2024-04-16T19:11:58.769 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T19:11:58.770 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi --archive /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462 --owner scheduled_smanjara@teuthology --description rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi} -- /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/orig.config.yaml 2024-04-16T19:11:58.810 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T19:11:58.901 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462 branch: wip-shilpa-multisite-tests description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi} email: null first_in_suite: false job_id: '7658462' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-shilpa-multisite-tests ceph: conf: client: debug rgw: 20 debug rgw lifecycle: 20 debug rgw notification: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw data sync poll interval: 5 rgw md log max shards: 4 rgw meta sync poll interval: 5 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: b0c7b917267c471208541fdeb652f5e897d00086 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: b0c7b917267c471208541fdeb652f5e897d00086 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding - notification_v2 endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 rgw-multisite-tests: args: - tests.py extra_args: - -a !fails_on_rgw selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-shilpa-multisite-tests sha1: 9fbf23e1cbe3573736447d8751b80c9b42cadc24 owner: scheduled_smanjara@teuthology priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - - c1.mgr.x - c1.osd.3 - c1.osd.4 - c1.osd.5 - c1.client.1 - - c2.mon.a - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - - c2.mgr.x - c2.osd.3 - c2.osd.4 - c2.osd.5 - c2.client.1 sha1: b0c7b917267c471208541fdeb652f5e897d00086 sleep_before_teardown: 0 suite: rgw:multisite suite_branch: wip-shilpa-multisite-tests suite_path: /home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa suite_relpath: qa suite_repo: https://github.com/smanjara/ceph.git suite_sha1: 9fbf23e1cbe3573736447d8751b80c9b42cadc24 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFeEZqyh9FanBgXANxWy0pvFnKNo2hNbUFcqc4b4YSTlDKxftRrW04MORnc+2d/6ALkEqGynnxTKwFB0BDqSt6E= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM0TlHyXmVZ7jqn/rZpGycELmAJqs913ebWmQ2A0/heI3vGrv35mMKLp+X4G0DUqcRgFGEnaTPo3DjdbG/3Sps= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBn36P5Qab1DiTDnz/Z06PhUxTXcAilRT0rHTDZBY3vuVopfxS/BCCXXESn35uqPsSYOEgNDMs9qS+wcQpSIaBQ= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCVR1+NFSII3eGCZ2LQqmYNr8DPxw8UcAqsFGaEiPTDm+W8eOyUOvkV+Q0w306QFGyg9qJh1CvoFhvHYvh5aac= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_19:05:08 tube: smithi user: smanjara verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T19:11:58.901 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa; will attempt to use it 2024-04-16T19:11:59.029 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks 2024-04-16T19:11:59.030 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T19:11:59.031 INFO:teuthology.task.internal:Checking packages... 2024-04-16T19:11:59.303 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b0c7b917267c471208541fdeb652f5e897d00086' 2024-04-16T19:11:59.304 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T19:11:59.304 INFO:teuthology.packaging:ref: None 2024-04-16T19:11:59.304 INFO:teuthology.packaging:tag: None 2024-04-16T19:11:59.304 INFO:teuthology.packaging:branch: wip-shilpa-multisite-tests 2024-04-16T19:11:59.304 INFO:teuthology.packaging:sha1: b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:11:59.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-shilpa-multisite-tests 2024-04-16T19:11:59.570 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3089.gb0c7b917 2024-04-16T19:11:59.572 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T19:11:59.579 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T19:11:59.579 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T19:11:59.587 INFO:teuthology.task.internal:Saving configuration 2024-04-16T19:11:59.602 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T19:11:59.609 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T19:11:59.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.798557', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFeEZqyh9FanBgXANxWy0pvFnKNo2hNbUFcqc4b4YSTlDKxftRrW04MORnc+2d/6ALkEqGynnxTKwFB0BDqSt6E='} 2024-04-16T19:11:59.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.799674', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM0TlHyXmVZ7jqn/rZpGycELmAJqs913ebWmQ2A0/heI3vGrv35mMKLp+X4G0DUqcRgFGEnaTPo3DjdbG/3Sps='} 2024-04-16T19:11:59.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.800802', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBn36P5Qab1DiTDnz/Z06PhUxTXcAilRT0rHTDZBY3vuVopfxS/BCCXXESn35uqPsSYOEgNDMs9qS+wcQpSIaBQ='} 2024-04-16T19:11:59.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.796882', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCVR1+NFSII3eGCZ2LQqmYNr8DPxw8UcAqsFGaEiPTDm+W8eOyUOvkV+Q0w306QFGyg9qJh1CvoFhvHYvh5aac='} 2024-04-16T19:11:59.700 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T19:11:59.707 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['c1.mon.a', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'] 2024-04-16T19:11:59.707 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['c1.mgr.x', 'c1.osd.3', 'c1.osd.4', 'c1.osd.5', 'c1.client.1'] 2024-04-16T19:11:59.707 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['c2.mon.a', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0'] 2024-04-16T19:11:59.707 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['c2.mgr.x', 'c2.osd.3', 'c2.osd.4', 'c2.osd.5', 'c2.client.1'] 2024-04-16T19:11:59.707 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T19:11:59.832 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f19449a3f70>, signals=[15]) 2024-04-16T19:11:59.833 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T19:11:59.840 INFO:teuthology.task.internal:Opening connections... 2024-04-16T19:11:59.840 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:11:59.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:11:59.954 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:11:59.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:12:00.046 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:12:00.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:12:00.133 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:12:00.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:12:00.223 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T19:12:00.231 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-04-16T19:12:00.249 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-04-16T19:12:00.249 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-04-16T19:12:00.307 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-04-16T19:12:00.307 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T19:12:00.308 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T19:12:00.309 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-04-16T19:12:00.336 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-16T19:12:00.353 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-16T19:12:00.354 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-16T19:12:00.410 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-04-16T19:12:00.410 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-04-16T19:12:00.410 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T19:12:00.411 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T19:12:00.411 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-16T19:12:00.435 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-04-16T19:12:00.453 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-04-16T19:12:00.453 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T19:12:00.509 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T19:12:00.510 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T19:12:00.510 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-04-16T19:12:00.543 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-04-16T19:12:00.561 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-04-16T19:12:00.561 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-04-16T19:12:00.618 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-04-16T19:12:00.618 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-04-16T19:12:00.618 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-04-16T19:12:00.618 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-04-16T19:12:00.618 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T19:12:00.619 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T19:12:00.619 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-04-16T19:12:00.652 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T19:12:00.662 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T19:12:00.669 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T19:12:00.669 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-04-16T19:12:00.672 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-16T19:12:00.674 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-04-16T19:12:00.676 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-04-16T19:12:00.693 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T19:12:00.702 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T19:12:00.702 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-04-16T19:12:00.729 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-16T19:12:00.732 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-04-16T19:12:00.734 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-04-16T19:12:00.840 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T19:12:00.938 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T19:12:00.970 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T19:12:00.970 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T19:12:00.970 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T19:12:00.971 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T19:12:00.971 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T19:12:00.972 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T19:12:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:12:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:12:00.989 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-16T19:12:00.990 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-04-16T19:12:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:12:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:12:00.991 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-04-16T19:12:00.992 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-04-16T19:12:01.046 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-361.el9.x86_64 2024-04-16T19:12:01.046 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-361.el9.x86_64 2024-04-16T19:12:01.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-04-16T19:12:01.047 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-04-16T19:12:01.047 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-04-16T19:12:01.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-04-16T19:12:01.048 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-04-16T19:12:01.049 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-04-16T19:12:01.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-04-16T19:12:01.050 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-04-16T19:12:01.050 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-361.el9.x86_64 2024-04-16T19:12:01.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-04-16T19:12:03.575 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 8.1 MB/s | 6.7 MB 00:00 2024-04-16T19:12:03.580 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.7 MB 00:00 2024-04-16T19:12:03.625 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-04-16T19:12:03.755 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.7 MB 00:01 2024-04-16T19:12:05.133 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-04-16T19:12:05.239 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-04-16T19:12:05.317 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-04-16T19:12:05.455 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-04-16T19:12:06.238 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-16T19:12:06.498 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-04-16T19:12:06.591 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.0 MB 00:00 2024-04-16T19:12:06.629 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-16T19:12:08.455 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-16T19:12:09.167 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-16T19:12:12.782 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 19 MB 00:05 2024-04-16T19:12:13.600 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.2 MB 00:00 2024-04-16T19:12:15.201 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-16T19:12:16.605 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 6.2 MB 00:04 2024-04-16T19:12:17.564 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 21 MB 00:02 2024-04-16T19:12:18.072 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-16T19:12:19.883 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T19:12:20.541 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.2 MB 00:04 2024-04-16T19:12:22.050 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-16T19:12:22.703 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-16T19:12:23.460 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-16T19:12:24.949 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-16T19:12:25.060 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T19:12:25.060 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T19:12:25.165 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:12:25.172 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:12:25.172 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:12:25.172 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 77 M 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-04-16T19:12:25.173 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:12:25.354 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-16T19:12:26.062 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-04-16T19:12:26.888 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-04-16T19:12:27.339 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T19:12:27.339 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T19:12:27.379 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 14 MB/s | 28 MB 00:02 2024-04-16T19:12:27.382 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:12:27.383 INFO:teuthology.orchestra.run.smithi151.stdout:Total 35 MB/s | 77 MB 00:02 2024-04-16T19:12:27.383 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:12:27.433 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:12:27.444 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:12:27.444 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-04-16T19:12:27.445 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:12:27.446 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 77 M 2024-04-16T19:12:27.446 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-04-16T19:12:27.446 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:12:27.616 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.6 MB/s | 277 kB 00:00 2024-04-16T19:12:27.766 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:12:27.766 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:12:28.317 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-04-16T19:12:28.602 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:12:28.602 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:12:28.660 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-04-16T19:12:29.242 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-04-16T19:12:29.950 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-16T19:12:29.954 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:12:29.954 INFO:teuthology.orchestra.run.smithi069.stdout:Total 31 MB/s | 77 MB 00:02 2024-04-16T19:12:29.954 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:12:30.303 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 850 kB/s | 19 MB 00:22 2024-04-16T19:12:30.338 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:12:30.338 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:12:30.923 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:12:31.141 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T19:12:31.142 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T19:12:31.192 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:12:31.193 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:12:31.233 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:12:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:12:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:12:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:12:31.243 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:12:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-04-16T19:12:31.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:12:31.245 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 77 M 2024-04-16T19:12:31.245 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 123 M 2024-04-16T19:12:31.245 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:12:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:12:31.561 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 878 kB/s | 277 kB 00:00 2024-04-16T19:12:31.613 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:33.481 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:12:34.061 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:12:34.151 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:34.362 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:34.499 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:34.936 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-modules-core-5.14.0-438.el9.x86_6 8.2 MB/s | 28 MB 00:03 2024-04-16T19:12:35.086 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-core-5.14.0-438.el9.x86_64.rpm 4.0 MB/s | 15 MB 00:03 2024-04-16T19:12:35.520 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 7.9 MB/s | 34 MB 00:04 2024-04-16T19:12:35.523 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:12:35.523 INFO:teuthology.orchestra.run.smithi026.stdout:Total 18 MB/s | 77 MB 00:04 2024-04-16T19:12:35.523 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:12:35.980 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:12:35.980 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:12:36.814 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:12:36.815 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:12:36.902 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:37.118 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:39.396 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:12:40.031 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:12:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:41.769 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 804 kB/s | 6.2 MB 00:07 2024-04-16T19:12:41.794 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:41.892 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:43.002 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:12:43.138 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:43.336 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-16T19:12:44.041 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:44.530 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-16T19:12:46.617 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:48.220 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:12:48.270 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:49.729 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-16T19:12:52.111 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T19:12:52.111 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T19:12:52.192 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T19:12:52.202 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 77 M 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 123 M 2024-04-16T19:12:52.203 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:12:52.364 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-16T19:12:52.948 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-16T19:12:53.370 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:12:53.948 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-04-16T19:12:54.298 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 14 MB/s | 28 MB 00:01 2024-04-16T19:12:54.301 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:12:54.301 INFO:teuthology.orchestra.run.smithi119.stdout:Total 37 MB/s | 77 MB 00:02 2024-04-16T19:12:54.301 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:12:54.690 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:12:54.690 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:12:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:12:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:12:57.829 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:12:58.433 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:12:58.517 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:01.260 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:01.420 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:06.165 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:06.294 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:11.060 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:13.262 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:13.262 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T19:13:13.262 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:13:13.307 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:13.665 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:13.665 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:13:13.665 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:13.665 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:13:14.092 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:13:14.422 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:14.525 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.526 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:14.526 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T19:13:14.526 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T19:13:14.526 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T19:13:14.526 INFO:teuthology.task.kernel:Installing distro kernel on c2.mgr.x... 2024-04-16T19:13:14.527 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T19:13:14.527 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:14.606 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:14.607 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:14.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-04-16T19:13:14.640 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-04-16T19:13:14.640 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-04-16T19:13:14.640 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T19:13:14.641 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:13:14.642 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T19:13:14.643 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T19:13:14.643 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-438.el9.x86_64 2024-04-16T19:13:14.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T19:13:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-04-16T19:13:16.194 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:16.195 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T19:13:16.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:13:16.239 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:13:16.591 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:16.592 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:16.615 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T19:13:16.639 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-04-16T19:13:16.642 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-04-16T19:13:16.657 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.tz48NdDX99 2024-04-16T19:13:16.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tz48NdDX99 2024-04-16T19:13:16.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.tz48NdDX99 2024-04-16T19:13:16.867 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.tz48NdDX99 is 6KB 2024-04-16T19:13:16.916 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.tz48NdDX99 2024-04-16T19:13:16.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T19:13:16.996 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T19:13:16.996 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T19:13:16.996 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T19:13:16.996 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T19:13:16.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.028 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.029 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.029 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:13:17.029 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:13:17.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-04-16T19:13:17.373 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:17.477 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:17.478 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T19:13:17.478 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T19:13:17.478 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T19:13:17.478 INFO:teuthology.task.kernel:Installing distro kernel on c1.mgr.x... 2024-04-16T19:13:17.478 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T19:13:17.478 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:17.559 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:17.559 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:17.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T19:13:17.593 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:13:17.594 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T19:13:17.595 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T19:13:17.595 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-438.el9.x86_64 2024-04-16T19:13:17.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T19:13:18.216 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-04-16T19:13:19.597 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T19:13:19.628 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-04-16T19:13:19.630 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-16T19:13:19.646 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.p4AE2Zp54T 2024-04-16T19:13:19.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p4AE2Zp54T 2024-04-16T19:13:19.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.p4AE2Zp54T 2024-04-16T19:13:19.851 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.p4AE2Zp54T is 6KB 2024-04-16T19:13:19.899 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.p4AE2Zp54T 2024-04-16T19:13:19.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T19:13:19.977 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T19:13:19.977 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T19:13:19.977 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T19:13:19.977 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T19:13:19.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T19:13:20.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-04-16T19:13:24.291 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:24.292 INFO:teuthology.orchestra.run.smithi026.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T19:13:24.292 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:13:24.336 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:13:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:24.770 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:13:25.201 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:13:25.508 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:25.617 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.617 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:25.617 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T19:13:25.617 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T19:13:25.617 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T19:13:25.617 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-04-16T19:13:25.617 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T19:13:25.617 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:25.700 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:25.700 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:25.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-04-16T19:13:25.741 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-04-16T19:13:25.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:13:25.743 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T19:13:25.743 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T19:13:25.743 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-438.el9.x86_64 2024-04-16T19:13:25.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T19:13:26.374 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-04-16T19:13:27.776 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T19:13:27.814 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-04-16T19:13:27.815 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-04-16T19:13:27.830 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.yQd9RTW89u 2024-04-16T19:13:27.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yQd9RTW89u 2024-04-16T19:13:27.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.yQd9RTW89u 2024-04-16T19:13:28.042 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.yQd9RTW89u is 6KB 2024-04-16T19:13:28.090 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.yQd9RTW89u 2024-04-16T19:13:28.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T19:13:28.173 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T19:13:28.173 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T19:13:28.173 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T19:13:28.173 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T19:13:28.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T19:13:28.319 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-04-16T19:13:40.489 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:40.489 INFO:teuthology.orchestra.run.smithi119.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T19:13:40.489 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:13:40.531 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T19:13:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T19:13:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T19:13:41.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T19:13:41.336 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:13:41.337 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:13:41.668 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:41.773 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:41.774 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T19:13:41.774 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T19:13:41.774 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T19:13:41.775 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-04-16T19:13:41.775 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T19:13:41.775 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T19:13:41.855 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T19:13:41.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T19:13:41.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 68.el9 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8264254 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T19:13:41.900 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:13:41.901 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T19:13:41.902 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T19:13:41.902 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-438.el9.x86_64 2024-04-16T19:13:41.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T19:13:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-04-16T19:13:43.876 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T19:13:43.905 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-04-16T19:13:43.914 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-04-16T19:13:43.928 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.5MXKAhVEbY 2024-04-16T19:13:43.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5MXKAhVEbY 2024-04-16T19:13:43.998 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.5MXKAhVEbY 2024-04-16T19:13:44.133 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.5MXKAhVEbY is 6KB 2024-04-16T19:13:44.180 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.5MXKAhVEbY 2024-04-16T19:13:44.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T19:13:44.259 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T19:13:44.259 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T19:13:44.259 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T19:13:44.259 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T19:13:44.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T19:13:44.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-04-16T19:13:47.136 INFO:teuthology.task.kernel:Checking client c2.mgr.x for new kernel version... 2024-04-16T19:13:47.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:13:47.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:13:50.122 INFO:teuthology.task.kernel:Checking client c1.mgr.x for new kernel version... 2024-04-16T19:13:50.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:13:50.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:13:58.327 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-04-16T19:13:58.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:13:58.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:05.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-04-16T19:14:08.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-04-16T19:14:14.419 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-04-16T19:14:14.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:14:14.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:14.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:14:14.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:16.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-04-16T19:14:17.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-04-16T19:14:17.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:14:17.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:20.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-04-16T19:14:25.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:14:25.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:29.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:14:29.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:32.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:14:32.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:32.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-04-16T19:14:41.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:14:41.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:44.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-04-16T19:14:44.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-04-16T19:14:56.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:14:56.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:56.595 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-04-16T19:14:56.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:14:56.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:14:56.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:14:56.994 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T19:14:56.994 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-04-16T19:14:57.008 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-438.el9.x86_64 2024-04-16T19:14:57.008 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T19:14:57.008 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T19:14:57.008 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T19:14:58.009 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-16T19:14:58.009 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T19:14:58.067 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-04-16T19:14:58.108 DEBUG:teuthology.parallel:result is None 2024-04-16T19:15:04.539 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-16T19:15:04.963 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:15:04.963 INFO:teuthology.task.kernel:Checking kernel version of c1.mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T19:15:04.963 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-16T19:15:04.986 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-438.el9.x86_64 2024-04-16T19:15:04.986 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T19:15:04.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T19:15:04.986 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T19:15:05.987 INFO:teuthology.task.kernel:Enabling kdb on c1.mgr.x... 2024-04-16T19:15:06.134 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T19:15:06.243 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-04-16T19:15:06.292 DEBUG:teuthology.parallel:result is None 2024-04-16T19:15:15.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-04-16T19:15:29.542 DEBUG:teuthology.orchestra.remote:timed out 2024-04-16T19:15:30.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:15:30.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:15:30.820 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-16T19:15:31.211 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:15:31.211 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T19:15:31.212 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-04-16T19:15:31.226 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-438.el9.x86_64 2024-04-16T19:15:31.226 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T19:15:31.226 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T19:15:31.226 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T19:15:32.227 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-16T19:15:32.228 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T19:15:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-04-16T19:15:32.363 DEBUG:teuthology.parallel:result is None 2024-04-16T19:15:44.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:15:44.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:15:45.075 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-04-16T19:15:45.448 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:15:45.448 INFO:teuthology.task.kernel:Checking kernel version of c2.mgr.x, want "5.14.0-438.el9.x86_64"... 2024-04-16T19:15:45.449 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-04-16T19:15:45.465 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-438.el9.x86_64 2024-04-16T19:15:45.465 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T19:15:45.465 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T19:15:45.465 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T19:15:46.466 INFO:teuthology.task.kernel:Enabling kdb on c2.mgr.x... 2024-04-16T19:15:46.466 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T19:15:46.541 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-04-16T19:15:46.572 DEBUG:teuthology.parallel:result is None 2024-04-16T19:15:46.573 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T19:15:46.582 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T19:15:46.582 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T19:15:46.585 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T19:15:46.588 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T19:15:46.591 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T19:15:46.609 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T19:15:46.627 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T19:15:46.636 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T19:15:46.636 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T19:15:46.644 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T19:15:46.647 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T19:15:46.650 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T19:15:46.739 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T19:15:46.748 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T19:15:46.748 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T19:15:46.750 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T19:15:46.753 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T19:15:46.755 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T19:15:46.787 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.793 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.796 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.802 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.802 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.808 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.811 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.818 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T19:15:46.820 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T19:15:46.833 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T19:15:46.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T19:15:46.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T19:15:46.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T19:15:46.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T19:15:46.904 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T19:15:46.922 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T19:15:46.922 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T19:15:46.930 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T19:15:46.938 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T19:15:46.944 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T19:15:46.967 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.092 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.222 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:15:47.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T19:15:47.292 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.338 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.396 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.525 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:15:47.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T19:15:47.595 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.693 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:47.822 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:15:47.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T19:15:47.892 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T19:15:47.996 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:48.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T19:15:48.127 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:15:48.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T19:15:48.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-04-16T19:15:48.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-16T19:15:48.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-04-16T19:15:48.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-04-16T19:15:48.252 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T19:15:48.253 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T19:15:48.257 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T19:15:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T19:15:48.705 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T19:15:48.713 INFO:teuthology.task.internal:Starting timer... 2024-04-16T19:15:48.713 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T19:15:48.743 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T19:15:48.753 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T19:15:48.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-04-16T19:15:48.820 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-04-16T19:15:48.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-04-16T19:15:48.900 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-04-16T19:15:48.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-04-16T19:15:48.957 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-04-16T19:15:48.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-04-16T19:15:49.028 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-04-16T19:15:49.031 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T19:15:49.031 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-04-16T19:15:49.056 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-04-16T19:15:49.056 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-04-16T19:15:49.079 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-04-16T19:15:49.079 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-04-16T19:15:49.103 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-04-16T19:15:49.104 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-04-16T19:15:49.124 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-04-16T19:15:49.124 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2024-04-16T19:15:49.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T19:15:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:15:49.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T19:15:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:15:49.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T19:15:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:15:49.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T19:15:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T19:15:49.249 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T19:15:49.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-04-16T19:15:49.283 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-04-16T19:15:49.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-04-16T19:15:49.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-04-16T19:15:49.390 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T19:15:49.508 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T19:15:49.807 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T19:15:55.157 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T19:15:56.357 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T19:15:56.453 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi069.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T19:20:45.074 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-04-16T19:20:45.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:20:45.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:20:45.170 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-04-16T19:20:45.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-04-16T19:20:45.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:20:45.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:20:45.342 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-16T19:20:45.424 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-16T19:20:45.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:20:45.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:20:45.509 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-16T19:20:45.586 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-16T19:20:45.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:20:45.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T19:20:45.677 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-04-16T19:20:45.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-04-16T19:20:45.760 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T19:20:45.779 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T19:20:45.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T19:20:45.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T19:20:45.782 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T19:20:45.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T19:20:45.785 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T19:20:45.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T19:20:45.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T19:20:45.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T19:20:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T19:20:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T19:20:45.822 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T19:20:45.834 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T19:20:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T19:20:45.839 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T19:20:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T19:20:45.853 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T19:20:45.870 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-04-16T19:20:45.872 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-04-16T19:20:45.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-04-16T19:20:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-04-16T19:20:45.885 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-04-16T19:20:45.887 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-04-16T19:20:45.889 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-04-16T19:20:45.899 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-04-16T19:20:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T19:20:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T19:20:45.908 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T19:20:45.917 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T19:20:45.921 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T19:20:45.922 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T19:20:45.926 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T19:20:45.934 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T19:20:45.976 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-16T19:20:45.979 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.980 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.984 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-16T19:20:45.987 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T19:20:45.987 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-16T19:20:45.988 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.988 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.988 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.988 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.991 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-04-16T19:20:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T19:20:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-04-16T19:20:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T19:20:45.996 INFO:teuthology.run_tasks:Running task install... 2024-04-16T19:20:46.004 DEBUG:teuthology.task.install:project ceph 2024-04-16T19:20:46.004 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}} 2024-04-16T19:20:46.004 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'} 2024-04-16T19:20:46.004 INFO:teuthology.task.install:Using flavor: default 2024-04-16T19:20:46.085 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T19:20:46.085 INFO:teuthology.task.install:extra packages: [] 2024-04-16T19:20:46.086 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'tag': None, 'wait_for_package': False} 2024-04-16T19:20:46.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:20:46.088 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'tag': None, 'wait_for_package': False} 2024-04-16T19:20:46.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:20:46.089 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'tag': None, 'wait_for_package': False} 2024-04-16T19:20:46.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:20:46.090 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'tag': None, 'wait_for_package': False} 2024-04-16T19:20:46.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:20:46.237 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/ 2024-04-16T19:20:46.237 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3089.gb0c7b917 2024-04-16T19:20:46.242 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/ 2024-04-16T19:20:46.243 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3089.gb0c7b917 2024-04-16T19:20:46.354 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T19:20:46.355 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:20:46.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T19:20:46.368 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/ 2024-04-16T19:20:46.368 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3089.gb0c7b917 2024-04-16T19:20:46.376 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/ 2024-04-16T19:20:46.376 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3089.gb0c7b917 2024-04-16T19:20:46.385 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T19:20:46.385 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0c7b917267c471208541fdeb652f5e897d00086/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T19:20:46.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T19:20:46.461 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T19:20:46.461 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:20:46.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T19:20:46.481 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T19:20:46.481 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:20:46.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T19:20:46.492 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T19:20:46.492 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0c7b917267c471208541fdeb652f5e897d00086/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T19:20:46.513 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T19:20:46.513 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0c7b917267c471208541fdeb652f5e897d00086/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T19:20:46.537 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T19:20:46.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T19:20:46.573 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-shilpa-multisite-tests/b0c7b917267c471208541fdeb652f5e897d00086/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T19:20:46.573 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:20:46.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T19:20:46.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T19:20:46.605 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T19:20:46.605 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0c7b917267c471208541fdeb652f5e897d00086/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T19:20:46.606 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-04-16T19:20:46.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-04-16T19:20:46.645 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T19:20:46.667 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T19:20:46.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T19:20:46.713 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-04-16T19:20:46.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-04-16T19:20:46.736 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-04-16T19:20:46.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-04-16T19:20:46.760 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T19:20:46.829 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-04-16T19:20:46.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-04-16T19:20:46.860 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-04-16T19:20:46.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-04-16T19:20:46.959 INFO:teuthology.orchestra.run.smithi069.stdout:74 files removed 2024-04-16T19:20:46.983 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-04-16T19:20:46.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2024-04-16T19:20:47.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-04-16T19:20:47.085 INFO:teuthology.orchestra.run.smithi026.stdout:74 files removed 2024-04-16T19:20:47.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-04-16T19:20:47.670 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 174 kB/s | 83 kB 00:00 2024-04-16T19:20:47.686 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 220 kB/s | 83 kB 00:00 2024-04-16T19:20:47.692 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-04-16T19:20:47.771 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 240 kB/s | 83 kB 00:00 2024-04-16T19:20:48.018 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-04-16T19:20:48.037 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-16T19:20:48.196 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-16T19:20:48.422 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-16T19:20:48.669 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-16T19:20:48.788 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-04-16T19:20:49.154 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 8.1 kB/s | 12 kB 00:01 2024-04-16T19:20:49.266 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T19:20:49.432 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.7 MB 00:00 2024-04-16T19:20:49.446 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 9.1 MB/s | 6.7 MB 00:00 2024-04-16T19:20:49.504 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-16T19:20:50.131 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T19:20:50.875 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-04-16T19:20:50.924 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-04-16T19:20:51.096 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-04-16T19:20:51.632 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-04-16T19:20:51.889 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-16T19:20:52.098 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-16T19:20:52.679 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-04-16T19:20:54.257 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-16T19:20:55.291 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-04-16T19:20:56.457 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-04-16T19:20:58.709 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.2 MB 00:00 2024-04-16T19:20:59.484 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-16T19:21:00.716 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-04-16T19:21:00.830 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 929 kB/s | 8.0 MB 00:08 2024-04-16T19:21:01.653 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 9.3 kB/s | 16 kB 00:01 2024-04-16T19:21:02.652 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-16T19:21:03.680 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 5.4 kB/s | 16 kB 00:02 2024-04-16T19:21:04.539 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-04-16T19:21:04.714 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-16T19:21:04.853 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 6.0 kB/s | 16 kB 00:02 2024-04-16T19:21:06.063 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-16T19:21:07.652 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-04-16T19:21:09.049 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.2 MB 00:00 2024-04-16T19:21:10.014 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-16T19:21:10.390 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 11 M 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.4 M 2024-04-16T19:21:10.400 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 21 M 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 25 k 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T19:21:10.401 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 746 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 501 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:10.402 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 45 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 135 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 162 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 321 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 298 k 2024-04-16T19:21:10.403 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 99 k 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:10.404 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-04-16T19:21:10.405 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-04-16T19:21:10.405 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:10.405 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 60 M 2024-04-16T19:21:10.406 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:21:11.120 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-base-19.0.0-3089.gb0c7b917.el9.x86 9.6 MB/s | 5.4 MB 00:00 2024-04-16T19:21:11.178 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-selinux-19.0.0-3089.gb0c7b917.el9. 433 kB/s | 25 kB 00:00 2024-04-16T19:21:11.186 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-16T19:21:11.270 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): libcephfs2-19.0.0-3089.gb0c7b917.el9.x8 8.0 MB/s | 746 kB 00:00 2024-04-16T19:21:11.354 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libradosstriper1-19.0.0-3089.gb0c7b917. 5.9 MB/s | 501 kB 00:00 2024-04-16T19:21:11.416 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-16T19:21:11.829 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): librgw2-19.0.0-3089.gb0c7b917.el9.x86_6 11 MB/s | 5.4 MB 00:00 2024-04-16T19:21:11.888 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.0.0-3089.gb0c7 775 kB/s | 45 kB 00:00 2024-04-16T19:21:11.954 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): python3-ceph-common-19.0.0-3089.gb0c7b9 2.0 MB/s | 135 kB 00:00 2024-04-16T19:21:12.021 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): python3-cephfs-19.0.0-3089.gb0c7b917.el 2.4 MB/s | 162 kB 00:00 2024-04-16T19:21:12.096 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): python3-rados-19.0.0-3089.gb0c7b917.el9 4.2 MB/s | 321 kB 00:00 2024-04-16T19:21:12.172 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): python3-rbd-19.0.0-3089.gb0c7b917.el9. 3.9 MB/s | 298 kB 00:00 2024-04-16T19:21:12.238 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rgw-19.0.0-3089.gb0c7b917.el9. 1.5 MB/s | 99 kB 00:00 2024-04-16T19:21:12.381 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-04-16T19:21:12.565 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): ceph-common-19.0.0-3089.gb0c7b917.el9. 11 MB/s | 21 MB 00:02 2024-04-16T19:21:12.589 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 764 kB/s | 159 kB 00:00 2024-04-16T19:21:12.615 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 890 kB/s | 44 kB 00:00 2024-04-16T19:21:12.732 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): ceph-radosgw-19.0.0-3089.gb0c7b917.el9 4.9 MB/s | 11 MB 00:02 2024-04-16T19:21:12.757 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 661 kB 00:00 2024-04-16T19:21:12.782 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-04-16T19:21:12.807 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 556 kB/s | 41 kB 00:00 2024-04-16T19:21:12.833 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-04-16T19:21:12.866 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-04-16T19:21:12.891 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 553 kB/s | 46 kB 00:00 2024-04-16T19:21:12.947 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:21:12.960 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 11 M 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.4 M 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:12.961 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 21 M 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 25 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 746 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 501 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T19:21:12.962 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 45 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 135 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 162 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 321 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 298 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 99 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T19:21:12.963 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:Install 30 Packages 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 60 M 2024-04-16T19:21:12.964 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:21:13.083 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-04-16T19:21:13.151 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 974 kB/s | 309 kB 00:00 2024-04-16T19:21:13.176 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2024-04-16T19:21:13.209 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-16T19:21:13.284 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-16T19:21:13.359 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-04-16T19:21:13.401 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-04-16T19:21:13.485 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-16T19:21:13.660 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): librados2-19.0.0-3089.gb0c7b917.el9.x8 11 MB/s | 3.4 MB 00:00 2024-04-16T19:21:13.760 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): librbd1-19.0.0-3089.gb0c7b917.el9.x86_ 8.8 MB/s | 3.1 MB 00:00 2024-04-16T19:21:13.766 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:13.766 INFO:teuthology.orchestra.run.smithi151.stdout:Total 18 MB/s | 60 MB 00:03 2024-04-16T19:21:13.766 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:21:13.800 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:21:13.800 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:21:13.980 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 11 M 2024-04-16T19:21:13.987 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.4 M 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T19:21:13.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 21 M 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 25 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 746 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 501 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T19:21:13.989 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 45 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 135 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 162 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 321 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 298 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 99 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T19:21:13.990 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:Install 30 Packages 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 60 M 2024-04-16T19:21:13.991 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:21:14.094 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-04-16T19:21:14.107 INFO:teuthology.orchestra.run.smithi026.stdout:(1/32): ceph-base-19.0.0-3089.gb0c7b917.el9.x86 7.1 MB/s | 5.4 MB 00:00 2024-04-16T19:21:14.165 INFO:teuthology.orchestra.run.smithi026.stdout:(2/32): ceph-selinux-19.0.0-3089.gb0c7b917.el9. 432 kB/s | 25 kB 00:00 2024-04-16T19:21:14.257 INFO:teuthology.orchestra.run.smithi026.stdout:(3/32): libcephfs2-19.0.0-3089.gb0c7b917.el9.x8 8.0 MB/s | 746 kB 00:00 2024-04-16T19:21:14.284 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:21:14.284 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:21:14.333 INFO:teuthology.orchestra.run.smithi026.stdout:(4/32): libradosstriper1-19.0.0-3089.gb0c7b917. 6.5 MB/s | 501 kB 00:00 2024-04-16T19:21:14.733 INFO:teuthology.orchestra.run.smithi026.stdout:(5/32): librgw2-19.0.0-3089.gb0c7b917.el9.x86_6 13 MB/s | 5.4 MB 00:00 2024-04-16T19:21:14.800 INFO:teuthology.orchestra.run.smithi026.stdout:(6/32): python3-ceph-argparse-19.0.0-3089.gb0c7 678 kB/s | 45 kB 00:00 2024-04-16T19:21:14.866 INFO:teuthology.orchestra.run.smithi026.stdout:(7/32): python3-ceph-common-19.0.0-3089.gb0c7b9 2.0 MB/s | 135 kB 00:00 2024-04-16T19:21:14.903 INFO:teuthology.orchestra.run.smithi069.stdout:(1/32): ceph-base-19.0.0-3089.gb0c7b917.el9.x86 6.5 MB/s | 5.4 MB 00:00 2024-04-16T19:21:14.933 INFO:teuthology.orchestra.run.smithi026.stdout:(8/32): python3-cephfs-19.0.0-3089.gb0c7b917.el 2.4 MB/s | 162 kB 00:00 2024-04-16T19:21:14.945 INFO:teuthology.orchestra.run.smithi069.stdout:(2/32): ceph-selinux-19.0.0-3089.gb0c7b917.el9. 605 kB/s | 25 kB 00:00 2024-04-16T19:21:15.009 INFO:teuthology.orchestra.run.smithi026.stdout:(9/32): python3-rados-19.0.0-3089.gb0c7b917.el9 4.2 MB/s | 321 kB 00:00 2024-04-16T19:21:15.076 INFO:teuthology.orchestra.run.smithi026.stdout:(10/32): python3-rbd-19.0.0-3089.gb0c7b917.el9. 4.4 MB/s | 298 kB 00:00 2024-04-16T19:21:15.143 INFO:teuthology.orchestra.run.smithi026.stdout:(11/32): python3-rgw-19.0.0-3089.gb0c7b917.el9. 1.5 MB/s | 99 kB 00:00 2024-04-16T19:21:15.260 INFO:teuthology.orchestra.run.smithi026.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 907 kB/s | 106 kB 00:00 2024-04-16T19:21:15.262 INFO:teuthology.orchestra.run.smithi069.stdout:(3/32): libcephfs2-19.0.0-3089.gb0c7b917.el9.x8 2.3 MB/s | 746 kB 00:00 2024-04-16T19:21:15.318 INFO:teuthology.orchestra.run.smithi026.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-04-16T19:21:15.352 INFO:teuthology.orchestra.run.smithi026.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T19:21:15.362 INFO:teuthology.orchestra.run.smithi069.stdout:(4/32): libradosstriper1-19.0.0-3089.gb0c7b917. 4.9 MB/s | 501 kB 00:00 2024-04-16T19:21:15.435 INFO:teuthology.orchestra.run.smithi026.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-04-16T19:21:15.478 INFO:teuthology.orchestra.run.smithi026.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-16T19:21:15.511 INFO:teuthology.orchestra.run.smithi026.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T19:21:15.553 INFO:teuthology.orchestra.run.smithi026.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-04-16T19:21:15.587 INFO:teuthology.orchestra.run.smithi026.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T19:21:15.621 INFO:teuthology.orchestra.run.smithi026.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-04-16T19:21:15.832 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:21:15.888 INFO:teuthology.orchestra.run.smithi026.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-04-16T19:21:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T19:21:15.965 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-16T19:21:15.990 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-16T19:21:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:16.055 INFO:teuthology.orchestra.run.smithi069.stdout:(5/32): librgw2-19.0.0-3089.gb0c7b917.el9.x86_6 7.8 MB/s | 5.4 MB 00:00 2024-04-16T19:21:16.096 INFO:teuthology.orchestra.run.smithi069.stdout:(6/32): python3-ceph-argparse-19.0.0-3089.gb0c7 1.1 MB/s | 45 kB 00:00 2024-04-16T19:21:16.146 INFO:teuthology.orchestra.run.smithi026.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-16T19:21:16.147 INFO:teuthology.orchestra.run.smithi069.stdout:(7/32): python3-ceph-common-19.0.0-3089.gb0c7b9 2.6 MB/s | 135 kB 00:00 2024-04-16T19:21:16.205 INFO:teuthology.orchestra.run.smithi026.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-16T19:21:16.205 INFO:teuthology.orchestra.run.smithi069.stdout:(8/32): python3-cephfs-19.0.0-3089.gb0c7b917.el 2.7 MB/s | 162 kB 00:00 2024-04-16T19:21:16.232 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:16.264 INFO:teuthology.orchestra.run.smithi069.stdout:(9/32): python3-rados-19.0.0-3089.gb0c7b917.el9 5.4 MB/s | 321 kB 00:00 2024-04-16T19:21:16.291 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:16.314 INFO:teuthology.orchestra.run.smithi069.stdout:(10/32): python3-rbd-19.0.0-3089.gb0c7b917.el9. 5.8 MB/s | 298 kB 00:00 2024-04-16T19:21:16.355 INFO:teuthology.orchestra.run.smithi026.stdout:(24/32): ceph-radosgw-19.0.0-3089.gb0c7b917.el9 3.5 MB/s | 11 MB 00:03 2024-04-16T19:21:16.365 INFO:teuthology.orchestra.run.smithi069.stdout:(11/32): python3-rgw-19.0.0-3089.gb0c7b917.el9. 2.0 MB/s | 99 kB 00:00 2024-04-16T19:21:16.372 INFO:teuthology.orchestra.run.smithi026.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-04-16T19:21:16.472 INFO:teuthology.orchestra.run.smithi026.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-16T19:21:16.473 INFO:teuthology.orchestra.run.smithi069.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-04-16T19:21:16.515 INFO:teuthology.orchestra.run.smithi069.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-04-16T19:21:16.531 INFO:teuthology.orchestra.run.smithi026.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-16T19:21:16.549 INFO:teuthology.orchestra.run.smithi069.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-16T19:21:16.556 INFO:teuthology.orchestra.run.smithi026.stdout:(28/32): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-04-16T19:21:16.615 INFO:teuthology.orchestra.run.smithi026.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-16T19:21:16.616 INFO:teuthology.orchestra.run.smithi069.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-16T19:21:16.657 INFO:teuthology.orchestra.run.smithi069.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-16T19:21:16.661 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-16T19:21:16.691 INFO:teuthology.orchestra.run.smithi069.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-04-16T19:21:16.701 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:16.733 INFO:teuthology.orchestra.run.smithi069.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-04-16T19:21:16.767 INFO:teuthology.orchestra.run.smithi069.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-04-16T19:21:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:16.800 INFO:teuthology.orchestra.run.smithi069.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-04-16T19:21:16.881 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 8/34 2024-04-16T19:21:16.918 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:16.981 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:16.990 INFO:teuthology.orchestra.run.smithi026.stdout:(30/32): librados2-19.0.0-3089.gb0c7b917.el9.x8 7.8 MB/s | 3.4 MB 00:00 2024-04-16T19:21:17.010 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-16T19:21:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 11/34 2024-04-16T19:21:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:17.209 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:17.224 INFO:teuthology.orchestra.run.smithi026.stdout:(31/32): librbd1-19.0.0-3089.gb0c7b917.el9.x86_ 5.2 MB/s | 3.1 MB 00:00 2024-04-16T19:21:17.244 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-16T19:21:17.259 INFO:teuthology.orchestra.run.smithi069.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 674 kB/s | 309 kB 00:00 2024-04-16T19:21:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-16T19:21:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-16T19:21:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-16T19:21:17.468 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-16T19:21:17.497 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-16T19:21:17.530 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-16T19:21:17.642 INFO:teuthology.orchestra.run.smithi069.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-16T19:21:17.734 INFO:teuthology.orchestra.run.smithi069.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-16T19:21:17.826 INFO:teuthology.orchestra.run.smithi069.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-16T19:21:17.851 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-16T19:21:17.884 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:17.907 INFO:teuthology.orchestra.run.smithi026.stdout:(32/32): ceph-common-19.0.0-3089.gb0c7b917.el9. 4.6 MB/s | 21 MB 00:04 2024-04-16T19:21:17.911 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:17.911 INFO:teuthology.orchestra.run.smithi026.stdout:Total 12 MB/s | 60 MB 00:04 2024-04-16T19:21:17.912 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:21:17.927 INFO:teuthology.orchestra.run.smithi069.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-16T19:21:17.955 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:21:17.955 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:21:17.957 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:17.989 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 23/34 2024-04-16T19:21:18.033 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T19:21:18.035 INFO:teuthology.orchestra.run.smithi069.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-16T19:21:18.136 INFO:teuthology.orchestra.run.smithi069.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-16T19:21:18.370 INFO:teuthology.orchestra.run.smithi069.stdout:(28/32): ceph-common-19.0.0-3089.gb0c7b917.el9. 4.9 MB/s | 21 MB 00:04 2024-04-16T19:21:18.465 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:21:18.465 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:21:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T19:21:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:18.617 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:18.678 INFO:teuthology.orchestra.run.smithi069.stdout:(29/32): librados2-19.0.0-3089.gb0c7b917.el9.x8 11 MB/s | 3.4 MB 00:00 2024-04-16T19:21:18.684 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 27/34 2024-04-16T19:21:18.714 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 28/34 2024-04-16T19:21:18.879 INFO:teuthology.orchestra.run.smithi069.stdout:(30/32): librbd1-19.0.0-3089.gb0c7b917.el9.x86_ 16 MB/s | 3.1 MB 00:00 2024-04-16T19:21:18.920 INFO:teuthology.orchestra.run.smithi069.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-04-16T19:21:19.084 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-16T19:21:19.354 INFO:teuthology.orchestra.run.smithi069.stdout:(32/32): ceph-radosgw-19.0.0-3089.gb0c7b917.el9 2.0 MB/s | 11 MB 00:05 2024-04-16T19:21:19.357 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:19.358 INFO:teuthology.orchestra.run.smithi069.stdout:Total 11 MB/s | 60 MB 00:05 2024-04-16T19:21:19.358 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:21:19.396 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:21:19.396 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:21:19.904 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:21:19.905 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:21:20.128 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:21:20.220 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T19:21:20.245 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-16T19:21:20.279 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-16T19:21:20.302 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:20.318 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:20.512 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:20.574 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:20.918 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-16T19:21:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:20.980 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:21.056 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:21.146 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 8/34 2024-04-16T19:21:21.173 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:21.219 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-16T19:21:21.293 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 11/34 2024-04-16T19:21:21.363 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:21.392 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:21.393 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:21:21.456 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:21.482 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T19:21:21.491 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-16T19:21:21.517 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-16T19:21:21.542 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-16T19:21:21.565 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:21.582 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:21.601 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-16T19:21:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:21.642 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-16T19:21:21.647 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:21.706 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-16T19:21:21.725 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:21.725 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T19:21:21.726 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T19:21:21.726 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:21.740 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-16T19:21:21.750 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:21.757 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:21.769 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-16T19:21:21.802 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-16T19:21:21.821 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:21.887 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 11 M 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.4 M 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options 2024-04-16T19:21:21.902 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 21 M 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 25 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 746 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 501 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T19:21:21.903 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 5.4 M 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 45 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 135 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 162 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 321 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 298 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 99 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T19:21:21.904 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout:Install 30 Packages 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-04-16T19:21:21.905 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:21.906 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 60 M 2024-04-16T19:21:21.906 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:21:22.131 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-16T19:21:22.165 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:22.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-16T19:21:22.255 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:22.260 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:22.286 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 23/34 2024-04-16T19:21:22.334 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T19:21:22.355 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:22.441 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 8/34 2024-04-16T19:21:22.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:22.541 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:22.570 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-16T19:21:22.614 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 11/34 2024-04-16T19:21:22.683 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:22.714 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:22.761 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:22.804 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-16T19:21:22.836 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T19:21:22.867 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:22.914 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-16T19:21:22.922 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:22.955 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-16T19:21:22.992 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 27/34 2024-04-16T19:21:22.996 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-16T19:21:23.028 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-16T19:21:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 28/34 2024-04-16T19:21:23.057 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-16T19:21:23.090 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-16T19:21:23.402 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-16T19:21:23.436 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:23.527 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:23.558 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 23/34 2024-04-16T19:21:23.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T19:21:23.818 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-base-19.0.0-3089.gb0c7b917.el9.x86 3.1 MB/s | 5.4 MB 00:01 2024-04-16T19:21:24.018 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-common-19.0.0-3089.gb0c7b917.el9.x 11 MB/s | 21 MB 00:01 2024-04-16T19:21:24.035 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): ceph-selinux-19.0.0-3089.gb0c7b917.el9. 116 kB/s | 25 kB 00:00 2024-04-16T19:21:24.070 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T19:21:24.097 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:24.102 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): libcephfs2-19.0.0-3089.gb0c7b917.el9.x8 8.7 MB/s | 746 kB 00:00 2024-04-16T19:21:24.168 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:24.177 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): libradosstriper1-19.0.0-3089.gb0c7b917. 3.4 MB/s | 501 kB 00:00 2024-04-16T19:21:24.236 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 27/34 2024-04-16T19:21:24.253 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): python3-ceph-argparse-19.0.0-3089.gb0c7 601 kB/s | 45 kB 00:00 2024-04-16T19:21:24.266 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 28/34 2024-04-16T19:21:24.361 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-ceph-common-19.0.0-3089.gb0c7b9 1.2 MB/s | 135 kB 00:00 2024-04-16T19:21:24.445 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): librgw2-19.0.0-3089.gb0c7b917.el9.x86_6 16 MB/s | 5.4 MB 00:00 2024-04-16T19:21:24.554 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): ceph-radosgw-19.0.0-3089.gb0c7b917.el9. 4.3 MB/s | 11 MB 00:02 2024-04-16T19:21:24.579 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): python3-cephfs-19.0.0-3089.gb0c7b917.e 745 kB/s | 162 kB 00:00 2024-04-16T19:21:24.613 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-rados-19.0.0-3089.gb0c7b917.el 1.9 MB/s | 321 kB 00:00 2024-04-16T19:21:24.638 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): python3-rbd-19.0.0-3089.gb0c7b917.el9. 3.5 MB/s | 298 kB 00:00 2024-04-16T19:21:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): python3-rgw-19.0.0-3089.gb0c7b917.el9. 910 kB/s | 99 kB 00:00 2024-04-16T19:21:24.747 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 791 kB/s | 106 kB 00:00 2024-04-16T19:21:24.772 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-04-16T19:21:24.806 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-04-16T19:21:24.831 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-04-16T19:21:24.856 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-04-16T19:21:24.881 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 550 kB/s | 41 kB 00:00 2024-04-16T19:21:24.907 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-04-16T19:21:24.932 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-16T19:21:24.957 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 614 kB/s | 46 kB 00:00 2024-04-16T19:21:25.124 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-04-16T19:21:25.249 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-04-16T19:21:25.275 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 841 kB/s | 309 kB 00:00 2024-04-16T19:21:25.367 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-04-16T19:21:25.406 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:25.442 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-16T19:21:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:25.509 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-04-16T19:21:25.534 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-16T19:21:25.676 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-04-16T19:21:25.793 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): librbd1-19.0.0-3089.gb0c7b917.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-04-16T19:21:26.101 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): librados2-19.0.0-3089.gb0c7b917.el9.x8 5.7 MB/s | 3.4 MB 00:00 2024-04-16T19:21:26.106 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:26.106 INFO:teuthology.orchestra.run.smithi119.stdout:Total 14 MB/s | 60 MB 00:04 2024-04-16T19:21:26.106 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:21:26.133 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:26.140 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:21:26.140 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:21:26.178 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:26.256 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T19:21:26.256 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T19:21:26.256 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:26.544 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:26.588 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:26.626 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:21:26.626 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:21:27.209 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:27.259 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:27.353 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:27.354 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T19:21:27.354 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T19:21:27.354 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:27.402 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:21:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T19:21:28.189 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-16T19:21:28.215 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-16T19:21:28.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:28.256 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-16T19:21:28.397 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:28.450 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-16T19:21:28.834 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-16T19:21:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:28.967 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:29.053 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 8/34 2024-04-16T19:21:29.076 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 9/34 2024-04-16T19:21:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-16T19:21:29.178 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 11/34 2024-04-16T19:21:29.246 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:29.270 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:29.326 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 13/34 2024-04-16T19:21:29.360 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-16T19:21:29.467 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-16T19:21:29.495 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-16T19:21:29.535 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-16T19:21:29.560 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-16T19:21:29.589 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-16T19:21:29.621 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-16T19:21:29.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-16T19:21:29.960 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:30.050 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 22/34 2024-04-16T19:21:30.081 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 23/34 2024-04-16T19:21:30.119 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T19:21:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T19:21:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:30.692 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 26/34 2024-04-16T19:21:30.760 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 27/34 2024-04-16T19:21:30.790 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 28/34 2024-04-16T19:21:31.449 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:31.449 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-04-16T19:21:31.450 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:32.464 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:32.465 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T19:21:32.465 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:32.465 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:32.550 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:32.551 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:33.136 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:33.178 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:33.726 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 6/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:33.727 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 8/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 9/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 10/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 11/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 13/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T19:21:33.728 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T19:21:33.729 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T19:21:33.730 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/34 2024-04-16T19:21:33.755 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 29/34 2024-04-16T19:21:33.790 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:33.883 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 30/34 2024-04-16T19:21:33.883 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T19:21:33.884 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T19:21:33.884 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:33.916 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:21:34.790 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T19:21:34.791 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.792 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:34.793 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:21:35.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-04-16T19:21:35.646 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:28 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:21:35.806 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 49 M 2024-04-16T19:21:35.807 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 164 k 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-04-16T19:21:35.808 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:35.809 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 50 M 2024-04-16T19:21:35.809 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 212 M 2024-04-16T19:21:35.809 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:21:36.494 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:36.494 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-04-16T19:21:36.495 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:36.825 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:36.825 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-04-16T19:21:36.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:37.492 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:37.517 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:37.519 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:37.602 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:37.602 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:37.817 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:37.843 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:37.845 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:37.911 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:37.912 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:38.165 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-04-16T19:21:38.215 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-16T19:21:38.241 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-04-16T19:21:38.283 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): libcephsqlite-19.0.0-3089.gb0c7b917.el9. 611 kB/s | 164 kB 00:00 2024-04-16T19:21:38.408 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-16T19:21:38.743 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 6/34 2024-04-16T19:21:38.744 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 8/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 9/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 10/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 11/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 13/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T19:21:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T19:21:38.746 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T19:21:38.747 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T19:21:38.748 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/34 2024-04-16T19:21:39.029 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:39.029 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/34 2024-04-16T19:21:39.029 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/34 2024-04-16T19:21:39.029 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/34 2024-04-16T19:21:39.029 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 6/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 8/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 9/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 10/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 11/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:39.030 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 13/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T19:21:39.031 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T19:21:39.032 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/34 2024-04-16T19:21:39.928 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:39.928 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:39.928 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-04-16T19:21:39.928 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.928 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T19:21:39.929 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T19:21:39.930 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T19:21:39.931 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T19:21:39.932 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T19:21:39.932 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T19:21:39.932 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T19:21:39.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:39.932 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:21:40.097 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.098 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.099 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.100 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:40.101 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:21:40.263 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-04-16T19:21:40.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2024-04-16T19:21:40.817 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): ceph-test-19.0.0-3089.gb0c7b917.el9.x86_ 18 MB/s | 49 MB 00:02 2024-04-16T19:21:40.820 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:40.820 INFO:teuthology.orchestra.run.smithi151.stdout:Total 10 MB/s | 50 MB 00:05 2024-04-16T19:21:40.835 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:21:40.857 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:21:40.857 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:21:40.889 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:31 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:21:41.034 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:30 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:21:41.057 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:21:41.058 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 49 M 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 164 k 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T19:21:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T19:21:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:21:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:21:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-04-16T19:21:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 50 M 2024-04-16T19:21:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 212 M 2024-04-16T19:21:41.061 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:21:41.194 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 49 M 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T19:21:41.195 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 164 k 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-04-16T19:21:41.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 50 M 2024-04-16T19:21:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 212 M 2024-04-16T19:21:41.197 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:21:41.245 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:21:41.246 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:21:41.432 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-16T19:21:41.474 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-16T19:21:41.497 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-16T19:21:41.499 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-04-16T19:21:41.548 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-16T19:21:41.590 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T19:21:41.650 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): libcephsqlite-19.0.0-3089.gb0c7b917.el9. 424 kB/s | 164 kB 00:00 2024-04-16T19:21:41.698 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): libcephsqlite-19.0.0-3089.gb0c7b917.el9. 455 kB/s | 164 kB 00:00 2024-04-16T19:21:41.757 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-16T19:21:41.823 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:21:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T19:21:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T19:21:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:42.050 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T19:21:42.070 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:42.342 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 222 kB/s | 187 kB 00:00 2024-04-16T19:21:43.401 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:43.401 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-04-16T19:21:43.401 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-04-16T19:21:43.401 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-04-16T19:21:43.402 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-04-16T19:21:43.402 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-04-16T19:21:43.402 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-04-16T19:21:43.402 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-04-16T19:21:43.402 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:44.340 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/34 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T19:21:44.367 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:44.369 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:44.459 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T19:21:44.459 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:44.908 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-19.0.0-3089.gb0c7b917.el9.x86_ 13 MB/s | 49 MB 00:03 2024-04-16T19:21:44.912 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:44.912 INFO:teuthology.orchestra.run.smithi069.stdout:Total 13 MB/s | 50 MB 00:03 2024-04-16T19:21:44.928 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:21:44.949 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:21:44.949 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:21:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:21:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:21:45.668 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_6 6/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 7/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9. 8/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x8 9/34 2024-04-16T19:21:45.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 10/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 11/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 12/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 13/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T19:21:45.670 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 31/34 2024-04-16T19:21:45.671 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T19:21:45.672 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/34 2024-04-16T19:21:45.790 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-19.0.0-3089.gb0c7b917.el9.x86_ 11 MB/s | 49 MB 00:04 2024-04-16T19:21:45.793 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:45.794 INFO:teuthology.orchestra.run.smithi026.stdout:Total 11 MB/s | 50 MB 00:04 2024-04-16T19:21:45.808 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:21:45.831 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:21:45.831 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:21:45.933 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:21:46.023 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T19:21:46.059 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T19:21:46.090 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:46.138 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T19:21:46.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:46.241 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:21:46.241 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:21:46.753 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T19:21:46.756 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T19:21:46.757 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.758 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:46.759 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:21:46.837 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:46.841 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:21:46.933 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T19:21:46.975 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T19:21:47.015 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:47.068 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T19:21:47.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-04-16T19:21:47.102 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:47.658 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:29 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:21:47.809 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:21:47.832 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:47.832 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:21:47.832 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:47.832 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:21:47.832 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 49 M 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 164 k 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 50 M 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 212 M 2024-04-16T19:21:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:21:48.027 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-04-16T19:21:48.077 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-16T19:21:48.110 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-04-16T19:21:48.136 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): libcephsqlite-19.0.0-3089.gb0c7b917.el9. 631 kB/s | 164 kB 00:00 2024-04-16T19:21:48.319 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 897 kB/s | 187 kB 00:00 2024-04-16T19:21:49.474 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:49.474 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/6 2024-04-16T19:21:49.474 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/6 2024-04-16T19:21:49.475 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:49.475 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T19:21:49.475 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-16.el9.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T19:21:49.843 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:49.844 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:21:50.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-04-16T19:21:50.578 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): ceph-test-19.0.0-3089.gb0c7b917.el9.x86_ 18 MB/s | 49 MB 00:02 2024-04-16T19:21:50.581 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:50.581 INFO:teuthology.orchestra.run.smithi119.stdout:Total 18 MB/s | 50 MB 00:02 2024-04-16T19:21:50.595 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:21:50.617 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:21:50.617 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:21:50.760 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:43 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:21:50.905 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:50.925 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:21:50.927 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 6.6 k 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 2.4 M 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 1.5 M 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 249 k 2024-04-16T19:21:50.928 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 4.7 M 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 17 M 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T19:21:50.929 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T19:21:50.930 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T19:21:50.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout:Install 36 Packages 2024-04-16T19:21:50.932 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:50.933 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-04-16T19:21:50.933 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 107 M 2024-04-16T19:21:50.933 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:21:50.986 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:51.012 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:21:51.012 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:21:51.526 INFO:teuthology.orchestra.run.smithi151.stdout:(1/36): ceph-19.0.0-3089.gb0c7b917.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-04-16T19:21:51.599 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:21:51.690 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T19:21:51.718 INFO:teuthology.orchestra.run.smithi151.stdout:(2/36): ceph-mgr-19.0.0-3089.gb0c7b917.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-04-16T19:21:51.728 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T19:21:51.760 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:51.812 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T19:21:51.839 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:51.860 INFO:teuthology.orchestra.run.smithi151.stdout:(3/36): ceph-mds-19.0.0-3089.gb0c7b917.el9.x86_ 4.6 MB/s | 2.4 MB 00:00 2024-04-16T19:21:51.960 INFO:teuthology.orchestra.run.smithi151.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3089.gb0c7 2.4 MB/s | 249 kB 00:00 2024-04-16T19:21:52.029 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:52.077 INFO:teuthology.orchestra.run.smithi151.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T19:21:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:52.119 INFO:teuthology.orchestra.run.smithi151.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T19:21:52.153 INFO:teuthology.orchestra.run.smithi151.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 952 kB/s | 32 kB 00:00 2024-04-16T19:21:52.186 INFO:teuthology.orchestra.run.smithi151.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T19:21:52.220 INFO:teuthology.orchestra.run.smithi151.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T19:21:52.263 INFO:teuthology.orchestra.run.smithi151.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-16T19:21:52.329 INFO:teuthology.orchestra.run.smithi151.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-16T19:21:52.363 INFO:teuthology.orchestra.run.smithi151.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-04-16T19:21:52.397 INFO:teuthology.orchestra.run.smithi151.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T19:21:52.464 INFO:teuthology.orchestra.run.smithi151.stdout:(14/36): ceph-mon-19.0.0-3089.gb0c7b917.el9.x86 5.0 MB/s | 4.7 MB 00:00 2024-04-16T19:21:52.489 INFO:teuthology.orchestra.run.smithi151.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 187 kB 00:00 2024-04-16T19:21:52.657 INFO:teuthology.orchestra.run.smithi151.stdout:(16/36): ceph-osd-19.0.0-3089.gb0c7b917.el9.x86 18 MB/s | 17 MB 00:00 2024-04-16T19:21:52.707 INFO:teuthology.orchestra.run.smithi151.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-04-16T19:21:53.152 INFO:teuthology.orchestra.run.smithi151.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 98 kB/s | 43 kB 00:00 2024-04-16T19:21:53.247 INFO:teuthology.orchestra.run.smithi151.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 256 kB/s | 151 kB 00:00 2024-04-16T19:21:53.349 INFO:teuthology.orchestra.run.smithi151.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 141 kB/s | 125 kB 00:00 2024-04-16T19:21:53.483 INFO:teuthology.orchestra.run.smithi151.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 519 kB/s | 172 kB 00:00 2024-04-16T19:21:53.508 INFO:teuthology.orchestra.run.smithi151.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-04-16T19:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/6 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/6 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T19:21:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T19:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 184 kB/s | 23 kB 00:00 2024-04-16T19:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout:(25/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 35 kB/s | 11 kB 00:00 2024-04-16T19:21:53.718 INFO:teuthology.orchestra.run.smithi151.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-04-16T19:21:53.744 INFO:teuthology.orchestra.run.smithi151.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-04-16T19:21:53.852 INFO:teuthology.orchestra.run.smithi151.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 589 kB/s | 79 kB 00:00 2024-04-16T19:21:53.878 INFO:teuthology.orchestra.run.smithi151.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 212 kB/s | 46 kB 00:00 2024-04-16T19:21:53.903 INFO:teuthology.orchestra.run.smithi151.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-04-16T19:21:53.961 INFO:teuthology.orchestra.run.smithi151.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-04-16T19:21:53.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T19:21:53.984 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:53.984 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:53.985 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:21:54.003 INFO:teuthology.orchestra.run.smithi151.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-04-16T19:21:54.120 INFO:teuthology.orchestra.run.smithi151.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 370 kB/s | 90 kB 00:00 2024-04-16T19:21:54.146 INFO:teuthology.orchestra.run.smithi151.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-16T19:21:54.188 INFO:teuthology.orchestra.run.smithi151.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-16T19:21:54.246 INFO:teuthology.orchestra.run.smithi151.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-04-16T19:21:54.252 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:54.252 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.0 MB/s | 30 MB 00:03 2024-04-16T19:21:54.275 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:21:54.315 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:21:54.315 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:21:54.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2024-04-16T19:21:54.588 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:54.589 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/6 2024-04-16T19:21:54.589 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/6 2024-04-16T19:21:54.589 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:54.589 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T19:21:54.589 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T19:21:54.716 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:21:54.717 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:21:54.933 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:43 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-16.el9.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T19:21:54.951 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:21:55.089 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 6.6 k 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 2.4 M 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 1.5 M 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 249 k 2024-04-16T19:21:55.092 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 4.7 M 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 17 M 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T19:21:55.093 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T19:21:55.094 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout:========================================================================================= 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout:Install 36 Packages 2024-04-16T19:21:55.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:21:55.096 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2024-04-16T19:21:55.096 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 107 M 2024-04-16T19:21:55.096 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:21:55.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-04-16T19:21:55.596 INFO:teuthology.orchestra.run.smithi069.stdout:(1/36): ceph-19.0.0-3089.gb0c7b917.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-04-16T19:21:55.843 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:46 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:21:55.847 INFO:teuthology.orchestra.run.smithi069.stdout:(2/36): ceph-mds-19.0.0-3089.gb0c7b917.el9.x86_ 5.7 MB/s | 2.4 MB 00:00 2024-04-16T19:21:55.964 INFO:teuthology.orchestra.run.smithi069.stdout:(3/36): ceph-mgr-19.0.0-3089.gb0c7b917.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-04-16T19:21:56.018 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:21:56.020 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-04-16T19:21:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 6.6 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 2.4 M 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 1.5 M 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 249 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 4.7 M 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 17 M 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T19:21:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T19:21:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:21:56.023 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-04-16T19:21:56.024 INFO:teuthology.orchestra.run.smithi026.stdout:Install 36 Packages 2024-04-16T19:21:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:21:56.024 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 30 M 2024-04-16T19:21:56.024 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 107 M 2024-04-16T19:21:56.024 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:21:56.040 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:21:56.064 INFO:teuthology.orchestra.run.smithi069.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3089.gb0c7 2.4 MB/s | 249 kB 00:00 2024-04-16T19:21:56.107 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T19:21:56.149 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T19:21:56.177 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T19:21:56.189 INFO:teuthology.orchestra.run.smithi069.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-04-16T19:21:56.231 INFO:teuthology.orchestra.run.smithi069.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T19:21:56.247 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T19:21:56.265 INFO:teuthology.orchestra.run.smithi069.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-04-16T19:21:56.298 INFO:teuthology.orchestra.run.smithi069.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T19:21:56.309 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T19:21:56.342 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T19:21:56.365 INFO:teuthology.orchestra.run.smithi069.stdout:(9/36): ceph-mon-19.0.0-3089.gb0c7b917.el9.x86_ 6.1 MB/s | 4.7 MB 00:00 2024-04-16T19:21:56.391 INFO:teuthology.orchestra.run.smithi069.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 419 kB/s | 39 kB 00:00 2024-04-16T19:21:56.416 INFO:teuthology.orchestra.run.smithi069.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-16T19:21:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T19:21:56.475 INFO:teuthology.orchestra.run.smithi069.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-04-16T19:21:56.493 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/6 2024-04-16T19:21:56.500 INFO:teuthology.orchestra.run.smithi069.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-16T19:21:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T19:21:56.525 INFO:teuthology.orchestra.run.smithi069.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-16T19:21:56.546 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T19:21:56.551 INFO:teuthology.orchestra.run.smithi069.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-04-16T19:21:56.573 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:56.617 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T19:21:56.672 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T19:21:56.710 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T19:21:56.751 INFO:teuthology.orchestra.run.smithi069.stdout:(16/36): ceph-osd-19.0.0-3089.gb0c7b917.el9.x86 19 MB/s | 17 MB 00:00 2024-04-16T19:21:56.751 INFO:teuthology.orchestra.run.smithi026.stdout:(1/36): ceph-19.0.0-3089.gb0c7b917.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-04-16T19:21:56.759 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T19:21:56.776 INFO:teuthology.orchestra.run.smithi069.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-04-16T19:21:56.814 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T19:21:56.893 INFO:teuthology.orchestra.run.smithi069.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-04-16T19:21:56.918 INFO:teuthology.orchestra.run.smithi069.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-04-16T19:21:56.985 INFO:teuthology.orchestra.run.smithi069.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-16T19:21:57.020 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T19:21:57.027 INFO:teuthology.orchestra.run.smithi069.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-04-16T19:21:57.053 INFO:teuthology.orchestra.run.smithi069.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 159 kB/s | 11 kB 00:00 2024-04-16T19:21:57.075 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T19:21:57.078 INFO:teuthology.orchestra.run.smithi069.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 358 kB/s | 18 kB 00:00 2024-04-16T19:21:57.103 INFO:teuthology.orchestra.run.smithi069.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-04-16T19:21:57.113 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T19:21:57.129 INFO:teuthology.orchestra.run.smithi069.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-04-16T19:21:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T19:21:57.154 INFO:teuthology.orchestra.run.smithi069.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-04-16T19:21:57.177 INFO:teuthology.orchestra.run.smithi026.stdout:(2/36): ceph-mon-19.0.0-3089.gb0c7b917.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-04-16T19:21:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T19:21:57.179 INFO:teuthology.orchestra.run.smithi069.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-04-16T19:21:57.205 INFO:teuthology.orchestra.run.smithi069.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-16T19:21:57.218 INFO:teuthology.orchestra.run.smithi026.stdout:(3/36): ceph-mgr-19.0.0-3089.gb0c7b917.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-04-16T19:21:57.230 INFO:teuthology.orchestra.run.smithi069.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-16T19:21:57.255 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T19:21:57.255 INFO:teuthology.orchestra.run.smithi069.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-04-16T19:21:57.277 INFO:teuthology.orchestra.run.smithi026.stdout:(4/36): ceph-mds-19.0.0-3089.gb0c7b917.el9.x86_ 3.4 MB/s | 2.4 MB 00:00 2024-04-16T19:21:57.280 INFO:teuthology.orchestra.run.smithi069.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-16T19:21:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:21:57.305 INFO:teuthology.orchestra.run.smithi069.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-04-16T19:21:57.311 INFO:teuthology.orchestra.run.smithi026.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3089.gb0c7 2.6 MB/s | 249 kB 00:00 2024-04-16T19:21:57.322 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T19:21:57.331 INFO:teuthology.orchestra.run.smithi069.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-16T19:21:57.354 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T19:21:57.364 INFO:teuthology.orchestra.run.smithi069.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 600 kB/s | 20 kB 00:00 2024-04-16T19:21:57.390 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T19:21:57.425 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T19:21:57.445 INFO:teuthology.orchestra.run.smithi026.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-16T19:21:57.456 INFO:teuthology.orchestra.run.smithi069.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-04-16T19:21:57.470 INFO:teuthology.orchestra.run.smithi026.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-04-16T19:21:57.481 INFO:teuthology.orchestra.run.smithi069.stdout:(36/36): python3-requests-2.25.1-8.el9.noarch.r 131 kB/s | 125 kB 00:00 2024-04-16T19:21:57.485 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:57.485 INFO:teuthology.orchestra.run.smithi069.stdout:Total 13 MB/s | 30 MB 00:02 2024-04-16T19:21:57.495 INFO:teuthology.orchestra.run.smithi026.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-04-16T19:21:57.505 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T19:21:57.510 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:21:57.520 INFO:teuthology.orchestra.run.smithi026.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-04-16T19:21:57.545 INFO:teuthology.orchestra.run.smithi026.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-04-16T19:21:57.547 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T19:21:57.549 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:21:57.549 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:21:57.581 INFO:teuthology.orchestra.run.smithi026.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.9 MB/s | 241 kB 00:00 2024-04-16T19:21:57.621 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T19:21:57.632 INFO:teuthology.orchestra.run.smithi026.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-04-16T19:21:57.657 INFO:teuthology.orchestra.run.smithi026.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-04-16T19:21:57.682 INFO:teuthology.orchestra.run.smithi026.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-16T19:21:57.716 INFO:teuthology.orchestra.run.smithi026.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-04-16T19:21:57.757 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T19:21:57.791 INFO:teuthology.orchestra.run.smithi026.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-04-16T19:21:57.875 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:21:57.946 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:21:57.947 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:21:58.025 INFO:teuthology.orchestra.run.smithi026.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 648 kB/s | 151 kB 00:00 2024-04-16T19:21:58.038 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 31/36 2024-04-16T19:21:58.063 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:21:58.090 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:58.212 INFO:teuthology.orchestra.run.smithi026.stdout:(18/36): ceph-osd-19.0.0-3089.gb0c7b917.el9.x86 16 MB/s | 17 MB 00:01 2024-04-16T19:21:58.234 INFO:teuthology.orchestra.run.smithi026.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2024-04-16T19:21:58.351 INFO:teuthology.orchestra.run.smithi026.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-04-16T19:21:58.401 INFO:teuthology.orchestra.run.smithi026.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-04-16T19:21:58.451 INFO:teuthology.orchestra.run.smithi026.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 722 kB/s | 172 kB 00:00 2024-04-16T19:21:58.468 INFO:teuthology.orchestra.run.smithi026.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-04-16T19:21:58.502 INFO:teuthology.orchestra.run.smithi026.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-04-16T19:21:58.519 INFO:teuthology.orchestra.run.smithi026.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-16T19:21:58.552 INFO:teuthology.orchestra.run.smithi026.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-04-16T19:21:58.586 INFO:teuthology.orchestra.run.smithi026.stdout:(27/36): python3-requests-2.25.1-8.el9.noarch.r 138 kB/s | 125 kB 00:00 2024-04-16T19:21:58.611 INFO:teuthology.orchestra.run.smithi026.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-04-16T19:21:58.636 INFO:teuthology.orchestra.run.smithi026.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-04-16T19:21:58.670 INFO:teuthology.orchestra.run.smithi026.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-04-16T19:21:58.729 INFO:teuthology.orchestra.run.smithi026.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 978 kB/s | 90 kB 00:00 2024-04-16T19:21:58.754 INFO:teuthology.orchestra.run.smithi026.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-04-16T19:21:58.829 INFO:teuthology.orchestra.run.smithi026.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-16T19:21:58.854 INFO:teuthology.orchestra.run.smithi026.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-04-16T19:21:58.879 INFO:teuthology.orchestra.run.smithi026.stdout:(35/36): python3-pecan-1.4.2-3.el9.noarch.rpm 929 kB/s | 272 kB 00:00 2024-04-16T19:21:58.905 INFO:teuthology.orchestra.run.smithi026.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-04-16T19:21:58.908 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:21:58.908 INFO:teuthology.orchestra.run.smithi026.stdout:Total 10 MB/s | 30 MB 00:02 2024-04-16T19:21:58.935 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:21:58.980 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:21:58.980 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 6/6 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/6 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/6 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T19:21:59.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T19:21:59.278 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:21:59.343 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T19:21:59.383 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T19:21:59.412 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T19:21:59.423 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:21:59.423 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:21:59.481 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T19:21:59.482 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:21:59.483 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:21:59.518 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T19:21:59.539 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:21:59.559 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T19:21:59.566 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:21:59.566 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:21:59.567 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T19:21:59.567 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:21:59.567 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:21:59.567 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:21:59.658 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T19:21:59.745 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T19:21:59.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-04-16T19:21:59.787 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T19:21:59.858 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T19:21:59.921 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T19:21:59.960 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T19:22:00.008 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T19:22:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:00.039 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:00.039 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:00.039 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T19:22:00.040 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:00.040 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:00.040 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:00.064 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T19:22:00.250 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T19:22:00.266 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:00.290 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T19:22:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T19:22:00.363 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:00.373 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:42 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T19:22:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T19:22:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T19:22:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:00.530 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:00.532 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-16T19:22:00.532 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:00.532 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-16T19:22:00.532 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:00.532 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 6.6 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 2.4 M 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 1.5 M 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 249 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 4.7 M 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 17 M 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T19:22:00.533 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T19:22:00.534 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout:Install 36 Packages 2024-04-16T19:22:00.535 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 30 M 2024-04-16T19:22:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 107 M 2024-04-16T19:22:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T19:22:00.602 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T19:22:00.637 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T19:22:00.680 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T19:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:00.768 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T19:22:00.808 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T19:22:00.810 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T19:22:00.841 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T19:22:00.869 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T19:22:00.890 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T19:22:00.942 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T19:22:00.992 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T19:22:01.025 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T19:22:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T19:22:01.120 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T19:22:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:01.205 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T19:22:01.253 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T19:22:01.272 INFO:teuthology.orchestra.run.smithi119.stdout:(1/36): ceph-19.0.0-3089.gb0c7b917.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-04-16T19:22:01.338 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 31/36 2024-04-16T19:22:01.360 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T19:22:01.366 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:01.392 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:01.392 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:01.392 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T19:22:01.393 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:01.393 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:01.393 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:01.422 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T19:22:01.459 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T19:22:01.517 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T19:22:01.547 INFO:teuthology.orchestra.run.smithi119.stdout:(2/36): ceph-mds-19.0.0-3089.gb0c7b917.el9.x86_ 4.6 MB/s | 2.4 MB 00:00 2024-04-16T19:22:01.582 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T19:22:01.589 INFO:teuthology.orchestra.run.smithi119.stdout:(3/36): ceph-mgr-19.0.0-3089.gb0c7b917.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-04-16T19:22:01.706 INFO:teuthology.orchestra.run.smithi119.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3089.gb0c7 2.1 MB/s | 249 kB 00:00 2024-04-16T19:22:01.780 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T19:22:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T19:22:01.830 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T19:22:01.865 INFO:teuthology.orchestra.run.smithi119.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T19:22:01.869 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T19:22:01.898 INFO:teuthology.orchestra.run.smithi119.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-04-16T19:22:01.903 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T19:22:01.932 INFO:teuthology.orchestra.run.smithi119.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T19:22:01.933 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T19:22:01.965 INFO:teuthology.orchestra.run.smithi119.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T19:22:02.013 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T19:22:02.016 INFO:teuthology.orchestra.run.smithi119.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-16T19:22:02.052 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:02.085 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T19:22:02.126 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T19:22:02.133 INFO:teuthology.orchestra.run.smithi119.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-04-16T19:22:02.169 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T19:22:02.175 INFO:teuthology.orchestra.run.smithi119.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 6/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T19:22:02.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T19:22:02.185 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T19:22:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T19:22:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T19:22:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T19:22:02.186 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T19:22:02.187 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T19:22:02.212 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T19:22:02.217 INFO:teuthology.orchestra.run.smithi119.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-04-16T19:22:02.267 INFO:teuthology.orchestra.run.smithi119.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-16T19:22:02.313 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T19:22:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T19:22:02.435 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T19:22:02.476 INFO:teuthology.orchestra.run.smithi119.stdout:(15/36): ceph-osd-19.0.0-3089.gb0c7b917.el9.x86 18 MB/s | 17 MB 00:00 2024-04-16T19:22:02.553 INFO:teuthology.orchestra.run.smithi119.stdout:(16/36): ceph-mon-19.0.0-3089.gb0c7b917.el9.x86 3.7 MB/s | 4.7 MB 00:01 2024-04-16T19:22:02.576 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T19:22:02.576 INFO:teuthology.orchestra.run.smithi119.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 404 kB/s | 125 kB 00:00 2024-04-16T19:22:02.618 INFO:teuthology.orchestra.run.smithi119.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-04-16T19:22:02.698 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:02.802 INFO:teuthology.orchestra.run.smithi119.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2024-04-16T19:22:02.827 INFO:teuthology.orchestra.run.smithi119.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 553 kB/s | 151 kB 00:00 2024-04-16T19:22:02.842 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:02.860 INFO:teuthology.orchestra.run.smithi119.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2024-04-16T19:22:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:02.870 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:02.871 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T19:22:02.871 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:02.871 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 31/36 2024-04-16T19:22:02.886 INFO:teuthology.orchestra.run.smithi119.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 780 kB/s | 172 kB 00:00 2024-04-16T19:22:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:02.903 INFO:teuthology.orchestra.run.smithi119.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-04-16T19:22:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:02.923 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:02.923 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T19:22:02.923 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:02.924 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:02.937 INFO:teuthology.orchestra.run.smithi119.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-04-16T19:22:02.953 INFO:teuthology.orchestra.run.smithi119.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-04-16T19:22:02.970 INFO:teuthology.orchestra.run.smithi119.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-04-16T19:22:02.987 INFO:teuthology.orchestra.run.smithi119.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-04-16T19:22:03.012 INFO:teuthology.orchestra.run.smithi119.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-04-16T19:22:03.037 INFO:teuthology.orchestra.run.smithi119.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-04-16T19:22:03.063 INFO:teuthology.orchestra.run.smithi119.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-04-16T19:22:03.088 INFO:teuthology.orchestra.run.smithi119.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-16T19:22:03.113 INFO:teuthology.orchestra.run.smithi119.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi119.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T19:22:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T19:22:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T19:22:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T19:22:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T19:22:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T19:22:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:03.142 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:03.164 INFO:teuthology.orchestra.run.smithi119.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-16T19:22:03.189 INFO:teuthology.orchestra.run.smithi119.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-04-16T19:22:03.214 INFO:teuthology.orchestra.run.smithi119.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-04-16T19:22:03.219 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:03.219 INFO:teuthology.orchestra.run.smithi119.stdout:Total 11 MB/s | 30 MB 00:02 2024-04-16T19:22:03.242 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:03.280 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:03.280 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:03.310 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:03.334 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:03.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-04-16T19:22:03.552 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:03.576 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:03.576 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:03.576 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T19:22:03.577 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:03.577 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:03.577 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:03.641 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:03.669 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:03.670 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:04.097 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:57 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:04.198 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:04.250 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:04.251 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:04.251 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:04.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-04-16T19:22:04.448 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:04.475 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:04.924 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:57 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:04.973 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:04.973 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T19:22:04.973 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:04.973 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:04.989 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:05.058 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T19:22:05.079 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 762 k 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:05.080 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:05.081 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 762 k 2024-04-16T19:22:05.081 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 769 k 2024-04-16T19:22:05.081 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:05.091 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T19:22:05.119 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T19:22:05.192 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T19:22:05.208 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:05.232 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:05.232 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T19:22:05.232 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:05.232 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:05.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:05.245 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T19:22:05.284 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T19:22:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:05.376 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T19:22:05.455 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T19:22:05.496 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T19:22:05.521 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.0.0-3089.gb0c7b917.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:05.521 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:05.522 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:05.522 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:05.527 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:05.527 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:05.533 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:05.533 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:05.544 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:05.544 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 6/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T19:22:05.545 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T19:22:05.546 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T19:22:05.547 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T19:22:05.547 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T19:22:05.548 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T19:22:05.567 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T19:22:05.610 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:05.621 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T19:22:05.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T19:22:05.708 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T19:22:05.763 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T19:22:05.951 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T19:22:06.005 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T19:22:06.032 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:06.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T19:22:06.069 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T19:22:06.099 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T19:22:06.118 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:06.176 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T19:22:06.210 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:06.243 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T19:22:06.275 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T19:22:06.310 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T19:22:06.345 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T19:22:06.434 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T19:22:06.475 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T19:22:06.548 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T19:22:06.557 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:06.561 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T19:22:06.562 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T19:22:06.563 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T19:22:06.564 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T19:22:06.565 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T19:22:06.565 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:06.565 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:06.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T19:22:06.812 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:06.866 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:06.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2024-04-16T19:22:06.997 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 31/36 2024-04-16T19:22:07.047 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 32/36 2024-04-16T19:22:07.073 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:07.073 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T19:22:07.073 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:07.073 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T19:22:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:07.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T19:22:07.292 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:07.292 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/36 2024-04-16T19:22:07.292 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 6/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T19:22:07.293 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T19:22:07.294 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T19:22:07.295 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T19:22:07.296 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T19:22:07.296 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T19:22:07.296 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T19:22:07.296 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T19:22:07.297 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T19:22:07.519 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:56 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:07.621 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:07.673 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:07.674 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:07.675 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:07.679 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:07.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2024-04-16T19:22:07.835 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:07.835 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:07.835 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 145 k 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:07.836 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:07.837 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-04-16T19:22:07.837 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 435 k 2024-04-16T19:22:07.837 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:08.190 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.0.0-3089.gb0c7b9 411 kB/s | 145 kB 00:00 2024-04-16T19:22:08.191 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:08.191 INFO:teuthology.orchestra.run.smithi151.stdout:Total 409 kB/s | 145 kB 00:00 2024-04-16T19:22:08.191 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:08.198 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:08.198 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:08.250 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:08.250 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T19:22:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T19:22:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T19:22:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T19:22:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:08.281 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:08.310 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:57 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:08.390 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:08.462 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 762 k 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 762 k 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 769 k 2024-04-16T19:22:08.463 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:08.506 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:08.512 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:08.513 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:08.513 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T19:22:08.513 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 33/36 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T19:22:08.533 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:08.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-04-16T19:22:08.743 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-19.0.0-3089.gb0c7b917.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-04-16T19:22:08.744 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:08.744 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-16T19:22:08.744 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:08.749 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:08.750 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:08.755 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:08.755 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:08.825 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:08.982 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 34/36 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T19:22:09.006 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:09.232 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:09.238 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:09.256 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 35/36 2024-04-16T19:22:09.256 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:09.256 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T19:22:09.256 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:09.256 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T19:22:09.257 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:09.295 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:09.329 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:09.343 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:09.389 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:09.400 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:09.401 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:09.401 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:09.519 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:09.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-04-16T19:22:09.752 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:09.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:10.056 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:10.179 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:01 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:10.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T19:22:10.339 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 762 k 2024-04-16T19:22:10.340 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 762 k 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 769 k 2024-04-16T19:22:10.341 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:10.366 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:03 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:10.467 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:10.521 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:10.521 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:10.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T19:22:10.773 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-19.0.0-3089.gb0c7b917.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:10.773 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:10.774 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:10.774 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:10.779 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:10.779 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:10.786 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:10.858 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:10.860 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:59 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 145 k 2024-04-16T19:22:11.016 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 145 k 2024-04-16T19:22:11.017 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 435 k 2024-04-16T19:22:11.018 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 36/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 3/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 4/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 5/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9. 6/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T19:22:11.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T19:22:11.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T19:22:11.095 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T19:22:11.175 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:04 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:11.249 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-19.0.0-3089.gb0c7b9 626 kB/s | 145 kB 00:00 2024-04-16T19:22:11.249 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:11.250 INFO:teuthology.orchestra.run.smithi069.stdout:Total 622 kB/s | 145 kB 00:00 2024-04-16T19:22:11.250 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:11.258 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:11.258 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:11.310 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:11.310 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:11.329 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:11.330 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-04-16T19:22:11.330 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:11.330 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-04-16T19:22:11.330 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:11.330 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 3.5 M 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 34 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 16 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:11.331 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 8.6 M 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 104 M 2024-04-16T19:22:11.332 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:11.447 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:11.452 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:11.517 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:11.543 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:11.543 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:11.543 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T19:22:11.543 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:12.022 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3089.gb0c 64 kB/s | 16 kB 00:00 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T19:22:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T19:22:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:12.063 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:12.236 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:12.236 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:12.237 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:12.237 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:12.266 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): ceph-mgr-dashboard-19.0.0-3089.gb0c7b91 7.1 MB/s | 3.5 MB 00:00 2024-04-16T19:22:12.306 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-04-16T19:22:12.331 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.7 MB/s | 238 kB 00:00 2024-04-16T19:22:12.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-04-16T19:22:12.407 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): ceph-grafana-dashboards-19.0.0-3089.gb0 53 kB/s | 34 kB 00:00 2024-04-16T19:22:12.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:12.574 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:12.590 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-04-16T19:22:12.807 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 48 kB/s | 19 kB 00:00 2024-04-16T19:22:12.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2024-04-16T19:22:12.990 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:54 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:13.066 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 746 kB/s | 548 kB 00:00 2024-04-16T19:22:13.073 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:04 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:13.091 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:13.133 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 444 kB/s | 144 kB 00:00 2024-04-16T19:22:13.143 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:13.144 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:22:13.144 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:13.184 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 262 kB/s | 31 kB 00:00 2024-04-16T19:22:13.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-04-16T19:22:13.231 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:13.231 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 145 k 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 145 k 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 435 k 2024-04-16T19:22:13.232 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:13.342 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 903 kB/s | 188 kB 00:00 2024-04-16T19:22:13.429 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:02 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:13.517 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-04-16T19:22:13.523 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:13.523 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.9 MB/s | 8.6 MB 00:02 2024-04-16T19:22:13.530 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:13.541 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:13.558 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:13.559 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:13.582 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:13.583 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:13.583 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:13.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T19:22:13.738 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:13.738 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:13.779 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-19.0.0-3089.gb0c7b9 265 kB/s | 145 kB 00:00 2024-04-16T19:22:13.780 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:13.780 INFO:teuthology.orchestra.run.smithi026.stdout:Total 264 kB/s | 145 kB 00:00 2024-04-16T19:22:13.780 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:13.788 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:13.789 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:13.791 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:55 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:13.843 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:13.843 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:13.943 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:13.944 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:13.944 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:13.944 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:13.944 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:13.944 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 762 k 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 762 k 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 769 k 2024-04-16T19:22:13.945 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:13.983 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:14.046 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:14.071 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:14.071 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T19:22:14.071 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:14.234 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:03 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:14.252 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-16T19:22:14.382 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-19.0.0-3089.gb0c7b917.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:14.382 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:14.383 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-16T19:22:14.383 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:14.388 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:14.388 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:14.389 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:14.390 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-04-16T19:22:14.390 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 3.5 M 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 34 k 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 16 k 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T19:22:14.391 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout:====================================================================================== 2024-04-16T19:22:14.392 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2024-04-16T19:22:14.393 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:14.393 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 8.6 M 2024-04-16T19:22:14.393 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 104 M 2024-04-16T19:22:14.393 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:14.394 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:14.394 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-16T19:22:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-16T19:22:14.469 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-16T19:22:14.592 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-16T19:22:14.724 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-16T19:22:14.819 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-16T19:22:14.862 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3089.gb0c 64 kB/s | 16 kB 00:00 2024-04-16T19:22:14.862 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-16T19:22:14.887 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3089.gb0 121 kB/s | 34 kB 00:00 2024-04-16T19:22:14.893 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:14.925 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 10/12 2024-04-16T19:22:14.977 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:15.029 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:15.063 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:15.130 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): ceph-mgr-dashboard-19.0.0-3089.gb0c7b91 6.7 MB/s | 3.5 MB 00:00 2024-04-16T19:22:15.238 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-04-16T19:22:15.280 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-04-16T19:22:15.330 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-04-16T19:22:15.333 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-04-16T19:22:15.343 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:15.364 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-04-16T19:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 527 kB/s | 31 kB 00:00 2024-04-16T19:22:15.423 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-04-16T19:22:15.456 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.0 MB/s | 863 kB 00:00 2024-04-16T19:22:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:Total 8.1 MB/s | 8.6 MB 00:01 2024-04-16T19:22:15.480 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:15.497 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:15.497 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/1 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:15.614 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:15.708 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:15.708 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:15.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T19:22:15.941 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:06 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:15.969 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 11/12 2024-04-16T19:22:16.010 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:16.047 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:16.103 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:16.104 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:16.104 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:16.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T19:22:16.233 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:16.315 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-16T19:22:16.350 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-16T19:22:16.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-16T19:22:16.419 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:58 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:16.444 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-16T19:22:16.549 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-16T19:22:16.574 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:16.574 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 145 k 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:22:16.575 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:16.576 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 145 k 2024-04-16T19:22:16.576 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 435 k 2024-04-16T19:22:16.576 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:16.676 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-16T19:22:16.763 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-16T19:22:16.782 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:07 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:16.815 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-16T19:22:16.844 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:16.876 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 10/12 2024-04-16T19:22:16.879 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-19.0.0-3089.gb0c7b9 478 kB/s | 145 kB 00:00 2024-04-16T19:22:16.880 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:16.880 INFO:teuthology.orchestra.run.smithi119.stdout:Total 476 kB/s | 145 kB 00:00 2024-04-16T19:22:16.880 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:16.887 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:16.888 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:16.939 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:16.939 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:16.941 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 3.5 M 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 34 k 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 16 k 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T19:22:16.943 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-04-16T19:22:16.944 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:16.945 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 8.6 M 2024-04-16T19:22:16.945 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 104 M 2024-04-16T19:22:16.945 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:17.156 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:17.156 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:17.156 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T19:22:17.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:17.432 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3089.gb0c 75 kB/s | 16 kB 00:00 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 1/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 2/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 3/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-16T19:22:17.517 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-16T19:22:17.518 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-16T19:22:17.518 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-16T19:22:17.518 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:17.518 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-16T19:22:17.518 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-16T19:22:17.633 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): ceph-mgr-dashboard-19.0.0-3089.gb0c7b91 8.4 MB/s | 3.5 MB 00:00 2024-04-16T19:22:17.658 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): ceph-grafana-dashboards-19.0.0-3089.gb0 77 kB/s | 34 kB 00:00 2024-04-16T19:22:17.742 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-04-16T19:22:17.808 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-04-16T19:22:17.878 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-16T19:22:17.878 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:17.878 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:17.878 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T19:22:17.878 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T19:22:17.879 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T19:22:17.880 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:17.880 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:17.900 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-04-16T19:22:17.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 11/12 2024-04-16T19:22:17.926 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-04-16T19:22:17.960 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:18.043 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-04-16T19:22:18.076 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 960 kB/s | 144 kB 00:00 2024-04-16T19:22:18.093 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 612 kB/s | 31 kB 00:00 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917. 1/1 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:18.096 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:18.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T19:22:18.168 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-16T19:22:18.293 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.2 MB/s | 2.0 MB 00:00 2024-04-16T19:22:18.297 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:18.297 INFO:teuthology.orchestra.run.smithi026.stdout:Total 6.4 MB/s | 8.6 MB 00:01 2024-04-16T19:22:18.316 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:18.336 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:18.336 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:18.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-04-16T19:22:18.542 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:18.550 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:18.733 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:11 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:18.887 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:18.888 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-04-16T19:22:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:18.888 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-04-16T19:22:18.888 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 7.4 M 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T19:22:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-04-16T19:22:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:18.891 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2024-04-16T19:22:18.891 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-04-16T19:22:18.891 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:18.988 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:19.074 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:19.092 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:19.144 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:19.145 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:22:19.145 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:19.153 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-16T19:22:19.196 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-16T19:22:19.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T19:22:19.252 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-16T19:22:19.289 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 1/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 2/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 3/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-16T19:22:19.297 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-16T19:22:19.403 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-16T19:22:19.416 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2024-04-16T19:22:19.450 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-04-16T19:22:19.492 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 34 kB 00:00 2024-04-16T19:22:19.542 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-16T19:22:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-16T19:22:19.638 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-16T19:22:19.639 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T19:22:19.640 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T19:22:19.641 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:19.641 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-16T19:22:19.709 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-04-16T19:22:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 10/12 2024-04-16T19:22:19.776 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-04-16T19:22:19.796 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:01 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:19.801 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-04-16T19:22:19.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T19:22:19.949 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:19.950 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-04-16T19:22:19.950 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:19.950 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-04-16T19:22:19.950 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 3.5 M 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 34 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 16 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T19:22:19.951 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 8.6 M 2024-04-16T19:22:19.952 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 104 M 2024-04-16T19:22:19.953 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:20.085 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-04-16T19:22:20.119 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-16T19:22:20.203 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-04-16T19:22:20.245 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-16T19:22:20.486 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:20.641 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:20.642 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 7.4 M 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T19:22:20.643 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:20.644 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-04-16T19:22:20.645 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2024-04-16T19:22:20.645 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:20.645 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 37 M 2024-04-16T19:22:20.645 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 209 M 2024-04-16T19:22:20.646 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:20.775 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3089.gb0c 86 kB/s | 16 kB 00:00 2024-04-16T19:22:20.833 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3089.gb0 136 kB/s | 34 kB 00:00 2024-04-16T19:22:20.867 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 11/12 2024-04-16T19:22:20.914 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:20.934 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.4 MB/s | 238 kB 00:00 2024-04-16T19:22:20.967 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-04-16T19:22:21.031 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 242 kB/s | 32 kB 00:00 2024-04-16T19:22:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-04-16T19:22:21.098 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T19:22:21.201 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-04-16T19:22:21.260 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): ceph-mgr-dashboard-19.0.0-3089.gb0c7b91 5.2 MB/s | 3.5 MB 00:00 2024-04-16T19:22:21.265 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2024-04-16T19:22:21.301 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-16T19:22:21.324 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.1 MB/s | 3.0 MB 00:00 2024-04-16T19:22:21.335 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-04-16T19:22:21.416 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-04-16T19:22:21.441 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-04-16T19:22:21.444 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-04-16T19:22:21.485 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 786 kB/s | 144 kB 00:00 2024-04-16T19:22:21.577 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-04-16T19:22:21.703 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.6 MB/s | 2.0 MB 00:00 2024-04-16T19:22:21.707 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:21.707 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.9 MB/s | 8.6 MB 00:01 2024-04-16T19:22:21.725 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:21.743 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:21.743 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:21.811 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 2.9 MB/s | 7.4 MB 00:02 2024-04-16T19:22:21.814 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:21.815 INFO:teuthology.orchestra.run.smithi151.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-16T19:22:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-16T19:22:21.860 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:21.886 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:21.886 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:21.923 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:21.923 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:21.925 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-04-16T19:22:22.009 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-16T19:22:22.084 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-04-16T19:22:22.184 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 5.8 MB/s | 7.4 MB 00:01 2024-04-16T19:22:22.186 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:22.187 INFO:teuthology.orchestra.run.smithi069.stdout:Total 24 MB/s | 37 MB 00:01 2024-04-16T19:22:22.229 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:22.247 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:22.247 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:22.253 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:22.253 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 1/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 2/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 3/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:22.336 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-16T19:22:22.337 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-16T19:22:22.460 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:22.536 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-16T19:22:22.580 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-16T19:22:22.617 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:22.618 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:22.626 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-16T19:22:22.665 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-16T19:22:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T19:22:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T19:22:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:22.700 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:22.757 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-16T19:22:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-16T19:22:22.948 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T19:22:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:22.984 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-16T19:22:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T19:22:23.045 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-16T19:22:23.073 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:23.074 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T19:22:23.097 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 10/12 2024-04-16T19:22:23.168 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:23.212 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T19:22:23.327 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:23.407 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T19:22:23.436 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:23.438 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T19:22:23.521 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T19:22:23.532 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:23.558 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:14 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:23.591 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T19:22:23.723 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:23.724 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-04-16T19:22:23.724 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:23.724 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-04-16T19:22:23.724 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:23.724 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 7.4 M 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T19:22:23.725 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 37 M 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-04-16T19:22:23.726 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:23.769 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:23.799 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:23.875 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T19:22:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-04-16T19:22:24.094 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 548 kB/s | 17 kB 00:00 2024-04-16T19:22:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 11/12 2024-04-16T19:22:24.127 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-16T19:22:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:24.175 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T19:22:24.286 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-04-16T19:22:24.292 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T19:22:24.420 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-04-16T19:22:24.470 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-04-16T19:22:24.544 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T19:22:24.662 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T19:22:24.695 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): ceph-mgr-diskprediction-local-19.0.0-30 9.5 MB/s | 7.4 MB 00:00 2024-04-16T19:22:24.863 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-04-16T19:22:24.921 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:01 2024-04-16T19:22:25.063 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 982 kB/s | 195 kB 00:00 2024-04-16T19:22:25.088 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-04-16T19:22:25.130 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-16T19:22:25.132 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:25.132 INFO:teuthology.orchestra.run.smithi026.stdout:Total 26 MB/s | 37 MB 00:01 2024-04-16T19:22:25.174 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:25.199 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:25.199 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:25.506 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 12/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el 1/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noa 2/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9 3/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-16T19:22:25.507 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-16T19:22:25.529 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T19:22:25.576 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:25.577 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-16T19:22:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T19:22:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:25.857 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:25.892 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T19:22:26.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T19:22:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:26.378 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T19:22:26.409 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T19:22:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:26.546 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T19:22:26.564 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T19:22:26.591 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:26.698 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:26.730 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:26.762 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:26.849 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T19:22:26.862 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 7.4 M 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:26.863 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T19:22:26.864 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-04-16T19:22:26.865 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:26.866 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 37 M 2024-04-16T19:22:26.866 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 209 M 2024-04-16T19:22:26.866 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:26.942 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T19:22:26.973 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:27.538 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T19:22:27.544 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 252 kB/s | 32 kB 00:00 2024-04-16T19:22:27.577 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-04-16T19:22:27.644 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-04-16T19:22:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T19:22:27.669 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 34 kB 00:00 2024-04-16T19:22:27.837 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-04-16T19:22:27.895 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-04-16T19:22:27.920 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-04-16T19:22:28.204 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-04-16T19:22:28.288 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-30 8.5 MB/s | 7.4 MB 00:00 2024-04-16T19:22:28.972 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T19:22:29.138 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 209 kB/s | 195 kB 00:00 2024-04-16T19:22:29.145 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:29.145 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 1/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T19:22:29.146 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T19:22:29.255 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 260 kB/s | 245 kB 00:00 2024-04-16T19:22:29.578 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T19:22:29.578 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:29.578 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:29.579 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 1/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:29.701 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:29.702 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T19:22:29.702 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T19:22:29.702 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T19:22:29.702 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T19:22:29.702 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T19:22:29.705 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 451 kB/s | 803 kB 00:01 2024-04-16T19:22:29.708 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:29.708 INFO:teuthology.orchestra.run.smithi119.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-16T19:22:29.752 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:29.778 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:29.778 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:29.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-04-16T19:22:30.047 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T19:22:30.078 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:30.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T19:22:30.141 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T19:22:30.142 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:30.143 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:30.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2024-04-16T19:22:30.556 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:23 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:30.712 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 49 k 2024-04-16T19:22:30.713 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:30.714 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-04-16T19:22:30.715 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:30.840 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:30.911 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T19:22:30.912 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 24 kB 00:00 2024-04-16T19:22:30.929 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-04-16T19:22:30.951 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T19:22:30.979 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-04-16T19:22:31.005 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T19:22:31.030 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T19:22:31.042 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:31.064 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 728 kB/s | 43 kB 00:00 2024-04-16T19:22:31.079 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T19:22:31.090 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): ceph-mgr-rook-19.0.0-3089.gb0c7b917.el9 167 kB/s | 49 kB 00:00 2024-04-16T19:22:31.122 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:20 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:31.273 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-04-16T19:22:31.277 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 49 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T19:22:31.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2024-04-16T19:22:31.280 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:31.281 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2024-04-16T19:22:31.281 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2024-04-16T19:22:31.281 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:31.298 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-04-16T19:22:31.362 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T19:22:31.399 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 656 kB/s | 203 kB 00:00 2024-04-16T19:22:31.424 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-04-16T19:22:31.466 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-04-16T19:22:31.508 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.4 MB/s | 1.0 MB 00:00 2024-04-16T19:22:31.512 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:31.512 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-04-16T19:22:31.513 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:31.616 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2024-04-16T19:22:31.616 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:31.617 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:31.641 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-04-16T19:22:31.692 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T19:22:31.717 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T19:22:31.742 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T19:22:31.759 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-04-16T19:22:31.826 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-04-16T19:22:31.843 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-04-16T19:22:31.926 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.0 MB/s | 203 kB 00:00 2024-04-16T19:22:31.952 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): ceph-mgr-rook-19.0.0-3089.gb0c7b917.el 104 kB/s | 49 kB 00:00 2024-04-16T19:22:31.977 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-16T19:22:31.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T19:22:32.102 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 598 kB/s | 90 kB 00:00 2024-04-16T19:22:32.114 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T19:22:32.149 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:32.219 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-04-16T19:22:32.221 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:32.221 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-04-16T19:22:32.221 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:32.234 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:32.234 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:32.314 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T19:22:32.325 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:32.325 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:32.368 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T19:22:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T19:22:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T19:22:32.503 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T19:22:32.530 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 1/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T19:22:32.531 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T19:22:32.596 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T19:22:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T19:22:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T19:22:32.832 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:32.999 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T19:22:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T19:22:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:33.018 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:33.018 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T19:22:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T19:22:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T19:22:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T19:22:33.019 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T19:22:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T19:22:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T19:22:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T19:22:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:33.020 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T19:22:33.021 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:33.021 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:33.062 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T19:22:33.126 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T19:22:33.163 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T19:22:33.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T19:22:33.307 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T19:22:33.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-04-16T19:22:33.354 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T19:22:33.407 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T19:22:33.456 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T19:22:33.466 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T19:22:33.511 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T19:22:33.546 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T19:22:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T19:22:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:33.956 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:24 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:34.117 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:34.119 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:34.119 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:34.119 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 49 k 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T19:22:34.120 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:34.121 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:34.122 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-04-16T19:22:34.122 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:34.123 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-04-16T19:22:34.123 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-04-16T19:22:34.123 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:34.177 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T19:22:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T19:22:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T19:22:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T19:22:34.336 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 18 kB 00:00 2024-04-16T19:22:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:34.369 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 24 kB 00:00 2024-04-16T19:22:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T19:22:34.419 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:34.420 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 1/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T19:22:34.445 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T19:22:34.446 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T19:22:34.447 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T19:22:34.487 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-04-16T19:22:34.512 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 646 kB/s | 43 kB 00:00 2024-04-16T19:22:34.537 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): ceph-mgr-rook-19.0.0-3089.gb0c7b917.el9 149 kB/s | 49 kB 00:00 2024-04-16T19:22:34.779 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-04-16T19:22:34.804 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T19:22:34.818 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T19:22:34.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:34.820 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:34.930 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-04-16T19:22:34.963 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-04-16T19:22:35.047 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-04-16T19:22:35.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T19:22:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 1/13 2024-04-16T19:22:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T19:22:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T19:22:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T19:22:35.142 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-04-16T19:22:35.143 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:35.143 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-04-16T19:22:35.144 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:35.158 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:35.158 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:35.257 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:35.257 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T19:22:35.514 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:35.515 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:35.797 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T19:22:35.854 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:35.855 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:35.855 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:35.855 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:35.855 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 142 k 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:35.856 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-04-16T19:22:35.857 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-04-16T19:22:35.857 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:35.857 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2024-04-16T19:22:35.857 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-04-16T19:22:35.858 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:36.102 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T19:22:36.167 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T19:22:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T19:22:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T19:22:36.309 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T19:22:36.383 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:36.415 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T19:22:36.523 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T19:22:36.536 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 142 k 2024-04-16T19:22:36.538 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-04-16T19:22:36.539 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-04-16T19:22:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:36.540 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.8 M 2024-04-16T19:22:36.540 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 33 M 2024-04-16T19:22:36.540 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:36.557 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-16T19:22:36.578 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T19:22:36.582 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 833 kB/s | 248 kB 00:00 2024-04-16T19:22:36.674 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3089.gb0c7b917.e 365 kB/s | 142 kB 00:00 2024-04-16T19:22:36.800 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-04-16T19:22:36.825 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 342 kB/s | 51 kB 00:00 2024-04-16T19:22:36.850 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-04-16T19:22:36.852 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:36.853 INFO:teuthology.orchestra.run.smithi151.stdout:Total 6.8 MB/s | 6.8 MB 00:00 2024-04-16T19:22:36.866 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:36.876 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:36.876 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 12/12 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b 1/12 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T19:22:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T19:22:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T19:22:36.965 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:36.966 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:36.997 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-04-16T19:22:37.130 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-04-16T19:22:37.156 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3089.gb0c7b917.e 447 kB/s | 142 kB 00:00 2024-04-16T19:22:37.267 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:37.314 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 316 kB/s | 58 kB 00:00 2024-04-16T19:22:37.339 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 318 kB/s | 51 kB 00:00 2024-04-16T19:22:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T19:22:37.364 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-04-16T19:22:37.368 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:37.368 INFO:teuthology.orchestra.run.smithi069.stdout:Total 8.2 MB/s | 6.8 MB 00:00 2024-04-16T19:22:37.381 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:37.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T19:22:37.383 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:37.384 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:37.391 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:37.392 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:37.398 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T19:22:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T19:22:37.432 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T19:22:37.477 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T19:22:37.479 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:37.479 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:37.481 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T19:22:37.510 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:37.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-04-16T19:22:37.787 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:37.963 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T19:22:37.986 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T19:22:38.016 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T19:22:38.071 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T19:22:38.135 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T19:22:38.166 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:38.226 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:38.226 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 1/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T19:22:38.227 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T19:22:38.228 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T19:22:38.228 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T19:22:38.352 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:20 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:38.507 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:38.508 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:38.508 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:38.508 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:38.508 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 49 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T19:22:38.509 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-04-16T19:22:38.510 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:38.511 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.0 M 2024-04-16T19:22:38.511 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-04-16T19:22:38.511 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:38.535 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T19:22:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:38.600 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:38.618 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T19:22:38.681 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T19:22:38.702 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 168 kB/s | 18 kB 00:00 2024-04-16T19:22:38.703 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:38.727 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-04-16T19:22:38.777 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T19:22:38.802 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T19:22:38.827 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T19:22:38.853 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-04-16T19:22:38.865 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T19:22:38.961 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): ceph-mgr-rook-19.0.0-3089.gb0c7b917.el9 134 kB/s | 49 kB 00:00 2024-04-16T19:22:38.987 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-04-16T19:22:39.029 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-04-16T19:22:39.062 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 2.0 MB/s | 203 kB 00:00 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/6 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T19:22:39.097 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T19:22:39.138 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-04-16T19:22:39.163 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-04-16T19:22:39.188 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2024-04-16T19:22:39.192 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:39.192 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-04-16T19:22:39.193 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:39.205 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:39.205 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:39.294 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:39.294 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T19:22:39.389 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T19:22:39.390 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T19:22:39.390 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T19:22:39.390 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:39.390 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:39.467 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:39.493 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:39.493 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/6 2024-04-16T19:22:39.494 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T19:22:39.494 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T19:22:39.494 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T19:22:39.494 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T19:22:39.628 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:39.629 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:39.629 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:39.629 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 142 k 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T19:22:39.630 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T19:22:39.631 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:39.631 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:39.631 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-04-16T19:22:39.631 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-04-16T19:22:39.631 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:39.632 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.8 M 2024-04-16T19:22:39.632 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-04-16T19:22:39.632 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:39.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T19:22:39.793 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T19:22:39.794 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T19:22:39.794 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T19:22:39.794 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T19:22:39.794 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:39.794 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:39.801 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:39.976 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T19:22:40.030 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T19:22:40.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2024-04-16T19:22:40.086 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T19:22:40.125 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T19:22:40.167 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T19:22:40.236 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:33 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:40.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T19:22:40.316 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-16T19:22:40.341 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3089.gb0c7b917.e 335 kB/s | 142 kB 00:00 2024-04-16T19:22:40.377 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T19:22:40.392 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:40.393 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:40.393 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:40.393 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 890 k 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:40.394 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-04-16T19:22:40.395 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:40.395 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 968 k 2024-04-16T19:22:40.395 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.8 M 2024-04-16T19:22:40.395 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T19:22:40.513 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 664 kB/s | 79 kB 00:00 2024-04-16T19:22:40.566 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 256 kB/s | 58 kB 00:00 2024-04-16T19:22:40.642 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:40.658 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-04-16T19:22:40.692 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-04-16T19:22:40.797 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 890 k 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:40.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:40.800 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-04-16T19:22:40.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:40.800 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 968 k 2024-04-16T19:22:40.800 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.8 M 2024-04-16T19:22:40.800 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:40.846 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.0.0-3089.gb0c7b917.el9.x86_ 1.9 MB/s | 890 kB 00:00 2024-04-16T19:22:40.848 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:40.848 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-04-16T19:22:40.848 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:40.869 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:40.870 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:40.917 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 248 kB/s | 248 kB 00:00 2024-04-16T19:22:40.919 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:40.919 INFO:teuthology.orchestra.run.smithi026.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-04-16T19:22:40.933 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 588 kB/s | 79 kB 00:00 2024-04-16T19:22:40.934 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:40.942 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:40.943 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:40.944 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:40.944 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:41.037 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:41.039 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:41.133 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-19.0.0-3089.gb0c7b917.el9.x86_ 2.6 MB/s | 890 kB 00:00 2024-04-16T19:22:41.135 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:41.135 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.8 MB/s | 968 kB 00:00 2024-04-16T19:22:41.135 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:41.143 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T19:22:41.147 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:41.156 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:41.156 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:41.190 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T19:22:41.223 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T19:22:41.229 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:41.229 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:41.271 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T19:22:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T19:22:41.294 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:41.343 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:41.347 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:41.422 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:41.517 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T19:22:41.559 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T19:22:41.570 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T19:22:41.633 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:42.060 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 13/13 2024-04-16T19:22:42.060 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 1/13 2024-04-16T19:22:42.060 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T19:22:42.060 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T19:22:42.060 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T19:22:42.061 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T19:22:42.073 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:42.073 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/2 2024-04-16T19:22:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T19:22:42.223 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T19:22:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T19:22:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:42.327 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/2 2024-04-16T19:22:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T19:22:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:42.401 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T19:22:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:42.402 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:42.420 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T19:22:42.420 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:42.420 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T19:22:42.421 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T19:22:42.422 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T19:22:42.423 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:42.423 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:42.628 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:42.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T19:22:42.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-volume 2024-04-16T19:22:42.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2024-04-16T19:22:43.120 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:43.120 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/6 2024-04-16T19:22:43.121 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T19:22:43.121 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T19:22:43.121 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T19:22:43.121 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T19:22:43.266 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:43.319 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:36 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:43.418 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:43.419 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:43.419 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 142 k 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T19:22:43.420 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T19:22:43.421 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:43.421 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:43.421 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-04-16T19:22:43.421 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-04-16T19:22:43.421 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:43.422 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.8 M 2024-04-16T19:22:43.422 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-04-16T19:22:43.422 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T19:22:43.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:43.425 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:43.474 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 264 k 2024-04-16T19:22:43.475 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-04-16T19:22:43.476 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:43.477 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 332 k 2024-04-16T19:22:43.477 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.5 M 2024-04-16T19:22:43.477 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:43.546 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:32 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:43.588 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 617 kB/s | 69 kB 00:00 2024-04-16T19:22:43.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 264 k 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T19:22:43.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 332 k 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.5 M 2024-04-16T19:22:43.706 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:43.830 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 551 kB/s | 69 kB 00:00 2024-04-16T19:22:43.980 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-volume-19.0.0-3089.gb0c7b917.el9.no 960 kB/s | 264 kB 00:00 2024-04-16T19:22:43.982 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:43.982 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-04-16T19:22:43.982 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:44.029 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:44.029 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:44.063 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-volume-19.0.0-3089.gb0c7b917.el9.no 449 kB/s | 264 kB 00:00 2024-04-16T19:22:44.065 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:44.065 INFO:teuthology.orchestra.run.smithi151.stdout:Total 564 kB/s | 332 kB 00:00 2024-04-16T19:22:44.065 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:44.074 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:44.074 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:44.091 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-16T19:22:44.116 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:44.116 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:44.166 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-04-16T19:22:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:44.267 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:44.300 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3089.gb0c7b917.e 361 kB/s | 142 kB 00:00 2024-04-16T19:22:44.316 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:35 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T19:22:44.400 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:44.417 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:44.417 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:44.417 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T19:22:44.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:44.446 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T19:22:44.478 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:44.479 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:44.479 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:44.479 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:44.479 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 890 k 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 968 k 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.8 M 2024-04-16T19:22:44.480 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:44.483 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 197 kB/s | 58 kB 00:00 2024-04-16T19:22:44.491 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:44.508 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:44.508 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:44.508 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T19:22:44.509 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:44.600 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-04-16T19:22:44.610 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-04-16T19:22:44.642 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-16T19:22:44.646 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:44.646 INFO:teuthology.orchestra.run.smithi119.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-04-16T19:22:44.659 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:44.669 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:44.669 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:44.757 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:44.757 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:44.935 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-19.0.0-3089.gb0c7b917.el9.x86_ 1.9 MB/s | 890 kB 00:00 2024-04-16T19:22:44.937 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:44.937 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-04-16T19:22:44.937 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:44.958 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:44.959 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:45.034 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:45.034 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:45.048 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:45.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 1/2 2024-04-16T19:22:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T19:22:45.231 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 1/2 2024-04-16T19:22:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T19:22:45.366 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T19:22:45.438 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:45.487 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T19:22:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:45.593 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T19:22:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:45.594 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:45.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T19:22:45.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2024-04-16T19:22:45.913 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T19:22:45.976 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T19:22:45.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-04-16T19:22:46.006 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:46.128 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:46.128 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/2 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:46.417 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:46.423 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:35 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:46.576 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:46.576 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:46.576 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 127 k 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:46.577 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:46.578 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 127 k 2024-04-16T19:22:46.578 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 456 k 2024-04-16T19:22:46.578 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:46.594 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:39 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:46.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-04-16T19:22:46.757 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:46.758 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:46.758 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-04-16T19:22:46.758 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:46.758 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 127 k 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-04-16T19:22:46.759 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 6/6 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 1/6 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T19:22:46.804 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T19:22:46.896 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-19.0.0-3089.gb0c7b917.el9.x86_64 398 kB/s | 127 kB 00:00 2024-04-16T19:22:46.897 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:46.897 INFO:teuthology.orchestra.run.smithi069.stdout:Total 396 kB/s | 127 kB 00:00 2024-04-16T19:22:46.897 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:46.903 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:46.903 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:46.959 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:46.959 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:47.074 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.0.0-3089.gb0c7b917.el9.x86_64 402 kB/s | 127 kB 00:00 2024-04-16T19:22:47.077 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:47.078 INFO:teuthology.orchestra.run.smithi151.stdout:Total 399 kB/s | 127 kB 00:00 2024-04-16T19:22:47.078 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:47.086 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:47.086 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:47.099 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:47.110 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:47.152 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:47.152 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:47.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:47.292 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:47.296 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:47.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-04-16T19:22:47.407 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:47.454 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 264 k 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:47.455 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-04-16T19:22:47.456 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:47.456 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 332 k 2024-04-16T19:22:47.456 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.5 M 2024-04-16T19:22:47.456 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:47.582 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 549 kB/s | 69 kB 00:00 2024-04-16T19:22:47.639 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:47.903 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:47.919 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:47.961 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:48.117 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:48.117 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:48.117 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:48.117 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 890 k 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-04-16T19:22:48.118 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:48.119 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 968 k 2024-04-16T19:22:48.119 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.8 M 2024-04-16T19:22:48.119 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:48.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2024-04-16T19:22:48.205 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:48.205 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:48.205 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:48.205 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:48.206 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:48.206 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:48.244 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 629 kB/s | 79 kB 00:00 2024-04-16T19:22:48.282 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-volume-19.0.0-3089.gb0c7b917.el9.no 319 kB/s | 264 kB 00:00 2024-04-16T19:22:48.283 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:48.284 INFO:teuthology.orchestra.run.smithi026.stdout:Total 401 kB/s | 332 kB 00:00 2024-04-16T19:22:48.284 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:48.294 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:48.294 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:48.335 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:48.336 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:48.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-04-16T19:22:48.492 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:48.681 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T19:22:48.709 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:48.725 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:48.725 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:48.725 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T19:22:48.726 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:48.768 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:48.869 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-19.0.0-3089.gb0c7b917.el9.x86_ 1.2 MB/s | 890 kB 00:00 2024-04-16T19:22:48.870 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:48.871 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.3 MB/s | 968 kB 00:00 2024-04-16T19:22:48.871 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:48.873 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:48.895 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:48.895 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:48.927 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:48.928 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:48.928 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:48.971 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:48.971 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:48.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2024-04-16T19:22:49.045 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:42 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:49.151 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:49.197 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:49.207 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:49.208 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:49.209 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:49.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-04-16T19:22:49.336 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T19:22:49.402 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:49.411 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 1/2 2024-04-16T19:22:49.579 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:49.733 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 32 k 2024-04-16T19:22:49.734 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 32 k 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 142 k 2024-04-16T19:22:49.735 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T19:22:49.761 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:49.762 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:49.879 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:42 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:49.976 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-19.0.0-3089.gb0c7b917.el9.x86_6 131 kB/s | 32 kB 00:00 2024-04-16T19:22:49.977 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:49.977 INFO:teuthology.orchestra.run.smithi069.stdout:Total 131 kB/s | 32 kB 00:00 2024-04-16T19:22:49.977 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:49.980 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:49.980 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:50.009 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:50.009 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:50.033 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:50.034 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-04-16T19:22:50.034 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:50.034 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 32 k 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 32 k 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 142 k 2024-04-16T19:22:50.035 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:50.115 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:50.174 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2/2 2024-04-16T19:22:50.174 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/2 2024-04-16T19:22:50.214 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:50.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-04-16T19:22:50.448 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.0.0-3089.gb0c7b917.el9.x86_6 77 kB/s | 32 kB 00:00 2024-04-16T19:22:50.449 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:50.449 INFO:teuthology.orchestra.run.smithi151.stdout:Total 76 kB/s | 32 kB 00:00 2024-04-16T19:22:50.449 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:50.453 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:50.453 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:50.475 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:50.476 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:50.482 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T19:22:50.482 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:50.482 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:50.483 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T19:22:50.483 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:50.483 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:50.630 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:50.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-04-16T19:22:50.890 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:50.891 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:50.891 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:50.891 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:50.891 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:50.891 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:50.904 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:51.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2024-04-16T19:22:51.068 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:51.068 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:51.068 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T19:22:51.068 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:51.068 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 127 k 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 127 k 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 456 k 2024-04-16T19:22:51.069 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:51.304 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-19.0.0-3089.gb0c7b917.el9.x86_64 540 kB/s | 127 kB 00:00 2024-04-16T19:22:51.304 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:51.304 INFO:teuthology.orchestra.run.smithi026.stdout:Total 537 kB/s | 127 kB 00:00 2024-04-16T19:22:51.304 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:51.311 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:51.311 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:51.362 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:51.362 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:51.362 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:51.362 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:51.362 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:51.363 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:51.369 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:51.369 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:51.381 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:33 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:51.518 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:51.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-04-16T19:22:51.535 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-3089.gb0c7b917.el9 ceph-noarch 264 k 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:51.536 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:51.537 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:51.537 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-04-16T19:22:51.537 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:51.537 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 332 k 2024-04-16T19:22:51.537 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.5 M 2024-04-16T19:22:51.538 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:51.646 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:40 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:51.656 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 580 kB/s | 69 kB 00:00 2024-04-16T19:22:51.747 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:51.789 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-volume-19.0.0-3089.gb0c7b917.el9.no 1.0 MB/s | 264 kB 00:00 2024-04-16T19:22:51.791 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:51.791 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-04-16T19:22:51.791 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:51.799 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:51.800 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:51.835 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:51.835 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:51.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2024-04-16T19:22:51.982 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:52.124 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:52.162 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T19:22:52.192 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2/2 2024-04-16T19:22:52.208 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:22:52.208 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T19:22:52.209 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:52.229 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:52.280 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:52.282 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:52.282 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:52.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:52.373 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:52.455 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:41 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:52.556 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:52.610 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:52.611 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:52.611 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:52.617 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-04-16T19:22:52.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2024-04-16T19:22:52.911 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 1/2 2024-04-16T19:22:52.948 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:53.049 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:53.101 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:53.102 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:53.102 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:53.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-04-16T19:22:53.227 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:53.251 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-3089.gb0c7b917.el9.noarch 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:53.252 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:53.259 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:42 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:53.333 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:53.359 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:53.390 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:53.391 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:53.391 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:53.410 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:53.412 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:53.412 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:53.457 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2024-04-16T19:22:53.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-04-16T19:22:53.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-04-16T19:22:53.747 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:46 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:53.849 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:53.900 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:53.902 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:53.902 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:53.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-04-16T19:22:54.049 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:43 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:54.081 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:54.150 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:54.174 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:36 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:54.202 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:54.203 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:54.203 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:54.248 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 32 k 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:54.249 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 32 k 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 142 k 2024-04-16T19:22:54.250 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:22:54.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 127 k 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:54.330 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 127 k 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 456 k 2024-04-16T19:22:54.331 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:54.493 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-19.0.0-3089.gb0c7b917.el9.x86_6 131 kB/s | 32 kB 00:00 2024-04-16T19:22:54.494 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:54.494 INFO:teuthology.orchestra.run.smithi026.stdout:Total 130 kB/s | 32 kB 00:00 2024-04-16T19:22:54.494 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:22:54.498 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:22:54.498 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:22:54.525 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:22:54.525 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:22:54.550 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:47 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:54.645 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-19.0.0-3089.gb0c7b917.el9.x86_64 402 kB/s | 127 kB 00:00 2024-04-16T19:22:54.646 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:54.646 INFO:teuthology.orchestra.run.smithi119.stdout:Total 400 kB/s | 127 kB 00:00 2024-04-16T19:22:54.647 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:54.651 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:54.652 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:22:54.652 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:54.652 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:54.703 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:54.704 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:54.704 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:54.707 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:54.707 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:54.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-04-16T19:22:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:54.851 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:43 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:54.852 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:54.951 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:54.956 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:55.009 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:55.010 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:55.010 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:55.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2024-04-16T19:22:55.173 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:55.354 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:55.414 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:55.436 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:22:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:22:55.437 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:55.457 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:55.509 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:55.510 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:55.510 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:55.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-04-16T19:22:55.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-04-16T19:22:55.666 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:55.691 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:55.768 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:55.819 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:55.820 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-16T19:22:55.821 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:55.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2024-04-16T19:22:55.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-04-16T19:22:56.159 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:49 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:56.232 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:56.260 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:56.313 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:56.314 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-04-16T19:22:56.314 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:56.337 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:56.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-04-16T19:22:56.392 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:56.393 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:56.393 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:56.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-04-16T19:22:56.472 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:56.564 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:56.627 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:56.628 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:56.628 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-04-16T19:22:56.628 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 85 k 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 85 k 2024-04-16T19:22:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 231 k 2024-04-16T19:22:56.630 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:56.673 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:56.732 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:56.733 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:22:56.733 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:56.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-04-16T19:22:56.945 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 270 kB/s | 85 kB 00:00 2024-04-16T19:22:56.946 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:56.946 INFO:teuthology.orchestra.run.smithi069.stdout:Total 269 kB/s | 85 kB 00:00 2024-04-16T19:22:56.946 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:56.953 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:56.953 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:56.979 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:49 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:57.008 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:57.009 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:57.053 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:57.135 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:57.136 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 85 k 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:57.138 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-04-16T19:22:57.138 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-04-16T19:22:57.138 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:57.157 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:22:57.158 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:57.212 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:57.213 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:57.213 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:57.268 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:57.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-04-16T19:22:57.414 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:39 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:57.444 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 278 kB/s | 85 kB 00:00 2024-04-16T19:22:57.445 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:57.445 INFO:teuthology.orchestra.run.smithi151.stdout:Total 277 kB/s | 85 kB 00:00 2024-04-16T19:22:57.445 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:22:57.452 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:22:57.452 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:22:57.507 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:22:57.508 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:22:57.568 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 32 k 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 32 k 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 142 k 2024-04-16T19:22:57.571 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:22:57.666 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:22:57.685 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:57.767 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:57.824 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-19.0.0-3089.gb0c7b917.el9.x86_6 125 kB/s | 32 kB 00:00 2024-04-16T19:22:57.825 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:57.826 INFO:teuthology.orchestra.run.smithi119.stdout:Total 124 kB/s | 32 kB 00:00 2024-04-16T19:22:57.826 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:22:57.829 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:22:57.829 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:22:57.860 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:22:57.860 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:22:57.878 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:57.982 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:22:57.983 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:58.005 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:22:58.038 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:58.039 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:58.040 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:58.089 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-04-16T19:22:58.248 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2024-04-16T19:22:58.477 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:58.545 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:22:58.706 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:49 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:22:58.748 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:58.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-04-16T19:22:58.811 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:58.865 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:58.867 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:58.867 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:58.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-04-16T19:22:58.922 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:47 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:22:58.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-04-16T19:22:59.079 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:22:59.080 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.1 M 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 13 M 2024-04-16T19:22:59.081 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:22:59.387 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:22:59.514 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:41 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:22:59.518 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:50 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:22:59.547 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:22:59.547 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-04-16T19:22:59.547 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:59.547 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:22:59.548 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-04-16T19:22:59.549 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-04-16T19:22:59.549 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:22:59.615 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:59.622 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:22:59.632 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-04-16T19:22:59.633 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:22:59.633 INFO:teuthology.orchestra.run.smithi069.stdout:Total 5.6 MB/s | 3.1 MB 00:00 2024-04-16T19:22:59.633 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:22:59.642 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:22:59.642 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:22:59.665 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:22:59.666 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:22:59.666 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:22:59.677 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:22:59.678 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:22:59.678 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:22:59.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-04-16T19:22:59.714 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:22:59.714 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:22:59.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-04-16T19:23:00.101 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-04-16T19:23:00.101 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:00.101 INFO:teuthology.orchestra.run.smithi151.stdout:Total 5.6 MB/s | 3.1 MB 00:00 2024-04-16T19:23:00.102 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:23:00.111 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:23:00.111 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:23:00.145 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:23:00.184 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:23:00.184 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:23:00.208 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:00.234 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:00.300 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:42 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:00.353 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:23:00.399 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:00.451 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:00.452 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:23:00.452 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:00.459 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:00.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-04-16T19:23:00.513 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:23:00.514 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-04-16T19:23:00.514 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:23:00.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-04-16T19:23:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:23:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:00.693 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:00.693 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:23:00.694 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T19:23:00.694 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:00.694 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:00.694 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:01.087 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:01.179 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:23:01.188 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:01.240 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:01.241 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:23:01.241 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:01.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-04-16T19:23:01.339 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 85 k 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:01.340 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:23:01.341 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:01.341 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 85 k 2024-04-16T19:23:01.341 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 231 k 2024-04-16T19:23:01.341 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:23:01.375 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:01.375 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:01.375 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:23:01.376 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:01.376 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:01.376 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:23:01.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:01.768 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:23:01.887 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 156 kB/s | 85 kB 00:00 2024-04-16T19:23:01.888 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:01.889 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:01.889 INFO:teuthology.orchestra.run.smithi026.stdout:Total 155 kB/s | 85 kB 00:00 2024-04-16T19:23:01.889 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:23:01.896 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:23:01.896 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:23:01.955 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:23:01.955 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:23:01.988 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:02.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-04-16T19:23:02.040 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:02.041 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:23:02.041 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:02.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-04-16T19:23:02.102 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:23:02.214 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:02.271 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:51 ago on Tue 16 Apr 2024 07:21:11 PM UTC. 2024-04-16T19:23:02.422 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 170 k 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-16T19:23:02.424 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-04-16T19:23:02.425 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:02.425 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 170 k 2024-04-16T19:23:02.425 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 494 k 2024-04-16T19:23:02.425 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-16T19:23:02.598 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:02.628 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:55 ago on Tue 16 Apr 2024 07:21:07 PM UTC. 2024-04-16T19:23:02.644 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 777 kB/s | 170 kB 00:00 2024-04-16T19:23:02.645 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:02.645 INFO:teuthology.orchestra.run.smithi069.stdout:Total 772 kB/s | 170 kB 00:00 2024-04-16T19:23:02.645 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-16T19:23:02.653 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-16T19:23:02.653 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-16T19:23:02.667 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:02.709 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-16T19:23:02.709 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-16T19:23:02.766 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:02.780 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 170 k 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:02.781 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 170 k 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-04-16T19:23:02.782 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-04-16T19:23:02.818 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:02.819 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:23:02.819 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:02.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-04-16T19:23:02.864 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-16T19:23:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:02.879 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:23:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:02.879 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:23:02.976 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:02.999 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 788 kB/s | 170 kB 00:00 2024-04-16T19:23:02.999 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:03.000 INFO:teuthology.orchestra.run.smithi151.stdout:Total 782 kB/s | 170 kB 00:00 2024-04-16T19:23:03.000 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-04-16T19:23:03.008 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-04-16T19:23:03.008 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-04-16T19:23:03.064 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-04-16T19:23:03.064 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-04-16T19:23:03.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-04-16T19:23:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-04-16T19:23:03.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:03.449 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:03.451 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:03.551 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 is already installed. 2024-04-16T19:23:03.603 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:03.604 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-16T19:23:03.604 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:03.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:03.757 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-16T19:23:03.759 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:03.789 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:54 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:23:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:23:03.949 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.1 M 2024-04-16T19:23:03.950 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 13 M 2024-04-16T19:23:03.951 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:23:03.965 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:04.037 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-04-16T19:23:04.246 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:46 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:04.288 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 85 k 2024-04-16T19:23:04.401 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 85 k 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 231 k 2024-04-16T19:23:04.402 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:23:04.490 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-04-16T19:23:04.491 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:04.492 INFO:teuthology.orchestra.run.smithi026.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-04-16T19:23:04.492 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:23:04.501 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:23:04.501 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:23:04.577 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:23:04.578 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:23:04.969 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 150 kB/s | 85 kB 00:00 2024-04-16T19:23:04.970 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:04.970 INFO:teuthology.orchestra.run.smithi119.stdout:Total 150 kB/s | 85 kB 00:00 2024-04-16T19:23:04.971 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:23:04.977 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:23:04.977 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:23:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:23:05.033 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:23:05.033 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:23:05.083 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:05.108 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:23:05.108 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T19:23:05.109 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:05.109 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:05.177 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:23:05.276 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:05.666 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:05.980 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:06.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-04-16T19:23:06.244 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:06.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:23:06.244 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:06.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:06.245 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:23:06.486 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-04-16T19:23:06.820 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:06.984 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:06.985 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 3.1 M 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:23:06.986 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:06.987 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.1 M 2024-04-16T19:23:06.987 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 13 M 2024-04-16T19:23:06.987 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:23:07.105 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:58 ago on Tue 16 Apr 2024 07:21:09 PM UTC. 2024-04-16T19:23:07.264 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 170 k 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:07.266 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 170 k 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 494 k 2024-04-16T19:23:07.267 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-04-16T19:23:07.579 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 5.2 MB/s | 3.1 MB 00:00 2024-04-16T19:23:07.581 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:07.581 INFO:teuthology.orchestra.run.smithi119.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-04-16T19:23:07.581 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:23:07.589 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:23:07.590 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:23:07.643 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 452 kB/s | 170 kB 00:00 2024-04-16T19:23:07.644 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:07.645 INFO:teuthology.orchestra.run.smithi026.stdout:Total 450 kB/s | 170 kB 00:00 2024-04-16T19:23:07.645 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-04-16T19:23:07.653 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-04-16T19:23:07.653 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-04-16T19:23:07.665 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:23:07.665 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:23:07.712 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-04-16T19:23:07.712 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-04-16T19:23:07.865 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-04-16T19:23:07.970 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:08.096 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:23:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:08.181 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T19:23:08.181 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T19:23:08.181 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:08.181 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T19:23:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:08.388 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:08.697 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:08.697 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-04-16T19:23:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:08.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:08.698 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-04-16T19:23:08.960 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:09.265 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:09.266 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:09.266 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:23:09.266 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:09.266 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:09.266 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:09.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-04-16T19:23:10.124 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Apr 2024 07:21:18 PM UTC. 2024-04-16T19:23:10.282 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-16T19:23:10.283 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-3089.gb0c7b917.el9 ceph 170 k 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-16T19:23:10.284 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:10.285 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 170 k 2024-04-16T19:23:10.285 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 494 k 2024-04-16T19:23:10.285 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-16T19:23:10.517 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-19.0.0-3089.gb0c7b917.el9.x86_64.rpm 735 kB/s | 170 kB 00:00 2024-04-16T19:23:10.517 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-16T19:23:10.518 INFO:teuthology.orchestra.run.smithi119.stdout:Total 729 kB/s | 170 kB 00:00 2024-04-16T19:23:10.518 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-16T19:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-16T19:23:10.526 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-16T19:23:10.581 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-16T19:23:10.582 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-16T19:23:10.723 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-16T19:23:10.840 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:11.295 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:11.590 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 1/1 2024-04-16T19:23:11.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:11.590 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-16T19:23:11.590 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-3089.gb0c7b917.el9.x86_64 2024-04-16T19:23:11.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:11.591 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-16T19:23:11.840 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:11.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:23:12.061 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T19:23:12.087 INFO:teuthology.orchestra.run.smithi026.stdout:19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.088 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.088 INFO:teuthology.task.install:The correct ceph version 19.0.0-3089.gb0c7b917 is installed. 2024-04-16T19:23:12.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:23:12.225 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T19:23:12.252 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.252 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-3089.gb0c7b917 is installed. 2024-04-16T19:23:12.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:23:12.395 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T19:23:12.417 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.417 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.418 INFO:teuthology.task.install:The correct ceph version 19.0.0-3089.gb0c7b917 is installed. 2024-04-16T19:23:12.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0c7b917267c471208541fdeb652f5e897d00086 2024-04-16T19:23:12.632 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T19:23:12.658 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.658 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3089.gb0c7b917.el9 2024-04-16T19:23:12.658 INFO:teuthology.task.install:The correct ceph version 19.0.0-3089.gb0c7b917 is installed. 2024-04-16T19:23:12.660 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T19:23:12.672 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:12.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T19:23:12.706 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:12.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T19:23:12.740 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:12.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T19:23:12.770 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:12.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T19:23:12.803 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T19:23:12.822 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:12.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T19:23:12.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T19:23:12.926 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:12.926 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T19:23:12.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T19:23:13.031 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:13.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T19:23:13.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T19:23:13.131 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:13.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T19:23:13.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T19:23:13.236 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T19:23:13.236 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:13.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T19:23:13.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T19:23:13.374 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:13.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T19:23:13.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T19:23:13.481 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:13.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T19:23:13.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T19:23:13.584 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:13.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T19:23:13.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T19:23:13.688 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T19:23:13.688 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:13.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T19:23:13.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T19:23:13.795 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:13.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T19:23:13.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T19:23:13.901 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:13.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T19:23:13.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T19:23:14.007 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:14.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T19:23:14.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T19:23:14.111 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T19:23:15.044 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T19:23:15.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-04-16T19:23:15.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-04-16T19:23:15.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-04-16T19:23:15.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2024-04-16T19:23:15.086 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T19:23:15.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T19:23:15.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T19:23:15.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T19:23:15.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T19:23:15.158 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T19:23:15.159 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T19:23:15.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T19:23:15.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T19:23:15.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T19:23:15.243 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-16T19:23:15.243 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T19:23:15.243 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'branch': 'wip-shilpa-multisite-tests', 'description': 'rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7658462', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-shilpa-multisite-tests'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding', 'notification_v2'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py'], 'extra_args': ['-a !fails_on_rgw']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-shilpa-multisite-tests', 'sha1': '9fbf23e1cbe3573736447d8751b80c9b42cadc24'}}, 'owner': 'scheduled_smanjara@teuthology', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'], ['c1.mgr.x', 'c1.osd.3', 'c1.osd.4', 'c1.osd.5', 'c1.client.1'], ['c2.mon.a', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0'], ['c2.mgr.x', 'c2.osd.3', 'c2.osd.4', 'c2.osd.5', 'c2.client.1']], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'wip-shilpa-multisite-tests', 'suite_path': '/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/smanjara/ceph.git', 'suite_sha1': '9fbf23e1cbe3573736447d8751b80c9b42cadc24', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFeEZqyh9FanBgXANxWy0pvFnKNo2hNbUFcqc4b4YSTlDKxftRrW04MORnc+2d/6ALkEqGynnxTKwFB0BDqSt6E=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM0TlHyXmVZ7jqn/rZpGycELmAJqs913ebWmQ2A0/heI3vGrv35mMKLp+X4G0DUqcRgFGEnaTPo3DjdbG/3Sps=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBn36P5Qab1DiTDnz/Z06PhUxTXcAilRT0rHTDZBY3vuVopfxS/BCCXXESn35uqPsSYOEgNDMs9qS+wcQpSIaBQ=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCVR1+NFSII3eGCZ2LQqmYNr8DPxw8UcAqsFGaEiPTDm+W8eOyUOvkV+Q0w306QFGyg9qJh1CvoFhvHYvh5aac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 90}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_19:05:08', 'tube': 'smithi', 'user': 'smanjara', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T19:23:15.244 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-16T19:23:15.265 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-16T19:23:15.268 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-16T19:23:15.272 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-16T19:23:15.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:15.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:15.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:15.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:15.361 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:15.361 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T19:23:15.407 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T19:23:15.407 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-04-16T19:23:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T19:23:15.463 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:15.463 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-16T19:23:15.463 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:15.463 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:15.464 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 19:23:05.354773001 +0000 2024-04-16T19:23:15.464 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 19:20:20.152810950 +0000 2024-04-16T19:23:15.464 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 19:20:20.152810950 +0000 2024-04-16T19:23:15.464 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 19:20:20.152810950 +0000 2024-04-16T19:23:15.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T19:23:15.532 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T19:23:15.532 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T19:23:15.532 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00010589 s, 4.8 MB/s 2024-04-16T19:23:15.533 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T19:23:15.591 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 19:23:05.355772988 +0000 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 19:20:20.492806484 +0000 2024-04-16T19:23:15.648 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 19:20:20.492806484 +0000 2024-04-16T19:23:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 19:20:20.492806484 +0000 2024-04-16T19:23:15.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T19:23:15.717 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T19:23:15.717 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T19:23:15.717 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000101979 s, 5.0 MB/s 2024-04-16T19:23:15.718 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T19:23:15.775 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 19:23:05.355772988 +0000 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 19:20:20.817802213 +0000 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 19:20:20.817802213 +0000 2024-04-16T19:23:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 19:20:20.817802213 +0000 2024-04-16T19:23:15.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T19:23:15.899 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T19:23:15.899 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T19:23:15.899 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.00010012 s, 5.1 MB/s 2024-04-16T19:23:15.901 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T19:23:15.958 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-04-16 19:23:05.355772988 +0000 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-04-16 19:20:21.156797759 +0000 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-04-16 19:20:21.156797759 +0000 2024-04-16T19:23:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-04-16 19:20:21.156797759 +0000 2024-04-16T19:23:16.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T19:23:16.081 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-04-16T19:23:16.081 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-04-16T19:23:16.082 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.7824e-05 s, 5.8 MB/s 2024-04-16T19:23:16.083 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T19:23:16.139 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:16.139 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:16.139 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T19:23:16.159 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T19:23:16.159 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-16T19:23:16.217 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-16 19:23:00.474227190 +0000 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-16 19:20:21.057805252 +0000 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-16 19:20:21.057805252 +0000 2024-04-16T19:23:16.218 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-16 19:20:21.057805252 +0000 2024-04-16T19:23:16.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T19:23:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-16T19:23:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-16T19:23:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000259169 s, 2.0 MB/s 2024-04-16T19:23:16.290 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T19:23:16.350 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-16T19:23:16.407 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T19:23:16.407 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:16.407 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-16T19:23:16.407 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:16.408 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:16.408 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-16 19:23:00.474227190 +0000 2024-04-16T19:23:16.408 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-16 19:20:21.410797169 +0000 2024-04-16T19:23:16.408 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-16 19:20:21.410797169 +0000 2024-04-16T19:23:16.408 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-16 19:20:21.410797169 +0000 2024-04-16T19:23:16.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T19:23:16.477 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-16T19:23:16.477 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-16T19:23:16.478 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000492703 s, 1.0 MB/s 2024-04-16T19:23:16.479 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T19:23:16.538 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-16T19:23:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T19:23:16.595 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-16 19:23:00.475227167 +0000 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-16 19:20:21.740789613 +0000 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-16 19:20:21.740789613 +0000 2024-04-16T19:23:16.596 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-16 19:20:21.740789613 +0000 2024-04-16T19:23:16.596 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T19:23:16.664 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-16T19:23:16.664 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-16T19:23:16.664 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000231915 s, 2.2 MB/s 2024-04-16T19:23:16.665 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T19:23:16.723 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:16.779 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-16 19:23:00.475227167 +0000 2024-04-16T19:23:16.780 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-16 19:20:22.081781805 +0000 2024-04-16T19:23:16.780 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-16 19:20:22.081781805 +0000 2024-04-16T19:23:16.780 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-16 19:20:22.081781805 +0000 2024-04-16T19:23:16.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T19:23:16.847 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-16T19:23:16.847 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-16T19:23:16.847 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000209082 s, 2.4 MB/s 2024-04-16T19:23:16.848 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T19:23:16.906 INFO:tasks.ceph:osd dev map: {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:16.906 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-16T19:23:16.906 INFO:tasks.ceph:Generating config... 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] debug rgw notification = 20 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-16T19:23:16.918 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw data sync poll interval = 5 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw meta sync poll interval = 5 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client] setuser = ceph 2024-04-16T19:23:16.919 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-16T19:23:16.920 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T19:23:16.920 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T19:23:16.920 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T19:23:16.920 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T19:23:16.920 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:16.920 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:16.921 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-16T19:23:16.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-16T19:23:16.982 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/c1.keyring 2024-04-16T19:23:16.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-16T19:23:17.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-16T19:23:17.064 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.26')] 2024-04-16T19:23:17.064 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.26'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-16T19:23:17.065 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:17.065 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T19:23:17.120 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.26 --print /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:23:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-04-16T19:23:17.202111+0000 2024-04-16T19:23:17.202 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-04-16T19:23:17.202111+0000 2024-04-16T19:23:17.203 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 17 (quincy) 2024-04-16T19:23:17.203 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 1 2024-04-16T19:23:17.203 INFO:teuthology.orchestra.run.smithi026.stdout:0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2024-04-16T19:23:17.203 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-16T19:23:17.204 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T19:23:17.260 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 46f1113d-f19a-4ad9-a219-ddfdf9343e7d... 2024-04-16T19:23:17.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-16T19:23:17.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-16T19:23:17.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-16T19:23:17.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-16T19:23:17.354 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-04-16T19:23:17.354 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.355 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-04-16T19:23:17.356 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-04-16T19:23:17.357 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = 172.21.15.26 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.358 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-04-16T19:23:17.359 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-04-16T19:23:17.360 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:17.361 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.362 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.363 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw = 20 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw notification = 20 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw sync = 20 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt require ssl = False 2024-04-16T19:23:17.364 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw curl low speed time = 300 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data log num shards = 4 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw md log max shards = 4 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: setgroup = ceph 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout: setuser = ceph 2024-04-16T19:23:17.365 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-04-16T19:23:17.366 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.367 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:17.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:17.369 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = 172.21.15.26 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.370 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:17.371 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug trim objects = true 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-04-16T19:23:17.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-04-16T19:23:17.373 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-16T19:23:17.374 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw = 20 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw notification = 20 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw sync = 20 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt require ssl = False 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw curl low speed time = 300 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data log num shards = 4 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw md log max shards = 4 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: setgroup = ceph 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: setuser = ceph 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-04-16T19:23:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.379 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-16T19:23:17.380 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-16T19:23:17.381 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.26 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:17.382 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.384 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-16T19:23:17.385 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:17.386 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-16T19:23:17.387 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw = 20 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw notification = 20 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw sync = 20 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt require ssl = False 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw curl low speed time = 300 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data log num shards = 4 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw md log max shards = 4 2024-04-16T19:23:17.388 INFO:teuthology.orchestra.run.smithi069.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: setgroup = ceph 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: setuser = ceph 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-04-16T19:23:17.389 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:17.392 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-04-16T19:23:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-04-16T19:23:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-04-16T19:23:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-04-16T19:23:17.394 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = 172.21.15.26 2024-04-16T19:23:17.395 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 46f1113d-f19a-4ad9-a219-ddfdf9343e7d 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-04-16T19:23:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug trim objects = true 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-04-16T19:23:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw = 20 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw notification = 20 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw sync = 20 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt require ssl = False 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw curl low speed time = 300 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data log num shards = 4 2024-04-16T19:23:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw md log max shards = 4 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: setgroup = ceph 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: setuser = ceph 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:17.402 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-04-16T19:23:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:17.403 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-04-16T19:23:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:17.403 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-16T19:23:17.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-16T19:23:17.490 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T19:23:17.491 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:17.491 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-16T19:23:17.506 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:17.506 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-16T19:23:17.562 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:23:17.562 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:17.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/c1.keyring 2024-04-16T19:23:17.562 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-16T19:23:17.643 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:17.643 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:17.699 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:23:17.699 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:17.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/c1.keyring 2024-04-16T19:23:17.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-16T19:23:17.744 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:17.744 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:17.801 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:23:17.801 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:17.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/c1.keyring 2024-04-16T19:23:17.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-16T19:23:17.848 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:17.848 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:17.905 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:23:17.906 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:17.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/c1.keyring 2024-04-16T19:23:17.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-16T19:23:17.949 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:17.949 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:18.005 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T19:23:18.005 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T19:23:18.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-16T19:23:18.076 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-16T19:23:18.079 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T19:23:18.079 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T19:23:18.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-16T19:23:18.126 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-16T19:23:18.142 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-16T19:23:18.187 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-16T19:23:18.203 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T19:23:18.204 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-16T19:23:18.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-16T19:23:18.232 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:18.233 INFO:tasks.ceph:role: c1.osd.0 2024-04-16T19:23:18.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:23:18.233 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T19:23:18.300 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:18.301 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:18.333 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-04-16T19:23:18.336 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-04-16T19:23:18.336 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-16T19:23:18.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-16T19:23:18.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-16T19:23:18.548 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:18.548 INFO:tasks.ceph:role: c1.osd.1 2024-04-16T19:23:18.548 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:23:18.548 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:18.616 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:18.617 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:18.617 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:18.617 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:18.617 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:18.649 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-04-16T19:23:18.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-04-16T19:23:18.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-16T19:23:18.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-16T19:23:18.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-16T19:23:18.863 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:18.863 INFO:tasks.ceph:role: c1.osd.2 2024-04-16T19:23:18.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:23:18.864 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:18.933 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:18.933 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:18.933 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:18.965 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-04-16T19:23:18.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-04-16T19:23:18.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-16T19:23:19.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-16T19:23:19.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-16T19:23:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:19.214+0000 7f48c6773640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-16T19:23:21.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-16T19:23:21.333 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-16T19:23:21.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:21.426+0000 7f9e7e214640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-16T19:23:23.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-16T19:23:23.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:23.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.626+0000 7f0258ea0640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-16T19:23:23.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.626+0000 7f0258ea0640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-16T19:23:23.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.626+0000 7f0258ea0640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-16T19:23:23.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.626+0000 7f0258ea0640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-16T19:23:23.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.626+0000 7f0258ea0640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-16T19:23:23.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:23:23.627+0000 7f0258ea0640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-16T19:23:25.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-16T19:23:25.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c1-3 2024-04-16T19:23:25.767 INFO:tasks.ceph:roles_to_devs: {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:25.767 INFO:tasks.ceph:role: c1.osd.3 2024-04-16T19:23:25.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:23:25.768 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T19:23:25.846 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:25.846 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:25.846 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:25.846 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:25.847 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:26.341 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-16T19:23:26.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-16T19:23:26.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-3 2024-04-16T19:23:26.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-3 2024-04-16T19:23:26.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c1-4 2024-04-16T19:23:26.597 INFO:tasks.ceph:roles_to_devs: {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:26.597 INFO:tasks.ceph:role: c1.osd.4 2024-04-16T19:23:26.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:23:26.597 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:26.675 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:27.116 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-16T19:23:27.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-16T19:23:27.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-4 2024-04-16T19:23:27.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-4 2024-04-16T19:23:27.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/c1-5 2024-04-16T19:23:27.319 INFO:tasks.ceph:roles_to_devs: {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:27.319 INFO:tasks.ceph:role: c1.osd.5 2024-04-16T19:23:27.319 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:23:27.319 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:27.397 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:27.884 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-16T19:23:27.886 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-16T19:23:27.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-5 2024-04-16T19:23:28.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-5 2024-04-16T19:23:28.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:28.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.168+0000 7f263f3aa640 -1 auth: error reading file: /var/lib/ceph/osd/c1-3/keyring: can't open /var/lib/ceph/osd/c1-3/keyring: (2) No such file or directory 2024-04-16T19:23:28.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.171+0000 7f263f3aa640 -1 created new key in keyring /var/lib/ceph/osd/c1-3/keyring 2024-04-16T19:23:28.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.171+0000 7f263f3aa640 -1 bluestore(/var/lib/ceph/osd/c1-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-3/block: (2) No such file or directory 2024-04-16T19:23:28.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.171+0000 7f263f3aa640 -1 bluestore(/var/lib/ceph/osd/c1-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-3/block: (2) No such file or directory 2024-04-16T19:23:28.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.171+0000 7f263f3aa640 -1 bluestore(/var/lib/ceph/osd/c1-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-3/block: (2) No such file or directory 2024-04-16T19:23:28.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:28.178+0000 7f263f3aa640 -1 bluestore(/var/lib/ceph/osd/c1-3) _read_fsid unparsable uuid 2024-04-16T19:23:30.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-3 2024-04-16T19:23:30.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:30.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.392+0000 7fe353ca8640 -1 auth: error reading file: /var/lib/ceph/osd/c1-4/keyring: can't open /var/lib/ceph/osd/c1-4/keyring: (2) No such file or directory 2024-04-16T19:23:30.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.392+0000 7fe353ca8640 -1 created new key in keyring /var/lib/ceph/osd/c1-4/keyring 2024-04-16T19:23:30.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.392+0000 7fe353ca8640 -1 bluestore(/var/lib/ceph/osd/c1-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-4/block: (2) No such file or directory 2024-04-16T19:23:30.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.392+0000 7fe353ca8640 -1 bluestore(/var/lib/ceph/osd/c1-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-4/block: (2) No such file or directory 2024-04-16T19:23:30.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.392+0000 7fe353ca8640 -1 bluestore(/var/lib/ceph/osd/c1-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-4/block: (2) No such file or directory 2024-04-16T19:23:30.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:30.393+0000 7fe353ca8640 -1 bluestore(/var/lib/ceph/osd/c1-4) _read_fsid unparsable uuid 2024-04-16T19:23:32.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-4 2024-04-16T19:23:32.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.601+0000 7f0694415640 -1 auth: error reading file: /var/lib/ceph/osd/c1-5/keyring: can't open /var/lib/ceph/osd/c1-5/keyring: (2) No such file or directory 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.601+0000 7f0694415640 -1 created new key in keyring /var/lib/ceph/osd/c1-5/keyring 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.601+0000 7f0694415640 -1 bluestore(/var/lib/ceph/osd/c1-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-5/block: (2) No such file or directory 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.601+0000 7f0694415640 -1 bluestore(/var/lib/ceph/osd/c1-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-5/block: (2) No such file or directory 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.601+0000 7f0694415640 -1 bluestore(/var/lib/ceph/osd/c1-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-5/block: (2) No such file or directory 2024-04-16T19:23:32.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:23:32.602+0000 7f0694415640 -1 bluestore(/var/lib/ceph/osd/c1-5) _read_fsid unparsable uuid 2024-04-16T19:23:34.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-5 2024-04-16T19:23:34.722 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T19:23:34.723 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:34.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-16T19:23:34.757 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:34.757 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-16T19:23:34.829 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:34.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-16T19:23:34.902 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:34.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-16T19:23:34.935 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:34.935 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-3/keyring of=/dev/stdout 2024-04-16T19:23:35.007 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:35.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-4/keyring of=/dev/stdout 2024-04-16T19:23:35.077 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:35.077 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-5/keyring of=/dev/stdout 2024-04-16T19:23:35.146 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:35.146 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-16T19:23:35.165 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:35.165 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-16T19:23:35.203 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T19:23:35.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/c1.keyring 2024-04-16T19:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-04-16T19:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAn0B5mWKHQDBAApNKNqWYv5d6iBhzWe0gNNQ== 2024-04-16T19:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-04-16T19:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAp0B5mFFVzGRAAuuTzYUEd3vUVNJ8WKzRpNQ== 2024-04-16T19:23:35.235 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAr0B5mBhlnJRAAYBw74OFwbfknUxT/ja02Zg== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAm0B5md9SMBBAAAQ9F1/zP6Qv/zx6qxxKZHw== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAw0B5mmWMeChAA6YcUp6abLaqPeLfXYGJuoA== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAy0B5m6kxuFxAAf+2OCkIjJenuvxlLn/QZ1w== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQA00B5mZp7oIxAAk5aDIpVOeYvcCmI4MHKOTQ== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAm0B5mGsWGBxAAs57pXQf4YwDwVOE/E2G/Pg== 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout:[client.1] 2024-04-16T19:23:35.236 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQAm0B5mbDguCxAAm+vy+NN0JIAAqESG4uoLRA== 2024-04-16T19:23:35.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.448 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T19:23:35.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:23:35.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T19:23:35.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T19:23:35.779 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T19:23:35.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-16T19:23:35.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-16T19:23:35.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-16T19:23:35.932 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-16T19:23:35.987 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-16T19:23:35.988 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-16T19:23:35.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-16T19:23:36.030 INFO:tasks.ceph.c1.mon.a:Started 2024-04-16T19:23:36.030 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-16T19:23:36.030 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-16T19:23:36.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-16T19:23:36.033 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-16T19:23:36.033 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T19:23:36.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd crush tunables default 2024-04-16T19:23:36.211 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-04-16T19:23:36.223 INFO:tasks.ceph:check_enable_crimson: False 2024-04-16T19:23:36.223 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-16T19:23:36.223 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:36.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-16T19:23:36.251 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:36.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-16T19:23:36.319 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:36.319 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-16T19:23:36.387 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:36.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-3/fsid of=/dev/stdout 2024-04-16T19:23:36.416 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:36.416 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-4/fsid of=/dev/stdout 2024-04-16T19:23:36.484 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:36.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/c1-5/fsid of=/dev/stdout 2024-04-16T19:23:36.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new 097ebd2b-2937-4f76-8474-06350d6fcb0a 0 2024-04-16T19:23:36.617 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:36.616+0000 7fc79923b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T19:23:36.673 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:36.672+0000 7fc79923b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T19:23:36.778 INFO:teuthology.orchestra.run.smithi069.stdout:0 2024-04-16T19:23:36.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c 1 2024-04-16T19:23:37.029 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-16T19:23:37.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new 0965bd9d-6a0a-4f22-b3a5-e78191cf4526 2 2024-04-16T19:23:37.222 INFO:teuthology.orchestra.run.smithi069.stdout:2 2024-04-16T19:23:37.230 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.229+0000 7fc79923b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T19:23:37.234 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new 485c1994-8091-434c-b395-8a6aef81561d 3 2024-04-16T19:23:37.289 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.288+0000 7fc79923b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T19:23:37.337 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.336+0000 7fc79923b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T19:23:37.393 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.392+0000 7fc79923b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T19:23:37.421 INFO:teuthology.orchestra.run.smithi069.stdout:3 2024-04-16T19:23:37.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new 0da3d944-0b26-4daa-93f4-c5b7de1d72ce 4 2024-04-16T19:23:37.437 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.436+0000 7fc79923b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T19:23:37.560 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.559+0000 7fc79923b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T19:23:37.615 INFO:teuthology.orchestra.run.smithi069.stdout:4 2024-04-16T19:23:37.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster c1 osd new 82f781fe-193b-4e5f-813b-8645acd09ab9 5 2024-04-16T19:23:37.632 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.631+0000 7fc79923b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T19:23:37.681 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.680+0000 7fc79923b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T19:23:37.737 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.736+0000 7fc79923b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T19:23:37.786 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:37.785+0000 7fc79923b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T19:23:37.816 INFO:teuthology.orchestra.run.smithi069.stdout:5 2024-04-16T19:23:37.829 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-16T19:23:37.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-16T19:23:37.832 INFO:tasks.ceph.c1.osd.0:Started 2024-04-16T19:23:37.832 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-16T19:23:37.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-16T19:23:37.834 INFO:tasks.ceph.c1.osd.1:Started 2024-04-16T19:23:37.835 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-16T19:23:37.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-16T19:23:37.837 INFO:tasks.ceph.c1.osd.2:Started 2024-04-16T19:23:37.837 INFO:tasks.ceph.c1.osd.3:Restarting daemon 2024-04-16T19:23:37.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 3 2024-04-16T19:23:37.840 INFO:tasks.ceph.c1.osd.3:Started 2024-04-16T19:23:37.840 INFO:tasks.ceph.c1.osd.4:Restarting daemon 2024-04-16T19:23:37.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 4 2024-04-16T19:23:37.842 INFO:tasks.ceph.c1.osd.4:Started 2024-04-16T19:23:37.842 INFO:tasks.ceph.c1.osd.5:Restarting daemon 2024-04-16T19:23:37.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 5 2024-04-16T19:23:37.845 INFO:tasks.ceph.c1.osd.5:Started 2024-04-16T19:23:37.845 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-16T19:23:38.033 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:38.033 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:23:37.815629+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:23:38.044 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-16T19:23:38.045 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T19:23:38.182 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.181+0000 7fc79923b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T19:23:38.317 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.316+0000 7fc79923b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T19:23:38.433 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.431+0000 7fc79923b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T19:23:38.439 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2024-04-16T19:23:38.438+0000 7f1ecf011640 -1 Falling back to public interface 2024-04-16T19:23:38.439 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2024-04-16T19:23:38.438+0000 7fc45f60d640 -1 Falling back to public interface 2024-04-16T19:23:38.440 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2024-04-16T19:23:38.438+0000 7fc45df96640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-04-16T19:23:38.440 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2024-04-16T19:23:38.438+0000 7f1ecd928640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-04-16T19:23:38.443 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2024-04-16T19:23:38.442+0000 7f9b1fdff640 -1 Falling back to public interface 2024-04-16T19:23:38.464 INFO:tasks.ceph.c1.osd.5.smithi069.stderr:2024-04-16T19:23:38.463+0000 7fc634619640 -1 Falling back to public interface 2024-04-16T19:23:38.465 INFO:tasks.ceph.c1.osd.3.smithi069.stderr:2024-04-16T19:23:38.464+0000 7fda56549640 -1 Falling back to public interface 2024-04-16T19:23:38.465 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:38.464+0000 7fc4fe76b640 -1 Falling back to public interface 2024-04-16T19:23:38.465 INFO:tasks.ceph.c1.osd.3.smithi069.stderr:2024-04-16T19:23:38.464+0000 7fda554f8640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2024-04-16T19:23:38.466 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:38.464+0000 7fc4fd71a640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6806/0: (98) Address already in use 2024-04-16T19:23:38.466 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:38.464+0000 7fc4fd71a640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6807/0: (98) Address already in use 2024-04-16T19:23:38.481 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.480+0000 7fc79923b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T19:23:38.593 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.592+0000 7fc79923b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T19:23:38.649 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-16T19:23:38.687 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.686+0000 7fc79923b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T19:23:38.816 INFO:teuthology.misc.health.smithi026.stdout: 2024-04-16T19:23:38.816 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":8,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:23:37.815629+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:23:38.826 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-16T19:23:38.843 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.842+0000 7fc79923b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T19:23:38.885 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:38.884+0000 7fc79923b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T19:23:39.041 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:39.040+0000 7fc79923b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T19:23:39.858 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:39.857+0000 7fc79923b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T19:23:40.132 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:40.131+0000 7fc79923b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T19:23:40.182 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:40.181+0000 7fc79923b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T19:23:40.295 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:40.294+0000 7fc79923b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T19:23:41.110 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2024-04-16T19:23:41.109+0000 7fc45f60d640 -1 osd.1 0 log_to_monitors true 2024-04-16T19:23:41.154 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2024-04-16T19:23:41.153+0000 7f1ecf011640 -1 osd.0 0 log_to_monitors true 2024-04-16T19:23:41.155 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:41.154+0000 7fc4fe76b640 -1 osd.4 0 log_to_monitors true 2024-04-16T19:23:41.157 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2024-04-16T19:23:41.156+0000 7f9b1fdff640 -1 osd.2 0 log_to_monitors true 2024-04-16T19:23:41.203 INFO:tasks.ceph.c1.osd.5.smithi069.stderr:2024-04-16T19:23:41.202+0000 7fc634619640 -1 osd.5 0 log_to_monitors true 2024-04-16T19:23:41.205 INFO:tasks.ceph.c1.osd.3.smithi069.stderr:2024-04-16T19:23:41.204+0000 7fda56549640 -1 osd.3 0 log_to_monitors true 2024-04-16T19:23:41.304 INFO:tasks.ceph.c1.mgr.x.smithi069.stderr:2024-04-16T19:23:41.303+0000 7fc739e67640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T19:23:43.097 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2024-04-16T19:23:43.096+0000 7f1eca90f640 -1 osd.0 0 waiting for initial osdmap 2024-04-16T19:23:43.100 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2024-04-16T19:23:43.099+0000 7f9b1bd95640 -1 osd.2 0 waiting for initial osdmap 2024-04-16T19:23:43.106 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2024-04-16T19:23:43.105+0000 7fc45af7d640 -1 osd.1 0 waiting for initial osdmap 2024-04-16T19:23:43.107 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2024-04-16T19:23:43.107+0000 7f1ec272f640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:43.112 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2024-04-16T19:23:43.111+0000 7f9b13bb5640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:43.118 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2024-04-16T19:23:43.117+0000 7fc452d9d640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:43.563 INFO:tasks.ceph.c1.osd.5.smithi069.stderr:2024-04-16T19:23:43.561+0000 7fc630891640 -1 osd.5 0 waiting for initial osdmap 2024-04-16T19:23:43.565 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:43.564+0000 7fc4faf15640 -1 osd.4 0 waiting for initial osdmap 2024-04-16T19:23:43.579 INFO:tasks.ceph.c1.osd.4.smithi069.stderr:2024-04-16T19:23:43.578+0000 7fc4f2521640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:43.579 INFO:tasks.ceph.c1.osd.5.smithi069.stderr:2024-04-16T19:23:43.578+0000 7fc627e9d640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:43.595 INFO:tasks.ceph.c1.osd.3.smithi069.stderr:2024-04-16T19:23:43.594+0000 7fda524df640 -1 osd.3 0 waiting for initial osdmap 2024-04-16T19:23:43.605 INFO:tasks.ceph.c1.osd.3.smithi069.stderr:2024-04-16T19:23:43.604+0000 7fda4a2ff640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:23:45.429 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-16T19:23:45.729 INFO:teuthology.misc.health.smithi026.stdout: 2024-04-16T19:23:45.729 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":12,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:23:45.108082+0000","last_up_change":"2024-04-16T19:23:44.107341+0000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T19:23:44.317491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6802","nonce":562010544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6806","nonce":562010544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6814","nonce":562010544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6810","nonce":562010544}]},"public_addr":"172.21.15.26:6802/562010544","cluster_addr":"172.21.15.26:6806/562010544","heartbeat_back_addr":"172.21.15.26:6814/562010544","heartbeat_front_addr":"172.21.15.26:6810/562010544","state":["exists","up"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6801","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6803","nonce":930307441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6807","nonce":930307441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6815","nonce":930307441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6811","nonce":930307441}]},"public_addr":"172.21.15.26:6803/930307441","cluster_addr":"172.21.15.26:6807/930307441","heartbeat_back_addr":"172.21.15.26:6815/930307441","heartbeat_front_addr":"172.21.15.26:6811/930307441","state":["exists","up"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6817","nonce":1193744923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6819","nonce":1193744923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6823","nonce":1193744923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6821","nonce":1193744923}]},"public_addr":"172.21.15.26:6817/1193744923","cluster_addr":"172.21.15.26:6819/1193744923","heartbeat_back_addr":"172.21.15.26:6823/1193744923","heartbeat_front_addr":"172.21.15.26:6821/1193744923","state":["exists","up"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6806","nonce":800096592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6811","nonce":800096592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6821","nonce":800096592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6817","nonce":800096592}]},"public_addr":"172.21.15.69:6806/800096592","cluster_addr":"172.21.15.69:6811/800096592","heartbeat_back_addr":"172.21.15.69:6821/800096592","heartbeat_front_addr":"172.21.15.69:6817/800096592","state":["exists","up"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6808","nonce":3493128902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6816","nonce":3493128902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6823","nonce":3493128902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6820","nonce":3493128902}]},"public_addr":"172.21.15.69:6808/3493128902","cluster_addr":"172.21.15.69:6816/3493128902","heartbeat_back_addr":"172.21.15.69:6823/3493128902","heartbeat_front_addr":"172.21.15.69:6820/3493128902","state":["exists","up"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6801","nonce":3895908262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6805","nonce":3895908262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6815","nonce":3895908262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6810","nonce":3895908262}]},"public_addr":"172.21.15.69:6801/3895908262","cluster_addr":"172.21.15.69:6805/3895908262","heartbeat_back_addr":"172.21.15.69:6815/3895908262","heartbeat_front_addr":"172.21.15.69:6810/3895908262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.125284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.152292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.113122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.161231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.162925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.192596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:23:45.741 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-16T19:23:45.741 INFO:tasks.ceph:Creating RBD pool 2024-04-16T19:23:45.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-16T19:23:46.110 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2024-04-16T19:23:46.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T19:23:47.114 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T19:23:47.130 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-16T19:23:47.131 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-16T19:23:47.131 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-16T19:23:47.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-16T19:23:47.132 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T19:23:47.489 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:47.500 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2159975856},{"type":"v1","addr":"172.21.15.69:6825","nonce":2159975856}]},"active_addr":"172.21.15.69:6825/2159975856","active_change":"2024-04-16T19:23:40.297318+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3390529766}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3178623861}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":994462894}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":4047488659}]}]} 2024-04-16T19:23:47.503 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-16T19:23:47.503 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-16T19:23:47.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-16T19:23:47.802 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:47.803 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:23:47.112796+0000","last_up_change":"2024-04-16T19:23:44.107341+0000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T19:23:44.317491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T19:23:46.042846+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6802","nonce":562010544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6806","nonce":562010544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6814","nonce":562010544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6810","nonce":562010544}]},"public_addr":"172.21.15.26:6802/562010544","cluster_addr":"172.21.15.26:6806/562010544","heartbeat_back_addr":"172.21.15.26:6814/562010544","heartbeat_front_addr":"172.21.15.26:6810/562010544","state":["exists","up"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6801","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6803","nonce":930307441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6807","nonce":930307441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6815","nonce":930307441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6811","nonce":930307441}]},"public_addr":"172.21.15.26:6803/930307441","cluster_addr":"172.21.15.26:6807/930307441","heartbeat_back_addr":"172.21.15.26:6815/930307441","heartbeat_front_addr":"172.21.15.26:6811/930307441","state":["exists","up"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6817","nonce":1193744923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6819","nonce":1193744923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6823","nonce":1193744923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6821","nonce":1193744923}]},"public_addr":"172.21.15.26:6817/1193744923","cluster_addr":"172.21.15.26:6819/1193744923","heartbeat_back_addr":"172.21.15.26:6823/1193744923","heartbeat_front_addr":"172.21.15.26:6821/1193744923","state":["exists","up"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6806","nonce":800096592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6811","nonce":800096592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6821","nonce":800096592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6817","nonce":800096592}]},"public_addr":"172.21.15.69:6806/800096592","cluster_addr":"172.21.15.69:6811/800096592","heartbeat_back_addr":"172.21.15.69:6821/800096592","heartbeat_front_addr":"172.21.15.69:6817/800096592","state":["exists","up"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6808","nonce":3493128902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6816","nonce":3493128902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6823","nonce":3493128902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6820","nonce":3493128902}]},"public_addr":"172.21.15.69:6808/3493128902","cluster_addr":"172.21.15.69:6816/3493128902","heartbeat_back_addr":"172.21.15.69:6823/3493128902","heartbeat_front_addr":"172.21.15.69:6820/3493128902","state":["exists","up"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6801","nonce":3895908262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6805","nonce":3895908262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6815","nonce":3895908262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6810","nonce":3895908262}]},"public_addr":"172.21.15.69:6801/3895908262","cluster_addr":"172.21.15.69:6805/3895908262","heartbeat_back_addr":"172.21.15.69:6815/3895908262","heartbeat_front_addr":"172.21.15.69:6810/3895908262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.125284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.152292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.113122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.161231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.162925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.192596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:23:47.815 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-16T19:23:47.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-16T19:23:48.118 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:48.118 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:23:47.112796+0000","last_up_change":"2024-04-16T19:23:44.107341+0000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T19:23:44.317491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T19:23:46.042846+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6802","nonce":562010544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6806","nonce":562010544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6814","nonce":562010544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6810","nonce":562010544}]},"public_addr":"172.21.15.26:6802/562010544","cluster_addr":"172.21.15.26:6806/562010544","heartbeat_back_addr":"172.21.15.26:6814/562010544","heartbeat_front_addr":"172.21.15.26:6810/562010544","state":["exists","up"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6801","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6803","nonce":930307441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6807","nonce":930307441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6815","nonce":930307441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6811","nonce":930307441}]},"public_addr":"172.21.15.26:6803/930307441","cluster_addr":"172.21.15.26:6807/930307441","heartbeat_back_addr":"172.21.15.26:6815/930307441","heartbeat_front_addr":"172.21.15.26:6811/930307441","state":["exists","up"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6817","nonce":1193744923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6819","nonce":1193744923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6823","nonce":1193744923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6821","nonce":1193744923}]},"public_addr":"172.21.15.26:6817/1193744923","cluster_addr":"172.21.15.26:6819/1193744923","heartbeat_back_addr":"172.21.15.26:6823/1193744923","heartbeat_front_addr":"172.21.15.26:6821/1193744923","state":["exists","up"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6806","nonce":800096592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6811","nonce":800096592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6821","nonce":800096592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6817","nonce":800096592}]},"public_addr":"172.21.15.69:6806/800096592","cluster_addr":"172.21.15.69:6811/800096592","heartbeat_back_addr":"172.21.15.69:6821/800096592","heartbeat_front_addr":"172.21.15.69:6817/800096592","state":["exists","up"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6808","nonce":3493128902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6816","nonce":3493128902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6823","nonce":3493128902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6820","nonce":3493128902}]},"public_addr":"172.21.15.69:6808/3493128902","cluster_addr":"172.21.15.69:6816/3493128902","heartbeat_back_addr":"172.21.15.69:6823/3493128902","heartbeat_front_addr":"172.21.15.69:6820/3493128902","state":["exists","up"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6801","nonce":3895908262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6805","nonce":3895908262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6815","nonce":3895908262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6810","nonce":3895908262}]},"public_addr":"172.21.15.69:6801/3895908262","cluster_addr":"172.21.15.69:6805/3895908262","heartbeat_back_addr":"172.21.15.69:6815/3895908262","heartbeat_front_addr":"172.21.15.69:6810/3895908262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.125284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.152292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.113122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.161231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.162925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.192596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:23:48.130 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-16T19:23:48.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-16T19:23:48.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-16T19:23:48.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.3 flush_pg_stats 2024-04-16T19:23:48.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.4 flush_pg_stats 2024-04-16T19:23:48.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.5 flush_pg_stats 2024-04-16T19:23:48.316 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:48.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-16T19:23:48.317 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.5 2024-04-16T19:23:48.341 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:48.342 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.4 2024-04-16T19:23:48.349 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.3 2024-04-16T19:23:48.355 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.355 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-16T19:23:48.359 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:48.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-16T19:23:48.722 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-04-16T19:23:48.740 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640258 for osd.5 2024-04-16T19:23:48.783 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-04-16T19:23:48.797 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640258 for osd.0 2024-04-16T19:23:48.807 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.822 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640259 for osd.2 2024-04-16T19:23:48.859 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-04-16T19:23:48.865 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.870 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:48.872 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640258 for osd.3 2024-04-16T19:23:48.878 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640259 for osd.1 2024-04-16T19:23:48.884 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640259 for osd.4 2024-04-16T19:23:49.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.5 2024-04-16T19:23:49.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-16T19:23:49.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-16T19:23:49.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.3 2024-04-16T19:23:49.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-16T19:23:49.885 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.4 2024-04-16T19:23:50.096 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:50.109 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640259 for osd.5 2024-04-16T19:23:50.110 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:50.254 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-04-16T19:23:50.268 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640258 for osd.3 2024-04-16T19:23:50.284 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:50.297 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640259 for osd.1 2024-04-16T19:23:50.306 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-04-16T19:23:50.320 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640259 for osd.4 2024-04-16T19:23:50.321 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-04-16T19:23:50.335 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640258 for osd.0 2024-04-16T19:23:50.357 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:50.370 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640260 for osd.2 2024-04-16T19:23:50.371 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:51.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.3 2024-04-16T19:23:51.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-16T19:23:51.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.4 2024-04-16T19:23:51.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-16T19:23:51.617 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:51.634 INFO:teuthology.orchestra.run.smithi026.stdout:47244640261 2024-04-16T19:23:51.635 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640260 for osd.3 2024-04-16T19:23:51.635 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:51.648 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640261 for osd.1 2024-04-16T19:23:51.648 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:51.684 INFO:teuthology.orchestra.run.smithi026.stdout:47244640261 2024-04-16T19:23:51.697 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640260 got 47244640261 for osd.4 2024-04-16T19:23:51.697 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:51.699 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-04-16T19:23:51.712 INFO:tasks.ceph.ceph_manager.c1:need seq 47244640259 got 47244640260 for osd.0 2024-04-16T19:23:51.712 DEBUG:teuthology.parallel:result is None 2024-04-16T19:23:51.712 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-16T19:23:51.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T19:23:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T19:23:52.066 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T19:23:50.302526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":164836,"kb_used_data":1952,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":628980764,"statfs":{"total":644245094400,"available":644076302336,"internally_reserved":0,"allocated":1998848,"data_stored":1421810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.177982+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T19:23:47.177982+0000","last_peered":"2024-04-16T19:23:47.177982+0000","last_clean":"2024-04-16T19:23:47.177982+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T19:23:47.177982+0000","last_undegraded":"2024-04-16T19:23:47.177982+0000","last_fullsized":"2024-04-16T19:23:47.177982+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:49.312948+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T19:23:49.312948+0000","last_peered":"2024-04-16T19:23:49.312948+0000","last_clean":"2024-04-16T19:23:49.312948+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T19:23:49.312948+0000","last_undegraded":"2024-04-16T19:23:49.312948+0000","last_fullsized":"2024-04-16T19:23:49.312948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.187320+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T19:23:47.187320+0000","last_peered":"2024-04-16T19:23:47.187320+0000","last_clean":"2024-04-16T19:23:47.187320+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T19:23:47.187320+0000","last_undegraded":"2024-04-16T19:23:47.187320+0000","last_fullsized":"2024-04-16T19:23:47.187320+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:49.312959+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T19:23:49.312959+0000","last_peered":"2024-04-16T19:23:49.312959+0000","last_clean":"2024-04-16T19:23:49.312959+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T19:23:49.312959+0000","last_undegraded":"2024-04-16T19:23:49.312959+0000","last_fullsized":"2024-04-16T19:23:49.312959+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:48.118679+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T19:23:48.118679+0000","last_peered":"2024-04-16T19:23:48.118679+0000","last_clean":"2024-04-16T19:23:48.118679+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T19:23:48.118679+0000","last_undegraded":"2024-04-16T19:23:48.118679+0000","last_fullsized":"2024-04-16T19:23:48.118679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.120183+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T19:23:47.120183+0000","last_peered":"2024-04-16T19:23:47.120183+0000","last_clean":"2024-04-16T19:23:47.120183+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T19:23:47.120183+0000","last_undegraded":"2024-04-16T19:23:47.120183+0000","last_fullsized":"2024-04-16T19:23:47.120183+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:17.699137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.124733+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T19:23:47.124733+0000","last_peered":"2024-04-16T19:23:47.124733+0000","last_clean":"2024-04-16T19:23:47.124733+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T19:23:47.124733+0000","last_undegraded":"2024-04-16T19:23:47.124733+0000","last_fullsized":"2024-04-16T19:23:47.124733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:48.118720+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T19:23:48.118720+0000","last_peered":"2024-04-16T19:23:48.118720+0000","last_clean":"2024-04-16T19:23:48.118720+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T19:23:48.118720+0000","last_undegraded":"2024-04-16T19:23:48.118720+0000","last_fullsized":"2024-04-16T19:23:48.118720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":162,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.117474+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T19:23:47.117474+0000","last_peered":"2024-04-16T19:23:47.117474+0000","last_clean":"2024-04-16T19:23:47.117474+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T19:23:47.117474+0000","last_undegraded":"2024-04-16T19:23:47.117474+0000","last_fullsized":"2024-04-16T19:23:47.117474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27168,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830432,"statfs":{"total":107374182400,"available":107346362368,"internally_reserved":0,"allocated":143360,"data_stored":43435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28292,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829308,"statfs":{"total":107374182400,"available":107345211392,"internally_reserved":0,"allocated":724992,"data_stored":628919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27932,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829668,"statfs":{"total":107374182400,"available":107345580032,"internally_reserved":0,"allocated":737280,"data_stored":633803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T19:23:52.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T19:23:52.366 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T19:23:52.379 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T19:23:52.303040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":164836,"kb_used_data":1952,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":628980764,"statfs":{"total":644245094400,"available":644076302336,"internally_reserved":0,"allocated":1998848,"data_stored":1421810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.177982+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T19:23:47.177982+0000","last_peered":"2024-04-16T19:23:47.177982+0000","last_clean":"2024-04-16T19:23:47.177982+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T19:23:47.177982+0000","last_undegraded":"2024-04-16T19:23:47.177982+0000","last_fullsized":"2024-04-16T19:23:47.177982+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:49.312948+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T19:23:49.312948+0000","last_peered":"2024-04-16T19:23:49.312948+0000","last_clean":"2024-04-16T19:23:49.312948+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T19:23:49.312948+0000","last_undegraded":"2024-04-16T19:23:49.312948+0000","last_fullsized":"2024-04-16T19:23:49.312948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.187320+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T19:23:47.187320+0000","last_peered":"2024-04-16T19:23:47.187320+0000","last_clean":"2024-04-16T19:23:47.187320+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T19:23:47.187320+0000","last_undegraded":"2024-04-16T19:23:47.187320+0000","last_fullsized":"2024-04-16T19:23:47.187320+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:49.312959+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T19:23:49.312959+0000","last_peered":"2024-04-16T19:23:49.312959+0000","last_clean":"2024-04-16T19:23:49.312959+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T19:23:49.312959+0000","last_undegraded":"2024-04-16T19:23:49.312959+0000","last_fullsized":"2024-04-16T19:23:49.312959+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:48.118679+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T19:23:48.118679+0000","last_peered":"2024-04-16T19:23:48.118679+0000","last_clean":"2024-04-16T19:23:48.118679+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T19:23:48.118679+0000","last_undegraded":"2024-04-16T19:23:48.118679+0000","last_fullsized":"2024-04-16T19:23:48.118679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.120183+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T19:23:47.120183+0000","last_peered":"2024-04-16T19:23:47.120183+0000","last_clean":"2024-04-16T19:23:47.120183+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T19:23:47.120183+0000","last_undegraded":"2024-04-16T19:23:47.120183+0000","last_fullsized":"2024-04-16T19:23:47.120183+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:17.699137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.124733+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T19:23:47.124733+0000","last_peered":"2024-04-16T19:23:47.124733+0000","last_clean":"2024-04-16T19:23:47.124733+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T19:23:47.124733+0000","last_undegraded":"2024-04-16T19:23:47.124733+0000","last_fullsized":"2024-04-16T19:23:47.124733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:23:48.118720+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T19:23:48.118720+0000","last_peered":"2024-04-16T19:23:48.118720+0000","last_clean":"2024-04-16T19:23:48.118720+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T19:23:48.118720+0000","last_undegraded":"2024-04-16T19:23:48.118720+0000","last_fullsized":"2024-04-16T19:23:48.118720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":162,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:23:47.117474+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T19:23:47.117474+0000","last_peered":"2024-04-16T19:23:47.117474+0000","last_clean":"2024-04-16T19:23:47.117474+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T19:23:47.117474+0000","last_undegraded":"2024-04-16T19:23:47.117474+0000","last_fullsized":"2024-04-16T19:23:47.117474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27168,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830432,"statfs":{"total":107374182400,"available":107346362368,"internally_reserved":0,"allocated":143360,"data_stored":43435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28292,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829308,"statfs":{"total":107374182400,"available":107345211392,"internally_reserved":0,"allocated":724992,"data_stored":628919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":38551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27932,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829668,"statfs":{"total":107374182400,"available":107345580032,"internally_reserved":0,"allocated":737280,"data_stored":633803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T19:23:52.379 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-16T19:23:52.380 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-16T19:23:52.380 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-16T19:23:52.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-16T19:23:52.719 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:52.720 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T19:23:52.732 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-16T19:23:52.733 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T19:23:52.742 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-16T19:23:52.742 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T19:23:52.742 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'branch': 'wip-shilpa-multisite-tests', 'description': 'rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7658462', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-shilpa-multisite-tests'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding', 'notification_v2'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py'], 'extra_args': ['-a !fails_on_rgw']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-shilpa-multisite-tests', 'sha1': '9fbf23e1cbe3573736447d8751b80c9b42cadc24'}}, 'owner': 'scheduled_smanjara@teuthology', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'], ['c1.mgr.x', 'c1.osd.3', 'c1.osd.4', 'c1.osd.5', 'c1.client.1'], ['c2.mon.a', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0'], ['c2.mgr.x', 'c2.osd.3', 'c2.osd.4', 'c2.osd.5', 'c2.client.1']], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'wip-shilpa-multisite-tests', 'suite_path': '/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/smanjara/ceph.git', 'suite_sha1': '9fbf23e1cbe3573736447d8751b80c9b42cadc24', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFeEZqyh9FanBgXANxWy0pvFnKNo2hNbUFcqc4b4YSTlDKxftRrW04MORnc+2d/6ALkEqGynnxTKwFB0BDqSt6E=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM0TlHyXmVZ7jqn/rZpGycELmAJqs913ebWmQ2A0/heI3vGrv35mMKLp+X4G0DUqcRgFGEnaTPo3DjdbG/3Sps=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBn36P5Qab1DiTDnz/Z06PhUxTXcAilRT0rHTDZBY3vuVopfxS/BCCXXESn35uqPsSYOEgNDMs9qS+wcQpSIaBQ=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCVR1+NFSII3eGCZ2LQqmYNr8DPxw8UcAqsFGaEiPTDm+W8eOyUOvkV+Q0w306QFGyg9qJh1CvoFhvHYvh5aac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b0c7b917267c471208541fdeb652f5e897d00086'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 90}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_19:05:08', 'tube': 'smithi', 'user': 'smanjara', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T19:23:52.743 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-16T19:23:52.746 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-16T19:23:52.749 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-16T19:23:52.752 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-16T19:23:52.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:52.801 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:52.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:52.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T19:23:52.847 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:52.847 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T19:23:52.901 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T19:23:52.902 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-16 19:23:08.426140078 +0000 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-16 19:20:21.584690600 +0000 2024-04-16T19:23:52.959 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-16 19:20:21.584690600 +0000 2024-04-16T19:23:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-16 19:20:21.584690600 +0000 2024-04-16T19:23:52.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T19:23:53.029 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-16T19:23:53.029 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-16T19:23:53.029 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000213352 s, 2.4 MB/s 2024-04-16T19:23:53.030 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T19:23:53.089 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-16 19:23:08.426140078 +0000 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-16 19:20:21.951681603 +0000 2024-04-16T19:23:53.145 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-16 19:20:21.951681603 +0000 2024-04-16T19:23:53.146 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-16 19:20:21.951681603 +0000 2024-04-16T19:23:53.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T19:23:53.214 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-16T19:23:53.214 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-16T19:23:53.214 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000180346 s, 2.8 MB/s 2024-04-16T19:23:53.215 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T19:23:53.272 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-04-16T19:23:53.328 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T19:23:53.328 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:53.328 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-16 19:23:08.426140078 +0000 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-16 19:20:22.281673514 +0000 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-16 19:20:22.281673514 +0000 2024-04-16T19:23:53.329 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-16 19:20:22.281673514 +0000 2024-04-16T19:23:53.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T19:23:53.396 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-16T19:23:53.396 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-16T19:23:53.396 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000163068 s, 3.1 MB/s 2024-04-16T19:23:53.397 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T19:23:53.454 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-04-16T19:23:53.511 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T19:23:53.511 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:53.511 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T19:23:53.511 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:53.512 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:53.512 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-16 19:23:08.426140078 +0000 2024-04-16T19:23:53.512 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-16 19:20:22.636664810 +0000 2024-04-16T19:23:53.512 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-16 19:20:22.636664810 +0000 2024-04-16T19:23:53.512 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-16 19:20:22.636664810 +0000 2024-04-16T19:23:53.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T19:23:53.580 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-16T19:23:53.580 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-16T19:23:53.580 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000189351 s, 2.7 MB/s 2024-04-16T19:23:53.581 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T19:23:53.642 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:53.642 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:53.642 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T19:23:53.658 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T19:23:53.659 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-04-16T19:23:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T19:23:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:53.714 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-16T19:23:53.714 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:53.715 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:53.715 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 19:23:00.953748382 +0000 2024-04-16T19:23:53.715 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 19:20:21.968985496 +0000 2024-04-16T19:23:53.715 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 19:20:21.968985496 +0000 2024-04-16T19:23:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 19:20:21.968985496 +0000 2024-04-16T19:23:53.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T19:23:53.782 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T19:23:53.782 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T19:23:53.782 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000270557 s, 1.9 MB/s 2024-04-16T19:23:53.784 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T19:23:53.840 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 19:23:00.954748370 +0000 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 19:20:22.314981167 +0000 2024-04-16T19:23:53.895 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 19:20:22.314981167 +0000 2024-04-16T19:23:53.896 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 19:20:22.314981167 +0000 2024-04-16T19:23:53.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T19:23:53.961 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T19:23:53.961 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T19:23:53.961 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000215025 s, 2.4 MB/s 2024-04-16T19:23:53.962 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T19:23:54.019 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 19:23:00.954748370 +0000 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 19:20:22.660976837 +0000 2024-04-16T19:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 19:20:22.660976837 +0000 2024-04-16T19:23:54.075 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 19:20:22.660976837 +0000 2024-04-16T19:23:54.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T19:23:54.140 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T19:23:54.141 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T19:23:54.141 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000209679 s, 2.4 MB/s 2024-04-16T19:23:54.142 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T19:23:54.197 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-04-16T19:23:54.253 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T19:23:54.253 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-04-16 19:23:00.955748357 +0000 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-04-16 19:20:22.999972595 +0000 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-04-16 19:20:22.999972595 +0000 2024-04-16T19:23:54.254 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-04-16 19:20:22.999972595 +0000 2024-04-16T19:23:54.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T19:23:54.320 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-04-16T19:23:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-04-16T19:23:54.321 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000191246 s, 2.7 MB/s 2024-04-16T19:23:54.322 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T19:23:54.379 INFO:tasks.ceph:osd dev map: {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:54.379 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-16T19:23:54.379 INFO:tasks.ceph:Generating config... 2024-04-16T19:23:54.380 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-16T19:23:54.380 INFO:tasks.ceph:[client] debug rgw lifecycle = 20 2024-04-16T19:23:54.380 INFO:tasks.ceph:[client] debug rgw notification = 20 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw data sync poll interval = 5 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw meta sync poll interval = 5 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client] setuser = ceph 2024-04-16T19:23:54.381 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-16T19:23:54.382 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T19:23:54.382 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T19:23:54.382 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T19:23:54.382 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T19:23:54.382 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:54.382 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:54.382 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-16T19:23:54.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-16T19:23:54.447 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/c2.keyring 2024-04-16T19:23:54.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-16T19:23:54.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-16T19:23:54.521 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.119')] 2024-04-16T19:23:54.522 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.119'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw lifecycle': 20, 'debug rgw notification': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw data sync poll interval': 5, 'rgw md log max shards': 4, 'rgw meta sync poll interval': 5, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-16T19:23:54.523 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:54.523 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T19:23:54.577 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.119 --print /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:54.656 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:23:54.656 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:23:54.656 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:54.656 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: generated fsid 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = quincy 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 0 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:fsid 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:last_changed 2024-04-16T19:23:54.656906+0000 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:created 2024-04-16T19:23:54.656906+0000 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:min_mon_release 17 (quincy) 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:election_strategy: 1 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-04-16T19:23:54.657 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-16T19:23:54.658 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T19:23:54.714 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 292b9761-2cd3-4afe-805f-b344b0a40a39... 2024-04-16T19:23:54.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-16T19:23:54.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-16T19:23:54.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-16T19:23:54.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.776 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.777 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-04-16T19:23:54.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = 172.21.15.119 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:54.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-04-16T19:23:54.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-04-16T19:23:54.781 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.782 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-04-16T19:23:54.783 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw = 20 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw notification = 20 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw sync = 20 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt require ssl = False 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw curl low speed time = 300 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data log num shards = 4 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:54.784 INFO:teuthology.orchestra.run.smithi026.stdout: rgw md log max shards = 4 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: setgroup = ceph 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: setuser = ceph 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-04-16T19:23:54.785 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.786 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:54.787 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.788 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = 172.21.15.119 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-16T19:23:54.789 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-16T19:23:54.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:54.791 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-16T19:23:54.792 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-16T19:23:54.793 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw = 20 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw notification = 20 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: debug rgw sync = 20 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt require ssl = False 2024-04-16T19:23:54.794 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw curl low speed time = 300 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data log num shards = 4 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw md log max shards = 4 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: setgroup = ceph 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: setuser = ceph 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-16T19:23:54.795 INFO:teuthology.orchestra.run.smithi069.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-04-16T19:23:54.796 INFO:teuthology.orchestra.run.smithi069.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:54.807 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-04-16T19:23:54.807 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-04-16T19:23:54.807 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:54.807 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-04-16T19:23:54.807 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:54.808 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:54.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = 172.21.15.119 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-04-16T19:23:54.810 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug trim objects = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-04-16T19:23:54.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-16T19:23:54.812 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.813 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-04-16T19:23:54.814 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw = 20 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw notification = 20 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw sync = 20 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt require ssl = False 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw curl low speed time = 300 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data log num shards = 4 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw md log max shards = 4 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:54.815 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: setgroup = ceph 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: setuser = ceph 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-04-16T19:23:54.816 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-04-16T19:23:54.817 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-04-16T19:23:54.818 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = 172.21.15.119 2024-04-16T19:23:54.819 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 292b9761-2cd3-4afe-805f-b344b0a40a39 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug trim objects = true 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.820 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: rocksdb delete range threshold = 0 2024-04-16T19:23:54.821 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-04-16T19:23:54.822 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.823 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw = 20 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw lifecycle = 20 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw notification = 20 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: debug rgw sync = 20 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt require ssl = False 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt s3 kms backend = testing 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw curl low speed time = 300 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data log num shards = 4 2024-04-16T19:23:54.824 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data sync poll interval = 5 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw md log max shards = 4 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw meta sync poll interval = 5 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync data inject err probability = 0.1 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync log trim interval = 0 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync meta inject err probability = 0.1 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw sync obj etag verify = True 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: setgroup = ceph 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: setuser = ceph 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: rgw data notify interval msec = 0 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 10 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 1000 2024-04-16T19:23:54.825 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-04-16T19:23:54.826 INFO:teuthology.orchestra.run.smithi151.stdout: osd_max_omap_entries_per_request = 10000 2024-04-16T19:23:54.827 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-16T19:23:54.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-16T19:23:54.884 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T19:23:54.884 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:54.884 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-16T19:23:54.900 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:54.900 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-16T19:23:54.955 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T19:23:54.955 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:54.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/c2.keyring 2024-04-16T19:23:54.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-16T19:23:55.001 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-04-16T19:23:55.002 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:55.058 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T19:23:55.058 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:55.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/c2.keyring 2024-04-16T19:23:55.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-16T19:23:55.105 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-16T19:23:55.105 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:55.163 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:23:55.163 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:55.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/c2.keyring 2024-04-16T19:23:55.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-16T19:23:55.204 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:23:55.205 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:55.263 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:23:55.263 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:55.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/c2.keyring 2024-04-16T19:23:55.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-16T19:23:55.307 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:23:55.307 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:55.363 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T19:23:55.363 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T19:23:55.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-16T19:23:55.469 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-16T19:23:55.472 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T19:23:55.472 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T19:23:55.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-16T19:23:55.516 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-16T19:23:55.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-16T19:23:55.576 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-16T19:23:55.592 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T19:23:55.592 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'c1.osd.3': '/dev/vg_nvme/lv_1', 'c1.osd.4': '/dev/vg_nvme/lv_2', 'c1.osd.5': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-16T19:23:55.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-16T19:23:55.620 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:55.621 INFO:tasks.ceph:role: c2.osd.0 2024-04-16T19:23:55.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:23:55.621 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:55.696 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:55.697 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:55.697 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:55.697 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:55.697 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:56.204 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-16T19:23:56.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-16T19:23:56.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-16T19:23:56.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-16T19:23:56.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-16T19:23:56.424 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:56.424 INFO:tasks.ceph:role: c2.osd.1 2024-04-16T19:23:56.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:23:56.424 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T19:23:56.499 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:56.500 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:56.962 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-16T19:23:56.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-16T19:23:56.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-16T19:23:57.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-16T19:23:57.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-16T19:23:57.161 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T19:23:57.161 INFO:tasks.ceph:role: c2.osd.2 2024-04-16T19:23:57.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T19:23:57.161 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:23:57.238 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:23:57.239 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:23:57.239 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:23:57.727 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-16T19:23:57.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-16T19:23:57.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-16T19:23:57.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-16T19:23:57.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:23:58.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.023+0000 7f72f200f640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-16T19:23:58.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.028+0000 7f72f200f640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-16T19:23:58.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.028+0000 7f72f200f640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-16T19:23:58.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.028+0000 7f72f200f640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-16T19:23:58.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.028+0000 7f72f200f640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-16T19:23:58.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:23:58.035+0000 7f72f200f640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-16T19:24:00.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-16T19:24:00.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-16T19:24:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:00.245+0000 7f9aa3ac2640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-16T19:24:02.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-16T19:24:02.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.455+0000 7ff04860c640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.455+0000 7ff04860c640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.455+0000 7ff04860c640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.455+0000 7ff04860c640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.455+0000 7ff04860c640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-16T19:24:02.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:02.456+0000 7ff04860c640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-16T19:24:04.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-16T19:24:04.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/c2-3 2024-04-16T19:24:04.604 INFO:tasks.ceph:roles_to_devs: {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:24:04.604 INFO:tasks.ceph:role: c2.osd.3 2024-04-16T19:24:04.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:24:04.604 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:24:04.681 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:24:04.682 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:24:05.197 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-04-16T19:24:05.200 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-04-16T19:24:05.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-3 2024-04-16T19:24:05.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-3 2024-04-16T19:24:05.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/c2-4 2024-04-16T19:24:05.438 INFO:tasks.ceph:roles_to_devs: {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:24:05.438 INFO:tasks.ceph:role: c2.osd.4 2024-04-16T19:24:05.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:24:05.438 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T19:24:05.515 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:24:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:24:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:24:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:24:05.516 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:24:05.973 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-04-16T19:24:05.975 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-04-16T19:24:05.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-4 2024-04-16T19:24:06.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-4 2024-04-16T19:24:06.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/c2-5 2024-04-16T19:24:06.175 INFO:tasks.ceph:roles_to_devs: {'c2.osd.3': '/dev/vg_nvme/lv_1', 'c2.osd.4': '/dev/vg_nvme/lv_2', 'c2.osd.5': '/dev/vg_nvme/lv_3'} 2024-04-16T19:24:06.175 INFO:tasks.ceph:role: c2.osd.5 2024-04-16T19:24:06.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T19:24:06.175 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T19:24:06.252 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T19:24:06.252 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T19:24:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T19:24:06.742 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-04-16T19:24:06.744 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-04-16T19:24:06.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-5 2024-04-16T19:24:06.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-5 2024-04-16T19:24:06.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:07.048 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.047+0000 7ff9b1e24640 -1 auth: error reading file: /var/lib/ceph/osd/c2-3/keyring: can't open /var/lib/ceph/osd/c2-3/keyring: (2) No such file or directory 2024-04-16T19:24:07.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.049+0000 7ff9b1e24640 -1 created new key in keyring /var/lib/ceph/osd/c2-3/keyring 2024-04-16T19:24:07.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.049+0000 7ff9b1e24640 -1 bluestore(/var/lib/ceph/osd/c2-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-3/block: (2) No such file or directory 2024-04-16T19:24:07.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.049+0000 7ff9b1e24640 -1 bluestore(/var/lib/ceph/osd/c2-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-3/block: (2) No such file or directory 2024-04-16T19:24:07.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.049+0000 7ff9b1e24640 -1 bluestore(/var/lib/ceph/osd/c2-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-3/block: (2) No such file or directory 2024-04-16T19:24:07.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:07.057+0000 7ff9b1e24640 -1 bluestore(/var/lib/ceph/osd/c2-3) _read_fsid unparsable uuid 2024-04-16T19:24:09.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-3 2024-04-16T19:24:09.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:09.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 auth: error reading file: /var/lib/ceph/osd/c2-4/keyring: can't open /var/lib/ceph/osd/c2-4/keyring: (2) No such file or directory 2024-04-16T19:24:09.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 created new key in keyring /var/lib/ceph/osd/c2-4/keyring 2024-04-16T19:24:09.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 bluestore(/var/lib/ceph/osd/c2-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-4/block: (2) No such file or directory 2024-04-16T19:24:09.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 bluestore(/var/lib/ceph/osd/c2-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-4/block: (2) No such file or directory 2024-04-16T19:24:09.261 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 bluestore(/var/lib/ceph/osd/c2-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-4/block: (2) No such file or directory 2024-04-16T19:24:09.262 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:09.260+0000 7f0d61b06640 -1 bluestore(/var/lib/ceph/osd/c2-4) _read_fsid unparsable uuid 2024-04-16T19:24:11.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-4 2024-04-16T19:24:11.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.465+0000 7f491ef8f640 -1 auth: error reading file: /var/lib/ceph/osd/c2-5/keyring: can't open /var/lib/ceph/osd/c2-5/keyring: (2) No such file or directory 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.466+0000 7f491ef8f640 -1 created new key in keyring /var/lib/ceph/osd/c2-5/keyring 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.466+0000 7f491ef8f640 -1 bluestore(/var/lib/ceph/osd/c2-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-5/block: (2) No such file or directory 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.466+0000 7f491ef8f640 -1 bluestore(/var/lib/ceph/osd/c2-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-5/block: (2) No such file or directory 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.466+0000 7f491ef8f640 -1 bluestore(/var/lib/ceph/osd/c2-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-5/block: (2) No such file or directory 2024-04-16T19:24:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:24:11.466+0000 7f491ef8f640 -1 bluestore(/var/lib/ceph/osd/c2-5) _read_fsid unparsable uuid 2024-04-16T19:24:13.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-5 2024-04-16T19:24:13.582 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T19:24:13.582 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:13.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-16T19:24:13.615 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:13.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-16T19:24:13.687 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:13.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-16T19:24:13.757 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:13.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-16T19:24:13.791 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:13.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-3/keyring of=/dev/stdout 2024-04-16T19:24:13.861 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:13.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-4/keyring of=/dev/stdout 2024-04-16T19:24:13.932 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:13.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-5/keyring of=/dev/stdout 2024-04-16T19:24:14.003 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:14.004 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-16T19:24:14.021 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:14.029 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-16T19:24:14.061 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T19:24:14.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/c2.keyring 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBO0B5mJlB9ARAA78eyhjMcFtNczaQqj6Hw6A== 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBQ0B5mDpmpDhAAVKnqvhzE6+ypWmpIkenHAw== 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBS0B5msDw2GxAAk98yCNS6+E6CI581EcKEIw== 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-04-16T19:24:14.144 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBL0B5mLcX9GxAAM+fV9cfNuwmvWb6gm/L1tQ== 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBX0B5mcJXjAhAA7QD9ii/mxzTotOZYBXz+aQ== 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBZ0B5mG1OQDxAABNGFEXFMnxtoIx5IJqNDfg== 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBb0B5mjonRGxAAXh7ioHlTReQkwnAStWrdsw== 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBL0B5mw7TKHhAAgG95XvHiVZf0eOybsiT62g== 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2024-04-16T19:24:14.145 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBL0B5mWepbIhAAwlD5KbEu5fTROAxR+UHTWg== 2024-04-16T19:24:14.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T19:24:14.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.513 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T19:24:14.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T19:24:14.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T19:24:14.679 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T19:24:14.679 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-16T19:24:14.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-16T19:24:14.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-16T19:24:14.834 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-16T19:24:14.890 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-16T19:24:14.891 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-16T19:24:14.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-16T19:24:14.933 INFO:tasks.ceph.c2.mon.a:Started 2024-04-16T19:24:14.933 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-16T19:24:14.933 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-16T19:24:14.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-16T19:24:14.936 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-16T19:24:14.936 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T19:24:14.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster c2 osd crush tunables default 2024-04-16T19:24:15.112 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-04-16T19:24:15.123 INFO:tasks.ceph:check_enable_crimson: False 2024-04-16T19:24:15.123 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-16T19:24:15.124 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:15.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-16T19:24:15.152 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:15.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-16T19:24:15.219 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-16T19:24:15.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-16T19:24:15.288 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:15.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-3/fsid of=/dev/stdout 2024-04-16T19:24:15.316 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:15.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-4/fsid of=/dev/stdout 2024-04-16T19:24:15.385 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-04-16T19:24:15.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/c2-5/fsid of=/dev/stdout 2024-04-16T19:24:15.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new 5dde3bd7-758d-45cc-aaa1-2473b9305c69 0 2024-04-16T19:24:15.576 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:15.575+0000 7ff1c6040040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T19:24:15.632 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:15.631+0000 7ff1c6040040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T19:24:15.678 INFO:teuthology.orchestra.run.smithi151.stdout:0 2024-04-16T19:24:15.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new 386f4573-851e-43ba-83ee-271aceb9c938 1 2024-04-16T19:24:15.873 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-04-16T19:24:15.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new 2c8f1bf0-f552-4a71-b677-7b0d79023b68 2 2024-04-16T19:24:16.067 INFO:teuthology.orchestra.run.smithi151.stdout:2 2024-04-16T19:24:16.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new 42ddd759-5608-4e04-b824-98256168b086 3 2024-04-16T19:24:16.196 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.196+0000 7ff1c6040040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T19:24:16.253 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.252+0000 7ff1c6040040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T19:24:16.265 INFO:teuthology.orchestra.run.smithi151.stdout:3 2024-04-16T19:24:16.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new 8f75871e-d39f-4c40-a165-357b2777aa1b 4 2024-04-16T19:24:16.301 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.300+0000 7ff1c6040040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T19:24:16.358 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.357+0000 7ff1c6040040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T19:24:16.403 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.402+0000 7ff1c6040040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T19:24:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:4 2024-04-16T19:24:16.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster c2 osd new d293d827-c614-438c-af83-430c9e1cb640 5 2024-04-16T19:24:16.524 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.524+0000 7ff1c6040040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T19:24:16.599 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.598+0000 7ff1c6040040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T19:24:16.647 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.646+0000 7ff1c6040040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T19:24:16.701 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.700+0000 7ff1c6040040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T19:24:16.702 INFO:teuthology.orchestra.run.smithi151.stdout:5 2024-04-16T19:24:16.713 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-16T19:24:16.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-16T19:24:16.716 INFO:tasks.ceph.c2.osd.0:Started 2024-04-16T19:24:16.716 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-16T19:24:16.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-16T19:24:16.718 INFO:tasks.ceph.c2.osd.1:Started 2024-04-16T19:24:16.719 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-16T19:24:16.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-16T19:24:16.721 INFO:tasks.ceph.c2.osd.2:Started 2024-04-16T19:24:16.721 INFO:tasks.ceph.c2.osd.3:Restarting daemon 2024-04-16T19:24:16.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 3 2024-04-16T19:24:16.751 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:16.750+0000 7ff1c6040040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T19:24:16.753 INFO:tasks.ceph.c2.osd.3:Started 2024-04-16T19:24:16.753 INFO:tasks.ceph.c2.osd.4:Restarting daemon 2024-04-16T19:24:16.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 4 2024-04-16T19:24:16.755 INFO:tasks.ceph.c2.osd.4:Started 2024-04-16T19:24:16.755 INFO:tasks.ceph.c2.osd.5:Restarting daemon 2024-04-16T19:24:16.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 5 2024-04-16T19:24:16.757 INFO:tasks.ceph.c2.osd.5:Started 2024-04-16T19:24:16.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-16T19:24:16.934 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:16.935 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":8,"fsid":"292b9761-2cd3-4afe-805f-b344b0a40a39","created":"2024-04-16T19:24:15.022592+0000","modified":"2024-04-16T19:24:16.698863+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T19:24:16.698863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5dde3bd7-758d-45cc-aaa1-2473b9305c69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"386f4573-851e-43ba-83ee-271aceb9c938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c8f1bf0-f552-4a71-b677-7b0d79023b68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42ddd759-5608-4e04-b824-98256168b086","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f75871e-d39f-4c40-a165-357b2777aa1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d293d827-c614-438c-af83-430c9e1cb640","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:24:16.946 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-16T19:24:16.946 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T19:24:17.144 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.143+0000 7ff1c6040040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T19:24:17.277 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.276+0000 7ff1c6040040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T19:24:17.321 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T19:24:17.320+0000 7ffbb76a6640 -1 Falling back to public interface 2024-04-16T19:24:17.329 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T19:24:17.328+0000 7fe871e13640 -1 Falling back to public interface 2024-04-16T19:24:17.329 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T19:24:17.328+0000 7fe85ec11640 -1 Falling back to public interface 2024-04-16T19:24:17.329 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T19:24:17.328+0000 7fe870796640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-04-16T19:24:17.355 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T19:24:17.354+0000 7f7faa581640 -1 Falling back to public interface 2024-04-16T19:24:17.358 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T19:24:17.357+0000 7f242f299640 -1 Falling back to public interface 2024-04-16T19:24:17.360 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T19:24:17.358+0000 7f92fba9b640 -1 Falling back to public interface 2024-04-16T19:24:17.407 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.407+0000 7ff1c6040040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T19:24:17.454 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.453+0000 7ff1c6040040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T19:24:17.551 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-16T19:24:17.566 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.565+0000 7ff1c6040040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T19:24:17.663 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.662+0000 7ff1c6040040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T19:24:17.713 INFO:teuthology.misc.health.smithi119.stdout: 2024-04-16T19:24:17.713 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":8,"fsid":"292b9761-2cd3-4afe-805f-b344b0a40a39","created":"2024-04-16T19:24:15.022592+0000","modified":"2024-04-16T19:24:16.698863+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T19:24:16.698863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5dde3bd7-758d-45cc-aaa1-2473b9305c69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"386f4573-851e-43ba-83ee-271aceb9c938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c8f1bf0-f552-4a71-b677-7b0d79023b68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42ddd759-5608-4e04-b824-98256168b086","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f75871e-d39f-4c40-a165-357b2777aa1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d293d827-c614-438c-af83-430c9e1cb640","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:24:17.722 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-16T19:24:17.817 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.817+0000 7ff1c6040040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T19:24:17.860 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:17.859+0000 7ff1c6040040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T19:24:18.016 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:18.015+0000 7ff1c6040040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T19:24:18.843 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:18.842+0000 7ff1c6040040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T19:24:19.117 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:19.116+0000 7ff1c6040040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T19:24:19.168 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:19.167+0000 7ff1c6040040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T19:24:19.285 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:19.284+0000 7ff1c6040040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T19:24:19.987 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T19:24:19.986+0000 7ffbb76a6640 -1 osd.0 0 log_to_monitors true 2024-04-16T19:24:20.029 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T19:24:20.028+0000 7fe871e13640 -1 osd.1 0 log_to_monitors true 2024-04-16T19:24:20.055 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T19:24:20.054+0000 7fe85ec11640 -1 osd.2 0 log_to_monitors true 2024-04-16T19:24:20.102 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T19:24:20.101+0000 7f242f299640 -1 osd.5 0 log_to_monitors true 2024-04-16T19:24:20.111 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T19:24:20.110+0000 7f92fba9b640 -1 osd.4 0 log_to_monitors true 2024-04-16T19:24:20.112 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T19:24:20.110+0000 7f7faa581640 -1 osd.3 0 log_to_monitors true 2024-04-16T19:24:20.294 INFO:tasks.ceph.c2.mgr.x.smithi151.stderr:2024-04-16T19:24:20.292+0000 7ff167455640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T19:24:22.567 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T19:24:22.566+0000 7f242ba43640 -1 osd.5 0 waiting for initial osdmap 2024-04-16T19:24:22.573 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T19:24:22.573+0000 7fe86d77d640 -1 osd.1 0 waiting for initial osdmap 2024-04-16T19:24:22.580 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T19:24:22.579+0000 7f242304f640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:22.584 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T19:24:22.583+0000 7fe86559d640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:22.584 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T19:24:22.583+0000 7fe85a50f640 -1 osd.2 0 waiting for initial osdmap 2024-04-16T19:24:22.599 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T19:24:22.599+0000 7fe85232f640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:22.606 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T19:24:22.605+0000 7f7fa6517640 -1 osd.3 0 waiting for initial osdmap 2024-04-16T19:24:22.607 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T19:24:22.607+0000 7ffbb363c640 -1 osd.0 0 waiting for initial osdmap 2024-04-16T19:24:22.621 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T19:24:22.620+0000 7f7f9e337640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:22.625 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T19:24:22.624+0000 7f92f8245640 -1 osd.4 0 waiting for initial osdmap 2024-04-16T19:24:22.627 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T19:24:22.627+0000 7ffbab45c640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:22.636 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T19:24:22.635+0000 7f92ef851640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T19:24:24.329 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-16T19:24:24.620 INFO:teuthology.misc.health.smithi119.stdout: 2024-04-16T19:24:24.621 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":12,"fsid":"292b9761-2cd3-4afe-805f-b344b0a40a39","created":"2024-04-16T19:24:15.022592+0000","modified":"2024-04-16T19:24:24.576615+0000","last_up_change":"2024-04-16T19:24:23.575811+0000","last_in_change":"2024-04-16T19:24:16.698863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5dde3bd7-758d-45cc-aaa1-2473b9305c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6801","nonce":1903075924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6803","nonce":1903075924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6807","nonce":1903075924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6805","nonce":1903075924}]},"public_addr":"172.21.15.119:6801/1903075924","cluster_addr":"172.21.15.119:6803/1903075924","heartbeat_back_addr":"172.21.15.119:6807/1903075924","heartbeat_front_addr":"172.21.15.119:6805/1903075924","state":["exists","up"]},{"osd":1,"uuid":"386f4573-851e-43ba-83ee-271aceb9c938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6811","nonce":1709392723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6815","nonce":1709392723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6823","nonce":1709392723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6819","nonce":1709392723}]},"public_addr":"172.21.15.119:6811/1709392723","cluster_addr":"172.21.15.119:6815/1709392723","heartbeat_back_addr":"172.21.15.119:6823/1709392723","heartbeat_front_addr":"172.21.15.119:6819/1709392723","state":["exists","up"]},{"osd":2,"uuid":"2c8f1bf0-f552-4a71-b677-7b0d79023b68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6810","nonce":3329554284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6814","nonce":3329554284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6822","nonce":3329554284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6818","nonce":3329554284}]},"public_addr":"172.21.15.119:6810/3329554284","cluster_addr":"172.21.15.119:6814/3329554284","heartbeat_back_addr":"172.21.15.119:6822/3329554284","heartbeat_front_addr":"172.21.15.119:6818/3329554284","state":["exists","up"]},{"osd":3,"uuid":"42ddd759-5608-4e04-b824-98256168b086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6801","nonce":1199260872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6803","nonce":1199260872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6807","nonce":1199260872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6805","nonce":1199260872}]},"public_addr":"172.21.15.151:6801/1199260872","cluster_addr":"172.21.15.151:6803/1199260872","heartbeat_back_addr":"172.21.15.151:6807/1199260872","heartbeat_front_addr":"172.21.15.151:6805/1199260872","state":["exists","up"]},{"osd":4,"uuid":"8f75871e-d39f-4c40-a165-357b2777aa1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6815","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6817","nonce":1085854805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6819","nonce":1085854805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6823","nonce":1085854805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6821","nonce":1085854805}]},"public_addr":"172.21.15.151:6817/1085854805","cluster_addr":"172.21.15.151:6819/1085854805","heartbeat_back_addr":"172.21.15.151:6823/1085854805","heartbeat_front_addr":"172.21.15.151:6821/1085854805","state":["exists","up"]},{"osd":5,"uuid":"d293d827-c614-438c-af83-430c9e1cb640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6809","nonce":1117823092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6811","nonce":1117823092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6816","nonce":1117823092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6813","nonce":1117823092}]},"public_addr":"172.21.15.151:6809/1117823092","cluster_addr":"172.21.15.151:6811/1117823092","heartbeat_back_addr":"172.21.15.151:6816/1117823092","heartbeat_front_addr":"172.21.15.151:6813/1117823092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:20.963555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.072291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.023246+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.140847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.162713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.063139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:24:24.632 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-16T19:24:24.632 INFO:tasks.ceph:Creating RBD pool 2024-04-16T19:24:24.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-16T19:24:25.593 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'rbd' created 2024-04-16T19:24:25.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T19:24:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T19:24:26.611 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-16T19:24:26.611 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-16T19:24:26.612 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-16T19:24:26.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-16T19:24:26.612 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T19:24:26.977 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:26.988 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":4,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":3440446959},{"type":"v1","addr":"172.21.15.151:6825","nonce":3440446959}]},"active_addr":"172.21.15.151:6825/3440446959","active_change":"2024-04-16T19:24:19.286646+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":994725210}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":4176342975}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2629561792}]}]} 2024-04-16T19:24:26.990 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-16T19:24:26.990 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-16T19:24:26.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-16T19:24:27.286 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:27.286 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":14,"fsid":"292b9761-2cd3-4afe-805f-b344b0a40a39","created":"2024-04-16T19:24:15.022592+0000","modified":"2024-04-16T19:24:26.593213+0000","last_up_change":"2024-04-16T19:24:23.575811+0000","last_in_change":"2024-04-16T19:24:16.698863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-16T19:24:24.921668+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-16T19:24:25.305978+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5dde3bd7-758d-45cc-aaa1-2473b9305c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6801","nonce":1903075924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6803","nonce":1903075924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6807","nonce":1903075924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6805","nonce":1903075924}]},"public_addr":"172.21.15.119:6801/1903075924","cluster_addr":"172.21.15.119:6803/1903075924","heartbeat_back_addr":"172.21.15.119:6807/1903075924","heartbeat_front_addr":"172.21.15.119:6805/1903075924","state":["exists","up"]},{"osd":1,"uuid":"386f4573-851e-43ba-83ee-271aceb9c938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6811","nonce":1709392723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6815","nonce":1709392723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6823","nonce":1709392723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6819","nonce":1709392723}]},"public_addr":"172.21.15.119:6811/1709392723","cluster_addr":"172.21.15.119:6815/1709392723","heartbeat_back_addr":"172.21.15.119:6823/1709392723","heartbeat_front_addr":"172.21.15.119:6819/1709392723","state":["exists","up"]},{"osd":2,"uuid":"2c8f1bf0-f552-4a71-b677-7b0d79023b68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6810","nonce":3329554284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6814","nonce":3329554284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6822","nonce":3329554284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6818","nonce":3329554284}]},"public_addr":"172.21.15.119:6810/3329554284","cluster_addr":"172.21.15.119:6814/3329554284","heartbeat_back_addr":"172.21.15.119:6822/3329554284","heartbeat_front_addr":"172.21.15.119:6818/3329554284","state":["exists","up"]},{"osd":3,"uuid":"42ddd759-5608-4e04-b824-98256168b086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6801","nonce":1199260872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6803","nonce":1199260872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6807","nonce":1199260872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6805","nonce":1199260872}]},"public_addr":"172.21.15.151:6801/1199260872","cluster_addr":"172.21.15.151:6803/1199260872","heartbeat_back_addr":"172.21.15.151:6807/1199260872","heartbeat_front_addr":"172.21.15.151:6805/1199260872","state":["exists","up"]},{"osd":4,"uuid":"8f75871e-d39f-4c40-a165-357b2777aa1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6815","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6817","nonce":1085854805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6819","nonce":1085854805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6823","nonce":1085854805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6821","nonce":1085854805}]},"public_addr":"172.21.15.151:6817/1085854805","cluster_addr":"172.21.15.151:6819/1085854805","heartbeat_back_addr":"172.21.15.151:6823/1085854805","heartbeat_front_addr":"172.21.15.151:6821/1085854805","state":["exists","up"]},{"osd":5,"uuid":"d293d827-c614-438c-af83-430c9e1cb640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6809","nonce":1117823092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6811","nonce":1117823092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6816","nonce":1117823092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6813","nonce":1117823092}]},"public_addr":"172.21.15.151:6809/1117823092","cluster_addr":"172.21.15.151:6811/1117823092","heartbeat_back_addr":"172.21.15.151:6816/1117823092","heartbeat_front_addr":"172.21.15.151:6813/1117823092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:20.963555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.072291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.023246+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.140847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.162713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.063139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:24:27.299 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-16T19:24:27.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-16T19:24:27.589 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:27.589 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":14,"fsid":"292b9761-2cd3-4afe-805f-b344b0a40a39","created":"2024-04-16T19:24:15.022592+0000","modified":"2024-04-16T19:24:26.593213+0000","last_up_change":"2024-04-16T19:24:23.575811+0000","last_in_change":"2024-04-16T19:24:16.698863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-16T19:24:24.921668+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-16T19:24:25.305978+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5dde3bd7-758d-45cc-aaa1-2473b9305c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6801","nonce":1903075924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6803","nonce":1903075924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6807","nonce":1903075924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1903075924},{"type":"v1","addr":"172.21.15.119:6805","nonce":1903075924}]},"public_addr":"172.21.15.119:6801/1903075924","cluster_addr":"172.21.15.119:6803/1903075924","heartbeat_back_addr":"172.21.15.119:6807/1903075924","heartbeat_front_addr":"172.21.15.119:6805/1903075924","state":["exists","up"]},{"osd":1,"uuid":"386f4573-851e-43ba-83ee-271aceb9c938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6811","nonce":1709392723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6815","nonce":1709392723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6823","nonce":1709392723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":1709392723},{"type":"v1","addr":"172.21.15.119:6819","nonce":1709392723}]},"public_addr":"172.21.15.119:6811/1709392723","cluster_addr":"172.21.15.119:6815/1709392723","heartbeat_back_addr":"172.21.15.119:6823/1709392723","heartbeat_front_addr":"172.21.15.119:6819/1709392723","state":["exists","up"]},{"osd":2,"uuid":"2c8f1bf0-f552-4a71-b677-7b0d79023b68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6810","nonce":3329554284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6814","nonce":3329554284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6822","nonce":3329554284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":3329554284},{"type":"v1","addr":"172.21.15.119:6818","nonce":3329554284}]},"public_addr":"172.21.15.119:6810/3329554284","cluster_addr":"172.21.15.119:6814/3329554284","heartbeat_back_addr":"172.21.15.119:6822/3329554284","heartbeat_front_addr":"172.21.15.119:6818/3329554284","state":["exists","up"]},{"osd":3,"uuid":"42ddd759-5608-4e04-b824-98256168b086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6801","nonce":1199260872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6803","nonce":1199260872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6807","nonce":1199260872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1199260872},{"type":"v1","addr":"172.21.15.151:6805","nonce":1199260872}]},"public_addr":"172.21.15.151:6801/1199260872","cluster_addr":"172.21.15.151:6803/1199260872","heartbeat_back_addr":"172.21.15.151:6807/1199260872","heartbeat_front_addr":"172.21.15.151:6805/1199260872","state":["exists","up"]},{"osd":4,"uuid":"8f75871e-d39f-4c40-a165-357b2777aa1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6815","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6817","nonce":1085854805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6819","nonce":1085854805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6823","nonce":1085854805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":1085854805},{"type":"v1","addr":"172.21.15.151:6821","nonce":1085854805}]},"public_addr":"172.21.15.151:6817/1085854805","cluster_addr":"172.21.15.151:6819/1085854805","heartbeat_back_addr":"172.21.15.151:6823/1085854805","heartbeat_front_addr":"172.21.15.151:6821/1085854805","state":["exists","up"]},{"osd":5,"uuid":"d293d827-c614-438c-af83-430c9e1cb640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6809","nonce":1117823092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6811","nonce":1117823092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6816","nonce":1117823092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":1117823092},{"type":"v1","addr":"172.21.15.151:6813","nonce":1117823092}]},"public_addr":"172.21.15.151:6809/1117823092","cluster_addr":"172.21.15.151:6811/1117823092","heartbeat_back_addr":"172.21.15.151:6816/1117823092","heartbeat_front_addr":"172.21.15.151:6813/1117823092","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:20.963555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.072291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.023246+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.140847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.162713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:24:21.063139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T19:24:27.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-16T19:24:27.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-16T19:24:27.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-16T19:24:27.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.3 flush_pg_stats 2024-04-16T19:24:27.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.4 flush_pg_stats 2024-04-16T19:24:27.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.5 flush_pg_stats 2024-04-16T19:24:27.760 INFO:teuthology.orchestra.run.smithi119.stdout:47244640261 2024-04-16T19:24:27.760 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-16T19:24:27.768 INFO:teuthology.orchestra.run.smithi119.stdout:47244640258 2024-04-16T19:24:27.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-16T19:24:27.808 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:27.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-16T19:24:27.809 INFO:teuthology.orchestra.run.smithi119.stdout:47244640261 2024-04-16T19:24:27.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.3 2024-04-16T19:24:27.825 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:27.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.5 2024-04-16T19:24:27.831 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:27.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.4 2024-04-16T19:24:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:28.158 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640261 got 47244640260 for osd.1 2024-04-16T19:24:28.233 INFO:teuthology.orchestra.run.smithi119.stdout:47244640259 2024-04-16T19:24:28.251 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640259 for osd.4 2024-04-16T19:24:28.255 INFO:teuthology.orchestra.run.smithi119.stdout:47244640257 2024-04-16T19:24:28.268 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:28.268 INFO:teuthology.orchestra.run.smithi119.stdout:47244640259 2024-04-16T19:24:28.272 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640258 got 47244640257 for osd.2 2024-04-16T19:24:28.281 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640261 got 47244640260 for osd.3 2024-04-16T19:24:28.282 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640259 for osd.0 2024-04-16T19:24:28.346 INFO:teuthology.orchestra.run.smithi119.stdout:47244640259 2024-04-16T19:24:28.359 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640259 for osd.5 2024-04-16T19:24:29.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-16T19:24:29.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.4 2024-04-16T19:24:29.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-16T19:24:29.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.3 2024-04-16T19:24:29.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-16T19:24:29.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.5 2024-04-16T19:24:29.593 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:29.613 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640261 got 47244640260 for osd.1 2024-04-16T19:24:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:47244640258 2024-04-16T19:24:29.688 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640258 got 47244640258 for osd.2 2024-04-16T19:24:29.688 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:29.705 INFO:teuthology.orchestra.run.smithi119.stdout:47244640261 2024-04-16T19:24:29.718 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640261 got 47244640261 for osd.3 2024-04-16T19:24:29.719 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:29.741 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:29.753 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640260 for osd.5 2024-04-16T19:24:29.753 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:29.760 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:29.774 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640260 for osd.4 2024-04-16T19:24:29.774 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:29.792 INFO:teuthology.orchestra.run.smithi119.stdout:47244640260 2024-04-16T19:24:29.804 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640260 got 47244640260 for osd.0 2024-04-16T19:24:29.805 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:30.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-16T19:24:30.914 INFO:teuthology.orchestra.run.smithi119.stdout:47244640261 2024-04-16T19:24:30.927 INFO:tasks.ceph.ceph_manager.c2:need seq 47244640261 got 47244640261 for osd.1 2024-04-16T19:24:30.927 DEBUG:teuthology.parallel:result is None 2024-04-16T19:24:30.927 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-16T19:24:30.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-16T19:24:31.217 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:31.218 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-04-16T19:24:31.230 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T19:24:29.292313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":164932,"kb_used_data":1928,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":628980668,"statfs":{"total":644245094400,"available":644076204032,"internally_reserved":0,"allocated":1974272,"data_stored":1407458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.601119+0000","last_change":"2024-04-16T19:24:26.859991+0000","last_active":"2024-04-16T19:24:27.601119+0000","last_peered":"2024-04-16T19:24:27.601119+0000","last_clean":"2024-04-16T19:24:27.601119+0000","last_became_active":"2024-04-16T19:24:26.859761+0000","last_became_peered":"2024-04-16T19:24:26.859761+0000","last_unstale":"2024-04-16T19:24:27.601119+0000","last_undegraded":"2024-04-16T19:24:27.601119+0000","last_fullsized":"2024-04-16T19:24:27.601119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715052+0000","last_change":"2024-04-16T19:24:26.917359+0000","last_active":"2024-04-16T19:24:27.715052+0000","last_peered":"2024-04-16T19:24:27.715052+0000","last_clean":"2024-04-16T19:24:27.715052+0000","last_became_active":"2024-04-16T19:24:26.917048+0000","last_became_peered":"2024-04-16T19:24:26.917048+0000","last_unstale":"2024-04-16T19:24:27.715052+0000","last_undegraded":"2024-04-16T19:24:27.715052+0000","last_fullsized":"2024-04-16T19:24:27.715052+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:07.582985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:24:26.860637+0000","last_change":"2024-04-16T19:24:26.860300+0000","last_active":"2024-04-16T19:24:26.860637+0000","last_peered":"2024-04-16T19:24:26.860637+0000","last_clean":"2024-04-16T19:24:26.860637+0000","last_became_active":"2024-04-16T19:24:26.860017+0000","last_became_peered":"2024-04-16T19:24:26.860017+0000","last_unstale":"2024-04-16T19:24:26.860637+0000","last_undegraded":"2024-04-16T19:24:26.860637+0000","last_fullsized":"2024-04-16T19:24:26.860637+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:28.486230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:24:26.605985+0000","last_change":"2024-04-16T19:24:26.605029+0000","last_active":"2024-04-16T19:24:26.605985+0000","last_peered":"2024-04-16T19:24:26.605985+0000","last_clean":"2024-04-16T19:24:26.605985+0000","last_became_active":"2024-04-16T19:24:26.604809+0000","last_became_peered":"2024-04-16T19:24:26.604809+0000","last_unstale":"2024-04-16T19:24:26.605985+0000","last_undegraded":"2024-04-16T19:24:26.605985+0000","last_fullsized":"2024-04-16T19:24:26.605985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:46.171011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715072+0000","last_change":"2024-04-16T19:24:26.601696+0000","last_active":"2024-04-16T19:24:27.715072+0000","last_peered":"2024-04-16T19:24:27.715072+0000","last_clean":"2024-04-16T19:24:27.715072+0000","last_became_active":"2024-04-16T19:24:26.601437+0000","last_became_peered":"2024-04-16T19:24:26.601437+0000","last_unstale":"2024-04-16T19:24:27.715072+0000","last_undegraded":"2024-04-16T19:24:27.715072+0000","last_fullsized":"2024-04-16T19:24:27.715072+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:53.015728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:24:26.602672+0000","last_change":"2024-04-16T19:24:26.602312+0000","last_active":"2024-04-16T19:24:26.602672+0000","last_peered":"2024-04-16T19:24:26.602672+0000","last_clean":"2024-04-16T19:24:26.602672+0000","last_became_active":"2024-04-16T19:24:26.602073+0000","last_became_peered":"2024-04-16T19:24:26.602073+0000","last_unstale":"2024-04-16T19:24:26.602672+0000","last_undegraded":"2024-04-16T19:24:26.602672+0000","last_fullsized":"2024-04-16T19:24:26.602672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:27:16.437023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-16T19:24:26.917424+0000","last_change":"2024-04-16T19:24:26.916733+0000","last_active":"2024-04-16T19:24:26.917424+0000","last_peered":"2024-04-16T19:24:26.917424+0000","last_clean":"2024-04-16T19:24:26.917424+0000","last_became_active":"2024-04-16T19:24:26.916534+0000","last_became_peered":"2024-04-16T19:24:26.916534+0000","last_unstale":"2024-04-16T19:24:26.917424+0000","last_undegraded":"2024-04-16T19:24:26.917424+0000","last_fullsized":"2024-04-16T19:24:26.917424+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:46.171011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.601067+0000","last_change":"2024-04-16T19:24:26.917240+0000","last_active":"2024-04-16T19:24:27.601067+0000","last_peered":"2024-04-16T19:24:27.601067+0000","last_clean":"2024-04-16T19:24:27.601067+0000","last_became_active":"2024-04-16T19:24:26.916933+0000","last_became_peered":"2024-04-16T19:24:26.916933+0000","last_unstale":"2024-04-16T19:24:27.601067+0000","last_undegraded":"2024-04-16T19:24:27.601067+0000","last_fullsized":"2024-04-16T19:24:27.601067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:07.582985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715066+0000","last_change":"2024-04-16T19:24:26.917228+0000","last_active":"2024-04-16T19:24:27.715066+0000","last_peered":"2024-04-16T19:24:27.715066+0000","last_clean":"2024-04-16T19:24:27.715066+0000","last_became_active":"2024-04-16T19:24:26.916904+0000","last_became_peered":"2024-04-16T19:24:26.916904+0000","last_unstale":"2024-04-16T19:24:27.715066+0000","last_undegraded":"2024-04-16T19:24:27.715066+0000","last_fullsized":"2024-04-16T19:24:27.715066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":131072,"data_stored":37787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":131072,"data_stored":37787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28160,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829440,"statfs":{"total":107374182400,"available":107345346560,"internally_reserved":0,"allocated":724992,"data_stored":628155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830444,"statfs":{"total":107374182400,"available":107346374656,"internally_reserved":0,"allocated":131072,"data_stored":37787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28160,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829440,"statfs":{"total":107374182400,"available":107345346560,"internally_reserved":0,"allocated":724992,"data_stored":628155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":131072,"data_stored":37787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T19:24:31.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-16T19:24:31.519 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:31.519 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-04-16T19:24:31.531 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-16T19:24:31.292728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":164264,"kb_used_data":2000,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":628981336,"statfs":{"total":644245094400,"available":644076888064,"internally_reserved":0,"allocated":2048000,"data_stored":1438982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000414"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.601119+0000","last_change":"2024-04-16T19:24:26.859991+0000","last_active":"2024-04-16T19:24:27.601119+0000","last_peered":"2024-04-16T19:24:27.601119+0000","last_clean":"2024-04-16T19:24:27.601119+0000","last_became_active":"2024-04-16T19:24:26.859761+0000","last_became_peered":"2024-04-16T19:24:26.859761+0000","last_unstale":"2024-04-16T19:24:27.601119+0000","last_undegraded":"2024-04-16T19:24:27.601119+0000","last_fullsized":"2024-04-16T19:24:27.601119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715052+0000","last_change":"2024-04-16T19:24:26.917359+0000","last_active":"2024-04-16T19:24:27.715052+0000","last_peered":"2024-04-16T19:24:27.715052+0000","last_clean":"2024-04-16T19:24:27.715052+0000","last_became_active":"2024-04-16T19:24:26.917048+0000","last_became_peered":"2024-04-16T19:24:26.917048+0000","last_unstale":"2024-04-16T19:24:27.715052+0000","last_undegraded":"2024-04-16T19:24:27.715052+0000","last_fullsized":"2024-04-16T19:24:27.715052+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:07.582985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:28.015935+0000","last_change":"2024-04-16T19:24:26.860300+0000","last_active":"2024-04-16T19:24:28.015935+0000","last_peered":"2024-04-16T19:24:28.015935+0000","last_clean":"2024-04-16T19:24:28.015935+0000","last_became_active":"2024-04-16T19:24:26.860017+0000","last_became_peered":"2024-04-16T19:24:26.860017+0000","last_unstale":"2024-04-16T19:24:28.015935+0000","last_undegraded":"2024-04-16T19:24:28.015935+0000","last_fullsized":"2024-04-16T19:24:28.015935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:53.015728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:28.516773+0000","last_change":"2024-04-16T19:24:26.605029+0000","last_active":"2024-04-16T19:24:28.516773+0000","last_peered":"2024-04-16T19:24:28.516773+0000","last_clean":"2024-04-16T19:24:28.516773+0000","last_became_active":"2024-04-16T19:24:26.604809+0000","last_became_peered":"2024-04-16T19:24:26.604809+0000","last_unstale":"2024-04-16T19:24:28.516773+0000","last_undegraded":"2024-04-16T19:24:28.516773+0000","last_fullsized":"2024-04-16T19:24:28.516773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:07.582985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715072+0000","last_change":"2024-04-16T19:24:26.601696+0000","last_active":"2024-04-16T19:24:27.715072+0000","last_peered":"2024-04-16T19:24:27.715072+0000","last_clean":"2024-04-16T19:24:27.715072+0000","last_became_active":"2024-04-16T19:24:26.601437+0000","last_became_peered":"2024-04-16T19:24:26.601437+0000","last_unstale":"2024-04-16T19:24:27.715072+0000","last_undegraded":"2024-04-16T19:24:27.715072+0000","last_fullsized":"2024-04-16T19:24:27.715072+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:53.015728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:28.516960+0000","last_change":"2024-04-16T19:24:26.602312+0000","last_active":"2024-04-16T19:24:28.516960+0000","last_peered":"2024-04-16T19:24:28.516960+0000","last_clean":"2024-04-16T19:24:28.516960+0000","last_became_active":"2024-04-16T19:24:26.602073+0000","last_became_peered":"2024-04-16T19:24:26.602073+0000","last_unstale":"2024-04-16T19:24:28.516960+0000","last_undegraded":"2024-04-16T19:24:28.516960+0000","last_fullsized":"2024-04-16T19:24:28.516960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:28.015909+0000","last_change":"2024-04-16T19:24:26.916733+0000","last_active":"2024-04-16T19:24:28.015909+0000","last_peered":"2024-04-16T19:24:28.015909+0000","last_clean":"2024-04-16T19:24:28.015909+0000","last_became_active":"2024-04-16T19:24:26.916534+0000","last_became_peered":"2024-04-16T19:24:26.916534+0000","last_unstale":"2024-04-16T19:24:28.015909+0000","last_undegraded":"2024-04-16T19:24:28.015909+0000","last_fullsized":"2024-04-16T19:24:28.015909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.601067+0000","last_change":"2024-04-16T19:24:26.917240+0000","last_active":"2024-04-16T19:24:27.601067+0000","last_peered":"2024-04-16T19:24:27.601067+0000","last_clean":"2024-04-16T19:24:27.601067+0000","last_became_active":"2024-04-16T19:24:26.916933+0000","last_became_peered":"2024-04-16T19:24:26.916933+0000","last_unstale":"2024-04-16T19:24:27.601067+0000","last_undegraded":"2024-04-16T19:24:27.601067+0000","last_fullsized":"2024-04-16T19:24:27.601067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:07.582985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-16T19:24:27.715066+0000","last_change":"2024-04-16T19:24:26.917228+0000","last_active":"2024-04-16T19:24:27.715066+0000","last_peered":"2024-04-16T19:24:27.715066+0000","last_clean":"2024-04-16T19:24:27.715066+0000","last_became_active":"2024-04-16T19:24:26.916904+0000","last_became_peered":"2024-04-16T19:24:26.916904+0000","last_unstale":"2024-04-16T19:24:27.715066+0000","last_undegraded":"2024-04-16T19:24:27.715066+0000","last_fullsized":"2024-04-16T19:24:27.715066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:25.587691+0000","last_clean_scrub_stamp":"2024-04-16T19:24:25.587691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:53.164390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27168,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830432,"statfs":{"total":107374182400,"available":107346362368,"internally_reserved":0,"allocated":143360,"data_stored":43041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27168,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830432,"statfs":{"total":107374182400,"available":107346362368,"internally_reserved":0,"allocated":143360,"data_stored":43041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27868,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829732,"statfs":{"total":107374182400,"available":107345645568,"internally_reserved":0,"allocated":737280,"data_stored":633409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830440,"statfs":{"total":107374182400,"available":107346370560,"internally_reserved":0,"allocated":143360,"data_stored":43041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829860,"statfs":{"total":107374182400,"available":107345776640,"internally_reserved":0,"allocated":737280,"data_stored":633409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830440,"statfs":{"total":107374182400,"available":107346370560,"internally_reserved":0,"allocated":143360,"data_stored":43041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T19:24:31.532 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-16T19:24:31.532 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-16T19:24:31.532 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-16T19:24:31.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-16T19:24:31.856 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-16T19:24:31.857 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T19:24:31.869 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-16T19:24:31.870 INFO:teuthology.run_tasks:Running task rgw... 2024-04-16T19:24:31.924 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-16T19:24:31.924 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-16T19:24:31.924 INFO:tasks.rgw:Creating data pools 2024-04-16T19:24:31.924 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-16T19:24:31.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-16T19:24:32.371 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.data' created 2024-04-16T19:24:32.388 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-16T19:24:33.372 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-16T19:24:33.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-16T19:24:34.379 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.index' created 2024-04-16T19:24:34.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-16T19:24:35.380 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-16T19:24:35.409 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-16T19:24:35.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-16T19:24:35.704 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-16T19:24:35.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-16T19:24:36.386 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-16T19:24:36.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-16T19:24:36.688 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-16T19:24:36.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-16T19:24:37.387 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-16T19:24:37.401 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-16T19:24:37.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-16T19:24:38.616 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'default.rgw.buckets.data' created 2024-04-16T19:24:38.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-16T19:24:39.619 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-16T19:24:39.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-16T19:24:40.624 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'default.rgw.buckets.index' created 2024-04-16T19:24:40.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-16T19:24:41.630 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-16T19:24:41.653 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-16T19:24:41.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-16T19:24:41.950 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-16T19:24:41.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-16T19:24:42.634 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-16T19:24:42.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-16T19:24:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-16T19:24:42.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-16T19:24:43.637 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-16T19:24:43.652 DEBUG:tasks.rgw:Pools created 2024-04-16T19:24:43.652 INFO:tasks.rgw:Configuring compression type = random 2024-04-16T19:24:43.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-16T19:24:43.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-16T19:24:43.653 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-16T19:24:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:24:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:24:45.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.444+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:45.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 20 realm 2024-04-16T19:24:45.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:45.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-16T19:24:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.445+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.446+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.446+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.447+0000 7ffa17b4b800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:45.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.447+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.448+0000 7ffa17b4b800 20 rados_obj.operate() r=0 bl.length=993 2024-04-16T19:24:45.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.448+0000 7ffa17b4b800 20 searching for the correct realm 2024-04-16T19:24:45.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got zone_info.648d3679-4f76-42da-a9ee-3ca9ad0fef3d 2024-04-16T19:24:45.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-16T19:24:45.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got default.zone. 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got zonegroup_info.275e75a5-46d7-4f27-99cb-698fef49c40b 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got zone_names.default 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.459+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 rados_obj.operate() r=0 bl.length=436 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 zone default found 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 4 Realm: () 2024-04-16T19:24:45.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 4 ZoneGroup: default (275e75a5-46d7-4f27-99cb-698fef49c40b) 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 4 Zone: default (648d3679-4f76-42da-a9ee-3ca9ad0fef3d) 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 zonegroup default 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.460+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.461+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:45.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:45.461+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:47.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:47.422+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:47.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:47.422+0000 7ffa17b4b800 20 rados->read ofs=0 len=0 2024-04-16T19:24:47.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:47.423+0000 7ffa17b4b800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:47.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:47.423+0000 7ffa17b4b800 20 started sync module instance, tier type = 2024-04-16T19:24:47.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:47.423+0000 7ffa17b4b800 20 started zone id=648d3679-4f76-42da-a9ee-3ca9ad0fef3d (name=default) with tier type = 2024-04-16T19:24:49.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=0 2024-04-16T19:24:49.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=1 2024-04-16T19:24:49.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=2 2024-04-16T19:24:49.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=3 2024-04-16T19:24:49.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=4 2024-04-16T19:24:49.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=5 2024-04-16T19:24:49.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.453+0000 7ffa17b4b800 20 add_watcher() i=6 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.454+0000 7ffa17b4b800 20 add_watcher() i=7 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.454+0000 7ffa17b4b800 2 all 8 watchers are set, enabling cache 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.454+0000 7ffa17b4b800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.454+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.454+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-16T19:24:49.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.455+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:49.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.456+0000 7ffa17b4b800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-16T19:24:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.456+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-04-16T19:24:49.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.457+0000 7ffa17b4b800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.465+0000 7ff8c7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:24:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.465+0000 7ffa17b4b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:24:49.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:49.465+0000 7ffa17b4b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.430+0000 7ffa17b4b800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.431+0000 7ffa17b4b800 5 note: GC not initialized 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.431+0000 7ff8c6fe5640 20 reqs_thread_entry: start 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.431+0000 7ffa17b4b800 20 init_complete bucket index max shards: 11 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.431+0000 7ffa17b4b800 20 Filter name: none 2024-04-16T19:24:52.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.431+0000 7ff8c4fe1640 20 reqs_thread_entry: start 2024-04-16T19:24:52.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.456+0000 7ff8c7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:24:52.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.459+0000 7ffa17b4b800 20 remove_watcher() i=0 2024-04-16T19:24:52.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.459+0000 7ffa17b4b800 2 removed watcher, disabling cache 2024-04-16T19:24:52.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.462+0000 7ffa17b4b800 20 remove_watcher() i=1 2024-04-16T19:24:52.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.463+0000 7ffa17b4b800 20 remove_watcher() i=2 2024-04-16T19:24:52.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.467+0000 7ffa17b4b800 20 remove_watcher() i=3 2024-04-16T19:24:52.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.468+0000 7ffa17b4b800 20 remove_watcher() i=4 2024-04-16T19:24:52.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.470+0000 7ffa17b4b800 20 remove_watcher() i=5 2024-04-16T19:24:52.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.471+0000 7ffa17b4b800 20 remove_watcher() i=6 2024-04-16T19:24:52.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.472+0000 7ffa17b4b800 20 remove_watcher() i=7 2024-04-16T19:24:52.492 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:24:52.492 DEBUG:tasks.util.rgw: json result: [] 2024-04-16T19:24:52.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:24:52.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:24:52.492 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-16T19:24:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:24:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:24:52.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.571+0000 7f94c300f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:24:52.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.571+0000 7f94c300f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:24:52.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:24:52.573+0000 7f9380fe1640 20 reqs_thread_entry: start 2024-04-16T19:24:52.590 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"648d3679-4f76-42da-a9ee-3ca9ad0fef3d","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-16T19:24:52.590 DEBUG:tasks.util.rgw: json result: {'id': '648d3679-4f76-42da-a9ee-3ca9ad0fef3d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-16T19:24:52.590 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-16T19:24:52.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-16T19:24:52.591 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-16T19:24:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:24:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:24:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.730+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 20 realm 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.731+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.732+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.732+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.732+0000 7efd08067800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:52.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.732+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.733+0000 7efd08067800 20 rados_obj.operate() r=0 bl.length=1009 2024-04-16T19:24:52.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.733+0000 7efd08067800 20 searching for the correct realm 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got zone_info.648d3679-4f76-42da-a9ee-3ca9ad0fef3d 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got default.zone. 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got zonegroup_info.275e75a5-46d7-4f27-99cb-698fef49c40b 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got zone_names.default 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.743+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados_obj.operate() r=0 bl.length=436 2024-04-16T19:24:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 zone default found 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 4 Realm: () 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 4 ZoneGroup: default (275e75a5-46d7-4f27-99cb-698fef49c40b) 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 4 Zone: default (648d3679-4f76-42da-a9ee-3ca9ad0fef3d) 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 zonegroup default 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.744+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.745+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.745+0000 7efd08067800 20 rados->read ofs=0 len=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.745+0000 7efd08067800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.745+0000 7efd08067800 20 started sync module instance, tier type = 2024-04-16T19:24:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.745+0000 7efd08067800 20 started zone id=648d3679-4f76-42da-a9ee-3ca9ad0fef3d (name=default) with tier type = 2024-04-16T19:24:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=0 2024-04-16T19:24:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=1 2024-04-16T19:24:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=2 2024-04-16T19:24:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=3 2024-04-16T19:24:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=4 2024-04-16T19:24:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=5 2024-04-16T19:24:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.753+0000 7efd08067800 20 add_watcher() i=6 2024-04-16T19:24:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.754+0000 7efd08067800 20 add_watcher() i=7 2024-04-16T19:24:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.754+0000 7efd08067800 2 all 8 watchers are set, enabling cache 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.756+0000 7efbb77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.756+0000 7efd08067800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.756+0000 7efd08067800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.756+0000 7efd08067800 5 note: GC not initialized 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.757+0000 7efbb67fc640 20 reqs_thread_entry: start 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.757+0000 7efd08067800 20 init_complete bucket index max shards: 11 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.757+0000 7efd08067800 20 Filter name: none 2024-04-16T19:24:52.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.757+0000 7efb27fff640 20 reqs_thread_entry: start 2024-04-16T19:24:52.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.767+0000 7efbb77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:24:52.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.769+0000 7efd08067800 20 remove_watcher() i=0 2024-04-16T19:24:52.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.769+0000 7efd08067800 2 removed watcher, disabling cache 2024-04-16T19:24:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.770+0000 7efd08067800 20 remove_watcher() i=1 2024-04-16T19:24:52.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.771+0000 7efd08067800 20 remove_watcher() i=2 2024-04-16T19:24:52.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.771+0000 7efd08067800 20 remove_watcher() i=3 2024-04-16T19:24:52.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.772+0000 7efd08067800 20 remove_watcher() i=4 2024-04-16T19:24:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.773+0000 7efd08067800 20 remove_watcher() i=5 2024-04-16T19:24:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.774+0000 7efd08067800 20 remove_watcher() i=6 2024-04-16T19:24:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.775+0000 7efd08067800 20 remove_watcher() i=7 2024-04-16T19:24:52.789 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-04-16T19:24:52.789 DEBUG:tasks.util.rgw: json result: [] 2024-04-16T19:24:52.789 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:24:52.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:24:52.790 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-16T19:24:52.839 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:24:52.839 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:24:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.865+0000 7f1244727800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:24:52.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.865+0000 7f1244727800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:24:52.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T19:24:52.868+0000 7f10ae7fc640 20 reqs_thread_entry: start 2024-04-16T19:24:52.883 INFO:teuthology.orchestra.run.smithi069.stdout:{"id":"648d3679-4f76-42da-a9ee-3ca9ad0fef3d","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-16T19:24:52.884 DEBUG:tasks.util.rgw: json result: {'id': '648d3679-4f76-42da-a9ee-3ca9ad0fef3d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-16T19:24:52.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-16T19:24:52.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-16T19:24:52.884 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-16T19:24:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:24:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:24:54.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.698+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:54.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 realm 2024-04-16T19:24:54.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:54.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-16T19:24:54.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:54.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.699+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.700+0000 7fc16cc44800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:54.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.700+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.700+0000 7fc16cc44800 20 rados_obj.operate() r=0 bl.length=993 2024-04-16T19:24:54.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.700+0000 7fc16cc44800 20 searching for the correct realm 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got zonegroup_info.415d1141-b79e-46c8-a660-518b9f2d542c 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got default.zone. 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got zone_info.4df7a835-0aea-4ad6-a2db-32592cf718e0 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got zone_names.default 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.711+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:24:54.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 rados_obj.operate() r=0 bl.length=436 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 zone default found 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 4 Realm: () 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 4 ZoneGroup: default (415d1141-b79e-46c8-a660-518b9f2d542c) 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 4 Zone: default (4df7a835-0aea-4ad6-a2db-32592cf718e0) 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 zonegroup default 2024-04-16T19:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.712+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:54.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.713+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:54.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:54.713+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:56.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:56.675+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:56.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:56.675+0000 7fc16cc44800 20 rados->read ofs=0 len=0 2024-04-16T19:24:56.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:56.676+0000 7fc16cc44800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:24:56.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:56.676+0000 7fc16cc44800 20 started sync module instance, tier type = 2024-04-16T19:24:56.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:56.676+0000 7fc16cc44800 20 started zone id=4df7a835-0aea-4ad6-a2db-32592cf718e0 (name=default) with tier type = 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=0 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=1 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=2 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=3 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=4 2024-04-16T19:24:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=5 2024-04-16T19:24:58.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.709+0000 7fc16cc44800 20 add_watcher() i=6 2024-04-16T19:24:58.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 add_watcher() i=7 2024-04-16T19:24:58.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 2 all 8 watchers are set, enabling cache 2024-04-16T19:24:58.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-16T19:24:58.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-16T19:24:58.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:58.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.712+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:58.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-16T19:24:58.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.713+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-04-16T19:24:58.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.714+0000 7fc16cc44800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-04-16T19:24:58.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.717+0000 7fc01cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:24:58.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.717+0000 7fc16cc44800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:24:58.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:24:58.717+0000 7fc16cc44800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.686+0000 7fc16cc44800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.686+0000 7fc16cc44800 5 note: GC not initialized 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.687+0000 7fbf877fe640 20 reqs_thread_entry: start 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.687+0000 7fc16cc44800 20 init_complete bucket index max shards: 11 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.687+0000 7fc16cc44800 20 Filter name: none 2024-04-16T19:25:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.687+0000 7fbf857fa640 20 reqs_thread_entry: start 2024-04-16T19:25:01.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.712+0000 7fc01cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:25:01.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.715+0000 7fc16cc44800 20 remove_watcher() i=0 2024-04-16T19:25:01.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.715+0000 7fc16cc44800 2 removed watcher, disabling cache 2024-04-16T19:25:01.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.716+0000 7fc16cc44800 20 remove_watcher() i=1 2024-04-16T19:25:01.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.718+0000 7fc16cc44800 20 remove_watcher() i=2 2024-04-16T19:25:01.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.721+0000 7fc16cc44800 20 remove_watcher() i=3 2024-04-16T19:25:01.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.723+0000 7fc16cc44800 20 remove_watcher() i=4 2024-04-16T19:25:01.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.725+0000 7fc16cc44800 20 remove_watcher() i=5 2024-04-16T19:25:01.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.726+0000 7fc16cc44800 20 remove_watcher() i=6 2024-04-16T19:25:01.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.727+0000 7fc16cc44800 20 remove_watcher() i=7 2024-04-16T19:25:01.782 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-04-16T19:25:01.782 DEBUG:tasks.util.rgw: json result: [] 2024-04-16T19:25:01.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:25:01.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:25:01.783 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-16T19:25:01.834 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:01.834 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:01.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.858+0000 7fd0dbd3b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:25:01.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.858+0000 7fd0dbd3b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:25:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:01.861+0000 7fcf4dffb640 20 reqs_thread_entry: start 2024-04-16T19:25:01.877 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"4df7a835-0aea-4ad6-a2db-32592cf718e0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-16T19:25:01.878 DEBUG:tasks.util.rgw: json result: {'id': '4df7a835-0aea-4ad6-a2db-32592cf718e0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-16T19:25:01.878 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-16T19:25:01.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-16T19:25:01.878 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-16T19:25:01.976 INFO:teuthology.orchestra.run.smithi151.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:01.976 INFO:teuthology.orchestra.run.smithi151.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.011+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.013 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 20 realm 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.012+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.013+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.013+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.013+0000 7fb627a10800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:25:02.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.013+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.014+0000 7fb627a10800 20 rados_obj.operate() r=0 bl.length=1009 2024-04-16T19:25:02.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.014+0000 7fb627a10800 20 searching for the correct realm 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got zonegroup_info.415d1141-b79e-46c8-a660-518b9f2d542c 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got default.zone. 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got zone_info.4df7a835-0aea-4ad6-a2db-32592cf718e0 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got zone_names.default 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-16T19:25:02.025 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T19:25:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.024+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados_obj.operate() r=0 bl.length=436 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 zone default found 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 4 Realm: () 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 4 ZoneGroup: default (415d1141-b79e-46c8-a660-518b9f2d542c) 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 4 Zone: default (4df7a835-0aea-4ad6-a2db-32592cf718e0) 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 zonegroup default 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.027 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.025+0000 7fb627a10800 20 rados->read ofs=0 len=0 2024-04-16T19:25:02.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.026+0000 7fb627a10800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T19:25:02.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.026+0000 7fb627a10800 20 started sync module instance, tier type = 2024-04-16T19:25:02.028 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.026+0000 7fb627a10800 20 started zone id=4df7a835-0aea-4ad6-a2db-32592cf718e0 (name=default) with tier type = 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=0 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=1 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=2 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=3 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=4 2024-04-16T19:25:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=5 2024-04-16T19:25:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.036+0000 7fb627a10800 20 add_watcher() i=6 2024-04-16T19:25:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.037+0000 7fb627a10800 20 add_watcher() i=7 2024-04-16T19:25:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.037+0000 7fb627a10800 2 all 8 watchers are set, enabling cache 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.039+0000 7fb4d97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.039+0000 7fb627a10800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.039+0000 7fb627a10800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.039+0000 7fb627a10800 5 note: GC not initialized 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.039+0000 7fb453fff640 20 reqs_thread_entry: start 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.040+0000 7fb627a10800 20 init_complete bucket index max shards: 11 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.040+0000 7fb627a10800 20 Filter name: none 2024-04-16T19:25:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.040+0000 7fb451ffb640 20 reqs_thread_entry: start 2024-04-16T19:25:02.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.049+0000 7fb4d97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T19:25:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.051+0000 7fb627a10800 20 remove_watcher() i=0 2024-04-16T19:25:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.051+0000 7fb627a10800 2 removed watcher, disabling cache 2024-04-16T19:25:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.052+0000 7fb627a10800 20 remove_watcher() i=1 2024-04-16T19:25:02.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.053+0000 7fb627a10800 20 remove_watcher() i=2 2024-04-16T19:25:02.054 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.054+0000 7fb627a10800 20 remove_watcher() i=3 2024-04-16T19:25:02.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.055+0000 7fb627a10800 20 remove_watcher() i=4 2024-04-16T19:25:02.056 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.055+0000 7fb627a10800 20 remove_watcher() i=5 2024-04-16T19:25:02.057 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.056+0000 7fb627a10800 20 remove_watcher() i=6 2024-04-16T19:25:02.058 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.057+0000 7fb627a10800 20 remove_watcher() i=7 2024-04-16T19:25:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:[] 2024-04-16T19:25:02.070 DEBUG:tasks.util.rgw: json result: [] 2024-04-16T19:25:02.070 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:25:02.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-16T19:25:02.070 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-16T19:25:02.119 INFO:teuthology.orchestra.run.smithi151.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:02.119 INFO:teuthology.orchestra.run.smithi151.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:02.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.144+0000 7f32d3f33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T19:25:02.145 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.144+0000 7f32d3f33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T19:25:02.147 INFO:teuthology.orchestra.run.smithi151.stderr:2024-04-16T19:25:02.146+0000 7f314dffb640 20 reqs_thread_entry: start 2024-04-16T19:25:02.162 INFO:teuthology.orchestra.run.smithi151.stdout:{"id":"4df7a835-0aea-4ad6-a2db-32592cf718e0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-16T19:25:02.163 DEBUG:tasks.util.rgw: json result: {'id': '4df7a835-0aea-4ad6-a2db-32592cf718e0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-16T19:25:02.163 INFO:tasks.rgw:Starting rgw... 2024-04-16T19:25:02.163 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-16T19:25:02.163 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-16T19:25:02.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo echo -n http://smithi026.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.199 INFO:teuthology.orchestra.run.smithi026.stdout:http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:02.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.270 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-16T19:25:02.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-16T19:25:02.312 INFO:tasks.rgw.c1.client.0:Started 2024-04-16T19:25:02.312 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-16T19:25:02.312 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-16T19:25:02.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo echo -n http://smithi069.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.347 INFO:teuthology.orchestra.run.smithi069.stdout:http://smithi069.front.sepia.ceph.com:8001 2024-04-16T19:25:02.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.418 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-16T19:25:02.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-16T19:25:02.461 INFO:tasks.rgw.c1.client.1:Started 2024-04-16T19:25:02.461 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-16T19:25:02.461 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-16T19:25:02.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo echo -n http://smithi119.front.sepia.ceph.com:8000 | sudo tee /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.492 INFO:teuthology.orchestra.run.smithi119.stdout:http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:02.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.501 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:25:02.500+0000 7f0614d23880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:02.560 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T19:25:02.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T19:25:02.603 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T19:25:02.603 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-16T19:25:02.603 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-16T19:25:02.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo echo -n http://smithi151.front.sepia.ceph.com:8001 | sudo tee /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.635 INFO:teuthology.orchestra.run.smithi151.stdout:http://smithi151.front.sepia.ceph.com:8001 2024-04-16T19:25:02.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-04-16T19:25:02.659 INFO:tasks.rgw.c1.client.1.smithi069.stdout:2024-04-16T19:25:02.658+0000 7f7016f22880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:02.704 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-16T19:25:02.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-16T19:25:02.747 INFO:tasks.rgw.c2.client.1:Started 2024-04-16T19:25:02.747 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi026.front.sepia.ceph.com:8000/ 2024-04-16T19:25:02.747 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000/ 2024-04-16T19:25:02.780 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:02.780 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:02.781 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:02.782 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2024-04-16T19:25:02.782 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi069.front.sepia.ceph.com:8001/ 2024-04-16T19:25:02.782 DEBUG:teuthology.orchestra.run.smithi069:> curl http://smithi069.front.sepia.ceph.com:8001/ 2024-04-16T19:25:02.796 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:25:02.795+0000 7f6af1520880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:02.806 INFO:teuthology.orchestra.run.smithi069.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:02.807 INFO:teuthology.orchestra.run.smithi069.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:02.807 INFO:teuthology.orchestra.run.smithi069.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:02.808 INFO:teuthology.orchestra.run.smithi069.stdout:anonymous 2024-04-16T19:25:02.808 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi119.front.sepia.ceph.com:8000/ 2024-04-16T19:25:02.809 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000/ 2024-04-16T19:25:02.841 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:02.841 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:02.842 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:02.842 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T19:25:02.843 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi151.front.sepia.ceph.com:8001/ 2024-04-16T19:25:02.843 DEBUG:teuthology.orchestra.run.smithi151:> curl http://smithi151.front.sepia.ceph.com:8001/ 2024-04-16T19:25:02.875 INFO:teuthology.orchestra.run.smithi151.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T19:25:02.878 INFO:teuthology.orchestra.run.smithi151.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:02.878 INFO:teuthology.orchestra.run.smithi151.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T19:25:02.878 INFO:teuthology.orchestra.run.smithi151.stderr:curl: (7) Failed to connect to smithi151.front.sepia.ceph.com port 8001: Connection refused 2024-04-16T19:25:02.996 INFO:tasks.rgw.c2.client.1.smithi151.stdout:2024-04-16T19:25:02.995+0000 7ffbd942f880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:03.879 DEBUG:teuthology.orchestra.run.smithi151:> curl http://smithi151.front.sepia.ceph.com:8001/ 2024-04-16T19:25:03.899 INFO:teuthology.orchestra.run.smithi151.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:03.899 INFO:teuthology.orchestra.run.smithi151.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:03.900 INFO:teuthology.orchestra.run.smithi151.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:03.901 INFO:teuthology.orchestra.run.smithi151.stdout:anonymous 2024-04-16T19:25:03.901 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-16T19:25:07.280 INFO:tasks.rgw_multisite:creating realm.. 2024-04-16T19:25:07.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.281 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:07.387 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"82a6ab2b-1604-42ea-987b-3570b8df3977","name":"test-realm","current_period":"05d34c3b-cf84-4753-a65b-67914d39a038","epoch":1} 2024-04-16T19:25:07.411 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-16T19:25:07.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.412 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:07.461 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:07.461 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:07.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:25:07.489+0000 7f3774217800 -1 old format 2024-04-16T19:25:07.506 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2024-04-16T19:25:07.506 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-16T19:25:07.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:07.507 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key W5L6GRR79K371291 --secret jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY --endpoints http://smithi026.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:07.602 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:07.602 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:07.633 INFO:teuthology.orchestra.run.smithi026.stderr:NOTICE: set zone's realm_id=82a6ab2b-1604-42ea-987b-3570b8df3977 2024-04-16T19:25:07.661 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","notif_pool":"test-zone1.rgw.log:notif","topics_pool":"test-zone1.rgw.meta:topics","account_pool":"test-zone1.rgw.meta:accounts","group_pool":"test-zone1.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:25:07.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-04-16T19:25:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-04-16T19:25:08.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-04-16T19:25:09.445 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-04-16T19:25:09.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:09.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:09.478 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:09.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:09.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:09.573 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","notif_pool":"test-zone1.rgw.log:notif","topics_pool":"test-zone1.rgw.meta:topics","account_pool":"test-zone1.rgw.meta:accounts","group_pool":"test-zone1.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:25:09.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:09.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:09.574 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:25:09.654+0000 7f5ffcf2b800 0 period (05d34c3b-cf84-4753-a65b-67914d39a038 does not have zone b482a2a9-ef36-488b-9666-ada656453328 configured 2024-04-16T19:25:16.567 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":1,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:25:16.568 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-16T19:25:16.568 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T19:25:16.568 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:25:16.567+0000 7f06114ca640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 37916) UID: 0 2024-04-16T19:25:16.568 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:25:16.568+0000 7f0614d23880 -1 shutting down 2024-04-16T19:25:21.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-16T19:25:22.670 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-16T19:25:22.670 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-16T19:25:22.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-16T19:25:22.673 INFO:tasks.rgw.c1.client.0:Started 2024-04-16T19:25:22.674 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T19:25:22.697 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:22.697 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:22.697 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T19:25:22.697 INFO:teuthology.orchestra.run.smithi026.stderr:curl: (7) Failed to connect to smithi026.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T19:25:22.872 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:25:22.870+0000 7fab25f34880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:23.698 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:23.719 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:23.719 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:23.720 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:23.720 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2024-04-16T19:25:23.720 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-16T19:25:23.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:23.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:23.721 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key W5L6GRR79K371291 --secret jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:23.874 INFO:teuthology.orchestra.run.smithi026.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY","active":true,"create_date":"2024-04-16T19:25:23.870246Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-16T19:25:23.870235Z","tags":[],"group_ids":[]} 2024-04-16T19:25:23.895 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-16T19:25:23.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:23.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:23.895 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi026.front.sepia.ceph.com:8000 --access-key W5L6GRR79K371291 --secret jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:23.953 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:23.953 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:24.021 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"82a6ab2b-1604-42ea-987b-3570b8df3977","name":"test-realm","current_period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":2} 2024-04-16T19:25:24.022 INFO:tasks.rgw_multisite:creating zone.. 2024-04-16T19:25:24.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--endpoints', 'http://smithi119.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:24.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--endpoints', 'http://smithi119.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:24.022 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key W5L6GRR79K371291 --secret jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY --endpoints http://smithi119.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:24.076 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:24.076 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:24.106 INFO:teuthology.orchestra.run.smithi119.stderr:NOTICE: set zone's realm_id=82a6ab2b-1604-42ea-987b-3570b8df3977 2024-04-16T19:25:24.134 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:25:24.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-04-16T19:25:24.714 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-04-16T19:25:24.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-04-16T19:25:25.720 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-04-16T19:25:25.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:25.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:25.749 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:25.802 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:25.844 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:25:25.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:25.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:25.845 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:25.891 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:25.891 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:25.925+0000 7f040e74b800 0 period (f1f5b060-9fe8-4227-8567-18236dc68d0a does not have zone eafd0fcf-abef-4654-893b-ccd3b2eb5bea configured 2024-04-16T19:25:27.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-16T19:25:32.795 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:25:32.847 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":2,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:25:32.848 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T19:25:32.848 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T19:25:32.849 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:25:32.848+0000 7f6aebcf1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 38079) UID: 0 2024-04-16T19:25:32.849 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:25:32.848+0000 7f6af1520880 -1 shutting down 2024-04-16T19:25:38.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T19:25:38.950 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T19:25:38.950 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T19:25:38.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T19:25:38.953 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T19:25:38.953 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T19:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T19:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T19:25:39.209 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:25:39.209+0000 7fe48fccc880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:25:39.977 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:39.997 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:25:39.998 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:25:39.999 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:25:40.000 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T19:25:40.000 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-16T19:25:40.000 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-16T19:25:40.015 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi 2024-04-16T19:25:40.117 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-16T19:25:40.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['account', 'create', '--account-id', 'RGW11111111111111111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:40.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'account', 'create', '--account-id', 'RGW11111111111111111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:40.118 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 account create --account-id RGW11111111111111111 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:40.256 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"RGW11111111111111111","tenant":"","name":"","email":"","quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"max_users":1000,"max_roles":1000,"max_groups":1000,"max_buckets":1000,"max_access_keys":4} 2024-04-16T19:25:40.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--account-id', 'RGW11111111111111111', '--account-root', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'TestUser', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:40.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--account-id', 'RGW11111111111111111', '--account-root', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'TestUser', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:25:40.256 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --account-id RGW11111111111111111 --account-root --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name TestUser --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:25:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:40.371 INFO:teuthology.orchestra.run.smithi026.stdout:{"user_id":"rgw-multisite-test-user","display_name":"TestUser","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"VS71ZU627N05UVO7JCG7","secret_key":"IfQ8nA8s7QqApgncPdzgMpGsRE1CA6MLn4wQ465d","active":true,"create_date":"2024-04-16T19:25:40.364705Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"root","mfa_ids":[],"account_id":"RGW11111111111111111","path":"/","create_date":"2024-04-16T19:25:40.364410Z","tags":[],"group_ids":[]} 2024-04-16T19:25:40.394 INFO:rgw_multi.tests:meta checkpoint 2024-04-16T19:25:40.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:40.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:40.395 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:40.441 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:40.442 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:25:40.524 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:25:40.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:25:40.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:40.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:40.525 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:40.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:40.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:40.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:40.633+0000 7f0f1fe24800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:40.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:40.635+0000 7f0f1fe24800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:40.659 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"building-full-sync-maps","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":0,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":0}},{"key":1,"val":{"state":0,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":0}},{"key":2,"val":{"state":0,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":0}},{"key":3,"val":{"state":0,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":0}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:25:40Z"} 2024-04-16T19:25:40.659 WARNING:rgw_multi.tests:zone test-zone2 has not started sync yet, state=building-full-sync-maps 2024-04-16T19:25:44.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T19:25:45.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.661 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:45.715 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:45.715 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:45.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:45.770+0000 7fa7e040f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:45.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:45.772+0000 7fa7e040f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:45.793 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:45Z"} 2024-04-16T19:25:45.793 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:25:45.793 INFO:rgw_multi.tests:meta checkpoint 2024-04-16T19:25:45.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.794 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:45.848 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:45.848 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:45.904 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:25:45.926 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:25:45.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:25:45.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:45.927 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:45.977 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:45.977 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:46.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:46.038+0000 7fd91a733800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:46.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:46.039+0000 7fd91a733800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:46.061 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:46Z"} 2024-04-16T19:25:46.062 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:25:46.062 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-16T19:25:46.497 DEBUG:boto:Using access key provided by client. 2024-04-16T19:25:46.497 DEBUG:boto:Using secret key provided by client. 2024-04-16T19:25:46.498 DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2024-04-16T19:25:46.500 DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway 2024-04-16T19:25:46.501 DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2024-04-16T19:25:46.502 DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2024-04-16T19:25:46.502 DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2024-04-16T19:25:46.503 DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2024-04-16T19:25:46.504 DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2024-04-16T19:25:46.507 DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2024-04-16T19:25:46.507 DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section 2024-04-16T19:25:46.507 DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2024-04-16T19:25:46.508 DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section 2024-04-16T19:25:46.652 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/endpoints.json 2024-04-16T19:25:46.682 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/sdk-default-configuration.json 2024-04-16T19:25:46.683 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:47.657 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/iam/2010-05-08/service-2.json.gz 2024-04-16T19:25:47.723 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/iam/2010-05-08/endpoint-rule-set-1.json.gz 2024-04-16T19:25:47.740 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/partitions.json 2024-04-16T19:25:47.744 DEBUG:botocore.hooks:Event creating-client-class.iam: calling handler 2024-04-16T19:25:47.745 DEBUG:botocore.regions:Using partition endpoint for iam, test-zonegroup: aws-global 2024-04-16T19:25:47.747 DEBUG:botocore.endpoint:Setting iam timeout as (60, 60) 2024-04-16T19:25:47.758 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/_retry.json 2024-04-16T19:25:47.758 DEBUG:botocore.client:Registering retry handlers for service: iam 2024-04-16T19:25:47.759 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:47.786 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/service-2.json.gz 2024-04-16T19:25:47.809 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz 2024-04-16T19:25:47.815 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler 2024-04-16T19:25:47.815 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler ._handler at 0x7f19438464c0> 2024-04-16T19:25:48.099 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler 2024-04-16T19:25:48.100 DEBUG:botocore.regions:Creating a regex based endpoint for s3, test-zonegroup 2024-04-16T19:25:48.101 DEBUG:botocore.endpoint:Setting s3 timeout as (60, 60) 2024-04-16T19:25:48.103 DEBUG:botocore.client:Registering retry handlers for service: s3 2024-04-16T19:25:48.103 DEBUG:botocore.utils:Registering S3 region redirector handler 2024-04-16T19:25:48.104 DEBUG:botocore.utils:Registering S3Express Identity Resolver 2024-04-16T19:25:48.104 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:48.108 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/sns/2010-03-31/service-2.json.gz 2024-04-16T19:25:48.112 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/sns/2010-03-31/endpoint-rule-set-1.json.gz 2024-04-16T19:25:48.114 DEBUG:botocore.hooks:Event creating-client-class.sns: calling handler 2024-04-16T19:25:48.114 DEBUG:botocore.regions:Creating a regex based endpoint for sns, test-zonegroup 2024-04-16T19:25:48.115 DEBUG:botocore.endpoint:Setting sns timeout as (60, 60) 2024-04-16T19:25:48.116 DEBUG:botocore.client:Registering retry handlers for service: sns 2024-04-16T19:25:48.117 DEBUG:boto:Using access key provided by client. 2024-04-16T19:25:48.117 DEBUG:boto:Using secret key provided by client. 2024-04-16T19:25:48.117 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:48.119 DEBUG:botocore.hooks:Event creating-client-class.iam: calling handler 2024-04-16T19:25:48.119 DEBUG:botocore.regions:Using partition endpoint for iam, test-zonegroup: aws-global 2024-04-16T19:25:48.121 DEBUG:botocore.endpoint:Setting iam timeout as (60, 60) 2024-04-16T19:25:48.122 DEBUG:botocore.client:Registering retry handlers for service: iam 2024-04-16T19:25:48.123 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:48.124 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler 2024-04-16T19:25:48.124 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler ._handler at 0x7f19438464c0> 2024-04-16T19:25:48.124 DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler 2024-04-16T19:25:48.124 DEBUG:botocore.regions:Creating a regex based endpoint for s3, test-zonegroup 2024-04-16T19:25:48.125 DEBUG:botocore.endpoint:Setting s3 timeout as (60, 60) 2024-04-16T19:25:48.127 DEBUG:botocore.client:Registering retry handlers for service: s3 2024-04-16T19:25:48.127 DEBUG:botocore.utils:Registering S3 region redirector handler 2024-04-16T19:25:48.127 DEBUG:botocore.utils:Registering S3Express Identity Resolver 2024-04-16T19:25:48.128 DEBUG:botocore.hooks:Event choose-service-name: calling handler 2024-04-16T19:25:48.128 DEBUG:botocore.hooks:Event creating-client-class.sns: calling handler 2024-04-16T19:25:48.129 DEBUG:botocore.regions:Creating a regex based endpoint for sns, test-zonegroup 2024-04-16T19:25:48.130 DEBUG:botocore.endpoint:Setting sns timeout as (60, 60) 2024-04-16T19:25:48.131 DEBUG:botocore.client:Registering retry handlers for service: sns 2024-04-16T19:25:48.131 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-1 2024-04-16T19:25:48.131 DEBUG:boto:path=/yrjefh-1/ 2024-04-16T19:25:48.131 DEBUG:boto:auth_path=/yrjefh-1/ 2024-04-16T19:25:48.131 DEBUG:boto:Method: PUT 2024-04-16T19:25:48.131 DEBUG:boto:Path: /yrjefh-1/ 2024-04-16T19:25:48.131 DEBUG:boto:Data: 2024-04-16T19:25:48.132 DEBUG:boto:Headers: {} 2024-04-16T19:25:48.132 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:48.132 DEBUG:boto:Port: 8000 2024-04-16T19:25:48.132 DEBUG:boto:Params: {} 2024-04-16T19:25:48.132 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:25:48.132 DEBUG:boto:Token: None 2024-04-16T19:25:48.132 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:48 GMT /yrjefh-1/ 2024-04-16T19:25:48.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:neV8y7I+6444nassS7JLLZ/EQYg= 2024-04-16T19:25:48.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:48 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:neV8y7I+6444nassS7JLLZ/EQYg=', 'Content-Length': '0'} 2024-04-16T19:25:50.538 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed0698c3e41839eb-00661ed0bc-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:50.538 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-2 2024-04-16T19:25:50.538 DEBUG:boto:path=/yrjefh-2/ 2024-04-16T19:25:50.538 DEBUG:boto:auth_path=/yrjefh-2/ 2024-04-16T19:25:50.539 DEBUG:boto:Method: PUT 2024-04-16T19:25:50.539 DEBUG:boto:Path: /yrjefh-2/ 2024-04-16T19:25:50.539 DEBUG:boto:Data: 2024-04-16T19:25:50.539 DEBUG:boto:Headers: {} 2024-04-16T19:25:50.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:50.539 DEBUG:boto:Port: 8000 2024-04-16T19:25:50.539 DEBUG:boto:Params: {} 2024-04-16T19:25:50.539 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:25:50.539 DEBUG:boto:Token: None 2024-04-16T19:25:50.539 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:50 GMT /yrjefh-2/ 2024-04-16T19:25:50.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j2sXQNftIVBn9s6u8Hx7kuMrqyI= 2024-04-16T19:25:50.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j2sXQNftIVBn9s6u8Hx7kuMrqyI=', 'Content-Length': '0'} 2024-04-16T19:25:52.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021dfe8245e842c72-00661ed0be-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:52.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:52.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:52.810 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:25:52.939 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295550.557816_65.1","last_update":"2024-04-16T19:25:50.557816Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:25:52.940 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295550.557816_65.1', 3: ''} 2024-04-16T19:25:52.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:25:52.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:52.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:52.940 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:52.995 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:52.996 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:53.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:53.048+0000 7f54f5f31800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:53.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:53.050+0000 7f54f5f31800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:53.071 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:53Z"} 2024-04-16T19:25:53.071 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:53Z"} 2024-04-16T19:25:53.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:25:53.072 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295550.557816_65.1', 3: ''} 2024-04-16T19:25:53.072 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '', 3: ''} 2024-04-16T19:25:53.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713295550.557816_65.1 target= 2024-04-16T19:25:58.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.073 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:58.128 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:58.128 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:58.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.192+0000 7fc937e0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.193+0000 7fc937e0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.226 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295550.557816_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:50.557816Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.226 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295550.557816_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:50.557816Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295550.557816_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:25:50.557816Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:25:58.226 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295550.557816_65.1', 3: ''} 2024-04-16T19:25:58.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295550.557816_65.1', 3: ''} 2024-04-16T19:25:58.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:25:58.227 DEBUG:boto:path=/yrjefh-1/ 2024-04-16T19:25:58.227 DEBUG:boto:auth_path=/yrjefh-1/ 2024-04-16T19:25:58.227 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.227 DEBUG:boto:Path: /yrjefh-1/ 2024-04-16T19:25:58.227 DEBUG:boto:Data: 2024-04-16T19:25:58.227 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.227 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.227 DEBUG:boto:Params: {} 2024-04-16T19:25:58.227 DEBUG:boto:Token: None 2024-04-16T19:25:58.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-1/ 2024-04-16T19:25:58.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LXXOYMAvhBA1DhlF6Ry0gqtNctg= 2024-04-16T19:25:58.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LXXOYMAvhBA1DhlF6Ry0gqtNctg=', 'Content-Length': '0'} 2024-04-16T19:25:58.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000074f89d450859c6d-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.231 DEBUG:boto:path=/yrjefh-2/ 2024-04-16T19:25:58.231 DEBUG:boto:auth_path=/yrjefh-2/ 2024-04-16T19:25:58.231 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.231 DEBUG:boto:Path: /yrjefh-2/ 2024-04-16T19:25:58.231 DEBUG:boto:Data: 2024-04-16T19:25:58.231 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.231 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.231 DEBUG:boto:Params: {} 2024-04-16T19:25:58.232 DEBUG:boto:Token: None 2024-04-16T19:25:58.232 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-2/ 2024-04-16T19:25:58.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JGaIpxU1rubVP7EHi8JlgfjXUa0= 2024-04-16T19:25:58.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JGaIpxU1rubVP7EHi8JlgfjXUa0=', 'Content-Length': '0'} 2024-04-16T19:25:58.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027319f0bb8bb8752-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.235 DEBUG:boto:path=/yrjefh-1/ 2024-04-16T19:25:58.235 DEBUG:boto:auth_path=/yrjefh-1/ 2024-04-16T19:25:58.235 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.235 DEBUG:boto:Path: /yrjefh-1/ 2024-04-16T19:25:58.235 DEBUG:boto:Data: 2024-04-16T19:25:58.235 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.235 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.235 DEBUG:boto:Params: {} 2024-04-16T19:25:58.235 DEBUG:boto:Token: None 2024-04-16T19:25:58.235 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-1/ 2024-04-16T19:25:58.236 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LXXOYMAvhBA1DhlF6Ry0gqtNctg= 2024-04-16T19:25:58.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LXXOYMAvhBA1DhlF6Ry0gqtNctg=', 'Content-Length': '0'} 2024-04-16T19:25:58.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a664a63b9a2b3790-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.238 DEBUG:boto:path=/yrjefh-2/ 2024-04-16T19:25:58.238 DEBUG:boto:auth_path=/yrjefh-2/ 2024-04-16T19:25:58.238 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.238 DEBUG:boto:Path: /yrjefh-2/ 2024-04-16T19:25:58.239 DEBUG:boto:Data: 2024-04-16T19:25:58.239 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.239 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.239 DEBUG:boto:Params: {} 2024-04-16T19:25:58.239 DEBUG:boto:Token: None 2024-04-16T19:25:58.239 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-2/ 2024-04-16T19:25:58.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JGaIpxU1rubVP7EHi8JlgfjXUa0= 2024-04-16T19:25:58.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JGaIpxU1rubVP7EHi8JlgfjXUa0=', 'Content-Length': '0'} 2024-04-16T19:25:58.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbe6db50f2f8a657-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-16T19:25:58.243 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-3 2024-04-16T19:25:58.243 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.243 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.243 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.243 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.243 DEBUG:boto:Data: 2024-04-16T19:25:58.243 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.243 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.243 DEBUG:boto:Params: {} 2024-04-16T19:25:58.243 DEBUG:boto:Token: None 2024-04-16T19:25:58.244 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E= 2024-04-16T19:25:58.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E=', 'Content-Length': '0'} 2024-04-16T19:25:58.259 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e30347494c70b117-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.259 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-4 2024-04-16T19:25:58.259 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.260 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.260 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.260 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.260 DEBUG:boto:Data: 2024-04-16T19:25:58.260 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.260 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.260 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.260 DEBUG:boto:Params: {} 2024-04-16T19:25:58.260 DEBUG:boto:Token: None 2024-04-16T19:25:58.260 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg= 2024-04-16T19:25:58.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg=', 'Content-Length': '0'} 2024-04-16T19:25:58.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe60cb501008cdd5-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.293 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:58.346 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:58.347 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:58.400 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:25:58.423 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295558.276248_80.1","last_update":"2024-04-16T19:25:58.276248Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:25:58.423 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.276248_80.1', 3: ''} 2024-04-16T19:25:58.423 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:25:58.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.424 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:58.472 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:58.472 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:58.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.518+0000 7f658f22e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.519+0000 7f658f22e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.540 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.276248_80.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.276248Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.541 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.276248_80.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.276248Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295558.276248_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:25:58.276248Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:25:58.541 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.276248_80.1', 3: ''} 2024-04-16T19:25:58.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.276248_80.1', 3: ''} 2024-04-16T19:25:58.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:25:58.542 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.542 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.542 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.542 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.542 DEBUG:boto:Data: 2024-04-16T19:25:58.542 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.542 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.542 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.542 DEBUG:boto:Params: {} 2024-04-16T19:25:58.542 DEBUG:boto:Token: None 2024-04-16T19:25:58.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE= 2024-04-16T19:25:58.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE=', 'Content-Length': '0'} 2024-04-16T19:25:58.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223fbf4d055259af-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.545 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.545 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.546 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.546 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.546 DEBUG:boto:Data: 2024-04-16T19:25:58.546 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.546 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.546 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.546 DEBUG:boto:Params: {} 2024-04-16T19:25:58.546 DEBUG:boto:Token: None 2024-04-16T19:25:58.546 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg= 2024-04-16T19:25:58.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg=', 'Content-Length': '0'} 2024-04-16T19:25:58.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006972d0973bef450d-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.548 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.548 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.548 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.549 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.549 DEBUG:boto:Data: 2024-04-16T19:25:58.549 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.549 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.549 DEBUG:boto:Params: {} 2024-04-16T19:25:58.549 DEBUG:boto:Token: None 2024-04-16T19:25:58.549 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE= 2024-04-16T19:25:58.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE=', 'Content-Length': '0'} 2024-04-16T19:25:58.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d02d54775facdf-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.552 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.552 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.552 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.552 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.552 DEBUG:boto:Data: 2024-04-16T19:25:58.552 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.552 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.552 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.552 DEBUG:boto:Params: {} 2024-04-16T19:25:58.552 DEBUG:boto:Token: None 2024-04-16T19:25:58.553 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg= 2024-04-16T19:25:58.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg=', 'Content-Length': '0'} 2024-04-16T19:25:58.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008477221240e1629e-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.555 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.555 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.555 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.555 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.555 DEBUG:boto:Data: 2024-04-16T19:25:58.556 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.556 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.556 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.556 DEBUG:boto:Params: {} 2024-04-16T19:25:58.556 DEBUG:boto:Token: None 2024-04-16T19:25:58.556 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E= 2024-04-16T19:25:58.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E=', 'Content-Length': '0'} 2024-04-16T19:25:58.581 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b56dd1f0a42f884e-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.581 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.581 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.581 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.581 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.581 DEBUG:boto:Data: 2024-04-16T19:25:58.581 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.582 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.582 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.582 DEBUG:boto:Params: {} 2024-04-16T19:25:58.582 DEBUG:boto:Token: None 2024-04-16T19:25:58.582 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.582 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg= 2024-04-16T19:25:58.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg=', 'Content-Length': '0'} 2024-04-16T19:25:58.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c9bc6c54bbe9ed57-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.598 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.598 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.598 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.599 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.599 DEBUG:boto:Data: 2024-04-16T19:25:58.599 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.599 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.599 DEBUG:boto:Params: {} 2024-04-16T19:25:58.599 DEBUG:boto:Token: None 2024-04-16T19:25:58.599 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E= 2024-04-16T19:25:58.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f2/7EKTJ/zlKD4Y9GlH+jRDpf6E=', 'Content-Length': '0'} 2024-04-16T19:25:58.629 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d94b1886bd459b66-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.630 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.630 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.630 DEBUG:boto:Method: PUT 2024-04-16T19:25:58.630 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.630 DEBUG:boto:Data: 2024-04-16T19:25:58.630 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.630 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.630 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.630 DEBUG:boto:Params: {} 2024-04-16T19:25:58.631 DEBUG:boto:Token: None 2024-04-16T19:25:58.631 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg= 2024-04-16T19:25:58.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Df4ve2y6AbJ4Q/NXNvUXaAhFMVg=', 'Content-Length': '0'} 2024-04-16T19:25:58.697 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9e24b928819a24c-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.697 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.697 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.698 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.698 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.698 DEBUG:boto:Data: 2024-04-16T19:25:58.698 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.698 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.698 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.698 DEBUG:boto:Params: {} 2024-04-16T19:25:58.698 DEBUG:boto:Token: None 2024-04-16T19:25:58.698 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE= 2024-04-16T19:25:58.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE=', 'Content-Length': '0'} 2024-04-16T19:25:58.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090565ee3eb887980-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.701 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.701 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.701 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.701 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.702 DEBUG:boto:Data: 2024-04-16T19:25:58.702 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.702 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:25:58.702 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.702 DEBUG:boto:Params: {} 2024-04-16T19:25:58.702 DEBUG:boto:Token: None 2024-04-16T19:25:58.702 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg= 2024-04-16T19:25:58.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg=', 'Content-Length': '0'} 2024-04-16T19:25:58.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000485c463222993145-00661ed0c6-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.705 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:25:58.705 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:25:58.705 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.705 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:25:58.705 DEBUG:boto:Data: 2024-04-16T19:25:58.705 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.705 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.705 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.705 DEBUG:boto:Params: {} 2024-04-16T19:25:58.705 DEBUG:boto:Token: None 2024-04-16T19:25:58.706 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-3/ 2024-04-16T19:25:58.706 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE= 2024-04-16T19:25:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vKWwBLWue/BYSn875YpidzxpooE=', 'Content-Length': '0'} 2024-04-16T19:25:58.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073e18a3ee8f1f251-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.708 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:25:58.709 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:25:58.709 DEBUG:boto:Method: HEAD 2024-04-16T19:25:58.709 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:25:58.709 DEBUG:boto:Data: 2024-04-16T19:25:58.709 DEBUG:boto:Headers: {} 2024-04-16T19:25:58.709 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:25:58.709 DEBUG:boto:Port: 8000 2024-04-16T19:25:58.709 DEBUG:boto:Params: {} 2024-04-16T19:25:58.709 DEBUG:boto:Token: None 2024-04-16T19:25:58.709 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:25:58 GMT /yrjefh-4/ 2024-04-16T19:25:58.710 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg= 2024-04-16T19:25:58.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:25:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ri6F8ShXquq5yTVj19iZb+DlUYg=', 'Content-Length': '0'} 2024-04-16T19:25:58.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091abdfac0f260cae-00661ed0c6-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:25:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:25:58.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.713 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:58.760 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:58.760 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:58.809 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:25:58.830 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295558.642797_98.1","last_update":"2024-04-16T19:25:58.642797Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:25:58.831 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.642797_98.1', 3: ''} 2024-04-16T19:25:58.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:25:58.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:25:58.831 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:25:58.879 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:25:58.879 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:25:58.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.926+0000 7f6365644800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:25:58.927+0000 7f6365644800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:25:58.949 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.639802_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.639802Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.949 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.639802_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.639802Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:25:58Z"} 2024-04-16T19:25:58.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295558.639802_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:25:58.639802Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:25:58.950 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.642797_98.1', 3: ''} 2024-04-16T19:25:58.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.639802_96.1', 3: ''} 2024-04-16T19:25:58.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713295558.642797_98.1 target=1_1713295558.639802_96.1 2024-04-16T19:26:03.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:03.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:03.951 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:04.006 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:04.007 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:04.072+0000 7fa61bfa2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:04.074+0000 7fa61bfa2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:04.106 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.642797_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.642797Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:04Z"} 2024-04-16T19:26:04.107 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295558.642797_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:25:58.642797Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:04Z"} 2024-04-16T19:26:04.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295558.642797_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:25:58.642797Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:04.107 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.642797_98.1', 3: ''} 2024-04-16T19:26:04.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295558.642797_98.1', 3: ''} 2024-04-16T19:26:04.107 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:04.107 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:26:04.107 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:26:04.108 DEBUG:boto:Method: HEAD 2024-04-16T19:26:04.108 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:26:04.108 DEBUG:boto:Data: 2024-04-16T19:26:04.108 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.108 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:04.108 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.108 DEBUG:boto:Params: {} 2024-04-16T19:26:04.109 DEBUG:boto:Token: None 2024-04-16T19:26:04.109 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-3/ 2024-04-16T19:26:04.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WPQuOqIB0TAadbbWGzQygWK5lGk= 2024-04-16T19:26:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WPQuOqIB0TAadbbWGzQygWK5lGk=', 'Content-Length': '0'} 2024-04-16T19:26:04.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008763efe4c6a79bf7-00661ed0cc-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.112 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:26:04.112 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:26:04.112 DEBUG:boto:Method: HEAD 2024-04-16T19:26:04.112 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:26:04.112 DEBUG:boto:Data: 2024-04-16T19:26:04.112 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.112 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:04.112 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.112 DEBUG:boto:Params: {} 2024-04-16T19:26:04.113 DEBUG:boto:Token: None 2024-04-16T19:26:04.113 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-4/ 2024-04-16T19:26:04.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ew6BW6aeC+VYgMd5FvpAD/KMLyA= 2024-04-16T19:26:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ew6BW6aeC+VYgMd5FvpAD/KMLyA=', 'Content-Length': '0'} 2024-04-16T19:26:04.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca67805cf89e11fc-00661ed0cc-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.115 DEBUG:boto:path=/yrjefh-3/ 2024-04-16T19:26:04.115 DEBUG:boto:auth_path=/yrjefh-3/ 2024-04-16T19:26:04.115 DEBUG:boto:Method: HEAD 2024-04-16T19:26:04.115 DEBUG:boto:Path: /yrjefh-3/ 2024-04-16T19:26:04.116 DEBUG:boto:Data: 2024-04-16T19:26:04.116 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:04.116 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.116 DEBUG:boto:Params: {} 2024-04-16T19:26:04.116 DEBUG:boto:Token: None 2024-04-16T19:26:04.116 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-3/ 2024-04-16T19:26:04.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WPQuOqIB0TAadbbWGzQygWK5lGk= 2024-04-16T19:26:04.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WPQuOqIB0TAadbbWGzQygWK5lGk=', 'Content-Length': '0'} 2024-04-16T19:26:04.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a02f0cff6142f034-00661ed0cc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.119 DEBUG:boto:path=/yrjefh-4/ 2024-04-16T19:26:04.119 DEBUG:boto:auth_path=/yrjefh-4/ 2024-04-16T19:26:04.119 DEBUG:boto:Method: HEAD 2024-04-16T19:26:04.119 DEBUG:boto:Path: /yrjefh-4/ 2024-04-16T19:26:04.119 DEBUG:boto:Data: 2024-04-16T19:26:04.119 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:04.119 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.119 DEBUG:boto:Params: {} 2024-04-16T19:26:04.120 DEBUG:boto:Token: None 2024-04-16T19:26:04.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-4/ 2024-04-16T19:26:04.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ew6BW6aeC+VYgMd5FvpAD/KMLyA= 2024-04-16T19:26:04.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ew6BW6aeC+VYgMd5FvpAD/KMLyA=', 'Content-Length': '0'} 2024-04-16T19:26:04.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004537e0ef9a42e153-00661ed0cc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-16T19:26:04.123 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-5 2024-04-16T19:26:04.123 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:04.123 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:04.123 DEBUG:boto:Method: PUT 2024-04-16T19:26:04.123 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:04.123 DEBUG:boto:Data: 2024-04-16T19:26:04.123 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.124 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:04.124 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.124 DEBUG:boto:Params: {} 2024-04-16T19:26:04.124 DEBUG:boto:Token: None 2024-04-16T19:26:04.124 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-5/ 2024-04-16T19:26:04.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bE2lpaFj283IrwNvqztxu8Z91nw= 2024-04-16T19:26:04.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bE2lpaFj283IrwNvqztxu8Z91nw=', 'Content-Length': '0'} 2024-04-16T19:26:04.141 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3bd9e48fcd2e840-00661ed0cc-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.141 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-6 2024-04-16T19:26:04.141 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:04.141 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:04.142 DEBUG:boto:Method: PUT 2024-04-16T19:26:04.142 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:04.142 DEBUG:boto:Data: 2024-04-16T19:26:04.142 DEBUG:boto:Headers: {} 2024-04-16T19:26:04.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:04.142 DEBUG:boto:Port: 8000 2024-04-16T19:26:04.142 DEBUG:boto:Params: {} 2024-04-16T19:26:04.142 DEBUG:boto:Token: None 2024-04-16T19:26:04.142 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:04 GMT /yrjefh-6/ 2024-04-16T19:26:04.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/o/+ufoyBlvwD2ScbxF/4Z2Ig5o= 2024-04-16T19:26:04.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/o/+ufoyBlvwD2ScbxF/4Z2Ig5o=', 'Content-Length': '0'} 2024-04-16T19:26:04.174 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085631b28458da5ec-00661ed0cc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:04.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:04.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:04.175 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:04.227 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:04.227 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:26:04.302 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295564.159430_108.1","last_update":"2024-04-16T19:26:04.159430Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:26:04.302 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295564.159430_108.1', 3: ''} 2024-04-16T19:26:04.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:26:04.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:04.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:04.303 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:04.351 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:04.352 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:04.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:04.412+0000 7f92ead33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:04.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:04.413+0000 7f92ead33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:04.434 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295564.130351_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:04.130351Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:04Z"} 2024-04-16T19:26:04.434 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295564.130351_99.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:04.130351Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:04Z"} 2024-04-16T19:26:04.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295564.130351_99.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:04.130351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:04.434 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295564.159430_108.1', 3: ''} 2024-04-16T19:26:04.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295564.130351_99.1', 3: ''} 2024-04-16T19:26:04.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713295564.159430_108.1 target=1_1713295564.130351_99.1 2024-04-16T19:26:09.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.436 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:09.491 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:09.491 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:09.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:09.548+0000 7f0804b2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:09.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:09.549+0000 7f0804b2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:09.570 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295564.159430_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:04.159430Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:09Z"} 2024-04-16T19:26:09.570 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295564.159430_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:04.159430Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:09Z"} 2024-04-16T19:26:09.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295564.159430_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:04.159430Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:09.571 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295564.159430_108.1', 3: ''} 2024-04-16T19:26:09.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295564.159430_108.1', 3: ''} 2024-04-16T19:26:09.571 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:09.571 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:09.571 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:09.571 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.571 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:09.572 DEBUG:boto:Data: 2024-04-16T19:26:09.572 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.572 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.572 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.572 DEBUG:boto:Params: {} 2024-04-16T19:26:09.572 DEBUG:boto:Token: None 2024-04-16T19:26:09.572 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-5/ 2024-04-16T19:26:09.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c= 2024-04-16T19:26:09.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c=', 'Content-Length': '0'} 2024-04-16T19:26:09.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000647bea1bf49faf68-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.575 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:09.575 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:09.575 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.575 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:09.575 DEBUG:boto:Data: 2024-04-16T19:26:09.575 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.576 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.576 DEBUG:boto:Params: {} 2024-04-16T19:26:09.576 DEBUG:boto:Token: None 2024-04-16T19:26:09.576 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-6/ 2024-04-16T19:26:09.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo= 2024-04-16T19:26:09.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo=', 'Content-Length': '0'} 2024-04-16T19:26:09.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ad7ee9ec188c155-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.578 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:09.579 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:09.579 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.579 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:09.579 DEBUG:boto:Data: 2024-04-16T19:26:09.579 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.579 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.579 DEBUG:boto:Params: {} 2024-04-16T19:26:09.579 DEBUG:boto:Token: None 2024-04-16T19:26:09.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-5/ 2024-04-16T19:26:09.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c= 2024-04-16T19:26:09.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c=', 'Content-Length': '0'} 2024-04-16T19:26:09.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c0b3e423c50a50-00661ed0d1-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.582 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:09.582 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:09.582 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.582 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:09.582 DEBUG:boto:Data: 2024-04-16T19:26:09.582 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.582 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.582 DEBUG:boto:Params: {} 2024-04-16T19:26:09.583 DEBUG:boto:Token: None 2024-04-16T19:26:09.583 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-6/ 2024-04-16T19:26:09.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo= 2024-04-16T19:26:09.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo=', 'Content-Length': '0'} 2024-04-16T19:26:09.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e549cccf0b26de29-00661ed0d1-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.585 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:09.585 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:09.585 DEBUG:boto:Method: DELETE 2024-04-16T19:26:09.585 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:09.585 DEBUG:boto:Data: 2024-04-16T19:26:09.586 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.586 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.586 DEBUG:boto:Params: {} 2024-04-16T19:26:09.586 DEBUG:boto:Token: None 2024-04-16T19:26:09.586 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-5/ 2024-04-16T19:26:09.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VJQ1k5OGrberuE8kbe8NkNJVu2g= 2024-04-16T19:26:09.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VJQ1k5OGrberuE8kbe8NkNJVu2g=', 'Content-Length': '0'} 2024-04-16T19:26:09.615 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ce6f9e06ca371df-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.615 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:09.615 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:09.616 DEBUG:boto:Method: DELETE 2024-04-16T19:26:09.616 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:09.616 DEBUG:boto:Data: 2024-04-16T19:26:09.616 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.616 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.616 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.616 DEBUG:boto:Params: {} 2024-04-16T19:26:09.616 DEBUG:boto:Token: None 2024-04-16T19:26:09.616 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-6/ 2024-04-16T19:26:09.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vJcohSggoRmxoptJZaXMv6/ZgOI= 2024-04-16T19:26:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vJcohSggoRmxoptJZaXMv6/ZgOI=', 'Content-Length': '0'} 2024-04-16T19:26:09.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ce91d89a5829462-00661ed0d1-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.680 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:26:09.802 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295569.649417_114.1","last_update":"2024-04-16T19:26:09.649417Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:26:09.803 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.649417_114.1', 3: ''} 2024-04-16T19:26:09.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:26:09.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:09.803 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:09.851 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:09.851 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:09.897+0000 7ff6b8e2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:09.898+0000 7ff6b8e2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:09.919 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.649417_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.649417Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:09Z"} 2024-04-16T19:26:09.919 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.649417_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.649417Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:09Z"} 2024-04-16T19:26:09.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295569.649417_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:09.649417Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:09.919 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.649417_114.1', 3: ''} 2024-04-16T19:26:09.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.649417_114.1', 3: ''} 2024-04-16T19:26:09.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:09.920 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:09.920 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:09.920 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.920 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:09.920 DEBUG:boto:Data: 2024-04-16T19:26:09.920 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.920 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.920 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.920 DEBUG:boto:Params: {} 2024-04-16T19:26:09.921 DEBUG:boto:Token: None 2024-04-16T19:26:09.921 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-5/ 2024-04-16T19:26:09.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c= 2024-04-16T19:26:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c=', 'Content-Length': '0'} 2024-04-16T19:26:09.922 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000ce388eb18d6177d3-00661ed0d1-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.922 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:09.923 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:09.923 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.923 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:09.923 DEBUG:boto:Data: 2024-04-16T19:26:09.923 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.923 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.923 DEBUG:boto:Params: {} 2024-04-16T19:26:09.923 DEBUG:boto:Token: None 2024-04-16T19:26:09.923 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-6/ 2024-04-16T19:26:09.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo= 2024-04-16T19:26:09.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo=', 'Content-Length': '0'} 2024-04-16T19:26:09.924 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000fc3b888de019f7e8-00661ed0d1-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.924 DEBUG:boto:path=/yrjefh-5/ 2024-04-16T19:26:09.925 DEBUG:boto:auth_path=/yrjefh-5/ 2024-04-16T19:26:09.925 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.925 DEBUG:boto:Path: /yrjefh-5/ 2024-04-16T19:26:09.925 DEBUG:boto:Data: 2024-04-16T19:26:09.925 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.925 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.925 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.925 DEBUG:boto:Params: {} 2024-04-16T19:26:09.925 DEBUG:boto:Token: None 2024-04-16T19:26:09.925 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-5/ 2024-04-16T19:26:09.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c= 2024-04-16T19:26:09.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zc8wpFWPbZi7sshJtudVdeO9B8c=', 'Content-Length': '0'} 2024-04-16T19:26:09.927 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000009fd720e84a08ec34-00661ed0d1-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.927 DEBUG:boto:path=/yrjefh-6/ 2024-04-16T19:26:09.928 DEBUG:boto:auth_path=/yrjefh-6/ 2024-04-16T19:26:09.928 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.928 DEBUG:boto:Path: /yrjefh-6/ 2024-04-16T19:26:09.928 DEBUG:boto:Data: 2024-04-16T19:26:09.928 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.928 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.928 DEBUG:boto:Params: {} 2024-04-16T19:26:09.928 DEBUG:boto:Token: None 2024-04-16T19:26:09.928 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-6/ 2024-04-16T19:26:09.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo= 2024-04-16T19:26:09.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbLtsbSe//kOMvFnjh2F17Ztneo=', 'Content-Length': '0'} 2024-04-16T19:26:09.929 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000dd3cc24d8b014652-00661ed0d1-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-16T19:26:09.930 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-7 2024-04-16T19:26:09.931 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:09.931 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:09.931 DEBUG:boto:Method: PUT 2024-04-16T19:26:09.931 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:09.931 DEBUG:boto:Data: 2024-04-16T19:26:09.931 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.931 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.931 DEBUG:boto:Params: {} 2024-04-16T19:26:09.931 DEBUG:boto:Token: None 2024-04-16T19:26:09.932 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-7/ 2024-04-16T19:26:09.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bMBzPVKIxPKrzB7cEpIICHxKr0k= 2024-04-16T19:26:09.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bMBzPVKIxPKrzB7cEpIICHxKr0k=', 'Content-Length': '0'} 2024-04-16T19:26:09.950 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b3f979b2af82c1d-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.950 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-8 2024-04-16T19:26:09.950 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:09.950 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:09.950 DEBUG:boto:Method: PUT 2024-04-16T19:26:09.950 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:09.950 DEBUG:boto:Data: 2024-04-16T19:26:09.950 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.950 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:09.951 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.951 DEBUG:boto:Params: {} 2024-04-16T19:26:09.951 DEBUG:boto:Token: None 2024-04-16T19:26:09.951 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-8/ 2024-04-16T19:26:09.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n6dJayCig/pJ7u9RM2/Y4XrLLr4= 2024-04-16T19:26:09.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n6dJayCig/pJ7u9RM2/Y4XrLLr4=', 'Content-Length': '0'} 2024-04-16T19:26:09.976 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007869a139dc0b0c85-00661ed0d1-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.976 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:09.976 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:09.976 DEBUG:boto:Method: HEAD 2024-04-16T19:26:09.977 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:09.977 DEBUG:boto:Data: 2024-04-16T19:26:09.977 DEBUG:boto:Headers: {} 2024-04-16T19:26:09.977 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.977 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.977 DEBUG:boto:Params: {} 2024-04-16T19:26:09.977 DEBUG:boto:Token: None 2024-04-16T19:26:09.977 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-7/ 2024-04-16T19:26:09.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dT1VSSOQuNsSJfwpbvpRt+Ow/eQ= 2024-04-16T19:26:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dT1VSSOQuNsSJfwpbvpRt+Ow/eQ=', 'Content-Length': '0'} 2024-04-16T19:26:09.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe951ce9e4e9f0e9-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:09.980 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:09.980 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:09.980 DEBUG:boto:Method: PUT 2024-04-16T19:26:09.980 DEBUG:boto:Path: /yrjefh-7/myobj 2024-04-16T19:26:09.980 DEBUG:boto:Data: 2024-04-16T19:26:09.980 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:09.980 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:09.981 DEBUG:boto:Port: 8000 2024-04-16T19:26:09.981 DEBUG:boto:Params: {} 2024-04-16T19:26:09.981 DEBUG:boto:Token: None 2024-04-16T19:26:09.981 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:09 GMT /yrjefh-7/myobj 2024-04-16T19:26:09.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6MfdtwyceB77ZoNdwEHfgebOsRw= 2024-04-16T19:26:09.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6MfdtwyceB77ZoNdwEHfgebOsRw='} 2024-04-16T19:26:10.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d761b0ea3fc890b3-00661ed0d1-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.023 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.023 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.023 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.023 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.023 DEBUG:boto:Data: 2024-04-16T19:26:10.023 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.024 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.024 DEBUG:boto:Params: {} 2024-04-16T19:26:10.024 DEBUG:boto:Token: None 2024-04-16T19:26:10.024 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c92571ab4e00bb3d-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.026 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:10.027 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:10.027 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.027 DEBUG:boto:Path: /yrjefh-7/_myobj 2024-04-16T19:26:10.027 DEBUG:boto:Data: 2024-04-16T19:26:10.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.027 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.027 DEBUG:boto:Params: {} 2024-04-16T19:26:10.027 DEBUG:boto:Token: None 2024-04-16T19:26:10.027 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/_myobj 2024-04-16T19:26:10.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bFB5tjnosMV8+qXgkMMlXeVlYoU= 2024-04-16T19:26:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bFB5tjnosMV8+qXgkMMlXeVlYoU='} 2024-04-16T19:26:10.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de3abf4e4ce3eca1-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.039 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.039 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.039 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.039 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.039 DEBUG:boto:Data: 2024-04-16T19:26:10.039 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.039 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.039 DEBUG:boto:Params: {} 2024-04-16T19:26:10.040 DEBUG:boto:Token: None 2024-04-16T19:26:10.040 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce05af4076560df5-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.042 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:10.042 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:10.042 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.042 DEBUG:boto:Path: /yrjefh-7/%3A 2024-04-16T19:26:10.042 DEBUG:boto:Data: 2024-04-16T19:26:10.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.042 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.043 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.043 DEBUG:boto:Params: {} 2024-04-16T19:26:10.043 DEBUG:boto:Token: None 2024-04-16T19:26:10.043 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/%3A 2024-04-16T19:26:10.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:78aZAAvXX+24+9iDYqhUYv9X9RA= 2024-04-16T19:26:10.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:78aZAAvXX+24+9iDYqhUYv9X9RA='} 2024-04-16T19:26:10.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d74471e0c0127b56-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.051 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.051 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.051 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.052 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.052 DEBUG:boto:Data: 2024-04-16T19:26:10.052 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.052 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.052 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.052 DEBUG:boto:Params: {} 2024-04-16T19:26:10.052 DEBUG:boto:Token: None 2024-04-16T19:26:10.052 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea6024b4f296fddd-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.055 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:10.055 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:10.055 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.055 DEBUG:boto:Path: /yrjefh-7/%26 2024-04-16T19:26:10.055 DEBUG:boto:Data: 2024-04-16T19:26:10.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.056 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.056 DEBUG:boto:Params: {} 2024-04-16T19:26:10.056 DEBUG:boto:Token: None 2024-04-16T19:26:10.056 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/%26 2024-04-16T19:26:10.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8zN3uxdwN5FsBJYvFGE2gxCv8Lg= 2024-04-16T19:26:10.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8zN3uxdwN5FsBJYvFGE2gxCv8Lg='} 2024-04-16T19:26:10.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f72f0bac86ba5bd-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.099 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.099 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.099 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.099 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.099 DEBUG:boto:Data: 2024-04-16T19:26:10.099 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.099 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.099 DEBUG:boto:Params: {} 2024-04-16T19:26:10.099 DEBUG:boto:Token: None 2024-04-16T19:26:10.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1f258e7738ff3ba-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.102 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:10.102 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:10.102 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.102 DEBUG:boto:Path: /yrjefh-7/. 2024-04-16T19:26:10.103 DEBUG:boto:Data: 2024-04-16T19:26:10.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.103 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.103 DEBUG:boto:Params: {} 2024-04-16T19:26:10.103 DEBUG:boto:Token: None 2024-04-16T19:26:10.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/. 2024-04-16T19:26:10.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cItodKvzGvpUk8o3ZZGmRc0DXhM= 2024-04-16T19:26:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cItodKvzGvpUk8o3ZZGmRc0DXhM='} 2024-04-16T19:26:10.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009956f72c53a1a131-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.111 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.111 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.111 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.111 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.111 DEBUG:boto:Data: 2024-04-16T19:26:10.111 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.112 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.112 DEBUG:boto:Params: {} 2024-04-16T19:26:10.112 DEBUG:boto:Token: None 2024-04-16T19:26:10.112 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04ac98f9ac66a2e-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.114 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:10.114 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:10.115 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.115 DEBUG:boto:Path: /yrjefh-7/.. 2024-04-16T19:26:10.115 DEBUG:boto:Data: 2024-04-16T19:26:10.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.115 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.115 DEBUG:boto:Params: {} 2024-04-16T19:26:10.115 DEBUG:boto:Token: None 2024-04-16T19:26:10.115 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/.. 2024-04-16T19:26:10.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uHWiNgm6aR7AjKD8A6Iu/Oqwl7c= 2024-04-16T19:26:10.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uHWiNgm6aR7AjKD8A6Iu/Oqwl7c='} 2024-04-16T19:26:10.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c4c82eb151d1bb9-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.121 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.121 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.121 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.121 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.121 DEBUG:boto:Data: 2024-04-16T19:26:10.121 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.122 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.122 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.122 DEBUG:boto:Params: {} 2024-04-16T19:26:10.122 DEBUG:boto:Token: None 2024-04-16T19:26:10.122 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.122 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067ac284c90647e32-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.124 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:10.124 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:10.124 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.124 DEBUG:boto:Path: /yrjefh-7/... 2024-04-16T19:26:10.124 DEBUG:boto:Data: 2024-04-16T19:26:10.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.125 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.125 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.125 DEBUG:boto:Params: {} 2024-04-16T19:26:10.125 DEBUG:boto:Token: None 2024-04-16T19:26:10.125 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/... 2024-04-16T19:26:10.125 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GqmCnS/hYPxplRrxb2PiqaXYPhs= 2024-04-16T19:26:10.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GqmCnS/hYPxplRrxb2PiqaXYPhs='} 2024-04-16T19:26:10.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069057ceaaa83cf40-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.167 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.167 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.167 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.167 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.167 DEBUG:boto:Data: 2024-04-16T19:26:10.167 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.167 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.167 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.167 DEBUG:boto:Params: {} 2024-04-16T19:26:10.167 DEBUG:boto:Token: None 2024-04-16T19:26:10.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078ae885c323c8445-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.170 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:10.170 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:10.170 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.170 DEBUG:boto:Path: /yrjefh-7/.o 2024-04-16T19:26:10.170 DEBUG:boto:Data: 2024-04-16T19:26:10.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.170 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.170 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.170 DEBUG:boto:Params: {} 2024-04-16T19:26:10.171 DEBUG:boto:Token: None 2024-04-16T19:26:10.171 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/.o 2024-04-16T19:26:10.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0wpxIx0iUGN1biil0wGASIBJhK8= 2024-04-16T19:26:10.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0wpxIx0iUGN1biil0wGASIBJhK8='} 2024-04-16T19:26:10.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009442641c546de95b-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.175 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:10.175 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:10.175 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.175 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:10.175 DEBUG:boto:Data: 2024-04-16T19:26:10.175 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.175 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.175 DEBUG:boto:Params: {} 2024-04-16T19:26:10.175 DEBUG:boto:Token: None 2024-04-16T19:26:10.176 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/ 2024-04-16T19:26:10.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g= 2024-04-16T19:26:10.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nk3XZOQkcEOFLJCMKhEZscllw6g=', 'Content-Length': '0'} 2024-04-16T19:26:10.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edf090dde1af405f-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.178 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:10.178 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:10.178 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.178 DEBUG:boto:Path: /yrjefh-7/.o. 2024-04-16T19:26:10.178 DEBUG:boto:Data: 2024-04-16T19:26:10.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.178 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:10.178 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.178 DEBUG:boto:Params: {} 2024-04-16T19:26:10.179 DEBUG:boto:Token: None 2024-04-16T19:26:10.179 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-7/.o. 2024-04-16T19:26:10.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i3Zc++PmvTytVuIzi9yBEBWUKFI= 2024-04-16T19:26:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i3Zc++PmvTytVuIzi9yBEBWUKFI='} 2024-04-16T19:26:10.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000250cea345194f854-00661ed0d2-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.184 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.184 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.184 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.185 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.185 DEBUG:boto:Data: 2024-04-16T19:26:10.185 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.185 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.185 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.185 DEBUG:boto:Params: {} 2024-04-16T19:26:10.185 DEBUG:boto:Token: None 2024-04-16T19:26:10.185 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.185 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a66c4f0e07c5bbf1-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.188 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:10.188 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:10.188 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.188 DEBUG:boto:Path: /yrjefh-8/myobj 2024-04-16T19:26:10.188 DEBUG:boto:Data: 2024-04-16T19:26:10.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.188 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.188 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.188 DEBUG:boto:Params: {} 2024-04-16T19:26:10.189 DEBUG:boto:Token: None 2024-04-16T19:26:10.189 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/myobj 2024-04-16T19:26:10.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5QpE5Rgu/++GJ86k3A2lGVF6Arc= 2024-04-16T19:26:10.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5QpE5Rgu/++GJ86k3A2lGVF6Arc='} 2024-04-16T19:26:10.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc34bdc6845ed0d4-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.231 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.231 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.231 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.231 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.231 DEBUG:boto:Data: 2024-04-16T19:26:10.231 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.231 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.231 DEBUG:boto:Params: {} 2024-04-16T19:26:10.231 DEBUG:boto:Token: None 2024-04-16T19:26:10.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6a2d007bb8754e7-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.234 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:10.234 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:10.234 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.234 DEBUG:boto:Path: /yrjefh-8/_myobj 2024-04-16T19:26:10.234 DEBUG:boto:Data: 2024-04-16T19:26:10.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.234 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.235 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.235 DEBUG:boto:Params: {} 2024-04-16T19:26:10.235 DEBUG:boto:Token: None 2024-04-16T19:26:10.235 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/_myobj 2024-04-16T19:26:10.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2P25GdJpM2UFVNDTYicG1zzsAtw= 2024-04-16T19:26:10.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2P25GdJpM2UFVNDTYicG1zzsAtw='} 2024-04-16T19:26:10.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cd663f3ab2036c4-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.244 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.244 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.244 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.244 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.244 DEBUG:boto:Data: 2024-04-16T19:26:10.244 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.245 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.245 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.245 DEBUG:boto:Params: {} 2024-04-16T19:26:10.245 DEBUG:boto:Token: None 2024-04-16T19:26:10.245 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8780be0c0cb630a-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.249 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:10.249 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:10.249 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.249 DEBUG:boto:Path: /yrjefh-8/%3A 2024-04-16T19:26:10.249 DEBUG:boto:Data: 2024-04-16T19:26:10.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.249 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.249 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.250 DEBUG:boto:Params: {} 2024-04-16T19:26:10.250 DEBUG:boto:Token: None 2024-04-16T19:26:10.250 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/%3A 2024-04-16T19:26:10.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ednXvpOqR5EHrk04UQJ+0JwyGuA= 2024-04-16T19:26:10.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ednXvpOqR5EHrk04UQJ+0JwyGuA='} 2024-04-16T19:26:10.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dadcb391b3b30f30-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.264 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.264 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.265 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.265 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.265 DEBUG:boto:Data: 2024-04-16T19:26:10.265 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.265 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.265 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.265 DEBUG:boto:Params: {} 2024-04-16T19:26:10.266 DEBUG:boto:Token: None 2024-04-16T19:26:10.266 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.266 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000044620b81f03e338-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.269 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:10.269 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:10.269 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.269 DEBUG:boto:Path: /yrjefh-8/%26 2024-04-16T19:26:10.270 DEBUG:boto:Data: 2024-04-16T19:26:10.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.270 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.270 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.270 DEBUG:boto:Params: {} 2024-04-16T19:26:10.270 DEBUG:boto:Token: None 2024-04-16T19:26:10.270 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/%26 2024-04-16T19:26:10.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9bai7KGYbVkFwZWw/Akge1bF3pw= 2024-04-16T19:26:10.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9bai7KGYbVkFwZWw/Akge1bF3pw='} 2024-04-16T19:26:10.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f258a5f96a92f4a-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.315 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.315 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.315 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.315 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.315 DEBUG:boto:Data: 2024-04-16T19:26:10.315 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.315 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.315 DEBUG:boto:Params: {} 2024-04-16T19:26:10.316 DEBUG:boto:Token: None 2024-04-16T19:26:10.316 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084cb5aa0f22ae94b-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.318 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:10.319 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:10.319 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.319 DEBUG:boto:Path: /yrjefh-8/. 2024-04-16T19:26:10.319 DEBUG:boto:Data: 2024-04-16T19:26:10.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.319 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.319 DEBUG:boto:Params: {} 2024-04-16T19:26:10.319 DEBUG:boto:Token: None 2024-04-16T19:26:10.319 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/. 2024-04-16T19:26:10.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MRAFxBBUtcsUkdOQmF/34rplvwE= 2024-04-16T19:26:10.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MRAFxBBUtcsUkdOQmF/34rplvwE='} 2024-04-16T19:26:10.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae4f22b5e1a5ea98-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.332 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.332 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.332 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.332 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.333 DEBUG:boto:Data: 2024-04-16T19:26:10.333 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.333 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.333 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.333 DEBUG:boto:Params: {} 2024-04-16T19:26:10.333 DEBUG:boto:Token: None 2024-04-16T19:26:10.333 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.333 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e43f7854750a6d9-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.336 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:10.336 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:10.336 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.336 DEBUG:boto:Path: /yrjefh-8/.. 2024-04-16T19:26:10.336 DEBUG:boto:Data: 2024-04-16T19:26:10.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.336 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.336 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.336 DEBUG:boto:Params: {} 2024-04-16T19:26:10.336 DEBUG:boto:Token: None 2024-04-16T19:26:10.337 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/.. 2024-04-16T19:26:10.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A1Rn+n+cgUU0ALdvKDt9UUMjbwM= 2024-04-16T19:26:10.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A1Rn+n+cgUU0ALdvKDt9UUMjbwM='} 2024-04-16T19:26:10.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012d7e238eb2ae474-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.378 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.379 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.379 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.379 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.379 DEBUG:boto:Data: 2024-04-16T19:26:10.379 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.379 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.379 DEBUG:boto:Params: {} 2024-04-16T19:26:10.379 DEBUG:boto:Token: None 2024-04-16T19:26:10.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7bbe24e812103f7-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.382 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:10.382 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:10.382 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.382 DEBUG:boto:Path: /yrjefh-8/... 2024-04-16T19:26:10.382 DEBUG:boto:Data: 2024-04-16T19:26:10.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.383 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.383 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.383 DEBUG:boto:Params: {} 2024-04-16T19:26:10.383 DEBUG:boto:Token: None 2024-04-16T19:26:10.383 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/... 2024-04-16T19:26:10.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tg6Rntdsox4P1Ij5R6VYybLe4E4= 2024-04-16T19:26:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tg6Rntdsox4P1Ij5R6VYybLe4E4='} 2024-04-16T19:26:10.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae0d494f1c5d5d8e-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.387 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.387 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.387 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.388 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.388 DEBUG:boto:Data: 2024-04-16T19:26:10.388 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.388 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.388 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.388 DEBUG:boto:Params: {} 2024-04-16T19:26:10.388 DEBUG:boto:Token: None 2024-04-16T19:26:10.388 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d443ef7f1f34a1ed-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.390 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:10.390 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:10.391 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.391 DEBUG:boto:Path: /yrjefh-8/.o 2024-04-16T19:26:10.391 DEBUG:boto:Data: 2024-04-16T19:26:10.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.391 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.391 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.391 DEBUG:boto:Params: {} 2024-04-16T19:26:10.391 DEBUG:boto:Token: None 2024-04-16T19:26:10.391 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/.o 2024-04-16T19:26:10.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FgLnKLvueiacTQW6ASh1tD6i0b8= 2024-04-16T19:26:10.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FgLnKLvueiacTQW6ASh1tD6i0b8='} 2024-04-16T19:26:10.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bce7623b022f094-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.396 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:10.396 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:10.396 DEBUG:boto:Method: HEAD 2024-04-16T19:26:10.396 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:10.396 DEBUG:boto:Data: 2024-04-16T19:26:10.396 DEBUG:boto:Headers: {} 2024-04-16T19:26:10.396 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.396 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.396 DEBUG:boto:Params: {} 2024-04-16T19:26:10.396 DEBUG:boto:Token: None 2024-04-16T19:26:10.397 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/ 2024-04-16T19:26:10.397 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ= 2024-04-16T19:26:10.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:00IICr5yX/dHzI1hEvsrxPX3wXQ=', 'Content-Length': '0'} 2024-04-16T19:26:10.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e180d53c1c79fdd-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.398 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:10.399 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:10.399 DEBUG:boto:Method: PUT 2024-04-16T19:26:10.399 DEBUG:boto:Path: /yrjefh-8/.o. 2024-04-16T19:26:10.399 DEBUG:boto:Data: 2024-04-16T19:26:10.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:10.399 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:10.399 DEBUG:boto:Port: 8000 2024-04-16T19:26:10.399 DEBUG:boto:Params: {} 2024-04-16T19:26:10.399 DEBUG:boto:Token: None 2024-04-16T19:26:10.400 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:10 GMT /yrjefh-8/.o. 2024-04-16T19:26:10.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PqBH8qsn38ebr0JMxQx7qrd4E9I= 2024-04-16T19:26:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PqBH8qsn38ebr0JMxQx7qrd4E9I='} 2024-04-16T19:26:10.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000440a029f4a3d6b58-00661ed0d2-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:10.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:10.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:10.443 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:10.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:10.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:10.540 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:26:10.561 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295569.965150_124.1","last_update":"2024-04-16T19:26:09.965150Z"},{"marker":"","last_update":"0.000000"}] 2024-04-16T19:26:10.562 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.965150_124.1', 3: ''} 2024-04-16T19:26:10.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:26:10.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:10.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:10.562 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:10.611 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:10.611 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:10.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:10.659+0000 7f785cc24800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:10.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:10.660+0000 7f785cc24800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:10.682 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.939567_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.939567Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:10Z"} 2024-04-16T19:26:10.682 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.939567_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.939567Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:10Z"} 2024-04-16T19:26:10.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295569.939567_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:09.939567Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:10.682 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.965150_124.1', 3: ''} 2024-04-16T19:26:10.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.939567_115.1', 3: ''} 2024-04-16T19:26:10.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713295569.965150_124.1 target=1_1713295569.939567_115.1 2024-04-16T19:26:15.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:15.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:15.684 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:15.738 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:15.739 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:15.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:15.791+0000 7fcc5220f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:15.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:15.792+0000 7fcc5220f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:15.816 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.965150_124.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.965150Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:15Z"} 2024-04-16T19:26:15.816 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295569.965150_124.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:09.965150Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:15Z"} 2024-04-16T19:26:15.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295569.965150_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:09.965150Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-16T19:26:15.816 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.965150_124.1', 3: ''} 2024-04-16T19:26:15.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295569.965150_124.1', 3: ''} 2024-04-16T19:26:15.817 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:15.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-7', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:15.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-7', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:15.817 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-7 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:15.872 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:15.872 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:15.923+0000 7f120df2c800 1 waiting to reach incremental sync.. 2024-04-16T19:26:20.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:20.934+0000 7f120df2c800 1 bucket sync caught up with source: 2024-04-16T19:26:20.936 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000002.6.6, , 00000000001.7.6, , 00000000001.7.6, 00000000002.9.6, 00000000001.5.6, 00000000001.9.6, 00000000001.8.6, ] 2024-04-16T19:26:20.936 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000002.6.6, , 00000000001.7.6, , 00000000001.7.6, 00000000002.9.6, 00000000001.5.6, 00000000001.9.6, 00000000001.8.6, ] 2024-04-16T19:26:20.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:20.934+0000 7f120df2c800 0 bucket checkpoint complete 2024-04-16T19:26:20.959 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-7 zones={test-zone2, test-zone1} 2024-04-16T19:26:20.959 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.959 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.960 DEBUG:boto:Method: HEAD 2024-04-16T19:26:20.960 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:20.960 DEBUG:boto:Data: 2024-04-16T19:26:20.960 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:20.960 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.960 DEBUG:boto:Params: {} 2024-04-16T19:26:20.960 DEBUG:boto:Token: None 2024-04-16T19:26:20.961 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/ 2024-04-16T19:26:20.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZFqbBykWy/2q44G16hVtAiIeKXg= 2024-04-16T19:26:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZFqbBykWy/2q44G16hVtAiIeKXg=', 'Content-Length': '0'} 2024-04-16T19:26:20.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cbf953c18c0e131-00661ed0dc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.963 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.963 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.964 DEBUG:boto:Method: HEAD 2024-04-16T19:26:20.964 DEBUG:boto:Path: /yrjefh-7/ 2024-04-16T19:26:20.964 DEBUG:boto:Data: 2024-04-16T19:26:20.964 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:20.964 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.964 DEBUG:boto:Params: {} 2024-04-16T19:26:20.964 DEBUG:boto:Token: None 2024-04-16T19:26:20.964 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/ 2024-04-16T19:26:20.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZFqbBykWy/2q44G16hVtAiIeKXg= 2024-04-16T19:26:20.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZFqbBykWy/2q44G16hVtAiIeKXg=', 'Content-Length': '0'} 2024-04-16T19:26:20.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab80eb588bbd014-00661ed0dc-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.967 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:20.967 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.967 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.967 DEBUG:boto:path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.967 DEBUG:boto:auth_path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.967 DEBUG:boto:Method: GET 2024-04-16T19:26:20.968 DEBUG:boto:Path: /yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.968 DEBUG:boto:Data: 2024-04-16T19:26:20.968 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.968 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:20.968 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.968 DEBUG:boto:Params: {} 2024-04-16T19:26:20.968 DEBUG:boto:Token: None 2024-04-16T19:26:20.968 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/?versions 2024-04-16T19:26:20.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI= 2024-04-16T19:26:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI=', 'Content-Length': '0'} 2024-04-16T19:26:20.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c107e04f54129a52-00661ed0dc-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.971 DEBUG:boto:b'yrjefh-7999false&nulltrue2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:20.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:20.980 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.980 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.980 DEBUG:boto:path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.980 DEBUG:boto:auth_path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.980 DEBUG:boto:Method: GET 2024-04-16T19:26:20.980 DEBUG:boto:Path: /yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.980 DEBUG:boto:Data: 2024-04-16T19:26:20.980 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.980 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:20.980 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.980 DEBUG:boto:Params: {} 2024-04-16T19:26:20.980 DEBUG:boto:Token: None 2024-04-16T19:26:20.981 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/?versions 2024-04-16T19:26:20.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI= 2024-04-16T19:26:20.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI=', 'Content-Length': '0'} 2024-04-16T19:26:20.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650b69b10c63d00d-00661ed0dc-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.983 DEBUG:boto:b'yrjefh-7999false&nulltrue2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-16T19:26:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-04-16T19:26:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-04-16T19:26:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-04-16T19:26:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-04-16T19:26:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-04-16T19:26:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-16T19:26:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-16T19:26:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:20.985 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.985 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.985 DEBUG:boto:path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.985 DEBUG:boto:auth_path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.985 DEBUG:boto:Method: GET 2024-04-16T19:26:20.985 DEBUG:boto:Path: /yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.985 DEBUG:boto:Data: 2024-04-16T19:26:20.985 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.986 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:20.986 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.986 DEBUG:boto:Params: {} 2024-04-16T19:26:20.986 DEBUG:boto:Token: None 2024-04-16T19:26:20.986 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/?versions 2024-04-16T19:26:20.986 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI= 2024-04-16T19:26:20.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI=', 'Content-Length': '0'} 2024-04-16T19:26:20.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000330ae1965295f0a8-00661ed0dc-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.988 DEBUG:boto:b'yrjefh-7999false&nulltrue2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:20.989 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:20.989 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:20.989 DEBUG:boto:path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.989 DEBUG:boto:auth_path=/yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.990 DEBUG:boto:Method: GET 2024-04-16T19:26:20.990 DEBUG:boto:Path: /yrjefh-7/?versions&max-keys=999 2024-04-16T19:26:20.990 DEBUG:boto:Data: 2024-04-16T19:26:20.990 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:20.990 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.990 DEBUG:boto:Params: {} 2024-04-16T19:26:20.990 DEBUG:boto:Token: None 2024-04-16T19:26:20.990 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/?versions 2024-04-16T19:26:20.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI= 2024-04-16T19:26:20.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enPkJyLbejq+bjeJmRzOVt4XxYI=', 'Content-Length': '0'} 2024-04-16T19:26:20.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000520f568311b565a2-00661ed0dc-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.993 DEBUG:boto:b'yrjefh-7999false&nulltrue2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:20.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:20.994 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:20.994 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:20.994 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.994 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.994 DEBUG:boto:Method: GET 2024-04-16T19:26:20.994 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.995 DEBUG:boto:Data: 2024-04-16T19:26:20.995 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:20.995 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.995 DEBUG:boto:Params: {} 2024-04-16T19:26:20.995 DEBUG:boto:Token: None 2024-04-16T19:26:20.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MD/VVzc0RO53EKoITDnMu86NLKM= 2024-04-16T19:26:20.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MD/VVzc0RO53EKoITDnMu86NLKM=', 'Content-Length': '0'} 2024-04-16T19:26:20.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5650c9e04ad3683-00661ed0dc-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:20.997 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:20.997 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:20.998 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.998 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.998 DEBUG:boto:Method: GET 2024-04-16T19:26:20.998 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.998 DEBUG:boto:Data: 2024-04-16T19:26:20.998 DEBUG:boto:Headers: {} 2024-04-16T19:26:20.998 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:20.998 DEBUG:boto:Port: 8000 2024-04-16T19:26:20.998 DEBUG:boto:Params: {} 2024-04-16T19:26:20.998 DEBUG:boto:Token: None 2024-04-16T19:26:20.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:20 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:20.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MD/VVzc0RO53EKoITDnMu86NLKM= 2024-04-16T19:26:20.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:20 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MD/VVzc0RO53EKoITDnMu86NLKM=', 'Content-Length': '0'} 2024-04-16T19:26:21.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005abec52a5a0b2176-00661ed0dc-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.042 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.042 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.043 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.043 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.043 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.043 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.043 DEBUG:boto:Data: 2024-04-16T19:26:21.043 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.043 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.043 DEBUG:boto:Params: {} 2024-04-16T19:26:21.043 DEBUG:boto:Token: None 2024-04-16T19:26:21.044 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sjCh+JoF4+qhkc/2jd/DiDFu10c= 2024-04-16T19:26:21.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sjCh+JoF4+qhkc/2jd/DiDFu10c=', 'Content-Length': '0'} 2024-04-16T19:26:21.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca34c0f472004413-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.046 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.046 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.046 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.046 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.046 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.046 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.046 DEBUG:boto:Data: 2024-04-16T19:26:21.047 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.047 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.047 DEBUG:boto:Params: {} 2024-04-16T19:26:21.047 DEBUG:boto:Token: None 2024-04-16T19:26:21.047 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sjCh+JoF4+qhkc/2jd/DiDFu10c= 2024-04-16T19:26:21.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sjCh+JoF4+qhkc/2jd/DiDFu10c=', 'Content-Length': '0'} 2024-04-16T19:26:21.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000750e6a8b36b1ef23-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:21.049 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.050 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.050 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.050 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.050 DEBUG:boto:Method: GET 2024-04-16T19:26:21.050 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.050 DEBUG:boto:Data: 2024-04-16T19:26:21.050 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.050 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.050 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.050 DEBUG:boto:Params: {} 2024-04-16T19:26:21.050 DEBUG:boto:Token: None 2024-04-16T19:26:21.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEhLzfausHpEXG0FHJcLpVI0J6g= 2024-04-16T19:26:21.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEhLzfausHpEXG0FHJcLpVI0J6g=', 'Content-Length': '0'} 2024-04-16T19:26:21.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3c6c4b36e20201a-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.094 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.094 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.095 DEBUG:boto:path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.095 DEBUG:boto:auth_path=/yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.095 DEBUG:boto:Method: GET 2024-04-16T19:26:21.095 DEBUG:boto:Path: /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.095 DEBUG:boto:Data: 2024-04-16T19:26:21.095 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.095 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.095 DEBUG:boto:Params: {} 2024-04-16T19:26:21.095 DEBUG:boto:Token: None 2024-04-16T19:26:21.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26?versionId=null 2024-04-16T19:26:21.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEhLzfausHpEXG0FHJcLpVI0J6g= 2024-04-16T19:26:21.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEhLzfausHpEXG0FHJcLpVI0J6g=', 'Content-Length': '0'} 2024-04-16T19:26:21.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5ed46ea81e34715-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.098 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.098 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.098 DEBUG:boto:path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.098 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.098 DEBUG:boto:Method: GET 2024-04-16T19:26:21.098 DEBUG:boto:Path: /yrjefh-7/?prefix=%26 2024-04-16T19:26:21.098 DEBUG:boto:Data: 2024-04-16T19:26:21.098 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.099 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.099 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.099 DEBUG:boto:Params: {} 2024-04-16T19:26:21.099 DEBUG:boto:Token: None 2024-04-16T19:26:21.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c51dd8d3409800ab-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.142 DEBUG:boto:b'yrjefh-7&1000false&2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.143 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.143 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.143 DEBUG:boto:path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.143 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.143 DEBUG:boto:Method: GET 2024-04-16T19:26:21.143 DEBUG:boto:Path: /yrjefh-7/?prefix=%26 2024-04-16T19:26:21.143 DEBUG:boto:Data: 2024-04-16T19:26:21.143 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.143 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.144 DEBUG:boto:Params: {} 2024-04-16T19:26:21.144 DEBUG:boto:Token: None 2024-04-16T19:26:21.144 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fa054972fd93d2-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.146 DEBUG:boto:b'yrjefh-7&1000false&2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.147 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.147 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.147 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.147 DEBUG:boto:Path: /yrjefh-7/%26 2024-04-16T19:26:21.147 DEBUG:boto:Data: 2024-04-16T19:26:21.147 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.147 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.147 DEBUG:boto:Params: {} 2024-04-16T19:26:21.148 DEBUG:boto:Token: None 2024-04-16T19:26:21.148 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26 2024-04-16T19:26:21.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eM1Gy6/RP68oFIVN1LrlP9CW+Pg= 2024-04-16T19:26:21.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eM1Gy6/RP68oFIVN1LrlP9CW+Pg=', 'Content-Length': '0'} 2024-04-16T19:26:21.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005567e7a55b94048a-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.150 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.150 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.150 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.150 DEBUG:boto:Path: /yrjefh-7/%26 2024-04-16T19:26:21.150 DEBUG:boto:Data: 2024-04-16T19:26:21.151 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.151 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.151 DEBUG:boto:Params: {} 2024-04-16T19:26:21.151 DEBUG:boto:Token: None 2024-04-16T19:26:21.151 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26 2024-04-16T19:26:21.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eM1Gy6/RP68oFIVN1LrlP9CW+Pg= 2024-04-16T19:26:21.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eM1Gy6/RP68oFIVN1LrlP9CW+Pg=', 'Content-Length': '0'} 2024-04-16T19:26:21.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007667e93366608515-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:21.153 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.153 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.153 DEBUG:boto:Method: GET 2024-04-16T19:26:21.153 DEBUG:boto:Path: /yrjefh-7/%26 2024-04-16T19:26:21.154 DEBUG:boto:Data: 2024-04-16T19:26:21.154 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.154 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.154 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.154 DEBUG:boto:Params: {} 2024-04-16T19:26:21.154 DEBUG:boto:Token: None 2024-04-16T19:26:21.154 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26 2024-04-16T19:26:21.154 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pwSYDfvuszTwF7CIfEkiJKmA+m0= 2024-04-16T19:26:21.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pwSYDfvuszTwF7CIfEkiJKmA+m0=', 'Content-Length': '0'} 2024-04-16T19:26:21.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e37e069d431cd9-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.198 DEBUG:boto:path=/yrjefh-7/%26 2024-04-16T19:26:21.198 DEBUG:boto:auth_path=/yrjefh-7/%26 2024-04-16T19:26:21.199 DEBUG:boto:Method: GET 2024-04-16T19:26:21.199 DEBUG:boto:Path: /yrjefh-7/%26 2024-04-16T19:26:21.199 DEBUG:boto:Data: 2024-04-16T19:26:21.199 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.199 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.199 DEBUG:boto:Params: {} 2024-04-16T19:26:21.199 DEBUG:boto:Token: None 2024-04-16T19:26:21.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%26 2024-04-16T19:26:21.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pwSYDfvuszTwF7CIfEkiJKmA+m0= 2024-04-16T19:26:21.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pwSYDfvuszTwF7CIfEkiJKmA+m0=', 'Content-Length': '0'} 2024-04-16T19:26:21.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002afc7bd0a0582b28-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.202 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.202 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.202 DEBUG:boto:path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.202 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.202 DEBUG:boto:Method: GET 2024-04-16T19:26:21.202 DEBUG:boto:Path: /yrjefh-7/?prefix=%26 2024-04-16T19:26:21.202 DEBUG:boto:Data: 2024-04-16T19:26:21.202 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.202 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.202 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.202 DEBUG:boto:Params: {} 2024-04-16T19:26:21.203 DEBUG:boto:Token: None 2024-04-16T19:26:21.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097075b5e6be42b46-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.205 DEBUG:boto:b'yrjefh-7&1000false&2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.206 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.206 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.206 DEBUG:boto:path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.206 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%26 2024-04-16T19:26:21.206 DEBUG:boto:Method: GET 2024-04-16T19:26:21.206 DEBUG:boto:Path: /yrjefh-7/?prefix=%26 2024-04-16T19:26:21.206 DEBUG:boto:Data: 2024-04-16T19:26:21.206 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.206 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.206 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.206 DEBUG:boto:Params: {} 2024-04-16T19:26:21.207 DEBUG:boto:Token: None 2024-04-16T19:26:21.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ac1cf9b89712ba-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.209 DEBUG:boto:b'yrjefh-7&1000false&2024-04-16T19:26:10.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:21.210 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.210 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.210 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.210 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.210 DEBUG:boto:Method: GET 2024-04-16T19:26:21.210 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.210 DEBUG:boto:Data: 2024-04-16T19:26:21.210 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.210 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.211 DEBUG:boto:Params: {} 2024-04-16T19:26:21.211 DEBUG:boto:Token: None 2024-04-16T19:26:21.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM= 2024-04-16T19:26:21.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM=', 'Content-Length': '0'} 2024-04-16T19:26:21.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f83b4ec49a77230e-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.254 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.254 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.254 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.255 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.255 DEBUG:boto:Method: GET 2024-04-16T19:26:21.255 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.255 DEBUG:boto:Data: 2024-04-16T19:26:21.255 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.255 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.255 DEBUG:boto:Params: {} 2024-04-16T19:26:21.255 DEBUG:boto:Token: None 2024-04-16T19:26:21.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM= 2024-04-16T19:26:21.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM=', 'Content-Length': '0'} 2024-04-16T19:26:21.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059f6e698d960561a-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.258 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.258 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.258 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.258 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.258 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.258 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.258 DEBUG:boto:Data: 2024-04-16T19:26:21.259 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.259 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.259 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.259 DEBUG:boto:Params: {} 2024-04-16T19:26:21.259 DEBUG:boto:Token: None 2024-04-16T19:26:21.259 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvDX80T2JGuo0g3xSh/jwEpVBc8= 2024-04-16T19:26:21.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvDX80T2JGuo0g3xSh/jwEpVBc8=', 'Content-Length': '0'} 2024-04-16T19:26:21.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000028b6d061517d9fe-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.261 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.262 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.262 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.262 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.262 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.262 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.262 DEBUG:boto:Data: 2024-04-16T19:26:21.262 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.262 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.262 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.262 DEBUG:boto:Params: {} 2024-04-16T19:26:21.263 DEBUG:boto:Token: None 2024-04-16T19:26:21.263 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvDX80T2JGuo0g3xSh/jwEpVBc8= 2024-04-16T19:26:21.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvDX80T2JGuo0g3xSh/jwEpVBc8=', 'Content-Length': '0'} 2024-04-16T19:26:21.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c906bad08db02bb-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:21.265 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.265 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.265 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.266 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.266 DEBUG:boto:Method: GET 2024-04-16T19:26:21.266 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.266 DEBUG:boto:Data: 2024-04-16T19:26:21.266 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.266 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.266 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.266 DEBUG:boto:Params: {} 2024-04-16T19:26:21.266 DEBUG:boto:Token: None 2024-04-16T19:26:21.266 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM= 2024-04-16T19:26:21.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM=', 'Content-Length': '0'} 2024-04-16T19:26:21.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7155fd37c3a4a55-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.310 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.310 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.310 DEBUG:boto:path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.311 DEBUG:boto:auth_path=/yrjefh-7/.?versionId=null 2024-04-16T19:26:21.311 DEBUG:boto:Method: GET 2024-04-16T19:26:21.311 DEBUG:boto:Path: /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.311 DEBUG:boto:Data: 2024-04-16T19:26:21.311 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.311 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.311 DEBUG:boto:Params: {} 2024-04-16T19:26:21.311 DEBUG:boto:Token: None 2024-04-16T19:26:21.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.?versionId=null 2024-04-16T19:26:21.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM= 2024-04-16T19:26:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JYBOiDd2ik0FaDQqFPKkVDYGDlM=', 'Content-Length': '0'} 2024-04-16T19:26:21.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000827bf311a53888f1-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.314 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.314 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.314 DEBUG:boto:path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.314 DEBUG:boto:auth_path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.314 DEBUG:boto:Method: GET 2024-04-16T19:26:21.314 DEBUG:boto:Path: /yrjefh-7/?prefix=. 2024-04-16T19:26:21.314 DEBUG:boto:Data: 2024-04-16T19:26:21.314 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.314 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.315 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.315 DEBUG:boto:Params: {} 2024-04-16T19:26:21.315 DEBUG:boto:Token: None 2024-04-16T19:26:21.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627bddfe5a68d2ae-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.318 DEBUG:boto:b'yrjefh-7.1000false.2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.318 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.318 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.318 DEBUG:boto:path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.319 DEBUG:boto:auth_path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.319 DEBUG:boto:Method: GET 2024-04-16T19:26:21.319 DEBUG:boto:Path: /yrjefh-7/?prefix=. 2024-04-16T19:26:21.319 DEBUG:boto:Data: 2024-04-16T19:26:21.319 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.319 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.319 DEBUG:boto:Params: {} 2024-04-16T19:26:21.319 DEBUG:boto:Token: None 2024-04-16T19:26:21.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5409d93d553c46c-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.322 DEBUG:boto:b'yrjefh-7.1000false.2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.322 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.323 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.323 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.323 DEBUG:boto:Path: /yrjefh-7/. 2024-04-16T19:26:21.323 DEBUG:boto:Data: 2024-04-16T19:26:21.323 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.323 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.323 DEBUG:boto:Params: {} 2024-04-16T19:26:21.323 DEBUG:boto:Token: None 2024-04-16T19:26:21.323 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/. 2024-04-16T19:26:21.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hZjMg/iXxJFUgBbtps+cT8K4toQ= 2024-04-16T19:26:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hZjMg/iXxJFUgBbtps+cT8K4toQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007524da09383ba3a6-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.326 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.326 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.326 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.326 DEBUG:boto:Path: /yrjefh-7/. 2024-04-16T19:26:21.326 DEBUG:boto:Data: 2024-04-16T19:26:21.326 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.326 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.326 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.326 DEBUG:boto:Params: {} 2024-04-16T19:26:21.327 DEBUG:boto:Token: None 2024-04-16T19:26:21.327 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/. 2024-04-16T19:26:21.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hZjMg/iXxJFUgBbtps+cT8K4toQ= 2024-04-16T19:26:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hZjMg/iXxJFUgBbtps+cT8K4toQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000810e11d40132f333-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:21.329 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.329 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.330 DEBUG:boto:Method: GET 2024-04-16T19:26:21.330 DEBUG:boto:Path: /yrjefh-7/. 2024-04-16T19:26:21.330 DEBUG:boto:Data: 2024-04-16T19:26:21.330 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.330 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.330 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.330 DEBUG:boto:Params: {} 2024-04-16T19:26:21.330 DEBUG:boto:Token: None 2024-04-16T19:26:21.330 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/. 2024-04-16T19:26:21.330 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HV8YOKZVNrjFGv+WJesuGSduE0= 2024-04-16T19:26:21.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HV8YOKZVNrjFGv+WJesuGSduE0=', 'Content-Length': '0'} 2024-04-16T19:26:21.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a7d24a891f12bcd-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.333 DEBUG:boto:path=/yrjefh-7/. 2024-04-16T19:26:21.333 DEBUG:boto:auth_path=/yrjefh-7/. 2024-04-16T19:26:21.333 DEBUG:boto:Method: GET 2024-04-16T19:26:21.333 DEBUG:boto:Path: /yrjefh-7/. 2024-04-16T19:26:21.333 DEBUG:boto:Data: 2024-04-16T19:26:21.333 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.333 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.334 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.334 DEBUG:boto:Params: {} 2024-04-16T19:26:21.334 DEBUG:boto:Token: None 2024-04-16T19:26:21.334 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/. 2024-04-16T19:26:21.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HV8YOKZVNrjFGv+WJesuGSduE0= 2024-04-16T19:26:21.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HV8YOKZVNrjFGv+WJesuGSduE0=', 'Content-Length': '0'} 2024-04-16T19:26:21.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f936281a4030f8ec-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.336 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.336 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.336 DEBUG:boto:path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.337 DEBUG:boto:auth_path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.337 DEBUG:boto:Method: GET 2024-04-16T19:26:21.337 DEBUG:boto:Path: /yrjefh-7/?prefix=. 2024-04-16T19:26:21.337 DEBUG:boto:Data: 2024-04-16T19:26:21.337 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.337 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.337 DEBUG:boto:Params: {} 2024-04-16T19:26:21.337 DEBUG:boto:Token: None 2024-04-16T19:26:21.337 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd5bd74cbbcb58e-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.340 DEBUG:boto:b'yrjefh-7.1000false.2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.340 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.341 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.341 DEBUG:boto:path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.341 DEBUG:boto:auth_path=/yrjefh-7/?prefix=. 2024-04-16T19:26:21.341 DEBUG:boto:Method: GET 2024-04-16T19:26:21.341 DEBUG:boto:Path: /yrjefh-7/?prefix=. 2024-04-16T19:26:21.341 DEBUG:boto:Data: 2024-04-16T19:26:21.341 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.341 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.341 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.341 DEBUG:boto:Params: {} 2024-04-16T19:26:21.342 DEBUG:boto:Token: None 2024-04-16T19:26:21.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582c397f9e363e64-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.344 DEBUG:boto:b'yrjefh-7.1000false.2024-04-16T19:26:10.104Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:21.345 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.345 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.345 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.345 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.345 DEBUG:boto:Method: GET 2024-04-16T19:26:21.345 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.346 DEBUG:boto:Data: 2024-04-16T19:26:21.346 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.346 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.346 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.346 DEBUG:boto:Params: {} 2024-04-16T19:26:21.346 DEBUG:boto:Token: None 2024-04-16T19:26:21.346 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ= 2024-04-16T19:26:21.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ccaf5422e2e5918-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.349 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.349 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.349 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.349 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.349 DEBUG:boto:Method: GET 2024-04-16T19:26:21.349 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.349 DEBUG:boto:Data: 2024-04-16T19:26:21.349 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.349 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.349 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.349 DEBUG:boto:Params: {} 2024-04-16T19:26:21.350 DEBUG:boto:Token: None 2024-04-16T19:26:21.350 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.350 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ= 2024-04-16T19:26:21.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6b0ff5181d3de5a-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.352 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.352 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.352 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.352 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.352 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.352 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.353 DEBUG:boto:Data: 2024-04-16T19:26:21.353 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.353 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.353 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.353 DEBUG:boto:Params: {} 2024-04-16T19:26:21.353 DEBUG:boto:Token: None 2024-04-16T19:26:21.353 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBCN3ZJj7w6b5x/F6Bxfz9a4uX4= 2024-04-16T19:26:21.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBCN3ZJj7w6b5x/F6Bxfz9a4uX4=', 'Content-Length': '0'} 2024-04-16T19:26:21.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d4af307120bb19-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.355 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.355 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.355 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.356 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.356 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.356 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.356 DEBUG:boto:Data: 2024-04-16T19:26:21.356 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.356 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.356 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.356 DEBUG:boto:Params: {} 2024-04-16T19:26:21.356 DEBUG:boto:Token: None 2024-04-16T19:26:21.356 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.357 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBCN3ZJj7w6b5x/F6Bxfz9a4uX4= 2024-04-16T19:26:21.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBCN3ZJj7w6b5x/F6Bxfz9a4uX4=', 'Content-Length': '0'} 2024-04-16T19:26:21.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a87b2e26bc3bc4a2-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:21.359 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.359 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.359 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.359 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.359 DEBUG:boto:Method: GET 2024-04-16T19:26:21.359 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.359 DEBUG:boto:Data: 2024-04-16T19:26:21.359 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.359 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.360 DEBUG:boto:Params: {} 2024-04-16T19:26:21.360 DEBUG:boto:Token: None 2024-04-16T19:26:21.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ= 2024-04-16T19:26:21.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b126ff0ca9cbe203-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.402 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.402 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.402 DEBUG:boto:path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.403 DEBUG:boto:auth_path=/yrjefh-7/..?versionId=null 2024-04-16T19:26:21.403 DEBUG:boto:Method: GET 2024-04-16T19:26:21.403 DEBUG:boto:Path: /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.403 DEBUG:boto:Data: 2024-04-16T19:26:21.403 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.403 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.403 DEBUG:boto:Params: {} 2024-04-16T19:26:21.403 DEBUG:boto:Token: None 2024-04-16T19:26:21.403 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/..?versionId=null 2024-04-16T19:26:21.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ= 2024-04-16T19:26:21.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmqEJdtEwDsqNL7ejkasQI6T0dQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2078f36347abaff-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.406 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.406 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.406 DEBUG:boto:path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.406 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.406 DEBUG:boto:Method: GET 2024-04-16T19:26:21.406 DEBUG:boto:Path: /yrjefh-7/?prefix=.. 2024-04-16T19:26:21.406 DEBUG:boto:Data: 2024-04-16T19:26:21.406 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.407 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.407 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.407 DEBUG:boto:Params: {} 2024-04-16T19:26:21.407 DEBUG:boto:Token: None 2024-04-16T19:26:21.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce45eefd698631d9-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.409 DEBUG:boto:b'yrjefh-7..1000false..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.410 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.410 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.410 DEBUG:boto:path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.410 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.410 DEBUG:boto:Method: GET 2024-04-16T19:26:21.410 DEBUG:boto:Path: /yrjefh-7/?prefix=.. 2024-04-16T19:26:21.410 DEBUG:boto:Data: 2024-04-16T19:26:21.410 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.411 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.411 DEBUG:boto:Params: {} 2024-04-16T19:26:21.411 DEBUG:boto:Token: None 2024-04-16T19:26:21.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daaba6ee0c0d7b95-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.414 DEBUG:boto:b'yrjefh-7..1000false..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.414 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.414 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.414 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.414 DEBUG:boto:Path: /yrjefh-7/.. 2024-04-16T19:26:21.414 DEBUG:boto:Data: 2024-04-16T19:26:21.415 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.415 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.415 DEBUG:boto:Params: {} 2024-04-16T19:26:21.415 DEBUG:boto:Token: None 2024-04-16T19:26:21.415 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.. 2024-04-16T19:26:21.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gMU6rThk3atD8nXmN3k3jMNsvkc= 2024-04-16T19:26:21.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gMU6rThk3atD8nXmN3k3jMNsvkc=', 'Content-Length': '0'} 2024-04-16T19:26:21.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c001521c2d6f09c-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.417 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.418 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.418 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.418 DEBUG:boto:Path: /yrjefh-7/.. 2024-04-16T19:26:21.418 DEBUG:boto:Data: 2024-04-16T19:26:21.418 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.418 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.418 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.418 DEBUG:boto:Params: {} 2024-04-16T19:26:21.418 DEBUG:boto:Token: None 2024-04-16T19:26:21.418 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.. 2024-04-16T19:26:21.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gMU6rThk3atD8nXmN3k3jMNsvkc= 2024-04-16T19:26:21.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gMU6rThk3atD8nXmN3k3jMNsvkc=', 'Content-Length': '0'} 2024-04-16T19:26:21.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afc45cc54a3ea312-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:21.420 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.421 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.421 DEBUG:boto:Method: GET 2024-04-16T19:26:21.421 DEBUG:boto:Path: /yrjefh-7/.. 2024-04-16T19:26:21.421 DEBUG:boto:Data: 2024-04-16T19:26:21.421 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.421 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.421 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.421 DEBUG:boto:Params: {} 2024-04-16T19:26:21.421 DEBUG:boto:Token: None 2024-04-16T19:26:21.421 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.. 2024-04-16T19:26:21.422 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3L1QL2C2YzTp125vpmIU8GLNig= 2024-04-16T19:26:21.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3L1QL2C2YzTp125vpmIU8GLNig=', 'Content-Length': '0'} 2024-04-16T19:26:21.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c12b1c84661120c8-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.424 DEBUG:boto:path=/yrjefh-7/.. 2024-04-16T19:26:21.424 DEBUG:boto:auth_path=/yrjefh-7/.. 2024-04-16T19:26:21.424 DEBUG:boto:Method: GET 2024-04-16T19:26:21.424 DEBUG:boto:Path: /yrjefh-7/.. 2024-04-16T19:26:21.424 DEBUG:boto:Data: 2024-04-16T19:26:21.424 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.424 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.424 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.424 DEBUG:boto:Params: {} 2024-04-16T19:26:21.425 DEBUG:boto:Token: None 2024-04-16T19:26:21.425 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.. 2024-04-16T19:26:21.425 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3L1QL2C2YzTp125vpmIU8GLNig= 2024-04-16T19:26:21.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3L1QL2C2YzTp125vpmIU8GLNig=', 'Content-Length': '0'} 2024-04-16T19:26:21.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000617a335f359e22b0-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.427 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.427 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.427 DEBUG:boto:path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.427 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.427 DEBUG:boto:Method: GET 2024-04-16T19:26:21.428 DEBUG:boto:Path: /yrjefh-7/?prefix=.. 2024-04-16T19:26:21.428 DEBUG:boto:Data: 2024-04-16T19:26:21.428 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.428 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.428 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.428 DEBUG:boto:Params: {} 2024-04-16T19:26:21.428 DEBUG:boto:Token: None 2024-04-16T19:26:21.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b5b555e452c84b-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.431 DEBUG:boto:b'yrjefh-7..1000false..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.431 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.431 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.431 DEBUG:boto:path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.431 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.. 2024-04-16T19:26:21.431 DEBUG:boto:Method: GET 2024-04-16T19:26:21.432 DEBUG:boto:Path: /yrjefh-7/?prefix=.. 2024-04-16T19:26:21.432 DEBUG:boto:Data: 2024-04-16T19:26:21.432 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.432 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.432 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.432 DEBUG:boto:Params: {} 2024-04-16T19:26:21.432 DEBUG:boto:Token: None 2024-04-16T19:26:21.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057fdad9d3f2d28e1-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.478 DEBUG:boto:b'yrjefh-7..1000false..2024-04-16T19:26:10.117Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:21.479 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.479 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.479 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.479 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.479 DEBUG:boto:Method: GET 2024-04-16T19:26:21.479 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.479 DEBUG:boto:Data: 2024-04-16T19:26:21.479 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.480 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.480 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.480 DEBUG:boto:Params: {} 2024-04-16T19:26:21.480 DEBUG:boto:Token: None 2024-04-16T19:26:21.480 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ= 2024-04-16T19:26:21.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000892658edc499955f-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.483 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.483 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.483 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.483 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.483 DEBUG:boto:Method: GET 2024-04-16T19:26:21.483 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.483 DEBUG:boto:Data: 2024-04-16T19:26:21.483 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.483 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.484 DEBUG:boto:Params: {} 2024-04-16T19:26:21.484 DEBUG:boto:Token: None 2024-04-16T19:26:21.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ= 2024-04-16T19:26:21.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2150198f9e72934-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.486 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.487 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.487 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.487 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.487 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.487 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.487 DEBUG:boto:Data: 2024-04-16T19:26:21.487 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.487 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.487 DEBUG:boto:Params: {} 2024-04-16T19:26:21.487 DEBUG:boto:Token: None 2024-04-16T19:26:21.488 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u7Pj9pzl+tk2skDK9PMZtH66nJU= 2024-04-16T19:26:21.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u7Pj9pzl+tk2skDK9PMZtH66nJU=', 'Content-Length': '0'} 2024-04-16T19:26:21.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c432125fd5e387d6-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.490 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.490 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.490 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.491 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.491 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.491 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.491 DEBUG:boto:Data: 2024-04-16T19:26:21.491 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.491 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.491 DEBUG:boto:Params: {} 2024-04-16T19:26:21.491 DEBUG:boto:Token: None 2024-04-16T19:26:21.491 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u7Pj9pzl+tk2skDK9PMZtH66nJU= 2024-04-16T19:26:21.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u7Pj9pzl+tk2skDK9PMZtH66nJU=', 'Content-Length': '0'} 2024-04-16T19:26:21.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1d839f90e2df096-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:21.494 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.494 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.494 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.494 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.494 DEBUG:boto:Method: GET 2024-04-16T19:26:21.494 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.494 DEBUG:boto:Data: 2024-04-16T19:26:21.495 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.495 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.495 DEBUG:boto:Params: {} 2024-04-16T19:26:21.495 DEBUG:boto:Token: None 2024-04-16T19:26:21.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ= 2024-04-16T19:26:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b88a57d022efde40-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.498 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.498 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.498 DEBUG:boto:path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.498 DEBUG:boto:auth_path=/yrjefh-7/...?versionId=null 2024-04-16T19:26:21.498 DEBUG:boto:Method: GET 2024-04-16T19:26:21.498 DEBUG:boto:Path: /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.498 DEBUG:boto:Data: 2024-04-16T19:26:21.498 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.498 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.499 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.499 DEBUG:boto:Params: {} 2024-04-16T19:26:21.499 DEBUG:boto:Token: None 2024-04-16T19:26:21.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/...?versionId=null 2024-04-16T19:26:21.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ= 2024-04-16T19:26:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:soVvOb/8rsaujWp61jhIk1vAszQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004935ee4f428aa186-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.501 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.501 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.501 DEBUG:boto:path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.501 DEBUG:boto:auth_path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.501 DEBUG:boto:Method: GET 2024-04-16T19:26:21.502 DEBUG:boto:Path: /yrjefh-7/?prefix=... 2024-04-16T19:26:21.502 DEBUG:boto:Data: 2024-04-16T19:26:21.502 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.502 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.502 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.502 DEBUG:boto:Params: {} 2024-04-16T19:26:21.502 DEBUG:boto:Token: None 2024-04-16T19:26:21.502 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767400c419ee4666-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.505 DEBUG:boto:b'yrjefh-7...1000false...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.505 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.505 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.505 DEBUG:boto:path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.506 DEBUG:boto:auth_path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.506 DEBUG:boto:Method: GET 2024-04-16T19:26:21.506 DEBUG:boto:Path: /yrjefh-7/?prefix=... 2024-04-16T19:26:21.506 DEBUG:boto:Data: 2024-04-16T19:26:21.506 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.506 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.506 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.506 DEBUG:boto:Params: {} 2024-04-16T19:26:21.506 DEBUG:boto:Token: None 2024-04-16T19:26:21.506 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6ecc46408f8e14-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.509 DEBUG:boto:b'yrjefh-7...1000false...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.509 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.509 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.509 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.509 DEBUG:boto:Path: /yrjefh-7/... 2024-04-16T19:26:21.510 DEBUG:boto:Data: 2024-04-16T19:26:21.510 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.510 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.510 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.510 DEBUG:boto:Params: {} 2024-04-16T19:26:21.510 DEBUG:boto:Token: None 2024-04-16T19:26:21.510 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/... 2024-04-16T19:26:21.510 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zywu7uz7vupwBnJWyK4Kud+kyT4= 2024-04-16T19:26:21.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zywu7uz7vupwBnJWyK4Kud+kyT4=', 'Content-Length': '0'} 2024-04-16T19:26:21.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a0c5e435bd02661-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.513 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.513 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.513 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.513 DEBUG:boto:Path: /yrjefh-7/... 2024-04-16T19:26:21.513 DEBUG:boto:Data: 2024-04-16T19:26:21.513 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.513 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.513 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.513 DEBUG:boto:Params: {} 2024-04-16T19:26:21.514 DEBUG:boto:Token: None 2024-04-16T19:26:21.514 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/... 2024-04-16T19:26:21.514 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zywu7uz7vupwBnJWyK4Kud+kyT4= 2024-04-16T19:26:21.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zywu7uz7vupwBnJWyK4Kud+kyT4=', 'Content-Length': '0'} 2024-04-16T19:26:21.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a08b0f5d913623d-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:21.516 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.516 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.516 DEBUG:boto:Method: GET 2024-04-16T19:26:21.516 DEBUG:boto:Path: /yrjefh-7/... 2024-04-16T19:26:21.516 DEBUG:boto:Data: 2024-04-16T19:26:21.516 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.516 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.516 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.516 DEBUG:boto:Params: {} 2024-04-16T19:26:21.517 DEBUG:boto:Token: None 2024-04-16T19:26:21.517 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/... 2024-04-16T19:26:21.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c+ru3tokH90dVC6RgSPV0uvap78= 2024-04-16T19:26:21.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c+ru3tokH90dVC6RgSPV0uvap78=', 'Content-Length': '0'} 2024-04-16T19:26:21.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047f6c4a893f8629e-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.519 DEBUG:boto:path=/yrjefh-7/... 2024-04-16T19:26:21.519 DEBUG:boto:auth_path=/yrjefh-7/... 2024-04-16T19:26:21.520 DEBUG:boto:Method: GET 2024-04-16T19:26:21.520 DEBUG:boto:Path: /yrjefh-7/... 2024-04-16T19:26:21.520 DEBUG:boto:Data: 2024-04-16T19:26:21.520 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.520 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.520 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.520 DEBUG:boto:Params: {} 2024-04-16T19:26:21.520 DEBUG:boto:Token: None 2024-04-16T19:26:21.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/... 2024-04-16T19:26:21.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c+ru3tokH90dVC6RgSPV0uvap78= 2024-04-16T19:26:21.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c+ru3tokH90dVC6RgSPV0uvap78=', 'Content-Length': '0'} 2024-04-16T19:26:21.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a28edb731799fc2c-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.522 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.522 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.523 DEBUG:boto:path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.523 DEBUG:boto:auth_path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.523 DEBUG:boto:Method: GET 2024-04-16T19:26:21.523 DEBUG:boto:Path: /yrjefh-7/?prefix=... 2024-04-16T19:26:21.523 DEBUG:boto:Data: 2024-04-16T19:26:21.523 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.523 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.523 DEBUG:boto:Params: {} 2024-04-16T19:26:21.523 DEBUG:boto:Token: None 2024-04-16T19:26:21.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8db671f75345c88-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.566 DEBUG:boto:b'yrjefh-7...1000false...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.567 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.567 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.567 DEBUG:boto:path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.567 DEBUG:boto:auth_path=/yrjefh-7/?prefix=... 2024-04-16T19:26:21.567 DEBUG:boto:Method: GET 2024-04-16T19:26:21.567 DEBUG:boto:Path: /yrjefh-7/?prefix=... 2024-04-16T19:26:21.567 DEBUG:boto:Data: 2024-04-16T19:26:21.567 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.567 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.567 DEBUG:boto:Params: {} 2024-04-16T19:26:21.567 DEBUG:boto:Token: None 2024-04-16T19:26:21.568 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a431129f12b97491-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.570 DEBUG:boto:b'yrjefh-7...1000false...2024-04-16T19:26:10.127Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:21.571 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.571 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.571 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.571 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.571 DEBUG:boto:Method: GET 2024-04-16T19:26:21.571 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.571 DEBUG:boto:Data: 2024-04-16T19:26:21.571 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.571 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.572 DEBUG:boto:Params: {} 2024-04-16T19:26:21.572 DEBUG:boto:Token: None 2024-04-16T19:26:21.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No= 2024-04-16T19:26:21.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No=', 'Content-Length': '0'} 2024-04-16T19:26:21.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:19 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000148becf68ab4f97a-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.575 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.575 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.575 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.575 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.575 DEBUG:boto:Method: GET 2024-04-16T19:26:21.575 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.575 DEBUG:boto:Data: 2024-04-16T19:26:21.575 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.576 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.576 DEBUG:boto:Params: {} 2024-04-16T19:26:21.576 DEBUG:boto:Token: None 2024-04-16T19:26:21.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No= 2024-04-16T19:26:21.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No=', 'Content-Length': '0'} 2024-04-16T19:26:21.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056872ba1ae67e22e-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.622 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.622 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.622 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.623 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.623 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.623 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.623 DEBUG:boto:Data: 2024-04-16T19:26:21.623 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.623 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.623 DEBUG:boto:Params: {} 2024-04-16T19:26:21.623 DEBUG:boto:Token: None 2024-04-16T19:26:21.623 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WBDONrRkyAHDpAaDyruZAJjUe8= 2024-04-16T19:26:21.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WBDONrRkyAHDpAaDyruZAJjUe8=', 'Content-Length': '0'} 2024-04-16T19:26:21.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:19 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000208fb83df4d35ef8-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.626 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.626 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.626 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.626 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.626 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.626 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.627 DEBUG:boto:Data: 2024-04-16T19:26:21.627 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.627 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.627 DEBUG:boto:Params: {} 2024-04-16T19:26:21.627 DEBUG:boto:Token: None 2024-04-16T19:26:21.627 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WBDONrRkyAHDpAaDyruZAJjUe8= 2024-04-16T19:26:21.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WBDONrRkyAHDpAaDyruZAJjUe8=', 'Content-Length': '0'} 2024-04-16T19:26:21.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d71472072245f820-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:21.630 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.630 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.630 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.630 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.630 DEBUG:boto:Method: GET 2024-04-16T19:26:21.630 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.630 DEBUG:boto:Data: 2024-04-16T19:26:21.630 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.630 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.630 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.630 DEBUG:boto:Params: {} 2024-04-16T19:26:21.630 DEBUG:boto:Token: None 2024-04-16T19:26:21.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No= 2024-04-16T19:26:21.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No=', 'Content-Length': '0'} 2024-04-16T19:26:21.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:19 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecc46a8af34fb22d-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.633 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.633 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.633 DEBUG:boto:path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.633 DEBUG:boto:auth_path=/yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.634 DEBUG:boto:Method: GET 2024-04-16T19:26:21.634 DEBUG:boto:Path: /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.634 DEBUG:boto:Data: 2024-04-16T19:26:21.634 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.634 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.634 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.634 DEBUG:boto:Params: {} 2024-04-16T19:26:21.634 DEBUG:boto:Token: None 2024-04-16T19:26:21.634 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o?versionId=null 2024-04-16T19:26:21.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No= 2024-04-16T19:26:21.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O5C9kDreLu8Hm0I/CXUHaenE3No=', 'Content-Length': '0'} 2024-04-16T19:26:21.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe2aa9ff4d56b933-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.637 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.637 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.637 DEBUG:boto:path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.637 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.637 DEBUG:boto:Method: GET 2024-04-16T19:26:21.637 DEBUG:boto:Path: /yrjefh-7/?prefix=.o 2024-04-16T19:26:21.637 DEBUG:boto:Data: 2024-04-16T19:26:21.638 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.638 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.638 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.638 DEBUG:boto:Params: {} 2024-04-16T19:26:21.638 DEBUG:boto:Token: None 2024-04-16T19:26:21.638 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.638 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe062bb89642749-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.641 DEBUG:boto:b'yrjefh-7.o1000false.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.641 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.642 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.642 DEBUG:boto:path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.642 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.642 DEBUG:boto:Method: GET 2024-04-16T19:26:21.642 DEBUG:boto:Path: /yrjefh-7/?prefix=.o 2024-04-16T19:26:21.642 DEBUG:boto:Data: 2024-04-16T19:26:21.642 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.642 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.642 DEBUG:boto:Params: {} 2024-04-16T19:26:21.643 DEBUG:boto:Token: None 2024-04-16T19:26:21.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033764241e417a529-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.645 DEBUG:boto:b'yrjefh-7.o1000false.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.646 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.646 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.646 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.646 DEBUG:boto:Path: /yrjefh-7/.o 2024-04-16T19:26:21.646 DEBUG:boto:Data: 2024-04-16T19:26:21.646 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.646 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.646 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.646 DEBUG:boto:Params: {} 2024-04-16T19:26:21.646 DEBUG:boto:Token: None 2024-04-16T19:26:21.647 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o 2024-04-16T19:26:21.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beutQ7JiMrJwJ7r8iBg32rntcEA= 2024-04-16T19:26:21.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beutQ7JiMrJwJ7r8iBg32rntcEA=', 'Content-Length': '0'} 2024-04-16T19:26:21.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:19 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000566b64f06fb39e42-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.649 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.649 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.649 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.650 DEBUG:boto:Path: /yrjefh-7/.o 2024-04-16T19:26:21.650 DEBUG:boto:Data: 2024-04-16T19:26:21.650 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.650 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.650 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.650 DEBUG:boto:Params: {} 2024-04-16T19:26:21.650 DEBUG:boto:Token: None 2024-04-16T19:26:21.650 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o 2024-04-16T19:26:21.650 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beutQ7JiMrJwJ7r8iBg32rntcEA= 2024-04-16T19:26:21.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beutQ7JiMrJwJ7r8iBg32rntcEA=', 'Content-Length': '0'} 2024-04-16T19:26:21.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b12dada397119eb0-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:21.653 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.653 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.653 DEBUG:boto:Method: GET 2024-04-16T19:26:21.653 DEBUG:boto:Path: /yrjefh-7/.o 2024-04-16T19:26:21.653 DEBUG:boto:Data: 2024-04-16T19:26:21.653 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.653 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.653 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.654 DEBUG:boto:Params: {} 2024-04-16T19:26:21.654 DEBUG:boto:Token: None 2024-04-16T19:26:21.654 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o 2024-04-16T19:26:21.654 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YqBfCvyoIB/rD3XmgIPDxgAEDnw= 2024-04-16T19:26:21.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YqBfCvyoIB/rD3XmgIPDxgAEDnw=', 'Content-Length': '0'} 2024-04-16T19:26:21.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:19 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000706ab22017ff98cd-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.657 DEBUG:boto:path=/yrjefh-7/.o 2024-04-16T19:26:21.657 DEBUG:boto:auth_path=/yrjefh-7/.o 2024-04-16T19:26:21.657 DEBUG:boto:Method: GET 2024-04-16T19:26:21.657 DEBUG:boto:Path: /yrjefh-7/.o 2024-04-16T19:26:21.657 DEBUG:boto:Data: 2024-04-16T19:26:21.657 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.657 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.658 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.658 DEBUG:boto:Params: {} 2024-04-16T19:26:21.658 DEBUG:boto:Token: None 2024-04-16T19:26:21.658 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o 2024-04-16T19:26:21.658 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YqBfCvyoIB/rD3XmgIPDxgAEDnw= 2024-04-16T19:26:21.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YqBfCvyoIB/rD3XmgIPDxgAEDnw=', 'Content-Length': '0'} 2024-04-16T19:26:21.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7f843289824e348-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.660 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.661 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.661 DEBUG:boto:path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.661 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.661 DEBUG:boto:Method: GET 2024-04-16T19:26:21.661 DEBUG:boto:Path: /yrjefh-7/?prefix=.o 2024-04-16T19:26:21.661 DEBUG:boto:Data: 2024-04-16T19:26:21.661 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.661 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.661 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.661 DEBUG:boto:Params: {} 2024-04-16T19:26:21.661 DEBUG:boto:Token: None 2024-04-16T19:26:21.662 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c61aac69e9f37055-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.664 DEBUG:boto:b'yrjefh-7.o1000false.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.665 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.665 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.665 DEBUG:boto:path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.665 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o 2024-04-16T19:26:21.665 DEBUG:boto:Method: GET 2024-04-16T19:26:21.665 DEBUG:boto:Path: /yrjefh-7/?prefix=.o 2024-04-16T19:26:21.665 DEBUG:boto:Data: 2024-04-16T19:26:21.665 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.665 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.665 DEBUG:boto:Params: {} 2024-04-16T19:26:21.666 DEBUG:boto:Token: None 2024-04-16T19:26:21.666 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007561c2a1f4dce979-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.668 DEBUG:boto:b'yrjefh-7.o1000false.o2024-04-16T19:26:10.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:21.669 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.669 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.669 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.669 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.669 DEBUG:boto:Method: GET 2024-04-16T19:26:21.669 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.669 DEBUG:boto:Data: 2024-04-16T19:26:21.669 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.670 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.670 DEBUG:boto:Params: {} 2024-04-16T19:26:21.670 DEBUG:boto:Token: None 2024-04-16T19:26:21.670 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU= 2024-04-16T19:26:21.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU=', 'Content-Length': '0'} 2024-04-16T19:26:21.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000733ec4fdeef4a9e0-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.673 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.673 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.673 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.673 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.673 DEBUG:boto:Method: GET 2024-04-16T19:26:21.673 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.673 DEBUG:boto:Data: 2024-04-16T19:26:21.673 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.673 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.673 DEBUG:boto:Params: {} 2024-04-16T19:26:21.673 DEBUG:boto:Token: None 2024-04-16T19:26:21.674 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.674 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU= 2024-04-16T19:26:21.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU=', 'Content-Length': '0'} 2024-04-16T19:26:21.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c66edfae189a0ee-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.676 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.676 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.676 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.676 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.676 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.677 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.677 DEBUG:boto:Data: 2024-04-16T19:26:21.677 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.677 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.677 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.677 DEBUG:boto:Params: {} 2024-04-16T19:26:21.677 DEBUG:boto:Token: None 2024-04-16T19:26:21.677 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnN5lRkBDBJhkRMdxe0GWOWBG6o= 2024-04-16T19:26:21.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnN5lRkBDBJhkRMdxe0GWOWBG6o=', 'Content-Length': '0'} 2024-04-16T19:26:21.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed1ec8ee4bbbe5cd-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.679 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.679 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.679 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.680 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.680 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.680 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.680 DEBUG:boto:Data: 2024-04-16T19:26:21.680 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.680 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.680 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.680 DEBUG:boto:Params: {} 2024-04-16T19:26:21.680 DEBUG:boto:Token: None 2024-04-16T19:26:21.680 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnN5lRkBDBJhkRMdxe0GWOWBG6o= 2024-04-16T19:26:21.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnN5lRkBDBJhkRMdxe0GWOWBG6o=', 'Content-Length': '0'} 2024-04-16T19:26:21.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b09c6533163d8e-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:21.683 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.683 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.683 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.683 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.683 DEBUG:boto:Method: GET 2024-04-16T19:26:21.683 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.683 DEBUG:boto:Data: 2024-04-16T19:26:21.683 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.683 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.683 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.684 DEBUG:boto:Params: {} 2024-04-16T19:26:21.684 DEBUG:boto:Token: None 2024-04-16T19:26:21.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU= 2024-04-16T19:26:21.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU=', 'Content-Length': '0'} 2024-04-16T19:26:21.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ebd1b83c7c039fd-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.686 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.686 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.686 DEBUG:boto:path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.686 DEBUG:boto:auth_path=/yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.686 DEBUG:boto:Method: GET 2024-04-16T19:26:21.687 DEBUG:boto:Path: /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.687 DEBUG:boto:Data: 2024-04-16T19:26:21.687 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.687 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.687 DEBUG:boto:Params: {} 2024-04-16T19:26:21.687 DEBUG:boto:Token: None 2024-04-16T19:26:21.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o.?versionId=null 2024-04-16T19:26:21.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU= 2024-04-16T19:26:21.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNJP4lKmfqp4tS0e1Nz/KHEbytU=', 'Content-Length': '0'} 2024-04-16T19:26:21.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062f45c8825fe050e-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.689 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.689 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.690 DEBUG:boto:path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.690 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.690 DEBUG:boto:Method: GET 2024-04-16T19:26:21.690 DEBUG:boto:Path: /yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.690 DEBUG:boto:Data: 2024-04-16T19:26:21.690 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.690 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.690 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.690 DEBUG:boto:Params: {} 2024-04-16T19:26:21.690 DEBUG:boto:Token: None 2024-04-16T19:26:21.691 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.691 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2df123efad4db96-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.693 DEBUG:boto:b'yrjefh-7.o.1000false.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.694 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.694 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.694 DEBUG:boto:path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.694 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.694 DEBUG:boto:Method: GET 2024-04-16T19:26:21.694 DEBUG:boto:Path: /yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.694 DEBUG:boto:Data: 2024-04-16T19:26:21.694 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.694 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.694 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.694 DEBUG:boto:Params: {} 2024-04-16T19:26:21.695 DEBUG:boto:Token: None 2024-04-16T19:26:21.695 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401f8e92af157637-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.697 DEBUG:boto:b'yrjefh-7.o.1000false.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.698 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.698 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.698 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.698 DEBUG:boto:Path: /yrjefh-7/.o. 2024-04-16T19:26:21.698 DEBUG:boto:Data: 2024-04-16T19:26:21.698 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.698 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.698 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.698 DEBUG:boto:Params: {} 2024-04-16T19:26:21.699 DEBUG:boto:Token: None 2024-04-16T19:26:21.699 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o. 2024-04-16T19:26:21.699 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MxeZc5SVLm2EY0UhJS9q9hTunco= 2024-04-16T19:26:21.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MxeZc5SVLm2EY0UhJS9q9hTunco=', 'Content-Length': '0'} 2024-04-16T19:26:21.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dcfc97dd22c083d-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.701 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.701 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.701 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.701 DEBUG:boto:Path: /yrjefh-7/.o. 2024-04-16T19:26:21.701 DEBUG:boto:Data: 2024-04-16T19:26:21.701 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.701 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.701 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.702 DEBUG:boto:Params: {} 2024-04-16T19:26:21.702 DEBUG:boto:Token: None 2024-04-16T19:26:21.702 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o. 2024-04-16T19:26:21.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MxeZc5SVLm2EY0UhJS9q9hTunco= 2024-04-16T19:26:21.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MxeZc5SVLm2EY0UhJS9q9hTunco=', 'Content-Length': '0'} 2024-04-16T19:26:21.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051030746f79ac937-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:21.704 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.704 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.704 DEBUG:boto:Method: GET 2024-04-16T19:26:21.704 DEBUG:boto:Path: /yrjefh-7/.o. 2024-04-16T19:26:21.705 DEBUG:boto:Data: 2024-04-16T19:26:21.705 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.705 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.705 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.705 DEBUG:boto:Params: {} 2024-04-16T19:26:21.705 DEBUG:boto:Token: None 2024-04-16T19:26:21.705 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o. 2024-04-16T19:26:21.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xmcjwVAIo1b8m+ykwUdHXGb8JtQ= 2024-04-16T19:26:21.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xmcjwVAIo1b8m+ykwUdHXGb8JtQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b176aab305baa0c-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.708 DEBUG:boto:path=/yrjefh-7/.o. 2024-04-16T19:26:21.708 DEBUG:boto:auth_path=/yrjefh-7/.o. 2024-04-16T19:26:21.708 DEBUG:boto:Method: GET 2024-04-16T19:26:21.708 DEBUG:boto:Path: /yrjefh-7/.o. 2024-04-16T19:26:21.708 DEBUG:boto:Data: 2024-04-16T19:26:21.708 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.708 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.708 DEBUG:boto:Params: {} 2024-04-16T19:26:21.708 DEBUG:boto:Token: None 2024-04-16T19:26:21.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/.o. 2024-04-16T19:26:21.709 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xmcjwVAIo1b8m+ykwUdHXGb8JtQ= 2024-04-16T19:26:21.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xmcjwVAIo1b8m+ykwUdHXGb8JtQ=', 'Content-Length': '0'} 2024-04-16T19:26:21.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dd447813ff556c6-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.711 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.711 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.711 DEBUG:boto:path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.711 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.711 DEBUG:boto:Method: GET 2024-04-16T19:26:21.711 DEBUG:boto:Path: /yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.711 DEBUG:boto:Data: 2024-04-16T19:26:21.711 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.712 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.712 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.712 DEBUG:boto:Params: {} 2024-04-16T19:26:21.712 DEBUG:boto:Token: None 2024-04-16T19:26:21.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc7feb37725b39ad-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.758 DEBUG:boto:b'yrjefh-7.o.1000false.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.759 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.759 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.759 DEBUG:boto:path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.759 DEBUG:boto:auth_path=/yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.759 DEBUG:boto:Method: GET 2024-04-16T19:26:21.759 DEBUG:boto:Path: /yrjefh-7/?prefix=.o. 2024-04-16T19:26:21.759 DEBUG:boto:Data: 2024-04-16T19:26:21.760 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.760 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.760 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.760 DEBUG:boto:Params: {} 2024-04-16T19:26:21.760 DEBUG:boto:Token: None 2024-04-16T19:26:21.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aadfe89df8c4d422-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.763 DEBUG:boto:b'yrjefh-7.o.1000false.o.2024-04-16T19:26:10.180Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:21.764 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.764 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.764 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.764 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.764 DEBUG:boto:Method: GET 2024-04-16T19:26:21.764 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.764 DEBUG:boto:Data: 2024-04-16T19:26:21.764 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.764 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.764 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.764 DEBUG:boto:Params: {} 2024-04-16T19:26:21.764 DEBUG:boto:Token: None 2024-04-16T19:26:21.765 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0= 2024-04-16T19:26:21.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0=', 'Content-Length': '0'} 2024-04-16T19:26:21.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f1e37389508521-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.767 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.767 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.767 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.767 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.767 DEBUG:boto:Method: GET 2024-04-16T19:26:21.767 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.768 DEBUG:boto:Data: 2024-04-16T19:26:21.768 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.768 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.768 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.768 DEBUG:boto:Params: {} 2024-04-16T19:26:21.768 DEBUG:boto:Token: None 2024-04-16T19:26:21.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0= 2024-04-16T19:26:21.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0=', 'Content-Length': '0'} 2024-04-16T19:26:21.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc20badaa116e277-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.814 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.815 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.815 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.815 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.815 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.815 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.815 DEBUG:boto:Data: 2024-04-16T19:26:21.815 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.815 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.815 DEBUG:boto:Params: {} 2024-04-16T19:26:21.815 DEBUG:boto:Token: None 2024-04-16T19:26:21.816 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yLanzkbcZ/NN9FgHLZIyk3lPIUI= 2024-04-16T19:26:21.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yLanzkbcZ/NN9FgHLZIyk3lPIUI=', 'Content-Length': '0'} 2024-04-16T19:26:21.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c22e96f378b9948e-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.818 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.818 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.818 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.818 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.818 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.818 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.818 DEBUG:boto:Data: 2024-04-16T19:26:21.818 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.818 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.819 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.819 DEBUG:boto:Params: {} 2024-04-16T19:26:21.819 DEBUG:boto:Token: None 2024-04-16T19:26:21.819 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yLanzkbcZ/NN9FgHLZIyk3lPIUI= 2024-04-16T19:26:21.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yLanzkbcZ/NN9FgHLZIyk3lPIUI=', 'Content-Length': '0'} 2024-04-16T19:26:21.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdc9594904e78dac-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:21.821 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.821 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.821 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.821 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.821 DEBUG:boto:Method: GET 2024-04-16T19:26:21.822 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.822 DEBUG:boto:Data: 2024-04-16T19:26:21.822 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.822 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.822 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.822 DEBUG:boto:Params: {} 2024-04-16T19:26:21.822 DEBUG:boto:Token: None 2024-04-16T19:26:21.822 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.822 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0= 2024-04-16T19:26:21.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0=', 'Content-Length': '0'} 2024-04-16T19:26:21.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004da8eb166321ce42-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.825 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.825 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.825 DEBUG:boto:path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.825 DEBUG:boto:auth_path=/yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.825 DEBUG:boto:Method: GET 2024-04-16T19:26:21.825 DEBUG:boto:Path: /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.825 DEBUG:boto:Data: 2024-04-16T19:26:21.825 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.826 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.826 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.826 DEBUG:boto:Params: {} 2024-04-16T19:26:21.826 DEBUG:boto:Token: None 2024-04-16T19:26:21.826 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A?versionId=null 2024-04-16T19:26:21.826 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0= 2024-04-16T19:26:21.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8fXWLKRqZ25rMsLRi+ufmxHaKI0=', 'Content-Length': '0'} 2024-04-16T19:26:21.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a08c4f1a3523750-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.828 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.828 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.828 DEBUG:boto:path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.828 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.829 DEBUG:boto:Method: GET 2024-04-16T19:26:21.829 DEBUG:boto:Path: /yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.829 DEBUG:boto:Data: 2024-04-16T19:26:21.829 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.829 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.829 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.829 DEBUG:boto:Params: {} 2024-04-16T19:26:21.829 DEBUG:boto:Token: None 2024-04-16T19:26:21.829 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.829 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001836081a25c441a5-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.832 DEBUG:boto:b'yrjefh-7:1000false:2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.832 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.832 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.833 DEBUG:boto:path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.833 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.833 DEBUG:boto:Method: GET 2024-04-16T19:26:21.833 DEBUG:boto:Path: /yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.833 DEBUG:boto:Data: 2024-04-16T19:26:21.833 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.833 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.833 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.833 DEBUG:boto:Params: {} 2024-04-16T19:26:21.833 DEBUG:boto:Token: None 2024-04-16T19:26:21.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a7412cc6b4ebde-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.836 DEBUG:boto:b'yrjefh-7:1000false:2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.837 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.837 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.837 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.837 DEBUG:boto:Path: /yrjefh-7/%3A 2024-04-16T19:26:21.837 DEBUG:boto:Data: 2024-04-16T19:26:21.837 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.837 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.837 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.837 DEBUG:boto:Params: {} 2024-04-16T19:26:21.837 DEBUG:boto:Token: None 2024-04-16T19:26:21.838 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A 2024-04-16T19:26:21.838 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TVTHTP+R9lpP/ygoWGGDawgq0EE= 2024-04-16T19:26:21.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVTHTP+R9lpP/ygoWGGDawgq0EE=', 'Content-Length': '0'} 2024-04-16T19:26:21.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f1ed071c392f886-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.840 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.840 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.840 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.840 DEBUG:boto:Path: /yrjefh-7/%3A 2024-04-16T19:26:21.840 DEBUG:boto:Data: 2024-04-16T19:26:21.840 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.840 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.840 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.841 DEBUG:boto:Params: {} 2024-04-16T19:26:21.841 DEBUG:boto:Token: None 2024-04-16T19:26:21.841 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A 2024-04-16T19:26:21.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TVTHTP+R9lpP/ygoWGGDawgq0EE= 2024-04-16T19:26:21.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVTHTP+R9lpP/ygoWGGDawgq0EE=', 'Content-Length': '0'} 2024-04-16T19:26:21.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b558ebb86c565409-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:21.843 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.843 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.843 DEBUG:boto:Method: GET 2024-04-16T19:26:21.843 DEBUG:boto:Path: /yrjefh-7/%3A 2024-04-16T19:26:21.844 DEBUG:boto:Data: 2024-04-16T19:26:21.844 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.844 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.844 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.844 DEBUG:boto:Params: {} 2024-04-16T19:26:21.844 DEBUG:boto:Token: None 2024-04-16T19:26:21.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A 2024-04-16T19:26:21.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m65s0Xj0042P00SIlv15MToHjvo= 2024-04-16T19:26:21.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m65s0Xj0042P00SIlv15MToHjvo=', 'Content-Length': '0'} 2024-04-16T19:26:21.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c30c1c3413f21b96-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.846 DEBUG:boto:path=/yrjefh-7/%3A 2024-04-16T19:26:21.847 DEBUG:boto:auth_path=/yrjefh-7/%3A 2024-04-16T19:26:21.847 DEBUG:boto:Method: GET 2024-04-16T19:26:21.847 DEBUG:boto:Path: /yrjefh-7/%3A 2024-04-16T19:26:21.847 DEBUG:boto:Data: 2024-04-16T19:26:21.847 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.847 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.847 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.847 DEBUG:boto:Params: {} 2024-04-16T19:26:21.847 DEBUG:boto:Token: None 2024-04-16T19:26:21.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/%3A 2024-04-16T19:26:21.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m65s0Xj0042P00SIlv15MToHjvo= 2024-04-16T19:26:21.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m65s0Xj0042P00SIlv15MToHjvo=', 'Content-Length': '0'} 2024-04-16T19:26:21.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6cbfd45b995d785-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.850 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.850 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.850 DEBUG:boto:path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.850 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.850 DEBUG:boto:Method: GET 2024-04-16T19:26:21.850 DEBUG:boto:Path: /yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.850 DEBUG:boto:Data: 2024-04-16T19:26:21.850 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.851 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.851 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.851 DEBUG:boto:Params: {} 2024-04-16T19:26:21.851 DEBUG:boto:Token: None 2024-04-16T19:26:21.851 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4eeab6e0eba3249-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.894 DEBUG:boto:b'yrjefh-7:1000false:2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.895 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.895 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.895 DEBUG:boto:path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.895 DEBUG:boto:auth_path=/yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.895 DEBUG:boto:Method: GET 2024-04-16T19:26:21.895 DEBUG:boto:Path: /yrjefh-7/?prefix=%3A 2024-04-16T19:26:21.896 DEBUG:boto:Data: 2024-04-16T19:26:21.896 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.896 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.896 DEBUG:boto:Params: {} 2024-04-16T19:26:21.896 DEBUG:boto:Token: None 2024-04-16T19:26:21.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/ 2024-04-16T19:26:21.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk= 2024-04-16T19:26:21.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xD4x85jbJR7BzUI4lm+7Aav0Hrk=', 'Content-Length': '0'} 2024-04-16T19:26:21.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a0c4d2fb30ff75f-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.899 DEBUG:boto:b'yrjefh-7:1000false:2024-04-16T19:26:10.045Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:21.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:21.899 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.900 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.900 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.900 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.900 DEBUG:boto:Method: GET 2024-04-16T19:26:21.900 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.900 DEBUG:boto:Data: 2024-04-16T19:26:21.900 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.900 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.900 DEBUG:boto:Params: {} 2024-04-16T19:26:21.901 DEBUG:boto:Token: None 2024-04-16T19:26:21.901 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw= 2024-04-16T19:26:21.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw=', 'Content-Length': '0'} 2024-04-16T19:26:21.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000491dfe807be8a43d-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.903 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.903 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.904 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.904 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.904 DEBUG:boto:Method: GET 2024-04-16T19:26:21.904 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.904 DEBUG:boto:Data: 2024-04-16T19:26:21.904 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.904 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.904 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.904 DEBUG:boto:Params: {} 2024-04-16T19:26:21.904 DEBUG:boto:Token: None 2024-04-16T19:26:21.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw= 2024-04-16T19:26:21.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw=', 'Content-Length': '0'} 2024-04-16T19:26:21.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000395aba61819a82cf-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.950 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.951 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.951 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.951 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.951 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.951 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.951 DEBUG:boto:Data: 2024-04-16T19:26:21.951 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.951 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.951 DEBUG:boto:Params: {} 2024-04-16T19:26:21.952 DEBUG:boto:Token: None 2024-04-16T19:26:21.952 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CtFjDTfMKoIhCMlPjW2qZFmjReY= 2024-04-16T19:26:21.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CtFjDTfMKoIhCMlPjW2qZFmjReY=', 'Content-Length': '0'} 2024-04-16T19:26:21.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee2d3500fb349bbb-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.954 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.955 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.955 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.955 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.955 DEBUG:boto:Method: HEAD 2024-04-16T19:26:21.955 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.955 DEBUG:boto:Data: 2024-04-16T19:26:21.955 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.955 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.955 DEBUG:boto:Params: {} 2024-04-16T19:26:21.955 DEBUG:boto:Token: None 2024-04-16T19:26:21.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CtFjDTfMKoIhCMlPjW2qZFmjReY= 2024-04-16T19:26:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CtFjDTfMKoIhCMlPjW2qZFmjReY=', 'Content-Length': '0'} 2024-04-16T19:26:21.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0e24c03ae532a42-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:21.958 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.959 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.959 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.959 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.959 DEBUG:boto:Method: GET 2024-04-16T19:26:21.959 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.959 DEBUG:boto:Data: 2024-04-16T19:26:21.959 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.959 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.959 DEBUG:boto:Params: {} 2024-04-16T19:26:21.960 DEBUG:boto:Token: None 2024-04-16T19:26:21.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw= 2024-04-16T19:26:21.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw=', 'Content-Length': '0'} 2024-04-16T19:26:21.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a787011a82bb322-00661ed0dd-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.962 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:21.962 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:21.962 DEBUG:boto:path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.963 DEBUG:boto:auth_path=/yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.963 DEBUG:boto:Method: GET 2024-04-16T19:26:21.963 DEBUG:boto:Path: /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.963 DEBUG:boto:Data: 2024-04-16T19:26:21.963 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:21.963 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.963 DEBUG:boto:Params: {} 2024-04-16T19:26:21.963 DEBUG:boto:Token: None 2024-04-16T19:26:21.963 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:21 GMT /yrjefh-7/_myobj?versionId=null 2024-04-16T19:26:21.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw= 2024-04-16T19:26:21.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iLg5kCNSoAf/GV4uT7fhE2Kl9xw=', 'Content-Length': '0'} 2024-04-16T19:26:21.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f756b3255cbfc92e-00661ed0dd-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:21.966 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:21.966 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:21.966 DEBUG:boto:path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:21.966 DEBUG:boto:auth_path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:21.966 DEBUG:boto:Method: GET 2024-04-16T19:26:21.966 DEBUG:boto:Path: /yrjefh-7/?prefix=_myobj 2024-04-16T19:26:21.966 DEBUG:boto:Data: 2024-04-16T19:26:21.966 DEBUG:boto:Headers: {} 2024-04-16T19:26:21.966 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:21.966 DEBUG:boto:Port: 8000 2024-04-16T19:26:21.967 DEBUG:boto:Params: {} 2024-04-16T19:26:21.967 DEBUG:boto:Token: None 2024-04-16T19:26:22.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0d5e7db937d1a86-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.016 DEBUG:boto:b'yrjefh-7_myobj1000false_myobj2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.016 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.016 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.016 DEBUG:boto:path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.016 DEBUG:boto:auth_path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.016 DEBUG:boto:Method: GET 2024-04-16T19:26:22.016 DEBUG:boto:Path: /yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.017 DEBUG:boto:Data: 2024-04-16T19:26:22.017 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.017 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.017 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.017 DEBUG:boto:Params: {} 2024-04-16T19:26:22.017 DEBUG:boto:Token: None 2024-04-16T19:26:22.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086bcebd81f4e0a2f-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.020 DEBUG:boto:b'yrjefh-7_myobj1000false_myobj2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.020 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:22.020 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:22.020 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.020 DEBUG:boto:Path: /yrjefh-7/_myobj 2024-04-16T19:26:22.020 DEBUG:boto:Data: 2024-04-16T19:26:22.021 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.021 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.021 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.021 DEBUG:boto:Params: {} 2024-04-16T19:26:22.021 DEBUG:boto:Token: None 2024-04-16T19:26:22.021 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/_myobj 2024-04-16T19:26:22.021 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:roopWsNaT6tI9uICoVzKTv7LOYU= 2024-04-16T19:26:22.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:roopWsNaT6tI9uICoVzKTv7LOYU=', 'Content-Length': '0'} 2024-04-16T19:26:22.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000935ee39827e6f4a4-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.024 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:22.024 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:22.024 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.024 DEBUG:boto:Path: /yrjefh-7/_myobj 2024-04-16T19:26:22.024 DEBUG:boto:Data: 2024-04-16T19:26:22.024 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.024 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.024 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.024 DEBUG:boto:Params: {} 2024-04-16T19:26:22.024 DEBUG:boto:Token: None 2024-04-16T19:26:22.025 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/_myobj 2024-04-16T19:26:22.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:roopWsNaT6tI9uICoVzKTv7LOYU= 2024-04-16T19:26:22.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:roopWsNaT6tI9uICoVzKTv7LOYU=', 'Content-Length': '0'} 2024-04-16T19:26:22.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347e1d4cb5739866-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:22.027 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:22.027 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:22.027 DEBUG:boto:Method: GET 2024-04-16T19:26:22.027 DEBUG:boto:Path: /yrjefh-7/_myobj 2024-04-16T19:26:22.027 DEBUG:boto:Data: 2024-04-16T19:26:22.028 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.028 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.028 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.028 DEBUG:boto:Params: {} 2024-04-16T19:26:22.028 DEBUG:boto:Token: None 2024-04-16T19:26:22.028 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/_myobj 2024-04-16T19:26:22.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Epi+iLeigvsQOg3K3iebeGKKavg= 2024-04-16T19:26:22.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Epi+iLeigvsQOg3K3iebeGKKavg=', 'Content-Length': '0'} 2024-04-16T19:26:22.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebfe965bfe8c4ecc-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.074 DEBUG:boto:path=/yrjefh-7/_myobj 2024-04-16T19:26:22.075 DEBUG:boto:auth_path=/yrjefh-7/_myobj 2024-04-16T19:26:22.075 DEBUG:boto:Method: GET 2024-04-16T19:26:22.075 DEBUG:boto:Path: /yrjefh-7/_myobj 2024-04-16T19:26:22.075 DEBUG:boto:Data: 2024-04-16T19:26:22.075 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.075 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.075 DEBUG:boto:Params: {} 2024-04-16T19:26:22.076 DEBUG:boto:Token: None 2024-04-16T19:26:22.076 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/_myobj 2024-04-16T19:26:22.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Epi+iLeigvsQOg3K3iebeGKKavg= 2024-04-16T19:26:22.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Epi+iLeigvsQOg3K3iebeGKKavg=', 'Content-Length': '0'} 2024-04-16T19:26:22.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d8886cc26725f07-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.078 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.079 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.079 DEBUG:boto:path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.079 DEBUG:boto:auth_path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.079 DEBUG:boto:Method: GET 2024-04-16T19:26:22.079 DEBUG:boto:Path: /yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.079 DEBUG:boto:Data: 2024-04-16T19:26:22.079 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.079 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.079 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.079 DEBUG:boto:Params: {} 2024-04-16T19:26:22.079 DEBUG:boto:Token: None 2024-04-16T19:26:22.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab36c9ad070c8fd1-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.082 DEBUG:boto:b'yrjefh-7_myobj1000false_myobj2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.083 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.083 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.083 DEBUG:boto:path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.083 DEBUG:boto:auth_path=/yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.083 DEBUG:boto:Method: GET 2024-04-16T19:26:22.083 DEBUG:boto:Path: /yrjefh-7/?prefix=_myobj 2024-04-16T19:26:22.083 DEBUG:boto:Data: 2024-04-16T19:26:22.083 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.083 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.084 DEBUG:boto:Params: {} 2024-04-16T19:26:22.084 DEBUG:boto:Token: None 2024-04-16T19:26:22.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000241512a6b6a33fef-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.086 DEBUG:boto:b'yrjefh-7_myobj1000false_myobj2024-04-16T19:26:10.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:22.087 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.087 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.087 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.087 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.087 DEBUG:boto:Method: GET 2024-04-16T19:26:22.087 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.087 DEBUG:boto:Data: 2024-04-16T19:26:22.087 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.087 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.087 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.088 DEBUG:boto:Params: {} 2024-04-16T19:26:22.088 DEBUG:boto:Token: None 2024-04-16T19:26:22.088 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E= 2024-04-16T19:26:22.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E=', 'Content-Length': '0'} 2024-04-16T19:26:22.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087da047a55521027-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.090 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.090 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.091 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.091 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.091 DEBUG:boto:Method: GET 2024-04-16T19:26:22.091 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.091 DEBUG:boto:Data: 2024-04-16T19:26:22.091 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.091 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.091 DEBUG:boto:Params: {} 2024-04-16T19:26:22.091 DEBUG:boto:Token: None 2024-04-16T19:26:22.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E= 2024-04-16T19:26:22.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E=', 'Content-Length': '0'} 2024-04-16T19:26:22.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ede78e2c42c8286-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.134 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.134 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.134 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.135 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.135 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.135 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.135 DEBUG:boto:Data: 2024-04-16T19:26:22.135 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.135 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.135 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.135 DEBUG:boto:Params: {} 2024-04-16T19:26:22.135 DEBUG:boto:Token: None 2024-04-16T19:26:22.135 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dEiYQGvLrc2RnIzfRd2nFIgLZNQ= 2024-04-16T19:26:22.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dEiYQGvLrc2RnIzfRd2nFIgLZNQ=', 'Content-Length': '0'} 2024-04-16T19:26:22.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d309ea1913e136-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.138 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.138 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.138 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.138 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.138 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.138 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.138 DEBUG:boto:Data: 2024-04-16T19:26:22.138 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.138 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.139 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.139 DEBUG:boto:Params: {} 2024-04-16T19:26:22.139 DEBUG:boto:Token: None 2024-04-16T19:26:22.139 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dEiYQGvLrc2RnIzfRd2nFIgLZNQ= 2024-04-16T19:26:22.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dEiYQGvLrc2RnIzfRd2nFIgLZNQ=', 'Content-Length': '0'} 2024-04-16T19:26:22.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d777008d2db1fee7-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:22.141 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.141 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.141 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.142 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.142 DEBUG:boto:Method: GET 2024-04-16T19:26:22.142 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.142 DEBUG:boto:Data: 2024-04-16T19:26:22.142 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.142 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.142 DEBUG:boto:Params: {} 2024-04-16T19:26:22.142 DEBUG:boto:Token: None 2024-04-16T19:26:22.142 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E= 2024-04-16T19:26:22.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E=', 'Content-Length': '0'} 2024-04-16T19:26:22.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b732281d9f0dd62-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.145 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.145 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.145 DEBUG:boto:path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.145 DEBUG:boto:auth_path=/yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.145 DEBUG:boto:Method: GET 2024-04-16T19:26:22.145 DEBUG:boto:Path: /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.145 DEBUG:boto:Data: 2024-04-16T19:26:22.145 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.145 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.146 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.146 DEBUG:boto:Params: {} 2024-04-16T19:26:22.146 DEBUG:boto:Token: None 2024-04-16T19:26:22.146 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj?versionId=null 2024-04-16T19:26:22.146 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E= 2024-04-16T19:26:22.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/eyHw9cP3vqL7OvMuiAhfPE2s1E=', 'Content-Length': '0'} 2024-04-16T19:26:22.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c46ab565647c7f-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.148 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.148 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.148 DEBUG:boto:path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.148 DEBUG:boto:auth_path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.148 DEBUG:boto:Method: GET 2024-04-16T19:26:22.149 DEBUG:boto:Path: /yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.149 DEBUG:boto:Data: 2024-04-16T19:26:22.149 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.149 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.149 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.149 DEBUG:boto:Params: {} 2024-04-16T19:26:22.149 DEBUG:boto:Token: None 2024-04-16T19:26:22.149 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.149 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000710444775a46ad36-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.152 DEBUG:boto:b'yrjefh-7myobj1000falsemyobj2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.152 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.152 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.152 DEBUG:boto:path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.152 DEBUG:boto:auth_path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.152 DEBUG:boto:Method: GET 2024-04-16T19:26:22.152 DEBUG:boto:Path: /yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.153 DEBUG:boto:Data: 2024-04-16T19:26:22.153 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.153 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.153 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.153 DEBUG:boto:Params: {} 2024-04-16T19:26:22.153 DEBUG:boto:Token: None 2024-04-16T19:26:22.153 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07fec593aef4314-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.155 DEBUG:boto:b'yrjefh-7myobj1000falsemyobj2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.156 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.156 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.156 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.156 DEBUG:boto:Path: /yrjefh-7/myobj 2024-04-16T19:26:22.156 DEBUG:boto:Data: 2024-04-16T19:26:22.156 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.156 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.157 DEBUG:boto:Params: {} 2024-04-16T19:26:22.157 DEBUG:boto:Token: None 2024-04-16T19:26:22.157 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj 2024-04-16T19:26:22.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jk1ANfAZ4Oe1545OcAtqyF5+Ek4= 2024-04-16T19:26:22.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jk1ANfAZ4Oe1545OcAtqyF5+Ek4=', 'Content-Length': '0'} 2024-04-16T19:26:22.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000524c81b1f07cee77-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.159 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.159 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.159 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.159 DEBUG:boto:Path: /yrjefh-7/myobj 2024-04-16T19:26:22.160 DEBUG:boto:Data: 2024-04-16T19:26:22.160 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.160 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.160 DEBUG:boto:Params: {} 2024-04-16T19:26:22.160 DEBUG:boto:Token: None 2024-04-16T19:26:22.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj 2024-04-16T19:26:22.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jk1ANfAZ4Oe1545OcAtqyF5+Ek4= 2024-04-16T19:26:22.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jk1ANfAZ4Oe1545OcAtqyF5+Ek4=', 'Content-Length': '0'} 2024-04-16T19:26:22.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b13a2c3d7f770541-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:22.162 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.163 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.163 DEBUG:boto:Method: GET 2024-04-16T19:26:22.163 DEBUG:boto:Path: /yrjefh-7/myobj 2024-04-16T19:26:22.163 DEBUG:boto:Data: 2024-04-16T19:26:22.163 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.163 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.163 DEBUG:boto:Params: {} 2024-04-16T19:26:22.163 DEBUG:boto:Token: None 2024-04-16T19:26:22.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj 2024-04-16T19:26:22.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rAjTm9mLt/wFGNdjXUix6hlO7+4= 2024-04-16T19:26:22.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rAjTm9mLt/wFGNdjXUix6hlO7+4=', 'Content-Length': '0'} 2024-04-16T19:26:22.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:14 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028e02fdc094c16d1-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.206 DEBUG:boto:path=/yrjefh-7/myobj 2024-04-16T19:26:22.206 DEBUG:boto:auth_path=/yrjefh-7/myobj 2024-04-16T19:26:22.206 DEBUG:boto:Method: GET 2024-04-16T19:26:22.207 DEBUG:boto:Path: /yrjefh-7/myobj 2024-04-16T19:26:22.207 DEBUG:boto:Data: 2024-04-16T19:26:22.207 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.207 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.207 DEBUG:boto:Params: {} 2024-04-16T19:26:22.207 DEBUG:boto:Token: None 2024-04-16T19:26:22.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/myobj 2024-04-16T19:26:22.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rAjTm9mLt/wFGNdjXUix6hlO7+4= 2024-04-16T19:26:22.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rAjTm9mLt/wFGNdjXUix6hlO7+4=', 'Content-Length': '0'} 2024-04-16T19:26:22.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3bb7523a124706e-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.210 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.210 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.210 DEBUG:boto:path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.210 DEBUG:boto:auth_path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.210 DEBUG:boto:Method: GET 2024-04-16T19:26:22.210 DEBUG:boto:Path: /yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.210 DEBUG:boto:Data: 2024-04-16T19:26:22.211 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.211 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.211 DEBUG:boto:Params: {} 2024-04-16T19:26:22.211 DEBUG:boto:Token: None 2024-04-16T19:26:22.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d0ce4a2fedd84b-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.214 DEBUG:boto:b'yrjefh-7myobj1000falsemyobj2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.214 DEBUG:boto:path=/yrjefh-7/ 2024-04-16T19:26:22.214 DEBUG:boto:auth_path=/yrjefh-7/ 2024-04-16T19:26:22.214 DEBUG:boto:path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.214 DEBUG:boto:auth_path=/yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.215 DEBUG:boto:Method: GET 2024-04-16T19:26:22.215 DEBUG:boto:Path: /yrjefh-7/?prefix=myobj 2024-04-16T19:26:22.215 DEBUG:boto:Data: 2024-04-16T19:26:22.215 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.215 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.215 DEBUG:boto:Params: {} 2024-04-16T19:26:22.215 DEBUG:boto:Token: None 2024-04-16T19:26:22.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-7/ 2024-04-16T19:26:22.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc= 2024-04-16T19:26:22.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/mCvWHSCQLwah5mmflsdJ2+UAoc=', 'Content-Length': '0'} 2024-04-16T19:26:22.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c63b5413a97629-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.218 DEBUG:boto:b'yrjefh-7myobj1000falsemyobj2024-04-16T19:26:09.984Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-7 zones={test-zone2, test-zone1} 2024-04-16T19:26:22.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-8', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:22.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-8', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:22.219 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-8 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:22.341+0000 7efcaba10800 1 bucket sync caught up with source: 2024-04-16T19:26:22.342 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.5.6, , 00000000001.10.6, , 00000000001.5.6, 00000000002.31.6, 00000000001.7.6, 00000000001.4.6, 00000000001.16.6, ] 2024-04-16T19:26:22.343 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.5.6, , 00000000001.10.6, , 00000000001.5.6, 00000000002.31.6, 00000000001.7.6, 00000000001.4.6, 00000000001.16.6, ] 2024-04-16T19:26:22.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:22.341+0000 7efcaba10800 0 bucket checkpoint complete 2024-04-16T19:26:22.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-8 zones={test-zone1, test-zone2} 2024-04-16T19:26:22.363 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.363 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.363 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.364 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:22.364 DEBUG:boto:Data: 2024-04-16T19:26:22.364 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.364 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.364 DEBUG:boto:Params: {} 2024-04-16T19:26:22.364 DEBUG:boto:Token: None 2024-04-16T19:26:22.364 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jRqSJU8uosiz8oNqwfsbZMxrWlg= 2024-04-16T19:26:22.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jRqSJU8uosiz8oNqwfsbZMxrWlg=', 'Content-Length': '0'} 2024-04-16T19:26:22.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004157bb3e482e238a-00661ed0de-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.366 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.367 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.367 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.367 DEBUG:boto:Path: /yrjefh-8/ 2024-04-16T19:26:22.367 DEBUG:boto:Data: 2024-04-16T19:26:22.367 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.367 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.367 DEBUG:boto:Params: {} 2024-04-16T19:26:22.367 DEBUG:boto:Token: None 2024-04-16T19:26:22.367 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jRqSJU8uosiz8oNqwfsbZMxrWlg= 2024-04-16T19:26:22.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jRqSJU8uosiz8oNqwfsbZMxrWlg=', 'Content-Length': '0'} 2024-04-16T19:26:22.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c7e81aa58ee1670-00661ed0de-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.370 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:22.370 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.370 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.370 DEBUG:boto:path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.370 DEBUG:boto:auth_path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.370 DEBUG:boto:Method: GET 2024-04-16T19:26:22.370 DEBUG:boto:Path: /yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.371 DEBUG:boto:Data: 2024-04-16T19:26:22.371 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.371 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.371 DEBUG:boto:Params: {} 2024-04-16T19:26:22.371 DEBUG:boto:Token: None 2024-04-16T19:26:22.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/?versions 2024-04-16T19:26:22.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg= 2024-04-16T19:26:22.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg=', 'Content-Length': '0'} 2024-04-16T19:26:22.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2c48ddee1967f7-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.374 DEBUG:boto:b'yrjefh-8999false&nulltrue2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.374 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-16T19:26:22.374 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-04-16T19:26:22.374 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-04-16T19:26:22.374 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-04-16T19:26:22.374 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-04-16T19:26:22.375 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-04-16T19:26:22.375 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-16T19:26:22.375 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-16T19:26:22.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:22.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:22.375 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.375 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.375 DEBUG:boto:path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.375 DEBUG:boto:auth_path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.375 DEBUG:boto:Method: GET 2024-04-16T19:26:22.375 DEBUG:boto:Path: /yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.375 DEBUG:boto:Data: 2024-04-16T19:26:22.376 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.376 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.376 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.376 DEBUG:boto:Params: {} 2024-04-16T19:26:22.376 DEBUG:boto:Token: None 2024-04-16T19:26:22.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/?versions 2024-04-16T19:26:22.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg= 2024-04-16T19:26:22.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg=', 'Content-Length': '0'} 2024-04-16T19:26:22.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb80e418a7fd87bf-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.379 DEBUG:boto:b'yrjefh-8999false&nulltrue2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.379 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=. 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=.. 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=... 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=.o 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=.o. 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-16T19:26:22.380 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:22.380 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.380 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.380 DEBUG:boto:path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.381 DEBUG:boto:auth_path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.381 DEBUG:boto:Method: GET 2024-04-16T19:26:22.381 DEBUG:boto:Path: /yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.381 DEBUG:boto:Data: 2024-04-16T19:26:22.381 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.381 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.381 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.381 DEBUG:boto:Params: {} 2024-04-16T19:26:22.381 DEBUG:boto:Token: None 2024-04-16T19:26:22.381 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/?versions 2024-04-16T19:26:22.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg= 2024-04-16T19:26:22.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg=', 'Content-Length': '0'} 2024-04-16T19:26:22.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a7a839b18c46353-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.383 DEBUG:boto:b'yrjefh-8999false&nulltrue2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.384 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.384 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.384 DEBUG:boto:path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.384 DEBUG:boto:auth_path=/yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.384 DEBUG:boto:Method: GET 2024-04-16T19:26:22.384 DEBUG:boto:Path: /yrjefh-8/?versions&max-keys=999 2024-04-16T19:26:22.384 DEBUG:boto:Data: 2024-04-16T19:26:22.385 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.385 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.385 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.385 DEBUG:boto:Params: {} 2024-04-16T19:26:22.385 DEBUG:boto:Token: None 2024-04-16T19:26:22.385 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/?versions 2024-04-16T19:26:22.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg= 2024-04-16T19:26:22.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nd2lLLJw6QsTrjftepbS5jf6vHg=', 'Content-Length': '0'} 2024-04-16T19:26:22.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085eb0cf6fd96794e-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.388 DEBUG:boto:b'yrjefh-8999false&nulltrue2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.nulltrue2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..nulltrue2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...nulltrue2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.onulltrue2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.nulltrue2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal:nulltrue2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal_myobjnulltrue2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalmyobjnulltrue2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:22.389 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.389 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.389 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.389 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.389 DEBUG:boto:Method: GET 2024-04-16T19:26:22.389 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.389 DEBUG:boto:Data: 2024-04-16T19:26:22.389 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.389 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.389 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.389 DEBUG:boto:Params: {} 2024-04-16T19:26:22.390 DEBUG:boto:Token: None 2024-04-16T19:26:22.390 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.390 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU= 2024-04-16T19:26:22.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU=', 'Content-Length': '0'} 2024-04-16T19:26:22.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013c35cc12eefe8b5-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.434 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.434 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.435 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.435 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.435 DEBUG:boto:Method: GET 2024-04-16T19:26:22.435 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.435 DEBUG:boto:Data: 2024-04-16T19:26:22.435 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.435 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.435 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.435 DEBUG:boto:Params: {} 2024-04-16T19:26:22.435 DEBUG:boto:Token: None 2024-04-16T19:26:22.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU= 2024-04-16T19:26:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU=', 'Content-Length': '0'} 2024-04-16T19:26:22.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bce1649befc16a0-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.438 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.438 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.438 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.439 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.439 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.439 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.439 DEBUG:boto:Data: 2024-04-16T19:26:22.439 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.439 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.439 DEBUG:boto:Params: {} 2024-04-16T19:26:22.439 DEBUG:boto:Token: None 2024-04-16T19:26:22.439 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+UrdIkp6leF4Sn4mlIDWEUJHvSI= 2024-04-16T19:26:22.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+UrdIkp6leF4Sn4mlIDWEUJHvSI=', 'Content-Length': '0'} 2024-04-16T19:26:22.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006163df0241fe28b7-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.442 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.442 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.442 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.442 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.442 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.442 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.442 DEBUG:boto:Data: 2024-04-16T19:26:22.443 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.443 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.443 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.443 DEBUG:boto:Params: {} 2024-04-16T19:26:22.443 DEBUG:boto:Token: None 2024-04-16T19:26:22.443 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+UrdIkp6leF4Sn4mlIDWEUJHvSI= 2024-04-16T19:26:22.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+UrdIkp6leF4Sn4mlIDWEUJHvSI=', 'Content-Length': '0'} 2024-04-16T19:26:22.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dd40c0a81752a5e-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:22.446 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.446 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.446 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.446 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.446 DEBUG:boto:Method: GET 2024-04-16T19:26:22.446 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.447 DEBUG:boto:Data: 2024-04-16T19:26:22.447 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.447 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.447 DEBUG:boto:Params: {} 2024-04-16T19:26:22.447 DEBUG:boto:Token: None 2024-04-16T19:26:22.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU= 2024-04-16T19:26:22.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU=', 'Content-Length': '0'} 2024-04-16T19:26:22.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd8dffdd74fe24be-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.490 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.490 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.490 DEBUG:boto:path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.491 DEBUG:boto:auth_path=/yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.491 DEBUG:boto:Method: GET 2024-04-16T19:26:22.491 DEBUG:boto:Path: /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.491 DEBUG:boto:Data: 2024-04-16T19:26:22.491 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.491 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.491 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.491 DEBUG:boto:Params: {} 2024-04-16T19:26:22.491 DEBUG:boto:Token: None 2024-04-16T19:26:22.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26?versionId=null 2024-04-16T19:26:22.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU= 2024-04-16T19:26:22.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MB2IrFnZED4JVMcf4M/+zWIcCgU=', 'Content-Length': '0'} 2024-04-16T19:26:22.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9185a61574c7dac-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.494 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.494 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.494 DEBUG:boto:path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.494 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.494 DEBUG:boto:Method: GET 2024-04-16T19:26:22.495 DEBUG:boto:Path: /yrjefh-8/?prefix=%26 2024-04-16T19:26:22.495 DEBUG:boto:Data: 2024-04-16T19:26:22.495 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.495 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.495 DEBUG:boto:Params: {} 2024-04-16T19:26:22.495 DEBUG:boto:Token: None 2024-04-16T19:26:22.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a172582a2e4a658-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.498 DEBUG:boto:b'yrjefh-8&1000false&2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.498 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.498 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.498 DEBUG:boto:path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.498 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.498 DEBUG:boto:Method: GET 2024-04-16T19:26:22.498 DEBUG:boto:Path: /yrjefh-8/?prefix=%26 2024-04-16T19:26:22.498 DEBUG:boto:Data: 2024-04-16T19:26:22.498 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.499 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.499 DEBUG:boto:Params: {} 2024-04-16T19:26:22.499 DEBUG:boto:Token: None 2024-04-16T19:26:22.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d580c1b29a9166-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.502 DEBUG:boto:b'yrjefh-8&1000false&2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.502 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.502 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.503 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.503 DEBUG:boto:Path: /yrjefh-8/%26 2024-04-16T19:26:22.503 DEBUG:boto:Data: 2024-04-16T19:26:22.503 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.503 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.503 DEBUG:boto:Params: {} 2024-04-16T19:26:22.503 DEBUG:boto:Token: None 2024-04-16T19:26:22.503 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26 2024-04-16T19:26:22.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UkOZ+PVg+Whj36Yo8b8bchqVeRE= 2024-04-16T19:26:22.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UkOZ+PVg+Whj36Yo8b8bchqVeRE=', 'Content-Length': '0'} 2024-04-16T19:26:22.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021451954f9f03558-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.506 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.506 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.506 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.506 DEBUG:boto:Path: /yrjefh-8/%26 2024-04-16T19:26:22.506 DEBUG:boto:Data: 2024-04-16T19:26:22.506 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.506 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.506 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.506 DEBUG:boto:Params: {} 2024-04-16T19:26:22.506 DEBUG:boto:Token: None 2024-04-16T19:26:22.507 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26 2024-04-16T19:26:22.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UkOZ+PVg+Whj36Yo8b8bchqVeRE= 2024-04-16T19:26:22.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UkOZ+PVg+Whj36Yo8b8bchqVeRE=', 'Content-Length': '0'} 2024-04-16T19:26:22.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9f53de21f9144d9-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-16T19:26:22.509 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.509 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.509 DEBUG:boto:Method: GET 2024-04-16T19:26:22.509 DEBUG:boto:Path: /yrjefh-8/%26 2024-04-16T19:26:22.509 DEBUG:boto:Data: 2024-04-16T19:26:22.510 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.510 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.510 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.510 DEBUG:boto:Params: {} 2024-04-16T19:26:22.510 DEBUG:boto:Token: None 2024-04-16T19:26:22.510 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26 2024-04-16T19:26:22.510 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:440qhq8l/y1409sV+7LMdxH+DCU= 2024-04-16T19:26:22.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:440qhq8l/y1409sV+7LMdxH+DCU=', 'Content-Length': '0'} 2024-04-16T19:26:22.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb069cbe72229e30-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.554 DEBUG:boto:path=/yrjefh-8/%26 2024-04-16T19:26:22.554 DEBUG:boto:auth_path=/yrjefh-8/%26 2024-04-16T19:26:22.555 DEBUG:boto:Method: GET 2024-04-16T19:26:22.555 DEBUG:boto:Path: /yrjefh-8/%26 2024-04-16T19:26:22.555 DEBUG:boto:Data: 2024-04-16T19:26:22.555 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.555 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.555 DEBUG:boto:Params: {} 2024-04-16T19:26:22.555 DEBUG:boto:Token: None 2024-04-16T19:26:22.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/%26 2024-04-16T19:26:22.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:440qhq8l/y1409sV+7LMdxH+DCU= 2024-04-16T19:26:22.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:440qhq8l/y1409sV+7LMdxH+DCU=', 'Content-Length': '0'} 2024-04-16T19:26:22.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000897f24d94e9e0a4d-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.558 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.558 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.558 DEBUG:boto:path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.558 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.558 DEBUG:boto:Method: GET 2024-04-16T19:26:22.558 DEBUG:boto:Path: /yrjefh-8/?prefix=%26 2024-04-16T19:26:22.558 DEBUG:boto:Data: 2024-04-16T19:26:22.559 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.559 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.559 DEBUG:boto:Params: {} 2024-04-16T19:26:22.559 DEBUG:boto:Token: None 2024-04-16T19:26:22.559 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6abfc2e2fdecc4-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.602 DEBUG:boto:b'yrjefh-8&1000false&2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.603 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.603 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.603 DEBUG:boto:path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.603 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%26 2024-04-16T19:26:22.603 DEBUG:boto:Method: GET 2024-04-16T19:26:22.603 DEBUG:boto:Path: /yrjefh-8/?prefix=%26 2024-04-16T19:26:22.603 DEBUG:boto:Data: 2024-04-16T19:26:22.603 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.603 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.603 DEBUG:boto:Params: {} 2024-04-16T19:26:22.604 DEBUG:boto:Token: None 2024-04-16T19:26:22.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a071b872124458b7-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.606 DEBUG:boto:b'yrjefh-8&1000false&2024-04-16T19:26:10.273Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:22.607 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.607 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.607 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.607 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.607 DEBUG:boto:Method: GET 2024-04-16T19:26:22.607 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.607 DEBUG:boto:Data: 2024-04-16T19:26:22.607 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.607 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.608 DEBUG:boto:Params: {} 2024-04-16T19:26:22.608 DEBUG:boto:Token: None 2024-04-16T19:26:22.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q= 2024-04-16T19:26:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8dfab00a52c603c-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.610 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.610 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.610 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.610 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.611 DEBUG:boto:Method: GET 2024-04-16T19:26:22.611 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.611 DEBUG:boto:Data: 2024-04-16T19:26:22.611 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.611 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.611 DEBUG:boto:Params: {} 2024-04-16T19:26:22.611 DEBUG:boto:Token: None 2024-04-16T19:26:22.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q= 2024-04-16T19:26:22.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000999d5a2aeca27a8a-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.658 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.658 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.659 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.659 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.659 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.659 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.659 DEBUG:boto:Data: 2024-04-16T19:26:22.659 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.659 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.659 DEBUG:boto:Params: {} 2024-04-16T19:26:22.659 DEBUG:boto:Token: None 2024-04-16T19:26:22.660 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7bixpPlXN+Fbu4Q2dEq0G2iGko4= 2024-04-16T19:26:22.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7bixpPlXN+Fbu4Q2dEq0G2iGko4=', 'Content-Length': '0'} 2024-04-16T19:26:22.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000085c697534ada50b-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.662 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.662 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.662 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.662 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.663 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.663 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.663 DEBUG:boto:Data: 2024-04-16T19:26:22.663 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.663 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.663 DEBUG:boto:Params: {} 2024-04-16T19:26:22.663 DEBUG:boto:Token: None 2024-04-16T19:26:22.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7bixpPlXN+Fbu4Q2dEq0G2iGko4= 2024-04-16T19:26:22.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7bixpPlXN+Fbu4Q2dEq0G2iGko4=', 'Content-Length': '0'} 2024-04-16T19:26:22.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae009698193e0f9b-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:22.666 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.666 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.666 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.666 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.666 DEBUG:boto:Method: GET 2024-04-16T19:26:22.666 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.666 DEBUG:boto:Data: 2024-04-16T19:26:22.666 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.666 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.666 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.666 DEBUG:boto:Params: {} 2024-04-16T19:26:22.666 DEBUG:boto:Token: None 2024-04-16T19:26:22.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q= 2024-04-16T19:26:22.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5ce59e505a8e275-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.669 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.669 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.669 DEBUG:boto:path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.669 DEBUG:boto:auth_path=/yrjefh-8/.?versionId=null 2024-04-16T19:26:22.669 DEBUG:boto:Method: GET 2024-04-16T19:26:22.669 DEBUG:boto:Path: /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.669 DEBUG:boto:Data: 2024-04-16T19:26:22.669 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.670 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.670 DEBUG:boto:Params: {} 2024-04-16T19:26:22.670 DEBUG:boto:Token: None 2024-04-16T19:26:22.670 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.?versionId=null 2024-04-16T19:26:22.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q= 2024-04-16T19:26:22.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Pqtaai0GvAzxQjjmxA9bM40R8Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d577052b9c25755-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.672 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.672 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.672 DEBUG:boto:path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.672 DEBUG:boto:auth_path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.672 DEBUG:boto:Method: GET 2024-04-16T19:26:22.673 DEBUG:boto:Path: /yrjefh-8/?prefix=. 2024-04-16T19:26:22.673 DEBUG:boto:Data: 2024-04-16T19:26:22.673 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.673 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.673 DEBUG:boto:Params: {} 2024-04-16T19:26:22.673 DEBUG:boto:Token: None 2024-04-16T19:26:22.673 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c050338ea1942871-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.676 DEBUG:boto:b'yrjefh-8.1000false.2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.676 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.676 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.676 DEBUG:boto:path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.676 DEBUG:boto:auth_path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.677 DEBUG:boto:Method: GET 2024-04-16T19:26:22.677 DEBUG:boto:Path: /yrjefh-8/?prefix=. 2024-04-16T19:26:22.677 DEBUG:boto:Data: 2024-04-16T19:26:22.677 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.677 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.677 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.677 DEBUG:boto:Params: {} 2024-04-16T19:26:22.677 DEBUG:boto:Token: None 2024-04-16T19:26:22.677 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d0fabafeb6e58d-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.680 DEBUG:boto:b'yrjefh-8.1000false.2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.680 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.680 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.680 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.681 DEBUG:boto:Path: /yrjefh-8/. 2024-04-16T19:26:22.681 DEBUG:boto:Data: 2024-04-16T19:26:22.681 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.681 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.681 DEBUG:boto:Params: {} 2024-04-16T19:26:22.681 DEBUG:boto:Token: None 2024-04-16T19:26:22.681 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/. 2024-04-16T19:26:22.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FInuLGNBsrBdkZuduHuFvEn9UbU= 2024-04-16T19:26:22.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FInuLGNBsrBdkZuduHuFvEn9UbU=', 'Content-Length': '0'} 2024-04-16T19:26:22.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f412a54f17ff0253-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.683 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.683 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.683 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.684 DEBUG:boto:Path: /yrjefh-8/. 2024-04-16T19:26:22.684 DEBUG:boto:Data: 2024-04-16T19:26:22.684 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.684 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.684 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.684 DEBUG:boto:Params: {} 2024-04-16T19:26:22.684 DEBUG:boto:Token: None 2024-04-16T19:26:22.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/. 2024-04-16T19:26:22.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FInuLGNBsrBdkZuduHuFvEn9UbU= 2024-04-16T19:26:22.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FInuLGNBsrBdkZuduHuFvEn9UbU=', 'Content-Length': '0'} 2024-04-16T19:26:22.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000901cf709facbe641-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=. 2024-04-16T19:26:22.687 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.687 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.687 DEBUG:boto:Method: GET 2024-04-16T19:26:22.687 DEBUG:boto:Path: /yrjefh-8/. 2024-04-16T19:26:22.687 DEBUG:boto:Data: 2024-04-16T19:26:22.687 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.687 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.687 DEBUG:boto:Params: {} 2024-04-16T19:26:22.687 DEBUG:boto:Token: None 2024-04-16T19:26:22.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/. 2024-04-16T19:26:22.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iSuQcINwxTgaGiURj3Vkoq9ADE8= 2024-04-16T19:26:22.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iSuQcINwxTgaGiURj3Vkoq9ADE8=', 'Content-Length': '0'} 2024-04-16T19:26:22.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000359d3797f2e5943a-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.730 DEBUG:boto:path=/yrjefh-8/. 2024-04-16T19:26:22.730 DEBUG:boto:auth_path=/yrjefh-8/. 2024-04-16T19:26:22.731 DEBUG:boto:Method: GET 2024-04-16T19:26:22.731 DEBUG:boto:Path: /yrjefh-8/. 2024-04-16T19:26:22.731 DEBUG:boto:Data: 2024-04-16T19:26:22.731 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.731 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.731 DEBUG:boto:Params: {} 2024-04-16T19:26:22.731 DEBUG:boto:Token: None 2024-04-16T19:26:22.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/. 2024-04-16T19:26:22.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iSuQcINwxTgaGiURj3Vkoq9ADE8= 2024-04-16T19:26:22.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iSuQcINwxTgaGiURj3Vkoq9ADE8=', 'Content-Length': '0'} 2024-04-16T19:26:22.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066c1506b2009b192-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.734 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.734 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.734 DEBUG:boto:path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.734 DEBUG:boto:auth_path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.734 DEBUG:boto:Method: GET 2024-04-16T19:26:22.734 DEBUG:boto:Path: /yrjefh-8/?prefix=. 2024-04-16T19:26:22.734 DEBUG:boto:Data: 2024-04-16T19:26:22.734 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.734 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.734 DEBUG:boto:Params: {} 2024-04-16T19:26:22.735 DEBUG:boto:Token: None 2024-04-16T19:26:22.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077942663607accc-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.737 DEBUG:boto:b'yrjefh-8.1000false.2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.737 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.737 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.738 DEBUG:boto:path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.738 DEBUG:boto:auth_path=/yrjefh-8/?prefix=. 2024-04-16T19:26:22.738 DEBUG:boto:Method: GET 2024-04-16T19:26:22.738 DEBUG:boto:Path: /yrjefh-8/?prefix=. 2024-04-16T19:26:22.738 DEBUG:boto:Data: 2024-04-16T19:26:22.738 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.738 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.738 DEBUG:boto:Params: {} 2024-04-16T19:26:22.738 DEBUG:boto:Token: None 2024-04-16T19:26:22.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000523c3d1198aaf681-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.741 DEBUG:boto:b'yrjefh-8.1000false.2024-04-16T19:26:10.321Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:22.742 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.742 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.742 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.742 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.742 DEBUG:boto:Method: GET 2024-04-16T19:26:22.742 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.742 DEBUG:boto:Data: 2024-04-16T19:26:22.742 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.742 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.742 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.742 DEBUG:boto:Params: {} 2024-04-16T19:26:22.743 DEBUG:boto:Token: None 2024-04-16T19:26:22.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I= 2024-04-16T19:26:22.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I=', 'Content-Length': '0'} 2024-04-16T19:26:22.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f43759e76b05400e-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.745 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.745 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.745 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.745 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.745 DEBUG:boto:Method: GET 2024-04-16T19:26:22.745 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.745 DEBUG:boto:Data: 2024-04-16T19:26:22.745 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.745 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.745 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.745 DEBUG:boto:Params: {} 2024-04-16T19:26:22.746 DEBUG:boto:Token: None 2024-04-16T19:26:22.746 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.746 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I= 2024-04-16T19:26:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I=', 'Content-Length': '0'} 2024-04-16T19:26:22.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b6167b4423abc01-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.748 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.748 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.748 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.748 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.748 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.748 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.748 DEBUG:boto:Data: 2024-04-16T19:26:22.748 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.748 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.748 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.748 DEBUG:boto:Params: {} 2024-04-16T19:26:22.749 DEBUG:boto:Token: None 2024-04-16T19:26:22.749 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.749 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8w+ChwcnwU2Lc/d/HZ+brZ6ePR4= 2024-04-16T19:26:22.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8w+ChwcnwU2Lc/d/HZ+brZ6ePR4=', 'Content-Length': '0'} 2024-04-16T19:26:22.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e4d12bbb2df05ab-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.751 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.751 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.751 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.751 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.751 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.751 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.751 DEBUG:boto:Data: 2024-04-16T19:26:22.751 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.752 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.752 DEBUG:boto:Params: {} 2024-04-16T19:26:22.752 DEBUG:boto:Token: None 2024-04-16T19:26:22.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8w+ChwcnwU2Lc/d/HZ+brZ6ePR4= 2024-04-16T19:26:22.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8w+ChwcnwU2Lc/d/HZ+brZ6ePR4=', 'Content-Length': '0'} 2024-04-16T19:26:22.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c11520af9febd36-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:22.754 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.754 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.754 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.754 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.754 DEBUG:boto:Method: GET 2024-04-16T19:26:22.754 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.755 DEBUG:boto:Data: 2024-04-16T19:26:22.755 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.755 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.755 DEBUG:boto:Params: {} 2024-04-16T19:26:22.755 DEBUG:boto:Token: None 2024-04-16T19:26:22.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I= 2024-04-16T19:26:22.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I=', 'Content-Length': '0'} 2024-04-16T19:26:22.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7d872d66758c5dd-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.757 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.757 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.757 DEBUG:boto:path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.758 DEBUG:boto:auth_path=/yrjefh-8/..?versionId=null 2024-04-16T19:26:22.758 DEBUG:boto:Method: GET 2024-04-16T19:26:22.758 DEBUG:boto:Path: /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.758 DEBUG:boto:Data: 2024-04-16T19:26:22.758 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.758 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.758 DEBUG:boto:Params: {} 2024-04-16T19:26:22.758 DEBUG:boto:Token: None 2024-04-16T19:26:22.758 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/..?versionId=null 2024-04-16T19:26:22.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I= 2024-04-16T19:26:22.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F4j/s/sNpbxuY7R2K423IJeo96I=', 'Content-Length': '0'} 2024-04-16T19:26:22.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d9cef8c5df38abb-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.761 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.761 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.761 DEBUG:boto:path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.761 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.761 DEBUG:boto:Method: GET 2024-04-16T19:26:22.761 DEBUG:boto:Path: /yrjefh-8/?prefix=.. 2024-04-16T19:26:22.761 DEBUG:boto:Data: 2024-04-16T19:26:22.761 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.762 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.762 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.762 DEBUG:boto:Params: {} 2024-04-16T19:26:22.762 DEBUG:boto:Token: None 2024-04-16T19:26:22.762 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e75e527a25a23a5-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.764 DEBUG:boto:b'yrjefh-8..1000false..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.765 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.765 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.765 DEBUG:boto:path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.765 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.765 DEBUG:boto:Method: GET 2024-04-16T19:26:22.765 DEBUG:boto:Path: /yrjefh-8/?prefix=.. 2024-04-16T19:26:22.765 DEBUG:boto:Data: 2024-04-16T19:26:22.765 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.766 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.766 DEBUG:boto:Params: {} 2024-04-16T19:26:22.766 DEBUG:boto:Token: None 2024-04-16T19:26:22.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb73fd46aa25ab0e-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.768 DEBUG:boto:b'yrjefh-8..1000false..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.769 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.769 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.769 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.769 DEBUG:boto:Path: /yrjefh-8/.. 2024-04-16T19:26:22.769 DEBUG:boto:Data: 2024-04-16T19:26:22.769 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.769 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.769 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.769 DEBUG:boto:Params: {} 2024-04-16T19:26:22.770 DEBUG:boto:Token: None 2024-04-16T19:26:22.770 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.. 2024-04-16T19:26:22.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TW3bMUp5APqopzugewm83/YNeD4= 2024-04-16T19:26:22.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TW3bMUp5APqopzugewm83/YNeD4=', 'Content-Length': '0'} 2024-04-16T19:26:22.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c43cdfffe03bd821-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.772 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.772 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.772 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.772 DEBUG:boto:Path: /yrjefh-8/.. 2024-04-16T19:26:22.772 DEBUG:boto:Data: 2024-04-16T19:26:22.772 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.772 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.772 DEBUG:boto:Params: {} 2024-04-16T19:26:22.772 DEBUG:boto:Token: None 2024-04-16T19:26:22.773 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.. 2024-04-16T19:26:22.773 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TW3bMUp5APqopzugewm83/YNeD4= 2024-04-16T19:26:22.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TW3bMUp5APqopzugewm83/YNeD4=', 'Content-Length': '0'} 2024-04-16T19:26:22.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab1dbce62e450931-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.. 2024-04-16T19:26:22.775 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.775 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.775 DEBUG:boto:Method: GET 2024-04-16T19:26:22.775 DEBUG:boto:Path: /yrjefh-8/.. 2024-04-16T19:26:22.775 DEBUG:boto:Data: 2024-04-16T19:26:22.775 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.775 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.776 DEBUG:boto:Params: {} 2024-04-16T19:26:22.776 DEBUG:boto:Token: None 2024-04-16T19:26:22.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.. 2024-04-16T19:26:22.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C98raiZhIyxaQjslLMUqYlcA2SQ= 2024-04-16T19:26:22.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C98raiZhIyxaQjslLMUqYlcA2SQ=', 'Content-Length': '0'} 2024-04-16T19:26:22.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707e37959aef4d58-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.778 DEBUG:boto:path=/yrjefh-8/.. 2024-04-16T19:26:22.778 DEBUG:boto:auth_path=/yrjefh-8/.. 2024-04-16T19:26:22.778 DEBUG:boto:Method: GET 2024-04-16T19:26:22.778 DEBUG:boto:Path: /yrjefh-8/.. 2024-04-16T19:26:22.778 DEBUG:boto:Data: 2024-04-16T19:26:22.778 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.778 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.778 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.778 DEBUG:boto:Params: {} 2024-04-16T19:26:22.779 DEBUG:boto:Token: None 2024-04-16T19:26:22.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.. 2024-04-16T19:26:22.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C98raiZhIyxaQjslLMUqYlcA2SQ= 2024-04-16T19:26:22.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C98raiZhIyxaQjslLMUqYlcA2SQ=', 'Content-Length': '0'} 2024-04-16T19:26:22.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a3443ffb1e7b278-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.822 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.822 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.823 DEBUG:boto:path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.823 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.823 DEBUG:boto:Method: GET 2024-04-16T19:26:22.823 DEBUG:boto:Path: /yrjefh-8/?prefix=.. 2024-04-16T19:26:22.823 DEBUG:boto:Data: 2024-04-16T19:26:22.823 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.823 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.823 DEBUG:boto:Params: {} 2024-04-16T19:26:22.823 DEBUG:boto:Token: None 2024-04-16T19:26:22.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b41d4c930ff90a-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.826 DEBUG:boto:b'yrjefh-8..1000false..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.826 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.826 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.826 DEBUG:boto:path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.826 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.. 2024-04-16T19:26:22.826 DEBUG:boto:Method: GET 2024-04-16T19:26:22.827 DEBUG:boto:Path: /yrjefh-8/?prefix=.. 2024-04-16T19:26:22.827 DEBUG:boto:Data: 2024-04-16T19:26:22.827 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.827 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.827 DEBUG:boto:Params: {} 2024-04-16T19:26:22.827 DEBUG:boto:Token: None 2024-04-16T19:26:22.827 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e66ffe39eb81527-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.830 DEBUG:boto:b'yrjefh-8..1000false..2024-04-16T19:26:10.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:22.831 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.831 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.831 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.831 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.831 DEBUG:boto:Method: GET 2024-04-16T19:26:22.831 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.831 DEBUG:boto:Data: 2024-04-16T19:26:22.831 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.832 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.832 DEBUG:boto:Params: {} 2024-04-16T19:26:22.832 DEBUG:boto:Token: None 2024-04-16T19:26:22.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg= 2024-04-16T19:26:22.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg=', 'Content-Length': '0'} 2024-04-16T19:26:22.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084711f9b2be1c732-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.835 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.835 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.835 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.835 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.835 DEBUG:boto:Method: GET 2024-04-16T19:26:22.835 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.835 DEBUG:boto:Data: 2024-04-16T19:26:22.835 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.835 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.835 DEBUG:boto:Params: {} 2024-04-16T19:26:22.836 DEBUG:boto:Token: None 2024-04-16T19:26:22.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg= 2024-04-16T19:26:22.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg=', 'Content-Length': '0'} 2024-04-16T19:26:22.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff74241ec6122620-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.839 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.839 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.839 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.839 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.839 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.839 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.839 DEBUG:boto:Data: 2024-04-16T19:26:22.839 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.839 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.839 DEBUG:boto:Params: {} 2024-04-16T19:26:22.840 DEBUG:boto:Token: None 2024-04-16T19:26:22.840 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aj40BaLgRcuYrzbG5KBlzYulEWw= 2024-04-16T19:26:22.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aj40BaLgRcuYrzbG5KBlzYulEWw=', 'Content-Length': '0'} 2024-04-16T19:26:22.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060510b50d03921ce-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.842 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.842 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.842 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.842 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.842 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.842 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.842 DEBUG:boto:Data: 2024-04-16T19:26:22.843 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.843 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.843 DEBUG:boto:Params: {} 2024-04-16T19:26:22.843 DEBUG:boto:Token: None 2024-04-16T19:26:22.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aj40BaLgRcuYrzbG5KBlzYulEWw= 2024-04-16T19:26:22.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aj40BaLgRcuYrzbG5KBlzYulEWw=', 'Content-Length': '0'} 2024-04-16T19:26:22.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e4e66b8613b8229-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:22.845 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.846 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.846 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.846 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.846 DEBUG:boto:Method: GET 2024-04-16T19:26:22.846 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.846 DEBUG:boto:Data: 2024-04-16T19:26:22.846 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.846 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.846 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.846 DEBUG:boto:Params: {} 2024-04-16T19:26:22.846 DEBUG:boto:Token: None 2024-04-16T19:26:22.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg= 2024-04-16T19:26:22.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg=', 'Content-Length': '0'} 2024-04-16T19:26:22.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7f5a98ac2c8a199-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.890 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.890 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.891 DEBUG:boto:path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.891 DEBUG:boto:auth_path=/yrjefh-8/...?versionId=null 2024-04-16T19:26:22.891 DEBUG:boto:Method: GET 2024-04-16T19:26:22.891 DEBUG:boto:Path: /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.891 DEBUG:boto:Data: 2024-04-16T19:26:22.891 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.891 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.891 DEBUG:boto:Params: {} 2024-04-16T19:26:22.891 DEBUG:boto:Token: None 2024-04-16T19:26:22.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/...?versionId=null 2024-04-16T19:26:22.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg= 2024-04-16T19:26:22.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zR8bXNIDcfr3VOmM5lq18WKZUGg=', 'Content-Length': '0'} 2024-04-16T19:26:22.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062147f80572d07d8-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.894 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.894 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.894 DEBUG:boto:path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.894 DEBUG:boto:auth_path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.894 DEBUG:boto:Method: GET 2024-04-16T19:26:22.895 DEBUG:boto:Path: /yrjefh-8/?prefix=... 2024-04-16T19:26:22.895 DEBUG:boto:Data: 2024-04-16T19:26:22.895 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.895 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.895 DEBUG:boto:Params: {} 2024-04-16T19:26:22.895 DEBUG:boto:Token: None 2024-04-16T19:26:22.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006734277d854a54e-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.897 DEBUG:boto:b'yrjefh-8...1000false...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.898 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.898 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.898 DEBUG:boto:path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.898 DEBUG:boto:auth_path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.898 DEBUG:boto:Method: GET 2024-04-16T19:26:22.898 DEBUG:boto:Path: /yrjefh-8/?prefix=... 2024-04-16T19:26:22.898 DEBUG:boto:Data: 2024-04-16T19:26:22.898 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.898 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.899 DEBUG:boto:Params: {} 2024-04-16T19:26:22.899 DEBUG:boto:Token: None 2024-04-16T19:26:22.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d52ca8837937938-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.902 DEBUG:boto:b'yrjefh-8...1000false...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.902 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.902 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.902 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.902 DEBUG:boto:Path: /yrjefh-8/... 2024-04-16T19:26:22.902 DEBUG:boto:Data: 2024-04-16T19:26:22.902 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.902 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.902 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.903 DEBUG:boto:Params: {} 2024-04-16T19:26:22.903 DEBUG:boto:Token: None 2024-04-16T19:26:22.903 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/... 2024-04-16T19:26:22.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EmZRTfm0rMqtRinfcvOP4WnspkM= 2024-04-16T19:26:22.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EmZRTfm0rMqtRinfcvOP4WnspkM=', 'Content-Length': '0'} 2024-04-16T19:26:22.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000803fa88da5a9707e-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.905 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.905 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.905 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.906 DEBUG:boto:Path: /yrjefh-8/... 2024-04-16T19:26:22.906 DEBUG:boto:Data: 2024-04-16T19:26:22.906 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.906 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.906 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.906 DEBUG:boto:Params: {} 2024-04-16T19:26:22.906 DEBUG:boto:Token: None 2024-04-16T19:26:22.906 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/... 2024-04-16T19:26:22.906 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EmZRTfm0rMqtRinfcvOP4WnspkM= 2024-04-16T19:26:22.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EmZRTfm0rMqtRinfcvOP4WnspkM=', 'Content-Length': '0'} 2024-04-16T19:26:22.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb6efc98995a6ebd-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=... 2024-04-16T19:26:22.909 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.909 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.909 DEBUG:boto:Method: GET 2024-04-16T19:26:22.909 DEBUG:boto:Path: /yrjefh-8/... 2024-04-16T19:26:22.909 DEBUG:boto:Data: 2024-04-16T19:26:22.909 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.910 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.910 DEBUG:boto:Params: {} 2024-04-16T19:26:22.910 DEBUG:boto:Token: None 2024-04-16T19:26:22.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/... 2024-04-16T19:26:22.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BBTiPf6CCoVQKFU1cLZ0w7RwLNo= 2024-04-16T19:26:22.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BBTiPf6CCoVQKFU1cLZ0w7RwLNo=', 'Content-Length': '0'} 2024-04-16T19:26:22.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082d11fd2d3be3c12-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.912 DEBUG:boto:path=/yrjefh-8/... 2024-04-16T19:26:22.912 DEBUG:boto:auth_path=/yrjefh-8/... 2024-04-16T19:26:22.912 DEBUG:boto:Method: GET 2024-04-16T19:26:22.913 DEBUG:boto:Path: /yrjefh-8/... 2024-04-16T19:26:22.913 DEBUG:boto:Data: 2024-04-16T19:26:22.913 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.913 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.913 DEBUG:boto:Params: {} 2024-04-16T19:26:22.913 DEBUG:boto:Token: None 2024-04-16T19:26:22.913 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/... 2024-04-16T19:26:22.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BBTiPf6CCoVQKFU1cLZ0w7RwLNo= 2024-04-16T19:26:22.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BBTiPf6CCoVQKFU1cLZ0w7RwLNo=', 'Content-Length': '0'} 2024-04-16T19:26:22.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003cfca51ee98759d-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.916 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.916 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.916 DEBUG:boto:path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.916 DEBUG:boto:auth_path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.916 DEBUG:boto:Method: GET 2024-04-16T19:26:22.916 DEBUG:boto:Path: /yrjefh-8/?prefix=... 2024-04-16T19:26:22.917 DEBUG:boto:Data: 2024-04-16T19:26:22.917 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.917 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.917 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.917 DEBUG:boto:Params: {} 2024-04-16T19:26:22.917 DEBUG:boto:Token: None 2024-04-16T19:26:22.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3bcf66a46cbbeb-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.919 DEBUG:boto:b'yrjefh-8...1000false...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.920 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.920 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.920 DEBUG:boto:path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.920 DEBUG:boto:auth_path=/yrjefh-8/?prefix=... 2024-04-16T19:26:22.920 DEBUG:boto:Method: GET 2024-04-16T19:26:22.920 DEBUG:boto:Path: /yrjefh-8/?prefix=... 2024-04-16T19:26:22.920 DEBUG:boto:Data: 2024-04-16T19:26:22.920 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.920 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.921 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.921 DEBUG:boto:Params: {} 2024-04-16T19:26:22.921 DEBUG:boto:Token: None 2024-04-16T19:26:22.921 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d23e02ac83ef9a-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.923 DEBUG:boto:b'yrjefh-8...1000false...2024-04-16T19:26:10.384Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:22.924 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.924 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.924 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.924 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.924 DEBUG:boto:Method: GET 2024-04-16T19:26:22.924 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.924 DEBUG:boto:Data: 2024-04-16T19:26:22.924 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.924 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.924 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.925 DEBUG:boto:Params: {} 2024-04-16T19:26:22.925 DEBUG:boto:Token: None 2024-04-16T19:26:22.925 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q= 2024-04-16T19:26:22.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032a5e5e388989582-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.927 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.927 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.927 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.927 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.927 DEBUG:boto:Method: GET 2024-04-16T19:26:22.927 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.927 DEBUG:boto:Data: 2024-04-16T19:26:22.928 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.928 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.928 DEBUG:boto:Params: {} 2024-04-16T19:26:22.928 DEBUG:boto:Token: None 2024-04-16T19:26:22.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q= 2024-04-16T19:26:22.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbce8b0b7adee83f-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.930 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.930 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.930 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.931 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.931 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.931 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.931 DEBUG:boto:Data: 2024-04-16T19:26:22.931 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.931 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.931 DEBUG:boto:Params: {} 2024-04-16T19:26:22.931 DEBUG:boto:Token: None 2024-04-16T19:26:22.931 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oiWOlaAPXky53Hci3NmrjY4zRtA= 2024-04-16T19:26:22.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oiWOlaAPXky53Hci3NmrjY4zRtA=', 'Content-Length': '0'} 2024-04-16T19:26:22.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c22d013ea8a8af3a-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.933 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.933 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.934 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.934 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.934 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.934 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.934 DEBUG:boto:Data: 2024-04-16T19:26:22.934 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.934 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.934 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.934 DEBUG:boto:Params: {} 2024-04-16T19:26:22.934 DEBUG:boto:Token: None 2024-04-16T19:26:22.935 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oiWOlaAPXky53Hci3NmrjY4zRtA= 2024-04-16T19:26:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oiWOlaAPXky53Hci3NmrjY4zRtA=', 'Content-Length': '0'} 2024-04-16T19:26:22.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb12b1cfa467a334-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:22.937 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.937 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.937 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.937 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.937 DEBUG:boto:Method: GET 2024-04-16T19:26:22.937 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.937 DEBUG:boto:Data: 2024-04-16T19:26:22.937 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.937 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.937 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.937 DEBUG:boto:Params: {} 2024-04-16T19:26:22.938 DEBUG:boto:Token: None 2024-04-16T19:26:22.938 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.938 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q= 2024-04-16T19:26:22.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c1b3c598fc2e517-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.982 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.982 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.983 DEBUG:boto:path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.983 DEBUG:boto:auth_path=/yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.983 DEBUG:boto:Method: GET 2024-04-16T19:26:22.983 DEBUG:boto:Path: /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.983 DEBUG:boto:Data: 2024-04-16T19:26:22.983 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.983 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.983 DEBUG:boto:Params: {} 2024-04-16T19:26:22.983 DEBUG:boto:Token: None 2024-04-16T19:26:22.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o?versionId=null 2024-04-16T19:26:22.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q= 2024-04-16T19:26:22.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JRZzRejcDiuIPLfZNMBzcyNmH5Q=', 'Content-Length': '0'} 2024-04-16T19:26:22.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029f9355ea859772a-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.986 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.986 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.986 DEBUG:boto:path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:22.986 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:22.986 DEBUG:boto:Method: GET 2024-04-16T19:26:22.986 DEBUG:boto:Path: /yrjefh-8/?prefix=.o 2024-04-16T19:26:22.986 DEBUG:boto:Data: 2024-04-16T19:26:22.986 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.987 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.987 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.987 DEBUG:boto:Params: {} 2024-04-16T19:26:22.987 DEBUG:boto:Token: None 2024-04-16T19:26:22.987 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc945c25ae17465-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.989 DEBUG:boto:b'yrjefh-8.o1000false.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.989 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:22.990 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:22.990 DEBUG:boto:path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:22.990 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:22.990 DEBUG:boto:Method: GET 2024-04-16T19:26:22.990 DEBUG:boto:Path: /yrjefh-8/?prefix=.o 2024-04-16T19:26:22.990 DEBUG:boto:Data: 2024-04-16T19:26:22.990 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.990 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.990 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.990 DEBUG:boto:Params: {} 2024-04-16T19:26:22.990 DEBUG:boto:Token: None 2024-04-16T19:26:22.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/ 2024-04-16T19:26:22.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0= 2024-04-16T19:26:22.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YLHUqs9Uf9FZCPfyhJBQzpjvrC0=', 'Content-Length': '0'} 2024-04-16T19:26:22.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c09422aa78925929-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.993 DEBUG:boto:b'yrjefh-8.o1000false.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:22.994 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.994 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.994 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.994 DEBUG:boto:Path: /yrjefh-8/.o 2024-04-16T19:26:22.994 DEBUG:boto:Data: 2024-04-16T19:26:22.994 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.994 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:22.994 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.994 DEBUG:boto:Params: {} 2024-04-16T19:26:22.994 DEBUG:boto:Token: None 2024-04-16T19:26:22.995 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o 2024-04-16T19:26:22.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81QhAVZwsCawJC3KLx2gDNLvRBk= 2024-04-16T19:26:22.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81QhAVZwsCawJC3KLx2gDNLvRBk=', 'Content-Length': '0'} 2024-04-16T19:26:22.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f656ecfbb08430b-00661ed0de-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:22.996 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:22.997 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:22.997 DEBUG:boto:Method: HEAD 2024-04-16T19:26:22.997 DEBUG:boto:Path: /yrjefh-8/.o 2024-04-16T19:26:22.997 DEBUG:boto:Data: 2024-04-16T19:26:22.997 DEBUG:boto:Headers: {} 2024-04-16T19:26:22.997 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:22.997 DEBUG:boto:Port: 8000 2024-04-16T19:26:22.997 DEBUG:boto:Params: {} 2024-04-16T19:26:22.997 DEBUG:boto:Token: None 2024-04-16T19:26:22.997 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:22 GMT /yrjefh-8/.o 2024-04-16T19:26:22.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81QhAVZwsCawJC3KLx2gDNLvRBk= 2024-04-16T19:26:22.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81QhAVZwsCawJC3KLx2gDNLvRBk=', 'Content-Length': '0'} 2024-04-16T19:26:22.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000633dd68d54abd9cb-00661ed0de-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o 2024-04-16T19:26:23.000 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:23.000 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:23.000 DEBUG:boto:Method: GET 2024-04-16T19:26:23.000 DEBUG:boto:Path: /yrjefh-8/.o 2024-04-16T19:26:23.000 DEBUG:boto:Data: 2024-04-16T19:26:23.000 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.001 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.001 DEBUG:boto:Params: {} 2024-04-16T19:26:23.001 DEBUG:boto:Token: None 2024-04-16T19:26:23.001 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o 2024-04-16T19:26:23.001 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gr2Lt/bpZFtc4LidDgBUs1dBj9c= 2024-04-16T19:26:23.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gr2Lt/bpZFtc4LidDgBUs1dBj9c=', 'Content-Length': '0'} 2024-04-16T19:26:23.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927d18af5bcd5308-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.003 DEBUG:boto:path=/yrjefh-8/.o 2024-04-16T19:26:23.003 DEBUG:boto:auth_path=/yrjefh-8/.o 2024-04-16T19:26:23.004 DEBUG:boto:Method: GET 2024-04-16T19:26:23.004 DEBUG:boto:Path: /yrjefh-8/.o 2024-04-16T19:26:23.004 DEBUG:boto:Data: 2024-04-16T19:26:23.004 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.004 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.004 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.004 DEBUG:boto:Params: {} 2024-04-16T19:26:23.004 DEBUG:boto:Token: None 2024-04-16T19:26:23.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o 2024-04-16T19:26:23.005 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gr2Lt/bpZFtc4LidDgBUs1dBj9c= 2024-04-16T19:26:23.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gr2Lt/bpZFtc4LidDgBUs1dBj9c=', 'Content-Length': '0'} 2024-04-16T19:26:23.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c39d96302f6b5303-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.052 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.052 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.053 DEBUG:boto:path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:23.053 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:23.053 DEBUG:boto:Method: GET 2024-04-16T19:26:23.053 DEBUG:boto:Path: /yrjefh-8/?prefix=.o 2024-04-16T19:26:23.053 DEBUG:boto:Data: 2024-04-16T19:26:23.053 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.053 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.053 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.053 DEBUG:boto:Params: {} 2024-04-16T19:26:23.054 DEBUG:boto:Token: None 2024-04-16T19:26:23.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aca63880f701ce9-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.058 DEBUG:boto:b'yrjefh-8.o1000false.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.058 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.059 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.059 DEBUG:boto:path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:23.059 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o 2024-04-16T19:26:23.059 DEBUG:boto:Method: GET 2024-04-16T19:26:23.059 DEBUG:boto:Path: /yrjefh-8/?prefix=.o 2024-04-16T19:26:23.059 DEBUG:boto:Data: 2024-04-16T19:26:23.059 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.059 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.059 DEBUG:boto:Params: {} 2024-04-16T19:26:23.060 DEBUG:boto:Token: None 2024-04-16T19:26:23.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d3ce71afe07bab-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.065 DEBUG:boto:b'yrjefh-8.o1000false.o2024-04-16T19:26:10.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:23.066 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.066 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.066 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.066 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.066 DEBUG:boto:Method: GET 2024-04-16T19:26:23.066 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.067 DEBUG:boto:Data: 2024-04-16T19:26:23.067 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.067 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.067 DEBUG:boto:Params: {} 2024-04-16T19:26:23.067 DEBUG:boto:Token: None 2024-04-16T19:26:23.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM= 2024-04-16T19:26:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM=', 'Content-Length': '0'} 2024-04-16T19:26:23.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2f5e32eee572b48-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.072 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.072 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.072 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.072 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.072 DEBUG:boto:Method: GET 2024-04-16T19:26:23.072 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.072 DEBUG:boto:Data: 2024-04-16T19:26:23.072 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.072 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.072 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.073 DEBUG:boto:Params: {} 2024-04-16T19:26:23.073 DEBUG:boto:Token: None 2024-04-16T19:26:23.073 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM= 2024-04-16T19:26:23.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM=', 'Content-Length': '0'} 2024-04-16T19:26:23.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000282ce14714412007-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.077 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.077 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.077 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.077 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.077 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.078 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.078 DEBUG:boto:Data: 2024-04-16T19:26:23.078 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.078 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.078 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.078 DEBUG:boto:Params: {} 2024-04-16T19:26:23.078 DEBUG:boto:Token: None 2024-04-16T19:26:23.078 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.078 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVtTY8bvIOC54IZbQHofLfFVWxA= 2024-04-16T19:26:23.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVtTY8bvIOC54IZbQHofLfFVWxA=', 'Content-Length': '0'} 2024-04-16T19:26:23.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d43b3cf9a97d6fb-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.081 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.081 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.081 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.081 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.081 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.081 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.082 DEBUG:boto:Data: 2024-04-16T19:26:23.082 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.082 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.082 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.082 DEBUG:boto:Params: {} 2024-04-16T19:26:23.082 DEBUG:boto:Token: None 2024-04-16T19:26:23.082 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVtTY8bvIOC54IZbQHofLfFVWxA= 2024-04-16T19:26:23.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVtTY8bvIOC54IZbQHofLfFVWxA=', 'Content-Length': '0'} 2024-04-16T19:26:23.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbbaac834a40fcdb-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:23.088 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.088 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.088 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.089 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.089 DEBUG:boto:Method: GET 2024-04-16T19:26:23.089 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.089 DEBUG:boto:Data: 2024-04-16T19:26:23.089 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.089 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.089 DEBUG:boto:Params: {} 2024-04-16T19:26:23.089 DEBUG:boto:Token: None 2024-04-16T19:26:23.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM= 2024-04-16T19:26:23.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM=', 'Content-Length': '0'} 2024-04-16T19:26:23.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb374e3c6b0966ce-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.093 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.093 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.093 DEBUG:boto:path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.093 DEBUG:boto:auth_path=/yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.093 DEBUG:boto:Method: GET 2024-04-16T19:26:23.093 DEBUG:boto:Path: /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.094 DEBUG:boto:Data: 2024-04-16T19:26:23.094 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.094 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.094 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.094 DEBUG:boto:Params: {} 2024-04-16T19:26:23.094 DEBUG:boto:Token: None 2024-04-16T19:26:23.094 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o.?versionId=null 2024-04-16T19:26:23.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM= 2024-04-16T19:26:23.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WQEPn1wwXpCoZqMBNc44T/oSqTM=', 'Content-Length': '0'} 2024-04-16T19:26:23.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c85c8d1346f4de7f-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.140 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.141 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.141 DEBUG:boto:path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.141 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.141 DEBUG:boto:Method: GET 2024-04-16T19:26:23.141 DEBUG:boto:Path: /yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.141 DEBUG:boto:Data: 2024-04-16T19:26:23.141 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.141 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.141 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.141 DEBUG:boto:Params: {} 2024-04-16T19:26:23.142 DEBUG:boto:Token: None 2024-04-16T19:26:23.142 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007459e1b4a07c5b43-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.146 DEBUG:boto:b'yrjefh-8.o.1000false.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.146 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.146 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.146 DEBUG:boto:path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.146 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.147 DEBUG:boto:Method: GET 2024-04-16T19:26:23.147 DEBUG:boto:Path: /yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.147 DEBUG:boto:Data: 2024-04-16T19:26:23.147 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.147 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.147 DEBUG:boto:Params: {} 2024-04-16T19:26:23.147 DEBUG:boto:Token: None 2024-04-16T19:26:23.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f13b8b31b3745a62-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.152 DEBUG:boto:b'yrjefh-8.o.1000false.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.152 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.152 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.153 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.153 DEBUG:boto:Path: /yrjefh-8/.o. 2024-04-16T19:26:23.153 DEBUG:boto:Data: 2024-04-16T19:26:23.153 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.153 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.153 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.153 DEBUG:boto:Params: {} 2024-04-16T19:26:23.153 DEBUG:boto:Token: None 2024-04-16T19:26:23.153 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o. 2024-04-16T19:26:23.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xu91Y1O73vUuMBLzMdB7p1m/+3Q= 2024-04-16T19:26:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xu91Y1O73vUuMBLzMdB7p1m/+3Q=', 'Content-Length': '0'} 2024-04-16T19:26:23.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34ccd50a1475677-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.156 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.156 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.156 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.156 DEBUG:boto:Path: /yrjefh-8/.o. 2024-04-16T19:26:23.157 DEBUG:boto:Data: 2024-04-16T19:26:23.157 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.157 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.157 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.157 DEBUG:boto:Params: {} 2024-04-16T19:26:23.157 DEBUG:boto:Token: None 2024-04-16T19:26:23.157 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o. 2024-04-16T19:26:23.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xu91Y1O73vUuMBLzMdB7p1m/+3Q= 2024-04-16T19:26:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xu91Y1O73vUuMBLzMdB7p1m/+3Q=', 'Content-Length': '0'} 2024-04-16T19:26:23.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7850e4202c49e19-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=.o. 2024-04-16T19:26:23.160 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.160 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.160 DEBUG:boto:Method: GET 2024-04-16T19:26:23.160 DEBUG:boto:Path: /yrjefh-8/.o. 2024-04-16T19:26:23.160 DEBUG:boto:Data: 2024-04-16T19:26:23.161 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.161 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.161 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.161 DEBUG:boto:Params: {} 2024-04-16T19:26:23.161 DEBUG:boto:Token: None 2024-04-16T19:26:23.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o. 2024-04-16T19:26:23.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e50+tEYq53K99TgwoJsbkbFCsz8= 2024-04-16T19:26:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e50+tEYq53K99TgwoJsbkbFCsz8=', 'Content-Length': '0'} 2024-04-16T19:26:23.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb23a73df6e672c7-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.209 DEBUG:boto:path=/yrjefh-8/.o. 2024-04-16T19:26:23.209 DEBUG:boto:auth_path=/yrjefh-8/.o. 2024-04-16T19:26:23.209 DEBUG:boto:Method: GET 2024-04-16T19:26:23.209 DEBUG:boto:Path: /yrjefh-8/.o. 2024-04-16T19:26:23.209 DEBUG:boto:Data: 2024-04-16T19:26:23.209 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.210 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.210 DEBUG:boto:Params: {} 2024-04-16T19:26:23.210 DEBUG:boto:Token: None 2024-04-16T19:26:23.210 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/.o. 2024-04-16T19:26:23.210 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e50+tEYq53K99TgwoJsbkbFCsz8= 2024-04-16T19:26:23.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e50+tEYq53K99TgwoJsbkbFCsz8=', 'Content-Length': '0'} 2024-04-16T19:26:23.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cb260d3131d9d51-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.215 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.215 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.215 DEBUG:boto:path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.215 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.215 DEBUG:boto:Method: GET 2024-04-16T19:26:23.215 DEBUG:boto:Path: /yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.216 DEBUG:boto:Data: 2024-04-16T19:26:23.216 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.216 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.216 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.216 DEBUG:boto:Params: {} 2024-04-16T19:26:23.216 DEBUG:boto:Token: None 2024-04-16T19:26:23.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e5645ea6ab9a33d-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.219 DEBUG:boto:b'yrjefh-8.o.1000false.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.220 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.220 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.220 DEBUG:boto:path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.220 DEBUG:boto:auth_path=/yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.220 DEBUG:boto:Method: GET 2024-04-16T19:26:23.220 DEBUG:boto:Path: /yrjefh-8/?prefix=.o. 2024-04-16T19:26:23.220 DEBUG:boto:Data: 2024-04-16T19:26:23.220 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.220 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.221 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.221 DEBUG:boto:Params: {} 2024-04-16T19:26:23.221 DEBUG:boto:Token: None 2024-04-16T19:26:23.221 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009321494c9023b81-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.271 DEBUG:boto:b'yrjefh-8.o.1000false.o.2024-04-16T19:26:10.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:23.272 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.272 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.272 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.272 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.273 DEBUG:boto:Method: GET 2024-04-16T19:26:23.273 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.273 DEBUG:boto:Data: 2024-04-16T19:26:23.273 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.273 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.273 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.273 DEBUG:boto:Params: {} 2024-04-16T19:26:23.273 DEBUG:boto:Token: None 2024-04-16T19:26:23.273 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo= 2024-04-16T19:26:23.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo=', 'Content-Length': '0'} 2024-04-16T19:26:23.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5e5301c5be14581-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.276 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.276 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.276 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.276 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.277 DEBUG:boto:Method: GET 2024-04-16T19:26:23.277 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.277 DEBUG:boto:Data: 2024-04-16T19:26:23.277 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.277 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.277 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.277 DEBUG:boto:Params: {} 2024-04-16T19:26:23.277 DEBUG:boto:Token: None 2024-04-16T19:26:23.277 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.278 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo= 2024-04-16T19:26:23.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo=', 'Content-Length': '0'} 2024-04-16T19:26:23.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2793adf802f1d8b-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.280 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.280 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.280 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.280 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.280 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.281 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.281 DEBUG:boto:Data: 2024-04-16T19:26:23.281 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.281 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.281 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.281 DEBUG:boto:Params: {} 2024-04-16T19:26:23.281 DEBUG:boto:Token: None 2024-04-16T19:26:23.281 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GbUEJSKf01HBJpMwxmj72sYp61M= 2024-04-16T19:26:23.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GbUEJSKf01HBJpMwxmj72sYp61M=', 'Content-Length': '0'} 2024-04-16T19:26:23.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf05383c034c5a2e-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.285 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.285 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.285 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.286 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.286 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.286 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.286 DEBUG:boto:Data: 2024-04-16T19:26:23.286 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.286 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.286 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.286 DEBUG:boto:Params: {} 2024-04-16T19:26:23.287 DEBUG:boto:Token: None 2024-04-16T19:26:23.287 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GbUEJSKf01HBJpMwxmj72sYp61M= 2024-04-16T19:26:23.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GbUEJSKf01HBJpMwxmj72sYp61M=', 'Content-Length': '0'} 2024-04-16T19:26:23.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ed12e43f362dd7-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:23.289 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.289 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.290 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.290 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.290 DEBUG:boto:Method: GET 2024-04-16T19:26:23.290 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.290 DEBUG:boto:Data: 2024-04-16T19:26:23.290 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.290 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.290 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.290 DEBUG:boto:Params: {} 2024-04-16T19:26:23.290 DEBUG:boto:Token: None 2024-04-16T19:26:23.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo= 2024-04-16T19:26:23.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo=', 'Content-Length': '0'} 2024-04-16T19:26:23.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4757db9094f6ce4-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.293 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.293 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.293 DEBUG:boto:path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.293 DEBUG:boto:auth_path=/yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.294 DEBUG:boto:Method: GET 2024-04-16T19:26:23.294 DEBUG:boto:Path: /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.294 DEBUG:boto:Data: 2024-04-16T19:26:23.294 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.294 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.294 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.294 DEBUG:boto:Params: {} 2024-04-16T19:26:23.294 DEBUG:boto:Token: None 2024-04-16T19:26:23.294 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A?versionId=null 2024-04-16T19:26:23.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo= 2024-04-16T19:26:23.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1Vny5+iYtuhtRo1N+Ch1VwqsTo=', 'Content-Length': '0'} 2024-04-16T19:26:23.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb0e8df9e4dfbb82-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.297 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.298 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.298 DEBUG:boto:path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.298 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.298 DEBUG:boto:Method: GET 2024-04-16T19:26:23.298 DEBUG:boto:Path: /yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.298 DEBUG:boto:Data: 2024-04-16T19:26:23.298 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.298 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.298 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.298 DEBUG:boto:Params: {} 2024-04-16T19:26:23.299 DEBUG:boto:Token: None 2024-04-16T19:26:23.299 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f347fa6827fcd58-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.302 DEBUG:boto:b'yrjefh-8:1000false:2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.306 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.306 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.307 DEBUG:boto:path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.307 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.307 DEBUG:boto:Method: GET 2024-04-16T19:26:23.307 DEBUG:boto:Path: /yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.307 DEBUG:boto:Data: 2024-04-16T19:26:23.307 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.307 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.307 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.307 DEBUG:boto:Params: {} 2024-04-16T19:26:23.308 DEBUG:boto:Token: None 2024-04-16T19:26:23.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c443c5f169eb0c0-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.312 DEBUG:boto:b'yrjefh-8:1000false:2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.312 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.313 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.313 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.313 DEBUG:boto:Path: /yrjefh-8/%3A 2024-04-16T19:26:23.313 DEBUG:boto:Data: 2024-04-16T19:26:23.313 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.313 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.314 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.314 DEBUG:boto:Params: {} 2024-04-16T19:26:23.314 DEBUG:boto:Token: None 2024-04-16T19:26:23.314 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A 2024-04-16T19:26:23.314 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LRYYvo1BeNwxZQV5SeXJ/cJRBZs= 2024-04-16T19:26:23.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LRYYvo1BeNwxZQV5SeXJ/cJRBZs=', 'Content-Length': '0'} 2024-04-16T19:26:23.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014b3ba0674257043-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.317 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.317 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.317 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.318 DEBUG:boto:Path: /yrjefh-8/%3A 2024-04-16T19:26:23.318 DEBUG:boto:Data: 2024-04-16T19:26:23.318 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.318 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.318 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.318 DEBUG:boto:Params: {} 2024-04-16T19:26:23.318 DEBUG:boto:Token: None 2024-04-16T19:26:23.319 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A 2024-04-16T19:26:23.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LRYYvo1BeNwxZQV5SeXJ/cJRBZs= 2024-04-16T19:26:23.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LRYYvo1BeNwxZQV5SeXJ/cJRBZs=', 'Content-Length': '0'} 2024-04-16T19:26:23.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1dc21370ea38286-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-16T19:26:23.322 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.322 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.322 DEBUG:boto:Method: GET 2024-04-16T19:26:23.322 DEBUG:boto:Path: /yrjefh-8/%3A 2024-04-16T19:26:23.322 DEBUG:boto:Data: 2024-04-16T19:26:23.322 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.322 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.323 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.323 DEBUG:boto:Params: {} 2024-04-16T19:26:23.323 DEBUG:boto:Token: None 2024-04-16T19:26:23.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A 2024-04-16T19:26:23.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMTPynwWa3zCC69KbJ5dfLslm8= 2024-04-16T19:26:23.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMTPynwWa3zCC69KbJ5dfLslm8=', 'Content-Length': '0'} 2024-04-16T19:26:23.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000760cb35e1f76b480-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.326 DEBUG:boto:path=/yrjefh-8/%3A 2024-04-16T19:26:23.326 DEBUG:boto:auth_path=/yrjefh-8/%3A 2024-04-16T19:26:23.326 DEBUG:boto:Method: GET 2024-04-16T19:26:23.326 DEBUG:boto:Path: /yrjefh-8/%3A 2024-04-16T19:26:23.326 DEBUG:boto:Data: 2024-04-16T19:26:23.326 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.326 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.327 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.327 DEBUG:boto:Params: {} 2024-04-16T19:26:23.327 DEBUG:boto:Token: None 2024-04-16T19:26:23.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/%3A 2024-04-16T19:26:23.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMTPynwWa3zCC69KbJ5dfLslm8= 2024-04-16T19:26:23.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMTPynwWa3zCC69KbJ5dfLslm8=', 'Content-Length': '0'} 2024-04-16T19:26:23.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2c143e7f19fcb8c-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.330 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.330 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.330 DEBUG:boto:path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.330 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.330 DEBUG:boto:Method: GET 2024-04-16T19:26:23.330 DEBUG:boto:Path: /yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.330 DEBUG:boto:Data: 2024-04-16T19:26:23.331 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.331 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.331 DEBUG:boto:Params: {} 2024-04-16T19:26:23.331 DEBUG:boto:Token: None 2024-04-16T19:26:23.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fc4af001492ec5-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.334 DEBUG:boto:b'yrjefh-8:1000false:2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.334 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.335 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.335 DEBUG:boto:path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.335 DEBUG:boto:auth_path=/yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.335 DEBUG:boto:Method: GET 2024-04-16T19:26:23.335 DEBUG:boto:Path: /yrjefh-8/?prefix=%3A 2024-04-16T19:26:23.335 DEBUG:boto:Data: 2024-04-16T19:26:23.335 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.335 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.335 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.336 DEBUG:boto:Params: {} 2024-04-16T19:26:23.336 DEBUG:boto:Token: None 2024-04-16T19:26:23.336 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003950cf1d6b58b51d-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.340 DEBUG:boto:b'yrjefh-8:1000false:2024-04-16T19:26:10.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:23.340 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.340 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.340 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.341 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.341 DEBUG:boto:Method: GET 2024-04-16T19:26:23.341 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.341 DEBUG:boto:Data: 2024-04-16T19:26:23.341 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.341 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.341 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.341 DEBUG:boto:Params: {} 2024-04-16T19:26:23.341 DEBUG:boto:Token: None 2024-04-16T19:26:23.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA= 2024-04-16T19:26:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA=', 'Content-Length': '0'} 2024-04-16T19:26:23.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072b895799dda1f89-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.346 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.347 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.347 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.347 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.347 DEBUG:boto:Method: GET 2024-04-16T19:26:23.347 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.347 DEBUG:boto:Data: 2024-04-16T19:26:23.347 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.347 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.347 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.347 DEBUG:boto:Params: {} 2024-04-16T19:26:23.347 DEBUG:boto:Token: None 2024-04-16T19:26:23.348 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA= 2024-04-16T19:26:23.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA=', 'Content-Length': '0'} 2024-04-16T19:26:23.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa86b17c57c5d699-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.395 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.395 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.395 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.395 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.396 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.396 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.396 DEBUG:boto:Data: 2024-04-16T19:26:23.396 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.396 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.396 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.396 DEBUG:boto:Params: {} 2024-04-16T19:26:23.396 DEBUG:boto:Token: None 2024-04-16T19:26:23.396 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.397 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tCT4ngUNKSYeP9qRGzpXOOYpa4I= 2024-04-16T19:26:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tCT4ngUNKSYeP9qRGzpXOOYpa4I=', 'Content-Length': '0'} 2024-04-16T19:26:23.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009901bdc51e9f5414-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.399 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.399 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.399 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.399 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.399 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.399 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.400 DEBUG:boto:Data: 2024-04-16T19:26:23.400 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.400 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.400 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.400 DEBUG:boto:Params: {} 2024-04-16T19:26:23.400 DEBUG:boto:Token: None 2024-04-16T19:26:23.400 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tCT4ngUNKSYeP9qRGzpXOOYpa4I= 2024-04-16T19:26:23.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tCT4ngUNKSYeP9qRGzpXOOYpa4I=', 'Content-Length': '0'} 2024-04-16T19:26:23.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d0ad0084d95d194-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:23.403 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.403 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.403 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.403 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.403 DEBUG:boto:Method: GET 2024-04-16T19:26:23.404 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.404 DEBUG:boto:Data: 2024-04-16T19:26:23.404 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.404 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.404 DEBUG:boto:Params: {} 2024-04-16T19:26:23.404 DEBUG:boto:Token: None 2024-04-16T19:26:23.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA= 2024-04-16T19:26:23.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA=', 'Content-Length': '0'} 2024-04-16T19:26:23.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000033d372fc2ef8733-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.450 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.451 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.451 DEBUG:boto:path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.451 DEBUG:boto:auth_path=/yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.451 DEBUG:boto:Method: GET 2024-04-16T19:26:23.451 DEBUG:boto:Path: /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.451 DEBUG:boto:Data: 2024-04-16T19:26:23.451 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.451 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.451 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.451 DEBUG:boto:Params: {} 2024-04-16T19:26:23.452 DEBUG:boto:Token: None 2024-04-16T19:26:23.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj?versionId=null 2024-04-16T19:26:23.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA= 2024-04-16T19:26:23.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BIzvtfef4jhR9JHlctiqfcTKXNA=', 'Content-Length': '0'} 2024-04-16T19:26:23.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000610cd6209f4a32af-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.454 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.454 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.454 DEBUG:boto:path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.455 DEBUG:boto:auth_path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.455 DEBUG:boto:Method: GET 2024-04-16T19:26:23.455 DEBUG:boto:Path: /yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.455 DEBUG:boto:Data: 2024-04-16T19:26:23.455 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.455 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.455 DEBUG:boto:Params: {} 2024-04-16T19:26:23.455 DEBUG:boto:Token: None 2024-04-16T19:26:23.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efce761ef823ca13-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.458 DEBUG:boto:b'yrjefh-8_myobj1000false_myobj2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.458 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.458 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.458 DEBUG:boto:path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.458 DEBUG:boto:auth_path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.458 DEBUG:boto:Method: GET 2024-04-16T19:26:23.458 DEBUG:boto:Path: /yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.459 DEBUG:boto:Data: 2024-04-16T19:26:23.459 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.459 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.459 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.459 DEBUG:boto:Params: {} 2024-04-16T19:26:23.459 DEBUG:boto:Token: None 2024-04-16T19:26:23.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004851cc6ec342b29-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.461 DEBUG:boto:b'yrjefh-8_myobj1000false_myobj2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.462 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.462 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.462 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.462 DEBUG:boto:Path: /yrjefh-8/_myobj 2024-04-16T19:26:23.462 DEBUG:boto:Data: 2024-04-16T19:26:23.462 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.462 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.462 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.462 DEBUG:boto:Params: {} 2024-04-16T19:26:23.462 DEBUG:boto:Token: None 2024-04-16T19:26:23.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj 2024-04-16T19:26:23.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uWM8oDaiRyBjMLxPLs4pk+yrkXU= 2024-04-16T19:26:23.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uWM8oDaiRyBjMLxPLs4pk+yrkXU=', 'Content-Length': '0'} 2024-04-16T19:26:23.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005257f1f9ce403b53-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.464 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.465 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.465 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.465 DEBUG:boto:Path: /yrjefh-8/_myobj 2024-04-16T19:26:23.465 DEBUG:boto:Data: 2024-04-16T19:26:23.465 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.465 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.465 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.465 DEBUG:boto:Params: {} 2024-04-16T19:26:23.465 DEBUG:boto:Token: None 2024-04-16T19:26:23.465 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj 2024-04-16T19:26:23.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uWM8oDaiRyBjMLxPLs4pk+yrkXU= 2024-04-16T19:26:23.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uWM8oDaiRyBjMLxPLs4pk+yrkXU=', 'Content-Length': '0'} 2024-04-16T19:26:23.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b23b0b7bbc0b1ae-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-16T19:26:23.468 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.468 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.468 DEBUG:boto:Method: GET 2024-04-16T19:26:23.468 DEBUG:boto:Path: /yrjefh-8/_myobj 2024-04-16T19:26:23.468 DEBUG:boto:Data: 2024-04-16T19:26:23.468 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.468 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.468 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.468 DEBUG:boto:Params: {} 2024-04-16T19:26:23.468 DEBUG:boto:Token: None 2024-04-16T19:26:23.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj 2024-04-16T19:26:23.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xi1H1riTBeLlJ/6A3jE+os1dd6c= 2024-04-16T19:26:23.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xi1H1riTBeLlJ/6A3jE+os1dd6c=', 'Content-Length': '0'} 2024-04-16T19:26:23.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2d58eecd9e210ab-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.471 DEBUG:boto:path=/yrjefh-8/_myobj 2024-04-16T19:26:23.471 DEBUG:boto:auth_path=/yrjefh-8/_myobj 2024-04-16T19:26:23.471 DEBUG:boto:Method: GET 2024-04-16T19:26:23.471 DEBUG:boto:Path: /yrjefh-8/_myobj 2024-04-16T19:26:23.471 DEBUG:boto:Data: 2024-04-16T19:26:23.471 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.471 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.471 DEBUG:boto:Params: {} 2024-04-16T19:26:23.471 DEBUG:boto:Token: None 2024-04-16T19:26:23.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/_myobj 2024-04-16T19:26:23.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xi1H1riTBeLlJ/6A3jE+os1dd6c= 2024-04-16T19:26:23.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xi1H1riTBeLlJ/6A3jE+os1dd6c=', 'Content-Length': '0'} 2024-04-16T19:26:23.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d40831edfbeea2c6-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.474 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.474 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.474 DEBUG:boto:path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.474 DEBUG:boto:auth_path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.474 DEBUG:boto:Method: GET 2024-04-16T19:26:23.474 DEBUG:boto:Path: /yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.475 DEBUG:boto:Data: 2024-04-16T19:26:23.475 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.475 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.475 DEBUG:boto:Params: {} 2024-04-16T19:26:23.475 DEBUG:boto:Token: None 2024-04-16T19:26:23.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ad2215327cb67bd-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.477 DEBUG:boto:b'yrjefh-8_myobj1000false_myobj2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.478 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.478 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.478 DEBUG:boto:path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.478 DEBUG:boto:auth_path=/yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.478 DEBUG:boto:Method: GET 2024-04-16T19:26:23.478 DEBUG:boto:Path: /yrjefh-8/?prefix=_myobj 2024-04-16T19:26:23.478 DEBUG:boto:Data: 2024-04-16T19:26:23.478 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.478 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.478 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.478 DEBUG:boto:Params: {} 2024-04-16T19:26:23.479 DEBUG:boto:Token: None 2024-04-16T19:26:23.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f353b7d4ad48aca6-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.481 DEBUG:boto:b'yrjefh-8_myobj1000false_myobj2024-04-16T19:26:10.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:23.482 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.482 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.482 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.482 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.482 DEBUG:boto:Method: GET 2024-04-16T19:26:23.482 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.482 DEBUG:boto:Data: 2024-04-16T19:26:23.483 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.483 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.483 DEBUG:boto:Params: {} 2024-04-16T19:26:23.483 DEBUG:boto:Token: None 2024-04-16T19:26:23.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU= 2024-04-16T19:26:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU=', 'Content-Length': '0'} 2024-04-16T19:26:23.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cca602a59b5ad6be-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.485 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.486 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.486 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.486 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.486 DEBUG:boto:Method: GET 2024-04-16T19:26:23.486 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.486 DEBUG:boto:Data: 2024-04-16T19:26:23.486 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.486 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.486 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.486 DEBUG:boto:Params: {} 2024-04-16T19:26:23.486 DEBUG:boto:Token: None 2024-04-16T19:26:23.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU= 2024-04-16T19:26:23.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU=', 'Content-Length': '0'} 2024-04-16T19:26:23.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027b37b7fd2149d5f-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.489 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.489 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.489 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.489 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.489 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.489 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.490 DEBUG:boto:Data: 2024-04-16T19:26:23.490 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.490 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.490 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.490 DEBUG:boto:Params: {} 2024-04-16T19:26:23.490 DEBUG:boto:Token: None 2024-04-16T19:26:23.490 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.490 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CUSmv26HhIsBp86klrRluTTY7H8= 2024-04-16T19:26:23.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CUSmv26HhIsBp86klrRluTTY7H8=', 'Content-Length': '0'} 2024-04-16T19:26:23.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ff6487e444c1fb0-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.493 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.493 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.493 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.493 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.493 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.493 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.493 DEBUG:boto:Data: 2024-04-16T19:26:23.493 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.493 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.493 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.493 DEBUG:boto:Params: {} 2024-04-16T19:26:23.493 DEBUG:boto:Token: None 2024-04-16T19:26:23.494 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CUSmv26HhIsBp86klrRluTTY7H8= 2024-04-16T19:26:23.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CUSmv26HhIsBp86klrRluTTY7H8=', 'Content-Length': '0'} 2024-04-16T19:26:23.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4f788ae5a8142b2-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:23.496 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.497 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.497 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.497 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.497 DEBUG:boto:Method: GET 2024-04-16T19:26:23.497 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.497 DEBUG:boto:Data: 2024-04-16T19:26:23.497 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.497 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.497 DEBUG:boto:Params: {} 2024-04-16T19:26:23.497 DEBUG:boto:Token: None 2024-04-16T19:26:23.497 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU= 2024-04-16T19:26:23.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU=', 'Content-Length': '0'} 2024-04-16T19:26:23.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb7eaf35e64b0b4d-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.500 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.500 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.500 DEBUG:boto:path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.500 DEBUG:boto:auth_path=/yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.500 DEBUG:boto:Method: GET 2024-04-16T19:26:23.500 DEBUG:boto:Path: /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.500 DEBUG:boto:Data: 2024-04-16T19:26:23.501 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.501 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.501 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.501 DEBUG:boto:Params: {} 2024-04-16T19:26:23.501 DEBUG:boto:Token: None 2024-04-16T19:26:23.501 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj?versionId=null 2024-04-16T19:26:23.501 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU= 2024-04-16T19:26:23.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SXY3BzKfLPsmJ5okFg8JXk+2ztU=', 'Content-Length': '0'} 2024-04-16T19:26:23.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067941d233bbb079a-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.504 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.504 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.504 DEBUG:boto:path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.504 DEBUG:boto:auth_path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.504 DEBUG:boto:Method: GET 2024-04-16T19:26:23.504 DEBUG:boto:Path: /yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.504 DEBUG:boto:Data: 2024-04-16T19:26:23.504 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.504 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.504 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.505 DEBUG:boto:Params: {} 2024-04-16T19:26:23.505 DEBUG:boto:Token: None 2024-04-16T19:26:23.505 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.505 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8c9d0048fd04a1-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.511 DEBUG:boto:b'yrjefh-8myobj1000falsemyobj2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.511 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.511 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.511 DEBUG:boto:path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.512 DEBUG:boto:auth_path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.512 DEBUG:boto:Method: GET 2024-04-16T19:26:23.512 DEBUG:boto:Path: /yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.512 DEBUG:boto:Data: 2024-04-16T19:26:23.512 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.512 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.512 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.512 DEBUG:boto:Params: {} 2024-04-16T19:26:23.512 DEBUG:boto:Token: None 2024-04-16T19:26:23.512 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d65ac8e06b0c5ec1-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.562 DEBUG:boto:b'yrjefh-8myobj1000falsemyobj2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.563 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.563 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.563 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.563 DEBUG:boto:Path: /yrjefh-8/myobj 2024-04-16T19:26:23.563 DEBUG:boto:Data: 2024-04-16T19:26:23.563 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.563 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.563 DEBUG:boto:Params: {} 2024-04-16T19:26:23.563 DEBUG:boto:Token: None 2024-04-16T19:26:23.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj 2024-04-16T19:26:23.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dx+QHfjJfT95Jhd+qC9/BKuyIDM= 2024-04-16T19:26:23.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dx+QHfjJfT95Jhd+qC9/BKuyIDM=', 'Content-Length': '0'} 2024-04-16T19:26:23.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e241fae72a7a75fd-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.566 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.566 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.566 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.566 DEBUG:boto:Path: /yrjefh-8/myobj 2024-04-16T19:26:23.567 DEBUG:boto:Data: 2024-04-16T19:26:23.567 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.567 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.567 DEBUG:boto:Params: {} 2024-04-16T19:26:23.567 DEBUG:boto:Token: None 2024-04-16T19:26:23.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj 2024-04-16T19:26:23.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dx+QHfjJfT95Jhd+qC9/BKuyIDM= 2024-04-16T19:26:23.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dx+QHfjJfT95Jhd+qC9/BKuyIDM=', 'Content-Length': '0'} 2024-04-16T19:26:23.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022fb79dacc0cafd6-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:23.570 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.570 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.570 DEBUG:boto:Method: GET 2024-04-16T19:26:23.570 DEBUG:boto:Path: /yrjefh-8/myobj 2024-04-16T19:26:23.570 DEBUG:boto:Data: 2024-04-16T19:26:23.570 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.570 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.570 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.571 DEBUG:boto:Params: {} 2024-04-16T19:26:23.571 DEBUG:boto:Token: None 2024-04-16T19:26:23.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj 2024-04-16T19:26:23.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6uADIJvvAapg4T9gYG01dDSzcA= 2024-04-16T19:26:23.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6uADIJvvAapg4T9gYG01dDSzcA=', 'Content-Length': '0'} 2024-04-16T19:26:23.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:13 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76f78d2110f2c0c-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.573 DEBUG:boto:path=/yrjefh-8/myobj 2024-04-16T19:26:23.573 DEBUG:boto:auth_path=/yrjefh-8/myobj 2024-04-16T19:26:23.573 DEBUG:boto:Method: GET 2024-04-16T19:26:23.573 DEBUG:boto:Path: /yrjefh-8/myobj 2024-04-16T19:26:23.573 DEBUG:boto:Data: 2024-04-16T19:26:23.574 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.574 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.574 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.574 DEBUG:boto:Params: {} 2024-04-16T19:26:23.574 DEBUG:boto:Token: None 2024-04-16T19:26:23.574 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/myobj 2024-04-16T19:26:23.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6uADIJvvAapg4T9gYG01dDSzcA= 2024-04-16T19:26:23.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6uADIJvvAapg4T9gYG01dDSzcA=', 'Content-Length': '0'} 2024-04-16T19:26:23.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afa19594f4433d69-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.577 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.577 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.577 DEBUG:boto:path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.577 DEBUG:boto:auth_path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.577 DEBUG:boto:Method: GET 2024-04-16T19:26:23.577 DEBUG:boto:Path: /yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.577 DEBUG:boto:Data: 2024-04-16T19:26:23.577 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.577 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.578 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.578 DEBUG:boto:Params: {} 2024-04-16T19:26:23.578 DEBUG:boto:Token: None 2024-04-16T19:26:23.578 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375d502b41002b29-00661ed0df-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.580 DEBUG:boto:b'yrjefh-8myobj1000falsemyobj2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.580 DEBUG:boto:path=/yrjefh-8/ 2024-04-16T19:26:23.580 DEBUG:boto:auth_path=/yrjefh-8/ 2024-04-16T19:26:23.581 DEBUG:boto:path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.581 DEBUG:boto:auth_path=/yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.581 DEBUG:boto:Method: GET 2024-04-16T19:26:23.581 DEBUG:boto:Path: /yrjefh-8/?prefix=myobj 2024-04-16T19:26:23.581 DEBUG:boto:Data: 2024-04-16T19:26:23.581 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.581 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.581 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.581 DEBUG:boto:Params: {} 2024-04-16T19:26:23.581 DEBUG:boto:Token: None 2024-04-16T19:26:23.582 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-8/ 2024-04-16T19:26:23.582 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0= 2024-04-16T19:26:23.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:siAFN1hm22RedsInvVxm3wmlGc0=', 'Content-Length': '0'} 2024-04-16T19:26:23.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fd1d516b2be9c2-00661ed0df-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.584 DEBUG:boto:b'yrjefh-8myobj1000falsemyobj2024-04-16T19:26:10.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:23.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-8 zones={test-zone1, test-zone2} 2024-04-16T19:26:23.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-16T19:26:23.586 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-9 2024-04-16T19:26:23.586 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:23.586 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:23.587 DEBUG:boto:Method: PUT 2024-04-16T19:26:23.587 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:23.587 DEBUG:boto:Data: 2024-04-16T19:26:23.587 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.587 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.587 DEBUG:boto:Params: {} 2024-04-16T19:26:23.587 DEBUG:boto:Token: None 2024-04-16T19:26:23.587 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-9/ 2024-04-16T19:26:23.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QscHk2cCD6ZK+5Oczjolyzn1vRA= 2024-04-16T19:26:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QscHk2cCD6ZK+5Oczjolyzn1vRA=', 'Content-Length': '0'} 2024-04-16T19:26:23.605 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b08be0b49c835cc-00661ed0df-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.606 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-10 2024-04-16T19:26:23.606 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:23.606 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:23.606 DEBUG:boto:Method: PUT 2024-04-16T19:26:23.606 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:23.606 DEBUG:boto:Data: 2024-04-16T19:26:23.606 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.606 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.606 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.606 DEBUG:boto:Params: {} 2024-04-16T19:26:23.607 DEBUG:boto:Token: None 2024-04-16T19:26:23.607 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-10/ 2024-04-16T19:26:23.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WOSgzr1sXrPxeURKebOXNqIiMYI= 2024-04-16T19:26:23.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WOSgzr1sXrPxeURKebOXNqIiMYI=', 'Content-Length': '0'} 2024-04-16T19:26:23.636 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a6fbc6d11b8fdbe-00661ed0df-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.636 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:23.636 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:23.636 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.636 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:23.636 DEBUG:boto:Data: 2024-04-16T19:26:23.636 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.636 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.636 DEBUG:boto:Params: {} 2024-04-16T19:26:23.637 DEBUG:boto:Token: None 2024-04-16T19:26:23.637 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-9/ 2024-04-16T19:26:23.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rfdi+mF1IXkyJ2lS2O9JiOmUT/o= 2024-04-16T19:26:23.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rfdi+mF1IXkyJ2lS2O9JiOmUT/o=', 'Content-Length': '0'} 2024-04-16T19:26:23.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dedbc674e23d9f4-00661ed0df-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.640 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:23.640 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:23.640 DEBUG:boto:Method: PUT 2024-04-16T19:26:23.640 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:23.640 DEBUG:boto:Data: 2024-04-16T19:26:23.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:23.640 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:23.640 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.640 DEBUG:boto:Params: {} 2024-04-16T19:26:23.641 DEBUG:boto:Token: None 2024-04-16T19:26:23.641 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-9/myobj 2024-04-16T19:26:23.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N0YXzJxlgNC7v4bElMcJdRLLhHA= 2024-04-16T19:26:23.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N0YXzJxlgNC7v4bElMcJdRLLhHA='} 2024-04-16T19:26:23.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f4dfc0ed488e7df-00661ed0df-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.683 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:23.683 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:23.683 DEBUG:boto:Method: HEAD 2024-04-16T19:26:23.683 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:23.683 DEBUG:boto:Data: 2024-04-16T19:26:23.683 DEBUG:boto:Headers: {} 2024-04-16T19:26:23.683 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.683 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.683 DEBUG:boto:Params: {} 2024-04-16T19:26:23.683 DEBUG:boto:Token: None 2024-04-16T19:26:23.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-10/ 2024-04-16T19:26:23.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwlsJ8IeNGcqQ2zWQRRgoyX4BYw= 2024-04-16T19:26:23.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwlsJ8IeNGcqQ2zWQRRgoyX4BYw=', 'Content-Length': '0'} 2024-04-16T19:26:23.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa90e77114558cd0-00661ed0df-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.686 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:23.687 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:23.687 DEBUG:boto:Method: PUT 2024-04-16T19:26:23.687 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:23.687 DEBUG:boto:Data: 2024-04-16T19:26:23.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:23.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:23.687 DEBUG:boto:Port: 8000 2024-04-16T19:26:23.687 DEBUG:boto:Params: {} 2024-04-16T19:26:23.687 DEBUG:boto:Token: None 2024-04-16T19:26:23.687 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:23 GMT /yrjefh-10/myobj 2024-04-16T19:26:23.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K7LA+nlqOhgy1F1X+VO8oeY1JqY= 2024-04-16T19:26:23.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K7LA+nlqOhgy1F1X+VO8oeY1JqY='} 2024-04-16T19:26:23.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f8f7e886d3bc529-00661ed0df-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:23.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:23.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:23.696 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:23.745 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:23.745 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:26:23.816 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713295583.623976_48.1","last_update":"2024-04-16T19:26:23.623976Z"}] 2024-04-16T19:26:23.816 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295583.623976_48.1'} 2024-04-16T19:26:23.816 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:26:23.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:23.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:23.817 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:23.872 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:23.872 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:23.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:23.922+0000 7f8929731800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:23.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:23.923+0000 7f8929731800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:23.944 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.595014_135.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.595014Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295583.623976_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.623976Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:23Z"} 2024-04-16T19:26:23.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.595014_135.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.595014Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295583.623976_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.623976Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:23Z"} 2024-04-16T19:26:23.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.595014_135.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.595014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295583.623976_48.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.623976Z', 'realm_epoch': 2}}] 2024-04-16T19:26:23.945 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295583.623976_48.1'} 2024-04-16T19:26:23.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.595014_135.1', 3: '1_1713295583.623976_48.1'} 2024-04-16T19:26:23.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713295583.603379_140.1 target=1_1713295583.595014_135.1 2024-04-16T19:26:28.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:28.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:28.946 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:29.001 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:29.001 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:29.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:29.054+0000 7f8b3bc0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:29.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:29.056+0000 7f8b3bc0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:29.077 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295583.623976_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.623976Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:29Z"} 2024-04-16T19:26:29.078 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295583.623976_48.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.623976Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:29Z"} 2024-04-16T19:26:29.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295583.623976_48.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.623976Z', 'realm_epoch': 2}}] 2024-04-16T19:26:29.078 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295583.623976_48.1'} 2024-04-16T19:26:29.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295583.623976_48.1'} 2024-04-16T19:26:29.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:29.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-9', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-9', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.078 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-9 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:29.187+0000 7f24aff31800 1 bucket sync caught up with source: 2024-04-16T19:26:29.188 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2024-04-16T19:26:29.188 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.11.6, , , , , , , , , ] 2024-04-16T19:26:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:29.187+0000 7f24aff31800 0 bucket checkpoint complete 2024-04-16T19:26:29.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-9 zones={test-zone2, test-zone1} 2024-04-16T19:26:29.207 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.207 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.208 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.208 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:29.208 DEBUG:boto:Data: 2024-04-16T19:26:29.208 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.208 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.208 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.208 DEBUG:boto:Params: {} 2024-04-16T19:26:29.208 DEBUG:boto:Token: None 2024-04-16T19:26:29.208 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM= 2024-04-16T19:26:29.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM=', 'Content-Length': '0'} 2024-04-16T19:26:29.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017f8007055ea0c41-00661ed0e5-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.211 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.211 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.211 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.211 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:29.211 DEBUG:boto:Data: 2024-04-16T19:26:29.212 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.212 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.212 DEBUG:boto:Params: {} 2024-04-16T19:26:29.212 DEBUG:boto:Token: None 2024-04-16T19:26:29.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM= 2024-04-16T19:26:29.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM=', 'Content-Length': '0'} 2024-04-16T19:26:29.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ab33088c9a43ac4-00661ed0e5-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:29.215 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.215 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.215 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.215 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.215 DEBUG:boto:Method: GET 2024-04-16T19:26:29.215 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.215 DEBUG:boto:Data: 2024-04-16T19:26:29.215 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.215 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.216 DEBUG:boto:Params: {} 2024-04-16T19:26:29.216 DEBUG:boto:Token: None 2024-04-16T19:26:29.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/?versions 2024-04-16T19:26:29.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI= 2024-04-16T19:26:29.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI=', 'Content-Length': '0'} 2024-04-16T19:26:29.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e750555ef9b1c0-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.218 DEBUG:boto:b'yrjefh-9999falsemyobjnulltrue2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.219 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:29.219 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:29.219 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.219 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.219 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.219 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.219 DEBUG:boto:Method: GET 2024-04-16T19:26:29.219 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.219 DEBUG:boto:Data: 2024-04-16T19:26:29.219 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.220 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.220 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.220 DEBUG:boto:Params: {} 2024-04-16T19:26:29.220 DEBUG:boto:Token: None 2024-04-16T19:26:29.220 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/?versions 2024-04-16T19:26:29.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI= 2024-04-16T19:26:29.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI=', 'Content-Length': '0'} 2024-04-16T19:26:29.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023bcea8f80a3e3ef-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.222 DEBUG:boto:b'yrjefh-9999falsemyobjnulltrue2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.223 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:29.223 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.223 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.223 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.223 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.223 DEBUG:boto:Method: GET 2024-04-16T19:26:29.223 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.223 DEBUG:boto:Data: 2024-04-16T19:26:29.223 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.223 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.224 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.224 DEBUG:boto:Params: {} 2024-04-16T19:26:29.224 DEBUG:boto:Token: None 2024-04-16T19:26:29.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/?versions 2024-04-16T19:26:29.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI= 2024-04-16T19:26:29.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI=', 'Content-Length': '0'} 2024-04-16T19:26:29.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000956523ccf090a50a-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.226 DEBUG:boto:b'yrjefh-9999falsemyobjnulltrue2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.226 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.227 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.227 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.227 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.227 DEBUG:boto:Method: GET 2024-04-16T19:26:29.227 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:29.227 DEBUG:boto:Data: 2024-04-16T19:26:29.227 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.227 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.227 DEBUG:boto:Params: {} 2024-04-16T19:26:29.227 DEBUG:boto:Token: None 2024-04-16T19:26:29.228 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/?versions 2024-04-16T19:26:29.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI= 2024-04-16T19:26:29.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U00x7JM4Io5XNQTZnw0KYo/9dpI=', 'Content-Length': '0'} 2024-04-16T19:26:29.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bfce99642a449d5-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.230 DEBUG:boto:b'yrjefh-9999falsemyobjnulltrue2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.231 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.231 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.231 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.231 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.231 DEBUG:boto:Method: GET 2024-04-16T19:26:29.231 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.231 DEBUG:boto:Data: 2024-04-16T19:26:29.231 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.231 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.231 DEBUG:boto:Params: {} 2024-04-16T19:26:29.231 DEBUG:boto:Token: None 2024-04-16T19:26:29.232 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs= 2024-04-16T19:26:29.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs=', 'Content-Length': '0'} 2024-04-16T19:26:29.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021ae0fd0ce68e99c-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.234 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.234 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.234 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.234 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.234 DEBUG:boto:Method: GET 2024-04-16T19:26:29.234 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.234 DEBUG:boto:Data: 2024-04-16T19:26:29.234 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.234 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.235 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.235 DEBUG:boto:Params: {} 2024-04-16T19:26:29.235 DEBUG:boto:Token: None 2024-04-16T19:26:29.235 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs= 2024-04-16T19:26:29.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs=', 'Content-Length': '0'} 2024-04-16T19:26:29.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b064cba011587e5f-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.237 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.237 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.237 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.237 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.238 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.238 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.238 DEBUG:boto:Data: 2024-04-16T19:26:29.238 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.238 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.238 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.238 DEBUG:boto:Params: {} 2024-04-16T19:26:29.238 DEBUG:boto:Token: None 2024-04-16T19:26:29.238 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3xI7gw2qMnOGKhkpZVU52ryL3w= 2024-04-16T19:26:29.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3xI7gw2qMnOGKhkpZVU52ryL3w=', 'Content-Length': '0'} 2024-04-16T19:26:29.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050bb69113cf16b51-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.240 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.240 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.240 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.241 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.241 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.241 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.241 DEBUG:boto:Data: 2024-04-16T19:26:29.241 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.241 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.241 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.241 DEBUG:boto:Params: {} 2024-04-16T19:26:29.241 DEBUG:boto:Token: None 2024-04-16T19:26:29.241 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3xI7gw2qMnOGKhkpZVU52ryL3w= 2024-04-16T19:26:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3xI7gw2qMnOGKhkpZVU52ryL3w=', 'Content-Length': '0'} 2024-04-16T19:26:29.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082b6a2dc3e23f936-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.244 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.244 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.244 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.244 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.244 DEBUG:boto:Method: GET 2024-04-16T19:26:29.244 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.244 DEBUG:boto:Data: 2024-04-16T19:26:29.244 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.244 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.244 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.244 DEBUG:boto:Params: {} 2024-04-16T19:26:29.244 DEBUG:boto:Token: None 2024-04-16T19:26:29.245 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs= 2024-04-16T19:26:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs=', 'Content-Length': '0'} 2024-04-16T19:26:29.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd90094d80692f20-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.246 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.247 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.247 DEBUG:boto:path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.247 DEBUG:boto:auth_path=/yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.247 DEBUG:boto:Method: GET 2024-04-16T19:26:29.247 DEBUG:boto:Path: /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.247 DEBUG:boto:Data: 2024-04-16T19:26:29.247 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.247 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.247 DEBUG:boto:Params: {} 2024-04-16T19:26:29.248 DEBUG:boto:Token: None 2024-04-16T19:26:29.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj?versionId=null 2024-04-16T19:26:29.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs= 2024-04-16T19:26:29.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFGDyXymfMVnAC41ECtvj1OBEIs=', 'Content-Length': '0'} 2024-04-16T19:26:29.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d178e82d6bd908a-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.290 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.290 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.290 DEBUG:boto:path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.291 DEBUG:boto:auth_path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.291 DEBUG:boto:Method: GET 2024-04-16T19:26:29.291 DEBUG:boto:Path: /yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.291 DEBUG:boto:Data: 2024-04-16T19:26:29.291 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.291 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.291 DEBUG:boto:Params: {} 2024-04-16T19:26:29.291 DEBUG:boto:Token: None 2024-04-16T19:26:29.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY= 2024-04-16T19:26:29.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY=', 'Content-Length': '0'} 2024-04-16T19:26:29.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1c3f2e008136ea-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.294 DEBUG:boto:b'yrjefh-9myobj1000falsemyobj2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.295 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.295 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.295 DEBUG:boto:path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.295 DEBUG:boto:auth_path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.295 DEBUG:boto:Method: GET 2024-04-16T19:26:29.295 DEBUG:boto:Path: /yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.295 DEBUG:boto:Data: 2024-04-16T19:26:29.295 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.295 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.295 DEBUG:boto:Params: {} 2024-04-16T19:26:29.296 DEBUG:boto:Token: None 2024-04-16T19:26:29.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY= 2024-04-16T19:26:29.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY=', 'Content-Length': '0'} 2024-04-16T19:26:29.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6baecedd962a3d-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.298 DEBUG:boto:b'yrjefh-9myobj1000falsemyobj2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.298 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.298 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.299 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.299 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.299 DEBUG:boto:Data: 2024-04-16T19:26:29.299 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.299 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.299 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.299 DEBUG:boto:Params: {} 2024-04-16T19:26:29.299 DEBUG:boto:Token: None 2024-04-16T19:26:29.299 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo= 2024-04-16T19:26:29.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo=', 'Content-Length': '0'} 2024-04-16T19:26:29.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ca54bcfbd936719-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.301 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.301 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.301 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.301 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.301 DEBUG:boto:Data: 2024-04-16T19:26:29.302 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.302 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.302 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.302 DEBUG:boto:Params: {} 2024-04-16T19:26:29.302 DEBUG:boto:Token: None 2024-04-16T19:26:29.302 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo= 2024-04-16T19:26:29.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo=', 'Content-Length': '0'} 2024-04-16T19:26:29.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054496a406e991532-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.304 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.304 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.304 DEBUG:boto:Method: GET 2024-04-16T19:26:29.305 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.305 DEBUG:boto:Data: 2024-04-16T19:26:29.305 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.305 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.305 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.305 DEBUG:boto:Params: {} 2024-04-16T19:26:29.305 DEBUG:boto:Token: None 2024-04-16T19:26:29.305 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F8ZRiSnJz6aDeuJVsh+Ru4ZBwR8= 2024-04-16T19:26:29.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F8ZRiSnJz6aDeuJVsh+Ru4ZBwR8=', 'Content-Length': '0'} 2024-04-16T19:26:29.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c10fc124de7b9e1-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.350 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.350 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.350 DEBUG:boto:Method: GET 2024-04-16T19:26:29.351 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.351 DEBUG:boto:Data: 2024-04-16T19:26:29.351 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.351 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.351 DEBUG:boto:Params: {} 2024-04-16T19:26:29.351 DEBUG:boto:Token: None 2024-04-16T19:26:29.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F8ZRiSnJz6aDeuJVsh+Ru4ZBwR8= 2024-04-16T19:26:29.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F8ZRiSnJz6aDeuJVsh+Ru4ZBwR8=', 'Content-Length': '0'} 2024-04-16T19:26:29.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9868494d6bd0b80-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.353 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.353 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.353 DEBUG:boto:path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.353 DEBUG:boto:auth_path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.354 DEBUG:boto:Method: GET 2024-04-16T19:26:29.354 DEBUG:boto:Path: /yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.354 DEBUG:boto:Data: 2024-04-16T19:26:29.354 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.354 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.354 DEBUG:boto:Params: {} 2024-04-16T19:26:29.354 DEBUG:boto:Token: None 2024-04-16T19:26:29.354 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.354 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY= 2024-04-16T19:26:29.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY=', 'Content-Length': '0'} 2024-04-16T19:26:29.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e97ff9dfff530b-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.398 DEBUG:boto:b'yrjefh-9myobj1000falsemyobj2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.399 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.399 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.399 DEBUG:boto:path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.399 DEBUG:boto:auth_path=/yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.399 DEBUG:boto:Method: GET 2024-04-16T19:26:29.399 DEBUG:boto:Path: /yrjefh-9/?prefix=myobj 2024-04-16T19:26:29.399 DEBUG:boto:Data: 2024-04-16T19:26:29.399 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.399 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.399 DEBUG:boto:Params: {} 2024-04-16T19:26:29.400 DEBUG:boto:Token: None 2024-04-16T19:26:29.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY= 2024-04-16T19:26:29.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:laoa0/rp85rpxJzOoO1JXebSJWY=', 'Content-Length': '0'} 2024-04-16T19:26:29.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a99ff851e46a108-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.402 DEBUG:boto:b'yrjefh-9myobj1000falsemyobj2024-04-16T19:26:23.642Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.402 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-9 zones={test-zone2, test-zone1} 2024-04-16T19:26:29.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-10', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-10', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.403 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-10 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:29.469 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:29.531+0000 7f0a16a10800 1 bucket sync caught up with source: 2024-04-16T19:26:29.532 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2024-04-16T19:26:29.532 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.14.6, , , , , , , , , ] 2024-04-16T19:26:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:29.531+0000 7f0a16a10800 0 bucket checkpoint complete 2024-04-16T19:26:29.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-10 zones={test-zone1, test-zone2} 2024-04-16T19:26:29.552 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.553 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.553 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.553 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:29.553 DEBUG:boto:Data: 2024-04-16T19:26:29.553 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.553 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.553 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.553 DEBUG:boto:Params: {} 2024-04-16T19:26:29.553 DEBUG:boto:Token: None 2024-04-16T19:26:29.553 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NUW+BfCzjKktvnwoXnMa5PH2Pro= 2024-04-16T19:26:29.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUW+BfCzjKktvnwoXnMa5PH2Pro=', 'Content-Length': '0'} 2024-04-16T19:26:29.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005925947affd93e97-00661ed0e5-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.556 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.556 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.556 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.556 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:29.557 DEBUG:boto:Data: 2024-04-16T19:26:29.557 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.557 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.557 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.557 DEBUG:boto:Params: {} 2024-04-16T19:26:29.557 DEBUG:boto:Token: None 2024-04-16T19:26:29.557 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.557 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NUW+BfCzjKktvnwoXnMa5PH2Pro= 2024-04-16T19:26:29.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUW+BfCzjKktvnwoXnMa5PH2Pro=', 'Content-Length': '0'} 2024-04-16T19:26:29.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ee2d4ba206e65bf-00661ed0e5-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:29.560 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.560 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.560 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.560 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.560 DEBUG:boto:Method: GET 2024-04-16T19:26:29.561 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.561 DEBUG:boto:Data: 2024-04-16T19:26:29.561 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.561 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.561 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.561 DEBUG:boto:Params: {} 2024-04-16T19:26:29.561 DEBUG:boto:Token: None 2024-04-16T19:26:29.561 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/?versions 2024-04-16T19:26:29.561 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w= 2024-04-16T19:26:29.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w=', 'Content-Length': '0'} 2024-04-16T19:26:29.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000925e94acb56da10a-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.564 DEBUG:boto:b'yrjefh-10999falsemyobjnulltrue2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.564 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:29.564 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:29.564 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.564 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.564 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.565 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.565 DEBUG:boto:Method: GET 2024-04-16T19:26:29.565 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.565 DEBUG:boto:Data: 2024-04-16T19:26:29.565 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.565 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.565 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.565 DEBUG:boto:Params: {} 2024-04-16T19:26:29.565 DEBUG:boto:Token: None 2024-04-16T19:26:29.565 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/?versions 2024-04-16T19:26:29.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w= 2024-04-16T19:26:29.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w=', 'Content-Length': '0'} 2024-04-16T19:26:29.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3273611ba0450b1-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.568 DEBUG:boto:b'yrjefh-10999falsemyobjnulltrue2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.569 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-16T19:26:29.569 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.569 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.569 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.569 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.569 DEBUG:boto:Method: GET 2024-04-16T19:26:29.569 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.569 DEBUG:boto:Data: 2024-04-16T19:26:29.569 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.570 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.570 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.570 DEBUG:boto:Params: {} 2024-04-16T19:26:29.570 DEBUG:boto:Token: None 2024-04-16T19:26:29.570 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/?versions 2024-04-16T19:26:29.570 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w= 2024-04-16T19:26:29.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w=', 'Content-Length': '0'} 2024-04-16T19:26:29.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee9ddddbc253b21-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.572 DEBUG:boto:b'yrjefh-10999falsemyobjnulltrue2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.573 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.573 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.573 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.573 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.573 DEBUG:boto:Method: GET 2024-04-16T19:26:29.573 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:29.573 DEBUG:boto:Data: 2024-04-16T19:26:29.573 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.573 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.573 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.573 DEBUG:boto:Params: {} 2024-04-16T19:26:29.574 DEBUG:boto:Token: None 2024-04-16T19:26:29.574 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/?versions 2024-04-16T19:26:29.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w= 2024-04-16T19:26:29.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o2YJSFeR5ZU7UvXYhPSeeaXV40w=', 'Content-Length': '0'} 2024-04-16T19:26:29.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c7cba4bcf7bdbe-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.576 DEBUG:boto:b'yrjefh-10999falsemyobjnulltrue2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.577 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.577 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.577 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.577 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.577 DEBUG:boto:Method: GET 2024-04-16T19:26:29.577 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.577 DEBUG:boto:Data: 2024-04-16T19:26:29.577 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.577 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.578 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.578 DEBUG:boto:Params: {} 2024-04-16T19:26:29.578 DEBUG:boto:Token: None 2024-04-16T19:26:29.578 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0= 2024-04-16T19:26:29.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0=', 'Content-Length': '0'} 2024-04-16T19:26:29.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:28 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb972e13164b2057-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.580 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.580 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.580 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.580 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.581 DEBUG:boto:Method: GET 2024-04-16T19:26:29.581 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.581 DEBUG:boto:Data: 2024-04-16T19:26:29.581 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.581 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.581 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.581 DEBUG:boto:Params: {} 2024-04-16T19:26:29.581 DEBUG:boto:Token: None 2024-04-16T19:26:29.581 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.581 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0= 2024-04-16T19:26:29.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0=', 'Content-Length': '0'} 2024-04-16T19:26:29.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cb74605f5cf9e6d-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.583 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.583 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.584 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.584 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.584 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.584 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.584 DEBUG:boto:Data: 2024-04-16T19:26:29.584 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.584 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.584 DEBUG:boto:Params: {} 2024-04-16T19:26:29.584 DEBUG:boto:Token: None 2024-04-16T19:26:29.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lRW6h4cByQkiOcGqV5BaJySlmQE= 2024-04-16T19:26:29.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lRW6h4cByQkiOcGqV5BaJySlmQE=', 'Content-Length': '0'} 2024-04-16T19:26:29.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:28 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028fe5c8582dfbc6f-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.586 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.586 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.586 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.587 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.587 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.587 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.587 DEBUG:boto:Data: 2024-04-16T19:26:29.587 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.587 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.587 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.587 DEBUG:boto:Params: {} 2024-04-16T19:26:29.587 DEBUG:boto:Token: None 2024-04-16T19:26:29.587 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lRW6h4cByQkiOcGqV5BaJySlmQE= 2024-04-16T19:26:29.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lRW6h4cByQkiOcGqV5BaJySlmQE=', 'Content-Length': '0'} 2024-04-16T19:26:29.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7d1ae402ecace02-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.589 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.590 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.590 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.590 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.590 DEBUG:boto:Method: GET 2024-04-16T19:26:29.590 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.590 DEBUG:boto:Data: 2024-04-16T19:26:29.590 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.590 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.590 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.590 DEBUG:boto:Params: {} 2024-04-16T19:26:29.591 DEBUG:boto:Token: None 2024-04-16T19:26:29.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0= 2024-04-16T19:26:29.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0=', 'Content-Length': '0'} 2024-04-16T19:26:29.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:28 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a006a899aa47abf-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.593 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.593 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.593 DEBUG:boto:path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.593 DEBUG:boto:auth_path=/yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.593 DEBUG:boto:Method: GET 2024-04-16T19:26:29.593 DEBUG:boto:Path: /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.593 DEBUG:boto:Data: 2024-04-16T19:26:29.593 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.593 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.593 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.594 DEBUG:boto:Params: {} 2024-04-16T19:26:29.594 DEBUG:boto:Token: None 2024-04-16T19:26:29.594 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj?versionId=null 2024-04-16T19:26:29.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0= 2024-04-16T19:26:29.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JrWo0MqKHgM7PK96fvnrUg9MiK0=', 'Content-Length': '0'} 2024-04-16T19:26:29.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e3cb3684d5ab65f-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.638 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.639 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.639 DEBUG:boto:path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.639 DEBUG:boto:auth_path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.639 DEBUG:boto:Method: GET 2024-04-16T19:26:29.639 DEBUG:boto:Path: /yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.639 DEBUG:boto:Data: 2024-04-16T19:26:29.639 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.639 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.639 DEBUG:boto:Params: {} 2024-04-16T19:26:29.639 DEBUG:boto:Token: None 2024-04-16T19:26:29.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww= 2024-04-16T19:26:29.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww=', 'Content-Length': '0'} 2024-04-16T19:26:29.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b6d9b54643d64b-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.642 DEBUG:boto:b'yrjefh-10myobj1000falsemyobj2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.642 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.643 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.643 DEBUG:boto:path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.643 DEBUG:boto:auth_path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.643 DEBUG:boto:Method: GET 2024-04-16T19:26:29.643 DEBUG:boto:Path: /yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.643 DEBUG:boto:Data: 2024-04-16T19:26:29.643 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.643 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.643 DEBUG:boto:Params: {} 2024-04-16T19:26:29.644 DEBUG:boto:Token: None 2024-04-16T19:26:29.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww= 2024-04-16T19:26:29.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww=', 'Content-Length': '0'} 2024-04-16T19:26:29.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a6054816b170ea3-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.646 DEBUG:boto:b'yrjefh-10myobj1000falsemyobj2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.646 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.647 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.647 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.647 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:29.647 DEBUG:boto:Data: 2024-04-16T19:26:29.647 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.647 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.647 DEBUG:boto:Params: {} 2024-04-16T19:26:29.647 DEBUG:boto:Token: None 2024-04-16T19:26:29.647 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj 2024-04-16T19:26:29.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k4mteYkEmnn0RIMHQ0CMKMWUREw= 2024-04-16T19:26:29.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k4mteYkEmnn0RIMHQ0CMKMWUREw=', 'Content-Length': '0'} 2024-04-16T19:26:29.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:28 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0a799edd43a0b30-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.649 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.649 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.649 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.650 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:29.650 DEBUG:boto:Data: 2024-04-16T19:26:29.650 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.650 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.650 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.650 DEBUG:boto:Params: {} 2024-04-16T19:26:29.650 DEBUG:boto:Token: None 2024-04-16T19:26:29.650 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj 2024-04-16T19:26:29.650 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k4mteYkEmnn0RIMHQ0CMKMWUREw= 2024-04-16T19:26:29.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k4mteYkEmnn0RIMHQ0CMKMWUREw=', 'Content-Length': '0'} 2024-04-16T19:26:29.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001d3a4b954788bff-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-16T19:26:29.652 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.652 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.653 DEBUG:boto:Method: GET 2024-04-16T19:26:29.653 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:29.653 DEBUG:boto:Data: 2024-04-16T19:26:29.653 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.653 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.653 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.653 DEBUG:boto:Params: {} 2024-04-16T19:26:29.653 DEBUG:boto:Token: None 2024-04-16T19:26:29.653 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj 2024-04-16T19:26:29.653 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lj5FBBlmZlatEWphDSoy+mokdUI= 2024-04-16T19:26:29.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lj5FBBlmZlatEWphDSoy+mokdUI=', 'Content-Length': '0'} 2024-04-16T19:26:29.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:26:28 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d42a07b9561d923c-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.655 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:29.655 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:29.656 DEBUG:boto:Method: GET 2024-04-16T19:26:29.656 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:29.656 DEBUG:boto:Data: 2024-04-16T19:26:29.656 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.656 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.656 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.656 DEBUG:boto:Params: {} 2024-04-16T19:26:29.656 DEBUG:boto:Token: None 2024-04-16T19:26:29.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/myobj 2024-04-16T19:26:29.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lj5FBBlmZlatEWphDSoy+mokdUI= 2024-04-16T19:26:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lj5FBBlmZlatEWphDSoy+mokdUI=', 'Content-Length': '0'} 2024-04-16T19:26:29.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000809456399302a88d-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.658 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.658 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.658 DEBUG:boto:path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.658 DEBUG:boto:auth_path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.659 DEBUG:boto:Method: GET 2024-04-16T19:26:29.659 DEBUG:boto:Path: /yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.659 DEBUG:boto:Data: 2024-04-16T19:26:29.659 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.659 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.659 DEBUG:boto:Params: {} 2024-04-16T19:26:29.659 DEBUG:boto:Token: None 2024-04-16T19:26:29.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww= 2024-04-16T19:26:29.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww=', 'Content-Length': '0'} 2024-04-16T19:26:29.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0e2e5af3d55813-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.702 DEBUG:boto:b'yrjefh-10myobj1000falsemyobj2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.703 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:29.703 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:29.703 DEBUG:boto:path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.703 DEBUG:boto:auth_path=/yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.703 DEBUG:boto:Method: GET 2024-04-16T19:26:29.703 DEBUG:boto:Path: /yrjefh-10/?prefix=myobj 2024-04-16T19:26:29.703 DEBUG:boto:Data: 2024-04-16T19:26:29.703 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:29.703 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.703 DEBUG:boto:Params: {} 2024-04-16T19:26:29.704 DEBUG:boto:Token: None 2024-04-16T19:26:29.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-10/ 2024-04-16T19:26:29.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww= 2024-04-16T19:26:29.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w8Y8HGDD/qFnzqMq8HrKmouTKww=', 'Content-Length': '0'} 2024-04-16T19:26:29.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043fac01176c5c5e2-00661ed0e5-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.706 DEBUG:boto:b'yrjefh-10myobj1000falsemyobj2024-04-16T19:26:23.689Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:26:29.707 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-10 zones={test-zone1, test-zone2} 2024-04-16T19:26:29.707 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:29.707 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:29.707 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.707 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:29.707 DEBUG:boto:Data: 2024-04-16T19:26:29.707 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.708 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.708 DEBUG:boto:Params: {} 2024-04-16T19:26:29.708 DEBUG:boto:Token: None 2024-04-16T19:26:29.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/ 2024-04-16T19:26:29.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM= 2024-04-16T19:26:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEz6n6AGObjU4ZD0Q2N00YTCIiM=', 'Content-Length': '0'} 2024-04-16T19:26:29.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d878d52f68085cc3-00661ed0e5-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.710 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.710 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.710 DEBUG:boto:Method: HEAD 2024-04-16T19:26:29.710 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.711 DEBUG:boto:Data: 2024-04-16T19:26:29.711 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.711 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.711 DEBUG:boto:Params: {} 2024-04-16T19:26:29.711 DEBUG:boto:Token: None 2024-04-16T19:26:29.711 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo= 2024-04-16T19:26:29.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qPCanAcMsQ03fbv0LLKu3pZVWEo=', 'Content-Length': '0'} 2024-04-16T19:26:29.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004269a6274294de1-00661ed0e5-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.713 DEBUG:boto:path=/yrjefh-9/myobj 2024-04-16T19:26:29.713 DEBUG:boto:auth_path=/yrjefh-9/myobj 2024-04-16T19:26:29.713 DEBUG:boto:Method: DELETE 2024-04-16T19:26:29.713 DEBUG:boto:Path: /yrjefh-9/myobj 2024-04-16T19:26:29.713 DEBUG:boto:Data: 2024-04-16T19:26:29.713 DEBUG:boto:Headers: {} 2024-04-16T19:26:29.714 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:29.714 DEBUG:boto:Port: 8000 2024-04-16T19:26:29.714 DEBUG:boto:Params: {} 2024-04-16T19:26:29.714 DEBUG:boto:Token: None 2024-04-16T19:26:29.714 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:26:29 GMT /yrjefh-9/myobj 2024-04-16T19:26:29.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KD+l8VypdzCJNK24R02S+l7mDRI= 2024-04-16T19:26:29.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KD+l8VypdzCJNK24R02S+l7mDRI=', 'Content-Length': '0'} 2024-04-16T19:26:29.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff76f8c039b24152-00661ed0e5-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:29.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-9', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-9', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:29.719 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-9 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:29.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:29.829+0000 7f150cd4b800 1 waiting for incremental sync to catch up: 2024-04-16T19:26:29.830 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2024-04-16T19:26:29.830 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2024-04-16T19:26:34.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:34.830+0000 7f150cd4b800 1 waiting for incremental sync to catch up: 2024-04-16T19:26:34.832 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2024-04-16T19:26:34.832 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2024-04-16T19:26:39.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:39.831+0000 7f150cd4b800 1 bucket sync caught up with source: 2024-04-16T19:26:39.833 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000002.13.6, , , , , , , , , ] 2024-04-16T19:26:39.833 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2024-04-16T19:26:39.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:39.831+0000 7f150cd4b800 0 bucket checkpoint complete 2024-04-16T19:26:39.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-9 zones={test-zone2, test-zone1} 2024-04-16T19:26:39.857 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.857 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.857 DEBUG:boto:Method: HEAD 2024-04-16T19:26:39.857 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:39.857 DEBUG:boto:Data: 2024-04-16T19:26:39.857 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.857 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.858 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.858 DEBUG:boto:Params: {} 2024-04-16T19:26:39.858 DEBUG:boto:Token: None 2024-04-16T19:26:39.858 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/ 2024-04-16T19:26:39.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JkALosGdRVKqS9v1bNAq0L4m10o= 2024-04-16T19:26:39.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JkALosGdRVKqS9v1bNAq0L4m10o=', 'Content-Length': '0'} 2024-04-16T19:26:39.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a4258344a511bbb-00661ed0ef-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.861 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.861 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.861 DEBUG:boto:Method: HEAD 2024-04-16T19:26:39.862 DEBUG:boto:Path: /yrjefh-9/ 2024-04-16T19:26:39.862 DEBUG:boto:Data: 2024-04-16T19:26:39.862 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:39.862 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.862 DEBUG:boto:Params: {} 2024-04-16T19:26:39.862 DEBUG:boto:Token: None 2024-04-16T19:26:39.862 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/ 2024-04-16T19:26:39.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JkALosGdRVKqS9v1bNAq0L4m10o= 2024-04-16T19:26:39.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JkALosGdRVKqS9v1bNAq0L4m10o=', 'Content-Length': '0'} 2024-04-16T19:26:39.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce84fd1dbffaa759-00661ed0ef-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.865 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:39.865 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.865 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.865 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.866 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.866 DEBUG:boto:Method: GET 2024-04-16T19:26:39.866 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.866 DEBUG:boto:Data: 2024-04-16T19:26:39.866 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.866 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.866 DEBUG:boto:Params: {} 2024-04-16T19:26:39.866 DEBUG:boto:Token: None 2024-04-16T19:26:39.866 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/?versions 2024-04-16T19:26:39.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA= 2024-04-16T19:26:39.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA=', 'Content-Length': '0'} 2024-04-16T19:26:39.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb7a2a365516c1d-00661ed0ef-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.869 DEBUG:boto:b'yrjefh-9999false' 2024-04-16T19:26:39.869 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:39.870 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.870 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.870 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.870 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.870 DEBUG:boto:Method: GET 2024-04-16T19:26:39.870 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.870 DEBUG:boto:Data: 2024-04-16T19:26:39.870 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.870 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:39.870 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.870 DEBUG:boto:Params: {} 2024-04-16T19:26:39.871 DEBUG:boto:Token: None 2024-04-16T19:26:39.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/?versions 2024-04-16T19:26:39.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA= 2024-04-16T19:26:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA=', 'Content-Length': '0'} 2024-04-16T19:26:39.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9e965bcac8ecad-00661ed0ef-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.874 DEBUG:boto:b'yrjefh-9999false' 2024-04-16T19:26:39.874 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.874 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.874 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.874 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.874 DEBUG:boto:Method: GET 2024-04-16T19:26:39.874 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.874 DEBUG:boto:Data: 2024-04-16T19:26:39.875 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.875 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.875 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.875 DEBUG:boto:Params: {} 2024-04-16T19:26:39.875 DEBUG:boto:Token: None 2024-04-16T19:26:39.875 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/?versions 2024-04-16T19:26:39.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA= 2024-04-16T19:26:39.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA=', 'Content-Length': '0'} 2024-04-16T19:26:39.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad04737eca65dcb9-00661ed0ef-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.878 DEBUG:boto:b'yrjefh-9999false' 2024-04-16T19:26:39.878 DEBUG:boto:path=/yrjefh-9/ 2024-04-16T19:26:39.878 DEBUG:boto:auth_path=/yrjefh-9/ 2024-04-16T19:26:39.879 DEBUG:boto:path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.879 DEBUG:boto:auth_path=/yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.879 DEBUG:boto:Method: GET 2024-04-16T19:26:39.879 DEBUG:boto:Path: /yrjefh-9/?versions&max-keys=999 2024-04-16T19:26:39.879 DEBUG:boto:Data: 2024-04-16T19:26:39.879 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:39.879 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.879 DEBUG:boto:Params: {} 2024-04-16T19:26:39.879 DEBUG:boto:Token: None 2024-04-16T19:26:39.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-9/?versions 2024-04-16T19:26:39.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA= 2024-04-16T19:26:39.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Jyd0Zz6p1Qgvhy71X2EY5O+bqA=', 'Content-Length': '0'} 2024-04-16T19:26:39.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d743d5726fd8b7e4-00661ed0ef-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.884 DEBUG:boto:b'yrjefh-9999false' 2024-04-16T19:26:39.884 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-9 zones={test-zone2, test-zone1} 2024-04-16T19:26:39.884 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:39.884 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:39.884 DEBUG:boto:Method: HEAD 2024-04-16T19:26:39.884 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:39.884 DEBUG:boto:Data: 2024-04-16T19:26:39.884 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.885 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.885 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.885 DEBUG:boto:Params: {} 2024-04-16T19:26:39.885 DEBUG:boto:Token: None 2024-04-16T19:26:39.885 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-10/ 2024-04-16T19:26:39.885 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u/qvhxYq23Hhoy6tlLUYSddSvrY= 2024-04-16T19:26:39.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u/qvhxYq23Hhoy6tlLUYSddSvrY=', 'Content-Length': '0'} 2024-04-16T19:26:39.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f45a54fddb9e5833-00661ed0ef-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.888 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:39.888 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:39.888 DEBUG:boto:Method: HEAD 2024-04-16T19:26:39.888 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:39.888 DEBUG:boto:Data: 2024-04-16T19:26:39.888 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.888 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.888 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.888 DEBUG:boto:Params: {} 2024-04-16T19:26:39.889 DEBUG:boto:Token: None 2024-04-16T19:26:39.889 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-10/myobj 2024-04-16T19:26:39.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wkj4PEMY53V/gL98bRbNHl/MudI= 2024-04-16T19:26:39.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wkj4PEMY53V/gL98bRbNHl/MudI=', 'Content-Length': '0'} 2024-04-16T19:26:39.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:26:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2c546c900420540-00661ed0ef-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.891 DEBUG:boto:path=/yrjefh-10/myobj 2024-04-16T19:26:39.891 DEBUG:boto:auth_path=/yrjefh-10/myobj 2024-04-16T19:26:39.891 DEBUG:boto:Method: DELETE 2024-04-16T19:26:39.891 DEBUG:boto:Path: /yrjefh-10/myobj 2024-04-16T19:26:39.891 DEBUG:boto:Data: 2024-04-16T19:26:39.891 DEBUG:boto:Headers: {} 2024-04-16T19:26:39.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:39.892 DEBUG:boto:Port: 8000 2024-04-16T19:26:39.892 DEBUG:boto:Params: {} 2024-04-16T19:26:39.892 DEBUG:boto:Token: None 2024-04-16T19:26:39.892 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:26:39 GMT /yrjefh-10/myobj 2024-04-16T19:26:39.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s5rMhr+eUVHcKi2ObUKMJdr6ZU8= 2024-04-16T19:26:39.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s5rMhr+eUVHcKi2ObUKMJdr6ZU8=', 'Content-Length': '0'} 2024-04-16T19:26:39.896 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027b4e9cc0defa4fe-00661ed0ef-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:39.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-10', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:39.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-10', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:39.897 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-10 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:39.959 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:39.960 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:40.019+0000 7fa25b22d800 1 waiting for incremental sync to catch up: 2024-04-16T19:26:40.021 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2024-04-16T19:26:40.022 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-04-16T19:26:45.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:45.020+0000 7fa25b22d800 1 waiting for incremental sync to catch up: 2024-04-16T19:26:45.022 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2024-04-16T19:26:45.023 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-04-16T19:26:50.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:50.021+0000 7fa25b22d800 1 bucket sync caught up with source: 2024-04-16T19:26:50.024 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.17.6, , , , , , , , , ] 2024-04-16T19:26:50.024 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-04-16T19:26:50.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:26:50.021+0000 7fa25b22d800 0 bucket checkpoint complete 2024-04-16T19:26:50.048 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-10 zones={test-zone1, test-zone2} 2024-04-16T19:26:50.048 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.048 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.048 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.048 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:50.048 DEBUG:boto:Data: 2024-04-16T19:26:50.048 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.048 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.048 DEBUG:boto:Params: {} 2024-04-16T19:26:50.049 DEBUG:boto:Token: None 2024-04-16T19:26:50.049 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/ 2024-04-16T19:26:50.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6d92/EG4rU34GBLZzGoCm8pF2Rw= 2024-04-16T19:26:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6d92/EG4rU34GBLZzGoCm8pF2Rw=', 'Content-Length': '0'} 2024-04-16T19:26:50.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ec8f26b1a2324be-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.051 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.052 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.052 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.052 DEBUG:boto:Path: /yrjefh-10/ 2024-04-16T19:26:50.052 DEBUG:boto:Data: 2024-04-16T19:26:50.052 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.052 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:50.052 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.052 DEBUG:boto:Params: {} 2024-04-16T19:26:50.052 DEBUG:boto:Token: None 2024-04-16T19:26:50.052 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/ 2024-04-16T19:26:50.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6d92/EG4rU34GBLZzGoCm8pF2Rw= 2024-04-16T19:26:50.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6d92/EG4rU34GBLZzGoCm8pF2Rw=', 'Content-Length': '0'} 2024-04-16T19:26:50.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c289540801b5ba58-00661ed0fa-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:26:50.055 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.055 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.055 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.055 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.056 DEBUG:boto:Method: GET 2024-04-16T19:26:50.056 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.056 DEBUG:boto:Data: 2024-04-16T19:26:50.056 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.056 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.056 DEBUG:boto:Params: {} 2024-04-16T19:26:50.056 DEBUG:boto:Token: None 2024-04-16T19:26:50.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/?versions 2024-04-16T19:26:50.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw= 2024-04-16T19:26:50.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw=', 'Content-Length': '0'} 2024-04-16T19:26:50.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea19b9b8d5a3d5a-00661ed0fa-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.059 DEBUG:boto:b'yrjefh-10999false' 2024-04-16T19:26:50.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:26:50.059 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.059 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.060 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.060 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.060 DEBUG:boto:Method: GET 2024-04-16T19:26:50.060 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.060 DEBUG:boto:Data: 2024-04-16T19:26:50.060 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.060 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:50.060 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.060 DEBUG:boto:Params: {} 2024-04-16T19:26:50.060 DEBUG:boto:Token: None 2024-04-16T19:26:50.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/?versions 2024-04-16T19:26:50.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw= 2024-04-16T19:26:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw=', 'Content-Length': '0'} 2024-04-16T19:26:50.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce3b3805eae0a85-00661ed0fa-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.063 DEBUG:boto:b'yrjefh-10999false' 2024-04-16T19:26:50.063 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.063 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.064 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.064 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.064 DEBUG:boto:Method: GET 2024-04-16T19:26:50.064 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.064 DEBUG:boto:Data: 2024-04-16T19:26:50.064 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.064 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.064 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.064 DEBUG:boto:Params: {} 2024-04-16T19:26:50.064 DEBUG:boto:Token: None 2024-04-16T19:26:50.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/?versions 2024-04-16T19:26:50.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw= 2024-04-16T19:26:50.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw=', 'Content-Length': '0'} 2024-04-16T19:26:50.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae123b1725849963-00661ed0fa-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.110 DEBUG:boto:b'yrjefh-10999false' 2024-04-16T19:26:50.111 DEBUG:boto:path=/yrjefh-10/ 2024-04-16T19:26:50.111 DEBUG:boto:auth_path=/yrjefh-10/ 2024-04-16T19:26:50.111 DEBUG:boto:path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.111 DEBUG:boto:auth_path=/yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.111 DEBUG:boto:Method: GET 2024-04-16T19:26:50.111 DEBUG:boto:Path: /yrjefh-10/?versions&max-keys=999 2024-04-16T19:26:50.111 DEBUG:boto:Data: 2024-04-16T19:26:50.111 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:50.111 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.111 DEBUG:boto:Params: {} 2024-04-16T19:26:50.111 DEBUG:boto:Token: None 2024-04-16T19:26:50.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-10/?versions 2024-04-16T19:26:50.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw= 2024-04-16T19:26:50.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfOHTFE+rzUJPyQONuSnRoXkqHw=', 'Content-Length': '0'} 2024-04-16T19:26:50.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a866fc4561035ef-00661ed0fa-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.114 DEBUG:boto:b'yrjefh-10999false' 2024-04-16T19:26:50.115 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-10 zones={test-zone1, test-zone2} 2024-04-16T19:26:50.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-16T19:26:50.116 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-11 2024-04-16T19:26:50.116 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.116 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.116 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.116 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.116 DEBUG:boto:Data: 2024-04-16T19:26:50.116 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.116 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.116 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.116 DEBUG:boto:Params: {} 2024-04-16T19:26:50.117 DEBUG:boto:Token: None 2024-04-16T19:26:50.117 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QEQBaF+ahEVrwB3QSxMN1c9Ecm4= 2024-04-16T19:26:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QEQBaF+ahEVrwB3QSxMN1c9Ecm4=', 'Content-Length': '0'} 2024-04-16T19:26:50.133 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008cdca3691f685361-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.133 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-12 2024-04-16T19:26:50.133 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:50.133 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:50.133 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.133 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:50.133 DEBUG:boto:Data: 2024-04-16T19:26:50.133 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.133 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:50.133 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.133 DEBUG:boto:Params: {} 2024-04-16T19:26:50.134 DEBUG:boto:Token: None 2024-04-16T19:26:50.134 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-12/ 2024-04-16T19:26:50.134 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Sfnj14g6QYwvujKaYBf3agh7qgY= 2024-04-16T19:26:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Sfnj14g6QYwvujKaYBf3agh7qgY=', 'Content-Length': '0'} 2024-04-16T19:26:50.167 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5b0951cc15c6d79-00661ed0fa-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.167 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.167 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.168 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.168 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.168 DEBUG:boto:Data: 2024-04-16T19:26:50.168 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.168 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.168 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.168 DEBUG:boto:Params: {} 2024-04-16T19:26:50.168 DEBUG:boto:Token: None 2024-04-16T19:26:50.168 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0881a826cebd79c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.171 DEBUG:boto:path=/yrjefh-11/obj1 2024-04-16T19:26:50.171 DEBUG:boto:auth_path=/yrjefh-11/obj1 2024-04-16T19:26:50.172 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.172 DEBUG:boto:Path: /yrjefh-11/obj1 2024-04-16T19:26:50.172 DEBUG:boto:Data: 2024-04-16T19:26:50.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.172 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.172 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.172 DEBUG:boto:Params: {} 2024-04-16T19:26:50.172 DEBUG:boto:Token: None 2024-04-16T19:26:50.172 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj1 2024-04-16T19:26:50.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H+PMlk07VEbYt9nkOUwTYEF0FeU= 2024-04-16T19:26:50.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H+PMlk07VEbYt9nkOUwTYEF0FeU='} 2024-04-16T19:26:50.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004da4898423efa264-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.215 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.215 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.215 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.215 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.215 DEBUG:boto:Data: 2024-04-16T19:26:50.215 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.215 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.215 DEBUG:boto:Params: {} 2024-04-16T19:26:50.215 DEBUG:boto:Token: None 2024-04-16T19:26:50.216 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff258eeef90375ef-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.218 DEBUG:boto:path=/yrjefh-11/obj2 2024-04-16T19:26:50.218 DEBUG:boto:auth_path=/yrjefh-11/obj2 2024-04-16T19:26:50.218 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.218 DEBUG:boto:Path: /yrjefh-11/obj2 2024-04-16T19:26:50.218 DEBUG:boto:Data: 2024-04-16T19:26:50.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.219 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.219 DEBUG:boto:Params: {} 2024-04-16T19:26:50.219 DEBUG:boto:Token: None 2024-04-16T19:26:50.219 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj2 2024-04-16T19:26:50.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OOsLE9mFlyRMMQOXi+dwYOWDWEE= 2024-04-16T19:26:50.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OOsLE9mFlyRMMQOXi+dwYOWDWEE='} 2024-04-16T19:26:50.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af4f3a64b7cb0f8f-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.226 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.226 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.226 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.226 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.226 DEBUG:boto:Data: 2024-04-16T19:26:50.226 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.227 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.227 DEBUG:boto:Params: {} 2024-04-16T19:26:50.227 DEBUG:boto:Token: None 2024-04-16T19:26:50.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e6f4c3e40316d07-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.229 DEBUG:boto:path=/yrjefh-11/obj3 2024-04-16T19:26:50.230 DEBUG:boto:auth_path=/yrjefh-11/obj3 2024-04-16T19:26:50.230 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.230 DEBUG:boto:Path: /yrjefh-11/obj3 2024-04-16T19:26:50.230 DEBUG:boto:Data: 2024-04-16T19:26:50.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.230 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.230 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.230 DEBUG:boto:Params: {} 2024-04-16T19:26:50.230 DEBUG:boto:Token: None 2024-04-16T19:26:50.230 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj3 2024-04-16T19:26:50.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zIbASO3lho5H1JJySGPFFAKIIoU= 2024-04-16T19:26:50.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zIbASO3lho5H1JJySGPFFAKIIoU='} 2024-04-16T19:26:50.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013c9d9b502c0e344-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.237 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.237 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.237 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.237 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.237 DEBUG:boto:Data: 2024-04-16T19:26:50.237 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.237 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.237 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.237 DEBUG:boto:Params: {} 2024-04-16T19:26:50.237 DEBUG:boto:Token: None 2024-04-16T19:26:50.237 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081202dd1cac1c30c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.240 DEBUG:boto:path=/yrjefh-11/obj4 2024-04-16T19:26:50.240 DEBUG:boto:auth_path=/yrjefh-11/obj4 2024-04-16T19:26:50.240 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.240 DEBUG:boto:Path: /yrjefh-11/obj4 2024-04-16T19:26:50.240 DEBUG:boto:Data: 2024-04-16T19:26:50.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.240 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.240 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.241 DEBUG:boto:Params: {} 2024-04-16T19:26:50.241 DEBUG:boto:Token: None 2024-04-16T19:26:50.241 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj4 2024-04-16T19:26:50.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6SzmzIxv+qFTNGP2ANxciNGnxV8= 2024-04-16T19:26:50.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6SzmzIxv+qFTNGP2ANxciNGnxV8='} 2024-04-16T19:26:50.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd6c419ca6e11ddf-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.283 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.283 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.283 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.283 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.283 DEBUG:boto:Data: 2024-04-16T19:26:50.283 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.283 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.283 DEBUG:boto:Params: {} 2024-04-16T19:26:50.283 DEBUG:boto:Token: None 2024-04-16T19:26:50.283 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021418df9289a1a62-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.286 DEBUG:boto:path=/yrjefh-11/obj5 2024-04-16T19:26:50.286 DEBUG:boto:auth_path=/yrjefh-11/obj5 2024-04-16T19:26:50.286 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.286 DEBUG:boto:Path: /yrjefh-11/obj5 2024-04-16T19:26:50.286 DEBUG:boto:Data: 2024-04-16T19:26:50.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.287 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.287 DEBUG:boto:Params: {} 2024-04-16T19:26:50.287 DEBUG:boto:Token: None 2024-04-16T19:26:50.287 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj5 2024-04-16T19:26:50.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2+fUM5Puvz26T3V+qWA1Gli37y8= 2024-04-16T19:26:50.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2+fUM5Puvz26T3V+qWA1Gli37y8='} 2024-04-16T19:26:50.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c975a4ec6f6a10ff-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.292 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.292 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.292 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.292 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.292 DEBUG:boto:Data: 2024-04-16T19:26:50.292 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.292 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.292 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.292 DEBUG:boto:Params: {} 2024-04-16T19:26:50.293 DEBUG:boto:Token: None 2024-04-16T19:26:50.293 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055ff5c8e082ec9da-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.295 DEBUG:boto:path=/yrjefh-11/obj6 2024-04-16T19:26:50.295 DEBUG:boto:auth_path=/yrjefh-11/obj6 2024-04-16T19:26:50.295 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.295 DEBUG:boto:Path: /yrjefh-11/obj6 2024-04-16T19:26:50.295 DEBUG:boto:Data: 2024-04-16T19:26:50.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.296 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.296 DEBUG:boto:Params: {} 2024-04-16T19:26:50.296 DEBUG:boto:Token: None 2024-04-16T19:26:50.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj6 2024-04-16T19:26:50.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gUtj3p+S8NxKoDWjwkBrznYfeMY= 2024-04-16T19:26:50.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gUtj3p+S8NxKoDWjwkBrznYfeMY='} 2024-04-16T19:26:50.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021f24912cb6e3cf6-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.339 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.339 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.339 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.339 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.339 DEBUG:boto:Data: 2024-04-16T19:26:50.339 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.339 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.339 DEBUG:boto:Params: {} 2024-04-16T19:26:50.340 DEBUG:boto:Token: None 2024-04-16T19:26:50.340 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6e6a8c9470e5d7c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.342 DEBUG:boto:path=/yrjefh-11/obj7 2024-04-16T19:26:50.342 DEBUG:boto:auth_path=/yrjefh-11/obj7 2024-04-16T19:26:50.343 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.343 DEBUG:boto:Path: /yrjefh-11/obj7 2024-04-16T19:26:50.343 DEBUG:boto:Data: 2024-04-16T19:26:50.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.343 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.343 DEBUG:boto:Params: {} 2024-04-16T19:26:50.343 DEBUG:boto:Token: None 2024-04-16T19:26:50.343 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj7 2024-04-16T19:26:50.344 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SMnYfVAc4YT0uaSuDDMehFQR1lg= 2024-04-16T19:26:50.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SMnYfVAc4YT0uaSuDDMehFQR1lg='} 2024-04-16T19:26:50.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4c70df07dafd5d2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.349 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.349 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.349 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.349 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.349 DEBUG:boto:Data: 2024-04-16T19:26:50.349 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.349 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.349 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.349 DEBUG:boto:Params: {} 2024-04-16T19:26:50.349 DEBUG:boto:Token: None 2024-04-16T19:26:50.350 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.350 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003410257b4fdb8c7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.352 DEBUG:boto:path=/yrjefh-11/obj8 2024-04-16T19:26:50.352 DEBUG:boto:auth_path=/yrjefh-11/obj8 2024-04-16T19:26:50.352 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.352 DEBUG:boto:Path: /yrjefh-11/obj8 2024-04-16T19:26:50.352 DEBUG:boto:Data: 2024-04-16T19:26:50.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.352 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.353 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.353 DEBUG:boto:Params: {} 2024-04-16T19:26:50.353 DEBUG:boto:Token: None 2024-04-16T19:26:50.353 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj8 2024-04-16T19:26:50.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:meg2u+c5qQ9U/oL+6LQt79XhDCU= 2024-04-16T19:26:50.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:meg2u+c5qQ9U/oL+6LQt79XhDCU='} 2024-04-16T19:26:50.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057ff3e738cbe88be-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.357 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.357 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.358 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.358 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.358 DEBUG:boto:Data: 2024-04-16T19:26:50.358 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.358 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.358 DEBUG:boto:Params: {} 2024-04-16T19:26:50.358 DEBUG:boto:Token: None 2024-04-16T19:26:50.358 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076a2372e4d2280df-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.361 DEBUG:boto:path=/yrjefh-11/obj9 2024-04-16T19:26:50.361 DEBUG:boto:auth_path=/yrjefh-11/obj9 2024-04-16T19:26:50.361 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.361 DEBUG:boto:Path: /yrjefh-11/obj9 2024-04-16T19:26:50.361 DEBUG:boto:Data: 2024-04-16T19:26:50.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.362 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.362 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.362 DEBUG:boto:Params: {} 2024-04-16T19:26:50.362 DEBUG:boto:Token: None 2024-04-16T19:26:50.362 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj9 2024-04-16T19:26:50.362 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:itVjPyJXUGzRUbI9xzweYsBvZCc= 2024-04-16T19:26:50.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:itVjPyJXUGzRUbI9xzweYsBvZCc='} 2024-04-16T19:26:50.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000220a26c288c561ef-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.407 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.407 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.407 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.407 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.407 DEBUG:boto:Data: 2024-04-16T19:26:50.407 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.407 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.407 DEBUG:boto:Params: {} 2024-04-16T19:26:50.407 DEBUG:boto:Token: None 2024-04-16T19:26:50.408 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1b9cd4f73fc5ce-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.410 DEBUG:boto:path=/yrjefh-11/obj10 2024-04-16T19:26:50.410 DEBUG:boto:auth_path=/yrjefh-11/obj10 2024-04-16T19:26:50.410 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.410 DEBUG:boto:Path: /yrjefh-11/obj10 2024-04-16T19:26:50.411 DEBUG:boto:Data: 2024-04-16T19:26:50.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.411 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.411 DEBUG:boto:Params: {} 2024-04-16T19:26:50.411 DEBUG:boto:Token: None 2024-04-16T19:26:50.411 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj10 2024-04-16T19:26:50.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CSGlPy4oKPkBwnXIgO2xvmPJ1nA= 2024-04-16T19:26:50.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CSGlPy4oKPkBwnXIgO2xvmPJ1nA='} 2024-04-16T19:26:50.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4309ffdd7efad61-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.416 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.416 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.416 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.416 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.416 DEBUG:boto:Data: 2024-04-16T19:26:50.416 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.417 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.417 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.417 DEBUG:boto:Params: {} 2024-04-16T19:26:50.417 DEBUG:boto:Token: None 2024-04-16T19:26:50.417 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bd1770dba7aff6b-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.419 DEBUG:boto:path=/yrjefh-11/obj11 2024-04-16T19:26:50.420 DEBUG:boto:auth_path=/yrjefh-11/obj11 2024-04-16T19:26:50.420 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.420 DEBUG:boto:Path: /yrjefh-11/obj11 2024-04-16T19:26:50.420 DEBUG:boto:Data: 2024-04-16T19:26:50.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.420 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.420 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.420 DEBUG:boto:Params: {} 2024-04-16T19:26:50.420 DEBUG:boto:Token: None 2024-04-16T19:26:50.420 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj11 2024-04-16T19:26:50.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s9qXXbuQYX67yfyQL4dgPtdACBQ= 2024-04-16T19:26:50.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s9qXXbuQYX67yfyQL4dgPtdACBQ='} 2024-04-16T19:26:50.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f18f628432dfd011-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.463 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.463 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.463 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.463 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.463 DEBUG:boto:Data: 2024-04-16T19:26:50.463 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.463 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.463 DEBUG:boto:Params: {} 2024-04-16T19:26:50.463 DEBUG:boto:Token: None 2024-04-16T19:26:50.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006562266b8809cdcf-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.466 DEBUG:boto:path=/yrjefh-11/obj12 2024-04-16T19:26:50.466 DEBUG:boto:auth_path=/yrjefh-11/obj12 2024-04-16T19:26:50.466 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.467 DEBUG:boto:Path: /yrjefh-11/obj12 2024-04-16T19:26:50.467 DEBUG:boto:Data: 2024-04-16T19:26:50.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.467 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.467 DEBUG:boto:Params: {} 2024-04-16T19:26:50.467 DEBUG:boto:Token: None 2024-04-16T19:26:50.467 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj12 2024-04-16T19:26:50.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kUBn6eAC0uVd6ulI6KJQUqryjSI= 2024-04-16T19:26:50.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kUBn6eAC0uVd6ulI6KJQUqryjSI='} 2024-04-16T19:26:50.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e24a70f5815601b7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.472 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.472 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.472 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.473 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.473 DEBUG:boto:Data: 2024-04-16T19:26:50.473 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.473 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.473 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.473 DEBUG:boto:Params: {} 2024-04-16T19:26:50.473 DEBUG:boto:Token: None 2024-04-16T19:26:50.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e90e77e3458b91ce-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.476 DEBUG:boto:path=/yrjefh-11/obj13 2024-04-16T19:26:50.476 DEBUG:boto:auth_path=/yrjefh-11/obj13 2024-04-16T19:26:50.476 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.476 DEBUG:boto:Path: /yrjefh-11/obj13 2024-04-16T19:26:50.476 DEBUG:boto:Data: 2024-04-16T19:26:50.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.476 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.477 DEBUG:boto:Params: {} 2024-04-16T19:26:50.477 DEBUG:boto:Token: None 2024-04-16T19:26:50.477 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj13 2024-04-16T19:26:50.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zyB4icBzPYCmJO64sveJMfWMRV4= 2024-04-16T19:26:50.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zyB4icBzPYCmJO64sveJMfWMRV4='} 2024-04-16T19:26:50.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df6548f4e79feb74-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.523 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.523 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.523 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.523 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.523 DEBUG:boto:Data: 2024-04-16T19:26:50.523 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.523 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.523 DEBUG:boto:Params: {} 2024-04-16T19:26:50.523 DEBUG:boto:Token: None 2024-04-16T19:26:50.523 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c035391ccb9f6d61-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.526 DEBUG:boto:path=/yrjefh-11/obj14 2024-04-16T19:26:50.526 DEBUG:boto:auth_path=/yrjefh-11/obj14 2024-04-16T19:26:50.526 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.526 DEBUG:boto:Path: /yrjefh-11/obj14 2024-04-16T19:26:50.526 DEBUG:boto:Data: 2024-04-16T19:26:50.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.527 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.527 DEBUG:boto:Params: {} 2024-04-16T19:26:50.527 DEBUG:boto:Token: None 2024-04-16T19:26:50.527 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj14 2024-04-16T19:26:50.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:35WWd0N6er8aiUr1S8ABcNEdLZs= 2024-04-16T19:26:50.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35WWd0N6er8aiUr1S8ABcNEdLZs='} 2024-04-16T19:26:50.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008de1c9640d54bffb-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.531 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.532 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.532 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.532 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.532 DEBUG:boto:Data: 2024-04-16T19:26:50.532 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.532 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.532 DEBUG:boto:Params: {} 2024-04-16T19:26:50.532 DEBUG:boto:Token: None 2024-04-16T19:26:50.532 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006552993c0e4baf84-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.535 DEBUG:boto:path=/yrjefh-11/obj15 2024-04-16T19:26:50.535 DEBUG:boto:auth_path=/yrjefh-11/obj15 2024-04-16T19:26:50.535 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.535 DEBUG:boto:Path: /yrjefh-11/obj15 2024-04-16T19:26:50.535 DEBUG:boto:Data: 2024-04-16T19:26:50.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.536 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.536 DEBUG:boto:Params: {} 2024-04-16T19:26:50.536 DEBUG:boto:Token: None 2024-04-16T19:26:50.536 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj15 2024-04-16T19:26:50.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M4wlcANluCLUBZE1SbI3kgV7sf0= 2024-04-16T19:26:50.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M4wlcANluCLUBZE1SbI3kgV7sf0='} 2024-04-16T19:26:50.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6636f0d7a94ba43-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.579 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.579 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.579 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.579 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.579 DEBUG:boto:Data: 2024-04-16T19:26:50.579 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.579 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.579 DEBUG:boto:Params: {} 2024-04-16T19:26:50.579 DEBUG:boto:Token: None 2024-04-16T19:26:50.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd746df397ef182d-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.582 DEBUG:boto:path=/yrjefh-11/obj16 2024-04-16T19:26:50.583 DEBUG:boto:auth_path=/yrjefh-11/obj16 2024-04-16T19:26:50.583 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.583 DEBUG:boto:Path: /yrjefh-11/obj16 2024-04-16T19:26:50.583 DEBUG:boto:Data: 2024-04-16T19:26:50.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.583 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.583 DEBUG:boto:Params: {} 2024-04-16T19:26:50.583 DEBUG:boto:Token: None 2024-04-16T19:26:50.583 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj16 2024-04-16T19:26:50.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8RP78nYp/BgXZIb7XHfBN27SpnQ= 2024-04-16T19:26:50.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8RP78nYp/BgXZIb7XHfBN27SpnQ='} 2024-04-16T19:26:50.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019a3ec9a4b75d01c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.590 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.590 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.590 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.590 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.590 DEBUG:boto:Data: 2024-04-16T19:26:50.590 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.590 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.590 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.591 DEBUG:boto:Params: {} 2024-04-16T19:26:50.591 DEBUG:boto:Token: None 2024-04-16T19:26:50.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007978e01f77a2ac72-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.593 DEBUG:boto:path=/yrjefh-11/obj17 2024-04-16T19:26:50.593 DEBUG:boto:auth_path=/yrjefh-11/obj17 2024-04-16T19:26:50.594 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.594 DEBUG:boto:Path: /yrjefh-11/obj17 2024-04-16T19:26:50.594 DEBUG:boto:Data: 2024-04-16T19:26:50.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.594 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.594 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.594 DEBUG:boto:Params: {} 2024-04-16T19:26:50.594 DEBUG:boto:Token: None 2024-04-16T19:26:50.594 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj17 2024-04-16T19:26:50.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EKUd4UGR0ydeSK4SEf6IzToMNiY= 2024-04-16T19:26:50.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EKUd4UGR0ydeSK4SEf6IzToMNiY='} 2024-04-16T19:26:50.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006026850401352a06-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.638 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.639 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.639 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.639 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.639 DEBUG:boto:Data: 2024-04-16T19:26:50.639 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.639 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.639 DEBUG:boto:Params: {} 2024-04-16T19:26:50.639 DEBUG:boto:Token: None 2024-04-16T19:26:50.639 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beebedb56ad967fd-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.642 DEBUG:boto:path=/yrjefh-11/obj18 2024-04-16T19:26:50.642 DEBUG:boto:auth_path=/yrjefh-11/obj18 2024-04-16T19:26:50.642 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.642 DEBUG:boto:Path: /yrjefh-11/obj18 2024-04-16T19:26:50.642 DEBUG:boto:Data: 2024-04-16T19:26:50.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.642 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.643 DEBUG:boto:Params: {} 2024-04-16T19:26:50.643 DEBUG:boto:Token: None 2024-04-16T19:26:50.643 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj18 2024-04-16T19:26:50.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GRGYeWa6pCMZrGzp+MlFpIdf1q4= 2024-04-16T19:26:50.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRGYeWa6pCMZrGzp+MlFpIdf1q4='} 2024-04-16T19:26:50.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009922147c34580a1b-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.648 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.648 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.648 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.648 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.648 DEBUG:boto:Data: 2024-04-16T19:26:50.648 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.648 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.648 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.649 DEBUG:boto:Params: {} 2024-04-16T19:26:50.649 DEBUG:boto:Token: None 2024-04-16T19:26:50.649 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015042aa8b509270f-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.651 DEBUG:boto:path=/yrjefh-11/obj19 2024-04-16T19:26:50.651 DEBUG:boto:auth_path=/yrjefh-11/obj19 2024-04-16T19:26:50.651 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.651 DEBUG:boto:Path: /yrjefh-11/obj19 2024-04-16T19:26:50.651 DEBUG:boto:Data: 2024-04-16T19:26:50.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.652 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.652 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.652 DEBUG:boto:Params: {} 2024-04-16T19:26:50.652 DEBUG:boto:Token: None 2024-04-16T19:26:50.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj19 2024-04-16T19:26:50.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lrsmStDMJ+9s/Wj+9Bu48Qe7Xzs= 2024-04-16T19:26:50.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lrsmStDMJ+9s/Wj+9Bu48Qe7Xzs='} 2024-04-16T19:26:50.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cbcda8a726c6aa9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.695 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.695 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.695 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.695 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.695 DEBUG:boto:Data: 2024-04-16T19:26:50.695 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.695 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.695 DEBUG:boto:Params: {} 2024-04-16T19:26:50.696 DEBUG:boto:Token: None 2024-04-16T19:26:50.696 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1f21b5b73fed0ba-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.699 DEBUG:boto:path=/yrjefh-11/obj20 2024-04-16T19:26:50.699 DEBUG:boto:auth_path=/yrjefh-11/obj20 2024-04-16T19:26:50.699 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.699 DEBUG:boto:Path: /yrjefh-11/obj20 2024-04-16T19:26:50.699 DEBUG:boto:Data: 2024-04-16T19:26:50.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.699 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.699 DEBUG:boto:Params: {} 2024-04-16T19:26:50.699 DEBUG:boto:Token: None 2024-04-16T19:26:50.700 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj20 2024-04-16T19:26:50.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ls3RTUi5Zc2pnhAJF++XdhuFIPU= 2024-04-16T19:26:50.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ls3RTUi5Zc2pnhAJF++XdhuFIPU='} 2024-04-16T19:26:50.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44efa74e900b8e6-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.706 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.706 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.706 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.706 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.706 DEBUG:boto:Data: 2024-04-16T19:26:50.706 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.706 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.706 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.707 DEBUG:boto:Params: {} 2024-04-16T19:26:50.707 DEBUG:boto:Token: None 2024-04-16T19:26:50.707 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7d82dfcc409927e-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.711 DEBUG:boto:path=/yrjefh-11/obj21 2024-04-16T19:26:50.711 DEBUG:boto:auth_path=/yrjefh-11/obj21 2024-04-16T19:26:50.711 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.711 DEBUG:boto:Path: /yrjefh-11/obj21 2024-04-16T19:26:50.711 DEBUG:boto:Data: 2024-04-16T19:26:50.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.712 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.712 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.712 DEBUG:boto:Params: {} 2024-04-16T19:26:50.712 DEBUG:boto:Token: None 2024-04-16T19:26:50.712 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj21 2024-04-16T19:26:50.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bWlnCPY5oFlEtJ4U0wdjs7NwjAw= 2024-04-16T19:26:50.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bWlnCPY5oFlEtJ4U0wdjs7NwjAw='} 2024-04-16T19:26:50.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000c29c3ece3567fe-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.755 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.755 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.755 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.756 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.756 DEBUG:boto:Data: 2024-04-16T19:26:50.756 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.756 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.756 DEBUG:boto:Params: {} 2024-04-16T19:26:50.756 DEBUG:boto:Token: None 2024-04-16T19:26:50.756 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa89db627720a518-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.759 DEBUG:boto:path=/yrjefh-11/obj22 2024-04-16T19:26:50.759 DEBUG:boto:auth_path=/yrjefh-11/obj22 2024-04-16T19:26:50.760 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.760 DEBUG:boto:Path: /yrjefh-11/obj22 2024-04-16T19:26:50.760 DEBUG:boto:Data: 2024-04-16T19:26:50.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.760 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.760 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.760 DEBUG:boto:Params: {} 2024-04-16T19:26:50.760 DEBUG:boto:Token: None 2024-04-16T19:26:50.760 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj22 2024-04-16T19:26:50.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c5BQ6mUaFpKx3wsGPUXzI5JnUZo= 2024-04-16T19:26:50.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c5BQ6mUaFpKx3wsGPUXzI5JnUZo='} 2024-04-16T19:26:50.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c012246ace0a916-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.765 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.765 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.765 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.765 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.765 DEBUG:boto:Data: 2024-04-16T19:26:50.765 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.766 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.766 DEBUG:boto:Params: {} 2024-04-16T19:26:50.766 DEBUG:boto:Token: None 2024-04-16T19:26:50.766 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000840b6fb7057b42e2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.769 DEBUG:boto:path=/yrjefh-11/obj23 2024-04-16T19:26:50.769 DEBUG:boto:auth_path=/yrjefh-11/obj23 2024-04-16T19:26:50.770 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.770 DEBUG:boto:Path: /yrjefh-11/obj23 2024-04-16T19:26:50.770 DEBUG:boto:Data: 2024-04-16T19:26:50.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.770 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.770 DEBUG:boto:Params: {} 2024-04-16T19:26:50.770 DEBUG:boto:Token: None 2024-04-16T19:26:50.770 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj23 2024-04-16T19:26:50.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7ULAnS06NKgMfq0VFD7QecvlcwY= 2024-04-16T19:26:50.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7ULAnS06NKgMfq0VFD7QecvlcwY='} 2024-04-16T19:26:50.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d2282ccd4e97843-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.816 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.816 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.817 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.817 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.817 DEBUG:boto:Data: 2024-04-16T19:26:50.817 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.817 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.817 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.817 DEBUG:boto:Params: {} 2024-04-16T19:26:50.817 DEBUG:boto:Token: None 2024-04-16T19:26:50.817 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.817 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a200e2ccc852e7f9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.820 DEBUG:boto:path=/yrjefh-11/obj24 2024-04-16T19:26:50.820 DEBUG:boto:auth_path=/yrjefh-11/obj24 2024-04-16T19:26:50.820 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.821 DEBUG:boto:Path: /yrjefh-11/obj24 2024-04-16T19:26:50.821 DEBUG:boto:Data: 2024-04-16T19:26:50.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.821 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.821 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.821 DEBUG:boto:Params: {} 2024-04-16T19:26:50.821 DEBUG:boto:Token: None 2024-04-16T19:26:50.821 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj24 2024-04-16T19:26:50.821 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kr/04c1a8D9n36uZXMoOnYpjrPw= 2024-04-16T19:26:50.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kr/04c1a8D9n36uZXMoOnYpjrPw='} 2024-04-16T19:26:50.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2fcd22a9ca34491-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.827 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.827 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.827 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.827 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.827 DEBUG:boto:Data: 2024-04-16T19:26:50.827 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.828 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.828 DEBUG:boto:Params: {} 2024-04-16T19:26:50.828 DEBUG:boto:Token: None 2024-04-16T19:26:50.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dddecbdd24d61dc-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.831 DEBUG:boto:path=/yrjefh-11/obj25 2024-04-16T19:26:50.831 DEBUG:boto:auth_path=/yrjefh-11/obj25 2024-04-16T19:26:50.831 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.831 DEBUG:boto:Path: /yrjefh-11/obj25 2024-04-16T19:26:50.832 DEBUG:boto:Data: 2024-04-16T19:26:50.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.832 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.832 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.832 DEBUG:boto:Params: {} 2024-04-16T19:26:50.832 DEBUG:boto:Token: None 2024-04-16T19:26:50.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj25 2024-04-16T19:26:50.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tZLjaP6q1RYk6Gr0UfpWrb1dyHk= 2024-04-16T19:26:50.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tZLjaP6q1RYk6Gr0UfpWrb1dyHk='} 2024-04-16T19:26:50.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3d5ba4e3a2079c9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.879 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.880 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.880 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.881 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.881 DEBUG:boto:Data: 2024-04-16T19:26:50.881 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.882 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.882 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.882 DEBUG:boto:Params: {} 2024-04-16T19:26:50.883 DEBUG:boto:Token: None 2024-04-16T19:26:50.883 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000681847a383af12c7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.888 DEBUG:boto:path=/yrjefh-11/obj26 2024-04-16T19:26:50.888 DEBUG:boto:auth_path=/yrjefh-11/obj26 2024-04-16T19:26:50.889 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.889 DEBUG:boto:Path: /yrjefh-11/obj26 2024-04-16T19:26:50.889 DEBUG:boto:Data: 2024-04-16T19:26:50.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.890 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.890 DEBUG:boto:Params: {} 2024-04-16T19:26:50.890 DEBUG:boto:Token: None 2024-04-16T19:26:50.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj26 2024-04-16T19:26:50.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jsFBx1EVuI1XgQm0DeIUerLQ4FQ= 2024-04-16T19:26:50.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jsFBx1EVuI1XgQm0DeIUerLQ4FQ='} 2024-04-16T19:26:50.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b0c3eee95c0e8ab-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.896 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.896 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.897 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.897 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.897 DEBUG:boto:Data: 2024-04-16T19:26:50.897 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.897 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.897 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.897 DEBUG:boto:Params: {} 2024-04-16T19:26:50.897 DEBUG:boto:Token: None 2024-04-16T19:26:50.897 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000243c32de476ef6a2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.900 DEBUG:boto:path=/yrjefh-11/obj27 2024-04-16T19:26:50.900 DEBUG:boto:auth_path=/yrjefh-11/obj27 2024-04-16T19:26:50.900 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.900 DEBUG:boto:Path: /yrjefh-11/obj27 2024-04-16T19:26:50.900 DEBUG:boto:Data: 2024-04-16T19:26:50.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.901 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.901 DEBUG:boto:Params: {} 2024-04-16T19:26:50.901 DEBUG:boto:Token: None 2024-04-16T19:26:50.901 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj27 2024-04-16T19:26:50.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pZ74elxSY5R6qcHB+RQ8RFC3YCk= 2024-04-16T19:26:50.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pZ74elxSY5R6qcHB+RQ8RFC3YCk='} 2024-04-16T19:26:50.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052fd208104cec402-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.947 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.947 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.947 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.947 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.947 DEBUG:boto:Data: 2024-04-16T19:26:50.948 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.948 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.948 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.948 DEBUG:boto:Params: {} 2024-04-16T19:26:50.948 DEBUG:boto:Token: None 2024-04-16T19:26:50.948 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f50063c2052ce985-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.951 DEBUG:boto:path=/yrjefh-11/obj28 2024-04-16T19:26:50.951 DEBUG:boto:auth_path=/yrjefh-11/obj28 2024-04-16T19:26:50.951 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.951 DEBUG:boto:Path: /yrjefh-11/obj28 2024-04-16T19:26:50.952 DEBUG:boto:Data: 2024-04-16T19:26:50.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.952 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.952 DEBUG:boto:Params: {} 2024-04-16T19:26:50.952 DEBUG:boto:Token: None 2024-04-16T19:26:50.952 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj28 2024-04-16T19:26:50.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wB/Y8+56N8Fa4RCR4jmtqIoRD3w= 2024-04-16T19:26:50.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wB/Y8+56N8Fa4RCR4jmtqIoRD3w='} 2024-04-16T19:26:50.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067fc803444995def-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.958 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:50.958 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:50.959 DEBUG:boto:Method: HEAD 2024-04-16T19:26:50.959 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:50.959 DEBUG:boto:Data: 2024-04-16T19:26:50.959 DEBUG:boto:Headers: {} 2024-04-16T19:26:50.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.959 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.959 DEBUG:boto:Params: {} 2024-04-16T19:26:50.959 DEBUG:boto:Token: None 2024-04-16T19:26:50.959 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/ 2024-04-16T19:26:50.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T19:26:50.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T19:26:50.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3c6589116b14dc-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:50.962 DEBUG:boto:path=/yrjefh-11/obj29 2024-04-16T19:26:50.962 DEBUG:boto:auth_path=/yrjefh-11/obj29 2024-04-16T19:26:50.962 DEBUG:boto:Method: PUT 2024-04-16T19:26:50.962 DEBUG:boto:Path: /yrjefh-11/obj29 2024-04-16T19:26:50.963 DEBUG:boto:Data: 2024-04-16T19:26:50.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:50.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:50.963 DEBUG:boto:Port: 8000 2024-04-16T19:26:50.963 DEBUG:boto:Params: {} 2024-04-16T19:26:50.963 DEBUG:boto:Token: None 2024-04-16T19:26:50.963 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:50 GMT /yrjefh-11/obj29 2024-04-16T19:26:50.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YA/tqq/KO1e62BYkkAe5ulILFZ8= 2024-04-16T19:26:50.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YA/tqq/KO1e62BYkkAe5ulILFZ8='} 2024-04-16T19:26:51.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd09747b9b5a4100-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.008 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.008 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.008 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.008 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.008 DEBUG:boto:Data: 2024-04-16T19:26:51.008 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.008 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.008 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.008 DEBUG:boto:Params: {} 2024-04-16T19:26:51.009 DEBUG:boto:Token: None 2024-04-16T19:26:51.009 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.009 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000820002739544a41-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.012 DEBUG:boto:path=/yrjefh-11/obj30 2024-04-16T19:26:51.012 DEBUG:boto:auth_path=/yrjefh-11/obj30 2024-04-16T19:26:51.012 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.012 DEBUG:boto:Path: /yrjefh-11/obj30 2024-04-16T19:26:51.012 DEBUG:boto:Data: 2024-04-16T19:26:51.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.013 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.013 DEBUG:boto:Params: {} 2024-04-16T19:26:51.013 DEBUG:boto:Token: None 2024-04-16T19:26:51.013 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj30 2024-04-16T19:26:51.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jVZK/lY9PstIy9sVRnlHVUzZ7P4= 2024-04-16T19:26:51.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jVZK/lY9PstIy9sVRnlHVUzZ7P4='} 2024-04-16T19:26:51.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000568185cb48e92ea7-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.018 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.019 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.019 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.019 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.019 DEBUG:boto:Data: 2024-04-16T19:26:51.019 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.019 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.019 DEBUG:boto:Params: {} 2024-04-16T19:26:51.019 DEBUG:boto:Token: None 2024-04-16T19:26:51.019 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b87edf8b4113cb42-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.023 DEBUG:boto:path=/yrjefh-11/obj31 2024-04-16T19:26:51.024 DEBUG:boto:auth_path=/yrjefh-11/obj31 2024-04-16T19:26:51.024 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.024 DEBUG:boto:Path: /yrjefh-11/obj31 2024-04-16T19:26:51.024 DEBUG:boto:Data: 2024-04-16T19:26:51.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.024 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.024 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.024 DEBUG:boto:Params: {} 2024-04-16T19:26:51.024 DEBUG:boto:Token: None 2024-04-16T19:26:51.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj31 2024-04-16T19:26:51.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q0/Pt+z6Xn1+osiNP6y8gSDeHAE= 2024-04-16T19:26:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q0/Pt+z6Xn1+osiNP6y8gSDeHAE='} 2024-04-16T19:26:51.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3ce578c90760054-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.071 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.071 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.071 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.071 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.071 DEBUG:boto:Data: 2024-04-16T19:26:51.071 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.072 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.072 DEBUG:boto:Params: {} 2024-04-16T19:26:51.072 DEBUG:boto:Token: None 2024-04-16T19:26:51.072 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c7caac0276a4c6d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.075 DEBUG:boto:path=/yrjefh-11/obj32 2024-04-16T19:26:51.075 DEBUG:boto:auth_path=/yrjefh-11/obj32 2024-04-16T19:26:51.075 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.076 DEBUG:boto:Path: /yrjefh-11/obj32 2024-04-16T19:26:51.076 DEBUG:boto:Data: 2024-04-16T19:26:51.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.076 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.076 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.076 DEBUG:boto:Params: {} 2024-04-16T19:26:51.076 DEBUG:boto:Token: None 2024-04-16T19:26:51.076 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj32 2024-04-16T19:26:51.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:COZdk1jOfOXjyqxBW/jGES1W5/g= 2024-04-16T19:26:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:COZdk1jOfOXjyqxBW/jGES1W5/g='} 2024-04-16T19:26:51.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058fab48682da643e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.082 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.082 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.082 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.082 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.082 DEBUG:boto:Data: 2024-04-16T19:26:51.082 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.082 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.083 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.083 DEBUG:boto:Params: {} 2024-04-16T19:26:51.083 DEBUG:boto:Token: None 2024-04-16T19:26:51.083 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f77d33ae9bba893-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.086 DEBUG:boto:path=/yrjefh-11/obj33 2024-04-16T19:26:51.086 DEBUG:boto:auth_path=/yrjefh-11/obj33 2024-04-16T19:26:51.086 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.086 DEBUG:boto:Path: /yrjefh-11/obj33 2024-04-16T19:26:51.086 DEBUG:boto:Data: 2024-04-16T19:26:51.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.087 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.087 DEBUG:boto:Params: {} 2024-04-16T19:26:51.087 DEBUG:boto:Token: None 2024-04-16T19:26:51.087 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj33 2024-04-16T19:26:51.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2uMytdFc0mRquimE1p+ZoFPdLLk= 2024-04-16T19:26:51.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2uMytdFc0mRquimE1p+ZoFPdLLk='} 2024-04-16T19:26:51.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb964f8024248a40-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.131 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.131 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.131 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.131 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.131 DEBUG:boto:Data: 2024-04-16T19:26:51.131 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.131 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.132 DEBUG:boto:Params: {} 2024-04-16T19:26:51.132 DEBUG:boto:Token: None 2024-04-16T19:26:51.132 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c4808f06965f544-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.134 DEBUG:boto:path=/yrjefh-11/obj34 2024-04-16T19:26:51.134 DEBUG:boto:auth_path=/yrjefh-11/obj34 2024-04-16T19:26:51.135 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.135 DEBUG:boto:Path: /yrjefh-11/obj34 2024-04-16T19:26:51.135 DEBUG:boto:Data: 2024-04-16T19:26:51.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.135 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.136 DEBUG:boto:Params: {} 2024-04-16T19:26:51.136 DEBUG:boto:Token: None 2024-04-16T19:26:51.136 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj34 2024-04-16T19:26:51.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t2WCYni02X4ug9QHmC+BSXlbBa0= 2024-04-16T19:26:51.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t2WCYni02X4ug9QHmC+BSXlbBa0='} 2024-04-16T19:26:51.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f0b89af60cafcb9-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.141 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.141 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.141 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.141 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.141 DEBUG:boto:Data: 2024-04-16T19:26:51.141 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.141 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.141 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.141 DEBUG:boto:Params: {} 2024-04-16T19:26:51.141 DEBUG:boto:Token: None 2024-04-16T19:26:51.142 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc95099407256ac-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.144 DEBUG:boto:path=/yrjefh-11/obj35 2024-04-16T19:26:51.144 DEBUG:boto:auth_path=/yrjefh-11/obj35 2024-04-16T19:26:51.144 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.144 DEBUG:boto:Path: /yrjefh-11/obj35 2024-04-16T19:26:51.144 DEBUG:boto:Data: 2024-04-16T19:26:51.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.144 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.145 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.145 DEBUG:boto:Params: {} 2024-04-16T19:26:51.145 DEBUG:boto:Token: None 2024-04-16T19:26:51.145 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj35 2024-04-16T19:26:51.145 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T2KxNjm+r9ZpY4jxOOSsHgaiDOo= 2024-04-16T19:26:51.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T2KxNjm+r9ZpY4jxOOSsHgaiDOo='} 2024-04-16T19:26:51.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006025c2bc1d922935-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.149 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.150 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.150 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.150 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.150 DEBUG:boto:Data: 2024-04-16T19:26:51.150 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.150 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.150 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.150 DEBUG:boto:Params: {} 2024-04-16T19:26:51.150 DEBUG:boto:Token: None 2024-04-16T19:26:51.150 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e23a8f315c3e1dd-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.153 DEBUG:boto:path=/yrjefh-11/obj36 2024-04-16T19:26:51.153 DEBUG:boto:auth_path=/yrjefh-11/obj36 2024-04-16T19:26:51.153 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.153 DEBUG:boto:Path: /yrjefh-11/obj36 2024-04-16T19:26:51.153 DEBUG:boto:Data: 2024-04-16T19:26:51.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.153 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.153 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.154 DEBUG:boto:Params: {} 2024-04-16T19:26:51.154 DEBUG:boto:Token: None 2024-04-16T19:26:51.154 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj36 2024-04-16T19:26:51.154 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bTUOSovdQ1fMD1uZDoex3FjrotQ= 2024-04-16T19:26:51.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTUOSovdQ1fMD1uZDoex3FjrotQ='} 2024-04-16T19:26:51.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d2f8bc52446924e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.199 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.199 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.199 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.199 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.199 DEBUG:boto:Data: 2024-04-16T19:26:51.199 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.199 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.199 DEBUG:boto:Params: {} 2024-04-16T19:26:51.199 DEBUG:boto:Token: None 2024-04-16T19:26:51.200 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007996871eaca9b0ec-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.202 DEBUG:boto:path=/yrjefh-11/obj37 2024-04-16T19:26:51.202 DEBUG:boto:auth_path=/yrjefh-11/obj37 2024-04-16T19:26:51.202 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.202 DEBUG:boto:Path: /yrjefh-11/obj37 2024-04-16T19:26:51.202 DEBUG:boto:Data: 2024-04-16T19:26:51.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.203 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.203 DEBUG:boto:Params: {} 2024-04-16T19:26:51.203 DEBUG:boto:Token: None 2024-04-16T19:26:51.203 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj37 2024-04-16T19:26:51.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HkxyCSOFf4MAeRkRigonR58p0pI= 2024-04-16T19:26:51.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HkxyCSOFf4MAeRkRigonR58p0pI='} 2024-04-16T19:26:51.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000271203cc03272857-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.208 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.208 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.208 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.208 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.208 DEBUG:boto:Data: 2024-04-16T19:26:51.209 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.209 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.209 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.209 DEBUG:boto:Params: {} 2024-04-16T19:26:51.209 DEBUG:boto:Token: None 2024-04-16T19:26:51.209 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caacbb34f9bd9cb3-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.212 DEBUG:boto:path=/yrjefh-11/obj38 2024-04-16T19:26:51.212 DEBUG:boto:auth_path=/yrjefh-11/obj38 2024-04-16T19:26:51.212 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.212 DEBUG:boto:Path: /yrjefh-11/obj38 2024-04-16T19:26:51.212 DEBUG:boto:Data: 2024-04-16T19:26:51.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.212 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.212 DEBUG:boto:Params: {} 2024-04-16T19:26:51.212 DEBUG:boto:Token: None 2024-04-16T19:26:51.213 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj38 2024-04-16T19:26:51.213 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EUezPiRm1t9NbSc76fAe5INgeeA= 2024-04-16T19:26:51.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EUezPiRm1t9NbSc76fAe5INgeeA='} 2024-04-16T19:26:51.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f3e3a6e5fe5de53-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.254 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.255 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.255 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.255 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.255 DEBUG:boto:Data: 2024-04-16T19:26:51.255 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.255 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.255 DEBUG:boto:Params: {} 2024-04-16T19:26:51.255 DEBUG:boto:Token: None 2024-04-16T19:26:51.255 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000138c012f4b13fbee-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.258 DEBUG:boto:path=/yrjefh-11/obj39 2024-04-16T19:26:51.258 DEBUG:boto:auth_path=/yrjefh-11/obj39 2024-04-16T19:26:51.258 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.259 DEBUG:boto:Path: /yrjefh-11/obj39 2024-04-16T19:26:51.259 DEBUG:boto:Data: 2024-04-16T19:26:51.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.259 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.259 DEBUG:boto:Params: {} 2024-04-16T19:26:51.259 DEBUG:boto:Token: None 2024-04-16T19:26:51.259 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj39 2024-04-16T19:26:51.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tatDDQ3f2zRwQ5w7h/xYFi407cA= 2024-04-16T19:26:51.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tatDDQ3f2zRwQ5w7h/xYFi407cA='} 2024-04-16T19:26:51.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075391506984427dc-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.264 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.264 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.264 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.264 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.264 DEBUG:boto:Data: 2024-04-16T19:26:51.264 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.264 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.264 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.265 DEBUG:boto:Params: {} 2024-04-16T19:26:51.265 DEBUG:boto:Token: None 2024-04-16T19:26:51.265 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f500b752301e93d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.268 DEBUG:boto:path=/yrjefh-11/obj40 2024-04-16T19:26:51.268 DEBUG:boto:auth_path=/yrjefh-11/obj40 2024-04-16T19:26:51.268 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.268 DEBUG:boto:Path: /yrjefh-11/obj40 2024-04-16T19:26:51.268 DEBUG:boto:Data: 2024-04-16T19:26:51.268 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.268 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.268 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.268 DEBUG:boto:Params: {} 2024-04-16T19:26:51.268 DEBUG:boto:Token: None 2024-04-16T19:26:51.269 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj40 2024-04-16T19:26:51.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:puF0B/2TjvBup59eHmMTZmUsA9I= 2024-04-16T19:26:51.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:puF0B/2TjvBup59eHmMTZmUsA9I='} 2024-04-16T19:26:51.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e744b68a834962e4-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.311 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.311 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.311 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.311 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.311 DEBUG:boto:Data: 2024-04-16T19:26:51.311 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.311 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.311 DEBUG:boto:Params: {} 2024-04-16T19:26:51.312 DEBUG:boto:Token: None 2024-04-16T19:26:51.312 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e62de0588313e152-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.314 DEBUG:boto:path=/yrjefh-11/obj41 2024-04-16T19:26:51.315 DEBUG:boto:auth_path=/yrjefh-11/obj41 2024-04-16T19:26:51.315 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.315 DEBUG:boto:Path: /yrjefh-11/obj41 2024-04-16T19:26:51.315 DEBUG:boto:Data: 2024-04-16T19:26:51.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.315 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.315 DEBUG:boto:Params: {} 2024-04-16T19:26:51.315 DEBUG:boto:Token: None 2024-04-16T19:26:51.315 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj41 2024-04-16T19:26:51.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N0wl1SgdvT3zXsZkgxXGZclsf0Y= 2024-04-16T19:26:51.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N0wl1SgdvT3zXsZkgxXGZclsf0Y='} 2024-04-16T19:26:51.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aebd55b9e59bb631-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.321 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.321 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.321 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.321 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.321 DEBUG:boto:Data: 2024-04-16T19:26:51.321 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.321 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.322 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.322 DEBUG:boto:Params: {} 2024-04-16T19:26:51.322 DEBUG:boto:Token: None 2024-04-16T19:26:51.322 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.322 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4fd194eb51f329d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.324 DEBUG:boto:path=/yrjefh-11/obj42 2024-04-16T19:26:51.324 DEBUG:boto:auth_path=/yrjefh-11/obj42 2024-04-16T19:26:51.325 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.325 DEBUG:boto:Path: /yrjefh-11/obj42 2024-04-16T19:26:51.325 DEBUG:boto:Data: 2024-04-16T19:26:51.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.325 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.325 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.325 DEBUG:boto:Params: {} 2024-04-16T19:26:51.325 DEBUG:boto:Token: None 2024-04-16T19:26:51.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj42 2024-04-16T19:26:51.325 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PFsLqXTnUD/LE2UjAA7RShd0L7g= 2024-04-16T19:26:51.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PFsLqXTnUD/LE2UjAA7RShd0L7g='} 2024-04-16T19:26:51.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed363655cf7d7c50-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.331 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.331 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.331 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.331 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.331 DEBUG:boto:Data: 2024-04-16T19:26:51.331 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.331 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.331 DEBUG:boto:Params: {} 2024-04-16T19:26:51.331 DEBUG:boto:Token: None 2024-04-16T19:26:51.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000223cbc4ef9eb824-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.334 DEBUG:boto:path=/yrjefh-11/obj43 2024-04-16T19:26:51.334 DEBUG:boto:auth_path=/yrjefh-11/obj43 2024-04-16T19:26:51.334 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.334 DEBUG:boto:Path: /yrjefh-11/obj43 2024-04-16T19:26:51.335 DEBUG:boto:Data: 2024-04-16T19:26:51.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.335 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.335 DEBUG:boto:Params: {} 2024-04-16T19:26:51.335 DEBUG:boto:Token: None 2024-04-16T19:26:51.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj43 2024-04-16T19:26:51.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:euBqNB0KQH5NX3PJnIGejZ+5awM= 2024-04-16T19:26:51.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:euBqNB0KQH5NX3PJnIGejZ+5awM='} 2024-04-16T19:26:51.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2804274db182b2b-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.379 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.379 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.379 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.379 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.379 DEBUG:boto:Data: 2024-04-16T19:26:51.379 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.379 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.379 DEBUG:boto:Params: {} 2024-04-16T19:26:51.379 DEBUG:boto:Token: None 2024-04-16T19:26:51.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9be922075e04ebb-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.382 DEBUG:boto:path=/yrjefh-11/obj44 2024-04-16T19:26:51.382 DEBUG:boto:auth_path=/yrjefh-11/obj44 2024-04-16T19:26:51.382 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.383 DEBUG:boto:Path: /yrjefh-11/obj44 2024-04-16T19:26:51.383 DEBUG:boto:Data: 2024-04-16T19:26:51.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.383 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.383 DEBUG:boto:Params: {} 2024-04-16T19:26:51.383 DEBUG:boto:Token: None 2024-04-16T19:26:51.383 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj44 2024-04-16T19:26:51.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cf4ZP//V6evi9GrBrh5RFJu1Jws= 2024-04-16T19:26:51.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cf4ZP//V6evi9GrBrh5RFJu1Jws='} 2024-04-16T19:26:51.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9928da302da153e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.390 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.390 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.390 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.390 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.390 DEBUG:boto:Data: 2024-04-16T19:26:51.391 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.391 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.391 DEBUG:boto:Params: {} 2024-04-16T19:26:51.391 DEBUG:boto:Token: None 2024-04-16T19:26:51.391 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d32d1c65a0958ef0-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.393 DEBUG:boto:path=/yrjefh-11/obj45 2024-04-16T19:26:51.393 DEBUG:boto:auth_path=/yrjefh-11/obj45 2024-04-16T19:26:51.394 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.394 DEBUG:boto:Path: /yrjefh-11/obj45 2024-04-16T19:26:51.394 DEBUG:boto:Data: 2024-04-16T19:26:51.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.394 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.394 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.394 DEBUG:boto:Params: {} 2024-04-16T19:26:51.394 DEBUG:boto:Token: None 2024-04-16T19:26:51.394 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj45 2024-04-16T19:26:51.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fkoqEq+kfCMMxJAZsIKbY1ef7AU= 2024-04-16T19:26:51.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fkoqEq+kfCMMxJAZsIKbY1ef7AU='} 2024-04-16T19:26:51.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d5b412eeebdad69-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.399 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.399 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.399 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.399 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.399 DEBUG:boto:Data: 2024-04-16T19:26:51.400 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.400 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.400 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.400 DEBUG:boto:Params: {} 2024-04-16T19:26:51.400 DEBUG:boto:Token: None 2024-04-16T19:26:51.400 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2809a99684d07c0-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.402 DEBUG:boto:path=/yrjefh-11/obj46 2024-04-16T19:26:51.402 DEBUG:boto:auth_path=/yrjefh-11/obj46 2024-04-16T19:26:51.402 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.403 DEBUG:boto:Path: /yrjefh-11/obj46 2024-04-16T19:26:51.403 DEBUG:boto:Data: 2024-04-16T19:26:51.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.403 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.403 DEBUG:boto:Params: {} 2024-04-16T19:26:51.403 DEBUG:boto:Token: None 2024-04-16T19:26:51.403 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj46 2024-04-16T19:26:51.403 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:loYLKDsZExFIYNmSoWebanfBbEk= 2024-04-16T19:26:51.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:loYLKDsZExFIYNmSoWebanfBbEk='} 2024-04-16T19:26:51.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c2d14c816acb523-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.446 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.447 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.447 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.447 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.447 DEBUG:boto:Data: 2024-04-16T19:26:51.447 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.447 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.447 DEBUG:boto:Params: {} 2024-04-16T19:26:51.447 DEBUG:boto:Token: None 2024-04-16T19:26:51.447 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007388e397720a0f0b-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.450 DEBUG:boto:path=/yrjefh-11/obj47 2024-04-16T19:26:51.450 DEBUG:boto:auth_path=/yrjefh-11/obj47 2024-04-16T19:26:51.450 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.450 DEBUG:boto:Path: /yrjefh-11/obj47 2024-04-16T19:26:51.450 DEBUG:boto:Data: 2024-04-16T19:26:51.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.450 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.450 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.450 DEBUG:boto:Params: {} 2024-04-16T19:26:51.451 DEBUG:boto:Token: None 2024-04-16T19:26:51.451 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj47 2024-04-16T19:26:51.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5XeqYK0zi9zIYX+3Q5bYnRWz+RM= 2024-04-16T19:26:51.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5XeqYK0zi9zIYX+3Q5bYnRWz+RM='} 2024-04-16T19:26:51.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c54abf7947999ed5-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.455 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.456 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.456 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.456 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.456 DEBUG:boto:Data: 2024-04-16T19:26:51.456 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.456 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.456 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.456 DEBUG:boto:Params: {} 2024-04-16T19:26:51.456 DEBUG:boto:Token: None 2024-04-16T19:26:51.456 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cce0f1b3d4b85cac-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.459 DEBUG:boto:path=/yrjefh-11/obj48 2024-04-16T19:26:51.459 DEBUG:boto:auth_path=/yrjefh-11/obj48 2024-04-16T19:26:51.459 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.459 DEBUG:boto:Path: /yrjefh-11/obj48 2024-04-16T19:26:51.459 DEBUG:boto:Data: 2024-04-16T19:26:51.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.459 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.459 DEBUG:boto:Params: {} 2024-04-16T19:26:51.459 DEBUG:boto:Token: None 2024-04-16T19:26:51.460 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj48 2024-04-16T19:26:51.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n80ChO8YNo9wELOlUoltOSklkwA= 2024-04-16T19:26:51.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n80ChO8YNo9wELOlUoltOSklkwA='} 2024-04-16T19:26:51.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000480a27f80564c62d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.503 DEBUG:boto:path=/yrjefh-11/ 2024-04-16T19:26:51.503 DEBUG:boto:auth_path=/yrjefh-11/ 2024-04-16T19:26:51.503 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.503 DEBUG:boto:Path: /yrjefh-11/ 2024-04-16T19:26:51.503 DEBUG:boto:Data: 2024-04-16T19:26:51.503 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.503 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.503 DEBUG:boto:Params: {} 2024-04-16T19:26:51.504 DEBUG:boto:Token: None 2024-04-16T19:26:51.504 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/ 2024-04-16T19:26:51.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T19:26:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T19:26:51.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dde1586c736688f1-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.506 DEBUG:boto:path=/yrjefh-11/obj49 2024-04-16T19:26:51.506 DEBUG:boto:auth_path=/yrjefh-11/obj49 2024-04-16T19:26:51.507 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.507 DEBUG:boto:Path: /yrjefh-11/obj49 2024-04-16T19:26:51.507 DEBUG:boto:Data: 2024-04-16T19:26:51.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:26:51.507 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.507 DEBUG:boto:Params: {} 2024-04-16T19:26:51.507 DEBUG:boto:Token: None 2024-04-16T19:26:51.507 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-11/obj49 2024-04-16T19:26:51.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8FlPUC540V6egdA26pIuWbpfXd0= 2024-04-16T19:26:51.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8FlPUC540V6egdA26pIuWbpfXd0='} 2024-04-16T19:26:51.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c18cc138f9931ad9-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.512 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.512 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.512 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.512 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.513 DEBUG:boto:Data: 2024-04-16T19:26:51.513 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.513 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.513 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.513 DEBUG:boto:Params: {} 2024-04-16T19:26:51.513 DEBUG:boto:Token: None 2024-04-16T19:26:51.513 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f79c09b56c167f15-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.516 DEBUG:boto:path=/yrjefh-12/obj1 2024-04-16T19:26:51.516 DEBUG:boto:auth_path=/yrjefh-12/obj1 2024-04-16T19:26:51.516 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.517 DEBUG:boto:Path: /yrjefh-12/obj1 2024-04-16T19:26:51.517 DEBUG:boto:Data: 2024-04-16T19:26:51.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.517 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.517 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.517 DEBUG:boto:Params: {} 2024-04-16T19:26:51.517 DEBUG:boto:Token: None 2024-04-16T19:26:51.517 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj1 2024-04-16T19:26:51.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lB+yq2t9zQVZnELqxhfJFFc5y6U= 2024-04-16T19:26:51.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lB+yq2t9zQVZnELqxhfJFFc5y6U='} 2024-04-16T19:26:51.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b4dfae00c177d1-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.562 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.563 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.563 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.563 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.563 DEBUG:boto:Data: 2024-04-16T19:26:51.563 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.563 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.563 DEBUG:boto:Params: {} 2024-04-16T19:26:51.563 DEBUG:boto:Token: None 2024-04-16T19:26:51.563 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008265a3ac3424a503-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.566 DEBUG:boto:path=/yrjefh-12/obj2 2024-04-16T19:26:51.566 DEBUG:boto:auth_path=/yrjefh-12/obj2 2024-04-16T19:26:51.566 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.567 DEBUG:boto:Path: /yrjefh-12/obj2 2024-04-16T19:26:51.567 DEBUG:boto:Data: 2024-04-16T19:26:51.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.567 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.567 DEBUG:boto:Params: {} 2024-04-16T19:26:51.567 DEBUG:boto:Token: None 2024-04-16T19:26:51.567 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj2 2024-04-16T19:26:51.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SUzZ0fjH57HvhI0INJpiD3ER/Q8= 2024-04-16T19:26:51.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SUzZ0fjH57HvhI0INJpiD3ER/Q8='} 2024-04-16T19:26:51.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041e93ee1e89adbe2-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.573 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.574 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.574 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.574 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.574 DEBUG:boto:Data: 2024-04-16T19:26:51.574 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.574 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.574 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.574 DEBUG:boto:Params: {} 2024-04-16T19:26:51.574 DEBUG:boto:Token: None 2024-04-16T19:26:51.574 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040acc17718f11a55-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.577 DEBUG:boto:path=/yrjefh-12/obj3 2024-04-16T19:26:51.577 DEBUG:boto:auth_path=/yrjefh-12/obj3 2024-04-16T19:26:51.577 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.577 DEBUG:boto:Path: /yrjefh-12/obj3 2024-04-16T19:26:51.577 DEBUG:boto:Data: 2024-04-16T19:26:51.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.578 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.578 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.578 DEBUG:boto:Params: {} 2024-04-16T19:26:51.578 DEBUG:boto:Token: None 2024-04-16T19:26:51.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj3 2024-04-16T19:26:51.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E/8uE9xBtKBtFBRuAYi5VKg9dVw= 2024-04-16T19:26:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/8uE9xBtKBtFBRuAYi5VKg9dVw='} 2024-04-16T19:26:51.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec6792dfd61cd960-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.585 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.585 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.585 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.585 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.585 DEBUG:boto:Data: 2024-04-16T19:26:51.585 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.585 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.585 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.585 DEBUG:boto:Params: {} 2024-04-16T19:26:51.585 DEBUG:boto:Token: None 2024-04-16T19:26:51.586 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c636a5f742cf30e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.588 DEBUG:boto:path=/yrjefh-12/obj4 2024-04-16T19:26:51.588 DEBUG:boto:auth_path=/yrjefh-12/obj4 2024-04-16T19:26:51.588 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.589 DEBUG:boto:Path: /yrjefh-12/obj4 2024-04-16T19:26:51.589 DEBUG:boto:Data: 2024-04-16T19:26:51.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.589 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.589 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.589 DEBUG:boto:Params: {} 2024-04-16T19:26:51.589 DEBUG:boto:Token: None 2024-04-16T19:26:51.589 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj4 2024-04-16T19:26:51.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DVwvg1JsenrBFtxiY+pXOP2+7RQ= 2024-04-16T19:26:51.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DVwvg1JsenrBFtxiY+pXOP2+7RQ='} 2024-04-16T19:26:51.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000426d47799f2b205a-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.634 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.635 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.635 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.635 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.635 DEBUG:boto:Data: 2024-04-16T19:26:51.635 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.635 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.635 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.635 DEBUG:boto:Params: {} 2024-04-16T19:26:51.635 DEBUG:boto:Token: None 2024-04-16T19:26:51.635 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5826dd855eca384-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.638 DEBUG:boto:path=/yrjefh-12/obj5 2024-04-16T19:26:51.638 DEBUG:boto:auth_path=/yrjefh-12/obj5 2024-04-16T19:26:51.638 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.638 DEBUG:boto:Path: /yrjefh-12/obj5 2024-04-16T19:26:51.638 DEBUG:boto:Data: 2024-04-16T19:26:51.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.638 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.638 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.639 DEBUG:boto:Params: {} 2024-04-16T19:26:51.639 DEBUG:boto:Token: None 2024-04-16T19:26:51.639 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj5 2024-04-16T19:26:51.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UeVPE5DCpR56lTJR2bfTOdAymDg= 2024-04-16T19:26:51.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UeVPE5DCpR56lTJR2bfTOdAymDg='} 2024-04-16T19:26:51.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9f055347b15eeb6-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.644 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.644 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.644 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.644 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.645 DEBUG:boto:Data: 2024-04-16T19:26:51.645 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.645 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.645 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.645 DEBUG:boto:Params: {} 2024-04-16T19:26:51.645 DEBUG:boto:Token: None 2024-04-16T19:26:51.645 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.645 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da6fa241c4501da0-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.648 DEBUG:boto:path=/yrjefh-12/obj6 2024-04-16T19:26:51.648 DEBUG:boto:auth_path=/yrjefh-12/obj6 2024-04-16T19:26:51.648 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.648 DEBUG:boto:Path: /yrjefh-12/obj6 2024-04-16T19:26:51.648 DEBUG:boto:Data: 2024-04-16T19:26:51.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.648 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.648 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.648 DEBUG:boto:Params: {} 2024-04-16T19:26:51.648 DEBUG:boto:Token: None 2024-04-16T19:26:51.649 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj6 2024-04-16T19:26:51.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jovyVKCGe6JvCwDPkKQPxLDINvo= 2024-04-16T19:26:51.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jovyVKCGe6JvCwDPkKQPxLDINvo='} 2024-04-16T19:26:51.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007357c4f1c1288282-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.691 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.691 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.691 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.691 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.691 DEBUG:boto:Data: 2024-04-16T19:26:51.691 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.691 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.691 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.691 DEBUG:boto:Params: {} 2024-04-16T19:26:51.691 DEBUG:boto:Token: None 2024-04-16T19:26:51.691 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005894ded2c90fc373-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.694 DEBUG:boto:path=/yrjefh-12/obj7 2024-04-16T19:26:51.694 DEBUG:boto:auth_path=/yrjefh-12/obj7 2024-04-16T19:26:51.694 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.694 DEBUG:boto:Path: /yrjefh-12/obj7 2024-04-16T19:26:51.694 DEBUG:boto:Data: 2024-04-16T19:26:51.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.695 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.695 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.695 DEBUG:boto:Params: {} 2024-04-16T19:26:51.695 DEBUG:boto:Token: None 2024-04-16T19:26:51.695 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj7 2024-04-16T19:26:51.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7aaBFKitnMndt1jhXFmXd4czAMs= 2024-04-16T19:26:51.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7aaBFKitnMndt1jhXFmXd4czAMs='} 2024-04-16T19:26:51.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b098a71d29ac8bf-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.700 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.700 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.700 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.700 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.700 DEBUG:boto:Data: 2024-04-16T19:26:51.700 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.700 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.701 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.701 DEBUG:boto:Params: {} 2024-04-16T19:26:51.701 DEBUG:boto:Token: None 2024-04-16T19:26:51.701 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.701 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076cb5c42e700b676-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.703 DEBUG:boto:path=/yrjefh-12/obj8 2024-04-16T19:26:51.703 DEBUG:boto:auth_path=/yrjefh-12/obj8 2024-04-16T19:26:51.703 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.704 DEBUG:boto:Path: /yrjefh-12/obj8 2024-04-16T19:26:51.704 DEBUG:boto:Data: 2024-04-16T19:26:51.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.704 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.704 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.704 DEBUG:boto:Params: {} 2024-04-16T19:26:51.704 DEBUG:boto:Token: None 2024-04-16T19:26:51.704 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj8 2024-04-16T19:26:51.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qENy4y2z3HUYcko3eDZWdDZPIds= 2024-04-16T19:26:51.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qENy4y2z3HUYcko3eDZWdDZPIds='} 2024-04-16T19:26:51.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d14b12e3ea618913-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.747 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.747 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.747 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.747 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.747 DEBUG:boto:Data: 2024-04-16T19:26:51.747 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.747 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.747 DEBUG:boto:Params: {} 2024-04-16T19:26:51.747 DEBUG:boto:Token: None 2024-04-16T19:26:51.748 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d45ff30b7d1e8e27-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.750 DEBUG:boto:path=/yrjefh-12/obj9 2024-04-16T19:26:51.750 DEBUG:boto:auth_path=/yrjefh-12/obj9 2024-04-16T19:26:51.750 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.750 DEBUG:boto:Path: /yrjefh-12/obj9 2024-04-16T19:26:51.750 DEBUG:boto:Data: 2024-04-16T19:26:51.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.751 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.751 DEBUG:boto:Params: {} 2024-04-16T19:26:51.751 DEBUG:boto:Token: None 2024-04-16T19:26:51.751 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj9 2024-04-16T19:26:51.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1FWQCY99lexn42p27izoe8IghM= 2024-04-16T19:26:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1FWQCY99lexn42p27izoe8IghM='} 2024-04-16T19:26:51.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c624fb0133256d05-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.757 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.757 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.757 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.757 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.757 DEBUG:boto:Data: 2024-04-16T19:26:51.758 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.758 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.758 DEBUG:boto:Params: {} 2024-04-16T19:26:51.758 DEBUG:boto:Token: None 2024-04-16T19:26:51.758 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.758 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000931977e10db72c77-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.761 DEBUG:boto:path=/yrjefh-12/obj10 2024-04-16T19:26:51.761 DEBUG:boto:auth_path=/yrjefh-12/obj10 2024-04-16T19:26:51.761 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.761 DEBUG:boto:Path: /yrjefh-12/obj10 2024-04-16T19:26:51.761 DEBUG:boto:Data: 2024-04-16T19:26:51.761 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.761 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.761 DEBUG:boto:Params: {} 2024-04-16T19:26:51.762 DEBUG:boto:Token: None 2024-04-16T19:26:51.762 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj10 2024-04-16T19:26:51.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sFb6zUI1p+tlhWygMxsjlly/7YY= 2024-04-16T19:26:51.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFb6zUI1p+tlhWygMxsjlly/7YY='} 2024-04-16T19:26:51.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a14b57efadc9ac5d-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.766 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.766 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.767 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.767 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.767 DEBUG:boto:Data: 2024-04-16T19:26:51.767 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.767 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.767 DEBUG:boto:Params: {} 2024-04-16T19:26:51.767 DEBUG:boto:Token: None 2024-04-16T19:26:51.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081243e58e0639f16-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.770 DEBUG:boto:path=/yrjefh-12/obj11 2024-04-16T19:26:51.770 DEBUG:boto:auth_path=/yrjefh-12/obj11 2024-04-16T19:26:51.770 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.770 DEBUG:boto:Path: /yrjefh-12/obj11 2024-04-16T19:26:51.770 DEBUG:boto:Data: 2024-04-16T19:26:51.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.770 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.770 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.770 DEBUG:boto:Params: {} 2024-04-16T19:26:51.770 DEBUG:boto:Token: None 2024-04-16T19:26:51.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj11 2024-04-16T19:26:51.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xmWms5aT00S79zylPmZxnEHRUyc= 2024-04-16T19:26:51.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xmWms5aT00S79zylPmZxnEHRUyc='} 2024-04-16T19:26:51.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d417d2a9ffb34d2e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.777 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.777 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.777 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.777 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.777 DEBUG:boto:Data: 2024-04-16T19:26:51.777 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.777 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.777 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.777 DEBUG:boto:Params: {} 2024-04-16T19:26:51.778 DEBUG:boto:Token: None 2024-04-16T19:26:51.778 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.778 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51a0781cb60ca98-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.780 DEBUG:boto:path=/yrjefh-12/obj12 2024-04-16T19:26:51.780 DEBUG:boto:auth_path=/yrjefh-12/obj12 2024-04-16T19:26:51.780 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.780 DEBUG:boto:Path: /yrjefh-12/obj12 2024-04-16T19:26:51.781 DEBUG:boto:Data: 2024-04-16T19:26:51.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.781 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.781 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.781 DEBUG:boto:Params: {} 2024-04-16T19:26:51.781 DEBUG:boto:Token: None 2024-04-16T19:26:51.781 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj12 2024-04-16T19:26:51.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YcokwFF7/PmXUgWJekSBND080tg= 2024-04-16T19:26:51.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YcokwFF7/PmXUgWJekSBND080tg='} 2024-04-16T19:26:51.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013863407d16b77fd-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.827 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.827 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.827 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.827 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.827 DEBUG:boto:Data: 2024-04-16T19:26:51.827 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.827 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.827 DEBUG:boto:Params: {} 2024-04-16T19:26:51.828 DEBUG:boto:Token: None 2024-04-16T19:26:51.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e5573a89ee35b07-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.830 DEBUG:boto:path=/yrjefh-12/obj13 2024-04-16T19:26:51.831 DEBUG:boto:auth_path=/yrjefh-12/obj13 2024-04-16T19:26:51.831 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.831 DEBUG:boto:Path: /yrjefh-12/obj13 2024-04-16T19:26:51.831 DEBUG:boto:Data: 2024-04-16T19:26:51.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.831 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.831 DEBUG:boto:Params: {} 2024-04-16T19:26:51.831 DEBUG:boto:Token: None 2024-04-16T19:26:51.831 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj13 2024-04-16T19:26:51.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B5m5P6bipZkUFb5gh700QNFLDEA= 2024-04-16T19:26:51.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B5m5P6bipZkUFb5gh700QNFLDEA='} 2024-04-16T19:26:51.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1c726728c411c43-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.836 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.836 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.836 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.836 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.836 DEBUG:boto:Data: 2024-04-16T19:26:51.837 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.837 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.837 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.837 DEBUG:boto:Params: {} 2024-04-16T19:26:51.837 DEBUG:boto:Token: None 2024-04-16T19:26:51.837 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.837 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e6b4a78521fe33e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.840 DEBUG:boto:path=/yrjefh-12/obj14 2024-04-16T19:26:51.840 DEBUG:boto:auth_path=/yrjefh-12/obj14 2024-04-16T19:26:51.840 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.840 DEBUG:boto:Path: /yrjefh-12/obj14 2024-04-16T19:26:51.840 DEBUG:boto:Data: 2024-04-16T19:26:51.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.840 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.841 DEBUG:boto:Params: {} 2024-04-16T19:26:51.841 DEBUG:boto:Token: None 2024-04-16T19:26:51.841 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj14 2024-04-16T19:26:51.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lMI7AyDbnGGY5AU2Sqdya5/zvg4= 2024-04-16T19:26:51.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lMI7AyDbnGGY5AU2Sqdya5/zvg4='} 2024-04-16T19:26:51.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c173659f785f13e0-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.883 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.883 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.883 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.883 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.883 DEBUG:boto:Data: 2024-04-16T19:26:51.883 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.883 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.883 DEBUG:boto:Params: {} 2024-04-16T19:26:51.884 DEBUG:boto:Token: None 2024-04-16T19:26:51.884 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e63db72aa2957016-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.886 DEBUG:boto:path=/yrjefh-12/obj15 2024-04-16T19:26:51.887 DEBUG:boto:auth_path=/yrjefh-12/obj15 2024-04-16T19:26:51.887 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.887 DEBUG:boto:Path: /yrjefh-12/obj15 2024-04-16T19:26:51.887 DEBUG:boto:Data: 2024-04-16T19:26:51.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.887 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.887 DEBUG:boto:Params: {} 2024-04-16T19:26:51.887 DEBUG:boto:Token: None 2024-04-16T19:26:51.887 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj15 2024-04-16T19:26:51.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s/6+1wokHKqdt6+nU+3TOmVZVp4= 2024-04-16T19:26:51.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s/6+1wokHKqdt6+nU+3TOmVZVp4='} 2024-04-16T19:26:51.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dad90c8cd42d38ba-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.894 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.894 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.894 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.894 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.894 DEBUG:boto:Data: 2024-04-16T19:26:51.894 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.894 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.895 DEBUG:boto:Params: {} 2024-04-16T19:26:51.895 DEBUG:boto:Token: None 2024-04-16T19:26:51.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfd34cd1d36720c7-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.898 DEBUG:boto:path=/yrjefh-12/obj16 2024-04-16T19:26:51.898 DEBUG:boto:auth_path=/yrjefh-12/obj16 2024-04-16T19:26:51.898 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.898 DEBUG:boto:Path: /yrjefh-12/obj16 2024-04-16T19:26:51.898 DEBUG:boto:Data: 2024-04-16T19:26:51.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.899 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.899 DEBUG:boto:Params: {} 2024-04-16T19:26:51.899 DEBUG:boto:Token: None 2024-04-16T19:26:51.899 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj16 2024-04-16T19:26:51.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mM5ipUgKWYgPYF1+awOOx4UcQRc= 2024-04-16T19:26:51.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM5ipUgKWYgPYF1+awOOx4UcQRc='} 2024-04-16T19:26:51.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e39a36a1b49d179-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.943 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.943 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.943 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.943 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.943 DEBUG:boto:Data: 2024-04-16T19:26:51.943 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.943 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.944 DEBUG:boto:Params: {} 2024-04-16T19:26:51.944 DEBUG:boto:Token: None 2024-04-16T19:26:51.944 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b616d8d453b7e53-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.947 DEBUG:boto:path=/yrjefh-12/obj17 2024-04-16T19:26:51.947 DEBUG:boto:auth_path=/yrjefh-12/obj17 2024-04-16T19:26:51.947 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.947 DEBUG:boto:Path: /yrjefh-12/obj17 2024-04-16T19:26:51.947 DEBUG:boto:Data: 2024-04-16T19:26:51.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.947 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.947 DEBUG:boto:Params: {} 2024-04-16T19:26:51.948 DEBUG:boto:Token: None 2024-04-16T19:26:51.948 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj17 2024-04-16T19:26:51.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J2UWDR4meVeX43/kNPdDUiXfFq4= 2024-04-16T19:26:51.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J2UWDR4meVeX43/kNPdDUiXfFq4='} 2024-04-16T19:26:51.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005321fc056bcb05f5-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.954 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:51.954 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:51.954 DEBUG:boto:Method: HEAD 2024-04-16T19:26:51.955 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:51.955 DEBUG:boto:Data: 2024-04-16T19:26:51.955 DEBUG:boto:Headers: {} 2024-04-16T19:26:51.955 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.955 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.955 DEBUG:boto:Params: {} 2024-04-16T19:26:51.955 DEBUG:boto:Token: None 2024-04-16T19:26:51.955 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/ 2024-04-16T19:26:51.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T19:26:51.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T19:26:51.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072f1eb38054d6455-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:51.958 DEBUG:boto:path=/yrjefh-12/obj18 2024-04-16T19:26:51.958 DEBUG:boto:auth_path=/yrjefh-12/obj18 2024-04-16T19:26:51.958 DEBUG:boto:Method: PUT 2024-04-16T19:26:51.958 DEBUG:boto:Path: /yrjefh-12/obj18 2024-04-16T19:26:51.958 DEBUG:boto:Data: 2024-04-16T19:26:51.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:51.958 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:51.958 DEBUG:boto:Port: 8000 2024-04-16T19:26:51.958 DEBUG:boto:Params: {} 2024-04-16T19:26:51.958 DEBUG:boto:Token: None 2024-04-16T19:26:51.959 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:51 GMT /yrjefh-12/obj18 2024-04-16T19:26:51.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bdBL9gaVRT6ULqrEDrLyNLkvTms= 2024-04-16T19:26:51.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bdBL9gaVRT6ULqrEDrLyNLkvTms='} 2024-04-16T19:26:52.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de4fd4801cd4a32b-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.003 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.003 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.003 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.003 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.003 DEBUG:boto:Data: 2024-04-16T19:26:52.003 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.003 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.004 DEBUG:boto:Params: {} 2024-04-16T19:26:52.004 DEBUG:boto:Token: None 2024-04-16T19:26:52.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d6349272d2476d7-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.007 DEBUG:boto:path=/yrjefh-12/obj19 2024-04-16T19:26:52.007 DEBUG:boto:auth_path=/yrjefh-12/obj19 2024-04-16T19:26:52.007 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.007 DEBUG:boto:Path: /yrjefh-12/obj19 2024-04-16T19:26:52.007 DEBUG:boto:Data: 2024-04-16T19:26:52.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.007 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.007 DEBUG:boto:Params: {} 2024-04-16T19:26:52.007 DEBUG:boto:Token: None 2024-04-16T19:26:52.007 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj19 2024-04-16T19:26:52.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IjPDJVYeU0tgHtBrsuWlLkQSqJU= 2024-04-16T19:26:52.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IjPDJVYeU0tgHtBrsuWlLkQSqJU='} 2024-04-16T19:26:52.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfd09137cc9d07d7-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.012 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.012 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.012 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.013 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.013 DEBUG:boto:Data: 2024-04-16T19:26:52.013 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.013 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.013 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.013 DEBUG:boto:Params: {} 2024-04-16T19:26:52.013 DEBUG:boto:Token: None 2024-04-16T19:26:52.013 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9b4bea1abcfd319-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.016 DEBUG:boto:path=/yrjefh-12/obj20 2024-04-16T19:26:52.016 DEBUG:boto:auth_path=/yrjefh-12/obj20 2024-04-16T19:26:52.016 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.016 DEBUG:boto:Path: /yrjefh-12/obj20 2024-04-16T19:26:52.016 DEBUG:boto:Data: 2024-04-16T19:26:52.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.016 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.016 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.016 DEBUG:boto:Params: {} 2024-04-16T19:26:52.017 DEBUG:boto:Token: None 2024-04-16T19:26:52.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj20 2024-04-16T19:26:52.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g9/VL4U9q+2m4YtXNw6CBeBt15E= 2024-04-16T19:26:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g9/VL4U9q+2m4YtXNw6CBeBt15E='} 2024-04-16T19:26:52.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000452938e616097205-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.059 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.059 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.059 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.059 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.059 DEBUG:boto:Data: 2024-04-16T19:26:52.060 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.060 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.060 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.060 DEBUG:boto:Params: {} 2024-04-16T19:26:52.060 DEBUG:boto:Token: None 2024-04-16T19:26:52.060 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016cd98fb4cb7443d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.063 DEBUG:boto:path=/yrjefh-12/obj21 2024-04-16T19:26:52.063 DEBUG:boto:auth_path=/yrjefh-12/obj21 2024-04-16T19:26:52.063 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.063 DEBUG:boto:Path: /yrjefh-12/obj21 2024-04-16T19:26:52.063 DEBUG:boto:Data: 2024-04-16T19:26:52.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.063 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.063 DEBUG:boto:Params: {} 2024-04-16T19:26:52.064 DEBUG:boto:Token: None 2024-04-16T19:26:52.064 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj21 2024-04-16T19:26:52.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jMCT0B0iFmgMCFTR29sALeVU37c= 2024-04-16T19:26:52.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMCT0B0iFmgMCFTR29sALeVU37c='} 2024-04-16T19:26:52.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028ee90c5e90bd884-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.069 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.069 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.070 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.070 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.070 DEBUG:boto:Data: 2024-04-16T19:26:52.070 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.070 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.070 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.070 DEBUG:boto:Params: {} 2024-04-16T19:26:52.070 DEBUG:boto:Token: None 2024-04-16T19:26:52.070 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bfec42b76085c08-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.073 DEBUG:boto:path=/yrjefh-12/obj22 2024-04-16T19:26:52.073 DEBUG:boto:auth_path=/yrjefh-12/obj22 2024-04-16T19:26:52.073 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.073 DEBUG:boto:Path: /yrjefh-12/obj22 2024-04-16T19:26:52.074 DEBUG:boto:Data: 2024-04-16T19:26:52.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.074 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.074 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.074 DEBUG:boto:Params: {} 2024-04-16T19:26:52.074 DEBUG:boto:Token: None 2024-04-16T19:26:52.074 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj22 2024-04-16T19:26:52.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5dMhF6/lo+vsuXbHYv9cC6Et37I= 2024-04-16T19:26:52.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5dMhF6/lo+vsuXbHYv9cC6Et37I='} 2024-04-16T19:26:52.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3424ebb9f786ff1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.119 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.119 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.119 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.119 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.119 DEBUG:boto:Data: 2024-04-16T19:26:52.119 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.120 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.120 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.120 DEBUG:boto:Params: {} 2024-04-16T19:26:52.120 DEBUG:boto:Token: None 2024-04-16T19:26:52.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5836c89b88bcbdc-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.123 DEBUG:boto:path=/yrjefh-12/obj23 2024-04-16T19:26:52.123 DEBUG:boto:auth_path=/yrjefh-12/obj23 2024-04-16T19:26:52.123 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.123 DEBUG:boto:Path: /yrjefh-12/obj23 2024-04-16T19:26:52.123 DEBUG:boto:Data: 2024-04-16T19:26:52.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.124 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.124 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.124 DEBUG:boto:Params: {} 2024-04-16T19:26:52.124 DEBUG:boto:Token: None 2024-04-16T19:26:52.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj23 2024-04-16T19:26:52.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DRJAYwpmkO18sdmw3NuQkmhEsXk= 2024-04-16T19:26:52.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRJAYwpmkO18sdmw3NuQkmhEsXk='} 2024-04-16T19:26:52.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b3f43e24030f2e-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.128 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.128 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.129 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.129 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.129 DEBUG:boto:Data: 2024-04-16T19:26:52.129 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.129 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.129 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.129 DEBUG:boto:Params: {} 2024-04-16T19:26:52.129 DEBUG:boto:Token: None 2024-04-16T19:26:52.129 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099cabeb1b4329ebf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.132 DEBUG:boto:path=/yrjefh-12/obj24 2024-04-16T19:26:52.132 DEBUG:boto:auth_path=/yrjefh-12/obj24 2024-04-16T19:26:52.133 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.133 DEBUG:boto:Path: /yrjefh-12/obj24 2024-04-16T19:26:52.133 DEBUG:boto:Data: 2024-04-16T19:26:52.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.133 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.133 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.133 DEBUG:boto:Params: {} 2024-04-16T19:26:52.133 DEBUG:boto:Token: None 2024-04-16T19:26:52.133 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj24 2024-04-16T19:26:52.133 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:An4uizaRtX4pCajGku5HNU1bvcQ= 2024-04-16T19:26:52.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:An4uizaRtX4pCajGku5HNU1bvcQ='} 2024-04-16T19:26:52.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000603f91a159e72681-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.179 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.179 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.179 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.179 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.179 DEBUG:boto:Data: 2024-04-16T19:26:52.179 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.179 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.179 DEBUG:boto:Params: {} 2024-04-16T19:26:52.179 DEBUG:boto:Token: None 2024-04-16T19:26:52.180 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3842f21c878c59-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.182 DEBUG:boto:path=/yrjefh-12/obj25 2024-04-16T19:26:52.182 DEBUG:boto:auth_path=/yrjefh-12/obj25 2024-04-16T19:26:52.183 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.183 DEBUG:boto:Path: /yrjefh-12/obj25 2024-04-16T19:26:52.183 DEBUG:boto:Data: 2024-04-16T19:26:52.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.183 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.183 DEBUG:boto:Params: {} 2024-04-16T19:26:52.183 DEBUG:boto:Token: None 2024-04-16T19:26:52.183 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj25 2024-04-16T19:26:52.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jtc6641C8fejLdrHzNc62ln5Des= 2024-04-16T19:26:52.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jtc6641C8fejLdrHzNc62ln5Des='} 2024-04-16T19:26:52.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c3c35e158fb4c7a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.188 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.188 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.188 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.188 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.188 DEBUG:boto:Data: 2024-04-16T19:26:52.188 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.188 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.188 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.189 DEBUG:boto:Params: {} 2024-04-16T19:26:52.189 DEBUG:boto:Token: None 2024-04-16T19:26:52.189 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077257a2d9e5c5c1b-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.191 DEBUG:boto:path=/yrjefh-12/obj26 2024-04-16T19:26:52.191 DEBUG:boto:auth_path=/yrjefh-12/obj26 2024-04-16T19:26:52.191 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.192 DEBUG:boto:Path: /yrjefh-12/obj26 2024-04-16T19:26:52.192 DEBUG:boto:Data: 2024-04-16T19:26:52.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.192 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.192 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.192 DEBUG:boto:Params: {} 2024-04-16T19:26:52.192 DEBUG:boto:Token: None 2024-04-16T19:26:52.192 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj26 2024-04-16T19:26:52.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5FWzFDQtKOIMNAdIs4NKv0wAckk= 2024-04-16T19:26:52.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5FWzFDQtKOIMNAdIs4NKv0wAckk='} 2024-04-16T19:26:52.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095d90e3d6309fb86-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.239 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.239 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.239 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.239 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.239 DEBUG:boto:Data: 2024-04-16T19:26:52.239 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.239 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.239 DEBUG:boto:Params: {} 2024-04-16T19:26:52.240 DEBUG:boto:Token: None 2024-04-16T19:26:52.240 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007695a81adc4a2aaf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.242 DEBUG:boto:path=/yrjefh-12/obj27 2024-04-16T19:26:52.242 DEBUG:boto:auth_path=/yrjefh-12/obj27 2024-04-16T19:26:52.243 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.243 DEBUG:boto:Path: /yrjefh-12/obj27 2024-04-16T19:26:52.243 DEBUG:boto:Data: 2024-04-16T19:26:52.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.243 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.243 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.243 DEBUG:boto:Params: {} 2024-04-16T19:26:52.243 DEBUG:boto:Token: None 2024-04-16T19:26:52.243 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj27 2024-04-16T19:26:52.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JQfb71mnKv+A5Z0SyXSoFX00sf8= 2024-04-16T19:26:52.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JQfb71mnKv+A5Z0SyXSoFX00sf8='} 2024-04-16T19:26:52.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005215f1c8f515aa6-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.248 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.248 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.248 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.248 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.248 DEBUG:boto:Data: 2024-04-16T19:26:52.248 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.248 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.248 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.249 DEBUG:boto:Params: {} 2024-04-16T19:26:52.249 DEBUG:boto:Token: None 2024-04-16T19:26:52.249 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.249 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc5605e835e72134-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.251 DEBUG:boto:path=/yrjefh-12/obj28 2024-04-16T19:26:52.251 DEBUG:boto:auth_path=/yrjefh-12/obj28 2024-04-16T19:26:52.251 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.251 DEBUG:boto:Path: /yrjefh-12/obj28 2024-04-16T19:26:52.252 DEBUG:boto:Data: 2024-04-16T19:26:52.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.252 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.252 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.252 DEBUG:boto:Params: {} 2024-04-16T19:26:52.252 DEBUG:boto:Token: None 2024-04-16T19:26:52.252 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj28 2024-04-16T19:26:52.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fzHe+mbw4gYMSPvP73L1Lr5EenE= 2024-04-16T19:26:52.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fzHe+mbw4gYMSPvP73L1Lr5EenE='} 2024-04-16T19:26:52.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016bd362be56a85b1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.295 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.295 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.295 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.295 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.295 DEBUG:boto:Data: 2024-04-16T19:26:52.295 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.295 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.295 DEBUG:boto:Params: {} 2024-04-16T19:26:52.296 DEBUG:boto:Token: None 2024-04-16T19:26:52.296 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6978081ae912898-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.299 DEBUG:boto:path=/yrjefh-12/obj29 2024-04-16T19:26:52.299 DEBUG:boto:auth_path=/yrjefh-12/obj29 2024-04-16T19:26:52.299 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.299 DEBUG:boto:Path: /yrjefh-12/obj29 2024-04-16T19:26:52.299 DEBUG:boto:Data: 2024-04-16T19:26:52.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.299 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.299 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.299 DEBUG:boto:Params: {} 2024-04-16T19:26:52.299 DEBUG:boto:Token: None 2024-04-16T19:26:52.300 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj29 2024-04-16T19:26:52.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5vPnXkN6lZf6PdRPFjSEDfM2zC4= 2024-04-16T19:26:52.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5vPnXkN6lZf6PdRPFjSEDfM2zC4='} 2024-04-16T19:26:52.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b6d20662521ea0a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.304 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.304 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.304 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.304 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.305 DEBUG:boto:Data: 2024-04-16T19:26:52.305 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.305 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.305 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.305 DEBUG:boto:Params: {} 2024-04-16T19:26:52.305 DEBUG:boto:Token: None 2024-04-16T19:26:52.305 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaa5358d5e2ca639-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.308 DEBUG:boto:path=/yrjefh-12/obj30 2024-04-16T19:26:52.308 DEBUG:boto:auth_path=/yrjefh-12/obj30 2024-04-16T19:26:52.308 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.308 DEBUG:boto:Path: /yrjefh-12/obj30 2024-04-16T19:26:52.308 DEBUG:boto:Data: 2024-04-16T19:26:52.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.308 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.308 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.308 DEBUG:boto:Params: {} 2024-04-16T19:26:52.309 DEBUG:boto:Token: None 2024-04-16T19:26:52.309 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj30 2024-04-16T19:26:52.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Feo1CcHsteWU4/JaPdKTHbxfhXQ= 2024-04-16T19:26:52.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Feo1CcHsteWU4/JaPdKTHbxfhXQ='} 2024-04-16T19:26:52.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008713638177885e07-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.351 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.351 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.351 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.351 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.351 DEBUG:boto:Data: 2024-04-16T19:26:52.351 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.351 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.351 DEBUG:boto:Params: {} 2024-04-16T19:26:52.351 DEBUG:boto:Token: None 2024-04-16T19:26:52.352 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ac4691d1e918cab-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.354 DEBUG:boto:path=/yrjefh-12/obj31 2024-04-16T19:26:52.354 DEBUG:boto:auth_path=/yrjefh-12/obj31 2024-04-16T19:26:52.355 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.355 DEBUG:boto:Path: /yrjefh-12/obj31 2024-04-16T19:26:52.355 DEBUG:boto:Data: 2024-04-16T19:26:52.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.355 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.355 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.355 DEBUG:boto:Params: {} 2024-04-16T19:26:52.355 DEBUG:boto:Token: None 2024-04-16T19:26:52.355 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj31 2024-04-16T19:26:52.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EdGWjQXq9U8sh5AyLAXC1Uz+0no= 2024-04-16T19:26:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EdGWjQXq9U8sh5AyLAXC1Uz+0no='} 2024-04-16T19:26:52.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003818d81cedc712d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.360 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.360 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.360 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.360 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.360 DEBUG:boto:Data: 2024-04-16T19:26:52.361 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.361 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.361 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.361 DEBUG:boto:Params: {} 2024-04-16T19:26:52.361 DEBUG:boto:Token: None 2024-04-16T19:26:52.361 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.361 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad4bc6a9de932a16-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.363 DEBUG:boto:path=/yrjefh-12/obj32 2024-04-16T19:26:52.363 DEBUG:boto:auth_path=/yrjefh-12/obj32 2024-04-16T19:26:52.364 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.364 DEBUG:boto:Path: /yrjefh-12/obj32 2024-04-16T19:26:52.364 DEBUG:boto:Data: 2024-04-16T19:26:52.364 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.364 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.364 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.364 DEBUG:boto:Params: {} 2024-04-16T19:26:52.364 DEBUG:boto:Token: None 2024-04-16T19:26:52.364 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj32 2024-04-16T19:26:52.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SIzMT6xK0jddiqA0ko7tAE4W8EI= 2024-04-16T19:26:52.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SIzMT6xK0jddiqA0ko7tAE4W8EI='} 2024-04-16T19:26:52.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001409a20bb8b551ac-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.407 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.407 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.407 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.407 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.407 DEBUG:boto:Data: 2024-04-16T19:26:52.407 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.407 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.407 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.408 DEBUG:boto:Params: {} 2024-04-16T19:26:52.408 DEBUG:boto:Token: None 2024-04-16T19:26:52.408 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb61bc932dc3d120-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.410 DEBUG:boto:path=/yrjefh-12/obj33 2024-04-16T19:26:52.410 DEBUG:boto:auth_path=/yrjefh-12/obj33 2024-04-16T19:26:52.411 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.411 DEBUG:boto:Path: /yrjefh-12/obj33 2024-04-16T19:26:52.411 DEBUG:boto:Data: 2024-04-16T19:26:52.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.411 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.411 DEBUG:boto:Params: {} 2024-04-16T19:26:52.411 DEBUG:boto:Token: None 2024-04-16T19:26:52.411 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj33 2024-04-16T19:26:52.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WqLR/CJu80dd6qN78/xlaYTEA5k= 2024-04-16T19:26:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WqLR/CJu80dd6qN78/xlaYTEA5k='} 2024-04-16T19:26:52.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003aab63a42988ba87-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.416 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.417 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.417 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.417 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.417 DEBUG:boto:Data: 2024-04-16T19:26:52.417 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.417 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.417 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.417 DEBUG:boto:Params: {} 2024-04-16T19:26:52.417 DEBUG:boto:Token: None 2024-04-16T19:26:52.417 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e847e4c9f1badf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.420 DEBUG:boto:path=/yrjefh-12/obj34 2024-04-16T19:26:52.420 DEBUG:boto:auth_path=/yrjefh-12/obj34 2024-04-16T19:26:52.420 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.420 DEBUG:boto:Path: /yrjefh-12/obj34 2024-04-16T19:26:52.420 DEBUG:boto:Data: 2024-04-16T19:26:52.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.420 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.420 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.420 DEBUG:boto:Params: {} 2024-04-16T19:26:52.421 DEBUG:boto:Token: None 2024-04-16T19:26:52.421 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj34 2024-04-16T19:26:52.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gPqlc4xa3BlKfnJ00O9ByV2sYxA= 2024-04-16T19:26:52.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gPqlc4xa3BlKfnJ00O9ByV2sYxA='} 2024-04-16T19:26:52.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dce1ee357641af4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.467 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.467 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.467 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.467 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.467 DEBUG:boto:Data: 2024-04-16T19:26:52.467 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.467 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.467 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.467 DEBUG:boto:Params: {} 2024-04-16T19:26:52.468 DEBUG:boto:Token: None 2024-04-16T19:26:52.468 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003444405d08aaeaca-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.471 DEBUG:boto:path=/yrjefh-12/obj35 2024-04-16T19:26:52.471 DEBUG:boto:auth_path=/yrjefh-12/obj35 2024-04-16T19:26:52.471 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.471 DEBUG:boto:Path: /yrjefh-12/obj35 2024-04-16T19:26:52.471 DEBUG:boto:Data: 2024-04-16T19:26:52.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.471 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.471 DEBUG:boto:Params: {} 2024-04-16T19:26:52.472 DEBUG:boto:Token: None 2024-04-16T19:26:52.472 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj35 2024-04-16T19:26:52.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beFTrp4oKRZbGex5TbiPvVl1ax4= 2024-04-16T19:26:52.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beFTrp4oKRZbGex5TbiPvVl1ax4='} 2024-04-16T19:26:52.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c34d2f3f73807fac-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.477 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.477 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.477 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.477 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.477 DEBUG:boto:Data: 2024-04-16T19:26:52.477 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.477 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.477 DEBUG:boto:Params: {} 2024-04-16T19:26:52.477 DEBUG:boto:Token: None 2024-04-16T19:26:52.478 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.478 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efab9757960f4dc4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.480 DEBUG:boto:path=/yrjefh-12/obj36 2024-04-16T19:26:52.480 DEBUG:boto:auth_path=/yrjefh-12/obj36 2024-04-16T19:26:52.480 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.480 DEBUG:boto:Path: /yrjefh-12/obj36 2024-04-16T19:26:52.480 DEBUG:boto:Data: 2024-04-16T19:26:52.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.481 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.481 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.481 DEBUG:boto:Params: {} 2024-04-16T19:26:52.481 DEBUG:boto:Token: None 2024-04-16T19:26:52.481 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj36 2024-04-16T19:26:52.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RkLk/GKnoEKM9An9hIMwq1wRrF4= 2024-04-16T19:26:52.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkLk/GKnoEKM9An9hIMwq1wRrF4='} 2024-04-16T19:26:52.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1821f16e58884ea-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.527 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.527 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.527 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.527 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.527 DEBUG:boto:Data: 2024-04-16T19:26:52.527 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.527 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.527 DEBUG:boto:Params: {} 2024-04-16T19:26:52.527 DEBUG:boto:Token: None 2024-04-16T19:26:52.528 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adbb1333f2e9e86c-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.530 DEBUG:boto:path=/yrjefh-12/obj37 2024-04-16T19:26:52.530 DEBUG:boto:auth_path=/yrjefh-12/obj37 2024-04-16T19:26:52.530 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.531 DEBUG:boto:Path: /yrjefh-12/obj37 2024-04-16T19:26:52.531 DEBUG:boto:Data: 2024-04-16T19:26:52.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.531 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.531 DEBUG:boto:Params: {} 2024-04-16T19:26:52.531 DEBUG:boto:Token: None 2024-04-16T19:26:52.531 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj37 2024-04-16T19:26:52.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:43ZBrRwTvT6SG6EdWhMfFxICVRc= 2024-04-16T19:26:52.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:43ZBrRwTvT6SG6EdWhMfFxICVRc='} 2024-04-16T19:26:52.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012ef11fccf4fff34-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.536 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.536 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.536 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.536 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.536 DEBUG:boto:Data: 2024-04-16T19:26:52.536 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.536 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.536 DEBUG:boto:Params: {} 2024-04-16T19:26:52.537 DEBUG:boto:Token: None 2024-04-16T19:26:52.537 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fcd4a743e7392a2-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.539 DEBUG:boto:path=/yrjefh-12/obj38 2024-04-16T19:26:52.539 DEBUG:boto:auth_path=/yrjefh-12/obj38 2024-04-16T19:26:52.540 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.540 DEBUG:boto:Path: /yrjefh-12/obj38 2024-04-16T19:26:52.540 DEBUG:boto:Data: 2024-04-16T19:26:52.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.540 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.540 DEBUG:boto:Params: {} 2024-04-16T19:26:52.540 DEBUG:boto:Token: None 2024-04-16T19:26:52.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj38 2024-04-16T19:26:52.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JTMVngw6lMAXGH4eUxax3WPgxUQ= 2024-04-16T19:26:52.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JTMVngw6lMAXGH4eUxax3WPgxUQ='} 2024-04-16T19:26:52.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000840e028ed75dfbaf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.583 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.583 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.583 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.583 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.583 DEBUG:boto:Data: 2024-04-16T19:26:52.583 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.583 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.583 DEBUG:boto:Params: {} 2024-04-16T19:26:52.584 DEBUG:boto:Token: None 2024-04-16T19:26:52.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000449c45e94cf3bd52-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.587 DEBUG:boto:path=/yrjefh-12/obj39 2024-04-16T19:26:52.587 DEBUG:boto:auth_path=/yrjefh-12/obj39 2024-04-16T19:26:52.587 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.587 DEBUG:boto:Path: /yrjefh-12/obj39 2024-04-16T19:26:52.587 DEBUG:boto:Data: 2024-04-16T19:26:52.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.588 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.588 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.588 DEBUG:boto:Params: {} 2024-04-16T19:26:52.588 DEBUG:boto:Token: None 2024-04-16T19:26:52.588 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj39 2024-04-16T19:26:52.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qGH+mzCMA5Bgbp9ddQuYn+FZag= 2024-04-16T19:26:52.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qGH+mzCMA5Bgbp9ddQuYn+FZag='} 2024-04-16T19:26:52.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d8330378999aca5-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.593 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.593 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.593 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.593 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.593 DEBUG:boto:Data: 2024-04-16T19:26:52.593 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.594 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.594 DEBUG:boto:Params: {} 2024-04-16T19:26:52.594 DEBUG:boto:Token: None 2024-04-16T19:26:52.594 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dfce296130ef09a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.597 DEBUG:boto:path=/yrjefh-12/obj40 2024-04-16T19:26:52.597 DEBUG:boto:auth_path=/yrjefh-12/obj40 2024-04-16T19:26:52.597 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.597 DEBUG:boto:Path: /yrjefh-12/obj40 2024-04-16T19:26:52.597 DEBUG:boto:Data: 2024-04-16T19:26:52.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.597 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.597 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.597 DEBUG:boto:Params: {} 2024-04-16T19:26:52.598 DEBUG:boto:Token: None 2024-04-16T19:26:52.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj40 2024-04-16T19:26:52.598 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8JE8ZuD5QsoFScnskmTOZ0Nfgs0= 2024-04-16T19:26:52.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8JE8ZuD5QsoFScnskmTOZ0Nfgs0='} 2024-04-16T19:26:52.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad75e194f685332f-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.643 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.643 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.643 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.643 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.643 DEBUG:boto:Data: 2024-04-16T19:26:52.643 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.643 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.644 DEBUG:boto:Params: {} 2024-04-16T19:26:52.644 DEBUG:boto:Token: None 2024-04-16T19:26:52.644 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a0608047bf6684-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.647 DEBUG:boto:path=/yrjefh-12/obj41 2024-04-16T19:26:52.647 DEBUG:boto:auth_path=/yrjefh-12/obj41 2024-04-16T19:26:52.647 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.647 DEBUG:boto:Path: /yrjefh-12/obj41 2024-04-16T19:26:52.647 DEBUG:boto:Data: 2024-04-16T19:26:52.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.647 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.647 DEBUG:boto:Params: {} 2024-04-16T19:26:52.648 DEBUG:boto:Token: None 2024-04-16T19:26:52.648 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj41 2024-04-16T19:26:52.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cmBkZaOzdVotsKbaxS6ndPRXbuo= 2024-04-16T19:26:52.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cmBkZaOzdVotsKbaxS6ndPRXbuo='} 2024-04-16T19:26:52.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b1b0a4c0c162b02-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.653 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.653 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.653 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.653 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.653 DEBUG:boto:Data: 2024-04-16T19:26:52.653 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.653 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.653 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.653 DEBUG:boto:Params: {} 2024-04-16T19:26:52.654 DEBUG:boto:Token: None 2024-04-16T19:26:52.654 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.654 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047bba4c9c1920eeb-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.656 DEBUG:boto:path=/yrjefh-12/obj42 2024-04-16T19:26:52.656 DEBUG:boto:auth_path=/yrjefh-12/obj42 2024-04-16T19:26:52.656 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.656 DEBUG:boto:Path: /yrjefh-12/obj42 2024-04-16T19:26:52.657 DEBUG:boto:Data: 2024-04-16T19:26:52.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.657 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.657 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.657 DEBUG:boto:Params: {} 2024-04-16T19:26:52.657 DEBUG:boto:Token: None 2024-04-16T19:26:52.657 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj42 2024-04-16T19:26:52.657 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bUvo9sE7KkNsI++IpRBvx5lOIxs= 2024-04-16T19:26:52.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bUvo9sE7KkNsI++IpRBvx5lOIxs='} 2024-04-16T19:26:52.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c766a4b5a9f7d9fc-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.703 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.703 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.703 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.703 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.703 DEBUG:boto:Data: 2024-04-16T19:26:52.703 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.704 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.704 DEBUG:boto:Params: {} 2024-04-16T19:26:52.704 DEBUG:boto:Token: None 2024-04-16T19:26:52.704 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b160af09bf9e0b1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.707 DEBUG:boto:path=/yrjefh-12/obj43 2024-04-16T19:26:52.707 DEBUG:boto:auth_path=/yrjefh-12/obj43 2024-04-16T19:26:52.707 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.707 DEBUG:boto:Path: /yrjefh-12/obj43 2024-04-16T19:26:52.707 DEBUG:boto:Data: 2024-04-16T19:26:52.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.707 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.707 DEBUG:boto:Params: {} 2024-04-16T19:26:52.708 DEBUG:boto:Token: None 2024-04-16T19:26:52.708 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj43 2024-04-16T19:26:52.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5TmOjQAsYw7c6nElH+Rcy0G8mY= 2024-04-16T19:26:52.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5TmOjQAsYw7c6nElH+Rcy0G8mY='} 2024-04-16T19:26:52.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f611e44dc903b716-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.713 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.713 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.713 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.713 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.713 DEBUG:boto:Data: 2024-04-16T19:26:52.713 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.713 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.713 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.713 DEBUG:boto:Params: {} 2024-04-16T19:26:52.714 DEBUG:boto:Token: None 2024-04-16T19:26:52.714 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000423b95d1a2e4cd97-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.716 DEBUG:boto:path=/yrjefh-12/obj44 2024-04-16T19:26:52.716 DEBUG:boto:auth_path=/yrjefh-12/obj44 2024-04-16T19:26:52.716 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.717 DEBUG:boto:Path: /yrjefh-12/obj44 2024-04-16T19:26:52.717 DEBUG:boto:Data: 2024-04-16T19:26:52.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.717 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.717 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.717 DEBUG:boto:Params: {} 2024-04-16T19:26:52.717 DEBUG:boto:Token: None 2024-04-16T19:26:52.717 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj44 2024-04-16T19:26:52.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XtmSlbkW3IiN+lblprroRIwZ/bM= 2024-04-16T19:26:52.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtmSlbkW3IiN+lblprroRIwZ/bM='} 2024-04-16T19:26:52.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db8783374d54c636-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.763 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.763 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.763 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.763 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.763 DEBUG:boto:Data: 2024-04-16T19:26:52.763 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.763 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.763 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.763 DEBUG:boto:Params: {} 2024-04-16T19:26:52.763 DEBUG:boto:Token: None 2024-04-16T19:26:52.764 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063b2b65ac4fab42a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.766 DEBUG:boto:path=/yrjefh-12/obj45 2024-04-16T19:26:52.766 DEBUG:boto:auth_path=/yrjefh-12/obj45 2024-04-16T19:26:52.766 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.767 DEBUG:boto:Path: /yrjefh-12/obj45 2024-04-16T19:26:52.767 DEBUG:boto:Data: 2024-04-16T19:26:52.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.767 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.767 DEBUG:boto:Params: {} 2024-04-16T19:26:52.767 DEBUG:boto:Token: None 2024-04-16T19:26:52.767 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj45 2024-04-16T19:26:52.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uFM1QVVUPkENMgwrYamRxWK/DIQ= 2024-04-16T19:26:52.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uFM1QVVUPkENMgwrYamRxWK/DIQ='} 2024-04-16T19:26:52.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000526bbf3175fcf3f2-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.772 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.772 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.772 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.772 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.772 DEBUG:boto:Data: 2024-04-16T19:26:52.772 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.772 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.773 DEBUG:boto:Params: {} 2024-04-16T19:26:52.773 DEBUG:boto:Token: None 2024-04-16T19:26:52.773 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.773 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005058262dd505ec4e-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.775 DEBUG:boto:path=/yrjefh-12/obj46 2024-04-16T19:26:52.775 DEBUG:boto:auth_path=/yrjefh-12/obj46 2024-04-16T19:26:52.776 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.776 DEBUG:boto:Path: /yrjefh-12/obj46 2024-04-16T19:26:52.776 DEBUG:boto:Data: 2024-04-16T19:26:52.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.776 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.776 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.776 DEBUG:boto:Params: {} 2024-04-16T19:26:52.776 DEBUG:boto:Token: None 2024-04-16T19:26:52.776 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj46 2024-04-16T19:26:52.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w3+sdGwv6oHf8tZ1uWd5BIL5kN8= 2024-04-16T19:26:52.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w3+sdGwv6oHf8tZ1uWd5BIL5kN8='} 2024-04-16T19:26:52.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f202fb61fb839ab4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.819 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.819 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.819 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.819 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.819 DEBUG:boto:Data: 2024-04-16T19:26:52.819 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.820 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.820 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.820 DEBUG:boto:Params: {} 2024-04-16T19:26:52.820 DEBUG:boto:Token: None 2024-04-16T19:26:52.820 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790705ef94130a1d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.823 DEBUG:boto:path=/yrjefh-12/obj47 2024-04-16T19:26:52.823 DEBUG:boto:auth_path=/yrjefh-12/obj47 2024-04-16T19:26:52.823 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.823 DEBUG:boto:Path: /yrjefh-12/obj47 2024-04-16T19:26:52.823 DEBUG:boto:Data: 2024-04-16T19:26:52.823 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.823 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.823 DEBUG:boto:Params: {} 2024-04-16T19:26:52.824 DEBUG:boto:Token: None 2024-04-16T19:26:52.824 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj47 2024-04-16T19:26:52.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1d6unHf/rew/15rU/kT0zplb5W4= 2024-04-16T19:26:52.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1d6unHf/rew/15rU/kT0zplb5W4='} 2024-04-16T19:26:52.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000072dcba10270888d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.828 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.828 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.828 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.828 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.828 DEBUG:boto:Data: 2024-04-16T19:26:52.828 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.829 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.829 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.829 DEBUG:boto:Params: {} 2024-04-16T19:26:52.829 DEBUG:boto:Token: None 2024-04-16T19:26:52.829 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.829 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005092bd3be141c096-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.831 DEBUG:boto:path=/yrjefh-12/obj48 2024-04-16T19:26:52.831 DEBUG:boto:auth_path=/yrjefh-12/obj48 2024-04-16T19:26:52.832 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.832 DEBUG:boto:Path: /yrjefh-12/obj48 2024-04-16T19:26:52.832 DEBUG:boto:Data: 2024-04-16T19:26:52.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.832 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.832 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.832 DEBUG:boto:Params: {} 2024-04-16T19:26:52.832 DEBUG:boto:Token: None 2024-04-16T19:26:52.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj48 2024-04-16T19:26:52.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wglv1teydWA1vQhO4XVCRq5+6Fc= 2024-04-16T19:26:52.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wglv1teydWA1vQhO4XVCRq5+6Fc='} 2024-04-16T19:26:52.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000209d47a0e91e7aa4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.875 DEBUG:boto:path=/yrjefh-12/ 2024-04-16T19:26:52.875 DEBUG:boto:auth_path=/yrjefh-12/ 2024-04-16T19:26:52.875 DEBUG:boto:Method: HEAD 2024-04-16T19:26:52.875 DEBUG:boto:Path: /yrjefh-12/ 2024-04-16T19:26:52.875 DEBUG:boto:Data: 2024-04-16T19:26:52.875 DEBUG:boto:Headers: {} 2024-04-16T19:26:52.875 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.876 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.876 DEBUG:boto:Params: {} 2024-04-16T19:26:52.876 DEBUG:boto:Token: None 2024-04-16T19:26:52.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/ 2024-04-16T19:26:52.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T19:26:52.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T19:26:52.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000769ff5e825148efb-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.879 DEBUG:boto:path=/yrjefh-12/obj49 2024-04-16T19:26:52.879 DEBUG:boto:auth_path=/yrjefh-12/obj49 2024-04-16T19:26:52.879 DEBUG:boto:Method: PUT 2024-04-16T19:26:52.879 DEBUG:boto:Path: /yrjefh-12/obj49 2024-04-16T19:26:52.879 DEBUG:boto:Data: 2024-04-16T19:26:52.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:26:52.880 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:26:52.880 DEBUG:boto:Port: 8000 2024-04-16T19:26:52.880 DEBUG:boto:Params: {} 2024-04-16T19:26:52.880 DEBUG:boto:Token: None 2024-04-16T19:26:52.880 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:26:52 GMT /yrjefh-12/obj49 2024-04-16T19:26:52.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dmUHahZl++QVE+NwQ1Gk+RMKHIA= 2024-04-16T19:26:52.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dmUHahZl++QVE+NwQ1Gk+RMKHIA='} 2024-04-16T19:26:52.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000596e61dd55fe04db-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:26:52.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:52.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:52.885 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:52.940 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:52.996 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:26:53.018 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713295610.154792_56.1","last_update":"2024-04-16T19:26:50.154792Z"}] 2024-04-16T19:26:53.018 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T19:26:53.018 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:26:53.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:53.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:26:53.019 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:26:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:53.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:53.119+0000 7efeb2a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:53.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:53.121+0000 7efeb2a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:26:53.141 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295610.154792_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:50.154792Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:53Z"} 2024-04-16T19:26:53.142 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295610.154792_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:50.154792Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:53Z"} 2024-04-16T19:26:53.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295610.154792_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:50.154792Z', 'realm_epoch': 2}}] 2024-04-16T19:26:53.142 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T19:26:53.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T19:26:53.143 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:26:53.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-11', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:53.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-11', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:26:53.143 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-11 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:26:53.198 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:26:53.198 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:26:53.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:53.255+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:26:58.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:26:58.256+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:03.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:03.258+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:08.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:08.258+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:13.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:13.260+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:18.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:18.260+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:23.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:23.261+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:28.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:28.262+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:33.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:33.263+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:38.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:38.264+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:43.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:43.265+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:48.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:48.267+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:53.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:53.267+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:27:58.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:27:58.268+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:03.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:03.269+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:08.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:08.270+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:13.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:13.271+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:18.272+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:23.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:23.273+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:28.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:28.274+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:33.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:33.275+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:38.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:38.275+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:43.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:43.276+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:48.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:48.277+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:53.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:53.278+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:28:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:28:58.279+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:03.280+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:08.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:13.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:13.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:18.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:18.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:23.284+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:28.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:28.285+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:33.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:33.286+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:38.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:38.287+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:43.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:43.288+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:48.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:48.289+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:53.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:53.290+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:29:58.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:29:58.290+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:03.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:03.292+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:08.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:08.293+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:13.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:13.294+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:18.295+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:23.296+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:28.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:28.296+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:33.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:33.297+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:38.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:38.298+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:43.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:43.299+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:48.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:48.300+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:53.301+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:30:58.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:30:58.302+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:03.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:03.303+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:08.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:08.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:13.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:13.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:18.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:18.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:23.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:23.307+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:28.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:28.308+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:33.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:33.309+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:38.310+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:43.311+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T19:31:48.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:48.312+0000 7f9d44424800 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-16T19:31:48.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:48.312+0000 7f9d44424800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T19:31:48.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:48.312+0000 7f9d44424800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T19:31:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-16T19:31:48.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-16T19:31:48.375 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-13 2024-04-16T19:31:48.375 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:31:48.375 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:31:48.376 DEBUG:boto:Method: PUT 2024-04-16T19:31:48.376 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:31:48.376 DEBUG:boto:Data: 2024-04-16T19:31:48.376 DEBUG:boto:Headers: {} 2024-04-16T19:31:48.376 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:31:48.376 DEBUG:boto:Port: 8000 2024-04-16T19:31:48.376 DEBUG:boto:Params: {} 2024-04-16T19:31:48.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:31:48.377 DEBUG:boto:Token: None 2024-04-16T19:31:48.378 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:31:48 GMT /yrjefh-13/ 2024-04-16T19:31:48.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FkfAAu2YT/woDCnFLq/nABUkfNU= 2024-04-16T19:31:48.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:48 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FkfAAu2YT/woDCnFLq/nABUkfNU=', 'Content-Length': '0'} 2024-04-16T19:31:48.399 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ec2609d1ba2504b-00661ed224-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:48.399 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-14 2024-04-16T19:31:48.399 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:31:48.399 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:31:48.400 DEBUG:boto:Method: PUT 2024-04-16T19:31:48.400 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:31:48.400 DEBUG:boto:Data: 2024-04-16T19:31:48.400 DEBUG:boto:Headers: {} 2024-04-16T19:31:48.400 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:31:48.400 DEBUG:boto:Port: 8000 2024-04-16T19:31:48.400 DEBUG:boto:Params: {} 2024-04-16T19:31:48.401 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:31:48.401 DEBUG:boto:Token: None 2024-04-16T19:31:48.401 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:31:48 GMT /yrjefh-14/ 2024-04-16T19:31:48.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jHGygDsAO8KfCTixWgcbTnKCAbU= 2024-04-16T19:31:48.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:48 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jHGygDsAO8KfCTixWgcbTnKCAbU=', 'Content-Length': '0'} 2024-04-16T19:31:48.431 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000faeb87e70ea45149-00661ed224-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:48.431 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:31:48.431 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:31:48.431 DEBUG:boto:path=/yrjefh-13/?versioning 2024-04-16T19:31:48.431 DEBUG:boto:auth_path=/yrjefh-13/?versioning 2024-04-16T19:31:48.432 DEBUG:boto:Method: PUT 2024-04-16T19:31:48.432 DEBUG:boto:Path: /yrjefh-13/?versioning 2024-04-16T19:31:48.432 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:31:48.432 DEBUG:boto:Headers: {} 2024-04-16T19:31:48.432 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:31:48.432 DEBUG:boto:Port: 8000 2024-04-16T19:31:48.432 DEBUG:boto:Params: {} 2024-04-16T19:31:48.432 DEBUG:boto:Token: None 2024-04-16T19:31:48.433 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:31:48 GMT /yrjefh-13/?versioning 2024-04-16T19:31:48.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7AQiQZkwB6sP0zE6D/Kpek0JvBw= 2024-04-16T19:31:48.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:48 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7AQiQZkwB6sP0zE6D/Kpek0JvBw=', 'Content-Length': '228'} 2024-04-16T19:31:48.440 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bdcdd706b28b8ce7-00661ed224-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:48.440 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:31:48.440 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:31:48.440 DEBUG:boto:path=/yrjefh-14/?versioning 2024-04-16T19:31:48.440 DEBUG:boto:auth_path=/yrjefh-14/?versioning 2024-04-16T19:31:48.440 DEBUG:boto:Method: PUT 2024-04-16T19:31:48.441 DEBUG:boto:Path: /yrjefh-14/?versioning 2024-04-16T19:31:48.441 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:31:48.441 DEBUG:boto:Headers: {} 2024-04-16T19:31:48.441 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:31:48.441 DEBUG:boto:Port: 8000 2024-04-16T19:31:48.441 DEBUG:boto:Params: {} 2024-04-16T19:31:48.441 DEBUG:boto:Token: None 2024-04-16T19:31:48.441 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:31:48 GMT /yrjefh-14/?versioning 2024-04-16T19:31:48.442 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n6uNjrp2WXnX71n0BZVv32LBXC4= 2024-04-16T19:31:48.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:48 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n6uNjrp2WXnX71n0BZVv32LBXC4=', 'Content-Length': '228'} 2024-04-16T19:31:48.452 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f263c1ce1d063a4f-00661ed224-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:48.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:48.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:48.453 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:31:48.507 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:31:48.507 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:31:48.562 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:31:48.583 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713295908.447955_120.1","last_update":"2024-04-16T19:31:48.447955Z"}] 2024-04-16T19:31:48.584 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295908.447955_120.1'} 2024-04-16T19:31:48.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:31:48.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:48.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:48.585 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:31:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:31:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:31:48.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:48.690+0000 7f22f6216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:31:48.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:48.691+0000 7f22f6216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:31:48.714 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295908.444323_119.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:31:48.444323Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:31:48Z"} 2024-04-16T19:31:48.715 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295908.444323_119.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:31:48.444323Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:31:48Z"} 2024-04-16T19:31:48.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295908.444323_119.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:31:48.444323Z', 'realm_epoch': 2}}] 2024-04-16T19:31:48.715 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295908.447955_120.1'} 2024-04-16T19:31:48.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295908.444323_119.1'} 2024-04-16T19:31:48.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713295908.447955_120.1 target=1_1713295908.444323_119.1 2024-04-16T19:31:53.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:53.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:31:53.718 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:31:53.774 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:31:53.774 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:31:53.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:53.825+0000 7f3e2b52b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:31:53.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:53.826+0000 7f3e2b52b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:31:53.849 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295908.447955_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:31:48.447955Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:31:53Z"} 2024-04-16T19:31:53.849 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295908.447955_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:31:48.447955Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:31:53Z"} 2024-04-16T19:31:53.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295908.447955_120.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:31:48.447955Z', 'realm_epoch': 2}}] 2024-04-16T19:31:53.849 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295908.447955_120.1'} 2024-04-16T19:31:53.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295908.447955_120.1'} 2024-04-16T19:31:53.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:31:53.850 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:31:53.850 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:31:53.850 DEBUG:boto:Method: HEAD 2024-04-16T19:31:53.850 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:31:53.850 DEBUG:boto:Data: 2024-04-16T19:31:53.851 DEBUG:boto:Headers: {} 2024-04-16T19:31:53.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:31:53.851 DEBUG:boto:Port: 8000 2024-04-16T19:31:53.851 DEBUG:boto:Params: {} 2024-04-16T19:31:53.851 DEBUG:boto:Token: None 2024-04-16T19:31:53.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:31:53 GMT /yrjefh-13/ 2024-04-16T19:31:53.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xgKUAnlALmi3Ue6wEgO6JEwZriQ= 2024-04-16T19:31:53.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xgKUAnlALmi3Ue6wEgO6JEwZriQ=', 'Content-Length': '0'} 2024-04-16T19:31:53.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3b2f85e86d7f2ec-00661ed229-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:53.855 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:31:53.855 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:31:53.855 DEBUG:boto:Method: PUT 2024-04-16T19:31:53.855 DEBUG:boto:Path: /yrjefh-13/dummy 2024-04-16T19:31:53.855 DEBUG:boto:Data: 2024-04-16T19:31:53.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:31:53.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:31:53.855 DEBUG:boto:Port: 8000 2024-04-16T19:31:53.856 DEBUG:boto:Params: {} 2024-04-16T19:31:53.856 DEBUG:boto:Token: None 2024-04-16T19:31:53.856 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:31:53 GMT /yrjefh-13/dummy 2024-04-16T19:31:53.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NXDOBUI6jg9OrZ4TBkLziUE91kw= 2024-04-16T19:31:53.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:31:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NXDOBUI6jg9OrZ4TBkLziUE91kw='} 2024-04-16T19:31:53.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-amz-request-id', 'tx00000b4df150a70433b40-00661ed229-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:53.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:31:53.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:31:53.875 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:31:53.930 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:31:53.931 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:31:53.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:53.984+0000 7fab435af800 1 waiting to reach incremental sync.. 2024-04-16T19:31:58.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:58.985+0000 7fab435af800 1 bucket sync caught up with source: 2024-04-16T19:31:58.987 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2024-04-16T19:31:58.987 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000002.10.11, , , , , ] 2024-04-16T19:31:58.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:31:58.985+0000 7fab435af800 0 bucket checkpoint complete 2024-04-16T19:31:59.011 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:31:59.012 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:31:59.012 DEBUG:boto:Method: HEAD 2024-04-16T19:31:59.012 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:31:59.012 DEBUG:boto:Data: 2024-04-16T19:31:59.012 DEBUG:boto:Headers: {} 2024-04-16T19:31:59.013 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:31:59.013 DEBUG:boto:Port: 8000 2024-04-16T19:31:59.013 DEBUG:boto:Params: {} 2024-04-16T19:31:59.013 DEBUG:boto:Token: None 2024-04-16T19:31:59.014 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:31:59 GMT /yrjefh-14/ 2024-04-16T19:31:59.014 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5mMXqRYVsYq2X2nHVe7c0phD5y4= 2024-04-16T19:31:59.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:31:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5mMXqRYVsYq2X2nHVe7c0phD5y4=', 'Content-Length': '0'} 2024-04-16T19:31:59.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f0284a7a09e6d26-00661ed22f-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:59.017 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:31:59.017 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:31:59.017 DEBUG:boto:Method: PUT 2024-04-16T19:31:59.017 DEBUG:boto:Path: /yrjefh-14/dummy 2024-04-16T19:31:59.017 DEBUG:boto:Data: 2024-04-16T19:31:59.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:31:59.017 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:31:59.017 DEBUG:boto:Port: 8000 2024-04-16T19:31:59.017 DEBUG:boto:Params: {} 2024-04-16T19:31:59.017 DEBUG:boto:Token: None 2024-04-16T19:31:59.018 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:31:59 GMT /yrjefh-14/dummy 2024-04-16T19:31:59.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5UTH52w7jHIGoCYH0vBvv3yPmV4= 2024-04-16T19:31:59.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:31:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5UTH52w7jHIGoCYH0vBvv3yPmV4='} 2024-04-16T19:31:59.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-amz-request-id', 'tx00000f3302a9f71bfbc30-00661ed22f-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:31:59.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:31:59.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:31:59.031 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:31:59.094 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:31:59.094 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:31:59.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:31:59.151+0000 7f1112867800 1 waiting to reach incremental sync.. 2024-04-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:32:04.153+0000 7f1112867800 1 bucket sync caught up with source: 2024-04-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000002.41.11, , , , , ] 2024-04-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.41.11, , , , , ] 2024-04-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:32:04.153+0000 7f1112867800 0 bucket checkpoint complete 2024-04-16T19:32:04.181 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:04.181 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:04.181 DEBUG:boto:Method: HEAD 2024-04-16T19:32:04.181 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:32:04.181 DEBUG:boto:Data: 2024-04-16T19:32:04.181 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.182 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.182 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.182 DEBUG:boto:Params: {} 2024-04-16T19:32:04.182 DEBUG:boto:Token: None 2024-04-16T19:32:04.182 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/ 2024-04-16T19:32:04.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hQTHsZ6QMdrqvh3CMaKcaoAarhA= 2024-04-16T19:32:04.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hQTHsZ6QMdrqvh3CMaKcaoAarhA=', 'Content-Length': '0'} 2024-04-16T19:32:04.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8a532bba8ba4d0e-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.186 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.186 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.186 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.186 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.186 DEBUG:boto:Data: 2024-04-16T19:32:04.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.187 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.187 DEBUG:boto:Params: {} 2024-04-16T19:32:04.187 DEBUG:boto:Token: None 2024-04-16T19:32:04.187 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKKrsSBhXUgYAuGd79c6MtBvz0w= 2024-04-16T19:32:04.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKKrsSBhXUgYAuGd79c6MtBvz0w='} 2024-04-16T19:32:04.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-amz-request-id', 'tx000000952a04279e4a5ef-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.201 DEBUG:rgw_multi.tests:version1 id=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:04.201 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.201 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.201 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.202 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.202 DEBUG:boto:Data: 2024-04-16T19:32:04.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.202 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.202 DEBUG:boto:Params: {} 2024-04-16T19:32:04.202 DEBUG:boto:Token: None 2024-04-16T19:32:04.203 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M9o3WSiFIt2alyrEwzUKFJjFg2E= 2024-04-16T19:32:04.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M9o3WSiFIt2alyrEwzUKFJjFg2E='} 2024-04-16T19:32:04.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hHCkeYClO82CfG-.OtpBHBmUVAe8VC-'), ('x-amz-request-id', 'tx00000abc6c178dc77955d-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.213 DEBUG:rgw_multi.tests:version2 id=hHCkeYClO82CfG-.OtpBHBmUVAe8VC- 2024-04-16T19:32:04.213 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.213 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.214 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=hHCkeYClO82CfG-.OtpBHBmUVAe8VC- 2024-04-16T19:32:04.214 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=hHCkeYClO82CfG-.OtpBHBmUVAe8VC- 2024-04-16T19:32:04.214 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.214 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=hHCkeYClO82CfG-.OtpBHBmUVAe8VC- 2024-04-16T19:32:04.214 DEBUG:boto:Data: 2024-04-16T19:32:04.214 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.214 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.214 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.215 DEBUG:boto:Params: {} 2024-04-16T19:32:04.215 DEBUG:boto:Token: None 2024-04-16T19:32:04.215 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone1?versionId=hHCkeYClO82CfG-.OtpBHBmUVAe8VC- 2024-04-16T19:32:04.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:of5+cQqjv1/CQGMhWlkYypH4br4= 2024-04-16T19:32:04.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:of5+cQqjv1/CQGMhWlkYypH4br4=', 'Content-Length': '0'} 2024-04-16T19:32:04.226 DEBUG:boto:Response headers: [('x-amz-version-id', 'hHCkeYClO82CfG-.OtpBHBmUVAe8VC-'), ('x-amz-request-id', 'tx00000c7af1f5c55ae81e7-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.226 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.226 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.227 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.227 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.227 DEBUG:boto:Data: 2024-04-16T19:32:04.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.227 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.227 DEBUG:boto:Params: {} 2024-04-16T19:32:04.227 DEBUG:boto:Token: None 2024-04-16T19:32:04.228 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Zt5uCU9Y9zT/Adu1ocIEC+E2R8= 2024-04-16T19:32:04.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Zt5uCU9Y9zT/Adu1ocIEC+E2R8='} 2024-04-16T19:32:04.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dDMcVO3NEojqseZpL989ZpqeUB1SLZf'), ('x-amz-request-id', 'tx000003504c92ec655ee44-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.271 DEBUG:rgw_multi.tests:version3 id=dDMcVO3NEojqseZpL989ZpqeUB1SLZf 2024-04-16T19:32:04.271 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.271 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:04.271 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=dDMcVO3NEojqseZpL989ZpqeUB1SLZf 2024-04-16T19:32:04.271 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=dDMcVO3NEojqseZpL989ZpqeUB1SLZf 2024-04-16T19:32:04.272 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.272 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=dDMcVO3NEojqseZpL989ZpqeUB1SLZf 2024-04-16T19:32:04.272 DEBUG:boto:Data: 2024-04-16T19:32:04.272 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.272 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.272 DEBUG:boto:Params: {} 2024-04-16T19:32:04.272 DEBUG:boto:Token: None 2024-04-16T19:32:04.273 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone1?versionId=dDMcVO3NEojqseZpL989ZpqeUB1SLZf 2024-04-16T19:32:04.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6ogZoteqPrGGZX4UyEKIU1OeU78= 2024-04-16T19:32:04.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6ogZoteqPrGGZX4UyEKIU1OeU78=', 'Content-Length': '0'} 2024-04-16T19:32:04.283 DEBUG:boto:Response headers: [('x-amz-version-id', 'dDMcVO3NEojqseZpL989ZpqeUB1SLZf'), ('x-amz-request-id', 'tx00000481f442ad8ecc45b-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.284 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:04.284 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:04.284 DEBUG:boto:Method: HEAD 2024-04-16T19:32:04.284 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:32:04.284 DEBUG:boto:Data: 2024-04-16T19:32:04.284 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.284 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.284 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.285 DEBUG:boto:Params: {} 2024-04-16T19:32:04.285 DEBUG:boto:Token: None 2024-04-16T19:32:04.285 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/ 2024-04-16T19:32:04.285 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hQTHsZ6QMdrqvh3CMaKcaoAarhA= 2024-04-16T19:32:04.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hQTHsZ6QMdrqvh3CMaKcaoAarhA=', 'Content-Length': '0'} 2024-04-16T19:32:04.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008938cfabbcb3225e-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.288 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.288 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.289 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.289 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.289 DEBUG:boto:Data: 2024-04-16T19:32:04.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.289 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.289 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.289 DEBUG:boto:Params: {} 2024-04-16T19:32:04.289 DEBUG:boto:Token: None 2024-04-16T19:32:04.290 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.290 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aF7gMdIGHOtIL5riuhys4hZHhOE= 2024-04-16T19:32:04.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aF7gMdIGHOtIL5riuhys4hZHhOE='} 2024-04-16T19:32:04.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-amz-request-id', 'tx0000060e4958ffa9d559b-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.305 DEBUG:rgw_multi.tests:version1 id=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:04.305 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.305 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.305 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.305 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.305 DEBUG:boto:Data: 2024-04-16T19:32:04.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.306 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.306 DEBUG:boto:Params: {} 2024-04-16T19:32:04.306 DEBUG:boto:Token: None 2024-04-16T19:32:04.306 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.306 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3G8yKlSIQ1UrkUtZKTi+2kgM0kc= 2024-04-16T19:32:04.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3G8yKlSIQ1UrkUtZKTi+2kgM0kc='} 2024-04-16T19:32:04.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TMZmFwjercb0LrlgAPCUggqhDHHP0W3'), ('x-amz-request-id', 'tx00000a012705834d432d1-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.351 DEBUG:rgw_multi.tests:version2 id=TMZmFwjercb0LrlgAPCUggqhDHHP0W3 2024-04-16T19:32:04.351 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.351 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.351 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=TMZmFwjercb0LrlgAPCUggqhDHHP0W3 2024-04-16T19:32:04.351 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=TMZmFwjercb0LrlgAPCUggqhDHHP0W3 2024-04-16T19:32:04.351 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.352 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=TMZmFwjercb0LrlgAPCUggqhDHHP0W3 2024-04-16T19:32:04.352 DEBUG:boto:Data: 2024-04-16T19:32:04.352 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.352 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.352 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.352 DEBUG:boto:Params: {} 2024-04-16T19:32:04.352 DEBUG:boto:Token: None 2024-04-16T19:32:04.352 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone2?versionId=TMZmFwjercb0LrlgAPCUggqhDHHP0W3 2024-04-16T19:32:04.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bpt8xQ4qBw2TmNKISBZmZMesf5A= 2024-04-16T19:32:04.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bpt8xQ4qBw2TmNKISBZmZMesf5A=', 'Content-Length': '0'} 2024-04-16T19:32:04.365 DEBUG:boto:Response headers: [('x-amz-version-id', 'TMZmFwjercb0LrlgAPCUggqhDHHP0W3'), ('x-amz-request-id', 'tx000008b06e1bbb5828cd8-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.366 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.366 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.366 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.366 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.366 DEBUG:boto:Data: 2024-04-16T19:32:04.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.367 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.367 DEBUG:boto:Params: {} 2024-04-16T19:32:04.367 DEBUG:boto:Token: None 2024-04-16T19:32:04.368 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CAc/EJT453uMuZEju1e4hr29QUY= 2024-04-16T19:32:04.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CAc/EJT453uMuZEju1e4hr29QUY='} 2024-04-16T19:32:04.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO'), ('x-amz-request-id', 'tx000008ee5060a004d3b01-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.380 DEBUG:rgw_multi.tests:version3 id=ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO 2024-04-16T19:32:04.380 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.380 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:04.380 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO 2024-04-16T19:32:04.380 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO 2024-04-16T19:32:04.380 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.380 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO 2024-04-16T19:32:04.381 DEBUG:boto:Data: 2024-04-16T19:32:04.381 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.381 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.381 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.381 DEBUG:boto:Params: {} 2024-04-16T19:32:04.381 DEBUG:boto:Token: None 2024-04-16T19:32:04.381 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-13/obj-test-zone2?versionId=ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO 2024-04-16T19:32:04.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z/+w1GpcojAq5/YpiHZ3Z/wON8I= 2024-04-16T19:32:04.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z/+w1GpcojAq5/YpiHZ3Z/wON8I=', 'Content-Length': '0'} 2024-04-16T19:32:04.394 DEBUG:boto:Response headers: [('x-amz-version-id', 'ZMRWMHPYkFUkmrRHhnod5RGxEg90dHO'), ('x-amz-request-id', 'tx00000b2b09d3a0784e53e-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.394 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:32:04.394 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:32:04.395 DEBUG:boto:Method: HEAD 2024-04-16T19:32:04.395 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:32:04.395 DEBUG:boto:Data: 2024-04-16T19:32:04.395 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.395 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.395 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.395 DEBUG:boto:Params: {} 2024-04-16T19:32:04.395 DEBUG:boto:Token: None 2024-04-16T19:32:04.395 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/ 2024-04-16T19:32:04.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VsJ2FYfCTMmI1Nug2TCfe7ed7rY= 2024-04-16T19:32:04.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VsJ2FYfCTMmI1Nug2TCfe7ed7rY=', 'Content-Length': '0'} 2024-04-16T19:32:04.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c048eeece76b102-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.398 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.398 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.398 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.398 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.399 DEBUG:boto:Data: 2024-04-16T19:32:04.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.399 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.399 DEBUG:boto:Params: {} 2024-04-16T19:32:04.399 DEBUG:boto:Token: None 2024-04-16T19:32:04.399 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2i0P8iq4xqiDDmd4clE4FFyBKNw= 2024-04-16T19:32:04.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2i0P8iq4xqiDDmd4clE4FFyBKNw='} 2024-04-16T19:32:04.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-amz-request-id', 'tx00000ce3736545cc655f0-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.411 DEBUG:rgw_multi.tests:version1 id=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:32:04.411 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.412 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.412 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.412 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.412 DEBUG:boto:Data: 2024-04-16T19:32:04.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.412 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.412 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.412 DEBUG:boto:Params: {} 2024-04-16T19:32:04.412 DEBUG:boto:Token: None 2024-04-16T19:32:04.413 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kGxZKr+11uO72IqivKTcwfSLpC8= 2024-04-16T19:32:04.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kGxZKr+11uO72IqivKTcwfSLpC8='} 2024-04-16T19:32:04.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fGAwcvybDb549nrI0FTHppAIHJ5uBLV'), ('x-amz-request-id', 'tx000000aed9b3e3f1fe6c7-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.424 DEBUG:rgw_multi.tests:version2 id=fGAwcvybDb549nrI0FTHppAIHJ5uBLV 2024-04-16T19:32:04.425 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.425 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.425 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fGAwcvybDb549nrI0FTHppAIHJ5uBLV 2024-04-16T19:32:04.425 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fGAwcvybDb549nrI0FTHppAIHJ5uBLV 2024-04-16T19:32:04.425 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.425 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fGAwcvybDb549nrI0FTHppAIHJ5uBLV 2024-04-16T19:32:04.425 DEBUG:boto:Data: 2024-04-16T19:32:04.425 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.425 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.425 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.426 DEBUG:boto:Params: {} 2024-04-16T19:32:04.426 DEBUG:boto:Token: None 2024-04-16T19:32:04.426 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone1?versionId=fGAwcvybDb549nrI0FTHppAIHJ5uBLV 2024-04-16T19:32:04.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b0uRV475Ng3MpjV3VqZ3WhKac9o= 2024-04-16T19:32:04.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b0uRV475Ng3MpjV3VqZ3WhKac9o=', 'Content-Length': '0'} 2024-04-16T19:32:04.437 DEBUG:boto:Response headers: [('x-amz-version-id', 'fGAwcvybDb549nrI0FTHppAIHJ5uBLV'), ('x-amz-request-id', 'tx0000073ebeaaa3b48d3ef-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.438 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.438 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.438 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.438 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.438 DEBUG:boto:Data: 2024-04-16T19:32:04.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.438 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.439 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.439 DEBUG:boto:Params: {} 2024-04-16T19:32:04.439 DEBUG:boto:Token: None 2024-04-16T19:32:04.439 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y5+PUWo8Vob4KSIhat0PyORlXq0= 2024-04-16T19:32:04.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y5+PUWo8Vob4KSIhat0PyORlXq0='} 2024-04-16T19:32:04.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aVDa0-4XRu43wTBGjGBop90hJqAo2hP'), ('x-amz-request-id', 'tx000001d38f0019546c634-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.483 DEBUG:rgw_multi.tests:version3 id=aVDa0-4XRu43wTBGjGBop90hJqAo2hP 2024-04-16T19:32:04.483 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.483 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:32:04.483 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=aVDa0-4XRu43wTBGjGBop90hJqAo2hP 2024-04-16T19:32:04.483 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=aVDa0-4XRu43wTBGjGBop90hJqAo2hP 2024-04-16T19:32:04.483 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.484 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=aVDa0-4XRu43wTBGjGBop90hJqAo2hP 2024-04-16T19:32:04.484 DEBUG:boto:Data: 2024-04-16T19:32:04.484 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.484 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:04.484 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.484 DEBUG:boto:Params: {} 2024-04-16T19:32:04.484 DEBUG:boto:Token: None 2024-04-16T19:32:04.484 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone1?versionId=aVDa0-4XRu43wTBGjGBop90hJqAo2hP 2024-04-16T19:32:04.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Sc6upMpyC9hsmHLzbK9b7grSxGU= 2024-04-16T19:32:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Sc6upMpyC9hsmHLzbK9b7grSxGU=', 'Content-Length': '0'} 2024-04-16T19:32:04.496 DEBUG:boto:Response headers: [('x-amz-version-id', 'aVDa0-4XRu43wTBGjGBop90hJqAo2hP'), ('x-amz-request-id', 'tx000000c28d16a66f65668-00661ed234-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.497 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:32:04.497 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:32:04.497 DEBUG:boto:Method: HEAD 2024-04-16T19:32:04.497 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:32:04.497 DEBUG:boto:Data: 2024-04-16T19:32:04.497 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.497 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.498 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.498 DEBUG:boto:Params: {} 2024-04-16T19:32:04.498 DEBUG:boto:Token: None 2024-04-16T19:32:04.498 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/ 2024-04-16T19:32:04.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VsJ2FYfCTMmI1Nug2TCfe7ed7rY= 2024-04-16T19:32:04.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VsJ2FYfCTMmI1Nug2TCfe7ed7rY=', 'Content-Length': '0'} 2024-04-16T19:32:04.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000897b5733db492998-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.501 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.501 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.501 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.501 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.501 DEBUG:boto:Data: 2024-04-16T19:32:04.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.502 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.502 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.502 DEBUG:boto:Params: {} 2024-04-16T19:32:04.502 DEBUG:boto:Token: None 2024-04-16T19:32:04.502 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Cu1CpxlkTwBx2j/xZ8HaAKFO7Oc= 2024-04-16T19:32:04.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Cu1CpxlkTwBx2j/xZ8HaAKFO7Oc='} 2024-04-16T19:32:04.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-amz-request-id', 'tx00000c721a0b044f5df9e-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.515 DEBUG:rgw_multi.tests:version1 id=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:32:04.515 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.516 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.516 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.516 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.516 DEBUG:boto:Data: 2024-04-16T19:32:04.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.516 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.516 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.516 DEBUG:boto:Params: {} 2024-04-16T19:32:04.517 DEBUG:boto:Token: None 2024-04-16T19:32:04.517 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q+iFcEX0zBOYGlIrkBy/i9d/AE8= 2024-04-16T19:32:04.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q+iFcEX0zBOYGlIrkBy/i9d/AE8='} 2024-04-16T19:32:04.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kn-jcgchiEEChN4paj5e27DtOrv9z60'), ('x-amz-request-id', 'tx000001a94ad3b439fe29d-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.530 DEBUG:rgw_multi.tests:version2 id=kn-jcgchiEEChN4paj5e27DtOrv9z60 2024-04-16T19:32:04.530 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.530 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.530 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=kn-jcgchiEEChN4paj5e27DtOrv9z60 2024-04-16T19:32:04.530 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=kn-jcgchiEEChN4paj5e27DtOrv9z60 2024-04-16T19:32:04.530 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.530 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=kn-jcgchiEEChN4paj5e27DtOrv9z60 2024-04-16T19:32:04.530 DEBUG:boto:Data: 2024-04-16T19:32:04.530 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.531 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.531 DEBUG:boto:Params: {} 2024-04-16T19:32:04.531 DEBUG:boto:Token: None 2024-04-16T19:32:04.531 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone2?versionId=kn-jcgchiEEChN4paj5e27DtOrv9z60 2024-04-16T19:32:04.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aX7xf2Rtb8irb74e1JkLUWp7gjg= 2024-04-16T19:32:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aX7xf2Rtb8irb74e1JkLUWp7gjg=', 'Content-Length': '0'} 2024-04-16T19:32:04.542 DEBUG:boto:Response headers: [('x-amz-version-id', 'kn-jcgchiEEChN4paj5e27DtOrv9z60'), ('x-amz-request-id', 'tx00000c076624bb688623a-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.543 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.543 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.543 DEBUG:boto:Method: PUT 2024-04-16T19:32:04.543 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.543 DEBUG:boto:Data: 2024-04-16T19:32:04.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:32:04.543 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.544 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.544 DEBUG:boto:Params: {} 2024-04-16T19:32:04.544 DEBUG:boto:Token: None 2024-04-16T19:32:04.544 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UUsddct+9c32xHXq7PmPgfplQeI= 2024-04-16T19:32:04.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UUsddct+9c32xHXq7PmPgfplQeI='} 2024-04-16T19:32:04.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HoPRBbWv770t6mLBH0P4dr319FkDsfE'), ('x-amz-request-id', 'tx000003e8c85b1fd097ee8-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.587 DEBUG:rgw_multi.tests:version3 id=HoPRBbWv770t6mLBH0P4dr319FkDsfE 2024-04-16T19:32:04.587 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.587 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:32:04.587 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=HoPRBbWv770t6mLBH0P4dr319FkDsfE 2024-04-16T19:32:04.587 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=HoPRBbWv770t6mLBH0P4dr319FkDsfE 2024-04-16T19:32:04.587 DEBUG:boto:Method: DELETE 2024-04-16T19:32:04.588 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=HoPRBbWv770t6mLBH0P4dr319FkDsfE 2024-04-16T19:32:04.588 DEBUG:boto:Data: 2024-04-16T19:32:04.588 DEBUG:boto:Headers: {} 2024-04-16T19:32:04.588 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:04.588 DEBUG:boto:Port: 8000 2024-04-16T19:32:04.588 DEBUG:boto:Params: {} 2024-04-16T19:32:04.588 DEBUG:boto:Token: None 2024-04-16T19:32:04.588 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:32:04 GMT /yrjefh-14/obj-test-zone2?versionId=HoPRBbWv770t6mLBH0P4dr319FkDsfE 2024-04-16T19:32:04.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grrXvFdBFEeDfAVFOjPxiQ3m2sU= 2024-04-16T19:32:04.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grrXvFdBFEeDfAVFOjPxiQ3m2sU=', 'Content-Length': '0'} 2024-04-16T19:32:04.600 DEBUG:boto:Response headers: [('x-amz-version-id', 'HoPRBbWv770t6mLBH0P4dr319FkDsfE'), ('x-amz-request-id', 'tx00000bcce7427b920da6b-00661ed234-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:04.600 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-13 2024-04-16T19:32:04.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:04.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:04.601 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:32:04.660 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:32:04.660 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:32:04.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:04.715+0000 7f88ef726800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:04.717 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2024-04-16T19:32:04.717 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.32.6, , , , 00000000003.12.10, , , , , ] 2024-04-16T19:32:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:09.717+0000 7f88ef726800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:09.718 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000010.32.6, , 00000000002.16.11, , 00000000002.10.11, , , , , ] 2024-04-16T19:32:09.718 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.32.6, , , , 00000000003.12.10, , , , , ] 2024-04-16T19:32:14.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:14.718+0000 7f88ef726800 1 bucket sync caught up with source: 2024-04-16T19:32:14.720 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000010.32.6, , 00000000002.16.11, , 00000000003.12.10, , , , , ] 2024-04-16T19:32:14.720 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.32.6, , , , 00000000003.12.10, , , , , ] 2024-04-16T19:32:14.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:14.718+0000 7f88ef726800 0 bucket checkpoint complete 2024-04-16T19:32:14.744 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-13 2024-04-16T19:32:14.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:14.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:14.744 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-13 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:32:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:32:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:32:14.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:32:14.867+0000 7f585c928800 1 bucket sync caught up with source: 2024-04-16T19:32:14.869 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.17.11, , 00000000010.38.6, , 00000000002.20.11, , , , , ] 2024-04-16T19:32:14.869 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.17.11, , 00000000010.38.6, , 00000000002.20.11, , , , , ] 2024-04-16T19:32:14.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:32:14.867+0000 7f585c928800 0 bucket checkpoint complete 2024-04-16T19:32:14.889 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-13 zones={test-zone2, test-zone1} 2024-04-16T19:32:14.890 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.890 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.890 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.890 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:32:14.890 DEBUG:boto:Data: 2024-04-16T19:32:14.890 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.890 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.890 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.890 DEBUG:boto:Params: {} 2024-04-16T19:32:14.890 DEBUG:boto:Token: None 2024-04-16T19:32:14.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/ 2024-04-16T19:32:14.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kilVT1XqYWZXvMSP4OKEjB23s20= 2024-04-16T19:32:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kilVT1XqYWZXvMSP4OKEjB23s20=', 'Content-Length': '0'} 2024-04-16T19:32:14.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040c8c0f074c3de6b-00661ed23e-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.894 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.894 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.894 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.894 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:32:14.894 DEBUG:boto:Data: 2024-04-16T19:32:14.894 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.894 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.894 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.894 DEBUG:boto:Params: {} 2024-04-16T19:32:14.894 DEBUG:boto:Token: None 2024-04-16T19:32:14.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/ 2024-04-16T19:32:14.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kilVT1XqYWZXvMSP4OKEjB23s20= 2024-04-16T19:32:14.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kilVT1XqYWZXvMSP4OKEjB23s20=', 'Content-Length': '0'} 2024-04-16T19:32:14.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2a0bf10443f1ea0-00661ed23e-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:32:14.898 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.898 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.898 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.898 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.898 DEBUG:boto:Method: GET 2024-04-16T19:32:14.898 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.898 DEBUG:boto:Data: 2024-04-16T19:32:14.898 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.898 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.898 DEBUG:boto:Params: {} 2024-04-16T19:32:14.899 DEBUG:boto:Token: None 2024-04-16T19:32:14.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/?versions 2024-04-16T19:32:14.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4= 2024-04-16T19:32:14.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4=', 'Content-Length': '0'} 2024-04-16T19:32:14.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474844e316256254-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.902 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:32:14.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:32:14.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:32:14.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:32:14.903 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.903 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.903 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.904 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.904 DEBUG:boto:Method: GET 2024-04-16T19:32:14.904 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.904 DEBUG:boto:Data: 2024-04-16T19:32:14.904 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.904 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.904 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.904 DEBUG:boto:Params: {} 2024-04-16T19:32:14.904 DEBUG:boto:Token: None 2024-04-16T19:32:14.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/?versions 2024-04-16T19:32:14.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4= 2024-04-16T19:32:14.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4=', 'Content-Length': '0'} 2024-04-16T19:32:14.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0462bff5cd8f6f-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.907 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.907 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:32:14.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:32:14.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:32:14.907 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.907 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.907 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.908 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.908 DEBUG:boto:Method: GET 2024-04-16T19:32:14.908 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.908 DEBUG:boto:Data: 2024-04-16T19:32:14.908 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.908 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.908 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.908 DEBUG:boto:Params: {} 2024-04-16T19:32:14.908 DEBUG:boto:Token: None 2024-04-16T19:32:14.908 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/?versions 2024-04-16T19:32:14.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4= 2024-04-16T19:32:14.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4=', 'Content-Length': '0'} 2024-04-16T19:32:14.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b26c0896dbbcacf-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.911 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.911 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.911 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.912 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.912 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.912 DEBUG:boto:Method: GET 2024-04-16T19:32:14.912 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:32:14.912 DEBUG:boto:Data: 2024-04-16T19:32:14.912 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.912 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.912 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.912 DEBUG:boto:Params: {} 2024-04-16T19:32:14.912 DEBUG:boto:Token: None 2024-04-16T19:32:14.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/?versions 2024-04-16T19:32:14.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4= 2024-04-16T19:32:14.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:orL2iO5g9/PbQEA4hAcqY4Lnzu4=', 'Content-Length': '0'} 2024-04-16T19:32:14.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068075814543364ba-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.915 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:32:14.916 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.916 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.916 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.916 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.916 DEBUG:boto:Method: GET 2024-04-16T19:32:14.916 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.916 DEBUG:boto:Data: 2024-04-16T19:32:14.916 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.916 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.916 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.917 DEBUG:boto:Params: {} 2024-04-16T19:32:14.917 DEBUG:boto:Token: None 2024-04-16T19:32:14.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:14.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:14.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a03fbae2ef178b0-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.919 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.919 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.919 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.919 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.920 DEBUG:boto:Method: GET 2024-04-16T19:32:14.920 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.920 DEBUG:boto:Data: 2024-04-16T19:32:14.920 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.920 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.920 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.920 DEBUG:boto:Params: {} 2024-04-16T19:32:14.920 DEBUG:boto:Token: None 2024-04-16T19:32:14.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:14.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdfee981dd2fd5a7-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.923 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.923 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.923 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.923 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.923 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.923 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.923 DEBUG:boto:Data: 2024-04-16T19:32:14.923 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.923 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.924 DEBUG:boto:Params: {} 2024-04-16T19:32:14.924 DEBUG:boto:Token: None 2024-04-16T19:32:14.924 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DWRxXySxLpkU//0wXB8peWXKkWg= 2024-04-16T19:32:14.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWRxXySxLpkU//0wXB8peWXKkWg=', 'Content-Length': '0'} 2024-04-16T19:32:14.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bc2525a565171f7-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.926 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.926 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.926 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.926 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.926 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.926 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.927 DEBUG:boto:Data: 2024-04-16T19:32:14.927 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.927 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.927 DEBUG:boto:Params: {} 2024-04-16T19:32:14.927 DEBUG:boto:Token: None 2024-04-16T19:32:14.927 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DWRxXySxLpkU//0wXB8peWXKkWg= 2024-04-16T19:32:14.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWRxXySxLpkU//0wXB8peWXKkWg=', 'Content-Length': '0'} 2024-04-16T19:32:14.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004cb6190b5b75751b-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:32:14.929 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.929 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.929 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.930 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.930 DEBUG:boto:Method: GET 2024-04-16T19:32:14.930 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.930 DEBUG:boto:Data: 2024-04-16T19:32:14.930 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.930 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.930 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.930 DEBUG:boto:Params: {} 2024-04-16T19:32:14.930 DEBUG:boto:Token: None 2024-04-16T19:32:14.930 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.930 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:14.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:14.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b891340ba7a4218-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.933 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.933 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.933 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.933 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.933 DEBUG:boto:Method: GET 2024-04-16T19:32:14.933 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.933 DEBUG:boto:Data: 2024-04-16T19:32:14.933 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.934 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.934 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.934 DEBUG:boto:Params: {} 2024-04-16T19:32:14.934 DEBUG:boto:Token: None 2024-04-16T19:32:14.934 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.934 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:14.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:14.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000388bf856f88b0b72-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.936 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.936 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.936 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.936 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.937 DEBUG:boto:Method: GET 2024-04-16T19:32:14.937 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.937 DEBUG:boto:Data: 2024-04-16T19:32:14.937 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.937 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.937 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.937 DEBUG:boto:Params: {} 2024-04-16T19:32:14.937 DEBUG:boto:Token: None 2024-04-16T19:32:14.937 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/ 2024-04-16T19:32:14.937 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x587lWc4+zXoW/viIy0lZ24AiAk= 2024-04-16T19:32:14.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x587lWc4+zXoW/viIy0lZ24AiAk=', 'Content-Length': '0'} 2024-04-16T19:32:14.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006193f9c76d98b4da-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.982 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.983 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:14.983 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:14.983 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.983 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.983 DEBUG:boto:Method: GET 2024-04-16T19:32:14.983 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:32:14.983 DEBUG:boto:Data: 2024-04-16T19:32:14.983 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.984 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.984 DEBUG:boto:Params: {} 2024-04-16T19:32:14.984 DEBUG:boto:Token: None 2024-04-16T19:32:14.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/ 2024-04-16T19:32:14.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x587lWc4+zXoW/viIy0lZ24AiAk= 2024-04-16T19:32:14.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x587lWc4+zXoW/viIy0lZ24AiAk=', 'Content-Length': '0'} 2024-04-16T19:32:14.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe0a12a69692132-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.986 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:14.987 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.987 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.987 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.987 DEBUG:boto:Path: /yrjefh-13/dummy 2024-04-16T19:32:14.987 DEBUG:boto:Data: 2024-04-16T19:32:14.987 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.987 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.987 DEBUG:boto:Params: {} 2024-04-16T19:32:14.987 DEBUG:boto:Token: None 2024-04-16T19:32:14.988 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy 2024-04-16T19:32:14.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JuVRfU7n/+3tuZF5HID66tEXFk0= 2024-04-16T19:32:14.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JuVRfU7n/+3tuZF5HID66tEXFk0=', 'Content-Length': '0'} 2024-04-16T19:32:14.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea65577a1d1279d4-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.991 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.991 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.991 DEBUG:boto:Method: HEAD 2024-04-16T19:32:14.991 DEBUG:boto:Path: /yrjefh-13/dummy 2024-04-16T19:32:14.991 DEBUG:boto:Data: 2024-04-16T19:32:14.991 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.991 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.991 DEBUG:boto:Params: {} 2024-04-16T19:32:14.992 DEBUG:boto:Token: None 2024-04-16T19:32:14.992 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy 2024-04-16T19:32:14.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JuVRfU7n/+3tuZF5HID66tEXFk0= 2024-04-16T19:32:14.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JuVRfU7n/+3tuZF5HID66tEXFk0=', 'Content-Length': '0'} 2024-04-16T19:32:14.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b9326a1654f85c5-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:32:14.995 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.995 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.995 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.995 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.995 DEBUG:boto:Method: GET 2024-04-16T19:32:14.995 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.995 DEBUG:boto:Data: 2024-04-16T19:32:14.995 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:14.995 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.995 DEBUG:boto:Params: {} 2024-04-16T19:32:14.996 DEBUG:boto:Token: None 2024-04-16T19:32:14.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:14.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:14.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009859fc25960c7b27-00661ed23e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:14.998 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:32:14.998 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:32:14.999 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.999 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.999 DEBUG:boto:Method: GET 2024-04-16T19:32:14.999 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:14.999 DEBUG:boto:Data: 2024-04-16T19:32:14.999 DEBUG:boto:Headers: {} 2024-04-16T19:32:14.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:14.999 DEBUG:boto:Port: 8000 2024-04-16T19:32:14.999 DEBUG:boto:Params: {} 2024-04-16T19:32:14.999 DEBUG:boto:Token: None 2024-04-16T19:32:14.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:14 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:32:15.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM= 2024-04-16T19:32:15.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VM5Cn6RhgL1yyazAOyo8GpybyIM=', 'Content-Length': '0'} 2024-04-16T19:32:15.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d546de44ea46358-00661ed23e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.002 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.002 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.002 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.002 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.002 DEBUG:boto:Method: GET 2024-04-16T19:32:15.002 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.002 DEBUG:boto:Data: 2024-04-16T19:32:15.003 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.003 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.003 DEBUG:boto:Params: {} 2024-04-16T19:32:15.003 DEBUG:boto:Token: None 2024-04-16T19:32:15.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e07ef6695cceadc2-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.006 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.006 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.006 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.007 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.007 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.007 DEBUG:boto:Method: GET 2024-04-16T19:32:15.007 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:32:15.007 DEBUG:boto:Data: 2024-04-16T19:32:15.007 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.007 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.007 DEBUG:boto:Params: {} 2024-04-16T19:32:15.007 DEBUG:boto:Token: None 2024-04-16T19:32:15.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004454d2fe287e441b-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.010 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:32:15.010 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.011 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.011 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.011 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.011 DEBUG:boto:Method: GET 2024-04-16T19:32:15.011 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.011 DEBUG:boto:Data: 2024-04-16T19:32:15.011 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.011 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.011 DEBUG:boto:Params: {} 2024-04-16T19:32:15.011 DEBUG:boto:Token: None 2024-04-16T19:32:15.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2badc2e88b2d995-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.054 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.054 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.055 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.055 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.055 DEBUG:boto:Method: GET 2024-04-16T19:32:15.055 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.055 DEBUG:boto:Data: 2024-04-16T19:32:15.055 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.055 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.055 DEBUG:boto:Params: {} 2024-04-16T19:32:15.055 DEBUG:boto:Token: None 2024-04-16T19:32:15.055 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3e4aa1b83280429-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.058 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.058 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.058 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.058 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.058 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.058 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.058 DEBUG:boto:Data: 2024-04-16T19:32:15.058 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.058 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.059 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.059 DEBUG:boto:Params: {} 2024-04-16T19:32:15.059 DEBUG:boto:Token: None 2024-04-16T19:32:15.059 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A7s7AWQF9Gn79MDMQc71mP9IYGY= 2024-04-16T19:32:15.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A7s7AWQF9Gn79MDMQc71mP9IYGY=', 'Content-Length': '0'} 2024-04-16T19:32:15.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e11757c382ac2208-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.061 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.061 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.061 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.062 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.062 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.062 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.062 DEBUG:boto:Data: 2024-04-16T19:32:15.062 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.062 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.062 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.062 DEBUG:boto:Params: {} 2024-04-16T19:32:15.062 DEBUG:boto:Token: None 2024-04-16T19:32:15.062 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A7s7AWQF9Gn79MDMQc71mP9IYGY= 2024-04-16T19:32:15.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A7s7AWQF9Gn79MDMQc71mP9IYGY=', 'Content-Length': '0'} 2024-04-16T19:32:15.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000526af38c0d8c659a-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:32:15.065 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.065 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.065 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.065 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.065 DEBUG:boto:Method: GET 2024-04-16T19:32:15.065 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.065 DEBUG:boto:Data: 2024-04-16T19:32:15.065 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.065 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.065 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.065 DEBUG:boto:Params: {} 2024-04-16T19:32:15.066 DEBUG:boto:Token: None 2024-04-16T19:32:15.066 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.066 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ca2081710fb70d9-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.110 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.110 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.111 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.111 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.111 DEBUG:boto:Method: GET 2024-04-16T19:32:15.111 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.111 DEBUG:boto:Data: 2024-04-16T19:32:15.111 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.111 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.111 DEBUG:boto:Params: {} 2024-04-16T19:32:15.111 DEBUG:boto:Token: None 2024-04-16T19:32:15.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e99956b2fe94cc-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.114 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.114 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.114 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.114 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.114 DEBUG:boto:Method: GET 2024-04-16T19:32:15.114 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.114 DEBUG:boto:Data: 2024-04-16T19:32:15.114 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.115 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.115 DEBUG:boto:Params: {} 2024-04-16T19:32:15.115 DEBUG:boto:Token: None 2024-04-16T19:32:15.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021cba617fadcf0e6-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.118 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.118 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.118 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.118 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.118 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.118 DEBUG:boto:Method: GET 2024-04-16T19:32:15.119 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.119 DEBUG:boto:Data: 2024-04-16T19:32:15.119 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.119 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.119 DEBUG:boto:Params: {} 2024-04-16T19:32:15.119 DEBUG:boto:Token: None 2024-04-16T19:32:15.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007caf4f86c060da29-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.122 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.122 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.122 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.122 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.122 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.123 DEBUG:boto:Data: 2024-04-16T19:32:15.123 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.123 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.123 DEBUG:boto:Params: {} 2024-04-16T19:32:15.123 DEBUG:boto:Token: None 2024-04-16T19:32:15.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.123 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:la8rFLpSwFCHe0QHwQIEL0SuTB8= 2024-04-16T19:32:15.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:la8rFLpSwFCHe0QHwQIEL0SuTB8=', 'Content-Length': '0'} 2024-04-16T19:32:15.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e948790c04e7e2e4-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.126 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.126 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.126 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.126 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.126 DEBUG:boto:Data: 2024-04-16T19:32:15.126 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.126 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.126 DEBUG:boto:Params: {} 2024-04-16T19:32:15.127 DEBUG:boto:Token: None 2024-04-16T19:32:15.127 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:la8rFLpSwFCHe0QHwQIEL0SuTB8= 2024-04-16T19:32:15.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:la8rFLpSwFCHe0QHwQIEL0SuTB8=', 'Content-Length': '0'} 2024-04-16T19:32:15.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a213f86f11e95086-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:32:15.129 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.129 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.130 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.130 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.130 DEBUG:boto:Method: GET 2024-04-16T19:32:15.130 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.130 DEBUG:boto:Data: 2024-04-16T19:32:15.130 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.130 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.130 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.130 DEBUG:boto:Params: {} 2024-04-16T19:32:15.130 DEBUG:boto:Token: None 2024-04-16T19:32:15.131 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c90fa8d321f96d3-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.178 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.178 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:32:15.179 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.179 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.179 DEBUG:boto:Method: GET 2024-04-16T19:32:15.179 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.179 DEBUG:boto:Data: 2024-04-16T19:32:15.179 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.179 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.179 DEBUG:boto:Params: {} 2024-04-16T19:32:15.179 DEBUG:boto:Token: None 2024-04-16T19:32:15.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:32:15.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po= 2024-04-16T19:32:15.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0hfK0Eg2J3wLcCx3wnyDrEfI3Po=', 'Content-Length': '0'} 2024-04-16T19:32:15.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019d7c623e2c40e90-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.182 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.182 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.182 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.182 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.182 DEBUG:boto:Method: GET 2024-04-16T19:32:15.182 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.182 DEBUG:boto:Data: 2024-04-16T19:32:15.182 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.183 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.183 DEBUG:boto:Params: {} 2024-04-16T19:32:15.183 DEBUG:boto:Token: None 2024-04-16T19:32:15.183 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7f3fbf28676b579-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.186 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.186 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.186 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.186 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.186 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.187 DEBUG:boto:Method: GET 2024-04-16T19:32:15.187 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:32:15.187 DEBUG:boto:Data: 2024-04-16T19:32:15.187 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.187 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.187 DEBUG:boto:Params: {} 2024-04-16T19:32:15.187 DEBUG:boto:Token: None 2024-04-16T19:32:15.187 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef357abd835b34c-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.190 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:32:15.190 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.190 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.190 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.190 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.190 DEBUG:boto:Method: GET 2024-04-16T19:32:15.190 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.190 DEBUG:boto:Data: 2024-04-16T19:32:15.191 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.191 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.191 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.191 DEBUG:boto:Params: {} 2024-04-16T19:32:15.191 DEBUG:boto:Token: None 2024-04-16T19:32:15.191 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc4314255ecfc88-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.194 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.194 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.194 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.194 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.194 DEBUG:boto:Method: GET 2024-04-16T19:32:15.194 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.194 DEBUG:boto:Data: 2024-04-16T19:32:15.195 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.195 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.195 DEBUG:boto:Params: {} 2024-04-16T19:32:15.195 DEBUG:boto:Token: None 2024-04-16T19:32:15.195 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca22c29e37bf9d0-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.198 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.198 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.198 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.199 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.199 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.199 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.199 DEBUG:boto:Data: 2024-04-16T19:32:15.199 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.199 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.199 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.199 DEBUG:boto:Params: {} 2024-04-16T19:32:15.199 DEBUG:boto:Token: None 2024-04-16T19:32:15.199 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pLJWrh6s1v82bf7KUNn/3qgyJ9Q= 2024-04-16T19:32:15.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pLJWrh6s1v82bf7KUNn/3qgyJ9Q=', 'Content-Length': '0'} 2024-04-16T19:32:15.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b57bcc904f79da9d-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.202 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.202 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.202 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.202 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.202 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.203 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.203 DEBUG:boto:Data: 2024-04-16T19:32:15.203 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.203 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.203 DEBUG:boto:Params: {} 2024-04-16T19:32:15.203 DEBUG:boto:Token: None 2024-04-16T19:32:15.203 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pLJWrh6s1v82bf7KUNn/3qgyJ9Q= 2024-04-16T19:32:15.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pLJWrh6s1v82bf7KUNn/3qgyJ9Q=', 'Content-Length': '0'} 2024-04-16T19:32:15.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064a1545b24cea2c4-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:32:15.206 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.206 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.206 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.206 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.206 DEBUG:boto:Method: GET 2024-04-16T19:32:15.206 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.206 DEBUG:boto:Data: 2024-04-16T19:32:15.206 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.207 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.207 DEBUG:boto:Params: {} 2024-04-16T19:32:15.207 DEBUG:boto:Token: None 2024-04-16T19:32:15.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000841f1ea81db127a0-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.210 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.210 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.210 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.210 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.210 DEBUG:boto:Method: GET 2024-04-16T19:32:15.210 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.210 DEBUG:boto:Data: 2024-04-16T19:32:15.210 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.210 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.210 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.210 DEBUG:boto:Params: {} 2024-04-16T19:32:15.211 DEBUG:boto:Token: None 2024-04-16T19:32:15.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000221a3f85ca18535d-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.254 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.255 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.255 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.255 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.255 DEBUG:boto:Method: GET 2024-04-16T19:32:15.255 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.255 DEBUG:boto:Data: 2024-04-16T19:32:15.255 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.255 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.255 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.255 DEBUG:boto:Params: {} 2024-04-16T19:32:15.255 DEBUG:boto:Token: None 2024-04-16T19:32:15.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd316bd63aa5b2d-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.258 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.259 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.259 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.259 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.259 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.259 DEBUG:boto:Method: GET 2024-04-16T19:32:15.259 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.259 DEBUG:boto:Data: 2024-04-16T19:32:15.259 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.259 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.260 DEBUG:boto:Params: {} 2024-04-16T19:32:15.260 DEBUG:boto:Token: None 2024-04-16T19:32:15.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e88d76399c73259-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.306 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.307 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.307 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.307 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.307 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.307 DEBUG:boto:Data: 2024-04-16T19:32:15.307 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.307 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.307 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.307 DEBUG:boto:Params: {} 2024-04-16T19:32:15.307 DEBUG:boto:Token: None 2024-04-16T19:32:15.308 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P5Nx7Ofn+C3rFmyZLQYaPp81hl0= 2024-04-16T19:32:15.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P5Nx7Ofn+C3rFmyZLQYaPp81hl0=', 'Content-Length': '0'} 2024-04-16T19:32:15.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2a6696a2c0f714-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.310 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.311 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.311 DEBUG:boto:Method: HEAD 2024-04-16T19:32:15.311 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.311 DEBUG:boto:Data: 2024-04-16T19:32:15.311 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.311 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.311 DEBUG:boto:Params: {} 2024-04-16T19:32:15.311 DEBUG:boto:Token: None 2024-04-16T19:32:15.311 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P5Nx7Ofn+C3rFmyZLQYaPp81hl0= 2024-04-16T19:32:15.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P5Nx7Ofn+C3rFmyZLQYaPp81hl0=', 'Content-Length': '0'} 2024-04-16T19:32:15.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027ab02b3f9d19004-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:32:15.314 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.314 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.314 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.315 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.315 DEBUG:boto:Method: GET 2024-04-16T19:32:15.315 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.315 DEBUG:boto:Data: 2024-04-16T19:32:15.315 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.315 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.315 DEBUG:boto:Params: {} 2024-04-16T19:32:15.315 DEBUG:boto:Token: None 2024-04-16T19:32:15.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000186c0aaad8b0f121-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.318 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.318 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:32:15.318 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.319 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.319 DEBUG:boto:Method: GET 2024-04-16T19:32:15.319 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.319 DEBUG:boto:Data: 2024-04-16T19:32:15.319 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.319 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.319 DEBUG:boto:Params: {} 2024-04-16T19:32:15.319 DEBUG:boto:Token: None 2024-04-16T19:32:15.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:32:15.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0= 2024-04-16T19:32:15.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nwHMDI6PBZxlxzKCbkR6dSsQAf0=', 'Content-Length': '0'} 2024-04-16T19:32:15.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe41edcb3bed4e7-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.362 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.362 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.363 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.363 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.363 DEBUG:boto:Method: GET 2024-04-16T19:32:15.363 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.363 DEBUG:boto:Data: 2024-04-16T19:32:15.363 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.363 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:32:15.363 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.363 DEBUG:boto:Params: {} 2024-04-16T19:32:15.363 DEBUG:boto:Token: None 2024-04-16T19:32:15.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d37b30252c2b1b-00661ed23f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.366 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.366 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:32:15.367 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:32:15.367 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.367 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.367 DEBUG:boto:Method: GET 2024-04-16T19:32:15.367 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:32:15.367 DEBUG:boto:Data: 2024-04-16T19:32:15.367 DEBUG:boto:Headers: {} 2024-04-16T19:32:15.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:32:15.367 DEBUG:boto:Port: 8000 2024-04-16T19:32:15.367 DEBUG:boto:Params: {} 2024-04-16T19:32:15.367 DEBUG:boto:Token: None 2024-04-16T19:32:15.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:32:15 GMT /yrjefh-13/ 2024-04-16T19:32:15.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs= 2024-04-16T19:32:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:32:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Q3NCBdh7Gmpy4diqjAEJXlUYLs=', 'Content-Length': '0'} 2024-04-16T19:32:15.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe8842514d88771-00661ed23f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:32:15.370 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:32:15.370 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-13 zones={test-zone2, test-zone1} 2024-04-16T19:32:15.371 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-14 2024-04-16T19:32:15.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:15.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:32:15.371 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-14 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:32:15.433 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:32:15.434 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:32:15.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:15.492+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:15.493 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:15.493 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:20.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:20.493+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:20.495 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:20.496 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:25.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:25.495+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:25.497 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:25.497 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:30.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:30.497+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:30.498 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:30.499 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:35.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:35.497+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:35.501 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:35.501 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:40.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:40.500+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:40.502 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:40.502 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:45.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:45.502+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:45.504 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:45.504 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:50.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:50.503+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:50.505 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:50.505 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:55.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:32:55.505+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:32:55.507 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:32:55.507 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:00.506+0000 7f79aa975800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:00.508 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000008.33.12, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:00.508 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:05.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:05.507+0000 7f79aa975800 1 bucket sync caught up with source: 2024-04-16T19:33:05.509 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:05.509 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000010.37.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:05.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:05.507+0000 7f79aa975800 0 bucket checkpoint complete 2024-04-16T19:33:05.533 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-14 2024-04-16T19:33:05.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:05.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:05.533 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:05.656+0000 7fb3a7824800 1 bucket sync caught up with source: 2024-04-16T19:33:05.658 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.24.11, , 00000000010.40.6, , 00000000003.43.10, , , , , ] 2024-04-16T19:33:05.658 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.24.11, , 00000000010.40.6, , 00000000003.43.10, , , , , ] 2024-04-16T19:33:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:05.656+0000 7fb3a7824800 0 bucket checkpoint complete 2024-04-16T19:33:05.679 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-14 zones={test-zone2, test-zone1} 2024-04-16T19:33:05.679 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.679 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.679 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.679 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:05.679 DEBUG:boto:Data: 2024-04-16T19:33:05.679 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.680 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.680 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.680 DEBUG:boto:Params: {} 2024-04-16T19:33:05.680 DEBUG:boto:Token: None 2024-04-16T19:33:05.680 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y2GW0o6uWTwhWHYsiSm315PWDsE= 2024-04-16T19:33:05.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y2GW0o6uWTwhWHYsiSm315PWDsE=', 'Content-Length': '0'} 2024-04-16T19:33:05.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046e575c4e9992943-00661ed271-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.683 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.683 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.683 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.683 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:05.683 DEBUG:boto:Data: 2024-04-16T19:33:05.684 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.684 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.684 DEBUG:boto:Params: {} 2024-04-16T19:33:05.684 DEBUG:boto:Token: None 2024-04-16T19:33:05.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y2GW0o6uWTwhWHYsiSm315PWDsE= 2024-04-16T19:33:05.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y2GW0o6uWTwhWHYsiSm315PWDsE=', 'Content-Length': '0'} 2024-04-16T19:33:05.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cafcae8d7a603e1a-00661ed271-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.686 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:33:05.686 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.687 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.687 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.687 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.687 DEBUG:boto:Method: GET 2024-04-16T19:33:05.687 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.687 DEBUG:boto:Data: 2024-04-16T19:33:05.687 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.687 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.687 DEBUG:boto:Params: {} 2024-04-16T19:33:05.687 DEBUG:boto:Token: None 2024-04-16T19:33:05.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/?versions 2024-04-16T19:33:05.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4= 2024-04-16T19:33:05.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4=', 'Content-Length': '0'} 2024-04-16T19:33:05.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dccd8cd502a30e5-00661ed271-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.730 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:05.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:05.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:05.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:33:05.731 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.732 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.732 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.732 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.732 DEBUG:boto:Method: GET 2024-04-16T19:33:05.732 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.732 DEBUG:boto:Data: 2024-04-16T19:33:05.732 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.732 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.732 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.732 DEBUG:boto:Params: {} 2024-04-16T19:33:05.732 DEBUG:boto:Token: None 2024-04-16T19:33:05.733 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/?versions 2024-04-16T19:33:05.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4= 2024-04-16T19:33:05.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4=', 'Content-Length': '0'} 2024-04-16T19:33:05.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2f949d6ad0fe51-00661ed271-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.735 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.736 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:05.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:05.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:05.736 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.736 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.736 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.736 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.736 DEBUG:boto:Method: GET 2024-04-16T19:33:05.736 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.736 DEBUG:boto:Data: 2024-04-16T19:33:05.736 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.737 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.737 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.737 DEBUG:boto:Params: {} 2024-04-16T19:33:05.737 DEBUG:boto:Token: None 2024-04-16T19:33:05.737 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/?versions 2024-04-16T19:33:05.737 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4= 2024-04-16T19:33:05.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4=', 'Content-Length': '0'} 2024-04-16T19:33:05.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cbd798c03f88811-00661ed271-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.740 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.740 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.741 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.741 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.741 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.741 DEBUG:boto:Method: GET 2024-04-16T19:33:05.741 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:05.741 DEBUG:boto:Data: 2024-04-16T19:33:05.741 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.741 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.741 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.741 DEBUG:boto:Params: {} 2024-04-16T19:33:05.741 DEBUG:boto:Token: None 2024-04-16T19:33:05.741 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/?versions 2024-04-16T19:33:05.742 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4= 2024-04-16T19:33:05.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VfZTMIHVPws456J2VWpVCdOqXL4=', 'Content-Length': '0'} 2024-04-16T19:33:05.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d991f13812a2d1d8-00661ed271-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.786 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:05.787 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.787 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.787 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.787 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.787 DEBUG:boto:Method: GET 2024-04-16T19:33:05.787 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.787 DEBUG:boto:Data: 2024-04-16T19:33:05.787 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.788 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.788 DEBUG:boto:Params: {} 2024-04-16T19:33:05.788 DEBUG:boto:Token: None 2024-04-16T19:33:05.788 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000671a1459a5f5605e-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.790 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.790 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.790 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.790 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.791 DEBUG:boto:Method: GET 2024-04-16T19:33:05.791 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.791 DEBUG:boto:Data: 2024-04-16T19:33:05.791 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.791 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.791 DEBUG:boto:Params: {} 2024-04-16T19:33:05.791 DEBUG:boto:Token: None 2024-04-16T19:33:05.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044dc5e10abd6ef7f-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.793 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.793 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.793 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.793 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.794 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.794 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.794 DEBUG:boto:Data: 2024-04-16T19:33:05.794 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.794 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.794 DEBUG:boto:Params: {} 2024-04-16T19:33:05.794 DEBUG:boto:Token: None 2024-04-16T19:33:05.794 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aMkn+8OBV/WhKygKbsowB9PrE8Y= 2024-04-16T19:33:05.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aMkn+8OBV/WhKygKbsowB9PrE8Y=', 'Content-Length': '0'} 2024-04-16T19:33:05.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b0a8b60965ef85e-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.797 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.797 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.797 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.797 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.797 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.797 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.797 DEBUG:boto:Data: 2024-04-16T19:33:05.797 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.797 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.797 DEBUG:boto:Params: {} 2024-04-16T19:33:05.798 DEBUG:boto:Token: None 2024-04-16T19:33:05.798 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aMkn+8OBV/WhKygKbsowB9PrE8Y= 2024-04-16T19:33:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aMkn+8OBV/WhKygKbsowB9PrE8Y=', 'Content-Length': '0'} 2024-04-16T19:33:05.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7c572586899c4e5-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:05.800 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.800 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.800 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.800 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.800 DEBUG:boto:Method: GET 2024-04-16T19:33:05.800 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.800 DEBUG:boto:Data: 2024-04-16T19:33:05.800 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.800 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.801 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.801 DEBUG:boto:Params: {} 2024-04-16T19:33:05.801 DEBUG:boto:Token: None 2024-04-16T19:33:05.801 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092d527c004f71fa7-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.803 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.803 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.803 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.803 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.803 DEBUG:boto:Method: GET 2024-04-16T19:33:05.803 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.804 DEBUG:boto:Data: 2024-04-16T19:33:05.804 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.804 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.804 DEBUG:boto:Params: {} 2024-04-16T19:33:05.804 DEBUG:boto:Token: None 2024-04-16T19:33:05.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebdbdfb9b10e5c2f-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.806 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.806 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.806 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.806 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.806 DEBUG:boto:Method: GET 2024-04-16T19:33:05.807 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.807 DEBUG:boto:Data: 2024-04-16T19:33:05.807 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.807 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.807 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.807 DEBUG:boto:Params: {} 2024-04-16T19:33:05.807 DEBUG:boto:Token: None 2024-04-16T19:33:05.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62fda32e68097ee-00661ed271-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.810 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.810 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.810 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.810 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.810 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.810 DEBUG:boto:Method: GET 2024-04-16T19:33:05.811 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.811 DEBUG:boto:Data: 2024-04-16T19:33:05.811 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.811 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.811 DEBUG:boto:Params: {} 2024-04-16T19:33:05.811 DEBUG:boto:Token: None 2024-04-16T19:33:05.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95791a842dcd40b-00661ed271-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.814 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.814 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.814 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.814 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.814 DEBUG:boto:Path: /yrjefh-14/dummy 2024-04-16T19:33:05.814 DEBUG:boto:Data: 2024-04-16T19:33:05.814 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.815 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.815 DEBUG:boto:Params: {} 2024-04-16T19:33:05.815 DEBUG:boto:Token: None 2024-04-16T19:33:05.815 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy 2024-04-16T19:33:05.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlGLqttqq0Pf5k8ksdPaQSYuYTY= 2024-04-16T19:33:05.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlGLqttqq0Pf5k8ksdPaQSYuYTY=', 'Content-Length': '0'} 2024-04-16T19:33:05.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017c558b4db709d5a-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.818 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.818 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.818 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.818 DEBUG:boto:Path: /yrjefh-14/dummy 2024-04-16T19:33:05.818 DEBUG:boto:Data: 2024-04-16T19:33:05.819 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.819 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.819 DEBUG:boto:Params: {} 2024-04-16T19:33:05.819 DEBUG:boto:Token: None 2024-04-16T19:33:05.819 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy 2024-04-16T19:33:05.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlGLqttqq0Pf5k8ksdPaQSYuYTY= 2024-04-16T19:33:05.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlGLqttqq0Pf5k8ksdPaQSYuYTY=', 'Content-Length': '0'} 2024-04-16T19:33:05.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036a8e06eb7aa6873-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:05.821 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.822 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.822 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.822 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.822 DEBUG:boto:Method: GET 2024-04-16T19:33:05.822 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.822 DEBUG:boto:Data: 2024-04-16T19:33:05.822 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.822 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.822 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.822 DEBUG:boto:Params: {} 2024-04-16T19:33:05.822 DEBUG:boto:Token: None 2024-04-16T19:33:05.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001b04ca8343063f1-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.825 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:05.825 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:05.825 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.825 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.825 DEBUG:boto:Method: GET 2024-04-16T19:33:05.825 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.826 DEBUG:boto:Data: 2024-04-16T19:33:05.826 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.826 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.826 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.826 DEBUG:boto:Params: {} 2024-04-16T19:33:05.826 DEBUG:boto:Token: None 2024-04-16T19:33:05.826 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:05.826 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8= 2024-04-16T19:33:05.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8NZpd6wK1SyNT+rlwE2EVDvVf8=', 'Content-Length': '0'} 2024-04-16T19:33:05.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087f349764aecf519-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.828 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.828 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.828 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.828 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.828 DEBUG:boto:Method: GET 2024-04-16T19:33:05.828 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.828 DEBUG:boto:Data: 2024-04-16T19:33:05.829 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.829 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.829 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.829 DEBUG:boto:Params: {} 2024-04-16T19:33:05.829 DEBUG:boto:Token: None 2024-04-16T19:33:05.829 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.829 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa047f61727e7244-00661ed271-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.831 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.832 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.832 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.832 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.832 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.832 DEBUG:boto:Method: GET 2024-04-16T19:33:05.832 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:05.832 DEBUG:boto:Data: 2024-04-16T19:33:05.832 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.832 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.832 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.833 DEBUG:boto:Params: {} 2024-04-16T19:33:05.833 DEBUG:boto:Token: None 2024-04-16T19:33:05.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea20e10080d5300-00661ed271-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.835 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:05.836 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.836 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.836 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.836 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.836 DEBUG:boto:Method: GET 2024-04-16T19:33:05.836 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.836 DEBUG:boto:Data: 2024-04-16T19:33:05.836 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.836 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.836 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.836 DEBUG:boto:Params: {} 2024-04-16T19:33:05.837 DEBUG:boto:Token: None 2024-04-16T19:33:05.837 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.837 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE= 2024-04-16T19:33:05.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE=', 'Content-Length': '0'} 2024-04-16T19:33:05.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036545712dcf09f84-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.882 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.883 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.883 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.883 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.883 DEBUG:boto:Method: GET 2024-04-16T19:33:05.883 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.883 DEBUG:boto:Data: 2024-04-16T19:33:05.883 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.883 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.883 DEBUG:boto:Params: {} 2024-04-16T19:33:05.883 DEBUG:boto:Token: None 2024-04-16T19:33:05.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE= 2024-04-16T19:33:05.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE=', 'Content-Length': '0'} 2024-04-16T19:33:05.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000418bff821638ab68-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.886 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.886 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.886 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.886 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.886 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.886 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.886 DEBUG:boto:Data: 2024-04-16T19:33:05.886 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.886 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.887 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.887 DEBUG:boto:Params: {} 2024-04-16T19:33:05.887 DEBUG:boto:Token: None 2024-04-16T19:33:05.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8tlMTWYuNAyl9a7k02VMyZJH690= 2024-04-16T19:33:05.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8tlMTWYuNAyl9a7k02VMyZJH690=', 'Content-Length': '0'} 2024-04-16T19:33:05.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af93da68e9839ea3-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.889 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.889 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.889 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.889 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.889 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.890 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.890 DEBUG:boto:Data: 2024-04-16T19:33:05.890 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.890 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.890 DEBUG:boto:Params: {} 2024-04-16T19:33:05.890 DEBUG:boto:Token: None 2024-04-16T19:33:05.890 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.890 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8tlMTWYuNAyl9a7k02VMyZJH690= 2024-04-16T19:33:05.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8tlMTWYuNAyl9a7k02VMyZJH690=', 'Content-Length': '0'} 2024-04-16T19:33:05.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071afc660cf54b7e5-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:05.892 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.892 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.893 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.893 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.893 DEBUG:boto:Method: GET 2024-04-16T19:33:05.893 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.893 DEBUG:boto:Data: 2024-04-16T19:33:05.893 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.893 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.893 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.893 DEBUG:boto:Params: {} 2024-04-16T19:33:05.893 DEBUG:boto:Token: None 2024-04-16T19:33:05.893 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE= 2024-04-16T19:33:05.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE=', 'Content-Length': '0'} 2024-04-16T19:33:05.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003094054cc980a462-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.938 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.938 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.939 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.939 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.939 DEBUG:boto:Method: GET 2024-04-16T19:33:05.939 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.939 DEBUG:boto:Data: 2024-04-16T19:33:05.939 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.939 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.939 DEBUG:boto:Params: {} 2024-04-16T19:33:05.939 DEBUG:boto:Token: None 2024-04-16T19:33:05.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE= 2024-04-16T19:33:05.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE=', 'Content-Length': '0'} 2024-04-16T19:33:05.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009293536bc248c31c-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.942 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.942 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.942 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.942 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.942 DEBUG:boto:Method: GET 2024-04-16T19:33:05.942 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.942 DEBUG:boto:Data: 2024-04-16T19:33:05.942 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.942 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.943 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.943 DEBUG:boto:Params: {} 2024-04-16T19:33:05.943 DEBUG:boto:Token: None 2024-04-16T19:33:05.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b043526567b4a6-00661ed271-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.945 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.946 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:05.946 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:05.946 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.946 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.946 DEBUG:boto:Method: GET 2024-04-16T19:33:05.946 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:05.946 DEBUG:boto:Data: 2024-04-16T19:33:05.946 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.946 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.946 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.947 DEBUG:boto:Params: {} 2024-04-16T19:33:05.947 DEBUG:boto:Token: None 2024-04-16T19:33:05.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/ 2024-04-16T19:33:05.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0= 2024-04-16T19:33:05.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FT6/QMvy3kwYLdBmnuNvlIwfwY0=', 'Content-Length': '0'} 2024-04-16T19:33:05.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001baa5d042cd88ed4-00661ed271-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.949 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:05.950 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.950 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.950 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.950 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.950 DEBUG:boto:Data: 2024-04-16T19:33:05.950 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.950 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.950 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.950 DEBUG:boto:Params: {} 2024-04-16T19:33:05.950 DEBUG:boto:Token: None 2024-04-16T19:33:05.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JOJDZvkceB+6Dd4zE+wUyTZd4ss= 2024-04-16T19:33:05.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JOJDZvkceB+6Dd4zE+wUyTZd4ss=', 'Content-Length': '0'} 2024-04-16T19:33:05.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013ab32ae4a6c47c4-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.954 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.954 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.954 DEBUG:boto:Method: HEAD 2024-04-16T19:33:05.954 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.954 DEBUG:boto:Data: 2024-04-16T19:33:05.954 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.954 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:05.954 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.954 DEBUG:boto:Params: {} 2024-04-16T19:33:05.954 DEBUG:boto:Token: None 2024-04-16T19:33:05.955 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JOJDZvkceB+6Dd4zE+wUyTZd4ss= 2024-04-16T19:33:05.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JOJDZvkceB+6Dd4zE+wUyTZd4ss=', 'Content-Length': '0'} 2024-04-16T19:33:05.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b48320ad47a75d0-00661ed271-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:05.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:05.957 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.957 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:05.957 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.958 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.958 DEBUG:boto:Method: GET 2024-04-16T19:33:05.958 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.958 DEBUG:boto:Data: 2024-04-16T19:33:05.958 DEBUG:boto:Headers: {} 2024-04-16T19:33:05.958 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:05.958 DEBUG:boto:Port: 8000 2024-04-16T19:33:05.958 DEBUG:boto:Params: {} 2024-04-16T19:33:05.958 DEBUG:boto:Token: None 2024-04-16T19:33:05.958 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:05 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:05.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE= 2024-04-16T19:33:05.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:85R1f0+miRZDig5amhKl3QiSEzE=', 'Content-Length': '0'} 2024-04-16T19:33:05.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004daa4aab17706cb0-00661ed271-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.002 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:06.002 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:06.003 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:06.003 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:06.003 DEBUG:boto:Method: GET 2024-04-16T19:33:06.003 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:06.003 DEBUG:boto:Data: 2024-04-16T19:33:06.003 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.003 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.003 DEBUG:boto:Params: {} 2024-04-16T19:33:06.003 DEBUG:boto:Token: None 2024-04-16T19:33:06.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:06.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oAyja7bh50MqklTRMBglKivr1D0= 2024-04-16T19:33:06.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oAyja7bh50MqklTRMBglKivr1D0=', 'Content-Length': '0'} 2024-04-16T19:33:06.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac5b635396412e9e-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.006 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.006 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.006 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.006 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.006 DEBUG:boto:Method: GET 2024-04-16T19:33:06.006 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.006 DEBUG:boto:Data: 2024-04-16T19:33:06.007 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.007 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.007 DEBUG:boto:Params: {} 2024-04-16T19:33:06.007 DEBUG:boto:Token: None 2024-04-16T19:33:06.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbfc7918a8f5ec4e-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.050 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.051 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.051 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.051 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.051 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.051 DEBUG:boto:Method: GET 2024-04-16T19:33:06.051 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:06.051 DEBUG:boto:Data: 2024-04-16T19:33:06.051 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.051 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.051 DEBUG:boto:Params: {} 2024-04-16T19:33:06.052 DEBUG:boto:Token: None 2024-04-16T19:33:06.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e7cd0640ee5a07-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.054 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:06.055 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.055 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.055 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.055 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.055 DEBUG:boto:Method: GET 2024-04-16T19:33:06.055 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.055 DEBUG:boto:Data: 2024-04-16T19:33:06.055 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.055 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.055 DEBUG:boto:Params: {} 2024-04-16T19:33:06.056 DEBUG:boto:Token: None 2024-04-16T19:33:06.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011520028f7c95a44-00661ed272-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.058 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.058 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.058 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.059 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.059 DEBUG:boto:Method: GET 2024-04-16T19:33:06.059 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.059 DEBUG:boto:Data: 2024-04-16T19:33:06.059 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.059 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.059 DEBUG:boto:Params: {} 2024-04-16T19:33:06.059 DEBUG:boto:Token: None 2024-04-16T19:33:06.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e190fcdee552ad52-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.106 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.107 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.107 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.107 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.107 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.107 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.107 DEBUG:boto:Data: 2024-04-16T19:33:06.107 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.107 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.107 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.108 DEBUG:boto:Params: {} 2024-04-16T19:33:06.108 DEBUG:boto:Token: None 2024-04-16T19:33:06.108 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dTY6gP+arUza5meik6HxgloYCrE= 2024-04-16T19:33:06.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dTY6gP+arUza5meik6HxgloYCrE=', 'Content-Length': '0'} 2024-04-16T19:33:06.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ac34d680d10e4c0-00661ed272-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.110 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.110 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.111 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.111 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.111 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.111 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.111 DEBUG:boto:Data: 2024-04-16T19:33:06.111 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.111 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.111 DEBUG:boto:Params: {} 2024-04-16T19:33:06.111 DEBUG:boto:Token: None 2024-04-16T19:33:06.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dTY6gP+arUza5meik6HxgloYCrE= 2024-04-16T19:33:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dTY6gP+arUza5meik6HxgloYCrE=', 'Content-Length': '0'} 2024-04-16T19:33:06.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3bd7e1bcc513b97-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:06.114 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.114 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.114 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.114 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.114 DEBUG:boto:Method: GET 2024-04-16T19:33:06.114 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.114 DEBUG:boto:Data: 2024-04-16T19:33:06.114 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.115 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.115 DEBUG:boto:Params: {} 2024-04-16T19:33:06.115 DEBUG:boto:Token: None 2024-04-16T19:33:06.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070d73df1c2ec79f1-00661ed272-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.118 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.118 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.118 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.118 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.118 DEBUG:boto:Method: GET 2024-04-16T19:33:06.118 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.118 DEBUG:boto:Data: 2024-04-16T19:33:06.118 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.118 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.118 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.119 DEBUG:boto:Params: {} 2024-04-16T19:33:06.119 DEBUG:boto:Token: None 2024-04-16T19:33:06.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000287a58114e6ed4ba-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.162 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.163 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.163 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.163 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.163 DEBUG:boto:Method: GET 2024-04-16T19:33:06.163 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.163 DEBUG:boto:Data: 2024-04-16T19:33:06.163 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.163 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.163 DEBUG:boto:Params: {} 2024-04-16T19:33:06.164 DEBUG:boto:Token: None 2024-04-16T19:33:06.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b70fa7c7ed7e459-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.167 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.167 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.167 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.167 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.167 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.167 DEBUG:boto:Method: GET 2024-04-16T19:33:06.168 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.168 DEBUG:boto:Data: 2024-04-16T19:33:06.168 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.168 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.168 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.168 DEBUG:boto:Params: {} 2024-04-16T19:33:06.168 DEBUG:boto:Token: None 2024-04-16T19:33:06.168 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a0eab940a1220f-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.171 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.171 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.171 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.171 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.172 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.172 DEBUG:boto:Data: 2024-04-16T19:33:06.172 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.172 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.172 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.172 DEBUG:boto:Params: {} 2024-04-16T19:33:06.172 DEBUG:boto:Token: None 2024-04-16T19:33:06.172 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZMq5X8UxUYxWWFpUJm1Nxhxayz0= 2024-04-16T19:33:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZMq5X8UxUYxWWFpUJm1Nxhxayz0=', 'Content-Length': '0'} 2024-04-16T19:33:06.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac6649322d3abf9-00661ed272-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.175 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.175 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.175 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.176 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.176 DEBUG:boto:Data: 2024-04-16T19:33:06.176 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.176 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.176 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.176 DEBUG:boto:Params: {} 2024-04-16T19:33:06.176 DEBUG:boto:Token: None 2024-04-16T19:33:06.176 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZMq5X8UxUYxWWFpUJm1Nxhxayz0= 2024-04-16T19:33:06.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZMq5X8UxUYxWWFpUJm1Nxhxayz0=', 'Content-Length': '0'} 2024-04-16T19:33:06.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000737d0c52df75c7b2-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:06.179 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.179 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.179 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.179 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.179 DEBUG:boto:Method: GET 2024-04-16T19:33:06.179 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.180 DEBUG:boto:Data: 2024-04-16T19:33:06.180 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.180 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.180 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.180 DEBUG:boto:Params: {} 2024-04-16T19:33:06.180 DEBUG:boto:Token: None 2024-04-16T19:33:06.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009819423b6433ff41-00661ed272-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.222 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.222 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.223 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.223 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.223 DEBUG:boto:Method: GET 2024-04-16T19:33:06.223 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.223 DEBUG:boto:Data: 2024-04-16T19:33:06.223 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.223 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.223 DEBUG:boto:Params: {} 2024-04-16T19:33:06.223 DEBUG:boto:Token: None 2024-04-16T19:33:06.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:06.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s= 2024-04-16T19:33:06.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d8oAu3Vtfm7xfCDWrduGmEuRv8s=', 'Content-Length': '0'} 2024-04-16T19:33:06.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea74c49d0660ef02-00661ed272-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.266 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.267 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.267 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.267 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.267 DEBUG:boto:Method: GET 2024-04-16T19:33:06.267 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.267 DEBUG:boto:Data: 2024-04-16T19:33:06.267 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.267 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.267 DEBUG:boto:Params: {} 2024-04-16T19:33:06.268 DEBUG:boto:Token: None 2024-04-16T19:33:06.268 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d158513eb542073-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.270 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.271 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.271 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.271 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.271 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.271 DEBUG:boto:Method: GET 2024-04-16T19:33:06.271 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:06.271 DEBUG:boto:Data: 2024-04-16T19:33:06.271 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.272 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.272 DEBUG:boto:Params: {} 2024-04-16T19:33:06.272 DEBUG:boto:Token: None 2024-04-16T19:33:06.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4= 2024-04-16T19:33:06.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nItv7ivJ9C3W2miLG7s5XHqZTQ4=', 'Content-Length': '0'} 2024-04-16T19:33:06.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae3d57c9c1aa3ca8-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.275 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.275 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-14 zones={test-zone2, test-zone1} 2024-04-16T19:33:06.275 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:06.275 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:06.275 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.275 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:33:06.275 DEBUG:boto:Data: 2024-04-16T19:33:06.276 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.276 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.276 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.276 DEBUG:boto:Params: {} 2024-04-16T19:33:06.276 DEBUG:boto:Token: None 2024-04-16T19:33:06.276 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-13/ 2024-04-16T19:33:06.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rG1XJtcc9Q1+QHy8nxQ3x0jTTNA= 2024-04-16T19:33:06.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rG1XJtcc9Q1+QHy8nxQ3x0jTTNA=', 'Content-Length': '0'} 2024-04-16T19:33:06.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f20b11bda164053-00661ed272-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.279 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:06.279 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:06.279 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.279 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.279 DEBUG:boto:Method: GET 2024-04-16T19:33:06.279 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.279 DEBUG:boto:Data: 2024-04-16T19:33:06.279 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.280 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.280 DEBUG:boto:Params: {} 2024-04-16T19:33:06.280 DEBUG:boto:Token: None 2024-04-16T19:33:06.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-13/?versions 2024-04-16T19:33:06.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:66PavVXHgeyCM0D3hhM1FqW1UxQ= 2024-04-16T19:33:06.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:66PavVXHgeyCM0D3hhM1FqW1UxQ=', 'Content-Length': '0'} 2024-04-16T19:33:06.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18b974040c3840b-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.326 DEBUG:boto:b'yrjefh-13obj-test-zone1999falseobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.327 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:06.327 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:06.327 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?acl 2024-04-16T19:33:06.327 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?acl 2024-04-16T19:33:06.327 DEBUG:boto:Method: PUT 2024-04-16T19:33:06.327 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?acl 2024-04-16T19:33:06.327 DEBUG:boto:Data: 2024-04-16T19:33:06.327 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:33:06.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.328 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.328 DEBUG:boto:Params: {} 2024-04-16T19:33:06.328 DEBUG:boto:Token: None 2024-04-16T19:33:06.328 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:06 GMT x-amz-acl:public-read /yrjefh-13/obj-test-zone1?acl 2024-04-16T19:33:06.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BsS9LJ5Nq6+3WZX1w9oGzbzsQp0= 2024-04-16T19:33:06.328 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BsS9LJ5Nq6+3WZX1w9oGzbzsQp0=', 'Content-Length': '0'} 2024-04-16T19:33:06.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e656671ccbc12223-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.337 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:06.337 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:06.337 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.337 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:33:06.337 DEBUG:boto:Data: 2024-04-16T19:33:06.337 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.337 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.337 DEBUG:boto:Params: {} 2024-04-16T19:33:06.338 DEBUG:boto:Token: None 2024-04-16T19:33:06.338 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-13/ 2024-04-16T19:33:06.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rG1XJtcc9Q1+QHy8nxQ3x0jTTNA= 2024-04-16T19:33:06.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rG1XJtcc9Q1+QHy8nxQ3x0jTTNA=', 'Content-Length': '0'} 2024-04-16T19:33:06.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b34591f5ebcfd82f-00661ed272-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.341 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:06.341 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:06.341 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.341 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.341 DEBUG:boto:Method: GET 2024-04-16T19:33:06.341 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.341 DEBUG:boto:Data: 2024-04-16T19:33:06.341 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.341 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.341 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.341 DEBUG:boto:Params: {} 2024-04-16T19:33:06.342 DEBUG:boto:Token: None 2024-04-16T19:33:06.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-13/?versions 2024-04-16T19:33:06.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:66PavVXHgeyCM0D3hhM1FqW1UxQ= 2024-04-16T19:33:06.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:66PavVXHgeyCM0D3hhM1FqW1UxQ=', 'Content-Length': '0'} 2024-04-16T19:33:06.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006484adfaf3636ae7-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.344 DEBUG:boto:b'yrjefh-13obj-test-zone2999falseobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.345 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:06.345 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:06.345 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?acl 2024-04-16T19:33:06.345 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?acl 2024-04-16T19:33:06.345 DEBUG:boto:Method: PUT 2024-04-16T19:33:06.345 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?acl 2024-04-16T19:33:06.345 DEBUG:boto:Data: 2024-04-16T19:33:06.345 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:33:06.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.346 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.346 DEBUG:boto:Params: {} 2024-04-16T19:33:06.346 DEBUG:boto:Token: None 2024-04-16T19:33:06.346 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:06 GMT x-amz-acl:public-read /yrjefh-13/obj-test-zone2?acl 2024-04-16T19:33:06.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Lb1slQ88KXa8qNlXUK8hfyJB5XA= 2024-04-16T19:33:06.346 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Lb1slQ88KXa8qNlXUK8hfyJB5XA=', 'Content-Length': '0'} 2024-04-16T19:33:06.353 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005561ab4c28e4c624-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.354 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.354 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.354 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.354 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:06.354 DEBUG:boto:Data: 2024-04-16T19:33:06.354 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.354 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.354 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.354 DEBUG:boto:Params: {} 2024-04-16T19:33:06.354 DEBUG:boto:Token: None 2024-04-16T19:33:06.355 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VFuV9ELP78puB4AcBMatOXNmRDk= 2024-04-16T19:33:06.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VFuV9ELP78puB4AcBMatOXNmRDk=', 'Content-Length': '0'} 2024-04-16T19:33:06.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f7f4f9420881b83-00661ed272-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.357 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.357 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.357 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.357 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.358 DEBUG:boto:Method: GET 2024-04-16T19:33:06.358 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-16T19:33:06.358 DEBUG:boto:Data: 2024-04-16T19:33:06.358 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.358 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.358 DEBUG:boto:Params: {} 2024-04-16T19:33:06.358 DEBUG:boto:Token: None 2024-04-16T19:33:06.358 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/?versions 2024-04-16T19:33:06.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fJFVQiDloDXzNqClMzlTlRqos+o= 2024-04-16T19:33:06.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fJFVQiDloDXzNqClMzlTlRqos+o=', 'Content-Length': '0'} 2024-04-16T19:33:06.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983e6a2f726367ef-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.361 DEBUG:boto:b'yrjefh-14obj-test-zone1999falseobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.361 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:06.361 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:06.362 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?acl 2024-04-16T19:33:06.362 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?acl 2024-04-16T19:33:06.362 DEBUG:boto:Method: PUT 2024-04-16T19:33:06.362 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?acl 2024-04-16T19:33:06.362 DEBUG:boto:Data: 2024-04-16T19:33:06.362 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:33:06.362 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:06.362 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.362 DEBUG:boto:Params: {} 2024-04-16T19:33:06.362 DEBUG:boto:Token: None 2024-04-16T19:33:06.363 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:06 GMT x-amz-acl:public-read /yrjefh-14/obj-test-zone1?acl 2024-04-16T19:33:06.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KstdiWeXHZfxTUyIkh2Gj+nZGiI= 2024-04-16T19:33:06.363 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KstdiWeXHZfxTUyIkh2Gj+nZGiI=', 'Content-Length': '0'} 2024-04-16T19:33:06.370 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb1d853191b4981c-00661ed272-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.370 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.370 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.370 DEBUG:boto:Method: HEAD 2024-04-16T19:33:06.370 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:06.370 DEBUG:boto:Data: 2024-04-16T19:33:06.370 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.370 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.370 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.370 DEBUG:boto:Params: {} 2024-04-16T19:33:06.371 DEBUG:boto:Token: None 2024-04-16T19:33:06.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/ 2024-04-16T19:33:06.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VFuV9ELP78puB4AcBMatOXNmRDk= 2024-04-16T19:33:06.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VFuV9ELP78puB4AcBMatOXNmRDk=', 'Content-Length': '0'} 2024-04-16T19:33:06.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000106bea4cf81773c0-00661ed272-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.373 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:06.373 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:06.373 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.373 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.373 DEBUG:boto:Method: GET 2024-04-16T19:33:06.374 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-16T19:33:06.374 DEBUG:boto:Data: 2024-04-16T19:33:06.374 DEBUG:boto:Headers: {} 2024-04-16T19:33:06.374 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.374 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.374 DEBUG:boto:Params: {} 2024-04-16T19:33:06.374 DEBUG:boto:Token: None 2024-04-16T19:33:06.374 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:06 GMT /yrjefh-14/?versions 2024-04-16T19:33:06.374 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fJFVQiDloDXzNqClMzlTlRqos+o= 2024-04-16T19:33:06.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fJFVQiDloDXzNqClMzlTlRqos+o=', 'Content-Length': '0'} 2024-04-16T19:33:06.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002286beceee9779e6-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.377 DEBUG:boto:b'yrjefh-14obj-test-zone2999falseobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:06.377 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.377 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:06.377 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?acl 2024-04-16T19:33:06.377 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?acl 2024-04-16T19:33:06.378 DEBUG:boto:Method: PUT 2024-04-16T19:33:06.378 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?acl 2024-04-16T19:33:06.378 DEBUG:boto:Data: 2024-04-16T19:33:06.378 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:33:06.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:06.378 DEBUG:boto:Port: 8000 2024-04-16T19:33:06.378 DEBUG:boto:Params: {} 2024-04-16T19:33:06.378 DEBUG:boto:Token: None 2024-04-16T19:33:06.378 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:06 GMT x-amz-acl:public-read /yrjefh-14/obj-test-zone2?acl 2024-04-16T19:33:06.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:05p03upI8k6bdiohKp8NaaR+cj4= 2024-04-16T19:33:06.379 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:05p03upI8k6bdiohKp8NaaR+cj4=', 'Content-Length': '0'} 2024-04-16T19:33:06.385 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001101acc345e44e1b-00661ed272-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:06.386 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-13 2024-04-16T19:33:06.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:06.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:06.386 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:06.447 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:06.447 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:06.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:06.503+0000 7f40b803d800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:06.505 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000010.32.6, , 00000000002.16.11, , 00000000003.12.10, , , , , ] 2024-04-16T19:33:06.505 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000011.35.6, , 00000000002.16.11, , 00000000003.12.10, , , , , ] 2024-04-16T19:33:11.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:11.505+0000 7f40b803d800 1 bucket sync caught up with source: 2024-04-16T19:33:11.506 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000011.35.6, , 00000000002.16.11, , 00000000003.12.10, , , , , ] 2024-04-16T19:33:11.507 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000011.35.6, , 00000000002.16.11, , 00000000003.12.10, , , , , ] 2024-04-16T19:33:11.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:11.505+0000 7f40b803d800 0 bucket checkpoint complete 2024-04-16T19:33:11.530 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-13 2024-04-16T19:33:11.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:11.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-13', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:11.530 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-13 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:11.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:11.654+0000 7f5cec938800 1 bucket sync caught up with source: 2024-04-16T19:33:11.656 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.17.11, , 00000000011.41.6, , 00000000002.20.11, , , , , ] 2024-04-16T19:33:11.656 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.17.11, , 00000000011.41.6, , 00000000002.20.11, , , , , ] 2024-04-16T19:33:11.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:11.654+0000 7f5cec938800 0 bucket checkpoint complete 2024-04-16T19:33:11.677 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-13 zones={test-zone2, test-zone1} 2024-04-16T19:33:11.677 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.678 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.678 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.678 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:33:11.678 DEBUG:boto:Data: 2024-04-16T19:33:11.678 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.678 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.678 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.678 DEBUG:boto:Params: {} 2024-04-16T19:33:11.678 DEBUG:boto:Token: None 2024-04-16T19:33:11.679 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8TwUjb0EnySB0/tXhI8iuT9BT2g= 2024-04-16T19:33:11.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8TwUjb0EnySB0/tXhI8iuT9BT2g=', 'Content-Length': '0'} 2024-04-16T19:33:11.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e07866367f2134d-00661ed277-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.681 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.681 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.682 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.682 DEBUG:boto:Path: /yrjefh-13/ 2024-04-16T19:33:11.682 DEBUG:boto:Data: 2024-04-16T19:33:11.682 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.682 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.682 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.682 DEBUG:boto:Params: {} 2024-04-16T19:33:11.682 DEBUG:boto:Token: None 2024-04-16T19:33:11.682 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8TwUjb0EnySB0/tXhI8iuT9BT2g= 2024-04-16T19:33:11.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8TwUjb0EnySB0/tXhI8iuT9BT2g=', 'Content-Length': '0'} 2024-04-16T19:33:11.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c314f8ab1f52bb05-00661ed277-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:33:11.685 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.686 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.686 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.686 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.686 DEBUG:boto:Method: GET 2024-04-16T19:33:11.686 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.686 DEBUG:boto:Data: 2024-04-16T19:33:11.686 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.686 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.686 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.686 DEBUG:boto:Params: {} 2024-04-16T19:33:11.687 DEBUG:boto:Token: None 2024-04-16T19:33:11.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/?versions 2024-04-16T19:33:11.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU= 2024-04-16T19:33:11.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU=', 'Content-Length': '0'} 2024-04-16T19:33:11.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abef4561a3adb03b-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.689 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.690 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:11.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:11.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:11.690 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:33:11.690 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.690 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.690 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.691 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.691 DEBUG:boto:Method: GET 2024-04-16T19:33:11.691 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.691 DEBUG:boto:Data: 2024-04-16T19:33:11.691 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.691 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.691 DEBUG:boto:Params: {} 2024-04-16T19:33:11.691 DEBUG:boto:Token: None 2024-04-16T19:33:11.691 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/?versions 2024-04-16T19:33:11.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU= 2024-04-16T19:33:11.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU=', 'Content-Length': '0'} 2024-04-16T19:33:11.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ba1b5b85cd05fd-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.694 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.694 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:11.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:11.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:11.695 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.695 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.695 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.695 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.695 DEBUG:boto:Method: GET 2024-04-16T19:33:11.695 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.695 DEBUG:boto:Data: 2024-04-16T19:33:11.695 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.695 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.695 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.695 DEBUG:boto:Params: {} 2024-04-16T19:33:11.696 DEBUG:boto:Token: None 2024-04-16T19:33:11.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/?versions 2024-04-16T19:33:11.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU= 2024-04-16T19:33:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU=', 'Content-Length': '0'} 2024-04-16T19:33:11.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b964ee213d0ac58a-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.698 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.699 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.699 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.699 DEBUG:boto:path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.699 DEBUG:boto:auth_path=/yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.699 DEBUG:boto:Method: GET 2024-04-16T19:33:11.699 DEBUG:boto:Path: /yrjefh-13/?versions&max-keys=999 2024-04-16T19:33:11.699 DEBUG:boto:Data: 2024-04-16T19:33:11.700 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.700 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.700 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.700 DEBUG:boto:Params: {} 2024-04-16T19:33:11.700 DEBUG:boto:Token: None 2024-04-16T19:33:11.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/?versions 2024-04-16T19:33:11.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU= 2024-04-16T19:33:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/A/4Li9gygcvIz6Uq+PAfNlPEU=', 'Content-Length': '0'} 2024-04-16T19:33:11.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040939c7533bc61a7-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.703 DEBUG:boto:b'yrjefh-13999falsedummyQVjkXVmTmztsnjfNUJx5p8B3Miv-2Vytrue2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1.vlzI6-on87pK3c1H7RH5J53v03EqDhtrue2024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2Jht8HWUgi1-j2o49sjX9hYHoGIC.iQdtrue2024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:11.703 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.703 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.704 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.704 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.704 DEBUG:boto:Method: GET 2024-04-16T19:33:11.704 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.704 DEBUG:boto:Data: 2024-04-16T19:33:11.704 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.704 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.704 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.704 DEBUG:boto:Params: {} 2024-04-16T19:33:11.704 DEBUG:boto:Token: None 2024-04-16T19:33:11.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027887354c0763fc5-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.707 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.707 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.707 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.707 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.707 DEBUG:boto:Method: GET 2024-04-16T19:33:11.708 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.708 DEBUG:boto:Data: 2024-04-16T19:33:11.708 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.708 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.708 DEBUG:boto:Params: {} 2024-04-16T19:33:11.708 DEBUG:boto:Token: None 2024-04-16T19:33:11.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011cd7529c858b00d-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.711 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.711 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.711 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.711 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.711 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.711 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.711 DEBUG:boto:Data: 2024-04-16T19:33:11.711 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.712 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.712 DEBUG:boto:Params: {} 2024-04-16T19:33:11.712 DEBUG:boto:Token: None 2024-04-16T19:33:11.712 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3jf/NP91hCGs54Ral+DzMU7fKM= 2024-04-16T19:33:11.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3jf/NP91hCGs54Ral+DzMU7fKM=', 'Content-Length': '0'} 2024-04-16T19:33:11.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d7711d6328db471-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.714 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.714 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.714 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.714 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.715 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.715 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.715 DEBUG:boto:Data: 2024-04-16T19:33:11.715 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.715 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.715 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.715 DEBUG:boto:Params: {} 2024-04-16T19:33:11.715 DEBUG:boto:Token: None 2024-04-16T19:33:11.715 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3jf/NP91hCGs54Ral+DzMU7fKM= 2024-04-16T19:33:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3jf/NP91hCGs54Ral+DzMU7fKM=', 'Content-Length': '0'} 2024-04-16T19:33:11.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adee6ba8c80517cd-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:11.718 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.718 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.718 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.718 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.718 DEBUG:boto:Method: GET 2024-04-16T19:33:11.718 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.718 DEBUG:boto:Data: 2024-04-16T19:33:11.719 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.719 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.719 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.719 DEBUG:boto:Params: {} 2024-04-16T19:33:11.719 DEBUG:boto:Token: None 2024-04-16T19:33:11.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082deb0fc48f26e9f-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.722 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.722 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.722 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.722 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.722 DEBUG:boto:Method: GET 2024-04-16T19:33:11.722 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.722 DEBUG:boto:Data: 2024-04-16T19:33:11.722 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.722 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.722 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.723 DEBUG:boto:Params: {} 2024-04-16T19:33:11.723 DEBUG:boto:Token: None 2024-04-16T19:33:11.723 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aee60ed8c81c1651-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.725 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.725 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.725 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.725 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.726 DEBUG:boto:Method: GET 2024-04-16T19:33:11.726 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.726 DEBUG:boto:Data: 2024-04-16T19:33:11.726 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.726 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.726 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.726 DEBUG:boto:Params: {} 2024-04-16T19:33:11.726 DEBUG:boto:Token: None 2024-04-16T19:33:11.726 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.726 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007657e4e7af8ca4b2-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.770 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.771 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.771 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.771 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.771 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.771 DEBUG:boto:Method: GET 2024-04-16T19:33:11.771 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.771 DEBUG:boto:Data: 2024-04-16T19:33:11.771 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.771 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.772 DEBUG:boto:Params: {} 2024-04-16T19:33:11.772 DEBUG:boto:Token: None 2024-04-16T19:33:11.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083952bc7de8810ad-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.774 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.775 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.775 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.775 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.775 DEBUG:boto:Path: /yrjefh-13/dummy 2024-04-16T19:33:11.775 DEBUG:boto:Data: 2024-04-16T19:33:11.775 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.775 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.775 DEBUG:boto:Params: {} 2024-04-16T19:33:11.776 DEBUG:boto:Token: None 2024-04-16T19:33:11.776 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy 2024-04-16T19:33:11.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F2taUCIru9/qEoOl3nu4wlj02RE= 2024-04-16T19:33:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F2taUCIru9/qEoOl3nu4wlj02RE=', 'Content-Length': '0'} 2024-04-16T19:33:11.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca755dc914ee5742-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.778 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.779 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.779 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.779 DEBUG:boto:Path: /yrjefh-13/dummy 2024-04-16T19:33:11.779 DEBUG:boto:Data: 2024-04-16T19:33:11.779 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.779 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.779 DEBUG:boto:Params: {} 2024-04-16T19:33:11.779 DEBUG:boto:Token: None 2024-04-16T19:33:11.779 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy 2024-04-16T19:33:11.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F2taUCIru9/qEoOl3nu4wlj02RE= 2024-04-16T19:33:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F2taUCIru9/qEoOl3nu4wlj02RE=', 'Content-Length': '0'} 2024-04-16T19:33:11.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055fe5d128528bda6-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:11.782 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.783 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.783 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.783 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.783 DEBUG:boto:Method: GET 2024-04-16T19:33:11.783 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.783 DEBUG:boto:Data: 2024-04-16T19:33:11.783 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.783 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.783 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.783 DEBUG:boto:Params: {} 2024-04-16T19:33:11.783 DEBUG:boto:Token: None 2024-04-16T19:33:11.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:54 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009131e819023442b3-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.786 DEBUG:boto:path=/yrjefh-13/dummy 2024-04-16T19:33:11.786 DEBUG:boto:auth_path=/yrjefh-13/dummy 2024-04-16T19:33:11.786 DEBUG:boto:path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.786 DEBUG:boto:auth_path=/yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.787 DEBUG:boto:Method: GET 2024-04-16T19:33:11.787 DEBUG:boto:Path: /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.787 DEBUG:boto:Data: 2024-04-16T19:33:11.787 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.787 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.787 DEBUG:boto:Params: {} 2024-04-16T19:33:11.787 DEBUG:boto:Token: None 2024-04-16T19:33:11.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/dummy?versionId=QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy 2024-04-16T19:33:11.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA= 2024-04-16T19:33:11.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J76uJ2zzVQsfNfwEXm50kugvgA=', 'Content-Length': '0'} 2024-04-16T19:33:11.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:53 GMT'), ('x-amz-version-id', 'QVjkXVmTmztsnjfNUJx5p8B3Miv-2Vy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dfa456b93de57402-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.789 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.790 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.790 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.790 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.790 DEBUG:boto:Method: GET 2024-04-16T19:33:11.790 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.790 DEBUG:boto:Data: 2024-04-16T19:33:11.790 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.790 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.790 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.790 DEBUG:boto:Params: {} 2024-04-16T19:33:11.791 DEBUG:boto:Token: None 2024-04-16T19:33:11.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec71ce8895b60a3-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.793 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.794 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.794 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.794 DEBUG:boto:path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.794 DEBUG:boto:auth_path=/yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.794 DEBUG:boto:Method: GET 2024-04-16T19:33:11.794 DEBUG:boto:Path: /yrjefh-13/?prefix=dummy 2024-04-16T19:33:11.794 DEBUG:boto:Data: 2024-04-16T19:33:11.794 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.794 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.794 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.794 DEBUG:boto:Params: {} 2024-04-16T19:33:11.795 DEBUG:boto:Token: None 2024-04-16T19:33:11.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcc881932c7d5d1-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.797 DEBUG:boto:b'yrjefh-13dummy1000falsedummy2024-04-16T19:31:53.857Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:11.798 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.798 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.798 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.798 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.798 DEBUG:boto:Method: GET 2024-04-16T19:33:11.798 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.798 DEBUG:boto:Data: 2024-04-16T19:33:11.798 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.798 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.798 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.799 DEBUG:boto:Params: {} 2024-04-16T19:33:11.799 DEBUG:boto:Token: None 2024-04-16T19:33:11.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df6ee14fd35c86cc-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.842 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.842 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.843 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.843 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.843 DEBUG:boto:Method: GET 2024-04-16T19:33:11.843 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.843 DEBUG:boto:Data: 2024-04-16T19:33:11.843 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.843 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.843 DEBUG:boto:Params: {} 2024-04-16T19:33:11.843 DEBUG:boto:Token: None 2024-04-16T19:33:11.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073e83986d6927528-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.846 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.846 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.846 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.846 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.847 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.847 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.847 DEBUG:boto:Data: 2024-04-16T19:33:11.847 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.847 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.847 DEBUG:boto:Params: {} 2024-04-16T19:33:11.847 DEBUG:boto:Token: None 2024-04-16T19:33:11.847 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F2Z+Ih0I8SU7txYUNe2XV1Cf0YE= 2024-04-16T19:33:11.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F2Z+Ih0I8SU7txYUNe2XV1Cf0YE=', 'Content-Length': '0'} 2024-04-16T19:33:11.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028b6ef33e1496e17-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.850 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.850 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.850 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.851 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.851 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.851 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.851 DEBUG:boto:Data: 2024-04-16T19:33:11.851 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.851 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.851 DEBUG:boto:Params: {} 2024-04-16T19:33:11.851 DEBUG:boto:Token: None 2024-04-16T19:33:11.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F2Z+Ih0I8SU7txYUNe2XV1Cf0YE= 2024-04-16T19:33:11.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F2Z+Ih0I8SU7txYUNe2XV1Cf0YE=', 'Content-Length': '0'} 2024-04-16T19:33:11.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009507b13103e7af92-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:11.854 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.854 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.854 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.854 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.854 DEBUG:boto:Method: GET 2024-04-16T19:33:11.855 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.855 DEBUG:boto:Data: 2024-04-16T19:33:11.855 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.855 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.855 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.855 DEBUG:boto:Params: {} 2024-04-16T19:33:11.855 DEBUG:boto:Token: None 2024-04-16T19:33:11.855 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003ece7acdd088309-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.898 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.899 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.899 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.899 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.899 DEBUG:boto:Method: GET 2024-04-16T19:33:11.899 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.899 DEBUG:boto:Data: 2024-04-16T19:33:11.899 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.899 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.899 DEBUG:boto:Params: {} 2024-04-16T19:33:11.900 DEBUG:boto:Token: None 2024-04-16T19:33:11.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ef0b5ddfeaba4b-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.902 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.902 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.903 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.903 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.903 DEBUG:boto:Method: GET 2024-04-16T19:33:11.903 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.903 DEBUG:boto:Data: 2024-04-16T19:33:11.903 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.903 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.903 DEBUG:boto:Params: {} 2024-04-16T19:33:11.903 DEBUG:boto:Token: None 2024-04-16T19:33:11.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b3bfc937198d77f-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.906 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.906 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.907 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.907 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.907 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.907 DEBUG:boto:Method: GET 2024-04-16T19:33:11.907 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.907 DEBUG:boto:Data: 2024-04-16T19:33:11.907 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.907 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.907 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.907 DEBUG:boto:Params: {} 2024-04-16T19:33:11.907 DEBUG:boto:Token: None 2024-04-16T19:33:11.908 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a676720aba9b0e78-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.910 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.910 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.910 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.911 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.911 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.911 DEBUG:boto:Data: 2024-04-16T19:33:11.911 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.911 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.911 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.911 DEBUG:boto:Params: {} 2024-04-16T19:33:11.911 DEBUG:boto:Token: None 2024-04-16T19:33:11.911 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msLCXhA6NObJjKksX9vDY++czWE= 2024-04-16T19:33:11.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msLCXhA6NObJjKksX9vDY++czWE=', 'Content-Length': '0'} 2024-04-16T19:33:11.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000150cd6c8279ec50d-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.914 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.915 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.915 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.915 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.915 DEBUG:boto:Data: 2024-04-16T19:33:11.915 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.915 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.915 DEBUG:boto:Params: {} 2024-04-16T19:33:11.915 DEBUG:boto:Token: None 2024-04-16T19:33:11.915 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msLCXhA6NObJjKksX9vDY++czWE= 2024-04-16T19:33:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msLCXhA6NObJjKksX9vDY++czWE=', 'Content-Length': '0'} 2024-04-16T19:33:11.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008532474ebb081c5b-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:11.918 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.918 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.918 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.919 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.919 DEBUG:boto:Method: GET 2024-04-16T19:33:11.919 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.919 DEBUG:boto:Data: 2024-04-16T19:33:11.919 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.919 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.919 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.919 DEBUG:boto:Params: {} 2024-04-16T19:33:11.919 DEBUG:boto:Token: None 2024-04-16T19:33:11.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0651c489ab5f82b-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.962 DEBUG:boto:path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.962 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1 2024-04-16T19:33:11.962 DEBUG:boto:path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.963 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.963 DEBUG:boto:Method: GET 2024-04-16T19:33:11.963 DEBUG:boto:Path: /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.963 DEBUG:boto:Data: 2024-04-16T19:33:11.963 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.963 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.963 DEBUG:boto:Params: {} 2024-04-16T19:33:11.963 DEBUG:boto:Token: None 2024-04-16T19:33:11.963 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone1?versionId=.vlzI6-on87pK3c1H7RH5J53v03EqDh 2024-04-16T19:33:11.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs= 2024-04-16T19:33:11.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7I8BdzTeCvBuEqK1PR/Dww9Hdqs=', 'Content-Length': '0'} 2024-04-16T19:33:11.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', '.vlzI6-on87pK3c1H7RH5J53v03EqDh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a747b0cc0a3437be-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.966 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.966 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.966 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.966 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.966 DEBUG:boto:Method: GET 2024-04-16T19:33:11.967 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.967 DEBUG:boto:Data: 2024-04-16T19:33:11.967 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.967 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.967 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.967 DEBUG:boto:Params: {} 2024-04-16T19:33:11.967 DEBUG:boto:Token: None 2024-04-16T19:33:11.967 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d05e4cb87c9fd65-00661ed277-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.970 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.970 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:11.971 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:11.971 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.971 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.971 DEBUG:boto:Method: GET 2024-04-16T19:33:11.971 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone1 2024-04-16T19:33:11.971 DEBUG:boto:Data: 2024-04-16T19:33:11.971 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.971 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.971 DEBUG:boto:Params: {} 2024-04-16T19:33:11.972 DEBUG:boto:Token: None 2024-04-16T19:33:11.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/ 2024-04-16T19:33:11.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg= 2024-04-16T19:33:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5uZAkLIcQWNAHArKvyNcX1NYmvg=', 'Content-Length': '0'} 2024-04-16T19:33:11.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356176ce4df8bd3e-00661ed277-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.974 DEBUG:boto:b'yrjefh-13obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:11.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:11.975 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.975 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.975 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.975 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.975 DEBUG:boto:Method: GET 2024-04-16T19:33:11.975 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.975 DEBUG:boto:Data: 2024-04-16T19:33:11.976 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.976 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.976 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.976 DEBUG:boto:Params: {} 2024-04-16T19:33:11.976 DEBUG:boto:Token: None 2024-04-16T19:33:11.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8= 2024-04-16T19:33:11.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8=', 'Content-Length': '0'} 2024-04-16T19:33:11.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000974cab8077eebb61-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.979 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.980 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.980 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.980 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.980 DEBUG:boto:Method: GET 2024-04-16T19:33:11.980 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.980 DEBUG:boto:Data: 2024-04-16T19:33:11.980 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.980 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.980 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.981 DEBUG:boto:Params: {} 2024-04-16T19:33:11.981 DEBUG:boto:Token: None 2024-04-16T19:33:11.981 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8= 2024-04-16T19:33:11.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8=', 'Content-Length': '0'} 2024-04-16T19:33:11.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044151fcb7d619fc1-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.983 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.983 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.983 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.984 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.984 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.984 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.984 DEBUG:boto:Data: 2024-04-16T19:33:11.984 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.984 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.984 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.984 DEBUG:boto:Params: {} 2024-04-16T19:33:11.984 DEBUG:boto:Token: None 2024-04-16T19:33:11.984 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bQzIF36B7tK9q1ZDlCnLRPL8Bs0= 2024-04-16T19:33:11.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bQzIF36B7tK9q1ZDlCnLRPL8Bs0=', 'Content-Length': '0'} 2024-04-16T19:33:11.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1ec0b701819583f-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.987 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.987 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.987 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.987 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.987 DEBUG:boto:Method: HEAD 2024-04-16T19:33:11.987 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.987 DEBUG:boto:Data: 2024-04-16T19:33:11.987 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.987 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:11.987 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.988 DEBUG:boto:Params: {} 2024-04-16T19:33:11.988 DEBUG:boto:Token: None 2024-04-16T19:33:11.988 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bQzIF36B7tK9q1ZDlCnLRPL8Bs0= 2024-04-16T19:33:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bQzIF36B7tK9q1ZDlCnLRPL8Bs0=', 'Content-Length': '0'} 2024-04-16T19:33:11.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2b5e626bd5d40c6-00661ed277-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:11.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:11.990 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.990 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:11.990 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.990 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.991 DEBUG:boto:Method: GET 2024-04-16T19:33:11.991 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.991 DEBUG:boto:Data: 2024-04-16T19:33:11.991 DEBUG:boto:Headers: {} 2024-04-16T19:33:11.991 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:11.991 DEBUG:boto:Port: 8000 2024-04-16T19:33:11.991 DEBUG:boto:Params: {} 2024-04-16T19:33:11.991 DEBUG:boto:Token: None 2024-04-16T19:33:11.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:11 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:11.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8= 2024-04-16T19:33:11.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bvAGPxlrTzIHT17X/lZDtsIz1X8=', 'Content-Length': '0'} 2024-04-16T19:33:11.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9353c25472e6519-00661ed277-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.034 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.034 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.035 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.035 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.035 DEBUG:boto:Method: GET 2024-04-16T19:33:12.035 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.035 DEBUG:boto:Data: 2024-04-16T19:33:12.035 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.035 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.035 DEBUG:boto:Params: {} 2024-04-16T19:33:12.035 DEBUG:boto:Token: None 2024-04-16T19:33:12.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0= 2024-04-16T19:33:12.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0=', 'Content-Length': '0'} 2024-04-16T19:33:12.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000807331b013be0b0-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.078 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:12.078 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:12.078 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.079 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.079 DEBUG:boto:Method: GET 2024-04-16T19:33:12.079 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.079 DEBUG:boto:Data: 2024-04-16T19:33:12.079 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.079 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.079 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.079 DEBUG:boto:Params: {} 2024-04-16T19:33:12.079 DEBUG:boto:Token: None 2024-04-16T19:33:12.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/ 2024-04-16T19:33:12.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0= 2024-04-16T19:33:12.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0=', 'Content-Length': '0'} 2024-04-16T19:33:12.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a6a7d1360f7c86-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.082 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.082 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:12.082 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:12.083 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.083 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.083 DEBUG:boto:Method: GET 2024-04-16T19:33:12.083 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.083 DEBUG:boto:Data: 2024-04-16T19:33:12.083 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.083 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.083 DEBUG:boto:Params: {} 2024-04-16T19:33:12.083 DEBUG:boto:Token: None 2024-04-16T19:33:12.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/ 2024-04-16T19:33:12.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0= 2024-04-16T19:33:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0=', 'Content-Length': '0'} 2024-04-16T19:33:12.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064526460a930148a-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.086 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.086 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.086 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.086 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.087 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.087 DEBUG:boto:Data: 2024-04-16T19:33:12.087 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.087 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.087 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.087 DEBUG:boto:Params: {} 2024-04-16T19:33:12.087 DEBUG:boto:Token: None 2024-04-16T19:33:12.087 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eF3BsgRjGBctyGvyjxO6BjBPigU= 2024-04-16T19:33:12.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eF3BsgRjGBctyGvyjxO6BjBPigU=', 'Content-Length': '0'} 2024-04-16T19:33:12.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000911b89db09840fa-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.090 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.090 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.090 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.090 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.090 DEBUG:boto:Data: 2024-04-16T19:33:12.091 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.091 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.091 DEBUG:boto:Params: {} 2024-04-16T19:33:12.091 DEBUG:boto:Token: None 2024-04-16T19:33:12.091 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eF3BsgRjGBctyGvyjxO6BjBPigU= 2024-04-16T19:33:12.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eF3BsgRjGBctyGvyjxO6BjBPigU=', 'Content-Length': '0'} 2024-04-16T19:33:12.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a15eaa4e168f129-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:12.094 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.094 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.094 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.094 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.094 DEBUG:boto:Method: GET 2024-04-16T19:33:12.094 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.094 DEBUG:boto:Data: 2024-04-16T19:33:12.095 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.095 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.095 DEBUG:boto:Params: {} 2024-04-16T19:33:12.095 DEBUG:boto:Token: None 2024-04-16T19:33:12.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0= 2024-04-16T19:33:12.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0=', 'Content-Length': '0'} 2024-04-16T19:33:12.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000179406dd904dae5c-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.098 DEBUG:boto:path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.098 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2 2024-04-16T19:33:12.098 DEBUG:boto:path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.098 DEBUG:boto:auth_path=/yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.098 DEBUG:boto:Method: GET 2024-04-16T19:33:12.098 DEBUG:boto:Path: /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.098 DEBUG:boto:Data: 2024-04-16T19:33:12.098 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.098 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.098 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.098 DEBUG:boto:Params: {} 2024-04-16T19:33:12.099 DEBUG:boto:Token: None 2024-04-16T19:33:12.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/obj-test-zone2?versionId=Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd 2024-04-16T19:33:12.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0= 2024-04-16T19:33:12.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4m7to6UR1SqpUH9ZO3luXW5cii0=', 'Content-Length': '0'} 2024-04-16T19:33:12.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'Jht8HWUgi1-j2o49sjX9hYHoGIC.iQd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c74418f9728b6ff-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.142 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:12.142 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:12.143 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.143 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.143 DEBUG:boto:Method: GET 2024-04-16T19:33:12.143 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.143 DEBUG:boto:Data: 2024-04-16T19:33:12.143 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.143 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.143 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.143 DEBUG:boto:Params: {} 2024-04-16T19:33:12.143 DEBUG:boto:Token: None 2024-04-16T19:33:12.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/ 2024-04-16T19:33:12.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0= 2024-04-16T19:33:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0=', 'Content-Length': '0'} 2024-04-16T19:33:12.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005121ee6f50d3548c-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.146 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.146 DEBUG:boto:path=/yrjefh-13/ 2024-04-16T19:33:12.146 DEBUG:boto:auth_path=/yrjefh-13/ 2024-04-16T19:33:12.147 DEBUG:boto:path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.147 DEBUG:boto:auth_path=/yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.147 DEBUG:boto:Method: GET 2024-04-16T19:33:12.147 DEBUG:boto:Path: /yrjefh-13/?prefix=obj-test-zone2 2024-04-16T19:33:12.147 DEBUG:boto:Data: 2024-04-16T19:33:12.147 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.147 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.147 DEBUG:boto:Params: {} 2024-04-16T19:33:12.147 DEBUG:boto:Token: None 2024-04-16T19:33:12.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-13/ 2024-04-16T19:33:12.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0= 2024-04-16T19:33:12.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lym2x0ryRiPJzaadFpUBeU4OX0=', 'Content-Length': '0'} 2024-04-16T19:33:12.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af7b9dd37b819ad3-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.150 DEBUG:boto:b'yrjefh-13obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-13 zones={test-zone2, test-zone1} 2024-04-16T19:33:12.150 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-14 2024-04-16T19:33:12.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:12.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:12.151 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-14 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:12.213 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:12.213 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:12.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:12.271+0000 7f7fc4ba2800 1 bucket sync caught up with source: 2024-04-16T19:33:12.273 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000011.40.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:12.273 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000011.40.6, , 00000000002.16.11, , 00000000002.24.11, , , , , ] 2024-04-16T19:33:12.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:12.271+0000 7f7fc4ba2800 0 bucket checkpoint complete 2024-04-16T19:33:12.292 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-14 2024-04-16T19:33:12.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:12.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:12.293 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:12.352 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:12.352 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:12.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:12.408+0000 7fdaca038800 1 bucket sync caught up with source: 2024-04-16T19:33:12.410 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.24.11, , 00000000011.43.6, , 00000000003.43.10, , , , , ] 2024-04-16T19:33:12.410 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.24.11, , 00000000011.43.6, , 00000000003.43.10, , , , , ] 2024-04-16T19:33:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:12.408+0000 7fdaca038800 0 bucket checkpoint complete 2024-04-16T19:33:12.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-14 zones={test-zone2, test-zone1} 2024-04-16T19:33:12.430 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.430 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.430 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.430 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:12.430 DEBUG:boto:Data: 2024-04-16T19:33:12.430 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.430 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.430 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.431 DEBUG:boto:Params: {} 2024-04-16T19:33:12.431 DEBUG:boto:Token: None 2024-04-16T19:33:12.431 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nPxsy7GZsaleMZLSQG5rRtfFwtY= 2024-04-16T19:33:12.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPxsy7GZsaleMZLSQG5rRtfFwtY=', 'Content-Length': '0'} 2024-04-16T19:33:12.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0c78843346b757a-00661ed278-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.433 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.434 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.434 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.434 DEBUG:boto:Path: /yrjefh-14/ 2024-04-16T19:33:12.434 DEBUG:boto:Data: 2024-04-16T19:33:12.434 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.434 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.434 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.434 DEBUG:boto:Params: {} 2024-04-16T19:33:12.434 DEBUG:boto:Token: None 2024-04-16T19:33:12.434 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nPxsy7GZsaleMZLSQG5rRtfFwtY= 2024-04-16T19:33:12.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPxsy7GZsaleMZLSQG5rRtfFwtY=', 'Content-Length': '0'} 2024-04-16T19:33:12.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b433b8548e6efe4-00661ed278-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:33:12.437 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.437 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.437 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.437 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.437 DEBUG:boto:Method: GET 2024-04-16T19:33:12.438 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.438 DEBUG:boto:Data: 2024-04-16T19:33:12.438 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.438 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.438 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.438 DEBUG:boto:Params: {} 2024-04-16T19:33:12.438 DEBUG:boto:Token: None 2024-04-16T19:33:12.438 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/?versions 2024-04-16T19:33:12.438 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg= 2024-04-16T19:33:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg=', 'Content-Length': '0'} 2024-04-16T19:33:12.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db1e993d7f463586-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.441 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.441 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:12.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:12.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:12.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:33:12.442 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.442 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.442 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.442 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.442 DEBUG:boto:Method: GET 2024-04-16T19:33:12.442 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.442 DEBUG:boto:Data: 2024-04-16T19:33:12.442 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.443 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.443 DEBUG:boto:Params: {} 2024-04-16T19:33:12.443 DEBUG:boto:Token: None 2024-04-16T19:33:12.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/?versions 2024-04-16T19:33:12.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg= 2024-04-16T19:33:12.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg=', 'Content-Length': '0'} 2024-04-16T19:33:12.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c95214453237355-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.445 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.446 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:12.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-16T19:33:12.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-16T19:33:12.446 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.446 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.446 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.446 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.446 DEBUG:boto:Method: GET 2024-04-16T19:33:12.447 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.447 DEBUG:boto:Data: 2024-04-16T19:33:12.447 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.447 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.447 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.447 DEBUG:boto:Params: {} 2024-04-16T19:33:12.447 DEBUG:boto:Token: None 2024-04-16T19:33:12.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/?versions 2024-04-16T19:33:12.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg= 2024-04-16T19:33:12.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg=', 'Content-Length': '0'} 2024-04-16T19:33:12.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89e2ed32fb342bb-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.450 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.450 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.450 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.451 DEBUG:boto:path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.451 DEBUG:boto:auth_path=/yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.451 DEBUG:boto:Method: GET 2024-04-16T19:33:12.451 DEBUG:boto:Path: /yrjefh-14/?versions&max-keys=999 2024-04-16T19:33:12.451 DEBUG:boto:Data: 2024-04-16T19:33:12.451 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.451 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.451 DEBUG:boto:Params: {} 2024-04-16T19:33:12.451 DEBUG:boto:Token: None 2024-04-16T19:33:12.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/?versions 2024-04-16T19:33:12.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg= 2024-04-16T19:33:12.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GuVqZxdQh9FvU16/FhMTYZq+8Yg=', 'Content-Length': '0'} 2024-04-16T19:33:12.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075710ae64b42dad6-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.454 DEBUG:boto:b'yrjefh-14999falsedummyUVGKycmzB5nu8t70EdJ2ZTtR46HxNAhtrue2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobj-test-zone1fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5xtrue2024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj-test-zone2CtLdn-jZwuC3ShuunWtb1zm1roPFMdhtrue2024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:12.455 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.455 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.455 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.455 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.455 DEBUG:boto:Method: GET 2024-04-16T19:33:12.455 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.455 DEBUG:boto:Data: 2024-04-16T19:33:12.455 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.455 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.455 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.455 DEBUG:boto:Params: {} 2024-04-16T19:33:12.455 DEBUG:boto:Token: None 2024-04-16T19:33:12.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000540e8ac3bd38cc1-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.458 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.458 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.458 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.458 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.458 DEBUG:boto:Method: GET 2024-04-16T19:33:12.458 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.458 DEBUG:boto:Data: 2024-04-16T19:33:12.458 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.458 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.458 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.458 DEBUG:boto:Params: {} 2024-04-16T19:33:12.459 DEBUG:boto:Token: None 2024-04-16T19:33:12.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017edb64e8f013862-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.461 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.461 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.461 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.461 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.461 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.461 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.461 DEBUG:boto:Data: 2024-04-16T19:33:12.461 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.462 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.462 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.462 DEBUG:boto:Params: {} 2024-04-16T19:33:12.462 DEBUG:boto:Token: None 2024-04-16T19:33:12.462 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.462 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Qoy/KXXLzxpRv5obwoDbw5LBF4= 2024-04-16T19:33:12.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Qoy/KXXLzxpRv5obwoDbw5LBF4=', 'Content-Length': '0'} 2024-04-16T19:33:12.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045039c6d4c184be4-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.464 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.464 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.464 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.464 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.464 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.464 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.464 DEBUG:boto:Data: 2024-04-16T19:33:12.464 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.465 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.465 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.465 DEBUG:boto:Params: {} 2024-04-16T19:33:12.465 DEBUG:boto:Token: None 2024-04-16T19:33:12.465 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1Qoy/KXXLzxpRv5obwoDbw5LBF4= 2024-04-16T19:33:12.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1Qoy/KXXLzxpRv5obwoDbw5LBF4=', 'Content-Length': '0'} 2024-04-16T19:33:12.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6802be07783ec7e-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:12.467 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.467 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.467 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.467 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.467 DEBUG:boto:Method: GET 2024-04-16T19:33:12.467 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.468 DEBUG:boto:Data: 2024-04-16T19:33:12.468 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.468 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.468 DEBUG:boto:Params: {} 2024-04-16T19:33:12.468 DEBUG:boto:Token: None 2024-04-16T19:33:12.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081e5e6b8b1876b9d-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.470 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.470 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.470 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.470 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.470 DEBUG:boto:Method: GET 2024-04-16T19:33:12.470 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.471 DEBUG:boto:Data: 2024-04-16T19:33:12.471 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.471 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.471 DEBUG:boto:Params: {} 2024-04-16T19:33:12.471 DEBUG:boto:Token: None 2024-04-16T19:33:12.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b2ab395d43e5aef-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.473 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.474 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.474 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.474 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.474 DEBUG:boto:Method: GET 2024-04-16T19:33:12.474 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.474 DEBUG:boto:Data: 2024-04-16T19:33:12.474 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.474 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.474 DEBUG:boto:Params: {} 2024-04-16T19:33:12.474 DEBUG:boto:Token: None 2024-04-16T19:33:12.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d3e7236a8bc700-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.477 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.477 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.477 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.478 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.478 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.478 DEBUG:boto:Method: GET 2024-04-16T19:33:12.478 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.478 DEBUG:boto:Data: 2024-04-16T19:33:12.478 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.478 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.478 DEBUG:boto:Params: {} 2024-04-16T19:33:12.478 DEBUG:boto:Token: None 2024-04-16T19:33:12.478 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d09a86135f6ab3f-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.522 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.522 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.523 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.523 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.523 DEBUG:boto:Path: /yrjefh-14/dummy 2024-04-16T19:33:12.523 DEBUG:boto:Data: 2024-04-16T19:33:12.523 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.523 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.523 DEBUG:boto:Params: {} 2024-04-16T19:33:12.523 DEBUG:boto:Token: None 2024-04-16T19:33:12.523 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy 2024-04-16T19:33:12.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0J7BfrUi3Qoa2bI5NpSj18Lkdh0= 2024-04-16T19:33:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0J7BfrUi3Qoa2bI5NpSj18Lkdh0=', 'Content-Length': '0'} 2024-04-16T19:33:12.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b42862823e180735-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.526 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.526 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.526 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.526 DEBUG:boto:Path: /yrjefh-14/dummy 2024-04-16T19:33:12.526 DEBUG:boto:Data: 2024-04-16T19:33:12.526 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.526 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.527 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.527 DEBUG:boto:Params: {} 2024-04-16T19:33:12.527 DEBUG:boto:Token: None 2024-04-16T19:33:12.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy 2024-04-16T19:33:12.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0J7BfrUi3Qoa2bI5NpSj18Lkdh0= 2024-04-16T19:33:12.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0J7BfrUi3Qoa2bI5NpSj18Lkdh0=', 'Content-Length': '0'} 2024-04-16T19:33:12.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c1c949d2051a7a4-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:12.530 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.530 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.530 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.530 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.530 DEBUG:boto:Method: GET 2024-04-16T19:33:12.530 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.531 DEBUG:boto:Data: 2024-04-16T19:33:12.531 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.531 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.531 DEBUG:boto:Params: {} 2024-04-16T19:33:12.531 DEBUG:boto:Token: None 2024-04-16T19:33:12.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f693d51cecf0e59-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.533 DEBUG:boto:path=/yrjefh-14/dummy 2024-04-16T19:33:12.533 DEBUG:boto:auth_path=/yrjefh-14/dummy 2024-04-16T19:33:12.533 DEBUG:boto:path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.533 DEBUG:boto:auth_path=/yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.533 DEBUG:boto:Method: GET 2024-04-16T19:33:12.533 DEBUG:boto:Path: /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.534 DEBUG:boto:Data: 2024-04-16T19:33:12.534 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.534 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.534 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.534 DEBUG:boto:Params: {} 2024-04-16T19:33:12.534 DEBUG:boto:Token: None 2024-04-16T19:33:12.534 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/dummy?versionId=UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh 2024-04-16T19:33:12.534 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A= 2024-04-16T19:33:12.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NTSNGG2zt/TOifQSE+MsCgSsl3A=', 'Content-Length': '0'} 2024-04-16T19:33:12.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('x-amz-version-id', 'UVGKycmzB5nu8t70EdJ2ZTtR46HxNAh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:31:59 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bdf1b18595c0ee1-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.536 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.536 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.536 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.536 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.536 DEBUG:boto:Method: GET 2024-04-16T19:33:12.536 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.537 DEBUG:boto:Data: 2024-04-16T19:33:12.537 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.537 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.537 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.537 DEBUG:boto:Params: {} 2024-04-16T19:33:12.537 DEBUG:boto:Token: None 2024-04-16T19:33:12.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055273181abab3385-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.540 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.540 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.540 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.541 DEBUG:boto:path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.541 DEBUG:boto:auth_path=/yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.541 DEBUG:boto:Method: GET 2024-04-16T19:33:12.541 DEBUG:boto:Path: /yrjefh-14/?prefix=dummy 2024-04-16T19:33:12.541 DEBUG:boto:Data: 2024-04-16T19:33:12.541 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.541 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.541 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.541 DEBUG:boto:Params: {} 2024-04-16T19:33:12.541 DEBUG:boto:Token: None 2024-04-16T19:33:12.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1c12a991abca45-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.544 DEBUG:boto:b'yrjefh-14dummy1000falsedummy2024-04-16T19:31:59.019Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:12.544 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.544 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.544 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.544 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.545 DEBUG:boto:Method: GET 2024-04-16T19:33:12.545 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.545 DEBUG:boto:Data: 2024-04-16T19:33:12.545 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.545 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.545 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.545 DEBUG:boto:Params: {} 2024-04-16T19:33:12.545 DEBUG:boto:Token: None 2024-04-16T19:33:12.545 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd9f7353281e8474-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.547 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.547 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.547 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.548 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.548 DEBUG:boto:Method: GET 2024-04-16T19:33:12.548 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.548 DEBUG:boto:Data: 2024-04-16T19:33:12.548 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.548 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.548 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.548 DEBUG:boto:Params: {} 2024-04-16T19:33:12.548 DEBUG:boto:Token: None 2024-04-16T19:33:12.548 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090ab6e398af7a90e-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.550 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.551 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.551 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.551 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.551 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.551 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.551 DEBUG:boto:Data: 2024-04-16T19:33:12.551 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.551 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.551 DEBUG:boto:Params: {} 2024-04-16T19:33:12.551 DEBUG:boto:Token: None 2024-04-16T19:33:12.552 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4BBmtLCtxeEWTF+pCsI6mf4PyHE= 2024-04-16T19:33:12.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4BBmtLCtxeEWTF+pCsI6mf4PyHE=', 'Content-Length': '0'} 2024-04-16T19:33:12.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac860b6276c7719-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.554 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.554 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.554 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.554 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.554 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.554 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.554 DEBUG:boto:Data: 2024-04-16T19:33:12.554 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.554 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.555 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.555 DEBUG:boto:Params: {} 2024-04-16T19:33:12.555 DEBUG:boto:Token: None 2024-04-16T19:33:12.555 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4BBmtLCtxeEWTF+pCsI6mf4PyHE= 2024-04-16T19:33:12.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4BBmtLCtxeEWTF+pCsI6mf4PyHE=', 'Content-Length': '0'} 2024-04-16T19:33:12.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc97a1b69bd268b-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:12.557 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.557 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.557 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.557 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.557 DEBUG:boto:Method: GET 2024-04-16T19:33:12.557 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.557 DEBUG:boto:Data: 2024-04-16T19:33:12.557 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.557 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.557 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.558 DEBUG:boto:Params: {} 2024-04-16T19:33:12.558 DEBUG:boto:Token: None 2024-04-16T19:33:12.558 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.558 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1107b55fde461da-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.602 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.603 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.603 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.603 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.603 DEBUG:boto:Method: GET 2024-04-16T19:33:12.603 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.603 DEBUG:boto:Data: 2024-04-16T19:33:12.603 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.603 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.603 DEBUG:boto:Params: {} 2024-04-16T19:33:12.603 DEBUG:boto:Token: None 2024-04-16T19:33:12.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e1607f61872e845-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.606 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.606 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.606 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.606 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.606 DEBUG:boto:Method: GET 2024-04-16T19:33:12.606 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.606 DEBUG:boto:Data: 2024-04-16T19:33:12.606 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.607 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.607 DEBUG:boto:Params: {} 2024-04-16T19:33:12.607 DEBUG:boto:Token: None 2024-04-16T19:33:12.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30cc0c16855d358-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.650 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.651 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.651 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.651 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.651 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.651 DEBUG:boto:Method: GET 2024-04-16T19:33:12.651 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.651 DEBUG:boto:Data: 2024-04-16T19:33:12.651 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.651 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.651 DEBUG:boto:Params: {} 2024-04-16T19:33:12.652 DEBUG:boto:Token: None 2024-04-16T19:33:12.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c8155e68c587913-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.654 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.654 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.654 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.654 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.655 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.655 DEBUG:boto:Data: 2024-04-16T19:33:12.655 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.655 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.655 DEBUG:boto:Params: {} 2024-04-16T19:33:12.655 DEBUG:boto:Token: None 2024-04-16T19:33:12.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeaAAJWcWGxSM4U5rbW0y9/4FLw= 2024-04-16T19:33:12.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeaAAJWcWGxSM4U5rbW0y9/4FLw=', 'Content-Length': '0'} 2024-04-16T19:33:12.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000883db683763e63cd-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.658 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.658 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.658 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.658 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.658 DEBUG:boto:Data: 2024-04-16T19:33:12.658 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.658 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.659 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.659 DEBUG:boto:Params: {} 2024-04-16T19:33:12.659 DEBUG:boto:Token: None 2024-04-16T19:33:12.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeaAAJWcWGxSM4U5rbW0y9/4FLw= 2024-04-16T19:33:12.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeaAAJWcWGxSM4U5rbW0y9/4FLw=', 'Content-Length': '0'} 2024-04-16T19:33:12.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000240cfaaf1008abdc-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-16T19:33:12.661 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.661 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.661 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.661 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.661 DEBUG:boto:Method: GET 2024-04-16T19:33:12.662 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.662 DEBUG:boto:Data: 2024-04-16T19:33:12.662 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.662 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.662 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.662 DEBUG:boto:Params: {} 2024-04-16T19:33:12.662 DEBUG:boto:Token: None 2024-04-16T19:33:12.662 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:05 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d1e73291fbcfd05-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.706 DEBUG:boto:path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.706 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1 2024-04-16T19:33:12.706 DEBUG:boto:path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.706 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.707 DEBUG:boto:Method: GET 2024-04-16T19:33:12.707 DEBUG:boto:Path: /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.707 DEBUG:boto:Data: 2024-04-16T19:33:12.707 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.707 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.707 DEBUG:boto:Params: {} 2024-04-16T19:33:12.707 DEBUG:boto:Token: None 2024-04-16T19:33:12.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone1?versionId=fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x 2024-04-16T19:33:12.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4= 2024-04-16T19:33:12.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfAOllcj2JWBE9U39HzRh7KfvO4=', 'Content-Length': '0'} 2024-04-16T19:33:12.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'fPNNC7sS1CM7zQ3lrP9jRLy7Lxi0c5x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d407c0f142141a4-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.710 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.710 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.710 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.710 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.710 DEBUG:boto:Method: GET 2024-04-16T19:33:12.710 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.710 DEBUG:boto:Data: 2024-04-16T19:33:12.710 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.710 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.710 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.711 DEBUG:boto:Params: {} 2024-04-16T19:33:12.711 DEBUG:boto:Token: None 2024-04-16T19:33:12.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008257d9e3a1e6b780-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.713 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.714 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.714 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.714 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.714 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.714 DEBUG:boto:Method: GET 2024-04-16T19:33:12.714 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone1 2024-04-16T19:33:12.714 DEBUG:boto:Data: 2024-04-16T19:33:12.714 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.714 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.714 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.714 DEBUG:boto:Params: {} 2024-04-16T19:33:12.715 DEBUG:boto:Token: None 2024-04-16T19:33:12.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000146e5b8260bd225-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.717 DEBUG:boto:b'yrjefh-14obj-test-zone11000falseobj-test-zone12024-04-16T19:32:04.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:12.717 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.717 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.717 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.717 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.718 DEBUG:boto:Method: GET 2024-04-16T19:33:12.718 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.718 DEBUG:boto:Data: 2024-04-16T19:33:12.718 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.718 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.718 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.718 DEBUG:boto:Params: {} 2024-04-16T19:33:12.718 DEBUG:boto:Token: None 2024-04-16T19:33:12.718 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030f59235d0ce9b20-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.720 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.720 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.720 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.721 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.721 DEBUG:boto:Method: GET 2024-04-16T19:33:12.721 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.721 DEBUG:boto:Data: 2024-04-16T19:33:12.721 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.721 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.721 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.721 DEBUG:boto:Params: {} 2024-04-16T19:33:12.721 DEBUG:boto:Token: None 2024-04-16T19:33:12.721 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086b1a26d55791035-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.766 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.766 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.766 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.767 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.767 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.767 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.767 DEBUG:boto:Data: 2024-04-16T19:33:12.767 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.767 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.767 DEBUG:boto:Params: {} 2024-04-16T19:33:12.767 DEBUG:boto:Token: None 2024-04-16T19:33:12.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qoIddwZgA4Ljp8ovoeSD1nixYYw= 2024-04-16T19:33:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qoIddwZgA4Ljp8ovoeSD1nixYYw=', 'Content-Length': '0'} 2024-04-16T19:33:12.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0180d7815dc04af-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.769 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.769 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.769 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.770 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.770 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.770 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.770 DEBUG:boto:Data: 2024-04-16T19:33:12.770 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.770 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.770 DEBUG:boto:Params: {} 2024-04-16T19:33:12.770 DEBUG:boto:Token: None 2024-04-16T19:33:12.770 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qoIddwZgA4Ljp8ovoeSD1nixYYw= 2024-04-16T19:33:12.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qoIddwZgA4Ljp8ovoeSD1nixYYw=', 'Content-Length': '0'} 2024-04-16T19:33:12.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008437be87be9ebcd4-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:12.773 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.773 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.773 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.773 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.773 DEBUG:boto:Method: GET 2024-04-16T19:33:12.773 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.773 DEBUG:boto:Data: 2024-04-16T19:33:12.773 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.773 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.773 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.773 DEBUG:boto:Params: {} 2024-04-16T19:33:12.774 DEBUG:boto:Token: None 2024-04-16T19:33:12.774 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055a679f4f288fe45-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.776 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.776 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.776 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.776 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.776 DEBUG:boto:Method: GET 2024-04-16T19:33:12.776 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.776 DEBUG:boto:Data: 2024-04-16T19:33:12.776 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.776 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.776 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.776 DEBUG:boto:Params: {} 2024-04-16T19:33:12.777 DEBUG:boto:Token: None 2024-04-16T19:33:12.777 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000561e4cc5e60ce22a-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.822 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.822 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.823 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.823 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.823 DEBUG:boto:Method: GET 2024-04-16T19:33:12.823 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.823 DEBUG:boto:Data: 2024-04-16T19:33:12.823 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.823 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.823 DEBUG:boto:Params: {} 2024-04-16T19:33:12.823 DEBUG:boto:Token: None 2024-04-16T19:33:12.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a0cb7752994fe7-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.826 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.826 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.827 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.827 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.827 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.827 DEBUG:boto:Method: GET 2024-04-16T19:33:12.827 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.827 DEBUG:boto:Data: 2024-04-16T19:33:12.827 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.827 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.827 DEBUG:boto:Params: {} 2024-04-16T19:33:12.827 DEBUG:boto:Token: None 2024-04-16T19:33:12.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030972d1e67ca5b74-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.830 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.830 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.831 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.831 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.831 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.831 DEBUG:boto:Data: 2024-04-16T19:33:12.831 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.831 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.831 DEBUG:boto:Params: {} 2024-04-16T19:33:12.831 DEBUG:boto:Token: None 2024-04-16T19:33:12.831 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SCq6o7Y2vUf+eARy40brw3DQIBw= 2024-04-16T19:33:12.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SCq6o7Y2vUf+eARy40brw3DQIBw=', 'Content-Length': '0'} 2024-04-16T19:33:12.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb5174793429597c-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.834 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.834 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.834 DEBUG:boto:Method: HEAD 2024-04-16T19:33:12.834 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.834 DEBUG:boto:Data: 2024-04-16T19:33:12.834 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.834 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.835 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.835 DEBUG:boto:Params: {} 2024-04-16T19:33:12.835 DEBUG:boto:Token: None 2024-04-16T19:33:12.835 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SCq6o7Y2vUf+eARy40brw3DQIBw= 2024-04-16T19:33:12.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SCq6o7Y2vUf+eARy40brw3DQIBw=', 'Content-Length': '0'} 2024-04-16T19:33:12.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f28d834f871d6e69-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-16T19:33:12.838 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.838 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.838 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.838 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.838 DEBUG:boto:Method: GET 2024-04-16T19:33:12.838 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.838 DEBUG:boto:Data: 2024-04-16T19:33:12.839 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.839 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.839 DEBUG:boto:Params: {} 2024-04-16T19:33:12.839 DEBUG:boto:Token: None 2024-04-16T19:33:12.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051ef4cf9a3914742-00661ed278-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.841 DEBUG:boto:path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.841 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2 2024-04-16T19:33:12.841 DEBUG:boto:path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.841 DEBUG:boto:auth_path=/yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.842 DEBUG:boto:Method: GET 2024-04-16T19:33:12.842 DEBUG:boto:Path: /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.842 DEBUG:boto:Data: 2024-04-16T19:33:12.842 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.842 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.842 DEBUG:boto:Params: {} 2024-04-16T19:33:12.842 DEBUG:boto:Token: None 2024-04-16T19:33:12.842 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/obj-test-zone2?versionId=CtLdn-jZwuC3ShuunWtb1zm1roPFMdh 2024-04-16T19:33:12.842 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0= 2024-04-16T19:33:12.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cU92by9wM4Dmf7TIOQlNj7SvRq0=', 'Content-Length': '0'} 2024-04-16T19:33:12.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:32:04 GMT'), ('x-amz-version-id', 'CtLdn-jZwuC3ShuunWtb1zm1roPFMdh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:32:08 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbf9584bae45dc18-00661ed278-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.886 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.886 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.887 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.887 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.887 DEBUG:boto:Method: GET 2024-04-16T19:33:12.887 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.887 DEBUG:boto:Data: 2024-04-16T19:33:12.887 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:12.887 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.887 DEBUG:boto:Params: {} 2024-04-16T19:33:12.887 DEBUG:boto:Token: None 2024-04-16T19:33:12.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5737dbd8cbd1896-00661ed278-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.890 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.891 DEBUG:boto:path=/yrjefh-14/ 2024-04-16T19:33:12.891 DEBUG:boto:auth_path=/yrjefh-14/ 2024-04-16T19:33:12.891 DEBUG:boto:path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.891 DEBUG:boto:auth_path=/yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.891 DEBUG:boto:Method: GET 2024-04-16T19:33:12.891 DEBUG:boto:Path: /yrjefh-14/?prefix=obj-test-zone2 2024-04-16T19:33:12.891 DEBUG:boto:Data: 2024-04-16T19:33:12.891 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.891 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.891 DEBUG:boto:Params: {} 2024-04-16T19:33:12.892 DEBUG:boto:Token: None 2024-04-16T19:33:12.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-14/ 2024-04-16T19:33:12.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ= 2024-04-16T19:33:12.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uxG++1WJLu58OhVfJDAu/dKCmTQ=', 'Content-Length': '0'} 2024-04-16T19:33:12.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001db23fbacc25cc0-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.894 DEBUG:boto:b'yrjefh-14obj-test-zone21000falseobj-test-zone22024-04-16T19:32:04.504Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:12.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-14 zones={test-zone2, test-zone1} 2024-04-16T19:33:12.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-16T19:33:12.896 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:12.896 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:12.897 DEBUG:boto:Method: PUT 2024-04-16T19:33:12.897 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:12.897 DEBUG:boto:Data: 2024-04-16T19:33:12.897 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.897 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.897 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.897 DEBUG:boto:Params: {} 2024-04-16T19:33:12.897 DEBUG:boto:Token: None 2024-04-16T19:33:12.897 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-15/ 2024-04-16T19:33:12.898 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wI0i5ZO5H8LsJU5WaW6TfAMXIcE= 2024-04-16T19:33:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wI0i5ZO5H8LsJU5WaW6TfAMXIcE=', 'Content-Length': '0'} 2024-04-16T19:33:12.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3d2079f149fea4f-00661ed278-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.916 DEBUG:rgw_multi.tests:created bucket=yrjefh-15 2024-04-16T19:33:12.916 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:12.916 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:12.916 DEBUG:boto:path=/yrjefh-15/?versioning 2024-04-16T19:33:12.916 DEBUG:boto:auth_path=/yrjefh-15/?versioning 2024-04-16T19:33:12.916 DEBUG:boto:Method: PUT 2024-04-16T19:33:12.917 DEBUG:boto:Path: /yrjefh-15/?versioning 2024-04-16T19:33:12.917 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:33:12.917 DEBUG:boto:Headers: {} 2024-04-16T19:33:12.917 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:12.917 DEBUG:boto:Port: 8000 2024-04-16T19:33:12.917 DEBUG:boto:Params: {} 2024-04-16T19:33:12.917 DEBUG:boto:Token: None 2024-04-16T19:33:12.917 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:12 GMT /yrjefh-15/?versioning 2024-04-16T19:33:12.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tx+lYUAmHTRejFUbIJJe3cO1E7o= 2024-04-16T19:33:12.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tx+lYUAmHTRejFUbIJJe3cO1E7o=', 'Content-Length': '228'} 2024-04-16T19:33:12.923 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000385d3b9e8076146d-00661ed278-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:12.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:12.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:12.924 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:12.972 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:12.973 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:33:13.046 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713295992.922527_129.1","last_update":"2024-04-16T19:33:12.922527Z"}] 2024-04-16T19:33:13.047 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295992.922527_129.1'} 2024-04-16T19:33:13.047 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:33:13.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:13.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:13.047 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:13.098 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:13.099 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:13.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:13.147+0000 7fb43fd45800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:33:13.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:13.148+0000 7fb43fd45800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:33:13.169 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295992.922527_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:12.922527Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:33:13Z"} 2024-04-16T19:33:13.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295992.922527_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:12.922527Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:33:13Z"} 2024-04-16T19:33:13.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295992.922527_129.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:12.922527Z', 'realm_epoch': 2}}] 2024-04-16T19:33:13.170 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295992.922527_129.1'} 2024-04-16T19:33:13.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295992.922527_129.1'} 2024-04-16T19:33:13.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:33:13.170 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:13.170 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:13.170 DEBUG:boto:Method: HEAD 2024-04-16T19:33:13.170 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:13.171 DEBUG:boto:Data: 2024-04-16T19:33:13.171 DEBUG:boto:Headers: {} 2024-04-16T19:33:13.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:13.171 DEBUG:boto:Port: 8000 2024-04-16T19:33:13.171 DEBUG:boto:Params: {} 2024-04-16T19:33:13.171 DEBUG:boto:Token: None 2024-04-16T19:33:13.171 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:13 GMT /yrjefh-15/ 2024-04-16T19:33:13.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kqEKiECZvC9J86xb00WjS++8odc= 2024-04-16T19:33:13.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kqEKiECZvC9J86xb00WjS++8odc=', 'Content-Length': '0'} 2024-04-16T19:33:13.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066ce4d60493c2f71-00661ed279-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:13.174 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:13.174 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:13.174 DEBUG:boto:Method: PUT 2024-04-16T19:33:13.174 DEBUG:boto:Path: /yrjefh-15/dummy 2024-04-16T19:33:13.174 DEBUG:boto:Data: 2024-04-16T19:33:13.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:33:13.174 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:13.175 DEBUG:boto:Port: 8000 2024-04-16T19:33:13.175 DEBUG:boto:Params: {} 2024-04-16T19:33:13.175 DEBUG:boto:Token: None 2024-04-16T19:33:13.175 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:33:13 GMT /yrjefh-15/dummy 2024-04-16T19:33:13.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBsGlD245IkeCQYlzLjW/bRfY90= 2024-04-16T19:33:13.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBsGlD245IkeCQYlzLjW/bRfY90='} 2024-04-16T19:33:13.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-amz-request-id', 'tx000008c7f92e1e50c76fc-00661ed279-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:13.188 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-15 2024-04-16T19:33:13.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:13.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:13.188 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-15 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:13.243 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:13.243 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:13.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:13.296+0000 7f403411f800 1 waiting to reach incremental sync.. 2024-04-16T19:33:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:18.297+0000 7f403411f800 1 bucket sync caught up with source: 2024-04-16T19:33:18.299 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-04-16T19:33:18.299 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2024-04-16T19:33:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:18.297+0000 7f403411f800 0 bucket checkpoint complete 2024-04-16T19:33:18.323 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-15 2024-04-16T19:33:18.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:18.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:18.324 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-15 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:18.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:18.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:18.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:18.445+0000 7f8fac12a800 1 waiting to reach incremental sync.. 2024-04-16T19:33:23.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:23.447+0000 7f8fac12a800 1 bucket sync caught up with source: 2024-04-16T19:33:23.449 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-16T19:33:23.449 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.37.11, , , , , ] 2024-04-16T19:33:23.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:23.447+0000 7f8fac12a800 0 bucket checkpoint complete 2024-04-16T19:33:23.473 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-15 zones={test-zone2, test-zone1} 2024-04-16T19:33:23.474 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.474 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.474 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.474 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.474 DEBUG:boto:Data: 2024-04-16T19:33:23.474 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.474 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.474 DEBUG:boto:Params: {} 2024-04-16T19:33:23.475 DEBUG:boto:Token: None 2024-04-16T19:33:23.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d676f296dcacb7e-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.478 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.478 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.478 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.478 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.478 DEBUG:boto:Data: 2024-04-16T19:33:23.478 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.478 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.478 DEBUG:boto:Params: {} 2024-04-16T19:33:23.479 DEBUG:boto:Token: None 2024-04-16T19:33:23.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085be44d00638a8da-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:33:23.481 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.482 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.482 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.482 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.482 DEBUG:boto:Method: GET 2024-04-16T19:33:23.482 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.482 DEBUG:boto:Data: 2024-04-16T19:33:23.482 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.482 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.482 DEBUG:boto:Params: {} 2024-04-16T19:33:23.482 DEBUG:boto:Token: None 2024-04-16T19:33:23.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/?versions 2024-04-16T19:33:23.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ= 2024-04-16T19:33:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ=', 'Content-Length': '0'} 2024-04-16T19:33:23.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061fe525377a5f1bf-00661ed283-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.485 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.485 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:23.486 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:33:23.486 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.486 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.486 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.486 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.486 DEBUG:boto:Method: GET 2024-04-16T19:33:23.486 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.486 DEBUG:boto:Data: 2024-04-16T19:33:23.487 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.487 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.487 DEBUG:boto:Params: {} 2024-04-16T19:33:23.487 DEBUG:boto:Token: None 2024-04-16T19:33:23.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/?versions 2024-04-16T19:33:23.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ= 2024-04-16T19:33:23.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ=', 'Content-Length': '0'} 2024-04-16T19:33:23.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67b2a6ce6536696-00661ed283-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.489 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.490 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:23.490 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.490 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.490 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.490 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.490 DEBUG:boto:Method: GET 2024-04-16T19:33:23.490 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.490 DEBUG:boto:Data: 2024-04-16T19:33:23.490 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.490 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.491 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.491 DEBUG:boto:Params: {} 2024-04-16T19:33:23.491 DEBUG:boto:Token: None 2024-04-16T19:33:23.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/?versions 2024-04-16T19:33:23.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ= 2024-04-16T19:33:23.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ=', 'Content-Length': '0'} 2024-04-16T19:33:23.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009459390a89fd403e-00661ed283-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.493 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.494 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.494 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.494 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.494 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.494 DEBUG:boto:Method: GET 2024-04-16T19:33:23.494 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:23.494 DEBUG:boto:Data: 2024-04-16T19:33:23.494 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.494 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.495 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.495 DEBUG:boto:Params: {} 2024-04-16T19:33:23.495 DEBUG:boto:Token: None 2024-04-16T19:33:23.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/?versions 2024-04-16T19:33:23.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ= 2024-04-16T19:33:23.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2J4zNuwN9vNc/oGbsFOy5DZzlrQ=', 'Content-Length': '0'} 2024-04-16T19:33:23.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e535915eaebeeb-00661ed283-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.497 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:23.498 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.498 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.498 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.498 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.498 DEBUG:boto:Method: GET 2024-04-16T19:33:23.498 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.498 DEBUG:boto:Data: 2024-04-16T19:33:23.498 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.498 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.498 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.499 DEBUG:boto:Params: {} 2024-04-16T19:33:23.499 DEBUG:boto:Token: None 2024-04-16T19:33:23.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008afaf68c98b78436-00661ed283-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.501 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.501 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.502 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.502 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.502 DEBUG:boto:Method: GET 2024-04-16T19:33:23.502 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.502 DEBUG:boto:Data: 2024-04-16T19:33:23.502 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.502 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.502 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.502 DEBUG:boto:Params: {} 2024-04-16T19:33:23.502 DEBUG:boto:Token: None 2024-04-16T19:33:23.502 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007448448a17f82d68-00661ed283-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.505 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.505 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.505 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.505 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.505 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.505 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.505 DEBUG:boto:Data: 2024-04-16T19:33:23.505 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.505 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.506 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.506 DEBUG:boto:Params: {} 2024-04-16T19:33:23.506 DEBUG:boto:Token: None 2024-04-16T19:33:23.506 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.506 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jNIhzCQeyjvhx8E/s761S3diPMU= 2024-04-16T19:33:23.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jNIhzCQeyjvhx8E/s761S3diPMU=', 'Content-Length': '0'} 2024-04-16T19:33:23.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033ae37fd9703a015-00661ed283-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.508 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.508 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.508 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.508 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.509 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.509 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.509 DEBUG:boto:Data: 2024-04-16T19:33:23.509 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.509 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.509 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.509 DEBUG:boto:Params: {} 2024-04-16T19:33:23.509 DEBUG:boto:Token: None 2024-04-16T19:33:23.509 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jNIhzCQeyjvhx8E/s761S3diPMU= 2024-04-16T19:33:23.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jNIhzCQeyjvhx8E/s761S3diPMU=', 'Content-Length': '0'} 2024-04-16T19:33:23.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdd5011d5f273aef-00661ed283-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:23.512 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.512 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.512 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.512 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.512 DEBUG:boto:Method: GET 2024-04-16T19:33:23.512 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.512 DEBUG:boto:Data: 2024-04-16T19:33:23.512 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.512 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.512 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.512 DEBUG:boto:Params: {} 2024-04-16T19:33:23.513 DEBUG:boto:Token: None 2024-04-16T19:33:23.513 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036292af8b8e491d4-00661ed283-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.515 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.515 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.515 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.515 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.515 DEBUG:boto:Method: GET 2024-04-16T19:33:23.515 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.515 DEBUG:boto:Data: 2024-04-16T19:33:23.516 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.516 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.516 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.516 DEBUG:boto:Params: {} 2024-04-16T19:33:23.516 DEBUG:boto:Token: None 2024-04-16T19:33:23.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f5805343a1bd0e3-00661ed283-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.518 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.518 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.518 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.518 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.519 DEBUG:boto:Method: GET 2024-04-16T19:33:23.519 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.519 DEBUG:boto:Data: 2024-04-16T19:33:23.519 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.519 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.519 DEBUG:boto:Params: {} 2024-04-16T19:33:23.519 DEBUG:boto:Token: None 2024-04-16T19:33:23.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0= 2024-04-16T19:33:23.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0=', 'Content-Length': '0'} 2024-04-16T19:33:23.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a56c28de2a5407d-00661ed283-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.562 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.563 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.563 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.563 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.563 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.563 DEBUG:boto:Method: GET 2024-04-16T19:33:23.563 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.563 DEBUG:boto:Data: 2024-04-16T19:33:23.563 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.563 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.563 DEBUG:boto:Params: {} 2024-04-16T19:33:23.563 DEBUG:boto:Token: None 2024-04-16T19:33:23.564 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0= 2024-04-16T19:33:23.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0=', 'Content-Length': '0'} 2024-04-16T19:33:23.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8baff7e76cf56ec-00661ed283-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.566 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.566 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.567 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.567 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.567 DEBUG:boto:Path: /yrjefh-15/dummy 2024-04-16T19:33:23.567 DEBUG:boto:Data: 2024-04-16T19:33:23.567 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.567 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.567 DEBUG:boto:Params: {} 2024-04-16T19:33:23.567 DEBUG:boto:Token: None 2024-04-16T19:33:23.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy 2024-04-16T19:33:23.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4eTc1p7UKdZFx45sqLTRNDQYbI0= 2024-04-16T19:33:23.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4eTc1p7UKdZFx45sqLTRNDQYbI0=', 'Content-Length': '0'} 2024-04-16T19:33:23.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094acc81ae45a0cd9-00661ed283-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.570 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.571 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.571 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.571 DEBUG:boto:Path: /yrjefh-15/dummy 2024-04-16T19:33:23.571 DEBUG:boto:Data: 2024-04-16T19:33:23.571 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.571 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.571 DEBUG:boto:Params: {} 2024-04-16T19:33:23.571 DEBUG:boto:Token: None 2024-04-16T19:33:23.571 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy 2024-04-16T19:33:23.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4eTc1p7UKdZFx45sqLTRNDQYbI0= 2024-04-16T19:33:23.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4eTc1p7UKdZFx45sqLTRNDQYbI0=', 'Content-Length': '0'} 2024-04-16T19:33:23.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000210a5b66d9c48450-00661ed283-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:23.574 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.575 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.575 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.575 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.575 DEBUG:boto:Method: GET 2024-04-16T19:33:23.575 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.575 DEBUG:boto:Data: 2024-04-16T19:33:23.575 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.575 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.575 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.575 DEBUG:boto:Params: {} 2024-04-16T19:33:23.575 DEBUG:boto:Token: None 2024-04-16T19:33:23.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007001024ce479cc63-00661ed283-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.578 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:23.578 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:23.578 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.578 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.578 DEBUG:boto:Method: GET 2024-04-16T19:33:23.578 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.578 DEBUG:boto:Data: 2024-04-16T19:33:23.578 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.578 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.579 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.579 DEBUG:boto:Params: {} 2024-04-16T19:33:23.579 DEBUG:boto:Token: None 2024-04-16T19:33:23.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:23.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA= 2024-04-16T19:33:23.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKvRsmdxyDQOIZfEV5NX2uPX0SA=', 'Content-Length': '0'} 2024-04-16T19:33:23.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbc4e10c861540e7-00661ed283-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.581 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.581 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.581 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.582 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.582 DEBUG:boto:Method: GET 2024-04-16T19:33:23.582 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.582 DEBUG:boto:Data: 2024-04-16T19:33:23.582 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.582 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.582 DEBUG:boto:Params: {} 2024-04-16T19:33:23.582 DEBUG:boto:Token: None 2024-04-16T19:33:23.582 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.582 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0= 2024-04-16T19:33:23.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0=', 'Content-Length': '0'} 2024-04-16T19:33:23.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6564d3f5e909ba-00661ed283-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.585 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.585 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.585 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.586 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.586 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.586 DEBUG:boto:Method: GET 2024-04-16T19:33:23.586 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:23.586 DEBUG:boto:Data: 2024-04-16T19:33:23.586 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.586 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.586 DEBUG:boto:Params: {} 2024-04-16T19:33:23.586 DEBUG:boto:Token: None 2024-04-16T19:33:23.586 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0= 2024-04-16T19:33:23.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53+E+79A4pHmdKvMmk+Al8C0AW0=', 'Content-Length': '0'} 2024-04-16T19:33:23.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb94d506a23feb0-00661ed283-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.589 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:23.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-15 zones={test-zone2, test-zone1} 2024-04-16T19:33:23.590 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.590 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.590 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.590 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.590 DEBUG:boto:Data: 2024-04-16T19:33:23.590 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.590 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.590 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.590 DEBUG:boto:Params: {} 2024-04-16T19:33:23.590 DEBUG:boto:Token: None 2024-04-16T19:33:23.590 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e1e26e782a0058c-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.593 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.593 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.593 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.593 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.593 DEBUG:boto:Data: 2024-04-16T19:33:23.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.593 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.593 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.593 DEBUG:boto:Params: {} 2024-04-16T19:33:23.593 DEBUG:boto:Token: None 2024-04-16T19:33:23.594 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-amz-request-id', 'tx00000556708fd6963aff9-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.607 DEBUG:rgw_multi.tests:zone=test-zone1 version=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:23.607 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.607 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.607 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.607 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.607 DEBUG:boto:Data: 2024-04-16T19:33:23.608 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.608 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.608 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.608 DEBUG:boto:Params: {} 2024-04-16T19:33:23.608 DEBUG:boto:Token: None 2024-04-16T19:33:23.608 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004727a79f5c3e208a-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.611 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.611 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.611 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.611 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.611 DEBUG:boto:Data: 2024-04-16T19:33:23.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.611 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.611 DEBUG:boto:Params: {} 2024-04-16T19:33:23.611 DEBUG:boto:Token: None 2024-04-16T19:33:23.611 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-amz-request-id', 'tx00000b3c9dc504419a660-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.627 DEBUG:rgw_multi.tests:zone=test-zone2 version=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:23.627 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.627 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.627 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.627 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.627 DEBUG:boto:Data: 2024-04-16T19:33:23.627 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.627 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.627 DEBUG:boto:Params: {} 2024-04-16T19:33:23.627 DEBUG:boto:Token: None 2024-04-16T19:33:23.628 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffc679077f51ab29-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.630 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.631 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.631 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.631 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.631 DEBUG:boto:Data: 2024-04-16T19:33:23.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.631 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.631 DEBUG:boto:Params: {} 2024-04-16T19:33:23.631 DEBUG:boto:Token: None 2024-04-16T19:33:23.632 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-amz-request-id', 'tx00000ae8cdcb109a91e65-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.643 DEBUG:rgw_multi.tests:zone=test-zone1 version=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:23.643 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.643 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.643 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.643 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.644 DEBUG:boto:Data: 2024-04-16T19:33:23.644 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.644 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.644 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.644 DEBUG:boto:Params: {} 2024-04-16T19:33:23.644 DEBUG:boto:Token: None 2024-04-16T19:33:23.644 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa7a23996789a9ee-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.647 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.647 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.647 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.647 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.647 DEBUG:boto:Data: 2024-04-16T19:33:23.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.647 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.647 DEBUG:boto:Params: {} 2024-04-16T19:33:23.647 DEBUG:boto:Token: None 2024-04-16T19:33:23.648 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-amz-request-id', 'tx00000a37aa5ce844f4e11-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.691 DEBUG:rgw_multi.tests:zone=test-zone2 version=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:23.691 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.691 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.691 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.691 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.691 DEBUG:boto:Data: 2024-04-16T19:33:23.691 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.691 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.691 DEBUG:boto:Params: {} 2024-04-16T19:33:23.692 DEBUG:boto:Token: None 2024-04-16T19:33:23.692 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae7e1df0f9c9bad6-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.694 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.694 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.694 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.695 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.695 DEBUG:boto:Data: 2024-04-16T19:33:23.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.695 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.695 DEBUG:boto:Params: {} 2024-04-16T19:33:23.695 DEBUG:boto:Token: None 2024-04-16T19:33:23.695 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-amz-request-id', 'tx0000025326342095c2087-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.708 DEBUG:rgw_multi.tests:zone=test-zone1 version=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:23.708 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.708 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.708 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.708 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.708 DEBUG:boto:Data: 2024-04-16T19:33:23.708 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.708 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.708 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.708 DEBUG:boto:Params: {} 2024-04-16T19:33:23.709 DEBUG:boto:Token: None 2024-04-16T19:33:23.709 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.709 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83e1b3a69e68ec4-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.711 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.711 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.711 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.712 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.712 DEBUG:boto:Data: 2024-04-16T19:33:23.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.712 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.712 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.712 DEBUG:boto:Params: {} 2024-04-16T19:33:23.712 DEBUG:boto:Token: None 2024-04-16T19:33:23.712 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-amz-request-id', 'tx000005366c0a43048f78c-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.724 DEBUG:rgw_multi.tests:zone=test-zone2 version=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:23.724 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.725 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.725 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.725 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.725 DEBUG:boto:Data: 2024-04-16T19:33:23.725 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.725 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.725 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.725 DEBUG:boto:Params: {} 2024-04-16T19:33:23.725 DEBUG:boto:Token: None 2024-04-16T19:33:23.725 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbc4df80854a2a6f-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.728 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.728 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.728 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.728 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.729 DEBUG:boto:Data: 2024-04-16T19:33:23.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.729 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.729 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.729 DEBUG:boto:Params: {} 2024-04-16T19:33:23.729 DEBUG:boto:Token: None 2024-04-16T19:33:23.729 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-amz-request-id', 'tx00000a85005d49a2026c2-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.775 DEBUG:rgw_multi.tests:zone=test-zone1 version=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:23.775 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.775 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.775 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.775 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.775 DEBUG:boto:Data: 2024-04-16T19:33:23.775 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.775 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.775 DEBUG:boto:Params: {} 2024-04-16T19:33:23.775 DEBUG:boto:Token: None 2024-04-16T19:33:23.776 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003ebf843e24093ae-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.778 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.778 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.778 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.778 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.778 DEBUG:boto:Data: 2024-04-16T19:33:23.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.779 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.779 DEBUG:boto:Params: {} 2024-04-16T19:33:23.779 DEBUG:boto:Token: None 2024-04-16T19:33:23.779 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-amz-request-id', 'tx000003fd746d91d7c8c27-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.790 DEBUG:rgw_multi.tests:zone=test-zone2 version=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:23.790 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.791 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.791 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.791 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.791 DEBUG:boto:Data: 2024-04-16T19:33:23.791 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.791 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.791 DEBUG:boto:Params: {} 2024-04-16T19:33:23.791 DEBUG:boto:Token: None 2024-04-16T19:33:23.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cf92f54433ff93c-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.794 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.794 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.794 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.794 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.794 DEBUG:boto:Data: 2024-04-16T19:33:23.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.794 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.794 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.794 DEBUG:boto:Params: {} 2024-04-16T19:33:23.795 DEBUG:boto:Token: None 2024-04-16T19:33:23.795 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-amz-request-id', 'tx00000272edfc1500d1c46-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.806 DEBUG:rgw_multi.tests:zone=test-zone1 version=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:23.806 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.806 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.806 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.806 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.807 DEBUG:boto:Data: 2024-04-16T19:33:23.807 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.807 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.807 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.807 DEBUG:boto:Params: {} 2024-04-16T19:33:23.807 DEBUG:boto:Token: None 2024-04-16T19:33:23.807 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050ce1478d00fbff-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.810 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.810 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.810 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.810 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.810 DEBUG:boto:Data: 2024-04-16T19:33:23.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.810 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.810 DEBUG:boto:Params: {} 2024-04-16T19:33:23.811 DEBUG:boto:Token: None 2024-04-16T19:33:23.811 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-amz-request-id', 'tx000003280a16aebc92107-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.855 DEBUG:rgw_multi.tests:zone=test-zone2 version=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:23.855 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.855 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.855 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.855 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.855 DEBUG:boto:Data: 2024-04-16T19:33:23.855 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.855 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.855 DEBUG:boto:Params: {} 2024-04-16T19:33:23.855 DEBUG:boto:Token: None 2024-04-16T19:33:23.856 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3258b945f4acfe8-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.858 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.858 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.858 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.859 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.859 DEBUG:boto:Data: 2024-04-16T19:33:23.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.859 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.859 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.859 DEBUG:boto:Params: {} 2024-04-16T19:33:23.859 DEBUG:boto:Token: None 2024-04-16T19:33:23.859 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-amz-request-id', 'tx000007bc14ffab1d056bd-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.870 DEBUG:rgw_multi.tests:zone=test-zone1 version=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:23.870 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.870 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.870 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.870 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.871 DEBUG:boto:Data: 2024-04-16T19:33:23.871 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.871 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.871 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.871 DEBUG:boto:Params: {} 2024-04-16T19:33:23.871 DEBUG:boto:Token: None 2024-04-16T19:33:23.871 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b03607c31599c94d-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.874 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.874 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.874 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.874 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.874 DEBUG:boto:Data: 2024-04-16T19:33:23.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.874 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.874 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.874 DEBUG:boto:Params: {} 2024-04-16T19:33:23.875 DEBUG:boto:Token: None 2024-04-16T19:33:23.875 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-amz-request-id', 'tx00000b790dc45b09a47b7-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.886 DEBUG:rgw_multi.tests:zone=test-zone2 version=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:23.886 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.886 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.886 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.886 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.886 DEBUG:boto:Data: 2024-04-16T19:33:23.886 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.887 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.887 DEBUG:boto:Params: {} 2024-04-16T19:33:23.887 DEBUG:boto:Token: None 2024-04-16T19:33:23.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b32d695b880d0bed-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.890 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.890 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.890 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.890 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.890 DEBUG:boto:Data: 2024-04-16T19:33:23.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.890 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.890 DEBUG:boto:Params: {} 2024-04-16T19:33:23.890 DEBUG:boto:Token: None 2024-04-16T19:33:23.891 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-amz-request-id', 'tx00000e10ad6b21902de6c-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.902 DEBUG:rgw_multi.tests:zone=test-zone1 version=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:23.903 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.903 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.903 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.903 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.903 DEBUG:boto:Data: 2024-04-16T19:33:23.903 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.903 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.903 DEBUG:boto:Params: {} 2024-04-16T19:33:23.903 DEBUG:boto:Token: None 2024-04-16T19:33:23.903 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008efd7e950d0f3448-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.906 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.906 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.906 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.906 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.906 DEBUG:boto:Data: 2024-04-16T19:33:23.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.907 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.907 DEBUG:boto:Params: {} 2024-04-16T19:33:23.907 DEBUG:boto:Token: None 2024-04-16T19:33:23.907 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-amz-request-id', 'tx0000004ac4b16843a7e72-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.918 DEBUG:rgw_multi.tests:zone=test-zone2 version=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:23.918 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.918 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.918 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.918 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.919 DEBUG:boto:Data: 2024-04-16T19:33:23.919 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.919 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.919 DEBUG:boto:Params: {} 2024-04-16T19:33:23.919 DEBUG:boto:Token: None 2024-04-16T19:33:23.919 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007827baf49028ebaa-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.922 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.922 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.922 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.922 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.922 DEBUG:boto:Data: 2024-04-16T19:33:23.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.922 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.922 DEBUG:boto:Params: {} 2024-04-16T19:33:23.923 DEBUG:boto:Token: None 2024-04-16T19:33:23.923 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-amz-request-id', 'tx000005141c245a36d1288-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.967 DEBUG:rgw_multi.tests:zone=test-zone1 version=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:23.967 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.967 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.967 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.967 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.967 DEBUG:boto:Data: 2024-04-16T19:33:23.967 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.967 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.967 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.967 DEBUG:boto:Params: {} 2024-04-16T19:33:23.968 DEBUG:boto:Token: None 2024-04-16T19:33:23.968 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5bd0b80c58c247d-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.970 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.970 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.970 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.971 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.971 DEBUG:boto:Data: 2024-04-16T19:33:23.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.971 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.971 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.971 DEBUG:boto:Params: {} 2024-04-16T19:33:23.971 DEBUG:boto:Token: None 2024-04-16T19:33:23.971 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-amz-request-id', 'tx00000e4fa67da538a5005-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.982 DEBUG:rgw_multi.tests:zone=test-zone2 version=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:23.982 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.982 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.983 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.983 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.983 DEBUG:boto:Data: 2024-04-16T19:33:23.983 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.983 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.983 DEBUG:boto:Params: {} 2024-04-16T19:33:23.983 DEBUG:boto:Token: None 2024-04-16T19:33:23.983 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:23.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5475cb6ab15dd28-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.986 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:23.986 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:23.986 DEBUG:boto:Method: PUT 2024-04-16T19:33:23.986 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:23.986 DEBUG:boto:Data: 2024-04-16T19:33:23.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:23.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:23.986 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.986 DEBUG:boto:Params: {} 2024-04-16T19:33:23.987 DEBUG:boto:Token: None 2024-04-16T19:33:23.987 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/obj 2024-04-16T19:33:23.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk= 2024-04-16T19:33:23.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:30Tc9JtZSQs381ylkUZmxKB2TKk='} 2024-04-16T19:33:23.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-amz-request-id', 'tx000002a1afbec7ab78659-00661ed283-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:23.998 DEBUG:rgw_multi.tests:zone=test-zone1 version=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:23.998 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:23.998 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:23.998 DEBUG:boto:Method: HEAD 2024-04-16T19:33:23.998 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:23.999 DEBUG:boto:Data: 2024-04-16T19:33:23.999 DEBUG:boto:Headers: {} 2024-04-16T19:33:23.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:23.999 DEBUG:boto:Port: 8000 2024-04-16T19:33:23.999 DEBUG:boto:Params: {} 2024-04-16T19:33:23.999 DEBUG:boto:Token: None 2024-04-16T19:33:23.999 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:23 GMT /yrjefh-15/ 2024-04-16T19:33:23.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8= 2024-04-16T19:33:23.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toUnpDzQ/8wVG+eg2PVk+bIx4u8=', 'Content-Length': '0'} 2024-04-16T19:33:24.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf734b0676bb68a8-00661ed283-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.002 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:24.002 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:24.002 DEBUG:boto:Method: PUT 2024-04-16T19:33:24.002 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:24.002 DEBUG:boto:Data: 2024-04-16T19:33:24.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:24.002 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:24.002 DEBUG:boto:Port: 8000 2024-04-16T19:33:24.003 DEBUG:boto:Params: {} 2024-04-16T19:33:24.003 DEBUG:boto:Token: None 2024-04-16T19:33:24.003 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:24 GMT /yrjefh-15/obj 2024-04-16T19:33:24.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag= 2024-04-16T19:33:24.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag='} 2024-04-16T19:33:24.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-amz-request-id', 'tx00000987dcd447469cb42-00661ed284-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.047 DEBUG:rgw_multi.tests:zone=test-zone2 version=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:24.047 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:24.047 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:24.047 DEBUG:boto:Method: HEAD 2024-04-16T19:33:24.047 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:24.047 DEBUG:boto:Data: 2024-04-16T19:33:24.047 DEBUG:boto:Headers: {} 2024-04-16T19:33:24.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:24.047 DEBUG:boto:Port: 8000 2024-04-16T19:33:24.047 DEBUG:boto:Params: {} 2024-04-16T19:33:24.048 DEBUG:boto:Token: None 2024-04-16T19:33:24.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:24 GMT /yrjefh-15/ 2024-04-16T19:33:24.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ixb4YrUaPrGCdpmXWyZEaA7aKnk= 2024-04-16T19:33:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ixb4YrUaPrGCdpmXWyZEaA7aKnk=', 'Content-Length': '0'} 2024-04-16T19:33:24.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d14b71ae1b21b0f-00661ed284-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.050 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:24.051 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:24.051 DEBUG:boto:Method: PUT 2024-04-16T19:33:24.051 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:24.051 DEBUG:boto:Data: 2024-04-16T19:33:24.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:24.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:24.051 DEBUG:boto:Port: 8000 2024-04-16T19:33:24.051 DEBUG:boto:Params: {} 2024-04-16T19:33:24.051 DEBUG:boto:Token: None 2024-04-16T19:33:24.051 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:24 GMT /yrjefh-15/obj 2024-04-16T19:33:24.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag= 2024-04-16T19:33:24.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag='} 2024-04-16T19:33:24.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-amz-request-id', 'tx0000029c18cc09cf9df45-00661ed284-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.063 DEBUG:rgw_multi.tests:zone=test-zone1 version=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:24.063 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:24.063 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:24.063 DEBUG:boto:Method: HEAD 2024-04-16T19:33:24.064 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:24.064 DEBUG:boto:Data: 2024-04-16T19:33:24.064 DEBUG:boto:Headers: {} 2024-04-16T19:33:24.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:24.064 DEBUG:boto:Port: 8000 2024-04-16T19:33:24.064 DEBUG:boto:Params: {} 2024-04-16T19:33:24.064 DEBUG:boto:Token: None 2024-04-16T19:33:24.064 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:24 GMT /yrjefh-15/ 2024-04-16T19:33:24.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ixb4YrUaPrGCdpmXWyZEaA7aKnk= 2024-04-16T19:33:24.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ixb4YrUaPrGCdpmXWyZEaA7aKnk=', 'Content-Length': '0'} 2024-04-16T19:33:24.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000231a9d7a44479a91-00661ed284-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.067 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:24.067 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:24.067 DEBUG:boto:Method: PUT 2024-04-16T19:33:24.067 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:24.067 DEBUG:boto:Data: 2024-04-16T19:33:24.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-16T19:33:24.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:24.068 DEBUG:boto:Port: 8000 2024-04-16T19:33:24.068 DEBUG:boto:Params: {} 2024-04-16T19:33:24.068 DEBUG:boto:Token: None 2024-04-16T19:33:24.068 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 16 Apr 2024 19:33:24 GMT /yrjefh-15/obj 2024-04-16T19:33:24.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag= 2024-04-16T19:33:24.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UwONmNdo+zigVA7NfSKkk7y/bag='} 2024-04-16T19:33:24.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-amz-request-id', 'tx000004c3e1e70d389f477-00661ed284-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:24.079 DEBUG:rgw_multi.tests:zone=test-zone2 version=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:24.079 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-15 2024-04-16T19:33:24.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:24.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:24.079 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-15 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:24.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:24.196+0000 7fb0cd8c0800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:24.197 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-04-16T19:33:24.197 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000003.29.10, , , 00000000020.56.12, , ] 2024-04-16T19:33:29.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:29.197+0000 7fb0cd8c0800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:29.199 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000020.56.12, , ] 2024-04-16T19:33:29.199 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000003.29.10, , , 00000000020.56.12, , ] 2024-04-16T19:33:34.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:34.198+0000 7fb0cd8c0800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:34.200 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000020.56.12, , ] 2024-04-16T19:33:34.200 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000003.29.10, , , 00000000020.56.12, , ] 2024-04-16T19:33:39.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:39.199+0000 7fb0cd8c0800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:39.202 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000020.56.12, , ] 2024-04-16T19:33:39.202 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000003.29.10, , , 00000000020.56.12, , ] 2024-04-16T19:33:44.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:44.201+0000 7fb0cd8c0800 1 bucket sync caught up with source: 2024-04-16T19:33:44.203 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000003.29.10, , , 00000000024.68.6, , ] 2024-04-16T19:33:44.203 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000003.29.10, , , 00000000020.56.12, , ] 2024-04-16T19:33:44.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:44.201+0000 7fb0cd8c0800 0 bucket checkpoint complete 2024-04-16T19:33:44.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-15 2024-04-16T19:33:44.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:44.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-15', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:44.228 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-15 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:44.291 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:44.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:44.352+0000 7f4f43e39800 1 waiting for incremental sync to catch up: 2024-04-16T19:33:44.354 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000002.37.11, , , 00000000009.34.6, , ] 2024-04-16T19:33:44.354 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.37.11, , , 00000000031.86.9, , ] 2024-04-16T19:33:49.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:49.353+0000 7f4f43e39800 1 bucket sync caught up with source: 2024-04-16T19:33:49.355 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000002.37.11, , , 00000000031.86.9, , ] 2024-04-16T19:33:49.356 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.37.11, , , 00000000031.86.9, , ] 2024-04-16T19:33:49.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:33:49.353+0000 7f4f43e39800 0 bucket checkpoint complete 2024-04-16T19:33:49.380 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-15 zones={test-zone2, test-zone1} 2024-04-16T19:33:49.380 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.381 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.381 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.381 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:49.381 DEBUG:boto:Data: 2024-04-16T19:33:49.381 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.381 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.381 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.381 DEBUG:boto:Params: {} 2024-04-16T19:33:49.381 DEBUG:boto:Token: None 2024-04-16T19:33:49.382 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.382 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9t0BF+Lr/4YFD1MgGwoR7BTn0g0= 2024-04-16T19:33:49.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9t0BF+Lr/4YFD1MgGwoR7BTn0g0=', 'Content-Length': '0'} 2024-04-16T19:33:49.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cdc096ff0707038-00661ed29d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.385 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.385 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.386 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.386 DEBUG:boto:Path: /yrjefh-15/ 2024-04-16T19:33:49.386 DEBUG:boto:Data: 2024-04-16T19:33:49.386 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.386 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.386 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.386 DEBUG:boto:Params: {} 2024-04-16T19:33:49.386 DEBUG:boto:Token: None 2024-04-16T19:33:49.386 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.386 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9t0BF+Lr/4YFD1MgGwoR7BTn0g0= 2024-04-16T19:33:49.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9t0BF+Lr/4YFD1MgGwoR7BTn0g0=', 'Content-Length': '0'} 2024-04-16T19:33:49.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b5b142723a4780a-00661ed29d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.389 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:33:49.389 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.390 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.390 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.390 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.390 DEBUG:boto:Method: GET 2024-04-16T19:33:49.390 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.390 DEBUG:boto:Data: 2024-04-16T19:33:49.390 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.390 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.390 DEBUG:boto:Params: {} 2024-04-16T19:33:49.391 DEBUG:boto:Token: None 2024-04-16T19:33:49.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/?versions 2024-04-16T19:33:49.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M= 2024-04-16T19:33:49.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M=', 'Content-Length': '0'} 2024-04-16T19:33:49.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd43920d4948f652-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.394 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111NormalobjCMUP7qgLDH-ugNM44s2aPhnev-gtpbTtrue2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjHrsL5oz7VPiuzzM-jPvC7pNxy3XcWTtfalse2024-04-16T19:33:24.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnufalse2024-04-16T19:33:23.988Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJF3QhPh4rjNtlWbfOOCOAgIBw58tClufalse2024-04-16T19:33:24.004Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjBE5KRw60iPLiIC0pRor7iVGgCFHPOMXfalse2024-04-16T19:33:23.924Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjz5NEQlCe9sj9xqcXfSR.CdpxO0XYz-Gfalse2024-04-16T19:33:23.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8sfalse2024-04-16T19:33:23.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjCYAOhFv8bg7kzgAKSBFmemvPHi04V8rfalse2024-04-16T19:33:23.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjN4TTl.jGFPrOisqoPfBas3tagTGN6P8false2024-04-16T19:33:23.876Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjcMEGTbElZiAsg21zVl0Gu3D.rK1-aRmfalse2024-04-16T19:33:23.860Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPNCSB4IaD-TcrJja6nFQ.S9oWsdHA5cfalse2024-04-16T19:33:23.796Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjuNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5false2024-04-16T19:33:23.812Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2false2024-04-16T19:33:23.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjdRPU.SBoWM6qiICyhzAojFNNU.nHDyAfalse2024-04-16T19:33:23.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjnCXWVN92LLTZ7uScP.40eRgAyqJsfbKfalse2024-04-16T19:33:23.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjw8fgzqLyp-vAWbVK094EB5Cn6ffO05Mfalse2024-04-16T19:33:23.696Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6qxMhlyhJvcIxiHpEYtnaj9M6riEhpgfalse2024-04-16T19:33:23.633Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TRfalse2024-04-16T19:33:23.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJfyqEJD8sqQTmgMG3Eku8wfrvQvaTFifalse2024-04-16T19:33:23.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjgFRR0OYRsUkl6HocRmPvamoSSS8O0Tufalse2024-04-16T19:33:23.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:33:49.398 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.398 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.398 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.398 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.398 DEBUG:boto:Method: GET 2024-04-16T19:33:49.398 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.398 DEBUG:boto:Data: 2024-04-16T19:33:49.399 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.399 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.399 DEBUG:boto:Params: {} 2024-04-16T19:33:49.399 DEBUG:boto:Token: None 2024-04-16T19:33:49.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/?versions 2024-04-16T19:33:49.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M= 2024-04-16T19:33:49.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M=', 'Content-Length': '0'} 2024-04-16T19:33:49.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dc5c7981053da4-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.402 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111NormalobjCMUP7qgLDH-ugNM44s2aPhnev-gtpbTtrue2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjHrsL5oz7VPiuzzM-jPvC7pNxy3XcWTtfalse2024-04-16T19:33:24.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnufalse2024-04-16T19:33:23.988Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJF3QhPh4rjNtlWbfOOCOAgIBw58tClufalse2024-04-16T19:33:24.004Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjBE5KRw60iPLiIC0pRor7iVGgCFHPOMXfalse2024-04-16T19:33:23.924Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjz5NEQlCe9sj9xqcXfSR.CdpxO0XYz-Gfalse2024-04-16T19:33:23.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8sfalse2024-04-16T19:33:23.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjCYAOhFv8bg7kzgAKSBFmemvPHi04V8rfalse2024-04-16T19:33:23.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjN4TTl.jGFPrOisqoPfBas3tagTGN6P8false2024-04-16T19:33:23.876Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjcMEGTbElZiAsg21zVl0Gu3D.rK1-aRmfalse2024-04-16T19:33:23.860Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPNCSB4IaD-TcrJja6nFQ.S9oWsdHA5cfalse2024-04-16T19:33:23.796Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjuNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5false2024-04-16T19:33:23.812Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2false2024-04-16T19:33:23.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjdRPU.SBoWM6qiICyhzAojFNNU.nHDyAfalse2024-04-16T19:33:23.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjnCXWVN92LLTZ7uScP.40eRgAyqJsfbKfalse2024-04-16T19:33:23.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjw8fgzqLyp-vAWbVK094EB5Cn6ffO05Mfalse2024-04-16T19:33:23.696Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6qxMhlyhJvcIxiHpEYtnaj9M6riEhpgfalse2024-04-16T19:33:23.633Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TRfalse2024-04-16T19:33:23.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJfyqEJD8sqQTmgMG3Eku8wfrvQvaTFifalse2024-04-16T19:33:23.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjgFRR0OYRsUkl6HocRmPvamoSSS8O0Tufalse2024-04-16T19:33:23.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.403 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T19:33:49.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:33:49.405 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.405 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.405 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.406 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.406 DEBUG:boto:Method: GET 2024-04-16T19:33:49.406 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.406 DEBUG:boto:Data: 2024-04-16T19:33:49.406 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.406 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.406 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.406 DEBUG:boto:Params: {} 2024-04-16T19:33:49.406 DEBUG:boto:Token: None 2024-04-16T19:33:49.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/?versions 2024-04-16T19:33:49.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M= 2024-04-16T19:33:49.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M=', 'Content-Length': '0'} 2024-04-16T19:33:49.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c074a4795397b7-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.410 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111NormalobjCMUP7qgLDH-ugNM44s2aPhnev-gtpbTtrue2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjHrsL5oz7VPiuzzM-jPvC7pNxy3XcWTtfalse2024-04-16T19:33:24.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnufalse2024-04-16T19:33:23.988Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJF3QhPh4rjNtlWbfOOCOAgIBw58tClufalse2024-04-16T19:33:24.004Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjBE5KRw60iPLiIC0pRor7iVGgCFHPOMXfalse2024-04-16T19:33:23.924Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjz5NEQlCe9sj9xqcXfSR.CdpxO0XYz-Gfalse2024-04-16T19:33:23.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8sfalse2024-04-16T19:33:23.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjCYAOhFv8bg7kzgAKSBFmemvPHi04V8rfalse2024-04-16T19:33:23.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjN4TTl.jGFPrOisqoPfBas3tagTGN6P8false2024-04-16T19:33:23.876Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjcMEGTbElZiAsg21zVl0Gu3D.rK1-aRmfalse2024-04-16T19:33:23.860Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPNCSB4IaD-TcrJja6nFQ.S9oWsdHA5cfalse2024-04-16T19:33:23.796Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjuNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5false2024-04-16T19:33:23.812Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2false2024-04-16T19:33:23.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjdRPU.SBoWM6qiICyhzAojFNNU.nHDyAfalse2024-04-16T19:33:23.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjnCXWVN92LLTZ7uScP.40eRgAyqJsfbKfalse2024-04-16T19:33:23.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjw8fgzqLyp-vAWbVK094EB5Cn6ffO05Mfalse2024-04-16T19:33:23.696Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6qxMhlyhJvcIxiHpEYtnaj9M6riEhpgfalse2024-04-16T19:33:23.633Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TRfalse2024-04-16T19:33:23.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJfyqEJD8sqQTmgMG3Eku8wfrvQvaTFifalse2024-04-16T19:33:23.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjgFRR0OYRsUkl6HocRmPvamoSSS8O0Tufalse2024-04-16T19:33:23.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.411 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.411 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.412 DEBUG:boto:path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.412 DEBUG:boto:auth_path=/yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.412 DEBUG:boto:Method: GET 2024-04-16T19:33:49.412 DEBUG:boto:Path: /yrjefh-15/?versions&max-keys=999 2024-04-16T19:33:49.412 DEBUG:boto:Data: 2024-04-16T19:33:49.412 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.412 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.412 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.412 DEBUG:boto:Params: {} 2024-04-16T19:33:49.412 DEBUG:boto:Token: None 2024-04-16T19:33:49.413 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/?versions 2024-04-16T19:33:49.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M= 2024-04-16T19:33:49.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WUlp9aRouBbP5b9jD6gAH7qly2M=', 'Content-Length': '0'} 2024-04-16T19:33:49.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cfed8a1aa8698c1-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.416 DEBUG:boto:b'yrjefh-15999falsedummyHnLrVAl.QXugJroaYfxL8VdbRBHLVSbtrue2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111NormalobjCMUP7qgLDH-ugNM44s2aPhnev-gtpbTtrue2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjHrsL5oz7VPiuzzM-jPvC7pNxy3XcWTtfalse2024-04-16T19:33:24.053Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnufalse2024-04-16T19:33:23.988Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJF3QhPh4rjNtlWbfOOCOAgIBw58tClufalse2024-04-16T19:33:24.004Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjBE5KRw60iPLiIC0pRor7iVGgCFHPOMXfalse2024-04-16T19:33:23.924Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjz5NEQlCe9sj9xqcXfSR.CdpxO0XYz-Gfalse2024-04-16T19:33:23.972Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8sfalse2024-04-16T19:33:23.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjCYAOhFv8bg7kzgAKSBFmemvPHi04V8rfalse2024-04-16T19:33:23.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjN4TTl.jGFPrOisqoPfBas3tagTGN6P8false2024-04-16T19:33:23.876Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjcMEGTbElZiAsg21zVl0Gu3D.rK1-aRmfalse2024-04-16T19:33:23.860Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPNCSB4IaD-TcrJja6nFQ.S9oWsdHA5cfalse2024-04-16T19:33:23.796Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjuNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5false2024-04-16T19:33:23.812Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2false2024-04-16T19:33:23.780Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjdRPU.SBoWM6qiICyhzAojFNNU.nHDyAfalse2024-04-16T19:33:23.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjnCXWVN92LLTZ7uScP.40eRgAyqJsfbKfalse2024-04-16T19:33:23.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobjw8fgzqLyp-vAWbVK094EB5Cn6ffO05Mfalse2024-04-16T19:33:23.696Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normalobj6qxMhlyhJvcIxiHpEYtnaj9M6riEhpgfalse2024-04-16T19:33:23.633Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjPLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TRfalse2024-04-16T19:33:23.649Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjJfyqEJD8sqQTmgMG3Eku8wfrvQvaTFifalse2024-04-16T19:33:23.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111NormalobjgFRR0OYRsUkl6HocRmPvamoSSS8O0Tufalse2024-04-16T19:33:23.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:49.417 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.417 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.417 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.417 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.417 DEBUG:boto:Method: GET 2024-04-16T19:33:49.417 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.417 DEBUG:boto:Data: 2024-04-16T19:33:49.417 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.418 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.418 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.418 DEBUG:boto:Params: {} 2024-04-16T19:33:49.418 DEBUG:boto:Token: None 2024-04-16T19:33:49.418 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f28a443adcf5dcd-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.420 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.421 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.421 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.421 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.421 DEBUG:boto:Method: GET 2024-04-16T19:33:49.421 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.421 DEBUG:boto:Data: 2024-04-16T19:33:49.421 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.421 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.421 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.421 DEBUG:boto:Params: {} 2024-04-16T19:33:49.422 DEBUG:boto:Token: None 2024-04-16T19:33:49.422 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.422 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcf1bbeb110d47b2-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.424 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.424 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.424 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.425 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.425 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.425 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.425 DEBUG:boto:Data: 2024-04-16T19:33:49.425 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.425 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.425 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.425 DEBUG:boto:Params: {} 2024-04-16T19:33:49.425 DEBUG:boto:Token: None 2024-04-16T19:33:49.425 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HRHLFMC53WXnpu7lMooW71qwalg= 2024-04-16T19:33:49.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HRHLFMC53WXnpu7lMooW71qwalg=', 'Content-Length': '0'} 2024-04-16T19:33:49.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a59a99407b1f0533-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.428 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.428 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.428 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.428 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.428 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.428 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.428 DEBUG:boto:Data: 2024-04-16T19:33:49.428 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.428 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.429 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.429 DEBUG:boto:Params: {} 2024-04-16T19:33:49.429 DEBUG:boto:Token: None 2024-04-16T19:33:49.429 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.429 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HRHLFMC53WXnpu7lMooW71qwalg= 2024-04-16T19:33:49.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HRHLFMC53WXnpu7lMooW71qwalg=', 'Content-Length': '0'} 2024-04-16T19:33:49.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e537afd941189cfa-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:49.431 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.431 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.431 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.431 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.431 DEBUG:boto:Method: GET 2024-04-16T19:33:49.431 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.432 DEBUG:boto:Data: 2024-04-16T19:33:49.432 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.432 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.432 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.432 DEBUG:boto:Params: {} 2024-04-16T19:33:49.432 DEBUG:boto:Token: None 2024-04-16T19:33:49.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0d176cc4c5ac5c4-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.435 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.435 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.435 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.435 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.435 DEBUG:boto:Method: GET 2024-04-16T19:33:49.435 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.435 DEBUG:boto:Data: 2024-04-16T19:33:49.435 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.436 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.436 DEBUG:boto:Params: {} 2024-04-16T19:33:49.436 DEBUG:boto:Token: None 2024-04-16T19:33:49.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fbedf3ab967c835-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.438 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.438 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.438 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.439 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.439 DEBUG:boto:Method: GET 2024-04-16T19:33:49.439 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.439 DEBUG:boto:Data: 2024-04-16T19:33:49.439 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.439 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.439 DEBUG:boto:Params: {} 2024-04-16T19:33:49.439 DEBUG:boto:Token: None 2024-04-16T19:33:49.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a5e531e640a48f-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.490 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.491 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.491 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.491 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.491 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.491 DEBUG:boto:Method: GET 2024-04-16T19:33:49.491 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.492 DEBUG:boto:Data: 2024-04-16T19:33:49.492 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.492 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.492 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.492 DEBUG:boto:Params: {} 2024-04-16T19:33:49.492 DEBUG:boto:Token: None 2024-04-16T19:33:49.492 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfcf8179e177321-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.495 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.495 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.495 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.495 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.496 DEBUG:boto:Path: /yrjefh-15/dummy 2024-04-16T19:33:49.496 DEBUG:boto:Data: 2024-04-16T19:33:49.496 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.496 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.496 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.496 DEBUG:boto:Params: {} 2024-04-16T19:33:49.496 DEBUG:boto:Token: None 2024-04-16T19:33:49.496 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy 2024-04-16T19:33:49.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gMJeJsqvIUi4B65D4flg5pr0haA= 2024-04-16T19:33:49.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gMJeJsqvIUi4B65D4flg5pr0haA=', 'Content-Length': '0'} 2024-04-16T19:33:49.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007b771fc9a5b720d-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.499 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.499 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.499 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.499 DEBUG:boto:Path: /yrjefh-15/dummy 2024-04-16T19:33:49.499 DEBUG:boto:Data: 2024-04-16T19:33:49.499 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.500 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.500 DEBUG:boto:Params: {} 2024-04-16T19:33:49.500 DEBUG:boto:Token: None 2024-04-16T19:33:49.500 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy 2024-04-16T19:33:49.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gMJeJsqvIUi4B65D4flg5pr0haA= 2024-04-16T19:33:49.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gMJeJsqvIUi4B65D4flg5pr0haA=', 'Content-Length': '0'} 2024-04-16T19:33:49.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac006037695af429-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T19:33:49.503 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.503 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.503 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.503 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.503 DEBUG:boto:Method: GET 2024-04-16T19:33:49.503 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.503 DEBUG:boto:Data: 2024-04-16T19:33:49.503 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.503 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.504 DEBUG:boto:Params: {} 2024-04-16T19:33:49.504 DEBUG:boto:Token: None 2024-04-16T19:33:49.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:15 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e595b745fa36abb-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.506 DEBUG:boto:path=/yrjefh-15/dummy 2024-04-16T19:33:49.506 DEBUG:boto:auth_path=/yrjefh-15/dummy 2024-04-16T19:33:49.506 DEBUG:boto:path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.506 DEBUG:boto:auth_path=/yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.507 DEBUG:boto:Method: GET 2024-04-16T19:33:49.507 DEBUG:boto:Path: /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.507 DEBUG:boto:Data: 2024-04-16T19:33:49.507 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.507 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.507 DEBUG:boto:Params: {} 2024-04-16T19:33:49.507 DEBUG:boto:Token: None 2024-04-16T19:33:49.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/dummy?versionId=HnLrVAl.QXugJroaYfxL8VdbRBHLVSb 2024-04-16T19:33:49.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg= 2024-04-16T19:33:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eDBzAPdja37HA4SRV4SmsXhZpg=', 'Content-Length': '0'} 2024-04-16T19:33:49.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:13 GMT'), ('x-amz-version-id', 'HnLrVAl.QXugJroaYfxL8VdbRBHLVSb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000019234f0455a60a2-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.509 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.510 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.510 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.510 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.510 DEBUG:boto:Method: GET 2024-04-16T19:33:49.510 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.510 DEBUG:boto:Data: 2024-04-16T19:33:49.510 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.510 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.510 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.510 DEBUG:boto:Params: {} 2024-04-16T19:33:49.511 DEBUG:boto:Token: None 2024-04-16T19:33:49.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c245de27ea1c5f-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.513 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.514 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.514 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.514 DEBUG:boto:path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.514 DEBUG:boto:auth_path=/yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.514 DEBUG:boto:Method: GET 2024-04-16T19:33:49.514 DEBUG:boto:Path: /yrjefh-15/?prefix=dummy 2024-04-16T19:33:49.514 DEBUG:boto:Data: 2024-04-16T19:33:49.514 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.515 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.515 DEBUG:boto:Params: {} 2024-04-16T19:33:49.515 DEBUG:boto:Token: None 2024-04-16T19:33:49.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d8c26a0c247ed7f-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.517 DEBUG:boto:b'yrjefh-15dummy1000falsedummy2024-04-16T19:33:13.176Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.518 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.518 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.518 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.518 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.518 DEBUG:boto:Method: GET 2024-04-16T19:33:49.518 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.519 DEBUG:boto:Data: 2024-04-16T19:33:49.519 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.519 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.519 DEBUG:boto:Params: {} 2024-04-16T19:33:49.519 DEBUG:boto:Token: None 2024-04-16T19:33:49.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000983a736e69f95121-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.522 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.522 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.522 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.522 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.522 DEBUG:boto:Method: GET 2024-04-16T19:33:49.522 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.522 DEBUG:boto:Data: 2024-04-16T19:33:49.523 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.523 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.523 DEBUG:boto:Params: {} 2024-04-16T19:33:49.523 DEBUG:boto:Token: None 2024-04-16T19:33:49.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000086391526025839a-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.526 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.526 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.526 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.526 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.526 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.526 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.526 DEBUG:boto:Data: 2024-04-16T19:33:49.526 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.527 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.527 DEBUG:boto:Params: {} 2024-04-16T19:33:49.527 DEBUG:boto:Token: None 2024-04-16T19:33:49.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNcvBi/dQ6Mybm/OqatM3Fw5O+k= 2024-04-16T19:33:49.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNcvBi/dQ6Mybm/OqatM3Fw5O+k=', 'Content-Length': '0'} 2024-04-16T19:33:49.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b5dcbad88585f06-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.530 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.530 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.530 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.530 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.530 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.530 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.530 DEBUG:boto:Data: 2024-04-16T19:33:49.530 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.531 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.531 DEBUG:boto:Params: {} 2024-04-16T19:33:49.531 DEBUG:boto:Token: None 2024-04-16T19:33:49.531 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNcvBi/dQ6Mybm/OqatM3Fw5O+k= 2024-04-16T19:33:49.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNcvBi/dQ6Mybm/OqatM3Fw5O+k=', 'Content-Length': '0'} 2024-04-16T19:33:49.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1e7a13f947b723d-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.534 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.534 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.534 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.534 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.534 DEBUG:boto:Method: GET 2024-04-16T19:33:49.534 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.534 DEBUG:boto:Data: 2024-04-16T19:33:49.534 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.534 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.535 DEBUG:boto:Params: {} 2024-04-16T19:33:49.535 DEBUG:boto:Token: None 2024-04-16T19:33:49.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f14044fb9fe46ccb-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.537 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.538 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.538 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.538 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.538 DEBUG:boto:Method: GET 2024-04-16T19:33:49.538 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.538 DEBUG:boto:Data: 2024-04-16T19:33:49.538 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.538 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.538 DEBUG:boto:Params: {} 2024-04-16T19:33:49.539 DEBUG:boto:Token: None 2024-04-16T19:33:49.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000752a8c9ba441f7df-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.582 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.583 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.583 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.583 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.583 DEBUG:boto:Method: GET 2024-04-16T19:33:49.583 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.583 DEBUG:boto:Data: 2024-04-16T19:33:49.583 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.583 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.583 DEBUG:boto:Params: {} 2024-04-16T19:33:49.583 DEBUG:boto:Token: None 2024-04-16T19:33:49.584 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006125068b070eb111-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.587 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.587 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.587 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.587 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.588 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.588 DEBUG:boto:Method: GET 2024-04-16T19:33:49.588 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.588 DEBUG:boto:Data: 2024-04-16T19:33:49.588 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.588 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.588 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.588 DEBUG:boto:Params: {} 2024-04-16T19:33:49.588 DEBUG:boto:Token: None 2024-04-16T19:33:49.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c4c4dba79d5a88e-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.634 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.635 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.635 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.635 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.635 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:49.635 DEBUG:boto:Data: 2024-04-16T19:33:49.635 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.635 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.635 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.635 DEBUG:boto:Params: {} 2024-04-16T19:33:49.635 DEBUG:boto:Token: None 2024-04-16T19:33:49.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj 2024-04-16T19:33:49.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc= 2024-04-16T19:33:49.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc=', 'Content-Length': '0'} 2024-04-16T19:33:49.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000b172cf810ba0d7-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.640 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.640 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.641 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.641 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:49.641 DEBUG:boto:Data: 2024-04-16T19:33:49.641 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.641 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.641 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.641 DEBUG:boto:Params: {} 2024-04-16T19:33:49.641 DEBUG:boto:Token: None 2024-04-16T19:33:49.641 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj 2024-04-16T19:33:49.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc= 2024-04-16T19:33:49.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc=', 'Content-Length': '0'} 2024-04-16T19:33:49.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000803b456365a606b0-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.646 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.646 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.646 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.646 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.646 DEBUG:boto:Method: GET 2024-04-16T19:33:49.646 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.646 DEBUG:boto:Data: 2024-04-16T19:33:49.646 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.646 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.647 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.647 DEBUG:boto:Params: {} 2024-04-16T19:33:49.647 DEBUG:boto:Token: None 2024-04-16T19:33:49.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f1898d57351bf59-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.690 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.690 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.691 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.691 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.691 DEBUG:boto:Method: GET 2024-04-16T19:33:49.691 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.691 DEBUG:boto:Data: 2024-04-16T19:33:49.691 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.691 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.691 DEBUG:boto:Params: {} 2024-04-16T19:33:49.691 DEBUG:boto:Token: None 2024-04-16T19:33:49.691 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066c85f03be8bf32e-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.734 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.735 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.735 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.735 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.735 DEBUG:boto:Method: GET 2024-04-16T19:33:49.735 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.735 DEBUG:boto:Data: 2024-04-16T19:33:49.735 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.735 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.735 DEBUG:boto:Params: {} 2024-04-16T19:33:49.736 DEBUG:boto:Token: None 2024-04-16T19:33:49.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b0c4911faf1a22f-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.739 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.739 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.739 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.739 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.740 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.740 DEBUG:boto:Method: GET 2024-04-16T19:33:49.740 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.740 DEBUG:boto:Data: 2024-04-16T19:33:49.740 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.740 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.740 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.740 DEBUG:boto:Params: {} 2024-04-16T19:33:49.740 DEBUG:boto:Token: None 2024-04-16T19:33:49.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001732bd3d2b8185b9-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.786 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.787 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.787 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.787 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.787 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.787 DEBUG:boto:Method: GET 2024-04-16T19:33:49.787 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.788 DEBUG:boto:Data: 2024-04-16T19:33:49.788 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.788 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.788 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.788 DEBUG:boto:Params: {} 2024-04-16T19:33:49.788 DEBUG:boto:Token: None 2024-04-16T19:33:49.788 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8= 2024-04-16T19:33:49.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8=', 'Content-Length': '0'} 2024-04-16T19:33:49.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f0f198e9472a661-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.834 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.835 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.835 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.835 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.835 DEBUG:boto:Method: GET 2024-04-16T19:33:49.835 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.835 DEBUG:boto:Data: 2024-04-16T19:33:49.835 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.835 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.835 DEBUG:boto:Params: {} 2024-04-16T19:33:49.836 DEBUG:boto:Token: None 2024-04-16T19:33:49.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8= 2024-04-16T19:33:49.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8=', 'Content-Length': '0'} 2024-04-16T19:33:49.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009de662e109346731-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.839 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.839 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.839 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.839 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.839 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.839 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.839 DEBUG:boto:Data: 2024-04-16T19:33:49.839 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.840 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.840 DEBUG:boto:Params: {} 2024-04-16T19:33:49.840 DEBUG:boto:Token: None 2024-04-16T19:33:49.840 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gl4YgSPJ4XY0yiGR5up/9+wAqjE= 2024-04-16T19:33:49.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gl4YgSPJ4XY0yiGR5up/9+wAqjE=', 'Content-Length': '0'} 2024-04-16T19:33:49.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064cde3a8ff04f97e-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.843 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.843 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.843 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.843 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.843 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.843 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.843 DEBUG:boto:Data: 2024-04-16T19:33:49.843 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.844 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.844 DEBUG:boto:Params: {} 2024-04-16T19:33:49.844 DEBUG:boto:Token: None 2024-04-16T19:33:49.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gl4YgSPJ4XY0yiGR5up/9+wAqjE= 2024-04-16T19:33:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gl4YgSPJ4XY0yiGR5up/9+wAqjE=', 'Content-Length': '0'} 2024-04-16T19:33:49.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000befcfabea973094e-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.847 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.847 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.847 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.847 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.847 DEBUG:boto:Method: GET 2024-04-16T19:33:49.847 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.847 DEBUG:boto:Data: 2024-04-16T19:33:49.847 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.848 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.848 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.848 DEBUG:boto:Params: {} 2024-04-16T19:33:49.848 DEBUG:boto:Token: None 2024-04-16T19:33:49.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8= 2024-04-16T19:33:49.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8=', 'Content-Length': '0'} 2024-04-16T19:33:49.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3218178ecf31ce8-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.894 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.895 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.895 DEBUG:boto:path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.895 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.895 DEBUG:boto:Method: GET 2024-04-16T19:33:49.895 DEBUG:boto:Path: /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.895 DEBUG:boto:Data: 2024-04-16T19:33:49.895 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.895 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.895 DEBUG:boto:Params: {} 2024-04-16T19:33:49.896 DEBUG:boto:Token: None 2024-04-16T19:33:49.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt 2024-04-16T19:33:49.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8= 2024-04-16T19:33:49.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W56R050xivb6EMvSpejabLOFW+8=', 'Content-Length': '0'} 2024-04-16T19:33:49.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'HrsL5oz7VPiuzzM-jPvC7pNxy3XcWTt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053772f759dbe34ac-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.899 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.899 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.899 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.899 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.899 DEBUG:boto:Method: GET 2024-04-16T19:33:49.900 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.900 DEBUG:boto:Data: 2024-04-16T19:33:49.900 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.900 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.900 DEBUG:boto:Params: {} 2024-04-16T19:33:49.900 DEBUG:boto:Token: None 2024-04-16T19:33:49.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001267bf1c68c92844-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.946 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.947 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:49.947 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:49.947 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.947 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:49.947 DEBUG:boto:Method: GET 2024-04-16T19:33:49.947 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:49.947 DEBUG:boto:Data: 2024-04-16T19:33:49.948 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.948 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.948 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.948 DEBUG:boto:Params: {} 2024-04-16T19:33:49.948 DEBUG:boto:Token: None 2024-04-16T19:33:49.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/ 2024-04-16T19:33:49.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk= 2024-04-16T19:33:49.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Tr3By/vrbSGljVSA9gk75QNDdk=', 'Content-Length': '0'} 2024-04-16T19:33:49.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cba2553aeea3c67-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.951 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:49.951 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.951 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.951 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.951 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:49.952 DEBUG:boto:Data: 2024-04-16T19:33:49.952 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.952 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.952 DEBUG:boto:Params: {} 2024-04-16T19:33:49.952 DEBUG:boto:Token: None 2024-04-16T19:33:49.952 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj 2024-04-16T19:33:49.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc= 2024-04-16T19:33:49.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc=', 'Content-Length': '0'} 2024-04-16T19:33:49.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000037437c3f821afd9-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.957 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.957 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.957 DEBUG:boto:Method: HEAD 2024-04-16T19:33:49.957 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:49.957 DEBUG:boto:Data: 2024-04-16T19:33:49.957 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.958 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.958 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.958 DEBUG:boto:Params: {} 2024-04-16T19:33:49.958 DEBUG:boto:Token: None 2024-04-16T19:33:49.958 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj 2024-04-16T19:33:49.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc= 2024-04-16T19:33:49.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sEtHzdUw3q4oxA3OI3s7s6FcPzc=', 'Content-Length': '0'} 2024-04-16T19:33:49.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a52350f84ea70643-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:49.963 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.963 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.963 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.963 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.963 DEBUG:boto:Method: GET 2024-04-16T19:33:49.963 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.963 DEBUG:boto:Data: 2024-04-16T19:33:49.963 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.963 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:49.963 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.964 DEBUG:boto:Params: {} 2024-04-16T19:33:49.964 DEBUG:boto:Token: None 2024-04-16T19:33:49.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbc4964f7fc182fb-00661ed29d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:49.966 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:49.966 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:49.967 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.967 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.967 DEBUG:boto:Method: GET 2024-04-16T19:33:49.967 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.967 DEBUG:boto:Data: 2024-04-16T19:33:49.967 DEBUG:boto:Headers: {} 2024-04-16T19:33:49.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:49.967 DEBUG:boto:Port: 8000 2024-04-16T19:33:49.967 DEBUG:boto:Params: {} 2024-04-16T19:33:49.968 DEBUG:boto:Token: None 2024-04-16T19:33:49.968 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:49 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:49.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ= 2024-04-16T19:33:49.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOZUEn9JJ/7RZXpBWJ25CqHccwQ=', 'Content-Length': '0'} 2024-04-16T19:33:49.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048992e3c5ef1a40d-00661ed29d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.010 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.011 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.011 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.011 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.011 DEBUG:boto:Method: GET 2024-04-16T19:33:50.011 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.011 DEBUG:boto:Data: 2024-04-16T19:33:50.011 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.011 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.012 DEBUG:boto:Params: {} 2024-04-16T19:33:50.012 DEBUG:boto:Token: None 2024-04-16T19:33:50.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003274b5294e71518c-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.015 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.015 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.015 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.016 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.016 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.016 DEBUG:boto:Method: GET 2024-04-16T19:33:50.016 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.016 DEBUG:boto:Data: 2024-04-16T19:33:50.016 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.016 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.016 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.016 DEBUG:boto:Params: {} 2024-04-16T19:33:50.017 DEBUG:boto:Token: None 2024-04-16T19:33:50.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096b72a7b4bb1e056-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.062 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.063 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.063 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.063 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.063 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.063 DEBUG:boto:Method: GET 2024-04-16T19:33:50.063 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.064 DEBUG:boto:Data: 2024-04-16T19:33:50.064 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.064 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.064 DEBUG:boto:Params: {} 2024-04-16T19:33:50.064 DEBUG:boto:Token: None 2024-04-16T19:33:50.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI= 2024-04-16T19:33:50.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI=', 'Content-Length': '0'} 2024-04-16T19:33:50.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa0be728debe9f78-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.111 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.111 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.111 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.111 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.111 DEBUG:boto:Method: GET 2024-04-16T19:33:50.111 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.111 DEBUG:boto:Data: 2024-04-16T19:33:50.111 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.112 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.112 DEBUG:boto:Params: {} 2024-04-16T19:33:50.112 DEBUG:boto:Token: None 2024-04-16T19:33:50.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI= 2024-04-16T19:33:50.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI=', 'Content-Length': '0'} 2024-04-16T19:33:50.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f135349c7351144a-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.115 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.115 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.115 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.115 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.115 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.115 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.116 DEBUG:boto:Data: 2024-04-16T19:33:50.116 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.116 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.116 DEBUG:boto:Params: {} 2024-04-16T19:33:50.116 DEBUG:boto:Token: None 2024-04-16T19:33:50.116 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ee1uOn9ua0C+RTAysbxNOdQ3to0= 2024-04-16T19:33:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ee1uOn9ua0C+RTAysbxNOdQ3to0=', 'Content-Length': '0'} 2024-04-16T19:33:50.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087aef61e4086d563-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.119 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.119 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.119 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.119 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.119 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.119 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.119 DEBUG:boto:Data: 2024-04-16T19:33:50.119 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.120 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.120 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.120 DEBUG:boto:Params: {} 2024-04-16T19:33:50.120 DEBUG:boto:Token: None 2024-04-16T19:33:50.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ee1uOn9ua0C+RTAysbxNOdQ3to0= 2024-04-16T19:33:50.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ee1uOn9ua0C+RTAysbxNOdQ3to0=', 'Content-Length': '0'} 2024-04-16T19:33:50.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c0e2ee15e21920-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.123 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.123 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.123 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.123 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.123 DEBUG:boto:Method: GET 2024-04-16T19:33:50.123 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.123 DEBUG:boto:Data: 2024-04-16T19:33:50.123 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.124 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.124 DEBUG:boto:Params: {} 2024-04-16T19:33:50.124 DEBUG:boto:Token: None 2024-04-16T19:33:50.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI= 2024-04-16T19:33:50.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI=', 'Content-Length': '0'} 2024-04-16T19:33:50.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000273fcd0efe609ee4-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.170 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.171 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.171 DEBUG:boto:path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.171 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.171 DEBUG:boto:Method: GET 2024-04-16T19:33:50.171 DEBUG:boto:Path: /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.171 DEBUG:boto:Data: 2024-04-16T19:33:50.171 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.171 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.171 DEBUG:boto:Params: {} 2024-04-16T19:33:50.172 DEBUG:boto:Token: None 2024-04-16T19:33:50.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu 2024-04-16T19:33:50.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI= 2024-04-16T19:33:50.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PSfeBjWy+cXqJI6KzzNI913B7vI=', 'Content-Length': '0'} 2024-04-16T19:33:50.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '9NhIdfxEbvHx5ygmguvd1.Ud7NXgUnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5f489d39b3acfa5-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.174 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.174 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.175 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.175 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.175 DEBUG:boto:Method: GET 2024-04-16T19:33:50.175 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.175 DEBUG:boto:Data: 2024-04-16T19:33:50.175 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.175 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.175 DEBUG:boto:Params: {} 2024-04-16T19:33:50.175 DEBUG:boto:Token: None 2024-04-16T19:33:50.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004247464ea195c8c5-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.179 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.179 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.179 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.180 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.180 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.180 DEBUG:boto:Method: GET 2024-04-16T19:33:50.180 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.180 DEBUG:boto:Data: 2024-04-16T19:33:50.180 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.180 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.180 DEBUG:boto:Params: {} 2024-04-16T19:33:50.181 DEBUG:boto:Token: None 2024-04-16T19:33:50.181 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000146834df392dfdb0-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.184 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.184 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.184 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.185 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.185 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.185 DEBUG:boto:Data: 2024-04-16T19:33:50.185 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.185 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.185 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.185 DEBUG:boto:Params: {} 2024-04-16T19:33:50.185 DEBUG:boto:Token: None 2024-04-16T19:33:50.185 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.186 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093f1b01870fbd3e0-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.190 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.190 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.190 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.190 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.190 DEBUG:boto:Data: 2024-04-16T19:33:50.191 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.191 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.191 DEBUG:boto:Params: {} 2024-04-16T19:33:50.191 DEBUG:boto:Token: None 2024-04-16T19:33:50.191 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c914d237c7bd27d-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.196 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.196 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.196 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.196 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.196 DEBUG:boto:Method: GET 2024-04-16T19:33:50.196 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.197 DEBUG:boto:Data: 2024-04-16T19:33:50.197 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.197 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.197 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.197 DEBUG:boto:Params: {} 2024-04-16T19:33:50.197 DEBUG:boto:Token: None 2024-04-16T19:33:50.197 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a6036db1be3f3b8-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.200 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.200 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.200 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.200 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.200 DEBUG:boto:Method: GET 2024-04-16T19:33:50.201 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.201 DEBUG:boto:Data: 2024-04-16T19:33:50.201 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.201 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.201 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.201 DEBUG:boto:Params: {} 2024-04-16T19:33:50.201 DEBUG:boto:Token: None 2024-04-16T19:33:50.201 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.202 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000348d1261612fede2-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.247 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.247 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.247 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.247 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.247 DEBUG:boto:Method: GET 2024-04-16T19:33:50.247 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.247 DEBUG:boto:Data: 2024-04-16T19:33:50.247 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.248 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.248 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.248 DEBUG:boto:Params: {} 2024-04-16T19:33:50.248 DEBUG:boto:Token: None 2024-04-16T19:33:50.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e0270d9ac74efeb-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.251 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.252 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.252 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.252 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.252 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.252 DEBUG:boto:Method: GET 2024-04-16T19:33:50.252 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.252 DEBUG:boto:Data: 2024-04-16T19:33:50.252 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.253 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.253 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.253 DEBUG:boto:Params: {} 2024-04-16T19:33:50.253 DEBUG:boto:Token: None 2024-04-16T19:33:50.253 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.253 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc58f84e98df133-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.256 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.257 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.257 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.257 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.257 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.257 DEBUG:boto:Method: GET 2024-04-16T19:33:50.258 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.258 DEBUG:boto:Data: 2024-04-16T19:33:50.258 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.258 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.258 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.258 DEBUG:boto:Params: {} 2024-04-16T19:33:50.258 DEBUG:boto:Token: None 2024-04-16T19:33:50.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A= 2024-04-16T19:33:50.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A=', 'Content-Length': '0'} 2024-04-16T19:33:50.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfa40addf6fe23a5-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.302 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.303 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.303 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.303 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.303 DEBUG:boto:Method: GET 2024-04-16T19:33:50.303 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.303 DEBUG:boto:Data: 2024-04-16T19:33:50.303 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.304 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.304 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.304 DEBUG:boto:Params: {} 2024-04-16T19:33:50.304 DEBUG:boto:Token: None 2024-04-16T19:33:50.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A= 2024-04-16T19:33:50.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A=', 'Content-Length': '0'} 2024-04-16T19:33:50.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c58cc4092304a0b9-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.351 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.351 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.351 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.351 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.351 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.351 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.351 DEBUG:boto:Data: 2024-04-16T19:33:50.351 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.352 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.352 DEBUG:boto:Params: {} 2024-04-16T19:33:50.352 DEBUG:boto:Token: None 2024-04-16T19:33:50.352 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beWDM6ptezePiHi3Ah/cBsm1wRs= 2024-04-16T19:33:50.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beWDM6ptezePiHi3Ah/cBsm1wRs=', 'Content-Length': '0'} 2024-04-16T19:33:50.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004251dc8f0f8838f5-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.355 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.355 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.355 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.355 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.355 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.355 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.355 DEBUG:boto:Data: 2024-04-16T19:33:50.355 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.356 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.356 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.356 DEBUG:boto:Params: {} 2024-04-16T19:33:50.356 DEBUG:boto:Token: None 2024-04-16T19:33:50.356 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beWDM6ptezePiHi3Ah/cBsm1wRs= 2024-04-16T19:33:50.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beWDM6ptezePiHi3Ah/cBsm1wRs=', 'Content-Length': '0'} 2024-04-16T19:33:50.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003929cfc36b67d6f1-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.359 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.359 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.359 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.359 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.359 DEBUG:boto:Method: GET 2024-04-16T19:33:50.359 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.360 DEBUG:boto:Data: 2024-04-16T19:33:50.360 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.360 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.360 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.360 DEBUG:boto:Params: {} 2024-04-16T19:33:50.360 DEBUG:boto:Token: None 2024-04-16T19:33:50.361 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.361 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A= 2024-04-16T19:33:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A=', 'Content-Length': '0'} 2024-04-16T19:33:50.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f6def08371881c9-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.363 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.363 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.363 DEBUG:boto:path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.363 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.364 DEBUG:boto:Method: GET 2024-04-16T19:33:50.364 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.364 DEBUG:boto:Data: 2024-04-16T19:33:50.364 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.364 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.364 DEBUG:boto:Params: {} 2024-04-16T19:33:50.364 DEBUG:boto:Token: None 2024-04-16T19:33:50.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=JF3QhPh4rjNtlWbfOOCOAgIBw58tClu 2024-04-16T19:33:50.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A= 2024-04-16T19:33:50.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kDNlSR955vEct0fBuRF9QC0kp1A=', 'Content-Length': '0'} 2024-04-16T19:33:50.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'JF3QhPh4rjNtlWbfOOCOAgIBw58tClu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042a31449b686c4b1-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.410 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.411 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.411 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.411 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.411 DEBUG:boto:Method: GET 2024-04-16T19:33:50.411 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.411 DEBUG:boto:Data: 2024-04-16T19:33:50.411 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.412 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.412 DEBUG:boto:Params: {} 2024-04-16T19:33:50.412 DEBUG:boto:Token: None 2024-04-16T19:33:50.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7257a807e87e348-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.415 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.416 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.416 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.416 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.416 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.416 DEBUG:boto:Method: GET 2024-04-16T19:33:50.416 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.416 DEBUG:boto:Data: 2024-04-16T19:33:50.416 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.416 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.417 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.417 DEBUG:boto:Params: {} 2024-04-16T19:33:50.417 DEBUG:boto:Token: None 2024-04-16T19:33:50.417 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f200c9415b3a7bcd-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.420 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.420 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.421 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.421 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.421 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.421 DEBUG:boto:Data: 2024-04-16T19:33:50.421 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.421 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.421 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.422 DEBUG:boto:Params: {} 2024-04-16T19:33:50.422 DEBUG:boto:Token: None 2024-04-16T19:33:50.422 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.422 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009223766ce3dc63f3-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.427 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.427 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.427 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.427 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.427 DEBUG:boto:Data: 2024-04-16T19:33:50.427 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.427 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.427 DEBUG:boto:Params: {} 2024-04-16T19:33:50.428 DEBUG:boto:Token: None 2024-04-16T19:33:50.428 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d7cdf664c6b5d35-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.433 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.433 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.433 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.433 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.433 DEBUG:boto:Method: GET 2024-04-16T19:33:50.434 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.434 DEBUG:boto:Data: 2024-04-16T19:33:50.434 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.434 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.434 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.434 DEBUG:boto:Params: {} 2024-04-16T19:33:50.434 DEBUG:boto:Token: None 2024-04-16T19:33:50.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce7fad40069338cd-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.437 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.438 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.438 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.438 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.438 DEBUG:boto:Method: GET 2024-04-16T19:33:50.438 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.438 DEBUG:boto:Data: 2024-04-16T19:33:50.438 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.438 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.438 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.439 DEBUG:boto:Params: {} 2024-04-16T19:33:50.439 DEBUG:boto:Token: None 2024-04-16T19:33:50.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021c08995c80feba1-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.482 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.483 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.483 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.483 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.483 DEBUG:boto:Method: GET 2024-04-16T19:33:50.483 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.483 DEBUG:boto:Data: 2024-04-16T19:33:50.483 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.484 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.484 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.484 DEBUG:boto:Params: {} 2024-04-16T19:33:50.484 DEBUG:boto:Token: None 2024-04-16T19:33:50.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210daca10d4aeebd-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.487 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.488 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.488 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.488 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.488 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.489 DEBUG:boto:Method: GET 2024-04-16T19:33:50.489 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.489 DEBUG:boto:Data: 2024-04-16T19:33:50.489 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.489 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.489 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.489 DEBUG:boto:Params: {} 2024-04-16T19:33:50.489 DEBUG:boto:Token: None 2024-04-16T19:33:50.489 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.490 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e579a48394623a4-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.493 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.493 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.493 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.493 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.494 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.494 DEBUG:boto:Method: GET 2024-04-16T19:33:50.494 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.494 DEBUG:boto:Data: 2024-04-16T19:33:50.494 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.494 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.495 DEBUG:boto:Params: {} 2024-04-16T19:33:50.495 DEBUG:boto:Token: None 2024-04-16T19:33:50.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo= 2024-04-16T19:33:50.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo=', 'Content-Length': '0'} 2024-04-16T19:33:50.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046be8a8e73fc47da-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.539 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.539 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.539 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.539 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.539 DEBUG:boto:Method: GET 2024-04-16T19:33:50.539 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.540 DEBUG:boto:Data: 2024-04-16T19:33:50.540 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.540 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.540 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.540 DEBUG:boto:Params: {} 2024-04-16T19:33:50.540 DEBUG:boto:Token: None 2024-04-16T19:33:50.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo= 2024-04-16T19:33:50.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo=', 'Content-Length': '0'} 2024-04-16T19:33:50.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000832a37c204beb629-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.543 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.543 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.543 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.543 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.543 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.543 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.544 DEBUG:boto:Data: 2024-04-16T19:33:50.544 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.544 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.544 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.544 DEBUG:boto:Params: {} 2024-04-16T19:33:50.544 DEBUG:boto:Token: None 2024-04-16T19:33:50.544 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HoCQtztvlneM5iyB497k0yTMyf0= 2024-04-16T19:33:50.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HoCQtztvlneM5iyB497k0yTMyf0=', 'Content-Length': '0'} 2024-04-16T19:33:50.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084b164daa1e54a4f-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.547 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.547 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.547 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.547 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.547 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.548 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.548 DEBUG:boto:Data: 2024-04-16T19:33:50.548 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.548 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.548 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.548 DEBUG:boto:Params: {} 2024-04-16T19:33:50.548 DEBUG:boto:Token: None 2024-04-16T19:33:50.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HoCQtztvlneM5iyB497k0yTMyf0= 2024-04-16T19:33:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HoCQtztvlneM5iyB497k0yTMyf0=', 'Content-Length': '0'} 2024-04-16T19:33:50.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ebf25ae5389e330-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.551 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.552 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.552 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.552 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.552 DEBUG:boto:Method: GET 2024-04-16T19:33:50.552 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.552 DEBUG:boto:Data: 2024-04-16T19:33:50.552 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.552 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.552 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.552 DEBUG:boto:Params: {} 2024-04-16T19:33:50.553 DEBUG:boto:Token: None 2024-04-16T19:33:50.553 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo= 2024-04-16T19:33:50.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo=', 'Content-Length': '0'} 2024-04-16T19:33:50.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704cfdb35c15ebd2-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.598 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.599 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.599 DEBUG:boto:path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.599 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.599 DEBUG:boto:Method: GET 2024-04-16T19:33:50.599 DEBUG:boto:Path: /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.599 DEBUG:boto:Data: 2024-04-16T19:33:50.599 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.599 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.600 DEBUG:boto:Params: {} 2024-04-16T19:33:50.600 DEBUG:boto:Token: None 2024-04-16T19:33:50.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=BE5KRw60iPLiIC0pRor7iVGgCFHPOMX 2024-04-16T19:33:50.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo= 2024-04-16T19:33:50.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3QQf4QbY2ic/snQOhH2DUtKtBKo=', 'Content-Length': '0'} 2024-04-16T19:33:50.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'BE5KRw60iPLiIC0pRor7iVGgCFHPOMX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb5d7b501f22f636-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.603 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.603 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.603 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.603 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.603 DEBUG:boto:Method: GET 2024-04-16T19:33:50.603 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.604 DEBUG:boto:Data: 2024-04-16T19:33:50.604 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.604 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.604 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.604 DEBUG:boto:Params: {} 2024-04-16T19:33:50.604 DEBUG:boto:Token: None 2024-04-16T19:33:50.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef26c086f5bc8377-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.608 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.608 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.608 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.608 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.608 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.609 DEBUG:boto:Method: GET 2024-04-16T19:33:50.609 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.609 DEBUG:boto:Data: 2024-04-16T19:33:50.609 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.609 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.609 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.609 DEBUG:boto:Params: {} 2024-04-16T19:33:50.609 DEBUG:boto:Token: None 2024-04-16T19:33:50.610 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.610 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d4c0f1243ddf56-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.654 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.655 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.655 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.655 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.655 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.655 DEBUG:boto:Data: 2024-04-16T19:33:50.656 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.656 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.656 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.656 DEBUG:boto:Params: {} 2024-04-16T19:33:50.656 DEBUG:boto:Token: None 2024-04-16T19:33:50.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1f8dd794b47f55d-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.662 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.662 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.662 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.662 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.662 DEBUG:boto:Data: 2024-04-16T19:33:50.663 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.663 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.663 DEBUG:boto:Params: {} 2024-04-16T19:33:50.663 DEBUG:boto:Token: None 2024-04-16T19:33:50.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ede4d1405b3c444c-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.670 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.671 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.671 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.671 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.671 DEBUG:boto:Method: GET 2024-04-16T19:33:50.671 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.671 DEBUG:boto:Data: 2024-04-16T19:33:50.671 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.672 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.672 DEBUG:boto:Params: {} 2024-04-16T19:33:50.672 DEBUG:boto:Token: None 2024-04-16T19:33:50.672 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b5bd5eac7052fc5-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.675 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.675 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.676 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.676 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.676 DEBUG:boto:Method: GET 2024-04-16T19:33:50.676 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.676 DEBUG:boto:Data: 2024-04-16T19:33:50.676 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.676 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.676 DEBUG:boto:Params: {} 2024-04-16T19:33:50.677 DEBUG:boto:Token: None 2024-04-16T19:33:50.677 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f9a4c18b78c434c-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.722 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.723 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.723 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.723 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.723 DEBUG:boto:Method: GET 2024-04-16T19:33:50.723 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.723 DEBUG:boto:Data: 2024-04-16T19:33:50.723 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.724 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.724 DEBUG:boto:Params: {} 2024-04-16T19:33:50.724 DEBUG:boto:Token: None 2024-04-16T19:33:50.724 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce6a2193f9efbb3-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.727 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.728 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.728 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.728 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.728 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.728 DEBUG:boto:Method: GET 2024-04-16T19:33:50.728 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.729 DEBUG:boto:Data: 2024-04-16T19:33:50.729 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.729 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.729 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.729 DEBUG:boto:Params: {} 2024-04-16T19:33:50.729 DEBUG:boto:Token: None 2024-04-16T19:33:50.729 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d684e26a051abb-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.778 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.779 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.779 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.779 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.779 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.780 DEBUG:boto:Method: GET 2024-04-16T19:33:50.780 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.780 DEBUG:boto:Data: 2024-04-16T19:33:50.780 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.780 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.780 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.780 DEBUG:boto:Params: {} 2024-04-16T19:33:50.780 DEBUG:boto:Token: None 2024-04-16T19:33:50.781 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o= 2024-04-16T19:33:50.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o=', 'Content-Length': '0'} 2024-04-16T19:33:50.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e495e7c848f4146a-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.784 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.784 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.784 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.784 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.784 DEBUG:boto:Method: GET 2024-04-16T19:33:50.785 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.785 DEBUG:boto:Data: 2024-04-16T19:33:50.785 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.785 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.785 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.785 DEBUG:boto:Params: {} 2024-04-16T19:33:50.785 DEBUG:boto:Token: None 2024-04-16T19:33:50.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.786 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o= 2024-04-16T19:33:50.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o=', 'Content-Length': '0'} 2024-04-16T19:33:50.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edad232b4d0f3301-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.830 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.831 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.831 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.831 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.831 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.831 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.831 DEBUG:boto:Data: 2024-04-16T19:33:50.831 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.832 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.832 DEBUG:boto:Params: {} 2024-04-16T19:33:50.832 DEBUG:boto:Token: None 2024-04-16T19:33:50.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5mxC8M0XxHY8el0onMbd0XjBGgY= 2024-04-16T19:33:50.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5mxC8M0XxHY8el0onMbd0XjBGgY=', 'Content-Length': '0'} 2024-04-16T19:33:50.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052eb37df5596e0c1-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.835 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.835 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.835 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.835 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.835 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.836 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.836 DEBUG:boto:Data: 2024-04-16T19:33:50.836 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.836 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.836 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.836 DEBUG:boto:Params: {} 2024-04-16T19:33:50.837 DEBUG:boto:Token: None 2024-04-16T19:33:50.837 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.837 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5mxC8M0XxHY8el0onMbd0XjBGgY= 2024-04-16T19:33:50.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5mxC8M0XxHY8el0onMbd0XjBGgY=', 'Content-Length': '0'} 2024-04-16T19:33:50.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c57319de7985b0d6-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.840 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.840 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.840 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.840 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.840 DEBUG:boto:Method: GET 2024-04-16T19:33:50.840 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.841 DEBUG:boto:Data: 2024-04-16T19:33:50.841 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.841 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.841 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.841 DEBUG:boto:Params: {} 2024-04-16T19:33:50.841 DEBUG:boto:Token: None 2024-04-16T19:33:50.841 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o= 2024-04-16T19:33:50.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o=', 'Content-Length': '0'} 2024-04-16T19:33:50.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000131df438daa218bc-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.887 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.887 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.887 DEBUG:boto:path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.887 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.887 DEBUG:boto:Method: GET 2024-04-16T19:33:50.887 DEBUG:boto:Path: /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.887 DEBUG:boto:Data: 2024-04-16T19:33:50.887 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.888 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.888 DEBUG:boto:Params: {} 2024-04-16T19:33:50.888 DEBUG:boto:Token: None 2024-04-16T19:33:50.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G 2024-04-16T19:33:50.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o= 2024-04-16T19:33:50.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WuVivuq955HyV3YSeaU4qtATM2o=', 'Content-Length': '0'} 2024-04-16T19:33:50.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'z5NEQlCe9sj9xqcXfSR.CdpxO0XYz-G'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c50f0d3c2a22e1f6-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.935 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.935 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.935 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.935 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.935 DEBUG:boto:Method: GET 2024-04-16T19:33:50.935 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.935 DEBUG:boto:Data: 2024-04-16T19:33:50.936 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.936 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.936 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.936 DEBUG:boto:Params: {} 2024-04-16T19:33:50.936 DEBUG:boto:Token: None 2024-04-16T19:33:50.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000410c42ba13b1c115-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.940 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.940 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:50.940 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:50.940 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.940 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:50.941 DEBUG:boto:Method: GET 2024-04-16T19:33:50.941 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:50.941 DEBUG:boto:Data: 2024-04-16T19:33:50.941 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.941 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.941 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.941 DEBUG:boto:Params: {} 2024-04-16T19:33:50.941 DEBUG:boto:Token: None 2024-04-16T19:33:50.942 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/ 2024-04-16T19:33:50.942 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY= 2024-04-16T19:33:50.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2beS7A6S2T5k9oBkdBLQZ7nytKY=', 'Content-Length': '0'} 2024-04-16T19:33:50.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260b77665d6a5fcd-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.945 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:50.945 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.945 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.946 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.946 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.946 DEBUG:boto:Data: 2024-04-16T19:33:50.946 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.946 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.946 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.946 DEBUG:boto:Params: {} 2024-04-16T19:33:50.946 DEBUG:boto:Token: None 2024-04-16T19:33:50.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa8d5daca3cf157e-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.952 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.952 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.952 DEBUG:boto:Method: HEAD 2024-04-16T19:33:50.952 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:50.952 DEBUG:boto:Data: 2024-04-16T19:33:50.953 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.953 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.953 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.953 DEBUG:boto:Params: {} 2024-04-16T19:33:50.953 DEBUG:boto:Token: None 2024-04-16T19:33:50.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj 2024-04-16T19:33:50.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ= 2024-04-16T19:33:50.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oeJYs5s4Uh2IfcHPEts22qs9JCQ=', 'Content-Length': '0'} 2024-04-16T19:33:50.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019488b08ee1e272f-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:50.959 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.959 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.959 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.959 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.959 DEBUG:boto:Method: GET 2024-04-16T19:33:50.959 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.959 DEBUG:boto:Data: 2024-04-16T19:33:50.959 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:50.960 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.960 DEBUG:boto:Params: {} 2024-04-16T19:33:50.960 DEBUG:boto:Token: None 2024-04-16T19:33:50.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b51317b51cc98ff-00661ed29e-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:50.963 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:50.963 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:50.964 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.964 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.964 DEBUG:boto:Method: GET 2024-04-16T19:33:50.964 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.964 DEBUG:boto:Data: 2024-04-16T19:33:50.964 DEBUG:boto:Headers: {} 2024-04-16T19:33:50.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:50.964 DEBUG:boto:Port: 8000 2024-04-16T19:33:50.965 DEBUG:boto:Params: {} 2024-04-16T19:33:50.965 DEBUG:boto:Token: None 2024-04-16T19:33:50.965 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:50 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:50.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0= 2024-04-16T19:33:50.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hg+p4ObGBx1IeDJwE/JVbt0xYW0=', 'Content-Length': '0'} 2024-04-16T19:33:50.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83be986e9d566c8-00661ed29e-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.010 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.011 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.011 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.011 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.011 DEBUG:boto:Method: GET 2024-04-16T19:33:51.011 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.011 DEBUG:boto:Data: 2024-04-16T19:33:51.011 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.012 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.012 DEBUG:boto:Params: {} 2024-04-16T19:33:51.012 DEBUG:boto:Token: None 2024-04-16T19:33:51.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605877321e92b585-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.015 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.016 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.016 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.016 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.016 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.016 DEBUG:boto:Method: GET 2024-04-16T19:33:51.017 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.017 DEBUG:boto:Data: 2024-04-16T19:33:51.017 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.017 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.017 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.017 DEBUG:boto:Params: {} 2024-04-16T19:33:51.017 DEBUG:boto:Token: None 2024-04-16T19:33:51.018 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb53440a4212f9f-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.062 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.063 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.063 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.063 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.063 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.063 DEBUG:boto:Method: GET 2024-04-16T19:33:51.064 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.064 DEBUG:boto:Data: 2024-04-16T19:33:51.064 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.064 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.064 DEBUG:boto:Params: {} 2024-04-16T19:33:51.064 DEBUG:boto:Token: None 2024-04-16T19:33:51.065 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8= 2024-04-16T19:33:51.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8=', 'Content-Length': '0'} 2024-04-16T19:33:51.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025291b1346bd35e7-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.111 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.111 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.111 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.111 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.111 DEBUG:boto:Method: GET 2024-04-16T19:33:51.111 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.111 DEBUG:boto:Data: 2024-04-16T19:33:51.111 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.112 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.112 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.112 DEBUG:boto:Params: {} 2024-04-16T19:33:51.112 DEBUG:boto:Token: None 2024-04-16T19:33:51.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8= 2024-04-16T19:33:51.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8=', 'Content-Length': '0'} 2024-04-16T19:33:51.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de858d6429f8c14a-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.115 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.115 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.115 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.115 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.115 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.116 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.116 DEBUG:boto:Data: 2024-04-16T19:33:51.116 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.116 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.116 DEBUG:boto:Params: {} 2024-04-16T19:33:51.116 DEBUG:boto:Token: None 2024-04-16T19:33:51.116 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RmKWlZunxsoLw/sw1EWCOu6z3ls= 2024-04-16T19:33:51.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RmKWlZunxsoLw/sw1EWCOu6z3ls=', 'Content-Length': '0'} 2024-04-16T19:33:51.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031a784f493764267-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.119 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.119 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.119 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.120 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.120 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.120 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.120 DEBUG:boto:Data: 2024-04-16T19:33:51.120 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.120 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.120 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.120 DEBUG:boto:Params: {} 2024-04-16T19:33:51.120 DEBUG:boto:Token: None 2024-04-16T19:33:51.121 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RmKWlZunxsoLw/sw1EWCOu6z3ls= 2024-04-16T19:33:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RmKWlZunxsoLw/sw1EWCOu6z3ls=', 'Content-Length': '0'} 2024-04-16T19:33:51.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8efe35436aebed0-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.123 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.123 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.124 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.124 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.124 DEBUG:boto:Method: GET 2024-04-16T19:33:51.124 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.124 DEBUG:boto:Data: 2024-04-16T19:33:51.124 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.124 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.124 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.124 DEBUG:boto:Params: {} 2024-04-16T19:33:51.124 DEBUG:boto:Token: None 2024-04-16T19:33:51.125 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.125 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8= 2024-04-16T19:33:51.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8=', 'Content-Length': '0'} 2024-04-16T19:33:51.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0ace525cb401585-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.174 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.175 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.175 DEBUG:boto:path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.175 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.175 DEBUG:boto:Method: GET 2024-04-16T19:33:51.175 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.175 DEBUG:boto:Data: 2024-04-16T19:33:51.175 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.175 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.176 DEBUG:boto:Params: {} 2024-04-16T19:33:51.176 DEBUG:boto:Token: None 2024-04-16T19:33:51.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s 2024-04-16T19:33:51.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8= 2024-04-16T19:33:51.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RZmD72IlmXRw8RMzI/K6lWay7o8=', 'Content-Length': '0'} 2024-04-16T19:33:51.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6ZyOishsjGIlC5qLBtS5tbhd6d0Ou8s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026ce021f22a3250f-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.178 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.179 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.179 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.179 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.179 DEBUG:boto:Method: GET 2024-04-16T19:33:51.179 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.179 DEBUG:boto:Data: 2024-04-16T19:33:51.179 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.180 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.180 DEBUG:boto:Params: {} 2024-04-16T19:33:51.180 DEBUG:boto:Token: None 2024-04-16T19:33:51.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8f575a6522ba4a9-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.183 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.183 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.184 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.184 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.184 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.184 DEBUG:boto:Method: GET 2024-04-16T19:33:51.184 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.184 DEBUG:boto:Data: 2024-04-16T19:33:51.184 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.184 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.184 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.185 DEBUG:boto:Params: {} 2024-04-16T19:33:51.185 DEBUG:boto:Token: None 2024-04-16T19:33:51.185 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.185 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd0a657396f4b69-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.187 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.188 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.188 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.188 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.188 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.188 DEBUG:boto:Data: 2024-04-16T19:33:51.188 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.188 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.189 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.189 DEBUG:boto:Params: {} 2024-04-16T19:33:51.189 DEBUG:boto:Token: None 2024-04-16T19:33:51.189 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e30a85f283998a9e-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.194 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.194 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.194 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.194 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.194 DEBUG:boto:Data: 2024-04-16T19:33:51.194 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.195 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.195 DEBUG:boto:Params: {} 2024-04-16T19:33:51.195 DEBUG:boto:Token: None 2024-04-16T19:33:51.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7acd5e4e96cbc60-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.200 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.200 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.200 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.200 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.200 DEBUG:boto:Method: GET 2024-04-16T19:33:51.200 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.200 DEBUG:boto:Data: 2024-04-16T19:33:51.200 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.200 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.200 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.201 DEBUG:boto:Params: {} 2024-04-16T19:33:51.201 DEBUG:boto:Token: None 2024-04-16T19:33:51.201 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.201 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdd937a42846650a-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.246 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.247 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.247 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.247 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.247 DEBUG:boto:Method: GET 2024-04-16T19:33:51.247 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.247 DEBUG:boto:Data: 2024-04-16T19:33:51.247 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.248 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.248 DEBUG:boto:Params: {} 2024-04-16T19:33:51.248 DEBUG:boto:Token: None 2024-04-16T19:33:51.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f1ed38325257e06-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.295 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.295 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.295 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.295 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.295 DEBUG:boto:Method: GET 2024-04-16T19:33:51.295 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.295 DEBUG:boto:Data: 2024-04-16T19:33:51.296 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.296 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.296 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.296 DEBUG:boto:Params: {} 2024-04-16T19:33:51.296 DEBUG:boto:Token: None 2024-04-16T19:33:51.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e270d5973207203-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.300 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.300 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.300 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.300 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.301 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.301 DEBUG:boto:Method: GET 2024-04-16T19:33:51.301 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.301 DEBUG:boto:Data: 2024-04-16T19:33:51.301 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.301 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.301 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.301 DEBUG:boto:Params: {} 2024-04-16T19:33:51.301 DEBUG:boto:Token: None 2024-04-16T19:33:51.302 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a222d4cab10f2b8-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.304 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.305 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.305 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.305 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.305 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.306 DEBUG:boto:Method: GET 2024-04-16T19:33:51.306 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.306 DEBUG:boto:Data: 2024-04-16T19:33:51.306 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.306 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.306 DEBUG:boto:Params: {} 2024-04-16T19:33:51.306 DEBUG:boto:Token: None 2024-04-16T19:33:51.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY= 2024-04-16T19:33:51.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY=', 'Content-Length': '0'} 2024-04-16T19:33:51.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004af7f47c8dc9b4fb-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.310 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.310 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.310 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.310 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.310 DEBUG:boto:Method: GET 2024-04-16T19:33:51.310 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.311 DEBUG:boto:Data: 2024-04-16T19:33:51.311 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.311 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.311 DEBUG:boto:Params: {} 2024-04-16T19:33:51.311 DEBUG:boto:Token: None 2024-04-16T19:33:51.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY= 2024-04-16T19:33:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY=', 'Content-Length': '0'} 2024-04-16T19:33:51.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5d8f50a161dba9b-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.354 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.355 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.355 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.355 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.355 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.355 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.355 DEBUG:boto:Data: 2024-04-16T19:33:51.355 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.355 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.356 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.356 DEBUG:boto:Params: {} 2024-04-16T19:33:51.356 DEBUG:boto:Token: None 2024-04-16T19:33:51.356 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rCIH0laJdD6Dwq1yWbUG+VVUqjs= 2024-04-16T19:33:51.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rCIH0laJdD6Dwq1yWbUG+VVUqjs=', 'Content-Length': '0'} 2024-04-16T19:33:51.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e52fbfd7af05300-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.359 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.359 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.359 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.360 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.360 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.360 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.360 DEBUG:boto:Data: 2024-04-16T19:33:51.360 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.360 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.360 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.360 DEBUG:boto:Params: {} 2024-04-16T19:33:51.361 DEBUG:boto:Token: None 2024-04-16T19:33:51.361 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.361 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rCIH0laJdD6Dwq1yWbUG+VVUqjs= 2024-04-16T19:33:51.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rCIH0laJdD6Dwq1yWbUG+VVUqjs=', 'Content-Length': '0'} 2024-04-16T19:33:51.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000808a70f4ffd45536-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.364 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.364 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.364 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.364 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.364 DEBUG:boto:Method: GET 2024-04-16T19:33:51.364 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.365 DEBUG:boto:Data: 2024-04-16T19:33:51.365 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.365 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.365 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.365 DEBUG:boto:Params: {} 2024-04-16T19:33:51.365 DEBUG:boto:Token: None 2024-04-16T19:33:51.365 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.366 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY= 2024-04-16T19:33:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY=', 'Content-Length': '0'} 2024-04-16T19:33:51.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a97eabb4d07d791-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.369 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.369 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.369 DEBUG:boto:path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.369 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.369 DEBUG:boto:Method: GET 2024-04-16T19:33:51.370 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.370 DEBUG:boto:Data: 2024-04-16T19:33:51.370 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.370 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.370 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.370 DEBUG:boto:Params: {} 2024-04-16T19:33:51.370 DEBUG:boto:Token: None 2024-04-16T19:33:51.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CYAOhFv8bg7kzgAKSBFmemvPHi04V8r 2024-04-16T19:33:51.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY= 2024-04-16T19:33:51.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ukI3izPRc5Fh6DRbC08WtVNQtEY=', 'Content-Length': '0'} 2024-04-16T19:33:51.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'CYAOhFv8bg7kzgAKSBFmemvPHi04V8r'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000703ca5cafa60890c-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.414 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.415 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.415 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.415 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.415 DEBUG:boto:Method: GET 2024-04-16T19:33:51.415 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.415 DEBUG:boto:Data: 2024-04-16T19:33:51.415 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.416 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.416 DEBUG:boto:Params: {} 2024-04-16T19:33:51.416 DEBUG:boto:Token: None 2024-04-16T19:33:51.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fbb411cfeab349-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.419 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.420 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.420 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.420 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.420 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.420 DEBUG:boto:Method: GET 2024-04-16T19:33:51.420 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.420 DEBUG:boto:Data: 2024-04-16T19:33:51.420 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.421 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.421 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.421 DEBUG:boto:Params: {} 2024-04-16T19:33:51.421 DEBUG:boto:Token: None 2024-04-16T19:33:51.421 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108d60d22169c554-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.424 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.425 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.425 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.425 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.425 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.425 DEBUG:boto:Data: 2024-04-16T19:33:51.425 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.425 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.425 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.426 DEBUG:boto:Params: {} 2024-04-16T19:33:51.426 DEBUG:boto:Token: None 2024-04-16T19:33:51.426 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000621a8af570c936b0-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.431 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.431 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.431 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.432 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.432 DEBUG:boto:Data: 2024-04-16T19:33:51.432 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.432 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.432 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.432 DEBUG:boto:Params: {} 2024-04-16T19:33:51.432 DEBUG:boto:Token: None 2024-04-16T19:33:51.432 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078912cbe23de2e7b-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.437 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.437 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.437 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.437 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.437 DEBUG:boto:Method: GET 2024-04-16T19:33:51.438 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.438 DEBUG:boto:Data: 2024-04-16T19:33:51.438 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.438 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.438 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.438 DEBUG:boto:Params: {} 2024-04-16T19:33:51.438 DEBUG:boto:Token: None 2024-04-16T19:33:51.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046278de1850ab63d-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.482 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.483 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.483 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.483 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.483 DEBUG:boto:Method: GET 2024-04-16T19:33:51.483 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.484 DEBUG:boto:Data: 2024-04-16T19:33:51.484 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.484 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.484 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.484 DEBUG:boto:Params: {} 2024-04-16T19:33:51.485 DEBUG:boto:Token: None 2024-04-16T19:33:51.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4324eaa550c4e6-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.535 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.535 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.535 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.535 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.535 DEBUG:boto:Method: GET 2024-04-16T19:33:51.535 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.535 DEBUG:boto:Data: 2024-04-16T19:33:51.536 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.536 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.536 DEBUG:boto:Params: {} 2024-04-16T19:33:51.536 DEBUG:boto:Token: None 2024-04-16T19:33:51.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883f89c5e5e07c88-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.540 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.540 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.540 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.540 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.540 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.541 DEBUG:boto:Method: GET 2024-04-16T19:33:51.541 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.541 DEBUG:boto:Data: 2024-04-16T19:33:51.541 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.541 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.541 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.541 DEBUG:boto:Params: {} 2024-04-16T19:33:51.542 DEBUG:boto:Token: None 2024-04-16T19:33:51.542 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b461055dc9e7123-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.545 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.546 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.546 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.546 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.546 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.546 DEBUG:boto:Method: GET 2024-04-16T19:33:51.546 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.547 DEBUG:boto:Data: 2024-04-16T19:33:51.547 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.547 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.547 DEBUG:boto:Params: {} 2024-04-16T19:33:51.547 DEBUG:boto:Token: None 2024-04-16T19:33:51.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc= 2024-04-16T19:33:51.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc=', 'Content-Length': '0'} 2024-04-16T19:33:51.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c381227b97fc419-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.550 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.550 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.550 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.550 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.551 DEBUG:boto:Method: GET 2024-04-16T19:33:51.551 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.551 DEBUG:boto:Data: 2024-04-16T19:33:51.551 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.551 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.551 DEBUG:boto:Params: {} 2024-04-16T19:33:51.551 DEBUG:boto:Token: None 2024-04-16T19:33:51.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc= 2024-04-16T19:33:51.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc=', 'Content-Length': '0'} 2024-04-16T19:33:51.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3af5bc30ca80474-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.594 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.595 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.595 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.595 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.595 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.595 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.595 DEBUG:boto:Data: 2024-04-16T19:33:51.595 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.595 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.596 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.596 DEBUG:boto:Params: {} 2024-04-16T19:33:51.596 DEBUG:boto:Token: None 2024-04-16T19:33:51.596 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:18wN+aqoeZ1zHi1YXpG0elKceUQ= 2024-04-16T19:33:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:18wN+aqoeZ1zHi1YXpG0elKceUQ=', 'Content-Length': '0'} 2024-04-16T19:33:51.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006152084d9bf7baf8-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.598 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.599 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.599 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.599 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.599 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.599 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.599 DEBUG:boto:Data: 2024-04-16T19:33:51.599 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.599 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.600 DEBUG:boto:Params: {} 2024-04-16T19:33:51.600 DEBUG:boto:Token: None 2024-04-16T19:33:51.600 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:18wN+aqoeZ1zHi1YXpG0elKceUQ= 2024-04-16T19:33:51.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:18wN+aqoeZ1zHi1YXpG0elKceUQ=', 'Content-Length': '0'} 2024-04-16T19:33:51.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f56f665973973042-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.603 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.603 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.603 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.603 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.603 DEBUG:boto:Method: GET 2024-04-16T19:33:51.603 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.603 DEBUG:boto:Data: 2024-04-16T19:33:51.603 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.604 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.604 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.604 DEBUG:boto:Params: {} 2024-04-16T19:33:51.604 DEBUG:boto:Token: None 2024-04-16T19:33:51.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc= 2024-04-16T19:33:51.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc=', 'Content-Length': '0'} 2024-04-16T19:33:51.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c90406e32ab9acd-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.607 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.607 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.607 DEBUG:boto:path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.607 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.607 DEBUG:boto:Method: GET 2024-04-16T19:33:51.607 DEBUG:boto:Path: /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.607 DEBUG:boto:Data: 2024-04-16T19:33:51.607 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.608 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.608 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.608 DEBUG:boto:Params: {} 2024-04-16T19:33:51.608 DEBUG:boto:Token: None 2024-04-16T19:33:51.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=N4TTl.jGFPrOisqoPfBas3tagTGN6P8 2024-04-16T19:33:51.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc= 2024-04-16T19:33:51.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v8ej6vhIJzH0BWDGhDjuiqnxCzc=', 'Content-Length': '0'} 2024-04-16T19:33:51.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'N4TTl.jGFPrOisqoPfBas3tagTGN6P8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bdeb957673bb4ce-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.654 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.655 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.655 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.655 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.655 DEBUG:boto:Method: GET 2024-04-16T19:33:51.655 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.655 DEBUG:boto:Data: 2024-04-16T19:33:51.655 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.656 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.656 DEBUG:boto:Params: {} 2024-04-16T19:33:51.656 DEBUG:boto:Token: None 2024-04-16T19:33:51.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f218afd11d1e8fe0-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.660 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.660 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.660 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.661 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.661 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.661 DEBUG:boto:Method: GET 2024-04-16T19:33:51.661 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.661 DEBUG:boto:Data: 2024-04-16T19:33:51.661 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.661 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.661 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.662 DEBUG:boto:Params: {} 2024-04-16T19:33:51.662 DEBUG:boto:Token: None 2024-04-16T19:33:51.662 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c750b4c7ea5ecc-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.706 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.707 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.707 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.707 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.707 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.708 DEBUG:boto:Data: 2024-04-16T19:33:51.708 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.708 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.708 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.708 DEBUG:boto:Params: {} 2024-04-16T19:33:51.708 DEBUG:boto:Token: None 2024-04-16T19:33:51.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.709 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000504f6c9c5b8ad3b6-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.714 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.714 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.714 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.714 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.714 DEBUG:boto:Data: 2024-04-16T19:33:51.715 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.715 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.715 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.715 DEBUG:boto:Params: {} 2024-04-16T19:33:51.715 DEBUG:boto:Token: None 2024-04-16T19:33:51.715 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa2276b970bb10d4-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.720 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.720 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.720 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.720 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.721 DEBUG:boto:Method: GET 2024-04-16T19:33:51.721 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.721 DEBUG:boto:Data: 2024-04-16T19:33:51.721 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.721 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.721 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.721 DEBUG:boto:Params: {} 2024-04-16T19:33:51.721 DEBUG:boto:Token: None 2024-04-16T19:33:51.722 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d294a3af6ebd87b8-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.725 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.725 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.725 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.725 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.725 DEBUG:boto:Method: GET 2024-04-16T19:33:51.725 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.725 DEBUG:boto:Data: 2024-04-16T19:33:51.726 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.726 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.726 DEBUG:boto:Params: {} 2024-04-16T19:33:51.726 DEBUG:boto:Token: None 2024-04-16T19:33:51.726 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8f418a47f2eabaa-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.770 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.771 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.771 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.771 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.771 DEBUG:boto:Method: GET 2024-04-16T19:33:51.771 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.771 DEBUG:boto:Data: 2024-04-16T19:33:51.771 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.772 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.772 DEBUG:boto:Params: {} 2024-04-16T19:33:51.772 DEBUG:boto:Token: None 2024-04-16T19:33:51.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de801dde6054f288-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.775 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.775 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.776 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.776 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.776 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.776 DEBUG:boto:Method: GET 2024-04-16T19:33:51.776 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.776 DEBUG:boto:Data: 2024-04-16T19:33:51.776 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.776 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.777 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.777 DEBUG:boto:Params: {} 2024-04-16T19:33:51.777 DEBUG:boto:Token: None 2024-04-16T19:33:51.777 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000509aa74ff053a666-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.822 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.823 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.823 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.823 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.823 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.824 DEBUG:boto:Method: GET 2024-04-16T19:33:51.824 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.824 DEBUG:boto:Data: 2024-04-16T19:33:51.824 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.824 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.824 DEBUG:boto:Params: {} 2024-04-16T19:33:51.824 DEBUG:boto:Token: None 2024-04-16T19:33:51.825 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM= 2024-04-16T19:33:51.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM=', 'Content-Length': '0'} 2024-04-16T19:33:51.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db0ece7ee5f61ee5-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.870 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.871 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.871 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.871 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.871 DEBUG:boto:Method: GET 2024-04-16T19:33:51.871 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.871 DEBUG:boto:Data: 2024-04-16T19:33:51.871 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.872 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.872 DEBUG:boto:Params: {} 2024-04-16T19:33:51.872 DEBUG:boto:Token: None 2024-04-16T19:33:51.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM= 2024-04-16T19:33:51.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM=', 'Content-Length': '0'} 2024-04-16T19:33:51.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0e8e28ecf447395-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.875 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.875 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.875 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.875 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.876 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.876 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.876 DEBUG:boto:Data: 2024-04-16T19:33:51.876 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.876 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.876 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.876 DEBUG:boto:Params: {} 2024-04-16T19:33:51.876 DEBUG:boto:Token: None 2024-04-16T19:33:51.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+IRqI/vWMjr3pdHpqVIYLNVaazk= 2024-04-16T19:33:51.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+IRqI/vWMjr3pdHpqVIYLNVaazk=', 'Content-Length': '0'} 2024-04-16T19:33:51.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000746535866fa70c03-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.879 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.879 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.879 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.879 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.880 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.880 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.880 DEBUG:boto:Data: 2024-04-16T19:33:51.880 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.880 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.880 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.880 DEBUG:boto:Params: {} 2024-04-16T19:33:51.880 DEBUG:boto:Token: None 2024-04-16T19:33:51.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+IRqI/vWMjr3pdHpqVIYLNVaazk= 2024-04-16T19:33:51.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+IRqI/vWMjr3pdHpqVIYLNVaazk=', 'Content-Length': '0'} 2024-04-16T19:33:51.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008807e0e15a5c7b1c-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.883 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.883 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.883 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.883 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.883 DEBUG:boto:Method: GET 2024-04-16T19:33:51.884 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.884 DEBUG:boto:Data: 2024-04-16T19:33:51.884 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.884 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.884 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.884 DEBUG:boto:Params: {} 2024-04-16T19:33:51.884 DEBUG:boto:Token: None 2024-04-16T19:33:51.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM= 2024-04-16T19:33:51.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM=', 'Content-Length': '0'} 2024-04-16T19:33:51.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b50e8c0647de9449-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.930 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.931 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.931 DEBUG:boto:path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.931 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.931 DEBUG:boto:Method: GET 2024-04-16T19:33:51.931 DEBUG:boto:Path: /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.931 DEBUG:boto:Data: 2024-04-16T19:33:51.931 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.931 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.931 DEBUG:boto:Params: {} 2024-04-16T19:33:51.932 DEBUG:boto:Token: None 2024-04-16T19:33:51.932 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm 2024-04-16T19:33:51.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM= 2024-04-16T19:33:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qSxOAr+nBwkECjZVr1/eJJSqiHM=', 'Content-Length': '0'} 2024-04-16T19:33:51.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'cMEGTbElZiAsg21zVl0Gu3D.rK1-aRm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cefbae6c9f2042b-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.935 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.935 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.935 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.935 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.935 DEBUG:boto:Method: GET 2024-04-16T19:33:51.935 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.935 DEBUG:boto:Data: 2024-04-16T19:33:51.935 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.935 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.936 DEBUG:boto:Params: {} 2024-04-16T19:33:51.936 DEBUG:boto:Token: None 2024-04-16T19:33:51.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b451d2e4020795-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.938 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.939 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:51.939 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:51.939 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.939 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:51.939 DEBUG:boto:Method: GET 2024-04-16T19:33:51.939 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:51.939 DEBUG:boto:Data: 2024-04-16T19:33:51.940 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.940 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.940 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.940 DEBUG:boto:Params: {} 2024-04-16T19:33:51.940 DEBUG:boto:Token: None 2024-04-16T19:33:51.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/ 2024-04-16T19:33:51.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c= 2024-04-16T19:33:51.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v6ojD6GvTQ6Nma7rm/qZ9dJG16c=', 'Content-Length': '0'} 2024-04-16T19:33:51.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086647575c12769db-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.943 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:51.943 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.943 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.943 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.943 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.943 DEBUG:boto:Data: 2024-04-16T19:33:51.943 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.944 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.944 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.944 DEBUG:boto:Params: {} 2024-04-16T19:33:51.944 DEBUG:boto:Token: None 2024-04-16T19:33:51.944 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f09e8e06b088378-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.948 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.949 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.949 DEBUG:boto:Method: HEAD 2024-04-16T19:33:51.949 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:51.949 DEBUG:boto:Data: 2024-04-16T19:33:51.949 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.949 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.949 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.949 DEBUG:boto:Params: {} 2024-04-16T19:33:51.949 DEBUG:boto:Token: None 2024-04-16T19:33:51.949 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj 2024-04-16T19:33:51.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU= 2024-04-16T19:33:51.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K5rIRVa6gYZqlY4LFQ3AXn5rSRU=', 'Content-Length': '0'} 2024-04-16T19:33:51.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b77a8e9ac0695332-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:51.954 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.954 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.954 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.954 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.954 DEBUG:boto:Method: GET 2024-04-16T19:33:51.954 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.954 DEBUG:boto:Data: 2024-04-16T19:33:51.954 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.954 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:51.954 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.954 DEBUG:boto:Params: {} 2024-04-16T19:33:51.955 DEBUG:boto:Token: None 2024-04-16T19:33:51.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:51.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:51.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038dd45c3aaaa3448-00661ed29f-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:51.998 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:51.999 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:51.999 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.999 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.999 DEBUG:boto:Method: GET 2024-04-16T19:33:51.999 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:51.999 DEBUG:boto:Data: 2024-04-16T19:33:51.999 DEBUG:boto:Headers: {} 2024-04-16T19:33:51.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:51.999 DEBUG:boto:Port: 8000 2024-04-16T19:33:51.999 DEBUG:boto:Params: {} 2024-04-16T19:33:51.999 DEBUG:boto:Token: None 2024-04-16T19:33:52.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:51 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8= 2024-04-16T19:33:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J4C4n+ceF5baGHa+IiwIeCogEJ8=', 'Content-Length': '0'} 2024-04-16T19:33:52.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c5888aedf8a647e-00661ed29f-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.042 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.042 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.043 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.043 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.043 DEBUG:boto:Method: GET 2024-04-16T19:33:52.043 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.043 DEBUG:boto:Data: 2024-04-16T19:33:52.043 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.043 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.043 DEBUG:boto:Params: {} 2024-04-16T19:33:52.043 DEBUG:boto:Token: None 2024-04-16T19:33:52.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7e382dfd3e23f9-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.046 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.047 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.047 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.047 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.047 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.047 DEBUG:boto:Method: GET 2024-04-16T19:33:52.047 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.047 DEBUG:boto:Data: 2024-04-16T19:33:52.047 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.048 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.048 DEBUG:boto:Params: {} 2024-04-16T19:33:52.048 DEBUG:boto:Token: None 2024-04-16T19:33:52.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0ce5555518cf7a-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.050 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.051 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.051 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.051 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.051 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.051 DEBUG:boto:Method: GET 2024-04-16T19:33:52.051 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.051 DEBUG:boto:Data: 2024-04-16T19:33:52.052 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.052 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.052 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.052 DEBUG:boto:Params: {} 2024-04-16T19:33:52.052 DEBUG:boto:Token: None 2024-04-16T19:33:52.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc= 2024-04-16T19:33:52.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc=', 'Content-Length': '0'} 2024-04-16T19:33:52.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d9679084d6f186d-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.098 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.099 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.099 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.099 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.099 DEBUG:boto:Method: GET 2024-04-16T19:33:52.099 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.099 DEBUG:boto:Data: 2024-04-16T19:33:52.099 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.099 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.099 DEBUG:boto:Params: {} 2024-04-16T19:33:52.099 DEBUG:boto:Token: None 2024-04-16T19:33:52.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc= 2024-04-16T19:33:52.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc=', 'Content-Length': '0'} 2024-04-16T19:33:52.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c558f48590fdff2-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.102 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.102 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.102 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.102 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.102 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.102 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.103 DEBUG:boto:Data: 2024-04-16T19:33:52.103 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.103 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.103 DEBUG:boto:Params: {} 2024-04-16T19:33:52.103 DEBUG:boto:Token: None 2024-04-16T19:33:52.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouYKYvHPZLd8+Z23KqrdtNmlpyo= 2024-04-16T19:33:52.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouYKYvHPZLd8+Z23KqrdtNmlpyo=', 'Content-Length': '0'} 2024-04-16T19:33:52.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002deab1575e97bc30-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.105 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.105 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.105 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.105 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.106 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.106 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.106 DEBUG:boto:Data: 2024-04-16T19:33:52.106 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.106 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.106 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.106 DEBUG:boto:Params: {} 2024-04-16T19:33:52.106 DEBUG:boto:Token: None 2024-04-16T19:33:52.106 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouYKYvHPZLd8+Z23KqrdtNmlpyo= 2024-04-16T19:33:52.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouYKYvHPZLd8+Z23KqrdtNmlpyo=', 'Content-Length': '0'} 2024-04-16T19:33:52.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef5186d0e591b307-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.108 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.109 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.109 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.109 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.109 DEBUG:boto:Method: GET 2024-04-16T19:33:52.109 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.109 DEBUG:boto:Data: 2024-04-16T19:33:52.109 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.109 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.109 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.109 DEBUG:boto:Params: {} 2024-04-16T19:33:52.109 DEBUG:boto:Token: None 2024-04-16T19:33:52.110 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.110 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc= 2024-04-16T19:33:52.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc=', 'Content-Length': '0'} 2024-04-16T19:33:52.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ad843f7ed4df80f-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.154 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.155 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.155 DEBUG:boto:path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.155 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.155 DEBUG:boto:Method: GET 2024-04-16T19:33:52.155 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.155 DEBUG:boto:Data: 2024-04-16T19:33:52.155 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.155 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.155 DEBUG:boto:Params: {} 2024-04-16T19:33:52.156 DEBUG:boto:Token: None 2024-04-16T19:33:52.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c 2024-04-16T19:33:52.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc= 2024-04-16T19:33:52.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9CipsbQs8VWz8KYYyx9T8LNBPc=', 'Content-Length': '0'} 2024-04-16T19:33:52.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PNCSB4IaD-TcrJja6nFQ.S9oWsdHA5c'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c3dbb8de0d39851-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.158 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.159 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.159 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.159 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.159 DEBUG:boto:Method: GET 2024-04-16T19:33:52.159 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.159 DEBUG:boto:Data: 2024-04-16T19:33:52.159 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.159 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.159 DEBUG:boto:Params: {} 2024-04-16T19:33:52.159 DEBUG:boto:Token: None 2024-04-16T19:33:52.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056023a54807f78be-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.210 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.211 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.211 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.211 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.211 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.211 DEBUG:boto:Method: GET 2024-04-16T19:33:52.211 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.211 DEBUG:boto:Data: 2024-04-16T19:33:52.211 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.211 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.211 DEBUG:boto:Params: {} 2024-04-16T19:33:52.212 DEBUG:boto:Token: None 2024-04-16T19:33:52.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024b7ad2c9ab7c59-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.214 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.214 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.215 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.215 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.215 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.215 DEBUG:boto:Data: 2024-04-16T19:33:52.215 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.215 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.215 DEBUG:boto:Params: {} 2024-04-16T19:33:52.215 DEBUG:boto:Token: None 2024-04-16T19:33:52.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2e39c354ebd9eb6-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.220 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.220 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.220 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.220 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.220 DEBUG:boto:Data: 2024-04-16T19:33:52.220 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.220 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.220 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.220 DEBUG:boto:Params: {} 2024-04-16T19:33:52.220 DEBUG:boto:Token: None 2024-04-16T19:33:52.221 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009c6c06507e797df-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.225 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.225 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.225 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.225 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.225 DEBUG:boto:Method: GET 2024-04-16T19:33:52.225 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.225 DEBUG:boto:Data: 2024-04-16T19:33:52.225 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.225 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.225 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.226 DEBUG:boto:Params: {} 2024-04-16T19:33:52.226 DEBUG:boto:Token: None 2024-04-16T19:33:52.226 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.226 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011b99a04997726a1-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.228 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.228 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.228 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.228 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.228 DEBUG:boto:Method: GET 2024-04-16T19:33:52.229 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.229 DEBUG:boto:Data: 2024-04-16T19:33:52.229 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.229 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.229 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.229 DEBUG:boto:Params: {} 2024-04-16T19:33:52.229 DEBUG:boto:Token: None 2024-04-16T19:33:52.229 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.229 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba8d8f33dd35cd93-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.274 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.274 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.275 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.275 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.275 DEBUG:boto:Method: GET 2024-04-16T19:33:52.275 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.275 DEBUG:boto:Data: 2024-04-16T19:33:52.275 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.275 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.275 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.275 DEBUG:boto:Params: {} 2024-04-16T19:33:52.275 DEBUG:boto:Token: None 2024-04-16T19:33:52.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ea3d295103aec4-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.278 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.278 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.278 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.279 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.279 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.279 DEBUG:boto:Method: GET 2024-04-16T19:33:52.279 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.279 DEBUG:boto:Data: 2024-04-16T19:33:52.279 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.279 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.279 DEBUG:boto:Params: {} 2024-04-16T19:33:52.279 DEBUG:boto:Token: None 2024-04-16T19:33:52.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e3a70750478a42-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.282 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.283 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.283 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.283 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.283 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.283 DEBUG:boto:Method: GET 2024-04-16T19:33:52.283 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.283 DEBUG:boto:Data: 2024-04-16T19:33:52.283 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.283 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.284 DEBUG:boto:Params: {} 2024-04-16T19:33:52.284 DEBUG:boto:Token: None 2024-04-16T19:33:52.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU= 2024-04-16T19:33:52.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU=', 'Content-Length': '0'} 2024-04-16T19:33:52.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da597f27e9f3aa05-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.286 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.287 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.287 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.287 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.287 DEBUG:boto:Method: GET 2024-04-16T19:33:52.287 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.287 DEBUG:boto:Data: 2024-04-16T19:33:52.287 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.287 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.287 DEBUG:boto:Params: {} 2024-04-16T19:33:52.287 DEBUG:boto:Token: None 2024-04-16T19:33:52.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU= 2024-04-16T19:33:52.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU=', 'Content-Length': '0'} 2024-04-16T19:33:52.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000034585ad05807003-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.330 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.330 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.331 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.331 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.331 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.331 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.331 DEBUG:boto:Data: 2024-04-16T19:33:52.331 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.331 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.331 DEBUG:boto:Params: {} 2024-04-16T19:33:52.331 DEBUG:boto:Token: None 2024-04-16T19:33:52.331 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tszAEPXT39rizJQ2KRII3s4YlWI= 2024-04-16T19:33:52.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tszAEPXT39rizJQ2KRII3s4YlWI=', 'Content-Length': '0'} 2024-04-16T19:33:52.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f95b758f619c3be4-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.334 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.334 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.334 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.334 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.334 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.334 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.335 DEBUG:boto:Data: 2024-04-16T19:33:52.335 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.335 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.335 DEBUG:boto:Params: {} 2024-04-16T19:33:52.335 DEBUG:boto:Token: None 2024-04-16T19:33:52.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tszAEPXT39rizJQ2KRII3s4YlWI= 2024-04-16T19:33:52.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tszAEPXT39rizJQ2KRII3s4YlWI=', 'Content-Length': '0'} 2024-04-16T19:33:52.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a13e5ddb86375c2-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.338 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.338 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.338 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.338 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.338 DEBUG:boto:Method: GET 2024-04-16T19:33:52.338 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.338 DEBUG:boto:Data: 2024-04-16T19:33:52.339 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.339 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.339 DEBUG:boto:Params: {} 2024-04-16T19:33:52.339 DEBUG:boto:Token: None 2024-04-16T19:33:52.339 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU= 2024-04-16T19:33:52.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU=', 'Content-Length': '0'} 2024-04-16T19:33:52.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b878c0e71da0c22e-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.342 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.342 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.342 DEBUG:boto:path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.342 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.342 DEBUG:boto:Method: GET 2024-04-16T19:33:52.342 DEBUG:boto:Path: /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.342 DEBUG:boto:Data: 2024-04-16T19:33:52.343 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.343 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.343 DEBUG:boto:Params: {} 2024-04-16T19:33:52.343 DEBUG:boto:Token: None 2024-04-16T19:33:52.343 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5 2024-04-16T19:33:52.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU= 2024-04-16T19:33:52.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YEyEvv3Orrq4a6ZVVHSt4DcBYvU=', 'Content-Length': '0'} 2024-04-16T19:33:52.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'uNl8ikLGAIhQJioc5ClK7HsEUE3Kgc5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a4883023401896d-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.386 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.387 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.387 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.387 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.387 DEBUG:boto:Method: GET 2024-04-16T19:33:52.387 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.387 DEBUG:boto:Data: 2024-04-16T19:33:52.387 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.387 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.387 DEBUG:boto:Params: {} 2024-04-16T19:33:52.387 DEBUG:boto:Token: None 2024-04-16T19:33:52.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010f9e374c7a23a2-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.391 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.391 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.391 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.391 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.391 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.391 DEBUG:boto:Method: GET 2024-04-16T19:33:52.391 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.391 DEBUG:boto:Data: 2024-04-16T19:33:52.391 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.392 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.392 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.392 DEBUG:boto:Params: {} 2024-04-16T19:33:52.392 DEBUG:boto:Token: None 2024-04-16T19:33:52.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab53979bccc1225f-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.395 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.395 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.395 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.395 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.395 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.395 DEBUG:boto:Data: 2024-04-16T19:33:52.396 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.396 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.396 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.396 DEBUG:boto:Params: {} 2024-04-16T19:33:52.396 DEBUG:boto:Token: None 2024-04-16T19:33:52.396 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000561e6bea76059037-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.400 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.400 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.400 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.400 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.401 DEBUG:boto:Data: 2024-04-16T19:33:52.401 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.401 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.401 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.401 DEBUG:boto:Params: {} 2024-04-16T19:33:52.401 DEBUG:boto:Token: None 2024-04-16T19:33:52.401 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4ae6dc43c9ab2f2-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.405 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.406 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.406 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.406 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.406 DEBUG:boto:Method: GET 2024-04-16T19:33:52.406 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.406 DEBUG:boto:Data: 2024-04-16T19:33:52.406 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.406 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.406 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.406 DEBUG:boto:Params: {} 2024-04-16T19:33:52.406 DEBUG:boto:Token: None 2024-04-16T19:33:52.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e11ce42cdcaa57-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.450 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.450 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.450 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.451 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.451 DEBUG:boto:Method: GET 2024-04-16T19:33:52.451 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.451 DEBUG:boto:Data: 2024-04-16T19:33:52.451 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.451 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.451 DEBUG:boto:Params: {} 2024-04-16T19:33:52.451 DEBUG:boto:Token: None 2024-04-16T19:33:52.451 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000303f7da46fa633d6-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.494 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.495 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.495 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.495 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.495 DEBUG:boto:Method: GET 2024-04-16T19:33:52.495 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.495 DEBUG:boto:Data: 2024-04-16T19:33:52.495 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.495 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.495 DEBUG:boto:Params: {} 2024-04-16T19:33:52.496 DEBUG:boto:Token: None 2024-04-16T19:33:52.496 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079870e1b3d43eb59-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.499 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.499 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.499 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.499 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.499 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.499 DEBUG:boto:Method: GET 2024-04-16T19:33:52.500 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.500 DEBUG:boto:Data: 2024-04-16T19:33:52.500 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.500 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.500 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.500 DEBUG:boto:Params: {} 2024-04-16T19:33:52.500 DEBUG:boto:Token: None 2024-04-16T19:33:52.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee16578e417008c-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.505 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.506 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.506 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.506 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.506 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.506 DEBUG:boto:Method: GET 2024-04-16T19:33:52.506 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.507 DEBUG:boto:Data: 2024-04-16T19:33:52.507 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.507 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.507 DEBUG:boto:Params: {} 2024-04-16T19:33:52.507 DEBUG:boto:Token: None 2024-04-16T19:33:52.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE= 2024-04-16T19:33:52.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE=', 'Content-Length': '0'} 2024-04-16T19:33:52.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e254d98ed7f7273-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.510 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.510 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.510 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.510 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.510 DEBUG:boto:Method: GET 2024-04-16T19:33:52.510 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.510 DEBUG:boto:Data: 2024-04-16T19:33:52.510 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.510 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.511 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.511 DEBUG:boto:Params: {} 2024-04-16T19:33:52.511 DEBUG:boto:Token: None 2024-04-16T19:33:52.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE= 2024-04-16T19:33:52.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE=', 'Content-Length': '0'} 2024-04-16T19:33:52.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a63e86ca0d1012b-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.554 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.555 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.555 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.555 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.555 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.555 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.555 DEBUG:boto:Data: 2024-04-16T19:33:52.555 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.555 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.555 DEBUG:boto:Params: {} 2024-04-16T19:33:52.555 DEBUG:boto:Token: None 2024-04-16T19:33:52.556 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1txDrJ11UzeYg2o1K1mZneVOt5c= 2024-04-16T19:33:52.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1txDrJ11UzeYg2o1K1mZneVOt5c=', 'Content-Length': '0'} 2024-04-16T19:33:52.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b127e61ca061257-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.558 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.558 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.559 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.559 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.559 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.559 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.559 DEBUG:boto:Data: 2024-04-16T19:33:52.559 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.559 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.559 DEBUG:boto:Params: {} 2024-04-16T19:33:52.559 DEBUG:boto:Token: None 2024-04-16T19:33:52.559 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1txDrJ11UzeYg2o1K1mZneVOt5c= 2024-04-16T19:33:52.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1txDrJ11UzeYg2o1K1mZneVOt5c=', 'Content-Length': '0'} 2024-04-16T19:33:52.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002595e1810fb8dc23-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.562 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.562 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.562 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.562 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.563 DEBUG:boto:Method: GET 2024-04-16T19:33:52.563 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.563 DEBUG:boto:Data: 2024-04-16T19:33:52.563 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.563 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.563 DEBUG:boto:Params: {} 2024-04-16T19:33:52.563 DEBUG:boto:Token: None 2024-04-16T19:33:52.563 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE= 2024-04-16T19:33:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE=', 'Content-Length': '0'} 2024-04-16T19:33:52.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011f0be41e0f31921-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.566 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.566 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.566 DEBUG:boto:path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.566 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.566 DEBUG:boto:Method: GET 2024-04-16T19:33:52.566 DEBUG:boto:Path: /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.566 DEBUG:boto:Data: 2024-04-16T19:33:52.567 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.567 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.567 DEBUG:boto:Params: {} 2024-04-16T19:33:52.567 DEBUG:boto:Token: None 2024-04-16T19:33:52.567 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2 2024-04-16T19:33:52.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE= 2024-04-16T19:33:52.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n0cvI4aH2ZNEH46TEz8CIr/vbRE=', 'Content-Length': '0'} 2024-04-16T19:33:52.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'ZU3NZ.mvzQMST-G3Wkv4GmgRlq2IRF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050c7dc5d1f8aa356-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.610 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.611 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.611 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.611 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.611 DEBUG:boto:Method: GET 2024-04-16T19:33:52.611 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.611 DEBUG:boto:Data: 2024-04-16T19:33:52.611 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.611 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.611 DEBUG:boto:Params: {} 2024-04-16T19:33:52.611 DEBUG:boto:Token: None 2024-04-16T19:33:52.612 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c2129a719e92c9c-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.615 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.615 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.615 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.615 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.615 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.615 DEBUG:boto:Method: GET 2024-04-16T19:33:52.616 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.616 DEBUG:boto:Data: 2024-04-16T19:33:52.616 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.616 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.616 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.616 DEBUG:boto:Params: {} 2024-04-16T19:33:52.616 DEBUG:boto:Token: None 2024-04-16T19:33:52.616 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018b918f95d493fa1-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.619 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.619 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.619 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.619 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.620 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.620 DEBUG:boto:Data: 2024-04-16T19:33:52.620 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.620 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.620 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.620 DEBUG:boto:Params: {} 2024-04-16T19:33:52.620 DEBUG:boto:Token: None 2024-04-16T19:33:52.620 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f5527c4c1ff719-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.624 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.625 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.625 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.625 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.625 DEBUG:boto:Data: 2024-04-16T19:33:52.625 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.625 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.625 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.625 DEBUG:boto:Params: {} 2024-04-16T19:33:52.625 DEBUG:boto:Token: None 2024-04-16T19:33:52.626 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.626 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c47afee53c003738-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.630 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.630 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.630 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.630 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.631 DEBUG:boto:Method: GET 2024-04-16T19:33:52.631 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.631 DEBUG:boto:Data: 2024-04-16T19:33:52.631 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.631 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.631 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.631 DEBUG:boto:Params: {} 2024-04-16T19:33:52.631 DEBUG:boto:Token: None 2024-04-16T19:33:52.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0bb1ebd36d55248-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.674 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.675 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.675 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.675 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.675 DEBUG:boto:Method: GET 2024-04-16T19:33:52.675 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.675 DEBUG:boto:Data: 2024-04-16T19:33:52.675 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.675 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.675 DEBUG:boto:Params: {} 2024-04-16T19:33:52.676 DEBUG:boto:Token: None 2024-04-16T19:33:52.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a768237aae55d84-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.723 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.723 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.723 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.723 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.723 DEBUG:boto:Method: GET 2024-04-16T19:33:52.723 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.723 DEBUG:boto:Data: 2024-04-16T19:33:52.724 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.724 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.724 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.724 DEBUG:boto:Params: {} 2024-04-16T19:33:52.724 DEBUG:boto:Token: None 2024-04-16T19:33:52.724 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815edfe87202c14d-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.727 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.727 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.727 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.728 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.728 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.728 DEBUG:boto:Method: GET 2024-04-16T19:33:52.728 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.728 DEBUG:boto:Data: 2024-04-16T19:33:52.728 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.728 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.728 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.728 DEBUG:boto:Params: {} 2024-04-16T19:33:52.728 DEBUG:boto:Token: None 2024-04-16T19:33:52.729 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac6ce2130f2bd08b-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.774 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.775 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.775 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.775 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.775 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.775 DEBUG:boto:Method: GET 2024-04-16T19:33:52.775 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.775 DEBUG:boto:Data: 2024-04-16T19:33:52.776 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.776 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.776 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.776 DEBUG:boto:Params: {} 2024-04-16T19:33:52.776 DEBUG:boto:Token: None 2024-04-16T19:33:52.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks= 2024-04-16T19:33:52.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks=', 'Content-Length': '0'} 2024-04-16T19:33:52.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9a29724dc5b71e4-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.822 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.823 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.823 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.823 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.823 DEBUG:boto:Method: GET 2024-04-16T19:33:52.823 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.823 DEBUG:boto:Data: 2024-04-16T19:33:52.823 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.823 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.823 DEBUG:boto:Params: {} 2024-04-16T19:33:52.823 DEBUG:boto:Token: None 2024-04-16T19:33:52.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks= 2024-04-16T19:33:52.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks=', 'Content-Length': '0'} 2024-04-16T19:33:52.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000861e42e940fee776-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.827 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.827 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.827 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.827 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.827 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.827 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.827 DEBUG:boto:Data: 2024-04-16T19:33:52.827 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.827 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.828 DEBUG:boto:Params: {} 2024-04-16T19:33:52.828 DEBUG:boto:Token: None 2024-04-16T19:33:52.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HRWwEFRQgIftndvz1jiLT4NUElc= 2024-04-16T19:33:52.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HRWwEFRQgIftndvz1jiLT4NUElc=', 'Content-Length': '0'} 2024-04-16T19:33:52.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac575d7c861a7974-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.830 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.830 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.830 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.831 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.831 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.831 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.831 DEBUG:boto:Data: 2024-04-16T19:33:52.831 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.831 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.831 DEBUG:boto:Params: {} 2024-04-16T19:33:52.831 DEBUG:boto:Token: None 2024-04-16T19:33:52.831 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HRWwEFRQgIftndvz1jiLT4NUElc= 2024-04-16T19:33:52.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HRWwEFRQgIftndvz1jiLT4NUElc=', 'Content-Length': '0'} 2024-04-16T19:33:52.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057bf317bb3982272-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.834 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.834 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.834 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.834 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.834 DEBUG:boto:Method: GET 2024-04-16T19:33:52.835 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.835 DEBUG:boto:Data: 2024-04-16T19:33:52.835 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.835 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.835 DEBUG:boto:Params: {} 2024-04-16T19:33:52.835 DEBUG:boto:Token: None 2024-04-16T19:33:52.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks= 2024-04-16T19:33:52.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks=', 'Content-Length': '0'} 2024-04-16T19:33:52.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9bfac5026988106-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.878 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.878 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.879 DEBUG:boto:path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.879 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.879 DEBUG:boto:Method: GET 2024-04-16T19:33:52.879 DEBUG:boto:Path: /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.879 DEBUG:boto:Data: 2024-04-16T19:33:52.879 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.879 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.879 DEBUG:boto:Params: {} 2024-04-16T19:33:52.879 DEBUG:boto:Token: None 2024-04-16T19:33:52.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=dRPU.SBoWM6qiICyhzAojFNNU.nHDyA 2024-04-16T19:33:52.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks= 2024-04-16T19:33:52.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pQNliUNO1TnRi8iKDgCN0LEdjks=', 'Content-Length': '0'} 2024-04-16T19:33:52.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'dRPU.SBoWM6qiICyhzAojFNNU.nHDyA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1b14c49974c6d33-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.882 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.882 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.882 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.882 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.882 DEBUG:boto:Method: GET 2024-04-16T19:33:52.883 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.883 DEBUG:boto:Data: 2024-04-16T19:33:52.883 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.883 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.883 DEBUG:boto:Params: {} 2024-04-16T19:33:52.883 DEBUG:boto:Token: None 2024-04-16T19:33:52.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088965e4a6db9b8ae-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.886 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.887 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.887 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.887 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.887 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.887 DEBUG:boto:Method: GET 2024-04-16T19:33:52.887 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.887 DEBUG:boto:Data: 2024-04-16T19:33:52.887 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.888 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.888 DEBUG:boto:Params: {} 2024-04-16T19:33:52.888 DEBUG:boto:Token: None 2024-04-16T19:33:52.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2ac04a30c4812f-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.891 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:52.891 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.891 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.891 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.892 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.892 DEBUG:boto:Data: 2024-04-16T19:33:52.892 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.892 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.892 DEBUG:boto:Params: {} 2024-04-16T19:33:52.892 DEBUG:boto:Token: None 2024-04-16T19:33:52.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040058e43d968f246-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.897 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.897 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.897 DEBUG:boto:Method: HEAD 2024-04-16T19:33:52.897 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:52.897 DEBUG:boto:Data: 2024-04-16T19:33:52.898 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.898 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.898 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.898 DEBUG:boto:Params: {} 2024-04-16T19:33:52.898 DEBUG:boto:Token: None 2024-04-16T19:33:52.898 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj 2024-04-16T19:33:52.898 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg= 2024-04-16T19:33:52.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LDXcBgKA6XzyEF80I/ISiGT2Bpg=', 'Content-Length': '0'} 2024-04-16T19:33:52.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000001cc52ec1380cb4-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:52.903 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.903 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.903 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.903 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.903 DEBUG:boto:Method: GET 2024-04-16T19:33:52.903 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.903 DEBUG:boto:Data: 2024-04-16T19:33:52.903 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.903 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.904 DEBUG:boto:Params: {} 2024-04-16T19:33:52.904 DEBUG:boto:Token: None 2024-04-16T19:33:52.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000745232852f9eaaa6-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.946 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:52.946 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:52.947 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.947 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.947 DEBUG:boto:Method: GET 2024-04-16T19:33:52.947 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.947 DEBUG:boto:Data: 2024-04-16T19:33:52.947 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:52.947 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.947 DEBUG:boto:Params: {} 2024-04-16T19:33:52.947 DEBUG:boto:Token: None 2024-04-16T19:33:52.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:52.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc= 2024-04-16T19:33:52.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuaAgr+g4ZPPuttJhiBBqDNvjZc=', 'Content-Length': '0'} 2024-04-16T19:33:52.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f7fa8cbee489785-00661ed2a0-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.994 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:52.995 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:52.995 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.995 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:52.995 DEBUG:boto:Method: GET 2024-04-16T19:33:52.995 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:52.995 DEBUG:boto:Data: 2024-04-16T19:33:52.995 DEBUG:boto:Headers: {} 2024-04-16T19:33:52.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:52.996 DEBUG:boto:Port: 8000 2024-04-16T19:33:52.996 DEBUG:boto:Params: {} 2024-04-16T19:33:52.996 DEBUG:boto:Token: None 2024-04-16T19:33:52.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:52 GMT /yrjefh-15/ 2024-04-16T19:33:52.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk= 2024-04-16T19:33:52.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2YknatFKWOIdEHkEoHn1abHLAnk=', 'Content-Length': '0'} 2024-04-16T19:33:52.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979f8138733daae6-00661ed2a0-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:52.999 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.000 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.000 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.000 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.000 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.000 DEBUG:boto:Method: GET 2024-04-16T19:33:53.001 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.001 DEBUG:boto:Data: 2024-04-16T19:33:53.001 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.001 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.001 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.001 DEBUG:boto:Params: {} 2024-04-16T19:33:53.001 DEBUG:boto:Token: None 2024-04-16T19:33:53.001 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.002 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005119d31a287ead4c-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.046 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.047 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.047 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.047 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.047 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.048 DEBUG:boto:Method: GET 2024-04-16T19:33:53.048 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.048 DEBUG:boto:Data: 2024-04-16T19:33:53.048 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.048 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.048 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.048 DEBUG:boto:Params: {} 2024-04-16T19:33:53.048 DEBUG:boto:Token: None 2024-04-16T19:33:53.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8= 2024-04-16T19:33:53.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8=', 'Content-Length': '0'} 2024-04-16T19:33:53.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007099f94c5715f15b-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.051 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.052 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.052 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.052 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.052 DEBUG:boto:Method: GET 2024-04-16T19:33:53.052 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.052 DEBUG:boto:Data: 2024-04-16T19:33:53.052 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.052 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.052 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.053 DEBUG:boto:Params: {} 2024-04-16T19:33:53.053 DEBUG:boto:Token: None 2024-04-16T19:33:53.053 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.053 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8= 2024-04-16T19:33:53.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8=', 'Content-Length': '0'} 2024-04-16T19:33:53.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007b3676507c4dbbe-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.098 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.099 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.099 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.099 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.099 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.099 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.099 DEBUG:boto:Data: 2024-04-16T19:33:53.099 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.100 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.100 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.100 DEBUG:boto:Params: {} 2024-04-16T19:33:53.100 DEBUG:boto:Token: None 2024-04-16T19:33:53.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rBgACHPPMb40htuku75iFXA4yyk= 2024-04-16T19:33:53.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rBgACHPPMb40htuku75iFXA4yyk=', 'Content-Length': '0'} 2024-04-16T19:33:53.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b030acb5d68a546-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.103 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.103 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.103 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.103 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.103 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.103 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.104 DEBUG:boto:Data: 2024-04-16T19:33:53.104 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.104 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.104 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.104 DEBUG:boto:Params: {} 2024-04-16T19:33:53.104 DEBUG:boto:Token: None 2024-04-16T19:33:53.104 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rBgACHPPMb40htuku75iFXA4yyk= 2024-04-16T19:33:53.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rBgACHPPMb40htuku75iFXA4yyk=', 'Content-Length': '0'} 2024-04-16T19:33:53.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb043f0bdb80be33-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.107 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.107 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.107 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.107 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.108 DEBUG:boto:Method: GET 2024-04-16T19:33:53.108 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.108 DEBUG:boto:Data: 2024-04-16T19:33:53.108 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.108 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.108 DEBUG:boto:Params: {} 2024-04-16T19:33:53.108 DEBUG:boto:Token: None 2024-04-16T19:33:53.109 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8= 2024-04-16T19:33:53.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8=', 'Content-Length': '0'} 2024-04-16T19:33:53.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b748c05a3578b2da-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.111 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.111 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.111 DEBUG:boto:path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.112 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.112 DEBUG:boto:Method: GET 2024-04-16T19:33:53.112 DEBUG:boto:Path: /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.112 DEBUG:boto:Data: 2024-04-16T19:33:53.112 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.112 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.112 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.112 DEBUG:boto:Params: {} 2024-04-16T19:33:53.113 DEBUG:boto:Token: None 2024-04-16T19:33:53.113 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=nCXWVN92LLTZ7uScP.40eRgAyqJsfbK 2024-04-16T19:33:53.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8= 2024-04-16T19:33:53.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmBcwtJD4MiMbf4IQXujT7u7EN8=', 'Content-Length': '0'} 2024-04-16T19:33:53.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'nCXWVN92LLTZ7uScP.40eRgAyqJsfbK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047cb291b36c4030a-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.158 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.159 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.159 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.159 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.159 DEBUG:boto:Method: GET 2024-04-16T19:33:53.159 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.159 DEBUG:boto:Data: 2024-04-16T19:33:53.159 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.160 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.160 DEBUG:boto:Params: {} 2024-04-16T19:33:53.160 DEBUG:boto:Token: None 2024-04-16T19:33:53.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3028e9e6dbac810-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.164 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.164 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.164 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.164 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.165 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.165 DEBUG:boto:Method: GET 2024-04-16T19:33:53.165 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.165 DEBUG:boto:Data: 2024-04-16T19:33:53.165 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.165 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.165 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.165 DEBUG:boto:Params: {} 2024-04-16T19:33:53.165 DEBUG:boto:Token: None 2024-04-16T19:33:53.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000864addbf211f2b80-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.169 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.169 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.169 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.169 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.170 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.170 DEBUG:boto:Data: 2024-04-16T19:33:53.170 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.170 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.170 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.170 DEBUG:boto:Params: {} 2024-04-16T19:33:53.170 DEBUG:boto:Token: None 2024-04-16T19:33:53.171 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028f4ed6c2c4b9286-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.175 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.175 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.176 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.176 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.176 DEBUG:boto:Data: 2024-04-16T19:33:53.176 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.176 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.176 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.176 DEBUG:boto:Params: {} 2024-04-16T19:33:53.176 DEBUG:boto:Token: None 2024-04-16T19:33:53.177 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.177 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fd5b7aa90c607ec-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.181 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.181 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.181 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.182 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.182 DEBUG:boto:Method: GET 2024-04-16T19:33:53.182 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.182 DEBUG:boto:Data: 2024-04-16T19:33:53.182 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.182 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.182 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.183 DEBUG:boto:Params: {} 2024-04-16T19:33:53.183 DEBUG:boto:Token: None 2024-04-16T19:33:53.183 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6791c2c386fb2b9-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.226 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.227 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.227 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.227 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.227 DEBUG:boto:Method: GET 2024-04-16T19:33:53.227 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.227 DEBUG:boto:Data: 2024-04-16T19:33:53.227 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.228 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.228 DEBUG:boto:Params: {} 2024-04-16T19:33:53.228 DEBUG:boto:Token: None 2024-04-16T19:33:53.228 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000441be58df6895181-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.274 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.275 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.275 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.275 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.275 DEBUG:boto:Method: GET 2024-04-16T19:33:53.275 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.275 DEBUG:boto:Data: 2024-04-16T19:33:53.275 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.276 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.276 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.276 DEBUG:boto:Params: {} 2024-04-16T19:33:53.276 DEBUG:boto:Token: None 2024-04-16T19:33:53.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e87bb80882ff9ba-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.279 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.280 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.280 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.280 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.280 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.280 DEBUG:boto:Method: GET 2024-04-16T19:33:53.280 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.280 DEBUG:boto:Data: 2024-04-16T19:33:53.281 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.281 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.281 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.281 DEBUG:boto:Params: {} 2024-04-16T19:33:53.281 DEBUG:boto:Token: None 2024-04-16T19:33:53.281 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e2420a113252849-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.326 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.327 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.327 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.327 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.327 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.328 DEBUG:boto:Method: GET 2024-04-16T19:33:53.328 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.328 DEBUG:boto:Data: 2024-04-16T19:33:53.328 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.328 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.328 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.328 DEBUG:boto:Params: {} 2024-04-16T19:33:53.328 DEBUG:boto:Token: None 2024-04-16T19:33:53.329 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.329 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4= 2024-04-16T19:33:53.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4=', 'Content-Length': '0'} 2024-04-16T19:33:53.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7aed3dd4ef70440-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.375 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.375 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.375 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.375 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.375 DEBUG:boto:Method: GET 2024-04-16T19:33:53.375 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.375 DEBUG:boto:Data: 2024-04-16T19:33:53.376 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.376 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.376 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.376 DEBUG:boto:Params: {} 2024-04-16T19:33:53.376 DEBUG:boto:Token: None 2024-04-16T19:33:53.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.377 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4= 2024-04-16T19:33:53.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4=', 'Content-Length': '0'} 2024-04-16T19:33:53.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd7d2cc3941ab056-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.379 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.379 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.379 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.379 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.380 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.380 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.380 DEBUG:boto:Data: 2024-04-16T19:33:53.380 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.380 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.380 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.380 DEBUG:boto:Params: {} 2024-04-16T19:33:53.380 DEBUG:boto:Token: None 2024-04-16T19:33:53.381 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iBopN0mZb7D76ZEiQNl5RffduT8= 2024-04-16T19:33:53.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iBopN0mZb7D76ZEiQNl5RffduT8=', 'Content-Length': '0'} 2024-04-16T19:33:53.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df6cacb0b1cbf3d4-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.383 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.383 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.384 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.384 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.384 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.384 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.384 DEBUG:boto:Data: 2024-04-16T19:33:53.384 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.384 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.384 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.384 DEBUG:boto:Params: {} 2024-04-16T19:33:53.385 DEBUG:boto:Token: None 2024-04-16T19:33:53.385 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iBopN0mZb7D76ZEiQNl5RffduT8= 2024-04-16T19:33:53.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iBopN0mZb7D76ZEiQNl5RffduT8=', 'Content-Length': '0'} 2024-04-16T19:33:53.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325f7577e8b513a7-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.387 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.388 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.388 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.388 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.388 DEBUG:boto:Method: GET 2024-04-16T19:33:53.388 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.388 DEBUG:boto:Data: 2024-04-16T19:33:53.388 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.388 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.389 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.389 DEBUG:boto:Params: {} 2024-04-16T19:33:53.389 DEBUG:boto:Token: None 2024-04-16T19:33:53.389 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4= 2024-04-16T19:33:53.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4=', 'Content-Length': '0'} 2024-04-16T19:33:53.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee09b31ec57a289d-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.435 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.435 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.435 DEBUG:boto:path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.435 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.435 DEBUG:boto:Method: GET 2024-04-16T19:33:53.435 DEBUG:boto:Path: /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.435 DEBUG:boto:Data: 2024-04-16T19:33:53.435 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.436 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.436 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.436 DEBUG:boto:Params: {} 2024-04-16T19:33:53.436 DEBUG:boto:Token: None 2024-04-16T19:33:53.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=w8fgzqLyp-vAWbVK094EB5Cn6ffO05M 2024-04-16T19:33:53.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4= 2024-04-16T19:33:53.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W2to85KEhMLfx9h708Z3R3Hr/b4=', 'Content-Length': '0'} 2024-04-16T19:33:53.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'w8fgzqLyp-vAWbVK094EB5Cn6ffO05M'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003867931a4e37d250-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.439 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.439 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.439 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.439 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.439 DEBUG:boto:Method: GET 2024-04-16T19:33:53.440 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.440 DEBUG:boto:Data: 2024-04-16T19:33:53.440 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.440 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.440 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.440 DEBUG:boto:Params: {} 2024-04-16T19:33:53.440 DEBUG:boto:Token: None 2024-04-16T19:33:53.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.441 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b2a8756fd91d09-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.486 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.487 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.487 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.487 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.487 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.487 DEBUG:boto:Method: GET 2024-04-16T19:33:53.488 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.488 DEBUG:boto:Data: 2024-04-16T19:33:53.488 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.488 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.488 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.488 DEBUG:boto:Params: {} 2024-04-16T19:33:53.488 DEBUG:boto:Token: None 2024-04-16T19:33:53.489 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007635896710cb2e44-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.492 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.492 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.493 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.493 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.493 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.493 DEBUG:boto:Data: 2024-04-16T19:33:53.493 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.493 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.493 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.494 DEBUG:boto:Params: {} 2024-04-16T19:33:53.494 DEBUG:boto:Token: None 2024-04-16T19:33:53.494 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db980235c315b758-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.499 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.499 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.499 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.499 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.500 DEBUG:boto:Data: 2024-04-16T19:33:53.500 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.500 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.500 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.500 DEBUG:boto:Params: {} 2024-04-16T19:33:53.500 DEBUG:boto:Token: None 2024-04-16T19:33:53.500 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.501 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b2bb5cbf8aab0f5-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.505 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.505 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.506 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.506 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.506 DEBUG:boto:Method: GET 2024-04-16T19:33:53.506 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.506 DEBUG:boto:Data: 2024-04-16T19:33:53.506 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.507 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.507 DEBUG:boto:Params: {} 2024-04-16T19:33:53.507 DEBUG:boto:Token: None 2024-04-16T19:33:53.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000775fe31c92f0fe6d-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.510 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.511 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.511 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.511 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.511 DEBUG:boto:Method: GET 2024-04-16T19:33:53.511 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.511 DEBUG:boto:Data: 2024-04-16T19:33:53.511 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.512 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.512 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.512 DEBUG:boto:Params: {} 2024-04-16T19:33:53.512 DEBUG:boto:Token: None 2024-04-16T19:33:53.512 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a888c39f33a5a305-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.558 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.559 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.559 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.559 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.559 DEBUG:boto:Method: GET 2024-04-16T19:33:53.559 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.559 DEBUG:boto:Data: 2024-04-16T19:33:53.559 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.559 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.560 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.560 DEBUG:boto:Params: {} 2024-04-16T19:33:53.560 DEBUG:boto:Token: None 2024-04-16T19:33:53.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020433937b9b694ee-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.563 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.564 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.564 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.564 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.564 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.564 DEBUG:boto:Method: GET 2024-04-16T19:33:53.565 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.565 DEBUG:boto:Data: 2024-04-16T19:33:53.565 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.565 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.565 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.565 DEBUG:boto:Params: {} 2024-04-16T19:33:53.565 DEBUG:boto:Token: None 2024-04-16T19:33:53.565 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.566 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee894e4736d5f7f-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.569 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.569 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.569 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.569 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.569 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.570 DEBUG:boto:Method: GET 2024-04-16T19:33:53.570 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.570 DEBUG:boto:Data: 2024-04-16T19:33:53.570 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.570 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.571 DEBUG:boto:Params: {} 2024-04-16T19:33:53.571 DEBUG:boto:Token: None 2024-04-16T19:33:53.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU= 2024-04-16T19:33:53.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU=', 'Content-Length': '0'} 2024-04-16T19:33:53.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f91ac37eb68cce-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.615 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.615 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.615 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.615 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.615 DEBUG:boto:Method: GET 2024-04-16T19:33:53.615 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.615 DEBUG:boto:Data: 2024-04-16T19:33:53.615 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.616 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.616 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.616 DEBUG:boto:Params: {} 2024-04-16T19:33:53.616 DEBUG:boto:Token: None 2024-04-16T19:33:53.616 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU= 2024-04-16T19:33:53.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU=', 'Content-Length': '0'} 2024-04-16T19:33:53.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004352ba9dcb69d5ad-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.619 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.619 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.619 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.619 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.619 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.620 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.620 DEBUG:boto:Data: 2024-04-16T19:33:53.620 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.620 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.620 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.620 DEBUG:boto:Params: {} 2024-04-16T19:33:53.620 DEBUG:boto:Token: None 2024-04-16T19:33:53.620 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.621 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SMIxqSa02oho38fTqjYe93EmvqU= 2024-04-16T19:33:53.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SMIxqSa02oho38fTqjYe93EmvqU=', 'Content-Length': '0'} 2024-04-16T19:33:53.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005384e0b023645f23-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.623 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.623 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.624 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.624 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.624 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.624 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.624 DEBUG:boto:Data: 2024-04-16T19:33:53.624 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.624 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.624 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.624 DEBUG:boto:Params: {} 2024-04-16T19:33:53.625 DEBUG:boto:Token: None 2024-04-16T19:33:53.625 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.625 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SMIxqSa02oho38fTqjYe93EmvqU= 2024-04-16T19:33:53.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SMIxqSa02oho38fTqjYe93EmvqU=', 'Content-Length': '0'} 2024-04-16T19:33:53.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2aedfa92303ecea-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.627 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.628 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.628 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.628 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.628 DEBUG:boto:Method: GET 2024-04-16T19:33:53.628 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.628 DEBUG:boto:Data: 2024-04-16T19:33:53.628 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.628 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.629 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.629 DEBUG:boto:Params: {} 2024-04-16T19:33:53.629 DEBUG:boto:Token: None 2024-04-16T19:33:53.629 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU= 2024-04-16T19:33:53.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU=', 'Content-Length': '0'} 2024-04-16T19:33:53.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df5eb03e8592a4d2-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.675 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.675 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.675 DEBUG:boto:path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.675 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.675 DEBUG:boto:Method: GET 2024-04-16T19:33:53.675 DEBUG:boto:Path: /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.676 DEBUG:boto:Data: 2024-04-16T19:33:53.676 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.676 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.676 DEBUG:boto:Params: {} 2024-04-16T19:33:53.676 DEBUG:boto:Token: None 2024-04-16T19:33:53.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg 2024-04-16T19:33:53.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU= 2024-04-16T19:33:53.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwlLpe+dkDJ9ocJdEZ5R8aepvQU=', 'Content-Length': '0'} 2024-04-16T19:33:53.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', '6qxMhlyhJvcIxiHpEYtnaj9M6riEhpg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000afb123aa6e78152-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.679 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.680 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.680 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.680 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.680 DEBUG:boto:Method: GET 2024-04-16T19:33:53.680 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.680 DEBUG:boto:Data: 2024-04-16T19:33:53.680 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.680 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.680 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.681 DEBUG:boto:Params: {} 2024-04-16T19:33:53.681 DEBUG:boto:Token: None 2024-04-16T19:33:53.681 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b7b61a714b95c8-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.726 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.727 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.727 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.727 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.727 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.727 DEBUG:boto:Method: GET 2024-04-16T19:33:53.728 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.728 DEBUG:boto:Data: 2024-04-16T19:33:53.728 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.728 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.728 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.728 DEBUG:boto:Params: {} 2024-04-16T19:33:53.728 DEBUG:boto:Token: None 2024-04-16T19:33:53.728 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ee37e43f771f27-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.731 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.732 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.732 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.732 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.732 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.732 DEBUG:boto:Data: 2024-04-16T19:33:53.732 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.733 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.733 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.733 DEBUG:boto:Params: {} 2024-04-16T19:33:53.733 DEBUG:boto:Token: None 2024-04-16T19:33:53.733 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012494e4d4c429f4f-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.738 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.738 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.738 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.739 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.739 DEBUG:boto:Data: 2024-04-16T19:33:53.739 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.739 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.739 DEBUG:boto:Params: {} 2024-04-16T19:33:53.739 DEBUG:boto:Token: None 2024-04-16T19:33:53.739 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072cd6d2c4e41125d-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.744 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.744 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.745 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.745 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.745 DEBUG:boto:Method: GET 2024-04-16T19:33:53.745 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.745 DEBUG:boto:Data: 2024-04-16T19:33:53.745 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.745 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.745 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.745 DEBUG:boto:Params: {} 2024-04-16T19:33:53.746 DEBUG:boto:Token: None 2024-04-16T19:33:53.746 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.746 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc1f06a4c1622f71-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.749 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.749 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.749 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.749 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.749 DEBUG:boto:Method: GET 2024-04-16T19:33:53.750 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.750 DEBUG:boto:Data: 2024-04-16T19:33:53.750 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.750 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.750 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.750 DEBUG:boto:Params: {} 2024-04-16T19:33:53.750 DEBUG:boto:Token: None 2024-04-16T19:33:53.750 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d908a4c428de20f-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.794 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.795 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.795 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.795 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.795 DEBUG:boto:Method: GET 2024-04-16T19:33:53.795 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.795 DEBUG:boto:Data: 2024-04-16T19:33:53.795 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.795 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.795 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.795 DEBUG:boto:Params: {} 2024-04-16T19:33:53.796 DEBUG:boto:Token: None 2024-04-16T19:33:53.796 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dabab96cc6e2e52-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.799 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.799 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.799 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.800 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.800 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.800 DEBUG:boto:Method: GET 2024-04-16T19:33:53.800 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.800 DEBUG:boto:Data: 2024-04-16T19:33:53.800 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.800 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.800 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.800 DEBUG:boto:Params: {} 2024-04-16T19:33:53.800 DEBUG:boto:Token: None 2024-04-16T19:33:53.801 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6f8f59e49a9e430-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.803 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.804 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.804 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.804 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.804 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.804 DEBUG:boto:Method: GET 2024-04-16T19:33:53.804 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.804 DEBUG:boto:Data: 2024-04-16T19:33:53.805 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.805 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.805 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.805 DEBUG:boto:Params: {} 2024-04-16T19:33:53.805 DEBUG:boto:Token: None 2024-04-16T19:33:53.805 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8= 2024-04-16T19:33:53.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8=', 'Content-Length': '0'} 2024-04-16T19:33:53.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f4404fff83925c7-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.850 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.851 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.851 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.851 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.851 DEBUG:boto:Method: GET 2024-04-16T19:33:53.851 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.851 DEBUG:boto:Data: 2024-04-16T19:33:53.851 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.852 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.852 DEBUG:boto:Params: {} 2024-04-16T19:33:53.852 DEBUG:boto:Token: None 2024-04-16T19:33:53.852 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8= 2024-04-16T19:33:53.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8=', 'Content-Length': '0'} 2024-04-16T19:33:53.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045d4cfd995e1af98-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.899 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.899 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.899 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.899 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.899 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.899 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.899 DEBUG:boto:Data: 2024-04-16T19:33:53.899 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.900 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.900 DEBUG:boto:Params: {} 2024-04-16T19:33:53.900 DEBUG:boto:Token: None 2024-04-16T19:33:53.900 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gk6atTL71x2LXTcu0UQF51vP8U= 2024-04-16T19:33:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gk6atTL71x2LXTcu0UQF51vP8U=', 'Content-Length': '0'} 2024-04-16T19:33:53.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007df3c724a7614922-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.903 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.903 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.903 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.903 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.904 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.904 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.904 DEBUG:boto:Data: 2024-04-16T19:33:53.904 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.904 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.904 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.904 DEBUG:boto:Params: {} 2024-04-16T19:33:53.904 DEBUG:boto:Token: None 2024-04-16T19:33:53.905 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gk6atTL71x2LXTcu0UQF51vP8U= 2024-04-16T19:33:53.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gk6atTL71x2LXTcu0UQF51vP8U=', 'Content-Length': '0'} 2024-04-16T19:33:53.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a1102bebd52e499-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.908 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.908 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.908 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.908 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.908 DEBUG:boto:Method: GET 2024-04-16T19:33:53.908 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.908 DEBUG:boto:Data: 2024-04-16T19:33:53.909 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.909 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.909 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.909 DEBUG:boto:Params: {} 2024-04-16T19:33:53.909 DEBUG:boto:Token: None 2024-04-16T19:33:53.909 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.909 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8= 2024-04-16T19:33:53.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8=', 'Content-Length': '0'} 2024-04-16T19:33:53.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b55ef10e02ce9f46-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.913 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.913 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.913 DEBUG:boto:path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.913 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.913 DEBUG:boto:Method: GET 2024-04-16T19:33:53.914 DEBUG:boto:Path: /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.914 DEBUG:boto:Data: 2024-04-16T19:33:53.914 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.914 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.914 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.914 DEBUG:boto:Params: {} 2024-04-16T19:33:53.914 DEBUG:boto:Token: None 2024-04-16T19:33:53.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR 2024-04-16T19:33:53.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8= 2024-04-16T19:33:53.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tUJoykO7ljxrKJ1GQrHm9jtaWB8=', 'Content-Length': '0'} 2024-04-16T19:33:53.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'PLGsVNJxIrV2XrvVVQ-JZh8K0NJ62TR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd788f10f611dd73-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.959 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.959 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.959 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.959 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.959 DEBUG:boto:Method: GET 2024-04-16T19:33:53.959 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.959 DEBUG:boto:Data: 2024-04-16T19:33:53.960 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.960 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.960 DEBUG:boto:Params: {} 2024-04-16T19:33:53.960 DEBUG:boto:Token: None 2024-04-16T19:33:53.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000056e069adf2f3f60-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.964 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.964 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:53.965 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:53.965 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.965 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:53.965 DEBUG:boto:Method: GET 2024-04-16T19:33:53.965 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:53.965 DEBUG:boto:Data: 2024-04-16T19:33:53.965 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.965 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.966 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.966 DEBUG:boto:Params: {} 2024-04-16T19:33:53.966 DEBUG:boto:Token: None 2024-04-16T19:33:53.966 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/ 2024-04-16T19:33:53.966 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c= 2024-04-16T19:33:53.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fd9w/fGZ4UiRzcjWu+ZMVmKx07c=', 'Content-Length': '0'} 2024-04-16T19:33:53.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c9e17936bc40c9c-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.969 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:53.970 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.970 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.970 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.970 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.970 DEBUG:boto:Data: 2024-04-16T19:33:53.971 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.971 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.971 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.971 DEBUG:boto:Params: {} 2024-04-16T19:33:53.971 DEBUG:boto:Token: None 2024-04-16T19:33:53.971 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0ce0fa3edbb1e1-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.976 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.976 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.976 DEBUG:boto:Method: HEAD 2024-04-16T19:33:53.977 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:53.977 DEBUG:boto:Data: 2024-04-16T19:33:53.977 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.977 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:53.977 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.977 DEBUG:boto:Params: {} 2024-04-16T19:33:53.977 DEBUG:boto:Token: None 2024-04-16T19:33:53.978 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj 2024-04-16T19:33:53.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY= 2024-04-16T19:33:53.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oGC5sOyEyI7ir8HjzDxbvFfoJEY=', 'Content-Length': '0'} 2024-04-16T19:33:53.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd39f4eefb1a7d15-00661ed2a1-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:53.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:53.983 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:53.983 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:53.983 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.984 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.984 DEBUG:boto:Method: GET 2024-04-16T19:33:53.984 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.984 DEBUG:boto:Data: 2024-04-16T19:33:53.984 DEBUG:boto:Headers: {} 2024-04-16T19:33:53.984 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:53.984 DEBUG:boto:Port: 8000 2024-04-16T19:33:53.984 DEBUG:boto:Params: {} 2024-04-16T19:33:53.984 DEBUG:boto:Token: None 2024-04-16T19:33:53.985 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:53 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:53.985 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E= 2024-04-16T19:33:53.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5CJd4ACnehorHCEaxN52ai5M4E=', 'Content-Length': '0'} 2024-04-16T19:33:53.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079f4c82c0332260e-00661ed2a1-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.035 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.035 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.035 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.035 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.035 DEBUG:boto:Method: GET 2024-04-16T19:33:54.035 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.035 DEBUG:boto:Data: 2024-04-16T19:33:54.036 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.036 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.036 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.036 DEBUG:boto:Params: {} 2024-04-16T19:33:54.036 DEBUG:boto:Token: None 2024-04-16T19:33:54.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs= 2024-04-16T19:33:54.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs=', 'Content-Length': '0'} 2024-04-16T19:33:54.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e42d117484f9655c-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.082 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.083 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.083 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.083 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.083 DEBUG:boto:Method: GET 2024-04-16T19:33:54.083 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.083 DEBUG:boto:Data: 2024-04-16T19:33:54.083 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.084 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.084 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.084 DEBUG:boto:Params: {} 2024-04-16T19:33:54.084 DEBUG:boto:Token: None 2024-04-16T19:33:54.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ff933fa23dcffc-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.088 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.088 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.088 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.088 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.088 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.089 DEBUG:boto:Method: GET 2024-04-16T19:33:54.089 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.089 DEBUG:boto:Data: 2024-04-16T19:33:54.089 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.089 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.089 DEBUG:boto:Params: {} 2024-04-16T19:33:54.089 DEBUG:boto:Token: None 2024-04-16T19:33:54.090 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd83a7cac6ee8353-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.134 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.135 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.135 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.135 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.136 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.136 DEBUG:boto:Method: GET 2024-04-16T19:33:54.136 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.136 DEBUG:boto:Data: 2024-04-16T19:33:54.136 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.136 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.136 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.136 DEBUG:boto:Params: {} 2024-04-16T19:33:54.137 DEBUG:boto:Token: None 2024-04-16T19:33:54.137 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.137 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8= 2024-04-16T19:33:54.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8=', 'Content-Length': '0'} 2024-04-16T19:33:54.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fcf5a46cd22cfcd-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.140 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.140 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.140 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.140 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.140 DEBUG:boto:Method: GET 2024-04-16T19:33:54.140 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.141 DEBUG:boto:Data: 2024-04-16T19:33:54.141 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.141 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.141 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.141 DEBUG:boto:Params: {} 2024-04-16T19:33:54.141 DEBUG:boto:Token: None 2024-04-16T19:33:54.141 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8= 2024-04-16T19:33:54.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8=', 'Content-Length': '0'} 2024-04-16T19:33:54.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000998823950b91af7-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.186 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.187 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.187 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.187 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.187 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.187 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.187 DEBUG:boto:Data: 2024-04-16T19:33:54.187 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.188 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.188 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.188 DEBUG:boto:Params: {} 2024-04-16T19:33:54.188 DEBUG:boto:Token: None 2024-04-16T19:33:54.188 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XBLw6DtdXKxMddQDbn5wETvIsfw= 2024-04-16T19:33:54.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XBLw6DtdXKxMddQDbn5wETvIsfw=', 'Content-Length': '0'} 2024-04-16T19:33:54.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006517004afdce269d-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.191 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.192 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.192 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.192 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.192 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.192 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.192 DEBUG:boto:Data: 2024-04-16T19:33:54.192 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.193 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.193 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.193 DEBUG:boto:Params: {} 2024-04-16T19:33:54.193 DEBUG:boto:Token: None 2024-04-16T19:33:54.193 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XBLw6DtdXKxMddQDbn5wETvIsfw= 2024-04-16T19:33:54.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XBLw6DtdXKxMddQDbn5wETvIsfw=', 'Content-Length': '0'} 2024-04-16T19:33:54.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fcc398072772bf9-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.196 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.197 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.197 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.197 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.197 DEBUG:boto:Method: GET 2024-04-16T19:33:54.197 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.197 DEBUG:boto:Data: 2024-04-16T19:33:54.197 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.197 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.198 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.198 DEBUG:boto:Params: {} 2024-04-16T19:33:54.198 DEBUG:boto:Token: None 2024-04-16T19:33:54.198 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.198 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8= 2024-04-16T19:33:54.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8=', 'Content-Length': '0'} 2024-04-16T19:33:54.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c9173270ffa9a00-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.201 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.201 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.201 DEBUG:boto:path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.202 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.202 DEBUG:boto:Method: GET 2024-04-16T19:33:54.202 DEBUG:boto:Path: /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.202 DEBUG:boto:Data: 2024-04-16T19:33:54.202 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.202 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.202 DEBUG:boto:Params: {} 2024-04-16T19:33:54.203 DEBUG:boto:Token: None 2024-04-16T19:33:54.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi 2024-04-16T19:33:54.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8= 2024-04-16T19:33:54.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iY05LJHohWUCKxqFi6CbCGc/3f8=', 'Content-Length': '0'} 2024-04-16T19:33:54.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'JfyqEJD8sqQTmgMG3Eku8wfrvQvaTFi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:25 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042b5a4dc3bdf869d-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.246 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.247 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.247 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.247 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.247 DEBUG:boto:Method: GET 2024-04-16T19:33:54.247 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.247 DEBUG:boto:Data: 2024-04-16T19:33:54.247 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.248 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.248 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.248 DEBUG:boto:Params: {} 2024-04-16T19:33:54.248 DEBUG:boto:Token: None 2024-04-16T19:33:54.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650f18246fb957f4-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.251 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.252 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.252 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.252 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.252 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.252 DEBUG:boto:Method: GET 2024-04-16T19:33:54.253 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.253 DEBUG:boto:Data: 2024-04-16T19:33:54.253 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.253 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.253 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.253 DEBUG:boto:Params: {} 2024-04-16T19:33:54.253 DEBUG:boto:Token: None 2024-04-16T19:33:54.253 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.254 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b66c21121037bba-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.256 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.257 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.257 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.257 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.257 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:54.257 DEBUG:boto:Data: 2024-04-16T19:33:54.257 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.258 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.258 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.258 DEBUG:boto:Params: {} 2024-04-16T19:33:54.258 DEBUG:boto:Token: None 2024-04-16T19:33:54.258 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj 2024-04-16T19:33:54.258 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk= 2024-04-16T19:33:54.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk=', 'Content-Length': '0'} 2024-04-16T19:33:54.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023cf553b728cbdba-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.263 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.263 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.263 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.263 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:54.264 DEBUG:boto:Data: 2024-04-16T19:33:54.264 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.264 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.264 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.264 DEBUG:boto:Params: {} 2024-04-16T19:33:54.264 DEBUG:boto:Token: None 2024-04-16T19:33:54.264 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj 2024-04-16T19:33:54.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk= 2024-04-16T19:33:54.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk=', 'Content-Length': '0'} 2024-04-16T19:33:54.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000757593b51e9453b-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.269 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.269 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.269 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.269 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.270 DEBUG:boto:Method: GET 2024-04-16T19:33:54.270 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.270 DEBUG:boto:Data: 2024-04-16T19:33:54.270 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.270 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.270 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.270 DEBUG:boto:Params: {} 2024-04-16T19:33:54.270 DEBUG:boto:Token: None 2024-04-16T19:33:54.271 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs= 2024-04-16T19:33:54.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs=', 'Content-Length': '0'} 2024-04-16T19:33:54.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5720f4350cbc1e9-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.315 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.315 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.315 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.315 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.315 DEBUG:boto:Method: GET 2024-04-16T19:33:54.315 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.315 DEBUG:boto:Data: 2024-04-16T19:33:54.315 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.316 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.316 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.316 DEBUG:boto:Params: {} 2024-04-16T19:33:54.316 DEBUG:boto:Token: None 2024-04-16T19:33:54.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs= 2024-04-16T19:33:54.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs=', 'Content-Length': '0'} 2024-04-16T19:33:54.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b69d4fe54968d4e-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.363 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.363 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.363 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.363 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.363 DEBUG:boto:Method: GET 2024-04-16T19:33:54.363 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.364 DEBUG:boto:Data: 2024-04-16T19:33:54.364 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.364 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.364 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.364 DEBUG:boto:Params: {} 2024-04-16T19:33:54.364 DEBUG:boto:Token: None 2024-04-16T19:33:54.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed25eece762e7894-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.368 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.368 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.368 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.369 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.369 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.369 DEBUG:boto:Method: GET 2024-04-16T19:33:54.369 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.369 DEBUG:boto:Data: 2024-04-16T19:33:54.369 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.369 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.369 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.370 DEBUG:boto:Params: {} 2024-04-16T19:33:54.370 DEBUG:boto:Token: None 2024-04-16T19:33:54.370 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.370 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98f14581ac89abb-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.414 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.415 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.415 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.415 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.415 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.416 DEBUG:boto:Method: GET 2024-04-16T19:33:54.416 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.416 DEBUG:boto:Data: 2024-04-16T19:33:54.416 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.416 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.416 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.416 DEBUG:boto:Params: {} 2024-04-16T19:33:54.417 DEBUG:boto:Token: None 2024-04-16T19:33:54.417 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU= 2024-04-16T19:33:54.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU=', 'Content-Length': '0'} 2024-04-16T19:33:54.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f1e2f6510fc1c6a-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.462 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.463 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.463 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.463 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.463 DEBUG:boto:Method: GET 2024-04-16T19:33:54.463 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.463 DEBUG:boto:Data: 2024-04-16T19:33:54.463 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.464 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.464 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.464 DEBUG:boto:Params: {} 2024-04-16T19:33:54.464 DEBUG:boto:Token: None 2024-04-16T19:33:54.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU= 2024-04-16T19:33:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU=', 'Content-Length': '0'} 2024-04-16T19:33:54.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000818d75fed647a77e-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.467 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.467 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.467 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.468 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.468 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.468 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.468 DEBUG:boto:Data: 2024-04-16T19:33:54.468 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.468 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.468 DEBUG:boto:Params: {} 2024-04-16T19:33:54.469 DEBUG:boto:Token: None 2024-04-16T19:33:54.469 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NBpOyxshRXRWmIJMxYxeFfowclo= 2024-04-16T19:33:54.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NBpOyxshRXRWmIJMxYxeFfowclo=', 'Content-Length': '0'} 2024-04-16T19:33:54.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28713f49a449613-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.472 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.472 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.472 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.472 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.472 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.472 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.473 DEBUG:boto:Data: 2024-04-16T19:33:54.473 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.473 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.473 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.473 DEBUG:boto:Params: {} 2024-04-16T19:33:54.473 DEBUG:boto:Token: None 2024-04-16T19:33:54.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NBpOyxshRXRWmIJMxYxeFfowclo= 2024-04-16T19:33:54.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NBpOyxshRXRWmIJMxYxeFfowclo=', 'Content-Length': '0'} 2024-04-16T19:33:54.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095443edb11e6a0f4-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.476 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.476 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.477 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.477 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.477 DEBUG:boto:Method: GET 2024-04-16T19:33:54.477 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.477 DEBUG:boto:Data: 2024-04-16T19:33:54.477 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.477 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.478 DEBUG:boto:Params: {} 2024-04-16T19:33:54.478 DEBUG:boto:Token: None 2024-04-16T19:33:54.478 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.478 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU= 2024-04-16T19:33:54.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU=', 'Content-Length': '0'} 2024-04-16T19:33:54.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070859678629ee3fe-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.522 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.523 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.523 DEBUG:boto:path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.523 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.523 DEBUG:boto:Method: GET 2024-04-16T19:33:54.523 DEBUG:boto:Path: /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.523 DEBUG:boto:Data: 2024-04-16T19:33:54.523 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.524 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.524 DEBUG:boto:Params: {} 2024-04-16T19:33:54.524 DEBUG:boto:Token: None 2024-04-16T19:33:54.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu 2024-04-16T19:33:54.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU= 2024-04-16T19:33:54.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wnOeG2Y6lPxMUuQU4PjOW7ypzTU=', 'Content-Length': '0'} 2024-04-16T19:33:54.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:23 GMT'), ('x-amz-version-id', 'gFRR0OYRsUkl6HocRmPvamoSSS8O0Tu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000289770863a5bc5aa-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.527 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.527 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.527 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.527 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.527 DEBUG:boto:Method: GET 2024-04-16T19:33:54.527 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.527 DEBUG:boto:Data: 2024-04-16T19:33:54.528 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.528 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.528 DEBUG:boto:Params: {} 2024-04-16T19:33:54.528 DEBUG:boto:Token: None 2024-04-16T19:33:54.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a0d897a357d19b-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.574 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.575 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.575 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.575 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.575 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.575 DEBUG:boto:Method: GET 2024-04-16T19:33:54.576 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.576 DEBUG:boto:Data: 2024-04-16T19:33:54.576 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.576 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.576 DEBUG:boto:Params: {} 2024-04-16T19:33:54.576 DEBUG:boto:Token: None 2024-04-16T19:33:54.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.577 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1657c2155de1b66-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.580 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.580 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.580 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.580 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.580 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:54.581 DEBUG:boto:Data: 2024-04-16T19:33:54.581 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.581 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.581 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.581 DEBUG:boto:Params: {} 2024-04-16T19:33:54.581 DEBUG:boto:Token: None 2024-04-16T19:33:54.581 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj 2024-04-16T19:33:54.581 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk= 2024-04-16T19:33:54.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk=', 'Content-Length': '0'} 2024-04-16T19:33:54.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006571b87f844a43a7-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.587 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.587 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.587 DEBUG:boto:Method: HEAD 2024-04-16T19:33:54.587 DEBUG:boto:Path: /yrjefh-15/obj 2024-04-16T19:33:54.587 DEBUG:boto:Data: 2024-04-16T19:33:54.587 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.587 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.588 DEBUG:boto:Params: {} 2024-04-16T19:33:54.588 DEBUG:boto:Token: None 2024-04-16T19:33:54.588 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj 2024-04-16T19:33:54.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk= 2024-04-16T19:33:54.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6InybsRlZnM/2snZoJxNWSARhlk=', 'Content-Length': '0'} 2024-04-16T19:33:54.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfeaf172d37b0a00-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:33:54.593 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.593 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.594 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.594 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.594 DEBUG:boto:Method: GET 2024-04-16T19:33:54.594 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.594 DEBUG:boto:Data: 2024-04-16T19:33:54.594 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.594 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.595 DEBUG:boto:Params: {} 2024-04-16T19:33:54.595 DEBUG:boto:Token: None 2024-04-16T19:33:54.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs= 2024-04-16T19:33:54.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs=', 'Content-Length': '0'} 2024-04-16T19:33:54.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d01bbf56daf23a1a-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.598 DEBUG:boto:path=/yrjefh-15/obj 2024-04-16T19:33:54.598 DEBUG:boto:auth_path=/yrjefh-15/obj 2024-04-16T19:33:54.598 DEBUG:boto:path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.598 DEBUG:boto:auth_path=/yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.598 DEBUG:boto:Method: GET 2024-04-16T19:33:54.599 DEBUG:boto:Path: /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.599 DEBUG:boto:Data: 2024-04-16T19:33:54.599 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.599 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.599 DEBUG:boto:Params: {} 2024-04-16T19:33:54.599 DEBUG:boto:Token: None 2024-04-16T19:33:54.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/obj?versionId=CMUP7qgLDH-ugNM44s2aPhnev-gtpbT 2024-04-16T19:33:54.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs= 2024-04-16T19:33:54.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc0yYFNR9Wr548snFgQN4vJhrPs=', 'Content-Length': '0'} 2024-04-16T19:33:54.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:24 GMT'), ('x-amz-version-id', 'CMUP7qgLDH-ugNM44s2aPhnev-gtpbT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:45 GMT'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d31afa0e53c51cb8-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.642 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.643 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.643 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.643 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.643 DEBUG:boto:Method: GET 2024-04-16T19:33:54.643 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.643 DEBUG:boto:Data: 2024-04-16T19:33:54.643 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:33:54.644 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.644 DEBUG:boto:Params: {} 2024-04-16T19:33:54.644 DEBUG:boto:Token: None 2024-04-16T19:33:54.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f756a332d6290d-00661ed2a2-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.647 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.648 DEBUG:boto:path=/yrjefh-15/ 2024-04-16T19:33:54.648 DEBUG:boto:auth_path=/yrjefh-15/ 2024-04-16T19:33:54.648 DEBUG:boto:path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.648 DEBUG:boto:auth_path=/yrjefh-15/?prefix=obj 2024-04-16T19:33:54.648 DEBUG:boto:Method: GET 2024-04-16T19:33:54.648 DEBUG:boto:Path: /yrjefh-15/?prefix=obj 2024-04-16T19:33:54.648 DEBUG:boto:Data: 2024-04-16T19:33:54.648 DEBUG:boto:Headers: {} 2024-04-16T19:33:54.649 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:54.649 DEBUG:boto:Port: 8000 2024-04-16T19:33:54.649 DEBUG:boto:Params: {} 2024-04-16T19:33:54.649 DEBUG:boto:Token: None 2024-04-16T19:33:54.649 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:33:54 GMT /yrjefh-15/ 2024-04-16T19:33:54.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE= 2024-04-16T19:33:54.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kCrDDGlQCvObVec+hvDqh1TeEkE=', 'Content-Length': '0'} 2024-04-16T19:33:54.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008c0809d0977174-00661ed2a2-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:54.652 DEBUG:boto:b'yrjefh-15obj1000falseobj2024-04-16T19:33:24.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDRGW11111111111111111Normal' 2024-04-16T19:33:54.653 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-15 zones={test-zone2, test-zone1} 2024-04-16T19:33:54.653 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T19:33:54.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.654 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:54.708 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:54.708 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:54.791 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000003466","last_update":"2024-04-16T19:27:00.929227Z"},{"marker":"00000000000000000000:00000000000000002623","last_update":"2024-04-16T19:32:08.938133Z"},{"marker":"00000000000000000000:00000000000000002905","last_update":"2024-04-16T19:33:36.944758Z"},{"marker":"00000000000000000000:00000000000000004458","last_update":"2024-04-16T19:33:36.946564Z"}] 2024-04-16T19:33:54.792 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:33:54.792 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:33:54.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.792 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:54.850 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:54.850 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:54.926 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:27:00.929227Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.944758Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004458","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.946564Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:33:54Z"} 2024-04-16T19:33:54.926 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:27:00.929227Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.944758Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004458","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.946564Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:33:54Z"} 2024-04-16T19:33:54.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:27:00.929227Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:36.944758Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004458', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:36.946564Z'}}] 2024-04-16T19:33:54.926 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:33:54.927 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:33:54.927 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:33:54.927 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T19:33:54.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:54.927 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:54.974 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:54.975 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:55.049 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000003466","last_update":"2024-04-16T19:27:07.112638Z"},{"marker":"00000000000000000000:00000000000000002905","last_update":"2024-04-16T19:33:06.384483Z"},{"marker":"00000000000000000000:00000000000000002905","last_update":"2024-04-16T19:33:43.128578Z"},{"marker":"00000000000000000000:00000000000000004035","last_update":"2024-04-16T19:33:21.126136Z"}] 2024-04-16T19:33:55.050 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:33:55.050 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:33:55.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.051 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:55.103 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:55.103 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:55.177 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:27:07.112638Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:43.128578Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:33:55Z"} 2024-04-16T19:33:55.177 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:27:07.112638Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:43.128578Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:33:55Z"} 2024-04-16T19:33:55.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:27:07.112638Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:06.384483Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:43.128578Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004035', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:33:21.126136Z'}}] 2024-04-16T19:33:55.177 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:33:55.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003466', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:33:55.178 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:33:55.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-16T19:33:55.180 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:33:55.180 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:33:55.180 DEBUG:boto:Method: PUT 2024-04-16T19:33:55.180 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:33:55.180 DEBUG:boto:Data: 2024-04-16T19:33:55.180 DEBUG:boto:Headers: {} 2024-04-16T19:33:55.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:55.181 DEBUG:boto:Port: 8000 2024-04-16T19:33:55.181 DEBUG:boto:Params: {} 2024-04-16T19:33:55.181 DEBUG:boto:Token: None 2024-04-16T19:33:55.181 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:33:55 GMT /yrjefh-16/ 2024-04-16T19:33:55.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jvedaBCy0Relul7pOgb4lUnOXLA= 2024-04-16T19:33:55.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jvedaBCy0Relul7pOgb4lUnOXLA=', 'Content-Length': '0'} 2024-04-16T19:33:55.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005cb7846235cc11ac-00661ed2a3-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:55.198 DEBUG:rgw_multi.tests:created bucket=yrjefh-16 2024-04-16T19:33:55.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.198 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:55.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:55.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:55.294 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:33:55.316 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296035.194919_139.1","last_update":"2024-04-16T19:33:55.194919Z"}] 2024-04-16T19:33:55.316 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296035.194919_139.1'} 2024-04-16T19:33:55.316 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:33:55.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:33:55.317 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:33:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:55.411+0000 7fe966d2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:33:55.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:55.412+0000 7fe966d2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:33:55.434 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296035.194919_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:55.194919Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:33:55Z"} 2024-04-16T19:33:55.435 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296035.194919_139.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:55.194919Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:33:55Z"} 2024-04-16T19:33:55.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296035.194919_139.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:55.194919Z', 'realm_epoch': 2}}] 2024-04-16T19:33:55.435 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296035.194919_139.1'} 2024-04-16T19:33:55.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296035.194919_139.1'} 2024-04-16T19:33:55.435 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:33:55.435 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:33:55.436 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:33:55.436 DEBUG:boto:Method: HEAD 2024-04-16T19:33:55.436 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:33:55.436 DEBUG:boto:Data: 2024-04-16T19:33:55.436 DEBUG:boto:Headers: {} 2024-04-16T19:33:55.436 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:55.436 DEBUG:boto:Port: 8000 2024-04-16T19:33:55.436 DEBUG:boto:Params: {} 2024-04-16T19:33:55.436 DEBUG:boto:Token: None 2024-04-16T19:33:55.437 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:33:55 GMT /yrjefh-16/ 2024-04-16T19:33:55.437 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jASyZI5kmqcWUuXqwDsijNz0RO0= 2024-04-16T19:33:55.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:33:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jASyZI5kmqcWUuXqwDsijNz0RO0=', 'Content-Length': '0'} 2024-04-16T19:33:55.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005865693bf22b809c-00661ed2a3-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:55.440 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:33:55.440 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:33:55.440 DEBUG:boto:Method: PUT 2024-04-16T19:33:55.440 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:33:55.440 DEBUG:boto:Data: 2024-04-16T19:33:55.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:33:55.440 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:33:55.440 DEBUG:boto:Port: 8000 2024-04-16T19:33:55.440 DEBUG:boto:Params: {} 2024-04-16T19:33:55.440 DEBUG:boto:Token: None 2024-04-16T19:33:55.441 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:33:55 GMT /yrjefh-16/obj 2024-04-16T19:33:55.441 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+zyKyse1KMK6Z8EEcmDuGX1ggUI= 2024-04-16T19:33:55.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:33:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+zyKyse1KMK6Z8EEcmDuGX1ggUI='} 2024-04-16T19:33:55.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f930760bd46d504-00661ed2a3-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:33:55.447 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-16T19:33:55.447 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-16 2024-04-16T19:33:55.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:55.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:33:55.448 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:33:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:33:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:33:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:33:55.556+0000 7f6525844800 1 waiting to reach incremental sync.. 2024-04-16T19:34:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:00.559+0000 7f6525844800 1 bucket sync caught up with source: 2024-04-16T19:34:00.561 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:00.561 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:00.559+0000 7f6525844800 0 bucket checkpoint complete 2024-04-16T19:34:00.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-16 2024-04-16T19:34:00.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:34:00.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:34:00.585 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:34:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:34:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:34:00.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:34:00.707+0000 7f258ba3c800 1 bucket sync caught up with source: 2024-04-16T19:34:00.708 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-16T19:34:00.709 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.30.6, , ] 2024-04-16T19:34:00.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:34:00.707+0000 7f258ba3c800 0 bucket checkpoint complete 2024-04-16T19:34:00.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:34:00.730 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.730 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.730 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.730 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:34:00.730 DEBUG:boto:Data: 2024-04-16T19:34:00.730 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.730 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.730 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.730 DEBUG:boto:Params: {} 2024-04-16T19:34:00.731 DEBUG:boto:Token: None 2024-04-16T19:34:00.731 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IvSK8wKMQKJ2qdPeepvz19GGlJA= 2024-04-16T19:34:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IvSK8wKMQKJ2qdPeepvz19GGlJA=', 'Content-Length': '0'} 2024-04-16T19:34:00.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000738d052c106ac17c-00661ed2a8-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.734 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.734 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.734 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.734 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:34:00.734 DEBUG:boto:Data: 2024-04-16T19:34:00.734 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.734 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.734 DEBUG:boto:Params: {} 2024-04-16T19:34:00.734 DEBUG:boto:Token: None 2024-04-16T19:34:00.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IvSK8wKMQKJ2qdPeepvz19GGlJA= 2024-04-16T19:34:00.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IvSK8wKMQKJ2qdPeepvz19GGlJA=', 'Content-Length': '0'} 2024-04-16T19:34:00.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070caed3b720fc1a3-00661ed2a8-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.737 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:34:00.737 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.737 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.738 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.738 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.738 DEBUG:boto:Method: GET 2024-04-16T19:34:00.738 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.738 DEBUG:boto:Data: 2024-04-16T19:34:00.738 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.738 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.738 DEBUG:boto:Params: {} 2024-04-16T19:34:00.738 DEBUG:boto:Token: None 2024-04-16T19:34:00.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/?versions 2024-04-16T19:34:00.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU= 2024-04-16T19:34:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU=', 'Content-Length': '0'} 2024-04-16T19:34:00.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4836f5cb03282a-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.741 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:34:00.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:34:00.742 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.742 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.742 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.742 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.742 DEBUG:boto:Method: GET 2024-04-16T19:34:00.742 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.742 DEBUG:boto:Data: 2024-04-16T19:34:00.742 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.743 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.743 DEBUG:boto:Params: {} 2024-04-16T19:34:00.743 DEBUG:boto:Token: None 2024-04-16T19:34:00.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/?versions 2024-04-16T19:34:00.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU= 2024-04-16T19:34:00.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU=', 'Content-Length': '0'} 2024-04-16T19:34:00.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096c05d55035fdfbe-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.746 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:34:00.746 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.746 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.747 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.747 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.747 DEBUG:boto:Method: GET 2024-04-16T19:34:00.747 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.747 DEBUG:boto:Data: 2024-04-16T19:34:00.747 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.747 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.747 DEBUG:boto:Params: {} 2024-04-16T19:34:00.747 DEBUG:boto:Token: None 2024-04-16T19:34:00.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/?versions 2024-04-16T19:34:00.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU= 2024-04-16T19:34:00.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU=', 'Content-Length': '0'} 2024-04-16T19:34:00.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbb6edcf7ecf841-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.750 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.750 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.750 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.750 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.751 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.751 DEBUG:boto:Method: GET 2024-04-16T19:34:00.751 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:34:00.751 DEBUG:boto:Data: 2024-04-16T19:34:00.751 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.751 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.751 DEBUG:boto:Params: {} 2024-04-16T19:34:00.751 DEBUG:boto:Token: None 2024-04-16T19:34:00.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/?versions 2024-04-16T19:34:00.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU= 2024-04-16T19:34:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hvm33Lj5eJ04L4/OszeaDPP/IKU=', 'Content-Length': '0'} 2024-04-16T19:34:00.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009680052303a4cb08-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.754 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:34:00.754 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.755 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.755 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.755 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.755 DEBUG:boto:Method: GET 2024-04-16T19:34:00.755 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.755 DEBUG:boto:Data: 2024-04-16T19:34:00.755 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.755 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.755 DEBUG:boto:Params: {} 2024-04-16T19:34:00.755 DEBUG:boto:Token: None 2024-04-16T19:34:00.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0= 2024-04-16T19:34:00.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0=', 'Content-Length': '0'} 2024-04-16T19:34:00.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a5b59dbbd639838-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.758 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.758 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.758 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.758 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.758 DEBUG:boto:Method: GET 2024-04-16T19:34:00.758 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.758 DEBUG:boto:Data: 2024-04-16T19:34:00.758 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.758 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.758 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.758 DEBUG:boto:Params: {} 2024-04-16T19:34:00.758 DEBUG:boto:Token: None 2024-04-16T19:34:00.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0= 2024-04-16T19:34:00.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0=', 'Content-Length': '0'} 2024-04-16T19:34:00.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a92010c24bdad8be-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.760 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.761 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.761 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.761 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.761 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.761 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.761 DEBUG:boto:Data: 2024-04-16T19:34:00.761 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.761 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.761 DEBUG:boto:Params: {} 2024-04-16T19:34:00.761 DEBUG:boto:Token: None 2024-04-16T19:34:00.762 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ufCDcK2lygdpt5LS5DNkO08r3ZU= 2024-04-16T19:34:00.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ufCDcK2lygdpt5LS5DNkO08r3ZU=', 'Content-Length': '0'} 2024-04-16T19:34:00.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005fbf7727a753525f-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.764 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.764 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.764 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.764 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.764 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.764 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.764 DEBUG:boto:Data: 2024-04-16T19:34:00.764 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.764 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.764 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.764 DEBUG:boto:Params: {} 2024-04-16T19:34:00.765 DEBUG:boto:Token: None 2024-04-16T19:34:00.765 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ufCDcK2lygdpt5LS5DNkO08r3ZU= 2024-04-16T19:34:00.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ufCDcK2lygdpt5LS5DNkO08r3ZU=', 'Content-Length': '0'} 2024-04-16T19:34:00.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d8c59b65c7bb8ce-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:34:00.766 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.767 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.767 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.767 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.767 DEBUG:boto:Method: GET 2024-04-16T19:34:00.767 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.767 DEBUG:boto:Data: 2024-04-16T19:34:00.767 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.767 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.767 DEBUG:boto:Params: {} 2024-04-16T19:34:00.767 DEBUG:boto:Token: None 2024-04-16T19:34:00.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0= 2024-04-16T19:34:00.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0=', 'Content-Length': '0'} 2024-04-16T19:34:00.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b0318ab8485937d-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.770 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.770 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.770 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.770 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.770 DEBUG:boto:Method: GET 2024-04-16T19:34:00.770 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.770 DEBUG:boto:Data: 2024-04-16T19:34:00.770 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.770 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.770 DEBUG:boto:Params: {} 2024-04-16T19:34:00.771 DEBUG:boto:Token: None 2024-04-16T19:34:00.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:34:00.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0= 2024-04-16T19:34:00.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:spXvFskahAWxGN8qb8bSZncdLZ0=', 'Content-Length': '0'} 2024-04-16T19:34:00.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049ce5183e95b6e75-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.772 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.773 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.773 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.773 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.773 DEBUG:boto:Method: GET 2024-04-16T19:34:00.773 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:34:00.773 DEBUG:boto:Data: 2024-04-16T19:34:00.773 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.773 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.773 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.773 DEBUG:boto:Params: {} 2024-04-16T19:34:00.773 DEBUG:boto:Token: None 2024-04-16T19:34:00.773 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek= 2024-04-16T19:34:00.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek=', 'Content-Length': '0'} 2024-04-16T19:34:00.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f90d0ee3af3714-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.776 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.776 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.777 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.777 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.777 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.777 DEBUG:boto:Method: GET 2024-04-16T19:34:00.777 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:34:00.777 DEBUG:boto:Data: 2024-04-16T19:34:00.777 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.777 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.777 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.777 DEBUG:boto:Params: {} 2024-04-16T19:34:00.777 DEBUG:boto:Token: None 2024-04-16T19:34:00.778 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.778 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek= 2024-04-16T19:34:00.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek=', 'Content-Length': '0'} 2024-04-16T19:34:00.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be68ee09bc26bd64-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.780 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.780 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.781 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.781 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.781 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:34:00.781 DEBUG:boto:Data: 2024-04-16T19:34:00.781 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.781 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.781 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.781 DEBUG:boto:Params: {} 2024-04-16T19:34:00.781 DEBUG:boto:Token: None 2024-04-16T19:34:00.781 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj 2024-04-16T19:34:00.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WxpDD/Lhot9j/tfGRl5tC9/ddYI= 2024-04-16T19:34:00.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WxpDD/Lhot9j/tfGRl5tC9/ddYI=', 'Content-Length': '0'} 2024-04-16T19:34:00.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072fd8245f6f3386d-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.783 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.784 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.784 DEBUG:boto:Method: HEAD 2024-04-16T19:34:00.784 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:34:00.784 DEBUG:boto:Data: 2024-04-16T19:34:00.784 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.784 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.784 DEBUG:boto:Params: {} 2024-04-16T19:34:00.784 DEBUG:boto:Token: None 2024-04-16T19:34:00.784 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj 2024-04-16T19:34:00.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WxpDD/Lhot9j/tfGRl5tC9/ddYI= 2024-04-16T19:34:00.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WxpDD/Lhot9j/tfGRl5tC9/ddYI=', 'Content-Length': '0'} 2024-04-16T19:34:00.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cec478c34e9a5797-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:34:00.786 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.787 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.787 DEBUG:boto:Method: GET 2024-04-16T19:34:00.787 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:34:00.787 DEBUG:boto:Data: 2024-04-16T19:34:00.787 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.787 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.787 DEBUG:boto:Params: {} 2024-04-16T19:34:00.787 DEBUG:boto:Token: None 2024-04-16T19:34:00.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj 2024-04-16T19:34:00.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9m4DRNQU+yhMhhZoxfj9uRw6Jp4= 2024-04-16T19:34:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9m4DRNQU+yhMhhZoxfj9uRw6Jp4=', 'Content-Length': '0'} 2024-04-16T19:34:00.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7f1921629993c68-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.790 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:00.790 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:00.790 DEBUG:boto:Method: GET 2024-04-16T19:34:00.790 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:34:00.790 DEBUG:boto:Data: 2024-04-16T19:34:00.790 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.790 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.790 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.790 DEBUG:boto:Params: {} 2024-04-16T19:34:00.791 DEBUG:boto:Token: None 2024-04-16T19:34:00.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/obj 2024-04-16T19:34:00.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9m4DRNQU+yhMhhZoxfj9uRw6Jp4= 2024-04-16T19:34:00.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9m4DRNQU+yhMhhZoxfj9uRw6Jp4=', 'Content-Length': '0'} 2024-04-16T19:34:00.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb5fed7940429f02-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.793 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.793 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.793 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.793 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.793 DEBUG:boto:Method: GET 2024-04-16T19:34:00.793 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:34:00.793 DEBUG:boto:Data: 2024-04-16T19:34:00.793 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.793 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:34:00.793 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.794 DEBUG:boto:Params: {} 2024-04-16T19:34:00.794 DEBUG:boto:Token: None 2024-04-16T19:34:00.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek= 2024-04-16T19:34:00.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek=', 'Content-Length': '0'} 2024-04-16T19:34:00.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef8d6022974475d-00661ed2a8-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.838 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.839 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.839 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.839 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.839 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:34:00.839 DEBUG:boto:Method: GET 2024-04-16T19:34:00.839 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:34:00.839 DEBUG:boto:Data: 2024-04-16T19:34:00.839 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.839 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.840 DEBUG:boto:Params: {} 2024-04-16T19:34:00.840 DEBUG:boto:Token: None 2024-04-16T19:34:00.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/ 2024-04-16T19:34:00.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek= 2024-04-16T19:34:00.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JJlmy/MNd8U1gqQXYwSJ89lI9Ek=', 'Content-Length': '0'} 2024-04-16T19:34:00.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000910a5494b646d0bd-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.842 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:34:00.842 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:34:00.843 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:00.843 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:00.843 DEBUG:boto:path=/yrjefh-16/?versioning 2024-04-16T19:34:00.843 DEBUG:boto:auth_path=/yrjefh-16/?versioning 2024-04-16T19:34:00.843 DEBUG:boto:Method: PUT 2024-04-16T19:34:00.843 DEBUG:boto:Path: /yrjefh-16/?versioning 2024-04-16T19:34:00.843 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:34:00.843 DEBUG:boto:Headers: {} 2024-04-16T19:34:00.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:00.843 DEBUG:boto:Port: 8000 2024-04-16T19:34:00.843 DEBUG:boto:Params: {} 2024-04-16T19:34:00.844 DEBUG:boto:Token: None 2024-04-16T19:34:00.844 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:34:00 GMT /yrjefh-16/?versioning 2024-04-16T19:34:00.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c5YME8lx4bo260I78Sb7r7nI++0= 2024-04-16T19:34:00.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c5YME8lx4bo260I78Sb7r7nI++0=', 'Content-Length': '228'} 2024-04-16T19:34:00.849 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a08d8540541af65f-00661ed2a8-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:00.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:00.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:00.850 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:34:00.898 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:34:00.898 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:34:00.948 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:34:00.970 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296040.848395_144.1","last_update":"2024-04-16T19:34:00.848395Z"}] 2024-04-16T19:34:00.970 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296040.848395_144.1'} 2024-04-16T19:34:00.970 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:34:00.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:00.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:00.970 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:34:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:34:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:34:01.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:01.077+0000 7ffb58933800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:34:01.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:01.079+0000 7ffb58933800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:34:01.103 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296040.845203_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:34:00.845203Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:34:01Z"} 2024-04-16T19:34:01.103 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296040.845203_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:34:00.845203Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:34:01Z"} 2024-04-16T19:34:01.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296040.845203_143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:34:00.845203Z', 'realm_epoch': 2}}] 2024-04-16T19:34:01.103 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296040.848395_144.1'} 2024-04-16T19:34:01.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296040.845203_143.1'} 2024-04-16T19:34:01.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296040.848395_144.1 target=1_1713296040.845203_143.1 2024-04-16T19:34:06.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:06.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:34:06.105 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:34:06.160 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:34:06.160 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:34:06.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:06.214+0000 7fb730b33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:34:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:06.216+0000 7fb730b33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:34:06.239 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296040.848395_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:34:00.848395Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:34:06Z"} 2024-04-16T19:34:06.239 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296040.848395_144.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:34:00.848395Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:34:06Z"} 2024-04-16T19:34:06.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296040.848395_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:34:00.848395Z', 'realm_epoch': 2}}] 2024-04-16T19:34:06.239 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296040.848395_144.1'} 2024-04-16T19:34:06.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296040.848395_144.1'} 2024-04-16T19:34:06.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:34:06.239 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:34:06.240 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:34:06.240 DEBUG:boto:Method: HEAD 2024-04-16T19:34:06.240 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:34:06.240 DEBUG:boto:Data: 2024-04-16T19:34:06.240 DEBUG:boto:Headers: {} 2024-04-16T19:34:06.240 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:06.240 DEBUG:boto:Port: 8000 2024-04-16T19:34:06.240 DEBUG:boto:Params: {} 2024-04-16T19:34:06.240 DEBUG:boto:Token: None 2024-04-16T19:34:06.241 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:34:06 GMT /yrjefh-16/ 2024-04-16T19:34:06.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+X0QJKXB7J/uLW7YKEL5nK8/ryQ= 2024-04-16T19:34:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:34:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+X0QJKXB7J/uLW7YKEL5nK8/ryQ=', 'Content-Length': '0'} 2024-04-16T19:34:06.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034f5fdc02ff594c4-00661ed2ae-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:06.244 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:34:06.244 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:34:06.244 DEBUG:boto:Method: PUT 2024-04-16T19:34:06.244 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:34:06.244 DEBUG:boto:Data: 2024-04-16T19:34:06.244 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:34:06.244 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:34:06.244 DEBUG:boto:Port: 8000 2024-04-16T19:34:06.244 DEBUG:boto:Params: {} 2024-04-16T19:34:06.244 DEBUG:boto:Token: None 2024-04-16T19:34:06.245 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:34:06 GMT /yrjefh-16/obj 2024-04-16T19:34:06.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mWbmRHZmZ9uCdpmy8T3PbskyITY= 2024-04-16T19:34:06.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:34:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mWbmRHZmZ9uCdpmy8T3PbskyITY='} 2024-04-16T19:34:06.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-amz-request-id', 'tx00000b146b6bdedf6cff3-00661ed2ae-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:34:06.287 DEBUG:rgw_multi.tests:created new version id=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:34:06.287 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-16 2024-04-16T19:34:06.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:34:06.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:34:06.287 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:34:06.343 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:34:06.343 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:34:06.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:06.398+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:06.399 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:06.399 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:11.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:11.399+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:11.401 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:11.401 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:16.401+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:16.402 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:16.402 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:21.402+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:21.404 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.45.6, , ] 2024-04-16T19:34:21.404 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:26.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:26.403+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:26.405 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:26.405 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:31.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:31.404+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:31.406 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:31.406 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:36.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:36.405+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:36.407 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:36.407 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:41.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:41.407+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:41.410 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:41.410 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:46.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:46.408+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:46.411 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:46.411 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:51.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:51.410+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:51.412 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:51.412 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:34:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:34:56.411+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:34:56.413 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:34:56.413 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:01.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:01.412+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:01.414 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:01.414 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:06.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:06.413+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:06.415 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:06.415 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:11.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:11.414+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:11.417 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:11.417 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:16.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:16.416+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:16.418 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:16.418 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:21.417+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:21.419 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:21.420 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:26.418+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:26.420 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:26.420 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:31.420+0000 7fc4e3b22800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:31.421 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.47.6, , ] 2024-04-16T19:35:31.422 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:36.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:36.421+0000 7fc4e3b22800 1 bucket sync caught up with source: 2024-04-16T19:35:36.423 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:36.423 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:36.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:36.421+0000 7fc4e3b22800 0 bucket checkpoint complete 2024-04-16T19:35:36.447 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-16 2024-04-16T19:35:36.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:36.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:36.448 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:35:36.511 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:36.511 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:36.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:36.572+0000 7fc15e638800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:36.574 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-16T19:35:36.574 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.36.13, , ] 2024-04-16T19:35:41.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:41.573+0000 7fc15e638800 1 bucket sync caught up with source: 2024-04-16T19:35:41.576 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.36.13, , ] 2024-04-16T19:35:41.576 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.36.13, , ] 2024-04-16T19:35:41.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:41.573+0000 7fc15e638800 0 bucket checkpoint complete 2024-04-16T19:35:41.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:35:41.604 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.604 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.604 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.604 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:35:41.604 DEBUG:boto:Data: 2024-04-16T19:35:41.604 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.604 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.605 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.605 DEBUG:boto:Params: {} 2024-04-16T19:35:41.605 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:35:41.606 DEBUG:boto:Token: None 2024-04-16T19:35:41.606 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.606 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bcEDBUebX31zb9mFnKXQAjrh/NU= 2024-04-16T19:35:41.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bcEDBUebX31zb9mFnKXQAjrh/NU=', 'Content-Length': '0'} 2024-04-16T19:35:41.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a8d00ced42e92e8-00661ed30d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.613 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.613 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.614 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.614 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:35:41.614 DEBUG:boto:Data: 2024-04-16T19:35:41.614 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.614 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.614 DEBUG:boto:Params: {} 2024-04-16T19:35:41.615 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:35:41.615 DEBUG:boto:Token: None 2024-04-16T19:35:41.615 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bcEDBUebX31zb9mFnKXQAjrh/NU= 2024-04-16T19:35:41.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bcEDBUebX31zb9mFnKXQAjrh/NU=', 'Content-Length': '0'} 2024-04-16T19:35:41.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e3f13b7c5539e55-00661ed30d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:35:41.620 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.620 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.620 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.621 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.621 DEBUG:boto:Method: GET 2024-04-16T19:35:41.621 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.621 DEBUG:boto:Data: 2024-04-16T19:35:41.621 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.621 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.621 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.621 DEBUG:boto:Params: {} 2024-04-16T19:35:41.621 DEBUG:boto:Token: None 2024-04-16T19:35:41.622 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/?versions 2024-04-16T19:35:41.622 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE= 2024-04-16T19:35:41.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE=', 'Content-Length': '0'} 2024-04-16T19:35:41.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3bbc6066aa00b76-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.625 DEBUG:boto:b'yrjefh-16999falseobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZtrue2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjnullfalse2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:41.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:41.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:35:41.627 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.627 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.627 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.627 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.628 DEBUG:boto:Method: GET 2024-04-16T19:35:41.628 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.628 DEBUG:boto:Data: 2024-04-16T19:35:41.628 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.628 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.628 DEBUG:boto:Params: {} 2024-04-16T19:35:41.628 DEBUG:boto:Token: None 2024-04-16T19:35:41.629 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/?versions 2024-04-16T19:35:41.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE= 2024-04-16T19:35:41.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE=', 'Content-Length': '0'} 2024-04-16T19:35:41.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b2eb7da03b9d68-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.632 DEBUG:boto:b'yrjefh-16999falseobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZtrue2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjnullfalse2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:41.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:41.633 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.633 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.633 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.633 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.633 DEBUG:boto:Method: GET 2024-04-16T19:35:41.633 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.633 DEBUG:boto:Data: 2024-04-16T19:35:41.633 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.633 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.634 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.634 DEBUG:boto:Params: {} 2024-04-16T19:35:41.634 DEBUG:boto:Token: None 2024-04-16T19:35:41.634 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/?versions 2024-04-16T19:35:41.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE= 2024-04-16T19:35:41.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE=', 'Content-Length': '0'} 2024-04-16T19:35:41.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008395602e7967eb74-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.637 DEBUG:boto:b'yrjefh-16999falseobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZtrue2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjnullfalse2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.638 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.638 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.638 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.638 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.638 DEBUG:boto:Method: GET 2024-04-16T19:35:41.638 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:41.639 DEBUG:boto:Data: 2024-04-16T19:35:41.639 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.639 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.639 DEBUG:boto:Params: {} 2024-04-16T19:35:41.639 DEBUG:boto:Token: None 2024-04-16T19:35:41.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/?versions 2024-04-16T19:35:41.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE= 2024-04-16T19:35:41.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4CUC+cue3K/27YW/j4uQXRVhaSE=', 'Content-Length': '0'} 2024-04-16T19:35:41.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb2065b04b71872-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.643 DEBUG:boto:b'yrjefh-16999falseobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZtrue2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjnullfalse2024-04-16T19:33:55.442Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.643 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.643 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.644 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.644 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.644 DEBUG:boto:Method: GET 2024-04-16T19:35:41.644 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.644 DEBUG:boto:Data: 2024-04-16T19:35:41.644 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.644 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.644 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.644 DEBUG:boto:Params: {} 2024-04-16T19:35:41.644 DEBUG:boto:Token: None 2024-04-16T19:35:41.645 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.645 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bf4999fff52090b-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.647 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.648 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.648 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.648 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.648 DEBUG:boto:Method: GET 2024-04-16T19:35:41.648 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.648 DEBUG:boto:Data: 2024-04-16T19:35:41.648 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.648 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.648 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.648 DEBUG:boto:Params: {} 2024-04-16T19:35:41.649 DEBUG:boto:Token: None 2024-04-16T19:35:41.649 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065d75879a2877053-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.651 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.652 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.652 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.652 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.652 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.652 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.652 DEBUG:boto:Data: 2024-04-16T19:35:41.652 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.653 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.653 DEBUG:boto:Params: {} 2024-04-16T19:35:41.653 DEBUG:boto:Token: None 2024-04-16T19:35:41.653 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.653 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+RPjdyb7T8RgmohBwaAgAX5BfyE= 2024-04-16T19:35:41.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+RPjdyb7T8RgmohBwaAgAX5BfyE=', 'Content-Length': '0'} 2024-04-16T19:35:41.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006be65b749164c418-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.656 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.656 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.656 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.656 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.656 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.656 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.656 DEBUG:boto:Data: 2024-04-16T19:35:41.656 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.656 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.657 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.657 DEBUG:boto:Params: {} 2024-04-16T19:35:41.657 DEBUG:boto:Token: None 2024-04-16T19:35:41.657 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.657 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+RPjdyb7T8RgmohBwaAgAX5BfyE= 2024-04-16T19:35:41.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+RPjdyb7T8RgmohBwaAgAX5BfyE=', 'Content-Length': '0'} 2024-04-16T19:35:41.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbb92bd8a4f9d789-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.659 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.660 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.660 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.660 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.660 DEBUG:boto:Method: GET 2024-04-16T19:35:41.660 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.660 DEBUG:boto:Data: 2024-04-16T19:35:41.660 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.660 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.661 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.661 DEBUG:boto:Params: {} 2024-04-16T19:35:41.661 DEBUG:boto:Token: None 2024-04-16T19:35:41.661 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.661 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eca758458a8ed978-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.664 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.664 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.664 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.664 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.664 DEBUG:boto:Method: GET 2024-04-16T19:35:41.664 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.664 DEBUG:boto:Data: 2024-04-16T19:35:41.665 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.665 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.665 DEBUG:boto:Params: {} 2024-04-16T19:35:41.665 DEBUG:boto:Token: None 2024-04-16T19:35:41.665 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7e2087551ad5c45-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.668 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.668 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.668 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.668 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.668 DEBUG:boto:Method: GET 2024-04-16T19:35:41.668 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.668 DEBUG:boto:Data: 2024-04-16T19:35:41.669 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.669 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.669 DEBUG:boto:Params: {} 2024-04-16T19:35:41.669 DEBUG:boto:Token: None 2024-04-16T19:35:41.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002916701cdbdbecd3-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.714 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.715 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.715 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.715 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.715 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.715 DEBUG:boto:Method: GET 2024-04-16T19:35:41.715 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.716 DEBUG:boto:Data: 2024-04-16T19:35:41.716 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.716 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.716 DEBUG:boto:Params: {} 2024-04-16T19:35:41.716 DEBUG:boto:Token: None 2024-04-16T19:35:41.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b698ceaefc8a41-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.719 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.720 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.720 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.720 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.720 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:41.720 DEBUG:boto:Data: 2024-04-16T19:35:41.720 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.721 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.721 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.721 DEBUG:boto:Params: {} 2024-04-16T19:35:41.721 DEBUG:boto:Token: None 2024-04-16T19:35:41.721 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj 2024-04-16T19:35:41.721 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE= 2024-04-16T19:35:41.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE=', 'Content-Length': '0'} 2024-04-16T19:35:41.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcfa5a8bbcb7a618-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.724 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.724 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.724 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.724 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:41.725 DEBUG:boto:Data: 2024-04-16T19:35:41.725 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.725 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.725 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.725 DEBUG:boto:Params: {} 2024-04-16T19:35:41.725 DEBUG:boto:Token: None 2024-04-16T19:35:41.725 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj 2024-04-16T19:35:41.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE= 2024-04-16T19:35:41.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE=', 'Content-Length': '0'} 2024-04-16T19:35:41.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e754bac93a330f7-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.729 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.729 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.729 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.729 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.729 DEBUG:boto:Method: GET 2024-04-16T19:35:41.729 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.729 DEBUG:boto:Data: 2024-04-16T19:35:41.730 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.730 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.730 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.730 DEBUG:boto:Params: {} 2024-04-16T19:35:41.730 DEBUG:boto:Token: None 2024-04-16T19:35:41.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a64e751c38b3705-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.733 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.733 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.733 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.733 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.733 DEBUG:boto:Method: GET 2024-04-16T19:35:41.734 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.734 DEBUG:boto:Data: 2024-04-16T19:35:41.734 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.734 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.734 DEBUG:boto:Params: {} 2024-04-16T19:35:41.734 DEBUG:boto:Token: None 2024-04-16T19:35:41.734 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fdf2ccdca70af7a-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.737 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.737 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.737 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.737 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.737 DEBUG:boto:Method: GET 2024-04-16T19:35:41.737 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.738 DEBUG:boto:Data: 2024-04-16T19:35:41.738 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.738 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.738 DEBUG:boto:Params: {} 2024-04-16T19:35:41.738 DEBUG:boto:Token: None 2024-04-16T19:35:41.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db72c0a5a7c41392-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.742 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.742 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.742 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.742 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.743 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.743 DEBUG:boto:Method: GET 2024-04-16T19:35:41.743 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.743 DEBUG:boto:Data: 2024-04-16T19:35:41.743 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.743 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.743 DEBUG:boto:Params: {} 2024-04-16T19:35:41.743 DEBUG:boto:Token: None 2024-04-16T19:35:41.744 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002091d8950bfd7ec6-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.747 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.747 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.748 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.748 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.748 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.748 DEBUG:boto:Method: GET 2024-04-16T19:35:41.748 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.748 DEBUG:boto:Data: 2024-04-16T19:35:41.748 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.748 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.749 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.749 DEBUG:boto:Params: {} 2024-04-16T19:35:41.749 DEBUG:boto:Token: None 2024-04-16T19:35:41.749 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.749 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI= 2024-04-16T19:35:41.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI=', 'Content-Length': '0'} 2024-04-16T19:35:41.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000416c1bd0bb7d0578-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.751 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.751 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.752 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.752 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.752 DEBUG:boto:Method: GET 2024-04-16T19:35:41.752 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.752 DEBUG:boto:Data: 2024-04-16T19:35:41.752 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.752 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.752 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.752 DEBUG:boto:Params: {} 2024-04-16T19:35:41.753 DEBUG:boto:Token: None 2024-04-16T19:35:41.753 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.753 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI= 2024-04-16T19:35:41.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI=', 'Content-Length': '0'} 2024-04-16T19:35:41.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a39f4ae7cf6c4f6-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.755 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.755 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.756 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.756 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.756 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.756 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.756 DEBUG:boto:Data: 2024-04-16T19:35:41.756 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.756 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.756 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.756 DEBUG:boto:Params: {} 2024-04-16T19:35:41.757 DEBUG:boto:Token: None 2024-04-16T19:35:41.757 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.757 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EQNMLtEhK8gim+GmQruJ0vwAXAQ= 2024-04-16T19:35:41.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EQNMLtEhK8gim+GmQruJ0vwAXAQ=', 'Content-Length': '0'} 2024-04-16T19:35:41.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf1aed61054453e6-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.759 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.759 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.759 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.759 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.759 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.760 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.760 DEBUG:boto:Data: 2024-04-16T19:35:41.760 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.760 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.760 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.760 DEBUG:boto:Params: {} 2024-04-16T19:35:41.760 DEBUG:boto:Token: None 2024-04-16T19:35:41.760 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EQNMLtEhK8gim+GmQruJ0vwAXAQ= 2024-04-16T19:35:41.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EQNMLtEhK8gim+GmQruJ0vwAXAQ=', 'Content-Length': '0'} 2024-04-16T19:35:41.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094be6fcaf24395cb-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.763 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.763 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.763 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.763 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.763 DEBUG:boto:Method: GET 2024-04-16T19:35:41.763 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.763 DEBUG:boto:Data: 2024-04-16T19:35:41.764 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.764 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.764 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.764 DEBUG:boto:Params: {} 2024-04-16T19:35:41.764 DEBUG:boto:Token: None 2024-04-16T19:35:41.764 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI= 2024-04-16T19:35:41.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI=', 'Content-Length': '0'} 2024-04-16T19:35:41.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046154e4929394a95-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.766 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.767 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.767 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.767 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.767 DEBUG:boto:Method: GET 2024-04-16T19:35:41.767 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.767 DEBUG:boto:Data: 2024-04-16T19:35:41.767 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.767 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.767 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.768 DEBUG:boto:Params: {} 2024-04-16T19:35:41.768 DEBUG:boto:Token: None 2024-04-16T19:35:41.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:41.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI= 2024-04-16T19:35:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pVySI4D5WBi2+GtIwdvdg2ff/DI=', 'Content-Length': '0'} 2024-04-16T19:35:41.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:33:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000321d67d008a028d3-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.770 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.771 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.771 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.771 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.771 DEBUG:boto:Method: GET 2024-04-16T19:35:41.771 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.771 DEBUG:boto:Data: 2024-04-16T19:35:41.771 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.771 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.771 DEBUG:boto:Params: {} 2024-04-16T19:35:41.772 DEBUG:boto:Token: None 2024-04-16T19:35:41.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e542319924c08e-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.818 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.819 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.819 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.819 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.819 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.819 DEBUG:boto:Method: GET 2024-04-16T19:35:41.819 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.820 DEBUG:boto:Data: 2024-04-16T19:35:41.820 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.820 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.820 DEBUG:boto:Params: {} 2024-04-16T19:35:41.820 DEBUG:boto:Token: None 2024-04-16T19:35:41.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2e0f4d2d27e217-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.823 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.824 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.824 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.824 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.824 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:41.824 DEBUG:boto:Data: 2024-04-16T19:35:41.825 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.825 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.825 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.825 DEBUG:boto:Params: {} 2024-04-16T19:35:41.825 DEBUG:boto:Token: None 2024-04-16T19:35:41.825 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj 2024-04-16T19:35:41.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE= 2024-04-16T19:35:41.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE=', 'Content-Length': '0'} 2024-04-16T19:35:41.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ddbc3b9c6aedc4ab-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.828 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.828 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.828 DEBUG:boto:Method: HEAD 2024-04-16T19:35:41.829 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:41.829 DEBUG:boto:Data: 2024-04-16T19:35:41.829 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.829 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.829 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.829 DEBUG:boto:Params: {} 2024-04-16T19:35:41.829 DEBUG:boto:Token: None 2024-04-16T19:35:41.829 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj 2024-04-16T19:35:41.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE= 2024-04-16T19:35:41.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LZLy4gI5WbJEpOSTARZVT77V9iE=', 'Content-Length': '0'} 2024-04-16T19:35:41.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005566c028c409e50c-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:41.833 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.833 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.833 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.833 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.833 DEBUG:boto:Method: GET 2024-04-16T19:35:41.833 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.833 DEBUG:boto:Data: 2024-04-16T19:35:41.833 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.834 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.834 DEBUG:boto:Params: {} 2024-04-16T19:35:41.834 DEBUG:boto:Token: None 2024-04-16T19:35:41.834 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010e8af73f50b6686-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.837 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:41.837 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:41.837 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.837 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.837 DEBUG:boto:Method: GET 2024-04-16T19:35:41.837 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.837 DEBUG:boto:Data: 2024-04-16T19:35:41.837 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.838 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.838 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.838 DEBUG:boto:Params: {} 2024-04-16T19:35:41.838 DEBUG:boto:Token: None 2024-04-16T19:35:41.838 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:41.838 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg= 2024-04-16T19:35:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p/RjmEO4nyppjHqcEaUoO0C9tUg=', 'Content-Length': '0'} 2024-04-16T19:35:41.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d99d4fde675b6185-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.841 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.841 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.842 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.842 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.842 DEBUG:boto:Method: GET 2024-04-16T19:35:41.842 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.842 DEBUG:boto:Data: 2024-04-16T19:35:41.842 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.842 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:41.842 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.843 DEBUG:boto:Params: {} 2024-04-16T19:35:41.843 DEBUG:boto:Token: None 2024-04-16T19:35:41.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2bcdf6130e9e927-00661ed30d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.847 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.847 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.847 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.848 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.848 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:41.848 DEBUG:boto:Method: GET 2024-04-16T19:35:41.848 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:41.848 DEBUG:boto:Data: 2024-04-16T19:35:41.848 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.848 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.848 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.848 DEBUG:boto:Params: {} 2024-04-16T19:35:41.849 DEBUG:boto:Token: None 2024-04-16T19:35:41.849 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/ 2024-04-16T19:35:41.849 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk= 2024-04-16T19:35:41.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7HlP2/XCLliOeOvUayGr9ukycgk=', 'Content-Length': '0'} 2024-04-16T19:35:41.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a0166893be05c92-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.852 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:41.852 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:35:41.853 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:41.853 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:41.853 DEBUG:boto:path=/yrjefh-16/?versioning 2024-04-16T19:35:41.853 DEBUG:boto:auth_path=/yrjefh-16/?versioning 2024-04-16T19:35:41.853 DEBUG:boto:Method: PUT 2024-04-16T19:35:41.853 DEBUG:boto:Path: /yrjefh-16/?versioning 2024-04-16T19:35:41.853 DEBUG:boto:Data: Suspended Disabled 2024-04-16T19:35:41.853 DEBUG:boto:Headers: {} 2024-04-16T19:35:41.853 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:41.854 DEBUG:boto:Port: 8000 2024-04-16T19:35:41.854 DEBUG:boto:Params: {} 2024-04-16T19:35:41.854 DEBUG:boto:Token: None 2024-04-16T19:35:41.854 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:35:41 GMT /yrjefh-16/?versioning 2024-04-16T19:35:41.854 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dj5CxETMXCzHbA9/NhDkRjYcXV0= 2024-04-16T19:35:41.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dj5CxETMXCzHbA9/NhDkRjYcXV0=', 'Content-Length': '230'} 2024-04-16T19:35:41.862 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000444d950b71b3447b-00661ed30d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:41.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:41.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:41.863 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:35:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:41.972 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:35:41.994 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296141.860216_162.1","last_update":"2024-04-16T19:35:41.860216Z"}] 2024-04-16T19:35:41.995 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296141.860216_162.1'} 2024-04-16T19:35:41.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:35:41.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:41.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:41.995 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:35:42.050 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:42.050 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:42.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:42.100+0000 7fef04e39800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:35:42.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:42.101+0000 7fef04e39800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:35:42.123 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296141.860216_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:41.860216Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:35:42Z"} 2024-04-16T19:35:42.124 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296141.860216_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:41.860216Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:35:42Z"} 2024-04-16T19:35:42.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296141.860216_162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:35:41.860216Z', 'realm_epoch': 2}}] 2024-04-16T19:35:42.124 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296141.860216_162.1'} 2024-04-16T19:35:42.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296141.860216_162.1'} 2024-04-16T19:35:42.124 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:35:42.125 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:42.125 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:42.125 DEBUG:boto:Method: HEAD 2024-04-16T19:35:42.125 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:35:42.125 DEBUG:boto:Data: 2024-04-16T19:35:42.125 DEBUG:boto:Headers: {} 2024-04-16T19:35:42.125 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:42.126 DEBUG:boto:Port: 8000 2024-04-16T19:35:42.126 DEBUG:boto:Params: {} 2024-04-16T19:35:42.126 DEBUG:boto:Token: None 2024-04-16T19:35:42.126 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:42 GMT /yrjefh-16/ 2024-04-16T19:35:42.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T1EEy6lRe0Ov7dj8LRHVncBYFKk= 2024-04-16T19:35:42.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T1EEy6lRe0Ov7dj8LRHVncBYFKk=', 'Content-Length': '0'} 2024-04-16T19:35:42.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f9fcf71cb5aa539-00661ed30e-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:42.130 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:42.130 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:42.130 DEBUG:boto:Method: PUT 2024-04-16T19:35:42.130 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:42.130 DEBUG:boto:Data: 2024-04-16T19:35:42.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:35:42.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:42.131 DEBUG:boto:Port: 8000 2024-04-16T19:35:42.131 DEBUG:boto:Params: {} 2024-04-16T19:35:42.131 DEBUG:boto:Token: None 2024-04-16T19:35:42.131 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:35:42 GMT /yrjefh-16/obj 2024-04-16T19:35:42.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBBR1gVz+dbX1PpuZPiRFHtXO1E= 2024-04-16T19:35:42.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:35:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBBR1gVz+dbX1PpuZPiRFHtXO1E='} 2024-04-16T19:35:42.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f99e6eb9ba7901f2-00661ed30e-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:42.146 DEBUG:rgw_multi.tests:created null version id=None 2024-04-16T19:35:42.146 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-16 2024-04-16T19:35:42.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:42.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:42.147 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:35:42.202 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:42.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:42.255+0000 7f82e0a16800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:42.256 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000003.48.13, , ] 2024-04-16T19:35:42.256 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-16T19:35:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:47.256+0000 7f82e0a16800 1 bucket sync caught up with source: 2024-04-16T19:35:47.258 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000005.52.18, , ] 2024-04-16T19:35:47.258 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000005.52.18, , ] 2024-04-16T19:35:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:47.256+0000 7f82e0a16800 0 bucket checkpoint complete 2024-04-16T19:35:47.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-16 2024-04-16T19:35:47.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:47.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:35:47.284 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:35:47.347 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:47.347 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:47.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:47.405+0000 7f16bb83d800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:47.407 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.36.13, , ] 2024-04-16T19:35:47.407 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-04-16T19:35:52.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:52.407+0000 7f16bb83d800 1 waiting for incremental sync to catch up: 2024-04-16T19:35:52.408 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.36.13, , ] 2024-04-16T19:35:52.408 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-04-16T19:35:57.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:57.408+0000 7f16bb83d800 1 bucket sync caught up with source: 2024-04-16T19:35:57.410 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000005.43.18, , ] 2024-04-16T19:35:57.410 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000005.43.18, , ] 2024-04-16T19:35:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:35:57.408+0000 7f16bb83d800 0 bucket checkpoint complete 2024-04-16T19:35:57.436 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:35:57.436 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.436 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.437 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.437 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:35:57.437 DEBUG:boto:Data: 2024-04-16T19:35:57.437 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.437 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.437 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.437 DEBUG:boto:Params: {} 2024-04-16T19:35:57.437 DEBUG:boto:Token: None 2024-04-16T19:35:57.437 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.438 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bn+afxmvjhVrY4oIPnhLkLLKosA= 2024-04-16T19:35:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bn+afxmvjhVrY4oIPnhLkLLKosA=', 'Content-Length': '0'} 2024-04-16T19:35:57.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b58825072459cf0-00661ed31d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.441 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.441 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.441 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.441 DEBUG:boto:Path: /yrjefh-16/ 2024-04-16T19:35:57.441 DEBUG:boto:Data: 2024-04-16T19:35:57.441 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.442 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.442 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.442 DEBUG:boto:Params: {} 2024-04-16T19:35:57.442 DEBUG:boto:Token: None 2024-04-16T19:35:57.442 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.442 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bn+afxmvjhVrY4oIPnhLkLLKosA= 2024-04-16T19:35:57.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bn+afxmvjhVrY4oIPnhLkLLKosA=', 'Content-Length': '0'} 2024-04-16T19:35:57.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9eb5f46afcdab6b-00661ed31d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.445 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:35:57.445 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.445 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.445 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.445 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.445 DEBUG:boto:Method: GET 2024-04-16T19:35:57.445 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.446 DEBUG:boto:Data: 2024-04-16T19:35:57.446 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.446 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.446 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.446 DEBUG:boto:Params: {} 2024-04-16T19:35:57.446 DEBUG:boto:Token: None 2024-04-16T19:35:57.446 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/?versions 2024-04-16T19:35:57.446 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o= 2024-04-16T19:35:57.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o=', 'Content-Length': '0'} 2024-04-16T19:35:57.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db60d4c5f6fb664-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.449 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZfalse2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:57.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:57.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:35:57.450 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.450 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.450 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.450 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.450 DEBUG:boto:Method: GET 2024-04-16T19:35:57.450 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.450 DEBUG:boto:Data: 2024-04-16T19:35:57.450 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.451 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.451 DEBUG:boto:Params: {} 2024-04-16T19:35:57.451 DEBUG:boto:Token: None 2024-04-16T19:35:57.451 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/?versions 2024-04-16T19:35:57.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o= 2024-04-16T19:35:57.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o=', 'Content-Length': '0'} 2024-04-16T19:35:57.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ef9c93bc636b72-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.454 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZfalse2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:57.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:35:57.454 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.454 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.454 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.454 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.454 DEBUG:boto:Method: GET 2024-04-16T19:35:57.455 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.455 DEBUG:boto:Data: 2024-04-16T19:35:57.455 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.455 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.455 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.455 DEBUG:boto:Params: {} 2024-04-16T19:35:57.455 DEBUG:boto:Token: None 2024-04-16T19:35:57.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/?versions 2024-04-16T19:35:57.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o= 2024-04-16T19:35:57.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o=', 'Content-Length': '0'} 2024-04-16T19:35:57.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c067ad222b93c7ac-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.458 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZfalse2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.458 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.459 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.459 DEBUG:boto:path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.459 DEBUG:boto:auth_path=/yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.459 DEBUG:boto:Method: GET 2024-04-16T19:35:57.459 DEBUG:boto:Path: /yrjefh-16/?versions&max-keys=999 2024-04-16T19:35:57.459 DEBUG:boto:Data: 2024-04-16T19:35:57.459 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.459 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.459 DEBUG:boto:Params: {} 2024-04-16T19:35:57.459 DEBUG:boto:Token: None 2024-04-16T19:35:57.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/?versions 2024-04-16T19:35:57.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o= 2024-04-16T19:35:57.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zE1e2LbVHrYOIp1T0yXIJw7jj+o=', 'Content-Length': '0'} 2024-04-16T19:35:57.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000793ae1a1ac7ce36c-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.462 DEBUG:boto:b'yrjefh-16999falseobjnulltrue2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normalobjy-3sMAnytzoii-9z2QwxUyMqZ3gOgkZfalse2024-04-16T19:34:06.246Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.462 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.462 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.463 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.463 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.463 DEBUG:boto:Method: GET 2024-04-16T19:35:57.463 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.463 DEBUG:boto:Data: 2024-04-16T19:35:57.463 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.463 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.463 DEBUG:boto:Params: {} 2024-04-16T19:35:57.463 DEBUG:boto:Token: None 2024-04-16T19:35:57.463 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ= 2024-04-16T19:35:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ=', 'Content-Length': '0'} 2024-04-16T19:35:57.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000070dac69892e1be1-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.466 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.466 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.466 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.466 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.466 DEBUG:boto:Method: GET 2024-04-16T19:35:57.466 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.466 DEBUG:boto:Data: 2024-04-16T19:35:57.466 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.467 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.467 DEBUG:boto:Params: {} 2024-04-16T19:35:57.467 DEBUG:boto:Token: None 2024-04-16T19:35:57.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ= 2024-04-16T19:35:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ=', 'Content-Length': '0'} 2024-04-16T19:35:57.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b458c4988e45a3c2-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.469 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.469 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.469 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.469 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.469 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.469 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.470 DEBUG:boto:Data: 2024-04-16T19:35:57.470 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.470 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.470 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.470 DEBUG:boto:Params: {} 2024-04-16T19:35:57.470 DEBUG:boto:Token: None 2024-04-16T19:35:57.470 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.470 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bwyfh4fHNXHatwckRwmVsUm2Co8= 2024-04-16T19:35:57.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bwyfh4fHNXHatwckRwmVsUm2Co8=', 'Content-Length': '0'} 2024-04-16T19:35:57.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085422e3c62cc6592-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.472 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.472 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.472 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.473 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.473 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.473 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.473 DEBUG:boto:Data: 2024-04-16T19:35:57.473 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.473 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.473 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.473 DEBUG:boto:Params: {} 2024-04-16T19:35:57.473 DEBUG:boto:Token: None 2024-04-16T19:35:57.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bwyfh4fHNXHatwckRwmVsUm2Co8= 2024-04-16T19:35:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bwyfh4fHNXHatwckRwmVsUm2Co8=', 'Content-Length': '0'} 2024-04-16T19:35:57.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ab27a492c2c69c9-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.475 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.476 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.476 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.476 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.476 DEBUG:boto:Method: GET 2024-04-16T19:35:57.476 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.476 DEBUG:boto:Data: 2024-04-16T19:35:57.476 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.476 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.476 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.476 DEBUG:boto:Params: {} 2024-04-16T19:35:57.476 DEBUG:boto:Token: None 2024-04-16T19:35:57.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ= 2024-04-16T19:35:57.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ=', 'Content-Length': '0'} 2024-04-16T19:35:57.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008fb6969af9db38c2-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.479 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.479 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.479 DEBUG:boto:path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.479 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.479 DEBUG:boto:Method: GET 2024-04-16T19:35:57.479 DEBUG:boto:Path: /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.479 DEBUG:boto:Data: 2024-04-16T19:35:57.479 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.479 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.479 DEBUG:boto:Params: {} 2024-04-16T19:35:57.480 DEBUG:boto:Token: None 2024-04-16T19:35:57.480 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=null 2024-04-16T19:35:57.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ= 2024-04-16T19:35:57.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ioDSszXqMEcB8qkdqXGMJCC3ieQ=', 'Content-Length': '0'} 2024-04-16T19:35:57.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009af58082ebbc995b-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.483 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.483 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.483 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.483 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.483 DEBUG:boto:Method: GET 2024-04-16T19:35:57.483 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.483 DEBUG:boto:Data: 2024-04-16T19:35:57.483 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.483 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.483 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.483 DEBUG:boto:Params: {} 2024-04-16T19:35:57.483 DEBUG:boto:Token: None 2024-04-16T19:35:57.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d31a4e597bd2525-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.530 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.531 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.531 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.531 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.531 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.531 DEBUG:boto:Method: GET 2024-04-16T19:35:57.531 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.531 DEBUG:boto:Data: 2024-04-16T19:35:57.531 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.531 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.531 DEBUG:boto:Params: {} 2024-04-16T19:35:57.531 DEBUG:boto:Token: None 2024-04-16T19:35:57.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000573328e962bab009-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.534 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.534 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.535 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.535 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.535 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.535 DEBUG:boto:Data: 2024-04-16T19:35:57.535 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.535 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.535 DEBUG:boto:Params: {} 2024-04-16T19:35:57.535 DEBUG:boto:Token: None 2024-04-16T19:35:57.535 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI= 2024-04-16T19:35:57.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI=', 'Content-Length': '0'} 2024-04-16T19:35:57.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050e384bc4eb22d8e-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.537 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.538 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.538 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.538 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.538 DEBUG:boto:Data: 2024-04-16T19:35:57.538 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.538 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.538 DEBUG:boto:Params: {} 2024-04-16T19:35:57.538 DEBUG:boto:Token: None 2024-04-16T19:35:57.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI= 2024-04-16T19:35:57.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI=', 'Content-Length': '0'} 2024-04-16T19:35:57.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056339c5e53f0edd5-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.540 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.540 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.540 DEBUG:boto:Method: GET 2024-04-16T19:35:57.541 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.541 DEBUG:boto:Data: 2024-04-16T19:35:57.541 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.541 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.541 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.541 DEBUG:boto:Params: {} 2024-04-16T19:35:57.541 DEBUG:boto:Token: None 2024-04-16T19:35:57.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0= 2024-04-16T19:35:57.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0=', 'Content-Length': '0'} 2024-04-16T19:35:57.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006aac9a7c28f3d181-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.543 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.544 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.544 DEBUG:boto:Method: GET 2024-04-16T19:35:57.544 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.544 DEBUG:boto:Data: 2024-04-16T19:35:57.544 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.544 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.544 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.544 DEBUG:boto:Params: {} 2024-04-16T19:35:57.544 DEBUG:boto:Token: None 2024-04-16T19:35:57.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0= 2024-04-16T19:35:57.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0=', 'Content-Length': '0'} 2024-04-16T19:35:57.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b37fdce851066c2-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.546 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.546 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.547 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.547 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.547 DEBUG:boto:Method: GET 2024-04-16T19:35:57.547 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.547 DEBUG:boto:Data: 2024-04-16T19:35:57.547 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.547 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.547 DEBUG:boto:Params: {} 2024-04-16T19:35:57.547 DEBUG:boto:Token: None 2024-04-16T19:35:57.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fbe581634196ca0-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.550 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.550 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.550 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.551 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.551 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.551 DEBUG:boto:Method: GET 2024-04-16T19:35:57.551 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.551 DEBUG:boto:Data: 2024-04-16T19:35:57.551 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.551 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.551 DEBUG:boto:Params: {} 2024-04-16T19:35:57.551 DEBUG:boto:Token: None 2024-04-16T19:35:57.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7bbacb545a3e8ae-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.554 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.554 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.554 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.554 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.555 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.555 DEBUG:boto:Method: GET 2024-04-16T19:35:57.555 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.555 DEBUG:boto:Data: 2024-04-16T19:35:57.555 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.555 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.555 DEBUG:boto:Params: {} 2024-04-16T19:35:57.555 DEBUG:boto:Token: None 2024-04-16T19:35:57.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw= 2024-04-16T19:35:57.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw=', 'Content-Length': '0'} 2024-04-16T19:35:57.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adea57cab0acbf9d-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.558 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.558 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.558 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.558 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.558 DEBUG:boto:Method: GET 2024-04-16T19:35:57.558 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.558 DEBUG:boto:Data: 2024-04-16T19:35:57.558 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.558 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.558 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.558 DEBUG:boto:Params: {} 2024-04-16T19:35:57.559 DEBUG:boto:Token: None 2024-04-16T19:35:57.559 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw= 2024-04-16T19:35:57.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw=', 'Content-Length': '0'} 2024-04-16T19:35:57.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000313af6e5676384a2-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.561 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.561 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.561 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.561 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.561 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.561 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.561 DEBUG:boto:Data: 2024-04-16T19:35:57.561 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.561 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.561 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.562 DEBUG:boto:Params: {} 2024-04-16T19:35:57.562 DEBUG:boto:Token: None 2024-04-16T19:35:57.562 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.562 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ogui1K4uGFoJJoj0QFo8eIaBh9Y= 2024-04-16T19:35:57.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ogui1K4uGFoJJoj0QFo8eIaBh9Y=', 'Content-Length': '0'} 2024-04-16T19:35:57.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f55bdcf36ea246b2-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.564 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.564 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.564 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.564 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.564 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.565 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.565 DEBUG:boto:Data: 2024-04-16T19:35:57.565 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.565 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.565 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.565 DEBUG:boto:Params: {} 2024-04-16T19:35:57.565 DEBUG:boto:Token: None 2024-04-16T19:35:57.565 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ogui1K4uGFoJJoj0QFo8eIaBh9Y= 2024-04-16T19:35:57.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ogui1K4uGFoJJoj0QFo8eIaBh9Y=', 'Content-Length': '0'} 2024-04-16T19:35:57.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040c097bfc55424de-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.567 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.567 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.567 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.568 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.568 DEBUG:boto:Method: GET 2024-04-16T19:35:57.568 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.568 DEBUG:boto:Data: 2024-04-16T19:35:57.568 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.568 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.568 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.568 DEBUG:boto:Params: {} 2024-04-16T19:35:57.568 DEBUG:boto:Token: None 2024-04-16T19:35:57.568 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw= 2024-04-16T19:35:57.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw=', 'Content-Length': '0'} 2024-04-16T19:35:57.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:35 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9191c9dbae998aa-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.570 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.571 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.571 DEBUG:boto:path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.571 DEBUG:boto:auth_path=/yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.571 DEBUG:boto:Method: GET 2024-04-16T19:35:57.571 DEBUG:boto:Path: /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.571 DEBUG:boto:Data: 2024-04-16T19:35:57.571 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.571 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.571 DEBUG:boto:Params: {} 2024-04-16T19:35:57.571 DEBUG:boto:Token: None 2024-04-16T19:35:57.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj?versionId=y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ 2024-04-16T19:35:57.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw= 2024-04-16T19:35:57.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nuldI5mpDpNmmCLnQ0ukWtN4yTw=', 'Content-Length': '0'} 2024-04-16T19:35:57.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:34:06 GMT'), ('x-amz-version-id', 'y-3sMAnytzoii-9z2QwxUyMqZ3gOgkZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca15160ab4631e4e-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.573 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.573 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.574 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.574 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.574 DEBUG:boto:Method: GET 2024-04-16T19:35:57.574 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.574 DEBUG:boto:Data: 2024-04-16T19:35:57.574 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.574 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.574 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.574 DEBUG:boto:Params: {} 2024-04-16T19:35:57.574 DEBUG:boto:Token: None 2024-04-16T19:35:57.574 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f60e2fa0627dbb0-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.618 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.619 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.619 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.619 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.619 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.619 DEBUG:boto:Method: GET 2024-04-16T19:35:57.619 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.619 DEBUG:boto:Data: 2024-04-16T19:35:57.619 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.619 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.619 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.619 DEBUG:boto:Params: {} 2024-04-16T19:35:57.619 DEBUG:boto:Token: None 2024-04-16T19:35:57.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd6de2737ca865f9-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.622 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.622 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.622 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.623 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.623 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.623 DEBUG:boto:Data: 2024-04-16T19:35:57.623 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.623 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.623 DEBUG:boto:Params: {} 2024-04-16T19:35:57.623 DEBUG:boto:Token: None 2024-04-16T19:35:57.623 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI= 2024-04-16T19:35:57.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI=', 'Content-Length': '0'} 2024-04-16T19:35:57.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbe82861bdc8f253-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.625 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.625 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.625 DEBUG:boto:Method: HEAD 2024-04-16T19:35:57.626 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.626 DEBUG:boto:Data: 2024-04-16T19:35:57.626 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.626 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.626 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.626 DEBUG:boto:Params: {} 2024-04-16T19:35:57.626 DEBUG:boto:Token: None 2024-04-16T19:35:57.626 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.626 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI= 2024-04-16T19:35:57.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNqYAZbbjqjVEafoZoLAYPt0MdI=', 'Content-Length': '0'} 2024-04-16T19:35:57.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089ba3434188c0bd8-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:35:57.628 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.628 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.629 DEBUG:boto:Method: GET 2024-04-16T19:35:57.629 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.629 DEBUG:boto:Data: 2024-04-16T19:35:57.629 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.629 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.629 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.629 DEBUG:boto:Params: {} 2024-04-16T19:35:57.629 DEBUG:boto:Token: None 2024-04-16T19:35:57.629 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0= 2024-04-16T19:35:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0=', 'Content-Length': '0'} 2024-04-16T19:35:57.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:35:45 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077fffaebad670f8d-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.631 DEBUG:boto:path=/yrjefh-16/obj 2024-04-16T19:35:57.631 DEBUG:boto:auth_path=/yrjefh-16/obj 2024-04-16T19:35:57.631 DEBUG:boto:Method: GET 2024-04-16T19:35:57.632 DEBUG:boto:Path: /yrjefh-16/obj 2024-04-16T19:35:57.632 DEBUG:boto:Data: 2024-04-16T19:35:57.632 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.632 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.632 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.632 DEBUG:boto:Params: {} 2024-04-16T19:35:57.632 DEBUG:boto:Token: None 2024-04-16T19:35:57.632 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/obj 2024-04-16T19:35:57.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0= 2024-04-16T19:35:57.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NIGShMA3SvGFDfVABwZV39xuWi0=', 'Content-Length': '0'} 2024-04-16T19:35:57.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:35:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065f9923c2ffaaa2f-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.634 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.634 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.635 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.635 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.635 DEBUG:boto:Method: GET 2024-04-16T19:35:57.635 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.635 DEBUG:boto:Data: 2024-04-16T19:35:57.635 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.635 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.635 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.635 DEBUG:boto:Params: {} 2024-04-16T19:35:57.635 DEBUG:boto:Token: None 2024-04-16T19:35:57.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d6afbc51fc3ebf3-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.638 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.639 DEBUG:boto:path=/yrjefh-16/ 2024-04-16T19:35:57.639 DEBUG:boto:auth_path=/yrjefh-16/ 2024-04-16T19:35:57.639 DEBUG:boto:path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.639 DEBUG:boto:auth_path=/yrjefh-16/?prefix=obj 2024-04-16T19:35:57.639 DEBUG:boto:Method: GET 2024-04-16T19:35:57.639 DEBUG:boto:Path: /yrjefh-16/?prefix=obj 2024-04-16T19:35:57.639 DEBUG:boto:Data: 2024-04-16T19:35:57.639 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.639 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.639 DEBUG:boto:Params: {} 2024-04-16T19:35:57.639 DEBUG:boto:Token: None 2024-04-16T19:35:57.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-16/ 2024-04-16T19:35:57.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A= 2024-04-16T19:35:57.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:chvbqESGFJHKvbhj0Rz203zz98A=', 'Content-Length': '0'} 2024-04-16T19:35:57.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9512eaa1c6cb68-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.642 DEBUG:boto:b'yrjefh-16obj1000falseobj2024-04-16T19:35:42.132Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:35:57.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-16 zones={test-zone2, test-zone1} 2024-04-16T19:35:57.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-16T19:35:57.645 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-17 2024-04-16T19:35:57.645 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:35:57.645 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:35:57.645 DEBUG:boto:Method: PUT 2024-04-16T19:35:57.645 DEBUG:boto:Path: /yrjefh-17/ 2024-04-16T19:35:57.645 DEBUG:boto:Data: 2024-04-16T19:35:57.645 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.645 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.645 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.645 DEBUG:boto:Params: {} 2024-04-16T19:35:57.646 DEBUG:boto:Token: None 2024-04-16T19:35:57.646 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-17/ 2024-04-16T19:35:57.646 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yi4IG3ldEhlhRC3Z8PXkCfM7J4w= 2024-04-16T19:35:57.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yi4IG3ldEhlhRC3Z8PXkCfM7J4w=', 'Content-Length': '0'} 2024-04-16T19:35:57.661 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdacf85c571af694-00661ed31d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.661 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-18 2024-04-16T19:35:57.661 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:35:57.661 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:35:57.662 DEBUG:boto:Method: PUT 2024-04-16T19:35:57.662 DEBUG:boto:Path: /yrjefh-18/ 2024-04-16T19:35:57.662 DEBUG:boto:Data: 2024-04-16T19:35:57.662 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.662 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.662 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.662 DEBUG:boto:Params: {} 2024-04-16T19:35:57.662 DEBUG:boto:Token: None 2024-04-16T19:35:57.662 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-18/ 2024-04-16T19:35:57.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mTOrIhmJRvXswwjs7Px7zcrV6gc= 2024-04-16T19:35:57.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mTOrIhmJRvXswwjs7Px7zcrV6gc=', 'Content-Length': '0'} 2024-04-16T19:35:57.694 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007899a66c7c9564cf-00661ed31d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.695 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:35:57.695 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:35:57.695 DEBUG:boto:path=/yrjefh-17/?versioning 2024-04-16T19:35:57.695 DEBUG:boto:auth_path=/yrjefh-17/?versioning 2024-04-16T19:35:57.695 DEBUG:boto:Method: PUT 2024-04-16T19:35:57.695 DEBUG:boto:Path: /yrjefh-17/?versioning 2024-04-16T19:35:57.695 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:35:57.695 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:35:57.695 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.695 DEBUG:boto:Params: {} 2024-04-16T19:35:57.696 DEBUG:boto:Token: None 2024-04-16T19:35:57.696 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-17/?versioning 2024-04-16T19:35:57.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r54o4kcXj+ocQ52nYEd2R7F3ShY= 2024-04-16T19:35:57.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r54o4kcXj+ocQ52nYEd2R7F3ShY=', 'Content-Length': '228'} 2024-04-16T19:35:57.701 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a73ebc28726dd5f2-00661ed31d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.701 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:35:57.702 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:35:57.702 DEBUG:boto:path=/yrjefh-18/?versioning 2024-04-16T19:35:57.702 DEBUG:boto:auth_path=/yrjefh-18/?versioning 2024-04-16T19:35:57.702 DEBUG:boto:Method: PUT 2024-04-16T19:35:57.702 DEBUG:boto:Path: /yrjefh-18/?versioning 2024-04-16T19:35:57.702 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:35:57.702 DEBUG:boto:Headers: {} 2024-04-16T19:35:57.702 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:35:57.702 DEBUG:boto:Port: 8000 2024-04-16T19:35:57.702 DEBUG:boto:Params: {} 2024-04-16T19:35:57.702 DEBUG:boto:Token: None 2024-04-16T19:35:57.703 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:35:57 GMT /yrjefh-18/?versioning 2024-04-16T19:35:57.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ifocz0Uq3uhEw3pwIy59kt2YCDE= 2024-04-16T19:35:57.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:35:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ifocz0Uq3uhEw3pwIy59kt2YCDE=', 'Content-Length': '228'} 2024-04-16T19:35:57.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd9e639d5cc533b7-00661ed31d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:35:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:35:57.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:57.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:57.714 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:35:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:57.813 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:35:57.835 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296157.708325_177.1","last_update":"2024-04-16T19:35:57.708325Z"}] 2024-04-16T19:35:57.835 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296157.708325_177.1'} 2024-04-16T19:35:57.835 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:35:57.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:57.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:35:57.835 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:35:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:35:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:35:57.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:57.942+0000 7f0d0460f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:35:57.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:35:57.943+0000 7f0d0460f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:35:57.969 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296157.650848_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:57.650848Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:35:57Z"} 2024-04-16T19:35:57.969 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296157.650848_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:57.650848Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:35:57Z"} 2024-04-16T19:35:57.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296157.650848_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:35:57.650848Z', 'realm_epoch': 2}}] 2024-04-16T19:35:57.969 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296157.708325_177.1'} 2024-04-16T19:35:57.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296157.650848_166.1'} 2024-04-16T19:35:57.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296157.708325_177.1 target=1_1713296157.650848_166.1 2024-04-16T19:36:02.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:02.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:02.971 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:03.027 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:03.027 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:03.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:03.080+0000 7fa5f7d2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:03.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:03.082+0000 7fa5f7d2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:03.104 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296157.708325_177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:57.708325Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:03Z"} 2024-04-16T19:36:03.105 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296157.708325_177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:35:57.708325Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:03Z"} 2024-04-16T19:36:03.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296157.708325_177.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:35:57.708325Z', 'realm_epoch': 2}}] 2024-04-16T19:36:03.105 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296157.708325_177.1'} 2024-04-16T19:36:03.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296157.708325_177.1'} 2024-04-16T19:36:03.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:36:03.105 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:03.105 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:03.106 DEBUG:boto:Method: HEAD 2024-04-16T19:36:03.106 DEBUG:boto:Path: /yrjefh-17/ 2024-04-16T19:36:03.106 DEBUG:boto:Data: 2024-04-16T19:36:03.106 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.106 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:03.106 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.106 DEBUG:boto:Params: {} 2024-04-16T19:36:03.106 DEBUG:boto:Token: None 2024-04-16T19:36:03.107 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-17/ 2024-04-16T19:36:03.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ebdlji81hxk4fb2MSpMSGjlQzM= 2024-04-16T19:36:03.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ebdlji81hxk4fb2MSpMSGjlQzM=', 'Content-Length': '0'} 2024-04-16T19:36:03.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b1f787811bff0cf-00661ed323-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.110 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:03.110 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:03.110 DEBUG:boto:Method: PUT 2024-04-16T19:36:03.110 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:03.110 DEBUG:boto:Data: 2024-04-16T19:36:03.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:36:03.110 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:03.110 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.110 DEBUG:boto:Params: {} 2024-04-16T19:36:03.110 DEBUG:boto:Token: None 2024-04-16T19:36:03.111 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-17/obj 2024-04-16T19:36:03.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kks+EkiKaCazPP2ZgLywU5GGCkk= 2024-04-16T19:36:03.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kks+EkiKaCazPP2ZgLywU5GGCkk='} 2024-04-16T19:36:03.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-amz-request-id', 'tx0000034b4ec8fd15acafc-00661ed323-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.126 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:03.126 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:03.126 DEBUG:boto:Method: HEAD 2024-04-16T19:36:03.126 DEBUG:boto:Path: /yrjefh-17/ 2024-04-16T19:36:03.126 DEBUG:boto:Data: 2024-04-16T19:36:03.127 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:03.127 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.127 DEBUG:boto:Params: {} 2024-04-16T19:36:03.127 DEBUG:boto:Token: None 2024-04-16T19:36:03.127 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-17/ 2024-04-16T19:36:03.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ebdlji81hxk4fb2MSpMSGjlQzM= 2024-04-16T19:36:03.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ebdlji81hxk4fb2MSpMSGjlQzM=', 'Content-Length': '0'} 2024-04-16T19:36:03.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cce3b256be173327-00661ed323-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.129 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:03.129 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:03.130 DEBUG:boto:Method: DELETE 2024-04-16T19:36:03.130 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:03.130 DEBUG:boto:Data: 2024-04-16T19:36:03.130 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.130 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:03.130 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.130 DEBUG:boto:Params: {} 2024-04-16T19:36:03.130 DEBUG:boto:Token: None 2024-04-16T19:36:03.130 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-17/obj 2024-04-16T19:36:03.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6ywDLNAi7rI59LaGZ1jbK5GTJ3U= 2024-04-16T19:36:03.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6ywDLNAi7rI59LaGZ1jbK5GTJ3U=', 'Content-Length': '0'} 2024-04-16T19:36:03.138 DEBUG:boto:Response headers: [('x-amz-version-id', '6rd-Qw0l3074qMhqNL4JYtNr0TxMlwC'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000002bc5d88564fd9749-00661ed323-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.138 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:03.138 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:03.139 DEBUG:boto:Method: HEAD 2024-04-16T19:36:03.139 DEBUG:boto:Path: /yrjefh-18/ 2024-04-16T19:36:03.139 DEBUG:boto:Data: 2024-04-16T19:36:03.139 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:03.139 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.139 DEBUG:boto:Params: {} 2024-04-16T19:36:03.139 DEBUG:boto:Token: None 2024-04-16T19:36:03.139 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-18/ 2024-04-16T19:36:03.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3p9VYOvCwcXliNJHf/HUuzjOzLk= 2024-04-16T19:36:03.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3p9VYOvCwcXliNJHf/HUuzjOzLk=', 'Content-Length': '0'} 2024-04-16T19:36:03.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b1740b63ad675b1-00661ed323-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.142 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:03.142 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:03.142 DEBUG:boto:Method: PUT 2024-04-16T19:36:03.142 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:03.143 DEBUG:boto:Data: 2024-04-16T19:36:03.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:36:03.143 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:03.143 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.143 DEBUG:boto:Params: {} 2024-04-16T19:36:03.143 DEBUG:boto:Token: None 2024-04-16T19:36:03.143 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-18/obj 2024-04-16T19:36:03.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZZKjVOerCUu1+D6Zza070R5sa5c= 2024-04-16T19:36:03.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZZKjVOerCUu1+D6Zza070R5sa5c='} 2024-04-16T19:36:03.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-amz-request-id', 'tx0000053f81217b6f8dfaa-00661ed323-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.156 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:03.156 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:03.156 DEBUG:boto:Method: HEAD 2024-04-16T19:36:03.156 DEBUG:boto:Path: /yrjefh-18/ 2024-04-16T19:36:03.156 DEBUG:boto:Data: 2024-04-16T19:36:03.156 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:03.156 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.156 DEBUG:boto:Params: {} 2024-04-16T19:36:03.156 DEBUG:boto:Token: None 2024-04-16T19:36:03.157 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-18/ 2024-04-16T19:36:03.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3p9VYOvCwcXliNJHf/HUuzjOzLk= 2024-04-16T19:36:03.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3p9VYOvCwcXliNJHf/HUuzjOzLk=', 'Content-Length': '0'} 2024-04-16T19:36:03.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a4a0c1f1d9c0ca2-00661ed323-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.159 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:03.159 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:03.159 DEBUG:boto:Method: DELETE 2024-04-16T19:36:03.160 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:03.160 DEBUG:boto:Data: 2024-04-16T19:36:03.160 DEBUG:boto:Headers: {} 2024-04-16T19:36:03.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:03.160 DEBUG:boto:Port: 8000 2024-04-16T19:36:03.160 DEBUG:boto:Params: {} 2024-04-16T19:36:03.160 DEBUG:boto:Token: None 2024-04-16T19:36:03.160 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:03 GMT /yrjefh-18/obj 2024-04-16T19:36:03.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4NAM25bfkcDPKbms57SEtqzo5vs= 2024-04-16T19:36:03.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4NAM25bfkcDPKbms57SEtqzo5vs=', 'Content-Length': '0'} 2024-04-16T19:36:03.168 DEBUG:boto:Response headers: [('x-amz-version-id', '7wd2AlZMFnbMsWCsXf9evlG0GJ-Rftf'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000949288096dfa932f-00661ed323-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:03.168 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-17 2024-04-16T19:36:03.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:03.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:03.168 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:03.223 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:03.223 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:03.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:03.277+0000 7f1b6c143800 1 waiting to reach incremental sync.. 2024-04-16T19:36:08.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:08.279+0000 7f1b6c143800 1 bucket sync caught up with source: 2024-04-16T19:36:08.281 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000003.10.12, , ] 2024-04-16T19:36:08.281 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.10.12, , ] 2024-04-16T19:36:08.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:08.279+0000 7f1b6c143800 0 bucket checkpoint complete 2024-04-16T19:36:08.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-17 2024-04-16T19:36:08.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:08.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:08.307 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:08.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:08.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:08.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:08.429+0000 7f5d2292d800 1 waiting to reach incremental sync.. 2024-04-16T19:36:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:13.431+0000 7f5d2292d800 1 bucket sync caught up with source: 2024-04-16T19:36:13.433 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.21.6, , ] 2024-04-16T19:36:13.433 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.21.6, , ] 2024-04-16T19:36:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:13.431+0000 7f5d2292d800 0 bucket checkpoint complete 2024-04-16T19:36:13.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-17 zones={test-zone2, test-zone1} 2024-04-16T19:36:13.459 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.459 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.460 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.460 DEBUG:boto:Path: /yrjefh-17/ 2024-04-16T19:36:13.460 DEBUG:boto:Data: 2024-04-16T19:36:13.460 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.460 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.460 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.460 DEBUG:boto:Params: {} 2024-04-16T19:36:13.461 DEBUG:boto:Token: None 2024-04-16T19:36:13.461 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/ 2024-04-16T19:36:13.461 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NEM6Mm5RKXxiN3Bf63spGY5SKGI= 2024-04-16T19:36:13.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NEM6Mm5RKXxiN3Bf63spGY5SKGI=', 'Content-Length': '0'} 2024-04-16T19:36:13.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075956e20f48e7964-00661ed32d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.464 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.464 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.464 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.464 DEBUG:boto:Path: /yrjefh-17/ 2024-04-16T19:36:13.464 DEBUG:boto:Data: 2024-04-16T19:36:13.464 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.465 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.465 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.465 DEBUG:boto:Params: {} 2024-04-16T19:36:13.465 DEBUG:boto:Token: None 2024-04-16T19:36:13.465 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/ 2024-04-16T19:36:13.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NEM6Mm5RKXxiN3Bf63spGY5SKGI= 2024-04-16T19:36:13.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NEM6Mm5RKXxiN3Bf63spGY5SKGI=', 'Content-Length': '0'} 2024-04-16T19:36:13.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db4c24845ab5cd3b-00661ed32d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.468 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:36:13.468 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.469 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.469 DEBUG:boto:path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.469 DEBUG:boto:auth_path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.469 DEBUG:boto:Method: GET 2024-04-16T19:36:13.469 DEBUG:boto:Path: /yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.469 DEBUG:boto:Data: 2024-04-16T19:36:13.469 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.469 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.469 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.470 DEBUG:boto:Params: {} 2024-04-16T19:36:13.470 DEBUG:boto:Token: None 2024-04-16T19:36:13.470 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/?versions 2024-04-16T19:36:13.470 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg= 2024-04-16T19:36:13.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg=', 'Content-Length': '0'} 2024-04-16T19:36:13.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bcfff2579b25cb4-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.514 DEBUG:boto:b'yrjefh-17999falseobj6rd-Qw0l3074qMhqNL4JYtNr0TxMlwCtrue2024-04-16T19:36:03.132ZRGW11111111111111111NormalobjRawJNlJVVB1arOpDw6zUFO8KelA4nJ5false2024-04-16T19:36:03.112Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:36:13.516 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.516 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.516 DEBUG:boto:path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.516 DEBUG:boto:auth_path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.516 DEBUG:boto:Method: GET 2024-04-16T19:36:13.516 DEBUG:boto:Path: /yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.516 DEBUG:boto:Data: 2024-04-16T19:36:13.516 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.517 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.517 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.517 DEBUG:boto:Params: {} 2024-04-16T19:36:13.517 DEBUG:boto:Token: None 2024-04-16T19:36:13.517 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/?versions 2024-04-16T19:36:13.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg= 2024-04-16T19:36:13.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg=', 'Content-Length': '0'} 2024-04-16T19:36:13.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d63d231136b7d33-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.521 DEBUG:boto:b'yrjefh-17999falseobj6rd-Qw0l3074qMhqNL4JYtNr0TxMlwCtrue2024-04-16T19:36:03.132ZRGW11111111111111111NormalobjRawJNlJVVB1arOpDw6zUFO8KelA4nJ5false2024-04-16T19:36:03.112Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.521 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.522 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.522 DEBUG:boto:path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.522 DEBUG:boto:auth_path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.522 DEBUG:boto:Method: GET 2024-04-16T19:36:13.522 DEBUG:boto:Path: /yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.522 DEBUG:boto:Data: 2024-04-16T19:36:13.522 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.523 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.523 DEBUG:boto:Params: {} 2024-04-16T19:36:13.523 DEBUG:boto:Token: None 2024-04-16T19:36:13.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/?versions 2024-04-16T19:36:13.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg= 2024-04-16T19:36:13.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg=', 'Content-Length': '0'} 2024-04-16T19:36:13.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda960447f3bb108-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.526 DEBUG:boto:b'yrjefh-17999falseobj6rd-Qw0l3074qMhqNL4JYtNr0TxMlwCtrue2024-04-16T19:36:03.132ZRGW11111111111111111NormalobjRawJNlJVVB1arOpDw6zUFO8KelA4nJ5false2024-04-16T19:36:03.112Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.527 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.527 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.527 DEBUG:boto:path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.527 DEBUG:boto:auth_path=/yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.527 DEBUG:boto:Method: GET 2024-04-16T19:36:13.528 DEBUG:boto:Path: /yrjefh-17/?versions&max-keys=999 2024-04-16T19:36:13.528 DEBUG:boto:Data: 2024-04-16T19:36:13.528 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.528 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.528 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.528 DEBUG:boto:Params: {} 2024-04-16T19:36:13.528 DEBUG:boto:Token: None 2024-04-16T19:36:13.529 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/?versions 2024-04-16T19:36:13.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg= 2024-04-16T19:36:13.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0eXIzTG90PB4BkeKN9bjmsET+kg=', 'Content-Length': '0'} 2024-04-16T19:36:13.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ae429ee679657b-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.531 DEBUG:boto:b'yrjefh-17999falseobj6rd-Qw0l3074qMhqNL4JYtNr0TxMlwCtrue2024-04-16T19:36:03.132ZRGW11111111111111111NormalobjRawJNlJVVB1arOpDw6zUFO8KelA4nJ5false2024-04-16T19:36:03.112Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.532 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.532 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.532 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.532 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:13.533 DEBUG:boto:Data: 2024-04-16T19:36:13.533 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.533 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.533 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.533 DEBUG:boto:Params: {} 2024-04-16T19:36:13.533 DEBUG:boto:Token: None 2024-04-16T19:36:13.533 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj 2024-04-16T19:36:13.534 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A= 2024-04-16T19:36:13.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A=', 'Content-Length': '0'} 2024-04-16T19:36:13.536 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000315f68c218304404-00661ed32d-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.536 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.536 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.537 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.537 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:13.537 DEBUG:boto:Data: 2024-04-16T19:36:13.537 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.537 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.537 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.537 DEBUG:boto:Params: {} 2024-04-16T19:36:13.537 DEBUG:boto:Token: None 2024-04-16T19:36:13.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj 2024-04-16T19:36:13.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A= 2024-04-16T19:36:13.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A=', 'Content-Length': '0'} 2024-04-16T19:36:13.540 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000019ad7dac3375f55a-00661ed32d-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.541 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.541 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.541 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.541 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.541 DEBUG:boto:Method: GET 2024-04-16T19:36:13.541 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.541 DEBUG:boto:Data: 2024-04-16T19:36:13.541 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.542 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.542 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.542 DEBUG:boto:Params: {} 2024-04-16T19:36:13.542 DEBUG:boto:Token: None 2024-04-16T19:36:13.542 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI= 2024-04-16T19:36:13.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI=', 'Content-Length': '0'} 2024-04-16T19:36:13.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eec470410a3d7cd6-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.546 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.546 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.546 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.546 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.546 DEBUG:boto:Method: GET 2024-04-16T19:36:13.549 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.550 DEBUG:boto:Data: 2024-04-16T19:36:13.550 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.550 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.550 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.550 DEBUG:boto:Params: {} 2024-04-16T19:36:13.550 DEBUG:boto:Token: None 2024-04-16T19:36:13.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI= 2024-04-16T19:36:13.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI=', 'Content-Length': '0'} 2024-04-16T19:36:13.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000628dfd0ea58b3f06-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.553 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.554 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.554 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.554 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.554 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.554 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.554 DEBUG:boto:Data: 2024-04-16T19:36:13.554 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.554 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.554 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.555 DEBUG:boto:Params: {} 2024-04-16T19:36:13.555 DEBUG:boto:Token: None 2024-04-16T19:36:13.555 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ao7aJ5DjcNELpJU69qvRLTb5/QQ= 2024-04-16T19:36:13.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ao7aJ5DjcNELpJU69qvRLTb5/QQ=', 'Content-Length': '0'} 2024-04-16T19:36:13.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000682f2fbfc3147ef6-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.558 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.558 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.558 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.558 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.559 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.559 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.559 DEBUG:boto:Data: 2024-04-16T19:36:13.559 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.559 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.559 DEBUG:boto:Params: {} 2024-04-16T19:36:13.559 DEBUG:boto:Token: None 2024-04-16T19:36:13.560 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ao7aJ5DjcNELpJU69qvRLTb5/QQ= 2024-04-16T19:36:13.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ao7aJ5DjcNELpJU69qvRLTb5/QQ=', 'Content-Length': '0'} 2024-04-16T19:36:13.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc53993b57971fc6-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.563 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.563 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.563 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.563 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.563 DEBUG:boto:Method: GET 2024-04-16T19:36:13.563 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.563 DEBUG:boto:Data: 2024-04-16T19:36:13.563 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.564 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.564 DEBUG:boto:Params: {} 2024-04-16T19:36:13.564 DEBUG:boto:Token: None 2024-04-16T19:36:13.564 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI= 2024-04-16T19:36:13.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI=', 'Content-Length': '0'} 2024-04-16T19:36:13.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005eb49caecc643f53-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.567 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.567 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.567 DEBUG:boto:path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.567 DEBUG:boto:auth_path=/yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.568 DEBUG:boto:Method: GET 2024-04-16T19:36:13.568 DEBUG:boto:Path: /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.568 DEBUG:boto:Data: 2024-04-16T19:36:13.568 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.568 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.568 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.568 DEBUG:boto:Params: {} 2024-04-16T19:36:13.568 DEBUG:boto:Token: None 2024-04-16T19:36:13.569 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj?versionId=RawJNlJVVB1arOpDw6zUFO8KelA4nJ5 2024-04-16T19:36:13.569 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI= 2024-04-16T19:36:13.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrDQOVV20MV5tReSHvYcCymnXvI=', 'Content-Length': '0'} 2024-04-16T19:36:13.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'RawJNlJVVB1arOpDw6zUFO8KelA4nJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d784ffd7db60e9fe-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.571 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.572 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.572 DEBUG:boto:path=/yrjefh-17/?prefix=obj 2024-04-16T19:36:13.572 DEBUG:boto:auth_path=/yrjefh-17/?prefix=obj 2024-04-16T19:36:13.572 DEBUG:boto:Method: GET 2024-04-16T19:36:13.572 DEBUG:boto:Path: /yrjefh-17/?prefix=obj 2024-04-16T19:36:13.572 DEBUG:boto:Data: 2024-04-16T19:36:13.572 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.572 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.572 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.573 DEBUG:boto:Params: {} 2024-04-16T19:36:13.573 DEBUG:boto:Token: None 2024-04-16T19:36:13.573 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/ 2024-04-16T19:36:13.573 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZHOn0ue2obQRa3jdyePIoR6TgXE= 2024-04-16T19:36:13.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZHOn0ue2obQRa3jdyePIoR6TgXE=', 'Content-Length': '0'} 2024-04-16T19:36:13.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136fe71c619cb953-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.576 DEBUG:boto:b'yrjefh-17obj1000false' 2024-04-16T19:36:13.576 DEBUG:boto:path=/yrjefh-17/ 2024-04-16T19:36:13.576 DEBUG:boto:auth_path=/yrjefh-17/ 2024-04-16T19:36:13.577 DEBUG:boto:path=/yrjefh-17/?prefix=obj 2024-04-16T19:36:13.577 DEBUG:boto:auth_path=/yrjefh-17/?prefix=obj 2024-04-16T19:36:13.577 DEBUG:boto:Method: GET 2024-04-16T19:36:13.577 DEBUG:boto:Path: /yrjefh-17/?prefix=obj 2024-04-16T19:36:13.577 DEBUG:boto:Data: 2024-04-16T19:36:13.577 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.577 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.577 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.577 DEBUG:boto:Params: {} 2024-04-16T19:36:13.578 DEBUG:boto:Token: None 2024-04-16T19:36:13.578 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/ 2024-04-16T19:36:13.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZHOn0ue2obQRa3jdyePIoR6TgXE= 2024-04-16T19:36:13.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZHOn0ue2obQRa3jdyePIoR6TgXE=', 'Content-Length': '0'} 2024-04-16T19:36:13.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c400a4b0fc80b5d-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.622 DEBUG:boto:b'yrjefh-17obj1000false' 2024-04-16T19:36:13.623 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.623 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.623 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.623 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:13.623 DEBUG:boto:Data: 2024-04-16T19:36:13.623 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.624 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.624 DEBUG:boto:Params: {} 2024-04-16T19:36:13.624 DEBUG:boto:Token: None 2024-04-16T19:36:13.624 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj 2024-04-16T19:36:13.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A= 2024-04-16T19:36:13.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A=', 'Content-Length': '0'} 2024-04-16T19:36:13.627 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000007a63d754780ec50-00661ed32d-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.627 DEBUG:boto:path=/yrjefh-17/obj 2024-04-16T19:36:13.627 DEBUG:boto:auth_path=/yrjefh-17/obj 2024-04-16T19:36:13.627 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.627 DEBUG:boto:Path: /yrjefh-17/obj 2024-04-16T19:36:13.627 DEBUG:boto:Data: 2024-04-16T19:36:13.627 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.628 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.628 DEBUG:boto:Params: {} 2024-04-16T19:36:13.628 DEBUG:boto:Token: None 2024-04-16T19:36:13.628 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-17/obj 2024-04-16T19:36:13.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A= 2024-04-16T19:36:13.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VqmPEusec25KMpkZvkKVWnh6T2A=', 'Content-Length': '0'} 2024-04-16T19:36:13.631 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ea772bca41dec906-00661ed32d-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.631 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-17 zones={test-zone2, test-zone1} 2024-04-16T19:36:13.631 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-18 2024-04-16T19:36:13.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:13.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:13.632 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:13.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:13.753+0000 7f152333b800 1 bucket sync caught up with source: 2024-04-16T19:36:13.754 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.32.6, , ] 2024-04-16T19:36:13.754 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000002.32.6, , ] 2024-04-16T19:36:13.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:13.753+0000 7f152333b800 0 bucket checkpoint complete 2024-04-16T19:36:13.775 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-18 2024-04-16T19:36:13.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:13.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:13.776 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:13.900+0000 7f6ba952d800 1 bucket sync caught up with source: 2024-04-16T19:36:13.901 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.48.12, , ] 2024-04-16T19:36:13.902 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.48.12, , ] 2024-04-16T19:36:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:13.900+0000 7f6ba952d800 0 bucket checkpoint complete 2024-04-16T19:36:13.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-18 zones={test-zone2, test-zone1} 2024-04-16T19:36:13.922 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.922 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.923 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.923 DEBUG:boto:Path: /yrjefh-18/ 2024-04-16T19:36:13.923 DEBUG:boto:Data: 2024-04-16T19:36:13.923 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.923 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.923 DEBUG:boto:Params: {} 2024-04-16T19:36:13.923 DEBUG:boto:Token: None 2024-04-16T19:36:13.923 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/ 2024-04-16T19:36:13.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fIEx1lxA9ecdXxCH3ZiREd+aQhA= 2024-04-16T19:36:13.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fIEx1lxA9ecdXxCH3ZiREd+aQhA=', 'Content-Length': '0'} 2024-04-16T19:36:13.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077d10bc29b73b0dd-00661ed32d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.926 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.926 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.926 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.926 DEBUG:boto:Path: /yrjefh-18/ 2024-04-16T19:36:13.927 DEBUG:boto:Data: 2024-04-16T19:36:13.927 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.927 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.927 DEBUG:boto:Params: {} 2024-04-16T19:36:13.927 DEBUG:boto:Token: None 2024-04-16T19:36:13.927 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/ 2024-04-16T19:36:13.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fIEx1lxA9ecdXxCH3ZiREd+aQhA= 2024-04-16T19:36:13.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fIEx1lxA9ecdXxCH3ZiREd+aQhA=', 'Content-Length': '0'} 2024-04-16T19:36:13.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceb80bb6ef356c6d-00661ed32d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:36:13.930 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.930 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.930 DEBUG:boto:path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.930 DEBUG:boto:auth_path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.930 DEBUG:boto:Method: GET 2024-04-16T19:36:13.930 DEBUG:boto:Path: /yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.930 DEBUG:boto:Data: 2024-04-16T19:36:13.931 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.931 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.931 DEBUG:boto:Params: {} 2024-04-16T19:36:13.931 DEBUG:boto:Token: None 2024-04-16T19:36:13.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/?versions 2024-04-16T19:36:13.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g= 2024-04-16T19:36:13.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g=', 'Content-Length': '0'} 2024-04-16T19:36:13.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e94cd3b7c0b33e7-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.934 DEBUG:boto:b'yrjefh-18999falseobj7wd2AlZMFnbMsWCsXf9evlG0GJ-Rftftrue2024-04-16T19:36:03.162ZRGW11111111111111111NormalobjLaos7TY1xkkCTnNdXF3Fy8QV0UbNJTZfalse2024-04-16T19:36:03.144Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:36:13.935 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.935 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.935 DEBUG:boto:path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.935 DEBUG:boto:auth_path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.935 DEBUG:boto:Method: GET 2024-04-16T19:36:13.935 DEBUG:boto:Path: /yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.935 DEBUG:boto:Data: 2024-04-16T19:36:13.935 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.935 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.935 DEBUG:boto:Params: {} 2024-04-16T19:36:13.936 DEBUG:boto:Token: None 2024-04-16T19:36:13.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/?versions 2024-04-16T19:36:13.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g= 2024-04-16T19:36:13.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g=', 'Content-Length': '0'} 2024-04-16T19:36:13.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000067aabe0a569d8b0-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.938 DEBUG:boto:b'yrjefh-18999falseobj7wd2AlZMFnbMsWCsXf9evlG0GJ-Rftftrue2024-04-16T19:36:03.162ZRGW11111111111111111NormalobjLaos7TY1xkkCTnNdXF3Fy8QV0UbNJTZfalse2024-04-16T19:36:03.144Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:13.939 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.939 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.939 DEBUG:boto:path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.939 DEBUG:boto:auth_path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.939 DEBUG:boto:Method: GET 2024-04-16T19:36:13.939 DEBUG:boto:Path: /yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.939 DEBUG:boto:Data: 2024-04-16T19:36:13.939 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.940 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.940 DEBUG:boto:Params: {} 2024-04-16T19:36:13.940 DEBUG:boto:Token: None 2024-04-16T19:36:13.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/?versions 2024-04-16T19:36:13.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g= 2024-04-16T19:36:13.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g=', 'Content-Length': '0'} 2024-04-16T19:36:13.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa4f0450ca26d4b-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.943 DEBUG:boto:b'yrjefh-18999falseobj7wd2AlZMFnbMsWCsXf9evlG0GJ-Rftftrue2024-04-16T19:36:03.162ZRGW11111111111111111NormalobjLaos7TY1xkkCTnNdXF3Fy8QV0UbNJTZfalse2024-04-16T19:36:03.144Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.943 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.943 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.943 DEBUG:boto:path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.943 DEBUG:boto:auth_path=/yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.943 DEBUG:boto:Method: GET 2024-04-16T19:36:13.943 DEBUG:boto:Path: /yrjefh-18/?versions&max-keys=999 2024-04-16T19:36:13.944 DEBUG:boto:Data: 2024-04-16T19:36:13.944 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.944 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.944 DEBUG:boto:Params: {} 2024-04-16T19:36:13.944 DEBUG:boto:Token: None 2024-04-16T19:36:13.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/?versions 2024-04-16T19:36:13.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g= 2024-04-16T19:36:13.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81E+VTrK1iZgz76ijAuQztxbi/g=', 'Content-Length': '0'} 2024-04-16T19:36:13.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c332b6e5604365e7-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.947 DEBUG:boto:b'yrjefh-18999falseobj7wd2AlZMFnbMsWCsXf9evlG0GJ-Rftftrue2024-04-16T19:36:03.162ZRGW11111111111111111NormalobjLaos7TY1xkkCTnNdXF3Fy8QV0UbNJTZfalse2024-04-16T19:36:03.144Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDRGW11111111111111111Normal' 2024-04-16T19:36:13.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.947 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.947 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.947 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.948 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:13.948 DEBUG:boto:Data: 2024-04-16T19:36:13.948 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.948 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.948 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.948 DEBUG:boto:Params: {} 2024-04-16T19:36:13.948 DEBUG:boto:Token: None 2024-04-16T19:36:13.948 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj 2024-04-16T19:36:13.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GCKvuRy2W/SG0mObdHU9qtV3eIc= 2024-04-16T19:36:13.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GCKvuRy2W/SG0mObdHU9qtV3eIc=', 'Content-Length': '0'} 2024-04-16T19:36:13.951 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000044c25b04d4c2e9ec-00661ed32d-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.951 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.951 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.951 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.951 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:13.952 DEBUG:boto:Data: 2024-04-16T19:36:13.952 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.952 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.952 DEBUG:boto:Params: {} 2024-04-16T19:36:13.952 DEBUG:boto:Token: None 2024-04-16T19:36:13.952 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj 2024-04-16T19:36:13.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GCKvuRy2W/SG0mObdHU9qtV3eIc= 2024-04-16T19:36:13.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GCKvuRy2W/SG0mObdHU9qtV3eIc=', 'Content-Length': '0'} 2024-04-16T19:36:13.955 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000676483aed2720e3c-00661ed32d-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.955 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.955 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.955 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.955 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.955 DEBUG:boto:Method: GET 2024-04-16T19:36:13.955 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.955 DEBUG:boto:Data: 2024-04-16T19:36:13.956 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.956 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.956 DEBUG:boto:Params: {} 2024-04-16T19:36:13.956 DEBUG:boto:Token: None 2024-04-16T19:36:13.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo= 2024-04-16T19:36:13.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo=', 'Content-Length': '0'} 2024-04-16T19:36:13.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001344f58edb977dc0-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.958 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.958 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.958 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.958 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.958 DEBUG:boto:Method: GET 2024-04-16T19:36:13.958 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.959 DEBUG:boto:Data: 2024-04-16T19:36:13.959 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.959 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.959 DEBUG:boto:Params: {} 2024-04-16T19:36:13.959 DEBUG:boto:Token: None 2024-04-16T19:36:13.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo= 2024-04-16T19:36:13.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo=', 'Content-Length': '0'} 2024-04-16T19:36:13.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc0480dac740706a-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.961 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.961 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.962 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.962 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.962 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.962 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.962 DEBUG:boto:Data: 2024-04-16T19:36:13.962 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.962 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.962 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.962 DEBUG:boto:Params: {} 2024-04-16T19:36:13.962 DEBUG:boto:Token: None 2024-04-16T19:36:13.963 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d3m4OHRcSoK9B+Y2WlJMKKqkpiU= 2024-04-16T19:36:13.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d3m4OHRcSoK9B+Y2WlJMKKqkpiU=', 'Content-Length': '0'} 2024-04-16T19:36:13.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f53833daa4ec6d5-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.965 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.965 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.965 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.965 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.965 DEBUG:boto:Method: HEAD 2024-04-16T19:36:13.965 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.965 DEBUG:boto:Data: 2024-04-16T19:36:13.965 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.965 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.966 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.966 DEBUG:boto:Params: {} 2024-04-16T19:36:13.966 DEBUG:boto:Token: None 2024-04-16T19:36:13.966 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.966 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d3m4OHRcSoK9B+Y2WlJMKKqkpiU= 2024-04-16T19:36:13.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d3m4OHRcSoK9B+Y2WlJMKKqkpiU=', 'Content-Length': '0'} 2024-04-16T19:36:13.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000449a6e97db3ccb41-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:13.968 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.968 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.968 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.968 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.968 DEBUG:boto:Method: GET 2024-04-16T19:36:13.968 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.969 DEBUG:boto:Data: 2024-04-16T19:36:13.969 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.969 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.969 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.969 DEBUG:boto:Params: {} 2024-04-16T19:36:13.969 DEBUG:boto:Token: None 2024-04-16T19:36:13.969 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.969 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo= 2024-04-16T19:36:13.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo=', 'Content-Length': '0'} 2024-04-16T19:36:13.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004cb3eb85bd88ba5c-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.971 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:13.971 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:13.971 DEBUG:boto:path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.971 DEBUG:boto:auth_path=/yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.972 DEBUG:boto:Method: GET 2024-04-16T19:36:13.972 DEBUG:boto:Path: /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.972 DEBUG:boto:Data: 2024-04-16T19:36:13.972 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.972 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:13.972 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.972 DEBUG:boto:Params: {} 2024-04-16T19:36:13.972 DEBUG:boto:Token: None 2024-04-16T19:36:13.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/obj?versionId=Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ 2024-04-16T19:36:13.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo= 2024-04-16T19:36:13.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vZYHnHBHVB6hnc7TaZ9gV97KcZo=', 'Content-Length': '0'} 2024-04-16T19:36:13.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:36:03 GMT'), ('x-amz-version-id', 'Laos7TY1xkkCTnNdXF3Fy8QV0UbNJTZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:36:05 GMT'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba655841bd781576-00661ed32d-4361-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:13.974 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:13.975 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:13.975 DEBUG:boto:path=/yrjefh-18/?prefix=obj 2024-04-16T19:36:13.975 DEBUG:boto:auth_path=/yrjefh-18/?prefix=obj 2024-04-16T19:36:13.975 DEBUG:boto:Method: GET 2024-04-16T19:36:13.975 DEBUG:boto:Path: /yrjefh-18/?prefix=obj 2024-04-16T19:36:13.975 DEBUG:boto:Data: 2024-04-16T19:36:13.975 DEBUG:boto:Headers: {} 2024-04-16T19:36:13.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:13.975 DEBUG:boto:Port: 8000 2024-04-16T19:36:13.975 DEBUG:boto:Params: {} 2024-04-16T19:36:13.975 DEBUG:boto:Token: None 2024-04-16T19:36:13.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:13 GMT /yrjefh-18/ 2024-04-16T19:36:13.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5WkziZofkY8Tmn2ALxs2p35vSK8= 2024-04-16T19:36:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5WkziZofkY8Tmn2ALxs2p35vSK8=', 'Content-Length': '0'} 2024-04-16T19:36:13.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b2390f96ba379e-00661ed32d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.018 DEBUG:boto:b'yrjefh-18obj1000false' 2024-04-16T19:36:14.019 DEBUG:boto:path=/yrjefh-18/ 2024-04-16T19:36:14.019 DEBUG:boto:auth_path=/yrjefh-18/ 2024-04-16T19:36:14.019 DEBUG:boto:path=/yrjefh-18/?prefix=obj 2024-04-16T19:36:14.019 DEBUG:boto:auth_path=/yrjefh-18/?prefix=obj 2024-04-16T19:36:14.019 DEBUG:boto:Method: GET 2024-04-16T19:36:14.019 DEBUG:boto:Path: /yrjefh-18/?prefix=obj 2024-04-16T19:36:14.019 DEBUG:boto:Data: 2024-04-16T19:36:14.019 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:14.019 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.020 DEBUG:boto:Params: {} 2024-04-16T19:36:14.020 DEBUG:boto:Token: None 2024-04-16T19:36:14.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-18/ 2024-04-16T19:36:14.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VUqsbh6EiVNQ6Vxovkg5hy2agTs= 2024-04-16T19:36:14.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VUqsbh6EiVNQ6Vxovkg5hy2agTs=', 'Content-Length': '0'} 2024-04-16T19:36:14.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2dc1c37b647d4b-00661ed32e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.022 DEBUG:boto:b'yrjefh-18obj1000false' 2024-04-16T19:36:14.023 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:14.023 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:14.023 DEBUG:boto:Method: HEAD 2024-04-16T19:36:14.023 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:14.023 DEBUG:boto:Data: 2024-04-16T19:36:14.023 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.023 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:14.023 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.023 DEBUG:boto:Params: {} 2024-04-16T19:36:14.023 DEBUG:boto:Token: None 2024-04-16T19:36:14.023 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-18/obj 2024-04-16T19:36:14.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7iYaSUtdEk8GwwFAsBpS2vTsOWI= 2024-04-16T19:36:14.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7iYaSUtdEk8GwwFAsBpS2vTsOWI=', 'Content-Length': '0'} 2024-04-16T19:36:14.026 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000022491ae8d1e7672b-00661ed32e-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.026 DEBUG:boto:path=/yrjefh-18/obj 2024-04-16T19:36:14.026 DEBUG:boto:auth_path=/yrjefh-18/obj 2024-04-16T19:36:14.027 DEBUG:boto:Method: HEAD 2024-04-16T19:36:14.027 DEBUG:boto:Path: /yrjefh-18/obj 2024-04-16T19:36:14.027 DEBUG:boto:Data: 2024-04-16T19:36:14.027 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:14.027 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.027 DEBUG:boto:Params: {} 2024-04-16T19:36:14.027 DEBUG:boto:Token: None 2024-04-16T19:36:14.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-18/obj 2024-04-16T19:36:14.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7iYaSUtdEk8GwwFAsBpS2vTsOWI= 2024-04-16T19:36:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7iYaSUtdEk8GwwFAsBpS2vTsOWI=', 'Content-Length': '0'} 2024-04-16T19:36:14.030 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004dcacb66c2b59584-00661ed32e-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-18 zones={test-zone2, test-zone1} 2024-04-16T19:36:14.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-16T19:36:14.031 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-19 2024-04-16T19:36:14.031 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:14.031 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:14.031 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.031 DEBUG:boto:Path: /yrjefh-19/ 2024-04-16T19:36:14.031 DEBUG:boto:Data: 2024-04-16T19:36:14.031 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:14.031 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.031 DEBUG:boto:Params: {} 2024-04-16T19:36:14.032 DEBUG:boto:Token: None 2024-04-16T19:36:14.032 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-19/ 2024-04-16T19:36:14.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GlBIxSfY0vP8fgjfSKDrn+fIQ8I= 2024-04-16T19:36:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GlBIxSfY0vP8fgjfSKDrn+fIQ8I=', 'Content-Length': '0'} 2024-04-16T19:36:14.047 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7945b48bb9c2bb8-00661ed32e-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.047 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-20 2024-04-16T19:36:14.047 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:14.047 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:14.047 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.047 DEBUG:boto:Path: /yrjefh-20/ 2024-04-16T19:36:14.047 DEBUG:boto:Data: 2024-04-16T19:36:14.047 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.048 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:14.048 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.048 DEBUG:boto:Params: {} 2024-04-16T19:36:14.048 DEBUG:boto:Token: None 2024-04-16T19:36:14.048 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-20/ 2024-04-16T19:36:14.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nnIWhwDCPqxLpzN7fQNj7kn4nog= 2024-04-16T19:36:14.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nnIWhwDCPqxLpzN7fQNj7kn4nog=', 'Content-Length': '0'} 2024-04-16T19:36:14.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab6e80061976d0c4-00661ed32e-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.074 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:14.075 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:14.075 DEBUG:boto:path=/yrjefh-19/?versioning 2024-04-16T19:36:14.075 DEBUG:boto:auth_path=/yrjefh-19/?versioning 2024-04-16T19:36:14.075 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.075 DEBUG:boto:Path: /yrjefh-19/?versioning 2024-04-16T19:36:14.075 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:36:14.075 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:14.075 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.075 DEBUG:boto:Params: {} 2024-04-16T19:36:14.075 DEBUG:boto:Token: None 2024-04-16T19:36:14.075 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-19/?versioning 2024-04-16T19:36:14.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rk4cHG4TCt/Y1y7q/Z6bi2s8R8Y= 2024-04-16T19:36:14.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rk4cHG4TCt/Y1y7q/Z6bi2s8R8Y=', 'Content-Length': '228'} 2024-04-16T19:36:14.081 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009147609505108aa1-00661ed32e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.081 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:14.081 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:14.081 DEBUG:boto:path=/yrjefh-19/?versioning 2024-04-16T19:36:14.081 DEBUG:boto:auth_path=/yrjefh-19/?versioning 2024-04-16T19:36:14.082 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.082 DEBUG:boto:Path: /yrjefh-19/?versioning 2024-04-16T19:36:14.082 DEBUG:boto:Data: Suspended Disabled 2024-04-16T19:36:14.082 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.082 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:14.082 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.082 DEBUG:boto:Params: {} 2024-04-16T19:36:14.082 DEBUG:boto:Token: None 2024-04-16T19:36:14.082 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-19/?versioning 2024-04-16T19:36:14.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rk4cHG4TCt/Y1y7q/Z6bi2s8R8Y= 2024-04-16T19:36:14.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rk4cHG4TCt/Y1y7q/Z6bi2s8R8Y=', 'Content-Length': '230'} 2024-04-16T19:36:14.087 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5634ac61df9a611-00661ed32e-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.088 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:14.088 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:14.088 DEBUG:boto:path=/yrjefh-20/?versioning 2024-04-16T19:36:14.088 DEBUG:boto:auth_path=/yrjefh-20/?versioning 2024-04-16T19:36:14.088 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.088 DEBUG:boto:Path: /yrjefh-20/?versioning 2024-04-16T19:36:14.088 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:36:14.088 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.088 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:14.088 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.088 DEBUG:boto:Params: {} 2024-04-16T19:36:14.088 DEBUG:boto:Token: None 2024-04-16T19:36:14.089 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-20/?versioning 2024-04-16T19:36:14.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D9YrLs4jFb0XbWzsjlR25wCOZmI= 2024-04-16T19:36:14.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D9YrLs4jFb0XbWzsjlR25wCOZmI=', 'Content-Length': '228'} 2024-04-16T19:36:14.097 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abd6fb0faea03e6e-00661ed32e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.098 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:14.098 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:14.098 DEBUG:boto:path=/yrjefh-20/?versioning 2024-04-16T19:36:14.098 DEBUG:boto:auth_path=/yrjefh-20/?versioning 2024-04-16T19:36:14.098 DEBUG:boto:Method: PUT 2024-04-16T19:36:14.098 DEBUG:boto:Path: /yrjefh-20/?versioning 2024-04-16T19:36:14.098 DEBUG:boto:Data: Suspended Disabled 2024-04-16T19:36:14.098 DEBUG:boto:Headers: {} 2024-04-16T19:36:14.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:14.098 DEBUG:boto:Port: 8000 2024-04-16T19:36:14.098 DEBUG:boto:Params: {} 2024-04-16T19:36:14.099 DEBUG:boto:Token: None 2024-04-16T19:36:14.099 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:14 GMT /yrjefh-20/?versioning 2024-04-16T19:36:14.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D9YrLs4jFb0XbWzsjlR25wCOZmI= 2024-04-16T19:36:14.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D9YrLs4jFb0XbWzsjlR25wCOZmI=', 'Content-Length': '230'} 2024-04-16T19:36:14.108 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005ad9d5773b353ed4-00661ed32e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:14.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:14.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:14.109 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:14.156 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:14.156 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:14.205 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:36:14.225 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296174.104097_198.1","last_update":"2024-04-16T19:36:14.104097Z"}] 2024-04-16T19:36:14.226 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296174.104097_198.1'} 2024-04-16T19:36:14.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:36:14.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:14.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:14.226 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:14.275 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:14.275 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:14.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:14.323+0000 7f6480dad800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:14.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:14.324+0000 7f6480dad800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:14.345 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296174.062979_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:14.062979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:14Z"} 2024-04-16T19:36:14.345 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296174.062979_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:14.062979Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:14Z"} 2024-04-16T19:36:14.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296174.062979_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:14.062979Z', 'realm_epoch': 2}}] 2024-04-16T19:36:14.346 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296174.104097_198.1'} 2024-04-16T19:36:14.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296174.062979_190.1'} 2024-04-16T19:36:14.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296174.104097_198.1 target=1_1713296174.062979_190.1 2024-04-16T19:36:19.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:19.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:19.347 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:19.402 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:19.402 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:19.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:19.454+0000 7fab3623d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:19.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:19.455+0000 7fab3623d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:19.479 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296174.104097_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:14.104097Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:19Z"} 2024-04-16T19:36:19.479 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296174.104097_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:14.104097Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:19Z"} 2024-04-16T19:36:19.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296174.104097_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:14.104097Z', 'realm_epoch': 2}}] 2024-04-16T19:36:19.479 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296174.104097_198.1'} 2024-04-16T19:36:19.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296174.104097_198.1'} 2024-04-16T19:36:19.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:36:19.480 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:19.480 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:19.480 DEBUG:boto:Method: HEAD 2024-04-16T19:36:19.480 DEBUG:boto:Path: /yrjefh-19/ 2024-04-16T19:36:19.480 DEBUG:boto:Data: 2024-04-16T19:36:19.480 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.480 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:19.480 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.480 DEBUG:boto:Params: {} 2024-04-16T19:36:19.481 DEBUG:boto:Token: None 2024-04-16T19:36:19.481 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-19/ 2024-04-16T19:36:19.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VrKmYHfFgsB+odX12SO7fYo3Yao= 2024-04-16T19:36:19.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VrKmYHfFgsB+odX12SO7fYo3Yao=', 'Content-Length': '0'} 2024-04-16T19:36:19.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bd07c29d9ce1261-00661ed333-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.484 DEBUG:boto:path=/yrjefh-19/obj 2024-04-16T19:36:19.484 DEBUG:boto:auth_path=/yrjefh-19/obj 2024-04-16T19:36:19.484 DEBUG:boto:Method: PUT 2024-04-16T19:36:19.484 DEBUG:boto:Path: /yrjefh-19/obj 2024-04-16T19:36:19.484 DEBUG:boto:Data: 2024-04-16T19:36:19.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:36:19.484 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:19.484 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.484 DEBUG:boto:Params: {} 2024-04-16T19:36:19.485 DEBUG:boto:Token: None 2024-04-16T19:36:19.485 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-19/obj 2024-04-16T19:36:19.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:phPEOtjGhSpohFmq8NvWz3fc5nQ= 2024-04-16T19:36:19.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:phPEOtjGhSpohFmq8NvWz3fc5nQ='} 2024-04-16T19:36:19.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003553a40c37208b1c-00661ed333-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.493 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:19.493 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:19.493 DEBUG:boto:Method: HEAD 2024-04-16T19:36:19.493 DEBUG:boto:Path: /yrjefh-19/ 2024-04-16T19:36:19.493 DEBUG:boto:Data: 2024-04-16T19:36:19.493 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.493 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:19.493 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.493 DEBUG:boto:Params: {} 2024-04-16T19:36:19.493 DEBUG:boto:Token: None 2024-04-16T19:36:19.493 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-19/ 2024-04-16T19:36:19.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VrKmYHfFgsB+odX12SO7fYo3Yao= 2024-04-16T19:36:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VrKmYHfFgsB+odX12SO7fYo3Yao=', 'Content-Length': '0'} 2024-04-16T19:36:19.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb64820179981c85-00661ed333-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.496 DEBUG:boto:path=/yrjefh-19/obj 2024-04-16T19:36:19.496 DEBUG:boto:auth_path=/yrjefh-19/obj 2024-04-16T19:36:19.496 DEBUG:boto:Method: DELETE 2024-04-16T19:36:19.496 DEBUG:boto:Path: /yrjefh-19/obj 2024-04-16T19:36:19.497 DEBUG:boto:Data: 2024-04-16T19:36:19.497 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:19.497 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.497 DEBUG:boto:Params: {} 2024-04-16T19:36:19.497 DEBUG:boto:Token: None 2024-04-16T19:36:19.497 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-19/obj 2024-04-16T19:36:19.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XcJWGEmKQORUaqysIqKPpdF1A30= 2024-04-16T19:36:19.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XcJWGEmKQORUaqysIqKPpdF1A30=', 'Content-Length': '0'} 2024-04-16T19:36:19.508 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000004bb98c25d65b4941-00661ed333-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.508 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:19.509 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:19.509 DEBUG:boto:Method: HEAD 2024-04-16T19:36:19.509 DEBUG:boto:Path: /yrjefh-20/ 2024-04-16T19:36:19.509 DEBUG:boto:Data: 2024-04-16T19:36:19.509 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.509 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:19.509 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.509 DEBUG:boto:Params: {} 2024-04-16T19:36:19.509 DEBUG:boto:Token: None 2024-04-16T19:36:19.509 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-20/ 2024-04-16T19:36:19.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kKcN0L6UPIRu6ksoaZi7mKs1mAQ= 2024-04-16T19:36:19.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kKcN0L6UPIRu6ksoaZi7mKs1mAQ=', 'Content-Length': '0'} 2024-04-16T19:36:19.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e9d97c77e3a5dd0-00661ed333-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.512 DEBUG:boto:path=/yrjefh-20/obj 2024-04-16T19:36:19.512 DEBUG:boto:auth_path=/yrjefh-20/obj 2024-04-16T19:36:19.512 DEBUG:boto:Method: PUT 2024-04-16T19:36:19.512 DEBUG:boto:Path: /yrjefh-20/obj 2024-04-16T19:36:19.513 DEBUG:boto:Data: 2024-04-16T19:36:19.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-16T19:36:19.513 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:19.513 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.513 DEBUG:boto:Params: {} 2024-04-16T19:36:19.513 DEBUG:boto:Token: None 2024-04-16T19:36:19.513 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-20/obj 2024-04-16T19:36:19.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DFQ8XdOAr8UnxSKBJjSU7yhpbnQ= 2024-04-16T19:36:19.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DFQ8XdOAr8UnxSKBJjSU7yhpbnQ='} 2024-04-16T19:36:19.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000542d46d580571727-00661ed333-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.520 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:19.520 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:19.520 DEBUG:boto:Method: HEAD 2024-04-16T19:36:19.520 DEBUG:boto:Path: /yrjefh-20/ 2024-04-16T19:36:19.520 DEBUG:boto:Data: 2024-04-16T19:36:19.520 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.520 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:19.520 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.520 DEBUG:boto:Params: {} 2024-04-16T19:36:19.520 DEBUG:boto:Token: None 2024-04-16T19:36:19.521 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-20/ 2024-04-16T19:36:19.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kKcN0L6UPIRu6ksoaZi7mKs1mAQ= 2024-04-16T19:36:19.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kKcN0L6UPIRu6ksoaZi7mKs1mAQ=', 'Content-Length': '0'} 2024-04-16T19:36:19.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083b2f7f6a6a12362-00661ed333-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.523 DEBUG:boto:path=/yrjefh-20/obj 2024-04-16T19:36:19.523 DEBUG:boto:auth_path=/yrjefh-20/obj 2024-04-16T19:36:19.523 DEBUG:boto:Method: DELETE 2024-04-16T19:36:19.523 DEBUG:boto:Path: /yrjefh-20/obj 2024-04-16T19:36:19.523 DEBUG:boto:Data: 2024-04-16T19:36:19.523 DEBUG:boto:Headers: {} 2024-04-16T19:36:19.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:19.523 DEBUG:boto:Port: 8000 2024-04-16T19:36:19.523 DEBUG:boto:Params: {} 2024-04-16T19:36:19.523 DEBUG:boto:Token: None 2024-04-16T19:36:19.523 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:19 GMT /yrjefh-20/obj 2024-04-16T19:36:19.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nAAxupadn6qWdgoEn02PUUAozgA= 2024-04-16T19:36:19.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:19 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nAAxupadn6qWdgoEn02PUUAozgA=', 'Content-Length': '0'} 2024-04-16T19:36:19.533 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000000e43792b268257b5-00661ed333-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:19.533 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-19 2024-04-16T19:36:19.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:19.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:19.534 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:19.588 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:19.589 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:19.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:19.640+0000 7f370672c800 1 waiting to reach incremental sync.. 2024-04-16T19:36:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:24.641+0000 7f370672c800 1 waiting to reach incremental sync.. 2024-04-16T19:36:29.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:29.643+0000 7f370672c800 1 waiting to reach incremental sync.. 2024-04-16T19:36:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:34.643+0000 7f370672c800 1 waiting to reach incremental sync.. 2024-04-16T19:36:39.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:39.646+0000 7f370672c800 1 bucket sync caught up with source: 2024-04-16T19:36:39.648 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000003.13.7, , ] 2024-04-16T19:36:39.648 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000003.13.7, , ] 2024-04-16T19:36:39.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:39.646+0000 7f370672c800 0 bucket checkpoint complete 2024-04-16T19:36:39.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-19 2024-04-16T19:36:39.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:39.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:39.673 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:39.736 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:39.736 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:39.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:39.796+0000 7f6b0ba24800 1 waiting to reach incremental sync.. 2024-04-16T19:36:44.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:44.797+0000 7f6b0ba24800 1 bucket sync caught up with source: 2024-04-16T19:36:44.800 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.16.14, , ] 2024-04-16T19:36:44.800 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.16.14, , ] 2024-04-16T19:36:44.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:44.797+0000 7f6b0ba24800 0 bucket checkpoint complete 2024-04-16T19:36:44.826 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-19 zones={test-zone2, test-zone1} 2024-04-16T19:36:44.826 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.826 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.826 DEBUG:boto:Method: HEAD 2024-04-16T19:36:44.826 DEBUG:boto:Path: /yrjefh-19/ 2024-04-16T19:36:44.827 DEBUG:boto:Data: 2024-04-16T19:36:44.827 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:44.827 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.827 DEBUG:boto:Params: {} 2024-04-16T19:36:44.827 DEBUG:boto:Token: None 2024-04-16T19:36:44.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/ 2024-04-16T19:36:44.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YbGkzh5dXo3oZ2dz9kOpr6Bm9D0= 2024-04-16T19:36:44.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YbGkzh5dXo3oZ2dz9kOpr6Bm9D0=', 'Content-Length': '0'} 2024-04-16T19:36:44.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c41e865e4fbcb4e-00661ed34c-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.830 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.830 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.830 DEBUG:boto:Method: HEAD 2024-04-16T19:36:44.830 DEBUG:boto:Path: /yrjefh-19/ 2024-04-16T19:36:44.830 DEBUG:boto:Data: 2024-04-16T19:36:44.830 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.830 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:44.830 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.830 DEBUG:boto:Params: {} 2024-04-16T19:36:44.831 DEBUG:boto:Token: None 2024-04-16T19:36:44.831 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/ 2024-04-16T19:36:44.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YbGkzh5dXo3oZ2dz9kOpr6Bm9D0= 2024-04-16T19:36:44.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YbGkzh5dXo3oZ2dz9kOpr6Bm9D0=', 'Content-Length': '0'} 2024-04-16T19:36:44.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068976e30b92a5f0b-00661ed34c-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.834 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:36:44.834 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.834 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.834 DEBUG:boto:path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.834 DEBUG:boto:auth_path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.834 DEBUG:boto:Method: GET 2024-04-16T19:36:44.834 DEBUG:boto:Path: /yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.834 DEBUG:boto:Data: 2024-04-16T19:36:44.834 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:44.834 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.834 DEBUG:boto:Params: {} 2024-04-16T19:36:44.835 DEBUG:boto:Token: None 2024-04-16T19:36:44.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/?versions 2024-04-16T19:36:44.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg= 2024-04-16T19:36:44.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg=', 'Content-Length': '0'} 2024-04-16T19:36:44.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9fcce3360ab8d5-00661ed34c-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.837 DEBUG:boto:b'yrjefh-19999falseobjnulltrue2024-04-16T19:36:19.498ZRGW11111111111111111Normal' 2024-04-16T19:36:44.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:44.838 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:36:44.838 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.838 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.838 DEBUG:boto:path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.838 DEBUG:boto:auth_path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.839 DEBUG:boto:Method: GET 2024-04-16T19:36:44.839 DEBUG:boto:Path: /yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.839 DEBUG:boto:Data: 2024-04-16T19:36:44.839 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:44.839 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.839 DEBUG:boto:Params: {} 2024-04-16T19:36:44.839 DEBUG:boto:Token: None 2024-04-16T19:36:44.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/?versions 2024-04-16T19:36:44.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg= 2024-04-16T19:36:44.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg=', 'Content-Length': '0'} 2024-04-16T19:36:44.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5df946ffb9a06b6-00661ed34c-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.842 DEBUG:boto:b'yrjefh-19999falseobjnulltrue2024-04-16T19:36:19.498ZRGW11111111111111111Normal' 2024-04-16T19:36:44.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:44.842 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.842 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.843 DEBUG:boto:path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.843 DEBUG:boto:auth_path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.843 DEBUG:boto:Method: GET 2024-04-16T19:36:44.843 DEBUG:boto:Path: /yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.843 DEBUG:boto:Data: 2024-04-16T19:36:44.843 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:44.843 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.843 DEBUG:boto:Params: {} 2024-04-16T19:36:44.843 DEBUG:boto:Token: None 2024-04-16T19:36:44.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/?versions 2024-04-16T19:36:44.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg= 2024-04-16T19:36:44.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg=', 'Content-Length': '0'} 2024-04-16T19:36:44.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8530051612b2a07-00661ed34c-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.846 DEBUG:boto:b'yrjefh-19999falseobjnulltrue2024-04-16T19:36:19.498ZRGW11111111111111111Normal' 2024-04-16T19:36:44.846 DEBUG:boto:path=/yrjefh-19/ 2024-04-16T19:36:44.846 DEBUG:boto:auth_path=/yrjefh-19/ 2024-04-16T19:36:44.846 DEBUG:boto:path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.847 DEBUG:boto:auth_path=/yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.847 DEBUG:boto:Method: GET 2024-04-16T19:36:44.847 DEBUG:boto:Path: /yrjefh-19/?versions&max-keys=999 2024-04-16T19:36:44.847 DEBUG:boto:Data: 2024-04-16T19:36:44.847 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.847 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:44.847 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.847 DEBUG:boto:Params: {} 2024-04-16T19:36:44.847 DEBUG:boto:Token: None 2024-04-16T19:36:44.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/?versions 2024-04-16T19:36:44.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg= 2024-04-16T19:36:44.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ohDRmrSKODx3kzTz+FNHWW0Ypg=', 'Content-Length': '0'} 2024-04-16T19:36:44.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692cb061c52e94ce-00661ed34c-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.850 DEBUG:boto:b'yrjefh-19999falseobjnulltrue2024-04-16T19:36:19.498ZRGW11111111111111111Normal' 2024-04-16T19:36:44.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:44.850 DEBUG:boto:path=/yrjefh-19/obj 2024-04-16T19:36:44.851 DEBUG:boto:auth_path=/yrjefh-19/obj 2024-04-16T19:36:44.851 DEBUG:boto:Method: HEAD 2024-04-16T19:36:44.851 DEBUG:boto:Path: /yrjefh-19/obj 2024-04-16T19:36:44.851 DEBUG:boto:Data: 2024-04-16T19:36:44.851 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.851 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:44.851 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.851 DEBUG:boto:Params: {} 2024-04-16T19:36:44.851 DEBUG:boto:Token: None 2024-04-16T19:36:44.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/obj 2024-04-16T19:36:44.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fsh2i5i2Eq2Od8eGxKKqVgio4Xk= 2024-04-16T19:36:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fsh2i5i2Eq2Od8eGxKKqVgio4Xk=', 'Content-Length': '0'} 2024-04-16T19:36:44.853 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009838efd92334090e-00661ed34c-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.854 DEBUG:boto:path=/yrjefh-19/obj 2024-04-16T19:36:44.854 DEBUG:boto:auth_path=/yrjefh-19/obj 2024-04-16T19:36:44.854 DEBUG:boto:Method: HEAD 2024-04-16T19:36:44.854 DEBUG:boto:Path: /yrjefh-19/obj 2024-04-16T19:36:44.854 DEBUG:boto:Data: 2024-04-16T19:36:44.854 DEBUG:boto:Headers: {} 2024-04-16T19:36:44.854 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:44.854 DEBUG:boto:Port: 8000 2024-04-16T19:36:44.854 DEBUG:boto:Params: {} 2024-04-16T19:36:44.854 DEBUG:boto:Token: None 2024-04-16T19:36:44.854 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:44 GMT /yrjefh-19/obj 2024-04-16T19:36:44.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fsh2i5i2Eq2Od8eGxKKqVgio4Xk= 2024-04-16T19:36:44.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fsh2i5i2Eq2Od8eGxKKqVgio4Xk=', 'Content-Length': '0'} 2024-04-16T19:36:44.856 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002ac082e7dea33e36-00661ed34c-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:44.856 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-19 zones={test-zone2, test-zone1} 2024-04-16T19:36:44.856 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-20 2024-04-16T19:36:44.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:44.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:44.857 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:44.919 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:44.919 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:44.976+0000 7f5cb152a800 1 bucket sync caught up with source: 2024-04-16T19:36:44.978 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.19.14, , ] 2024-04-16T19:36:44.978 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000001.19.14, , ] 2024-04-16T19:36:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:44.976+0000 7f5cb152a800 0 bucket checkpoint complete 2024-04-16T19:36:44.998 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-20 2024-04-16T19:36:44.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:44.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:36:44.998 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:36:45.060 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:45.061 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:45.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:45.119+0000 7fca8394b800 1 bucket sync caught up with source: 2024-04-16T19:36:45.122 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.53.7, , ] 2024-04-16T19:36:45.122 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.53.7, , ] 2024-04-16T19:36:45.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:36:45.119+0000 7fca8394b800 0 bucket checkpoint complete 2024-04-16T19:36:45.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-20 zones={test-zone2, test-zone1} 2024-04-16T19:36:45.143 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.143 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.143 DEBUG:boto:Method: HEAD 2024-04-16T19:36:45.143 DEBUG:boto:Path: /yrjefh-20/ 2024-04-16T19:36:45.143 DEBUG:boto:Data: 2024-04-16T19:36:45.143 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.143 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.143 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.143 DEBUG:boto:Params: {} 2024-04-16T19:36:45.143 DEBUG:boto:Token: None 2024-04-16T19:36:45.144 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/ 2024-04-16T19:36:45.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vsC0OBL12Ryz8KXPUxk03gf9Ixg= 2024-04-16T19:36:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsC0OBL12Ryz8KXPUxk03gf9Ixg=', 'Content-Length': '0'} 2024-04-16T19:36:45.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008070c8bed758e5a-00661ed34d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.146 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.146 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.147 DEBUG:boto:Method: HEAD 2024-04-16T19:36:45.147 DEBUG:boto:Path: /yrjefh-20/ 2024-04-16T19:36:45.147 DEBUG:boto:Data: 2024-04-16T19:36:45.147 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.147 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.147 DEBUG:boto:Params: {} 2024-04-16T19:36:45.147 DEBUG:boto:Token: None 2024-04-16T19:36:45.147 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/ 2024-04-16T19:36:45.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vsC0OBL12Ryz8KXPUxk03gf9Ixg= 2024-04-16T19:36:45.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsC0OBL12Ryz8KXPUxk03gf9Ixg=', 'Content-Length': '0'} 2024-04-16T19:36:45.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060a07efd0f61d77d-00661ed34d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:36:45.150 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.150 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.150 DEBUG:boto:path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.150 DEBUG:boto:auth_path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.150 DEBUG:boto:Method: GET 2024-04-16T19:36:45.150 DEBUG:boto:Path: /yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.151 DEBUG:boto:Data: 2024-04-16T19:36:45.151 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.151 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.151 DEBUG:boto:Params: {} 2024-04-16T19:36:45.151 DEBUG:boto:Token: None 2024-04-16T19:36:45.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/?versions 2024-04-16T19:36:45.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs= 2024-04-16T19:36:45.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs=', 'Content-Length': '0'} 2024-04-16T19:36:45.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000691e678ff1ca53b1-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.154 DEBUG:boto:b'yrjefh-20999falseobjnulltrue2024-04-16T19:36:19.525ZRGW11111111111111111Normal' 2024-04-16T19:36:45.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:45.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:36:45.155 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.155 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.155 DEBUG:boto:path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.155 DEBUG:boto:auth_path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.155 DEBUG:boto:Method: GET 2024-04-16T19:36:45.155 DEBUG:boto:Path: /yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.155 DEBUG:boto:Data: 2024-04-16T19:36:45.156 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.156 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.156 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.156 DEBUG:boto:Params: {} 2024-04-16T19:36:45.156 DEBUG:boto:Token: None 2024-04-16T19:36:45.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/?versions 2024-04-16T19:36:45.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs= 2024-04-16T19:36:45.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs=', 'Content-Length': '0'} 2024-04-16T19:36:45.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d602217af3c30a8f-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.159 DEBUG:boto:b'yrjefh-20999falseobjnulltrue2024-04-16T19:36:19.525ZRGW11111111111111111Normal' 2024-04-16T19:36:45.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-16T19:36:45.159 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.159 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.159 DEBUG:boto:path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.159 DEBUG:boto:auth_path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.159 DEBUG:boto:Method: GET 2024-04-16T19:36:45.159 DEBUG:boto:Path: /yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.160 DEBUG:boto:Data: 2024-04-16T19:36:45.160 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.160 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.160 DEBUG:boto:Params: {} 2024-04-16T19:36:45.160 DEBUG:boto:Token: None 2024-04-16T19:36:45.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/?versions 2024-04-16T19:36:45.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs= 2024-04-16T19:36:45.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs=', 'Content-Length': '0'} 2024-04-16T19:36:45.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000711593c6bae09e3-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.163 DEBUG:boto:b'yrjefh-20999falseobjnulltrue2024-04-16T19:36:19.525ZRGW11111111111111111Normal' 2024-04-16T19:36:45.163 DEBUG:boto:path=/yrjefh-20/ 2024-04-16T19:36:45.163 DEBUG:boto:auth_path=/yrjefh-20/ 2024-04-16T19:36:45.163 DEBUG:boto:path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.163 DEBUG:boto:auth_path=/yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.163 DEBUG:boto:Method: GET 2024-04-16T19:36:45.164 DEBUG:boto:Path: /yrjefh-20/?versions&max-keys=999 2024-04-16T19:36:45.164 DEBUG:boto:Data: 2024-04-16T19:36:45.164 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.164 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.164 DEBUG:boto:Params: {} 2024-04-16T19:36:45.164 DEBUG:boto:Token: None 2024-04-16T19:36:45.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/?versions 2024-04-16T19:36:45.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs= 2024-04-16T19:36:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SJqgsg45AkaoYznl4uofsae45bs=', 'Content-Length': '0'} 2024-04-16T19:36:45.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1ae5811937a3398-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.210 DEBUG:boto:b'yrjefh-20999falseobjnulltrue2024-04-16T19:36:19.525ZRGW11111111111111111Normal' 2024-04-16T19:36:45.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-16T19:36:45.211 DEBUG:boto:path=/yrjefh-20/obj 2024-04-16T19:36:45.211 DEBUG:boto:auth_path=/yrjefh-20/obj 2024-04-16T19:36:45.211 DEBUG:boto:Method: HEAD 2024-04-16T19:36:45.211 DEBUG:boto:Path: /yrjefh-20/obj 2024-04-16T19:36:45.211 DEBUG:boto:Data: 2024-04-16T19:36:45.211 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.211 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.212 DEBUG:boto:Params: {} 2024-04-16T19:36:45.212 DEBUG:boto:Token: None 2024-04-16T19:36:45.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/obj 2024-04-16T19:36:45.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9FyFAEyIFRpajNGfzxyGciLl9Oc= 2024-04-16T19:36:45.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9FyFAEyIFRpajNGfzxyGciLl9Oc=', 'Content-Length': '0'} 2024-04-16T19:36:45.214 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c899e132babb4e59-00661ed34d-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.214 DEBUG:boto:path=/yrjefh-20/obj 2024-04-16T19:36:45.214 DEBUG:boto:auth_path=/yrjefh-20/obj 2024-04-16T19:36:45.214 DEBUG:boto:Method: HEAD 2024-04-16T19:36:45.215 DEBUG:boto:Path: /yrjefh-20/obj 2024-04-16T19:36:45.215 DEBUG:boto:Data: 2024-04-16T19:36:45.215 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.215 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.215 DEBUG:boto:Params: {} 2024-04-16T19:36:45.215 DEBUG:boto:Token: None 2024-04-16T19:36:45.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-20/obj 2024-04-16T19:36:45.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9FyFAEyIFRpajNGfzxyGciLl9Oc= 2024-04-16T19:36:45.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9FyFAEyIFRpajNGfzxyGciLl9Oc=', 'Content-Length': '0'} 2024-04-16T19:36:45.217 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000705bd59f78e12de-00661ed34d-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-20 zones={test-zone2, test-zone1} 2024-04-16T19:36:45.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-16T19:36:45.218 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-21 2024-04-16T19:36:45.219 DEBUG:boto:path=/yrjefh-21/ 2024-04-16T19:36:45.219 DEBUG:boto:auth_path=/yrjefh-21/ 2024-04-16T19:36:45.219 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.219 DEBUG:boto:Path: /yrjefh-21/ 2024-04-16T19:36:45.219 DEBUG:boto:Data: 2024-04-16T19:36:45.219 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.219 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.219 DEBUG:boto:Params: {} 2024-04-16T19:36:45.219 DEBUG:boto:Token: None 2024-04-16T19:36:45.220 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-21/ 2024-04-16T19:36:45.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5zy8X3rtXH98MARoz4HDyYx53Xk= 2024-04-16T19:36:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5zy8X3rtXH98MARoz4HDyYx53Xk=', 'Content-Length': '0'} 2024-04-16T19:36:45.236 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005f6600d2c5e99e71-00661ed34d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.236 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-22 2024-04-16T19:36:45.236 DEBUG:boto:path=/yrjefh-22/ 2024-04-16T19:36:45.236 DEBUG:boto:auth_path=/yrjefh-22/ 2024-04-16T19:36:45.236 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.237 DEBUG:boto:Path: /yrjefh-22/ 2024-04-16T19:36:45.237 DEBUG:boto:Data: 2024-04-16T19:36:45.237 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.237 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.237 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.237 DEBUG:boto:Params: {} 2024-04-16T19:36:45.237 DEBUG:boto:Token: None 2024-04-16T19:36:45.237 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-22/ 2024-04-16T19:36:45.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GhMk7LAcYEEQnXIj7fiiffy+uW0= 2024-04-16T19:36:45.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GhMk7LAcYEEQnXIj7fiiffy+uW0=', 'Content-Length': '0'} 2024-04-16T19:36:45.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000582bcd6a66c84152-00661ed34d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.266 DEBUG:boto:path=/yrjefh-21/ 2024-04-16T19:36:45.266 DEBUG:boto:auth_path=/yrjefh-21/ 2024-04-16T19:36:45.266 DEBUG:boto:path=/yrjefh-21/?versioning 2024-04-16T19:36:45.266 DEBUG:boto:auth_path=/yrjefh-21/?versioning 2024-04-16T19:36:45.266 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.266 DEBUG:boto:Path: /yrjefh-21/?versioning 2024-04-16T19:36:45.266 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:36:45.266 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.266 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.266 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.267 DEBUG:boto:Params: {} 2024-04-16T19:36:45.267 DEBUG:boto:Token: None 2024-04-16T19:36:45.267 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-21/?versioning 2024-04-16T19:36:45.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xHLCqlrbJ+t46PUS1cPTQ8rYRo4= 2024-04-16T19:36:45.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xHLCqlrbJ+t46PUS1cPTQ8rYRo4=', 'Content-Length': '228'} 2024-04-16T19:36:45.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083a3a8794c86f08b-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.273 DEBUG:boto:path=/yrjefh-21/ 2024-04-16T19:36:45.273 DEBUG:boto:auth_path=/yrjefh-21/ 2024-04-16T19:36:45.273 DEBUG:boto:path=/yrjefh-21/?versioning 2024-04-16T19:36:45.273 DEBUG:boto:auth_path=/yrjefh-21/?versioning 2024-04-16T19:36:45.273 DEBUG:boto:Method: GET 2024-04-16T19:36:45.273 DEBUG:boto:Path: /yrjefh-21/?versioning 2024-04-16T19:36:45.274 DEBUG:boto:Data: 2024-04-16T19:36:45.274 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.274 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.274 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.274 DEBUG:boto:Params: {} 2024-04-16T19:36:45.274 DEBUG:boto:Token: None 2024-04-16T19:36:45.274 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-21/?versioning 2024-04-16T19:36:45.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PV9JbTqvL4Wn/zK+xv998oF49EQ= 2024-04-16T19:36:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PV9JbTqvL4Wn/zK+xv998oF49EQ=', 'Content-Length': '0'} 2024-04-16T19:36:45.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec5b605b94bc7186-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.318 DEBUG:boto:EnabledDisabled 2024-04-16T19:36:45.319 DEBUG:boto:path=/yrjefh-22/ 2024-04-16T19:36:45.319 DEBUG:boto:auth_path=/yrjefh-22/ 2024-04-16T19:36:45.319 DEBUG:boto:path=/yrjefh-22/?versioning 2024-04-16T19:36:45.319 DEBUG:boto:auth_path=/yrjefh-22/?versioning 2024-04-16T19:36:45.320 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.320 DEBUG:boto:Path: /yrjefh-22/?versioning 2024-04-16T19:36:45.320 DEBUG:boto:Data: Enabled Disabled 2024-04-16T19:36:45.320 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.320 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.320 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.320 DEBUG:boto:Params: {} 2024-04-16T19:36:45.320 DEBUG:boto:Token: None 2024-04-16T19:36:45.320 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-22/?versioning 2024-04-16T19:36:45.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q2xfYABZvTv7koWMA19YCbX2+cM= 2024-04-16T19:36:45.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q2xfYABZvTv7koWMA19YCbX2+cM=', 'Content-Length': '228'} 2024-04-16T19:36:45.329 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e2990a91d0afe7f-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.329 DEBUG:boto:path=/yrjefh-22/ 2024-04-16T19:36:45.330 DEBUG:boto:auth_path=/yrjefh-22/ 2024-04-16T19:36:45.330 DEBUG:boto:path=/yrjefh-22/?versioning 2024-04-16T19:36:45.330 DEBUG:boto:auth_path=/yrjefh-22/?versioning 2024-04-16T19:36:45.330 DEBUG:boto:Method: GET 2024-04-16T19:36:45.330 DEBUG:boto:Path: /yrjefh-22/?versioning 2024-04-16T19:36:45.330 DEBUG:boto:Data: 2024-04-16T19:36:45.330 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.330 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.330 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.331 DEBUG:boto:Params: {} 2024-04-16T19:36:45.331 DEBUG:boto:Token: None 2024-04-16T19:36:45.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-22/?versioning 2024-04-16T19:36:45.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WpQH8QHBEmXKlWTyhr2f5WS+jD0= 2024-04-16T19:36:45.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WpQH8QHBEmXKlWTyhr2f5WS+jD0=', 'Content-Length': '0'} 2024-04-16T19:36:45.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a45b65edc796b4ed-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '192'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.332 DEBUG:boto:EnabledDisabled 2024-04-16T19:36:45.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-16T19:36:45.333 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-23 2024-04-16T19:36:45.333 DEBUG:boto:path=/yrjefh-23/ 2024-04-16T19:36:45.333 DEBUG:boto:auth_path=/yrjefh-23/ 2024-04-16T19:36:45.333 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.333 DEBUG:boto:Path: /yrjefh-23/ 2024-04-16T19:36:45.334 DEBUG:boto:Data: 2024-04-16T19:36:45.334 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.334 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.334 DEBUG:boto:Params: {} 2024-04-16T19:36:45.334 DEBUG:boto:Token: None 2024-04-16T19:36:45.334 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-23/ 2024-04-16T19:36:45.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eqebk3YNq0MzAGxEAQBrdjZJGU0= 2024-04-16T19:36:45.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eqebk3YNq0MzAGxEAQBrdjZJGU0=', 'Content-Length': '0'} 2024-04-16T19:36:45.349 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1976e7ed354b7a8-00661ed34d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.349 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-24 2024-04-16T19:36:45.349 DEBUG:boto:path=/yrjefh-24/ 2024-04-16T19:36:45.349 DEBUG:boto:auth_path=/yrjefh-24/ 2024-04-16T19:36:45.349 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.349 DEBUG:boto:Path: /yrjefh-24/ 2024-04-16T19:36:45.349 DEBUG:boto:Data: 2024-04-16T19:36:45.349 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.350 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.350 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.350 DEBUG:boto:Params: {} 2024-04-16T19:36:45.350 DEBUG:boto:Token: None 2024-04-16T19:36:45.350 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-24/ 2024-04-16T19:36:45.350 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6l4kdcaGPzTrI/RBglNzFCQMNtU= 2024-04-16T19:36:45.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6l4kdcaGPzTrI/RBglNzFCQMNtU=', 'Content-Length': '0'} 2024-04-16T19:36:45.416 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d8ae3ddbcc0e889-00661ed34d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.416 DEBUG:boto:path=/yrjefh-23/ 2024-04-16T19:36:45.416 DEBUG:boto:auth_path=/yrjefh-23/ 2024-04-16T19:36:45.416 DEBUG:boto:path=/yrjefh-23/?acl 2024-04-16T19:36:45.416 DEBUG:boto:auth_path=/yrjefh-23/?acl 2024-04-16T19:36:45.416 DEBUG:boto:Method: GET 2024-04-16T19:36:45.416 DEBUG:boto:Path: /yrjefh-23/?acl 2024-04-16T19:36:45.416 DEBUG:boto:Data: 2024-04-16T19:36:45.416 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.417 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.417 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.417 DEBUG:boto:Params: {} 2024-04-16T19:36:45.417 DEBUG:boto:Token: None 2024-04-16T19:36:45.417 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-23/?acl 2024-04-16T19:36:45.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNR3uCgjEn7TKI3kLEG+v6RKJcY= 2024-04-16T19:36:45.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNR3uCgjEn7TKI3kLEG+v6RKJcY=', 'Content-Length': '0'} 2024-04-16T19:36:45.419 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ccc2f172f1fb704-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '391'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.419 DEBUG:boto:path=/yrjefh-23/ 2024-04-16T19:36:45.419 DEBUG:boto:auth_path=/yrjefh-23/ 2024-04-16T19:36:45.419 DEBUG:boto:path=/yrjefh-23/?acl 2024-04-16T19:36:45.419 DEBUG:boto:auth_path=/yrjefh-23/?acl 2024-04-16T19:36:45.420 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.420 DEBUG:boto:Path: /yrjefh-23/?acl 2024-04-16T19:36:45.420 DEBUG:boto:Data: 2024-04-16T19:36:45.420 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:36:45.420 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.420 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.420 DEBUG:boto:Params: {} 2024-04-16T19:36:45.420 DEBUG:boto:Token: None 2024-04-16T19:36:45.420 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT x-amz-acl:public-read /yrjefh-23/?acl 2024-04-16T19:36:45.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4MusLuf2qdoBD77Ki8D7lTcliSg= 2024-04-16T19:36:45.421 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4MusLuf2qdoBD77Ki8D7lTcliSg=', 'Content-Length': '0'} 2024-04-16T19:36:45.426 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ea1dcb10eb9b56a-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.426 DEBUG:boto:path=/yrjefh-23/ 2024-04-16T19:36:45.426 DEBUG:boto:auth_path=/yrjefh-23/ 2024-04-16T19:36:45.426 DEBUG:boto:path=/yrjefh-23/?acl 2024-04-16T19:36:45.426 DEBUG:boto:auth_path=/yrjefh-23/?acl 2024-04-16T19:36:45.426 DEBUG:boto:Method: GET 2024-04-16T19:36:45.426 DEBUG:boto:Path: /yrjefh-23/?acl 2024-04-16T19:36:45.427 DEBUG:boto:Data: 2024-04-16T19:36:45.427 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.427 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.427 DEBUG:boto:Params: {} 2024-04-16T19:36:45.427 DEBUG:boto:Token: None 2024-04-16T19:36:45.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-23/?acl 2024-04-16T19:36:45.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNR3uCgjEn7TKI3kLEG+v6RKJcY= 2024-04-16T19:36:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNR3uCgjEn7TKI3kLEG+v6RKJcY=', 'Content-Length': '0'} 2024-04-16T19:36:45.428 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d67021095d766da-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '583'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.470 DEBUG:boto:path=/yrjefh-24/ 2024-04-16T19:36:45.471 DEBUG:boto:auth_path=/yrjefh-24/ 2024-04-16T19:36:45.471 DEBUG:boto:path=/yrjefh-24/?acl 2024-04-16T19:36:45.471 DEBUG:boto:auth_path=/yrjefh-24/?acl 2024-04-16T19:36:45.471 DEBUG:boto:Method: GET 2024-04-16T19:36:45.471 DEBUG:boto:Path: /yrjefh-24/?acl 2024-04-16T19:36:45.471 DEBUG:boto:Data: 2024-04-16T19:36:45.471 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.471 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.472 DEBUG:boto:Params: {} 2024-04-16T19:36:45.472 DEBUG:boto:Token: None 2024-04-16T19:36:45.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-24/?acl 2024-04-16T19:36:45.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WSPiSi28o4GsgphGWf8cnDmdwU= 2024-04-16T19:36:45.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WSPiSi28o4GsgphGWf8cnDmdwU=', 'Content-Length': '0'} 2024-04-16T19:36:45.473 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe25e6e4dc1fc405-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '391'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.474 DEBUG:boto:path=/yrjefh-24/ 2024-04-16T19:36:45.474 DEBUG:boto:auth_path=/yrjefh-24/ 2024-04-16T19:36:45.474 DEBUG:boto:path=/yrjefh-24/?acl 2024-04-16T19:36:45.474 DEBUG:boto:auth_path=/yrjefh-24/?acl 2024-04-16T19:36:45.474 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.474 DEBUG:boto:Path: /yrjefh-24/?acl 2024-04-16T19:36:45.474 DEBUG:boto:Data: 2024-04-16T19:36:45.474 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-16T19:36:45.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.474 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.475 DEBUG:boto:Params: {} 2024-04-16T19:36:45.475 DEBUG:boto:Token: None 2024-04-16T19:36:45.475 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT x-amz-acl:public-read /yrjefh-24/?acl 2024-04-16T19:36:45.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1n/wsRrI4SyAC90hxDqAT6ECz1k= 2024-04-16T19:36:45.475 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1n/wsRrI4SyAC90hxDqAT6ECz1k=', 'Content-Length': '0'} 2024-04-16T19:36:45.484 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000582a732e353777f9-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.484 DEBUG:boto:path=/yrjefh-24/ 2024-04-16T19:36:45.484 DEBUG:boto:auth_path=/yrjefh-24/ 2024-04-16T19:36:45.484 DEBUG:boto:path=/yrjefh-24/?acl 2024-04-16T19:36:45.484 DEBUG:boto:auth_path=/yrjefh-24/?acl 2024-04-16T19:36:45.484 DEBUG:boto:Method: GET 2024-04-16T19:36:45.484 DEBUG:boto:Path: /yrjefh-24/?acl 2024-04-16T19:36:45.484 DEBUG:boto:Data: 2024-04-16T19:36:45.484 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.485 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.485 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.485 DEBUG:boto:Params: {} 2024-04-16T19:36:45.485 DEBUG:boto:Token: None 2024-04-16T19:36:45.485 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-24/?acl 2024-04-16T19:36:45.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WSPiSi28o4GsgphGWf8cnDmdwU= 2024-04-16T19:36:45.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WSPiSi28o4GsgphGWf8cnDmdwU=', 'Content-Length': '0'} 2024-04-16T19:36:45.486 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2512347b08520f9-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '583'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-16T19:36:45.531 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-25 2024-04-16T19:36:45.531 DEBUG:boto:path=/yrjefh-25/ 2024-04-16T19:36:45.531 DEBUG:boto:auth_path=/yrjefh-25/ 2024-04-16T19:36:45.532 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.532 DEBUG:boto:Path: /yrjefh-25/ 2024-04-16T19:36:45.532 DEBUG:boto:Data: 2024-04-16T19:36:45.532 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.532 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.532 DEBUG:boto:Params: {} 2024-04-16T19:36:45.532 DEBUG:boto:Token: None 2024-04-16T19:36:45.532 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-25/ 2024-04-16T19:36:45.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JGhxx/iHBBzIHFV7OqHApAz04Fo= 2024-04-16T19:36:45.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JGhxx/iHBBzIHFV7OqHApAz04Fo=', 'Content-Length': '0'} 2024-04-16T19:36:45.547 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d1f0679ab496152-00661ed34d-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.548 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-26 2024-04-16T19:36:45.548 DEBUG:boto:path=/yrjefh-26/ 2024-04-16T19:36:45.548 DEBUG:boto:auth_path=/yrjefh-26/ 2024-04-16T19:36:45.548 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.548 DEBUG:boto:Path: /yrjefh-26/ 2024-04-16T19:36:45.548 DEBUG:boto:Data: 2024-04-16T19:36:45.548 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.548 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.548 DEBUG:boto:Params: {} 2024-04-16T19:36:45.549 DEBUG:boto:Token: None 2024-04-16T19:36:45.549 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-26/ 2024-04-16T19:36:45.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IFiNnwos2hYHuvgRRCHBxwjoOc4= 2024-04-16T19:36:45.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IFiNnwos2hYHuvgRRCHBxwjoOc4=', 'Content-Length': '0'} 2024-04-16T19:36:45.575 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a340c796d6de69a5-00661ed34d-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.575 DEBUG:boto:path=/yrjefh-25/ 2024-04-16T19:36:45.575 DEBUG:boto:auth_path=/yrjefh-25/ 2024-04-16T19:36:45.576 DEBUG:boto:path=/yrjefh-25/?cors 2024-04-16T19:36:45.576 DEBUG:boto:auth_path=/yrjefh-25/?cors 2024-04-16T19:36:45.576 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.576 DEBUG:boto:Path: /yrjefh-25/?cors 2024-04-16T19:36:45.576 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-16T19:36:45.576 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-16T19:36:45.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.576 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.576 DEBUG:boto:Params: {} 2024-04-16T19:36:45.576 DEBUG:boto:Token: None 2024-04-16T19:36:45.577 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-25/?cors 2024-04-16T19:36:45.577 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lsRzZcxYconVQLdNii0UYAyV8Fs= 2024-04-16T19:36:45.577 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lsRzZcxYconVQLdNii0UYAyV8Fs=', 'Content-Length': '218'} 2024-04-16T19:36:45.582 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b002f844660a246-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.582 DEBUG:boto:path=/yrjefh-25/ 2024-04-16T19:36:45.582 DEBUG:boto:auth_path=/yrjefh-25/ 2024-04-16T19:36:45.582 DEBUG:boto:path=/yrjefh-25/?cors 2024-04-16T19:36:45.583 DEBUG:boto:auth_path=/yrjefh-25/?cors 2024-04-16T19:36:45.583 DEBUG:boto:Method: GET 2024-04-16T19:36:45.583 DEBUG:boto:Path: /yrjefh-25/?cors 2024-04-16T19:36:45.583 DEBUG:boto:Data: 2024-04-16T19:36:45.583 DEBUG:boto:Headers: {} 2024-04-16T19:36:45.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:45.583 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.583 DEBUG:boto:Params: {} 2024-04-16T19:36:45.583 DEBUG:boto:Token: None 2024-04-16T19:36:45.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-25/?cors 2024-04-16T19:36:45.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QEHTuV8P8/IgIlmR+r/Jld1ZUIU= 2024-04-16T19:36:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QEHTuV8P8/IgIlmR+r/Jld1ZUIU=', 'Content-Length': '0'} 2024-04-16T19:36:45.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be8e2b91d5671eda-00661ed34d-4361-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Tue, 16 Apr 2024 19:36:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:45.585 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-16T19:36:45.585 DEBUG:boto:path=/yrjefh-26/ 2024-04-16T19:36:45.586 DEBUG:boto:auth_path=/yrjefh-26/ 2024-04-16T19:36:45.586 DEBUG:boto:path=/yrjefh-26/?cors 2024-04-16T19:36:45.586 DEBUG:boto:auth_path=/yrjefh-26/?cors 2024-04-16T19:36:45.586 DEBUG:boto:Method: PUT 2024-04-16T19:36:45.586 DEBUG:boto:Path: /yrjefh-26/?cors 2024-04-16T19:36:45.586 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-16T19:36:45.586 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-16T19:36:45.586 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:45.586 DEBUG:boto:Port: 8000 2024-04-16T19:36:45.586 DEBUG:boto:Params: {} 2024-04-16T19:36:45.587 DEBUG:boto:Token: None 2024-04-16T19:36:45.587 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 16 Apr 2024 19:36:45 GMT /yrjefh-26/?cors 2024-04-16T19:36:45.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L7gy/vgkWATMjQPy8fG0RE4LKdE= 2024-04-16T19:36:45.587 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L7gy/vgkWATMjQPy8fG0RE4LKdE=', 'Content-Length': '218'} 2024-04-16T19:36:46.603 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bf9f90ae0b3f2b5-00661ed34d-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:46.604 DEBUG:boto:path=/yrjefh-26/ 2024-04-16T19:36:46.604 DEBUG:boto:auth_path=/yrjefh-26/ 2024-04-16T19:36:46.604 DEBUG:boto:path=/yrjefh-26/?cors 2024-04-16T19:36:46.604 DEBUG:boto:auth_path=/yrjefh-26/?cors 2024-04-16T19:36:46.604 DEBUG:boto:Method: GET 2024-04-16T19:36:46.604 DEBUG:boto:Path: /yrjefh-26/?cors 2024-04-16T19:36:46.604 DEBUG:boto:Data: 2024-04-16T19:36:46.605 DEBUG:boto:Headers: {} 2024-04-16T19:36:46.605 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:46.605 DEBUG:boto:Port: 8000 2024-04-16T19:36:46.605 DEBUG:boto:Params: {} 2024-04-16T19:36:46.605 DEBUG:boto:Token: None 2024-04-16T19:36:46.605 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:36:46 GMT /yrjefh-26/?cors 2024-04-16T19:36:46.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eY2waeWW99kffqV/0LzZSaBEY3Q= 2024-04-16T19:36:46.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eY2waeWW99kffqV/0LzZSaBEY3Q=', 'Content-Length': '0'} 2024-04-16T19:36:46.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af0ec6f02a151220-00661ed34e-4339-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '266'), ('Date', 'Tue, 16 Apr 2024 19:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:46.650 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-16T19:36:46.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-16T19:36:46.652 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-27 2024-04-16T19:36:46.652 DEBUG:boto:path=/yrjefh-27/ 2024-04-16T19:36:46.652 DEBUG:boto:auth_path=/yrjefh-27/ 2024-04-16T19:36:46.652 DEBUG:boto:Method: PUT 2024-04-16T19:36:46.653 DEBUG:boto:Path: /yrjefh-27/ 2024-04-16T19:36:46.653 DEBUG:boto:Data: 2024-04-16T19:36:46.653 DEBUG:boto:Headers: {} 2024-04-16T19:36:46.653 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:46.653 DEBUG:boto:Port: 8000 2024-04-16T19:36:46.653 DEBUG:boto:Params: {} 2024-04-16T19:36:46.653 DEBUG:boto:Token: None 2024-04-16T19:36:46.654 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:46 GMT /yrjefh-27/ 2024-04-16T19:36:46.654 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hU1pct3kFUVkeVrjE2NjqCxId1A= 2024-04-16T19:36:46.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hU1pct3kFUVkeVrjE2NjqCxId1A=', 'Content-Length': '0'} 2024-04-16T19:36:46.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009eafee0de5ae685c-00661ed34e-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:46.671 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-28 2024-04-16T19:36:46.671 DEBUG:boto:path=/yrjefh-28/ 2024-04-16T19:36:46.671 DEBUG:boto:auth_path=/yrjefh-28/ 2024-04-16T19:36:46.671 DEBUG:boto:Method: PUT 2024-04-16T19:36:46.672 DEBUG:boto:Path: /yrjefh-28/ 2024-04-16T19:36:46.672 DEBUG:boto:Data: 2024-04-16T19:36:46.672 DEBUG:boto:Headers: {} 2024-04-16T19:36:46.672 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:46.672 DEBUG:boto:Port: 8000 2024-04-16T19:36:46.672 DEBUG:boto:Params: {} 2024-04-16T19:36:46.672 DEBUG:boto:Token: None 2024-04-16T19:36:46.672 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:46 GMT /yrjefh-28/ 2024-04-16T19:36:46.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w22Mii/aEWVQ5CaCaEdlYqWXHpw= 2024-04-16T19:36:46.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w22Mii/aEWVQ5CaCaEdlYqWXHpw=', 'Content-Length': '0'} 2024-04-16T19:36:46.701 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d526aaa16681624f-00661ed34e-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:46.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:46.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:46.702 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:46.755 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:46.755 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:46.808 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:36:46.830 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296206.689648_251.1","last_update":"2024-04-16T19:36:46.689648Z"}] 2024-04-16T19:36:46.831 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296206.689648_251.1'} 2024-04-16T19:36:46.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:36:46.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:46.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:46.831 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:46.886 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:46.886 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:46.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:46.938+0000 7f3c8db2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:46.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:46.939+0000 7f3c8db2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:46.961 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296205.581095_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:45.581095Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:46Z"} 2024-04-16T19:36:46.962 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296205.581095_241.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:45.581095Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:46Z"} 2024-04-16T19:36:46.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296205.581095_241.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:45.581095Z', 'realm_epoch': 2}}] 2024-04-16T19:36:46.962 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296206.689648_251.1'} 2024-04-16T19:36:46.962 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296205.581095_241.1'} 2024-04-16T19:36:46.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296206.689648_251.1 target=1_1713296205.581095_241.1 2024-04-16T19:36:51.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:51.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:51.964 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:52.019 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:52.019 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:52.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:52.072+0000 7f3101f33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:52.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:52.073+0000 7f3101f33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:52.096 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296206.689648_251.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:46.689648Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:52Z"} 2024-04-16T19:36:52.096 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296206.689648_251.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:46.689648Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:52Z"} 2024-04-16T19:36:52.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296206.689648_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:46.689648Z', 'realm_epoch': 2}}] 2024-04-16T19:36:52.096 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296206.689648_251.1'} 2024-04-16T19:36:52.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296206.689648_251.1'} 2024-04-16T19:36:52.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:36:52.097 DEBUG:boto:path=/yrjefh-27/ 2024-04-16T19:36:52.097 DEBUG:boto:auth_path=/yrjefh-27/ 2024-04-16T19:36:52.097 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.097 DEBUG:boto:Path: /yrjefh-27/ 2024-04-16T19:36:52.097 DEBUG:boto:Data: 2024-04-16T19:36:52.097 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.097 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.097 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.097 DEBUG:boto:Params: {} 2024-04-16T19:36:52.098 DEBUG:boto:Token: None 2024-04-16T19:36:52.098 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-27/ 2024-04-16T19:36:52.098 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UxHaCjYtV/zCKxl6nvS80b1z28s= 2024-04-16T19:36:52.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UxHaCjYtV/zCKxl6nvS80b1z28s=', 'Content-Length': '0'} 2024-04-16T19:36:52.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba30694597e00c68-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.100 DEBUG:boto:path=/yrjefh-27/foo 2024-04-16T19:36:52.101 DEBUG:boto:auth_path=/yrjefh-27/foo 2024-04-16T19:36:52.101 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.101 DEBUG:boto:Path: /yrjefh-27/foo 2024-04-16T19:36:52.101 DEBUG:boto:Data: 2024-04-16T19:36:52.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:36:52.101 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.101 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.101 DEBUG:boto:Params: {} 2024-04-16T19:36:52.101 DEBUG:boto:Token: None 2024-04-16T19:36:52.102 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-27/foo 2024-04-16T19:36:52.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mG+YzBMi6EyP5sHLmS05QKH9uj8= 2024-04-16T19:36:52.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mG+YzBMi6EyP5sHLmS05QKH9uj8='} 2024-04-16T19:36:52.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000133bf82ebf561ff7-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.110 DEBUG:boto:path=/yrjefh-27/ 2024-04-16T19:36:52.110 DEBUG:boto:auth_path=/yrjefh-27/ 2024-04-16T19:36:52.110 DEBUG:boto:Method: DELETE 2024-04-16T19:36:52.110 DEBUG:boto:Path: /yrjefh-27/ 2024-04-16T19:36:52.110 DEBUG:boto:Data: 2024-04-16T19:36:52.110 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.110 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.110 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.110 DEBUG:boto:Params: {} 2024-04-16T19:36:52.111 DEBUG:boto:Token: None 2024-04-16T19:36:52.111 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-27/ 2024-04-16T19:36:52.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yOmX6RXZX66oUxn2+HVvD31bibU= 2024-04-16T19:36:52.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yOmX6RXZX66oUxn2+HVvD31bibU=', 'Content-Length': '0'} 2024-04-16T19:36:52.120 DEBUG:boto:Response headers: [('Content-Length', '253'), ('x-amz-request-id', 'tx000007fb7863354baf0a2-00661ed354-4361-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.163 DEBUG:boto:path=/yrjefh-28/ 2024-04-16T19:36:52.163 DEBUG:boto:auth_path=/yrjefh-28/ 2024-04-16T19:36:52.163 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.163 DEBUG:boto:Path: /yrjefh-28/ 2024-04-16T19:36:52.163 DEBUG:boto:Data: 2024-04-16T19:36:52.163 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.163 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.163 DEBUG:boto:Params: {} 2024-04-16T19:36:52.163 DEBUG:boto:Token: None 2024-04-16T19:36:52.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-28/ 2024-04-16T19:36:52.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ueggNEw2D4wc7u9kIKNE69EsA8Q= 2024-04-16T19:36:52.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ueggNEw2D4wc7u9kIKNE69EsA8Q=', 'Content-Length': '0'} 2024-04-16T19:36:52.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa9739bab243db94-00661ed354-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.166 DEBUG:boto:path=/yrjefh-28/foo 2024-04-16T19:36:52.166 DEBUG:boto:auth_path=/yrjefh-28/foo 2024-04-16T19:36:52.166 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.166 DEBUG:boto:Path: /yrjefh-28/foo 2024-04-16T19:36:52.166 DEBUG:boto:Data: 2024-04-16T19:36:52.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:36:52.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.167 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.167 DEBUG:boto:Params: {} 2024-04-16T19:36:52.167 DEBUG:boto:Token: None 2024-04-16T19:36:52.167 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-28/foo 2024-04-16T19:36:52.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tk/bEiPouGRy692GVKBBQMVsA7g= 2024-04-16T19:36:52.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tk/bEiPouGRy692GVKBBQMVsA7g='} 2024-04-16T19:36:52.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000015f04b684e1034f-00661ed354-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.174 DEBUG:boto:path=/yrjefh-28/ 2024-04-16T19:36:52.174 DEBUG:boto:auth_path=/yrjefh-28/ 2024-04-16T19:36:52.174 DEBUG:boto:Method: DELETE 2024-04-16T19:36:52.174 DEBUG:boto:Path: /yrjefh-28/ 2024-04-16T19:36:52.174 DEBUG:boto:Data: 2024-04-16T19:36:52.174 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.175 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.175 DEBUG:boto:Params: {} 2024-04-16T19:36:52.175 DEBUG:boto:Token: None 2024-04-16T19:36:52.175 DEBUG:boto:StringToSign: DELETE Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-28/ 2024-04-16T19:36:52.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7://1AWPq+nz/3f6wNszWi+XXoSpU= 2024-04-16T19:36:52.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7://1AWPq+nz/3f6wNszWi+XXoSpU=', 'Content-Length': '0'} 2024-04-16T19:36:52.182 DEBUG:boto:Response headers: [('Content-Length', '253'), ('x-amz-request-id', 'tx0000002c7f61120b8186a-00661ed354-4339-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.183 DEBUG:boto:path=/yrjefh-27/ 2024-04-16T19:36:52.183 DEBUG:boto:auth_path=/yrjefh-27/ 2024-04-16T19:36:52.183 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.183 DEBUG:boto:Path: /yrjefh-27/ 2024-04-16T19:36:52.183 DEBUG:boto:Data: 2024-04-16T19:36:52.183 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.183 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.183 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.183 DEBUG:boto:Params: {} 2024-04-16T19:36:52.184 DEBUG:boto:Token: None 2024-04-16T19:36:52.184 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-27/ 2024-04-16T19:36:52.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UxHaCjYtV/zCKxl6nvS80b1z28s= 2024-04-16T19:36:52.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UxHaCjYtV/zCKxl6nvS80b1z28s=', 'Content-Length': '0'} 2024-04-16T19:36:52.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af2ed91d9ca79be8-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.186 DEBUG:boto:path=/yrjefh-28/ 2024-04-16T19:36:52.186 DEBUG:boto:auth_path=/yrjefh-28/ 2024-04-16T19:36:52.186 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.187 DEBUG:boto:Path: /yrjefh-28/ 2024-04-16T19:36:52.187 DEBUG:boto:Data: 2024-04-16T19:36:52.187 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.187 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.187 DEBUG:boto:Params: {} 2024-04-16T19:36:52.187 DEBUG:boto:Token: None 2024-04-16T19:36:52.187 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-28/ 2024-04-16T19:36:52.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ueggNEw2D4wc7u9kIKNE69EsA8Q= 2024-04-16T19:36:52.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ueggNEw2D4wc7u9kIKNE69EsA8Q=', 'Content-Length': '0'} 2024-04-16T19:36:52.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb44e80664a932a8-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2024-04-16T19:36:52.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2024-04-16T19:36:52.191 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-29 2024-04-16T19:36:52.191 DEBUG:boto:path=/yrjefh-29/ 2024-04-16T19:36:52.191 DEBUG:boto:auth_path=/yrjefh-29/ 2024-04-16T19:36:52.191 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.191 DEBUG:boto:Path: /yrjefh-29/ 2024-04-16T19:36:52.191 DEBUG:boto:Data: 2024-04-16T19:36:52.191 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.191 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.191 DEBUG:boto:Params: {} 2024-04-16T19:36:52.191 DEBUG:boto:Token: None 2024-04-16T19:36:52.192 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-29/ 2024-04-16T19:36:52.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N3d7lCcw6VjzEELwbvD606oSIdM= 2024-04-16T19:36:52.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N3d7lCcw6VjzEELwbvD606oSIdM=', 'Content-Length': '0'} 2024-04-16T19:36:52.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e04ca9ca11148d3-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.209 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-30 2024-04-16T19:36:52.209 DEBUG:boto:path=/yrjefh-30/ 2024-04-16T19:36:52.210 DEBUG:boto:auth_path=/yrjefh-30/ 2024-04-16T19:36:52.210 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.210 DEBUG:boto:Path: /yrjefh-30/ 2024-04-16T19:36:52.210 DEBUG:boto:Data: 2024-04-16T19:36:52.210 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.210 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.210 DEBUG:boto:Params: {} 2024-04-16T19:36:52.210 DEBUG:boto:Token: None 2024-04-16T19:36:52.210 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-30/ 2024-04-16T19:36:52.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/20GnB+e8/kF8uY8fU/seqhGoLw= 2024-04-16T19:36:52.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/20GnB+e8/kF8uY8fU/seqhGoLw=', 'Content-Length': '0'} 2024-04-16T19:36:52.237 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000612707d70e4cf50b-00661ed354-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.237 DEBUG:boto:path=/yrjefh-29/ 2024-04-16T19:36:52.237 DEBUG:boto:auth_path=/yrjefh-29/ 2024-04-16T19:36:52.238 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.238 DEBUG:boto:Path: /yrjefh-29/ 2024-04-16T19:36:52.238 DEBUG:boto:Data: 2024-04-16T19:36:52.238 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.238 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.238 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.238 DEBUG:boto:Params: {} 2024-04-16T19:36:52.238 DEBUG:boto:Token: None 2024-04-16T19:36:52.238 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-29/ 2024-04-16T19:36:52.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AYk9QrgKur8U1fIDkr4N0p8lx+Q= 2024-04-16T19:36:52.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AYk9QrgKur8U1fIDkr4N0p8lx+Q=', 'Content-Length': '0'} 2024-04-16T19:36:52.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027752284d8867391-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.241 DEBUG:boto:path=/yrjefh-29/key 2024-04-16T19:36:52.241 DEBUG:boto:auth_path=/yrjefh-29/key 2024-04-16T19:36:52.241 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.241 DEBUG:boto:Path: /yrjefh-29/key 2024-04-16T19:36:52.241 DEBUG:boto:Data: 2024-04-16T19:36:52.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-16T19:36:52.242 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:36:52.242 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.242 DEBUG:boto:Params: {} 2024-04-16T19:36:52.242 DEBUG:boto:Token: None 2024-04-16T19:36:52.242 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-29/key 2024-04-16T19:36:52.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4tN/kD9bddoXHhCLZ5JsrvNAHHc= 2024-04-16T19:36:52.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4tN/kD9bddoXHhCLZ5JsrvNAHHc='} 2024-04-16T19:36:52.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b17e2186d9b3834-00661ed354-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.249 DEBUG:boto:path=/yrjefh-30/ 2024-04-16T19:36:52.249 DEBUG:boto:auth_path=/yrjefh-30/ 2024-04-16T19:36:52.249 DEBUG:boto:Method: HEAD 2024-04-16T19:36:52.249 DEBUG:boto:Path: /yrjefh-30/ 2024-04-16T19:36:52.249 DEBUG:boto:Data: 2024-04-16T19:36:52.249 DEBUG:boto:Headers: {} 2024-04-16T19:36:52.249 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.249 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.249 DEBUG:boto:Params: {} 2024-04-16T19:36:52.250 DEBUG:boto:Token: None 2024-04-16T19:36:52.250 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-30/ 2024-04-16T19:36:52.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mQSP9CcHMK7JUHavFx4/MBA8DcU= 2024-04-16T19:36:52.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mQSP9CcHMK7JUHavFx4/MBA8DcU=', 'Content-Length': '0'} 2024-04-16T19:36:52.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c36d050d4725b10-00661ed354-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.252 DEBUG:boto:path=/yrjefh-30/key 2024-04-16T19:36:52.252 DEBUG:boto:auth_path=/yrjefh-30/key 2024-04-16T19:36:52.252 DEBUG:boto:Method: PUT 2024-04-16T19:36:52.252 DEBUG:boto:Path: /yrjefh-30/key 2024-04-16T19:36:52.252 DEBUG:boto:Data: 2024-04-16T19:36:52.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-16T19:36:52.253 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:36:52.253 DEBUG:boto:Port: 8000 2024-04-16T19:36:52.253 DEBUG:boto:Params: {} 2024-04-16T19:36:52.253 DEBUG:boto:Token: None 2024-04-16T19:36:52.253 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 16 Apr 2024 19:36:52 GMT /yrjefh-30/key 2024-04-16T19:36:52.253 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GPE4vF2rC0vNYbvB7XIuyR2YuMc= 2024-04-16T19:36:52.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:36:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GPE4vF2rC0vNYbvB7XIuyR2YuMc='} 2024-04-16T19:36:52.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b041390f53102ef6-00661ed354-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:36:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:36:52.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:52.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:52.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:52.348 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:52.348 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:52.400 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:36:52.422 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296212.226200_259.1","last_update":"2024-04-16T19:36:52.226200Z"}] 2024-04-16T19:36:52.422 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:36:52.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:36:52.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:52.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:52.422 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:52.470 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:52.470 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:52.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:52.516+0000 7f5d18833800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:52.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:52.517+0000 7f5d18833800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:52.539 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.197362_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.197362Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:52Z"} 2024-04-16T19:36:52.539 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.197362_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.197362Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:52Z"} 2024-04-16T19:36:52.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.197362_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.197362Z', 'realm_epoch': 2}}] 2024-04-16T19:36:52.539 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:36:52.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.197362_252.1'} 2024-04-16T19:36:52.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296212.226200_259.1 target=1_1713296212.197362_252.1 2024-04-16T19:36:57.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.541 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:57.596 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:57.596 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:57.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:57.648+0000 7f0fc9a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:57.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:36:57.649+0000 7f0fc9a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:36:57.673 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:57Z"} 2024-04-16T19:36:57.673 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:36:57Z"} 2024-04-16T19:36:57.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:36:57.673 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:36:57.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:36:57.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:36:57.674 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T19:36:57.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.674 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:57.812 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-04-16T19:36:54.423318Z"},{"marker":"00000000000000000000:00000000000000002623","last_update":"2024-04-16T19:32:08.938133Z"},{"marker":"00000000000000000000:00000000000000005020","last_update":"2024-04-16T19:36:55.370182Z"},{"marker":"00000000000000000000:00000000000000004458","last_update":"2024-04-16T19:33:36.946564Z"}] 2024-04-16T19:36:57.813 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:36:57.813 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:36:57.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.813 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:57.864 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:57.864 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:57.934 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004458","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.946564Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:36:57Z"} 2024-04-16T19:36:57.934 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004458","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:36.946564Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:36:57Z"} 2024-04-16T19:36:57.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:54.423318Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.370182Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004458', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:36.946564Z'}}] 2024-04-16T19:36:57.935 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:36:57.935 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004458'} 2024-04-16T19:36:57.935 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:36:57.935 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T19:36:57.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:57.936 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:57.982 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:57.982 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:58.057 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-04-16T19:36:55.481773Z"},{"marker":"00000000000000000000:00000000000000002905","last_update":"2024-04-16T19:33:06.384483Z"},{"marker":"00000000000000000000:00000000000000004738","last_update":"2024-04-16T19:36:56.708240Z"},{"marker":"00000000000000000000:00000000000000004035","last_update":"2024-04-16T19:33:21.126136Z"}] 2024-04-16T19:36:58.057 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000004738', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:36:58.058 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:36:58.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:58.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:36:58.058 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:36:58.111 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:36:58.111 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:36:58.186 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003608","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.173093Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004597","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.257713Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:36:58Z"} 2024-04-16T19:36:58.186 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003608","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.173093Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004597","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.257713Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:36:58Z"} 2024-04-16T19:36:58.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003608', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.173093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:06.384483Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.257713Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004035', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:33:21.126136Z'}}] 2024-04-16T19:36:58.187 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000004738', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:36:58.187 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003608', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:36:58.187 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000003608, shard=2 master=00000000000000000000:00000000000000004738 target=00000000000000000000:00000000000000004597 2024-04-16T19:37:03.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.189 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:03.247 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.247 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.325 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.481773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004738","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:56.708240Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:37:03Z"} 2024-04-16T19:37:03.325 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.481773Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002905","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:33:06.384483Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004738","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:56.708240Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004035","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:33:21.126136Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:37:03Z"} 2024-04-16T19:37:03.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.481773Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:33:06.384483Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004738', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:56.708240Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004035', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:33:21.126136Z'}}] 2024-04-16T19:37:03.325 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000004738', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:37:03.326 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002905', 2: '00000000000000000000:00000000000000004738', 3: '00000000000000000000:00000000000000004035'} 2024-04-16T19:37:03.326 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:37:03.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.326 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.448 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-04-16T19:36:54.423318Z"},{"marker":"00000000000000000000:00000000000000002623","last_update":"2024-04-16T19:32:08.938133Z"},{"marker":"00000000000000000000:00000000000000005020","last_update":"2024-04-16T19:36:55.370182Z"},{"marker":"00000000000000000000:00000000000000004458","last_update":"2024-04-16T19:33:36.946564Z"}] 2024-04-16T19:37:03.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:03.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:03.449 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:37:03.492 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.492 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.573 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:03.660 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.660 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.730 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:37:03.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.850 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:37:03.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.851 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:03.969 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:37:03.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:03.970 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.087 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:37:04.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.087 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.222 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000003750","last_update":"2024-04-16T19:36:55.481773Z"},{"marker":"00000000000000000000:00000000000000002905","last_update":"2024-04-16T19:33:06.384483Z"},{"marker":"00000000000000000000:00000000000000004738","last_update":"2024-04-16T19:36:56.708240Z"},{"marker":"00000000000000000000:00000000000000004035","last_update":"2024-04-16T19:33:21.126136Z"}] 2024-04-16T19:37:04.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.222 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:37:04.266 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.266 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.345 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.432 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.432 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.503 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-04-16T19:37:04.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.504 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 1 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.623 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-04-16T19:37:04.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.624 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 2 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.671 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.671 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.741 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-04-16T19:37:04.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:37:04.742 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 3 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:37:04.789 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.789 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.859 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-04-16T19:37:04.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2024-04-16T19:37:04.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.861 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:37:04.907 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.907 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:04.951 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:37:04.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:37:04.952 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:37:04.998 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:37:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:37:05.064 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:37:05.113 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":3,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:37:05.113 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-04-16T19:38:35.114 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-31 2024-04-16T19:38:35.115 DEBUG:boto:path=/yrjefh-31/ 2024-04-16T19:38:35.115 DEBUG:boto:auth_path=/yrjefh-31/ 2024-04-16T19:38:35.116 DEBUG:boto:Method: PUT 2024-04-16T19:38:35.116 DEBUG:boto:Path: /yrjefh-31/ 2024-04-16T19:38:35.116 DEBUG:boto:Data: 2024-04-16T19:38:35.116 DEBUG:boto:Headers: {} 2024-04-16T19:38:35.116 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:38:35.116 DEBUG:boto:Port: 8000 2024-04-16T19:38:35.116 DEBUG:boto:Params: {} 2024-04-16T19:38:35.117 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:38:35.117 DEBUG:boto:Token: None 2024-04-16T19:38:35.118 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:38:35 GMT /yrjefh-31/ 2024-04-16T19:38:35.118 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5+qeEnMIF7vpkHvLypS0A8zhPpo= 2024-04-16T19:38:35.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:38:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5+qeEnMIF7vpkHvLypS0A8zhPpo=', 'Content-Length': '0'} 2024-04-16T19:38:35.139 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a11448957103ec7b-00661ed3bb-4707-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:38:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:38:35.140 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:38:35.140 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:38:35.140 DEBUG:boto:Method: PUT 2024-04-16T19:38:35.140 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:38:35.140 DEBUG:boto:Data: 2024-04-16T19:38:35.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:38:35.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:38:35.141 DEBUG:boto:Port: 8000 2024-04-16T19:38:35.141 DEBUG:boto:Params: {} 2024-04-16T19:38:35.141 DEBUG:boto:Token: None 2024-04-16T19:38:35.141 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 16 Apr 2024 19:38:35 GMT /yrjefh-31/testredirect 2024-04-16T19:38:35.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQdjvDSoKzf81FEaw5WRNPfOf38= 2024-04-16T19:38:35.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:38:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQdjvDSoKzf81FEaw5WRNPfOf38='} 2024-04-16T19:38:35.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000555cacb4192b662f-00661ed3bb-4707-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:38:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:38:35.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:35.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:35.152 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:35.205 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:35.205 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:38:35.281 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296315.136221_288.1","last_update":"2024-04-16T19:38:35.136221Z"}] 2024-04-16T19:38:35.281 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:35.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:38:35.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:35.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:35.282 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:35.336 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:35.336 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:35.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:35.387+0000 7f5d94520800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:35.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:35.389+0000 7f5d94520800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:35.413 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:35Z"} 2024-04-16T19:38:35.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:35Z"} 2024-04-16T19:38:35.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:38:35.414 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:35.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:38:35.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:38:40.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:40.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:40.415 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:40.470 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:40.470 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:40.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:40.521+0000 7f0bd3d2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:40.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:40.523+0000 7f0bd3d2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:40.545 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:40Z"} 2024-04-16T19:38:40.545 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:40Z"} 2024-04-16T19:38:40.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:38:40.546 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:40.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:38:40.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:38:45.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:45.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:45.547 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:45.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:45.654+0000 7fda9ad33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:45.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:45.656+0000 7fda9ad33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:45.679 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:45Z"} 2024-04-16T19:38:45.680 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:45Z"} 2024-04-16T19:38:45.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:38:45.680 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:45.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:38:45.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:38:50.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:50.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:50.682 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:50.737 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:50.737 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:50.789+0000 7faa3412c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:50.790+0000 7faa3412c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:50.812 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:50Z"} 2024-04-16T19:38:50.812 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:50Z"} 2024-04-16T19:38:50.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:38:50.813 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:50.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:38:50.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:38:55.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:55.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:38:55.815 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:38:55.870 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:38:55.870 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:38:55.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:55.922+0000 7fa7ca224800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:55.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:38:55.924+0000 7fa7ca224800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:38:55.947 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:55Z"} 2024-04-16T19:38:55.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:38:55Z"} 2024-04-16T19:38:55.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:38:55.947 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:38:55.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:38:55.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:39:00.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:39:00.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:39:00.950 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:39:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:39:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:39:01.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:39:01.056+0000 7f51b7044800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:39:01.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:39:01.058+0000 7f51b7044800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:39:01.080 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:39:01Z"} 2024-04-16T19:39:01.081 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296212.226200_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:52.226200Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:39:01Z"} 2024-04-16T19:39:01.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296212.226200_259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:52.226200Z', 'realm_epoch': 2}}] 2024-04-16T19:39:01.081 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:39:01.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296212.226200_259.1'} 2024-04-16T19:39:01.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296315.136221_288.1 target=1_1713296212.226200_259.1 2024-04-16T19:39:06.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:39:06.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:39:06.083 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:39:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:39:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:39:06.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:39:06.193+0000 7f2d93543800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:39:06.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:39:06.195+0000 7f2d93543800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:39:06.217 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:39:06Z"} 2024-04-16T19:39:06.217 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:39:06Z"} 2024-04-16T19:39:06.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:39:06.218 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:39:06.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:39:06.218 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:39:06.218 DEBUG:boto:path=/yrjefh-31/ 2024-04-16T19:39:06.218 DEBUG:boto:auth_path=/yrjefh-31/ 2024-04-16T19:39:06.218 DEBUG:boto:Method: HEAD 2024-04-16T19:39:06.219 DEBUG:boto:Path: /yrjefh-31/ 2024-04-16T19:39:06.219 DEBUG:boto:Data: 2024-04-16T19:39:06.219 DEBUG:boto:Headers: {} 2024-04-16T19:39:06.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:39:06.219 DEBUG:boto:Port: 8000 2024-04-16T19:39:06.219 DEBUG:boto:Params: {} 2024-04-16T19:39:06.219 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:39:06.219 DEBUG:boto:Token: None 2024-04-16T19:39:06.220 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:39:06 GMT /yrjefh-31/ 2024-04-16T19:39:06.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8hne2Xg3oUcy+GvMQfzVHrkoC4U= 2024-04-16T19:39:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:39:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8hne2Xg3oUcy+GvMQfzVHrkoC4U=', 'Content-Length': '0'} 2024-04-16T19:39:06.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f6539a31fc31eae-00661ed3da-4771-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:39:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:39:06.226 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:39:06.226 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:39:06.226 DEBUG:boto:Method: HEAD 2024-04-16T19:39:06.226 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:39:06.226 DEBUG:boto:Data: 2024-04-16T19:39:06.227 DEBUG:boto:Headers: {} 2024-04-16T19:39:06.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:39:06.227 DEBUG:boto:Port: 8000 2024-04-16T19:39:06.227 DEBUG:boto:Params: {} 2024-04-16T19:39:06.227 DEBUG:boto:Token: None 2024-04-16T19:39:06.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:39:06 GMT /yrjefh-31/testredirect 2024-04-16T19:39:06.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fGtFvBeQpL8cAELlT99hhM2Fn2o= 2024-04-16T19:39:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:39:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fGtFvBeQpL8cAELlT99hhM2Fn2o=', 'Content-Length': '0'} 2024-04-16T19:39:06.228 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000058ec88033dfa7d52-00661ed3da-4771-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:39:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:39:06.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=b482a2a9-ef36-488b-9666-ada656453328', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:39:06.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=b482a2a9-ef36-488b-9666-ada656453328', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:39:06.229 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=b482a2a9-ef36-488b-9666-ada656453328 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:39:06.277 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:39:06.277 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:39:06.320 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:39:06.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:39:06.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:39:06.321 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:39:06.366 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:39:06.367 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:39:06.429 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:39:06.479 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":4,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":false,"sync_from":[],"redirect_zone":"b482a2a9-ef36-488b-9666-ada656453328","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:39:06.480 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "b482a2a9-ef36-488b-9666-ada656453328" 2024-04-16T19:40:36.480 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.481 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.481 DEBUG:boto:Method: HEAD 2024-04-16T19:40:36.481 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.481 DEBUG:boto:Data: 2024-04-16T19:40:36.481 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.481 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.481 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.481 DEBUG:boto:Params: {} 2024-04-16T19:40:36.482 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.482 DEBUG:boto:Token: None 2024-04-16T19:40:36.482 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.482 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+MnVLFl1SIueF6ipu0n0aZcCpmw= 2024-04-16T19:40:36.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+MnVLFl1SIueF6ipu0n0aZcCpmw=', 'Content-Length': '0'} 2024-04-16T19:40:36.490 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx000006cdb85564cb18668-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.490 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.491 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2024-04-16T19:40:36.491 DEBUG:boto:Token: None 2024-04-16T19:40:36.491 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+MnVLFl1SIueF6ipu0n0aZcCpmw= 2024-04-16T19:40:36.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+MnVLFl1SIueF6ipu0n0aZcCpmw=', 'Content-Length': '0'} 2024-04-16T19:40:36.499 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:38:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000054ff6379c1bb8d0c-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.499 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.499 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.499 DEBUG:boto:Method: GET 2024-04-16T19:40:36.499 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.499 DEBUG:boto:Data: 2024-04-16T19:40:36.499 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.500 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.500 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.500 DEBUG:boto:Params: {} 2024-04-16T19:40:36.500 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.500 DEBUG:boto:Token: None 2024-04-16T19:40:36.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.503 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx000005b97d7d1ad563d40-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.503 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.503 DEBUG:boto:Token: None 2024-04-16T19:40:36.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.506 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:38:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000f1d303a361df81f2-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.506 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.506 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.507 DEBUG:boto:Method: PUT 2024-04-16T19:40:36.507 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.507 DEBUG:boto:Data: 2024-04-16T19:40:36.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:40:36.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:40:36.507 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.507 DEBUG:boto:Params: {} 2024-04-16T19:40:36.507 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.507 DEBUG:boto:Token: None 2024-04-16T19:40:36.508 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XO3NmDkREIf/ptEvWQt+eQKxloY= 2024-04-16T19:40:36.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XO3NmDkREIf/ptEvWQt+eQKxloY='} 2024-04-16T19:40:36.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021b95bf419a3166b-00661ed434-4717-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.521 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.521 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.521 DEBUG:boto:Method: GET 2024-04-16T19:40:36.521 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.521 DEBUG:boto:Data: 2024-04-16T19:40:36.521 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.521 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.521 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.522 DEBUG:boto:Params: {} 2024-04-16T19:40:36.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.522 DEBUG:boto:Token: None 2024-04-16T19:40:36.522 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.522 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.525 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx000002b7f55803b8fe6e9-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.525 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.525 DEBUG:boto:Token: None 2024-04-16T19:40:36.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.527 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx0000095d99c90ac07da2a-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.528 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.528 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.528 DEBUG:boto:Method: PUT 2024-04-16T19:40:36.528 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.528 DEBUG:boto:Data: 2024-04-16T19:40:36.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:40:36.528 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:40:36.528 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.528 DEBUG:boto:Params: {} 2024-04-16T19:40:36.528 DEBUG:boto:Token: None 2024-04-16T19:40:36.529 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf4rY/Oe6T1IDjgMowqVfgKRo5w= 2024-04-16T19:40:36.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf4rY/Oe6T1IDjgMowqVfgKRo5w='} 2024-04-16T19:40:36.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000819db9f9adc4bf1d-00661ed434-4717-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.535 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.536 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.536 DEBUG:boto:Method: GET 2024-04-16T19:40:36.536 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.536 DEBUG:boto:Data: 2024-04-16T19:40:36.536 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.536 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.536 DEBUG:boto:Params: {} 2024-04-16T19:40:36.536 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.536 DEBUG:boto:Token: None 2024-04-16T19:40:36.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.539 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx0000000f8e44ba4c4e2f8-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.539 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.540 DEBUG:boto:Token: None 2024-04-16T19:40:36.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.542 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000b8eabad411774825-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.542 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.542 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.542 DEBUG:boto:Method: PUT 2024-04-16T19:40:36.543 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.543 DEBUG:boto:Data: 2024-04-16T19:40:36.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:40:36.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:40:36.543 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.543 DEBUG:boto:Params: {} 2024-04-16T19:40:36.543 DEBUG:boto:Token: None 2024-04-16T19:40:36.543 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c7nW5z+3uNhasShPYFwIkxvhrpw= 2024-04-16T19:40:36.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c7nW5z+3uNhasShPYFwIkxvhrpw='} 2024-04-16T19:40:36.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098addb639987bd69-00661ed434-4717-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.550 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.550 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.550 DEBUG:boto:Method: GET 2024-04-16T19:40:36.550 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.550 DEBUG:boto:Data: 2024-04-16T19:40:36.550 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.551 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.551 DEBUG:boto:Params: {} 2024-04-16T19:40:36.551 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.551 DEBUG:boto:Token: None 2024-04-16T19:40:36.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.554 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx000003a29b0e02590c15a-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.554 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.554 DEBUG:boto:Token: None 2024-04-16T19:40:36.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.557 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000ddc699493bdc5fb8-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.557 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.557 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.557 DEBUG:boto:Method: PUT 2024-04-16T19:40:36.557 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.557 DEBUG:boto:Data: 2024-04-16T19:40:36.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:40:36.557 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:40:36.558 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.558 DEBUG:boto:Params: {} 2024-04-16T19:40:36.558 DEBUG:boto:Token: None 2024-04-16T19:40:36.558 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.558 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PUfStxKuOlyNQJrjDgL4DusSdB8= 2024-04-16T19:40:36.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PUfStxKuOlyNQJrjDgL4DusSdB8='} 2024-04-16T19:40:36.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000188b6c8670d7108-00661ed434-4717-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.603 DEBUG:boto:path=/yrjefh-31/testredirect 2024-04-16T19:40:36.603 DEBUG:boto:auth_path=/yrjefh-31/testredirect 2024-04-16T19:40:36.603 DEBUG:boto:Method: GET 2024-04-16T19:40:36.603 DEBUG:boto:Path: /yrjefh-31/testredirect 2024-04-16T19:40:36.603 DEBUG:boto:Data: 2024-04-16T19:40:36.603 DEBUG:boto:Headers: {} 2024-04-16T19:40:36.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:40:36.603 DEBUG:boto:Port: 8000 2024-04-16T19:40:36.603 DEBUG:boto:Params: {} 2024-04-16T19:40:36.604 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:40:36.604 DEBUG:boto:Token: None 2024-04-16T19:40:36.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.607 DEBUG:boto:Response headers: [('Location', 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-31/testredirect?'), ('x-amz-request-id', 'tx0000045c56ac530b4fa1b-00661ed434-4837-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.607 DEBUG:boto:Redirecting: http://smithi026.front.sepia.ceph.com/yrjefh-31/testredirect 2024-04-16T19:40:36.607 DEBUG:boto:Token: None 2024-04-16T19:40:36.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:40:36 GMT /yrjefh-31/testredirect 2024-04-16T19:40:36.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA= 2024-04-16T19:40:36.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:40:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2g27QRBtSF3yF7HT+AKLPhwOZA=', 'Content-Length': '0'} 2024-04-16T19:40:36.609 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000000d6b9c507218b02f-00661ed434-4717-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:40:36.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:40:36.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:40:36.611 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:40:36.666 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:40:36.667 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:40:36.715 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:40:36.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:40:36.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:40:36.716 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:40:36.763 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:40:36.763 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:40:36.828 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:40:36.877 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":5,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"b482a2a9-ef36-488b-9666-ada656453328","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:40:36.877 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2024-04-16T19:40:37.130 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:40:37.127+0000 7fe36c233640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T19:41:43.151 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:41:43.147+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T19:42:06.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:42:06.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:42:06.879 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:42:06.934 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:42:06.935 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:42:06.983 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","notif_pool":"test-zone2.rgw.log:notif","topics_pool":"test-zone2.rgw.meta:topics","account_pool":"test-zone2.rgw.meta:accounts","group_pool":"test-zone2.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:42:06.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:42:06.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:42:06.984 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:42:07.096 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:42:07.143 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":6,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:42:07.144 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2024-04-16T19:43:37.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-04-16T19:43:37.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:37.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:37.148 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:37.203 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:37.259 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:43:37.282 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296315.136221_288.1","last_update":"2024-04-16T19:38:35.136221Z"}] 2024-04-16T19:43:37.283 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:37.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:43:37.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:37.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:37.283 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:37.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:37.390+0000 7f81a5728800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:37.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:37.391+0000 7f81a5728800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:37.413 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:37Z"} 2024-04-16T19:43:37.414 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:37Z"} 2024-04-16T19:43:37.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:43:37.414 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:37.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:37.414 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:43:37.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:37.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'W5L6GRR79K371291', '--secret', 'jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:37.415 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key W5L6GRR79K371291 --secret jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:37.494 INFO:teuthology.orchestra.run.smithi119.stderr:NOTICE: set zone's realm_id=82a6ab2b-1604-42ea-987b-3570b8df3977 2024-04-16T19:43:37.521 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"3be12cb8-2063-44e2-a437-ac0c9bc9d6f1","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","notif_pool":"remove.rgw.log:notif","topics_pool":"remove.rgw.meta:topics","account_pool":"remove.rgw.meta:accounts","group_pool":"remove.rgw.meta:groups","system_key":{"access_key":"W5L6GRR79K371291","secret_key":"jl5fLOT6RjFOwso7ZE9PyEX3Zz09knXY"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:43:37.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:37.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:37.522 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:37.568 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:37.568 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:37.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:37.602+0000 7fc5b792a800 0 period (f1f5b060-9fe8-4227-8567-18236dc68d0a does not have zone 3be12cb8-2063-44e2-a437-ac0c9bc9d6f1 configured 2024-04-16T19:43:43.913 INFO:teuthology.orchestra.run.smithi119.stderr:Sending period to new master zone b482a2a9-ef36-488b-9666-ada656453328 2024-04-16T19:43:43.963 INFO:teuthology.orchestra.run.smithi119.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":7,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"3be12cb8-2063-44e2-a437-ac0c9bc9d6f1","name":"remove","endpoints":[],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"3be12cb8-2063-44e2-a437-ac0c9bc9d6f1","val":1414074070},{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:43:43.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:43.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:43.964 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.016 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.016 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.063 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2024-04-16T19:43:44.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.063 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.114 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.114 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.150 INFO:teuthology.orchestra.run.smithi026.stderr:zone name remove not found in zonegroup test-zonegroup 2024-04-16T19:43:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-16T19:43:44.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.164 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.214 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.214 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:44.258+0000 7f6d32816800 0 Removed zone from zonegroup test-zonegroup 2024-04-16T19:43:44.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.277 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.328 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.328 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:43:44.368+0000 7fb6c2d41800 0 WARNING: can't generate connection for zone 3be12cb8-2063-44e2-a437-ac0c9bc9d6f1 id remove: no endpoints defined 2024-04-16T19:43:44.428 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":8,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:43:44.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-16T19:43:44.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.430 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.475 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.475 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.519 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"168c1590-5c84-425c-a4cc-db2e523fd3e3","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2024-04-16T19:43:44.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.519 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.564 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.564 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:43:44.607+0000 7f59faac0800 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-16T19:43:44.625 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"e8149fe8-83a1-439d-a876-663ecf724932","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","notif_pool":"remove.rgw.log:notif","topics_pool":"remove.rgw.meta:topics","account_pool":"remove.rgw.meta:accounts","group_pool":"remove.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977"} 2024-04-16T19:43:44.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.626 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.680 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.680 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.794 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":9,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"168c1590-5c84-425c-a4cc-db2e523fd3e3","name":"remove_zg","api_name":"remove_zg","is_master":false,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"e8149fe8-83a1-439d-a876-663ecf724932","zones":[{"id":"e8149fe8-83a1-439d-a876-663ecf724932","name":"remove","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"e8149fe8-83a1-439d-a876-663ecf724932","val":549113659},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:43:44.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.795 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.842 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.842 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:44.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:43:44.885+0000 7fba88533800 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-16T19:43:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:43:44.886+0000 7fba88533800 0 Removed zone from zonegroup remove_zg 2024-04-16T19:43:44.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:44.905 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:44.954 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:44.955 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:45.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:43:45.000+0000 7f836ba44800 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-16T19:43:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-16T19:43:45.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:45.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:45.015 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:45.068 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:45.068 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:45.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:45.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:43:45.121 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:43:45.172 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:45.173 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:45.270 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":10,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T19:43:45.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-16T19:43:45.271 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-32 2024-04-16T19:43:45.272 DEBUG:boto:path=/yrjefh-32/ 2024-04-16T19:43:45.272 DEBUG:boto:auth_path=/yrjefh-32/ 2024-04-16T19:43:45.272 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.272 DEBUG:boto:Path: /yrjefh-32/ 2024-04-16T19:43:45.272 DEBUG:boto:Data: 2024-04-16T19:43:45.272 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:45.272 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.273 DEBUG:boto:Params: {} 2024-04-16T19:43:45.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:43:45.273 DEBUG:boto:Token: None 2024-04-16T19:43:45.274 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-32/ 2024-04-16T19:43:45.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9nY40dTV9ghACHAA1UZgKE0IiBs= 2024-04-16T19:43:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9nY40dTV9ghACHAA1UZgKE0IiBs=', 'Content-Length': '0'} 2024-04-16T19:43:45.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c76b37e08299656f-00661ed4f1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.522 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-33 2024-04-16T19:43:45.522 DEBUG:boto:path=/yrjefh-33/ 2024-04-16T19:43:45.523 DEBUG:boto:auth_path=/yrjefh-33/ 2024-04-16T19:43:45.523 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.523 DEBUG:boto:Path: /yrjefh-33/ 2024-04-16T19:43:45.523 DEBUG:boto:Data: 2024-04-16T19:43:45.523 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:43:45.523 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.523 DEBUG:boto:Params: {} 2024-04-16T19:43:45.523 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:43:45.524 DEBUG:boto:Token: None 2024-04-16T19:43:45.524 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-33/ 2024-04-16T19:43:45.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:16d/CgJ6xXvrYmgIHO9VMJpbFcc= 2024-04-16T19:43:45.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:16d/CgJ6xXvrYmgIHO9VMJpbFcc=', 'Content-Length': '0'} 2024-04-16T19:43:45.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000442b67ae6a9b4ed0-00661ed4f1-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.595 DEBUG:boto:path=/yrjefh-32/ 2024-04-16T19:43:45.595 DEBUG:boto:auth_path=/yrjefh-32/ 2024-04-16T19:43:45.595 DEBUG:boto:path=/yrjefh-32/?website 2024-04-16T19:43:45.595 DEBUG:boto:auth_path=/yrjefh-32/?website 2024-04-16T19:43:45.595 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.595 DEBUG:boto:Path: /yrjefh-32/?website 2024-04-16T19:43:45.595 DEBUG:boto:Data: index.htmlerror.html 2024-04-16T19:43:45.595 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:45.596 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.596 DEBUG:boto:Params: {} 2024-04-16T19:43:45.596 DEBUG:boto:Token: None 2024-04-16T19:43:45.597 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-32/?website 2024-04-16T19:43:45.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OLt9GGkogjbUgrkGC4eUFrh+A9o= 2024-04-16T19:43:45.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OLt9GGkogjbUgrkGC4eUFrh+A9o=', 'Content-Length': '241'} 2024-04-16T19:43:45.598 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000003a48dfb6a38c0efb-00661ed4f1-4812-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-16T19:43:45.600 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-34 2024-04-16T19:43:45.600 DEBUG:boto:path=/yrjefh-34/ 2024-04-16T19:43:45.600 DEBUG:boto:auth_path=/yrjefh-34/ 2024-04-16T19:43:45.600 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.600 DEBUG:boto:Path: /yrjefh-34/ 2024-04-16T19:43:45.600 DEBUG:boto:Data: 2024-04-16T19:43:45.601 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.601 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:45.601 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.601 DEBUG:boto:Params: {} 2024-04-16T19:43:45.601 DEBUG:boto:Token: None 2024-04-16T19:43:45.601 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-34/ 2024-04-16T19:43:45.601 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eMUgYBj8TK22ficlQ8SJbtbyIYo= 2024-04-16T19:43:45.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eMUgYBj8TK22ficlQ8SJbtbyIYo=', 'Content-Length': '0'} 2024-04-16T19:43:45.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b067eab6ab27fe63-00661ed4f1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.614 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-35 2024-04-16T19:43:45.615 DEBUG:boto:path=/yrjefh-35/ 2024-04-16T19:43:45.615 DEBUG:boto:auth_path=/yrjefh-35/ 2024-04-16T19:43:45.615 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.615 DEBUG:boto:Path: /yrjefh-35/ 2024-04-16T19:43:45.615 DEBUG:boto:Data: 2024-04-16T19:43:45.615 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.615 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:43:45.615 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.615 DEBUG:boto:Params: {} 2024-04-16T19:43:45.615 DEBUG:boto:Token: None 2024-04-16T19:43:45.616 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-35/ 2024-04-16T19:43:45.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e/cZ+62yuHGVaLdlNugWfuU4BVI= 2024-04-16T19:43:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e/cZ+62yuHGVaLdlNugWfuU4BVI=', 'Content-Length': '0'} 2024-04-16T19:43:45.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072b642d26ce2c641-00661ed4f1-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.680 DEBUG:boto:path=/yrjefh-34/ 2024-04-16T19:43:45.680 DEBUG:boto:auth_path=/yrjefh-34/ 2024-04-16T19:43:45.680 DEBUG:boto:path=/yrjefh-34/?policy 2024-04-16T19:43:45.680 DEBUG:boto:auth_path=/yrjefh-34/?policy 2024-04-16T19:43:45.680 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.680 DEBUG:boto:Path: /yrjefh-34/?policy 2024-04-16T19:43:45.681 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-16T19:43:45.681 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:45.681 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.681 DEBUG:boto:Params: {} 2024-04-16T19:43:45.681 DEBUG:boto:Token: None 2024-04-16T19:43:45.681 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-34/?policy 2024-04-16T19:43:45.682 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dzOgQEyTEwUfjPLnnm8Rkw7XlcA= 2024-04-16T19:43:45.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dzOgQEyTEwUfjPLnnm8Rkw7XlcA=', 'Content-Length': '97'} 2024-04-16T19:43:45.687 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000040bdb1e3c8df678-00661ed4f1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.687 DEBUG:boto:path=/yrjefh-34/ 2024-04-16T19:43:45.687 DEBUG:boto:auth_path=/yrjefh-34/ 2024-04-16T19:43:45.687 DEBUG:boto:path=/yrjefh-34/?policy 2024-04-16T19:43:45.687 DEBUG:boto:auth_path=/yrjefh-34/?policy 2024-04-16T19:43:45.687 DEBUG:boto:Method: GET 2024-04-16T19:43:45.688 DEBUG:boto:Path: /yrjefh-34/?policy 2024-04-16T19:43:45.688 DEBUG:boto:Data: 2024-04-16T19:43:45.688 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.688 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:45.688 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.688 DEBUG:boto:Params: {} 2024-04-16T19:43:45.688 DEBUG:boto:Token: None 2024-04-16T19:43:45.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-34/?policy 2024-04-16T19:43:45.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ADKeyEz4ym89+Ufi9WZS9Zjyt/k= 2024-04-16T19:43:45.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ADKeyEz4ym89+Ufi9WZS9Zjyt/k=', 'Content-Length': '0'} 2024-04-16T19:43:45.690 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8ffa0d4a933f576-00661ed4f1-4812-test-zone1'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Tue, 16 Apr 2024 19:43:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:45.690 DEBUG:boto:path=/yrjefh-35/ 2024-04-16T19:43:45.690 DEBUG:boto:auth_path=/yrjefh-35/ 2024-04-16T19:43:45.690 DEBUG:boto:path=/yrjefh-35/?policy 2024-04-16T19:43:45.690 DEBUG:boto:auth_path=/yrjefh-35/?policy 2024-04-16T19:43:45.691 DEBUG:boto:Method: PUT 2024-04-16T19:43:45.691 DEBUG:boto:Path: /yrjefh-35/?policy 2024-04-16T19:43:45.691 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-16T19:43:45.691 DEBUG:boto:Headers: {} 2024-04-16T19:43:45.691 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:43:45.691 DEBUG:boto:Port: 8000 2024-04-16T19:43:45.691 DEBUG:boto:Params: {} 2024-04-16T19:43:45.691 DEBUG:boto:Token: None 2024-04-16T19:43:45.692 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:45 GMT /yrjefh-35/?policy 2024-04-16T19:43:45.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DrwgU7rx0XE+yXM0Vi0gQOYC6jk= 2024-04-16T19:43:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DrwgU7rx0XE+yXM0Vi0gQOYC6jk=', 'Content-Length': '97'} 2024-04-16T19:43:46.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014085d35921da829-00661ed4f1-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:43:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:46.702 DEBUG:boto:path=/yrjefh-35/ 2024-04-16T19:43:46.703 DEBUG:boto:auth_path=/yrjefh-35/ 2024-04-16T19:43:46.703 DEBUG:boto:path=/yrjefh-35/?policy 2024-04-16T19:43:46.703 DEBUG:boto:auth_path=/yrjefh-35/?policy 2024-04-16T19:43:46.703 DEBUG:boto:Method: GET 2024-04-16T19:43:46.703 DEBUG:boto:Path: /yrjefh-35/?policy 2024-04-16T19:43:46.703 DEBUG:boto:Data: 2024-04-16T19:43:46.703 DEBUG:boto:Headers: {} 2024-04-16T19:43:46.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:43:46.703 DEBUG:boto:Port: 8000 2024-04-16T19:43:46.703 DEBUG:boto:Params: {} 2024-04-16T19:43:46.704 DEBUG:boto:Token: None 2024-04-16T19:43:46.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:43:46 GMT /yrjefh-35/?policy 2024-04-16T19:43:46.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kloaKPuCTtC6idyE4IRvBXoZerQ= 2024-04-16T19:43:46.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kloaKPuCTtC6idyE4IRvBXoZerQ=', 'Content-Length': '0'} 2024-04-16T19:43:46.706 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000794d5ceba6eb64e3-00661ed4f2-4911-test-zone2'), ('Content-Type', 'application/json'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '97'), ('Date', 'Tue, 16 Apr 2024 19:43:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:46.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-16T19:43:46.747 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-36 2024-04-16T19:43:46.747 DEBUG:boto:path=/yrjefh-36/ 2024-04-16T19:43:46.747 DEBUG:boto:auth_path=/yrjefh-36/ 2024-04-16T19:43:46.747 DEBUG:boto:Method: PUT 2024-04-16T19:43:46.748 DEBUG:boto:Path: /yrjefh-36/ 2024-04-16T19:43:46.748 DEBUG:boto:Data: 2024-04-16T19:43:46.748 DEBUG:boto:Headers: {} 2024-04-16T19:43:46.748 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:43:46.748 DEBUG:boto:Port: 8000 2024-04-16T19:43:46.748 DEBUG:boto:Params: {} 2024-04-16T19:43:46.748 DEBUG:boto:Token: None 2024-04-16T19:43:46.748 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:46 GMT /yrjefh-36/ 2024-04-16T19:43:46.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:96JtgXsccRL1/cV15vpt+ZNwzwU= 2024-04-16T19:43:46.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:96JtgXsccRL1/cV15vpt+ZNwzwU=', 'Content-Length': '0'} 2024-04-16T19:43:46.765 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba39a02ed1a326cb-00661ed4f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:46.765 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-37 2024-04-16T19:43:46.765 DEBUG:boto:path=/yrjefh-37/ 2024-04-16T19:43:46.765 DEBUG:boto:auth_path=/yrjefh-37/ 2024-04-16T19:43:46.765 DEBUG:boto:Method: PUT 2024-04-16T19:43:46.765 DEBUG:boto:Path: /yrjefh-37/ 2024-04-16T19:43:46.765 DEBUG:boto:Data: 2024-04-16T19:43:46.766 DEBUG:boto:Headers: {} 2024-04-16T19:43:46.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:43:46.766 DEBUG:boto:Port: 8000 2024-04-16T19:43:46.766 DEBUG:boto:Params: {} 2024-04-16T19:43:46.766 DEBUG:boto:Token: None 2024-04-16T19:43:46.766 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:43:46 GMT /yrjefh-37/ 2024-04-16T19:43:46.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7YOZU05xjyoXuNEbnWAsbRTEjNA= 2024-04-16T19:43:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:43:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7YOZU05xjyoXuNEbnWAsbRTEjNA=', 'Content-Length': '0'} 2024-04-16T19:43:46.795 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d03a820c218e28b-00661ed4f2-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:43:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:43:46.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:46.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:46.796 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:46.849 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:46.849 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:46.903 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:43:46.925 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296626.782664_424.1","last_update":"2024-04-16T19:43:46.782664Z"}] 2024-04-16T19:43:46.925 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:43:46.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:43:46.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:46.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:46.926 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:47.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:47.032+0000 7fdc59b4b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:47.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:47.033+0000 7fdc59b4b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:47Z"} 2024-04-16T19:43:47.057 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:47Z"} 2024-04-16T19:43:47.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:43:47.057 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:43:47.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:47.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:43:52.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:52.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:52.059 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:52.116 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:52.116 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:52.167+0000 7f8735ba2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:52.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:52.169+0000 7f8735ba2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:52.191 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:52Z"} 2024-04-16T19:43:52.191 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:52Z"} 2024-04-16T19:43:52.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:43:52.192 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:43:52.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:52.192 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:43:57.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:57.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:43:57.193 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:43:57.251 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:43:57.251 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:43:57.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:57.306+0000 7ffa06611800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:57.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:43:57.307+0000 7ffa06611800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:43:57.330 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:57Z"} 2024-04-16T19:43:57.331 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:43:57Z"} 2024-04-16T19:43:57.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:43:57.331 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:43:57.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:43:57.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:02.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:02.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:02.334 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:02.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:02.446+0000 7f6507617800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:02.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:02.448+0000 7f6507617800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:02.471 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:02Z"} 2024-04-16T19:44:02.471 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:02Z"} 2024-04-16T19:44:02.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:02.472 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:02.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:02.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:07.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:07.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:07.473 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:07.529 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:07.529 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:07.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:07.580+0000 7f522cb33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:07.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:07.581+0000 7f522cb33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:07.605 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:07Z"} 2024-04-16T19:44:07.606 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:07Z"} 2024-04-16T19:44:07.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:07.606 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:07.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:07.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:12.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:12.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:12.608 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:12.663 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:12.663 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:12.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:12.715+0000 7f200574b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:12.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:12.716+0000 7f200574b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:12.739 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:12Z"} 2024-04-16T19:44:12.739 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:12Z"} 2024-04-16T19:44:12.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:12.739 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:12.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:12.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:17.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:17.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:17.741 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:17.797 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:17.797 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:17.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:17.851+0000 7f0e8c9a2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:17.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:17.852+0000 7f0e8c9a2800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:17.875 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:17Z"} 2024-04-16T19:44:17.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:17Z"} 2024-04-16T19:44:17.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:17.875 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:17.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:17.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:22.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:22.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:22.877 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:22.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:22.981+0000 7fd7a3527800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:22.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:22.982+0000 7fd7a3527800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:23.006 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:22Z"} 2024-04-16T19:44:23.006 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:22Z"} 2024-04-16T19:44:23.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:23.007 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:23.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:23.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:28.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:28.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:28.009 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:28.064 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:28.064 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:28.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:28.115+0000 7f91ba824800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:28.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:28.116+0000 7f91ba824800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:28.139 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:28Z"} 2024-04-16T19:44:28.139 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:28Z"} 2024-04-16T19:44:28.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:28.140 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:28.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:28.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:33.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:33.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:33.141 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:33.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:33.247+0000 7f0b7692a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:33.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:33.248+0000 7f0b7692a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:33.271 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:33Z"} 2024-04-16T19:44:33.271 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:33Z"} 2024-04-16T19:44:33.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:33.271 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:33.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:33.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:38.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:38.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:38.273 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:38.326 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:38.326 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:38.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:38.377+0000 7f269671f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:38.379+0000 7f269671f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:38.402 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:38Z"} 2024-04-16T19:44:38.403 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:38Z"} 2024-04-16T19:44:38.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:38.403 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:38.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:38.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:43.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:43.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:43.405 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:43.461 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:43.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:43.513+0000 7f9e12f2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:43.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:43.514+0000 7f9e12f2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:43.537 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:43Z"} 2024-04-16T19:44:43.538 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:43Z"} 2024-04-16T19:44:43.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:43.538 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:43.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:43.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:48.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:48.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:48.540 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:48.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:48.646+0000 7f0e1629d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:48.648+0000 7f0e1629d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:48.671 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:48Z"} 2024-04-16T19:44:48.671 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:48Z"} 2024-04-16T19:44:48.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:48.672 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:48.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:48.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:53.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:53.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:53.675 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:53.730 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:53.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:53.783+0000 7fdc66a67800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:53.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:53.784+0000 7fdc66a67800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:53.808 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:53Z"} 2024-04-16T19:44:53.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:53Z"} 2024-04-16T19:44:53.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:53.809 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:53.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:53.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:44:58.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:58.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:44:58.811 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:44:58.868 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:44:58.868 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:58.919+0000 7f0814b1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:44:58.920+0000 7f0814b1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:44:58.944 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:58Z"} 2024-04-16T19:44:58.945 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:44:58Z"} 2024-04-16T19:44:58.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:44:58.945 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:44:58.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:44:58.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:45:03.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:03.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:03.948 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:04.000 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:04.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:04.050+0000 7f1c03b2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:04.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:04.051+0000 7f1c03b2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:04.072 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:04Z"} 2024-04-16T19:45:04.072 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:04Z"} 2024-04-16T19:45:04.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:45:04.073 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:04.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:45:04.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:45:09.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:09.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:09.074 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:09.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:09.182+0000 7f34a32c0800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:09.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:09.183+0000 7f34a32c0800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:09.206 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:09Z"} 2024-04-16T19:45:09.206 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:09Z"} 2024-04-16T19:45:09.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:45:09.206 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:09.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:45:09.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:45:14.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:14.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:14.207 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:14.263 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:14.263 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:14.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:14.314+0000 7f0cdda67800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:14.316+0000 7f0cdda67800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:14.339 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:14Z"} 2024-04-16T19:45:14.340 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296315.136221_288.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:38:35.136221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:14Z"} 2024-04-16T19:45:14.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296315.136221_288.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:38:35.136221Z', 'realm_epoch': 2}}] 2024-04-16T19:45:14.340 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:14.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296315.136221_288.1'} 2024-04-16T19:45:14.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296315.136221_288.1 2024-04-16T19:45:19.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:19.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:19.341 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:19.396 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:19.420 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:19.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:19.448+0000 7f15fb131800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:19.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:19.450+0000 7f15fb131800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:19.473 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296625.518702_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:45.518702Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:19Z"} 2024-04-16T19:45:19.473 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296625.518702_389.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:45.518702Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:19Z"} 2024-04-16T19:45:19.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296625.518702_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:43:45.518702Z', 'realm_epoch': 2}}] 2024-04-16T19:45:19.473 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:19.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296625.518702_389.1'} 2024-04-16T19:45:19.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296625.518702_389.1 2024-04-16T19:45:24.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:24.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:24.475 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:24.535 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:24.535 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:24.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:24.595+0000 7ffa7c930800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:24.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:24.596+0000 7ffa7c930800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:24.618 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296625.627910_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:45.627910Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:24Z"} 2024-04-16T19:45:24.618 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296625.627910_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:45.627910Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:24Z"} 2024-04-16T19:45:24.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296625.627910_404.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:43:45.627910Z', 'realm_epoch': 2}}] 2024-04-16T19:45:24.619 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:24.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296625.627910_404.1'} 2024-04-16T19:45:24.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296626.782664_424.1 target=1_1713296625.627910_404.1 2024-04-16T19:45:29.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:29.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:29.620 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:29.675 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:29.675 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:29.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:29.729+0000 7f4064b21800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:29.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:29.731+0000 7f4064b21800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:29.753 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296626.782664_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:46.782664Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:29Z"} 2024-04-16T19:45:29.753 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296626.782664_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:43:46.782664Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:29Z"} 2024-04-16T19:45:29.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296626.782664_424.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:43:46.782664Z', 'realm_epoch': 2}}] 2024-04-16T19:45:29.754 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:29.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296626.782664_424.1'} 2024-04-16T19:45:29.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:45:29.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-36', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:29.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-36', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:29.754 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-36 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:29.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-37', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:29.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-37', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:29.918 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-37 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:30.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.064 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:30.178 INFO:teuthology.orchestra.run.smithi026.stdout:["bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:8","sync.error-log.29","obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:10","sync.error-log.23","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:10","obj_delete_at_hint.0000000026","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:2","sync.error-log.8","obj_delete_at_hint.0000000028","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:4","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:7","sync.error-log.30","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:0","obj_delete_at_hint.0000000040","sync.error-log.6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:3","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:4","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:7","obj_delete_at_hint.0000000035","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:1","obj_delete_at_hint.0000000037","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:8","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.1.retry","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:9","data_log.3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:8","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:4","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:5","data_loggenerations_metadata","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:6","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.2","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:0","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.3","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","sync.error-log.21","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:5","obj_delete_at_hint.0000000088","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:3","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:2","obj_delete_at_hint.0000000110","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:4","sync.error-log.0","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1","obj_delete_at_hint.0000000096","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:1","obj_delete_at_hint.0000000087","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:1","obj_delete_at_hint.0000000008","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:0","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:4","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:7","bilog.trim","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:3","obj_delete_at_hint.0000000016","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.1","obj_delete_at_hint.0000000041","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:6","obj_delete_at_hint.0000000018","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:7","obj_delete_at_hint.0000000092","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:3","sync.error-log.27","sync.error-log.4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:5","obj_delete_at_hint.0000000014","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:1","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","sync.error-log.16","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:4","obj_delete_at_hint.0000000064","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:10","obj_delete_at_hint.0000000071","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:8","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:0","obj_delete_at_hint.0000000074","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:7","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:1","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.3.retry","obj_delete_at_hint.0000000121","data_log.1","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.2.retry","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:0","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:4","sync.error-log.17","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:4","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:6","sync.error-log.24","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:1","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:4","data_log.2.ay7B/3ApHQp.0","obj_delete_at_hint.0000000053","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:9","obj_delete_at_hint.0000000091","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:8","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:0","sync.error-log.7","sync.error-log.3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:5","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:0","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:8","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","datalog.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:8","sync.error-log.12","obj_delete_at_hint.0000000116","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:6","sync.error-log.25","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:4","obj_delete_at_hint.0000000034","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:2","obj_delete_at_hint.0000000101","sync.error-log.13","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:5","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:6","sync.error-log.14","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:10","obj_delete_at_hint.0000000066","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:1","obj_delete_at_hint.0000000068","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:5","obj_delete_at_hint.0000000085","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:8","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:3","sync.error-log.10","obj_delete_at_hint.0000000118","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:3","obj_delete_at_hint.0000000036","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:4","obj_delete_at_hint.0000000103","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:0","obj_delete_at_hint.0000000119","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:7","obj_delete_at_hint.0000000098","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:9","obj_delete_at_hint.0000000027","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:0","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:3","obj_delete_at_hint.0000000100","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:4","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:4","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:3","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:7","obj_delete_at_hint.0000000093","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:1","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32","obj_delete_at_hint.0000000057","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:5","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:5","sync.error-log.2","data_log.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:4","obj_delete_at_hint.0000000012","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:2","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","sync.error-log.11","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:10","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16","data_log.1.nAiYC66DPpP.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:10","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:9","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data_log.0.kVt0nH2P0n2.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:6","sync.error-log.19","sync.error-log.28","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:0","obj_delete_at_hint.0000000072","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:9","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:5","obj_delete_at_hint.0000000025","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:10","obj_delete_at_hint.0000000123","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:1","sync.error-log.31","sync.error-log.18","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:2","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:5","obj_delete_at_hint.0000000060","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:2","obj_delete_at_hint.0000000109","sync.error-log.15","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:7","obj_delete_at_hint.0000000013","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:3","obj_delete_at_hint.0000000051","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:10","obj_delete_at_hint.0000000106","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:6","datalog.sync-status.shard.eafd0fcf-abef-4654-893b-ccd3b2eb5bea.0.retry","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:9","obj_delete_at_hint.0000000017","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:9","obj_delete_at_hint.0000000023","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:7","obj_delete_at_hint.0000000107","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:0","obj_delete_at_hint.0000000046","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:9","sync.error-log.26","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:1","sync.error-log.22","data_log.2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:6","obj_delete_at_hint.0000000067","sync.error-log.9","obj_delete_at_hint.0000000086","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:7","sync.error-log.5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:6","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:1","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:9","sync.error-log.20","bucket.full-sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:2","data_log.3.kpk0Q5Za2DG.0","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:7","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:8","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:7","obj_delete_at_hint.0000000050","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:3","obj_delete_at_hint.0000000102","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:9","obj_delete_at_hint.0000000124","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:2","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:9","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:9","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:1","obj_delete_at_hint.0000000097","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:5","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:10","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:10","obj_delete_at_hint.0000000042","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:3","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:9","obj_delete_at_hint.0000000075","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:7","obj_delete_at_hint.0000000120","bucket.sync-status.eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:0","obj_delete_at_hint.0000000030"] 2024-04-16T19:45:30.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.198 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:30.247 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:30.247 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:30.322 INFO:teuthology.orchestra.run.smithi119.stdout:["sync.error-log.29","obj_delete_at_hint.0000000078","meta.history","sync.error-log.23","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:4","sync.error-log.8","obj_delete_at_hint.0000000028","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.0.retry","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:0","sync.error-log.30","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:8","obj_delete_at_hint.0000000040","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:9","sync.error-log.6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:2","obj_delete_at_hint.0000000015","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:4","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:2","obj_delete_at_hint.0000000037","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:0","data_log.0.JNgxvSGns04.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:1","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:0","obj_delete_at_hint.0000000024","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:2","obj_delete_at_hint.0000000000","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:4","obj_delete_at_hint.0000000031","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:8","data_log.3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:6","data_loggenerations_metadata","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:10","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34","obj_delete_at_hint.0000000011","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:2","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:7","sync.error-log.21","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:9","sync.error-log.0","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:2","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.3.retry","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:1","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:8","obj_delete_at_hint.0000000041","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:4","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22","sync.error-log.27","sync.error-log.4","obj_delete_at_hint.0000000014","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:3","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:3","sync.error-log.16","obj_delete_at_hint.0000000064","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:10","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:5","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:2","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:5","data_log.3.a5LtfhKCDDN.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:8","obj_delete_at_hint.0000000121","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:0","data_log.1","sync.error-log.17","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:6","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:4","sync.error-log.24","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:10","obj_delete_at_hint.0000000053","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:10","obj_delete_at_hint.0000000091","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:3","meta.log.f1f5b060-9fe8-4227-8567-18236dc68d0a.3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:3","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:1","sync.error-log.7","sync.error-log.3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:0","mdlog.sync-status.shard.2","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:10","obj_delete_at_hint.0000000045","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:10","obj_delete_at_hint.0000000002","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:3","sync.error-log.12","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:8","obj_delete_at_hint.0000000116","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:3","sync.error-log.25","obj_delete_at_hint.0000000034","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:0","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17","obj_delete_at_hint.0000000101","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:1","sync.error-log.13","mdlog.sync-status.shard.1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","sync.error-log.14","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:6","obj_delete_at_hint.0000000066","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:6","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:9","obj_delete_at_hint.0000000068","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:5","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:0","sync.error-log.10","obj_delete_at_hint.0000000118","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:8","obj_delete_at_hint.0000000036","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.1.retry","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:7","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:1","mdlog.sync-status.shard.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:6","obj_delete_at_hint.0000000027","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:3","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:1","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:5","obj_delete_at_hint.0000000100","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:10","obj_delete_at_hint.0000000093","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:5","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:7","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","sync.error-log.2","data_log.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:9","obj_delete_at_hint.0000000012","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:5","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:7","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:5","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20","sync.error-log.11","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:1","obj_delete_at_hint.0000000020","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:3","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:1","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18","sync.error-log.19","sync.error-log.28","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:9","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:4","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:1","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:3","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","sync.error-log.31","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:4","sync.error-log.18","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:0","obj_delete_at_hint.0000000108","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:6","obj_delete_at_hint.0000000094","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-18:b482a2a9-ef36-488b-9666-ada656453328.4361.22:1","datalog.sync-status.shard.b482a2a9-ef36-488b-9666-ada656453328.2.retry","obj_delete_at_hint.0000000060","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:7","obj_delete_at_hint.0000000109","sync.error-log.15","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:9","obj_delete_at_hint.0000000013","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:3","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:6","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-8:b482a2a9-ef36-488b-9666-ada656453328.4361.12:4","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:2","obj_delete_at_hint.0000000046","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:6","sync.error-log.26","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:7","sync.error-log.22","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:6","data_log.2","obj_delete_at_hint.0000000067","datalog.sync-status.b482a2a9-ef36-488b-9666-ada656453328","data_log.1.BRI9EeSPMsI.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:0","sync.error-log.9","obj_delete_at_hint.0000000086","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:10","sync.error-log.5","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:0","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-29:b482a2a9-ef36-488b-9666-ada656453328.4361.33:0","sync.error-log.20","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-31:b482a2a9-ef36-488b-9666-ada656453328.4707.1:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-12:b482a2a9-ef36-488b-9666-ada656453328.4361.16:6","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21:10","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:7","obj_delete_at_hint.0000000050","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:4","obj_delete_at_hint.0000000102","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-20:b482a2a9-ef36-488b-9666-ada656453328.4361.24:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:9","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:5","obj_delete_at_hint.0000000124","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-13:b482a2a9-ef36-488b-9666-ada656453328.4361.17:10","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-7:b482a2a9-ef36-488b-9666-ada656453328.4361.11:5","obj_delete_at_hint.0000000061","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-16:b482a2a9-ef36-488b-9666-ada656453328.4361.20:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:4","obj_delete_at_hint.0000000097","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-15:b482a2a9-ef36-488b-9666-ada656453328.4361.19","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-28:b482a2a9-ef36-488b-9666-ada656453328.4361.32:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-10:b482a2a9-ef36-488b-9666-ada656453328.4361.14:7","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:2","obj_delete_at_hint.0000000042","bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-17:b482a2a9-ef36-488b-9666-ada656453328.4361.21","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-11:b482a2a9-ef36-488b-9666-ada656453328.4361.15:3","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:8","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-30:b482a2a9-ef36-488b-9666-ada656453328.4361.34:0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-9:b482a2a9-ef36-488b-9666-ada656453328.4361.13:1","obj_delete_at_hint.0000000075","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-27:b482a2a9-ef36-488b-9666-ada656453328.4361.31:3","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","data_log.2.Li5qnyEZgJF.0","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-14:b482a2a9-ef36-488b-9666-ada656453328.4361.18:2","bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-19:b482a2a9-ef36-488b-9666-ada656453328.4361.23:6"] 2024-04-16T19:45:30.342 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T19:45:30.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.342 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:30.468 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000004593","last_update":"2024-04-16T19:45:30.037412Z"},{"marker":"00000000000000000000:00000000000000003184","last_update":"2024-04-16T19:45:30.033557Z"},{"marker":"00000000000000000000:00000000000000005861","last_update":"2024-04-16T19:45:30.034916Z"},{"marker":"00000000000000000000:00000000000000005719","last_update":"2024-04-16T19:45:30.036241Z"}] 2024-04-16T19:45:30.468 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:30.468 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:45:30.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:30.468 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:30.518 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:30.518 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:30.589 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:30Z"} 2024-04-16T19:45:30.589 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:30Z"} 2024-04-16T19:45:30.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:54.423318Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.370182Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004879', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:40:37.024835Z'}}] 2024-04-16T19:45:30.590 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:30.590 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004879'} 2024-04-16T19:45:30.590 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004593 target=00000000000000000000:00000000000000003750, shard=1 master=00000000000000000000:00000000000000003184 target=00000000000000000000:00000000000000002623, shard=2 master=00000000000000000000:00000000000000005861 target=00000000000000000000:00000000000000005020, shard=3 master=00000000000000000000:00000000000000005719 target=00000000000000000000:00000000000000004879 2024-04-16T19:45:35.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:35.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:35.591 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:35.727 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:35Z"} 2024-04-16T19:45:35.727 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:35Z"} 2024-04-16T19:45:35.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:54.423318Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.370182Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004879', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:40:37.024835Z'}}] 2024-04-16T19:45:35.727 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:35.727 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004879'} 2024-04-16T19:45:35.728 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004593 target=00000000000000000000:00000000000000003750, shard=1 master=00000000000000000000:00000000000000003184 target=00000000000000000000:00000000000000002623, shard=2 master=00000000000000000000:00000000000000005861 target=00000000000000000000:00000000000000005020, shard=3 master=00000000000000000000:00000000000000005719 target=00000000000000000000:00000000000000004879 2024-04-16T19:45:36.537 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:45:36.534+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T19:45:40.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:40.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:40.730 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:40.788 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:40.788 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:40.865 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:40Z"} 2024-04-16T19:45:40.865 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:40Z"} 2024-04-16T19:45:40.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:54.423318Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.370182Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004879', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:40:37.024835Z'}}] 2024-04-16T19:45:40.865 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:40.865 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004879'} 2024-04-16T19:45:40.866 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004593 target=00000000000000000000:00000000000000003750, shard=1 master=00000000000000000000:00000000000000003184 target=00000000000000000000:00000000000000002623, shard=2 master=00000000000000000000:00000000000000005861 target=00000000000000000000:00000000000000005020, shard=3 master=00000000000000000000:00000000000000005719 target=00000000000000000000:00000000000000004879 2024-04-16T19:45:45.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:45.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:45.867 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:45.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:45.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:46.001 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:45Z"} 2024-04-16T19:45:46.001 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003750","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:54.423318Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002623","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:32:08.938133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005020","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:36:55.370182Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004879","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:40:37.024835Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:45Z"} 2024-04-16T19:45:46.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:54.423318Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:32:08.938133Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:36:55.370182Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004879', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:40:37.024835Z'}}] 2024-04-16T19:45:46.001 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:46.002 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000005020', 3: '00000000000000000000:00000000000000004879'} 2024-04-16T19:45:46.002 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004593 target=00000000000000000000:00000000000000003750, shard=1 master=00000000000000000000:00000000000000003184 target=00000000000000000000:00000000000000002623, shard=2 master=00000000000000000000:00000000000000005861 target=00000000000000000000:00000000000000005020, shard=3 master=00000000000000000000:00000000000000005719 target=00000000000000000000:00000000000000004879 2024-04-16T19:45:51.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.004 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:51.061 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:51.061 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:51.139 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004593","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.037412Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003184","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.033557Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005861","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.034916Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005719","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.036241Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.140 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004593","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.037412Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003184","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.033557Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005861","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.034916Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005719","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:30.036241Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:30.037412Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:30.033557Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005861', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:30.034916Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005719', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:30.036241Z'}}] 2024-04-16T19:45:51.140 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:51.140 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003184', 2: '00000000000000000000:00000000000000005861', 3: '00000000000000000000:00000000000000005719'} 2024-04-16T19:45:51.140 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:45:51.141 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T19:45:51.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.141 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:51.189 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:51.189 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:51.269 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000004593","last_update":"2024-04-16T19:45:32.973545Z"},{"marker":"00000000000000000000:00000000000000003466","last_update":"2024-04-16T19:45:32.969629Z"},{"marker":"00000000000000000000:00000000000000005579","last_update":"2024-04-16T19:45:32.970840Z"},{"marker":"00000000000000000000:00000000000000005016","last_update":"2024-04-16T19:45:32.972030Z"}] 2024-04-16T19:45:51.270 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003466', 2: '00000000000000000000:00000000000000005579', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:45:51.270 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:45:51.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.270 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:51.329 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:51.329 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:51.408 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004593","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.973545Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.969629Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.970840Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005016","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:45:32.972030Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.408 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004593","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.973545Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003466","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.969629Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005579","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:32.970840Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005016","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:45:32.972030Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:32.973545Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:32.969629Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005579', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:32.970840Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005016', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:45:32.972030Z'}}] 2024-04-16T19:45:51.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003466', 2: '00000000000000000000:00000000000000005579', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:45:51.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000003466', 2: '00000000000000000000:00000000000000005579', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:45:51.409 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:45:51.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2024-04-16T19:45:51.412 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-38 2024-04-16T19:45:51.413 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:51.413 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:51.414 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.414 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:51.414 DEBUG:boto:Data: 2024-04-16T19:45:51.415 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.415 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.415 DEBUG:boto:Params: {} 2024-04-16T19:45:51.416 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:45:51.417 DEBUG:boto:Token: None 2024-04-16T19:45:51.417 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/ 2024-04-16T19:45:51.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QeeF1UNN4YMROaw35LzxlXm5Dhg= 2024-04-16T19:45:51.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QeeF1UNN4YMROaw35LzxlXm5Dhg=', 'Content-Length': '0'} 2024-04-16T19:45:51.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065ba69c7cc055ac1-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.438 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-39 2024-04-16T19:45:51.438 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:51.438 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:51.438 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.439 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:51.439 DEBUG:boto:Data: 2024-04-16T19:45:51.439 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.439 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.439 DEBUG:boto:Params: {} 2024-04-16T19:45:51.439 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:45:51.440 DEBUG:boto:Token: None 2024-04-16T19:45:51.440 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/ 2024-04-16T19:45:51.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HhJXwltuPsBIahWs2lza1Ok7QfA= 2024-04-16T19:45:51.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HhJXwltuPsBIahWs2lza1Ok7QfA=', 'Content-Length': '0'} 2024-04-16T19:45:51.472 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7123d6520b42d59-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.472 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:51.472 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:51.473 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.473 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:51.473 DEBUG:boto:Data: 2024-04-16T19:45:51.473 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.473 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.473 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.473 DEBUG:boto:Params: {} 2024-04-16T19:45:51.473 DEBUG:boto:Token: None 2024-04-16T19:45:51.474 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/ 2024-04-16T19:45:51.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY= 2024-04-16T19:45:51.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY=', 'Content-Length': '0'} 2024-04-16T19:45:51.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd70b0996cf46860-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.477 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:51.477 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:51.477 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.477 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:45:51.477 DEBUG:boto:Data: 2024-04-16T19:45:51.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.477 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.478 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.478 DEBUG:boto:Params: {} 2024-04-16T19:45:51.478 DEBUG:boto:Token: None 2024-04-16T19:45:51.478 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/obj1 2024-04-16T19:45:51.478 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nCPOvTxaVWDmC4WIy8gLpQwJcOM= 2024-04-16T19:45:51.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nCPOvTxaVWDmC4WIy8gLpQwJcOM='} 2024-04-16T19:45:51.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c001e35577e38b01-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.486 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:51.486 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:51.486 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.486 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:51.486 DEBUG:boto:Data: 2024-04-16T19:45:51.486 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.486 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.486 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.487 DEBUG:boto:Params: {} 2024-04-16T19:45:51.487 DEBUG:boto:Token: None 2024-04-16T19:45:51.487 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/ 2024-04-16T19:45:51.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY= 2024-04-16T19:45:51.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY=', 'Content-Length': '0'} 2024-04-16T19:45:51.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085215012079caf4d-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.489 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:51.490 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:51.490 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.490 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:45:51.490 DEBUG:boto:Data: 2024-04-16T19:45:51.490 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.490 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.491 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.491 DEBUG:boto:Params: {} 2024-04-16T19:45:51.491 DEBUG:boto:Token: None 2024-04-16T19:45:51.491 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/obj2 2024-04-16T19:45:51.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Av6DdJUa7ixL4A8fHcPKM9HhixU= 2024-04-16T19:45:51.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Av6DdJUa7ixL4A8fHcPKM9HhixU='} 2024-04-16T19:45:51.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068873a3869e3714b-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.497 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:51.497 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:51.498 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.498 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:51.498 DEBUG:boto:Data: 2024-04-16T19:45:51.498 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.498 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.498 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.498 DEBUG:boto:Params: {} 2024-04-16T19:45:51.498 DEBUG:boto:Token: None 2024-04-16T19:45:51.499 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/ 2024-04-16T19:45:51.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY= 2024-04-16T19:45:51.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY=', 'Content-Length': '0'} 2024-04-16T19:45:51.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d790e04cf2abaadc-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.501 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:51.501 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:51.501 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.502 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:45:51.502 DEBUG:boto:Data: 2024-04-16T19:45:51.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.502 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.502 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.502 DEBUG:boto:Params: {} 2024-04-16T19:45:51.502 DEBUG:boto:Token: None 2024-04-16T19:45:51.502 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/obj3 2024-04-16T19:45:51.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jTcySrS/cL7LZerQCmllCK9yWX0= 2024-04-16T19:45:51.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jTcySrS/cL7LZerQCmllCK9yWX0='} 2024-04-16T19:45:51.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dd2ca7c7c8eaaea-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.509 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:51.509 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:51.509 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.509 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:51.509 DEBUG:boto:Data: 2024-04-16T19:45:51.510 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.510 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.510 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.510 DEBUG:boto:Params: {} 2024-04-16T19:45:51.510 DEBUG:boto:Token: None 2024-04-16T19:45:51.510 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/ 2024-04-16T19:45:51.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY= 2024-04-16T19:45:51.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/WV2tnw1DPCkdfPIDwwIul/GwZY=', 'Content-Length': '0'} 2024-04-16T19:45:51.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e19ce09d7e4d3f95-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.513 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:51.513 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:51.513 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.514 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:45:51.514 DEBUG:boto:Data: 2024-04-16T19:45:51.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.514 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:51.514 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.514 DEBUG:boto:Params: {} 2024-04-16T19:45:51.514 DEBUG:boto:Token: None 2024-04-16T19:45:51.514 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-38/obj4 2024-04-16T19:45:51.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:isjQpCLwMrNjv2bqhkiRy6+0m04= 2024-04-16T19:45:51.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:isjQpCLwMrNjv2bqhkiRy6+0m04='} 2024-04-16T19:45:51.535 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:45:51.534+0000 7fe37924d640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-16T19:45:51.536 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:45:51.534+0000 7fe376a48640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-16T19:45:51.536 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:45:51.534+0000 7fe37a24f640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-16T19:45:51.537 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:45:51.534+0000 7fe373a42640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-16T19:45:51.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055ce78197525227b-00661ed56f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.559 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:51.559 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:51.559 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.559 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:51.559 DEBUG:boto:Data: 2024-04-16T19:45:51.560 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.560 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.560 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.560 DEBUG:boto:Params: {} 2024-04-16T19:45:51.560 DEBUG:boto:Token: None 2024-04-16T19:45:51.560 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/ 2024-04-16T19:45:51.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8= 2024-04-16T19:45:51.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8=', 'Content-Length': '0'} 2024-04-16T19:45:51.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000575f46658291aa0e-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.563 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:51.563 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:51.564 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.564 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:45:51.564 DEBUG:boto:Data: 2024-04-16T19:45:51.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.564 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.564 DEBUG:boto:Params: {} 2024-04-16T19:45:51.564 DEBUG:boto:Token: None 2024-04-16T19:45:51.565 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/obj1 2024-04-16T19:45:51.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JpuX2IPABDPXHEXY4dLGlGItZAc= 2024-04-16T19:45:51.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JpuX2IPABDPXHEXY4dLGlGItZAc='} 2024-04-16T19:45:51.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fe70e48b51b478a-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.572 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:51.573 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:51.573 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.573 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:51.573 DEBUG:boto:Data: 2024-04-16T19:45:51.573 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.573 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.573 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.573 DEBUG:boto:Params: {} 2024-04-16T19:45:51.574 DEBUG:boto:Token: None 2024-04-16T19:45:51.574 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/ 2024-04-16T19:45:51.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8= 2024-04-16T19:45:51.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8=', 'Content-Length': '0'} 2024-04-16T19:45:51.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006481aec5e9ed44ec-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.577 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:51.577 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:51.577 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.577 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:45:51.577 DEBUG:boto:Data: 2024-04-16T19:45:51.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.578 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.578 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.578 DEBUG:boto:Params: {} 2024-04-16T19:45:51.578 DEBUG:boto:Token: None 2024-04-16T19:45:51.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/obj2 2024-04-16T19:45:51.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GeYF/oAt4sNeZDH+xT0kURpW4iU= 2024-04-16T19:45:51.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GeYF/oAt4sNeZDH+xT0kURpW4iU='} 2024-04-16T19:45:51.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5fb765a6843e13a-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.589 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:51.589 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:51.589 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.590 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:51.590 DEBUG:boto:Data: 2024-04-16T19:45:51.590 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.590 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.590 DEBUG:boto:Params: {} 2024-04-16T19:45:51.590 DEBUG:boto:Token: None 2024-04-16T19:45:51.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/ 2024-04-16T19:45:51.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8= 2024-04-16T19:45:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8=', 'Content-Length': '0'} 2024-04-16T19:45:51.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e348b187e5e0ecd-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.593 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:51.593 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:51.593 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.594 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:45:51.594 DEBUG:boto:Data: 2024-04-16T19:45:51.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.594 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.594 DEBUG:boto:Params: {} 2024-04-16T19:45:51.594 DEBUG:boto:Token: None 2024-04-16T19:45:51.595 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/obj3 2024-04-16T19:45:51.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6siskZ+kXoqnQW6fsd1TdheF9Bg= 2024-04-16T19:45:51.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6siskZ+kXoqnQW6fsd1TdheF9Bg='} 2024-04-16T19:45:51.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c04eaf82da91d8f7-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.639 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:51.639 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:51.639 DEBUG:boto:Method: HEAD 2024-04-16T19:45:51.639 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:51.639 DEBUG:boto:Data: 2024-04-16T19:45:51.639 DEBUG:boto:Headers: {} 2024-04-16T19:45:51.640 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.640 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.640 DEBUG:boto:Params: {} 2024-04-16T19:45:51.640 DEBUG:boto:Token: None 2024-04-16T19:45:51.640 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/ 2024-04-16T19:45:51.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8= 2024-04-16T19:45:51.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3qC+upxW/Qn05AL0eOMQH6wcY8=', 'Content-Length': '0'} 2024-04-16T19:45:51.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015880dccd8643bfe-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.643 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:51.643 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:51.643 DEBUG:boto:Method: PUT 2024-04-16T19:45:51.643 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:45:51.643 DEBUG:boto:Data: 2024-04-16T19:45:51.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:51.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:51.644 DEBUG:boto:Port: 8000 2024-04-16T19:45:51.644 DEBUG:boto:Params: {} 2024-04-16T19:45:51.644 DEBUG:boto:Token: None 2024-04-16T19:45:51.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:51 GMT /yrjefh-39/obj4 2024-04-16T19:45:51.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qMFeaWh+LKAlleqPagN2I2qyXKo= 2024-04-16T19:45:51.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qMFeaWh+LKAlleqPagN2I2qyXKo='} 2024-04-16T19:45:51.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5285fdd06bc93e4-00661ed56f-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:51.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.652 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:51.698 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:45:51.770 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296751.456560_453.1","last_update":"2024-04-16T19:45:51.456560Z"}] 2024-04-16T19:45:51.771 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296751.456560_453.1'} 2024-04-16T19:45:51.771 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:45:51.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:51.771 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:51.821 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:51.821 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:51.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:51.870+0000 7f4700c0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:51.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:51.871+0000 7f4700c0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:51.893 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296751.427026_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:51.427026Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.893 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296751.427026_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:51.427026Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:51Z"} 2024-04-16T19:45:51.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296751.427026_446.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:51.427026Z', 'realm_epoch': 2}}] 2024-04-16T19:45:51.893 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296751.456560_453.1'} 2024-04-16T19:45:51.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296751.427026_446.1'} 2024-04-16T19:45:51.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296751.456560_453.1 target=1_1713296751.427026_446.1 2024-04-16T19:45:56.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:56.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:45:56.895 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:45:56.949 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:56.950 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:57.002+0000 7f177ba2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:57.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:57.003+0000 7f177ba2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:45:57.026 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296751.456560_453.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:51.456560Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:57Z"} 2024-04-16T19:45:57.026 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296751.456560_453.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:51.456560Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:45:57Z"} 2024-04-16T19:45:57.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296751.456560_453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:51.456560Z', 'realm_epoch': 2}}] 2024-04-16T19:45:57.027 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296751.456560_453.1'} 2024-04-16T19:45:57.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296751.456560_453.1'} 2024-04-16T19:45:57.027 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:45:57.027 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-38 2024-04-16T19:45:57.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:57.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:57.027 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-38 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:45:57.082 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:57.082 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:57.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:57.136+0000 7f910793b800 1 bucket sync caught up with source: 2024-04-16T19:45:57.137 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.32.6, , 00000000001.46.6, , 00000000001.34.6, , , 00000000001.65.6, , , ] 2024-04-16T19:45:57.137 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.32.6, , 00000000001.46.6, , 00000000001.34.6, , , 00000000001.65.6, , , ] 2024-04-16T19:45:57.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:57.136+0000 7f910793b800 0 bucket checkpoint complete 2024-04-16T19:45:57.157 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-38 2024-04-16T19:45:57.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:57.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:57.157 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-38 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:45:57.219 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:57.219 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:57.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:45:57.281+0000 7fd384ba2800 1 bucket sync caught up with source: 2024-04-16T19:45:57.282 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:45:57.282 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:45:57.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:45:57.281+0000 7fd384ba2800 0 bucket checkpoint complete 2024-04-16T19:45:57.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-38 zones={test-zone2, test-zone1} 2024-04-16T19:45:57.304 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.304 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.304 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.304 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:57.305 DEBUG:boto:Data: 2024-04-16T19:45:57.305 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.305 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.305 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.305 DEBUG:boto:Params: {} 2024-04-16T19:45:57.305 DEBUG:boto:Token: None 2024-04-16T19:45:57.305 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3xxuVvRV1e8itnBxkJQDiP3/xR0= 2024-04-16T19:45:57.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xxuVvRV1e8itnBxkJQDiP3/xR0=', 'Content-Length': '0'} 2024-04-16T19:45:57.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018e06bd6880e4aab-00661ed575-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.308 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.308 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.308 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.308 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:57.308 DEBUG:boto:Data: 2024-04-16T19:45:57.308 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.308 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.308 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.308 DEBUG:boto:Params: {} 2024-04-16T19:45:57.308 DEBUG:boto:Token: None 2024-04-16T19:45:57.309 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3xxuVvRV1e8itnBxkJQDiP3/xR0= 2024-04-16T19:45:57.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xxuVvRV1e8itnBxkJQDiP3/xR0=', 'Content-Length': '0'} 2024-04-16T19:45:57.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb441001f5e073da-00661ed575-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:45:57.311 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.311 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.311 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.311 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.312 DEBUG:boto:Method: GET 2024-04-16T19:45:57.312 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.312 DEBUG:boto:Data: 2024-04-16T19:45:57.312 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.312 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.312 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.312 DEBUG:boto:Params: {} 2024-04-16T19:45:57.312 DEBUG:boto:Token: None 2024-04-16T19:45:57.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/?versions 2024-04-16T19:45:57.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg= 2024-04-16T19:45:57.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg=', 'Content-Length': '0'} 2024-04-16T19:45:57.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644a02b9dccb653b-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.315 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:45:57.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:45:57.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:45:57.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:45:57.316 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:45:57.316 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.316 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.316 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.316 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.316 DEBUG:boto:Method: GET 2024-04-16T19:45:57.317 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.317 DEBUG:boto:Data: 2024-04-16T19:45:57.317 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.317 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.317 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.317 DEBUG:boto:Params: {} 2024-04-16T19:45:57.317 DEBUG:boto:Token: None 2024-04-16T19:45:57.317 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/?versions 2024-04-16T19:45:57.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg= 2024-04-16T19:45:57.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg=', 'Content-Length': '0'} 2024-04-16T19:45:57.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c45a7e5d10d9922-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.319 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:45:57.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:45:57.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:45:57.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:45:57.320 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.320 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.320 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.320 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.320 DEBUG:boto:Method: GET 2024-04-16T19:45:57.320 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.320 DEBUG:boto:Data: 2024-04-16T19:45:57.320 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.321 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.321 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.321 DEBUG:boto:Params: {} 2024-04-16T19:45:57.321 DEBUG:boto:Token: None 2024-04-16T19:45:57.321 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/?versions 2024-04-16T19:45:57.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg= 2024-04-16T19:45:57.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg=', 'Content-Length': '0'} 2024-04-16T19:45:57.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a22dad7540f9b65c-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.324 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.324 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.324 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.324 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.325 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.325 DEBUG:boto:Method: GET 2024-04-16T19:45:57.325 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:45:57.325 DEBUG:boto:Data: 2024-04-16T19:45:57.325 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.325 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.325 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.325 DEBUG:boto:Params: {} 2024-04-16T19:45:57.325 DEBUG:boto:Token: None 2024-04-16T19:45:57.325 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/?versions 2024-04-16T19:45:57.325 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg= 2024-04-16T19:45:57.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XPrX/tUbA1B39jDzNMY99l76qHg=', 'Content-Length': '0'} 2024-04-16T19:45:57.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003e655987e620d3b-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.328 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:57.328 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.328 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.328 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.328 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.329 DEBUG:boto:Method: GET 2024-04-16T19:45:57.329 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.329 DEBUG:boto:Data: 2024-04-16T19:45:57.329 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.329 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.329 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.329 DEBUG:boto:Params: {} 2024-04-16T19:45:57.329 DEBUG:boto:Token: None 2024-04-16T19:45:57.329 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.329 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY= 2024-04-16T19:45:57.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY=', 'Content-Length': '0'} 2024-04-16T19:45:57.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017dbf3f8f36d4b19-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.374 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.375 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.375 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.375 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.375 DEBUG:boto:Method: GET 2024-04-16T19:45:57.375 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.375 DEBUG:boto:Data: 2024-04-16T19:45:57.375 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.375 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.375 DEBUG:boto:Params: {} 2024-04-16T19:45:57.375 DEBUG:boto:Token: None 2024-04-16T19:45:57.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY= 2024-04-16T19:45:57.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY=', 'Content-Length': '0'} 2024-04-16T19:45:57.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066f76a835f0784b9-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.377 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.377 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.378 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.378 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.378 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.378 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.378 DEBUG:boto:Data: 2024-04-16T19:45:57.378 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.378 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.378 DEBUG:boto:Params: {} 2024-04-16T19:45:57.378 DEBUG:boto:Token: None 2024-04-16T19:45:57.378 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EoT5NUZ/a3xiold8KeCsleJ+t0w= 2024-04-16T19:45:57.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EoT5NUZ/a3xiold8KeCsleJ+t0w=', 'Content-Length': '0'} 2024-04-16T19:45:57.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064a7e486027a5039-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.380 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.381 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.381 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.381 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.381 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.381 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.381 DEBUG:boto:Data: 2024-04-16T19:45:57.381 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.381 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.381 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.381 DEBUG:boto:Params: {} 2024-04-16T19:45:57.381 DEBUG:boto:Token: None 2024-04-16T19:45:57.381 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.382 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EoT5NUZ/a3xiold8KeCsleJ+t0w= 2024-04-16T19:45:57.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EoT5NUZ/a3xiold8KeCsleJ+t0w=', 'Content-Length': '0'} 2024-04-16T19:45:57.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000981702d0bd1a2af6-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:57.383 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.384 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.384 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.384 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.384 DEBUG:boto:Method: GET 2024-04-16T19:45:57.384 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.384 DEBUG:boto:Data: 2024-04-16T19:45:57.384 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.384 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.384 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.384 DEBUG:boto:Params: {} 2024-04-16T19:45:57.384 DEBUG:boto:Token: None 2024-04-16T19:45:57.385 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY= 2024-04-16T19:45:57.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY=', 'Content-Length': '0'} 2024-04-16T19:45:57.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f493d25bd763af9a-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.387 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.387 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.387 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.387 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.387 DEBUG:boto:Method: GET 2024-04-16T19:45:57.387 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.387 DEBUG:boto:Data: 2024-04-16T19:45:57.387 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.388 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.388 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.388 DEBUG:boto:Params: {} 2024-04-16T19:45:57.388 DEBUG:boto:Token: None 2024-04-16T19:45:57.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:45:57.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY= 2024-04-16T19:45:57.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/h83wPJyCc7AEI3sZbcsw/NFvY=', 'Content-Length': '0'} 2024-04-16T19:45:57.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076d054347350f6ca-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.390 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.390 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.390 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.390 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.390 DEBUG:boto:Method: GET 2024-04-16T19:45:57.390 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.390 DEBUG:boto:Data: 2024-04-16T19:45:57.390 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.391 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.391 DEBUG:boto:Params: {} 2024-04-16T19:45:57.391 DEBUG:boto:Token: None 2024-04-16T19:45:57.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b043257950e457-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.393 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.394 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.394 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.394 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.394 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.394 DEBUG:boto:Method: GET 2024-04-16T19:45:57.394 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.394 DEBUG:boto:Data: 2024-04-16T19:45:57.394 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.394 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.394 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.394 DEBUG:boto:Params: {} 2024-04-16T19:45:57.395 DEBUG:boto:Token: None 2024-04-16T19:45:57.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1dd662f9570f58-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.397 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.397 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.398 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.398 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.398 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:45:57.398 DEBUG:boto:Data: 2024-04-16T19:45:57.398 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.398 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.398 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.398 DEBUG:boto:Params: {} 2024-04-16T19:45:57.398 DEBUG:boto:Token: None 2024-04-16T19:45:57.398 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1 2024-04-16T19:45:57.398 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ucyBwey2WtqkO2jknK+zO+kSivQ= 2024-04-16T19:45:57.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ucyBwey2WtqkO2jknK+zO+kSivQ=', 'Content-Length': '0'} 2024-04-16T19:45:57.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f47e2f6a3ec10135-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.400 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.400 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.401 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.401 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:45:57.401 DEBUG:boto:Data: 2024-04-16T19:45:57.401 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.401 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.401 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.401 DEBUG:boto:Params: {} 2024-04-16T19:45:57.401 DEBUG:boto:Token: None 2024-04-16T19:45:57.401 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1 2024-04-16T19:45:57.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ucyBwey2WtqkO2jknK+zO+kSivQ= 2024-04-16T19:45:57.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ucyBwey2WtqkO2jknK+zO+kSivQ=', 'Content-Length': '0'} 2024-04-16T19:45:57.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007aec80d50e4c02e6-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:57.403 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.403 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.404 DEBUG:boto:Method: GET 2024-04-16T19:45:57.404 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:45:57.404 DEBUG:boto:Data: 2024-04-16T19:45:57.404 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.404 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.404 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.404 DEBUG:boto:Params: {} 2024-04-16T19:45:57.404 DEBUG:boto:Token: None 2024-04-16T19:45:57.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1 2024-04-16T19:45:57.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rAhxD/C14esPW8PJ8/TPL11Psdw= 2024-04-16T19:45:57.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rAhxD/C14esPW8PJ8/TPL11Psdw=', 'Content-Length': '0'} 2024-04-16T19:45:57.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051f4ebac8566c006-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.450 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:45:57.450 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:45:57.451 DEBUG:boto:Method: GET 2024-04-16T19:45:57.451 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:45:57.451 DEBUG:boto:Data: 2024-04-16T19:45:57.451 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.451 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.451 DEBUG:boto:Params: {} 2024-04-16T19:45:57.451 DEBUG:boto:Token: None 2024-04-16T19:45:57.451 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj1 2024-04-16T19:45:57.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rAhxD/C14esPW8PJ8/TPL11Psdw= 2024-04-16T19:45:57.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rAhxD/C14esPW8PJ8/TPL11Psdw=', 'Content-Length': '0'} 2024-04-16T19:45:57.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0baedc75db8e6b3-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.453 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.453 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.454 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.454 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.454 DEBUG:boto:Method: GET 2024-04-16T19:45:57.454 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.454 DEBUG:boto:Data: 2024-04-16T19:45:57.454 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.454 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.454 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.454 DEBUG:boto:Params: {} 2024-04-16T19:45:57.454 DEBUG:boto:Token: None 2024-04-16T19:45:57.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa12bd66e05a476b-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.498 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.499 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.499 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.499 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.499 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.499 DEBUG:boto:Method: GET 2024-04-16T19:45:57.499 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:45:57.499 DEBUG:boto:Data: 2024-04-16T19:45:57.499 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.499 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.499 DEBUG:boto:Params: {} 2024-04-16T19:45:57.499 DEBUG:boto:Token: None 2024-04-16T19:45:57.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3464b6d2aed492c-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.502 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:57.502 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.502 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.502 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.503 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.503 DEBUG:boto:Method: GET 2024-04-16T19:45:57.503 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.503 DEBUG:boto:Data: 2024-04-16T19:45:57.503 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.503 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.503 DEBUG:boto:Params: {} 2024-04-16T19:45:57.503 DEBUG:boto:Token: None 2024-04-16T19:45:57.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA= 2024-04-16T19:45:57.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA=', 'Content-Length': '0'} 2024-04-16T19:45:57.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002b079b8154c9b98-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.546 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.546 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.547 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.547 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.547 DEBUG:boto:Method: GET 2024-04-16T19:45:57.547 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.547 DEBUG:boto:Data: 2024-04-16T19:45:57.547 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.547 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.547 DEBUG:boto:Params: {} 2024-04-16T19:45:57.547 DEBUG:boto:Token: None 2024-04-16T19:45:57.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA= 2024-04-16T19:45:57.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA=', 'Content-Length': '0'} 2024-04-16T19:45:57.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034e4932e4164107e-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.549 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.549 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.549 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.550 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.550 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.550 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.550 DEBUG:boto:Data: 2024-04-16T19:45:57.550 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.550 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.550 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.550 DEBUG:boto:Params: {} 2024-04-16T19:45:57.550 DEBUG:boto:Token: None 2024-04-16T19:45:57.550 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KF8VNtLLrKlGva/ASFNL8WiJy9s= 2024-04-16T19:45:57.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KF8VNtLLrKlGva/ASFNL8WiJy9s=', 'Content-Length': '0'} 2024-04-16T19:45:57.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7a4172ccd4e27b0-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.552 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.553 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.553 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.553 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.553 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.553 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.553 DEBUG:boto:Data: 2024-04-16T19:45:57.553 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.553 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.553 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.553 DEBUG:boto:Params: {} 2024-04-16T19:45:57.553 DEBUG:boto:Token: None 2024-04-16T19:45:57.553 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KF8VNtLLrKlGva/ASFNL8WiJy9s= 2024-04-16T19:45:57.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KF8VNtLLrKlGva/ASFNL8WiJy9s=', 'Content-Length': '0'} 2024-04-16T19:45:57.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b3687d26545b090-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:57.555 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.555 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.556 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.556 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.556 DEBUG:boto:Method: GET 2024-04-16T19:45:57.556 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.556 DEBUG:boto:Data: 2024-04-16T19:45:57.556 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.556 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.556 DEBUG:boto:Params: {} 2024-04-16T19:45:57.556 DEBUG:boto:Token: None 2024-04-16T19:45:57.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA= 2024-04-16T19:45:57.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA=', 'Content-Length': '0'} 2024-04-16T19:45:57.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac128b443aa2cb58-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.602 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.602 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.603 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.603 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.603 DEBUG:boto:Method: GET 2024-04-16T19:45:57.603 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.603 DEBUG:boto:Data: 2024-04-16T19:45:57.603 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.603 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.603 DEBUG:boto:Params: {} 2024-04-16T19:45:57.603 DEBUG:boto:Token: None 2024-04-16T19:45:57.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:45:57.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA= 2024-04-16T19:45:57.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W8VcfJZEc71OItTIJQeCBis/fMA=', 'Content-Length': '0'} 2024-04-16T19:45:57.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fc52c8c34cb38c3-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.605 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.606 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.606 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.606 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.606 DEBUG:boto:Method: GET 2024-04-16T19:45:57.606 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.606 DEBUG:boto:Data: 2024-04-16T19:45:57.606 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.606 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.606 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.606 DEBUG:boto:Params: {} 2024-04-16T19:45:57.606 DEBUG:boto:Token: None 2024-04-16T19:45:57.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a744fae8a66a6e8-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.650 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.650 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.651 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.651 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.651 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.651 DEBUG:boto:Method: GET 2024-04-16T19:45:57.651 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.651 DEBUG:boto:Data: 2024-04-16T19:45:57.651 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.651 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.651 DEBUG:boto:Params: {} 2024-04-16T19:45:57.651 DEBUG:boto:Token: None 2024-04-16T19:45:57.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1dbcde2c3f36d4-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.654 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.654 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.654 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.654 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.655 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:45:57.655 DEBUG:boto:Data: 2024-04-16T19:45:57.655 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.655 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.655 DEBUG:boto:Params: {} 2024-04-16T19:45:57.655 DEBUG:boto:Token: None 2024-04-16T19:45:57.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2 2024-04-16T19:45:57.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fJT6yzKjzxDpvuTEjGkm7WoBCjQ= 2024-04-16T19:45:57.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fJT6yzKjzxDpvuTEjGkm7WoBCjQ=', 'Content-Length': '0'} 2024-04-16T19:45:57.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c61b1def242b4fc7-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.657 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.657 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.657 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.657 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:45:57.657 DEBUG:boto:Data: 2024-04-16T19:45:57.657 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.658 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.658 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.658 DEBUG:boto:Params: {} 2024-04-16T19:45:57.658 DEBUG:boto:Token: None 2024-04-16T19:45:57.658 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2 2024-04-16T19:45:57.658 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fJT6yzKjzxDpvuTEjGkm7WoBCjQ= 2024-04-16T19:45:57.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fJT6yzKjzxDpvuTEjGkm7WoBCjQ=', 'Content-Length': '0'} 2024-04-16T19:45:57.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005beba563a7c15fca-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:57.660 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.660 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.660 DEBUG:boto:Method: GET 2024-04-16T19:45:57.660 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:45:57.660 DEBUG:boto:Data: 2024-04-16T19:45:57.661 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.661 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.661 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.661 DEBUG:boto:Params: {} 2024-04-16T19:45:57.661 DEBUG:boto:Token: None 2024-04-16T19:45:57.661 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2 2024-04-16T19:45:57.661 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XP+xETsK709MfXVpaVp9r6Qpg38= 2024-04-16T19:45:57.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XP+xETsK709MfXVpaVp9r6Qpg38=', 'Content-Length': '0'} 2024-04-16T19:45:57.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b86754c2ae4113b5-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.706 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:45:57.706 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:45:57.706 DEBUG:boto:Method: GET 2024-04-16T19:45:57.707 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:45:57.707 DEBUG:boto:Data: 2024-04-16T19:45:57.707 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.707 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.707 DEBUG:boto:Params: {} 2024-04-16T19:45:57.707 DEBUG:boto:Token: None 2024-04-16T19:45:57.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj2 2024-04-16T19:45:57.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XP+xETsK709MfXVpaVp9r6Qpg38= 2024-04-16T19:45:57.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XP+xETsK709MfXVpaVp9r6Qpg38=', 'Content-Length': '0'} 2024-04-16T19:45:57.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c954547259b02bb-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.709 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.709 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.710 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.710 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.710 DEBUG:boto:Method: GET 2024-04-16T19:45:57.710 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.710 DEBUG:boto:Data: 2024-04-16T19:45:57.710 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.710 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.710 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.710 DEBUG:boto:Params: {} 2024-04-16T19:45:57.710 DEBUG:boto:Token: None 2024-04-16T19:45:57.710 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b1f411bf594053-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.713 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.713 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.713 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.713 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.714 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.714 DEBUG:boto:Method: GET 2024-04-16T19:45:57.714 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:45:57.714 DEBUG:boto:Data: 2024-04-16T19:45:57.714 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.714 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.714 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.714 DEBUG:boto:Params: {} 2024-04-16T19:45:57.714 DEBUG:boto:Token: None 2024-04-16T19:45:57.714 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000363c0280e2a17737-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.717 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:57.717 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.717 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.717 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.717 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.717 DEBUG:boto:Method: GET 2024-04-16T19:45:57.717 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.717 DEBUG:boto:Data: 2024-04-16T19:45:57.718 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.718 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.718 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.718 DEBUG:boto:Params: {} 2024-04-16T19:45:57.718 DEBUG:boto:Token: None 2024-04-16T19:45:57.718 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8= 2024-04-16T19:45:57.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8=', 'Content-Length': '0'} 2024-04-16T19:45:57.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6cffd8d270d03e-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.762 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.763 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.763 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.763 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.763 DEBUG:boto:Method: GET 2024-04-16T19:45:57.763 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.763 DEBUG:boto:Data: 2024-04-16T19:45:57.763 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.763 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.763 DEBUG:boto:Params: {} 2024-04-16T19:45:57.763 DEBUG:boto:Token: None 2024-04-16T19:45:57.764 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8= 2024-04-16T19:45:57.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8=', 'Content-Length': '0'} 2024-04-16T19:45:57.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da939a8936c278f0-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.766 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.767 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.767 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.767 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.767 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.767 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.767 DEBUG:boto:Data: 2024-04-16T19:45:57.767 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.767 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.767 DEBUG:boto:Params: {} 2024-04-16T19:45:57.767 DEBUG:boto:Token: None 2024-04-16T19:45:57.768 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mc6jFTg/U045Esp7JX4Bnd1Rtnw= 2024-04-16T19:45:57.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mc6jFTg/U045Esp7JX4Bnd1Rtnw=', 'Content-Length': '0'} 2024-04-16T19:45:57.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093d8819006e486d1-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.770 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.770 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.770 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.770 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.770 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.770 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.770 DEBUG:boto:Data: 2024-04-16T19:45:57.771 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.771 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.771 DEBUG:boto:Params: {} 2024-04-16T19:45:57.771 DEBUG:boto:Token: None 2024-04-16T19:45:57.771 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mc6jFTg/U045Esp7JX4Bnd1Rtnw= 2024-04-16T19:45:57.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mc6jFTg/U045Esp7JX4Bnd1Rtnw=', 'Content-Length': '0'} 2024-04-16T19:45:57.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006027f3893d2d2b79-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:57.774 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.774 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.774 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.774 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.774 DEBUG:boto:Method: GET 2024-04-16T19:45:57.774 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.775 DEBUG:boto:Data: 2024-04-16T19:45:57.775 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.775 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.775 DEBUG:boto:Params: {} 2024-04-16T19:45:57.775 DEBUG:boto:Token: None 2024-04-16T19:45:57.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8= 2024-04-16T19:45:57.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8=', 'Content-Length': '0'} 2024-04-16T19:45:57.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3bf81fd54e2bf48-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.818 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.819 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.819 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.819 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.819 DEBUG:boto:Method: GET 2024-04-16T19:45:57.819 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.819 DEBUG:boto:Data: 2024-04-16T19:45:57.819 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.819 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.819 DEBUG:boto:Params: {} 2024-04-16T19:45:57.819 DEBUG:boto:Token: None 2024-04-16T19:45:57.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:45:57.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8= 2024-04-16T19:45:57.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Il2KJ3ftdh/WUk4ZtmwZ0UK1mj8=', 'Content-Length': '0'} 2024-04-16T19:45:57.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072b6f5de48b5abba-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.823 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.823 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.823 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.823 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.823 DEBUG:boto:Method: GET 2024-04-16T19:45:57.823 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.823 DEBUG:boto:Data: 2024-04-16T19:45:57.823 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.823 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.823 DEBUG:boto:Params: {} 2024-04-16T19:45:57.824 DEBUG:boto:Token: None 2024-04-16T19:45:57.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cebd3e409c18e0-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.826 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.827 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.827 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.827 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.827 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.827 DEBUG:boto:Method: GET 2024-04-16T19:45:57.827 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.827 DEBUG:boto:Data: 2024-04-16T19:45:57.827 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.828 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.828 DEBUG:boto:Params: {} 2024-04-16T19:45:57.828 DEBUG:boto:Token: None 2024-04-16T19:45:57.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000743216f8b83dce68-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.830 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.831 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.831 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.831 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.831 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:45:57.831 DEBUG:boto:Data: 2024-04-16T19:45:57.831 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.831 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.831 DEBUG:boto:Params: {} 2024-04-16T19:45:57.832 DEBUG:boto:Token: None 2024-04-16T19:45:57.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3 2024-04-16T19:45:57.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9/EWosPr1PsisLP1T+GwBOtiz4E= 2024-04-16T19:45:57.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9/EWosPr1PsisLP1T+GwBOtiz4E=', 'Content-Length': '0'} 2024-04-16T19:45:57.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c82727a3649d7d8e-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.834 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.834 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.834 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.834 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:45:57.834 DEBUG:boto:Data: 2024-04-16T19:45:57.834 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.835 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.835 DEBUG:boto:Params: {} 2024-04-16T19:45:57.835 DEBUG:boto:Token: None 2024-04-16T19:45:57.835 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3 2024-04-16T19:45:57.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9/EWosPr1PsisLP1T+GwBOtiz4E= 2024-04-16T19:45:57.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9/EWosPr1PsisLP1T+GwBOtiz4E=', 'Content-Length': '0'} 2024-04-16T19:45:57.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac183a1e24c0e462-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:57.838 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.838 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.838 DEBUG:boto:Method: GET 2024-04-16T19:45:57.838 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:45:57.838 DEBUG:boto:Data: 2024-04-16T19:45:57.838 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.838 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.838 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.838 DEBUG:boto:Params: {} 2024-04-16T19:45:57.839 DEBUG:boto:Token: None 2024-04-16T19:45:57.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3 2024-04-16T19:45:57.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3zWYinbLWqzdIlD1ZDTPjrDw5qE= 2024-04-16T19:45:57.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3zWYinbLWqzdIlD1ZDTPjrDw5qE=', 'Content-Length': '0'} 2024-04-16T19:45:57.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aad60c90ea68ac4-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.882 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:45:57.882 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:45:57.883 DEBUG:boto:Method: GET 2024-04-16T19:45:57.883 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:45:57.883 DEBUG:boto:Data: 2024-04-16T19:45:57.883 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.883 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.883 DEBUG:boto:Params: {} 2024-04-16T19:45:57.883 DEBUG:boto:Token: None 2024-04-16T19:45:57.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj3 2024-04-16T19:45:57.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3zWYinbLWqzdIlD1ZDTPjrDw5qE= 2024-04-16T19:45:57.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3zWYinbLWqzdIlD1ZDTPjrDw5qE=', 'Content-Length': '0'} 2024-04-16T19:45:57.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a965c5c30ba0ae50-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.886 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.886 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.886 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.886 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.886 DEBUG:boto:Method: GET 2024-04-16T19:45:57.887 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.887 DEBUG:boto:Data: 2024-04-16T19:45:57.887 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.887 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.887 DEBUG:boto:Params: {} 2024-04-16T19:45:57.887 DEBUG:boto:Token: None 2024-04-16T19:45:57.887 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eabcc51bf3721102-00661ed575-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.930 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.931 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:57.931 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:57.931 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.931 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.931 DEBUG:boto:Method: GET 2024-04-16T19:45:57.931 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:45:57.931 DEBUG:boto:Data: 2024-04-16T19:45:57.931 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.932 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.932 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.932 DEBUG:boto:Params: {} 2024-04-16T19:45:57.932 DEBUG:boto:Token: None 2024-04-16T19:45:57.932 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/ 2024-04-16T19:45:57.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY= 2024-04-16T19:45:57.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q7wByNeKzyOr8HDwATk/wpCa2bY=', 'Content-Length': '0'} 2024-04-16T19:45:57.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f5f2750a2cab37-00661ed575-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.935 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:57.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:57.935 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:57.935 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:57.935 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.936 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.936 DEBUG:boto:Method: GET 2024-04-16T19:45:57.936 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.936 DEBUG:boto:Data: 2024-04-16T19:45:57.936 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.936 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.936 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.936 DEBUG:boto:Params: {} 2024-04-16T19:45:57.936 DEBUG:boto:Token: None 2024-04-16T19:45:57.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.937 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU= 2024-04-16T19:45:57.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU=', 'Content-Length': '0'} 2024-04-16T19:45:57.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009559b6b794711c47-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.983 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:57.983 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:57.983 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.983 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.983 DEBUG:boto:Method: GET 2024-04-16T19:45:57.983 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.983 DEBUG:boto:Data: 2024-04-16T19:45:57.983 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.984 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.984 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.984 DEBUG:boto:Params: {} 2024-04-16T19:45:57.984 DEBUG:boto:Token: None 2024-04-16T19:45:57.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU= 2024-04-16T19:45:57.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU=', 'Content-Length': '0'} 2024-04-16T19:45:57.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea9714361f073c5f-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.987 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:57.987 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:57.987 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.987 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.987 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.987 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.987 DEBUG:boto:Data: 2024-04-16T19:45:57.987 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.988 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.988 DEBUG:boto:Params: {} 2024-04-16T19:45:57.988 DEBUG:boto:Token: None 2024-04-16T19:45:57.988 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fbn40+gVCrtYNbIL4FBO+n5pPpk= 2024-04-16T19:45:57.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fbn40+gVCrtYNbIL4FBO+n5pPpk=', 'Content-Length': '0'} 2024-04-16T19:45:57.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8b553764e22e078-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.990 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:57.991 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:57.991 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.991 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.991 DEBUG:boto:Method: HEAD 2024-04-16T19:45:57.991 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.991 DEBUG:boto:Data: 2024-04-16T19:45:57.991 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:57.991 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.991 DEBUG:boto:Params: {} 2024-04-16T19:45:57.992 DEBUG:boto:Token: None 2024-04-16T19:45:57.992 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fbn40+gVCrtYNbIL4FBO+n5pPpk= 2024-04-16T19:45:57.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fbn40+gVCrtYNbIL4FBO+n5pPpk=', 'Content-Length': '0'} 2024-04-16T19:45:57.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0c6720425b8ca43-00661ed575-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:57.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:57.994 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:57.994 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:57.994 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.995 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.995 DEBUG:boto:Method: GET 2024-04-16T19:45:57.995 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.995 DEBUG:boto:Data: 2024-04-16T19:45:57.995 DEBUG:boto:Headers: {} 2024-04-16T19:45:57.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:57.995 DEBUG:boto:Port: 8000 2024-04-16T19:45:57.995 DEBUG:boto:Params: {} 2024-04-16T19:45:57.995 DEBUG:boto:Token: None 2024-04-16T19:45:57.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:57 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:57.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU= 2024-04-16T19:45:57.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jzT9zc78WqNlaEi3qYcrSAzJvFU=', 'Content-Length': '0'} 2024-04-16T19:45:57.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085ae27f9c2ce8f48-00661ed575-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.039 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:58.039 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:58.039 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:58.039 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:45:58.039 DEBUG:boto:Method: GET 2024-04-16T19:45:58.039 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:58.039 DEBUG:boto:Data: 2024-04-16T19:45:58.039 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.040 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.040 DEBUG:boto:Params: {} 2024-04-16T19:45:58.040 DEBUG:boto:Token: None 2024-04-16T19:45:58.040 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:45:58.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4nHgqcHDvp7A2OnSoWX68XZapSY= 2024-04-16T19:45:58.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4nHgqcHDvp7A2OnSoWX68XZapSY=', 'Content-Length': '0'} 2024-04-16T19:45:58.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028987e3fc4afa9ca-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.043 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:58.043 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:58.043 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.043 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.043 DEBUG:boto:Method: GET 2024-04-16T19:45:58.043 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.043 DEBUG:boto:Data: 2024-04-16T19:45:58.044 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.044 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.044 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.044 DEBUG:boto:Params: {} 2024-04-16T19:45:58.044 DEBUG:boto:Token: None 2024-04-16T19:45:58.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/ 2024-04-16T19:45:58.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U= 2024-04-16T19:45:58.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U=', 'Content-Length': '0'} 2024-04-16T19:45:58.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f616ce99ce545513-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.090 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.091 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:58.091 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:58.091 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.091 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.092 DEBUG:boto:Method: GET 2024-04-16T19:45:58.092 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.092 DEBUG:boto:Data: 2024-04-16T19:45:58.092 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.092 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.092 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.092 DEBUG:boto:Params: {} 2024-04-16T19:45:58.092 DEBUG:boto:Token: None 2024-04-16T19:45:58.093 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/ 2024-04-16T19:45:58.093 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U= 2024-04-16T19:45:58.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U=', 'Content-Length': '0'} 2024-04-16T19:45:58.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48d3af0f60029a5-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.095 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.096 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:58.096 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:58.096 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.096 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:45:58.096 DEBUG:boto:Data: 2024-04-16T19:45:58.096 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.097 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.097 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.097 DEBUG:boto:Params: {} 2024-04-16T19:45:58.097 DEBUG:boto:Token: None 2024-04-16T19:45:58.097 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/obj4 2024-04-16T19:45:58.097 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2aDDwFC37VmRF6sRaRT0OzcW4cE= 2024-04-16T19:45:58.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2aDDwFC37VmRF6sRaRT0OzcW4cE=', 'Content-Length': '0'} 2024-04-16T19:45:58.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dde9016a611dc59-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.099 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:58.099 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:58.100 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.100 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:45:58.100 DEBUG:boto:Data: 2024-04-16T19:45:58.100 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.100 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.100 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.100 DEBUG:boto:Params: {} 2024-04-16T19:45:58.100 DEBUG:boto:Token: None 2024-04-16T19:45:58.101 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/obj4 2024-04-16T19:45:58.101 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2aDDwFC37VmRF6sRaRT0OzcW4cE= 2024-04-16T19:45:58.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2aDDwFC37VmRF6sRaRT0OzcW4cE=', 'Content-Length': '0'} 2024-04-16T19:45:58.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002da64708486a526e-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:58.104 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:58.104 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:58.104 DEBUG:boto:Method: GET 2024-04-16T19:45:58.104 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:45:58.104 DEBUG:boto:Data: 2024-04-16T19:45:58.104 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.104 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.104 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.104 DEBUG:boto:Params: {} 2024-04-16T19:45:58.105 DEBUG:boto:Token: None 2024-04-16T19:45:58.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/obj4 2024-04-16T19:45:58.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Amv8xbu2ZqcC0Kww4dmaXCJU8AY= 2024-04-16T19:45:58.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Amv8xbu2ZqcC0Kww4dmaXCJU8AY=', 'Content-Length': '0'} 2024-04-16T19:45:58.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e47f02f90b66bc6-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.150 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:45:58.151 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:45:58.151 DEBUG:boto:Method: GET 2024-04-16T19:45:58.151 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:45:58.151 DEBUG:boto:Data: 2024-04-16T19:45:58.151 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.151 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.151 DEBUG:boto:Params: {} 2024-04-16T19:45:58.152 DEBUG:boto:Token: None 2024-04-16T19:45:58.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/obj4 2024-04-16T19:45:58.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Amv8xbu2ZqcC0Kww4dmaXCJU8AY= 2024-04-16T19:45:58.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Amv8xbu2ZqcC0Kww4dmaXCJU8AY=', 'Content-Length': '0'} 2024-04-16T19:45:58.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b431beec0b16ba3d-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.155 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:58.155 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:58.155 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.155 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.155 DEBUG:boto:Method: GET 2024-04-16T19:45:58.155 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.156 DEBUG:boto:Data: 2024-04-16T19:45:58.156 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.156 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.156 DEBUG:boto:Params: {} 2024-04-16T19:45:58.156 DEBUG:boto:Token: None 2024-04-16T19:45:58.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/ 2024-04-16T19:45:58.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U= 2024-04-16T19:45:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U=', 'Content-Length': '0'} 2024-04-16T19:45:58.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e15876936858819-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.202 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.203 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:58.203 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:58.203 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.203 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.204 DEBUG:boto:Method: GET 2024-04-16T19:45:58.204 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:45:58.204 DEBUG:boto:Data: 2024-04-16T19:45:58.204 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.204 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.204 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.204 DEBUG:boto:Params: {} 2024-04-16T19:45:58.204 DEBUG:boto:Token: None 2024-04-16T19:45:58.205 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-38/ 2024-04-16T19:45:58.205 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U= 2024-04-16T19:45:58.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x7B73LquYtETgvH7MpJISxQ9o/U=', 'Content-Length': '0'} 2024-04-16T19:45:58.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d573a5fee162e425-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.208 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.208 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-38 zones={test-zone2, test-zone1} 2024-04-16T19:45:58.208 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-39 2024-04-16T19:45:58.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:58.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:58.209 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-39 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:45:58.265 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:58.266 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:58.319+0000 7f8914d1f800 1 bucket sync caught up with source: 2024-04-16T19:45:58.321 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.30.6, , 00000000001.24.6, , 00000000001.16.6, , , 00000000001.38.6, , , ] 2024-04-16T19:45:58.321 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.30.6, , 00000000001.24.6, , 00000000001.16.6, , , 00000000001.38.6, , , ] 2024-04-16T19:45:58.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:45:58.319+0000 7f8914d1f800 0 bucket checkpoint complete 2024-04-16T19:45:58.341 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-39 2024-04-16T19:45:58.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:58.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:58.341 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-39 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:45:58.404 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:58.404 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:58.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:45:58.465+0000 7f7e032e4800 1 bucket sync caught up with source: 2024-04-16T19:45:58.466 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.61.6, , 00000000001.118.6, , 00000000001.102.6, , , 00000000001.140.6, , , ] 2024-04-16T19:45:58.467 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.61.6, , 00000000001.118.6, , 00000000001.102.6, , , 00000000001.140.6, , , ] 2024-04-16T19:45:58.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:45:58.465+0000 7f7e032e4800 0 bucket checkpoint complete 2024-04-16T19:45:58.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-39 zones={test-zone2, test-zone1} 2024-04-16T19:45:58.488 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.488 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.488 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.488 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:58.488 DEBUG:boto:Data: 2024-04-16T19:45:58.489 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.489 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.489 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.489 DEBUG:boto:Params: {} 2024-04-16T19:45:58.489 DEBUG:boto:Token: None 2024-04-16T19:45:58.489 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Duajpq7pj5Gg/rDRG502VS/zOG4= 2024-04-16T19:45:58.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Duajpq7pj5Gg/rDRG502VS/zOG4=', 'Content-Length': '0'} 2024-04-16T19:45:58.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012282297494dda91-00661ed576-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.492 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.492 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.492 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.492 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:58.492 DEBUG:boto:Data: 2024-04-16T19:45:58.493 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.493 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.493 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.493 DEBUG:boto:Params: {} 2024-04-16T19:45:58.493 DEBUG:boto:Token: None 2024-04-16T19:45:58.493 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.493 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Duajpq7pj5Gg/rDRG502VS/zOG4= 2024-04-16T19:45:58.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Duajpq7pj5Gg/rDRG502VS/zOG4=', 'Content-Length': '0'} 2024-04-16T19:45:58.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fe36f9d19dc3e63-00661ed576-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:45:58.496 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.497 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.497 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.497 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.497 DEBUG:boto:Method: GET 2024-04-16T19:45:58.497 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.497 DEBUG:boto:Data: 2024-04-16T19:45:58.497 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.497 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.497 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.498 DEBUG:boto:Params: {} 2024-04-16T19:45:58.498 DEBUG:boto:Token: None 2024-04-16T19:45:58.498 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/?versions 2024-04-16T19:45:58.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y= 2024-04-16T19:45:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y=', 'Content-Length': '0'} 2024-04-16T19:45:58.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c656ebb74fc223d4-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.501 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:45:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:45:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:45:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:45:58.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:45:58.502 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.502 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.503 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.503 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.503 DEBUG:boto:Method: GET 2024-04-16T19:45:58.503 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.503 DEBUG:boto:Data: 2024-04-16T19:45:58.503 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.503 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.503 DEBUG:boto:Params: {} 2024-04-16T19:45:58.503 DEBUG:boto:Token: None 2024-04-16T19:45:58.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/?versions 2024-04-16T19:45:58.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y= 2024-04-16T19:45:58.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y=', 'Content-Length': '0'} 2024-04-16T19:45:58.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0a8bc46c945dd9-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.507 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:45:58.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:45:58.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:45:58.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:45:58.508 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.508 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.508 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.508 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.508 DEBUG:boto:Method: GET 2024-04-16T19:45:58.508 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.508 DEBUG:boto:Data: 2024-04-16T19:45:58.508 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.509 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.509 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.509 DEBUG:boto:Params: {} 2024-04-16T19:45:58.509 DEBUG:boto:Token: None 2024-04-16T19:45:58.509 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/?versions 2024-04-16T19:45:58.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y= 2024-04-16T19:45:58.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y=', 'Content-Length': '0'} 2024-04-16T19:45:58.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc3c401fde40dbb-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.512 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.513 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.513 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.513 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.513 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.513 DEBUG:boto:Method: GET 2024-04-16T19:45:58.513 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:45:58.513 DEBUG:boto:Data: 2024-04-16T19:45:58.513 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.513 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.514 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.514 DEBUG:boto:Params: {} 2024-04-16T19:45:58.514 DEBUG:boto:Token: None 2024-04-16T19:45:58.514 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/?versions 2024-04-16T19:45:58.514 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y= 2024-04-16T19:45:58.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aT60BxS8KM2jFl4DWfkpaN+QC0Y=', 'Content-Length': '0'} 2024-04-16T19:45:58.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005909b484342f5dba-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.517 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:58.518 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.518 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.518 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.518 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.518 DEBUG:boto:Method: GET 2024-04-16T19:45:58.518 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.519 DEBUG:boto:Data: 2024-04-16T19:45:58.519 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.519 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.519 DEBUG:boto:Params: {} 2024-04-16T19:45:58.519 DEBUG:boto:Token: None 2024-04-16T19:45:58.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8= 2024-04-16T19:45:58.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8=', 'Content-Length': '0'} 2024-04-16T19:45:58.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000262b138f9376c06d-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.522 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.522 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.522 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.522 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.523 DEBUG:boto:Method: GET 2024-04-16T19:45:58.523 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.523 DEBUG:boto:Data: 2024-04-16T19:45:58.523 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.523 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.523 DEBUG:boto:Params: {} 2024-04-16T19:45:58.523 DEBUG:boto:Token: None 2024-04-16T19:45:58.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8= 2024-04-16T19:45:58.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8=', 'Content-Length': '0'} 2024-04-16T19:45:58.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c838b350e450a0d-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.566 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.567 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.567 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.567 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.567 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.567 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.567 DEBUG:boto:Data: 2024-04-16T19:45:58.567 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.568 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.568 DEBUG:boto:Params: {} 2024-04-16T19:45:58.568 DEBUG:boto:Token: None 2024-04-16T19:45:58.568 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mN/LfoLcPQEqJ+2k8rz1RSrADqk= 2024-04-16T19:45:58.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mN/LfoLcPQEqJ+2k8rz1RSrADqk=', 'Content-Length': '0'} 2024-04-16T19:45:58.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003f867eec2fd5c29-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.571 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.571 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.571 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.572 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.572 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.572 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.572 DEBUG:boto:Data: 2024-04-16T19:45:58.572 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.572 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.572 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.572 DEBUG:boto:Params: {} 2024-04-16T19:45:58.573 DEBUG:boto:Token: None 2024-04-16T19:45:58.573 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.573 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mN/LfoLcPQEqJ+2k8rz1RSrADqk= 2024-04-16T19:45:58.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mN/LfoLcPQEqJ+2k8rz1RSrADqk=', 'Content-Length': '0'} 2024-04-16T19:45:58.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000388720467c942528-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:58.576 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.576 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.576 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.576 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.576 DEBUG:boto:Method: GET 2024-04-16T19:45:58.577 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.577 DEBUG:boto:Data: 2024-04-16T19:45:58.577 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.577 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.577 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.577 DEBUG:boto:Params: {} 2024-04-16T19:45:58.577 DEBUG:boto:Token: None 2024-04-16T19:45:58.577 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8= 2024-04-16T19:45:58.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8=', 'Content-Length': '0'} 2024-04-16T19:45:58.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ec1beb3689851c4-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.622 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.623 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.623 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.623 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.623 DEBUG:boto:Method: GET 2024-04-16T19:45:58.623 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.623 DEBUG:boto:Data: 2024-04-16T19:45:58.623 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.624 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.624 DEBUG:boto:Params: {} 2024-04-16T19:45:58.624 DEBUG:boto:Token: None 2024-04-16T19:45:58.624 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:45:58.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8= 2024-04-16T19:45:58.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qboPRoF/mUU/LmneXVMpWxYq3Z8=', 'Content-Length': '0'} 2024-04-16T19:45:58.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005134bbc77556b80f-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.627 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.627 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.627 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.627 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.627 DEBUG:boto:Method: GET 2024-04-16T19:45:58.627 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.627 DEBUG:boto:Data: 2024-04-16T19:45:58.628 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.628 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.628 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.628 DEBUG:boto:Params: {} 2024-04-16T19:45:58.628 DEBUG:boto:Token: None 2024-04-16T19:45:58.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002166db42c9ef0e47-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.634 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.635 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.635 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.635 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.635 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.635 DEBUG:boto:Method: GET 2024-04-16T19:45:58.635 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.636 DEBUG:boto:Data: 2024-04-16T19:45:58.636 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.636 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.636 DEBUG:boto:Params: {} 2024-04-16T19:45:58.636 DEBUG:boto:Token: None 2024-04-16T19:45:58.636 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c294f7ea79f0430-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.640 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.640 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.640 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.640 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.641 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:45:58.641 DEBUG:boto:Data: 2024-04-16T19:45:58.641 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.641 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.641 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.641 DEBUG:boto:Params: {} 2024-04-16T19:45:58.641 DEBUG:boto:Token: None 2024-04-16T19:45:58.641 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1 2024-04-16T19:45:58.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GimtSDf8GD6GKtbWB1WOD95v9e8= 2024-04-16T19:45:58.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GimtSDf8GD6GKtbWB1WOD95v9e8=', 'Content-Length': '0'} 2024-04-16T19:45:58.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013db518cf8210fc6-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.644 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.644 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.644 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.644 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:45:58.645 DEBUG:boto:Data: 2024-04-16T19:45:58.645 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.645 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.645 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.645 DEBUG:boto:Params: {} 2024-04-16T19:45:58.646 DEBUG:boto:Token: None 2024-04-16T19:45:58.646 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1 2024-04-16T19:45:58.646 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GimtSDf8GD6GKtbWB1WOD95v9e8= 2024-04-16T19:45:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GimtSDf8GD6GKtbWB1WOD95v9e8=', 'Content-Length': '0'} 2024-04-16T19:45:58.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc28b9363a8f3b72-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:45:58.649 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.649 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.649 DEBUG:boto:Method: GET 2024-04-16T19:45:58.649 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:45:58.650 DEBUG:boto:Data: 2024-04-16T19:45:58.650 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.650 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.650 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.650 DEBUG:boto:Params: {} 2024-04-16T19:45:58.650 DEBUG:boto:Token: None 2024-04-16T19:45:58.650 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1 2024-04-16T19:45:58.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qOiUKC3NMqhp9kWpaVmyTSz/bQ= 2024-04-16T19:45:58.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qOiUKC3NMqhp9kWpaVmyTSz/bQ=', 'Content-Length': '0'} 2024-04-16T19:45:58.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edfa46f85cad7392-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.653 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:45:58.654 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:45:58.654 DEBUG:boto:Method: GET 2024-04-16T19:45:58.654 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:45:58.654 DEBUG:boto:Data: 2024-04-16T19:45:58.654 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.654 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.655 DEBUG:boto:Params: {} 2024-04-16T19:45:58.655 DEBUG:boto:Token: None 2024-04-16T19:45:58.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj1 2024-04-16T19:45:58.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qOiUKC3NMqhp9kWpaVmyTSz/bQ= 2024-04-16T19:45:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qOiUKC3NMqhp9kWpaVmyTSz/bQ=', 'Content-Length': '0'} 2024-04-16T19:45:58.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026dd5f5803c84299-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.699 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.699 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.699 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.699 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.699 DEBUG:boto:Method: GET 2024-04-16T19:45:58.699 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.699 DEBUG:boto:Data: 2024-04-16T19:45:58.699 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.700 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.700 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.700 DEBUG:boto:Params: {} 2024-04-16T19:45:58.700 DEBUG:boto:Token: None 2024-04-16T19:45:58.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba6d1dedba13941-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.703 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.703 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.703 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.704 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.704 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.704 DEBUG:boto:Method: GET 2024-04-16T19:45:58.704 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:45:58.704 DEBUG:boto:Data: 2024-04-16T19:45:58.704 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.704 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.704 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.704 DEBUG:boto:Params: {} 2024-04-16T19:45:58.705 DEBUG:boto:Token: None 2024-04-16T19:45:58.705 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075dae88d164920e3-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.707 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:58.708 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.708 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.708 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.708 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.709 DEBUG:boto:Method: GET 2024-04-16T19:45:58.709 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.709 DEBUG:boto:Data: 2024-04-16T19:45:58.709 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.709 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.709 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.709 DEBUG:boto:Params: {} 2024-04-16T19:45:58.709 DEBUG:boto:Token: None 2024-04-16T19:45:58.709 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.710 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs= 2024-04-16T19:45:58.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs=', 'Content-Length': '0'} 2024-04-16T19:45:58.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5b1b99c578eb06-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.755 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.755 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.755 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.755 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.755 DEBUG:boto:Method: GET 2024-04-16T19:45:58.755 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.755 DEBUG:boto:Data: 2024-04-16T19:45:58.755 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.756 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.756 DEBUG:boto:Params: {} 2024-04-16T19:45:58.756 DEBUG:boto:Token: None 2024-04-16T19:45:58.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs= 2024-04-16T19:45:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs=', 'Content-Length': '0'} 2024-04-16T19:45:58.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085bfa3ef3a4b04d1-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.798 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.799 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.799 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.799 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.799 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.799 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.799 DEBUG:boto:Data: 2024-04-16T19:45:58.799 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.800 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.800 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.800 DEBUG:boto:Params: {} 2024-04-16T19:45:58.800 DEBUG:boto:Token: None 2024-04-16T19:45:58.800 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DKIYxBvDoO18SUgYU/lkT3wC400= 2024-04-16T19:45:58.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DKIYxBvDoO18SUgYU/lkT3wC400=', 'Content-Length': '0'} 2024-04-16T19:45:58.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d3c6a483ddbed39-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.803 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.803 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.803 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.803 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.803 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.803 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.804 DEBUG:boto:Data: 2024-04-16T19:45:58.804 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.804 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.804 DEBUG:boto:Params: {} 2024-04-16T19:45:58.804 DEBUG:boto:Token: None 2024-04-16T19:45:58.804 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DKIYxBvDoO18SUgYU/lkT3wC400= 2024-04-16T19:45:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DKIYxBvDoO18SUgYU/lkT3wC400=', 'Content-Length': '0'} 2024-04-16T19:45:58.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b4e6fca67e34a78-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:58.807 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.807 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.807 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.808 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.808 DEBUG:boto:Method: GET 2024-04-16T19:45:58.808 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.808 DEBUG:boto:Data: 2024-04-16T19:45:58.808 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.808 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.808 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.808 DEBUG:boto:Params: {} 2024-04-16T19:45:58.809 DEBUG:boto:Token: None 2024-04-16T19:45:58.809 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.809 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs= 2024-04-16T19:45:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs=', 'Content-Length': '0'} 2024-04-16T19:45:58.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c29d731ad36d2d45-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.811 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.811 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.812 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.812 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.812 DEBUG:boto:Method: GET 2024-04-16T19:45:58.812 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.812 DEBUG:boto:Data: 2024-04-16T19:45:58.812 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.812 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.812 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.812 DEBUG:boto:Params: {} 2024-04-16T19:45:58.813 DEBUG:boto:Token: None 2024-04-16T19:45:58.813 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:45:58.813 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs= 2024-04-16T19:45:58.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyMUvjZCDpYnZFbc3X+A8NXqLbs=', 'Content-Length': '0'} 2024-04-16T19:45:58.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ffe093e9e2b0036-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.858 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.859 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.859 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.859 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.859 DEBUG:boto:Method: GET 2024-04-16T19:45:58.859 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.859 DEBUG:boto:Data: 2024-04-16T19:45:58.859 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.860 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.860 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.860 DEBUG:boto:Params: {} 2024-04-16T19:45:58.860 DEBUG:boto:Token: None 2024-04-16T19:45:58.860 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7003b560471f85a-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.863 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.864 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.864 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.864 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.864 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.864 DEBUG:boto:Method: GET 2024-04-16T19:45:58.865 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.865 DEBUG:boto:Data: 2024-04-16T19:45:58.865 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.865 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.865 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.865 DEBUG:boto:Params: {} 2024-04-16T19:45:58.865 DEBUG:boto:Token: None 2024-04-16T19:45:58.865 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.866 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552b853363bf3958-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.869 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.869 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.869 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.869 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.870 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:45:58.870 DEBUG:boto:Data: 2024-04-16T19:45:58.870 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.870 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.870 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.870 DEBUG:boto:Params: {} 2024-04-16T19:45:58.870 DEBUG:boto:Token: None 2024-04-16T19:45:58.870 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2 2024-04-16T19:45:58.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GKqe0JOES+g2cmFVoV8jcx3e9gI= 2024-04-16T19:45:58.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GKqe0JOES+g2cmFVoV8jcx3e9gI=', 'Content-Length': '0'} 2024-04-16T19:45:58.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032b2234a4f8e0a55-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.873 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.873 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.873 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.873 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:45:58.873 DEBUG:boto:Data: 2024-04-16T19:45:58.873 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.873 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.873 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.873 DEBUG:boto:Params: {} 2024-04-16T19:45:58.874 DEBUG:boto:Token: None 2024-04-16T19:45:58.874 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2 2024-04-16T19:45:58.874 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GKqe0JOES+g2cmFVoV8jcx3e9gI= 2024-04-16T19:45:58.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GKqe0JOES+g2cmFVoV8jcx3e9gI=', 'Content-Length': '0'} 2024-04-16T19:45:58.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20684f5ebef66f8-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:45:58.876 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.877 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.877 DEBUG:boto:Method: GET 2024-04-16T19:45:58.877 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:45:58.877 DEBUG:boto:Data: 2024-04-16T19:45:58.877 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.877 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.877 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.877 DEBUG:boto:Params: {} 2024-04-16T19:45:58.878 DEBUG:boto:Token: None 2024-04-16T19:45:58.878 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2 2024-04-16T19:45:58.878 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wA86f1LW7KoTRIs2QurFFdWjIDc= 2024-04-16T19:45:58.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wA86f1LW7KoTRIs2QurFFdWjIDc=', 'Content-Length': '0'} 2024-04-16T19:45:58.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d99c4c3c62346eb0-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.880 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:45:58.880 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:45:58.880 DEBUG:boto:Method: GET 2024-04-16T19:45:58.881 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:45:58.881 DEBUG:boto:Data: 2024-04-16T19:45:58.881 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.881 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.881 DEBUG:boto:Params: {} 2024-04-16T19:45:58.881 DEBUG:boto:Token: None 2024-04-16T19:45:58.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj2 2024-04-16T19:45:58.882 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wA86f1LW7KoTRIs2QurFFdWjIDc= 2024-04-16T19:45:58.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wA86f1LW7KoTRIs2QurFFdWjIDc=', 'Content-Length': '0'} 2024-04-16T19:45:58.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e165fd96585caf2-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.884 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.884 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.884 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.884 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.884 DEBUG:boto:Method: GET 2024-04-16T19:45:58.884 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.884 DEBUG:boto:Data: 2024-04-16T19:45:58.885 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.885 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.885 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.885 DEBUG:boto:Params: {} 2024-04-16T19:45:58.885 DEBUG:boto:Token: None 2024-04-16T19:45:58.885 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.885 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5c855854fe48a1-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.888 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.888 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.888 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.889 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.889 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.889 DEBUG:boto:Method: GET 2024-04-16T19:45:58.889 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:45:58.889 DEBUG:boto:Data: 2024-04-16T19:45:58.889 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.889 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.889 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.889 DEBUG:boto:Params: {} 2024-04-16T19:45:58.889 DEBUG:boto:Token: None 2024-04-16T19:45:58.890 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.890 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000028487b98bc36c5a-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.893 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:58.893 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.893 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.893 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.893 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.894 DEBUG:boto:Method: GET 2024-04-16T19:45:58.894 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.894 DEBUG:boto:Data: 2024-04-16T19:45:58.894 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.894 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.894 DEBUG:boto:Params: {} 2024-04-16T19:45:58.894 DEBUG:boto:Token: None 2024-04-16T19:45:58.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4= 2024-04-16T19:45:58.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4=', 'Content-Length': '0'} 2024-04-16T19:45:58.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9de59cae23b911c-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.938 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.939 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.939 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.939 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.939 DEBUG:boto:Method: GET 2024-04-16T19:45:58.939 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.939 DEBUG:boto:Data: 2024-04-16T19:45:58.939 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.940 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.940 DEBUG:boto:Params: {} 2024-04-16T19:45:58.940 DEBUG:boto:Token: None 2024-04-16T19:45:58.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4= 2024-04-16T19:45:58.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4=', 'Content-Length': '0'} 2024-04-16T19:45:58.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6c087713920716-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.942 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.943 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.943 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.943 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.943 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.943 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.943 DEBUG:boto:Data: 2024-04-16T19:45:58.943 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.943 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.944 DEBUG:boto:Params: {} 2024-04-16T19:45:58.944 DEBUG:boto:Token: None 2024-04-16T19:45:58.944 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6/CxLEZQ3RdlyMToj715/CEvWNk= 2024-04-16T19:45:58.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6/CxLEZQ3RdlyMToj715/CEvWNk=', 'Content-Length': '0'} 2024-04-16T19:45:58.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e97b055f71532dbb-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.946 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.946 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.947 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.947 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.947 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.947 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.947 DEBUG:boto:Data: 2024-04-16T19:45:58.947 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.947 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.947 DEBUG:boto:Params: {} 2024-04-16T19:45:58.948 DEBUG:boto:Token: None 2024-04-16T19:45:58.948 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6/CxLEZQ3RdlyMToj715/CEvWNk= 2024-04-16T19:45:58.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6/CxLEZQ3RdlyMToj715/CEvWNk=', 'Content-Length': '0'} 2024-04-16T19:45:58.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cacc71e13d626e8b-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:58.950 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.950 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.951 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.951 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.951 DEBUG:boto:Method: GET 2024-04-16T19:45:58.951 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.951 DEBUG:boto:Data: 2024-04-16T19:45:58.951 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.951 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.951 DEBUG:boto:Params: {} 2024-04-16T19:45:58.951 DEBUG:boto:Token: None 2024-04-16T19:45:58.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4= 2024-04-16T19:45:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4=', 'Content-Length': '0'} 2024-04-16T19:45:58.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd7ef585be595a4c-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.954 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.954 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.954 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.954 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.955 DEBUG:boto:Method: GET 2024-04-16T19:45:58.955 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.955 DEBUG:boto:Data: 2024-04-16T19:45:58.955 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.955 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.955 DEBUG:boto:Params: {} 2024-04-16T19:45:58.955 DEBUG:boto:Token: None 2024-04-16T19:45:58.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:45:58.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4= 2024-04-16T19:45:58.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B7Q7065rePrvOydjGzwivr2bxE4=', 'Content-Length': '0'} 2024-04-16T19:45:58.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009c326c92f425cc3-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.958 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.958 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.958 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.958 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.958 DEBUG:boto:Method: GET 2024-04-16T19:45:58.959 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.959 DEBUG:boto:Data: 2024-04-16T19:45:58.959 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.959 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.959 DEBUG:boto:Params: {} 2024-04-16T19:45:58.959 DEBUG:boto:Token: None 2024-04-16T19:45:58.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76fe252e72b420d-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.962 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.963 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.963 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.963 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.963 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.963 DEBUG:boto:Method: GET 2024-04-16T19:45:58.963 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.963 DEBUG:boto:Data: 2024-04-16T19:45:58.963 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.964 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.964 DEBUG:boto:Params: {} 2024-04-16T19:45:58.964 DEBUG:boto:Token: None 2024-04-16T19:45:58.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987639151164a6ff-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.967 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.968 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.968 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.968 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.968 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:45:58.968 DEBUG:boto:Data: 2024-04-16T19:45:58.968 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.968 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.968 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.968 DEBUG:boto:Params: {} 2024-04-16T19:45:58.968 DEBUG:boto:Token: None 2024-04-16T19:45:58.969 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3 2024-04-16T19:45:58.969 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oboLZww/HTGJuYuMe7jVWydD5S0= 2024-04-16T19:45:58.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oboLZww/HTGJuYuMe7jVWydD5S0=', 'Content-Length': '0'} 2024-04-16T19:45:58.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006145ea72e531928-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.971 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.971 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.971 DEBUG:boto:Method: HEAD 2024-04-16T19:45:58.971 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:45:58.971 DEBUG:boto:Data: 2024-04-16T19:45:58.971 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.972 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.972 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.972 DEBUG:boto:Params: {} 2024-04-16T19:45:58.972 DEBUG:boto:Token: None 2024-04-16T19:45:58.972 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3 2024-04-16T19:45:58.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oboLZww/HTGJuYuMe7jVWydD5S0= 2024-04-16T19:45:58.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oboLZww/HTGJuYuMe7jVWydD5S0=', 'Content-Length': '0'} 2024-04-16T19:45:58.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7674fc9e1b437b4-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:45:58.975 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.975 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.975 DEBUG:boto:Method: GET 2024-04-16T19:45:58.975 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:45:58.975 DEBUG:boto:Data: 2024-04-16T19:45:58.975 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.975 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.975 DEBUG:boto:Params: {} 2024-04-16T19:45:58.976 DEBUG:boto:Token: None 2024-04-16T19:45:58.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3 2024-04-16T19:45:58.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0chiptOB9xsLPptt65995WHqzFs= 2024-04-16T19:45:58.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0chiptOB9xsLPptt65995WHqzFs=', 'Content-Length': '0'} 2024-04-16T19:45:58.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b8dda2b6da7adc3-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.978 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:45:58.978 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:45:58.978 DEBUG:boto:Method: GET 2024-04-16T19:45:58.978 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:45:58.978 DEBUG:boto:Data: 2024-04-16T19:45:58.978 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.979 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.979 DEBUG:boto:Params: {} 2024-04-16T19:45:58.979 DEBUG:boto:Token: None 2024-04-16T19:45:58.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj3 2024-04-16T19:45:58.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0chiptOB9xsLPptt65995WHqzFs= 2024-04-16T19:45:58.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0chiptOB9xsLPptt65995WHqzFs=', 'Content-Length': '0'} 2024-04-16T19:45:58.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1299386bc86210f-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.981 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.981 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.982 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.982 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.982 DEBUG:boto:Method: GET 2024-04-16T19:45:58.982 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.982 DEBUG:boto:Data: 2024-04-16T19:45:58.982 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.982 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.982 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.982 DEBUG:boto:Params: {} 2024-04-16T19:45:58.983 DEBUG:boto:Token: None 2024-04-16T19:45:58.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dfea7207b2357e7-00661ed576-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.985 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.986 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:58.986 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:58.986 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.986 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.986 DEBUG:boto:Method: GET 2024-04-16T19:45:58.987 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:45:58.987 DEBUG:boto:Data: 2024-04-16T19:45:58.987 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.987 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.987 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.987 DEBUG:boto:Params: {} 2024-04-16T19:45:58.987 DEBUG:boto:Token: None 2024-04-16T19:45:58.987 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/ 2024-04-16T19:45:58.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU= 2024-04-16T19:45:58.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81L2PiHKxXd16RZDnJIoOK/xdEU=', 'Content-Length': '0'} 2024-04-16T19:45:58.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c5728282ad8cba-00661ed576-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.990 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:58.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:58.991 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:58.991 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:58.991 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.991 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.991 DEBUG:boto:Method: GET 2024-04-16T19:45:58.991 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.991 DEBUG:boto:Data: 2024-04-16T19:45:58.992 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.992 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:58.992 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.992 DEBUG:boto:Params: {} 2024-04-16T19:45:58.992 DEBUG:boto:Token: None 2024-04-16T19:45:58.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3a09JSlUBDrnrJt4AJuV41LwFew= 2024-04-16T19:45:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3a09JSlUBDrnrJt4AJuV41LwFew=', 'Content-Length': '0'} 2024-04-16T19:45:58.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013f8764dea5c72ae-00661ed576-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:58.995 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:58.995 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:58.995 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.995 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.995 DEBUG:boto:Method: GET 2024-04-16T19:45:58.995 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.996 DEBUG:boto:Data: 2024-04-16T19:45:58.996 DEBUG:boto:Headers: {} 2024-04-16T19:45:58.996 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:58.996 DEBUG:boto:Port: 8000 2024-04-16T19:45:58.996 DEBUG:boto:Params: {} 2024-04-16T19:45:58.996 DEBUG:boto:Token: None 2024-04-16T19:45:58.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:58 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:58.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3a09JSlUBDrnrJt4AJuV41LwFew= 2024-04-16T19:45:58.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3a09JSlUBDrnrJt4AJuV41LwFew=', 'Content-Length': '0'} 2024-04-16T19:45:58.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eab56988ad751dd7-00661ed576-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.042 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.043 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.043 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.043 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.043 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.043 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.043 DEBUG:boto:Data: 2024-04-16T19:45:59.043 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.043 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.044 DEBUG:boto:Params: {} 2024-04-16T19:45:59.044 DEBUG:boto:Token: None 2024-04-16T19:45:59.044 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrSnkYYvMZwCzHN6G+mnee0hKS8= 2024-04-16T19:45:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrSnkYYvMZwCzHN6G+mnee0hKS8=', 'Content-Length': '0'} 2024-04-16T19:45:59.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f956875b5ac34b-00661ed577-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.046 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.047 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.047 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.047 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.047 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.047 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.047 DEBUG:boto:Data: 2024-04-16T19:45:59.047 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.047 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.047 DEBUG:boto:Params: {} 2024-04-16T19:45:59.048 DEBUG:boto:Token: None 2024-04-16T19:45:59.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrSnkYYvMZwCzHN6G+mnee0hKS8= 2024-04-16T19:45:59.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrSnkYYvMZwCzHN6G+mnee0hKS8=', 'Content-Length': '0'} 2024-04-16T19:45:59.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d92968c2f69da82-00661ed577-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:59.051 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.051 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.051 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.051 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.051 DEBUG:boto:Method: GET 2024-04-16T19:45:59.051 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.051 DEBUG:boto:Data: 2024-04-16T19:45:59.051 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.051 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.052 DEBUG:boto:Params: {} 2024-04-16T19:45:59.052 DEBUG:boto:Token: None 2024-04-16T19:45:59.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU2HWJOnoamIge8wSV40BAD5A4A= 2024-04-16T19:45:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU2HWJOnoamIge8wSV40BAD5A4A=', 'Content-Length': '0'} 2024-04-16T19:45:59.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b96661a7fb93fdec-00661ed577-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.054 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.055 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.055 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.055 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.055 DEBUG:boto:Method: GET 2024-04-16T19:45:59.055 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.055 DEBUG:boto:Data: 2024-04-16T19:45:59.055 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.055 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.055 DEBUG:boto:Params: {} 2024-04-16T19:45:59.056 DEBUG:boto:Token: None 2024-04-16T19:45:59.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:45:59.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU2HWJOnoamIge8wSV40BAD5A4A= 2024-04-16T19:45:59.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU2HWJOnoamIge8wSV40BAD5A4A=', 'Content-Length': '0'} 2024-04-16T19:45:59.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3fb9ae066febe43-00661ed577-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.058 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.058 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.059 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.059 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.059 DEBUG:boto:Method: GET 2024-04-16T19:45:59.059 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.059 DEBUG:boto:Data: 2024-04-16T19:45:59.059 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.059 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.059 DEBUG:boto:Params: {} 2024-04-16T19:45:59.059 DEBUG:boto:Token: None 2024-04-16T19:45:59.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU= 2024-04-16T19:45:59.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU=', 'Content-Length': '0'} 2024-04-16T19:45:59.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d414c0e3ee1c4b-00661ed577-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.062 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:59.063 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.063 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.063 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.063 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.063 DEBUG:boto:Method: GET 2024-04-16T19:45:59.063 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.063 DEBUG:boto:Data: 2024-04-16T19:45:59.063 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.063 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.064 DEBUG:boto:Params: {} 2024-04-16T19:45:59.064 DEBUG:boto:Token: None 2024-04-16T19:45:59.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU= 2024-04-16T19:45:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU=', 'Content-Length': '0'} 2024-04-16T19:45:59.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b13f76ceb64a6b5-00661ed577-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.067 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:59.067 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.067 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.067 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.068 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:45:59.068 DEBUG:boto:Data: 2024-04-16T19:45:59.068 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.068 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.068 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.068 DEBUG:boto:Params: {} 2024-04-16T19:45:59.068 DEBUG:boto:Token: None 2024-04-16T19:45:59.068 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4 2024-04-16T19:45:59.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GJkUxU/AQPPBxn0g42yhO9zBjbA= 2024-04-16T19:45:59.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJkUxU/AQPPBxn0g42yhO9zBjbA=', 'Content-Length': '0'} 2024-04-16T19:45:59.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aea0d288c63bb6ea-00661ed577-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.071 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.071 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.071 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.071 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:45:59.071 DEBUG:boto:Data: 2024-04-16T19:45:59.072 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.072 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.072 DEBUG:boto:Params: {} 2024-04-16T19:45:59.072 DEBUG:boto:Token: None 2024-04-16T19:45:59.072 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4 2024-04-16T19:45:59.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GJkUxU/AQPPBxn0g42yhO9zBjbA= 2024-04-16T19:45:59.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJkUxU/AQPPBxn0g42yhO9zBjbA=', 'Content-Length': '0'} 2024-04-16T19:45:59.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5fb6d72a3b2f8a2-00661ed577-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:45:59.075 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.075 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.075 DEBUG:boto:Method: GET 2024-04-16T19:45:59.075 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:45:59.075 DEBUG:boto:Data: 2024-04-16T19:45:59.075 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.076 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.076 DEBUG:boto:Params: {} 2024-04-16T19:45:59.076 DEBUG:boto:Token: None 2024-04-16T19:45:59.076 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4 2024-04-16T19:45:59.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdkS1IJstOqnJtX9UjMNRoQMKao= 2024-04-16T19:45:59.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdkS1IJstOqnJtX9UjMNRoQMKao=', 'Content-Length': '0'} 2024-04-16T19:45:59.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca7619a2f5453247-00661ed577-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.119 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:45:59.119 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:45:59.119 DEBUG:boto:Method: GET 2024-04-16T19:45:59.119 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:45:59.119 DEBUG:boto:Data: 2024-04-16T19:45:59.119 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.119 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.119 DEBUG:boto:Params: {} 2024-04-16T19:45:59.120 DEBUG:boto:Token: None 2024-04-16T19:45:59.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj4 2024-04-16T19:45:59.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdkS1IJstOqnJtX9UjMNRoQMKao= 2024-04-16T19:45:59.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdkS1IJstOqnJtX9UjMNRoQMKao=', 'Content-Length': '0'} 2024-04-16T19:45:59.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e3da3a5889109b8-00661ed577-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.123 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.123 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.123 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.123 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.123 DEBUG:boto:Method: GET 2024-04-16T19:45:59.123 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.123 DEBUG:boto:Data: 2024-04-16T19:45:59.123 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.124 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.124 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.124 DEBUG:boto:Params: {} 2024-04-16T19:45:59.124 DEBUG:boto:Token: None 2024-04-16T19:45:59.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU= 2024-04-16T19:45:59.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU=', 'Content-Length': '0'} 2024-04-16T19:45:59.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a9c1b0b731a223-00661ed577-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.127 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:59.127 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.128 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.128 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.128 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.128 DEBUG:boto:Method: GET 2024-04-16T19:45:59.128 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:45:59.128 DEBUG:boto:Data: 2024-04-16T19:45:59.128 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.128 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.128 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.128 DEBUG:boto:Params: {} 2024-04-16T19:45:59.129 DEBUG:boto:Token: None 2024-04-16T19:45:59.129 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU= 2024-04-16T19:45:59.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VHgS7TSOT1zlqytYdCyBYRKr3RU=', 'Content-Length': '0'} 2024-04-16T19:45:59.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074270b618e1dee11-00661ed577-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.174 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:45:59.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-39 zones={test-zone2, test-zone1} 2024-04-16T19:45:59.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-38', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-38', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.175 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-38 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:59.228 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:59.228 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:59.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'yrjefh-38', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'yrjefh-38', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.332 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket yrjefh-38 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:59.381 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:59.381 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:59.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-39', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-39', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.480 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-39 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:59.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'yrjefh-39', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'yrjefh-39', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:45:59.626 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket yrjefh-39 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:45:59.675 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:45:59.675 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:45:59.773 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:59.773 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:59.773 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.773 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:59.773 DEBUG:boto:Data: 2024-04-16T19:45:59.774 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.774 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.774 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.774 DEBUG:boto:Params: {} 2024-04-16T19:45:59.774 DEBUG:boto:Token: None 2024-04-16T19:45:59.774 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/ 2024-04-16T19:45:59.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E= 2024-04-16T19:45:59.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E=', 'Content-Length': '0'} 2024-04-16T19:45:59.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080838c90f6f6e2a1-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.777 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:45:59.777 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:45:59.777 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.778 DEBUG:boto:Path: /yrjefh-38/obj5 2024-04-16T19:45:59.778 DEBUG:boto:Data: 2024-04-16T19:45:59.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.778 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.778 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.778 DEBUG:boto:Params: {} 2024-04-16T19:45:59.778 DEBUG:boto:Token: None 2024-04-16T19:45:59.778 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/obj5 2024-04-16T19:45:59.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8pfzyJAb5vgNGx4aynCA5lVUQp0= 2024-04-16T19:45:59.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8pfzyJAb5vgNGx4aynCA5lVUQp0='} 2024-04-16T19:45:59.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf24279b60bb6a69-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.783 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:59.783 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:59.783 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.784 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:59.784 DEBUG:boto:Data: 2024-04-16T19:45:59.784 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.784 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.784 DEBUG:boto:Params: {} 2024-04-16T19:45:59.784 DEBUG:boto:Token: None 2024-04-16T19:45:59.784 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/ 2024-04-16T19:45:59.785 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E= 2024-04-16T19:45:59.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E=', 'Content-Length': '0'} 2024-04-16T19:45:59.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad07622bcb6161be-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.787 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:45:59.787 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:45:59.787 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.787 DEBUG:boto:Path: /yrjefh-38/obj6 2024-04-16T19:45:59.787 DEBUG:boto:Data: 2024-04-16T19:45:59.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.788 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.788 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.788 DEBUG:boto:Params: {} 2024-04-16T19:45:59.788 DEBUG:boto:Token: None 2024-04-16T19:45:59.788 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/obj6 2024-04-16T19:45:59.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V8Jld/dbCEl/RNzIkJq5wNPop6w= 2024-04-16T19:45:59.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V8Jld/dbCEl/RNzIkJq5wNPop6w='} 2024-04-16T19:45:59.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd4cd9108b4d4147-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.792 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:59.793 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:59.793 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.793 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:59.793 DEBUG:boto:Data: 2024-04-16T19:45:59.793 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.793 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.793 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.793 DEBUG:boto:Params: {} 2024-04-16T19:45:59.793 DEBUG:boto:Token: None 2024-04-16T19:45:59.794 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/ 2024-04-16T19:45:59.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E= 2024-04-16T19:45:59.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E=', 'Content-Length': '0'} 2024-04-16T19:45:59.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009289c4b2db7cd698-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.796 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:45:59.796 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:45:59.796 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.797 DEBUG:boto:Path: /yrjefh-38/obj7 2024-04-16T19:45:59.797 DEBUG:boto:Data: 2024-04-16T19:45:59.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.797 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.797 DEBUG:boto:Params: {} 2024-04-16T19:45:59.797 DEBUG:boto:Token: None 2024-04-16T19:45:59.797 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/obj7 2024-04-16T19:45:59.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IhnreUR0vdUldub66fhGzCs/X4s= 2024-04-16T19:45:59.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IhnreUR0vdUldub66fhGzCs/X4s='} 2024-04-16T19:45:59.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086e98b2b81bbd300-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.843 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:45:59.843 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:45:59.843 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.843 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:45:59.843 DEBUG:boto:Data: 2024-04-16T19:45:59.843 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.843 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.843 DEBUG:boto:Params: {} 2024-04-16T19:45:59.844 DEBUG:boto:Token: None 2024-04-16T19:45:59.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/ 2024-04-16T19:45:59.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E= 2024-04-16T19:45:59.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T1JYfWU0EB6IkzbpE+0kC/Q0Y1E=', 'Content-Length': '0'} 2024-04-16T19:45:59.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6ccf54418c47088-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.847 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:45:59.847 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:45:59.847 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.847 DEBUG:boto:Path: /yrjefh-38/obj8 2024-04-16T19:45:59.847 DEBUG:boto:Data: 2024-04-16T19:45:59.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.847 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:45:59.847 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.848 DEBUG:boto:Params: {} 2024-04-16T19:45:59.848 DEBUG:boto:Token: None 2024-04-16T19:45:59.848 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-38/obj8 2024-04-16T19:45:59.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUeUbLzBfYzJ2oGeUORRow/56F8= 2024-04-16T19:45:59.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUeUbLzBfYzJ2oGeUORRow/56F8='} 2024-04-16T19:45:59.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006dd1550bdd09725b-00661ed577-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.852 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.852 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.852 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.852 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:59.853 DEBUG:boto:Data: 2024-04-16T19:45:59.853 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.853 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.853 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.853 DEBUG:boto:Params: {} 2024-04-16T19:45:59.853 DEBUG:boto:Token: None 2024-04-16T19:45:59.853 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q= 2024-04-16T19:45:59.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q=', 'Content-Length': '0'} 2024-04-16T19:45:59.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e82bf18b9951f51b-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.856 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:45:59.856 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:45:59.857 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.857 DEBUG:boto:Path: /yrjefh-39/obj5 2024-04-16T19:45:59.857 DEBUG:boto:Data: 2024-04-16T19:45:59.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.857 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.857 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.857 DEBUG:boto:Params: {} 2024-04-16T19:45:59.857 DEBUG:boto:Token: None 2024-04-16T19:45:59.857 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj5 2024-04-16T19:45:59.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gibG7kko9PiitfhPmyCKqTzJmVU= 2024-04-16T19:45:59.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gibG7kko9PiitfhPmyCKqTzJmVU='} 2024-04-16T19:45:59.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be8db5d19903d55a-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.863 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.863 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.863 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.863 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:59.863 DEBUG:boto:Data: 2024-04-16T19:45:59.863 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.863 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.863 DEBUG:boto:Params: {} 2024-04-16T19:45:59.863 DEBUG:boto:Token: None 2024-04-16T19:45:59.864 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q= 2024-04-16T19:45:59.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q=', 'Content-Length': '0'} 2024-04-16T19:45:59.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddf831adab2dda78-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.866 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:45:59.866 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:45:59.866 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.867 DEBUG:boto:Path: /yrjefh-39/obj6 2024-04-16T19:45:59.867 DEBUG:boto:Data: 2024-04-16T19:45:59.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.867 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.867 DEBUG:boto:Params: {} 2024-04-16T19:45:59.867 DEBUG:boto:Token: None 2024-04-16T19:45:59.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj6 2024-04-16T19:45:59.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kJlpIg0g8xoSHhOVGVUOJr4FAk= 2024-04-16T19:45:59.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kJlpIg0g8xoSHhOVGVUOJr4FAk='} 2024-04-16T19:45:59.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e40b179a184c5c90-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.911 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.911 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.911 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.911 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:59.911 DEBUG:boto:Data: 2024-04-16T19:45:59.911 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.911 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.911 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.911 DEBUG:boto:Params: {} 2024-04-16T19:45:59.912 DEBUG:boto:Token: None 2024-04-16T19:45:59.912 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q= 2024-04-16T19:45:59.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q=', 'Content-Length': '0'} 2024-04-16T19:45:59.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddae3b40bcb2d7aa-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.914 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:45:59.914 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:45:59.915 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.915 DEBUG:boto:Path: /yrjefh-39/obj7 2024-04-16T19:45:59.915 DEBUG:boto:Data: 2024-04-16T19:45:59.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.915 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.915 DEBUG:boto:Params: {} 2024-04-16T19:45:59.915 DEBUG:boto:Token: None 2024-04-16T19:45:59.915 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj7 2024-04-16T19:45:59.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fKfMynKVNUXhBfC98t1V9jxHRWY= 2024-04-16T19:45:59.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fKfMynKVNUXhBfC98t1V9jxHRWY='} 2024-04-16T19:45:59.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ae21d2a5a4b87c9-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.920 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:45:59.920 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:45:59.920 DEBUG:boto:Method: HEAD 2024-04-16T19:45:59.921 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:45:59.921 DEBUG:boto:Data: 2024-04-16T19:45:59.921 DEBUG:boto:Headers: {} 2024-04-16T19:45:59.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.921 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.921 DEBUG:boto:Params: {} 2024-04-16T19:45:59.921 DEBUG:boto:Token: None 2024-04-16T19:45:59.921 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/ 2024-04-16T19:45:59.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q= 2024-04-16T19:45:59.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3KXkoAH5aTPDjOUHKv8fZKzTv5Q=', 'Content-Length': '0'} 2024-04-16T19:45:59.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a01dab7977aeee-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.924 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:45:59.924 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:45:59.924 DEBUG:boto:Method: PUT 2024-04-16T19:45:59.924 DEBUG:boto:Path: /yrjefh-39/obj8 2024-04-16T19:45:59.924 DEBUG:boto:Data: 2024-04-16T19:45:59.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:45:59.924 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:45:59.924 DEBUG:boto:Port: 8000 2024-04-16T19:45:59.924 DEBUG:boto:Params: {} 2024-04-16T19:45:59.924 DEBUG:boto:Token: None 2024-04-16T19:45:59.925 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:45:59 GMT /yrjefh-39/obj8 2024-04-16T19:45:59.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c96nAQefsqn2ARMgqLv69y5oKpU= 2024-04-16T19:45:59.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:45:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c96nAQefsqn2ARMgqLv69y5oKpU='} 2024-04-16T19:45:59.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073ec39a6e3cade6c-00661ed577-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:45:59.967 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-38 2024-04-16T19:45:59.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:59.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:45:59.967 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-38 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:00.024 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:00.024 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:00.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:00.077+0000 7f2507810800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:00.078 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.32.6, , 00000000001.46.6, , 00000000001.34.6, , , 00000000001.65.6, , , ] 2024-04-16T19:46:00.079 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.38.6, 00000000002.21.1, 00000000004.50.6, 00000000002.18.1, 00000000004.40.6, 00000000002.22.1, 00000000002.36.1, 00000000004.69.6, 00000000002.63.1, 00000000002.24.1, 00000000002.47.1] 2024-04-16T19:46:05.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:05.078+0000 7f2507810800 1 bucket sync caught up with source: 2024-04-16T19:46:05.080 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.38.6, 00000000002.21.1, 00000000004.50.6, 00000000002.18.1, 00000000004.40.6, 00000000002.22.1, 00000000002.36.1, 00000000004.69.6, 00000000002.63.1, 00000000002.24.1, 00000000002.47.1] 2024-04-16T19:46:05.080 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.38.6, 00000000002.21.1, 00000000004.50.6, 00000000002.18.1, 00000000004.40.6, 00000000002.22.1, 00000000002.36.1, 00000000004.69.6, 00000000002.63.1, 00000000002.24.1, 00000000002.47.1] 2024-04-16T19:46:05.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:05.078+0000 7f2507810800 0 bucket checkpoint complete 2024-04-16T19:46:05.105 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-38 2024-04-16T19:46:05.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:05.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-38', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:05.106 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-38 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:05.169 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:05.169 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:05.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:05.229+0000 7fe64472a800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:05.231 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:46:05.231 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:10.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:10.231+0000 7fe64472a800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:10.233 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:46:10.233 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:15.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:15.232+0000 7fe64472a800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:15.234 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:46:15.235 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:20.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:20.233+0000 7fe64472a800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:20.236 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.162.6, , 00000000001.169.6, , 00000000001.142.6, , , 00000000001.175.6, , , ] 2024-04-16T19:46:20.236 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:25.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:25.235+0000 7fe64472a800 1 bucket sync caught up with source: 2024-04-16T19:46:25.237 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:25.237 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.170.6, , 00000000002.172.6, , 00000000002.146.6, , , 00000000002.178.6, , , ] 2024-04-16T19:46:25.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:25.235+0000 7fe64472a800 0 bucket checkpoint complete 2024-04-16T19:46:25.263 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-38 zones={test-zone2, test-zone1} 2024-04-16T19:46:25.264 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.264 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.264 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.264 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:46:25.264 DEBUG:boto:Data: 2024-04-16T19:46:25.264 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.264 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.264 DEBUG:boto:Params: {} 2024-04-16T19:46:25.265 DEBUG:boto:Token: None 2024-04-16T19:46:25.265 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fOP4bIJcg6mS+XoHQcFy/qu0zBI= 2024-04-16T19:46:25.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fOP4bIJcg6mS+XoHQcFy/qu0zBI=', 'Content-Length': '0'} 2024-04-16T19:46:25.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062fd5b88c314fcbc-00661ed591-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.268 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.268 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.268 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.268 DEBUG:boto:Path: /yrjefh-38/ 2024-04-16T19:46:25.268 DEBUG:boto:Data: 2024-04-16T19:46:25.268 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.269 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.269 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.269 DEBUG:boto:Params: {} 2024-04-16T19:46:25.269 DEBUG:boto:Token: None 2024-04-16T19:46:25.269 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fOP4bIJcg6mS+XoHQcFy/qu0zBI= 2024-04-16T19:46:25.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fOP4bIJcg6mS+XoHQcFy/qu0zBI=', 'Content-Length': '0'} 2024-04-16T19:46:25.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf386597424d07b0-00661ed591-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.272 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:46:25.272 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.272 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.272 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.272 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.272 DEBUG:boto:Method: GET 2024-04-16T19:46:25.272 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.272 DEBUG:boto:Data: 2024-04-16T19:46:25.273 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.273 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.273 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.273 DEBUG:boto:Params: {} 2024-04-16T19:46:25.273 DEBUG:boto:Token: None 2024-04-16T19:46:25.273 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/?versions 2024-04-16T19:46:25.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A= 2024-04-16T19:46:25.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A=', 'Content-Length': '0'} 2024-04-16T19:46:25.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48b2edf693bb814-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.276 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:25.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:25.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:25.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:25.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:46:25.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:46:25.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:46:25.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:46:25.278 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:46:25.278 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.278 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.278 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.278 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.278 DEBUG:boto:Method: GET 2024-04-16T19:46:25.278 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.278 DEBUG:boto:Data: 2024-04-16T19:46:25.279 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.279 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.279 DEBUG:boto:Params: {} 2024-04-16T19:46:25.279 DEBUG:boto:Token: None 2024-04-16T19:46:25.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/?versions 2024-04-16T19:46:25.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A= 2024-04-16T19:46:25.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A=', 'Content-Length': '0'} 2024-04-16T19:46:25.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f912fa4d3595d486-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.282 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:25.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:25.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:25.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:25.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:46:25.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:46:25.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:46:25.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:46:25.283 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.283 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.283 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.283 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.283 DEBUG:boto:Method: GET 2024-04-16T19:46:25.283 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.283 DEBUG:boto:Data: 2024-04-16T19:46:25.283 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.284 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.284 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.284 DEBUG:boto:Params: {} 2024-04-16T19:46:25.284 DEBUG:boto:Token: None 2024-04-16T19:46:25.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/?versions 2024-04-16T19:46:25.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A= 2024-04-16T19:46:25.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A=', 'Content-Length': '0'} 2024-04-16T19:46:25.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee90336940015e30-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.287 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.287 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.287 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.288 DEBUG:boto:path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.288 DEBUG:boto:auth_path=/yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.288 DEBUG:boto:Method: GET 2024-04-16T19:46:25.288 DEBUG:boto:Path: /yrjefh-38/?versions&max-keys=999 2024-04-16T19:46:25.288 DEBUG:boto:Data: 2024-04-16T19:46:25.288 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.288 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.288 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.288 DEBUG:boto:Params: {} 2024-04-16T19:46:25.288 DEBUG:boto:Token: None 2024-04-16T19:46:25.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/?versions 2024-04-16T19:46:25.289 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A= 2024-04-16T19:46:25.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ba6mSmUvdA6j2K8pKsbLi4H7e3A=', 'Content-Length': '0'} 2024-04-16T19:46:25.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000537290623f01802a-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.291 DEBUG:boto:b'yrjefh-38999falseobj1nulltrue2024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:25.292 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.292 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.292 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.292 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.292 DEBUG:boto:Method: GET 2024-04-16T19:46:25.292 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.292 DEBUG:boto:Data: 2024-04-16T19:46:25.292 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.293 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.293 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.293 DEBUG:boto:Params: {} 2024-04-16T19:46:25.293 DEBUG:boto:Token: None 2024-04-16T19:46:25.293 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo= 2024-04-16T19:46:25.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo=', 'Content-Length': '0'} 2024-04-16T19:46:25.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f10ca9d6a895a4e4-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.338 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.339 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.339 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.339 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.339 DEBUG:boto:Method: GET 2024-04-16T19:46:25.339 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.339 DEBUG:boto:Data: 2024-04-16T19:46:25.339 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.339 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.339 DEBUG:boto:Params: {} 2024-04-16T19:46:25.340 DEBUG:boto:Token: None 2024-04-16T19:46:25.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo= 2024-04-16T19:46:25.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo=', 'Content-Length': '0'} 2024-04-16T19:46:25.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf70a3a19a4e5673-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.342 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.342 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.342 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.342 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.343 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.343 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.343 DEBUG:boto:Data: 2024-04-16T19:46:25.343 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.343 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.343 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.343 DEBUG:boto:Params: {} 2024-04-16T19:46:25.343 DEBUG:boto:Token: None 2024-04-16T19:46:25.343 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IV8hYLHVLbg5sb/my8UYDzLClS8= 2024-04-16T19:46:25.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IV8hYLHVLbg5sb/my8UYDzLClS8=', 'Content-Length': '0'} 2024-04-16T19:46:25.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c3f44e0ea8a60d8-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.346 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.346 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.346 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.346 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.346 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.346 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.346 DEBUG:boto:Data: 2024-04-16T19:46:25.346 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.347 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.347 DEBUG:boto:Params: {} 2024-04-16T19:46:25.347 DEBUG:boto:Token: None 2024-04-16T19:46:25.347 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.347 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IV8hYLHVLbg5sb/my8UYDzLClS8= 2024-04-16T19:46:25.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IV8hYLHVLbg5sb/my8UYDzLClS8=', 'Content-Length': '0'} 2024-04-16T19:46:25.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005adeb375ba36a5ea-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:25.349 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.350 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.350 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.350 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.350 DEBUG:boto:Method: GET 2024-04-16T19:46:25.350 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.350 DEBUG:boto:Data: 2024-04-16T19:46:25.350 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.350 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.350 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.350 DEBUG:boto:Params: {} 2024-04-16T19:46:25.350 DEBUG:boto:Token: None 2024-04-16T19:46:25.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo= 2024-04-16T19:46:25.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo=', 'Content-Length': '0'} 2024-04-16T19:46:25.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e65b2e223efa7a3f-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.353 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.353 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.353 DEBUG:boto:path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.353 DEBUG:boto:auth_path=/yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.354 DEBUG:boto:Method: GET 2024-04-16T19:46:25.354 DEBUG:boto:Path: /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.354 DEBUG:boto:Data: 2024-04-16T19:46:25.354 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.354 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.354 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.354 DEBUG:boto:Params: {} 2024-04-16T19:46:25.354 DEBUG:boto:Token: None 2024-04-16T19:46:25.354 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1?versionId=null 2024-04-16T19:46:25.354 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo= 2024-04-16T19:46:25.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oJkrN+l1a3Ccs6QghdqSekvmgSo=', 'Content-Length': '0'} 2024-04-16T19:46:25.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e6733fc9970e590-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.357 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.357 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.357 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.357 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.357 DEBUG:boto:Method: GET 2024-04-16T19:46:25.357 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.357 DEBUG:boto:Data: 2024-04-16T19:46:25.357 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.357 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.357 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.357 DEBUG:boto:Params: {} 2024-04-16T19:46:25.358 DEBUG:boto:Token: None 2024-04-16T19:46:25.358 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a39aab0572eb78-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.361 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.361 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.361 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.361 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.361 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.361 DEBUG:boto:Method: GET 2024-04-16T19:46:25.362 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.362 DEBUG:boto:Data: 2024-04-16T19:46:25.362 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.362 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.362 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.362 DEBUG:boto:Params: {} 2024-04-16T19:46:25.362 DEBUG:boto:Token: None 2024-04-16T19:46:25.362 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.362 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896ec2d067b75a48-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.365 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.365 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.365 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.365 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.365 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:46:25.365 DEBUG:boto:Data: 2024-04-16T19:46:25.365 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.365 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.366 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.366 DEBUG:boto:Params: {} 2024-04-16T19:46:25.366 DEBUG:boto:Token: None 2024-04-16T19:46:25.366 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1 2024-04-16T19:46:25.366 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f0PfgTQyL4k/jY1hv5/2yF1t2BQ= 2024-04-16T19:46:25.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f0PfgTQyL4k/jY1hv5/2yF1t2BQ=', 'Content-Length': '0'} 2024-04-16T19:46:25.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb7430f224a611a9-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.368 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.369 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.369 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.369 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:46:25.369 DEBUG:boto:Data: 2024-04-16T19:46:25.369 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.369 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.369 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.369 DEBUG:boto:Params: {} 2024-04-16T19:46:25.369 DEBUG:boto:Token: None 2024-04-16T19:46:25.369 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1 2024-04-16T19:46:25.369 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f0PfgTQyL4k/jY1hv5/2yF1t2BQ= 2024-04-16T19:46:25.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f0PfgTQyL4k/jY1hv5/2yF1t2BQ=', 'Content-Length': '0'} 2024-04-16T19:46:25.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b02bfb12ee340e63-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:25.372 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.372 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.372 DEBUG:boto:Method: GET 2024-04-16T19:46:25.372 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:46:25.372 DEBUG:boto:Data: 2024-04-16T19:46:25.372 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.372 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.372 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.372 DEBUG:boto:Params: {} 2024-04-16T19:46:25.372 DEBUG:boto:Token: None 2024-04-16T19:46:25.373 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1 2024-04-16T19:46:25.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hta6+Jf8TY7XhFF2RI2YJc+NKIQ= 2024-04-16T19:46:25.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hta6+Jf8TY7XhFF2RI2YJc+NKIQ=', 'Content-Length': '0'} 2024-04-16T19:46:25.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c4dea026d4747ae-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.418 DEBUG:boto:path=/yrjefh-38/obj1 2024-04-16T19:46:25.419 DEBUG:boto:auth_path=/yrjefh-38/obj1 2024-04-16T19:46:25.419 DEBUG:boto:Method: GET 2024-04-16T19:46:25.419 DEBUG:boto:Path: /yrjefh-38/obj1 2024-04-16T19:46:25.419 DEBUG:boto:Data: 2024-04-16T19:46:25.419 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.419 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.419 DEBUG:boto:Params: {} 2024-04-16T19:46:25.419 DEBUG:boto:Token: None 2024-04-16T19:46:25.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj1 2024-04-16T19:46:25.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hta6+Jf8TY7XhFF2RI2YJc+NKIQ= 2024-04-16T19:46:25.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hta6+Jf8TY7XhFF2RI2YJc+NKIQ=', 'Content-Length': '0'} 2024-04-16T19:46:25.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae45c3ec136f262a-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.422 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.422 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.422 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.422 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.422 DEBUG:boto:Method: GET 2024-04-16T19:46:25.422 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.422 DEBUG:boto:Data: 2024-04-16T19:46:25.422 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.423 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.423 DEBUG:boto:Params: {} 2024-04-16T19:46:25.423 DEBUG:boto:Token: None 2024-04-16T19:46:25.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f901ab1bd50e9ba5-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.426 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.426 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.426 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.426 DEBUG:boto:path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.426 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.427 DEBUG:boto:Method: GET 2024-04-16T19:46:25.427 DEBUG:boto:Path: /yrjefh-38/?prefix=obj1 2024-04-16T19:46:25.427 DEBUG:boto:Data: 2024-04-16T19:46:25.427 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.427 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.427 DEBUG:boto:Params: {} 2024-04-16T19:46:25.427 DEBUG:boto:Token: None 2024-04-16T19:46:25.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da2b8611563f334-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.430 DEBUG:boto:b'yrjefh-38obj11000falseobj12024-04-16T19:45:51.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:25.431 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.431 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.431 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.431 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.431 DEBUG:boto:Method: GET 2024-04-16T19:46:25.431 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.431 DEBUG:boto:Data: 2024-04-16T19:46:25.431 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.431 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.431 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.431 DEBUG:boto:Params: {} 2024-04-16T19:46:25.431 DEBUG:boto:Token: None 2024-04-16T19:46:25.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis= 2024-04-16T19:46:25.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis=', 'Content-Length': '0'} 2024-04-16T19:46:25.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f9b4328163f8547-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.474 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.475 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.475 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.475 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.475 DEBUG:boto:Method: GET 2024-04-16T19:46:25.475 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.475 DEBUG:boto:Data: 2024-04-16T19:46:25.475 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.475 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.475 DEBUG:boto:Params: {} 2024-04-16T19:46:25.476 DEBUG:boto:Token: None 2024-04-16T19:46:25.476 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis= 2024-04-16T19:46:25.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis=', 'Content-Length': '0'} 2024-04-16T19:46:25.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e00b1c4c72bf4d7-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.478 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.478 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.478 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.479 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.479 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.479 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.479 DEBUG:boto:Data: 2024-04-16T19:46:25.479 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.479 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.479 DEBUG:boto:Params: {} 2024-04-16T19:46:25.479 DEBUG:boto:Token: None 2024-04-16T19:46:25.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7qbbaCSbkz1AoFs76qRCXnnGFdM= 2024-04-16T19:46:25.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qbbaCSbkz1AoFs76qRCXnnGFdM=', 'Content-Length': '0'} 2024-04-16T19:46:25.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030da5673d95b0ef0-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.482 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.482 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.482 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.482 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.482 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.482 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.482 DEBUG:boto:Data: 2024-04-16T19:46:25.482 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.482 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.482 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.482 DEBUG:boto:Params: {} 2024-04-16T19:46:25.482 DEBUG:boto:Token: None 2024-04-16T19:46:25.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7qbbaCSbkz1AoFs76qRCXnnGFdM= 2024-04-16T19:46:25.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qbbaCSbkz1AoFs76qRCXnnGFdM=', 'Content-Length': '0'} 2024-04-16T19:46:25.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021d1c4b21e4a9770-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:25.485 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.485 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.485 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.485 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.485 DEBUG:boto:Method: GET 2024-04-16T19:46:25.485 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.485 DEBUG:boto:Data: 2024-04-16T19:46:25.486 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.486 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.486 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.486 DEBUG:boto:Params: {} 2024-04-16T19:46:25.486 DEBUG:boto:Token: None 2024-04-16T19:46:25.486 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis= 2024-04-16T19:46:25.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis=', 'Content-Length': '0'} 2024-04-16T19:46:25.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017d88d4434e33d69-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.530 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.530 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.531 DEBUG:boto:path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.531 DEBUG:boto:auth_path=/yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.531 DEBUG:boto:Method: GET 2024-04-16T19:46:25.531 DEBUG:boto:Path: /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.531 DEBUG:boto:Data: 2024-04-16T19:46:25.531 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.531 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.531 DEBUG:boto:Params: {} 2024-04-16T19:46:25.531 DEBUG:boto:Token: None 2024-04-16T19:46:25.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2?versionId=null 2024-04-16T19:46:25.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis= 2024-04-16T19:46:25.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A2LKm5QV818krG8HcicJWYrVAis=', 'Content-Length': '0'} 2024-04-16T19:46:25.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013a4e14b0cf2d67a-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.534 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.534 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.534 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.534 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.534 DEBUG:boto:Method: GET 2024-04-16T19:46:25.534 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.535 DEBUG:boto:Data: 2024-04-16T19:46:25.535 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.535 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.535 DEBUG:boto:Params: {} 2024-04-16T19:46:25.535 DEBUG:boto:Token: None 2024-04-16T19:46:25.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cefa89af08e3be61-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.538 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.538 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.538 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.538 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.538 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.539 DEBUG:boto:Method: GET 2024-04-16T19:46:25.539 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.539 DEBUG:boto:Data: 2024-04-16T19:46:25.539 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.539 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.539 DEBUG:boto:Params: {} 2024-04-16T19:46:25.539 DEBUG:boto:Token: None 2024-04-16T19:46:25.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bc9e677b962a3d-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.542 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.542 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.542 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.542 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.543 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:46:25.543 DEBUG:boto:Data: 2024-04-16T19:46:25.543 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.543 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.543 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.543 DEBUG:boto:Params: {} 2024-04-16T19:46:25.543 DEBUG:boto:Token: None 2024-04-16T19:46:25.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2 2024-04-16T19:46:25.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cBHQowUVbVTiCCLwVABomhcpIZs= 2024-04-16T19:46:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cBHQowUVbVTiCCLwVABomhcpIZs=', 'Content-Length': '0'} 2024-04-16T19:46:25.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000189665f991a8817d-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.545 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.545 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.545 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.546 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:46:25.546 DEBUG:boto:Data: 2024-04-16T19:46:25.546 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.546 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.546 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.546 DEBUG:boto:Params: {} 2024-04-16T19:46:25.546 DEBUG:boto:Token: None 2024-04-16T19:46:25.546 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2 2024-04-16T19:46:25.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cBHQowUVbVTiCCLwVABomhcpIZs= 2024-04-16T19:46:25.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cBHQowUVbVTiCCLwVABomhcpIZs=', 'Content-Length': '0'} 2024-04-16T19:46:25.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dccaa990875d683b-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:25.549 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.549 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.549 DEBUG:boto:Method: GET 2024-04-16T19:46:25.549 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:46:25.549 DEBUG:boto:Data: 2024-04-16T19:46:25.549 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.549 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.549 DEBUG:boto:Params: {} 2024-04-16T19:46:25.550 DEBUG:boto:Token: None 2024-04-16T19:46:25.550 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2 2024-04-16T19:46:25.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qT2TKIJesiuLgTrOoUDxEUtvCSM= 2024-04-16T19:46:25.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qT2TKIJesiuLgTrOoUDxEUtvCSM=', 'Content-Length': '0'} 2024-04-16T19:46:25.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007497f385758b74ed-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.594 DEBUG:boto:path=/yrjefh-38/obj2 2024-04-16T19:46:25.594 DEBUG:boto:auth_path=/yrjefh-38/obj2 2024-04-16T19:46:25.595 DEBUG:boto:Method: GET 2024-04-16T19:46:25.595 DEBUG:boto:Path: /yrjefh-38/obj2 2024-04-16T19:46:25.595 DEBUG:boto:Data: 2024-04-16T19:46:25.595 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.595 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.595 DEBUG:boto:Params: {} 2024-04-16T19:46:25.595 DEBUG:boto:Token: None 2024-04-16T19:46:25.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj2 2024-04-16T19:46:25.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qT2TKIJesiuLgTrOoUDxEUtvCSM= 2024-04-16T19:46:25.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qT2TKIJesiuLgTrOoUDxEUtvCSM=', 'Content-Length': '0'} 2024-04-16T19:46:25.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d005e7832746ebc-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.598 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.598 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.598 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.599 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.599 DEBUG:boto:Method: GET 2024-04-16T19:46:25.599 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.599 DEBUG:boto:Data: 2024-04-16T19:46:25.599 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.599 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.600 DEBUG:boto:Params: {} 2024-04-16T19:46:25.600 DEBUG:boto:Token: None 2024-04-16T19:46:25.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d14ae74fd6e935-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.606 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.606 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.606 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.606 DEBUG:boto:path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.606 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.606 DEBUG:boto:Method: GET 2024-04-16T19:46:25.606 DEBUG:boto:Path: /yrjefh-38/?prefix=obj2 2024-04-16T19:46:25.607 DEBUG:boto:Data: 2024-04-16T19:46:25.607 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.607 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.607 DEBUG:boto:Params: {} 2024-04-16T19:46:25.607 DEBUG:boto:Token: None 2024-04-16T19:46:25.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb97433941346ae-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.609 DEBUG:boto:b'yrjefh-38obj21000falseobj22024-04-16T19:45:51.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:25.610 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.610 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.610 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.610 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.610 DEBUG:boto:Method: GET 2024-04-16T19:46:25.610 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.610 DEBUG:boto:Data: 2024-04-16T19:46:25.611 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.611 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.611 DEBUG:boto:Params: {} 2024-04-16T19:46:25.611 DEBUG:boto:Token: None 2024-04-16T19:46:25.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg= 2024-04-16T19:46:25.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg=', 'Content-Length': '0'} 2024-04-16T19:46:25.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002171246b1057ed87-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.654 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.654 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.655 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.655 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.655 DEBUG:boto:Method: GET 2024-04-16T19:46:25.655 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.655 DEBUG:boto:Data: 2024-04-16T19:46:25.655 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.655 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.655 DEBUG:boto:Params: {} 2024-04-16T19:46:25.655 DEBUG:boto:Token: None 2024-04-16T19:46:25.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg= 2024-04-16T19:46:25.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg=', 'Content-Length': '0'} 2024-04-16T19:46:25.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013b26aecfe4a855c-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.658 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.658 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.658 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.658 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.658 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.658 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.658 DEBUG:boto:Data: 2024-04-16T19:46:25.659 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.659 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.659 DEBUG:boto:Params: {} 2024-04-16T19:46:25.659 DEBUG:boto:Token: None 2024-04-16T19:46:25.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0MeFUXYBhXD78kviOMyeF2DJ1iw= 2024-04-16T19:46:25.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0MeFUXYBhXD78kviOMyeF2DJ1iw=', 'Content-Length': '0'} 2024-04-16T19:46:25.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025745f72e07bd84f-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.662 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.662 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.662 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.662 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.662 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.662 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.662 DEBUG:boto:Data: 2024-04-16T19:46:25.662 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.662 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.662 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.663 DEBUG:boto:Params: {} 2024-04-16T19:46:25.663 DEBUG:boto:Token: None 2024-04-16T19:46:25.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0MeFUXYBhXD78kviOMyeF2DJ1iw= 2024-04-16T19:46:25.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0MeFUXYBhXD78kviOMyeF2DJ1iw=', 'Content-Length': '0'} 2024-04-16T19:46:25.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057e7b6e4fb936df5-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:25.665 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.665 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.666 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.666 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.666 DEBUG:boto:Method: GET 2024-04-16T19:46:25.666 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.666 DEBUG:boto:Data: 2024-04-16T19:46:25.666 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.666 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.666 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.666 DEBUG:boto:Params: {} 2024-04-16T19:46:25.666 DEBUG:boto:Token: None 2024-04-16T19:46:25.666 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg= 2024-04-16T19:46:25.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg=', 'Content-Length': '0'} 2024-04-16T19:46:25.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8d49c93d9981b9f-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.710 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.710 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.711 DEBUG:boto:path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.711 DEBUG:boto:auth_path=/yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.711 DEBUG:boto:Method: GET 2024-04-16T19:46:25.711 DEBUG:boto:Path: /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.711 DEBUG:boto:Data: 2024-04-16T19:46:25.711 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.711 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.711 DEBUG:boto:Params: {} 2024-04-16T19:46:25.711 DEBUG:boto:Token: None 2024-04-16T19:46:25.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3?versionId=null 2024-04-16T19:46:25.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg= 2024-04-16T19:46:25.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQek9uYq9AnCyp/IaXKPk+HENg=', 'Content-Length': '0'} 2024-04-16T19:46:25.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6fb6285881793f8-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.714 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.714 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.714 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.715 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.715 DEBUG:boto:Method: GET 2024-04-16T19:46:25.715 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.715 DEBUG:boto:Data: 2024-04-16T19:46:25.715 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.715 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.715 DEBUG:boto:Params: {} 2024-04-16T19:46:25.715 DEBUG:boto:Token: None 2024-04-16T19:46:25.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305eb3b0f1543913-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.718 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.719 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.719 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.719 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.719 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.719 DEBUG:boto:Method: GET 2024-04-16T19:46:25.719 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.719 DEBUG:boto:Data: 2024-04-16T19:46:25.719 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.719 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.720 DEBUG:boto:Params: {} 2024-04-16T19:46:25.720 DEBUG:boto:Token: None 2024-04-16T19:46:25.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6780a1a6be1e7a3-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.722 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.723 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.723 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.723 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.723 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:46:25.723 DEBUG:boto:Data: 2024-04-16T19:46:25.723 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.723 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.723 DEBUG:boto:Params: {} 2024-04-16T19:46:25.723 DEBUG:boto:Token: None 2024-04-16T19:46:25.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3 2024-04-16T19:46:25.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7mRrQlf5rdnnQkmvhbrlF4zltJE= 2024-04-16T19:46:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7mRrQlf5rdnnQkmvhbrlF4zltJE=', 'Content-Length': '0'} 2024-04-16T19:46:25.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4f3c7118d594cf3-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.726 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.726 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.726 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.726 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:46:25.726 DEBUG:boto:Data: 2024-04-16T19:46:25.726 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.727 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.727 DEBUG:boto:Params: {} 2024-04-16T19:46:25.727 DEBUG:boto:Token: None 2024-04-16T19:46:25.727 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3 2024-04-16T19:46:25.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7mRrQlf5rdnnQkmvhbrlF4zltJE= 2024-04-16T19:46:25.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7mRrQlf5rdnnQkmvhbrlF4zltJE=', 'Content-Length': '0'} 2024-04-16T19:46:25.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae87e73ffeaa940-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:25.729 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.729 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.730 DEBUG:boto:Method: GET 2024-04-16T19:46:25.730 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:46:25.730 DEBUG:boto:Data: 2024-04-16T19:46:25.730 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.730 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.730 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.730 DEBUG:boto:Params: {} 2024-04-16T19:46:25.730 DEBUG:boto:Token: None 2024-04-16T19:46:25.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3 2024-04-16T19:46:25.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kX0Qt0DdCYoFRVtD2kdgIeYcrbA= 2024-04-16T19:46:25.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kX0Qt0DdCYoFRVtD2kdgIeYcrbA=', 'Content-Length': '0'} 2024-04-16T19:46:25.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac8f3db42dfdb038-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.774 DEBUG:boto:path=/yrjefh-38/obj3 2024-04-16T19:46:25.774 DEBUG:boto:auth_path=/yrjefh-38/obj3 2024-04-16T19:46:25.775 DEBUG:boto:Method: GET 2024-04-16T19:46:25.775 DEBUG:boto:Path: /yrjefh-38/obj3 2024-04-16T19:46:25.775 DEBUG:boto:Data: 2024-04-16T19:46:25.775 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.775 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.775 DEBUG:boto:Params: {} 2024-04-16T19:46:25.775 DEBUG:boto:Token: None 2024-04-16T19:46:25.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj3 2024-04-16T19:46:25.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kX0Qt0DdCYoFRVtD2kdgIeYcrbA= 2024-04-16T19:46:25.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kX0Qt0DdCYoFRVtD2kdgIeYcrbA=', 'Content-Length': '0'} 2024-04-16T19:46:25.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e54f753f71fb2034-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.778 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.778 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.778 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.778 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.778 DEBUG:boto:Method: GET 2024-04-16T19:46:25.778 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.778 DEBUG:boto:Data: 2024-04-16T19:46:25.779 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.779 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.779 DEBUG:boto:Params: {} 2024-04-16T19:46:25.779 DEBUG:boto:Token: None 2024-04-16T19:46:25.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000779f6a45a7784d2e-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.782 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.782 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.783 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.783 DEBUG:boto:path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.783 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.783 DEBUG:boto:Method: GET 2024-04-16T19:46:25.783 DEBUG:boto:Path: /yrjefh-38/?prefix=obj3 2024-04-16T19:46:25.783 DEBUG:boto:Data: 2024-04-16T19:46:25.783 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.783 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.783 DEBUG:boto:Params: {} 2024-04-16T19:46:25.783 DEBUG:boto:Token: None 2024-04-16T19:46:25.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ad0a935e70e914-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.786 DEBUG:boto:b'yrjefh-38obj31000falseobj32024-04-16T19:45:51.504Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:25.786 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.787 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.787 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.787 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.787 DEBUG:boto:Method: GET 2024-04-16T19:46:25.787 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.787 DEBUG:boto:Data: 2024-04-16T19:46:25.787 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.787 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.787 DEBUG:boto:Params: {} 2024-04-16T19:46:25.787 DEBUG:boto:Token: None 2024-04-16T19:46:25.788 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw= 2024-04-16T19:46:25.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw=', 'Content-Length': '0'} 2024-04-16T19:46:25.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6faab9c0a4a2e27-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.834 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.835 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.835 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.835 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.835 DEBUG:boto:Method: GET 2024-04-16T19:46:25.835 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.835 DEBUG:boto:Data: 2024-04-16T19:46:25.835 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.835 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.835 DEBUG:boto:Params: {} 2024-04-16T19:46:25.835 DEBUG:boto:Token: None 2024-04-16T19:46:25.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw= 2024-04-16T19:46:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw=', 'Content-Length': '0'} 2024-04-16T19:46:25.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c61f9f0ff38c8656-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.838 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.838 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.838 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.838 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.838 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.839 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.839 DEBUG:boto:Data: 2024-04-16T19:46:25.839 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.839 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.839 DEBUG:boto:Params: {} 2024-04-16T19:46:25.839 DEBUG:boto:Token: None 2024-04-16T19:46:25.839 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7oji2M7wcfR2dl0dYwgGzKrvqrY= 2024-04-16T19:46:25.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7oji2M7wcfR2dl0dYwgGzKrvqrY=', 'Content-Length': '0'} 2024-04-16T19:46:25.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cabcc5c033c5c9f1-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.842 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.842 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.842 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.842 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.842 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.842 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.842 DEBUG:boto:Data: 2024-04-16T19:46:25.842 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.842 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.842 DEBUG:boto:Params: {} 2024-04-16T19:46:25.843 DEBUG:boto:Token: None 2024-04-16T19:46:25.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7oji2M7wcfR2dl0dYwgGzKrvqrY= 2024-04-16T19:46:25.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7oji2M7wcfR2dl0dYwgGzKrvqrY=', 'Content-Length': '0'} 2024-04-16T19:46:25.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edfef3ae4268408d-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:25.845 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.845 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.845 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.845 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.845 DEBUG:boto:Method: GET 2024-04-16T19:46:25.845 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.845 DEBUG:boto:Data: 2024-04-16T19:46:25.845 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.846 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.846 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.846 DEBUG:boto:Params: {} 2024-04-16T19:46:25.846 DEBUG:boto:Token: None 2024-04-16T19:46:25.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw= 2024-04-16T19:46:25.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw=', 'Content-Length': '0'} 2024-04-16T19:46:25.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6f19a1fc3b85a37-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.890 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.890 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.891 DEBUG:boto:path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.891 DEBUG:boto:auth_path=/yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.891 DEBUG:boto:Method: GET 2024-04-16T19:46:25.891 DEBUG:boto:Path: /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.891 DEBUG:boto:Data: 2024-04-16T19:46:25.891 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.891 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.891 DEBUG:boto:Params: {} 2024-04-16T19:46:25.891 DEBUG:boto:Token: None 2024-04-16T19:46:25.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4?versionId=null 2024-04-16T19:46:25.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw= 2024-04-16T19:46:25.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFT+SwszKBIlnFULX/47TWI9dTw=', 'Content-Length': '0'} 2024-04-16T19:46:25.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068f72349ce8083a7-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.894 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.894 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.894 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.894 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.894 DEBUG:boto:Method: GET 2024-04-16T19:46:25.895 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.895 DEBUG:boto:Data: 2024-04-16T19:46:25.895 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.895 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.895 DEBUG:boto:Params: {} 2024-04-16T19:46:25.895 DEBUG:boto:Token: None 2024-04-16T19:46:25.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de882967008f2185-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.938 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.939 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.939 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.939 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.939 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.939 DEBUG:boto:Method: GET 2024-04-16T19:46:25.939 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.939 DEBUG:boto:Data: 2024-04-16T19:46:25.939 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.940 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.940 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.940 DEBUG:boto:Params: {} 2024-04-16T19:46:25.940 DEBUG:boto:Token: None 2024-04-16T19:46:25.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:25.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b6daf17ae53327-00661ed591-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.943 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:25.943 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.943 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.943 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.943 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:46:25.943 DEBUG:boto:Data: 2024-04-16T19:46:25.943 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.943 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.944 DEBUG:boto:Params: {} 2024-04-16T19:46:25.944 DEBUG:boto:Token: None 2024-04-16T19:46:25.944 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4 2024-04-16T19:46:25.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SN080efLs2bQTIK8ZkXwwy0R5kg= 2024-04-16T19:46:25.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SN080efLs2bQTIK8ZkXwwy0R5kg=', 'Content-Length': '0'} 2024-04-16T19:46:25.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027b272484a108cd3-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.946 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.946 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.947 DEBUG:boto:Method: HEAD 2024-04-16T19:46:25.947 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:46:25.947 DEBUG:boto:Data: 2024-04-16T19:46:25.947 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.947 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.947 DEBUG:boto:Params: {} 2024-04-16T19:46:25.947 DEBUG:boto:Token: None 2024-04-16T19:46:25.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4 2024-04-16T19:46:25.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SN080efLs2bQTIK8ZkXwwy0R5kg= 2024-04-16T19:46:25.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SN080efLs2bQTIK8ZkXwwy0R5kg=', 'Content-Length': '0'} 2024-04-16T19:46:25.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8796b8295203958-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:25.950 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.950 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.950 DEBUG:boto:Method: GET 2024-04-16T19:46:25.950 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:46:25.950 DEBUG:boto:Data: 2024-04-16T19:46:25.950 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.950 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.951 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.951 DEBUG:boto:Params: {} 2024-04-16T19:46:25.951 DEBUG:boto:Token: None 2024-04-16T19:46:25.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4 2024-04-16T19:46:25.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TNhxrHj0oeHEP5uq23QPS+ixGHU= 2024-04-16T19:46:25.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TNhxrHj0oeHEP5uq23QPS+ixGHU=', 'Content-Length': '0'} 2024-04-16T19:46:25.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cbe2f42812a0609-00661ed591-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.994 DEBUG:boto:path=/yrjefh-38/obj4 2024-04-16T19:46:25.994 DEBUG:boto:auth_path=/yrjefh-38/obj4 2024-04-16T19:46:25.995 DEBUG:boto:Method: GET 2024-04-16T19:46:25.995 DEBUG:boto:Path: /yrjefh-38/obj4 2024-04-16T19:46:25.995 DEBUG:boto:Data: 2024-04-16T19:46:25.995 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:25.995 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.995 DEBUG:boto:Params: {} 2024-04-16T19:46:25.995 DEBUG:boto:Token: None 2024-04-16T19:46:25.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/obj4 2024-04-16T19:46:25.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TNhxrHj0oeHEP5uq23QPS+ixGHU= 2024-04-16T19:46:25.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TNhxrHj0oeHEP5uq23QPS+ixGHU=', 'Content-Length': '0'} 2024-04-16T19:46:25.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e081a6553d9b4bfd-00661ed591-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:25.998 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:25.998 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:25.998 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.998 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.998 DEBUG:boto:Method: GET 2024-04-16T19:46:25.998 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:46:25.998 DEBUG:boto:Data: 2024-04-16T19:46:25.999 DEBUG:boto:Headers: {} 2024-04-16T19:46:25.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:25.999 DEBUG:boto:Port: 8000 2024-04-16T19:46:25.999 DEBUG:boto:Params: {} 2024-04-16T19:46:25.999 DEBUG:boto:Token: None 2024-04-16T19:46:25.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:25 GMT /yrjefh-38/ 2024-04-16T19:46:25.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8= 2024-04-16T19:46:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBCDp+ptwYr1mybLVNHUWrHgGJ8=', 'Content-Length': '0'} 2024-04-16T19:46:26.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003db7d236e7b5ab8-00661ed591-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.002 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.002 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.002 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.002 DEBUG:boto:path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:26.003 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj4 2024-04-16T19:46:26.003 DEBUG:boto:Method: GET 2024-04-16T19:46:26.003 DEBUG:boto:Path: /yrjefh-38/?prefix=obj4 2024-04-16T19:46:26.003 DEBUG:boto:Data: 2024-04-16T19:46:26.003 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.003 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.003 DEBUG:boto:Params: {} 2024-04-16T19:46:26.003 DEBUG:boto:Token: None 2024-04-16T19:46:26.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ccd9351c5d153f6-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.006 DEBUG:boto:b'yrjefh-38obj41000falseobj42024-04-16T19:45:51.515Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:26.006 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.007 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.007 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.007 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.007 DEBUG:boto:Method: GET 2024-04-16T19:46:26.007 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.007 DEBUG:boto:Data: 2024-04-16T19:46:26.007 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.007 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.007 DEBUG:boto:Params: {} 2024-04-16T19:46:26.007 DEBUG:boto:Token: None 2024-04-16T19:46:26.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg= 2024-04-16T19:46:26.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg=', 'Content-Length': '0'} 2024-04-16T19:46:26.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0d81ee2d64a2b44-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.050 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.050 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.051 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.051 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.051 DEBUG:boto:Method: GET 2024-04-16T19:46:26.051 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.051 DEBUG:boto:Data: 2024-04-16T19:46:26.051 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.051 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.051 DEBUG:boto:Params: {} 2024-04-16T19:46:26.051 DEBUG:boto:Token: None 2024-04-16T19:46:26.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg= 2024-04-16T19:46:26.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg=', 'Content-Length': '0'} 2024-04-16T19:46:26.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e26ec23021666c45-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.054 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.054 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.054 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.054 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.054 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.054 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.054 DEBUG:boto:Data: 2024-04-16T19:46:26.054 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.054 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.055 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.055 DEBUG:boto:Params: {} 2024-04-16T19:46:26.055 DEBUG:boto:Token: None 2024-04-16T19:46:26.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AzamQgM5cNpyvvK/8HzEZQDJykM= 2024-04-16T19:46:26.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AzamQgM5cNpyvvK/8HzEZQDJykM=', 'Content-Length': '0'} 2024-04-16T19:46:26.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1fe1743f95e5003-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.057 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.058 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.058 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.058 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.058 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.058 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.058 DEBUG:boto:Data: 2024-04-16T19:46:26.058 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.058 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.058 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.058 DEBUG:boto:Params: {} 2024-04-16T19:46:26.058 DEBUG:boto:Token: None 2024-04-16T19:46:26.058 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AzamQgM5cNpyvvK/8HzEZQDJykM= 2024-04-16T19:46:26.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AzamQgM5cNpyvvK/8HzEZQDJykM=', 'Content-Length': '0'} 2024-04-16T19:46:26.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc6bf83ef75a100-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:26.061 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.061 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.061 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.061 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.062 DEBUG:boto:Method: GET 2024-04-16T19:46:26.062 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.062 DEBUG:boto:Data: 2024-04-16T19:46:26.062 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.062 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.062 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.062 DEBUG:boto:Params: {} 2024-04-16T19:46:26.062 DEBUG:boto:Token: None 2024-04-16T19:46:26.062 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.062 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg= 2024-04-16T19:46:26.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg=', 'Content-Length': '0'} 2024-04-16T19:46:26.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062de3dfd4df80fc9-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.106 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.106 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.107 DEBUG:boto:path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.107 DEBUG:boto:auth_path=/yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.107 DEBUG:boto:Method: GET 2024-04-16T19:46:26.107 DEBUG:boto:Path: /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.107 DEBUG:boto:Data: 2024-04-16T19:46:26.107 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.107 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.107 DEBUG:boto:Params: {} 2024-04-16T19:46:26.107 DEBUG:boto:Token: None 2024-04-16T19:46:26.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5?versionId=null 2024-04-16T19:46:26.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg= 2024-04-16T19:46:26.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TegiaoaoDVKgR47e+FQwlEDjeUg=', 'Content-Length': '0'} 2024-04-16T19:46:26.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eedbbf6efe7eea3a-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.110 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.110 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.110 DEBUG:boto:path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.111 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.111 DEBUG:boto:Method: GET 2024-04-16T19:46:26.111 DEBUG:boto:Path: /yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.111 DEBUG:boto:Data: 2024-04-16T19:46:26.111 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.111 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.111 DEBUG:boto:Params: {} 2024-04-16T19:46:26.111 DEBUG:boto:Token: None 2024-04-16T19:46:26.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c1fd37b8a9f4e7-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.114 DEBUG:boto:b'yrjefh-38obj51000falseobj52024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.115 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.115 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.115 DEBUG:boto:path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.115 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.115 DEBUG:boto:Method: GET 2024-04-16T19:46:26.115 DEBUG:boto:Path: /yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.115 DEBUG:boto:Data: 2024-04-16T19:46:26.115 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.116 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.116 DEBUG:boto:Params: {} 2024-04-16T19:46:26.116 DEBUG:boto:Token: None 2024-04-16T19:46:26.116 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd4157fba51267c-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.118 DEBUG:boto:b'yrjefh-38obj51000falseobj52024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.119 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.119 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.119 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.119 DEBUG:boto:Path: /yrjefh-38/obj5 2024-04-16T19:46:26.119 DEBUG:boto:Data: 2024-04-16T19:46:26.119 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.119 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.119 DEBUG:boto:Params: {} 2024-04-16T19:46:26.120 DEBUG:boto:Token: None 2024-04-16T19:46:26.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5 2024-04-16T19:46:26.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/hrXeAm9U8o9YI8iLRmMykn4rM= 2024-04-16T19:46:26.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/hrXeAm9U8o9YI8iLRmMykn4rM=', 'Content-Length': '0'} 2024-04-16T19:46:26.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004322f382cab776ae-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.122 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.122 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.122 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.122 DEBUG:boto:Path: /yrjefh-38/obj5 2024-04-16T19:46:26.123 DEBUG:boto:Data: 2024-04-16T19:46:26.123 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.123 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.123 DEBUG:boto:Params: {} 2024-04-16T19:46:26.123 DEBUG:boto:Token: None 2024-04-16T19:46:26.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5 2024-04-16T19:46:26.123 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/hrXeAm9U8o9YI8iLRmMykn4rM= 2024-04-16T19:46:26.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/hrXeAm9U8o9YI8iLRmMykn4rM=', 'Content-Length': '0'} 2024-04-16T19:46:26.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f188af8dc35c32ef-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:26.126 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.126 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.126 DEBUG:boto:Method: GET 2024-04-16T19:46:26.126 DEBUG:boto:Path: /yrjefh-38/obj5 2024-04-16T19:46:26.126 DEBUG:boto:Data: 2024-04-16T19:46:26.126 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.126 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.126 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.126 DEBUG:boto:Params: {} 2024-04-16T19:46:26.126 DEBUG:boto:Token: None 2024-04-16T19:46:26.127 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5 2024-04-16T19:46:26.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+yr3qYtxqT0dhZy6e2rLJ4c12Y= 2024-04-16T19:46:26.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+yr3qYtxqT0dhZy6e2rLJ4c12Y=', 'Content-Length': '0'} 2024-04-16T19:46:26.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003544978f7f62f7ea-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.170 DEBUG:boto:path=/yrjefh-38/obj5 2024-04-16T19:46:26.170 DEBUG:boto:auth_path=/yrjefh-38/obj5 2024-04-16T19:46:26.170 DEBUG:boto:Method: GET 2024-04-16T19:46:26.171 DEBUG:boto:Path: /yrjefh-38/obj5 2024-04-16T19:46:26.171 DEBUG:boto:Data: 2024-04-16T19:46:26.171 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.171 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.171 DEBUG:boto:Params: {} 2024-04-16T19:46:26.171 DEBUG:boto:Token: None 2024-04-16T19:46:26.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj5 2024-04-16T19:46:26.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+yr3qYtxqT0dhZy6e2rLJ4c12Y= 2024-04-16T19:46:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+yr3qYtxqT0dhZy6e2rLJ4c12Y=', 'Content-Length': '0'} 2024-04-16T19:46:26.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1fad4136d0a5df3-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.174 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.174 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.174 DEBUG:boto:path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.174 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.175 DEBUG:boto:Method: GET 2024-04-16T19:46:26.175 DEBUG:boto:Path: /yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.175 DEBUG:boto:Data: 2024-04-16T19:46:26.175 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.175 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.175 DEBUG:boto:Params: {} 2024-04-16T19:46:26.175 DEBUG:boto:Token: None 2024-04-16T19:46:26.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8c8d85630ea42f-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.218 DEBUG:boto:b'yrjefh-38obj51000falseobj52024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.219 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.219 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.219 DEBUG:boto:path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.219 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.219 DEBUG:boto:Method: GET 2024-04-16T19:46:26.219 DEBUG:boto:Path: /yrjefh-38/?prefix=obj5 2024-04-16T19:46:26.219 DEBUG:boto:Data: 2024-04-16T19:46:26.219 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.219 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.219 DEBUG:boto:Params: {} 2024-04-16T19:46:26.220 DEBUG:boto:Token: None 2024-04-16T19:46:26.220 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af5a5ef56eb9d485-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.222 DEBUG:boto:b'yrjefh-38obj51000falseobj52024-04-16T19:45:59.780Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:26.223 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.223 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.223 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.223 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.223 DEBUG:boto:Method: GET 2024-04-16T19:46:26.223 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.223 DEBUG:boto:Data: 2024-04-16T19:46:26.223 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.224 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.224 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.224 DEBUG:boto:Params: {} 2024-04-16T19:46:26.224 DEBUG:boto:Token: None 2024-04-16T19:46:26.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0= 2024-04-16T19:46:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0=', 'Content-Length': '0'} 2024-04-16T19:46:26.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aff4ed21323f4b2-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.270 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.271 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.271 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.271 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.271 DEBUG:boto:Method: GET 2024-04-16T19:46:26.271 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.271 DEBUG:boto:Data: 2024-04-16T19:46:26.271 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.271 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.271 DEBUG:boto:Params: {} 2024-04-16T19:46:26.272 DEBUG:boto:Token: None 2024-04-16T19:46:26.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0= 2024-04-16T19:46:26.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0=', 'Content-Length': '0'} 2024-04-16T19:46:26.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077ef3d70979a1748-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.274 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.274 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.275 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.275 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.275 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.275 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.275 DEBUG:boto:Data: 2024-04-16T19:46:26.275 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.275 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.275 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.275 DEBUG:boto:Params: {} 2024-04-16T19:46:26.275 DEBUG:boto:Token: None 2024-04-16T19:46:26.275 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wLggkhBOlm6anA5CxVIUvc1zMNA= 2024-04-16T19:46:26.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wLggkhBOlm6anA5CxVIUvc1zMNA=', 'Content-Length': '0'} 2024-04-16T19:46:26.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec85dfd5e040fa9b-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.278 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.278 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.278 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.278 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.279 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.279 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.279 DEBUG:boto:Data: 2024-04-16T19:46:26.279 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.279 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.279 DEBUG:boto:Params: {} 2024-04-16T19:46:26.279 DEBUG:boto:Token: None 2024-04-16T19:46:26.279 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wLggkhBOlm6anA5CxVIUvc1zMNA= 2024-04-16T19:46:26.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wLggkhBOlm6anA5CxVIUvc1zMNA=', 'Content-Length': '0'} 2024-04-16T19:46:26.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ad1068fd6c94b6f-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:26.282 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.282 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.282 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.282 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.283 DEBUG:boto:Method: GET 2024-04-16T19:46:26.283 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.283 DEBUG:boto:Data: 2024-04-16T19:46:26.283 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.283 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.283 DEBUG:boto:Params: {} 2024-04-16T19:46:26.283 DEBUG:boto:Token: None 2024-04-16T19:46:26.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0= 2024-04-16T19:46:26.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0=', 'Content-Length': '0'} 2024-04-16T19:46:26.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffe0529ec123178f-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.326 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.326 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.327 DEBUG:boto:path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.327 DEBUG:boto:auth_path=/yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.327 DEBUG:boto:Method: GET 2024-04-16T19:46:26.327 DEBUG:boto:Path: /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.327 DEBUG:boto:Data: 2024-04-16T19:46:26.327 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.327 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.327 DEBUG:boto:Params: {} 2024-04-16T19:46:26.327 DEBUG:boto:Token: None 2024-04-16T19:46:26.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6?versionId=null 2024-04-16T19:46:26.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0= 2024-04-16T19:46:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k2yx1L0qDHvg+lD58bj8gh0VSX0=', 'Content-Length': '0'} 2024-04-16T19:46:26.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eddc210f66904319-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.331 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.331 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.331 DEBUG:boto:path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.331 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.331 DEBUG:boto:Method: GET 2024-04-16T19:46:26.331 DEBUG:boto:Path: /yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.331 DEBUG:boto:Data: 2024-04-16T19:46:26.331 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.331 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.332 DEBUG:boto:Params: {} 2024-04-16T19:46:26.332 DEBUG:boto:Token: None 2024-04-16T19:46:26.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602525074cb0f3e2-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.378 DEBUG:boto:b'yrjefh-38obj61000falseobj62024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.379 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.379 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.379 DEBUG:boto:path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.379 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.379 DEBUG:boto:Method: GET 2024-04-16T19:46:26.379 DEBUG:boto:Path: /yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.379 DEBUG:boto:Data: 2024-04-16T19:46:26.379 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.379 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.380 DEBUG:boto:Params: {} 2024-04-16T19:46:26.380 DEBUG:boto:Token: None 2024-04-16T19:46:26.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033220356af8af23f-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.382 DEBUG:boto:b'yrjefh-38obj61000falseobj62024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.383 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.383 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.383 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.383 DEBUG:boto:Path: /yrjefh-38/obj6 2024-04-16T19:46:26.383 DEBUG:boto:Data: 2024-04-16T19:46:26.383 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.383 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.383 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.383 DEBUG:boto:Params: {} 2024-04-16T19:46:26.383 DEBUG:boto:Token: None 2024-04-16T19:46:26.383 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6 2024-04-16T19:46:26.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NRfPf/kgeQCKEMy7RLJxBbIRULg= 2024-04-16T19:46:26.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NRfPf/kgeQCKEMy7RLJxBbIRULg=', 'Content-Length': '0'} 2024-04-16T19:46:26.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b179bf869a4a35d-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.386 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.386 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.386 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.386 DEBUG:boto:Path: /yrjefh-38/obj6 2024-04-16T19:46:26.386 DEBUG:boto:Data: 2024-04-16T19:46:26.387 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.387 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.387 DEBUG:boto:Params: {} 2024-04-16T19:46:26.387 DEBUG:boto:Token: None 2024-04-16T19:46:26.387 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6 2024-04-16T19:46:26.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NRfPf/kgeQCKEMy7RLJxBbIRULg= 2024-04-16T19:46:26.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NRfPf/kgeQCKEMy7RLJxBbIRULg=', 'Content-Length': '0'} 2024-04-16T19:46:26.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b27497233f19292-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:26.390 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.390 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.390 DEBUG:boto:Method: GET 2024-04-16T19:46:26.390 DEBUG:boto:Path: /yrjefh-38/obj6 2024-04-16T19:46:26.390 DEBUG:boto:Data: 2024-04-16T19:46:26.390 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.390 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.390 DEBUG:boto:Params: {} 2024-04-16T19:46:26.390 DEBUG:boto:Token: None 2024-04-16T19:46:26.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6 2024-04-16T19:46:26.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OosGRpDX9QL1Ts2LlyOhiOJFXos= 2024-04-16T19:46:26.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OosGRpDX9QL1Ts2LlyOhiOJFXos=', 'Content-Length': '0'} 2024-04-16T19:46:26.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016d18f36d4dce3f4-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.434 DEBUG:boto:path=/yrjefh-38/obj6 2024-04-16T19:46:26.434 DEBUG:boto:auth_path=/yrjefh-38/obj6 2024-04-16T19:46:26.435 DEBUG:boto:Method: GET 2024-04-16T19:46:26.435 DEBUG:boto:Path: /yrjefh-38/obj6 2024-04-16T19:46:26.435 DEBUG:boto:Data: 2024-04-16T19:46:26.435 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.435 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.435 DEBUG:boto:Params: {} 2024-04-16T19:46:26.435 DEBUG:boto:Token: None 2024-04-16T19:46:26.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj6 2024-04-16T19:46:26.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OosGRpDX9QL1Ts2LlyOhiOJFXos= 2024-04-16T19:46:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OosGRpDX9QL1Ts2LlyOhiOJFXos=', 'Content-Length': '0'} 2024-04-16T19:46:26.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f84332fa2686bcb3-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.438 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.438 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.439 DEBUG:boto:path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.439 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.439 DEBUG:boto:Method: GET 2024-04-16T19:46:26.439 DEBUG:boto:Path: /yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.439 DEBUG:boto:Data: 2024-04-16T19:46:26.439 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.439 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.439 DEBUG:boto:Params: {} 2024-04-16T19:46:26.439 DEBUG:boto:Token: None 2024-04-16T19:46:26.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6d0d5b0f5e7e71-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.442 DEBUG:boto:b'yrjefh-38obj61000falseobj62024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.443 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.443 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.443 DEBUG:boto:path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.443 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.443 DEBUG:boto:Method: GET 2024-04-16T19:46:26.443 DEBUG:boto:Path: /yrjefh-38/?prefix=obj6 2024-04-16T19:46:26.443 DEBUG:boto:Data: 2024-04-16T19:46:26.443 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.444 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.444 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.444 DEBUG:boto:Params: {} 2024-04-16T19:46:26.444 DEBUG:boto:Token: None 2024-04-16T19:46:26.444 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a72a17eeddce50-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.446 DEBUG:boto:b'yrjefh-38obj61000falseobj62024-04-16T19:45:59.789Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:26.447 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.447 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.447 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.447 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.447 DEBUG:boto:Method: GET 2024-04-16T19:46:26.447 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.447 DEBUG:boto:Data: 2024-04-16T19:46:26.447 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.448 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.448 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.448 DEBUG:boto:Params: {} 2024-04-16T19:46:26.448 DEBUG:boto:Token: None 2024-04-16T19:46:26.448 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo= 2024-04-16T19:46:26.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo=', 'Content-Length': '0'} 2024-04-16T19:46:26.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:02 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc78c70ac8c7ff49-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.494 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.495 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.495 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.495 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.495 DEBUG:boto:Method: GET 2024-04-16T19:46:26.495 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.495 DEBUG:boto:Data: 2024-04-16T19:46:26.495 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.495 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.495 DEBUG:boto:Params: {} 2024-04-16T19:46:26.495 DEBUG:boto:Token: None 2024-04-16T19:46:26.496 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo= 2024-04-16T19:46:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo=', 'Content-Length': '0'} 2024-04-16T19:46:26.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aece998f404f50d4-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.498 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.498 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.498 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.499 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.499 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.499 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.499 DEBUG:boto:Data: 2024-04-16T19:46:26.499 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.499 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.499 DEBUG:boto:Params: {} 2024-04-16T19:46:26.499 DEBUG:boto:Token: None 2024-04-16T19:46:26.499 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/pDgYvD4IkV5PSEuZ264kZ/P64= 2024-04-16T19:46:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/pDgYvD4IkV5PSEuZ264kZ/P64=', 'Content-Length': '0'} 2024-04-16T19:46:26.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:02 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6b117524e226bdf-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.502 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.502 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.502 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.502 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.502 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.502 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.503 DEBUG:boto:Data: 2024-04-16T19:46:26.503 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.503 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.503 DEBUG:boto:Params: {} 2024-04-16T19:46:26.503 DEBUG:boto:Token: None 2024-04-16T19:46:26.503 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/pDgYvD4IkV5PSEuZ264kZ/P64= 2024-04-16T19:46:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/pDgYvD4IkV5PSEuZ264kZ/P64=', 'Content-Length': '0'} 2024-04-16T19:46:26.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044217712d19c2f53-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:26.506 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.506 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.506 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.506 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.506 DEBUG:boto:Method: GET 2024-04-16T19:46:26.506 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.507 DEBUG:boto:Data: 2024-04-16T19:46:26.507 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.507 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.507 DEBUG:boto:Params: {} 2024-04-16T19:46:26.507 DEBUG:boto:Token: None 2024-04-16T19:46:26.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo= 2024-04-16T19:46:26.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo=', 'Content-Length': '0'} 2024-04-16T19:46:26.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:02 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5562da4e44715d4-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.550 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.550 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.551 DEBUG:boto:path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.551 DEBUG:boto:auth_path=/yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.551 DEBUG:boto:Method: GET 2024-04-16T19:46:26.551 DEBUG:boto:Path: /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.551 DEBUG:boto:Data: 2024-04-16T19:46:26.551 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.551 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.551 DEBUG:boto:Params: {} 2024-04-16T19:46:26.551 DEBUG:boto:Token: None 2024-04-16T19:46:26.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7?versionId=null 2024-04-16T19:46:26.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo= 2024-04-16T19:46:26.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C4xvzzOIC2iHZ9sESHilHluV8jo=', 'Content-Length': '0'} 2024-04-16T19:46:26.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001480fb6eedf2d1c4-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.554 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.554 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.554 DEBUG:boto:path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.554 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.554 DEBUG:boto:Method: GET 2024-04-16T19:46:26.554 DEBUG:boto:Path: /yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.555 DEBUG:boto:Data: 2024-04-16T19:46:26.555 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.555 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.555 DEBUG:boto:Params: {} 2024-04-16T19:46:26.555 DEBUG:boto:Token: None 2024-04-16T19:46:26.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038819850a66837ac-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.558 DEBUG:boto:b'yrjefh-38obj71000falseobj72024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.558 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.558 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.559 DEBUG:boto:path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.559 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.559 DEBUG:boto:Method: GET 2024-04-16T19:46:26.559 DEBUG:boto:Path: /yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.559 DEBUG:boto:Data: 2024-04-16T19:46:26.559 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.559 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.559 DEBUG:boto:Params: {} 2024-04-16T19:46:26.559 DEBUG:boto:Token: None 2024-04-16T19:46:26.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a45bbbe8773255-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.562 DEBUG:boto:b'yrjefh-38obj71000falseobj72024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.563 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.563 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.563 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.563 DEBUG:boto:Path: /yrjefh-38/obj7 2024-04-16T19:46:26.563 DEBUG:boto:Data: 2024-04-16T19:46:26.563 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.563 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.563 DEBUG:boto:Params: {} 2024-04-16T19:46:26.563 DEBUG:boto:Token: None 2024-04-16T19:46:26.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7 2024-04-16T19:46:26.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b3NBlj4mOrEbChQ9F+LmgNmBsiA= 2024-04-16T19:46:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b3NBlj4mOrEbChQ9F+LmgNmBsiA=', 'Content-Length': '0'} 2024-04-16T19:46:26.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:02 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039daa4589b4dcb5e-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.566 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.566 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.566 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.566 DEBUG:boto:Path: /yrjefh-38/obj7 2024-04-16T19:46:26.566 DEBUG:boto:Data: 2024-04-16T19:46:26.567 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.567 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.567 DEBUG:boto:Params: {} 2024-04-16T19:46:26.567 DEBUG:boto:Token: None 2024-04-16T19:46:26.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7 2024-04-16T19:46:26.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b3NBlj4mOrEbChQ9F+LmgNmBsiA= 2024-04-16T19:46:26.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b3NBlj4mOrEbChQ9F+LmgNmBsiA=', 'Content-Length': '0'} 2024-04-16T19:46:26.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060640f7e10db9f89-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:26.570 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.570 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.570 DEBUG:boto:Method: GET 2024-04-16T19:46:26.570 DEBUG:boto:Path: /yrjefh-38/obj7 2024-04-16T19:46:26.570 DEBUG:boto:Data: 2024-04-16T19:46:26.570 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.570 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.570 DEBUG:boto:Params: {} 2024-04-16T19:46:26.570 DEBUG:boto:Token: None 2024-04-16T19:46:26.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7 2024-04-16T19:46:26.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nknUmF2PXwk9SQlSm+JrAIIBYnw= 2024-04-16T19:46:26.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nknUmF2PXwk9SQlSm+JrAIIBYnw=', 'Content-Length': '0'} 2024-04-16T19:46:26.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:02 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000695531e58b2a9120-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.618 DEBUG:boto:path=/yrjefh-38/obj7 2024-04-16T19:46:26.619 DEBUG:boto:auth_path=/yrjefh-38/obj7 2024-04-16T19:46:26.619 DEBUG:boto:Method: GET 2024-04-16T19:46:26.619 DEBUG:boto:Path: /yrjefh-38/obj7 2024-04-16T19:46:26.619 DEBUG:boto:Data: 2024-04-16T19:46:26.619 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.619 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.619 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.619 DEBUG:boto:Params: {} 2024-04-16T19:46:26.619 DEBUG:boto:Token: None 2024-04-16T19:46:26.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj7 2024-04-16T19:46:26.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nknUmF2PXwk9SQlSm+JrAIIBYnw= 2024-04-16T19:46:26.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nknUmF2PXwk9SQlSm+JrAIIBYnw=', 'Content-Length': '0'} 2024-04-16T19:46:26.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008083f9d52f0085ef-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.622 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.622 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.622 DEBUG:boto:path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.623 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.623 DEBUG:boto:Method: GET 2024-04-16T19:46:26.623 DEBUG:boto:Path: /yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.623 DEBUG:boto:Data: 2024-04-16T19:46:26.623 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.623 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.623 DEBUG:boto:Params: {} 2024-04-16T19:46:26.623 DEBUG:boto:Token: None 2024-04-16T19:46:26.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047127957ff706aef-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.666 DEBUG:boto:b'yrjefh-38obj71000falseobj72024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.667 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.667 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.667 DEBUG:boto:path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.667 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.667 DEBUG:boto:Method: GET 2024-04-16T19:46:26.667 DEBUG:boto:Path: /yrjefh-38/?prefix=obj7 2024-04-16T19:46:26.667 DEBUG:boto:Data: 2024-04-16T19:46:26.667 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.667 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.667 DEBUG:boto:Params: {} 2024-04-16T19:46:26.668 DEBUG:boto:Token: None 2024-04-16T19:46:26.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0206249fbe3b58-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.671 DEBUG:boto:b'yrjefh-38obj71000falseobj72024-04-16T19:45:59.798Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:26.671 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.671 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.671 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.671 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.672 DEBUG:boto:Method: GET 2024-04-16T19:46:26.672 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.672 DEBUG:boto:Data: 2024-04-16T19:46:26.672 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.672 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.672 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.672 DEBUG:boto:Params: {} 2024-04-16T19:46:26.672 DEBUG:boto:Token: None 2024-04-16T19:46:26.672 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro= 2024-04-16T19:46:26.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro=', 'Content-Length': '0'} 2024-04-16T19:46:26.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d9d6c37fde34665-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.718 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.718 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.719 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.719 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.719 DEBUG:boto:Method: GET 2024-04-16T19:46:26.719 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.719 DEBUG:boto:Data: 2024-04-16T19:46:26.719 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.719 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.719 DEBUG:boto:Params: {} 2024-04-16T19:46:26.719 DEBUG:boto:Token: None 2024-04-16T19:46:26.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro= 2024-04-16T19:46:26.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro=', 'Content-Length': '0'} 2024-04-16T19:46:26.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0b89466094dd9f1-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.722 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.723 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.723 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.723 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.723 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.723 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.723 DEBUG:boto:Data: 2024-04-16T19:46:26.723 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.723 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.723 DEBUG:boto:Params: {} 2024-04-16T19:46:26.723 DEBUG:boto:Token: None 2024-04-16T19:46:26.724 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZg7aZIsqoVT7vQVWGpiIVbpTLw= 2024-04-16T19:46:26.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZg7aZIsqoVT7vQVWGpiIVbpTLw=', 'Content-Length': '0'} 2024-04-16T19:46:26.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000352f23157e94de90-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.726 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.726 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.727 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.727 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.727 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.727 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.727 DEBUG:boto:Data: 2024-04-16T19:46:26.727 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.727 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.727 DEBUG:boto:Params: {} 2024-04-16T19:46:26.727 DEBUG:boto:Token: None 2024-04-16T19:46:26.727 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZg7aZIsqoVT7vQVWGpiIVbpTLw= 2024-04-16T19:46:26.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZg7aZIsqoVT7vQVWGpiIVbpTLw=', 'Content-Length': '0'} 2024-04-16T19:46:26.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c479cc4e9ec49436-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:26.731 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.731 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.731 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.731 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.731 DEBUG:boto:Method: GET 2024-04-16T19:46:26.731 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.731 DEBUG:boto:Data: 2024-04-16T19:46:26.731 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.732 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.732 DEBUG:boto:Params: {} 2024-04-16T19:46:26.732 DEBUG:boto:Token: None 2024-04-16T19:46:26.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro= 2024-04-16T19:46:26.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro=', 'Content-Length': '0'} 2024-04-16T19:46:26.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0129dff4c3ee0af-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.778 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.779 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.779 DEBUG:boto:path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.779 DEBUG:boto:auth_path=/yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.779 DEBUG:boto:Method: GET 2024-04-16T19:46:26.779 DEBUG:boto:Path: /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.779 DEBUG:boto:Data: 2024-04-16T19:46:26.779 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.779 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.779 DEBUG:boto:Params: {} 2024-04-16T19:46:26.780 DEBUG:boto:Token: None 2024-04-16T19:46:26.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8?versionId=null 2024-04-16T19:46:26.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro= 2024-04-16T19:46:26.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l382kZcUacPQkjNuwffevwV00Ro=', 'Content-Length': '0'} 2024-04-16T19:46:26.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000440dcc26bb760307-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.783 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.783 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.783 DEBUG:boto:path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.783 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.783 DEBUG:boto:Method: GET 2024-04-16T19:46:26.783 DEBUG:boto:Path: /yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.783 DEBUG:boto:Data: 2024-04-16T19:46:26.783 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.784 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.784 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.784 DEBUG:boto:Params: {} 2024-04-16T19:46:26.784 DEBUG:boto:Token: None 2024-04-16T19:46:26.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005caa6561fb23d9f9-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.834 DEBUG:boto:b'yrjefh-38obj81000falseobj82024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.835 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.835 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.835 DEBUG:boto:path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.835 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.835 DEBUG:boto:Method: GET 2024-04-16T19:46:26.835 DEBUG:boto:Path: /yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.835 DEBUG:boto:Data: 2024-04-16T19:46:26.836 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.836 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.836 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.836 DEBUG:boto:Params: {} 2024-04-16T19:46:26.836 DEBUG:boto:Token: None 2024-04-16T19:46:26.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cefc800c8a86cdea-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.839 DEBUG:boto:b'yrjefh-38obj81000falseobj82024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.839 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.839 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.840 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.840 DEBUG:boto:Path: /yrjefh-38/obj8 2024-04-16T19:46:26.840 DEBUG:boto:Data: 2024-04-16T19:46:26.840 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.840 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.840 DEBUG:boto:Params: {} 2024-04-16T19:46:26.840 DEBUG:boto:Token: None 2024-04-16T19:46:26.840 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8 2024-04-16T19:46:26.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kBDi6lk+bAW9xSXC42GpZzCCfKk= 2024-04-16T19:46:26.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kBDi6lk+bAW9xSXC42GpZzCCfKk=', 'Content-Length': '0'} 2024-04-16T19:46:26.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b89139b32a8452e5-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.843 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.843 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.843 DEBUG:boto:Method: HEAD 2024-04-16T19:46:26.843 DEBUG:boto:Path: /yrjefh-38/obj8 2024-04-16T19:46:26.843 DEBUG:boto:Data: 2024-04-16T19:46:26.844 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.844 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.844 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.844 DEBUG:boto:Params: {} 2024-04-16T19:46:26.844 DEBUG:boto:Token: None 2024-04-16T19:46:26.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8 2024-04-16T19:46:26.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kBDi6lk+bAW9xSXC42GpZzCCfKk= 2024-04-16T19:46:26.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kBDi6lk+bAW9xSXC42GpZzCCfKk=', 'Content-Length': '0'} 2024-04-16T19:46:26.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006587bfb7d05a01ed-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:26.847 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.847 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.847 DEBUG:boto:Method: GET 2024-04-16T19:46:26.847 DEBUG:boto:Path: /yrjefh-38/obj8 2024-04-16T19:46:26.847 DEBUG:boto:Data: 2024-04-16T19:46:26.847 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.847 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.848 DEBUG:boto:Params: {} 2024-04-16T19:46:26.848 DEBUG:boto:Token: None 2024-04-16T19:46:26.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8 2024-04-16T19:46:26.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X43Ejxr+2dS2FGiZKhA5rJkMmfo= 2024-04-16T19:46:26.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X43Ejxr+2dS2FGiZKhA5rJkMmfo=', 'Content-Length': '0'} 2024-04-16T19:46:26.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-38:b482a2a9-ef36-488b-9666-ada656453328.4812.7'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:01 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e53bb1b1cf59665-00661ed592-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.890 DEBUG:boto:path=/yrjefh-38/obj8 2024-04-16T19:46:26.891 DEBUG:boto:auth_path=/yrjefh-38/obj8 2024-04-16T19:46:26.891 DEBUG:boto:Method: GET 2024-04-16T19:46:26.891 DEBUG:boto:Path: /yrjefh-38/obj8 2024-04-16T19:46:26.891 DEBUG:boto:Data: 2024-04-16T19:46:26.891 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.891 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.891 DEBUG:boto:Params: {} 2024-04-16T19:46:26.891 DEBUG:boto:Token: None 2024-04-16T19:46:26.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/obj8 2024-04-16T19:46:26.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X43Ejxr+2dS2FGiZKhA5rJkMmfo= 2024-04-16T19:46:26.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X43Ejxr+2dS2FGiZKhA5rJkMmfo=', 'Content-Length': '0'} 2024-04-16T19:46:26.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001781e8a8459bd3f8-00661ed592-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.894 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.894 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.895 DEBUG:boto:path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.895 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.895 DEBUG:boto:Method: GET 2024-04-16T19:46:26.895 DEBUG:boto:Path: /yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.895 DEBUG:boto:Data: 2024-04-16T19:46:26.895 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:26.895 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.895 DEBUG:boto:Params: {} 2024-04-16T19:46:26.895 DEBUG:boto:Token: None 2024-04-16T19:46:26.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ac7db1d42e62e9e-00661ed592-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.942 DEBUG:boto:b'yrjefh-38obj81000falseobj82024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.943 DEBUG:boto:path=/yrjefh-38/ 2024-04-16T19:46:26.943 DEBUG:boto:auth_path=/yrjefh-38/ 2024-04-16T19:46:26.943 DEBUG:boto:path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.943 DEBUG:boto:auth_path=/yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.943 DEBUG:boto:Method: GET 2024-04-16T19:46:26.943 DEBUG:boto:Path: /yrjefh-38/?prefix=obj8 2024-04-16T19:46:26.943 DEBUG:boto:Data: 2024-04-16T19:46:26.943 DEBUG:boto:Headers: {} 2024-04-16T19:46:26.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:26.944 DEBUG:boto:Port: 8000 2024-04-16T19:46:26.944 DEBUG:boto:Params: {} 2024-04-16T19:46:26.944 DEBUG:boto:Token: None 2024-04-16T19:46:26.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:26 GMT /yrjefh-38/ 2024-04-16T19:46:26.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4= 2024-04-16T19:46:26.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JNYM8wEAawSvRIVcXhotLiT7Lu4=', 'Content-Length': '0'} 2024-04-16T19:46:26.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd884f4389f9854-00661ed592-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:26.947 DEBUG:boto:b'yrjefh-38obj81000falseobj82024-04-16T19:45:59.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:26.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-38 zones={test-zone2, test-zone1} 2024-04-16T19:46:26.948 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-39 2024-04-16T19:46:26.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:26.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:26.948 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-39 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:27.013 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:27.013 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:27.071+0000 7f942652a800 1 waiting for incremental sync to catch up: 2024-04-16T19:46:27.073 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.33.1, 00000000002.103.1, 00000000003.26.1, 00000000002.33.1, 00000000003.18.1, 00000000002.104.1, 00000000002.26.1, 00000000004.42.6, 00000000002.34.1, 00000000002.16.1, 00000000002.38.1] 2024-04-16T19:46:27.073 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.36.6, 00000000002.103.1, 00000000004.28.6, 00000000002.33.1, 00000000004.20.6, 00000000002.104.1, 00000000002.26.1, 00000000004.42.6, 00000000002.34.1, 00000000002.16.1, 00000000002.38.1] 2024-04-16T19:46:32.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:32.072+0000 7f942652a800 1 bucket sync caught up with source: 2024-04-16T19:46:32.074 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.36.6, 00000000002.103.1, 00000000004.28.6, 00000000002.33.1, 00000000004.20.6, 00000000002.104.1, 00000000002.26.1, 00000000004.42.6, 00000000002.34.1, 00000000002.16.1, 00000000002.38.1] 2024-04-16T19:46:32.074 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.36.6, 00000000002.103.1, 00000000004.28.6, 00000000002.33.1, 00000000004.20.6, 00000000002.104.1, 00000000002.26.1, 00000000004.42.6, 00000000002.34.1, 00000000002.16.1, 00000000002.38.1] 2024-04-16T19:46:32.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:32.072+0000 7f942652a800 0 bucket checkpoint complete 2024-04-16T19:46:32.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-39 2024-04-16T19:46:32.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:32.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-39', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:32.100 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-39 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:32.223+0000 7f0d72730800 1 bucket sync caught up with source: 2024-04-16T19:46:32.225 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.65.6, , 00000000002.121.6, , 00000000002.106.6, , , 00000000002.143.6, , , ] 2024-04-16T19:46:32.225 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.65.6, , 00000000002.121.6, , 00000000002.106.6, , , 00000000002.143.6, , , ] 2024-04-16T19:46:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:32.223+0000 7f0d72730800 0 bucket checkpoint complete 2024-04-16T19:46:32.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-39 zones={test-zone2, test-zone1} 2024-04-16T19:46:32.247 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.247 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.247 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.247 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:46:32.247 DEBUG:boto:Data: 2024-04-16T19:46:32.247 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.247 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.247 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.247 DEBUG:boto:Params: {} 2024-04-16T19:46:32.248 DEBUG:boto:Token: None 2024-04-16T19:46:32.248 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ShIJO0pZGZCMfDuu0/7NwZe05oE= 2024-04-16T19:46:32.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ShIJO0pZGZCMfDuu0/7NwZe05oE=', 'Content-Length': '0'} 2024-04-16T19:46:32.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c14a9c790db7ecd-00661ed598-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.250 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.250 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.250 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.250 DEBUG:boto:Path: /yrjefh-39/ 2024-04-16T19:46:32.251 DEBUG:boto:Data: 2024-04-16T19:46:32.251 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.251 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.251 DEBUG:boto:Params: {} 2024-04-16T19:46:32.251 DEBUG:boto:Token: None 2024-04-16T19:46:32.251 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ShIJO0pZGZCMfDuu0/7NwZe05oE= 2024-04-16T19:46:32.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ShIJO0pZGZCMfDuu0/7NwZe05oE=', 'Content-Length': '0'} 2024-04-16T19:46:32.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000508f65412a44e104-00661ed598-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.254 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:46:32.254 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.254 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.254 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.254 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.254 DEBUG:boto:Method: GET 2024-04-16T19:46:32.254 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.254 DEBUG:boto:Data: 2024-04-16T19:46:32.254 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.254 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.254 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.255 DEBUG:boto:Params: {} 2024-04-16T19:46:32.255 DEBUG:boto:Token: None 2024-04-16T19:46:32.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/?versions 2024-04-16T19:46:32.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk= 2024-04-16T19:46:32.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk=', 'Content-Length': '0'} 2024-04-16T19:46:32.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e561026ef255319-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.257 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:46:32.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:46:32.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:46:32.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:46:32.259 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.259 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.259 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.259 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.259 DEBUG:boto:Method: GET 2024-04-16T19:46:32.259 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.259 DEBUG:boto:Data: 2024-04-16T19:46:32.259 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.259 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.260 DEBUG:boto:Params: {} 2024-04-16T19:46:32.260 DEBUG:boto:Token: None 2024-04-16T19:46:32.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/?versions 2024-04-16T19:46:32.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk= 2024-04-16T19:46:32.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk=', 'Content-Length': '0'} 2024-04-16T19:46:32.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd63005c67d6259-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.262 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:46:32.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:46:32.263 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.264 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.264 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.264 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.264 DEBUG:boto:Method: GET 2024-04-16T19:46:32.264 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.264 DEBUG:boto:Data: 2024-04-16T19:46:32.264 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.264 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.264 DEBUG:boto:Params: {} 2024-04-16T19:46:32.264 DEBUG:boto:Token: None 2024-04-16T19:46:32.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/?versions 2024-04-16T19:46:32.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk= 2024-04-16T19:46:32.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk=', 'Content-Length': '0'} 2024-04-16T19:46:32.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003558fa9bf3f8890-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.267 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.268 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.268 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.268 DEBUG:boto:path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.268 DEBUG:boto:auth_path=/yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.268 DEBUG:boto:Method: GET 2024-04-16T19:46:32.268 DEBUG:boto:Path: /yrjefh-39/?versions&max-keys=999 2024-04-16T19:46:32.268 DEBUG:boto:Data: 2024-04-16T19:46:32.268 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.268 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.268 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.268 DEBUG:boto:Params: {} 2024-04-16T19:46:32.268 DEBUG:boto:Token: None 2024-04-16T19:46:32.269 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/?versions 2024-04-16T19:46:32.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk= 2024-04-16T19:46:32.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvsJrw4SY0LT+PqlEYu3q7UHYLk=', 'Content-Length': '0'} 2024-04-16T19:46:32.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23a4378f561820b-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.271 DEBUG:boto:b'yrjefh-39999falseobj1nulltrue2024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:32.272 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.272 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.272 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.272 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.272 DEBUG:boto:Method: GET 2024-04-16T19:46:32.272 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.272 DEBUG:boto:Data: 2024-04-16T19:46:32.272 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.272 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.272 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.273 DEBUG:boto:Params: {} 2024-04-16T19:46:32.273 DEBUG:boto:Token: None 2024-04-16T19:46:32.273 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0= 2024-04-16T19:46:32.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0=', 'Content-Length': '0'} 2024-04-16T19:46:32.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1997ecbe243ce54-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.275 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.275 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.276 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.276 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.276 DEBUG:boto:Method: GET 2024-04-16T19:46:32.276 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.276 DEBUG:boto:Data: 2024-04-16T19:46:32.276 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.276 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.276 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.276 DEBUG:boto:Params: {} 2024-04-16T19:46:32.276 DEBUG:boto:Token: None 2024-04-16T19:46:32.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.277 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0= 2024-04-16T19:46:32.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0=', 'Content-Length': '0'} 2024-04-16T19:46:32.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000441597a5bad8d2c5-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.322 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.322 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.322 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.323 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.323 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.323 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.323 DEBUG:boto:Data: 2024-04-16T19:46:32.323 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.323 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.323 DEBUG:boto:Params: {} 2024-04-16T19:46:32.323 DEBUG:boto:Token: None 2024-04-16T19:46:32.323 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YlIIX7YgmnPX8nhuxuQOYAAW2ok= 2024-04-16T19:46:32.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YlIIX7YgmnPX8nhuxuQOYAAW2ok=', 'Content-Length': '0'} 2024-04-16T19:46:32.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b40133d325e0b9d1-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.326 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.326 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.326 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.326 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.326 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.326 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.326 DEBUG:boto:Data: 2024-04-16T19:46:32.326 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.326 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.326 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.327 DEBUG:boto:Params: {} 2024-04-16T19:46:32.327 DEBUG:boto:Token: None 2024-04-16T19:46:32.327 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YlIIX7YgmnPX8nhuxuQOYAAW2ok= 2024-04-16T19:46:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YlIIX7YgmnPX8nhuxuQOYAAW2ok=', 'Content-Length': '0'} 2024-04-16T19:46:32.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002010d753e30e9cd2-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:32.329 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.329 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.329 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.329 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.330 DEBUG:boto:Method: GET 2024-04-16T19:46:32.330 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.330 DEBUG:boto:Data: 2024-04-16T19:46:32.330 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.330 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.330 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.330 DEBUG:boto:Params: {} 2024-04-16T19:46:32.330 DEBUG:boto:Token: None 2024-04-16T19:46:32.330 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.330 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0= 2024-04-16T19:46:32.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0=', 'Content-Length': '0'} 2024-04-16T19:46:32.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa3763f98c2e4e68-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.333 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.333 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.333 DEBUG:boto:path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.333 DEBUG:boto:auth_path=/yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.333 DEBUG:boto:Method: GET 2024-04-16T19:46:32.333 DEBUG:boto:Path: /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.333 DEBUG:boto:Data: 2024-04-16T19:46:32.334 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.334 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.334 DEBUG:boto:Params: {} 2024-04-16T19:46:32.334 DEBUG:boto:Token: None 2024-04-16T19:46:32.334 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1?versionId=null 2024-04-16T19:46:32.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0= 2024-04-16T19:46:32.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XlAtO1NediVl2nzYnzPx5EtWgw0=', 'Content-Length': '0'} 2024-04-16T19:46:32.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2d906b9a12f9259-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.336 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.337 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.337 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.337 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.337 DEBUG:boto:Method: GET 2024-04-16T19:46:32.337 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.337 DEBUG:boto:Data: 2024-04-16T19:46:32.337 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.337 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.337 DEBUG:boto:Params: {} 2024-04-16T19:46:32.337 DEBUG:boto:Token: None 2024-04-16T19:46:32.337 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb430e7551f6a38-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.340 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.341 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.341 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.341 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.341 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.341 DEBUG:boto:Method: GET 2024-04-16T19:46:32.341 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.341 DEBUG:boto:Data: 2024-04-16T19:46:32.341 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.341 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.341 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.341 DEBUG:boto:Params: {} 2024-04-16T19:46:32.341 DEBUG:boto:Token: None 2024-04-16T19:46:32.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a888cd2eb8a010a-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.344 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.344 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.345 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.345 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.345 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:46:32.345 DEBUG:boto:Data: 2024-04-16T19:46:32.345 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.345 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.345 DEBUG:boto:Params: {} 2024-04-16T19:46:32.345 DEBUG:boto:Token: None 2024-04-16T19:46:32.345 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1 2024-04-16T19:46:32.345 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9Y/BUlyVBFmVOPUp8utxT9rDFo= 2024-04-16T19:46:32.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9Y/BUlyVBFmVOPUp8utxT9rDFo=', 'Content-Length': '0'} 2024-04-16T19:46:32.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5aa01c0e20dc0da-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.347 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.347 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.348 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.348 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:46:32.348 DEBUG:boto:Data: 2024-04-16T19:46:32.348 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.348 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.348 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.348 DEBUG:boto:Params: {} 2024-04-16T19:46:32.348 DEBUG:boto:Token: None 2024-04-16T19:46:32.348 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1 2024-04-16T19:46:32.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p9Y/BUlyVBFmVOPUp8utxT9rDFo= 2024-04-16T19:46:32.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p9Y/BUlyVBFmVOPUp8utxT9rDFo=', 'Content-Length': '0'} 2024-04-16T19:46:32.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e52f151f5777aed-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:32.350 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.351 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.351 DEBUG:boto:Method: GET 2024-04-16T19:46:32.351 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:46:32.351 DEBUG:boto:Data: 2024-04-16T19:46:32.351 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.351 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.351 DEBUG:boto:Params: {} 2024-04-16T19:46:32.351 DEBUG:boto:Token: None 2024-04-16T19:46:32.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1 2024-04-16T19:46:32.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Tozmp96rAwoMm+AM0NA04MwO4M= 2024-04-16T19:46:32.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Tozmp96rAwoMm+AM0NA04MwO4M=', 'Content-Length': '0'} 2024-04-16T19:46:32.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054519d1322f97d93-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.398 DEBUG:boto:path=/yrjefh-39/obj1 2024-04-16T19:46:32.398 DEBUG:boto:auth_path=/yrjefh-39/obj1 2024-04-16T19:46:32.398 DEBUG:boto:Method: GET 2024-04-16T19:46:32.399 DEBUG:boto:Path: /yrjefh-39/obj1 2024-04-16T19:46:32.399 DEBUG:boto:Data: 2024-04-16T19:46:32.399 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.399 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.399 DEBUG:boto:Params: {} 2024-04-16T19:46:32.399 DEBUG:boto:Token: None 2024-04-16T19:46:32.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj1 2024-04-16T19:46:32.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Tozmp96rAwoMm+AM0NA04MwO4M= 2024-04-16T19:46:32.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Tozmp96rAwoMm+AM0NA04MwO4M=', 'Content-Length': '0'} 2024-04-16T19:46:32.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e0f251050597eef-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.442 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.442 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.442 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.443 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.443 DEBUG:boto:Method: GET 2024-04-16T19:46:32.443 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.443 DEBUG:boto:Data: 2024-04-16T19:46:32.443 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.443 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.443 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.443 DEBUG:boto:Params: {} 2024-04-16T19:46:32.443 DEBUG:boto:Token: None 2024-04-16T19:46:32.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007246fb9cdb47cfd8-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.446 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.447 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.447 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.447 DEBUG:boto:path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.447 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.447 DEBUG:boto:Method: GET 2024-04-16T19:46:32.447 DEBUG:boto:Path: /yrjefh-39/?prefix=obj1 2024-04-16T19:46:32.447 DEBUG:boto:Data: 2024-04-16T19:46:32.447 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.447 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.447 DEBUG:boto:Params: {} 2024-04-16T19:46:32.448 DEBUG:boto:Token: None 2024-04-16T19:46:32.448 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bda7e2dbbbb160d-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.451 DEBUG:boto:b'yrjefh-39obj11000falseobj12024-04-16T19:45:51.567Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:32.451 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.451 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.451 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.451 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.451 DEBUG:boto:Method: GET 2024-04-16T19:46:32.452 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.452 DEBUG:boto:Data: 2024-04-16T19:46:32.452 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.452 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.452 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.452 DEBUG:boto:Params: {} 2024-04-16T19:46:32.452 DEBUG:boto:Token: None 2024-04-16T19:46:32.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA= 2024-04-16T19:46:32.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA=', 'Content-Length': '0'} 2024-04-16T19:46:32.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083f0ed456d1137d2-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.455 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.455 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.455 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.455 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.455 DEBUG:boto:Method: GET 2024-04-16T19:46:32.455 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.455 DEBUG:boto:Data: 2024-04-16T19:46:32.456 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.456 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.456 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.456 DEBUG:boto:Params: {} 2024-04-16T19:46:32.456 DEBUG:boto:Token: None 2024-04-16T19:46:32.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA= 2024-04-16T19:46:32.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA=', 'Content-Length': '0'} 2024-04-16T19:46:32.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee2c7db19d31f03f-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.498 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.499 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.499 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.499 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.499 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.499 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.499 DEBUG:boto:Data: 2024-04-16T19:46:32.499 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.499 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.499 DEBUG:boto:Params: {} 2024-04-16T19:46:32.499 DEBUG:boto:Token: None 2024-04-16T19:46:32.500 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Llq2KECKqP/zwiQUGGpOoGnmygQ= 2024-04-16T19:46:32.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Llq2KECKqP/zwiQUGGpOoGnmygQ=', 'Content-Length': '0'} 2024-04-16T19:46:32.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c46b93ee53cee6c2-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.502 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.502 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.502 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.502 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.502 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.502 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.502 DEBUG:boto:Data: 2024-04-16T19:46:32.502 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.502 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.502 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.502 DEBUG:boto:Params: {} 2024-04-16T19:46:32.503 DEBUG:boto:Token: None 2024-04-16T19:46:32.503 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Llq2KECKqP/zwiQUGGpOoGnmygQ= 2024-04-16T19:46:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Llq2KECKqP/zwiQUGGpOoGnmygQ=', 'Content-Length': '0'} 2024-04-16T19:46:32.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000e5cf6d5b92ff13-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:32.505 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.505 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.505 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.505 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.505 DEBUG:boto:Method: GET 2024-04-16T19:46:32.505 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.505 DEBUG:boto:Data: 2024-04-16T19:46:32.505 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.505 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.505 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.506 DEBUG:boto:Params: {} 2024-04-16T19:46:32.506 DEBUG:boto:Token: None 2024-04-16T19:46:32.506 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.506 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA= 2024-04-16T19:46:32.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA=', 'Content-Length': '0'} 2024-04-16T19:46:32.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f87ad3f0adf967ad-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.508 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.508 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.508 DEBUG:boto:path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.508 DEBUG:boto:auth_path=/yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.508 DEBUG:boto:Method: GET 2024-04-16T19:46:32.508 DEBUG:boto:Path: /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.509 DEBUG:boto:Data: 2024-04-16T19:46:32.509 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.509 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.509 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.509 DEBUG:boto:Params: {} 2024-04-16T19:46:32.509 DEBUG:boto:Token: None 2024-04-16T19:46:32.509 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2?versionId=null 2024-04-16T19:46:32.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA= 2024-04-16T19:46:32.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1OT0n8kg1WaGgk5OxzuSDDRAmIA=', 'Content-Length': '0'} 2024-04-16T19:46:32.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b3c3980f0d768f0-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.554 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.554 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.555 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.555 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.555 DEBUG:boto:Method: GET 2024-04-16T19:46:32.555 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.555 DEBUG:boto:Data: 2024-04-16T19:46:32.555 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.555 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.555 DEBUG:boto:Params: {} 2024-04-16T19:46:32.555 DEBUG:boto:Token: None 2024-04-16T19:46:32.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e76bb8de84c10b54-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.558 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.559 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.559 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.559 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.559 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.559 DEBUG:boto:Method: GET 2024-04-16T19:46:32.559 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.559 DEBUG:boto:Data: 2024-04-16T19:46:32.559 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.560 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.560 DEBUG:boto:Params: {} 2024-04-16T19:46:32.560 DEBUG:boto:Token: None 2024-04-16T19:46:32.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd354fda7948b3ed-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.563 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.563 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.563 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.563 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.563 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:46:32.563 DEBUG:boto:Data: 2024-04-16T19:46:32.563 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.564 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.564 DEBUG:boto:Params: {} 2024-04-16T19:46:32.564 DEBUG:boto:Token: None 2024-04-16T19:46:32.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2 2024-04-16T19:46:32.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y8k6RcR+L8IZ12r3wAO2sdzjDWc= 2024-04-16T19:46:32.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y8k6RcR+L8IZ12r3wAO2sdzjDWc=', 'Content-Length': '0'} 2024-04-16T19:46:32.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041bf481ce332b7c5-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.566 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.566 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.566 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.566 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:46:32.566 DEBUG:boto:Data: 2024-04-16T19:46:32.566 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.567 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.567 DEBUG:boto:Params: {} 2024-04-16T19:46:32.567 DEBUG:boto:Token: None 2024-04-16T19:46:32.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2 2024-04-16T19:46:32.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y8k6RcR+L8IZ12r3wAO2sdzjDWc= 2024-04-16T19:46:32.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y8k6RcR+L8IZ12r3wAO2sdzjDWc=', 'Content-Length': '0'} 2024-04-16T19:46:32.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000995d6896a187e981-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:32.569 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.569 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.569 DEBUG:boto:Method: GET 2024-04-16T19:46:32.569 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:46:32.570 DEBUG:boto:Data: 2024-04-16T19:46:32.570 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.570 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.570 DEBUG:boto:Params: {} 2024-04-16T19:46:32.570 DEBUG:boto:Token: None 2024-04-16T19:46:32.570 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2 2024-04-16T19:46:32.570 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EAF9fghBQ8PZTPMddfe64nDkHFA= 2024-04-16T19:46:32.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EAF9fghBQ8PZTPMddfe64nDkHFA=', 'Content-Length': '0'} 2024-04-16T19:46:32.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd21e8ff060bac7e-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.572 DEBUG:boto:path=/yrjefh-39/obj2 2024-04-16T19:46:32.573 DEBUG:boto:auth_path=/yrjefh-39/obj2 2024-04-16T19:46:32.573 DEBUG:boto:Method: GET 2024-04-16T19:46:32.573 DEBUG:boto:Path: /yrjefh-39/obj2 2024-04-16T19:46:32.573 DEBUG:boto:Data: 2024-04-16T19:46:32.573 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.573 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.573 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.573 DEBUG:boto:Params: {} 2024-04-16T19:46:32.573 DEBUG:boto:Token: None 2024-04-16T19:46:32.573 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj2 2024-04-16T19:46:32.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EAF9fghBQ8PZTPMddfe64nDkHFA= 2024-04-16T19:46:32.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EAF9fghBQ8PZTPMddfe64nDkHFA=', 'Content-Length': '0'} 2024-04-16T19:46:32.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edffebab419666b0-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.622 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.622 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.623 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.623 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.623 DEBUG:boto:Method: GET 2024-04-16T19:46:32.623 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.623 DEBUG:boto:Data: 2024-04-16T19:46:32.623 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.623 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.623 DEBUG:boto:Params: {} 2024-04-16T19:46:32.623 DEBUG:boto:Token: None 2024-04-16T19:46:32.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b617c148cac90212-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.626 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.626 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.627 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.627 DEBUG:boto:path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.627 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.627 DEBUG:boto:Method: GET 2024-04-16T19:46:32.627 DEBUG:boto:Path: /yrjefh-39/?prefix=obj2 2024-04-16T19:46:32.627 DEBUG:boto:Data: 2024-04-16T19:46:32.627 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.627 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.627 DEBUG:boto:Params: {} 2024-04-16T19:46:32.627 DEBUG:boto:Token: None 2024-04-16T19:46:32.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004883293914036673-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.630 DEBUG:boto:b'yrjefh-39obj21000falseobj22024-04-16T19:45:51.579Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:32.630 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.630 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.631 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.631 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.631 DEBUG:boto:Method: GET 2024-04-16T19:46:32.631 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.631 DEBUG:boto:Data: 2024-04-16T19:46:32.631 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.631 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.631 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.631 DEBUG:boto:Params: {} 2024-04-16T19:46:32.631 DEBUG:boto:Token: None 2024-04-16T19:46:32.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc= 2024-04-16T19:46:32.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc=', 'Content-Length': '0'} 2024-04-16T19:46:32.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058c89c70520c2baf-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.634 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.634 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.634 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.634 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.634 DEBUG:boto:Method: GET 2024-04-16T19:46:32.634 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.634 DEBUG:boto:Data: 2024-04-16T19:46:32.634 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.634 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.635 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.635 DEBUG:boto:Params: {} 2024-04-16T19:46:32.635 DEBUG:boto:Token: None 2024-04-16T19:46:32.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc= 2024-04-16T19:46:32.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc=', 'Content-Length': '0'} 2024-04-16T19:46:32.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e4523b181407618-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.678 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.678 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.678 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.679 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.679 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.679 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.679 DEBUG:boto:Data: 2024-04-16T19:46:32.679 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.679 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.679 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.679 DEBUG:boto:Params: {} 2024-04-16T19:46:32.679 DEBUG:boto:Token: None 2024-04-16T19:46:32.679 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80sLma4B7eK8raLac19u6FM5sk0= 2024-04-16T19:46:32.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80sLma4B7eK8raLac19u6FM5sk0=', 'Content-Length': '0'} 2024-04-16T19:46:32.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a766fd5d976ece6d-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.681 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.682 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.682 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.682 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.682 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.682 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.682 DEBUG:boto:Data: 2024-04-16T19:46:32.682 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.682 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.682 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.682 DEBUG:boto:Params: {} 2024-04-16T19:46:32.682 DEBUG:boto:Token: None 2024-04-16T19:46:32.683 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80sLma4B7eK8raLac19u6FM5sk0= 2024-04-16T19:46:32.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80sLma4B7eK8raLac19u6FM5sk0=', 'Content-Length': '0'} 2024-04-16T19:46:32.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009179eddaf1f83311-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:32.685 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.685 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.685 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.685 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.685 DEBUG:boto:Method: GET 2024-04-16T19:46:32.685 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.685 DEBUG:boto:Data: 2024-04-16T19:46:32.685 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.686 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.686 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.686 DEBUG:boto:Params: {} 2024-04-16T19:46:32.686 DEBUG:boto:Token: None 2024-04-16T19:46:32.686 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.686 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc= 2024-04-16T19:46:32.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc=', 'Content-Length': '0'} 2024-04-16T19:46:32.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d91b5f3bc3ae4ec-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.688 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.689 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.689 DEBUG:boto:path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.689 DEBUG:boto:auth_path=/yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.689 DEBUG:boto:Method: GET 2024-04-16T19:46:32.689 DEBUG:boto:Path: /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.689 DEBUG:boto:Data: 2024-04-16T19:46:32.689 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.689 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.689 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.689 DEBUG:boto:Params: {} 2024-04-16T19:46:32.689 DEBUG:boto:Token: None 2024-04-16T19:46:32.689 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3?versionId=null 2024-04-16T19:46:32.690 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc= 2024-04-16T19:46:32.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:53Pks7Frb5O+/HNXDNWEmrfcKEc=', 'Content-Length': '0'} 2024-04-16T19:46:32.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d21a97d3576485a3-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.734 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.734 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.734 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.735 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.735 DEBUG:boto:Method: GET 2024-04-16T19:46:32.735 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.735 DEBUG:boto:Data: 2024-04-16T19:46:32.735 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.735 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.735 DEBUG:boto:Params: {} 2024-04-16T19:46:32.735 DEBUG:boto:Token: None 2024-04-16T19:46:32.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5aefc53afb705e5-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.738 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.738 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.738 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.738 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.738 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.739 DEBUG:boto:Method: GET 2024-04-16T19:46:32.739 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.739 DEBUG:boto:Data: 2024-04-16T19:46:32.739 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.739 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.739 DEBUG:boto:Params: {} 2024-04-16T19:46:32.739 DEBUG:boto:Token: None 2024-04-16T19:46:32.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7f233fe9cab4ed-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.782 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.783 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.783 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.783 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.783 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:46:32.783 DEBUG:boto:Data: 2024-04-16T19:46:32.783 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.783 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.783 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.783 DEBUG:boto:Params: {} 2024-04-16T19:46:32.783 DEBUG:boto:Token: None 2024-04-16T19:46:32.783 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3 2024-04-16T19:46:32.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOkfzQ17M0w5cNmVowqACt+yzfg= 2024-04-16T19:46:32.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOkfzQ17M0w5cNmVowqACt+yzfg=', 'Content-Length': '0'} 2024-04-16T19:46:32.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ded2c1d84233ff7-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.785 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.786 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.786 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.786 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:46:32.786 DEBUG:boto:Data: 2024-04-16T19:46:32.786 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.786 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.786 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.786 DEBUG:boto:Params: {} 2024-04-16T19:46:32.786 DEBUG:boto:Token: None 2024-04-16T19:46:32.786 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3 2024-04-16T19:46:32.786 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jOkfzQ17M0w5cNmVowqACt+yzfg= 2024-04-16T19:46:32.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOkfzQ17M0w5cNmVowqACt+yzfg=', 'Content-Length': '0'} 2024-04-16T19:46:32.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b8e7d7fa19fa33-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:32.789 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.789 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.789 DEBUG:boto:Method: GET 2024-04-16T19:46:32.789 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:46:32.789 DEBUG:boto:Data: 2024-04-16T19:46:32.789 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.789 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.789 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.789 DEBUG:boto:Params: {} 2024-04-16T19:46:32.789 DEBUG:boto:Token: None 2024-04-16T19:46:32.790 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3 2024-04-16T19:46:32.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AGSpQcha7lShv91wPpwlTkY2tcQ= 2024-04-16T19:46:32.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AGSpQcha7lShv91wPpwlTkY2tcQ=', 'Content-Length': '0'} 2024-04-16T19:46:32.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abf6e87b8e5923c7-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.792 DEBUG:boto:path=/yrjefh-39/obj3 2024-04-16T19:46:32.792 DEBUG:boto:auth_path=/yrjefh-39/obj3 2024-04-16T19:46:32.792 DEBUG:boto:Method: GET 2024-04-16T19:46:32.792 DEBUG:boto:Path: /yrjefh-39/obj3 2024-04-16T19:46:32.792 DEBUG:boto:Data: 2024-04-16T19:46:32.792 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.793 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.793 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.793 DEBUG:boto:Params: {} 2024-04-16T19:46:32.793 DEBUG:boto:Token: None 2024-04-16T19:46:32.793 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj3 2024-04-16T19:46:32.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AGSpQcha7lShv91wPpwlTkY2tcQ= 2024-04-16T19:46:32.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AGSpQcha7lShv91wPpwlTkY2tcQ=', 'Content-Length': '0'} 2024-04-16T19:46:32.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d551960e699df7-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.838 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.838 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.838 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.839 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.839 DEBUG:boto:Method: GET 2024-04-16T19:46:32.839 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.839 DEBUG:boto:Data: 2024-04-16T19:46:32.839 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.839 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.839 DEBUG:boto:Params: {} 2024-04-16T19:46:32.839 DEBUG:boto:Token: None 2024-04-16T19:46:32.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb7eb6d76a34e47c-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.842 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.843 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.843 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.843 DEBUG:boto:path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.843 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.843 DEBUG:boto:Method: GET 2024-04-16T19:46:32.843 DEBUG:boto:Path: /yrjefh-39/?prefix=obj3 2024-04-16T19:46:32.843 DEBUG:boto:Data: 2024-04-16T19:46:32.843 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.843 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.843 DEBUG:boto:Params: {} 2024-04-16T19:46:32.843 DEBUG:boto:Token: None 2024-04-16T19:46:32.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4496f1a71181766-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.846 DEBUG:boto:b'yrjefh-39obj31000falseobj32024-04-16T19:45:51.596Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:32.847 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.847 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.847 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.847 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.847 DEBUG:boto:Method: GET 2024-04-16T19:46:32.847 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.847 DEBUG:boto:Data: 2024-04-16T19:46:32.847 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.847 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.847 DEBUG:boto:Params: {} 2024-04-16T19:46:32.848 DEBUG:boto:Token: None 2024-04-16T19:46:32.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4= 2024-04-16T19:46:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4=', 'Content-Length': '0'} 2024-04-16T19:46:32.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075789a552bd81df4-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.851 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.851 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.851 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.851 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.851 DEBUG:boto:Method: GET 2024-04-16T19:46:32.851 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.851 DEBUG:boto:Data: 2024-04-16T19:46:32.851 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.851 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.851 DEBUG:boto:Params: {} 2024-04-16T19:46:32.852 DEBUG:boto:Token: None 2024-04-16T19:46:32.852 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4= 2024-04-16T19:46:32.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4=', 'Content-Length': '0'} 2024-04-16T19:46:32.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e2e72f3fe407723-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.894 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.894 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.894 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.895 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.895 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.895 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.895 DEBUG:boto:Data: 2024-04-16T19:46:32.895 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.895 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.895 DEBUG:boto:Params: {} 2024-04-16T19:46:32.895 DEBUG:boto:Token: None 2024-04-16T19:46:32.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jTbrWdZ2V/Uuxxww5V7VVtS7ilg= 2024-04-16T19:46:32.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jTbrWdZ2V/Uuxxww5V7VVtS7ilg=', 'Content-Length': '0'} 2024-04-16T19:46:32.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000936bf12c599fdf1f-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.898 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.898 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.898 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.898 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.898 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.898 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.899 DEBUG:boto:Data: 2024-04-16T19:46:32.899 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.899 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.899 DEBUG:boto:Params: {} 2024-04-16T19:46:32.899 DEBUG:boto:Token: None 2024-04-16T19:46:32.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jTbrWdZ2V/Uuxxww5V7VVtS7ilg= 2024-04-16T19:46:32.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jTbrWdZ2V/Uuxxww5V7VVtS7ilg=', 'Content-Length': '0'} 2024-04-16T19:46:32.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091c76a40af061937-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:32.901 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.902 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.902 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.902 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.902 DEBUG:boto:Method: GET 2024-04-16T19:46:32.902 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.902 DEBUG:boto:Data: 2024-04-16T19:46:32.902 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.902 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.902 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.902 DEBUG:boto:Params: {} 2024-04-16T19:46:32.903 DEBUG:boto:Token: None 2024-04-16T19:46:32.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4= 2024-04-16T19:46:32.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4=', 'Content-Length': '0'} 2024-04-16T19:46:32.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000343893ae0bed7328-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.905 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.905 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.905 DEBUG:boto:path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.906 DEBUG:boto:auth_path=/yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.906 DEBUG:boto:Method: GET 2024-04-16T19:46:32.906 DEBUG:boto:Path: /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.906 DEBUG:boto:Data: 2024-04-16T19:46:32.906 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.906 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.906 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.906 DEBUG:boto:Params: {} 2024-04-16T19:46:32.906 DEBUG:boto:Token: None 2024-04-16T19:46:32.906 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4?versionId=null 2024-04-16T19:46:32.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4= 2024-04-16T19:46:32.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKIy4vke0djejlZq/linAjdsPJ4=', 'Content-Length': '0'} 2024-04-16T19:46:32.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4c34dafb1395213-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.950 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.950 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.951 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.951 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.951 DEBUG:boto:Method: GET 2024-04-16T19:46:32.951 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.951 DEBUG:boto:Data: 2024-04-16T19:46:32.951 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.951 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.951 DEBUG:boto:Params: {} 2024-04-16T19:46:32.951 DEBUG:boto:Token: None 2024-04-16T19:46:32.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd0b7004bfc84a7-00661ed598-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.954 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.955 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:32.955 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:32.955 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.955 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.955 DEBUG:boto:Method: GET 2024-04-16T19:46:32.955 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:46:32.955 DEBUG:boto:Data: 2024-04-16T19:46:32.955 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.955 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.955 DEBUG:boto:Params: {} 2024-04-16T19:46:32.955 DEBUG:boto:Token: None 2024-04-16T19:46:32.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/ 2024-04-16T19:46:32.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo= 2024-04-16T19:46:32.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:npkjXxNj2ctkUS56KSjwXVueRyo=', 'Content-Length': '0'} 2024-04-16T19:46:32.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af33efa438bc7cc-00661ed598-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.958 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:32.959 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.959 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.959 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.959 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:46:32.959 DEBUG:boto:Data: 2024-04-16T19:46:32.959 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.959 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.959 DEBUG:boto:Params: {} 2024-04-16T19:46:32.959 DEBUG:boto:Token: None 2024-04-16T19:46:32.960 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4 2024-04-16T19:46:32.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ysSDFqGGnwQcXjKzvf2XqHrEBS8= 2024-04-16T19:46:32.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ysSDFqGGnwQcXjKzvf2XqHrEBS8=', 'Content-Length': '0'} 2024-04-16T19:46:32.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ade32a790268f187-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.962 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.962 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.962 DEBUG:boto:Method: HEAD 2024-04-16T19:46:32.962 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:46:32.962 DEBUG:boto:Data: 2024-04-16T19:46:32.962 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.963 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.963 DEBUG:boto:Params: {} 2024-04-16T19:46:32.963 DEBUG:boto:Token: None 2024-04-16T19:46:32.963 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4 2024-04-16T19:46:32.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ysSDFqGGnwQcXjKzvf2XqHrEBS8= 2024-04-16T19:46:32.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ysSDFqGGnwQcXjKzvf2XqHrEBS8=', 'Content-Length': '0'} 2024-04-16T19:46:32.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7a9e77588a5d93-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:32.965 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.966 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.966 DEBUG:boto:Method: GET 2024-04-16T19:46:32.966 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:46:32.966 DEBUG:boto:Data: 2024-04-16T19:46:32.966 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.966 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:32.966 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.966 DEBUG:boto:Params: {} 2024-04-16T19:46:32.966 DEBUG:boto:Token: None 2024-04-16T19:46:32.966 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4 2024-04-16T19:46:32.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1iYlZTm56v4ljZwOdgx6f65ixM0= 2024-04-16T19:46:32.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1iYlZTm56v4ljZwOdgx6f65ixM0=', 'Content-Length': '0'} 2024-04-16T19:46:32.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c809e431e2a4ea21-00661ed598-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:32.969 DEBUG:boto:path=/yrjefh-39/obj4 2024-04-16T19:46:32.969 DEBUG:boto:auth_path=/yrjefh-39/obj4 2024-04-16T19:46:32.969 DEBUG:boto:Method: GET 2024-04-16T19:46:32.969 DEBUG:boto:Path: /yrjefh-39/obj4 2024-04-16T19:46:32.969 DEBUG:boto:Data: 2024-04-16T19:46:32.969 DEBUG:boto:Headers: {} 2024-04-16T19:46:32.969 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:32.969 DEBUG:boto:Port: 8000 2024-04-16T19:46:32.970 DEBUG:boto:Params: {} 2024-04-16T19:46:32.970 DEBUG:boto:Token: None 2024-04-16T19:46:32.970 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:32 GMT /yrjefh-39/obj4 2024-04-16T19:46:32.970 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1iYlZTm56v4ljZwOdgx6f65ixM0= 2024-04-16T19:46:32.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1iYlZTm56v4ljZwOdgx6f65ixM0=', 'Content-Length': '0'} 2024-04-16T19:46:32.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:45:52 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9ecc25465f8b9b-00661ed598-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.014 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.014 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.014 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.015 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.015 DEBUG:boto:Method: GET 2024-04-16T19:46:33.015 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.015 DEBUG:boto:Data: 2024-04-16T19:46:33.015 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.015 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.015 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.015 DEBUG:boto:Params: {} 2024-04-16T19:46:33.015 DEBUG:boto:Token: None 2024-04-16T19:46:33.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6782f58df1d706-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.018 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.018 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.018 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.018 DEBUG:boto:path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.018 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.018 DEBUG:boto:Method: GET 2024-04-16T19:46:33.018 DEBUG:boto:Path: /yrjefh-39/?prefix=obj4 2024-04-16T19:46:33.019 DEBUG:boto:Data: 2024-04-16T19:46:33.019 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.019 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.019 DEBUG:boto:Params: {} 2024-04-16T19:46:33.019 DEBUG:boto:Token: None 2024-04-16T19:46:33.019 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.019 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b74f8380ad62f10-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.022 DEBUG:boto:b'yrjefh-39obj41000falseobj42024-04-16T19:45:51.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:33.022 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.022 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.023 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.023 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.023 DEBUG:boto:Method: GET 2024-04-16T19:46:33.023 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.023 DEBUG:boto:Data: 2024-04-16T19:46:33.023 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.023 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.023 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.023 DEBUG:boto:Params: {} 2024-04-16T19:46:33.023 DEBUG:boto:Token: None 2024-04-16T19:46:33.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0= 2024-04-16T19:46:33.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0=', 'Content-Length': '0'} 2024-04-16T19:46:33.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f8b6d567742471a-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.026 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.026 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.026 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.026 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.026 DEBUG:boto:Method: GET 2024-04-16T19:46:33.026 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.026 DEBUG:boto:Data: 2024-04-16T19:46:33.026 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.026 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.027 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.027 DEBUG:boto:Params: {} 2024-04-16T19:46:33.027 DEBUG:boto:Token: None 2024-04-16T19:46:33.027 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0= 2024-04-16T19:46:33.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0=', 'Content-Length': '0'} 2024-04-16T19:46:33.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000228550cc32e3ef9-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.070 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.070 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.071 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.071 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.071 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.071 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.071 DEBUG:boto:Data: 2024-04-16T19:46:33.071 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.071 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.071 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.071 DEBUG:boto:Params: {} 2024-04-16T19:46:33.071 DEBUG:boto:Token: None 2024-04-16T19:46:33.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jmZB1zNdK8yi2k8gKYBn4savnX8= 2024-04-16T19:46:33.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jmZB1zNdK8yi2k8gKYBn4savnX8=', 'Content-Length': '0'} 2024-04-16T19:46:33.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dcd07982827ac36-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.073 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.074 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.074 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.074 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.074 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.074 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.074 DEBUG:boto:Data: 2024-04-16T19:46:33.074 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.074 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.074 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.074 DEBUG:boto:Params: {} 2024-04-16T19:46:33.074 DEBUG:boto:Token: None 2024-04-16T19:46:33.075 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jmZB1zNdK8yi2k8gKYBn4savnX8= 2024-04-16T19:46:33.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jmZB1zNdK8yi2k8gKYBn4savnX8=', 'Content-Length': '0'} 2024-04-16T19:46:33.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ca6e779b0b4c37e-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:33.077 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.077 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.077 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.077 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.077 DEBUG:boto:Method: GET 2024-04-16T19:46:33.077 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.077 DEBUG:boto:Data: 2024-04-16T19:46:33.077 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.077 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.077 DEBUG:boto:Params: {} 2024-04-16T19:46:33.078 DEBUG:boto:Token: None 2024-04-16T19:46:33.078 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.078 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0= 2024-04-16T19:46:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0=', 'Content-Length': '0'} 2024-04-16T19:46:33.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8a6ff3f94e6b592-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.080 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.080 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.080 DEBUG:boto:path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.080 DEBUG:boto:auth_path=/yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.080 DEBUG:boto:Method: GET 2024-04-16T19:46:33.080 DEBUG:boto:Path: /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.081 DEBUG:boto:Data: 2024-04-16T19:46:33.081 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.081 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.081 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.081 DEBUG:boto:Params: {} 2024-04-16T19:46:33.081 DEBUG:boto:Token: None 2024-04-16T19:46:33.081 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5?versionId=null 2024-04-16T19:46:33.081 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0= 2024-04-16T19:46:33.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvGzsIjyh842+e9fYAvJ4KcSrw0=', 'Content-Length': '0'} 2024-04-16T19:46:33.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052878a1ab6db180e-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.130 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.131 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.131 DEBUG:boto:path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.131 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.131 DEBUG:boto:Method: GET 2024-04-16T19:46:33.131 DEBUG:boto:Path: /yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.131 DEBUG:boto:Data: 2024-04-16T19:46:33.131 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.131 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.131 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.131 DEBUG:boto:Params: {} 2024-04-16T19:46:33.131 DEBUG:boto:Token: None 2024-04-16T19:46:33.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deeb718ceb2d6ee7-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.135 DEBUG:boto:b'yrjefh-39obj51000falseobj52024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.135 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.135 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.135 DEBUG:boto:path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.135 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.135 DEBUG:boto:Method: GET 2024-04-16T19:46:33.135 DEBUG:boto:Path: /yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.135 DEBUG:boto:Data: 2024-04-16T19:46:33.136 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.136 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.136 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.136 DEBUG:boto:Params: {} 2024-04-16T19:46:33.136 DEBUG:boto:Token: None 2024-04-16T19:46:33.136 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d964291b980063ec-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.182 DEBUG:boto:b'yrjefh-39obj51000falseobj52024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.183 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.183 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.183 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.183 DEBUG:boto:Path: /yrjefh-39/obj5 2024-04-16T19:46:33.183 DEBUG:boto:Data: 2024-04-16T19:46:33.183 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.183 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.183 DEBUG:boto:Params: {} 2024-04-16T19:46:33.184 DEBUG:boto:Token: None 2024-04-16T19:46:33.184 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5 2024-04-16T19:46:33.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ikvTYRYV69KObxV+fjN86ey3ocY= 2024-04-16T19:46:33.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ikvTYRYV69KObxV+fjN86ey3ocY=', 'Content-Length': '0'} 2024-04-16T19:46:33.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004595e483514464d6-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.186 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.186 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.186 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.186 DEBUG:boto:Path: /yrjefh-39/obj5 2024-04-16T19:46:33.186 DEBUG:boto:Data: 2024-04-16T19:46:33.186 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.187 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.187 DEBUG:boto:Params: {} 2024-04-16T19:46:33.187 DEBUG:boto:Token: None 2024-04-16T19:46:33.187 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5 2024-04-16T19:46:33.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ikvTYRYV69KObxV+fjN86ey3ocY= 2024-04-16T19:46:33.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ikvTYRYV69KObxV+fjN86ey3ocY=', 'Content-Length': '0'} 2024-04-16T19:46:33.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cc8d37256427ed6-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:46:33.189 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.189 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.190 DEBUG:boto:Method: GET 2024-04-16T19:46:33.190 DEBUG:boto:Path: /yrjefh-39/obj5 2024-04-16T19:46:33.190 DEBUG:boto:Data: 2024-04-16T19:46:33.190 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.190 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.190 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.190 DEBUG:boto:Params: {} 2024-04-16T19:46:33.190 DEBUG:boto:Token: None 2024-04-16T19:46:33.190 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5 2024-04-16T19:46:33.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T+xaL1sN7BB/i5UuWSxmoak2KPk= 2024-04-16T19:46:33.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T+xaL1sN7BB/i5UuWSxmoak2KPk=', 'Content-Length': '0'} 2024-04-16T19:46:33.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb1b18e3f394ab04-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.193 DEBUG:boto:path=/yrjefh-39/obj5 2024-04-16T19:46:33.193 DEBUG:boto:auth_path=/yrjefh-39/obj5 2024-04-16T19:46:33.193 DEBUG:boto:Method: GET 2024-04-16T19:46:33.193 DEBUG:boto:Path: /yrjefh-39/obj5 2024-04-16T19:46:33.193 DEBUG:boto:Data: 2024-04-16T19:46:33.193 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.193 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.193 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.193 DEBUG:boto:Params: {} 2024-04-16T19:46:33.193 DEBUG:boto:Token: None 2024-04-16T19:46:33.193 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj5 2024-04-16T19:46:33.194 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T+xaL1sN7BB/i5UuWSxmoak2KPk= 2024-04-16T19:46:33.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T+xaL1sN7BB/i5UuWSxmoak2KPk=', 'Content-Length': '0'} 2024-04-16T19:46:33.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026d6252dceedb2c8-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.238 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.238 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.239 DEBUG:boto:path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.239 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.239 DEBUG:boto:Method: GET 2024-04-16T19:46:33.239 DEBUG:boto:Path: /yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.239 DEBUG:boto:Data: 2024-04-16T19:46:33.239 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.239 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.239 DEBUG:boto:Params: {} 2024-04-16T19:46:33.239 DEBUG:boto:Token: None 2024-04-16T19:46:33.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c13dcbb47e5ec2-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.242 DEBUG:boto:b'yrjefh-39obj51000falseobj52024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.242 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.243 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.243 DEBUG:boto:path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.243 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.243 DEBUG:boto:Method: GET 2024-04-16T19:46:33.243 DEBUG:boto:Path: /yrjefh-39/?prefix=obj5 2024-04-16T19:46:33.243 DEBUG:boto:Data: 2024-04-16T19:46:33.243 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.243 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.243 DEBUG:boto:Params: {} 2024-04-16T19:46:33.243 DEBUG:boto:Token: None 2024-04-16T19:46:33.243 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca550fac03aed1f-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.246 DEBUG:boto:b'yrjefh-39obj51000falseobj52024-04-16T19:45:59.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:33.246 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.247 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.247 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.247 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.247 DEBUG:boto:Method: GET 2024-04-16T19:46:33.247 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.247 DEBUG:boto:Data: 2024-04-16T19:46:33.247 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.247 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.247 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.247 DEBUG:boto:Params: {} 2024-04-16T19:46:33.247 DEBUG:boto:Token: None 2024-04-16T19:46:33.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM= 2024-04-16T19:46:33.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM=', 'Content-Length': '0'} 2024-04-16T19:46:33.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afdc0e2870dd92c8-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.250 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.250 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.250 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.250 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.250 DEBUG:boto:Method: GET 2024-04-16T19:46:33.250 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.250 DEBUG:boto:Data: 2024-04-16T19:46:33.250 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.250 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.251 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.251 DEBUG:boto:Params: {} 2024-04-16T19:46:33.251 DEBUG:boto:Token: None 2024-04-16T19:46:33.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM= 2024-04-16T19:46:33.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM=', 'Content-Length': '0'} 2024-04-16T19:46:33.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cad3fd311bf23b5-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.294 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.294 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.294 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.295 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.295 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.295 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.295 DEBUG:boto:Data: 2024-04-16T19:46:33.295 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.295 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.295 DEBUG:boto:Params: {} 2024-04-16T19:46:33.295 DEBUG:boto:Token: None 2024-04-16T19:46:33.295 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4U0GU03reQ4pCvN4f/ZjentYruY= 2024-04-16T19:46:33.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4U0GU03reQ4pCvN4f/ZjentYruY=', 'Content-Length': '0'} 2024-04-16T19:46:33.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009119efcc37371558-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.298 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.298 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.298 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.298 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.298 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.298 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.298 DEBUG:boto:Data: 2024-04-16T19:46:33.298 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.299 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.299 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.299 DEBUG:boto:Params: {} 2024-04-16T19:46:33.299 DEBUG:boto:Token: None 2024-04-16T19:46:33.299 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4U0GU03reQ4pCvN4f/ZjentYruY= 2024-04-16T19:46:33.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4U0GU03reQ4pCvN4f/ZjentYruY=', 'Content-Length': '0'} 2024-04-16T19:46:33.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e706970b27360949-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:33.301 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.302 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.302 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.302 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.302 DEBUG:boto:Method: GET 2024-04-16T19:46:33.302 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.302 DEBUG:boto:Data: 2024-04-16T19:46:33.302 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.302 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.302 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.302 DEBUG:boto:Params: {} 2024-04-16T19:46:33.302 DEBUG:boto:Token: None 2024-04-16T19:46:33.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM= 2024-04-16T19:46:33.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM=', 'Content-Length': '0'} 2024-04-16T19:46:33.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9cc595846cce8da-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.305 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.305 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.305 DEBUG:boto:path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.305 DEBUG:boto:auth_path=/yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.305 DEBUG:boto:Method: GET 2024-04-16T19:46:33.305 DEBUG:boto:Path: /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.305 DEBUG:boto:Data: 2024-04-16T19:46:33.306 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.306 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.306 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.306 DEBUG:boto:Params: {} 2024-04-16T19:46:33.306 DEBUG:boto:Token: None 2024-04-16T19:46:33.306 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6?versionId=null 2024-04-16T19:46:33.306 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM= 2024-04-16T19:46:33.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z6552D5R0V4kwAKMdN1tF48jwiM=', 'Content-Length': '0'} 2024-04-16T19:46:33.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fd3f7200abc27ba-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.350 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.350 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.351 DEBUG:boto:path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.351 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.351 DEBUG:boto:Method: GET 2024-04-16T19:46:33.351 DEBUG:boto:Path: /yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.351 DEBUG:boto:Data: 2024-04-16T19:46:33.351 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.351 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.351 DEBUG:boto:Params: {} 2024-04-16T19:46:33.351 DEBUG:boto:Token: None 2024-04-16T19:46:33.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000845b19ce608c32d9-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.354 DEBUG:boto:b'yrjefh-39obj61000falseobj62024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.355 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.355 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.355 DEBUG:boto:path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.355 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.355 DEBUG:boto:Method: GET 2024-04-16T19:46:33.355 DEBUG:boto:Path: /yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.355 DEBUG:boto:Data: 2024-04-16T19:46:33.355 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.355 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.355 DEBUG:boto:Params: {} 2024-04-16T19:46:33.356 DEBUG:boto:Token: None 2024-04-16T19:46:33.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1b9732d45b27f9-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.358 DEBUG:boto:b'yrjefh-39obj61000falseobj62024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.358 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.359 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.359 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.359 DEBUG:boto:Path: /yrjefh-39/obj6 2024-04-16T19:46:33.359 DEBUG:boto:Data: 2024-04-16T19:46:33.359 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.359 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.359 DEBUG:boto:Params: {} 2024-04-16T19:46:33.359 DEBUG:boto:Token: None 2024-04-16T19:46:33.359 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6 2024-04-16T19:46:33.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w0RSFhghiiBW80Kw0xrx9nMwNKM= 2024-04-16T19:46:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w0RSFhghiiBW80Kw0xrx9nMwNKM=', 'Content-Length': '0'} 2024-04-16T19:46:33.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b900779b3759f067-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.362 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.362 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.362 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.362 DEBUG:boto:Path: /yrjefh-39/obj6 2024-04-16T19:46:33.362 DEBUG:boto:Data: 2024-04-16T19:46:33.362 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.363 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.363 DEBUG:boto:Params: {} 2024-04-16T19:46:33.363 DEBUG:boto:Token: None 2024-04-16T19:46:33.363 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6 2024-04-16T19:46:33.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w0RSFhghiiBW80Kw0xrx9nMwNKM= 2024-04-16T19:46:33.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w0RSFhghiiBW80Kw0xrx9nMwNKM=', 'Content-Length': '0'} 2024-04-16T19:46:33.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000448a733728fbbfcc-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:46:33.366 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.366 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.366 DEBUG:boto:Method: GET 2024-04-16T19:46:33.366 DEBUG:boto:Path: /yrjefh-39/obj6 2024-04-16T19:46:33.366 DEBUG:boto:Data: 2024-04-16T19:46:33.366 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.366 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.366 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.366 DEBUG:boto:Params: {} 2024-04-16T19:46:33.366 DEBUG:boto:Token: None 2024-04-16T19:46:33.366 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6 2024-04-16T19:46:33.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xuQXixTBYwLwEMFbq4rXAmHRqXs= 2024-04-16T19:46:33.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xuQXixTBYwLwEMFbq4rXAmHRqXs=', 'Content-Length': '0'} 2024-04-16T19:46:33.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a21677a259bf55d-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.410 DEBUG:boto:path=/yrjefh-39/obj6 2024-04-16T19:46:33.410 DEBUG:boto:auth_path=/yrjefh-39/obj6 2024-04-16T19:46:33.411 DEBUG:boto:Method: GET 2024-04-16T19:46:33.411 DEBUG:boto:Path: /yrjefh-39/obj6 2024-04-16T19:46:33.411 DEBUG:boto:Data: 2024-04-16T19:46:33.411 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.411 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.411 DEBUG:boto:Params: {} 2024-04-16T19:46:33.411 DEBUG:boto:Token: None 2024-04-16T19:46:33.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj6 2024-04-16T19:46:33.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xuQXixTBYwLwEMFbq4rXAmHRqXs= 2024-04-16T19:46:33.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xuQXixTBYwLwEMFbq4rXAmHRqXs=', 'Content-Length': '0'} 2024-04-16T19:46:33.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb4612b45dd78924-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.454 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.455 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.455 DEBUG:boto:path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.455 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.455 DEBUG:boto:Method: GET 2024-04-16T19:46:33.455 DEBUG:boto:Path: /yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.455 DEBUG:boto:Data: 2024-04-16T19:46:33.455 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.455 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.455 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.455 DEBUG:boto:Params: {} 2024-04-16T19:46:33.455 DEBUG:boto:Token: None 2024-04-16T19:46:33.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c084982d837744-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.458 DEBUG:boto:b'yrjefh-39obj61000falseobj62024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.458 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.459 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.459 DEBUG:boto:path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.459 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.459 DEBUG:boto:Method: GET 2024-04-16T19:46:33.459 DEBUG:boto:Path: /yrjefh-39/?prefix=obj6 2024-04-16T19:46:33.459 DEBUG:boto:Data: 2024-04-16T19:46:33.459 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.459 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.459 DEBUG:boto:Params: {} 2024-04-16T19:46:33.459 DEBUG:boto:Token: None 2024-04-16T19:46:33.460 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ecca1851dd06f85-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.502 DEBUG:boto:b'yrjefh-39obj61000falseobj62024-04-16T19:45:59.868Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:33.503 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.503 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.503 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.503 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.503 DEBUG:boto:Method: GET 2024-04-16T19:46:33.503 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.503 DEBUG:boto:Data: 2024-04-16T19:46:33.503 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.504 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.504 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.504 DEBUG:boto:Params: {} 2024-04-16T19:46:33.504 DEBUG:boto:Token: None 2024-04-16T19:46:33.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4= 2024-04-16T19:46:33.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4=', 'Content-Length': '0'} 2024-04-16T19:46:33.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae5ac03056971f8d-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.506 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.506 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.507 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.507 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.507 DEBUG:boto:Method: GET 2024-04-16T19:46:33.507 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.507 DEBUG:boto:Data: 2024-04-16T19:46:33.507 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.507 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.507 DEBUG:boto:Params: {} 2024-04-16T19:46:33.507 DEBUG:boto:Token: None 2024-04-16T19:46:33.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4= 2024-04-16T19:46:33.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4=', 'Content-Length': '0'} 2024-04-16T19:46:33.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006406cf7b66259965-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.550 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.550 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.551 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.551 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.551 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.551 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.551 DEBUG:boto:Data: 2024-04-16T19:46:33.551 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.551 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.551 DEBUG:boto:Params: {} 2024-04-16T19:46:33.551 DEBUG:boto:Token: None 2024-04-16T19:46:33.552 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hHWdvmMgvxqoURHMD+Iu6mluWjc= 2024-04-16T19:46:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hHWdvmMgvxqoURHMD+Iu6mluWjc=', 'Content-Length': '0'} 2024-04-16T19:46:33.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed2a760dc8b2cdce-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.554 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.554 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.555 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.555 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.555 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.555 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.555 DEBUG:boto:Data: 2024-04-16T19:46:33.555 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.555 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.555 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.555 DEBUG:boto:Params: {} 2024-04-16T19:46:33.555 DEBUG:boto:Token: None 2024-04-16T19:46:33.556 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hHWdvmMgvxqoURHMD+Iu6mluWjc= 2024-04-16T19:46:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hHWdvmMgvxqoURHMD+Iu6mluWjc=', 'Content-Length': '0'} 2024-04-16T19:46:33.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076cc2c68d1c05373-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:33.558 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.558 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.559 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.559 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.559 DEBUG:boto:Method: GET 2024-04-16T19:46:33.559 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.559 DEBUG:boto:Data: 2024-04-16T19:46:33.559 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.559 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.559 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.559 DEBUG:boto:Params: {} 2024-04-16T19:46:33.559 DEBUG:boto:Token: None 2024-04-16T19:46:33.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4= 2024-04-16T19:46:33.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4=', 'Content-Length': '0'} 2024-04-16T19:46:33.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b38e0221122645e7-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.562 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.562 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.562 DEBUG:boto:path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.562 DEBUG:boto:auth_path=/yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.563 DEBUG:boto:Method: GET 2024-04-16T19:46:33.563 DEBUG:boto:Path: /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.563 DEBUG:boto:Data: 2024-04-16T19:46:33.563 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.563 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.563 DEBUG:boto:Params: {} 2024-04-16T19:46:33.563 DEBUG:boto:Token: None 2024-04-16T19:46:33.563 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7?versionId=null 2024-04-16T19:46:33.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4= 2024-04-16T19:46:33.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ/L37uPaTtB0iOyZgWViUGruH4=', 'Content-Length': '0'} 2024-04-16T19:46:33.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027dc1c56a82120af-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.606 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.607 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.607 DEBUG:boto:path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.607 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.607 DEBUG:boto:Method: GET 2024-04-16T19:46:33.607 DEBUG:boto:Path: /yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.607 DEBUG:boto:Data: 2024-04-16T19:46:33.607 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.607 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.607 DEBUG:boto:Params: {} 2024-04-16T19:46:33.607 DEBUG:boto:Token: None 2024-04-16T19:46:33.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6176f6e80cf7f97-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.611 DEBUG:boto:b'yrjefh-39obj71000falseobj72024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.611 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.611 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.611 DEBUG:boto:path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.611 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.611 DEBUG:boto:Method: GET 2024-04-16T19:46:33.612 DEBUG:boto:Path: /yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.612 DEBUG:boto:Data: 2024-04-16T19:46:33.612 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.612 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.612 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.612 DEBUG:boto:Params: {} 2024-04-16T19:46:33.612 DEBUG:boto:Token: None 2024-04-16T19:46:33.612 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d98544c21329ee-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.658 DEBUG:boto:b'yrjefh-39obj71000falseobj72024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.659 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.659 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.659 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.659 DEBUG:boto:Path: /yrjefh-39/obj7 2024-04-16T19:46:33.659 DEBUG:boto:Data: 2024-04-16T19:46:33.659 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.659 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.660 DEBUG:boto:Params: {} 2024-04-16T19:46:33.660 DEBUG:boto:Token: None 2024-04-16T19:46:33.660 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7 2024-04-16T19:46:33.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NXp3Wt9qmmtJFBtYrExKvkmzS6A= 2024-04-16T19:46:33.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NXp3Wt9qmmtJFBtYrExKvkmzS6A=', 'Content-Length': '0'} 2024-04-16T19:46:33.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb19c1e823af8859-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.662 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.663 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.663 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.663 DEBUG:boto:Path: /yrjefh-39/obj7 2024-04-16T19:46:33.663 DEBUG:boto:Data: 2024-04-16T19:46:33.663 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.663 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.663 DEBUG:boto:Params: {} 2024-04-16T19:46:33.663 DEBUG:boto:Token: None 2024-04-16T19:46:33.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7 2024-04-16T19:46:33.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NXp3Wt9qmmtJFBtYrExKvkmzS6A= 2024-04-16T19:46:33.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NXp3Wt9qmmtJFBtYrExKvkmzS6A=', 'Content-Length': '0'} 2024-04-16T19:46:33.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e57ea67125cbad33-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:46:33.666 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.666 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.667 DEBUG:boto:Method: GET 2024-04-16T19:46:33.667 DEBUG:boto:Path: /yrjefh-39/obj7 2024-04-16T19:46:33.667 DEBUG:boto:Data: 2024-04-16T19:46:33.667 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.667 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.667 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.667 DEBUG:boto:Params: {} 2024-04-16T19:46:33.667 DEBUG:boto:Token: None 2024-04-16T19:46:33.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7 2024-04-16T19:46:33.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qpG9bQG9y4PGBMJc4X6b7NPAcdI= 2024-04-16T19:46:33.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qpG9bQG9y4PGBMJc4X6b7NPAcdI=', 'Content-Length': '0'} 2024-04-16T19:46:33.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a9752e21f2a4716-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.670 DEBUG:boto:path=/yrjefh-39/obj7 2024-04-16T19:46:33.670 DEBUG:boto:auth_path=/yrjefh-39/obj7 2024-04-16T19:46:33.670 DEBUG:boto:Method: GET 2024-04-16T19:46:33.670 DEBUG:boto:Path: /yrjefh-39/obj7 2024-04-16T19:46:33.670 DEBUG:boto:Data: 2024-04-16T19:46:33.670 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.671 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.671 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.671 DEBUG:boto:Params: {} 2024-04-16T19:46:33.671 DEBUG:boto:Token: None 2024-04-16T19:46:33.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj7 2024-04-16T19:46:33.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qpG9bQG9y4PGBMJc4X6b7NPAcdI= 2024-04-16T19:46:33.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qpG9bQG9y4PGBMJc4X6b7NPAcdI=', 'Content-Length': '0'} 2024-04-16T19:46:33.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:24 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034265355afb9d1a6-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.714 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.714 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.715 DEBUG:boto:path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.715 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.715 DEBUG:boto:Method: GET 2024-04-16T19:46:33.715 DEBUG:boto:Path: /yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.715 DEBUG:boto:Data: 2024-04-16T19:46:33.715 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.715 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.715 DEBUG:boto:Params: {} 2024-04-16T19:46:33.716 DEBUG:boto:Token: None 2024-04-16T19:46:33.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f32131a04401456b-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.719 DEBUG:boto:b'yrjefh-39obj71000falseobj72024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.719 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.720 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.720 DEBUG:boto:path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.720 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.720 DEBUG:boto:Method: GET 2024-04-16T19:46:33.720 DEBUG:boto:Path: /yrjefh-39/?prefix=obj7 2024-04-16T19:46:33.720 DEBUG:boto:Data: 2024-04-16T19:46:33.720 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.720 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.721 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.721 DEBUG:boto:Params: {} 2024-04-16T19:46:33.721 DEBUG:boto:Token: None 2024-04-16T19:46:33.721 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.721 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c21e8b223e09f56e-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.724 DEBUG:boto:b'yrjefh-39obj71000falseobj72024-04-16T19:45:59.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:33.724 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.724 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.724 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.725 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.725 DEBUG:boto:Method: GET 2024-04-16T19:46:33.725 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.725 DEBUG:boto:Data: 2024-04-16T19:46:33.725 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.725 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.725 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.725 DEBUG:boto:Params: {} 2024-04-16T19:46:33.725 DEBUG:boto:Token: None 2024-04-16T19:46:33.726 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.726 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c= 2024-04-16T19:46:33.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c=', 'Content-Length': '0'} 2024-04-16T19:46:33.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000831f79410227a1d4-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.729 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.729 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.729 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.729 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.729 DEBUG:boto:Method: GET 2024-04-16T19:46:33.729 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.729 DEBUG:boto:Data: 2024-04-16T19:46:33.729 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.730 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.730 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.730 DEBUG:boto:Params: {} 2024-04-16T19:46:33.730 DEBUG:boto:Token: None 2024-04-16T19:46:33.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c= 2024-04-16T19:46:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c=', 'Content-Length': '0'} 2024-04-16T19:46:33.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004937d073b9cead95-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.774 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.775 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.775 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.775 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.775 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.775 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.775 DEBUG:boto:Data: 2024-04-16T19:46:33.775 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.775 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.776 DEBUG:boto:Params: {} 2024-04-16T19:46:33.776 DEBUG:boto:Token: None 2024-04-16T19:46:33.776 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QTR7mjV92AXkU1WMRejKSzuvryk= 2024-04-16T19:46:33.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QTR7mjV92AXkU1WMRejKSzuvryk=', 'Content-Length': '0'} 2024-04-16T19:46:33.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060e3f071a5ef40ef-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.779 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.779 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.779 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.779 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.779 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.779 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.780 DEBUG:boto:Data: 2024-04-16T19:46:33.780 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.780 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.780 DEBUG:boto:Params: {} 2024-04-16T19:46:33.780 DEBUG:boto:Token: None 2024-04-16T19:46:33.780 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QTR7mjV92AXkU1WMRejKSzuvryk= 2024-04-16T19:46:33.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QTR7mjV92AXkU1WMRejKSzuvryk=', 'Content-Length': '0'} 2024-04-16T19:46:33.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d50bbbd8968ff77-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:33.784 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.784 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.784 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.784 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.784 DEBUG:boto:Method: GET 2024-04-16T19:46:33.784 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.785 DEBUG:boto:Data: 2024-04-16T19:46:33.785 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.785 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.785 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.785 DEBUG:boto:Params: {} 2024-04-16T19:46:33.785 DEBUG:boto:Token: None 2024-04-16T19:46:33.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.786 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c= 2024-04-16T19:46:33.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c=', 'Content-Length': '0'} 2024-04-16T19:46:33.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000941221f914a2a066-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.788 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.788 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.788 DEBUG:boto:path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.789 DEBUG:boto:auth_path=/yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.789 DEBUG:boto:Method: GET 2024-04-16T19:46:33.789 DEBUG:boto:Path: /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.789 DEBUG:boto:Data: 2024-04-16T19:46:33.789 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.789 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.789 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.789 DEBUG:boto:Params: {} 2024-04-16T19:46:33.789 DEBUG:boto:Token: None 2024-04-16T19:46:33.790 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8?versionId=null 2024-04-16T19:46:33.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c= 2024-04-16T19:46:33.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3iG1Bm2iJLqPltLrMFQZSJxth4c=', 'Content-Length': '0'} 2024-04-16T19:46:33.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca9b8d712a5b15f9-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.838 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.839 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.839 DEBUG:boto:path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.839 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.839 DEBUG:boto:Method: GET 2024-04-16T19:46:33.839 DEBUG:boto:Path: /yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.839 DEBUG:boto:Data: 2024-04-16T19:46:33.839 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.840 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.840 DEBUG:boto:Params: {} 2024-04-16T19:46:33.840 DEBUG:boto:Token: None 2024-04-16T19:46:33.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006337763f0f8c1939-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.843 DEBUG:boto:b'yrjefh-39obj81000falseobj82024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.844 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.844 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.844 DEBUG:boto:path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.844 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.844 DEBUG:boto:Method: GET 2024-04-16T19:46:33.844 DEBUG:boto:Path: /yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.845 DEBUG:boto:Data: 2024-04-16T19:46:33.845 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.845 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.845 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.845 DEBUG:boto:Params: {} 2024-04-16T19:46:33.845 DEBUG:boto:Token: None 2024-04-16T19:46:33.845 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf0880cbf2919c3-00661ed599-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.890 DEBUG:boto:b'yrjefh-39obj81000falseobj82024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.891 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.891 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.891 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.891 DEBUG:boto:Path: /yrjefh-39/obj8 2024-04-16T19:46:33.891 DEBUG:boto:Data: 2024-04-16T19:46:33.891 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.892 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.892 DEBUG:boto:Params: {} 2024-04-16T19:46:33.892 DEBUG:boto:Token: None 2024-04-16T19:46:33.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8 2024-04-16T19:46:33.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhjGSydVpm8DDk3t3l+W2GsCUnM= 2024-04-16T19:46:33.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhjGSydVpm8DDk3t3l+W2GsCUnM=', 'Content-Length': '0'} 2024-04-16T19:46:33.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd2efc17babbdcf-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.895 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.895 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.895 DEBUG:boto:Method: HEAD 2024-04-16T19:46:33.895 DEBUG:boto:Path: /yrjefh-39/obj8 2024-04-16T19:46:33.895 DEBUG:boto:Data: 2024-04-16T19:46:33.896 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.896 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.896 DEBUG:boto:Params: {} 2024-04-16T19:46:33.896 DEBUG:boto:Token: None 2024-04-16T19:46:33.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8 2024-04-16T19:46:33.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhjGSydVpm8DDk3t3l+W2GsCUnM= 2024-04-16T19:46:33.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhjGSydVpm8DDk3t3l+W2GsCUnM=', 'Content-Length': '0'} 2024-04-16T19:46:33.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000466b0dde230beaff-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:46:33.900 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.900 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.900 DEBUG:boto:Method: GET 2024-04-16T19:46:33.900 DEBUG:boto:Path: /yrjefh-39/obj8 2024-04-16T19:46:33.900 DEBUG:boto:Data: 2024-04-16T19:46:33.900 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.901 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.901 DEBUG:boto:Params: {} 2024-04-16T19:46:33.901 DEBUG:boto:Token: None 2024-04-16T19:46:33.901 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8 2024-04-16T19:46:33.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2YRwSmdsjkJ491K2Ixmv/8eQ6E= 2024-04-16T19:46:33.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2YRwSmdsjkJ491K2Ixmv/8eQ6E=', 'Content-Length': '0'} 2024-04-16T19:46:33.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074ace78b347910bf-00661ed599-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.947 DEBUG:boto:path=/yrjefh-39/obj8 2024-04-16T19:46:33.947 DEBUG:boto:auth_path=/yrjefh-39/obj8 2024-04-16T19:46:33.947 DEBUG:boto:Method: GET 2024-04-16T19:46:33.947 DEBUG:boto:Path: /yrjefh-39/obj8 2024-04-16T19:46:33.947 DEBUG:boto:Data: 2024-04-16T19:46:33.947 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:33.947 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.948 DEBUG:boto:Params: {} 2024-04-16T19:46:33.948 DEBUG:boto:Token: None 2024-04-16T19:46:33.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/obj8 2024-04-16T19:46:33.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2YRwSmdsjkJ491K2Ixmv/8eQ6E= 2024-04-16T19:46:33.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2YRwSmdsjkJ491K2Ixmv/8eQ6E=', 'Content-Length': '0'} 2024-04-16T19:46:33.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:45:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-39:b482a2a9-ef36-488b-9666-ada656453328.4812.8'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:23 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c6c3cb85a5cb81b-00661ed599-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.994 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.995 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.995 DEBUG:boto:path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.995 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.995 DEBUG:boto:Method: GET 2024-04-16T19:46:33.995 DEBUG:boto:Path: /yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.995 DEBUG:boto:Data: 2024-04-16T19:46:33.995 DEBUG:boto:Headers: {} 2024-04-16T19:46:33.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:33.995 DEBUG:boto:Port: 8000 2024-04-16T19:46:33.995 DEBUG:boto:Params: {} 2024-04-16T19:46:33.996 DEBUG:boto:Token: None 2024-04-16T19:46:33.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:33 GMT /yrjefh-39/ 2024-04-16T19:46:33.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw= 2024-04-16T19:46:33.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VXp+KmxQBze3n5lK7VFpqgozBRw=', 'Content-Length': '0'} 2024-04-16T19:46:33.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a4b319bd427dd4-00661ed599-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:33.999 DEBUG:boto:b'yrjefh-39obj81000falseobj82024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:33.999 DEBUG:boto:path=/yrjefh-39/ 2024-04-16T19:46:33.999 DEBUG:boto:auth_path=/yrjefh-39/ 2024-04-16T19:46:33.999 DEBUG:boto:path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.999 DEBUG:boto:auth_path=/yrjefh-39/?prefix=obj8 2024-04-16T19:46:33.999 DEBUG:boto:Method: GET 2024-04-16T19:46:34.000 DEBUG:boto:Path: /yrjefh-39/?prefix=obj8 2024-04-16T19:46:34.000 DEBUG:boto:Data: 2024-04-16T19:46:34.000 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.000 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.000 DEBUG:boto:Params: {} 2024-04-16T19:46:34.000 DEBUG:boto:Token: None 2024-04-16T19:46:34.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-39/ 2024-04-16T19:46:34.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sjYlPOODEU9QxjPhXKZJwByxfc8= 2024-04-16T19:46:34.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sjYlPOODEU9QxjPhXKZJwByxfc8=', 'Content-Length': '0'} 2024-04-16T19:46:34.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006180e4432411f09f-00661ed59a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.003 DEBUG:boto:b'yrjefh-39obj81000falseobj82024-04-16T19:45:59.925Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:34.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-39 zones={test-zone2, test-zone1} 2024-04-16T19:46:34.004 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T19:46:34.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.004 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.142 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000006838","last_update":"2024-04-16T19:46:23.938148Z"},{"marker":"00000000000000000000:00000000000000004864","last_update":"2024-04-16T19:46:23.888882Z"},{"marker":"00000000000000000000:00000000000000008661","last_update":"2024-04-16T19:46:24.921362Z"},{"marker":"00000000000000000000:00000000000000007399","last_update":"2024-04-16T19:45:59.746175Z"}] 2024-04-16T19:46:34.142 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000006838', 1: '00000000000000000000:00000000000000004864', 2: '00000000000000000000:00000000000000008661', 3: '00000000000000000000:00000000000000007399'} 2024-04-16T19:46:34.142 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:46:34.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.143 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.199 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.199 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006838","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:23.938148Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004864","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:23.888882Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008661","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:24.921362Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007399","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:59.746175Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:34.276 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006838","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:23.938148Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004864","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:23.888882Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008661","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:24.921362Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007399","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:45:59.746175Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:34.276 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006838', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:23.938148Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004864', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:23.888882Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008661', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:24.921362Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007399', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:45:59.746175Z'}}] 2024-04-16T19:46:34.277 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006838', 1: '00000000000000000000:00000000000000004864', 2: '00000000000000000000:00000000000000008661', 3: '00000000000000000000:00000000000000007399'} 2024-04-16T19:46:34.277 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006838', 1: '00000000000000000000:00000000000000004864', 2: '00000000000000000000:00000000000000008661', 3: '00000000000000000000:00000000000000007399'} 2024-04-16T19:46:34.277 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:46:34.277 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T19:46:34.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.277 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.324 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.324 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.399 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000005154","last_update":"2024-04-16T19:46:19.401056Z"},{"marker":"00000000000000000000:00000000000000004026","last_update":"2024-04-16T19:46:19.402716Z"},{"marker":"00000000000000000000:00000000000000006699","last_update":"2024-04-16T19:46:19.404888Z"},{"marker":"00000000000000000000:00000000000000005016","last_update":"2024-04-16T19:45:32.972030Z"}] 2024-04-16T19:46:34.399 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000005154', 1: '00000000000000000000:00000000000000004026', 2: '00000000000000000000:00000000000000006699', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:46:34.399 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:46:34.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.400 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.532 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005154","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.401056Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004026","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.402716Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006699","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.404888Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005016","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:45:32.972030Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:34.532 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005154","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.401056Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004026","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.402716Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006699","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:19.404888Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005016","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:45:32.972030Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:34.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005154', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:19.401056Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004026', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:19.402716Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006699', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:19.404888Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005016', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:45:32.972030Z'}}] 2024-04-16T19:46:34.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005154', 1: '00000000000000000000:00000000000000004026', 2: '00000000000000000000:00000000000000006699', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:46:34.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005154', 1: '00000000000000000000:00000000000000004026', 2: '00000000000000000000:00000000000000006699', 3: '00000000000000000000:00000000000000005016'} 2024-04-16T19:46:34.533 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:46:34.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-04-16T19:46:34.534 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-40 2024-04-16T19:46:34.534 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:34.534 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:34.534 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.534 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:34.535 DEBUG:boto:Data: 2024-04-16T19:46:34.535 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.535 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.535 DEBUG:boto:Params: {} 2024-04-16T19:46:34.535 DEBUG:boto:Token: None 2024-04-16T19:46:34.535 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/ 2024-04-16T19:46:34.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B2r5XCZxvej5XeUoC5NxdJ16Guc= 2024-04-16T19:46:34.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B2r5XCZxvej5XeUoC5NxdJ16Guc=', 'Content-Length': '0'} 2024-04-16T19:46:34.552 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038da5cb92c9277bc-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.552 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-41 2024-04-16T19:46:34.552 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:34.552 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:34.553 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.553 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:34.553 DEBUG:boto:Data: 2024-04-16T19:46:34.553 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.553 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.553 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.553 DEBUG:boto:Params: {} 2024-04-16T19:46:34.553 DEBUG:boto:Token: None 2024-04-16T19:46:34.553 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/ 2024-04-16T19:46:34.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FBHH8Bnt6V/gcB/TdSIK5mweTqo= 2024-04-16T19:46:34.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FBHH8Bnt6V/gcB/TdSIK5mweTqo=', 'Content-Length': '0'} 2024-04-16T19:46:34.577 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000053fb6441eafc2b0-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.578 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:34.578 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:34.578 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.578 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:34.578 DEBUG:boto:Data: 2024-04-16T19:46:34.578 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.578 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.578 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.578 DEBUG:boto:Params: {} 2024-04-16T19:46:34.578 DEBUG:boto:Token: None 2024-04-16T19:46:34.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/ 2024-04-16T19:46:34.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o= 2024-04-16T19:46:34.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o=', 'Content-Length': '0'} 2024-04-16T19:46:34.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009356ca0e807e365d-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.581 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:34.581 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:34.581 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.581 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:46:34.581 DEBUG:boto:Data: 2024-04-16T19:46:34.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.582 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.582 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.582 DEBUG:boto:Params: {} 2024-04-16T19:46:34.582 DEBUG:boto:Token: None 2024-04-16T19:46:34.582 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/obj1 2024-04-16T19:46:34.582 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HsLocApc0OI0UoyVvRTB4GCOvDI= 2024-04-16T19:46:34.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HsLocApc0OI0UoyVvRTB4GCOvDI='} 2024-04-16T19:46:34.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015851023cfb96e21-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.627 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:34.627 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:34.627 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.627 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:34.627 DEBUG:boto:Data: 2024-04-16T19:46:34.627 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.627 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.627 DEBUG:boto:Params: {} 2024-04-16T19:46:34.627 DEBUG:boto:Token: None 2024-04-16T19:46:34.628 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/ 2024-04-16T19:46:34.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o= 2024-04-16T19:46:34.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o=', 'Content-Length': '0'} 2024-04-16T19:46:34.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bb6532def464c06-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.630 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:34.630 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:34.630 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.630 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:46:34.631 DEBUG:boto:Data: 2024-04-16T19:46:34.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.631 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.631 DEBUG:boto:Params: {} 2024-04-16T19:46:34.631 DEBUG:boto:Token: None 2024-04-16T19:46:34.631 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/obj2 2024-04-16T19:46:34.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:15qcRlmVQ2Wx88D50bSSAWem1RI= 2024-04-16T19:46:34.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:15qcRlmVQ2Wx88D50bSSAWem1RI='} 2024-04-16T19:46:34.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003269701506d5dc13-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.640 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:34.640 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:34.640 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.640 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:34.640 DEBUG:boto:Data: 2024-04-16T19:46:34.640 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.640 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.640 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.640 DEBUG:boto:Params: {} 2024-04-16T19:46:34.640 DEBUG:boto:Token: None 2024-04-16T19:46:34.641 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/ 2024-04-16T19:46:34.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o= 2024-04-16T19:46:34.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o=', 'Content-Length': '0'} 2024-04-16T19:46:34.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4cf337bab3fca70-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.643 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:34.643 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:34.643 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.643 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:46:34.643 DEBUG:boto:Data: 2024-04-16T19:46:34.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.644 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.644 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.644 DEBUG:boto:Params: {} 2024-04-16T19:46:34.644 DEBUG:boto:Token: None 2024-04-16T19:46:34.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/obj3 2024-04-16T19:46:34.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tov3j/QaOKd0pa3PDY+aa9Sahxw= 2024-04-16T19:46:34.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tov3j/QaOKd0pa3PDY+aa9Sahxw='} 2024-04-16T19:46:34.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dae6a3e94c9b5c7-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.686 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:34.687 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:34.687 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.687 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:34.687 DEBUG:boto:Data: 2024-04-16T19:46:34.687 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.687 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.687 DEBUG:boto:Params: {} 2024-04-16T19:46:34.687 DEBUG:boto:Token: None 2024-04-16T19:46:34.688 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/ 2024-04-16T19:46:34.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o= 2024-04-16T19:46:34.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FREht3mr5zK0HZfhWg4kRYVPC/o=', 'Content-Length': '0'} 2024-04-16T19:46:34.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001920998771fd1317-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.690 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:34.690 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:34.690 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.690 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:46:34.690 DEBUG:boto:Data: 2024-04-16T19:46:34.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:34.691 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.691 DEBUG:boto:Params: {} 2024-04-16T19:46:34.691 DEBUG:boto:Token: None 2024-04-16T19:46:34.691 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-40/obj4 2024-04-16T19:46:34.691 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+z4bEtF+LnK3VzwWUlls8zm2L6U= 2024-04-16T19:46:34.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+z4bEtF+LnK3VzwWUlls8zm2L6U='} 2024-04-16T19:46:34.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e5544e1a6e75928-00661ed59a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.696 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:34.697 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:34.697 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.697 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:34.697 DEBUG:boto:Data: 2024-04-16T19:46:34.697 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.697 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.697 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.697 DEBUG:boto:Params: {} 2024-04-16T19:46:34.697 DEBUG:boto:Token: None 2024-04-16T19:46:34.697 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/ 2024-04-16T19:46:34.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA= 2024-04-16T19:46:34.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA=', 'Content-Length': '0'} 2024-04-16T19:46:34.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088b54480c675e669-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.700 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:34.700 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:34.700 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.701 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:46:34.701 DEBUG:boto:Data: 2024-04-16T19:46:34.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.701 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.701 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.701 DEBUG:boto:Params: {} 2024-04-16T19:46:34.701 DEBUG:boto:Token: None 2024-04-16T19:46:34.701 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/obj1 2024-04-16T19:46:34.701 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jRrF02L78IHzDNU3lcCEq3RqQ3M= 2024-04-16T19:46:34.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jRrF02L78IHzDNU3lcCEq3RqQ3M='} 2024-04-16T19:46:34.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029cbf9fd75e3a9f2-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.707 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:34.708 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:34.708 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.708 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:34.708 DEBUG:boto:Data: 2024-04-16T19:46:34.708 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.708 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.708 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.708 DEBUG:boto:Params: {} 2024-04-16T19:46:34.708 DEBUG:boto:Token: None 2024-04-16T19:46:34.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/ 2024-04-16T19:46:34.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA= 2024-04-16T19:46:34.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA=', 'Content-Length': '0'} 2024-04-16T19:46:34.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e50efd82e22856c-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.711 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:34.711 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:34.711 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.711 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:46:34.711 DEBUG:boto:Data: 2024-04-16T19:46:34.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.711 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.711 DEBUG:boto:Params: {} 2024-04-16T19:46:34.711 DEBUG:boto:Token: None 2024-04-16T19:46:34.712 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/obj2 2024-04-16T19:46:34.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n7QKcl1PKeN+4Q9YgBpaRZj15CQ= 2024-04-16T19:46:34.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n7QKcl1PKeN+4Q9YgBpaRZj15CQ='} 2024-04-16T19:46:34.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000793d58e4703a6427-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.755 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:34.755 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:34.755 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.755 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:34.755 DEBUG:boto:Data: 2024-04-16T19:46:34.755 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.755 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.755 DEBUG:boto:Params: {} 2024-04-16T19:46:34.755 DEBUG:boto:Token: None 2024-04-16T19:46:34.755 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/ 2024-04-16T19:46:34.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA= 2024-04-16T19:46:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA=', 'Content-Length': '0'} 2024-04-16T19:46:34.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b914d084a6bb207b-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.758 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:34.758 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:34.758 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.758 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:46:34.759 DEBUG:boto:Data: 2024-04-16T19:46:34.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.759 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.759 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.759 DEBUG:boto:Params: {} 2024-04-16T19:46:34.759 DEBUG:boto:Token: None 2024-04-16T19:46:34.759 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/obj3 2024-04-16T19:46:34.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W6cpcYfC9rzo2Xzpd5uW5hSqi48= 2024-04-16T19:46:34.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W6cpcYfC9rzo2Xzpd5uW5hSqi48='} 2024-04-16T19:46:34.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000067493c405504aed-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.767 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:34.767 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:34.767 DEBUG:boto:Method: HEAD 2024-04-16T19:46:34.767 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:34.767 DEBUG:boto:Data: 2024-04-16T19:46:34.767 DEBUG:boto:Headers: {} 2024-04-16T19:46:34.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.767 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.767 DEBUG:boto:Params: {} 2024-04-16T19:46:34.767 DEBUG:boto:Token: None 2024-04-16T19:46:34.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/ 2024-04-16T19:46:34.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA= 2024-04-16T19:46:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1IQQTn1iUX7OD2u5ff0D/uHVEdA=', 'Content-Length': '0'} 2024-04-16T19:46:34.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c700fb3d86dc74e4-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.770 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:34.770 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:34.770 DEBUG:boto:Method: PUT 2024-04-16T19:46:34.770 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:46:34.770 DEBUG:boto:Data: 2024-04-16T19:46:34.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:34.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:34.771 DEBUG:boto:Port: 8000 2024-04-16T19:46:34.771 DEBUG:boto:Params: {} 2024-04-16T19:46:34.771 DEBUG:boto:Token: None 2024-04-16T19:46:34.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:34 GMT /yrjefh-41/obj4 2024-04-16T19:46:34.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gye/QKhUNrJPsQnYxev3Ey8+eGE= 2024-04-16T19:46:34.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gye/QKhUNrJPsQnYxev3Ey8+eGE='} 2024-04-16T19:46:34.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a610e487a5d61d32-00661ed59a-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:34.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.777 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.874 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:46:34.896 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296794.566388_501.1","last_update":"2024-04-16T19:46:34.566388Z"}] 2024-04-16T19:46:34.897 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296794.566388_501.1'} 2024-04-16T19:46:34.897 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:46:34.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:34.897 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:34.945 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:34.945 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:34.993+0000 7fbc17b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:34.995+0000 7fbc17b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:35.017 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296794.566388_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:34.566388Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:35.017 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296794.566388_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:34.566388Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:34Z"} 2024-04-16T19:46:35.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296794.566388_501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:34.566388Z', 'realm_epoch': 2}}] 2024-04-16T19:46:35.017 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296794.566388_501.1'} 2024-04-16T19:46:35.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296794.566388_501.1'} 2024-04-16T19:46:35.017 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:46:35.017 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-40 2024-04-16T19:46:35.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:35.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:35.018 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-40 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:35.072 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:35.072 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:35.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:35.126+0000 7f0e67f1f800 1 waiting to reach incremental sync.. 2024-04-16T19:46:40.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:40.128+0000 7f0e67f1f800 1 bucket sync caught up with source: 2024-04-16T19:46:40.130 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.45.6, , 00000000001.45.6, , 00000000001.50.6, , , 00000000001.72.6, , , ] 2024-04-16T19:46:40.130 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.45.6, , 00000000001.45.6, , 00000000001.50.6, , , 00000000001.72.6, , , ] 2024-04-16T19:46:40.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:40.128+0000 7f0e67f1f800 0 bucket checkpoint complete 2024-04-16T19:46:40.156 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-40 2024-04-16T19:46:40.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:40.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:40.156 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-40 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:40.218 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:40.218 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:40.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:40.277+0000 7f692623c800 1 waiting to reach incremental sync.. 2024-04-16T19:46:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:45.279+0000 7f692623c800 1 bucket sync caught up with source: 2024-04-16T19:46:45.283 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.83.6, , 00000000001.143.6, , 00000000001.101.6, , , 00000000001.183.6, , , ] 2024-04-16T19:46:45.283 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.83.6, , 00000000001.143.6, , 00000000001.101.6, , , 00000000001.183.6, , , ] 2024-04-16T19:46:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:45.279+0000 7f692623c800 0 bucket checkpoint complete 2024-04-16T19:46:45.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-40 zones={test-zone2, test-zone1} 2024-04-16T19:46:45.308 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.309 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.309 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.309 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:45.309 DEBUG:boto:Data: 2024-04-16T19:46:45.309 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.309 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.309 DEBUG:boto:Params: {} 2024-04-16T19:46:45.310 DEBUG:boto:Token: None 2024-04-16T19:46:45.310 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.310 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FzkH1L7PLtgySNuUJOKv7bFNB5k= 2024-04-16T19:46:45.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FzkH1L7PLtgySNuUJOKv7bFNB5k=', 'Content-Length': '0'} 2024-04-16T19:46:45.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f740bf871abd41c6-00661ed5a5-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.313 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.314 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.314 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.314 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:45.314 DEBUG:boto:Data: 2024-04-16T19:46:45.314 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.314 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.314 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.315 DEBUG:boto:Params: {} 2024-04-16T19:46:45.315 DEBUG:boto:Token: None 2024-04-16T19:46:45.315 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FzkH1L7PLtgySNuUJOKv7bFNB5k= 2024-04-16T19:46:45.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FzkH1L7PLtgySNuUJOKv7bFNB5k=', 'Content-Length': '0'} 2024-04-16T19:46:45.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cb309c809c3e38d-00661ed5a5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.318 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:46:45.318 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.319 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.319 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.319 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.319 DEBUG:boto:Method: GET 2024-04-16T19:46:45.319 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.319 DEBUG:boto:Data: 2024-04-16T19:46:45.319 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.320 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.320 DEBUG:boto:Params: {} 2024-04-16T19:46:45.320 DEBUG:boto:Token: None 2024-04-16T19:46:45.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/?versions 2024-04-16T19:46:45.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0= 2024-04-16T19:46:45.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0=', 'Content-Length': '0'} 2024-04-16T19:46:45.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a037f51ef656d3-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.323 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:45.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:45.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:45.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:45.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:46:45.325 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.325 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.325 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.325 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.325 DEBUG:boto:Method: GET 2024-04-16T19:46:45.325 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.325 DEBUG:boto:Data: 2024-04-16T19:46:45.325 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.326 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.326 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.326 DEBUG:boto:Params: {} 2024-04-16T19:46:45.326 DEBUG:boto:Token: None 2024-04-16T19:46:45.326 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/?versions 2024-04-16T19:46:45.326 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0= 2024-04-16T19:46:45.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0=', 'Content-Length': '0'} 2024-04-16T19:46:45.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f36d8ea76ee81d-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.329 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:45.330 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.330 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.331 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.331 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.331 DEBUG:boto:Method: GET 2024-04-16T19:46:45.331 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.331 DEBUG:boto:Data: 2024-04-16T19:46:45.331 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.331 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.331 DEBUG:boto:Params: {} 2024-04-16T19:46:45.332 DEBUG:boto:Token: None 2024-04-16T19:46:45.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/?versions 2024-04-16T19:46:45.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0= 2024-04-16T19:46:45.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0=', 'Content-Length': '0'} 2024-04-16T19:46:45.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004e0bb5de8d7482d-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.335 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.335 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.336 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.336 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.336 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.336 DEBUG:boto:Method: GET 2024-04-16T19:46:45.336 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:46:45.336 DEBUG:boto:Data: 2024-04-16T19:46:45.336 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.336 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.336 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.337 DEBUG:boto:Params: {} 2024-04-16T19:46:45.337 DEBUG:boto:Token: None 2024-04-16T19:46:45.337 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/?versions 2024-04-16T19:46:45.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0= 2024-04-16T19:46:45.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uk805akmqpQPzkzBfSI4HA3Kpg0=', 'Content-Length': '0'} 2024-04-16T19:46:45.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b412c5f3a8d164e-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.340 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:45.341 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.341 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.341 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.341 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.341 DEBUG:boto:Method: GET 2024-04-16T19:46:45.341 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.341 DEBUG:boto:Data: 2024-04-16T19:46:45.341 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.341 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.342 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.342 DEBUG:boto:Params: {} 2024-04-16T19:46:45.342 DEBUG:boto:Token: None 2024-04-16T19:46:45.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA= 2024-04-16T19:46:45.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA=', 'Content-Length': '0'} 2024-04-16T19:46:45.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009264feb27f8958cd-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.386 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.387 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.387 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.387 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.387 DEBUG:boto:Method: GET 2024-04-16T19:46:45.387 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.387 DEBUG:boto:Data: 2024-04-16T19:46:45.387 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.388 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.388 DEBUG:boto:Params: {} 2024-04-16T19:46:45.388 DEBUG:boto:Token: None 2024-04-16T19:46:45.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA= 2024-04-16T19:46:45.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA=', 'Content-Length': '0'} 2024-04-16T19:46:45.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f25d3d354549dcb4-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.391 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.391 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.391 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.391 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.391 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.392 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.392 DEBUG:boto:Data: 2024-04-16T19:46:45.392 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.392 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.392 DEBUG:boto:Params: {} 2024-04-16T19:46:45.392 DEBUG:boto:Token: None 2024-04-16T19:46:45.392 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MpDzEAQEqjJ6yVmuV+syQfQiiJE= 2024-04-16T19:46:45.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MpDzEAQEqjJ6yVmuV+syQfQiiJE=', 'Content-Length': '0'} 2024-04-16T19:46:45.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a8bf0bc68c717d-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.395 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.395 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.395 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.395 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.396 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.396 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.396 DEBUG:boto:Data: 2024-04-16T19:46:45.396 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.396 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.396 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.396 DEBUG:boto:Params: {} 2024-04-16T19:46:45.396 DEBUG:boto:Token: None 2024-04-16T19:46:45.397 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.397 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MpDzEAQEqjJ6yVmuV+syQfQiiJE= 2024-04-16T19:46:45.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MpDzEAQEqjJ6yVmuV+syQfQiiJE=', 'Content-Length': '0'} 2024-04-16T19:46:45.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005daaa00e633b165d-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:45.399 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.399 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.400 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.400 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.400 DEBUG:boto:Method: GET 2024-04-16T19:46:45.400 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.400 DEBUG:boto:Data: 2024-04-16T19:46:45.400 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.400 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.400 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.401 DEBUG:boto:Params: {} 2024-04-16T19:46:45.401 DEBUG:boto:Token: None 2024-04-16T19:46:45.401 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA= 2024-04-16T19:46:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA=', 'Content-Length': '0'} 2024-04-16T19:46:45.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f618e0995e234770-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.403 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.404 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.404 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.404 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.404 DEBUG:boto:Method: GET 2024-04-16T19:46:45.404 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.404 DEBUG:boto:Data: 2024-04-16T19:46:45.404 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.404 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.405 DEBUG:boto:Params: {} 2024-04-16T19:46:45.405 DEBUG:boto:Token: None 2024-04-16T19:46:45.405 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:46:45.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA= 2024-04-16T19:46:45.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S1RSWUNchwayCoD5pOYmzCqyirA=', 'Content-Length': '0'} 2024-04-16T19:46:45.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057768c0471ec271c-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.451 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.451 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.451 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.451 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.451 DEBUG:boto:Method: GET 2024-04-16T19:46:45.451 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.451 DEBUG:boto:Data: 2024-04-16T19:46:45.451 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.452 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.452 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.452 DEBUG:boto:Params: {} 2024-04-16T19:46:45.452 DEBUG:boto:Token: None 2024-04-16T19:46:45.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f97e4908bde489a7-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.455 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.456 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.456 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.456 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.456 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.456 DEBUG:boto:Method: GET 2024-04-16T19:46:45.456 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.456 DEBUG:boto:Data: 2024-04-16T19:46:45.457 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.457 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.457 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.457 DEBUG:boto:Params: {} 2024-04-16T19:46:45.457 DEBUG:boto:Token: None 2024-04-16T19:46:45.457 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.457 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d56a2846834f2346-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.461 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.461 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.462 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.462 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.462 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:46:45.462 DEBUG:boto:Data: 2024-04-16T19:46:45.462 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.462 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.462 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.463 DEBUG:boto:Params: {} 2024-04-16T19:46:45.463 DEBUG:boto:Token: None 2024-04-16T19:46:45.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1 2024-04-16T19:46:45.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5a8Uoq5Yo8ignzQNUjdRTbwbmWI= 2024-04-16T19:46:45.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5a8Uoq5Yo8ignzQNUjdRTbwbmWI=', 'Content-Length': '0'} 2024-04-16T19:46:45.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035007bbe9176fa3a-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.465 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.466 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.466 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.466 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:46:45.466 DEBUG:boto:Data: 2024-04-16T19:46:45.466 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.466 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.466 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.466 DEBUG:boto:Params: {} 2024-04-16T19:46:45.467 DEBUG:boto:Token: None 2024-04-16T19:46:45.467 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1 2024-04-16T19:46:45.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5a8Uoq5Yo8ignzQNUjdRTbwbmWI= 2024-04-16T19:46:45.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5a8Uoq5Yo8ignzQNUjdRTbwbmWI=', 'Content-Length': '0'} 2024-04-16T19:46:45.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008613e670f0aec4bc-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:45.470 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.470 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.470 DEBUG:boto:Method: GET 2024-04-16T19:46:45.470 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:46:45.470 DEBUG:boto:Data: 2024-04-16T19:46:45.470 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.470 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.471 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.471 DEBUG:boto:Params: {} 2024-04-16T19:46:45.471 DEBUG:boto:Token: None 2024-04-16T19:46:45.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1 2024-04-16T19:46:45.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sNIfC6kO9fXNd1i3tf+uLSHfvts= 2024-04-16T19:46:45.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sNIfC6kO9fXNd1i3tf+uLSHfvts=', 'Content-Length': '0'} 2024-04-16T19:46:45.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f272a096bd65b08-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.514 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:46:45.515 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:46:45.515 DEBUG:boto:Method: GET 2024-04-16T19:46:45.515 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:46:45.515 DEBUG:boto:Data: 2024-04-16T19:46:45.515 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.515 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.515 DEBUG:boto:Params: {} 2024-04-16T19:46:45.516 DEBUG:boto:Token: None 2024-04-16T19:46:45.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj1 2024-04-16T19:46:45.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sNIfC6kO9fXNd1i3tf+uLSHfvts= 2024-04-16T19:46:45.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sNIfC6kO9fXNd1i3tf+uLSHfvts=', 'Content-Length': '0'} 2024-04-16T19:46:45.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa4a20e3c84600c6-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.519 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.519 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.519 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.519 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.519 DEBUG:boto:Method: GET 2024-04-16T19:46:45.519 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.520 DEBUG:boto:Data: 2024-04-16T19:46:45.520 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.520 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.520 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.520 DEBUG:boto:Params: {} 2024-04-16T19:46:45.520 DEBUG:boto:Token: None 2024-04-16T19:46:45.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2528401057141a-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.523 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.524 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.524 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.524 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.524 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.524 DEBUG:boto:Method: GET 2024-04-16T19:46:45.525 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:46:45.525 DEBUG:boto:Data: 2024-04-16T19:46:45.525 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.525 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.525 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.525 DEBUG:boto:Params: {} 2024-04-16T19:46:45.525 DEBUG:boto:Token: None 2024-04-16T19:46:45.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493a3dfe789078c6-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.570 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:45.571 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.571 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.571 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.571 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.572 DEBUG:boto:Method: GET 2024-04-16T19:46:45.572 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.572 DEBUG:boto:Data: 2024-04-16T19:46:45.572 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.572 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.572 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.572 DEBUG:boto:Params: {} 2024-04-16T19:46:45.572 DEBUG:boto:Token: None 2024-04-16T19:46:45.573 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.573 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ= 2024-04-16T19:46:45.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ=', 'Content-Length': '0'} 2024-04-16T19:46:45.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028ed12dca52ef683-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.618 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.619 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.619 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.619 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.619 DEBUG:boto:Method: GET 2024-04-16T19:46:45.619 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.619 DEBUG:boto:Data: 2024-04-16T19:46:45.619 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.620 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.620 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.620 DEBUG:boto:Params: {} 2024-04-16T19:46:45.620 DEBUG:boto:Token: None 2024-04-16T19:46:45.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ= 2024-04-16T19:46:45.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ=', 'Content-Length': '0'} 2024-04-16T19:46:45.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a63b75787a71f49-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.623 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.623 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.623 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.623 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.623 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.624 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.624 DEBUG:boto:Data: 2024-04-16T19:46:45.624 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.624 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.624 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.624 DEBUG:boto:Params: {} 2024-04-16T19:46:45.624 DEBUG:boto:Token: None 2024-04-16T19:46:45.624 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.625 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZWhKnnUwLQQSnSV24Uaua4mZmWs= 2024-04-16T19:46:45.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZWhKnnUwLQQSnSV24Uaua4mZmWs=', 'Content-Length': '0'} 2024-04-16T19:46:45.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000941a77825ad0b6ae-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.627 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.627 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.628 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.628 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.628 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.628 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.628 DEBUG:boto:Data: 2024-04-16T19:46:45.628 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.628 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.629 DEBUG:boto:Params: {} 2024-04-16T19:46:45.629 DEBUG:boto:Token: None 2024-04-16T19:46:45.629 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZWhKnnUwLQQSnSV24Uaua4mZmWs= 2024-04-16T19:46:45.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZWhKnnUwLQQSnSV24Uaua4mZmWs=', 'Content-Length': '0'} 2024-04-16T19:46:45.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf8f9182d6f8ea8c-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:45.632 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.632 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.632 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.632 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.633 DEBUG:boto:Method: GET 2024-04-16T19:46:45.633 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.633 DEBUG:boto:Data: 2024-04-16T19:46:45.633 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.633 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.633 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.633 DEBUG:boto:Params: {} 2024-04-16T19:46:45.633 DEBUG:boto:Token: None 2024-04-16T19:46:45.634 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ= 2024-04-16T19:46:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ=', 'Content-Length': '0'} 2024-04-16T19:46:45.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e4ada9def825bfb-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.678 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.679 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.679 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.679 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.679 DEBUG:boto:Method: GET 2024-04-16T19:46:45.679 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.679 DEBUG:boto:Data: 2024-04-16T19:46:45.679 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.679 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.679 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.680 DEBUG:boto:Params: {} 2024-04-16T19:46:45.680 DEBUG:boto:Token: None 2024-04-16T19:46:45.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:46:45.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ= 2024-04-16T19:46:45.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6NBMuaSyytmZPWPQtFgsz0+6etQ=', 'Content-Length': '0'} 2024-04-16T19:46:45.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acb316be906acce6-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.682 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.683 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.683 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.683 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.683 DEBUG:boto:Method: GET 2024-04-16T19:46:45.683 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.683 DEBUG:boto:Data: 2024-04-16T19:46:45.683 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.683 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.683 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.684 DEBUG:boto:Params: {} 2024-04-16T19:46:45.684 DEBUG:boto:Token: None 2024-04-16T19:46:45.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b524b91527560610-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.687 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.687 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.687 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.687 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.688 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.688 DEBUG:boto:Method: GET 2024-04-16T19:46:45.688 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.688 DEBUG:boto:Data: 2024-04-16T19:46:45.688 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.688 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.688 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.688 DEBUG:boto:Params: {} 2024-04-16T19:46:45.688 DEBUG:boto:Token: None 2024-04-16T19:46:45.689 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072351bdc309d5efb-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.692 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.692 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.692 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.692 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.692 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:46:45.692 DEBUG:boto:Data: 2024-04-16T19:46:45.692 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.693 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.693 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.693 DEBUG:boto:Params: {} 2024-04-16T19:46:45.693 DEBUG:boto:Token: None 2024-04-16T19:46:45.693 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2 2024-04-16T19:46:45.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qqD5fVQzuFcqiCn0MwWpUg2EOkA= 2024-04-16T19:46:45.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qqD5fVQzuFcqiCn0MwWpUg2EOkA=', 'Content-Length': '0'} 2024-04-16T19:46:45.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099a5696e8ec66e95-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.695 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.696 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.696 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.696 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:46:45.696 DEBUG:boto:Data: 2024-04-16T19:46:45.696 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.696 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.696 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.696 DEBUG:boto:Params: {} 2024-04-16T19:46:45.696 DEBUG:boto:Token: None 2024-04-16T19:46:45.697 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2 2024-04-16T19:46:45.697 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qqD5fVQzuFcqiCn0MwWpUg2EOkA= 2024-04-16T19:46:45.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qqD5fVQzuFcqiCn0MwWpUg2EOkA=', 'Content-Length': '0'} 2024-04-16T19:46:45.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016f0feeef41e5744-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:45.699 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.699 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.699 DEBUG:boto:Method: GET 2024-04-16T19:46:45.699 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:46:45.700 DEBUG:boto:Data: 2024-04-16T19:46:45.700 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.700 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.700 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.700 DEBUG:boto:Params: {} 2024-04-16T19:46:45.700 DEBUG:boto:Token: None 2024-04-16T19:46:45.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2 2024-04-16T19:46:45.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NindEIM6J3yni+fOPNCZUp0wpD8= 2024-04-16T19:46:45.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NindEIM6J3yni+fOPNCZUp0wpD8=', 'Content-Length': '0'} 2024-04-16T19:46:45.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cdb68a2fdf7f6f4-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.746 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:46:45.746 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:46:45.747 DEBUG:boto:Method: GET 2024-04-16T19:46:45.747 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:46:45.747 DEBUG:boto:Data: 2024-04-16T19:46:45.747 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.747 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.747 DEBUG:boto:Params: {} 2024-04-16T19:46:45.747 DEBUG:boto:Token: None 2024-04-16T19:46:45.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj2 2024-04-16T19:46:45.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NindEIM6J3yni+fOPNCZUp0wpD8= 2024-04-16T19:46:45.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NindEIM6J3yni+fOPNCZUp0wpD8=', 'Content-Length': '0'} 2024-04-16T19:46:45.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030d9aa558115b29e-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.750 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.750 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.750 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.750 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.750 DEBUG:boto:Method: GET 2024-04-16T19:46:45.750 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.751 DEBUG:boto:Data: 2024-04-16T19:46:45.751 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.751 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.751 DEBUG:boto:Params: {} 2024-04-16T19:46:45.751 DEBUG:boto:Token: None 2024-04-16T19:46:45.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402f9a046993d996-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.754 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.755 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.755 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.755 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.755 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.755 DEBUG:boto:Method: GET 2024-04-16T19:46:45.755 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:46:45.755 DEBUG:boto:Data: 2024-04-16T19:46:45.755 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.756 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.756 DEBUG:boto:Params: {} 2024-04-16T19:46:45.756 DEBUG:boto:Token: None 2024-04-16T19:46:45.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d87c649a3ebae6-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.759 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:45.759 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.759 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.760 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.760 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.760 DEBUG:boto:Method: GET 2024-04-16T19:46:45.760 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.760 DEBUG:boto:Data: 2024-04-16T19:46:45.760 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.760 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.760 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.760 DEBUG:boto:Params: {} 2024-04-16T19:46:45.761 DEBUG:boto:Token: None 2024-04-16T19:46:45.761 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY= 2024-04-16T19:46:45.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY=', 'Content-Length': '0'} 2024-04-16T19:46:45.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fc1b5852de3e671-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.806 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.806 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.806 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.807 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.807 DEBUG:boto:Method: GET 2024-04-16T19:46:45.807 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.807 DEBUG:boto:Data: 2024-04-16T19:46:45.807 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.807 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.807 DEBUG:boto:Params: {} 2024-04-16T19:46:45.807 DEBUG:boto:Token: None 2024-04-16T19:46:45.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY= 2024-04-16T19:46:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY=', 'Content-Length': '0'} 2024-04-16T19:46:45.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005be4b2091737b81a-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.810 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.810 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.810 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.810 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.810 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.810 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.810 DEBUG:boto:Data: 2024-04-16T19:46:45.810 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.810 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.810 DEBUG:boto:Params: {} 2024-04-16T19:46:45.811 DEBUG:boto:Token: None 2024-04-16T19:46:45.811 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z3KNjcUd0+MUgNgsutK20h/QU9M= 2024-04-16T19:46:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z3KNjcUd0+MUgNgsutK20h/QU9M=', 'Content-Length': '0'} 2024-04-16T19:46:45.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc9ad3b4bf1aa690-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.813 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.813 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.813 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.813 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.813 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.814 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.814 DEBUG:boto:Data: 2024-04-16T19:46:45.814 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.814 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.814 DEBUG:boto:Params: {} 2024-04-16T19:46:45.814 DEBUG:boto:Token: None 2024-04-16T19:46:45.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z3KNjcUd0+MUgNgsutK20h/QU9M= 2024-04-16T19:46:45.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z3KNjcUd0+MUgNgsutK20h/QU9M=', 'Content-Length': '0'} 2024-04-16T19:46:45.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053de45507bb99764-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:45.816 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.817 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.817 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.817 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.817 DEBUG:boto:Method: GET 2024-04-16T19:46:45.817 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.817 DEBUG:boto:Data: 2024-04-16T19:46:45.817 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.817 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.817 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.817 DEBUG:boto:Params: {} 2024-04-16T19:46:45.817 DEBUG:boto:Token: None 2024-04-16T19:46:45.818 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY= 2024-04-16T19:46:45.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY=', 'Content-Length': '0'} 2024-04-16T19:46:45.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6447ebafca9bf53-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.862 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.862 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.863 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.863 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.863 DEBUG:boto:Method: GET 2024-04-16T19:46:45.863 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.863 DEBUG:boto:Data: 2024-04-16T19:46:45.863 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.863 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.863 DEBUG:boto:Params: {} 2024-04-16T19:46:45.863 DEBUG:boto:Token: None 2024-04-16T19:46:45.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:46:45.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY= 2024-04-16T19:46:45.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dvftccYgMgb2+PRkNsCUFoJx4XY=', 'Content-Length': '0'} 2024-04-16T19:46:45.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f6b3372f0511a1e-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.866 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.866 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.866 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.866 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.866 DEBUG:boto:Method: GET 2024-04-16T19:46:45.866 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.867 DEBUG:boto:Data: 2024-04-16T19:46:45.867 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.867 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.867 DEBUG:boto:Params: {} 2024-04-16T19:46:45.867 DEBUG:boto:Token: None 2024-04-16T19:46:45.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4d876f0ebab9a8-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.910 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.911 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.911 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.911 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.911 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.911 DEBUG:boto:Method: GET 2024-04-16T19:46:45.911 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.911 DEBUG:boto:Data: 2024-04-16T19:46:45.911 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.911 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.911 DEBUG:boto:Params: {} 2024-04-16T19:46:45.912 DEBUG:boto:Token: None 2024-04-16T19:46:45.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004129a20c23f644e7-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.914 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.914 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.915 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.915 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.915 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:46:45.915 DEBUG:boto:Data: 2024-04-16T19:46:45.915 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.915 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.915 DEBUG:boto:Params: {} 2024-04-16T19:46:45.915 DEBUG:boto:Token: None 2024-04-16T19:46:45.915 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3 2024-04-16T19:46:45.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ICGh5n9nqPY7dzjYLMA4wPUgvzA= 2024-04-16T19:46:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ICGh5n9nqPY7dzjYLMA4wPUgvzA=', 'Content-Length': '0'} 2024-04-16T19:46:45.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad230fbdfd3385b4-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.918 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.918 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.918 DEBUG:boto:Method: HEAD 2024-04-16T19:46:45.918 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:46:45.918 DEBUG:boto:Data: 2024-04-16T19:46:45.918 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.918 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.918 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.918 DEBUG:boto:Params: {} 2024-04-16T19:46:45.918 DEBUG:boto:Token: None 2024-04-16T19:46:45.919 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3 2024-04-16T19:46:45.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ICGh5n9nqPY7dzjYLMA4wPUgvzA= 2024-04-16T19:46:45.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ICGh5n9nqPY7dzjYLMA4wPUgvzA=', 'Content-Length': '0'} 2024-04-16T19:46:45.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c46d082292cd4f1-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:45.921 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.921 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.921 DEBUG:boto:Method: GET 2024-04-16T19:46:45.921 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:46:45.921 DEBUG:boto:Data: 2024-04-16T19:46:45.921 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.921 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.921 DEBUG:boto:Params: {} 2024-04-16T19:46:45.922 DEBUG:boto:Token: None 2024-04-16T19:46:45.922 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3 2024-04-16T19:46:45.922 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ADuPQn/s4Abqdqw4RZGFApmAvNc= 2024-04-16T19:46:45.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ADuPQn/s4Abqdqw4RZGFApmAvNc=', 'Content-Length': '0'} 2024-04-16T19:46:45.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f0bc0f27d2345af-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.966 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:46:45.966 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:46:45.967 DEBUG:boto:Method: GET 2024-04-16T19:46:45.967 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:46:45.967 DEBUG:boto:Data: 2024-04-16T19:46:45.967 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.967 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.967 DEBUG:boto:Params: {} 2024-04-16T19:46:45.967 DEBUG:boto:Token: None 2024-04-16T19:46:45.967 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj3 2024-04-16T19:46:45.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ADuPQn/s4Abqdqw4RZGFApmAvNc= 2024-04-16T19:46:45.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ADuPQn/s4Abqdqw4RZGFApmAvNc=', 'Content-Length': '0'} 2024-04-16T19:46:45.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000271ae7a49250ce75-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.970 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.970 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.970 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.970 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.970 DEBUG:boto:Method: GET 2024-04-16T19:46:45.970 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.970 DEBUG:boto:Data: 2024-04-16T19:46:45.970 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.970 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.970 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.970 DEBUG:boto:Params: {} 2024-04-16T19:46:45.971 DEBUG:boto:Token: None 2024-04-16T19:46:45.971 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e461c86c8dd73142-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.973 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.974 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:45.974 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:45.974 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.974 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.974 DEBUG:boto:Method: GET 2024-04-16T19:46:45.974 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:46:45.974 DEBUG:boto:Data: 2024-04-16T19:46:45.974 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.974 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:45.975 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.975 DEBUG:boto:Params: {} 2024-04-16T19:46:45.975 DEBUG:boto:Token: None 2024-04-16T19:46:45.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/ 2024-04-16T19:46:45.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g= 2024-04-16T19:46:45.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlRAeqknYW+8BEl9uIiFpLn//9g=', 'Content-Length': '0'} 2024-04-16T19:46:45.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cf09e38be85a948-00661ed5a5-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:45.977 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:45.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:45.978 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:45.978 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:45.978 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:45.978 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:45.978 DEBUG:boto:Method: GET 2024-04-16T19:46:45.978 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:45.978 DEBUG:boto:Data: 2024-04-16T19:46:45.978 DEBUG:boto:Headers: {} 2024-04-16T19:46:45.978 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:45.978 DEBUG:boto:Port: 8000 2024-04-16T19:46:45.979 DEBUG:boto:Params: {} 2024-04-16T19:46:45.979 DEBUG:boto:Token: None 2024-04-16T19:46:45.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:45 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:45.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A9pGPUUjHvQfqy3ctJLCApDEmjI= 2024-04-16T19:46:45.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A9pGPUUjHvQfqy3ctJLCApDEmjI=', 'Content-Length': '0'} 2024-04-16T19:46:45.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7964734f7e4846a-00661ed5a5-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.022 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.022 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.022 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.023 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.023 DEBUG:boto:Method: GET 2024-04-16T19:46:46.023 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.023 DEBUG:boto:Data: 2024-04-16T19:46:46.023 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.023 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.023 DEBUG:boto:Params: {} 2024-04-16T19:46:46.023 DEBUG:boto:Token: None 2024-04-16T19:46:46.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY= 2024-04-16T19:46:46.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY=', 'Content-Length': '0'} 2024-04-16T19:46:46.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5f223c72050880d-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.026 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.026 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.026 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.026 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.026 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.026 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.026 DEBUG:boto:Data: 2024-04-16T19:46:46.026 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.027 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.027 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.027 DEBUG:boto:Params: {} 2024-04-16T19:46:46.027 DEBUG:boto:Token: None 2024-04-16T19:46:46.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KfTcYWwsDHd8qml8WeaBcoFfibs= 2024-04-16T19:46:46.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KfTcYWwsDHd8qml8WeaBcoFfibs=', 'Content-Length': '0'} 2024-04-16T19:46:46.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a8e14c801b508c2-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.029 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.029 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.029 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.029 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.029 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.029 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.030 DEBUG:boto:Data: 2024-04-16T19:46:46.030 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.030 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.030 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.030 DEBUG:boto:Params: {} 2024-04-16T19:46:46.030 DEBUG:boto:Token: None 2024-04-16T19:46:46.030 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.030 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KfTcYWwsDHd8qml8WeaBcoFfibs= 2024-04-16T19:46:46.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KfTcYWwsDHd8qml8WeaBcoFfibs=', 'Content-Length': '0'} 2024-04-16T19:46:46.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e032cd7ecde30ee-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:46.032 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.033 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.033 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.033 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.033 DEBUG:boto:Method: GET 2024-04-16T19:46:46.033 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.033 DEBUG:boto:Data: 2024-04-16T19:46:46.033 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.033 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.033 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.033 DEBUG:boto:Params: {} 2024-04-16T19:46:46.033 DEBUG:boto:Token: None 2024-04-16T19:46:46.034 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.034 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY= 2024-04-16T19:46:46.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY=', 'Content-Length': '0'} 2024-04-16T19:46:46.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030cec9c8ed08e476-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.078 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.079 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.079 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.079 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.079 DEBUG:boto:Method: GET 2024-04-16T19:46:46.079 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.079 DEBUG:boto:Data: 2024-04-16T19:46:46.079 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.079 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.079 DEBUG:boto:Params: {} 2024-04-16T19:46:46.079 DEBUG:boto:Token: None 2024-04-16T19:46:46.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:46:46.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY= 2024-04-16T19:46:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9c+yMDryUxLhWeV/MAgS6XY0OPY=', 'Content-Length': '0'} 2024-04-16T19:46:46.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b37de90350c0de7-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.082 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:46.082 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:46.082 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.082 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.083 DEBUG:boto:Method: GET 2024-04-16T19:46:46.083 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.083 DEBUG:boto:Data: 2024-04-16T19:46:46.083 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.083 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.083 DEBUG:boto:Params: {} 2024-04-16T19:46:46.083 DEBUG:boto:Token: None 2024-04-16T19:46:46.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/ 2024-04-16T19:46:46.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw= 2024-04-16T19:46:46.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw=', 'Content-Length': '0'} 2024-04-16T19:46:46.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf3c00805bd0b63-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.086 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.086 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:46.086 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:46.086 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.087 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.087 DEBUG:boto:Method: GET 2024-04-16T19:46:46.087 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.087 DEBUG:boto:Data: 2024-04-16T19:46:46.087 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.087 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.087 DEBUG:boto:Params: {} 2024-04-16T19:46:46.087 DEBUG:boto:Token: None 2024-04-16T19:46:46.087 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/ 2024-04-16T19:46:46.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw= 2024-04-16T19:46:46.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw=', 'Content-Length': '0'} 2024-04-16T19:46:46.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256f590f4cc31507-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.090 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.090 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.090 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.090 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.091 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:46:46.091 DEBUG:boto:Data: 2024-04-16T19:46:46.091 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.091 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.091 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.091 DEBUG:boto:Params: {} 2024-04-16T19:46:46.091 DEBUG:boto:Token: None 2024-04-16T19:46:46.091 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4 2024-04-16T19:46:46.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gfRu5naly8+GwoTs62gdHkKDMJg= 2024-04-16T19:46:46.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gfRu5naly8+GwoTs62gdHkKDMJg=', 'Content-Length': '0'} 2024-04-16T19:46:46.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acc77f7207e4eb5e-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.094 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.094 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.094 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.094 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:46:46.094 DEBUG:boto:Data: 2024-04-16T19:46:46.094 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.094 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.094 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.094 DEBUG:boto:Params: {} 2024-04-16T19:46:46.094 DEBUG:boto:Token: None 2024-04-16T19:46:46.095 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4 2024-04-16T19:46:46.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gfRu5naly8+GwoTs62gdHkKDMJg= 2024-04-16T19:46:46.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gfRu5naly8+GwoTs62gdHkKDMJg=', 'Content-Length': '0'} 2024-04-16T19:46:46.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051cb3874a2a349ab-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:46.097 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.097 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.097 DEBUG:boto:Method: GET 2024-04-16T19:46:46.097 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:46:46.097 DEBUG:boto:Data: 2024-04-16T19:46:46.097 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.097 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.098 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.098 DEBUG:boto:Params: {} 2024-04-16T19:46:46.098 DEBUG:boto:Token: None 2024-04-16T19:46:46.098 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4 2024-04-16T19:46:46.098 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:48TolC9tDXT4XBFJwZHaWi88j7c= 2024-04-16T19:46:46.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:48TolC9tDXT4XBFJwZHaWi88j7c=', 'Content-Length': '0'} 2024-04-16T19:46:46.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063ee7d30ed01b043-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.142 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:46:46.142 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:46:46.143 DEBUG:boto:Method: GET 2024-04-16T19:46:46.143 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:46:46.143 DEBUG:boto:Data: 2024-04-16T19:46:46.143 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.143 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.143 DEBUG:boto:Params: {} 2024-04-16T19:46:46.143 DEBUG:boto:Token: None 2024-04-16T19:46:46.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/obj4 2024-04-16T19:46:46.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:48TolC9tDXT4XBFJwZHaWi88j7c= 2024-04-16T19:46:46.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:48TolC9tDXT4XBFJwZHaWi88j7c=', 'Content-Length': '0'} 2024-04-16T19:46:46.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce1784db9bcc1dd5-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.145 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:46.146 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:46.146 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.146 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.146 DEBUG:boto:Method: GET 2024-04-16T19:46:46.146 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.146 DEBUG:boto:Data: 2024-04-16T19:46:46.146 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.146 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.146 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.146 DEBUG:boto:Params: {} 2024-04-16T19:46:46.147 DEBUG:boto:Token: None 2024-04-16T19:46:46.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/ 2024-04-16T19:46:46.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw= 2024-04-16T19:46:46.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw=', 'Content-Length': '0'} 2024-04-16T19:46:46.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2aa7c7bc2fee8d5-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.149 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.150 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:46.150 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:46.150 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.150 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.150 DEBUG:boto:Method: GET 2024-04-16T19:46:46.150 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:46:46.150 DEBUG:boto:Data: 2024-04-16T19:46:46.150 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.151 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.151 DEBUG:boto:Params: {} 2024-04-16T19:46:46.151 DEBUG:boto:Token: None 2024-04-16T19:46:46.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-40/ 2024-04-16T19:46:46.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw= 2024-04-16T19:46:46.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kfr6vJFbr7Q47el1PrfUMkmUqqw=', 'Content-Length': '0'} 2024-04-16T19:46:46.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026321cf52f4e684b-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.154 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-40 zones={test-zone2, test-zone1} 2024-04-16T19:46:46.154 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-41 2024-04-16T19:46:46.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:46.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:46.155 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-41 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:46.217 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:46.217 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:46.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:46.277+0000 7f33a4131800 1 bucket sync caught up with source: 2024-04-16T19:46:46.279 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.26.6, , 00000000001.25.6, , 00000000001.19.6, , , 00000000001.21.6, , , ] 2024-04-16T19:46:46.279 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.26.6, , 00000000001.25.6, , 00000000001.19.6, , , 00000000001.21.6, , , ] 2024-04-16T19:46:46.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:46.277+0000 7f33a4131800 0 bucket checkpoint complete 2024-04-16T19:46:46.300 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-41 2024-04-16T19:46:46.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:46.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:46.300 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-41 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:46.362 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:46.362 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:46.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:46.423+0000 7f47c272c800 1 bucket sync caught up with source: 2024-04-16T19:46:46.426 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.70.6, , 00000000001.64.6, , 00000000001.119.6, , , 00000000001.108.6, , , ] 2024-04-16T19:46:46.426 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.70.6, , 00000000001.64.6, , 00000000001.119.6, , , 00000000001.108.6, , , ] 2024-04-16T19:46:46.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:46:46.423+0000 7f47c272c800 0 bucket checkpoint complete 2024-04-16T19:46:46.447 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-41 zones={test-zone2, test-zone1} 2024-04-16T19:46:46.447 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.447 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.448 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.448 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:46.448 DEBUG:boto:Data: 2024-04-16T19:46:46.448 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.448 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.448 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.448 DEBUG:boto:Params: {} 2024-04-16T19:46:46.448 DEBUG:boto:Token: None 2024-04-16T19:46:46.448 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VPhEO0W03OFxgZAmTHAHU+zokfE= 2024-04-16T19:46:46.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VPhEO0W03OFxgZAmTHAHU+zokfE=', 'Content-Length': '0'} 2024-04-16T19:46:46.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8ccab9c66fe1f9b-00661ed5a6-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.451 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.451 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.451 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.451 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:46.451 DEBUG:boto:Data: 2024-04-16T19:46:46.452 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.452 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.452 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.452 DEBUG:boto:Params: {} 2024-04-16T19:46:46.452 DEBUG:boto:Token: None 2024-04-16T19:46:46.452 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VPhEO0W03OFxgZAmTHAHU+zokfE= 2024-04-16T19:46:46.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VPhEO0W03OFxgZAmTHAHU+zokfE=', 'Content-Length': '0'} 2024-04-16T19:46:46.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001417414bb5231a3a-00661ed5a6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:46:46.455 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.455 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.455 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.455 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.455 DEBUG:boto:Method: GET 2024-04-16T19:46:46.455 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.455 DEBUG:boto:Data: 2024-04-16T19:46:46.455 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.455 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.456 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.456 DEBUG:boto:Params: {} 2024-04-16T19:46:46.456 DEBUG:boto:Token: None 2024-04-16T19:46:46.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/?versions 2024-04-16T19:46:46.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME= 2024-04-16T19:46:46.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME=', 'Content-Length': '0'} 2024-04-16T19:46:46.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033cbb531369b6d3c-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.459 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:46.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:46.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:46.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:46.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:46:46.460 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.460 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.460 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.460 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.460 DEBUG:boto:Method: GET 2024-04-16T19:46:46.460 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.460 DEBUG:boto:Data: 2024-04-16T19:46:46.460 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.460 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.460 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.461 DEBUG:boto:Params: {} 2024-04-16T19:46:46.461 DEBUG:boto:Token: None 2024-04-16T19:46:46.461 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/?versions 2024-04-16T19:46:46.461 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME= 2024-04-16T19:46:46.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME=', 'Content-Length': '0'} 2024-04-16T19:46:46.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6214371ab239cf6-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.464 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:46:46.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:46:46.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:46:46.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:46:46.464 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.465 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.465 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.465 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.465 DEBUG:boto:Method: GET 2024-04-16T19:46:46.465 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.465 DEBUG:boto:Data: 2024-04-16T19:46:46.465 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.465 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.465 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.465 DEBUG:boto:Params: {} 2024-04-16T19:46:46.465 DEBUG:boto:Token: None 2024-04-16T19:46:46.465 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/?versions 2024-04-16T19:46:46.466 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME= 2024-04-16T19:46:46.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME=', 'Content-Length': '0'} 2024-04-16T19:46:46.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035eaae8bad28d312-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.469 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.469 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.469 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.469 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.469 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.469 DEBUG:boto:Method: GET 2024-04-16T19:46:46.469 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:46:46.470 DEBUG:boto:Data: 2024-04-16T19:46:46.470 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.470 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.470 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.470 DEBUG:boto:Params: {} 2024-04-16T19:46:46.470 DEBUG:boto:Token: None 2024-04-16T19:46:46.470 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/?versions 2024-04-16T19:46:46.470 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME= 2024-04-16T19:46:46.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h/63eswDsUXKv7K6l2ADoM32+ME=', 'Content-Length': '0'} 2024-04-16T19:46:46.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62fb3ef52e0fdeb-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.473 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:46.473 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.473 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.473 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.474 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.474 DEBUG:boto:Method: GET 2024-04-16T19:46:46.474 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.474 DEBUG:boto:Data: 2024-04-16T19:46:46.474 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.474 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.474 DEBUG:boto:Params: {} 2024-04-16T19:46:46.474 DEBUG:boto:Token: None 2024-04-16T19:46:46.474 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo= 2024-04-16T19:46:46.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo=', 'Content-Length': '0'} 2024-04-16T19:46:46.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8d75ed9a3fae5e-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.518 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.518 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.519 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.519 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.519 DEBUG:boto:Method: GET 2024-04-16T19:46:46.519 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.519 DEBUG:boto:Data: 2024-04-16T19:46:46.519 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.519 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.519 DEBUG:boto:Params: {} 2024-04-16T19:46:46.519 DEBUG:boto:Token: None 2024-04-16T19:46:46.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo= 2024-04-16T19:46:46.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo=', 'Content-Length': '0'} 2024-04-16T19:46:46.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b005b8e5d0abd6-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.521 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.522 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.522 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.522 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.522 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.522 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.522 DEBUG:boto:Data: 2024-04-16T19:46:46.522 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.522 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.522 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.522 DEBUG:boto:Params: {} 2024-04-16T19:46:46.522 DEBUG:boto:Token: None 2024-04-16T19:46:46.523 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7://a9re9xmiK/a7MZSPDOx/s6Z5k= 2024-04-16T19:46:46.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7://a9re9xmiK/a7MZSPDOx/s6Z5k=', 'Content-Length': '0'} 2024-04-16T19:46:46.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000183ca64a8aa08b8b-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.525 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.525 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.525 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.525 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.525 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.525 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.525 DEBUG:boto:Data: 2024-04-16T19:46:46.525 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.525 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.525 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.525 DEBUG:boto:Params: {} 2024-04-16T19:46:46.526 DEBUG:boto:Token: None 2024-04-16T19:46:46.526 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7://a9re9xmiK/a7MZSPDOx/s6Z5k= 2024-04-16T19:46:46.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7://a9re9xmiK/a7MZSPDOx/s6Z5k=', 'Content-Length': '0'} 2024-04-16T19:46:46.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000008cafe026aab8f6-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:46.528 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.528 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.528 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.528 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.528 DEBUG:boto:Method: GET 2024-04-16T19:46:46.528 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.528 DEBUG:boto:Data: 2024-04-16T19:46:46.528 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.529 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.529 DEBUG:boto:Params: {} 2024-04-16T19:46:46.529 DEBUG:boto:Token: None 2024-04-16T19:46:46.529 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo= 2024-04-16T19:46:46.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo=', 'Content-Length': '0'} 2024-04-16T19:46:46.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7f92ec6310e66a-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.531 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.531 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.531 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.531 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.531 DEBUG:boto:Method: GET 2024-04-16T19:46:46.531 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.531 DEBUG:boto:Data: 2024-04-16T19:46:46.531 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.531 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.532 DEBUG:boto:Params: {} 2024-04-16T19:46:46.532 DEBUG:boto:Token: None 2024-04-16T19:46:46.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:46:46.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo= 2024-04-16T19:46:46.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:blbuHRnvegEVXYqdFNxujibyXGo=', 'Content-Length': '0'} 2024-04-16T19:46:46.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066a8ba4c041c6f83-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.534 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.534 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.534 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.534 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.534 DEBUG:boto:Method: GET 2024-04-16T19:46:46.534 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.534 DEBUG:boto:Data: 2024-04-16T19:46:46.534 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.535 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.535 DEBUG:boto:Params: {} 2024-04-16T19:46:46.535 DEBUG:boto:Token: None 2024-04-16T19:46:46.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049765bf14d44389c-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.538 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.538 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.538 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.538 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.538 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.538 DEBUG:boto:Method: GET 2024-04-16T19:46:46.538 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.538 DEBUG:boto:Data: 2024-04-16T19:46:46.538 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.539 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.539 DEBUG:boto:Params: {} 2024-04-16T19:46:46.539 DEBUG:boto:Token: None 2024-04-16T19:46:46.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c828d2983f9405bc-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.541 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.542 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.542 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.542 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.542 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:46:46.542 DEBUG:boto:Data: 2024-04-16T19:46:46.542 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.542 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.542 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.542 DEBUG:boto:Params: {} 2024-04-16T19:46:46.542 DEBUG:boto:Token: None 2024-04-16T19:46:46.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1 2024-04-16T19:46:46.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YrV9/h1qBoj1vCswwnzhhCa6MLY= 2024-04-16T19:46:46.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YrV9/h1qBoj1vCswwnzhhCa6MLY=', 'Content-Length': '0'} 2024-04-16T19:46:46.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022250c17d3e62b9c-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.545 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.545 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.545 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.545 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:46:46.545 DEBUG:boto:Data: 2024-04-16T19:46:46.545 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.545 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.545 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.545 DEBUG:boto:Params: {} 2024-04-16T19:46:46.545 DEBUG:boto:Token: None 2024-04-16T19:46:46.545 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1 2024-04-16T19:46:46.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YrV9/h1qBoj1vCswwnzhhCa6MLY= 2024-04-16T19:46:46.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YrV9/h1qBoj1vCswwnzhhCa6MLY=', 'Content-Length': '0'} 2024-04-16T19:46:46.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c13ed4d50319f5a-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:46:46.548 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.548 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.548 DEBUG:boto:Method: GET 2024-04-16T19:46:46.548 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:46:46.548 DEBUG:boto:Data: 2024-04-16T19:46:46.548 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.548 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.548 DEBUG:boto:Params: {} 2024-04-16T19:46:46.548 DEBUG:boto:Token: None 2024-04-16T19:46:46.549 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1 2024-04-16T19:46:46.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cSgdVcUT5dLEuFSe+EJkP7w2UaM= 2024-04-16T19:46:46.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSgdVcUT5dLEuFSe+EJkP7w2UaM=', 'Content-Length': '0'} 2024-04-16T19:46:46.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7692468b208b89d-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.550 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:46:46.551 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:46:46.551 DEBUG:boto:Method: GET 2024-04-16T19:46:46.551 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:46:46.551 DEBUG:boto:Data: 2024-04-16T19:46:46.551 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.551 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.551 DEBUG:boto:Params: {} 2024-04-16T19:46:46.551 DEBUG:boto:Token: None 2024-04-16T19:46:46.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj1 2024-04-16T19:46:46.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cSgdVcUT5dLEuFSe+EJkP7w2UaM= 2024-04-16T19:46:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSgdVcUT5dLEuFSe+EJkP7w2UaM=', 'Content-Length': '0'} 2024-04-16T19:46:46.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e80b07ab38207894-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.594 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.594 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.595 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.595 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.595 DEBUG:boto:Method: GET 2024-04-16T19:46:46.595 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.595 DEBUG:boto:Data: 2024-04-16T19:46:46.595 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.595 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.595 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.595 DEBUG:boto:Params: {} 2024-04-16T19:46:46.595 DEBUG:boto:Token: None 2024-04-16T19:46:46.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000509431eaa062c688-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.598 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.598 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.598 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.598 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.599 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.599 DEBUG:boto:Method: GET 2024-04-16T19:46:46.599 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:46:46.599 DEBUG:boto:Data: 2024-04-16T19:46:46.599 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.599 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.599 DEBUG:boto:Params: {} 2024-04-16T19:46:46.599 DEBUG:boto:Token: None 2024-04-16T19:46:46.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005edf86f94eb01888-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.602 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:46.602 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.603 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.603 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.603 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.603 DEBUG:boto:Method: GET 2024-04-16T19:46:46.603 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.603 DEBUG:boto:Data: 2024-04-16T19:46:46.603 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.603 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.603 DEBUG:boto:Params: {} 2024-04-16T19:46:46.603 DEBUG:boto:Token: None 2024-04-16T19:46:46.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY= 2024-04-16T19:46:46.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY=', 'Content-Length': '0'} 2024-04-16T19:46:46.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d1c5262c8588711-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.606 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.606 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.606 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.606 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.606 DEBUG:boto:Method: GET 2024-04-16T19:46:46.606 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.606 DEBUG:boto:Data: 2024-04-16T19:46:46.606 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.606 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.606 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.606 DEBUG:boto:Params: {} 2024-04-16T19:46:46.606 DEBUG:boto:Token: None 2024-04-16T19:46:46.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY= 2024-04-16T19:46:46.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY=', 'Content-Length': '0'} 2024-04-16T19:46:46.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c430155d50fc046-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.650 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.650 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.651 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.651 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.651 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.651 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.651 DEBUG:boto:Data: 2024-04-16T19:46:46.651 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.651 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.651 DEBUG:boto:Params: {} 2024-04-16T19:46:46.651 DEBUG:boto:Token: None 2024-04-16T19:46:46.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L61d/432BsPni2CDZC3a/CD05qw= 2024-04-16T19:46:46.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L61d/432BsPni2CDZC3a/CD05qw=', 'Content-Length': '0'} 2024-04-16T19:46:46.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2f09f999b3d17ba-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.653 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.653 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.654 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.654 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.654 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.654 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.654 DEBUG:boto:Data: 2024-04-16T19:46:46.654 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.654 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.654 DEBUG:boto:Params: {} 2024-04-16T19:46:46.654 DEBUG:boto:Token: None 2024-04-16T19:46:46.654 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L61d/432BsPni2CDZC3a/CD05qw= 2024-04-16T19:46:46.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L61d/432BsPni2CDZC3a/CD05qw=', 'Content-Length': '0'} 2024-04-16T19:46:46.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c1bbdf1525d4b7-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:46.656 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.656 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.657 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.657 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.657 DEBUG:boto:Method: GET 2024-04-16T19:46:46.657 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.657 DEBUG:boto:Data: 2024-04-16T19:46:46.657 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.657 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.657 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.657 DEBUG:boto:Params: {} 2024-04-16T19:46:46.657 DEBUG:boto:Token: None 2024-04-16T19:46:46.657 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.657 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY= 2024-04-16T19:46:46.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY=', 'Content-Length': '0'} 2024-04-16T19:46:46.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a9cfe3fe77c8b09-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.659 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.659 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.659 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.659 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.659 DEBUG:boto:Method: GET 2024-04-16T19:46:46.660 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.660 DEBUG:boto:Data: 2024-04-16T19:46:46.660 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.660 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.660 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.660 DEBUG:boto:Params: {} 2024-04-16T19:46:46.660 DEBUG:boto:Token: None 2024-04-16T19:46:46.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:46:46.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY= 2024-04-16T19:46:46.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XYxzGD2YnHuS0ZUdcPqFaLTBRDY=', 'Content-Length': '0'} 2024-04-16T19:46:46.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002879bc367e80f811-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.702 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.702 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.702 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.703 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.703 DEBUG:boto:Method: GET 2024-04-16T19:46:46.703 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.703 DEBUG:boto:Data: 2024-04-16T19:46:46.703 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.703 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.703 DEBUG:boto:Params: {} 2024-04-16T19:46:46.703 DEBUG:boto:Token: None 2024-04-16T19:46:46.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e6dc6b4786a2e89-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.706 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.706 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.706 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.706 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.707 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.707 DEBUG:boto:Method: GET 2024-04-16T19:46:46.707 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.707 DEBUG:boto:Data: 2024-04-16T19:46:46.707 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.707 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.707 DEBUG:boto:Params: {} 2024-04-16T19:46:46.707 DEBUG:boto:Token: None 2024-04-16T19:46:46.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78de26829e2bbc5-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.750 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.751 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.751 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.751 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.751 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:46:46.751 DEBUG:boto:Data: 2024-04-16T19:46:46.751 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.751 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.751 DEBUG:boto:Params: {} 2024-04-16T19:46:46.751 DEBUG:boto:Token: None 2024-04-16T19:46:46.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2 2024-04-16T19:46:46.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nz1j3QniK/L0CkYEvGWDZKQiQdk= 2024-04-16T19:46:46.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nz1j3QniK/L0CkYEvGWDZKQiQdk=', 'Content-Length': '0'} 2024-04-16T19:46:46.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af87e3c5c6d8ea79-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.754 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.754 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.754 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.754 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:46:46.754 DEBUG:boto:Data: 2024-04-16T19:46:46.754 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.754 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.754 DEBUG:boto:Params: {} 2024-04-16T19:46:46.754 DEBUG:boto:Token: None 2024-04-16T19:46:46.755 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2 2024-04-16T19:46:46.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nz1j3QniK/L0CkYEvGWDZKQiQdk= 2024-04-16T19:46:46.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nz1j3QniK/L0CkYEvGWDZKQiQdk=', 'Content-Length': '0'} 2024-04-16T19:46:46.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad3031003903e8d5-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:46:46.757 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.757 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.757 DEBUG:boto:Method: GET 2024-04-16T19:46:46.757 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:46:46.757 DEBUG:boto:Data: 2024-04-16T19:46:46.757 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.757 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.757 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.757 DEBUG:boto:Params: {} 2024-04-16T19:46:46.757 DEBUG:boto:Token: None 2024-04-16T19:46:46.758 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2 2024-04-16T19:46:46.758 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mPcmVUdFillcd7hOVrQDotaqPio= 2024-04-16T19:46:46.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mPcmVUdFillcd7hOVrQDotaqPio=', 'Content-Length': '0'} 2024-04-16T19:46:46.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014438715e9a5a4cb-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.760 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:46:46.760 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:46:46.760 DEBUG:boto:Method: GET 2024-04-16T19:46:46.760 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:46:46.760 DEBUG:boto:Data: 2024-04-16T19:46:46.760 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.760 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.760 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.760 DEBUG:boto:Params: {} 2024-04-16T19:46:46.760 DEBUG:boto:Token: None 2024-04-16T19:46:46.761 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj2 2024-04-16T19:46:46.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mPcmVUdFillcd7hOVrQDotaqPio= 2024-04-16T19:46:46.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mPcmVUdFillcd7hOVrQDotaqPio=', 'Content-Length': '0'} 2024-04-16T19:46:46.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f599caceb153968-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.806 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.806 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.807 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.807 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.807 DEBUG:boto:Method: GET 2024-04-16T19:46:46.807 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.807 DEBUG:boto:Data: 2024-04-16T19:46:46.807 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.807 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.807 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.807 DEBUG:boto:Params: {} 2024-04-16T19:46:46.807 DEBUG:boto:Token: None 2024-04-16T19:46:46.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9e071dad55385c-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.810 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.810 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.810 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.810 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.810 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.811 DEBUG:boto:Method: GET 2024-04-16T19:46:46.811 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:46:46.811 DEBUG:boto:Data: 2024-04-16T19:46:46.811 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.811 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.811 DEBUG:boto:Params: {} 2024-04-16T19:46:46.811 DEBUG:boto:Token: None 2024-04-16T19:46:46.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f54e154de0a77b-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.814 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:46.814 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.815 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.815 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.815 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.815 DEBUG:boto:Method: GET 2024-04-16T19:46:46.815 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.815 DEBUG:boto:Data: 2024-04-16T19:46:46.815 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.815 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.815 DEBUG:boto:Params: {} 2024-04-16T19:46:46.815 DEBUG:boto:Token: None 2024-04-16T19:46:46.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo= 2024-04-16T19:46:46.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo=', 'Content-Length': '0'} 2024-04-16T19:46:46.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b827dae1a089b56a-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.818 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.818 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.818 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.818 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.818 DEBUG:boto:Method: GET 2024-04-16T19:46:46.818 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.818 DEBUG:boto:Data: 2024-04-16T19:46:46.818 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.819 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.819 DEBUG:boto:Params: {} 2024-04-16T19:46:46.819 DEBUG:boto:Token: None 2024-04-16T19:46:46.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo= 2024-04-16T19:46:46.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo=', 'Content-Length': '0'} 2024-04-16T19:46:46.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036cf9c1d94e6f48b-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.862 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.863 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.863 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.863 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.863 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.863 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.863 DEBUG:boto:Data: 2024-04-16T19:46:46.863 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.863 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.863 DEBUG:boto:Params: {} 2024-04-16T19:46:46.863 DEBUG:boto:Token: None 2024-04-16T19:46:46.864 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:acb5CkoZdPKse+okZSbpdPNpqf4= 2024-04-16T19:46:46.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:acb5CkoZdPKse+okZSbpdPNpqf4=', 'Content-Length': '0'} 2024-04-16T19:46:46.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee7e7a5fa93204df-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.866 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.866 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.866 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.866 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.866 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.866 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.866 DEBUG:boto:Data: 2024-04-16T19:46:46.867 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.867 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.867 DEBUG:boto:Params: {} 2024-04-16T19:46:46.867 DEBUG:boto:Token: None 2024-04-16T19:46:46.867 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:acb5CkoZdPKse+okZSbpdPNpqf4= 2024-04-16T19:46:46.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:acb5CkoZdPKse+okZSbpdPNpqf4=', 'Content-Length': '0'} 2024-04-16T19:46:46.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf13ac70ecf1108f-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:46.869 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.869 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.869 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.870 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.870 DEBUG:boto:Method: GET 2024-04-16T19:46:46.870 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.870 DEBUG:boto:Data: 2024-04-16T19:46:46.870 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.870 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.870 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.870 DEBUG:boto:Params: {} 2024-04-16T19:46:46.870 DEBUG:boto:Token: None 2024-04-16T19:46:46.870 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo= 2024-04-16T19:46:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo=', 'Content-Length': '0'} 2024-04-16T19:46:46.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee5a9a0f4309fbf5-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.873 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.873 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.873 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.873 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.873 DEBUG:boto:Method: GET 2024-04-16T19:46:46.873 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.873 DEBUG:boto:Data: 2024-04-16T19:46:46.873 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.874 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.874 DEBUG:boto:Params: {} 2024-04-16T19:46:46.874 DEBUG:boto:Token: None 2024-04-16T19:46:46.874 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:46:46.874 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo= 2024-04-16T19:46:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:162pwM0S5xRtVVSjTjAZUA6GPQo=', 'Content-Length': '0'} 2024-04-16T19:46:46.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b9bc4ead0cb9b4f-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.918 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.919 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.919 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.919 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.919 DEBUG:boto:Method: GET 2024-04-16T19:46:46.919 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.919 DEBUG:boto:Data: 2024-04-16T19:46:46.919 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.919 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.919 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.919 DEBUG:boto:Params: {} 2024-04-16T19:46:46.919 DEBUG:boto:Token: None 2024-04-16T19:46:46.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449e48a2d4ed7373-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.922 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.923 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.923 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.923 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.923 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.923 DEBUG:boto:Method: GET 2024-04-16T19:46:46.923 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.923 DEBUG:boto:Data: 2024-04-16T19:46:46.923 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.923 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.923 DEBUG:boto:Params: {} 2024-04-16T19:46:46.924 DEBUG:boto:Token: None 2024-04-16T19:46:46.924 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5ce04f77a1addb7-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.927 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.927 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.927 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.927 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.927 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:46:46.927 DEBUG:boto:Data: 2024-04-16T19:46:46.927 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.927 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.928 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.928 DEBUG:boto:Params: {} 2024-04-16T19:46:46.928 DEBUG:boto:Token: None 2024-04-16T19:46:46.928 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3 2024-04-16T19:46:46.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xsv3k4smnBLmhh4elXXiJ621GOU= 2024-04-16T19:46:46.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xsv3k4smnBLmhh4elXXiJ621GOU=', 'Content-Length': '0'} 2024-04-16T19:46:46.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008482e7e64a840df2-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.930 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.930 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.930 DEBUG:boto:Method: HEAD 2024-04-16T19:46:46.930 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:46:46.930 DEBUG:boto:Data: 2024-04-16T19:46:46.931 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.931 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.931 DEBUG:boto:Params: {} 2024-04-16T19:46:46.931 DEBUG:boto:Token: None 2024-04-16T19:46:46.931 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3 2024-04-16T19:46:46.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xsv3k4smnBLmhh4elXXiJ621GOU= 2024-04-16T19:46:46.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xsv3k4smnBLmhh4elXXiJ621GOU=', 'Content-Length': '0'} 2024-04-16T19:46:46.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000469b241fcc55de9f-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:46:46.933 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.933 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.934 DEBUG:boto:Method: GET 2024-04-16T19:46:46.934 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:46:46.934 DEBUG:boto:Data: 2024-04-16T19:46:46.934 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.934 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.934 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.934 DEBUG:boto:Params: {} 2024-04-16T19:46:46.934 DEBUG:boto:Token: None 2024-04-16T19:46:46.934 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3 2024-04-16T19:46:46.934 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cqCQ62p++PKs1uB+75G/DkInrLs= 2024-04-16T19:46:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cqCQ62p++PKs1uB+75G/DkInrLs=', 'Content-Length': '0'} 2024-04-16T19:46:46.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000415bc291d917fc31-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.937 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:46:46.937 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:46:46.937 DEBUG:boto:Method: GET 2024-04-16T19:46:46.937 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:46:46.937 DEBUG:boto:Data: 2024-04-16T19:46:46.937 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.937 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.937 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.937 DEBUG:boto:Params: {} 2024-04-16T19:46:46.937 DEBUG:boto:Token: None 2024-04-16T19:46:46.938 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/obj3 2024-04-16T19:46:46.938 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cqCQ62p++PKs1uB+75G/DkInrLs= 2024-04-16T19:46:46.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cqCQ62p++PKs1uB+75G/DkInrLs=', 'Content-Length': '0'} 2024-04-16T19:46:46.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032452441e98af7f0-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.982 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.982 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.983 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.983 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.983 DEBUG:boto:Method: GET 2024-04-16T19:46:46.983 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.983 DEBUG:boto:Data: 2024-04-16T19:46:46.983 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:46.983 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.983 DEBUG:boto:Params: {} 2024-04-16T19:46:46.983 DEBUG:boto:Token: None 2024-04-16T19:46:46.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b939289fca52696e-00661ed5a6-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:46.987 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:46.987 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:46.987 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:46.987 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.987 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.988 DEBUG:boto:Method: GET 2024-04-16T19:46:46.988 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:46:46.988 DEBUG:boto:Data: 2024-04-16T19:46:46.988 DEBUG:boto:Headers: {} 2024-04-16T19:46:46.988 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:46.988 DEBUG:boto:Port: 8000 2024-04-16T19:46:46.988 DEBUG:boto:Params: {} 2024-04-16T19:46:46.988 DEBUG:boto:Token: None 2024-04-16T19:46:46.988 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:46 GMT /yrjefh-41/ 2024-04-16T19:46:46.989 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII= 2024-04-16T19:46:46.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:46 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fVIOlAmeZMted9baD3Ode4wtxII=', 'Content-Length': '0'} 2024-04-16T19:46:46.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8a54b7dd5768a2-00661ed5a6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.034 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:47.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:47.035 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.035 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.035 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.035 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.035 DEBUG:boto:Method: GET 2024-04-16T19:46:47.035 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.035 DEBUG:boto:Data: 2024-04-16T19:46:47.035 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.035 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.036 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.036 DEBUG:boto:Params: {} 2024-04-16T19:46:47.036 DEBUG:boto:Token: None 2024-04-16T19:46:47.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA= 2024-04-16T19:46:47.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA=', 'Content-Length': '0'} 2024-04-16T19:46:47.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e188cb3631938f7-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.039 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.039 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.039 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.039 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.039 DEBUG:boto:Method: GET 2024-04-16T19:46:47.039 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.039 DEBUG:boto:Data: 2024-04-16T19:46:47.039 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.039 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.040 DEBUG:boto:Params: {} 2024-04-16T19:46:47.040 DEBUG:boto:Token: None 2024-04-16T19:46:47.040 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA= 2024-04-16T19:46:47.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA=', 'Content-Length': '0'} 2024-04-16T19:46:47.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cb8e092285ca36e-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.082 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.083 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.083 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.083 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.083 DEBUG:boto:Method: HEAD 2024-04-16T19:46:47.083 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.083 DEBUG:boto:Data: 2024-04-16T19:46:47.083 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.083 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.083 DEBUG:boto:Params: {} 2024-04-16T19:46:47.083 DEBUG:boto:Token: None 2024-04-16T19:46:47.084 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lzT1EgIC5G3EpyT7glUUsdUefKU= 2024-04-16T19:46:47.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lzT1EgIC5G3EpyT7glUUsdUefKU=', 'Content-Length': '0'} 2024-04-16T19:46:47.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ab9a41498a24543-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.086 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.086 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.086 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.086 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.086 DEBUG:boto:Method: HEAD 2024-04-16T19:46:47.086 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.087 DEBUG:boto:Data: 2024-04-16T19:46:47.087 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.087 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.087 DEBUG:boto:Params: {} 2024-04-16T19:46:47.087 DEBUG:boto:Token: None 2024-04-16T19:46:47.087 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lzT1EgIC5G3EpyT7glUUsdUefKU= 2024-04-16T19:46:47.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lzT1EgIC5G3EpyT7glUUsdUefKU=', 'Content-Length': '0'} 2024-04-16T19:46:47.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de43d6aa7678c142-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:47.090 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.090 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.090 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.090 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.090 DEBUG:boto:Method: GET 2024-04-16T19:46:47.090 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.090 DEBUG:boto:Data: 2024-04-16T19:46:47.090 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.090 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.091 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.091 DEBUG:boto:Params: {} 2024-04-16T19:46:47.091 DEBUG:boto:Token: None 2024-04-16T19:46:47.091 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA= 2024-04-16T19:46:47.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA=', 'Content-Length': '0'} 2024-04-16T19:46:47.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae9f65508aee3b13-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.093 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.093 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.093 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.093 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.093 DEBUG:boto:Method: GET 2024-04-16T19:46:47.094 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.094 DEBUG:boto:Data: 2024-04-16T19:46:47.094 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.094 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.094 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.094 DEBUG:boto:Params: {} 2024-04-16T19:46:47.094 DEBUG:boto:Token: None 2024-04-16T19:46:47.094 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:46:47.094 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA= 2024-04-16T19:46:47.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCQwyFe8fcBWvZO+KAdcbklVNWA=', 'Content-Length': '0'} 2024-04-16T19:46:47.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f884e761d8e1638-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.138 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:47.139 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:47.139 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.139 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.139 DEBUG:boto:Method: GET 2024-04-16T19:46:47.139 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.139 DEBUG:boto:Data: 2024-04-16T19:46:47.139 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.139 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.139 DEBUG:boto:Params: {} 2024-04-16T19:46:47.139 DEBUG:boto:Token: None 2024-04-16T19:46:47.140 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/ 2024-04-16T19:46:47.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo= 2024-04-16T19:46:47.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo=', 'Content-Length': '0'} 2024-04-16T19:46:47.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d9561c8fe4cc15-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.142 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:47.143 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:47.143 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:47.143 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.143 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.143 DEBUG:boto:Method: GET 2024-04-16T19:46:47.143 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.143 DEBUG:boto:Data: 2024-04-16T19:46:47.143 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.143 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.143 DEBUG:boto:Params: {} 2024-04-16T19:46:47.144 DEBUG:boto:Token: None 2024-04-16T19:46:47.144 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/ 2024-04-16T19:46:47.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo= 2024-04-16T19:46:47.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo=', 'Content-Length': '0'} 2024-04-16T19:46:47.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068dbbd0fc14548c7-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.190 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:47.191 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.191 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.191 DEBUG:boto:Method: HEAD 2024-04-16T19:46:47.191 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:46:47.191 DEBUG:boto:Data: 2024-04-16T19:46:47.191 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.191 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.191 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.191 DEBUG:boto:Params: {} 2024-04-16T19:46:47.192 DEBUG:boto:Token: None 2024-04-16T19:46:47.192 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4 2024-04-16T19:46:47.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RqJZw0y7GIX0PFqixXyfugjlb4A= 2024-04-16T19:46:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RqJZw0y7GIX0PFqixXyfugjlb4A=', 'Content-Length': '0'} 2024-04-16T19:46:47.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006798dc7fab06addc-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.194 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.194 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.194 DEBUG:boto:Method: HEAD 2024-04-16T19:46:47.195 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:46:47.195 DEBUG:boto:Data: 2024-04-16T19:46:47.195 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.195 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.195 DEBUG:boto:Params: {} 2024-04-16T19:46:47.195 DEBUG:boto:Token: None 2024-04-16T19:46:47.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4 2024-04-16T19:46:47.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RqJZw0y7GIX0PFqixXyfugjlb4A= 2024-04-16T19:46:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RqJZw0y7GIX0PFqixXyfugjlb4A=', 'Content-Length': '0'} 2024-04-16T19:46:47.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c686cdcd5e4ac061-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:46:47.198 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.198 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.198 DEBUG:boto:Method: GET 2024-04-16T19:46:47.198 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:46:47.198 DEBUG:boto:Data: 2024-04-16T19:46:47.198 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.198 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.198 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.198 DEBUG:boto:Params: {} 2024-04-16T19:46:47.199 DEBUG:boto:Token: None 2024-04-16T19:46:47.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4 2024-04-16T19:46:47.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jeetjDCK67WHK/9MHnEr+04fctM= 2024-04-16T19:46:47.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jeetjDCK67WHK/9MHnEr+04fctM=', 'Content-Length': '0'} 2024-04-16T19:46:47.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000828a748a95abe79-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.201 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:46:47.202 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:46:47.202 DEBUG:boto:Method: GET 2024-04-16T19:46:47.202 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:46:47.202 DEBUG:boto:Data: 2024-04-16T19:46:47.202 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.202 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.202 DEBUG:boto:Params: {} 2024-04-16T19:46:47.202 DEBUG:boto:Token: None 2024-04-16T19:46:47.202 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/obj4 2024-04-16T19:46:47.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jeetjDCK67WHK/9MHnEr+04fctM= 2024-04-16T19:46:47.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jeetjDCK67WHK/9MHnEr+04fctM=', 'Content-Length': '0'} 2024-04-16T19:46:47.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022e60144c9231264-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.246 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:47.247 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:47.247 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.247 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.247 DEBUG:boto:Method: GET 2024-04-16T19:46:47.247 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.247 DEBUG:boto:Data: 2024-04-16T19:46:47.247 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.247 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:47.247 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.247 DEBUG:boto:Params: {} 2024-04-16T19:46:47.247 DEBUG:boto:Token: None 2024-04-16T19:46:47.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/ 2024-04-16T19:46:47.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo= 2024-04-16T19:46:47.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo=', 'Content-Length': '0'} 2024-04-16T19:46:47.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d15bb77773689647-00661ed5a7-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.250 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:47.251 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:47.251 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:47.251 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.251 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.251 DEBUG:boto:Method: GET 2024-04-16T19:46:47.251 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:46:47.251 DEBUG:boto:Data: 2024-04-16T19:46:47.252 DEBUG:boto:Headers: {} 2024-04-16T19:46:47.252 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:47.252 DEBUG:boto:Port: 8000 2024-04-16T19:46:47.252 DEBUG:boto:Params: {} 2024-04-16T19:46:47.252 DEBUG:boto:Token: None 2024-04-16T19:46:47.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:46:47 GMT /yrjefh-41/ 2024-04-16T19:46:47.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo= 2024-04-16T19:46:47.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:47 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6pcxaHbFi0ILoxRb6po9NcvYlmo=', 'Content-Length': '0'} 2024-04-16T19:46:47.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3b2c833ce44032-00661ed5a7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:47.255 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:46:47.255 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-41 zones={test-zone2, test-zone1} 2024-04-16T19:46:47.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:47.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:47.256 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-40 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:46:47.311 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:47.312 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:47.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'yrjefh-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:47.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'yrjefh-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:47.418 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket yrjefh-41 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:46:47.467 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:47.467 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:47.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:47.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:47.567 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:47.664 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:46:47.685 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296807.544531_524.1","last_update":"2024-04-16T19:46:47.544531Z"}] 2024-04-16T19:46:47.686 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296807.544531_524.1'} 2024-04-16T19:46:47.686 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:46:47.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:47.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:47.687 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:47.743 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:47.743 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:47.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:47.795+0000 7f387429d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:47.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:47.796+0000 7f387429d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:47.819 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296794.566388_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:34.566388Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:47Z"} 2024-04-16T19:46:47.820 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296794.566388_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:34.566388Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:47Z"} 2024-04-16T19:46:47.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296794.566388_501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:34.566388Z', 'realm_epoch': 2}}] 2024-04-16T19:46:47.820 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296807.544531_524.1'} 2024-04-16T19:46:47.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296794.566388_501.1'} 2024-04-16T19:46:47.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296807.544531_524.1 target=1_1713296794.566388_501.1 2024-04-16T19:46:52.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:52.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:46:52.822 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:46:52.878 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:52.878 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:52.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:52.929+0000 7fcc89c11800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:52.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:52.931+0000 7fcc89c11800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:46:52.954 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296807.544531_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:47.544531Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:52Z"} 2024-04-16T19:46:52.954 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296807.544531_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:47.544531Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:46:52Z"} 2024-04-16T19:46:52.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296807.544531_524.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:47.544531Z', 'realm_epoch': 2}}] 2024-04-16T19:46:52.955 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296807.544531_524.1'} 2024-04-16T19:46:52.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296807.544531_524.1'} 2024-04-16T19:46:52.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:46:52.955 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:52.955 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:52.955 DEBUG:boto:Method: HEAD 2024-04-16T19:46:52.955 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:52.955 DEBUG:boto:Data: 2024-04-16T19:46:52.956 DEBUG:boto:Headers: {} 2024-04-16T19:46:52.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:52.956 DEBUG:boto:Port: 8000 2024-04-16T19:46:52.956 DEBUG:boto:Params: {} 2024-04-16T19:46:52.956 DEBUG:boto:Token: None 2024-04-16T19:46:52.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:52 GMT /yrjefh-40/ 2024-04-16T19:46:52.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FpW+NhpXN2uHsbxqi3ZtHIWlM0s= 2024-04-16T19:46:52.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FpW+NhpXN2uHsbxqi3ZtHIWlM0s=', 'Content-Length': '0'} 2024-04-16T19:46:52.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f0f36e5037dd12c-00661ed5ac-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:52.959 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:46:52.960 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:46:52.960 DEBUG:boto:Method: PUT 2024-04-16T19:46:52.960 DEBUG:boto:Path: /yrjefh-40/obj5 2024-04-16T19:46:52.960 DEBUG:boto:Data: 2024-04-16T19:46:52.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:52.960 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:52.960 DEBUG:boto:Port: 8000 2024-04-16T19:46:52.960 DEBUG:boto:Params: {} 2024-04-16T19:46:52.960 DEBUG:boto:Token: None 2024-04-16T19:46:52.960 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:52 GMT /yrjefh-40/obj5 2024-04-16T19:46:52.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lB1ukt2k6M2wObUS3fBrjqHEe+4= 2024-04-16T19:46:52.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lB1ukt2k6M2wObUS3fBrjqHEe+4='} 2024-04-16T19:46:53.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcf4b237d5f0fb32-00661ed5ac-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.003 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:53.003 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:53.003 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.003 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:53.003 DEBUG:boto:Data: 2024-04-16T19:46:53.003 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.003 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.003 DEBUG:boto:Params: {} 2024-04-16T19:46:53.004 DEBUG:boto:Token: None 2024-04-16T19:46:53.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/ 2024-04-16T19:46:53.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw= 2024-04-16T19:46:53.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw=', 'Content-Length': '0'} 2024-04-16T19:46:53.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6981eed1fcefffd-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.006 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:46:53.006 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:46:53.007 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.007 DEBUG:boto:Path: /yrjefh-40/obj6 2024-04-16T19:46:53.007 DEBUG:boto:Data: 2024-04-16T19:46:53.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.007 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.007 DEBUG:boto:Params: {} 2024-04-16T19:46:53.007 DEBUG:boto:Token: None 2024-04-16T19:46:53.007 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/obj6 2024-04-16T19:46:53.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5D7SZpEjGVBNJtsATNhBYSiVUhs= 2024-04-16T19:46:53.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5D7SZpEjGVBNJtsATNhBYSiVUhs='} 2024-04-16T19:46:53.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f9f801142a5623d-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.013 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:53.013 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:53.013 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.013 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:53.013 DEBUG:boto:Data: 2024-04-16T19:46:53.013 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.013 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.013 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.013 DEBUG:boto:Params: {} 2024-04-16T19:46:53.014 DEBUG:boto:Token: None 2024-04-16T19:46:53.014 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/ 2024-04-16T19:46:53.014 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw= 2024-04-16T19:46:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw=', 'Content-Length': '0'} 2024-04-16T19:46:53.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b81452829c1b6b3a-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.016 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:46:53.016 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:46:53.016 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.016 DEBUG:boto:Path: /yrjefh-40/obj7 2024-04-16T19:46:53.016 DEBUG:boto:Data: 2024-04-16T19:46:53.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.017 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.017 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.017 DEBUG:boto:Params: {} 2024-04-16T19:46:53.017 DEBUG:boto:Token: None 2024-04-16T19:46:53.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/obj7 2024-04-16T19:46:53.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vJMiRRQJa11wHdL5uZ3FtEvpxvo= 2024-04-16T19:46:53.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vJMiRRQJa11wHdL5uZ3FtEvpxvo='} 2024-04-16T19:46:53.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0f2da04ebc75dbb-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.022 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:46:53.022 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:46:53.022 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.022 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:46:53.023 DEBUG:boto:Data: 2024-04-16T19:46:53.023 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.023 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.023 DEBUG:boto:Params: {} 2024-04-16T19:46:53.023 DEBUG:boto:Token: None 2024-04-16T19:46:53.023 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/ 2024-04-16T19:46:53.023 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw= 2024-04-16T19:46:53.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l7aJjtgreD6nrqYQDbr8LQS1+jw=', 'Content-Length': '0'} 2024-04-16T19:46:53.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3997940f36c1533-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.026 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:46:53.026 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:46:53.026 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.026 DEBUG:boto:Path: /yrjefh-40/obj8 2024-04-16T19:46:53.026 DEBUG:boto:Data: 2024-04-16T19:46:53.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.026 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:46:53.026 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.027 DEBUG:boto:Params: {} 2024-04-16T19:46:53.027 DEBUG:boto:Token: None 2024-04-16T19:46:53.027 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-40/obj8 2024-04-16T19:46:53.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Koy3LraIkjNgONxlFhsv8x8ezjY= 2024-04-16T19:46:53.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Koy3LraIkjNgONxlFhsv8x8ezjY='} 2024-04-16T19:46:53.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f86ad0ebf7ccf92a-00661ed5ad-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.071 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:53.071 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:53.071 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.071 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:53.071 DEBUG:boto:Data: 2024-04-16T19:46:53.071 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.071 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.071 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.071 DEBUG:boto:Params: {} 2024-04-16T19:46:53.071 DEBUG:boto:Token: None 2024-04-16T19:46:53.072 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/ 2024-04-16T19:46:53.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U= 2024-04-16T19:46:53.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U=', 'Content-Length': '0'} 2024-04-16T19:46:53.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048d9e5680f61d962-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.075 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:46:53.075 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:46:53.075 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.075 DEBUG:boto:Path: /yrjefh-41/obj5 2024-04-16T19:46:53.075 DEBUG:boto:Data: 2024-04-16T19:46:53.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.075 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.075 DEBUG:boto:Params: {} 2024-04-16T19:46:53.076 DEBUG:boto:Token: None 2024-04-16T19:46:53.076 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/obj5 2024-04-16T19:46:53.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UEpZqMMJWEoqVvnvhEaOosljoOE= 2024-04-16T19:46:53.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UEpZqMMJWEoqVvnvhEaOosljoOE='} 2024-04-16T19:46:53.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056cb6a8dae54d6ac-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.119 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:53.119 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:53.119 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.119 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:53.119 DEBUG:boto:Data: 2024-04-16T19:46:53.119 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.119 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.119 DEBUG:boto:Params: {} 2024-04-16T19:46:53.120 DEBUG:boto:Token: None 2024-04-16T19:46:53.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/ 2024-04-16T19:46:53.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U= 2024-04-16T19:46:53.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U=', 'Content-Length': '0'} 2024-04-16T19:46:53.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b88fc3916aa4c70f-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.122 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:46:53.123 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:46:53.123 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.123 DEBUG:boto:Path: /yrjefh-41/obj6 2024-04-16T19:46:53.123 DEBUG:boto:Data: 2024-04-16T19:46:53.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.123 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.123 DEBUG:boto:Params: {} 2024-04-16T19:46:53.123 DEBUG:boto:Token: None 2024-04-16T19:46:53.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/obj6 2024-04-16T19:46:53.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/ZXNReGBq3eBD+4upwWqqHHCSI4= 2024-04-16T19:46:53.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/ZXNReGBq3eBD+4upwWqqHHCSI4='} 2024-04-16T19:46:53.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000568c4c1f4f837efa-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.129 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:53.129 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:53.129 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.129 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:53.129 DEBUG:boto:Data: 2024-04-16T19:46:53.129 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.129 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.129 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.129 DEBUG:boto:Params: {} 2024-04-16T19:46:53.129 DEBUG:boto:Token: None 2024-04-16T19:46:53.130 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/ 2024-04-16T19:46:53.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U= 2024-04-16T19:46:53.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U=', 'Content-Length': '0'} 2024-04-16T19:46:53.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000583526fe3252e224-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.132 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:46:53.133 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:46:53.133 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.133 DEBUG:boto:Path: /yrjefh-41/obj7 2024-04-16T19:46:53.133 DEBUG:boto:Data: 2024-04-16T19:46:53.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.133 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.133 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.133 DEBUG:boto:Params: {} 2024-04-16T19:46:53.133 DEBUG:boto:Token: None 2024-04-16T19:46:53.133 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/obj7 2024-04-16T19:46:53.134 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y6hTnAIC2fXC0OPxm5h4rEaUwQg= 2024-04-16T19:46:53.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y6hTnAIC2fXC0OPxm5h4rEaUwQg='} 2024-04-16T19:46:53.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb7d41bbeadb958f-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.138 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:46:53.138 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:46:53.139 DEBUG:boto:Method: HEAD 2024-04-16T19:46:53.139 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:46:53.139 DEBUG:boto:Data: 2024-04-16T19:46:53.139 DEBUG:boto:Headers: {} 2024-04-16T19:46:53.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.139 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.139 DEBUG:boto:Params: {} 2024-04-16T19:46:53.139 DEBUG:boto:Token: None 2024-04-16T19:46:53.139 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/ 2024-04-16T19:46:53.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U= 2024-04-16T19:46:53.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jBivqXQSOgYMwYfRxdVxHorfM4U=', 'Content-Length': '0'} 2024-04-16T19:46:53.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bd1ff400052ed47-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.142 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:46:53.142 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:46:53.143 DEBUG:boto:Method: PUT 2024-04-16T19:46:53.143 DEBUG:boto:Path: /yrjefh-41/obj8 2024-04-16T19:46:53.143 DEBUG:boto:Data: 2024-04-16T19:46:53.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T19:46:53.143 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:46:53.143 DEBUG:boto:Port: 8000 2024-04-16T19:46:53.143 DEBUG:boto:Params: {} 2024-04-16T19:46:53.143 DEBUG:boto:Token: None 2024-04-16T19:46:53.143 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 16 Apr 2024 19:46:53 GMT /yrjefh-41/obj8 2024-04-16T19:46:53.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P8YyGq0Lp4AhV0O9uFAjoaOwa9s= 2024-04-16T19:46:53.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:46:53 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P8YyGq0Lp4AhV0O9uFAjoaOwa9s='} 2024-04-16T19:46:53.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000890fe68a41939a05-00661ed5ad-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:46:53.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'yrjefh-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:53.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'yrjefh-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:53.149 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket yrjefh-40 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:46:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:53.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'yrjefh-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:53.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'yrjefh-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:46:53.311 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket yrjefh-41 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:46:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:53.460 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-40 2024-04-16T19:46:53.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:53.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:46:53.460 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-40 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:46:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:46:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:46:53.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:53.575+0000 7f6825daf800 1 waiting to reach incremental sync.. 2024-04-16T19:46:58.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:46:58.575+0000 7f6825daf800 1 waiting to reach incremental sync.. 2024-04-16T19:47:03.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:03.578+0000 7f6825daf800 1 bucket sync caught up with source: 2024-04-16T19:47:03.580 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.49.1, 00000000002.29.1, 00000000004.49.1, 00000000002.25.1, 00000000004.54.1, 00000000002.38.1, 00000000002.15.1, 00000000004.76.1, 00000000002.22.1, 00000000002.31.1, 00000000002.17.1] 2024-04-16T19:47:03.580 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.49.1, 00000000002.29.1, 00000000004.49.1, 00000000002.25.1, 00000000004.54.1, 00000000002.38.1, 00000000002.15.1, 00000000004.76.1, 00000000002.22.1, 00000000002.31.1, 00000000002.17.1] 2024-04-16T19:47:03.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:03.578+0000 7f6825daf800 0 bucket checkpoint complete 2024-04-16T19:47:03.605 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-40 2024-04-16T19:47:03.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:03.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-40', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:03.606 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-40 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:03.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:03.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:03.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:03.735+0000 7fc571634800 1 waiting to reach incremental sync.. 2024-04-16T19:47:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:08.737+0000 7fc571634800 1 bucket sync caught up with source: 2024-04-16T19:47:08.739 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.90.1, 00000000002.81.1, 00000000004.150.1, 00000000002.52.1, 00000000004.109.6, 00000000002.68.1, 00000000002.81.1, 00000000004.190.1, 00000000002.130.1, 00000000002.125.1, 00000000002.81.1] 2024-04-16T19:47:08.739 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.90.1, 00000000002.81.1, 00000000004.150.1, 00000000002.52.1, 00000000004.109.6, 00000000002.68.1, 00000000002.81.1, 00000000004.190.1, 00000000002.130.1, 00000000002.125.1, 00000000002.81.1] 2024-04-16T19:47:08.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:08.737+0000 7fc571634800 0 bucket checkpoint complete 2024-04-16T19:47:08.766 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-40 zones={test-zone2, test-zone1} 2024-04-16T19:47:08.766 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.766 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.766 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.766 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:47:08.766 DEBUG:boto:Data: 2024-04-16T19:47:08.766 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.767 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.767 DEBUG:boto:Params: {} 2024-04-16T19:47:08.767 DEBUG:boto:Token: None 2024-04-16T19:47:08.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4VAvAUmoUmHVHQGHEOyOP4BGoHs= 2024-04-16T19:47:08.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4VAvAUmoUmHVHQGHEOyOP4BGoHs=', 'Content-Length': '0'} 2024-04-16T19:47:08.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066bc09d26b633ff8-00661ed5bc-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.770 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.770 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.770 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.770 DEBUG:boto:Path: /yrjefh-40/ 2024-04-16T19:47:08.770 DEBUG:boto:Data: 2024-04-16T19:47:08.770 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.771 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.771 DEBUG:boto:Params: {} 2024-04-16T19:47:08.771 DEBUG:boto:Token: None 2024-04-16T19:47:08.771 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4VAvAUmoUmHVHQGHEOyOP4BGoHs= 2024-04-16T19:47:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4VAvAUmoUmHVHQGHEOyOP4BGoHs=', 'Content-Length': '0'} 2024-04-16T19:47:08.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec83d648d4c710d7-00661ed5bc-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:47:08.774 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.774 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.774 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.774 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.774 DEBUG:boto:Method: GET 2024-04-16T19:47:08.774 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.774 DEBUG:boto:Data: 2024-04-16T19:47:08.774 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.774 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.774 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.774 DEBUG:boto:Params: {} 2024-04-16T19:47:08.775 DEBUG:boto:Token: None 2024-04-16T19:47:08.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/?versions 2024-04-16T19:47:08.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs= 2024-04-16T19:47:08.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs=', 'Content-Length': '0'} 2024-04-16T19:47:08.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002013102086c3ef17-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.778 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:47:08.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:47:08.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:47:08.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:47:08.779 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.779 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.779 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.779 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.779 DEBUG:boto:Method: GET 2024-04-16T19:47:08.779 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.779 DEBUG:boto:Data: 2024-04-16T19:47:08.780 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.780 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.780 DEBUG:boto:Params: {} 2024-04-16T19:47:08.780 DEBUG:boto:Token: None 2024-04-16T19:47:08.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/?versions 2024-04-16T19:47:08.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs= 2024-04-16T19:47:08.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs=', 'Content-Length': '0'} 2024-04-16T19:47:08.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d939c7865bd0efaf-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.782 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:47:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:47:08.784 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.784 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.784 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.784 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.784 DEBUG:boto:Method: GET 2024-04-16T19:47:08.784 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.784 DEBUG:boto:Data: 2024-04-16T19:47:08.784 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.784 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.784 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.784 DEBUG:boto:Params: {} 2024-04-16T19:47:08.784 DEBUG:boto:Token: None 2024-04-16T19:47:08.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/?versions 2024-04-16T19:47:08.785 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs= 2024-04-16T19:47:08.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs=', 'Content-Length': '0'} 2024-04-16T19:47:08.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014dbb20a739fbe1-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.787 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.788 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.788 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.788 DEBUG:boto:path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.788 DEBUG:boto:auth_path=/yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.788 DEBUG:boto:Method: GET 2024-04-16T19:47:08.788 DEBUG:boto:Path: /yrjefh-40/?versions&max-keys=999 2024-04-16T19:47:08.788 DEBUG:boto:Data: 2024-04-16T19:47:08.788 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.788 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.788 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.788 DEBUG:boto:Params: {} 2024-04-16T19:47:08.789 DEBUG:boto:Token: None 2024-04-16T19:47:08.789 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/?versions 2024-04-16T19:47:08.789 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs= 2024-04-16T19:47:08.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vSzRPvmNzmxh5qFZpyfDDDP/bcs=', 'Content-Length': '0'} 2024-04-16T19:47:08.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a3a34cd1e064d0-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.791 DEBUG:boto:b'yrjefh-40999falseobj1nulltrue2024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:08.792 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.792 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.792 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.792 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.792 DEBUG:boto:Method: GET 2024-04-16T19:47:08.792 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.792 DEBUG:boto:Data: 2024-04-16T19:47:08.792 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.792 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.793 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.793 DEBUG:boto:Params: {} 2024-04-16T19:47:08.793 DEBUG:boto:Token: None 2024-04-16T19:47:08.793 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU= 2024-04-16T19:47:08.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU=', 'Content-Length': '0'} 2024-04-16T19:47:08.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9a56e19912c4f9b-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.838 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.838 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.838 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.839 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.839 DEBUG:boto:Method: GET 2024-04-16T19:47:08.839 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.839 DEBUG:boto:Data: 2024-04-16T19:47:08.839 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.839 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.839 DEBUG:boto:Params: {} 2024-04-16T19:47:08.839 DEBUG:boto:Token: None 2024-04-16T19:47:08.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU= 2024-04-16T19:47:08.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU=', 'Content-Length': '0'} 2024-04-16T19:47:08.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002d9beb93f8a110c-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.842 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.842 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.842 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.842 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.842 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.842 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.842 DEBUG:boto:Data: 2024-04-16T19:47:08.842 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.842 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.842 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.842 DEBUG:boto:Params: {} 2024-04-16T19:47:08.843 DEBUG:boto:Token: None 2024-04-16T19:47:08.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1izPBnw0HRQh+B9MqQwTq4pX7CY= 2024-04-16T19:47:08.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1izPBnw0HRQh+B9MqQwTq4pX7CY=', 'Content-Length': '0'} 2024-04-16T19:47:08.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8691d73f8779977-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.845 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.845 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.845 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.845 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.845 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.845 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.845 DEBUG:boto:Data: 2024-04-16T19:47:08.845 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.845 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.846 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.846 DEBUG:boto:Params: {} 2024-04-16T19:47:08.846 DEBUG:boto:Token: None 2024-04-16T19:47:08.846 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1izPBnw0HRQh+B9MqQwTq4pX7CY= 2024-04-16T19:47:08.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1izPBnw0HRQh+B9MqQwTq4pX7CY=', 'Content-Length': '0'} 2024-04-16T19:47:08.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000671397b2d24be311-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:08.848 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.848 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.848 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.848 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.849 DEBUG:boto:Method: GET 2024-04-16T19:47:08.849 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.849 DEBUG:boto:Data: 2024-04-16T19:47:08.849 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.849 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.849 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.849 DEBUG:boto:Params: {} 2024-04-16T19:47:08.849 DEBUG:boto:Token: None 2024-04-16T19:47:08.849 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.849 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU= 2024-04-16T19:47:08.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU=', 'Content-Length': '0'} 2024-04-16T19:47:08.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db64fedd19a0760f-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.852 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.852 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.852 DEBUG:boto:path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.852 DEBUG:boto:auth_path=/yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.852 DEBUG:boto:Method: GET 2024-04-16T19:47:08.852 DEBUG:boto:Path: /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.852 DEBUG:boto:Data: 2024-04-16T19:47:08.852 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.852 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.852 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.852 DEBUG:boto:Params: {} 2024-04-16T19:47:08.853 DEBUG:boto:Token: None 2024-04-16T19:47:08.853 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1?versionId=null 2024-04-16T19:47:08.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU= 2024-04-16T19:47:08.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ht33TFKSNJdJt+XMjgDbfv4IuMU=', 'Content-Length': '0'} 2024-04-16T19:47:08.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051ea1d2b7a9b0bb1-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.855 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.855 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.855 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.855 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.855 DEBUG:boto:Method: GET 2024-04-16T19:47:08.855 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.855 DEBUG:boto:Data: 2024-04-16T19:47:08.855 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.855 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.856 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.856 DEBUG:boto:Params: {} 2024-04-16T19:47:08.856 DEBUG:boto:Token: None 2024-04-16T19:47:08.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM= 2024-04-16T19:47:08.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM=', 'Content-Length': '0'} 2024-04-16T19:47:08.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e282f4c37d1e857d-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.859 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.859 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.859 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.859 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.859 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.859 DEBUG:boto:Method: GET 2024-04-16T19:47:08.859 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.859 DEBUG:boto:Data: 2024-04-16T19:47:08.860 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.860 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.860 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.860 DEBUG:boto:Params: {} 2024-04-16T19:47:08.860 DEBUG:boto:Token: None 2024-04-16T19:47:08.860 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM= 2024-04-16T19:47:08.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM=', 'Content-Length': '0'} 2024-04-16T19:47:08.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ed22c3655981e8-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.862 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.862 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.863 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.863 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.863 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:47:08.863 DEBUG:boto:Data: 2024-04-16T19:47:08.863 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.863 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.863 DEBUG:boto:Params: {} 2024-04-16T19:47:08.863 DEBUG:boto:Token: None 2024-04-16T19:47:08.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1 2024-04-16T19:47:08.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ka8y4pdAi5J5mQTi3vId7JZerJE= 2024-04-16T19:47:08.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ka8y4pdAi5J5mQTi3vId7JZerJE=', 'Content-Length': '0'} 2024-04-16T19:47:08.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000501ee89979f721a6-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.865 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.866 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.866 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.866 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:47:08.866 DEBUG:boto:Data: 2024-04-16T19:47:08.866 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.866 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.866 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.866 DEBUG:boto:Params: {} 2024-04-16T19:47:08.866 DEBUG:boto:Token: None 2024-04-16T19:47:08.866 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1 2024-04-16T19:47:08.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ka8y4pdAi5J5mQTi3vId7JZerJE= 2024-04-16T19:47:08.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ka8y4pdAi5J5mQTi3vId7JZerJE=', 'Content-Length': '0'} 2024-04-16T19:47:08.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000990bdab94e5c8ea6-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:08.869 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.869 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.869 DEBUG:boto:Method: GET 2024-04-16T19:47:08.869 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:47:08.869 DEBUG:boto:Data: 2024-04-16T19:47:08.869 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.869 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.869 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.869 DEBUG:boto:Params: {} 2024-04-16T19:47:08.869 DEBUG:boto:Token: None 2024-04-16T19:47:08.869 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1 2024-04-16T19:47:08.869 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SuUgA9WIvLju+EAxvkmS0q3zFHk= 2024-04-16T19:47:08.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SuUgA9WIvLju+EAxvkmS0q3zFHk=', 'Content-Length': '0'} 2024-04-16T19:47:08.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027c2b47420a0319c-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.914 DEBUG:boto:path=/yrjefh-40/obj1 2024-04-16T19:47:08.914 DEBUG:boto:auth_path=/yrjefh-40/obj1 2024-04-16T19:47:08.915 DEBUG:boto:Method: GET 2024-04-16T19:47:08.915 DEBUG:boto:Path: /yrjefh-40/obj1 2024-04-16T19:47:08.915 DEBUG:boto:Data: 2024-04-16T19:47:08.915 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.915 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.915 DEBUG:boto:Params: {} 2024-04-16T19:47:08.915 DEBUG:boto:Token: None 2024-04-16T19:47:08.915 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj1 2024-04-16T19:47:08.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SuUgA9WIvLju+EAxvkmS0q3zFHk= 2024-04-16T19:47:08.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SuUgA9WIvLju+EAxvkmS0q3zFHk=', 'Content-Length': '0'} 2024-04-16T19:47:08.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e0539f93a2d7d01-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.918 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.918 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.918 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.918 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.918 DEBUG:boto:Method: GET 2024-04-16T19:47:08.918 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.918 DEBUG:boto:Data: 2024-04-16T19:47:08.918 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.918 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.918 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.919 DEBUG:boto:Params: {} 2024-04-16T19:47:08.919 DEBUG:boto:Token: None 2024-04-16T19:47:08.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM= 2024-04-16T19:47:08.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM=', 'Content-Length': '0'} 2024-04-16T19:47:08.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ee8b1c32252b81-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.921 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.922 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:08.922 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:08.922 DEBUG:boto:path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.922 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.922 DEBUG:boto:Method: GET 2024-04-16T19:47:08.922 DEBUG:boto:Path: /yrjefh-40/?prefix=obj1 2024-04-16T19:47:08.922 DEBUG:boto:Data: 2024-04-16T19:47:08.922 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.923 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.923 DEBUG:boto:Params: {} 2024-04-16T19:47:08.923 DEBUG:boto:Token: None 2024-04-16T19:47:08.923 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/ 2024-04-16T19:47:08.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM= 2024-04-16T19:47:08.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:duHLtd3Cek5jDae5Zowk82wbDJM=', 'Content-Length': '0'} 2024-04-16T19:47:08.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a9337b0c40f7223-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.926 DEBUG:boto:b'yrjefh-40obj11000falseobj12024-04-16T19:46:34.583Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:08.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:08.926 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:08.926 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:08.926 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.926 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.926 DEBUG:boto:Method: GET 2024-04-16T19:47:08.926 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.927 DEBUG:boto:Data: 2024-04-16T19:47:08.927 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.927 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.927 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.927 DEBUG:boto:Params: {} 2024-04-16T19:47:08.927 DEBUG:boto:Token: None 2024-04-16T19:47:08.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0= 2024-04-16T19:47:08.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0=', 'Content-Length': '0'} 2024-04-16T19:47:08.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000443cd23359262854-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.970 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:08.970 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:08.970 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.971 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.971 DEBUG:boto:Method: GET 2024-04-16T19:47:08.971 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.971 DEBUG:boto:Data: 2024-04-16T19:47:08.971 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.971 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.971 DEBUG:boto:Params: {} 2024-04-16T19:47:08.971 DEBUG:boto:Token: None 2024-04-16T19:47:08.971 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0= 2024-04-16T19:47:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0=', 'Content-Length': '0'} 2024-04-16T19:47:08.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000273ca0ba5dc64f4e-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.974 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:08.974 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:08.974 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.974 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.974 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.974 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.974 DEBUG:boto:Data: 2024-04-16T19:47:08.974 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.975 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.975 DEBUG:boto:Params: {} 2024-04-16T19:47:08.975 DEBUG:boto:Token: None 2024-04-16T19:47:08.975 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gGcvDvSnf9T29PiFONiH+il3ToE= 2024-04-16T19:47:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gGcvDvSnf9T29PiFONiH+il3ToE=', 'Content-Length': '0'} 2024-04-16T19:47:08.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086dbf7180b090acf-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.977 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:08.977 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:08.977 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.977 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.977 DEBUG:boto:Method: HEAD 2024-04-16T19:47:08.977 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.977 DEBUG:boto:Data: 2024-04-16T19:47:08.977 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.978 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:08.978 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.978 DEBUG:boto:Params: {} 2024-04-16T19:47:08.978 DEBUG:boto:Token: None 2024-04-16T19:47:08.978 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gGcvDvSnf9T29PiFONiH+il3ToE= 2024-04-16T19:47:08.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gGcvDvSnf9T29PiFONiH+il3ToE=', 'Content-Length': '0'} 2024-04-16T19:47:08.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096959ae31583549e-00661ed5bc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:08.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:08.980 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:08.980 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:08.980 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.980 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.980 DEBUG:boto:Method: GET 2024-04-16T19:47:08.981 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.981 DEBUG:boto:Data: 2024-04-16T19:47:08.981 DEBUG:boto:Headers: {} 2024-04-16T19:47:08.981 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:08.981 DEBUG:boto:Port: 8000 2024-04-16T19:47:08.981 DEBUG:boto:Params: {} 2024-04-16T19:47:08.981 DEBUG:boto:Token: None 2024-04-16T19:47:08.981 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:08 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:08.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0= 2024-04-16T19:47:08.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rjrAy/66rVgFGOCjauiR+JkG1W0=', 'Content-Length': '0'} 2024-04-16T19:47:08.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fb3a8d87956f06e-00661ed5bc-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.026 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:09.026 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:09.027 DEBUG:boto:path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:09.027 DEBUG:boto:auth_path=/yrjefh-40/obj2?versionId=null 2024-04-16T19:47:09.027 DEBUG:boto:Method: GET 2024-04-16T19:47:09.027 DEBUG:boto:Path: /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:09.027 DEBUG:boto:Data: 2024-04-16T19:47:09.027 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.027 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.027 DEBUG:boto:Params: {} 2024-04-16T19:47:09.027 DEBUG:boto:Token: None 2024-04-16T19:47:09.027 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj2?versionId=null 2024-04-16T19:47:09.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QaZclV2TvAOQ2u4H/zHTZFkXIA8= 2024-04-16T19:47:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QaZclV2TvAOQ2u4H/zHTZFkXIA8=', 'Content-Length': '0'} 2024-04-16T19:47:09.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023f389c303e5438a-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.030 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.030 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.030 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.030 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.030 DEBUG:boto:Method: GET 2024-04-16T19:47:09.030 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.030 DEBUG:boto:Data: 2024-04-16T19:47:09.030 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.030 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.031 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.031 DEBUG:boto:Params: {} 2024-04-16T19:47:09.031 DEBUG:boto:Token: None 2024-04-16T19:47:09.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f41bbc7cc185395-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.078 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.079 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.079 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.079 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.079 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.079 DEBUG:boto:Method: GET 2024-04-16T19:47:09.079 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.079 DEBUG:boto:Data: 2024-04-16T19:47:09.079 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.079 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.079 DEBUG:boto:Params: {} 2024-04-16T19:47:09.079 DEBUG:boto:Token: None 2024-04-16T19:47:09.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe5fa12077440d1-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.082 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.083 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:09.083 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:09.083 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.083 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:47:09.083 DEBUG:boto:Data: 2024-04-16T19:47:09.083 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.083 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.083 DEBUG:boto:Params: {} 2024-04-16T19:47:09.083 DEBUG:boto:Token: None 2024-04-16T19:47:09.084 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj2 2024-04-16T19:47:09.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B8hzSwmZB0vJxsnxSj/DRDALVek= 2024-04-16T19:47:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B8hzSwmZB0vJxsnxSj/DRDALVek=', 'Content-Length': '0'} 2024-04-16T19:47:09.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e45e4093be4de184-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.086 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:09.086 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:09.086 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.086 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:47:09.086 DEBUG:boto:Data: 2024-04-16T19:47:09.086 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.086 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.086 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.086 DEBUG:boto:Params: {} 2024-04-16T19:47:09.087 DEBUG:boto:Token: None 2024-04-16T19:47:09.087 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj2 2024-04-16T19:47:09.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B8hzSwmZB0vJxsnxSj/DRDALVek= 2024-04-16T19:47:09.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B8hzSwmZB0vJxsnxSj/DRDALVek=', 'Content-Length': '0'} 2024-04-16T19:47:09.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010f670a702e25852-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:09.089 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:09.089 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:09.089 DEBUG:boto:Method: GET 2024-04-16T19:47:09.089 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:47:09.089 DEBUG:boto:Data: 2024-04-16T19:47:09.089 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.089 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.089 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.090 DEBUG:boto:Params: {} 2024-04-16T19:47:09.090 DEBUG:boto:Token: None 2024-04-16T19:47:09.090 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj2 2024-04-16T19:47:09.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:51hfma9OunFzAI0Eknxzemb94pw= 2024-04-16T19:47:09.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:51hfma9OunFzAI0Eknxzemb94pw=', 'Content-Length': '0'} 2024-04-16T19:47:09.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccefd67178137402-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.134 DEBUG:boto:path=/yrjefh-40/obj2 2024-04-16T19:47:09.134 DEBUG:boto:auth_path=/yrjefh-40/obj2 2024-04-16T19:47:09.134 DEBUG:boto:Method: GET 2024-04-16T19:47:09.135 DEBUG:boto:Path: /yrjefh-40/obj2 2024-04-16T19:47:09.135 DEBUG:boto:Data: 2024-04-16T19:47:09.135 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.135 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.135 DEBUG:boto:Params: {} 2024-04-16T19:47:09.135 DEBUG:boto:Token: None 2024-04-16T19:47:09.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj2 2024-04-16T19:47:09.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:51hfma9OunFzAI0Eknxzemb94pw= 2024-04-16T19:47:09.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:51hfma9OunFzAI0Eknxzemb94pw=', 'Content-Length': '0'} 2024-04-16T19:47:09.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005836ed2b1b3ef082-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.137 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.138 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.138 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.138 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.138 DEBUG:boto:Method: GET 2024-04-16T19:47:09.138 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.138 DEBUG:boto:Data: 2024-04-16T19:47:09.138 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.138 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.138 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.138 DEBUG:boto:Params: {} 2024-04-16T19:47:09.138 DEBUG:boto:Token: None 2024-04-16T19:47:09.139 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c223acfddfd91166-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.141 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.141 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.142 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.142 DEBUG:boto:path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.142 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.142 DEBUG:boto:Method: GET 2024-04-16T19:47:09.142 DEBUG:boto:Path: /yrjefh-40/?prefix=obj2 2024-04-16T19:47:09.142 DEBUG:boto:Data: 2024-04-16T19:47:09.142 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.142 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.142 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.142 DEBUG:boto:Params: {} 2024-04-16T19:47:09.142 DEBUG:boto:Token: None 2024-04-16T19:47:09.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa07fb3e5bfc0c6-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.145 DEBUG:boto:b'yrjefh-40obj21000falseobj22024-04-16T19:46:34.632Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:09.146 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.146 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.146 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.146 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.146 DEBUG:boto:Method: GET 2024-04-16T19:47:09.146 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.146 DEBUG:boto:Data: 2024-04-16T19:47:09.146 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.146 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.146 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.147 DEBUG:boto:Params: {} 2024-04-16T19:47:09.147 DEBUG:boto:Token: None 2024-04-16T19:47:09.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM= 2024-04-16T19:47:09.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM=', 'Content-Length': '0'} 2024-04-16T19:47:09.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8b91027b8c6404b-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.190 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.190 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.190 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.191 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.191 DEBUG:boto:Method: GET 2024-04-16T19:47:09.191 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.191 DEBUG:boto:Data: 2024-04-16T19:47:09.191 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.191 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.191 DEBUG:boto:Params: {} 2024-04-16T19:47:09.191 DEBUG:boto:Token: None 2024-04-16T19:47:09.191 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM= 2024-04-16T19:47:09.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM=', 'Content-Length': '0'} 2024-04-16T19:47:09.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000785755857a03ccc5-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.193 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.194 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.194 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.194 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.194 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.194 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.194 DEBUG:boto:Data: 2024-04-16T19:47:09.194 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.194 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.194 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.194 DEBUG:boto:Params: {} 2024-04-16T19:47:09.194 DEBUG:boto:Token: None 2024-04-16T19:47:09.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fCX2OVe4ae0mKxqzn+bBrG73c9c= 2024-04-16T19:47:09.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fCX2OVe4ae0mKxqzn+bBrG73c9c=', 'Content-Length': '0'} 2024-04-16T19:47:09.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c8c74009ff87769-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.197 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.197 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.197 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.197 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.197 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.197 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.197 DEBUG:boto:Data: 2024-04-16T19:47:09.197 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.198 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.198 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.198 DEBUG:boto:Params: {} 2024-04-16T19:47:09.198 DEBUG:boto:Token: None 2024-04-16T19:47:09.198 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.198 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fCX2OVe4ae0mKxqzn+bBrG73c9c= 2024-04-16T19:47:09.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fCX2OVe4ae0mKxqzn+bBrG73c9c=', 'Content-Length': '0'} 2024-04-16T19:47:09.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cf256a5e0666a83-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:09.200 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.200 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.200 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.200 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.200 DEBUG:boto:Method: GET 2024-04-16T19:47:09.200 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.201 DEBUG:boto:Data: 2024-04-16T19:47:09.201 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.201 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.201 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.201 DEBUG:boto:Params: {} 2024-04-16T19:47:09.201 DEBUG:boto:Token: None 2024-04-16T19:47:09.201 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.201 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM= 2024-04-16T19:47:09.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM=', 'Content-Length': '0'} 2024-04-16T19:47:09.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a81f437298a61fa-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.246 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.246 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.246 DEBUG:boto:path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.247 DEBUG:boto:auth_path=/yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.247 DEBUG:boto:Method: GET 2024-04-16T19:47:09.247 DEBUG:boto:Path: /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.247 DEBUG:boto:Data: 2024-04-16T19:47:09.247 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.247 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.247 DEBUG:boto:Params: {} 2024-04-16T19:47:09.247 DEBUG:boto:Token: None 2024-04-16T19:47:09.247 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3?versionId=null 2024-04-16T19:47:09.247 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM= 2024-04-16T19:47:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xNNLOhpSAUxGc+3c7eAJ9MCBqxM=', 'Content-Length': '0'} 2024-04-16T19:47:09.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb1a2f6b1bddf7b-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.250 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.250 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.250 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.250 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.250 DEBUG:boto:Method: GET 2024-04-16T19:47:09.250 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.250 DEBUG:boto:Data: 2024-04-16T19:47:09.250 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.250 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.250 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.250 DEBUG:boto:Params: {} 2024-04-16T19:47:09.251 DEBUG:boto:Token: None 2024-04-16T19:47:09.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069bbf862c071e8ab-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.253 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.253 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.254 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.254 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.254 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.254 DEBUG:boto:Method: GET 2024-04-16T19:47:09.254 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.254 DEBUG:boto:Data: 2024-04-16T19:47:09.254 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.254 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.254 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.254 DEBUG:boto:Params: {} 2024-04-16T19:47:09.254 DEBUG:boto:Token: None 2024-04-16T19:47:09.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca3822252dc9aae-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.257 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.257 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.258 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.258 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.258 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:47:09.258 DEBUG:boto:Data: 2024-04-16T19:47:09.258 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.258 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.258 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.258 DEBUG:boto:Params: {} 2024-04-16T19:47:09.258 DEBUG:boto:Token: None 2024-04-16T19:47:09.258 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3 2024-04-16T19:47:09.258 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EkgZvj7dZXLbwDIxbmysQ/4PvLs= 2024-04-16T19:47:09.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EkgZvj7dZXLbwDIxbmysQ/4PvLs=', 'Content-Length': '0'} 2024-04-16T19:47:09.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a6f0e33185fe928-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.261 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.261 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.261 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.261 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:47:09.261 DEBUG:boto:Data: 2024-04-16T19:47:09.261 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.261 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.261 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.261 DEBUG:boto:Params: {} 2024-04-16T19:47:09.261 DEBUG:boto:Token: None 2024-04-16T19:47:09.261 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3 2024-04-16T19:47:09.262 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EkgZvj7dZXLbwDIxbmysQ/4PvLs= 2024-04-16T19:47:09.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EkgZvj7dZXLbwDIxbmysQ/4PvLs=', 'Content-Length': '0'} 2024-04-16T19:47:09.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026d58746fdf31aa7-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:09.264 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.264 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.264 DEBUG:boto:Method: GET 2024-04-16T19:47:09.264 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:47:09.264 DEBUG:boto:Data: 2024-04-16T19:47:09.264 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.264 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.264 DEBUG:boto:Params: {} 2024-04-16T19:47:09.264 DEBUG:boto:Token: None 2024-04-16T19:47:09.265 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3 2024-04-16T19:47:09.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zgK6v2f4WJm9BNBAON2ttLDot3A= 2024-04-16T19:47:09.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zgK6v2f4WJm9BNBAON2ttLDot3A=', 'Content-Length': '0'} 2024-04-16T19:47:09.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000446b70d0c1ca9a8b-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.310 DEBUG:boto:path=/yrjefh-40/obj3 2024-04-16T19:47:09.310 DEBUG:boto:auth_path=/yrjefh-40/obj3 2024-04-16T19:47:09.310 DEBUG:boto:Method: GET 2024-04-16T19:47:09.311 DEBUG:boto:Path: /yrjefh-40/obj3 2024-04-16T19:47:09.311 DEBUG:boto:Data: 2024-04-16T19:47:09.311 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.311 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.311 DEBUG:boto:Params: {} 2024-04-16T19:47:09.311 DEBUG:boto:Token: None 2024-04-16T19:47:09.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj3 2024-04-16T19:47:09.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zgK6v2f4WJm9BNBAON2ttLDot3A= 2024-04-16T19:47:09.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zgK6v2f4WJm9BNBAON2ttLDot3A=', 'Content-Length': '0'} 2024-04-16T19:47:09.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a759472b8f3c8deb-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.313 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.314 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.314 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.314 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.314 DEBUG:boto:Method: GET 2024-04-16T19:47:09.314 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.314 DEBUG:boto:Data: 2024-04-16T19:47:09.314 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.314 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.314 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.314 DEBUG:boto:Params: {} 2024-04-16T19:47:09.314 DEBUG:boto:Token: None 2024-04-16T19:47:09.314 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb4d61e55daf331-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.317 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.317 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.318 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.318 DEBUG:boto:path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.318 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.318 DEBUG:boto:Method: GET 2024-04-16T19:47:09.318 DEBUG:boto:Path: /yrjefh-40/?prefix=obj3 2024-04-16T19:47:09.318 DEBUG:boto:Data: 2024-04-16T19:47:09.318 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.318 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.318 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.318 DEBUG:boto:Params: {} 2024-04-16T19:47:09.318 DEBUG:boto:Token: None 2024-04-16T19:47:09.318 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167c45da97ab595f-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.321 DEBUG:boto:b'yrjefh-40obj31000falseobj32024-04-16T19:46:34.645Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:09.322 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.322 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.322 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.322 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.322 DEBUG:boto:Method: GET 2024-04-16T19:47:09.322 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.322 DEBUG:boto:Data: 2024-04-16T19:47:09.322 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.322 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.322 DEBUG:boto:Params: {} 2024-04-16T19:47:09.323 DEBUG:boto:Token: None 2024-04-16T19:47:09.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI= 2024-04-16T19:47:09.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI=', 'Content-Length': '0'} 2024-04-16T19:47:09.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000316d4cf037f7c35e-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.366 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.366 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.366 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.367 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.367 DEBUG:boto:Method: GET 2024-04-16T19:47:09.367 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.367 DEBUG:boto:Data: 2024-04-16T19:47:09.367 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.367 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.367 DEBUG:boto:Params: {} 2024-04-16T19:47:09.367 DEBUG:boto:Token: None 2024-04-16T19:47:09.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI= 2024-04-16T19:47:09.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI=', 'Content-Length': '0'} 2024-04-16T19:47:09.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000881e4b17d9311274-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.369 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.370 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.370 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.370 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.370 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.370 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.370 DEBUG:boto:Data: 2024-04-16T19:47:09.370 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.370 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.370 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.370 DEBUG:boto:Params: {} 2024-04-16T19:47:09.370 DEBUG:boto:Token: None 2024-04-16T19:47:09.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FsXwSesQ8Hkjdt3/oIRI/e6nh20= 2024-04-16T19:47:09.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FsXwSesQ8Hkjdt3/oIRI/e6nh20=', 'Content-Length': '0'} 2024-04-16T19:47:09.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b1800145cfb4356-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.373 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.373 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.373 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.373 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.373 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.373 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.373 DEBUG:boto:Data: 2024-04-16T19:47:09.374 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.374 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.374 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.374 DEBUG:boto:Params: {} 2024-04-16T19:47:09.374 DEBUG:boto:Token: None 2024-04-16T19:47:09.374 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.374 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FsXwSesQ8Hkjdt3/oIRI/e6nh20= 2024-04-16T19:47:09.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FsXwSesQ8Hkjdt3/oIRI/e6nh20=', 'Content-Length': '0'} 2024-04-16T19:47:09.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eca51f9b66eb7a52-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:09.376 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.376 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.376 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.377 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.377 DEBUG:boto:Method: GET 2024-04-16T19:47:09.377 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.377 DEBUG:boto:Data: 2024-04-16T19:47:09.377 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.377 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.377 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.377 DEBUG:boto:Params: {} 2024-04-16T19:47:09.377 DEBUG:boto:Token: None 2024-04-16T19:47:09.377 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI= 2024-04-16T19:47:09.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI=', 'Content-Length': '0'} 2024-04-16T19:47:09.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050adfdf25b3fd8f3-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.422 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.422 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.423 DEBUG:boto:path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.423 DEBUG:boto:auth_path=/yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.423 DEBUG:boto:Method: GET 2024-04-16T19:47:09.423 DEBUG:boto:Path: /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.423 DEBUG:boto:Data: 2024-04-16T19:47:09.423 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.423 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.423 DEBUG:boto:Params: {} 2024-04-16T19:47:09.423 DEBUG:boto:Token: None 2024-04-16T19:47:09.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4?versionId=null 2024-04-16T19:47:09.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI= 2024-04-16T19:47:09.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xID1dUSYdRHK+3yG7FkuRtRi3VI=', 'Content-Length': '0'} 2024-04-16T19:47:09.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002db72712eff0b6ae-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.426 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.426 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.426 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.426 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.426 DEBUG:boto:Method: GET 2024-04-16T19:47:09.427 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.427 DEBUG:boto:Data: 2024-04-16T19:47:09.427 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.427 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.427 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.427 DEBUG:boto:Params: {} 2024-04-16T19:47:09.427 DEBUG:boto:Token: None 2024-04-16T19:47:09.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec05a2051ba16ae-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.430 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.430 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.430 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.431 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.431 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.431 DEBUG:boto:Method: GET 2024-04-16T19:47:09.431 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.431 DEBUG:boto:Data: 2024-04-16T19:47:09.431 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.431 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.431 DEBUG:boto:Params: {} 2024-04-16T19:47:09.431 DEBUG:boto:Token: None 2024-04-16T19:47:09.431 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1488f8fe0a7849-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.434 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.434 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.435 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.435 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.435 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:47:09.435 DEBUG:boto:Data: 2024-04-16T19:47:09.435 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.435 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.435 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.435 DEBUG:boto:Params: {} 2024-04-16T19:47:09.435 DEBUG:boto:Token: None 2024-04-16T19:47:09.435 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4 2024-04-16T19:47:09.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B+7c5Ud2sAMoM3Ze+DswmiFFL1c= 2024-04-16T19:47:09.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B+7c5Ud2sAMoM3Ze+DswmiFFL1c=', 'Content-Length': '0'} 2024-04-16T19:47:09.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b33130e5784057d5-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.438 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.438 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.438 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.438 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:47:09.438 DEBUG:boto:Data: 2024-04-16T19:47:09.438 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.438 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.438 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.438 DEBUG:boto:Params: {} 2024-04-16T19:47:09.438 DEBUG:boto:Token: None 2024-04-16T19:47:09.439 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4 2024-04-16T19:47:09.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B+7c5Ud2sAMoM3Ze+DswmiFFL1c= 2024-04-16T19:47:09.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B+7c5Ud2sAMoM3Ze+DswmiFFL1c=', 'Content-Length': '0'} 2024-04-16T19:47:09.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cab613ce44c783f6-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:09.441 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.441 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.441 DEBUG:boto:Method: GET 2024-04-16T19:47:09.441 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:47:09.441 DEBUG:boto:Data: 2024-04-16T19:47:09.441 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.441 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.442 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.442 DEBUG:boto:Params: {} 2024-04-16T19:47:09.442 DEBUG:boto:Token: None 2024-04-16T19:47:09.442 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4 2024-04-16T19:47:09.442 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RjkK9jimfBycnZ1GSFsFUbk3gag= 2024-04-16T19:47:09.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RjkK9jimfBycnZ1GSFsFUbk3gag=', 'Content-Length': '0'} 2024-04-16T19:47:09.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:36 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e33e10d8729b980-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.486 DEBUG:boto:path=/yrjefh-40/obj4 2024-04-16T19:47:09.486 DEBUG:boto:auth_path=/yrjefh-40/obj4 2024-04-16T19:47:09.487 DEBUG:boto:Method: GET 2024-04-16T19:47:09.487 DEBUG:boto:Path: /yrjefh-40/obj4 2024-04-16T19:47:09.487 DEBUG:boto:Data: 2024-04-16T19:47:09.487 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.487 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.487 DEBUG:boto:Params: {} 2024-04-16T19:47:09.487 DEBUG:boto:Token: None 2024-04-16T19:47:09.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj4 2024-04-16T19:47:09.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RjkK9jimfBycnZ1GSFsFUbk3gag= 2024-04-16T19:47:09.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RjkK9jimfBycnZ1GSFsFUbk3gag=', 'Content-Length': '0'} 2024-04-16T19:47:09.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d79eddcc9f7ed4d-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.490 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.490 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.490 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.490 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.490 DEBUG:boto:Method: GET 2024-04-16T19:47:09.490 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.490 DEBUG:boto:Data: 2024-04-16T19:47:09.490 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.490 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.491 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.491 DEBUG:boto:Params: {} 2024-04-16T19:47:09.491 DEBUG:boto:Token: None 2024-04-16T19:47:09.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d1e1a3cd25ef96-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.493 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.494 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.494 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.494 DEBUG:boto:path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.494 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.494 DEBUG:boto:Method: GET 2024-04-16T19:47:09.494 DEBUG:boto:Path: /yrjefh-40/?prefix=obj4 2024-04-16T19:47:09.494 DEBUG:boto:Data: 2024-04-16T19:47:09.494 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.494 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.495 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.495 DEBUG:boto:Params: {} 2024-04-16T19:47:09.495 DEBUG:boto:Token: None 2024-04-16T19:47:09.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004daf4f08aa1f883c-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.498 DEBUG:boto:b'yrjefh-40obj41000falseobj42024-04-16T19:46:34.692Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:09.498 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.498 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.498 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.498 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.498 DEBUG:boto:Method: GET 2024-04-16T19:47:09.499 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.499 DEBUG:boto:Data: 2024-04-16T19:47:09.499 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.499 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.499 DEBUG:boto:Params: {} 2024-04-16T19:47:09.499 DEBUG:boto:Token: None 2024-04-16T19:47:09.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk= 2024-04-16T19:47:09.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk=', 'Content-Length': '0'} 2024-04-16T19:47:09.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3ff5c3dd4077bce-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.542 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.542 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.543 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.543 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.543 DEBUG:boto:Method: GET 2024-04-16T19:47:09.543 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.543 DEBUG:boto:Data: 2024-04-16T19:47:09.543 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.543 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.543 DEBUG:boto:Params: {} 2024-04-16T19:47:09.543 DEBUG:boto:Token: None 2024-04-16T19:47:09.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk= 2024-04-16T19:47:09.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk=', 'Content-Length': '0'} 2024-04-16T19:47:09.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cf7180450b79e29-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.546 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.546 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.546 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.546 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.546 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.547 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.547 DEBUG:boto:Data: 2024-04-16T19:47:09.547 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.547 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.547 DEBUG:boto:Params: {} 2024-04-16T19:47:09.547 DEBUG:boto:Token: None 2024-04-16T19:47:09.547 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8hFzlurdBB8LU7Zw9W19X/NLZjs= 2024-04-16T19:47:09.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8hFzlurdBB8LU7Zw9W19X/NLZjs=', 'Content-Length': '0'} 2024-04-16T19:47:09.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef9d10a6a30869c-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.549 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.550 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.550 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.550 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.550 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.550 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.550 DEBUG:boto:Data: 2024-04-16T19:47:09.550 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.550 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.550 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.550 DEBUG:boto:Params: {} 2024-04-16T19:47:09.550 DEBUG:boto:Token: None 2024-04-16T19:47:09.551 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8hFzlurdBB8LU7Zw9W19X/NLZjs= 2024-04-16T19:47:09.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8hFzlurdBB8LU7Zw9W19X/NLZjs=', 'Content-Length': '0'} 2024-04-16T19:47:09.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002319c07452eda4d4-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:09.553 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.553 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.553 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.553 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.553 DEBUG:boto:Method: GET 2024-04-16T19:47:09.553 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.553 DEBUG:boto:Data: 2024-04-16T19:47:09.553 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.553 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.553 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.554 DEBUG:boto:Params: {} 2024-04-16T19:47:09.554 DEBUG:boto:Token: None 2024-04-16T19:47:09.554 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk= 2024-04-16T19:47:09.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk=', 'Content-Length': '0'} 2024-04-16T19:47:09.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000350a529b9d43f8b-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.598 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.598 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.599 DEBUG:boto:path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.599 DEBUG:boto:auth_path=/yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.599 DEBUG:boto:Method: GET 2024-04-16T19:47:09.599 DEBUG:boto:Path: /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.599 DEBUG:boto:Data: 2024-04-16T19:47:09.599 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.599 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.599 DEBUG:boto:Params: {} 2024-04-16T19:47:09.599 DEBUG:boto:Token: None 2024-04-16T19:47:09.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5?versionId=null 2024-04-16T19:47:09.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk= 2024-04-16T19:47:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7djSsPQmwG6FEOwQJ6WMN5apPnk=', 'Content-Length': '0'} 2024-04-16T19:47:09.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7a32164af4171d0-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.602 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.602 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.602 DEBUG:boto:path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.602 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.602 DEBUG:boto:Method: GET 2024-04-16T19:47:09.602 DEBUG:boto:Path: /yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.602 DEBUG:boto:Data: 2024-04-16T19:47:09.602 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.603 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.603 DEBUG:boto:Params: {} 2024-04-16T19:47:09.603 DEBUG:boto:Token: None 2024-04-16T19:47:09.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc1aaed0874bc71-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.606 DEBUG:boto:b'yrjefh-40obj51000falseobj52024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.606 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.606 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.606 DEBUG:boto:path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.606 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.606 DEBUG:boto:Method: GET 2024-04-16T19:47:09.606 DEBUG:boto:Path: /yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.607 DEBUG:boto:Data: 2024-04-16T19:47:09.607 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.607 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.607 DEBUG:boto:Params: {} 2024-04-16T19:47:09.607 DEBUG:boto:Token: None 2024-04-16T19:47:09.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b7efb4bad5a5c5-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.610 DEBUG:boto:b'yrjefh-40obj51000falseobj52024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.610 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.610 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.610 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.610 DEBUG:boto:Path: /yrjefh-40/obj5 2024-04-16T19:47:09.610 DEBUG:boto:Data: 2024-04-16T19:47:09.610 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.611 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.611 DEBUG:boto:Params: {} 2024-04-16T19:47:09.611 DEBUG:boto:Token: None 2024-04-16T19:47:09.611 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5 2024-04-16T19:47:09.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ov7PD0H0blLeyM4iAOJZgPDrMfU= 2024-04-16T19:47:09.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ov7PD0H0blLeyM4iAOJZgPDrMfU=', 'Content-Length': '0'} 2024-04-16T19:47:09.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ba2c8c4c9cb35f1-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.613 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.613 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.613 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.613 DEBUG:boto:Path: /yrjefh-40/obj5 2024-04-16T19:47:09.613 DEBUG:boto:Data: 2024-04-16T19:47:09.614 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.614 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.614 DEBUG:boto:Params: {} 2024-04-16T19:47:09.614 DEBUG:boto:Token: None 2024-04-16T19:47:09.614 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5 2024-04-16T19:47:09.614 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ov7PD0H0blLeyM4iAOJZgPDrMfU= 2024-04-16T19:47:09.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ov7PD0H0blLeyM4iAOJZgPDrMfU=', 'Content-Length': '0'} 2024-04-16T19:47:09.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6599f719281f38a-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:09.616 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.616 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.617 DEBUG:boto:Method: GET 2024-04-16T19:47:09.617 DEBUG:boto:Path: /yrjefh-40/obj5 2024-04-16T19:47:09.617 DEBUG:boto:Data: 2024-04-16T19:47:09.617 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.617 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.617 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.617 DEBUG:boto:Params: {} 2024-04-16T19:47:09.617 DEBUG:boto:Token: None 2024-04-16T19:47:09.617 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5 2024-04-16T19:47:09.617 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CWCifBCN35H+/zW7jJx7Q8pWMmk= 2024-04-16T19:47:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CWCifBCN35H+/zW7jJx7Q8pWMmk=', 'Content-Length': '0'} 2024-04-16T19:47:09.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bfccb3513109354-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.662 DEBUG:boto:path=/yrjefh-40/obj5 2024-04-16T19:47:09.662 DEBUG:boto:auth_path=/yrjefh-40/obj5 2024-04-16T19:47:09.663 DEBUG:boto:Method: GET 2024-04-16T19:47:09.663 DEBUG:boto:Path: /yrjefh-40/obj5 2024-04-16T19:47:09.663 DEBUG:boto:Data: 2024-04-16T19:47:09.663 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.663 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.663 DEBUG:boto:Params: {} 2024-04-16T19:47:09.663 DEBUG:boto:Token: None 2024-04-16T19:47:09.663 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj5 2024-04-16T19:47:09.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CWCifBCN35H+/zW7jJx7Q8pWMmk= 2024-04-16T19:47:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CWCifBCN35H+/zW7jJx7Q8pWMmk=', 'Content-Length': '0'} 2024-04-16T19:47:09.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055a6e6edaa6be940-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.665 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.666 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.666 DEBUG:boto:path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.666 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.666 DEBUG:boto:Method: GET 2024-04-16T19:47:09.666 DEBUG:boto:Path: /yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.666 DEBUG:boto:Data: 2024-04-16T19:47:09.666 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.666 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.666 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.666 DEBUG:boto:Params: {} 2024-04-16T19:47:09.666 DEBUG:boto:Token: None 2024-04-16T19:47:09.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18bf75f520ed7f2-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.669 DEBUG:boto:b'yrjefh-40obj51000falseobj52024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.670 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.670 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.670 DEBUG:boto:path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.670 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.670 DEBUG:boto:Method: GET 2024-04-16T19:47:09.670 DEBUG:boto:Path: /yrjefh-40/?prefix=obj5 2024-04-16T19:47:09.670 DEBUG:boto:Data: 2024-04-16T19:47:09.670 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.670 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.670 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.670 DEBUG:boto:Params: {} 2024-04-16T19:47:09.671 DEBUG:boto:Token: None 2024-04-16T19:47:09.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c1cb098aeaa6a3f-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.673 DEBUG:boto:b'yrjefh-40obj51000falseobj52024-04-16T19:46:52.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:09.674 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.674 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.674 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.674 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.674 DEBUG:boto:Method: GET 2024-04-16T19:47:09.674 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.674 DEBUG:boto:Data: 2024-04-16T19:47:09.674 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.674 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.675 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.675 DEBUG:boto:Params: {} 2024-04-16T19:47:09.675 DEBUG:boto:Token: None 2024-04-16T19:47:09.675 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw= 2024-04-16T19:47:09.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw=', 'Content-Length': '0'} 2024-04-16T19:47:09.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df0a134920cba504-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.718 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.718 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.718 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.719 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.719 DEBUG:boto:Method: GET 2024-04-16T19:47:09.719 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.719 DEBUG:boto:Data: 2024-04-16T19:47:09.719 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.719 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.719 DEBUG:boto:Params: {} 2024-04-16T19:47:09.719 DEBUG:boto:Token: None 2024-04-16T19:47:09.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw= 2024-04-16T19:47:09.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw=', 'Content-Length': '0'} 2024-04-16T19:47:09.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b747d69cbd992992-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.723 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.723 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.723 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.723 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.723 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.723 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.723 DEBUG:boto:Data: 2024-04-16T19:47:09.724 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.724 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.724 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.724 DEBUG:boto:Params: {} 2024-04-16T19:47:09.724 DEBUG:boto:Token: None 2024-04-16T19:47:09.724 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kXJg37y8Oe7LSCPNCZr5TdnVHTI= 2024-04-16T19:47:09.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kXJg37y8Oe7LSCPNCZr5TdnVHTI=', 'Content-Length': '0'} 2024-04-16T19:47:09.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d526459a32c7656-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.726 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.727 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.727 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.727 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.727 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.727 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.727 DEBUG:boto:Data: 2024-04-16T19:47:09.727 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.727 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.727 DEBUG:boto:Params: {} 2024-04-16T19:47:09.727 DEBUG:boto:Token: None 2024-04-16T19:47:09.728 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kXJg37y8Oe7LSCPNCZr5TdnVHTI= 2024-04-16T19:47:09.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kXJg37y8Oe7LSCPNCZr5TdnVHTI=', 'Content-Length': '0'} 2024-04-16T19:47:09.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba13c369e6b886b2-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:09.730 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.730 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.730 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.730 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.731 DEBUG:boto:Method: GET 2024-04-16T19:47:09.731 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.731 DEBUG:boto:Data: 2024-04-16T19:47:09.731 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.731 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.731 DEBUG:boto:Params: {} 2024-04-16T19:47:09.731 DEBUG:boto:Token: None 2024-04-16T19:47:09.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw= 2024-04-16T19:47:09.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw=', 'Content-Length': '0'} 2024-04-16T19:47:09.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e623f0586a648148-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.774 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.774 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.775 DEBUG:boto:path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.775 DEBUG:boto:auth_path=/yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.775 DEBUG:boto:Method: GET 2024-04-16T19:47:09.775 DEBUG:boto:Path: /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.775 DEBUG:boto:Data: 2024-04-16T19:47:09.775 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.775 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.775 DEBUG:boto:Params: {} 2024-04-16T19:47:09.775 DEBUG:boto:Token: None 2024-04-16T19:47:09.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6?versionId=null 2024-04-16T19:47:09.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw= 2024-04-16T19:47:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ze/tfjzKvOqgUY2+ZI90El5WTOw=', 'Content-Length': '0'} 2024-04-16T19:47:09.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b152b7e096266a0-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.778 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.778 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.778 DEBUG:boto:path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.778 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.779 DEBUG:boto:Method: GET 2024-04-16T19:47:09.779 DEBUG:boto:Path: /yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.779 DEBUG:boto:Data: 2024-04-16T19:47:09.779 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.779 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.779 DEBUG:boto:Params: {} 2024-04-16T19:47:09.779 DEBUG:boto:Token: None 2024-04-16T19:47:09.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7e0609b384eed5f-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.783 DEBUG:boto:b'yrjefh-40obj61000falseobj62024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.783 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.783 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.784 DEBUG:boto:path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.784 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.784 DEBUG:boto:Method: GET 2024-04-16T19:47:09.784 DEBUG:boto:Path: /yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.784 DEBUG:boto:Data: 2024-04-16T19:47:09.784 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.784 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.784 DEBUG:boto:Params: {} 2024-04-16T19:47:09.784 DEBUG:boto:Token: None 2024-04-16T19:47:09.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50117a3a16ceabc-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.787 DEBUG:boto:b'yrjefh-40obj61000falseobj62024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.787 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.787 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.787 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.788 DEBUG:boto:Path: /yrjefh-40/obj6 2024-04-16T19:47:09.788 DEBUG:boto:Data: 2024-04-16T19:47:09.788 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.788 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.788 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.788 DEBUG:boto:Params: {} 2024-04-16T19:47:09.788 DEBUG:boto:Token: None 2024-04-16T19:47:09.788 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6 2024-04-16T19:47:09.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gIQ2PyRuXzOg+Y1UDeUNd5SosGA= 2024-04-16T19:47:09.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gIQ2PyRuXzOg+Y1UDeUNd5SosGA=', 'Content-Length': '0'} 2024-04-16T19:47:09.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f808feb44cf6e20-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.790 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.790 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.790 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.790 DEBUG:boto:Path: /yrjefh-40/obj6 2024-04-16T19:47:09.791 DEBUG:boto:Data: 2024-04-16T19:47:09.791 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.791 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.791 DEBUG:boto:Params: {} 2024-04-16T19:47:09.791 DEBUG:boto:Token: None 2024-04-16T19:47:09.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6 2024-04-16T19:47:09.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gIQ2PyRuXzOg+Y1UDeUNd5SosGA= 2024-04-16T19:47:09.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gIQ2PyRuXzOg+Y1UDeUNd5SosGA=', 'Content-Length': '0'} 2024-04-16T19:47:09.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c608aa19adb64c6-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:09.793 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.794 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.794 DEBUG:boto:Method: GET 2024-04-16T19:47:09.794 DEBUG:boto:Path: /yrjefh-40/obj6 2024-04-16T19:47:09.794 DEBUG:boto:Data: 2024-04-16T19:47:09.794 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.794 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.794 DEBUG:boto:Params: {} 2024-04-16T19:47:09.794 DEBUG:boto:Token: None 2024-04-16T19:47:09.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6 2024-04-16T19:47:09.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AZH/AuE2soVJ41ahlmxbyd4CDdI= 2024-04-16T19:47:09.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AZH/AuE2soVJ41ahlmxbyd4CDdI=', 'Content-Length': '0'} 2024-04-16T19:47:09.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056ad1ee324da761e-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.838 DEBUG:boto:path=/yrjefh-40/obj6 2024-04-16T19:47:09.838 DEBUG:boto:auth_path=/yrjefh-40/obj6 2024-04-16T19:47:09.838 DEBUG:boto:Method: GET 2024-04-16T19:47:09.839 DEBUG:boto:Path: /yrjefh-40/obj6 2024-04-16T19:47:09.839 DEBUG:boto:Data: 2024-04-16T19:47:09.839 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.839 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.839 DEBUG:boto:Params: {} 2024-04-16T19:47:09.839 DEBUG:boto:Token: None 2024-04-16T19:47:09.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj6 2024-04-16T19:47:09.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AZH/AuE2soVJ41ahlmxbyd4CDdI= 2024-04-16T19:47:09.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AZH/AuE2soVJ41ahlmxbyd4CDdI=', 'Content-Length': '0'} 2024-04-16T19:47:09.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d867eaf5be16bf-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.842 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.842 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.842 DEBUG:boto:path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.842 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.842 DEBUG:boto:Method: GET 2024-04-16T19:47:09.843 DEBUG:boto:Path: /yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.843 DEBUG:boto:Data: 2024-04-16T19:47:09.843 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.843 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.843 DEBUG:boto:Params: {} 2024-04-16T19:47:09.843 DEBUG:boto:Token: None 2024-04-16T19:47:09.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007576015fd6d2467a-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.886 DEBUG:boto:b'yrjefh-40obj61000falseobj62024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.887 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.887 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.887 DEBUG:boto:path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.887 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.887 DEBUG:boto:Method: GET 2024-04-16T19:47:09.887 DEBUG:boto:Path: /yrjefh-40/?prefix=obj6 2024-04-16T19:47:09.887 DEBUG:boto:Data: 2024-04-16T19:47:09.887 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.887 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.887 DEBUG:boto:Params: {} 2024-04-16T19:47:09.888 DEBUG:boto:Token: None 2024-04-16T19:47:09.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a54ce70225cc05-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.890 DEBUG:boto:b'yrjefh-40obj61000falseobj62024-04-16T19:46:53.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:09.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:09.891 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.891 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.891 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.891 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.891 DEBUG:boto:Method: GET 2024-04-16T19:47:09.891 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.891 DEBUG:boto:Data: 2024-04-16T19:47:09.891 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.892 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.892 DEBUG:boto:Params: {} 2024-04-16T19:47:09.892 DEBUG:boto:Token: None 2024-04-16T19:47:09.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A= 2024-04-16T19:47:09.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A=', 'Content-Length': '0'} 2024-04-16T19:47:09.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1439a5ad58af8ea-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.934 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.934 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.934 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.935 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.935 DEBUG:boto:Method: GET 2024-04-16T19:47:09.935 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.935 DEBUG:boto:Data: 2024-04-16T19:47:09.935 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.935 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.935 DEBUG:boto:Params: {} 2024-04-16T19:47:09.935 DEBUG:boto:Token: None 2024-04-16T19:47:09.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A= 2024-04-16T19:47:09.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A=', 'Content-Length': '0'} 2024-04-16T19:47:09.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9cecf61e9eada4a-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.938 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.938 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.938 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.938 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.938 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.938 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.938 DEBUG:boto:Data: 2024-04-16T19:47:09.938 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.939 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.939 DEBUG:boto:Params: {} 2024-04-16T19:47:09.939 DEBUG:boto:Token: None 2024-04-16T19:47:09.939 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vdExoAirw3TOv+YvJwTghFXV064= 2024-04-16T19:47:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vdExoAirw3TOv+YvJwTghFXV064=', 'Content-Length': '0'} 2024-04-16T19:47:09.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004967c6d5d2c00c1a-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.941 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.941 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.942 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.942 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.942 DEBUG:boto:Method: HEAD 2024-04-16T19:47:09.942 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.942 DEBUG:boto:Data: 2024-04-16T19:47:09.942 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.942 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.942 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.942 DEBUG:boto:Params: {} 2024-04-16T19:47:09.942 DEBUG:boto:Token: None 2024-04-16T19:47:09.942 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vdExoAirw3TOv+YvJwTghFXV064= 2024-04-16T19:47:09.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vdExoAirw3TOv+YvJwTghFXV064=', 'Content-Length': '0'} 2024-04-16T19:47:09.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de45b036695447bc-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:09.945 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.945 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.945 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.945 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.945 DEBUG:boto:Method: GET 2024-04-16T19:47:09.945 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.945 DEBUG:boto:Data: 2024-04-16T19:47:09.945 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.945 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.945 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.946 DEBUG:boto:Params: {} 2024-04-16T19:47:09.946 DEBUG:boto:Token: None 2024-04-16T19:47:09.946 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.946 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A= 2024-04-16T19:47:09.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A=', 'Content-Length': '0'} 2024-04-16T19:47:09.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f285d14694c95cc7-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.990 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:09.990 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:09.990 DEBUG:boto:path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.991 DEBUG:boto:auth_path=/yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.991 DEBUG:boto:Method: GET 2024-04-16T19:47:09.991 DEBUG:boto:Path: /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.991 DEBUG:boto:Data: 2024-04-16T19:47:09.991 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:09.991 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.991 DEBUG:boto:Params: {} 2024-04-16T19:47:09.991 DEBUG:boto:Token: None 2024-04-16T19:47:09.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/obj7?versionId=null 2024-04-16T19:47:09.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A= 2024-04-16T19:47:09.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n7K9m1afguvwLToGr0RlzMlxr+A=', 'Content-Length': '0'} 2024-04-16T19:47:09.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d832264da81c6acc-00661ed5bd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:09.994 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:09.994 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:09.994 DEBUG:boto:path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:09.994 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:09.994 DEBUG:boto:Method: GET 2024-04-16T19:47:09.994 DEBUG:boto:Path: /yrjefh-40/?prefix=obj7 2024-04-16T19:47:09.994 DEBUG:boto:Data: 2024-04-16T19:47:09.994 DEBUG:boto:Headers: {} 2024-04-16T19:47:09.994 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:09.994 DEBUG:boto:Port: 8000 2024-04-16T19:47:09.994 DEBUG:boto:Params: {} 2024-04-16T19:47:09.995 DEBUG:boto:Token: None 2024-04-16T19:47:09.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:09 GMT /yrjefh-40/ 2024-04-16T19:47:09.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ= 2024-04-16T19:47:09.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDbDe3BZk2vW/POdcUnH3h8AjSQ=', 'Content-Length': '0'} 2024-04-16T19:47:09.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9ebf3797be00b4-00661ed5bd-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.038 DEBUG:boto:b'yrjefh-40obj71000falseobj72024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.039 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.039 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.039 DEBUG:boto:path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.039 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.039 DEBUG:boto:Method: GET 2024-04-16T19:47:10.039 DEBUG:boto:Path: /yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.039 DEBUG:boto:Data: 2024-04-16T19:47:10.039 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.039 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.039 DEBUG:boto:Params: {} 2024-04-16T19:47:10.040 DEBUG:boto:Token: None 2024-04-16T19:47:10.040 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022c2e7e0c8ff8f58-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.042 DEBUG:boto:b'yrjefh-40obj71000falseobj72024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.043 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:10.043 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:10.043 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.043 DEBUG:boto:Path: /yrjefh-40/obj7 2024-04-16T19:47:10.043 DEBUG:boto:Data: 2024-04-16T19:47:10.043 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.043 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.043 DEBUG:boto:Params: {} 2024-04-16T19:47:10.043 DEBUG:boto:Token: None 2024-04-16T19:47:10.044 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj7 2024-04-16T19:47:10.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VPGPr8pBs4OA6n7BdkBLwZumhek= 2024-04-16T19:47:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VPGPr8pBs4OA6n7BdkBLwZumhek=', 'Content-Length': '0'} 2024-04-16T19:47:10.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b313dc51ed7e6a94-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.046 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:10.046 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:10.046 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.046 DEBUG:boto:Path: /yrjefh-40/obj7 2024-04-16T19:47:10.046 DEBUG:boto:Data: 2024-04-16T19:47:10.046 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.047 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.047 DEBUG:boto:Params: {} 2024-04-16T19:47:10.047 DEBUG:boto:Token: None 2024-04-16T19:47:10.047 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj7 2024-04-16T19:47:10.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VPGPr8pBs4OA6n7BdkBLwZumhek= 2024-04-16T19:47:10.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VPGPr8pBs4OA6n7BdkBLwZumhek=', 'Content-Length': '0'} 2024-04-16T19:47:10.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023f25f66eae1af4b-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:10.049 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:10.050 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:10.050 DEBUG:boto:Method: GET 2024-04-16T19:47:10.050 DEBUG:boto:Path: /yrjefh-40/obj7 2024-04-16T19:47:10.050 DEBUG:boto:Data: 2024-04-16T19:47:10.050 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.050 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.050 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.050 DEBUG:boto:Params: {} 2024-04-16T19:47:10.050 DEBUG:boto:Token: None 2024-04-16T19:47:10.050 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj7 2024-04-16T19:47:10.050 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yYeTYGPjMBgO8O/i4fe4gDqgd8A= 2024-04-16T19:47:10.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yYeTYGPjMBgO8O/i4fe4gDqgd8A=', 'Content-Length': '0'} 2024-04-16T19:47:10.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076fc711992da8029-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.094 DEBUG:boto:path=/yrjefh-40/obj7 2024-04-16T19:47:10.094 DEBUG:boto:auth_path=/yrjefh-40/obj7 2024-04-16T19:47:10.094 DEBUG:boto:Method: GET 2024-04-16T19:47:10.095 DEBUG:boto:Path: /yrjefh-40/obj7 2024-04-16T19:47:10.095 DEBUG:boto:Data: 2024-04-16T19:47:10.095 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.095 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.095 DEBUG:boto:Params: {} 2024-04-16T19:47:10.095 DEBUG:boto:Token: None 2024-04-16T19:47:10.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj7 2024-04-16T19:47:10.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yYeTYGPjMBgO8O/i4fe4gDqgd8A= 2024-04-16T19:47:10.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yYeTYGPjMBgO8O/i4fe4gDqgd8A=', 'Content-Length': '0'} 2024-04-16T19:47:10.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efe21f098d429d70-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.098 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.098 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.098 DEBUG:boto:path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.098 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.098 DEBUG:boto:Method: GET 2024-04-16T19:47:10.098 DEBUG:boto:Path: /yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.098 DEBUG:boto:Data: 2024-04-16T19:47:10.098 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.098 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.098 DEBUG:boto:Params: {} 2024-04-16T19:47:10.099 DEBUG:boto:Token: None 2024-04-16T19:47:10.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a0113f4918e319-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.101 DEBUG:boto:b'yrjefh-40obj71000falseobj72024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.102 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.102 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.102 DEBUG:boto:path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.102 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.102 DEBUG:boto:Method: GET 2024-04-16T19:47:10.102 DEBUG:boto:Path: /yrjefh-40/?prefix=obj7 2024-04-16T19:47:10.102 DEBUG:boto:Data: 2024-04-16T19:47:10.102 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.102 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.102 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.102 DEBUG:boto:Params: {} 2024-04-16T19:47:10.103 DEBUG:boto:Token: None 2024-04-16T19:47:10.103 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008861becaf0e55f1d-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.105 DEBUG:boto:b'yrjefh-40obj71000falseobj72024-04-16T19:46:53.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:10.106 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.106 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.106 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.106 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.106 DEBUG:boto:Method: GET 2024-04-16T19:47:10.106 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.106 DEBUG:boto:Data: 2024-04-16T19:47:10.106 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.106 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.106 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.106 DEBUG:boto:Params: {} 2024-04-16T19:47:10.106 DEBUG:boto:Token: None 2024-04-16T19:47:10.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ= 2024-04-16T19:47:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ=', 'Content-Length': '0'} 2024-04-16T19:47:10.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077ae7bbbe1c2bb59-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.150 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.150 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.151 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.151 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.151 DEBUG:boto:Method: GET 2024-04-16T19:47:10.151 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.151 DEBUG:boto:Data: 2024-04-16T19:47:10.151 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.151 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.151 DEBUG:boto:Params: {} 2024-04-16T19:47:10.151 DEBUG:boto:Token: None 2024-04-16T19:47:10.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ= 2024-04-16T19:47:10.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ=', 'Content-Length': '0'} 2024-04-16T19:47:10.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c3f3712e375f94c-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.154 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.154 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.154 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.154 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.154 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.154 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.154 DEBUG:boto:Data: 2024-04-16T19:47:10.154 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.155 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.155 DEBUG:boto:Params: {} 2024-04-16T19:47:10.155 DEBUG:boto:Token: None 2024-04-16T19:47:10.155 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FkG/YIUr5808Nxs0o8WmcnRbGd4= 2024-04-16T19:47:10.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FkG/YIUr5808Nxs0o8WmcnRbGd4=', 'Content-Length': '0'} 2024-04-16T19:47:10.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065278c537a3efaa3-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.157 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.157 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.157 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.157 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.157 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.157 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.157 DEBUG:boto:Data: 2024-04-16T19:47:10.158 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.158 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.158 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.158 DEBUG:boto:Params: {} 2024-04-16T19:47:10.158 DEBUG:boto:Token: None 2024-04-16T19:47:10.158 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FkG/YIUr5808Nxs0o8WmcnRbGd4= 2024-04-16T19:47:10.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FkG/YIUr5808Nxs0o8WmcnRbGd4=', 'Content-Length': '0'} 2024-04-16T19:47:10.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abf4945c5ca0f5cf-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:10.160 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.160 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.160 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.160 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.161 DEBUG:boto:Method: GET 2024-04-16T19:47:10.161 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.161 DEBUG:boto:Data: 2024-04-16T19:47:10.161 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.161 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.161 DEBUG:boto:Params: {} 2024-04-16T19:47:10.161 DEBUG:boto:Token: None 2024-04-16T19:47:10.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ= 2024-04-16T19:47:10.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ=', 'Content-Length': '0'} 2024-04-16T19:47:10.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fe8dd7d02a7b192-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.206 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.206 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.207 DEBUG:boto:path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.207 DEBUG:boto:auth_path=/yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.207 DEBUG:boto:Method: GET 2024-04-16T19:47:10.207 DEBUG:boto:Path: /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.207 DEBUG:boto:Data: 2024-04-16T19:47:10.207 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.207 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.207 DEBUG:boto:Params: {} 2024-04-16T19:47:10.207 DEBUG:boto:Token: None 2024-04-16T19:47:10.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8?versionId=null 2024-04-16T19:47:10.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ= 2024-04-16T19:47:10.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xELvm41oTlGjQaFLhQDOBsdLVgQ=', 'Content-Length': '0'} 2024-04-16T19:47:10.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df9eae7757290ae8-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.210 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.210 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.210 DEBUG:boto:path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.210 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.210 DEBUG:boto:Method: GET 2024-04-16T19:47:10.210 DEBUG:boto:Path: /yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.210 DEBUG:boto:Data: 2024-04-16T19:47:10.210 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.211 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.211 DEBUG:boto:Params: {} 2024-04-16T19:47:10.211 DEBUG:boto:Token: None 2024-04-16T19:47:10.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbfc949337293f3-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.213 DEBUG:boto:b'yrjefh-40obj81000falseobj82024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.214 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.214 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.214 DEBUG:boto:path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.214 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.214 DEBUG:boto:Method: GET 2024-04-16T19:47:10.214 DEBUG:boto:Path: /yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.214 DEBUG:boto:Data: 2024-04-16T19:47:10.214 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.215 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.215 DEBUG:boto:Params: {} 2024-04-16T19:47:10.215 DEBUG:boto:Token: None 2024-04-16T19:47:10.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9106260c1168bff-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.217 DEBUG:boto:b'yrjefh-40obj81000falseobj82024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.218 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.218 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.218 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.218 DEBUG:boto:Path: /yrjefh-40/obj8 2024-04-16T19:47:10.218 DEBUG:boto:Data: 2024-04-16T19:47:10.218 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.218 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.218 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.218 DEBUG:boto:Params: {} 2024-04-16T19:47:10.219 DEBUG:boto:Token: None 2024-04-16T19:47:10.219 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8 2024-04-16T19:47:10.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yhvLzmU0WLyGQMNP1b01TcQwGtY= 2024-04-16T19:47:10.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yhvLzmU0WLyGQMNP1b01TcQwGtY=', 'Content-Length': '0'} 2024-04-16T19:47:10.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb39b84aa8017a82-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.221 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.221 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.221 DEBUG:boto:Method: HEAD 2024-04-16T19:47:10.221 DEBUG:boto:Path: /yrjefh-40/obj8 2024-04-16T19:47:10.221 DEBUG:boto:Data: 2024-04-16T19:47:10.221 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.221 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.221 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.221 DEBUG:boto:Params: {} 2024-04-16T19:47:10.222 DEBUG:boto:Token: None 2024-04-16T19:47:10.222 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8 2024-04-16T19:47:10.222 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yhvLzmU0WLyGQMNP1b01TcQwGtY= 2024-04-16T19:47:10.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yhvLzmU0WLyGQMNP1b01TcQwGtY=', 'Content-Length': '0'} 2024-04-16T19:47:10.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003778caf5d8c7494e-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:10.224 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.224 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.224 DEBUG:boto:Method: GET 2024-04-16T19:47:10.225 DEBUG:boto:Path: /yrjefh-40/obj8 2024-04-16T19:47:10.225 DEBUG:boto:Data: 2024-04-16T19:47:10.225 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.225 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.225 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.225 DEBUG:boto:Params: {} 2024-04-16T19:47:10.225 DEBUG:boto:Token: None 2024-04-16T19:47:10.225 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8 2024-04-16T19:47:10.225 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBIPPde7uluNEQRDRtnmT2azNRo= 2024-04-16T19:47:10.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBIPPde7uluNEQRDRtnmT2azNRo=', 'Content-Length': '0'} 2024-04-16T19:47:10.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-40:b482a2a9-ef36-488b-9666-ada656453328.4812.9'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000205c7d45aed76b14-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.270 DEBUG:boto:path=/yrjefh-40/obj8 2024-04-16T19:47:10.270 DEBUG:boto:auth_path=/yrjefh-40/obj8 2024-04-16T19:47:10.270 DEBUG:boto:Method: GET 2024-04-16T19:47:10.271 DEBUG:boto:Path: /yrjefh-40/obj8 2024-04-16T19:47:10.271 DEBUG:boto:Data: 2024-04-16T19:47:10.271 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.271 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.271 DEBUG:boto:Params: {} 2024-04-16T19:47:10.271 DEBUG:boto:Token: None 2024-04-16T19:47:10.271 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/obj8 2024-04-16T19:47:10.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sBIPPde7uluNEQRDRtnmT2azNRo= 2024-04-16T19:47:10.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sBIPPde7uluNEQRDRtnmT2azNRo=', 'Content-Length': '0'} 2024-04-16T19:47:10.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b67ac70393730d8-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.273 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.274 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.274 DEBUG:boto:path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.274 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.274 DEBUG:boto:Method: GET 2024-04-16T19:47:10.274 DEBUG:boto:Path: /yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.274 DEBUG:boto:Data: 2024-04-16T19:47:10.274 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.274 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:10.274 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.274 DEBUG:boto:Params: {} 2024-04-16T19:47:10.274 DEBUG:boto:Token: None 2024-04-16T19:47:10.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6bc1062eeeefe4-00661ed5be-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.277 DEBUG:boto:b'yrjefh-40obj81000falseobj82024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.278 DEBUG:boto:path=/yrjefh-40/ 2024-04-16T19:47:10.278 DEBUG:boto:auth_path=/yrjefh-40/ 2024-04-16T19:47:10.278 DEBUG:boto:path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.278 DEBUG:boto:auth_path=/yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.278 DEBUG:boto:Method: GET 2024-04-16T19:47:10.278 DEBUG:boto:Path: /yrjefh-40/?prefix=obj8 2024-04-16T19:47:10.278 DEBUG:boto:Data: 2024-04-16T19:47:10.278 DEBUG:boto:Headers: {} 2024-04-16T19:47:10.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:10.279 DEBUG:boto:Port: 8000 2024-04-16T19:47:10.279 DEBUG:boto:Params: {} 2024-04-16T19:47:10.279 DEBUG:boto:Token: None 2024-04-16T19:47:10.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:10 GMT /yrjefh-40/ 2024-04-16T19:47:10.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0= 2024-04-16T19:47:10.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZCol0/AThPbF+cUb+tV2DtbIO0=', 'Content-Length': '0'} 2024-04-16T19:47:10.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64a21d400c101a2-00661ed5be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:10.282 DEBUG:boto:b'yrjefh-40obj81000falseobj82024-04-16T19:46:53.028Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:10.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-40 zones={test-zone2, test-zone1} 2024-04-16T19:47:10.282 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-41 2024-04-16T19:47:10.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:10.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:10.282 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-41 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:10.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:10.408+0000 7f3f49b35800 1 waiting for incremental sync to catch up: 2024-04-16T19:47:10.410 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.32.6, 00000000002.19.1, 00000000003.28.1, 00000000002.40.1, 00000000003.23.1, 00000000002.53.1, 00000000002.29.1, 00000000003.23.1, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:10.410 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:15.409+0000 7f3f49b35800 1 waiting for incremental sync to catch up: 2024-04-16T19:47:15.411 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.32.6, 00000000002.19.1, 00000000003.28.1, 00000000002.40.1, 00000000003.23.1, 00000000002.53.1, 00000000002.29.1, 00000000003.23.1, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:15.412 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:20.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:20.411+0000 7f3f49b35800 1 waiting for incremental sync to catch up: 2024-04-16T19:47:20.413 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.32.6, 00000000002.19.1, 00000000003.28.1, 00000000002.40.1, 00000000003.23.1, 00000000002.53.1, 00000000002.29.1, 00000000003.23.1, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:20.413 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:25.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:25.412+0000 7f3f49b35800 1 waiting for incremental sync to catch up: 2024-04-16T19:47:25.414 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.32.6, 00000000002.19.1, 00000000003.28.1, 00000000002.40.1, 00000000003.23.1, 00000000002.53.1, 00000000002.29.1, 00000000003.23.1, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:25.414 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:30.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:30.413+0000 7f3f49b35800 1 bucket sync caught up with source: 2024-04-16T19:47:30.416 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:30.416 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000004.32.6, 00000000002.19.1, 00000000004.31.6, 00000000002.40.1, 00000000004.25.6, 00000000002.53.1, 00000000002.29.1, 00000000004.25.6, 00000000002.41.1, 00000000002.32.1, 00000000002.42.1] 2024-04-16T19:47:30.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:30.413+0000 7f3f49b35800 0 bucket checkpoint complete 2024-04-16T19:47:30.441 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-41 2024-04-16T19:47:30.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:30.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-41', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:30.442 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-41 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:30.505 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:30.506 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:30.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:30.566+0000 7f5887328800 1 bucket sync caught up with source: 2024-04-16T19:47:30.568 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.82.1, 00000000002.76.1, 00000000004.75.1, 00000000002.136.1, 00000000004.131.1, 00000000002.178.1, 00000000002.76.1, 00000000004.115.1, 00000000002.146.1, 00000000002.131.1, 00000000002.147.1] 2024-04-16T19:47:30.568 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.82.1, 00000000002.76.1, 00000000004.75.1, 00000000002.136.1, 00000000004.131.1, 00000000002.178.1, 00000000002.76.1, 00000000004.115.1, 00000000002.146.1, 00000000002.131.1, 00000000002.147.1] 2024-04-16T19:47:30.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:30.566+0000 7f5887328800 0 bucket checkpoint complete 2024-04-16T19:47:30.590 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-41 zones={test-zone2, test-zone1} 2024-04-16T19:47:30.590 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.590 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.590 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.590 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:47:30.590 DEBUG:boto:Data: 2024-04-16T19:47:30.590 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.591 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.591 DEBUG:boto:Params: {} 2024-04-16T19:47:30.591 DEBUG:boto:Token: None 2024-04-16T19:47:30.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HINybRTWw3u6IASDhNLIjciVc8U= 2024-04-16T19:47:30.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HINybRTWw3u6IASDhNLIjciVc8U=', 'Content-Length': '0'} 2024-04-16T19:47:30.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7bd2352c5c32d2f-00661ed5d2-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.594 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.594 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.594 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.594 DEBUG:boto:Path: /yrjefh-41/ 2024-04-16T19:47:30.594 DEBUG:boto:Data: 2024-04-16T19:47:30.594 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.594 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.595 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.595 DEBUG:boto:Params: {} 2024-04-16T19:47:30.595 DEBUG:boto:Token: None 2024-04-16T19:47:30.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HINybRTWw3u6IASDhNLIjciVc8U= 2024-04-16T19:47:30.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HINybRTWw3u6IASDhNLIjciVc8U=', 'Content-Length': '0'} 2024-04-16T19:47:30.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006adc7ce937a90c34-00661ed5d2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.598 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:47:30.598 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.598 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.598 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.598 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.598 DEBUG:boto:Method: GET 2024-04-16T19:47:30.599 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.599 DEBUG:boto:Data: 2024-04-16T19:47:30.599 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.599 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.599 DEBUG:boto:Params: {} 2024-04-16T19:47:30.599 DEBUG:boto:Token: None 2024-04-16T19:47:30.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/?versions 2024-04-16T19:47:30.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig= 2024-04-16T19:47:30.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig=', 'Content-Length': '0'} 2024-04-16T19:47:30.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed2ff9b01cdfae9-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.602 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:47:30.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:47:30.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:47:30.604 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:47:30.604 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.604 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.604 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.604 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.604 DEBUG:boto:Method: GET 2024-04-16T19:47:30.604 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.604 DEBUG:boto:Data: 2024-04-16T19:47:30.604 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.604 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.604 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.604 DEBUG:boto:Params: {} 2024-04-16T19:47:30.605 DEBUG:boto:Token: None 2024-04-16T19:47:30.605 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/?versions 2024-04-16T19:47:30.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig= 2024-04-16T19:47:30.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig=', 'Content-Length': '0'} 2024-04-16T19:47:30.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049725e59c158644d-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.607 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-16T19:47:30.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-16T19:47:30.608 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.609 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.609 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.609 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.609 DEBUG:boto:Method: GET 2024-04-16T19:47:30.609 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.609 DEBUG:boto:Data: 2024-04-16T19:47:30.609 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.609 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.609 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.609 DEBUG:boto:Params: {} 2024-04-16T19:47:30.609 DEBUG:boto:Token: None 2024-04-16T19:47:30.610 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/?versions 2024-04-16T19:47:30.610 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig= 2024-04-16T19:47:30.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig=', 'Content-Length': '0'} 2024-04-16T19:47:30.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051db763cf6f2bf31-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.613 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.613 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.613 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.613 DEBUG:boto:path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.613 DEBUG:boto:auth_path=/yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.613 DEBUG:boto:Method: GET 2024-04-16T19:47:30.614 DEBUG:boto:Path: /yrjefh-41/?versions&max-keys=999 2024-04-16T19:47:30.614 DEBUG:boto:Data: 2024-04-16T19:47:30.614 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.614 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.614 DEBUG:boto:Params: {} 2024-04-16T19:47:30.614 DEBUG:boto:Token: None 2024-04-16T19:47:30.614 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/?versions 2024-04-16T19:47:30.614 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig= 2024-04-16T19:47:30.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Vv6ifouSrLu5pgcBu2TZ/xwsig=', 'Content-Length': '0'} 2024-04-16T19:47:30.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fcafedff11ba531-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.617 DEBUG:boto:b'yrjefh-41999falseobj1nulltrue2024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj2nulltrue2024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj3nulltrue2024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj4nulltrue2024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj5nulltrue2024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj6nulltrue2024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj7nulltrue2024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normalobj8nulltrue2024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:30.617 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.617 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.617 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.617 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.618 DEBUG:boto:Method: GET 2024-04-16T19:47:30.618 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.618 DEBUG:boto:Data: 2024-04-16T19:47:30.618 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.618 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.618 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.618 DEBUG:boto:Params: {} 2024-04-16T19:47:30.618 DEBUG:boto:Token: None 2024-04-16T19:47:30.618 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.618 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ= 2024-04-16T19:47:30.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc1bb9060fb49d85-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.662 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.662 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.663 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.663 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.663 DEBUG:boto:Method: GET 2024-04-16T19:47:30.663 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.663 DEBUG:boto:Data: 2024-04-16T19:47:30.663 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.663 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.663 DEBUG:boto:Params: {} 2024-04-16T19:47:30.663 DEBUG:boto:Token: None 2024-04-16T19:47:30.663 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ= 2024-04-16T19:47:30.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000404ea744d655afdb-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.666 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.666 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.666 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.666 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.666 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.666 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.666 DEBUG:boto:Data: 2024-04-16T19:47:30.666 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.666 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.666 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.667 DEBUG:boto:Params: {} 2024-04-16T19:47:30.667 DEBUG:boto:Token: None 2024-04-16T19:47:30.667 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3hejNRbmUNao6cbsXDFuB6WVDIs= 2024-04-16T19:47:30.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3hejNRbmUNao6cbsXDFuB6WVDIs=', 'Content-Length': '0'} 2024-04-16T19:47:30.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afe5db88f7cb5237-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.669 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.669 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.669 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.669 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.669 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.669 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.669 DEBUG:boto:Data: 2024-04-16T19:47:30.669 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.669 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.669 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.670 DEBUG:boto:Params: {} 2024-04-16T19:47:30.670 DEBUG:boto:Token: None 2024-04-16T19:47:30.670 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3hejNRbmUNao6cbsXDFuB6WVDIs= 2024-04-16T19:47:30.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3hejNRbmUNao6cbsXDFuB6WVDIs=', 'Content-Length': '0'} 2024-04-16T19:47:30.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad7f705c5d7e41b-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:30.672 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.672 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.672 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.672 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.672 DEBUG:boto:Method: GET 2024-04-16T19:47:30.673 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.673 DEBUG:boto:Data: 2024-04-16T19:47:30.673 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.673 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.673 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.673 DEBUG:boto:Params: {} 2024-04-16T19:47:30.673 DEBUG:boto:Token: None 2024-04-16T19:47:30.673 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ= 2024-04-16T19:47:30.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5ec46c13e84e45a-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.675 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.676 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.676 DEBUG:boto:path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.676 DEBUG:boto:auth_path=/yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.676 DEBUG:boto:Method: GET 2024-04-16T19:47:30.676 DEBUG:boto:Path: /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.676 DEBUG:boto:Data: 2024-04-16T19:47:30.676 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.676 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.676 DEBUG:boto:Params: {} 2024-04-16T19:47:30.676 DEBUG:boto:Token: None 2024-04-16T19:47:30.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1?versionId=null 2024-04-16T19:47:30.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ= 2024-04-16T19:47:30.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IQyGRwQQMW6ql3WNb1OisbI+YQQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0e0e2e895224982-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.679 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.679 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.679 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.679 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.679 DEBUG:boto:Method: GET 2024-04-16T19:47:30.679 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.679 DEBUG:boto:Data: 2024-04-16T19:47:30.679 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.679 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.679 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.680 DEBUG:boto:Params: {} 2024-04-16T19:47:30.680 DEBUG:boto:Token: None 2024-04-16T19:47:30.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7432630b5f69446-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.683 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.683 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.683 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.683 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.683 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.683 DEBUG:boto:Method: GET 2024-04-16T19:47:30.683 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.684 DEBUG:boto:Data: 2024-04-16T19:47:30.684 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.684 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.684 DEBUG:boto:Params: {} 2024-04-16T19:47:30.684 DEBUG:boto:Token: None 2024-04-16T19:47:30.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368e4a5b152eca8f-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.687 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.687 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.687 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.687 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.687 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:47:30.687 DEBUG:boto:Data: 2024-04-16T19:47:30.687 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.688 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.688 DEBUG:boto:Params: {} 2024-04-16T19:47:30.688 DEBUG:boto:Token: None 2024-04-16T19:47:30.688 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1 2024-04-16T19:47:30.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+kCqlFpit/N+D9xJG4nRQEx1Ig= 2024-04-16T19:47:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+kCqlFpit/N+D9xJG4nRQEx1Ig=', 'Content-Length': '0'} 2024-04-16T19:47:30.690 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa9a597b6c6be363-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.690 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.690 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.690 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.690 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:47:30.690 DEBUG:boto:Data: 2024-04-16T19:47:30.691 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.691 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.691 DEBUG:boto:Params: {} 2024-04-16T19:47:30.691 DEBUG:boto:Token: None 2024-04-16T19:47:30.691 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1 2024-04-16T19:47:30.691 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+kCqlFpit/N+D9xJG4nRQEx1Ig= 2024-04-16T19:47:30.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+kCqlFpit/N+D9xJG4nRQEx1Ig=', 'Content-Length': '0'} 2024-04-16T19:47:30.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005783d8c5440b19f4-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-16T19:47:30.693 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.693 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.693 DEBUG:boto:Method: GET 2024-04-16T19:47:30.693 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:47:30.693 DEBUG:boto:Data: 2024-04-16T19:47:30.693 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.694 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.694 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.694 DEBUG:boto:Params: {} 2024-04-16T19:47:30.694 DEBUG:boto:Token: None 2024-04-16T19:47:30.694 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1 2024-04-16T19:47:30.694 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tYt5r9TA9TN9t5u+tYfCQ+3n2hg= 2024-04-16T19:47:30.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tYt5r9TA9TN9t5u+tYfCQ+3n2hg=', 'Content-Length': '0'} 2024-04-16T19:47:30.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa557471974dd1de-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.738 DEBUG:boto:path=/yrjefh-41/obj1 2024-04-16T19:47:30.738 DEBUG:boto:auth_path=/yrjefh-41/obj1 2024-04-16T19:47:30.739 DEBUG:boto:Method: GET 2024-04-16T19:47:30.739 DEBUG:boto:Path: /yrjefh-41/obj1 2024-04-16T19:47:30.739 DEBUG:boto:Data: 2024-04-16T19:47:30.739 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.739 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.739 DEBUG:boto:Params: {} 2024-04-16T19:47:30.739 DEBUG:boto:Token: None 2024-04-16T19:47:30.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj1 2024-04-16T19:47:30.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tYt5r9TA9TN9t5u+tYfCQ+3n2hg= 2024-04-16T19:47:30.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tYt5r9TA9TN9t5u+tYfCQ+3n2hg=', 'Content-Length': '0'} 2024-04-16T19:47:30.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9e9358473cf0146-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.782 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.782 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.782 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.782 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.783 DEBUG:boto:Method: GET 2024-04-16T19:47:30.783 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.783 DEBUG:boto:Data: 2024-04-16T19:47:30.783 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.783 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.783 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.783 DEBUG:boto:Params: {} 2024-04-16T19:47:30.783 DEBUG:boto:Token: None 2024-04-16T19:47:30.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c6104301931cf58-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.786 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.786 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.786 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.786 DEBUG:boto:path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.786 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.786 DEBUG:boto:Method: GET 2024-04-16T19:47:30.787 DEBUG:boto:Path: /yrjefh-41/?prefix=obj1 2024-04-16T19:47:30.787 DEBUG:boto:Data: 2024-04-16T19:47:30.787 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.787 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.787 DEBUG:boto:Params: {} 2024-04-16T19:47:30.787 DEBUG:boto:Token: None 2024-04-16T19:47:30.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003144a2885d038aca-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.830 DEBUG:boto:b'yrjefh-41obj11000falseobj12024-04-16T19:46:34.702Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:30.831 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.831 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.831 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.831 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.831 DEBUG:boto:Method: GET 2024-04-16T19:47:30.831 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.831 DEBUG:boto:Data: 2024-04-16T19:47:30.831 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.831 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.831 DEBUG:boto:Params: {} 2024-04-16T19:47:30.832 DEBUG:boto:Token: None 2024-04-16T19:47:30.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E= 2024-04-16T19:47:30.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E=', 'Content-Length': '0'} 2024-04-16T19:47:30.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af2e010aadd76d69-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.834 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.834 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.834 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.834 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.834 DEBUG:boto:Method: GET 2024-04-16T19:47:30.835 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.835 DEBUG:boto:Data: 2024-04-16T19:47:30.835 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.835 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.835 DEBUG:boto:Params: {} 2024-04-16T19:47:30.835 DEBUG:boto:Token: None 2024-04-16T19:47:30.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E= 2024-04-16T19:47:30.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E=', 'Content-Length': '0'} 2024-04-16T19:47:30.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013e7b6e7d393ed21-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.878 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.879 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.879 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.879 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.879 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.879 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.879 DEBUG:boto:Data: 2024-04-16T19:47:30.879 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.879 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.879 DEBUG:boto:Params: {} 2024-04-16T19:47:30.879 DEBUG:boto:Token: None 2024-04-16T19:47:30.879 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:khcTHY15OnO28T0XDiz3ch/vEE4= 2024-04-16T19:47:30.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:khcTHY15OnO28T0XDiz3ch/vEE4=', 'Content-Length': '0'} 2024-04-16T19:47:30.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005775ebec9fd1982b-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.882 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.882 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.882 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.882 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.882 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.882 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.882 DEBUG:boto:Data: 2024-04-16T19:47:30.882 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.882 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.882 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.882 DEBUG:boto:Params: {} 2024-04-16T19:47:30.882 DEBUG:boto:Token: None 2024-04-16T19:47:30.883 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:khcTHY15OnO28T0XDiz3ch/vEE4= 2024-04-16T19:47:30.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:khcTHY15OnO28T0XDiz3ch/vEE4=', 'Content-Length': '0'} 2024-04-16T19:47:30.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000408dd5b13d5aa0f9-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:30.885 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.885 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.885 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.885 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.885 DEBUG:boto:Method: GET 2024-04-16T19:47:30.885 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.885 DEBUG:boto:Data: 2024-04-16T19:47:30.885 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.885 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.885 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.885 DEBUG:boto:Params: {} 2024-04-16T19:47:30.886 DEBUG:boto:Token: None 2024-04-16T19:47:30.886 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.886 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E= 2024-04-16T19:47:30.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E=', 'Content-Length': '0'} 2024-04-16T19:47:30.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052f4928a21b05593-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.888 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.888 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.888 DEBUG:boto:path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.888 DEBUG:boto:auth_path=/yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.889 DEBUG:boto:Method: GET 2024-04-16T19:47:30.889 DEBUG:boto:Path: /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.889 DEBUG:boto:Data: 2024-04-16T19:47:30.889 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.889 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.889 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.889 DEBUG:boto:Params: {} 2024-04-16T19:47:30.889 DEBUG:boto:Token: None 2024-04-16T19:47:30.889 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2?versionId=null 2024-04-16T19:47:30.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E= 2024-04-16T19:47:30.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FGyL+Nr6K+ujQCqYoLW7FpK4p/E=', 'Content-Length': '0'} 2024-04-16T19:47:30.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6ead3b00a3709b0-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.934 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.935 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.935 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.935 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.935 DEBUG:boto:Method: GET 2024-04-16T19:47:30.935 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.935 DEBUG:boto:Data: 2024-04-16T19:47:30.935 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.935 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.935 DEBUG:boto:Params: {} 2024-04-16T19:47:30.935 DEBUG:boto:Token: None 2024-04-16T19:47:30.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab9fa5254daa68ec-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.938 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.938 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:30.939 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:30.939 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.939 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.939 DEBUG:boto:Method: GET 2024-04-16T19:47:30.939 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:47:30.939 DEBUG:boto:Data: 2024-04-16T19:47:30.939 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.939 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.939 DEBUG:boto:Params: {} 2024-04-16T19:47:30.939 DEBUG:boto:Token: None 2024-04-16T19:47:30.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/ 2024-04-16T19:47:30.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ= 2024-04-16T19:47:30.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uut8U/+93XTs19XsINkfzc0U2PQ=', 'Content-Length': '0'} 2024-04-16T19:47:30.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613dcfcf3d36aad2-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.982 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:30.983 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.983 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.983 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.983 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:47:30.983 DEBUG:boto:Data: 2024-04-16T19:47:30.983 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.983 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.983 DEBUG:boto:Params: {} 2024-04-16T19:47:30.983 DEBUG:boto:Token: None 2024-04-16T19:47:30.984 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2 2024-04-16T19:47:30.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFUBKbcEMfTB43WKHEWwl/Q/Uvw= 2024-04-16T19:47:30.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFUBKbcEMfTB43WKHEWwl/Q/Uvw=', 'Content-Length': '0'} 2024-04-16T19:47:30.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e01fe8d59d62463e-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.986 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.986 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.986 DEBUG:boto:Method: HEAD 2024-04-16T19:47:30.986 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:47:30.986 DEBUG:boto:Data: 2024-04-16T19:47:30.986 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.986 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.986 DEBUG:boto:Params: {} 2024-04-16T19:47:30.987 DEBUG:boto:Token: None 2024-04-16T19:47:30.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2 2024-04-16T19:47:30.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFUBKbcEMfTB43WKHEWwl/Q/Uvw= 2024-04-16T19:47:30.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFUBKbcEMfTB43WKHEWwl/Q/Uvw=', 'Content-Length': '0'} 2024-04-16T19:47:30.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000234c4ada3e5e690-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-16T19:47:30.989 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.989 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.989 DEBUG:boto:Method: GET 2024-04-16T19:47:30.989 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:47:30.989 DEBUG:boto:Data: 2024-04-16T19:47:30.989 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.989 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:30.989 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.990 DEBUG:boto:Params: {} 2024-04-16T19:47:30.990 DEBUG:boto:Token: None 2024-04-16T19:47:30.990 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2 2024-04-16T19:47:30.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Q2VqQQVB4IZ/eUa7M4NzJjNNjU= 2024-04-16T19:47:30.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Q2VqQQVB4IZ/eUa7M4NzJjNNjU=', 'Content-Length': '0'} 2024-04-16T19:47:30.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d8c4dd0d660d15f-00661ed5d2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:30.992 DEBUG:boto:path=/yrjefh-41/obj2 2024-04-16T19:47:30.992 DEBUG:boto:auth_path=/yrjefh-41/obj2 2024-04-16T19:47:30.992 DEBUG:boto:Method: GET 2024-04-16T19:47:30.992 DEBUG:boto:Path: /yrjefh-41/obj2 2024-04-16T19:47:30.992 DEBUG:boto:Data: 2024-04-16T19:47:30.992 DEBUG:boto:Headers: {} 2024-04-16T19:47:30.993 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:30.993 DEBUG:boto:Port: 8000 2024-04-16T19:47:30.993 DEBUG:boto:Params: {} 2024-04-16T19:47:30.993 DEBUG:boto:Token: None 2024-04-16T19:47:30.993 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:30 GMT /yrjefh-41/obj2 2024-04-16T19:47:30.993 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Q2VqQQVB4IZ/eUa7M4NzJjNNjU= 2024-04-16T19:47:30.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Q2VqQQVB4IZ/eUa7M4NzJjNNjU=', 'Content-Length': '0'} 2024-04-16T19:47:30.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061bfefbbf92a2cda-00661ed5d2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.038 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.038 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.038 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.039 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.039 DEBUG:boto:Method: GET 2024-04-16T19:47:31.039 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.039 DEBUG:boto:Data: 2024-04-16T19:47:31.039 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.039 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.039 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.039 DEBUG:boto:Params: {} 2024-04-16T19:47:31.039 DEBUG:boto:Token: None 2024-04-16T19:47:31.039 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e140148ee722132-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.042 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.043 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.043 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.043 DEBUG:boto:path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.043 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.043 DEBUG:boto:Method: GET 2024-04-16T19:47:31.043 DEBUG:boto:Path: /yrjefh-41/?prefix=obj2 2024-04-16T19:47:31.043 DEBUG:boto:Data: 2024-04-16T19:47:31.043 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.043 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.043 DEBUG:boto:Params: {} 2024-04-16T19:47:31.044 DEBUG:boto:Token: None 2024-04-16T19:47:31.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a12b1b57a599822-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.046 DEBUG:boto:b'yrjefh-41obj21000falseobj22024-04-16T19:46:34.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:31.047 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.047 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.047 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.047 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.047 DEBUG:boto:Method: GET 2024-04-16T19:47:31.047 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.047 DEBUG:boto:Data: 2024-04-16T19:47:31.047 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.048 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.048 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.048 DEBUG:boto:Params: {} 2024-04-16T19:47:31.048 DEBUG:boto:Token: None 2024-04-16T19:47:31.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc= 2024-04-16T19:47:31.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc=', 'Content-Length': '0'} 2024-04-16T19:47:31.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000071485ca0c65534d-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.050 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.050 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.050 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.050 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.051 DEBUG:boto:Method: GET 2024-04-16T19:47:31.051 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.051 DEBUG:boto:Data: 2024-04-16T19:47:31.051 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.051 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.051 DEBUG:boto:Params: {} 2024-04-16T19:47:31.051 DEBUG:boto:Token: None 2024-04-16T19:47:31.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc= 2024-04-16T19:47:31.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc=', 'Content-Length': '0'} 2024-04-16T19:47:31.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c397b50f2b845c73-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.094 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.094 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.095 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.095 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.095 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.095 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.095 DEBUG:boto:Data: 2024-04-16T19:47:31.095 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.095 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.095 DEBUG:boto:Params: {} 2024-04-16T19:47:31.095 DEBUG:boto:Token: None 2024-04-16T19:47:31.095 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aUlrDplFzuUYMuaoiuDslEoLqRk= 2024-04-16T19:47:31.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aUlrDplFzuUYMuaoiuDslEoLqRk=', 'Content-Length': '0'} 2024-04-16T19:47:31.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008318ad3785c7ca8e-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.098 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.098 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.098 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.098 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.098 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.098 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.098 DEBUG:boto:Data: 2024-04-16T19:47:31.099 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.099 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.099 DEBUG:boto:Params: {} 2024-04-16T19:47:31.099 DEBUG:boto:Token: None 2024-04-16T19:47:31.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aUlrDplFzuUYMuaoiuDslEoLqRk= 2024-04-16T19:47:31.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aUlrDplFzuUYMuaoiuDslEoLqRk=', 'Content-Length': '0'} 2024-04-16T19:47:31.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d3f0845159b3e6-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:31.101 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.101 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.101 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.101 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.101 DEBUG:boto:Method: GET 2024-04-16T19:47:31.101 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.102 DEBUG:boto:Data: 2024-04-16T19:47:31.102 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.102 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.102 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.102 DEBUG:boto:Params: {} 2024-04-16T19:47:31.102 DEBUG:boto:Token: None 2024-04-16T19:47:31.102 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc= 2024-04-16T19:47:31.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc=', 'Content-Length': '0'} 2024-04-16T19:47:31.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f6e693c75c4b277-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.104 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.104 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.105 DEBUG:boto:path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.105 DEBUG:boto:auth_path=/yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.105 DEBUG:boto:Method: GET 2024-04-16T19:47:31.105 DEBUG:boto:Path: /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.105 DEBUG:boto:Data: 2024-04-16T19:47:31.105 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.105 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.105 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.105 DEBUG:boto:Params: {} 2024-04-16T19:47:31.105 DEBUG:boto:Token: None 2024-04-16T19:47:31.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3?versionId=null 2024-04-16T19:47:31.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc= 2024-04-16T19:47:31.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6zT39fMQo1vBUX9ajoesNrVDKc=', 'Content-Length': '0'} 2024-04-16T19:47:31.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072a94192c17ceb59-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.150 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.150 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.151 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.151 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.151 DEBUG:boto:Method: GET 2024-04-16T19:47:31.151 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.151 DEBUG:boto:Data: 2024-04-16T19:47:31.151 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.151 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.151 DEBUG:boto:Params: {} 2024-04-16T19:47:31.151 DEBUG:boto:Token: None 2024-04-16T19:47:31.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336fb80f897bdb09-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.154 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.155 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.155 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.155 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.155 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.155 DEBUG:boto:Method: GET 2024-04-16T19:47:31.155 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.155 DEBUG:boto:Data: 2024-04-16T19:47:31.155 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.155 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.155 DEBUG:boto:Params: {} 2024-04-16T19:47:31.155 DEBUG:boto:Token: None 2024-04-16T19:47:31.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001282f6d4de2aad35-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.158 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.159 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.159 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.159 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.159 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:47:31.159 DEBUG:boto:Data: 2024-04-16T19:47:31.159 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.159 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.159 DEBUG:boto:Params: {} 2024-04-16T19:47:31.160 DEBUG:boto:Token: None 2024-04-16T19:47:31.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3 2024-04-16T19:47:31.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E7Ky0wK4sT/6bXgLGMKxJ7+ls9M= 2024-04-16T19:47:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E7Ky0wK4sT/6bXgLGMKxJ7+ls9M=', 'Content-Length': '0'} 2024-04-16T19:47:31.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a62893cd4b5f57e1-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.162 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.162 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.162 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.162 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:47:31.162 DEBUG:boto:Data: 2024-04-16T19:47:31.162 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.162 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.162 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.162 DEBUG:boto:Params: {} 2024-04-16T19:47:31.163 DEBUG:boto:Token: None 2024-04-16T19:47:31.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3 2024-04-16T19:47:31.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E7Ky0wK4sT/6bXgLGMKxJ7+ls9M= 2024-04-16T19:47:31.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E7Ky0wK4sT/6bXgLGMKxJ7+ls9M=', 'Content-Length': '0'} 2024-04-16T19:47:31.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aa8ceb0d69111a1-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-16T19:47:31.165 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.165 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.165 DEBUG:boto:Method: GET 2024-04-16T19:47:31.165 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:47:31.165 DEBUG:boto:Data: 2024-04-16T19:47:31.165 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.165 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.165 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.165 DEBUG:boto:Params: {} 2024-04-16T19:47:31.165 DEBUG:boto:Token: None 2024-04-16T19:47:31.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3 2024-04-16T19:47:31.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8U44kqXGIW705vmGt5Dtuov+R3I= 2024-04-16T19:47:31.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8U44kqXGIW705vmGt5Dtuov+R3I=', 'Content-Length': '0'} 2024-04-16T19:47:31.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ac7ee0a0c9c5732-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.168 DEBUG:boto:path=/yrjefh-41/obj3 2024-04-16T19:47:31.168 DEBUG:boto:auth_path=/yrjefh-41/obj3 2024-04-16T19:47:31.169 DEBUG:boto:Method: GET 2024-04-16T19:47:31.169 DEBUG:boto:Path: /yrjefh-41/obj3 2024-04-16T19:47:31.169 DEBUG:boto:Data: 2024-04-16T19:47:31.169 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.169 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.169 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.169 DEBUG:boto:Params: {} 2024-04-16T19:47:31.169 DEBUG:boto:Token: None 2024-04-16T19:47:31.169 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj3 2024-04-16T19:47:31.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8U44kqXGIW705vmGt5Dtuov+R3I= 2024-04-16T19:47:31.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8U44kqXGIW705vmGt5Dtuov+R3I=', 'Content-Length': '0'} 2024-04-16T19:47:31.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:35 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec9187495a4f7c8a-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.214 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.215 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.215 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.215 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.215 DEBUG:boto:Method: GET 2024-04-16T19:47:31.215 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.215 DEBUG:boto:Data: 2024-04-16T19:47:31.215 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.215 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.215 DEBUG:boto:Params: {} 2024-04-16T19:47:31.215 DEBUG:boto:Token: None 2024-04-16T19:47:31.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb48db846b7736f-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.218 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.218 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.219 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.219 DEBUG:boto:path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.219 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.219 DEBUG:boto:Method: GET 2024-04-16T19:47:31.219 DEBUG:boto:Path: /yrjefh-41/?prefix=obj3 2024-04-16T19:47:31.219 DEBUG:boto:Data: 2024-04-16T19:47:31.219 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.219 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.219 DEBUG:boto:Params: {} 2024-04-16T19:47:31.219 DEBUG:boto:Token: None 2024-04-16T19:47:31.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf6145eb5920509-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.222 DEBUG:boto:b'yrjefh-41obj31000falseobj32024-04-16T19:46:34.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:31.223 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.223 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.223 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.223 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.223 DEBUG:boto:Method: GET 2024-04-16T19:47:31.223 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.223 DEBUG:boto:Data: 2024-04-16T19:47:31.223 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.223 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.224 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.224 DEBUG:boto:Params: {} 2024-04-16T19:47:31.224 DEBUG:boto:Token: None 2024-04-16T19:47:31.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8= 2024-04-16T19:47:31.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8=', 'Content-Length': '0'} 2024-04-16T19:47:31.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e554b6faa422d147-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.227 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.227 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.227 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.227 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.227 DEBUG:boto:Method: GET 2024-04-16T19:47:31.227 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.227 DEBUG:boto:Data: 2024-04-16T19:47:31.227 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.227 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.227 DEBUG:boto:Params: {} 2024-04-16T19:47:31.228 DEBUG:boto:Token: None 2024-04-16T19:47:31.228 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8= 2024-04-16T19:47:31.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8=', 'Content-Length': '0'} 2024-04-16T19:47:31.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072c78ef9cbdb003e-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.270 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.270 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.271 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.271 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.271 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.271 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.271 DEBUG:boto:Data: 2024-04-16T19:47:31.271 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.271 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.271 DEBUG:boto:Params: {} 2024-04-16T19:47:31.271 DEBUG:boto:Token: None 2024-04-16T19:47:31.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nl3Rduw0g0vfpSbCQILCNJFnbB4= 2024-04-16T19:47:31.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nl3Rduw0g0vfpSbCQILCNJFnbB4=', 'Content-Length': '0'} 2024-04-16T19:47:31.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b229c3c2df5937a-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.274 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.274 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.274 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.274 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.274 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.274 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.274 DEBUG:boto:Data: 2024-04-16T19:47:31.275 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.275 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.275 DEBUG:boto:Params: {} 2024-04-16T19:47:31.275 DEBUG:boto:Token: None 2024-04-16T19:47:31.275 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nl3Rduw0g0vfpSbCQILCNJFnbB4= 2024-04-16T19:47:31.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nl3Rduw0g0vfpSbCQILCNJFnbB4=', 'Content-Length': '0'} 2024-04-16T19:47:31.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057789fa48290b35e-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:31.278 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.278 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.278 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.278 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.278 DEBUG:boto:Method: GET 2024-04-16T19:47:31.278 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.278 DEBUG:boto:Data: 2024-04-16T19:47:31.278 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.278 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.278 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.279 DEBUG:boto:Params: {} 2024-04-16T19:47:31.279 DEBUG:boto:Token: None 2024-04-16T19:47:31.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8= 2024-04-16T19:47:31.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8=', 'Content-Length': '0'} 2024-04-16T19:47:31.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fd08ad029529d1e-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.281 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.281 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.281 DEBUG:boto:path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.282 DEBUG:boto:auth_path=/yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.282 DEBUG:boto:Method: GET 2024-04-16T19:47:31.282 DEBUG:boto:Path: /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.282 DEBUG:boto:Data: 2024-04-16T19:47:31.282 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.282 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.282 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.282 DEBUG:boto:Params: {} 2024-04-16T19:47:31.282 DEBUG:boto:Token: None 2024-04-16T19:47:31.282 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4?versionId=null 2024-04-16T19:47:31.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8= 2024-04-16T19:47:31.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Ed51i59yPJwyzSXrbihlRgIuR8=', 'Content-Length': '0'} 2024-04-16T19:47:31.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738409e9753f18ba-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.330 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.331 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.331 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.331 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.331 DEBUG:boto:Method: GET 2024-04-16T19:47:31.331 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.331 DEBUG:boto:Data: 2024-04-16T19:47:31.331 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.331 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.331 DEBUG:boto:Params: {} 2024-04-16T19:47:31.331 DEBUG:boto:Token: None 2024-04-16T19:47:31.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000893f390a867934db-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.334 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.335 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.335 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.335 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.335 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.335 DEBUG:boto:Method: GET 2024-04-16T19:47:31.335 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.335 DEBUG:boto:Data: 2024-04-16T19:47:31.335 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.336 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.336 DEBUG:boto:Params: {} 2024-04-16T19:47:31.336 DEBUG:boto:Token: None 2024-04-16T19:47:31.336 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005896cd3ec69ff2ad-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.339 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.339 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.339 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.339 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.339 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:47:31.339 DEBUG:boto:Data: 2024-04-16T19:47:31.339 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.340 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.340 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.340 DEBUG:boto:Params: {} 2024-04-16T19:47:31.340 DEBUG:boto:Token: None 2024-04-16T19:47:31.340 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4 2024-04-16T19:47:31.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:guW1sAKDT6BfN2lFDdbkAeZSVik= 2024-04-16T19:47:31.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:guW1sAKDT6BfN2lFDdbkAeZSVik=', 'Content-Length': '0'} 2024-04-16T19:47:31.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c83fff84a26936b-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.342 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.342 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.342 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.343 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:47:31.343 DEBUG:boto:Data: 2024-04-16T19:47:31.343 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.343 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.343 DEBUG:boto:Params: {} 2024-04-16T19:47:31.343 DEBUG:boto:Token: None 2024-04-16T19:47:31.343 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4 2024-04-16T19:47:31.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:guW1sAKDT6BfN2lFDdbkAeZSVik= 2024-04-16T19:47:31.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:guW1sAKDT6BfN2lFDdbkAeZSVik=', 'Content-Length': '0'} 2024-04-16T19:47:31.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e315c9e3cabab00-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-16T19:47:31.346 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.346 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.346 DEBUG:boto:Method: GET 2024-04-16T19:47:31.346 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:47:31.346 DEBUG:boto:Data: 2024-04-16T19:47:31.346 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.346 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.346 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.346 DEBUG:boto:Params: {} 2024-04-16T19:47:31.346 DEBUG:boto:Token: None 2024-04-16T19:47:31.346 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4 2024-04-16T19:47:31.347 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LJVGBS+cbl3eS034A3FZ4NBR1hU= 2024-04-16T19:47:31.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LJVGBS+cbl3eS034A3FZ4NBR1hU=', 'Content-Length': '0'} 2024-04-16T19:47:31.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004614cae165d51684-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.390 DEBUG:boto:path=/yrjefh-41/obj4 2024-04-16T19:47:31.390 DEBUG:boto:auth_path=/yrjefh-41/obj4 2024-04-16T19:47:31.391 DEBUG:boto:Method: GET 2024-04-16T19:47:31.391 DEBUG:boto:Path: /yrjefh-41/obj4 2024-04-16T19:47:31.391 DEBUG:boto:Data: 2024-04-16T19:47:31.391 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.391 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.391 DEBUG:boto:Params: {} 2024-04-16T19:47:31.391 DEBUG:boto:Token: None 2024-04-16T19:47:31.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj4 2024-04-16T19:47:31.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LJVGBS+cbl3eS034A3FZ4NBR1hU= 2024-04-16T19:47:31.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LJVGBS+cbl3eS034A3FZ4NBR1hU=', 'Content-Length': '0'} 2024-04-16T19:47:31.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:46:34 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed1b1985680197f-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.434 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.434 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.435 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.435 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.435 DEBUG:boto:Method: GET 2024-04-16T19:47:31.435 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.435 DEBUG:boto:Data: 2024-04-16T19:47:31.435 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.435 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.435 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.435 DEBUG:boto:Params: {} 2024-04-16T19:47:31.435 DEBUG:boto:Token: None 2024-04-16T19:47:31.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f6fef3a1dc6dac7-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.438 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.438 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.439 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.439 DEBUG:boto:path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.439 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.439 DEBUG:boto:Method: GET 2024-04-16T19:47:31.439 DEBUG:boto:Path: /yrjefh-41/?prefix=obj4 2024-04-16T19:47:31.439 DEBUG:boto:Data: 2024-04-16T19:47:31.439 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.439 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.439 DEBUG:boto:Params: {} 2024-04-16T19:47:31.439 DEBUG:boto:Token: None 2024-04-16T19:47:31.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb48868a3fde9e4-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.486 DEBUG:boto:b'yrjefh-41obj41000falseobj42024-04-16T19:46:34.772Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:31.487 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.487 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.487 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.487 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.487 DEBUG:boto:Method: GET 2024-04-16T19:47:31.487 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.487 DEBUG:boto:Data: 2024-04-16T19:47:31.487 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.488 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.488 DEBUG:boto:Params: {} 2024-04-16T19:47:31.488 DEBUG:boto:Token: None 2024-04-16T19:47:31.488 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M= 2024-04-16T19:47:31.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M=', 'Content-Length': '0'} 2024-04-16T19:47:31.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa2f3ed47ed2b9af-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.490 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.490 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.490 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.491 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.491 DEBUG:boto:Method: GET 2024-04-16T19:47:31.491 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.491 DEBUG:boto:Data: 2024-04-16T19:47:31.491 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.491 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.491 DEBUG:boto:Params: {} 2024-04-16T19:47:31.491 DEBUG:boto:Token: None 2024-04-16T19:47:31.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M= 2024-04-16T19:47:31.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M=', 'Content-Length': '0'} 2024-04-16T19:47:31.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e00258dff444e88-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.534 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.534 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.535 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.535 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.535 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.535 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.535 DEBUG:boto:Data: 2024-04-16T19:47:31.535 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.535 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.535 DEBUG:boto:Params: {} 2024-04-16T19:47:31.535 DEBUG:boto:Token: None 2024-04-16T19:47:31.535 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PD2o+d3QMg1yqCrCxYr76dfoE50= 2024-04-16T19:47:31.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PD2o+d3QMg1yqCrCxYr76dfoE50=', 'Content-Length': '0'} 2024-04-16T19:47:31.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbc9b5dc4061d92c-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.538 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.538 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.538 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.538 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.538 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.538 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.538 DEBUG:boto:Data: 2024-04-16T19:47:31.538 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.539 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.539 DEBUG:boto:Params: {} 2024-04-16T19:47:31.539 DEBUG:boto:Token: None 2024-04-16T19:47:31.539 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PD2o+d3QMg1yqCrCxYr76dfoE50= 2024-04-16T19:47:31.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PD2o+d3QMg1yqCrCxYr76dfoE50=', 'Content-Length': '0'} 2024-04-16T19:47:31.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000016f42fdd97158e5-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:31.541 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.541 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.542 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.542 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.542 DEBUG:boto:Method: GET 2024-04-16T19:47:31.542 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.542 DEBUG:boto:Data: 2024-04-16T19:47:31.542 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.542 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.542 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.542 DEBUG:boto:Params: {} 2024-04-16T19:47:31.542 DEBUG:boto:Token: None 2024-04-16T19:47:31.542 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M= 2024-04-16T19:47:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M=', 'Content-Length': '0'} 2024-04-16T19:47:31.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053f79cf37ae5559c-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.545 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.545 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.545 DEBUG:boto:path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.545 DEBUG:boto:auth_path=/yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.546 DEBUG:boto:Method: GET 2024-04-16T19:47:31.546 DEBUG:boto:Path: /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.546 DEBUG:boto:Data: 2024-04-16T19:47:31.546 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.546 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.546 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.546 DEBUG:boto:Params: {} 2024-04-16T19:47:31.546 DEBUG:boto:Token: None 2024-04-16T19:47:31.546 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5?versionId=null 2024-04-16T19:47:31.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M= 2024-04-16T19:47:31.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EbRZ3REFTOr6RI3hwKV5XF++92M=', 'Content-Length': '0'} 2024-04-16T19:47:31.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1f6ad70ac1177c0-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.590 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.590 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.591 DEBUG:boto:path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.591 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.591 DEBUG:boto:Method: GET 2024-04-16T19:47:31.591 DEBUG:boto:Path: /yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.591 DEBUG:boto:Data: 2024-04-16T19:47:31.591 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.591 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.591 DEBUG:boto:Params: {} 2024-04-16T19:47:31.591 DEBUG:boto:Token: None 2024-04-16T19:47:31.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a89b40fdaf253be9-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.594 DEBUG:boto:b'yrjefh-41obj51000falseobj52024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.595 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.595 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.595 DEBUG:boto:path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.595 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.595 DEBUG:boto:Method: GET 2024-04-16T19:47:31.595 DEBUG:boto:Path: /yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.595 DEBUG:boto:Data: 2024-04-16T19:47:31.595 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.596 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.596 DEBUG:boto:Params: {} 2024-04-16T19:47:31.596 DEBUG:boto:Token: None 2024-04-16T19:47:31.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028dba5693523388e-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.642 DEBUG:boto:b'yrjefh-41obj51000falseobj52024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.643 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.643 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.643 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.643 DEBUG:boto:Path: /yrjefh-41/obj5 2024-04-16T19:47:31.643 DEBUG:boto:Data: 2024-04-16T19:47:31.643 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.643 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.643 DEBUG:boto:Params: {} 2024-04-16T19:47:31.643 DEBUG:boto:Token: None 2024-04-16T19:47:31.644 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5 2024-04-16T19:47:31.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/1BbVr1ubsEEY30Rn5OFspttbU8= 2024-04-16T19:47:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/1BbVr1ubsEEY30Rn5OFspttbU8=', 'Content-Length': '0'} 2024-04-16T19:47:31.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006db4514b7bbadb18-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.646 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.646 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.646 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.646 DEBUG:boto:Path: /yrjefh-41/obj5 2024-04-16T19:47:31.647 DEBUG:boto:Data: 2024-04-16T19:47:31.647 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.647 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.647 DEBUG:boto:Params: {} 2024-04-16T19:47:31.647 DEBUG:boto:Token: None 2024-04-16T19:47:31.647 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5 2024-04-16T19:47:31.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/1BbVr1ubsEEY30Rn5OFspttbU8= 2024-04-16T19:47:31.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/1BbVr1ubsEEY30Rn5OFspttbU8=', 'Content-Length': '0'} 2024-04-16T19:47:31.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6e8b67714a6ffea-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-16T19:47:31.650 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.650 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.650 DEBUG:boto:Method: GET 2024-04-16T19:47:31.650 DEBUG:boto:Path: /yrjefh-41/obj5 2024-04-16T19:47:31.650 DEBUG:boto:Data: 2024-04-16T19:47:31.650 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.651 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.651 DEBUG:boto:Params: {} 2024-04-16T19:47:31.651 DEBUG:boto:Token: None 2024-04-16T19:47:31.651 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5 2024-04-16T19:47:31.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VZUpt1yTU32ExQDXTZVCluOCv70= 2024-04-16T19:47:31.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VZUpt1yTU32ExQDXTZVCluOCv70=', 'Content-Length': '0'} 2024-04-16T19:47:31.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caf6ee6187602d5c-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.653 DEBUG:boto:path=/yrjefh-41/obj5 2024-04-16T19:47:31.654 DEBUG:boto:auth_path=/yrjefh-41/obj5 2024-04-16T19:47:31.654 DEBUG:boto:Method: GET 2024-04-16T19:47:31.654 DEBUG:boto:Path: /yrjefh-41/obj5 2024-04-16T19:47:31.654 DEBUG:boto:Data: 2024-04-16T19:47:31.654 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.654 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.654 DEBUG:boto:Params: {} 2024-04-16T19:47:31.654 DEBUG:boto:Token: None 2024-04-16T19:47:31.654 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj5 2024-04-16T19:47:31.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VZUpt1yTU32ExQDXTZVCluOCv70= 2024-04-16T19:47:31.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VZUpt1yTU32ExQDXTZVCluOCv70=', 'Content-Length': '0'} 2024-04-16T19:47:31.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee2f4a290ca2314d-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.698 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.699 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.699 DEBUG:boto:path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.699 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.699 DEBUG:boto:Method: GET 2024-04-16T19:47:31.699 DEBUG:boto:Path: /yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.699 DEBUG:boto:Data: 2024-04-16T19:47:31.699 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.699 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.699 DEBUG:boto:Params: {} 2024-04-16T19:47:31.699 DEBUG:boto:Token: None 2024-04-16T19:47:31.699 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073dd4fc93dceebd6-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.702 DEBUG:boto:b'yrjefh-41obj51000falseobj52024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.703 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.703 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.703 DEBUG:boto:path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.703 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.703 DEBUG:boto:Method: GET 2024-04-16T19:47:31.703 DEBUG:boto:Path: /yrjefh-41/?prefix=obj5 2024-04-16T19:47:31.703 DEBUG:boto:Data: 2024-04-16T19:47:31.703 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.703 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.703 DEBUG:boto:Params: {} 2024-04-16T19:47:31.704 DEBUG:boto:Token: None 2024-04-16T19:47:31.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e8da44f101b74d-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.706 DEBUG:boto:b'yrjefh-41obj51000falseobj52024-04-16T19:46:53.077Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:31.707 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.707 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.707 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.707 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.707 DEBUG:boto:Method: GET 2024-04-16T19:47:31.707 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.707 DEBUG:boto:Data: 2024-04-16T19:47:31.707 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.708 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.708 DEBUG:boto:Params: {} 2024-04-16T19:47:31.708 DEBUG:boto:Token: None 2024-04-16T19:47:31.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE= 2024-04-16T19:47:31.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE=', 'Content-Length': '0'} 2024-04-16T19:47:31.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d98ee32501cead-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.710 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.711 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.711 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.711 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.711 DEBUG:boto:Method: GET 2024-04-16T19:47:31.711 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.711 DEBUG:boto:Data: 2024-04-16T19:47:31.711 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.711 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.711 DEBUG:boto:Params: {} 2024-04-16T19:47:31.711 DEBUG:boto:Token: None 2024-04-16T19:47:31.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE= 2024-04-16T19:47:31.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE=', 'Content-Length': '0'} 2024-04-16T19:47:31.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e2333f5b9c447ac-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.754 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.754 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.755 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.755 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.755 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.755 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.755 DEBUG:boto:Data: 2024-04-16T19:47:31.755 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.755 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.755 DEBUG:boto:Params: {} 2024-04-16T19:47:31.755 DEBUG:boto:Token: None 2024-04-16T19:47:31.755 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bd3kke8BoelWD9pzchcxtmyd7eU= 2024-04-16T19:47:31.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bd3kke8BoelWD9pzchcxtmyd7eU=', 'Content-Length': '0'} 2024-04-16T19:47:31.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000592cb9b3a9f62c19-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.758 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.758 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.758 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.758 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.759 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.759 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.759 DEBUG:boto:Data: 2024-04-16T19:47:31.759 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.759 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.759 DEBUG:boto:Params: {} 2024-04-16T19:47:31.759 DEBUG:boto:Token: None 2024-04-16T19:47:31.759 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bd3kke8BoelWD9pzchcxtmyd7eU= 2024-04-16T19:47:31.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bd3kke8BoelWD9pzchcxtmyd7eU=', 'Content-Length': '0'} 2024-04-16T19:47:31.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000911eca62b479485e-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:31.762 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.762 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.762 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.762 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.762 DEBUG:boto:Method: GET 2024-04-16T19:47:31.762 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.762 DEBUG:boto:Data: 2024-04-16T19:47:31.762 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.762 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.762 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.762 DEBUG:boto:Params: {} 2024-04-16T19:47:31.763 DEBUG:boto:Token: None 2024-04-16T19:47:31.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE= 2024-04-16T19:47:31.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE=', 'Content-Length': '0'} 2024-04-16T19:47:31.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080e3ab404273903f-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.765 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.765 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.765 DEBUG:boto:path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.765 DEBUG:boto:auth_path=/yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.766 DEBUG:boto:Method: GET 2024-04-16T19:47:31.766 DEBUG:boto:Path: /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.766 DEBUG:boto:Data: 2024-04-16T19:47:31.766 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.766 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.766 DEBUG:boto:Params: {} 2024-04-16T19:47:31.766 DEBUG:boto:Token: None 2024-04-16T19:47:31.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6?versionId=null 2024-04-16T19:47:31.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE= 2024-04-16T19:47:31.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PK/77j2Dlwyvql+iaozCgF2KkxE=', 'Content-Length': '0'} 2024-04-16T19:47:31.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cafc088ea53ec563-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.811 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.811 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.811 DEBUG:boto:path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.811 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.811 DEBUG:boto:Method: GET 2024-04-16T19:47:31.811 DEBUG:boto:Path: /yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.811 DEBUG:boto:Data: 2024-04-16T19:47:31.812 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.812 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.812 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.812 DEBUG:boto:Params: {} 2024-04-16T19:47:31.812 DEBUG:boto:Token: None 2024-04-16T19:47:31.812 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce5c8bcd824df610-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.814 DEBUG:boto:b'yrjefh-41obj61000falseobj62024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.815 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.815 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.815 DEBUG:boto:path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.815 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.815 DEBUG:boto:Method: GET 2024-04-16T19:47:31.815 DEBUG:boto:Path: /yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.815 DEBUG:boto:Data: 2024-04-16T19:47:31.816 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.816 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.816 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.816 DEBUG:boto:Params: {} 2024-04-16T19:47:31.816 DEBUG:boto:Token: None 2024-04-16T19:47:31.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a65a8c786e65b8-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.819 DEBUG:boto:b'yrjefh-41obj61000falseobj62024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.819 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.819 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.819 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.819 DEBUG:boto:Path: /yrjefh-41/obj6 2024-04-16T19:47:31.819 DEBUG:boto:Data: 2024-04-16T19:47:31.820 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.820 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.820 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.820 DEBUG:boto:Params: {} 2024-04-16T19:47:31.820 DEBUG:boto:Token: None 2024-04-16T19:47:31.820 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6 2024-04-16T19:47:31.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ISaJuiQR3izL5//lnYB/Vek4Sfk= 2024-04-16T19:47:31.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ISaJuiQR3izL5//lnYB/Vek4Sfk=', 'Content-Length': '0'} 2024-04-16T19:47:31.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059191f8e1c6a6cae-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.822 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.823 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.823 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.823 DEBUG:boto:Path: /yrjefh-41/obj6 2024-04-16T19:47:31.823 DEBUG:boto:Data: 2024-04-16T19:47:31.823 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.823 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.823 DEBUG:boto:Params: {} 2024-04-16T19:47:31.823 DEBUG:boto:Token: None 2024-04-16T19:47:31.823 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6 2024-04-16T19:47:31.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ISaJuiQR3izL5//lnYB/Vek4Sfk= 2024-04-16T19:47:31.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ISaJuiQR3izL5//lnYB/Vek4Sfk=', 'Content-Length': '0'} 2024-04-16T19:47:31.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a679f6658859e1ca-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-16T19:47:31.826 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.826 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.826 DEBUG:boto:Method: GET 2024-04-16T19:47:31.826 DEBUG:boto:Path: /yrjefh-41/obj6 2024-04-16T19:47:31.826 DEBUG:boto:Data: 2024-04-16T19:47:31.826 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.826 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.826 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.827 DEBUG:boto:Params: {} 2024-04-16T19:47:31.827 DEBUG:boto:Token: None 2024-04-16T19:47:31.827 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6 2024-04-16T19:47:31.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VDYjyE5L0kPWfDx22bSnSfg3WKs= 2024-04-16T19:47:31.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VDYjyE5L0kPWfDx22bSnSfg3WKs=', 'Content-Length': '0'} 2024-04-16T19:47:31.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88058cb028d5cb6-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.870 DEBUG:boto:path=/yrjefh-41/obj6 2024-04-16T19:47:31.870 DEBUG:boto:auth_path=/yrjefh-41/obj6 2024-04-16T19:47:31.871 DEBUG:boto:Method: GET 2024-04-16T19:47:31.871 DEBUG:boto:Path: /yrjefh-41/obj6 2024-04-16T19:47:31.871 DEBUG:boto:Data: 2024-04-16T19:47:31.871 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.871 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.871 DEBUG:boto:Params: {} 2024-04-16T19:47:31.871 DEBUG:boto:Token: None 2024-04-16T19:47:31.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj6 2024-04-16T19:47:31.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VDYjyE5L0kPWfDx22bSnSfg3WKs= 2024-04-16T19:47:31.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VDYjyE5L0kPWfDx22bSnSfg3WKs=', 'Content-Length': '0'} 2024-04-16T19:47:31.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075f4c2615839d5eb-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.914 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.915 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.915 DEBUG:boto:path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.915 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.915 DEBUG:boto:Method: GET 2024-04-16T19:47:31.915 DEBUG:boto:Path: /yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.915 DEBUG:boto:Data: 2024-04-16T19:47:31.915 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.915 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.915 DEBUG:boto:Params: {} 2024-04-16T19:47:31.915 DEBUG:boto:Token: None 2024-04-16T19:47:31.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb5c9e5e0dd2ae8b-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.918 DEBUG:boto:b'yrjefh-41obj61000falseobj62024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.919 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:31.919 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:31.919 DEBUG:boto:path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.919 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.919 DEBUG:boto:Method: GET 2024-04-16T19:47:31.919 DEBUG:boto:Path: /yrjefh-41/?prefix=obj6 2024-04-16T19:47:31.919 DEBUG:boto:Data: 2024-04-16T19:47:31.919 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.919 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.919 DEBUG:boto:Params: {} 2024-04-16T19:47:31.920 DEBUG:boto:Token: None 2024-04-16T19:47:31.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/ 2024-04-16T19:47:31.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM= 2024-04-16T19:47:31.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4aUEiITHE3OFaR+uz8zpeLL6CM=', 'Content-Length': '0'} 2024-04-16T19:47:31.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007252aafd5966e349-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.925 DEBUG:boto:b'yrjefh-41obj61000falseobj62024-04-16T19:46:53.125Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:31.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:31.925 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.926 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.926 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.926 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.926 DEBUG:boto:Method: GET 2024-04-16T19:47:31.926 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.926 DEBUG:boto:Data: 2024-04-16T19:47:31.926 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.926 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.926 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.926 DEBUG:boto:Params: {} 2024-04-16T19:47:31.926 DEBUG:boto:Token: None 2024-04-16T19:47:31.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg= 2024-04-16T19:47:31.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg=', 'Content-Length': '0'} 2024-04-16T19:47:31.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e66f33258367b1d-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.929 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.929 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.930 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.930 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.930 DEBUG:boto:Method: GET 2024-04-16T19:47:31.930 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.930 DEBUG:boto:Data: 2024-04-16T19:47:31.930 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.930 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.930 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.930 DEBUG:boto:Params: {} 2024-04-16T19:47:31.930 DEBUG:boto:Token: None 2024-04-16T19:47:31.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg= 2024-04-16T19:47:31.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg=', 'Content-Length': '0'} 2024-04-16T19:47:31.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:05 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008de982b4836482c8-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.974 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.974 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.975 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.975 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.975 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.975 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.975 DEBUG:boto:Data: 2024-04-16T19:47:31.975 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.975 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.975 DEBUG:boto:Params: {} 2024-04-16T19:47:31.975 DEBUG:boto:Token: None 2024-04-16T19:47:31.975 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdGu9e9NBptn/ILk9cHK4I8Q2as= 2024-04-16T19:47:31.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdGu9e9NBptn/ILk9cHK4I8Q2as=', 'Content-Length': '0'} 2024-04-16T19:47:31.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8e8576557e6054a-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.978 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.978 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.978 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.978 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.978 DEBUG:boto:Method: HEAD 2024-04-16T19:47:31.979 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.979 DEBUG:boto:Data: 2024-04-16T19:47:31.979 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.979 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.979 DEBUG:boto:Params: {} 2024-04-16T19:47:31.979 DEBUG:boto:Token: None 2024-04-16T19:47:31.979 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdGu9e9NBptn/ILk9cHK4I8Q2as= 2024-04-16T19:47:31.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdGu9e9NBptn/ILk9cHK4I8Q2as=', 'Content-Length': '0'} 2024-04-16T19:47:31.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:05 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f0bed7cf3fbfb67-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:31.982 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.982 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.982 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.982 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.982 DEBUG:boto:Method: GET 2024-04-16T19:47:31.983 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.983 DEBUG:boto:Data: 2024-04-16T19:47:31.983 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:31.983 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.983 DEBUG:boto:Params: {} 2024-04-16T19:47:31.983 DEBUG:boto:Token: None 2024-04-16T19:47:31.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg= 2024-04-16T19:47:31.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg=', 'Content-Length': '0'} 2024-04-16T19:47:31.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d7fee36de172a9-00661ed5d3-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:31.986 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:31.986 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:31.986 DEBUG:boto:path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.986 DEBUG:boto:auth_path=/yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.986 DEBUG:boto:Method: GET 2024-04-16T19:47:31.986 DEBUG:boto:Path: /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.986 DEBUG:boto:Data: 2024-04-16T19:47:31.986 DEBUG:boto:Headers: {} 2024-04-16T19:47:31.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:31.987 DEBUG:boto:Port: 8000 2024-04-16T19:47:31.987 DEBUG:boto:Params: {} 2024-04-16T19:47:31.987 DEBUG:boto:Token: None 2024-04-16T19:47:31.987 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:31 GMT /yrjefh-41/obj7?versionId=null 2024-04-16T19:47:31.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg= 2024-04-16T19:47:31.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n+mASFGFxtcBihg32AIkit5sWXg=', 'Content-Length': '0'} 2024-04-16T19:47:31.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:05 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df4a03cf6b741a9-00661ed5d3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.030 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.030 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.030 DEBUG:boto:path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.031 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.031 DEBUG:boto:Method: GET 2024-04-16T19:47:32.031 DEBUG:boto:Path: /yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.031 DEBUG:boto:Data: 2024-04-16T19:47:32.031 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.031 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.031 DEBUG:boto:Params: {} 2024-04-16T19:47:32.031 DEBUG:boto:Token: None 2024-04-16T19:47:32.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db6c021bb2666f6-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.034 DEBUG:boto:b'yrjefh-41obj71000falseobj72024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.035 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.035 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.035 DEBUG:boto:path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.035 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.035 DEBUG:boto:Method: GET 2024-04-16T19:47:32.035 DEBUG:boto:Path: /yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.035 DEBUG:boto:Data: 2024-04-16T19:47:32.035 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.035 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.035 DEBUG:boto:Params: {} 2024-04-16T19:47:32.036 DEBUG:boto:Token: None 2024-04-16T19:47:32.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9f923bcdf85bb9-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.038 DEBUG:boto:b'yrjefh-41obj71000falseobj72024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.039 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:32.039 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:32.039 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.039 DEBUG:boto:Path: /yrjefh-41/obj7 2024-04-16T19:47:32.039 DEBUG:boto:Data: 2024-04-16T19:47:32.039 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.039 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.039 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.039 DEBUG:boto:Params: {} 2024-04-16T19:47:32.039 DEBUG:boto:Token: None 2024-04-16T19:47:32.040 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj7 2024-04-16T19:47:32.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Kd1BMkOmzzqFv6fozFkXVU0go4= 2024-04-16T19:47:32.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Kd1BMkOmzzqFv6fozFkXVU0go4=', 'Content-Length': '0'} 2024-04-16T19:47:32.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c079f7818cec25da-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.042 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:32.042 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:32.042 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.042 DEBUG:boto:Path: /yrjefh-41/obj7 2024-04-16T19:47:32.042 DEBUG:boto:Data: 2024-04-16T19:47:32.042 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.042 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.043 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.043 DEBUG:boto:Params: {} 2024-04-16T19:47:32.043 DEBUG:boto:Token: None 2024-04-16T19:47:32.043 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj7 2024-04-16T19:47:32.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/Kd1BMkOmzzqFv6fozFkXVU0go4= 2024-04-16T19:47:32.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/Kd1BMkOmzzqFv6fozFkXVU0go4=', 'Content-Length': '0'} 2024-04-16T19:47:32.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:05 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea494332b694068c-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-16T19:47:32.045 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:32.045 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:32.046 DEBUG:boto:Method: GET 2024-04-16T19:47:32.046 DEBUG:boto:Path: /yrjefh-41/obj7 2024-04-16T19:47:32.046 DEBUG:boto:Data: 2024-04-16T19:47:32.046 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.046 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.046 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.046 DEBUG:boto:Params: {} 2024-04-16T19:47:32.046 DEBUG:boto:Token: None 2024-04-16T19:47:32.046 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj7 2024-04-16T19:47:32.046 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aV9wdBr2hn5H3eZT/AspVrffh00= 2024-04-16T19:47:32.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aV9wdBr2hn5H3eZT/AspVrffh00=', 'Content-Length': '0'} 2024-04-16T19:47:32.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006598c6988bcb2209-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.049 DEBUG:boto:path=/yrjefh-41/obj7 2024-04-16T19:47:32.049 DEBUG:boto:auth_path=/yrjefh-41/obj7 2024-04-16T19:47:32.049 DEBUG:boto:Method: GET 2024-04-16T19:47:32.049 DEBUG:boto:Path: /yrjefh-41/obj7 2024-04-16T19:47:32.049 DEBUG:boto:Data: 2024-04-16T19:47:32.050 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.050 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.050 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.050 DEBUG:boto:Params: {} 2024-04-16T19:47:32.050 DEBUG:boto:Token: None 2024-04-16T19:47:32.050 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj7 2024-04-16T19:47:32.050 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aV9wdBr2hn5H3eZT/AspVrffh00= 2024-04-16T19:47:32.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aV9wdBr2hn5H3eZT/AspVrffh00=', 'Content-Length': '0'} 2024-04-16T19:47:32.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:05 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000899da8ea75052bfc-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.094 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.094 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.095 DEBUG:boto:path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.095 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.095 DEBUG:boto:Method: GET 2024-04-16T19:47:32.095 DEBUG:boto:Path: /yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.095 DEBUG:boto:Data: 2024-04-16T19:47:32.095 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.095 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.095 DEBUG:boto:Params: {} 2024-04-16T19:47:32.095 DEBUG:boto:Token: None 2024-04-16T19:47:32.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021976c35b2254db7-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.099 DEBUG:boto:b'yrjefh-41obj71000falseobj72024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.099 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.099 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.099 DEBUG:boto:path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.099 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.099 DEBUG:boto:Method: GET 2024-04-16T19:47:32.099 DEBUG:boto:Path: /yrjefh-41/?prefix=obj7 2024-04-16T19:47:32.099 DEBUG:boto:Data: 2024-04-16T19:47:32.100 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.100 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.100 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.100 DEBUG:boto:Params: {} 2024-04-16T19:47:32.100 DEBUG:boto:Token: None 2024-04-16T19:47:32.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5b2b28e3734445-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.146 DEBUG:boto:b'yrjefh-41obj71000falseobj72024-04-16T19:46:53.134Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:32.147 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.147 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.147 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.147 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.147 DEBUG:boto:Method: GET 2024-04-16T19:47:32.147 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.147 DEBUG:boto:Data: 2024-04-16T19:47:32.147 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.148 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.148 DEBUG:boto:Params: {} 2024-04-16T19:47:32.148 DEBUG:boto:Token: None 2024-04-16T19:47:32.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ= 2024-04-16T19:47:32.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ=', 'Content-Length': '0'} 2024-04-16T19:47:32.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003422038c79a6a4e2-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.150 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.150 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.150 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.150 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.150 DEBUG:boto:Method: GET 2024-04-16T19:47:32.151 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.151 DEBUG:boto:Data: 2024-04-16T19:47:32.151 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.151 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.151 DEBUG:boto:Params: {} 2024-04-16T19:47:32.151 DEBUG:boto:Token: None 2024-04-16T19:47:32.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ= 2024-04-16T19:47:32.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ=', 'Content-Length': '0'} 2024-04-16T19:47:32.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc50863e23fc9e83-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.194 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.194 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.195 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.195 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.195 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.195 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.195 DEBUG:boto:Data: 2024-04-16T19:47:32.195 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.195 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.195 DEBUG:boto:Params: {} 2024-04-16T19:47:32.195 DEBUG:boto:Token: None 2024-04-16T19:47:32.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tRbZuNB6Vg+s45H2FYBG3vro2i4= 2024-04-16T19:47:32.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tRbZuNB6Vg+s45H2FYBG3vro2i4=', 'Content-Length': '0'} 2024-04-16T19:47:32.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000221f03afbbde5e4-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.197 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.198 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.198 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.198 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.198 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.198 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.198 DEBUG:boto:Data: 2024-04-16T19:47:32.198 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.198 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.198 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.198 DEBUG:boto:Params: {} 2024-04-16T19:47:32.198 DEBUG:boto:Token: None 2024-04-16T19:47:32.199 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tRbZuNB6Vg+s45H2FYBG3vro2i4= 2024-04-16T19:47:32.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tRbZuNB6Vg+s45H2FYBG3vro2i4=', 'Content-Length': '0'} 2024-04-16T19:47:32.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0f4cc286b50b4f-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:32.201 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.201 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.201 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.201 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.201 DEBUG:boto:Method: GET 2024-04-16T19:47:32.201 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.201 DEBUG:boto:Data: 2024-04-16T19:47:32.201 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.201 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.201 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.201 DEBUG:boto:Params: {} 2024-04-16T19:47:32.202 DEBUG:boto:Token: None 2024-04-16T19:47:32.202 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.202 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ= 2024-04-16T19:47:32.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ=', 'Content-Length': '0'} 2024-04-16T19:47:32.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8ab765c1f2456da-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.204 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.204 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.204 DEBUG:boto:path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.204 DEBUG:boto:auth_path=/yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.204 DEBUG:boto:Method: GET 2024-04-16T19:47:32.204 DEBUG:boto:Path: /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.205 DEBUG:boto:Data: 2024-04-16T19:47:32.205 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.205 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.205 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.205 DEBUG:boto:Params: {} 2024-04-16T19:47:32.205 DEBUG:boto:Token: None 2024-04-16T19:47:32.205 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8?versionId=null 2024-04-16T19:47:32.205 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ= 2024-04-16T19:47:32.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsgC29IxiwrkBza6IAG1k8UWupQ=', 'Content-Length': '0'} 2024-04-16T19:47:32.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000415a4a4c181fa799-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.250 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.250 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.251 DEBUG:boto:path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.251 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.251 DEBUG:boto:Method: GET 2024-04-16T19:47:32.251 DEBUG:boto:Path: /yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.251 DEBUG:boto:Data: 2024-04-16T19:47:32.251 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.251 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.251 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.251 DEBUG:boto:Params: {} 2024-04-16T19:47:32.251 DEBUG:boto:Token: None 2024-04-16T19:47:32.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e1cb359242d0c3-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.255 DEBUG:boto:b'yrjefh-41obj81000falseobj82024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.255 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.255 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.255 DEBUG:boto:path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.255 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.255 DEBUG:boto:Method: GET 2024-04-16T19:47:32.255 DEBUG:boto:Path: /yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.255 DEBUG:boto:Data: 2024-04-16T19:47:32.256 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.256 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.256 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.256 DEBUG:boto:Params: {} 2024-04-16T19:47:32.256 DEBUG:boto:Token: None 2024-04-16T19:47:32.256 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420d5c54ee1d3845-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.302 DEBUG:boto:b'yrjefh-41obj81000falseobj82024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.303 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.303 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.303 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.303 DEBUG:boto:Path: /yrjefh-41/obj8 2024-04-16T19:47:32.303 DEBUG:boto:Data: 2024-04-16T19:47:32.303 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.303 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.303 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.303 DEBUG:boto:Params: {} 2024-04-16T19:47:32.303 DEBUG:boto:Token: None 2024-04-16T19:47:32.304 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8 2024-04-16T19:47:32.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cn7MNUyYpe/ikG86DbLRYejWkdw= 2024-04-16T19:47:32.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cn7MNUyYpe/ikG86DbLRYejWkdw=', 'Content-Length': '0'} 2024-04-16T19:47:32.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3ad341c31bebd4b-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.306 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.306 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.306 DEBUG:boto:Method: HEAD 2024-04-16T19:47:32.306 DEBUG:boto:Path: /yrjefh-41/obj8 2024-04-16T19:47:32.306 DEBUG:boto:Data: 2024-04-16T19:47:32.306 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.306 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.307 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.307 DEBUG:boto:Params: {} 2024-04-16T19:47:32.307 DEBUG:boto:Token: None 2024-04-16T19:47:32.307 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8 2024-04-16T19:47:32.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cn7MNUyYpe/ikG86DbLRYejWkdw= 2024-04-16T19:47:32.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cn7MNUyYpe/ikG86DbLRYejWkdw=', 'Content-Length': '0'} 2024-04-16T19:47:32.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064a95e67c445efe9-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-16T19:47:32.310 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.310 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.310 DEBUG:boto:Method: GET 2024-04-16T19:47:32.310 DEBUG:boto:Path: /yrjefh-41/obj8 2024-04-16T19:47:32.310 DEBUG:boto:Data: 2024-04-16T19:47:32.310 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.310 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.310 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.310 DEBUG:boto:Params: {} 2024-04-16T19:47:32.310 DEBUG:boto:Token: None 2024-04-16T19:47:32.310 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8 2024-04-16T19:47:32.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUoKzNb1tNZAGbwkaaIK6ZpdU+8= 2024-04-16T19:47:32.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUoKzNb1tNZAGbwkaaIK6ZpdU+8=', 'Content-Length': '0'} 2024-04-16T19:47:32.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aa289f9df9833d4-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.313 DEBUG:boto:path=/yrjefh-41/obj8 2024-04-16T19:47:32.313 DEBUG:boto:auth_path=/yrjefh-41/obj8 2024-04-16T19:47:32.313 DEBUG:boto:Method: GET 2024-04-16T19:47:32.313 DEBUG:boto:Path: /yrjefh-41/obj8 2024-04-16T19:47:32.313 DEBUG:boto:Data: 2024-04-16T19:47:32.313 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.313 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.313 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.313 DEBUG:boto:Params: {} 2024-04-16T19:47:32.314 DEBUG:boto:Token: None 2024-04-16T19:47:32.314 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/obj8 2024-04-16T19:47:32.314 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUoKzNb1tNZAGbwkaaIK6ZpdU+8= 2024-04-16T19:47:32.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUoKzNb1tNZAGbwkaaIK6ZpdU+8=', 'Content-Length': '0'} 2024-04-16T19:47:32.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:46:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'eafd0fcf-abef-4654-893b-ccd3b2eb5bea:yrjefh-41:b482a2a9-ef36-488b-9666-ada656453328.4812.10'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:03 GMT'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e4c6b2d1cf006da-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.358 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.359 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.359 DEBUG:boto:path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.359 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.359 DEBUG:boto:Method: GET 2024-04-16T19:47:32.359 DEBUG:boto:Path: /yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.359 DEBUG:boto:Data: 2024-04-16T19:47:32.359 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.359 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.359 DEBUG:boto:Params: {} 2024-04-16T19:47:32.359 DEBUG:boto:Token: None 2024-04-16T19:47:32.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae12c13bea7c6a9-00661ed5d4-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.363 DEBUG:boto:b'yrjefh-41obj81000falseobj82024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.363 DEBUG:boto:path=/yrjefh-41/ 2024-04-16T19:47:32.363 DEBUG:boto:auth_path=/yrjefh-41/ 2024-04-16T19:47:32.363 DEBUG:boto:path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.363 DEBUG:boto:auth_path=/yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.363 DEBUG:boto:Method: GET 2024-04-16T19:47:32.363 DEBUG:boto:Path: /yrjefh-41/?prefix=obj8 2024-04-16T19:47:32.363 DEBUG:boto:Data: 2024-04-16T19:47:32.364 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.364 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.364 DEBUG:boto:Params: {} 2024-04-16T19:47:32.364 DEBUG:boto:Token: None 2024-04-16T19:47:32.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-41/ 2024-04-16T19:47:32.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA= 2024-04-16T19:47:32.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XraIFCzlD3Fafd4/icX48OdQNMA=', 'Content-Length': '0'} 2024-04-16T19:47:32.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328f1fd75aab31f3-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.367 DEBUG:boto:b'yrjefh-41obj81000falseobj82024-04-16T19:46:53.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:32.367 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-41 zones={test-zone2, test-zone1} 2024-04-16T19:47:32.368 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T19:47:32.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.368 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:32.422 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:32.422 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:32.507 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000009089","last_update":"2024-04-16T19:47:25.473955Z"},{"marker":"00000000000000000000:00000000000000006549","last_update":"2024-04-16T19:47:25.475565Z"},{"marker":"00000000000000000000:00000000000000011470","last_update":"2024-04-16T19:47:25.477260Z"},{"marker":"00000000000000000000:00000000000000009084","last_update":"2024-04-16T19:46:53.431985Z"}] 2024-04-16T19:47:32.508 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000009089', 1: '00000000000000000000:00000000000000006549', 2: '00000000000000000000:00000000000000011470', 3: '00000000000000000000:00000000000000009084'} 2024-04-16T19:47:32.508 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:47:32.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.508 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:32.565 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:32.565 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:32.644 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009089","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.473955Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.475565Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011470","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.477260Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009084","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:53.431985Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:47:32Z"} 2024-04-16T19:47:32.644 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4249039607640047526},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009089","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.473955Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006549","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.475565Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000011470","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:25.477260Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009084","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:46:53.431985Z"}}]},"full_sync":{"total":0,"complete":0},"current_time":"2024-04-16T19:47:32Z"} 2024-04-16T19:47:32.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009089', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:25.473955Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006549', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:25.475565Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011470', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:25.477260Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009084', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:46:53.431985Z'}}] 2024-04-16T19:47:32.645 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009089', 1: '00000000000000000000:00000000000000006549', 2: '00000000000000000000:00000000000000011470', 3: '00000000000000000000:00000000000000009084'} 2024-04-16T19:47:32.645 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009089', 1: '00000000000000000000:00000000000000006549', 2: '00000000000000000000:00000000000000011470', 3: '00000000000000000000:00000000000000009084'} 2024-04-16T19:47:32.645 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T19:47:32.645 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T19:47:32.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.645 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:32.692 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:32.770 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000006842","last_update":"2024-04-16T19:47:03.407368Z"},{"marker":"00000000000000000000:00000000000000005149","last_update":"2024-04-16T19:47:03.409089Z"},{"marker":"00000000000000000000:00000000000000008384","last_update":"2024-04-16T19:47:03.411254Z"},{"marker":"00000000000000000000:00000000000000006701","last_update":"2024-04-16T19:47:03.413299Z"}] 2024-04-16T19:47:32.771 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000006842', 1: '00000000000000000000:00000000000000005149', 2: '00000000000000000000:00000000000000008384', 3: '00000000000000000000:00000000000000006701'} 2024-04-16T19:47:32.771 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:47:32.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:32.771 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:32.902 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006842","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.407368Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005149","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.409089Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008384","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.411254Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006701","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:47:03.413299Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:47:32Z"} 2024-04-16T19:47:32.902 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006842","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.407368Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005149","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.409089Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008384","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:03.411254Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006701","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T19:47:03.413299Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T19:47:32Z"} 2024-04-16T19:47:32.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:03.407368Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005149', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:03.409089Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008384', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:03.411254Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006701', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T19:47:03.413299Z'}}] 2024-04-16T19:47:32.902 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006842', 1: '00000000000000000000:00000000000000005149', 2: '00000000000000000000:00000000000000008384', 3: '00000000000000000000:00000000000000006701'} 2024-04-16T19:47:32.903 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006842', 1: '00000000000000000000:00000000000000005149', 2: '00000000000000000000:00000000000000008384', 3: '00000000000000000000:00000000000000006701'} 2024-04-16T19:47:32.903 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T19:47:32.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-04-16T19:47:32.905 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-42 2024-04-16T19:47:32.905 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:32.905 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:32.905 DEBUG:boto:Method: PUT 2024-04-16T19:47:32.905 DEBUG:boto:Path: /yrjefh-42/ 2024-04-16T19:47:32.905 DEBUG:boto:Data: 2024-04-16T19:47:32.905 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.906 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.906 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.906 DEBUG:boto:Params: {} 2024-04-16T19:47:32.906 DEBUG:boto:Token: None 2024-04-16T19:47:32.906 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-42/ 2024-04-16T19:47:32.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XwjLBJXGnOVzK2rlkhBYB6yuiCs= 2024-04-16T19:47:32.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XwjLBJXGnOVzK2rlkhBYB6yuiCs=', 'Content-Length': '0'} 2024-04-16T19:47:32.920 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ddef3bed92d590ef-00661ed5d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.920 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=yrjefh-43 2024-04-16T19:47:32.920 DEBUG:boto:path=/yrjefh-43/ 2024-04-16T19:47:32.920 DEBUG:boto:auth_path=/yrjefh-43/ 2024-04-16T19:47:32.920 DEBUG:boto:Method: PUT 2024-04-16T19:47:32.920 DEBUG:boto:Path: /yrjefh-43/ 2024-04-16T19:47:32.921 DEBUG:boto:Data: 2024-04-16T19:47:32.921 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:32.921 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.921 DEBUG:boto:Params: {} 2024-04-16T19:47:32.921 DEBUG:boto:Token: None 2024-04-16T19:47:32.921 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-43/ 2024-04-16T19:47:32.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i44kr8fn2sjb9+SOlMjxSE40/Vs= 2024-04-16T19:47:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i44kr8fn2sjb9+SOlMjxSE40/Vs=', 'Content-Length': '0'} 2024-04-16T19:47:32.948 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000069902450c641654a-00661ed5d4-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:32.949 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:32.949 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:32.949 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploads 2024-04-16T19:47:32.949 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploads 2024-04-16T19:47:32.949 DEBUG:boto:Method: POST 2024-04-16T19:47:32.949 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploads 2024-04-16T19:47:32.949 DEBUG:boto:Data: 2024-04-16T19:47:32.949 DEBUG:boto:Headers: {} 2024-04-16T19:47:32.949 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:32.949 DEBUG:boto:Port: 8000 2024-04-16T19:47:32.950 DEBUG:boto:Params: {} 2024-04-16T19:47:32.950 DEBUG:boto:Token: None 2024-04-16T19:47:32.950 DEBUG:boto:StringToSign: POST Tue, 16 Apr 2024 19:47:32 GMT /yrjefh-42/MULTIPART?uploads 2024-04-16T19:47:32.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YEf5VehVUiy5BUxCWydy+cm7Gog= 2024-04-16T19:47:32.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YEf5VehVUiy5BUxCWydy+cm7Gog=', 'Content-Length': '0'} 2024-04-16T19:47:33.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005122545f851a9bec-00661ed5d4-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '249'), ('Date', 'Tue, 16 Apr 2024 19:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.034 DEBUG:boto:b'yrjefh-42MULTIPART2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd' 2024-04-16T19:47:34.083 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.084 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.084 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=1 2024-04-16T19:47:34.084 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=1 2024-04-16T19:47:34.084 DEBUG:boto:Method: PUT 2024-04-16T19:47:34.084 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=1 2024-04-16T19:47:34.084 DEBUG:boto:Data: 2024-04-16T19:47:34.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-16T19:47:34.084 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.085 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.085 DEBUG:boto:Params: {} 2024-04-16T19:47:34.085 DEBUG:boto:Token: None 2024-04-16T19:47:34.085 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?partNumber=1&uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XzhJHiOMxQXYxRLeyPbJJbs+x3E= 2024-04-16T19:47:34.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XzhJHiOMxQXYxRLeyPbJJbs+x3E='} 2024-04-16T19:47:34.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000348160a961dd66a6-00661ed5d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.152 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.153 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.153 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=2 2024-04-16T19:47:34.153 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=2 2024-04-16T19:47:34.153 DEBUG:boto:Method: PUT 2024-04-16T19:47:34.153 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=2 2024-04-16T19:47:34.153 DEBUG:boto:Data: 2024-04-16T19:47:34.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-16T19:47:34.153 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.153 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.153 DEBUG:boto:Params: {} 2024-04-16T19:47:34.154 DEBUG:boto:Token: None 2024-04-16T19:47:34.154 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?partNumber=2&uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.154 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zjRxQhUK63pnY5qSqfW/rY3J0HY= 2024-04-16T19:47:34.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zjRxQhUK63pnY5qSqfW/rY3J0HY='} 2024-04-16T19:47:34.189 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c1b2dbcc843b095-00661ed5d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.207 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.207 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.207 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=3 2024-04-16T19:47:34.207 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=3 2024-04-16T19:47:34.207 DEBUG:boto:Method: PUT 2024-04-16T19:47:34.207 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=3 2024-04-16T19:47:34.207 DEBUG:boto:Data: 2024-04-16T19:47:34.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-16T19:47:34.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.208 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.208 DEBUG:boto:Params: {} 2024-04-16T19:47:34.208 DEBUG:boto:Token: None 2024-04-16T19:47:34.208 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?partNumber=3&uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/vUG12ItsFOVj+I1/eS/QaPy0g= 2024-04-16T19:47:34.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/vUG12ItsFOVj+I1/eS/QaPy0g='} 2024-04-16T19:47:34.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000622e1441f5b3199-00661ed5d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.256 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.256 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.256 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=4 2024-04-16T19:47:34.256 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=4 2024-04-16T19:47:34.256 DEBUG:boto:Method: PUT 2024-04-16T19:47:34.256 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd&partNumber=4 2024-04-16T19:47:34.256 DEBUG:boto:Data: 2024-04-16T19:47:34.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-16T19:47:34.257 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.257 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.257 DEBUG:boto:Params: {} 2024-04-16T19:47:34.257 DEBUG:boto:Token: None 2024-04-16T19:47:34.257 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?partNumber=4&uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dFHedGfuJ1xjh4e1Anq/MxFa7kE= 2024-04-16T19:47:34.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dFHedGfuJ1xjh4e1Anq/MxFa7kE='} 2024-04-16T19:47:34.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cea902c0b946b19c-00661ed5d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.288 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.288 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.288 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.288 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.288 DEBUG:boto:Method: GET 2024-04-16T19:47:34.288 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.288 DEBUG:boto:Data: 2024-04-16T19:47:34.288 DEBUG:boto:Headers: {} 2024-04-16T19:47:34.288 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.288 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.288 DEBUG:boto:Params: {} 2024-04-16T19:47:34.289 DEBUG:boto:Token: None 2024-04-16T19:47:34.289 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.289 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dbp+JQroKaZYoPmO5ih+KR9jedo= 2024-04-16T19:47:34.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dbp+JQroKaZYoPmO5ih+KR9jedo=', 'Content-Length': '0'} 2024-04-16T19:47:34.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d943780853f7eb1-00661ed5d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.335 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.335 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:34.335 DEBUG:boto:path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.335 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.335 DEBUG:boto:Method: POST 2024-04-16T19:47:34.335 DEBUG:boto:Path: /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.335 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-16T19:47:34.335 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-16T19:47:34.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:34.335 DEBUG:boto:Port: 8000 2024-04-16T19:47:34.335 DEBUG:boto:Params: {} 2024-04-16T19:47:34.336 DEBUG:boto:Token: None 2024-04-16T19:47:34.336 DEBUG:boto:StringToSign: POST text/xml Tue, 16 Apr 2024 19:47:34 GMT /yrjefh-42/MULTIPART?uploadId=2~zLskSiU4vLrPw-3NRMQSBihKWibSLLd 2024-04-16T19:47:34.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CMoC/JDjvVyb7XyK7oPILRMKfKI= 2024-04-16T19:47:34.336 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CMoC/JDjvVyb7XyK7oPILRMKfKI=', 'Content-Length': '460'} 2024-04-16T19:47:34.349 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038fd377c0c973b98-00661ed5d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '325'), ('Date', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:34.349 DEBUG:boto:http://smithi026.front.sepia.ceph.com:8000/yrjefh-42/MULTIPARTyrjefh-42MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-16T19:47:34.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:34.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:34.350 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:34.398 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:34.398 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:47:34.471 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296852.935758_551.1","last_update":"2024-04-16T19:47:32.935758Z"}] 2024-04-16T19:47:34.472 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296852.935758_551.1'} 2024-04-16T19:47:34.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:47:34.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:34.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:34.472 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:34.527 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:34.527 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:34.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:34.579+0000 7f25b7b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:34.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:34.580+0000 7f25b7b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:34.604 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296852.935758_551.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:32.935758Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:34Z"} 2024-04-16T19:47:34.604 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296852.935758_551.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:32.935758Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:34Z"} 2024-04-16T19:47:34.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296852.935758_551.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:32.935758Z', 'realm_epoch': 2}}] 2024-04-16T19:47:34.604 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296852.935758_551.1'} 2024-04-16T19:47:34.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296852.935758_551.1'} 2024-04-16T19:47:34.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:47:34.605 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-42 2024-04-16T19:47:34.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-42', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:34.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-42', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:34.605 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-42 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:34.715+0000 7ff9d6412800 1 waiting to reach incremental sync.. 2024-04-16T19:47:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:39.716+0000 7ff9d6412800 1 bucket sync caught up with source: 2024-04-16T19:47:39.720 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000006.47.6, , , , , ] 2024-04-16T19:47:39.720 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000006.47.6, , , , , ] 2024-04-16T19:47:39.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:39.716+0000 7ff9d6412800 0 bucket checkpoint complete 2024-04-16T19:47:39.744 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-42 2024-04-16T19:47:39.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-42', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:39.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-42', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:39.745 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-42 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:39.809 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:39.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:39.877+0000 7fdf3ae12800 1 waiting to reach incremental sync.. 2024-04-16T19:47:44.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:44.879+0000 7fdf3ae12800 1 bucket sync caught up with source: 2024-04-16T19:47:44.881 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.98.6, , , , , ] 2024-04-16T19:47:44.881 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.98.6, , , , , ] 2024-04-16T19:47:44.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:44.879+0000 7fdf3ae12800 0 bucket checkpoint complete 2024-04-16T19:47:44.908 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-42 zones={test-zone2, test-zone1} 2024-04-16T19:47:44.908 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.908 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.909 DEBUG:boto:Method: HEAD 2024-04-16T19:47:44.909 DEBUG:boto:Path: /yrjefh-42/ 2024-04-16T19:47:44.909 DEBUG:boto:Data: 2024-04-16T19:47:44.909 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.909 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:44.909 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.909 DEBUG:boto:Params: {} 2024-04-16T19:47:44.910 DEBUG:boto:Token: None 2024-04-16T19:47:44.910 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/ 2024-04-16T19:47:44.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TxVeExMsld3M0jsTT4+ls4ATF1Y= 2024-04-16T19:47:44.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TxVeExMsld3M0jsTT4+ls4ATF1Y=', 'Content-Length': '0'} 2024-04-16T19:47:44.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db54de5a54540720-00661ed5e0-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.914 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.914 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.914 DEBUG:boto:Method: HEAD 2024-04-16T19:47:44.914 DEBUG:boto:Path: /yrjefh-42/ 2024-04-16T19:47:44.914 DEBUG:boto:Data: 2024-04-16T19:47:44.914 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:44.915 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.915 DEBUG:boto:Params: {} 2024-04-16T19:47:44.915 DEBUG:boto:Token: None 2024-04-16T19:47:44.915 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/ 2024-04-16T19:47:44.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TxVeExMsld3M0jsTT4+ls4ATF1Y= 2024-04-16T19:47:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TxVeExMsld3M0jsTT4+ls4ATF1Y=', 'Content-Length': '0'} 2024-04-16T19:47:44.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093ae9a908993024b-00661ed5e0-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:47:44.919 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.919 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.919 DEBUG:boto:path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.919 DEBUG:boto:auth_path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.919 DEBUG:boto:Method: GET 2024-04-16T19:47:44.919 DEBUG:boto:Path: /yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.920 DEBUG:boto:Data: 2024-04-16T19:47:44.920 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.920 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:44.920 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.920 DEBUG:boto:Params: {} 2024-04-16T19:47:44.920 DEBUG:boto:Token: None 2024-04-16T19:47:44.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/?versions 2024-04-16T19:47:44.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM= 2024-04-16T19:47:44.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM=', 'Content-Length': '0'} 2024-04-16T19:47:44.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29819811507ba61-00661ed5e0-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.924 DEBUG:boto:b'yrjefh-42999falseMULTIPARTnulltrue2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:44.924 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-16T19:47:44.924 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:47:44.925 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.925 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.925 DEBUG:boto:path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.925 DEBUG:boto:auth_path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.925 DEBUG:boto:Method: GET 2024-04-16T19:47:44.925 DEBUG:boto:Path: /yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.925 DEBUG:boto:Data: 2024-04-16T19:47:44.926 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.926 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:44.926 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.926 DEBUG:boto:Params: {} 2024-04-16T19:47:44.926 DEBUG:boto:Token: None 2024-04-16T19:47:44.926 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/?versions 2024-04-16T19:47:44.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM= 2024-04-16T19:47:44.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM=', 'Content-Length': '0'} 2024-04-16T19:47:44.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ebc6c902e0668fb-00661ed5e0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.929 DEBUG:boto:b'yrjefh-42999falseMULTIPARTnulltrue2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:44.930 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-16T19:47:44.930 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.930 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.930 DEBUG:boto:path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.930 DEBUG:boto:auth_path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.930 DEBUG:boto:Method: GET 2024-04-16T19:47:44.931 DEBUG:boto:Path: /yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.931 DEBUG:boto:Data: 2024-04-16T19:47:44.931 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:44.931 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.931 DEBUG:boto:Params: {} 2024-04-16T19:47:44.931 DEBUG:boto:Token: None 2024-04-16T19:47:44.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/?versions 2024-04-16T19:47:44.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM= 2024-04-16T19:47:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM=', 'Content-Length': '0'} 2024-04-16T19:47:44.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001305cef0f89379d3-00661ed5e0-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.975 DEBUG:boto:b'yrjefh-42999falseMULTIPARTnulltrue2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:44.975 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:44.975 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:44.975 DEBUG:boto:path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.976 DEBUG:boto:auth_path=/yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.976 DEBUG:boto:Method: GET 2024-04-16T19:47:44.976 DEBUG:boto:Path: /yrjefh-42/?versions&max-keys=999 2024-04-16T19:47:44.976 DEBUG:boto:Data: 2024-04-16T19:47:44.976 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.976 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:44.976 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.976 DEBUG:boto:Params: {} 2024-04-16T19:47:44.977 DEBUG:boto:Token: None 2024-04-16T19:47:44.977 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/?versions 2024-04-16T19:47:44.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM= 2024-04-16T19:47:44.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jJubzy+vhMGRsp3SBXgpNa7CdrM=', 'Content-Length': '0'} 2024-04-16T19:47:44.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f79336f9e5b904f-00661ed5e0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:44.980 DEBUG:boto:b'yrjefh-42999falseMULTIPARTnulltrue2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:44.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-16T19:47:44.981 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:44.981 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:44.981 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:44.981 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:44.981 DEBUG:boto:Method: GET 2024-04-16T19:47:44.981 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:44.982 DEBUG:boto:Data: 2024-04-16T19:47:44.982 DEBUG:boto:Headers: {} 2024-04-16T19:47:44.982 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:44.982 DEBUG:boto:Port: 8000 2024-04-16T19:47:44.982 DEBUG:boto:Params: {} 2024-04-16T19:47:44.982 DEBUG:boto:Token: None 2024-04-16T19:47:44.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:44 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:44.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AJQVzzPNZ+8ZInYLhctR5VXnt6M= 2024-04-16T19:47:44.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AJQVzzPNZ+8ZInYLhctR5VXnt6M=', 'Content-Length': '0'} 2024-04-16T19:47:44.987 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-42:b482a2a9-ef36-488b-9666-ada656453328.4812.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:38 GMT'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009fac56a358292516-00661ed5e0-4911-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.068 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.068 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.069 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.069 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.069 DEBUG:boto:Method: GET 2024-04-16T19:47:45.069 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.069 DEBUG:boto:Data: 2024-04-16T19:47:45.069 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.069 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.069 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.069 DEBUG:boto:Params: {} 2024-04-16T19:47:45.069 DEBUG:boto:Token: None 2024-04-16T19:47:45.070 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs= 2024-04-16T19:47:45.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs=', 'Content-Length': '0'} 2024-04-16T19:47:45.073 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000006b0b2d05a9468a2-00661ed5e1-4812-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.168 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.168 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.169 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.169 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.169 DEBUG:boto:Method: HEAD 2024-04-16T19:47:45.169 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.169 DEBUG:boto:Data: 2024-04-16T19:47:45.169 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.169 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.169 DEBUG:boto:Params: {} 2024-04-16T19:47:45.169 DEBUG:boto:Token: None 2024-04-16T19:47:45.170 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.170 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xgcP/UIh3wMM1ADupr7vZ/siquk= 2024-04-16T19:47:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xgcP/UIh3wMM1ADupr7vZ/siquk=', 'Content-Length': '0'} 2024-04-16T19:47:45.172 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-42:b482a2a9-ef36-488b-9666-ada656453328.4812.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:38 GMT'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000730967e410e37225-00661ed5e1-4911-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.172 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.173 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.173 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.173 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.173 DEBUG:boto:Method: HEAD 2024-04-16T19:47:45.173 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.173 DEBUG:boto:Data: 2024-04-16T19:47:45.173 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.173 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.173 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.173 DEBUG:boto:Params: {} 2024-04-16T19:47:45.173 DEBUG:boto:Token: None 2024-04-16T19:47:45.174 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xgcP/UIh3wMM1ADupr7vZ/siquk= 2024-04-16T19:47:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xgcP/UIh3wMM1ADupr7vZ/siquk=', 'Content-Length': '0'} 2024-04-16T19:47:45.176 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000064bc68b0080a0272-00661ed5e1-4812-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-16T19:47:45.176 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.176 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.177 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.177 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.177 DEBUG:boto:Method: GET 2024-04-16T19:47:45.177 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.177 DEBUG:boto:Data: 2024-04-16T19:47:45.177 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.177 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.177 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.177 DEBUG:boto:Params: {} 2024-04-16T19:47:45.177 DEBUG:boto:Token: None 2024-04-16T19:47:45.177 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.178 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs= 2024-04-16T19:47:45.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs=', 'Content-Length': '0'} 2024-04-16T19:47:45.180 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-42:b482a2a9-ef36-488b-9666-ada656453328.4812.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:38 GMT'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000cd54387873415f6f-00661ed5e1-4911-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.262 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.262 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.262 DEBUG:boto:path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.262 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.262 DEBUG:boto:Method: GET 2024-04-16T19:47:45.263 DEBUG:boto:Path: /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.263 DEBUG:boto:Data: 2024-04-16T19:47:45.263 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.263 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.263 DEBUG:boto:Params: {} 2024-04-16T19:47:45.263 DEBUG:boto:Token: None 2024-04-16T19:47:45.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART?versionId=null 2024-04-16T19:47:45.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs= 2024-04-16T19:47:45.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qmil6YlEhMERKoxPGokyviV2cCs=', 'Content-Length': '0'} 2024-04-16T19:47:45.267 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001093c34067a0c4cc-00661ed5e1-4812-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.352 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:45.352 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:45.352 DEBUG:boto:path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.352 DEBUG:boto:auth_path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.353 DEBUG:boto:Method: GET 2024-04-16T19:47:45.353 DEBUG:boto:Path: /yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.353 DEBUG:boto:Data: 2024-04-16T19:47:45.353 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.353 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.353 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.353 DEBUG:boto:Params: {} 2024-04-16T19:47:45.353 DEBUG:boto:Token: None 2024-04-16T19:47:45.353 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/ 2024-04-16T19:47:45.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY= 2024-04-16T19:47:45.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY=', 'Content-Length': '0'} 2024-04-16T19:47:45.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084789478fed72966-00661ed5e1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.356 DEBUG:boto:b'yrjefh-42MULTIPART1000falseMULTIPART2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:45.357 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:45.357 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:45.357 DEBUG:boto:path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.357 DEBUG:boto:auth_path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.357 DEBUG:boto:Method: GET 2024-04-16T19:47:45.357 DEBUG:boto:Path: /yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.357 DEBUG:boto:Data: 2024-04-16T19:47:45.357 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.357 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.357 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.357 DEBUG:boto:Params: {} 2024-04-16T19:47:45.358 DEBUG:boto:Token: None 2024-04-16T19:47:45.358 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/ 2024-04-16T19:47:45.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY= 2024-04-16T19:47:45.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY=', 'Content-Length': '0'} 2024-04-16T19:47:45.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f67e8caa0425d0-00661ed5e1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.360 DEBUG:boto:b'yrjefh-42MULTIPART1000falseMULTIPART2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:45.361 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.361 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.361 DEBUG:boto:Method: HEAD 2024-04-16T19:47:45.361 DEBUG:boto:Path: /yrjefh-42/MULTIPART 2024-04-16T19:47:45.361 DEBUG:boto:Data: 2024-04-16T19:47:45.361 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.361 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.361 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.361 DEBUG:boto:Params: {} 2024-04-16T19:47:45.361 DEBUG:boto:Token: None 2024-04-16T19:47:45.362 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART 2024-04-16T19:47:45.362 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q5p6350xssB8TKxZxbMe5IPw09I= 2024-04-16T19:47:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q5p6350xssB8TKxZxbMe5IPw09I=', 'Content-Length': '0'} 2024-04-16T19:47:45.364 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-42:b482a2a9-ef36-488b-9666-ada656453328.4812.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:38 GMT'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000317c55936dbbed31-00661ed5e1-4911-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.364 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.364 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.364 DEBUG:boto:Method: HEAD 2024-04-16T19:47:45.364 DEBUG:boto:Path: /yrjefh-42/MULTIPART 2024-04-16T19:47:45.364 DEBUG:boto:Data: 2024-04-16T19:47:45.364 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.365 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.365 DEBUG:boto:Params: {} 2024-04-16T19:47:45.365 DEBUG:boto:Token: None 2024-04-16T19:47:45.365 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART 2024-04-16T19:47:45.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q5p6350xssB8TKxZxbMe5IPw09I= 2024-04-16T19:47:45.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q5p6350xssB8TKxZxbMe5IPw09I=', 'Content-Length': '0'} 2024-04-16T19:47:45.367 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000718e97ec4af7ac96-00661ed5e1-4812-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-16T19:47:45.367 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.367 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.367 DEBUG:boto:Method: GET 2024-04-16T19:47:45.367 DEBUG:boto:Path: /yrjefh-42/MULTIPART 2024-04-16T19:47:45.367 DEBUG:boto:Data: 2024-04-16T19:47:45.368 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.368 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.368 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.368 DEBUG:boto:Params: {} 2024-04-16T19:47:45.368 DEBUG:boto:Token: None 2024-04-16T19:47:45.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART 2024-04-16T19:47:45.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h9IxTeOl7Kk7c827F97GFt+QZaw= 2024-04-16T19:47:45.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h9IxTeOl7Kk7c827F97GFt+QZaw=', 'Content-Length': '0'} 2024-04-16T19:47:45.371 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-42:b482a2a9-ef36-488b-9666-ada656453328.4812.11'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:38 GMT'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000174cd9700f7b83ec-00661ed5e1-4911-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.448 DEBUG:boto:path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.449 DEBUG:boto:auth_path=/yrjefh-42/MULTIPART 2024-04-16T19:47:45.449 DEBUG:boto:Method: GET 2024-04-16T19:47:45.449 DEBUG:boto:Path: /yrjefh-42/MULTIPART 2024-04-16T19:47:45.449 DEBUG:boto:Data: 2024-04-16T19:47:45.449 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.449 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.449 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.449 DEBUG:boto:Params: {} 2024-04-16T19:47:45.449 DEBUG:boto:Token: None 2024-04-16T19:47:45.450 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/MULTIPART 2024-04-16T19:47:45.450 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h9IxTeOl7Kk7c827F97GFt+QZaw= 2024-04-16T19:47:45.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h9IxTeOl7Kk7c827F97GFt+QZaw=', 'Content-Length': '0'} 2024-04-16T19:47:45.453 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000dcf9f287475ee4d2-00661ed5e1-4812-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.535 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:45.535 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:45.535 DEBUG:boto:path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.535 DEBUG:boto:auth_path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.535 DEBUG:boto:Method: GET 2024-04-16T19:47:45.535 DEBUG:boto:Path: /yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.535 DEBUG:boto:Data: 2024-04-16T19:47:45.536 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:45.536 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.536 DEBUG:boto:Params: {} 2024-04-16T19:47:45.536 DEBUG:boto:Token: None 2024-04-16T19:47:45.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/ 2024-04-16T19:47:45.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY= 2024-04-16T19:47:45.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY=', 'Content-Length': '0'} 2024-04-16T19:47:45.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d23fa1795625b8b-00661ed5e1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.539 DEBUG:boto:b'yrjefh-42MULTIPART1000falseMULTIPART2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:45.539 DEBUG:boto:path=/yrjefh-42/ 2024-04-16T19:47:45.539 DEBUG:boto:auth_path=/yrjefh-42/ 2024-04-16T19:47:45.539 DEBUG:boto:path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.539 DEBUG:boto:auth_path=/yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.540 DEBUG:boto:Method: GET 2024-04-16T19:47:45.540 DEBUG:boto:Path: /yrjefh-42/?prefix=MULTIPART 2024-04-16T19:47:45.540 DEBUG:boto:Data: 2024-04-16T19:47:45.540 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.540 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.540 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.540 DEBUG:boto:Params: {} 2024-04-16T19:47:45.540 DEBUG:boto:Token: None 2024-04-16T19:47:45.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-42/ 2024-04-16T19:47:45.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY= 2024-04-16T19:47:45.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EA4tbGNVOhJFlr+O9QJGPAX68EY=', 'Content-Length': '0'} 2024-04-16T19:47:45.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002372eba60dceb7a4-00661ed5e1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.543 DEBUG:boto:b'yrjefh-42MULTIPART1000falseMULTIPART2024-04-16T19:47:34.339Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDRGW11111111111111111Normal' 2024-04-16T19:47:45.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-42 zones={test-zone2, test-zone1} 2024-04-16T19:47:45.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2024-04-16T19:47:45.544 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-44 2024-04-16T19:47:45.544 DEBUG:boto:path=/yrjefh-44/ 2024-04-16T19:47:45.544 DEBUG:boto:auth_path=/yrjefh-44/ 2024-04-16T19:47:45.545 DEBUG:boto:Method: PUT 2024-04-16T19:47:45.545 DEBUG:boto:Path: /yrjefh-44/ 2024-04-16T19:47:45.545 DEBUG:boto:Data: 2024-04-16T19:47:45.545 DEBUG:boto:Headers: {} 2024-04-16T19:47:45.545 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.545 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.545 DEBUG:boto:Params: {} 2024-04-16T19:47:45.545 DEBUG:boto:Token: None 2024-04-16T19:47:45.545 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:47:45 GMT /yrjefh-44/ 2024-04-16T19:47:45.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:edYZehFj8J5RUaia8BlkQrnmOuk= 2024-04-16T19:47:45.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:edYZehFj8J5RUaia8BlkQrnmOuk=', 'Content-Length': '0'} 2024-04-16T19:47:45.562 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004334c919702b37c1-00661ed5e1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.562 DEBUG:boto:path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:45.563 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:45.563 DEBUG:boto:Method: PUT 2024-04-16T19:47:45.563 DEBUG:boto:Path: /yrjefh-44/testobj-sse-c 2024-04-16T19:47:45.563 DEBUG:boto:Data: 2024-04-16T19:47:45.563 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:47:45.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.563 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.563 DEBUG:boto:Params: {} 2024-04-16T19:47:45.563 DEBUG:boto:Token: None 2024-04-16T19:47:45.563 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 16 Apr 2024 19:47:45 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /yrjefh-44/testobj-sse-c 2024-04-16T19:47:45.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:prWmG84w6Pb+zLdF1Yp/tD6Q22g= 2024-04-16T19:47:45.564 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:prWmG84w6Pb+zLdF1Yp/tD6Q22g='} 2024-04-16T19:47:45.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000014a793a3b8fdf03e-00661ed5e1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.607 DEBUG:boto:path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:45.607 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:45.607 DEBUG:boto:Method: PUT 2024-04-16T19:47:45.607 DEBUG:boto:Path: /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:45.607 DEBUG:boto:Data: 2024-04-16T19:47:45.607 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-16T19:47:45.608 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:45.608 DEBUG:boto:Port: 8000 2024-04-16T19:47:45.608 DEBUG:boto:Params: {} 2024-04-16T19:47:45.608 DEBUG:boto:Token: None 2024-04-16T19:47:45.608 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 16 Apr 2024 19:47:45 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:45.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Bf2xAXfp44sfo0qiQRkuCKgHXBA= 2024-04-16T19:47:45.608 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:45 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Bf2xAXfp44sfo0qiQRkuCKgHXBA='} 2024-04-16T19:47:45.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eXJqZWZoLTQ0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000cf9f83d624960350-00661ed5e1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:45.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:45.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:45.632 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:45.683 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:45.736 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:47:45.758 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296865.559312_555.1","last_update":"2024-04-16T19:47:45.559312Z"}] 2024-04-16T19:47:45.759 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296865.559312_555.1'} 2024-04-16T19:47:45.759 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:47:45.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:45.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:45.759 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:45.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:45.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:45.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:45.862+0000 7ff70069d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:45.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:45.864+0000 7ff70069d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:45.887 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296865.559312_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:45.559312Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:45Z"} 2024-04-16T19:47:45.887 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296865.559312_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:45.559312Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:45Z"} 2024-04-16T19:47:45.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296865.559312_555.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:45.559312Z', 'realm_epoch': 2}}] 2024-04-16T19:47:45.888 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296865.559312_555.1'} 2024-04-16T19:47:45.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296865.559312_555.1'} 2024-04-16T19:47:45.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:47:45.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-44', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:45.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-44', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:45.888 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-44 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:45.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:45.997+0000 7f8df5b2a800 1 waiting to reach incremental sync.. 2024-04-16T19:47:51.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:50.999+0000 7f8df5b2a800 1 bucket sync caught up with source: 2024-04-16T19:47:51.001 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.47.6, , , , 00000000001.22.6, , , , , ] 2024-04-16T19:47:51.002 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.22.6, , , , , ] 2024-04-16T19:47:51.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:50.999+0000 7f8df5b2a800 0 bucket checkpoint complete 2024-04-16T19:47:51.027 DEBUG:boto:path=/yrjefh-44/ 2024-04-16T19:47:51.027 DEBUG:boto:auth_path=/yrjefh-44/ 2024-04-16T19:47:51.027 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.027 DEBUG:boto:Path: /yrjefh-44/ 2024-04-16T19:47:51.027 DEBUG:boto:Data: 2024-04-16T19:47:51.027 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.027 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:51.027 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.027 DEBUG:boto:Params: {} 2024-04-16T19:47:51.028 DEBUG:boto:Token: None 2024-04-16T19:47:51.028 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-44/ 2024-04-16T19:47:51.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V5d6Q3st/dbIdZ0/7l3p1G+WMrI= 2024-04-16T19:47:51.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V5d6Q3st/dbIdZ0/7l3p1G+WMrI=', 'Content-Length': '0'} 2024-04-16T19:47:51.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7b62f74b4f50e64-00661ed5e7-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.031 DEBUG:boto:path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.031 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.031 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.031 DEBUG:boto:Path: /yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.031 DEBUG:boto:Data: 2024-04-16T19:47:51.032 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-16T19:47:51.032 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:51.032 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.032 DEBUG:boto:Params: {} 2024-04-16T19:47:51.032 DEBUG:boto:Token: None 2024-04-16T19:47:51.032 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNce7yQEN/XjlTImBNW02aj+S4k= 2024-04-16T19:47:51.033 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNce7yQEN/XjlTImBNW02aj+S4k=', 'Content-Length': '0'} 2024-04-16T19:47:51.034 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-44:b482a2a9-ef36-488b-9666-ada656453328.4812.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:48 GMT'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000076785f31d785a7c-00661ed5e7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.035 DEBUG:boto:path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.035 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.035 DEBUG:boto:Method: GET 2024-04-16T19:47:51.035 DEBUG:boto:Path: /yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.035 DEBUG:boto:Data: 2024-04-16T19:47:51.035 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-16T19:47:51.035 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:51.035 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.036 DEBUG:boto:Params: {} 2024-04-16T19:47:51.036 DEBUG:boto:Token: None 2024-04-16T19:47:51.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:51 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /yrjefh-44/testobj-sse-c 2024-04-16T19:47:51.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YlBfHW6oqRuvFrg8F5ZtVVlG7Fo= 2024-04-16T19:47:51.036 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YlBfHW6oqRuvFrg8F5ZtVVlG7Fo=', 'Content-Length': '0'} 2024-04-16T19:47:51.038 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-44:b482a2a9-ef36-488b-9666-ada656453328.4812.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:48 GMT'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000e376af5dfc0f86a0-00661ed5e7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.083 DEBUG:boto:path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.083 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.083 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.083 DEBUG:boto:Path: /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.083 DEBUG:boto:Data: 2024-04-16T19:47:51.083 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:51.083 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.083 DEBUG:boto:Params: {} 2024-04-16T19:47:51.083 DEBUG:boto:Token: None 2024-04-16T19:47:51.084 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HxvRomDILoZKt4bd4BcgPIjoTus= 2024-04-16T19:47:51.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HxvRomDILoZKt4bd4BcgPIjoTus=', 'Content-Length': '0'} 2024-04-16T19:47:51.086 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-44:b482a2a9-ef36-488b-9666-ada656453328.4812.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:48 GMT'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000006cd456e89ddbd094-00661ed5e7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.086 DEBUG:boto:path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.086 DEBUG:boto:auth_path=/yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.087 DEBUG:boto:Method: GET 2024-04-16T19:47:51.087 DEBUG:boto:Path: /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.087 DEBUG:boto:Data: 2024-04-16T19:47:51.087 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.087 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:47:51.087 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.087 DEBUG:boto:Params: {} 2024-04-16T19:47:51.087 DEBUG:boto:Token: None 2024-04-16T19:47:51.087 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-44/testobj-sse-kms 2024-04-16T19:47:51.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M5NSNTFLMo5JPlw1fVpA1ShzQ5E= 2024-04-16T19:47:51.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M5NSNTFLMo5JPlw1fVpA1ShzQ5E=', 'Content-Length': '0'} 2024-04-16T19:47:51.090 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-44:b482a2a9-ef36-488b-9666-ada656453328.4812.13'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:48 GMT'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000004d550c6f5ff3fca6-00661ed5e7-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2024-04-16T19:47:51.091 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-45 2024-04-16T19:47:51.091 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:47:51.092 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:47:51.092 DEBUG:boto:Method: PUT 2024-04-16T19:47:51.092 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:47:51.092 DEBUG:boto:Data: 2024-04-16T19:47:51.092 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.092 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.092 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.092 DEBUG:boto:Params: {} 2024-04-16T19:47:51.092 DEBUG:boto:Token: None 2024-04-16T19:47:51.092 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/ 2024-04-16T19:47:51.093 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wswXrl6teOo4d+jN5suvdp4NGw4= 2024-04-16T19:47:51.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wswXrl6teOo4d+jN5suvdp4NGw4=', 'Content-Length': '0'} 2024-04-16T19:47:51.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b32f3c1b776c376f-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.110 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:47:51.110 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:47:51.110 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.110 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:47:51.110 DEBUG:boto:Data: 2024-04-16T19:47:51.111 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.111 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.111 DEBUG:boto:Params: {} 2024-04-16T19:47:51.111 DEBUG:boto:Token: None 2024-04-16T19:47:51.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/ 2024-04-16T19:47:51.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM= 2024-04-16T19:47:51.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM=', 'Content-Length': '0'} 2024-04-16T19:47:51.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000353bb87b72ab4019-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.114 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:47:51.115 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:47:51.115 DEBUG:boto:Method: PUT 2024-04-16T19:47:51.115 DEBUG:boto:Path: /yrjefh-45/a 2024-04-16T19:47:51.115 DEBUG:boto:Data: 2024-04-16T19:47:51.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:47:51.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.115 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.115 DEBUG:boto:Params: {} 2024-04-16T19:47:51.115 DEBUG:boto:Token: None 2024-04-16T19:47:51.115 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/a 2024-04-16T19:47:51.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qg5bE9o12S0Wj1QDuratWwHsoOQ= 2024-04-16T19:47:51.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qg5bE9o12S0Wj1QDuratWwHsoOQ='} 2024-04-16T19:47:51.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e478eec06d24fffc-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.122 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:47:51.122 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:47:51.122 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.123 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:47:51.123 DEBUG:boto:Data: 2024-04-16T19:47:51.123 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.123 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.123 DEBUG:boto:Params: {} 2024-04-16T19:47:51.123 DEBUG:boto:Token: None 2024-04-16T19:47:51.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/ 2024-04-16T19:47:51.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM= 2024-04-16T19:47:51.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM=', 'Content-Length': '0'} 2024-04-16T19:47:51.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3d88153e6c67602-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.126 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:47:51.126 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:47:51.126 DEBUG:boto:Method: PUT 2024-04-16T19:47:51.126 DEBUG:boto:Path: /yrjefh-45/b 2024-04-16T19:47:51.126 DEBUG:boto:Data: 2024-04-16T19:47:51.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:47:51.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.127 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.127 DEBUG:boto:Params: {} 2024-04-16T19:47:51.127 DEBUG:boto:Token: None 2024-04-16T19:47:51.127 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/b 2024-04-16T19:47:51.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o2gaQKAn6a+KLNhFdZjuTBuqtSI= 2024-04-16T19:47:51.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o2gaQKAn6a+KLNhFdZjuTBuqtSI='} 2024-04-16T19:47:51.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2351aecb0249870-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.171 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:47:51.171 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:47:51.171 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.171 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:47:51.171 DEBUG:boto:Data: 2024-04-16T19:47:51.171 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.172 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.172 DEBUG:boto:Params: {} 2024-04-16T19:47:51.172 DEBUG:boto:Token: None 2024-04-16T19:47:51.172 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/ 2024-04-16T19:47:51.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM= 2024-04-16T19:47:51.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM=', 'Content-Length': '0'} 2024-04-16T19:47:51.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b7e23d11ecc03ea-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.175 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:47:51.175 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:47:51.175 DEBUG:boto:Method: PUT 2024-04-16T19:47:51.175 DEBUG:boto:Path: /yrjefh-45/c 2024-04-16T19:47:51.175 DEBUG:boto:Data: 2024-04-16T19:47:51.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:47:51.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.175 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.176 DEBUG:boto:Params: {} 2024-04-16T19:47:51.176 DEBUG:boto:Token: None 2024-04-16T19:47:51.176 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/c 2024-04-16T19:47:51.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rrdwjectPPf8zGI50/xbvsttymU= 2024-04-16T19:47:51.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rrdwjectPPf8zGI50/xbvsttymU='} 2024-04-16T19:47:51.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017f11fd9cf001d39-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.183 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:47:51.183 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:47:51.183 DEBUG:boto:Method: HEAD 2024-04-16T19:47:51.183 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:47:51.183 DEBUG:boto:Data: 2024-04-16T19:47:51.184 DEBUG:boto:Headers: {} 2024-04-16T19:47:51.184 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.184 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.184 DEBUG:boto:Params: {} 2024-04-16T19:47:51.184 DEBUG:boto:Token: None 2024-04-16T19:47:51.184 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/ 2024-04-16T19:47:51.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM= 2024-04-16T19:47:51.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wp45wxLRnHJ3U0vRTawTmp+ChM=', 'Content-Length': '0'} 2024-04-16T19:47:51.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1200782f71ca5ae-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.187 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:47:51.187 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:47:51.187 DEBUG:boto:Method: PUT 2024-04-16T19:47:51.187 DEBUG:boto:Path: /yrjefh-45/d 2024-04-16T19:47:51.188 DEBUG:boto:Data: 2024-04-16T19:47:51.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:47:51.188 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:47:51.188 DEBUG:boto:Port: 8000 2024-04-16T19:47:51.188 DEBUG:boto:Params: {} 2024-04-16T19:47:51.188 DEBUG:boto:Token: None 2024-04-16T19:47:51.188 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:47:51 GMT /yrjefh-45/d 2024-04-16T19:47:51.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwBjIRdCqiIxv6g+GbZWH8PZ2gw= 2024-04-16T19:47:51.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:47:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwBjIRdCqiIxv6g+GbZWH8PZ2gw='} 2024-04-16T19:47:51.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9f75273b0b2b202-00661ed5e7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:47:51.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:51.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:51.232 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:51.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:51.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:51.341 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:47:51.366 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296871.106894_560.1","last_update":"2024-04-16T19:47:51.106894Z"}] 2024-04-16T19:47:51.366 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296871.106894_560.1'} 2024-04-16T19:47:51.367 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:47:51.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:51.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:51.367 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:51.422 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:51.422 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:51.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:51.475+0000 7f6e1de3c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:51.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:51.476+0000 7f6e1de3c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:51.500 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296871.098255_557.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:51.098255Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:51Z"} 2024-04-16T19:47:51.500 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296871.098255_557.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:51.098255Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:51Z"} 2024-04-16T19:47:51.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296871.098255_557.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:51.098255Z', 'realm_epoch': 2}}] 2024-04-16T19:47:51.501 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296871.106894_560.1'} 2024-04-16T19:47:51.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296871.098255_557.1'} 2024-04-16T19:47:51.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713296871.106894_560.1 target=1_1713296871.098255_557.1 2024-04-16T19:47:56.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:56.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:47:56.503 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:47:56.559 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:56.559 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:56.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:56.611+0000 7fa3a1c3c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:56.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:56.612+0000 7fa3a1c3c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:47:56.636 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296871.106894_560.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:51.106894Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:56Z"} 2024-04-16T19:47:56.636 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296871.106894_560.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:47:51.106894Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:47:56Z"} 2024-04-16T19:47:56.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296871.106894_560.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:47:51.106894Z', 'realm_epoch': 2}}] 2024-04-16T19:47:56.637 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296871.106894_560.1'} 2024-04-16T19:47:56.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296871.106894_560.1'} 2024-04-16T19:47:56.637 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:47:56.637 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-45 2024-04-16T19:47:56.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-45', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:56.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-45', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:56.638 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-45 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:56.693 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:56.693 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:56.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:56.748+0000 7ff5af74b800 1 bucket sync caught up with source: 2024-04-16T19:47:56.750 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.29.6, , , , 00000000001.18.6, , , 00000000001.59.6, , 00000000001.25.6] 2024-04-16T19:47:56.750 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.29.6, , , , 00000000001.18.6, , , 00000000001.59.6, , 00000000001.25.6] 2024-04-16T19:47:56.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:47:56.748+0000 7ff5af74b800 0 bucket checkpoint complete 2024-04-16T19:47:56.770 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-45 2024-04-16T19:47:56.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-45', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:56.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-45', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:47:56.770 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-45 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:47:56.832 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:47:56.833 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:47:56.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:47:56.890+0000 7f5ed4743800 1 waiting to reach incremental sync.. 2024-04-16T19:48:01.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:01.892+0000 7f5ed4743800 1 bucket sync caught up with source: 2024-04-16T19:48:01.895 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.139.6, , , , 00000000001.82.6, , , 00000000001.194.6, , 00000000001.116.6] 2024-04-16T19:48:01.895 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.139.6, , , , 00000000001.82.6, , , 00000000001.194.6, , 00000000001.116.6] 2024-04-16T19:48:01.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:01.892+0000 7f5ed4743800 0 bucket checkpoint complete 2024-04-16T19:48:01.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-45 zones={test-zone2, test-zone1} 2024-04-16T19:48:01.922 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.922 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.922 DEBUG:boto:Method: HEAD 2024-04-16T19:48:01.923 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:48:01.923 DEBUG:boto:Data: 2024-04-16T19:48:01.923 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:01.923 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.923 DEBUG:boto:Params: {} 2024-04-16T19:48:01.923 DEBUG:boto:Token: None 2024-04-16T19:48:01.923 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/ 2024-04-16T19:48:01.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7O24Vuvjkb0yHG5kRzPtZNNHs/I= 2024-04-16T19:48:01.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7O24Vuvjkb0yHG5kRzPtZNNHs/I=', 'Content-Length': '0'} 2024-04-16T19:48:01.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a176535f38e06944-00661ed5f1-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.927 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.927 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.927 DEBUG:boto:Method: HEAD 2024-04-16T19:48:01.928 DEBUG:boto:Path: /yrjefh-45/ 2024-04-16T19:48:01.928 DEBUG:boto:Data: 2024-04-16T19:48:01.928 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.928 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:01.928 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.929 DEBUG:boto:Params: {} 2024-04-16T19:48:01.929 DEBUG:boto:Token: None 2024-04-16T19:48:01.929 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/ 2024-04-16T19:48:01.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7O24Vuvjkb0yHG5kRzPtZNNHs/I= 2024-04-16T19:48:01.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7O24Vuvjkb0yHG5kRzPtZNNHs/I=', 'Content-Length': '0'} 2024-04-16T19:48:01.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5c1e96884815d74-00661ed5f1-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:48:01.932 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.932 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.932 DEBUG:boto:path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.932 DEBUG:boto:auth_path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.932 DEBUG:boto:Method: GET 2024-04-16T19:48:01.932 DEBUG:boto:Path: /yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.933 DEBUG:boto:Data: 2024-04-16T19:48:01.933 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.933 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:01.933 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.933 DEBUG:boto:Params: {} 2024-04-16T19:48:01.933 DEBUG:boto:Token: None 2024-04-16T19:48:01.933 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/?versions 2024-04-16T19:48:01.933 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A= 2024-04-16T19:48:01.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A=', 'Content-Length': '0'} 2024-04-16T19:48:01.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a5b76f5ac51879-00661ed5f1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.937 DEBUG:boto:b'yrjefh-45999falseanulltrue2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:01.938 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:48:01.938 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:48:01.938 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:48:01.938 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:48:01.938 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:48:01.938 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.939 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.939 DEBUG:boto:path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.939 DEBUG:boto:auth_path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.939 DEBUG:boto:Method: GET 2024-04-16T19:48:01.939 DEBUG:boto:Path: /yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.939 DEBUG:boto:Data: 2024-04-16T19:48:01.939 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:01.939 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.939 DEBUG:boto:Params: {} 2024-04-16T19:48:01.940 DEBUG:boto:Token: None 2024-04-16T19:48:01.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/?versions 2024-04-16T19:48:01.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A= 2024-04-16T19:48:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A=', 'Content-Length': '0'} 2024-04-16T19:48:01.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d8bdf3dbdd26b3-00661ed5f1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.943 DEBUG:boto:b'yrjefh-45999falseanulltrue2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:01.944 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:48:01.944 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:48:01.944 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:48:01.944 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:48:01.944 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.944 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.944 DEBUG:boto:path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.945 DEBUG:boto:auth_path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.945 DEBUG:boto:Method: GET 2024-04-16T19:48:01.945 DEBUG:boto:Path: /yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.945 DEBUG:boto:Data: 2024-04-16T19:48:01.945 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.945 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:01.945 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.945 DEBUG:boto:Params: {} 2024-04-16T19:48:01.945 DEBUG:boto:Token: None 2024-04-16T19:48:01.946 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/?versions 2024-04-16T19:48:01.946 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A= 2024-04-16T19:48:01.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A=', 'Content-Length': '0'} 2024-04-16T19:48:01.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043572ff514bf877d-00661ed5f1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.948 DEBUG:boto:b'yrjefh-45999falseanulltrue2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:01.949 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:01.949 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:01.949 DEBUG:boto:path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.949 DEBUG:boto:auth_path=/yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.949 DEBUG:boto:Method: GET 2024-04-16T19:48:01.949 DEBUG:boto:Path: /yrjefh-45/?versions&max-keys=999 2024-04-16T19:48:01.949 DEBUG:boto:Data: 2024-04-16T19:48:01.950 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.950 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:01.950 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.950 DEBUG:boto:Params: {} 2024-04-16T19:48:01.950 DEBUG:boto:Token: None 2024-04-16T19:48:01.950 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/?versions 2024-04-16T19:48:01.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A= 2024-04-16T19:48:01.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Bz1o83Jf4U3y3L8Xi9sH+MF16A=', 'Content-Length': '0'} 2024-04-16T19:48:01.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088076c5f1e9c4788-00661ed5f1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.953 DEBUG:boto:b'yrjefh-45999falseanulltrue2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:01.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:48:01.953 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:01.954 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:01.954 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:01.954 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:01.954 DEBUG:boto:Method: GET 2024-04-16T19:48:01.954 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:01.954 DEBUG:boto:Data: 2024-04-16T19:48:01.954 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.954 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:01.954 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.954 DEBUG:boto:Params: {} 2024-04-16T19:48:01.955 DEBUG:boto:Token: None 2024-04-16T19:48:01.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:01.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GCNrjwRkVGokjbUpMI4zZFvaBkA= 2024-04-16T19:48:01.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GCNrjwRkVGokjbUpMI4zZFvaBkA=', 'Content-Length': '0'} 2024-04-16T19:48:01.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be08761543061b4-00661ed5f1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:01.998 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:01.998 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:01.999 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:01.999 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:01.999 DEBUG:boto:Method: GET 2024-04-16T19:48:01.999 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:01.999 DEBUG:boto:Data: 2024-04-16T19:48:01.999 DEBUG:boto:Headers: {} 2024-04-16T19:48:01.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:01.999 DEBUG:boto:Port: 8000 2024-04-16T19:48:01.999 DEBUG:boto:Params: {} 2024-04-16T19:48:01.999 DEBUG:boto:Token: None 2024-04-16T19:48:01.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:01 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GCNrjwRkVGokjbUpMI4zZFvaBkA= 2024-04-16T19:48:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GCNrjwRkVGokjbUpMI4zZFvaBkA=', 'Content-Length': '0'} 2024-04-16T19:48:02.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17669536423f673-00661ed5f1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.002 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.002 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.002 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.002 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.002 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.002 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.002 DEBUG:boto:Data: 2024-04-16T19:48:02.002 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.003 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.003 DEBUG:boto:Params: {} 2024-04-16T19:48:02.003 DEBUG:boto:Token: None 2024-04-16T19:48:02.003 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NeU07tJ86oj/VnKKLxaFFqDTgb4= 2024-04-16T19:48:02.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NeU07tJ86oj/VnKKLxaFFqDTgb4=', 'Content-Length': '0'} 2024-04-16T19:48:02.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12a0ce99ac5ea7f-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.005 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.005 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.005 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.005 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.006 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.006 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.006 DEBUG:boto:Data: 2024-04-16T19:48:02.006 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.006 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.006 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.006 DEBUG:boto:Params: {} 2024-04-16T19:48:02.006 DEBUG:boto:Token: None 2024-04-16T19:48:02.006 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.006 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NeU07tJ86oj/VnKKLxaFFqDTgb4= 2024-04-16T19:48:02.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NeU07tJ86oj/VnKKLxaFFqDTgb4=', 'Content-Length': '0'} 2024-04-16T19:48:02.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200af195dcc9f9e0-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:48:02.008 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.009 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.009 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.009 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.009 DEBUG:boto:Method: GET 2024-04-16T19:48:02.009 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.009 DEBUG:boto:Data: 2024-04-16T19:48:02.009 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.009 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.009 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.009 DEBUG:boto:Params: {} 2024-04-16T19:48:02.009 DEBUG:boto:Token: None 2024-04-16T19:48:02.009 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.010 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BRlaQ5QWWD1kgOd+ffpa/DJulsA= 2024-04-16T19:48:02.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BRlaQ5QWWD1kgOd+ffpa/DJulsA=', 'Content-Length': '0'} 2024-04-16T19:48:02.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822e4ca2529b5de6-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.012 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.012 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.012 DEBUG:boto:path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.012 DEBUG:boto:auth_path=/yrjefh-45/a?versionId=null 2024-04-16T19:48:02.012 DEBUG:boto:Method: GET 2024-04-16T19:48:02.012 DEBUG:boto:Path: /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.012 DEBUG:boto:Data: 2024-04-16T19:48:02.012 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.012 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.013 DEBUG:boto:Params: {} 2024-04-16T19:48:02.013 DEBUG:boto:Token: None 2024-04-16T19:48:02.013 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a?versionId=null 2024-04-16T19:48:02.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BRlaQ5QWWD1kgOd+ffpa/DJulsA= 2024-04-16T19:48:02.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BRlaQ5QWWD1kgOd+ffpa/DJulsA=', 'Content-Length': '0'} 2024-04-16T19:48:02.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001325174183a2a68-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.015 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.015 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.015 DEBUG:boto:path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.015 DEBUG:boto:auth_path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.015 DEBUG:boto:Method: GET 2024-04-16T19:48:02.016 DEBUG:boto:Path: /yrjefh-45/?prefix=a 2024-04-16T19:48:02.016 DEBUG:boto:Data: 2024-04-16T19:48:02.016 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.016 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.016 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.016 DEBUG:boto:Params: {} 2024-04-16T19:48:02.016 DEBUG:boto:Token: None 2024-04-16T19:48:02.016 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000989d476118ffad1f-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.019 DEBUG:boto:b'yrjefh-45a1000falsea2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.019 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.019 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.019 DEBUG:boto:path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.019 DEBUG:boto:auth_path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.019 DEBUG:boto:Method: GET 2024-04-16T19:48:02.019 DEBUG:boto:Path: /yrjefh-45/?prefix=a 2024-04-16T19:48:02.020 DEBUG:boto:Data: 2024-04-16T19:48:02.020 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.020 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.020 DEBUG:boto:Params: {} 2024-04-16T19:48:02.020 DEBUG:boto:Token: None 2024-04-16T19:48:02.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0875b7a0af345a6-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.066 DEBUG:boto:b'yrjefh-45a1000falsea2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.067 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.067 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.067 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.067 DEBUG:boto:Path: /yrjefh-45/a 2024-04-16T19:48:02.067 DEBUG:boto:Data: 2024-04-16T19:48:02.067 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.067 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.067 DEBUG:boto:Params: {} 2024-04-16T19:48:02.067 DEBUG:boto:Token: None 2024-04-16T19:48:02.068 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a 2024-04-16T19:48:02.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0dr2zFwtLhprpGim7Fxa0h77OIM= 2024-04-16T19:48:02.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0dr2zFwtLhprpGim7Fxa0h77OIM=', 'Content-Length': '0'} 2024-04-16T19:48:02.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc91e3e9aa0713f-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.070 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.070 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.070 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.070 DEBUG:boto:Path: /yrjefh-45/a 2024-04-16T19:48:02.070 DEBUG:boto:Data: 2024-04-16T19:48:02.070 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.070 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.070 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.070 DEBUG:boto:Params: {} 2024-04-16T19:48:02.070 DEBUG:boto:Token: None 2024-04-16T19:48:02.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a 2024-04-16T19:48:02.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0dr2zFwtLhprpGim7Fxa0h77OIM= 2024-04-16T19:48:02.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0dr2zFwtLhprpGim7Fxa0h77OIM=', 'Content-Length': '0'} 2024-04-16T19:48:02.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a0835bb1a08d4f-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:48:02.073 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.073 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.073 DEBUG:boto:Method: GET 2024-04-16T19:48:02.073 DEBUG:boto:Path: /yrjefh-45/a 2024-04-16T19:48:02.073 DEBUG:boto:Data: 2024-04-16T19:48:02.073 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.073 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.073 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.073 DEBUG:boto:Params: {} 2024-04-16T19:48:02.074 DEBUG:boto:Token: None 2024-04-16T19:48:02.074 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a 2024-04-16T19:48:02.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grN+EhmkqGjLNwJaqQ1Zw1qjht8= 2024-04-16T19:48:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grN+EhmkqGjLNwJaqQ1Zw1qjht8=', 'Content-Length': '0'} 2024-04-16T19:48:02.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d467baf3faaa50e4-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.118 DEBUG:boto:path=/yrjefh-45/a 2024-04-16T19:48:02.118 DEBUG:boto:auth_path=/yrjefh-45/a 2024-04-16T19:48:02.119 DEBUG:boto:Method: GET 2024-04-16T19:48:02.119 DEBUG:boto:Path: /yrjefh-45/a 2024-04-16T19:48:02.119 DEBUG:boto:Data: 2024-04-16T19:48:02.119 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.119 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.119 DEBUG:boto:Params: {} 2024-04-16T19:48:02.119 DEBUG:boto:Token: None 2024-04-16T19:48:02.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/a 2024-04-16T19:48:02.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grN+EhmkqGjLNwJaqQ1Zw1qjht8= 2024-04-16T19:48:02.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grN+EhmkqGjLNwJaqQ1Zw1qjht8=', 'Content-Length': '0'} 2024-04-16T19:48:02.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af4005706b5f1a0-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.122 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.122 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.122 DEBUG:boto:path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.122 DEBUG:boto:auth_path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.122 DEBUG:boto:Method: GET 2024-04-16T19:48:02.122 DEBUG:boto:Path: /yrjefh-45/?prefix=a 2024-04-16T19:48:02.122 DEBUG:boto:Data: 2024-04-16T19:48:02.122 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.122 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.122 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.123 DEBUG:boto:Params: {} 2024-04-16T19:48:02.123 DEBUG:boto:Token: None 2024-04-16T19:48:02.123 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.123 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000914855ed82e3bfb7-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.125 DEBUG:boto:b'yrjefh-45a1000falsea2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.126 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.126 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.126 DEBUG:boto:path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.126 DEBUG:boto:auth_path=/yrjefh-45/?prefix=a 2024-04-16T19:48:02.126 DEBUG:boto:Method: GET 2024-04-16T19:48:02.126 DEBUG:boto:Path: /yrjefh-45/?prefix=a 2024-04-16T19:48:02.126 DEBUG:boto:Data: 2024-04-16T19:48:02.126 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.127 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.127 DEBUG:boto:Params: {} 2024-04-16T19:48:02.127 DEBUG:boto:Token: None 2024-04-16T19:48:02.127 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aacbd6632da35dff-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.129 DEBUG:boto:b'yrjefh-45a1000falsea2024-04-16T19:47:51.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:48:02.130 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.130 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.130 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.130 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.130 DEBUG:boto:Method: GET 2024-04-16T19:48:02.130 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.130 DEBUG:boto:Data: 2024-04-16T19:48:02.130 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.130 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.130 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.131 DEBUG:boto:Params: {} 2024-04-16T19:48:02.131 DEBUG:boto:Token: None 2024-04-16T19:48:02.131 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184= 2024-04-16T19:48:02.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184=', 'Content-Length': '0'} 2024-04-16T19:48:02.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001708dc763160b708-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.174 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.174 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.175 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.175 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.175 DEBUG:boto:Method: GET 2024-04-16T19:48:02.175 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.175 DEBUG:boto:Data: 2024-04-16T19:48:02.175 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.175 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.175 DEBUG:boto:Params: {} 2024-04-16T19:48:02.175 DEBUG:boto:Token: None 2024-04-16T19:48:02.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184= 2024-04-16T19:48:02.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184=', 'Content-Length': '0'} 2024-04-16T19:48:02.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cb16145b01345a-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.178 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.178 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.178 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.178 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.178 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.178 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.178 DEBUG:boto:Data: 2024-04-16T19:48:02.178 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.178 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.178 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.179 DEBUG:boto:Params: {} 2024-04-16T19:48:02.179 DEBUG:boto:Token: None 2024-04-16T19:48:02.179 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srBrN0UWDgcL2DGe8jb0Y8aVHEk= 2024-04-16T19:48:02.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srBrN0UWDgcL2DGe8jb0Y8aVHEk=', 'Content-Length': '0'} 2024-04-16T19:48:02.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344de36679e1e813-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.181 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.181 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.181 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.181 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.181 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.182 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.182 DEBUG:boto:Data: 2024-04-16T19:48:02.182 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.182 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.182 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.182 DEBUG:boto:Params: {} 2024-04-16T19:48:02.182 DEBUG:boto:Token: None 2024-04-16T19:48:02.182 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.182 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srBrN0UWDgcL2DGe8jb0Y8aVHEk= 2024-04-16T19:48:02.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srBrN0UWDgcL2DGe8jb0Y8aVHEk=', 'Content-Length': '0'} 2024-04-16T19:48:02.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013eb16eb8eaf78a7-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:48:02.185 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.185 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.185 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.185 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.185 DEBUG:boto:Method: GET 2024-04-16T19:48:02.185 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.185 DEBUG:boto:Data: 2024-04-16T19:48:02.185 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.186 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.186 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.186 DEBUG:boto:Params: {} 2024-04-16T19:48:02.186 DEBUG:boto:Token: None 2024-04-16T19:48:02.186 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.186 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184= 2024-04-16T19:48:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184=', 'Content-Length': '0'} 2024-04-16T19:48:02.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6970ad0b246dfb-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.230 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.230 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.231 DEBUG:boto:path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.231 DEBUG:boto:auth_path=/yrjefh-45/b?versionId=null 2024-04-16T19:48:02.231 DEBUG:boto:Method: GET 2024-04-16T19:48:02.231 DEBUG:boto:Path: /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.231 DEBUG:boto:Data: 2024-04-16T19:48:02.231 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.231 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.231 DEBUG:boto:Params: {} 2024-04-16T19:48:02.231 DEBUG:boto:Token: None 2024-04-16T19:48:02.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b?versionId=null 2024-04-16T19:48:02.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184= 2024-04-16T19:48:02.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:edf8nvNzS24xyQ9LVP/EbhCa184=', 'Content-Length': '0'} 2024-04-16T19:48:02.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0b3395b6a4c4e7-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.234 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.234 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.234 DEBUG:boto:path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.234 DEBUG:boto:auth_path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.234 DEBUG:boto:Method: GET 2024-04-16T19:48:02.235 DEBUG:boto:Path: /yrjefh-45/?prefix=b 2024-04-16T19:48:02.235 DEBUG:boto:Data: 2024-04-16T19:48:02.235 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.235 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.235 DEBUG:boto:Params: {} 2024-04-16T19:48:02.235 DEBUG:boto:Token: None 2024-04-16T19:48:02.235 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada09f4561f159e0-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.238 DEBUG:boto:b'yrjefh-45b1000falseb2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.238 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.238 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.238 DEBUG:boto:path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.239 DEBUG:boto:auth_path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.239 DEBUG:boto:Method: GET 2024-04-16T19:48:02.239 DEBUG:boto:Path: /yrjefh-45/?prefix=b 2024-04-16T19:48:02.239 DEBUG:boto:Data: 2024-04-16T19:48:02.239 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.239 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.239 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.239 DEBUG:boto:Params: {} 2024-04-16T19:48:02.239 DEBUG:boto:Token: None 2024-04-16T19:48:02.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf88ed8c8f26f57a-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.242 DEBUG:boto:b'yrjefh-45b1000falseb2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.242 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.243 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.243 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.243 DEBUG:boto:Path: /yrjefh-45/b 2024-04-16T19:48:02.243 DEBUG:boto:Data: 2024-04-16T19:48:02.243 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.243 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.243 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.243 DEBUG:boto:Params: {} 2024-04-16T19:48:02.243 DEBUG:boto:Token: None 2024-04-16T19:48:02.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b 2024-04-16T19:48:02.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6kxpGeKdEPU9FbfB13niWdnWsxw= 2024-04-16T19:48:02.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6kxpGeKdEPU9FbfB13niWdnWsxw=', 'Content-Length': '0'} 2024-04-16T19:48:02.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e808273b41acc9-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.246 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.246 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.246 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.246 DEBUG:boto:Path: /yrjefh-45/b 2024-04-16T19:48:02.246 DEBUG:boto:Data: 2024-04-16T19:48:02.246 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.246 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.246 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.247 DEBUG:boto:Params: {} 2024-04-16T19:48:02.247 DEBUG:boto:Token: None 2024-04-16T19:48:02.247 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b 2024-04-16T19:48:02.247 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6kxpGeKdEPU9FbfB13niWdnWsxw= 2024-04-16T19:48:02.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6kxpGeKdEPU9FbfB13niWdnWsxw=', 'Content-Length': '0'} 2024-04-16T19:48:02.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63f8d3c33a7911e-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:48:02.249 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.249 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.249 DEBUG:boto:Method: GET 2024-04-16T19:48:02.250 DEBUG:boto:Path: /yrjefh-45/b 2024-04-16T19:48:02.250 DEBUG:boto:Data: 2024-04-16T19:48:02.250 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.250 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.250 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.250 DEBUG:boto:Params: {} 2024-04-16T19:48:02.250 DEBUG:boto:Token: None 2024-04-16T19:48:02.250 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b 2024-04-16T19:48:02.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:geqrWTWln1rjYbBJvSBAEPEhVSo= 2024-04-16T19:48:02.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:geqrWTWln1rjYbBJvSBAEPEhVSo=', 'Content-Length': '0'} 2024-04-16T19:48:02.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002f1a050fee20f15-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.294 DEBUG:boto:path=/yrjefh-45/b 2024-04-16T19:48:02.294 DEBUG:boto:auth_path=/yrjefh-45/b 2024-04-16T19:48:02.294 DEBUG:boto:Method: GET 2024-04-16T19:48:02.295 DEBUG:boto:Path: /yrjefh-45/b 2024-04-16T19:48:02.295 DEBUG:boto:Data: 2024-04-16T19:48:02.295 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.295 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.295 DEBUG:boto:Params: {} 2024-04-16T19:48:02.295 DEBUG:boto:Token: None 2024-04-16T19:48:02.295 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/b 2024-04-16T19:48:02.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:geqrWTWln1rjYbBJvSBAEPEhVSo= 2024-04-16T19:48:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:geqrWTWln1rjYbBJvSBAEPEhVSo=', 'Content-Length': '0'} 2024-04-16T19:48:02.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd418427dbccfc8-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.297 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.297 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.297 DEBUG:boto:path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.298 DEBUG:boto:auth_path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.298 DEBUG:boto:Method: GET 2024-04-16T19:48:02.298 DEBUG:boto:Path: /yrjefh-45/?prefix=b 2024-04-16T19:48:02.298 DEBUG:boto:Data: 2024-04-16T19:48:02.298 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.298 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.298 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.298 DEBUG:boto:Params: {} 2024-04-16T19:48:02.298 DEBUG:boto:Token: None 2024-04-16T19:48:02.298 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f11b1fc6931f998-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.301 DEBUG:boto:b'yrjefh-45b1000falseb2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.301 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.301 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.302 DEBUG:boto:path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.302 DEBUG:boto:auth_path=/yrjefh-45/?prefix=b 2024-04-16T19:48:02.302 DEBUG:boto:Method: GET 2024-04-16T19:48:02.302 DEBUG:boto:Path: /yrjefh-45/?prefix=b 2024-04-16T19:48:02.302 DEBUG:boto:Data: 2024-04-16T19:48:02.302 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.302 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.302 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.302 DEBUG:boto:Params: {} 2024-04-16T19:48:02.302 DEBUG:boto:Token: None 2024-04-16T19:48:02.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081c5505afb2a1bf5-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.305 DEBUG:boto:b'yrjefh-45b1000falseb2024-04-16T19:47:51.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:48:02.306 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.306 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.306 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.306 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.306 DEBUG:boto:Method: GET 2024-04-16T19:48:02.306 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.306 DEBUG:boto:Data: 2024-04-16T19:48:02.306 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.306 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.306 DEBUG:boto:Params: {} 2024-04-16T19:48:02.307 DEBUG:boto:Token: None 2024-04-16T19:48:02.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4= 2024-04-16T19:48:02.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4=', 'Content-Length': '0'} 2024-04-16T19:48:02.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068441d2d7ea06ad8-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.350 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.350 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.351 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.351 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.351 DEBUG:boto:Method: GET 2024-04-16T19:48:02.351 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.351 DEBUG:boto:Data: 2024-04-16T19:48:02.351 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.351 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.351 DEBUG:boto:Params: {} 2024-04-16T19:48:02.351 DEBUG:boto:Token: None 2024-04-16T19:48:02.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4= 2024-04-16T19:48:02.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4=', 'Content-Length': '0'} 2024-04-16T19:48:02.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f0b2810cec57f0-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.353 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.354 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.354 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.354 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.354 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.354 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.354 DEBUG:boto:Data: 2024-04-16T19:48:02.354 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.354 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.354 DEBUG:boto:Params: {} 2024-04-16T19:48:02.354 DEBUG:boto:Token: None 2024-04-16T19:48:02.355 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVc1gaqYzGuJFbBRbeymCIOyhWU= 2024-04-16T19:48:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVc1gaqYzGuJFbBRbeymCIOyhWU=', 'Content-Length': '0'} 2024-04-16T19:48:02.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057cf712f08e32540-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.357 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.357 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.357 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.357 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.357 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.357 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.357 DEBUG:boto:Data: 2024-04-16T19:48:02.357 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.357 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.357 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.357 DEBUG:boto:Params: {} 2024-04-16T19:48:02.357 DEBUG:boto:Token: None 2024-04-16T19:48:02.358 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVc1gaqYzGuJFbBRbeymCIOyhWU= 2024-04-16T19:48:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVc1gaqYzGuJFbBRbeymCIOyhWU=', 'Content-Length': '0'} 2024-04-16T19:48:02.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b157f2e3452038-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:48:02.360 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.360 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.360 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.360 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.360 DEBUG:boto:Method: GET 2024-04-16T19:48:02.361 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.361 DEBUG:boto:Data: 2024-04-16T19:48:02.361 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.361 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.361 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.361 DEBUG:boto:Params: {} 2024-04-16T19:48:02.361 DEBUG:boto:Token: None 2024-04-16T19:48:02.361 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.361 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4= 2024-04-16T19:48:02.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4=', 'Content-Length': '0'} 2024-04-16T19:48:02.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab511d690781f844-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.406 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.406 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.406 DEBUG:boto:path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.407 DEBUG:boto:auth_path=/yrjefh-45/c?versionId=null 2024-04-16T19:48:02.407 DEBUG:boto:Method: GET 2024-04-16T19:48:02.407 DEBUG:boto:Path: /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.407 DEBUG:boto:Data: 2024-04-16T19:48:02.407 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.407 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.407 DEBUG:boto:Params: {} 2024-04-16T19:48:02.407 DEBUG:boto:Token: None 2024-04-16T19:48:02.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c?versionId=null 2024-04-16T19:48:02.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4= 2024-04-16T19:48:02.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z4atboErUuaHY2g4r+RaldlQqr4=', 'Content-Length': '0'} 2024-04-16T19:48:02.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47f2ce8a0c1b138-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.409 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.409 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.410 DEBUG:boto:path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.410 DEBUG:boto:auth_path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.410 DEBUG:boto:Method: GET 2024-04-16T19:48:02.410 DEBUG:boto:Path: /yrjefh-45/?prefix=c 2024-04-16T19:48:02.410 DEBUG:boto:Data: 2024-04-16T19:48:02.410 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.410 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.410 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.410 DEBUG:boto:Params: {} 2024-04-16T19:48:02.410 DEBUG:boto:Token: None 2024-04-16T19:48:02.410 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000518600ef233b836-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.413 DEBUG:boto:b'yrjefh-45c1000falsec2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.413 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.414 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.414 DEBUG:boto:path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.414 DEBUG:boto:auth_path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.414 DEBUG:boto:Method: GET 2024-04-16T19:48:02.414 DEBUG:boto:Path: /yrjefh-45/?prefix=c 2024-04-16T19:48:02.414 DEBUG:boto:Data: 2024-04-16T19:48:02.414 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.414 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.414 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.414 DEBUG:boto:Params: {} 2024-04-16T19:48:02.414 DEBUG:boto:Token: None 2024-04-16T19:48:02.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dde11bc85465f56-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.417 DEBUG:boto:b'yrjefh-45c1000falsec2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.417 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.417 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.418 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.418 DEBUG:boto:Path: /yrjefh-45/c 2024-04-16T19:48:02.418 DEBUG:boto:Data: 2024-04-16T19:48:02.418 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.418 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.418 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.418 DEBUG:boto:Params: {} 2024-04-16T19:48:02.418 DEBUG:boto:Token: None 2024-04-16T19:48:02.418 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c 2024-04-16T19:48:02.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afD1jE2oZ0XYEiYcmQbseuPaXJQ= 2024-04-16T19:48:02.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afD1jE2oZ0XYEiYcmQbseuPaXJQ=', 'Content-Length': '0'} 2024-04-16T19:48:02.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2c1560f3af920e-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.421 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.421 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.421 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.421 DEBUG:boto:Path: /yrjefh-45/c 2024-04-16T19:48:02.421 DEBUG:boto:Data: 2024-04-16T19:48:02.421 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.421 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.421 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.421 DEBUG:boto:Params: {} 2024-04-16T19:48:02.421 DEBUG:boto:Token: None 2024-04-16T19:48:02.422 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c 2024-04-16T19:48:02.422 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afD1jE2oZ0XYEiYcmQbseuPaXJQ= 2024-04-16T19:48:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afD1jE2oZ0XYEiYcmQbseuPaXJQ=', 'Content-Length': '0'} 2024-04-16T19:48:02.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022fe027218adb224-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:48:02.424 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.424 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.424 DEBUG:boto:Method: GET 2024-04-16T19:48:02.424 DEBUG:boto:Path: /yrjefh-45/c 2024-04-16T19:48:02.424 DEBUG:boto:Data: 2024-04-16T19:48:02.424 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.424 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.424 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.424 DEBUG:boto:Params: {} 2024-04-16T19:48:02.425 DEBUG:boto:Token: None 2024-04-16T19:48:02.425 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c 2024-04-16T19:48:02.425 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d1lqbWF6Jm1yyOkbCQtcaigRwqk= 2024-04-16T19:48:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d1lqbWF6Jm1yyOkbCQtcaigRwqk=', 'Content-Length': '0'} 2024-04-16T19:48:02.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000578f956748f686d4-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.470 DEBUG:boto:path=/yrjefh-45/c 2024-04-16T19:48:02.470 DEBUG:boto:auth_path=/yrjefh-45/c 2024-04-16T19:48:02.471 DEBUG:boto:Method: GET 2024-04-16T19:48:02.471 DEBUG:boto:Path: /yrjefh-45/c 2024-04-16T19:48:02.471 DEBUG:boto:Data: 2024-04-16T19:48:02.471 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.471 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.471 DEBUG:boto:Params: {} 2024-04-16T19:48:02.471 DEBUG:boto:Token: None 2024-04-16T19:48:02.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/c 2024-04-16T19:48:02.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d1lqbWF6Jm1yyOkbCQtcaigRwqk= 2024-04-16T19:48:02.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d1lqbWF6Jm1yyOkbCQtcaigRwqk=', 'Content-Length': '0'} 2024-04-16T19:48:02.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091151c203d812341-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.473 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.473 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.473 DEBUG:boto:path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.474 DEBUG:boto:auth_path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.474 DEBUG:boto:Method: GET 2024-04-16T19:48:02.474 DEBUG:boto:Path: /yrjefh-45/?prefix=c 2024-04-16T19:48:02.474 DEBUG:boto:Data: 2024-04-16T19:48:02.474 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.474 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.474 DEBUG:boto:Params: {} 2024-04-16T19:48:02.474 DEBUG:boto:Token: None 2024-04-16T19:48:02.474 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb03cc8f00d2d9f-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.477 DEBUG:boto:b'yrjefh-45c1000falsec2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.477 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.477 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.478 DEBUG:boto:path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.478 DEBUG:boto:auth_path=/yrjefh-45/?prefix=c 2024-04-16T19:48:02.478 DEBUG:boto:Method: GET 2024-04-16T19:48:02.478 DEBUG:boto:Path: /yrjefh-45/?prefix=c 2024-04-16T19:48:02.478 DEBUG:boto:Data: 2024-04-16T19:48:02.478 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.478 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.478 DEBUG:boto:Params: {} 2024-04-16T19:48:02.478 DEBUG:boto:Token: None 2024-04-16T19:48:02.478 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0ea6a47b84b441-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.481 DEBUG:boto:b'yrjefh-45c1000falsec2024-04-16T19:47:51.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:48:02.481 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.481 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.482 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.482 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.482 DEBUG:boto:Method: GET 2024-04-16T19:48:02.482 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.482 DEBUG:boto:Data: 2024-04-16T19:48:02.482 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.482 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.482 DEBUG:boto:Params: {} 2024-04-16T19:48:02.482 DEBUG:boto:Token: None 2024-04-16T19:48:02.482 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.482 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY= 2024-04-16T19:48:02.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY=', 'Content-Length': '0'} 2024-04-16T19:48:02.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae229bfc0bf17ca-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.530 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.530 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.530 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.530 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.531 DEBUG:boto:Method: GET 2024-04-16T19:48:02.531 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.531 DEBUG:boto:Data: 2024-04-16T19:48:02.531 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.531 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.531 DEBUG:boto:Params: {} 2024-04-16T19:48:02.531 DEBUG:boto:Token: None 2024-04-16T19:48:02.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY= 2024-04-16T19:48:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY=', 'Content-Length': '0'} 2024-04-16T19:48:02.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06746d5571d82f1-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.533 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.534 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.534 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.534 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.534 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.534 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.534 DEBUG:boto:Data: 2024-04-16T19:48:02.534 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.534 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.534 DEBUG:boto:Params: {} 2024-04-16T19:48:02.534 DEBUG:boto:Token: None 2024-04-16T19:48:02.535 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tK5AvMHJgfLBMejG34NDr9NdYCg= 2024-04-16T19:48:02.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK5AvMHJgfLBMejG34NDr9NdYCg=', 'Content-Length': '0'} 2024-04-16T19:48:02.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006bc7d25e686e78b-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.537 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.537 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.537 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.537 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.537 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.537 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.537 DEBUG:boto:Data: 2024-04-16T19:48:02.537 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.537 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.538 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.538 DEBUG:boto:Params: {} 2024-04-16T19:48:02.538 DEBUG:boto:Token: None 2024-04-16T19:48:02.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tK5AvMHJgfLBMejG34NDr9NdYCg= 2024-04-16T19:48:02.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK5AvMHJgfLBMejG34NDr9NdYCg=', 'Content-Length': '0'} 2024-04-16T19:48:02.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c73b98ef16a3c5b-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:48:02.540 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.540 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.540 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.540 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.540 DEBUG:boto:Method: GET 2024-04-16T19:48:02.540 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.540 DEBUG:boto:Data: 2024-04-16T19:48:02.541 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.541 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.541 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.541 DEBUG:boto:Params: {} 2024-04-16T19:48:02.541 DEBUG:boto:Token: None 2024-04-16T19:48:02.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY= 2024-04-16T19:48:02.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY=', 'Content-Length': '0'} 2024-04-16T19:48:02.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f542aae2051f7b-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.586 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.586 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.587 DEBUG:boto:path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.587 DEBUG:boto:auth_path=/yrjefh-45/d?versionId=null 2024-04-16T19:48:02.587 DEBUG:boto:Method: GET 2024-04-16T19:48:02.587 DEBUG:boto:Path: /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.587 DEBUG:boto:Data: 2024-04-16T19:48:02.587 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.587 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.587 DEBUG:boto:Params: {} 2024-04-16T19:48:02.587 DEBUG:boto:Token: None 2024-04-16T19:48:02.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d?versionId=null 2024-04-16T19:48:02.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY= 2024-04-16T19:48:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WAuIaSBSDPJYwqBzmxU5htGk9wY=', 'Content-Length': '0'} 2024-04-16T19:48:02.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c8538384a483d45-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.590 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.590 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.590 DEBUG:boto:path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.590 DEBUG:boto:auth_path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.590 DEBUG:boto:Method: GET 2024-04-16T19:48:02.590 DEBUG:boto:Path: /yrjefh-45/?prefix=d 2024-04-16T19:48:02.590 DEBUG:boto:Data: 2024-04-16T19:48:02.590 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.590 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.590 DEBUG:boto:Params: {} 2024-04-16T19:48:02.591 DEBUG:boto:Token: None 2024-04-16T19:48:02.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f28c0775c7bc9c-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.594 DEBUG:boto:b'yrjefh-45d1000falsed2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.594 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.594 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.594 DEBUG:boto:path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.594 DEBUG:boto:auth_path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.594 DEBUG:boto:Method: GET 2024-04-16T19:48:02.594 DEBUG:boto:Path: /yrjefh-45/?prefix=d 2024-04-16T19:48:02.595 DEBUG:boto:Data: 2024-04-16T19:48:02.595 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.595 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.595 DEBUG:boto:Params: {} 2024-04-16T19:48:02.595 DEBUG:boto:Token: None 2024-04-16T19:48:02.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a48a3092c6cfb7cf-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.598 DEBUG:boto:b'yrjefh-45d1000falsed2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.598 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.598 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.598 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.598 DEBUG:boto:Path: /yrjefh-45/d 2024-04-16T19:48:02.598 DEBUG:boto:Data: 2024-04-16T19:48:02.598 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.599 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.599 DEBUG:boto:Params: {} 2024-04-16T19:48:02.599 DEBUG:boto:Token: None 2024-04-16T19:48:02.599 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d 2024-04-16T19:48:02.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0gcdknDjqwR7ZmJHRyMmAMgARhQ= 2024-04-16T19:48:02.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0gcdknDjqwR7ZmJHRyMmAMgARhQ=', 'Content-Length': '0'} 2024-04-16T19:48:02.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe3d86b2f9ba5fa9-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.601 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.601 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.601 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.601 DEBUG:boto:Path: /yrjefh-45/d 2024-04-16T19:48:02.601 DEBUG:boto:Data: 2024-04-16T19:48:02.601 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.601 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.602 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.602 DEBUG:boto:Params: {} 2024-04-16T19:48:02.602 DEBUG:boto:Token: None 2024-04-16T19:48:02.602 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d 2024-04-16T19:48:02.602 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0gcdknDjqwR7ZmJHRyMmAMgARhQ= 2024-04-16T19:48:02.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0gcdknDjqwR7ZmJHRyMmAMgARhQ=', 'Content-Length': '0'} 2024-04-16T19:48:02.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5816e6db3027a7-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:48:02.604 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.604 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.604 DEBUG:boto:Method: GET 2024-04-16T19:48:02.604 DEBUG:boto:Path: /yrjefh-45/d 2024-04-16T19:48:02.604 DEBUG:boto:Data: 2024-04-16T19:48:02.604 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.605 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.605 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.605 DEBUG:boto:Params: {} 2024-04-16T19:48:02.605 DEBUG:boto:Token: None 2024-04-16T19:48:02.605 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d 2024-04-16T19:48:02.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vVnDYwssQnbsEljwNNxgiRHIEhs= 2024-04-16T19:48:02.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vVnDYwssQnbsEljwNNxgiRHIEhs=', 'Content-Length': '0'} 2024-04-16T19:48:02.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:47:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b1591cf1dffc6b-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.650 DEBUG:boto:path=/yrjefh-45/d 2024-04-16T19:48:02.650 DEBUG:boto:auth_path=/yrjefh-45/d 2024-04-16T19:48:02.651 DEBUG:boto:Method: GET 2024-04-16T19:48:02.651 DEBUG:boto:Path: /yrjefh-45/d 2024-04-16T19:48:02.651 DEBUG:boto:Data: 2024-04-16T19:48:02.651 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.651 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.651 DEBUG:boto:Params: {} 2024-04-16T19:48:02.651 DEBUG:boto:Token: None 2024-04-16T19:48:02.651 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/d 2024-04-16T19:48:02.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vVnDYwssQnbsEljwNNxgiRHIEhs= 2024-04-16T19:48:02.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vVnDYwssQnbsEljwNNxgiRHIEhs=', 'Content-Length': '0'} 2024-04-16T19:48:02.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:47:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701e2e6cda7cedc1-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.653 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.654 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.654 DEBUG:boto:path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.654 DEBUG:boto:auth_path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.654 DEBUG:boto:Method: GET 2024-04-16T19:48:02.654 DEBUG:boto:Path: /yrjefh-45/?prefix=d 2024-04-16T19:48:02.654 DEBUG:boto:Data: 2024-04-16T19:48:02.654 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.654 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:48:02.654 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.654 DEBUG:boto:Params: {} 2024-04-16T19:48:02.654 DEBUG:boto:Token: None 2024-04-16T19:48:02.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f522c8dc28da712c-00661ed5f2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.657 DEBUG:boto:b'yrjefh-45d1000falsed2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.657 DEBUG:boto:path=/yrjefh-45/ 2024-04-16T19:48:02.657 DEBUG:boto:auth_path=/yrjefh-45/ 2024-04-16T19:48:02.657 DEBUG:boto:path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.657 DEBUG:boto:auth_path=/yrjefh-45/?prefix=d 2024-04-16T19:48:02.657 DEBUG:boto:Method: GET 2024-04-16T19:48:02.658 DEBUG:boto:Path: /yrjefh-45/?prefix=d 2024-04-16T19:48:02.658 DEBUG:boto:Data: 2024-04-16T19:48:02.658 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.658 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.658 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.658 DEBUG:boto:Params: {} 2024-04-16T19:48:02.658 DEBUG:boto:Token: None 2024-04-16T19:48:02.658 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-45/ 2024-04-16T19:48:02.658 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY= 2024-04-16T19:48:02.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1vb/Mm+3dzOToV/LKdnpHfQp5pY=', 'Content-Length': '0'} 2024-04-16T19:48:02.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d97ae7d8c55bec-00661ed5f2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.661 DEBUG:boto:b'yrjefh-45d1000falsed2024-04-16T19:47:51.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:48:02.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-45 zones={test-zone2, test-zone1} 2024-04-16T19:48:02.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:48:02.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:48:02.661 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:48:02.714 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:48:02.714 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:48:02.797 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-46 2024-04-16T19:48:02.797 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:48:02.797 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:48:02.797 DEBUG:boto:Method: PUT 2024-04-16T19:48:02.797 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:48:02.797 DEBUG:boto:Data: 2024-04-16T19:48:02.797 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.797 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.797 DEBUG:boto:Params: {} 2024-04-16T19:48:02.797 DEBUG:boto:Token: None 2024-04-16T19:48:02.798 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/ 2024-04-16T19:48:02.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4GvjiCM9Apli5kkSU5sO3GUCvLQ= 2024-04-16T19:48:02.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4GvjiCM9Apli5kkSU5sO3GUCvLQ=', 'Content-Length': '0'} 2024-04-16T19:48:02.813 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009bbc0ea8bfdf2323-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.813 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:48:02.813 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:48:02.814 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.814 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:48:02.814 DEBUG:boto:Data: 2024-04-16T19:48:02.814 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.814 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.814 DEBUG:boto:Params: {} 2024-04-16T19:48:02.814 DEBUG:boto:Token: None 2024-04-16T19:48:02.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/ 2024-04-16T19:48:02.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM= 2024-04-16T19:48:02.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM=', 'Content-Length': '0'} 2024-04-16T19:48:02.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020106fcc2df65927-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.817 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:48:02.817 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:48:02.817 DEBUG:boto:Method: PUT 2024-04-16T19:48:02.817 DEBUG:boto:Path: /yrjefh-46/a 2024-04-16T19:48:02.817 DEBUG:boto:Data: 2024-04-16T19:48:02.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:48:02.817 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.817 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.817 DEBUG:boto:Params: {} 2024-04-16T19:48:02.818 DEBUG:boto:Token: None 2024-04-16T19:48:02.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/a 2024-04-16T19:48:02.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+nz7vsQmanIIaSPLh8Q5LwD8qA= 2024-04-16T19:48:02.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+nz7vsQmanIIaSPLh8Q5LwD8qA='} 2024-04-16T19:48:02.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a12af17b8ffeb2b4-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.824 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:48:02.824 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:48:02.824 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.824 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:48:02.824 DEBUG:boto:Data: 2024-04-16T19:48:02.824 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.825 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.825 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.825 DEBUG:boto:Params: {} 2024-04-16T19:48:02.825 DEBUG:boto:Token: None 2024-04-16T19:48:02.825 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/ 2024-04-16T19:48:02.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM= 2024-04-16T19:48:02.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM=', 'Content-Length': '0'} 2024-04-16T19:48:02.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032f04797ae17efd6-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.827 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:48:02.827 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:48:02.827 DEBUG:boto:Method: PUT 2024-04-16T19:48:02.828 DEBUG:boto:Path: /yrjefh-46/b 2024-04-16T19:48:02.828 DEBUG:boto:Data: 2024-04-16T19:48:02.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:48:02.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.828 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.828 DEBUG:boto:Params: {} 2024-04-16T19:48:02.828 DEBUG:boto:Token: None 2024-04-16T19:48:02.828 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/b 2024-04-16T19:48:02.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6AFEdkP0AsfVhfhKMY7eQTWglus= 2024-04-16T19:48:02.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6AFEdkP0AsfVhfhKMY7eQTWglus='} 2024-04-16T19:48:02.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001db8012c3fca7c7f-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.871 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:48:02.871 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:48:02.871 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.871 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:48:02.871 DEBUG:boto:Data: 2024-04-16T19:48:02.871 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.871 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.871 DEBUG:boto:Params: {} 2024-04-16T19:48:02.871 DEBUG:boto:Token: None 2024-04-16T19:48:02.871 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/ 2024-04-16T19:48:02.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM= 2024-04-16T19:48:02.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM=', 'Content-Length': '0'} 2024-04-16T19:48:02.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a402db3edf2bbcb-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.874 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:48:02.874 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:48:02.874 DEBUG:boto:Method: PUT 2024-04-16T19:48:02.874 DEBUG:boto:Path: /yrjefh-46/c 2024-04-16T19:48:02.874 DEBUG:boto:Data: 2024-04-16T19:48:02.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:48:02.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.874 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.874 DEBUG:boto:Params: {} 2024-04-16T19:48:02.874 DEBUG:boto:Token: None 2024-04-16T19:48:02.875 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/c 2024-04-16T19:48:02.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n1tknP2DCDgFzsD6Nm4ADXBooFE= 2024-04-16T19:48:02.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n1tknP2DCDgFzsD6Nm4ADXBooFE='} 2024-04-16T19:48:02.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d81c8687113d4cf-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.880 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:48:02.880 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:48:02.880 DEBUG:boto:Method: HEAD 2024-04-16T19:48:02.880 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:48:02.880 DEBUG:boto:Data: 2024-04-16T19:48:02.881 DEBUG:boto:Headers: {} 2024-04-16T19:48:02.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.881 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.881 DEBUG:boto:Params: {} 2024-04-16T19:48:02.881 DEBUG:boto:Token: None 2024-04-16T19:48:02.881 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/ 2024-04-16T19:48:02.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM= 2024-04-16T19:48:02.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNz5KikJnx8aLwXFmvRF2vsdzQM=', 'Content-Length': '0'} 2024-04-16T19:48:02.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bf415cfbbb98583-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.883 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:48:02.884 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:48:02.884 DEBUG:boto:Method: PUT 2024-04-16T19:48:02.884 DEBUG:boto:Path: /yrjefh-46/d 2024-04-16T19:48:02.884 DEBUG:boto:Data: 2024-04-16T19:48:02.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:48:02.884 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:48:02.884 DEBUG:boto:Port: 8000 2024-04-16T19:48:02.884 DEBUG:boto:Params: {} 2024-04-16T19:48:02.884 DEBUG:boto:Token: None 2024-04-16T19:48:02.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:48:02 GMT /yrjefh-46/d 2024-04-16T19:48:02.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e8HMG/sTGhOtGPIjpEfQN6FYJio= 2024-04-16T19:48:02.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:48:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e8HMG/sTGhOtGPIjpEfQN6FYJio='} 2024-04-16T19:48:02.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a51118ae76952472-00661ed5f2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:48:02.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:48:02.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:48:02.927 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:48:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:48:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:48:03.027 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:48:03.048 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713296882.810474_567.1","last_update":"2024-04-16T19:48:02.810474Z"}] 2024-04-16T19:48:03.048 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296882.810474_567.1'} 2024-04-16T19:48:03.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:48:03.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:48:03.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:48:03.049 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:48:03.104 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:48:03.104 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:48:03.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:48:03.156+0000 7f9f4e54b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:48:03.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:48:03.157+0000 7f9f4e54b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:48:03.180 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296882.810474_567.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:48:02.810474Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:48:03Z"} 2024-04-16T19:48:03.180 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713296882.810474_567.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:48:02.810474Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:48:03Z"} 2024-04-16T19:48:03.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713296882.810474_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:48:02.810474Z', 'realm_epoch': 2}}] 2024-04-16T19:48:03.181 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296882.810474_567.1'} 2024-04-16T19:48:03.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713296882.810474_567.1'} 2024-04-16T19:48:03.181 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:48:03.181 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-46 2024-04-16T19:48:03.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-46', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:48:03.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-46', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:48:03.181 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-46 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:48:03.236 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:48:03.237 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:48:03.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:48:03.290+0000 7f68afe44800 1 waiting to reach incremental sync.. 2024-04-16T19:48:08.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:48:08.292+0000 7f68afe44800 1 bucket sync caught up with source: 2024-04-16T19:48:08.295 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.32.6, , , , 00000000001.31.6, , , 00000000001.54.6, , 00000000001.22.6] 2024-04-16T19:48:08.295 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.32.6, , , , 00000000001.31.6, , , 00000000001.54.6, , 00000000001.22.6] 2024-04-16T19:48:08.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:48:08.292+0000 7f68afe44800 0 bucket checkpoint complete 2024-04-16T19:48:08.320 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-46 2024-04-16T19:48:08.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-46', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:48:08.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-46', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:48:08.320 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-46 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:48:08.381 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:48:08.382 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:48:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:08.441+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:13.442+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:18.443+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:23.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:23.444+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:28.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:28.445+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:33.446+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:38.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:38.447+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:43.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:43.448+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:48.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:48.448+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:53.449+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:48:58.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:48:58.451+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:03.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:03.452+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:08.451+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:13.452+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:18.454+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:23.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:23.455+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:28.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:28.456+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:33.457+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:38.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:38.458+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:43.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:43.459+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:48.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:48.460+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:53.461+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:49:58.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:49:58.462+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:50:03.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:03.462+0000 7fd9c6ee4800 1 waiting to reach incremental sync.. 2024-04-16T19:50:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:08.465+0000 7fd9c6ee4800 1 bucket sync caught up with source: 2024-04-16T19:50:08.467 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.142.6, , , , 00000000001.126.6, , , 00000000001.97.6, , 00000000001.144.6] 2024-04-16T19:50:08.467 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.142.6, , , , 00000000001.126.6, , , 00000000001.97.6, , 00000000001.144.6] 2024-04-16T19:50:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:08.465+0000 7fd9c6ee4800 0 bucket checkpoint complete 2024-04-16T19:50:08.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-46 zones={test-zone2, test-zone1} 2024-04-16T19:50:08.494 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.494 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.494 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.494 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:50:08.494 DEBUG:boto:Data: 2024-04-16T19:50:08.494 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.495 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.495 DEBUG:boto:Params: {} 2024-04-16T19:50:08.495 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:50:08.495 DEBUG:boto:Token: None 2024-04-16T19:50:08.496 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cgMDs8j1WH5jMRkJa3X51AV/7sw= 2024-04-16T19:50:08.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cgMDs8j1WH5jMRkJa3X51AV/7sw=', 'Content-Length': '0'} 2024-04-16T19:50:08.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005b380ab00baf9ed-00661ed670-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.501 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.501 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.501 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.501 DEBUG:boto:Path: /yrjefh-46/ 2024-04-16T19:50:08.501 DEBUG:boto:Data: 2024-04-16T19:50:08.501 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.502 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.502 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.502 DEBUG:boto:Params: {} 2024-04-16T19:50:08.502 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:50:08.502 DEBUG:boto:Token: None 2024-04-16T19:50:08.502 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cgMDs8j1WH5jMRkJa3X51AV/7sw= 2024-04-16T19:50:08.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cgMDs8j1WH5jMRkJa3X51AV/7sw=', 'Content-Length': '0'} 2024-04-16T19:50:08.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fb39540af4ae0f6-00661ed670-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.507 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:50:08.507 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.507 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.507 DEBUG:boto:path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.507 DEBUG:boto:auth_path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.507 DEBUG:boto:Method: GET 2024-04-16T19:50:08.507 DEBUG:boto:Path: /yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.507 DEBUG:boto:Data: 2024-04-16T19:50:08.507 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.507 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.508 DEBUG:boto:Params: {} 2024-04-16T19:50:08.508 DEBUG:boto:Token: None 2024-04-16T19:50:08.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/?versions 2024-04-16T19:50:08.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU= 2024-04-16T19:50:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU=', 'Content-Length': '0'} 2024-04-16T19:50:08.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001171b1c238d3fb9e-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.510 DEBUG:boto:b'yrjefh-46999falseanulltrue2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.511 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:50:08.511 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:50:08.512 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:50:08.512 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:50:08.512 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:50:08.512 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.512 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.512 DEBUG:boto:path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.512 DEBUG:boto:auth_path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.512 DEBUG:boto:Method: GET 2024-04-16T19:50:08.512 DEBUG:boto:Path: /yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.512 DEBUG:boto:Data: 2024-04-16T19:50:08.512 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.512 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.513 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.513 DEBUG:boto:Params: {} 2024-04-16T19:50:08.513 DEBUG:boto:Token: None 2024-04-16T19:50:08.513 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/?versions 2024-04-16T19:50:08.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU= 2024-04-16T19:50:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU=', 'Content-Length': '0'} 2024-04-16T19:50:08.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb09d6c419f26a3-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.515 DEBUG:boto:b'yrjefh-46999falseanulltrue2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.516 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:50:08.516 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:50:08.516 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:50:08.516 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:50:08.516 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.516 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.516 DEBUG:boto:path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.516 DEBUG:boto:auth_path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.517 DEBUG:boto:Method: GET 2024-04-16T19:50:08.517 DEBUG:boto:Path: /yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.517 DEBUG:boto:Data: 2024-04-16T19:50:08.517 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.517 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.517 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.517 DEBUG:boto:Params: {} 2024-04-16T19:50:08.517 DEBUG:boto:Token: None 2024-04-16T19:50:08.517 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/?versions 2024-04-16T19:50:08.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU= 2024-04-16T19:50:08.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU=', 'Content-Length': '0'} 2024-04-16T19:50:08.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e36e3f93288ae64-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.520 DEBUG:boto:b'yrjefh-46999falseanulltrue2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.520 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.520 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.520 DEBUG:boto:path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.520 DEBUG:boto:auth_path=/yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.520 DEBUG:boto:Method: GET 2024-04-16T19:50:08.520 DEBUG:boto:Path: /yrjefh-46/?versions&max-keys=999 2024-04-16T19:50:08.520 DEBUG:boto:Data: 2024-04-16T19:50:08.521 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.521 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.521 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.521 DEBUG:boto:Params: {} 2024-04-16T19:50:08.521 DEBUG:boto:Token: None 2024-04-16T19:50:08.521 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/?versions 2024-04-16T19:50:08.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU= 2024-04-16T19:50:08.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AX0jxiOTWmfV0KpDgD4qVF+ZQgU=', 'Content-Length': '0'} 2024-04-16T19:50:08.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000327f4a4bbbd11510-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.523 DEBUG:boto:b'yrjefh-46999falseanulltrue2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:08.524 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.524 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.524 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.524 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.524 DEBUG:boto:Method: GET 2024-04-16T19:50:08.525 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.525 DEBUG:boto:Data: 2024-04-16T19:50:08.525 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.525 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.525 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.525 DEBUG:boto:Params: {} 2024-04-16T19:50:08.525 DEBUG:boto:Token: None 2024-04-16T19:50:08.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk= 2024-04-16T19:50:08.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk=', 'Content-Length': '0'} 2024-04-16T19:50:08.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241be47fb5fdf3b0-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.528 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.528 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.528 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.528 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.528 DEBUG:boto:Method: GET 2024-04-16T19:50:08.528 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.528 DEBUG:boto:Data: 2024-04-16T19:50:08.528 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.528 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.528 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.529 DEBUG:boto:Params: {} 2024-04-16T19:50:08.529 DEBUG:boto:Token: None 2024-04-16T19:50:08.529 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk= 2024-04-16T19:50:08.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk=', 'Content-Length': '0'} 2024-04-16T19:50:08.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36a9b162a307fb4-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.574 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.574 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.574 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.575 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.575 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.575 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.575 DEBUG:boto:Data: 2024-04-16T19:50:08.575 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.575 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.575 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.575 DEBUG:boto:Params: {} 2024-04-16T19:50:08.575 DEBUG:boto:Token: None 2024-04-16T19:50:08.575 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E14yEhwXBFuo5WAXGGirTJ4grdw= 2024-04-16T19:50:08.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E14yEhwXBFuo5WAXGGirTJ4grdw=', 'Content-Length': '0'} 2024-04-16T19:50:08.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e46a4ba46b9ee3-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.578 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.578 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.578 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.578 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.578 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.578 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.578 DEBUG:boto:Data: 2024-04-16T19:50:08.579 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.579 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.579 DEBUG:boto:Params: {} 2024-04-16T19:50:08.579 DEBUG:boto:Token: None 2024-04-16T19:50:08.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E14yEhwXBFuo5WAXGGirTJ4grdw= 2024-04-16T19:50:08.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E14yEhwXBFuo5WAXGGirTJ4grdw=', 'Content-Length': '0'} 2024-04-16T19:50:08.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021369b180b26c24d-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:08.581 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.582 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.582 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.582 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.582 DEBUG:boto:Method: GET 2024-04-16T19:50:08.582 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.582 DEBUG:boto:Data: 2024-04-16T19:50:08.582 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.582 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.582 DEBUG:boto:Params: {} 2024-04-16T19:50:08.582 DEBUG:boto:Token: None 2024-04-16T19:50:08.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk= 2024-04-16T19:50:08.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk=', 'Content-Length': '0'} 2024-04-16T19:50:08.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb54dad29294f42-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.585 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.585 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.585 DEBUG:boto:path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.585 DEBUG:boto:auth_path=/yrjefh-46/a?versionId=null 2024-04-16T19:50:08.585 DEBUG:boto:Method: GET 2024-04-16T19:50:08.585 DEBUG:boto:Path: /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.585 DEBUG:boto:Data: 2024-04-16T19:50:08.585 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.586 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.586 DEBUG:boto:Params: {} 2024-04-16T19:50:08.586 DEBUG:boto:Token: None 2024-04-16T19:50:08.586 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a?versionId=null 2024-04-16T19:50:08.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk= 2024-04-16T19:50:08.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zXnik4jLR2GI/biil6KJQWVibXk=', 'Content-Length': '0'} 2024-04-16T19:50:08.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35ade3057f9d0ae-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.588 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.588 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.588 DEBUG:boto:path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.588 DEBUG:boto:auth_path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.589 DEBUG:boto:Method: GET 2024-04-16T19:50:08.589 DEBUG:boto:Path: /yrjefh-46/?prefix=a 2024-04-16T19:50:08.589 DEBUG:boto:Data: 2024-04-16T19:50:08.589 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.589 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.589 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.589 DEBUG:boto:Params: {} 2024-04-16T19:50:08.589 DEBUG:boto:Token: None 2024-04-16T19:50:08.589 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b272f50a68ae063-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.592 DEBUG:boto:b'yrjefh-46a1000falsea2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.592 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.592 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.592 DEBUG:boto:path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.592 DEBUG:boto:auth_path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.592 DEBUG:boto:Method: GET 2024-04-16T19:50:08.592 DEBUG:boto:Path: /yrjefh-46/?prefix=a 2024-04-16T19:50:08.592 DEBUG:boto:Data: 2024-04-16T19:50:08.593 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.593 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.593 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.593 DEBUG:boto:Params: {} 2024-04-16T19:50:08.593 DEBUG:boto:Token: None 2024-04-16T19:50:08.593 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.593 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb7e3a006a46402-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.596 DEBUG:boto:b'yrjefh-46a1000falsea2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.596 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.596 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.596 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.596 DEBUG:boto:Path: /yrjefh-46/a 2024-04-16T19:50:08.596 DEBUG:boto:Data: 2024-04-16T19:50:08.596 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.596 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.596 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.597 DEBUG:boto:Params: {} 2024-04-16T19:50:08.597 DEBUG:boto:Token: None 2024-04-16T19:50:08.597 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a 2024-04-16T19:50:08.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cBko0CyE+bg1MG8eCVHmqSqtbnU= 2024-04-16T19:50:08.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cBko0CyE+bg1MG8eCVHmqSqtbnU=', 'Content-Length': '0'} 2024-04-16T19:50:08.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea5e123be2aeeae-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.599 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.599 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.599 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.599 DEBUG:boto:Path: /yrjefh-46/a 2024-04-16T19:50:08.599 DEBUG:boto:Data: 2024-04-16T19:50:08.599 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.599 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.599 DEBUG:boto:Params: {} 2024-04-16T19:50:08.600 DEBUG:boto:Token: None 2024-04-16T19:50:08.600 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a 2024-04-16T19:50:08.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cBko0CyE+bg1MG8eCVHmqSqtbnU= 2024-04-16T19:50:08.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cBko0CyE+bg1MG8eCVHmqSqtbnU=', 'Content-Length': '0'} 2024-04-16T19:50:08.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a356cb23a10c977e-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:08.602 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.602 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.602 DEBUG:boto:Method: GET 2024-04-16T19:50:08.602 DEBUG:boto:Path: /yrjefh-46/a 2024-04-16T19:50:08.603 DEBUG:boto:Data: 2024-04-16T19:50:08.603 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.603 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.603 DEBUG:boto:Params: {} 2024-04-16T19:50:08.603 DEBUG:boto:Token: None 2024-04-16T19:50:08.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a 2024-04-16T19:50:08.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PN4al47VwCoKCkoQyaCkm3pBlMQ= 2024-04-16T19:50:08.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PN4al47VwCoKCkoQyaCkm3pBlMQ=', 'Content-Length': '0'} 2024-04-16T19:50:08.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1050f8b226d5188-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.646 DEBUG:boto:path=/yrjefh-46/a 2024-04-16T19:50:08.646 DEBUG:boto:auth_path=/yrjefh-46/a 2024-04-16T19:50:08.647 DEBUG:boto:Method: GET 2024-04-16T19:50:08.647 DEBUG:boto:Path: /yrjefh-46/a 2024-04-16T19:50:08.647 DEBUG:boto:Data: 2024-04-16T19:50:08.647 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.647 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.647 DEBUG:boto:Params: {} 2024-04-16T19:50:08.647 DEBUG:boto:Token: None 2024-04-16T19:50:08.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/a 2024-04-16T19:50:08.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PN4al47VwCoKCkoQyaCkm3pBlMQ= 2024-04-16T19:50:08.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PN4al47VwCoKCkoQyaCkm3pBlMQ=', 'Content-Length': '0'} 2024-04-16T19:50:08.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0f07b52d4b2dee-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.649 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.650 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.650 DEBUG:boto:path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.650 DEBUG:boto:auth_path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.650 DEBUG:boto:Method: GET 2024-04-16T19:50:08.650 DEBUG:boto:Path: /yrjefh-46/?prefix=a 2024-04-16T19:50:08.650 DEBUG:boto:Data: 2024-04-16T19:50:08.650 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.650 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.650 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.650 DEBUG:boto:Params: {} 2024-04-16T19:50:08.650 DEBUG:boto:Token: None 2024-04-16T19:50:08.651 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050e0c852677f447-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.653 DEBUG:boto:b'yrjefh-46a1000falsea2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.654 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.654 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.654 DEBUG:boto:path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.654 DEBUG:boto:auth_path=/yrjefh-46/?prefix=a 2024-04-16T19:50:08.654 DEBUG:boto:Method: GET 2024-04-16T19:50:08.654 DEBUG:boto:Path: /yrjefh-46/?prefix=a 2024-04-16T19:50:08.654 DEBUG:boto:Data: 2024-04-16T19:50:08.654 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.655 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.655 DEBUG:boto:Params: {} 2024-04-16T19:50:08.655 DEBUG:boto:Token: None 2024-04-16T19:50:08.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bff293529d8a855-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.657 DEBUG:boto:b'yrjefh-46a1000falsea2024-04-16T19:48:02.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:08.658 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.658 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.658 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.658 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.658 DEBUG:boto:Method: GET 2024-04-16T19:50:08.658 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.658 DEBUG:boto:Data: 2024-04-16T19:50:08.658 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.658 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.659 DEBUG:boto:Params: {} 2024-04-16T19:50:08.659 DEBUG:boto:Token: None 2024-04-16T19:50:08.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno= 2024-04-16T19:50:08.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno=', 'Content-Length': '0'} 2024-04-16T19:50:08.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045787c10bc519c1f-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.702 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.703 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.703 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.703 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.703 DEBUG:boto:Method: GET 2024-04-16T19:50:08.703 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.703 DEBUG:boto:Data: 2024-04-16T19:50:08.703 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.703 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.703 DEBUG:boto:Params: {} 2024-04-16T19:50:08.703 DEBUG:boto:Token: None 2024-04-16T19:50:08.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno= 2024-04-16T19:50:08.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno=', 'Content-Length': '0'} 2024-04-16T19:50:08.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64217af413d137e-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.706 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.706 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.706 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.706 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.706 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.706 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.706 DEBUG:boto:Data: 2024-04-16T19:50:08.706 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.707 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.707 DEBUG:boto:Params: {} 2024-04-16T19:50:08.707 DEBUG:boto:Token: None 2024-04-16T19:50:08.707 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0goWhcNMcmw7MNhz0eUlOkxtv44= 2024-04-16T19:50:08.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0goWhcNMcmw7MNhz0eUlOkxtv44=', 'Content-Length': '0'} 2024-04-16T19:50:08.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f45f66ef791838-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.709 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.709 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.709 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.709 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.710 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.710 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.710 DEBUG:boto:Data: 2024-04-16T19:50:08.710 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.710 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.710 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.710 DEBUG:boto:Params: {} 2024-04-16T19:50:08.710 DEBUG:boto:Token: None 2024-04-16T19:50:08.710 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.710 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0goWhcNMcmw7MNhz0eUlOkxtv44= 2024-04-16T19:50:08.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0goWhcNMcmw7MNhz0eUlOkxtv44=', 'Content-Length': '0'} 2024-04-16T19:50:08.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8c1a651c04b385-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:08.712 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.712 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.713 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.713 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.713 DEBUG:boto:Method: GET 2024-04-16T19:50:08.713 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.713 DEBUG:boto:Data: 2024-04-16T19:50:08.713 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.713 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.713 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.713 DEBUG:boto:Params: {} 2024-04-16T19:50:08.713 DEBUG:boto:Token: None 2024-04-16T19:50:08.713 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno= 2024-04-16T19:50:08.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno=', 'Content-Length': '0'} 2024-04-16T19:50:08.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b564e945156d85-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.758 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.758 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.759 DEBUG:boto:path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.759 DEBUG:boto:auth_path=/yrjefh-46/b?versionId=null 2024-04-16T19:50:08.759 DEBUG:boto:Method: GET 2024-04-16T19:50:08.759 DEBUG:boto:Path: /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.759 DEBUG:boto:Data: 2024-04-16T19:50:08.759 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.759 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.759 DEBUG:boto:Params: {} 2024-04-16T19:50:08.759 DEBUG:boto:Token: None 2024-04-16T19:50:08.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b?versionId=null 2024-04-16T19:50:08.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno= 2024-04-16T19:50:08.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JBHOA6lVqIU3iu36mFv5muYsDno=', 'Content-Length': '0'} 2024-04-16T19:50:08.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba95bb8d0e442a55-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.762 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.762 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.762 DEBUG:boto:path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.762 DEBUG:boto:auth_path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.762 DEBUG:boto:Method: GET 2024-04-16T19:50:08.762 DEBUG:boto:Path: /yrjefh-46/?prefix=b 2024-04-16T19:50:08.762 DEBUG:boto:Data: 2024-04-16T19:50:08.762 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.763 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.763 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.763 DEBUG:boto:Params: {} 2024-04-16T19:50:08.763 DEBUG:boto:Token: None 2024-04-16T19:50:08.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b031dffa76bfd7-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.806 DEBUG:boto:b'yrjefh-46b1000falseb2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.807 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.807 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.807 DEBUG:boto:path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.807 DEBUG:boto:auth_path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.807 DEBUG:boto:Method: GET 2024-04-16T19:50:08.807 DEBUG:boto:Path: /yrjefh-46/?prefix=b 2024-04-16T19:50:08.807 DEBUG:boto:Data: 2024-04-16T19:50:08.807 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.807 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.807 DEBUG:boto:Params: {} 2024-04-16T19:50:08.807 DEBUG:boto:Token: None 2024-04-16T19:50:08.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee89b0d769728f1-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.810 DEBUG:boto:b'yrjefh-46b1000falseb2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.811 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.811 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.811 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.811 DEBUG:boto:Path: /yrjefh-46/b 2024-04-16T19:50:08.811 DEBUG:boto:Data: 2024-04-16T19:50:08.811 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.811 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.811 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.811 DEBUG:boto:Params: {} 2024-04-16T19:50:08.812 DEBUG:boto:Token: None 2024-04-16T19:50:08.812 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b 2024-04-16T19:50:08.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKLFA+xMmX5IHh9kDEwMKxRRQtk= 2024-04-16T19:50:08.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKLFA+xMmX5IHh9kDEwMKxRRQtk=', 'Content-Length': '0'} 2024-04-16T19:50:08.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d57e30654964fe7-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.814 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.814 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.814 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.814 DEBUG:boto:Path: /yrjefh-46/b 2024-04-16T19:50:08.815 DEBUG:boto:Data: 2024-04-16T19:50:08.815 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.815 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.815 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.815 DEBUG:boto:Params: {} 2024-04-16T19:50:08.815 DEBUG:boto:Token: None 2024-04-16T19:50:08.815 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b 2024-04-16T19:50:08.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKLFA+xMmX5IHh9kDEwMKxRRQtk= 2024-04-16T19:50:08.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKLFA+xMmX5IHh9kDEwMKxRRQtk=', 'Content-Length': '0'} 2024-04-16T19:50:08.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0920438e26177d8-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:08.817 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.817 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.817 DEBUG:boto:Method: GET 2024-04-16T19:50:08.817 DEBUG:boto:Path: /yrjefh-46/b 2024-04-16T19:50:08.818 DEBUG:boto:Data: 2024-04-16T19:50:08.818 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.818 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.818 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.818 DEBUG:boto:Params: {} 2024-04-16T19:50:08.818 DEBUG:boto:Token: None 2024-04-16T19:50:08.818 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b 2024-04-16T19:50:08.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCa8JdDzVHSbCT0US8gvt8ZwRbs= 2024-04-16T19:50:08.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCa8JdDzVHSbCT0US8gvt8ZwRbs=', 'Content-Length': '0'} 2024-04-16T19:50:08.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94d65adab6b4027-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.862 DEBUG:boto:path=/yrjefh-46/b 2024-04-16T19:50:08.863 DEBUG:boto:auth_path=/yrjefh-46/b 2024-04-16T19:50:08.863 DEBUG:boto:Method: GET 2024-04-16T19:50:08.863 DEBUG:boto:Path: /yrjefh-46/b 2024-04-16T19:50:08.863 DEBUG:boto:Data: 2024-04-16T19:50:08.863 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.863 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.863 DEBUG:boto:Params: {} 2024-04-16T19:50:08.863 DEBUG:boto:Token: None 2024-04-16T19:50:08.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/b 2024-04-16T19:50:08.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCa8JdDzVHSbCT0US8gvt8ZwRbs= 2024-04-16T19:50:08.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCa8JdDzVHSbCT0US8gvt8ZwRbs=', 'Content-Length': '0'} 2024-04-16T19:50:08.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992d83389150407e-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.865 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.866 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.866 DEBUG:boto:path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.866 DEBUG:boto:auth_path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.866 DEBUG:boto:Method: GET 2024-04-16T19:50:08.866 DEBUG:boto:Path: /yrjefh-46/?prefix=b 2024-04-16T19:50:08.866 DEBUG:boto:Data: 2024-04-16T19:50:08.866 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.866 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.866 DEBUG:boto:Params: {} 2024-04-16T19:50:08.866 DEBUG:boto:Token: None 2024-04-16T19:50:08.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde673bcd1db0234-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.869 DEBUG:boto:b'yrjefh-46b1000falseb2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.869 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.870 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.870 DEBUG:boto:path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.870 DEBUG:boto:auth_path=/yrjefh-46/?prefix=b 2024-04-16T19:50:08.870 DEBUG:boto:Method: GET 2024-04-16T19:50:08.870 DEBUG:boto:Path: /yrjefh-46/?prefix=b 2024-04-16T19:50:08.870 DEBUG:boto:Data: 2024-04-16T19:50:08.870 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.870 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.870 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.870 DEBUG:boto:Params: {} 2024-04-16T19:50:08.870 DEBUG:boto:Token: None 2024-04-16T19:50:08.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e98e32e1472d7fa-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.873 DEBUG:boto:b'yrjefh-46b1000falseb2024-04-16T19:48:02.829Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:08.874 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.874 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.874 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.874 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.874 DEBUG:boto:Method: GET 2024-04-16T19:50:08.874 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.874 DEBUG:boto:Data: 2024-04-16T19:50:08.874 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.874 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.874 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.874 DEBUG:boto:Params: {} 2024-04-16T19:50:08.875 DEBUG:boto:Token: None 2024-04-16T19:50:08.875 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k= 2024-04-16T19:50:08.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k=', 'Content-Length': '0'} 2024-04-16T19:50:08.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006671fb65f0998ecf-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.918 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.919 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.919 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.919 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.919 DEBUG:boto:Method: GET 2024-04-16T19:50:08.919 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.919 DEBUG:boto:Data: 2024-04-16T19:50:08.919 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.919 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.919 DEBUG:boto:Params: {} 2024-04-16T19:50:08.919 DEBUG:boto:Token: None 2024-04-16T19:50:08.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k= 2024-04-16T19:50:08.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k=', 'Content-Length': '0'} 2024-04-16T19:50:08.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001096bdf26ec65243-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.922 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.922 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.922 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.922 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.922 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.922 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.922 DEBUG:boto:Data: 2024-04-16T19:50:08.922 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.923 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.923 DEBUG:boto:Params: {} 2024-04-16T19:50:08.923 DEBUG:boto:Token: None 2024-04-16T19:50:08.923 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6kG3UA+D7WGpkt8sFvYRNsBnZM= 2024-04-16T19:50:08.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6kG3UA+D7WGpkt8sFvYRNsBnZM=', 'Content-Length': '0'} 2024-04-16T19:50:08.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76cd2de555feb37-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.925 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.925 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.925 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.925 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.926 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.926 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.926 DEBUG:boto:Data: 2024-04-16T19:50:08.926 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.926 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.926 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.926 DEBUG:boto:Params: {} 2024-04-16T19:50:08.926 DEBUG:boto:Token: None 2024-04-16T19:50:08.926 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6kG3UA+D7WGpkt8sFvYRNsBnZM= 2024-04-16T19:50:08.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6kG3UA+D7WGpkt8sFvYRNsBnZM=', 'Content-Length': '0'} 2024-04-16T19:50:08.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec81d0e14b6d48c4-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:08.928 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.929 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.929 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.929 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.929 DEBUG:boto:Method: GET 2024-04-16T19:50:08.929 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.929 DEBUG:boto:Data: 2024-04-16T19:50:08.929 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.929 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.929 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.929 DEBUG:boto:Params: {} 2024-04-16T19:50:08.929 DEBUG:boto:Token: None 2024-04-16T19:50:08.930 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.930 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k= 2024-04-16T19:50:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k=', 'Content-Length': '0'} 2024-04-16T19:50:08.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481699b0672f0f0f-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.974 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.974 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.974 DEBUG:boto:path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.975 DEBUG:boto:auth_path=/yrjefh-46/c?versionId=null 2024-04-16T19:50:08.975 DEBUG:boto:Method: GET 2024-04-16T19:50:08.975 DEBUG:boto:Path: /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.975 DEBUG:boto:Data: 2024-04-16T19:50:08.975 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.975 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.975 DEBUG:boto:Params: {} 2024-04-16T19:50:08.975 DEBUG:boto:Token: None 2024-04-16T19:50:08.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c?versionId=null 2024-04-16T19:50:08.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k= 2024-04-16T19:50:08.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rYQJxUB/o4ih+enTkSzbDz0f04k=', 'Content-Length': '0'} 2024-04-16T19:50:08.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c5e418e6ecb3d70-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.978 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.978 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.978 DEBUG:boto:path=/yrjefh-46/?prefix=c 2024-04-16T19:50:08.978 DEBUG:boto:auth_path=/yrjefh-46/?prefix=c 2024-04-16T19:50:08.978 DEBUG:boto:Method: GET 2024-04-16T19:50:08.978 DEBUG:boto:Path: /yrjefh-46/?prefix=c 2024-04-16T19:50:08.978 DEBUG:boto:Data: 2024-04-16T19:50:08.978 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.979 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.979 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.979 DEBUG:boto:Params: {} 2024-04-16T19:50:08.979 DEBUG:boto:Token: None 2024-04-16T19:50:08.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db52eee056395fda-00661ed670-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.982 DEBUG:boto:b'yrjefh-46c1000falsec2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.982 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:08.983 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:08.983 DEBUG:boto:path=/yrjefh-46/?prefix=c 2024-04-16T19:50:08.983 DEBUG:boto:auth_path=/yrjefh-46/?prefix=c 2024-04-16T19:50:08.983 DEBUG:boto:Method: GET 2024-04-16T19:50:08.983 DEBUG:boto:Path: /yrjefh-46/?prefix=c 2024-04-16T19:50:08.983 DEBUG:boto:Data: 2024-04-16T19:50:08.983 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.983 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.983 DEBUG:boto:Params: {} 2024-04-16T19:50:08.983 DEBUG:boto:Token: None 2024-04-16T19:50:08.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/ 2024-04-16T19:50:08.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw= 2024-04-16T19:50:08.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xe8i98EspKjaihYjomDMBfoM8rw=', 'Content-Length': '0'} 2024-04-16T19:50:08.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1e3f5b1b2d9539-00661ed670-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.986 DEBUG:boto:b'yrjefh-46c1000falsec2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:08.986 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.986 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.986 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.986 DEBUG:boto:Path: /yrjefh-46/c 2024-04-16T19:50:08.987 DEBUG:boto:Data: 2024-04-16T19:50:08.987 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.987 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.987 DEBUG:boto:Params: {} 2024-04-16T19:50:08.987 DEBUG:boto:Token: None 2024-04-16T19:50:08.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c 2024-04-16T19:50:08.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E1daGpDw/ilDFWoaOL9TzUDVX9k= 2024-04-16T19:50:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E1daGpDw/ilDFWoaOL9TzUDVX9k=', 'Content-Length': '0'} 2024-04-16T19:50:08.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041714b31e43b5a87-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.989 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.989 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.989 DEBUG:boto:Method: HEAD 2024-04-16T19:50:08.990 DEBUG:boto:Path: /yrjefh-46/c 2024-04-16T19:50:08.990 DEBUG:boto:Data: 2024-04-16T19:50:08.990 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:08.990 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.990 DEBUG:boto:Params: {} 2024-04-16T19:50:08.990 DEBUG:boto:Token: None 2024-04-16T19:50:08.990 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c 2024-04-16T19:50:08.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E1daGpDw/ilDFWoaOL9TzUDVX9k= 2024-04-16T19:50:08.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E1daGpDw/ilDFWoaOL9TzUDVX9k=', 'Content-Length': '0'} 2024-04-16T19:50:08.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7ae8954a8e1861-00661ed670-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:08.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:08.992 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:08.993 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:08.993 DEBUG:boto:Method: GET 2024-04-16T19:50:08.993 DEBUG:boto:Path: /yrjefh-46/c 2024-04-16T19:50:08.993 DEBUG:boto:Data: 2024-04-16T19:50:08.993 DEBUG:boto:Headers: {} 2024-04-16T19:50:08.993 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:08.993 DEBUG:boto:Port: 8000 2024-04-16T19:50:08.993 DEBUG:boto:Params: {} 2024-04-16T19:50:08.993 DEBUG:boto:Token: None 2024-04-16T19:50:08.993 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:08 GMT /yrjefh-46/c 2024-04-16T19:50:08.993 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:njMoMNyd6T7esmBs9+SGOcEn7ro= 2024-04-16T19:50:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:njMoMNyd6T7esmBs9+SGOcEn7ro=', 'Content-Length': '0'} 2024-04-16T19:50:08.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c589bbca94f2ac5-00661ed670-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.038 DEBUG:boto:path=/yrjefh-46/c 2024-04-16T19:50:09.038 DEBUG:boto:auth_path=/yrjefh-46/c 2024-04-16T19:50:09.039 DEBUG:boto:Method: GET 2024-04-16T19:50:09.039 DEBUG:boto:Path: /yrjefh-46/c 2024-04-16T19:50:09.039 DEBUG:boto:Data: 2024-04-16T19:50:09.039 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.039 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.039 DEBUG:boto:Params: {} 2024-04-16T19:50:09.039 DEBUG:boto:Token: None 2024-04-16T19:50:09.039 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/c 2024-04-16T19:50:09.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fXZOFV4dABnPBYb+AoIvp+Uu3b8= 2024-04-16T19:50:09.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fXZOFV4dABnPBYb+AoIvp+Uu3b8=', 'Content-Length': '0'} 2024-04-16T19:50:09.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f7778bf9b55448e-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.041 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.042 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.042 DEBUG:boto:path=/yrjefh-46/?prefix=c 2024-04-16T19:50:09.042 DEBUG:boto:auth_path=/yrjefh-46/?prefix=c 2024-04-16T19:50:09.042 DEBUG:boto:Method: GET 2024-04-16T19:50:09.042 DEBUG:boto:Path: /yrjefh-46/?prefix=c 2024-04-16T19:50:09.042 DEBUG:boto:Data: 2024-04-16T19:50:09.042 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.042 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.042 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.042 DEBUG:boto:Params: {} 2024-04-16T19:50:09.043 DEBUG:boto:Token: None 2024-04-16T19:50:09.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e826fbd78e27d30e-00661ed671-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.045 DEBUG:boto:b'yrjefh-46c1000falsec2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.045 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.046 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.046 DEBUG:boto:path=/yrjefh-46/?prefix=c 2024-04-16T19:50:09.046 DEBUG:boto:auth_path=/yrjefh-46/?prefix=c 2024-04-16T19:50:09.046 DEBUG:boto:Method: GET 2024-04-16T19:50:09.046 DEBUG:boto:Path: /yrjefh-46/?prefix=c 2024-04-16T19:50:09.046 DEBUG:boto:Data: 2024-04-16T19:50:09.046 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.046 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.046 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.046 DEBUG:boto:Params: {} 2024-04-16T19:50:09.046 DEBUG:boto:Token: None 2024-04-16T19:50:09.046 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075de033cadd6e7c7-00661ed671-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.049 DEBUG:boto:b'yrjefh-46c1000falsec2024-04-16T19:48:02.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:09.050 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.050 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.050 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.050 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.050 DEBUG:boto:Method: GET 2024-04-16T19:50:09.050 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.050 DEBUG:boto:Data: 2024-04-16T19:50:09.050 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.050 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.050 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.050 DEBUG:boto:Params: {} 2024-04-16T19:50:09.051 DEBUG:boto:Token: None 2024-04-16T19:50:09.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg= 2024-04-16T19:50:09.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg=', 'Content-Length': '0'} 2024-04-16T19:50:09.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f34826b83ec6fe2-00661ed671-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.094 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.094 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.094 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.095 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.095 DEBUG:boto:Method: GET 2024-04-16T19:50:09.095 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.095 DEBUG:boto:Data: 2024-04-16T19:50:09.095 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.095 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.095 DEBUG:boto:Params: {} 2024-04-16T19:50:09.095 DEBUG:boto:Token: None 2024-04-16T19:50:09.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg= 2024-04-16T19:50:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg=', 'Content-Length': '0'} 2024-04-16T19:50:09.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f88db4205c278a9-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.098 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.098 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.098 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.098 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.098 DEBUG:boto:Method: HEAD 2024-04-16T19:50:09.098 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.098 DEBUG:boto:Data: 2024-04-16T19:50:09.099 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.099 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.099 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.099 DEBUG:boto:Params: {} 2024-04-16T19:50:09.099 DEBUG:boto:Token: None 2024-04-16T19:50:09.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9w5rgSpt5Bfd4KoM/HC+hCbzE24= 2024-04-16T19:50:09.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9w5rgSpt5Bfd4KoM/HC+hCbzE24=', 'Content-Length': '0'} 2024-04-16T19:50:09.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab78cea862aeb348-00661ed671-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.102 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.102 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.102 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.102 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.102 DEBUG:boto:Method: HEAD 2024-04-16T19:50:09.102 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.102 DEBUG:boto:Data: 2024-04-16T19:50:09.102 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.102 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.102 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.103 DEBUG:boto:Params: {} 2024-04-16T19:50:09.103 DEBUG:boto:Token: None 2024-04-16T19:50:09.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9w5rgSpt5Bfd4KoM/HC+hCbzE24= 2024-04-16T19:50:09.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9w5rgSpt5Bfd4KoM/HC+hCbzE24=', 'Content-Length': '0'} 2024-04-16T19:50:09.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6b56f5ae87c768-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:09.105 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.106 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.106 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.106 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.106 DEBUG:boto:Method: GET 2024-04-16T19:50:09.106 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.106 DEBUG:boto:Data: 2024-04-16T19:50:09.106 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.106 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.106 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.106 DEBUG:boto:Params: {} 2024-04-16T19:50:09.106 DEBUG:boto:Token: None 2024-04-16T19:50:09.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg= 2024-04-16T19:50:09.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg=', 'Content-Length': '0'} 2024-04-16T19:50:09.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f4d52d09dc64c0-00661ed671-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.150 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.150 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.151 DEBUG:boto:path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.151 DEBUG:boto:auth_path=/yrjefh-46/d?versionId=null 2024-04-16T19:50:09.151 DEBUG:boto:Method: GET 2024-04-16T19:50:09.151 DEBUG:boto:Path: /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.151 DEBUG:boto:Data: 2024-04-16T19:50:09.151 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.151 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.151 DEBUG:boto:Params: {} 2024-04-16T19:50:09.151 DEBUG:boto:Token: None 2024-04-16T19:50:09.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d?versionId=null 2024-04-16T19:50:09.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg= 2024-04-16T19:50:09.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MIKR5PlG2H4EBnBzrK/eRxZEMhg=', 'Content-Length': '0'} 2024-04-16T19:50:09.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003065ac4a048e6054-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.154 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.154 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.154 DEBUG:boto:path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.154 DEBUG:boto:auth_path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.154 DEBUG:boto:Method: GET 2024-04-16T19:50:09.154 DEBUG:boto:Path: /yrjefh-46/?prefix=d 2024-04-16T19:50:09.154 DEBUG:boto:Data: 2024-04-16T19:50:09.155 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.155 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.155 DEBUG:boto:Params: {} 2024-04-16T19:50:09.155 DEBUG:boto:Token: None 2024-04-16T19:50:09.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c86a76c46c2204-00661ed671-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.158 DEBUG:boto:b'yrjefh-46d1000falsed2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.158 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.158 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.159 DEBUG:boto:path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.159 DEBUG:boto:auth_path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.159 DEBUG:boto:Method: GET 2024-04-16T19:50:09.159 DEBUG:boto:Path: /yrjefh-46/?prefix=d 2024-04-16T19:50:09.159 DEBUG:boto:Data: 2024-04-16T19:50:09.159 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.159 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.159 DEBUG:boto:Params: {} 2024-04-16T19:50:09.159 DEBUG:boto:Token: None 2024-04-16T19:50:09.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cd62ee8f010dbe-00661ed671-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.162 DEBUG:boto:b'yrjefh-46d1000falsed2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.162 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.162 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.162 DEBUG:boto:Method: HEAD 2024-04-16T19:50:09.162 DEBUG:boto:Path: /yrjefh-46/d 2024-04-16T19:50:09.162 DEBUG:boto:Data: 2024-04-16T19:50:09.163 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.163 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.163 DEBUG:boto:Params: {} 2024-04-16T19:50:09.163 DEBUG:boto:Token: None 2024-04-16T19:50:09.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d 2024-04-16T19:50:09.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1uHj9BDrw2NEw2r/HRjCY9ejISg= 2024-04-16T19:50:09.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1uHj9BDrw2NEw2r/HRjCY9ejISg=', 'Content-Length': '0'} 2024-04-16T19:50:09.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ec5c615198a2dd-00661ed671-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.166 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.166 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.166 DEBUG:boto:Method: HEAD 2024-04-16T19:50:09.166 DEBUG:boto:Path: /yrjefh-46/d 2024-04-16T19:50:09.166 DEBUG:boto:Data: 2024-04-16T19:50:09.166 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.166 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.166 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.166 DEBUG:boto:Params: {} 2024-04-16T19:50:09.166 DEBUG:boto:Token: None 2024-04-16T19:50:09.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d 2024-04-16T19:50:09.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1uHj9BDrw2NEw2r/HRjCY9ejISg= 2024-04-16T19:50:09.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1uHj9BDrw2NEw2r/HRjCY9ejISg=', 'Content-Length': '0'} 2024-04-16T19:50:09.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b95b30833ee4d8-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:09.169 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.169 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.169 DEBUG:boto:Method: GET 2024-04-16T19:50:09.169 DEBUG:boto:Path: /yrjefh-46/d 2024-04-16T19:50:09.169 DEBUG:boto:Data: 2024-04-16T19:50:09.169 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.170 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.170 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.170 DEBUG:boto:Params: {} 2024-04-16T19:50:09.170 DEBUG:boto:Token: None 2024-04-16T19:50:09.170 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d 2024-04-16T19:50:09.170 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T63Y3fVIX4M01iUE9sJloBf3XBg= 2024-04-16T19:50:09.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T63Y3fVIX4M01iUE9sJloBf3XBg=', 'Content-Length': '0'} 2024-04-16T19:50:09.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-46:b482a2a9-ef36-488b-9666-ada656453328.4812.15'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:48:03 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fc4875e003b772-00661ed671-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.214 DEBUG:boto:path=/yrjefh-46/d 2024-04-16T19:50:09.215 DEBUG:boto:auth_path=/yrjefh-46/d 2024-04-16T19:50:09.215 DEBUG:boto:Method: GET 2024-04-16T19:50:09.215 DEBUG:boto:Path: /yrjefh-46/d 2024-04-16T19:50:09.215 DEBUG:boto:Data: 2024-04-16T19:50:09.215 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.215 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.215 DEBUG:boto:Params: {} 2024-04-16T19:50:09.215 DEBUG:boto:Token: None 2024-04-16T19:50:09.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/d 2024-04-16T19:50:09.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T63Y3fVIX4M01iUE9sJloBf3XBg= 2024-04-16T19:50:09.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T63Y3fVIX4M01iUE9sJloBf3XBg=', 'Content-Length': '0'} 2024-04-16T19:50:09.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7c3d53b94f3a5e-00661ed671-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.218 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.218 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.218 DEBUG:boto:path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.218 DEBUG:boto:auth_path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.218 DEBUG:boto:Method: GET 2024-04-16T19:50:09.218 DEBUG:boto:Path: /yrjefh-46/?prefix=d 2024-04-16T19:50:09.219 DEBUG:boto:Data: 2024-04-16T19:50:09.219 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:09.219 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.219 DEBUG:boto:Params: {} 2024-04-16T19:50:09.219 DEBUG:boto:Token: None 2024-04-16T19:50:09.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e966108731f99d90-00661ed671-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.222 DEBUG:boto:b'yrjefh-46d1000falsed2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.222 DEBUG:boto:path=/yrjefh-46/ 2024-04-16T19:50:09.222 DEBUG:boto:auth_path=/yrjefh-46/ 2024-04-16T19:50:09.223 DEBUG:boto:path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.223 DEBUG:boto:auth_path=/yrjefh-46/?prefix=d 2024-04-16T19:50:09.223 DEBUG:boto:Method: GET 2024-04-16T19:50:09.223 DEBUG:boto:Path: /yrjefh-46/?prefix=d 2024-04-16T19:50:09.223 DEBUG:boto:Data: 2024-04-16T19:50:09.223 DEBUG:boto:Headers: {} 2024-04-16T19:50:09.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:09.223 DEBUG:boto:Port: 8000 2024-04-16T19:50:09.223 DEBUG:boto:Params: {} 2024-04-16T19:50:09.223 DEBUG:boto:Token: None 2024-04-16T19:50:09.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:09 GMT /yrjefh-46/ 2024-04-16T19:50:09.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00= 2024-04-16T19:50:09.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LfcZDSX6otTanVFeQbiVy9kGw00=', 'Content-Length': '0'} 2024-04-16T19:50:09.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34aa721c71e1f0c-00661ed671-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:09.226 DEBUG:boto:b'yrjefh-46d1000falsed2024-04-16T19:48:02.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:09.227 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-46 zones={test-zone2, test-zone1} 2024-04-16T19:50:09.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:09.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:09.227 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:50:09.280 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:09.280 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:09.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:09.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:09.371 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-46 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:50:09.417 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:09.418 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:09.493 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:50:09.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:09.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:09.493 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:50:09.539 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:09.539 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:09.610 INFO:teuthology.orchestra.run.smithi026.stdout:[{"op_id":"1#00000000001.29.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16463170948085121020","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:47:51.116847775Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14"}]},{"op_id":"5#00000000001.18.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16859034386923958786","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:47:51.189869954Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14"}]},{"op_id":"8#00000000001.59.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1725195152600866105","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:47:51.177304608Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14"}]},{"op_id":"10#00000000001.25.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17452885534814017648","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:47:51.128451699Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-45:b482a2a9-ef36-488b-9666-ada656453328.4812.14"}]}] 2024-04-16T19:50:09.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:09.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:09.611 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:50:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:09.656 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:09.923 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:09.921+0000 7fe39c293640 -1 req 13452010420255220480 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:09.926 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:09.924+0000 7fe3e0b1c640 -1 req 13584470350607304459 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:09.929 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:09.928+0000 7fe42ebb8640 -1 req 13486778800486481422 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:09.930 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:09.928+0000 7fe4253a5640 -1 req 6064871077730565200 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.022 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.021+0000 7fe40a36f640 -1 req 13257811256340884607 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.070 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.069+0000 7fe457c0a640 -1 req 12730255985784521437 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.121 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.120+0000 7fe3acab4640 -1 req 10273609375449366556 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.166 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.164+0000 7fe389a6e640 -1 req 4332568192554919522 0.000999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.169 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.168+0000 7fe446be8640 -1 req 4950853757368274800 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.215 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.213+0000 7fe3fe357640 -1 req 13369754121983795558 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-5:b482a2a9-ef36-488b-9666-ada656453328.4361.9,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-5:b482a2a9-ef36-488b-9666-ada656453328.4361.9,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.215 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.213+0000 7fe4263a7640 -1 req 6299871622816650536 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-25:b482a2a9-ef36-488b-9666-ada656453328.4361.29,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-25:b482a2a9-ef36-488b-9666-ada656453328.4361.29,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.257 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.255+0000 7fe455c06640 -1 req 12164588064753745548 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-1:b482a2a9-ef36-488b-9666-ada656453328.4361.1,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.259 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.257+0000 7fe418b8c640 -1 req 1262079685474072156 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-34:b482a2a9-ef36-488b-9666-ada656453328.4812.3,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-34:b482a2a9-ef36-488b-9666-ada656453328.4812.3,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.309 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.308+0000 7fe46ec38640 -1 req 15444861933257684626 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-24:b482a2a9-ef36-488b-9666-ada656453328.4361.28,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-24:b482a2a9-ef36-488b-9666-ada656453328.4361.28,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.397 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.395+0000 7fe39fa9a640 -1 req 11472293971366811702 0.001999950s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-23:b482a2a9-ef36-488b-9666-ada656453328.4361.27,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-23:b482a2a9-ef36-488b-9666-ada656453328.4361.27,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.397 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:50:10.395+0000 7fe3a0a9c640 -1 req 2472476955022928642 0.001999950s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-26:b482a2a9-ef36-488b-9666-ada656453328.4361.30,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-26:b482a2a9-ef36-488b-9666-ada656453328.4361.30,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:50:10.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.464 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:50:10.510 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:10.510 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:10.580 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:50:10.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2024-04-16T19:50:10.583 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:10.583 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:10.583 DEBUG:boto:Method: PUT 2024-04-16T19:50:10.583 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:10.583 DEBUG:boto:Data: 2024-04-16T19:50:10.583 DEBUG:boto:Headers: {} 2024-04-16T19:50:10.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.583 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.584 DEBUG:boto:Params: {} 2024-04-16T19:50:10.584 DEBUG:boto:Token: None 2024-04-16T19:50:10.584 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/ 2024-04-16T19:50:10.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UrqPEKE0P7EmmZ00ErmFVczJVGw= 2024-04-16T19:50:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UrqPEKE0P7EmmZ00ErmFVczJVGw=', 'Content-Length': '0'} 2024-04-16T19:50:10.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033b3eb35035ed4a6-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.598 DEBUG:rgw_multi.tests:created bucket=yrjefh-47 2024-04-16T19:50:10.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.599 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:50:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:10.694 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:50:10.715 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297010.595252_582.1","last_update":"2024-04-16T19:50:10.595252Z"}] 2024-04-16T19:50:10.715 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297010.595252_582.1'} 2024-04-16T19:50:10.715 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:50:10.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:50:10.716 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:50:10.774 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:10.774 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:10.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:10.825+0000 7f82aa624800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:50:10.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:10.827+0000 7f82aa624800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:50:10.849 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297010.595252_582.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:50:10.595252Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:50:10Z"} 2024-04-16T19:50:10.850 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297010.595252_582.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:50:10.595252Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:50:10Z"} 2024-04-16T19:50:10.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297010.595252_582.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:50:10.595252Z', 'realm_epoch': 2}}] 2024-04-16T19:50:10.850 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297010.595252_582.1'} 2024-04-16T19:50:10.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297010.595252_582.1'} 2024-04-16T19:50:10.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:50:10.850 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:10.851 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:10.851 DEBUG:boto:Method: HEAD 2024-04-16T19:50:10.851 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:10.851 DEBUG:boto:Data: 2024-04-16T19:50:10.851 DEBUG:boto:Headers: {} 2024-04-16T19:50:10.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.851 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.851 DEBUG:boto:Params: {} 2024-04-16T19:50:10.851 DEBUG:boto:Token: None 2024-04-16T19:50:10.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/ 2024-04-16T19:50:10.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk= 2024-04-16T19:50:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk=', 'Content-Length': '0'} 2024-04-16T19:50:10.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f69cb7d57b63334f-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.854 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:10.854 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:10.854 DEBUG:boto:Method: PUT 2024-04-16T19:50:10.854 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:50:10.854 DEBUG:boto:Data: 2024-04-16T19:50:10.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:10.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.855 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.855 DEBUG:boto:Params: {} 2024-04-16T19:50:10.855 DEBUG:boto:Token: None 2024-04-16T19:50:10.855 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/a 2024-04-16T19:50:10.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RfrA7oGUxwh/3eg7fMKee1TVP/w= 2024-04-16T19:50:10.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RfrA7oGUxwh/3eg7fMKee1TVP/w='} 2024-04-16T19:50:10.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e1056f6cd3a4b5b-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.861 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:10.861 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:10.861 DEBUG:boto:Method: HEAD 2024-04-16T19:50:10.861 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:10.862 DEBUG:boto:Data: 2024-04-16T19:50:10.862 DEBUG:boto:Headers: {} 2024-04-16T19:50:10.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.862 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.862 DEBUG:boto:Params: {} 2024-04-16T19:50:10.862 DEBUG:boto:Token: None 2024-04-16T19:50:10.862 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/ 2024-04-16T19:50:10.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk= 2024-04-16T19:50:10.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk=', 'Content-Length': '0'} 2024-04-16T19:50:10.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c6ddcd38fc3c719-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.864 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:10.864 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:10.864 DEBUG:boto:Method: PUT 2024-04-16T19:50:10.864 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:50:10.865 DEBUG:boto:Data: 2024-04-16T19:50:10.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:10.865 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.865 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.865 DEBUG:boto:Params: {} 2024-04-16T19:50:10.865 DEBUG:boto:Token: None 2024-04-16T19:50:10.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/b 2024-04-16T19:50:10.865 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI47Pz8Ry2E1zc7/nqpIAU/i1q4= 2024-04-16T19:50:10.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI47Pz8Ry2E1zc7/nqpIAU/i1q4='} 2024-04-16T19:50:10.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067c2f3d44121a9f4-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.871 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:10.871 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:10.871 DEBUG:boto:Method: HEAD 2024-04-16T19:50:10.871 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:10.871 DEBUG:boto:Data: 2024-04-16T19:50:10.871 DEBUG:boto:Headers: {} 2024-04-16T19:50:10.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.871 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.871 DEBUG:boto:Params: {} 2024-04-16T19:50:10.871 DEBUG:boto:Token: None 2024-04-16T19:50:10.872 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/ 2024-04-16T19:50:10.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk= 2024-04-16T19:50:10.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk=', 'Content-Length': '0'} 2024-04-16T19:50:10.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb48c1c97e3581c0-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.874 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:10.874 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:10.874 DEBUG:boto:Method: PUT 2024-04-16T19:50:10.874 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:50:10.874 DEBUG:boto:Data: 2024-04-16T19:50:10.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:10.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.874 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.874 DEBUG:boto:Params: {} 2024-04-16T19:50:10.875 DEBUG:boto:Token: None 2024-04-16T19:50:10.875 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/c 2024-04-16T19:50:10.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1hKyfTDQ28HEnFD78YU+CTjo2Xo= 2024-04-16T19:50:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1hKyfTDQ28HEnFD78YU+CTjo2Xo='} 2024-04-16T19:50:10.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9427d0b7338c939-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.880 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:10.880 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:10.880 DEBUG:boto:Method: HEAD 2024-04-16T19:50:10.880 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:10.880 DEBUG:boto:Data: 2024-04-16T19:50:10.881 DEBUG:boto:Headers: {} 2024-04-16T19:50:10.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.881 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.881 DEBUG:boto:Params: {} 2024-04-16T19:50:10.881 DEBUG:boto:Token: None 2024-04-16T19:50:10.881 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/ 2024-04-16T19:50:10.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk= 2024-04-16T19:50:10.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9wD/Fih2elJW1xAspg4ziw4Rdqk=', 'Content-Length': '0'} 2024-04-16T19:50:10.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c10c2c21082e6b2-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.883 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:10.883 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:10.884 DEBUG:boto:Method: PUT 2024-04-16T19:50:10.884 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:50:10.884 DEBUG:boto:Data: 2024-04-16T19:50:10.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:10.884 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:10.884 DEBUG:boto:Port: 8000 2024-04-16T19:50:10.884 DEBUG:boto:Params: {} 2024-04-16T19:50:10.884 DEBUG:boto:Token: None 2024-04-16T19:50:10.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:10 GMT /yrjefh-47/d 2024-04-16T19:50:10.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c51dLOSagkMy6iPuepIpXaxowiY= 2024-04-16T19:50:10.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c51dLOSagkMy6iPuepIpXaxowiY='} 2024-04-16T19:50:10.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c2b0d832c5d97a6-00661ed672-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:10.927 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-47 2024-04-16T19:50:10.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:10.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:10.927 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-47 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:11.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:11.035+0000 7f7d12f2c800 1 waiting to reach incremental sync.. 2024-04-16T19:50:16.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:16.036+0000 7f7d12f2c800 1 bucket sync caught up with source: 2024-04-16T19:50:16.039 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.42.6, , , , 00000000001.90.6, , , 00000000001.88.6, , 00000000001.31.6] 2024-04-16T19:50:16.039 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.42.6, , , , 00000000001.90.6, , , 00000000001.88.6, , 00000000001.31.6] 2024-04-16T19:50:16.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:16.036+0000 7f7d12f2c800 0 bucket checkpoint complete 2024-04-16T19:50:16.065 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-47 2024-04-16T19:50:16.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:16.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:16.066 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-47 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:50:16.129 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:16.129 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:16.191+0000 7f865712a800 1 waiting to reach incremental sync.. 2024-04-16T19:50:21.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:21.192+0000 7f865712a800 1 bucket sync caught up with source: 2024-04-16T19:50:21.195 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.146.6, , , , 00000000001.201.6, , , 00000000001.202.6, , 00000000001.60.6] 2024-04-16T19:50:21.195 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.146.6, , , , 00000000001.201.6, , , 00000000001.202.6, , 00000000001.60.6] 2024-04-16T19:50:21.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:50:21.192+0000 7f865712a800 0 bucket checkpoint complete 2024-04-16T19:50:21.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-47 zones={test-zone2, test-zone1} 2024-04-16T19:50:21.223 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.223 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.223 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.223 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:21.223 DEBUG:boto:Data: 2024-04-16T19:50:21.223 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.224 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.224 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.224 DEBUG:boto:Params: {} 2024-04-16T19:50:21.224 DEBUG:boto:Token: None 2024-04-16T19:50:21.224 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hBQE3ZHfA7ql7pJBfqdIiCkfbwg= 2024-04-16T19:50:21.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hBQE3ZHfA7ql7pJBfqdIiCkfbwg=', 'Content-Length': '0'} 2024-04-16T19:50:21.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094f1b870bf4387fc-00661ed67d-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.228 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.228 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.228 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.228 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:21.228 DEBUG:boto:Data: 2024-04-16T19:50:21.228 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.228 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.228 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.228 DEBUG:boto:Params: {} 2024-04-16T19:50:21.229 DEBUG:boto:Token: None 2024-04-16T19:50:21.229 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.229 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hBQE3ZHfA7ql7pJBfqdIiCkfbwg= 2024-04-16T19:50:21.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hBQE3ZHfA7ql7pJBfqdIiCkfbwg=', 'Content-Length': '0'} 2024-04-16T19:50:21.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031c634d5d0557e36-00661ed67d-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:50:21.232 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.232 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.232 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.232 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.232 DEBUG:boto:Method: GET 2024-04-16T19:50:21.233 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.233 DEBUG:boto:Data: 2024-04-16T19:50:21.233 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.233 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.233 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.233 DEBUG:boto:Params: {} 2024-04-16T19:50:21.233 DEBUG:boto:Token: None 2024-04-16T19:50:21.233 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/?versions 2024-04-16T19:50:21.233 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU= 2024-04-16T19:50:21.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU=', 'Content-Length': '0'} 2024-04-16T19:50:21.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dfe59a0d363abf4-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.236 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.237 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:50:21.237 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:50:21.237 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:50:21.237 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:50:21.238 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:50:21.238 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.238 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.238 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.238 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.238 DEBUG:boto:Method: GET 2024-04-16T19:50:21.238 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.238 DEBUG:boto:Data: 2024-04-16T19:50:21.238 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.239 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.239 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.239 DEBUG:boto:Params: {} 2024-04-16T19:50:21.239 DEBUG:boto:Token: None 2024-04-16T19:50:21.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/?versions 2024-04-16T19:50:21.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU= 2024-04-16T19:50:21.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU=', 'Content-Length': '0'} 2024-04-16T19:50:21.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe27bf18ce4adf0f-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.242 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.243 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:50:21.243 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:50:21.243 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:50:21.243 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:50:21.243 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.243 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.243 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.243 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.244 DEBUG:boto:Method: GET 2024-04-16T19:50:21.244 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.244 DEBUG:boto:Data: 2024-04-16T19:50:21.244 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.244 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.244 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.244 DEBUG:boto:Params: {} 2024-04-16T19:50:21.244 DEBUG:boto:Token: None 2024-04-16T19:50:21.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/?versions 2024-04-16T19:50:21.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU= 2024-04-16T19:50:21.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU=', 'Content-Length': '0'} 2024-04-16T19:50:21.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9c9c315535eb89-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.247 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.248 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.248 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.248 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.248 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.248 DEBUG:boto:Method: GET 2024-04-16T19:50:21.248 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:50:21.249 DEBUG:boto:Data: 2024-04-16T19:50:21.249 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.249 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.249 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.249 DEBUG:boto:Params: {} 2024-04-16T19:50:21.249 DEBUG:boto:Token: None 2024-04-16T19:50:21.249 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/?versions 2024-04-16T19:50:21.249 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU= 2024-04-16T19:50:21.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:co97adZAkT3QKZX0zd8YtX3T9YU=', 'Content-Length': '0'} 2024-04-16T19:50:21.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6396ac7ee496a7-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.252 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:21.253 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.253 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.253 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.253 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.253 DEBUG:boto:Method: GET 2024-04-16T19:50:21.253 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.253 DEBUG:boto:Data: 2024-04-16T19:50:21.253 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.253 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.254 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.254 DEBUG:boto:Params: {} 2024-04-16T19:50:21.254 DEBUG:boto:Token: None 2024-04-16T19:50:21.254 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.254 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg= 2024-04-16T19:50:21.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg=', 'Content-Length': '0'} 2024-04-16T19:50:21.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094fb08e0061d951b-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.298 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.299 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.299 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.299 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.299 DEBUG:boto:Method: GET 2024-04-16T19:50:21.299 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.299 DEBUG:boto:Data: 2024-04-16T19:50:21.299 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.299 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.299 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.299 DEBUG:boto:Params: {} 2024-04-16T19:50:21.300 DEBUG:boto:Token: None 2024-04-16T19:50:21.300 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg= 2024-04-16T19:50:21.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg=', 'Content-Length': '0'} 2024-04-16T19:50:21.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3fd4b74a4e239d-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.302 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.303 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.303 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.303 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.303 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.303 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.303 DEBUG:boto:Data: 2024-04-16T19:50:21.303 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.303 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.303 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.303 DEBUG:boto:Params: {} 2024-04-16T19:50:21.304 DEBUG:boto:Token: None 2024-04-16T19:50:21.304 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1UX6Qe78xSihLDvECPSUvAjxvQ= 2024-04-16T19:50:21.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1UX6Qe78xSihLDvECPSUvAjxvQ=', 'Content-Length': '0'} 2024-04-16T19:50:21.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b2f5ee2f9489bc-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.306 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.306 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.307 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.307 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.307 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.307 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.307 DEBUG:boto:Data: 2024-04-16T19:50:21.307 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.307 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.307 DEBUG:boto:Params: {} 2024-04-16T19:50:21.308 DEBUG:boto:Token: None 2024-04-16T19:50:21.308 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1UX6Qe78xSihLDvECPSUvAjxvQ= 2024-04-16T19:50:21.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1UX6Qe78xSihLDvECPSUvAjxvQ=', 'Content-Length': '0'} 2024-04-16T19:50:21.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acece1b733fca276-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:21.310 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.310 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.310 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.311 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.311 DEBUG:boto:Method: GET 2024-04-16T19:50:21.311 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.311 DEBUG:boto:Data: 2024-04-16T19:50:21.311 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.311 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.311 DEBUG:boto:Params: {} 2024-04-16T19:50:21.312 DEBUG:boto:Token: None 2024-04-16T19:50:21.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg= 2024-04-16T19:50:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg=', 'Content-Length': '0'} 2024-04-16T19:50:21.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a10f89958201de9-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.314 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.314 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.315 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.315 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:50:21.315 DEBUG:boto:Method: GET 2024-04-16T19:50:21.315 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.315 DEBUG:boto:Data: 2024-04-16T19:50:21.315 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.315 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.315 DEBUG:boto:Params: {} 2024-04-16T19:50:21.315 DEBUG:boto:Token: None 2024-04-16T19:50:21.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:50:21.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg= 2024-04-16T19:50:21.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/QPzbPkFQ6kfX17/mePWwp+Bg=', 'Content-Length': '0'} 2024-04-16T19:50:21.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d6617603d7e628-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.318 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.318 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.319 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.319 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.319 DEBUG:boto:Method: GET 2024-04-16T19:50:21.319 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:50:21.319 DEBUG:boto:Data: 2024-04-16T19:50:21.319 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.319 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.319 DEBUG:boto:Params: {} 2024-04-16T19:50:21.319 DEBUG:boto:Token: None 2024-04-16T19:50:21.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da3d17c2d8d517b6-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.322 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.323 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.323 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.323 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.323 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.323 DEBUG:boto:Method: GET 2024-04-16T19:50:21.323 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:50:21.323 DEBUG:boto:Data: 2024-04-16T19:50:21.324 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.324 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.324 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.324 DEBUG:boto:Params: {} 2024-04-16T19:50:21.324 DEBUG:boto:Token: None 2024-04-16T19:50:21.324 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddf0927c4298472-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.327 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.328 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.328 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.328 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.328 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:50:21.328 DEBUG:boto:Data: 2024-04-16T19:50:21.328 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.328 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.328 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.328 DEBUG:boto:Params: {} 2024-04-16T19:50:21.328 DEBUG:boto:Token: None 2024-04-16T19:50:21.329 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a 2024-04-16T19:50:21.329 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZomNQhG+nEtDlJGPYxx8XaS2+g= 2024-04-16T19:50:21.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZomNQhG+nEtDlJGPYxx8XaS2+g=', 'Content-Length': '0'} 2024-04-16T19:50:21.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d808044f8c895b8f-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.331 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.331 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.331 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.331 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:50:21.331 DEBUG:boto:Data: 2024-04-16T19:50:21.331 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.332 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.332 DEBUG:boto:Params: {} 2024-04-16T19:50:21.332 DEBUG:boto:Token: None 2024-04-16T19:50:21.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a 2024-04-16T19:50:21.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZomNQhG+nEtDlJGPYxx8XaS2+g= 2024-04-16T19:50:21.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZomNQhG+nEtDlJGPYxx8XaS2+g=', 'Content-Length': '0'} 2024-04-16T19:50:21.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069afec3012b88a86-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:50:21.334 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.335 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.335 DEBUG:boto:Method: GET 2024-04-16T19:50:21.335 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:50:21.335 DEBUG:boto:Data: 2024-04-16T19:50:21.335 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.335 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.335 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.335 DEBUG:boto:Params: {} 2024-04-16T19:50:21.335 DEBUG:boto:Token: None 2024-04-16T19:50:21.336 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a 2024-04-16T19:50:21.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BkDZq9l7pPOeyJTmgkc1lsyb7Ew= 2024-04-16T19:50:21.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkDZq9l7pPOeyJTmgkc1lsyb7Ew=', 'Content-Length': '0'} 2024-04-16T19:50:21.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a30f6326024c7ad-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.378 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:50:21.379 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:50:21.379 DEBUG:boto:Method: GET 2024-04-16T19:50:21.379 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:50:21.379 DEBUG:boto:Data: 2024-04-16T19:50:21.379 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.379 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.379 DEBUG:boto:Params: {} 2024-04-16T19:50:21.380 DEBUG:boto:Token: None 2024-04-16T19:50:21.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/a 2024-04-16T19:50:21.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BkDZq9l7pPOeyJTmgkc1lsyb7Ew= 2024-04-16T19:50:21.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkDZq9l7pPOeyJTmgkc1lsyb7Ew=', 'Content-Length': '0'} 2024-04-16T19:50:21.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2273aded27a8983-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.382 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.383 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.383 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.383 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.383 DEBUG:boto:Method: GET 2024-04-16T19:50:21.383 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:50:21.383 DEBUG:boto:Data: 2024-04-16T19:50:21.383 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.383 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.383 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.383 DEBUG:boto:Params: {} 2024-04-16T19:50:21.384 DEBUG:boto:Token: None 2024-04-16T19:50:21.384 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa52c34835de5f5c-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.386 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.387 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.387 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.387 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.387 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:50:21.387 DEBUG:boto:Method: GET 2024-04-16T19:50:21.387 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:50:21.387 DEBUG:boto:Data: 2024-04-16T19:50:21.388 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.388 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.388 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.388 DEBUG:boto:Params: {} 2024-04-16T19:50:21.388 DEBUG:boto:Token: None 2024-04-16T19:50:21.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ff26a14a85b796e-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.391 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:21.391 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.392 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.392 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.392 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.392 DEBUG:boto:Method: GET 2024-04-16T19:50:21.392 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.392 DEBUG:boto:Data: 2024-04-16T19:50:21.392 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.392 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.392 DEBUG:boto:Params: {} 2024-04-16T19:50:21.393 DEBUG:boto:Token: None 2024-04-16T19:50:21.393 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM= 2024-04-16T19:50:21.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM=', 'Content-Length': '0'} 2024-04-16T19:50:21.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2075420362d750c-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.438 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.438 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.439 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.439 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.439 DEBUG:boto:Method: GET 2024-04-16T19:50:21.439 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.439 DEBUG:boto:Data: 2024-04-16T19:50:21.439 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.439 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.439 DEBUG:boto:Params: {} 2024-04-16T19:50:21.440 DEBUG:boto:Token: None 2024-04-16T19:50:21.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM= 2024-04-16T19:50:21.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM=', 'Content-Length': '0'} 2024-04-16T19:50:21.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0074c533e2dde7-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.443 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.443 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.443 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.443 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.443 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.443 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.443 DEBUG:boto:Data: 2024-04-16T19:50:21.443 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.443 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.443 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.444 DEBUG:boto:Params: {} 2024-04-16T19:50:21.444 DEBUG:boto:Token: None 2024-04-16T19:50:21.444 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TF4sSGEc+ExO5RgS45XS7/wCWww= 2024-04-16T19:50:21.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TF4sSGEc+ExO5RgS45XS7/wCWww=', 'Content-Length': '0'} 2024-04-16T19:50:21.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80e4dd485fa0d80-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.446 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.446 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.447 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.447 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.447 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.447 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.447 DEBUG:boto:Data: 2024-04-16T19:50:21.447 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.447 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.447 DEBUG:boto:Params: {} 2024-04-16T19:50:21.447 DEBUG:boto:Token: None 2024-04-16T19:50:21.448 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TF4sSGEc+ExO5RgS45XS7/wCWww= 2024-04-16T19:50:21.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TF4sSGEc+ExO5RgS45XS7/wCWww=', 'Content-Length': '0'} 2024-04-16T19:50:21.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005156b9bae41dbb15-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:21.451 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.451 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.451 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.451 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.451 DEBUG:boto:Method: GET 2024-04-16T19:50:21.451 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.451 DEBUG:boto:Data: 2024-04-16T19:50:21.452 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.452 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.452 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.452 DEBUG:boto:Params: {} 2024-04-16T19:50:21.452 DEBUG:boto:Token: None 2024-04-16T19:50:21.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM= 2024-04-16T19:50:21.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM=', 'Content-Length': '0'} 2024-04-16T19:50:21.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a4096bc6406ae6-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.498 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.498 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.499 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.499 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:50:21.499 DEBUG:boto:Method: GET 2024-04-16T19:50:21.499 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.499 DEBUG:boto:Data: 2024-04-16T19:50:21.499 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.499 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.499 DEBUG:boto:Params: {} 2024-04-16T19:50:21.500 DEBUG:boto:Token: None 2024-04-16T19:50:21.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:50:21.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM= 2024-04-16T19:50:21.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+gYOHJZRCylkwStpYnUA/lg0QfM=', 'Content-Length': '0'} 2024-04-16T19:50:21.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68e0a2c1446d0df-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.503 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.503 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.503 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.503 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.503 DEBUG:boto:Method: GET 2024-04-16T19:50:21.503 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:50:21.503 DEBUG:boto:Data: 2024-04-16T19:50:21.504 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.504 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.504 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.504 DEBUG:boto:Params: {} 2024-04-16T19:50:21.504 DEBUG:boto:Token: None 2024-04-16T19:50:21.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000218985c311341910-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.507 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.507 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.507 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.508 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.508 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.508 DEBUG:boto:Method: GET 2024-04-16T19:50:21.508 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:50:21.508 DEBUG:boto:Data: 2024-04-16T19:50:21.508 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.508 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.508 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.509 DEBUG:boto:Params: {} 2024-04-16T19:50:21.509 DEBUG:boto:Token: None 2024-04-16T19:50:21.509 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088279fba2232b0f7-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.513 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.514 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.514 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.514 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.514 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:50:21.514 DEBUG:boto:Data: 2024-04-16T19:50:21.515 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.515 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.515 DEBUG:boto:Params: {} 2024-04-16T19:50:21.515 DEBUG:boto:Token: None 2024-04-16T19:50:21.515 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b 2024-04-16T19:50:21.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8TWcjSLeZjAGcvBJPd2J95dngnw= 2024-04-16T19:50:21.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8TWcjSLeZjAGcvBJPd2J95dngnw=', 'Content-Length': '0'} 2024-04-16T19:50:21.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183ccd979ec84ecd-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.518 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.518 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.518 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.518 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:50:21.518 DEBUG:boto:Data: 2024-04-16T19:50:21.518 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.519 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.519 DEBUG:boto:Params: {} 2024-04-16T19:50:21.519 DEBUG:boto:Token: None 2024-04-16T19:50:21.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b 2024-04-16T19:50:21.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8TWcjSLeZjAGcvBJPd2J95dngnw= 2024-04-16T19:50:21.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8TWcjSLeZjAGcvBJPd2J95dngnw=', 'Content-Length': '0'} 2024-04-16T19:50:21.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f3b91df02f2b05-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:50:21.522 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.522 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.522 DEBUG:boto:Method: GET 2024-04-16T19:50:21.522 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:50:21.522 DEBUG:boto:Data: 2024-04-16T19:50:21.522 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.522 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.523 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.523 DEBUG:boto:Params: {} 2024-04-16T19:50:21.523 DEBUG:boto:Token: None 2024-04-16T19:50:21.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b 2024-04-16T19:50:21.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wN0tialtpan8K04aiAIWkEygUo0= 2024-04-16T19:50:21.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wN0tialtpan8K04aiAIWkEygUo0=', 'Content-Length': '0'} 2024-04-16T19:50:21.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59bb9610ee2f019-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.566 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:50:21.566 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:50:21.567 DEBUG:boto:Method: GET 2024-04-16T19:50:21.567 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:50:21.567 DEBUG:boto:Data: 2024-04-16T19:50:21.567 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.567 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.567 DEBUG:boto:Params: {} 2024-04-16T19:50:21.567 DEBUG:boto:Token: None 2024-04-16T19:50:21.567 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/b 2024-04-16T19:50:21.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wN0tialtpan8K04aiAIWkEygUo0= 2024-04-16T19:50:21.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wN0tialtpan8K04aiAIWkEygUo0=', 'Content-Length': '0'} 2024-04-16T19:50:21.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5cfeed3f555538e-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.570 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.570 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.570 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.571 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.571 DEBUG:boto:Method: GET 2024-04-16T19:50:21.571 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:50:21.571 DEBUG:boto:Data: 2024-04-16T19:50:21.571 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.571 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.571 DEBUG:boto:Params: {} 2024-04-16T19:50:21.571 DEBUG:boto:Token: None 2024-04-16T19:50:21.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d24aef6d254e68-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.614 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.615 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.615 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.615 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.616 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:50:21.616 DEBUG:boto:Method: GET 2024-04-16T19:50:21.616 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:50:21.616 DEBUG:boto:Data: 2024-04-16T19:50:21.616 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.616 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.616 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.616 DEBUG:boto:Params: {} 2024-04-16T19:50:21.616 DEBUG:boto:Token: None 2024-04-16T19:50:21.617 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.617 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63c022922770716-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.620 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:21.620 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.621 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.621 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.621 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.621 DEBUG:boto:Method: GET 2024-04-16T19:50:21.621 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.621 DEBUG:boto:Data: 2024-04-16T19:50:21.621 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.621 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.621 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.622 DEBUG:boto:Params: {} 2024-04-16T19:50:21.622 DEBUG:boto:Token: None 2024-04-16T19:50:21.622 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.622 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88= 2024-04-16T19:50:21.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88=', 'Content-Length': '0'} 2024-04-16T19:50:21.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700377f227633d17-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.667 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.667 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.667 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.667 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.667 DEBUG:boto:Method: GET 2024-04-16T19:50:21.667 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.667 DEBUG:boto:Data: 2024-04-16T19:50:21.668 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.668 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.668 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.668 DEBUG:boto:Params: {} 2024-04-16T19:50:21.668 DEBUG:boto:Token: None 2024-04-16T19:50:21.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88= 2024-04-16T19:50:21.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88=', 'Content-Length': '0'} 2024-04-16T19:50:21.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfd42feba7e9c91-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.671 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.672 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.672 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.672 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.672 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.672 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.672 DEBUG:boto:Data: 2024-04-16T19:50:21.672 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.672 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.673 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.673 DEBUG:boto:Params: {} 2024-04-16T19:50:21.673 DEBUG:boto:Token: None 2024-04-16T19:50:21.673 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MVYRWGZMCEcoHkx8GCgIcRidgyU= 2024-04-16T19:50:21.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MVYRWGZMCEcoHkx8GCgIcRidgyU=', 'Content-Length': '0'} 2024-04-16T19:50:21.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b01435d13d6bf4a-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.676 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.676 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.676 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.676 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.676 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.676 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.676 DEBUG:boto:Data: 2024-04-16T19:50:21.677 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.677 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.677 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.677 DEBUG:boto:Params: {} 2024-04-16T19:50:21.677 DEBUG:boto:Token: None 2024-04-16T19:50:21.677 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MVYRWGZMCEcoHkx8GCgIcRidgyU= 2024-04-16T19:50:21.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MVYRWGZMCEcoHkx8GCgIcRidgyU=', 'Content-Length': '0'} 2024-04-16T19:50:21.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004134da66650a9cf6-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:21.680 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.680 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.680 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.680 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.681 DEBUG:boto:Method: GET 2024-04-16T19:50:21.681 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.681 DEBUG:boto:Data: 2024-04-16T19:50:21.681 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.681 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.681 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.681 DEBUG:boto:Params: {} 2024-04-16T19:50:21.682 DEBUG:boto:Token: None 2024-04-16T19:50:21.682 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.682 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88= 2024-04-16T19:50:21.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88=', 'Content-Length': '0'} 2024-04-16T19:50:21.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1ac0e3a670dbc5-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.726 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.727 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.727 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.727 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:50:21.727 DEBUG:boto:Method: GET 2024-04-16T19:50:21.727 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.727 DEBUG:boto:Data: 2024-04-16T19:50:21.727 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.728 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.728 DEBUG:boto:Params: {} 2024-04-16T19:50:21.728 DEBUG:boto:Token: None 2024-04-16T19:50:21.728 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:50:21.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88= 2024-04-16T19:50:21.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7GMYWddtoQNCEyylBsFAQB+T88=', 'Content-Length': '0'} 2024-04-16T19:50:21.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d1857b25e0373c-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.731 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.731 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.731 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.731 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.732 DEBUG:boto:Method: GET 2024-04-16T19:50:21.732 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:50:21.732 DEBUG:boto:Data: 2024-04-16T19:50:21.732 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.732 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.732 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.732 DEBUG:boto:Params: {} 2024-04-16T19:50:21.732 DEBUG:boto:Token: None 2024-04-16T19:50:21.733 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014db3b84b6d49a7d-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.736 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.736 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.736 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.737 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.737 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.737 DEBUG:boto:Method: GET 2024-04-16T19:50:21.737 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:50:21.737 DEBUG:boto:Data: 2024-04-16T19:50:21.737 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.737 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.738 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.738 DEBUG:boto:Params: {} 2024-04-16T19:50:21.738 DEBUG:boto:Token: None 2024-04-16T19:50:21.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.738 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034dd8584a35d679-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.741 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.742 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.742 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.742 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.742 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:50:21.742 DEBUG:boto:Data: 2024-04-16T19:50:21.742 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.742 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.743 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.743 DEBUG:boto:Params: {} 2024-04-16T19:50:21.743 DEBUG:boto:Token: None 2024-04-16T19:50:21.743 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c 2024-04-16T19:50:21.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0fiFWxmRtHh9A/RBed+lW6xzTfU= 2024-04-16T19:50:21.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0fiFWxmRtHh9A/RBed+lW6xzTfU=', 'Content-Length': '0'} 2024-04-16T19:50:21.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1c42ce005b6b1d-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.746 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.746 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.746 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.746 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:50:21.746 DEBUG:boto:Data: 2024-04-16T19:50:21.746 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.746 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.747 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.747 DEBUG:boto:Params: {} 2024-04-16T19:50:21.747 DEBUG:boto:Token: None 2024-04-16T19:50:21.747 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c 2024-04-16T19:50:21.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0fiFWxmRtHh9A/RBed+lW6xzTfU= 2024-04-16T19:50:21.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0fiFWxmRtHh9A/RBed+lW6xzTfU=', 'Content-Length': '0'} 2024-04-16T19:50:21.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d99f7a1b83a8f60-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:50:21.750 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.750 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.750 DEBUG:boto:Method: GET 2024-04-16T19:50:21.750 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:50:21.750 DEBUG:boto:Data: 2024-04-16T19:50:21.751 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.751 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.751 DEBUG:boto:Params: {} 2024-04-16T19:50:21.751 DEBUG:boto:Token: None 2024-04-16T19:50:21.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c 2024-04-16T19:50:21.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e+Pt8qhdxVIdhgD9FMjAs2+TSAk= 2024-04-16T19:50:21.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e+Pt8qhdxVIdhgD9FMjAs2+TSAk=', 'Content-Length': '0'} 2024-04-16T19:50:21.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1bb1dc18085d64-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.794 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:50:21.795 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:50:21.795 DEBUG:boto:Method: GET 2024-04-16T19:50:21.795 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:50:21.795 DEBUG:boto:Data: 2024-04-16T19:50:21.795 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.795 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.796 DEBUG:boto:Params: {} 2024-04-16T19:50:21.796 DEBUG:boto:Token: None 2024-04-16T19:50:21.796 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/c 2024-04-16T19:50:21.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e+Pt8qhdxVIdhgD9FMjAs2+TSAk= 2024-04-16T19:50:21.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e+Pt8qhdxVIdhgD9FMjAs2+TSAk=', 'Content-Length': '0'} 2024-04-16T19:50:21.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41feff109070853-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.799 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.799 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.799 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.799 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.799 DEBUG:boto:Method: GET 2024-04-16T19:50:21.799 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:50:21.800 DEBUG:boto:Data: 2024-04-16T19:50:21.800 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.800 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.800 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.800 DEBUG:boto:Params: {} 2024-04-16T19:50:21.800 DEBUG:boto:Token: None 2024-04-16T19:50:21.800 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88586688a641459-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.846 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.847 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.847 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.847 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.847 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:50:21.847 DEBUG:boto:Method: GET 2024-04-16T19:50:21.847 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:50:21.848 DEBUG:boto:Data: 2024-04-16T19:50:21.848 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.848 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.848 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.848 DEBUG:boto:Params: {} 2024-04-16T19:50:21.848 DEBUG:boto:Token: None 2024-04-16T19:50:21.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.849 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120ae15ec879169e-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.851 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:21.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:21.852 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.852 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.852 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.852 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.852 DEBUG:boto:Method: GET 2024-04-16T19:50:21.852 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.853 DEBUG:boto:Data: 2024-04-16T19:50:21.853 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.853 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.853 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.853 DEBUG:boto:Params: {} 2024-04-16T19:50:21.853 DEBUG:boto:Token: None 2024-04-16T19:50:21.853 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.854 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA= 2024-04-16T19:50:21.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA=', 'Content-Length': '0'} 2024-04-16T19:50:21.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001336ef7caa6e44d8-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.898 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.898 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.899 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.899 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.899 DEBUG:boto:Method: GET 2024-04-16T19:50:21.899 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.899 DEBUG:boto:Data: 2024-04-16T19:50:21.899 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.899 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.899 DEBUG:boto:Params: {} 2024-04-16T19:50:21.899 DEBUG:boto:Token: None 2024-04-16T19:50:21.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA= 2024-04-16T19:50:21.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA=', 'Content-Length': '0'} 2024-04-16T19:50:21.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006549b4a1fec0988-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.903 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.903 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.903 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.903 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.903 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.903 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.903 DEBUG:boto:Data: 2024-04-16T19:50:21.903 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.904 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.904 DEBUG:boto:Params: {} 2024-04-16T19:50:21.904 DEBUG:boto:Token: None 2024-04-16T19:50:21.904 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVglkUcbp72sE2H0vCknfsyi3Pw= 2024-04-16T19:50:21.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVglkUcbp72sE2H0vCknfsyi3Pw=', 'Content-Length': '0'} 2024-04-16T19:50:21.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f0a0090f732451-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.906 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.906 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.907 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.907 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.907 DEBUG:boto:Method: HEAD 2024-04-16T19:50:21.907 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.907 DEBUG:boto:Data: 2024-04-16T19:50:21.907 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.907 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.907 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.907 DEBUG:boto:Params: {} 2024-04-16T19:50:21.907 DEBUG:boto:Token: None 2024-04-16T19:50:21.908 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVglkUcbp72sE2H0vCknfsyi3Pw= 2024-04-16T19:50:21.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVglkUcbp72sE2H0vCknfsyi3Pw=', 'Content-Length': '0'} 2024-04-16T19:50:21.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ac4e6ecdf2daa3-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:21.910 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.910 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.910 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.911 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.911 DEBUG:boto:Method: GET 2024-04-16T19:50:21.911 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.911 DEBUG:boto:Data: 2024-04-16T19:50:21.911 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.911 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.911 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.911 DEBUG:boto:Params: {} 2024-04-16T19:50:21.911 DEBUG:boto:Token: None 2024-04-16T19:50:21.911 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA= 2024-04-16T19:50:21.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA=', 'Content-Length': '0'} 2024-04-16T19:50:21.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc06294380e94ca8-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.954 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:21.954 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:21.955 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.955 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:50:21.955 DEBUG:boto:Method: GET 2024-04-16T19:50:21.955 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.955 DEBUG:boto:Data: 2024-04-16T19:50:21.955 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:21.955 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.955 DEBUG:boto:Params: {} 2024-04-16T19:50:21.955 DEBUG:boto:Token: None 2024-04-16T19:50:21.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:50:21.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA= 2024-04-16T19:50:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9LYCc+0xMG1Y6t+4jktSr5vUoA=', 'Content-Length': '0'} 2024-04-16T19:50:21.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a10985e7ec48ddc9-00661ed67d-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:21.958 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:21.958 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:21.958 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:50:21.958 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:50:21.959 DEBUG:boto:Method: GET 2024-04-16T19:50:21.959 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:50:21.959 DEBUG:boto:Data: 2024-04-16T19:50:21.959 DEBUG:boto:Headers: {} 2024-04-16T19:50:21.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:21.959 DEBUG:boto:Port: 8000 2024-04-16T19:50:21.959 DEBUG:boto:Params: {} 2024-04-16T19:50:21.959 DEBUG:boto:Token: None 2024-04-16T19:50:21.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:21 GMT /yrjefh-47/ 2024-04-16T19:50:21.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0= 2024-04-16T19:50:21.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vS11Q2DvDjjMxg9gckb5xE5kEl0=', 'Content-Length': '0'} 2024-04-16T19:50:21.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2b56c691efab8a-00661ed67d-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.002 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:22.003 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.003 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.003 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.003 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.003 DEBUG:boto:Method: GET 2024-04-16T19:50:22.003 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:50:22.003 DEBUG:boto:Data: 2024-04-16T19:50:22.003 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.003 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.003 DEBUG:boto:Params: {} 2024-04-16T19:50:22.004 DEBUG:boto:Token: None 2024-04-16T19:50:22.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM= 2024-04-16T19:50:22.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM=', 'Content-Length': '0'} 2024-04-16T19:50:22.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d90608e0d469b6-00661ed67e-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.006 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:22.007 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:22.007 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:22.007 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.007 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:50:22.007 DEBUG:boto:Data: 2024-04-16T19:50:22.007 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:22.007 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.007 DEBUG:boto:Params: {} 2024-04-16T19:50:22.007 DEBUG:boto:Token: None 2024-04-16T19:50:22.008 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/d 2024-04-16T19:50:22.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DHDHwk7nQkJ94aEQOBxlYiCMYOU= 2024-04-16T19:50:22.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DHDHwk7nQkJ94aEQOBxlYiCMYOU=', 'Content-Length': '0'} 2024-04-16T19:50:22.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfd4d2c9fb9d996-00661ed67e-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.010 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:22.010 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:22.010 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.010 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:50:22.010 DEBUG:boto:Data: 2024-04-16T19:50:22.010 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.011 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.011 DEBUG:boto:Params: {} 2024-04-16T19:50:22.011 DEBUG:boto:Token: None 2024-04-16T19:50:22.011 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/d 2024-04-16T19:50:22.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DHDHwk7nQkJ94aEQOBxlYiCMYOU= 2024-04-16T19:50:22.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DHDHwk7nQkJ94aEQOBxlYiCMYOU=', 'Content-Length': '0'} 2024-04-16T19:50:22.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8b3af9eb706aec-00661ed67e-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:50:22.014 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:22.014 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:22.014 DEBUG:boto:Method: GET 2024-04-16T19:50:22.014 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:50:22.014 DEBUG:boto:Data: 2024-04-16T19:50:22.014 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.014 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:22.014 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.014 DEBUG:boto:Params: {} 2024-04-16T19:50:22.014 DEBUG:boto:Token: None 2024-04-16T19:50:22.014 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/d 2024-04-16T19:50:22.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tVPXHwYIL2tIOHhoVUuiLXbYowo= 2024-04-16T19:50:22.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tVPXHwYIL2tIOHhoVUuiLXbYowo=', 'Content-Length': '0'} 2024-04-16T19:50:22.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3e87a5f37d87a1-00661ed67e-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.058 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:50:22.058 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:50:22.059 DEBUG:boto:Method: GET 2024-04-16T19:50:22.059 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:50:22.059 DEBUG:boto:Data: 2024-04-16T19:50:22.059 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.059 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.059 DEBUG:boto:Params: {} 2024-04-16T19:50:22.059 DEBUG:boto:Token: None 2024-04-16T19:50:22.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/d 2024-04-16T19:50:22.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tVPXHwYIL2tIOHhoVUuiLXbYowo= 2024-04-16T19:50:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tVPXHwYIL2tIOHhoVUuiLXbYowo=', 'Content-Length': '0'} 2024-04-16T19:50:22.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ddcce489f635c62-00661ed67e-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.062 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.062 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.062 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.062 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.062 DEBUG:boto:Method: GET 2024-04-16T19:50:22.063 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:50:22.063 DEBUG:boto:Data: 2024-04-16T19:50:22.063 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:50:22.063 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.063 DEBUG:boto:Params: {} 2024-04-16T19:50:22.063 DEBUG:boto:Token: None 2024-04-16T19:50:22.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM= 2024-04-16T19:50:22.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM=', 'Content-Length': '0'} 2024-04-16T19:50:22.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007533776436fab103-00661ed67e-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.066 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:22.066 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.066 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.066 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.066 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:50:22.067 DEBUG:boto:Method: GET 2024-04-16T19:50:22.067 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:50:22.067 DEBUG:boto:Data: 2024-04-16T19:50:22.067 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.067 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.067 DEBUG:boto:Params: {} 2024-04-16T19:50:22.067 DEBUG:boto:Token: None 2024-04-16T19:50:22.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.067 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM= 2024-04-16T19:50:22.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OitfmSjaZhR6iKtVuBGX0OEqIbM=', 'Content-Length': '0'} 2024-04-16T19:50:22.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e480afb86721afb4-00661ed67e-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.070 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:50:22.071 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-47 zones={test-zone2, test-zone1} 2024-04-16T19:50:22.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-47', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:22.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-47', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:22.071 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-47 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:50:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:22.191 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:50:22.191 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-47 2024-04-16T19:50:22.196 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 4 2024-04-16T19:50:22.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-47', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:22.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'reshard', '--bucket', 'yrjefh-47', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:50:22.247 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket reshard --bucket yrjefh-47 --num-shards 3 --yes-i-really-mean-it --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:50:22.302 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:22.302 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:22.365 INFO:teuthology.orchestra.run.smithi119.stdout:tenant: 2024-04-16T19:50:22.365 INFO:teuthology.orchestra.run.smithi119.stdout:bucket name: yrjefh-47 2024-04-16T19:50:22.369 INFO:teuthology.orchestra.run.smithi119.stdout:total entries: 4 2024-04-16T19:50:22.418 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.419 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.419 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.419 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:22.419 DEBUG:boto:Data: 2024-04-16T19:50:22.419 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.419 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.419 DEBUG:boto:Params: {} 2024-04-16T19:50:22.419 DEBUG:boto:Token: None 2024-04-16T19:50:22.420 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs= 2024-04-16T19:50:22.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs=', 'Content-Length': '0'} 2024-04-16T19:50:22.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176c6bd44085429a-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.422 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:50:22.422 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:50:22.422 DEBUG:boto:Method: PUT 2024-04-16T19:50:22.422 DEBUG:boto:Path: /yrjefh-47/e 2024-04-16T19:50:22.422 DEBUG:boto:Data: 2024-04-16T19:50:22.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:22.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.423 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.423 DEBUG:boto:Params: {} 2024-04-16T19:50:22.423 DEBUG:boto:Token: None 2024-04-16T19:50:22.423 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/e 2024-04-16T19:50:22.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c0ty3v+oRofCNHXVIXAmLgaIKeo= 2024-04-16T19:50:22.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c0ty3v+oRofCNHXVIXAmLgaIKeo='} 2024-04-16T19:50:22.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bf0b19d1e7802df-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.429 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.429 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.429 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.429 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:22.429 DEBUG:boto:Data: 2024-04-16T19:50:22.429 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.429 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.429 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.430 DEBUG:boto:Params: {} 2024-04-16T19:50:22.430 DEBUG:boto:Token: None 2024-04-16T19:50:22.430 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.430 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs= 2024-04-16T19:50:22.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs=', 'Content-Length': '0'} 2024-04-16T19:50:22.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6247eb153643bd4-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.432 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:50:22.432 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:50:22.432 DEBUG:boto:Method: PUT 2024-04-16T19:50:22.432 DEBUG:boto:Path: /yrjefh-47/f 2024-04-16T19:50:22.432 DEBUG:boto:Data: 2024-04-16T19:50:22.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:22.433 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.433 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.433 DEBUG:boto:Params: {} 2024-04-16T19:50:22.433 DEBUG:boto:Token: None 2024-04-16T19:50:22.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/f 2024-04-16T19:50:22.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LKSh8I4ivN5o8/neX0jX6iV1P5A= 2024-04-16T19:50:22.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LKSh8I4ivN5o8/neX0jX6iV1P5A='} 2024-04-16T19:50:22.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053d25bc1dfc428a7-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.439 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.439 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.439 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.439 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:22.439 DEBUG:boto:Data: 2024-04-16T19:50:22.439 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.439 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.439 DEBUG:boto:Params: {} 2024-04-16T19:50:22.440 DEBUG:boto:Token: None 2024-04-16T19:50:22.440 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs= 2024-04-16T19:50:22.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs=', 'Content-Length': '0'} 2024-04-16T19:50:22.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000437f8448438e60a4-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.442 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:50:22.442 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:50:22.442 DEBUG:boto:Method: PUT 2024-04-16T19:50:22.442 DEBUG:boto:Path: /yrjefh-47/g 2024-04-16T19:50:22.442 DEBUG:boto:Data: 2024-04-16T19:50:22.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:22.442 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.442 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.442 DEBUG:boto:Params: {} 2024-04-16T19:50:22.443 DEBUG:boto:Token: None 2024-04-16T19:50:22.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/g 2024-04-16T19:50:22.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ELznTVkqPOKNc7uDe9powkLUrBU= 2024-04-16T19:50:22.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ELznTVkqPOKNc7uDe9powkLUrBU='} 2024-04-16T19:50:22.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015991eb418f953d5-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.449 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:50:22.449 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:50:22.449 DEBUG:boto:Method: HEAD 2024-04-16T19:50:22.449 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:50:22.449 DEBUG:boto:Data: 2024-04-16T19:50:22.449 DEBUG:boto:Headers: {} 2024-04-16T19:50:22.449 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.449 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.449 DEBUG:boto:Params: {} 2024-04-16T19:50:22.449 DEBUG:boto:Token: None 2024-04-16T19:50:22.450 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/ 2024-04-16T19:50:22.450 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs= 2024-04-16T19:50:22.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLIkULxquk7v3/uSwJTB/XnYsPs=', 'Content-Length': '0'} 2024-04-16T19:50:22.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de4367953d80e630-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.452 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:50:22.452 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:50:22.452 DEBUG:boto:Method: PUT 2024-04-16T19:50:22.452 DEBUG:boto:Path: /yrjefh-47/h 2024-04-16T19:50:22.453 DEBUG:boto:Data: 2024-04-16T19:50:22.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:50:22.453 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:50:22.453 DEBUG:boto:Port: 8000 2024-04-16T19:50:22.453 DEBUG:boto:Params: {} 2024-04-16T19:50:22.453 DEBUG:boto:Token: None 2024-04-16T19:50:22.453 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:50:22 GMT /yrjefh-47/h 2024-04-16T19:50:22.453 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AJ18aFwUkLGmgijNgxAKa7DbxtE= 2024-04-16T19:50:22.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:50:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AJ18aFwUkLGmgijNgxAKa7DbxtE='} 2024-04-16T19:50:22.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000497af04a2680a55b-00661ed67e-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:50:22.499 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-47 2024-04-16T19:50:22.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:22.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:50:22.499 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-47 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:50:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:50:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:50:22.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:22.608+0000 7fe82ab1f800 1 waiting to reach latest gen 1, on 0.. 2024-04-16T19:50:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:27.610+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:27.612 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , ] 2024-04-16T19:50:27.612 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:32.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:32.611+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:32.613 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , ] 2024-04-16T19:50:32.613 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:37.612+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:37.614 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , ] 2024-04-16T19:50:37.614 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:39.381 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:50:39.379+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T19:50:42.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:42.613+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:42.615 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , ] 2024-04-16T19:50:42.615 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:47.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:47.614+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:47.616 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.25.6, , ] 2024-04-16T19:50:47.616 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:52.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:52.614+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:52.617 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.25.6, , ] 2024-04-16T19:50:52.617 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:50:57.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:50:57.616+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:50:57.618 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2024-04-16T19:50:57.619 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:51:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:02.616+0000 7fe82ab1f800 1 waiting for incremental sync to catch up: 2024-04-16T19:51:02.619 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2024-04-16T19:51:02.620 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:51:07.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:07.618+0000 7fe82ab1f800 1 bucket sync caught up with source: 2024-04-16T19:51:07.620 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:51:07.620 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.25.6, 00000000002.36.6, 00000000002.27.6] 2024-04-16T19:51:07.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:07.618+0000 7fe82ab1f800 0 bucket checkpoint complete 2024-04-16T19:51:07.648 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-47 2024-04-16T19:51:07.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:51:07.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-47', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:51:07.649 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-47 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:51:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:51:07.712 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:51:07.773+0000 7fbb4cb3f800 1 waiting for incremental sync to catch up: 2024-04-16T19:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.88.6, 00000000002.131.6, 00000000002.89.6] 2024-04-16T19:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000003.90.6, 00000000002.131.6, 00000000002.89.6] 2024-04-16T19:51:12.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:51:12.774+0000 7fbb4cb3f800 1 bucket sync caught up with source: 2024-04-16T19:51:12.776 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000003.90.6, 00000000002.131.6, 00000000002.89.6] 2024-04-16T19:51:12.776 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000003.90.6, 00000000002.131.6, 00000000002.89.6] 2024-04-16T19:51:12.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:51:12.774+0000 7fbb4cb3f800 0 bucket checkpoint complete 2024-04-16T19:51:12.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-47 zones={test-zone2, test-zone1} 2024-04-16T19:51:12.803 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.803 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.804 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.804 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:51:12.804 DEBUG:boto:Data: 2024-04-16T19:51:12.804 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.804 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.804 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.804 DEBUG:boto:Params: {} 2024-04-16T19:51:12.805 DEBUG:boto:Token: None 2024-04-16T19:51:12.805 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/ 2024-04-16T19:51:12.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xw6pdEF/4/a9MeCLzQjz5Y64UvQ= 2024-04-16T19:51:12.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xw6pdEF/4/a9MeCLzQjz5Y64UvQ=', 'Content-Length': '0'} 2024-04-16T19:51:12.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfa579aa331a8e9e-00661ed6b0-4911-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.808 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.808 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.808 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.808 DEBUG:boto:Path: /yrjefh-47/ 2024-04-16T19:51:12.808 DEBUG:boto:Data: 2024-04-16T19:51:12.808 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.808 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.809 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.809 DEBUG:boto:Params: {} 2024-04-16T19:51:12.809 DEBUG:boto:Token: None 2024-04-16T19:51:12.809 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/ 2024-04-16T19:51:12.809 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xw6pdEF/4/a9MeCLzQjz5Y64UvQ= 2024-04-16T19:51:12.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xw6pdEF/4/a9MeCLzQjz5Y64UvQ=', 'Content-Length': '0'} 2024-04-16T19:51:12.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a069635e4c569f4-00661ed6b0-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:51:12.812 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.812 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.813 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.813 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.813 DEBUG:boto:Method: GET 2024-04-16T19:51:12.813 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.813 DEBUG:boto:Data: 2024-04-16T19:51:12.813 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.813 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.813 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.813 DEBUG:boto:Params: {} 2024-04-16T19:51:12.813 DEBUG:boto:Token: None 2024-04-16T19:51:12.814 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/?versions 2024-04-16T19:51:12.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE= 2024-04-16T19:51:12.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE=', 'Content-Length': '0'} 2024-04-16T19:51:12.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215853b2d65d2e98-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.816 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.817 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:51:12.817 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:51:12.817 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:51:12.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:51:12.818 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.818 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.818 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.819 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.819 DEBUG:boto:Method: GET 2024-04-16T19:51:12.819 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.819 DEBUG:boto:Data: 2024-04-16T19:51:12.819 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.819 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.819 DEBUG:boto:Params: {} 2024-04-16T19:51:12.819 DEBUG:boto:Token: None 2024-04-16T19:51:12.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/?versions 2024-04-16T19:51:12.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE= 2024-04-16T19:51:12.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE=', 'Content-Length': '0'} 2024-04-16T19:51:12.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002247f445baccd30c-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.822 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:51:12.823 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:51:12.824 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.824 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.824 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.824 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.824 DEBUG:boto:Method: GET 2024-04-16T19:51:12.824 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.824 DEBUG:boto:Data: 2024-04-16T19:51:12.824 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.825 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.825 DEBUG:boto:Params: {} 2024-04-16T19:51:12.825 DEBUG:boto:Token: None 2024-04-16T19:51:12.825 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/?versions 2024-04-16T19:51:12.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE= 2024-04-16T19:51:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE=', 'Content-Length': '0'} 2024-04-16T19:51:12.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ef0a15a94491b2-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.828 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.828 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.829 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.829 DEBUG:boto:path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.829 DEBUG:boto:auth_path=/yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.829 DEBUG:boto:Method: GET 2024-04-16T19:51:12.829 DEBUG:boto:Path: /yrjefh-47/?versions&max-keys=999 2024-04-16T19:51:12.829 DEBUG:boto:Data: 2024-04-16T19:51:12.829 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.829 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.829 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.830 DEBUG:boto:Params: {} 2024-04-16T19:51:12.830 DEBUG:boto:Token: None 2024-04-16T19:51:12.830 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/?versions 2024-04-16T19:51:12.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE= 2024-04-16T19:51:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9eAnncTE6KJbtwogJ87bvWicHXE=', 'Content-Length': '0'} 2024-04-16T19:51:12.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d349bd644f3e95c4-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.833 DEBUG:boto:b'yrjefh-47999falseanulltrue2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:51:12.833 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.833 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.834 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.834 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.834 DEBUG:boto:Method: GET 2024-04-16T19:51:12.834 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.834 DEBUG:boto:Data: 2024-04-16T19:51:12.834 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.834 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.834 DEBUG:boto:Params: {} 2024-04-16T19:51:12.835 DEBUG:boto:Token: None 2024-04-16T19:51:12.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic= 2024-04-16T19:51:12.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic=', 'Content-Length': '0'} 2024-04-16T19:51:12.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aca2b0b9279adba-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.878 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.879 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.879 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.879 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.879 DEBUG:boto:Method: GET 2024-04-16T19:51:12.879 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.879 DEBUG:boto:Data: 2024-04-16T19:51:12.879 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.879 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.880 DEBUG:boto:Params: {} 2024-04-16T19:51:12.880 DEBUG:boto:Token: None 2024-04-16T19:51:12.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic= 2024-04-16T19:51:12.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic=', 'Content-Length': '0'} 2024-04-16T19:51:12.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9990a7eb69f7566-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.883 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.883 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.883 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.883 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.883 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.883 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.883 DEBUG:boto:Data: 2024-04-16T19:51:12.883 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.884 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.884 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.884 DEBUG:boto:Params: {} 2024-04-16T19:51:12.884 DEBUG:boto:Token: None 2024-04-16T19:51:12.884 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nZq+FRmTUCH78ChTcE2cCwTcsaA= 2024-04-16T19:51:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nZq+FRmTUCH78ChTcE2cCwTcsaA=', 'Content-Length': '0'} 2024-04-16T19:51:12.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dee6e09493c6440-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.887 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.887 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.887 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.887 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.887 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.887 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.887 DEBUG:boto:Data: 2024-04-16T19:51:12.888 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.888 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.888 DEBUG:boto:Params: {} 2024-04-16T19:51:12.888 DEBUG:boto:Token: None 2024-04-16T19:51:12.888 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nZq+FRmTUCH78ChTcE2cCwTcsaA= 2024-04-16T19:51:12.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nZq+FRmTUCH78ChTcE2cCwTcsaA=', 'Content-Length': '0'} 2024-04-16T19:51:12.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4a7efb14836365e-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:51:12.891 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.891 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.891 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.891 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.891 DEBUG:boto:Method: GET 2024-04-16T19:51:12.891 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.892 DEBUG:boto:Data: 2024-04-16T19:51:12.892 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.892 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.892 DEBUG:boto:Params: {} 2024-04-16T19:51:12.892 DEBUG:boto:Token: None 2024-04-16T19:51:12.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic= 2024-04-16T19:51:12.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic=', 'Content-Length': '0'} 2024-04-16T19:51:12.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd935a5973d75e23-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.895 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.895 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.895 DEBUG:boto:path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.895 DEBUG:boto:auth_path=/yrjefh-47/a?versionId=null 2024-04-16T19:51:12.895 DEBUG:boto:Method: GET 2024-04-16T19:51:12.895 DEBUG:boto:Path: /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.895 DEBUG:boto:Data: 2024-04-16T19:51:12.896 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.896 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.896 DEBUG:boto:Params: {} 2024-04-16T19:51:12.896 DEBUG:boto:Token: None 2024-04-16T19:51:12.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a?versionId=null 2024-04-16T19:51:12.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic= 2024-04-16T19:51:12.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3CaKDIr/Hd6OG1TDtyJFGxTqEic=', 'Content-Length': '0'} 2024-04-16T19:51:12.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a50710d051c1a76-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.942 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.943 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.943 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:51:12.943 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:51:12.943 DEBUG:boto:Method: GET 2024-04-16T19:51:12.943 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:51:12.943 DEBUG:boto:Data: 2024-04-16T19:51:12.943 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.944 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.944 DEBUG:boto:Params: {} 2024-04-16T19:51:12.944 DEBUG:boto:Token: None 2024-04-16T19:51:12.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/ 2024-04-16T19:51:12.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rtWmyJ4DRffhs7qfNHuOShnMBxY= 2024-04-16T19:51:12.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rtWmyJ4DRffhs7qfNHuOShnMBxY=', 'Content-Length': '0'} 2024-04-16T19:51:12.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd961e6cdefa0649-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.947 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.948 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:12.948 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:12.948 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:51:12.948 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:51:12.948 DEBUG:boto:Method: GET 2024-04-16T19:51:12.948 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:51:12.948 DEBUG:boto:Data: 2024-04-16T19:51:12.949 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.949 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.949 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.949 DEBUG:boto:Params: {} 2024-04-16T19:51:12.949 DEBUG:boto:Token: None 2024-04-16T19:51:12.949 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/ 2024-04-16T19:51:12.949 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rtWmyJ4DRffhs7qfNHuOShnMBxY= 2024-04-16T19:51:12.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rtWmyJ4DRffhs7qfNHuOShnMBxY=', 'Content-Length': '0'} 2024-04-16T19:51:12.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec04bd2ea6313f2b-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.952 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:12.953 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.953 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.953 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.953 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:51:12.953 DEBUG:boto:Data: 2024-04-16T19:51:12.953 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.953 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.953 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.954 DEBUG:boto:Params: {} 2024-04-16T19:51:12.954 DEBUG:boto:Token: None 2024-04-16T19:51:12.954 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a 2024-04-16T19:51:12.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lMQnDuFkuOf6y+4vFYz59x44DFg= 2024-04-16T19:51:12.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lMQnDuFkuOf6y+4vFYz59x44DFg=', 'Content-Length': '0'} 2024-04-16T19:51:12.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049691ef5833be55c-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.956 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.956 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.957 DEBUG:boto:Method: HEAD 2024-04-16T19:51:12.957 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:51:12.957 DEBUG:boto:Data: 2024-04-16T19:51:12.957 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.957 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:12.957 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.957 DEBUG:boto:Params: {} 2024-04-16T19:51:12.957 DEBUG:boto:Token: None 2024-04-16T19:51:12.958 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a 2024-04-16T19:51:12.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lMQnDuFkuOf6y+4vFYz59x44DFg= 2024-04-16T19:51:12.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lMQnDuFkuOf6y+4vFYz59x44DFg=', 'Content-Length': '0'} 2024-04-16T19:51:12.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b494a9da47b611-00661ed6b0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:12.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:51:12.960 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:12.960 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:12.960 DEBUG:boto:Method: GET 2024-04-16T19:51:12.961 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:51:12.961 DEBUG:boto:Data: 2024-04-16T19:51:12.961 DEBUG:boto:Headers: {} 2024-04-16T19:51:12.961 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:12.961 DEBUG:boto:Port: 8000 2024-04-16T19:51:12.961 DEBUG:boto:Params: {} 2024-04-16T19:51:12.961 DEBUG:boto:Token: None 2024-04-16T19:51:12.962 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:12 GMT /yrjefh-47/a 2024-04-16T19:51:12.962 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vVbxdZz8DE5/X2p9yIIUW8MHNAU= 2024-04-16T19:51:12.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vVbxdZz8DE5/X2p9yIIUW8MHNAU=', 'Content-Length': '0'} 2024-04-16T19:51:12.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049227f0c4afd2f14-00661ed6b0-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.006 DEBUG:boto:path=/yrjefh-47/a 2024-04-16T19:51:13.007 DEBUG:boto:auth_path=/yrjefh-47/a 2024-04-16T19:51:13.007 DEBUG:boto:Method: GET 2024-04-16T19:51:13.007 DEBUG:boto:Path: /yrjefh-47/a 2024-04-16T19:51:13.007 DEBUG:boto:Data: 2024-04-16T19:51:13.007 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.007 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.007 DEBUG:boto:Params: {} 2024-04-16T19:51:13.008 DEBUG:boto:Token: None 2024-04-16T19:51:13.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/a 2024-04-16T19:51:13.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uJV7sQI82LtsVGs3qfS/tOp7N4Q= 2024-04-16T19:51:13.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uJV7sQI82LtsVGs3qfS/tOp7N4Q=', 'Content-Length': '0'} 2024-04-16T19:51:13.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad53f0540e519cc-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.011 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.011 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.011 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:51:13.011 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:51:13.011 DEBUG:boto:Method: GET 2024-04-16T19:51:13.011 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:51:13.011 DEBUG:boto:Data: 2024-04-16T19:51:13.011 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.012 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.012 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.012 DEBUG:boto:Params: {} 2024-04-16T19:51:13.012 DEBUG:boto:Token: None 2024-04-16T19:51:13.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094a95871a1380956-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.015 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.015 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.016 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.016 DEBUG:boto:path=/yrjefh-47/?prefix=a 2024-04-16T19:51:13.016 DEBUG:boto:auth_path=/yrjefh-47/?prefix=a 2024-04-16T19:51:13.016 DEBUG:boto:Method: GET 2024-04-16T19:51:13.016 DEBUG:boto:Path: /yrjefh-47/?prefix=a 2024-04-16T19:51:13.016 DEBUG:boto:Data: 2024-04-16T19:51:13.016 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.016 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.016 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.017 DEBUG:boto:Params: {} 2024-04-16T19:51:13.017 DEBUG:boto:Token: None 2024-04-16T19:51:13.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9d54cac5e22b99-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.062 DEBUG:boto:b'yrjefh-47a1000falsea2024-04-16T19:50:10.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:51:13.063 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.063 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.063 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.063 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.064 DEBUG:boto:Method: GET 2024-04-16T19:51:13.064 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.064 DEBUG:boto:Data: 2024-04-16T19:51:13.064 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.064 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.064 DEBUG:boto:Params: {} 2024-04-16T19:51:13.064 DEBUG:boto:Token: None 2024-04-16T19:51:13.065 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY= 2024-04-16T19:51:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY=', 'Content-Length': '0'} 2024-04-16T19:51:13.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2d33777c686b29-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.111 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.111 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.111 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.111 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.111 DEBUG:boto:Method: GET 2024-04-16T19:51:13.111 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.111 DEBUG:boto:Data: 2024-04-16T19:51:13.111 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.112 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.112 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.112 DEBUG:boto:Params: {} 2024-04-16T19:51:13.112 DEBUG:boto:Token: None 2024-04-16T19:51:13.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY= 2024-04-16T19:51:13.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY=', 'Content-Length': '0'} 2024-04-16T19:51:13.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c2c72890376edcd-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.115 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.115 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.115 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.116 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.116 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.116 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.116 DEBUG:boto:Data: 2024-04-16T19:51:13.116 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.116 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.116 DEBUG:boto:Params: {} 2024-04-16T19:51:13.117 DEBUG:boto:Token: None 2024-04-16T19:51:13.117 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QlsvIzhFvVEV2wbsqyyz6uPdtoA= 2024-04-16T19:51:13.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QlsvIzhFvVEV2wbsqyyz6uPdtoA=', 'Content-Length': '0'} 2024-04-16T19:51:13.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf7122ebdb96417d-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.120 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.120 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.120 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.120 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.120 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.120 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.120 DEBUG:boto:Data: 2024-04-16T19:51:13.120 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.121 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.121 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.121 DEBUG:boto:Params: {} 2024-04-16T19:51:13.121 DEBUG:boto:Token: None 2024-04-16T19:51:13.121 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QlsvIzhFvVEV2wbsqyyz6uPdtoA= 2024-04-16T19:51:13.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QlsvIzhFvVEV2wbsqyyz6uPdtoA=', 'Content-Length': '0'} 2024-04-16T19:51:13.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab6c96c41eb291a-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:51:13.124 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.124 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.125 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.125 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.125 DEBUG:boto:Method: GET 2024-04-16T19:51:13.125 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.125 DEBUG:boto:Data: 2024-04-16T19:51:13.125 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.125 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.125 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.125 DEBUG:boto:Params: {} 2024-04-16T19:51:13.126 DEBUG:boto:Token: None 2024-04-16T19:51:13.126 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY= 2024-04-16T19:51:13.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY=', 'Content-Length': '0'} 2024-04-16T19:51:13.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09bcbe98664d476-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.170 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.171 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.171 DEBUG:boto:path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.171 DEBUG:boto:auth_path=/yrjefh-47/b?versionId=null 2024-04-16T19:51:13.171 DEBUG:boto:Method: GET 2024-04-16T19:51:13.171 DEBUG:boto:Path: /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.171 DEBUG:boto:Data: 2024-04-16T19:51:13.171 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.172 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.172 DEBUG:boto:Params: {} 2024-04-16T19:51:13.172 DEBUG:boto:Token: None 2024-04-16T19:51:13.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b?versionId=null 2024-04-16T19:51:13.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY= 2024-04-16T19:51:13.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:htdgdIKxoRMUVZ/RxFefwbi8zCY=', 'Content-Length': '0'} 2024-04-16T19:51:13.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54d2567c2737d29-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.175 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.175 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.176 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.176 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.176 DEBUG:boto:Method: GET 2024-04-16T19:51:13.176 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:51:13.176 DEBUG:boto:Data: 2024-04-16T19:51:13.176 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.176 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.176 DEBUG:boto:Params: {} 2024-04-16T19:51:13.177 DEBUG:boto:Token: None 2024-04-16T19:51:13.177 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.177 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044fbcbf9ca2aeff9-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.180 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.180 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.180 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.180 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.180 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.181 DEBUG:boto:Method: GET 2024-04-16T19:51:13.181 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:51:13.181 DEBUG:boto:Data: 2024-04-16T19:51:13.181 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.181 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.181 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.181 DEBUG:boto:Params: {} 2024-04-16T19:51:13.181 DEBUG:boto:Token: None 2024-04-16T19:51:13.181 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.182 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14ee87761c7c03a-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.185 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.185 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.185 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.185 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.185 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:51:13.185 DEBUG:boto:Data: 2024-04-16T19:51:13.186 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.186 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.186 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.186 DEBUG:boto:Params: {} 2024-04-16T19:51:13.186 DEBUG:boto:Token: None 2024-04-16T19:51:13.186 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b 2024-04-16T19:51:13.186 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A/YBNmtQjzYhc624cxLCXNAKsyU= 2024-04-16T19:51:13.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/YBNmtQjzYhc624cxLCXNAKsyU=', 'Content-Length': '0'} 2024-04-16T19:51:13.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ff2cbbd9a89296-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.189 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.189 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.189 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.189 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:51:13.190 DEBUG:boto:Data: 2024-04-16T19:51:13.190 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.190 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.190 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.190 DEBUG:boto:Params: {} 2024-04-16T19:51:13.190 DEBUG:boto:Token: None 2024-04-16T19:51:13.190 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b 2024-04-16T19:51:13.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A/YBNmtQjzYhc624cxLCXNAKsyU= 2024-04-16T19:51:13.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/YBNmtQjzYhc624cxLCXNAKsyU=', 'Content-Length': '0'} 2024-04-16T19:51:13.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200abe09c25a6f68-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:51:13.194 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.194 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.195 DEBUG:boto:Method: GET 2024-04-16T19:51:13.195 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:51:13.195 DEBUG:boto:Data: 2024-04-16T19:51:13.195 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.195 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.195 DEBUG:boto:Params: {} 2024-04-16T19:51:13.196 DEBUG:boto:Token: None 2024-04-16T19:51:13.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b 2024-04-16T19:51:13.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qjZyzPN1r/Qzcz9h0WXRMx0qGog= 2024-04-16T19:51:13.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qjZyzPN1r/Qzcz9h0WXRMx0qGog=', 'Content-Length': '0'} 2024-04-16T19:51:13.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1480af25c9d05e-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.242 DEBUG:boto:path=/yrjefh-47/b 2024-04-16T19:51:13.243 DEBUG:boto:auth_path=/yrjefh-47/b 2024-04-16T19:51:13.243 DEBUG:boto:Method: GET 2024-04-16T19:51:13.243 DEBUG:boto:Path: /yrjefh-47/b 2024-04-16T19:51:13.243 DEBUG:boto:Data: 2024-04-16T19:51:13.243 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.244 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.244 DEBUG:boto:Params: {} 2024-04-16T19:51:13.244 DEBUG:boto:Token: None 2024-04-16T19:51:13.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/b 2024-04-16T19:51:13.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qjZyzPN1r/Qzcz9h0WXRMx0qGog= 2024-04-16T19:51:13.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qjZyzPN1r/Qzcz9h0WXRMx0qGog=', 'Content-Length': '0'} 2024-04-16T19:51:13.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004723e89bf06a60a9-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.247 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.248 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.248 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.248 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.248 DEBUG:boto:Method: GET 2024-04-16T19:51:13.248 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:51:13.248 DEBUG:boto:Data: 2024-04-16T19:51:13.248 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.248 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.249 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.249 DEBUG:boto:Params: {} 2024-04-16T19:51:13.249 DEBUG:boto:Token: None 2024-04-16T19:51:13.249 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.249 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4db8bad3ff3416b-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.294 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.295 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.295 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.296 DEBUG:boto:path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.296 DEBUG:boto:auth_path=/yrjefh-47/?prefix=b 2024-04-16T19:51:13.296 DEBUG:boto:Method: GET 2024-04-16T19:51:13.296 DEBUG:boto:Path: /yrjefh-47/?prefix=b 2024-04-16T19:51:13.296 DEBUG:boto:Data: 2024-04-16T19:51:13.296 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.296 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.297 DEBUG:boto:Params: {} 2024-04-16T19:51:13.297 DEBUG:boto:Token: None 2024-04-16T19:51:13.297 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.297 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007042db8429a9bb8c-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.300 DEBUG:boto:b'yrjefh-47b1000falseb2024-04-16T19:50:10.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:51:13.301 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.301 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.301 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.301 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.302 DEBUG:boto:Method: GET 2024-04-16T19:51:13.302 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.302 DEBUG:boto:Data: 2024-04-16T19:51:13.302 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.302 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.302 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.302 DEBUG:boto:Params: {} 2024-04-16T19:51:13.303 DEBUG:boto:Token: None 2024-04-16T19:51:13.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk= 2024-04-16T19:51:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk=', 'Content-Length': '0'} 2024-04-16T19:51:13.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003925905abdbb0938-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.346 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.347 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.347 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.347 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.347 DEBUG:boto:Method: GET 2024-04-16T19:51:13.347 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.348 DEBUG:boto:Data: 2024-04-16T19:51:13.348 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.348 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.348 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.348 DEBUG:boto:Params: {} 2024-04-16T19:51:13.348 DEBUG:boto:Token: None 2024-04-16T19:51:13.349 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.349 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk= 2024-04-16T19:51:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk=', 'Content-Length': '0'} 2024-04-16T19:51:13.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15328770fc1fe99-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.352 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.352 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.352 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.352 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.352 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.352 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.353 DEBUG:boto:Data: 2024-04-16T19:51:13.353 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.353 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.353 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.353 DEBUG:boto:Params: {} 2024-04-16T19:51:13.353 DEBUG:boto:Token: None 2024-04-16T19:51:13.354 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.354 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flkScUpnkbzvNIaFE8bxWqKKBso= 2024-04-16T19:51:13.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flkScUpnkbzvNIaFE8bxWqKKBso=', 'Content-Length': '0'} 2024-04-16T19:51:13.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c675790a7ae8b897-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.356 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.357 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.357 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.357 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.357 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.357 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.357 DEBUG:boto:Data: 2024-04-16T19:51:13.357 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.358 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.358 DEBUG:boto:Params: {} 2024-04-16T19:51:13.358 DEBUG:boto:Token: None 2024-04-16T19:51:13.358 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flkScUpnkbzvNIaFE8bxWqKKBso= 2024-04-16T19:51:13.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flkScUpnkbzvNIaFE8bxWqKKBso=', 'Content-Length': '0'} 2024-04-16T19:51:13.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e309b457989329-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:51:13.361 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.362 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.362 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.362 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.362 DEBUG:boto:Method: GET 2024-04-16T19:51:13.362 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.362 DEBUG:boto:Data: 2024-04-16T19:51:13.362 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.363 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.363 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.363 DEBUG:boto:Params: {} 2024-04-16T19:51:13.363 DEBUG:boto:Token: None 2024-04-16T19:51:13.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk= 2024-04-16T19:51:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk=', 'Content-Length': '0'} 2024-04-16T19:51:13.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3257d5d58148f9-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.406 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.407 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.407 DEBUG:boto:path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.407 DEBUG:boto:auth_path=/yrjefh-47/c?versionId=null 2024-04-16T19:51:13.407 DEBUG:boto:Method: GET 2024-04-16T19:51:13.407 DEBUG:boto:Path: /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.407 DEBUG:boto:Data: 2024-04-16T19:51:13.407 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.408 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.408 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.408 DEBUG:boto:Params: {} 2024-04-16T19:51:13.408 DEBUG:boto:Token: None 2024-04-16T19:51:13.408 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c?versionId=null 2024-04-16T19:51:13.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk= 2024-04-16T19:51:13.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6lCKL9+5EOcoF0Ura0wfWtrgDMk=', 'Content-Length': '0'} 2024-04-16T19:51:13.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e10c51fe5cd0e6-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.411 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.412 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.412 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.412 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.412 DEBUG:boto:Method: GET 2024-04-16T19:51:13.412 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:51:13.412 DEBUG:boto:Data: 2024-04-16T19:51:13.412 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.413 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.413 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.413 DEBUG:boto:Params: {} 2024-04-16T19:51:13.413 DEBUG:boto:Token: None 2024-04-16T19:51:13.413 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a9f1a557ca54f2-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.458 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.459 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.459 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.459 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.459 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.460 DEBUG:boto:Method: GET 2024-04-16T19:51:13.460 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:51:13.460 DEBUG:boto:Data: 2024-04-16T19:51:13.460 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.460 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.460 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.460 DEBUG:boto:Params: {} 2024-04-16T19:51:13.461 DEBUG:boto:Token: None 2024-04-16T19:51:13.461 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.461 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e140317ef5f9408-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.464 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.464 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.465 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.465 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.465 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:51:13.465 DEBUG:boto:Data: 2024-04-16T19:51:13.465 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.465 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.465 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.466 DEBUG:boto:Params: {} 2024-04-16T19:51:13.466 DEBUG:boto:Token: None 2024-04-16T19:51:13.466 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c 2024-04-16T19:51:13.466 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P84vAVwZVV0C+N5XEB8b6WTThj4= 2024-04-16T19:51:13.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P84vAVwZVV0C+N5XEB8b6WTThj4=', 'Content-Length': '0'} 2024-04-16T19:51:13.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afece691149bf77-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.469 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.469 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.469 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.469 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:51:13.469 DEBUG:boto:Data: 2024-04-16T19:51:13.470 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.470 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.470 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.470 DEBUG:boto:Params: {} 2024-04-16T19:51:13.470 DEBUG:boto:Token: None 2024-04-16T19:51:13.470 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c 2024-04-16T19:51:13.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P84vAVwZVV0C+N5XEB8b6WTThj4= 2024-04-16T19:51:13.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P84vAVwZVV0C+N5XEB8b6WTThj4=', 'Content-Length': '0'} 2024-04-16T19:51:13.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003112844da90e9c88-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:51:13.473 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.474 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.474 DEBUG:boto:Method: GET 2024-04-16T19:51:13.474 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:51:13.474 DEBUG:boto:Data: 2024-04-16T19:51:13.474 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.474 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.475 DEBUG:boto:Params: {} 2024-04-16T19:51:13.475 DEBUG:boto:Token: None 2024-04-16T19:51:13.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c 2024-04-16T19:51:13.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrtjBi+qCRAdtRGjfBeJxHWeh1c= 2024-04-16T19:51:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrtjBi+qCRAdtRGjfBeJxHWeh1c=', 'Content-Length': '0'} 2024-04-16T19:51:13.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f23dedf7b906c2-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.518 DEBUG:boto:path=/yrjefh-47/c 2024-04-16T19:51:13.519 DEBUG:boto:auth_path=/yrjefh-47/c 2024-04-16T19:51:13.519 DEBUG:boto:Method: GET 2024-04-16T19:51:13.519 DEBUG:boto:Path: /yrjefh-47/c 2024-04-16T19:51:13.519 DEBUG:boto:Data: 2024-04-16T19:51:13.519 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.519 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.520 DEBUG:boto:Params: {} 2024-04-16T19:51:13.520 DEBUG:boto:Token: None 2024-04-16T19:51:13.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/c 2024-04-16T19:51:13.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrtjBi+qCRAdtRGjfBeJxHWeh1c= 2024-04-16T19:51:13.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrtjBi+qCRAdtRGjfBeJxHWeh1c=', 'Content-Length': '0'} 2024-04-16T19:51:13.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b37c708c0f92f1c3-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.523 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.524 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.524 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.524 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.524 DEBUG:boto:Method: GET 2024-04-16T19:51:13.524 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:51:13.524 DEBUG:boto:Data: 2024-04-16T19:51:13.525 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.525 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.525 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.525 DEBUG:boto:Params: {} 2024-04-16T19:51:13.525 DEBUG:boto:Token: None 2024-04-16T19:51:13.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006207629b9d99c03e-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.528 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.529 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.529 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.529 DEBUG:boto:path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.529 DEBUG:boto:auth_path=/yrjefh-47/?prefix=c 2024-04-16T19:51:13.529 DEBUG:boto:Method: GET 2024-04-16T19:51:13.530 DEBUG:boto:Path: /yrjefh-47/?prefix=c 2024-04-16T19:51:13.530 DEBUG:boto:Data: 2024-04-16T19:51:13.530 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.530 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.530 DEBUG:boto:Params: {} 2024-04-16T19:51:13.531 DEBUG:boto:Token: None 2024-04-16T19:51:13.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca4235b258a17eb-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.534 DEBUG:boto:b'yrjefh-47c1000falsec2024-04-16T19:50:10.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:51:13.534 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.534 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.534 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.535 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.535 DEBUG:boto:Method: GET 2024-04-16T19:51:13.535 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.535 DEBUG:boto:Data: 2024-04-16T19:51:13.535 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.535 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.535 DEBUG:boto:Params: {} 2024-04-16T19:51:13.536 DEBUG:boto:Token: None 2024-04-16T19:51:13.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs= 2024-04-16T19:51:13.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs=', 'Content-Length': '0'} 2024-04-16T19:51:13.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057417b953b2c99d0-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.582 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.583 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.583 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.583 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.583 DEBUG:boto:Method: GET 2024-04-16T19:51:13.583 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.583 DEBUG:boto:Data: 2024-04-16T19:51:13.584 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.584 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.584 DEBUG:boto:Params: {} 2024-04-16T19:51:13.584 DEBUG:boto:Token: None 2024-04-16T19:51:13.584 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs= 2024-04-16T19:51:13.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs=', 'Content-Length': '0'} 2024-04-16T19:51:13.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d290b17068a3447e-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.588 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.588 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.588 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.588 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.588 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.589 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.589 DEBUG:boto:Data: 2024-04-16T19:51:13.589 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.589 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.589 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.589 DEBUG:boto:Params: {} 2024-04-16T19:51:13.589 DEBUG:boto:Token: None 2024-04-16T19:51:13.590 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.590 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a3+Mvy52nHa970NsAy0irAAY3YE= 2024-04-16T19:51:13.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a3+Mvy52nHa970NsAy0irAAY3YE=', 'Content-Length': '0'} 2024-04-16T19:51:13.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057cb2963d6e9838e-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.593 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.593 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.593 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.593 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.593 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.594 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.594 DEBUG:boto:Data: 2024-04-16T19:51:13.594 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.594 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.594 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.594 DEBUG:boto:Params: {} 2024-04-16T19:51:13.594 DEBUG:boto:Token: None 2024-04-16T19:51:13.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a3+Mvy52nHa970NsAy0irAAY3YE= 2024-04-16T19:51:13.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a3+Mvy52nHa970NsAy0irAAY3YE=', 'Content-Length': '0'} 2024-04-16T19:51:13.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf64e26b13099926-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:51:13.598 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.598 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.598 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.599 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.599 DEBUG:boto:Method: GET 2024-04-16T19:51:13.599 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.599 DEBUG:boto:Data: 2024-04-16T19:51:13.599 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.599 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.600 DEBUG:boto:Params: {} 2024-04-16T19:51:13.600 DEBUG:boto:Token: None 2024-04-16T19:51:13.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs= 2024-04-16T19:51:13.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs=', 'Content-Length': '0'} 2024-04-16T19:51:13.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be4cc04d489a0f4a-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.646 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.647 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.647 DEBUG:boto:path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.647 DEBUG:boto:auth_path=/yrjefh-47/d?versionId=null 2024-04-16T19:51:13.647 DEBUG:boto:Method: GET 2024-04-16T19:51:13.647 DEBUG:boto:Path: /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.647 DEBUG:boto:Data: 2024-04-16T19:51:13.648 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.648 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.648 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.648 DEBUG:boto:Params: {} 2024-04-16T19:51:13.648 DEBUG:boto:Token: None 2024-04-16T19:51:13.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d?versionId=null 2024-04-16T19:51:13.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs= 2024-04-16T19:51:13.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIEItQFvR1RHLtQooLalR4/0DQs=', 'Content-Length': '0'} 2024-04-16T19:51:13.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f524ee09427f78d6-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.652 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.652 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.652 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.652 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.652 DEBUG:boto:Method: GET 2024-04-16T19:51:13.653 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:51:13.653 DEBUG:boto:Data: 2024-04-16T19:51:13.653 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.653 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.653 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.653 DEBUG:boto:Params: {} 2024-04-16T19:51:13.653 DEBUG:boto:Token: None 2024-04-16T19:51:13.654 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.654 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896f5eab37a536f1-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.657 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.657 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.657 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.657 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.658 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.658 DEBUG:boto:Method: GET 2024-04-16T19:51:13.658 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:51:13.658 DEBUG:boto:Data: 2024-04-16T19:51:13.658 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.658 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.658 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.658 DEBUG:boto:Params: {} 2024-04-16T19:51:13.659 DEBUG:boto:Token: None 2024-04-16T19:51:13.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f2a6683f4b8f8d6-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.661 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.662 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.662 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.662 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.662 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:51:13.662 DEBUG:boto:Data: 2024-04-16T19:51:13.663 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.663 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.663 DEBUG:boto:Params: {} 2024-04-16T19:51:13.663 DEBUG:boto:Token: None 2024-04-16T19:51:13.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d 2024-04-16T19:51:13.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jSyksA6PNGoyvbx0I6TMn6kkIIw= 2024-04-16T19:51:13.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jSyksA6PNGoyvbx0I6TMn6kkIIw=', 'Content-Length': '0'} 2024-04-16T19:51:13.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00f55a2dd443df6-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.666 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.666 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.666 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.666 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:51:13.666 DEBUG:boto:Data: 2024-04-16T19:51:13.667 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.667 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.667 DEBUG:boto:Params: {} 2024-04-16T19:51:13.667 DEBUG:boto:Token: None 2024-04-16T19:51:13.667 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d 2024-04-16T19:51:13.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jSyksA6PNGoyvbx0I6TMn6kkIIw= 2024-04-16T19:51:13.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jSyksA6PNGoyvbx0I6TMn6kkIIw=', 'Content-Length': '0'} 2024-04-16T19:51:13.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000723118f3a1f78e42-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:51:13.670 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.671 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.671 DEBUG:boto:Method: GET 2024-04-16T19:51:13.671 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:51:13.671 DEBUG:boto:Data: 2024-04-16T19:51:13.671 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.671 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.671 DEBUG:boto:Params: {} 2024-04-16T19:51:13.672 DEBUG:boto:Token: None 2024-04-16T19:51:13.672 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d 2024-04-16T19:51:13.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yigl6rWM7SUrxvcuuF2ov6Ce6CM= 2024-04-16T19:51:13.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yigl6rWM7SUrxvcuuF2ov6Ce6CM=', 'Content-Length': '0'} 2024-04-16T19:51:13.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c634551e12b4e0f-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.718 DEBUG:boto:path=/yrjefh-47/d 2024-04-16T19:51:13.718 DEBUG:boto:auth_path=/yrjefh-47/d 2024-04-16T19:51:13.719 DEBUG:boto:Method: GET 2024-04-16T19:51:13.719 DEBUG:boto:Path: /yrjefh-47/d 2024-04-16T19:51:13.719 DEBUG:boto:Data: 2024-04-16T19:51:13.719 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.719 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.719 DEBUG:boto:Params: {} 2024-04-16T19:51:13.719 DEBUG:boto:Token: None 2024-04-16T19:51:13.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/d 2024-04-16T19:51:13.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yigl6rWM7SUrxvcuuF2ov6Ce6CM= 2024-04-16T19:51:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yigl6rWM7SUrxvcuuF2ov6Ce6CM=', 'Content-Length': '0'} 2024-04-16T19:51:13.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e77f9aa73712e28-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.722 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.723 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.723 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.723 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.723 DEBUG:boto:Method: GET 2024-04-16T19:51:13.723 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:51:13.723 DEBUG:boto:Data: 2024-04-16T19:51:13.723 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.723 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.723 DEBUG:boto:Params: {} 2024-04-16T19:51:13.724 DEBUG:boto:Token: None 2024-04-16T19:51:13.724 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a9a48b143cf58f-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.726 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.727 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.727 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.727 DEBUG:boto:path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.727 DEBUG:boto:auth_path=/yrjefh-47/?prefix=d 2024-04-16T19:51:13.727 DEBUG:boto:Method: GET 2024-04-16T19:51:13.727 DEBUG:boto:Path: /yrjefh-47/?prefix=d 2024-04-16T19:51:13.727 DEBUG:boto:Data: 2024-04-16T19:51:13.728 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.728 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.728 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.728 DEBUG:boto:Params: {} 2024-04-16T19:51:13.728 DEBUG:boto:Token: None 2024-04-16T19:51:13.728 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54d9612fc3bd13d-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.731 DEBUG:boto:b'yrjefh-47d1000falsed2024-04-16T19:50:10.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:51:13.732 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.732 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.732 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.732 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.732 DEBUG:boto:Method: GET 2024-04-16T19:51:13.732 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.732 DEBUG:boto:Data: 2024-04-16T19:51:13.732 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.732 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.733 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.733 DEBUG:boto:Params: {} 2024-04-16T19:51:13.733 DEBUG:boto:Token: None 2024-04-16T19:51:13.733 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU= 2024-04-16T19:51:13.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU=', 'Content-Length': '0'} 2024-04-16T19:51:13.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:51:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f955de028371461c-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.778 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.779 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.779 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.779 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.779 DEBUG:boto:Method: GET 2024-04-16T19:51:13.779 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.779 DEBUG:boto:Data: 2024-04-16T19:51:13.779 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.779 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.780 DEBUG:boto:Params: {} 2024-04-16T19:51:13.780 DEBUG:boto:Token: None 2024-04-16T19:51:13.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU= 2024-04-16T19:51:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU=', 'Content-Length': '0'} 2024-04-16T19:51:13.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cb1b3c45a6907d-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.783 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.783 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.783 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.783 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.783 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.783 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.783 DEBUG:boto:Data: 2024-04-16T19:51:13.784 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.784 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.784 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.784 DEBUG:boto:Params: {} 2024-04-16T19:51:13.784 DEBUG:boto:Token: None 2024-04-16T19:51:13.784 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KxhDgQR03fLHRG0B8K5T3My6BLA= 2024-04-16T19:51:13.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KxhDgQR03fLHRG0B8K5T3My6BLA=', 'Content-Length': '0'} 2024-04-16T19:51:13.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:51:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eaea7debe650b2d-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.787 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.787 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.787 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.787 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.787 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.787 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.787 DEBUG:boto:Data: 2024-04-16T19:51:13.787 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.788 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.788 DEBUG:boto:Params: {} 2024-04-16T19:51:13.788 DEBUG:boto:Token: None 2024-04-16T19:51:13.788 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KxhDgQR03fLHRG0B8K5T3My6BLA= 2024-04-16T19:51:13.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KxhDgQR03fLHRG0B8K5T3My6BLA=', 'Content-Length': '0'} 2024-04-16T19:51:13.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005698f8714f5b44bb-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:51:13.791 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.791 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.791 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.791 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.791 DEBUG:boto:Method: GET 2024-04-16T19:51:13.791 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.792 DEBUG:boto:Data: 2024-04-16T19:51:13.792 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.792 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.792 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.792 DEBUG:boto:Params: {} 2024-04-16T19:51:13.792 DEBUG:boto:Token: None 2024-04-16T19:51:13.792 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU= 2024-04-16T19:51:13.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU=', 'Content-Length': '0'} 2024-04-16T19:51:13.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:51:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092aadbb453d341a9-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.838 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.839 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.839 DEBUG:boto:path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.839 DEBUG:boto:auth_path=/yrjefh-47/e?versionId=null 2024-04-16T19:51:13.839 DEBUG:boto:Method: GET 2024-04-16T19:51:13.839 DEBUG:boto:Path: /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.839 DEBUG:boto:Data: 2024-04-16T19:51:13.839 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.840 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.840 DEBUG:boto:Params: {} 2024-04-16T19:51:13.840 DEBUG:boto:Token: None 2024-04-16T19:51:13.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e?versionId=null 2024-04-16T19:51:13.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU= 2024-04-16T19:51:13.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SDtoZ9it6S2En1ePuKEz+2JVGaU=', 'Content-Length': '0'} 2024-04-16T19:51:13.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d54698eec6c7922-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.843 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.843 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.843 DEBUG:boto:path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.843 DEBUG:boto:auth_path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.844 DEBUG:boto:Method: GET 2024-04-16T19:51:13.844 DEBUG:boto:Path: /yrjefh-47/?prefix=e 2024-04-16T19:51:13.844 DEBUG:boto:Data: 2024-04-16T19:51:13.844 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.844 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.844 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.844 DEBUG:boto:Params: {} 2024-04-16T19:51:13.844 DEBUG:boto:Token: None 2024-04-16T19:51:13.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.845 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51c2d56255bda00-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.890 DEBUG:boto:b'yrjefh-47e1000falsee2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.891 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.891 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.891 DEBUG:boto:path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.891 DEBUG:boto:auth_path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.891 DEBUG:boto:Method: GET 2024-04-16T19:51:13.891 DEBUG:boto:Path: /yrjefh-47/?prefix=e 2024-04-16T19:51:13.891 DEBUG:boto:Data: 2024-04-16T19:51:13.892 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.892 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.892 DEBUG:boto:Params: {} 2024-04-16T19:51:13.892 DEBUG:boto:Token: None 2024-04-16T19:51:13.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8aa855883898f47-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.895 DEBUG:boto:b'yrjefh-47e1000falsee2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:13.895 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.895 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.896 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.896 DEBUG:boto:Path: /yrjefh-47/e 2024-04-16T19:51:13.896 DEBUG:boto:Data: 2024-04-16T19:51:13.896 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.896 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.896 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.896 DEBUG:boto:Params: {} 2024-04-16T19:51:13.896 DEBUG:boto:Token: None 2024-04-16T19:51:13.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e 2024-04-16T19:51:13.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtyKVPdMZXjicCo+Aio/owrEd1s= 2024-04-16T19:51:13.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtyKVPdMZXjicCo+Aio/owrEd1s=', 'Content-Length': '0'} 2024-04-16T19:51:13.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:51:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082736a7ac4dbc842-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.899 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.900 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.900 DEBUG:boto:Method: HEAD 2024-04-16T19:51:13.900 DEBUG:boto:Path: /yrjefh-47/e 2024-04-16T19:51:13.900 DEBUG:boto:Data: 2024-04-16T19:51:13.900 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.900 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.900 DEBUG:boto:Params: {} 2024-04-16T19:51:13.900 DEBUG:boto:Token: None 2024-04-16T19:51:13.901 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e 2024-04-16T19:51:13.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtyKVPdMZXjicCo+Aio/owrEd1s= 2024-04-16T19:51:13.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtyKVPdMZXjicCo+Aio/owrEd1s=', 'Content-Length': '0'} 2024-04-16T19:51:13.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e3c64b6e516161-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:51:13.904 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.904 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.904 DEBUG:boto:Method: GET 2024-04-16T19:51:13.904 DEBUG:boto:Path: /yrjefh-47/e 2024-04-16T19:51:13.904 DEBUG:boto:Data: 2024-04-16T19:51:13.904 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.904 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.904 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.904 DEBUG:boto:Params: {} 2024-04-16T19:51:13.905 DEBUG:boto:Token: None 2024-04-16T19:51:13.905 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e 2024-04-16T19:51:13.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZmnT5fpvstzoHkhpdo5iqXgB5w= 2024-04-16T19:51:13.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZmnT5fpvstzoHkhpdo5iqXgB5w=', 'Content-Length': '0'} 2024-04-16T19:51:13.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:51:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000efc08a74d4887f0-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.950 DEBUG:boto:path=/yrjefh-47/e 2024-04-16T19:51:13.951 DEBUG:boto:auth_path=/yrjefh-47/e 2024-04-16T19:51:13.951 DEBUG:boto:Method: GET 2024-04-16T19:51:13.951 DEBUG:boto:Path: /yrjefh-47/e 2024-04-16T19:51:13.951 DEBUG:boto:Data: 2024-04-16T19:51:13.951 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:13.951 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.951 DEBUG:boto:Params: {} 2024-04-16T19:51:13.952 DEBUG:boto:Token: None 2024-04-16T19:51:13.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/e 2024-04-16T19:51:13.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZmnT5fpvstzoHkhpdo5iqXgB5w= 2024-04-16T19:51:13.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZmnT5fpvstzoHkhpdo5iqXgB5w=', 'Content-Length': '0'} 2024-04-16T19:51:13.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726e024fdefd58be-00661ed6b1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:13.955 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:13.955 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:13.955 DEBUG:boto:path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.955 DEBUG:boto:auth_path=/yrjefh-47/?prefix=e 2024-04-16T19:51:13.955 DEBUG:boto:Method: GET 2024-04-16T19:51:13.955 DEBUG:boto:Path: /yrjefh-47/?prefix=e 2024-04-16T19:51:13.955 DEBUG:boto:Data: 2024-04-16T19:51:13.955 DEBUG:boto:Headers: {} 2024-04-16T19:51:13.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:13.956 DEBUG:boto:Port: 8000 2024-04-16T19:51:13.956 DEBUG:boto:Params: {} 2024-04-16T19:51:13.956 DEBUG:boto:Token: None 2024-04-16T19:51:13.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:13 GMT /yrjefh-47/ 2024-04-16T19:51:13.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8= 2024-04-16T19:51:13.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wIV4v5Xa9JqRert+FzIQyr6z6f8=', 'Content-Length': '0'} 2024-04-16T19:51:13.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4e539b376ccc20-00661ed6b1-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.002 DEBUG:boto:b'yrjefh-47e1000falsee2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.003 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.003 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.003 DEBUG:boto:path=/yrjefh-47/?prefix=e 2024-04-16T19:51:14.003 DEBUG:boto:auth_path=/yrjefh-47/?prefix=e 2024-04-16T19:51:14.003 DEBUG:boto:Method: GET 2024-04-16T19:51:14.003 DEBUG:boto:Path: /yrjefh-47/?prefix=e 2024-04-16T19:51:14.003 DEBUG:boto:Data: 2024-04-16T19:51:14.004 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.004 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.004 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.004 DEBUG:boto:Params: {} 2024-04-16T19:51:14.004 DEBUG:boto:Token: None 2024-04-16T19:51:14.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e489e2ad1eddc990-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.007 DEBUG:boto:b'yrjefh-47e1000falsee2024-04-16T19:50:22.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:51:14.008 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.008 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.008 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.008 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.008 DEBUG:boto:Method: GET 2024-04-16T19:51:14.008 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.008 DEBUG:boto:Data: 2024-04-16T19:51:14.008 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.008 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.009 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.009 DEBUG:boto:Params: {} 2024-04-16T19:51:14.009 DEBUG:boto:Token: None 2024-04-16T19:51:14.009 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.009 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o= 2024-04-16T19:51:14.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o=', 'Content-Length': '0'} 2024-04-16T19:51:14.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:45 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d1d75185d8a1b7-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.054 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.055 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.055 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.055 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.055 DEBUG:boto:Method: GET 2024-04-16T19:51:14.055 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.055 DEBUG:boto:Data: 2024-04-16T19:51:14.055 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.056 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.056 DEBUG:boto:Params: {} 2024-04-16T19:51:14.056 DEBUG:boto:Token: None 2024-04-16T19:51:14.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o= 2024-04-16T19:51:14.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o=', 'Content-Length': '0'} 2024-04-16T19:51:14.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b837402989cb67-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.059 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.060 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.060 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.060 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.060 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.060 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.060 DEBUG:boto:Data: 2024-04-16T19:51:14.060 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.061 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.061 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.061 DEBUG:boto:Params: {} 2024-04-16T19:51:14.061 DEBUG:boto:Token: None 2024-04-16T19:51:14.061 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3r7i3ZZclUdwATWKBmrcIC5D2R8= 2024-04-16T19:51:14.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3r7i3ZZclUdwATWKBmrcIC5D2R8=', 'Content-Length': '0'} 2024-04-16T19:51:14.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:45 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072228b042b20f878-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.064 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.064 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.064 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.064 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.064 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.065 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.065 DEBUG:boto:Data: 2024-04-16T19:51:14.065 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.065 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.065 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.065 DEBUG:boto:Params: {} 2024-04-16T19:51:14.065 DEBUG:boto:Token: None 2024-04-16T19:51:14.065 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.066 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3r7i3ZZclUdwATWKBmrcIC5D2R8= 2024-04-16T19:51:14.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3r7i3ZZclUdwATWKBmrcIC5D2R8=', 'Content-Length': '0'} 2024-04-16T19:51:14.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d18af478ad4a0a9-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:51:14.068 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.069 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.069 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.069 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.069 DEBUG:boto:Method: GET 2024-04-16T19:51:14.069 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.069 DEBUG:boto:Data: 2024-04-16T19:51:14.069 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.069 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.069 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.070 DEBUG:boto:Params: {} 2024-04-16T19:51:14.070 DEBUG:boto:Token: None 2024-04-16T19:51:14.070 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o= 2024-04-16T19:51:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o=', 'Content-Length': '0'} 2024-04-16T19:51:14.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:45 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be02291207a7ebe-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.114 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.115 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.115 DEBUG:boto:path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.115 DEBUG:boto:auth_path=/yrjefh-47/f?versionId=null 2024-04-16T19:51:14.115 DEBUG:boto:Method: GET 2024-04-16T19:51:14.115 DEBUG:boto:Path: /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.115 DEBUG:boto:Data: 2024-04-16T19:51:14.115 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.116 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.116 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.116 DEBUG:boto:Params: {} 2024-04-16T19:51:14.116 DEBUG:boto:Token: None 2024-04-16T19:51:14.116 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f?versionId=null 2024-04-16T19:51:14.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o= 2024-04-16T19:51:14.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flNn8qKnGwA1eAXNw1SjSFFaO1o=', 'Content-Length': '0'} 2024-04-16T19:51:14.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d1e803606a4b6d-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.119 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.119 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.119 DEBUG:boto:path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.119 DEBUG:boto:auth_path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.120 DEBUG:boto:Method: GET 2024-04-16T19:51:14.120 DEBUG:boto:Path: /yrjefh-47/?prefix=f 2024-04-16T19:51:14.120 DEBUG:boto:Data: 2024-04-16T19:51:14.120 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.120 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.120 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.120 DEBUG:boto:Params: {} 2024-04-16T19:51:14.120 DEBUG:boto:Token: None 2024-04-16T19:51:14.121 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000db6ba38781d0d8c-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.123 DEBUG:boto:b'yrjefh-47f1000falsef2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.124 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.124 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.124 DEBUG:boto:path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.124 DEBUG:boto:auth_path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.124 DEBUG:boto:Method: GET 2024-04-16T19:51:14.125 DEBUG:boto:Path: /yrjefh-47/?prefix=f 2024-04-16T19:51:14.125 DEBUG:boto:Data: 2024-04-16T19:51:14.125 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.125 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.125 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.125 DEBUG:boto:Params: {} 2024-04-16T19:51:14.125 DEBUG:boto:Token: None 2024-04-16T19:51:14.125 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588afd3ab7c3c579-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.128 DEBUG:boto:b'yrjefh-47f1000falsef2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.129 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.129 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.129 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.129 DEBUG:boto:Path: /yrjefh-47/f 2024-04-16T19:51:14.129 DEBUG:boto:Data: 2024-04-16T19:51:14.129 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.129 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.130 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.130 DEBUG:boto:Params: {} 2024-04-16T19:51:14.130 DEBUG:boto:Token: None 2024-04-16T19:51:14.130 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f 2024-04-16T19:51:14.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dHi7/dh8Pc+kaNUS4e0xKQ8I07k= 2024-04-16T19:51:14.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dHi7/dh8Pc+kaNUS4e0xKQ8I07k=', 'Content-Length': '0'} 2024-04-16T19:51:14.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:45 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67fffb640de1d3c-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.133 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.133 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.133 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.133 DEBUG:boto:Path: /yrjefh-47/f 2024-04-16T19:51:14.133 DEBUG:boto:Data: 2024-04-16T19:51:14.134 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.134 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.134 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.134 DEBUG:boto:Params: {} 2024-04-16T19:51:14.134 DEBUG:boto:Token: None 2024-04-16T19:51:14.134 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f 2024-04-16T19:51:14.134 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dHi7/dh8Pc+kaNUS4e0xKQ8I07k= 2024-04-16T19:51:14.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dHi7/dh8Pc+kaNUS4e0xKQ8I07k=', 'Content-Length': '0'} 2024-04-16T19:51:14.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5be4a04e915754-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:51:14.137 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.137 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.137 DEBUG:boto:Method: GET 2024-04-16T19:51:14.137 DEBUG:boto:Path: /yrjefh-47/f 2024-04-16T19:51:14.138 DEBUG:boto:Data: 2024-04-16T19:51:14.138 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.138 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.138 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.138 DEBUG:boto:Params: {} 2024-04-16T19:51:14.138 DEBUG:boto:Token: None 2024-04-16T19:51:14.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f 2024-04-16T19:51:14.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Kswz+8kShbX6m4sibcTcOshvXA= 2024-04-16T19:51:14.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Kswz+8kShbX6m4sibcTcOshvXA=', 'Content-Length': '0'} 2024-04-16T19:51:14.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:45 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016353451a5272c7c-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.183 DEBUG:boto:path=/yrjefh-47/f 2024-04-16T19:51:14.183 DEBUG:boto:auth_path=/yrjefh-47/f 2024-04-16T19:51:14.183 DEBUG:boto:Method: GET 2024-04-16T19:51:14.183 DEBUG:boto:Path: /yrjefh-47/f 2024-04-16T19:51:14.183 DEBUG:boto:Data: 2024-04-16T19:51:14.183 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.183 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.184 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.184 DEBUG:boto:Params: {} 2024-04-16T19:51:14.184 DEBUG:boto:Token: None 2024-04-16T19:51:14.184 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/f 2024-04-16T19:51:14.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Kswz+8kShbX6m4sibcTcOshvXA= 2024-04-16T19:51:14.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Kswz+8kShbX6m4sibcTcOshvXA=', 'Content-Length': '0'} 2024-04-16T19:51:14.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6a161b3dfd9a0b-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.187 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.187 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.187 DEBUG:boto:path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.187 DEBUG:boto:auth_path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.187 DEBUG:boto:Method: GET 2024-04-16T19:51:14.187 DEBUG:boto:Path: /yrjefh-47/?prefix=f 2024-04-16T19:51:14.188 DEBUG:boto:Data: 2024-04-16T19:51:14.188 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.188 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.188 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.188 DEBUG:boto:Params: {} 2024-04-16T19:51:14.188 DEBUG:boto:Token: None 2024-04-16T19:51:14.188 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0bd12bbd741676a-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.234 DEBUG:boto:b'yrjefh-47f1000falsef2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.235 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.235 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.235 DEBUG:boto:path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.235 DEBUG:boto:auth_path=/yrjefh-47/?prefix=f 2024-04-16T19:51:14.235 DEBUG:boto:Method: GET 2024-04-16T19:51:14.236 DEBUG:boto:Path: /yrjefh-47/?prefix=f 2024-04-16T19:51:14.236 DEBUG:boto:Data: 2024-04-16T19:51:14.236 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.236 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.236 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.236 DEBUG:boto:Params: {} 2024-04-16T19:51:14.236 DEBUG:boto:Token: None 2024-04-16T19:51:14.237 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee91a0647b2f9165-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.239 DEBUG:boto:b'yrjefh-47f1000falsef2024-04-16T19:50:22.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:51:14.240 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.240 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.240 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.240 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.241 DEBUG:boto:Method: GET 2024-04-16T19:51:14.241 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.241 DEBUG:boto:Data: 2024-04-16T19:51:14.241 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.241 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.241 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.241 DEBUG:boto:Params: {} 2024-04-16T19:51:14.241 DEBUG:boto:Token: None 2024-04-16T19:51:14.242 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek= 2024-04-16T19:51:14.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek=', 'Content-Length': '0'} 2024-04-16T19:51:14.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:54 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002786896345320b74-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.286 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.287 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.287 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.287 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.287 DEBUG:boto:Method: GET 2024-04-16T19:51:14.287 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.287 DEBUG:boto:Data: 2024-04-16T19:51:14.287 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.288 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.288 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.288 DEBUG:boto:Params: {} 2024-04-16T19:51:14.288 DEBUG:boto:Token: None 2024-04-16T19:51:14.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek= 2024-04-16T19:51:14.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek=', 'Content-Length': '0'} 2024-04-16T19:51:14.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb49a49af28846d0-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.291 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.292 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.292 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.292 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.292 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.292 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.292 DEBUG:boto:Data: 2024-04-16T19:51:14.292 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.293 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.293 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.293 DEBUG:boto:Params: {} 2024-04-16T19:51:14.293 DEBUG:boto:Token: None 2024-04-16T19:51:14.293 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7imcNUp/MvPy2sCP8ATNgFukY6E= 2024-04-16T19:51:14.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7imcNUp/MvPy2sCP8ATNgFukY6E=', 'Content-Length': '0'} 2024-04-16T19:51:14.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:54 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebed9f624b42fc76-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.296 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.297 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.297 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.297 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.297 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.297 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.297 DEBUG:boto:Data: 2024-04-16T19:51:14.297 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.297 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.298 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.298 DEBUG:boto:Params: {} 2024-04-16T19:51:14.298 DEBUG:boto:Token: None 2024-04-16T19:51:14.298 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.298 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7imcNUp/MvPy2sCP8ATNgFukY6E= 2024-04-16T19:51:14.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7imcNUp/MvPy2sCP8ATNgFukY6E=', 'Content-Length': '0'} 2024-04-16T19:51:14.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c07e90eae8c13938-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:51:14.301 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.301 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.301 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.302 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.302 DEBUG:boto:Method: GET 2024-04-16T19:51:14.302 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.302 DEBUG:boto:Data: 2024-04-16T19:51:14.302 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.302 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.302 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.303 DEBUG:boto:Params: {} 2024-04-16T19:51:14.303 DEBUG:boto:Token: None 2024-04-16T19:51:14.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek= 2024-04-16T19:51:14.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek=', 'Content-Length': '0'} 2024-04-16T19:51:14.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:54 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ffd8437fa0584e-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.346 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.347 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.347 DEBUG:boto:path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.347 DEBUG:boto:auth_path=/yrjefh-47/g?versionId=null 2024-04-16T19:51:14.347 DEBUG:boto:Method: GET 2024-04-16T19:51:14.347 DEBUG:boto:Path: /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.347 DEBUG:boto:Data: 2024-04-16T19:51:14.347 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.348 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.348 DEBUG:boto:Params: {} 2024-04-16T19:51:14.348 DEBUG:boto:Token: None 2024-04-16T19:51:14.348 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g?versionId=null 2024-04-16T19:51:14.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek= 2024-04-16T19:51:14.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mBxmNv/U4kx4GOJvedQUIdiKRek=', 'Content-Length': '0'} 2024-04-16T19:51:14.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c9f90330d0879a-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.351 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.351 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.351 DEBUG:boto:path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.352 DEBUG:boto:auth_path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.352 DEBUG:boto:Method: GET 2024-04-16T19:51:14.352 DEBUG:boto:Path: /yrjefh-47/?prefix=g 2024-04-16T19:51:14.352 DEBUG:boto:Data: 2024-04-16T19:51:14.352 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.352 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.352 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.352 DEBUG:boto:Params: {} 2024-04-16T19:51:14.353 DEBUG:boto:Token: None 2024-04-16T19:51:14.353 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036856abe883111c9-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.356 DEBUG:boto:b'yrjefh-47g1000falseg2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.356 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.357 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.357 DEBUG:boto:path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.357 DEBUG:boto:auth_path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.357 DEBUG:boto:Method: GET 2024-04-16T19:51:14.357 DEBUG:boto:Path: /yrjefh-47/?prefix=g 2024-04-16T19:51:14.357 DEBUG:boto:Data: 2024-04-16T19:51:14.357 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.358 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.358 DEBUG:boto:Params: {} 2024-04-16T19:51:14.358 DEBUG:boto:Token: None 2024-04-16T19:51:14.358 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a79ca29d9bbb7d-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.361 DEBUG:boto:b'yrjefh-47g1000falseg2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.362 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.362 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.362 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.362 DEBUG:boto:Path: /yrjefh-47/g 2024-04-16T19:51:14.362 DEBUG:boto:Data: 2024-04-16T19:51:14.362 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.362 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.363 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.363 DEBUG:boto:Params: {} 2024-04-16T19:51:14.363 DEBUG:boto:Token: None 2024-04-16T19:51:14.363 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g 2024-04-16T19:51:14.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5LzTx98093UxeL1BTqtbkTICKA= 2024-04-16T19:51:14.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5LzTx98093UxeL1BTqtbkTICKA=', 'Content-Length': '0'} 2024-04-16T19:51:14.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:54 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32bf757aa4b00f2-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.366 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.366 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.366 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.366 DEBUG:boto:Path: /yrjefh-47/g 2024-04-16T19:51:14.366 DEBUG:boto:Data: 2024-04-16T19:51:14.366 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.367 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.367 DEBUG:boto:Params: {} 2024-04-16T19:51:14.367 DEBUG:boto:Token: None 2024-04-16T19:51:14.367 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g 2024-04-16T19:51:14.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5LzTx98093UxeL1BTqtbkTICKA= 2024-04-16T19:51:14.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5LzTx98093UxeL1BTqtbkTICKA=', 'Content-Length': '0'} 2024-04-16T19:51:14.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621a40404a78ff5e-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:51:14.370 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.370 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.370 DEBUG:boto:Method: GET 2024-04-16T19:51:14.370 DEBUG:boto:Path: /yrjefh-47/g 2024-04-16T19:51:14.371 DEBUG:boto:Data: 2024-04-16T19:51:14.371 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.371 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.371 DEBUG:boto:Params: {} 2024-04-16T19:51:14.371 DEBUG:boto:Token: None 2024-04-16T19:51:14.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g 2024-04-16T19:51:14.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OWqvpq6J4nhXjiNZMvkQakOjjBo= 2024-04-16T19:51:14.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OWqvpq6J4nhXjiNZMvkQakOjjBo=', 'Content-Length': '0'} 2024-04-16T19:51:14.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:54 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd05f1a32e723fa-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.414 DEBUG:boto:path=/yrjefh-47/g 2024-04-16T19:51:14.415 DEBUG:boto:auth_path=/yrjefh-47/g 2024-04-16T19:51:14.415 DEBUG:boto:Method: GET 2024-04-16T19:51:14.415 DEBUG:boto:Path: /yrjefh-47/g 2024-04-16T19:51:14.415 DEBUG:boto:Data: 2024-04-16T19:51:14.415 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.415 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.415 DEBUG:boto:Params: {} 2024-04-16T19:51:14.416 DEBUG:boto:Token: None 2024-04-16T19:51:14.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/g 2024-04-16T19:51:14.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OWqvpq6J4nhXjiNZMvkQakOjjBo= 2024-04-16T19:51:14.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OWqvpq6J4nhXjiNZMvkQakOjjBo=', 'Content-Length': '0'} 2024-04-16T19:51:14.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ed299d26dcbbf1-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.419 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.419 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.419 DEBUG:boto:path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.419 DEBUG:boto:auth_path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.419 DEBUG:boto:Method: GET 2024-04-16T19:51:14.419 DEBUG:boto:Path: /yrjefh-47/?prefix=g 2024-04-16T19:51:14.420 DEBUG:boto:Data: 2024-04-16T19:51:14.420 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.420 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.420 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.420 DEBUG:boto:Params: {} 2024-04-16T19:51:14.420 DEBUG:boto:Token: None 2024-04-16T19:51:14.420 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bcd6c4da7cd96b7-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.466 DEBUG:boto:b'yrjefh-47g1000falseg2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.467 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.467 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.467 DEBUG:boto:path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.467 DEBUG:boto:auth_path=/yrjefh-47/?prefix=g 2024-04-16T19:51:14.467 DEBUG:boto:Method: GET 2024-04-16T19:51:14.468 DEBUG:boto:Path: /yrjefh-47/?prefix=g 2024-04-16T19:51:14.468 DEBUG:boto:Data: 2024-04-16T19:51:14.468 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.468 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.468 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.468 DEBUG:boto:Params: {} 2024-04-16T19:51:14.468 DEBUG:boto:Token: None 2024-04-16T19:51:14.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050fa9417a23b846-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.471 DEBUG:boto:b'yrjefh-47g1000falseg2024-04-16T19:50:22.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:51:14.472 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.472 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.472 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.472 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.473 DEBUG:boto:Method: GET 2024-04-16T19:51:14.473 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.473 DEBUG:boto:Data: 2024-04-16T19:51:14.473 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.473 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.473 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.473 DEBUG:boto:Params: {} 2024-04-16T19:51:14.473 DEBUG:boto:Token: None 2024-04-16T19:51:14.474 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0= 2024-04-16T19:51:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0=', 'Content-Length': '0'} 2024-04-16T19:51:14.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:35 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff5ed92b5ab6586-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.518 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.519 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.519 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.519 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.519 DEBUG:boto:Method: GET 2024-04-16T19:51:14.519 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.519 DEBUG:boto:Data: 2024-04-16T19:51:14.519 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.520 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.520 DEBUG:boto:Params: {} 2024-04-16T19:51:14.520 DEBUG:boto:Token: None 2024-04-16T19:51:14.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0= 2024-04-16T19:51:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0=', 'Content-Length': '0'} 2024-04-16T19:51:14.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a680e3c416a31b-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.523 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.523 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.524 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.524 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.524 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.524 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.524 DEBUG:boto:Data: 2024-04-16T19:51:14.524 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.524 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.524 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.524 DEBUG:boto:Params: {} 2024-04-16T19:51:14.525 DEBUG:boto:Token: None 2024-04-16T19:51:14.525 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEE25AmjjWRN1UV5kQtuqkemo8Q= 2024-04-16T19:51:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEE25AmjjWRN1UV5kQtuqkemo8Q=', 'Content-Length': '0'} 2024-04-16T19:51:14.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:35 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54fa6aa8b52238a-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.528 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.528 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.528 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.528 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.528 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.528 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.528 DEBUG:boto:Data: 2024-04-16T19:51:14.529 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.529 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.529 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.529 DEBUG:boto:Params: {} 2024-04-16T19:51:14.529 DEBUG:boto:Token: None 2024-04-16T19:51:14.529 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEE25AmjjWRN1UV5kQtuqkemo8Q= 2024-04-16T19:51:14.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEE25AmjjWRN1UV5kQtuqkemo8Q=', 'Content-Length': '0'} 2024-04-16T19:51:14.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbcbc8f710c250cf-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:51:14.533 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.533 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.533 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.533 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.533 DEBUG:boto:Method: GET 2024-04-16T19:51:14.533 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.533 DEBUG:boto:Data: 2024-04-16T19:51:14.534 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.534 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.534 DEBUG:boto:Params: {} 2024-04-16T19:51:14.534 DEBUG:boto:Token: None 2024-04-16T19:51:14.534 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0= 2024-04-16T19:51:14.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0=', 'Content-Length': '0'} 2024-04-16T19:51:14.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:35 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef78b8d5ff7eb7a1-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.578 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.579 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.579 DEBUG:boto:path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.579 DEBUG:boto:auth_path=/yrjefh-47/h?versionId=null 2024-04-16T19:51:14.579 DEBUG:boto:Method: GET 2024-04-16T19:51:14.579 DEBUG:boto:Path: /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.579 DEBUG:boto:Data: 2024-04-16T19:51:14.579 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.580 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.580 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.580 DEBUG:boto:Params: {} 2024-04-16T19:51:14.580 DEBUG:boto:Token: None 2024-04-16T19:51:14.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h?versionId=null 2024-04-16T19:51:14.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0= 2024-04-16T19:51:14.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aCXd5hxqGaaPsHX3uKSTYyhFCZ0=', 'Content-Length': '0'} 2024-04-16T19:51:14.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014494599dbcc42f9-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.583 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.583 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.584 DEBUG:boto:path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.584 DEBUG:boto:auth_path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.584 DEBUG:boto:Method: GET 2024-04-16T19:51:14.584 DEBUG:boto:Path: /yrjefh-47/?prefix=h 2024-04-16T19:51:14.584 DEBUG:boto:Data: 2024-04-16T19:51:14.584 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.584 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.584 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.584 DEBUG:boto:Params: {} 2024-04-16T19:51:14.585 DEBUG:boto:Token: None 2024-04-16T19:51:14.585 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b65384ff713f758-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.588 DEBUG:boto:b'yrjefh-47h1000falseh2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.588 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.588 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.588 DEBUG:boto:path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.589 DEBUG:boto:auth_path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.589 DEBUG:boto:Method: GET 2024-04-16T19:51:14.589 DEBUG:boto:Path: /yrjefh-47/?prefix=h 2024-04-16T19:51:14.589 DEBUG:boto:Data: 2024-04-16T19:51:14.589 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.589 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.589 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.589 DEBUG:boto:Params: {} 2024-04-16T19:51:14.590 DEBUG:boto:Token: None 2024-04-16T19:51:14.590 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.590 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3039d60f896d570-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.634 DEBUG:boto:b'yrjefh-47h1000falseh2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.635 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.635 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.635 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.635 DEBUG:boto:Path: /yrjefh-47/h 2024-04-16T19:51:14.635 DEBUG:boto:Data: 2024-04-16T19:51:14.635 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.636 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.636 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.636 DEBUG:boto:Params: {} 2024-04-16T19:51:14.636 DEBUG:boto:Token: None 2024-04-16T19:51:14.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h 2024-04-16T19:51:14.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oDiwU9fq2RqxRd3TVe7pp/Gs7vc= 2024-04-16T19:51:14.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oDiwU9fq2RqxRd3TVe7pp/Gs7vc=', 'Content-Length': '0'} 2024-04-16T19:51:14.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:35 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f6d6498901ac7b5-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.639 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.640 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.640 DEBUG:boto:Method: HEAD 2024-04-16T19:51:14.640 DEBUG:boto:Path: /yrjefh-47/h 2024-04-16T19:51:14.640 DEBUG:boto:Data: 2024-04-16T19:51:14.640 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.640 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.640 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.640 DEBUG:boto:Params: {} 2024-04-16T19:51:14.641 DEBUG:boto:Token: None 2024-04-16T19:51:14.641 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h 2024-04-16T19:51:14.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oDiwU9fq2RqxRd3TVe7pp/Gs7vc= 2024-04-16T19:51:14.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oDiwU9fq2RqxRd3TVe7pp/Gs7vc=', 'Content-Length': '0'} 2024-04-16T19:51:14.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008658ffcde0a26f3f-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:51:14.644 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.644 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.644 DEBUG:boto:Method: GET 2024-04-16T19:51:14.644 DEBUG:boto:Path: /yrjefh-47/h 2024-04-16T19:51:14.644 DEBUG:boto:Data: 2024-04-16T19:51:14.644 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.644 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.645 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.645 DEBUG:boto:Params: {} 2024-04-16T19:51:14.645 DEBUG:boto:Token: None 2024-04-16T19:51:14.645 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h 2024-04-16T19:51:14.645 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eqbOO0rj+t+ENt/3yRqEssTv+FY= 2024-04-16T19:51:14.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eqbOO0rj+t+ENt/3yRqEssTv+FY=', 'Content-Length': '0'} 2024-04-16T19:51:14.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-47:b482a2a9-ef36-488b-9666-ada656453328.4812.16'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:50:35 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d2e9b94893efb4-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.690 DEBUG:boto:path=/yrjefh-47/h 2024-04-16T19:51:14.691 DEBUG:boto:auth_path=/yrjefh-47/h 2024-04-16T19:51:14.691 DEBUG:boto:Method: GET 2024-04-16T19:51:14.691 DEBUG:boto:Path: /yrjefh-47/h 2024-04-16T19:51:14.691 DEBUG:boto:Data: 2024-04-16T19:51:14.691 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.691 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.692 DEBUG:boto:Params: {} 2024-04-16T19:51:14.692 DEBUG:boto:Token: None 2024-04-16T19:51:14.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/h 2024-04-16T19:51:14.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eqbOO0rj+t+ENt/3yRqEssTv+FY= 2024-04-16T19:51:14.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eqbOO0rj+t+ENt/3yRqEssTv+FY=', 'Content-Length': '0'} 2024-04-16T19:51:14.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:50:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811251d92f3003a9-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.695 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.695 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.695 DEBUG:boto:path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.696 DEBUG:boto:auth_path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.696 DEBUG:boto:Method: GET 2024-04-16T19:51:14.696 DEBUG:boto:Path: /yrjefh-47/?prefix=h 2024-04-16T19:51:14.696 DEBUG:boto:Data: 2024-04-16T19:51:14.696 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.696 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:14.696 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.696 DEBUG:boto:Params: {} 2024-04-16T19:51:14.697 DEBUG:boto:Token: None 2024-04-16T19:51:14.697 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.697 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089699b9f432bb173-00661ed6b2-4911-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.742 DEBUG:boto:b'yrjefh-47h1000falseh2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.743 DEBUG:boto:path=/yrjefh-47/ 2024-04-16T19:51:14.743 DEBUG:boto:auth_path=/yrjefh-47/ 2024-04-16T19:51:14.743 DEBUG:boto:path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.743 DEBUG:boto:auth_path=/yrjefh-47/?prefix=h 2024-04-16T19:51:14.743 DEBUG:boto:Method: GET 2024-04-16T19:51:14.744 DEBUG:boto:Path: /yrjefh-47/?prefix=h 2024-04-16T19:51:14.744 DEBUG:boto:Data: 2024-04-16T19:51:14.744 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.744 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.744 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.744 DEBUG:boto:Params: {} 2024-04-16T19:51:14.744 DEBUG:boto:Token: None 2024-04-16T19:51:14.744 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-47/ 2024-04-16T19:51:14.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84= 2024-04-16T19:51:14.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5i9mRBrNmRHMIHxlgjeQBDUgv84=', 'Content-Length': '0'} 2024-04-16T19:51:14.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a714b035e243f59-00661ed6b2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.747 DEBUG:boto:b'yrjefh-47h1000falseh2024-04-16T19:50:22.454Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:51:14.748 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-47 zones={test-zone2, test-zone1} 2024-04-16T19:51:14.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2024-04-16T19:51:14.750 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:14.750 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:14.750 DEBUG:boto:Method: PUT 2024-04-16T19:51:14.750 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:14.751 DEBUG:boto:Data: 2024-04-16T19:51:14.751 DEBUG:boto:Headers: {} 2024-04-16T19:51:14.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:14.751 DEBUG:boto:Port: 8000 2024-04-16T19:51:14.751 DEBUG:boto:Params: {} 2024-04-16T19:51:14.751 DEBUG:boto:Token: None 2024-04-16T19:51:14.751 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:51:14 GMT /yrjefh-48/ 2024-04-16T19:51:14.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jUNA3IxPfQKExec6Ry5cq7Y8n9s= 2024-04-16T19:51:14.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jUNA3IxPfQKExec6Ry5cq7Y8n9s=', 'Content-Length': '0'} 2024-04-16T19:51:14.770 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dea2e7603780b840-00661ed6b2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:14.770 DEBUG:rgw_multi.tests:created bucket=yrjefh-48 2024-04-16T19:51:14.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:51:14.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:51:14.771 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:51:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:51:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:51:14.880 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:51:14.903 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297074.766987_602.1","last_update":"2024-04-16T19:51:14.766987Z"}] 2024-04-16T19:51:14.904 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297074.766987_602.1'} 2024-04-16T19:51:14.904 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:51:14.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:51:14.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:51:14.905 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:51:14.960 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:51:14.961 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:51:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:15.014+0000 7efffd941800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:51:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:15.016+0000 7efffd941800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:51:15.040 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297074.766987_602.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:51:14.766987Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:51:15Z"} 2024-04-16T19:51:15.040 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297074.766987_602.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:51:14.766987Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:51:15Z"} 2024-04-16T19:51:15.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297074.766987_602.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:51:14.766987Z', 'realm_epoch': 2}}] 2024-04-16T19:51:15.040 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297074.766987_602.1'} 2024-04-16T19:51:15.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297074.766987_602.1'} 2024-04-16T19:51:15.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:51:15.041 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T19:51:15.041 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T19:51:15.042 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:51:15.040+0000 7fe48c473640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 41247) UID: 0 2024-04-16T19:51:15.042 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:51:15.040+0000 7fe48fccc880 -1 shutting down 2024-04-16T19:51:16.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T19:51:21.143 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T19:51:21.144 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.144 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.144 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.144 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.144 DEBUG:boto:Data: 2024-04-16T19:51:21.144 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.144 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.144 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.145 DEBUG:boto:Params: {} 2024-04-16T19:51:21.145 DEBUG:boto:Token: None 2024-04-16T19:51:21.145 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.145 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000803f435071bdb436-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.149 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:51:21.149 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:51:21.149 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.149 DEBUG:boto:Path: /yrjefh-48/a 2024-04-16T19:51:21.149 DEBUG:boto:Data: 2024-04-16T19:51:21.150 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.150 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.150 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.150 DEBUG:boto:Params: {} 2024-04-16T19:51:21.150 DEBUG:boto:Token: None 2024-04-16T19:51:21.150 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/a 2024-04-16T19:51:21.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:G1zGAfcYfWaK66zVlopgW3o5Cvg= 2024-04-16T19:51:21.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:G1zGAfcYfWaK66zVlopgW3o5Cvg='} 2024-04-16T19:51:21.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc58a18d4c4ce9c9-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.159 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.159 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.159 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.160 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.160 DEBUG:boto:Data: 2024-04-16T19:51:21.160 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.160 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.160 DEBUG:boto:Params: {} 2024-04-16T19:51:21.160 DEBUG:boto:Token: None 2024-04-16T19:51:21.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005736c5ddbd172c3a-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.164 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:51:21.164 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:51:21.164 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.164 DEBUG:boto:Path: /yrjefh-48/b 2024-04-16T19:51:21.164 DEBUG:boto:Data: 2024-04-16T19:51:21.164 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.164 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.165 DEBUG:boto:Params: {} 2024-04-16T19:51:21.165 DEBUG:boto:Token: None 2024-04-16T19:51:21.165 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/b 2024-04-16T19:51:21.165 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mi8KhU5eWi05Idzh99iDDGlmlHM= 2024-04-16T19:51:21.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mi8KhU5eWi05Idzh99iDDGlmlHM='} 2024-04-16T19:51:21.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000746c2f268368144a-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.211 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.211 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.211 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.211 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.211 DEBUG:boto:Data: 2024-04-16T19:51:21.211 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.212 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.212 DEBUG:boto:Params: {} 2024-04-16T19:51:21.212 DEBUG:boto:Token: None 2024-04-16T19:51:21.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000244f9b12bb5df363-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.216 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:51:21.216 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:51:21.216 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.216 DEBUG:boto:Path: /yrjefh-48/c 2024-04-16T19:51:21.216 DEBUG:boto:Data: 2024-04-16T19:51:21.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.216 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.216 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.217 DEBUG:boto:Params: {} 2024-04-16T19:51:21.217 DEBUG:boto:Token: None 2024-04-16T19:51:21.217 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/c 2024-04-16T19:51:21.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j1is8MAiVoFSdYbBNWOnUFBRls0= 2024-04-16T19:51:21.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j1is8MAiVoFSdYbBNWOnUFBRls0='} 2024-04-16T19:51:21.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d15711f9916145d6-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.224 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.224 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.225 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.225 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.225 DEBUG:boto:Data: 2024-04-16T19:51:21.225 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.225 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.225 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.225 DEBUG:boto:Params: {} 2024-04-16T19:51:21.225 DEBUG:boto:Token: None 2024-04-16T19:51:21.226 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.226 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acb7870fd72cf8fa-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.229 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:51:21.229 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:51:21.229 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.229 DEBUG:boto:Path: /yrjefh-48/d 2024-04-16T19:51:21.230 DEBUG:boto:Data: 2024-04-16T19:51:21.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.230 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.230 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.230 DEBUG:boto:Params: {} 2024-04-16T19:51:21.230 DEBUG:boto:Token: None 2024-04-16T19:51:21.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/d 2024-04-16T19:51:21.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YhmyysWn0YifLlX995Lpviv+k6Q= 2024-04-16T19:51:21.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YhmyysWn0YifLlX995Lpviv+k6Q='} 2024-04-16T19:51:21.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096cd537ea91685f0-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-48', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:51:21.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-48', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:51:21.275 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-48 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:51:21.331 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:51:21.331 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:51:21.398 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:51:21.398 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-48 2024-04-16T19:51:21.404 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 4 2024-04-16T19:51:21.459 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.459 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.460 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.460 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.460 DEBUG:boto:Data: 2024-04-16T19:51:21.460 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.460 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.460 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.460 DEBUG:boto:Params: {} 2024-04-16T19:51:21.460 DEBUG:boto:Token: None 2024-04-16T19:51:21.461 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.461 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da6f8aa4b833f33e-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.464 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:51:21.464 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:51:21.464 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.464 DEBUG:boto:Path: /yrjefh-48/e 2024-04-16T19:51:21.464 DEBUG:boto:Data: 2024-04-16T19:51:21.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.464 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.464 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.465 DEBUG:boto:Params: {} 2024-04-16T19:51:21.465 DEBUG:boto:Token: None 2024-04-16T19:51:21.465 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/e 2024-04-16T19:51:21.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aeiLSYSS2jVXQMjFhE5Sp7fdFbk= 2024-04-16T19:51:21.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aeiLSYSS2jVXQMjFhE5Sp7fdFbk='} 2024-04-16T19:51:21.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d7f1f2b33b3bbfc-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.471 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.471 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.472 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.472 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.472 DEBUG:boto:Data: 2024-04-16T19:51:21.472 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.472 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.472 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.472 DEBUG:boto:Params: {} 2024-04-16T19:51:21.472 DEBUG:boto:Token: None 2024-04-16T19:51:21.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002288c87f7a23da13-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.475 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:51:21.475 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:51:21.476 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.476 DEBUG:boto:Path: /yrjefh-48/f 2024-04-16T19:51:21.476 DEBUG:boto:Data: 2024-04-16T19:51:21.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.476 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.476 DEBUG:boto:Params: {} 2024-04-16T19:51:21.476 DEBUG:boto:Token: None 2024-04-16T19:51:21.477 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/f 2024-04-16T19:51:21.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m8rS9nEPCNCC4BW+DU0tEChMK7c= 2024-04-16T19:51:21.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m8rS9nEPCNCC4BW+DU0tEChMK7c='} 2024-04-16T19:51:21.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d822557e8bafe040-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.519 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.519 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.519 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.519 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.519 DEBUG:boto:Data: 2024-04-16T19:51:21.519 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.520 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.520 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.520 DEBUG:boto:Params: {} 2024-04-16T19:51:21.520 DEBUG:boto:Token: None 2024-04-16T19:51:21.520 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f405a60b6db8dcf-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.523 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:51:21.523 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:51:21.524 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.524 DEBUG:boto:Path: /yrjefh-48/g 2024-04-16T19:51:21.524 DEBUG:boto:Data: 2024-04-16T19:51:21.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.524 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.524 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.524 DEBUG:boto:Params: {} 2024-04-16T19:51:21.524 DEBUG:boto:Token: None 2024-04-16T19:51:21.525 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/g 2024-04-16T19:51:21.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zH2DqNh3wINSFhkmnSebYxgOF1I= 2024-04-16T19:51:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zH2DqNh3wINSFhkmnSebYxgOF1I='} 2024-04-16T19:51:21.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd135c8d8ff83a2e-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.531 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:51:21.531 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:51:21.531 DEBUG:boto:Method: HEAD 2024-04-16T19:51:21.531 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:51:21.531 DEBUG:boto:Data: 2024-04-16T19:51:21.531 DEBUG:boto:Headers: {} 2024-04-16T19:51:21.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.532 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.532 DEBUG:boto:Params: {} 2024-04-16T19:51:21.532 DEBUG:boto:Token: None 2024-04-16T19:51:21.532 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/ 2024-04-16T19:51:21.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg= 2024-04-16T19:51:21.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ASkVtcBACUY9Hyd4zToaISUhrkg=', 'Content-Length': '0'} 2024-04-16T19:51:21.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f00b388a2130be3f-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.535 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:51:21.535 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:51:21.535 DEBUG:boto:Method: PUT 2024-04-16T19:51:21.535 DEBUG:boto:Path: /yrjefh-48/h 2024-04-16T19:51:21.535 DEBUG:boto:Data: 2024-04-16T19:51:21.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:51:21.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:51:21.536 DEBUG:boto:Port: 8000 2024-04-16T19:51:21.536 DEBUG:boto:Params: {} 2024-04-16T19:51:21.536 DEBUG:boto:Token: None 2024-04-16T19:51:21.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:51:21 GMT /yrjefh-48/h 2024-04-16T19:51:21.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WK03IyRJmhYXG8qvoU4wZln1gFE= 2024-04-16T19:51:21.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:51:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WK03IyRJmhYXG8qvoU4wZln1gFE='} 2024-04-16T19:51:21.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c78291dcfc659e04-00661ed6b9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:51:21.579 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T19:51:21.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T19:51:21.582 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T19:51:21.583 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T19:51:21.607 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:51:21.607 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:51:21.608 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T19:51:21.608 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T19:51:21.778 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:51:21.776+0000 7f994602e880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:51:22.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T19:51:22.609 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:51:22.630 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:51:22.630 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:51:22.631 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:51:22.631 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T19:51:22.632 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-48 2024-04-16T19:51:22.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-48', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:51:22.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-48', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:51:22.633 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-48 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:51:22.727 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:51:22.727 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:51:22.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:22.782+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:27.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:27.782+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:32.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:32.782+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:37.784+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:42.785+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:47.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:47.786+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:52.787+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:51:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:51:57.788+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:02.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:02.788+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:07.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:07.789+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:12.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:12.789+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:17.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:17.791+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:22.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:22.792+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:27.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:27.793+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:32.794+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:37.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:37.794+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:42.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:42.795+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:47.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:47.796+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:52.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:52.796+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:52:57.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:52:57.798+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:02.799+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:07.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:07.800+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:12.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:12.800+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:17.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:17.801+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:22.802+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:27.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:27.803+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:32.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:32.804+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:37.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:37.805+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:42.805+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:47.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:47.806+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:52.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:52.806+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:53:57.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:53:57.807+0000 7fd45e920800 1 waiting to reach incremental sync.. 2024-04-16T19:54:02.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:02.809+0000 7fd45e920800 1 bucket sync caught up with source: 2024-04-16T19:54:02.812 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000003.34.6, 00000000002.33.6, 00000000002.32.6] 2024-04-16T19:54:02.812 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000003.34.6, 00000000002.33.6, 00000000002.32.6] 2024-04-16T19:54:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:02.809+0000 7fd45e920800 0 bucket checkpoint complete 2024-04-16T19:54:02.838 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-48 2024-04-16T19:54:02.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-48', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:02.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-48', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:02.839 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-48 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:54:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:02.903 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:54:02.962+0000 7fe9faa24800 1 bucket sync caught up with source: 2024-04-16T19:54:02.965 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.121.6, , , 00000000001.74.6, 00000000001.139.6, , , 00000000002.209.6, , 00000000002.152.6] 2024-04-16T19:54:02.965 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.121.6, , , 00000000001.74.6, 00000000001.139.6, , , 00000000002.209.6, , 00000000002.152.6] 2024-04-16T19:54:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:54:02.962+0000 7fe9faa24800 0 bucket checkpoint complete 2024-04-16T19:54:02.987 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-48 zones={test-zone2, test-zone1} 2024-04-16T19:54:02.987 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:02.987 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:02.987 DEBUG:boto:Method: HEAD 2024-04-16T19:54:02.987 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:54:02.988 DEBUG:boto:Data: 2024-04-16T19:54:02.988 DEBUG:boto:Headers: {} 2024-04-16T19:54:02.988 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:02.988 DEBUG:boto:Port: 8000 2024-04-16T19:54:02.988 DEBUG:boto:Params: {} 2024-04-16T19:54:02.989 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:54:02.989 DEBUG:boto:Token: None 2024-04-16T19:54:02.989 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:02 GMT /yrjefh-48/ 2024-04-16T19:54:02.989 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MHJWDLihfUg0001Cp3DD5Ke8+IM= 2024-04-16T19:54:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MHJWDLihfUg0001Cp3DD5Ke8+IM=', 'Content-Length': '0'} 2024-04-16T19:54:02.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009722a3f1e9eaa085-00661ed75a-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:02.997 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:02.997 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:02.997 DEBUG:boto:Method: HEAD 2024-04-16T19:54:02.997 DEBUG:boto:Path: /yrjefh-48/ 2024-04-16T19:54:02.997 DEBUG:boto:Data: 2024-04-16T19:54:02.997 DEBUG:boto:Headers: {} 2024-04-16T19:54:02.997 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:02.998 DEBUG:boto:Port: 8000 2024-04-16T19:54:02.998 DEBUG:boto:Params: {} 2024-04-16T19:54:02.998 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:54:02.998 DEBUG:boto:Token: None 2024-04-16T19:54:02.998 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:02 GMT /yrjefh-48/ 2024-04-16T19:54:02.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MHJWDLihfUg0001Cp3DD5Ke8+IM= 2024-04-16T19:54:02.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MHJWDLihfUg0001Cp3DD5Ke8+IM=', 'Content-Length': '0'} 2024-04-16T19:54:03.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012c9e0183c07a581-00661ed75a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.003 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:54:03.003 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.003 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.003 DEBUG:boto:path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.003 DEBUG:boto:auth_path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.003 DEBUG:boto:Method: GET 2024-04-16T19:54:03.003 DEBUG:boto:Path: /yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.003 DEBUG:boto:Data: 2024-04-16T19:54:03.004 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.004 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.004 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.004 DEBUG:boto:Params: {} 2024-04-16T19:54:03.004 DEBUG:boto:Token: None 2024-04-16T19:54:03.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/?versions 2024-04-16T19:54:03.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA= 2024-04-16T19:54:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA=', 'Content-Length': '0'} 2024-04-16T19:54:03.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b20ca099edde25e-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.007 DEBUG:boto:b'yrjefh-48999falseanulltrue2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.008 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:54:03.008 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:54:03.008 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:54:03.009 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:54:03.009 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.009 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.009 DEBUG:boto:path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.009 DEBUG:boto:auth_path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.010 DEBUG:boto:Method: GET 2024-04-16T19:54:03.010 DEBUG:boto:Path: /yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.010 DEBUG:boto:Data: 2024-04-16T19:54:03.010 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.010 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.010 DEBUG:boto:Params: {} 2024-04-16T19:54:03.010 DEBUG:boto:Token: None 2024-04-16T19:54:03.010 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/?versions 2024-04-16T19:54:03.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA= 2024-04-16T19:54:03.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA=', 'Content-Length': '0'} 2024-04-16T19:54:03.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003570d4fc2f9b809a-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.013 DEBUG:boto:b'yrjefh-48999falseanulltrue2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:54:03.014 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:54:03.015 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.015 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.015 DEBUG:boto:path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.015 DEBUG:boto:auth_path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.015 DEBUG:boto:Method: GET 2024-04-16T19:54:03.015 DEBUG:boto:Path: /yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.015 DEBUG:boto:Data: 2024-04-16T19:54:03.015 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.015 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.015 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.015 DEBUG:boto:Params: {} 2024-04-16T19:54:03.016 DEBUG:boto:Token: None 2024-04-16T19:54:03.016 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/?versions 2024-04-16T19:54:03.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA= 2024-04-16T19:54:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA=', 'Content-Length': '0'} 2024-04-16T19:54:03.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e37efb35bd44b973-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.018 DEBUG:boto:b'yrjefh-48999falseanulltrue2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.019 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.019 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.019 DEBUG:boto:path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.019 DEBUG:boto:auth_path=/yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.019 DEBUG:boto:Method: GET 2024-04-16T19:54:03.019 DEBUG:boto:Path: /yrjefh-48/?versions&max-keys=999 2024-04-16T19:54:03.020 DEBUG:boto:Data: 2024-04-16T19:54:03.020 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.020 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.020 DEBUG:boto:Params: {} 2024-04-16T19:54:03.020 DEBUG:boto:Token: None 2024-04-16T19:54:03.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/?versions 2024-04-16T19:54:03.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA= 2024-04-16T19:54:03.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YKIIZeassVmn5ah/qTJ4Wv4iBkA=', 'Content-Length': '0'} 2024-04-16T19:54:03.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007704a52847dd3988-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.023 DEBUG:boto:b'yrjefh-48999falseanulltrue2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:03.024 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.024 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.024 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.024 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.024 DEBUG:boto:Method: GET 2024-04-16T19:54:03.024 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.024 DEBUG:boto:Data: 2024-04-16T19:54:03.024 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.025 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.025 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.025 DEBUG:boto:Params: {} 2024-04-16T19:54:03.025 DEBUG:boto:Token: None 2024-04-16T19:54:03.025 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY= 2024-04-16T19:54:03.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY=', 'Content-Length': '0'} 2024-04-16T19:54:03.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f1495c3e496e7c6-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.070 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.070 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.071 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.071 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.071 DEBUG:boto:Method: GET 2024-04-16T19:54:03.071 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.071 DEBUG:boto:Data: 2024-04-16T19:54:03.071 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.071 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.071 DEBUG:boto:Params: {} 2024-04-16T19:54:03.071 DEBUG:boto:Token: None 2024-04-16T19:54:03.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY= 2024-04-16T19:54:03.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY=', 'Content-Length': '0'} 2024-04-16T19:54:03.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c7331aa6bcb1b0-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.074 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.074 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.074 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.074 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.074 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.075 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.075 DEBUG:boto:Data: 2024-04-16T19:54:03.075 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.075 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.075 DEBUG:boto:Params: {} 2024-04-16T19:54:03.075 DEBUG:boto:Token: None 2024-04-16T19:54:03.075 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:indNWg8L2gSX3k2j9yqraa+EHOM= 2024-04-16T19:54:03.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:indNWg8L2gSX3k2j9yqraa+EHOM=', 'Content-Length': '0'} 2024-04-16T19:54:03.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8a066c4b3326a7-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.078 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.078 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.078 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.078 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.078 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.078 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.078 DEBUG:boto:Data: 2024-04-16T19:54:03.078 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.078 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.079 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.079 DEBUG:boto:Params: {} 2024-04-16T19:54:03.079 DEBUG:boto:Token: None 2024-04-16T19:54:03.079 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:indNWg8L2gSX3k2j9yqraa+EHOM= 2024-04-16T19:54:03.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:indNWg8L2gSX3k2j9yqraa+EHOM=', 'Content-Length': '0'} 2024-04-16T19:54:03.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af75e16a0a6f27ce-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:03.081 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.081 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.081 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.081 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.082 DEBUG:boto:Method: GET 2024-04-16T19:54:03.082 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.082 DEBUG:boto:Data: 2024-04-16T19:54:03.082 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.082 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.082 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.082 DEBUG:boto:Params: {} 2024-04-16T19:54:03.082 DEBUG:boto:Token: None 2024-04-16T19:54:03.082 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY= 2024-04-16T19:54:03.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY=', 'Content-Length': '0'} 2024-04-16T19:54:03.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86e1bbd8808b73b-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.085 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.085 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.085 DEBUG:boto:path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.085 DEBUG:boto:auth_path=/yrjefh-48/a?versionId=null 2024-04-16T19:54:03.085 DEBUG:boto:Method: GET 2024-04-16T19:54:03.085 DEBUG:boto:Path: /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.085 DEBUG:boto:Data: 2024-04-16T19:54:03.085 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.085 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.086 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.086 DEBUG:boto:Params: {} 2024-04-16T19:54:03.086 DEBUG:boto:Token: None 2024-04-16T19:54:03.086 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a?versionId=null 2024-04-16T19:54:03.086 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY= 2024-04-16T19:54:03.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dqm1bNm9s5o71j+7KnY2UYumDBY=', 'Content-Length': '0'} 2024-04-16T19:54:03.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8476306cc5305d4-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.088 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.088 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.088 DEBUG:boto:path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.089 DEBUG:boto:auth_path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.089 DEBUG:boto:Method: GET 2024-04-16T19:54:03.089 DEBUG:boto:Path: /yrjefh-48/?prefix=a 2024-04-16T19:54:03.089 DEBUG:boto:Data: 2024-04-16T19:54:03.089 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.089 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.089 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.089 DEBUG:boto:Params: {} 2024-04-16T19:54:03.089 DEBUG:boto:Token: None 2024-04-16T19:54:03.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55157bc096b5784-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.093 DEBUG:boto:b'yrjefh-48a1000falsea2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.093 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.093 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.094 DEBUG:boto:path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.094 DEBUG:boto:auth_path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.094 DEBUG:boto:Method: GET 2024-04-16T19:54:03.094 DEBUG:boto:Path: /yrjefh-48/?prefix=a 2024-04-16T19:54:03.094 DEBUG:boto:Data: 2024-04-16T19:54:03.094 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.094 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.094 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.094 DEBUG:boto:Params: {} 2024-04-16T19:54:03.094 DEBUG:boto:Token: None 2024-04-16T19:54:03.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c3b5631265731b-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.097 DEBUG:boto:b'yrjefh-48a1000falsea2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.097 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.097 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.097 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.097 DEBUG:boto:Path: /yrjefh-48/a 2024-04-16T19:54:03.097 DEBUG:boto:Data: 2024-04-16T19:54:03.097 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.098 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.098 DEBUG:boto:Params: {} 2024-04-16T19:54:03.098 DEBUG:boto:Token: None 2024-04-16T19:54:03.098 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a 2024-04-16T19:54:03.098 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z6YaehUeC6r+h2d+zl9koZKEePU= 2024-04-16T19:54:03.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6YaehUeC6r+h2d+zl9koZKEePU=', 'Content-Length': '0'} 2024-04-16T19:54:03.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9481511d1a0c662-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.100 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.100 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.101 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.101 DEBUG:boto:Path: /yrjefh-48/a 2024-04-16T19:54:03.101 DEBUG:boto:Data: 2024-04-16T19:54:03.101 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.101 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.101 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.101 DEBUG:boto:Params: {} 2024-04-16T19:54:03.101 DEBUG:boto:Token: None 2024-04-16T19:54:03.101 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a 2024-04-16T19:54:03.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z6YaehUeC6r+h2d+zl9koZKEePU= 2024-04-16T19:54:03.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6YaehUeC6r+h2d+zl9koZKEePU=', 'Content-Length': '0'} 2024-04-16T19:54:03.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007968215ee12f073c-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:03.104 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.104 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.104 DEBUG:boto:Method: GET 2024-04-16T19:54:03.104 DEBUG:boto:Path: /yrjefh-48/a 2024-04-16T19:54:03.104 DEBUG:boto:Data: 2024-04-16T19:54:03.104 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.104 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.104 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.105 DEBUG:boto:Params: {} 2024-04-16T19:54:03.105 DEBUG:boto:Token: None 2024-04-16T19:54:03.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a 2024-04-16T19:54:03.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6Ks/xJE2ULzt04aLNpvAkjNgxdk= 2024-04-16T19:54:03.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Ks/xJE2ULzt04aLNpvAkjNgxdk=', 'Content-Length': '0'} 2024-04-16T19:54:03.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0518d3912dab5b4-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.150 DEBUG:boto:path=/yrjefh-48/a 2024-04-16T19:54:03.150 DEBUG:boto:auth_path=/yrjefh-48/a 2024-04-16T19:54:03.151 DEBUG:boto:Method: GET 2024-04-16T19:54:03.151 DEBUG:boto:Path: /yrjefh-48/a 2024-04-16T19:54:03.151 DEBUG:boto:Data: 2024-04-16T19:54:03.151 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.151 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.151 DEBUG:boto:Params: {} 2024-04-16T19:54:03.151 DEBUG:boto:Token: None 2024-04-16T19:54:03.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/a 2024-04-16T19:54:03.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6Ks/xJE2ULzt04aLNpvAkjNgxdk= 2024-04-16T19:54:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Ks/xJE2ULzt04aLNpvAkjNgxdk=', 'Content-Length': '0'} 2024-04-16T19:54:03.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04881ae2f54dc04-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.154 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.154 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.154 DEBUG:boto:path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.154 DEBUG:boto:auth_path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.154 DEBUG:boto:Method: GET 2024-04-16T19:54:03.154 DEBUG:boto:Path: /yrjefh-48/?prefix=a 2024-04-16T19:54:03.154 DEBUG:boto:Data: 2024-04-16T19:54:03.154 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.154 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.155 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.155 DEBUG:boto:Params: {} 2024-04-16T19:54:03.155 DEBUG:boto:Token: None 2024-04-16T19:54:03.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000843042b3681733df-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.198 DEBUG:boto:b'yrjefh-48a1000falsea2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.199 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.199 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.199 DEBUG:boto:path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.199 DEBUG:boto:auth_path=/yrjefh-48/?prefix=a 2024-04-16T19:54:03.199 DEBUG:boto:Method: GET 2024-04-16T19:54:03.199 DEBUG:boto:Path: /yrjefh-48/?prefix=a 2024-04-16T19:54:03.199 DEBUG:boto:Data: 2024-04-16T19:54:03.199 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.199 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.200 DEBUG:boto:Params: {} 2024-04-16T19:54:03.200 DEBUG:boto:Token: None 2024-04-16T19:54:03.200 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c63e2f6b20ab7f-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.202 DEBUG:boto:b'yrjefh-48a1000falsea2024-04-16T19:51:21.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:03.203 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.203 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.203 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.203 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.203 DEBUG:boto:Method: GET 2024-04-16T19:54:03.203 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.203 DEBUG:boto:Data: 2024-04-16T19:54:03.204 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.204 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.204 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.204 DEBUG:boto:Params: {} 2024-04-16T19:54:03.204 DEBUG:boto:Token: None 2024-04-16T19:54:03.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw= 2024-04-16T19:54:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw=', 'Content-Length': '0'} 2024-04-16T19:54:03.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ae6a9082f5797b-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.250 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.250 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.251 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.251 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.251 DEBUG:boto:Method: GET 2024-04-16T19:54:03.251 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.251 DEBUG:boto:Data: 2024-04-16T19:54:03.251 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.251 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.251 DEBUG:boto:Params: {} 2024-04-16T19:54:03.251 DEBUG:boto:Token: None 2024-04-16T19:54:03.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw= 2024-04-16T19:54:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw=', 'Content-Length': '0'} 2024-04-16T19:54:03.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c2f04cf950ee738-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.254 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.254 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.254 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.254 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.254 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.254 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.254 DEBUG:boto:Data: 2024-04-16T19:54:03.255 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.255 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.255 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.255 DEBUG:boto:Params: {} 2024-04-16T19:54:03.255 DEBUG:boto:Token: None 2024-04-16T19:54:03.255 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VZ3BLrNnuqHDBO1t6ov2nwjCwrs= 2024-04-16T19:54:03.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VZ3BLrNnuqHDBO1t6ov2nwjCwrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0f86410e307038-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.257 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.257 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.257 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.257 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.258 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.258 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.258 DEBUG:boto:Data: 2024-04-16T19:54:03.258 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.258 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.258 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.258 DEBUG:boto:Params: {} 2024-04-16T19:54:03.258 DEBUG:boto:Token: None 2024-04-16T19:54:03.258 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VZ3BLrNnuqHDBO1t6ov2nwjCwrs= 2024-04-16T19:54:03.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VZ3BLrNnuqHDBO1t6ov2nwjCwrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108ca3dcc081c782-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:03.261 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.261 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.261 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.261 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.261 DEBUG:boto:Method: GET 2024-04-16T19:54:03.261 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.261 DEBUG:boto:Data: 2024-04-16T19:54:03.261 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.261 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.261 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.261 DEBUG:boto:Params: {} 2024-04-16T19:54:03.262 DEBUG:boto:Token: None 2024-04-16T19:54:03.262 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.262 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw= 2024-04-16T19:54:03.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw=', 'Content-Length': '0'} 2024-04-16T19:54:03.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cecbc20a0cddc6-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.306 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.306 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.307 DEBUG:boto:path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.307 DEBUG:boto:auth_path=/yrjefh-48/b?versionId=null 2024-04-16T19:54:03.307 DEBUG:boto:Method: GET 2024-04-16T19:54:03.307 DEBUG:boto:Path: /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.307 DEBUG:boto:Data: 2024-04-16T19:54:03.307 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.307 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.307 DEBUG:boto:Params: {} 2024-04-16T19:54:03.307 DEBUG:boto:Token: None 2024-04-16T19:54:03.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b?versionId=null 2024-04-16T19:54:03.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw= 2024-04-16T19:54:03.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jkCtM0XF8yCs62zp0JH3K11qcKw=', 'Content-Length': '0'} 2024-04-16T19:54:03.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912edeb1f383d5ea-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.310 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.310 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.310 DEBUG:boto:path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.310 DEBUG:boto:auth_path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.310 DEBUG:boto:Method: GET 2024-04-16T19:54:03.310 DEBUG:boto:Path: /yrjefh-48/?prefix=b 2024-04-16T19:54:03.310 DEBUG:boto:Data: 2024-04-16T19:54:03.310 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.311 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.311 DEBUG:boto:Params: {} 2024-04-16T19:54:03.311 DEBUG:boto:Token: None 2024-04-16T19:54:03.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d92ebf2bb383c1-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.354 DEBUG:boto:b'yrjefh-48b1000falseb2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.355 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.355 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.355 DEBUG:boto:path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.355 DEBUG:boto:auth_path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.355 DEBUG:boto:Method: GET 2024-04-16T19:54:03.355 DEBUG:boto:Path: /yrjefh-48/?prefix=b 2024-04-16T19:54:03.356 DEBUG:boto:Data: 2024-04-16T19:54:03.356 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.356 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.356 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.356 DEBUG:boto:Params: {} 2024-04-16T19:54:03.356 DEBUG:boto:Token: None 2024-04-16T19:54:03.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1eddcc926bf70cc-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.359 DEBUG:boto:b'yrjefh-48b1000falseb2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.359 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.359 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.359 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.359 DEBUG:boto:Path: /yrjefh-48/b 2024-04-16T19:54:03.359 DEBUG:boto:Data: 2024-04-16T19:54:03.360 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.360 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.360 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.360 DEBUG:boto:Params: {} 2024-04-16T19:54:03.360 DEBUG:boto:Token: None 2024-04-16T19:54:03.360 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b 2024-04-16T19:54:03.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KDFYqpstvJgeh7gJLrlaj2eks2Q= 2024-04-16T19:54:03.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KDFYqpstvJgeh7gJLrlaj2eks2Q=', 'Content-Length': '0'} 2024-04-16T19:54:03.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e724d768abf764-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.362 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.363 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.363 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.363 DEBUG:boto:Path: /yrjefh-48/b 2024-04-16T19:54:03.363 DEBUG:boto:Data: 2024-04-16T19:54:03.363 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.363 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.363 DEBUG:boto:Params: {} 2024-04-16T19:54:03.363 DEBUG:boto:Token: None 2024-04-16T19:54:03.363 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b 2024-04-16T19:54:03.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KDFYqpstvJgeh7gJLrlaj2eks2Q= 2024-04-16T19:54:03.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KDFYqpstvJgeh7gJLrlaj2eks2Q=', 'Content-Length': '0'} 2024-04-16T19:54:03.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c48da2274b84f7-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:03.366 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.366 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.366 DEBUG:boto:Method: GET 2024-04-16T19:54:03.366 DEBUG:boto:Path: /yrjefh-48/b 2024-04-16T19:54:03.366 DEBUG:boto:Data: 2024-04-16T19:54:03.366 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.366 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.366 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.367 DEBUG:boto:Params: {} 2024-04-16T19:54:03.367 DEBUG:boto:Token: None 2024-04-16T19:54:03.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b 2024-04-16T19:54:03.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MUH2Or8k4e0WAJdS2YqSbHIKNXA= 2024-04-16T19:54:03.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MUH2Or8k4e0WAJdS2YqSbHIKNXA=', 'Content-Length': '0'} 2024-04-16T19:54:03.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009489642d3dfca18e-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.410 DEBUG:boto:path=/yrjefh-48/b 2024-04-16T19:54:03.411 DEBUG:boto:auth_path=/yrjefh-48/b 2024-04-16T19:54:03.411 DEBUG:boto:Method: GET 2024-04-16T19:54:03.411 DEBUG:boto:Path: /yrjefh-48/b 2024-04-16T19:54:03.411 DEBUG:boto:Data: 2024-04-16T19:54:03.411 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.411 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.411 DEBUG:boto:Params: {} 2024-04-16T19:54:03.411 DEBUG:boto:Token: None 2024-04-16T19:54:03.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/b 2024-04-16T19:54:03.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MUH2Or8k4e0WAJdS2YqSbHIKNXA= 2024-04-16T19:54:03.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MUH2Or8k4e0WAJdS2YqSbHIKNXA=', 'Content-Length': '0'} 2024-04-16T19:54:03.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085191a11519a8ffa-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.414 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.414 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.414 DEBUG:boto:path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.414 DEBUG:boto:auth_path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.414 DEBUG:boto:Method: GET 2024-04-16T19:54:03.414 DEBUG:boto:Path: /yrjefh-48/?prefix=b 2024-04-16T19:54:03.415 DEBUG:boto:Data: 2024-04-16T19:54:03.415 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.415 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.415 DEBUG:boto:Params: {} 2024-04-16T19:54:03.415 DEBUG:boto:Token: None 2024-04-16T19:54:03.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007875df24a3443380-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.418 DEBUG:boto:b'yrjefh-48b1000falseb2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.418 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.418 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.419 DEBUG:boto:path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.419 DEBUG:boto:auth_path=/yrjefh-48/?prefix=b 2024-04-16T19:54:03.419 DEBUG:boto:Method: GET 2024-04-16T19:54:03.419 DEBUG:boto:Path: /yrjefh-48/?prefix=b 2024-04-16T19:54:03.419 DEBUG:boto:Data: 2024-04-16T19:54:03.419 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.419 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.419 DEBUG:boto:Params: {} 2024-04-16T19:54:03.419 DEBUG:boto:Token: None 2024-04-16T19:54:03.420 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e8ae490e742583-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.422 DEBUG:boto:b'yrjefh-48b1000falseb2024-04-16T19:51:21.166Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:03.423 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.423 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.423 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.423 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.423 DEBUG:boto:Method: GET 2024-04-16T19:54:03.423 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.423 DEBUG:boto:Data: 2024-04-16T19:54:03.423 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.424 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.424 DEBUG:boto:Params: {} 2024-04-16T19:54:03.424 DEBUG:boto:Token: None 2024-04-16T19:54:03.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ= 2024-04-16T19:54:03.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ=', 'Content-Length': '0'} 2024-04-16T19:54:03.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a239a25265305f-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.470 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.470 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.471 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.471 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.471 DEBUG:boto:Method: GET 2024-04-16T19:54:03.471 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.471 DEBUG:boto:Data: 2024-04-16T19:54:03.471 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.471 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.471 DEBUG:boto:Params: {} 2024-04-16T19:54:03.471 DEBUG:boto:Token: None 2024-04-16T19:54:03.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ= 2024-04-16T19:54:03.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ=', 'Content-Length': '0'} 2024-04-16T19:54:03.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081ffceaf7ab7de1e-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.474 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.474 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.474 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.474 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.474 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.474 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.474 DEBUG:boto:Data: 2024-04-16T19:54:03.475 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.475 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.475 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.475 DEBUG:boto:Params: {} 2024-04-16T19:54:03.475 DEBUG:boto:Token: None 2024-04-16T19:54:03.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jy+Wd/xCRVZIuc2lu+Qq5MdRdko= 2024-04-16T19:54:03.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jy+Wd/xCRVZIuc2lu+Qq5MdRdko=', 'Content-Length': '0'} 2024-04-16T19:54:03.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4605bb852cd6793-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.477 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.478 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.478 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.478 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.478 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.478 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.478 DEBUG:boto:Data: 2024-04-16T19:54:03.478 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.478 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.478 DEBUG:boto:Params: {} 2024-04-16T19:54:03.479 DEBUG:boto:Token: None 2024-04-16T19:54:03.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jy+Wd/xCRVZIuc2lu+Qq5MdRdko= 2024-04-16T19:54:03.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jy+Wd/xCRVZIuc2lu+Qq5MdRdko=', 'Content-Length': '0'} 2024-04-16T19:54:03.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000161dbb869b8e60b6-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:03.481 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.481 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.481 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.481 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.481 DEBUG:boto:Method: GET 2024-04-16T19:54:03.481 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.482 DEBUG:boto:Data: 2024-04-16T19:54:03.482 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.482 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.482 DEBUG:boto:Params: {} 2024-04-16T19:54:03.482 DEBUG:boto:Token: None 2024-04-16T19:54:03.482 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.482 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ= 2024-04-16T19:54:03.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ=', 'Content-Length': '0'} 2024-04-16T19:54:03.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000528472fd21220f5e-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.526 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.527 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.527 DEBUG:boto:path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.527 DEBUG:boto:auth_path=/yrjefh-48/c?versionId=null 2024-04-16T19:54:03.527 DEBUG:boto:Method: GET 2024-04-16T19:54:03.527 DEBUG:boto:Path: /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.527 DEBUG:boto:Data: 2024-04-16T19:54:03.527 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.527 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.527 DEBUG:boto:Params: {} 2024-04-16T19:54:03.527 DEBUG:boto:Token: None 2024-04-16T19:54:03.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c?versionId=null 2024-04-16T19:54:03.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ= 2024-04-16T19:54:03.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv83jQTGs3wQf2Zjaq2w9k3P9MQ=', 'Content-Length': '0'} 2024-04-16T19:54:03.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a93121890c77a24-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.530 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.530 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.530 DEBUG:boto:path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.530 DEBUG:boto:auth_path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.530 DEBUG:boto:Method: GET 2024-04-16T19:54:03.530 DEBUG:boto:Path: /yrjefh-48/?prefix=c 2024-04-16T19:54:03.530 DEBUG:boto:Data: 2024-04-16T19:54:03.531 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.531 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.531 DEBUG:boto:Params: {} 2024-04-16T19:54:03.531 DEBUG:boto:Token: None 2024-04-16T19:54:03.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d481c2b5d428d513-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.534 DEBUG:boto:b'yrjefh-48c1000falsec2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.534 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.534 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.534 DEBUG:boto:path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.534 DEBUG:boto:auth_path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.534 DEBUG:boto:Method: GET 2024-04-16T19:54:03.534 DEBUG:boto:Path: /yrjefh-48/?prefix=c 2024-04-16T19:54:03.535 DEBUG:boto:Data: 2024-04-16T19:54:03.535 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.535 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.535 DEBUG:boto:Params: {} 2024-04-16T19:54:03.535 DEBUG:boto:Token: None 2024-04-16T19:54:03.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002543457db201f84b-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.538 DEBUG:boto:b'yrjefh-48c1000falsec2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.538 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.538 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.538 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.539 DEBUG:boto:Path: /yrjefh-48/c 2024-04-16T19:54:03.539 DEBUG:boto:Data: 2024-04-16T19:54:03.539 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.539 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.539 DEBUG:boto:Params: {} 2024-04-16T19:54:03.539 DEBUG:boto:Token: None 2024-04-16T19:54:03.539 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c 2024-04-16T19:54:03.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LA8JYg7vpAUMEnlEU/+i0ODk13o= 2024-04-16T19:54:03.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LA8JYg7vpAUMEnlEU/+i0ODk13o=', 'Content-Length': '0'} 2024-04-16T19:54:03.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a2aebb2e72e503-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.542 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.542 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.542 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.542 DEBUG:boto:Path: /yrjefh-48/c 2024-04-16T19:54:03.542 DEBUG:boto:Data: 2024-04-16T19:54:03.542 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.542 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.542 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.542 DEBUG:boto:Params: {} 2024-04-16T19:54:03.543 DEBUG:boto:Token: None 2024-04-16T19:54:03.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c 2024-04-16T19:54:03.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LA8JYg7vpAUMEnlEU/+i0ODk13o= 2024-04-16T19:54:03.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LA8JYg7vpAUMEnlEU/+i0ODk13o=', 'Content-Length': '0'} 2024-04-16T19:54:03.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd61f4867e88a47-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:03.545 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.545 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.545 DEBUG:boto:Method: GET 2024-04-16T19:54:03.545 DEBUG:boto:Path: /yrjefh-48/c 2024-04-16T19:54:03.545 DEBUG:boto:Data: 2024-04-16T19:54:03.545 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.545 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.545 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.545 DEBUG:boto:Params: {} 2024-04-16T19:54:03.546 DEBUG:boto:Token: None 2024-04-16T19:54:03.546 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c 2024-04-16T19:54:03.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fAHs3y6JRCye7DYJOHBfSn2V23Q= 2024-04-16T19:54:03.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fAHs3y6JRCye7DYJOHBfSn2V23Q=', 'Content-Length': '0'} 2024-04-16T19:54:03.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d2cf94fd2afc0a-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.590 DEBUG:boto:path=/yrjefh-48/c 2024-04-16T19:54:03.590 DEBUG:boto:auth_path=/yrjefh-48/c 2024-04-16T19:54:03.591 DEBUG:boto:Method: GET 2024-04-16T19:54:03.591 DEBUG:boto:Path: /yrjefh-48/c 2024-04-16T19:54:03.591 DEBUG:boto:Data: 2024-04-16T19:54:03.591 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.591 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.591 DEBUG:boto:Params: {} 2024-04-16T19:54:03.591 DEBUG:boto:Token: None 2024-04-16T19:54:03.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/c 2024-04-16T19:54:03.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fAHs3y6JRCye7DYJOHBfSn2V23Q= 2024-04-16T19:54:03.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fAHs3y6JRCye7DYJOHBfSn2V23Q=', 'Content-Length': '0'} 2024-04-16T19:54:03.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d21b9aeae015fc-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.594 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.594 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.594 DEBUG:boto:path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.594 DEBUG:boto:auth_path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.594 DEBUG:boto:Method: GET 2024-04-16T19:54:03.594 DEBUG:boto:Path: /yrjefh-48/?prefix=c 2024-04-16T19:54:03.594 DEBUG:boto:Data: 2024-04-16T19:54:03.594 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.594 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.595 DEBUG:boto:Params: {} 2024-04-16T19:54:03.595 DEBUG:boto:Token: None 2024-04-16T19:54:03.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047af3f18e5f3585a-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.598 DEBUG:boto:b'yrjefh-48c1000falsec2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.598 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.598 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.598 DEBUG:boto:path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.598 DEBUG:boto:auth_path=/yrjefh-48/?prefix=c 2024-04-16T19:54:03.599 DEBUG:boto:Method: GET 2024-04-16T19:54:03.599 DEBUG:boto:Path: /yrjefh-48/?prefix=c 2024-04-16T19:54:03.599 DEBUG:boto:Data: 2024-04-16T19:54:03.599 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.599 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.599 DEBUG:boto:Params: {} 2024-04-16T19:54:03.599 DEBUG:boto:Token: None 2024-04-16T19:54:03.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448c5c3323c20ca6-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.602 DEBUG:boto:b'yrjefh-48c1000falsec2024-04-16T19:51:21.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:03.602 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.602 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.602 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.603 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.603 DEBUG:boto:Method: GET 2024-04-16T19:54:03.603 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.603 DEBUG:boto:Data: 2024-04-16T19:54:03.603 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.603 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.603 DEBUG:boto:Params: {} 2024-04-16T19:54:03.603 DEBUG:boto:Token: None 2024-04-16T19:54:03.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs= 2024-04-16T19:54:03.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6bdb87ec842c849-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.646 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.646 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.646 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.647 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.647 DEBUG:boto:Method: GET 2024-04-16T19:54:03.647 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.647 DEBUG:boto:Data: 2024-04-16T19:54:03.647 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.647 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.647 DEBUG:boto:Params: {} 2024-04-16T19:54:03.647 DEBUG:boto:Token: None 2024-04-16T19:54:03.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs= 2024-04-16T19:54:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9e2535f2c22515-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.650 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.650 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.650 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.650 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.650 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.650 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.650 DEBUG:boto:Data: 2024-04-16T19:54:03.650 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.651 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.651 DEBUG:boto:Params: {} 2024-04-16T19:54:03.651 DEBUG:boto:Token: None 2024-04-16T19:54:03.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8gU5j0zYed85IPVv8J6xg9+7mhg= 2024-04-16T19:54:03.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8gU5j0zYed85IPVv8J6xg9+7mhg=', 'Content-Length': '0'} 2024-04-16T19:54:03.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015c5579246de305-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.654 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.654 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.654 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.654 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.654 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.654 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.654 DEBUG:boto:Data: 2024-04-16T19:54:03.654 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.654 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.655 DEBUG:boto:Params: {} 2024-04-16T19:54:03.655 DEBUG:boto:Token: None 2024-04-16T19:54:03.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8gU5j0zYed85IPVv8J6xg9+7mhg= 2024-04-16T19:54:03.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8gU5j0zYed85IPVv8J6xg9+7mhg=', 'Content-Length': '0'} 2024-04-16T19:54:03.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c814dc4082c779a-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:03.657 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.657 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.657 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.657 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.657 DEBUG:boto:Method: GET 2024-04-16T19:54:03.658 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.658 DEBUG:boto:Data: 2024-04-16T19:54:03.658 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.658 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.658 DEBUG:boto:Params: {} 2024-04-16T19:54:03.658 DEBUG:boto:Token: None 2024-04-16T19:54:03.658 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.658 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs= 2024-04-16T19:54:03.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056134fc8c36a49fd-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.702 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.702 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.703 DEBUG:boto:path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.703 DEBUG:boto:auth_path=/yrjefh-48/d?versionId=null 2024-04-16T19:54:03.703 DEBUG:boto:Method: GET 2024-04-16T19:54:03.703 DEBUG:boto:Path: /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.703 DEBUG:boto:Data: 2024-04-16T19:54:03.703 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.703 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.703 DEBUG:boto:Params: {} 2024-04-16T19:54:03.704 DEBUG:boto:Token: None 2024-04-16T19:54:03.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d?versionId=null 2024-04-16T19:54:03.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs= 2024-04-16T19:54:03.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LcCrIPtp+Ef1j70SYybhr70xxrs=', 'Content-Length': '0'} 2024-04-16T19:54:03.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029089aa5c3a011b9-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.706 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.706 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.706 DEBUG:boto:path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.706 DEBUG:boto:auth_path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.706 DEBUG:boto:Method: GET 2024-04-16T19:54:03.706 DEBUG:boto:Path: /yrjefh-48/?prefix=d 2024-04-16T19:54:03.706 DEBUG:boto:Data: 2024-04-16T19:54:03.707 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.707 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.707 DEBUG:boto:Params: {} 2024-04-16T19:54:03.707 DEBUG:boto:Token: None 2024-04-16T19:54:03.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaaa1a85e0c0f8fb-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.710 DEBUG:boto:b'yrjefh-48d1000falsed2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.711 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.711 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.711 DEBUG:boto:path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.711 DEBUG:boto:auth_path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.711 DEBUG:boto:Method: GET 2024-04-16T19:54:03.711 DEBUG:boto:Path: /yrjefh-48/?prefix=d 2024-04-16T19:54:03.711 DEBUG:boto:Data: 2024-04-16T19:54:03.711 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.711 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.711 DEBUG:boto:Params: {} 2024-04-16T19:54:03.712 DEBUG:boto:Token: None 2024-04-16T19:54:03.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e20a079c79bbf5-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.714 DEBUG:boto:b'yrjefh-48d1000falsed2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.715 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.715 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.715 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.715 DEBUG:boto:Path: /yrjefh-48/d 2024-04-16T19:54:03.715 DEBUG:boto:Data: 2024-04-16T19:54:03.715 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.715 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.715 DEBUG:boto:Params: {} 2024-04-16T19:54:03.716 DEBUG:boto:Token: None 2024-04-16T19:54:03.716 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d 2024-04-16T19:54:03.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JEm67u28GgsPHeqjs/uYDFxZXA8= 2024-04-16T19:54:03.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JEm67u28GgsPHeqjs/uYDFxZXA8=', 'Content-Length': '0'} 2024-04-16T19:54:03.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9033b958283333-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.718 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.718 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.718 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.719 DEBUG:boto:Path: /yrjefh-48/d 2024-04-16T19:54:03.719 DEBUG:boto:Data: 2024-04-16T19:54:03.719 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.719 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.719 DEBUG:boto:Params: {} 2024-04-16T19:54:03.719 DEBUG:boto:Token: None 2024-04-16T19:54:03.719 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d 2024-04-16T19:54:03.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JEm67u28GgsPHeqjs/uYDFxZXA8= 2024-04-16T19:54:03.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JEm67u28GgsPHeqjs/uYDFxZXA8=', 'Content-Length': '0'} 2024-04-16T19:54:03.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea308dca8aabc7e-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:03.721 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.721 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.722 DEBUG:boto:Method: GET 2024-04-16T19:54:03.722 DEBUG:boto:Path: /yrjefh-48/d 2024-04-16T19:54:03.722 DEBUG:boto:Data: 2024-04-16T19:54:03.722 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.722 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.722 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.722 DEBUG:boto:Params: {} 2024-04-16T19:54:03.722 DEBUG:boto:Token: None 2024-04-16T19:54:03.723 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d 2024-04-16T19:54:03.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k4HotYXu9T/ja8NXt2E0B8yMIX4= 2024-04-16T19:54:03.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k4HotYXu9T/ja8NXt2E0B8yMIX4=', 'Content-Length': '0'} 2024-04-16T19:54:03.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba8bbf876f72f69-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.766 DEBUG:boto:path=/yrjefh-48/d 2024-04-16T19:54:03.767 DEBUG:boto:auth_path=/yrjefh-48/d 2024-04-16T19:54:03.767 DEBUG:boto:Method: GET 2024-04-16T19:54:03.767 DEBUG:boto:Path: /yrjefh-48/d 2024-04-16T19:54:03.767 DEBUG:boto:Data: 2024-04-16T19:54:03.767 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.767 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.767 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.767 DEBUG:boto:Params: {} 2024-04-16T19:54:03.767 DEBUG:boto:Token: None 2024-04-16T19:54:03.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/d 2024-04-16T19:54:03.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k4HotYXu9T/ja8NXt2E0B8yMIX4= 2024-04-16T19:54:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k4HotYXu9T/ja8NXt2E0B8yMIX4=', 'Content-Length': '0'} 2024-04-16T19:54:03.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdabdc75c580ebda-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.770 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.770 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.770 DEBUG:boto:path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.770 DEBUG:boto:auth_path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.770 DEBUG:boto:Method: GET 2024-04-16T19:54:03.770 DEBUG:boto:Path: /yrjefh-48/?prefix=d 2024-04-16T19:54:03.770 DEBUG:boto:Data: 2024-04-16T19:54:03.771 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.771 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.771 DEBUG:boto:Params: {} 2024-04-16T19:54:03.771 DEBUG:boto:Token: None 2024-04-16T19:54:03.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45d2a738faaca34-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.774 DEBUG:boto:b'yrjefh-48d1000falsed2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.774 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.774 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.774 DEBUG:boto:path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.774 DEBUG:boto:auth_path=/yrjefh-48/?prefix=d 2024-04-16T19:54:03.775 DEBUG:boto:Method: GET 2024-04-16T19:54:03.775 DEBUG:boto:Path: /yrjefh-48/?prefix=d 2024-04-16T19:54:03.775 DEBUG:boto:Data: 2024-04-16T19:54:03.775 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.775 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.775 DEBUG:boto:Params: {} 2024-04-16T19:54:03.775 DEBUG:boto:Token: None 2024-04-16T19:54:03.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08ba3a6bd1ee807-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.778 DEBUG:boto:b'yrjefh-48d1000falsed2024-04-16T19:51:21.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:54:03.778 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.778 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.779 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.779 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.779 DEBUG:boto:Method: GET 2024-04-16T19:54:03.779 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.779 DEBUG:boto:Data: 2024-04-16T19:54:03.779 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.779 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.779 DEBUG:boto:Params: {} 2024-04-16T19:54:03.779 DEBUG:boto:Token: None 2024-04-16T19:54:03.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA= 2024-04-16T19:54:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA=', 'Content-Length': '0'} 2024-04-16T19:54:03.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a3e5c23df3af1e-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.822 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.822 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.823 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.823 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.823 DEBUG:boto:Method: GET 2024-04-16T19:54:03.823 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.823 DEBUG:boto:Data: 2024-04-16T19:54:03.823 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.823 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.823 DEBUG:boto:Params: {} 2024-04-16T19:54:03.823 DEBUG:boto:Token: None 2024-04-16T19:54:03.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA= 2024-04-16T19:54:03.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA=', 'Content-Length': '0'} 2024-04-16T19:54:03.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ec4aef1ea9a6ba-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.826 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.826 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.826 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.826 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.826 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.826 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.826 DEBUG:boto:Data: 2024-04-16T19:54:03.827 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.827 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.827 DEBUG:boto:Params: {} 2024-04-16T19:54:03.827 DEBUG:boto:Token: None 2024-04-16T19:54:03.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ku8b4wxdMDiMv2m2Y/QXQ8j2Ilg= 2024-04-16T19:54:03.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ku8b4wxdMDiMv2m2Y/QXQ8j2Ilg=', 'Content-Length': '0'} 2024-04-16T19:54:03.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003196fa6c97ca9f06-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.830 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.830 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.830 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.830 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.830 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.830 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.830 DEBUG:boto:Data: 2024-04-16T19:54:03.830 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.831 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.831 DEBUG:boto:Params: {} 2024-04-16T19:54:03.831 DEBUG:boto:Token: None 2024-04-16T19:54:03.831 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ku8b4wxdMDiMv2m2Y/QXQ8j2Ilg= 2024-04-16T19:54:03.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ku8b4wxdMDiMv2m2Y/QXQ8j2Ilg=', 'Content-Length': '0'} 2024-04-16T19:54:03.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e2aaaf56dd368d5-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:54:03.833 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.833 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.834 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.834 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.834 DEBUG:boto:Method: GET 2024-04-16T19:54:03.834 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.834 DEBUG:boto:Data: 2024-04-16T19:54:03.834 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.834 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.834 DEBUG:boto:Params: {} 2024-04-16T19:54:03.834 DEBUG:boto:Token: None 2024-04-16T19:54:03.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA= 2024-04-16T19:54:03.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA=', 'Content-Length': '0'} 2024-04-16T19:54:03.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90bc6e30430719d-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.878 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.878 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.879 DEBUG:boto:path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.879 DEBUG:boto:auth_path=/yrjefh-48/e?versionId=null 2024-04-16T19:54:03.879 DEBUG:boto:Method: GET 2024-04-16T19:54:03.879 DEBUG:boto:Path: /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.879 DEBUG:boto:Data: 2024-04-16T19:54:03.879 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.879 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.879 DEBUG:boto:Params: {} 2024-04-16T19:54:03.879 DEBUG:boto:Token: None 2024-04-16T19:54:03.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e?versionId=null 2024-04-16T19:54:03.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA= 2024-04-16T19:54:03.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+4/7m1AXMZdBu31OQR7TQhXQmA=', 'Content-Length': '0'} 2024-04-16T19:54:03.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd172c9404867485-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.882 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.882 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.883 DEBUG:boto:path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.883 DEBUG:boto:auth_path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.883 DEBUG:boto:Method: GET 2024-04-16T19:54:03.883 DEBUG:boto:Path: /yrjefh-48/?prefix=e 2024-04-16T19:54:03.883 DEBUG:boto:Data: 2024-04-16T19:54:03.883 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.883 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.883 DEBUG:boto:Params: {} 2024-04-16T19:54:03.883 DEBUG:boto:Token: None 2024-04-16T19:54:03.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d243b3bfec909daa-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.887 DEBUG:boto:b'yrjefh-48e1000falsee2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.887 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.887 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.887 DEBUG:boto:path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.887 DEBUG:boto:auth_path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.887 DEBUG:boto:Method: GET 2024-04-16T19:54:03.887 DEBUG:boto:Path: /yrjefh-48/?prefix=e 2024-04-16T19:54:03.888 DEBUG:boto:Data: 2024-04-16T19:54:03.888 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.888 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.888 DEBUG:boto:Params: {} 2024-04-16T19:54:03.888 DEBUG:boto:Token: None 2024-04-16T19:54:03.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973556f400ec652c-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.890 DEBUG:boto:b'yrjefh-48e1000falsee2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.891 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.891 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.891 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.891 DEBUG:boto:Path: /yrjefh-48/e 2024-04-16T19:54:03.891 DEBUG:boto:Data: 2024-04-16T19:54:03.891 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.891 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.892 DEBUG:boto:Params: {} 2024-04-16T19:54:03.892 DEBUG:boto:Token: None 2024-04-16T19:54:03.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e 2024-04-16T19:54:03.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:innuLxrGI8m/2NdgIvXy59yMC/8= 2024-04-16T19:54:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:innuLxrGI8m/2NdgIvXy59yMC/8=', 'Content-Length': '0'} 2024-04-16T19:54:03.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f8e4b41efa6f648-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.894 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.894 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.894 DEBUG:boto:Method: HEAD 2024-04-16T19:54:03.895 DEBUG:boto:Path: /yrjefh-48/e 2024-04-16T19:54:03.895 DEBUG:boto:Data: 2024-04-16T19:54:03.895 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.895 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.895 DEBUG:boto:Params: {} 2024-04-16T19:54:03.895 DEBUG:boto:Token: None 2024-04-16T19:54:03.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e 2024-04-16T19:54:03.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:innuLxrGI8m/2NdgIvXy59yMC/8= 2024-04-16T19:54:03.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:innuLxrGI8m/2NdgIvXy59yMC/8=', 'Content-Length': '0'} 2024-04-16T19:54:03.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a92576961085c40c-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:54:03.898 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.898 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.898 DEBUG:boto:Method: GET 2024-04-16T19:54:03.898 DEBUG:boto:Path: /yrjefh-48/e 2024-04-16T19:54:03.898 DEBUG:boto:Data: 2024-04-16T19:54:03.898 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.898 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.899 DEBUG:boto:Params: {} 2024-04-16T19:54:03.899 DEBUG:boto:Token: None 2024-04-16T19:54:03.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e 2024-04-16T19:54:03.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lrI2v3QcmFI1RwF1I7thC+/14js= 2024-04-16T19:54:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lrI2v3QcmFI1RwF1I7thC+/14js=', 'Content-Length': '0'} 2024-04-16T19:54:03.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bf5caa89bcc543-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.942 DEBUG:boto:path=/yrjefh-48/e 2024-04-16T19:54:03.943 DEBUG:boto:auth_path=/yrjefh-48/e 2024-04-16T19:54:03.943 DEBUG:boto:Method: GET 2024-04-16T19:54:03.943 DEBUG:boto:Path: /yrjefh-48/e 2024-04-16T19:54:03.943 DEBUG:boto:Data: 2024-04-16T19:54:03.943 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.943 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.943 DEBUG:boto:Params: {} 2024-04-16T19:54:03.943 DEBUG:boto:Token: None 2024-04-16T19:54:03.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/e 2024-04-16T19:54:03.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lrI2v3QcmFI1RwF1I7thC+/14js= 2024-04-16T19:54:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lrI2v3QcmFI1RwF1I7thC+/14js=', 'Content-Length': '0'} 2024-04-16T19:54:03.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92285b24e681a40-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.946 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.946 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.946 DEBUG:boto:path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.946 DEBUG:boto:auth_path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.946 DEBUG:boto:Method: GET 2024-04-16T19:54:03.947 DEBUG:boto:Path: /yrjefh-48/?prefix=e 2024-04-16T19:54:03.947 DEBUG:boto:Data: 2024-04-16T19:54:03.947 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.947 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.947 DEBUG:boto:Params: {} 2024-04-16T19:54:03.947 DEBUG:boto:Token: None 2024-04-16T19:54:03.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017cfe9b0c6af7a3d-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.990 DEBUG:boto:b'yrjefh-48e1000falsee2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.991 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:03.991 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:03.991 DEBUG:boto:path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.991 DEBUG:boto:auth_path=/yrjefh-48/?prefix=e 2024-04-16T19:54:03.991 DEBUG:boto:Method: GET 2024-04-16T19:54:03.991 DEBUG:boto:Path: /yrjefh-48/?prefix=e 2024-04-16T19:54:03.991 DEBUG:boto:Data: 2024-04-16T19:54:03.991 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:03.991 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.992 DEBUG:boto:Params: {} 2024-04-16T19:54:03.992 DEBUG:boto:Token: None 2024-04-16T19:54:03.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/ 2024-04-16T19:54:03.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk= 2024-04-16T19:54:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BENfWi85WeTkxUDkooGn4uEbmGk=', 'Content-Length': '0'} 2024-04-16T19:54:03.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036bc0cbd8ce162b7-00661ed75b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:03.994 DEBUG:boto:b'yrjefh-48e1000falsee2024-04-16T19:51:21.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:03.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:54:03.995 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:03.995 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:03.995 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:03.995 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:03.995 DEBUG:boto:Method: GET 2024-04-16T19:54:03.995 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:03.995 DEBUG:boto:Data: 2024-04-16T19:54:03.995 DEBUG:boto:Headers: {} 2024-04-16T19:54:03.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:03.996 DEBUG:boto:Port: 8000 2024-04-16T19:54:03.996 DEBUG:boto:Params: {} 2024-04-16T19:54:03.996 DEBUG:boto:Token: None 2024-04-16T19:54:03.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:03 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:03.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:POTaiDJhZjg6rGSSPO9LpR4LTec= 2024-04-16T19:54:03.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:POTaiDJhZjg6rGSSPO9LpR4LTec=', 'Content-Length': '0'} 2024-04-16T19:54:03.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fdcb1654493f2d3-00661ed75b-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.046 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.046 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.047 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.047 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.047 DEBUG:boto:Method: GET 2024-04-16T19:54:04.047 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.047 DEBUG:boto:Data: 2024-04-16T19:54:04.047 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.047 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.047 DEBUG:boto:Params: {} 2024-04-16T19:54:04.047 DEBUG:boto:Token: None 2024-04-16T19:54:04.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM= 2024-04-16T19:54:04.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM=', 'Content-Length': '0'} 2024-04-16T19:54:04.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b9a3a40867e6ea-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.050 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.050 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.050 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.050 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.051 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.051 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.051 DEBUG:boto:Data: 2024-04-16T19:54:04.051 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.051 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.051 DEBUG:boto:Params: {} 2024-04-16T19:54:04.051 DEBUG:boto:Token: None 2024-04-16T19:54:04.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TtCReut8ffEalCNHvXVmji4oYY4= 2024-04-16T19:54:04.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TtCReut8ffEalCNHvXVmji4oYY4=', 'Content-Length': '0'} 2024-04-16T19:54:04.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf0203f73ed42f1-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.054 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.054 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.054 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.054 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.054 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.054 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.054 DEBUG:boto:Data: 2024-04-16T19:54:04.054 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.054 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.054 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.054 DEBUG:boto:Params: {} 2024-04-16T19:54:04.055 DEBUG:boto:Token: None 2024-04-16T19:54:04.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TtCReut8ffEalCNHvXVmji4oYY4= 2024-04-16T19:54:04.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TtCReut8ffEalCNHvXVmji4oYY4=', 'Content-Length': '0'} 2024-04-16T19:54:04.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8c4985f6304365-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:54:04.057 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.057 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.057 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.058 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.058 DEBUG:boto:Method: GET 2024-04-16T19:54:04.058 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.058 DEBUG:boto:Data: 2024-04-16T19:54:04.058 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.058 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.058 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.058 DEBUG:boto:Params: {} 2024-04-16T19:54:04.058 DEBUG:boto:Token: None 2024-04-16T19:54:04.058 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM= 2024-04-16T19:54:04.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM=', 'Content-Length': '0'} 2024-04-16T19:54:04.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bdd76fbe1e7b713-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.102 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.102 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.103 DEBUG:boto:path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.103 DEBUG:boto:auth_path=/yrjefh-48/f?versionId=null 2024-04-16T19:54:04.103 DEBUG:boto:Method: GET 2024-04-16T19:54:04.103 DEBUG:boto:Path: /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.103 DEBUG:boto:Data: 2024-04-16T19:54:04.103 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.103 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.103 DEBUG:boto:Params: {} 2024-04-16T19:54:04.103 DEBUG:boto:Token: None 2024-04-16T19:54:04.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f?versionId=null 2024-04-16T19:54:04.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM= 2024-04-16T19:54:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvWACY0jH7LaNrMlNRnDgOaT7mM=', 'Content-Length': '0'} 2024-04-16T19:54:04.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b3296c13fffaf7-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.106 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.106 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.106 DEBUG:boto:path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.106 DEBUG:boto:auth_path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.107 DEBUG:boto:Method: GET 2024-04-16T19:54:04.107 DEBUG:boto:Path: /yrjefh-48/?prefix=f 2024-04-16T19:54:04.107 DEBUG:boto:Data: 2024-04-16T19:54:04.107 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.107 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.107 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.107 DEBUG:boto:Params: {} 2024-04-16T19:54:04.107 DEBUG:boto:Token: None 2024-04-16T19:54:04.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2748bf3f951b8e4-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.150 DEBUG:boto:b'yrjefh-48f1000falsef2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.151 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.151 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.151 DEBUG:boto:path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.151 DEBUG:boto:auth_path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.151 DEBUG:boto:Method: GET 2024-04-16T19:54:04.151 DEBUG:boto:Path: /yrjefh-48/?prefix=f 2024-04-16T19:54:04.151 DEBUG:boto:Data: 2024-04-16T19:54:04.151 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.151 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.152 DEBUG:boto:Params: {} 2024-04-16T19:54:04.152 DEBUG:boto:Token: None 2024-04-16T19:54:04.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031a58c7810bcb8e8-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.154 DEBUG:boto:b'yrjefh-48f1000falsef2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.155 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.155 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.155 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.155 DEBUG:boto:Path: /yrjefh-48/f 2024-04-16T19:54:04.155 DEBUG:boto:Data: 2024-04-16T19:54:04.155 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.155 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.155 DEBUG:boto:Params: {} 2024-04-16T19:54:04.155 DEBUG:boto:Token: None 2024-04-16T19:54:04.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f 2024-04-16T19:54:04.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rSRUk14VD4isNE7HFsjE9jZqDVo= 2024-04-16T19:54:04.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rSRUk14VD4isNE7HFsjE9jZqDVo=', 'Content-Length': '0'} 2024-04-16T19:54:04.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119027f2581cf743-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.158 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.158 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.158 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.159 DEBUG:boto:Path: /yrjefh-48/f 2024-04-16T19:54:04.159 DEBUG:boto:Data: 2024-04-16T19:54:04.159 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.159 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.159 DEBUG:boto:Params: {} 2024-04-16T19:54:04.159 DEBUG:boto:Token: None 2024-04-16T19:54:04.159 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f 2024-04-16T19:54:04.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rSRUk14VD4isNE7HFsjE9jZqDVo= 2024-04-16T19:54:04.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rSRUk14VD4isNE7HFsjE9jZqDVo=', 'Content-Length': '0'} 2024-04-16T19:54:04.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a388d96e509c22-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:54:04.162 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.162 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.162 DEBUG:boto:Method: GET 2024-04-16T19:54:04.162 DEBUG:boto:Path: /yrjefh-48/f 2024-04-16T19:54:04.162 DEBUG:boto:Data: 2024-04-16T19:54:04.162 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.162 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.162 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.162 DEBUG:boto:Params: {} 2024-04-16T19:54:04.163 DEBUG:boto:Token: None 2024-04-16T19:54:04.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f 2024-04-16T19:54:04.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ImYTgH5t61Ooxzw1c6jaS36K6Tk= 2024-04-16T19:54:04.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ImYTgH5t61Ooxzw1c6jaS36K6Tk=', 'Content-Length': '0'} 2024-04-16T19:54:04.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc15b93b7c1f55f-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.206 DEBUG:boto:path=/yrjefh-48/f 2024-04-16T19:54:04.206 DEBUG:boto:auth_path=/yrjefh-48/f 2024-04-16T19:54:04.207 DEBUG:boto:Method: GET 2024-04-16T19:54:04.207 DEBUG:boto:Path: /yrjefh-48/f 2024-04-16T19:54:04.207 DEBUG:boto:Data: 2024-04-16T19:54:04.207 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.207 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.207 DEBUG:boto:Params: {} 2024-04-16T19:54:04.207 DEBUG:boto:Token: None 2024-04-16T19:54:04.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/f 2024-04-16T19:54:04.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ImYTgH5t61Ooxzw1c6jaS36K6Tk= 2024-04-16T19:54:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ImYTgH5t61Ooxzw1c6jaS36K6Tk=', 'Content-Length': '0'} 2024-04-16T19:54:04.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fc11f3e20d17ccb-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.210 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.210 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.210 DEBUG:boto:path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.210 DEBUG:boto:auth_path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.210 DEBUG:boto:Method: GET 2024-04-16T19:54:04.210 DEBUG:boto:Path: /yrjefh-48/?prefix=f 2024-04-16T19:54:04.210 DEBUG:boto:Data: 2024-04-16T19:54:04.210 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.210 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.211 DEBUG:boto:Params: {} 2024-04-16T19:54:04.211 DEBUG:boto:Token: None 2024-04-16T19:54:04.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009260071973667b8e-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.214 DEBUG:boto:b'yrjefh-48f1000falsef2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.214 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.214 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.214 DEBUG:boto:path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.214 DEBUG:boto:auth_path=/yrjefh-48/?prefix=f 2024-04-16T19:54:04.214 DEBUG:boto:Method: GET 2024-04-16T19:54:04.214 DEBUG:boto:Path: /yrjefh-48/?prefix=f 2024-04-16T19:54:04.214 DEBUG:boto:Data: 2024-04-16T19:54:04.215 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.215 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.215 DEBUG:boto:Params: {} 2024-04-16T19:54:04.215 DEBUG:boto:Token: None 2024-04-16T19:54:04.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca785f019f6c9b1a-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.217 DEBUG:boto:b'yrjefh-48f1000falsef2024-04-16T19:51:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:54:04.218 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.218 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.218 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.218 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.218 DEBUG:boto:Method: GET 2024-04-16T19:54:04.218 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.218 DEBUG:boto:Data: 2024-04-16T19:54:04.219 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.219 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.219 DEBUG:boto:Params: {} 2024-04-16T19:54:04.219 DEBUG:boto:Token: None 2024-04-16T19:54:04.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo= 2024-04-16T19:54:04.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo=', 'Content-Length': '0'} 2024-04-16T19:54:04.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d574cee1f9a4dec4-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.262 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.262 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.263 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.263 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.263 DEBUG:boto:Method: GET 2024-04-16T19:54:04.263 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.263 DEBUG:boto:Data: 2024-04-16T19:54:04.263 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.263 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.263 DEBUG:boto:Params: {} 2024-04-16T19:54:04.263 DEBUG:boto:Token: None 2024-04-16T19:54:04.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo= 2024-04-16T19:54:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo=', 'Content-Length': '0'} 2024-04-16T19:54:04.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58711837704117d-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.266 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.266 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.267 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.267 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.267 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.267 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.267 DEBUG:boto:Data: 2024-04-16T19:54:04.267 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.267 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.267 DEBUG:boto:Params: {} 2024-04-16T19:54:04.267 DEBUG:boto:Token: None 2024-04-16T19:54:04.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ddUBEBMQOWkebI5TVkQPiLkrlYE= 2024-04-16T19:54:04.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ddUBEBMQOWkebI5TVkQPiLkrlYE=', 'Content-Length': '0'} 2024-04-16T19:54:04.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007975db916b6eb163-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.270 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.270 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.270 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.270 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.270 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.270 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.271 DEBUG:boto:Data: 2024-04-16T19:54:04.271 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.271 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.271 DEBUG:boto:Params: {} 2024-04-16T19:54:04.271 DEBUG:boto:Token: None 2024-04-16T19:54:04.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ddUBEBMQOWkebI5TVkQPiLkrlYE= 2024-04-16T19:54:04.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ddUBEBMQOWkebI5TVkQPiLkrlYE=', 'Content-Length': '0'} 2024-04-16T19:54:04.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c863e32b0b06c7-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:54:04.274 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.274 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.274 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.274 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.274 DEBUG:boto:Method: GET 2024-04-16T19:54:04.274 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.275 DEBUG:boto:Data: 2024-04-16T19:54:04.275 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.275 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.275 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.275 DEBUG:boto:Params: {} 2024-04-16T19:54:04.275 DEBUG:boto:Token: None 2024-04-16T19:54:04.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo= 2024-04-16T19:54:04.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo=', 'Content-Length': '0'} 2024-04-16T19:54:04.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001212a5f7c45235e0-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.318 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.319 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.319 DEBUG:boto:path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.319 DEBUG:boto:auth_path=/yrjefh-48/g?versionId=null 2024-04-16T19:54:04.319 DEBUG:boto:Method: GET 2024-04-16T19:54:04.319 DEBUG:boto:Path: /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.319 DEBUG:boto:Data: 2024-04-16T19:54:04.319 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.319 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.319 DEBUG:boto:Params: {} 2024-04-16T19:54:04.319 DEBUG:boto:Token: None 2024-04-16T19:54:04.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g?versionId=null 2024-04-16T19:54:04.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo= 2024-04-16T19:54:04.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6fBA6aX9i6IfVpCRhVHz2F/pUUo=', 'Content-Length': '0'} 2024-04-16T19:54:04.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cedfafc3c39d3db2-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.322 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.322 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.323 DEBUG:boto:path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.323 DEBUG:boto:auth_path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.323 DEBUG:boto:Method: GET 2024-04-16T19:54:04.323 DEBUG:boto:Path: /yrjefh-48/?prefix=g 2024-04-16T19:54:04.323 DEBUG:boto:Data: 2024-04-16T19:54:04.323 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.323 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.323 DEBUG:boto:Params: {} 2024-04-16T19:54:04.323 DEBUG:boto:Token: None 2024-04-16T19:54:04.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003397c11f91b1a9d2-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.326 DEBUG:boto:b'yrjefh-48g1000falseg2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.326 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.327 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.327 DEBUG:boto:path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.327 DEBUG:boto:auth_path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.327 DEBUG:boto:Method: GET 2024-04-16T19:54:04.327 DEBUG:boto:Path: /yrjefh-48/?prefix=g 2024-04-16T19:54:04.327 DEBUG:boto:Data: 2024-04-16T19:54:04.327 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.327 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.327 DEBUG:boto:Params: {} 2024-04-16T19:54:04.327 DEBUG:boto:Token: None 2024-04-16T19:54:04.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec4fac99148ff53-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.330 DEBUG:boto:b'yrjefh-48g1000falseg2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.330 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.331 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.331 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.331 DEBUG:boto:Path: /yrjefh-48/g 2024-04-16T19:54:04.331 DEBUG:boto:Data: 2024-04-16T19:54:04.331 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.331 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.331 DEBUG:boto:Params: {} 2024-04-16T19:54:04.331 DEBUG:boto:Token: None 2024-04-16T19:54:04.331 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g 2024-04-16T19:54:04.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ur6ccTQWw7DDTI2sivxUzbYNqjg= 2024-04-16T19:54:04.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ur6ccTQWw7DDTI2sivxUzbYNqjg=', 'Content-Length': '0'} 2024-04-16T19:54:04.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddcd040c0658c377-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.334 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.334 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.335 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.335 DEBUG:boto:Path: /yrjefh-48/g 2024-04-16T19:54:04.335 DEBUG:boto:Data: 2024-04-16T19:54:04.335 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.335 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.335 DEBUG:boto:Params: {} 2024-04-16T19:54:04.335 DEBUG:boto:Token: None 2024-04-16T19:54:04.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g 2024-04-16T19:54:04.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ur6ccTQWw7DDTI2sivxUzbYNqjg= 2024-04-16T19:54:04.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ur6ccTQWw7DDTI2sivxUzbYNqjg=', 'Content-Length': '0'} 2024-04-16T19:54:04.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8209d79ca0671db-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:54:04.339 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.339 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.340 DEBUG:boto:Method: GET 2024-04-16T19:54:04.340 DEBUG:boto:Path: /yrjefh-48/g 2024-04-16T19:54:04.340 DEBUG:boto:Data: 2024-04-16T19:54:04.340 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.340 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.340 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.340 DEBUG:boto:Params: {} 2024-04-16T19:54:04.340 DEBUG:boto:Token: None 2024-04-16T19:54:04.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g 2024-04-16T19:54:04.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jabx2rR8kN92XXqnPPQuMqaffBI= 2024-04-16T19:54:04.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jabx2rR8kN92XXqnPPQuMqaffBI=', 'Content-Length': '0'} 2024-04-16T19:54:04.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:52:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009222bf748606569d-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.386 DEBUG:boto:path=/yrjefh-48/g 2024-04-16T19:54:04.387 DEBUG:boto:auth_path=/yrjefh-48/g 2024-04-16T19:54:04.387 DEBUG:boto:Method: GET 2024-04-16T19:54:04.387 DEBUG:boto:Path: /yrjefh-48/g 2024-04-16T19:54:04.387 DEBUG:boto:Data: 2024-04-16T19:54:04.387 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.387 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.387 DEBUG:boto:Params: {} 2024-04-16T19:54:04.387 DEBUG:boto:Token: None 2024-04-16T19:54:04.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/g 2024-04-16T19:54:04.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jabx2rR8kN92XXqnPPQuMqaffBI= 2024-04-16T19:54:04.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jabx2rR8kN92XXqnPPQuMqaffBI=', 'Content-Length': '0'} 2024-04-16T19:54:04.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008281ae0a958dfff1-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.390 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.390 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.390 DEBUG:boto:path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.390 DEBUG:boto:auth_path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.391 DEBUG:boto:Method: GET 2024-04-16T19:54:04.391 DEBUG:boto:Path: /yrjefh-48/?prefix=g 2024-04-16T19:54:04.391 DEBUG:boto:Data: 2024-04-16T19:54:04.391 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.391 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.391 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.391 DEBUG:boto:Params: {} 2024-04-16T19:54:04.391 DEBUG:boto:Token: None 2024-04-16T19:54:04.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067566b8d980631a4-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.394 DEBUG:boto:b'yrjefh-48g1000falseg2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.395 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.395 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.395 DEBUG:boto:path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.395 DEBUG:boto:auth_path=/yrjefh-48/?prefix=g 2024-04-16T19:54:04.395 DEBUG:boto:Method: GET 2024-04-16T19:54:04.395 DEBUG:boto:Path: /yrjefh-48/?prefix=g 2024-04-16T19:54:04.395 DEBUG:boto:Data: 2024-04-16T19:54:04.395 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.395 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.396 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.396 DEBUG:boto:Params: {} 2024-04-16T19:54:04.396 DEBUG:boto:Token: None 2024-04-16T19:54:04.396 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcd9814c240cf8a3-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.398 DEBUG:boto:b'yrjefh-48g1000falseg2024-04-16T19:51:21.526Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:54:04.399 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.399 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.399 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.399 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.399 DEBUG:boto:Method: GET 2024-04-16T19:54:04.399 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.400 DEBUG:boto:Data: 2024-04-16T19:54:04.400 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.400 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.400 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.400 DEBUG:boto:Params: {} 2024-04-16T19:54:04.400 DEBUG:boto:Token: None 2024-04-16T19:54:04.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20= 2024-04-16T19:54:04.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20=', 'Content-Length': '0'} 2024-04-16T19:54:04.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aed0e2856c590f2-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.446 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.447 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.447 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.447 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.447 DEBUG:boto:Method: GET 2024-04-16T19:54:04.447 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.447 DEBUG:boto:Data: 2024-04-16T19:54:04.447 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.448 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.448 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.448 DEBUG:boto:Params: {} 2024-04-16T19:54:04.448 DEBUG:boto:Token: None 2024-04-16T19:54:04.448 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20= 2024-04-16T19:54:04.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20=', 'Content-Length': '0'} 2024-04-16T19:54:04.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540ab964c7f9a2ff-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.451 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.451 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.452 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.452 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.452 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.452 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.452 DEBUG:boto:Data: 2024-04-16T19:54:04.452 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.452 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.452 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.452 DEBUG:boto:Params: {} 2024-04-16T19:54:04.452 DEBUG:boto:Token: None 2024-04-16T19:54:04.453 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.453 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hEhEd1WGwOizGy+rENjK+rh3W/Y= 2024-04-16T19:54:04.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hEhEd1WGwOizGy+rENjK+rh3W/Y=', 'Content-Length': '0'} 2024-04-16T19:54:04.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee3e9ee8e974d6c-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.455 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.455 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.455 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.455 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.456 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.456 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.456 DEBUG:boto:Data: 2024-04-16T19:54:04.456 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.456 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.456 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.456 DEBUG:boto:Params: {} 2024-04-16T19:54:04.456 DEBUG:boto:Token: None 2024-04-16T19:54:04.456 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hEhEd1WGwOizGy+rENjK+rh3W/Y= 2024-04-16T19:54:04.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hEhEd1WGwOizGy+rENjK+rh3W/Y=', 'Content-Length': '0'} 2024-04-16T19:54:04.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266aed4b5256a361-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:54:04.459 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.459 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.459 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.459 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.459 DEBUG:boto:Method: GET 2024-04-16T19:54:04.459 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.459 DEBUG:boto:Data: 2024-04-16T19:54:04.460 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.460 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.460 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.460 DEBUG:boto:Params: {} 2024-04-16T19:54:04.460 DEBUG:boto:Token: None 2024-04-16T19:54:04.460 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20= 2024-04-16T19:54:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20=', 'Content-Length': '0'} 2024-04-16T19:54:04.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b892dcd2817656dc-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.506 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.506 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.507 DEBUG:boto:path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.507 DEBUG:boto:auth_path=/yrjefh-48/h?versionId=null 2024-04-16T19:54:04.507 DEBUG:boto:Method: GET 2024-04-16T19:54:04.507 DEBUG:boto:Path: /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.507 DEBUG:boto:Data: 2024-04-16T19:54:04.507 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.507 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.507 DEBUG:boto:Params: {} 2024-04-16T19:54:04.507 DEBUG:boto:Token: None 2024-04-16T19:54:04.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h?versionId=null 2024-04-16T19:54:04.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20= 2024-04-16T19:54:04.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuMllB7JQxGWVJMjnj0+8M26f20=', 'Content-Length': '0'} 2024-04-16T19:54:04.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f235d3cb0ef3e442-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.510 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.510 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.510 DEBUG:boto:path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.510 DEBUG:boto:auth_path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.511 DEBUG:boto:Method: GET 2024-04-16T19:54:04.511 DEBUG:boto:Path: /yrjefh-48/?prefix=h 2024-04-16T19:54:04.511 DEBUG:boto:Data: 2024-04-16T19:54:04.511 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.511 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.511 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.511 DEBUG:boto:Params: {} 2024-04-16T19:54:04.511 DEBUG:boto:Token: None 2024-04-16T19:54:04.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003af65eaf9dac2ea7-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.514 DEBUG:boto:b'yrjefh-48h1000falseh2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.514 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.515 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.515 DEBUG:boto:path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.515 DEBUG:boto:auth_path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.515 DEBUG:boto:Method: GET 2024-04-16T19:54:04.515 DEBUG:boto:Path: /yrjefh-48/?prefix=h 2024-04-16T19:54:04.515 DEBUG:boto:Data: 2024-04-16T19:54:04.515 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.515 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.515 DEBUG:boto:Params: {} 2024-04-16T19:54:04.515 DEBUG:boto:Token: None 2024-04-16T19:54:04.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfceb5ec38db6dcc-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.518 DEBUG:boto:b'yrjefh-48h1000falseh2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.518 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.518 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.518 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.518 DEBUG:boto:Path: /yrjefh-48/h 2024-04-16T19:54:04.519 DEBUG:boto:Data: 2024-04-16T19:54:04.519 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.519 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.519 DEBUG:boto:Params: {} 2024-04-16T19:54:04.519 DEBUG:boto:Token: None 2024-04-16T19:54:04.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h 2024-04-16T19:54:04.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6etCyKqyqWB/MUiZbTbOIFHZLJo= 2024-04-16T19:54:04.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6etCyKqyqWB/MUiZbTbOIFHZLJo=', 'Content-Length': '0'} 2024-04-16T19:54:04.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb34201509b4cd4-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.522 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.522 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.522 DEBUG:boto:Method: HEAD 2024-04-16T19:54:04.522 DEBUG:boto:Path: /yrjefh-48/h 2024-04-16T19:54:04.522 DEBUG:boto:Data: 2024-04-16T19:54:04.522 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.522 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.522 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.522 DEBUG:boto:Params: {} 2024-04-16T19:54:04.523 DEBUG:boto:Token: None 2024-04-16T19:54:04.523 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h 2024-04-16T19:54:04.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6etCyKqyqWB/MUiZbTbOIFHZLJo= 2024-04-16T19:54:04.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6etCyKqyqWB/MUiZbTbOIFHZLJo=', 'Content-Length': '0'} 2024-04-16T19:54:04.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005adf30547a4099bb-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:54:04.525 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.525 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.525 DEBUG:boto:Method: GET 2024-04-16T19:54:04.525 DEBUG:boto:Path: /yrjefh-48/h 2024-04-16T19:54:04.525 DEBUG:boto:Data: 2024-04-16T19:54:04.526 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.526 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.526 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.526 DEBUG:boto:Params: {} 2024-04-16T19:54:04.526 DEBUG:boto:Token: None 2024-04-16T19:54:04.526 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h 2024-04-16T19:54:04.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MbfeG+wJ8KKilDH3Sys4aAHSj4o= 2024-04-16T19:54:04.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MbfeG+wJ8KKilDH3Sys4aAHSj4o=', 'Content-Length': '0'} 2024-04-16T19:54:04.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-48:b482a2a9-ef36-488b-9666-ada656453328.4812.17'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:53:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001563384c747dc333-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.570 DEBUG:boto:path=/yrjefh-48/h 2024-04-16T19:54:04.571 DEBUG:boto:auth_path=/yrjefh-48/h 2024-04-16T19:54:04.571 DEBUG:boto:Method: GET 2024-04-16T19:54:04.571 DEBUG:boto:Path: /yrjefh-48/h 2024-04-16T19:54:04.571 DEBUG:boto:Data: 2024-04-16T19:54:04.571 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.571 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.571 DEBUG:boto:Params: {} 2024-04-16T19:54:04.571 DEBUG:boto:Token: None 2024-04-16T19:54:04.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/h 2024-04-16T19:54:04.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MbfeG+wJ8KKilDH3Sys4aAHSj4o= 2024-04-16T19:54:04.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MbfeG+wJ8KKilDH3Sys4aAHSj4o=', 'Content-Length': '0'} 2024-04-16T19:54:04.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:51:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5c14a2072f47ba-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.574 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.574 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.574 DEBUG:boto:path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.574 DEBUG:boto:auth_path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.574 DEBUG:boto:Method: GET 2024-04-16T19:54:04.574 DEBUG:boto:Path: /yrjefh-48/?prefix=h 2024-04-16T19:54:04.574 DEBUG:boto:Data: 2024-04-16T19:54:04.574 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.575 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.575 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.575 DEBUG:boto:Params: {} 2024-04-16T19:54:04.575 DEBUG:boto:Token: None 2024-04-16T19:54:04.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d58dedc35bd0a0e-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.577 DEBUG:boto:b'yrjefh-48h1000falseh2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.578 DEBUG:boto:path=/yrjefh-48/ 2024-04-16T19:54:04.578 DEBUG:boto:auth_path=/yrjefh-48/ 2024-04-16T19:54:04.578 DEBUG:boto:path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.578 DEBUG:boto:auth_path=/yrjefh-48/?prefix=h 2024-04-16T19:54:04.578 DEBUG:boto:Method: GET 2024-04-16T19:54:04.578 DEBUG:boto:Path: /yrjefh-48/?prefix=h 2024-04-16T19:54:04.578 DEBUG:boto:Data: 2024-04-16T19:54:04.579 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.579 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.579 DEBUG:boto:Params: {} 2024-04-16T19:54:04.579 DEBUG:boto:Token: None 2024-04-16T19:54:04.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT /yrjefh-48/ 2024-04-16T19:54:04.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg= 2024-04-16T19:54:04.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58B0G8yUEL+ViFO5PNIRfK2Ymg=', 'Content-Length': '0'} 2024-04-16T19:54:04.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e689b6611e999b-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.581 DEBUG:boto:b'yrjefh-48h1000falseh2024-04-16T19:51:21.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:04.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-48 zones={test-zone2, test-zone1} 2024-04-16T19:54:04.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2024-04-16T19:54:04.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.584 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:04.638 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:04.638 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:04.693 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:04.716 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297081.408227_608.1","last_update":"2024-04-16T19:51:21.408227Z"}] 2024-04-16T19:54:04.716 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297081.408227_608.1'} 2024-04-16T19:54:04.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:04.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.717 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:04.773 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:04.773 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:04.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:04.826+0000 7fb60a216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:04.827+0000 7fb60a216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:04.851 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297081.408227_608.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:51:21.408227Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:04Z"} 2024-04-16T19:54:04.852 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297081.408227_608.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:51:21.408227Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:04Z"} 2024-04-16T19:54:04.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297081.408227_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:51:21.408227Z', 'realm_epoch': 2}}] 2024-04-16T19:54:04.852 DEBUG:rgw_multi.tests:log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297081.408227_608.1'} 2024-04-16T19:54:04.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297081.408227_608.1'} 2024-04-16T19:54:04.852 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:04.852 DEBUG:boto:path=/ 2024-04-16T19:54:04.852 DEBUG:boto:auth_path=/ 2024-04-16T19:54:04.852 DEBUG:boto:Method: GET 2024-04-16T19:54:04.852 DEBUG:boto:Path: / 2024-04-16T19:54:04.853 DEBUG:boto:Data: 2024-04-16T19:54:04.853 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.853 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.853 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.853 DEBUG:boto:Params: {} 2024-04-16T19:54:04.853 DEBUG:boto:Token: None 2024-04-16T19:54:04.853 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT / 2024-04-16T19:54:04.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g5WLbFQmm60MLKbE4P1GvNmqhic= 2024-04-16T19:54:04.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g5WLbFQmm60MLKbE4P1GvNmqhic=', 'Content-Length': '0'} 2024-04-16T19:54:04.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095dd8fc532a004db-00661ed75c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.857 DEBUG:boto:path=/ 2024-04-16T19:54:04.857 DEBUG:boto:auth_path=/ 2024-04-16T19:54:04.857 DEBUG:boto:Method: GET 2024-04-16T19:54:04.857 DEBUG:boto:Path: / 2024-04-16T19:54:04.857 DEBUG:boto:Data: 2024-04-16T19:54:04.858 DEBUG:boto:Headers: {} 2024-04-16T19:54:04.858 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.858 DEBUG:boto:Port: 8000 2024-04-16T19:54:04.858 DEBUG:boto:Params: {} 2024-04-16T19:54:04.858 DEBUG:boto:Token: None 2024-04-16T19:54:04.858 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:04 GMT / 2024-04-16T19:54:04.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g5WLbFQmm60MLKbE4P1GvNmqhic= 2024-04-16T19:54:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g5WLbFQmm60MLKbE4P1GvNmqhic=', 'Content-Length': '0'} 2024-04-16T19:54:04.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9bb053c63f7671-00661ed75c-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:04.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-16T19:54:04.862 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-yrjefh-1 2024-04-16T19:54:04.863 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateRole: calling handler 2024-04-16T19:54:04.863 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:04.864 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:04.864 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.864 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.864 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1', 'AssumeRolePolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:04.865 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler > 2024-04-16T19:54:04.865 DEBUG:botocore.hooks:Event choose-signer.iam.CreateRole: calling handler 2024-04-16T19:54:04.866 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:04.866 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195404Z content-type;host;x-amz-date c2c67a7f81154f760976f767d622169d3438896fdcb35ae489afdc0e4d56b3dd 2024-04-16T19:54:04.866 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195404Z 20240416/test-zonegroup/iam/aws4_request fe06bd8368a21af2a1bd9a2e2d702d89b85b6ec577caaea2cf24ba54f9417f9b 2024-04-16T19:54:04.866 DEBUG:botocore.auth:Signature: fde43dbaa57fb8e5ea08c27437fad6d23530d4f2251a9bbefbdf0b11860bf864 2024-04-16T19:54:04.867 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler 2024-04-16T19:54:04.867 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:04.881 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000c09307cf678284da-00661ed75c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '785', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:04.881 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}tx00000c09307cf678284da-00661ed75c-4812-test-zone1' 2024-04-16T19:54:04.902 DEBUG:botocore.hooks:Event needs-retry.iam.CreateRole: calling handler 2024-04-16T19:54:04.902 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:04.902 DEBUG:botocore.hooks:Event after-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.902 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-yrjefh-2 2024-04-16T19:54:04.903 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateRole: calling handler 2024-04-16T19:54:04.903 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:04.903 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:04.904 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.904 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.904 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2', 'AssumeRolePolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:04.904 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler > 2024-04-16T19:54:04.904 DEBUG:botocore.hooks:Event choose-signer.iam.CreateRole: calling handler 2024-04-16T19:54:04.905 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:04.905 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195404Z content-type;host;x-amz-date 994d2cfea4d9056607c266b38d0c4363947a8f45d3e28a6b56ae8bebd868b0bd 2024-04-16T19:54:04.905 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195404Z 20240416/test-zonegroup/iam/aws4_request b31d02e3ee0aa7b66f816d7cf0143dcb35d8a38aa8182b0ba8b9fd543ed7f579 2024-04-16T19:54:04.905 DEBUG:botocore.auth:Signature: 9b98df954c83e3ed1da6d17b5f0fc5306a8a552d935783ff26b0be905d0ed509 2024-04-16T19:54:04.905 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler 2024-04-16T19:54:04.906 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:04.925 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006ee7d995b35e1fb7-00661ed75c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '785', 'Date': 'Tue, 16 Apr 2024 19:54:04 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:04.925 DEBUG:botocore.parsers:Response body: b'20b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}tx000006ee7d995b35e1fb7-00661ed75c-5331-test-zone2' 2024-04-16T19:54:04.926 DEBUG:botocore.hooks:Event needs-retry.iam.CreateRole: calling handler 2024-04-16T19:54:04.926 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:04.926 DEBUG:botocore.hooks:Event after-call.iam.CreateRole: calling handler 2024-04-16T19:54:04.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:04.927 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:04.975 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:04.976 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:05.027 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:05.050 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T19:54:05.050 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:05.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:05.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:05.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:05.050 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:05.100 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:05.100 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:05.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:05.148+0000 7f8e0b89d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:05.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:05.149+0000 7f8e0b89d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:05.172 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:05Z"} 2024-04-16T19:54:05.172 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:05Z"} 2024-04-16T19:54:05.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T19:54:05.172 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:05.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:05.173 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.030 DEBUG:botocore.loaders:Loading JSON file: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/data/iam/2010-05-08/paginators-1.json 2024-04-16T19:54:06.031 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:06.031 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.031 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.032 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.032 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.032 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.032 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:06.033 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:06.033 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.033 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:06.033 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 9ba0566446fb81246bc7af1cdc4b72b2cad3b895f9adc92ed9c291b19d7479e1 2024-04-16T19:54:06.033 DEBUG:botocore.auth:Signature: de0e824887c105e8940d2ee65fd5dee5a91e2c688e1617fde87c736baa71e3ac 2024-04-16T19:54:06.034 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:06.034 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.037 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004a0bdcabf95b2044-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1519', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.037 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111falsetx000004a0bdcabf95b2044-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.038 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:06.039 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.039 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.039 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:06.039 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.039 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.040 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.040 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.040 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.040 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:06.040 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:06.041 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.041 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:06.041 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 73edb8bc002e04343acbba7540d0e4f315ea8d6810f1ddb4d10873b5c9b40f43 2024-04-16T19:54:06.041 DEBUG:botocore.auth:Signature: d697d93b9018609703fc8db17fe30bc87de7d094ff38385a30a0fdbda0dfb0ec 2024-04-16T19:54:06.041 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:06.041 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.046 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000ab2e2db99cc1e0e5-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1519', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.046 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111falsetx00000ab2e2db99cc1e0e5-00661ed75e-5331-test-zone2' 2024-04-16T19:54:06.047 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:06.047 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.048 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:06.048 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.048 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.048 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.049 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.049 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.049 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.049 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.049 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.050 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.050 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:06.050 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 6f8c94dc11e3e70e024a99c098e0664416d2c940d14443ee02eb744f7c02cadf 2024-04-16T19:54:06.050 DEBUG:botocore.auth:Signature: 776e5aab9c15d5b1fcaf3fb971dc4eac543cfeaaf9f62d11afe59f6d99c8d846 2024-04-16T19:54:06.050 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.050 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.052 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000538bc0e42a1bca32-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.053 DEBUG:botocore.parsers:Response body: b'tx00000538bc0e42a1bca32-00661ed75e-4812-test-zone1bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.053 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.053 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.054 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.054 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.054 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.054 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.055 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.055 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.055 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.055 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.055 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.056 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.056 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:06.056 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request eb9ffd35535c1b8e79fa46fa4c706e50f531fa0f62492b2232bd41962be0dc23 2024-04-16T19:54:06.056 DEBUG:botocore.auth:Signature: 3d5f4f3e45263f02b08292b967185e5a0b6da3531d0b5e8046037a4ca76b4481 2024-04-16T19:54:06.056 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.056 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.058 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000001a8143b34f02c939-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.058 DEBUG:botocore.parsers:Response body: b'tx000001a8143b34f02c939-00661ed75e-5331-test-zone2bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.059 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.059 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.059 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.060 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.060 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.060 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.060 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.061 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.061 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.061 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:06.061 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.061 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.062 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:06.062 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request c32f29023b2494dd0aedcfe51d3d402d05d365163d004453897d22cae750f179 2024-04-16T19:54:06.062 DEBUG:botocore.auth:Signature: 0430902c18531cdc62ddf5f0bbfbd60c0cdeb0fe8a8174afc93155ee769df44e 2024-04-16T19:54:06.062 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.062 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.064 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b7f7249c252f74a1-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.064 DEBUG:botocore.parsers:Response body: b'tx00000b7f7249c252f74a1-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.065 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.065 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.065 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.065 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.065 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.065 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.066 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.066 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.066 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.066 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:06.066 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.067 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.067 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:06.067 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 1d796f25e02c9f73a90a4bd3abbc185863e5726eeef461a3835b11bbcea3d461 2024-04-16T19:54:06.067 DEBUG:botocore.auth:Signature: 5798e7a7f4eb7e0b14058eb313276d5a49558f0c1a76e13c9e53eab22d96b6a2 2024-04-16T19:54:06.067 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.067 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.070 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000015b3ee7ad90c3877-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.070 DEBUG:botocore.parsers:Response body: b'tx0000015b3ee7ad90c3877-00661ed75e-5331-test-zone2' 2024-04-16T19:54:06.070 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.070 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.070 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.071 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.071 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.071 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.072 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.072 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.072 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.072 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:06.072 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.073 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.073 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:06.073 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 632b72f1d5e1cb92f3895ea67353dddffa039ce864052ceb6b941206c54c031c 2024-04-16T19:54:06.073 DEBUG:botocore.auth:Signature: 5190afb01e0677bb10ba9173de5491f122631886b5b6724a8f998acedc8c8379 2024-04-16T19:54:06.073 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.073 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.075 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000005aa49197ea144182-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.076 DEBUG:botocore.parsers:Response body: b'tx000005aa49197ea144182-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.076 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.076 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.076 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.076 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.077 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.077 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.077 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.077 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.077 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.077 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:06.078 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.078 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.078 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:06.078 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 7f49ef5a6bf6ed79b60b5d486aae1a918e9ca8bda023a5ceb4bbb1b3cf0652e3 2024-04-16T19:54:06.078 DEBUG:botocore.auth:Signature: 2b4bcca9b8ba73aa9a2b2a83db51e8bfe7e5c51dfc3ae7542d780a7dd9330181 2024-04-16T19:54:06.079 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.079 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.081 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000abc36f05285063d1-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.081 DEBUG:botocore.parsers:Response body: b'tx00000abc36f05285063d1-00661ed75e-5331-test-zone2' 2024-04-16T19:54:06.081 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.081 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.082 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.082 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.082 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.082 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.082 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.082 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.082 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.083 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.083 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.083 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.083 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:06.083 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request b85cdcfef65f9f00313cc34c2d06742237aea6747bce88a0af3cb49c94319063 2024-04-16T19:54:06.083 DEBUG:botocore.auth:Signature: b5f33544aef213ccdf708f3802d68aa432b9404e025f81f27a64bce6486d5e9a 2024-04-16T19:54:06.084 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.084 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.085 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000c7453a6b06eb00d1-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.086 DEBUG:botocore.parsers:Response body: b'tx00000c7453a6b06eb00d1-00661ed75e-4812-test-zone120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.086 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.086 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.086 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.087 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.087 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.087 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.087 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.087 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.087 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.087 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.087 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.088 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.088 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:06.088 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request fb4cbae4c1a35400359a1690e834a8d88e6843bc4b4d162b0c12375da2bf26dd 2024-04-16T19:54:06.088 DEBUG:botocore.auth:Signature: 0343a8b0e2d38170c65c454e33a48dd977de095fdd2f0693ad8c64a31ee9257e 2024-04-16T19:54:06.088 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.088 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.090 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000da6abd25c1a88f83-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.090 DEBUG:botocore.parsers:Response body: b'tx00000da6abd25c1a88f83-00661ed75e-5331-test-zone220b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.091 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.091 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.091 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.091 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.092 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.092 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.092 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.092 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.092 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.092 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:06.092 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.093 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.093 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:06.093 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 83c511195948646f3a1629693fb89c1d2ad67e35325fcbba87a4c04a9488c754 2024-04-16T19:54:06.093 DEBUG:botocore.auth:Signature: e2a0901e4b50f96458aa8ff2a657303d18c311a34ff871ba3dd2e7e3bce1dc04 2024-04-16T19:54:06.093 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.093 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.095 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000018716089952c9ef7-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.096 DEBUG:botocore.parsers:Response body: b'tx0000018716089952c9ef7-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.096 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.096 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.096 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.096 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.096 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.096 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.097 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.097 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.097 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.097 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:06.097 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.097 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.097 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:06.098 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 60cc4a426be9adc06b25988de31c82a0a8989c0f21c7815e9a20e58d73d0f37a 2024-04-16T19:54:06.098 DEBUG:botocore.auth:Signature: 0af107b1cac655e1bdcb102059a7bfac8208bd54c3e4c6ec02b70ecabbef640e 2024-04-16T19:54:06.098 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.098 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.100 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000377cb36a59dc187f-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.100 DEBUG:botocore.parsers:Response body: b'tx00000377cb36a59dc187f-00661ed75e-5331-test-zone2' 2024-04-16T19:54:06.100 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.100 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.101 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:06.101 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.101 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.101 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.101 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.101 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.101 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.102 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:06.102 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.102 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.102 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:06.102 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 28cac86ea182d417ae0be4266609e97e9e2cd062bbe91eae60079fc2d79d1dd8 2024-04-16T19:54:06.102 DEBUG:botocore.auth:Signature: 8ad3dd80041874b3821db5f4ab1046f7c6e45a0a1d73ef64b3e5c8754a4d272e 2024-04-16T19:54:06.103 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.103 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.105 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000a76330312457ba0c-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.105 DEBUG:botocore.parsers:Response body: b'tx00000a76330312457ba0c-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.105 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.105 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.106 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.106 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.106 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.106 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.106 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.106 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.107 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.107 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:06.107 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.107 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.107 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:06.107 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request f72668018afb0777bbb89b1983904e58d6f3acdc8b35db2bfca2a1478bda1cfe 2024-04-16T19:54:06.107 DEBUG:botocore.auth:Signature: 3ce41c36fa9cb2c0fcbb6b0f5be813746cd0cccbda9a0e9ee73e0b722855fc95 2024-04-16T19:54:06.108 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.108 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.110 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000009e16f5c9726b9672-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.110 DEBUG:botocore.parsers:Response body: b'tx000009e16f5c9726b9672-00661ed75e-5331-test-zone2' 2024-04-16T19:54:06.111 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.111 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.111 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:06.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-16T19:54:06.112 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-yrjefh-3 2024-04-16T19:54:06.112 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateRole: calling handler 2024-04-16T19:54:06.112 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.112 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.112 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:06.112 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:06.113 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.113 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler > 2024-04-16T19:54:06.113 DEBUG:botocore.hooks:Event choose-signer.iam.CreateRole: calling handler 2024-04-16T19:54:06.113 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.113 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 1d2678887c1960b7e4f6ee4e5276ccceb1e2fb8af56c5ae7fc711cdc7bf0a666 2024-04-16T19:54:06.113 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 5d750fc9c86d8f865cd39f57afb3144f17bf6c10c414b5afd3cd57223e8d13f3 2024-04-16T19:54:06.114 DEBUG:botocore.auth:Signature: 54230d92a49387104377964cdbe94a2a04d88aa670eb554733c5f44229a85c3c 2024-04-16T19:54:06.114 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler 2024-04-16T19:54:06.114 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.127 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004e0567db51b6ae93-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '778', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.127 DEBUG:botocore.parsers:Response body: b'c7297c15-4f7e-4ce7-b694-95073750be5broles-yrjefh-3/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-32024-04-16T19:54:06.116Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000004e0567db51b6ae93-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.127 DEBUG:botocore.hooks:Event needs-retry.iam.CreateRole: calling handler 2024-04-16T19:54:06.127 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.127 DEBUG:botocore.hooks:Event after-call.iam.CreateRole: calling handler 2024-04-16T19:54:06.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.128 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.183 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.183 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.239 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:06.262 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.120972_114.1","last_update":"2024-04-16T19:54:06.120972Z"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T19:54:06.262 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.120972_114.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.263 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.319 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.319 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.370+0000 7f3deb53b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.372+0000 7f3deb53b800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.395 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.120972_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.120972Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.395 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.120972_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.120972Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.120972_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.120972Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T19:54:06.395 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.120972_114.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.120972_114.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.395 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.395 INFO:rgw_multi.tests:checking if zone: test-zone1 has role: roles-yrjefh-3 2024-04-16T19:54:06.396 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.396 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.396 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.396 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.396 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.396 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.397 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.397 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.397 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.397 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 77226483663849461d98f2bf5299f31895b10c98c209ccfc783a71492f688a39 2024-04-16T19:54:06.397 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 7690d92d4ad521ea44880d6ae62791e579d171a89d7c8b32e6c1b980eeb5bde7 2024-04-16T19:54:06.397 DEBUG:botocore.auth:Signature: 6dd6cf8d283bf33aafacd2bda7e6579368abdb47b3ab6d9aa2220b48738d6851 2024-04-16T19:54:06.398 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.398 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.400 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000008b55ae771f0daddc-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '766', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.400 DEBUG:botocore.parsers:Response body: b'tx000008b55ae771f0daddc-00661ed75e-4812-test-zone1c7297c15-4f7e-4ce7-b694-95073750be5broles-yrjefh-3/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-32024-04-16T19:54:06.116Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.401 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.401 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.401 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.401 INFO:rgw_multi.tests:success, zone: test-zone1 has role: roles-yrjefh-3 2024-04-16T19:54:06.401 INFO:rgw_multi.tests:checking if zone: test-zone2 has role: roles-yrjefh-3 2024-04-16T19:54:06.401 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.401 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.401 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.402 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.402 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.402 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.402 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.402 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.403 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.403 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 77226483663849461d98f2bf5299f31895b10c98c209ccfc783a71492f688a39 2024-04-16T19:54:06.403 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 711fc347a4d095192c5d19f9aafa81d7f5f99cf8e68f23c865c8b9788cc34d30 2024-04-16T19:54:06.403 DEBUG:botocore.auth:Signature: 39bbbb595969d71acccf15280a8d013f1662a8916b512c81dfef8fc7de5b8c22 2024-04-16T19:54:06.403 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.403 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.406 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b4f34720cbbb3956-00661ed75e-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '766', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.406 DEBUG:botocore.parsers:Response body: b'tx00000b4f34720cbbb3956-00661ed75e-5331-test-zone2c7297c15-4f7e-4ce7-b694-95073750be5broles-yrjefh-3/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-32024-04-16T19:54:06.116Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-16T19:54:06.407 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.407 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.407 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.407 INFO:rgw_multi.tests:success, zone: test-zone2 has role: roles-yrjefh-3 2024-04-16T19:54:06.407 INFO:rgw_multi.tests:deleting role: roles-yrjefh-3 2024-04-16T19:54:06.408 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteRole: calling handler 2024-04-16T19:54:06.408 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.408 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.408 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:06.408 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:06.408 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.409 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler > 2024-04-16T19:54:06.409 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteRole: calling handler 2024-04-16T19:54:06.409 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.409 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date e73bc4711a6522baaa93c0201ef67c988033dd017816392ff2b36db56bdbce46 2024-04-16T19:54:06.409 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 541b15cbce09ed0e2650daeba207961cf06469b0a507cec480ffef3791ff0812 2024-04-16T19:54:06.410 DEBUG:botocore.auth:Signature: 7a9fe6847ceb5f0d2f3a343b676cd031be828d9606aa6be726382d84b8a59146 2024-04-16T19:54:06.410 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler 2024-04-16T19:54:06.410 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.420 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f8a69915208211e7-00661ed75e-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '151', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.420 DEBUG:botocore.parsers:Response body: b'tx00000f8a69915208211e7-00661ed75e-4812-test-zone1' 2024-04-16T19:54:06.421 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteRole: calling handler 2024-04-16T19:54:06.421 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.421 DEBUG:botocore.hooks:Event after-call.iam.DeleteRole: calling handler 2024-04-16T19:54:06.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.421 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.471 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.471 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:06.547 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T19:54:06.547 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.548 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.548 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.596 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.597 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.643+0000 7f1a1371f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.645+0000 7f1a1371f800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.668 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.668 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T19:54:06.668 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.669 INFO:rgw_multi.tests:checking if zone: test-zone1 does not have role: roles-yrjefh-3 2024-04-16T19:54:06.673 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.673 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.674 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:06.674 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.674 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.674 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.674 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.674 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.675 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.675 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 77226483663849461d98f2bf5299f31895b10c98c209ccfc783a71492f688a39 2024-04-16T19:54:06.675 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 7690d92d4ad521ea44880d6ae62791e579d171a89d7c8b32e6c1b980eeb5bde7 2024-04-16T19:54:06.675 DEBUG:botocore.auth:Signature: 6dd6cf8d283bf33aafacd2bda7e6579368abdb47b3ab6d9aa2220b48738d6851 2024-04-16T19:54:06.675 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.675 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.678 DEBUG:botocore.parsers:Response headers: {'Content-Length': '328', 'x-amz-request-id': 'tx00000e9b0650cd0b65cf3-00661ed75e-4812-test-zone1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.678 DEBUG:botocore.parsers:Response body: b'NoSuchEntityNo such RoleName in the tenanttx00000e9b0650cd0b65cf3-00661ed75e-4812-test-zone14812-test-zone1-test-zonegroup' 2024-04-16T19:54:06.679 DEBUG:botocore.parsers:Response headers: {'Content-Length': '328', 'x-amz-request-id': 'tx00000e9b0650cd0b65cf3-00661ed75e-4812-test-zone1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.679 DEBUG:botocore.parsers:Response body: b'NoSuchEntityNo such RoleName in the tenanttx00000e9b0650cd0b65cf3-00661ed75e-4812-test-zone14812-test-zone1-test-zonegroup' 2024-04-16T19:54:06.679 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.679 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.679 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.679 INFO:rgw_multi.tests:success, zone: test-zone1 does not have role: roles-yrjefh-3 2024-04-16T19:54:06.679 INFO:rgw_multi.tests:checking if zone: test-zone2 does not have role: roles-yrjefh-3 2024-04-16T19:54:06.680 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:06.680 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:06.680 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:06.680 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.680 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:06.680 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-3'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:06.680 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:06.680 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:06.681 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:06.681 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195406Z content-type;host;x-amz-date 77226483663849461d98f2bf5299f31895b10c98c209ccfc783a71492f688a39 2024-04-16T19:54:06.681 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195406Z 20240416/test-zonegroup/iam/aws4_request 711fc347a4d095192c5d19f9aafa81d7f5f99cf8e68f23c865c8b9788cc34d30 2024-04-16T19:54:06.681 DEBUG:botocore.auth:Signature: 39bbbb595969d71acccf15280a8d013f1662a8916b512c81dfef8fc7de5b8c22 2024-04-16T19:54:06.681 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:06.681 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:06.684 DEBUG:botocore.parsers:Response headers: {'Content-Length': '328', 'x-amz-request-id': 'tx0000036e705b416b37da4-00661ed75e-5331-test-zone2', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.684 DEBUG:botocore.parsers:Response body: b'NoSuchEntityNo such RoleName in the tenanttx0000036e705b416b37da4-00661ed75e-5331-test-zone25331-test-zone2-test-zonegroup' 2024-04-16T19:54:06.686 DEBUG:botocore.parsers:Response headers: {'Content-Length': '328', 'x-amz-request-id': 'tx0000036e705b416b37da4-00661ed75e-5331-test-zone2', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:06 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:06.686 DEBUG:botocore.parsers:Response body: b'NoSuchEntityNo such RoleName in the tenanttx0000036e705b416b37da4-00661ed75e-5331-test-zone25331-test-zone2-test-zonegroup' 2024-04-16T19:54:06.686 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:06.686 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:06.686 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:06.686 INFO:rgw_multi.tests:success, zone: test-zone2 does not have role: roles-yrjefh-3 2024-04-16T19:54:06.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_delete_sync ... ok 2024-04-16T19:54:06.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.687 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.734 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.734 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.784 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:06.807 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T19:54:06.807 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:06.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:06.808 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:06.856 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:06.856 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:06.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.903+0000 7f0516216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:06.905+0000 7f0516216800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:06.929 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.929 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T19:54:06.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T19:54:06.929 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:06.930 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:11.930 INFO:rgw_multi.tests:create topic zone=test-zone1 name=topic-yrjefh-1 2024-04-16T19:54:11.931 DEBUG:botocore.hooks:Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-16T19:54:11.931 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:11.931 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:11.932 DEBUG:botocore.hooks:Event before-call.sns.CreateTopic: calling handler 2024-04-16T19:54:11.932 DEBUG:botocore.hooks:Event before-call.sns.CreateTopic: calling handler 2024-04-16T19:54:11.932 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'topic-yrjefh-1', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'http://kaboom:9999', 'Attributes.entry.2.key': 'persistent', 'Attributes.entry.2.value': 'true'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:11.932 DEBUG:botocore.hooks:Event request-created.sns.CreateTopic: calling handler > 2024-04-16T19:54:11.933 DEBUG:botocore.hooks:Event choose-signer.sns.CreateTopic: calling handler 2024-04-16T19:54:11.933 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:11.933 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195411Z content-type;host;x-amz-date 6ea285337f3d2901b9784063b7470502ca9dcbf9383f4b30bb8b2af9c15ccdd1 2024-04-16T19:54:11.933 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195411Z 20240416/test-zonegroup/sns/aws4_request 6eff7ed354e76a14e0c72b37a921274ac52c867592b5fe52aa11b28df393a852 2024-04-16T19:54:11.933 DEBUG:botocore.auth:Signature: 50dae9fb41b8097f7593b436c8571628370d01a9838007e5dc6363f40e66fa01 2024-04-16T19:54:11.934 DEBUG:botocore.hooks:Event request-created.sns.CreateTopic: calling handler 2024-04-16T19:54:11.934 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:11.949 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000edb00c9ccb5bc253-00661ed763-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '337', 'Date': 'Tue, 16 Apr 2024 19:54:11 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:11.949 DEBUG:botocore.parsers:Response body: b'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1b482a2a9-ef36-488b-9666-ada656453328.4812.17127203250456216147' 2024-04-16T19:54:11.949 DEBUG:botocore.hooks:Event needs-retry.sns.CreateTopic: calling handler 2024-04-16T19:54:11.949 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:11.950 INFO:rgw_multi.tests:create topic zone=test-zone2 name=topic-yrjefh-2 2024-04-16T19:54:11.950 DEBUG:botocore.hooks:Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-16T19:54:11.950 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:11.950 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:11.951 DEBUG:botocore.hooks:Event before-call.sns.CreateTopic: calling handler 2024-04-16T19:54:11.951 DEBUG:botocore.hooks:Event before-call.sns.CreateTopic: calling handler 2024-04-16T19:54:11.951 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'topic-yrjefh-2', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'http://kaboom:9999', 'Attributes.entry.2.key': 'persistent', 'Attributes.entry.2.value': 'true'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:11.951 DEBUG:botocore.hooks:Event request-created.sns.CreateTopic: calling handler > 2024-04-16T19:54:11.951 DEBUG:botocore.hooks:Event choose-signer.sns.CreateTopic: calling handler 2024-04-16T19:54:11.952 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:11.952 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195411Z content-type;host;x-amz-date 485e4ca69f91960253a643c532fe0db5b5e57cd613a06750d5157b31631406db 2024-04-16T19:54:11.952 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195411Z 20240416/test-zonegroup/sns/aws4_request 202e2eccc4e8e65e7bff985b385acb76f6f84766db8bcd621f49adf8a5b5ab0a 2024-04-16T19:54:11.952 DEBUG:botocore.auth:Signature: 3b13eeddbd39f4a1af39116185d59d17d13b7eaf863dc241e155432c094dd6b0 2024-04-16T19:54:11.952 DEBUG:botocore.hooks:Event request-created.sns.CreateTopic: calling handler 2024-04-16T19:54:11.952 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:11.974 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006dce9b2ff72424c0-00661ed763-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '336', 'Date': 'Tue, 16 Apr 2024 19:54:11 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:11.975 DEBUG:botocore.parsers:Response body: b'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2eafd0fcf-abef-4654-893b-ccd3b2eb5bea.5331.7912432225648649408' 2024-04-16T19:54:11.975 DEBUG:botocore.hooks:Event needs-retry.sns.CreateTopic: calling handler 2024-04-16T19:54:11.975 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:11.975 DEBUG:rgw_multi.tests:topic_arns: ['arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1', 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'] 2024-04-16T19:54:11.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:11.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:11.976 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:12.031 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:12.031 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:12.087 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:12.110 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713297251.965282_828.1","last_update":"2024-04-16T19:54:11.965282Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T19:54:12.110 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:12.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:12.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.110 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:12.165 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:12.165 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:12.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:12.219+0000 7f2f3af33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:12.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:12.221+0000 7f2f3af33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T19:54:12.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T19:54:12.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297251.965282_828.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:11.965282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T19:54:12.246 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:12.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T19:54:12.247 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:12.247 DEBUG:botocore.hooks:Event before-parameter-build.sns.ListTopics: calling handler 2024-04-16T19:54:12.247 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:12.247 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:12.248 DEBUG:botocore.hooks:Event before-call.sns.ListTopics: calling handler 2024-04-16T19:54:12.248 DEBUG:botocore.hooks:Event before-call.sns.ListTopics: calling handler 2024-04-16T19:54:12.248 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListTopics) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListTopics', 'Version': '2010-03-31'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:12.248 DEBUG:botocore.hooks:Event request-created.sns.ListTopics: calling handler > 2024-04-16T19:54:12.248 DEBUG:botocore.hooks:Event choose-signer.sns.ListTopics: calling handler 2024-04-16T19:54:12.249 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:12.249 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195412Z content-type;host;x-amz-date 55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 2024-04-16T19:54:12.249 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195412Z 20240416/test-zonegroup/sns/aws4_request bddf64a8f5d2b8f90555dcedd13fa9531557a2620b0710be0044e4df0ad7e4db 2024-04-16T19:54:12.249 DEBUG:botocore.auth:Signature: 41cf30b5719ddf0bab493b45f855c5a1baa49f64b085399451163a36ce9e725d 2024-04-16T19:54:12.249 DEBUG:botocore.hooks:Event request-created.sns.ListTopics: calling handler 2024-04-16T19:54:12.249 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:12.253 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000003d1358660dd4c313-00661ed764-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1414', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:12.253 DEBUG:botocore.parsers:Response body: b'RGW11111111111111111topic-yrjefh-1http://kaboom:9999Version=2010-03-31topic-yrjefh-1FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1RGW11111111111111111topic-yrjefh-2http://kaboom:9999Version=2010-03-31&rgwx-uid=RGW11111111111111111&rgwx-zonegroup=test-zonegrouptopic-yrjefh-2FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2b482a2a9-ef36-488b-9666-ada656453328.4812.4400958456213062419' 2024-04-16T19:54:12.253 DEBUG:botocore.hooks:Event needs-retry.sns.ListTopics: calling handler 2024-04-16T19:54:12.253 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:12.254 DEBUG:rgw_multi.tests:topics for zone=test-zone1 = [{'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1'}, {'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'}] 2024-04-16T19:54:12.254 DEBUG:botocore.hooks:Event before-parameter-build.sns.ListTopics: calling handler 2024-04-16T19:54:12.254 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:12.254 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:12.255 DEBUG:botocore.hooks:Event before-call.sns.ListTopics: calling handler 2024-04-16T19:54:12.255 DEBUG:botocore.hooks:Event before-call.sns.ListTopics: calling handler 2024-04-16T19:54:12.255 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListTopics) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListTopics', 'Version': '2010-03-31'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:12.255 DEBUG:botocore.hooks:Event request-created.sns.ListTopics: calling handler > 2024-04-16T19:54:12.255 DEBUG:botocore.hooks:Event choose-signer.sns.ListTopics: calling handler 2024-04-16T19:54:12.256 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:12.256 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195412Z content-type;host;x-amz-date 55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 2024-04-16T19:54:12.256 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195412Z 20240416/test-zonegroup/sns/aws4_request d85286e07b540e4fa298579bcb62f7de89ca95e537225788046606c560a9ccb3 2024-04-16T19:54:12.256 DEBUG:botocore.auth:Signature: 43703228d02333abb8a0d4ab5bf6bc2729300f8b5346ed0ce6920e292deb3a58 2024-04-16T19:54:12.256 DEBUG:botocore.hooks:Event request-created.sns.ListTopics: calling handler 2024-04-16T19:54:12.256 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:12.259 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000eb2899fcc08c7bd5-00661ed764-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1415', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:12.259 DEBUG:botocore.parsers:Response body: b'RGW11111111111111111topic-yrjefh-1http://kaboom:9999Version=2010-03-31topic-yrjefh-1FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1RGW11111111111111111topic-yrjefh-2http://kaboom:9999Version=2010-03-31&rgwx-uid=RGW11111111111111111&rgwx-zonegroup=test-zonegrouptopic-yrjefh-2FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2eafd0fcf-abef-4654-893b-ccd3b2eb5bea.5331.16944962908822731733' 2024-04-16T19:54:12.259 DEBUG:botocore.hooks:Event needs-retry.sns.ListTopics: calling handler 2024-04-16T19:54:12.259 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:12.260 DEBUG:rgw_multi.tests:topics for zone=test-zone2 = [{'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1'}, {'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'}] 2024-04-16T19:54:12.260 DEBUG:boto:path=/yrjefh-49/ 2024-04-16T19:54:12.260 DEBUG:boto:auth_path=/yrjefh-49/ 2024-04-16T19:54:12.260 DEBUG:boto:Method: PUT 2024-04-16T19:54:12.260 DEBUG:boto:Path: /yrjefh-49/ 2024-04-16T19:54:12.260 DEBUG:boto:Data: 2024-04-16T19:54:12.260 DEBUG:boto:Headers: {} 2024-04-16T19:54:12.260 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:12.260 DEBUG:boto:Port: 8000 2024-04-16T19:54:12.260 DEBUG:boto:Params: {} 2024-04-16T19:54:12.261 DEBUG:boto:Token: None 2024-04-16T19:54:12.261 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:54:12 GMT /yrjefh-49/ 2024-04-16T19:54:12.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6N4SwSnwJesEvhXD92ZAd8aH2oc= 2024-04-16T19:54:12.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6N4SwSnwJesEvhXD92ZAd8aH2oc=', 'Content-Length': '0'} 2024-04-16T19:54:12.279 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b53afeb19387562-00661ed764-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:12.279 DEBUG:rgw_multi.tests:created bucket=yrjefh-49 2024-04-16T19:54:12.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.280 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:12.328 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:12.401 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713297251.965282_828.1","last_update":"2024-04-16T19:54:11.965282Z"},{"marker":"1_1713297252.276314_637.1","last_update":"2024-04-16T19:54:12.276314Z"}] 2024-04-16T19:54:12.402 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T19:54:12.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:12.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:12.402 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:12.452 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:12.452 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:12.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:12.503+0000 7f9d8272a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:12.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:12.505+0000 7f9d8272a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297252.276314_637.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:12.276314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T19:54:12.529 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297252.276314_637.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:12.276314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T19:54:12.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297251.965282_828.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:11.965282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297252.276314_637.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:12.276314Z', 'realm_epoch': 2}}] 2024-04-16T19:54:12.529 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T19:54:12.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T19:54:12.529 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:12.529 INFO:rgw_multi.tests:creating bucket notification for zone=test-zone1 name=bn-yrjefh-1 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.530 DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler 2024-04-16T19:54:12.531 DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler > 2024-04-16T19:54:12.531 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'yrjefh-49', 'Region': 'test-zonegroup', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True, 'UseS3ExpressControlEndpoint': True} 2024-04-16T19:54:12.532 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000/yrjefh-49 2024-04-16T19:54:12.532 DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-16T19:54:12.532 DEBUG:botocore.regions:Selected auth type "v4" as "v4" with signing context params: {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-04-16T19:54:12.534 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.534 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.534 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.534 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.534 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': b'bn-yrjefh-1arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1s3:ObjectCreated:*', 'auth_path': '/yrjefh-49/', 'url': 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-49?notification', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'yrjefh-49', 'params': {'Bucket': 'yrjefh-49', 'NotificationConfiguration': {'TopicConfigurations': [{'Id': 'bn-yrjefh-1', 'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1', 'Events': ['s3:ObjectCreated:*']}]}}}, 'S3Express': {'bucket_name': 'yrjefh-49'}, 'signing': {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'test-zonegroup'}]}}} 2024-04-16T19:54:12.534 DEBUG:botocore.hooks:Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.535 DEBUG:botocore.hooks:Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.535 DEBUG:botocore.hooks:Event before-sign.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.535 DEBUG:botocore.hooks:Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.535 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:12.535 DEBUG:botocore.auth:CanonicalRequest: PUT /yrjefh-49 notification= host:smithi026.front.sepia.ceph.com:8000 x-amz-content-sha256:be87c4a3acfd35705d711d944633e64c1ba48da889de87610544d17cdea27e1e x-amz-date:20240416T195412Z host;x-amz-content-sha256;x-amz-date be87c4a3acfd35705d711d944633e64c1ba48da889de87610544d17cdea27e1e 2024-04-16T19:54:12.536 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195412Z 20240416/test-zonegroup/s3/aws4_request 226d73d40d3e32b0a4a709841ebbbf799f1eb0a9ebc2098bdeaa46acec794bb4 2024-04-16T19:54:12.536 DEBUG:botocore.auth:Signature: ad152cde17734ea01472cac197c735adeef270113bb4d56f0319f86002bbce81 2024-04-16T19:54:12.536 DEBUG:botocore.hooks:Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.536 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:12.546 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000bce598ca9790daa1-00661ed764-4812-test-zone1', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '0', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:12.546 DEBUG:botocore.parsers:Response body: b'' 2024-04-16T19:54:12.546 DEBUG:botocore.hooks:Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.546 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:12.546 DEBUG:botocore.hooks:Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.547 INFO:rgw_multi.tests:creating bucket notification for zone=test-zone2 name=bn-yrjefh-2 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler 2024-04-16T19:54:12.547 DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler > 2024-04-16T19:54:12.548 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'yrjefh-49', 'Region': 'test-zonegroup', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True, 'UseS3ExpressControlEndpoint': True} 2024-04-16T19:54:12.548 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000/yrjefh-49 2024-04-16T19:54:12.548 DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-16T19:54:12.548 DEBUG:botocore.regions:Selected auth type "v4" as "v4" with signing context params: {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-04-16T19:54:12.549 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.549 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.549 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.549 DEBUG:botocore.hooks:Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.549 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': b'bn-yrjefh-2arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2s3:ObjectCreated:*', 'auth_path': '/yrjefh-49/', 'url': 'http://smithi119.front.sepia.ceph.com:8000/yrjefh-49?notification', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'yrjefh-49', 'params': {'Bucket': 'yrjefh-49', 'NotificationConfiguration': {'TopicConfigurations': [{'Id': 'bn-yrjefh-2', 'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2', 'Events': ['s3:ObjectCreated:*']}]}}}, 'S3Express': {'bucket_name': 'yrjefh-49'}, 'signing': {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'test-zonegroup'}]}}} 2024-04-16T19:54:12.550 DEBUG:botocore.hooks:Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.550 DEBUG:botocore.hooks:Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.550 DEBUG:botocore.hooks:Event before-sign.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.550 DEBUG:botocore.hooks:Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:12.550 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:12.550 DEBUG:botocore.auth:CanonicalRequest: PUT /yrjefh-49 notification= host:smithi119.front.sepia.ceph.com:8000 x-amz-content-sha256:09fafb09dfdc35bb179c0f67bc7cfb4d7f29d8187ab56ed97957fecd46102881 x-amz-date:20240416T195412Z host;x-amz-content-sha256;x-amz-date 09fafb09dfdc35bb179c0f67bc7cfb4d7f29d8187ab56ed97957fecd46102881 2024-04-16T19:54:12.551 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195412Z 20240416/test-zonegroup/s3/aws4_request f817c808b14a1195f9b70dd7b79eea454fa090b7c0b41e01d9c81b1b5708a649 2024-04-16T19:54:12.551 DEBUG:botocore.auth:Signature: f62613d11e6117884d71f4887a1653eea86debab513057db386452716b9542d9 2024-04-16T19:54:12.551 DEBUG:botocore.hooks:Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:12.551 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.567 DEBUG:botocore.parsers:Response headers: {'Content-Length': '261', 'x-amz-request-id': 'tx00000a0b4a01262222d3b-00661ed764-5331-test-zone2', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.567 DEBUG:botocore.parsers:Response body: b'ConcurrentModificationyrjefh-49tx00000a0b4a01262222d3b-00661ed764-5331-test-zone25331-test-zone2-test-zonegroup' 2024-04-16T19:54:13.570 DEBUG:botocore.hooks:Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T19:54:13.570 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.570 DEBUG:botocore.hooks:Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T19:54:13.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_topic_notification_sync ... ERROR 2024-04-16T19:54:13.664 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateUser: calling handler 2024-04-16T19:54:13.664 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.664 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.664 DEBUG:botocore.hooks:Event before-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.665 DEBUG:botocore.hooks:Event before-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.665 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.665 DEBUG:botocore.hooks:Event request-created.iam.CreateUser: calling handler > 2024-04-16T19:54:13.665 DEBUG:botocore.hooks:Event choose-signer.iam.CreateUser: calling handler 2024-04-16T19:54:13.666 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.666 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date f7475b30454e8ffcb361ce7c694a8facda3014395dff383d48edb4ca989ef6b0 2024-04-16T19:54:13.666 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request d22939c7e0d9ed36af451c5337de54c8e65cf3f96419b6f26495948e5aa16e2d 2024-04-16T19:54:13.666 DEBUG:botocore.auth:Signature: b6a5b6686ce5cf9a2f3bb93265bd48a9c24fb70ca2aff58ffb5cfdf63a9aba42 2024-04-16T19:54:13.666 DEBUG:botocore.hooks:Event request-created.iam.CreateUser: calling handler 2024-04-16T19:54:13.666 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.677 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000070753a43feb55f40-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '499', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.677 DEBUG:botocore.parsers:Response body: b'/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone12024-04-16T19:54:13.669293Ztx0000070753a43feb55f40-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.678 DEBUG:botocore.hooks:Event needs-retry.iam.CreateUser: calling handler 2024-04-16T19:54:13.678 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.679 DEBUG:botocore.hooks:Event after-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.679 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.679 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.679 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.679 DEBUG:botocore.hooks:Event before-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.680 DEBUG:botocore.hooks:Event before-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.680 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateAccessKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateAccessKey', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.680 DEBUG:botocore.hooks:Event request-created.iam.CreateAccessKey: calling handler > 2024-04-16T19:54:13.680 DEBUG:botocore.hooks:Event choose-signer.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.680 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.681 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 2d56745a01c086ed0203077dd4eef7bcaf00472bb3389da335fbf28f3732fcd0 2024-04-16T19:54:13.681 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request f9ee35b493039eb5d9e21de95f0a2badbd5a22933199e332bd43a3f2ec4154ac 2024-04-16T19:54:13.681 DEBUG:botocore.auth:Signature: 6d0be9e6a50d5b375af286aa997deb0114bbaf90e138d954f092f870f579507d 2024-04-16T19:54:13.681 DEBUG:botocore.hooks:Event request-created.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.681 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.692 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004944ef2874502cf5-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '547', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.692 DEBUG:botocore.parsers:Response body: b'test-zone1IBVMW6I15HR0T84KDGA7Active2024-04-16T19:54:13.683670Zhl4LwaZptxhhn4RVPWNSejrJjUX4dyGc32t03DSotx000004944ef2874502cf5-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.692 DEBUG:botocore.hooks:Event needs-retry.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.693 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.693 DEBUG:botocore.hooks:Event after-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.693 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.693 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.693 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.694 DEBUG:botocore.hooks:Event before-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.694 DEBUG:botocore.hooks:Event before-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.694 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone1', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.694 DEBUG:botocore.hooks:Event request-created.iam.PutUserPolicy: calling handler > 2024-04-16T19:54:13.694 DEBUG:botocore.hooks:Event choose-signer.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.695 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.695 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 17879975ab2692d579dfbda99a028060b29570530de8c9f87e064005597017ac 2024-04-16T19:54:13.695 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request ba37de925b2a63e45ba9c648349a8257a9bf36319bc40670f4b1f962c30676db 2024-04-16T19:54:13.695 DEBUG:botocore.auth:Signature: 76d815d2330205e1390177b7404e627918eb0f8e94bed69b8663511edf0d0a4d 2024-04-16T19:54:13.695 DEBUG:botocore.hooks:Event request-created.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.695 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.706 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000f27137860429b86-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '207', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.706 DEBUG:botocore.parsers:Response body: b'tx000000f27137860429b86-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.708 DEBUG:botocore.hooks:Event needs-retry.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.708 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.708 DEBUG:botocore.hooks:Event after-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:13.708 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.708 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.708 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.709 DEBUG:botocore.hooks:Event before-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.709 DEBUG:botocore.hooks:Event before-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.709 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.709 DEBUG:botocore.hooks:Event request-created.iam.AttachUserPolicy: calling handler > 2024-04-16T19:54:13.709 DEBUG:botocore.hooks:Event choose-signer.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.710 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.710 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 6bae6a67a9faf220198a4254a71b23780816be124b7a0f3c4c5492c739caeaa2 2024-04-16T19:54:13.710 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 6c58b9de19d463cb41db7cae30e5f2d1e770e5b0b7b13623d1792bc6a60912eb 2024-04-16T19:54:13.710 DEBUG:botocore.auth:Signature: 1701a78692e1410fd282670b7901840b30a001e6de80d317cd9af67b53e2d305 2024-04-16T19:54:13.710 DEBUG:botocore.hooks:Event request-created.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.710 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.762 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000056889bc5eb34b897-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.763 DEBUG:botocore.parsers:Response body: b'tx0000056889bc5eb34b897-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.763 DEBUG:botocore.hooks:Event needs-retry.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.763 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.764 DEBUG:botocore.hooks:Event after-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:13.764 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateGroup: calling handler 2024-04-16T19:54:13.764 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.764 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.764 DEBUG:botocore.hooks:Event before-call.iam.CreateGroup: calling handler 2024-04-16T19:54:13.765 DEBUG:botocore.hooks:Event before-call.iam.CreateGroup: calling handler 2024-04-16T19:54:13.765 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.765 DEBUG:botocore.hooks:Event request-created.iam.CreateGroup: calling handler > 2024-04-16T19:54:13.765 DEBUG:botocore.hooks:Event choose-signer.iam.CreateGroup: calling handler 2024-04-16T19:54:13.766 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.766 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 624e928373ac8004a209ec47e507ab54a865f60c2f07522729232344f4ba6c8a 2024-04-16T19:54:13.766 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 82197dd9ad455c55a659d764a79e9c000335c9fe31d162c8ce1d3d782e81fc3d 2024-04-16T19:54:13.766 DEBUG:botocore.auth:Signature: ba0aed18881edc1f8c8485c69f8f895c3838e23337596d47bffe12a52f7b9275 2024-04-16T19:54:13.766 DEBUG:botocore.hooks:Event request-created.iam.CreateGroup: calling handler 2024-04-16T19:54:13.766 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.777 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000001762b55fc558e763-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '458', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.777 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1tx000001762b55fc558e763-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.778 DEBUG:botocore.hooks:Event needs-retry.iam.CreateGroup: calling handler 2024-04-16T19:54:13.778 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.778 DEBUG:botocore.hooks:Event after-call.iam.CreateGroup: calling handler 2024-04-16T19:54:13.778 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.779 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.779 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.779 DEBUG:botocore.hooks:Event before-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.779 DEBUG:botocore.hooks:Event before-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.779 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.779 DEBUG:botocore.hooks:Event request-created.iam.PutGroupPolicy: calling handler > 2024-04-16T19:54:13.780 DEBUG:botocore.hooks:Event choose-signer.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.780 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.780 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date ad4c82339605c3b58275f685e51aaa8f175f9d7bfa75f418f3b90f7c7b4bf620 2024-04-16T19:54:13.780 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 02afc140e60171e827e01c5e70cadc1b78068ff960bf9beaa64ccbbad08b0e9d 2024-04-16T19:54:13.780 DEBUG:botocore.auth:Signature: 6cdd3ad0320b3c54fa4cc8b114156cf115652437738b15d85ebff42421d07118 2024-04-16T19:54:13.781 DEBUG:botocore.hooks:Event request-created.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.781 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.787 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000a19f7d32552d796c-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '247', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.787 DEBUG:botocore.parsers:Response body: b'tx00000a19f7d32552d796c-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.787 DEBUG:botocore.hooks:Event needs-retry.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.788 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.788 DEBUG:botocore.hooks:Event after-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:13.788 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.788 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.788 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.789 DEBUG:botocore.hooks:Event before-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.789 DEBUG:botocore.hooks:Event before-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.789 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.789 DEBUG:botocore.hooks:Event request-created.iam.AttachGroupPolicy: calling handler > 2024-04-16T19:54:13.789 DEBUG:botocore.hooks:Event choose-signer.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.790 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.790 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 6650eb41f571a51390e5e4d8fe1d82748ac41b679798f9d70d71a7c097ba721b 2024-04-16T19:54:13.790 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 8130ccec85ea7de21455f8726a07caa935a447bcd252708e37690ce3d0c7d032 2024-04-16T19:54:13.790 DEBUG:botocore.auth:Signature: 5d135cf658d0fdd803ca8ef54eb5f572ad1e95e6a0427354fa20ffbdb4a8c0c2 2024-04-16T19:54:13.790 DEBUG:botocore.hooks:Event request-created.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.790 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.838 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000009ffa665260c03afc-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.839 DEBUG:botocore.parsers:Response body: b'tx000009ffa665260c03afc-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.839 DEBUG:botocore.hooks:Event needs-retry.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.839 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.839 DEBUG:botocore.hooks:Event after-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:13.839 DEBUG:botocore.hooks:Event before-parameter-build.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.840 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.840 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.840 DEBUG:botocore.hooks:Event before-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.840 DEBUG:botocore.hooks:Event before-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.840 DEBUG:botocore.endpoint:Making request for OperationModel(name=AddUserToGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AddUserToGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.841 DEBUG:botocore.hooks:Event request-created.iam.AddUserToGroup: calling handler > 2024-04-16T19:54:13.841 DEBUG:botocore.hooks:Event choose-signer.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.841 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.841 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date b828f4a28c8d0ee119b54a503ae785618cc0a7bd2f1ba220c61d60c88189b9de 2024-04-16T19:54:13.841 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 409dd53ad5739f19bfad706cc73976a8a712b0272dae65648518cdb3ec236665 2024-04-16T19:54:13.841 DEBUG:botocore.auth:Signature: e7bc5846b19eb1ea2111acc155fe5cbc770f1291c503981f736cec9784c96bc5 2024-04-16T19:54:13.842 DEBUG:botocore.hooks:Event request-created.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.842 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.851 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004ffbdbeb80586d36-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '247', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.851 DEBUG:botocore.parsers:Response body: b'tx000004ffbdbeb80586d36-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.851 DEBUG:botocore.hooks:Event needs-retry.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.851 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.851 DEBUG:botocore.hooks:Event after-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:13.851 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateRole: calling handler 2024-04-16T19:54:13.852 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.852 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.852 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:13.852 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:13.852 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateRole', 'Version': '2010-05-08', 'RoleName': 'test-zone1', 'AssumeRolePolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.852 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler > 2024-04-16T19:54:13.852 DEBUG:botocore.hooks:Event choose-signer.iam.CreateRole: calling handler 2024-04-16T19:54:13.853 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.853 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date b8dfbdc932156a00bb2505dfd736bc41e7bb2b32934a250726a1d2ae8ac7126c 2024-04-16T19:54:13.853 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request c497a0b57590f595e34387f6e8f68f9575b08b888f879b6ac16fa065171096e1 2024-04-16T19:54:13.853 DEBUG:botocore.auth:Signature: 2edd284c71e55a0d8ffc27762af402dc8cd87cc8a549f4e2b189af922edfd879 2024-04-16T19:54:13.853 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler 2024-04-16T19:54:13.853 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.865 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000001d3c03b72940592e-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '777', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.865 DEBUG:botocore.parsers:Response body: b'6e7ac441-491c-4598-8785-7de38e426f36test-zone1/arn:aws:iam::RGW11111111111111111:role/test-zone12024-04-16T19:54:13.855Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}tx000001d3c03b72940592e-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.866 DEBUG:botocore.hooks:Event needs-retry.iam.CreateRole: calling handler 2024-04-16T19:54:13.866 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.866 DEBUG:botocore.hooks:Event after-call.iam.CreateRole: calling handler 2024-04-16T19:54:13.866 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.867 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.867 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.867 DEBUG:botocore.hooks:Event before-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.867 DEBUG:botocore.hooks:Event before-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.867 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone1', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.867 DEBUG:botocore.hooks:Event request-created.iam.PutRolePolicy: calling handler > 2024-04-16T19:54:13.868 DEBUG:botocore.hooks:Event choose-signer.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.868 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.868 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date fbdda2fe5860cad2b2ea0a554f4430d72305dbdc0b7565c6cbd1cba689429a4d 2024-04-16T19:54:13.868 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request ec9657b51cf9ffa5b9cb5c0b087617520e72ada9990e87fff4d394b5f0dfcfb2 2024-04-16T19:54:13.868 DEBUG:botocore.auth:Signature: d8b1dd3c768c11b2fbc0cba792143d35a4d08eabefcc39bd7f31464580e45443 2024-04-16T19:54:13.868 DEBUG:botocore.hooks:Event request-created.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.869 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.918 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000aaacb0a1b4506f70-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '157', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.919 DEBUG:botocore.parsers:Response body: b'tx00000aaacb0a1b4506f70-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.919 DEBUG:botocore.hooks:Event needs-retry.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.919 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.919 DEBUG:botocore.hooks:Event after-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:13.919 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.920 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.920 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:13.920 DEBUG:botocore.hooks:Event before-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.920 DEBUG:botocore.hooks:Event before-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.920 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.920 DEBUG:botocore.hooks:Event request-created.iam.AttachRolePolicy: calling handler > 2024-04-16T19:54:13.921 DEBUG:botocore.hooks:Event choose-signer.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.921 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.921 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 98ed48432946a0db032c442aa66c08bd0e2e471826a731d2e6807950820ab209 2024-04-16T19:54:13.921 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 7a3fb6494255505a73fdb996f35d1b42d2dafad77cc67f00400a75cf59f7c44f 2024-04-16T19:54:13.921 DEBUG:botocore.auth:Signature: f319dc2edc367ec98273ea257da8998567182bc7ac2e5a739878a38845dd116f 2024-04-16T19:54:13.922 DEBUG:botocore.hooks:Event request-created.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.922 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.929 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e02aecdd43a146f4-00661ed765-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.929 DEBUG:botocore.parsers:Response body: b'tx00000e02aecdd43a146f4-00661ed765-4812-test-zone1' 2024-04-16T19:54:13.929 DEBUG:botocore.hooks:Event needs-retry.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.929 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.929 DEBUG:botocore.hooks:Event after-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:13.930 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateUser: calling handler 2024-04-16T19:54:13.930 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.930 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:13.930 DEBUG:botocore.hooks:Event before-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.931 DEBUG:botocore.hooks:Event before-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.931 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.931 DEBUG:botocore.hooks:Event request-created.iam.CreateUser: calling handler > 2024-04-16T19:54:13.931 DEBUG:botocore.hooks:Event choose-signer.iam.CreateUser: calling handler 2024-04-16T19:54:13.932 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.932 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date 2510cb179a00379b5bc09d3a2af02e7e7b61a5cb434f7fa528d9845bf96444f3 2024-04-16T19:54:13.932 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 2b85892d1afcdaf64b767534cef29f0c20bbc39af2df80ebaf5f33dffe83f46c 2024-04-16T19:54:13.932 DEBUG:botocore.auth:Signature: 1eec3181451c9f9d77c6ac7d86d5314337e9cc5d01b6ba5d03aa558ac96ecbd9 2024-04-16T19:54:13.932 DEBUG:botocore.hooks:Event request-created.iam.CreateUser: calling handler 2024-04-16T19:54:13.932 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:13.948 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000008160554db181a126-00661ed765-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '499', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:13.948 DEBUG:botocore.parsers:Response body: b'/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone22024-04-16T19:54:13.934672Ztx000008160554db181a126-00661ed765-5331-test-zone2' 2024-04-16T19:54:13.949 DEBUG:botocore.hooks:Event needs-retry.iam.CreateUser: calling handler 2024-04-16T19:54:13.949 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:13.949 DEBUG:botocore.hooks:Event after-call.iam.CreateUser: calling handler 2024-04-16T19:54:13.949 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.949 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:13.949 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:13.950 DEBUG:botocore.hooks:Event before-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.950 DEBUG:botocore.hooks:Event before-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.950 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateAccessKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateAccessKey', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:13.950 DEBUG:botocore.hooks:Event request-created.iam.CreateAccessKey: calling handler > 2024-04-16T19:54:13.950 DEBUG:botocore.hooks:Event choose-signer.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.951 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:13.951 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195413Z content-type;host;x-amz-date a0b5d5b3609514e9ba50d912aefb3a37850c6755f9d7985c4c4bb1c50f4d7f4b 2024-04-16T19:54:13.951 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195413Z 20240416/test-zonegroup/iam/aws4_request 3a8b0c8e4f3c04af56d39aab653909f2942926c31e3fbee94f891146e90fe2b4 2024-04-16T19:54:13.951 DEBUG:botocore.auth:Signature: cc250f7f6d86530d5cbe658f1cc970d10dd314ca29194047f7412e0cd5573e8d 2024-04-16T19:54:13.951 DEBUG:botocore.hooks:Event request-created.iam.CreateAccessKey: calling handler 2024-04-16T19:54:13.951 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.010 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000ca49fd4dd60540e2-00661ed765-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '547', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.010 DEBUG:botocore.parsers:Response body: b'test-zone231HO20TWHBMRQFU8RLUEActive2024-04-16T19:54:13.956210ZL2BhAGuSGgXqixFfvZOooLkjidIkRMO4dbnMhm7otx00000ca49fd4dd60540e2-00661ed765-5331-test-zone2' 2024-04-16T19:54:14.011 DEBUG:botocore.hooks:Event needs-retry.iam.CreateAccessKey: calling handler 2024-04-16T19:54:14.011 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.011 DEBUG:botocore.hooks:Event after-call.iam.CreateAccessKey: calling handler 2024-04-16T19:54:14.011 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.011 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.011 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.012 DEBUG:botocore.hooks:Event before-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.012 DEBUG:botocore.hooks:Event before-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.012 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone2', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.012 DEBUG:botocore.hooks:Event request-created.iam.PutUserPolicy: calling handler > 2024-04-16T19:54:14.012 DEBUG:botocore.hooks:Event choose-signer.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.013 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.013 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 85bc49ee9891ccaf4f94397e9650c15250b819764da6ef0a05cb2450c32e0d43 2024-04-16T19:54:14.013 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 8d5bd9af9b61733feae72c343dc1defd137b42530e876fee13fd462c0b4f4a19 2024-04-16T19:54:14.013 DEBUG:botocore.auth:Signature: 749b740b9519a6a2013a7d73923d1e73272878230c1f5a5a23bf6a6b67332461 2024-04-16T19:54:14.013 DEBUG:botocore.hooks:Event request-created.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.013 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.122 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000ed55733754ae040-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '207', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.123 DEBUG:botocore.parsers:Response body: b'tx000000ed55733754ae040-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.123 DEBUG:botocore.hooks:Event needs-retry.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.123 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.123 DEBUG:botocore.hooks:Event after-call.iam.PutUserPolicy: calling handler 2024-04-16T19:54:14.124 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.124 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.124 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.124 DEBUG:botocore.hooks:Event before-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.124 DEBUG:botocore.hooks:Event before-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.124 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.125 DEBUG:botocore.hooks:Event request-created.iam.AttachUserPolicy: calling handler > 2024-04-16T19:54:14.125 DEBUG:botocore.hooks:Event choose-signer.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.125 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.125 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 9fa30c027a5dbbf088ce980f0476d5a9a21aa2fdc24be06e3279cba8bcdc470b 2024-04-16T19:54:14.126 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request c2470a65de13a8ae34b88e55114f320b53e8eea2cfa97107814a5df4abd2ecaa 2024-04-16T19:54:14.126 DEBUG:botocore.auth:Signature: a175058b916e7befda653413da91feac05e65f4b83d296973f2a323727ca41ee 2024-04-16T19:54:14.126 DEBUG:botocore.hooks:Event request-created.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.126 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.146 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000009ba9ca6461a859c3-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.146 DEBUG:botocore.parsers:Response body: b'tx000009ba9ca6461a859c3-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.146 DEBUG:botocore.hooks:Event needs-retry.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.147 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.147 DEBUG:botocore.hooks:Event after-call.iam.AttachUserPolicy: calling handler 2024-04-16T19:54:14.147 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateGroup: calling handler 2024-04-16T19:54:14.147 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.147 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.148 DEBUG:botocore.hooks:Event before-call.iam.CreateGroup: calling handler 2024-04-16T19:54:14.148 DEBUG:botocore.hooks:Event before-call.iam.CreateGroup: calling handler 2024-04-16T19:54:14.148 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.148 DEBUG:botocore.hooks:Event request-created.iam.CreateGroup: calling handler > 2024-04-16T19:54:14.148 DEBUG:botocore.hooks:Event choose-signer.iam.CreateGroup: calling handler 2024-04-16T19:54:14.149 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.149 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 24a14e11b4493ec41142bbe4ace87bd6f70ba76e166d59411fd5edee5bcde329 2024-04-16T19:54:14.149 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 90581b8cf9619366d34d65f97622df8f25c41c7d87bb3041b16a6b81bfa2344a 2024-04-16T19:54:14.149 DEBUG:botocore.auth:Signature: 4c48b87aa0f4a31d24940f156e7d74b43c069750915abfef5d69ce550db79cc9 2024-04-16T19:54:14.149 DEBUG:botocore.hooks:Event request-created.iam.CreateGroup: calling handler 2024-04-16T19:54:14.149 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.209 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000626365b1cdc6e6f8-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '458', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.209 DEBUG:botocore.parsers:Response body: b'/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2tx00000626365b1cdc6e6f8-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.209 DEBUG:botocore.hooks:Event needs-retry.iam.CreateGroup: calling handler 2024-04-16T19:54:14.209 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.209 DEBUG:botocore.hooks:Event after-call.iam.CreateGroup: calling handler 2024-04-16T19:54:14.210 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.210 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.210 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.210 DEBUG:botocore.hooks:Event before-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.211 DEBUG:botocore.hooks:Event before-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.211 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.211 DEBUG:botocore.hooks:Event request-created.iam.PutGroupPolicy: calling handler > 2024-04-16T19:54:14.211 DEBUG:botocore.hooks:Event choose-signer.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.211 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.212 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date ab7d67234b7569246e67b8e19e7bdcd215174800a0c659b8f2e5f4ac095702b5 2024-04-16T19:54:14.212 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 129061c26c67d5cc1e1c0f986d4b7d2270e0404b101d2f83e7f4a616b0b26459 2024-04-16T19:54:14.212 DEBUG:botocore.auth:Signature: 1fdb6dde2b064acaa2ecfe17b82212720b3eea03e1f2f4dbc001d86023173312 2024-04-16T19:54:14.212 DEBUG:botocore.hooks:Event request-created.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.212 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.306 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000051db29534f810732-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '247', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.306 DEBUG:botocore.parsers:Response body: b'tx0000051db29534f810732-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.307 DEBUG:botocore.hooks:Event needs-retry.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.307 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.307 DEBUG:botocore.hooks:Event after-call.iam.PutGroupPolicy: calling handler 2024-04-16T19:54:14.307 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.308 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.308 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.308 DEBUG:botocore.hooks:Event before-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.308 DEBUG:botocore.hooks:Event before-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.308 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.309 DEBUG:botocore.hooks:Event request-created.iam.AttachGroupPolicy: calling handler > 2024-04-16T19:54:14.309 DEBUG:botocore.hooks:Event choose-signer.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.309 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.309 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 08d5ff802492dc9987af66bd4af8637b58974f1703349e3edf09538e4f34ef3a 2024-04-16T19:54:14.310 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 90bbed9dd1aa82650ab5ae119670a14e617e6dafe295ba84a4fb90737722cd9a 2024-04-16T19:54:14.310 DEBUG:botocore.auth:Signature: 02e5a54baad9ceafffb9a10538db45ada4feaa8a5fd3bd48212941c00ce933b1 2024-04-16T19:54:14.310 DEBUG:botocore.hooks:Event request-created.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.310 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.321 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000027b3d1727e28ce9e-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.322 DEBUG:botocore.parsers:Response body: b'tx0000027b3d1727e28ce9e-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.322 DEBUG:botocore.hooks:Event needs-retry.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.322 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.322 DEBUG:botocore.hooks:Event after-call.iam.AttachGroupPolicy: calling handler 2024-04-16T19:54:14.323 DEBUG:botocore.hooks:Event before-parameter-build.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.323 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.323 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.323 DEBUG:botocore.hooks:Event before-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.323 DEBUG:botocore.hooks:Event before-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.324 DEBUG:botocore.endpoint:Making request for OperationModel(name=AddUserToGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AddUserToGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.324 DEBUG:botocore.hooks:Event request-created.iam.AddUserToGroup: calling handler > 2024-04-16T19:54:14.324 DEBUG:botocore.hooks:Event choose-signer.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.324 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.325 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date b1a71af0ad1d86e1dce0f1ed3358241dd82bb321ef00b4200a506a73f65811f4 2024-04-16T19:54:14.325 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request fc8c20ad30d339b848628964fdd0b1bd86c1b545966d25249e38637a46343d19 2024-04-16T19:54:14.325 DEBUG:botocore.auth:Signature: 7dd03c6f61a1f87e556db9ba748c449de4099e4d6bb21b09599e9695075f2e23 2024-04-16T19:54:14.325 DEBUG:botocore.hooks:Event request-created.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.325 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.381 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000029d03771a2929ba-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '247', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.381 DEBUG:botocore.parsers:Response body: b'tx00000029d03771a2929ba-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.381 DEBUG:botocore.hooks:Event needs-retry.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.381 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.381 DEBUG:botocore.hooks:Event after-call.iam.AddUserToGroup: calling handler 2024-04-16T19:54:14.382 DEBUG:botocore.hooks:Event before-parameter-build.iam.CreateRole: calling handler 2024-04-16T19:54:14.382 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.382 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.382 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:14.382 DEBUG:botocore.hooks:Event before-call.iam.CreateRole: calling handler 2024-04-16T19:54:14.383 DEBUG:botocore.endpoint:Making request for OperationModel(name=CreateRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateRole', 'Version': '2010-05-08', 'RoleName': 'test-zone2', 'AssumeRolePolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.383 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler > 2024-04-16T19:54:14.383 DEBUG:botocore.hooks:Event choose-signer.iam.CreateRole: calling handler 2024-04-16T19:54:14.383 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.383 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 3b15dc3cd5d30d4e069c24bb6bdc8bba7e68b341ab2726d98d123919ff657ea8 2024-04-16T19:54:14.383 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 0f69c6a7190b0191d2351f78b29698dfb6529481ea9617d589d449011f764cdf 2024-04-16T19:54:14.384 DEBUG:botocore.auth:Signature: ceed69b640566caa34f7cc0cd99832cb1f96312bdf593827260e94a01b2491a5 2024-04-16T19:54:14.384 DEBUG:botocore.hooks:Event request-created.iam.CreateRole: calling handler 2024-04-16T19:54:14.384 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.490 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000041fabef96235e93f-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '777', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.490 DEBUG:botocore.parsers:Response body: b'33439692-517f-4a4e-a7c3-14a0cf35bc03test-zone2/arn:aws:iam::RGW11111111111111111:role/test-zone22024-04-16T19:54:14.388Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}tx0000041fabef96235e93f-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.491 DEBUG:botocore.hooks:Event needs-retry.iam.CreateRole: calling handler 2024-04-16T19:54:14.491 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.491 DEBUG:botocore.hooks:Event after-call.iam.CreateRole: calling handler 2024-04-16T19:54:14.492 DEBUG:botocore.hooks:Event before-parameter-build.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.492 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.492 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.492 DEBUG:botocore.hooks:Event before-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.492 DEBUG:botocore.hooks:Event before-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.493 DEBUG:botocore.endpoint:Making request for OperationModel(name=PutRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'PutRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone2', 'PolicyName': 'Allow', 'PolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.493 DEBUG:botocore.hooks:Event request-created.iam.PutRolePolicy: calling handler > 2024-04-16T19:54:14.493 DEBUG:botocore.hooks:Event choose-signer.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.493 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.494 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date d375b5364696b79344588bf03ce90d2c993e120cfc9ed210ac5505fa6b6852b7 2024-04-16T19:54:14.494 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 87592f732cb54271bc9fba1064d7986af04a14de2bd0e34a982a6f548ee80443 2024-04-16T19:54:14.494 DEBUG:botocore.auth:Signature: 6b40dbb77b15c9978d3105b63a8fee3b34756d89e3f7242b6be3e5493b2fdbef 2024-04-16T19:54:14.494 DEBUG:botocore.hooks:Event request-created.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.494 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.508 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000693007647bac44b0-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '157', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.508 DEBUG:botocore.parsers:Response body: b'tx00000693007647bac44b0-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.508 DEBUG:botocore.hooks:Event needs-retry.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.508 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.508 DEBUG:botocore.hooks:Event after-call.iam.PutRolePolicy: calling handler 2024-04-16T19:54:14.509 DEBUG:botocore.hooks:Event before-parameter-build.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.509 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:14.509 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:14.509 DEBUG:botocore.hooks:Event before-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.509 DEBUG:botocore.hooks:Event before-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.509 DEBUG:botocore.endpoint:Making request for OperationModel(name=AttachRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'AttachRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:14.510 DEBUG:botocore.hooks:Event request-created.iam.AttachRolePolicy: calling handler > 2024-04-16T19:54:14.510 DEBUG:botocore.hooks:Event choose-signer.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.510 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:14.511 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195414Z content-type;host;x-amz-date 769290fb146152a7afbdb9bd3a33d29951994cc1eb13e718908b74fc21378f03 2024-04-16T19:54:14.511 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195414Z 20240416/test-zonegroup/iam/aws4_request 9210bd5f510ee7a5471a3712ec9277409b88d532f474a62f78c2c7b325c0d9f5 2024-04-16T19:54:14.511 DEBUG:botocore.auth:Signature: 65d075b37492652f018f7fab6f26bb8a52be8d3fbbca22cc4c4c466a08cb3caf 2024-04-16T19:54:14.511 DEBUG:botocore.hooks:Event request-created.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.511 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:14.564 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004f090525f63f46a6-00661ed766-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:14 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:14.564 DEBUG:botocore.parsers:Response body: b'tx000004f090525f63f46a6-00661ed766-5331-test-zone2' 2024-04-16T19:54:14.564 DEBUG:botocore.hooks:Event needs-retry.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.564 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:14.564 DEBUG:botocore.hooks:Event after-call.iam.AttachRolePolicy: calling handler 2024-04-16T19:54:14.565 INFO:rgw_multi.tests:meta checkpoint 2024-04-16T19:54:14.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:14.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:14.565 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:14.677 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:14.701 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297254.332712_126.1","last_update":"2024-04-16T19:54:14.332712Z"},{"marker":"1_1713297253.847516_845.1","last_update":"2024-04-16T19:54:13.847516Z"},{"marker":"1_1713297254.517560_656.1","last_update":"2024-04-16T19:54:14.517560Z"}] 2024-04-16T19:54:14.701 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297254.332712_126.1', 2: '1_1713297253.847516_845.1', 3: '1_1713297254.517560_656.1'} 2024-04-16T19:54:14.701 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:14.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:14.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:14.701 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:14.757 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:14.757 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:14.809+0000 7fa4c2d2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:14.811+0000 7fa4c2d2c800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:14.834 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297254.134542_124.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.134542Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297253.847516_845.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:13.847516Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297254.393091_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.393091Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:14Z"} 2024-04-16T19:54:14.835 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297254.134542_124.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.134542Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297253.847516_845.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:13.847516Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297254.393091_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.393091Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:14Z"} 2024-04-16T19:54:14.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297254.134542_124.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:14.134542Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297253.847516_845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:13.847516Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297254.393091_652.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:14.393091Z', 'realm_epoch': 2}}] 2024-04-16T19:54:14.835 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297254.332712_126.1', 2: '1_1713297253.847516_845.1', 3: '1_1713297254.517560_656.1'} 2024-04-16T19:54:14.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297254.134542_124.1', 2: '1_1713297253.847516_845.1', 3: '1_1713297254.393091_652.1'} 2024-04-16T19:54:14.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1713297254.332712_126.1 target=1_1713297254.134542_124.1, shard=3 master=1_1713297254.517560_656.1 target=1_1713297254.393091_652.1 2024-04-16T19:54:19.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:19.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:19.837 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:19.892 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:19.892 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:19.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:19.942+0000 7fccfa213800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:19.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:19.944+0000 7fccfa213800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:19.968 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297254.332712_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.332712Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297253.847516_845.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:13.847516Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297254.517560_656.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.517560Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:19Z"} 2024-04-16T19:54:19.968 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297254.332712_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.332712Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297253.847516_845.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:13.847516Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297254.517560_656.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:14.517560Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:19Z"} 2024-04-16T19:54:19.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297254.332712_126.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:14.332712Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297253.847516_845.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:13.847516Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297254.517560_656.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:14.517560Z', 'realm_epoch': 2}}] 2024-04-16T19:54:19.968 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297254.332712_126.1', 2: '1_1713297253.847516_845.1', 3: '1_1713297254.517560_656.1'} 2024-04-16T19:54:19.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297254.332712_126.1', 2: '1_1713297253.847516_845.1', 3: '1_1713297254.517560_656.1'} 2024-04-16T19:54:19.968 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:19.969 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:19.969 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:19.969 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:19.969 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.970 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.970 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:19.970 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:19.970 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:19.971 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:19.971 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195419Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:19.971 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195419Z 20240416/test-zonegroup/iam/aws4_request 240c53b71e766646ae1d06bfd0eba8e2948f8b33d1c7d17dfb5d79dbcc10ae1e 2024-04-16T19:54:19.971 DEBUG:botocore.auth:Signature: c58e58833dd6a2d986d859af13a443ae72317091c7f5c2d586331a1f65f5dfa0 2024-04-16T19:54:19.971 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:19.971 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:19.975 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004626baea18d91431-00661ed76b-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '3633', 'Date': 'Tue, 16 Apr 2024 19:54:19 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:19.975 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW111111111111111116e7ac441-491c-4598-8785-7de38e426f36test-zone1/arn:aws:iam::RGW11111111111111111:role/test-zone12024-04-16T19:54:13.855Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111Allow{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}arn:aws:iam::aws:policy/AmazonS3FullAccess33439692-517f-4a4e-a7c3-14a0cf35bc03test-zone2/arn:aws:iam::RGW11111111111111111:role/test-zone22024-04-16T19:54:14.388Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111Allow{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}arn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx000004626baea18d91431-00661ed76b-4812-test-zone1' 2024-04-16T19:54:19.976 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:19.976 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:19.976 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.977 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:19.977 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:19.977 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:19.977 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.977 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.977 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:19.978 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:19.978 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:19.978 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:19.978 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195419Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:19.978 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195419Z 20240416/test-zonegroup/iam/aws4_request f8119944c5b60b28e8430182d7c1844c92725ba9a0e3d1a44d1f270de4ed0c10 2024-04-16T19:54:19.978 DEBUG:botocore.auth:Signature: 769fe594deabf1bd078cc53d9ea580521f81a6ab0d44b3d6207005329177ddb5 2024-04-16T19:54:19.979 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:19.979 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:19.983 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000fb37f870425335d7-00661ed76b-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '3633', 'Date': 'Tue, 16 Apr 2024 19:54:19 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:19.983 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW111111111111111116e7ac441-491c-4598-8785-7de38e426f36test-zone1/arn:aws:iam::RGW11111111111111111:role/test-zone12024-04-16T19:54:13.855Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111Allow{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}arn:aws:iam::aws:policy/AmazonS3FullAccess33439692-517f-4a4e-a7c3-14a0cf35bc03test-zone2/arn:aws:iam::RGW11111111111111111:role/test-zone22024-04-16T19:54:14.388Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111Allow{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]}arn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx00000fb37f870425335d7-00661ed76b-5331-test-zone2' 2024-04-16T19:54:19.984 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:19.984 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:19.984 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:19.985 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:19.985 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:19.985 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:19.985 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:19.985 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:19.985 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:19.985 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:19.986 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:19.986 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:19.986 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195419Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:19.986 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195419Z 20240416/test-zonegroup/iam/aws4_request 96b8fec3362e5d18a8b727e9b2130ed7e44901aa19365bd1881a07e8c7417819 2024-04-16T19:54:19.986 DEBUG:botocore.auth:Signature: fc463c8710220725f188d581fce85ce009de23d3e06e36f224fb43b4a1840d4a 2024-04-16T19:54:19.986 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:19.986 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:19.988 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000685e389762339597-00661ed76b-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:19 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:19.989 DEBUG:botocore.parsers:Response body: b'tx00000685e389762339597-00661ed76b-4812-test-zone1bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:19.989 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:19.989 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:19.989 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:19.989 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:19.989 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:19.990 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:19.990 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:19.990 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:19.990 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:19.990 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:19.990 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:19.991 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:19.991 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195419Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:19.991 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195419Z 20240416/test-zonegroup/iam/aws4_request 20b4ec4ac58a26e18e7e54e5b5c43437699268d9ec57523035232ea5881def7e 2024-04-16T19:54:19.991 DEBUG:botocore.auth:Signature: ba926ba74fd627fad0351a7f60440c110a8c60e83c9e0e9dca25eb7ddbf50f40 2024-04-16T19:54:19.991 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:19.991 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.034 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000010b842dc9f7a7c09-00661ed76b-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:19 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.034 DEBUG:botocore.parsers:Response body: b'tx0000010b842dc9f7a7c09-00661ed76b-5331-test-zone2bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.035 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.035 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.035 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.036 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.036 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.036 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.036 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.036 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.036 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.036 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.037 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.037 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.037 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:20.037 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 8ecf9ed37deec52671127a782a9d9e64c4ef32260968bfd87d083d2449ede3e9 2024-04-16T19:54:20.037 DEBUG:botocore.auth:Signature: fe2c719d3ae1695d7cb1423955c9a8b99ea72e1a4f962895878e1117e5c7e5f4 2024-04-16T19:54:20.037 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.038 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.040 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000002b8ed6282ee10f89-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.040 DEBUG:botocore.parsers:Response body: b'tx000002b8ed6282ee10f89-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.040 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.040 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.040 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.040 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.041 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.041 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.041 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.041 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.041 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.041 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.041 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.042 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.042 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:20.042 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 9d6728d219039d175e15b919d6290bb4d3d60f813c1efdc096bc25a38ea478ed 2024-04-16T19:54:20.042 DEBUG:botocore.auth:Signature: 0dc0a6d3f643de3e2b9dcd208d01b15e8c22c6b6e65d536cb49afd9445d9ea04 2024-04-16T19:54:20.042 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.042 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.045 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000097c05da34d88a64-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.045 DEBUG:botocore.parsers:Response body: b'tx00000097c05da34d88a64-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.045 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.045 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.045 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.045 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.046 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.046 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.046 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.046 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.046 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.046 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.046 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.047 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.047 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:20.047 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 233d64a29a44c27fde11de57d4fd8a2300b02cc83f9197a281fa4b13a7728e63 2024-04-16T19:54:20.047 DEBUG:botocore.auth:Signature: 46925c42c4f74fc3142ba352431c99c231b2f13a1f4f754787ab50bc15bc3d56 2024-04-16T19:54:20.047 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.047 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.050 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000009c6c05217ddc0c2c-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.050 DEBUG:botocore.parsers:Response body: b'tx000009c6c05217ddc0c2c-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.050 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.050 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.050 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.050 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.051 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.051 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.051 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.051 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.051 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.051 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.051 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.052 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.052 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:20.052 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 5000b4288570324a7e2b443aa1b4ec5f85617db8b0c30080affa0a9be39571d5 2024-04-16T19:54:20.052 DEBUG:botocore.auth:Signature: 2ec6988c5c9ed6593e1bd487beae39172dabc6f1a09c568184a83fdafe845dd3 2024-04-16T19:54:20.052 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.053 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.055 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000dc11c0ce32ced96a-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.055 DEBUG:botocore.parsers:Response body: b'tx00000dc11c0ce32ced96a-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.055 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.055 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.055 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.055 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.056 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.056 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.056 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.056 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.056 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.056 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.056 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.057 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.057 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:20.057 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request a9bf52d40d920c60810625d1d33aa94fb3c9cd1e477a64d84ab35a884aa91fc7 2024-04-16T19:54:20.057 DEBUG:botocore.auth:Signature: 2708648aba78a15a54211d87546e39f8fa8cee50a216cd33978b338d9523f964 2024-04-16T19:54:20.057 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.057 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.059 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000658687c5b05efff8-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.059 DEBUG:botocore.parsers:Response body: b'tx00000658687c5b05efff8-00661ed76c-4812-test-zone120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.060 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.060 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.060 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.060 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.060 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.060 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.061 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.061 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.061 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.061 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.061 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.061 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.061 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:20.062 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 9eeb5d003751575b20d5f184b8cb69a7dd053dad1999068efba99833ba0b40c5 2024-04-16T19:54:20.062 DEBUG:botocore.auth:Signature: d3e27e4316aefe5bfdda2c0972a0cf882609bf472bac68e2c6348b06af6c6989 2024-04-16T19:54:20.062 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.062 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.064 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000345479165a627ea-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.064 DEBUG:botocore.parsers:Response body: b'tx000000345479165a627ea-00661ed76c-5331-test-zone220b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.064 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.064 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.065 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.065 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.065 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.065 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.065 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.065 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.065 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.066 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.066 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.066 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.066 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:20.066 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request eb67b35b3b637935ed7b7f7993018119f72bb469be2c693a32c3167b3feb2315 2024-04-16T19:54:20.066 DEBUG:botocore.auth:Signature: 6fcf51062a90133c3e751c46584adecc829468bcb035fabba16c5ad23638ad2c 2024-04-16T19:54:20.066 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.067 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.110 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f3defe409a944c6d-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.111 DEBUG:botocore.parsers:Response body: b'tx00000f3defe409a944c6d-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.111 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.111 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.111 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.111 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.111 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.111 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.112 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.112 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.112 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.112 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.112 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.112 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.113 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:20.113 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 144b1e982649556b8bc9eada9426f8d106cc4f00ac486d098d74c1956127eaf2 2024-04-16T19:54:20.113 DEBUG:botocore.auth:Signature: c4fadc5c18e8d6e79e155dd2b93ad968714bd7a17b338581615e1b7b4b67e31c 2024-04-16T19:54:20.113 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.113 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.115 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000bec8291512e4b199-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.115 DEBUG:botocore.parsers:Response body: b'tx00000bec8291512e4b199-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.115 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.116 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.116 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.116 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.116 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.116 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.116 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.117 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.117 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.117 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.117 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.117 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.117 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:20.117 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request f2bee8360aac25a8976e074d4fddfc89a5a76cc79f5775c60550a8a6f34f69f0 2024-04-16T19:54:20.118 DEBUG:botocore.auth:Signature: 827d3d58aca855e27dc11cd719044a626bea5415dcff6aada112cd1ce66d185f 2024-04-16T19:54:20.118 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.118 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.120 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f0a499f24a5f6092-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.120 DEBUG:botocore.parsers:Response body: b'tx00000f0a499f24a5f6092-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.120 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.120 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.120 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.120 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.121 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.121 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.121 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.121 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.121 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.121 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.121 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.122 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.122 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:20.122 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request ce5944a3f25e9feba9081dc458e11668420bd989464d1007955852b0bf8b61f8 2024-04-16T19:54:20.122 DEBUG:botocore.auth:Signature: eb83a95eed77554c74052b0df14c0ffdd71c3963ff2dfd92ae2c204836fb41a0 2024-04-16T19:54:20.122 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.122 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.166 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f87d867c901e6682-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.167 DEBUG:botocore.parsers:Response body: b'tx00000f87d867c901e6682-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.167 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.167 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.167 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.167 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.167 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.167 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.168 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.168 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.168 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.168 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.168 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.168 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.168 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 7067bf4a2c2c30babee515f4ca93ddeff9e9141bcd70c45f18f5120d787f22f5 2024-04-16T19:54:20.169 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 2b365cf168d8613a9dfba44ba3ee4b6e44f232cc7381959214d5c5081489e64d 2024-04-16T19:54:20.169 DEBUG:botocore.auth:Signature: 3c5eee68553680d625f42340f94ef8d176ab3845d1d91089a404b09b68b1eca3 2024-04-16T19:54:20.169 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.169 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.171 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000005910148ff92d0393-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '765', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.171 DEBUG:botocore.parsers:Response body: b'tx000005910148ff92d0393-00661ed76c-4812-test-zone16e7ac441-491c-4598-8785-7de38e426f36test-zone1/arn:aws:iam::RGW11111111111111111:role/test-zone12024-04-16T19:54:13.855Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.172 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.172 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.172 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.172 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.172 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.172 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.172 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.173 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.173 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.173 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.173 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.173 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.173 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 7067bf4a2c2c30babee515f4ca93ddeff9e9141bcd70c45f18f5120d787f22f5 2024-04-16T19:54:20.173 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 0e246c09f34a68aedf653dfa784055cfae298b963819daf50673833343802217 2024-04-16T19:54:20.174 DEBUG:botocore.auth:Signature: 71c8e0ef040170b832e45b0806db8476e28f41450532873ab659887d46cd3a0e 2024-04-16T19:54:20.174 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.174 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.176 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e823161d0e78fba1-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '765', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.176 DEBUG:botocore.parsers:Response body: b'tx00000e823161d0e78fba1-00661ed76c-5331-test-zone26e7ac441-491c-4598-8785-7de38e426f36test-zone1/arn:aws:iam::RGW11111111111111111:role/test-zone12024-04-16T19:54:13.855Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.177 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.177 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.177 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.177 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.177 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.177 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.178 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.178 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.178 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.178 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.178 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.178 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.179 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 38569227d58e6809847b4e4238520661b8b38646b59cb058130b7fc6c1ed64be 2024-04-16T19:54:20.179 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request d35be2f47b2ee0fff5112eb1566bb9883b0875a7039be0fcd4a7716285619dcd 2024-04-16T19:54:20.179 DEBUG:botocore.auth:Signature: d90d3294fc42211799744b94b89aec95277a4c8fe0a72564652e99a82665d45c 2024-04-16T19:54:20.179 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.179 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.181 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b53447dca1e5624b-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '261', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.181 DEBUG:botocore.parsers:Response body: b'tx00000b53447dca1e5624b-00661ed76c-4812-test-zone1Allow' 2024-04-16T19:54:20.181 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.181 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.181 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.182 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.182 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.182 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.182 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.182 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.182 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.182 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.183 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.183 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.183 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 38569227d58e6809847b4e4238520661b8b38646b59cb058130b7fc6c1ed64be 2024-04-16T19:54:20.183 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 653342bbbe14a7e0619010c65880dd490ae10e0ad9ddae29fff73a820b249bbc 2024-04-16T19:54:20.183 DEBUG:botocore.auth:Signature: a2d6761b21bca0314fa7903a95f34785b8bfcf97f0e123bc21fb5cef24baa64b 2024-04-16T19:54:20.183 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.184 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.185 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000007c1c6193662476af-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '261', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.185 DEBUG:botocore.parsers:Response body: b'tx000007c1c6193662476af-00661ed76c-5331-test-zone2Allow' 2024-04-16T19:54:20.186 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.186 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.186 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.186 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.186 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.186 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.187 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.187 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.187 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.187 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.187 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.187 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.187 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 8b09555836446b433b1809feb29e77128d8e49e7527669bc33c462424a43b8b4 2024-04-16T19:54:20.188 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request b7fb140fe9ca58b4cf3c617a2ede3f8855c9bac5ba7481cd0dd6d50f86c4b45d 2024-04-16T19:54:20.188 DEBUG:botocore.auth:Signature: 1af3eb59f165a7ed2502e401944f4c925c8c28778ddf7a318b2adaad0ee1016d 2024-04-16T19:54:20.188 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.188 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.190 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000018982ec8fbeebbfa-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '456', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.190 DEBUG:botocore.parsers:Response body: b'tx0000018982ec8fbeebbfa-00661ed76c-4812-test-zone1AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccess' 2024-04-16T19:54:20.191 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.191 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.191 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.191 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.191 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.191 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.191 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.191 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.192 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.192 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.192 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.192 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.192 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 8b09555836446b433b1809feb29e77128d8e49e7527669bc33c462424a43b8b4 2024-04-16T19:54:20.192 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 421244447bcde87b038955c96db79e9cfb0245b301e1a7094e5ce84d7125fb99 2024-04-16T19:54:20.193 DEBUG:botocore.auth:Signature: 30bb3522c1cb6d51af01dc7d770ccf06a3822a5d658fc31e39889f42fde268b1 2024-04-16T19:54:20.193 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.193 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.194 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000bca891f44b6d1ce1-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '456', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.195 DEBUG:botocore.parsers:Response body: b'tx00000bca891f44b6d1ce1-00661ed76c-5331-test-zone2AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccess' 2024-04-16T19:54:20.195 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.195 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.195 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.195 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.195 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.195 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.196 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.196 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.196 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.196 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.196 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.196 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.197 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d1b3c02eab04a96ea4c1b62766790de704b6fcba4c966a27173500c60f5d6ff1 2024-04-16T19:54:20.197 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request e9bdb87f7571429c3a9a73ee7f720a2a1a21b16eaea7aa158cb60482f7b70d8d 2024-04-16T19:54:20.197 DEBUG:botocore.auth:Signature: 3d322c25c959ef5de8713383267bbd6b003349ca060fd437409206a0159004db 2024-04-16T19:54:20.197 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.197 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.199 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000267a6e3a45cc3587-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '765', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.199 DEBUG:botocore.parsers:Response body: b'tx00000267a6e3a45cc3587-00661ed76c-4812-test-zone133439692-517f-4a4e-a7c3-14a0cf35bc03test-zone2/arn:aws:iam::RGW11111111111111111:role/test-zone22024-04-16T19:54:14.388Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.200 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.200 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.200 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.200 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:20.200 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.200 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.200 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.201 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:20.201 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.201 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:20.201 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:20.201 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.201 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d1b3c02eab04a96ea4c1b62766790de704b6fcba4c966a27173500c60f5d6ff1 2024-04-16T19:54:20.201 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 12e0c92655866ccb1cbce713bae2fd4281ebfa4903d8a31bb64e9f5f8f02695e 2024-04-16T19:54:20.202 DEBUG:botocore.auth:Signature: ffacc32d92d41e795a9eeb3c5969255336894693d3dfe7f04ab1c451b31500f8 2024-04-16T19:54:20.202 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:20.202 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.204 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000998c05f8098a0aba-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '765', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.204 DEBUG:botocore.parsers:Response body: b'tx00000998c05f8098a0aba-00661ed76c-5331-test-zone233439692-517f-4a4e-a7c3-14a0cf35bc03test-zone2/arn:aws:iam::RGW11111111111111111:role/test-zone22024-04-16T19:54:14.388Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:20.204 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:20.205 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.205 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:20.205 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.205 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.205 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.205 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.205 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.206 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.206 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.206 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.206 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.206 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 60679f059cf1aaaffeb2ba196691292bb365c33a340fc79c0064b5b350fd568f 2024-04-16T19:54:20.206 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 8705aed61c265fef3e62b60979639ad286a2fdc8b5aff14e4f3464fe10b9e85b 2024-04-16T19:54:20.206 DEBUG:botocore.auth:Signature: 732451d8d6e8922ec466d8f8a0000a1a1327036b127d04592c211578fe7caf84 2024-04-16T19:54:20.207 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.207 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.254 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000947bd39c4916a496-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '261', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.254 DEBUG:botocore.parsers:Response body: b'tx00000947bd39c4916a496-00661ed76c-4812-test-zone1Allow' 2024-04-16T19:54:20.255 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.255 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.255 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.255 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.255 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.255 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.255 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.256 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.256 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.256 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:20.256 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.256 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.256 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 60679f059cf1aaaffeb2ba196691292bb365c33a340fc79c0064b5b350fd568f 2024-04-16T19:54:20.256 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request cff74091c86d5a922147fe19128baa3cd12fc3393f1da4f4ac03611a6801287c 2024-04-16T19:54:20.257 DEBUG:botocore.auth:Signature: a28b9e944c7837fa9f4ed12ad581281736c5a4a2c5d3a720c2c89d3d8cbe4a76 2024-04-16T19:54:20.257 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.257 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.259 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000ebdb79bdfcc12efc-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '261', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.259 DEBUG:botocore.parsers:Response body: b'tx00000ebdb79bdfcc12efc-00661ed76c-5331-test-zone2Allow' 2024-04-16T19:54:20.259 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.259 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.259 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:20.260 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.260 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.260 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.260 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.260 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.260 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.260 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.260 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.261 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.261 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date e0f61ac7a0d39bc58cdbe7966ed858e3c10a35a6e4ff05a95164010ed084bae5 2024-04-16T19:54:20.261 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 62024a923bfc23630cad84540c3d87c954fe489b21ff517eed082987b468fcae 2024-04-16T19:54:20.261 DEBUG:botocore.auth:Signature: af0a9fdb93a390a27143fb7c6e598ea4e2735d73e462d6c95847486470f6b712 2024-04-16T19:54:20.261 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.262 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.264 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000ef3dbf5540c5951a-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '456', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.264 DEBUG:botocore.parsers:Response body: b'tx00000ef3dbf5540c5951a-00661ed76c-4812-test-zone1AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccess' 2024-04-16T19:54:20.264 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.264 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.264 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.264 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.265 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.265 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.265 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.265 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.265 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.265 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:20.265 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.266 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.266 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date e0f61ac7a0d39bc58cdbe7966ed858e3c10a35a6e4ff05a95164010ed084bae5 2024-04-16T19:54:20.266 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 7e2d4dbe56d08eaae9266d050fb9209a80e31382f1649d041afac9c05e1112ae 2024-04-16T19:54:20.266 DEBUG:botocore.auth:Signature: 60d38c4877f8fb2eb2776cc4dc74652df096ae5f4fdbb6f256ebd1d1148e4935 2024-04-16T19:54:20.266 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.266 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.310 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000df922bae6c9e61c1-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '456', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.310 DEBUG:botocore.parsers:Response body: b'tx00000df922bae6c9e61c1-00661ed76c-5331-test-zone2AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccess' 2024-04-16T19:54:20.311 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.311 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.311 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:20.312 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUsers: calling handler 2024-04-16T19:54:20.312 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.312 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.313 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.313 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.313 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUsers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUsers', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.313 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler > 2024-04-16T19:54:20.313 DEBUG:botocore.hooks:Event choose-signer.iam.ListUsers: calling handler 2024-04-16T19:54:20.314 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.314 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2 2024-04-16T19:54:20.314 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 416fc0151f7cbf426834fe05379c72fe011337fba1523420007710f4b5dd2e56 2024-04-16T19:54:20.314 DEBUG:botocore.auth:Signature: 89c9372d90dda6d6e957ee7657848c441e1874ed20810adfbafbf018c92a3323 2024-04-16T19:54:20.314 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler 2024-04-16T19:54:20.315 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.317 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx000008a55dc1fec785ab0-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.317 DEBUG:botocore.parsers:Response body: b'/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone12024-04-16T19:54:13.669293Z/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone22024-04-16T19:54:13.936127Zfalse' 2024-04-16T19:54:20.318 DEBUG:botocore.hooks:Event needs-retry.iam.ListUsers: calling handler 2024-04-16T19:54:20.318 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.319 DEBUG:botocore.hooks:Event after-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.319 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUsers: calling handler 2024-04-16T19:54:20.319 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.319 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.319 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.320 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.320 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUsers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUsers', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.320 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler > 2024-04-16T19:54:20.320 DEBUG:botocore.hooks:Event choose-signer.iam.ListUsers: calling handler 2024-04-16T19:54:20.320 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.321 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2 2024-04-16T19:54:20.321 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request fc2743e1d0ae62f6c7a5db943eb33ca551ba2b9ba4b6bee649cb48d9bad3afab 2024-04-16T19:54:20.321 DEBUG:botocore.auth:Signature: e6bc4f727bd30bce4c28af82a9bcb5a9bfefad6b447f79dbed160a727d523aad 2024-04-16T19:54:20.321 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler 2024-04-16T19:54:20.321 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.324 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000189dbfe7d6cd8bf6-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.324 DEBUG:botocore.parsers:Response body: b'/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone12024-04-16T19:54:13.669293Z/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone22024-04-16T19:54:13.936127Zfalse' 2024-04-16T19:54:20.325 DEBUG:botocore.hooks:Event needs-retry.iam.ListUsers: calling handler 2024-04-16T19:54:20.325 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.325 DEBUG:botocore.hooks:Event after-call.iam.ListUsers: calling handler 2024-04-16T19:54:20.326 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetUser: calling handler 2024-04-16T19:54:20.326 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.326 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.326 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.326 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.326 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.327 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler > 2024-04-16T19:54:20.327 DEBUG:botocore.hooks:Event choose-signer.iam.GetUser: calling handler 2024-04-16T19:54:20.327 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.327 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 36be43c886972026dc9392cd2c57a48595fcb211bd6cc645e01ae503a91b5b02 2024-04-16T19:54:20.327 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request e79da62c4c9151b1b7dc75d5f3a27b34ab7b0ad6ba0ea5e5d9cb2fd30b3e8263 2024-04-16T19:54:20.328 DEBUG:botocore.auth:Signature: 59da02cc2ff01786598eb662cba3243dcbff3733000c7abfacffdd015b4a4f8b 2024-04-16T19:54:20.328 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler 2024-04-16T19:54:20.328 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.331 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000004cd33d0e4d4f257-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '487', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.331 DEBUG:botocore.parsers:Response body: b'/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone12024-04-16T19:54:13.669293Ztx0000004cd33d0e4d4f257-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.331 DEBUG:botocore.hooks:Event needs-retry.iam.GetUser: calling handler 2024-04-16T19:54:20.332 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.332 DEBUG:botocore.hooks:Event after-call.iam.GetUser: calling handler 2024-04-16T19:54:20.332 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetUser: calling handler 2024-04-16T19:54:20.332 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.332 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.333 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.333 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.333 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.333 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler > 2024-04-16T19:54:20.333 DEBUG:botocore.hooks:Event choose-signer.iam.GetUser: calling handler 2024-04-16T19:54:20.334 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.334 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 36be43c886972026dc9392cd2c57a48595fcb211bd6cc645e01ae503a91b5b02 2024-04-16T19:54:20.334 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 59db4784c7498d54a27a55c28dad8fd41dcd3d1c5e90aebcd60d49e403baedf7 2024-04-16T19:54:20.334 DEBUG:botocore.auth:Signature: 97c1859252e6b26c19eb5aa60cf7206480ea1495590b033b0cf97b9ada89b579 2024-04-16T19:54:20.334 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler 2024-04-16T19:54:20.334 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.337 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000002282cef869b96261-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '487', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.337 DEBUG:botocore.parsers:Response body: b'/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone12024-04-16T19:54:13.669293Ztx000002282cef869b96261-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.338 DEBUG:botocore.hooks:Event needs-retry.iam.GetUser: calling handler 2024-04-16T19:54:20.338 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.338 DEBUG:botocore.hooks:Event after-call.iam.GetUser: calling handler 2024-04-16T19:54:20.339 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.339 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.339 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.339 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.339 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.339 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.340 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:20.340 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.340 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.340 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 919fd997a1071a9eb8adeb0d8e2a852ea3984fffd61b7a1dc133995556ffb778 2024-04-16T19:54:20.340 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 035fd1e4027529a44339174d81e1bc677efe549549e9126cefa2dacb790e8e7a 2024-04-16T19:54:20.341 DEBUG:botocore.auth:Signature: 8c6788d37d60fce18e45a1b50a587ac7c99d12211398e7fd4b0e2b1e9c930908 2024-04-16T19:54:20.341 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.341 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.343 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000d46f0d423be6471d-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.344 DEBUG:botocore.parsers:Response body: b'test-zone1test-zone1IBVMW6I15HR0T84KDGA7Active2024-04-16T19:54:13.683670Zfalse' 2024-04-16T19:54:20.344 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.344 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.345 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.345 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.345 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.345 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.345 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.345 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.346 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.346 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:20.346 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.346 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.346 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 919fd997a1071a9eb8adeb0d8e2a852ea3984fffd61b7a1dc133995556ffb778 2024-04-16T19:54:20.347 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 61a99b0722b2758bde16eb21c6365a099f805e4139da303af9edc02a305cea9a 2024-04-16T19:54:20.347 DEBUG:botocore.auth:Signature: ec45151203b5e020a8fdb0ca9718d06c372cfd69538fb2e0805bea2e44ba7021 2024-04-16T19:54:20.347 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.347 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.349 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx0000051e9011fe8ef0ddd-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.350 DEBUG:botocore.parsers:Response body: b'test-zone1test-zone1IBVMW6I15HR0T84KDGA7Active2024-04-16T19:54:13.683670Zfalse' 2024-04-16T19:54:20.350 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.350 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.351 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.351 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.351 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.351 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.352 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.352 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.352 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupsForUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupsForUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.352 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler > 2024-04-16T19:54:20.352 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.353 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.353 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d4742f49a0da68c5bb72a1260c8ca465b619be78c1f1b6a677d6fd1e87406c72 2024-04-16T19:54:20.353 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request dc41f9feab9c09553b0f5f40e9e00b53672957c9c1d70f51a51bdf8972bd29f6 2024-04-16T19:54:20.353 DEBUG:botocore.auth:Signature: 08b89340a81b7cd7145e11cbe5fc9d181b0e751fb39ef5163c748ee5ca538917 2024-04-16T19:54:20.353 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.354 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.398 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000360327bc3a3facb0-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '533', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.398 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1falsetx00000360327bc3a3facb0-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.399 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.399 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.400 DEBUG:botocore.hooks:Event after-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.400 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.400 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.400 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.401 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.401 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.401 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupsForUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupsForUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.401 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler > 2024-04-16T19:54:20.401 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.402 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.402 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d4742f49a0da68c5bb72a1260c8ca465b619be78c1f1b6a677d6fd1e87406c72 2024-04-16T19:54:20.402 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 134465cc05bc52012d9e7f8ec6a41317ee9456dff0edecf016775f6f53580d7c 2024-04-16T19:54:20.402 DEBUG:botocore.auth:Signature: 39e7a96209f2eb7c92553e5862d46aaa0f7346609e2e51dc051f9d90ab33c675 2024-04-16T19:54:20.402 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.402 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.405 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000d3e3fd1c6a702992-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '533', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.406 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1falsetx00000d3e3fd1c6a702992-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.406 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.406 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.406 DEBUG:botocore.hooks:Event after-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.407 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.407 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.407 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.407 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.407 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.407 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.408 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler > 2024-04-16T19:54:20.408 DEBUG:botocore.hooks:Event choose-signer.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.408 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.408 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4d3f921f2ff07e303538de70576b9876f48b3a5ab3f9a483a2b0b18fb63ec13d 2024-04-16T19:54:20.408 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 30e14dc5aba55321f4e7b62e672d7223e7f705138c5dcf6cdaa43e216aa99d87 2024-04-16T19:54:20.409 DEBUG:botocore.auth:Signature: f6208493ab9bdfa2b0b7b11d149522ff09933e48ce489370ebf5327fba858b25 2024-04-16T19:54:20.409 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.409 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.412 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f2c8b27b615c955e-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '343', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.412 DEBUG:botocore.parsers:Response body: b'tx00000f2c8b27b615c955e-00661ed76c-4812-test-zone1Allowfalse' 2024-04-16T19:54:20.412 DEBUG:botocore.hooks:Event needs-retry.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.412 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.413 DEBUG:botocore.hooks:Event after-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.413 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.413 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.413 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.413 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.414 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.414 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.414 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler > 2024-04-16T19:54:20.414 DEBUG:botocore.hooks:Event choose-signer.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.415 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.415 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4d3f921f2ff07e303538de70576b9876f48b3a5ab3f9a483a2b0b18fb63ec13d 2024-04-16T19:54:20.415 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 3595d5cfbfcc36ba718fe847a2425333d8e7cd08b4e2372ef3e3051e7e1282b2 2024-04-16T19:54:20.415 DEBUG:botocore.auth:Signature: 891ef1b416a0debbc8f3e7d554830a10c11ed9883a4685445dd8fe66689ec3a7 2024-04-16T19:54:20.415 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.415 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.418 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000003fa5949420e6fd4-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '343', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.418 DEBUG:botocore.parsers:Response body: b'tx0000003fa5949420e6fd4-00661ed76c-5331-test-zone2Allowfalse' 2024-04-16T19:54:20.418 DEBUG:botocore.hooks:Event needs-retry.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.419 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.419 DEBUG:botocore.hooks:Event after-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.419 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.419 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.419 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.420 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.420 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.420 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.420 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler > 2024-04-16T19:54:20.420 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.421 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.421 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 49799e0d599ed5d7467f14f6a5435b14aad4e9b68dd234a6844ecab21760f18f 2024-04-16T19:54:20.421 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request cbf1fddf77c6326137ab9071ac932681c7e086665629ba44832aacbf8f8607eb 2024-04-16T19:54:20.421 DEBUG:botocore.auth:Signature: 39089d47d03dbb4d5dbe8931ac20018fd1566adbf3d6eb407e316b78ac77b695 2024-04-16T19:54:20.421 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.421 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.424 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000005bdcbed083753e97-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '488', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.425 DEBUG:botocore.parsers:Response body: b'tx000005bdcbed083753e97-00661ed76c-4812-test-zone1AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalse' 2024-04-16T19:54:20.425 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.425 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.425 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.426 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.426 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.426 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.426 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.426 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.426 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.427 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler > 2024-04-16T19:54:20.427 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.427 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.427 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 49799e0d599ed5d7467f14f6a5435b14aad4e9b68dd234a6844ecab21760f18f 2024-04-16T19:54:20.427 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 39ddd5bb77270aa81868c1ed51ef1b2ba9b4515f3a2932359d3416d4e30e159f 2024-04-16T19:54:20.428 DEBUG:botocore.auth:Signature: c62cfa6d8c3f0b6b56e776f01bbb8226a3f1fa16f6895ddd40fedcae7ed96d40 2024-04-16T19:54:20.428 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.428 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.430 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000063fa9dada5230b72-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '488', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.431 DEBUG:botocore.parsers:Response body: b'tx0000063fa9dada5230b72-00661ed76c-5331-test-zone2AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalse' 2024-04-16T19:54:20.431 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.431 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.431 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.432 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetUser: calling handler 2024-04-16T19:54:20.432 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.432 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.432 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.432 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.433 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.433 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler > 2024-04-16T19:54:20.433 DEBUG:botocore.hooks:Event choose-signer.iam.GetUser: calling handler 2024-04-16T19:54:20.433 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.433 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 8d30a3f507fe42ace157bf649ae227bc4005724392fed2f2ba1e19ac20f57f65 2024-04-16T19:54:20.434 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 165b061f1b2f372b20c445b149699fc2a5d93bc9616f15bfd0c76ce2de6c0864 2024-04-16T19:54:20.434 DEBUG:botocore.auth:Signature: 682f850d78315ebe6feeccefc2e86151aaa091e18dff01c2dceb9970a8ddbf17 2024-04-16T19:54:20.434 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler 2024-04-16T19:54:20.434 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.436 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000729a0c6f74e66486-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '487', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.437 DEBUG:botocore.parsers:Response body: b'/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone22024-04-16T19:54:13.936127Ztx00000729a0c6f74e66486-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.437 DEBUG:botocore.hooks:Event needs-retry.iam.GetUser: calling handler 2024-04-16T19:54:20.437 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.437 DEBUG:botocore.hooks:Event after-call.iam.GetUser: calling handler 2024-04-16T19:54:20.438 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetUser: calling handler 2024-04-16T19:54:20.438 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.438 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.438 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.438 DEBUG:botocore.hooks:Event before-call.iam.GetUser: calling handler 2024-04-16T19:54:20.438 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.438 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler > 2024-04-16T19:54:20.439 DEBUG:botocore.hooks:Event choose-signer.iam.GetUser: calling handler 2024-04-16T19:54:20.439 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.439 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 8d30a3f507fe42ace157bf649ae227bc4005724392fed2f2ba1e19ac20f57f65 2024-04-16T19:54:20.439 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 540dd38436e2dc401f87b290cc710f3ad94c832f370693e02d07e3b22b019adb 2024-04-16T19:54:20.439 DEBUG:botocore.auth:Signature: 0f5b66759ee496d3b3cc53e7f4b2065ba19fefd92dd63a0bd67025089e44d68c 2024-04-16T19:54:20.439 DEBUG:botocore.hooks:Event request-created.iam.GetUser: calling handler 2024-04-16T19:54:20.440 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.442 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000cb6e030f99613222-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '487', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.442 DEBUG:botocore.parsers:Response body: b'/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone22024-04-16T19:54:13.936127Ztx00000cb6e030f99613222-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.443 DEBUG:botocore.hooks:Event needs-retry.iam.GetUser: calling handler 2024-04-16T19:54:20.443 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.443 DEBUG:botocore.hooks:Event after-call.iam.GetUser: calling handler 2024-04-16T19:54:20.443 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.443 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.443 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.444 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.444 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.444 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.444 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:20.444 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.444 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.445 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 575b893e712663944c456f61f1aabd27df830cbe7212c952bf0cd211f66fc279 2024-04-16T19:54:20.445 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request f7eb1d379b748394026581c6b9d9676a6f5c33ee001eab6c4c823f6250d3c720 2024-04-16T19:54:20.445 DEBUG:botocore.auth:Signature: bd7aab285c67640fc72725a74586143699c1cf00081765f7993e9d541fe65bf8 2024-04-16T19:54:20.445 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.445 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.448 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx000006096d46f1a4fe067-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.448 DEBUG:botocore.parsers:Response body: b'test-zone2test-zone231HO20TWHBMRQFU8RLUEActive2024-04-16T19:54:13.956210Zfalse' 2024-04-16T19:54:20.449 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.449 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.449 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.449 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.449 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.449 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.449 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.449 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.450 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.450 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:20.450 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.450 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.450 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 575b893e712663944c456f61f1aabd27df830cbe7212c952bf0cd211f66fc279 2024-04-16T19:54:20.451 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 57d7859dfce4e8b28102770fc508abbd717424cb1d2cfcb8b7b852dd592ec203 2024-04-16T19:54:20.451 DEBUG:botocore.auth:Signature: 097deb51c9daa53a640f2d03c9a55081c55ad49f670be4e88fea7e2d72529687 2024-04-16T19:54:20.451 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.451 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.498 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000a36b266911a38a2a-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.499 DEBUG:botocore.parsers:Response body: b'test-zone2test-zone231HO20TWHBMRQFU8RLUEActive2024-04-16T19:54:13.956210Zfalse' 2024-04-16T19:54:20.499 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.499 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.499 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.500 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.500 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.500 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.500 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.500 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.500 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupsForUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupsForUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.501 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler > 2024-04-16T19:54:20.501 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.501 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.501 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date ecf8aee7706b736153a5a7d48afdc1ac23c4def3724f6ced3ccd7e69933ba573 2024-04-16T19:54:20.501 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 5fd7e1284f584d25e0b842e77b2865f70553dacd51117fb6015ef9a83331754a 2024-04-16T19:54:20.501 DEBUG:botocore.auth:Signature: 7010093712200a1c72bd2cf4817767e82d321a23f742b5418d4b7b4dfdcb062c 2024-04-16T19:54:20.502 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.502 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.504 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006aa0c68b98147c70-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '533', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.505 DEBUG:botocore.parsers:Response body: b'/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2falsetx000006aa0c68b98147c70-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.505 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.505 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.505 DEBUG:botocore.hooks:Event after-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.505 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.505 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.505 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.506 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.506 DEBUG:botocore.hooks:Event before-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.506 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupsForUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupsForUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.506 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler > 2024-04-16T19:54:20.506 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.507 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.507 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date ecf8aee7706b736153a5a7d48afdc1ac23c4def3724f6ced3ccd7e69933ba573 2024-04-16T19:54:20.507 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 0bfae4bd717b1d3e4e111ff5b09f5453a375b03086f51ad4ce2765c2c9564b03 2024-04-16T19:54:20.507 DEBUG:botocore.auth:Signature: 4ce7168041537e93eb1c4872373b9a7133cf642a8c64710aff725f8cbd071dea 2024-04-16T19:54:20.507 DEBUG:botocore.hooks:Event request-created.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.507 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.510 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000cf55aa87450a06d4-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '533', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.510 DEBUG:botocore.parsers:Response body: b'/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2falsetx00000cf55aa87450a06d4-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.510 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.510 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.511 DEBUG:botocore.hooks:Event after-call.iam.ListGroupsForUser: calling handler 2024-04-16T19:54:20.511 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.511 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.511 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.511 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.511 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.512 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.512 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler > 2024-04-16T19:54:20.512 DEBUG:botocore.hooks:Event choose-signer.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.512 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.512 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 683bb38491391414b27b590de03643d3067ab3a39c7a8909fc0ae8ca88576c7c 2024-04-16T19:54:20.512 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 3197534f1eac8234d9678d4bd4155afdc7c4eb4ed55c13d9954a38a82b85bc41 2024-04-16T19:54:20.512 DEBUG:botocore.auth:Signature: e86a15e00a3de56474f88e51e724343c79934eaf30f96939ea81c1828d3a6062 2024-04-16T19:54:20.513 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.513 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.515 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000ca24463c7e70a2d2-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '343', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.515 DEBUG:botocore.parsers:Response body: b'tx00000ca24463c7e70a2d2-00661ed76c-4812-test-zone1Allowfalse' 2024-04-16T19:54:20.516 DEBUG:botocore.hooks:Event needs-retry.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.516 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.516 DEBUG:botocore.hooks:Event after-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.516 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.516 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.516 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.516 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.517 DEBUG:botocore.hooks:Event before-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.517 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.517 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler > 2024-04-16T19:54:20.517 DEBUG:botocore.hooks:Event choose-signer.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.517 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.517 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 683bb38491391414b27b590de03643d3067ab3a39c7a8909fc0ae8ca88576c7c 2024-04-16T19:54:20.517 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 8c5af43c3df9e91facfe6a4817236179cfd30c02e58fe31a30cd10070f5431df 2024-04-16T19:54:20.518 DEBUG:botocore.auth:Signature: 27a075a973ff93609247fa324ebe2939636f9422f970e19f8330bfc02e032149 2024-04-16T19:54:20.518 DEBUG:botocore.hooks:Event request-created.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.518 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.521 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b186618994c45187-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '343', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.521 DEBUG:botocore.parsers:Response body: b'tx00000b186618994c45187-00661ed76c-5331-test-zone2Allowfalse' 2024-04-16T19:54:20.521 DEBUG:botocore.hooks:Event needs-retry.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.521 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.521 DEBUG:botocore.hooks:Event after-call.iam.ListUserPolicies: calling handler 2024-04-16T19:54:20.522 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.522 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.522 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.522 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.522 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.522 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.523 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler > 2024-04-16T19:54:20.523 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.523 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.523 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 3dace716e8edf34ab8446c6b1da0096d88964c8d134639d581b86139d8c0a818 2024-04-16T19:54:20.523 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 7a846cb39412163d4ce842f03598ff66950c0e53bd65f9288d9bcfa104518cfd 2024-04-16T19:54:20.523 DEBUG:botocore.auth:Signature: 1c9e1d336e2bd88f6c036a918ba85d85e0b679804cc172c90b423b086bed3515 2024-04-16T19:54:20.523 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.524 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.566 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000d61f12a14b87734e-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '488', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.566 DEBUG:botocore.parsers:Response body: b'tx00000d61f12a14b87734e-00661ed76c-4812-test-zone1AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalse' 2024-04-16T19:54:20.567 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.567 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.567 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.567 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.567 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.567 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.567 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.568 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.568 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedUserPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedUserPolicies', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.568 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler > 2024-04-16T19:54:20.568 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.568 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.568 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 3dace716e8edf34ab8446c6b1da0096d88964c8d134639d581b86139d8c0a818 2024-04-16T19:54:20.568 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 346d067f9b6cbad82522f2d66ecb3a61e957cd60c8259bb57bda1ec2adf60a10 2024-04-16T19:54:20.569 DEBUG:botocore.auth:Signature: 61b8ef499567199afd36e2cc732f992c5081407710ef997b62faa6d58f482f86 2024-04-16T19:54:20.569 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.569 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.572 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000254e6f90ebc41c7a-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '488', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.572 DEBUG:botocore.parsers:Response body: b'tx00000254e6f90ebc41c7a-00661ed76c-5331-test-zone2AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalse' 2024-04-16T19:54:20.572 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.572 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.572 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedUserPolicies: calling handler 2024-04-16T19:54:20.573 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroups: calling handler 2024-04-16T19:54:20.573 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.573 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.573 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.573 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.574 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroups', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.574 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler > 2024-04-16T19:54:20.574 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroups: calling handler 2024-04-16T19:54:20.574 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.574 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 5f776d91509b9c99b8cb5eb5d6d4a787a33ae41c8cd6e7b69effca69080e1e1f 2024-04-16T19:54:20.575 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 85fb1473a32e1823133a9f2769d6fc4ef95943a3787f3fd845a9d1fd78b38d35 2024-04-16T19:54:20.575 DEBUG:botocore.auth:Signature: ed9b14df656416f853d77d02b943b94220bb45e1bbf3ef3faa8294e7551f4940 2024-04-16T19:54:20.575 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler 2024-04-16T19:54:20.575 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.578 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000165be3587be659aa-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.578 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2false' 2024-04-16T19:54:20.578 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroups: calling handler 2024-04-16T19:54:20.578 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.579 DEBUG:botocore.hooks:Event after-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.579 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroups: calling handler 2024-04-16T19:54:20.579 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.579 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.580 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.580 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.580 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroups', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.580 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler > 2024-04-16T19:54:20.580 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroups: calling handler 2024-04-16T19:54:20.581 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.581 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 5f776d91509b9c99b8cb5eb5d6d4a787a33ae41c8cd6e7b69effca69080e1e1f 2024-04-16T19:54:20.581 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 5b759e69e2afe789c74a6d8b8363cb3d7f1d5b8574867d86630f34df0fb0bc94 2024-04-16T19:54:20.581 DEBUG:botocore.auth:Signature: f9208e021abc350cd527b8cc2ff7bb6fdc21ce8f2bb733142cfc361591520622 2024-04-16T19:54:20.581 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler 2024-04-16T19:54:20.581 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.584 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000c5a096eacda8fa7c-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.584 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2false' 2024-04-16T19:54:20.584 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroups: calling handler 2024-04-16T19:54:20.585 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.585 DEBUG:botocore.hooks:Event after-call.iam.ListGroups: calling handler 2024-04-16T19:54:20.585 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetGroup: calling handler 2024-04-16T19:54:20.585 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.585 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.586 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.586 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.586 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.586 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler > 2024-04-16T19:54:20.586 DEBUG:botocore.hooks:Event choose-signer.iam.GetGroup: calling handler 2024-04-16T19:54:20.587 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.587 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 96c30390354a0b67fe3fad2562a6dea8125351a0e25f82d61db2010f5b8b25a3 2024-04-16T19:54:20.587 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 75c1787f3847e1d73566b5bb0e540106dc47c3b8e531a6cbfcd7de6b06cfbfec 2024-04-16T19:54:20.587 DEBUG:botocore.auth:Signature: 48e24b7c3bb269f45bf3295227942356418fcd14f96260b38975413e7a1fb7c8 2024-04-16T19:54:20.587 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler 2024-04-16T19:54:20.587 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.590 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000005de882ffa04044ec-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '668', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.590 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone1falsetx000005de882ffa04044ec-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.591 DEBUG:botocore.hooks:Event needs-retry.iam.GetGroup: calling handler 2024-04-16T19:54:20.591 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.591 DEBUG:botocore.hooks:Event after-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.591 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetGroup: calling handler 2024-04-16T19:54:20.591 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.591 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.592 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.592 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.592 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.592 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler > 2024-04-16T19:54:20.593 DEBUG:botocore.hooks:Event choose-signer.iam.GetGroup: calling handler 2024-04-16T19:54:20.593 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.593 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 96c30390354a0b67fe3fad2562a6dea8125351a0e25f82d61db2010f5b8b25a3 2024-04-16T19:54:20.593 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request bba4505ec6c27c8c7e898c90c3e2d95a88b74b5ca8ada8256142f3fd719f8967 2024-04-16T19:54:20.593 DEBUG:botocore.auth:Signature: a2823d4eb9be3433849fd0a5b207dd9a66264e45bf927ae8d9ba290883eb8b26 2024-04-16T19:54:20.593 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler 2024-04-16T19:54:20.593 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.596 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000082e5f1d7013b3ef5-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '668', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.596 DEBUG:botocore.parsers:Response body: b'/test-zone1fd88b68e-3981-4975-9c49-58cf4a9bbee4arn:aws:iam::RGW11111111111111111:group/test-zone1/test-zone17c156b14-3284-41bf-8a51-5dfd45307f6barn:aws:iam::RGW11111111111111111:user/test-zone1falsetx0000082e5f1d7013b3ef5-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.597 DEBUG:botocore.hooks:Event needs-retry.iam.GetGroup: calling handler 2024-04-16T19:54:20.597 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.597 DEBUG:botocore.hooks:Event after-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.597 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.598 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.598 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.598 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.598 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.598 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.599 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler > 2024-04-16T19:54:20.599 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.599 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.599 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 7811b9d5c4ca8b1e9d0155d32c2077f72a8b205c19c3c2277df8f5e6e074f9aa 2024-04-16T19:54:20.599 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request d9430e82a0821825ef289b2421dd8693d4242348dcc8ea9d51eb89c343615882 2024-04-16T19:54:20.599 DEBUG:botocore.auth:Signature: e11fd3682ce6186396beac398c3f03e9a2ec2e0c3a33f677253a1f1acc91d1a6 2024-04-16T19:54:20.600 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.600 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.602 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000008ab28debf63f2407-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '385', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.602 DEBUG:botocore.parsers:Response body: b'Allowfalsetx000008ab28debf63f2407-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.602 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.602 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.603 DEBUG:botocore.hooks:Event after-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.603 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.603 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.603 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.603 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.604 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.604 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.604 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler > 2024-04-16T19:54:20.604 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.604 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.605 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 7811b9d5c4ca8b1e9d0155d32c2077f72a8b205c19c3c2277df8f5e6e074f9aa 2024-04-16T19:54:20.605 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 2c8e9d1e0e24180832d4a9f28c8b6d81cd0e9052255ce06315f8eee9149bd405 2024-04-16T19:54:20.605 DEBUG:botocore.auth:Signature: efc34d2a12be4de1d14ab188e4bff980c730d1c37202e764f7370886d8679e4b 2024-04-16T19:54:20.605 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.605 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.607 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000f392ac34768870a7-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '385', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.607 DEBUG:botocore.parsers:Response body: b'Allowfalsetx00000f392ac34768870a7-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.608 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.608 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.608 DEBUG:botocore.hooks:Event after-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.608 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.608 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.609 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.609 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.609 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.609 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.609 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler > 2024-04-16T19:54:20.610 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.610 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.610 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d3cad5863c5641759e4bcb9ff168c45d0bfbea13d35654ca45486df7605c4b98 2024-04-16T19:54:20.610 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 2b2180972a02ff91a17b733ebc65de8fdd2bab5861cdf4d9bbd0502d4f63717d 2024-04-16T19:54:20.610 DEBUG:botocore.auth:Signature: 3612dbd4dd8967345a9b6eb88057ecf241cb1fb90d5aa30becce410b2ab1b085 2024-04-16T19:54:20.611 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.611 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.613 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000884405943286b018-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '530', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.613 DEBUG:botocore.parsers:Response body: b'AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx00000884405943286b018-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.613 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.613 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.613 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.614 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.614 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.614 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.614 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.614 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.615 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.615 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler > 2024-04-16T19:54:20.615 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.615 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.615 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date d3cad5863c5641759e4bcb9ff168c45d0bfbea13d35654ca45486df7605c4b98 2024-04-16T19:54:20.616 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request d0af15db9847657a87d85653715a95861c1b2a67f1d76eb4ef74e4d7a03c1ad1 2024-04-16T19:54:20.616 DEBUG:botocore.auth:Signature: 5ed0afe1f344b61000830b64aac296781ddfa3149cfd4e40eee549cabcd56167 2024-04-16T19:54:20.616 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.616 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.618 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e623ce1358d108b0-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '530', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.618 DEBUG:botocore.parsers:Response body: b'AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx00000e623ce1358d108b0-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.619 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.619 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.619 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.619 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetGroup: calling handler 2024-04-16T19:54:20.619 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.619 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.620 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.620 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.620 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.620 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler > 2024-04-16T19:54:20.620 DEBUG:botocore.hooks:Event choose-signer.iam.GetGroup: calling handler 2024-04-16T19:54:20.620 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.620 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 0de1abeeecebec53ce7c53e52f1ad78c5754c2ec5c9b2c6a3da55177a99211d3 2024-04-16T19:54:20.621 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request e0c1682d3fd720fb1bb1f1ce75c568c64c1647e24a183a6b3c1acab2d2a1dfcb 2024-04-16T19:54:20.621 DEBUG:botocore.auth:Signature: 3438b8d36e2d92a2adc3989ee17021857548c2b1858ad2c911a396713adef1a6 2024-04-16T19:54:20.621 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler 2024-04-16T19:54:20.621 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.623 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000d27a3631e112a8d6-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '668', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.624 DEBUG:botocore.parsers:Response body: b'/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone2falsetx00000d27a3631e112a8d6-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.624 DEBUG:botocore.hooks:Event needs-retry.iam.GetGroup: calling handler 2024-04-16T19:54:20.624 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.624 DEBUG:botocore.hooks:Event after-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.624 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetGroup: calling handler 2024-04-16T19:54:20.624 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.625 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.625 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.625 DEBUG:botocore.hooks:Event before-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.625 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.625 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler > 2024-04-16T19:54:20.625 DEBUG:botocore.hooks:Event choose-signer.iam.GetGroup: calling handler 2024-04-16T19:54:20.626 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.626 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 0de1abeeecebec53ce7c53e52f1ad78c5754c2ec5c9b2c6a3da55177a99211d3 2024-04-16T19:54:20.626 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 16ca5218c8ca5425fb55461d74a51ffd20f87cab5a04342656f717244fdc999d 2024-04-16T19:54:20.626 DEBUG:botocore.auth:Signature: 15ae10ab2f6156a041f35e7123458c16d7215829ef78783fe7dc7c10a7cfd29b 2024-04-16T19:54:20.626 DEBUG:botocore.hooks:Event request-created.iam.GetGroup: calling handler 2024-04-16T19:54:20.626 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.629 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000cf0ea6ae598aded1-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '668', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.629 DEBUG:botocore.parsers:Response body: b'/test-zone2d13639ea-647a-4310-a26c-43ac90d6a01barn:aws:iam::RGW11111111111111111:group/test-zone2/test-zone2a2924d60-2a39-4727-a2cd-973baae602b3arn:aws:iam::RGW11111111111111111:user/test-zone2falsetx00000cf0ea6ae598aded1-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.629 DEBUG:botocore.hooks:Event needs-retry.iam.GetGroup: calling handler 2024-04-16T19:54:20.629 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.629 DEBUG:botocore.hooks:Event after-call.iam.GetGroup: calling handler 2024-04-16T19:54:20.630 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.630 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.630 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.630 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.630 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.630 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.631 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler > 2024-04-16T19:54:20.631 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.631 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.631 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date b8c9ebd42f207eccd0f1ac4735e5608d9b5aca42e16ba14ba414850016f19bb6 2024-04-16T19:54:20.631 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request a4ed6351c6eef480af3458f9f9a0da54d0fae006457801d9b467fb4a2e06d7b2 2024-04-16T19:54:20.631 DEBUG:botocore.auth:Signature: 36e286c4f5727cae4e1e0521b5d64d20fc9f1dbd0d90a9e0185cf2e5f39ae7c6 2024-04-16T19:54:20.631 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.632 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.634 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000007cec5af18edde856-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '385', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.634 DEBUG:botocore.parsers:Response body: b'Allowfalsetx000007cec5af18edde856-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.634 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.634 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.634 DEBUG:botocore.hooks:Event after-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.634 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.635 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.635 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.635 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.635 DEBUG:botocore.hooks:Event before-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.635 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.635 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler > 2024-04-16T19:54:20.635 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.636 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.636 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date b8c9ebd42f207eccd0f1ac4735e5608d9b5aca42e16ba14ba414850016f19bb6 2024-04-16T19:54:20.636 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request c17681ec3b738593b54c93e227b43c7f217a5ae242c14bad24c844316af3cc60 2024-04-16T19:54:20.636 DEBUG:botocore.auth:Signature: 9f3479c4247622a84ea5e3601f76653772c843f983acfa137bf3db98f606c476 2024-04-16T19:54:20.636 DEBUG:botocore.hooks:Event request-created.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.636 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.638 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000e50055a95df2bfd-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '385', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.638 DEBUG:botocore.parsers:Response body: b'Allowfalsetx000000e50055a95df2bfd-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.639 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.639 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.639 DEBUG:botocore.hooks:Event after-call.iam.ListGroupPolicies: calling handler 2024-04-16T19:54:20.639 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.639 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.639 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.640 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.640 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.640 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.640 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler > 2024-04-16T19:54:20.640 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.640 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.640 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4cdafc9ca13a31ddd9a7a3bfb5e4a2230e855a4b6f877fc7873c6f03bf7ad991 2024-04-16T19:54:20.640 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 56593d80f0047130e5a04489521326dbe7d9d4cdbef2878719f11fd2efda12a8 2024-04-16T19:54:20.641 DEBUG:botocore.auth:Signature: fb6ce485d620522f397bb32c5dd908d736ddaef7b6cb22cc93ec616e77a229cc 2024-04-16T19:54:20.641 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.641 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.643 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000663ca5c69661c71-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '530', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.643 DEBUG:botocore.parsers:Response body: b'AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx000000663ca5c69661c71-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.643 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.643 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.643 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.644 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.644 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.644 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.644 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.644 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.644 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedGroupPolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedGroupPolicies', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.644 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler > 2024-04-16T19:54:20.644 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.645 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.645 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 4cdafc9ca13a31ddd9a7a3bfb5e4a2230e855a4b6f877fc7873c6f03bf7ad991 2024-04-16T19:54:20.645 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 5d88b185da449624be670ecf5c3538915851df262fb820e4dc971c6261bd683d 2024-04-16T19:54:20.645 DEBUG:botocore.auth:Signature: 9d074c53b9c4eb5f4c2181da96d2d8c8cab3661f65fa2585aea76ee01a5b4788 2024-04-16T19:54:20.645 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.645 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.690 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000863858222165350b-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '530', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.690 DEBUG:botocore.parsers:Response body: b'AmazonS3FullAccessarn:aws:iam::aws:policy/AmazonS3FullAccessfalsetx00000863858222165350b-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.691 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.691 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.691 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedGroupPolicies: calling handler 2024-04-16T19:54:20.691 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.692 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.692 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.692 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.692 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.692 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListOpenIDConnectProviders) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListOpenIDConnectProviders', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.692 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler > 2024-04-16T19:54:20.693 DEBUG:botocore.hooks:Event choose-signer.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.693 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.693 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 6be253d5bef7fe9cb4ed0d00d8a18351f59593888e86906ef784f147279adb10 2024-04-16T19:54:20.693 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request ca2e21b0a2bbdf4c4bb64ea596436e7a0539ba70f4d8a46874ef3dfd71c63744 2024-04-16T19:54:20.693 DEBUG:botocore.auth:Signature: 24cfa9eddb731565aee055df494fee40265fb9658581a1415e129ae11c1027cf 2024-04-16T19:54:20.693 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.694 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.708 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000005d647f3dec2ae850-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '357', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.708 DEBUG:botocore.parsers:Response body: b'tx000005d647f3dec2ae850-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.709 DEBUG:botocore.hooks:Event needs-retry.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.709 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.709 DEBUG:botocore.hooks:Event after-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.709 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.709 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.710 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:20.710 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.710 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.710 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListOpenIDConnectProviders) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListOpenIDConnectProviders', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.710 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler > 2024-04-16T19:54:20.711 DEBUG:botocore.hooks:Event choose-signer.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.711 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.711 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 6be253d5bef7fe9cb4ed0d00d8a18351f59593888e86906ef784f147279adb10 2024-04-16T19:54:20.711 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 010dd3451ecc104cf30533c451c15c1ca95a716732068841c6370223a1607964 2024-04-16T19:54:20.711 DEBUG:botocore.auth:Signature: 04fc0a3ba1d6736cc79433582d05ca237eaa36692f32ac3701050c605a5725a3 2024-04-16T19:54:20.711 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.712 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.725 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000003a2290397dcbdc0c-00661ed76c-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '357', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.726 DEBUG:botocore.parsers:Response body: b'tx000003a2290397dcbdc0c-00661ed76c-5331-test-zone2' 2024-04-16T19:54:20.726 DEBUG:botocore.hooks:Event needs-retry.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.726 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.726 DEBUG:botocore.hooks:Event after-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:20.727 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.727 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.727 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.727 DEBUG:botocore.hooks:Event before-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.727 DEBUG:botocore.hooks:Event before-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.728 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.728 DEBUG:botocore.hooks:Event request-created.iam.DetachRolePolicy: calling handler > 2024-04-16T19:54:20.728 DEBUG:botocore.hooks:Event choose-signer.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.728 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.728 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date bd73383d5317fe146a37b32cc167dca318a97704e67083ae893e94c203174404 2024-04-16T19:54:20.729 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request bc719d804186f8d4eb96de1b829ebc0710ffe53d884594962ee2f74dd98be869 2024-04-16T19:54:20.729 DEBUG:botocore.auth:Signature: ffe65364595b83d339e6a31a215fbe9a2c348cb681149418fb3359664cf6fc3b 2024-04-16T19:54:20.729 DEBUG:botocore.hooks:Event request-created.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.729 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.778 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000011c245288b8c0b12-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.778 DEBUG:botocore.parsers:Response body: b'tx0000011c245288b8c0b12-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.779 DEBUG:botocore.hooks:Event needs-retry.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.779 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.779 DEBUG:botocore.hooks:Event after-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:20.779 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.779 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.780 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.780 DEBUG:botocore.hooks:Event before-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.780 DEBUG:botocore.hooks:Event before-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.780 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone1', 'PolicyName': 'Allow'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.780 DEBUG:botocore.hooks:Event request-created.iam.DeleteRolePolicy: calling handler > 2024-04-16T19:54:20.781 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.781 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.781 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date ff2b8e08f5bd2f6e968bc5d2892f7a2005ce31351a054fa3113914bd584e4a51 2024-04-16T19:54:20.781 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 950abf15e288591a22ea71e83eeb48f1a5cd7e555e085eb4bceb0146fcd45678 2024-04-16T19:54:20.781 DEBUG:botocore.auth:Signature: 81612bb21c553767e4936b18d20865fbc6760a5ad97a4966de3ce44308edd81b 2024-04-16T19:54:20.781 DEBUG:botocore.hooks:Event request-created.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.782 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.790 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000364ff7e81efa1c11-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '163', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.790 DEBUG:botocore.parsers:Response body: b'tx00000364ff7e81efa1c11-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.790 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.790 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.791 DEBUG:botocore.hooks:Event after-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:20.791 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteRole: calling handler 2024-04-16T19:54:20.791 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.791 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.791 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:20.791 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:20.791 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteRole', 'Version': '2010-05-08', 'RoleName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.792 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler > 2024-04-16T19:54:20.792 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteRole: calling handler 2024-04-16T19:54:20.792 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.792 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date eb0db1c076fced9cc6f71d7a5176de1b17d1027791a1e35ab50e96286833a6ef 2024-04-16T19:54:20.792 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request a3aed76bd37816015752cfaac73da729a5daebbbcc49331e8cd4cc73117c7502 2024-04-16T19:54:20.792 DEBUG:botocore.auth:Signature: c7be737879b9166065c62c220a1761701a0b01a42d0c5555a15469bcc7e1b8dd 2024-04-16T19:54:20.792 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler 2024-04-16T19:54:20.793 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.803 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000fc2a552d0dd67acc-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '151', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.803 DEBUG:botocore.parsers:Response body: b'tx00000fc2a552d0dd67acc-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.803 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteRole: calling handler 2024-04-16T19:54:20.803 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.803 DEBUG:botocore.hooks:Event after-call.iam.DeleteRole: calling handler 2024-04-16T19:54:20.804 DEBUG:botocore.hooks:Event before-parameter-build.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.804 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.804 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.804 DEBUG:botocore.hooks:Event before-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.804 DEBUG:botocore.hooks:Event before-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.805 DEBUG:botocore.endpoint:Making request for OperationModel(name=RemoveUserFromGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'RemoveUserFromGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.805 DEBUG:botocore.hooks:Event request-created.iam.RemoveUserFromGroup: calling handler > 2024-04-16T19:54:20.805 DEBUG:botocore.hooks:Event choose-signer.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.805 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.805 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 8101fbe751c2510228d0196ca3a189db821341cab247a56bf56b1a8de94be62f 2024-04-16T19:54:20.805 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request acfdd94bb110329b9baeb1824875327124ef97b26966df8041aa2927a3b48ca4 2024-04-16T19:54:20.806 DEBUG:botocore.auth:Signature: 5bc8f241c4aa444d67efd4a31fdee9eb9b9993aecbff0e001a0b70f4fb0a6b55 2024-04-16T19:54:20.806 DEBUG:botocore.hooks:Event request-created.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.806 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.858 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000aaa1c1d718f5c95c-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '257', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.858 DEBUG:botocore.parsers:Response body: b'tx00000aaa1c1d718f5c95c-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.859 DEBUG:botocore.hooks:Event needs-retry.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.859 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.859 DEBUG:botocore.hooks:Event after-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:20.859 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.860 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.860 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.860 DEBUG:botocore.hooks:Event before-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.860 DEBUG:botocore.hooks:Event before-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.860 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.860 DEBUG:botocore.hooks:Event request-created.iam.DetachGroupPolicy: calling handler > 2024-04-16T19:54:20.861 DEBUG:botocore.hooks:Event choose-signer.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.861 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.861 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 93eba8cc47d76370fe5425ab749ce1edbed126d87b12ab5216df76b39f891c29 2024-04-16T19:54:20.861 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 5435dff9f818c876f4a84424250487eef0de6c3c443655c72de76bb06b547708 2024-04-16T19:54:20.861 DEBUG:botocore.auth:Signature: 7aa891623e24d6eddd6546a1277519d181ad558615a11379d91006b9fb4f989e 2024-04-16T19:54:20.861 DEBUG:botocore.hooks:Event request-created.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.862 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.868 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000049c6cea68f3841e6-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.868 DEBUG:botocore.parsers:Response body: b'tx0000049c6cea68f3841e6-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.868 DEBUG:botocore.hooks:Event needs-retry.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.868 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.869 DEBUG:botocore.hooks:Event after-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:20.869 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.869 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.869 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.870 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.870 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.870 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone1', 'PolicyName': 'Allow'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.870 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroupPolicy: calling handler > 2024-04-16T19:54:20.871 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.871 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.872 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date bcb6e0830d4e4b57d0c03ed18cd3a27f9eed72fb600ce4b187dfdf4f5a23a0f1 2024-04-16T19:54:20.872 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request e9b20287b99e14adb5660daa8ca6f8b25f1e8fe03b37124b97ee5b1e4050389f 2024-04-16T19:54:20.872 DEBUG:botocore.auth:Signature: e410881ad9ea6966f8936e8324458afa9d2fb55ef1c107fe18d114349873abc0 2024-04-16T19:54:20.872 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.872 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.879 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000039c653a1b020d163-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.879 DEBUG:botocore.parsers:Response body: b'tx0000039c653a1b020d163-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.879 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.879 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.879 DEBUG:botocore.hooks:Event after-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:20.880 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.880 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.880 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.880 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.880 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.880 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.881 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroup: calling handler > 2024-04-16T19:54:20.881 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.881 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.881 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date fe2ed6547b0314458d41caf46f8196bcd62a800e25af60d633b29183f00efad6 2024-04-16T19:54:20.881 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 3d6b84df3245855c135f277721697e91adac9fc263c5c3348296a641fb3d447e 2024-04-16T19:54:20.882 DEBUG:botocore.auth:Signature: b5151308a424a50079dbb409766dd5265de3080547d56897c0392e6f4b60e2a3 2024-04-16T19:54:20.882 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.882 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.934 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000958d9c2fb1de1ad2-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '241', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.935 DEBUG:botocore.parsers:Response body: b'tx00000958d9c2fb1de1ad2-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.935 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.935 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.935 DEBUG:botocore.hooks:Event after-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:20.935 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.936 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.936 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.936 DEBUG:botocore.hooks:Event before-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.936 DEBUG:botocore.hooks:Event before-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.936 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone1', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.937 DEBUG:botocore.hooks:Event request-created.iam.DetachUserPolicy: calling handler > 2024-04-16T19:54:20.937 DEBUG:botocore.hooks:Event choose-signer.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.937 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.937 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date e4eb10f876ac99edb55b6e38ac859bdca8126307348276c5e6af1c89db67d5ef 2024-04-16T19:54:20.937 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 7c0e0fc712df7fcfcb4052a994ea5e443cca7f48f541d5f464f7176996279660 2024-04-16T19:54:20.938 DEBUG:botocore.auth:Signature: 67d05becf58485211962b81671a8ce1db96414776c8275c5d04f2ebadac31701 2024-04-16T19:54:20.938 DEBUG:botocore.hooks:Event request-created.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.938 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.950 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000009c0c96788ca593f0-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.950 DEBUG:botocore.parsers:Response body: b'tx000009c0c96788ca593f0-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.951 DEBUG:botocore.hooks:Event needs-retry.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.951 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.951 DEBUG:botocore.hooks:Event after-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:20.951 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.952 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.952 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.952 DEBUG:botocore.hooks:Event before-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.952 DEBUG:botocore.hooks:Event before-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.952 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone1', 'PolicyName': 'Allow'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.953 DEBUG:botocore.hooks:Event request-created.iam.DeleteUserPolicy: calling handler > 2024-04-16T19:54:20.953 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.953 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.953 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date a8978949ce38abd713a4ef1765a7b6091a66572aed9d67f96ac31a077d7e0d99 2024-04-16T19:54:20.953 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 08bf66d3d91db5dc55ca1912450899fdfe0fe18f85eeda07e3cb90d9516c85ba 2024-04-16T19:54:20.953 DEBUG:botocore.auth:Signature: 55fe6689d8a7b9a72a7380b58545d9dfbf0f5700485d8b3a23822bc0c7f86d74 2024-04-16T19:54:20.954 DEBUG:botocore.hooks:Event request-created.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.954 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:20.966 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000000f2805c6dc86d1b8-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '217', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:20.966 DEBUG:botocore.parsers:Response body: b'tx000000f2805c6dc86d1b8-00661ed76c-4812-test-zone1' 2024-04-16T19:54:20.967 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.967 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:20.967 DEBUG:botocore.hooks:Event after-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:20.967 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.967 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:20.967 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:20.967 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.968 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.968 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:20.968 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:20.968 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.968 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:20.968 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195420Z content-type;host;x-amz-date 919fd997a1071a9eb8adeb0d8e2a852ea3984fffd61b7a1dc133995556ffb778 2024-04-16T19:54:20.968 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195420Z 20240416/test-zonegroup/iam/aws4_request 035fd1e4027529a44339174d81e1bc677efe549549e9126cefa2dacb790e8e7a 2024-04-16T19:54:20.968 DEBUG:botocore.auth:Signature: 8c6788d37d60fce18e45a1b50a587ac7c99d12211398e7fd4b0e2b1e9c930908 2024-04-16T19:54:20.969 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:20.969 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.018 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000359d97bca4340c47-00661ed76c-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:20 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.018 DEBUG:botocore.parsers:Response body: b'test-zone1test-zone1IBVMW6I15HR0T84KDGA7Active2024-04-16T19:54:13.683670Zfalse' 2024-04-16T19:54:21.019 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.019 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.019 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.020 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.020 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.020 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:21.020 DEBUG:botocore.hooks:Event before-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.020 DEBUG:botocore.hooks:Event before-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.021 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteAccessKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteAccessKey', 'Version': '2010-05-08', 'UserName': 'test-zone1', 'AccessKeyId': 'IBVMW6I15HR0T84KDGA7'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.021 DEBUG:botocore.hooks:Event request-created.iam.DeleteAccessKey: calling handler > 2024-04-16T19:54:21.021 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.021 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.021 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date b01e6dcfe3fd8e188232b78c0097a807d40bb8e2255dbf7008f0af10ae4ca67a 2024-04-16T19:54:21.022 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 428b3a4eec5730f58aa969aaf2311d43f2181d9cf0bce0adaf611317238b6ee0 2024-04-16T19:54:21.022 DEBUG:botocore.auth:Signature: c6250d64d8d726cb0abae9a37b9cdd3ab29778c64a036ca1478aa392355911ad 2024-04-16T19:54:21.022 DEBUG:botocore.hooks:Event request-created.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.022 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.033 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000cdd57efdbfb5d165-00661ed76d-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '249', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.033 DEBUG:botocore.parsers:Response body: b'tx00000cdd57efdbfb5d165-00661ed76d-4812-test-zone1' 2024-04-16T19:54:21.033 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.033 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.034 DEBUG:botocore.hooks:Event after-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.034 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteUser: calling handler 2024-04-16T19:54:21.034 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.034 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:21.034 DEBUG:botocore.hooks:Event before-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.035 DEBUG:botocore.hooks:Event before-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.035 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteUser', 'Version': '2010-05-08', 'UserName': 'test-zone1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.035 DEBUG:botocore.hooks:Event request-created.iam.DeleteUser: calling handler > 2024-04-16T19:54:21.035 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteUser: calling handler 2024-04-16T19:54:21.036 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.036 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 73b6ad4b3e5d11fba46ac1e98e9ba27cc1964c1d8192f1fa1a1bf17be5282a09 2024-04-16T19:54:21.036 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 6c0892d794602d46c386e1ba2cb37aaddb15db2b8409135b8d010f37331b9a94 2024-04-16T19:54:21.036 DEBUG:botocore.auth:Signature: 70613badae1b591b926527f76e480311c40294d29f274222e58b93d310ef8111 2024-04-16T19:54:21.036 DEBUG:botocore.hooks:Event request-created.iam.DeleteUser: calling handler 2024-04-16T19:54:21.036 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.046 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000021d52e1dde391306-00661ed76d-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.046 DEBUG:botocore.parsers:Response body: b'tx0000021d52e1dde391306-00661ed76d-4812-test-zone1' 2024-04-16T19:54:21.046 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteUser: calling handler 2024-04-16T19:54:21.047 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.047 DEBUG:botocore.hooks:Event after-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.047 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.047 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.047 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.047 DEBUG:botocore.hooks:Event before-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.048 DEBUG:botocore.hooks:Event before-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.048 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.048 DEBUG:botocore.hooks:Event request-created.iam.DetachRolePolicy: calling handler > 2024-04-16T19:54:21.048 DEBUG:botocore.hooks:Event choose-signer.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.048 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.049 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date c325a97cec49c93483b73d102a4ee669ab364fdbcde01894c7995f875c5e42ea 2024-04-16T19:54:21.049 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 350ce9f54c1ccc55e1ba3ca4d65563bc3915528ea0a394ba1a5f663b5015c94e 2024-04-16T19:54:21.049 DEBUG:botocore.auth:Signature: 9b1c5735357b84c268cb9203d486a39e5763ac4c40cff583681081a09d0520d4 2024-04-16T19:54:21.049 DEBUG:botocore.hooks:Event request-created.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.049 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.060 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000c34b288ce97cf6eb-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.060 DEBUG:botocore.parsers:Response body: b'tx00000c34b288ce97cf6eb-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.061 DEBUG:botocore.hooks:Event needs-retry.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.061 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.061 DEBUG:botocore.hooks:Event after-call.iam.DetachRolePolicy: calling handler 2024-04-16T19:54:21.061 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.062 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.062 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.062 DEBUG:botocore.hooks:Event before-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.062 DEBUG:botocore.hooks:Event before-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.062 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteRolePolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteRolePolicy', 'Version': '2010-05-08', 'RoleName': 'test-zone2', 'PolicyName': 'Allow'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.062 DEBUG:botocore.hooks:Event request-created.iam.DeleteRolePolicy: calling handler > 2024-04-16T19:54:21.063 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.063 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.063 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 0e711179b95df7649427669dd02fc5b80c804175ee6ff14feb947361680d8f4c 2024-04-16T19:54:21.063 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 113eef6a75d8e39bf880c2cbd7cc68d9e28c8d18425e397a964a53f3761d0d08 2024-04-16T19:54:21.063 DEBUG:botocore.auth:Signature: 382a82fc0a0e79bb3d2fef903d491b82555fbb7105a7cc219b556c33d1fff0cf 2024-04-16T19:54:21.063 DEBUG:botocore.hooks:Event request-created.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.064 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.116 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000083e6a6b827feac06-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '163', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.116 DEBUG:botocore.parsers:Response body: b'tx0000083e6a6b827feac06-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.117 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.117 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.117 DEBUG:botocore.hooks:Event after-call.iam.DeleteRolePolicy: calling handler 2024-04-16T19:54:21.117 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteRole: calling handler 2024-04-16T19:54:21.117 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.118 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.118 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:21.118 DEBUG:botocore.hooks:Event before-call.iam.DeleteRole: calling handler 2024-04-16T19:54:21.118 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteRole', 'Version': '2010-05-08', 'RoleName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.119 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler > 2024-04-16T19:54:21.119 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteRole: calling handler 2024-04-16T19:54:21.119 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.119 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 832fae51b1d10776be58658abbb6e5d165d2e58ca758e9bedbbbabf3d441ea56 2024-04-16T19:54:21.119 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request ff6bbc17129d3d9ae3788e47f942819b1c7e86119e6ad8d345baeebd402fff41 2024-04-16T19:54:21.119 DEBUG:botocore.auth:Signature: a03c921474a9a2b14ffd8d33ace0c5d60a4e875741e1fc9666beb6457f27e09d 2024-04-16T19:54:21.119 DEBUG:botocore.hooks:Event request-created.iam.DeleteRole: calling handler 2024-04-16T19:54:21.120 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.226 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000007b319c2d7f443442-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '151', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.226 DEBUG:botocore.parsers:Response body: b'tx000007b319c2d7f443442-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.227 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteRole: calling handler 2024-04-16T19:54:21.227 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.227 DEBUG:botocore.hooks:Event after-call.iam.DeleteRole: calling handler 2024-04-16T19:54:21.227 DEBUG:botocore.hooks:Event before-parameter-build.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.227 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.228 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.228 DEBUG:botocore.hooks:Event before-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.228 DEBUG:botocore.hooks:Event before-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.228 DEBUG:botocore.endpoint:Making request for OperationModel(name=RemoveUserFromGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'RemoveUserFromGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.228 DEBUG:botocore.hooks:Event request-created.iam.RemoveUserFromGroup: calling handler > 2024-04-16T19:54:21.228 DEBUG:botocore.hooks:Event choose-signer.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.229 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.229 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date f37f0ec6c347966013e62f32fd007623cac80b2e0db8e71ca06331c77d3b3c28 2024-04-16T19:54:21.229 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 2fb2bb4a7797d5ed6e21e2dec65b2b8c59645e633f1d3c093ca08789f23c51a3 2024-04-16T19:54:21.229 DEBUG:botocore.auth:Signature: 14d535784eb75465410654bbc3dffe254df7c80e63a3469e2f5b358fb4257598 2024-04-16T19:54:21.229 DEBUG:botocore.hooks:Event request-created.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.229 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.246 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e8d77b38125b0728-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '257', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.246 DEBUG:botocore.parsers:Response body: b'tx00000e8d77b38125b0728-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.247 DEBUG:botocore.hooks:Event needs-retry.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.247 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.247 DEBUG:botocore.hooks:Event after-call.iam.RemoveUserFromGroup: calling handler 2024-04-16T19:54:21.247 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.247 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.248 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.248 DEBUG:botocore.hooks:Event before-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.248 DEBUG:botocore.hooks:Event before-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.248 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.248 DEBUG:botocore.hooks:Event request-created.iam.DetachGroupPolicy: calling handler > 2024-04-16T19:54:21.248 DEBUG:botocore.hooks:Event choose-signer.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.249 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.249 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date d10497e442384ad8daba8c183ca9a44d1658b63c4ded1005678646a32710f983 2024-04-16T19:54:21.249 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 9de95c4413865372513ffc51a7f14572b6bb1bb3f46add55dc4b8c388123356f 2024-04-16T19:54:21.249 DEBUG:botocore.auth:Signature: a9b3a4866f2c99424bca0aa209b1f38fce2f145ea98d1daf855fe292c62595cf 2024-04-16T19:54:21.249 DEBUG:botocore.hooks:Event request-created.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.249 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.301 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b4b97099223b9dca-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.301 DEBUG:botocore.parsers:Response body: b'tx00000b4b97099223b9dca-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.302 DEBUG:botocore.hooks:Event needs-retry.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.302 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.302 DEBUG:botocore.hooks:Event after-call.iam.DetachGroupPolicy: calling handler 2024-04-16T19:54:21.302 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.302 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.302 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.303 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.303 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.303 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteGroupPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteGroupPolicy', 'Version': '2010-05-08', 'GroupName': 'test-zone2', 'PolicyName': 'Allow'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.303 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroupPolicy: calling handler > 2024-04-16T19:54:21.304 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.304 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.304 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date bc90399dc585c9565b8e09c006ff5a452f25f284492b3177a00a6816d35ed452 2024-04-16T19:54:21.304 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 9de039d04afd98afa84a8206001fc37becbd365d4598ccc469719717faf9986e 2024-04-16T19:54:21.304 DEBUG:botocore.auth:Signature: 141216c438124aa66232d693c1553a14db94cd268a3515a52510370883cf57cf 2024-04-16T19:54:21.304 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.305 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.398 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e74a6d8cf1969123-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '253', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.398 DEBUG:botocore.parsers:Response body: b'tx00000e74a6d8cf1969123-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.399 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.399 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.399 DEBUG:botocore.hooks:Event after-call.iam.DeleteGroupPolicy: calling handler 2024-04-16T19:54:21.399 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.400 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.400 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.400 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.400 DEBUG:botocore.hooks:Event before-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.400 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteGroup) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteGroup', 'Version': '2010-05-08', 'GroupName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.400 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroup: calling handler > 2024-04-16T19:54:21.401 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.401 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.401 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 44bf744101969da17efb1509be4f7be6d0a4c5a9f8dcfdeffea997cc24d810c7 2024-04-16T19:54:21.401 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 41649091bd59adef3cf41a274b5153b7ba73d43b3a92d96c7ad1c17d6ef7cdf8 2024-04-16T19:54:21.401 DEBUG:botocore.auth:Signature: d1b8b9edc91a11a037162534f7f85505424176a820950c9fc6faeeae8440167d 2024-04-16T19:54:21.402 DEBUG:botocore.hooks:Event request-created.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.402 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.423 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000004ac24ef20a91fb3b-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '241', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.423 DEBUG:botocore.parsers:Response body: b'tx000004ac24ef20a91fb3b-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.424 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.424 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.424 DEBUG:botocore.hooks:Event after-call.iam.DeleteGroup: calling handler 2024-04-16T19:54:21.424 DEBUG:botocore.hooks:Event before-parameter-build.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.424 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.424 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.425 DEBUG:botocore.hooks:Event before-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.425 DEBUG:botocore.hooks:Event before-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.425 DEBUG:botocore.endpoint:Making request for OperationModel(name=DetachUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DetachUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone2', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonS3FullAccess'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.425 DEBUG:botocore.hooks:Event request-created.iam.DetachUserPolicy: calling handler > 2024-04-16T19:54:21.425 DEBUG:botocore.hooks:Event choose-signer.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.426 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.426 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date e19d91a90ba0f4a2add980542fe58bb8ea7de111e90548b27ec9798cc678a117 2024-04-16T19:54:21.426 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request a1bfc974bc0706e6b0bcc7502b5060ee64ed6176112dd6fb18ff45c6cc336a22 2024-04-16T19:54:21.426 DEBUG:botocore.auth:Signature: 75490d1950bc35dee32db57c01dcf9dcc9b977813efd62314dd2fbbb22b4beda 2024-04-16T19:54:21.426 DEBUG:botocore.hooks:Event request-created.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.427 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.484 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000079e8e5656ab7052-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '213', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.484 DEBUG:botocore.parsers:Response body: b'tx00000079e8e5656ab7052-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.484 DEBUG:botocore.hooks:Event needs-retry.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.484 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.485 DEBUG:botocore.hooks:Event after-call.iam.DetachUserPolicy: calling handler 2024-04-16T19:54:21.485 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.485 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.485 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.486 DEBUG:botocore.hooks:Event before-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.486 DEBUG:botocore.hooks:Event before-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.486 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteUserPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteUserPolicy', 'Version': '2010-05-08', 'UserName': 'test-zone2', 'PolicyName': 'Allow'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.486 DEBUG:botocore.hooks:Event request-created.iam.DeleteUserPolicy: calling handler > 2024-04-16T19:54:21.486 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.487 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.487 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date d2741fea5e8fde2129ad86dfc4cf8cd0689aa8930033a952888845632f37aeac 2024-04-16T19:54:21.487 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 88ee735c83ae91b0d8659811624d3a0e1faa8badc8d2baea3380d91032339a0a 2024-04-16T19:54:21.487 DEBUG:botocore.auth:Signature: 81f7dd317a1fac68712c9edab135cfca8328cb161b742f0e1d8577539fb84c72 2024-04-16T19:54:21.487 DEBUG:botocore.hooks:Event request-created.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.488 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.590 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000002446ddea400b8c25-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '217', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.590 DEBUG:botocore.parsers:Response body: b'tx000002446ddea400b8c25-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.591 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.591 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.591 DEBUG:botocore.hooks:Event after-call.iam.DeleteUserPolicy: calling handler 2024-04-16T19:54:21.591 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.591 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.592 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.592 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.592 DEBUG:botocore.hooks:Event before-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.592 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.592 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler > 2024-04-16T19:54:21.592 DEBUG:botocore.hooks:Event choose-signer.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.593 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.593 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 575b893e712663944c456f61f1aabd27df830cbe7212c952bf0cd211f66fc279 2024-04-16T19:54:21.593 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request c38c8f7f2712188eaae2f712db0330fd99bcc3e757c8c79a4e92af70877897b3 2024-04-16T19:54:21.593 DEBUG:botocore.auth:Signature: 9a37fc2ababc51acd70b97b7348a5691b549da47ab432e6d55c93c465f06408e 2024-04-16T19:54:21.593 DEBUG:botocore.hooks:Event request-created.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.594 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.597 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx000006e348144b9552495-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.597 DEBUG:botocore.parsers:Response body: b'test-zone2test-zone231HO20TWHBMRQFU8RLUEActive2024-04-16T19:54:13.956210Zfalse' 2024-04-16T19:54:21.598 DEBUG:botocore.hooks:Event needs-retry.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.598 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.598 DEBUG:botocore.hooks:Event after-call.iam.ListAccessKeys: calling handler 2024-04-16T19:54:21.598 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.598 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.599 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.599 DEBUG:botocore.hooks:Event before-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.599 DEBUG:botocore.hooks:Event before-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.599 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteAccessKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteAccessKey', 'Version': '2010-05-08', 'UserName': 'test-zone2', 'AccessKeyId': '31HO20TWHBMRQFU8RLUE'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.599 DEBUG:botocore.hooks:Event request-created.iam.DeleteAccessKey: calling handler > 2024-04-16T19:54:21.600 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.600 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.600 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 5b50936bce022f0659588b8952387822306d1cfda2a633781316d6a580a4ce5a 2024-04-16T19:54:21.600 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request c7b96bb609565221f36ee9e30d47210555add83c400a4e20cbdae549acb488f5 2024-04-16T19:54:21.600 DEBUG:botocore.auth:Signature: a24f27fe4e50524c31d02484e70b25d85dca64ff0ecff2fca2ada309a8d3994b 2024-04-16T19:54:21.600 DEBUG:botocore.hooks:Event request-created.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.601 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.618 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000081aa4ebd2187ca2d-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '249', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.618 DEBUG:botocore.parsers:Response body: b'tx0000081aa4ebd2187ca2d-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.619 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.619 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.619 DEBUG:botocore.hooks:Event after-call.iam.DeleteAccessKey: calling handler 2024-04-16T19:54:21.619 DEBUG:botocore.hooks:Event before-parameter-build.iam.DeleteUser: calling handler 2024-04-16T19:54:21.620 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:21.620 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:21.620 DEBUG:botocore.hooks:Event before-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.620 DEBUG:botocore.hooks:Event before-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.620 DEBUG:botocore.endpoint:Making request for OperationModel(name=DeleteUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'DeleteUser', 'Version': '2010-05-08', 'UserName': 'test-zone2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:21.621 DEBUG:botocore.hooks:Event request-created.iam.DeleteUser: calling handler > 2024-04-16T19:54:21.621 DEBUG:botocore.hooks:Event choose-signer.iam.DeleteUser: calling handler 2024-04-16T19:54:21.621 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:21.621 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195421Z content-type;host;x-amz-date 5cec3e80938f8d7af48456f647335d80c040d9c550731683093602f874824e23 2024-04-16T19:54:21.622 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195421Z 20240416/test-zonegroup/iam/aws4_request 10446b584015f601d322a0eda610f3db8e3aee3b631244fac6c95c265cf9c880 2024-04-16T19:54:21.622 DEBUG:botocore.auth:Signature: 3891f7a52dcef88a4acf4dea2d7e6e3e3e943f1827968df75279e0a9e2b3c90a 2024-04-16T19:54:21.622 DEBUG:botocore.hooks:Event request-created.iam.DeleteUser: calling handler 2024-04-16T19:54:21.622 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:21.675 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006a446bc3baede641-00661ed76d-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:21 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:21.675 DEBUG:botocore.parsers:Response body: b'tx000006a446bc3baede641-00661ed76d-5331-test-zone2' 2024-04-16T19:54:21.676 DEBUG:botocore.hooks:Event needs-retry.iam.DeleteUser: calling handler 2024-04-16T19:54:21.676 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:21.676 DEBUG:botocore.hooks:Event after-call.iam.DeleteUser: calling handler 2024-04-16T19:54:21.676 INFO:rgw_multi.tests:meta checkpoint 2024-04-16T19:54:21.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:21.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:21.677 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:21.731 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:21.731 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:21.788 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:21.812 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713297261.412882_672.1","last_update":"2024-04-16T19:54:21.412882Z"}] 2024-04-16T19:54:21.812 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297261.412882_672.1'} 2024-04-16T19:54:21.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:21.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:21.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:21.813 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:21.918+0000 7fb3df016800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:21.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:21.920+0000 7fb3df016800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:21.943 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.492555_131.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.492555Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297260.812494_849.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:20.812494Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297261.412882_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.412882Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:21Z"} 2024-04-16T19:54:21.944 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.492555_131.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.492555Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297260.812494_849.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:20.812494Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297261.412882_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.412882Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:21Z"} 2024-04-16T19:54:21.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.492555_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.492555Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297260.812494_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:20.812494Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297261.412882_672.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.412882Z', 'realm_epoch': 2}}] 2024-04-16T19:54:21.944 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297261.412882_672.1'} 2024-04-16T19:54:21.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.492555_131.1', 2: '1_1713297260.812494_849.1', 3: '1_1713297261.412882_672.1'} 2024-04-16T19:54:21.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1713297261.630622_136.1 target=1_1713297261.492555_131.1, shard=2 master=1_1713297261.044058_860.1 target=1_1713297260.812494_849.1 2024-04-16T19:54:26.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:26.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:26.945 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:27.002 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:27.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:27.054+0000 7f546a412800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:27.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:27.055+0000 7f546a412800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:27.078 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297261.412882_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.412882Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T19:54:27.079 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297261.412882_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.412882Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T19:54:27.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297261.412882_672.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.412882Z', 'realm_epoch': 2}}] 2024-04-16T19:54:27.079 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297261.412882_672.1'} 2024-04-16T19:54:27.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297261.412882_672.1'} 2024-04-16T19:54:27.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:27.080 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:27.081 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.081 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.081 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.081 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.082 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.082 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:27.082 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:27.083 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.083 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:27.083 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 7e25b0f19d455c358aa952cd798d3c7cfaa3de47d9f25b26f180803b942f382d 2024-04-16T19:54:27.083 DEBUG:botocore.auth:Signature: a7ad48407a18712a05837ae5e277a429460c17e39ba18492ecd4102486225c67 2024-04-16T19:54:27.083 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:27.084 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.088 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000933b0f93480d5db3-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1519', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.088 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111falsetx00000933b0f93480d5db3-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.089 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:27.089 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.089 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.090 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRoles: calling handler 2024-04-16T19:54:27.090 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.090 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.090 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.090 DEBUG:botocore.hooks:Event before-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.091 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRoles) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRoles', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.091 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler > 2024-04-16T19:54:27.091 DEBUG:botocore.hooks:Event choose-signer.iam.ListRoles: calling handler 2024-04-16T19:54:27.092 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.092 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 8e689dc6b69b17003f0460011582254d8643f0dfb48ac46e093a24df58170b3f 2024-04-16T19:54:27.092 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request c8c5eb619155ea7a4b6ef549bb7b256f6eb197c61172e622e0d84b059a4e3dcd 2024-04-16T19:54:27.092 DEBUG:botocore.auth:Signature: 40ac9ea587400d81468d81249a1eee8f9c0a95c59caa576a0d37693b5e07c311 2024-04-16T19:54:27.092 DEBUG:botocore.hooks:Event request-created.iam.ListRoles: calling handler 2024-04-16T19:54:27.093 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.096 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000069697cb3f34fdc66-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1519', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.096 DEBUG:botocore.parsers:Response body: b'bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW1111111111111111120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}RGW11111111111111111falsetx0000069697cb3f34fdc66-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.097 DEBUG:botocore.hooks:Event needs-retry.iam.ListRoles: calling handler 2024-04-16T19:54:27.097 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.097 DEBUG:botocore.hooks:Event after-call.iam.ListRoles: calling handler 2024-04-16T19:54:27.097 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:27.098 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.098 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.098 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.098 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.098 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.099 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:27.099 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:27.099 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.099 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:27.100 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 7879db766332785b99a6acb672232d889e13a0d13e7fb0570ca6a3081cf5c144 2024-04-16T19:54:27.100 DEBUG:botocore.auth:Signature: 8b15dad8732781640218a7f4c785e11b88f35524b019b9d6b63d5d6ed402b106 2024-04-16T19:54:27.100 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:27.100 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.146 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000fd9e5376d47ddbae-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.147 DEBUG:botocore.parsers:Response body: b'tx00000fd9e5376d47ddbae-00661ed773-4812-test-zone1bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:27.147 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:27.147 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.147 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:27.148 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:27.148 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.148 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.148 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.148 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.149 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.149 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:27.149 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:27.149 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.149 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 625efa5752e46ed22d33c1a9ac8e6a2027ae25c0c944ab70958e490a4727642c 2024-04-16T19:54:27.150 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request a785c9fd31c35da39c7ecdee28d6e09949ff4257c65df445cdb9385896c10fa2 2024-04-16T19:54:27.150 DEBUG:botocore.auth:Signature: 6c6f8de9997d3da08b3ffea08f45fe0990646234e0dc62ed84276f7bed099608 2024-04-16T19:54:27.150 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:27.150 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.152 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006601c8cb66459c8d-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.153 DEBUG:botocore.parsers:Response body: b'tx000006601c8cb66459c8d-00661ed773-5331-test-zone2bec9a453-046a-4ebd-a738-0cf1bd4c45c7roles-yrjefh-1/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-12024-04-16T19:54:04.871Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:27.153 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:27.153 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.153 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:27.154 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.154 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.154 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.154 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.155 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.155 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.155 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:27.155 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.156 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.156 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:27.156 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request f8278a145f809ddf972be1452af8043d165ec4be23a4cb7c657418bd0b21ae1c 2024-04-16T19:54:27.156 DEBUG:botocore.auth:Signature: 70f015912bbee40236ef96310247b8436b975a859fc8db663bdd909944198d00 2024-04-16T19:54:27.156 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.156 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.159 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000c6156f823a451d3f-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.159 DEBUG:botocore.parsers:Response body: b'tx00000c6156f823a451d3f-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.159 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.159 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.159 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.160 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.160 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.160 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.160 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.160 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.161 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.161 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:27.161 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.161 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.161 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date a510ce707483b5d320b0048ecc7bd7df1adb30decb8183b5cfc10f23640e6de6 2024-04-16T19:54:27.162 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 79b06ec07bcc94185a4b3431fc5578c518633b49b9c53a41723cd6752babf670 2024-04-16T19:54:27.162 DEBUG:botocore.auth:Signature: f3d5b857d3650f93391b065826e88799796faa6fe324d8d961f2cd5ef53d6d52 2024-04-16T19:54:27.162 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.162 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.164 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000801bfa7cd9466c6f-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.164 DEBUG:botocore.parsers:Response body: b'tx00000801bfa7cd9466c6f-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.165 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.165 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.165 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.165 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.166 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.166 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.166 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.166 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.166 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.167 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:27.167 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.167 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.167 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:27.167 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request aca248665bb3baa4e57dc33dcc29eaa0f901792f8767326907c3644754a2abb9 2024-04-16T19:54:27.168 DEBUG:botocore.auth:Signature: 128ebc2613baabc2be15dda2107715f628f80661b18637fe757ae770d5ca1e74 2024-04-16T19:54:27.168 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.168 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.170 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000e20573c11fd2f2c3-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.170 DEBUG:botocore.parsers:Response body: b'tx00000e20573c11fd2f2c3-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.170 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.171 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.171 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.171 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.171 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.171 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.172 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.172 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.172 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-1'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.172 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:27.172 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.173 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.173 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 5f580bf8d72d4ddd78e714181408beb937547d1261ab8a411695ed0f906f7ad3 2024-04-16T19:54:27.173 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 10c6763bdad3ccce064fed4bdd07903118d6f2d902da6c9551794d62a5f11246 2024-04-16T19:54:27.173 DEBUG:botocore.auth:Signature: 6f56f3cc6992ee1357952fd34883ee9198e9849ccef2128faf1adbf561baa81f 2024-04-16T19:54:27.174 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.174 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.176 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000006b60928f74eb825d-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.176 DEBUG:botocore.parsers:Response body: b'tx000006b60928f74eb825d-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.176 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.177 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.177 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.177 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:27.177 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.177 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.177 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.178 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.178 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.178 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:27.178 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:27.179 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.179 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:27.179 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 94eff5f754fc36206ef9191fbc859b14eadca6910ec33dcdbad94351bcd491fc 2024-04-16T19:54:27.179 DEBUG:botocore.auth:Signature: cd400c1bc0544b182fa31332cc9170769658301add2b804c223557c02752cc81 2024-04-16T19:54:27.179 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:27.179 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.182 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000095a51a29be1f732e-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.182 DEBUG:botocore.parsers:Response body: b'tx0000095a51a29be1f732e-00661ed773-4812-test-zone120b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:27.182 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:27.183 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.183 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:27.183 DEBUG:botocore.hooks:Event before-parameter-build.iam.GetRole: calling handler 2024-04-16T19:54:27.183 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.183 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.183 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.184 DEBUG:botocore.hooks:Event before-call.iam.GetRole: calling handler 2024-04-16T19:54:27.184 DEBUG:botocore.endpoint:Making request for OperationModel(name=GetRole) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.184 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler > 2024-04-16T19:54:27.184 DEBUG:botocore.hooks:Event choose-signer.iam.GetRole: calling handler 2024-04-16T19:54:27.184 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.185 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 61ca15a23c2fbd86783928dceba4b22aca6677e176b05aaec82d005063bd336e 2024-04-16T19:54:27.185 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request f61819663a77109e6c7a3fc161d2c1f1ec381513cba07cb0b4442b587acfd564 2024-04-16T19:54:27.185 DEBUG:botocore.auth:Signature: 553ec871947779e9b3aea3a56f18b40c1167a9e11492fa72238df526c8ac7c0e 2024-04-16T19:54:27.185 DEBUG:botocore.hooks:Event request-created.iam.GetRole: calling handler 2024-04-16T19:54:27.185 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.187 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000041faab92d842da1e-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '773', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.188 DEBUG:botocore.parsers:Response body: b'tx0000041faab92d842da1e-00661ed773-5331-test-zone220b0a753-e2e9-4908-b051-bdefda557f17roles-yrjefh-2/arn:aws:iam::RGW11111111111111111:role/roles-yrjefh-22024-04-16T19:54:04.911Z3600{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam:::user/testuser"}, "Action": ["sts:AssumeRole"]}]}' 2024-04-16T19:54:27.188 DEBUG:botocore.hooks:Event needs-retry.iam.GetRole: calling handler 2024-04-16T19:54:27.188 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.188 DEBUG:botocore.hooks:Event after-call.iam.GetRole: calling handler 2024-04-16T19:54:27.189 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.189 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.189 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.189 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.189 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.190 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.190 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:27.190 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.190 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.190 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:27.191 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 6913435a3ef75ca2eada8aec24ffc7a0c30e6a45f3f801080871b4e2f5321d94 2024-04-16T19:54:27.191 DEBUG:botocore.auth:Signature: 912307add0216b2aec84a383294ddb31dafa2e675d7ec311f0e41f102d11b310 2024-04-16T19:54:27.191 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.191 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.193 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx000008b691a2514c4d83c-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.194 DEBUG:botocore.parsers:Response body: b'tx000008b691a2514c4d83c-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.194 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.194 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.194 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.194 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.195 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.195 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.195 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.195 DEBUG:botocore.hooks:Event before-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.195 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.195 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler > 2024-04-16T19:54:27.196 DEBUG:botocore.hooks:Event choose-signer.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.196 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.196 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 4e2b97cc38cffcbc8fda1caf69b53c6f91f319599b072c3463ca4ef818ca6420 2024-04-16T19:54:27.196 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request b5e89b1cbb60d69b65ce90d0cc68fcf6f3ebabd978a0eefb748737153e7ceb48 2024-04-16T19:54:27.196 DEBUG:botocore.auth:Signature: d4e663637f9fe525e1a32e4a921a44afbf0bee62a71451fe6fa6c6afef808ce0 2024-04-16T19:54:27.197 DEBUG:botocore.hooks:Event request-created.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.197 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.199 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000076aa943c972b90f3-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '239', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.199 DEBUG:botocore.parsers:Response body: b'tx0000076aa943c972b90f3-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.199 DEBUG:botocore.hooks:Event needs-retry.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.200 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.200 DEBUG:botocore.hooks:Event after-call.iam.ListRolePolicies: calling handler 2024-04-16T19:54:27.200 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.200 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.200 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.201 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.201 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.201 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.201 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:27.201 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.201 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.202 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:27.202 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 03fa172aea8c34ec90baae471b5be03157463152f18e9e1458520312a87299ba 2024-04-16T19:54:27.202 DEBUG:botocore.auth:Signature: 2fa487878d3bb2c3ddae8c2384753b775a8dd36c5acf8eff39e75989966d784b 2024-04-16T19:54:27.202 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.202 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.204 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx0000027a0a481a89e7de0-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.205 DEBUG:botocore.parsers:Response body: b'tx0000027a0a481a89e7de0-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.205 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.205 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.205 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.205 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.205 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.206 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.206 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.206 DEBUG:botocore.hooks:Event before-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.206 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListAttachedRolePolicies) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListAttachedRolePolicies', 'Version': '2010-05-08', 'RoleName': 'roles-yrjefh-2'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.206 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler > 2024-04-16T19:54:27.206 DEBUG:botocore.hooks:Event choose-signer.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.207 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.207 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date cd87c3ac13cb6d83c596a6b989868219cbfd850f483bf246c5e24f85fcbd0361 2024-04-16T19:54:27.207 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request f2c0266ae1430c339f12e1dab0f6ac25809f78809719db5447effedd2c0d5d8f 2024-04-16T19:54:27.207 DEBUG:botocore.auth:Signature: 649ca65969443ea956a0f794867f0a620800d46cf67377fb81ca08d62c70cfff 2024-04-16T19:54:27.207 DEBUG:botocore.hooks:Event request-created.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.207 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.254 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000b53a88ca56edf11e-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '331', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.255 DEBUG:botocore.parsers:Response body: b'tx00000b53a88ca56edf11e-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.255 DEBUG:botocore.hooks:Event needs-retry.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.255 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.255 DEBUG:botocore.hooks:Event after-call.iam.ListAttachedRolePolicies: calling handler 2024-04-16T19:54:27.256 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUsers: calling handler 2024-04-16T19:54:27.256 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.256 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.256 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.256 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.257 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUsers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUsers', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.257 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler > 2024-04-16T19:54:27.257 DEBUG:botocore.hooks:Event choose-signer.iam.ListUsers: calling handler 2024-04-16T19:54:27.257 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.257 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2 2024-04-16T19:54:27.258 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 25fea33c8a4338b8ad927cc71909fad2b57be57ceac37d3ebd9c2d6ab14a0d73 2024-04-16T19:54:27.258 DEBUG:botocore.auth:Signature: f5c5682ea34f603273caddc147e00315358333d445c8ff80765b30a76c6091c8 2024-04-16T19:54:27.258 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler 2024-04-16T19:54:27.258 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.261 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000c668d8f60157f51d-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.261 DEBUG:botocore.parsers:Response body: b'false' 2024-04-16T19:54:27.262 DEBUG:botocore.hooks:Event needs-retry.iam.ListUsers: calling handler 2024-04-16T19:54:27.262 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.262 DEBUG:botocore.hooks:Event after-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.262 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListUsers: calling handler 2024-04-16T19:54:27.263 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.263 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.263 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.263 DEBUG:botocore.hooks:Event before-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.263 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListUsers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListUsers', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.264 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler > 2024-04-16T19:54:27.264 DEBUG:botocore.hooks:Event choose-signer.iam.ListUsers: calling handler 2024-04-16T19:54:27.264 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.264 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2 2024-04-16T19:54:27.264 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request d3cbc2f08d425aa1fec39629c762fc2060ecc0bb3a87542778eb79919e612191 2024-04-16T19:54:27.264 DEBUG:botocore.auth:Signature: 7210773992aa3b238675c2dd46dfbd3e978b9ab6808828fa0a9b77e0fc69ca80 2024-04-16T19:54:27.265 DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler 2024-04-16T19:54:27.265 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.267 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000659da4ab8ad70c8c-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.268 DEBUG:botocore.parsers:Response body: b'false' 2024-04-16T19:54:27.268 DEBUG:botocore.hooks:Event needs-retry.iam.ListUsers: calling handler 2024-04-16T19:54:27.268 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.268 DEBUG:botocore.hooks:Event after-call.iam.ListUsers: calling handler 2024-04-16T19:54:27.269 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroups: calling handler 2024-04-16T19:54:27.269 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.269 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.269 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.269 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.269 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroups', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.270 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler > 2024-04-16T19:54:27.270 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroups: calling handler 2024-04-16T19:54:27.270 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.270 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 5f776d91509b9c99b8cb5eb5d6d4a787a33ae41c8cd6e7b69effca69080e1e1f 2024-04-16T19:54:27.270 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 528fdae558979bcec57416905dbe69f5626f195546151a4a674876a0f8b66a84 2024-04-16T19:54:27.270 DEBUG:botocore.auth:Signature: 35c7c1ca8c4c2c742c95540e991eef23c45d13d8dddd3961cf7dd105719cef90 2024-04-16T19:54:27.271 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler 2024-04-16T19:54:27.271 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.273 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000e0bc8178ecc1cefc-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.274 DEBUG:botocore.parsers:Response body: b'false' 2024-04-16T19:54:27.274 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroups: calling handler 2024-04-16T19:54:27.274 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.274 DEBUG:botocore.hooks:Event after-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.274 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListGroups: calling handler 2024-04-16T19:54:27.275 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.275 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.275 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.275 DEBUG:botocore.hooks:Event before-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.275 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListGroups', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.275 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler > 2024-04-16T19:54:27.276 DEBUG:botocore.hooks:Event choose-signer.iam.ListGroups: calling handler 2024-04-16T19:54:27.276 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.276 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 5f776d91509b9c99b8cb5eb5d6d4a787a33ae41c8cd6e7b69effca69080e1e1f 2024-04-16T19:54:27.276 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request f4c02d3eb8dcd44fb2a416610e8e489da984153a8afb2b5411723f3263b15207 2024-04-16T19:54:27.276 DEBUG:botocore.auth:Signature: d2133b4736e4db86f46ce5fe99d0b28843c666436941ffa2af0ae453cfd5dd3a 2024-04-16T19:54:27.276 DEBUG:botocore.hooks:Event request-created.iam.ListGroups: calling handler 2024-04-16T19:54:27.277 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.279 DEBUG:botocore.parsers:Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx000001aa88af273d29e97-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.279 DEBUG:botocore.parsers:Response body: b'false' 2024-04-16T19:54:27.280 DEBUG:botocore.hooks:Event needs-retry.iam.ListGroups: calling handler 2024-04-16T19:54:27.280 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.280 DEBUG:botocore.hooks:Event after-call.iam.ListGroups: calling handler 2024-04-16T19:54:27.280 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.280 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.280 DEBUG:botocore.regions:Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.281 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.281 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.281 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListOpenIDConnectProviders) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListOpenIDConnectProviders', 'Version': '2010-05-08'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.281 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler > 2024-04-16T19:54:27.281 DEBUG:botocore.hooks:Event choose-signer.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.282 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.282 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi026.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 6be253d5bef7fe9cb4ed0d00d8a18351f59593888e86906ef784f147279adb10 2024-04-16T19:54:27.282 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 48ac4dc1dd40c710d7308103b2d64dec9409aa60bc288d39e073f67c8995e563 2024-04-16T19:54:27.282 DEBUG:botocore.auth:Signature: 4c1b4b609656504e7e5267fcdd8d355a51f91f193ad6183089b337bbf1228c81 2024-04-16T19:54:27.282 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.282 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.297 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000d5b7e3fce0d5c9d7-00661ed773-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '357', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.297 DEBUG:botocore.parsers:Response body: b'tx00000d5b7e3fce0d5c9d7-00661ed773-4812-test-zone1' 2024-04-16T19:54:27.298 DEBUG:botocore.hooks:Event needs-retry.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.298 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.298 DEBUG:botocore.hooks:Event after-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.298 DEBUG:botocore.hooks:Event before-parameter-build.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.298 DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T19:54:27.298 DEBUG:botocore.regions:Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:27.299 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.299 DEBUG:botocore.hooks:Event before-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.299 DEBUG:botocore.endpoint:Making request for OperationModel(name=ListOpenIDConnectProviders) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListOpenIDConnectProviders', 'Version': '2010-05-08'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T19:54:27.299 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler > 2024-04-16T19:54:27.299 DEBUG:botocore.hooks:Event choose-signer.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.300 DEBUG:botocore.auth:Calculating signature using v4 auth. 2024-04-16T19:54:27.300 DEBUG:botocore.auth:CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:smithi119.front.sepia.ceph.com:8000 x-amz-date:20240416T195427Z content-type;host;x-amz-date 6be253d5bef7fe9cb4ed0d00d8a18351f59593888e86906ef784f147279adb10 2024-04-16T19:54:27.300 DEBUG:botocore.auth:StringToSign: AWS4-HMAC-SHA256 20240416T195427Z 20240416/test-zonegroup/iam/aws4_request 278e4d1c4f88bf25a6c5cd2f3318f009ca316af0c5e564dcc0fe7ee6fb6ec8ec 2024-04-16T19:54:27.300 DEBUG:botocore.auth:Signature: 14b50fac9696b279c43275e49b9d5aee3fe51ec5488a48642ee75000a340507d 2024-04-16T19:54:27.300 DEBUG:botocore.hooks:Event request-created.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.301 DEBUG:botocore.endpoint:Sending http request: 2024-04-16T19:54:27.354 DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'tx00000c2f22d6a4741b717-00661ed773-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '357', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T19:54:27.355 DEBUG:botocore.parsers:Response body: b'tx00000c2f22d6a4741b717-00661ed773-5331-test-zone2' 2024-04-16T19:54:27.355 DEBUG:botocore.hooks:Event needs-retry.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.355 DEBUG:botocore.retryhandler:No retry needed. 2024-04-16T19:54:27.355 DEBUG:botocore.hooks:Event after-call.iam.ListOpenIDConnectProviders: calling handler 2024-04-16T19:54:27.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_account_metadata_sync ... ok 2024-04-16T19:54:27.356 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-50 2024-04-16T19:54:27.357 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:27.357 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:27.357 DEBUG:boto:Method: PUT 2024-04-16T19:54:27.357 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:27.357 DEBUG:boto:Data: 2024-04-16T19:54:27.357 DEBUG:boto:Headers: {} 2024-04-16T19:54:27.357 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.357 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.358 DEBUG:boto:Params: {} 2024-04-16T19:54:27.358 DEBUG:boto:Token: None 2024-04-16T19:54:27.358 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/ 2024-04-16T19:54:27.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D7KBXfPn9Efi4zAWcMj0t7gzF8= 2024-04-16T19:54:27.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D7KBXfPn9Efi4zAWcMj0t7gzF8=', 'Content-Length': '0'} 2024-04-16T19:54:27.379 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b26772a881962264-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.379 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:27.380 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:27.380 DEBUG:boto:Method: HEAD 2024-04-16T19:54:27.380 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:27.380 DEBUG:boto:Data: 2024-04-16T19:54:27.380 DEBUG:boto:Headers: {} 2024-04-16T19:54:27.380 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.380 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.380 DEBUG:boto:Params: {} 2024-04-16T19:54:27.380 DEBUG:boto:Token: None 2024-04-16T19:54:27.380 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/ 2024-04-16T19:54:27.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T19:54:27.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T19:54:27.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f83694e3ae0b2b3-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.383 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:27.384 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:27.384 DEBUG:boto:Method: PUT 2024-04-16T19:54:27.384 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:54:27.384 DEBUG:boto:Data: 2024-04-16T19:54:27.384 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:27.384 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.384 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.384 DEBUG:boto:Params: {} 2024-04-16T19:54:27.384 DEBUG:boto:Token: None 2024-04-16T19:54:27.384 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/a 2024-04-16T19:54:27.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OawIn4ucckKNFHeCdrJon1ciU+Q= 2024-04-16T19:54:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OawIn4ucckKNFHeCdrJon1ciU+Q='} 2024-04-16T19:54:27.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000476b57e09300983f-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.392 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:27.392 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:27.392 DEBUG:boto:Method: HEAD 2024-04-16T19:54:27.392 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:27.392 DEBUG:boto:Data: 2024-04-16T19:54:27.392 DEBUG:boto:Headers: {} 2024-04-16T19:54:27.392 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.392 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.392 DEBUG:boto:Params: {} 2024-04-16T19:54:27.393 DEBUG:boto:Token: None 2024-04-16T19:54:27.393 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/ 2024-04-16T19:54:27.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T19:54:27.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T19:54:27.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b07081aa92ee8a6-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.395 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:27.395 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:27.396 DEBUG:boto:Method: PUT 2024-04-16T19:54:27.396 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:54:27.396 DEBUG:boto:Data: 2024-04-16T19:54:27.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:27.396 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.396 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.396 DEBUG:boto:Params: {} 2024-04-16T19:54:27.396 DEBUG:boto:Token: None 2024-04-16T19:54:27.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/b 2024-04-16T19:54:27.397 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tW/jLtJqfJ/5LMBEwax+H6Uj+mE= 2024-04-16T19:54:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tW/jLtJqfJ/5LMBEwax+H6Uj+mE='} 2024-04-16T19:54:27.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea1a48a51f046db-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.439 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:27.439 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:27.439 DEBUG:boto:Method: HEAD 2024-04-16T19:54:27.439 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:27.439 DEBUG:boto:Data: 2024-04-16T19:54:27.439 DEBUG:boto:Headers: {} 2024-04-16T19:54:27.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.439 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.439 DEBUG:boto:Params: {} 2024-04-16T19:54:27.440 DEBUG:boto:Token: None 2024-04-16T19:54:27.440 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/ 2024-04-16T19:54:27.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T19:54:27.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T19:54:27.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006745302b41a79f99-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.443 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:27.443 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:27.443 DEBUG:boto:Method: PUT 2024-04-16T19:54:27.443 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:54:27.443 DEBUG:boto:Data: 2024-04-16T19:54:27.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:27.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.443 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.443 DEBUG:boto:Params: {} 2024-04-16T19:54:27.444 DEBUG:boto:Token: None 2024-04-16T19:54:27.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/c 2024-04-16T19:54:27.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T2JmAG52tRnr2/Yhg7thY7GkxE4= 2024-04-16T19:54:27.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T2JmAG52tRnr2/Yhg7thY7GkxE4='} 2024-04-16T19:54:27.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e2e0bf6a96fef8a-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.450 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:27.451 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:27.451 DEBUG:boto:Method: HEAD 2024-04-16T19:54:27.451 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:27.451 DEBUG:boto:Data: 2024-04-16T19:54:27.451 DEBUG:boto:Headers: {} 2024-04-16T19:54:27.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.451 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.451 DEBUG:boto:Params: {} 2024-04-16T19:54:27.451 DEBUG:boto:Token: None 2024-04-16T19:54:27.452 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/ 2024-04-16T19:54:27.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T19:54:27.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T19:54:27.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b7eccb6f585757-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.454 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:27.454 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:27.454 DEBUG:boto:Method: PUT 2024-04-16T19:54:27.454 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:54:27.455 DEBUG:boto:Data: 2024-04-16T19:54:27.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:27.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:27.455 DEBUG:boto:Port: 8000 2024-04-16T19:54:27.455 DEBUG:boto:Params: {} 2024-04-16T19:54:27.455 DEBUG:boto:Token: None 2024-04-16T19:54:27.455 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:27 GMT /yrjefh-50/d 2024-04-16T19:54:27.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ovhsf3NSedHasPFFrVL6oe/vPxE= 2024-04-16T19:54:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ovhsf3NSedHasPFFrVL6oe/vPxE='} 2024-04-16T19:54:27.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3fbb02a31ea2620-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:27.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:27.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:27.499 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:54:27.628 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713297267.375662_676.1","last_update":"2024-04-16T19:54:27.375662Z"}] 2024-04-16T19:54:27.628 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T19:54:27.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:54:27.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:27.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:27.629 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:27.679 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:27.679 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:27.728+0000 7f7cac42e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:27.730+0000 7f7cac42e800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:54:27.753 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297267.375662_676.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:27.375662Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T19:54:27.753 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297267.375662_676.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:27.375662Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T19:54:27.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297267.375662_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:27.375662Z', 'realm_epoch': 2}}] 2024-04-16T19:54:27.754 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T19:54:27.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T19:54:27.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:54:27.754 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T19:54:27.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:27.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:27.755 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:54:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:27.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:27.870+0000 7f3ab7f49800 1 waiting to reach incremental sync.. 2024-04-16T19:54:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:32.872+0000 7f3ab7f49800 1 bucket sync caught up with source: 2024-04-16T19:54:32.875 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.33.6, , , , 00000000001.22.6, , , 00000000001.65.6, , 00000000001.30.6] 2024-04-16T19:54:32.875 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, 00000000001.33.6, , , , 00000000001.22.6, , , 00000000001.65.6, , 00000000001.30.6] 2024-04-16T19:54:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:32.872+0000 7f3ab7f49800 0 bucket checkpoint complete 2024-04-16T19:54:32.900 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T19:54:32.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:32.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:32.901 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:54:32.964 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:32.964 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:54:33.022+0000 7f6ab8c33800 1 bucket sync caught up with source: 2024-04-16T19:54:33.024 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.158.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000001.101.6] 2024-04-16T19:54:33.024 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.158.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000001.101.6] 2024-04-16T19:54:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:54:33.022+0000 7f6ab8c33800 0 bucket checkpoint complete 2024-04-16T19:54:33.046 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:54:33.046 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.047 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.047 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.047 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:33.047 DEBUG:boto:Data: 2024-04-16T19:54:33.047 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.047 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.047 DEBUG:boto:Params: {} 2024-04-16T19:54:33.047 DEBUG:boto:Token: None 2024-04-16T19:54:33.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo= 2024-04-16T19:54:33.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo=', 'Content-Length': '0'} 2024-04-16T19:54:33.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8421e9bd8ca1b68-00661ed779-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.051 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.051 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.051 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.051 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:33.051 DEBUG:boto:Data: 2024-04-16T19:54:33.051 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.051 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.052 DEBUG:boto:Params: {} 2024-04-16T19:54:33.052 DEBUG:boto:Token: None 2024-04-16T19:54:33.052 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo= 2024-04-16T19:54:33.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo=', 'Content-Length': '0'} 2024-04-16T19:54:33.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd921ecc120362d9-00661ed779-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.054 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:54:33.055 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.055 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.055 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.055 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.055 DEBUG:boto:Method: GET 2024-04-16T19:54:33.055 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.055 DEBUG:boto:Data: 2024-04-16T19:54:33.055 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.055 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.055 DEBUG:boto:Params: {} 2024-04-16T19:54:33.056 DEBUG:boto:Token: None 2024-04-16T19:54:33.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/?versions 2024-04-16T19:54:33.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T19:54:33.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T19:54:33.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e06f0e23b47f69c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.059 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:54:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:54:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:54:33.060 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:54:33.060 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:54:33.060 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.060 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.060 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.060 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.060 DEBUG:boto:Method: GET 2024-04-16T19:54:33.060 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.060 DEBUG:boto:Data: 2024-04-16T19:54:33.060 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.061 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.061 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.061 DEBUG:boto:Params: {} 2024-04-16T19:54:33.061 DEBUG:boto:Token: None 2024-04-16T19:54:33.061 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/?versions 2024-04-16T19:54:33.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T19:54:33.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T19:54:33.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d663544bccdcae5d-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.063 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.064 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:54:33.064 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:54:33.064 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:54:33.064 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:54:33.064 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.064 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.064 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.064 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.064 DEBUG:boto:Method: GET 2024-04-16T19:54:33.065 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.065 DEBUG:boto:Data: 2024-04-16T19:54:33.065 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.065 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.065 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.065 DEBUG:boto:Params: {} 2024-04-16T19:54:33.065 DEBUG:boto:Token: None 2024-04-16T19:54:33.065 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/?versions 2024-04-16T19:54:33.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T19:54:33.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T19:54:33.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78d5b9d44c08b6f-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.068 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.068 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.068 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.069 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.069 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.069 DEBUG:boto:Method: GET 2024-04-16T19:54:33.069 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:54:33.069 DEBUG:boto:Data: 2024-04-16T19:54:33.069 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.069 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.069 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.069 DEBUG:boto:Params: {} 2024-04-16T19:54:33.069 DEBUG:boto:Token: None 2024-04-16T19:54:33.070 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/?versions 2024-04-16T19:54:33.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T19:54:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T19:54:33.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b92fe7790b50290-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.072 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:33.073 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.073 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.073 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.073 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.073 DEBUG:boto:Method: GET 2024-04-16T19:54:33.073 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.073 DEBUG:boto:Data: 2024-04-16T19:54:33.073 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.073 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.074 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.074 DEBUG:boto:Params: {} 2024-04-16T19:54:33.074 DEBUG:boto:Token: None 2024-04-16T19:54:33.074 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T19:54:33.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T19:54:33.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056950200471a350d-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.118 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.119 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.119 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.119 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.119 DEBUG:boto:Method: GET 2024-04-16T19:54:33.119 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.119 DEBUG:boto:Data: 2024-04-16T19:54:33.119 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.119 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.119 DEBUG:boto:Params: {} 2024-04-16T19:54:33.120 DEBUG:boto:Token: None 2024-04-16T19:54:33.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T19:54:33.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T19:54:33.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da628d660edb81fa-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.122 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.122 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.122 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.122 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.122 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.122 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.123 DEBUG:boto:Data: 2024-04-16T19:54:33.123 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.123 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.123 DEBUG:boto:Params: {} 2024-04-16T19:54:33.123 DEBUG:boto:Token: None 2024-04-16T19:54:33.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.123 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME= 2024-04-16T19:54:33.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME=', 'Content-Length': '0'} 2024-04-16T19:54:33.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c486108ae93b019a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.126 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.126 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.126 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.126 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.126 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.126 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.126 DEBUG:boto:Data: 2024-04-16T19:54:33.126 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.126 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.127 DEBUG:boto:Params: {} 2024-04-16T19:54:33.127 DEBUG:boto:Token: None 2024-04-16T19:54:33.127 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME= 2024-04-16T19:54:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME=', 'Content-Length': '0'} 2024-04-16T19:54:33.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b38e33f5abd8934-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:33.129 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.129 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.129 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.129 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.129 DEBUG:boto:Method: GET 2024-04-16T19:54:33.129 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.130 DEBUG:boto:Data: 2024-04-16T19:54:33.130 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.130 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.130 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.130 DEBUG:boto:Params: {} 2024-04-16T19:54:33.130 DEBUG:boto:Token: None 2024-04-16T19:54:33.130 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T19:54:33.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T19:54:33.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc21e34e595afb8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.133 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.133 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.133 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.133 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:54:33.133 DEBUG:boto:Method: GET 2024-04-16T19:54:33.133 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.133 DEBUG:boto:Data: 2024-04-16T19:54:33.134 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.134 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.134 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.134 DEBUG:boto:Params: {} 2024-04-16T19:54:33.134 DEBUG:boto:Token: None 2024-04-16T19:54:33.134 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:54:33.134 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T19:54:33.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T19:54:33.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601812f818ac0e7f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.136 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.136 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.137 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.137 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.137 DEBUG:boto:Method: GET 2024-04-16T19:54:33.137 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:54:33.137 DEBUG:boto:Data: 2024-04-16T19:54:33.137 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.137 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.137 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.137 DEBUG:boto:Params: {} 2024-04-16T19:54:33.138 DEBUG:boto:Token: None 2024-04-16T19:54:33.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.138 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d7d766319d9311a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.141 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.141 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.141 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.141 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.141 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.141 DEBUG:boto:Method: GET 2024-04-16T19:54:33.141 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:54:33.142 DEBUG:boto:Data: 2024-04-16T19:54:33.142 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.142 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.142 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.142 DEBUG:boto:Params: {} 2024-04-16T19:54:33.142 DEBUG:boto:Token: None 2024-04-16T19:54:33.142 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d678d331d64ebe-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.145 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.145 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.145 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.145 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.145 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:54:33.146 DEBUG:boto:Data: 2024-04-16T19:54:33.146 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.146 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.146 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.146 DEBUG:boto:Params: {} 2024-04-16T19:54:33.146 DEBUG:boto:Token: None 2024-04-16T19:54:33.146 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a 2024-04-16T19:54:33.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc= 2024-04-16T19:54:33.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc=', 'Content-Length': '0'} 2024-04-16T19:54:33.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4a93730105db4e-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.149 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.149 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.149 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.149 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:54:33.149 DEBUG:boto:Data: 2024-04-16T19:54:33.149 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.149 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.149 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.149 DEBUG:boto:Params: {} 2024-04-16T19:54:33.150 DEBUG:boto:Token: None 2024-04-16T19:54:33.150 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a 2024-04-16T19:54:33.150 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc= 2024-04-16T19:54:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc=', 'Content-Length': '0'} 2024-04-16T19:54:33.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffd10d7fec5a3f89-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:54:33.152 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.152 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.152 DEBUG:boto:Method: GET 2024-04-16T19:54:33.152 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:54:33.152 DEBUG:boto:Data: 2024-04-16T19:54:33.153 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.153 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.153 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.153 DEBUG:boto:Params: {} 2024-04-16T19:54:33.153 DEBUG:boto:Token: None 2024-04-16T19:54:33.153 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a 2024-04-16T19:54:33.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w= 2024-04-16T19:54:33.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w=', 'Content-Length': '0'} 2024-04-16T19:54:33.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921c4a99365c3ca3-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.202 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:54:33.203 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:54:33.203 DEBUG:boto:Method: GET 2024-04-16T19:54:33.203 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:54:33.203 DEBUG:boto:Data: 2024-04-16T19:54:33.203 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.203 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.203 DEBUG:boto:Params: {} 2024-04-16T19:54:33.203 DEBUG:boto:Token: None 2024-04-16T19:54:33.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/a 2024-04-16T19:54:33.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w= 2024-04-16T19:54:33.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w=', 'Content-Length': '0'} 2024-04-16T19:54:33.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdc500994a5a6bd-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.206 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.206 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.206 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.206 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.206 DEBUG:boto:Method: GET 2024-04-16T19:54:33.207 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:54:33.207 DEBUG:boto:Data: 2024-04-16T19:54:33.207 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.207 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.207 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.207 DEBUG:boto:Params: {} 2024-04-16T19:54:33.207 DEBUG:boto:Token: None 2024-04-16T19:54:33.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023969a605a8bfea-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.210 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.211 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.211 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.211 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.211 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:54:33.211 DEBUG:boto:Method: GET 2024-04-16T19:54:33.211 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:54:33.211 DEBUG:boto:Data: 2024-04-16T19:54:33.211 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.212 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.212 DEBUG:boto:Params: {} 2024-04-16T19:54:33.212 DEBUG:boto:Token: None 2024-04-16T19:54:33.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003174bada875c88af-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.214 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:33.215 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.215 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.215 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.215 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.215 DEBUG:boto:Method: GET 2024-04-16T19:54:33.215 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.216 DEBUG:boto:Data: 2024-04-16T19:54:33.216 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.216 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.216 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.216 DEBUG:boto:Params: {} 2024-04-16T19:54:33.216 DEBUG:boto:Token: None 2024-04-16T19:54:33.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T19:54:33.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T19:54:33.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574c663ad4671e32-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.262 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.262 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.263 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.263 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.263 DEBUG:boto:Method: GET 2024-04-16T19:54:33.263 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.263 DEBUG:boto:Data: 2024-04-16T19:54:33.263 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.263 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.263 DEBUG:boto:Params: {} 2024-04-16T19:54:33.263 DEBUG:boto:Token: None 2024-04-16T19:54:33.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T19:54:33.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T19:54:33.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004924b8f7ef45b14f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.266 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.266 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.266 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.266 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.266 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.266 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.267 DEBUG:boto:Data: 2024-04-16T19:54:33.267 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.267 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.267 DEBUG:boto:Params: {} 2024-04-16T19:54:33.267 DEBUG:boto:Token: None 2024-04-16T19:54:33.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A= 2024-04-16T19:54:33.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A=', 'Content-Length': '0'} 2024-04-16T19:54:33.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8b22d7490ed44b-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.270 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.270 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.270 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.270 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.270 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.270 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.270 DEBUG:boto:Data: 2024-04-16T19:54:33.270 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.271 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.271 DEBUG:boto:Params: {} 2024-04-16T19:54:33.271 DEBUG:boto:Token: None 2024-04-16T19:54:33.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A= 2024-04-16T19:54:33.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A=', 'Content-Length': '0'} 2024-04-16T19:54:33.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca42f12cb8ec0b5-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:33.273 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.273 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.273 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.274 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.274 DEBUG:boto:Method: GET 2024-04-16T19:54:33.274 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.274 DEBUG:boto:Data: 2024-04-16T19:54:33.274 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.274 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.274 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.274 DEBUG:boto:Params: {} 2024-04-16T19:54:33.274 DEBUG:boto:Token: None 2024-04-16T19:54:33.274 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T19:54:33.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T19:54:33.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004851f5885856e216-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.318 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.318 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.318 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.319 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:54:33.319 DEBUG:boto:Method: GET 2024-04-16T19:54:33.319 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.319 DEBUG:boto:Data: 2024-04-16T19:54:33.319 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.319 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.319 DEBUG:boto:Params: {} 2024-04-16T19:54:33.319 DEBUG:boto:Token: None 2024-04-16T19:54:33.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:54:33.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T19:54:33.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T19:54:33.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9835b59b7b9ef6f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.322 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.322 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.322 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.323 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.323 DEBUG:boto:Method: GET 2024-04-16T19:54:33.323 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:54:33.323 DEBUG:boto:Data: 2024-04-16T19:54:33.323 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.323 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.323 DEBUG:boto:Params: {} 2024-04-16T19:54:33.323 DEBUG:boto:Token: None 2024-04-16T19:54:33.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000322352c69192794-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.326 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.327 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.327 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.327 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.327 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.327 DEBUG:boto:Method: GET 2024-04-16T19:54:33.327 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:54:33.327 DEBUG:boto:Data: 2024-04-16T19:54:33.327 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.327 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.327 DEBUG:boto:Params: {} 2024-04-16T19:54:33.328 DEBUG:boto:Token: None 2024-04-16T19:54:33.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f98bc2b8a9f52a-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.330 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.331 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.331 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.331 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.331 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:54:33.331 DEBUG:boto:Data: 2024-04-16T19:54:33.331 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.331 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.331 DEBUG:boto:Params: {} 2024-04-16T19:54:33.331 DEBUG:boto:Token: None 2024-04-16T19:54:33.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b 2024-04-16T19:54:33.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo= 2024-04-16T19:54:33.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo=', 'Content-Length': '0'} 2024-04-16T19:54:33.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcfa702747df718-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.334 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.334 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.334 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.334 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:54:33.334 DEBUG:boto:Data: 2024-04-16T19:54:33.334 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.335 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.335 DEBUG:boto:Params: {} 2024-04-16T19:54:33.335 DEBUG:boto:Token: None 2024-04-16T19:54:33.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b 2024-04-16T19:54:33.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo= 2024-04-16T19:54:33.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo=', 'Content-Length': '0'} 2024-04-16T19:54:33.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9aaacebe1d68209-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:54:33.337 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.337 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.337 DEBUG:boto:Method: GET 2024-04-16T19:54:33.337 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:54:33.338 DEBUG:boto:Data: 2024-04-16T19:54:33.338 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.338 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.338 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.338 DEBUG:boto:Params: {} 2024-04-16T19:54:33.338 DEBUG:boto:Token: None 2024-04-16T19:54:33.338 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b 2024-04-16T19:54:33.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4= 2024-04-16T19:54:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4=', 'Content-Length': '0'} 2024-04-16T19:54:33.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099987ec66ef6a1f8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.382 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:54:33.382 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:54:33.383 DEBUG:boto:Method: GET 2024-04-16T19:54:33.383 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:54:33.383 DEBUG:boto:Data: 2024-04-16T19:54:33.383 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.383 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.383 DEBUG:boto:Params: {} 2024-04-16T19:54:33.383 DEBUG:boto:Token: None 2024-04-16T19:54:33.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/b 2024-04-16T19:54:33.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4= 2024-04-16T19:54:33.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4=', 'Content-Length': '0'} 2024-04-16T19:54:33.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008432ce83365f0e02-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.385 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.386 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.386 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.386 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.386 DEBUG:boto:Method: GET 2024-04-16T19:54:33.386 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:54:33.386 DEBUG:boto:Data: 2024-04-16T19:54:33.386 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.386 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.386 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.386 DEBUG:boto:Params: {} 2024-04-16T19:54:33.387 DEBUG:boto:Token: None 2024-04-16T19:54:33.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051af450a4cffaeeb-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.390 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.390 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.390 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.390 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.391 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:54:33.391 DEBUG:boto:Method: GET 2024-04-16T19:54:33.391 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:54:33.391 DEBUG:boto:Data: 2024-04-16T19:54:33.391 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.391 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.391 DEBUG:boto:Params: {} 2024-04-16T19:54:33.391 DEBUG:boto:Token: None 2024-04-16T19:54:33.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019fb149afe21f0f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.394 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:33.394 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.394 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.394 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.395 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.395 DEBUG:boto:Method: GET 2024-04-16T19:54:33.395 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.395 DEBUG:boto:Data: 2024-04-16T19:54:33.395 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.395 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.395 DEBUG:boto:Params: {} 2024-04-16T19:54:33.395 DEBUG:boto:Token: None 2024-04-16T19:54:33.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T19:54:33.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T19:54:33.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5c737781d1f2f8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.438 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.438 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.438 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.439 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.439 DEBUG:boto:Method: GET 2024-04-16T19:54:33.439 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.439 DEBUG:boto:Data: 2024-04-16T19:54:33.439 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.439 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.439 DEBUG:boto:Params: {} 2024-04-16T19:54:33.439 DEBUG:boto:Token: None 2024-04-16T19:54:33.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T19:54:33.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T19:54:33.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd796c95e7e24bc-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.441 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.442 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.442 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.442 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.442 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.442 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.442 DEBUG:boto:Data: 2024-04-16T19:54:33.442 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.442 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.442 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.443 DEBUG:boto:Params: {} 2024-04-16T19:54:33.443 DEBUG:boto:Token: None 2024-04-16T19:54:33.443 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg= 2024-04-16T19:54:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg=', 'Content-Length': '0'} 2024-04-16T19:54:33.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005750d9b81cc97758-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.445 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.445 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.445 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.445 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.445 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.445 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.445 DEBUG:boto:Data: 2024-04-16T19:54:33.446 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.446 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.446 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.446 DEBUG:boto:Params: {} 2024-04-16T19:54:33.446 DEBUG:boto:Token: None 2024-04-16T19:54:33.446 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.446 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg= 2024-04-16T19:54:33.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg=', 'Content-Length': '0'} 2024-04-16T19:54:33.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099006f6e732829ec-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:33.448 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.448 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.448 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.448 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.448 DEBUG:boto:Method: GET 2024-04-16T19:54:33.449 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.449 DEBUG:boto:Data: 2024-04-16T19:54:33.449 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.449 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.449 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.449 DEBUG:boto:Params: {} 2024-04-16T19:54:33.449 DEBUG:boto:Token: None 2024-04-16T19:54:33.449 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.449 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T19:54:33.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T19:54:33.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001834e2fe0f71f1e6-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.494 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.494 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.495 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.495 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:54:33.495 DEBUG:boto:Method: GET 2024-04-16T19:54:33.495 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.495 DEBUG:boto:Data: 2024-04-16T19:54:33.495 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.495 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.495 DEBUG:boto:Params: {} 2024-04-16T19:54:33.495 DEBUG:boto:Token: None 2024-04-16T19:54:33.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:54:33.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T19:54:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T19:54:33.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6cdd95aadce5c7-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.498 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.498 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.498 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.498 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.498 DEBUG:boto:Method: GET 2024-04-16T19:54:33.499 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:54:33.499 DEBUG:boto:Data: 2024-04-16T19:54:33.499 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.499 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.499 DEBUG:boto:Params: {} 2024-04-16T19:54:33.499 DEBUG:boto:Token: None 2024-04-16T19:54:33.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c4cf763b0d92497-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.542 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.543 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.543 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.543 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.543 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.543 DEBUG:boto:Method: GET 2024-04-16T19:54:33.543 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:54:33.543 DEBUG:boto:Data: 2024-04-16T19:54:33.543 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.543 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.543 DEBUG:boto:Params: {} 2024-04-16T19:54:33.544 DEBUG:boto:Token: None 2024-04-16T19:54:33.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aed17b9fbaf63cf-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.546 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.547 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.547 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.547 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.547 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:54:33.547 DEBUG:boto:Data: 2024-04-16T19:54:33.547 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.547 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.547 DEBUG:boto:Params: {} 2024-04-16T19:54:33.547 DEBUG:boto:Token: None 2024-04-16T19:54:33.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c 2024-04-16T19:54:33.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE= 2024-04-16T19:54:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE=', 'Content-Length': '0'} 2024-04-16T19:54:33.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035204177dcd40e7a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.550 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.550 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.550 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.551 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:54:33.551 DEBUG:boto:Data: 2024-04-16T19:54:33.551 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.551 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.551 DEBUG:boto:Params: {} 2024-04-16T19:54:33.551 DEBUG:boto:Token: None 2024-04-16T19:54:33.551 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c 2024-04-16T19:54:33.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE= 2024-04-16T19:54:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE=', 'Content-Length': '0'} 2024-04-16T19:54:33.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8875cf9fbb78852-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:54:33.553 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.553 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.554 DEBUG:boto:Method: GET 2024-04-16T19:54:33.554 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:54:33.554 DEBUG:boto:Data: 2024-04-16T19:54:33.554 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.554 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.554 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.554 DEBUG:boto:Params: {} 2024-04-16T19:54:33.554 DEBUG:boto:Token: None 2024-04-16T19:54:33.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c 2024-04-16T19:54:33.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY= 2024-04-16T19:54:33.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY=', 'Content-Length': '0'} 2024-04-16T19:54:33.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8794e8546417b71-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.598 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:54:33.598 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:54:33.599 DEBUG:boto:Method: GET 2024-04-16T19:54:33.599 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:54:33.599 DEBUG:boto:Data: 2024-04-16T19:54:33.599 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.599 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.599 DEBUG:boto:Params: {} 2024-04-16T19:54:33.599 DEBUG:boto:Token: None 2024-04-16T19:54:33.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/c 2024-04-16T19:54:33.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY= 2024-04-16T19:54:33.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY=', 'Content-Length': '0'} 2024-04-16T19:54:33.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008902efea9ba0b15d-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.602 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.602 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.602 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.602 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.602 DEBUG:boto:Method: GET 2024-04-16T19:54:33.602 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:54:33.602 DEBUG:boto:Data: 2024-04-16T19:54:33.603 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.603 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.603 DEBUG:boto:Params: {} 2024-04-16T19:54:33.603 DEBUG:boto:Token: None 2024-04-16T19:54:33.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c5650345c37595-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.606 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.606 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.607 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.607 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.607 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:54:33.607 DEBUG:boto:Method: GET 2024-04-16T19:54:33.607 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:54:33.607 DEBUG:boto:Data: 2024-04-16T19:54:33.607 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.607 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.607 DEBUG:boto:Params: {} 2024-04-16T19:54:33.608 DEBUG:boto:Token: None 2024-04-16T19:54:33.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0e84e41635d187-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.610 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:33.611 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.611 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.611 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.611 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.611 DEBUG:boto:Method: GET 2024-04-16T19:54:33.611 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.611 DEBUG:boto:Data: 2024-04-16T19:54:33.611 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.611 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.611 DEBUG:boto:Params: {} 2024-04-16T19:54:33.612 DEBUG:boto:Token: None 2024-04-16T19:54:33.612 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T19:54:33.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T19:54:33.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3717e3532674557-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.654 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.655 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.655 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.655 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.655 DEBUG:boto:Method: GET 2024-04-16T19:54:33.655 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.655 DEBUG:boto:Data: 2024-04-16T19:54:33.655 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.655 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.655 DEBUG:boto:Params: {} 2024-04-16T19:54:33.655 DEBUG:boto:Token: None 2024-04-16T19:54:33.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T19:54:33.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T19:54:33.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662191c5b34de497-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.658 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.658 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.658 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.658 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.658 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.659 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.659 DEBUG:boto:Data: 2024-04-16T19:54:33.659 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.659 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.659 DEBUG:boto:Params: {} 2024-04-16T19:54:33.659 DEBUG:boto:Token: None 2024-04-16T19:54:33.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs= 2024-04-16T19:54:33.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs=', 'Content-Length': '0'} 2024-04-16T19:54:33.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e534b3f43a012c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.662 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.662 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.662 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.662 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.663 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.663 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.663 DEBUG:boto:Data: 2024-04-16T19:54:33.663 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.663 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.663 DEBUG:boto:Params: {} 2024-04-16T19:54:33.663 DEBUG:boto:Token: None 2024-04-16T19:54:33.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs= 2024-04-16T19:54:33.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs=', 'Content-Length': '0'} 2024-04-16T19:54:33.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a660dc23abf1e5-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:33.665 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.666 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.666 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.666 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.666 DEBUG:boto:Method: GET 2024-04-16T19:54:33.666 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.666 DEBUG:boto:Data: 2024-04-16T19:54:33.666 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.666 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.666 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.666 DEBUG:boto:Params: {} 2024-04-16T19:54:33.667 DEBUG:boto:Token: None 2024-04-16T19:54:33.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T19:54:33.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T19:54:33.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970aab111ea18c62-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.710 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.710 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.710 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.711 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:54:33.711 DEBUG:boto:Method: GET 2024-04-16T19:54:33.711 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.711 DEBUG:boto:Data: 2024-04-16T19:54:33.711 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.711 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.711 DEBUG:boto:Params: {} 2024-04-16T19:54:33.711 DEBUG:boto:Token: None 2024-04-16T19:54:33.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:54:33.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T19:54:33.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T19:54:33.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ee202a274b83ba-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.714 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.714 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.714 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.714 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.715 DEBUG:boto:Method: GET 2024-04-16T19:54:33.715 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:54:33.715 DEBUG:boto:Data: 2024-04-16T19:54:33.715 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.715 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.715 DEBUG:boto:Params: {} 2024-04-16T19:54:33.715 DEBUG:boto:Token: None 2024-04-16T19:54:33.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4283f584c0fb121-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.718 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.718 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.718 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.718 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.718 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.719 DEBUG:boto:Method: GET 2024-04-16T19:54:33.719 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:54:33.719 DEBUG:boto:Data: 2024-04-16T19:54:33.719 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.719 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.719 DEBUG:boto:Params: {} 2024-04-16T19:54:33.719 DEBUG:boto:Token: None 2024-04-16T19:54:33.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f034391ea856e6d6-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.722 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.722 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.722 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.722 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.722 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:54:33.722 DEBUG:boto:Data: 2024-04-16T19:54:33.722 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.723 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.723 DEBUG:boto:Params: {} 2024-04-16T19:54:33.723 DEBUG:boto:Token: None 2024-04-16T19:54:33.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d 2024-04-16T19:54:33.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g= 2024-04-16T19:54:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g=', 'Content-Length': '0'} 2024-04-16T19:54:33.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d9acabd24bbf1c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.725 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.726 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.726 DEBUG:boto:Method: HEAD 2024-04-16T19:54:33.726 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:54:33.726 DEBUG:boto:Data: 2024-04-16T19:54:33.726 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.726 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.726 DEBUG:boto:Params: {} 2024-04-16T19:54:33.726 DEBUG:boto:Token: None 2024-04-16T19:54:33.727 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d 2024-04-16T19:54:33.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g= 2024-04-16T19:54:33.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g=', 'Content-Length': '0'} 2024-04-16T19:54:33.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6c210b002c8bcf-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:54:33.729 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.729 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.729 DEBUG:boto:Method: GET 2024-04-16T19:54:33.729 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:54:33.729 DEBUG:boto:Data: 2024-04-16T19:54:33.730 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.730 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.730 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.730 DEBUG:boto:Params: {} 2024-04-16T19:54:33.730 DEBUG:boto:Token: None 2024-04-16T19:54:33.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d 2024-04-16T19:54:33.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII= 2024-04-16T19:54:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII=', 'Content-Length': '0'} 2024-04-16T19:54:33.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba85475a1e43de9-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.774 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:54:33.775 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:54:33.775 DEBUG:boto:Method: GET 2024-04-16T19:54:33.775 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:54:33.775 DEBUG:boto:Data: 2024-04-16T19:54:33.775 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.775 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.775 DEBUG:boto:Params: {} 2024-04-16T19:54:33.775 DEBUG:boto:Token: None 2024-04-16T19:54:33.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/d 2024-04-16T19:54:33.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII= 2024-04-16T19:54:33.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII=', 'Content-Length': '0'} 2024-04-16T19:54:33.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef9397d1c93013-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.778 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.778 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.778 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.778 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.778 DEBUG:boto:Method: GET 2024-04-16T19:54:33.778 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:54:33.778 DEBUG:boto:Data: 2024-04-16T19:54:33.778 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:54:33.779 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.779 DEBUG:boto:Params: {} 2024-04-16T19:54:33.779 DEBUG:boto:Token: None 2024-04-16T19:54:33.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a98dc64e8ffb43-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.781 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.782 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:33.782 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:33.782 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.782 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:54:33.782 DEBUG:boto:Method: GET 2024-04-16T19:54:33.782 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:54:33.782 DEBUG:boto:Data: 2024-04-16T19:54:33.782 DEBUG:boto:Headers: {} 2024-04-16T19:54:33.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:33.783 DEBUG:boto:Port: 8000 2024-04-16T19:54:33.783 DEBUG:boto:Params: {} 2024-04-16T19:54:33.783 DEBUG:boto:Token: None 2024-04-16T19:54:33.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:54:33 GMT /yrjefh-50/ 2024-04-16T19:54:33.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T19:54:33.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T19:54:33.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be18496560aab6f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:33.786 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:54:33.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:54:33.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:33.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:33.786 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:54:33.838 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:33.838 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:33.916 INFO:teuthology.orchestra.run.smithi026.stdout:{"layout":{"resharding":"None","current_index":{"gen":0,"layout":{"type":"Normal","normal":{"num_shards":11,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}}]}} 2024-04-16T19:54:33.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:33.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:33.916 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-50 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:34.038 INFO:teuthology.orchestra.run.smithi026.stdout:[{"op_id":"1#00000000001.33.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5146303621251504191","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.385768586Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"5#00000000001.22.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16427917761036559904","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.456540920Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"8#00000000001.65.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5633453337919614858","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.444947649Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"10#00000000001.30.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18348127270871254747","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.397899049Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]}] 2024-04-16T19:54:34.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:34.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:34.038 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-50 --num-shards 13 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:54:34.085 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:34.085 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:34.147 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:54:34.147 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-50 2024-04-16T19:54:34.151 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 4 2024-04-16T19:54:34.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:34.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:54:34.198 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:54:34.243 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:34.243 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:34.314 INFO:teuthology.orchestra.run.smithi026.stdout:{"layout":{"resharding":"None","current_index":{"gen":1,"layout":{"type":"Normal","normal":{"num_shards":13,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}}]}} 2024-04-16T19:54:34.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:34.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:54:34.315 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-50 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:54:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:34.474 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2024-04-16T19:54:34.475 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:34.475 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:34.475 DEBUG:boto:Method: HEAD 2024-04-16T19:54:34.475 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:34.475 DEBUG:boto:Data: 2024-04-16T19:54:34.475 DEBUG:boto:Headers: {} 2024-04-16T19:54:34.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.475 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.476 DEBUG:boto:Params: {} 2024-04-16T19:54:34.476 DEBUG:boto:Token: None 2024-04-16T19:54:34.476 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/ 2024-04-16T19:54:34.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T19:54:34.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T19:54:34.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d4421f1d3120c37-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.479 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:54:34.479 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:54:34.479 DEBUG:boto:Method: PUT 2024-04-16T19:54:34.480 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:54:34.480 DEBUG:boto:Data: 2024-04-16T19:54:34.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:34.480 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.480 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.480 DEBUG:boto:Params: {} 2024-04-16T19:54:34.480 DEBUG:boto:Token: None 2024-04-16T19:54:34.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/e 2024-04-16T19:54:34.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TYHKDcCS6fzNYecgHITUYmxYSvI= 2024-04-16T19:54:34.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TYHKDcCS6fzNYecgHITUYmxYSvI='} 2024-04-16T19:54:34.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005455716a19bd63bd-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.486 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:34.486 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:34.487 DEBUG:boto:Method: HEAD 2024-04-16T19:54:34.487 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:34.487 DEBUG:boto:Data: 2024-04-16T19:54:34.487 DEBUG:boto:Headers: {} 2024-04-16T19:54:34.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.487 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.487 DEBUG:boto:Params: {} 2024-04-16T19:54:34.487 DEBUG:boto:Token: None 2024-04-16T19:54:34.487 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/ 2024-04-16T19:54:34.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T19:54:34.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T19:54:34.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a967f7b72bdcb97-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.490 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:54:34.490 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:54:34.490 DEBUG:boto:Method: PUT 2024-04-16T19:54:34.490 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:54:34.491 DEBUG:boto:Data: 2024-04-16T19:54:34.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:34.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.491 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.491 DEBUG:boto:Params: {} 2024-04-16T19:54:34.491 DEBUG:boto:Token: None 2024-04-16T19:54:34.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/f 2024-04-16T19:54:34.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p18Mzx5ZJbm933HXs+AubNP5XS8= 2024-04-16T19:54:34.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p18Mzx5ZJbm933HXs+AubNP5XS8='} 2024-04-16T19:54:34.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbd1f199b95578f4-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.497 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:34.497 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:34.497 DEBUG:boto:Method: HEAD 2024-04-16T19:54:34.497 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:34.497 DEBUG:boto:Data: 2024-04-16T19:54:34.497 DEBUG:boto:Headers: {} 2024-04-16T19:54:34.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.497 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.497 DEBUG:boto:Params: {} 2024-04-16T19:54:34.498 DEBUG:boto:Token: None 2024-04-16T19:54:34.498 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/ 2024-04-16T19:54:34.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T19:54:34.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T19:54:34.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b26f40b782495fc2-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.500 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:54:34.500 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:54:34.500 DEBUG:boto:Method: PUT 2024-04-16T19:54:34.501 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:54:34.501 DEBUG:boto:Data: 2024-04-16T19:54:34.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:34.501 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.501 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.501 DEBUG:boto:Params: {} 2024-04-16T19:54:34.501 DEBUG:boto:Token: None 2024-04-16T19:54:34.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/g 2024-04-16T19:54:34.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lWRjg0u+SZY78sdpNz9scQ6S11o= 2024-04-16T19:54:34.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lWRjg0u+SZY78sdpNz9scQ6S11o='} 2024-04-16T19:54:34.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cc1bb4b52d8150f-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.547 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:54:34.547 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:54:34.547 DEBUG:boto:Method: HEAD 2024-04-16T19:54:34.547 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:54:34.547 DEBUG:boto:Data: 2024-04-16T19:54:34.547 DEBUG:boto:Headers: {} 2024-04-16T19:54:34.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.547 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.547 DEBUG:boto:Params: {} 2024-04-16T19:54:34.547 DEBUG:boto:Token: None 2024-04-16T19:54:34.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/ 2024-04-16T19:54:34.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T19:54:34.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T19:54:34.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083dc07e2b5a581c4-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.550 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:54:34.550 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:54:34.550 DEBUG:boto:Method: PUT 2024-04-16T19:54:34.551 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:54:34.551 DEBUG:boto:Data: 2024-04-16T19:54:34.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:54:34.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:54:34.551 DEBUG:boto:Port: 8000 2024-04-16T19:54:34.551 DEBUG:boto:Params: {} 2024-04-16T19:54:34.551 DEBUG:boto:Token: None 2024-04-16T19:54:34.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:54:34 GMT /yrjefh-50/h 2024-04-16T19:54:34.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0gaMaRTjqfdC2bAXet/zNj/UEpM= 2024-04-16T19:54:34.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0gaMaRTjqfdC2bAXet/zNj/UEpM='} 2024-04-16T19:54:34.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b57ce3935e914c2-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:54:34.557 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T19:54:34.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:34.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:54:34.558 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:54:34.620 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:54:34.620 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:54:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:34.684+0000 7f768e52d800 1 waiting to reach latest gen 1, on 0.. 2024-04-16T19:54:39.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:39.686+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:54:39.689 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , ] 2024-04-16T19:54:39.689 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:54:44.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:44.688+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:54:44.690 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , ] 2024-04-16T19:54:44.690 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:54:49.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:49.688+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:54:49.691 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , ] 2024-04-16T19:54:49.691 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:54:54.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:54.690+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:54:54.692 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , ] 2024-04-16T19:54:54.692 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:54:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:54:59.692+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:54:59.694 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:54:59.694 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:04.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:04.693+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:04.695 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:55:04.695 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:09.695+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:09.696 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:55:09.696 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:14.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:14.696+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:14.698 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:55:14.698 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:19.698+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:19.699 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:55:19.699 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:24.699+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:24.701 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T19:55:24.701 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:29.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:29.700+0000 7f768e52d800 1 bucket sync caught up with source: 2024-04-16T19:55:29.702 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:29.702 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T19:55:29.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:29.700+0000 7f768e52d800 0 bucket checkpoint complete 2024-04-16T19:55:29.730 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T19:55:29.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:29.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:29.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:55:29.793 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:29.794 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:29.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:29.854+0000 7f30cec13800 1 waiting for incremental sync to catch up: 2024-04-16T19:55:29.856 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.162.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000002.105.6] 2024-04-16T19:55:29.856 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T19:55:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:34.856+0000 7f30cec13800 1 bucket sync caught up with source: 2024-04-16T19:55:34.857 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T19:55:34.857 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T19:55:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:34.856+0000 7f30cec13800 0 bucket checkpoint complete 2024-04-16T19:55:34.885 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:55:34.885 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.886 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.886 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.886 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:55:34.886 DEBUG:boto:Data: 2024-04-16T19:55:34.886 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.886 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.886 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.886 DEBUG:boto:Params: {} 2024-04-16T19:55:34.887 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:55:34.887 DEBUG:boto:Token: None 2024-04-16T19:55:34.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/ 2024-04-16T19:55:34.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w= 2024-04-16T19:55:34.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w=', 'Content-Length': '0'} 2024-04-16T19:55:34.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074164ea878b4b586-00661ed7b6-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.892 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.892 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.892 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.892 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:55:34.892 DEBUG:boto:Data: 2024-04-16T19:55:34.892 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.892 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.893 DEBUG:boto:Params: {} 2024-04-16T19:55:34.893 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:55:34.893 DEBUG:boto:Token: None 2024-04-16T19:55:34.893 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/ 2024-04-16T19:55:34.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w= 2024-04-16T19:55:34.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w=', 'Content-Length': '0'} 2024-04-16T19:55:34.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000144b041dc0cae123-00661ed7b6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:55:34.898 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.898 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.898 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.898 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.898 DEBUG:boto:Method: GET 2024-04-16T19:55:34.898 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.898 DEBUG:boto:Data: 2024-04-16T19:55:34.898 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.898 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.898 DEBUG:boto:Params: {} 2024-04-16T19:55:34.899 DEBUG:boto:Token: None 2024-04-16T19:55:34.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/?versions 2024-04-16T19:55:34.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T19:55:34.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T19:55:34.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a3c11d972812a95-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.901 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:55:34.902 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:55:34.903 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:55:34.903 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:55:34.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:55:34.903 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.903 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.903 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.903 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.903 DEBUG:boto:Method: GET 2024-04-16T19:55:34.903 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.903 DEBUG:boto:Data: 2024-04-16T19:55:34.903 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.903 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.904 DEBUG:boto:Params: {} 2024-04-16T19:55:34.904 DEBUG:boto:Token: None 2024-04-16T19:55:34.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/?versions 2024-04-16T19:55:34.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T19:55:34.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T19:55:34.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a8c0fc258a0efe-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.907 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.907 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:55:34.907 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:55:34.907 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:55:34.907 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:55:34.908 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:55:34.908 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:55:34.908 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:55:34.908 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:55:34.908 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.908 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.908 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.908 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.908 DEBUG:boto:Method: GET 2024-04-16T19:55:34.908 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.908 DEBUG:boto:Data: 2024-04-16T19:55:34.909 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.909 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.909 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.909 DEBUG:boto:Params: {} 2024-04-16T19:55:34.909 DEBUG:boto:Token: None 2024-04-16T19:55:34.909 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/?versions 2024-04-16T19:55:34.909 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T19:55:34.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T19:55:34.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec25bf2793be3c47-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.912 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.912 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.912 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.912 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.912 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.913 DEBUG:boto:Method: GET 2024-04-16T19:55:34.913 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:34.913 DEBUG:boto:Data: 2024-04-16T19:55:34.913 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.913 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.913 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.913 DEBUG:boto:Params: {} 2024-04-16T19:55:34.913 DEBUG:boto:Token: None 2024-04-16T19:55:34.913 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/?versions 2024-04-16T19:55:34.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T19:55:34.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T19:55:34.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce490b0701c5291b-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.916 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:34.917 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.917 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.917 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.917 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.917 DEBUG:boto:Method: GET 2024-04-16T19:55:34.917 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.917 DEBUG:boto:Data: 2024-04-16T19:55:34.917 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.917 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.917 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.917 DEBUG:boto:Params: {} 2024-04-16T19:55:34.918 DEBUG:boto:Token: None 2024-04-16T19:55:34.918 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.918 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T19:55:34.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T19:55:34.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e809398d4b2189e6-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.920 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.920 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.920 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.920 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.921 DEBUG:boto:Method: GET 2024-04-16T19:55:34.921 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.921 DEBUG:boto:Data: 2024-04-16T19:55:34.921 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.921 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.921 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.921 DEBUG:boto:Params: {} 2024-04-16T19:55:34.921 DEBUG:boto:Token: None 2024-04-16T19:55:34.921 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T19:55:34.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T19:55:34.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b4c1e7fb7d2fbb-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.923 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.924 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.924 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.924 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.924 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.924 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.924 DEBUG:boto:Data: 2024-04-16T19:55:34.924 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.924 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.924 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.924 DEBUG:boto:Params: {} 2024-04-16T19:55:34.924 DEBUG:boto:Token: None 2024-04-16T19:55:34.924 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo= 2024-04-16T19:55:34.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo=', 'Content-Length': '0'} 2024-04-16T19:55:34.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d56ef0e5777e236-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.927 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.927 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.927 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.927 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.927 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.927 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.927 DEBUG:boto:Data: 2024-04-16T19:55:34.927 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.927 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.927 DEBUG:boto:Params: {} 2024-04-16T19:55:34.927 DEBUG:boto:Token: None 2024-04-16T19:55:34.928 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo= 2024-04-16T19:55:34.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo=', 'Content-Length': '0'} 2024-04-16T19:55:34.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976c6446bca7b863-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:34.930 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.930 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.930 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.930 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.931 DEBUG:boto:Method: GET 2024-04-16T19:55:34.931 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.931 DEBUG:boto:Data: 2024-04-16T19:55:34.931 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.931 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.931 DEBUG:boto:Params: {} 2024-04-16T19:55:34.931 DEBUG:boto:Token: None 2024-04-16T19:55:34.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T19:55:34.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T19:55:34.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772564f8137bda24-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.974 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.974 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.974 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.974 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:34.975 DEBUG:boto:Method: GET 2024-04-16T19:55:34.975 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.975 DEBUG:boto:Data: 2024-04-16T19:55:34.975 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.975 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.975 DEBUG:boto:Params: {} 2024-04-16T19:55:34.975 DEBUG:boto:Token: None 2024-04-16T19:55:34.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:34.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T19:55:34.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T19:55:34.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bb29023e1607477-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.978 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.978 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.978 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:34.978 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:34.978 DEBUG:boto:Method: GET 2024-04-16T19:55:34.978 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:34.978 DEBUG:boto:Data: 2024-04-16T19:55:34.978 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.979 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.979 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.979 DEBUG:boto:Params: {} 2024-04-16T19:55:34.979 DEBUG:boto:Token: None 2024-04-16T19:55:34.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/ 2024-04-16T19:55:34.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM= 2024-04-16T19:55:34.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM=', 'Content-Length': '0'} 2024-04-16T19:55:34.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024aedd1d15b26a4e-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.982 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.982 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:34.982 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:34.982 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:34.982 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:34.983 DEBUG:boto:Method: GET 2024-04-16T19:55:34.983 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:34.983 DEBUG:boto:Data: 2024-04-16T19:55:34.983 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.983 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.983 DEBUG:boto:Params: {} 2024-04-16T19:55:34.983 DEBUG:boto:Token: None 2024-04-16T19:55:34.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/ 2024-04-16T19:55:34.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM= 2024-04-16T19:55:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM=', 'Content-Length': '0'} 2024-04-16T19:55:34.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ca1fe7f2c0f2e6-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.986 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:34.986 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.986 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.987 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.987 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:34.987 DEBUG:boto:Data: 2024-04-16T19:55:34.987 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.987 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.987 DEBUG:boto:Params: {} 2024-04-16T19:55:34.987 DEBUG:boto:Token: None 2024-04-16T19:55:34.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a 2024-04-16T19:55:34.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g= 2024-04-16T19:55:34.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g=', 'Content-Length': '0'} 2024-04-16T19:55:34.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73e797fb03f0c46-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.989 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.990 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.990 DEBUG:boto:Method: HEAD 2024-04-16T19:55:34.990 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:34.990 DEBUG:boto:Data: 2024-04-16T19:55:34.990 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:34.990 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.990 DEBUG:boto:Params: {} 2024-04-16T19:55:34.990 DEBUG:boto:Token: None 2024-04-16T19:55:34.990 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a 2024-04-16T19:55:34.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g= 2024-04-16T19:55:34.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g=', 'Content-Length': '0'} 2024-04-16T19:55:34.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d524103f90c09bf8-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:34.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:34.993 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:34.993 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:34.993 DEBUG:boto:Method: GET 2024-04-16T19:55:34.993 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:34.993 DEBUG:boto:Data: 2024-04-16T19:55:34.993 DEBUG:boto:Headers: {} 2024-04-16T19:55:34.993 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:34.993 DEBUG:boto:Port: 8000 2024-04-16T19:55:34.993 DEBUG:boto:Params: {} 2024-04-16T19:55:34.993 DEBUG:boto:Token: None 2024-04-16T19:55:34.993 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:34 GMT /yrjefh-50/a 2024-04-16T19:55:34.994 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZFqUTNvuhohpx+ys6fWhjZs3rU= 2024-04-16T19:55:34.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZFqUTNvuhohpx+ys6fWhjZs3rU=', 'Content-Length': '0'} 2024-04-16T19:55:34.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d7811c235d4e27-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.038 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:35.038 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:35.039 DEBUG:boto:Method: GET 2024-04-16T19:55:35.039 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:35.039 DEBUG:boto:Data: 2024-04-16T19:55:35.039 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.039 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.039 DEBUG:boto:Params: {} 2024-04-16T19:55:35.039 DEBUG:boto:Token: None 2024-04-16T19:55:35.039 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/a 2024-04-16T19:55:35.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gprj+si39b6GW6GUi75CPi7oCac= 2024-04-16T19:55:35.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gprj+si39b6GW6GUi75CPi7oCac=', 'Content-Length': '0'} 2024-04-16T19:55:35.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84acf5c0dbeced1-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.042 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.042 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.042 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:35.042 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:35.042 DEBUG:boto:Method: GET 2024-04-16T19:55:35.042 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:35.042 DEBUG:boto:Data: 2024-04-16T19:55:35.042 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.042 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.042 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.043 DEBUG:boto:Params: {} 2024-04-16T19:55:35.043 DEBUG:boto:Token: None 2024-04-16T19:55:35.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40746e6436537fb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.046 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.046 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.046 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.046 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:35.046 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:35.046 DEBUG:boto:Method: GET 2024-04-16T19:55:35.047 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:35.047 DEBUG:boto:Data: 2024-04-16T19:55:35.047 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.047 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.047 DEBUG:boto:Params: {} 2024-04-16T19:55:35.047 DEBUG:boto:Token: None 2024-04-16T19:55:35.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd1db27b7cf30c6-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.050 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:35.050 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.051 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.051 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.051 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.051 DEBUG:boto:Method: GET 2024-04-16T19:55:35.051 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.051 DEBUG:boto:Data: 2024-04-16T19:55:35.051 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.051 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.051 DEBUG:boto:Params: {} 2024-04-16T19:55:35.051 DEBUG:boto:Token: None 2024-04-16T19:55:35.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T19:55:35.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T19:55:35.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a13cc520b709b1-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.094 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.094 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.094 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.095 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.095 DEBUG:boto:Method: GET 2024-04-16T19:55:35.095 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.095 DEBUG:boto:Data: 2024-04-16T19:55:35.095 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.095 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.095 DEBUG:boto:Params: {} 2024-04-16T19:55:35.095 DEBUG:boto:Token: None 2024-04-16T19:55:35.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T19:55:35.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T19:55:35.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d00af871c072e32-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.098 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.098 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.098 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.098 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.098 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.098 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.098 DEBUG:boto:Data: 2024-04-16T19:55:35.098 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.098 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.099 DEBUG:boto:Params: {} 2024-04-16T19:55:35.099 DEBUG:boto:Token: None 2024-04-16T19:55:35.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc= 2024-04-16T19:55:35.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc=', 'Content-Length': '0'} 2024-04-16T19:55:35.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9b2968210d3f55-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.101 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.101 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.101 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.101 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.101 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.101 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.101 DEBUG:boto:Data: 2024-04-16T19:55:35.102 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.102 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.102 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.102 DEBUG:boto:Params: {} 2024-04-16T19:55:35.102 DEBUG:boto:Token: None 2024-04-16T19:55:35.102 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc= 2024-04-16T19:55:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc=', 'Content-Length': '0'} 2024-04-16T19:55:35.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b232ebcb1163cda7-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:35.104 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.104 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.104 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.104 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.104 DEBUG:boto:Method: GET 2024-04-16T19:55:35.105 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.105 DEBUG:boto:Data: 2024-04-16T19:55:35.105 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.105 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.105 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.105 DEBUG:boto:Params: {} 2024-04-16T19:55:35.105 DEBUG:boto:Token: None 2024-04-16T19:55:35.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T19:55:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T19:55:35.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d73298c319a7cd-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.150 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.150 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.151 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.151 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:35.151 DEBUG:boto:Method: GET 2024-04-16T19:55:35.151 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.151 DEBUG:boto:Data: 2024-04-16T19:55:35.151 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.151 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.151 DEBUG:boto:Params: {} 2024-04-16T19:55:35.151 DEBUG:boto:Token: None 2024-04-16T19:55:35.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:35.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T19:55:35.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T19:55:35.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50cabf251a9927f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.153 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.154 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.154 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.154 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.154 DEBUG:boto:Method: GET 2024-04-16T19:55:35.154 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:35.154 DEBUG:boto:Data: 2024-04-16T19:55:35.154 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.154 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.154 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.154 DEBUG:boto:Params: {} 2024-04-16T19:55:35.155 DEBUG:boto:Token: None 2024-04-16T19:55:35.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3147e04a080b81-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.157 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.158 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.158 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.158 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.158 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.158 DEBUG:boto:Method: GET 2024-04-16T19:55:35.158 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:35.158 DEBUG:boto:Data: 2024-04-16T19:55:35.158 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.158 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.158 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.159 DEBUG:boto:Params: {} 2024-04-16T19:55:35.159 DEBUG:boto:Token: None 2024-04-16T19:55:35.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b1f093b58bfe14-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.162 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.162 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.162 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.163 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.163 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:35.163 DEBUG:boto:Data: 2024-04-16T19:55:35.163 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.163 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.163 DEBUG:boto:Params: {} 2024-04-16T19:55:35.163 DEBUG:boto:Token: None 2024-04-16T19:55:35.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b 2024-04-16T19:55:35.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ= 2024-04-16T19:55:35.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ=', 'Content-Length': '0'} 2024-04-16T19:55:35.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001260be41d1531994-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.165 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.166 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.166 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.166 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:35.166 DEBUG:boto:Data: 2024-04-16T19:55:35.166 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.166 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.166 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.166 DEBUG:boto:Params: {} 2024-04-16T19:55:35.166 DEBUG:boto:Token: None 2024-04-16T19:55:35.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b 2024-04-16T19:55:35.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ= 2024-04-16T19:55:35.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ=', 'Content-Length': '0'} 2024-04-16T19:55:35.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1987190385a868-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:35.169 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.169 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.169 DEBUG:boto:Method: GET 2024-04-16T19:55:35.169 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:35.169 DEBUG:boto:Data: 2024-04-16T19:55:35.169 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.169 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.169 DEBUG:boto:Params: {} 2024-04-16T19:55:35.169 DEBUG:boto:Token: None 2024-04-16T19:55:35.170 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b 2024-04-16T19:55:35.170 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU= 2024-04-16T19:55:35.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU=', 'Content-Length': '0'} 2024-04-16T19:55:35.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093dc13705b92933a-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.214 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:35.214 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:35.214 DEBUG:boto:Method: GET 2024-04-16T19:55:35.215 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:35.215 DEBUG:boto:Data: 2024-04-16T19:55:35.215 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.215 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.215 DEBUG:boto:Params: {} 2024-04-16T19:55:35.215 DEBUG:boto:Token: None 2024-04-16T19:55:35.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/b 2024-04-16T19:55:35.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU= 2024-04-16T19:55:35.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU=', 'Content-Length': '0'} 2024-04-16T19:55:35.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ef0eb75fa5a5ef-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.217 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.217 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.217 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.218 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.218 DEBUG:boto:Method: GET 2024-04-16T19:55:35.218 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:35.218 DEBUG:boto:Data: 2024-04-16T19:55:35.218 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.218 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.218 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.218 DEBUG:boto:Params: {} 2024-04-16T19:55:35.218 DEBUG:boto:Token: None 2024-04-16T19:55:35.218 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.218 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9368bbdf3496a17-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.221 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.221 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.222 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.222 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.222 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:35.222 DEBUG:boto:Method: GET 2024-04-16T19:55:35.222 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:35.222 DEBUG:boto:Data: 2024-04-16T19:55:35.222 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.222 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.222 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.222 DEBUG:boto:Params: {} 2024-04-16T19:55:35.222 DEBUG:boto:Token: None 2024-04-16T19:55:35.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f805452a640f615-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.226 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:35.226 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.226 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.226 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.226 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.226 DEBUG:boto:Method: GET 2024-04-16T19:55:35.227 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.227 DEBUG:boto:Data: 2024-04-16T19:55:35.227 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.227 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.227 DEBUG:boto:Params: {} 2024-04-16T19:55:35.227 DEBUG:boto:Token: None 2024-04-16T19:55:35.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T19:55:35.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T19:55:35.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2367d7f46145a7-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.270 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.271 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.271 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.271 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.271 DEBUG:boto:Method: GET 2024-04-16T19:55:35.271 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.271 DEBUG:boto:Data: 2024-04-16T19:55:35.271 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.271 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.271 DEBUG:boto:Params: {} 2024-04-16T19:55:35.272 DEBUG:boto:Token: None 2024-04-16T19:55:35.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T19:55:35.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T19:55:35.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f00af875007824e-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.274 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.274 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.274 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.275 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.275 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.275 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.275 DEBUG:boto:Data: 2024-04-16T19:55:35.275 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.275 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.275 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.275 DEBUG:boto:Params: {} 2024-04-16T19:55:35.275 DEBUG:boto:Token: None 2024-04-16T19:55:35.275 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg= 2024-04-16T19:55:35.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg=', 'Content-Length': '0'} 2024-04-16T19:55:35.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbbca1dc4b72f77-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.278 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.278 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.278 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.278 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.278 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.278 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.278 DEBUG:boto:Data: 2024-04-16T19:55:35.278 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.278 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.278 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.278 DEBUG:boto:Params: {} 2024-04-16T19:55:35.279 DEBUG:boto:Token: None 2024-04-16T19:55:35.279 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg= 2024-04-16T19:55:35.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg=', 'Content-Length': '0'} 2024-04-16T19:55:35.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b44ee43480d636-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:35.281 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.281 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.281 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.281 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.281 DEBUG:boto:Method: GET 2024-04-16T19:55:35.281 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.281 DEBUG:boto:Data: 2024-04-16T19:55:35.282 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.282 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.282 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.282 DEBUG:boto:Params: {} 2024-04-16T19:55:35.282 DEBUG:boto:Token: None 2024-04-16T19:55:35.282 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.282 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T19:55:35.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T19:55:35.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987981d9c0947ef5-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.326 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.327 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.327 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.327 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:35.327 DEBUG:boto:Method: GET 2024-04-16T19:55:35.327 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.327 DEBUG:boto:Data: 2024-04-16T19:55:35.327 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.327 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.328 DEBUG:boto:Params: {} 2024-04-16T19:55:35.328 DEBUG:boto:Token: None 2024-04-16T19:55:35.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:35.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T19:55:35.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T19:55:35.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7acfba19cc72ac6-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.330 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.330 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.330 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.331 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.331 DEBUG:boto:Method: GET 2024-04-16T19:55:35.331 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:35.331 DEBUG:boto:Data: 2024-04-16T19:55:35.331 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.331 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.331 DEBUG:boto:Params: {} 2024-04-16T19:55:35.331 DEBUG:boto:Token: None 2024-04-16T19:55:35.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c35067489ba158-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.334 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.334 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.334 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.335 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.335 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.335 DEBUG:boto:Method: GET 2024-04-16T19:55:35.335 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:35.335 DEBUG:boto:Data: 2024-04-16T19:55:35.335 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.335 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.335 DEBUG:boto:Params: {} 2024-04-16T19:55:35.335 DEBUG:boto:Token: None 2024-04-16T19:55:35.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058974238029d2da7-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.338 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.338 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.339 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.339 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.339 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:35.339 DEBUG:boto:Data: 2024-04-16T19:55:35.339 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.339 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.339 DEBUG:boto:Params: {} 2024-04-16T19:55:35.339 DEBUG:boto:Token: None 2024-04-16T19:55:35.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c 2024-04-16T19:55:35.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw= 2024-04-16T19:55:35.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw=', 'Content-Length': '0'} 2024-04-16T19:55:35.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e054e0ce47a9dcb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.341 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.342 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.342 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.342 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:35.342 DEBUG:boto:Data: 2024-04-16T19:55:35.342 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.342 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.342 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.342 DEBUG:boto:Params: {} 2024-04-16T19:55:35.342 DEBUG:boto:Token: None 2024-04-16T19:55:35.342 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c 2024-04-16T19:55:35.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw= 2024-04-16T19:55:35.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw=', 'Content-Length': '0'} 2024-04-16T19:55:35.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea737cd6de6b82e-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:35.345 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.345 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.345 DEBUG:boto:Method: GET 2024-04-16T19:55:35.345 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:35.345 DEBUG:boto:Data: 2024-04-16T19:55:35.345 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.345 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.345 DEBUG:boto:Params: {} 2024-04-16T19:55:35.345 DEBUG:boto:Token: None 2024-04-16T19:55:35.346 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c 2024-04-16T19:55:35.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII= 2024-04-16T19:55:35.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII=', 'Content-Length': '0'} 2024-04-16T19:55:35.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f75ebb80fb705c2-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.390 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:35.390 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:35.391 DEBUG:boto:Method: GET 2024-04-16T19:55:35.391 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:35.391 DEBUG:boto:Data: 2024-04-16T19:55:35.391 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.391 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.391 DEBUG:boto:Params: {} 2024-04-16T19:55:35.391 DEBUG:boto:Token: None 2024-04-16T19:55:35.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/c 2024-04-16T19:55:35.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII= 2024-04-16T19:55:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII=', 'Content-Length': '0'} 2024-04-16T19:55:35.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2190939aaf912d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.394 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.394 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.394 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.394 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.394 DEBUG:boto:Method: GET 2024-04-16T19:55:35.394 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:35.394 DEBUG:boto:Data: 2024-04-16T19:55:35.394 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.394 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.394 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.395 DEBUG:boto:Params: {} 2024-04-16T19:55:35.395 DEBUG:boto:Token: None 2024-04-16T19:55:35.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bfd65c2a2523fab-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.397 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.398 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.398 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.398 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.398 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:35.398 DEBUG:boto:Method: GET 2024-04-16T19:55:35.398 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:35.398 DEBUG:boto:Data: 2024-04-16T19:55:35.398 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.398 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.399 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.399 DEBUG:boto:Params: {} 2024-04-16T19:55:35.399 DEBUG:boto:Token: None 2024-04-16T19:55:35.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b49d0f1a9429998-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.402 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:35.402 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.402 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.403 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.403 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.403 DEBUG:boto:Method: GET 2024-04-16T19:55:35.403 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.403 DEBUG:boto:Data: 2024-04-16T19:55:35.403 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.403 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.403 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.403 DEBUG:boto:Params: {} 2024-04-16T19:55:35.403 DEBUG:boto:Token: None 2024-04-16T19:55:35.403 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T19:55:35.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T19:55:35.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed6d8b99b05ab4d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.446 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.446 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.447 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.447 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.447 DEBUG:boto:Method: GET 2024-04-16T19:55:35.447 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.447 DEBUG:boto:Data: 2024-04-16T19:55:35.447 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.447 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.447 DEBUG:boto:Params: {} 2024-04-16T19:55:35.447 DEBUG:boto:Token: None 2024-04-16T19:55:35.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T19:55:35.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T19:55:35.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e7dbc778805f34-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.450 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.450 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.450 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.450 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.450 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.450 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.451 DEBUG:boto:Data: 2024-04-16T19:55:35.451 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.451 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.451 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.451 DEBUG:boto:Params: {} 2024-04-16T19:55:35.451 DEBUG:boto:Token: None 2024-04-16T19:55:35.451 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU= 2024-04-16T19:55:35.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU=', 'Content-Length': '0'} 2024-04-16T19:55:35.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9a3c72bb8721ea-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.454 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.454 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.454 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.454 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.454 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.454 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.454 DEBUG:boto:Data: 2024-04-16T19:55:35.455 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.455 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.455 DEBUG:boto:Params: {} 2024-04-16T19:55:35.455 DEBUG:boto:Token: None 2024-04-16T19:55:35.455 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU= 2024-04-16T19:55:35.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU=', 'Content-Length': '0'} 2024-04-16T19:55:35.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9625b40cb45fd4-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:35.458 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.458 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.458 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.458 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.458 DEBUG:boto:Method: GET 2024-04-16T19:55:35.458 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.458 DEBUG:boto:Data: 2024-04-16T19:55:35.458 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.458 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.458 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.458 DEBUG:boto:Params: {} 2024-04-16T19:55:35.458 DEBUG:boto:Token: None 2024-04-16T19:55:35.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T19:55:35.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T19:55:35.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8293d94f69d7ec-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.502 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.502 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.503 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.503 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:35.503 DEBUG:boto:Method: GET 2024-04-16T19:55:35.503 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.503 DEBUG:boto:Data: 2024-04-16T19:55:35.503 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.503 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.503 DEBUG:boto:Params: {} 2024-04-16T19:55:35.503 DEBUG:boto:Token: None 2024-04-16T19:55:35.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:35.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T19:55:35.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T19:55:35.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b400dc46790485f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.506 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.506 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.506 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.506 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.506 DEBUG:boto:Method: GET 2024-04-16T19:55:35.507 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:35.507 DEBUG:boto:Data: 2024-04-16T19:55:35.507 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.507 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.507 DEBUG:boto:Params: {} 2024-04-16T19:55:35.507 DEBUG:boto:Token: None 2024-04-16T19:55:35.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d48e1ad62a50c6-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.510 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.510 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.510 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.510 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.510 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.510 DEBUG:boto:Method: GET 2024-04-16T19:55:35.511 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:35.511 DEBUG:boto:Data: 2024-04-16T19:55:35.511 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.511 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.511 DEBUG:boto:Params: {} 2024-04-16T19:55:35.511 DEBUG:boto:Token: None 2024-04-16T19:55:35.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006611b1260ec557ec-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.514 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.515 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.515 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.515 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.515 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:35.515 DEBUG:boto:Data: 2024-04-16T19:55:35.515 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.515 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.515 DEBUG:boto:Params: {} 2024-04-16T19:55:35.515 DEBUG:boto:Token: None 2024-04-16T19:55:35.516 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d 2024-04-16T19:55:35.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8= 2024-04-16T19:55:35.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8=', 'Content-Length': '0'} 2024-04-16T19:55:35.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef722758a13d087-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.518 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.518 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.518 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.518 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:35.518 DEBUG:boto:Data: 2024-04-16T19:55:35.519 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.519 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.519 DEBUG:boto:Params: {} 2024-04-16T19:55:35.519 DEBUG:boto:Token: None 2024-04-16T19:55:35.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d 2024-04-16T19:55:35.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8= 2024-04-16T19:55:35.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8=', 'Content-Length': '0'} 2024-04-16T19:55:35.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181caffd2659f229-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:35.522 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.522 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.522 DEBUG:boto:Method: GET 2024-04-16T19:55:35.522 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:35.522 DEBUG:boto:Data: 2024-04-16T19:55:35.522 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.522 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.522 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.522 DEBUG:boto:Params: {} 2024-04-16T19:55:35.522 DEBUG:boto:Token: None 2024-04-16T19:55:35.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d 2024-04-16T19:55:35.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA= 2024-04-16T19:55:35.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA=', 'Content-Length': '0'} 2024-04-16T19:55:35.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ef8dac20bfaa92-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.570 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:35.570 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:35.571 DEBUG:boto:Method: GET 2024-04-16T19:55:35.571 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:35.571 DEBUG:boto:Data: 2024-04-16T19:55:35.571 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.571 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.571 DEBUG:boto:Params: {} 2024-04-16T19:55:35.571 DEBUG:boto:Token: None 2024-04-16T19:55:35.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/d 2024-04-16T19:55:35.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA= 2024-04-16T19:55:35.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA=', 'Content-Length': '0'} 2024-04-16T19:55:35.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eca006814999a923-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.574 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.574 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.574 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.574 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.574 DEBUG:boto:Method: GET 2024-04-16T19:55:35.574 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:35.575 DEBUG:boto:Data: 2024-04-16T19:55:35.575 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.575 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.575 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.575 DEBUG:boto:Params: {} 2024-04-16T19:55:35.575 DEBUG:boto:Token: None 2024-04-16T19:55:35.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007458b2e485306322-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.577 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.578 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.578 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.578 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.578 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:35.578 DEBUG:boto:Method: GET 2024-04-16T19:55:35.578 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:35.578 DEBUG:boto:Data: 2024-04-16T19:55:35.578 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.579 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.579 DEBUG:boto:Params: {} 2024-04-16T19:55:35.579 DEBUG:boto:Token: None 2024-04-16T19:55:35.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013856cd9eb455ab0-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.582 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:35.582 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.582 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.583 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.583 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.583 DEBUG:boto:Method: GET 2024-04-16T19:55:35.583 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.583 DEBUG:boto:Data: 2024-04-16T19:55:35.583 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.583 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.583 DEBUG:boto:Params: {} 2024-04-16T19:55:35.583 DEBUG:boto:Token: None 2024-04-16T19:55:35.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T19:55:35.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T19:55:35.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef471b9d0db319f4-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.630 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.631 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.631 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.631 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.631 DEBUG:boto:Method: GET 2024-04-16T19:55:35.631 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.631 DEBUG:boto:Data: 2024-04-16T19:55:35.631 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.631 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.631 DEBUG:boto:Params: {} 2024-04-16T19:55:35.631 DEBUG:boto:Token: None 2024-04-16T19:55:35.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T19:55:35.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T19:55:35.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c4f45161f50961-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.634 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.634 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.634 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.635 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.635 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.635 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.635 DEBUG:boto:Data: 2024-04-16T19:55:35.635 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.635 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.635 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.635 DEBUG:boto:Params: {} 2024-04-16T19:55:35.635 DEBUG:boto:Token: None 2024-04-16T19:55:35.635 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w= 2024-04-16T19:55:35.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w=', 'Content-Length': '0'} 2024-04-16T19:55:35.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c9165a5e938b43-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.638 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.638 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.638 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.638 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.638 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.638 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.638 DEBUG:boto:Data: 2024-04-16T19:55:35.638 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.638 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.638 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.639 DEBUG:boto:Params: {} 2024-04-16T19:55:35.639 DEBUG:boto:Token: None 2024-04-16T19:55:35.639 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w= 2024-04-16T19:55:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w=', 'Content-Length': '0'} 2024-04-16T19:55:35.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073de2cc2af2236d0-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:35.641 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.641 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.641 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.641 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.641 DEBUG:boto:Method: GET 2024-04-16T19:55:35.642 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.642 DEBUG:boto:Data: 2024-04-16T19:55:35.642 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.642 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.642 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.642 DEBUG:boto:Params: {} 2024-04-16T19:55:35.642 DEBUG:boto:Token: None 2024-04-16T19:55:35.642 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T19:55:35.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T19:55:35.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae97145747176811-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.686 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.686 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.686 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.687 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:35.687 DEBUG:boto:Method: GET 2024-04-16T19:55:35.687 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.687 DEBUG:boto:Data: 2024-04-16T19:55:35.687 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.687 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.687 DEBUG:boto:Params: {} 2024-04-16T19:55:35.687 DEBUG:boto:Token: None 2024-04-16T19:55:35.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:35.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T19:55:35.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T19:55:35.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064dce6e5b203de8a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.690 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.690 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.690 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.690 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.690 DEBUG:boto:Method: GET 2024-04-16T19:55:35.690 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:35.690 DEBUG:boto:Data: 2024-04-16T19:55:35.690 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.690 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.690 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.691 DEBUG:boto:Params: {} 2024-04-16T19:55:35.691 DEBUG:boto:Token: None 2024-04-16T19:55:35.691 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.691 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce6ba8fcc2ab174-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.693 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.694 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.694 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.694 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.694 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.694 DEBUG:boto:Method: GET 2024-04-16T19:55:35.694 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:35.694 DEBUG:boto:Data: 2024-04-16T19:55:35.694 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.694 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.695 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.695 DEBUG:boto:Params: {} 2024-04-16T19:55:35.695 DEBUG:boto:Token: None 2024-04-16T19:55:35.695 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860747e5f7e7cf81-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.698 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.698 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.698 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.698 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.699 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:35.699 DEBUG:boto:Data: 2024-04-16T19:55:35.699 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.699 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.699 DEBUG:boto:Params: {} 2024-04-16T19:55:35.699 DEBUG:boto:Token: None 2024-04-16T19:55:35.699 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e 2024-04-16T19:55:35.699 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg= 2024-04-16T19:55:35.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg=', 'Content-Length': '0'} 2024-04-16T19:55:35.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096670747b28458a1-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.701 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.701 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.701 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.701 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:35.702 DEBUG:boto:Data: 2024-04-16T19:55:35.702 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.702 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.702 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.702 DEBUG:boto:Params: {} 2024-04-16T19:55:35.702 DEBUG:boto:Token: None 2024-04-16T19:55:35.702 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e 2024-04-16T19:55:35.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg= 2024-04-16T19:55:35.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg=', 'Content-Length': '0'} 2024-04-16T19:55:35.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8b5969250bf49f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:35.704 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.704 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.704 DEBUG:boto:Method: GET 2024-04-16T19:55:35.705 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:35.705 DEBUG:boto:Data: 2024-04-16T19:55:35.705 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.705 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.705 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.705 DEBUG:boto:Params: {} 2024-04-16T19:55:35.705 DEBUG:boto:Token: None 2024-04-16T19:55:35.705 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e 2024-04-16T19:55:35.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U= 2024-04-16T19:55:35.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U=', 'Content-Length': '0'} 2024-04-16T19:55:35.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f1122927fd986d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.750 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:35.751 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:35.751 DEBUG:boto:Method: GET 2024-04-16T19:55:35.751 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:35.751 DEBUG:boto:Data: 2024-04-16T19:55:35.751 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.751 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.751 DEBUG:boto:Params: {} 2024-04-16T19:55:35.751 DEBUG:boto:Token: None 2024-04-16T19:55:35.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/e 2024-04-16T19:55:35.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U= 2024-04-16T19:55:35.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U=', 'Content-Length': '0'} 2024-04-16T19:55:35.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855f0c01b0a09147-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.754 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.754 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.754 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.754 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.755 DEBUG:boto:Method: GET 2024-04-16T19:55:35.755 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:35.755 DEBUG:boto:Data: 2024-04-16T19:55:35.755 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.755 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.755 DEBUG:boto:Params: {} 2024-04-16T19:55:35.755 DEBUG:boto:Token: None 2024-04-16T19:55:35.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4e6e89e53df938-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.758 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.758 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.758 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.758 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.759 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:35.759 DEBUG:boto:Method: GET 2024-04-16T19:55:35.759 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:35.759 DEBUG:boto:Data: 2024-04-16T19:55:35.759 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.759 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.759 DEBUG:boto:Params: {} 2024-04-16T19:55:35.759 DEBUG:boto:Token: None 2024-04-16T19:55:35.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31df3e1fa78e06d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.762 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:35.763 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.763 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.763 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.763 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.763 DEBUG:boto:Method: GET 2024-04-16T19:55:35.763 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.763 DEBUG:boto:Data: 2024-04-16T19:55:35.763 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.763 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.763 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.763 DEBUG:boto:Params: {} 2024-04-16T19:55:35.764 DEBUG:boto:Token: None 2024-04-16T19:55:35.764 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T19:55:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T19:55:35.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084990df426643815-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.806 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.806 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.807 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.807 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.807 DEBUG:boto:Method: GET 2024-04-16T19:55:35.807 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.807 DEBUG:boto:Data: 2024-04-16T19:55:35.807 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.807 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.807 DEBUG:boto:Params: {} 2024-04-16T19:55:35.807 DEBUG:boto:Token: None 2024-04-16T19:55:35.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T19:55:35.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T19:55:35.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8b523b975dfa6a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.810 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.810 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.810 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.810 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.811 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.811 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.811 DEBUG:boto:Data: 2024-04-16T19:55:35.811 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.811 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.811 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.811 DEBUG:boto:Params: {} 2024-04-16T19:55:35.811 DEBUG:boto:Token: None 2024-04-16T19:55:35.811 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I= 2024-04-16T19:55:35.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I=', 'Content-Length': '0'} 2024-04-16T19:55:35.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b253156a1018eb5-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.813 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.813 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.813 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.814 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.814 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.814 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.814 DEBUG:boto:Data: 2024-04-16T19:55:35.814 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.814 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.814 DEBUG:boto:Params: {} 2024-04-16T19:55:35.814 DEBUG:boto:Token: None 2024-04-16T19:55:35.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I= 2024-04-16T19:55:35.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I=', 'Content-Length': '0'} 2024-04-16T19:55:35.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6cd00b45eb192a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:35.817 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.817 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.817 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.817 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.817 DEBUG:boto:Method: GET 2024-04-16T19:55:35.818 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.818 DEBUG:boto:Data: 2024-04-16T19:55:35.818 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.818 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.818 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.818 DEBUG:boto:Params: {} 2024-04-16T19:55:35.818 DEBUG:boto:Token: None 2024-04-16T19:55:35.818 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T19:55:35.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T19:55:35.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d6b8d370a5ca58f-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.862 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.862 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.863 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.863 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:35.863 DEBUG:boto:Method: GET 2024-04-16T19:55:35.863 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.863 DEBUG:boto:Data: 2024-04-16T19:55:35.863 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.863 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.863 DEBUG:boto:Params: {} 2024-04-16T19:55:35.863 DEBUG:boto:Token: None 2024-04-16T19:55:35.864 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:35.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T19:55:35.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T19:55:35.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004754eff8955ad718-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.866 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.866 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.866 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.866 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.866 DEBUG:boto:Method: GET 2024-04-16T19:55:35.866 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:35.867 DEBUG:boto:Data: 2024-04-16T19:55:35.867 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.867 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.867 DEBUG:boto:Params: {} 2024-04-16T19:55:35.867 DEBUG:boto:Token: None 2024-04-16T19:55:35.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd8fb2ad5405c1d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.870 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.870 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.871 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.871 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.871 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.871 DEBUG:boto:Method: GET 2024-04-16T19:55:35.871 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:35.871 DEBUG:boto:Data: 2024-04-16T19:55:35.871 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.871 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.871 DEBUG:boto:Params: {} 2024-04-16T19:55:35.871 DEBUG:boto:Token: None 2024-04-16T19:55:35.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19c5d60bafe125a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.875 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.875 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.875 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.875 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.875 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:35.875 DEBUG:boto:Data: 2024-04-16T19:55:35.875 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.875 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.876 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.876 DEBUG:boto:Params: {} 2024-04-16T19:55:35.876 DEBUG:boto:Token: None 2024-04-16T19:55:35.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f 2024-04-16T19:55:35.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs= 2024-04-16T19:55:35.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs=', 'Content-Length': '0'} 2024-04-16T19:55:35.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c84f08a059f510-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.878 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.878 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.878 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.879 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:35.879 DEBUG:boto:Data: 2024-04-16T19:55:35.879 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.879 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.879 DEBUG:boto:Params: {} 2024-04-16T19:55:35.879 DEBUG:boto:Token: None 2024-04-16T19:55:35.879 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f 2024-04-16T19:55:35.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs= 2024-04-16T19:55:35.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs=', 'Content-Length': '0'} 2024-04-16T19:55:35.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8e6bc5f655507a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:35.881 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.881 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.882 DEBUG:boto:Method: GET 2024-04-16T19:55:35.882 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:35.882 DEBUG:boto:Data: 2024-04-16T19:55:35.882 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.882 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.882 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.882 DEBUG:boto:Params: {} 2024-04-16T19:55:35.882 DEBUG:boto:Token: None 2024-04-16T19:55:35.882 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f 2024-04-16T19:55:35.882 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI= 2024-04-16T19:55:35.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI=', 'Content-Length': '0'} 2024-04-16T19:55:35.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780fe4aed69f6afb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.926 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:35.927 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:35.927 DEBUG:boto:Method: GET 2024-04-16T19:55:35.927 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:35.927 DEBUG:boto:Data: 2024-04-16T19:55:35.927 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.927 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.927 DEBUG:boto:Params: {} 2024-04-16T19:55:35.927 DEBUG:boto:Token: None 2024-04-16T19:55:35.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/f 2024-04-16T19:55:35.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI= 2024-04-16T19:55:35.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI=', 'Content-Length': '0'} 2024-04-16T19:55:35.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699891c118b19958-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.930 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.930 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.930 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.930 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.930 DEBUG:boto:Method: GET 2024-04-16T19:55:35.930 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:35.930 DEBUG:boto:Data: 2024-04-16T19:55:35.930 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.931 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.931 DEBUG:boto:Params: {} 2024-04-16T19:55:35.931 DEBUG:boto:Token: None 2024-04-16T19:55:35.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05cf3242b4d0a35-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.934 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.934 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:35.934 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:35.934 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.934 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:35.934 DEBUG:boto:Method: GET 2024-04-16T19:55:35.935 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:35.935 DEBUG:boto:Data: 2024-04-16T19:55:35.935 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.935 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.935 DEBUG:boto:Params: {} 2024-04-16T19:55:35.935 DEBUG:boto:Token: None 2024-04-16T19:55:35.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/ 2024-04-16T19:55:35.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T19:55:35.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T19:55:35.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004239dfbac23865ef-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.938 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:35.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:35.938 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:35.939 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:35.939 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.939 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.939 DEBUG:boto:Method: GET 2024-04-16T19:55:35.939 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.939 DEBUG:boto:Data: 2024-04-16T19:55:35.939 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.939 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.939 DEBUG:boto:Params: {} 2024-04-16T19:55:35.939 DEBUG:boto:Token: None 2024-04-16T19:55:35.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T19:55:35.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T19:55:35.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1de076087f74707-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.982 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:35.983 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:35.983 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.983 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.983 DEBUG:boto:Method: GET 2024-04-16T19:55:35.983 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.983 DEBUG:boto:Data: 2024-04-16T19:55:35.983 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.983 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.983 DEBUG:boto:Params: {} 2024-04-16T19:55:35.983 DEBUG:boto:Token: None 2024-04-16T19:55:35.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T19:55:35.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T19:55:35.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008306bf94d3feb05d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.986 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:35.986 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:35.986 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.986 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.986 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.987 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.987 DEBUG:boto:Data: 2024-04-16T19:55:35.987 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.987 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.987 DEBUG:boto:Params: {} 2024-04-16T19:55:35.987 DEBUG:boto:Token: None 2024-04-16T19:55:35.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8= 2024-04-16T19:55:35.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8=', 'Content-Length': '0'} 2024-04-16T19:55:35.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb548b9b69e4c162-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.990 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:35.990 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:35.990 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.990 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.990 DEBUG:boto:Method: HEAD 2024-04-16T19:55:35.990 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.990 DEBUG:boto:Data: 2024-04-16T19:55:35.991 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:35.991 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.991 DEBUG:boto:Params: {} 2024-04-16T19:55:35.991 DEBUG:boto:Token: None 2024-04-16T19:55:35.991 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8= 2024-04-16T19:55:35.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8=', 'Content-Length': '0'} 2024-04-16T19:55:35.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c01809b3d5d4fe-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:35.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:35.993 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:35.994 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:35.994 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.994 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:35.994 DEBUG:boto:Method: GET 2024-04-16T19:55:35.994 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.994 DEBUG:boto:Data: 2024-04-16T19:55:35.994 DEBUG:boto:Headers: {} 2024-04-16T19:55:35.994 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:35.994 DEBUG:boto:Port: 8000 2024-04-16T19:55:35.994 DEBUG:boto:Params: {} 2024-04-16T19:55:35.994 DEBUG:boto:Token: None 2024-04-16T19:55:35.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:35 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:35.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T19:55:35.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T19:55:35.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d216dadaec9cbc4a-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.038 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:36.038 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:36.039 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:36.039 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:36.039 DEBUG:boto:Method: GET 2024-04-16T19:55:36.039 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:36.039 DEBUG:boto:Data: 2024-04-16T19:55:36.039 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.039 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.039 DEBUG:boto:Params: {} 2024-04-16T19:55:36.039 DEBUG:boto:Token: None 2024-04-16T19:55:36.039 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:36.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:poxmjJJ42bAZFatclaSVqMWWgsU= 2024-04-16T19:55:36.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:poxmjJJ42bAZFatclaSVqMWWgsU=', 'Content-Length': '0'} 2024-04-16T19:55:36.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055ab237f56a7354-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.042 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.042 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.042 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.042 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.043 DEBUG:boto:Method: GET 2024-04-16T19:55:36.043 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:36.043 DEBUG:boto:Data: 2024-04-16T19:55:36.043 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.043 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.043 DEBUG:boto:Params: {} 2024-04-16T19:55:36.043 DEBUG:boto:Token: None 2024-04-16T19:55:36.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720a804cd6b23a7a-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.046 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.046 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.046 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.047 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.047 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.047 DEBUG:boto:Method: GET 2024-04-16T19:55:36.047 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:36.047 DEBUG:boto:Data: 2024-04-16T19:55:36.047 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.047 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.047 DEBUG:boto:Params: {} 2024-04-16T19:55:36.047 DEBUG:boto:Token: None 2024-04-16T19:55:36.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632cecd4a82ee0e5-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.051 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.051 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:36.051 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:36.051 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.051 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:36.051 DEBUG:boto:Data: 2024-04-16T19:55:36.052 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.052 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.052 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.052 DEBUG:boto:Params: {} 2024-04-16T19:55:36.052 DEBUG:boto:Token: None 2024-04-16T19:55:36.052 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/g 2024-04-16T19:55:36.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo= 2024-04-16T19:55:36.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo=', 'Content-Length': '0'} 2024-04-16T19:55:36.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f626b805f760125b-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.054 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:36.055 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:36.055 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.055 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:36.055 DEBUG:boto:Data: 2024-04-16T19:55:36.055 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.055 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.055 DEBUG:boto:Params: {} 2024-04-16T19:55:36.055 DEBUG:boto:Token: None 2024-04-16T19:55:36.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/g 2024-04-16T19:55:36.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo= 2024-04-16T19:55:36.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo=', 'Content-Length': '0'} 2024-04-16T19:55:36.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b39e306fdf45e6-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:36.058 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:36.058 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:36.058 DEBUG:boto:Method: GET 2024-04-16T19:55:36.058 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:36.058 DEBUG:boto:Data: 2024-04-16T19:55:36.058 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.058 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.058 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.059 DEBUG:boto:Params: {} 2024-04-16T19:55:36.059 DEBUG:boto:Token: None 2024-04-16T19:55:36.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/g 2024-04-16T19:55:36.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI= 2024-04-16T19:55:36.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI=', 'Content-Length': '0'} 2024-04-16T19:55:36.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f895164625a63bc-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.102 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:36.102 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:36.103 DEBUG:boto:Method: GET 2024-04-16T19:55:36.103 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:36.103 DEBUG:boto:Data: 2024-04-16T19:55:36.103 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.103 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.103 DEBUG:boto:Params: {} 2024-04-16T19:55:36.103 DEBUG:boto:Token: None 2024-04-16T19:55:36.103 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/g 2024-04-16T19:55:36.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI= 2024-04-16T19:55:36.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI=', 'Content-Length': '0'} 2024-04-16T19:55:36.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ce2d2e56baf8c8-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.106 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.106 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.106 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.106 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.106 DEBUG:boto:Method: GET 2024-04-16T19:55:36.106 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:36.106 DEBUG:boto:Data: 2024-04-16T19:55:36.106 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.107 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.107 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.107 DEBUG:boto:Params: {} 2024-04-16T19:55:36.107 DEBUG:boto:Token: None 2024-04-16T19:55:36.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a366c567d2bbbb36-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.150 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.151 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.151 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.151 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.151 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:36.151 DEBUG:boto:Method: GET 2024-04-16T19:55:36.151 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:36.151 DEBUG:boto:Data: 2024-04-16T19:55:36.151 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.151 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.151 DEBUG:boto:Params: {} 2024-04-16T19:55:36.151 DEBUG:boto:Token: None 2024-04-16T19:55:36.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fabad3e61473c5cd-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.154 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:36.155 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.155 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.155 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.155 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.155 DEBUG:boto:Method: GET 2024-04-16T19:55:36.155 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.155 DEBUG:boto:Data: 2024-04-16T19:55:36.156 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.156 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.156 DEBUG:boto:Params: {} 2024-04-16T19:55:36.156 DEBUG:boto:Token: None 2024-04-16T19:55:36.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T19:55:36.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T19:55:36.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fbf7d828a9c95c-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.198 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.198 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.198 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.199 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.199 DEBUG:boto:Method: GET 2024-04-16T19:55:36.199 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.199 DEBUG:boto:Data: 2024-04-16T19:55:36.199 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.199 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.199 DEBUG:boto:Params: {} 2024-04-16T19:55:36.199 DEBUG:boto:Token: None 2024-04-16T19:55:36.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T19:55:36.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T19:55:36.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d1b971f688adcc-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.201 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.202 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.202 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.202 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.202 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.202 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.202 DEBUG:boto:Data: 2024-04-16T19:55:36.202 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.202 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.202 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.202 DEBUG:boto:Params: {} 2024-04-16T19:55:36.202 DEBUG:boto:Token: None 2024-04-16T19:55:36.203 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A= 2024-04-16T19:55:36.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A=', 'Content-Length': '0'} 2024-04-16T19:55:36.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24591668187355c-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.205 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.205 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.205 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.205 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.205 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.205 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.205 DEBUG:boto:Data: 2024-04-16T19:55:36.205 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.206 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.206 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.206 DEBUG:boto:Params: {} 2024-04-16T19:55:36.206 DEBUG:boto:Token: None 2024-04-16T19:55:36.206 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.206 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A= 2024-04-16T19:55:36.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A=', 'Content-Length': '0'} 2024-04-16T19:55:36.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084af76aba47463b7-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:36.208 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.208 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.208 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.208 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.208 DEBUG:boto:Method: GET 2024-04-16T19:55:36.208 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.209 DEBUG:boto:Data: 2024-04-16T19:55:36.209 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.209 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.209 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.209 DEBUG:boto:Params: {} 2024-04-16T19:55:36.209 DEBUG:boto:Token: None 2024-04-16T19:55:36.209 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T19:55:36.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T19:55:36.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070aaaba6e72dcf15-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.254 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.254 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.254 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.255 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:36.255 DEBUG:boto:Method: GET 2024-04-16T19:55:36.255 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.255 DEBUG:boto:Data: 2024-04-16T19:55:36.255 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.255 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.255 DEBUG:boto:Params: {} 2024-04-16T19:55:36.255 DEBUG:boto:Token: None 2024-04-16T19:55:36.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:36.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T19:55:36.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T19:55:36.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003117eccfef1999c7-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.258 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.258 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.258 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.258 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.258 DEBUG:boto:Method: GET 2024-04-16T19:55:36.258 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:36.258 DEBUG:boto:Data: 2024-04-16T19:55:36.258 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.258 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.258 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.259 DEBUG:boto:Params: {} 2024-04-16T19:55:36.259 DEBUG:boto:Token: None 2024-04-16T19:55:36.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81c01564ccb14aa-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.306 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.307 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.307 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.307 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.307 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.307 DEBUG:boto:Method: GET 2024-04-16T19:55:36.307 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:36.307 DEBUG:boto:Data: 2024-04-16T19:55:36.307 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.307 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.307 DEBUG:boto:Params: {} 2024-04-16T19:55:36.307 DEBUG:boto:Token: None 2024-04-16T19:55:36.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64b7b3db1e5d6a2-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.311 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.311 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.311 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.311 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.311 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:36.311 DEBUG:boto:Data: 2024-04-16T19:55:36.311 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.312 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.312 DEBUG:boto:Params: {} 2024-04-16T19:55:36.312 DEBUG:boto:Token: None 2024-04-16T19:55:36.312 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h 2024-04-16T19:55:36.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco= 2024-04-16T19:55:36.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco=', 'Content-Length': '0'} 2024-04-16T19:55:36.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f179c7d520ef1b-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.314 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.314 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.314 DEBUG:boto:Method: HEAD 2024-04-16T19:55:36.314 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:36.314 DEBUG:boto:Data: 2024-04-16T19:55:36.314 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.315 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.315 DEBUG:boto:Params: {} 2024-04-16T19:55:36.315 DEBUG:boto:Token: None 2024-04-16T19:55:36.315 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h 2024-04-16T19:55:36.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco= 2024-04-16T19:55:36.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco=', 'Content-Length': '0'} 2024-04-16T19:55:36.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b586293e1fe56d4e-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:36.317 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.317 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.317 DEBUG:boto:Method: GET 2024-04-16T19:55:36.317 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:36.317 DEBUG:boto:Data: 2024-04-16T19:55:36.318 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.318 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.318 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.318 DEBUG:boto:Params: {} 2024-04-16T19:55:36.318 DEBUG:boto:Token: None 2024-04-16T19:55:36.318 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h 2024-04-16T19:55:36.318 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4= 2024-04-16T19:55:36.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4=', 'Content-Length': '0'} 2024-04-16T19:55:36.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073db62cf40fc7d90-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.362 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:36.362 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:36.363 DEBUG:boto:Method: GET 2024-04-16T19:55:36.363 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:36.363 DEBUG:boto:Data: 2024-04-16T19:55:36.363 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.363 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.363 DEBUG:boto:Params: {} 2024-04-16T19:55:36.363 DEBUG:boto:Token: None 2024-04-16T19:55:36.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/h 2024-04-16T19:55:36.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4= 2024-04-16T19:55:36.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4=', 'Content-Length': '0'} 2024-04-16T19:55:36.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a231b47a51bb4cd2-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.365 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.365 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.366 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.366 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.366 DEBUG:boto:Method: GET 2024-04-16T19:55:36.366 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:36.366 DEBUG:boto:Data: 2024-04-16T19:55:36.366 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.366 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:36.366 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.366 DEBUG:boto:Params: {} 2024-04-16T19:55:36.366 DEBUG:boto:Token: None 2024-04-16T19:55:36.366 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380fdc21523b82fc-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.369 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.369 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:36.369 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:36.370 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.370 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:36.370 DEBUG:boto:Method: GET 2024-04-16T19:55:36.370 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:36.370 DEBUG:boto:Data: 2024-04-16T19:55:36.370 DEBUG:boto:Headers: {} 2024-04-16T19:55:36.370 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:36.370 DEBUG:boto:Port: 8000 2024-04-16T19:55:36.370 DEBUG:boto:Params: {} 2024-04-16T19:55:36.370 DEBUG:boto:Token: None 2024-04-16T19:55:36.370 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:36 GMT /yrjefh-50/ 2024-04-16T19:55:36.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T19:55:36.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T19:55:36.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031028ddff9575851-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:36.373 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:36.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:55:36.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:36.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:36.374 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-50 --num-shards 15 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:36.427 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:36.427 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:36.498 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:55:36.498 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-50 2024-04-16T19:55:36.504 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 8 2024-04-16T19:55:36.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:36.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:36.557 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:36.672 INFO:teuthology.orchestra.run.smithi026.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T19:55:36.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T19:55:36.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:36.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:36.673 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:55:36.734 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:36.735 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:36.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:36.795+0000 7f58b3f33800 1 waiting to reach latest gen 2, on 1.. 2024-04-16T19:55:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:41.796+0000 7f58b3f33800 1 bucket sync caught up with empty source 2024-04-16T19:55:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:41.796+0000 7f58b3f33800 0 bucket checkpoint complete 2024-04-16T19:55:41.824 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T19:55:41.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:41.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:41.825 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:55:41.887 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:41.887 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:41.949+0000 7f2750016800 1 bucket sync caught up with source: 2024-04-16T19:55:41.949 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T19:55:41.949 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T19:55:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:41.949+0000 7f2750016800 0 bucket checkpoint complete 2024-04-16T19:55:41.972 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:55:41.972 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.972 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.972 DEBUG:boto:Method: HEAD 2024-04-16T19:55:41.972 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:55:41.972 DEBUG:boto:Data: 2024-04-16T19:55:41.972 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.972 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:41.972 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.973 DEBUG:boto:Params: {} 2024-04-16T19:55:41.973 DEBUG:boto:Token: None 2024-04-16T19:55:41.973 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/ 2024-04-16T19:55:41.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8= 2024-04-16T19:55:41.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8=', 'Content-Length': '0'} 2024-04-16T19:55:41.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af355f4675315ec9-00661ed7bd-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.976 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.976 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.976 DEBUG:boto:Method: HEAD 2024-04-16T19:55:41.976 DEBUG:boto:Path: /yrjefh-50/ 2024-04-16T19:55:41.976 DEBUG:boto:Data: 2024-04-16T19:55:41.976 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.976 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:41.976 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.976 DEBUG:boto:Params: {} 2024-04-16T19:55:41.977 DEBUG:boto:Token: None 2024-04-16T19:55:41.977 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/ 2024-04-16T19:55:41.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8= 2024-04-16T19:55:41.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8=', 'Content-Length': '0'} 2024-04-16T19:55:41.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec1c3949b17cf011-00661ed7bd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:55:41.980 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.980 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.980 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.980 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.980 DEBUG:boto:Method: GET 2024-04-16T19:55:41.980 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.980 DEBUG:boto:Data: 2024-04-16T19:55:41.981 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.981 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:41.981 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.981 DEBUG:boto:Params: {} 2024-04-16T19:55:41.981 DEBUG:boto:Token: None 2024-04-16T19:55:41.981 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/?versions 2024-04-16T19:55:41.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T19:55:41.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T19:55:41.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dff54893410da9b-00661ed7bd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.984 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:41.984 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:55:41.984 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:55:41.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:55:41.985 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.985 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.985 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.985 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.985 DEBUG:boto:Method: GET 2024-04-16T19:55:41.986 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.986 DEBUG:boto:Data: 2024-04-16T19:55:41.986 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:41.986 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.986 DEBUG:boto:Params: {} 2024-04-16T19:55:41.986 DEBUG:boto:Token: None 2024-04-16T19:55:41.986 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/?versions 2024-04-16T19:55:41.986 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T19:55:41.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T19:55:41.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998c13f910b89a04-00661ed7bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.989 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-16T19:55:41.990 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-16T19:55:41.991 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.991 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.991 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.991 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.991 DEBUG:boto:Method: GET 2024-04-16T19:55:41.991 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.991 DEBUG:boto:Data: 2024-04-16T19:55:41.991 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.991 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:41.991 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.991 DEBUG:boto:Params: {} 2024-04-16T19:55:41.992 DEBUG:boto:Token: None 2024-04-16T19:55:41.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/?versions 2024-04-16T19:55:41.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T19:55:41.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T19:55:41.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f95229fb1d5ad5c-00661ed7bd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.994 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:41.995 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:41.995 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:41.995 DEBUG:boto:path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.995 DEBUG:boto:auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.995 DEBUG:boto:Method: GET 2024-04-16T19:55:41.996 DEBUG:boto:Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T19:55:41.996 DEBUG:boto:Data: 2024-04-16T19:55:41.996 DEBUG:boto:Headers: {} 2024-04-16T19:55:41.996 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:41.996 DEBUG:boto:Port: 8000 2024-04-16T19:55:41.996 DEBUG:boto:Params: {} 2024-04-16T19:55:41.996 DEBUG:boto:Token: None 2024-04-16T19:55:41.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:41 GMT /yrjefh-50/?versions 2024-04-16T19:55:41.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T19:55:41.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T19:55:41.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462fcc87bf7bc441-00661ed7bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:41.998 DEBUG:boto:b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:41.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:41.999 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:41.999 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:41.999 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:41.999 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.000 DEBUG:boto:Method: GET 2024-04-16T19:55:42.000 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.000 DEBUG:boto:Data: 2024-04-16T19:55:42.000 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.000 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.000 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.000 DEBUG:boto:Params: {} 2024-04-16T19:55:42.000 DEBUG:boto:Token: None 2024-04-16T19:55:42.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T19:55:42.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T19:55:42.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a0f0e339cbd03d-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.046 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.047 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.047 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.047 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.047 DEBUG:boto:Method: GET 2024-04-16T19:55:42.047 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.047 DEBUG:boto:Data: 2024-04-16T19:55:42.047 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.047 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.047 DEBUG:boto:Params: {} 2024-04-16T19:55:42.047 DEBUG:boto:Token: None 2024-04-16T19:55:42.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T19:55:42.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T19:55:42.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630edb97bcabae24-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.050 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.050 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.050 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.050 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.050 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.050 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.050 DEBUG:boto:Data: 2024-04-16T19:55:42.050 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.051 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.051 DEBUG:boto:Params: {} 2024-04-16T19:55:42.051 DEBUG:boto:Token: None 2024-04-16T19:55:42.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q= 2024-04-16T19:55:42.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q=', 'Content-Length': '0'} 2024-04-16T19:55:42.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057cf18e45b9b8037-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.053 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.053 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.053 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.053 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.053 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.053 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.054 DEBUG:boto:Data: 2024-04-16T19:55:42.054 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.054 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.054 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.054 DEBUG:boto:Params: {} 2024-04-16T19:55:42.054 DEBUG:boto:Token: None 2024-04-16T19:55:42.054 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q= 2024-04-16T19:55:42.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q=', 'Content-Length': '0'} 2024-04-16T19:55:42.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261b0c005af62261-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:42.056 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.056 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.056 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.057 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.057 DEBUG:boto:Method: GET 2024-04-16T19:55:42.057 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.057 DEBUG:boto:Data: 2024-04-16T19:55:42.057 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.057 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.057 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.057 DEBUG:boto:Params: {} 2024-04-16T19:55:42.057 DEBUG:boto:Token: None 2024-04-16T19:55:42.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T19:55:42.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T19:55:42.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093dfdf0dcddd0745-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.060 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.060 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.060 DEBUG:boto:path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.060 DEBUG:boto:auth_path=/yrjefh-50/a?versionId=null 2024-04-16T19:55:42.060 DEBUG:boto:Method: GET 2024-04-16T19:55:42.060 DEBUG:boto:Path: /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.060 DEBUG:boto:Data: 2024-04-16T19:55:42.060 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.060 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.061 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.061 DEBUG:boto:Params: {} 2024-04-16T19:55:42.061 DEBUG:boto:Token: None 2024-04-16T19:55:42.061 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a?versionId=null 2024-04-16T19:55:42.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T19:55:42.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T19:55:42.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca459da217e90c1-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.063 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.063 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.063 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.063 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.063 DEBUG:boto:Method: GET 2024-04-16T19:55:42.063 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:42.063 DEBUG:boto:Data: 2024-04-16T19:55:42.064 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.064 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.064 DEBUG:boto:Params: {} 2024-04-16T19:55:42.064 DEBUG:boto:Token: None 2024-04-16T19:55:42.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd2d46bd5a56ffb-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.067 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.067 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.067 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.067 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.067 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.067 DEBUG:boto:Method: GET 2024-04-16T19:55:42.067 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:42.067 DEBUG:boto:Data: 2024-04-16T19:55:42.068 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.068 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.068 DEBUG:boto:Params: {} 2024-04-16T19:55:42.068 DEBUG:boto:Token: None 2024-04-16T19:55:42.068 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a2a2e70862b244-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.114 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.115 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.115 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.115 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.115 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:42.115 DEBUG:boto:Data: 2024-04-16T19:55:42.115 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.115 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.115 DEBUG:boto:Params: {} 2024-04-16T19:55:42.115 DEBUG:boto:Token: None 2024-04-16T19:55:42.115 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a 2024-04-16T19:55:42.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E= 2024-04-16T19:55:42.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E=', 'Content-Length': '0'} 2024-04-16T19:55:42.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb795115e5cd081-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.118 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.118 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.118 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.118 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:42.118 DEBUG:boto:Data: 2024-04-16T19:55:42.118 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.118 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.119 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.119 DEBUG:boto:Params: {} 2024-04-16T19:55:42.119 DEBUG:boto:Token: None 2024-04-16T19:55:42.119 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a 2024-04-16T19:55:42.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E= 2024-04-16T19:55:42.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E=', 'Content-Length': '0'} 2024-04-16T19:55:42.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e175a69ee048feb6-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-16T19:55:42.121 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.121 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.121 DEBUG:boto:Method: GET 2024-04-16T19:55:42.121 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:42.121 DEBUG:boto:Data: 2024-04-16T19:55:42.121 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.121 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.121 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.122 DEBUG:boto:Params: {} 2024-04-16T19:55:42.122 DEBUG:boto:Token: None 2024-04-16T19:55:42.122 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a 2024-04-16T19:55:42.122 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU= 2024-04-16T19:55:42.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU=', 'Content-Length': '0'} 2024-04-16T19:55:42.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff67daf7f91d7b9-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.166 DEBUG:boto:path=/yrjefh-50/a 2024-04-16T19:55:42.166 DEBUG:boto:auth_path=/yrjefh-50/a 2024-04-16T19:55:42.167 DEBUG:boto:Method: GET 2024-04-16T19:55:42.167 DEBUG:boto:Path: /yrjefh-50/a 2024-04-16T19:55:42.167 DEBUG:boto:Data: 2024-04-16T19:55:42.167 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.167 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.167 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.167 DEBUG:boto:Params: {} 2024-04-16T19:55:42.167 DEBUG:boto:Token: None 2024-04-16T19:55:42.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/a 2024-04-16T19:55:42.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU= 2024-04-16T19:55:42.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU=', 'Content-Length': '0'} 2024-04-16T19:55:42.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d91fb433fc79296-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.169 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.170 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.170 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.170 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.170 DEBUG:boto:Method: GET 2024-04-16T19:55:42.170 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:42.170 DEBUG:boto:Data: 2024-04-16T19:55:42.170 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.170 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.170 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.170 DEBUG:boto:Params: {} 2024-04-16T19:55:42.171 DEBUG:boto:Token: None 2024-04-16T19:55:42.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001311fb7e36e4c347-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.214 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.215 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.215 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.215 DEBUG:boto:path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.215 DEBUG:boto:auth_path=/yrjefh-50/?prefix=a 2024-04-16T19:55:42.215 DEBUG:boto:Method: GET 2024-04-16T19:55:42.215 DEBUG:boto:Path: /yrjefh-50/?prefix=a 2024-04-16T19:55:42.215 DEBUG:boto:Data: 2024-04-16T19:55:42.215 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.215 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.215 DEBUG:boto:Params: {} 2024-04-16T19:55:42.216 DEBUG:boto:Token: None 2024-04-16T19:55:42.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083444184df440ffc-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.218 DEBUG:boto:b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:42.219 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.219 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.219 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.219 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.219 DEBUG:boto:Method: GET 2024-04-16T19:55:42.219 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.219 DEBUG:boto:Data: 2024-04-16T19:55:42.220 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.220 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.220 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.220 DEBUG:boto:Params: {} 2024-04-16T19:55:42.220 DEBUG:boto:Token: None 2024-04-16T19:55:42.220 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T19:55:42.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T19:55:42.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab70e066517be6e-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.262 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.262 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.263 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.263 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.263 DEBUG:boto:Method: GET 2024-04-16T19:55:42.263 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.263 DEBUG:boto:Data: 2024-04-16T19:55:42.263 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.263 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.263 DEBUG:boto:Params: {} 2024-04-16T19:55:42.263 DEBUG:boto:Token: None 2024-04-16T19:55:42.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T19:55:42.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T19:55:42.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a00f92063e9e4c-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.266 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.266 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.266 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.266 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.266 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.266 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.266 DEBUG:boto:Data: 2024-04-16T19:55:42.266 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.266 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.266 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.267 DEBUG:boto:Params: {} 2024-04-16T19:55:42.267 DEBUG:boto:Token: None 2024-04-16T19:55:42.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI= 2024-04-16T19:55:42.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI=', 'Content-Length': '0'} 2024-04-16T19:55:42.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0db546abba284f-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.269 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.270 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.270 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.270 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.270 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.270 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.270 DEBUG:boto:Data: 2024-04-16T19:55:42.270 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.270 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.270 DEBUG:boto:Params: {} 2024-04-16T19:55:42.270 DEBUG:boto:Token: None 2024-04-16T19:55:42.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI= 2024-04-16T19:55:42.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI=', 'Content-Length': '0'} 2024-04-16T19:55:42.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5158020d8ed520-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:42.273 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.273 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.273 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.273 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.273 DEBUG:boto:Method: GET 2024-04-16T19:55:42.273 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.273 DEBUG:boto:Data: 2024-04-16T19:55:42.273 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.274 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.274 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.274 DEBUG:boto:Params: {} 2024-04-16T19:55:42.274 DEBUG:boto:Token: None 2024-04-16T19:55:42.274 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T19:55:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T19:55:42.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ef86ca813a6055-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.318 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.319 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.319 DEBUG:boto:path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.319 DEBUG:boto:auth_path=/yrjefh-50/b?versionId=null 2024-04-16T19:55:42.319 DEBUG:boto:Method: GET 2024-04-16T19:55:42.319 DEBUG:boto:Path: /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.319 DEBUG:boto:Data: 2024-04-16T19:55:42.319 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.319 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.319 DEBUG:boto:Params: {} 2024-04-16T19:55:42.319 DEBUG:boto:Token: None 2024-04-16T19:55:42.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b?versionId=null 2024-04-16T19:55:42.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T19:55:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T19:55:42.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5adc3e4ef7c9d55-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.322 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.322 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.322 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.322 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.322 DEBUG:boto:Method: GET 2024-04-16T19:55:42.323 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:42.323 DEBUG:boto:Data: 2024-04-16T19:55:42.323 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.323 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.323 DEBUG:boto:Params: {} 2024-04-16T19:55:42.323 DEBUG:boto:Token: None 2024-04-16T19:55:42.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983113bc9b4b91a3-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.366 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.367 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.367 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.367 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.367 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.367 DEBUG:boto:Method: GET 2024-04-16T19:55:42.367 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:42.367 DEBUG:boto:Data: 2024-04-16T19:55:42.367 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.368 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.368 DEBUG:boto:Params: {} 2024-04-16T19:55:42.368 DEBUG:boto:Token: None 2024-04-16T19:55:42.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005796bcd374343013-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.371 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.371 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.371 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.371 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.371 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:42.371 DEBUG:boto:Data: 2024-04-16T19:55:42.371 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.372 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.372 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.372 DEBUG:boto:Params: {} 2024-04-16T19:55:42.372 DEBUG:boto:Token: None 2024-04-16T19:55:42.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b 2024-04-16T19:55:42.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo= 2024-04-16T19:55:42.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo=', 'Content-Length': '0'} 2024-04-16T19:55:42.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac41e47502f6a6b3-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.374 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.374 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.374 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.374 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:42.375 DEBUG:boto:Data: 2024-04-16T19:55:42.375 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.375 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.375 DEBUG:boto:Params: {} 2024-04-16T19:55:42.375 DEBUG:boto:Token: None 2024-04-16T19:55:42.375 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b 2024-04-16T19:55:42.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo= 2024-04-16T19:55:42.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo=', 'Content-Length': '0'} 2024-04-16T19:55:42.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606df98a267d6b0b-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-16T19:55:42.377 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.377 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.377 DEBUG:boto:Method: GET 2024-04-16T19:55:42.377 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:42.378 DEBUG:boto:Data: 2024-04-16T19:55:42.378 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.378 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.378 DEBUG:boto:Params: {} 2024-04-16T19:55:42.378 DEBUG:boto:Token: None 2024-04-16T19:55:42.378 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b 2024-04-16T19:55:42.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA= 2024-04-16T19:55:42.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA=', 'Content-Length': '0'} 2024-04-16T19:55:42.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b4bcbd84d534c4-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.422 DEBUG:boto:path=/yrjefh-50/b 2024-04-16T19:55:42.423 DEBUG:boto:auth_path=/yrjefh-50/b 2024-04-16T19:55:42.423 DEBUG:boto:Method: GET 2024-04-16T19:55:42.423 DEBUG:boto:Path: /yrjefh-50/b 2024-04-16T19:55:42.423 DEBUG:boto:Data: 2024-04-16T19:55:42.423 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.423 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.423 DEBUG:boto:Params: {} 2024-04-16T19:55:42.423 DEBUG:boto:Token: None 2024-04-16T19:55:42.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/b 2024-04-16T19:55:42.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA= 2024-04-16T19:55:42.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA=', 'Content-Length': '0'} 2024-04-16T19:55:42.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490c72b0fc4b0bb0-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.426 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.426 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.426 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.426 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.426 DEBUG:boto:Method: GET 2024-04-16T19:55:42.426 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:42.426 DEBUG:boto:Data: 2024-04-16T19:55:42.426 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.426 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.426 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.427 DEBUG:boto:Params: {} 2024-04-16T19:55:42.427 DEBUG:boto:Token: None 2024-04-16T19:55:42.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea57f273be80f265-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.429 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.430 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.430 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.430 DEBUG:boto:path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.430 DEBUG:boto:auth_path=/yrjefh-50/?prefix=b 2024-04-16T19:55:42.430 DEBUG:boto:Method: GET 2024-04-16T19:55:42.431 DEBUG:boto:Path: /yrjefh-50/?prefix=b 2024-04-16T19:55:42.431 DEBUG:boto:Data: 2024-04-16T19:55:42.431 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.431 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.431 DEBUG:boto:Params: {} 2024-04-16T19:55:42.431 DEBUG:boto:Token: None 2024-04-16T19:55:42.431 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b3b736c8b794f3-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.434 DEBUG:boto:b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:42.434 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.434 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.434 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.435 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.435 DEBUG:boto:Method: GET 2024-04-16T19:55:42.435 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.435 DEBUG:boto:Data: 2024-04-16T19:55:42.435 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.435 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.435 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.435 DEBUG:boto:Params: {} 2024-04-16T19:55:42.435 DEBUG:boto:Token: None 2024-04-16T19:55:42.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T19:55:42.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T19:55:42.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc33c082827bcf2b-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.478 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.478 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.479 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.479 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.479 DEBUG:boto:Method: GET 2024-04-16T19:55:42.479 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.479 DEBUG:boto:Data: 2024-04-16T19:55:42.479 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.479 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.479 DEBUG:boto:Params: {} 2024-04-16T19:55:42.479 DEBUG:boto:Token: None 2024-04-16T19:55:42.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T19:55:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T19:55:42.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee4b0ba548bdcee-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.482 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.482 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.482 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.482 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.482 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.482 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.482 DEBUG:boto:Data: 2024-04-16T19:55:42.482 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.482 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.483 DEBUG:boto:Params: {} 2024-04-16T19:55:42.483 DEBUG:boto:Token: None 2024-04-16T19:55:42.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA= 2024-04-16T19:55:42.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA=', 'Content-Length': '0'} 2024-04-16T19:55:42.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010aeabe9b47680c1-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.485 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.485 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.485 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.485 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.485 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.485 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.486 DEBUG:boto:Data: 2024-04-16T19:55:42.486 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.486 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.486 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.486 DEBUG:boto:Params: {} 2024-04-16T19:55:42.486 DEBUG:boto:Token: None 2024-04-16T19:55:42.486 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA= 2024-04-16T19:55:42.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA=', 'Content-Length': '0'} 2024-04-16T19:55:42.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007338fd6ef82acd84-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:42.488 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.488 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.489 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.489 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.489 DEBUG:boto:Method: GET 2024-04-16T19:55:42.489 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.489 DEBUG:boto:Data: 2024-04-16T19:55:42.489 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.489 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.489 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.489 DEBUG:boto:Params: {} 2024-04-16T19:55:42.489 DEBUG:boto:Token: None 2024-04-16T19:55:42.489 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.490 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T19:55:42.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T19:55:42.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e5df9a27de90354-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.534 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.534 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.535 DEBUG:boto:path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.535 DEBUG:boto:auth_path=/yrjefh-50/c?versionId=null 2024-04-16T19:55:42.535 DEBUG:boto:Method: GET 2024-04-16T19:55:42.535 DEBUG:boto:Path: /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.535 DEBUG:boto:Data: 2024-04-16T19:55:42.535 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.535 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.535 DEBUG:boto:Params: {} 2024-04-16T19:55:42.535 DEBUG:boto:Token: None 2024-04-16T19:55:42.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c?versionId=null 2024-04-16T19:55:42.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T19:55:42.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T19:55:42.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e07f37d5ffc9cb-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.538 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.538 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.538 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.538 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.538 DEBUG:boto:Method: GET 2024-04-16T19:55:42.538 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:42.538 DEBUG:boto:Data: 2024-04-16T19:55:42.539 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.539 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.539 DEBUG:boto:Params: {} 2024-04-16T19:55:42.539 DEBUG:boto:Token: None 2024-04-16T19:55:42.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7472325a1ba99e-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.542 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.543 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.543 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.543 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.543 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.543 DEBUG:boto:Method: GET 2024-04-16T19:55:42.543 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:42.543 DEBUG:boto:Data: 2024-04-16T19:55:42.543 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.544 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.544 DEBUG:boto:Params: {} 2024-04-16T19:55:42.544 DEBUG:boto:Token: None 2024-04-16T19:55:42.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002380b4791f8ebdb0-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.547 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.547 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.547 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.547 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.547 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:42.547 DEBUG:boto:Data: 2024-04-16T19:55:42.548 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.548 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.548 DEBUG:boto:Params: {} 2024-04-16T19:55:42.548 DEBUG:boto:Token: None 2024-04-16T19:55:42.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c 2024-04-16T19:55:42.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg= 2024-04-16T19:55:42.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg=', 'Content-Length': '0'} 2024-04-16T19:55:42.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fac2b6201fb471-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.550 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.550 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.551 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.551 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:42.551 DEBUG:boto:Data: 2024-04-16T19:55:42.551 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.551 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.551 DEBUG:boto:Params: {} 2024-04-16T19:55:42.551 DEBUG:boto:Token: None 2024-04-16T19:55:42.551 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c 2024-04-16T19:55:42.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg= 2024-04-16T19:55:42.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg=', 'Content-Length': '0'} 2024-04-16T19:55:42.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36fb24366858bca-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-16T19:55:42.553 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.553 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.554 DEBUG:boto:Method: GET 2024-04-16T19:55:42.554 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:42.554 DEBUG:boto:Data: 2024-04-16T19:55:42.554 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.554 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.554 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.554 DEBUG:boto:Params: {} 2024-04-16T19:55:42.554 DEBUG:boto:Token: None 2024-04-16T19:55:42.554 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c 2024-04-16T19:55:42.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA= 2024-04-16T19:55:42.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA=', 'Content-Length': '0'} 2024-04-16T19:55:42.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bd12d6674c971f-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.598 DEBUG:boto:path=/yrjefh-50/c 2024-04-16T19:55:42.598 DEBUG:boto:auth_path=/yrjefh-50/c 2024-04-16T19:55:42.599 DEBUG:boto:Method: GET 2024-04-16T19:55:42.599 DEBUG:boto:Path: /yrjefh-50/c 2024-04-16T19:55:42.599 DEBUG:boto:Data: 2024-04-16T19:55:42.599 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.599 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.599 DEBUG:boto:Params: {} 2024-04-16T19:55:42.599 DEBUG:boto:Token: None 2024-04-16T19:55:42.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/c 2024-04-16T19:55:42.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA= 2024-04-16T19:55:42.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA=', 'Content-Length': '0'} 2024-04-16T19:55:42.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d7048e057ead37-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.602 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.602 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.602 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.602 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.602 DEBUG:boto:Method: GET 2024-04-16T19:55:42.602 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:42.602 DEBUG:boto:Data: 2024-04-16T19:55:42.602 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.602 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.602 DEBUG:boto:Params: {} 2024-04-16T19:55:42.603 DEBUG:boto:Token: None 2024-04-16T19:55:42.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007875f20eacf76924-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.605 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.606 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.606 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.606 DEBUG:boto:path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.606 DEBUG:boto:auth_path=/yrjefh-50/?prefix=c 2024-04-16T19:55:42.606 DEBUG:boto:Method: GET 2024-04-16T19:55:42.606 DEBUG:boto:Path: /yrjefh-50/?prefix=c 2024-04-16T19:55:42.606 DEBUG:boto:Data: 2024-04-16T19:55:42.606 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.606 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.607 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.607 DEBUG:boto:Params: {} 2024-04-16T19:55:42.607 DEBUG:boto:Token: None 2024-04-16T19:55:42.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb71f28d1e11a4ed-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.609 DEBUG:boto:b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:42.610 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.610 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.610 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.610 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.610 DEBUG:boto:Method: GET 2024-04-16T19:55:42.610 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.610 DEBUG:boto:Data: 2024-04-16T19:55:42.610 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.610 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.610 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.610 DEBUG:boto:Params: {} 2024-04-16T19:55:42.611 DEBUG:boto:Token: None 2024-04-16T19:55:42.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T19:55:42.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T19:55:42.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a22f4dcb1a1087-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.654 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.654 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.655 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.655 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.655 DEBUG:boto:Method: GET 2024-04-16T19:55:42.655 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.655 DEBUG:boto:Data: 2024-04-16T19:55:42.655 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.655 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.655 DEBUG:boto:Params: {} 2024-04-16T19:55:42.655 DEBUG:boto:Token: None 2024-04-16T19:55:42.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T19:55:42.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T19:55:42.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13755c0aae3a1d2-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.658 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.658 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.658 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.658 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.658 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.659 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.659 DEBUG:boto:Data: 2024-04-16T19:55:42.659 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.659 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.659 DEBUG:boto:Params: {} 2024-04-16T19:55:42.659 DEBUG:boto:Token: None 2024-04-16T19:55:42.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI= 2024-04-16T19:55:42.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI=', 'Content-Length': '0'} 2024-04-16T19:55:42.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef78f5261e7693b-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.662 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.662 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.662 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.662 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.662 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.662 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.662 DEBUG:boto:Data: 2024-04-16T19:55:42.662 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.663 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.663 DEBUG:boto:Params: {} 2024-04-16T19:55:42.663 DEBUG:boto:Token: None 2024-04-16T19:55:42.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI= 2024-04-16T19:55:42.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI=', 'Content-Length': '0'} 2024-04-16T19:55:42.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d638068c9842f9-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:42.666 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.666 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.666 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.666 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.666 DEBUG:boto:Method: GET 2024-04-16T19:55:42.666 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.666 DEBUG:boto:Data: 2024-04-16T19:55:42.666 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.666 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.666 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.666 DEBUG:boto:Params: {} 2024-04-16T19:55:42.667 DEBUG:boto:Token: None 2024-04-16T19:55:42.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T19:55:42.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T19:55:42.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13ee5e26bd52490-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.710 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.710 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.711 DEBUG:boto:path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.711 DEBUG:boto:auth_path=/yrjefh-50/d?versionId=null 2024-04-16T19:55:42.711 DEBUG:boto:Method: GET 2024-04-16T19:55:42.711 DEBUG:boto:Path: /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.711 DEBUG:boto:Data: 2024-04-16T19:55:42.711 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.711 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.711 DEBUG:boto:Params: {} 2024-04-16T19:55:42.711 DEBUG:boto:Token: None 2024-04-16T19:55:42.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d?versionId=null 2024-04-16T19:55:42.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T19:55:42.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T19:55:42.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9072dc18896cbb-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.714 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.714 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.714 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.714 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.714 DEBUG:boto:Method: GET 2024-04-16T19:55:42.714 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:42.714 DEBUG:boto:Data: 2024-04-16T19:55:42.714 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.714 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.714 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.715 DEBUG:boto:Params: {} 2024-04-16T19:55:42.715 DEBUG:boto:Token: None 2024-04-16T19:55:42.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b07d83ab41ecc0-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.717 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.718 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.718 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.718 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.718 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.718 DEBUG:boto:Method: GET 2024-04-16T19:55:42.718 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:42.718 DEBUG:boto:Data: 2024-04-16T19:55:42.718 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.718 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.719 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.719 DEBUG:boto:Params: {} 2024-04-16T19:55:42.719 DEBUG:boto:Token: None 2024-04-16T19:55:42.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007699174c1f2e61dc-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.721 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.722 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.722 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.722 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.722 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:42.722 DEBUG:boto:Data: 2024-04-16T19:55:42.722 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.722 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.722 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.722 DEBUG:boto:Params: {} 2024-04-16T19:55:42.723 DEBUG:boto:Token: None 2024-04-16T19:55:42.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d 2024-04-16T19:55:42.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8= 2024-04-16T19:55:42.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8=', 'Content-Length': '0'} 2024-04-16T19:55:42.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003391c3de7f5e3202-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.725 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.725 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.726 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.726 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:42.726 DEBUG:boto:Data: 2024-04-16T19:55:42.726 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.726 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.726 DEBUG:boto:Params: {} 2024-04-16T19:55:42.726 DEBUG:boto:Token: None 2024-04-16T19:55:42.726 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d 2024-04-16T19:55:42.726 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8= 2024-04-16T19:55:42.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8=', 'Content-Length': '0'} 2024-04-16T19:55:42.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047555504c92d3090-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-16T19:55:42.729 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.729 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.729 DEBUG:boto:Method: GET 2024-04-16T19:55:42.729 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:42.729 DEBUG:boto:Data: 2024-04-16T19:55:42.729 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.729 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.729 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.729 DEBUG:boto:Params: {} 2024-04-16T19:55:42.729 DEBUG:boto:Token: None 2024-04-16T19:55:42.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d 2024-04-16T19:55:42.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho= 2024-04-16T19:55:42.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho=', 'Content-Length': '0'} 2024-04-16T19:55:42.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9cd03cc3e42a1a-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.774 DEBUG:boto:path=/yrjefh-50/d 2024-04-16T19:55:42.775 DEBUG:boto:auth_path=/yrjefh-50/d 2024-04-16T19:55:42.775 DEBUG:boto:Method: GET 2024-04-16T19:55:42.775 DEBUG:boto:Path: /yrjefh-50/d 2024-04-16T19:55:42.775 DEBUG:boto:Data: 2024-04-16T19:55:42.775 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.775 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.775 DEBUG:boto:Params: {} 2024-04-16T19:55:42.775 DEBUG:boto:Token: None 2024-04-16T19:55:42.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/d 2024-04-16T19:55:42.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho= 2024-04-16T19:55:42.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho=', 'Content-Length': '0'} 2024-04-16T19:55:42.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abb316ee4696a56-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.778 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.778 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.778 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.778 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.778 DEBUG:boto:Method: GET 2024-04-16T19:55:42.778 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:42.778 DEBUG:boto:Data: 2024-04-16T19:55:42.779 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.779 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.779 DEBUG:boto:Params: {} 2024-04-16T19:55:42.779 DEBUG:boto:Token: None 2024-04-16T19:55:42.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027527f3614f2133d-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.781 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.782 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.782 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.782 DEBUG:boto:path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.782 DEBUG:boto:auth_path=/yrjefh-50/?prefix=d 2024-04-16T19:55:42.782 DEBUG:boto:Method: GET 2024-04-16T19:55:42.782 DEBUG:boto:Path: /yrjefh-50/?prefix=d 2024-04-16T19:55:42.782 DEBUG:boto:Data: 2024-04-16T19:55:42.782 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.782 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.782 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.782 DEBUG:boto:Params: {} 2024-04-16T19:55:42.783 DEBUG:boto:Token: None 2024-04-16T19:55:42.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee38d3e1f11a727-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.785 DEBUG:boto:b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:42.786 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.786 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.786 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.786 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.786 DEBUG:boto:Method: GET 2024-04-16T19:55:42.786 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.786 DEBUG:boto:Data: 2024-04-16T19:55:42.786 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.786 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.786 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.787 DEBUG:boto:Params: {} 2024-04-16T19:55:42.787 DEBUG:boto:Token: None 2024-04-16T19:55:42.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T19:55:42.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T19:55:42.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccbda6ee3b7c686-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.830 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.831 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.831 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.831 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.831 DEBUG:boto:Method: GET 2024-04-16T19:55:42.831 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.831 DEBUG:boto:Data: 2024-04-16T19:55:42.831 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.831 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.831 DEBUG:boto:Params: {} 2024-04-16T19:55:42.832 DEBUG:boto:Token: None 2024-04-16T19:55:42.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T19:55:42.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T19:55:42.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73739dd9895cb3c-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.834 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.835 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.835 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.835 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.835 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.835 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.835 DEBUG:boto:Data: 2024-04-16T19:55:42.835 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.835 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.835 DEBUG:boto:Params: {} 2024-04-16T19:55:42.835 DEBUG:boto:Token: None 2024-04-16T19:55:42.835 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI= 2024-04-16T19:55:42.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI=', 'Content-Length': '0'} 2024-04-16T19:55:42.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61958e4bbd3fb3c-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.838 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.838 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.839 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.839 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.839 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.839 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.839 DEBUG:boto:Data: 2024-04-16T19:55:42.839 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.839 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.839 DEBUG:boto:Params: {} 2024-04-16T19:55:42.839 DEBUG:boto:Token: None 2024-04-16T19:55:42.839 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI= 2024-04-16T19:55:42.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI=', 'Content-Length': '0'} 2024-04-16T19:55:42.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097826f6e8abc2864-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:42.842 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.842 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.842 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.843 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.843 DEBUG:boto:Method: GET 2024-04-16T19:55:42.843 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.843 DEBUG:boto:Data: 2024-04-16T19:55:42.843 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.843 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.843 DEBUG:boto:Params: {} 2024-04-16T19:55:42.843 DEBUG:boto:Token: None 2024-04-16T19:55:42.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T19:55:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T19:55:42.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a43a83596a45954-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.886 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.886 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.886 DEBUG:boto:path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.887 DEBUG:boto:auth_path=/yrjefh-50/e?versionId=null 2024-04-16T19:55:42.887 DEBUG:boto:Method: GET 2024-04-16T19:55:42.887 DEBUG:boto:Path: /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.887 DEBUG:boto:Data: 2024-04-16T19:55:42.887 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.887 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.887 DEBUG:boto:Params: {} 2024-04-16T19:55:42.887 DEBUG:boto:Token: None 2024-04-16T19:55:42.887 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e?versionId=null 2024-04-16T19:55:42.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T19:55:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T19:55:42.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001386cf55a11104e4-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.890 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.890 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.890 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.890 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.891 DEBUG:boto:Method: GET 2024-04-16T19:55:42.891 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:42.891 DEBUG:boto:Data: 2024-04-16T19:55:42.891 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.891 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.891 DEBUG:boto:Params: {} 2024-04-16T19:55:42.891 DEBUG:boto:Token: None 2024-04-16T19:55:42.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599c115043fa9548-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.894 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.894 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.894 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.894 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.894 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.895 DEBUG:boto:Method: GET 2024-04-16T19:55:42.895 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:42.895 DEBUG:boto:Data: 2024-04-16T19:55:42.895 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.895 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.895 DEBUG:boto:Params: {} 2024-04-16T19:55:42.895 DEBUG:boto:Token: None 2024-04-16T19:55:42.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000378c1ab6402bd887-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.898 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.898 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.898 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.898 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.899 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:42.899 DEBUG:boto:Data: 2024-04-16T19:55:42.899 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.899 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.899 DEBUG:boto:Params: {} 2024-04-16T19:55:42.899 DEBUG:boto:Token: None 2024-04-16T19:55:42.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e 2024-04-16T19:55:42.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0= 2024-04-16T19:55:42.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0=', 'Content-Length': '0'} 2024-04-16T19:55:42.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd11e4cd4f6eaca-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.901 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.901 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.901 DEBUG:boto:Method: HEAD 2024-04-16T19:55:42.902 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:42.902 DEBUG:boto:Data: 2024-04-16T19:55:42.902 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.902 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.902 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.902 DEBUG:boto:Params: {} 2024-04-16T19:55:42.902 DEBUG:boto:Token: None 2024-04-16T19:55:42.902 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e 2024-04-16T19:55:42.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0= 2024-04-16T19:55:42.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0=', 'Content-Length': '0'} 2024-04-16T19:55:42.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bed10a5e20c2047-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-16T19:55:42.904 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.904 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.905 DEBUG:boto:Method: GET 2024-04-16T19:55:42.905 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:42.905 DEBUG:boto:Data: 2024-04-16T19:55:42.905 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.905 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.905 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.905 DEBUG:boto:Params: {} 2024-04-16T19:55:42.905 DEBUG:boto:Token: None 2024-04-16T19:55:42.905 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e 2024-04-16T19:55:42.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw= 2024-04-16T19:55:42.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw=', 'Content-Length': '0'} 2024-04-16T19:55:42.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb3672a2fa8aafa-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.951 DEBUG:boto:path=/yrjefh-50/e 2024-04-16T19:55:42.951 DEBUG:boto:auth_path=/yrjefh-50/e 2024-04-16T19:55:42.951 DEBUG:boto:Method: GET 2024-04-16T19:55:42.951 DEBUG:boto:Path: /yrjefh-50/e 2024-04-16T19:55:42.951 DEBUG:boto:Data: 2024-04-16T19:55:42.951 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.951 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.951 DEBUG:boto:Params: {} 2024-04-16T19:55:42.951 DEBUG:boto:Token: None 2024-04-16T19:55:42.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/e 2024-04-16T19:55:42.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw= 2024-04-16T19:55:42.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw=', 'Content-Length': '0'} 2024-04-16T19:55:42.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef7e8c4efc9df33-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.954 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.954 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.955 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.955 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.955 DEBUG:boto:Method: GET 2024-04-16T19:55:42.955 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:42.955 DEBUG:boto:Data: 2024-04-16T19:55:42.955 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.955 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.955 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.955 DEBUG:boto:Params: {} 2024-04-16T19:55:42.955 DEBUG:boto:Token: None 2024-04-16T19:55:42.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14562b4bf94f0f0-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.958 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.958 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:42.958 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:42.958 DEBUG:boto:path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.958 DEBUG:boto:auth_path=/yrjefh-50/?prefix=e 2024-04-16T19:55:42.958 DEBUG:boto:Method: GET 2024-04-16T19:55:42.958 DEBUG:boto:Path: /yrjefh-50/?prefix=e 2024-04-16T19:55:42.959 DEBUG:boto:Data: 2024-04-16T19:55:42.959 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:42.959 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.959 DEBUG:boto:Params: {} 2024-04-16T19:55:42.959 DEBUG:boto:Token: None 2024-04-16T19:55:42.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/ 2024-04-16T19:55:42.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T19:55:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T19:55:42.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f0786831235f88-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:42.962 DEBUG:boto:b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:42.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:42.962 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:42.962 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:42.963 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:42.963 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:42.963 DEBUG:boto:Method: GET 2024-04-16T19:55:42.963 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:42.963 DEBUG:boto:Data: 2024-04-16T19:55:42.963 DEBUG:boto:Headers: {} 2024-04-16T19:55:42.963 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:42.963 DEBUG:boto:Port: 8000 2024-04-16T19:55:42.963 DEBUG:boto:Params: {} 2024-04-16T19:55:42.963 DEBUG:boto:Token: None 2024-04-16T19:55:42.963 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:42 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:42.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iF/zG/rVkqChw0QMlNv0vV22/fc= 2024-04-16T19:55:42.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iF/zG/rVkqChw0QMlNv0vV22/fc=', 'Content-Length': '0'} 2024-04-16T19:55:42.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2fbdf15fffc868-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.006 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.006 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.007 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.007 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.007 DEBUG:boto:Method: GET 2024-04-16T19:55:43.007 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.007 DEBUG:boto:Data: 2024-04-16T19:55:43.007 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.007 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.007 DEBUG:boto:Params: {} 2024-04-16T19:55:43.007 DEBUG:boto:Token: None 2024-04-16T19:55:43.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T19:55:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T19:55:43.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a960dbeb068b8ddb-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.010 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.010 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.010 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.010 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.011 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.011 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.011 DEBUG:boto:Data: 2024-04-16T19:55:43.011 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.011 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.011 DEBUG:boto:Params: {} 2024-04-16T19:55:43.011 DEBUG:boto:Token: None 2024-04-16T19:55:43.011 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8= 2024-04-16T19:55:43.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8=', 'Content-Length': '0'} 2024-04-16T19:55:43.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2677f11c7f45983-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.014 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.014 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.014 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.014 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.014 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.014 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.014 DEBUG:boto:Data: 2024-04-16T19:55:43.015 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.015 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.015 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.015 DEBUG:boto:Params: {} 2024-04-16T19:55:43.015 DEBUG:boto:Token: None 2024-04-16T19:55:43.015 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8= 2024-04-16T19:55:43.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8=', 'Content-Length': '0'} 2024-04-16T19:55:43.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000499b2f0a2fef08fc-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:43.017 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.018 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.018 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.018 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.018 DEBUG:boto:Method: GET 2024-04-16T19:55:43.018 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.018 DEBUG:boto:Data: 2024-04-16T19:55:43.018 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.018 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.018 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.018 DEBUG:boto:Params: {} 2024-04-16T19:55:43.018 DEBUG:boto:Token: None 2024-04-16T19:55:43.019 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.019 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T19:55:43.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T19:55:43.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d9973f5d4ca7f9-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.062 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.063 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.063 DEBUG:boto:path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.063 DEBUG:boto:auth_path=/yrjefh-50/f?versionId=null 2024-04-16T19:55:43.063 DEBUG:boto:Method: GET 2024-04-16T19:55:43.063 DEBUG:boto:Path: /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.063 DEBUG:boto:Data: 2024-04-16T19:55:43.063 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.063 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.064 DEBUG:boto:Params: {} 2024-04-16T19:55:43.064 DEBUG:boto:Token: None 2024-04-16T19:55:43.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f?versionId=null 2024-04-16T19:55:43.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T19:55:43.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T19:55:43.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000560b1b4a8e2e61e3-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.067 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.067 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.067 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.067 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.067 DEBUG:boto:Method: GET 2024-04-16T19:55:43.067 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:43.067 DEBUG:boto:Data: 2024-04-16T19:55:43.067 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.067 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.068 DEBUG:boto:Params: {} 2024-04-16T19:55:43.068 DEBUG:boto:Token: None 2024-04-16T19:55:43.068 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007090b74a17a19518-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.071 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.071 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.071 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.071 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.072 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.072 DEBUG:boto:Method: GET 2024-04-16T19:55:43.072 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:43.072 DEBUG:boto:Data: 2024-04-16T19:55:43.072 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.072 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.072 DEBUG:boto:Params: {} 2024-04-16T19:55:43.072 DEBUG:boto:Token: None 2024-04-16T19:55:43.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211068b1f451bd04-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.118 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.119 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.119 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.119 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.119 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:43.119 DEBUG:boto:Data: 2024-04-16T19:55:43.119 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.119 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.120 DEBUG:boto:Params: {} 2024-04-16T19:55:43.120 DEBUG:boto:Token: None 2024-04-16T19:55:43.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f 2024-04-16T19:55:43.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg= 2024-04-16T19:55:43.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg=', 'Content-Length': '0'} 2024-04-16T19:55:43.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b908f9211658b5-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.122 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.122 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.123 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.123 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:43.123 DEBUG:boto:Data: 2024-04-16T19:55:43.123 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.123 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.123 DEBUG:boto:Params: {} 2024-04-16T19:55:43.123 DEBUG:boto:Token: None 2024-04-16T19:55:43.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f 2024-04-16T19:55:43.123 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg= 2024-04-16T19:55:43.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg=', 'Content-Length': '0'} 2024-04-16T19:55:43.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cbec0168b96f76-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-16T19:55:43.126 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.126 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.126 DEBUG:boto:Method: GET 2024-04-16T19:55:43.126 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:43.126 DEBUG:boto:Data: 2024-04-16T19:55:43.126 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.126 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.127 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.127 DEBUG:boto:Params: {} 2024-04-16T19:55:43.127 DEBUG:boto:Token: None 2024-04-16T19:55:43.127 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f 2024-04-16T19:55:43.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg= 2024-04-16T19:55:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg=', 'Content-Length': '0'} 2024-04-16T19:55:43.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032abd57693e80f06-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.170 DEBUG:boto:path=/yrjefh-50/f 2024-04-16T19:55:43.171 DEBUG:boto:auth_path=/yrjefh-50/f 2024-04-16T19:55:43.171 DEBUG:boto:Method: GET 2024-04-16T19:55:43.171 DEBUG:boto:Path: /yrjefh-50/f 2024-04-16T19:55:43.171 DEBUG:boto:Data: 2024-04-16T19:55:43.171 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.171 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.171 DEBUG:boto:Params: {} 2024-04-16T19:55:43.171 DEBUG:boto:Token: None 2024-04-16T19:55:43.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/f 2024-04-16T19:55:43.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg= 2024-04-16T19:55:43.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg=', 'Content-Length': '0'} 2024-04-16T19:55:43.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af2f1691bc52f7a-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.174 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.174 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.174 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.174 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.175 DEBUG:boto:Method: GET 2024-04-16T19:55:43.175 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:43.175 DEBUG:boto:Data: 2024-04-16T19:55:43.175 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.175 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.175 DEBUG:boto:Params: {} 2024-04-16T19:55:43.175 DEBUG:boto:Token: None 2024-04-16T19:55:43.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506055cb07c106b8-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.178 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.178 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.178 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.178 DEBUG:boto:path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.179 DEBUG:boto:auth_path=/yrjefh-50/?prefix=f 2024-04-16T19:55:43.179 DEBUG:boto:Method: GET 2024-04-16T19:55:43.179 DEBUG:boto:Path: /yrjefh-50/?prefix=f 2024-04-16T19:55:43.179 DEBUG:boto:Data: 2024-04-16T19:55:43.179 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.179 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.179 DEBUG:boto:Params: {} 2024-04-16T19:55:43.179 DEBUG:boto:Token: None 2024-04-16T19:55:43.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b566720c5bfc245f-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.182 DEBUG:boto:b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:43.183 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.183 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.183 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.183 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.183 DEBUG:boto:Method: GET 2024-04-16T19:55:43.183 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.183 DEBUG:boto:Data: 2024-04-16T19:55:43.183 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.183 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.183 DEBUG:boto:Params: {} 2024-04-16T19:55:43.184 DEBUG:boto:Token: None 2024-04-16T19:55:43.184 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T19:55:43.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T19:55:43.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000993747ab13a9a121-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.226 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.226 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.226 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.227 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.227 DEBUG:boto:Method: GET 2024-04-16T19:55:43.227 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.227 DEBUG:boto:Data: 2024-04-16T19:55:43.227 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.227 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.227 DEBUG:boto:Params: {} 2024-04-16T19:55:43.227 DEBUG:boto:Token: None 2024-04-16T19:55:43.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T19:55:43.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T19:55:43.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040147434fd61bcff-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.230 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.230 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.230 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.230 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.230 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.231 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.231 DEBUG:boto:Data: 2024-04-16T19:55:43.231 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.231 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.231 DEBUG:boto:Params: {} 2024-04-16T19:55:43.231 DEBUG:boto:Token: None 2024-04-16T19:55:43.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ= 2024-04-16T19:55:43.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ=', 'Content-Length': '0'} 2024-04-16T19:55:43.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488f7608e2e00aa0-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.233 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.234 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.234 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.234 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.234 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.234 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.234 DEBUG:boto:Data: 2024-04-16T19:55:43.234 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.234 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.234 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.234 DEBUG:boto:Params: {} 2024-04-16T19:55:43.234 DEBUG:boto:Token: None 2024-04-16T19:55:43.235 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ= 2024-04-16T19:55:43.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ=', 'Content-Length': '0'} 2024-04-16T19:55:43.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbc8b0343a313e9b-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:43.237 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.237 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.237 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.237 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.237 DEBUG:boto:Method: GET 2024-04-16T19:55:43.237 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.238 DEBUG:boto:Data: 2024-04-16T19:55:43.238 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.238 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.238 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.238 DEBUG:boto:Params: {} 2024-04-16T19:55:43.238 DEBUG:boto:Token: None 2024-04-16T19:55:43.238 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T19:55:43.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T19:55:43.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477004d2f8daaddd-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.282 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.283 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.283 DEBUG:boto:path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.283 DEBUG:boto:auth_path=/yrjefh-50/g?versionId=null 2024-04-16T19:55:43.283 DEBUG:boto:Method: GET 2024-04-16T19:55:43.283 DEBUG:boto:Path: /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.283 DEBUG:boto:Data: 2024-04-16T19:55:43.283 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.283 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.283 DEBUG:boto:Params: {} 2024-04-16T19:55:43.283 DEBUG:boto:Token: None 2024-04-16T19:55:43.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g?versionId=null 2024-04-16T19:55:43.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T19:55:43.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T19:55:43.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b50787ff2eda8d9-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.286 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.286 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.286 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.286 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.286 DEBUG:boto:Method: GET 2024-04-16T19:55:43.286 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:43.286 DEBUG:boto:Data: 2024-04-16T19:55:43.287 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.287 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.287 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.287 DEBUG:boto:Params: {} 2024-04-16T19:55:43.287 DEBUG:boto:Token: None 2024-04-16T19:55:43.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa716cb44f6c440c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.290 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.290 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.290 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.290 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.290 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.290 DEBUG:boto:Method: GET 2024-04-16T19:55:43.291 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:43.291 DEBUG:boto:Data: 2024-04-16T19:55:43.291 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.291 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.291 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.291 DEBUG:boto:Params: {} 2024-04-16T19:55:43.291 DEBUG:boto:Token: None 2024-04-16T19:55:43.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd840a17f97bf9b-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.294 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.294 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.294 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.294 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.294 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:43.294 DEBUG:boto:Data: 2024-04-16T19:55:43.295 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.295 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.295 DEBUG:boto:Params: {} 2024-04-16T19:55:43.295 DEBUG:boto:Token: None 2024-04-16T19:55:43.295 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g 2024-04-16T19:55:43.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE= 2024-04-16T19:55:43.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE=', 'Content-Length': '0'} 2024-04-16T19:55:43.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052456cbb4eb6bbaf-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.298 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.298 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.298 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.298 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:43.298 DEBUG:boto:Data: 2024-04-16T19:55:43.298 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.298 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.298 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.298 DEBUG:boto:Params: {} 2024-04-16T19:55:43.298 DEBUG:boto:Token: None 2024-04-16T19:55:43.299 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g 2024-04-16T19:55:43.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE= 2024-04-16T19:55:43.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE=', 'Content-Length': '0'} 2024-04-16T19:55:43.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e9d204f411b611-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-16T19:55:43.301 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.301 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.301 DEBUG:boto:Method: GET 2024-04-16T19:55:43.301 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:43.302 DEBUG:boto:Data: 2024-04-16T19:55:43.302 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.302 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.302 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.302 DEBUG:boto:Params: {} 2024-04-16T19:55:43.302 DEBUG:boto:Token: None 2024-04-16T19:55:43.302 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g 2024-04-16T19:55:43.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE= 2024-04-16T19:55:43.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE=', 'Content-Length': '0'} 2024-04-16T19:55:43.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce128e93105bfae-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.346 DEBUG:boto:path=/yrjefh-50/g 2024-04-16T19:55:43.347 DEBUG:boto:auth_path=/yrjefh-50/g 2024-04-16T19:55:43.347 DEBUG:boto:Method: GET 2024-04-16T19:55:43.347 DEBUG:boto:Path: /yrjefh-50/g 2024-04-16T19:55:43.347 DEBUG:boto:Data: 2024-04-16T19:55:43.347 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.347 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.347 DEBUG:boto:Params: {} 2024-04-16T19:55:43.347 DEBUG:boto:Token: None 2024-04-16T19:55:43.347 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/g 2024-04-16T19:55:43.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE= 2024-04-16T19:55:43.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE=', 'Content-Length': '0'} 2024-04-16T19:55:43.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c03cd118b0999f1-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.350 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.351 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.351 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.351 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.351 DEBUG:boto:Method: GET 2024-04-16T19:55:43.351 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:43.351 DEBUG:boto:Data: 2024-04-16T19:55:43.351 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.351 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.351 DEBUG:boto:Params: {} 2024-04-16T19:55:43.351 DEBUG:boto:Token: None 2024-04-16T19:55:43.352 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d26d53e2458a8f4e-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.354 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.355 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.355 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.355 DEBUG:boto:path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.355 DEBUG:boto:auth_path=/yrjefh-50/?prefix=g 2024-04-16T19:55:43.355 DEBUG:boto:Method: GET 2024-04-16T19:55:43.355 DEBUG:boto:Path: /yrjefh-50/?prefix=g 2024-04-16T19:55:43.355 DEBUG:boto:Data: 2024-04-16T19:55:43.355 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.355 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.355 DEBUG:boto:Params: {} 2024-04-16T19:55:43.356 DEBUG:boto:Token: None 2024-04-16T19:55:43.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043798b6333773f6c-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.359 DEBUG:boto:b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:43.359 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.359 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.359 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.359 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.359 DEBUG:boto:Method: GET 2024-04-16T19:55:43.359 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.360 DEBUG:boto:Data: 2024-04-16T19:55:43.360 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.360 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.360 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.360 DEBUG:boto:Params: {} 2024-04-16T19:55:43.360 DEBUG:boto:Token: None 2024-04-16T19:55:43.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T19:55:43.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T19:55:43.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc0a2f3ad1256b3-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.402 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.402 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.403 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.403 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.403 DEBUG:boto:Method: GET 2024-04-16T19:55:43.403 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.403 DEBUG:boto:Data: 2024-04-16T19:55:43.403 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.403 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.403 DEBUG:boto:Params: {} 2024-04-16T19:55:43.403 DEBUG:boto:Token: None 2024-04-16T19:55:43.403 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T19:55:43.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T19:55:43.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa33490d9138d92d-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.406 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.406 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.406 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.406 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.406 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.406 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.406 DEBUG:boto:Data: 2024-04-16T19:55:43.407 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.407 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.407 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.407 DEBUG:boto:Params: {} 2024-04-16T19:55:43.407 DEBUG:boto:Token: None 2024-04-16T19:55:43.407 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ= 2024-04-16T19:55:43.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ=', 'Content-Length': '0'} 2024-04-16T19:55:43.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e1c1261919018c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.409 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.409 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.409 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.409 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.410 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.410 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.410 DEBUG:boto:Data: 2024-04-16T19:55:43.410 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.410 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.410 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.410 DEBUG:boto:Params: {} 2024-04-16T19:55:43.410 DEBUG:boto:Token: None 2024-04-16T19:55:43.410 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.410 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ= 2024-04-16T19:55:43.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ=', 'Content-Length': '0'} 2024-04-16T19:55:43.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28a987b6819b877-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:43.413 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.413 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.413 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.413 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.413 DEBUG:boto:Method: GET 2024-04-16T19:55:43.413 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.413 DEBUG:boto:Data: 2024-04-16T19:55:43.413 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.413 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.414 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.414 DEBUG:boto:Params: {} 2024-04-16T19:55:43.414 DEBUG:boto:Token: None 2024-04-16T19:55:43.414 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.414 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T19:55:43.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T19:55:43.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000581bd69136d9db88-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.458 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.458 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.458 DEBUG:boto:path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.459 DEBUG:boto:auth_path=/yrjefh-50/h?versionId=null 2024-04-16T19:55:43.459 DEBUG:boto:Method: GET 2024-04-16T19:55:43.459 DEBUG:boto:Path: /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.459 DEBUG:boto:Data: 2024-04-16T19:55:43.459 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.459 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.459 DEBUG:boto:Params: {} 2024-04-16T19:55:43.459 DEBUG:boto:Token: None 2024-04-16T19:55:43.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h?versionId=null 2024-04-16T19:55:43.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T19:55:43.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T19:55:43.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a20adbdc65acf5f-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.462 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.462 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.462 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.462 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.462 DEBUG:boto:Method: GET 2024-04-16T19:55:43.462 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:43.462 DEBUG:boto:Data: 2024-04-16T19:55:43.462 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.463 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.463 DEBUG:boto:Params: {} 2024-04-16T19:55:43.463 DEBUG:boto:Token: None 2024-04-16T19:55:43.463 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd6f75b34b7b542-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.466 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.466 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.466 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.466 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.466 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.466 DEBUG:boto:Method: GET 2024-04-16T19:55:43.466 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:43.467 DEBUG:boto:Data: 2024-04-16T19:55:43.467 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.467 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.467 DEBUG:boto:Params: {} 2024-04-16T19:55:43.467 DEBUG:boto:Token: None 2024-04-16T19:55:43.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a164585c003b5154-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.470 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.470 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.470 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.470 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.470 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:43.471 DEBUG:boto:Data: 2024-04-16T19:55:43.471 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.471 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.471 DEBUG:boto:Params: {} 2024-04-16T19:55:43.471 DEBUG:boto:Token: None 2024-04-16T19:55:43.471 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h 2024-04-16T19:55:43.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0= 2024-04-16T19:55:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0=', 'Content-Length': '0'} 2024-04-16T19:55:43.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509e26a3bb7e2f0c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.473 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.473 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.473 DEBUG:boto:Method: HEAD 2024-04-16T19:55:43.473 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:43.473 DEBUG:boto:Data: 2024-04-16T19:55:43.474 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.474 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.474 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.474 DEBUG:boto:Params: {} 2024-04-16T19:55:43.474 DEBUG:boto:Token: None 2024-04-16T19:55:43.474 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h 2024-04-16T19:55:43.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0= 2024-04-16T19:55:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0=', 'Content-Length': '0'} 2024-04-16T19:55:43.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000572b3cbf5296eddd-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-16T19:55:43.476 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.477 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.477 DEBUG:boto:Method: GET 2024-04-16T19:55:43.477 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:43.477 DEBUG:boto:Data: 2024-04-16T19:55:43.477 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.477 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.477 DEBUG:boto:Params: {} 2024-04-16T19:55:43.477 DEBUG:boto:Token: None 2024-04-16T19:55:43.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h 2024-04-16T19:55:43.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0= 2024-04-16T19:55:43.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0=', 'Content-Length': '0'} 2024-04-16T19:55:43.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277968b089ab404e-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.522 DEBUG:boto:path=/yrjefh-50/h 2024-04-16T19:55:43.522 DEBUG:boto:auth_path=/yrjefh-50/h 2024-04-16T19:55:43.523 DEBUG:boto:Method: GET 2024-04-16T19:55:43.523 DEBUG:boto:Path: /yrjefh-50/h 2024-04-16T19:55:43.523 DEBUG:boto:Data: 2024-04-16T19:55:43.523 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.523 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.523 DEBUG:boto:Params: {} 2024-04-16T19:55:43.523 DEBUG:boto:Token: None 2024-04-16T19:55:43.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/h 2024-04-16T19:55:43.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0= 2024-04-16T19:55:43.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0=', 'Content-Length': '0'} 2024-04-16T19:55:43.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6dc1c6f9e30a332-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.526 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.526 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.526 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.526 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.526 DEBUG:boto:Method: GET 2024-04-16T19:55:43.526 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:43.527 DEBUG:boto:Data: 2024-04-16T19:55:43.527 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:43.527 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.527 DEBUG:boto:Params: {} 2024-04-16T19:55:43.527 DEBUG:boto:Token: None 2024-04-16T19:55:43.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f267aaa31ff802-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.530 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.530 DEBUG:boto:path=/yrjefh-50/ 2024-04-16T19:55:43.530 DEBUG:boto:auth_path=/yrjefh-50/ 2024-04-16T19:55:43.531 DEBUG:boto:path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.531 DEBUG:boto:auth_path=/yrjefh-50/?prefix=h 2024-04-16T19:55:43.531 DEBUG:boto:Method: GET 2024-04-16T19:55:43.531 DEBUG:boto:Path: /yrjefh-50/?prefix=h 2024-04-16T19:55:43.531 DEBUG:boto:Data: 2024-04-16T19:55:43.531 DEBUG:boto:Headers: {} 2024-04-16T19:55:43.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:43.531 DEBUG:boto:Port: 8000 2024-04-16T19:55:43.531 DEBUG:boto:Params: {} 2024-04-16T19:55:43.531 DEBUG:boto:Token: None 2024-04-16T19:55:43.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:43 GMT /yrjefh-50/ 2024-04-16T19:55:43.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T19:55:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T19:55:43.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771fcb84b457a5e1-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:43.534 DEBUG:boto:b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:43.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T19:55:43.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:43.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:43.535 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:43.586 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:43.586 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:43.770 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:43.769+0000 7f992a6c6640 -1 req 16214947220498892368 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:43.782 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:43.782+0000 7f99336d8640 -1 req 12603266879409288315 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:43.827 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:43.826+0000 7f983d4ec640 -1 req 11052363494949724008 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:43.827 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:43.826+0000 7f9939ee5640 -1 req 13017457097852656348 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:43.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:43.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:43.946 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:43.990 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:44.060 INFO:teuthology.orchestra.run.smithi026.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T19:55:44.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:44.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:44.061 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:44.104 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:44.104 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:44.250 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:44.249+0000 7f988557c640 -1 req 8480006673959211356 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:44.258 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:44.257+0000 7f98acdcb640 -1 req 9740684810442569463 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:44.305 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:44.305+0000 7f98c2df7640 -1 req 1317718920376686029 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:44.348 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:44.348+0000 7f98cd60c640 -1 req 7121165894475656509 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:44.350 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:44.350+0000 7f98e2636640 -1 req 17659004084310902894 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T19:55:44.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:44.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:55:44.422 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:55:44.506 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:44.506 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:44.576 INFO:teuthology.orchestra.run.smithi026.stdout:{"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T19:55:49.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-16T19:55:49.583 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:49.583 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:49.583 DEBUG:boto:Method: PUT 2024-04-16T19:55:49.583 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:49.583 DEBUG:boto:Data: 2024-04-16T19:55:49.584 DEBUG:boto:Headers: {} 2024-04-16T19:55:49.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:49.584 DEBUG:boto:Port: 8000 2024-04-16T19:55:49.584 DEBUG:boto:Params: {} 2024-04-16T19:55:49.584 DEBUG:boto:Token: None 2024-04-16T19:55:49.584 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:55:49 GMT /yrjefh-51/ 2024-04-16T19:55:49.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GF66LA56xFExcr80wuafLPsxCZY= 2024-04-16T19:55:49.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GF66LA56xFExcr80wuafLPsxCZY=', 'Content-Length': '0'} 2024-04-16T19:55:49.601 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000014cc52729143541-00661ed7c5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:49.602 DEBUG:rgw_multi.tests:created bucket=yrjefh-51 2024-04-16T19:55:49.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.602 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:55:49.654 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:49.654 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:49.709 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:55:49.733 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713297349.599966_705.1","last_update":"2024-04-16T19:55:49.599966Z"}] 2024-04-16T19:55:49.733 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:55:49.733 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:55:49.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.734 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:55:49.789 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:49.789 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:49.845+0000 7fce9d024800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:55:49.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:49.847+0000 7fce9d024800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:55:49Z"} 2024-04-16T19:55:49.869 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:55:49Z"} 2024-04-16T19:55:49.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:55:49.870 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:55:49.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:55:49.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:55:49.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:55:49.870 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:55:49.921 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:49.922 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:49.992 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000001","obj00000000000000000005","obj00000000000000000012","obj00000000000000000004","obj00000000000000000011","obj00000000000000000000","obj00000000000000000010","obj00000000000000000003","obj00000000000000000013","obj00000000000000000002","obj00000000000000000006"]} 2024-04-16T19:55:49.993 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:49.993 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:49.993 DEBUG:boto:Method: HEAD 2024-04-16T19:55:49.993 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:49.993 DEBUG:boto:Data: 2024-04-16T19:55:49.993 DEBUG:boto:Headers: {} 2024-04-16T19:55:49.994 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:49.994 DEBUG:boto:Port: 8000 2024-04-16T19:55:49.994 DEBUG:boto:Params: {} 2024-04-16T19:55:49.994 DEBUG:boto:Token: None 2024-04-16T19:55:49.994 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:49 GMT /yrjefh-51/ 2024-04-16T19:55:49.994 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ml8mLbhOdt9/y7eF/34w7YzV51E= 2024-04-16T19:55:49.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ml8mLbhOdt9/y7eF/34w7YzV51E=', 'Content-Length': '0'} 2024-04-16T19:55:49.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acd41fff01b37b2d-00661ed7c5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:49.997 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:49.998 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:49.998 DEBUG:boto:Method: PUT 2024-04-16T19:55:49.998 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:49.998 DEBUG:boto:Data: 2024-04-16T19:55:49.998 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:49.998 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:49.998 DEBUG:boto:Port: 8000 2024-04-16T19:55:49.998 DEBUG:boto:Params: {} 2024-04-16T19:55:49.999 DEBUG:boto:Token: None 2024-04-16T19:55:49.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:49 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:49.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tZS9Tq+qlIJuecsv82E90H6ZiOo= 2024-04-16T19:55:49.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:49 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tZS9Tq+qlIJuecsv82E90H6ZiOo='} 2024-04-16T19:55:50.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075de7defecc3f885-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.006 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.006 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.006 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.006 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.006 DEBUG:boto:Data: 2024-04-16T19:55:50.006 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.006 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.006 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.006 DEBUG:boto:Params: {} 2024-04-16T19:55:50.007 DEBUG:boto:Token: None 2024-04-16T19:55:50.007 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007884ca9a4a116406-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.009 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:50.009 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:50.010 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.010 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:50.010 DEBUG:boto:Data: 2024-04-16T19:55:50.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.010 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.010 DEBUG:boto:Params: {} 2024-04-16T19:55:50.010 DEBUG:boto:Token: None 2024-04-16T19:55:50.011 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:50.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hfvkiZG7DTJYDiOqjYhi5TpZdi0= 2024-04-16T19:55:50.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hfvkiZG7DTJYDiOqjYhi5TpZdi0='} 2024-04-16T19:55:50.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4b05d38dab2988a-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.016 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.016 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.016 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.016 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.016 DEBUG:boto:Data: 2024-04-16T19:55:50.016 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.016 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.017 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.017 DEBUG:boto:Params: {} 2024-04-16T19:55:50.017 DEBUG:boto:Token: None 2024-04-16T19:55:50.017 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a40322a503a4a4d-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.019 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:50.019 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:50.020 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.020 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:50.020 DEBUG:boto:Data: 2024-04-16T19:55:50.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.020 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.020 DEBUG:boto:Params: {} 2024-04-16T19:55:50.020 DEBUG:boto:Token: None 2024-04-16T19:55:50.021 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:50.021 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tIdhNhp+21y7T7/QoR5HCLecUng= 2024-04-16T19:55:50.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tIdhNhp+21y7T7/QoR5HCLecUng='} 2024-04-16T19:55:50.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f6bd734e44e0eda-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.063 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.063 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.063 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.063 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.063 DEBUG:boto:Data: 2024-04-16T19:55:50.063 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.063 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.063 DEBUG:boto:Params: {} 2024-04-16T19:55:50.064 DEBUG:boto:Token: None 2024-04-16T19:55:50.064 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000436cc2e9a52f1fc0-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.066 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:50.066 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:50.066 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.066 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:50.067 DEBUG:boto:Data: 2024-04-16T19:55:50.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.067 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.067 DEBUG:boto:Params: {} 2024-04-16T19:55:50.067 DEBUG:boto:Token: None 2024-04-16T19:55:50.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:50.067 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cRrCbn/IDhdLQhtj3u2k1jvzDWE= 2024-04-16T19:55:50.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cRrCbn/IDhdLQhtj3u2k1jvzDWE='} 2024-04-16T19:55:50.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e22d24fc7b184c0-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.073 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.073 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.073 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.073 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.073 DEBUG:boto:Data: 2024-04-16T19:55:50.073 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.073 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.074 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.074 DEBUG:boto:Params: {} 2024-04-16T19:55:50.074 DEBUG:boto:Token: None 2024-04-16T19:55:50.074 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005556ba230c53153f-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.076 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:50.076 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:50.076 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.077 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:50.077 DEBUG:boto:Data: 2024-04-16T19:55:50.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.077 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.077 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.077 DEBUG:boto:Params: {} 2024-04-16T19:55:50.077 DEBUG:boto:Token: None 2024-04-16T19:55:50.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:50.077 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6ZiyMe9E4nFndR6J0AjQ8v8DNuY= 2024-04-16T19:55:50.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6ZiyMe9E4nFndR6J0AjQ8v8DNuY='} 2024-04-16T19:55:50.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7d42d5db68f00ec-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.083 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.083 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.083 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.084 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.084 DEBUG:boto:Data: 2024-04-16T19:55:50.084 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.084 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.084 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.084 DEBUG:boto:Params: {} 2024-04-16T19:55:50.084 DEBUG:boto:Token: None 2024-04-16T19:55:50.084 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f172b9ad81cb08-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.087 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:50.087 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:50.087 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.087 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:50.087 DEBUG:boto:Data: 2024-04-16T19:55:50.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.087 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.087 DEBUG:boto:Params: {} 2024-04-16T19:55:50.088 DEBUG:boto:Token: None 2024-04-16T19:55:50.088 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:50.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcuV1kwtNScZatNy/Dn3FIUsUwo= 2024-04-16T19:55:50.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcuV1kwtNScZatNy/Dn3FIUsUwo='} 2024-04-16T19:55:50.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a4d824ee416c686-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.131 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.131 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.131 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.131 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.131 DEBUG:boto:Data: 2024-04-16T19:55:50.131 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.131 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.131 DEBUG:boto:Params: {} 2024-04-16T19:55:50.132 DEBUG:boto:Token: None 2024-04-16T19:55:50.132 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4cd64e3fe033043-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:50.135 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:50.135 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.135 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:50.135 DEBUG:boto:Data: 2024-04-16T19:55:50.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.135 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.135 DEBUG:boto:Params: {} 2024-04-16T19:55:50.135 DEBUG:boto:Token: None 2024-04-16T19:55:50.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:50.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b5tgHityjzR/fAZswizInSSzmaM= 2024-04-16T19:55:50.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b5tgHityjzR/fAZswizInSSzmaM='} 2024-04-16T19:55:50.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3c10dbfdfda5122-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.141 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.141 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.141 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.141 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.142 DEBUG:boto:Data: 2024-04-16T19:55:50.142 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.142 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.142 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.142 DEBUG:boto:Params: {} 2024-04-16T19:55:50.142 DEBUG:boto:Token: None 2024-04-16T19:55:50.142 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d6a4c4a85c4deea-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.144 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:50.144 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:50.145 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.145 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:50.145 DEBUG:boto:Data: 2024-04-16T19:55:50.145 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.145 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.145 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.145 DEBUG:boto:Params: {} 2024-04-16T19:55:50.145 DEBUG:boto:Token: None 2024-04-16T19:55:50.145 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:50.145 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dtv892/ttUvHtg+vAbmNBHTB06Q= 2024-04-16T19:55:50.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dtv892/ttUvHtg+vAbmNBHTB06Q='} 2024-04-16T19:55:50.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000613eb36331ed2f9b-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.151 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.152 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.152 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.152 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.152 DEBUG:boto:Data: 2024-04-16T19:55:50.152 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.152 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.152 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.152 DEBUG:boto:Params: {} 2024-04-16T19:55:50.152 DEBUG:boto:Token: None 2024-04-16T19:55:50.153 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003111f5afdfeaca81-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.155 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:50.155 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:50.155 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.155 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:50.155 DEBUG:boto:Data: 2024-04-16T19:55:50.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.156 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.156 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.156 DEBUG:boto:Params: {} 2024-04-16T19:55:50.156 DEBUG:boto:Token: None 2024-04-16T19:55:50.156 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:50.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xce+XHXgNvN4CUdJkgkfJoC9myc= 2024-04-16T19:55:50.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xce+XHXgNvN4CUdJkgkfJoC9myc='} 2024-04-16T19:55:50.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ce595a055efc7c7-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.199 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:50.199 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:50.199 DEBUG:boto:Method: HEAD 2024-04-16T19:55:50.199 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:50.199 DEBUG:boto:Data: 2024-04-16T19:55:50.199 DEBUG:boto:Headers: {} 2024-04-16T19:55:50.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.199 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.199 DEBUG:boto:Params: {} 2024-04-16T19:55:50.200 DEBUG:boto:Token: None 2024-04-16T19:55:50.200 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/ 2024-04-16T19:55:50.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE= 2024-04-16T19:55:50.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y/KP+uaebc7kw+s5Zcv+/avJHaE=', 'Content-Length': '0'} 2024-04-16T19:55:50.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a9c2495980db3c-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.202 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:50.202 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:50.202 DEBUG:boto:Method: PUT 2024-04-16T19:55:50.203 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:50.203 DEBUG:boto:Data: 2024-04-16T19:55:50.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:55:50.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:50.203 DEBUG:boto:Port: 8000 2024-04-16T19:55:50.203 DEBUG:boto:Params: {} 2024-04-16T19:55:50.203 DEBUG:boto:Token: None 2024-04-16T19:55:50.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:55:50 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:50.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l0q+HtWlBTbV4CI/zCtq4UXp2yo= 2024-04-16T19:55:50.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:55:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l0q+HtWlBTbV4CI/zCtq4UXp2yo='} 2024-04-16T19:55:50.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a517d89fecc134c8-00661ed7c6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:50.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-51 2024-04-16T19:55:50.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:50.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:50.209 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-51 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:55:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:50.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:50.322+0000 7facc7f4b800 1 waiting to reach incremental sync.. 2024-04-16T19:55:55.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:55.324+0000 7facc7f4b800 1 bucket sync caught up with source: 2024-04-16T19:55:55.325 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.31.6, 00000000001.46.6, 00000000001.50.6, 00000000001.76.6, 00000000001.110.6, , 00000000001.83.6, 00000000001.65.6, 00000000001.67.6, 00000000001.63.6, 00000000001.50.6] 2024-04-16T19:55:55.325 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.31.6, 00000000001.46.6, 00000000001.50.6, 00000000001.76.6, 00000000001.110.6, , 00000000001.83.6, 00000000001.65.6, 00000000001.67.6, 00000000001.63.6, 00000000001.50.6] 2024-04-16T19:55:55.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:55:55.324+0000 7facc7f4b800 0 bucket checkpoint complete 2024-04-16T19:55:55.350 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-51 2024-04-16T19:55:55.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:55.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:55:55.351 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-51 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:55:55.388 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T19:55:55.388+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T19:55:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:55:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:55:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:55.483+0000 7f081bf33800 1 bucket sync caught up with source: 2024-04-16T19:55:55.483 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.66.6, 00000000001.132.6, 00000000001.144.6, 00000000001.111.6, 00000000001.226.6, , 00000000001.147.6, 00000000001.162.6, 00000000001.163.6, 00000000001.161.6, 00000000001.153.6] 2024-04-16T19:55:55.483 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.66.6, 00000000001.132.6, 00000000001.144.6, 00000000001.111.6, 00000000001.226.6, , 00000000001.147.6, 00000000001.162.6, 00000000001.163.6, 00000000001.161.6, 00000000001.153.6] 2024-04-16T19:55:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:55:55.483+0000 7f081bf33800 0 bucket checkpoint complete 2024-04-16T19:55:55.506 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-51 zones={test-zone2, test-zone1} 2024-04-16T19:55:55.506 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.506 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.506 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.506 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:55.507 DEBUG:boto:Data: 2024-04-16T19:55:55.507 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.507 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.507 DEBUG:boto:Params: {} 2024-04-16T19:55:55.507 DEBUG:boto:Token: None 2024-04-16T19:55:55.507 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gy0ETP+3TJKcwB0b3ePaRbjPspY= 2024-04-16T19:55:55.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gy0ETP+3TJKcwB0b3ePaRbjPspY=', 'Content-Length': '0'} 2024-04-16T19:55:55.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000306e1612551e8dd2-00661ed7cb-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.511 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.511 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.511 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.511 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:55:55.511 DEBUG:boto:Data: 2024-04-16T19:55:55.511 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.512 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.512 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.512 DEBUG:boto:Params: {} 2024-04-16T19:55:55.512 DEBUG:boto:Token: None 2024-04-16T19:55:55.512 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gy0ETP+3TJKcwB0b3ePaRbjPspY= 2024-04-16T19:55:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gy0ETP+3TJKcwB0b3ePaRbjPspY=', 'Content-Length': '0'} 2024-04-16T19:55:55.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2c491182314f3ad-00661ed7cb-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.515 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:55:55.515 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.515 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.515 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.516 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.516 DEBUG:boto:Method: GET 2024-04-16T19:55:55.516 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.516 DEBUG:boto:Data: 2024-04-16T19:55:55.516 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.516 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.516 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.516 DEBUG:boto:Params: {} 2024-04-16T19:55:55.516 DEBUG:boto:Token: None 2024-04-16T19:55:55.517 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/?versions 2024-04-16T19:55:55.517 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI= 2024-04-16T19:55:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI=', 'Content-Length': '0'} 2024-04-16T19:55:55.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb8831b8c6f0452-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.520 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000001nulltrue2024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-16T19:55:55.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-16T19:55:55.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-16T19:55:55.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-16T19:55:55.522 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:55:55.522 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.523 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.523 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.523 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.523 DEBUG:boto:Method: GET 2024-04-16T19:55:55.523 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.523 DEBUG:boto:Data: 2024-04-16T19:55:55.523 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.523 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.523 DEBUG:boto:Params: {} 2024-04-16T19:55:55.524 DEBUG:boto:Token: None 2024-04-16T19:55:55.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/?versions 2024-04-16T19:55:55.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI= 2024-04-16T19:55:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI=', 'Content-Length': '0'} 2024-04-16T19:55:55.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6167f3da5a10ed5-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.527 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000001nulltrue2024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-16T19:55:55.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-16T19:55:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-16T19:55:55.528 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.529 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.529 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.529 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.529 DEBUG:boto:Method: GET 2024-04-16T19:55:55.529 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.529 DEBUG:boto:Data: 2024-04-16T19:55:55.529 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.529 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.529 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.529 DEBUG:boto:Params: {} 2024-04-16T19:55:55.530 DEBUG:boto:Token: None 2024-04-16T19:55:55.530 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/?versions 2024-04-16T19:55:55.530 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI= 2024-04-16T19:55:55.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI=', 'Content-Length': '0'} 2024-04-16T19:55:55.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81c11f1b54f220e-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.533 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000001nulltrue2024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.533 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.534 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.534 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.534 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.534 DEBUG:boto:Method: GET 2024-04-16T19:55:55.534 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:55:55.534 DEBUG:boto:Data: 2024-04-16T19:55:55.534 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.534 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.535 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.535 DEBUG:boto:Params: {} 2024-04-16T19:55:55.535 DEBUG:boto:Token: None 2024-04-16T19:55:55.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/?versions 2024-04-16T19:55:55.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI= 2024-04-16T19:55:55.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fcg/ToXtpi+4bNmM6YSO+byiQxI=', 'Content-Length': '0'} 2024-04-16T19:55:55.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dd732fbff4d7a2a-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.538 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000001nulltrue2024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:55:55.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.539 DEBUG:boto:Method: GET 2024-04-16T19:55:55.539 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.539 DEBUG:boto:Data: 2024-04-16T19:55:55.540 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.540 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.540 DEBUG:boto:Params: {} 2024-04-16T19:55:55.540 DEBUG:boto:Token: None 2024-04-16T19:55:55.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ= 2024-04-16T19:55:55.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ=', 'Content-Length': '0'} 2024-04-16T19:55:55.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2fabd6f8b6d9581-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.586 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.587 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.587 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.587 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.587 DEBUG:boto:Method: GET 2024-04-16T19:55:55.587 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.587 DEBUG:boto:Data: 2024-04-16T19:55:55.587 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.587 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.587 DEBUG:boto:Params: {} 2024-04-16T19:55:55.588 DEBUG:boto:Token: None 2024-04-16T19:55:55.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ= 2024-04-16T19:55:55.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ=', 'Content-Length': '0'} 2024-04-16T19:55:55.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008925f82812f88bc8-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.590 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.590 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.590 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.591 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.591 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.591 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.591 DEBUG:boto:Data: 2024-04-16T19:55:55.591 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.591 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.591 DEBUG:boto:Params: {} 2024-04-16T19:55:55.591 DEBUG:boto:Token: None 2024-04-16T19:55:55.592 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5VtsZSxyDUdX8j2AUBOT9j/6d9g= 2024-04-16T19:55:55.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VtsZSxyDUdX8j2AUBOT9j/6d9g=', 'Content-Length': '0'} 2024-04-16T19:55:55.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005704a6380868140a-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.594 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.595 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.595 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.595 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.595 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.595 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.595 DEBUG:boto:Data: 2024-04-16T19:55:55.595 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.596 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.596 DEBUG:boto:Params: {} 2024-04-16T19:55:55.596 DEBUG:boto:Token: None 2024-04-16T19:55:55.596 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5VtsZSxyDUdX8j2AUBOT9j/6d9g= 2024-04-16T19:55:55.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VtsZSxyDUdX8j2AUBOT9j/6d9g=', 'Content-Length': '0'} 2024-04-16T19:55:55.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a06ffaf0adca4cc6-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:55:55.598 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.598 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.599 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.599 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.599 DEBUG:boto:Method: GET 2024-04-16T19:55:55.599 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.599 DEBUG:boto:Data: 2024-04-16T19:55:55.599 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.599 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.599 DEBUG:boto:Params: {} 2024-04-16T19:55:55.599 DEBUG:boto:Token: None 2024-04-16T19:55:55.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ= 2024-04-16T19:55:55.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ=', 'Content-Length': '0'} 2024-04-16T19:55:55.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006936071a0f420fad-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.603 DEBUG:boto:Method: GET 2024-04-16T19:55:55.603 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.603 DEBUG:boto:Data: 2024-04-16T19:55:55.603 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.603 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.603 DEBUG:boto:Params: {} 2024-04-16T19:55:55.603 DEBUG:boto:Token: None 2024-04-16T19:55:55.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:55:55.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ= 2024-04-16T19:55:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M9KKBPOX2soVe1jfdyVapkUXBPQ=', 'Content-Length': '0'} 2024-04-16T19:55:55.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38ac7a48a4a3c1d-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.606 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.606 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.606 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.606 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.606 DEBUG:boto:Method: GET 2024-04-16T19:55:55.606 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.607 DEBUG:boto:Data: 2024-04-16T19:55:55.607 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.607 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.607 DEBUG:boto:Params: {} 2024-04-16T19:55:55.607 DEBUG:boto:Token: None 2024-04-16T19:55:55.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099613fc699c16bbf-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.610 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.611 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.611 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.611 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.611 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.611 DEBUG:boto:Method: GET 2024-04-16T19:55:55.611 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.612 DEBUG:boto:Data: 2024-04-16T19:55:55.612 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.612 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.612 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.612 DEBUG:boto:Params: {} 2024-04-16T19:55:55.612 DEBUG:boto:Token: None 2024-04-16T19:55:55.612 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d901fdf6bfa4dbb-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.615 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.615 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.615 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.615 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.616 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.616 DEBUG:boto:Data: 2024-04-16T19:55:55.616 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.616 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.616 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.616 DEBUG:boto:Params: {} 2024-04-16T19:55:55.616 DEBUG:boto:Token: None 2024-04-16T19:55:55.616 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6qmZFZTt7wd5mWDqxj1+HfRW/Sk= 2024-04-16T19:55:55.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6qmZFZTt7wd5mWDqxj1+HfRW/Sk=', 'Content-Length': '0'} 2024-04-16T19:55:55.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400cd4d594a00e82-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.619 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.619 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.619 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.619 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.619 DEBUG:boto:Data: 2024-04-16T19:55:55.619 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.619 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.619 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.619 DEBUG:boto:Params: {} 2024-04-16T19:55:55.619 DEBUG:boto:Token: None 2024-04-16T19:55:55.620 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6qmZFZTt7wd5mWDqxj1+HfRW/Sk= 2024-04-16T19:55:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6qmZFZTt7wd5mWDqxj1+HfRW/Sk=', 'Content-Length': '0'} 2024-04-16T19:55:55.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3785be7d7d811b-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:55:55.622 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.622 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.622 DEBUG:boto:Method: GET 2024-04-16T19:55:55.622 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.622 DEBUG:boto:Data: 2024-04-16T19:55:55.622 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.623 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.623 DEBUG:boto:Params: {} 2024-04-16T19:55:55.623 DEBUG:boto:Token: None 2024-04-16T19:55:55.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsxMUL/B1J/v0EO/9DvJ0n8AisA= 2024-04-16T19:55:55.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsxMUL/B1J/v0EO/9DvJ0n8AisA=', 'Content-Length': '0'} 2024-04-16T19:55:55.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006daf8eef0ae556d6-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.666 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.666 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.667 DEBUG:boto:Method: GET 2024-04-16T19:55:55.667 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.667 DEBUG:boto:Data: 2024-04-16T19:55:55.667 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.667 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.667 DEBUG:boto:Params: {} 2024-04-16T19:55:55.667 DEBUG:boto:Token: None 2024-04-16T19:55:55.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:55:55.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tsxMUL/B1J/v0EO/9DvJ0n8AisA= 2024-04-16T19:55:55.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tsxMUL/B1J/v0EO/9DvJ0n8AisA=', 'Content-Length': '0'} 2024-04-16T19:55:55.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f498c5f2e18da71-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.670 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.670 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.670 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.670 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.670 DEBUG:boto:Method: GET 2024-04-16T19:55:55.670 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.670 DEBUG:boto:Data: 2024-04-16T19:55:55.670 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.671 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.671 DEBUG:boto:Params: {} 2024-04-16T19:55:55.671 DEBUG:boto:Token: None 2024-04-16T19:55:55.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fa57af033beab12-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.674 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.674 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.675 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.675 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.675 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.675 DEBUG:boto:Method: GET 2024-04-16T19:55:55.675 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:55:55.675 DEBUG:boto:Data: 2024-04-16T19:55:55.675 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.675 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.675 DEBUG:boto:Params: {} 2024-04-16T19:55:55.676 DEBUG:boto:Token: None 2024-04-16T19:55:55.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a8851c4cd4c4f2-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.678 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:55:50.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:55:55.679 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.679 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.679 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.679 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.679 DEBUG:boto:Method: GET 2024-04-16T19:55:55.679 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.679 DEBUG:boto:Data: 2024-04-16T19:55:55.679 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.679 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.680 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.680 DEBUG:boto:Params: {} 2024-04-16T19:55:55.680 DEBUG:boto:Token: None 2024-04-16T19:55:55.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA= 2024-04-16T19:55:55.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA=', 'Content-Length': '0'} 2024-04-16T19:55:55.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000265c5550565943c4-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.722 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.723 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.723 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.723 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.723 DEBUG:boto:Method: GET 2024-04-16T19:55:55.723 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.723 DEBUG:boto:Data: 2024-04-16T19:55:55.723 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.723 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.723 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.723 DEBUG:boto:Params: {} 2024-04-16T19:55:55.723 DEBUG:boto:Token: None 2024-04-16T19:55:55.724 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA= 2024-04-16T19:55:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA=', 'Content-Length': '0'} 2024-04-16T19:55:55.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b5a8c1efdfc0b3-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.726 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.726 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.726 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.726 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.726 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.726 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.726 DEBUG:boto:Data: 2024-04-16T19:55:55.727 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.727 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.727 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.727 DEBUG:boto:Params: {} 2024-04-16T19:55:55.727 DEBUG:boto:Token: None 2024-04-16T19:55:55.727 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zPHdphkVQyxEkT6AdFASKFNxdM= 2024-04-16T19:55:55.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zPHdphkVQyxEkT6AdFASKFNxdM=', 'Content-Length': '0'} 2024-04-16T19:55:55.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be346c154e790f01-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.729 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.729 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.729 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.729 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.730 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.730 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.730 DEBUG:boto:Data: 2024-04-16T19:55:55.730 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.730 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.730 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.730 DEBUG:boto:Params: {} 2024-04-16T19:55:55.730 DEBUG:boto:Token: None 2024-04-16T19:55:55.730 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zPHdphkVQyxEkT6AdFASKFNxdM= 2024-04-16T19:55:55.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zPHdphkVQyxEkT6AdFASKFNxdM=', 'Content-Length': '0'} 2024-04-16T19:55:55.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c51c6fb0879266-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:55:55.732 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.732 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.732 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.733 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.733 DEBUG:boto:Method: GET 2024-04-16T19:55:55.733 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.733 DEBUG:boto:Data: 2024-04-16T19:55:55.733 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.733 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.733 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.733 DEBUG:boto:Params: {} 2024-04-16T19:55:55.733 DEBUG:boto:Token: None 2024-04-16T19:55:55.734 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.734 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA= 2024-04-16T19:55:55.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA=', 'Content-Length': '0'} 2024-04-16T19:55:55.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a0a9745729ac23-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.778 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.778 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.779 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.779 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.779 DEBUG:boto:Method: GET 2024-04-16T19:55:55.779 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.779 DEBUG:boto:Data: 2024-04-16T19:55:55.779 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.779 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.779 DEBUG:boto:Params: {} 2024-04-16T19:55:55.779 DEBUG:boto:Token: None 2024-04-16T19:55:55.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:55:55.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA= 2024-04-16T19:55:55.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qp7A2zxHO8jlQOwkXxCfouFq4gA=', 'Content-Length': '0'} 2024-04-16T19:55:55.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000673834bc993d661b-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.782 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.782 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.782 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.782 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.782 DEBUG:boto:Method: GET 2024-04-16T19:55:55.782 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.782 DEBUG:boto:Data: 2024-04-16T19:55:55.782 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.782 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.782 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.782 DEBUG:boto:Params: {} 2024-04-16T19:55:55.783 DEBUG:boto:Token: None 2024-04-16T19:55:55.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046b92933b19c9286-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.826 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.827 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.827 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.827 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.827 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.827 DEBUG:boto:Method: GET 2024-04-16T19:55:55.827 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.827 DEBUG:boto:Data: 2024-04-16T19:55:55.827 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.828 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.828 DEBUG:boto:Params: {} 2024-04-16T19:55:55.828 DEBUG:boto:Token: None 2024-04-16T19:55:55.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf7c7a052c6f39c2-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.831 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.831 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.831 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.831 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.831 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.831 DEBUG:boto:Data: 2024-04-16T19:55:55.832 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.832 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.832 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.832 DEBUG:boto:Params: {} 2024-04-16T19:55:55.832 DEBUG:boto:Token: None 2024-04-16T19:55:55.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRVPehC1MqUbr+mMBnpt5/Uamcg= 2024-04-16T19:55:55.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRVPehC1MqUbr+mMBnpt5/Uamcg=', 'Content-Length': '0'} 2024-04-16T19:55:55.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008172eb2d39925254-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.835 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.835 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.835 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.835 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.835 DEBUG:boto:Data: 2024-04-16T19:55:55.835 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.835 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.835 DEBUG:boto:Params: {} 2024-04-16T19:55:55.835 DEBUG:boto:Token: None 2024-04-16T19:55:55.836 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRVPehC1MqUbr+mMBnpt5/Uamcg= 2024-04-16T19:55:55.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRVPehC1MqUbr+mMBnpt5/Uamcg=', 'Content-Length': '0'} 2024-04-16T19:55:55.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002662b6321bc4eb1f-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:55:55.838 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.838 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.838 DEBUG:boto:Method: GET 2024-04-16T19:55:55.838 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.838 DEBUG:boto:Data: 2024-04-16T19:55:55.838 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.839 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.839 DEBUG:boto:Params: {} 2024-04-16T19:55:55.839 DEBUG:boto:Token: None 2024-04-16T19:55:55.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ycAnV5EbDxYBaw+hmQ88+BMkU9s= 2024-04-16T19:55:55.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ycAnV5EbDxYBaw+hmQ88+BMkU9s=', 'Content-Length': '0'} 2024-04-16T19:55:55.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b1f4c443b14f04-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.882 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.883 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.883 DEBUG:boto:Method: GET 2024-04-16T19:55:55.883 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.883 DEBUG:boto:Data: 2024-04-16T19:55:55.883 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.883 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.883 DEBUG:boto:Params: {} 2024-04-16T19:55:55.883 DEBUG:boto:Token: None 2024-04-16T19:55:55.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:55:55.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ycAnV5EbDxYBaw+hmQ88+BMkU9s= 2024-04-16T19:55:55.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ycAnV5EbDxYBaw+hmQ88+BMkU9s=', 'Content-Length': '0'} 2024-04-16T19:55:55.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32aa8acbea8600a-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.886 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.886 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.886 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.886 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.887 DEBUG:boto:Method: GET 2024-04-16T19:55:55.887 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.887 DEBUG:boto:Data: 2024-04-16T19:55:55.887 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.887 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.887 DEBUG:boto:Params: {} 2024-04-16T19:55:55.887 DEBUG:boto:Token: None 2024-04-16T19:55:55.887 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388b55efa54164ac-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.890 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.891 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:55.891 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:55.891 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.891 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.891 DEBUG:boto:Method: GET 2024-04-16T19:55:55.891 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:55:55.891 DEBUG:boto:Data: 2024-04-16T19:55:55.891 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.892 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.892 DEBUG:boto:Params: {} 2024-04-16T19:55:55.892 DEBUG:boto:Token: None 2024-04-16T19:55:55.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/ 2024-04-16T19:55:55.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g= 2024-04-16T19:55:55.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:otVViJfvQEQYcZz4xt4j8ku2j0g=', 'Content-Length': '0'} 2024-04-16T19:55:55.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb4a4383a41db02-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.895 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:55:50.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:55.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:55:55.895 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.895 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.895 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.895 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.895 DEBUG:boto:Method: GET 2024-04-16T19:55:55.895 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.896 DEBUG:boto:Data: 2024-04-16T19:55:55.896 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.896 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.896 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.896 DEBUG:boto:Params: {} 2024-04-16T19:55:55.896 DEBUG:boto:Token: None 2024-04-16T19:55:55.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc= 2024-04-16T19:55:55.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc=', 'Content-Length': '0'} 2024-04-16T19:55:55.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085bb13ff6351996c-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.942 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.943 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.943 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.943 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.943 DEBUG:boto:Method: GET 2024-04-16T19:55:55.943 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.943 DEBUG:boto:Data: 2024-04-16T19:55:55.943 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.943 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.943 DEBUG:boto:Params: {} 2024-04-16T19:55:55.943 DEBUG:boto:Token: None 2024-04-16T19:55:55.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc= 2024-04-16T19:55:55.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc=', 'Content-Length': '0'} 2024-04-16T19:55:55.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be23d4014be76de-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.946 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.946 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.946 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.946 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.946 DEBUG:boto:Data: 2024-04-16T19:55:55.946 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.946 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.946 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.946 DEBUG:boto:Params: {} 2024-04-16T19:55:55.947 DEBUG:boto:Token: None 2024-04-16T19:55:55.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MqCqZ3X9YPBt4P/vYEj7vSs+83g= 2024-04-16T19:55:55.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MqCqZ3X9YPBt4P/vYEj7vSs+83g=', 'Content-Length': '0'} 2024-04-16T19:55:55.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700106f066dace26-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.949 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.949 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.949 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.949 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.949 DEBUG:boto:Method: HEAD 2024-04-16T19:55:55.949 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.949 DEBUG:boto:Data: 2024-04-16T19:55:55.950 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.950 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.950 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.950 DEBUG:boto:Params: {} 2024-04-16T19:55:55.950 DEBUG:boto:Token: None 2024-04-16T19:55:55.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MqCqZ3X9YPBt4P/vYEj7vSs+83g= 2024-04-16T19:55:55.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MqCqZ3X9YPBt4P/vYEj7vSs+83g=', 'Content-Length': '0'} 2024-04-16T19:55:55.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042182cc9f9e99ac8-00661ed7cb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:55:55.952 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.952 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.953 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.953 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.953 DEBUG:boto:Method: GET 2024-04-16T19:55:55.953 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.953 DEBUG:boto:Data: 2024-04-16T19:55:55.953 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.953 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:55.953 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.953 DEBUG:boto:Params: {} 2024-04-16T19:55:55.953 DEBUG:boto:Token: None 2024-04-16T19:55:55.954 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:55 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc= 2024-04-16T19:55:55.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WEofWfxl0PIHMWII8eo5vh4rTTc=', 'Content-Length': '0'} 2024-04-16T19:55:55.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f674d3fb064d8d5-00661ed7cb-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:55.998 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.998 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:55.999 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.999 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.999 DEBUG:boto:Method: GET 2024-04-16T19:55:55.999 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:55.999 DEBUG:boto:Data: 2024-04-16T19:55:55.999 DEBUG:boto:Headers: {} 2024-04-16T19:55:55.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:55.999 DEBUG:boto:Port: 8000 2024-04-16T19:55:55.999 DEBUG:boto:Params: {} 2024-04-16T19:55:55.999 DEBUG:boto:Token: None 2024-04-16T19:55:56.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:55:56.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/aoPO/g9K7eRozYvcTdJrNGGpjs= 2024-04-16T19:55:56.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/aoPO/g9K7eRozYvcTdJrNGGpjs=', 'Content-Length': '0'} 2024-04-16T19:55:56.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeaca54fd682797c-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.002 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.002 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.002 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.002 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.002 DEBUG:boto:Method: GET 2024-04-16T19:55:56.002 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.003 DEBUG:boto:Data: 2024-04-16T19:55:56.003 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.003 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.003 DEBUG:boto:Params: {} 2024-04-16T19:55:56.003 DEBUG:boto:Token: None 2024-04-16T19:55:56.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a8b06d2d0af5576-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.006 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.006 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.007 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.007 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.007 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.007 DEBUG:boto:Method: GET 2024-04-16T19:55:56.007 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.007 DEBUG:boto:Data: 2024-04-16T19:55:56.007 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.007 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.007 DEBUG:boto:Params: {} 2024-04-16T19:55:56.007 DEBUG:boto:Token: None 2024-04-16T19:55:56.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f6e908be9f5e24-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.010 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.011 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.011 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.011 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.011 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.011 DEBUG:boto:Data: 2024-04-16T19:55:56.011 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.011 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.011 DEBUG:boto:Params: {} 2024-04-16T19:55:56.011 DEBUG:boto:Token: None 2024-04-16T19:55:56.012 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nLK5qEgODYmi6gQ5uQYygVEU1dg= 2024-04-16T19:55:56.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nLK5qEgODYmi6gQ5uQYygVEU1dg=', 'Content-Length': '0'} 2024-04-16T19:55:56.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51b5d661ab79c90-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.014 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.014 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.014 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.014 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.014 DEBUG:boto:Data: 2024-04-16T19:55:56.014 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.015 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.015 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.015 DEBUG:boto:Params: {} 2024-04-16T19:55:56.015 DEBUG:boto:Token: None 2024-04-16T19:55:56.015 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nLK5qEgODYmi6gQ5uQYygVEU1dg= 2024-04-16T19:55:56.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nLK5qEgODYmi6gQ5uQYygVEU1dg=', 'Content-Length': '0'} 2024-04-16T19:55:56.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f371afcb8beac65-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:55:56.017 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.017 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.017 DEBUG:boto:Method: GET 2024-04-16T19:55:56.018 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.018 DEBUG:boto:Data: 2024-04-16T19:55:56.018 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.018 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.018 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.018 DEBUG:boto:Params: {} 2024-04-16T19:55:56.018 DEBUG:boto:Token: None 2024-04-16T19:55:56.018 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dSttSn0veWFuW9kyAz6AflR8coQ= 2024-04-16T19:55:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dSttSn0veWFuW9kyAz6AflR8coQ=', 'Content-Length': '0'} 2024-04-16T19:55:56.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a7268f310c908cb-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.062 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.062 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.063 DEBUG:boto:Method: GET 2024-04-16T19:55:56.063 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.063 DEBUG:boto:Data: 2024-04-16T19:55:56.063 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.063 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.063 DEBUG:boto:Params: {} 2024-04-16T19:55:56.063 DEBUG:boto:Token: None 2024-04-16T19:55:56.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:55:56.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dSttSn0veWFuW9kyAz6AflR8coQ= 2024-04-16T19:55:56.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dSttSn0veWFuW9kyAz6AflR8coQ=', 'Content-Length': '0'} 2024-04-16T19:55:56.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9f6ed3dbcce44a-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.065 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.066 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.066 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.066 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.066 DEBUG:boto:Method: GET 2024-04-16T19:55:56.066 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.066 DEBUG:boto:Data: 2024-04-16T19:55:56.066 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.066 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.066 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.066 DEBUG:boto:Params: {} 2024-04-16T19:55:56.067 DEBUG:boto:Token: None 2024-04-16T19:55:56.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.067 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000732b0f64d354e092-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.110 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.111 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.111 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.111 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.111 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.111 DEBUG:boto:Method: GET 2024-04-16T19:55:56.111 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:55:56.111 DEBUG:boto:Data: 2024-04-16T19:55:56.111 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.111 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.112 DEBUG:boto:Params: {} 2024-04-16T19:55:56.112 DEBUG:boto:Token: None 2024-04-16T19:55:56.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0eec02f51725dd-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.114 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:55:50.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:55:56.115 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.115 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.115 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.115 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.115 DEBUG:boto:Method: GET 2024-04-16T19:55:56.115 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.115 DEBUG:boto:Data: 2024-04-16T19:55:56.115 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.115 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.115 DEBUG:boto:Params: {} 2024-04-16T19:55:56.116 DEBUG:boto:Token: None 2024-04-16T19:55:56.116 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M= 2024-04-16T19:55:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M=', 'Content-Length': '0'} 2024-04-16T19:55:56.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7a79973ba17645-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.158 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.158 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.158 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.159 DEBUG:boto:Method: GET 2024-04-16T19:55:56.159 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.159 DEBUG:boto:Data: 2024-04-16T19:55:56.159 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.159 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.159 DEBUG:boto:Params: {} 2024-04-16T19:55:56.159 DEBUG:boto:Token: None 2024-04-16T19:55:56.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M= 2024-04-16T19:55:56.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M=', 'Content-Length': '0'} 2024-04-16T19:55:56.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273556e420e2756b-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.162 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.162 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.162 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.162 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.162 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.162 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.162 DEBUG:boto:Data: 2024-04-16T19:55:56.162 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.163 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.163 DEBUG:boto:Params: {} 2024-04-16T19:55:56.163 DEBUG:boto:Token: None 2024-04-16T19:55:56.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hh639XgyP111pXFc6Nuu/wta79s= 2024-04-16T19:55:56.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hh639XgyP111pXFc6Nuu/wta79s=', 'Content-Length': '0'} 2024-04-16T19:55:56.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036df76f73679c5a0-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.165 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.165 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.165 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.165 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.166 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.166 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.166 DEBUG:boto:Data: 2024-04-16T19:55:56.166 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.166 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.166 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.166 DEBUG:boto:Params: {} 2024-04-16T19:55:56.166 DEBUG:boto:Token: None 2024-04-16T19:55:56.166 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hh639XgyP111pXFc6Nuu/wta79s= 2024-04-16T19:55:56.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hh639XgyP111pXFc6Nuu/wta79s=', 'Content-Length': '0'} 2024-04-16T19:55:56.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92efdf8938f6510-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:55:56.169 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.169 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.169 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.169 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.169 DEBUG:boto:Method: GET 2024-04-16T19:55:56.169 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.169 DEBUG:boto:Data: 2024-04-16T19:55:56.169 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.169 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.170 DEBUG:boto:Params: {} 2024-04-16T19:55:56.170 DEBUG:boto:Token: None 2024-04-16T19:55:56.170 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.170 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M= 2024-04-16T19:55:56.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M=', 'Content-Length': '0'} 2024-04-16T19:55:56.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6b55b9b9e81543-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.214 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.214 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.215 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.215 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.215 DEBUG:boto:Method: GET 2024-04-16T19:55:56.215 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.215 DEBUG:boto:Data: 2024-04-16T19:55:56.215 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.215 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.215 DEBUG:boto:Params: {} 2024-04-16T19:55:56.215 DEBUG:boto:Token: None 2024-04-16T19:55:56.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:55:56.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M= 2024-04-16T19:55:56.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IAVz8FDA+C3WyqqKCIWGldFG71M=', 'Content-Length': '0'} 2024-04-16T19:55:56.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679f6d58890b979d-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.218 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.218 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.218 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.218 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.218 DEBUG:boto:Method: GET 2024-04-16T19:55:56.218 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.218 DEBUG:boto:Data: 2024-04-16T19:55:56.218 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.218 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.219 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.219 DEBUG:boto:Params: {} 2024-04-16T19:55:56.219 DEBUG:boto:Token: None 2024-04-16T19:55:56.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c61dd3b607f8b050-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.262 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.263 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.263 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.263 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.263 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.263 DEBUG:boto:Method: GET 2024-04-16T19:55:56.263 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.263 DEBUG:boto:Data: 2024-04-16T19:55:56.263 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.264 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.264 DEBUG:boto:Params: {} 2024-04-16T19:55:56.264 DEBUG:boto:Token: None 2024-04-16T19:55:56.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c2d55bff821243-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.266 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.267 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.267 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.267 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.267 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.267 DEBUG:boto:Data: 2024-04-16T19:55:56.267 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.267 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.267 DEBUG:boto:Params: {} 2024-04-16T19:55:56.268 DEBUG:boto:Token: None 2024-04-16T19:55:56.268 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T5sT+AHEtin3hmw1foFY13l3Q68= 2024-04-16T19:55:56.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T5sT+AHEtin3hmw1foFY13l3Q68=', 'Content-Length': '0'} 2024-04-16T19:55:56.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2fd668cc8d7aed-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.270 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.270 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.270 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.271 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.271 DEBUG:boto:Data: 2024-04-16T19:55:56.271 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.271 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.271 DEBUG:boto:Params: {} 2024-04-16T19:55:56.271 DEBUG:boto:Token: None 2024-04-16T19:55:56.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T5sT+AHEtin3hmw1foFY13l3Q68= 2024-04-16T19:55:56.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T5sT+AHEtin3hmw1foFY13l3Q68=', 'Content-Length': '0'} 2024-04-16T19:55:56.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb03272dec22d97-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:55:56.274 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.274 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.274 DEBUG:boto:Method: GET 2024-04-16T19:55:56.274 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.274 DEBUG:boto:Data: 2024-04-16T19:55:56.274 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.274 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.274 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.274 DEBUG:boto:Params: {} 2024-04-16T19:55:56.275 DEBUG:boto:Token: None 2024-04-16T19:55:56.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eU4teg5Iqs399xnLYT7zG1+BHQU= 2024-04-16T19:55:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eU4teg5Iqs399xnLYT7zG1+BHQU=', 'Content-Length': '0'} 2024-04-16T19:55:56.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004225b41f6a133867-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.318 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.319 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.319 DEBUG:boto:Method: GET 2024-04-16T19:55:56.319 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.319 DEBUG:boto:Data: 2024-04-16T19:55:56.319 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.319 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.319 DEBUG:boto:Params: {} 2024-04-16T19:55:56.319 DEBUG:boto:Token: None 2024-04-16T19:55:56.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:55:56.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eU4teg5Iqs399xnLYT7zG1+BHQU= 2024-04-16T19:55:56.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eU4teg5Iqs399xnLYT7zG1+BHQU=', 'Content-Length': '0'} 2024-04-16T19:55:56.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007940395c69ecaeda-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.322 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.322 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.322 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.322 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.322 DEBUG:boto:Method: GET 2024-04-16T19:55:56.322 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.322 DEBUG:boto:Data: 2024-04-16T19:55:56.322 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.323 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.323 DEBUG:boto:Params: {} 2024-04-16T19:55:56.323 DEBUG:boto:Token: None 2024-04-16T19:55:56.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001673166be1be17e5-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.326 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.326 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.326 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.326 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.327 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.327 DEBUG:boto:Method: GET 2024-04-16T19:55:56.327 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:55:56.327 DEBUG:boto:Data: 2024-04-16T19:55:56.327 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.327 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.327 DEBUG:boto:Params: {} 2024-04-16T19:55:56.327 DEBUG:boto:Token: None 2024-04-16T19:55:56.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1d067506b57f18-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.330 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:55:50.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:55:56.331 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.331 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.331 DEBUG:boto:Method: GET 2024-04-16T19:55:56.331 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.331 DEBUG:boto:Data: 2024-04-16T19:55:56.332 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.332 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.332 DEBUG:boto:Params: {} 2024-04-16T19:55:56.332 DEBUG:boto:Token: None 2024-04-16T19:55:56.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic= 2024-04-16T19:55:56.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic=', 'Content-Length': '0'} 2024-04-16T19:55:56.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f66cddcaf47a07-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.374 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.375 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.375 DEBUG:boto:Method: GET 2024-04-16T19:55:56.375 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.375 DEBUG:boto:Data: 2024-04-16T19:55:56.375 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.375 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.375 DEBUG:boto:Params: {} 2024-04-16T19:55:56.375 DEBUG:boto:Token: None 2024-04-16T19:55:56.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic= 2024-04-16T19:55:56.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic=', 'Content-Length': '0'} 2024-04-16T19:55:56.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815b0dd3fc67cb82-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.379 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.379 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.379 DEBUG:boto:Data: 2024-04-16T19:55:56.379 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.379 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.379 DEBUG:boto:Params: {} 2024-04-16T19:55:56.379 DEBUG:boto:Token: None 2024-04-16T19:55:56.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HuJT6VTytaj017E27P0Ttr8RPuw= 2024-04-16T19:55:56.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HuJT6VTytaj017E27P0Ttr8RPuw=', 'Content-Length': '0'} 2024-04-16T19:55:56.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6adf048b03f469d-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.383 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.384 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.384 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.384 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.384 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.384 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.384 DEBUG:boto:Data: 2024-04-16T19:55:56.384 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.384 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.384 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.384 DEBUG:boto:Params: {} 2024-04-16T19:55:56.385 DEBUG:boto:Token: None 2024-04-16T19:55:56.385 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HuJT6VTytaj017E27P0Ttr8RPuw= 2024-04-16T19:55:56.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HuJT6VTytaj017E27P0Ttr8RPuw=', 'Content-Length': '0'} 2024-04-16T19:55:56.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61aec566abc6314-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:55:56.387 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.387 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.387 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.387 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.388 DEBUG:boto:Method: GET 2024-04-16T19:55:56.388 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.388 DEBUG:boto:Data: 2024-04-16T19:55:56.388 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.388 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.388 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.388 DEBUG:boto:Params: {} 2024-04-16T19:55:56.388 DEBUG:boto:Token: None 2024-04-16T19:55:56.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic= 2024-04-16T19:55:56.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic=', 'Content-Length': '0'} 2024-04-16T19:55:56.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022b8d30e9d6e564-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.434 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.434 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.435 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.435 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.435 DEBUG:boto:Method: GET 2024-04-16T19:55:56.435 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.435 DEBUG:boto:Data: 2024-04-16T19:55:56.435 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.435 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.435 DEBUG:boto:Params: {} 2024-04-16T19:55:56.435 DEBUG:boto:Token: None 2024-04-16T19:55:56.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:55:56.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic= 2024-04-16T19:55:56.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1GxagVuXQExYpTn8vOqGnx1Fic=', 'Content-Length': '0'} 2024-04-16T19:55:56.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3239678fe4d3582-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.438 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.438 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.438 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.438 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.439 DEBUG:boto:Method: GET 2024-04-16T19:55:56.439 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.439 DEBUG:boto:Data: 2024-04-16T19:55:56.439 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.439 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.439 DEBUG:boto:Params: {} 2024-04-16T19:55:56.439 DEBUG:boto:Token: None 2024-04-16T19:55:56.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c0fcd373297242a-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.442 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.443 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.443 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.443 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.443 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.443 DEBUG:boto:Method: GET 2024-04-16T19:55:56.443 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.443 DEBUG:boto:Data: 2024-04-16T19:55:56.443 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.444 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.444 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.444 DEBUG:boto:Params: {} 2024-04-16T19:55:56.444 DEBUG:boto:Token: None 2024-04-16T19:55:56.444 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074cd3c0f49110b08-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.446 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.447 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.447 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.447 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.447 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.447 DEBUG:boto:Data: 2024-04-16T19:55:56.447 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.447 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.447 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.448 DEBUG:boto:Params: {} 2024-04-16T19:55:56.448 DEBUG:boto:Token: None 2024-04-16T19:55:56.448 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:795u0IYO+Q7UnOnjD4YwOt/RiF4= 2024-04-16T19:55:56.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:795u0IYO+Q7UnOnjD4YwOt/RiF4=', 'Content-Length': '0'} 2024-04-16T19:55:56.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d268c55a706628-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.450 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.450 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.450 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.450 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.451 DEBUG:boto:Data: 2024-04-16T19:55:56.451 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.451 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.451 DEBUG:boto:Params: {} 2024-04-16T19:55:56.451 DEBUG:boto:Token: None 2024-04-16T19:55:56.451 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:795u0IYO+Q7UnOnjD4YwOt/RiF4= 2024-04-16T19:55:56.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:795u0IYO+Q7UnOnjD4YwOt/RiF4=', 'Content-Length': '0'} 2024-04-16T19:55:56.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c29b35752d3baaec-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:55:56.453 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.454 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.454 DEBUG:boto:Method: GET 2024-04-16T19:55:56.454 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.454 DEBUG:boto:Data: 2024-04-16T19:55:56.454 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.454 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.454 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.454 DEBUG:boto:Params: {} 2024-04-16T19:55:56.454 DEBUG:boto:Token: None 2024-04-16T19:55:56.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lQ480lqjJBFeNb5q5myrJVjnB0Q= 2024-04-16T19:55:56.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lQ480lqjJBFeNb5q5myrJVjnB0Q=', 'Content-Length': '0'} 2024-04-16T19:55:56.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2963f8ae1f4f17e-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.498 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.498 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.499 DEBUG:boto:Method: GET 2024-04-16T19:55:56.499 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.499 DEBUG:boto:Data: 2024-04-16T19:55:56.499 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.499 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.499 DEBUG:boto:Params: {} 2024-04-16T19:55:56.499 DEBUG:boto:Token: None 2024-04-16T19:55:56.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:55:56.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lQ480lqjJBFeNb5q5myrJVjnB0Q= 2024-04-16T19:55:56.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lQ480lqjJBFeNb5q5myrJVjnB0Q=', 'Content-Length': '0'} 2024-04-16T19:55:56.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ecc17cf374a54f-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.502 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.502 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.502 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.502 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.502 DEBUG:boto:Method: GET 2024-04-16T19:55:56.502 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.502 DEBUG:boto:Data: 2024-04-16T19:55:56.503 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.503 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.503 DEBUG:boto:Params: {} 2024-04-16T19:55:56.503 DEBUG:boto:Token: None 2024-04-16T19:55:56.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f469ee895bb31d16-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.550 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.551 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.551 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.551 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.551 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.551 DEBUG:boto:Method: GET 2024-04-16T19:55:56.551 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:55:56.551 DEBUG:boto:Data: 2024-04-16T19:55:56.551 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.551 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.552 DEBUG:boto:Params: {} 2024-04-16T19:55:56.552 DEBUG:boto:Token: None 2024-04-16T19:55:56.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141402518af8b64d-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.555 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:55:50.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:55:56.555 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.555 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.556 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.556 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.556 DEBUG:boto:Method: GET 2024-04-16T19:55:56.556 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.556 DEBUG:boto:Data: 2024-04-16T19:55:56.556 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.556 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.556 DEBUG:boto:Params: {} 2024-04-16T19:55:56.556 DEBUG:boto:Token: None 2024-04-16T19:55:56.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.557 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk= 2024-04-16T19:55:56.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk=', 'Content-Length': '0'} 2024-04-16T19:55:56.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d46fb0b08fcfaec-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.603 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.603 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.603 DEBUG:boto:Method: GET 2024-04-16T19:55:56.603 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.603 DEBUG:boto:Data: 2024-04-16T19:55:56.603 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.603 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.603 DEBUG:boto:Params: {} 2024-04-16T19:55:56.603 DEBUG:boto:Token: None 2024-04-16T19:55:56.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk= 2024-04-16T19:55:56.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk=', 'Content-Length': '0'} 2024-04-16T19:55:56.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70ceee107fdff95-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.606 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.606 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.606 DEBUG:boto:Data: 2024-04-16T19:55:56.607 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.607 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.607 DEBUG:boto:Params: {} 2024-04-16T19:55:56.607 DEBUG:boto:Token: None 2024-04-16T19:55:56.607 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IXL8mQEcMiKmTDQaQSAypnfJPgQ= 2024-04-16T19:55:56.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IXL8mQEcMiKmTDQaQSAypnfJPgQ=', 'Content-Length': '0'} 2024-04-16T19:55:56.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a721ff23b565a362-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.609 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.609 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.609 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.610 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.610 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.610 DEBUG:boto:Data: 2024-04-16T19:55:56.610 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.610 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.610 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.610 DEBUG:boto:Params: {} 2024-04-16T19:55:56.610 DEBUG:boto:Token: None 2024-04-16T19:55:56.610 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IXL8mQEcMiKmTDQaQSAypnfJPgQ= 2024-04-16T19:55:56.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IXL8mQEcMiKmTDQaQSAypnfJPgQ=', 'Content-Length': '0'} 2024-04-16T19:55:56.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435fa9fbeda420e6-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:55:56.613 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.613 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.613 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.613 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.613 DEBUG:boto:Method: GET 2024-04-16T19:55:56.613 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.613 DEBUG:boto:Data: 2024-04-16T19:55:56.613 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.613 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.614 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.614 DEBUG:boto:Params: {} 2024-04-16T19:55:56.614 DEBUG:boto:Token: None 2024-04-16T19:55:56.614 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.614 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk= 2024-04-16T19:55:56.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk=', 'Content-Length': '0'} 2024-04-16T19:55:56.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776a29c1d8101acf-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.658 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.658 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.659 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.659 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.659 DEBUG:boto:Method: GET 2024-04-16T19:55:56.659 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.659 DEBUG:boto:Data: 2024-04-16T19:55:56.659 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.659 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.659 DEBUG:boto:Params: {} 2024-04-16T19:55:56.659 DEBUG:boto:Token: None 2024-04-16T19:55:56.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:55:56.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk= 2024-04-16T19:55:56.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R9SSyrVmlInAsbOthWJ8zs6TXqk=', 'Content-Length': '0'} 2024-04-16T19:55:56.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643433172d3d38ac-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.662 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.662 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.662 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.662 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.662 DEBUG:boto:Method: GET 2024-04-16T19:55:56.662 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.663 DEBUG:boto:Data: 2024-04-16T19:55:56.663 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.663 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.663 DEBUG:boto:Params: {} 2024-04-16T19:55:56.663 DEBUG:boto:Token: None 2024-04-16T19:55:56.663 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a00b1ca1f2fc96c-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.706 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.707 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.707 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.707 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.707 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.707 DEBUG:boto:Method: GET 2024-04-16T19:55:56.707 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.707 DEBUG:boto:Data: 2024-04-16T19:55:56.707 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.707 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.708 DEBUG:boto:Params: {} 2024-04-16T19:55:56.708 DEBUG:boto:Token: None 2024-04-16T19:55:56.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053cf94f17339fedf-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.711 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.711 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.711 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.711 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.711 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.711 DEBUG:boto:Data: 2024-04-16T19:55:56.712 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.712 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.712 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.712 DEBUG:boto:Params: {} 2024-04-16T19:55:56.712 DEBUG:boto:Token: None 2024-04-16T19:55:56.712 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tYqpefnLh9e3iHYDCNkgkXKip74= 2024-04-16T19:55:56.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tYqpefnLh9e3iHYDCNkgkXKip74=', 'Content-Length': '0'} 2024-04-16T19:55:56.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b80510b3cb9c5d-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.714 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.715 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.715 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.715 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.715 DEBUG:boto:Data: 2024-04-16T19:55:56.715 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.715 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.715 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.715 DEBUG:boto:Params: {} 2024-04-16T19:55:56.715 DEBUG:boto:Token: None 2024-04-16T19:55:56.715 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tYqpefnLh9e3iHYDCNkgkXKip74= 2024-04-16T19:55:56.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tYqpefnLh9e3iHYDCNkgkXKip74=', 'Content-Length': '0'} 2024-04-16T19:55:56.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d313986d0aad3f81-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:55:56.718 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.718 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.718 DEBUG:boto:Method: GET 2024-04-16T19:55:56.718 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.718 DEBUG:boto:Data: 2024-04-16T19:55:56.718 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.718 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.718 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.718 DEBUG:boto:Params: {} 2024-04-16T19:55:56.718 DEBUG:boto:Token: None 2024-04-16T19:55:56.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mLgxhRqhHkFyrHciuRxnZXYNVTw= 2024-04-16T19:55:56.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mLgxhRqhHkFyrHciuRxnZXYNVTw=', 'Content-Length': '0'} 2024-04-16T19:55:56.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f40718202c5553-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.762 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.763 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.763 DEBUG:boto:Method: GET 2024-04-16T19:55:56.763 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.763 DEBUG:boto:Data: 2024-04-16T19:55:56.763 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.763 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.763 DEBUG:boto:Params: {} 2024-04-16T19:55:56.763 DEBUG:boto:Token: None 2024-04-16T19:55:56.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:55:56.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mLgxhRqhHkFyrHciuRxnZXYNVTw= 2024-04-16T19:55:56.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mLgxhRqhHkFyrHciuRxnZXYNVTw=', 'Content-Length': '0'} 2024-04-16T19:55:56.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fafc9d02d161bf73-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.766 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.766 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.766 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.766 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.766 DEBUG:boto:Method: GET 2024-04-16T19:55:56.766 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.766 DEBUG:boto:Data: 2024-04-16T19:55:56.766 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.766 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.767 DEBUG:boto:Params: {} 2024-04-16T19:55:56.767 DEBUG:boto:Token: None 2024-04-16T19:55:56.767 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027194d36bfd3e0a7-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.770 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.770 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.770 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.770 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.770 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.770 DEBUG:boto:Method: GET 2024-04-16T19:55:56.770 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:55:56.771 DEBUG:boto:Data: 2024-04-16T19:55:56.771 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.771 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.771 DEBUG:boto:Params: {} 2024-04-16T19:55:56.771 DEBUG:boto:Token: None 2024-04-16T19:55:56.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb791b5daa20bdb-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.774 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:55:50.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:55:56.774 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.774 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.775 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.775 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.775 DEBUG:boto:Method: GET 2024-04-16T19:55:56.775 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.775 DEBUG:boto:Data: 2024-04-16T19:55:56.775 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.775 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.775 DEBUG:boto:Params: {} 2024-04-16T19:55:56.775 DEBUG:boto:Token: None 2024-04-16T19:55:56.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo= 2024-04-16T19:55:56.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo=', 'Content-Length': '0'} 2024-04-16T19:55:56.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b48cf7d8571eaa-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.818 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.818 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.819 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.819 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.819 DEBUG:boto:Method: GET 2024-04-16T19:55:56.819 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.819 DEBUG:boto:Data: 2024-04-16T19:55:56.819 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.819 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.819 DEBUG:boto:Params: {} 2024-04-16T19:55:56.819 DEBUG:boto:Token: None 2024-04-16T19:55:56.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo= 2024-04-16T19:55:56.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo=', 'Content-Length': '0'} 2024-04-16T19:55:56.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020bd4741522d80aa-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.822 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.822 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.822 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.822 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.823 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.823 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.823 DEBUG:boto:Data: 2024-04-16T19:55:56.823 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.823 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.823 DEBUG:boto:Params: {} 2024-04-16T19:55:56.823 DEBUG:boto:Token: None 2024-04-16T19:55:56.823 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qW1UEuCC0cYzB8RyQdowL1sbedo= 2024-04-16T19:55:56.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qW1UEuCC0cYzB8RyQdowL1sbedo=', 'Content-Length': '0'} 2024-04-16T19:55:56.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cb45a2eb6488e17-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.826 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.826 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.826 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.826 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.826 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.826 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.826 DEBUG:boto:Data: 2024-04-16T19:55:56.826 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.827 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.827 DEBUG:boto:Params: {} 2024-04-16T19:55:56.827 DEBUG:boto:Token: None 2024-04-16T19:55:56.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qW1UEuCC0cYzB8RyQdowL1sbedo= 2024-04-16T19:55:56.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qW1UEuCC0cYzB8RyQdowL1sbedo=', 'Content-Length': '0'} 2024-04-16T19:55:56.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a729a04e9f133ad8-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:55:56.830 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.830 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.830 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.830 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.830 DEBUG:boto:Method: GET 2024-04-16T19:55:56.830 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.830 DEBUG:boto:Data: 2024-04-16T19:55:56.831 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.831 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.831 DEBUG:boto:Params: {} 2024-04-16T19:55:56.831 DEBUG:boto:Token: None 2024-04-16T19:55:56.831 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo= 2024-04-16T19:55:56.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo=', 'Content-Length': '0'} 2024-04-16T19:55:56.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004edacb91c16ba409-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.878 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.879 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.879 DEBUG:boto:Method: GET 2024-04-16T19:55:56.879 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.879 DEBUG:boto:Data: 2024-04-16T19:55:56.879 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.879 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.879 DEBUG:boto:Params: {} 2024-04-16T19:55:56.880 DEBUG:boto:Token: None 2024-04-16T19:55:56.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:55:56.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo= 2024-04-16T19:55:56.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+exAmPjIG2rY4oqjvRuoY45bvNo=', 'Content-Length': '0'} 2024-04-16T19:55:56.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5bac6547b7d491c-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.882 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.883 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.883 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.883 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.883 DEBUG:boto:Method: GET 2024-04-16T19:55:56.883 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.883 DEBUG:boto:Data: 2024-04-16T19:55:56.883 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.883 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.883 DEBUG:boto:Params: {} 2024-04-16T19:55:56.884 DEBUG:boto:Token: None 2024-04-16T19:55:56.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000182f1c871937ae7a-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.886 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.887 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.887 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.887 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.887 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.887 DEBUG:boto:Method: GET 2024-04-16T19:55:56.887 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.887 DEBUG:boto:Data: 2024-04-16T19:55:56.888 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.888 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.888 DEBUG:boto:Params: {} 2024-04-16T19:55:56.888 DEBUG:boto:Token: None 2024-04-16T19:55:56.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1150a5ef480876-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.891 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.891 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.891 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.892 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.892 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.892 DEBUG:boto:Data: 2024-04-16T19:55:56.892 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.892 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.892 DEBUG:boto:Params: {} 2024-04-16T19:55:56.892 DEBUG:boto:Token: None 2024-04-16T19:55:56.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rl3grSr//S+gILPeb/NZyxf7pxo= 2024-04-16T19:55:56.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rl3grSr//S+gILPeb/NZyxf7pxo=', 'Content-Length': '0'} 2024-04-16T19:55:56.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8ab9e5333bf3ce-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.895 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.895 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.895 DEBUG:boto:Method: HEAD 2024-04-16T19:55:56.895 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.895 DEBUG:boto:Data: 2024-04-16T19:55:56.895 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.895 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.895 DEBUG:boto:Params: {} 2024-04-16T19:55:56.896 DEBUG:boto:Token: None 2024-04-16T19:55:56.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rl3grSr//S+gILPeb/NZyxf7pxo= 2024-04-16T19:55:56.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rl3grSr//S+gILPeb/NZyxf7pxo=', 'Content-Length': '0'} 2024-04-16T19:55:56.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d590b5e0c8f0492-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:55:56.898 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.898 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.898 DEBUG:boto:Method: GET 2024-04-16T19:55:56.898 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.898 DEBUG:boto:Data: 2024-04-16T19:55:56.898 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.898 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.899 DEBUG:boto:Params: {} 2024-04-16T19:55:56.899 DEBUG:boto:Token: None 2024-04-16T19:55:56.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8xCzHQIUm149yvGwp+j4g1QU+mw= 2024-04-16T19:55:56.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8xCzHQIUm149yvGwp+j4g1QU+mw=', 'Content-Length': '0'} 2024-04-16T19:55:56.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009070af85ea66ae6f-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.947 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.947 DEBUG:boto:Method: GET 2024-04-16T19:55:56.947 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.947 DEBUG:boto:Data: 2024-04-16T19:55:56.947 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.947 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.947 DEBUG:boto:Params: {} 2024-04-16T19:55:56.947 DEBUG:boto:Token: None 2024-04-16T19:55:56.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:55:56.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8xCzHQIUm149yvGwp+j4g1QU+mw= 2024-04-16T19:55:56.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8xCzHQIUm149yvGwp+j4g1QU+mw=', 'Content-Length': '0'} 2024-04-16T19:55:56.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec95dd4b13931d7d-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.950 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.950 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.950 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.950 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.951 DEBUG:boto:Method: GET 2024-04-16T19:55:56.951 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.951 DEBUG:boto:Data: 2024-04-16T19:55:56.951 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:56.951 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.951 DEBUG:boto:Params: {} 2024-04-16T19:55:56.951 DEBUG:boto:Token: None 2024-04-16T19:55:56.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed619f4e12c0bc2c-00661ed7cc-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.994 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.995 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:56.995 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:56.995 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.995 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.995 DEBUG:boto:Method: GET 2024-04-16T19:55:56.995 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:55:56.995 DEBUG:boto:Data: 2024-04-16T19:55:56.995 DEBUG:boto:Headers: {} 2024-04-16T19:55:56.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:56.995 DEBUG:boto:Port: 8000 2024-04-16T19:55:56.996 DEBUG:boto:Params: {} 2024-04-16T19:55:56.996 DEBUG:boto:Token: None 2024-04-16T19:55:56.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:56 GMT /yrjefh-51/ 2024-04-16T19:55:56.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk= 2024-04-16T19:55:56.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DdHAVRVf5BNhRDGYw82tAj/38Pk=', 'Content-Length': '0'} 2024-04-16T19:55:56.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dddd271eccf5a006-00661ed7cc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:56.998 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:55:50.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:56.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:55:56.999 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:56.999 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:56.999 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:56.999 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:56.999 DEBUG:boto:Method: GET 2024-04-16T19:55:56.999 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:56.999 DEBUG:boto:Data: 2024-04-16T19:55:57.000 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.000 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.000 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.000 DEBUG:boto:Params: {} 2024-04-16T19:55:57.000 DEBUG:boto:Token: None 2024-04-16T19:55:57.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es= 2024-04-16T19:55:57.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es=', 'Content-Length': '0'} 2024-04-16T19:55:57.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009552bb76ffda7a1b-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.046 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.046 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.047 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.047 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.047 DEBUG:boto:Method: GET 2024-04-16T19:55:57.047 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.047 DEBUG:boto:Data: 2024-04-16T19:55:57.047 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.047 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.047 DEBUG:boto:Params: {} 2024-04-16T19:55:57.047 DEBUG:boto:Token: None 2024-04-16T19:55:57.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es= 2024-04-16T19:55:57.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es=', 'Content-Length': '0'} 2024-04-16T19:55:57.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af44c2dad9ba7b9-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.050 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.050 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.051 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.051 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.051 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.051 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.051 DEBUG:boto:Data: 2024-04-16T19:55:57.051 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.051 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.051 DEBUG:boto:Params: {} 2024-04-16T19:55:57.051 DEBUG:boto:Token: None 2024-04-16T19:55:57.052 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5sYfOIkgRk0ZCbmGEA+XTaAS1tI= 2024-04-16T19:55:57.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5sYfOIkgRk0ZCbmGEA+XTaAS1tI=', 'Content-Length': '0'} 2024-04-16T19:55:57.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b39a7db7f918e2d-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.054 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.054 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.054 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.054 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.055 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.055 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.055 DEBUG:boto:Data: 2024-04-16T19:55:57.055 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.055 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.055 DEBUG:boto:Params: {} 2024-04-16T19:55:57.055 DEBUG:boto:Token: None 2024-04-16T19:55:57.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5sYfOIkgRk0ZCbmGEA+XTaAS1tI= 2024-04-16T19:55:57.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5sYfOIkgRk0ZCbmGEA+XTaAS1tI=', 'Content-Length': '0'} 2024-04-16T19:55:57.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777cf9ff7e0fc529-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:55:57.058 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.058 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.058 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.058 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.059 DEBUG:boto:Method: GET 2024-04-16T19:55:57.059 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.059 DEBUG:boto:Data: 2024-04-16T19:55:57.059 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.059 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.059 DEBUG:boto:Params: {} 2024-04-16T19:55:57.059 DEBUG:boto:Token: None 2024-04-16T19:55:57.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es= 2024-04-16T19:55:57.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es=', 'Content-Length': '0'} 2024-04-16T19:55:57.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2bff54a6605d22b-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.102 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.103 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.103 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.103 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.103 DEBUG:boto:Method: GET 2024-04-16T19:55:57.103 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.103 DEBUG:boto:Data: 2024-04-16T19:55:57.103 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.103 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.103 DEBUG:boto:Params: {} 2024-04-16T19:55:57.103 DEBUG:boto:Token: None 2024-04-16T19:55:57.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:55:57.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es= 2024-04-16T19:55:57.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3HiLIR6NSf/ZmnDvI+jVREdP7es=', 'Content-Length': '0'} 2024-04-16T19:55:57.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b8bb12ad222e18-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.106 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.107 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.107 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.107 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.107 DEBUG:boto:Method: GET 2024-04-16T19:55:57.107 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.107 DEBUG:boto:Data: 2024-04-16T19:55:57.107 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.107 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.107 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.107 DEBUG:boto:Params: {} 2024-04-16T19:55:57.107 DEBUG:boto:Token: None 2024-04-16T19:55:57.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b70baca9e2c3d7a-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.154 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.155 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.155 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.155 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.155 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.155 DEBUG:boto:Method: GET 2024-04-16T19:55:57.155 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.155 DEBUG:boto:Data: 2024-04-16T19:55:57.155 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.156 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.156 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.156 DEBUG:boto:Params: {} 2024-04-16T19:55:57.156 DEBUG:boto:Token: None 2024-04-16T19:55:57.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd15b6a544d552a-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.159 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.159 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.159 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.159 DEBUG:boto:Data: 2024-04-16T19:55:57.159 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.160 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.160 DEBUG:boto:Params: {} 2024-04-16T19:55:57.160 DEBUG:boto:Token: None 2024-04-16T19:55:57.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nFF3vLS1P3knNlvnOtHzlTUkuyo= 2024-04-16T19:55:57.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nFF3vLS1P3knNlvnOtHzlTUkuyo=', 'Content-Length': '0'} 2024-04-16T19:55:57.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf19ef71831ace4-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.163 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.163 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.163 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.163 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.163 DEBUG:boto:Data: 2024-04-16T19:55:57.163 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.163 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.164 DEBUG:boto:Params: {} 2024-04-16T19:55:57.164 DEBUG:boto:Token: None 2024-04-16T19:55:57.164 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nFF3vLS1P3knNlvnOtHzlTUkuyo= 2024-04-16T19:55:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nFF3vLS1P3knNlvnOtHzlTUkuyo=', 'Content-Length': '0'} 2024-04-16T19:55:57.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001423aef994da96ed-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:55:57.167 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.167 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.167 DEBUG:boto:Method: GET 2024-04-16T19:55:57.167 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.167 DEBUG:boto:Data: 2024-04-16T19:55:57.167 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.168 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.168 DEBUG:boto:Params: {} 2024-04-16T19:55:57.168 DEBUG:boto:Token: None 2024-04-16T19:55:57.168 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dSAIVwHruonPY/F0C0namIgfVn8= 2024-04-16T19:55:57.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dSAIVwHruonPY/F0C0namIgfVn8=', 'Content-Length': '0'} 2024-04-16T19:55:57.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3882b8793a5b102-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.214 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.214 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.215 DEBUG:boto:Method: GET 2024-04-16T19:55:57.215 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.215 DEBUG:boto:Data: 2024-04-16T19:55:57.215 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.215 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.215 DEBUG:boto:Params: {} 2024-04-16T19:55:57.215 DEBUG:boto:Token: None 2024-04-16T19:55:57.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:55:57.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dSAIVwHruonPY/F0C0namIgfVn8= 2024-04-16T19:55:57.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dSAIVwHruonPY/F0C0namIgfVn8=', 'Content-Length': '0'} 2024-04-16T19:55:57.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000868ccbf17111dddd-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.218 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.218 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.218 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.218 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.219 DEBUG:boto:Method: GET 2024-04-16T19:55:57.219 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.219 DEBUG:boto:Data: 2024-04-16T19:55:57.219 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.219 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.219 DEBUG:boto:Params: {} 2024-04-16T19:55:57.219 DEBUG:boto:Token: None 2024-04-16T19:55:57.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02477a473c632db-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.262 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.263 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.263 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.263 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.263 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.263 DEBUG:boto:Method: GET 2024-04-16T19:55:57.263 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:55:57.263 DEBUG:boto:Data: 2024-04-16T19:55:57.264 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.264 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.264 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.264 DEBUG:boto:Params: {} 2024-04-16T19:55:57.264 DEBUG:boto:Token: None 2024-04-16T19:55:57.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050ab38d6ef99639-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.267 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:55:50.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:55:57.268 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.268 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.268 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.268 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.268 DEBUG:boto:Method: GET 2024-04-16T19:55:57.268 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.268 DEBUG:boto:Data: 2024-04-16T19:55:57.268 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.269 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.269 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.269 DEBUG:boto:Params: {} 2024-04-16T19:55:57.269 DEBUG:boto:Token: None 2024-04-16T19:55:57.269 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78= 2024-04-16T19:55:57.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78=', 'Content-Length': '0'} 2024-04-16T19:55:57.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce0946fa4e14c2e-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.314 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.315 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.315 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.315 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.315 DEBUG:boto:Method: GET 2024-04-16T19:55:57.315 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.315 DEBUG:boto:Data: 2024-04-16T19:55:57.315 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.315 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.316 DEBUG:boto:Params: {} 2024-04-16T19:55:57.316 DEBUG:boto:Token: None 2024-04-16T19:55:57.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78= 2024-04-16T19:55:57.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78=', 'Content-Length': '0'} 2024-04-16T19:55:57.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419f538e7fd15c99-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.318 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.319 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.319 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.319 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.319 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.319 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.319 DEBUG:boto:Data: 2024-04-16T19:55:57.319 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.319 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.319 DEBUG:boto:Params: {} 2024-04-16T19:55:57.320 DEBUG:boto:Token: None 2024-04-16T19:55:57.320 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yITboXYYHLSXaQ5SwfQOMrN6iDQ= 2024-04-16T19:55:57.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yITboXYYHLSXaQ5SwfQOMrN6iDQ=', 'Content-Length': '0'} 2024-04-16T19:55:57.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf348455c51e7b7-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.322 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.322 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.323 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.323 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.323 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.323 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.323 DEBUG:boto:Data: 2024-04-16T19:55:57.323 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.323 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.323 DEBUG:boto:Params: {} 2024-04-16T19:55:57.323 DEBUG:boto:Token: None 2024-04-16T19:55:57.324 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yITboXYYHLSXaQ5SwfQOMrN6iDQ= 2024-04-16T19:55:57.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yITboXYYHLSXaQ5SwfQOMrN6iDQ=', 'Content-Length': '0'} 2024-04-16T19:55:57.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000074b3676989caae2-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:55:57.326 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.326 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.326 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.326 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.327 DEBUG:boto:Method: GET 2024-04-16T19:55:57.327 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.327 DEBUG:boto:Data: 2024-04-16T19:55:57.327 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.327 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.327 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.327 DEBUG:boto:Params: {} 2024-04-16T19:55:57.327 DEBUG:boto:Token: None 2024-04-16T19:55:57.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78= 2024-04-16T19:55:57.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78=', 'Content-Length': '0'} 2024-04-16T19:55:57.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8ef20de530dcb8-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.374 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.375 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.375 DEBUG:boto:Method: GET 2024-04-16T19:55:57.375 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.375 DEBUG:boto:Data: 2024-04-16T19:55:57.375 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.375 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.376 DEBUG:boto:Params: {} 2024-04-16T19:55:57.376 DEBUG:boto:Token: None 2024-04-16T19:55:57.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:55:57.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78= 2024-04-16T19:55:57.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l80fcixQehfMATzJ0bA9gj6bn78=', 'Content-Length': '0'} 2024-04-16T19:55:57.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026c5e4a07932e2f6-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.379 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.379 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.379 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.379 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.379 DEBUG:boto:Method: GET 2024-04-16T19:55:57.379 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.379 DEBUG:boto:Data: 2024-04-16T19:55:57.379 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.379 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.380 DEBUG:boto:Params: {} 2024-04-16T19:55:57.380 DEBUG:boto:Token: None 2024-04-16T19:55:57.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b58d0ff5aa08ca6-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.426 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.427 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.427 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.427 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.427 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.427 DEBUG:boto:Method: GET 2024-04-16T19:55:57.427 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.427 DEBUG:boto:Data: 2024-04-16T19:55:57.428 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.428 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.428 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.428 DEBUG:boto:Params: {} 2024-04-16T19:55:57.428 DEBUG:boto:Token: None 2024-04-16T19:55:57.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d77fee4d41eff870-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.431 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.431 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.431 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.432 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.432 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.432 DEBUG:boto:Data: 2024-04-16T19:55:57.432 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.432 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.432 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.432 DEBUG:boto:Params: {} 2024-04-16T19:55:57.432 DEBUG:boto:Token: None 2024-04-16T19:55:57.432 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uvOywDnMQQm0vEzTZjrvnaGKsOA= 2024-04-16T19:55:57.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uvOywDnMQQm0vEzTZjrvnaGKsOA=', 'Content-Length': '0'} 2024-04-16T19:55:57.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb76dfd1e50a010-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.435 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.435 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.435 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.435 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.436 DEBUG:boto:Data: 2024-04-16T19:55:57.436 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.436 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.436 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.436 DEBUG:boto:Params: {} 2024-04-16T19:55:57.436 DEBUG:boto:Token: None 2024-04-16T19:55:57.436 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uvOywDnMQQm0vEzTZjrvnaGKsOA= 2024-04-16T19:55:57.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uvOywDnMQQm0vEzTZjrvnaGKsOA=', 'Content-Length': '0'} 2024-04-16T19:55:57.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009199d533ce93df9-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:55:57.439 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.439 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.439 DEBUG:boto:Method: GET 2024-04-16T19:55:57.439 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.439 DEBUG:boto:Data: 2024-04-16T19:55:57.439 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.439 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.439 DEBUG:boto:Params: {} 2024-04-16T19:55:57.440 DEBUG:boto:Token: None 2024-04-16T19:55:57.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o26M29E9SK/zFevMP7xbaY/BmEY= 2024-04-16T19:55:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o26M29E9SK/zFevMP7xbaY/BmEY=', 'Content-Length': '0'} 2024-04-16T19:55:57.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636adbe0eb691e9c-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.482 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.483 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.483 DEBUG:boto:Method: GET 2024-04-16T19:55:57.483 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.483 DEBUG:boto:Data: 2024-04-16T19:55:57.483 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.483 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.483 DEBUG:boto:Params: {} 2024-04-16T19:55:57.484 DEBUG:boto:Token: None 2024-04-16T19:55:57.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:55:57.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o26M29E9SK/zFevMP7xbaY/BmEY= 2024-04-16T19:55:57.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o26M29E9SK/zFevMP7xbaY/BmEY=', 'Content-Length': '0'} 2024-04-16T19:55:57.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b128a25a00d5beec-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.486 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.486 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.487 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.487 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.487 DEBUG:boto:Method: GET 2024-04-16T19:55:57.487 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.487 DEBUG:boto:Data: 2024-04-16T19:55:57.487 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.487 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.487 DEBUG:boto:Params: {} 2024-04-16T19:55:57.487 DEBUG:boto:Token: None 2024-04-16T19:55:57.488 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760c7f4684bf9f35-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.534 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.535 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.535 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.535 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.535 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.535 DEBUG:boto:Method: GET 2024-04-16T19:55:57.535 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:55:57.536 DEBUG:boto:Data: 2024-04-16T19:55:57.536 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.536 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.536 DEBUG:boto:Params: {} 2024-04-16T19:55:57.536 DEBUG:boto:Token: None 2024-04-16T19:55:57.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb3b3bd14ecb0e4-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.539 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:55:50.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:55:57.540 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.540 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.540 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.540 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.540 DEBUG:boto:Method: GET 2024-04-16T19:55:57.540 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.540 DEBUG:boto:Data: 2024-04-16T19:55:57.540 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.541 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.541 DEBUG:boto:Params: {} 2024-04-16T19:55:57.541 DEBUG:boto:Token: None 2024-04-16T19:55:57.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk= 2024-04-16T19:55:57.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk=', 'Content-Length': '0'} 2024-04-16T19:55:57.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8791ba4ac34ebe-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.586 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.587 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.587 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.587 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.587 DEBUG:boto:Method: GET 2024-04-16T19:55:57.587 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.587 DEBUG:boto:Data: 2024-04-16T19:55:57.587 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.587 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.588 DEBUG:boto:Params: {} 2024-04-16T19:55:57.588 DEBUG:boto:Token: None 2024-04-16T19:55:57.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk= 2024-04-16T19:55:57.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk=', 'Content-Length': '0'} 2024-04-16T19:55:57.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057efe9d730043f97-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.590 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.591 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.591 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.591 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.591 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.591 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.591 DEBUG:boto:Data: 2024-04-16T19:55:57.591 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.591 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.591 DEBUG:boto:Params: {} 2024-04-16T19:55:57.591 DEBUG:boto:Token: None 2024-04-16T19:55:57.592 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wDtz5nrbykz7uN57gofllT3qOrQ= 2024-04-16T19:55:57.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wDtz5nrbykz7uN57gofllT3qOrQ=', 'Content-Length': '0'} 2024-04-16T19:55:57.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76a4d87aa6a6ec3-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.594 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.594 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.594 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.594 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.594 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.594 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.595 DEBUG:boto:Data: 2024-04-16T19:55:57.595 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.595 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.595 DEBUG:boto:Params: {} 2024-04-16T19:55:57.595 DEBUG:boto:Token: None 2024-04-16T19:55:57.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wDtz5nrbykz7uN57gofllT3qOrQ= 2024-04-16T19:55:57.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wDtz5nrbykz7uN57gofllT3qOrQ=', 'Content-Length': '0'} 2024-04-16T19:55:57.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd5edef3898c665-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:55:57.597 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.598 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.598 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.598 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.598 DEBUG:boto:Method: GET 2024-04-16T19:55:57.598 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.598 DEBUG:boto:Data: 2024-04-16T19:55:57.598 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.598 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.598 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.598 DEBUG:boto:Params: {} 2024-04-16T19:55:57.599 DEBUG:boto:Token: None 2024-04-16T19:55:57.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk= 2024-04-16T19:55:57.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk=', 'Content-Length': '0'} 2024-04-16T19:55:57.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020cd4f678afb8f5c-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.642 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.643 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.643 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.643 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.643 DEBUG:boto:Method: GET 2024-04-16T19:55:57.643 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.643 DEBUG:boto:Data: 2024-04-16T19:55:57.643 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.643 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.643 DEBUG:boto:Params: {} 2024-04-16T19:55:57.644 DEBUG:boto:Token: None 2024-04-16T19:55:57.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:55:57.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk= 2024-04-16T19:55:57.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6TarOwUL5OlzMvjZA5lcSfU/llk=', 'Content-Length': '0'} 2024-04-16T19:55:57.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000094aed828732ba67-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.646 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.646 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.646 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.647 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.647 DEBUG:boto:Method: GET 2024-04-16T19:55:57.647 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.647 DEBUG:boto:Data: 2024-04-16T19:55:57.647 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.647 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.647 DEBUG:boto:Params: {} 2024-04-16T19:55:57.647 DEBUG:boto:Token: None 2024-04-16T19:55:57.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244f0b19f1882c0e-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.690 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.691 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.691 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.691 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.691 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.691 DEBUG:boto:Method: GET 2024-04-16T19:55:57.691 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.692 DEBUG:boto:Data: 2024-04-16T19:55:57.692 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.692 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.692 DEBUG:boto:Params: {} 2024-04-16T19:55:57.692 DEBUG:boto:Token: None 2024-04-16T19:55:57.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000668dd5ab3957f5dd-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.695 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.696 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.696 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.696 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.696 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.696 DEBUG:boto:Data: 2024-04-16T19:55:57.696 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.696 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.696 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.697 DEBUG:boto:Params: {} 2024-04-16T19:55:57.697 DEBUG:boto:Token: None 2024-04-16T19:55:57.697 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.697 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wzxy8qG0cGqrI3+i0Vufu+14k+w= 2024-04-16T19:55:57.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wzxy8qG0cGqrI3+i0Vufu+14k+w=', 'Content-Length': '0'} 2024-04-16T19:55:57.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c02a88ab505a0a0-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.699 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.700 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.700 DEBUG:boto:Method: HEAD 2024-04-16T19:55:57.700 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.700 DEBUG:boto:Data: 2024-04-16T19:55:57.700 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.700 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.700 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.700 DEBUG:boto:Params: {} 2024-04-16T19:55:57.701 DEBUG:boto:Token: None 2024-04-16T19:55:57.701 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.701 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wzxy8qG0cGqrI3+i0Vufu+14k+w= 2024-04-16T19:55:57.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wzxy8qG0cGqrI3+i0Vufu+14k+w=', 'Content-Length': '0'} 2024-04-16T19:55:57.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035639cafe7fa14ef-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:55:57.703 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.703 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.703 DEBUG:boto:Method: GET 2024-04-16T19:55:57.704 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.704 DEBUG:boto:Data: 2024-04-16T19:55:57.704 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.704 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.704 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.704 DEBUG:boto:Params: {} 2024-04-16T19:55:57.704 DEBUG:boto:Token: None 2024-04-16T19:55:57.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bDGp1cIVboUnAbl/UCqhxnnLl6I= 2024-04-16T19:55:57.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bDGp1cIVboUnAbl/UCqhxnnLl6I=', 'Content-Length': '0'} 2024-04-16T19:55:57.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:52 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4d89de8ff7689b-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.750 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.750 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.751 DEBUG:boto:Method: GET 2024-04-16T19:55:57.751 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.751 DEBUG:boto:Data: 2024-04-16T19:55:57.751 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.751 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.751 DEBUG:boto:Params: {} 2024-04-16T19:55:57.751 DEBUG:boto:Token: None 2024-04-16T19:55:57.752 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:55:57.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bDGp1cIVboUnAbl/UCqhxnnLl6I= 2024-04-16T19:55:57.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bDGp1cIVboUnAbl/UCqhxnnLl6I=', 'Content-Length': '0'} 2024-04-16T19:55:57.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8e28ff14160d21-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.754 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.754 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.754 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.754 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.755 DEBUG:boto:Method: GET 2024-04-16T19:55:57.755 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.755 DEBUG:boto:Data: 2024-04-16T19:55:57.755 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:55:57.755 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.755 DEBUG:boto:Params: {} 2024-04-16T19:55:57.755 DEBUG:boto:Token: None 2024-04-16T19:55:57.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5116dc22237fed-00661ed7cd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.802 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.803 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:55:57.803 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:55:57.803 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.803 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.803 DEBUG:boto:Method: GET 2024-04-16T19:55:57.803 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:55:57.804 DEBUG:boto:Data: 2024-04-16T19:55:57.804 DEBUG:boto:Headers: {} 2024-04-16T19:55:57.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:55:57.804 DEBUG:boto:Port: 8000 2024-04-16T19:55:57.804 DEBUG:boto:Params: {} 2024-04-16T19:55:57.804 DEBUG:boto:Token: None 2024-04-16T19:55:57.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:55:57 GMT /yrjefh-51/ 2024-04-16T19:55:57.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY= 2024-04-16T19:55:57.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrQuCzPcsgZs2vDsNk0cD2wVcnY=', 'Content-Length': '0'} 2024-04-16T19:55:57.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4bda5314a7ef6a-00661ed7cd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:55:57.807 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:55:50.158Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:55:57.808 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-51 zones={test-zone2, test-zone1} 2024-04-16T19:55:57.808 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T19:55:57.808 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T19:55:57.809 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:57.809+0000 7f993fef1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 125783) UID: 0 2024-04-16T19:55:57.810 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:55:57.809+0000 7f994602e880 -1 shutting down 2024-04-16T19:55:58.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T19:56:03.908 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T19:56:03.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:03.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:03.909 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:04.035 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:04.035 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:04.040 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 10 2024-04-16T19:56:04.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:04.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:04.093 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:04.143 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:04.143 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:04.213 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000007","obj00000000000000000016","obj00000000000000000024","obj00000000000000000020","obj00000000000000000028","obj00000000000000000000","obj00000000000000000032","obj00000000000000000001","obj00000000000000000010","obj00000000000000000019","obj00000000000000000013","obj00000000000000000021","obj00000000000000000030","obj00000000000000000026","obj00000000000000000033","obj00000000000000000004","obj00000000000000000076"]} 2024-04-16T19:56:04.213 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.213 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.213 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.213 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.213 DEBUG:boto:Data: 2024-04-16T19:56:04.213 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.213 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.214 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.214 DEBUG:boto:Params: {} 2024-04-16T19:56:04.214 DEBUG:boto:Token: None 2024-04-16T19:56:04.214 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.214 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2addcb31e342935-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.217 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.217 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.217 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.217 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.217 DEBUG:boto:Data: 2024-04-16T19:56:04.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.218 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.218 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.218 DEBUG:boto:Params: {} 2024-04-16T19:56:04.218 DEBUG:boto:Token: None 2024-04-16T19:56:04.218 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.218 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bYPzS+dtgI+35USqkdO+0sO+GQ4= 2024-04-16T19:56:04.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bYPzS+dtgI+35USqkdO+0sO+GQ4='} 2024-04-16T19:56:04.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ad7888ced2ef31f-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.226 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.226 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.226 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.226 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.226 DEBUG:boto:Data: 2024-04-16T19:56:04.226 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.226 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.226 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.226 DEBUG:boto:Params: {} 2024-04-16T19:56:04.226 DEBUG:boto:Token: None 2024-04-16T19:56:04.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000071bb0ebb989ef83-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.229 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:04.229 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:04.229 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.229 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:04.229 DEBUG:boto:Data: 2024-04-16T19:56:04.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.229 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.229 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.229 DEBUG:boto:Params: {} 2024-04-16T19:56:04.229 DEBUG:boto:Token: None 2024-04-16T19:56:04.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:04.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MXgzeKEvQYdEX8lShxPZhcDe+kY= 2024-04-16T19:56:04.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MXgzeKEvQYdEX8lShxPZhcDe+kY='} 2024-04-16T19:56:04.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ffbfed2b24011a3-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.236 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.236 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.236 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.237 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.237 DEBUG:boto:Data: 2024-04-16T19:56:04.237 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.237 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.237 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.237 DEBUG:boto:Params: {} 2024-04-16T19:56:04.237 DEBUG:boto:Token: None 2024-04-16T19:56:04.237 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1f714f341282128-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.240 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:04.240 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:04.240 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.240 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:04.240 DEBUG:boto:Data: 2024-04-16T19:56:04.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.240 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.240 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.241 DEBUG:boto:Params: {} 2024-04-16T19:56:04.241 DEBUG:boto:Token: None 2024-04-16T19:56:04.241 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:04.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LNNelBRi3KS3atnSJDBWhZeA+ew= 2024-04-16T19:56:04.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LNNelBRi3KS3atnSJDBWhZeA+ew='} 2024-04-16T19:56:04.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001644f5fdb9d06dea-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.283 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.283 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.283 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.283 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.283 DEBUG:boto:Data: 2024-04-16T19:56:04.283 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.283 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.283 DEBUG:boto:Params: {} 2024-04-16T19:56:04.283 DEBUG:boto:Token: None 2024-04-16T19:56:04.284 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004baf9a513cbb62a4-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.286 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:04.286 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:04.286 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.286 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:04.287 DEBUG:boto:Data: 2024-04-16T19:56:04.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.287 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.287 DEBUG:boto:Params: {} 2024-04-16T19:56:04.287 DEBUG:boto:Token: None 2024-04-16T19:56:04.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:04.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h9eSYiQoLQ+6VeJBhYAfey/qwYk= 2024-04-16T19:56:04.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h9eSYiQoLQ+6VeJBhYAfey/qwYk='} 2024-04-16T19:56:04.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001c12ef25cddab66-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.292 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.293 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.293 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.293 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.293 DEBUG:boto:Data: 2024-04-16T19:56:04.293 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.293 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.293 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.293 DEBUG:boto:Params: {} 2024-04-16T19:56:04.293 DEBUG:boto:Token: None 2024-04-16T19:56:04.293 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220896f93bbdb52e-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.296 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:04.296 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:04.296 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.296 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:04.296 DEBUG:boto:Data: 2024-04-16T19:56:04.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.296 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.296 DEBUG:boto:Params: {} 2024-04-16T19:56:04.297 DEBUG:boto:Token: None 2024-04-16T19:56:04.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:04.297 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jtg0HPb7wiI8I+lpdlzkPoAwJbQ= 2024-04-16T19:56:04.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jtg0HPb7wiI8I+lpdlzkPoAwJbQ='} 2024-04-16T19:56:04.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bef9939ceac6e13-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.339 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.339 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.339 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.339 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.339 DEBUG:boto:Data: 2024-04-16T19:56:04.339 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.339 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.339 DEBUG:boto:Params: {} 2024-04-16T19:56:04.339 DEBUG:boto:Token: None 2024-04-16T19:56:04.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba91126f2a612f4a-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.342 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:04.342 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:04.342 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.342 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:04.342 DEBUG:boto:Data: 2024-04-16T19:56:04.342 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.343 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.343 DEBUG:boto:Params: {} 2024-04-16T19:56:04.343 DEBUG:boto:Token: None 2024-04-16T19:56:04.343 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:04.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwhrSsq5Wh39MhRPfL+qOuhFfUI= 2024-04-16T19:56:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwhrSsq5Wh39MhRPfL+qOuhFfUI='} 2024-04-16T19:56:04.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000787667506919d293-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.348 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.348 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.348 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.349 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.349 DEBUG:boto:Data: 2024-04-16T19:56:04.349 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.349 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.349 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.349 DEBUG:boto:Params: {} 2024-04-16T19:56:04.349 DEBUG:boto:Token: None 2024-04-16T19:56:04.349 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.350 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000998c453ba9fc2063-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.352 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:04.352 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:04.352 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.352 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:04.353 DEBUG:boto:Data: 2024-04-16T19:56:04.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.353 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.353 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.353 DEBUG:boto:Params: {} 2024-04-16T19:56:04.353 DEBUG:boto:Token: None 2024-04-16T19:56:04.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:04.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aTAn2v+x4H8XCTm9E/QZnipZlgk= 2024-04-16T19:56:04.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aTAn2v+x4H8XCTm9E/QZnipZlgk='} 2024-04-16T19:56:04.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048e9642c81bb0bbf-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.402 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.403 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.403 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.403 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.403 DEBUG:boto:Data: 2024-04-16T19:56:04.403 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.403 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.403 DEBUG:boto:Params: {} 2024-04-16T19:56:04.403 DEBUG:boto:Token: None 2024-04-16T19:56:04.403 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000178cd2d6de9950f4-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.406 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:04.406 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:04.406 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.406 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:04.407 DEBUG:boto:Data: 2024-04-16T19:56:04.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.407 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.407 DEBUG:boto:Params: {} 2024-04-16T19:56:04.407 DEBUG:boto:Token: None 2024-04-16T19:56:04.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:04.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MdIRjVSLFrJs5hfZlaLkAIs5AzM= 2024-04-16T19:56:04.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MdIRjVSLFrJs5hfZlaLkAIs5AzM='} 2024-04-16T19:56:04.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a93d114b2b839ec-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.413 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.413 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.413 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.413 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.413 DEBUG:boto:Data: 2024-04-16T19:56:04.413 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.413 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.413 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.413 DEBUG:boto:Params: {} 2024-04-16T19:56:04.413 DEBUG:boto:Token: None 2024-04-16T19:56:04.414 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.414 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005386c3571aeb586b-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.416 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:04.416 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:04.416 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.416 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:04.417 DEBUG:boto:Data: 2024-04-16T19:56:04.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.417 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.417 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.417 DEBUG:boto:Params: {} 2024-04-16T19:56:04.417 DEBUG:boto:Token: None 2024-04-16T19:56:04.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:04.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Huwmjs2bA+e3RhHY4geBWsv259c= 2024-04-16T19:56:04.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Huwmjs2bA+e3RhHY4geBWsv259c='} 2024-04-16T19:56:04.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a39ff694a76df973-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.422 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.422 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.422 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.423 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.423 DEBUG:boto:Data: 2024-04-16T19:56:04.423 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.423 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.423 DEBUG:boto:Params: {} 2024-04-16T19:56:04.423 DEBUG:boto:Token: None 2024-04-16T19:56:04.423 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc2adce99394624-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.426 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:04.426 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:04.426 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.426 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:04.426 DEBUG:boto:Data: 2024-04-16T19:56:04.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.426 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.426 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.426 DEBUG:boto:Params: {} 2024-04-16T19:56:04.427 DEBUG:boto:Token: None 2024-04-16T19:56:04.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:04.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qk7JHudN+NXKWidOxA+7bZyLYiQ= 2024-04-16T19:56:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qk7JHudN+NXKWidOxA+7bZyLYiQ='} 2024-04-16T19:56:04.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0e890e12560cd48-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.471 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.471 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.471 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.471 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.471 DEBUG:boto:Data: 2024-04-16T19:56:04.471 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.471 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.471 DEBUG:boto:Params: {} 2024-04-16T19:56:04.471 DEBUG:boto:Token: None 2024-04-16T19:56:04.472 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc5c024e87caab2e-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.474 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:04.474 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:04.474 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.474 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:04.475 DEBUG:boto:Data: 2024-04-16T19:56:04.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.475 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.475 DEBUG:boto:Params: {} 2024-04-16T19:56:04.475 DEBUG:boto:Token: None 2024-04-16T19:56:04.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:04.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K8oqQxuGZtDl0JfIwB821z1NEQA= 2024-04-16T19:56:04.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K8oqQxuGZtDl0JfIwB821z1NEQA='} 2024-04-16T19:56:04.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a2b35d0ad08e480-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.480 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.480 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.480 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.481 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.481 DEBUG:boto:Data: 2024-04-16T19:56:04.481 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.481 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.481 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.481 DEBUG:boto:Params: {} 2024-04-16T19:56:04.481 DEBUG:boto:Token: None 2024-04-16T19:56:04.481 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e198f5887bebdb9-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.484 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.484 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.484 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.484 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.484 DEBUG:boto:Data: 2024-04-16T19:56:04.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.484 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.484 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.484 DEBUG:boto:Params: {} 2024-04-16T19:56:04.485 DEBUG:boto:Token: None 2024-04-16T19:56:04.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oli9CLG7uiUa0QW69rJ9yJx6D84= 2024-04-16T19:56:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oli9CLG7uiUa0QW69rJ9yJx6D84='} 2024-04-16T19:56:04.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T19:56:04.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5cea87c2df6df1b-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.526 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.527 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.527 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.527 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.527 DEBUG:boto:Data: 2024-04-16T19:56:04.527 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.527 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.527 DEBUG:boto:Params: {} 2024-04-16T19:56:04.527 DEBUG:boto:Token: None 2024-04-16T19:56:04.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4fc362fa0a5632f-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.530 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:04.530 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:04.530 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.530 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:04.530 DEBUG:boto:Data: 2024-04-16T19:56:04.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.530 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.530 DEBUG:boto:Params: {} 2024-04-16T19:56:04.531 DEBUG:boto:Token: None 2024-04-16T19:56:04.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:04.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:igiUDJDh0QzoUOBowicOpQ0yqMU= 2024-04-16T19:56:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:igiUDJDh0QzoUOBowicOpQ0yqMU='} 2024-04-16T19:56:04.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a14148fb053abf7-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.536 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.536 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.536 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.536 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.536 DEBUG:boto:Data: 2024-04-16T19:56:04.536 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.536 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.536 DEBUG:boto:Params: {} 2024-04-16T19:56:04.536 DEBUG:boto:Token: None 2024-04-16T19:56:04.537 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b40f17e2459894db-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:04.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:04.539 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.539 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:04.539 DEBUG:boto:Data: 2024-04-16T19:56:04.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.539 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.539 DEBUG:boto:Params: {} 2024-04-16T19:56:04.539 DEBUG:boto:Token: None 2024-04-16T19:56:04.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:04.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yfgWKMHpz7zerNJWd0/XervXRVE= 2024-04-16T19:56:04.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yfgWKMHpz7zerNJWd0/XervXRVE='} 2024-04-16T19:56:04.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a08dca49f782c79e-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.583 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.583 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.583 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.583 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.583 DEBUG:boto:Data: 2024-04-16T19:56:04.583 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.583 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.583 DEBUG:boto:Params: {} 2024-04-16T19:56:04.583 DEBUG:boto:Token: None 2024-04-16T19:56:04.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c33e64a3dd8c1a84-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.586 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:04.586 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:04.587 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.587 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:04.587 DEBUG:boto:Data: 2024-04-16T19:56:04.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.587 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.587 DEBUG:boto:Params: {} 2024-04-16T19:56:04.587 DEBUG:boto:Token: None 2024-04-16T19:56:04.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:04.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hEJ9yMngOYVeo+mvjl4sc4T2Vs4= 2024-04-16T19:56:04.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hEJ9yMngOYVeo+mvjl4sc4T2Vs4='} 2024-04-16T19:56:04.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626695d8f056c741-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.594 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.594 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.594 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.595 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.595 DEBUG:boto:Data: 2024-04-16T19:56:04.595 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.595 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.595 DEBUG:boto:Params: {} 2024-04-16T19:56:04.595 DEBUG:boto:Token: None 2024-04-16T19:56:04.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073313f53ffa608ae-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.598 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:04.598 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:04.598 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.598 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:04.598 DEBUG:boto:Data: 2024-04-16T19:56:04.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.598 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.598 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.598 DEBUG:boto:Params: {} 2024-04-16T19:56:04.599 DEBUG:boto:Token: None 2024-04-16T19:56:04.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:04.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:25wq3g1Yv+cM++0u7klVvyRGq/k= 2024-04-16T19:56:04.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:25wq3g1Yv+cM++0u7klVvyRGq/k='} 2024-04-16T19:56:04.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb1a460b6773e51d-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:04.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:04.604 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:04.714 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:04.715 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:04.722 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 22 2024-04-16T19:56:04.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:04.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:04.779 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:04.828 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:04.828 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:04.896 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000005","obj00000000000000000010","obj00000000000000000016","obj00000000000000000021","obj00000000000000000004","obj00000000000000000025","obj00000000000000000030","obj00000000000000000015","obj00000000000000000019","obj00000000000000000003","obj00000000000000000009","obj00000000000000000014","obj00000000000000000012","obj00000000000000000002","obj00000000000000000000","obj00000000000000000006","obj00000000000000000011","obj00000000000000000001","obj00000000000000000007"]} 2024-04-16T19:56:04.897 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.897 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.897 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.897 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.897 DEBUG:boto:Data: 2024-04-16T19:56:04.897 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.897 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.897 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.898 DEBUG:boto:Params: {} 2024-04-16T19:56:04.898 DEBUG:boto:Token: None 2024-04-16T19:56:04.898 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.898 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fbc5d0cf011e524-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.901 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.901 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.901 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.901 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.901 DEBUG:boto:Data: 2024-04-16T19:56:04.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.901 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.901 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.901 DEBUG:boto:Params: {} 2024-04-16T19:56:04.902 DEBUG:boto:Token: None 2024-04-16T19:56:04.902 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:04.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oli9CLG7uiUa0QW69rJ9yJx6D84= 2024-04-16T19:56:04.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oli9CLG7uiUa0QW69rJ9yJx6D84='} 2024-04-16T19:56:04.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5125b923c2e3d5d-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.909 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.909 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.909 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.909 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.909 DEBUG:boto:Data: 2024-04-16T19:56:04.909 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.909 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.909 DEBUG:boto:Params: {} 2024-04-16T19:56:04.909 DEBUG:boto:Token: None 2024-04-16T19:56:04.909 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000522fd4f23bcd007f-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.912 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:04.912 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:04.912 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.912 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:04.912 DEBUG:boto:Data: 2024-04-16T19:56:04.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.912 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.912 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.912 DEBUG:boto:Params: {} 2024-04-16T19:56:04.913 DEBUG:boto:Token: None 2024-04-16T19:56:04.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:04.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yJYT5IrUhkNpx2tR46wPuuT7SSk= 2024-04-16T19:56:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yJYT5IrUhkNpx2tR46wPuuT7SSk='} 2024-04-16T19:56:04.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016711e80df4f2cf5-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.955 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.955 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.955 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.955 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.955 DEBUG:boto:Data: 2024-04-16T19:56:04.955 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.955 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.955 DEBUG:boto:Params: {} 2024-04-16T19:56:04.955 DEBUG:boto:Token: None 2024-04-16T19:56:04.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003867fa7f833e6482-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.958 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:04.958 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:04.958 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.959 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:04.959 DEBUG:boto:Data: 2024-04-16T19:56:04.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.959 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.959 DEBUG:boto:Params: {} 2024-04-16T19:56:04.959 DEBUG:boto:Token: None 2024-04-16T19:56:04.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:04.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+/kG8TvT0i6gX1wGYtZsQR2Fp+A= 2024-04-16T19:56:04.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+/kG8TvT0i6gX1wGYtZsQR2Fp+A='} 2024-04-16T19:56:04.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e0ff959bd373071-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.966 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.966 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.966 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.966 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.966 DEBUG:boto:Data: 2024-04-16T19:56:04.966 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.966 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.966 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.967 DEBUG:boto:Params: {} 2024-04-16T19:56:04.967 DEBUG:boto:Token: None 2024-04-16T19:56:04.967 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db75df084137438e-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.969 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.969 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.969 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.969 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.969 DEBUG:boto:Data: 2024-04-16T19:56:04.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.970 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.970 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.970 DEBUG:boto:Params: {} 2024-04-16T19:56:04.970 DEBUG:boto:Token: None 2024-04-16T19:56:04.970 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:04.970 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bYPzS+dtgI+35USqkdO+0sO+GQ4= 2024-04-16T19:56:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bYPzS+dtgI+35USqkdO+0sO+GQ4='} 2024-04-16T19:56:04.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7335e563e22519c-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.977 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:04.977 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:04.977 DEBUG:boto:Method: HEAD 2024-04-16T19:56:04.977 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:04.977 DEBUG:boto:Data: 2024-04-16T19:56:04.977 DEBUG:boto:Headers: {} 2024-04-16T19:56:04.977 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.977 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.978 DEBUG:boto:Params: {} 2024-04-16T19:56:04.978 DEBUG:boto:Token: None 2024-04-16T19:56:04.978 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/ 2024-04-16T19:56:04.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4= 2024-04-16T19:56:04.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZn6BbTvZDiZ3573LPFT6Aykoo4=', 'Content-Length': '0'} 2024-04-16T19:56:04.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000305af0a4836e71f2-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:04.980 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:04.980 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:04.981 DEBUG:boto:Method: PUT 2024-04-16T19:56:04.981 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:04.981 DEBUG:boto:Data: 2024-04-16T19:56:04.981 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:04.981 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:04.981 DEBUG:boto:Port: 8000 2024-04-16T19:56:04.981 DEBUG:boto:Params: {} 2024-04-16T19:56:04.981 DEBUG:boto:Token: None 2024-04-16T19:56:04.981 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:04 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:04.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b+cAVRIjAPpQmDy/y88hWywm8+A= 2024-04-16T19:56:04.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b+cAVRIjAPpQmDy/y88hWywm8+A='} 2024-04-16T19:56:05.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0137d7f5ede4dbf-00661ed7d4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.027 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.027 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.027 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.027 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.027 DEBUG:boto:Data: 2024-04-16T19:56:05.027 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.027 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.027 DEBUG:boto:Params: {} 2024-04-16T19:56:05.027 DEBUG:boto:Token: None 2024-04-16T19:56:05.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006105b671f8068a4-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.030 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:05.030 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:05.030 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.030 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:05.030 DEBUG:boto:Data: 2024-04-16T19:56:05.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.031 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.031 DEBUG:boto:Params: {} 2024-04-16T19:56:05.031 DEBUG:boto:Token: None 2024-04-16T19:56:05.031 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:05.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9/YqC3LDCTRyv8DUPAu0laGV1dc= 2024-04-16T19:56:05.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9/YqC3LDCTRyv8DUPAu0laGV1dc='} 2024-04-16T19:56:05.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b33e9504d66b4db-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.038 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.038 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.038 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.038 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.038 DEBUG:boto:Data: 2024-04-16T19:56:05.038 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.038 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.038 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.038 DEBUG:boto:Params: {} 2024-04-16T19:56:05.039 DEBUG:boto:Token: None 2024-04-16T19:56:05.039 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010a11cf8c0661daf-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.041 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.041 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.041 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.041 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.042 DEBUG:boto:Data: 2024-04-16T19:56:05.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.042 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.042 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.042 DEBUG:boto:Params: {} 2024-04-16T19:56:05.042 DEBUG:boto:Token: None 2024-04-16T19:56:05.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ta8D+qYvlts7Mnu5L16EsTfs88Y= 2024-04-16T19:56:05.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ta8D+qYvlts7Mnu5L16EsTfs88Y='} 2024-04-16T19:56:05.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000dd1287e062f433-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.052 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.052 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.052 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.052 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.052 DEBUG:boto:Data: 2024-04-16T19:56:05.052 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.053 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.053 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.053 DEBUG:boto:Params: {} 2024-04-16T19:56:05.053 DEBUG:boto:Token: None 2024-04-16T19:56:05.053 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.053 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c8d42e1ca0fbcbc-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.055 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:05.056 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:05.056 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.056 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:05.056 DEBUG:boto:Data: 2024-04-16T19:56:05.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.056 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.056 DEBUG:boto:Params: {} 2024-04-16T19:56:05.056 DEBUG:boto:Token: None 2024-04-16T19:56:05.056 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:05.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QKu4otDrsBBBnB9l+9gyIv1yzvw= 2024-04-16T19:56:05.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QKu4otDrsBBBnB9l+9gyIv1yzvw='} 2024-04-16T19:56:05.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001eeb053a339cd3c6-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.099 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.099 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.099 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.099 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.099 DEBUG:boto:Data: 2024-04-16T19:56:05.099 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.099 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.099 DEBUG:boto:Params: {} 2024-04-16T19:56:05.099 DEBUG:boto:Token: None 2024-04-16T19:56:05.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fff525e66f593e1-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.102 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:05.102 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:05.102 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.102 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:05.102 DEBUG:boto:Data: 2024-04-16T19:56:05.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.102 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.103 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.103 DEBUG:boto:Params: {} 2024-04-16T19:56:05.103 DEBUG:boto:Token: None 2024-04-16T19:56:05.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:05.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YUrfhSy91Q8dThZxuj4pkP5/zOk= 2024-04-16T19:56:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YUrfhSy91Q8dThZxuj4pkP5/zOk='} 2024-04-16T19:56:05.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbebdfc745b5383d-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.110 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.110 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.110 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.110 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.110 DEBUG:boto:Data: 2024-04-16T19:56:05.110 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.110 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.111 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.111 DEBUG:boto:Params: {} 2024-04-16T19:56:05.111 DEBUG:boto:Token: None 2024-04-16T19:56:05.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079a9dc5996dd4368-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.113 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:05.113 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:05.114 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.114 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:05.114 DEBUG:boto:Data: 2024-04-16T19:56:05.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.114 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.114 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.114 DEBUG:boto:Params: {} 2024-04-16T19:56:05.114 DEBUG:boto:Token: None 2024-04-16T19:56:05.114 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:05.114 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6MhBqF/UcaePz+B4iz3ZoaSH01s= 2024-04-16T19:56:05.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6MhBqF/UcaePz+B4iz3ZoaSH01s='} 2024-04-16T19:56:05.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088c9ac593d7fb493-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.121 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.121 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.121 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.121 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.122 DEBUG:boto:Data: 2024-04-16T19:56:05.122 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.122 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.122 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.122 DEBUG:boto:Params: {} 2024-04-16T19:56:05.122 DEBUG:boto:Token: None 2024-04-16T19:56:05.122 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.122 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055e0ed326c66c6c8-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.125 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:05.125 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:05.125 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.125 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:05.125 DEBUG:boto:Data: 2024-04-16T19:56:05.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.125 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.125 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.126 DEBUG:boto:Params: {} 2024-04-16T19:56:05.126 DEBUG:boto:Token: None 2024-04-16T19:56:05.126 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:05.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A2WHX1ivDdygv6dTEu3GRG85NBc= 2024-04-16T19:56:05.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A2WHX1ivDdygv6dTEu3GRG85NBc='} 2024-04-16T19:56:05.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000588435a33dab8d27-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.175 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.175 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.175 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.175 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.175 DEBUG:boto:Data: 2024-04-16T19:56:05.175 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.175 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.175 DEBUG:boto:Params: {} 2024-04-16T19:56:05.175 DEBUG:boto:Token: None 2024-04-16T19:56:05.175 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f68bf562e31faf7-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.178 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.179 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.179 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.179 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.179 DEBUG:boto:Data: 2024-04-16T19:56:05.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.179 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.179 DEBUG:boto:Params: {} 2024-04-16T19:56:05.179 DEBUG:boto:Token: None 2024-04-16T19:56:05.179 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOOrZn5ELbE/esZYare40ltcSV4= 2024-04-16T19:56:05.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOOrZn5ELbE/esZYare40ltcSV4='} 2024-04-16T19:56:05.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008608431878bb439b-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.185 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.185 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.185 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.185 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.185 DEBUG:boto:Data: 2024-04-16T19:56:05.185 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.185 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.185 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.185 DEBUG:boto:Params: {} 2024-04-16T19:56:05.186 DEBUG:boto:Token: None 2024-04-16T19:56:05.186 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.186 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084543cae5b68e792-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.188 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:05.188 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:05.189 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.189 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:05.189 DEBUG:boto:Data: 2024-04-16T19:56:05.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.189 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.189 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.189 DEBUG:boto:Params: {} 2024-04-16T19:56:05.189 DEBUG:boto:Token: None 2024-04-16T19:56:05.189 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:05.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5F+oE/h4yBBQzfQroTx+w4x3OQE= 2024-04-16T19:56:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5F+oE/h4yBBQzfQroTx+w4x3OQE='} 2024-04-16T19:56:05.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a15be5bd2b22d255-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.196 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.196 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.196 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.196 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.197 DEBUG:boto:Data: 2024-04-16T19:56:05.197 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.197 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.197 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.197 DEBUG:boto:Params: {} 2024-04-16T19:56:05.197 DEBUG:boto:Token: None 2024-04-16T19:56:05.197 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049b8b97fd6532f91-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.200 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:05.200 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:05.200 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.200 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:05.200 DEBUG:boto:Data: 2024-04-16T19:56:05.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.200 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.200 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.200 DEBUG:boto:Params: {} 2024-04-16T19:56:05.201 DEBUG:boto:Token: None 2024-04-16T19:56:05.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:05.201 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:23QaPRM+zHKo58imONcw2ckKuOw= 2024-04-16T19:56:05.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:23QaPRM+zHKo58imONcw2ckKuOw='} 2024-04-16T19:56:05.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cdbc14f0b720f8e-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.243 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.243 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.243 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.243 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.243 DEBUG:boto:Data: 2024-04-16T19:56:05.243 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.243 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.243 DEBUG:boto:Params: {} 2024-04-16T19:56:05.243 DEBUG:boto:Token: None 2024-04-16T19:56:05.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031fcc3a24b2d6b0b-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.246 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:05.249 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:05.249 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.249 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:05.249 DEBUG:boto:Data: 2024-04-16T19:56:05.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.250 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.250 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.250 DEBUG:boto:Params: {} 2024-04-16T19:56:05.250 DEBUG:boto:Token: None 2024-04-16T19:56:05.250 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:05.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0sWVRT42xy8VpE5B4noKqSeGbmk= 2024-04-16T19:56:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0sWVRT42xy8VpE5B4noKqSeGbmk='} 2024-04-16T19:56:05.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7b07f05ea079357-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.257 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.257 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.257 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.257 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.257 DEBUG:boto:Data: 2024-04-16T19:56:05.257 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.257 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.257 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.257 DEBUG:boto:Params: {} 2024-04-16T19:56:05.258 DEBUG:boto:Token: None 2024-04-16T19:56:05.258 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.258 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000682294674bf14dd4-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.260 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:05.260 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:05.260 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.260 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:05.261 DEBUG:boto:Data: 2024-04-16T19:56:05.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.261 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.261 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.261 DEBUG:boto:Params: {} 2024-04-16T19:56:05.261 DEBUG:boto:Token: None 2024-04-16T19:56:05.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:05.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKMhuIirV0bgdhoHuLeef/w6GBw= 2024-04-16T19:56:05.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKMhuIirV0bgdhoHuLeef/w6GBw='} 2024-04-16T19:56:05.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdb3509418141833-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.266 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.266 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.266 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.266 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.266 DEBUG:boto:Data: 2024-04-16T19:56:05.266 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.266 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.266 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.267 DEBUG:boto:Params: {} 2024-04-16T19:56:05.267 DEBUG:boto:Token: None 2024-04-16T19:56:05.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078afc9da571cecb2-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.269 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:05.269 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:05.269 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.269 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:05.270 DEBUG:boto:Data: 2024-04-16T19:56:05.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.270 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.270 DEBUG:boto:Params: {} 2024-04-16T19:56:05.270 DEBUG:boto:Token: None 2024-04-16T19:56:05.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:05.270 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdckTmlkKa4Cm+qJsxeYsHM36g0= 2024-04-16T19:56:05.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdckTmlkKa4Cm+qJsxeYsHM36g0='} 2024-04-16T19:56:05.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e411f3269ca0b8b4-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.315 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.315 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.315 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.315 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.315 DEBUG:boto:Data: 2024-04-16T19:56:05.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.315 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.315 DEBUG:boto:Params: {} 2024-04-16T19:56:05.315 DEBUG:boto:Token: None 2024-04-16T19:56:05.315 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e211d1c87795679-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.318 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:05.318 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:05.318 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.318 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:05.318 DEBUG:boto:Data: 2024-04-16T19:56:05.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.319 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.319 DEBUG:boto:Params: {} 2024-04-16T19:56:05.319 DEBUG:boto:Token: None 2024-04-16T19:56:05.319 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:05.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kkY+THS4TCFG9TYl7TNGwUEC/+A= 2024-04-16T19:56:05.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kkY+THS4TCFG9TYl7TNGwUEC/+A='} 2024-04-16T19:56:05.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076b3eb4be16ba9f8-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:05.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:05.326 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:05.438 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:05.438 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:05.446 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 26 2024-04-16T19:56:05.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:05.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:05.507 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:05.558 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:05.558 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:05.631 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000058","obj00000000000000000026","obj00000000000000000011","obj00000000000000000046","obj00000000000000000014","obj00000000000000000050","obj00000000000000000032","obj00000000000000000002","obj00000000000000000039","obj00000000000000000007","obj00000000000000000006","obj00000000000000000040","obj00000000000000000009","obj00000000000000000045","obj00000000000000000013","obj00000000000000000012","obj00000000000000000033","obj00000000000000000001","obj00000000000000000000","obj00000000000000000021","obj00000000000000000003","obj00000000000000000055","obj00000000000000000008"]} 2024-04-16T19:56:05.631 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.631 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.631 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.631 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.632 DEBUG:boto:Data: 2024-04-16T19:56:05.632 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.632 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.632 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.632 DEBUG:boto:Params: {} 2024-04-16T19:56:05.632 DEBUG:boto:Token: None 2024-04-16T19:56:05.632 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5aea3db2257d9f9-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.636 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.636 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.636 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.636 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.636 DEBUG:boto:Data: 2024-04-16T19:56:05.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.636 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.636 DEBUG:boto:Params: {} 2024-04-16T19:56:05.636 DEBUG:boto:Token: None 2024-04-16T19:56:05.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:05.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ta8D+qYvlts7Mnu5L16EsTfs88Y= 2024-04-16T19:56:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ta8D+qYvlts7Mnu5L16EsTfs88Y='} 2024-04-16T19:56:05.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc75ff849be05e12-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.644 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.644 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.644 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.644 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.644 DEBUG:boto:Data: 2024-04-16T19:56:05.644 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.645 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.645 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.645 DEBUG:boto:Params: {} 2024-04-16T19:56:05.645 DEBUG:boto:Token: None 2024-04-16T19:56:05.645 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.645 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007316794bc492b243-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.648 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:05.648 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:05.648 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.648 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:05.648 DEBUG:boto:Data: 2024-04-16T19:56:05.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.648 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.649 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.649 DEBUG:boto:Params: {} 2024-04-16T19:56:05.649 DEBUG:boto:Token: None 2024-04-16T19:56:05.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:05.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WJQYDM6jkmZxXqRtqjjF3ynKuW4= 2024-04-16T19:56:05.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WJQYDM6jkmZxXqRtqjjF3ynKuW4='} 2024-04-16T19:56:05.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a925882399863f73-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.695 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.695 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.695 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.695 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.695 DEBUG:boto:Data: 2024-04-16T19:56:05.695 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.695 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.696 DEBUG:boto:Params: {} 2024-04-16T19:56:05.696 DEBUG:boto:Token: None 2024-04-16T19:56:05.696 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6a6ca8005d8c121-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.699 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:05.699 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:05.699 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.699 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:05.699 DEBUG:boto:Data: 2024-04-16T19:56:05.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.700 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.700 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.700 DEBUG:boto:Params: {} 2024-04-16T19:56:05.700 DEBUG:boto:Token: None 2024-04-16T19:56:05.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:05.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iVpKJIUNW73hhY09Y/5/Jbhuib0= 2024-04-16T19:56:05.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iVpKJIUNW73hhY09Y/5/Jbhuib0='} 2024-04-16T19:56:05.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003af854039ec714f1-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.707 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.707 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.707 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.707 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.707 DEBUG:boto:Data: 2024-04-16T19:56:05.708 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.708 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.708 DEBUG:boto:Params: {} 2024-04-16T19:56:05.708 DEBUG:boto:Token: None 2024-04-16T19:56:05.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a220efcfb8635c6-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.711 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:05.711 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:05.711 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.711 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:05.711 DEBUG:boto:Data: 2024-04-16T19:56:05.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.711 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.711 DEBUG:boto:Params: {} 2024-04-16T19:56:05.712 DEBUG:boto:Token: None 2024-04-16T19:56:05.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:05.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cgRVym6oCxK+LgdZQq8aBi+6kcg= 2024-04-16T19:56:05.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cgRVym6oCxK+LgdZQq8aBi+6kcg='} 2024-04-16T19:56:05.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a4c0fc1e60f2d7a-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.755 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.755 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.755 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.755 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.755 DEBUG:boto:Data: 2024-04-16T19:56:05.755 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.755 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.755 DEBUG:boto:Params: {} 2024-04-16T19:56:05.755 DEBUG:boto:Token: None 2024-04-16T19:56:05.756 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b671d81cfe98257-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.758 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:05.758 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:05.758 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.759 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:05.759 DEBUG:boto:Data: 2024-04-16T19:56:05.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.759 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.759 DEBUG:boto:Params: {} 2024-04-16T19:56:05.759 DEBUG:boto:Token: None 2024-04-16T19:56:05.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:05.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hHIt0RLV3nVcLWMp1NYfTiI+DFA= 2024-04-16T19:56:05.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hHIt0RLV3nVcLWMp1NYfTiI+DFA='} 2024-04-16T19:56:05.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fe72cfda4da74cd-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.765 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.765 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.765 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.765 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.766 DEBUG:boto:Data: 2024-04-16T19:56:05.766 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.766 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.766 DEBUG:boto:Params: {} 2024-04-16T19:56:05.766 DEBUG:boto:Token: None 2024-04-16T19:56:05.766 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029eacc94c48fceb7-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.769 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:05.769 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:05.769 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.769 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:05.769 DEBUG:boto:Data: 2024-04-16T19:56:05.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.770 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.770 DEBUG:boto:Params: {} 2024-04-16T19:56:05.770 DEBUG:boto:Token: None 2024-04-16T19:56:05.770 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:05.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pnuZ1DcfUqMTU2Uhdp3Nr1TLWuI= 2024-04-16T19:56:05.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pnuZ1DcfUqMTU2Uhdp3Nr1TLWuI='} 2024-04-16T19:56:05.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017fc7b5ba21c16d9-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.776 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.776 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.776 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.776 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.776 DEBUG:boto:Data: 2024-04-16T19:56:05.777 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.777 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.777 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.777 DEBUG:boto:Params: {} 2024-04-16T19:56:05.777 DEBUG:boto:Token: None 2024-04-16T19:56:05.777 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ddeb95fbb3e2673-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.780 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:05.780 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:05.780 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.780 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:05.780 DEBUG:boto:Data: 2024-04-16T19:56:05.780 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.781 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.781 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.781 DEBUG:boto:Params: {} 2024-04-16T19:56:05.781 DEBUG:boto:Token: None 2024-04-16T19:56:05.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:05.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uw9nzQimSVJ6B8uZAnDq6gKoXF4= 2024-04-16T19:56:05.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uw9nzQimSVJ6B8uZAnDq6gKoXF4='} 2024-04-16T19:56:05.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cd442d6138f8523-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.852 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.852 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.852 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.852 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.852 DEBUG:boto:Data: 2024-04-16T19:56:05.852 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.852 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.852 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.853 DEBUG:boto:Params: {} 2024-04-16T19:56:05.853 DEBUG:boto:Token: None 2024-04-16T19:56:05.853 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d17cad3d5abdf3e-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.856 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:05.856 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:05.856 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.856 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:05.856 DEBUG:boto:Data: 2024-04-16T19:56:05.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.857 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.857 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.857 DEBUG:boto:Params: {} 2024-04-16T19:56:05.857 DEBUG:boto:Token: None 2024-04-16T19:56:05.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:05.857 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W3+8avDRSrBTInWPk/Lecw6Xgek= 2024-04-16T19:56:05.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W3+8avDRSrBTInWPk/Lecw6Xgek='} 2024-04-16T19:56:05.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004158f3b5e86d9c96-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.864 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.864 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.864 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.864 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.864 DEBUG:boto:Data: 2024-04-16T19:56:05.864 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.864 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.864 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.864 DEBUG:boto:Params: {} 2024-04-16T19:56:05.864 DEBUG:boto:Token: None 2024-04-16T19:56:05.865 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.865 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036811854d0c542c6-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.867 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.868 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.868 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.868 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.868 DEBUG:boto:Data: 2024-04-16T19:56:05.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.868 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.868 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.868 DEBUG:boto:Params: {} 2024-04-16T19:56:05.868 DEBUG:boto:Token: None 2024-04-16T19:56:05.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:05.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOOrZn5ELbE/esZYare40ltcSV4= 2024-04-16T19:56:05.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOOrZn5ELbE/esZYare40ltcSV4='} 2024-04-16T19:56:05.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab7c85794a4e3d10-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.911 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.911 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.911 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.911 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.911 DEBUG:boto:Data: 2024-04-16T19:56:05.911 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.911 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.911 DEBUG:boto:Params: {} 2024-04-16T19:56:05.911 DEBUG:boto:Token: None 2024-04-16T19:56:05.911 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000021b6e1425425197-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.915 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:05.915 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:05.915 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.915 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:05.915 DEBUG:boto:Data: 2024-04-16T19:56:05.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.915 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.915 DEBUG:boto:Params: {} 2024-04-16T19:56:05.915 DEBUG:boto:Token: None 2024-04-16T19:56:05.915 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:05.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r2WZ+wF54qBzTGyUXM9zyuVFzcg= 2024-04-16T19:56:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r2WZ+wF54qBzTGyUXM9zyuVFzcg='} 2024-04-16T19:56:05.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c47e24bdd946f150-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.921 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.921 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.921 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.921 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.922 DEBUG:boto:Data: 2024-04-16T19:56:05.922 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.922 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.922 DEBUG:boto:Params: {} 2024-04-16T19:56:05.922 DEBUG:boto:Token: None 2024-04-16T19:56:05.922 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.922 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cfbfb6d327fc379-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.925 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:05.925 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:05.925 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.925 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:05.925 DEBUG:boto:Data: 2024-04-16T19:56:05.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.925 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.925 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.925 DEBUG:boto:Params: {} 2024-04-16T19:56:05.926 DEBUG:boto:Token: None 2024-04-16T19:56:05.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:05.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNUoVlsBhW8JluzMfLHQ475mNq4= 2024-04-16T19:56:05.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNUoVlsBhW8JluzMfLHQ475mNq4='} 2024-04-16T19:56:05.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b30ef820eb4f6165-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.975 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.975 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.975 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.975 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.975 DEBUG:boto:Data: 2024-04-16T19:56:05.975 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.975 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.975 DEBUG:boto:Params: {} 2024-04-16T19:56:05.975 DEBUG:boto:Token: None 2024-04-16T19:56:05.976 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0a34d609fd0885c-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.978 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:05.978 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:05.978 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.979 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:05.979 DEBUG:boto:Data: 2024-04-16T19:56:05.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.979 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.979 DEBUG:boto:Params: {} 2024-04-16T19:56:05.979 DEBUG:boto:Token: None 2024-04-16T19:56:05.979 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:05.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GLoA1VmeBEjt3asHFB3wXIDa86Y= 2024-04-16T19:56:05.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GLoA1VmeBEjt3asHFB3wXIDa86Y='} 2024-04-16T19:56:05.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000abff536d5f85393-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.986 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.986 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.986 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.986 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.986 DEBUG:boto:Data: 2024-04-16T19:56:05.986 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.987 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.987 DEBUG:boto:Params: {} 2024-04-16T19:56:05.987 DEBUG:boto:Token: None 2024-04-16T19:56:05.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:05.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d47145555f3c359d-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.989 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:05.990 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:05.990 DEBUG:boto:Method: PUT 2024-04-16T19:56:05.990 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:05.990 DEBUG:boto:Data: 2024-04-16T19:56:05.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:05.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.990 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.990 DEBUG:boto:Params: {} 2024-04-16T19:56:05.990 DEBUG:boto:Token: None 2024-04-16T19:56:05.990 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:05.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6WEKaCqTQocJk8U79fb9P2XPXrY= 2024-04-16T19:56:05.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6WEKaCqTQocJk8U79fb9P2XPXrY='} 2024-04-16T19:56:05.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009472c05ea19e5aa0-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:05.997 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:05.997 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:05.997 DEBUG:boto:Method: HEAD 2024-04-16T19:56:05.998 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:05.998 DEBUG:boto:Data: 2024-04-16T19:56:05.998 DEBUG:boto:Headers: {} 2024-04-16T19:56:05.998 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:05.998 DEBUG:boto:Port: 8000 2024-04-16T19:56:05.998 DEBUG:boto:Params: {} 2024-04-16T19:56:05.998 DEBUG:boto:Token: None 2024-04-16T19:56:05.998 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:05 GMT /yrjefh-51/ 2024-04-16T19:56:05.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E= 2024-04-16T19:56:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwJKi7AyFNnIOInEujh3YkAr/2E=', 'Content-Length': '0'} 2024-04-16T19:56:06.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006ff0778868b1b28-00661ed7d5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.001 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:06.001 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:06.001 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.001 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:06.001 DEBUG:boto:Data: 2024-04-16T19:56:06.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.001 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.001 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.001 DEBUG:boto:Params: {} 2024-04-16T19:56:06.001 DEBUG:boto:Token: None 2024-04-16T19:56:06.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:06.002 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JId7r+RfEnrPtj2hUwF6JcU+fS4= 2024-04-16T19:56:06.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JId7r+RfEnrPtj2hUwF6JcU+fS4='} 2024-04-16T19:56:06.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009836d7b2103cd720-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.046 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.047 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.047 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.047 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.047 DEBUG:boto:Data: 2024-04-16T19:56:06.047 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.047 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.047 DEBUG:boto:Params: {} 2024-04-16T19:56:06.047 DEBUG:boto:Token: None 2024-04-16T19:56:06.047 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ab40c4ada5858e8-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.050 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:06.050 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:06.050 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.050 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:06.050 DEBUG:boto:Data: 2024-04-16T19:56:06.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.050 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.051 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.051 DEBUG:boto:Params: {} 2024-04-16T19:56:06.051 DEBUG:boto:Token: None 2024-04-16T19:56:06.051 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:06.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7oigZbCs+CuYCrAEgRn0CEriHwI= 2024-04-16T19:56:06.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7oigZbCs+CuYCrAEgRn0CEriHwI='} 2024-04-16T19:56:06.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b48e20c2ebdc07c2-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.057 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.057 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.057 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.057 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.058 DEBUG:boto:Data: 2024-04-16T19:56:06.058 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.058 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.058 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.058 DEBUG:boto:Params: {} 2024-04-16T19:56:06.058 DEBUG:boto:Token: None 2024-04-16T19:56:06.058 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d094b7c0652a20d-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.061 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.061 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.061 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.061 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.061 DEBUG:boto:Data: 2024-04-16T19:56:06.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.061 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.061 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.062 DEBUG:boto:Params: {} 2024-04-16T19:56:06.062 DEBUG:boto:Token: None 2024-04-16T19:56:06.062 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.062 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Vl8Ld2b2SRUijYEX0RqEvk/eMy8= 2024-04-16T19:56:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Vl8Ld2b2SRUijYEX0RqEvk/eMy8='} 2024-04-16T19:56:06.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c64a6838e4da65b6-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.068 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.068 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.068 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.068 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.068 DEBUG:boto:Data: 2024-04-16T19:56:06.068 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.068 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.068 DEBUG:boto:Params: {} 2024-04-16T19:56:06.069 DEBUG:boto:Token: None 2024-04-16T19:56:06.069 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.069 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000902608e3218d1a04-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.071 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:06.072 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:06.072 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.072 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:06.072 DEBUG:boto:Data: 2024-04-16T19:56:06.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.072 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.072 DEBUG:boto:Params: {} 2024-04-16T19:56:06.072 DEBUG:boto:Token: None 2024-04-16T19:56:06.072 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:06.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YVHbPZICYNgUxqbyBjFJmTMobSA= 2024-04-16T19:56:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YVHbPZICYNgUxqbyBjFJmTMobSA='} 2024-04-16T19:56:06.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022bb94640728b7d4-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.114 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.115 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.115 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.115 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.115 DEBUG:boto:Data: 2024-04-16T19:56:06.115 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.115 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.115 DEBUG:boto:Params: {} 2024-04-16T19:56:06.115 DEBUG:boto:Token: None 2024-04-16T19:56:06.115 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acd39cdfbe747591-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.118 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.118 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.118 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.118 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.118 DEBUG:boto:Data: 2024-04-16T19:56:06.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.119 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.119 DEBUG:boto:Params: {} 2024-04-16T19:56:06.119 DEBUG:boto:Token: None 2024-04-16T19:56:06.119 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mG7uCH8ATbfO7qtZieQK7YL1nWE= 2024-04-16T19:56:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mG7uCH8ATbfO7qtZieQK7YL1nWE='} 2024-04-16T19:56:06.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086f6ca8a4cfd7fde-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.125 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.125 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.126 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.126 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.126 DEBUG:boto:Data: 2024-04-16T19:56:06.126 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.126 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.126 DEBUG:boto:Params: {} 2024-04-16T19:56:06.126 DEBUG:boto:Token: None 2024-04-16T19:56:06.126 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cf28770796c248c-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.129 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:06.129 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:06.129 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.129 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:06.130 DEBUG:boto:Data: 2024-04-16T19:56:06.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.130 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.130 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.130 DEBUG:boto:Params: {} 2024-04-16T19:56:06.130 DEBUG:boto:Token: None 2024-04-16T19:56:06.130 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:06.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aWzPtBzBkyz9c5Kdshts3cIfVYo= 2024-04-16T19:56:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aWzPtBzBkyz9c5Kdshts3cIfVYo='} 2024-04-16T19:56:06.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2c3834438b3ca73-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.136 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.136 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.136 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.136 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.136 DEBUG:boto:Data: 2024-04-16T19:56:06.136 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.136 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.137 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.137 DEBUG:boto:Params: {} 2024-04-16T19:56:06.137 DEBUG:boto:Token: None 2024-04-16T19:56:06.137 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.137 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069171dabb4df0a1c-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.140 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.140 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.140 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.140 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.140 DEBUG:boto:Data: 2024-04-16T19:56:06.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.140 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.140 DEBUG:boto:Params: {} 2024-04-16T19:56:06.140 DEBUG:boto:Token: None 2024-04-16T19:56:06.140 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lS3RBG4aYCaLcYRR0fESBfNfuBE= 2024-04-16T19:56:06.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lS3RBG4aYCaLcYRR0fESBfNfuBE='} 2024-04-16T19:56:06.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d0ef8aaaf94d8e4-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.183 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.183 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.183 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.183 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.183 DEBUG:boto:Data: 2024-04-16T19:56:06.183 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.183 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.183 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.183 DEBUG:boto:Params: {} 2024-04-16T19:56:06.183 DEBUG:boto:Token: None 2024-04-16T19:56:06.183 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be478f319ef8cad2-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.186 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:06.186 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:06.187 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.187 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:06.187 DEBUG:boto:Data: 2024-04-16T19:56:06.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.187 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.187 DEBUG:boto:Params: {} 2024-04-16T19:56:06.187 DEBUG:boto:Token: None 2024-04-16T19:56:06.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:06.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kxslWmDJRFQ4JqF8VwQqQ/QXZI= 2024-04-16T19:56:06.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kxslWmDJRFQ4JqF8VwQqQ/QXZI='} 2024-04-16T19:56:06.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b7a0b33c504f1fe-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:06.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:06.195 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:06.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:06.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:06.311 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:06.311 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 35 2024-04-16T19:56:06.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:06.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:06.388 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:06.437 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:06.438 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:06.506 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000006","obj00000000000000000059","obj00000000000000000027","obj00000000000000000004","obj00000000000000000011","obj00000000000000000002","obj00000000000000000019","obj00000000000000000049","obj00000000000000000017","obj00000000000000000008","obj00000000000000000015","obj00000000000000000031","obj00000000000000000022","obj00000000000000000013","obj00000000000000000020","obj00000000000000000037","obj00000000000000000005","obj00000000000000000012","obj00000000000000000003","obj00000000000000000042","obj00000000000000000010","obj00000000000000000001","obj00000000000000000009","obj00000000000000000025","obj00000000000000000016","obj00000000000000000000","obj00000000000000000014","obj00000000000000000030","obj00000000000000000021"]} 2024-04-16T19:56:06.506 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.507 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.507 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.507 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.507 DEBUG:boto:Data: 2024-04-16T19:56:06.507 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.507 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.507 DEBUG:boto:Params: {} 2024-04-16T19:56:06.507 DEBUG:boto:Token: None 2024-04-16T19:56:06.507 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016fdc7fb02b198c9-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.510 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:06.511 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:06.511 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.511 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:06.511 DEBUG:boto:Data: 2024-04-16T19:56:06.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.511 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.511 DEBUG:boto:Params: {} 2024-04-16T19:56:06.511 DEBUG:boto:Token: None 2024-04-16T19:56:06.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:06.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQ9deArAGgb8u7e1mNXwAnCz3L4= 2024-04-16T19:56:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQ9deArAGgb8u7e1mNXwAnCz3L4='} 2024-04-16T19:56:06.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8ab485dc78e73e8-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.518 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.518 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.518 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.518 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.519 DEBUG:boto:Data: 2024-04-16T19:56:06.519 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.519 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.519 DEBUG:boto:Params: {} 2024-04-16T19:56:06.519 DEBUG:boto:Token: None 2024-04-16T19:56:06.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001628000b5f963e33-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.522 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:06.522 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:06.522 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.522 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:06.522 DEBUG:boto:Data: 2024-04-16T19:56:06.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.523 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.523 DEBUG:boto:Params: {} 2024-04-16T19:56:06.523 DEBUG:boto:Token: None 2024-04-16T19:56:06.523 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:06.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4vztgBIS1vibYwYCYHbgr+KgiLU= 2024-04-16T19:56:06.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4vztgBIS1vibYwYCYHbgr+KgiLU='} 2024-04-16T19:56:06.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce237662bbd803e7-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.529 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.529 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.530 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.530 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.530 DEBUG:boto:Data: 2024-04-16T19:56:06.530 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.530 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.530 DEBUG:boto:Params: {} 2024-04-16T19:56:06.530 DEBUG:boto:Token: None 2024-04-16T19:56:06.530 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.530 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e46ae3adb96b5515-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.533 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:06.533 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:06.533 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.533 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:06.533 DEBUG:boto:Data: 2024-04-16T19:56:06.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.533 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.533 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.534 DEBUG:boto:Params: {} 2024-04-16T19:56:06.534 DEBUG:boto:Token: None 2024-04-16T19:56:06.534 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:06.534 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tFjq8krRZKsDNvYI5UHfPKyb4X4= 2024-04-16T19:56:06.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tFjq8krRZKsDNvYI5UHfPKyb4X4='} 2024-04-16T19:56:06.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000174ec17e09c7c1fa-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.579 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.579 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.579 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.579 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.579 DEBUG:boto:Data: 2024-04-16T19:56:06.579 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.579 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.579 DEBUG:boto:Params: {} 2024-04-16T19:56:06.579 DEBUG:boto:Token: None 2024-04-16T19:56:06.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a613ce9e247b659-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.582 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:06.582 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:06.582 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.583 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:06.583 DEBUG:boto:Data: 2024-04-16T19:56:06.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.583 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.583 DEBUG:boto:Params: {} 2024-04-16T19:56:06.583 DEBUG:boto:Token: None 2024-04-16T19:56:06.583 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:06.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LrOHEyxilM20j3znBpVrkJBjxdg= 2024-04-16T19:56:06.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LrOHEyxilM20j3znBpVrkJBjxdg='} 2024-04-16T19:56:06.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037f0385e184921d8-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.590 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.590 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.590 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.590 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.590 DEBUG:boto:Data: 2024-04-16T19:56:06.590 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.590 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.591 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.591 DEBUG:boto:Params: {} 2024-04-16T19:56:06.591 DEBUG:boto:Token: None 2024-04-16T19:56:06.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000246fe688cb5125a0-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.593 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:06.593 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:06.594 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.594 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:06.594 DEBUG:boto:Data: 2024-04-16T19:56:06.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.594 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.594 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.594 DEBUG:boto:Params: {} 2024-04-16T19:56:06.594 DEBUG:boto:Token: None 2024-04-16T19:56:06.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:06.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3687g+tdyTRsSgA5p7rzw3XamnA= 2024-04-16T19:56:06.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3687g+tdyTRsSgA5p7rzw3XamnA='} 2024-04-16T19:56:06.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007db904010489747d-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.601 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.602 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.602 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.602 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.602 DEBUG:boto:Data: 2024-04-16T19:56:06.602 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.602 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.602 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.602 DEBUG:boto:Params: {} 2024-04-16T19:56:06.602 DEBUG:boto:Token: None 2024-04-16T19:56:06.602 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.602 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094e5e0032c16ae23-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.605 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.605 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.605 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.606 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.606 DEBUG:boto:Data: 2024-04-16T19:56:06.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.606 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.606 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.606 DEBUG:boto:Params: {} 2024-04-16T19:56:06.606 DEBUG:boto:Token: None 2024-04-16T19:56:06.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:06.606 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mG7uCH8ATbfO7qtZieQK7YL1nWE= 2024-04-16T19:56:06.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mG7uCH8ATbfO7qtZieQK7YL1nWE='} 2024-04-16T19:56:06.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007887f643ea53bf88-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.650 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.651 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.651 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.651 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.651 DEBUG:boto:Data: 2024-04-16T19:56:06.651 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.651 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.651 DEBUG:boto:Params: {} 2024-04-16T19:56:06.651 DEBUG:boto:Token: None 2024-04-16T19:56:06.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf562656648829f4-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.654 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:06.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:06.655 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.655 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:06.655 DEBUG:boto:Data: 2024-04-16T19:56:06.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.655 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.655 DEBUG:boto:Params: {} 2024-04-16T19:56:06.655 DEBUG:boto:Token: None 2024-04-16T19:56:06.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:06.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jqtWqlh6SSh/mUph7AvrD8XNTtg= 2024-04-16T19:56:06.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jqtWqlh6SSh/mUph7AvrD8XNTtg='} 2024-04-16T19:56:06.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3e14e590a3b9a8c-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.661 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.661 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.661 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.662 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.662 DEBUG:boto:Data: 2024-04-16T19:56:06.662 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.662 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.662 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.662 DEBUG:boto:Params: {} 2024-04-16T19:56:06.662 DEBUG:boto:Token: None 2024-04-16T19:56:06.662 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092d4c0f1d03dfe1b-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.665 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:06.665 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:06.665 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.665 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:06.665 DEBUG:boto:Data: 2024-04-16T19:56:06.665 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.665 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.666 DEBUG:boto:Params: {} 2024-04-16T19:56:06.666 DEBUG:boto:Token: None 2024-04-16T19:56:06.666 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:06.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R3ivipcMWwYzxnoCks4tObrLtds= 2024-04-16T19:56:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R3ivipcMWwYzxnoCks4tObrLtds='} 2024-04-16T19:56:06.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef89540a83b7725f-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.672 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.672 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.672 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.672 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.672 DEBUG:boto:Data: 2024-04-16T19:56:06.672 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.672 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.672 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.672 DEBUG:boto:Params: {} 2024-04-16T19:56:06.672 DEBUG:boto:Token: None 2024-04-16T19:56:06.672 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9a7a916f63bae0-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.675 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:06.675 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:06.675 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.675 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:06.675 DEBUG:boto:Data: 2024-04-16T19:56:06.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.675 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.675 DEBUG:boto:Params: {} 2024-04-16T19:56:06.676 DEBUG:boto:Token: None 2024-04-16T19:56:06.676 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:06.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v7LQWHuNSksnME48kj3WuQFIG4Y= 2024-04-16T19:56:06.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v7LQWHuNSksnME48kj3WuQFIG4Y='} 2024-04-16T19:56:06.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005492fd10a9fdf59d-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.719 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.719 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.719 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.719 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.719 DEBUG:boto:Data: 2024-04-16T19:56:06.719 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.719 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.719 DEBUG:boto:Params: {} 2024-04-16T19:56:06.719 DEBUG:boto:Token: None 2024-04-16T19:56:06.720 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005aa06c20c9b58818-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.722 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:06.723 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:06.723 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.723 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:06.723 DEBUG:boto:Data: 2024-04-16T19:56:06.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.723 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.723 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.723 DEBUG:boto:Params: {} 2024-04-16T19:56:06.723 DEBUG:boto:Token: None 2024-04-16T19:56:06.723 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:06.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v1RzZXh0n8LmmPLWwkGDrbKX2eM= 2024-04-16T19:56:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v1RzZXh0n8LmmPLWwkGDrbKX2eM='} 2024-04-16T19:56:06.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067714df5e1a6dbcf-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.730 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.730 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.730 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.730 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.730 DEBUG:boto:Data: 2024-04-16T19:56:06.730 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.730 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.730 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.730 DEBUG:boto:Params: {} 2024-04-16T19:56:06.731 DEBUG:boto:Token: None 2024-04-16T19:56:06.731 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8954b16c097442c-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.733 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:06.733 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:06.734 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.734 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:06.734 DEBUG:boto:Data: 2024-04-16T19:56:06.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.734 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.734 DEBUG:boto:Params: {} 2024-04-16T19:56:06.734 DEBUG:boto:Token: None 2024-04-16T19:56:06.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:06.734 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VZ0FLc27b1dF/8on8yV9OOPWKFo= 2024-04-16T19:56:06.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VZ0FLc27b1dF/8on8yV9OOPWKFo='} 2024-04-16T19:56:06.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000846709efffe102ba-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.779 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.779 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.779 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.779 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.779 DEBUG:boto:Data: 2024-04-16T19:56:06.779 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.779 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.779 DEBUG:boto:Params: {} 2024-04-16T19:56:06.779 DEBUG:boto:Token: None 2024-04-16T19:56:06.779 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000374680150dec8110-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.782 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:06.783 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:06.783 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.783 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:06.783 DEBUG:boto:Data: 2024-04-16T19:56:06.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.783 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.783 DEBUG:boto:Params: {} 2024-04-16T19:56:06.783 DEBUG:boto:Token: None 2024-04-16T19:56:06.783 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:06.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wH/zCos8yiyiV7PD1ak8ReuCti8= 2024-04-16T19:56:06.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wH/zCos8yiyiV7PD1ak8ReuCti8='} 2024-04-16T19:56:06.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044a32b97567a2ba5-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.790 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.790 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.790 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.791 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.791 DEBUG:boto:Data: 2024-04-16T19:56:06.791 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.791 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.791 DEBUG:boto:Params: {} 2024-04-16T19:56:06.791 DEBUG:boto:Token: None 2024-04-16T19:56:06.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdf4b6e4c2572f4b-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.794 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:06.794 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:06.794 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.794 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:06.794 DEBUG:boto:Data: 2024-04-16T19:56:06.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.795 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.795 DEBUG:boto:Params: {} 2024-04-16T19:56:06.795 DEBUG:boto:Token: None 2024-04-16T19:56:06.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:06.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NZJIbyTOYNyViwuAsr9oPQYb6jw= 2024-04-16T19:56:06.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NZJIbyTOYNyViwuAsr9oPQYb6jw='} 2024-04-16T19:56:06.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1b79432cfeab9d1-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.801 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.801 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.801 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.801 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.801 DEBUG:boto:Data: 2024-04-16T19:56:06.801 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.801 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.801 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.801 DEBUG:boto:Params: {} 2024-04-16T19:56:06.801 DEBUG:boto:Token: None 2024-04-16T19:56:06.801 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.802 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001ee38267597f09c-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.804 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:06.804 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:06.804 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.804 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:06.804 DEBUG:boto:Data: 2024-04-16T19:56:06.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.805 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.805 DEBUG:boto:Params: {} 2024-04-16T19:56:06.805 DEBUG:boto:Token: None 2024-04-16T19:56:06.805 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:06.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j2pVgPpnotHyTKidcAbdS0ujcmI= 2024-04-16T19:56:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j2pVgPpnotHyTKidcAbdS0ujcmI='} 2024-04-16T19:56:06.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0d873096172e66d-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.847 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.847 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.847 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.847 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.847 DEBUG:boto:Data: 2024-04-16T19:56:06.847 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.847 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.847 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.847 DEBUG:boto:Params: {} 2024-04-16T19:56:06.847 DEBUG:boto:Token: None 2024-04-16T19:56:06.848 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d64ebb913cc118df-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.850 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.850 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.850 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.851 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.851 DEBUG:boto:Data: 2024-04-16T19:56:06.851 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.851 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.851 DEBUG:boto:Params: {} 2024-04-16T19:56:06.851 DEBUG:boto:Token: None 2024-04-16T19:56:06.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:06.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lS3RBG4aYCaLcYRR0fESBfNfuBE= 2024-04-16T19:56:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lS3RBG4aYCaLcYRR0fESBfNfuBE='} 2024-04-16T19:56:06.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d19d460f9253967a-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.857 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.857 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.858 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.858 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.858 DEBUG:boto:Data: 2024-04-16T19:56:06.858 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.858 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.858 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.858 DEBUG:boto:Params: {} 2024-04-16T19:56:06.858 DEBUG:boto:Token: None 2024-04-16T19:56:06.858 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fbeca908ed08c31-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.861 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:06.861 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:06.861 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.862 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:06.862 DEBUG:boto:Data: 2024-04-16T19:56:06.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.862 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.862 DEBUG:boto:Params: {} 2024-04-16T19:56:06.862 DEBUG:boto:Token: None 2024-04-16T19:56:06.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:06.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3WaS/ZMVcA6g10yDuLbrLSqLqI8= 2024-04-16T19:56:06.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3WaS/ZMVcA6g10yDuLbrLSqLqI8='} 2024-04-16T19:56:06.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024de856386952260-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.869 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.869 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.869 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.869 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.869 DEBUG:boto:Data: 2024-04-16T19:56:06.869 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.869 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.869 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.869 DEBUG:boto:Params: {} 2024-04-16T19:56:06.869 DEBUG:boto:Token: None 2024-04-16T19:56:06.870 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.870 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb98f5c21ffcb167-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.873 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:06.873 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:06.873 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.873 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:06.873 DEBUG:boto:Data: 2024-04-16T19:56:06.873 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.873 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.873 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.873 DEBUG:boto:Params: {} 2024-04-16T19:56:06.873 DEBUG:boto:Token: None 2024-04-16T19:56:06.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:06.874 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wYpKJqpmp7C4zl8w6o38NITbx8= 2024-04-16T19:56:06.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wYpKJqpmp7C4zl8w6o38NITbx8='} 2024-04-16T19:56:06.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1644fed1be92b3f-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.918 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.919 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.919 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.919 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.919 DEBUG:boto:Data: 2024-04-16T19:56:06.919 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.919 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.919 DEBUG:boto:Params: {} 2024-04-16T19:56:06.919 DEBUG:boto:Token: None 2024-04-16T19:56:06.920 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007364dba8779076d0-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.922 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:06.923 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:06.923 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.923 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:06.923 DEBUG:boto:Data: 2024-04-16T19:56:06.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.923 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.923 DEBUG:boto:Params: {} 2024-04-16T19:56:06.923 DEBUG:boto:Token: None 2024-04-16T19:56:06.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:06.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:977GgCx40Teu7k0+CNkDw9AYu/8= 2024-04-16T19:56:06.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:977GgCx40Teu7k0+CNkDw9AYu/8='} 2024-04-16T19:56:06.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bac9a9c3303d94f-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.930 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.930 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.930 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.930 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.931 DEBUG:boto:Data: 2024-04-16T19:56:06.931 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.931 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.931 DEBUG:boto:Params: {} 2024-04-16T19:56:06.931 DEBUG:boto:Token: None 2024-04-16T19:56:06.931 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f68c4f8a9146ac90-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.934 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:06.934 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:06.934 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.934 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:06.934 DEBUG:boto:Data: 2024-04-16T19:56:06.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.935 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.935 DEBUG:boto:Params: {} 2024-04-16T19:56:06.935 DEBUG:boto:Token: None 2024-04-16T19:56:06.935 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:06.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dw1RzBArn/DERzcWMk1eFDSqcfs= 2024-04-16T19:56:06.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dw1RzBArn/DERzcWMk1eFDSqcfs='} 2024-04-16T19:56:06.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061140f9453c6e899-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.943 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.943 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.943 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.943 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.943 DEBUG:boto:Data: 2024-04-16T19:56:06.943 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.943 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.943 DEBUG:boto:Params: {} 2024-04-16T19:56:06.943 DEBUG:boto:Token: None 2024-04-16T19:56:06.943 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fed37b336130cee8-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.946 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.946 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.946 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.947 DEBUG:boto:Data: 2024-04-16T19:56:06.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.947 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.947 DEBUG:boto:Params: {} 2024-04-16T19:56:06.947 DEBUG:boto:Token: None 2024-04-16T19:56:06.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:06.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Vl8Ld2b2SRUijYEX0RqEvk/eMy8= 2024-04-16T19:56:06.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Vl8Ld2b2SRUijYEX0RqEvk/eMy8='} 2024-04-16T19:56:06.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f0466015779fb0d-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.991 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:06.991 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:06.991 DEBUG:boto:Method: HEAD 2024-04-16T19:56:06.991 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:06.991 DEBUG:boto:Data: 2024-04-16T19:56:06.991 DEBUG:boto:Headers: {} 2024-04-16T19:56:06.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.991 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.991 DEBUG:boto:Params: {} 2024-04-16T19:56:06.991 DEBUG:boto:Token: None 2024-04-16T19:56:06.991 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/ 2024-04-16T19:56:06.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY= 2024-04-16T19:56:06.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6V6YCUWmxPWCr68Fce3h+NGaSY=', 'Content-Length': '0'} 2024-04-16T19:56:06.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a569169896f34870-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:06.994 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:06.994 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:06.995 DEBUG:boto:Method: PUT 2024-04-16T19:56:06.995 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:06.995 DEBUG:boto:Data: 2024-04-16T19:56:06.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:06.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:06.995 DEBUG:boto:Port: 8000 2024-04-16T19:56:06.995 DEBUG:boto:Params: {} 2024-04-16T19:56:06.995 DEBUG:boto:Token: None 2024-04-16T19:56:06.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:06 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:06.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OkeVVPncThRMFycuiWPTO1IzBwA= 2024-04-16T19:56:06.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OkeVVPncThRMFycuiWPTO1IzBwA='} 2024-04-16T19:56:07.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f646cb596aa0777-00661ed7d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.002 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.002 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.002 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.002 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.002 DEBUG:boto:Data: 2024-04-16T19:56:07.002 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.002 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.003 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.003 DEBUG:boto:Params: {} 2024-04-16T19:56:07.003 DEBUG:boto:Token: None 2024-04-16T19:56:07.003 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030208f2e92aee39a-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.006 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:07.006 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:07.006 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.006 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:07.006 DEBUG:boto:Data: 2024-04-16T19:56:07.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.006 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.006 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.006 DEBUG:boto:Params: {} 2024-04-16T19:56:07.006 DEBUG:boto:Token: None 2024-04-16T19:56:07.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:07.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:prEHWozJPPULkAfvBPOQQrNuXhk= 2024-04-16T19:56:07.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:prEHWozJPPULkAfvBPOQQrNuXhk='} 2024-04-16T19:56:07.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bea3f8250383ca2-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.012 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.012 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.012 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.012 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.012 DEBUG:boto:Data: 2024-04-16T19:56:07.012 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.012 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.012 DEBUG:boto:Params: {} 2024-04-16T19:56:07.012 DEBUG:boto:Token: None 2024-04-16T19:56:07.012 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007348eb68a38c4433-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.015 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:07.015 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:07.015 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.015 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:07.015 DEBUG:boto:Data: 2024-04-16T19:56:07.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.015 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.016 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.016 DEBUG:boto:Params: {} 2024-04-16T19:56:07.016 DEBUG:boto:Token: None 2024-04-16T19:56:07.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:07.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Js2eeHtXj75PnYeSiASXXWbTqKY= 2024-04-16T19:56:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Js2eeHtXj75PnYeSiASXXWbTqKY='} 2024-04-16T19:56:07.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b0e45189f7d2839-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.059 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.059 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.059 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.059 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.059 DEBUG:boto:Data: 2024-04-16T19:56:07.059 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.059 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.059 DEBUG:boto:Params: {} 2024-04-16T19:56:07.060 DEBUG:boto:Token: None 2024-04-16T19:56:07.060 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000294faa43b5286fae-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.062 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:07.062 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:07.063 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.063 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:07.063 DEBUG:boto:Data: 2024-04-16T19:56:07.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.063 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.063 DEBUG:boto:Params: {} 2024-04-16T19:56:07.063 DEBUG:boto:Token: None 2024-04-16T19:56:07.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:07.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A0Txs8vucC0tD6BIZbBbEDiaPEQ= 2024-04-16T19:56:07.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A0Txs8vucC0tD6BIZbBbEDiaPEQ='} 2024-04-16T19:56:07.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ce160882d8e2d6b-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.069 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.069 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.069 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.069 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.069 DEBUG:boto:Data: 2024-04-16T19:56:07.069 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.070 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.070 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.070 DEBUG:boto:Params: {} 2024-04-16T19:56:07.070 DEBUG:boto:Token: None 2024-04-16T19:56:07.070 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028126d0e83229e17-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:07.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:07.073 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.073 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:07.073 DEBUG:boto:Data: 2024-04-16T19:56:07.073 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.073 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.073 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.073 DEBUG:boto:Params: {} 2024-04-16T19:56:07.074 DEBUG:boto:Token: None 2024-04-16T19:56:07.074 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:07.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d4353frx5T7dY+CCryne+qp/A08= 2024-04-16T19:56:07.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d4353frx5T7dY+CCryne+qp/A08='} 2024-04-16T19:56:07.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cdf65ef49bd7568-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.079 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.079 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.079 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.079 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.079 DEBUG:boto:Data: 2024-04-16T19:56:07.079 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.080 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.080 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.080 DEBUG:boto:Params: {} 2024-04-16T19:56:07.080 DEBUG:boto:Token: None 2024-04-16T19:56:07.080 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005cac7ed70ec9abe-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.083 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:07.083 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:07.083 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.083 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:07.083 DEBUG:boto:Data: 2024-04-16T19:56:07.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.083 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.083 DEBUG:boto:Params: {} 2024-04-16T19:56:07.083 DEBUG:boto:Token: None 2024-04-16T19:56:07.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:07.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8gzJfgsfFGyiyr6OZc1M2kOU2tw= 2024-04-16T19:56:07.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8gzJfgsfFGyiyr6OZc1M2kOU2tw='} 2024-04-16T19:56:07.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005672752f8f0e8d7f-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.127 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.127 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.127 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.127 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.127 DEBUG:boto:Data: 2024-04-16T19:56:07.127 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.127 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.127 DEBUG:boto:Params: {} 2024-04-16T19:56:07.127 DEBUG:boto:Token: None 2024-04-16T19:56:07.128 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc844ef932ab4f2e-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.130 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.131 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.131 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.131 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.131 DEBUG:boto:Data: 2024-04-16T19:56:07.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.131 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.131 DEBUG:boto:Params: {} 2024-04-16T19:56:07.131 DEBUG:boto:Token: None 2024-04-16T19:56:07.131 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZLxraa5EjX2LxpEy+z/otcNzRO4= 2024-04-16T19:56:07.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZLxraa5EjX2LxpEy+z/otcNzRO4='} 2024-04-16T19:56:07.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a582e0e8d0eeb3aa-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:07.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:07.138 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:07.190 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:07.254 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:07.255 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:07.268 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 43 2024-04-16T19:56:07.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:07.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:07.336 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:07.385 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:07.385 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:07.454 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000019","obj00000000000000000006","obj00000000000000000467","obj00000000000000000005","obj00000000000000000056","obj00000000000000000004","obj00000000000000000023","obj00000000000000000317","obj00000000000000000022","obj00000000000000000000","obj00000000000000000021","obj00000000000000000009","obj00000000000000000271","obj00000000000000000008","obj00000000000000000018","obj00000000000000000007","obj00000000000000000017","obj00000000000000000162","obj00000000000000000016","obj00000000000000000003","obj00000000000000000423","obj00000000000000000002","obj00000000000000000012","obj00000000000000000001","obj00000000000000000011","obj00000000000000000283","obj00000000000000000010","obj00000000000000000061","obj00000000000000000050","obj00000000000000000029","obj00000000000000000133"]} 2024-04-16T19:56:07.454 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.454 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.454 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.455 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.455 DEBUG:boto:Data: 2024-04-16T19:56:07.455 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.455 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.455 DEBUG:boto:Params: {} 2024-04-16T19:56:07.455 DEBUG:boto:Token: None 2024-04-16T19:56:07.455 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091eaec8564f3104e-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.458 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:07.459 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:07.459 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.459 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:07.459 DEBUG:boto:Data: 2024-04-16T19:56:07.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.459 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.459 DEBUG:boto:Params: {} 2024-04-16T19:56:07.459 DEBUG:boto:Token: None 2024-04-16T19:56:07.459 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:07.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Er6F3h2mLAgQnLIzLXEIqU9mxis= 2024-04-16T19:56:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Er6F3h2mLAgQnLIzLXEIqU9mxis='} 2024-04-16T19:56:07.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a71590ead5669bdb-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.467 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.467 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.467 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.467 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.467 DEBUG:boto:Data: 2024-04-16T19:56:07.467 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.467 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.467 DEBUG:boto:Params: {} 2024-04-16T19:56:07.467 DEBUG:boto:Token: None 2024-04-16T19:56:07.467 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043bad9903158dfb1-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.470 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:07.470 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:07.471 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.471 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:07.471 DEBUG:boto:Data: 2024-04-16T19:56:07.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.471 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.472 DEBUG:boto:Params: {} 2024-04-16T19:56:07.472 DEBUG:boto:Token: None 2024-04-16T19:56:07.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:07.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jrRPJP5YquvbQi6bPfc4UzOo9v8= 2024-04-16T19:56:07.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jrRPJP5YquvbQi6bPfc4UzOo9v8='} 2024-04-16T19:56:07.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebd952dce23ccef0-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.479 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.479 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.479 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.479 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.479 DEBUG:boto:Data: 2024-04-16T19:56:07.479 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.479 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.479 DEBUG:boto:Params: {} 2024-04-16T19:56:07.480 DEBUG:boto:Token: None 2024-04-16T19:56:07.480 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0142bbbb8395f17-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.483 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:07.483 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:07.483 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.483 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:07.483 DEBUG:boto:Data: 2024-04-16T19:56:07.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.484 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.484 DEBUG:boto:Params: {} 2024-04-16T19:56:07.484 DEBUG:boto:Token: None 2024-04-16T19:56:07.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:07.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:blSb18diUhia0jBY6ljVcSnN+4g= 2024-04-16T19:56:07.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:blSb18diUhia0jBY6ljVcSnN+4g='} 2024-04-16T19:56:07.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065a1120dcccdd0e3-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.526 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.527 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.527 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.527 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.527 DEBUG:boto:Data: 2024-04-16T19:56:07.527 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.527 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.527 DEBUG:boto:Params: {} 2024-04-16T19:56:07.527 DEBUG:boto:Token: None 2024-04-16T19:56:07.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000129a478c5335e8d8-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.530 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:07.530 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:07.530 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.530 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:07.530 DEBUG:boto:Data: 2024-04-16T19:56:07.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.531 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.531 DEBUG:boto:Params: {} 2024-04-16T19:56:07.531 DEBUG:boto:Token: None 2024-04-16T19:56:07.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:07.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9bMIPFwEZkjE2xPazGzEi6VoIlA= 2024-04-16T19:56:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9bMIPFwEZkjE2xPazGzEi6VoIlA='} 2024-04-16T19:56:07.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4d2c2bf1e96cee0-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.538 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.538 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.539 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.539 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.539 DEBUG:boto:Data: 2024-04-16T19:56:07.539 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.539 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.539 DEBUG:boto:Params: {} 2024-04-16T19:56:07.539 DEBUG:boto:Token: None 2024-04-16T19:56:07.539 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfd627a72a3550f2-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.543 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:07.543 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:07.543 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.543 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:07.543 DEBUG:boto:Data: 2024-04-16T19:56:07.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.543 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.543 DEBUG:boto:Params: {} 2024-04-16T19:56:07.543 DEBUG:boto:Token: None 2024-04-16T19:56:07.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:07.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YAmB3U1d8l/B5rlKnJb57Ft035U= 2024-04-16T19:56:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YAmB3U1d8l/B5rlKnJb57Ft035U='} 2024-04-16T19:56:07.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000496a8b91be391e75-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.587 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.587 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.587 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.587 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.587 DEBUG:boto:Data: 2024-04-16T19:56:07.587 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.587 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.587 DEBUG:boto:Params: {} 2024-04-16T19:56:07.587 DEBUG:boto:Token: None 2024-04-16T19:56:07.587 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072abc1b389c69640-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.590 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:07.590 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:07.590 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.591 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:07.591 DEBUG:boto:Data: 2024-04-16T19:56:07.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.591 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.591 DEBUG:boto:Params: {} 2024-04-16T19:56:07.591 DEBUG:boto:Token: None 2024-04-16T19:56:07.591 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:07.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5oZYZfK2EjbqW5IKcBwZoliE/N4= 2024-04-16T19:56:07.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5oZYZfK2EjbqW5IKcBwZoliE/N4='} 2024-04-16T19:56:07.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004df61a0178739f7f-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.597 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.597 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.597 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.597 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.597 DEBUG:boto:Data: 2024-04-16T19:56:07.597 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.597 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.597 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.597 DEBUG:boto:Params: {} 2024-04-16T19:56:07.597 DEBUG:boto:Token: None 2024-04-16T19:56:07.598 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.598 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000195ff9fb72ac5e1-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.600 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:07.601 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:07.601 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.601 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:07.601 DEBUG:boto:Data: 2024-04-16T19:56:07.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.601 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.601 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.601 DEBUG:boto:Params: {} 2024-04-16T19:56:07.601 DEBUG:boto:Token: None 2024-04-16T19:56:07.601 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:07.601 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i2OdPP55fF0zqt/B4uL91ohsVI0= 2024-04-16T19:56:07.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i2OdPP55fF0zqt/B4uL91ohsVI0='} 2024-04-16T19:56:07.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000175dc31e3bf35bef-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.607 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.607 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.607 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.607 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.607 DEBUG:boto:Data: 2024-04-16T19:56:07.607 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.607 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.607 DEBUG:boto:Params: {} 2024-04-16T19:56:07.607 DEBUG:boto:Token: None 2024-04-16T19:56:07.608 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e64c574da2c53a5-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.610 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:07.611 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:07.611 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.611 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:07.611 DEBUG:boto:Data: 2024-04-16T19:56:07.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.612 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.612 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.612 DEBUG:boto:Params: {} 2024-04-16T19:56:07.612 DEBUG:boto:Token: None 2024-04-16T19:56:07.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:07.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Kro+B/G44lx+IkxLGNgIFdcEIw0= 2024-04-16T19:56:07.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kro+B/G44lx+IkxLGNgIFdcEIw0='} 2024-04-16T19:56:07.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007106bc459e4bb8c4-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.655 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.655 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.655 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.655 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.655 DEBUG:boto:Data: 2024-04-16T19:56:07.655 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.655 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.655 DEBUG:boto:Params: {} 2024-04-16T19:56:07.655 DEBUG:boto:Token: None 2024-04-16T19:56:07.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa7a8cd67d81358f-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.658 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:07.659 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:07.659 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.659 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:07.659 DEBUG:boto:Data: 2024-04-16T19:56:07.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.659 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.659 DEBUG:boto:Params: {} 2024-04-16T19:56:07.659 DEBUG:boto:Token: None 2024-04-16T19:56:07.659 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:07.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IMOkfljDZsx8rTZDIRVA60k2Was= 2024-04-16T19:56:07.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IMOkfljDZsx8rTZDIRVA60k2Was='} 2024-04-16T19:56:07.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da74d65d6f7b4018-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.665 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.665 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.665 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.665 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.665 DEBUG:boto:Data: 2024-04-16T19:56:07.665 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.665 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.665 DEBUG:boto:Params: {} 2024-04-16T19:56:07.666 DEBUG:boto:Token: None 2024-04-16T19:56:07.666 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000724834a57bd1a5ce-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.668 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:07.669 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:07.669 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.669 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:07.669 DEBUG:boto:Data: 2024-04-16T19:56:07.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.669 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.669 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.669 DEBUG:boto:Params: {} 2024-04-16T19:56:07.669 DEBUG:boto:Token: None 2024-04-16T19:56:07.669 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:07.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rS8Iqz72hGVtaWIh6Owmckn8fy4= 2024-04-16T19:56:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rS8Iqz72hGVtaWIh6Owmckn8fy4='} 2024-04-16T19:56:07.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec1796dc5108358b-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.676 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.676 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.676 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.676 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.676 DEBUG:boto:Data: 2024-04-16T19:56:07.676 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.676 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.676 DEBUG:boto:Params: {} 2024-04-16T19:56:07.677 DEBUG:boto:Token: None 2024-04-16T19:56:07.677 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f43c23575f84827-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.680 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:07.680 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:07.680 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.680 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:07.680 DEBUG:boto:Data: 2024-04-16T19:56:07.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.680 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.680 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.680 DEBUG:boto:Params: {} 2024-04-16T19:56:07.680 DEBUG:boto:Token: None 2024-04-16T19:56:07.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:07.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4H4VRmL5Hh1jc6Htr/SwdelNdpg= 2024-04-16T19:56:07.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4H4VRmL5Hh1jc6Htr/SwdelNdpg='} 2024-04-16T19:56:07.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019752dfed2692479-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.722 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.723 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.723 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.723 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.723 DEBUG:boto:Data: 2024-04-16T19:56:07.723 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.723 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.723 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.723 DEBUG:boto:Params: {} 2024-04-16T19:56:07.723 DEBUG:boto:Token: None 2024-04-16T19:56:07.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002020ce9213311074-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.727 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:07.727 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:07.727 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.727 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:07.727 DEBUG:boto:Data: 2024-04-16T19:56:07.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.727 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.727 DEBUG:boto:Params: {} 2024-04-16T19:56:07.727 DEBUG:boto:Token: None 2024-04-16T19:56:07.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:07.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h9D35Z6z2cKCuTaNIMpWODSJc2o= 2024-04-16T19:56:07.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h9D35Z6z2cKCuTaNIMpWODSJc2o='} 2024-04-16T19:56:07.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000863b2a7a93d94c23-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.734 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.734 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.734 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.734 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.735 DEBUG:boto:Data: 2024-04-16T19:56:07.735 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.735 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.735 DEBUG:boto:Params: {} 2024-04-16T19:56:07.735 DEBUG:boto:Token: None 2024-04-16T19:56:07.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bcabfe5c8d095d4-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.738 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:07.738 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:07.738 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.739 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:07.739 DEBUG:boto:Data: 2024-04-16T19:56:07.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.739 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.739 DEBUG:boto:Params: {} 2024-04-16T19:56:07.739 DEBUG:boto:Token: None 2024-04-16T19:56:07.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:07.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mf0umZKmXtJBF2YBSvU0bi54Gxs= 2024-04-16T19:56:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mf0umZKmXtJBF2YBSvU0bi54Gxs='} 2024-04-16T19:56:07.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe3e4323da5eab54-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.783 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.783 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.783 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.783 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.783 DEBUG:boto:Data: 2024-04-16T19:56:07.783 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.783 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.783 DEBUG:boto:Params: {} 2024-04-16T19:56:07.783 DEBUG:boto:Token: None 2024-04-16T19:56:07.784 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003220c476c202661d-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.786 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:07.786 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:07.787 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.787 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:07.787 DEBUG:boto:Data: 2024-04-16T19:56:07.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.787 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.787 DEBUG:boto:Params: {} 2024-04-16T19:56:07.787 DEBUG:boto:Token: None 2024-04-16T19:56:07.787 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:07.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4ayXvgRvKAoR11KupMo+IVJkix0= 2024-04-16T19:56:07.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4ayXvgRvKAoR11KupMo+IVJkix0='} 2024-04-16T19:56:07.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd2f48aee1df5ed4-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.793 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.793 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.793 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.793 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.793 DEBUG:boto:Data: 2024-04-16T19:56:07.793 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.793 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.793 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.793 DEBUG:boto:Params: {} 2024-04-16T19:56:07.794 DEBUG:boto:Token: None 2024-04-16T19:56:07.794 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044b0b19145f1d3f7-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.796 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:07.796 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:07.797 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.797 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:07.797 DEBUG:boto:Data: 2024-04-16T19:56:07.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.797 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.797 DEBUG:boto:Params: {} 2024-04-16T19:56:07.797 DEBUG:boto:Token: None 2024-04-16T19:56:07.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:07.797 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JFca4DYN09iMZvpGXCzCrK6VNzE= 2024-04-16T19:56:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JFca4DYN09iMZvpGXCzCrK6VNzE='} 2024-04-16T19:56:07.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b378164054252f1-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.802 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.802 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.802 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.803 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.803 DEBUG:boto:Data: 2024-04-16T19:56:07.803 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.803 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.803 DEBUG:boto:Params: {} 2024-04-16T19:56:07.803 DEBUG:boto:Token: None 2024-04-16T19:56:07.803 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aa275cb753dde5a-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:07.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:07.806 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.806 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:07.806 DEBUG:boto:Data: 2024-04-16T19:56:07.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.807 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.807 DEBUG:boto:Params: {} 2024-04-16T19:56:07.807 DEBUG:boto:Token: None 2024-04-16T19:56:07.807 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:07.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IE9wNNAPy/yMUzzzTqWZrYSCC8Q= 2024-04-16T19:56:07.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IE9wNNAPy/yMUzzzTqWZrYSCC8Q='} 2024-04-16T19:56:07.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000540b79be2a9ec7b9-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.851 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.851 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.851 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.851 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.851 DEBUG:boto:Data: 2024-04-16T19:56:07.851 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.851 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.851 DEBUG:boto:Params: {} 2024-04-16T19:56:07.852 DEBUG:boto:Token: None 2024-04-16T19:56:07.852 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004842f5db272b7305-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.855 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:07.855 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:07.855 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.855 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:07.855 DEBUG:boto:Data: 2024-04-16T19:56:07.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.856 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.856 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.856 DEBUG:boto:Params: {} 2024-04-16T19:56:07.856 DEBUG:boto:Token: None 2024-04-16T19:56:07.856 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:07.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GPbgR4JX0CMal1xoEzCDMjw+S0E= 2024-04-16T19:56:07.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GPbgR4JX0CMal1xoEzCDMjw+S0E='} 2024-04-16T19:56:07.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000878ac24aaf8a9f9b-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.862 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.862 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.862 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.862 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.862 DEBUG:boto:Data: 2024-04-16T19:56:07.862 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.862 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.862 DEBUG:boto:Params: {} 2024-04-16T19:56:07.862 DEBUG:boto:Token: None 2024-04-16T19:56:07.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cb44ee0107e1aad-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.865 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.866 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.866 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.866 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.866 DEBUG:boto:Data: 2024-04-16T19:56:07.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.866 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.866 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.866 DEBUG:boto:Params: {} 2024-04-16T19:56:07.866 DEBUG:boto:Token: None 2024-04-16T19:56:07.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:07.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZLxraa5EjX2LxpEy+z/otcNzRO4= 2024-04-16T19:56:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZLxraa5EjX2LxpEy+z/otcNzRO4='} 2024-04-16T19:56:07.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f79952310764a545-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.911 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.911 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.911 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.911 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.911 DEBUG:boto:Data: 2024-04-16T19:56:07.911 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.911 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.912 DEBUG:boto:Params: {} 2024-04-16T19:56:07.912 DEBUG:boto:Token: None 2024-04-16T19:56:07.912 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cb1424ae02be689-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.915 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:07.915 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:07.915 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.915 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:07.915 DEBUG:boto:Data: 2024-04-16T19:56:07.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.916 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.916 DEBUG:boto:Params: {} 2024-04-16T19:56:07.916 DEBUG:boto:Token: None 2024-04-16T19:56:07.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:07.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OwDVsrpHqocwZ0Hd9ZyhFX9V+j4= 2024-04-16T19:56:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OwDVsrpHqocwZ0Hd9ZyhFX9V+j4='} 2024-04-16T19:56:07.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065df56913bdce9a6-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.922 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.922 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.922 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.922 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.922 DEBUG:boto:Data: 2024-04-16T19:56:07.922 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.922 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.922 DEBUG:boto:Params: {} 2024-04-16T19:56:07.923 DEBUG:boto:Token: None 2024-04-16T19:56:07.923 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097caef8a393f84c7-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.925 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:07.926 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:07.926 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.926 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:07.926 DEBUG:boto:Data: 2024-04-16T19:56:07.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.926 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.926 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.926 DEBUG:boto:Params: {} 2024-04-16T19:56:07.926 DEBUG:boto:Token: None 2024-04-16T19:56:07.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:07.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a/z9M/TK5tSNQb+SDKRXV/AP2aI= 2024-04-16T19:56:07.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a/z9M/TK5tSNQb+SDKRXV/AP2aI='} 2024-04-16T19:56:07.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca268c8629109093-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.971 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.971 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.971 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.971 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.971 DEBUG:boto:Data: 2024-04-16T19:56:07.971 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.971 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.971 DEBUG:boto:Params: {} 2024-04-16T19:56:07.971 DEBUG:boto:Token: None 2024-04-16T19:56:07.971 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007663618213a17741-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.975 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:07.975 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:07.975 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.975 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:07.975 DEBUG:boto:Data: 2024-04-16T19:56:07.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.975 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.975 DEBUG:boto:Params: {} 2024-04-16T19:56:07.975 DEBUG:boto:Token: None 2024-04-16T19:56:07.975 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:07.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x+WOyqrOhR4HEdaPjwceBdVHjig= 2024-04-16T19:56:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x+WOyqrOhR4HEdaPjwceBdVHjig='} 2024-04-16T19:56:07.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd6a6ef3778e207d-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.981 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:07.981 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:07.981 DEBUG:boto:Method: HEAD 2024-04-16T19:56:07.981 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:07.981 DEBUG:boto:Data: 2024-04-16T19:56:07.981 DEBUG:boto:Headers: {} 2024-04-16T19:56:07.981 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.982 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.982 DEBUG:boto:Params: {} 2024-04-16T19:56:07.982 DEBUG:boto:Token: None 2024-04-16T19:56:07.982 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/ 2024-04-16T19:56:07.982 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw= 2024-04-16T19:56:07.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pFicxZAwfARjbK/trfDw4mKN1uw=', 'Content-Length': '0'} 2024-04-16T19:56:07.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb37dcbb341d9485-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:07.985 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:07.985 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:07.985 DEBUG:boto:Method: PUT 2024-04-16T19:56:07.985 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:07.985 DEBUG:boto:Data: 2024-04-16T19:56:07.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:07.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:07.986 DEBUG:boto:Port: 8000 2024-04-16T19:56:07.986 DEBUG:boto:Params: {} 2024-04-16T19:56:07.986 DEBUG:boto:Token: None 2024-04-16T19:56:07.986 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:07 GMT /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:07.986 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z4u914uG++Isn/jgwEvNHuAKhvg= 2024-04-16T19:56:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z4u914uG++Isn/jgwEvNHuAKhvg='} 2024-04-16T19:56:08.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d56aba74ce473038-00661ed7d7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.031 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.031 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.031 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.031 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.031 DEBUG:boto:Data: 2024-04-16T19:56:08.031 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.031 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.031 DEBUG:boto:Params: {} 2024-04-16T19:56:08.031 DEBUG:boto:Token: None 2024-04-16T19:56:08.031 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000530a12f4a89df3f2-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.035 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:08.035 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:08.035 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.035 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:08.035 DEBUG:boto:Data: 2024-04-16T19:56:08.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.035 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.035 DEBUG:boto:Params: {} 2024-04-16T19:56:08.035 DEBUG:boto:Token: None 2024-04-16T19:56:08.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:08.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9DbXDY6HddlDLWR/lWy/EN78AY0= 2024-04-16T19:56:08.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9DbXDY6HddlDLWR/lWy/EN78AY0='} 2024-04-16T19:56:08.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000437a16f7fb0cd77f-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.042 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.042 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.043 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.043 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.043 DEBUG:boto:Data: 2024-04-16T19:56:08.043 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.043 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.043 DEBUG:boto:Params: {} 2024-04-16T19:56:08.043 DEBUG:boto:Token: None 2024-04-16T19:56:08.043 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077208cc917754f89-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.046 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:08.046 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:08.046 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.046 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:08.047 DEBUG:boto:Data: 2024-04-16T19:56:08.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.047 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.047 DEBUG:boto:Params: {} 2024-04-16T19:56:08.047 DEBUG:boto:Token: None 2024-04-16T19:56:08.047 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:08.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9UzW/gQksvBF5axBDqQvRfzkf24= 2024-04-16T19:56:08.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9UzW/gQksvBF5axBDqQvRfzkf24='} 2024-04-16T19:56:08.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6730cf49ce6ab1a-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.091 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.091 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.091 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.091 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.091 DEBUG:boto:Data: 2024-04-16T19:56:08.091 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.091 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.091 DEBUG:boto:Params: {} 2024-04-16T19:56:08.092 DEBUG:boto:Token: None 2024-04-16T19:56:08.092 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18d339bf43d36f0-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.095 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.095 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.095 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.095 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.095 DEBUG:boto:Data: 2024-04-16T19:56:08.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.095 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.095 DEBUG:boto:Params: {} 2024-04-16T19:56:08.096 DEBUG:boto:Token: None 2024-04-16T19:56:08.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pfU0i2Ga4+H0ct7iaGrFpyeY0a8= 2024-04-16T19:56:08.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pfU0i2Ga4+H0ct7iaGrFpyeY0a8='} 2024-04-16T19:56:08.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034b3542affe90ec8-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.103 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.103 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.103 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.103 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.104 DEBUG:boto:Data: 2024-04-16T19:56:08.104 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.104 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.104 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.105 DEBUG:boto:Params: {} 2024-04-16T19:56:08.105 DEBUG:boto:Token: None 2024-04-16T19:56:08.105 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b6ffa329b677e12-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.108 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:08.109 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:08.109 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.109 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:08.109 DEBUG:boto:Data: 2024-04-16T19:56:08.109 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.110 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.110 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.110 DEBUG:boto:Params: {} 2024-04-16T19:56:08.110 DEBUG:boto:Token: None 2024-04-16T19:56:08.110 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:08.110 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MzcqKtNnFXvgE+l6ER1OxSM/Dbo= 2024-04-16T19:56:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MzcqKtNnFXvgE+l6ER1OxSM/Dbo='} 2024-04-16T19:56:08.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cfd07717bf3ecc0-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.155 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.155 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.155 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.155 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.155 DEBUG:boto:Data: 2024-04-16T19:56:08.155 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.155 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.155 DEBUG:boto:Params: {} 2024-04-16T19:56:08.155 DEBUG:boto:Token: None 2024-04-16T19:56:08.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bb3bbb77b048e33-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.159 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.159 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.159 DEBUG:boto:Data: 2024-04-16T19:56:08.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.159 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.159 DEBUG:boto:Params: {} 2024-04-16T19:56:08.159 DEBUG:boto:Token: None 2024-04-16T19:56:08.160 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U9gw3oYNXaVvBjbDkGAQGhDiefM= 2024-04-16T19:56:08.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U9gw3oYNXaVvBjbDkGAQGhDiefM='} 2024-04-16T19:56:08.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040892dd05a19f6ea-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.166 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.166 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.166 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.166 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.166 DEBUG:boto:Data: 2024-04-16T19:56:08.166 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.167 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.167 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.167 DEBUG:boto:Params: {} 2024-04-16T19:56:08.167 DEBUG:boto:Token: None 2024-04-16T19:56:08.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2b574a9a329cdb4-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.170 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:08.170 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:08.170 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.170 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:08.171 DEBUG:boto:Data: 2024-04-16T19:56:08.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.171 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.171 DEBUG:boto:Params: {} 2024-04-16T19:56:08.171 DEBUG:boto:Token: None 2024-04-16T19:56:08.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:08.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OvNbp1DsOqHe3TpfvEzoL3oR9SU= 2024-04-16T19:56:08.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OvNbp1DsOqHe3TpfvEzoL3oR9SU='} 2024-04-16T19:56:08.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d80085013439a4fc-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:08.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:08.215 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:08.331 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:56:08.331 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-51 2024-04-16T19:56:08.344 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 55 2024-04-16T19:56:08.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:08.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:08.426 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:08.476 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:08.476 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:08.548 INFO:teuthology.orchestra.run.smithi026.stdout:{"objs":["obj00000000000000000060","obj00000000000000000000","obj00000000000000000011","obj00000000000000000001","obj00000000000000000033","obj00000000000000000065","obj00000000000000000005","obj00000000000000000016","obj00000000000000000048","obj00000000000000000015","obj00000000000000000026","obj00000000000000000030","obj00000000000000000062","obj00000000000000000009","obj00000000000000000040","obj00000000000000000024","obj00000000000000000012","obj00000000000000000002","obj00000000000000000013","obj00000000000000000045","obj00000000000000000006","obj00000000000000000017","obj00000000000000000007","obj00000000000000000039","obj00000000000000000027","obj00000000000000000010","obj00000000000000000021","obj00000000000000000053","obj00000000000000000020","obj00000000000000000004","obj00000000000000000036","obj00000000000000000003","obj00000000000000000014","obj00000000000000000019","obj00000000000000000057","obj00000000000000000018","obj00000000000000000008"]} 2024-04-16T19:56:08.549 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.549 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.549 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.549 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.549 DEBUG:boto:Data: 2024-04-16T19:56:08.549 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.549 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.550 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.550 DEBUG:boto:Params: {} 2024-04-16T19:56:08.550 DEBUG:boto:Token: None 2024-04-16T19:56:08.550 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002878b69cf857cd31-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.553 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:08.554 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:08.554 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.554 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:08.554 DEBUG:boto:Data: 2024-04-16T19:56:08.554 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.554 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.554 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.554 DEBUG:boto:Params: {} 2024-04-16T19:56:08.554 DEBUG:boto:Token: None 2024-04-16T19:56:08.555 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:08.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eIhsMBSy9+WtFFOEZlTg1UROaZw= 2024-04-16T19:56:08.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eIhsMBSy9+WtFFOEZlTg1UROaZw='} 2024-04-16T19:56:08.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c82e0b5c6d22fc0f-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.562 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.562 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.562 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.562 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.562 DEBUG:boto:Data: 2024-04-16T19:56:08.562 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.562 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.562 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.562 DEBUG:boto:Params: {} 2024-04-16T19:56:08.563 DEBUG:boto:Token: None 2024-04-16T19:56:08.563 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44ea21ebbb32e0a-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.566 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:08.567 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:08.567 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.567 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:08.567 DEBUG:boto:Data: 2024-04-16T19:56:08.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.567 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.567 DEBUG:boto:Params: {} 2024-04-16T19:56:08.567 DEBUG:boto:Token: None 2024-04-16T19:56:08.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:08.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KJyItom0qu59Pwc0mXWZHgquqmI= 2024-04-16T19:56:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KJyItom0qu59Pwc0mXWZHgquqmI='} 2024-04-16T19:56:08.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f76301ee2c9f6a55-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.574 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.574 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.574 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.574 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.574 DEBUG:boto:Data: 2024-04-16T19:56:08.575 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.575 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.575 DEBUG:boto:Params: {} 2024-04-16T19:56:08.575 DEBUG:boto:Token: None 2024-04-16T19:56:08.575 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ede90c5006ae454b-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.578 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:08.578 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:08.579 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.579 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:08.579 DEBUG:boto:Data: 2024-04-16T19:56:08.579 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.579 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.579 DEBUG:boto:Params: {} 2024-04-16T19:56:08.579 DEBUG:boto:Token: None 2024-04-16T19:56:08.579 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:08.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lGB4CFmgzMMquvol9BcOQ357i0o= 2024-04-16T19:56:08.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lGB4CFmgzMMquvol9BcOQ357i0o='} 2024-04-16T19:56:08.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7c801fd38d8b99a-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.622 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.623 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.623 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.623 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.623 DEBUG:boto:Data: 2024-04-16T19:56:08.623 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.623 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.623 DEBUG:boto:Params: {} 2024-04-16T19:56:08.623 DEBUG:boto:Token: None 2024-04-16T19:56:08.623 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef5906f03e4c9904-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.627 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.627 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.627 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.627 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.627 DEBUG:boto:Data: 2024-04-16T19:56:08.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.627 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.627 DEBUG:boto:Params: {} 2024-04-16T19:56:08.627 DEBUG:boto:Token: None 2024-04-16T19:56:08.628 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:08.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U9gw3oYNXaVvBjbDkGAQGhDiefM= 2024-04-16T19:56:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U9gw3oYNXaVvBjbDkGAQGhDiefM='} 2024-04-16T19:56:08.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcd805f1218021cc-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.634 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.634 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.634 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.634 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.634 DEBUG:boto:Data: 2024-04-16T19:56:08.634 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.634 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.635 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.635 DEBUG:boto:Params: {} 2024-04-16T19:56:08.635 DEBUG:boto:Token: None 2024-04-16T19:56:08.635 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc761a7f69b5b5db-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.638 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:08.639 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:08.639 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.639 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:08.639 DEBUG:boto:Data: 2024-04-16T19:56:08.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.639 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.639 DEBUG:boto:Params: {} 2024-04-16T19:56:08.639 DEBUG:boto:Token: None 2024-04-16T19:56:08.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:08.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nYaahA7qtdIqn7KTpShW4e3S4vw= 2024-04-16T19:56:08.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nYaahA7qtdIqn7KTpShW4e3S4vw='} 2024-04-16T19:56:08.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003105b1729573982d-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.683 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.683 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.683 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.683 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.683 DEBUG:boto:Data: 2024-04-16T19:56:08.683 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.683 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.683 DEBUG:boto:Params: {} 2024-04-16T19:56:08.683 DEBUG:boto:Token: None 2024-04-16T19:56:08.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000729a67be23b9b31-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.687 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:08.687 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:08.687 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.687 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:08.687 DEBUG:boto:Data: 2024-04-16T19:56:08.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.687 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.687 DEBUG:boto:Params: {} 2024-04-16T19:56:08.688 DEBUG:boto:Token: None 2024-04-16T19:56:08.688 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:08.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ta7/czY0fjQTAxQ7ZJRzpwodxKY= 2024-04-16T19:56:08.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ta7/czY0fjQTAxQ7ZJRzpwodxKY='} 2024-04-16T19:56:08.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000467203da34eef9de-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.694 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.694 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.694 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.694 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.694 DEBUG:boto:Data: 2024-04-16T19:56:08.694 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.695 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.695 DEBUG:boto:Params: {} 2024-04-16T19:56:08.695 DEBUG:boto:Token: None 2024-04-16T19:56:08.695 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003195f97a94048311-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.698 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:08.698 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:08.698 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.698 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:08.699 DEBUG:boto:Data: 2024-04-16T19:56:08.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.699 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.699 DEBUG:boto:Params: {} 2024-04-16T19:56:08.699 DEBUG:boto:Token: None 2024-04-16T19:56:08.699 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:08.699 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6hjPsjNL1VVHL3Xe0AY6onLxxpM= 2024-04-16T19:56:08.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6hjPsjNL1VVHL3Xe0AY6onLxxpM='} 2024-04-16T19:56:08.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085a265eabb1ac9e6-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.743 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.743 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.743 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.743 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.743 DEBUG:boto:Data: 2024-04-16T19:56:08.743 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.743 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.743 DEBUG:boto:Params: {} 2024-04-16T19:56:08.743 DEBUG:boto:Token: None 2024-04-16T19:56:08.744 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b0ec3122018a151-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.747 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:08.747 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:08.747 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.747 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:08.747 DEBUG:boto:Data: 2024-04-16T19:56:08.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.748 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.748 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.748 DEBUG:boto:Params: {} 2024-04-16T19:56:08.748 DEBUG:boto:Token: None 2024-04-16T19:56:08.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:08.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jw9+3fxrRUNpXuudE4cbdxK6zZI= 2024-04-16T19:56:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jw9+3fxrRUNpXuudE4cbdxK6zZI='} 2024-04-16T19:56:08.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c636e9af6579829-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.753 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.753 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.754 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.754 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.754 DEBUG:boto:Data: 2024-04-16T19:56:08.754 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.754 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.754 DEBUG:boto:Params: {} 2024-04-16T19:56:08.754 DEBUG:boto:Token: None 2024-04-16T19:56:08.754 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.754 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050b434598ae247bf-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.757 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:08.758 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:08.758 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.758 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:08.758 DEBUG:boto:Data: 2024-04-16T19:56:08.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.758 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.758 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.758 DEBUG:boto:Params: {} 2024-04-16T19:56:08.758 DEBUG:boto:Token: None 2024-04-16T19:56:08.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:08.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uq201mpRr11HHFQFytTV/bk+JMM= 2024-04-16T19:56:08.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uq201mpRr11HHFQFytTV/bk+JMM='} 2024-04-16T19:56:08.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8cbb837aa2ab2ee-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.802 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.803 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.803 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.803 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.803 DEBUG:boto:Data: 2024-04-16T19:56:08.803 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.803 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.803 DEBUG:boto:Params: {} 2024-04-16T19:56:08.803 DEBUG:boto:Token: None 2024-04-16T19:56:08.803 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044e1e9a103dc38ca-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:08.807 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:08.807 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.807 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:08.807 DEBUG:boto:Data: 2024-04-16T19:56:08.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.807 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.807 DEBUG:boto:Params: {} 2024-04-16T19:56:08.807 DEBUG:boto:Token: None 2024-04-16T19:56:08.807 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:08.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xzglq9tbS+JGU+7qliqlPeZnY9I= 2024-04-16T19:56:08.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xzglq9tbS+JGU+7qliqlPeZnY9I='} 2024-04-16T19:56:08.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097f0fe093daab2a4-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.814 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.814 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.814 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.814 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.815 DEBUG:boto:Data: 2024-04-16T19:56:08.815 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.815 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.815 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.815 DEBUG:boto:Params: {} 2024-04-16T19:56:08.815 DEBUG:boto:Token: None 2024-04-16T19:56:08.815 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042f6900086f72ced-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.818 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:08.819 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:08.819 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.819 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:08.819 DEBUG:boto:Data: 2024-04-16T19:56:08.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.819 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.819 DEBUG:boto:Params: {} 2024-04-16T19:56:08.819 DEBUG:boto:Token: None 2024-04-16T19:56:08.819 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:08.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Frxi0F2GI27lAa5tzbMwMswMdpU= 2024-04-16T19:56:08.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Frxi0F2GI27lAa5tzbMwMswMdpU='} 2024-04-16T19:56:08.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e85d5f81f2e0e20f-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.862 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.863 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.863 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.863 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.863 DEBUG:boto:Data: 2024-04-16T19:56:08.863 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.863 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.863 DEBUG:boto:Params: {} 2024-04-16T19:56:08.863 DEBUG:boto:Token: None 2024-04-16T19:56:08.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c6c785418f01fdc-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.866 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:08.866 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:08.867 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.867 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:08.867 DEBUG:boto:Data: 2024-04-16T19:56:08.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.867 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.867 DEBUG:boto:Params: {} 2024-04-16T19:56:08.867 DEBUG:boto:Token: None 2024-04-16T19:56:08.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:08.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZxdJs90q5RuSnNyjhHTRiQgze/M= 2024-04-16T19:56:08.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZxdJs90q5RuSnNyjhHTRiQgze/M='} 2024-04-16T19:56:08.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054d7b39de745234b-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.874 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.874 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.874 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.874 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.874 DEBUG:boto:Data: 2024-04-16T19:56:08.874 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.874 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.875 DEBUG:boto:Params: {} 2024-04-16T19:56:08.875 DEBUG:boto:Token: None 2024-04-16T19:56:08.875 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000314f8985f45e089c-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.878 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:08.878 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:08.878 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.878 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:08.878 DEBUG:boto:Data: 2024-04-16T19:56:08.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.879 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.879 DEBUG:boto:Params: {} 2024-04-16T19:56:08.879 DEBUG:boto:Token: None 2024-04-16T19:56:08.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:08.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A+toVCvJLox80gdCgqT6mX4xtbM= 2024-04-16T19:56:08.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A+toVCvJLox80gdCgqT6mX4xtbM='} 2024-04-16T19:56:08.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdbbcfe713b0b14a-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.885 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.885 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.886 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.886 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.886 DEBUG:boto:Data: 2024-04-16T19:56:08.886 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.886 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.886 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.886 DEBUG:boto:Params: {} 2024-04-16T19:56:08.886 DEBUG:boto:Token: None 2024-04-16T19:56:08.886 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.886 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85bf00ab66c8302-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.889 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:08.889 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:08.890 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.890 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:08.890 DEBUG:boto:Data: 2024-04-16T19:56:08.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.890 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.890 DEBUG:boto:Params: {} 2024-04-16T19:56:08.890 DEBUG:boto:Token: None 2024-04-16T19:56:08.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:08.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sPTqEVm5yUHGYJZrj6as2hyv8m4= 2024-04-16T19:56:08.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sPTqEVm5yUHGYJZrj6as2hyv8m4='} 2024-04-16T19:56:08.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007afc9970cf8af40f-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.935 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.935 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.936 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.936 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.936 DEBUG:boto:Data: 2024-04-16T19:56:08.936 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.936 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.937 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.937 DEBUG:boto:Params: {} 2024-04-16T19:56:08.937 DEBUG:boto:Token: None 2024-04-16T19:56:08.937 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.938 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000042b81a377ec76e-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.942 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:08.942 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:08.943 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.943 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:08.943 DEBUG:boto:Data: 2024-04-16T19:56:08.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.943 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.943 DEBUG:boto:Params: {} 2024-04-16T19:56:08.943 DEBUG:boto:Token: None 2024-04-16T19:56:08.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:08.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZUz5CyxrL6y94H72qy2MYdhqWWA= 2024-04-16T19:56:08.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZUz5CyxrL6y94H72qy2MYdhqWWA='} 2024-04-16T19:56:08.949 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e0ee35c2ea61ca9-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.949 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.949 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.949 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.949 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.949 DEBUG:boto:Data: 2024-04-16T19:56:08.949 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.949 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.950 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.950 DEBUG:boto:Params: {} 2024-04-16T19:56:08.950 DEBUG:boto:Token: None 2024-04-16T19:56:08.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce6c875cb4a6da76-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.953 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:08.953 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:08.953 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.953 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:08.953 DEBUG:boto:Data: 2024-04-16T19:56:08.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.954 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.954 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.954 DEBUG:boto:Params: {} 2024-04-16T19:56:08.954 DEBUG:boto:Token: None 2024-04-16T19:56:08.954 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:08.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Vb/Bp/V6si3ZI+bKHGbE4bqwXf4= 2024-04-16T19:56:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Vb/Bp/V6si3ZI+bKHGbE4bqwXf4='} 2024-04-16T19:56:08.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcb544f04eb0d45e-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.960 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:08.961 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:08.961 DEBUG:boto:Method: HEAD 2024-04-16T19:56:08.961 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:08.961 DEBUG:boto:Data: 2024-04-16T19:56:08.961 DEBUG:boto:Headers: {} 2024-04-16T19:56:08.961 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.961 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.961 DEBUG:boto:Params: {} 2024-04-16T19:56:08.961 DEBUG:boto:Token: None 2024-04-16T19:56:08.961 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/ 2024-04-16T19:56:08.962 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ= 2024-04-16T19:56:08.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ohDEyb9tv1wpTXKeBLCmX9dMYBQ=', 'Content-Length': '0'} 2024-04-16T19:56:08.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006525395fc9202418-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:08.965 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.965 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.965 DEBUG:boto:Method: PUT 2024-04-16T19:56:08.965 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.965 DEBUG:boto:Data: 2024-04-16T19:56:08.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:08.965 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:08.965 DEBUG:boto:Port: 8000 2024-04-16T19:56:08.965 DEBUG:boto:Params: {} 2024-04-16T19:56:08.966 DEBUG:boto:Token: None 2024-04-16T19:56:08.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:08 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:08.966 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pfU0i2Ga4+H0ct7iaGrFpyeY0a8= 2024-04-16T19:56:08.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:08 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pfU0i2Ga4+H0ct7iaGrFpyeY0a8='} 2024-04-16T19:56:09.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009212336d89964786-00661ed7d8-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.011 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.011 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.011 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.011 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.011 DEBUG:boto:Data: 2024-04-16T19:56:09.011 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.011 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.012 DEBUG:boto:Params: {} 2024-04-16T19:56:09.012 DEBUG:boto:Token: None 2024-04-16T19:56:09.012 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002338fc46579165a8-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.015 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:09.016 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:09.016 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.016 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:09.016 DEBUG:boto:Data: 2024-04-16T19:56:09.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.016 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.016 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.016 DEBUG:boto:Params: {} 2024-04-16T19:56:09.016 DEBUG:boto:Token: None 2024-04-16T19:56:09.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:09.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wazXD3bidr9y9auqmZ0zQiu/wDM= 2024-04-16T19:56:09.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wazXD3bidr9y9auqmZ0zQiu/wDM='} 2024-04-16T19:56:09.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3d47e80782bbc6a-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.023 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.023 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.023 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.023 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.023 DEBUG:boto:Data: 2024-04-16T19:56:09.023 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.024 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.024 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.024 DEBUG:boto:Params: {} 2024-04-16T19:56:09.024 DEBUG:boto:Token: None 2024-04-16T19:56:09.024 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ab1c475aeec8fbc-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.027 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:09.027 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:09.027 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.028 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:09.028 DEBUG:boto:Data: 2024-04-16T19:56:09.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.028 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.028 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.028 DEBUG:boto:Params: {} 2024-04-16T19:56:09.028 DEBUG:boto:Token: None 2024-04-16T19:56:09.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:09.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gnWewJkIPS4Zy5qWulP3hFO/kvw= 2024-04-16T19:56:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gnWewJkIPS4Zy5qWulP3hFO/kvw='} 2024-04-16T19:56:09.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001dbfd3f5c2ea8746-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.071 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.071 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.071 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.071 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.071 DEBUG:boto:Data: 2024-04-16T19:56:09.071 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.071 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.071 DEBUG:boto:Params: {} 2024-04-16T19:56:09.071 DEBUG:boto:Token: None 2024-04-16T19:56:09.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000953302503af0415a-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.075 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:09.075 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:09.075 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.075 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:09.075 DEBUG:boto:Data: 2024-04-16T19:56:09.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.075 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.075 DEBUG:boto:Params: {} 2024-04-16T19:56:09.076 DEBUG:boto:Token: None 2024-04-16T19:56:09.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:09.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WCTra+jkixMdAa6mR9xdb2ynJDM= 2024-04-16T19:56:09.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WCTra+jkixMdAa6mR9xdb2ynJDM='} 2024-04-16T19:56:09.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c3750677623999a-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.083 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.083 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.083 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.083 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.083 DEBUG:boto:Data: 2024-04-16T19:56:09.083 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.083 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.083 DEBUG:boto:Params: {} 2024-04-16T19:56:09.083 DEBUG:boto:Token: None 2024-04-16T19:56:09.083 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003698c8e6f6f90312-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.087 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:09.087 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:09.087 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.087 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:09.087 DEBUG:boto:Data: 2024-04-16T19:56:09.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.087 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.087 DEBUG:boto:Params: {} 2024-04-16T19:56:09.087 DEBUG:boto:Token: None 2024-04-16T19:56:09.088 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:09.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DIQai2Eq3xCAgb4ESkiAZ6Oo+3s= 2024-04-16T19:56:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DIQai2Eq3xCAgb4ESkiAZ6Oo+3s='} 2024-04-16T19:56:09.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1e1d069f8f046e2-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.131 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.131 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.131 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.131 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.131 DEBUG:boto:Data: 2024-04-16T19:56:09.131 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.131 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.131 DEBUG:boto:Params: {} 2024-04-16T19:56:09.131 DEBUG:boto:Token: None 2024-04-16T19:56:09.132 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8bc27604ec6d4a-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.135 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:09.135 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:09.135 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.135 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:09.135 DEBUG:boto:Data: 2024-04-16T19:56:09.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.135 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.136 DEBUG:boto:Params: {} 2024-04-16T19:56:09.136 DEBUG:boto:Token: None 2024-04-16T19:56:09.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:09.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNe3w0xNf9VFbO1XZ/7GGjJjye8= 2024-04-16T19:56:09.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNe3w0xNf9VFbO1XZ/7GGjJjye8='} 2024-04-16T19:56:09.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9faeb84c9104177-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.142 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.142 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.142 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.142 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.143 DEBUG:boto:Data: 2024-04-16T19:56:09.143 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.143 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.143 DEBUG:boto:Params: {} 2024-04-16T19:56:09.143 DEBUG:boto:Token: None 2024-04-16T19:56:09.143 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000845fa5d03581b172-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.146 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:09.147 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:09.147 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.147 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:09.147 DEBUG:boto:Data: 2024-04-16T19:56:09.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.148 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.148 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.148 DEBUG:boto:Params: {} 2024-04-16T19:56:09.148 DEBUG:boto:Token: None 2024-04-16T19:56:09.148 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:09.149 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P2ewSODdrSN65JLJiqUE8S+A9Ko= 2024-04-16T19:56:09.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P2ewSODdrSN65JLJiqUE8S+A9Ko='} 2024-04-16T19:56:09.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095d610b2da6ed1fa-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.195 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.195 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.195 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.195 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.195 DEBUG:boto:Data: 2024-04-16T19:56:09.195 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.195 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.195 DEBUG:boto:Params: {} 2024-04-16T19:56:09.195 DEBUG:boto:Token: None 2024-04-16T19:56:09.196 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c9e380a9c6d783-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.199 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:09.199 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:09.199 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.199 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:09.199 DEBUG:boto:Data: 2024-04-16T19:56:09.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.200 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.200 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.200 DEBUG:boto:Params: {} 2024-04-16T19:56:09.200 DEBUG:boto:Token: None 2024-04-16T19:56:09.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:09.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NCI9Bp6I2ttlDhs2/31O5hJ8Z3U= 2024-04-16T19:56:09.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NCI9Bp6I2ttlDhs2/31O5hJ8Z3U='} 2024-04-16T19:56:09.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c57e794d4ea4d252-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.207 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.207 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.207 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.207 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.207 DEBUG:boto:Data: 2024-04-16T19:56:09.207 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.207 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.207 DEBUG:boto:Params: {} 2024-04-16T19:56:09.208 DEBUG:boto:Token: None 2024-04-16T19:56:09.208 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ed76a19e590bd45-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.211 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:09.211 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:09.211 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.211 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:09.212 DEBUG:boto:Data: 2024-04-16T19:56:09.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.212 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.212 DEBUG:boto:Params: {} 2024-04-16T19:56:09.212 DEBUG:boto:Token: None 2024-04-16T19:56:09.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:09.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z4C2oyzltimVUpZvf58uNkAJ2D8= 2024-04-16T19:56:09.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z4C2oyzltimVUpZvf58uNkAJ2D8='} 2024-04-16T19:56:09.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa41afbb73ed9188-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.255 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.255 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.255 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.255 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.255 DEBUG:boto:Data: 2024-04-16T19:56:09.255 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.255 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.255 DEBUG:boto:Params: {} 2024-04-16T19:56:09.255 DEBUG:boto:Token: None 2024-04-16T19:56:09.256 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff2d82320d02b60-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.259 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:09.259 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:09.260 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.260 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:09.260 DEBUG:boto:Data: 2024-04-16T19:56:09.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.260 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.260 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.260 DEBUG:boto:Params: {} 2024-04-16T19:56:09.260 DEBUG:boto:Token: None 2024-04-16T19:56:09.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:09.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/ASmM7tGqQ4sh8qA23HOHIaPMJQ= 2024-04-16T19:56:09.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/ASmM7tGqQ4sh8qA23HOHIaPMJQ='} 2024-04-16T19:56:09.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004916f03bdd927d8-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.267 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.267 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.267 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.268 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.268 DEBUG:boto:Data: 2024-04-16T19:56:09.268 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.268 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.268 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.268 DEBUG:boto:Params: {} 2024-04-16T19:56:09.268 DEBUG:boto:Token: None 2024-04-16T19:56:09.268 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc77f4e89d0378fc-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.271 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:09.271 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:09.272 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.272 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:09.272 DEBUG:boto:Data: 2024-04-16T19:56:09.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.272 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.272 DEBUG:boto:Params: {} 2024-04-16T19:56:09.272 DEBUG:boto:Token: None 2024-04-16T19:56:09.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:09.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iDp73ueqZ3yS6iHoJxg6ihR5p8I= 2024-04-16T19:56:09.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iDp73ueqZ3yS6iHoJxg6ihR5p8I='} 2024-04-16T19:56:09.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f33308a46a3078a5-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.314 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.315 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.315 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.315 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.315 DEBUG:boto:Data: 2024-04-16T19:56:09.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.315 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.315 DEBUG:boto:Params: {} 2024-04-16T19:56:09.315 DEBUG:boto:Token: None 2024-04-16T19:56:09.315 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000990a65740eee180d-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.319 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:09.319 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:09.319 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.319 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:09.319 DEBUG:boto:Data: 2024-04-16T19:56:09.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.319 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.320 DEBUG:boto:Params: {} 2024-04-16T19:56:09.320 DEBUG:boto:Token: None 2024-04-16T19:56:09.320 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:09.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8wF2YlA55pXfhxQa8LPFadXax4g= 2024-04-16T19:56:09.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8wF2YlA55pXfhxQa8LPFadXax4g='} 2024-04-16T19:56:09.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c9c1a6a26a587e0-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.328 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.328 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.328 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.328 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.329 DEBUG:boto:Data: 2024-04-16T19:56:09.329 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.329 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.329 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.329 DEBUG:boto:Params: {} 2024-04-16T19:56:09.329 DEBUG:boto:Token: None 2024-04-16T19:56:09.329 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.330 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f6628a33a53798f-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.333 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:09.333 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:09.333 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.333 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:09.333 DEBUG:boto:Data: 2024-04-16T19:56:09.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.334 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.334 DEBUG:boto:Params: {} 2024-04-16T19:56:09.334 DEBUG:boto:Token: None 2024-04-16T19:56:09.334 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:09.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Pta2DYF5+f3ujBFargmpCX/t8h0= 2024-04-16T19:56:09.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Pta2DYF5+f3ujBFargmpCX/t8h0='} 2024-04-16T19:56:09.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9053248243f7406-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.378 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.379 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.379 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.379 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.379 DEBUG:boto:Data: 2024-04-16T19:56:09.379 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.379 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.379 DEBUG:boto:Params: {} 2024-04-16T19:56:09.379 DEBUG:boto:Token: None 2024-04-16T19:56:09.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295baf6b5f45654b-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.383 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:09.383 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:09.383 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.383 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:09.383 DEBUG:boto:Data: 2024-04-16T19:56:09.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.383 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.383 DEBUG:boto:Params: {} 2024-04-16T19:56:09.384 DEBUG:boto:Token: None 2024-04-16T19:56:09.384 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:09.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TdmV/vi/O0VBbx5Yawdz2VcPzFo= 2024-04-16T19:56:09.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TdmV/vi/O0VBbx5Yawdz2VcPzFo='} 2024-04-16T19:56:09.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ffa070e2fe3debc-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.390 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.390 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.390 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.390 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.391 DEBUG:boto:Data: 2024-04-16T19:56:09.391 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.391 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.391 DEBUG:boto:Params: {} 2024-04-16T19:56:09.391 DEBUG:boto:Token: None 2024-04-16T19:56:09.391 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b0da9472f93971a-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.394 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:09.395 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:09.395 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.395 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:09.395 DEBUG:boto:Data: 2024-04-16T19:56:09.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.395 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.395 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.395 DEBUG:boto:Params: {} 2024-04-16T19:56:09.395 DEBUG:boto:Token: None 2024-04-16T19:56:09.395 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:09.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KZRFh3I4CaXSY0o31f8XMUCLri0= 2024-04-16T19:56:09.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KZRFh3I4CaXSY0o31f8XMUCLri0='} 2024-04-16T19:56:09.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008462f25a66a06f59-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.439 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.439 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.439 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.439 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.439 DEBUG:boto:Data: 2024-04-16T19:56:09.439 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.439 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.439 DEBUG:boto:Params: {} 2024-04-16T19:56:09.439 DEBUG:boto:Token: None 2024-04-16T19:56:09.439 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '62'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ea85fb965a9f5fe-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.443 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:09.443 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:09.443 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.443 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:09.443 DEBUG:boto:Data: 2024-04-16T19:56:09.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.443 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.444 DEBUG:boto:Params: {} 2024-04-16T19:56:09.444 DEBUG:boto:Token: None 2024-04-16T19:56:09.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:09.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nq3PUfw0bupiknYoMDwUfCPtKb8= 2024-04-16T19:56:09.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nq3PUfw0bupiknYoMDwUfCPtKb8='} 2024-04-16T19:56:09.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c166da62097e9d32-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.452 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.452 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.452 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.452 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.452 DEBUG:boto:Data: 2024-04-16T19:56:09.452 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.453 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.453 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.453 DEBUG:boto:Params: {} 2024-04-16T19:56:09.453 DEBUG:boto:Token: None 2024-04-16T19:56:09.453 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.453 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '62'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a983dfe37bb9468-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.456 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:09.456 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:09.456 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.456 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:09.457 DEBUG:boto:Data: 2024-04-16T19:56:09.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.457 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.457 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.457 DEBUG:boto:Params: {} 2024-04-16T19:56:09.457 DEBUG:boto:Token: None 2024-04-16T19:56:09.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:09.457 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IpMgzGJDNNVBC+F164kB0NNdiSY= 2024-04-16T19:56:09.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IpMgzGJDNNVBC+F164kB0NNdiSY='} 2024-04-16T19:56:09.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000395a4e89ca4ea08-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.498 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:09.499 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:09.499 DEBUG:boto:Method: HEAD 2024-04-16T19:56:09.499 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:09.499 DEBUG:boto:Data: 2024-04-16T19:56:09.499 DEBUG:boto:Headers: {} 2024-04-16T19:56:09.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.499 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.499 DEBUG:boto:Params: {} 2024-04-16T19:56:09.499 DEBUG:boto:Token: None 2024-04-16T19:56:09.499 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/ 2024-04-16T19:56:09.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM= 2024-04-16T19:56:09.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:maCMzxTMrGqdyQI/OEueVeqAcvM=', 'Content-Length': '0'} 2024-04-16T19:56:09.502 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '62'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7c2803805d87174-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.503 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:09.503 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:09.503 DEBUG:boto:Method: PUT 2024-04-16T19:56:09.503 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:09.503 DEBUG:boto:Data: 2024-04-16T19:56:09.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:56:09.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:09.503 DEBUG:boto:Port: 8000 2024-04-16T19:56:09.503 DEBUG:boto:Params: {} 2024-04-16T19:56:09.503 DEBUG:boto:Token: None 2024-04-16T19:56:09.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:56:09 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:09.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xqx/syfZxMyn5wzW0tC7ug2ivpk= 2024-04-16T19:56:09.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:56:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xqx/syfZxMyn5wzW0tC7ug2ivpk='} 2024-04-16T19:56:09.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059f5208ef085f934-00661ed7d9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:09.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:09.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:56:09.512 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T19:56:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.672+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.672+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.672+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.672+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 2 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.31.6 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 2 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-16T19:56:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 2 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000001.110.6 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 2 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.673+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 2 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000001.76.6 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 2 2024-04-16T19:56:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000001.67.6 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 2 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000001.83.6 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 2 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 2 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000001.46.6 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000001.65.6 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 2 2024-04-16T19:56:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 2 2024-04-16T19:56:09.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-16T19:56:09.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.715+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.716+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000020 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:04.601695+0000 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.57.6 op=0 op_state=1 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000020]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000020[0] 2024-04-16T19:56:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000001.57.6 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000007 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:04.356000+0000 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.53.6 op=0 op_state=1 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000028 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:04.542244+0000 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-16T19:56:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000016 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:04.299371+0000 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.39.6 op=0 op_state=1 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.53.6 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000028]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000028[0] 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000016]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000016[0] 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-16T19:56:09.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2/obj00000000000000000024 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: entry->timestamp=2024-04-16T19:56:04.345676+0000 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.54.6 op=0 op_state=1 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000007 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000024]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000024[0] 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000001.54.6 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000000 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:04.409813+0000 2024-04-16T19:56:09.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.73.6 op=0 op_state=1 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.718+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000001.73.6 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:09.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.719+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.720+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.720+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.720+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.720+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13/obj00000000000000000026 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: entry->timestamp=2024-04-16T19:56:04.533385+0000 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.59.6 op=0 op_state=1 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000026]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000026[0] 2024-04-16T19:56:09.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.721+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position=00000000001.59.6 2024-04-16T19:56:09.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.117.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:09.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000021 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:04.487397+0000 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.117.6 op=0 op_state=1 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.117.6 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000013 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:04.232278+0000 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.722+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000021 2024-04-16T19:56:09.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000013]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000013[0] 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000076 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:04.243458+0000 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000076]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000076[0] 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000001.115.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000030 2024-04-16T19:56:09.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:04.290017+0000 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.115.6 op=0 op_state=1 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000004 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:04.429450+0000 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000030]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000030[0] 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000001.115.6 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:09.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.723+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000004 2024-04-16T19:56:09.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.736+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000030]: success 2024-04-16T19:56:09.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.736+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:1 marker=00000000001.115.6 timestamp=2024-04-16T19:56:04.290017+0000 2024-04-16T19:56:09.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.736+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000016]: success 2024-04-16T19:56:09.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.736+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:1 marker=00000000001.39.6 timestamp=2024-04-16T19:56:04.299371+0000 2024-04-16T19:56:09.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.739+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000000]: success 2024-04-16T19:56:09.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.739+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:1 marker=00000000001.73.6 timestamp=2024-04-16T19:56:04.409813+0000 2024-04-16T19:56:09.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.739+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000076]: success 2024-04-16T19:56:09.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.739+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16:1 marker=00000000001.41.6 timestamp=2024-04-16T19:56:04.243458+0000 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000024]: success 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2:1 marker=00000000001.54.6 timestamp=2024-04-16T19:56:04.345676+0000 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000013]: success 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10:1 marker=00000000002.55.6 timestamp=2024-04-16T19:56:04.232278+0000 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000028]: success 2024-04-16T19:56:09.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.740+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:1 marker=00000000001.41.6 timestamp=2024-04-16T19:56:04.542244+0000 2024-04-16T19:56:09.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.744+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000026]: success 2024-04-16T19:56:09.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.744+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13:1 marker=00000000001.59.6 timestamp=2024-04-16T19:56:04.533385+0000 2024-04-16T19:56:09.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.744+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000020]: success 2024-04-16T19:56:09.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.744+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:1 marker=00000000001.57.6 timestamp=2024-04-16T19:56:04.601695+0000 2024-04-16T19:56:09.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:09.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000001 2024-04-16T19:56:09.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:04.590156+0000 2024-04-16T19:56:09.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-16T19:56:09.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:09.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.759+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000019 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:04.477907+0000 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-16T19:56:09.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000010 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:04.221209+0000 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000019 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000010]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000010[0] 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: a sync operation returned error 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: backing out with sync_status=-5 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:09.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.760+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:09.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.761+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.761+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:09.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.761+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000033 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:04.419822+0000 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.52.6 op=0 op_state=1 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000033]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000033[0] 2024-04-16T19:56:09.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.764+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000001.52.6 2024-04-16T19:56:09.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: a sync operation returned error 2024-04-16T19:56:09.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: a sync operation returned error 2024-04-16T19:56:09.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: backing out with sync_status=-5 2024-04-16T19:56:09.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: backing out with sync_status=-5 2024-04-16T19:56:09.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:09.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:09.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:09.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.765+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:09.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:09.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:09.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.768+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.774+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000010]: success 2024-04-16T19:56:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.774+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:1 marker=00000000002.39.6 timestamp=2024-04-16T19:56:04.221209+0000 2024-04-16T19:56:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.774+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000033]: success 2024-04-16T19:56:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.774+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14:1 marker=00000000001.52.6 timestamp=2024-04-16T19:56:04.419822+0000 2024-04-16T19:56:09.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.802+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: a sync operation returned error 2024-04-16T19:56:09.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.802+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: backing out with sync_status=-5 2024-04-16T19:56:09.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.802+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:09.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.802+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:09.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000007 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:04.356000+0000 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.53.6 op=0 op_state=1 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.53.6 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000007 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:09.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.803+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.805+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000001]: success 2024-04-16T19:56:09.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.805+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:1 marker=00000000002.41.6 timestamp=2024-04-16T19:56:04.590156+0000 2024-04-16T19:56:09.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.117.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:09.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000021 2024-04-16T19:56:09.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:04.487397+0000 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.117.6 op=0 op_state=1 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000021[0] 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.117.6 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000004 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:04.429450+0000 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000004[0] 2024-04-16T19:56:09.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.811+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-16T19:56:09.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.815+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000021]: success 2024-04-16T19:56:09.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.815+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:1 marker=00000000001.117.6 timestamp=2024-04-16T19:56:04.487397+0000 2024-04-16T19:56:09.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.816+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000004]: success 2024-04-16T19:56:09.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.816+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:1 marker=00000000002.57.6 timestamp=2024-04-16T19:56:04.429450+0000 2024-04-16T19:56:09.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: a sync operation returned error 2024-04-16T19:56:09.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:09.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000019 2024-04-16T19:56:09.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:04.477907+0000 2024-04-16T19:56:09.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000019[0] 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: backing out with sync_status=-5 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.845+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:09.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.846+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.846+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:09.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.846+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.850+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000019]: success 2024-04-16T19:56:09.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.850+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:1 marker=00000000001.40.6 timestamp=2024-04-16T19:56:04.477907+0000 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000007 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:04.356000+0000 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.53.6 op=0 op_state=1 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000007[0] 2024-04-16T19:56:09.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.888+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.53.6 2024-04-16T19:56:09.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.892+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000007]: success 2024-04-16T19:56:09.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.892+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:1 marker=00000000001.53.6 timestamp=2024-04-16T19:56:04.356000+0000 2024-04-16T19:56:09.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.935+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:09.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.936+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000021 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:04.904913+0000 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2/obj00000000000000000016 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: entry->timestamp=2024-04-16T19:56:05.105559+0000 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000021]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000021[0] 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-16T19:56:09.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000016]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000016[0] 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.937+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000025 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:05.128848+0000 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.50.6 op=0 op_state=1 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000025]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000025[0] 2024-04-16T19:56:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000002.121.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000004 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:05.322010+0000 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.121.6 op=0 op_state=1 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000010 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:04.973021+0000 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-16T19:56:09.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000004]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000004[0] 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000002.121.6 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000010]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000010[0] 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000015 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:05.263823+0000 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-16T19:56:09.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6/obj00000000000000000030 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: entry->timestamp=2024-04-16T19:56:05.192133+0000 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000015]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000015[0] 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.938+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000030]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000030[0] 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.939+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket shard: 0 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket shard: 0 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position= 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000021]: success 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:2 marker=00000000002.81.6 timestamp=2024-04-16T19:56:04.904913+0000 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000016]: success 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.940+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2:2 marker=00000000002.73.6 timestamp=2024-04-16T19:56:05.105559+0000 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.941+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000010]: success 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.941+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:2 marker=00000000002.57.6 timestamp=2024-04-16T19:56:04.973021+0000 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000004]: success 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:2 marker=00000000002.121.6 timestamp=2024-04-16T19:56:05.322010+0000 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000014 2024-04-16T19:56:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:05.181983+0000 2024-04-16T19:56:09.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.85.6 op=0 op_state=1 2024-04-16T19:56:09.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000014]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000014[0] 2024-04-16T19:56:09.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.942+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.85.6 2024-04-16T19:56:09.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-16T19:56:09.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13/obj00000000000000000002 2024-04-16T19:56:09.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: entry->timestamp=2024-04-16T19:56:05.272884+0000 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.70.6 op=0 op_state=1 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000030]: success 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6:2 marker=00000000002.62.6 timestamp=2024-04-16T19:56:05.192133+0000 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000002]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000002[0] 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position=00000000002.70.6 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000006 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:04.962050+0000 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-16T19:56:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17/obj00000000000000000001 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: entry->timestamp=2024-04-16T19:56:05.047737+0000 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.34.6 op=0 op_state=1 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000006]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000006[0] 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position=00000000002.34.6 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000000 2024-04-16T19:56:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:05.203500+0000 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.46.6 op=0 op_state=1 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.943+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000002.46.6 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000012 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:05.033734+0000 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000012]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000012[0] 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18/obj00000000000000000007 2024-04-16T19:56:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: entry->timestamp=2024-04-16T19:56:05.059296+0000 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000007]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000007[0] 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000025]: success 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.944+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:2 marker=00000000001.50.6 timestamp=2024-04-16T19:56:05.128848+0000 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000000]: success 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14:2 marker=00000000002.46.6 timestamp=2024-04-16T19:56:05.203500+0000 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: success 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17:2 marker=00000000002.34.6 timestamp=2024-04-16T19:56:05.047737+0000 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000007]: success 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.947+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18:2 marker=00000000002.83.6 timestamp=2024-04-16T19:56:05.059296+0000 2024-04-16T19:56:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.948+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000015]: success 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.948+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:2 marker=00000000002.57.6 timestamp=2024-04-16T19:56:05.263823+0000 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.948+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000002]: success 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.949+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13:2 marker=00000000002.70.6 timestamp=2024-04-16T19:56:05.272884+0000 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.949+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000014]: success 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.949+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:2 marker=00000000001.85.6 timestamp=2024-04-16T19:56:05.181983+0000 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.951+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000012]: success 2024-04-16T19:56:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.951+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:2 marker=00000000002.68.6 timestamp=2024-04-16T19:56:05.033734+0000 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000005 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:05.252812+0000 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000005]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000005[0] 2024-04-16T19:56:09.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.978+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000003 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:04.984365+0000 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000003]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000003[0] 2024-04-16T19:56:09.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.979+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-16T19:56:09.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:09.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000019 2024-04-16T19:56:09.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:05.116996+0000 2024-04-16T19:56:09.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.36.6 op=0 op_state=1 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000019]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000019[0] 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000002.36.6 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000009 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:04.915495+0000 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.60.6 op=0 op_state=1 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000009]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000009[0] 2024-04-16T19:56:09.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.980+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000001.60.6 2024-04-16T19:56:09.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.982+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000019]: success 2024-04-16T19:56:09.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.982+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:2 marker=00000000002.36.6 timestamp=2024-04-16T19:56:05.116996+0000 2024-04-16T19:56:09.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.987+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000005]: success 2024-04-16T19:56:09.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.987+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:2 marker=00000000002.75.6 timestamp=2024-04-16T19:56:05.252812+0000 2024-04-16T19:56:09.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.990+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000003]: success 2024-04-16T19:56:09.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.990+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:2 marker=00000000002.66.6 timestamp=2024-04-16T19:56:04.984365+0000 2024-04-16T19:56:09.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.990+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000009]: success 2024-04-16T19:56:09.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.990+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10:2 marker=00000000001.60.6 timestamp=2024-04-16T19:56:04.915495+0000 2024-04-16T19:56:09.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.995+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000006]: success 2024-04-16T19:56:09.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:09.995+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:2 marker=00000000002.30.6 timestamp=2024-04-16T19:56:04.962050+0000 2024-04-16T19:56:10.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.031+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.031+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.032+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.033+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6/obj00000000000000000032 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: entry->timestamp=2024-04-16T19:56:05.928651+0000 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000032]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000032[0] 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-16T19:56:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.034+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000014 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:05.871282+0000 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000014]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000014[0] 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-16T19:56:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000006 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:05.860055+0000 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000006]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000006[0] 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000039 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:06.133025+0000 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.32.6 op=0 op_state=1 2024-04-16T19:56:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000039]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000039[0] 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000001.32.6 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000002 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:06.053689+0000 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.88.6 op=0 op_state=1 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000002]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000002[0] 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000002.88.6 2024-04-16T19:56:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.035+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket shard: 0 2024-04-16T19:56:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket shard: 0 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket shard: 0 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket shard: 0 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.036+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket shard: 0 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket shard: 0 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-16T19:56:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13/obj00000000000000000045 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: entry->timestamp=2024-04-16T19:56:05.652028+0000 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000045]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000045[0] 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000014]: success 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.037+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:3 marker=00000000002.49.6 timestamp=2024-04-16T19:56:05.871282+0000 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.038+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000006]: success 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.038+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10:3 marker=00000000002.73.6 timestamp=2024-04-16T19:56:05.860055+0000 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000013 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:05.981792+0000 2024-04-16T19:56:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.38.6 op=0 op_state=1 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000013]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000013[0] 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000002.38.6 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000012 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:06.143244+0000 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000012]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000012[0] 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000050 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:05.773069+0000 2024-04-16T19:56:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.36.6 op=0 op_state=1 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000050]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000050[0] 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000001.36.6 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17/obj00000000000000000001 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: entry->timestamp=2024-04-16T19:56:05.639477+0000 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.039+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19/obj00000000000000000021 2024-04-16T19:56:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: entry->timestamp=2024-04-16T19:56:06.075253+0000 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000021]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000021[0] 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000002]: success 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:3 marker=00000000002.88.6 timestamp=2024-04-16T19:56:06.053689+0000 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: parsed entry: id=00000000001.30.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000055 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: entry->timestamp=2024-04-16T19:56:05.918304+0000 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.30.6 op=0 op_state=1 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000055]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000055[0] 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position=00000000001.30.6 2024-04-16T19:56:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18/obj00000000000000000000 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: entry->timestamp=2024-04-16T19:56:06.121680+0000 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.040+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20/obj00000000000000000003 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: entry->timestamp=2024-04-16T19:56:05.993010+0000 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.60.6 op=0 op_state=1 2024-04-16T19:56:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000003]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000003[0] 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position=00000000002.60.6 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000032]: success 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6:3 marker=00000000001.51.6 timestamp=2024-04-16T19:56:05.928651+0000 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000013]: success 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.041+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14:3 marker=00000000002.38.6 timestamp=2024-04-16T19:56:05.981792+0000 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.042+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000012]: success 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.042+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:3 marker=00000000002.62.6 timestamp=2024-04-16T19:56:06.143244+0000 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.042+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000001]: success 2024-04-16T19:56:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.042+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17:3 marker=00000000002.60.6 timestamp=2024-04-16T19:56:05.639477+0000 2024-04-16T19:56:10.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000039]: success 2024-04-16T19:56:10.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:3 marker=00000000001.32.6 timestamp=2024-04-16T19:56:06.133025+0000 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000021]: success 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19:3 marker=00000000002.45.6 timestamp=2024-04-16T19:56:06.075253+0000 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000003]: success 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20:3 marker=00000000002.60.6 timestamp=2024-04-16T19:56:05.993010+0000 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000000]: success 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.043+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18:3 marker=00000000002.62.6 timestamp=2024-04-16T19:56:06.121680+0000 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.052+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000050]: success 2024-04-16T19:56:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.052+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:3 marker=00000000001.36.6 timestamp=2024-04-16T19:56:05.773069+0000 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000001.91.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000058 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:05.783920+0000 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.91.6 op=0 op_state=1 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000058]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000058[0] 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.91.6 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000026 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:06.004360+0000 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000046 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:05.714574+0000 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000026]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000026[0] 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000046]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000046[0] 2024-04-16T19:56:10.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.074+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-16T19:56:10.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-16T19:56:10.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000040 2024-04-16T19:56:10.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:05.762358+0000 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000040]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000040[0] 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000007 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:06.190249+0000 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000002.126.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000009 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:05.703028+0000 2024-04-16T19:56:10.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.126.6 op=0 op_state=1 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000007]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000007[0] 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000009]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000009[0] 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.076+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000002.126.6 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.077+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000026]: success 2024-04-16T19:56:10.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.077+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:3 marker=00000000002.75.6 timestamp=2024-04-16T19:56:06.004360+0000 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.078+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000007]: success 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.078+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:3 marker=00000000002.53.6 timestamp=2024-04-16T19:56:06.190249+0000 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.078+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000009]: success 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.078+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:3 marker=00000000002.126.6 timestamp=2024-04-16T19:56:05.703028+0000 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-16T19:56:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22/obj00000000000000000008 2024-04-16T19:56:10.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: entry->timestamp=2024-04-16T19:56:06.064631+0000 2024-04-16T19:56:10.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.36.6 op=0 op_state=1 2024-04-16T19:56:10.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000008]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000008[0] 2024-04-16T19:56:10.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.080+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position=00000000002.36.6 2024-04-16T19:56:10.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.084+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000058]: success 2024-04-16T19:56:10.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.084+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:3 marker=00000000001.91.6 timestamp=2024-04-16T19:56:05.783920+0000 2024-04-16T19:56:10.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.086+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000008]: success 2024-04-16T19:56:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.086+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22:3 marker=00000000002.36.6 timestamp=2024-04-16T19:56:06.064631+0000 2024-04-16T19:56:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.086+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000045]: success 2024-04-16T19:56:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.086+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13:3 marker=00000000001.47.6 timestamp=2024-04-16T19:56:05.652028+0000 2024-04-16T19:56:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.087+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000040]: success 2024-04-16T19:56:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.087+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:3 marker=00000000001.43.6 timestamp=2024-04-16T19:56:05.762358+0000 2024-04-16T19:56:10.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.092+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000055]: success 2024-04-16T19:56:10.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.092+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21:3 marker=00000000001.30.6 timestamp=2024-04-16T19:56:05.918304+0000 2024-04-16T19:56:10.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.121+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000046]: success 2024-04-16T19:56:10.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.121+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:3 marker=00000000001.43.6 timestamp=2024-04-16T19:56:05.714574+0000 2024-04-16T19:56:10.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.146+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.146+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.146+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.146+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.146+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.147+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.148+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket shard: 0 2024-04-16T19:56:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket shard: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket shard: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.149+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket shard: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000001.129.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000017 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.150+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:07.066288+0000 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.129.6 op=0 op_state=1 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000017]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000017[0] 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000001.129.6 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000015 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:07.018572+0000 2024-04-16T19:56:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: : injecting data sync error on key=obj00000000000000000015 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000049 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:07.009195+0000 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket shard: 0 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket shard: 0 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000015 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000049]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000049[0] 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000022 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:06.658243+0000 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-04-16T19:56:10.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.96.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000031 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.076567+0000 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.96.6 op=0 op_state=1 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000022]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000022[0] 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000002.94.6 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: : injecting data sync error on key=obj00000000000000000031 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.96.6 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-16T19:56:10.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000020 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:06.786078+0000 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000005 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:06.586090+0000 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.151+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.72.6 op=0 op_state=1 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000031 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000020]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000020[0] 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-16T19:56:10.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000005 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket shard: 0 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket shard: 0 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket shard: 0 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.152+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket shard: 0 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket shard: 0 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket shard: 0 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.153+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22/obj00000000000000000009 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: entry->timestamp=2024-04-16T19:56:06.807509+0000 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.38.6 op=0 op_state=1 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000009]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000009[0] 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position=00000000002.38.6 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000001 2024-04-16T19:56:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: entry->timestamp=2024-04-16T19:56:06.865039+0000 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.154+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.155+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: a sync operation returned error 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.155+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: backing out with sync_status=-5 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.155+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.155+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23/obj00000000000000000025 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: entry->timestamp=2024-04-16T19:56:06.726266+0000 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-16T19:56:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000025]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000025[0] 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26/obj00000000000000000014 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: entry->timestamp=2024-04-16T19:56:06.525668+0000 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.44.6 op=0 op_state=1 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000014]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000014[0] 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position=00000000002.44.6 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24/obj00000000000000000016 2024-04-16T19:56:10.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: entry->timestamp=2024-04-16T19:56:06.597192+0000 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25/obj00000000000000000000 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: entry->timestamp=2024-04-16T19:56:06.609074+0000 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: a sync operation returned error 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000016]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000016[0] 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:10.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27/obj00000000000000000030 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: entry->timestamp=2024-04-16T19:56:06.997962+0000 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: backing out with sync_status=-5 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27/obj00000000000000000030 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.156+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000020]: success 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14:4 marker=00000000002.67.6 timestamp=2024-04-16T19:56:06.786078+0000 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000009]: success 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22:4 marker=00000000002.38.6 timestamp=2024-04-16T19:56:06.807509+0000 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000001]: success 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21:4 marker=00000000002.84.6 timestamp=2024-04-16T19:56:06.865039+0000 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000017]: success 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.157+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:4 marker=00000000001.129.6 timestamp=2024-04-16T19:56:07.066288+0000 2024-04-16T19:56:10.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.158+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000025]: success 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.158+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23:4 marker=00000000002.96.6 timestamp=2024-04-16T19:56:06.726266+0000 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.158+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000022]: success 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.158+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:4 marker=00000000002.94.6 timestamp=2024-04-16T19:56:06.658243+0000 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000016]: success 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24:4 marker=00000000002.83.6 timestamp=2024-04-16T19:56:06.597192+0000 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000014]: success 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26:4 marker=00000000002.44.6 timestamp=2024-04-16T19:56:06.525668+0000 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000000]: success 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25:4 marker=00000000002.83.6 timestamp=2024-04-16T19:56:06.609074+0000 2024-04-16T19:56:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000049]: success 2024-04-16T19:56:10.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:4 marker=00000000001.40.6 timestamp=2024-04-16T19:56:07.009195+0000 2024-04-16T19:56:10.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.159+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000001.30.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-04-16T19:56:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000059 2024-04-16T19:56:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:07.086250+0000 2024-04-16T19:56:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.30.6 op=0 op_state=1 2024-04-16T19:56:10.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000059]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000059[0] 2024-04-16T19:56:10.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.189+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000001.30.6 2024-04-16T19:56:10.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-16T19:56:10.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2/obj00000000000000000027 2024-04-16T19:56:10.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: entry->timestamp=2024-04-16T19:56:06.678406+0000 2024-04-16T19:56:10.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.82.6 op=0 op_state=1 2024-04-16T19:56:10.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000027]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000027[0] 2024-04-16T19:56:10.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.190+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000001.82.6 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000006 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:06.926007+0000 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.45.6 op=0 op_state=1 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000006]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000006[0] 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000002.45.6 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000011 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:06.876626+0000 2024-04-16T19:56:10.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000002 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:07.134380+0000 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000002.125.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000004 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:06.514147+0000 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.125.6 op=0 op_state=1 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6/obj00000000000000000019 2024-04-16T19:56:10.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: entry->timestamp=2024-04-16T19:56:06.938480+0000 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.80.6 op=0 op_state=1 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000002]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000002[0] 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000004]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000004[0] 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000002.125.6 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000019]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000019[0] 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000002.80.6 2024-04-16T19:56:10.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.191+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000011 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000001.127.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000037 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:06.797888+0000 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.127.6 op=0 op_state=1 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20/obj00000000000000000010 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: entry->timestamp=2024-04-16T19:56:06.536621+0000 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000037]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000037[0] 2024-04-16T19:56:10.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000001.127.6 2024-04-16T19:56:10.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000010]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000010[0] 2024-04-16T19:56:10.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.192+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-16T19:56:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-16T19:56:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17/obj00000000000000000012 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: entry->timestamp=2024-04-16T19:56:06.853923+0000 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000012]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000012[0] 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18/obj00000000000000000003 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: entry->timestamp=2024-04-16T19:56:06.736959+0000 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19/obj00000000000000000042 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: entry->timestamp=2024-04-16T19:56:06.668920+0000 2024-04-16T19:56:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.78.6 op=0 op_state=1 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000008 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:06.949699+0000 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000003]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000003[0] 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000042]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000042[0] 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position=00000000001.78.6 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000008]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000008[0] 2024-04-16T19:56:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.193+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: a sync operation returned error 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000059]: success 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:4 marker=00000000001.30.6 timestamp=2024-04-16T19:56:07.086250+0000 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000004]: success 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:4 marker=00000000002.125.6 timestamp=2024-04-16T19:56:06.514147+0000 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: backing out with sync_status=-5 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000002]: success 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.194+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:4 marker=00000000002.68.6 timestamp=2024-04-16T19:56:07.134380+0000 2024-04-16T19:56:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000006]: success 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:4 marker=00000000002.45.6 timestamp=2024-04-16T19:56:06.926007+0000 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000019]: success 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6:4 marker=00000000002.80.6 timestamp=2024-04-16T19:56:06.938480+0000 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000010]: success 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.195+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20:4 marker=00000000002.51.6 timestamp=2024-04-16T19:56:06.536621+0000 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000008]: success 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:4 marker=00000000002.85.6 timestamp=2024-04-16T19:56:06.949699+0000 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000003]: success 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18:4 marker=00000000002.68.6 timestamp=2024-04-16T19:56:06.736959+0000 2024-04-16T19:56:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000012]: success 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17:4 marker=00000000002.85.6 timestamp=2024-04-16T19:56:06.853923+0000 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.196+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.197+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000027]: success 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.197+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2:4 marker=00000000001.82.6 timestamp=2024-04-16T19:56:06.678406+0000 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.199+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: a sync operation returned error 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.199+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: backing out with sync_status=-5 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.199+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.199+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000001.96.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-04-16T19:56:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000031 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.076567+0000 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.96.6 op=0 op_state=1 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000031[0] 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000001.96.6 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000037]: success 2024-04-16T19:56:10.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.200+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:4 marker=00000000001.127.6 timestamp=2024-04-16T19:56:06.797888+0000 2024-04-16T19:56:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-16T19:56:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000005 2024-04-16T19:56:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:06.586090+0000 2024-04-16T19:56:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.72.6 op=0 op_state=1 2024-04-16T19:56:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000005[0] 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.201+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.202+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket shard: 0 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.202+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.202+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000005]: success 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.202+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16:4 marker=00000000002.72.6 timestamp=2024-04-16T19:56:06.586090+0000 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.208+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000031]: success 2024-04-16T19:56:10.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.208+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:4 marker=00000000001.96.6 timestamp=2024-04-16T19:56:07.076567+0000 2024-04-16T19:56:10.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.234+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: a sync operation returned error 2024-04-16T19:56:10.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.234+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: backing out with sync_status=-5 2024-04-16T19:56:10.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.234+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.234+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.235+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.235+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:10.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.235+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-16T19:56:10.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000015 2024-04-16T19:56:10.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:07.018572+0000 2024-04-16T19:56:10.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-16T19:56:10.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000015[0] 2024-04-16T19:56:10.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.238+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-16T19:56:10.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.239+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000015]: success 2024-04-16T19:56:10.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.239+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10:4 marker=00000000002.77.6 timestamp=2024-04-16T19:56:07.018572+0000 2024-04-16T19:56:10.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.241+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000042]: success 2024-04-16T19:56:10.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.241+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19:4 marker=00000000001.78.6 timestamp=2024-04-16T19:56:06.668920+0000 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27/obj00000000000000000030 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: entry->timestamp=2024-04-16T19:56:06.997962+0000 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000030[0] 2024-04-16T19:56:10.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.244+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-16T19:56:10.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.245+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000030]: success 2024-04-16T19:56:10.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.245+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27:4 marker=00000000002.66.6 timestamp=2024-04-16T19:56:06.997962+0000 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000011 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:06.876626+0000 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000011[0] 2024-04-16T19:56:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.277+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-16T19:56:10.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.282+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000011]: success 2024-04-16T19:56:10.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.282+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:4 marker=00000000002.74.6 timestamp=2024-04-16T19:56:06.876626+0000 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.325+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.325+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.325+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:10.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.326+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.327+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: parsed entry: id=00000000001.132.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2/obj00000000000000000467 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: entry->timestamp=2024-04-16T19:56:07.604193+0000 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.132.6 op=0 op_state=1 2024-04-16T19:56:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000467]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000467[0] 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000001.132.6 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.328+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-16T19:56:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000005 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:07.809677+0000 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000056 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:07.988743+0000 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000005]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000005[0] 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000002.91.6 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000056]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000056[0] 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000317 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:08.050048+0000 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.58.6 op=0 op_state=1 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: : injecting data sync error on key=obj00000000000000000317 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000001.58.6 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6/obj00000000000000000023 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: entry->timestamp=2024-04-16T19:56:07.800020+0000 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000004 2024-04-16T19:56:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:07.475006+0000 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000000 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:08.162389+0000 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000317 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.329+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000023]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000023[0] 2024-04-16T19:56:10.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000004]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000004[0] 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket shard: 0 2024-04-16T19:56:10.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.330+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket shard: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket shard: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket shard: 0 2024-04-16T19:56:10.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.331+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket shard: 0 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket shard: 0 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket shard: 0 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket shard: 0 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket shard: 0 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket shard: 0 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13/obj00000000000000000008 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: entry->timestamp=2024-04-16T19:56:07.742029+0000 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.42.6 op=0 op_state=1 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket shard: 0 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket shard: 0 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket shard: 0 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000008]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000008[0] 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position=00000000002.42.6 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.332+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: sync status for source bucket shard: 0 2024-04-16T19:56:10.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000018 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:07.978222+0000 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000005]: success 2024-04-16T19:56:10.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:5 marker=00000000002.91.6 timestamp=2024-04-16T19:56:07.809677+0000 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.333+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000018 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000017 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:08.113561+0000 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000467]: success 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2:5 marker=00000000001.132.6 timestamp=2024-04-16T19:56:07.604193+0000 2024-04-16T19:56:10.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: a sync operation returned error 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000017 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17/obj00000000000000000162 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: entry->timestamp=2024-04-16T19:56:07.593936+0000 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: backing out with sync_status=-5 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000162]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000162[0] 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.334+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000004]: success 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:5 marker=00000000002.56.6 timestamp=2024-04-16T19:56:07.475006+0000 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25/obj00000000000000000283 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: entry->timestamp=2024-04-16T19:56:07.662247+0000 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: : injecting data sync error on key=obj00000000000000000283 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25/obj00000000000000000283 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000000]: success 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.335+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:5 marker=00000000002.74.6 timestamp=2024-04-16T19:56:08.162389+0000 2024-04-16T19:56:10.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19/obj00000000000000000003 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: entry->timestamp=2024-04-16T19:56:07.546269+0000 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000003]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000003[0] 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.336+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28/obj00000000000000000050 2024-04-16T19:56:10.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: entry->timestamp=2024-04-16T19:56:08.038269+0000 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.81.6 op=0 op_state=1 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000050]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000050[0] 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: listing bilog for incremental sync; position=00000000002.81.6 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30/obj00000000000000000133 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: entry->timestamp=2024-04-16T19:56:07.919087+0000 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.37.6 op=0 op_state=1 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27/obj00000000000000000061 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: entry->timestamp=2024-04-16T19:56:07.790107+0000 2024-04-16T19:56:10.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.48.6 op=0 op_state=1 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29/obj00000000000000000029 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: entry->timestamp=2024-04-16T19:56:07.615583+0000 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.73.6 op=0 op_state=1 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000008]: success 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13:5 marker=00000000002.42.6 timestamp=2024-04-16T19:56:07.742029+0000 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: a sync operation returned error 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: : injecting data sync error on key=obj00000000000000000133 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000056]: success 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:5 marker=00000000001.45.6 timestamp=2024-04-16T19:56:07.988743+0000 2024-04-16T19:56:10.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000061]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000061[0] 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position=00000000001.48.6 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000029]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000029[0] 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: listing bilog for incremental sync; position=00000000001.73.6 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30/obj00000000000000000133 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: a sync operation returned error 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: backing out with sync_status=-5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.337+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: backing out with sync_status=-5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000023]: success 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6:5 marker=00000000002.51.6 timestamp=2024-04-16T19:56:07.800020+0000 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000003]: success 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.338+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19:5 marker=00000000002.65.6 timestamp=2024-04-16T19:56:07.546269+0000 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.339+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.339+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000050]: success 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.339+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28:5 marker=00000000002.81.6 timestamp=2024-04-16T19:56:08.038269+0000 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.339+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.340+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.340+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.340+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.340+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000017 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:08.113561+0000 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.341+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000017 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.346+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000029]: success 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.346+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29:5 marker=00000000001.73.6 timestamp=2024-04-16T19:56:07.615583+0000 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.347+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000162]: success 2024-04-16T19:56:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.347+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17:5 marker=00000000001.40.6 timestamp=2024-04-16T19:56:07.593936+0000 2024-04-16T19:56:10.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.350+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000061]: success 2024-04-16T19:56:10.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.350+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27:5 marker=00000000001.48.6 timestamp=2024-04-16T19:56:07.790107+0000 2024-04-16T19:56:10.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000006 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:08.098626+0000 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000006]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000006[0] 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000002.140.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000019 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:07.683513+0000 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.140.6 op=0 op_state=1 2024-04-16T19:56:10.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2024-04-16T19:56:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000002.140.6 2024-04-16T19:56:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.368+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000019 2024-04-16T19:56:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000006]: success 2024-04-16T19:56:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:5 marker=00000000002.50.6 timestamp=2024-04-16T19:56:08.098626+0000 2024-04-16T19:56:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000002.144.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-16T19:56:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000022 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:08.173952+0000 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.144.6 op=0 op_state=1 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000022]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000022[0] 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000002.144.6 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.672240+0000 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000271 2024-04-16T19:56:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:07.486748+0000 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.63.6 op=0 op_state=1 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000271]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000271[0] 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.370+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.372+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000022]: success 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.372+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:5 marker=00000000002.144.6 timestamp=2024-04-16T19:56:08.173952+0000 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000007 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:07.462289+0000 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000007]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000007[0] 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.373+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: parsed entry: id=00000000001.93.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-16T19:56:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20/obj00000000000000000423 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: entry->timestamp=2024-04-16T19:56:07.859067+0000 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.93.6 op=0 op_state=1 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000423]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000423[0] 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position=00000000001.93.6 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000007]: success 2024-04-16T19:56:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.374+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:5 marker=00000000002.63.6 timestamp=2024-04-16T19:56:07.462289+0000 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18/obj00000000000000000016 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: entry->timestamp=2024-04-16T19:56:07.533811+0000 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000016]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000016[0] 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position=00000000002.61.6 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000271]: success 2024-04-16T19:56:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.375+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:5 marker=00000000001.63.6 timestamp=2024-04-16T19:56:07.486748+0000 2024-04-16T19:56:10.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:10.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23/obj00000000000000000001 2024-04-16T19:56:10.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: entry->timestamp=2024-04-16T19:56:07.929284+0000 2024-04-16T19:56:10.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000002 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: entry->timestamp=2024-04-16T19:56:07.869469+0000 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: parsed entry: id=00000000002.142.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24/obj00000000000000000011 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: entry->timestamp=2024-04-16T19:56:07.730307+0000 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.142.6 op=0 op_state=1 2024-04-16T19:56:10.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:entry[obj00000000000000000016]: success 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18:5 marker=00000000002.61.6 timestamp=2024-04-16T19:56:07.533811+0000 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000002]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000002[0] 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000011]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000011[0] 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.377+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position=00000000002.142.6 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.378+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: a sync operation returned error 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.378+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: backing out with sync_status=-5 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.378+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.378+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000423]: success 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20:5 marker=00000000001.93.6 timestamp=2024-04-16T19:56:07.859067+0000 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000002]: success 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21:5 marker=00000000002.53.6 timestamp=2024-04-16T19:56:07.869469+0000 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000011]: success 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24:5 marker=00000000002.142.6 timestamp=2024-04-16T19:56:07.730307+0000 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.379+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000001]: success 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23:5 marker=00000000002.63.6 timestamp=2024-04-16T19:56:07.929284+0000 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: a sync operation returned error 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: backing out with sync_status=-5 2024-04-16T19:56:10.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket shard: 0 2024-04-16T19:56:10.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.380+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000317 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:08.050048+0000 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.58.6 op=0 op_state=1 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000317[0] 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.381+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000001.58.6 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.382+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket shard: 0 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-16T19:56:10.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14/obj00000000000000000018 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: entry->timestamp=2024-04-16T19:56:07.978222+0000 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000018[0] 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.383+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.384+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: a sync operation returned error 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.384+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: backing out with sync_status=-5 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.384+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.384+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.386+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.386+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.386+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.386+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000317]: success 2024-04-16T19:56:10.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.386+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:5 marker=00000000001.58.6 timestamp=2024-04-16T19:56:08.050048+0000 2024-04-16T19:56:10.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.389+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:entry[obj00000000000000000018]: success 2024-04-16T19:56:10.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.389+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14:5 marker=00000000002.50.6 timestamp=2024-04-16T19:56:07.978222+0000 2024-04-16T19:56:10.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.410+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: a sync operation returned error 2024-04-16T19:56:10.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.410+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: backing out with sync_status=-5 2024-04-16T19:56:10.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.410+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.410+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.411+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.411+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:10.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.411+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.413+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: a sync operation returned error 2024-04-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.413+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: backing out with sync_status=-5 2024-04-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.413+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.413+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.415+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.415+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.415+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25/obj00000000000000000283 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: entry->timestamp=2024-04-16T19:56:07.662247+0000 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.40.6 op=0 op_state=1 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000283[0] 2024-04-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.422+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position=00000000001.40.6 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30/obj00000000000000000133 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: entry->timestamp=2024-04-16T19:56:07.919087+0000 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.37.6 op=0 op_state=1 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000133[0] 2024-04-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.425+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position=00000000001.37.6 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000017 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:08.113561+0000 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000017[0] 2024-04-16T19:56:10.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.428+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-16T19:56:10.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.429+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000283]: success 2024-04-16T19:56:10.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.429+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25:5 marker=00000000001.40.6 timestamp=2024-04-16T19:56:07.662247+0000 2024-04-16T19:56:10.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.429+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000017]: success 2024-04-16T19:56:10.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.429+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16:5 marker=00000000002.48.6 timestamp=2024-04-16T19:56:08.113561+0000 2024-04-16T19:56:10.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.437+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000133]: success 2024-04-16T19:56:10.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.437+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30:5 marker=00000000001.37.6 timestamp=2024-04-16T19:56:07.919087+0000 2024-04-16T19:56:10.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000002.140.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:10.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000019 2024-04-16T19:56:10.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:07.683513+0000 2024-04-16T19:56:10.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.140.6 op=0 op_state=1 2024-04-16T19:56:10.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000019[0] 2024-04-16T19:56:10.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.454+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000002.140.6 2024-04-16T19:56:10.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.456+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000019]: success 2024-04-16T19:56:10.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.456+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:5 marker=00000000002.140.6 timestamp=2024-04-16T19:56:07.683513+0000 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.672240+0000 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.458+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: a sync operation returned error 2024-04-16T19:56:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: backing out with sync_status=-5 2024-04-16T19:56:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.500+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.672240+0000 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.542+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.584+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: a sync operation returned error 2024-04-16T19:56:10.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.584+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: backing out with sync_status=-5 2024-04-16T19:56:10.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.584+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.584+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.585+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.585+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.585+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000009 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:07.672240+0000 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000009[0] 2024-04-16T19:56:10.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.627+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.629+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000009]: success 2024-04-16T19:56:10.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.629+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:5 marker=00000000002.75.6 timestamp=2024-04-16T19:56:07.672240+0000 2024-04-16T19:56:10.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: sync status for source bucket shard: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: sync status for source bucket shard: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: sync status for source bucket shard: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: sync status for source bucket shard: 0 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.674+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: sync status for source bucket shard: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: sync status for source bucket shard: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: sync status for source bucket shard: 0 2024-04-16T19:56:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: sync status for source bucket shard: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: sync status for source bucket shard: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: sync status for source bucket shard: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.675+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: sync status for source bucket shard: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: sync status for source bucket shard: 0 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:14]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3/obj00000000000000000001 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: entry->timestamp=2024-04-16T19:56:09.019247+0000 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.676+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000001]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000001[0] 2024-04-16T19:56:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: listing bilog for incremental sync; position=00000000002.52.6 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6/obj00000000000000000005 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: entry->timestamp=2024-04-16T19:56:08.690522+0000 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000005]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000005[0] 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: parsed entry: id=00000000001.89.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.677+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8/obj00000000000000000048 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: entry->timestamp=2024-04-16T19:56:08.582181+0000 2024-04-16T19:56:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.89.6 op=0 op_state=1 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: sync status for source bucket shard: 0 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000048]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000048[0] 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: listing bilog for incremental sync; position=00000000001.89.6 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4/obj00000000000000000033 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: entry->timestamp=2024-04-16T19:56:09.446759+0000 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000033]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000033[0] 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: listing bilog for incremental sync; position=00000000002.93.6 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2/obj00000000000000000011 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: entry->timestamp=2024-04-16T19:56:08.570442+0000 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5/obj00000000000000000065 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: entry->timestamp=2024-04-16T19:56:08.893723+0000 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.50.6 op=0 op_state=1 2024-04-16T19:56:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000011]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000011[0] 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9/obj00000000000000000015 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: entry->timestamp=2024-04-16T19:56:09.138761+0000 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: parsed entry: id=00000000002.95.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10/obj00000000000000000026 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: entry->timestamp=2024-04-16T19:56:09.506436+0000 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.95.6 op=0 op_state=1 2024-04-16T19:56:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000065]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000065[0] 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: listing bilog for incremental sync; position=00000000001.50.6 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: parsed entry: id=00000000002.106.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7/obj00000000000000000016 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: entry->timestamp=2024-04-16T19:56:08.822165+0000 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.106.6 op=0 op_state=1 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000015]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000015[0] 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: listing bilog for incremental sync; position=00000000002.61.6 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000026]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000026[0] 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: listing bilog for incremental sync; position=00000000002.95.6 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: parsed entry: id=00000000002.140.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-16T19:56:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11/obj00000000000000000030 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: entry->timestamp=2024-04-16T19:56:08.881894+0000 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.140.6 op=0 op_state=1 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000016]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000016[0] 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.678+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: listing bilog for incremental sync; position=00000000002.106.6 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000030]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000030[0] 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: listing bilog for incremental sync; position=00000000002.140.6 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: sync status for source bucket shard: 0 2024-04-16T19:56:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.679+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: sync status for source bucket shard: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: sync status for source bucket shard: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: sync status for source bucket shard: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: sync status for source bucket shard: 0 2024-04-16T19:56:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.680+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: sync status for source bucket shard: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:entry[obj00000000000000000001]: success 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:3:6 marker=00000000002.52.6 timestamp=2024-04-16T19:56:09.019247+0000 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:entry[obj00000000000000000005]: success 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:6:6 marker=00000000002.85.6 timestamp=2024-04-16T19:56:08.690522+0000 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: sync status for source bucket shard: 0 2024-04-16T19:56:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: sync status for source bucket shard: 0 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket shard: 0 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:18]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: sync status for source bucket shard: 0 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: sync status for source bucket shard: 0 2024-04-16T19:56:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.681+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: sync status for source bucket shard: 0 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.682+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:entry[obj00000000000000000011]: success 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:2:6 marker=00000000002.83.6 timestamp=2024-04-16T19:56:08.570442+0000 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:entry[obj00000000000000000033]: success 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:4:6 marker=00000000002.93.6 timestamp=2024-04-16T19:56:09.446759+0000 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:36]: sync status for source bucket shard: 0 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:36]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:36]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.683+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: sync status for source bucket shard: 0 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:entry[obj00000000000000000015]: success 2024-04-16T19:56:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:9:6 marker=00000000002.61.6 timestamp=2024-04-16T19:56:09.138761+0000 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15/obj00000000000000000024 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: entry->timestamp=2024-04-16T19:56:09.386449+0000 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.93.6 op=0 op_state=1 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000024]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000024[0] 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: listing bilog for incremental sync; position=00000000002.93.6 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: parsed entry: id=00000000002.142.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17/obj00000000000000000002 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: entry->timestamp=2024-04-16T19:56:09.202872+0000 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.142.6 op=0 op_state=1 2024-04-16T19:56:10.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20/obj00000000000000000006 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.684+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: entry->timestamp=2024-04-16T19:56:08.968850+0000 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16/obj00000000000000000012 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: entry->timestamp=2024-04-16T19:56:09.078451+0000 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000002]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000002[0] 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: listing bilog for incremental sync; position=00000000002.142.6 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000006]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000006[0] 2024-04-16T19:56:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:entry[obj00000000000000000030]: success 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:11:6 marker=00000000002.140.6 timestamp=2024-04-16T19:56:08.881894+0000 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:entry[obj00000000000000000026]: success 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:10:6 marker=00000000002.95.6 timestamp=2024-04-16T19:56:09.506436+0000 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000012]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000012[0] 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.685+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19/obj00000000000000000045 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: entry->timestamp=2024-04-16T19:56:09.275079+0000 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.72.6 op=0 op_state=1 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000017 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: entry->timestamp=2024-04-16T19:56:09.215043+0000 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000045]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000045[0] 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:entry[obj00000000000000000048]: success 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:8:6 marker=00000000001.89.6 timestamp=2024-04-16T19:56:08.582181+0000 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:entry[obj00000000000000000016]: success 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:7:6 marker=00000000002.106.6 timestamp=2024-04-16T19:56:08.822165+0000 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-16T19:56:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:entry[obj00000000000000000065]: success 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:5:6 marker=00000000001.50.6 timestamp=2024-04-16T19:56:08.893723+0000 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1/obj00000000000000000000 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: entry->timestamp=2024-04-16T19:56:08.630368+0000 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000017 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000000]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000000[0] 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-16T19:56:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22/obj00000000000000000007 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: entry->timestamp=2024-04-16T19:56:09.263547+0000 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000007]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000007[0] 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.686+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27/obj00000000000000000053 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: entry->timestamp=2024-04-16T19:56:08.761455+0000 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000053]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000053[0] 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.687+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-04-16T19:56:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30/obj00000000000000000036 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: entry->timestamp=2024-04-16T19:56:08.750923+0000 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.57.6 op=0 op_state=1 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000036]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000036[0] 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: listing bilog for incremental sync; position=00000000001.57.6 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31/obj00000000000000000003 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: entry->timestamp=2024-04-16T19:56:08.557478+0000 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:entry[obj00000000000000000003]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000003[0] 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.688+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-16T19:56:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32/obj00000000000000000014 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: entry->timestamp=2024-04-16T19:56:09.459926+0000 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:entry[obj00000000000000000014]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000014[0] 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12/obj00000000000000000062 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: entry->timestamp=2024-04-16T19:56:08.946102+0000 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.68.6 op=0 op_state=1 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000062]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000062[0] 2024-04-16T19:56:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.689+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: listing bilog for incremental sync; position=00000000001.68.6 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26/obj00000000000000000021 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: entry->timestamp=2024-04-16T19:56:08.810331+0000 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:entry[obj00000000000000000006]: success 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:20:6 marker=00000000002.56.6 timestamp=2024-04-16T19:56:08.968850+0000 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T19:56:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 0 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: ERROR: failed to sync object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26/obj00000000000000000021 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:entry[obj00000000000000000012]: success 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:16:6 marker=00000000002.59.6 timestamp=2024-04-16T19:56:09.078451+0000 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23/obj00000000000000000039 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: entry->timestamp=2024-04-16T19:56:08.870110+0000 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.82.6 op=0 op_state=1 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.691+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000039]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000039[0] 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.692+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: listing bilog for incremental sync; position=00000000002.82.6 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.692+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:entry[obj00000000000000000007]: success 2024-04-16T19:56:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.692+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:22:6 marker=00000000002.48.6 timestamp=2024-04-16T19:56:09.263547+0000 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:entry[obj00000000000000000003]: success 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:31:6 marker=00000000002.63.6 timestamp=2024-04-16T19:56:08.557478+0000 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24/obj00000000000000000027 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: entry->timestamp=2024-04-16T19:56:09.337160+0000 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000027]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000027[0] 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:entry[obj00000000000000000000]: success 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:1:6 marker=00000000002.48.6 timestamp=2024-04-16T19:56:08.630368+0000 2024-04-16T19:56:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:entry[obj00000000000000000024]: success 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.693+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:15:6 marker=00000000002.93.6 timestamp=2024-04-16T19:56:09.386449+0000 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28/obj00000000000000000020 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: entry->timestamp=2024-04-16T19:56:08.642118+0000 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000020]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000020[0] 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: listing bilog for incremental sync; position=00000000002.91.6 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25/obj00000000000000000010 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: entry->timestamp=2024-04-16T19:56:08.702277+0000 2024-04-16T19:56:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.75.6 op=0 op_state=1 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000010]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000010[0] 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: listing bilog for incremental sync; position=00000000002.75.6 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:entry[obj00000000000000000045]: success 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:19:6 marker=00000000002.72.6 timestamp=2024-04-16T19:56:09.275079+0000 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:entry[obj00000000000000000002]: success 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.694+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:17:6 marker=00000000002.142.6 timestamp=2024-04-16T19:56:09.202872+0000 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.695+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:entry[obj00000000000000000014]: success 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.695+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:32:6 marker=00000000002.50.6 timestamp=2024-04-16T19:56:09.459926+0000 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.695+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:entry[obj00000000000000000039]: success 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.695+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:23:6 marker=00000000002.82.6 timestamp=2024-04-16T19:56:08.870110+0000 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:entry[obj00000000000000000027]: success 2024-04-16T19:56:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:24:6 marker=00000000002.67.6 timestamp=2024-04-16T19:56:09.337160+0000 2024-04-16T19:56:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:entry[obj00000000000000000020]: success 2024-04-16T19:56:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:28:6 marker=00000000002.91.6 timestamp=2024-04-16T19:56:08.642118+0000 2024-04-16T19:56:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:entry[obj00000000000000000010]: success 2024-04-16T19:56:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.696+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:25:6 marker=00000000002.75.6 timestamp=2024-04-16T19:56:08.702277+0000 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:entry[obj00000000000000000036]: success 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:30:6 marker=00000000001.57.6 timestamp=2024-04-16T19:56:08.750923+0000 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:entry[obj00000000000000000053]: success 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:27:6 marker=00000000002.65.6 timestamp=2024-04-16T19:56:08.761455+0000 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:entry[obj00000000000000000062]: success 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.698+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:12:6 marker=00000000001.68.6 timestamp=2024-04-16T19:56:08.946102+0000 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-04-16T19:56:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0/obj00000000000000000060 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: entry->timestamp=2024-04-16T19:56:09.398109+0000 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000060]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000060[0] 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.717+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13/obj00000000000000000009 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: entry->timestamp=2024-04-16T19:56:08.956986+0000 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000009]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000009[0] 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-16T19:56:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:entry[obj00000000000000000060]: success 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.722+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:0:6 marker=00000000001.47.6 timestamp=2024-04-16T19:56:09.398109+0000 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.724+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:entry[obj00000000000000000009]: success 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.724+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:13:6 marker=00000000002.56.6 timestamp=2024-04-16T19:56:08.956986+0000 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33/obj00000000000000000019 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: entry->timestamp=2024-04-16T19:56:09.323089+0000 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:entry[obj00000000000000000019]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000019[0] 2024-04-16T19:56:10.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.726+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-16T19:56:10.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.728+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:entry[obj00000000000000000019]: success 2024-04-16T19:56:10.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.728+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:33:6 marker=00000000002.74.6 timestamp=2024-04-16T19:56:09.323089+0000 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29/obj00000000000000000004 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: entry->timestamp=2024-04-16T19:56:09.152060+0000 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000004]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000004[0] 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-16T19:56:10.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35/obj00000000000000000018 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: entry->timestamp=2024-04-16T19:56:09.090568+0000 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:entry[obj00000000000000000018]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000018[0] 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.729+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.731+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:entry[obj00000000000000000004]: success 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.731+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:29:6 marker=00000000002.62.6 timestamp=2024-04-16T19:56:09.152060+0000 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.731+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:entry[obj00000000000000000018]: success 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.731+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:35:6 marker=00000000002.77.6 timestamp=2024-04-16T19:56:09.090568+0000 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: a sync operation returned error 2024-04-16T19:56:10.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: backing out with sync_status=-5 2024-04-16T19:56:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: sync status for source bucket shard: 0 2024-04-16T19:56:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.732+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-04-16T19:56:10.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34/obj00000000000000000057 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: entry->timestamp=2024-04-16T19:56:09.031246+0000 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000001.70.6 op=0 op_state=1 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:entry[obj00000000000000000057]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000057[0] 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: listing bilog for incremental sync; position=00000000001.70.6 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: a sync operation returned error 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: backing out with sync_status=-5 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: incremental sync on bucket failed, retcode=-5 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.734+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: ERROR: incremental sync failed. error: -5 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.735+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.735+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: sync status for source bucket shard: 0 2024-04-16T19:56:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.735+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position= 2024-04-16T19:56:10.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.738+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:entry[obj00000000000000000057]: success 2024-04-16T19:56:10.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.738+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:34:6 marker=00000000001.70.6 timestamp=2024-04-16T19:56:09.031246+0000 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21/obj00000000000000000017 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: entry->timestamp=2024-04-16T19:56:09.215043+0000 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.53.6 op=0 op_state=1 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000017[0] 2024-04-16T19:56:10.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.775+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: listing bilog for incremental sync; position=00000000002.53.6 2024-04-16T19:56:10.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.777+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:entry[obj00000000000000000017]: success 2024-04-16T19:56:10.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.777+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:21:6 marker=00000000002.53.6 timestamp=2024-04-16T19:56:09.215043+0000 2024-04-16T19:56:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-16T19:56:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: syncing object: yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26/obj00000000000000000021 2024-04-16T19:56:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: entry->timestamp=2024-04-16T19:56:08.810331+0000 2024-04-16T19:56:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-16T19:56:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 5 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-51[b482a2a9-ef36-488b-9666-ada656453328.4812.20])/obj00000000000000000021[0] 2024-04-16T19:56:10.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.778+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-16T19:56:10.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.779+0000 7f074c131800 10 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:entry[obj00000000000000000021]: success 2024-04-16T19:56:10.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:10.779+0000 7f074c131800 20 RGW-SYNC:bucket[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20<-yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]:inc_sync[yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26]: updating marker marker_oid=bucket.sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20:26:6 marker=00000000002.55.6 timestamp=2024-04-16T19:56:08.810331+0000 2024-04-16T19:56:10.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:10.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:10.850 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:10.905 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:10.905 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:10.981 INFO:teuthology.orchestra.run.smithi026.stdout:[{"name":"obj00000000000000000000","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.630368Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18219318825734644172","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000001","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.019247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15263964127302499434","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000002","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.202872Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14230945245476213330","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000003","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.557478Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14424479148110183439","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000004","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.152060Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.10796835517022261754","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000005","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.690522Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5076123965725800926","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000006","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.968850Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.10525531824671049606","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000007","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.263547Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.329166309635467224","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000008","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.742029Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18320154155456310100","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000009","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.956986Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18209536467245519966","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000010","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.702277Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.9629371012107389414","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000011","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.570442Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17826093872525044309","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000012","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.078451Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6644868180502354330","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000013","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:05.981792Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.774607276797875091","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000014","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.459926Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.258293872611093000","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000015","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.138761Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14554204101342675319","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000016","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.822165Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16743644001363616271","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000017","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.215043Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18032887602606870920","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000018","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.090568Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17429441186467170018","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000019","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.323089Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6673237791073470432","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000020","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.642118Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.3532424588433791021","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000021","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.810331Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.10948530009780433572","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000022","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.173952Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15564586552410088700","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000023","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.800020Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.4267021438541779697","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000024","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.386449Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6915847935295217340","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000025","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:06.726266Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.7453824576723016655","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000026","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.506436Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6482123036962453812","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000027","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.337160Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17943803575754781702","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000028","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:04.542244Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.11569125436818638750","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000029","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.615583Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8144403983340648644","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000030","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.881894Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14824671189794861386","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000031","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.076567Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.3233415135884047720","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000032","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:05.928651Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.12902522802733539685","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000033","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.446759Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.13936066211545521458","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000036","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.750923Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.892678762986444841","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000037","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:06.797888Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17417553029836224977","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000039","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.870110Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6113552509950042955","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000040","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:05.762358Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8063463125734290637","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000042","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:06.668920Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17260419451010380383","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000045","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.275079Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17524360077277952165","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000046","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:05.714574Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.9965357400926924154","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000048","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.582181Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15548679900588718490","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000049","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.009195Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.10081940544746437794","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000050","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.038269Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.4862224002024527743","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000053","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.761455Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.13315939272468640494","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000055","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:05.918304Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14158794676312142160","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000056","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.988743Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15378308888449724472","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000057","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.031246Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.2143665000142243654","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000058","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:05.783920Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.3230280319950095651","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000059","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.086250Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6229170081730825599","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000060","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.398109Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.9539453430806900569","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000061","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.790107Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.13632194513024802516","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000062","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.946102Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1012996951264074921","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000065","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.893723Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8862126876555670543","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000076","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:04.243458Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1604677837343714794","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000133","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.919087Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.7340681099411777958","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000162","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.593936Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5617706178809405311","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000271","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.486748Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.7323154319561052387","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000283","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.662247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15741442294357377048","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000317","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.050048Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17758552000395914010","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000423","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.859067Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.9766832367934807963","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000467","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.604193Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1683716370328673263","flags":0,"pending_map":[],"versioned_epoch":0}] 2024-04-16T19:56:10.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:10.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-51', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:10.982 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-51 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:11.033 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:11.033 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:11.103 INFO:teuthology.orchestra.run.smithi119.stdout:[{"name":"obj00000000000000000000","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.630368Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ACfsmKSgRBq6wILSHFz6Xk2YNeXJRuc","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000001","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.019247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_eWFhK2wly0juF0jh3UxqH1Kk8tdDLOn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000002","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.202872Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dw641QSdR4cqlDRmgtl-fJvZCHraF5J","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000003","instance":"","ver":{"pool":9,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.557478Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_miXL43hu6AvljtsHSuflJ-mhxNLjmcc","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000004","instance":"","ver":{"pool":9,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.152060Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_zf-lBpTnGM6v2UDSVVNm_9rpMAz1EmW","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000005","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.690522Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_cxlCyz3J9T_UcAF-FS6XXwOZjxPyvwv","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000006","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.968850Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_hsEqPw78yNDeFSVW9RTRqMKAiVq3m4j","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000007","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.263547Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_-tBiKr6_D6UAg5hML9s2pEId72uxliM","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000008","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.742029Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_XqzIArFm_0t-2HDH04HDeLuqXJAHQUc","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000009","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.956986Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_MS7mb3z6RebIpHwz_03bqU3d1saDnqQ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000010","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.702277Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Oem-Yeev57gvQDWp1gZNpXec1CJHl8L","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000011","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.570442Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_GCpKz4MG8i2DbsDuS7o2vwWhx9r-dt0","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000012","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.078451Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_y95ec8oIM0TATCSoPrd3bPAgqBXG6ef","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000013","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:05.981792Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_v9qNiXBZbU_AkMS5dWpofuTwH8K_7PR","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000014","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.459926Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uyUdtTK4JwWmgsvYzpuPZW_jlJ3doJn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000015","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.138761Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_cBfEjcmz5gwS8wkIcAMYGjkjq29rEbi","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000016","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.822165Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_7juDK1RSNXI4gj-wZomsEXWv-2sJFoA","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000017","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.215043Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_RHY-4TTx7xeRl67f2XE-yA8Vy6ASvWp","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000018","instance":"","ver":{"pool":9,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.090568Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_E6LxY5-JIFZa7evyytupM_gP93iMeY9","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000019","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.323089Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_8POlztBsiLqGA5_CFnT2W5camj9D6b8","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000020","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.642118Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dTW7mMjf9OmsS2gubLet_pt11v538lS","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000021","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.810331Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ylSD-cE-Trpr46gHrX04EX6gJBr05Ym","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000022","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.173952Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_dkFfeMEP8n5Z48sVn1EojDWhXR8V4N2","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000023","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.800020Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ZjMStz72-v_3fgIcreAHBspFB-99yw5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000024","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.386449Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_as33S1T3XPymhyY5BDNwNdLAG48lY7y","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000025","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:06.726266Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Ci2qoBncIApx_-hrn6bFmTX3O9pzmxN","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000026","instance":"","ver":{"pool":9,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.506436Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_eBj5URWLu3JEiKoC3IB1y2VeLTp35Q4","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000027","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:09.337160Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Ky1ofLM7eRH1Wx-e7wjVz1hWE8H0Rul","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000028","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:04.542244Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_fTz3GNJoSUFTdl-ni0sj3-65ohayIl9","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000029","instance":"","ver":{"pool":9,"epoch":2},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.615583Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_bi7q9aN0qZE-3Egn8qp1By0sPi1U7vn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000030","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.881894Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_YfG742-oCvV1j_iojCltEuDTig2aJsD","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000031","instance":"","ver":{"pool":9,"epoch":11},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.076567Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_VF_mitO_rhfSzmJJ_ewFctUlQGJGsY5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000032","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:05.928651Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Nw3WpCxP2ZWtF6oJaRk-afvSqshfedH","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000033","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.446759Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__xENeeuY7NWFZEtJW5xcS7_ofKSB2dV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000036","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.750923Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_e4MRnJejnNG9S3mMwbwQ2kNbgqnxbD0","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000037","instance":"","ver":{"pool":9,"epoch":1},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:06.797888Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_GVUpjMlkwxCsdcrSscSZNQmt7btpdnz","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000039","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:08.870110Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_MOoHFmvCj5AkYLOVjz_lEsJJcJLb3Xm","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000040","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:05.762358Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_LQBxEG3iQFNjoidtUnu7onmakvDL2pD","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000042","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:06.668920Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_r2338HKuAOhocRzQr1cCWyMTC3-3BEt","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000045","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:09.275079Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_N6JWN2-MuvqCUyTJd9tWavJbZ6oa_4s","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000046","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:05.714574Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Cek9wB3iIjR0eFBPAyfdKeuPw21VNk2","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000048","instance":"","ver":{"pool":9,"epoch":14},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.582181Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_mdvze86641lXN7Hi23DlQw09822BT-H","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000049","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.009195Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_02Q2i_F9aePSSQj8ltcxhIEq7Ul3diB","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000050","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.038269Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_AjhBuFtRmKeICbwW5fjCps5pmfu9Opo","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000053","instance":"","ver":{"pool":9,"epoch":15},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.761455Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_YRPBKxFHOcW6JPQQ_yP7M0bWkunLoOQ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000055","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:05.918304Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_59UBgeVGMZz5qdZfn7h-arDe9THwKZ5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000056","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.988743Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__eZ-IcVCLP9995irB38HeOEyg7N2AHy","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000057","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:09.031246Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_UiLirYctcCuicknfchzOVo2fKUkVaZG","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000058","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:05.783920Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ERHzguHMXvMZWLKkctMxyBv4oryNJUY","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000059","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.086250Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_MkWImrRdDgkW9GREDYsTgIu1uk4JEBV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000060","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:09.398109Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ZiA_yMin298DrDiii72kQs7uKArLvot","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000061","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.790107Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_4sLk8jlFH-6FRth8hsNxe5KH-zRgLyp","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000062","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:08.946102Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TgUilmJ_gAk1q8KZDUGmdV-hxyIJqQj","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000065","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:08.893723Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Tl4evB5psSSL9xhdPERWBaZWXufa_Kp","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000076","instance":"","ver":{"pool":9,"epoch":34},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:04.243458Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_VDZ3v5U5qdK-tYiW9jUbTg03tCMgSKZ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000133","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.919087Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_r3NE-EQb-7tm4HKQs-x0cPV-IoIAdYV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000162","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.593936Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_XJScbBI6RtyP8tsUBVj-7qneogzf9Tl","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000271","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:56:07.486748Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_NH3JntM_DipMAdcnexlSJP0CF4seL2H","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000283","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:56:07.662247Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_G-uk3NwO3nTw9b3NvLXDtTPGHekgU1N","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000317","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:08.050048Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_AMPfZ_N36GLG6rzZ6VnLTT0VFNmccsz","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000423","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:56:07.859067Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_19SQ87bKQEzIihS4nM-kDXVDmVQrojR","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj00000000000000000467","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:56:07.604193Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_3M0QJtIivRXT1O28_TcuMPlFHNaZy_H","flags":0,"pending_map":[],"versioned_epoch":0}] 2024-04-16T19:56:11.103 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T19:56:11.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T19:56:11.106 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T19:56:11.106 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T19:56:11.125 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:56:11.126 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:56:11.126 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T19:56:11.126 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T19:56:11.279 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T19:56:11.279+0000 7f2ab0e32880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T19:56:12.127 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:12.150 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T19:56:12.150 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T19:56:12.152 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T19:56:12.152 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T19:56:12.152 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-51 2024-04-16T19:56:12.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:56:12.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:56:12.153 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-51 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:56:12.252 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:12.252 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:12.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:12.314+0000 7f3ee18e4800 1 bucket sync caught up with source: 2024-04-16T19:56:12.315 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.47.6, 00000000002.48.6, 00000000002.83.6, 00000000002.52.6, 00000000002.93.6, 00000000001.50.6, 00000000002.85.6, 00000000002.106.6, 00000000001.89.6, 00000000002.61.6, 00000000002.95.6, 00000000002.140.6, 00000000001.68.6, 00000000002.56.6, , 00000000002.93.6, 00000000002.59.6, 00000000002.142.6, , 00000000002.72.6, 00000000002.56.6, 00000000002.53.6, 00000000002.48.6, 00000000002.82.6, 00000000002.67.6, 00000000002.75.6, 00000000002.55.6, 00000000002.65.6, 00000000002.91.6, 00000000002.62.6, 00000000001.57.6, 00000000002.63.6, 00000000002.50.6, 00000000002.74.6, 00000000001.70.6, 00000000002.77.6, ] 2024-04-16T19:56:12.315 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.47.6, 00000000002.48.6, 00000000002.83.6, 00000000002.52.6, 00000000002.93.6, 00000000001.50.6, 00000000002.85.6, 00000000002.106.6, 00000000001.89.6, 00000000002.61.6, 00000000002.95.6, 00000000002.140.6, 00000000001.68.6, 00000000002.56.6, , 00000000002.93.6, 00000000002.59.6, 00000000002.142.6, , 00000000002.72.6, 00000000002.56.6, 00000000002.53.6, 00000000002.48.6, 00000000002.82.6, 00000000002.67.6, 00000000002.75.6, 00000000002.55.6, 00000000002.65.6, 00000000002.91.6, 00000000002.62.6, 00000000001.57.6, 00000000002.63.6, 00000000002.50.6, 00000000002.74.6, 00000000001.70.6, 00000000002.77.6, ] 2024-04-16T19:56:12.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:12.314+0000 7f3ee18e4800 0 bucket checkpoint complete 2024-04-16T19:56:12.336 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-51 2024-04-16T19:56:12.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:56:12.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-51', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:56:12.337 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-51 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:56:12.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:12.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:12.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:56:12.465+0000 7f4ba120f800 1 waiting for incremental sync to catch up: 2024-04-16T19:56:12.465 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000005.74.6, 00000000007.144.6, 00000000001.144.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000001.147.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000001.153.6] 2024-04-16T19:56:12.465 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000005.74.6, 00000000007.144.6, 00000000006.154.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000008.161.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000004.159.6] 2024-04-16T19:56:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:56:17.466+0000 7f4ba120f800 1 waiting for incremental sync to catch up: 2024-04-16T19:56:17.466 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000005.74.6, 00000000007.144.6, 00000000001.144.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000001.147.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000001.153.6] 2024-04-16T19:56:17.466 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000005.74.6, 00000000007.144.6, 00000000006.154.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000008.161.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000004.159.6] 2024-04-16T19:56:22.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:56:22.467+0000 7f4ba120f800 1 bucket sync caught up with source: 2024-04-16T19:56:22.468 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000005.74.6, 00000000007.144.6, 00000000006.154.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000008.161.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000004.159.6] 2024-04-16T19:56:22.468 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000005.74.6, 00000000007.144.6, 00000000006.154.6, 00000000010.129.6, 00000000005.234.6, 00000000007.224.6, 00000000008.161.6, 00000000006.197.6, 00000000007.195.6, 00000000007.189.6, 00000000004.159.6] 2024-04-16T19:56:22.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:56:22.467+0000 7f4ba120f800 0 bucket checkpoint complete 2024-04-16T19:56:22.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-51 zones={test-zone2, test-zone1} 2024-04-16T19:56:22.496 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:22.496 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:22.496 DEBUG:boto:Method: HEAD 2024-04-16T19:56:22.496 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:22.496 DEBUG:boto:Data: 2024-04-16T19:56:22.497 DEBUG:boto:Headers: {} 2024-04-16T19:56:22.497 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:22.497 DEBUG:boto:Port: 8000 2024-04-16T19:56:22.497 DEBUG:boto:Params: {} 2024-04-16T19:56:22.497 DEBUG:boto:Token: None 2024-04-16T19:56:22.497 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:22 GMT /yrjefh-51/ 2024-04-16T19:56:22.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE= 2024-04-16T19:56:22.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE=', 'Content-Length': '0'} 2024-04-16T19:56:22.499 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-16T19:56:22.499 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:56:22.955 DEBUG:boto:Token: None 2024-04-16T19:56:22.955 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:22 GMT /yrjefh-51/ 2024-04-16T19:56:22.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE= 2024-04-16T19:56:22.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE=', 'Content-Length': '0'} 2024-04-16T19:56:22.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '62'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ea2a1111370f1d2-00661ed7e6-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:22.965 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:22.965 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:22.965 DEBUG:boto:Method: HEAD 2024-04-16T19:56:22.965 DEBUG:boto:Path: /yrjefh-51/ 2024-04-16T19:56:22.965 DEBUG:boto:Data: 2024-04-16T19:56:22.966 DEBUG:boto:Headers: {} 2024-04-16T19:56:22.966 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:22.966 DEBUG:boto:Port: 8000 2024-04-16T19:56:22.966 DEBUG:boto:Params: {} 2024-04-16T19:56:22.966 DEBUG:boto:Token: None 2024-04-16T19:56:22.967 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:22 GMT /yrjefh-51/ 2024-04-16T19:56:22.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE= 2024-04-16T19:56:22.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:riE77/J711YDd6470Qc2eNoscdE=', 'Content-Length': '0'} 2024-04-16T19:56:22.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '62'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b40adee69b3727fe-00661ed7e6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:22.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:56:22.971 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:22.971 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:22.972 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.972 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.972 DEBUG:boto:Method: GET 2024-04-16T19:56:22.972 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.972 DEBUG:boto:Data: 2024-04-16T19:56:22.972 DEBUG:boto:Headers: {} 2024-04-16T19:56:22.972 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:22.972 DEBUG:boto:Port: 8000 2024-04-16T19:56:22.973 DEBUG:boto:Params: {} 2024-04-16T19:56:22.973 DEBUG:boto:Token: None 2024-04-16T19:56:22.973 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:22 GMT /yrjefh-51/?versions 2024-04-16T19:56:22.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9c9nwOp3PN5S8eKeXtTVnVmEJM= 2024-04-16T19:56:22.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9c9nwOp3PN5S8eKeXtTVnVmEJM=', 'Content-Length': '0'} 2024-04-16T19:56:22.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5ebbcd1b7c6d27f-00661ed7e6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:22.978 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000000nulltrue2024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000001nulltrue2024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000007nulltrue2024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000008nulltrue2024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000009nulltrue2024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000014nulltrue2024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000015nulltrue2024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000016nulltrue2024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000017nulltrue2024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000018nulltrue2024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000019nulltrue2024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000020nulltrue2024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000021nulltrue2024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000022nulltrue2024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000023nulltrue2024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000024nulltrue2024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000025nulltrue2024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000026nulltrue2024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000027nulltrue2024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000028nulltrue2024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000029nulltrue2024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000030nulltrue2024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000031nulltrue2024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000032nulltrue2024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000033nulltrue2024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000036nulltrue2024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000037nulltrue2024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000039nulltrue2024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000040nulltrue2024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000042nulltrue2024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000045nulltrue2024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000046nulltrue2024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000048nulltrue2024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000049nulltrue2024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000050nulltrue2024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000053nulltrue2024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000055nulltrue2024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000056nulltrue2024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000057nulltrue2024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000058nulltrue2024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000059nulltrue2024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000060nulltrue2024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000061nulltrue2024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000062nulltrue2024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000065nulltrue2024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000076nulltrue2024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000133nulltrue2024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000162nulltrue2024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000271nulltrue2024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000283nulltrue2024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000317nulltrue2024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000423nulltrue2024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000467nulltrue2024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:22.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-16T19:56:22.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-16T19:56:22.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-16T19:56:22.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-16T19:56:22.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-16T19:56:22.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-16T19:56:22.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-16T19:56:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-16T19:56:22.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-16T19:56:22.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-16T19:56:22.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-16T19:56:22.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-16T19:56:22.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-16T19:56:22.989 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:56:22.989 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:22.989 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:22.989 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.990 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.990 DEBUG:boto:Method: GET 2024-04-16T19:56:22.990 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:22.990 DEBUG:boto:Data: 2024-04-16T19:56:22.990 DEBUG:boto:Headers: {} 2024-04-16T19:56:22.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:22.990 DEBUG:boto:Port: 8000 2024-04-16T19:56:22.990 DEBUG:boto:Params: {} 2024-04-16T19:56:22.991 DEBUG:boto:Token: None 2024-04-16T19:56:22.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:22 GMT /yrjefh-51/?versions 2024-04-16T19:56:22.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9c9nwOp3PN5S8eKeXtTVnVmEJM= 2024-04-16T19:56:22.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:22 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9c9nwOp3PN5S8eKeXtTVnVmEJM=', 'Content-Length': '0'} 2024-04-16T19:56:22.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b782238ca3d2372e-00661ed7e6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:22.996 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000000nulltrue2024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000001nulltrue2024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000007nulltrue2024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000008nulltrue2024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000009nulltrue2024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000014nulltrue2024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000015nulltrue2024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000016nulltrue2024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000017nulltrue2024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000018nulltrue2024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000019nulltrue2024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000020nulltrue2024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000021nulltrue2024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000022nulltrue2024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000023nulltrue2024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000024nulltrue2024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000025nulltrue2024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000026nulltrue2024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000027nulltrue2024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000028nulltrue2024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000029nulltrue2024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000030nulltrue2024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000031nulltrue2024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000032nulltrue2024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000033nulltrue2024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000036nulltrue2024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000037nulltrue2024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000039nulltrue2024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000040nulltrue2024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000042nulltrue2024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000045nulltrue2024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000046nulltrue2024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000048nulltrue2024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000049nulltrue2024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000050nulltrue2024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000053nulltrue2024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000055nulltrue2024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000056nulltrue2024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000057nulltrue2024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000058nulltrue2024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000059nulltrue2024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000060nulltrue2024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000061nulltrue2024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000062nulltrue2024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000065nulltrue2024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000076nulltrue2024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000133nulltrue2024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000162nulltrue2024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000271nulltrue2024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000283nulltrue2024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000317nulltrue2024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000423nulltrue2024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000467nulltrue2024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:22.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-16T19:56:22.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-16T19:56:22.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-16T19:56:23.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-16T19:56:23.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-16T19:56:23.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-16T19:56:23.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-16T19:56:23.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-16T19:56:23.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-16T19:56:23.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-16T19:56:23.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-16T19:56:23.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-16T19:56:23.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-16T19:56:23.005 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.005 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.006 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.006 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.006 DEBUG:boto:Method: GET 2024-04-16T19:56:23.006 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.006 DEBUG:boto:Data: 2024-04-16T19:56:23.006 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.006 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.007 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.007 DEBUG:boto:Params: {} 2024-04-16T19:56:23.007 DEBUG:boto:Token: None 2024-04-16T19:56:23.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/?versions 2024-04-16T19:56:23.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pI/KJLM4urHInGXWrCfHtXbYwNI= 2024-04-16T19:56:23.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pI/KJLM4urHInGXWrCfHtXbYwNI=', 'Content-Length': '0'} 2024-04-16T19:56:23.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000318086fb143c38d-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.011 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000000nulltrue2024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000001nulltrue2024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000007nulltrue2024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000008nulltrue2024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000009nulltrue2024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000014nulltrue2024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000015nulltrue2024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000016nulltrue2024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000017nulltrue2024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000018nulltrue2024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000019nulltrue2024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000020nulltrue2024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000021nulltrue2024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000022nulltrue2024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000023nulltrue2024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000024nulltrue2024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000025nulltrue2024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000026nulltrue2024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000027nulltrue2024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000028nulltrue2024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000029nulltrue2024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000030nulltrue2024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000031nulltrue2024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000032nulltrue2024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000033nulltrue2024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000036nulltrue2024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000037nulltrue2024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000039nulltrue2024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000040nulltrue2024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000042nulltrue2024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000045nulltrue2024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000046nulltrue2024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000048nulltrue2024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000049nulltrue2024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000050nulltrue2024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000053nulltrue2024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000055nulltrue2024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000056nulltrue2024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000057nulltrue2024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000058nulltrue2024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000059nulltrue2024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000060nulltrue2024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000061nulltrue2024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000062nulltrue2024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000065nulltrue2024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000076nulltrue2024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000133nulltrue2024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000162nulltrue2024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000271nulltrue2024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000283nulltrue2024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000317nulltrue2024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000423nulltrue2024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000467nulltrue2024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.013 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.014 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.014 DEBUG:boto:path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.014 DEBUG:boto:auth_path=/yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.014 DEBUG:boto:Method: GET 2024-04-16T19:56:23.014 DEBUG:boto:Path: /yrjefh-51/?versions&max-keys=999 2024-04-16T19:56:23.014 DEBUG:boto:Data: 2024-04-16T19:56:23.014 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.014 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.014 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.014 DEBUG:boto:Params: {} 2024-04-16T19:56:23.015 DEBUG:boto:Token: None 2024-04-16T19:56:23.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/?versions 2024-04-16T19:56:23.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pI/KJLM4urHInGXWrCfHtXbYwNI= 2024-04-16T19:56:23.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pI/KJLM4urHInGXWrCfHtXbYwNI=', 'Content-Length': '0'} 2024-04-16T19:56:23.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4c5346f8ad8ecee-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.019 DEBUG:boto:b'yrjefh-51999falseobj00000000000000000000nulltrue2024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000001nulltrue2024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000002nulltrue2024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000003nulltrue2024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000004nulltrue2024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000005nulltrue2024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000006nulltrue2024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000007nulltrue2024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000008nulltrue2024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000009nulltrue2024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000010nulltrue2024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000011nulltrue2024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000012nulltrue2024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000013nulltrue2024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000014nulltrue2024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000015nulltrue2024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000016nulltrue2024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000017nulltrue2024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000018nulltrue2024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000019nulltrue2024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000020nulltrue2024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000021nulltrue2024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000022nulltrue2024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000023nulltrue2024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000024nulltrue2024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000025nulltrue2024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000026nulltrue2024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000027nulltrue2024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000028nulltrue2024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000029nulltrue2024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000030nulltrue2024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000031nulltrue2024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000032nulltrue2024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000033nulltrue2024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000036nulltrue2024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000037nulltrue2024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000039nulltrue2024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000040nulltrue2024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000042nulltrue2024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000045nulltrue2024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000046nulltrue2024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000048nulltrue2024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000049nulltrue2024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000050nulltrue2024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000053nulltrue2024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000055nulltrue2024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000056nulltrue2024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000057nulltrue2024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000058nulltrue2024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000059nulltrue2024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000060nulltrue2024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000061nulltrue2024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000062nulltrue2024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000065nulltrue2024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000076nulltrue2024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000133nulltrue2024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000162nulltrue2024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000271nulltrue2024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000283nulltrue2024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000317nulltrue2024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000423nulltrue2024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj00000000000000000467nulltrue2024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-16T19:56:23.024 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.024 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.024 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.024 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.024 DEBUG:boto:Method: GET 2024-04-16T19:56:23.025 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.025 DEBUG:boto:Data: 2024-04-16T19:56:23.025 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.025 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.025 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.025 DEBUG:boto:Params: {} 2024-04-16T19:56:23.025 DEBUG:boto:Token: None 2024-04-16T19:56:23.025 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY= 2024-04-16T19:56:23.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY=', 'Content-Length': '0'} 2024-04-16T19:56:23.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227264a3fe91aa09-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.071 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.071 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.071 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.071 DEBUG:boto:Method: GET 2024-04-16T19:56:23.071 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.071 DEBUG:boto:Data: 2024-04-16T19:56:23.071 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.071 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.072 DEBUG:boto:Params: {} 2024-04-16T19:56:23.072 DEBUG:boto:Token: None 2024-04-16T19:56:23.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY= 2024-04-16T19:56:23.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY=', 'Content-Length': '0'} 2024-04-16T19:56:23.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6c0b226cee3c39-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.114 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.115 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.115 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.115 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.115 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.115 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.115 DEBUG:boto:Data: 2024-04-16T19:56:23.115 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.115 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.116 DEBUG:boto:Params: {} 2024-04-16T19:56:23.116 DEBUG:boto:Token: None 2024-04-16T19:56:23.116 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8tD9JskOJcigxEahtlfFMstztGI= 2024-04-16T19:56:23.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8tD9JskOJcigxEahtlfFMstztGI=', 'Content-Length': '0'} 2024-04-16T19:56:23.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e46f1656e47170-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.119 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.119 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.119 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.119 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.119 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.119 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.119 DEBUG:boto:Data: 2024-04-16T19:56:23.119 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.120 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.120 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.120 DEBUG:boto:Params: {} 2024-04-16T19:56:23.120 DEBUG:boto:Token: None 2024-04-16T19:56:23.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8tD9JskOJcigxEahtlfFMstztGI= 2024-04-16T19:56:23.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8tD9JskOJcigxEahtlfFMstztGI=', 'Content-Length': '0'} 2024-04-16T19:56:23.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007294f17e56a347d7-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-16T19:56:23.123 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.123 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.123 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.123 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.123 DEBUG:boto:Method: GET 2024-04-16T19:56:23.123 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.123 DEBUG:boto:Data: 2024-04-16T19:56:23.124 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.124 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.124 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.124 DEBUG:boto:Params: {} 2024-04-16T19:56:23.124 DEBUG:boto:Token: None 2024-04-16T19:56:23.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY= 2024-04-16T19:56:23.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY=', 'Content-Length': '0'} 2024-04-16T19:56:23.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4a761285f4fdf6-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.170 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.171 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.171 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.171 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.171 DEBUG:boto:Method: GET 2024-04-16T19:56:23.171 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.171 DEBUG:boto:Data: 2024-04-16T19:56:23.171 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.171 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.172 DEBUG:boto:Params: {} 2024-04-16T19:56:23.172 DEBUG:boto:Token: None 2024-04-16T19:56:23.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000?versionId=null 2024-04-16T19:56:23.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY= 2024-04-16T19:56:23.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9IF498RM3j2L6ZcmHPYxjpHDqRY=', 'Content-Length': '0'} 2024-04-16T19:56:23.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a6f851ff5916e1-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.174 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.175 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.175 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.175 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.175 DEBUG:boto:Method: GET 2024-04-16T19:56:23.175 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.175 DEBUG:boto:Data: 2024-04-16T19:56:23.175 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.175 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.176 DEBUG:boto:Params: {} 2024-04-16T19:56:23.176 DEBUG:boto:Token: None 2024-04-16T19:56:23.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f00b521730f420-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.179 DEBUG:boto:b'yrjefh-51obj000000000000000000001000falseobj000000000000000000002024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.179 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.180 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.180 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.180 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.180 DEBUG:boto:Method: GET 2024-04-16T19:56:23.180 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.180 DEBUG:boto:Data: 2024-04-16T19:56:23.180 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.180 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.181 DEBUG:boto:Params: {} 2024-04-16T19:56:23.181 DEBUG:boto:Token: None 2024-04-16T19:56:23.181 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001776167535a0f82e-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.185 DEBUG:boto:b'yrjefh-51obj000000000000000000001000falseobj000000000000000000002024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.185 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.186 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.186 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.186 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.186 DEBUG:boto:Data: 2024-04-16T19:56:23.186 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.186 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.186 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.186 DEBUG:boto:Params: {} 2024-04-16T19:56:23.186 DEBUG:boto:Token: None 2024-04-16T19:56:23.187 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jMF/XzIG58MbHeAA9ta+CI6Uewo= 2024-04-16T19:56:23.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMF/XzIG58MbHeAA9ta+CI6Uewo=', 'Content-Length': '0'} 2024-04-16T19:56:23.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e3ec0989ab7166-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.189 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.189 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.189 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.189 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.189 DEBUG:boto:Data: 2024-04-16T19:56:23.189 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.189 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.190 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.190 DEBUG:boto:Params: {} 2024-04-16T19:56:23.190 DEBUG:boto:Token: None 2024-04-16T19:56:23.190 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jMF/XzIG58MbHeAA9ta+CI6Uewo= 2024-04-16T19:56:23.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMF/XzIG58MbHeAA9ta+CI6Uewo=', 'Content-Length': '0'} 2024-04-16T19:56:23.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa9e3897c8b49e7d-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-16T19:56:23.193 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.193 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.193 DEBUG:boto:Method: GET 2024-04-16T19:56:23.193 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.193 DEBUG:boto:Data: 2024-04-16T19:56:23.193 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.193 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.193 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.193 DEBUG:boto:Params: {} 2024-04-16T19:56:23.193 DEBUG:boto:Token: None 2024-04-16T19:56:23.194 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.194 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2KrmBbf9+ukTwXQl57uyeU+fmk= 2024-04-16T19:56:23.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2KrmBbf9+ukTwXQl57uyeU+fmk=', 'Content-Length': '0'} 2024-04-16T19:56:23.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49df79d0b62b76e-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.238 DEBUG:boto:path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.239 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.239 DEBUG:boto:Method: GET 2024-04-16T19:56:23.239 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.239 DEBUG:boto:Data: 2024-04-16T19:56:23.239 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.239 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.239 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.239 DEBUG:boto:Params: {} 2024-04-16T19:56:23.239 DEBUG:boto:Token: None 2024-04-16T19:56:23.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000000 2024-04-16T19:56:23.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2KrmBbf9+ukTwXQl57uyeU+fmk= 2024-04-16T19:56:23.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2KrmBbf9+ukTwXQl57uyeU+fmk=', 'Content-Length': '0'} 2024-04-16T19:56:23.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e1f381baee6360-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.242 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.242 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.242 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.242 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.242 DEBUG:boto:Method: GET 2024-04-16T19:56:23.242 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.243 DEBUG:boto:Data: 2024-04-16T19:56:23.243 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.243 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.243 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.243 DEBUG:boto:Params: {} 2024-04-16T19:56:23.243 DEBUG:boto:Token: None 2024-04-16T19:56:23.243 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000563ccc898a3edb4c-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.286 DEBUG:boto:b'yrjefh-51obj000000000000000000001000falseobj000000000000000000002024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.287 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.287 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.287 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.287 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.287 DEBUG:boto:Method: GET 2024-04-16T19:56:23.287 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000000 2024-04-16T19:56:23.287 DEBUG:boto:Data: 2024-04-16T19:56:23.287 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.288 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.288 DEBUG:boto:Params: {} 2024-04-16T19:56:23.288 DEBUG:boto:Token: None 2024-04-16T19:56:23.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492298a5581de04e-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.292 DEBUG:boto:b'yrjefh-51obj000000000000000000001000falseobj000000000000000000002024-04-16T19:56:08.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:56:23.292 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.293 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.293 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.293 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.293 DEBUG:boto:Method: GET 2024-04-16T19:56:23.293 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.293 DEBUG:boto:Data: 2024-04-16T19:56:23.293 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.293 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.293 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.293 DEBUG:boto:Params: {} 2024-04-16T19:56:23.293 DEBUG:boto:Token: None 2024-04-16T19:56:23.294 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.294 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE= 2024-04-16T19:56:23.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE=', 'Content-Length': '0'} 2024-04-16T19:56:23.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006abb9f5fdccbe779-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.338 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.339 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.339 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.339 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.339 DEBUG:boto:Method: GET 2024-04-16T19:56:23.339 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.339 DEBUG:boto:Data: 2024-04-16T19:56:23.339 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.340 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.340 DEBUG:boto:Params: {} 2024-04-16T19:56:23.340 DEBUG:boto:Token: None 2024-04-16T19:56:23.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE= 2024-04-16T19:56:23.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE=', 'Content-Length': '0'} 2024-04-16T19:56:23.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8171f856bd42d4a-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.342 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.343 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.343 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.343 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.343 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.343 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.343 DEBUG:boto:Data: 2024-04-16T19:56:23.343 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.343 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.343 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.344 DEBUG:boto:Params: {} 2024-04-16T19:56:23.344 DEBUG:boto:Token: None 2024-04-16T19:56:23.344 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.344 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:goeroOzIdJ+NV4+OUbIqI16blp0= 2024-04-16T19:56:23.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:goeroOzIdJ+NV4+OUbIqI16blp0=', 'Content-Length': '0'} 2024-04-16T19:56:23.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d42a3e12f213a5a-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.346 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.347 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.347 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.347 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.347 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.347 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.347 DEBUG:boto:Data: 2024-04-16T19:56:23.347 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.347 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.348 DEBUG:boto:Params: {} 2024-04-16T19:56:23.348 DEBUG:boto:Token: None 2024-04-16T19:56:23.348 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:goeroOzIdJ+NV4+OUbIqI16blp0= 2024-04-16T19:56:23.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:goeroOzIdJ+NV4+OUbIqI16blp0=', 'Content-Length': '0'} 2024-04-16T19:56:23.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3102bf5c92f0018-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:56:23.351 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.351 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.351 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.351 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.351 DEBUG:boto:Method: GET 2024-04-16T19:56:23.351 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.351 DEBUG:boto:Data: 2024-04-16T19:56:23.351 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.351 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.352 DEBUG:boto:Params: {} 2024-04-16T19:56:23.352 DEBUG:boto:Token: None 2024-04-16T19:56:23.352 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE= 2024-04-16T19:56:23.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE=', 'Content-Length': '0'} 2024-04-16T19:56:23.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c14cf0443f489d9-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.354 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.354 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.354 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.355 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.355 DEBUG:boto:Method: GET 2024-04-16T19:56:23.355 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.355 DEBUG:boto:Data: 2024-04-16T19:56:23.355 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.355 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.355 DEBUG:boto:Params: {} 2024-04-16T19:56:23.355 DEBUG:boto:Token: None 2024-04-16T19:56:23.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001?versionId=null 2024-04-16T19:56:23.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE= 2024-04-16T19:56:23.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SStfzXe5BTPj49DCkVYTxbRjEOE=', 'Content-Length': '0'} 2024-04-16T19:56:23.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b56dc20f1c0a8c9f-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.398 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.398 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.399 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.399 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.399 DEBUG:boto:Method: GET 2024-04-16T19:56:23.399 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.399 DEBUG:boto:Data: 2024-04-16T19:56:23.399 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.399 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.399 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.400 DEBUG:boto:Params: {} 2024-04-16T19:56:23.400 DEBUG:boto:Token: None 2024-04-16T19:56:23.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f23fb69120dac388-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.403 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.403 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.403 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.404 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.404 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.404 DEBUG:boto:Method: GET 2024-04-16T19:56:23.404 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.404 DEBUG:boto:Data: 2024-04-16T19:56:23.404 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.404 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.404 DEBUG:boto:Params: {} 2024-04-16T19:56:23.404 DEBUG:boto:Token: None 2024-04-16T19:56:23.405 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62bd466165f6620-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.409 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.409 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.409 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.410 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.410 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.410 DEBUG:boto:Data: 2024-04-16T19:56:23.410 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.410 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.410 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.410 DEBUG:boto:Params: {} 2024-04-16T19:56:23.410 DEBUG:boto:Token: None 2024-04-16T19:56:23.411 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mUxtNbQgERNfksL+X7AgXt+x1Mc= 2024-04-16T19:56:23.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mUxtNbQgERNfksL+X7AgXt+x1Mc=', 'Content-Length': '0'} 2024-04-16T19:56:23.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95090610b2ece6f-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.413 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.413 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.413 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.414 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.414 DEBUG:boto:Data: 2024-04-16T19:56:23.414 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.414 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.414 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.414 DEBUG:boto:Params: {} 2024-04-16T19:56:23.414 DEBUG:boto:Token: None 2024-04-16T19:56:23.414 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mUxtNbQgERNfksL+X7AgXt+x1Mc= 2024-04-16T19:56:23.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mUxtNbQgERNfksL+X7AgXt+x1Mc=', 'Content-Length': '0'} 2024-04-16T19:56:23.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214ec93ff1c25050-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-16T19:56:23.417 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.417 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.417 DEBUG:boto:Method: GET 2024-04-16T19:56:23.417 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.417 DEBUG:boto:Data: 2024-04-16T19:56:23.418 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.418 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.418 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.418 DEBUG:boto:Params: {} 2024-04-16T19:56:23.418 DEBUG:boto:Token: None 2024-04-16T19:56:23.418 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGnj0jRSD2cjmUefTIoNCHWk0Ic= 2024-04-16T19:56:23.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGnj0jRSD2cjmUefTIoNCHWk0Ic=', 'Content-Length': '0'} 2024-04-16T19:56:23.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8fbe7f5008ae4d-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.462 DEBUG:boto:path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.462 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.463 DEBUG:boto:Method: GET 2024-04-16T19:56:23.463 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.463 DEBUG:boto:Data: 2024-04-16T19:56:23.463 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.463 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.463 DEBUG:boto:Params: {} 2024-04-16T19:56:23.464 DEBUG:boto:Token: None 2024-04-16T19:56:23.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000001 2024-04-16T19:56:23.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGnj0jRSD2cjmUefTIoNCHWk0Ic= 2024-04-16T19:56:23.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGnj0jRSD2cjmUefTIoNCHWk0Ic=', 'Content-Length': '0'} 2024-04-16T19:56:23.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adfa601c69911d62-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.466 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.467 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.467 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.467 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.467 DEBUG:boto:Method: GET 2024-04-16T19:56:23.467 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.467 DEBUG:boto:Data: 2024-04-16T19:56:23.467 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.468 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.468 DEBUG:boto:Params: {} 2024-04-16T19:56:23.468 DEBUG:boto:Token: None 2024-04-16T19:56:23.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1587d748bcaaade-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.471 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.472 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.472 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.472 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.472 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.472 DEBUG:boto:Method: GET 2024-04-16T19:56:23.472 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000001 2024-04-16T19:56:23.472 DEBUG:boto:Data: 2024-04-16T19:56:23.472 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.473 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.473 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.473 DEBUG:boto:Params: {} 2024-04-16T19:56:23.473 DEBUG:boto:Token: None 2024-04-16T19:56:23.473 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044de0211833f29a-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.477 DEBUG:boto:b'yrjefh-51obj000000000000000000011000falseobj000000000000000000012024-04-16T19:56:09.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:56:23.478 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.478 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.478 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.478 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.478 DEBUG:boto:Method: GET 2024-04-16T19:56:23.479 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.479 DEBUG:boto:Data: 2024-04-16T19:56:23.479 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.479 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.479 DEBUG:boto:Params: {} 2024-04-16T19:56:23.479 DEBUG:boto:Token: None 2024-04-16T19:56:23.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps= 2024-04-16T19:56:23.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps=', 'Content-Length': '0'} 2024-04-16T19:56:23.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000320f12b9aef10cc3-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.482 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.483 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.483 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.483 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.483 DEBUG:boto:Method: GET 2024-04-16T19:56:23.483 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.483 DEBUG:boto:Data: 2024-04-16T19:56:23.483 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.484 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.484 DEBUG:boto:Params: {} 2024-04-16T19:56:23.484 DEBUG:boto:Token: None 2024-04-16T19:56:23.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps= 2024-04-16T19:56:23.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps=', 'Content-Length': '0'} 2024-04-16T19:56:23.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df923ebdf664d2bc-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.530 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.531 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.531 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.531 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.531 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.531 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.531 DEBUG:boto:Data: 2024-04-16T19:56:23.531 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.531 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.532 DEBUG:boto:Params: {} 2024-04-16T19:56:23.532 DEBUG:boto:Token: None 2024-04-16T19:56:23.532 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rJg8gTRrQX6JlKH4VdJZpu8+lyo= 2024-04-16T19:56:23.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rJg8gTRrQX6JlKH4VdJZpu8+lyo=', 'Content-Length': '0'} 2024-04-16T19:56:23.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2aac46be7e27d9d-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.534 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.535 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.535 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.535 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.535 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.535 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.535 DEBUG:boto:Data: 2024-04-16T19:56:23.535 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.536 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.536 DEBUG:boto:Params: {} 2024-04-16T19:56:23.536 DEBUG:boto:Token: None 2024-04-16T19:56:23.536 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rJg8gTRrQX6JlKH4VdJZpu8+lyo= 2024-04-16T19:56:23.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rJg8gTRrQX6JlKH4VdJZpu8+lyo=', 'Content-Length': '0'} 2024-04-16T19:56:23.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b6c10de3a0bb86-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:56:23.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.539 DEBUG:boto:Method: GET 2024-04-16T19:56:23.539 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.539 DEBUG:boto:Data: 2024-04-16T19:56:23.540 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.540 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.540 DEBUG:boto:Params: {} 2024-04-16T19:56:23.540 DEBUG:boto:Token: None 2024-04-16T19:56:23.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps= 2024-04-16T19:56:23.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps=', 'Content-Length': '0'} 2024-04-16T19:56:23.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae442ff0c440acf5-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.543 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.543 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.543 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.543 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.543 DEBUG:boto:Method: GET 2024-04-16T19:56:23.544 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.544 DEBUG:boto:Data: 2024-04-16T19:56:23.544 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.544 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.544 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.544 DEBUG:boto:Params: {} 2024-04-16T19:56:23.544 DEBUG:boto:Token: None 2024-04-16T19:56:23.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002?versionId=null 2024-04-16T19:56:23.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps= 2024-04-16T19:56:23.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YOQAcKheL6s9fLfz1s3tSe5msps=', 'Content-Length': '0'} 2024-04-16T19:56:23.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004998ea54061e0a5e-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.547 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.547 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.547 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.547 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.547 DEBUG:boto:Method: GET 2024-04-16T19:56:23.548 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.548 DEBUG:boto:Data: 2024-04-16T19:56:23.548 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.548 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.548 DEBUG:boto:Params: {} 2024-04-16T19:56:23.548 DEBUG:boto:Token: None 2024-04-16T19:56:23.548 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f155b48b1e2d1adc-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.594 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.595 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.595 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.595 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.595 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.595 DEBUG:boto:Method: GET 2024-04-16T19:56:23.595 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.596 DEBUG:boto:Data: 2024-04-16T19:56:23.596 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.596 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.596 DEBUG:boto:Params: {} 2024-04-16T19:56:23.596 DEBUG:boto:Token: None 2024-04-16T19:56:23.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f798b40eb663e60-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.601 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.602 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.602 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.602 DEBUG:boto:Data: 2024-04-16T19:56:23.602 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.602 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.603 DEBUG:boto:Params: {} 2024-04-16T19:56:23.603 DEBUG:boto:Token: None 2024-04-16T19:56:23.603 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w/qwXSQ3B4FsluCxFW8LO/wrkn4= 2024-04-16T19:56:23.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w/qwXSQ3B4FsluCxFW8LO/wrkn4=', 'Content-Length': '0'} 2024-04-16T19:56:23.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e4ea96163a62cb-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.605 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.605 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.606 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.606 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.606 DEBUG:boto:Data: 2024-04-16T19:56:23.606 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.606 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.606 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.606 DEBUG:boto:Params: {} 2024-04-16T19:56:23.606 DEBUG:boto:Token: None 2024-04-16T19:56:23.607 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w/qwXSQ3B4FsluCxFW8LO/wrkn4= 2024-04-16T19:56:23.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w/qwXSQ3B4FsluCxFW8LO/wrkn4=', 'Content-Length': '0'} 2024-04-16T19:56:23.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042cf3a015a969269-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-16T19:56:23.609 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.610 DEBUG:boto:Method: GET 2024-04-16T19:56:23.610 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.610 DEBUG:boto:Data: 2024-04-16T19:56:23.610 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.610 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.610 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.610 DEBUG:boto:Params: {} 2024-04-16T19:56:23.610 DEBUG:boto:Token: None 2024-04-16T19:56:23.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xxs9yldWIynglSwg6FZJ0kehtCc= 2024-04-16T19:56:23.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xxs9yldWIynglSwg6FZJ0kehtCc=', 'Content-Length': '0'} 2024-04-16T19:56:23.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004806675618bc3155-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.613 DEBUG:boto:path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.614 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.614 DEBUG:boto:Method: GET 2024-04-16T19:56:23.614 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.614 DEBUG:boto:Data: 2024-04-16T19:56:23.614 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.614 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.614 DEBUG:boto:Params: {} 2024-04-16T19:56:23.615 DEBUG:boto:Token: None 2024-04-16T19:56:23.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000002 2024-04-16T19:56:23.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xxs9yldWIynglSwg6FZJ0kehtCc= 2024-04-16T19:56:23.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xxs9yldWIynglSwg6FZJ0kehtCc=', 'Content-Length': '0'} 2024-04-16T19:56:23.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a3c933c94fd3e7-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.658 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.659 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.659 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.659 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.659 DEBUG:boto:Method: GET 2024-04-16T19:56:23.659 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.659 DEBUG:boto:Data: 2024-04-16T19:56:23.659 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.660 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.660 DEBUG:boto:Params: {} 2024-04-16T19:56:23.660 DEBUG:boto:Token: None 2024-04-16T19:56:23.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f196eb4013a307-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.663 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.664 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.664 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.664 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.664 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.664 DEBUG:boto:Method: GET 2024-04-16T19:56:23.664 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000002 2024-04-16T19:56:23.664 DEBUG:boto:Data: 2024-04-16T19:56:23.664 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.664 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.665 DEBUG:boto:Params: {} 2024-04-16T19:56:23.665 DEBUG:boto:Token: None 2024-04-16T19:56:23.665 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0e27cb1877c68d-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.669 DEBUG:boto:b'yrjefh-51obj000000000000000000021000falseobj000000000000000000022024-04-16T19:56:09.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:56:23.670 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.670 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.670 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.670 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.670 DEBUG:boto:Method: GET 2024-04-16T19:56:23.670 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.671 DEBUG:boto:Data: 2024-04-16T19:56:23.671 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.671 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.671 DEBUG:boto:Params: {} 2024-04-16T19:56:23.671 DEBUG:boto:Token: None 2024-04-16T19:56:23.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q= 2024-04-16T19:56:23.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q=', 'Content-Length': '0'} 2024-04-16T19:56:23.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca4f58b3155bbdb-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.675 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.675 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.675 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.675 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.675 DEBUG:boto:Method: GET 2024-04-16T19:56:23.675 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.675 DEBUG:boto:Data: 2024-04-16T19:56:23.675 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.676 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.676 DEBUG:boto:Params: {} 2024-04-16T19:56:23.676 DEBUG:boto:Token: None 2024-04-16T19:56:23.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q= 2024-04-16T19:56:23.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q=', 'Content-Length': '0'} 2024-04-16T19:56:23.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce5eda26313348a-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.679 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.679 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.679 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.679 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.680 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.680 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.680 DEBUG:boto:Data: 2024-04-16T19:56:23.680 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.680 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.680 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.680 DEBUG:boto:Params: {} 2024-04-16T19:56:23.680 DEBUG:boto:Token: None 2024-04-16T19:56:23.680 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X6kBxKt0pdmTHKP8VRH2JGXB2Aw= 2024-04-16T19:56:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X6kBxKt0pdmTHKP8VRH2JGXB2Aw=', 'Content-Length': '0'} 2024-04-16T19:56:23.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000facfeb70e598b511-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.683 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.683 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.683 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.683 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.684 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.684 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.684 DEBUG:boto:Data: 2024-04-16T19:56:23.684 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.684 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.684 DEBUG:boto:Params: {} 2024-04-16T19:56:23.684 DEBUG:boto:Token: None 2024-04-16T19:56:23.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X6kBxKt0pdmTHKP8VRH2JGXB2Aw= 2024-04-16T19:56:23.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X6kBxKt0pdmTHKP8VRH2JGXB2Aw=', 'Content-Length': '0'} 2024-04-16T19:56:23.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4fec9ecb322dd0-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:56:23.687 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.687 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.688 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.688 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.688 DEBUG:boto:Method: GET 2024-04-16T19:56:23.688 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.688 DEBUG:boto:Data: 2024-04-16T19:56:23.688 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.688 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.688 DEBUG:boto:Params: {} 2024-04-16T19:56:23.689 DEBUG:boto:Token: None 2024-04-16T19:56:23.689 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q= 2024-04-16T19:56:23.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q=', 'Content-Length': '0'} 2024-04-16T19:56:23.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2c2569b9ea8ddd-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.734 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.735 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.735 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.735 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.735 DEBUG:boto:Method: GET 2024-04-16T19:56:23.735 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.735 DEBUG:boto:Data: 2024-04-16T19:56:23.735 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.735 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.736 DEBUG:boto:Params: {} 2024-04-16T19:56:23.736 DEBUG:boto:Token: None 2024-04-16T19:56:23.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003?versionId=null 2024-04-16T19:56:23.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q= 2024-04-16T19:56:23.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1hkWRzoFHErbjosWgD+yTHLfk5Q=', 'Content-Length': '0'} 2024-04-16T19:56:23.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c7d707f0cd432a4-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.738 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.739 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.739 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.739 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.739 DEBUG:boto:Method: GET 2024-04-16T19:56:23.739 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.739 DEBUG:boto:Data: 2024-04-16T19:56:23.739 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.739 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.739 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.740 DEBUG:boto:Params: {} 2024-04-16T19:56:23.740 DEBUG:boto:Token: None 2024-04-16T19:56:23.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e1ea1e2c1b516e-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.743 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.743 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.744 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.744 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.744 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.744 DEBUG:boto:Method: GET 2024-04-16T19:56:23.744 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.744 DEBUG:boto:Data: 2024-04-16T19:56:23.744 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.744 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.744 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.744 DEBUG:boto:Params: {} 2024-04-16T19:56:23.745 DEBUG:boto:Token: None 2024-04-16T19:56:23.745 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eaef73587158fbd-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.749 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.749 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.750 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.750 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.750 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.750 DEBUG:boto:Data: 2024-04-16T19:56:23.750 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.750 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.750 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.750 DEBUG:boto:Params: {} 2024-04-16T19:56:23.750 DEBUG:boto:Token: None 2024-04-16T19:56:23.751 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L9o6rgbAcGSXBh2NWoomAqk7RFs= 2024-04-16T19:56:23.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L9o6rgbAcGSXBh2NWoomAqk7RFs=', 'Content-Length': '0'} 2024-04-16T19:56:23.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f9a393b8e1812a-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.753 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.753 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.753 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.753 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.754 DEBUG:boto:Data: 2024-04-16T19:56:23.754 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.754 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.754 DEBUG:boto:Params: {} 2024-04-16T19:56:23.754 DEBUG:boto:Token: None 2024-04-16T19:56:23.754 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.754 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L9o6rgbAcGSXBh2NWoomAqk7RFs= 2024-04-16T19:56:23.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L9o6rgbAcGSXBh2NWoomAqk7RFs=', 'Content-Length': '0'} 2024-04-16T19:56:23.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f84d0fc53576988-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-16T19:56:23.757 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.757 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.757 DEBUG:boto:Method: GET 2024-04-16T19:56:23.757 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.757 DEBUG:boto:Data: 2024-04-16T19:56:23.757 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.758 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.758 DEBUG:boto:Params: {} 2024-04-16T19:56:23.758 DEBUG:boto:Token: None 2024-04-16T19:56:23.758 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.758 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flDBzzmg8Jjfm7namrYcRF5SKbM= 2024-04-16T19:56:23.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flDBzzmg8Jjfm7namrYcRF5SKbM=', 'Content-Length': '0'} 2024-04-16T19:56:23.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d574cb53f7d22db8-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.761 DEBUG:boto:path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.761 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.761 DEBUG:boto:Method: GET 2024-04-16T19:56:23.761 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.761 DEBUG:boto:Data: 2024-04-16T19:56:23.761 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.761 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.761 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.762 DEBUG:boto:Params: {} 2024-04-16T19:56:23.762 DEBUG:boto:Token: None 2024-04-16T19:56:23.762 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000003 2024-04-16T19:56:23.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:flDBzzmg8Jjfm7namrYcRF5SKbM= 2024-04-16T19:56:23.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:flDBzzmg8Jjfm7namrYcRF5SKbM=', 'Content-Length': '0'} 2024-04-16T19:56:23.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086410151127929aa-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.764 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.765 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.765 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.765 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.765 DEBUG:boto:Method: GET 2024-04-16T19:56:23.765 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.765 DEBUG:boto:Data: 2024-04-16T19:56:23.765 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.765 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.765 DEBUG:boto:Params: {} 2024-04-16T19:56:23.766 DEBUG:boto:Token: None 2024-04-16T19:56:23.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e6b4e29d405cfa5-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.769 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.769 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.769 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.769 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.769 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.770 DEBUG:boto:Method: GET 2024-04-16T19:56:23.770 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000003 2024-04-16T19:56:23.770 DEBUG:boto:Data: 2024-04-16T19:56:23.770 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.770 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.770 DEBUG:boto:Params: {} 2024-04-16T19:56:23.770 DEBUG:boto:Token: None 2024-04-16T19:56:23.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d18f6a14fa81f2-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.774 DEBUG:boto:b'yrjefh-51obj000000000000000000031000falseobj000000000000000000032024-04-16T19:56:08.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:56:23.775 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.775 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.775 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.775 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.775 DEBUG:boto:Method: GET 2024-04-16T19:56:23.776 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.776 DEBUG:boto:Data: 2024-04-16T19:56:23.776 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.776 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.776 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.776 DEBUG:boto:Params: {} 2024-04-16T19:56:23.776 DEBUG:boto:Token: None 2024-04-16T19:56:23.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw= 2024-04-16T19:56:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw=', 'Content-Length': '0'} 2024-04-16T19:56:23.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003900d6b7e81ef6f4-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.779 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.779 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.779 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.780 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.780 DEBUG:boto:Method: GET 2024-04-16T19:56:23.780 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.780 DEBUG:boto:Data: 2024-04-16T19:56:23.780 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.780 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.780 DEBUG:boto:Params: {} 2024-04-16T19:56:23.780 DEBUG:boto:Token: None 2024-04-16T19:56:23.781 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw= 2024-04-16T19:56:23.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw=', 'Content-Length': '0'} 2024-04-16T19:56:23.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd5387a1189728c-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.826 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.827 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.827 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.827 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.827 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.827 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.827 DEBUG:boto:Data: 2024-04-16T19:56:23.827 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.827 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.828 DEBUG:boto:Params: {} 2024-04-16T19:56:23.828 DEBUG:boto:Token: None 2024-04-16T19:56:23.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nRQGIOo5BcMipkoWRa1ixtElDfQ= 2024-04-16T19:56:23.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nRQGIOo5BcMipkoWRa1ixtElDfQ=', 'Content-Length': '0'} 2024-04-16T19:56:23.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d1b1fcd2b44e53-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.830 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.831 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.831 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.831 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.831 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.831 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.831 DEBUG:boto:Data: 2024-04-16T19:56:23.831 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.831 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.832 DEBUG:boto:Params: {} 2024-04-16T19:56:23.832 DEBUG:boto:Token: None 2024-04-16T19:56:23.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nRQGIOo5BcMipkoWRa1ixtElDfQ= 2024-04-16T19:56:23.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nRQGIOo5BcMipkoWRa1ixtElDfQ=', 'Content-Length': '0'} 2024-04-16T19:56:23.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37c702fef6477b4-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:56:23.835 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.835 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.835 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.835 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.835 DEBUG:boto:Method: GET 2024-04-16T19:56:23.835 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.835 DEBUG:boto:Data: 2024-04-16T19:56:23.835 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.835 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.836 DEBUG:boto:Params: {} 2024-04-16T19:56:23.836 DEBUG:boto:Token: None 2024-04-16T19:56:23.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw= 2024-04-16T19:56:23.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw=', 'Content-Length': '0'} 2024-04-16T19:56:23.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d4e927eedea9ce-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.839 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.839 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.839 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.839 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.839 DEBUG:boto:Method: GET 2024-04-16T19:56:23.839 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.839 DEBUG:boto:Data: 2024-04-16T19:56:23.839 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.840 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.840 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.840 DEBUG:boto:Params: {} 2024-04-16T19:56:23.840 DEBUG:boto:Token: None 2024-04-16T19:56:23.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004?versionId=null 2024-04-16T19:56:23.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw= 2024-04-16T19:56:23.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H/gw/P9YAOqfPZ6eutyz9WZQGOw=', 'Content-Length': '0'} 2024-04-16T19:56:23.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0622b9658b1b12e-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.843 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.843 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.843 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.843 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.843 DEBUG:boto:Method: GET 2024-04-16T19:56:23.843 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.843 DEBUG:boto:Data: 2024-04-16T19:56:23.843 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.844 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.844 DEBUG:boto:Params: {} 2024-04-16T19:56:23.844 DEBUG:boto:Token: None 2024-04-16T19:56:23.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bfb253e7aa80d81-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.890 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.891 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.891 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.891 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.891 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.891 DEBUG:boto:Method: GET 2024-04-16T19:56:23.891 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.891 DEBUG:boto:Data: 2024-04-16T19:56:23.892 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.892 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.892 DEBUG:boto:Params: {} 2024-04-16T19:56:23.892 DEBUG:boto:Token: None 2024-04-16T19:56:23.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4bad86afac4aa36-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.897 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.897 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.897 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.898 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.898 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.898 DEBUG:boto:Data: 2024-04-16T19:56:23.898 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.898 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.898 DEBUG:boto:Params: {} 2024-04-16T19:56:23.898 DEBUG:boto:Token: None 2024-04-16T19:56:23.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qne449OMIMHcRQbslNc5fkm+aEA= 2024-04-16T19:56:23.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qne449OMIMHcRQbslNc5fkm+aEA=', 'Content-Length': '0'} 2024-04-16T19:56:23.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea4fcec291e30e3-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.901 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.901 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.901 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.901 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.901 DEBUG:boto:Data: 2024-04-16T19:56:23.901 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.902 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.902 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.902 DEBUG:boto:Params: {} 2024-04-16T19:56:23.902 DEBUG:boto:Token: None 2024-04-16T19:56:23.902 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qne449OMIMHcRQbslNc5fkm+aEA= 2024-04-16T19:56:23.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qne449OMIMHcRQbslNc5fkm+aEA=', 'Content-Length': '0'} 2024-04-16T19:56:23.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a578924607b6d127-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-16T19:56:23.905 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.905 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.905 DEBUG:boto:Method: GET 2024-04-16T19:56:23.905 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.905 DEBUG:boto:Data: 2024-04-16T19:56:23.905 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.906 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.906 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.906 DEBUG:boto:Params: {} 2024-04-16T19:56:23.906 DEBUG:boto:Token: None 2024-04-16T19:56:23.906 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.906 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:POki1jtxhlZond95z+bT7Wkohi4= 2024-04-16T19:56:23.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:POki1jtxhlZond95z+bT7Wkohi4=', 'Content-Length': '0'} 2024-04-16T19:56:23.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df85547bd6eec64-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.909 DEBUG:boto:Method: GET 2024-04-16T19:56:23.909 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.909 DEBUG:boto:Data: 2024-04-16T19:56:23.909 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.909 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.910 DEBUG:boto:Params: {} 2024-04-16T19:56:23.910 DEBUG:boto:Token: None 2024-04-16T19:56:23.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000004 2024-04-16T19:56:23.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:POki1jtxhlZond95z+bT7Wkohi4= 2024-04-16T19:56:23.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:POki1jtxhlZond95z+bT7Wkohi4=', 'Content-Length': '0'} 2024-04-16T19:56:23.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092911d281fe04771-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.912 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.913 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.913 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.913 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.913 DEBUG:boto:Method: GET 2024-04-16T19:56:23.913 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.913 DEBUG:boto:Data: 2024-04-16T19:56:23.913 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.913 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.913 DEBUG:boto:Params: {} 2024-04-16T19:56:23.914 DEBUG:boto:Token: None 2024-04-16T19:56:23.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.914 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437902756d3de09f-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.917 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.917 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.918 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.918 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.918 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.918 DEBUG:boto:Method: GET 2024-04-16T19:56:23.918 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000004 2024-04-16T19:56:23.918 DEBUG:boto:Data: 2024-04-16T19:56:23.918 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.918 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.918 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.919 DEBUG:boto:Params: {} 2024-04-16T19:56:23.919 DEBUG:boto:Token: None 2024-04-16T19:56:23.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030afcb1a87bffa65-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.923 DEBUG:boto:b'yrjefh-51obj000000000000000000041000falseobj000000000000000000042024-04-16T19:56:09.152Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:56:23.924 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.924 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.924 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.924 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.924 DEBUG:boto:Method: GET 2024-04-16T19:56:23.924 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.925 DEBUG:boto:Data: 2024-04-16T19:56:23.925 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.925 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.925 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.925 DEBUG:boto:Params: {} 2024-04-16T19:56:23.925 DEBUG:boto:Token: None 2024-04-16T19:56:23.925 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0= 2024-04-16T19:56:23.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0=', 'Content-Length': '0'} 2024-04-16T19:56:23.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a31e8cf9c9225d5-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.928 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.928 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.928 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.928 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.928 DEBUG:boto:Method: GET 2024-04-16T19:56:23.928 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.929 DEBUG:boto:Data: 2024-04-16T19:56:23.929 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.929 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.929 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.929 DEBUG:boto:Params: {} 2024-04-16T19:56:23.929 DEBUG:boto:Token: None 2024-04-16T19:56:23.929 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0= 2024-04-16T19:56:23.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0=', 'Content-Length': '0'} 2024-04-16T19:56:23.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020c33ec561909f7-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.974 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.974 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.975 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.975 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.975 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.975 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.975 DEBUG:boto:Data: 2024-04-16T19:56:23.975 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.975 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.975 DEBUG:boto:Params: {} 2024-04-16T19:56:23.976 DEBUG:boto:Token: None 2024-04-16T19:56:23.976 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O6TI78tY1sQxvzNRm8BU+3fjn5w= 2024-04-16T19:56:23.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O6TI78tY1sQxvzNRm8BU+3fjn5w=', 'Content-Length': '0'} 2024-04-16T19:56:23.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac816ae21733da8-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.978 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.978 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.979 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.979 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.979 DEBUG:boto:Method: HEAD 2024-04-16T19:56:23.979 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.979 DEBUG:boto:Data: 2024-04-16T19:56:23.979 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.979 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.979 DEBUG:boto:Params: {} 2024-04-16T19:56:23.980 DEBUG:boto:Token: None 2024-04-16T19:56:23.980 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.980 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O6TI78tY1sQxvzNRm8BU+3fjn5w= 2024-04-16T19:56:23.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O6TI78tY1sQxvzNRm8BU+3fjn5w=', 'Content-Length': '0'} 2024-04-16T19:56:23.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948e36823702ed25-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:56:23.982 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.983 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.983 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.983 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.983 DEBUG:boto:Method: GET 2024-04-16T19:56:23.983 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.983 DEBUG:boto:Data: 2024-04-16T19:56:23.983 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.983 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.983 DEBUG:boto:Params: {} 2024-04-16T19:56:23.984 DEBUG:boto:Token: None 2024-04-16T19:56:23.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0= 2024-04-16T19:56:23.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0=', 'Content-Length': '0'} 2024-04-16T19:56:23.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabdc26475576178-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.986 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.986 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:23.987 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.987 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.987 DEBUG:boto:Method: GET 2024-04-16T19:56:23.987 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.987 DEBUG:boto:Data: 2024-04-16T19:56:23.987 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.987 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.987 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.987 DEBUG:boto:Params: {} 2024-04-16T19:56:23.987 DEBUG:boto:Token: None 2024-04-16T19:56:23.988 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/obj00000000000000000005?versionId=null 2024-04-16T19:56:23.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0= 2024-04-16T19:56:23.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tIZ6sEPxdSZsXiCv0GnbTKLMZd0=', 'Content-Length': '0'} 2024-04-16T19:56:23.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa421237266f7d61-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.990 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.990 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.990 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.991 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.991 DEBUG:boto:Method: GET 2024-04-16T19:56:23.991 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.991 DEBUG:boto:Data: 2024-04-16T19:56:23.991 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.991 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:23.991 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.991 DEBUG:boto:Params: {} 2024-04-16T19:56:23.991 DEBUG:boto:Token: None 2024-04-16T19:56:23.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:23.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d7a6ecb2c168936-00661ed7e7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:23.995 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:23.995 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:23.995 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:23.995 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.995 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.995 DEBUG:boto:Method: GET 2024-04-16T19:56:23.995 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:23.996 DEBUG:boto:Data: 2024-04-16T19:56:23.996 DEBUG:boto:Headers: {} 2024-04-16T19:56:23.996 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:23.996 DEBUG:boto:Port: 8000 2024-04-16T19:56:23.996 DEBUG:boto:Params: {} 2024-04-16T19:56:23.996 DEBUG:boto:Token: None 2024-04-16T19:56:23.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:23 GMT /yrjefh-51/ 2024-04-16T19:56:23.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g= 2024-04-16T19:56:23.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:23 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P750DI/jiakpmgB6kMM6xDPEf+g=', 'Content-Length': '0'} 2024-04-16T19:56:24.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023eeabfb8500115a-00661ed7e7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.000 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.000 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.000 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.001 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.001 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.001 DEBUG:boto:Data: 2024-04-16T19:56:24.001 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.001 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.001 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.001 DEBUG:boto:Params: {} 2024-04-16T19:56:24.001 DEBUG:boto:Token: None 2024-04-16T19:56:24.001 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.002 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SjBp8EoIZLPSAKVNxs2ToTJpvRI= 2024-04-16T19:56:24.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SjBp8EoIZLPSAKVNxs2ToTJpvRI=', 'Content-Length': '0'} 2024-04-16T19:56:24.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090acb6f5338ea1c2-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.004 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.004 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.004 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.004 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.004 DEBUG:boto:Data: 2024-04-16T19:56:24.005 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.005 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.005 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.005 DEBUG:boto:Params: {} 2024-04-16T19:56:24.005 DEBUG:boto:Token: None 2024-04-16T19:56:24.005 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.005 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SjBp8EoIZLPSAKVNxs2ToTJpvRI= 2024-04-16T19:56:24.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SjBp8EoIZLPSAKVNxs2ToTJpvRI=', 'Content-Length': '0'} 2024-04-16T19:56:24.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cceafd3a7b858517-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-16T19:56:24.007 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.008 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.008 DEBUG:boto:Method: GET 2024-04-16T19:56:24.008 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.008 DEBUG:boto:Data: 2024-04-16T19:56:24.008 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.008 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.008 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.008 DEBUG:boto:Params: {} 2024-04-16T19:56:24.008 DEBUG:boto:Token: None 2024-04-16T19:56:24.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.009 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:226blUo2vRxtdXLlviqMU+s5dDA= 2024-04-16T19:56:24.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:226blUo2vRxtdXLlviqMU+s5dDA=', 'Content-Length': '0'} 2024-04-16T19:56:24.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd9617513b8bf7d-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.011 DEBUG:boto:path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.011 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.011 DEBUG:boto:Method: GET 2024-04-16T19:56:24.012 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.012 DEBUG:boto:Data: 2024-04-16T19:56:24.012 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.012 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.012 DEBUG:boto:Params: {} 2024-04-16T19:56:24.012 DEBUG:boto:Token: None 2024-04-16T19:56:24.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000005 2024-04-16T19:56:24.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:226blUo2vRxtdXLlviqMU+s5dDA= 2024-04-16T19:56:24.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:226blUo2vRxtdXLlviqMU+s5dDA=', 'Content-Length': '0'} 2024-04-16T19:56:24.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cac092e126e3479-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.062 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.063 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.063 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.063 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.063 DEBUG:boto:Method: GET 2024-04-16T19:56:24.063 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.063 DEBUG:boto:Data: 2024-04-16T19:56:24.063 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.063 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.063 DEBUG:boto:Params: {} 2024-04-16T19:56:24.064 DEBUG:boto:Token: None 2024-04-16T19:56:24.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026aa2f0d4345ebb6-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.067 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.067 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.067 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.068 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.068 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.068 DEBUG:boto:Method: GET 2024-04-16T19:56:24.068 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000005 2024-04-16T19:56:24.068 DEBUG:boto:Data: 2024-04-16T19:56:24.068 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.068 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.068 DEBUG:boto:Params: {} 2024-04-16T19:56:24.068 DEBUG:boto:Token: None 2024-04-16T19:56:24.069 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.069 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000192e5909fef9088-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.073 DEBUG:boto:b'yrjefh-51obj000000000000000000051000falseobj000000000000000000052024-04-16T19:56:08.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:56:24.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.073 DEBUG:boto:Method: GET 2024-04-16T19:56:24.074 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.074 DEBUG:boto:Data: 2024-04-16T19:56:24.074 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.074 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.074 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.074 DEBUG:boto:Params: {} 2024-04-16T19:56:24.074 DEBUG:boto:Token: None 2024-04-16T19:56:24.074 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8= 2024-04-16T19:56:24.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8=', 'Content-Length': '0'} 2024-04-16T19:56:24.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3677de1319028c-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.118 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.119 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.119 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.119 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.119 DEBUG:boto:Method: GET 2024-04-16T19:56:24.119 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.119 DEBUG:boto:Data: 2024-04-16T19:56:24.119 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.119 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.119 DEBUG:boto:Params: {} 2024-04-16T19:56:24.120 DEBUG:boto:Token: None 2024-04-16T19:56:24.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8= 2024-04-16T19:56:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8=', 'Content-Length': '0'} 2024-04-16T19:56:24.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38e96adde555daa-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.122 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.123 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.123 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.123 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.123 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.123 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.123 DEBUG:boto:Data: 2024-04-16T19:56:24.123 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.123 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.123 DEBUG:boto:Params: {} 2024-04-16T19:56:24.124 DEBUG:boto:Token: None 2024-04-16T19:56:24.124 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/gBHNUp5Q8XV2G5PRHk/TiQkI/8= 2024-04-16T19:56:24.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/gBHNUp5Q8XV2G5PRHk/TiQkI/8=', 'Content-Length': '0'} 2024-04-16T19:56:24.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25d37aa02717d68-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.126 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.127 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.127 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.127 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.127 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.127 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.127 DEBUG:boto:Data: 2024-04-16T19:56:24.127 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.127 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.127 DEBUG:boto:Params: {} 2024-04-16T19:56:24.128 DEBUG:boto:Token: None 2024-04-16T19:56:24.128 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/gBHNUp5Q8XV2G5PRHk/TiQkI/8= 2024-04-16T19:56:24.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/gBHNUp5Q8XV2G5PRHk/TiQkI/8=', 'Content-Length': '0'} 2024-04-16T19:56:24.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f02e89973e244b-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:56:24.130 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.130 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.131 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.131 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.131 DEBUG:boto:Method: GET 2024-04-16T19:56:24.131 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.131 DEBUG:boto:Data: 2024-04-16T19:56:24.131 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.131 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.131 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.131 DEBUG:boto:Params: {} 2024-04-16T19:56:24.131 DEBUG:boto:Token: None 2024-04-16T19:56:24.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8= 2024-04-16T19:56:24.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8=', 'Content-Length': '0'} 2024-04-16T19:56:24.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbd8a62029c919e-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.134 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.134 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.135 DEBUG:boto:Method: GET 2024-04-16T19:56:24.135 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.135 DEBUG:boto:Data: 2024-04-16T19:56:24.135 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.135 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.135 DEBUG:boto:Params: {} 2024-04-16T19:56:24.135 DEBUG:boto:Token: None 2024-04-16T19:56:24.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006?versionId=null 2024-04-16T19:56:24.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8= 2024-04-16T19:56:24.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp5jnBYa4QxmZZuCY8vOSMSDIu8=', 'Content-Length': '0'} 2024-04-16T19:56:24.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d80ffedea41144c-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.138 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.138 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.138 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.138 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.138 DEBUG:boto:Method: GET 2024-04-16T19:56:24.139 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.139 DEBUG:boto:Data: 2024-04-16T19:56:24.139 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.139 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.139 DEBUG:boto:Params: {} 2024-04-16T19:56:24.139 DEBUG:boto:Token: None 2024-04-16T19:56:24.139 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051268fed30cd3b57-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.142 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.143 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.143 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.143 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.143 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.143 DEBUG:boto:Method: GET 2024-04-16T19:56:24.143 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.143 DEBUG:boto:Data: 2024-04-16T19:56:24.143 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.144 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.144 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.144 DEBUG:boto:Params: {} 2024-04-16T19:56:24.144 DEBUG:boto:Token: None 2024-04-16T19:56:24.144 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082597d2df8ef0a8b-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.148 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.148 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.148 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.149 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.149 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.149 DEBUG:boto:Data: 2024-04-16T19:56:24.149 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.149 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.149 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.149 DEBUG:boto:Params: {} 2024-04-16T19:56:24.149 DEBUG:boto:Token: None 2024-04-16T19:56:24.149 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.150 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t/H4pxPpARNN2bBsEOkNE/3y5sQ= 2024-04-16T19:56:24.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t/H4pxPpARNN2bBsEOkNE/3y5sQ=', 'Content-Length': '0'} 2024-04-16T19:56:24.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6349bb0f6e4526e-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.152 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.152 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.152 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.152 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.152 DEBUG:boto:Data: 2024-04-16T19:56:24.153 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.153 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.153 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.153 DEBUG:boto:Params: {} 2024-04-16T19:56:24.153 DEBUG:boto:Token: None 2024-04-16T19:56:24.153 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t/H4pxPpARNN2bBsEOkNE/3y5sQ= 2024-04-16T19:56:24.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t/H4pxPpARNN2bBsEOkNE/3y5sQ=', 'Content-Length': '0'} 2024-04-16T19:56:24.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490e4c56ef33e7ae-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-16T19:56:24.155 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.156 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.156 DEBUG:boto:Method: GET 2024-04-16T19:56:24.156 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.156 DEBUG:boto:Data: 2024-04-16T19:56:24.156 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.156 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.156 DEBUG:boto:Params: {} 2024-04-16T19:56:24.156 DEBUG:boto:Token: None 2024-04-16T19:56:24.157 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PS+qSqLUl/yNjI6HP68V0zE/MDY= 2024-04-16T19:56:24.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PS+qSqLUl/yNjI6HP68V0zE/MDY=', 'Content-Length': '0'} 2024-04-16T19:56:24.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6519e5817a22959-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.159 DEBUG:boto:Method: GET 2024-04-16T19:56:24.159 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.159 DEBUG:boto:Data: 2024-04-16T19:56:24.159 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.160 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.160 DEBUG:boto:Params: {} 2024-04-16T19:56:24.160 DEBUG:boto:Token: None 2024-04-16T19:56:24.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000006 2024-04-16T19:56:24.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PS+qSqLUl/yNjI6HP68V0zE/MDY= 2024-04-16T19:56:24.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PS+qSqLUl/yNjI6HP68V0zE/MDY=', 'Content-Length': '0'} 2024-04-16T19:56:24.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1245e9bf0f0420c-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.202 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.202 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.203 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.203 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.203 DEBUG:boto:Method: GET 2024-04-16T19:56:24.203 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.203 DEBUG:boto:Data: 2024-04-16T19:56:24.203 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.203 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.203 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.203 DEBUG:boto:Params: {} 2024-04-16T19:56:24.204 DEBUG:boto:Token: None 2024-04-16T19:56:24.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086442003ac1560c6-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.207 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.207 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.207 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.207 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.207 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.207 DEBUG:boto:Method: GET 2024-04-16T19:56:24.208 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000006 2024-04-16T19:56:24.208 DEBUG:boto:Data: 2024-04-16T19:56:24.208 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.208 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.208 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.208 DEBUG:boto:Params: {} 2024-04-16T19:56:24.208 DEBUG:boto:Token: None 2024-04-16T19:56:24.208 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3d8a102571d2f8-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.212 DEBUG:boto:b'yrjefh-51obj000000000000000000061000falseobj000000000000000000062024-04-16T19:56:08.968Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-16T19:56:24.213 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.213 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.213 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.213 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.213 DEBUG:boto:Method: GET 2024-04-16T19:56:24.214 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.214 DEBUG:boto:Data: 2024-04-16T19:56:24.214 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.214 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.214 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.214 DEBUG:boto:Params: {} 2024-04-16T19:56:24.214 DEBUG:boto:Token: None 2024-04-16T19:56:24.214 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE= 2024-04-16T19:56:24.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE=', 'Content-Length': '0'} 2024-04-16T19:56:24.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bdfd00f564fc11-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.258 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.259 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.259 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.259 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.259 DEBUG:boto:Method: GET 2024-04-16T19:56:24.259 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.259 DEBUG:boto:Data: 2024-04-16T19:56:24.259 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.259 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.260 DEBUG:boto:Params: {} 2024-04-16T19:56:24.260 DEBUG:boto:Token: None 2024-04-16T19:56:24.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE= 2024-04-16T19:56:24.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE=', 'Content-Length': '0'} 2024-04-16T19:56:24.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337196fcff19d432-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.263 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.263 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.263 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.263 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.263 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.263 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.263 DEBUG:boto:Data: 2024-04-16T19:56:24.263 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.264 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.264 DEBUG:boto:Params: {} 2024-04-16T19:56:24.264 DEBUG:boto:Token: None 2024-04-16T19:56:24.264 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2acBk7J/xmPAFgvEeC99dhU88Pw= 2024-04-16T19:56:24.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2acBk7J/xmPAFgvEeC99dhU88Pw=', 'Content-Length': '0'} 2024-04-16T19:56:24.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc3fadbd00575b9-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.267 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.267 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.267 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.267 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.267 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.267 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.267 DEBUG:boto:Data: 2024-04-16T19:56:24.267 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.268 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.268 DEBUG:boto:Params: {} 2024-04-16T19:56:24.268 DEBUG:boto:Token: None 2024-04-16T19:56:24.268 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2acBk7J/xmPAFgvEeC99dhU88Pw= 2024-04-16T19:56:24.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2acBk7J/xmPAFgvEeC99dhU88Pw=', 'Content-Length': '0'} 2024-04-16T19:56:24.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9bf8ff75841ab8-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-16T19:56:24.271 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.271 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.271 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.271 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.271 DEBUG:boto:Method: GET 2024-04-16T19:56:24.271 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.271 DEBUG:boto:Data: 2024-04-16T19:56:24.271 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.272 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.272 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.272 DEBUG:boto:Params: {} 2024-04-16T19:56:24.272 DEBUG:boto:Token: None 2024-04-16T19:56:24.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE= 2024-04-16T19:56:24.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE=', 'Content-Length': '0'} 2024-04-16T19:56:24.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c318d18e186b95a7-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.275 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.275 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.275 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.275 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.275 DEBUG:boto:Method: GET 2024-04-16T19:56:24.275 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.275 DEBUG:boto:Data: 2024-04-16T19:56:24.275 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.276 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.276 DEBUG:boto:Params: {} 2024-04-16T19:56:24.276 DEBUG:boto:Token: None 2024-04-16T19:56:24.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007?versionId=null 2024-04-16T19:56:24.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE= 2024-04-16T19:56:24.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9WrRZGALxUYyr+ARVwKoL9rBBSE=', 'Content-Length': '0'} 2024-04-16T19:56:24.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e08682ed3be4d7f2-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.279 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.279 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.279 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.279 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.279 DEBUG:boto:Method: GET 2024-04-16T19:56:24.279 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.279 DEBUG:boto:Data: 2024-04-16T19:56:24.279 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.280 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.280 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.280 DEBUG:boto:Params: {} 2024-04-16T19:56:24.280 DEBUG:boto:Token: None 2024-04-16T19:56:24.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4f6660e45aaf8f-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.283 DEBUG:boto:b'yrjefh-51obj000000000000000000071000falseobj000000000000000000072024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.283 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.284 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.284 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.284 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.284 DEBUG:boto:Method: GET 2024-04-16T19:56:24.284 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.284 DEBUG:boto:Data: 2024-04-16T19:56:24.284 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.284 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.284 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.285 DEBUG:boto:Params: {} 2024-04-16T19:56:24.285 DEBUG:boto:Token: None 2024-04-16T19:56:24.285 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.285 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b29e3f149c0d5d2-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.330 DEBUG:boto:b'yrjefh-51obj000000000000000000071000falseobj000000000000000000072024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.331 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.331 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.331 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.331 DEBUG:boto:Data: 2024-04-16T19:56:24.332 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.332 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.332 DEBUG:boto:Params: {} 2024-04-16T19:56:24.332 DEBUG:boto:Token: None 2024-04-16T19:56:24.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8n2yNxg45XrH1jf2D0AhPhhzYfs= 2024-04-16T19:56:24.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8n2yNxg45XrH1jf2D0AhPhhzYfs=', 'Content-Length': '0'} 2024-04-16T19:56:24.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df67fea74b62c890-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.335 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.335 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.335 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.335 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.335 DEBUG:boto:Data: 2024-04-16T19:56:24.335 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.336 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.336 DEBUG:boto:Params: {} 2024-04-16T19:56:24.336 DEBUG:boto:Token: None 2024-04-16T19:56:24.336 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8n2yNxg45XrH1jf2D0AhPhhzYfs= 2024-04-16T19:56:24.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8n2yNxg45XrH1jf2D0AhPhhzYfs=', 'Content-Length': '0'} 2024-04-16T19:56:24.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b534c0ac638d42-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-16T19:56:24.339 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.339 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.339 DEBUG:boto:Method: GET 2024-04-16T19:56:24.339 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.340 DEBUG:boto:Data: 2024-04-16T19:56:24.340 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.340 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.340 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.340 DEBUG:boto:Params: {} 2024-04-16T19:56:24.340 DEBUG:boto:Token: None 2024-04-16T19:56:24.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.341 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCC6a8wkPX0RrjDkWZYu8m0gJEo= 2024-04-16T19:56:24.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCC6a8wkPX0RrjDkWZYu8m0gJEo=', 'Content-Length': '0'} 2024-04-16T19:56:24.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7ac7dc99c7b306-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.386 DEBUG:boto:path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.387 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.387 DEBUG:boto:Method: GET 2024-04-16T19:56:24.387 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.387 DEBUG:boto:Data: 2024-04-16T19:56:24.387 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.388 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.388 DEBUG:boto:Params: {} 2024-04-16T19:56:24.388 DEBUG:boto:Token: None 2024-04-16T19:56:24.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000007 2024-04-16T19:56:24.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wCC6a8wkPX0RrjDkWZYu8m0gJEo= 2024-04-16T19:56:24.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wCC6a8wkPX0RrjDkWZYu8m0gJEo=', 'Content-Length': '0'} 2024-04-16T19:56:24.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35cadefc5cfe323-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.391 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.391 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.391 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.392 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.392 DEBUG:boto:Method: GET 2024-04-16T19:56:24.392 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.392 DEBUG:boto:Data: 2024-04-16T19:56:24.392 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.392 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.392 DEBUG:boto:Params: {} 2024-04-16T19:56:24.392 DEBUG:boto:Token: None 2024-04-16T19:56:24.393 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032abfd6328619ee3-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.396 DEBUG:boto:b'yrjefh-51obj000000000000000000071000falseobj000000000000000000072024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.396 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.396 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.397 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.397 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.397 DEBUG:boto:Method: GET 2024-04-16T19:56:24.397 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000007 2024-04-16T19:56:24.397 DEBUG:boto:Data: 2024-04-16T19:56:24.397 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.397 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.397 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.397 DEBUG:boto:Params: {} 2024-04-16T19:56:24.398 DEBUG:boto:Token: None 2024-04-16T19:56:24.398 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.398 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f588355e2cb296ce-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.402 DEBUG:boto:b'yrjefh-51obj000000000000000000071000falseobj000000000000000000072024-04-16T19:56:09.263Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-16T19:56:24.402 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.403 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.403 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.403 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.403 DEBUG:boto:Method: GET 2024-04-16T19:56:24.403 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.403 DEBUG:boto:Data: 2024-04-16T19:56:24.403 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.403 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.403 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.404 DEBUG:boto:Params: {} 2024-04-16T19:56:24.404 DEBUG:boto:Token: None 2024-04-16T19:56:24.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg= 2024-04-16T19:56:24.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg=', 'Content-Length': '0'} 2024-04-16T19:56:24.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b63322f2d5b2df9-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.408 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.408 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.408 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.409 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.409 DEBUG:boto:Method: GET 2024-04-16T19:56:24.409 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.409 DEBUG:boto:Data: 2024-04-16T19:56:24.409 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.409 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.409 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.409 DEBUG:boto:Params: {} 2024-04-16T19:56:24.409 DEBUG:boto:Token: None 2024-04-16T19:56:24.410 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.410 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg= 2024-04-16T19:56:24.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg=', 'Content-Length': '0'} 2024-04-16T19:56:24.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030ac4e0e3dc0f29-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.413 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.413 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.413 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.414 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.414 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.414 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.414 DEBUG:boto:Data: 2024-04-16T19:56:24.414 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.414 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.414 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.414 DEBUG:boto:Params: {} 2024-04-16T19:56:24.415 DEBUG:boto:Token: None 2024-04-16T19:56:24.415 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tpt4wJ2NzsIRGmBbttW06J5gsHU= 2024-04-16T19:56:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tpt4wJ2NzsIRGmBbttW06J5gsHU=', 'Content-Length': '0'} 2024-04-16T19:56:24.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ced24f6f678dcc8-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.417 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.417 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.418 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.418 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.418 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.418 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.418 DEBUG:boto:Data: 2024-04-16T19:56:24.418 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.418 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.418 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.419 DEBUG:boto:Params: {} 2024-04-16T19:56:24.419 DEBUG:boto:Token: None 2024-04-16T19:56:24.419 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tpt4wJ2NzsIRGmBbttW06J5gsHU= 2024-04-16T19:56:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tpt4wJ2NzsIRGmBbttW06J5gsHU=', 'Content-Length': '0'} 2024-04-16T19:56:24.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ec6412838bc28d-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-16T19:56:24.422 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.422 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.422 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.423 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.423 DEBUG:boto:Method: GET 2024-04-16T19:56:24.423 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.423 DEBUG:boto:Data: 2024-04-16T19:56:24.423 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.423 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.423 DEBUG:boto:Params: {} 2024-04-16T19:56:24.424 DEBUG:boto:Token: None 2024-04-16T19:56:24.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg= 2024-04-16T19:56:24.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg=', 'Content-Length': '0'} 2024-04-16T19:56:24.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000336d31653dd7efe3-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.470 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.471 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.471 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.471 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.471 DEBUG:boto:Method: GET 2024-04-16T19:56:24.471 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.471 DEBUG:boto:Data: 2024-04-16T19:56:24.471 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.472 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.472 DEBUG:boto:Params: {} 2024-04-16T19:56:24.472 DEBUG:boto:Token: None 2024-04-16T19:56:24.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008?versionId=null 2024-04-16T19:56:24.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg= 2024-04-16T19:56:24.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U75kRVKjySvN22PCfjsNnK85ueg=', 'Content-Length': '0'} 2024-04-16T19:56:24.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bbf65bbb6635ced-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.475 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.475 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.475 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.475 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.475 DEBUG:boto:Method: GET 2024-04-16T19:56:24.475 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.475 DEBUG:boto:Data: 2024-04-16T19:56:24.475 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.476 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.476 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.476 DEBUG:boto:Params: {} 2024-04-16T19:56:24.476 DEBUG:boto:Token: None 2024-04-16T19:56:24.476 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e233b06fd656ea6-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.479 DEBUG:boto:b'yrjefh-51obj000000000000000000081000falseobj000000000000000000082024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.480 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.480 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.480 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.480 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.480 DEBUG:boto:Method: GET 2024-04-16T19:56:24.480 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.480 DEBUG:boto:Data: 2024-04-16T19:56:24.480 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.481 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.481 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.481 DEBUG:boto:Params: {} 2024-04-16T19:56:24.481 DEBUG:boto:Token: None 2024-04-16T19:56:24.481 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf62e62b8e61eea-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.530 DEBUG:boto:b'yrjefh-51obj000000000000000000081000falseobj000000000000000000082024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.531 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.531 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.531 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.531 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.531 DEBUG:boto:Data: 2024-04-16T19:56:24.531 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.532 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.532 DEBUG:boto:Params: {} 2024-04-16T19:56:24.532 DEBUG:boto:Token: None 2024-04-16T19:56:24.532 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0spUg7cSOYq6fqk05BvJjZomdYI= 2024-04-16T19:56:24.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0spUg7cSOYq6fqk05BvJjZomdYI=', 'Content-Length': '0'} 2024-04-16T19:56:24.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002feecc2f0d83c70c-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.535 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.535 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.535 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.535 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.535 DEBUG:boto:Data: 2024-04-16T19:56:24.535 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.536 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.536 DEBUG:boto:Params: {} 2024-04-16T19:56:24.536 DEBUG:boto:Token: None 2024-04-16T19:56:24.536 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0spUg7cSOYq6fqk05BvJjZomdYI= 2024-04-16T19:56:24.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0spUg7cSOYq6fqk05BvJjZomdYI=', 'Content-Length': '0'} 2024-04-16T19:56:24.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d46aea959cf5ba16-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-16T19:56:24.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.539 DEBUG:boto:Method: GET 2024-04-16T19:56:24.539 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.539 DEBUG:boto:Data: 2024-04-16T19:56:24.539 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.540 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.540 DEBUG:boto:Params: {} 2024-04-16T19:56:24.540 DEBUG:boto:Token: None 2024-04-16T19:56:24.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rGApFaImywK/Do+SSaBT2EPmK98= 2024-04-16T19:56:24.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rGApFaImywK/Do+SSaBT2EPmK98=', 'Content-Length': '0'} 2024-04-16T19:56:24.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75a5dd81e2867e1-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.543 DEBUG:boto:path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.543 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.543 DEBUG:boto:Method: GET 2024-04-16T19:56:24.543 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.543 DEBUG:boto:Data: 2024-04-16T19:56:24.543 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.544 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.544 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.544 DEBUG:boto:Params: {} 2024-04-16T19:56:24.544 DEBUG:boto:Token: None 2024-04-16T19:56:24.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000008 2024-04-16T19:56:24.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rGApFaImywK/Do+SSaBT2EPmK98= 2024-04-16T19:56:24.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rGApFaImywK/Do+SSaBT2EPmK98=', 'Content-Length': '0'} 2024-04-16T19:56:24.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbe606a90611ba3-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.547 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.547 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.547 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.547 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.547 DEBUG:boto:Method: GET 2024-04-16T19:56:24.548 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.548 DEBUG:boto:Data: 2024-04-16T19:56:24.548 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.548 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.548 DEBUG:boto:Params: {} 2024-04-16T19:56:24.548 DEBUG:boto:Token: None 2024-04-16T19:56:24.548 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cc897fed800c3c-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.594 DEBUG:boto:b'yrjefh-51obj000000000000000000081000falseobj000000000000000000082024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.595 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.595 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.595 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.595 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.595 DEBUG:boto:Method: GET 2024-04-16T19:56:24.595 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000008 2024-04-16T19:56:24.595 DEBUG:boto:Data: 2024-04-16T19:56:24.596 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.596 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.596 DEBUG:boto:Params: {} 2024-04-16T19:56:24.596 DEBUG:boto:Token: None 2024-04-16T19:56:24.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db5436757e4d922-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.601 DEBUG:boto:b'yrjefh-51obj000000000000000000081000falseobj000000000000000000082024-04-16T19:56:07.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-16T19:56:24.601 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.601 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.601 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.602 DEBUG:boto:Method: GET 2024-04-16T19:56:24.602 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.602 DEBUG:boto:Data: 2024-04-16T19:56:24.602 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.602 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.602 DEBUG:boto:Params: {} 2024-04-16T19:56:24.603 DEBUG:boto:Token: None 2024-04-16T19:56:24.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc= 2024-04-16T19:56:24.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc=', 'Content-Length': '0'} 2024-04-16T19:56:24.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866fdc8b9a337843-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.606 DEBUG:boto:Method: GET 2024-04-16T19:56:24.606 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.606 DEBUG:boto:Data: 2024-04-16T19:56:24.606 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.607 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.607 DEBUG:boto:Params: {} 2024-04-16T19:56:24.607 DEBUG:boto:Token: None 2024-04-16T19:56:24.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc= 2024-04-16T19:56:24.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc=', 'Content-Length': '0'} 2024-04-16T19:56:24.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000874c61be321ad13e-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.650 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.651 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.651 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.651 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.651 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.651 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.651 DEBUG:boto:Data: 2024-04-16T19:56:24.651 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.652 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.652 DEBUG:boto:Params: {} 2024-04-16T19:56:24.652 DEBUG:boto:Token: None 2024-04-16T19:56:24.652 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A58JJB9flCQMF180VairJlCe0wU= 2024-04-16T19:56:24.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A58JJB9flCQMF180VairJlCe0wU=', 'Content-Length': '0'} 2024-04-16T19:56:24.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af36bf45a9cb610-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.655 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.655 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.656 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.656 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.656 DEBUG:boto:Data: 2024-04-16T19:56:24.656 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.656 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.656 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.656 DEBUG:boto:Params: {} 2024-04-16T19:56:24.656 DEBUG:boto:Token: None 2024-04-16T19:56:24.657 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.657 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A58JJB9flCQMF180VairJlCe0wU= 2024-04-16T19:56:24.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A58JJB9flCQMF180VairJlCe0wU=', 'Content-Length': '0'} 2024-04-16T19:56:24.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff2078145cd90b41-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-16T19:56:24.660 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.660 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.660 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.660 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.660 DEBUG:boto:Method: GET 2024-04-16T19:56:24.660 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.660 DEBUG:boto:Data: 2024-04-16T19:56:24.660 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.661 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.661 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.661 DEBUG:boto:Params: {} 2024-04-16T19:56:24.661 DEBUG:boto:Token: None 2024-04-16T19:56:24.661 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.661 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc= 2024-04-16T19:56:24.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc=', 'Content-Length': '0'} 2024-04-16T19:56:24.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b15588c207dd85-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.664 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.664 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.664 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.664 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.664 DEBUG:boto:Method: GET 2024-04-16T19:56:24.664 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.664 DEBUG:boto:Data: 2024-04-16T19:56:24.665 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.665 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.665 DEBUG:boto:Params: {} 2024-04-16T19:56:24.665 DEBUG:boto:Token: None 2024-04-16T19:56:24.665 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009?versionId=null 2024-04-16T19:56:24.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc= 2024-04-16T19:56:24.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TsuBjaoKaxRef4tO5J05b7NZNPc=', 'Content-Length': '0'} 2024-04-16T19:56:24.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b29aed34ea1326-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.668 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.668 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.668 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.668 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.669 DEBUG:boto:Method: GET 2024-04-16T19:56:24.669 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.669 DEBUG:boto:Data: 2024-04-16T19:56:24.669 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.669 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.669 DEBUG:boto:Params: {} 2024-04-16T19:56:24.669 DEBUG:boto:Token: None 2024-04-16T19:56:24.670 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d55d9df614e7f674-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.714 DEBUG:boto:b'yrjefh-51obj000000000000000000091000falseobj000000000000000000092024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.715 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.715 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.715 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.715 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.715 DEBUG:boto:Method: GET 2024-04-16T19:56:24.715 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.715 DEBUG:boto:Data: 2024-04-16T19:56:24.716 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.716 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.716 DEBUG:boto:Params: {} 2024-04-16T19:56:24.716 DEBUG:boto:Token: None 2024-04-16T19:56:24.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009636f32c8a5475fc-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.721 DEBUG:boto:b'yrjefh-51obj000000000000000000091000falseobj000000000000000000092024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.721 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.721 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.722 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.722 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.722 DEBUG:boto:Data: 2024-04-16T19:56:24.722 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.722 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.722 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.722 DEBUG:boto:Params: {} 2024-04-16T19:56:24.722 DEBUG:boto:Token: None 2024-04-16T19:56:24.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8KbR941hXEmFJqzWlo9rLMK0c= 2024-04-16T19:56:24.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8KbR941hXEmFJqzWlo9rLMK0c=', 'Content-Length': '0'} 2024-04-16T19:56:24.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3371f22631da60-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.725 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.725 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.725 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.725 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.725 DEBUG:boto:Data: 2024-04-16T19:56:24.726 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.726 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.726 DEBUG:boto:Params: {} 2024-04-16T19:56:24.726 DEBUG:boto:Token: None 2024-04-16T19:56:24.726 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.726 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8KbR941hXEmFJqzWlo9rLMK0c= 2024-04-16T19:56:24.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8KbR941hXEmFJqzWlo9rLMK0c=', 'Content-Length': '0'} 2024-04-16T19:56:24.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac24d5ec9c36e90-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-16T19:56:24.729 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.729 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.729 DEBUG:boto:Method: GET 2024-04-16T19:56:24.729 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.729 DEBUG:boto:Data: 2024-04-16T19:56:24.729 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.729 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.729 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.730 DEBUG:boto:Params: {} 2024-04-16T19:56:24.730 DEBUG:boto:Token: None 2024-04-16T19:56:24.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KnGai66nNYDUhCJkVr7DrAUQyHs= 2024-04-16T19:56:24.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KnGai66nNYDUhCJkVr7DrAUQyHs=', 'Content-Length': '0'} 2024-04-16T19:56:24.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e584435206b22943-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.733 DEBUG:boto:path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.733 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.733 DEBUG:boto:Method: GET 2024-04-16T19:56:24.733 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.733 DEBUG:boto:Data: 2024-04-16T19:56:24.733 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.734 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.734 DEBUG:boto:Params: {} 2024-04-16T19:56:24.734 DEBUG:boto:Token: None 2024-04-16T19:56:24.734 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000009 2024-04-16T19:56:24.734 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KnGai66nNYDUhCJkVr7DrAUQyHs= 2024-04-16T19:56:24.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KnGai66nNYDUhCJkVr7DrAUQyHs=', 'Content-Length': '0'} 2024-04-16T19:56:24.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d133b7b5f011c53-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.778 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.779 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.779 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.779 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.779 DEBUG:boto:Method: GET 2024-04-16T19:56:24.779 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.779 DEBUG:boto:Data: 2024-04-16T19:56:24.779 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.779 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.779 DEBUG:boto:Params: {} 2024-04-16T19:56:24.780 DEBUG:boto:Token: None 2024-04-16T19:56:24.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011331de6a113eb6b-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.783 DEBUG:boto:b'yrjefh-51obj000000000000000000091000falseobj000000000000000000092024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.783 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.783 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.783 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.784 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.784 DEBUG:boto:Method: GET 2024-04-16T19:56:24.784 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000009 2024-04-16T19:56:24.784 DEBUG:boto:Data: 2024-04-16T19:56:24.784 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.784 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.784 DEBUG:boto:Params: {} 2024-04-16T19:56:24.784 DEBUG:boto:Token: None 2024-04-16T19:56:24.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.785 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681fd0a1e0f6e7e0-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.788 DEBUG:boto:b'yrjefh-51obj000000000000000000091000falseobj000000000000000000092024-04-16T19:56:08.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:56:24.789 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.789 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.789 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.789 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.789 DEBUG:boto:Method: GET 2024-04-16T19:56:24.790 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.790 DEBUG:boto:Data: 2024-04-16T19:56:24.790 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.790 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.790 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.790 DEBUG:boto:Params: {} 2024-04-16T19:56:24.790 DEBUG:boto:Token: None 2024-04-16T19:56:24.790 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs= 2024-04-16T19:56:24.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs=', 'Content-Length': '0'} 2024-04-16T19:56:24.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa1ed66174a0985f-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.793 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.794 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.794 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.794 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.794 DEBUG:boto:Method: GET 2024-04-16T19:56:24.794 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.794 DEBUG:boto:Data: 2024-04-16T19:56:24.794 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.794 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.795 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.795 DEBUG:boto:Params: {} 2024-04-16T19:56:24.795 DEBUG:boto:Token: None 2024-04-16T19:56:24.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs= 2024-04-16T19:56:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs=', 'Content-Length': '0'} 2024-04-16T19:56:24.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce15dceec29e05c-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.798 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.798 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.798 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.798 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.798 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.798 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.799 DEBUG:boto:Data: 2024-04-16T19:56:24.799 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.799 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.799 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.799 DEBUG:boto:Params: {} 2024-04-16T19:56:24.799 DEBUG:boto:Token: None 2024-04-16T19:56:24.799 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bq2Ch0dF8ZyuDmVbLDsb0BHr6hY= 2024-04-16T19:56:24.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bq2Ch0dF8ZyuDmVbLDsb0BHr6hY=', 'Content-Length': '0'} 2024-04-16T19:56:24.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10ef96217f4a341-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.802 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.802 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.802 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.802 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.802 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.802 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.802 DEBUG:boto:Data: 2024-04-16T19:56:24.803 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.803 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.803 DEBUG:boto:Params: {} 2024-04-16T19:56:24.803 DEBUG:boto:Token: None 2024-04-16T19:56:24.803 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bq2Ch0dF8ZyuDmVbLDsb0BHr6hY= 2024-04-16T19:56:24.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bq2Ch0dF8ZyuDmVbLDsb0BHr6hY=', 'Content-Length': '0'} 2024-04-16T19:56:24.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d3e4885f344b1b-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:56:24.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.806 DEBUG:boto:Method: GET 2024-04-16T19:56:24.806 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.806 DEBUG:boto:Data: 2024-04-16T19:56:24.807 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.807 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.807 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.807 DEBUG:boto:Params: {} 2024-04-16T19:56:24.807 DEBUG:boto:Token: None 2024-04-16T19:56:24.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs= 2024-04-16T19:56:24.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs=', 'Content-Length': '0'} 2024-04-16T19:56:24.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000430235ad801cd9a-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.810 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.810 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.810 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.810 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.810 DEBUG:boto:Method: GET 2024-04-16T19:56:24.810 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.810 DEBUG:boto:Data: 2024-04-16T19:56:24.810 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.811 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.811 DEBUG:boto:Params: {} 2024-04-16T19:56:24.811 DEBUG:boto:Token: None 2024-04-16T19:56:24.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010?versionId=null 2024-04-16T19:56:24.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs= 2024-04-16T19:56:24.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mNIL0Yj84z4fd1Lybf5CBfpyxTs=', 'Content-Length': '0'} 2024-04-16T19:56:24.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052ab87472bcfcaea-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.814 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.814 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.814 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.814 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.815 DEBUG:boto:Method: GET 2024-04-16T19:56:24.815 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.815 DEBUG:boto:Data: 2024-04-16T19:56:24.815 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.815 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.815 DEBUG:boto:Params: {} 2024-04-16T19:56:24.815 DEBUG:boto:Token: None 2024-04-16T19:56:24.815 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004793888d2f86efc5-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.818 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.819 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.819 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.819 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.819 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.819 DEBUG:boto:Method: GET 2024-04-16T19:56:24.819 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.819 DEBUG:boto:Data: 2024-04-16T19:56:24.819 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.820 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.820 DEBUG:boto:Params: {} 2024-04-16T19:56:24.820 DEBUG:boto:Token: None 2024-04-16T19:56:24.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88261ec8dd024e7-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.825 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.825 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.825 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.825 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.825 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.826 DEBUG:boto:Data: 2024-04-16T19:56:24.826 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.826 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.826 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.826 DEBUG:boto:Params: {} 2024-04-16T19:56:24.826 DEBUG:boto:Token: None 2024-04-16T19:56:24.826 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.826 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UD+fQ72xS1H/KvmLcjJP0/sg2QQ= 2024-04-16T19:56:24.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UD+fQ72xS1H/KvmLcjJP0/sg2QQ=', 'Content-Length': '0'} 2024-04-16T19:56:24.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9ae8c673e6958c-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.828 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.829 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.829 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.829 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.829 DEBUG:boto:Data: 2024-04-16T19:56:24.829 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.829 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.829 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.829 DEBUG:boto:Params: {} 2024-04-16T19:56:24.829 DEBUG:boto:Token: None 2024-04-16T19:56:24.830 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UD+fQ72xS1H/KvmLcjJP0/sg2QQ= 2024-04-16T19:56:24.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UD+fQ72xS1H/KvmLcjJP0/sg2QQ=', 'Content-Length': '0'} 2024-04-16T19:56:24.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030944dc5eeab825c-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-16T19:56:24.832 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.832 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.832 DEBUG:boto:Method: GET 2024-04-16T19:56:24.832 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.833 DEBUG:boto:Data: 2024-04-16T19:56:24.833 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.833 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.833 DEBUG:boto:Params: {} 2024-04-16T19:56:24.833 DEBUG:boto:Token: None 2024-04-16T19:56:24.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cl/+oAQNbM4yPDVLav9QvQzytQ0= 2024-04-16T19:56:24.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cl/+oAQNbM4yPDVLav9QvQzytQ0=', 'Content-Length': '0'} 2024-04-16T19:56:24.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8568e00f3dacd9-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.878 DEBUG:boto:path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.879 DEBUG:boto:Method: GET 2024-04-16T19:56:24.879 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.879 DEBUG:boto:Data: 2024-04-16T19:56:24.879 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.879 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.879 DEBUG:boto:Params: {} 2024-04-16T19:56:24.879 DEBUG:boto:Token: None 2024-04-16T19:56:24.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000010 2024-04-16T19:56:24.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cl/+oAQNbM4yPDVLav9QvQzytQ0= 2024-04-16T19:56:24.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cl/+oAQNbM4yPDVLav9QvQzytQ0=', 'Content-Length': '0'} 2024-04-16T19:56:24.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec25380f4e9e6bc-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.882 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.882 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.882 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.883 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.883 DEBUG:boto:Method: GET 2024-04-16T19:56:24.883 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.883 DEBUG:boto:Data: 2024-04-16T19:56:24.883 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.883 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.883 DEBUG:boto:Params: {} 2024-04-16T19:56:24.883 DEBUG:boto:Token: None 2024-04-16T19:56:24.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfc7199991b6765-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.886 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.887 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.887 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.887 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.887 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.887 DEBUG:boto:Method: GET 2024-04-16T19:56:24.887 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000010 2024-04-16T19:56:24.887 DEBUG:boto:Data: 2024-04-16T19:56:24.887 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.888 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.888 DEBUG:boto:Params: {} 2024-04-16T19:56:24.888 DEBUG:boto:Token: None 2024-04-16T19:56:24.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000556b98434005ec3a-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.892 DEBUG:boto:b'yrjefh-51obj000000000000000000101000falseobj000000000000000000102024-04-16T19:56:08.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:56:24.893 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.893 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.893 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.893 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.893 DEBUG:boto:Method: GET 2024-04-16T19:56:24.893 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.893 DEBUG:boto:Data: 2024-04-16T19:56:24.893 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.894 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.894 DEBUG:boto:Params: {} 2024-04-16T19:56:24.894 DEBUG:boto:Token: None 2024-04-16T19:56:24.894 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4= 2024-04-16T19:56:24.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4=', 'Content-Length': '0'} 2024-04-16T19:56:24.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2050b814a62ec5-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.897 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.897 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.897 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.897 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.898 DEBUG:boto:Method: GET 2024-04-16T19:56:24.898 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.898 DEBUG:boto:Data: 2024-04-16T19:56:24.898 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.898 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.898 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.898 DEBUG:boto:Params: {} 2024-04-16T19:56:24.898 DEBUG:boto:Token: None 2024-04-16T19:56:24.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4= 2024-04-16T19:56:24.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4=', 'Content-Length': '0'} 2024-04-16T19:56:24.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad921540ed1d2704-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.901 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.901 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.901 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.902 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.902 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.902 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.902 DEBUG:boto:Data: 2024-04-16T19:56:24.902 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.902 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.902 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.902 DEBUG:boto:Params: {} 2024-04-16T19:56:24.902 DEBUG:boto:Token: None 2024-04-16T19:56:24.903 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+prcVlEPy3+gbbahV1CcVnFj0EY= 2024-04-16T19:56:24.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+prcVlEPy3+gbbahV1CcVnFj0EY=', 'Content-Length': '0'} 2024-04-16T19:56:24.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f1d1d20a454192b-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.905 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.905 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.906 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.906 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.906 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.906 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.906 DEBUG:boto:Data: 2024-04-16T19:56:24.906 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.906 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.906 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.906 DEBUG:boto:Params: {} 2024-04-16T19:56:24.907 DEBUG:boto:Token: None 2024-04-16T19:56:24.907 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+prcVlEPy3+gbbahV1CcVnFj0EY= 2024-04-16T19:56:24.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+prcVlEPy3+gbbahV1CcVnFj0EY=', 'Content-Length': '0'} 2024-04-16T19:56:24.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e030883c23c1cd70-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:56:24.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.910 DEBUG:boto:Method: GET 2024-04-16T19:56:24.910 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.910 DEBUG:boto:Data: 2024-04-16T19:56:24.910 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.910 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.910 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.910 DEBUG:boto:Params: {} 2024-04-16T19:56:24.910 DEBUG:boto:Token: None 2024-04-16T19:56:24.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.911 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4= 2024-04-16T19:56:24.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4=', 'Content-Length': '0'} 2024-04-16T19:56:24.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a02d799b926df51-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.913 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.914 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.914 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.914 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.914 DEBUG:boto:Method: GET 2024-04-16T19:56:24.914 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.914 DEBUG:boto:Data: 2024-04-16T19:56:24.914 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.914 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.914 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.915 DEBUG:boto:Params: {} 2024-04-16T19:56:24.915 DEBUG:boto:Token: None 2024-04-16T19:56:24.915 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011?versionId=null 2024-04-16T19:56:24.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4= 2024-04-16T19:56:24.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PkZhANAlRHlJ+/oTLM3AjM4lmX4=', 'Content-Length': '0'} 2024-04-16T19:56:24.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a728fe64cc09af33-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.917 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.918 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.918 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.918 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.918 DEBUG:boto:Method: GET 2024-04-16T19:56:24.918 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.918 DEBUG:boto:Data: 2024-04-16T19:56:24.918 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.918 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.918 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.918 DEBUG:boto:Params: {} 2024-04-16T19:56:24.919 DEBUG:boto:Token: None 2024-04-16T19:56:24.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542fc7dfc918c3b3-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.922 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.922 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.922 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.922 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.922 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.923 DEBUG:boto:Method: GET 2024-04-16T19:56:24.923 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.923 DEBUG:boto:Data: 2024-04-16T19:56:24.923 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.923 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.923 DEBUG:boto:Params: {} 2024-04-16T19:56:24.923 DEBUG:boto:Token: None 2024-04-16T19:56:24.923 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a338146360a74239-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.927 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.928 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.928 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.928 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.928 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.928 DEBUG:boto:Data: 2024-04-16T19:56:24.928 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.928 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.928 DEBUG:boto:Params: {} 2024-04-16T19:56:24.928 DEBUG:boto:Token: None 2024-04-16T19:56:24.929 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CyEuStG7Z1CdfC7hH+5xOG4nIRs= 2024-04-16T19:56:24.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CyEuStG7Z1CdfC7hH+5xOG4nIRs=', 'Content-Length': '0'} 2024-04-16T19:56:24.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929eccd0a4b847b4-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.931 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.932 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.932 DEBUG:boto:Method: HEAD 2024-04-16T19:56:24.932 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.932 DEBUG:boto:Data: 2024-04-16T19:56:24.932 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.932 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.932 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.932 DEBUG:boto:Params: {} 2024-04-16T19:56:24.932 DEBUG:boto:Token: None 2024-04-16T19:56:24.933 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.933 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CyEuStG7Z1CdfC7hH+5xOG4nIRs= 2024-04-16T19:56:24.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CyEuStG7Z1CdfC7hH+5xOG4nIRs=', 'Content-Length': '0'} 2024-04-16T19:56:24.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e80fdd21356232-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-16T19:56:24.935 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.935 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.935 DEBUG:boto:Method: GET 2024-04-16T19:56:24.935 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.935 DEBUG:boto:Data: 2024-04-16T19:56:24.935 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.936 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.936 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.936 DEBUG:boto:Params: {} 2024-04-16T19:56:24.936 DEBUG:boto:Token: None 2024-04-16T19:56:24.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K4NgUYKb+utRVXBadnQUwH4G2Go= 2024-04-16T19:56:24.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K4NgUYKb+utRVXBadnQUwH4G2Go=', 'Content-Length': '0'} 2024-04-16T19:56:24.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bebb626d0594c9f-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.978 DEBUG:boto:path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.979 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.979 DEBUG:boto:Method: GET 2024-04-16T19:56:24.979 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.979 DEBUG:boto:Data: 2024-04-16T19:56:24.979 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.979 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.979 DEBUG:boto:Params: {} 2024-04-16T19:56:24.979 DEBUG:boto:Token: None 2024-04-16T19:56:24.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000011 2024-04-16T19:56:24.980 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K4NgUYKb+utRVXBadnQUwH4G2Go= 2024-04-16T19:56:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K4NgUYKb+utRVXBadnQUwH4G2Go=', 'Content-Length': '0'} 2024-04-16T19:56:24.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4d128494c436c2-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.982 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.982 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.982 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.983 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.983 DEBUG:boto:Method: GET 2024-04-16T19:56:24.983 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.983 DEBUG:boto:Data: 2024-04-16T19:56:24.983 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.983 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.983 DEBUG:boto:Params: {} 2024-04-16T19:56:24.983 DEBUG:boto:Token: None 2024-04-16T19:56:24.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a56246231b01be-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.987 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.987 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:24.987 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:24.987 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.987 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.988 DEBUG:boto:Method: GET 2024-04-16T19:56:24.988 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000011 2024-04-16T19:56:24.988 DEBUG:boto:Data: 2024-04-16T19:56:24.988 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.988 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.988 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.988 DEBUG:boto:Params: {} 2024-04-16T19:56:24.988 DEBUG:boto:Token: None 2024-04-16T19:56:24.988 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/ 2024-04-16T19:56:24.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw= 2024-04-16T19:56:24.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5ypovBryf0r4ftG6QaTzE4ObNw=', 'Content-Length': '0'} 2024-04-16T19:56:24.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a30622ed44696e-00661ed7e8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.992 DEBUG:boto:b'yrjefh-51obj000000000000000000111000falseobj000000000000000000112024-04-16T19:56:08.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:24.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:56:24.993 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:24.993 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:24.993 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.993 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.993 DEBUG:boto:Method: GET 2024-04-16T19:56:24.993 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.994 DEBUG:boto:Data: 2024-04-16T19:56:24.994 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.994 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:24.994 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.994 DEBUG:boto:Params: {} 2024-04-16T19:56:24.994 DEBUG:boto:Token: None 2024-04-16T19:56:24.994 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.994 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MmHYwaYAHP1B+e0yeewrXBuJLSA= 2024-04-16T19:56:24.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MmHYwaYAHP1B+e0yeewrXBuJLSA=', 'Content-Length': '0'} 2024-04-16T19:56:24.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d197e1efce8886da-00661ed7e8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:24.997 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:24.997 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:24.997 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.997 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.997 DEBUG:boto:Method: GET 2024-04-16T19:56:24.997 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.997 DEBUG:boto:Data: 2024-04-16T19:56:24.997 DEBUG:boto:Headers: {} 2024-04-16T19:56:24.998 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:24.998 DEBUG:boto:Port: 8000 2024-04-16T19:56:24.998 DEBUG:boto:Params: {} 2024-04-16T19:56:24.998 DEBUG:boto:Token: None 2024-04-16T19:56:24.998 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:24 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:24.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MmHYwaYAHP1B+e0yeewrXBuJLSA= 2024-04-16T19:56:24.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MmHYwaYAHP1B+e0yeewrXBuJLSA=', 'Content-Length': '0'} 2024-04-16T19:56:25.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042098afb78e22988-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.042 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.043 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.043 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.043 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.043 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.043 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.043 DEBUG:boto:Data: 2024-04-16T19:56:25.043 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.043 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.043 DEBUG:boto:Params: {} 2024-04-16T19:56:25.044 DEBUG:boto:Token: None 2024-04-16T19:56:25.044 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:trUeH00XXjXaEmhzBPaLFCLzplA= 2024-04-16T19:56:25.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:trUeH00XXjXaEmhzBPaLFCLzplA=', 'Content-Length': '0'} 2024-04-16T19:56:25.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9da746302c847e-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.046 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.047 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.047 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.047 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.047 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.047 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.047 DEBUG:boto:Data: 2024-04-16T19:56:25.047 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.047 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.047 DEBUG:boto:Params: {} 2024-04-16T19:56:25.048 DEBUG:boto:Token: None 2024-04-16T19:56:25.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:trUeH00XXjXaEmhzBPaLFCLzplA= 2024-04-16T19:56:25.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:trUeH00XXjXaEmhzBPaLFCLzplA=', 'Content-Length': '0'} 2024-04-16T19:56:25.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a821d49fe9a14b3-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:56:25.050 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.050 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.051 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.051 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.051 DEBUG:boto:Method: GET 2024-04-16T19:56:25.051 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.051 DEBUG:boto:Data: 2024-04-16T19:56:25.051 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.051 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.051 DEBUG:boto:Params: {} 2024-04-16T19:56:25.051 DEBUG:boto:Token: None 2024-04-16T19:56:25.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wrkgG+ZZ45GbSxBvB/Gmm54m6lE= 2024-04-16T19:56:25.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wrkgG+ZZ45GbSxBvB/Gmm54m6lE=', 'Content-Length': '0'} 2024-04-16T19:56:25.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005feb1db48a2b9e4e-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.054 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.054 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.055 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.055 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.055 DEBUG:boto:Method: GET 2024-04-16T19:56:25.055 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.055 DEBUG:boto:Data: 2024-04-16T19:56:25.055 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.055 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.055 DEBUG:boto:Params: {} 2024-04-16T19:56:25.055 DEBUG:boto:Token: None 2024-04-16T19:56:25.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012?versionId=null 2024-04-16T19:56:25.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wrkgG+ZZ45GbSxBvB/Gmm54m6lE= 2024-04-16T19:56:25.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wrkgG+ZZ45GbSxBvB/Gmm54m6lE=', 'Content-Length': '0'} 2024-04-16T19:56:25.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fea789c5580adc7-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.058 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.058 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.059 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.059 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.059 DEBUG:boto:Method: GET 2024-04-16T19:56:25.059 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.059 DEBUG:boto:Data: 2024-04-16T19:56:25.059 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.059 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.059 DEBUG:boto:Params: {} 2024-04-16T19:56:25.059 DEBUG:boto:Token: None 2024-04-16T19:56:25.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71f9a6e1b2bb6d2-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.063 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.063 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.063 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.063 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.063 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.063 DEBUG:boto:Method: GET 2024-04-16T19:56:25.063 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.064 DEBUG:boto:Data: 2024-04-16T19:56:25.064 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.064 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.064 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.064 DEBUG:boto:Params: {} 2024-04-16T19:56:25.064 DEBUG:boto:Token: None 2024-04-16T19:56:25.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff46f7489b11ae16-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.069 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.069 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.069 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.069 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.069 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.069 DEBUG:boto:Data: 2024-04-16T19:56:25.070 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.070 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.070 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.070 DEBUG:boto:Params: {} 2024-04-16T19:56:25.070 DEBUG:boto:Token: None 2024-04-16T19:56:25.070 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1kjqbcSUEWupFbsBbW+KVbmSRFY= 2024-04-16T19:56:25.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1kjqbcSUEWupFbsBbW+KVbmSRFY=', 'Content-Length': '0'} 2024-04-16T19:56:25.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055504dab40f6942e-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.072 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.073 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.073 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.073 DEBUG:boto:Data: 2024-04-16T19:56:25.073 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.073 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.073 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.073 DEBUG:boto:Params: {} 2024-04-16T19:56:25.073 DEBUG:boto:Token: None 2024-04-16T19:56:25.074 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1kjqbcSUEWupFbsBbW+KVbmSRFY= 2024-04-16T19:56:25.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1kjqbcSUEWupFbsBbW+KVbmSRFY=', 'Content-Length': '0'} 2024-04-16T19:56:25.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4687d997e7c90b3-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-16T19:56:25.076 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.076 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.076 DEBUG:boto:Method: GET 2024-04-16T19:56:25.076 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.076 DEBUG:boto:Data: 2024-04-16T19:56:25.076 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.077 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.077 DEBUG:boto:Params: {} 2024-04-16T19:56:25.077 DEBUG:boto:Token: None 2024-04-16T19:56:25.077 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.077 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3qkcZH3yr1KtgHjEdQCYZvaAf7M= 2024-04-16T19:56:25.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3qkcZH3yr1KtgHjEdQCYZvaAf7M=', 'Content-Length': '0'} 2024-04-16T19:56:25.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c591293a6422e0b-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.122 DEBUG:boto:path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.123 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.123 DEBUG:boto:Method: GET 2024-04-16T19:56:25.123 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.123 DEBUG:boto:Data: 2024-04-16T19:56:25.123 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.123 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.123 DEBUG:boto:Params: {} 2024-04-16T19:56:25.123 DEBUG:boto:Token: None 2024-04-16T19:56:25.123 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000012 2024-04-16T19:56:25.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3qkcZH3yr1KtgHjEdQCYZvaAf7M= 2024-04-16T19:56:25.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3qkcZH3yr1KtgHjEdQCYZvaAf7M=', 'Content-Length': '0'} 2024-04-16T19:56:25.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020516d4d8278e559-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.126 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.126 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.126 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.126 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.126 DEBUG:boto:Method: GET 2024-04-16T19:56:25.127 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.127 DEBUG:boto:Data: 2024-04-16T19:56:25.127 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.127 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.127 DEBUG:boto:Params: {} 2024-04-16T19:56:25.127 DEBUG:boto:Token: None 2024-04-16T19:56:25.127 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019ddfe9413737a8-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.130 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.131 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.131 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.131 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.131 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.131 DEBUG:boto:Method: GET 2024-04-16T19:56:25.131 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000012 2024-04-16T19:56:25.131 DEBUG:boto:Data: 2024-04-16T19:56:25.131 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.132 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.132 DEBUG:boto:Params: {} 2024-04-16T19:56:25.132 DEBUG:boto:Token: None 2024-04-16T19:56:25.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6f42272703b4bc-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.137 DEBUG:boto:b'yrjefh-51obj000000000000000000121000falseobj000000000000000000122024-04-16T19:56:09.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:56:25.137 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.137 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.137 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.137 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.137 DEBUG:boto:Method: GET 2024-04-16T19:56:25.138 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.138 DEBUG:boto:Data: 2024-04-16T19:56:25.138 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.138 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.138 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.138 DEBUG:boto:Params: {} 2024-04-16T19:56:25.138 DEBUG:boto:Token: None 2024-04-16T19:56:25.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.138 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs= 2024-04-16T19:56:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs=', 'Content-Length': '0'} 2024-04-16T19:56:25.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014deb6c97fea49cd-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.141 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.141 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.141 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.141 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.141 DEBUG:boto:Method: GET 2024-04-16T19:56:25.141 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.141 DEBUG:boto:Data: 2024-04-16T19:56:25.141 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.142 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.142 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.142 DEBUG:boto:Params: {} 2024-04-16T19:56:25.142 DEBUG:boto:Token: None 2024-04-16T19:56:25.142 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs= 2024-04-16T19:56:25.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs=', 'Content-Length': '0'} 2024-04-16T19:56:25.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b722dd3047468a8f-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.186 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.187 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.187 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.187 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.187 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.187 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.187 DEBUG:boto:Data: 2024-04-16T19:56:25.187 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.187 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.187 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.187 DEBUG:boto:Params: {} 2024-04-16T19:56:25.188 DEBUG:boto:Token: None 2024-04-16T19:56:25.188 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BMpXghJv73Ke342y+UcYe5QxIcw= 2024-04-16T19:56:25.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BMpXghJv73Ke342y+UcYe5QxIcw=', 'Content-Length': '0'} 2024-04-16T19:56:25.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8e49326e9768b5-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.191 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.191 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.191 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.191 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.191 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.191 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.191 DEBUG:boto:Data: 2024-04-16T19:56:25.192 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.192 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.192 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.192 DEBUG:boto:Params: {} 2024-04-16T19:56:25.192 DEBUG:boto:Token: None 2024-04-16T19:56:25.192 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BMpXghJv73Ke342y+UcYe5QxIcw= 2024-04-16T19:56:25.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BMpXghJv73Ke342y+UcYe5QxIcw=', 'Content-Length': '0'} 2024-04-16T19:56:25.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f7ca299e13c3076-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:56:25.195 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.195 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.195 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.195 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.195 DEBUG:boto:Method: GET 2024-04-16T19:56:25.195 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.196 DEBUG:boto:Data: 2024-04-16T19:56:25.196 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.196 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.196 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.196 DEBUG:boto:Params: {} 2024-04-16T19:56:25.196 DEBUG:boto:Token: None 2024-04-16T19:56:25.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs= 2024-04-16T19:56:25.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs=', 'Content-Length': '0'} 2024-04-16T19:56:25.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ab084b30b1d3f6-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.199 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.199 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.199 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.199 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.200 DEBUG:boto:Method: GET 2024-04-16T19:56:25.200 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.200 DEBUG:boto:Data: 2024-04-16T19:56:25.200 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.200 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.200 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.200 DEBUG:boto:Params: {} 2024-04-16T19:56:25.200 DEBUG:boto:Token: None 2024-04-16T19:56:25.200 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013?versionId=null 2024-04-16T19:56:25.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs= 2024-04-16T19:56:25.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J8lXjKE/w3gr2h5eTERr6dUL9Fs=', 'Content-Length': '0'} 2024-04-16T19:56:25.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511be170366cd42a-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.203 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.203 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.203 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.203 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.203 DEBUG:boto:Method: GET 2024-04-16T19:56:25.203 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.203 DEBUG:boto:Data: 2024-04-16T19:56:25.203 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.203 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.204 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.204 DEBUG:boto:Params: {} 2024-04-16T19:56:25.204 DEBUG:boto:Token: None 2024-04-16T19:56:25.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c0e235bd82fb19d-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.250 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.251 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.251 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.251 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.251 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.251 DEBUG:boto:Method: GET 2024-04-16T19:56:25.251 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.251 DEBUG:boto:Data: 2024-04-16T19:56:25.251 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.251 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.252 DEBUG:boto:Params: {} 2024-04-16T19:56:25.252 DEBUG:boto:Token: None 2024-04-16T19:56:25.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000541d1c8b360847f-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.256 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.256 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.256 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.256 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.257 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.257 DEBUG:boto:Data: 2024-04-16T19:56:25.257 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.257 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.257 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.257 DEBUG:boto:Params: {} 2024-04-16T19:56:25.257 DEBUG:boto:Token: None 2024-04-16T19:56:25.257 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9W29OMNEC8A/0D7lzLlmkyGR7sI= 2024-04-16T19:56:25.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9W29OMNEC8A/0D7lzLlmkyGR7sI=', 'Content-Length': '0'} 2024-04-16T19:56:25.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1cc7ee30a45818-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.260 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.260 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.260 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.260 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.260 DEBUG:boto:Data: 2024-04-16T19:56:25.260 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.260 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.260 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.260 DEBUG:boto:Params: {} 2024-04-16T19:56:25.260 DEBUG:boto:Token: None 2024-04-16T19:56:25.261 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9W29OMNEC8A/0D7lzLlmkyGR7sI= 2024-04-16T19:56:25.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9W29OMNEC8A/0D7lzLlmkyGR7sI=', 'Content-Length': '0'} 2024-04-16T19:56:25.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c0d335add4c531-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-16T19:56:25.263 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.263 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.263 DEBUG:boto:Method: GET 2024-04-16T19:56:25.263 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.263 DEBUG:boto:Data: 2024-04-16T19:56:25.263 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.263 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.264 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.264 DEBUG:boto:Params: {} 2024-04-16T19:56:25.264 DEBUG:boto:Token: None 2024-04-16T19:56:25.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g0dOccm5wbdoZmeHs3bauPW76h4= 2024-04-16T19:56:25.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g0dOccm5wbdoZmeHs3bauPW76h4=', 'Content-Length': '0'} 2024-04-16T19:56:25.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e5017461f3718c-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.266 DEBUG:boto:path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.266 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.266 DEBUG:boto:Method: GET 2024-04-16T19:56:25.267 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.267 DEBUG:boto:Data: 2024-04-16T19:56:25.267 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.267 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.267 DEBUG:boto:Params: {} 2024-04-16T19:56:25.267 DEBUG:boto:Token: None 2024-04-16T19:56:25.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000013 2024-04-16T19:56:25.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g0dOccm5wbdoZmeHs3bauPW76h4= 2024-04-16T19:56:25.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g0dOccm5wbdoZmeHs3bauPW76h4=', 'Content-Length': '0'} 2024-04-16T19:56:25.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2c6c72b457fbe8-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.310 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.311 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.311 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.311 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.311 DEBUG:boto:Method: GET 2024-04-16T19:56:25.311 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.311 DEBUG:boto:Data: 2024-04-16T19:56:25.311 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.311 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.311 DEBUG:boto:Params: {} 2024-04-16T19:56:25.311 DEBUG:boto:Token: None 2024-04-16T19:56:25.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79ea1bf40a28964-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.315 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.315 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.315 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.315 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.315 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.315 DEBUG:boto:Method: GET 2024-04-16T19:56:25.315 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000013 2024-04-16T19:56:25.315 DEBUG:boto:Data: 2024-04-16T19:56:25.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.316 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.316 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.316 DEBUG:boto:Params: {} 2024-04-16T19:56:25.316 DEBUG:boto:Token: None 2024-04-16T19:56:25.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001131fa518b8d772-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.320 DEBUG:boto:b'yrjefh-51obj000000000000000000131000falseobj000000000000000000132024-04-16T19:56:05.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-16T19:56:25.321 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.321 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.321 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.321 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.321 DEBUG:boto:Method: GET 2024-04-16T19:56:25.321 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.321 DEBUG:boto:Data: 2024-04-16T19:56:25.322 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.322 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.322 DEBUG:boto:Params: {} 2024-04-16T19:56:25.322 DEBUG:boto:Token: None 2024-04-16T19:56:25.322 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.322 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw= 2024-04-16T19:56:25.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw=', 'Content-Length': '0'} 2024-04-16T19:56:25.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b1f06d797799471-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.366 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.366 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.366 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.367 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.367 DEBUG:boto:Method: GET 2024-04-16T19:56:25.367 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.367 DEBUG:boto:Data: 2024-04-16T19:56:25.367 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.367 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.367 DEBUG:boto:Params: {} 2024-04-16T19:56:25.367 DEBUG:boto:Token: None 2024-04-16T19:56:25.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw= 2024-04-16T19:56:25.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw=', 'Content-Length': '0'} 2024-04-16T19:56:25.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000473dd8c2bdcd7f64-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.370 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.370 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.371 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.371 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.371 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.371 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.371 DEBUG:boto:Data: 2024-04-16T19:56:25.371 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.371 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.371 DEBUG:boto:Params: {} 2024-04-16T19:56:25.372 DEBUG:boto:Token: None 2024-04-16T19:56:25.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NQcJJ7i7yab6KoqArF2E10py6nY= 2024-04-16T19:56:25.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NQcJJ7i7yab6KoqArF2E10py6nY=', 'Content-Length': '0'} 2024-04-16T19:56:25.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073cf438c097be980-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.374 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.374 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.374 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.374 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.375 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.375 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.375 DEBUG:boto:Data: 2024-04-16T19:56:25.375 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.375 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.375 DEBUG:boto:Params: {} 2024-04-16T19:56:25.375 DEBUG:boto:Token: None 2024-04-16T19:56:25.375 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NQcJJ7i7yab6KoqArF2E10py6nY= 2024-04-16T19:56:25.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NQcJJ7i7yab6KoqArF2E10py6nY=', 'Content-Length': '0'} 2024-04-16T19:56:25.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b9a02898a0d571-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-16T19:56:25.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.378 DEBUG:boto:Method: GET 2024-04-16T19:56:25.379 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.379 DEBUG:boto:Data: 2024-04-16T19:56:25.379 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.379 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.379 DEBUG:boto:Params: {} 2024-04-16T19:56:25.379 DEBUG:boto:Token: None 2024-04-16T19:56:25.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw= 2024-04-16T19:56:25.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw=', 'Content-Length': '0'} 2024-04-16T19:56:25.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f079d85cdcf3eaaf-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.382 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.382 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.382 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.382 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.383 DEBUG:boto:Method: GET 2024-04-16T19:56:25.383 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.383 DEBUG:boto:Data: 2024-04-16T19:56:25.383 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.383 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.383 DEBUG:boto:Params: {} 2024-04-16T19:56:25.383 DEBUG:boto:Token: None 2024-04-16T19:56:25.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014?versionId=null 2024-04-16T19:56:25.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw= 2024-04-16T19:56:25.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6h8DoFM3io7eXbBlKTUSfFIYBRw=', 'Content-Length': '0'} 2024-04-16T19:56:25.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff4e091cbb4651a-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.386 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.386 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.386 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.386 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.387 DEBUG:boto:Method: GET 2024-04-16T19:56:25.387 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.387 DEBUG:boto:Data: 2024-04-16T19:56:25.387 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.387 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.387 DEBUG:boto:Params: {} 2024-04-16T19:56:25.387 DEBUG:boto:Token: None 2024-04-16T19:56:25.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf517d8401082a9-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.390 DEBUG:boto:b'yrjefh-51obj000000000000000000141000falseobj000000000000000000142024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.390 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.391 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.391 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.391 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.391 DEBUG:boto:Method: GET 2024-04-16T19:56:25.391 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.391 DEBUG:boto:Data: 2024-04-16T19:56:25.391 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.391 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.391 DEBUG:boto:Params: {} 2024-04-16T19:56:25.392 DEBUG:boto:Token: None 2024-04-16T19:56:25.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85b4e7a63911f62-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.396 DEBUG:boto:b'yrjefh-51obj000000000000000000141000falseobj000000000000000000142024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.396 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.396 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.397 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.397 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.397 DEBUG:boto:Data: 2024-04-16T19:56:25.397 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.397 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.397 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.397 DEBUG:boto:Params: {} 2024-04-16T19:56:25.397 DEBUG:boto:Token: None 2024-04-16T19:56:25.397 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.398 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DfmD7JkADieOjMy2NgttZyAQqn0= 2024-04-16T19:56:25.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DfmD7JkADieOjMy2NgttZyAQqn0=', 'Content-Length': '0'} 2024-04-16T19:56:25.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35b05458c056f76-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.400 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.400 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.400 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.401 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.401 DEBUG:boto:Data: 2024-04-16T19:56:25.401 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.401 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.401 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.401 DEBUG:boto:Params: {} 2024-04-16T19:56:25.401 DEBUG:boto:Token: None 2024-04-16T19:56:25.401 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DfmD7JkADieOjMy2NgttZyAQqn0= 2024-04-16T19:56:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DfmD7JkADieOjMy2NgttZyAQqn0=', 'Content-Length': '0'} 2024-04-16T19:56:25.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fef4fec1135cec-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-16T19:56:25.404 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.404 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.404 DEBUG:boto:Method: GET 2024-04-16T19:56:25.404 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.405 DEBUG:boto:Data: 2024-04-16T19:56:25.405 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.405 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.405 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.405 DEBUG:boto:Params: {} 2024-04-16T19:56:25.405 DEBUG:boto:Token: None 2024-04-16T19:56:25.405 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f8vqoauoBcG71tAQNONVY+a913k= 2024-04-16T19:56:25.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f8vqoauoBcG71tAQNONVY+a913k=', 'Content-Length': '0'} 2024-04-16T19:56:25.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b7104ed4463d7b-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.408 DEBUG:boto:path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.408 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.408 DEBUG:boto:Method: GET 2024-04-16T19:56:25.408 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.408 DEBUG:boto:Data: 2024-04-16T19:56:25.408 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.409 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.409 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.409 DEBUG:boto:Params: {} 2024-04-16T19:56:25.409 DEBUG:boto:Token: None 2024-04-16T19:56:25.409 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000014 2024-04-16T19:56:25.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f8vqoauoBcG71tAQNONVY+a913k= 2024-04-16T19:56:25.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f8vqoauoBcG71tAQNONVY+a913k=', 'Content-Length': '0'} 2024-04-16T19:56:25.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e05954fe9077658-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.412 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.412 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.412 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.412 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.412 DEBUG:boto:Method: GET 2024-04-16T19:56:25.412 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.412 DEBUG:boto:Data: 2024-04-16T19:56:25.412 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.412 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.413 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.413 DEBUG:boto:Params: {} 2024-04-16T19:56:25.413 DEBUG:boto:Token: None 2024-04-16T19:56:25.413 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000993222f0f061d558-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.416 DEBUG:boto:b'yrjefh-51obj000000000000000000141000falseobj000000000000000000142024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.416 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.416 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.417 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.417 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.417 DEBUG:boto:Method: GET 2024-04-16T19:56:25.417 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000014 2024-04-16T19:56:25.417 DEBUG:boto:Data: 2024-04-16T19:56:25.417 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.417 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.417 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.417 DEBUG:boto:Params: {} 2024-04-16T19:56:25.417 DEBUG:boto:Token: None 2024-04-16T19:56:25.418 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcfc562c810d2186-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.462 DEBUG:boto:b'yrjefh-51obj000000000000000000141000falseobj000000000000000000142024-04-16T19:56:09.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-16T19:56:25.463 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.463 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.463 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.463 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.464 DEBUG:boto:Method: GET 2024-04-16T19:56:25.464 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.464 DEBUG:boto:Data: 2024-04-16T19:56:25.464 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.464 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.464 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.464 DEBUG:boto:Params: {} 2024-04-16T19:56:25.464 DEBUG:boto:Token: None 2024-04-16T19:56:25.465 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU= 2024-04-16T19:56:25.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU=', 'Content-Length': '0'} 2024-04-16T19:56:25.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4fb7f8542f7980b-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.467 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.468 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.468 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.468 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.468 DEBUG:boto:Method: GET 2024-04-16T19:56:25.468 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.468 DEBUG:boto:Data: 2024-04-16T19:56:25.468 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.468 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.468 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.469 DEBUG:boto:Params: {} 2024-04-16T19:56:25.469 DEBUG:boto:Token: None 2024-04-16T19:56:25.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU= 2024-04-16T19:56:25.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU=', 'Content-Length': '0'} 2024-04-16T19:56:25.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb653feefaff9c16-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.471 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.472 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.472 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.472 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.472 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.472 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.472 DEBUG:boto:Data: 2024-04-16T19:56:25.472 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.472 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.473 DEBUG:boto:Params: {} 2024-04-16T19:56:25.473 DEBUG:boto:Token: None 2024-04-16T19:56:25.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/AbCnU4UQjLuEnmdGh3flmRfjg= 2024-04-16T19:56:25.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/AbCnU4UQjLuEnmdGh3flmRfjg=', 'Content-Length': '0'} 2024-04-16T19:56:25.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034938e95b49eb296-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.475 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.475 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.475 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.476 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.476 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.476 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.476 DEBUG:boto:Data: 2024-04-16T19:56:25.476 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.476 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.476 DEBUG:boto:Params: {} 2024-04-16T19:56:25.476 DEBUG:boto:Token: None 2024-04-16T19:56:25.477 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/AbCnU4UQjLuEnmdGh3flmRfjg= 2024-04-16T19:56:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/AbCnU4UQjLuEnmdGh3flmRfjg=', 'Content-Length': '0'} 2024-04-16T19:56:25.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a708213371e25a-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-16T19:56:25.479 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.480 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.480 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.480 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.480 DEBUG:boto:Method: GET 2024-04-16T19:56:25.480 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.480 DEBUG:boto:Data: 2024-04-16T19:56:25.480 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.480 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.480 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.480 DEBUG:boto:Params: {} 2024-04-16T19:56:25.481 DEBUG:boto:Token: None 2024-04-16T19:56:25.481 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU= 2024-04-16T19:56:25.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU=', 'Content-Length': '0'} 2024-04-16T19:56:25.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f97e81b3db0410a-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.526 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.527 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.527 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.527 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.527 DEBUG:boto:Method: GET 2024-04-16T19:56:25.527 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.527 DEBUG:boto:Data: 2024-04-16T19:56:25.527 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.527 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.528 DEBUG:boto:Params: {} 2024-04-16T19:56:25.528 DEBUG:boto:Token: None 2024-04-16T19:56:25.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015?versionId=null 2024-04-16T19:56:25.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU= 2024-04-16T19:56:25.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:STofk6OO+OxjkDGVNvQ1/ufJ1AU=', 'Content-Length': '0'} 2024-04-16T19:56:25.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a741133325a55e7d-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.530 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.531 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.531 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.531 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.531 DEBUG:boto:Method: GET 2024-04-16T19:56:25.531 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.531 DEBUG:boto:Data: 2024-04-16T19:56:25.531 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.532 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.532 DEBUG:boto:Params: {} 2024-04-16T19:56:25.532 DEBUG:boto:Token: None 2024-04-16T19:56:25.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc671fcebb19f3df-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.535 DEBUG:boto:b'yrjefh-51obj000000000000000000151000falseobj000000000000000000152024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.535 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.535 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.536 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.536 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.536 DEBUG:boto:Method: GET 2024-04-16T19:56:25.536 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.536 DEBUG:boto:Data: 2024-04-16T19:56:25.536 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.536 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.536 DEBUG:boto:Params: {} 2024-04-16T19:56:25.537 DEBUG:boto:Token: None 2024-04-16T19:56:25.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d9ab094b7410dd-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.541 DEBUG:boto:b'yrjefh-51obj000000000000000000151000falseobj000000000000000000152024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.541 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.542 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.542 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.542 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.542 DEBUG:boto:Data: 2024-04-16T19:56:25.542 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.542 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.542 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.542 DEBUG:boto:Params: {} 2024-04-16T19:56:25.543 DEBUG:boto:Token: None 2024-04-16T19:56:25.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zwr6+R9Tcs+mKBDvirq8D+jbL1s= 2024-04-16T19:56:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zwr6+R9Tcs+mKBDvirq8D+jbL1s=', 'Content-Length': '0'} 2024-04-16T19:56:25.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d4c194fa928d6e-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.545 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.545 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.545 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.545 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.546 DEBUG:boto:Data: 2024-04-16T19:56:25.546 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.546 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.546 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.546 DEBUG:boto:Params: {} 2024-04-16T19:56:25.546 DEBUG:boto:Token: None 2024-04-16T19:56:25.546 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zwr6+R9Tcs+mKBDvirq8D+jbL1s= 2024-04-16T19:56:25.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zwr6+R9Tcs+mKBDvirq8D+jbL1s=', 'Content-Length': '0'} 2024-04-16T19:56:25.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065dee02fff270e9c-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-16T19:56:25.549 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.549 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.549 DEBUG:boto:Method: GET 2024-04-16T19:56:25.549 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.549 DEBUG:boto:Data: 2024-04-16T19:56:25.549 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.550 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.550 DEBUG:boto:Params: {} 2024-04-16T19:56:25.550 DEBUG:boto:Token: None 2024-04-16T19:56:25.550 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UBxtmzLVSKpxW6kRHfMRvHWW8s4= 2024-04-16T19:56:25.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UBxtmzLVSKpxW6kRHfMRvHWW8s4=', 'Content-Length': '0'} 2024-04-16T19:56:25.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870c98537b9918b4-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.553 DEBUG:boto:path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.553 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.553 DEBUG:boto:Method: GET 2024-04-16T19:56:25.553 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.553 DEBUG:boto:Data: 2024-04-16T19:56:25.553 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.553 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.553 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.553 DEBUG:boto:Params: {} 2024-04-16T19:56:25.554 DEBUG:boto:Token: None 2024-04-16T19:56:25.554 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000015 2024-04-16T19:56:25.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UBxtmzLVSKpxW6kRHfMRvHWW8s4= 2024-04-16T19:56:25.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UBxtmzLVSKpxW6kRHfMRvHWW8s4=', 'Content-Length': '0'} 2024-04-16T19:56:25.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa414f8ea2b4104-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.598 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.598 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.599 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.599 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.599 DEBUG:boto:Method: GET 2024-04-16T19:56:25.599 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.599 DEBUG:boto:Data: 2024-04-16T19:56:25.599 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.599 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.599 DEBUG:boto:Params: {} 2024-04-16T19:56:25.600 DEBUG:boto:Token: None 2024-04-16T19:56:25.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35737105865cf5b-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.603 DEBUG:boto:b'yrjefh-51obj000000000000000000151000falseobj000000000000000000152024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.603 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.603 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.603 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.604 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.604 DEBUG:boto:Method: GET 2024-04-16T19:56:25.604 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000015 2024-04-16T19:56:25.604 DEBUG:boto:Data: 2024-04-16T19:56:25.604 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.604 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.604 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.604 DEBUG:boto:Params: {} 2024-04-16T19:56:25.604 DEBUG:boto:Token: None 2024-04-16T19:56:25.605 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a509eda1ccd0bb-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.609 DEBUG:boto:b'yrjefh-51obj000000000000000000151000falseobj000000000000000000152024-04-16T19:56:09.138Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-16T19:56:25.609 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.610 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.610 DEBUG:boto:Method: GET 2024-04-16T19:56:25.610 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.610 DEBUG:boto:Data: 2024-04-16T19:56:25.610 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.610 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.610 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.611 DEBUG:boto:Params: {} 2024-04-16T19:56:25.611 DEBUG:boto:Token: None 2024-04-16T19:56:25.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw= 2024-04-16T19:56:25.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw=', 'Content-Length': '0'} 2024-04-16T19:56:25.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c698039d31e7636-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.654 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.655 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.655 DEBUG:boto:Method: GET 2024-04-16T19:56:25.655 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.655 DEBUG:boto:Data: 2024-04-16T19:56:25.655 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.655 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.656 DEBUG:boto:Params: {} 2024-04-16T19:56:25.656 DEBUG:boto:Token: None 2024-04-16T19:56:25.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw= 2024-04-16T19:56:25.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw=', 'Content-Length': '0'} 2024-04-16T19:56:25.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573557fd0173462e-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.659 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.659 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.659 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.659 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.659 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.659 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.659 DEBUG:boto:Data: 2024-04-16T19:56:25.659 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.660 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.660 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.660 DEBUG:boto:Params: {} 2024-04-16T19:56:25.660 DEBUG:boto:Token: None 2024-04-16T19:56:25.660 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bEvj9D1Cuqucj5K4U2qbRwT/E8s= 2024-04-16T19:56:25.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bEvj9D1Cuqucj5K4U2qbRwT/E8s=', 'Content-Length': '0'} 2024-04-16T19:56:25.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072d5895e7706ced9-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.663 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.663 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.663 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.664 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.664 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.664 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.664 DEBUG:boto:Data: 2024-04-16T19:56:25.664 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.664 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.664 DEBUG:boto:Params: {} 2024-04-16T19:56:25.664 DEBUG:boto:Token: None 2024-04-16T19:56:25.665 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bEvj9D1Cuqucj5K4U2qbRwT/E8s= 2024-04-16T19:56:25.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bEvj9D1Cuqucj5K4U2qbRwT/E8s=', 'Content-Length': '0'} 2024-04-16T19:56:25.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bab8204e31cd489-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-16T19:56:25.668 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.668 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.668 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.668 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.668 DEBUG:boto:Method: GET 2024-04-16T19:56:25.668 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.668 DEBUG:boto:Data: 2024-04-16T19:56:25.669 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.669 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.669 DEBUG:boto:Params: {} 2024-04-16T19:56:25.669 DEBUG:boto:Token: None 2024-04-16T19:56:25.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw= 2024-04-16T19:56:25.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw=', 'Content-Length': '0'} 2024-04-16T19:56:25.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cc56c4237775fd3-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.672 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.672 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.672 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.672 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.673 DEBUG:boto:Method: GET 2024-04-16T19:56:25.673 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.673 DEBUG:boto:Data: 2024-04-16T19:56:25.673 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.673 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.673 DEBUG:boto:Params: {} 2024-04-16T19:56:25.673 DEBUG:boto:Token: None 2024-04-16T19:56:25.674 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016?versionId=null 2024-04-16T19:56:25.674 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw= 2024-04-16T19:56:25.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F50w6QX9mgUK+nyMUml1OZ3L8pw=', 'Content-Length': '0'} 2024-04-16T19:56:25.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab94205f6564f74a-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.718 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.719 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.719 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.719 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.719 DEBUG:boto:Method: GET 2024-04-16T19:56:25.719 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.719 DEBUG:boto:Data: 2024-04-16T19:56:25.719 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.719 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.719 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.720 DEBUG:boto:Params: {} 2024-04-16T19:56:25.720 DEBUG:boto:Token: None 2024-04-16T19:56:25.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7adfea7619ec0a3-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.723 DEBUG:boto:b'yrjefh-51obj000000000000000000161000falseobj000000000000000000162024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.723 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.724 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.724 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.724 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.724 DEBUG:boto:Method: GET 2024-04-16T19:56:25.724 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.724 DEBUG:boto:Data: 2024-04-16T19:56:25.724 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.724 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.725 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.725 DEBUG:boto:Params: {} 2024-04-16T19:56:25.725 DEBUG:boto:Token: None 2024-04-16T19:56:25.725 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13af77107adb05b-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.729 DEBUG:boto:b'yrjefh-51obj000000000000000000161000falseobj000000000000000000162024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.730 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.730 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.730 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.730 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.730 DEBUG:boto:Data: 2024-04-16T19:56:25.730 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.730 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.730 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.731 DEBUG:boto:Params: {} 2024-04-16T19:56:25.731 DEBUG:boto:Token: None 2024-04-16T19:56:25.731 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zGcBslCOBsQAs+UzdYW2hfS36g= 2024-04-16T19:56:25.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zGcBslCOBsQAs+UzdYW2hfS36g=', 'Content-Length': '0'} 2024-04-16T19:56:25.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542602cac66c0c97-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.734 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.734 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.734 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.734 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.734 DEBUG:boto:Data: 2024-04-16T19:56:25.734 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.734 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.734 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.735 DEBUG:boto:Params: {} 2024-04-16T19:56:25.735 DEBUG:boto:Token: None 2024-04-16T19:56:25.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zGcBslCOBsQAs+UzdYW2hfS36g= 2024-04-16T19:56:25.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zGcBslCOBsQAs+UzdYW2hfS36g=', 'Content-Length': '0'} 2024-04-16T19:56:25.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fed91a1ec33b9e1-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-16T19:56:25.738 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.738 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.738 DEBUG:boto:Method: GET 2024-04-16T19:56:25.738 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.738 DEBUG:boto:Data: 2024-04-16T19:56:25.738 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.738 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.739 DEBUG:boto:Params: {} 2024-04-16T19:56:25.739 DEBUG:boto:Token: None 2024-04-16T19:56:25.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ZOIQ7L9QCSgHgha4+yuqTFBpDc= 2024-04-16T19:56:25.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ZOIQ7L9QCSgHgha4+yuqTFBpDc=', 'Content-Length': '0'} 2024-04-16T19:56:25.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac4b13203898436-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.782 DEBUG:boto:path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.783 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.783 DEBUG:boto:Method: GET 2024-04-16T19:56:25.783 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.783 DEBUG:boto:Data: 2024-04-16T19:56:25.783 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.783 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.783 DEBUG:boto:Params: {} 2024-04-16T19:56:25.783 DEBUG:boto:Token: None 2024-04-16T19:56:25.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000016 2024-04-16T19:56:25.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ZOIQ7L9QCSgHgha4+yuqTFBpDc= 2024-04-16T19:56:25.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ZOIQ7L9QCSgHgha4+yuqTFBpDc=', 'Content-Length': '0'} 2024-04-16T19:56:25.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff958772e58cd88a-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.786 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.787 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.787 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.787 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.787 DEBUG:boto:Method: GET 2024-04-16T19:56:25.787 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.787 DEBUG:boto:Data: 2024-04-16T19:56:25.787 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.787 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.788 DEBUG:boto:Params: {} 2024-04-16T19:56:25.788 DEBUG:boto:Token: None 2024-04-16T19:56:25.788 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000876f1ea882d16cab-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.791 DEBUG:boto:b'yrjefh-51obj000000000000000000161000falseobj000000000000000000162024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.791 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.791 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.791 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.791 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.792 DEBUG:boto:Method: GET 2024-04-16T19:56:25.792 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000016 2024-04-16T19:56:25.792 DEBUG:boto:Data: 2024-04-16T19:56:25.792 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.792 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.792 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.792 DEBUG:boto:Params: {} 2024-04-16T19:56:25.793 DEBUG:boto:Token: None 2024-04-16T19:56:25.793 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040aca2be092f8ddb-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.838 DEBUG:boto:b'yrjefh-51obj000000000000000000161000falseobj000000000000000000162024-04-16T19:56:08.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-16T19:56:25.839 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.839 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.839 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.839 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.839 DEBUG:boto:Method: GET 2024-04-16T19:56:25.839 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.839 DEBUG:boto:Data: 2024-04-16T19:56:25.839 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.840 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.840 DEBUG:boto:Params: {} 2024-04-16T19:56:25.840 DEBUG:boto:Token: None 2024-04-16T19:56:25.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ= 2024-04-16T19:56:25.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6b3a4109cf70d2-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.843 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.843 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.843 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.843 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.843 DEBUG:boto:Method: GET 2024-04-16T19:56:25.843 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.843 DEBUG:boto:Data: 2024-04-16T19:56:25.843 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.844 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.844 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.844 DEBUG:boto:Params: {} 2024-04-16T19:56:25.844 DEBUG:boto:Token: None 2024-04-16T19:56:25.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ= 2024-04-16T19:56:25.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e43ee7482a9f276-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.846 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.847 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.847 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.847 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.847 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.847 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.847 DEBUG:boto:Data: 2024-04-16T19:56:25.847 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.847 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.847 DEBUG:boto:Params: {} 2024-04-16T19:56:25.848 DEBUG:boto:Token: None 2024-04-16T19:56:25.848 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oRjP1eB20U+qnOyQV6j4fB0n8iQ= 2024-04-16T19:56:25.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oRjP1eB20U+qnOyQV6j4fB0n8iQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051efc32a602e846d-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.850 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.850 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.850 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.850 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.851 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.851 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.851 DEBUG:boto:Data: 2024-04-16T19:56:25.851 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.851 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.851 DEBUG:boto:Params: {} 2024-04-16T19:56:25.851 DEBUG:boto:Token: None 2024-04-16T19:56:25.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oRjP1eB20U+qnOyQV6j4fB0n8iQ= 2024-04-16T19:56:25.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oRjP1eB20U+qnOyQV6j4fB0n8iQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097170ebe79382516-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-16T19:56:25.854 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.854 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.854 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.854 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.854 DEBUG:boto:Method: GET 2024-04-16T19:56:25.854 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.854 DEBUG:boto:Data: 2024-04-16T19:56:25.854 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.854 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.855 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.855 DEBUG:boto:Params: {} 2024-04-16T19:56:25.855 DEBUG:boto:Token: None 2024-04-16T19:56:25.855 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ= 2024-04-16T19:56:25.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4e2f08a6ae00eb-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.857 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.857 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.857 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.858 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.858 DEBUG:boto:Method: GET 2024-04-16T19:56:25.858 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.858 DEBUG:boto:Data: 2024-04-16T19:56:25.858 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.858 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.858 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.858 DEBUG:boto:Params: {} 2024-04-16T19:56:25.858 DEBUG:boto:Token: None 2024-04-16T19:56:25.858 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017?versionId=null 2024-04-16T19:56:25.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ= 2024-04-16T19:56:25.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Tas2w7UhSvzdc9cPepbxmZUoWQ=', 'Content-Length': '0'} 2024-04-16T19:56:25.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326e988d2c08f6c8-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.861 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.861 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.861 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.861 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.861 DEBUG:boto:Method: GET 2024-04-16T19:56:25.862 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.862 DEBUG:boto:Data: 2024-04-16T19:56:25.862 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.862 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.862 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.862 DEBUG:boto:Params: {} 2024-04-16T19:56:25.862 DEBUG:boto:Token: None 2024-04-16T19:56:25.862 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e24c2529e10ab4c0-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.865 DEBUG:boto:b'yrjefh-51obj000000000000000000171000falseobj000000000000000000172024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.866 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.866 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.866 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.866 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.866 DEBUG:boto:Method: GET 2024-04-16T19:56:25.866 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.866 DEBUG:boto:Data: 2024-04-16T19:56:25.866 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.866 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.866 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.867 DEBUG:boto:Params: {} 2024-04-16T19:56:25.867 DEBUG:boto:Token: None 2024-04-16T19:56:25.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4b60f7de69b3e8-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.874 DEBUG:boto:b'yrjefh-51obj000000000000000000171000falseobj000000000000000000172024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.875 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.875 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.875 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.876 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.876 DEBUG:boto:Data: 2024-04-16T19:56:25.876 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.876 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.876 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.876 DEBUG:boto:Params: {} 2024-04-16T19:56:25.876 DEBUG:boto:Token: None 2024-04-16T19:56:25.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M3mCC11BH1FvVWccrpxyPSGODYU= 2024-04-16T19:56:25.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M3mCC11BH1FvVWccrpxyPSGODYU=', 'Content-Length': '0'} 2024-04-16T19:56:25.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075fe96f362d7fe29-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.879 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.879 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.879 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.879 DEBUG:boto:Data: 2024-04-16T19:56:25.879 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.879 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.880 DEBUG:boto:Params: {} 2024-04-16T19:56:25.880 DEBUG:boto:Token: None 2024-04-16T19:56:25.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M3mCC11BH1FvVWccrpxyPSGODYU= 2024-04-16T19:56:25.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M3mCC11BH1FvVWccrpxyPSGODYU=', 'Content-Length': '0'} 2024-04-16T19:56:25.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3b07c3f40ceb9b-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-16T19:56:25.882 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.882 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.883 DEBUG:boto:Method: GET 2024-04-16T19:56:25.883 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.883 DEBUG:boto:Data: 2024-04-16T19:56:25.883 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.883 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.883 DEBUG:boto:Params: {} 2024-04-16T19:56:25.883 DEBUG:boto:Token: None 2024-04-16T19:56:25.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H4KeICUI15LqS0+40LKAs+jPAxs= 2024-04-16T19:56:25.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H4KeICUI15LqS0+40LKAs+jPAxs=', 'Content-Length': '0'} 2024-04-16T19:56:25.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d656a9545144b6-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.926 DEBUG:boto:path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.926 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.927 DEBUG:boto:Method: GET 2024-04-16T19:56:25.927 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.927 DEBUG:boto:Data: 2024-04-16T19:56:25.927 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.927 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.927 DEBUG:boto:Params: {} 2024-04-16T19:56:25.927 DEBUG:boto:Token: None 2024-04-16T19:56:25.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000017 2024-04-16T19:56:25.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H4KeICUI15LqS0+40LKAs+jPAxs= 2024-04-16T19:56:25.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H4KeICUI15LqS0+40LKAs+jPAxs=', 'Content-Length': '0'} 2024-04-16T19:56:25.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001010e0be47e2c4b2-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.930 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.930 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.930 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.930 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.930 DEBUG:boto:Method: GET 2024-04-16T19:56:25.930 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.930 DEBUG:boto:Data: 2024-04-16T19:56:25.930 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.930 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.931 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.931 DEBUG:boto:Params: {} 2024-04-16T19:56:25.931 DEBUG:boto:Token: None 2024-04-16T19:56:25.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cefcfc854a771da-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.934 DEBUG:boto:b'yrjefh-51obj000000000000000000171000falseobj000000000000000000172024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.934 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:25.934 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:25.935 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.935 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.935 DEBUG:boto:Method: GET 2024-04-16T19:56:25.935 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000017 2024-04-16T19:56:25.935 DEBUG:boto:Data: 2024-04-16T19:56:25.935 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.935 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.935 DEBUG:boto:Params: {} 2024-04-16T19:56:25.935 DEBUG:boto:Token: None 2024-04-16T19:56:25.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/ 2024-04-16T19:56:25.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk= 2024-04-16T19:56:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BDlczQXh1vmFpEuO235xZkpRUk=', 'Content-Length': '0'} 2024-04-16T19:56:25.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3e551ebb974ac8-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.940 DEBUG:boto:b'yrjefh-51obj000000000000000000171000falseobj000000000000000000172024-04-16T19:56:09.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:25.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-16T19:56:25.940 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.940 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.940 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.941 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.941 DEBUG:boto:Method: GET 2024-04-16T19:56:25.941 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.941 DEBUG:boto:Data: 2024-04-16T19:56:25.941 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.941 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.941 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.941 DEBUG:boto:Params: {} 2024-04-16T19:56:25.941 DEBUG:boto:Token: None 2024-04-16T19:56:25.941 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.942 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U= 2024-04-16T19:56:25.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U=', 'Content-Length': '0'} 2024-04-16T19:56:25.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3922bfe94c7aea1-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.944 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.944 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.945 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.945 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.945 DEBUG:boto:Method: GET 2024-04-16T19:56:25.945 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.945 DEBUG:boto:Data: 2024-04-16T19:56:25.945 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.945 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.945 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.945 DEBUG:boto:Params: {} 2024-04-16T19:56:25.945 DEBUG:boto:Token: None 2024-04-16T19:56:25.945 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.946 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U= 2024-04-16T19:56:25.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U=', 'Content-Length': '0'} 2024-04-16T19:56:25.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c6c7f4b7caa375-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.948 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.948 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.948 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.948 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.948 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.949 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.949 DEBUG:boto:Data: 2024-04-16T19:56:25.949 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.949 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.949 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.949 DEBUG:boto:Params: {} 2024-04-16T19:56:25.949 DEBUG:boto:Token: None 2024-04-16T19:56:25.949 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.949 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UzqHtUSgX0qQ0rjCM4R05avNQYg= 2024-04-16T19:56:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UzqHtUSgX0qQ0rjCM4R05avNQYg=', 'Content-Length': '0'} 2024-04-16T19:56:25.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002198c0e4e0dd554e-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.952 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.952 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.952 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.952 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.952 DEBUG:boto:Method: HEAD 2024-04-16T19:56:25.953 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.953 DEBUG:boto:Data: 2024-04-16T19:56:25.953 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.953 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:25.953 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.953 DEBUG:boto:Params: {} 2024-04-16T19:56:25.953 DEBUG:boto:Token: None 2024-04-16T19:56:25.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UzqHtUSgX0qQ0rjCM4R05avNQYg= 2024-04-16T19:56:25.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UzqHtUSgX0qQ0rjCM4R05avNQYg=', 'Content-Length': '0'} 2024-04-16T19:56:25.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ea120d72e9dd58-00661ed7e9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:25.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-16T19:56:25.956 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.956 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:25.956 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.956 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.956 DEBUG:boto:Method: GET 2024-04-16T19:56:25.957 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.957 DEBUG:boto:Data: 2024-04-16T19:56:25.957 DEBUG:boto:Headers: {} 2024-04-16T19:56:25.957 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:25.957 DEBUG:boto:Port: 8000 2024-04-16T19:56:25.957 DEBUG:boto:Params: {} 2024-04-16T19:56:25.957 DEBUG:boto:Token: None 2024-04-16T19:56:25.957 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:25 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:25.957 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U= 2024-04-16T19:56:25.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3yM7mgenVDRiZetVBBW6lZFg4U=', 'Content-Length': '0'} 2024-04-16T19:56:25.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2dbb0689f02ad9-00661ed7e9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.002 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.002 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.003 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:26.003 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:26.003 DEBUG:boto:Method: GET 2024-04-16T19:56:26.003 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:26.003 DEBUG:boto:Data: 2024-04-16T19:56:26.003 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.003 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.003 DEBUG:boto:Params: {} 2024-04-16T19:56:26.003 DEBUG:boto:Token: None 2024-04-16T19:56:26.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000018?versionId=null 2024-04-16T19:56:26.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iZ/y/m5PadirSjrmgjZgD1jLCKU= 2024-04-16T19:56:26.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iZ/y/m5PadirSjrmgjZgD1jLCKU=', 'Content-Length': '0'} 2024-04-16T19:56:26.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53c27cdebc556c2-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.006 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.007 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.007 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.007 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.007 DEBUG:boto:Method: GET 2024-04-16T19:56:26.007 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.007 DEBUG:boto:Data: 2024-04-16T19:56:26.007 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.007 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.007 DEBUG:boto:Params: {} 2024-04-16T19:56:26.007 DEBUG:boto:Token: None 2024-04-16T19:56:26.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f507356b046eff66-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.010 DEBUG:boto:b'yrjefh-51obj000000000000000000181000falseobj000000000000000000182024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.011 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.011 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.011 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.011 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.011 DEBUG:boto:Method: GET 2024-04-16T19:56:26.011 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.011 DEBUG:boto:Data: 2024-04-16T19:56:26.011 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.012 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.012 DEBUG:boto:Params: {} 2024-04-16T19:56:26.012 DEBUG:boto:Token: None 2024-04-16T19:56:26.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a014c996e936faa-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.016 DEBUG:boto:b'yrjefh-51obj000000000000000000181000falseobj000000000000000000182024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.016 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.016 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.016 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.016 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.016 DEBUG:boto:Data: 2024-04-16T19:56:26.016 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.017 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.017 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.017 DEBUG:boto:Params: {} 2024-04-16T19:56:26.017 DEBUG:boto:Token: None 2024-04-16T19:56:26.017 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C6V2uESZfoxIJh2vshcmtiFByr8= 2024-04-16T19:56:26.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C6V2uESZfoxIJh2vshcmtiFByr8=', 'Content-Length': '0'} 2024-04-16T19:56:26.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964ab225aac356b0-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.019 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.019 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.020 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.020 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.020 DEBUG:boto:Data: 2024-04-16T19:56:26.020 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.020 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.020 DEBUG:boto:Params: {} 2024-04-16T19:56:26.020 DEBUG:boto:Token: None 2024-04-16T19:56:26.020 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C6V2uESZfoxIJh2vshcmtiFByr8= 2024-04-16T19:56:26.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C6V2uESZfoxIJh2vshcmtiFByr8=', 'Content-Length': '0'} 2024-04-16T19:56:26.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde58448170cca8e-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-16T19:56:26.023 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.023 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.023 DEBUG:boto:Method: GET 2024-04-16T19:56:26.023 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.023 DEBUG:boto:Data: 2024-04-16T19:56:26.023 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.023 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.023 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.023 DEBUG:boto:Params: {} 2024-04-16T19:56:26.023 DEBUG:boto:Token: None 2024-04-16T19:56:26.024 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cXaeJGRNbT0U7WMYPb5hAHnBzM0= 2024-04-16T19:56:26.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXaeJGRNbT0U7WMYPb5hAHnBzM0=', 'Content-Length': '0'} 2024-04-16T19:56:26.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b6f4e1523490be-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.070 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.071 DEBUG:boto:Method: GET 2024-04-16T19:56:26.071 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.071 DEBUG:boto:Data: 2024-04-16T19:56:26.071 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.071 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.071 DEBUG:boto:Params: {} 2024-04-16T19:56:26.071 DEBUG:boto:Token: None 2024-04-16T19:56:26.071 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000018 2024-04-16T19:56:26.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cXaeJGRNbT0U7WMYPb5hAHnBzM0= 2024-04-16T19:56:26.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXaeJGRNbT0U7WMYPb5hAHnBzM0=', 'Content-Length': '0'} 2024-04-16T19:56:26.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f634def73964e8b9-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.074 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.074 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.074 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.074 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.074 DEBUG:boto:Method: GET 2024-04-16T19:56:26.074 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.074 DEBUG:boto:Data: 2024-04-16T19:56:26.074 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.074 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.075 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.075 DEBUG:boto:Params: {} 2024-04-16T19:56:26.075 DEBUG:boto:Token: None 2024-04-16T19:56:26.075 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1869a73122b538-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.078 DEBUG:boto:b'yrjefh-51obj000000000000000000181000falseobj000000000000000000182024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.078 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.078 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.078 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.078 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.078 DEBUG:boto:Method: GET 2024-04-16T19:56:26.078 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000018 2024-04-16T19:56:26.079 DEBUG:boto:Data: 2024-04-16T19:56:26.079 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.079 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.079 DEBUG:boto:Params: {} 2024-04-16T19:56:26.079 DEBUG:boto:Token: None 2024-04-16T19:56:26.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f24172fa84c53689-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.083 DEBUG:boto:b'yrjefh-51obj000000000000000000181000falseobj000000000000000000182024-04-16T19:56:09.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-16T19:56:26.084 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.084 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.084 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.084 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.084 DEBUG:boto:Method: GET 2024-04-16T19:56:26.084 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.084 DEBUG:boto:Data: 2024-04-16T19:56:26.084 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.084 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.085 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.085 DEBUG:boto:Params: {} 2024-04-16T19:56:26.085 DEBUG:boto:Token: None 2024-04-16T19:56:26.085 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM= 2024-04-16T19:56:26.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM=', 'Content-Length': '0'} 2024-04-16T19:56:26.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009551e9e4945fee41-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.088 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.088 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.088 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.088 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.088 DEBUG:boto:Method: GET 2024-04-16T19:56:26.089 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.089 DEBUG:boto:Data: 2024-04-16T19:56:26.089 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.089 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.089 DEBUG:boto:Params: {} 2024-04-16T19:56:26.089 DEBUG:boto:Token: None 2024-04-16T19:56:26.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM= 2024-04-16T19:56:26.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM=', 'Content-Length': '0'} 2024-04-16T19:56:26.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007927da04ea1758f6-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.134 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.135 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.135 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.135 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.135 DEBUG:boto:Data: 2024-04-16T19:56:26.135 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.135 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.135 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.135 DEBUG:boto:Params: {} 2024-04-16T19:56:26.135 DEBUG:boto:Token: None 2024-04-16T19:56:26.135 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+Oj6Y14hiJ4Kn+8QlG6xsImLy7M= 2024-04-16T19:56:26.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+Oj6Y14hiJ4Kn+8QlG6xsImLy7M=', 'Content-Length': '0'} 2024-04-16T19:56:26.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27a399ba2d3044d-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.138 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.138 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.138 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.138 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.138 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.138 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.138 DEBUG:boto:Data: 2024-04-16T19:56:26.138 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.139 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.139 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.139 DEBUG:boto:Params: {} 2024-04-16T19:56:26.139 DEBUG:boto:Token: None 2024-04-16T19:56:26.139 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+Oj6Y14hiJ4Kn+8QlG6xsImLy7M= 2024-04-16T19:56:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+Oj6Y14hiJ4Kn+8QlG6xsImLy7M=', 'Content-Length': '0'} 2024-04-16T19:56:26.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0396d605394c158-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-16T19:56:26.142 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.142 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.142 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.142 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.142 DEBUG:boto:Method: GET 2024-04-16T19:56:26.143 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.143 DEBUG:boto:Data: 2024-04-16T19:56:26.143 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.143 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.143 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.143 DEBUG:boto:Params: {} 2024-04-16T19:56:26.143 DEBUG:boto:Token: None 2024-04-16T19:56:26.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM= 2024-04-16T19:56:26.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM=', 'Content-Length': '0'} 2024-04-16T19:56:26.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea423fac9dbf6fc9-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.146 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.146 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.146 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.146 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.146 DEBUG:boto:Method: GET 2024-04-16T19:56:26.146 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.146 DEBUG:boto:Data: 2024-04-16T19:56:26.146 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.146 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.146 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.147 DEBUG:boto:Params: {} 2024-04-16T19:56:26.147 DEBUG:boto:Token: None 2024-04-16T19:56:26.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019?versionId=null 2024-04-16T19:56:26.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM= 2024-04-16T19:56:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FWBaqK3UQrS1pIqryt9oYsWuSVM=', 'Content-Length': '0'} 2024-04-16T19:56:26.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdff53131354518-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.149 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.150 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.150 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.150 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.150 DEBUG:boto:Method: GET 2024-04-16T19:56:26.150 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.150 DEBUG:boto:Data: 2024-04-16T19:56:26.150 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.150 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.150 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.150 DEBUG:boto:Params: {} 2024-04-16T19:56:26.150 DEBUG:boto:Token: None 2024-04-16T19:56:26.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29bb33fadbe02d6-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.153 DEBUG:boto:b'yrjefh-51obj000000000000000000191000falseobj000000000000000000192024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.154 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.154 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.154 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.154 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.154 DEBUG:boto:Method: GET 2024-04-16T19:56:26.154 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.154 DEBUG:boto:Data: 2024-04-16T19:56:26.154 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.155 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.155 DEBUG:boto:Params: {} 2024-04-16T19:56:26.155 DEBUG:boto:Token: None 2024-04-16T19:56:26.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8cd4bf2795f95d-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.159 DEBUG:boto:b'yrjefh-51obj000000000000000000191000falseobj000000000000000000192024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.160 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.160 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.160 DEBUG:boto:Data: 2024-04-16T19:56:26.160 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.160 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.160 DEBUG:boto:Params: {} 2024-04-16T19:56:26.160 DEBUG:boto:Token: None 2024-04-16T19:56:26.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DBRnCxongikzd6oaQ/GquqITgDI= 2024-04-16T19:56:26.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DBRnCxongikzd6oaQ/GquqITgDI=', 'Content-Length': '0'} 2024-04-16T19:56:26.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0fda088f27a4bee-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.163 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.163 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.163 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.163 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.163 DEBUG:boto:Data: 2024-04-16T19:56:26.163 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.164 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.164 DEBUG:boto:Params: {} 2024-04-16T19:56:26.164 DEBUG:boto:Token: None 2024-04-16T19:56:26.164 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DBRnCxongikzd6oaQ/GquqITgDI= 2024-04-16T19:56:26.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DBRnCxongikzd6oaQ/GquqITgDI=', 'Content-Length': '0'} 2024-04-16T19:56:26.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062f472cfb16e200-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-16T19:56:26.166 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.167 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.167 DEBUG:boto:Method: GET 2024-04-16T19:56:26.167 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.167 DEBUG:boto:Data: 2024-04-16T19:56:26.167 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.167 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.167 DEBUG:boto:Params: {} 2024-04-16T19:56:26.167 DEBUG:boto:Token: None 2024-04-16T19:56:26.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0wGQtvAi0Wxs8D3L2pIt+jwQ+zc= 2024-04-16T19:56:26.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0wGQtvAi0Wxs8D3L2pIt+jwQ+zc=', 'Content-Length': '0'} 2024-04-16T19:56:26.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018aea8de02e6c523-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.170 DEBUG:boto:path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.170 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.170 DEBUG:boto:Method: GET 2024-04-16T19:56:26.170 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.171 DEBUG:boto:Data: 2024-04-16T19:56:26.171 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.171 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.171 DEBUG:boto:Params: {} 2024-04-16T19:56:26.171 DEBUG:boto:Token: None 2024-04-16T19:56:26.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000019 2024-04-16T19:56:26.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0wGQtvAi0Wxs8D3L2pIt+jwQ+zc= 2024-04-16T19:56:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0wGQtvAi0Wxs8D3L2pIt+jwQ+zc=', 'Content-Length': '0'} 2024-04-16T19:56:26.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045be619394f7fd45-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.174 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.174 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.174 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.174 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.174 DEBUG:boto:Method: GET 2024-04-16T19:56:26.174 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.174 DEBUG:boto:Data: 2024-04-16T19:56:26.175 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.175 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.175 DEBUG:boto:Params: {} 2024-04-16T19:56:26.175 DEBUG:boto:Token: None 2024-04-16T19:56:26.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe0581843eec566-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.178 DEBUG:boto:b'yrjefh-51obj000000000000000000191000falseobj000000000000000000192024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.178 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.178 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.178 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.178 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.179 DEBUG:boto:Method: GET 2024-04-16T19:56:26.179 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000019 2024-04-16T19:56:26.179 DEBUG:boto:Data: 2024-04-16T19:56:26.179 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.179 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.179 DEBUG:boto:Params: {} 2024-04-16T19:56:26.179 DEBUG:boto:Token: None 2024-04-16T19:56:26.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000957887b3624a793f-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.183 DEBUG:boto:b'yrjefh-51obj000000000000000000191000falseobj000000000000000000192024-04-16T19:56:09.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-16T19:56:26.184 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.184 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.184 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.184 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.184 DEBUG:boto:Method: GET 2024-04-16T19:56:26.184 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.184 DEBUG:boto:Data: 2024-04-16T19:56:26.185 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.185 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.185 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.185 DEBUG:boto:Params: {} 2024-04-16T19:56:26.185 DEBUG:boto:Token: None 2024-04-16T19:56:26.185 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.185 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw= 2024-04-16T19:56:26.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw=', 'Content-Length': '0'} 2024-04-16T19:56:26.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5058579871afcd-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.188 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.188 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.188 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.188 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.188 DEBUG:boto:Method: GET 2024-04-16T19:56:26.188 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.189 DEBUG:boto:Data: 2024-04-16T19:56:26.189 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.189 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.189 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.189 DEBUG:boto:Params: {} 2024-04-16T19:56:26.189 DEBUG:boto:Token: None 2024-04-16T19:56:26.189 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw= 2024-04-16T19:56:26.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw=', 'Content-Length': '0'} 2024-04-16T19:56:26.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff97dfb8996d1b4-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.238 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.238 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.239 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.239 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.239 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.239 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.239 DEBUG:boto:Data: 2024-04-16T19:56:26.239 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.239 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.239 DEBUG:boto:Params: {} 2024-04-16T19:56:26.239 DEBUG:boto:Token: None 2024-04-16T19:56:26.239 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zYyevaeKUZKD0qavUOXHOaUNUlE= 2024-04-16T19:56:26.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zYyevaeKUZKD0qavUOXHOaUNUlE=', 'Content-Length': '0'} 2024-04-16T19:56:26.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4752ba09f4c1bd-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.242 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.242 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.243 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.243 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.243 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.243 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.243 DEBUG:boto:Data: 2024-04-16T19:56:26.243 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.243 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.243 DEBUG:boto:Params: {} 2024-04-16T19:56:26.243 DEBUG:boto:Token: None 2024-04-16T19:56:26.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zYyevaeKUZKD0qavUOXHOaUNUlE= 2024-04-16T19:56:26.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zYyevaeKUZKD0qavUOXHOaUNUlE=', 'Content-Length': '0'} 2024-04-16T19:56:26.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078001f28bffcc02f-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-16T19:56:26.246 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.246 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.246 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.246 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.246 DEBUG:boto:Method: GET 2024-04-16T19:56:26.246 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.246 DEBUG:boto:Data: 2024-04-16T19:56:26.246 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.246 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.247 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.247 DEBUG:boto:Params: {} 2024-04-16T19:56:26.247 DEBUG:boto:Token: None 2024-04-16T19:56:26.247 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.247 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw= 2024-04-16T19:56:26.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw=', 'Content-Length': '0'} 2024-04-16T19:56:26.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76a05ac458fb2b0-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.249 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.250 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.250 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.250 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.250 DEBUG:boto:Method: GET 2024-04-16T19:56:26.250 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.250 DEBUG:boto:Data: 2024-04-16T19:56:26.250 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.250 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.250 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.250 DEBUG:boto:Params: {} 2024-04-16T19:56:26.251 DEBUG:boto:Token: None 2024-04-16T19:56:26.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020?versionId=null 2024-04-16T19:56:26.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw= 2024-04-16T19:56:26.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwU9ZbUVafl2llGGjsf0DS4Bwfw=', 'Content-Length': '0'} 2024-04-16T19:56:26.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13b6fb7a55bf486-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.253 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.253 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.253 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.253 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.254 DEBUG:boto:Method: GET 2024-04-16T19:56:26.254 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.254 DEBUG:boto:Data: 2024-04-16T19:56:26.254 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.254 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.254 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.254 DEBUG:boto:Params: {} 2024-04-16T19:56:26.254 DEBUG:boto:Token: None 2024-04-16T19:56:26.254 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.254 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf36c1884af1d15-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.257 DEBUG:boto:b'yrjefh-51obj000000000000000000201000falseobj000000000000000000202024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.258 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.258 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.258 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.258 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.258 DEBUG:boto:Method: GET 2024-04-16T19:56:26.258 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.258 DEBUG:boto:Data: 2024-04-16T19:56:26.258 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.258 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.258 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.258 DEBUG:boto:Params: {} 2024-04-16T19:56:26.259 DEBUG:boto:Token: None 2024-04-16T19:56:26.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12389a5a3b8e159-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.263 DEBUG:boto:b'yrjefh-51obj000000000000000000201000falseobj000000000000000000202024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.263 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.263 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.263 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.263 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.263 DEBUG:boto:Data: 2024-04-16T19:56:26.263 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.264 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.264 DEBUG:boto:Params: {} 2024-04-16T19:56:26.264 DEBUG:boto:Token: None 2024-04-16T19:56:26.264 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M7gWkNQy4EqIzFPP/l8x+WAfp3I= 2024-04-16T19:56:26.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M7gWkNQy4EqIzFPP/l8x+WAfp3I=', 'Content-Length': '0'} 2024-04-16T19:56:26.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5c0313842c19ffc-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.267 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.267 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.267 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.267 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.267 DEBUG:boto:Data: 2024-04-16T19:56:26.267 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.267 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.267 DEBUG:boto:Params: {} 2024-04-16T19:56:26.268 DEBUG:boto:Token: None 2024-04-16T19:56:26.268 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M7gWkNQy4EqIzFPP/l8x+WAfp3I= 2024-04-16T19:56:26.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M7gWkNQy4EqIzFPP/l8x+WAfp3I=', 'Content-Length': '0'} 2024-04-16T19:56:26.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f784a1c26b263ab-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-16T19:56:26.271 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.271 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.271 DEBUG:boto:Method: GET 2024-04-16T19:56:26.271 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.271 DEBUG:boto:Data: 2024-04-16T19:56:26.271 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.271 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.271 DEBUG:boto:Params: {} 2024-04-16T19:56:26.271 DEBUG:boto:Token: None 2024-04-16T19:56:26.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8oIbAnQOdf3wG52CK+ZbtjEvDjE= 2024-04-16T19:56:26.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8oIbAnQOdf3wG52CK+ZbtjEvDjE=', 'Content-Length': '0'} 2024-04-16T19:56:26.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007333476a8a5fe2b1-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.314 DEBUG:boto:path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.315 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.315 DEBUG:boto:Method: GET 2024-04-16T19:56:26.315 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.315 DEBUG:boto:Data: 2024-04-16T19:56:26.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.315 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.315 DEBUG:boto:Params: {} 2024-04-16T19:56:26.315 DEBUG:boto:Token: None 2024-04-16T19:56:26.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000020 2024-04-16T19:56:26.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8oIbAnQOdf3wG52CK+ZbtjEvDjE= 2024-04-16T19:56:26.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8oIbAnQOdf3wG52CK+ZbtjEvDjE=', 'Content-Length': '0'} 2024-04-16T19:56:26.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf0cc9cb64f8dc31-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.318 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.318 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.318 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.318 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.319 DEBUG:boto:Method: GET 2024-04-16T19:56:26.319 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.319 DEBUG:boto:Data: 2024-04-16T19:56:26.319 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.319 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.319 DEBUG:boto:Params: {} 2024-04-16T19:56:26.319 DEBUG:boto:Token: None 2024-04-16T19:56:26.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020998fd680fefc3d-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.322 DEBUG:boto:b'yrjefh-51obj000000000000000000201000falseobj000000000000000000202024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.322 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.323 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.323 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.323 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.323 DEBUG:boto:Method: GET 2024-04-16T19:56:26.323 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000020 2024-04-16T19:56:26.323 DEBUG:boto:Data: 2024-04-16T19:56:26.323 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.323 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.323 DEBUG:boto:Params: {} 2024-04-16T19:56:26.323 DEBUG:boto:Token: None 2024-04-16T19:56:26.324 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddecb610479a8665-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.328 DEBUG:boto:b'yrjefh-51obj000000000000000000201000falseobj000000000000000000202024-04-16T19:56:08.642Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-16T19:56:26.328 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.329 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.329 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.329 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.329 DEBUG:boto:Method: GET 2024-04-16T19:56:26.329 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.329 DEBUG:boto:Data: 2024-04-16T19:56:26.329 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.329 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.329 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.329 DEBUG:boto:Params: {} 2024-04-16T19:56:26.330 DEBUG:boto:Token: None 2024-04-16T19:56:26.330 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.330 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc= 2024-04-16T19:56:26.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc=', 'Content-Length': '0'} 2024-04-16T19:56:26.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053fd85190265a0f9-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.332 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.333 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.333 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.333 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.333 DEBUG:boto:Method: GET 2024-04-16T19:56:26.333 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.333 DEBUG:boto:Data: 2024-04-16T19:56:26.333 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.333 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.333 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.333 DEBUG:boto:Params: {} 2024-04-16T19:56:26.333 DEBUG:boto:Token: None 2024-04-16T19:56:26.334 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc= 2024-04-16T19:56:26.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc=', 'Content-Length': '0'} 2024-04-16T19:56:26.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9719daa1e884c22-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.379 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.379 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.379 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.379 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.379 DEBUG:boto:Data: 2024-04-16T19:56:26.379 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.379 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.379 DEBUG:boto:Params: {} 2024-04-16T19:56:26.379 DEBUG:boto:Token: None 2024-04-16T19:56:26.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cGgYT8jA4Xz/11MbsLRtjLzQYd0= 2024-04-16T19:56:26.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cGgYT8jA4Xz/11MbsLRtjLzQYd0=', 'Content-Length': '0'} 2024-04-16T19:56:26.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142cc82e3f61ff49-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.382 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.382 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.382 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.382 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.382 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.382 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.382 DEBUG:boto:Data: 2024-04-16T19:56:26.383 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.383 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.383 DEBUG:boto:Params: {} 2024-04-16T19:56:26.383 DEBUG:boto:Token: None 2024-04-16T19:56:26.383 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cGgYT8jA4Xz/11MbsLRtjLzQYd0= 2024-04-16T19:56:26.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cGgYT8jA4Xz/11MbsLRtjLzQYd0=', 'Content-Length': '0'} 2024-04-16T19:56:26.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0c8a4e426640e9c-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-16T19:56:26.386 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.386 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.386 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.386 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.386 DEBUG:boto:Method: GET 2024-04-16T19:56:26.386 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.386 DEBUG:boto:Data: 2024-04-16T19:56:26.386 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.387 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.387 DEBUG:boto:Params: {} 2024-04-16T19:56:26.387 DEBUG:boto:Token: None 2024-04-16T19:56:26.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc= 2024-04-16T19:56:26.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc=', 'Content-Length': '0'} 2024-04-16T19:56:26.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284327a600b41494-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.390 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.390 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.390 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.390 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.390 DEBUG:boto:Method: GET 2024-04-16T19:56:26.390 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.390 DEBUG:boto:Data: 2024-04-16T19:56:26.390 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.390 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.390 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.390 DEBUG:boto:Params: {} 2024-04-16T19:56:26.391 DEBUG:boto:Token: None 2024-04-16T19:56:26.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021?versionId=null 2024-04-16T19:56:26.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc= 2024-04-16T19:56:26.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8immWqJ5O5DOw/PeFFB/pkrSOVc=', 'Content-Length': '0'} 2024-04-16T19:56:26.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b90d8b7f6c6214b-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.393 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.393 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.393 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.393 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.394 DEBUG:boto:Method: GET 2024-04-16T19:56:26.394 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.394 DEBUG:boto:Data: 2024-04-16T19:56:26.394 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.394 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.394 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.394 DEBUG:boto:Params: {} 2024-04-16T19:56:26.394 DEBUG:boto:Token: None 2024-04-16T19:56:26.394 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ead0a4bda4a05a45-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.397 DEBUG:boto:b'yrjefh-51obj000000000000000000211000falseobj000000000000000000212024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.397 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.397 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.398 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.398 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.398 DEBUG:boto:Method: GET 2024-04-16T19:56:26.398 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.398 DEBUG:boto:Data: 2024-04-16T19:56:26.398 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.398 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.398 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.398 DEBUG:boto:Params: {} 2024-04-16T19:56:26.398 DEBUG:boto:Token: None 2024-04-16T19:56:26.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8c80527c47ff50-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.403 DEBUG:boto:b'yrjefh-51obj000000000000000000211000falseobj000000000000000000212024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.404 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.404 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.404 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.404 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.404 DEBUG:boto:Data: 2024-04-16T19:56:26.404 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.404 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.404 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.404 DEBUG:boto:Params: {} 2024-04-16T19:56:26.404 DEBUG:boto:Token: None 2024-04-16T19:56:26.405 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lRl4HyZNK3G16IRcdlxvDM9dVQY= 2024-04-16T19:56:26.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lRl4HyZNK3G16IRcdlxvDM9dVQY=', 'Content-Length': '0'} 2024-04-16T19:56:26.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3868e005f92650b-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.407 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.407 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.407 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.407 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.407 DEBUG:boto:Data: 2024-04-16T19:56:26.407 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.408 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.408 DEBUG:boto:Params: {} 2024-04-16T19:56:26.408 DEBUG:boto:Token: None 2024-04-16T19:56:26.408 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lRl4HyZNK3G16IRcdlxvDM9dVQY= 2024-04-16T19:56:26.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lRl4HyZNK3G16IRcdlxvDM9dVQY=', 'Content-Length': '0'} 2024-04-16T19:56:26.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ebccf7760eaccf-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-16T19:56:26.410 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.410 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.410 DEBUG:boto:Method: GET 2024-04-16T19:56:26.411 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.411 DEBUG:boto:Data: 2024-04-16T19:56:26.411 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.411 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.411 DEBUG:boto:Params: {} 2024-04-16T19:56:26.411 DEBUG:boto:Token: None 2024-04-16T19:56:26.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:opnfY+gWwsMAJijhjaboKVkdzRI= 2024-04-16T19:56:26.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:opnfY+gWwsMAJijhjaboKVkdzRI=', 'Content-Length': '0'} 2024-04-16T19:56:26.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7715936076d34f6-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.414 DEBUG:boto:path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.414 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.414 DEBUG:boto:Method: GET 2024-04-16T19:56:26.414 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.414 DEBUG:boto:Data: 2024-04-16T19:56:26.414 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.414 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.414 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.414 DEBUG:boto:Params: {} 2024-04-16T19:56:26.414 DEBUG:boto:Token: None 2024-04-16T19:56:26.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000021 2024-04-16T19:56:26.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:opnfY+gWwsMAJijhjaboKVkdzRI= 2024-04-16T19:56:26.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:opnfY+gWwsMAJijhjaboKVkdzRI=', 'Content-Length': '0'} 2024-04-16T19:56:26.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeaf21149c8fcdcb-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.417 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.417 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.417 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.417 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.418 DEBUG:boto:Method: GET 2024-04-16T19:56:26.418 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.418 DEBUG:boto:Data: 2024-04-16T19:56:26.418 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.418 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.418 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.418 DEBUG:boto:Params: {} 2024-04-16T19:56:26.418 DEBUG:boto:Token: None 2024-04-16T19:56:26.418 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.418 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e4af2217bb5dc9-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.421 DEBUG:boto:b'yrjefh-51obj000000000000000000211000falseobj000000000000000000212024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.422 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.422 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.422 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.422 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.422 DEBUG:boto:Method: GET 2024-04-16T19:56:26.422 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000021 2024-04-16T19:56:26.422 DEBUG:boto:Data: 2024-04-16T19:56:26.422 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.422 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.422 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.422 DEBUG:boto:Params: {} 2024-04-16T19:56:26.423 DEBUG:boto:Token: None 2024-04-16T19:56:26.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f8110e346f40049-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.427 DEBUG:boto:b'yrjefh-51obj000000000000000000211000falseobj000000000000000000212024-04-16T19:56:08.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-16T19:56:26.428 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.428 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.428 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.428 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.428 DEBUG:boto:Method: GET 2024-04-16T19:56:26.428 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.428 DEBUG:boto:Data: 2024-04-16T19:56:26.428 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.428 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.428 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.429 DEBUG:boto:Params: {} 2024-04-16T19:56:26.429 DEBUG:boto:Token: None 2024-04-16T19:56:26.429 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.429 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc= 2024-04-16T19:56:26.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc=', 'Content-Length': '0'} 2024-04-16T19:56:26.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ecc7da8ab7504b-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.431 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.431 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.432 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.432 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.432 DEBUG:boto:Method: GET 2024-04-16T19:56:26.432 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.432 DEBUG:boto:Data: 2024-04-16T19:56:26.432 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.432 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.432 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.432 DEBUG:boto:Params: {} 2024-04-16T19:56:26.432 DEBUG:boto:Token: None 2024-04-16T19:56:26.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc= 2024-04-16T19:56:26.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc=', 'Content-Length': '0'} 2024-04-16T19:56:26.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4c2bba946883d0-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.478 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.478 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.479 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.479 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.479 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.479 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.479 DEBUG:boto:Data: 2024-04-16T19:56:26.479 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.479 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.479 DEBUG:boto:Params: {} 2024-04-16T19:56:26.479 DEBUG:boto:Token: None 2024-04-16T19:56:26.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ7WVTt3TkcRi9GTUFtPMhNq1Ow= 2024-04-16T19:56:26.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ7WVTt3TkcRi9GTUFtPMhNq1Ow=', 'Content-Length': '0'} 2024-04-16T19:56:26.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f224f030659a0a08-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.481 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.482 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.482 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.482 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.482 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.482 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.482 DEBUG:boto:Data: 2024-04-16T19:56:26.482 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.482 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.482 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.482 DEBUG:boto:Params: {} 2024-04-16T19:56:26.483 DEBUG:boto:Token: None 2024-04-16T19:56:26.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ7WVTt3TkcRi9GTUFtPMhNq1Ow= 2024-04-16T19:56:26.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ7WVTt3TkcRi9GTUFtPMhNq1Ow=', 'Content-Length': '0'} 2024-04-16T19:56:26.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248facf31709000a-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-16T19:56:26.485 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.485 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.485 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.485 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.485 DEBUG:boto:Method: GET 2024-04-16T19:56:26.485 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.486 DEBUG:boto:Data: 2024-04-16T19:56:26.486 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.486 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.486 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.486 DEBUG:boto:Params: {} 2024-04-16T19:56:26.486 DEBUG:boto:Token: None 2024-04-16T19:56:26.486 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc= 2024-04-16T19:56:26.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc=', 'Content-Length': '0'} 2024-04-16T19:56:26.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f29c63f42a211b-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.488 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.488 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.488 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.489 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.489 DEBUG:boto:Method: GET 2024-04-16T19:56:26.489 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.489 DEBUG:boto:Data: 2024-04-16T19:56:26.489 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.489 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.489 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.489 DEBUG:boto:Params: {} 2024-04-16T19:56:26.489 DEBUG:boto:Token: None 2024-04-16T19:56:26.489 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022?versionId=null 2024-04-16T19:56:26.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc= 2024-04-16T19:56:26.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/a8wChu/tlHc/6RXbRE0J5AIvc=', 'Content-Length': '0'} 2024-04-16T19:56:26.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141e9e25fbc77f45-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.491 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.492 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.492 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.492 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.492 DEBUG:boto:Method: GET 2024-04-16T19:56:26.492 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.492 DEBUG:boto:Data: 2024-04-16T19:56:26.492 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.492 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.492 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.492 DEBUG:boto:Params: {} 2024-04-16T19:56:26.492 DEBUG:boto:Token: None 2024-04-16T19:56:26.493 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.493 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008003594537a86bcb-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.495 DEBUG:boto:b'yrjefh-51obj000000000000000000221000falseobj000000000000000000222024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.496 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.496 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.496 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.496 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.496 DEBUG:boto:Method: GET 2024-04-16T19:56:26.496 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.496 DEBUG:boto:Data: 2024-04-16T19:56:26.496 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.496 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.496 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.496 DEBUG:boto:Params: {} 2024-04-16T19:56:26.497 DEBUG:boto:Token: None 2024-04-16T19:56:26.497 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a1291c82a0d731-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.501 DEBUG:boto:b'yrjefh-51obj000000000000000000221000falseobj000000000000000000222024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.501 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.501 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.501 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.501 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.501 DEBUG:boto:Data: 2024-04-16T19:56:26.501 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.501 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.501 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.502 DEBUG:boto:Params: {} 2024-04-16T19:56:26.502 DEBUG:boto:Token: None 2024-04-16T19:56:26.502 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BgwDxx2QWwCzRb4BBgskD1oOZ6E= 2024-04-16T19:56:26.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BgwDxx2QWwCzRb4BBgskD1oOZ6E=', 'Content-Length': '0'} 2024-04-16T19:56:26.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5744b14362268f-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.504 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.504 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.504 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.504 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.504 DEBUG:boto:Data: 2024-04-16T19:56:26.505 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.505 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.505 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.505 DEBUG:boto:Params: {} 2024-04-16T19:56:26.505 DEBUG:boto:Token: None 2024-04-16T19:56:26.505 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.505 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BgwDxx2QWwCzRb4BBgskD1oOZ6E= 2024-04-16T19:56:26.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BgwDxx2QWwCzRb4BBgskD1oOZ6E=', 'Content-Length': '0'} 2024-04-16T19:56:26.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e780bbc1ce3118-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-16T19:56:26.507 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.507 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.508 DEBUG:boto:Method: GET 2024-04-16T19:56:26.508 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.508 DEBUG:boto:Data: 2024-04-16T19:56:26.508 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.508 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.508 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.508 DEBUG:boto:Params: {} 2024-04-16T19:56:26.508 DEBUG:boto:Token: None 2024-04-16T19:56:26.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cpct35RMyNR0Fgpb/KzVbNLj8Z4= 2024-04-16T19:56:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cpct35RMyNR0Fgpb/KzVbNLj8Z4=', 'Content-Length': '0'} 2024-04-16T19:56:26.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177a0fe33f6a5bfe-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.550 DEBUG:boto:path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.550 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.551 DEBUG:boto:Method: GET 2024-04-16T19:56:26.551 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.551 DEBUG:boto:Data: 2024-04-16T19:56:26.551 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.551 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.551 DEBUG:boto:Params: {} 2024-04-16T19:56:26.551 DEBUG:boto:Token: None 2024-04-16T19:56:26.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000022 2024-04-16T19:56:26.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cpct35RMyNR0Fgpb/KzVbNLj8Z4= 2024-04-16T19:56:26.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cpct35RMyNR0Fgpb/KzVbNLj8Z4=', 'Content-Length': '0'} 2024-04-16T19:56:26.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89048728eca42b0-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.554 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.554 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.554 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.554 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.554 DEBUG:boto:Method: GET 2024-04-16T19:56:26.554 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.554 DEBUG:boto:Data: 2024-04-16T19:56:26.554 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.554 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.554 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.555 DEBUG:boto:Params: {} 2024-04-16T19:56:26.555 DEBUG:boto:Token: None 2024-04-16T19:56:26.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070c1faf95b81f486-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.558 DEBUG:boto:b'yrjefh-51obj000000000000000000221000falseobj000000000000000000222024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.558 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.558 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.558 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.558 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.558 DEBUG:boto:Method: GET 2024-04-16T19:56:26.559 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000022 2024-04-16T19:56:26.559 DEBUG:boto:Data: 2024-04-16T19:56:26.559 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.559 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.559 DEBUG:boto:Params: {} 2024-04-16T19:56:26.559 DEBUG:boto:Token: None 2024-04-16T19:56:26.559 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eba875d2d300cbda-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.563 DEBUG:boto:b'yrjefh-51obj000000000000000000221000falseobj000000000000000000222024-04-16T19:56:08.173Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-16T19:56:26.564 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.564 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.564 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.564 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.564 DEBUG:boto:Method: GET 2024-04-16T19:56:26.564 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.564 DEBUG:boto:Data: 2024-04-16T19:56:26.564 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.565 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.565 DEBUG:boto:Params: {} 2024-04-16T19:56:26.565 DEBUG:boto:Token: None 2024-04-16T19:56:26.565 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ= 2024-04-16T19:56:26.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b422190b84c4b1e-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.568 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.568 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.568 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.568 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.568 DEBUG:boto:Method: GET 2024-04-16T19:56:26.568 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.568 DEBUG:boto:Data: 2024-04-16T19:56:26.568 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.568 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.568 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.568 DEBUG:boto:Params: {} 2024-04-16T19:56:26.569 DEBUG:boto:Token: None 2024-04-16T19:56:26.569 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.569 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ= 2024-04-16T19:56:26.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b7ebdfd8dfe3db-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.571 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.571 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.572 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.572 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.572 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.572 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.572 DEBUG:boto:Data: 2024-04-16T19:56:26.572 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.572 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.572 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.572 DEBUG:boto:Params: {} 2024-04-16T19:56:26.572 DEBUG:boto:Token: None 2024-04-16T19:56:26.572 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.573 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:heRZ2pP0faNGOpyfcBuqPYZGbkE= 2024-04-16T19:56:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heRZ2pP0faNGOpyfcBuqPYZGbkE=', 'Content-Length': '0'} 2024-04-16T19:56:26.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4bbc01c51ce06b-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.575 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.575 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.575 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.575 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.575 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.575 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.575 DEBUG:boto:Data: 2024-04-16T19:56:26.575 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.575 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.575 DEBUG:boto:Params: {} 2024-04-16T19:56:26.576 DEBUG:boto:Token: None 2024-04-16T19:56:26.576 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:heRZ2pP0faNGOpyfcBuqPYZGbkE= 2024-04-16T19:56:26.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heRZ2pP0faNGOpyfcBuqPYZGbkE=', 'Content-Length': '0'} 2024-04-16T19:56:26.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850e9f01719a0ef5-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-16T19:56:26.578 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.579 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.579 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.579 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.579 DEBUG:boto:Method: GET 2024-04-16T19:56:26.579 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.579 DEBUG:boto:Data: 2024-04-16T19:56:26.579 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.579 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.579 DEBUG:boto:Params: {} 2024-04-16T19:56:26.579 DEBUG:boto:Token: None 2024-04-16T19:56:26.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ= 2024-04-16T19:56:26.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000052f59afba950232-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.622 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.622 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.622 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.623 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.623 DEBUG:boto:Method: GET 2024-04-16T19:56:26.623 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.623 DEBUG:boto:Data: 2024-04-16T19:56:26.623 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.623 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.623 DEBUG:boto:Params: {} 2024-04-16T19:56:26.623 DEBUG:boto:Token: None 2024-04-16T19:56:26.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023?versionId=null 2024-04-16T19:56:26.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ= 2024-04-16T19:56:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xWS+4xHJMW4hkyJlU7USoEBgVYQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f96cfa3adebfb6-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.626 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.626 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.626 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.626 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.626 DEBUG:boto:Method: GET 2024-04-16T19:56:26.626 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.627 DEBUG:boto:Data: 2024-04-16T19:56:26.627 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.627 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.627 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.627 DEBUG:boto:Params: {} 2024-04-16T19:56:26.627 DEBUG:boto:Token: None 2024-04-16T19:56:26.627 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b73eed462bb7d63-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.630 DEBUG:boto:b'yrjefh-51obj000000000000000000231000falseobj000000000000000000232024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.630 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.630 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.630 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.631 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.631 DEBUG:boto:Method: GET 2024-04-16T19:56:26.631 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.631 DEBUG:boto:Data: 2024-04-16T19:56:26.631 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.631 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.631 DEBUG:boto:Params: {} 2024-04-16T19:56:26.631 DEBUG:boto:Token: None 2024-04-16T19:56:26.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733e594cb7b9f5b6-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.636 DEBUG:boto:b'yrjefh-51obj000000000000000000231000falseobj000000000000000000232024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.636 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.636 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.636 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.636 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.636 DEBUG:boto:Data: 2024-04-16T19:56:26.636 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.636 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.636 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.637 DEBUG:boto:Params: {} 2024-04-16T19:56:26.637 DEBUG:boto:Token: None 2024-04-16T19:56:26.637 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rO8IMdwUWsCpz0Us8JoEf6/UGnc= 2024-04-16T19:56:26.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rO8IMdwUWsCpz0Us8JoEf6/UGnc=', 'Content-Length': '0'} 2024-04-16T19:56:26.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa587b9684d478ec-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.639 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.639 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.639 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.639 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.639 DEBUG:boto:Data: 2024-04-16T19:56:26.639 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.640 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.640 DEBUG:boto:Params: {} 2024-04-16T19:56:26.640 DEBUG:boto:Token: None 2024-04-16T19:56:26.640 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rO8IMdwUWsCpz0Us8JoEf6/UGnc= 2024-04-16T19:56:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rO8IMdwUWsCpz0Us8JoEf6/UGnc=', 'Content-Length': '0'} 2024-04-16T19:56:26.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784bcceb0fc74090-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-16T19:56:26.642 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.642 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.643 DEBUG:boto:Method: GET 2024-04-16T19:56:26.643 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.643 DEBUG:boto:Data: 2024-04-16T19:56:26.643 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.643 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.643 DEBUG:boto:Params: {} 2024-04-16T19:56:26.643 DEBUG:boto:Token: None 2024-04-16T19:56:26.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcS5LxVl2mNXhkM/UWNPm7aoYI8= 2024-04-16T19:56:26.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcS5LxVl2mNXhkM/UWNPm7aoYI8=', 'Content-Length': '0'} 2024-04-16T19:56:26.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db1def27a80c0acf-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.646 DEBUG:boto:path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.646 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.646 DEBUG:boto:Method: GET 2024-04-16T19:56:26.646 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.646 DEBUG:boto:Data: 2024-04-16T19:56:26.646 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.646 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.646 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.646 DEBUG:boto:Params: {} 2024-04-16T19:56:26.647 DEBUG:boto:Token: None 2024-04-16T19:56:26.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000023 2024-04-16T19:56:26.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vcS5LxVl2mNXhkM/UWNPm7aoYI8= 2024-04-16T19:56:26.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcS5LxVl2mNXhkM/UWNPm7aoYI8=', 'Content-Length': '0'} 2024-04-16T19:56:26.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000902661c9cf2501cb-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.690 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.691 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.691 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.691 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.691 DEBUG:boto:Method: GET 2024-04-16T19:56:26.691 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.691 DEBUG:boto:Data: 2024-04-16T19:56:26.691 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.691 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.691 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.691 DEBUG:boto:Params: {} 2024-04-16T19:56:26.692 DEBUG:boto:Token: None 2024-04-16T19:56:26.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c91eb926dbc7bfd-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.695 DEBUG:boto:b'yrjefh-51obj000000000000000000231000falseobj000000000000000000232024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.695 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.695 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.695 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.695 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.695 DEBUG:boto:Method: GET 2024-04-16T19:56:26.696 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000023 2024-04-16T19:56:26.696 DEBUG:boto:Data: 2024-04-16T19:56:26.696 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.696 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.696 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.696 DEBUG:boto:Params: {} 2024-04-16T19:56:26.696 DEBUG:boto:Token: None 2024-04-16T19:56:26.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fadb3f6e34d9b4e4-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.700 DEBUG:boto:b'yrjefh-51obj000000000000000000231000falseobj000000000000000000232024-04-16T19:56:07.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-16T19:56:26.701 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.701 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.701 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.701 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.701 DEBUG:boto:Method: GET 2024-04-16T19:56:26.701 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.701 DEBUG:boto:Data: 2024-04-16T19:56:26.701 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.702 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.702 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.702 DEBUG:boto:Params: {} 2024-04-16T19:56:26.702 DEBUG:boto:Token: None 2024-04-16T19:56:26.702 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0= 2024-04-16T19:56:26.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0=', 'Content-Length': '0'} 2024-04-16T19:56:26.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1f4e7efc385d3d-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.746 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.746 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.746 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.747 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.747 DEBUG:boto:Method: GET 2024-04-16T19:56:26.747 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.747 DEBUG:boto:Data: 2024-04-16T19:56:26.747 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.747 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.747 DEBUG:boto:Params: {} 2024-04-16T19:56:26.747 DEBUG:boto:Token: None 2024-04-16T19:56:26.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0= 2024-04-16T19:56:26.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0=', 'Content-Length': '0'} 2024-04-16T19:56:26.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cc4a09f8f0d9d2-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.750 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.750 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.751 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.751 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.751 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.751 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.751 DEBUG:boto:Data: 2024-04-16T19:56:26.751 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.751 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.751 DEBUG:boto:Params: {} 2024-04-16T19:56:26.751 DEBUG:boto:Token: None 2024-04-16T19:56:26.751 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o9sNjAn331HOUYAyTgw0G/n421Y= 2024-04-16T19:56:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9sNjAn331HOUYAyTgw0G/n421Y=', 'Content-Length': '0'} 2024-04-16T19:56:26.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb3862b4d8d2eabb-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.754 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.754 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.754 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.754 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.754 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.754 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.754 DEBUG:boto:Data: 2024-04-16T19:56:26.754 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.754 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.755 DEBUG:boto:Params: {} 2024-04-16T19:56:26.755 DEBUG:boto:Token: None 2024-04-16T19:56:26.755 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o9sNjAn331HOUYAyTgw0G/n421Y= 2024-04-16T19:56:26.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9sNjAn331HOUYAyTgw0G/n421Y=', 'Content-Length': '0'} 2024-04-16T19:56:26.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da049717dea8650e-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-16T19:56:26.758 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.758 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.758 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.758 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.758 DEBUG:boto:Method: GET 2024-04-16T19:56:26.758 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.758 DEBUG:boto:Data: 2024-04-16T19:56:26.758 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.758 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.758 DEBUG:boto:Params: {} 2024-04-16T19:56:26.759 DEBUG:boto:Token: None 2024-04-16T19:56:26.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0= 2024-04-16T19:56:26.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0=', 'Content-Length': '0'} 2024-04-16T19:56:26.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5313a1570f787f1-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.762 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.762 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.762 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.762 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.762 DEBUG:boto:Method: GET 2024-04-16T19:56:26.762 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.762 DEBUG:boto:Data: 2024-04-16T19:56:26.763 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.763 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.763 DEBUG:boto:Params: {} 2024-04-16T19:56:26.763 DEBUG:boto:Token: None 2024-04-16T19:56:26.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024?versionId=null 2024-04-16T19:56:26.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0= 2024-04-16T19:56:26.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wpwXuENLxzoOgj7Y6tOuogQt4V0=', 'Content-Length': '0'} 2024-04-16T19:56:26.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d092ab3f582d03d-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.766 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.766 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.766 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.766 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.766 DEBUG:boto:Method: GET 2024-04-16T19:56:26.766 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.766 DEBUG:boto:Data: 2024-04-16T19:56:26.766 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.766 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.767 DEBUG:boto:Params: {} 2024-04-16T19:56:26.767 DEBUG:boto:Token: None 2024-04-16T19:56:26.767 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3ba9b67d1ad22e-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.770 DEBUG:boto:b'yrjefh-51obj000000000000000000241000falseobj000000000000000000242024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.770 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.770 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.770 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.770 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.770 DEBUG:boto:Method: GET 2024-04-16T19:56:26.770 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.771 DEBUG:boto:Data: 2024-04-16T19:56:26.771 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.771 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.771 DEBUG:boto:Params: {} 2024-04-16T19:56:26.771 DEBUG:boto:Token: None 2024-04-16T19:56:26.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c71b1f686eecea6d-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.775 DEBUG:boto:b'yrjefh-51obj000000000000000000241000falseobj000000000000000000242024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.775 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.775 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.775 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.775 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.775 DEBUG:boto:Data: 2024-04-16T19:56:26.775 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.776 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.776 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.776 DEBUG:boto:Params: {} 2024-04-16T19:56:26.776 DEBUG:boto:Token: None 2024-04-16T19:56:26.776 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dS3rh3qbzftVCXItycIkqC5qLHM= 2024-04-16T19:56:26.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dS3rh3qbzftVCXItycIkqC5qLHM=', 'Content-Length': '0'} 2024-04-16T19:56:26.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fb984b91aa5512-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.778 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.778 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.779 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.779 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.779 DEBUG:boto:Data: 2024-04-16T19:56:26.779 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.779 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.779 DEBUG:boto:Params: {} 2024-04-16T19:56:26.779 DEBUG:boto:Token: None 2024-04-16T19:56:26.779 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dS3rh3qbzftVCXItycIkqC5qLHM= 2024-04-16T19:56:26.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dS3rh3qbzftVCXItycIkqC5qLHM=', 'Content-Length': '0'} 2024-04-16T19:56:26.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49f9f8cc4cbba29-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-16T19:56:26.782 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.782 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.782 DEBUG:boto:Method: GET 2024-04-16T19:56:26.782 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.782 DEBUG:boto:Data: 2024-04-16T19:56:26.782 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.782 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.782 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.782 DEBUG:boto:Params: {} 2024-04-16T19:56:26.782 DEBUG:boto:Token: None 2024-04-16T19:56:26.782 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uJxXiJRZSobVjWPNdl4DBACHGMA= 2024-04-16T19:56:26.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uJxXiJRZSobVjWPNdl4DBACHGMA=', 'Content-Length': '0'} 2024-04-16T19:56:26.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645d1ab3a7954193-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.785 DEBUG:boto:path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.785 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.785 DEBUG:boto:Method: GET 2024-04-16T19:56:26.785 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.785 DEBUG:boto:Data: 2024-04-16T19:56:26.785 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.785 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.785 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.786 DEBUG:boto:Params: {} 2024-04-16T19:56:26.786 DEBUG:boto:Token: None 2024-04-16T19:56:26.786 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000024 2024-04-16T19:56:26.786 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uJxXiJRZSobVjWPNdl4DBACHGMA= 2024-04-16T19:56:26.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uJxXiJRZSobVjWPNdl4DBACHGMA=', 'Content-Length': '0'} 2024-04-16T19:56:26.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a448d602c299bfd-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.788 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.788 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.789 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.789 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.789 DEBUG:boto:Method: GET 2024-04-16T19:56:26.789 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.789 DEBUG:boto:Data: 2024-04-16T19:56:26.789 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.789 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.789 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.789 DEBUG:boto:Params: {} 2024-04-16T19:56:26.789 DEBUG:boto:Token: None 2024-04-16T19:56:26.789 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e87495bcda567e8f-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.792 DEBUG:boto:b'yrjefh-51obj000000000000000000241000falseobj000000000000000000242024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.793 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.793 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.793 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.793 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.793 DEBUG:boto:Method: GET 2024-04-16T19:56:26.793 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000024 2024-04-16T19:56:26.793 DEBUG:boto:Data: 2024-04-16T19:56:26.793 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.793 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.793 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.793 DEBUG:boto:Params: {} 2024-04-16T19:56:26.794 DEBUG:boto:Token: None 2024-04-16T19:56:26.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ff578de5a012fb-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.798 DEBUG:boto:b'yrjefh-51obj000000000000000000241000falseobj000000000000000000242024-04-16T19:56:09.386Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-16T19:56:26.798 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.798 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.799 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.799 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.799 DEBUG:boto:Method: GET 2024-04-16T19:56:26.799 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.799 DEBUG:boto:Data: 2024-04-16T19:56:26.799 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.799 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.799 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.799 DEBUG:boto:Params: {} 2024-04-16T19:56:26.799 DEBUG:boto:Token: None 2024-04-16T19:56:26.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ= 2024-04-16T19:56:26.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20915371c7e0ab4-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.802 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.802 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.802 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.802 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.802 DEBUG:boto:Method: GET 2024-04-16T19:56:26.802 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.803 DEBUG:boto:Data: 2024-04-16T19:56:26.803 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.803 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.803 DEBUG:boto:Params: {} 2024-04-16T19:56:26.803 DEBUG:boto:Token: None 2024-04-16T19:56:26.803 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ= 2024-04-16T19:56:26.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f0db3b0dd156d7-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.805 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.806 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.806 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.806 DEBUG:boto:Data: 2024-04-16T19:56:26.806 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.806 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.806 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.806 DEBUG:boto:Params: {} 2024-04-16T19:56:26.806 DEBUG:boto:Token: None 2024-04-16T19:56:26.807 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:67G603bRwHkbGUjTM5qDFVNs9Ng= 2024-04-16T19:56:26.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:67G603bRwHkbGUjTM5qDFVNs9Ng=', 'Content-Length': '0'} 2024-04-16T19:56:26.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004600323b55c40473-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.809 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.809 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.809 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.809 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.809 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.809 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.809 DEBUG:boto:Data: 2024-04-16T19:56:26.809 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.809 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.809 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.809 DEBUG:boto:Params: {} 2024-04-16T19:56:26.810 DEBUG:boto:Token: None 2024-04-16T19:56:26.810 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.810 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:67G603bRwHkbGUjTM5qDFVNs9Ng= 2024-04-16T19:56:26.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:67G603bRwHkbGUjTM5qDFVNs9Ng=', 'Content-Length': '0'} 2024-04-16T19:56:26.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8020b9cbcc6b91-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-16T19:56:26.812 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.812 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.812 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.812 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.812 DEBUG:boto:Method: GET 2024-04-16T19:56:26.812 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.812 DEBUG:boto:Data: 2024-04-16T19:56:26.813 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.813 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.813 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.813 DEBUG:boto:Params: {} 2024-04-16T19:56:26.813 DEBUG:boto:Token: None 2024-04-16T19:56:26.813 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.813 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ= 2024-04-16T19:56:26.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bac92012ebe759-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.815 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.815 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.816 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.816 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.816 DEBUG:boto:Method: GET 2024-04-16T19:56:26.816 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.816 DEBUG:boto:Data: 2024-04-16T19:56:26.816 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.816 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.816 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.816 DEBUG:boto:Params: {} 2024-04-16T19:56:26.816 DEBUG:boto:Token: None 2024-04-16T19:56:26.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025?versionId=null 2024-04-16T19:56:26.817 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ= 2024-04-16T19:56:26.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PwU4FnnUC4+9L797JT9nLYBPibQ=', 'Content-Length': '0'} 2024-04-16T19:56:26.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045763e62161b7c2c-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.819 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.819 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.819 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.819 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.819 DEBUG:boto:Method: GET 2024-04-16T19:56:26.819 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.819 DEBUG:boto:Data: 2024-04-16T19:56:26.819 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.820 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.820 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.820 DEBUG:boto:Params: {} 2024-04-16T19:56:26.820 DEBUG:boto:Token: None 2024-04-16T19:56:26.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004418273d8b111cb8-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.823 DEBUG:boto:b'yrjefh-51obj000000000000000000251000falseobj000000000000000000252024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.823 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.823 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.823 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.823 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.823 DEBUG:boto:Method: GET 2024-04-16T19:56:26.823 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.823 DEBUG:boto:Data: 2024-04-16T19:56:26.824 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.824 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.824 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.824 DEBUG:boto:Params: {} 2024-04-16T19:56:26.824 DEBUG:boto:Token: None 2024-04-16T19:56:26.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7bd94767fcb192-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.870 DEBUG:boto:b'yrjefh-51obj000000000000000000251000falseobj000000000000000000252024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.871 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.871 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.871 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.871 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.871 DEBUG:boto:Data: 2024-04-16T19:56:26.871 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.871 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.871 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.871 DEBUG:boto:Params: {} 2024-04-16T19:56:26.871 DEBUG:boto:Token: None 2024-04-16T19:56:26.872 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R48P8qLe6IskUK0a/4rEQyyQhCY= 2024-04-16T19:56:26.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R48P8qLe6IskUK0a/4rEQyyQhCY=', 'Content-Length': '0'} 2024-04-16T19:56:26.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48ae87ca522a60c-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.874 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.874 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.874 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.874 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.874 DEBUG:boto:Data: 2024-04-16T19:56:26.874 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.874 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.874 DEBUG:boto:Params: {} 2024-04-16T19:56:26.875 DEBUG:boto:Token: None 2024-04-16T19:56:26.875 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R48P8qLe6IskUK0a/4rEQyyQhCY= 2024-04-16T19:56:26.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R48P8qLe6IskUK0a/4rEQyyQhCY=', 'Content-Length': '0'} 2024-04-16T19:56:26.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b4eea4346b3b37-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-16T19:56:26.877 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.877 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.877 DEBUG:boto:Method: GET 2024-04-16T19:56:26.877 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.877 DEBUG:boto:Data: 2024-04-16T19:56:26.878 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.878 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.878 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.878 DEBUG:boto:Params: {} 2024-04-16T19:56:26.878 DEBUG:boto:Token: None 2024-04-16T19:56:26.878 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.878 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aY+XzmtA2uziayhm3bUlOCtrlFc= 2024-04-16T19:56:26.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aY+XzmtA2uziayhm3bUlOCtrlFc=', 'Content-Length': '0'} 2024-04-16T19:56:26.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078167c3538110b63-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.880 DEBUG:boto:path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.880 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.881 DEBUG:boto:Method: GET 2024-04-16T19:56:26.881 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.881 DEBUG:boto:Data: 2024-04-16T19:56:26.881 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.881 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.881 DEBUG:boto:Params: {} 2024-04-16T19:56:26.881 DEBUG:boto:Token: None 2024-04-16T19:56:26.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000025 2024-04-16T19:56:26.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aY+XzmtA2uziayhm3bUlOCtrlFc= 2024-04-16T19:56:26.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aY+XzmtA2uziayhm3bUlOCtrlFc=', 'Content-Length': '0'} 2024-04-16T19:56:26.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6e677face12b42-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.884 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.884 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.884 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.884 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.884 DEBUG:boto:Method: GET 2024-04-16T19:56:26.884 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.884 DEBUG:boto:Data: 2024-04-16T19:56:26.884 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.884 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.884 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.884 DEBUG:boto:Params: {} 2024-04-16T19:56:26.885 DEBUG:boto:Token: None 2024-04-16T19:56:26.885 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.885 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf368c45af8f2f8f-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.887 DEBUG:boto:b'yrjefh-51obj000000000000000000251000falseobj000000000000000000252024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.888 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.888 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.888 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.888 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.888 DEBUG:boto:Method: GET 2024-04-16T19:56:26.888 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000025 2024-04-16T19:56:26.888 DEBUG:boto:Data: 2024-04-16T19:56:26.888 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.888 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.888 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.888 DEBUG:boto:Params: {} 2024-04-16T19:56:26.888 DEBUG:boto:Token: None 2024-04-16T19:56:26.889 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:26.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:26.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:26.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f018949bfb2eed8-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.892 DEBUG:boto:b'yrjefh-51obj000000000000000000251000falseobj000000000000000000252024-04-16T19:56:06.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:26.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-16T19:56:26.893 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.893 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.893 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.893 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.893 DEBUG:boto:Method: GET 2024-04-16T19:56:26.893 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.893 DEBUG:boto:Data: 2024-04-16T19:56:26.893 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.894 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.894 DEBUG:boto:Params: {} 2024-04-16T19:56:26.894 DEBUG:boto:Token: None 2024-04-16T19:56:26.894 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE= 2024-04-16T19:56:26.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE=', 'Content-Length': '0'} 2024-04-16T19:56:26.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e8140d5f5b6fdf-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.938 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.939 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.939 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.939 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.939 DEBUG:boto:Method: GET 2024-04-16T19:56:26.939 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.939 DEBUG:boto:Data: 2024-04-16T19:56:26.939 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.939 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.939 DEBUG:boto:Params: {} 2024-04-16T19:56:26.939 DEBUG:boto:Token: None 2024-04-16T19:56:26.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE= 2024-04-16T19:56:26.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE=', 'Content-Length': '0'} 2024-04-16T19:56:26.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195f01d87368adc1-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.942 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.942 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.943 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.943 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.943 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.943 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.943 DEBUG:boto:Data: 2024-04-16T19:56:26.943 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.943 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.943 DEBUG:boto:Params: {} 2024-04-16T19:56:26.943 DEBUG:boto:Token: None 2024-04-16T19:56:26.943 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uohBzcqwz/Qey2zNegTlWXvBQ0= 2024-04-16T19:56:26.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uohBzcqwz/Qey2zNegTlWXvBQ0=', 'Content-Length': '0'} 2024-04-16T19:56:26.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001057fbb93f3f0289-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.946 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.946 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.946 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.946 DEBUG:boto:Method: HEAD 2024-04-16T19:56:26.946 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.946 DEBUG:boto:Data: 2024-04-16T19:56:26.947 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.947 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.947 DEBUG:boto:Params: {} 2024-04-16T19:56:26.947 DEBUG:boto:Token: None 2024-04-16T19:56:26.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uohBzcqwz/Qey2zNegTlWXvBQ0= 2024-04-16T19:56:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uohBzcqwz/Qey2zNegTlWXvBQ0=', 'Content-Length': '0'} 2024-04-16T19:56:26.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cc939ec8b020a4-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-16T19:56:26.950 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.950 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.950 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.950 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.950 DEBUG:boto:Method: GET 2024-04-16T19:56:26.950 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.950 DEBUG:boto:Data: 2024-04-16T19:56:26.950 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.950 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.951 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.951 DEBUG:boto:Params: {} 2024-04-16T19:56:26.951 DEBUG:boto:Token: None 2024-04-16T19:56:26.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE= 2024-04-16T19:56:26.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE=', 'Content-Length': '0'} 2024-04-16T19:56:26.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc97e7583f602cc-00661ed7ea-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.953 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.953 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:26.954 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.954 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.954 DEBUG:boto:Method: GET 2024-04-16T19:56:26.954 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.954 DEBUG:boto:Data: 2024-04-16T19:56:26.954 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.954 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:26.954 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.954 DEBUG:boto:Params: {} 2024-04-16T19:56:26.954 DEBUG:boto:Token: None 2024-04-16T19:56:26.954 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/obj00000000000000000026?versionId=null 2024-04-16T19:56:26.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE= 2024-04-16T19:56:26.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m4qOkj5I8YRbXYDR5pfa3qT/NZE=', 'Content-Length': '0'} 2024-04-16T19:56:26.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab75b756ccdaff90-00661ed7ea-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:26.998 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:26.998 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:26.999 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:26.999 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:26.999 DEBUG:boto:Method: GET 2024-04-16T19:56:26.999 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:26.999 DEBUG:boto:Data: 2024-04-16T19:56:26.999 DEBUG:boto:Headers: {} 2024-04-16T19:56:26.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:26.999 DEBUG:boto:Port: 8000 2024-04-16T19:56:26.999 DEBUG:boto:Params: {} 2024-04-16T19:56:26.999 DEBUG:boto:Token: None 2024-04-16T19:56:27.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:26 GMT /yrjefh-51/ 2024-04-16T19:56:27.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM= 2024-04-16T19:56:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gEYEmNQGGBD1lw3PA6mHl2p0MbM=', 'Content-Length': '0'} 2024-04-16T19:56:27.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca2bff0ef462a76-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.002 DEBUG:boto:b'yrjefh-51obj000000000000000000261000falseobj000000000000000000262024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.003 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.003 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.003 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.003 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.003 DEBUG:boto:Method: GET 2024-04-16T19:56:27.003 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.003 DEBUG:boto:Data: 2024-04-16T19:56:27.003 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.004 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.004 DEBUG:boto:Params: {} 2024-04-16T19:56:27.004 DEBUG:boto:Token: None 2024-04-16T19:56:27.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eab815600c4c2be-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.008 DEBUG:boto:b'yrjefh-51obj000000000000000000261000falseobj000000000000000000262024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.008 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.008 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.009 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.009 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.009 DEBUG:boto:Data: 2024-04-16T19:56:27.009 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.009 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.009 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.009 DEBUG:boto:Params: {} 2024-04-16T19:56:27.009 DEBUG:boto:Token: None 2024-04-16T19:56:27.009 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.009 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNTV5wO0nn4wZ8oyinc9WdKb73Y= 2024-04-16T19:56:27.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNTV5wO0nn4wZ8oyinc9WdKb73Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da80d1771492c23-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.011 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.011 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.012 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.012 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.012 DEBUG:boto:Data: 2024-04-16T19:56:27.012 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.012 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.012 DEBUG:boto:Params: {} 2024-04-16T19:56:27.012 DEBUG:boto:Token: None 2024-04-16T19:56:27.012 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNTV5wO0nn4wZ8oyinc9WdKb73Y= 2024-04-16T19:56:27.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNTV5wO0nn4wZ8oyinc9WdKb73Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8051e00afd8adc5-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-16T19:56:27.015 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.015 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.015 DEBUG:boto:Method: GET 2024-04-16T19:56:27.015 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.015 DEBUG:boto:Data: 2024-04-16T19:56:27.015 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.016 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.016 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.016 DEBUG:boto:Params: {} 2024-04-16T19:56:27.016 DEBUG:boto:Token: None 2024-04-16T19:56:27.016 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oG8s6+pQzcfBpTsTLxrrop5r6VM= 2024-04-16T19:56:27.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oG8s6+pQzcfBpTsTLxrrop5r6VM=', 'Content-Length': '0'} 2024-04-16T19:56:27.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dc1186cc978e5c-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.062 DEBUG:boto:path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.062 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.063 DEBUG:boto:Method: GET 2024-04-16T19:56:27.063 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.063 DEBUG:boto:Data: 2024-04-16T19:56:27.063 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.063 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.063 DEBUG:boto:Params: {} 2024-04-16T19:56:27.063 DEBUG:boto:Token: None 2024-04-16T19:56:27.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000026 2024-04-16T19:56:27.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oG8s6+pQzcfBpTsTLxrrop5r6VM= 2024-04-16T19:56:27.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oG8s6+pQzcfBpTsTLxrrop5r6VM=', 'Content-Length': '0'} 2024-04-16T19:56:27.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe0ab3e6e934062-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.066 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.066 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.066 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.066 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.066 DEBUG:boto:Method: GET 2024-04-16T19:56:27.066 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.066 DEBUG:boto:Data: 2024-04-16T19:56:27.066 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.067 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.067 DEBUG:boto:Params: {} 2024-04-16T19:56:27.067 DEBUG:boto:Token: None 2024-04-16T19:56:27.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.067 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b072242b699f6ce-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.070 DEBUG:boto:b'yrjefh-51obj000000000000000000261000falseobj000000000000000000262024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.070 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.070 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.070 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.070 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.071 DEBUG:boto:Method: GET 2024-04-16T19:56:27.071 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000026 2024-04-16T19:56:27.071 DEBUG:boto:Data: 2024-04-16T19:56:27.071 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.071 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.071 DEBUG:boto:Params: {} 2024-04-16T19:56:27.071 DEBUG:boto:Token: None 2024-04-16T19:56:27.071 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a4b9c080ab4f7ff-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.076 DEBUG:boto:b'yrjefh-51obj000000000000000000261000falseobj000000000000000000262024-04-16T19:56:09.506Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-16T19:56:27.076 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.076 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.076 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.076 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.076 DEBUG:boto:Method: GET 2024-04-16T19:56:27.076 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.077 DEBUG:boto:Data: 2024-04-16T19:56:27.077 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.077 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.077 DEBUG:boto:Params: {} 2024-04-16T19:56:27.077 DEBUG:boto:Token: None 2024-04-16T19:56:27.077 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.077 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo= 2024-04-16T19:56:27.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo=', 'Content-Length': '0'} 2024-04-16T19:56:27.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff4efb8137c2b0b-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.080 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.080 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.080 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.080 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.081 DEBUG:boto:Method: GET 2024-04-16T19:56:27.081 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.081 DEBUG:boto:Data: 2024-04-16T19:56:27.081 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.081 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.081 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.081 DEBUG:boto:Params: {} 2024-04-16T19:56:27.081 DEBUG:boto:Token: None 2024-04-16T19:56:27.081 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.081 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo= 2024-04-16T19:56:27.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo=', 'Content-Length': '0'} 2024-04-16T19:56:27.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d102987abedff5e-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.126 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.127 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.127 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.127 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.127 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.127 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.127 DEBUG:boto:Data: 2024-04-16T19:56:27.127 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.127 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.127 DEBUG:boto:Params: {} 2024-04-16T19:56:27.127 DEBUG:boto:Token: None 2024-04-16T19:56:27.128 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0PDpByHo/mK7oFYR3wBRxEOuC8= 2024-04-16T19:56:27.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0PDpByHo/mK7oFYR3wBRxEOuC8=', 'Content-Length': '0'} 2024-04-16T19:56:27.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056cf8a54cbc8db36-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.130 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.130 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.130 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.130 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.130 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.131 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.131 DEBUG:boto:Data: 2024-04-16T19:56:27.131 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.131 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.131 DEBUG:boto:Params: {} 2024-04-16T19:56:27.131 DEBUG:boto:Token: None 2024-04-16T19:56:27.131 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0PDpByHo/mK7oFYR3wBRxEOuC8= 2024-04-16T19:56:27.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0PDpByHo/mK7oFYR3wBRxEOuC8=', 'Content-Length': '0'} 2024-04-16T19:56:27.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f34f8af60937bf4-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-16T19:56:27.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.134 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.134 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.134 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.134 DEBUG:boto:Method: GET 2024-04-16T19:56:27.134 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.134 DEBUG:boto:Data: 2024-04-16T19:56:27.134 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.134 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.134 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.135 DEBUG:boto:Params: {} 2024-04-16T19:56:27.135 DEBUG:boto:Token: None 2024-04-16T19:56:27.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo= 2024-04-16T19:56:27.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo=', 'Content-Length': '0'} 2024-04-16T19:56:27.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbdcd4454e78c112-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.137 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.137 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.137 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.138 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.138 DEBUG:boto:Method: GET 2024-04-16T19:56:27.138 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.138 DEBUG:boto:Data: 2024-04-16T19:56:27.138 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.138 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.138 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.138 DEBUG:boto:Params: {} 2024-04-16T19:56:27.138 DEBUG:boto:Token: None 2024-04-16T19:56:27.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027?versionId=null 2024-04-16T19:56:27.138 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo= 2024-04-16T19:56:27.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vD6f7ffHd4q4iIDJM0TnctVteSo=', 'Content-Length': '0'} 2024-04-16T19:56:27.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003224780627f67a76-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.141 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.141 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.141 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.141 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.141 DEBUG:boto:Method: GET 2024-04-16T19:56:27.141 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.142 DEBUG:boto:Data: 2024-04-16T19:56:27.142 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.142 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.142 DEBUG:boto:Params: {} 2024-04-16T19:56:27.142 DEBUG:boto:Token: None 2024-04-16T19:56:27.142 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab846c0a579357a8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.145 DEBUG:boto:b'yrjefh-51obj000000000000000000271000falseobj000000000000000000272024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.145 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.145 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.146 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.146 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.146 DEBUG:boto:Method: GET 2024-04-16T19:56:27.146 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.146 DEBUG:boto:Data: 2024-04-16T19:56:27.146 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.146 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.146 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.146 DEBUG:boto:Params: {} 2024-04-16T19:56:27.146 DEBUG:boto:Token: None 2024-04-16T19:56:27.146 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9282b7053fdb11-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.151 DEBUG:boto:b'yrjefh-51obj000000000000000000271000falseobj000000000000000000272024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.151 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.151 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.151 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.151 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.151 DEBUG:boto:Data: 2024-04-16T19:56:27.151 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.152 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.152 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.152 DEBUG:boto:Params: {} 2024-04-16T19:56:27.152 DEBUG:boto:Token: None 2024-04-16T19:56:27.152 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:722zY/+Aow7hbfX4uqDNYTtkGwg= 2024-04-16T19:56:27.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:722zY/+Aow7hbfX4uqDNYTtkGwg=', 'Content-Length': '0'} 2024-04-16T19:56:27.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb84d5fe8deadac8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.154 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.155 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.155 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.155 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.155 DEBUG:boto:Data: 2024-04-16T19:56:27.155 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.155 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.155 DEBUG:boto:Params: {} 2024-04-16T19:56:27.155 DEBUG:boto:Token: None 2024-04-16T19:56:27.155 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:722zY/+Aow7hbfX4uqDNYTtkGwg= 2024-04-16T19:56:27.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:722zY/+Aow7hbfX4uqDNYTtkGwg=', 'Content-Length': '0'} 2024-04-16T19:56:27.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a63d5a1972d9cf94-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-16T19:56:27.158 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.158 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.158 DEBUG:boto:Method: GET 2024-04-16T19:56:27.158 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.158 DEBUG:boto:Data: 2024-04-16T19:56:27.158 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.158 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.158 DEBUG:boto:Params: {} 2024-04-16T19:56:27.159 DEBUG:boto:Token: None 2024-04-16T19:56:27.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nY31YIqT5IbL3MqTK3HccC8HT7E= 2024-04-16T19:56:27.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nY31YIqT5IbL3MqTK3HccC8HT7E=', 'Content-Length': '0'} 2024-04-16T19:56:27.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d125829626ade860-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.161 DEBUG:boto:path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.162 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.162 DEBUG:boto:Method: GET 2024-04-16T19:56:27.162 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.162 DEBUG:boto:Data: 2024-04-16T19:56:27.162 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.162 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.162 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.162 DEBUG:boto:Params: {} 2024-04-16T19:56:27.162 DEBUG:boto:Token: None 2024-04-16T19:56:27.162 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000027 2024-04-16T19:56:27.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nY31YIqT5IbL3MqTK3HccC8HT7E= 2024-04-16T19:56:27.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nY31YIqT5IbL3MqTK3HccC8HT7E=', 'Content-Length': '0'} 2024-04-16T19:56:27.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c40cecc5495f96-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.165 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.165 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.165 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.165 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.165 DEBUG:boto:Method: GET 2024-04-16T19:56:27.165 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.165 DEBUG:boto:Data: 2024-04-16T19:56:27.165 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.166 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.166 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.166 DEBUG:boto:Params: {} 2024-04-16T19:56:27.166 DEBUG:boto:Token: None 2024-04-16T19:56:27.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012543a03e609259b-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.169 DEBUG:boto:b'yrjefh-51obj000000000000000000271000falseobj000000000000000000272024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.169 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.169 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.169 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.170 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.170 DEBUG:boto:Method: GET 2024-04-16T19:56:27.170 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000027 2024-04-16T19:56:27.170 DEBUG:boto:Data: 2024-04-16T19:56:27.170 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.170 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.170 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.170 DEBUG:boto:Params: {} 2024-04-16T19:56:27.170 DEBUG:boto:Token: None 2024-04-16T19:56:27.170 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb2af34514fd6eb-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.175 DEBUG:boto:b'yrjefh-51obj000000000000000000271000falseobj000000000000000000272024-04-16T19:56:09.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-16T19:56:27.175 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.175 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.175 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.175 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.175 DEBUG:boto:Method: GET 2024-04-16T19:56:27.175 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.176 DEBUG:boto:Data: 2024-04-16T19:56:27.176 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.176 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.176 DEBUG:boto:Params: {} 2024-04-16T19:56:27.176 DEBUG:boto:Token: None 2024-04-16T19:56:27.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o= 2024-04-16T19:56:27.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o=', 'Content-Length': '0'} 2024-04-16T19:56:27.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074304b1f4c689507-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.179 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.179 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.179 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.179 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.179 DEBUG:boto:Method: GET 2024-04-16T19:56:27.179 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.179 DEBUG:boto:Data: 2024-04-16T19:56:27.179 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.179 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.180 DEBUG:boto:Params: {} 2024-04-16T19:56:27.180 DEBUG:boto:Token: None 2024-04-16T19:56:27.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o= 2024-04-16T19:56:27.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o=', 'Content-Length': '0'} 2024-04-16T19:56:27.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000bfb39e50540816-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.226 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.227 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.227 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.227 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.227 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.227 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.227 DEBUG:boto:Data: 2024-04-16T19:56:27.227 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.227 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.227 DEBUG:boto:Params: {} 2024-04-16T19:56:27.227 DEBUG:boto:Token: None 2024-04-16T19:56:27.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oL8Q00IEXXPksujiKlTnvWsvB/o= 2024-04-16T19:56:27.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oL8Q00IEXXPksujiKlTnvWsvB/o=', 'Content-Length': '0'} 2024-04-16T19:56:27.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff63c449604c31d8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.230 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.230 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.230 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.230 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.230 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.230 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.230 DEBUG:boto:Data: 2024-04-16T19:56:27.230 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.231 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.231 DEBUG:boto:Params: {} 2024-04-16T19:56:27.231 DEBUG:boto:Token: None 2024-04-16T19:56:27.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oL8Q00IEXXPksujiKlTnvWsvB/o= 2024-04-16T19:56:27.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oL8Q00IEXXPksujiKlTnvWsvB/o=', 'Content-Length': '0'} 2024-04-16T19:56:27.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3f15aeb2579fde-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-16T19:56:27.233 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.233 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.233 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.233 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.234 DEBUG:boto:Method: GET 2024-04-16T19:56:27.234 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.234 DEBUG:boto:Data: 2024-04-16T19:56:27.234 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.234 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.234 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.234 DEBUG:boto:Params: {} 2024-04-16T19:56:27.234 DEBUG:boto:Token: None 2024-04-16T19:56:27.234 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.234 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o= 2024-04-16T19:56:27.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o=', 'Content-Length': '0'} 2024-04-16T19:56:27.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001520a5b9d03e6d16-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.237 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.237 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.237 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.237 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.237 DEBUG:boto:Method: GET 2024-04-16T19:56:27.237 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.237 DEBUG:boto:Data: 2024-04-16T19:56:27.237 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.237 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.237 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.238 DEBUG:boto:Params: {} 2024-04-16T19:56:27.238 DEBUG:boto:Token: None 2024-04-16T19:56:27.238 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028?versionId=null 2024-04-16T19:56:27.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o= 2024-04-16T19:56:27.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iN9RHrzqqsZ7F79QtMNXcFHHl5o=', 'Content-Length': '0'} 2024-04-16T19:56:27.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af41692aa1bbe6bd-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.240 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.240 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.240 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.240 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.241 DEBUG:boto:Method: GET 2024-04-16T19:56:27.241 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.241 DEBUG:boto:Data: 2024-04-16T19:56:27.241 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.241 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.241 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.241 DEBUG:boto:Params: {} 2024-04-16T19:56:27.241 DEBUG:boto:Token: None 2024-04-16T19:56:27.241 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012408f41cff88373-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.244 DEBUG:boto:b'yrjefh-51obj000000000000000000281000falseobj000000000000000000282024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.244 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.245 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.245 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.245 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.245 DEBUG:boto:Method: GET 2024-04-16T19:56:27.245 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.245 DEBUG:boto:Data: 2024-04-16T19:56:27.245 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.245 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.245 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.245 DEBUG:boto:Params: {} 2024-04-16T19:56:27.245 DEBUG:boto:Token: None 2024-04-16T19:56:27.245 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a230674726d9e313-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.249 DEBUG:boto:b'yrjefh-51obj000000000000000000281000falseobj000000000000000000282024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.250 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.250 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.250 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.250 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.250 DEBUG:boto:Data: 2024-04-16T19:56:27.250 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.250 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.250 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.250 DEBUG:boto:Params: {} 2024-04-16T19:56:27.250 DEBUG:boto:Token: None 2024-04-16T19:56:27.251 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yg6IWn58bG8qbos8hWqgCKHs8go= 2024-04-16T19:56:27.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yg6IWn58bG8qbos8hWqgCKHs8go=', 'Content-Length': '0'} 2024-04-16T19:56:27.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e331250dd231f0a-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.253 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.253 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.253 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.253 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.253 DEBUG:boto:Data: 2024-04-16T19:56:27.253 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.253 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.253 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.254 DEBUG:boto:Params: {} 2024-04-16T19:56:27.254 DEBUG:boto:Token: None 2024-04-16T19:56:27.254 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.254 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yg6IWn58bG8qbos8hWqgCKHs8go= 2024-04-16T19:56:27.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yg6IWn58bG8qbos8hWqgCKHs8go=', 'Content-Length': '0'} 2024-04-16T19:56:27.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c18b855657ac056-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-16T19:56:27.256 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.256 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.256 DEBUG:boto:Method: GET 2024-04-16T19:56:27.256 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.256 DEBUG:boto:Data: 2024-04-16T19:56:27.256 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.257 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.257 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.257 DEBUG:boto:Params: {} 2024-04-16T19:56:27.257 DEBUG:boto:Token: None 2024-04-16T19:56:27.257 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cY+8JhJohOS1IgSiJdtMUZQKbKk= 2024-04-16T19:56:27.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cY+8JhJohOS1IgSiJdtMUZQKbKk=', 'Content-Length': '0'} 2024-04-16T19:56:27.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003417eb52cc5637b9-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.302 DEBUG:boto:path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.303 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.303 DEBUG:boto:Method: GET 2024-04-16T19:56:27.303 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.303 DEBUG:boto:Data: 2024-04-16T19:56:27.303 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.303 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.303 DEBUG:boto:Params: {} 2024-04-16T19:56:27.303 DEBUG:boto:Token: None 2024-04-16T19:56:27.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000028 2024-04-16T19:56:27.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cY+8JhJohOS1IgSiJdtMUZQKbKk= 2024-04-16T19:56:27.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cY+8JhJohOS1IgSiJdtMUZQKbKk=', 'Content-Length': '0'} 2024-04-16T19:56:27.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09a8bd7023e2313-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.306 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.306 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.306 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.306 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.306 DEBUG:boto:Method: GET 2024-04-16T19:56:27.306 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.306 DEBUG:boto:Data: 2024-04-16T19:56:27.306 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.306 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.306 DEBUG:boto:Params: {} 2024-04-16T19:56:27.307 DEBUG:boto:Token: None 2024-04-16T19:56:27.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004536dfe796717d02-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.310 DEBUG:boto:b'yrjefh-51obj000000000000000000281000falseobj000000000000000000282024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.310 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.310 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.310 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.310 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.310 DEBUG:boto:Method: GET 2024-04-16T19:56:27.310 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000028 2024-04-16T19:56:27.310 DEBUG:boto:Data: 2024-04-16T19:56:27.310 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.311 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.311 DEBUG:boto:Params: {} 2024-04-16T19:56:27.311 DEBUG:boto:Token: None 2024-04-16T19:56:27.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016cd1c6c8cd615e6-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.315 DEBUG:boto:b'yrjefh-51obj000000000000000000281000falseobj000000000000000000282024-04-16T19:56:04.542Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-16T19:56:27.315 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.316 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.316 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.316 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.316 DEBUG:boto:Method: GET 2024-04-16T19:56:27.316 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.316 DEBUG:boto:Data: 2024-04-16T19:56:27.316 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.316 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.316 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.316 DEBUG:boto:Params: {} 2024-04-16T19:56:27.316 DEBUG:boto:Token: None 2024-04-16T19:56:27.317 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y= 2024-04-16T19:56:27.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f7d4b9ff8175a8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.320 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.320 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.320 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.320 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.320 DEBUG:boto:Method: GET 2024-04-16T19:56:27.320 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.320 DEBUG:boto:Data: 2024-04-16T19:56:27.320 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.320 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.320 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.321 DEBUG:boto:Params: {} 2024-04-16T19:56:27.321 DEBUG:boto:Token: None 2024-04-16T19:56:27.321 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y= 2024-04-16T19:56:27.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cffea3e96d9bbac-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.323 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.323 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.324 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.324 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.324 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.324 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.324 DEBUG:boto:Data: 2024-04-16T19:56:27.324 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.324 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.324 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.324 DEBUG:boto:Params: {} 2024-04-16T19:56:27.324 DEBUG:boto:Token: None 2024-04-16T19:56:27.324 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.325 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BRvWecOz1Em8JJDnl3B+NFVT8Wk= 2024-04-16T19:56:27.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BRvWecOz1Em8JJDnl3B+NFVT8Wk=', 'Content-Length': '0'} 2024-04-16T19:56:27.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3db8c51f647df17-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.327 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.327 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.327 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.327 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.327 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.327 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.328 DEBUG:boto:Data: 2024-04-16T19:56:27.328 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.328 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.328 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.328 DEBUG:boto:Params: {} 2024-04-16T19:56:27.328 DEBUG:boto:Token: None 2024-04-16T19:56:27.328 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BRvWecOz1Em8JJDnl3B+NFVT8Wk= 2024-04-16T19:56:27.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BRvWecOz1Em8JJDnl3B+NFVT8Wk=', 'Content-Length': '0'} 2024-04-16T19:56:27.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd25caad03cc637-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-16T19:56:27.331 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.331 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.331 DEBUG:boto:Method: GET 2024-04-16T19:56:27.331 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.331 DEBUG:boto:Data: 2024-04-16T19:56:27.331 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.331 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.332 DEBUG:boto:Params: {} 2024-04-16T19:56:27.332 DEBUG:boto:Token: None 2024-04-16T19:56:27.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y= 2024-04-16T19:56:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f918e0ab3c2c85b8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.374 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.374 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.375 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.375 DEBUG:boto:Method: GET 2024-04-16T19:56:27.375 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.375 DEBUG:boto:Data: 2024-04-16T19:56:27.375 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.375 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.375 DEBUG:boto:Params: {} 2024-04-16T19:56:27.375 DEBUG:boto:Token: None 2024-04-16T19:56:27.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029?versionId=null 2024-04-16T19:56:27.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y= 2024-04-16T19:56:27.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NrAebDpNGgJn/UfhiXlXdUVvT3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94a333287a975f0-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.378 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.378 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.378 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.378 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.379 DEBUG:boto:Method: GET 2024-04-16T19:56:27.379 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.379 DEBUG:boto:Data: 2024-04-16T19:56:27.379 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.379 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.379 DEBUG:boto:Params: {} 2024-04-16T19:56:27.379 DEBUG:boto:Token: None 2024-04-16T19:56:27.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cffbb6f5ed6ee2f-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.382 DEBUG:boto:b'yrjefh-51obj000000000000000000291000falseobj000000000000000000292024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.382 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.382 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.382 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.383 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.383 DEBUG:boto:Method: GET 2024-04-16T19:56:27.383 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.383 DEBUG:boto:Data: 2024-04-16T19:56:27.383 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.383 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.383 DEBUG:boto:Params: {} 2024-04-16T19:56:27.383 DEBUG:boto:Token: None 2024-04-16T19:56:27.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e59063333e8341b-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.387 DEBUG:boto:b'yrjefh-51obj000000000000000000291000falseobj000000000000000000292024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.387 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.388 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.388 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.388 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.388 DEBUG:boto:Data: 2024-04-16T19:56:27.388 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.388 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.388 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.388 DEBUG:boto:Params: {} 2024-04-16T19:56:27.388 DEBUG:boto:Token: None 2024-04-16T19:56:27.388 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HjZYUdjmzrnJR005A2YZrC8YGT0= 2024-04-16T19:56:27.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HjZYUdjmzrnJR005A2YZrC8YGT0=', 'Content-Length': '0'} 2024-04-16T19:56:27.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa30338c5c5287df-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.391 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.391 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.391 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.391 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.391 DEBUG:boto:Data: 2024-04-16T19:56:27.392 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.392 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.392 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.392 DEBUG:boto:Params: {} 2024-04-16T19:56:27.392 DEBUG:boto:Token: None 2024-04-16T19:56:27.392 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HjZYUdjmzrnJR005A2YZrC8YGT0= 2024-04-16T19:56:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HjZYUdjmzrnJR005A2YZrC8YGT0=', 'Content-Length': '0'} 2024-04-16T19:56:27.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7aff9b3c5a865b3-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-16T19:56:27.394 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.395 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.395 DEBUG:boto:Method: GET 2024-04-16T19:56:27.395 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.395 DEBUG:boto:Data: 2024-04-16T19:56:27.395 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.395 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.395 DEBUG:boto:Params: {} 2024-04-16T19:56:27.395 DEBUG:boto:Token: None 2024-04-16T19:56:27.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f0Yn9+eytpD0TpTAPpsE6Qvqz6w= 2024-04-16T19:56:27.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f0Yn9+eytpD0TpTAPpsE6Qvqz6w=', 'Content-Length': '0'} 2024-04-16T19:56:27.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005268fd6e8c0a2891-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.398 DEBUG:boto:path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.398 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.398 DEBUG:boto:Method: GET 2024-04-16T19:56:27.398 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.398 DEBUG:boto:Data: 2024-04-16T19:56:27.398 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.398 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.398 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.398 DEBUG:boto:Params: {} 2024-04-16T19:56:27.399 DEBUG:boto:Token: None 2024-04-16T19:56:27.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000029 2024-04-16T19:56:27.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f0Yn9+eytpD0TpTAPpsE6Qvqz6w= 2024-04-16T19:56:27.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f0Yn9+eytpD0TpTAPpsE6Qvqz6w=', 'Content-Length': '0'} 2024-04-16T19:56:27.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000863b9ecfcf826bb5-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.401 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.401 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.401 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.401 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.401 DEBUG:boto:Method: GET 2024-04-16T19:56:27.402 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.402 DEBUG:boto:Data: 2024-04-16T19:56:27.402 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.402 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.402 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.402 DEBUG:boto:Params: {} 2024-04-16T19:56:27.402 DEBUG:boto:Token: None 2024-04-16T19:56:27.402 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.402 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce5efdbac34f2b94-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.405 DEBUG:boto:b'yrjefh-51obj000000000000000000291000falseobj000000000000000000292024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.405 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.406 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.406 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.406 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.406 DEBUG:boto:Method: GET 2024-04-16T19:56:27.406 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000029 2024-04-16T19:56:27.406 DEBUG:boto:Data: 2024-04-16T19:56:27.406 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.406 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.406 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.406 DEBUG:boto:Params: {} 2024-04-16T19:56:27.406 DEBUG:boto:Token: None 2024-04-16T19:56:27.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0e54cd058bd7ec-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.411 DEBUG:boto:b'yrjefh-51obj000000000000000000291000falseobj000000000000000000292024-04-16T19:56:07.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-16T19:56:27.411 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.411 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.411 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.412 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.412 DEBUG:boto:Method: GET 2024-04-16T19:56:27.412 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.412 DEBUG:boto:Data: 2024-04-16T19:56:27.412 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.412 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.412 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.412 DEBUG:boto:Params: {} 2024-04-16T19:56:27.412 DEBUG:boto:Token: None 2024-04-16T19:56:27.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s= 2024-04-16T19:56:27.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s=', 'Content-Length': '0'} 2024-04-16T19:56:27.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a00c3710855e68-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.415 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.415 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.415 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.415 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.415 DEBUG:boto:Method: GET 2024-04-16T19:56:27.415 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.415 DEBUG:boto:Data: 2024-04-16T19:56:27.415 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.416 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.416 DEBUG:boto:Params: {} 2024-04-16T19:56:27.416 DEBUG:boto:Token: None 2024-04-16T19:56:27.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s= 2024-04-16T19:56:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s=', 'Content-Length': '0'} 2024-04-16T19:56:27.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8838c0826b19e69-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.418 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.418 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.419 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.419 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.419 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.419 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.419 DEBUG:boto:Data: 2024-04-16T19:56:27.419 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.419 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.419 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.419 DEBUG:boto:Params: {} 2024-04-16T19:56:27.419 DEBUG:boto:Token: None 2024-04-16T19:56:27.419 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXe56n04ZtwmoexIhxbfBMT5vkQ= 2024-04-16T19:56:27.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXe56n04ZtwmoexIhxbfBMT5vkQ=', 'Content-Length': '0'} 2024-04-16T19:56:27.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf9b658fe7ff7b9-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.422 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.422 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.422 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.422 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.422 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.422 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.422 DEBUG:boto:Data: 2024-04-16T19:56:27.422 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.422 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.423 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.423 DEBUG:boto:Params: {} 2024-04-16T19:56:27.423 DEBUG:boto:Token: None 2024-04-16T19:56:27.423 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXe56n04ZtwmoexIhxbfBMT5vkQ= 2024-04-16T19:56:27.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXe56n04ZtwmoexIhxbfBMT5vkQ=', 'Content-Length': '0'} 2024-04-16T19:56:27.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f577e7a8d0383fc6-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-16T19:56:27.425 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.425 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.425 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.425 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.425 DEBUG:boto:Method: GET 2024-04-16T19:56:27.425 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.426 DEBUG:boto:Data: 2024-04-16T19:56:27.426 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.426 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.426 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.426 DEBUG:boto:Params: {} 2024-04-16T19:56:27.426 DEBUG:boto:Token: None 2024-04-16T19:56:27.426 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s= 2024-04-16T19:56:27.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s=', 'Content-Length': '0'} 2024-04-16T19:56:27.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4a2eafc1565317-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.470 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.471 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.471 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.471 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.471 DEBUG:boto:Method: GET 2024-04-16T19:56:27.471 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.471 DEBUG:boto:Data: 2024-04-16T19:56:27.471 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.471 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.471 DEBUG:boto:Params: {} 2024-04-16T19:56:27.471 DEBUG:boto:Token: None 2024-04-16T19:56:27.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030?versionId=null 2024-04-16T19:56:27.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s= 2024-04-16T19:56:27.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQiuFW4efoMYxMlWPZcBA4Ri72s=', 'Content-Length': '0'} 2024-04-16T19:56:27.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002820fd397b1e4b94-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.474 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.474 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.474 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.474 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.474 DEBUG:boto:Method: GET 2024-04-16T19:56:27.474 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.474 DEBUG:boto:Data: 2024-04-16T19:56:27.475 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.475 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.475 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.475 DEBUG:boto:Params: {} 2024-04-16T19:56:27.475 DEBUG:boto:Token: None 2024-04-16T19:56:27.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000064f84bc83eaa60-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.478 DEBUG:boto:b'yrjefh-51obj000000000000000000301000falseobj000000000000000000302024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.478 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.478 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.478 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.479 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.479 DEBUG:boto:Method: GET 2024-04-16T19:56:27.479 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.479 DEBUG:boto:Data: 2024-04-16T19:56:27.479 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.479 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.479 DEBUG:boto:Params: {} 2024-04-16T19:56:27.479 DEBUG:boto:Token: None 2024-04-16T19:56:27.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c9f81e1e5929056-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.484 DEBUG:boto:b'yrjefh-51obj000000000000000000301000falseobj000000000000000000302024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.484 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.484 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.484 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.484 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.484 DEBUG:boto:Data: 2024-04-16T19:56:27.485 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.485 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.485 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.485 DEBUG:boto:Params: {} 2024-04-16T19:56:27.485 DEBUG:boto:Token: None 2024-04-16T19:56:27.485 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yARa5m2sd9MgaCq12o/a6F6XBY= 2024-04-16T19:56:27.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yARa5m2sd9MgaCq12o/a6F6XBY=', 'Content-Length': '0'} 2024-04-16T19:56:27.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d13038822849be-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.487 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.487 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.487 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.487 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.487 DEBUG:boto:Data: 2024-04-16T19:56:27.488 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.488 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.488 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.488 DEBUG:boto:Params: {} 2024-04-16T19:56:27.488 DEBUG:boto:Token: None 2024-04-16T19:56:27.488 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/yARa5m2sd9MgaCq12o/a6F6XBY= 2024-04-16T19:56:27.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yARa5m2sd9MgaCq12o/a6F6XBY=', 'Content-Length': '0'} 2024-04-16T19:56:27.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2ee829fa607100-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-16T19:56:27.490 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.490 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.490 DEBUG:boto:Method: GET 2024-04-16T19:56:27.491 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.491 DEBUG:boto:Data: 2024-04-16T19:56:27.491 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.491 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.491 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.491 DEBUG:boto:Params: {} 2024-04-16T19:56:27.491 DEBUG:boto:Token: None 2024-04-16T19:56:27.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:08YIYjbht/03RN7EmqZXwMxKB/c= 2024-04-16T19:56:27.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:08YIYjbht/03RN7EmqZXwMxKB/c=', 'Content-Length': '0'} 2024-04-16T19:56:27.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a794659ce0e72dd1-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.493 DEBUG:boto:path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.493 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.494 DEBUG:boto:Method: GET 2024-04-16T19:56:27.494 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.494 DEBUG:boto:Data: 2024-04-16T19:56:27.494 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.494 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.494 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.494 DEBUG:boto:Params: {} 2024-04-16T19:56:27.494 DEBUG:boto:Token: None 2024-04-16T19:56:27.494 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000030 2024-04-16T19:56:27.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:08YIYjbht/03RN7EmqZXwMxKB/c= 2024-04-16T19:56:27.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:08YIYjbht/03RN7EmqZXwMxKB/c=', 'Content-Length': '0'} 2024-04-16T19:56:27.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008056894de148a815-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.497 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.497 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.497 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.497 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.497 DEBUG:boto:Method: GET 2024-04-16T19:56:27.497 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.498 DEBUG:boto:Data: 2024-04-16T19:56:27.498 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.498 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.498 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.498 DEBUG:boto:Params: {} 2024-04-16T19:56:27.498 DEBUG:boto:Token: None 2024-04-16T19:56:27.498 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.498 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8e1e0e9a78d9e0-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.501 DEBUG:boto:b'yrjefh-51obj000000000000000000301000falseobj000000000000000000302024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.501 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.501 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.501 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.501 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.501 DEBUG:boto:Method: GET 2024-04-16T19:56:27.502 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000030 2024-04-16T19:56:27.502 DEBUG:boto:Data: 2024-04-16T19:56:27.502 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.502 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.502 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.502 DEBUG:boto:Params: {} 2024-04-16T19:56:27.502 DEBUG:boto:Token: None 2024-04-16T19:56:27.502 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.502 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d66c00573f6a17c-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.506 DEBUG:boto:b'yrjefh-51obj000000000000000000301000falseobj000000000000000000302024-04-16T19:56:08.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-16T19:56:27.507 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.507 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.507 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.507 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.507 DEBUG:boto:Method: GET 2024-04-16T19:56:27.507 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.507 DEBUG:boto:Data: 2024-04-16T19:56:27.507 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.508 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.508 DEBUG:boto:Params: {} 2024-04-16T19:56:27.508 DEBUG:boto:Token: None 2024-04-16T19:56:27.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc= 2024-04-16T19:56:27.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc=', 'Content-Length': '0'} 2024-04-16T19:56:27.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec4d1483c001ddc-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.510 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.510 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.511 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.511 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.511 DEBUG:boto:Method: GET 2024-04-16T19:56:27.511 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.511 DEBUG:boto:Data: 2024-04-16T19:56:27.511 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.511 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.511 DEBUG:boto:Params: {} 2024-04-16T19:56:27.511 DEBUG:boto:Token: None 2024-04-16T19:56:27.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc= 2024-04-16T19:56:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc=', 'Content-Length': '0'} 2024-04-16T19:56:27.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b42c368ab4ba560-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.514 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.514 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.514 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.514 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.514 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.514 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.514 DEBUG:boto:Data: 2024-04-16T19:56:27.514 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.514 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.514 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.514 DEBUG:boto:Params: {} 2024-04-16T19:56:27.515 DEBUG:boto:Token: None 2024-04-16T19:56:27.515 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IcuuS6BHVJwj0aeLnPVNaxS5Ov8= 2024-04-16T19:56:27.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IcuuS6BHVJwj0aeLnPVNaxS5Ov8=', 'Content-Length': '0'} 2024-04-16T19:56:27.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a52d930a120c53-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.517 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.517 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.517 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.517 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.517 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.518 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.518 DEBUG:boto:Data: 2024-04-16T19:56:27.518 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.518 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.518 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.518 DEBUG:boto:Params: {} 2024-04-16T19:56:27.518 DEBUG:boto:Token: None 2024-04-16T19:56:27.518 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.518 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IcuuS6BHVJwj0aeLnPVNaxS5Ov8= 2024-04-16T19:56:27.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IcuuS6BHVJwj0aeLnPVNaxS5Ov8=', 'Content-Length': '0'} 2024-04-16T19:56:27.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001660135ca1ae46d6-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-16T19:56:27.520 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.520 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.521 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.521 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.521 DEBUG:boto:Method: GET 2024-04-16T19:56:27.521 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.521 DEBUG:boto:Data: 2024-04-16T19:56:27.521 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.521 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.521 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.521 DEBUG:boto:Params: {} 2024-04-16T19:56:27.521 DEBUG:boto:Token: None 2024-04-16T19:56:27.521 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.522 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc= 2024-04-16T19:56:27.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc=', 'Content-Length': '0'} 2024-04-16T19:56:27.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7b45ba6653f89e-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.524 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.524 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.524 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.524 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.524 DEBUG:boto:Method: GET 2024-04-16T19:56:27.524 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.524 DEBUG:boto:Data: 2024-04-16T19:56:27.525 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.525 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.525 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.525 DEBUG:boto:Params: {} 2024-04-16T19:56:27.525 DEBUG:boto:Token: None 2024-04-16T19:56:27.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031?versionId=null 2024-04-16T19:56:27.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc= 2024-04-16T19:56:27.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xeVO4WbYDmkRwcoQoQS119IKnNc=', 'Content-Length': '0'} 2024-04-16T19:56:27.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b9c666223cf839-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.528 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.528 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.528 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.528 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.528 DEBUG:boto:Method: GET 2024-04-16T19:56:27.528 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.528 DEBUG:boto:Data: 2024-04-16T19:56:27.528 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.528 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.528 DEBUG:boto:Params: {} 2024-04-16T19:56:27.528 DEBUG:boto:Token: None 2024-04-16T19:56:27.529 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1a0521662766b2-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.532 DEBUG:boto:b'yrjefh-51obj000000000000000000311000falseobj000000000000000000312024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.532 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.532 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.532 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.532 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.532 DEBUG:boto:Method: GET 2024-04-16T19:56:27.532 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.532 DEBUG:boto:Data: 2024-04-16T19:56:27.533 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.533 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.533 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.533 DEBUG:boto:Params: {} 2024-04-16T19:56:27.533 DEBUG:boto:Token: None 2024-04-16T19:56:27.533 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627e9b1141f4332f-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.578 DEBUG:boto:b'yrjefh-51obj000000000000000000311000falseobj000000000000000000312024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.579 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.579 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.579 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.579 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.579 DEBUG:boto:Data: 2024-04-16T19:56:27.579 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.579 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.579 DEBUG:boto:Params: {} 2024-04-16T19:56:27.579 DEBUG:boto:Token: None 2024-04-16T19:56:27.579 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lYgdNLcaGtqYLoa3Wd3h7eHFMxI= 2024-04-16T19:56:27.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lYgdNLcaGtqYLoa3Wd3h7eHFMxI=', 'Content-Length': '0'} 2024-04-16T19:56:27.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e04e3104db5eee-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.582 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.582 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.582 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.582 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.582 DEBUG:boto:Data: 2024-04-16T19:56:27.582 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.582 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.583 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.583 DEBUG:boto:Params: {} 2024-04-16T19:56:27.583 DEBUG:boto:Token: None 2024-04-16T19:56:27.583 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lYgdNLcaGtqYLoa3Wd3h7eHFMxI= 2024-04-16T19:56:27.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lYgdNLcaGtqYLoa3Wd3h7eHFMxI=', 'Content-Length': '0'} 2024-04-16T19:56:27.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69e53b6ce9ea700-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-16T19:56:27.585 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.585 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.586 DEBUG:boto:Method: GET 2024-04-16T19:56:27.586 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.586 DEBUG:boto:Data: 2024-04-16T19:56:27.586 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.586 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.586 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.586 DEBUG:boto:Params: {} 2024-04-16T19:56:27.586 DEBUG:boto:Token: None 2024-04-16T19:56:27.586 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yhagNpSO5NiTykuqlDsF51N1ZaE= 2024-04-16T19:56:27.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yhagNpSO5NiTykuqlDsF51N1ZaE=', 'Content-Length': '0'} 2024-04-16T19:56:27.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017013f3a49b78fa8-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.589 DEBUG:boto:path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.589 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.589 DEBUG:boto:Method: GET 2024-04-16T19:56:27.589 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.589 DEBUG:boto:Data: 2024-04-16T19:56:27.589 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.589 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.589 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.589 DEBUG:boto:Params: {} 2024-04-16T19:56:27.589 DEBUG:boto:Token: None 2024-04-16T19:56:27.589 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000031 2024-04-16T19:56:27.590 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yhagNpSO5NiTykuqlDsF51N1ZaE= 2024-04-16T19:56:27.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yhagNpSO5NiTykuqlDsF51N1ZaE=', 'Content-Length': '0'} 2024-04-16T19:56:27.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000735b7c33a901fff4-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.592 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.592 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.592 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.592 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.592 DEBUG:boto:Method: GET 2024-04-16T19:56:27.592 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.592 DEBUG:boto:Data: 2024-04-16T19:56:27.593 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.593 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.593 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.593 DEBUG:boto:Params: {} 2024-04-16T19:56:27.593 DEBUG:boto:Token: None 2024-04-16T19:56:27.593 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.593 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41d9e4f5895897c-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.596 DEBUG:boto:b'yrjefh-51obj000000000000000000311000falseobj000000000000000000312024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.596 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.596 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.596 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.597 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.597 DEBUG:boto:Method: GET 2024-04-16T19:56:27.597 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000031 2024-04-16T19:56:27.597 DEBUG:boto:Data: 2024-04-16T19:56:27.597 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.597 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.597 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.597 DEBUG:boto:Params: {} 2024-04-16T19:56:27.597 DEBUG:boto:Token: None 2024-04-16T19:56:27.597 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000865c7ea82b9138c-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.601 DEBUG:boto:b'yrjefh-51obj000000000000000000311000falseobj000000000000000000312024-04-16T19:56:07.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-16T19:56:27.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.602 DEBUG:boto:Method: GET 2024-04-16T19:56:27.602 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.602 DEBUG:boto:Data: 2024-04-16T19:56:27.602 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.602 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.603 DEBUG:boto:Params: {} 2024-04-16T19:56:27.603 DEBUG:boto:Token: None 2024-04-16T19:56:27.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y= 2024-04-16T19:56:27.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b02b964b3db69f-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.646 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.647 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.647 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.647 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.647 DEBUG:boto:Method: GET 2024-04-16T19:56:27.647 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.647 DEBUG:boto:Data: 2024-04-16T19:56:27.647 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.647 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.647 DEBUG:boto:Params: {} 2024-04-16T19:56:27.647 DEBUG:boto:Token: None 2024-04-16T19:56:27.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y= 2024-04-16T19:56:27.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e69373c81cbbaf-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.650 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.650 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.650 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.650 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.651 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.651 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.651 DEBUG:boto:Data: 2024-04-16T19:56:27.651 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.651 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.651 DEBUG:boto:Params: {} 2024-04-16T19:56:27.651 DEBUG:boto:Token: None 2024-04-16T19:56:27.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hhIp/m+dWPJ3i81LwKpMDjo0Dmc= 2024-04-16T19:56:27.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hhIp/m+dWPJ3i81LwKpMDjo0Dmc=', 'Content-Length': '0'} 2024-04-16T19:56:27.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d075af667e3cfc-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.653 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.654 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.654 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.654 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.654 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.654 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.654 DEBUG:boto:Data: 2024-04-16T19:56:27.654 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.654 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.654 DEBUG:boto:Params: {} 2024-04-16T19:56:27.654 DEBUG:boto:Token: None 2024-04-16T19:56:27.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hhIp/m+dWPJ3i81LwKpMDjo0Dmc= 2024-04-16T19:56:27.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hhIp/m+dWPJ3i81LwKpMDjo0Dmc=', 'Content-Length': '0'} 2024-04-16T19:56:27.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9ab8e9a559a318-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-16T19:56:27.657 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.657 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.657 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.657 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.657 DEBUG:boto:Method: GET 2024-04-16T19:56:27.657 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.658 DEBUG:boto:Data: 2024-04-16T19:56:27.658 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.658 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.658 DEBUG:boto:Params: {} 2024-04-16T19:56:27.658 DEBUG:boto:Token: None 2024-04-16T19:56:27.658 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.658 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y= 2024-04-16T19:56:27.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac48655c58ce6025-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.660 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.660 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.661 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.661 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.661 DEBUG:boto:Method: GET 2024-04-16T19:56:27.661 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.661 DEBUG:boto:Data: 2024-04-16T19:56:27.661 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.661 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.661 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.661 DEBUG:boto:Params: {} 2024-04-16T19:56:27.661 DEBUG:boto:Token: None 2024-04-16T19:56:27.661 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032?versionId=null 2024-04-16T19:56:27.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y= 2024-04-16T19:56:27.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M/jGvZlove8MEVGXQZ7WNYBIb3Y=', 'Content-Length': '0'} 2024-04-16T19:56:27.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de36f5ffdbc8906c-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.664 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.664 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.664 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.664 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.664 DEBUG:boto:Method: GET 2024-04-16T19:56:27.664 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.664 DEBUG:boto:Data: 2024-04-16T19:56:27.664 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.664 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.664 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.665 DEBUG:boto:Params: {} 2024-04-16T19:56:27.665 DEBUG:boto:Token: None 2024-04-16T19:56:27.665 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950d4afc633946eb-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.668 DEBUG:boto:b'yrjefh-51obj000000000000000000321000falseobj000000000000000000322024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.668 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.668 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.669 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.669 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.669 DEBUG:boto:Method: GET 2024-04-16T19:56:27.669 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.669 DEBUG:boto:Data: 2024-04-16T19:56:27.669 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.669 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.669 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.669 DEBUG:boto:Params: {} 2024-04-16T19:56:27.669 DEBUG:boto:Token: None 2024-04-16T19:56:27.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b5ce1bfd0609a58-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.673 DEBUG:boto:b'yrjefh-51obj000000000000000000321000falseobj000000000000000000322024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.673 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.674 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.674 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.674 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.674 DEBUG:boto:Data: 2024-04-16T19:56:27.674 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.674 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.674 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.674 DEBUG:boto:Params: {} 2024-04-16T19:56:27.674 DEBUG:boto:Token: None 2024-04-16T19:56:27.674 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QKYQ3KZSz1cLs/z6XmrqJxSO5Ow= 2024-04-16T19:56:27.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QKYQ3KZSz1cLs/z6XmrqJxSO5Ow=', 'Content-Length': '0'} 2024-04-16T19:56:27.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177a3ffcd8154fdd-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.677 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.677 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.677 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.677 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.677 DEBUG:boto:Data: 2024-04-16T19:56:27.677 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.677 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.677 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.677 DEBUG:boto:Params: {} 2024-04-16T19:56:27.677 DEBUG:boto:Token: None 2024-04-16T19:56:27.678 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.678 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QKYQ3KZSz1cLs/z6XmrqJxSO5Ow= 2024-04-16T19:56:27.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QKYQ3KZSz1cLs/z6XmrqJxSO5Ow=', 'Content-Length': '0'} 2024-04-16T19:56:27.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c08b4f92f3a104-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-16T19:56:27.680 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.680 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.680 DEBUG:boto:Method: GET 2024-04-16T19:56:27.680 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.680 DEBUG:boto:Data: 2024-04-16T19:56:27.680 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.680 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.680 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.680 DEBUG:boto:Params: {} 2024-04-16T19:56:27.681 DEBUG:boto:Token: None 2024-04-16T19:56:27.681 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cIdp0RyT2nRPkg71ujE9Hk62RKQ= 2024-04-16T19:56:27.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cIdp0RyT2nRPkg71ujE9Hk62RKQ=', 'Content-Length': '0'} 2024-04-16T19:56:27.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe732323f8476b8a-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.683 DEBUG:boto:path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.683 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.683 DEBUG:boto:Method: GET 2024-04-16T19:56:27.683 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.683 DEBUG:boto:Data: 2024-04-16T19:56:27.683 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.684 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.684 DEBUG:boto:Params: {} 2024-04-16T19:56:27.684 DEBUG:boto:Token: None 2024-04-16T19:56:27.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000032 2024-04-16T19:56:27.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cIdp0RyT2nRPkg71ujE9Hk62RKQ= 2024-04-16T19:56:27.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cIdp0RyT2nRPkg71ujE9Hk62RKQ=', 'Content-Length': '0'} 2024-04-16T19:56:27.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b849041622357c39-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.726 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.726 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.727 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.727 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.727 DEBUG:boto:Method: GET 2024-04-16T19:56:27.727 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.727 DEBUG:boto:Data: 2024-04-16T19:56:27.727 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.727 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.727 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.727 DEBUG:boto:Params: {} 2024-04-16T19:56:27.727 DEBUG:boto:Token: None 2024-04-16T19:56:27.727 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aceae1fe2d901fc-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.730 DEBUG:boto:b'yrjefh-51obj000000000000000000321000falseobj000000000000000000322024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.731 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.731 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.731 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.731 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.731 DEBUG:boto:Method: GET 2024-04-16T19:56:27.731 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000032 2024-04-16T19:56:27.731 DEBUG:boto:Data: 2024-04-16T19:56:27.731 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.732 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.732 DEBUG:boto:Params: {} 2024-04-16T19:56:27.732 DEBUG:boto:Token: None 2024-04-16T19:56:27.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b2b0d5e42816d4-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.736 DEBUG:boto:b'yrjefh-51obj000000000000000000321000falseobj000000000000000000322024-04-16T19:56:05.928Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-16T19:56:27.737 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.737 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.737 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.737 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.737 DEBUG:boto:Method: GET 2024-04-16T19:56:27.737 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.737 DEBUG:boto:Data: 2024-04-16T19:56:27.737 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.737 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.738 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.738 DEBUG:boto:Params: {} 2024-04-16T19:56:27.738 DEBUG:boto:Token: None 2024-04-16T19:56:27.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.738 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A= 2024-04-16T19:56:27.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A=', 'Content-Length': '0'} 2024-04-16T19:56:27.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58fea61208c2a76-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.782 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.783 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.783 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.783 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.783 DEBUG:boto:Method: GET 2024-04-16T19:56:27.783 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.783 DEBUG:boto:Data: 2024-04-16T19:56:27.783 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.783 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.783 DEBUG:boto:Params: {} 2024-04-16T19:56:27.783 DEBUG:boto:Token: None 2024-04-16T19:56:27.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A= 2024-04-16T19:56:27.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A=', 'Content-Length': '0'} 2024-04-16T19:56:27.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8681ffa767f4f6-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.786 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.786 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.786 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.786 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.786 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.786 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.787 DEBUG:boto:Data: 2024-04-16T19:56:27.787 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.787 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.787 DEBUG:boto:Params: {} 2024-04-16T19:56:27.787 DEBUG:boto:Token: None 2024-04-16T19:56:27.787 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m7xVP1XiKNinFQYd2pBDHtEECck= 2024-04-16T19:56:27.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m7xVP1XiKNinFQYd2pBDHtEECck=', 'Content-Length': '0'} 2024-04-16T19:56:27.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000528c8c4dc86dd3d1-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.789 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.789 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.790 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.790 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.790 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.790 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.790 DEBUG:boto:Data: 2024-04-16T19:56:27.790 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.790 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.790 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.790 DEBUG:boto:Params: {} 2024-04-16T19:56:27.790 DEBUG:boto:Token: None 2024-04-16T19:56:27.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m7xVP1XiKNinFQYd2pBDHtEECck= 2024-04-16T19:56:27.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m7xVP1XiKNinFQYd2pBDHtEECck=', 'Content-Length': '0'} 2024-04-16T19:56:27.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f68ad2f850e0cd-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-16T19:56:27.793 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.793 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.793 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.793 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.793 DEBUG:boto:Method: GET 2024-04-16T19:56:27.793 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.793 DEBUG:boto:Data: 2024-04-16T19:56:27.794 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.794 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.794 DEBUG:boto:Params: {} 2024-04-16T19:56:27.794 DEBUG:boto:Token: None 2024-04-16T19:56:27.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A= 2024-04-16T19:56:27.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A=', 'Content-Length': '0'} 2024-04-16T19:56:27.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b79a2068f39f4a-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.796 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.796 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.797 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.797 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.797 DEBUG:boto:Method: GET 2024-04-16T19:56:27.797 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.797 DEBUG:boto:Data: 2024-04-16T19:56:27.797 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.797 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.797 DEBUG:boto:Params: {} 2024-04-16T19:56:27.797 DEBUG:boto:Token: None 2024-04-16T19:56:27.797 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033?versionId=null 2024-04-16T19:56:27.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A= 2024-04-16T19:56:27.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hKyPamU4zxGnx3b0jjyNcnsKZ7A=', 'Content-Length': '0'} 2024-04-16T19:56:27.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b86badfc692b093-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.800 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.800 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.800 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.800 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.800 DEBUG:boto:Method: GET 2024-04-16T19:56:27.800 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.800 DEBUG:boto:Data: 2024-04-16T19:56:27.800 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.800 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.800 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.801 DEBUG:boto:Params: {} 2024-04-16T19:56:27.801 DEBUG:boto:Token: None 2024-04-16T19:56:27.801 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968a15d4ad18b487-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.804 DEBUG:boto:b'yrjefh-51obj000000000000000000331000falseobj000000000000000000332024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.804 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.804 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.804 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.804 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.804 DEBUG:boto:Method: GET 2024-04-16T19:56:27.804 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.804 DEBUG:boto:Data: 2024-04-16T19:56:27.805 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.805 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.805 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.805 DEBUG:boto:Params: {} 2024-04-16T19:56:27.805 DEBUG:boto:Token: None 2024-04-16T19:56:27.805 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab62785dee11a228-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.809 DEBUG:boto:b'yrjefh-51obj000000000000000000331000falseobj000000000000000000332024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.809 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.809 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.810 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.810 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.810 DEBUG:boto:Data: 2024-04-16T19:56:27.810 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.810 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.810 DEBUG:boto:Params: {} 2024-04-16T19:56:27.810 DEBUG:boto:Token: None 2024-04-16T19:56:27.810 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ja5GV6QF/VeqszA6AU9wAS+U2Mk= 2024-04-16T19:56:27.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ja5GV6QF/VeqszA6AU9wAS+U2Mk=', 'Content-Length': '0'} 2024-04-16T19:56:27.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed2335ef10caa976-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.813 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.813 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.813 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.813 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.813 DEBUG:boto:Data: 2024-04-16T19:56:27.813 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.814 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.814 DEBUG:boto:Params: {} 2024-04-16T19:56:27.814 DEBUG:boto:Token: None 2024-04-16T19:56:27.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ja5GV6QF/VeqszA6AU9wAS+U2Mk= 2024-04-16T19:56:27.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ja5GV6QF/VeqszA6AU9wAS+U2Mk=', 'Content-Length': '0'} 2024-04-16T19:56:27.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d03003dcbfe1aeb-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-16T19:56:27.816 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.816 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.816 DEBUG:boto:Method: GET 2024-04-16T19:56:27.816 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.816 DEBUG:boto:Data: 2024-04-16T19:56:27.817 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.817 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.817 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.817 DEBUG:boto:Params: {} 2024-04-16T19:56:27.817 DEBUG:boto:Token: None 2024-04-16T19:56:27.817 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.817 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HNDnMaomnZ2l9215cq5/x8w1N1I= 2024-04-16T19:56:27.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HNDnMaomnZ2l9215cq5/x8w1N1I=', 'Content-Length': '0'} 2024-04-16T19:56:27.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea7b0aa2be3c5a9-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.819 DEBUG:boto:path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.819 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.819 DEBUG:boto:Method: GET 2024-04-16T19:56:27.820 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.820 DEBUG:boto:Data: 2024-04-16T19:56:27.820 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.820 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.820 DEBUG:boto:Params: {} 2024-04-16T19:56:27.820 DEBUG:boto:Token: None 2024-04-16T19:56:27.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000033 2024-04-16T19:56:27.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HNDnMaomnZ2l9215cq5/x8w1N1I= 2024-04-16T19:56:27.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HNDnMaomnZ2l9215cq5/x8w1N1I=', 'Content-Length': '0'} 2024-04-16T19:56:27.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdde87ebc1fa6f3-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.823 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.823 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.823 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.823 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.823 DEBUG:boto:Method: GET 2024-04-16T19:56:27.823 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.823 DEBUG:boto:Data: 2024-04-16T19:56:27.823 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.823 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.823 DEBUG:boto:Params: {} 2024-04-16T19:56:27.824 DEBUG:boto:Token: None 2024-04-16T19:56:27.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8c6ee2a957aa81-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.870 DEBUG:boto:b'yrjefh-51obj000000000000000000331000falseobj000000000000000000332024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.871 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.871 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.871 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.871 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.871 DEBUG:boto:Method: GET 2024-04-16T19:56:27.871 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000033 2024-04-16T19:56:27.871 DEBUG:boto:Data: 2024-04-16T19:56:27.871 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.871 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.871 DEBUG:boto:Params: {} 2024-04-16T19:56:27.872 DEBUG:boto:Token: None 2024-04-16T19:56:27.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e8ea7a555089485-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.876 DEBUG:boto:b'yrjefh-51obj000000000000000000331000falseobj000000000000000000332024-04-16T19:56:09.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-16T19:56:27.876 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.876 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.877 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.877 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.877 DEBUG:boto:Method: GET 2024-04-16T19:56:27.877 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.877 DEBUG:boto:Data: 2024-04-16T19:56:27.877 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.877 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.877 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.877 DEBUG:boto:Params: {} 2024-04-16T19:56:27.877 DEBUG:boto:Token: None 2024-04-16T19:56:27.878 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.878 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc= 2024-04-16T19:56:27.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc=', 'Content-Length': '0'} 2024-04-16T19:56:27.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f6d21068d89660-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.881 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.881 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.881 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.881 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.881 DEBUG:boto:Method: GET 2024-04-16T19:56:27.881 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.881 DEBUG:boto:Data: 2024-04-16T19:56:27.881 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.881 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.881 DEBUG:boto:Params: {} 2024-04-16T19:56:27.882 DEBUG:boto:Token: None 2024-04-16T19:56:27.882 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.882 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc= 2024-04-16T19:56:27.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc=', 'Content-Length': '0'} 2024-04-16T19:56:27.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000603eb53be8e0874-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.926 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.927 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.927 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.927 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.927 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.927 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.927 DEBUG:boto:Data: 2024-04-16T19:56:27.927 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.927 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.927 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.927 DEBUG:boto:Params: {} 2024-04-16T19:56:27.927 DEBUG:boto:Token: None 2024-04-16T19:56:27.928 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KYD+2V5Ed/2zy6QceQiRByi561g= 2024-04-16T19:56:27.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KYD+2V5Ed/2zy6QceQiRByi561g=', 'Content-Length': '0'} 2024-04-16T19:56:27.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de87132171ef705-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.930 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.930 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.930 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.930 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.930 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.930 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.931 DEBUG:boto:Data: 2024-04-16T19:56:27.931 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.931 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.931 DEBUG:boto:Params: {} 2024-04-16T19:56:27.931 DEBUG:boto:Token: None 2024-04-16T19:56:27.931 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KYD+2V5Ed/2zy6QceQiRByi561g= 2024-04-16T19:56:27.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KYD+2V5Ed/2zy6QceQiRByi561g=', 'Content-Length': '0'} 2024-04-16T19:56:27.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c9414cd697587a4-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-16T19:56:27.934 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.934 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.934 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.935 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.935 DEBUG:boto:Method: GET 2024-04-16T19:56:27.935 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.935 DEBUG:boto:Data: 2024-04-16T19:56:27.935 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.935 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.935 DEBUG:boto:Params: {} 2024-04-16T19:56:27.935 DEBUG:boto:Token: None 2024-04-16T19:56:27.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc= 2024-04-16T19:56:27.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc=', 'Content-Length': '0'} 2024-04-16T19:56:27.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a6ec7fab495c21-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.938 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.938 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.938 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.938 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.938 DEBUG:boto:Method: GET 2024-04-16T19:56:27.938 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.939 DEBUG:boto:Data: 2024-04-16T19:56:27.939 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.939 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.939 DEBUG:boto:Params: {} 2024-04-16T19:56:27.939 DEBUG:boto:Token: None 2024-04-16T19:56:27.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036?versionId=null 2024-04-16T19:56:27.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc= 2024-04-16T19:56:27.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u1KFHGjOeqcyMiBEZfdGB1SScTc=', 'Content-Length': '0'} 2024-04-16T19:56:27.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd87cdb7e78b8c6-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.942 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.942 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.942 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.942 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.942 DEBUG:boto:Method: GET 2024-04-16T19:56:27.942 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.942 DEBUG:boto:Data: 2024-04-16T19:56:27.942 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.942 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.942 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.943 DEBUG:boto:Params: {} 2024-04-16T19:56:27.943 DEBUG:boto:Token: None 2024-04-16T19:56:27.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac41bf316fe0bfe-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.946 DEBUG:boto:b'yrjefh-51obj000000000000000000361000falseobj000000000000000000362024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.946 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:27.946 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:27.946 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.946 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.946 DEBUG:boto:Method: GET 2024-04-16T19:56:27.946 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:27.947 DEBUG:boto:Data: 2024-04-16T19:56:27.947 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.947 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.947 DEBUG:boto:Params: {} 2024-04-16T19:56:27.947 DEBUG:boto:Token: None 2024-04-16T19:56:27.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/ 2024-04-16T19:56:27.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0= 2024-04-16T19:56:27.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HpYyTdBEZxlP4KBd8qSJbB8mpP0=', 'Content-Length': '0'} 2024-04-16T19:56:27.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60c1920705e4087-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.951 DEBUG:boto:b'yrjefh-51obj000000000000000000361000falseobj000000000000000000362024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:27.952 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.952 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.952 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.952 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.952 DEBUG:boto:Data: 2024-04-16T19:56:27.952 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.952 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.952 DEBUG:boto:Params: {} 2024-04-16T19:56:27.952 DEBUG:boto:Token: None 2024-04-16T19:56:27.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DQWvfmjleaeXr+y1mn7nTmY4Zo8= 2024-04-16T19:56:27.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DQWvfmjleaeXr+y1mn7nTmY4Zo8=', 'Content-Length': '0'} 2024-04-16T19:56:27.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f950626c33b1480-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.955 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.955 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.956 DEBUG:boto:Method: HEAD 2024-04-16T19:56:27.956 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.956 DEBUG:boto:Data: 2024-04-16T19:56:27.956 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.956 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.956 DEBUG:boto:Params: {} 2024-04-16T19:56:27.956 DEBUG:boto:Token: None 2024-04-16T19:56:27.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DQWvfmjleaeXr+y1mn7nTmY4Zo8= 2024-04-16T19:56:27.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DQWvfmjleaeXr+y1mn7nTmY4Zo8=', 'Content-Length': '0'} 2024-04-16T19:56:27.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e540196011873309-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-16T19:56:27.959 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.959 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.959 DEBUG:boto:Method: GET 2024-04-16T19:56:27.960 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.960 DEBUG:boto:Data: 2024-04-16T19:56:27.960 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:27.960 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.960 DEBUG:boto:Params: {} 2024-04-16T19:56:27.960 DEBUG:boto:Token: None 2024-04-16T19:56:27.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+qEMk8wfG19NZvuHO1qBGkU9BA= 2024-04-16T19:56:27.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+qEMk8wfG19NZvuHO1qBGkU9BA=', 'Content-Length': '0'} 2024-04-16T19:56:27.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d67366630e4a58-00661ed7eb-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:27.963 DEBUG:boto:path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.963 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.963 DEBUG:boto:Method: GET 2024-04-16T19:56:27.963 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.963 DEBUG:boto:Data: 2024-04-16T19:56:27.963 DEBUG:boto:Headers: {} 2024-04-16T19:56:27.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:27.963 DEBUG:boto:Port: 8000 2024-04-16T19:56:27.964 DEBUG:boto:Params: {} 2024-04-16T19:56:27.964 DEBUG:boto:Token: None 2024-04-16T19:56:27.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:27 GMT /yrjefh-51/obj00000000000000000036 2024-04-16T19:56:27.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+qEMk8wfG19NZvuHO1qBGkU9BA= 2024-04-16T19:56:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+qEMk8wfG19NZvuHO1qBGkU9BA=', 'Content-Length': '0'} 2024-04-16T19:56:27.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d468eaef179c60-00661ed7eb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.006 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.007 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.007 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.007 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.007 DEBUG:boto:Method: GET 2024-04-16T19:56:28.007 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.007 DEBUG:boto:Data: 2024-04-16T19:56:28.007 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.007 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.007 DEBUG:boto:Params: {} 2024-04-16T19:56:28.007 DEBUG:boto:Token: None 2024-04-16T19:56:28.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6a8e96e9c84fa5-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.010 DEBUG:boto:b'yrjefh-51obj000000000000000000361000falseobj000000000000000000362024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.011 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.011 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.011 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.011 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.011 DEBUG:boto:Method: GET 2024-04-16T19:56:28.011 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000036 2024-04-16T19:56:28.011 DEBUG:boto:Data: 2024-04-16T19:56:28.011 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.011 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.012 DEBUG:boto:Params: {} 2024-04-16T19:56:28.012 DEBUG:boto:Token: None 2024-04-16T19:56:28.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801801914fbff5d8-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.016 DEBUG:boto:b'yrjefh-51obj000000000000000000361000falseobj000000000000000000362024-04-16T19:56:08.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-16T19:56:28.017 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.017 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.017 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.017 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.017 DEBUG:boto:Method: GET 2024-04-16T19:56:28.017 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.018 DEBUG:boto:Data: 2024-04-16T19:56:28.018 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.018 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.018 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.018 DEBUG:boto:Params: {} 2024-04-16T19:56:28.018 DEBUG:boto:Token: None 2024-04-16T19:56:28.018 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU= 2024-04-16T19:56:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU=', 'Content-Length': '0'} 2024-04-16T19:56:28.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001441dad20e030d21-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.062 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.062 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.063 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.063 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.063 DEBUG:boto:Method: GET 2024-04-16T19:56:28.063 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.063 DEBUG:boto:Data: 2024-04-16T19:56:28.063 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.063 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.063 DEBUG:boto:Params: {} 2024-04-16T19:56:28.063 DEBUG:boto:Token: None 2024-04-16T19:56:28.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU= 2024-04-16T19:56:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU=', 'Content-Length': '0'} 2024-04-16T19:56:28.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009688f16b1721efe1-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.066 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.066 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.066 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.066 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.067 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.067 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.067 DEBUG:boto:Data: 2024-04-16T19:56:28.067 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.067 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.067 DEBUG:boto:Params: {} 2024-04-16T19:56:28.067 DEBUG:boto:Token: None 2024-04-16T19:56:28.067 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/8uf0RzpFcZrz6h3ymhdpm3lnuM= 2024-04-16T19:56:28.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/8uf0RzpFcZrz6h3ymhdpm3lnuM=', 'Content-Length': '0'} 2024-04-16T19:56:28.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fdb5ca655d8337-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.070 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.070 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.070 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.070 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.071 DEBUG:boto:Data: 2024-04-16T19:56:28.071 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.071 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.071 DEBUG:boto:Params: {} 2024-04-16T19:56:28.071 DEBUG:boto:Token: None 2024-04-16T19:56:28.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/8uf0RzpFcZrz6h3ymhdpm3lnuM= 2024-04-16T19:56:28.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/8uf0RzpFcZrz6h3ymhdpm3lnuM=', 'Content-Length': '0'} 2024-04-16T19:56:28.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a554616c784165a3-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-16T19:56:28.074 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.074 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.074 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.075 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.075 DEBUG:boto:Method: GET 2024-04-16T19:56:28.075 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.075 DEBUG:boto:Data: 2024-04-16T19:56:28.075 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.075 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.075 DEBUG:boto:Params: {} 2024-04-16T19:56:28.075 DEBUG:boto:Token: None 2024-04-16T19:56:28.075 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU= 2024-04-16T19:56:28.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU=', 'Content-Length': '0'} 2024-04-16T19:56:28.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d6d957e3553d99-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.078 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.078 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.078 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.078 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.079 DEBUG:boto:Method: GET 2024-04-16T19:56:28.079 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.079 DEBUG:boto:Data: 2024-04-16T19:56:28.079 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.079 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.079 DEBUG:boto:Params: {} 2024-04-16T19:56:28.079 DEBUG:boto:Token: None 2024-04-16T19:56:28.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037?versionId=null 2024-04-16T19:56:28.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU= 2024-04-16T19:56:28.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaD5iIVzVGO/OKwrsXnsj1V6PRU=', 'Content-Length': '0'} 2024-04-16T19:56:28.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf10a2acde10f1e4-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.082 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.082 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.082 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.082 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.082 DEBUG:boto:Method: GET 2024-04-16T19:56:28.082 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.083 DEBUG:boto:Data: 2024-04-16T19:56:28.083 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.083 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.083 DEBUG:boto:Params: {} 2024-04-16T19:56:28.083 DEBUG:boto:Token: None 2024-04-16T19:56:28.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033635a89320cfe9b-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.086 DEBUG:boto:b'yrjefh-51obj000000000000000000371000falseobj000000000000000000372024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.087 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.087 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.087 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.087 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.087 DEBUG:boto:Method: GET 2024-04-16T19:56:28.087 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.087 DEBUG:boto:Data: 2024-04-16T19:56:28.087 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.087 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.087 DEBUG:boto:Params: {} 2024-04-16T19:56:28.087 DEBUG:boto:Token: None 2024-04-16T19:56:28.088 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d38da9b8a653e0-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.092 DEBUG:boto:b'yrjefh-51obj000000000000000000371000falseobj000000000000000000372024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.092 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.092 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.092 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.093 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.093 DEBUG:boto:Data: 2024-04-16T19:56:28.093 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.093 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.093 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.093 DEBUG:boto:Params: {} 2024-04-16T19:56:28.093 DEBUG:boto:Token: None 2024-04-16T19:56:28.093 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.093 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F5n6krp7EDKwtk67xFQ7WfTPYQg= 2024-04-16T19:56:28.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F5n6krp7EDKwtk67xFQ7WfTPYQg=', 'Content-Length': '0'} 2024-04-16T19:56:28.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651cbad16de8e504-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.096 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.096 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.096 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.096 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.096 DEBUG:boto:Data: 2024-04-16T19:56:28.096 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.096 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.096 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.097 DEBUG:boto:Params: {} 2024-04-16T19:56:28.097 DEBUG:boto:Token: None 2024-04-16T19:56:28.097 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.097 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F5n6krp7EDKwtk67xFQ7WfTPYQg= 2024-04-16T19:56:28.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F5n6krp7EDKwtk67xFQ7WfTPYQg=', 'Content-Length': '0'} 2024-04-16T19:56:28.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043dfffab3e95b4a5-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-16T19:56:28.099 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.099 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.100 DEBUG:boto:Method: GET 2024-04-16T19:56:28.100 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.100 DEBUG:boto:Data: 2024-04-16T19:56:28.100 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.100 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.100 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.100 DEBUG:boto:Params: {} 2024-04-16T19:56:28.100 DEBUG:boto:Token: None 2024-04-16T19:56:28.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.101 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9J+vePspPC499wgMDojxzs52I8Q= 2024-04-16T19:56:28.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9J+vePspPC499wgMDojxzs52I8Q=', 'Content-Length': '0'} 2024-04-16T19:56:28.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a72ad87dbf2d856f-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.146 DEBUG:boto:path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.146 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.147 DEBUG:boto:Method: GET 2024-04-16T19:56:28.147 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.147 DEBUG:boto:Data: 2024-04-16T19:56:28.147 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.147 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.147 DEBUG:boto:Params: {} 2024-04-16T19:56:28.147 DEBUG:boto:Token: None 2024-04-16T19:56:28.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000037 2024-04-16T19:56:28.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9J+vePspPC499wgMDojxzs52I8Q= 2024-04-16T19:56:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9J+vePspPC499wgMDojxzs52I8Q=', 'Content-Length': '0'} 2024-04-16T19:56:28.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714351c83050b83b-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.150 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.150 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.150 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.150 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.150 DEBUG:boto:Method: GET 2024-04-16T19:56:28.151 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.151 DEBUG:boto:Data: 2024-04-16T19:56:28.151 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.151 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.151 DEBUG:boto:Params: {} 2024-04-16T19:56:28.151 DEBUG:boto:Token: None 2024-04-16T19:56:28.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b541486accf0770a-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.154 DEBUG:boto:b'yrjefh-51obj000000000000000000371000falseobj000000000000000000372024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.155 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.155 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.155 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.155 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.155 DEBUG:boto:Method: GET 2024-04-16T19:56:28.155 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000037 2024-04-16T19:56:28.155 DEBUG:boto:Data: 2024-04-16T19:56:28.155 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.155 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.155 DEBUG:boto:Params: {} 2024-04-16T19:56:28.156 DEBUG:boto:Token: None 2024-04-16T19:56:28.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf61bc9ef5d17ce-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.160 DEBUG:boto:b'yrjefh-51obj000000000000000000371000falseobj000000000000000000372024-04-16T19:56:06.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-16T19:56:28.160 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.161 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.161 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.161 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.161 DEBUG:boto:Method: GET 2024-04-16T19:56:28.161 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.161 DEBUG:boto:Data: 2024-04-16T19:56:28.161 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.161 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.161 DEBUG:boto:Params: {} 2024-04-16T19:56:28.161 DEBUG:boto:Token: None 2024-04-16T19:56:28.162 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.162 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg= 2024-04-16T19:56:28.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg=', 'Content-Length': '0'} 2024-04-16T19:56:28.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002823ccaf6c17a5b3-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.164 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.165 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.165 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.165 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.165 DEBUG:boto:Method: GET 2024-04-16T19:56:28.165 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.165 DEBUG:boto:Data: 2024-04-16T19:56:28.165 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.165 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.165 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.165 DEBUG:boto:Params: {} 2024-04-16T19:56:28.166 DEBUG:boto:Token: None 2024-04-16T19:56:28.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg= 2024-04-16T19:56:28.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg=', 'Content-Length': '0'} 2024-04-16T19:56:28.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065207e7f5c213dae-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.210 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.210 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.210 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.211 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.211 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.211 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.211 DEBUG:boto:Data: 2024-04-16T19:56:28.211 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.211 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.211 DEBUG:boto:Params: {} 2024-04-16T19:56:28.211 DEBUG:boto:Token: None 2024-04-16T19:56:28.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j5nTkymKaqIsPQ+S8sU/d4h1otU= 2024-04-16T19:56:28.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j5nTkymKaqIsPQ+S8sU/d4h1otU=', 'Content-Length': '0'} 2024-04-16T19:56:28.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e3b27e462b68c3-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.214 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.214 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.214 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.214 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.214 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.215 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.215 DEBUG:boto:Data: 2024-04-16T19:56:28.215 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.215 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.215 DEBUG:boto:Params: {} 2024-04-16T19:56:28.215 DEBUG:boto:Token: None 2024-04-16T19:56:28.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j5nTkymKaqIsPQ+S8sU/d4h1otU= 2024-04-16T19:56:28.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j5nTkymKaqIsPQ+S8sU/d4h1otU=', 'Content-Length': '0'} 2024-04-16T19:56:28.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639b0843200c5ac7-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-16T19:56:28.218 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.218 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.218 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.219 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.219 DEBUG:boto:Method: GET 2024-04-16T19:56:28.219 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.219 DEBUG:boto:Data: 2024-04-16T19:56:28.219 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.219 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.219 DEBUG:boto:Params: {} 2024-04-16T19:56:28.219 DEBUG:boto:Token: None 2024-04-16T19:56:28.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg= 2024-04-16T19:56:28.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg=', 'Content-Length': '0'} 2024-04-16T19:56:28.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008646f710a0f3318b-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.222 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.222 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.222 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.222 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.222 DEBUG:boto:Method: GET 2024-04-16T19:56:28.222 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.223 DEBUG:boto:Data: 2024-04-16T19:56:28.223 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.223 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.223 DEBUG:boto:Params: {} 2024-04-16T19:56:28.223 DEBUG:boto:Token: None 2024-04-16T19:56:28.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039?versionId=null 2024-04-16T19:56:28.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg= 2024-04-16T19:56:28.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uA+o6NJ3foQC4R5roEoqY0JUZsg=', 'Content-Length': '0'} 2024-04-16T19:56:28.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f70f09861efd15-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.226 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.226 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.226 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.226 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.226 DEBUG:boto:Method: GET 2024-04-16T19:56:28.226 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.226 DEBUG:boto:Data: 2024-04-16T19:56:28.227 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.227 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.227 DEBUG:boto:Params: {} 2024-04-16T19:56:28.227 DEBUG:boto:Token: None 2024-04-16T19:56:28.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005b7730706e41c89-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.230 DEBUG:boto:b'yrjefh-51obj000000000000000000391000falseobj000000000000000000392024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.230 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.230 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.231 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.231 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.231 DEBUG:boto:Method: GET 2024-04-16T19:56:28.231 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.231 DEBUG:boto:Data: 2024-04-16T19:56:28.231 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.231 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.231 DEBUG:boto:Params: {} 2024-04-16T19:56:28.231 DEBUG:boto:Token: None 2024-04-16T19:56:28.232 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005836fba8a2ad9aef-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.236 DEBUG:boto:b'yrjefh-51obj000000000000000000391000falseobj000000000000000000392024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.236 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.236 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.236 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.236 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.236 DEBUG:boto:Data: 2024-04-16T19:56:28.236 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.237 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.237 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.237 DEBUG:boto:Params: {} 2024-04-16T19:56:28.237 DEBUG:boto:Token: None 2024-04-16T19:56:28.237 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j8eDioGzQ9c0YTpoVhqG653y4dM= 2024-04-16T19:56:28.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j8eDioGzQ9c0YTpoVhqG653y4dM=', 'Content-Length': '0'} 2024-04-16T19:56:28.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe195513ea1b28f-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.239 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.240 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.240 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.240 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.240 DEBUG:boto:Data: 2024-04-16T19:56:28.240 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.240 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.240 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.240 DEBUG:boto:Params: {} 2024-04-16T19:56:28.240 DEBUG:boto:Token: None 2024-04-16T19:56:28.241 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j8eDioGzQ9c0YTpoVhqG653y4dM= 2024-04-16T19:56:28.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j8eDioGzQ9c0YTpoVhqG653y4dM=', 'Content-Length': '0'} 2024-04-16T19:56:28.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d037b188b00742c-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-16T19:56:28.243 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.243 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.243 DEBUG:boto:Method: GET 2024-04-16T19:56:28.244 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.244 DEBUG:boto:Data: 2024-04-16T19:56:28.244 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.244 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.244 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.244 DEBUG:boto:Params: {} 2024-04-16T19:56:28.244 DEBUG:boto:Token: None 2024-04-16T19:56:28.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6VuL+600m3Mvw0iO2rtcJHRSVx4= 2024-04-16T19:56:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VuL+600m3Mvw0iO2rtcJHRSVx4=', 'Content-Length': '0'} 2024-04-16T19:56:28.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097546ad7f115d134-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.247 DEBUG:boto:path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.247 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.247 DEBUG:boto:Method: GET 2024-04-16T19:56:28.247 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.247 DEBUG:boto:Data: 2024-04-16T19:56:28.247 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.247 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.247 DEBUG:boto:Params: {} 2024-04-16T19:56:28.247 DEBUG:boto:Token: None 2024-04-16T19:56:28.247 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000039 2024-04-16T19:56:28.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6VuL+600m3Mvw0iO2rtcJHRSVx4= 2024-04-16T19:56:28.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VuL+600m3Mvw0iO2rtcJHRSVx4=', 'Content-Length': '0'} 2024-04-16T19:56:28.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bfecfc6dbfb327a-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.290 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.290 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.291 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.291 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.291 DEBUG:boto:Method: GET 2024-04-16T19:56:28.291 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.291 DEBUG:boto:Data: 2024-04-16T19:56:28.291 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.291 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.291 DEBUG:boto:Params: {} 2024-04-16T19:56:28.291 DEBUG:boto:Token: None 2024-04-16T19:56:28.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38f364a30315b5a-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.294 DEBUG:boto:b'yrjefh-51obj000000000000000000391000falseobj000000000000000000392024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.294 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.295 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.295 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.295 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.295 DEBUG:boto:Method: GET 2024-04-16T19:56:28.295 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000039 2024-04-16T19:56:28.295 DEBUG:boto:Data: 2024-04-16T19:56:28.295 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.295 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.295 DEBUG:boto:Params: {} 2024-04-16T19:56:28.295 DEBUG:boto:Token: None 2024-04-16T19:56:28.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080958ded32dd9cf1-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.300 DEBUG:boto:b'yrjefh-51obj000000000000000000391000falseobj000000000000000000392024-04-16T19:56:08.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-16T19:56:28.300 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.300 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.300 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.300 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.301 DEBUG:boto:Method: GET 2024-04-16T19:56:28.301 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.301 DEBUG:boto:Data: 2024-04-16T19:56:28.301 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.301 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.301 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.301 DEBUG:boto:Params: {} 2024-04-16T19:56:28.301 DEBUG:boto:Token: None 2024-04-16T19:56:28.301 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.301 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0= 2024-04-16T19:56:28.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0=', 'Content-Length': '0'} 2024-04-16T19:56:28.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009598861bcab97d15-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.304 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.304 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.304 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.304 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.304 DEBUG:boto:Method: GET 2024-04-16T19:56:28.304 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.304 DEBUG:boto:Data: 2024-04-16T19:56:28.305 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.305 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.305 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.305 DEBUG:boto:Params: {} 2024-04-16T19:56:28.305 DEBUG:boto:Token: None 2024-04-16T19:56:28.305 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0= 2024-04-16T19:56:28.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0=', 'Content-Length': '0'} 2024-04-16T19:56:28.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e13f9d4ca9e86a9-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.308 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.308 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.308 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.308 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.308 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.308 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.308 DEBUG:boto:Data: 2024-04-16T19:56:28.308 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.308 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.308 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.309 DEBUG:boto:Params: {} 2024-04-16T19:56:28.309 DEBUG:boto:Token: None 2024-04-16T19:56:28.309 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6bfWnNDgsHl5C6E94tcU9YyhaI= 2024-04-16T19:56:28.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6bfWnNDgsHl5C6E94tcU9YyhaI=', 'Content-Length': '0'} 2024-04-16T19:56:28.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915a0c04182fe5ca-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.311 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.311 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.311 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.311 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.311 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.312 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.312 DEBUG:boto:Data: 2024-04-16T19:56:28.312 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.312 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.312 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.312 DEBUG:boto:Params: {} 2024-04-16T19:56:28.312 DEBUG:boto:Token: None 2024-04-16T19:56:28.312 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6bfWnNDgsHl5C6E94tcU9YyhaI= 2024-04-16T19:56:28.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6bfWnNDgsHl5C6E94tcU9YyhaI=', 'Content-Length': '0'} 2024-04-16T19:56:28.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021cf9b4f027cf7d-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-16T19:56:28.315 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.315 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.315 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.315 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.315 DEBUG:boto:Method: GET 2024-04-16T19:56:28.315 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.315 DEBUG:boto:Data: 2024-04-16T19:56:28.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.315 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.316 DEBUG:boto:Params: {} 2024-04-16T19:56:28.316 DEBUG:boto:Token: None 2024-04-16T19:56:28.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0= 2024-04-16T19:56:28.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0=', 'Content-Length': '0'} 2024-04-16T19:56:28.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005a7aa0bee7b92eb-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.358 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.358 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.359 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.359 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.359 DEBUG:boto:Method: GET 2024-04-16T19:56:28.359 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.359 DEBUG:boto:Data: 2024-04-16T19:56:28.359 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.359 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.359 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.359 DEBUG:boto:Params: {} 2024-04-16T19:56:28.359 DEBUG:boto:Token: None 2024-04-16T19:56:28.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040?versionId=null 2024-04-16T19:56:28.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0= 2024-04-16T19:56:28.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1cPrVJLB4n5AfOg47Aw6cIrUfp0=', 'Content-Length': '0'} 2024-04-16T19:56:28.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa7321154f52854-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.362 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.362 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.362 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.362 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.362 DEBUG:boto:Method: GET 2024-04-16T19:56:28.362 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.362 DEBUG:boto:Data: 2024-04-16T19:56:28.363 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.363 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.363 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.363 DEBUG:boto:Params: {} 2024-04-16T19:56:28.363 DEBUG:boto:Token: None 2024-04-16T19:56:28.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e2a51a5c03afe9-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.366 DEBUG:boto:b'yrjefh-51obj000000000000000000401000falseobj000000000000000000402024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.366 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.366 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.366 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.366 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.367 DEBUG:boto:Method: GET 2024-04-16T19:56:28.367 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.367 DEBUG:boto:Data: 2024-04-16T19:56:28.367 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.367 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.367 DEBUG:boto:Params: {} 2024-04-16T19:56:28.367 DEBUG:boto:Token: None 2024-04-16T19:56:28.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9cb4000168441f2-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.371 DEBUG:boto:b'yrjefh-51obj000000000000000000401000falseobj000000000000000000402024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.372 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.372 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.372 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.372 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.372 DEBUG:boto:Data: 2024-04-16T19:56:28.372 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.372 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.372 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.372 DEBUG:boto:Params: {} 2024-04-16T19:56:28.372 DEBUG:boto:Token: None 2024-04-16T19:56:28.373 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nLfmtHpat5kiz8xR78FFAWO79gI= 2024-04-16T19:56:28.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nLfmtHpat5kiz8xR78FFAWO79gI=', 'Content-Length': '0'} 2024-04-16T19:56:28.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cdc4f02f430a8b-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.375 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.375 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.375 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.375 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.375 DEBUG:boto:Data: 2024-04-16T19:56:28.375 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.375 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.375 DEBUG:boto:Params: {} 2024-04-16T19:56:28.376 DEBUG:boto:Token: None 2024-04-16T19:56:28.376 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nLfmtHpat5kiz8xR78FFAWO79gI= 2024-04-16T19:56:28.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nLfmtHpat5kiz8xR78FFAWO79gI=', 'Content-Length': '0'} 2024-04-16T19:56:28.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f93ba56192a242-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-16T19:56:28.378 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.378 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.378 DEBUG:boto:Method: GET 2024-04-16T19:56:28.378 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.378 DEBUG:boto:Data: 2024-04-16T19:56:28.378 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.378 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.378 DEBUG:boto:Params: {} 2024-04-16T19:56:28.379 DEBUG:boto:Token: None 2024-04-16T19:56:28.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vjNAA23IKngmY/u0UJySjcUmbUM= 2024-04-16T19:56:28.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vjNAA23IKngmY/u0UJySjcUmbUM=', 'Content-Length': '0'} 2024-04-16T19:56:28.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a90656551a205cd-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.381 DEBUG:boto:path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.381 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.381 DEBUG:boto:Method: GET 2024-04-16T19:56:28.381 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.381 DEBUG:boto:Data: 2024-04-16T19:56:28.381 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.381 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.381 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.382 DEBUG:boto:Params: {} 2024-04-16T19:56:28.382 DEBUG:boto:Token: None 2024-04-16T19:56:28.382 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000040 2024-04-16T19:56:28.382 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vjNAA23IKngmY/u0UJySjcUmbUM= 2024-04-16T19:56:28.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vjNAA23IKngmY/u0UJySjcUmbUM=', 'Content-Length': '0'} 2024-04-16T19:56:28.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008082e9da23847660-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.386 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.386 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.386 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.386 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.386 DEBUG:boto:Method: GET 2024-04-16T19:56:28.386 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.386 DEBUG:boto:Data: 2024-04-16T19:56:28.386 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.386 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.386 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.386 DEBUG:boto:Params: {} 2024-04-16T19:56:28.387 DEBUG:boto:Token: None 2024-04-16T19:56:28.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d5d78bf51d04c38-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.390 DEBUG:boto:b'yrjefh-51obj000000000000000000401000falseobj000000000000000000402024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.390 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.390 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.390 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.390 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.390 DEBUG:boto:Method: GET 2024-04-16T19:56:28.390 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000040 2024-04-16T19:56:28.391 DEBUG:boto:Data: 2024-04-16T19:56:28.391 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.391 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.391 DEBUG:boto:Params: {} 2024-04-16T19:56:28.391 DEBUG:boto:Token: None 2024-04-16T19:56:28.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067007f430745c443-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.395 DEBUG:boto:b'yrjefh-51obj000000000000000000401000falseobj000000000000000000402024-04-16T19:56:05.762Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-16T19:56:28.395 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.395 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.395 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.395 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.395 DEBUG:boto:Method: GET 2024-04-16T19:56:28.396 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.396 DEBUG:boto:Data: 2024-04-16T19:56:28.396 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.396 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.396 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.396 DEBUG:boto:Params: {} 2024-04-16T19:56:28.396 DEBUG:boto:Token: None 2024-04-16T19:56:28.396 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ= 2024-04-16T19:56:28.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ=', 'Content-Length': '0'} 2024-04-16T19:56:28.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003987535160c1885d-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.399 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.399 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.399 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.399 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.399 DEBUG:boto:Method: GET 2024-04-16T19:56:28.399 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.399 DEBUG:boto:Data: 2024-04-16T19:56:28.399 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.399 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.400 DEBUG:boto:Params: {} 2024-04-16T19:56:28.400 DEBUG:boto:Token: None 2024-04-16T19:56:28.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ= 2024-04-16T19:56:28.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ=', 'Content-Length': '0'} 2024-04-16T19:56:28.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8ab05c6c5809de-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.402 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.402 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.402 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.402 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.402 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.403 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.403 DEBUG:boto:Data: 2024-04-16T19:56:28.403 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.403 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.403 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.403 DEBUG:boto:Params: {} 2024-04-16T19:56:28.403 DEBUG:boto:Token: None 2024-04-16T19:56:28.403 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.403 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VBOQznr5KbAAeddENilbSkGfKj8= 2024-04-16T19:56:28.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VBOQznr5KbAAeddENilbSkGfKj8=', 'Content-Length': '0'} 2024-04-16T19:56:28.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d4a1af1c335ad96-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.405 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.405 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.406 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.406 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.406 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.406 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.406 DEBUG:boto:Data: 2024-04-16T19:56:28.406 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.406 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.406 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.406 DEBUG:boto:Params: {} 2024-04-16T19:56:28.406 DEBUG:boto:Token: None 2024-04-16T19:56:28.406 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VBOQznr5KbAAeddENilbSkGfKj8= 2024-04-16T19:56:28.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VBOQznr5KbAAeddENilbSkGfKj8=', 'Content-Length': '0'} 2024-04-16T19:56:28.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb618da195bc34f-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-16T19:56:28.409 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.409 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.409 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.409 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.409 DEBUG:boto:Method: GET 2024-04-16T19:56:28.409 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.409 DEBUG:boto:Data: 2024-04-16T19:56:28.409 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.409 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.409 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.409 DEBUG:boto:Params: {} 2024-04-16T19:56:28.410 DEBUG:boto:Token: None 2024-04-16T19:56:28.410 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.410 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ= 2024-04-16T19:56:28.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ=', 'Content-Length': '0'} 2024-04-16T19:56:28.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa47149f981c5074-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.454 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.455 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.455 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.455 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.455 DEBUG:boto:Method: GET 2024-04-16T19:56:28.455 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.455 DEBUG:boto:Data: 2024-04-16T19:56:28.455 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.455 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.455 DEBUG:boto:Params: {} 2024-04-16T19:56:28.455 DEBUG:boto:Token: None 2024-04-16T19:56:28.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042?versionId=null 2024-04-16T19:56:28.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ= 2024-04-16T19:56:28.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ubv2XMz72KcVqrH/REE6BRVGoQ=', 'Content-Length': '0'} 2024-04-16T19:56:28.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbd051d28451659-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.458 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.458 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.458 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.458 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.458 DEBUG:boto:Method: GET 2024-04-16T19:56:28.458 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.458 DEBUG:boto:Data: 2024-04-16T19:56:28.458 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.458 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.458 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.458 DEBUG:boto:Params: {} 2024-04-16T19:56:28.459 DEBUG:boto:Token: None 2024-04-16T19:56:28.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c899e09a3a6d18c4-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.462 DEBUG:boto:b'yrjefh-51obj000000000000000000421000falseobj000000000000000000422024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.462 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.462 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.462 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.462 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.462 DEBUG:boto:Method: GET 2024-04-16T19:56:28.462 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.462 DEBUG:boto:Data: 2024-04-16T19:56:28.463 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.463 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.463 DEBUG:boto:Params: {} 2024-04-16T19:56:28.463 DEBUG:boto:Token: None 2024-04-16T19:56:28.463 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003490cb8d4a04f89e-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.467 DEBUG:boto:b'yrjefh-51obj000000000000000000421000falseobj000000000000000000422024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.467 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.467 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.468 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.468 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.468 DEBUG:boto:Data: 2024-04-16T19:56:28.468 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.468 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.468 DEBUG:boto:Params: {} 2024-04-16T19:56:28.468 DEBUG:boto:Token: None 2024-04-16T19:56:28.468 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:owqo3mGV1LsXYHS+BK6kjMMFj2Q= 2024-04-16T19:56:28.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:owqo3mGV1LsXYHS+BK6kjMMFj2Q=', 'Content-Length': '0'} 2024-04-16T19:56:28.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4fee2f69fd4752f-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.471 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.471 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.471 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.471 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.471 DEBUG:boto:Data: 2024-04-16T19:56:28.471 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.471 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.471 DEBUG:boto:Params: {} 2024-04-16T19:56:28.471 DEBUG:boto:Token: None 2024-04-16T19:56:28.471 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:owqo3mGV1LsXYHS+BK6kjMMFj2Q= 2024-04-16T19:56:28.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:owqo3mGV1LsXYHS+BK6kjMMFj2Q=', 'Content-Length': '0'} 2024-04-16T19:56:28.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29767cb19e3a60e-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-16T19:56:28.474 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.474 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.474 DEBUG:boto:Method: GET 2024-04-16T19:56:28.474 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.474 DEBUG:boto:Data: 2024-04-16T19:56:28.474 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.474 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.474 DEBUG:boto:Params: {} 2024-04-16T19:56:28.475 DEBUG:boto:Token: None 2024-04-16T19:56:28.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9U/AX5Q5eAkLgCmlr4cSZW3jrE4= 2024-04-16T19:56:28.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9U/AX5Q5eAkLgCmlr4cSZW3jrE4=', 'Content-Length': '0'} 2024-04-16T19:56:28.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82fa86020f3de1b-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.477 DEBUG:boto:path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.477 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.477 DEBUG:boto:Method: GET 2024-04-16T19:56:28.477 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.477 DEBUG:boto:Data: 2024-04-16T19:56:28.477 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.477 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.477 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.477 DEBUG:boto:Params: {} 2024-04-16T19:56:28.478 DEBUG:boto:Token: None 2024-04-16T19:56:28.478 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000042 2024-04-16T19:56:28.478 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9U/AX5Q5eAkLgCmlr4cSZW3jrE4= 2024-04-16T19:56:28.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9U/AX5Q5eAkLgCmlr4cSZW3jrE4=', 'Content-Length': '0'} 2024-04-16T19:56:28.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd06f751b4b23d0-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.522 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.523 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.523 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.523 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.523 DEBUG:boto:Method: GET 2024-04-16T19:56:28.523 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.523 DEBUG:boto:Data: 2024-04-16T19:56:28.523 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.523 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.523 DEBUG:boto:Params: {} 2024-04-16T19:56:28.523 DEBUG:boto:Token: None 2024-04-16T19:56:28.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9927055f3d335fc-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.527 DEBUG:boto:b'yrjefh-51obj000000000000000000421000falseobj000000000000000000422024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.527 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.527 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.527 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.527 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.527 DEBUG:boto:Method: GET 2024-04-16T19:56:28.527 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000042 2024-04-16T19:56:28.527 DEBUG:boto:Data: 2024-04-16T19:56:28.527 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.528 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.528 DEBUG:boto:Params: {} 2024-04-16T19:56:28.528 DEBUG:boto:Token: None 2024-04-16T19:56:28.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b8430a39932329-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.532 DEBUG:boto:b'yrjefh-51obj000000000000000000421000falseobj000000000000000000422024-04-16T19:56:06.668Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-16T19:56:28.532 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.532 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.532 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.532 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.532 DEBUG:boto:Method: GET 2024-04-16T19:56:28.533 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.533 DEBUG:boto:Data: 2024-04-16T19:56:28.533 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.533 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.533 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.533 DEBUG:boto:Params: {} 2024-04-16T19:56:28.533 DEBUG:boto:Token: None 2024-04-16T19:56:28.533 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg= 2024-04-16T19:56:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg=', 'Content-Length': '0'} 2024-04-16T19:56:28.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094486d83ecfc97cc-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.536 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.536 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.536 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.536 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.536 DEBUG:boto:Method: GET 2024-04-16T19:56:28.536 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.536 DEBUG:boto:Data: 2024-04-16T19:56:28.536 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.536 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.536 DEBUG:boto:Params: {} 2024-04-16T19:56:28.537 DEBUG:boto:Token: None 2024-04-16T19:56:28.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg= 2024-04-16T19:56:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg=', 'Content-Length': '0'} 2024-04-16T19:56:28.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3571a936a008d6-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.539 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.539 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.540 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.540 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.540 DEBUG:boto:Data: 2024-04-16T19:56:28.540 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.540 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.540 DEBUG:boto:Params: {} 2024-04-16T19:56:28.540 DEBUG:boto:Token: None 2024-04-16T19:56:28.540 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FrlPyito+8Kb8ackiw0/NWXRbwc= 2024-04-16T19:56:28.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FrlPyito+8Kb8ackiw0/NWXRbwc=', 'Content-Length': '0'} 2024-04-16T19:56:28.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098231c5bf1abae30-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.542 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.542 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.542 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.543 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.543 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.543 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.543 DEBUG:boto:Data: 2024-04-16T19:56:28.543 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.543 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.543 DEBUG:boto:Params: {} 2024-04-16T19:56:28.543 DEBUG:boto:Token: None 2024-04-16T19:56:28.543 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FrlPyito+8Kb8ackiw0/NWXRbwc= 2024-04-16T19:56:28.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FrlPyito+8Kb8ackiw0/NWXRbwc=', 'Content-Length': '0'} 2024-04-16T19:56:28.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095999cc1d9a0fb12-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-16T19:56:28.546 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.546 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.546 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.546 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.546 DEBUG:boto:Method: GET 2024-04-16T19:56:28.546 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.546 DEBUG:boto:Data: 2024-04-16T19:56:28.546 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.546 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.546 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.546 DEBUG:boto:Params: {} 2024-04-16T19:56:28.547 DEBUG:boto:Token: None 2024-04-16T19:56:28.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg= 2024-04-16T19:56:28.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg=', 'Content-Length': '0'} 2024-04-16T19:56:28.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843820d0710680ef-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.590 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.590 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.591 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.591 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.591 DEBUG:boto:Method: GET 2024-04-16T19:56:28.591 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.591 DEBUG:boto:Data: 2024-04-16T19:56:28.591 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.591 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.591 DEBUG:boto:Params: {} 2024-04-16T19:56:28.591 DEBUG:boto:Token: None 2024-04-16T19:56:28.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045?versionId=null 2024-04-16T19:56:28.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg= 2024-04-16T19:56:28.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BcQ9sTCMguMKvVQ928ltvoTTpyg=', 'Content-Length': '0'} 2024-04-16T19:56:28.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067427a875dec322e-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.594 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.594 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.594 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.594 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.594 DEBUG:boto:Method: GET 2024-04-16T19:56:28.594 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.594 DEBUG:boto:Data: 2024-04-16T19:56:28.594 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.595 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.595 DEBUG:boto:Params: {} 2024-04-16T19:56:28.595 DEBUG:boto:Token: None 2024-04-16T19:56:28.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003b4c68a93da88fc-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.598 DEBUG:boto:b'yrjefh-51obj000000000000000000451000falseobj000000000000000000452024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.598 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.598 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.598 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.598 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.598 DEBUG:boto:Method: GET 2024-04-16T19:56:28.598 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.598 DEBUG:boto:Data: 2024-04-16T19:56:28.599 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.599 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.599 DEBUG:boto:Params: {} 2024-04-16T19:56:28.599 DEBUG:boto:Token: None 2024-04-16T19:56:28.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb748ed4b4d505e-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.603 DEBUG:boto:b'yrjefh-51obj000000000000000000451000falseobj000000000000000000452024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.604 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.604 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.604 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.604 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.604 DEBUG:boto:Data: 2024-04-16T19:56:28.604 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.604 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.604 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.604 DEBUG:boto:Params: {} 2024-04-16T19:56:28.604 DEBUG:boto:Token: None 2024-04-16T19:56:28.604 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hth+nrVPZsoFcUxyOhFQaeWhGgg= 2024-04-16T19:56:28.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hth+nrVPZsoFcUxyOhFQaeWhGgg=', 'Content-Length': '0'} 2024-04-16T19:56:28.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d17a19660850815-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.607 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.607 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.607 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.607 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.607 DEBUG:boto:Data: 2024-04-16T19:56:28.607 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.608 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.608 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.608 DEBUG:boto:Params: {} 2024-04-16T19:56:28.608 DEBUG:boto:Token: None 2024-04-16T19:56:28.608 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hth+nrVPZsoFcUxyOhFQaeWhGgg= 2024-04-16T19:56:28.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hth+nrVPZsoFcUxyOhFQaeWhGgg=', 'Content-Length': '0'} 2024-04-16T19:56:28.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008726a667d857f71-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-16T19:56:28.610 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.610 DEBUG:boto:Method: GET 2024-04-16T19:56:28.611 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.611 DEBUG:boto:Data: 2024-04-16T19:56:28.611 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.611 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.611 DEBUG:boto:Params: {} 2024-04-16T19:56:28.611 DEBUG:boto:Token: None 2024-04-16T19:56:28.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhxkQUuAdIxplpXR4lPyCvrjtGc= 2024-04-16T19:56:28.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhxkQUuAdIxplpXR4lPyCvrjtGc=', 'Content-Length': '0'} 2024-04-16T19:56:28.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ce66cfac49f665-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.613 DEBUG:boto:path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.613 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.614 DEBUG:boto:Method: GET 2024-04-16T19:56:28.614 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.614 DEBUG:boto:Data: 2024-04-16T19:56:28.614 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.614 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.614 DEBUG:boto:Params: {} 2024-04-16T19:56:28.614 DEBUG:boto:Token: None 2024-04-16T19:56:28.614 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000045 2024-04-16T19:56:28.614 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhxkQUuAdIxplpXR4lPyCvrjtGc= 2024-04-16T19:56:28.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhxkQUuAdIxplpXR4lPyCvrjtGc=', 'Content-Length': '0'} 2024-04-16T19:56:28.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001472bfcda151f086-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.616 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.617 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.617 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.617 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.617 DEBUG:boto:Method: GET 2024-04-16T19:56:28.617 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.617 DEBUG:boto:Data: 2024-04-16T19:56:28.617 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.617 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.617 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.617 DEBUG:boto:Params: {} 2024-04-16T19:56:28.617 DEBUG:boto:Token: None 2024-04-16T19:56:28.618 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.618 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f36def1c1f263dec-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.621 DEBUG:boto:b'yrjefh-51obj000000000000000000451000falseobj000000000000000000452024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.621 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.621 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.621 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.621 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.621 DEBUG:boto:Method: GET 2024-04-16T19:56:28.621 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000045 2024-04-16T19:56:28.621 DEBUG:boto:Data: 2024-04-16T19:56:28.621 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.622 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.622 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.622 DEBUG:boto:Params: {} 2024-04-16T19:56:28.622 DEBUG:boto:Token: None 2024-04-16T19:56:28.622 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.622 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542919fcdc2f570e-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.626 DEBUG:boto:b'yrjefh-51obj000000000000000000451000falseobj000000000000000000452024-04-16T19:56:09.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-16T19:56:28.627 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.627 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.627 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.627 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.627 DEBUG:boto:Method: GET 2024-04-16T19:56:28.627 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.627 DEBUG:boto:Data: 2024-04-16T19:56:28.627 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.627 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.627 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.627 DEBUG:boto:Params: {} 2024-04-16T19:56:28.628 DEBUG:boto:Token: None 2024-04-16T19:56:28.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c= 2024-04-16T19:56:28.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c=', 'Content-Length': '0'} 2024-04-16T19:56:28.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13d0f96daa33794-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.630 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.630 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.630 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.630 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.630 DEBUG:boto:Method: GET 2024-04-16T19:56:28.631 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.631 DEBUG:boto:Data: 2024-04-16T19:56:28.631 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.631 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.631 DEBUG:boto:Params: {} 2024-04-16T19:56:28.631 DEBUG:boto:Token: None 2024-04-16T19:56:28.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c= 2024-04-16T19:56:28.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c=', 'Content-Length': '0'} 2024-04-16T19:56:28.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bdf22677d1ba920-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.633 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.634 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.634 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.634 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.634 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.634 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.634 DEBUG:boto:Data: 2024-04-16T19:56:28.634 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.634 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.634 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.634 DEBUG:boto:Params: {} 2024-04-16T19:56:28.634 DEBUG:boto:Token: None 2024-04-16T19:56:28.635 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diZU3+mI6+qrACC5Ia4CAlNe7jI= 2024-04-16T19:56:28.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diZU3+mI6+qrACC5Ia4CAlNe7jI=', 'Content-Length': '0'} 2024-04-16T19:56:28.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4909bef119b1613-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.637 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.637 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.637 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.637 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.637 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.637 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.637 DEBUG:boto:Data: 2024-04-16T19:56:28.637 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.638 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.638 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.638 DEBUG:boto:Params: {} 2024-04-16T19:56:28.638 DEBUG:boto:Token: None 2024-04-16T19:56:28.638 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.638 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diZU3+mI6+qrACC5Ia4CAlNe7jI= 2024-04-16T19:56:28.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diZU3+mI6+qrACC5Ia4CAlNe7jI=', 'Content-Length': '0'} 2024-04-16T19:56:28.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a21807f72712247-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-16T19:56:28.640 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.640 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.640 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.640 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.640 DEBUG:boto:Method: GET 2024-04-16T19:56:28.641 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.641 DEBUG:boto:Data: 2024-04-16T19:56:28.641 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.641 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.641 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.641 DEBUG:boto:Params: {} 2024-04-16T19:56:28.641 DEBUG:boto:Token: None 2024-04-16T19:56:28.641 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c= 2024-04-16T19:56:28.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c=', 'Content-Length': '0'} 2024-04-16T19:56:28.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66d2d95b9e609bc-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.643 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.643 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.644 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.644 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.644 DEBUG:boto:Method: GET 2024-04-16T19:56:28.644 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.644 DEBUG:boto:Data: 2024-04-16T19:56:28.644 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.644 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.644 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.644 DEBUG:boto:Params: {} 2024-04-16T19:56:28.644 DEBUG:boto:Token: None 2024-04-16T19:56:28.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046?versionId=null 2024-04-16T19:56:28.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c= 2024-04-16T19:56:28.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5X+fL+MvIGQxvsAEm4HfVEYvg3c=', 'Content-Length': '0'} 2024-04-16T19:56:28.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd579a452f518bf-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.646 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.647 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.647 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.647 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.647 DEBUG:boto:Method: GET 2024-04-16T19:56:28.647 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.647 DEBUG:boto:Data: 2024-04-16T19:56:28.647 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.647 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.647 DEBUG:boto:Params: {} 2024-04-16T19:56:28.647 DEBUG:boto:Token: None 2024-04-16T19:56:28.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368a33fffc8fc33d-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.690 DEBUG:boto:b'yrjefh-51obj000000000000000000461000falseobj000000000000000000462024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.691 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.691 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.691 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.691 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.691 DEBUG:boto:Method: GET 2024-04-16T19:56:28.691 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.691 DEBUG:boto:Data: 2024-04-16T19:56:28.691 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.692 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.692 DEBUG:boto:Params: {} 2024-04-16T19:56:28.692 DEBUG:boto:Token: None 2024-04-16T19:56:28.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ea9c3825c4847c-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.696 DEBUG:boto:b'yrjefh-51obj000000000000000000461000falseobj000000000000000000462024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.696 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.696 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.697 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.697 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.697 DEBUG:boto:Data: 2024-04-16T19:56:28.697 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.697 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.697 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.697 DEBUG:boto:Params: {} 2024-04-16T19:56:28.697 DEBUG:boto:Token: None 2024-04-16T19:56:28.697 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.697 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQUi3O8hdDqCrKW7ENb8qxxF6Z0= 2024-04-16T19:56:28.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQUi3O8hdDqCrKW7ENb8qxxF6Z0=', 'Content-Length': '0'} 2024-04-16T19:56:28.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99fdd74768c735c-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.699 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.699 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.700 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.700 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.700 DEBUG:boto:Data: 2024-04-16T19:56:28.700 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.700 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.700 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.700 DEBUG:boto:Params: {} 2024-04-16T19:56:28.700 DEBUG:boto:Token: None 2024-04-16T19:56:28.700 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQUi3O8hdDqCrKW7ENb8qxxF6Z0= 2024-04-16T19:56:28.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQUi3O8hdDqCrKW7ENb8qxxF6Z0=', 'Content-Length': '0'} 2024-04-16T19:56:28.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17ac1ab1e9a4642-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-16T19:56:28.703 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.703 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.703 DEBUG:boto:Method: GET 2024-04-16T19:56:28.703 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.703 DEBUG:boto:Data: 2024-04-16T19:56:28.703 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.703 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.703 DEBUG:boto:Params: {} 2024-04-16T19:56:28.703 DEBUG:boto:Token: None 2024-04-16T19:56:28.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YVX7NOSZaDSdZKdF2ib+faIo2KE= 2024-04-16T19:56:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YVX7NOSZaDSdZKdF2ib+faIo2KE=', 'Content-Length': '0'} 2024-04-16T19:56:28.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da38ac7914a7364d-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.706 DEBUG:boto:path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.706 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.706 DEBUG:boto:Method: GET 2024-04-16T19:56:28.706 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.706 DEBUG:boto:Data: 2024-04-16T19:56:28.706 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.706 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.706 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.707 DEBUG:boto:Params: {} 2024-04-16T19:56:28.707 DEBUG:boto:Token: None 2024-04-16T19:56:28.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000046 2024-04-16T19:56:28.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YVX7NOSZaDSdZKdF2ib+faIo2KE= 2024-04-16T19:56:28.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YVX7NOSZaDSdZKdF2ib+faIo2KE=', 'Content-Length': '0'} 2024-04-16T19:56:28.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004409c35129c634cc-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.750 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.750 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.751 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.751 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.751 DEBUG:boto:Method: GET 2024-04-16T19:56:28.751 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.751 DEBUG:boto:Data: 2024-04-16T19:56:28.751 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.751 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.751 DEBUG:boto:Params: {} 2024-04-16T19:56:28.751 DEBUG:boto:Token: None 2024-04-16T19:56:28.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063062b8f48147226-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.755 DEBUG:boto:b'yrjefh-51obj000000000000000000461000falseobj000000000000000000462024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.755 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.755 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.755 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.755 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.755 DEBUG:boto:Method: GET 2024-04-16T19:56:28.755 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000046 2024-04-16T19:56:28.755 DEBUG:boto:Data: 2024-04-16T19:56:28.756 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.756 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.756 DEBUG:boto:Params: {} 2024-04-16T19:56:28.756 DEBUG:boto:Token: None 2024-04-16T19:56:28.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f9f26b894d34528-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.760 DEBUG:boto:b'yrjefh-51obj000000000000000000461000falseobj000000000000000000462024-04-16T19:56:05.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-16T19:56:28.761 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.761 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.761 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.761 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.761 DEBUG:boto:Method: GET 2024-04-16T19:56:28.761 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.761 DEBUG:boto:Data: 2024-04-16T19:56:28.761 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.761 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.761 DEBUG:boto:Params: {} 2024-04-16T19:56:28.761 DEBUG:boto:Token: None 2024-04-16T19:56:28.762 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY= 2024-04-16T19:56:28.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY=', 'Content-Length': '0'} 2024-04-16T19:56:28.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa52c3b99649c0c-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.807 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.807 DEBUG:boto:Method: GET 2024-04-16T19:56:28.807 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.807 DEBUG:boto:Data: 2024-04-16T19:56:28.807 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.807 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.807 DEBUG:boto:Params: {} 2024-04-16T19:56:28.807 DEBUG:boto:Token: None 2024-04-16T19:56:28.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY= 2024-04-16T19:56:28.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY=', 'Content-Length': '0'} 2024-04-16T19:56:28.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e9bf84e0926c62-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.810 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.810 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.810 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.810 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.810 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.810 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.810 DEBUG:boto:Data: 2024-04-16T19:56:28.810 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.811 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.811 DEBUG:boto:Params: {} 2024-04-16T19:56:28.811 DEBUG:boto:Token: None 2024-04-16T19:56:28.811 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qqOAHzc0MNut0tsuCev5vza5idU= 2024-04-16T19:56:28.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qqOAHzc0MNut0tsuCev5vza5idU=', 'Content-Length': '0'} 2024-04-16T19:56:28.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f49aca57ea55c8d-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.813 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.813 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.813 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.813 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.813 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.814 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.814 DEBUG:boto:Data: 2024-04-16T19:56:28.814 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.814 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.814 DEBUG:boto:Params: {} 2024-04-16T19:56:28.814 DEBUG:boto:Token: None 2024-04-16T19:56:28.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qqOAHzc0MNut0tsuCev5vza5idU= 2024-04-16T19:56:28.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qqOAHzc0MNut0tsuCev5vza5idU=', 'Content-Length': '0'} 2024-04-16T19:56:28.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85067aa2c4da528-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-16T19:56:28.816 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.817 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.817 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.817 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.817 DEBUG:boto:Method: GET 2024-04-16T19:56:28.817 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.817 DEBUG:boto:Data: 2024-04-16T19:56:28.817 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.817 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.817 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.817 DEBUG:boto:Params: {} 2024-04-16T19:56:28.817 DEBUG:boto:Token: None 2024-04-16T19:56:28.817 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY= 2024-04-16T19:56:28.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY=', 'Content-Length': '0'} 2024-04-16T19:56:28.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc6391eea88b655-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.820 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.820 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.820 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.820 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.820 DEBUG:boto:Method: GET 2024-04-16T19:56:28.820 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.820 DEBUG:boto:Data: 2024-04-16T19:56:28.820 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.820 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.821 DEBUG:boto:Params: {} 2024-04-16T19:56:28.821 DEBUG:boto:Token: None 2024-04-16T19:56:28.821 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048?versionId=null 2024-04-16T19:56:28.821 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY= 2024-04-16T19:56:28.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k3XuXI3/bG0CW+Qa+0aZBYnENhY=', 'Content-Length': '0'} 2024-04-16T19:56:28.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43f184803ad3699-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.823 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.823 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.823 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.823 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.823 DEBUG:boto:Method: GET 2024-04-16T19:56:28.824 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.824 DEBUG:boto:Data: 2024-04-16T19:56:28.824 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.824 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.824 DEBUG:boto:Params: {} 2024-04-16T19:56:28.824 DEBUG:boto:Token: None 2024-04-16T19:56:28.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed812e81c2277baf-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.827 DEBUG:boto:b'yrjefh-51obj000000000000000000481000falseobj000000000000000000482024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.827 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.828 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.828 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.828 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.828 DEBUG:boto:Method: GET 2024-04-16T19:56:28.828 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.828 DEBUG:boto:Data: 2024-04-16T19:56:28.828 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.828 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.828 DEBUG:boto:Params: {} 2024-04-16T19:56:28.828 DEBUG:boto:Token: None 2024-04-16T19:56:28.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.829 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a50731a5627e7f28-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.833 DEBUG:boto:b'yrjefh-51obj000000000000000000481000falseobj000000000000000000482024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.833 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.833 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.833 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.833 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.833 DEBUG:boto:Data: 2024-04-16T19:56:28.833 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.833 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.834 DEBUG:boto:Params: {} 2024-04-16T19:56:28.834 DEBUG:boto:Token: None 2024-04-16T19:56:28.834 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5N7sVXumr7DWRPr0Z1FHP1Zc1ho= 2024-04-16T19:56:28.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5N7sVXumr7DWRPr0Z1FHP1Zc1ho=', 'Content-Length': '0'} 2024-04-16T19:56:28.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002963d8ef0edfafd6-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.836 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.836 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.836 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.836 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.837 DEBUG:boto:Data: 2024-04-16T19:56:28.837 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.837 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.837 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.837 DEBUG:boto:Params: {} 2024-04-16T19:56:28.837 DEBUG:boto:Token: None 2024-04-16T19:56:28.837 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.837 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5N7sVXumr7DWRPr0Z1FHP1Zc1ho= 2024-04-16T19:56:28.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5N7sVXumr7DWRPr0Z1FHP1Zc1ho=', 'Content-Length': '0'} 2024-04-16T19:56:28.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5635535a36d9f4-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-16T19:56:28.839 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.839 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.839 DEBUG:boto:Method: GET 2024-04-16T19:56:28.840 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.840 DEBUG:boto:Data: 2024-04-16T19:56:28.840 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.840 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.840 DEBUG:boto:Params: {} 2024-04-16T19:56:28.840 DEBUG:boto:Token: None 2024-04-16T19:56:28.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PEQalO276jREmbnVAumQRq/378E= 2024-04-16T19:56:28.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PEQalO276jREmbnVAumQRq/378E=', 'Content-Length': '0'} 2024-04-16T19:56:28.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000633a713516d48658-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.886 DEBUG:boto:path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.887 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.887 DEBUG:boto:Method: GET 2024-04-16T19:56:28.887 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.887 DEBUG:boto:Data: 2024-04-16T19:56:28.887 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.887 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.887 DEBUG:boto:Params: {} 2024-04-16T19:56:28.887 DEBUG:boto:Token: None 2024-04-16T19:56:28.887 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000048 2024-04-16T19:56:28.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PEQalO276jREmbnVAumQRq/378E= 2024-04-16T19:56:28.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PEQalO276jREmbnVAumQRq/378E=', 'Content-Length': '0'} 2024-04-16T19:56:28.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047b8a7c839403bc-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.890 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.890 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.890 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.890 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.890 DEBUG:boto:Method: GET 2024-04-16T19:56:28.890 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.890 DEBUG:boto:Data: 2024-04-16T19:56:28.890 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.890 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.890 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.890 DEBUG:boto:Params: {} 2024-04-16T19:56:28.891 DEBUG:boto:Token: None 2024-04-16T19:56:28.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a322b05de8636e8-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.893 DEBUG:boto:b'yrjefh-51obj000000000000000000481000falseobj000000000000000000482024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.894 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.894 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.894 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.894 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.894 DEBUG:boto:Method: GET 2024-04-16T19:56:28.894 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000048 2024-04-16T19:56:28.894 DEBUG:boto:Data: 2024-04-16T19:56:28.894 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.894 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.894 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.894 DEBUG:boto:Params: {} 2024-04-16T19:56:28.895 DEBUG:boto:Token: None 2024-04-16T19:56:28.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb441fd49e29a87-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.899 DEBUG:boto:b'yrjefh-51obj000000000000000000481000falseobj000000000000000000482024-04-16T19:56:08.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-16T19:56:28.899 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.899 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.899 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.899 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.900 DEBUG:boto:Method: GET 2024-04-16T19:56:28.900 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.900 DEBUG:boto:Data: 2024-04-16T19:56:28.900 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.900 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.900 DEBUG:boto:Params: {} 2024-04-16T19:56:28.900 DEBUG:boto:Token: None 2024-04-16T19:56:28.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM= 2024-04-16T19:56:28.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM=', 'Content-Length': '0'} 2024-04-16T19:56:28.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2794ea04a4ff237-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.903 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.903 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.903 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.903 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.903 DEBUG:boto:Method: GET 2024-04-16T19:56:28.903 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.903 DEBUG:boto:Data: 2024-04-16T19:56:28.903 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.904 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.904 DEBUG:boto:Params: {} 2024-04-16T19:56:28.904 DEBUG:boto:Token: None 2024-04-16T19:56:28.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM= 2024-04-16T19:56:28.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM=', 'Content-Length': '0'} 2024-04-16T19:56:28.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fa5d6f22301e7c-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.906 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.906 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.906 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.907 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.907 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.907 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.907 DEBUG:boto:Data: 2024-04-16T19:56:28.907 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.907 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.907 DEBUG:boto:Params: {} 2024-04-16T19:56:28.907 DEBUG:boto:Token: None 2024-04-16T19:56:28.907 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SWlNAzziBfS5lB6va4bjcR89N8= 2024-04-16T19:56:28.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SWlNAzziBfS5lB6va4bjcR89N8=', 'Content-Length': '0'} 2024-04-16T19:56:28.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000852460c91a0d2936-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.910 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.910 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.910 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.910 DEBUG:boto:Data: 2024-04-16T19:56:28.910 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.910 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.910 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.910 DEBUG:boto:Params: {} 2024-04-16T19:56:28.910 DEBUG:boto:Token: None 2024-04-16T19:56:28.910 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SWlNAzziBfS5lB6va4bjcR89N8= 2024-04-16T19:56:28.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SWlNAzziBfS5lB6va4bjcR89N8=', 'Content-Length': '0'} 2024-04-16T19:56:28.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a581ba75b9ecbd6b-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-16T19:56:28.912 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.913 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.913 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.913 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.913 DEBUG:boto:Method: GET 2024-04-16T19:56:28.913 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.913 DEBUG:boto:Data: 2024-04-16T19:56:28.913 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.913 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.913 DEBUG:boto:Params: {} 2024-04-16T19:56:28.913 DEBUG:boto:Token: None 2024-04-16T19:56:28.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.914 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM= 2024-04-16T19:56:28.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM=', 'Content-Length': '0'} 2024-04-16T19:56:28.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca90bba09e16710-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.916 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.916 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.916 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.916 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.916 DEBUG:boto:Method: GET 2024-04-16T19:56:28.916 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.916 DEBUG:boto:Data: 2024-04-16T19:56:28.916 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.917 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.917 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.917 DEBUG:boto:Params: {} 2024-04-16T19:56:28.917 DEBUG:boto:Token: None 2024-04-16T19:56:28.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049?versionId=null 2024-04-16T19:56:28.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM= 2024-04-16T19:56:28.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EaHR++qrVC0bvbfNje38xDZDYiM=', 'Content-Length': '0'} 2024-04-16T19:56:28.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3beb7abcd218567-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.919 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.919 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.919 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.919 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.919 DEBUG:boto:Method: GET 2024-04-16T19:56:28.919 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.920 DEBUG:boto:Data: 2024-04-16T19:56:28.920 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.920 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.920 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.920 DEBUG:boto:Params: {} 2024-04-16T19:56:28.920 DEBUG:boto:Token: None 2024-04-16T19:56:28.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a4ce31a1271a8cb-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.923 DEBUG:boto:b'yrjefh-51obj000000000000000000491000falseobj000000000000000000492024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.923 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.923 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.923 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.923 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.923 DEBUG:boto:Method: GET 2024-04-16T19:56:28.923 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.924 DEBUG:boto:Data: 2024-04-16T19:56:28.924 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.924 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.924 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.924 DEBUG:boto:Params: {} 2024-04-16T19:56:28.924 DEBUG:boto:Token: None 2024-04-16T19:56:28.924 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a3808d6a0b89da2-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.928 DEBUG:boto:b'yrjefh-51obj000000000000000000491000falseobj000000000000000000492024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.928 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.928 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.929 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.929 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.929 DEBUG:boto:Data: 2024-04-16T19:56:28.929 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.929 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.929 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.929 DEBUG:boto:Params: {} 2024-04-16T19:56:28.929 DEBUG:boto:Token: None 2024-04-16T19:56:28.929 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oHyIQF9UukY2oBv9Y3BURhmeyUU= 2024-04-16T19:56:28.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oHyIQF9UukY2oBv9Y3BURhmeyUU=', 'Content-Length': '0'} 2024-04-16T19:56:28.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06fa77522fa08fc-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.932 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.932 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.932 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.932 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.932 DEBUG:boto:Data: 2024-04-16T19:56:28.932 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.932 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.932 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.932 DEBUG:boto:Params: {} 2024-04-16T19:56:28.932 DEBUG:boto:Token: None 2024-04-16T19:56:28.933 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.933 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oHyIQF9UukY2oBv9Y3BURhmeyUU= 2024-04-16T19:56:28.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oHyIQF9UukY2oBv9Y3BURhmeyUU=', 'Content-Length': '0'} 2024-04-16T19:56:28.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f84777c6581d68-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2024-04-16T19:56:28.935 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.935 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.935 DEBUG:boto:Method: GET 2024-04-16T19:56:28.935 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.935 DEBUG:boto:Data: 2024-04-16T19:56:28.935 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.935 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.935 DEBUG:boto:Params: {} 2024-04-16T19:56:28.935 DEBUG:boto:Token: None 2024-04-16T19:56:28.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKi8tpzvYl+tOkaxLH18e8rBmO0= 2024-04-16T19:56:28.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKi8tpzvYl+tOkaxLH18e8rBmO0=', 'Content-Length': '0'} 2024-04-16T19:56:28.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee4ec29f3082041-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.978 DEBUG:boto:path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.979 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.979 DEBUG:boto:Method: GET 2024-04-16T19:56:28.979 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.979 DEBUG:boto:Data: 2024-04-16T19:56:28.979 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.979 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.979 DEBUG:boto:Params: {} 2024-04-16T19:56:28.979 DEBUG:boto:Token: None 2024-04-16T19:56:28.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000049 2024-04-16T19:56:28.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wKi8tpzvYl+tOkaxLH18e8rBmO0= 2024-04-16T19:56:28.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wKi8tpzvYl+tOkaxLH18e8rBmO0=', 'Content-Length': '0'} 2024-04-16T19:56:28.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066100b7a36847032-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.981 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.981 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.982 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.982 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.982 DEBUG:boto:Method: GET 2024-04-16T19:56:28.982 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.982 DEBUG:boto:Data: 2024-04-16T19:56:28.982 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.982 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.982 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.982 DEBUG:boto:Params: {} 2024-04-16T19:56:28.982 DEBUG:boto:Token: None 2024-04-16T19:56:28.982 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e963305db86f950e-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.985 DEBUG:boto:b'yrjefh-51obj000000000000000000491000falseobj000000000000000000492024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.985 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:28.986 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:28.986 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.986 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.986 DEBUG:boto:Method: GET 2024-04-16T19:56:28.986 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000049 2024-04-16T19:56:28.986 DEBUG:boto:Data: 2024-04-16T19:56:28.986 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.986 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.986 DEBUG:boto:Params: {} 2024-04-16T19:56:28.986 DEBUG:boto:Token: None 2024-04-16T19:56:28.986 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/ 2024-04-16T19:56:28.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk= 2024-04-16T19:56:28.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IEJRETyZw4tprlNsUQIA/lxScrk=', 'Content-Length': '0'} 2024-04-16T19:56:28.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000cbc392fffb03f5-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.991 DEBUG:boto:b'yrjefh-51obj000000000000000000491000falseobj000000000000000000492024-04-16T19:56:07.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:28.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-16T19:56:28.991 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.991 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.991 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.991 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.992 DEBUG:boto:Method: GET 2024-04-16T19:56:28.992 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.992 DEBUG:boto:Data: 2024-04-16T19:56:28.992 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.992 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.992 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.992 DEBUG:boto:Params: {} 2024-04-16T19:56:28.992 DEBUG:boto:Token: None 2024-04-16T19:56:28.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfHiARqXsnPAq0HSWsRUpAzuQg4= 2024-04-16T19:56:28.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfHiARqXsnPAq0HSWsRUpAzuQg4=', 'Content-Length': '0'} 2024-04-16T19:56:28.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a10d050e3766945-00661ed7ec-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.995 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.995 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.995 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.995 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.995 DEBUG:boto:Method: GET 2024-04-16T19:56:28.995 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.995 DEBUG:boto:Data: 2024-04-16T19:56:28.995 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:28.996 DEBUG:boto:Port: 8000 2024-04-16T19:56:28.996 DEBUG:boto:Params: {} 2024-04-16T19:56:28.996 DEBUG:boto:Token: None 2024-04-16T19:56:28.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:28 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZfHiARqXsnPAq0HSWsRUpAzuQg4= 2024-04-16T19:56:28.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZfHiARqXsnPAq0HSWsRUpAzuQg4=', 'Content-Length': '0'} 2024-04-16T19:56:28.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb51692dee5e5256-00661ed7ec-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:28.999 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.999 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:28.999 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.999 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.999 DEBUG:boto:Method: HEAD 2024-04-16T19:56:28.999 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:28.999 DEBUG:boto:Data: 2024-04-16T19:56:28.999 DEBUG:boto:Headers: {} 2024-04-16T19:56:28.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:28.999 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.000 DEBUG:boto:Params: {} 2024-04-16T19:56:29.000 DEBUG:boto:Token: None 2024-04-16T19:56:29.000 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3JnXTRxLWRIzMz7Gu43TxD4rGOA= 2024-04-16T19:56:29.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JnXTRxLWRIzMz7Gu43TxD4rGOA=', 'Content-Length': '0'} 2024-04-16T19:56:29.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080744b9ca451400a-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.002 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.002 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.003 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.003 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.003 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.003 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.003 DEBUG:boto:Data: 2024-04-16T19:56:29.003 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.003 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.003 DEBUG:boto:Params: {} 2024-04-16T19:56:29.003 DEBUG:boto:Token: None 2024-04-16T19:56:29.003 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3JnXTRxLWRIzMz7Gu43TxD4rGOA= 2024-04-16T19:56:29.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JnXTRxLWRIzMz7Gu43TxD4rGOA=', 'Content-Length': '0'} 2024-04-16T19:56:29.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88e847a070027e2-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-16T19:56:29.006 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.006 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.006 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.006 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.006 DEBUG:boto:Method: GET 2024-04-16T19:56:29.006 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.007 DEBUG:boto:Data: 2024-04-16T19:56:29.007 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.007 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.007 DEBUG:boto:Params: {} 2024-04-16T19:56:29.007 DEBUG:boto:Token: None 2024-04-16T19:56:29.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WwCBKnJDH1/54hg3BGdd25O9ors= 2024-04-16T19:56:29.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WwCBKnJDH1/54hg3BGdd25O9ors=', 'Content-Length': '0'} 2024-04-16T19:56:29.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323f1689cec5d8ba-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.010 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.010 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.010 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.010 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.010 DEBUG:boto:Method: GET 2024-04-16T19:56:29.010 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.010 DEBUG:boto:Data: 2024-04-16T19:56:29.010 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.011 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.011 DEBUG:boto:Params: {} 2024-04-16T19:56:29.011 DEBUG:boto:Token: None 2024-04-16T19:56:29.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050?versionId=null 2024-04-16T19:56:29.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WwCBKnJDH1/54hg3BGdd25O9ors= 2024-04-16T19:56:29.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WwCBKnJDH1/54hg3BGdd25O9ors=', 'Content-Length': '0'} 2024-04-16T19:56:29.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017491ebba5dd6fe1-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.014 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.014 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.014 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.014 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.014 DEBUG:boto:Method: GET 2024-04-16T19:56:29.014 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.014 DEBUG:boto:Data: 2024-04-16T19:56:29.014 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.014 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.014 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.014 DEBUG:boto:Params: {} 2024-04-16T19:56:29.015 DEBUG:boto:Token: None 2024-04-16T19:56:29.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c307635fa7fff0a4-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.018 DEBUG:boto:b'yrjefh-51obj000000000000000000501000falseobj000000000000000000502024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.018 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.018 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.018 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.018 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.018 DEBUG:boto:Method: GET 2024-04-16T19:56:29.018 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.019 DEBUG:boto:Data: 2024-04-16T19:56:29.019 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.019 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.019 DEBUG:boto:Params: {} 2024-04-16T19:56:29.019 DEBUG:boto:Token: None 2024-04-16T19:56:29.019 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.019 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6752e40948d1813-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.023 DEBUG:boto:b'yrjefh-51obj000000000000000000501000falseobj000000000000000000502024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.023 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.023 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.023 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.023 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.024 DEBUG:boto:Data: 2024-04-16T19:56:29.024 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.024 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.024 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.024 DEBUG:boto:Params: {} 2024-04-16T19:56:29.024 DEBUG:boto:Token: None 2024-04-16T19:56:29.024 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:36fVBDnmvcfQjFusO6liFXUw71I= 2024-04-16T19:56:29.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:36fVBDnmvcfQjFusO6liFXUw71I=', 'Content-Length': '0'} 2024-04-16T19:56:29.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ae38143f12839e-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.027 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.027 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.027 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.027 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.027 DEBUG:boto:Data: 2024-04-16T19:56:29.027 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.027 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.027 DEBUG:boto:Params: {} 2024-04-16T19:56:29.027 DEBUG:boto:Token: None 2024-04-16T19:56:29.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:36fVBDnmvcfQjFusO6liFXUw71I= 2024-04-16T19:56:29.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:36fVBDnmvcfQjFusO6liFXUw71I=', 'Content-Length': '0'} 2024-04-16T19:56:29.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad632fa9e01ed34-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-16T19:56:29.030 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.030 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.030 DEBUG:boto:Method: GET 2024-04-16T19:56:29.031 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.031 DEBUG:boto:Data: 2024-04-16T19:56:29.031 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.031 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.031 DEBUG:boto:Params: {} 2024-04-16T19:56:29.031 DEBUG:boto:Token: None 2024-04-16T19:56:29.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ISNxb5FSzwirdKBsGs3g4nanmEg= 2024-04-16T19:56:29.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ISNxb5FSzwirdKBsGs3g4nanmEg=', 'Content-Length': '0'} 2024-04-16T19:56:29.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5e57e43041c076-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.078 DEBUG:boto:path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.078 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.079 DEBUG:boto:Method: GET 2024-04-16T19:56:29.079 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.079 DEBUG:boto:Data: 2024-04-16T19:56:29.079 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.079 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.079 DEBUG:boto:Params: {} 2024-04-16T19:56:29.079 DEBUG:boto:Token: None 2024-04-16T19:56:29.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000050 2024-04-16T19:56:29.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ISNxb5FSzwirdKBsGs3g4nanmEg= 2024-04-16T19:56:29.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ISNxb5FSzwirdKBsGs3g4nanmEg=', 'Content-Length': '0'} 2024-04-16T19:56:29.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032ca86a4517318b-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.082 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.082 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.082 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.082 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.082 DEBUG:boto:Method: GET 2024-04-16T19:56:29.082 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.082 DEBUG:boto:Data: 2024-04-16T19:56:29.082 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.082 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.082 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.083 DEBUG:boto:Params: {} 2024-04-16T19:56:29.083 DEBUG:boto:Token: None 2024-04-16T19:56:29.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3a5bd4f1dce71f-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.085 DEBUG:boto:b'yrjefh-51obj000000000000000000501000falseobj000000000000000000502024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.086 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.086 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.086 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.086 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.086 DEBUG:boto:Method: GET 2024-04-16T19:56:29.086 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000050 2024-04-16T19:56:29.086 DEBUG:boto:Data: 2024-04-16T19:56:29.086 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.086 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.086 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.087 DEBUG:boto:Params: {} 2024-04-16T19:56:29.087 DEBUG:boto:Token: None 2024-04-16T19:56:29.087 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e6f3ef919871e7-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.090 DEBUG:boto:b'yrjefh-51obj000000000000000000501000falseobj000000000000000000502024-04-16T19:56:08.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-16T19:56:29.091 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.091 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.091 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.091 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.091 DEBUG:boto:Method: GET 2024-04-16T19:56:29.091 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.091 DEBUG:boto:Data: 2024-04-16T19:56:29.091 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.092 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.092 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.092 DEBUG:boto:Params: {} 2024-04-16T19:56:29.092 DEBUG:boto:Token: None 2024-04-16T19:56:29.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig= 2024-04-16T19:56:29.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig=', 'Content-Length': '0'} 2024-04-16T19:56:29.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f85bfaeeddbb78-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.095 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.095 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.095 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.095 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.095 DEBUG:boto:Method: GET 2024-04-16T19:56:29.095 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.095 DEBUG:boto:Data: 2024-04-16T19:56:29.095 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.095 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.095 DEBUG:boto:Params: {} 2024-04-16T19:56:29.095 DEBUG:boto:Token: None 2024-04-16T19:56:29.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig= 2024-04-16T19:56:29.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig=', 'Content-Length': '0'} 2024-04-16T19:56:29.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df12630a640275d6-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.098 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.098 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.098 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.098 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.099 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.099 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.099 DEBUG:boto:Data: 2024-04-16T19:56:29.099 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.099 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.099 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.099 DEBUG:boto:Params: {} 2024-04-16T19:56:29.099 DEBUG:boto:Token: None 2024-04-16T19:56:29.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GS83//1VYENOJwhMXmNEY1cwmEg= 2024-04-16T19:56:29.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GS83//1VYENOJwhMXmNEY1cwmEg=', 'Content-Length': '0'} 2024-04-16T19:56:29.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e42b4c9e1ce0e44-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.101 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.101 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.101 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.101 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.102 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.102 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.102 DEBUG:boto:Data: 2024-04-16T19:56:29.102 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.102 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.102 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.102 DEBUG:boto:Params: {} 2024-04-16T19:56:29.102 DEBUG:boto:Token: None 2024-04-16T19:56:29.102 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GS83//1VYENOJwhMXmNEY1cwmEg= 2024-04-16T19:56:29.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GS83//1VYENOJwhMXmNEY1cwmEg=', 'Content-Length': '0'} 2024-04-16T19:56:29.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db36ae4e5903ff89-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-16T19:56:29.105 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.105 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.105 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.105 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.105 DEBUG:boto:Method: GET 2024-04-16T19:56:29.105 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.105 DEBUG:boto:Data: 2024-04-16T19:56:29.105 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.105 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.105 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.106 DEBUG:boto:Params: {} 2024-04-16T19:56:29.106 DEBUG:boto:Token: None 2024-04-16T19:56:29.106 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig= 2024-04-16T19:56:29.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig=', 'Content-Length': '0'} 2024-04-16T19:56:29.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3de9066a23d9af0-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.108 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.108 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.108 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.108 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.108 DEBUG:boto:Method: GET 2024-04-16T19:56:29.108 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.109 DEBUG:boto:Data: 2024-04-16T19:56:29.109 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.109 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.109 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.109 DEBUG:boto:Params: {} 2024-04-16T19:56:29.109 DEBUG:boto:Token: None 2024-04-16T19:56:29.109 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053?versionId=null 2024-04-16T19:56:29.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig= 2024-04-16T19:56:29.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjAPqc14FFfR/sCEQniOwNSNCig=', 'Content-Length': '0'} 2024-04-16T19:56:29.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97ef85577cb4adb-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.111 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.111 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.112 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.112 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.112 DEBUG:boto:Method: GET 2024-04-16T19:56:29.112 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.112 DEBUG:boto:Data: 2024-04-16T19:56:29.112 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.112 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.112 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.112 DEBUG:boto:Params: {} 2024-04-16T19:56:29.112 DEBUG:boto:Token: None 2024-04-16T19:56:29.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b7471fb0605796-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.115 DEBUG:boto:b'yrjefh-51obj000000000000000000531000falseobj000000000000000000532024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.116 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.116 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.116 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.116 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.116 DEBUG:boto:Method: GET 2024-04-16T19:56:29.116 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.116 DEBUG:boto:Data: 2024-04-16T19:56:29.116 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.116 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.116 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.116 DEBUG:boto:Params: {} 2024-04-16T19:56:29.117 DEBUG:boto:Token: None 2024-04-16T19:56:29.117 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c52250c98f2fb9-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.121 DEBUG:boto:b'yrjefh-51obj000000000000000000531000falseobj000000000000000000532024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.121 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.121 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.121 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.121 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.121 DEBUG:boto:Data: 2024-04-16T19:56:29.121 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.122 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.122 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.122 DEBUG:boto:Params: {} 2024-04-16T19:56:29.122 DEBUG:boto:Token: None 2024-04-16T19:56:29.122 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.122 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XSoSjwnitu4bb59cHLvJp6nvT08= 2024-04-16T19:56:29.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XSoSjwnitu4bb59cHLvJp6nvT08=', 'Content-Length': '0'} 2024-04-16T19:56:29.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e173c6666a0144df-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.124 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.124 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.124 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.124 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.125 DEBUG:boto:Data: 2024-04-16T19:56:29.125 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.125 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.125 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.125 DEBUG:boto:Params: {} 2024-04-16T19:56:29.125 DEBUG:boto:Token: None 2024-04-16T19:56:29.125 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.125 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XSoSjwnitu4bb59cHLvJp6nvT08= 2024-04-16T19:56:29.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XSoSjwnitu4bb59cHLvJp6nvT08=', 'Content-Length': '0'} 2024-04-16T19:56:29.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068697c3d2912d3f9-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-16T19:56:29.127 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.127 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.127 DEBUG:boto:Method: GET 2024-04-16T19:56:29.127 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.128 DEBUG:boto:Data: 2024-04-16T19:56:29.128 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.128 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.128 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.128 DEBUG:boto:Params: {} 2024-04-16T19:56:29.128 DEBUG:boto:Token: None 2024-04-16T19:56:29.128 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GQB44hLMO+CGsygt8g4Gj/lhDXI= 2024-04-16T19:56:29.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GQB44hLMO+CGsygt8g4Gj/lhDXI=', 'Content-Length': '0'} 2024-04-16T19:56:29.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0753752bd349ba8-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.170 DEBUG:boto:path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.170 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.171 DEBUG:boto:Method: GET 2024-04-16T19:56:29.171 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.171 DEBUG:boto:Data: 2024-04-16T19:56:29.171 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.171 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.171 DEBUG:boto:Params: {} 2024-04-16T19:56:29.171 DEBUG:boto:Token: None 2024-04-16T19:56:29.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000053 2024-04-16T19:56:29.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GQB44hLMO+CGsygt8g4Gj/lhDXI= 2024-04-16T19:56:29.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GQB44hLMO+CGsygt8g4Gj/lhDXI=', 'Content-Length': '0'} 2024-04-16T19:56:29.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f98ad3842e0860-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.174 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.174 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.174 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.174 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.174 DEBUG:boto:Method: GET 2024-04-16T19:56:29.174 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.174 DEBUG:boto:Data: 2024-04-16T19:56:29.174 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.174 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.174 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.174 DEBUG:boto:Params: {} 2024-04-16T19:56:29.175 DEBUG:boto:Token: None 2024-04-16T19:56:29.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bce2e4fcb77a56-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.177 DEBUG:boto:b'yrjefh-51obj000000000000000000531000falseobj000000000000000000532024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.178 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.178 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.178 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.178 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.178 DEBUG:boto:Method: GET 2024-04-16T19:56:29.178 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000053 2024-04-16T19:56:29.178 DEBUG:boto:Data: 2024-04-16T19:56:29.178 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.178 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.178 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.178 DEBUG:boto:Params: {} 2024-04-16T19:56:29.179 DEBUG:boto:Token: None 2024-04-16T19:56:29.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a35a71d680e649-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.183 DEBUG:boto:b'yrjefh-51obj000000000000000000531000falseobj000000000000000000532024-04-16T19:56:08.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-16T19:56:29.183 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.183 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.183 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.183 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.183 DEBUG:boto:Method: GET 2024-04-16T19:56:29.184 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.184 DEBUG:boto:Data: 2024-04-16T19:56:29.184 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.184 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.184 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.184 DEBUG:boto:Params: {} 2024-04-16T19:56:29.184 DEBUG:boto:Token: None 2024-04-16T19:56:29.184 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE= 2024-04-16T19:56:29.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE=', 'Content-Length': '0'} 2024-04-16T19:56:29.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34fcab569cee234-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.187 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.187 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.187 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.187 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.187 DEBUG:boto:Method: GET 2024-04-16T19:56:29.187 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.187 DEBUG:boto:Data: 2024-04-16T19:56:29.187 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.188 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.188 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.188 DEBUG:boto:Params: {} 2024-04-16T19:56:29.188 DEBUG:boto:Token: None 2024-04-16T19:56:29.188 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE= 2024-04-16T19:56:29.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE=', 'Content-Length': '0'} 2024-04-16T19:56:29.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6681224fb61fc94-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.190 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.190 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.190 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.191 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.191 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.191 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.191 DEBUG:boto:Data: 2024-04-16T19:56:29.191 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.191 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.191 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.191 DEBUG:boto:Params: {} 2024-04-16T19:56:29.191 DEBUG:boto:Token: None 2024-04-16T19:56:29.191 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QygxLQJH+RvNSCewrQjJvMIVmYE= 2024-04-16T19:56:29.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QygxLQJH+RvNSCewrQjJvMIVmYE=', 'Content-Length': '0'} 2024-04-16T19:56:29.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd0eebb28906bf0-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.194 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.194 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.194 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.194 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.194 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.194 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.194 DEBUG:boto:Data: 2024-04-16T19:56:29.194 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.194 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.194 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.195 DEBUG:boto:Params: {} 2024-04-16T19:56:29.195 DEBUG:boto:Token: None 2024-04-16T19:56:29.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QygxLQJH+RvNSCewrQjJvMIVmYE= 2024-04-16T19:56:29.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QygxLQJH+RvNSCewrQjJvMIVmYE=', 'Content-Length': '0'} 2024-04-16T19:56:29.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f67160a54142065-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-16T19:56:29.197 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.197 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.197 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.197 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.198 DEBUG:boto:Method: GET 2024-04-16T19:56:29.198 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.198 DEBUG:boto:Data: 2024-04-16T19:56:29.198 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.198 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.198 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.198 DEBUG:boto:Params: {} 2024-04-16T19:56:29.198 DEBUG:boto:Token: None 2024-04-16T19:56:29.198 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.198 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE= 2024-04-16T19:56:29.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE=', 'Content-Length': '0'} 2024-04-16T19:56:29.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a06b8bd2e7a5d415-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.201 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.201 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.201 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.201 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.201 DEBUG:boto:Method: GET 2024-04-16T19:56:29.201 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.202 DEBUG:boto:Data: 2024-04-16T19:56:29.202 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.202 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.202 DEBUG:boto:Params: {} 2024-04-16T19:56:29.202 DEBUG:boto:Token: None 2024-04-16T19:56:29.202 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055?versionId=null 2024-04-16T19:56:29.202 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE= 2024-04-16T19:56:29.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ogk2yNr+aZ+EPoiTl7S3zyhigE=', 'Content-Length': '0'} 2024-04-16T19:56:29.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca08126a8801f8a-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.205 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.205 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.205 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.205 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.205 DEBUG:boto:Method: GET 2024-04-16T19:56:29.205 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.205 DEBUG:boto:Data: 2024-04-16T19:56:29.205 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.205 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.206 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.206 DEBUG:boto:Params: {} 2024-04-16T19:56:29.206 DEBUG:boto:Token: None 2024-04-16T19:56:29.206 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.206 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e304eccd277623-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.209 DEBUG:boto:b'yrjefh-51obj000000000000000000551000falseobj000000000000000000552024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.209 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.209 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.209 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.209 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.209 DEBUG:boto:Method: GET 2024-04-16T19:56:29.209 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.209 DEBUG:boto:Data: 2024-04-16T19:56:29.210 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.210 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.210 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.210 DEBUG:boto:Params: {} 2024-04-16T19:56:29.210 DEBUG:boto:Token: None 2024-04-16T19:56:29.210 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.210 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf977ac2059e2780-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.214 DEBUG:boto:b'yrjefh-51obj000000000000000000551000falseobj000000000000000000552024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.215 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.215 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.215 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.215 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.215 DEBUG:boto:Data: 2024-04-16T19:56:29.215 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.215 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.215 DEBUG:boto:Params: {} 2024-04-16T19:56:29.215 DEBUG:boto:Token: None 2024-04-16T19:56:29.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T797jvOccTtXvRTxhUxztdGZeTM= 2024-04-16T19:56:29.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T797jvOccTtXvRTxhUxztdGZeTM=', 'Content-Length': '0'} 2024-04-16T19:56:29.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca7f16e19196f9d-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.218 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.218 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.218 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.218 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.218 DEBUG:boto:Data: 2024-04-16T19:56:29.218 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.218 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.219 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.219 DEBUG:boto:Params: {} 2024-04-16T19:56:29.219 DEBUG:boto:Token: None 2024-04-16T19:56:29.219 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T797jvOccTtXvRTxhUxztdGZeTM= 2024-04-16T19:56:29.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T797jvOccTtXvRTxhUxztdGZeTM=', 'Content-Length': '0'} 2024-04-16T19:56:29.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006185a9f6969df107-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-16T19:56:29.221 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.221 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.221 DEBUG:boto:Method: GET 2024-04-16T19:56:29.221 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.222 DEBUG:boto:Data: 2024-04-16T19:56:29.222 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.222 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.222 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.222 DEBUG:boto:Params: {} 2024-04-16T19:56:29.222 DEBUG:boto:Token: None 2024-04-16T19:56:29.222 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.222 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pkEgTdDpJQyRBij75gWDxXI39Ww= 2024-04-16T19:56:29.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pkEgTdDpJQyRBij75gWDxXI39Ww=', 'Content-Length': '0'} 2024-04-16T19:56:29.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c10f1b50895b6d49-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.225 DEBUG:boto:path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.225 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.225 DEBUG:boto:Method: GET 2024-04-16T19:56:29.225 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.225 DEBUG:boto:Data: 2024-04-16T19:56:29.225 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.225 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.225 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.225 DEBUG:boto:Params: {} 2024-04-16T19:56:29.226 DEBUG:boto:Token: None 2024-04-16T19:56:29.226 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000055 2024-04-16T19:56:29.226 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pkEgTdDpJQyRBij75gWDxXI39Ww= 2024-04-16T19:56:29.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pkEgTdDpJQyRBij75gWDxXI39Ww=', 'Content-Length': '0'} 2024-04-16T19:56:29.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4270ff610794bae-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.228 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.228 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.229 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.229 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.229 DEBUG:boto:Method: GET 2024-04-16T19:56:29.229 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.229 DEBUG:boto:Data: 2024-04-16T19:56:29.229 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.229 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.229 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.229 DEBUG:boto:Params: {} 2024-04-16T19:56:29.229 DEBUG:boto:Token: None 2024-04-16T19:56:29.230 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3d9ee652219966-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.233 DEBUG:boto:b'yrjefh-51obj000000000000000000551000falseobj000000000000000000552024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.233 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.233 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.233 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.233 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.233 DEBUG:boto:Method: GET 2024-04-16T19:56:29.233 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000055 2024-04-16T19:56:29.234 DEBUG:boto:Data: 2024-04-16T19:56:29.234 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.234 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.234 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.234 DEBUG:boto:Params: {} 2024-04-16T19:56:29.234 DEBUG:boto:Token: None 2024-04-16T19:56:29.234 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.234 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea9ebff70e2a984-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.238 DEBUG:boto:b'yrjefh-51obj000000000000000000551000falseobj000000000000000000552024-04-16T19:56:05.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-16T19:56:29.238 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.238 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.238 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.239 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.239 DEBUG:boto:Method: GET 2024-04-16T19:56:29.239 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.239 DEBUG:boto:Data: 2024-04-16T19:56:29.239 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.239 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.239 DEBUG:boto:Params: {} 2024-04-16T19:56:29.239 DEBUG:boto:Token: None 2024-04-16T19:56:29.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw= 2024-04-16T19:56:29.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000938ac343a80a5ee4-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.242 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.242 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.242 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.242 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.242 DEBUG:boto:Method: GET 2024-04-16T19:56:29.242 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.242 DEBUG:boto:Data: 2024-04-16T19:56:29.243 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.243 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.243 DEBUG:boto:Params: {} 2024-04-16T19:56:29.243 DEBUG:boto:Token: None 2024-04-16T19:56:29.243 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw= 2024-04-16T19:56:29.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab3f2fc2d5d8aa8-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.245 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.245 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.245 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.246 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.246 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.246 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.246 DEBUG:boto:Data: 2024-04-16T19:56:29.246 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.246 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.246 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.246 DEBUG:boto:Params: {} 2024-04-16T19:56:29.246 DEBUG:boto:Token: None 2024-04-16T19:56:29.246 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tijzhovK0KTqR51uerwHcHm4KyA= 2024-04-16T19:56:29.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tijzhovK0KTqR51uerwHcHm4KyA=', 'Content-Length': '0'} 2024-04-16T19:56:29.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a09d5c1af3a7cd9-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.249 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.249 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.249 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.249 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.249 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.249 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.249 DEBUG:boto:Data: 2024-04-16T19:56:29.249 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.250 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.250 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.250 DEBUG:boto:Params: {} 2024-04-16T19:56:29.250 DEBUG:boto:Token: None 2024-04-16T19:56:29.250 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tijzhovK0KTqR51uerwHcHm4KyA= 2024-04-16T19:56:29.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tijzhovK0KTqR51uerwHcHm4KyA=', 'Content-Length': '0'} 2024-04-16T19:56:29.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d08f8a49bd4be8a-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-16T19:56:29.252 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.252 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.252 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.252 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.252 DEBUG:boto:Method: GET 2024-04-16T19:56:29.253 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.253 DEBUG:boto:Data: 2024-04-16T19:56:29.253 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.253 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.253 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.253 DEBUG:boto:Params: {} 2024-04-16T19:56:29.253 DEBUG:boto:Token: None 2024-04-16T19:56:29.253 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.253 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw= 2024-04-16T19:56:29.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195f31d38c03fd4f-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.255 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.255 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.256 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.256 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.256 DEBUG:boto:Method: GET 2024-04-16T19:56:29.256 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.256 DEBUG:boto:Data: 2024-04-16T19:56:29.256 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.256 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.256 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.256 DEBUG:boto:Params: {} 2024-04-16T19:56:29.256 DEBUG:boto:Token: None 2024-04-16T19:56:29.256 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056?versionId=null 2024-04-16T19:56:29.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw= 2024-04-16T19:56:29.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JA3fXdmVuWjNJFYAqUQSNUfRdCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c5d6f38a321529-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.258 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.259 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.259 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.259 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.259 DEBUG:boto:Method: GET 2024-04-16T19:56:29.259 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.259 DEBUG:boto:Data: 2024-04-16T19:56:29.259 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.259 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.259 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.259 DEBUG:boto:Params: {} 2024-04-16T19:56:29.259 DEBUG:boto:Token: None 2024-04-16T19:56:29.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e690ee6cd004dae-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.302 DEBUG:boto:b'yrjefh-51obj000000000000000000561000falseobj000000000000000000562024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.303 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.303 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.303 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.303 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.303 DEBUG:boto:Method: GET 2024-04-16T19:56:29.303 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.303 DEBUG:boto:Data: 2024-04-16T19:56:29.303 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.303 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.303 DEBUG:boto:Params: {} 2024-04-16T19:56:29.304 DEBUG:boto:Token: None 2024-04-16T19:56:29.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007617c885ce7e4a1c-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.308 DEBUG:boto:b'yrjefh-51obj000000000000000000561000falseobj000000000000000000562024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.308 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.308 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.308 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.308 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.308 DEBUG:boto:Data: 2024-04-16T19:56:29.308 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.309 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.309 DEBUG:boto:Params: {} 2024-04-16T19:56:29.309 DEBUG:boto:Token: None 2024-04-16T19:56:29.309 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oexTGB7wv48rUT8nciXj5JP8IjM= 2024-04-16T19:56:29.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oexTGB7wv48rUT8nciXj5JP8IjM=', 'Content-Length': '0'} 2024-04-16T19:56:29.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de8a55311dba018-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.311 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.311 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.311 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.311 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.311 DEBUG:boto:Data: 2024-04-16T19:56:29.312 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.312 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.312 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.312 DEBUG:boto:Params: {} 2024-04-16T19:56:29.312 DEBUG:boto:Token: None 2024-04-16T19:56:29.312 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oexTGB7wv48rUT8nciXj5JP8IjM= 2024-04-16T19:56:29.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oexTGB7wv48rUT8nciXj5JP8IjM=', 'Content-Length': '0'} 2024-04-16T19:56:29.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b885dd4ed6f4304c-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-16T19:56:29.314 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.314 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.314 DEBUG:boto:Method: GET 2024-04-16T19:56:29.314 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.315 DEBUG:boto:Data: 2024-04-16T19:56:29.315 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.315 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.315 DEBUG:boto:Params: {} 2024-04-16T19:56:29.315 DEBUG:boto:Token: None 2024-04-16T19:56:29.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z1ESvnHelEK5E43M29Dw5nM0eE4= 2024-04-16T19:56:29.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z1ESvnHelEK5E43M29Dw5nM0eE4=', 'Content-Length': '0'} 2024-04-16T19:56:29.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000386ffd639f0a2818-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.318 DEBUG:boto:path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.318 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.318 DEBUG:boto:Method: GET 2024-04-16T19:56:29.318 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.318 DEBUG:boto:Data: 2024-04-16T19:56:29.318 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.318 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.318 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.318 DEBUG:boto:Params: {} 2024-04-16T19:56:29.318 DEBUG:boto:Token: None 2024-04-16T19:56:29.318 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000056 2024-04-16T19:56:29.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z1ESvnHelEK5E43M29Dw5nM0eE4= 2024-04-16T19:56:29.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z1ESvnHelEK5E43M29Dw5nM0eE4=', 'Content-Length': '0'} 2024-04-16T19:56:29.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e21d75efefe15fcc-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.321 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.321 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.321 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.321 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.321 DEBUG:boto:Method: GET 2024-04-16T19:56:29.321 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.321 DEBUG:boto:Data: 2024-04-16T19:56:29.321 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.321 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.321 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.322 DEBUG:boto:Params: {} 2024-04-16T19:56:29.322 DEBUG:boto:Token: None 2024-04-16T19:56:29.322 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.322 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0981788c03cc38-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.325 DEBUG:boto:b'yrjefh-51obj000000000000000000561000falseobj000000000000000000562024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.325 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.325 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.325 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.325 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.325 DEBUG:boto:Method: GET 2024-04-16T19:56:29.325 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000056 2024-04-16T19:56:29.326 DEBUG:boto:Data: 2024-04-16T19:56:29.326 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.326 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.326 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.326 DEBUG:boto:Params: {} 2024-04-16T19:56:29.326 DEBUG:boto:Token: None 2024-04-16T19:56:29.326 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.326 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2bad2d9e411571-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.330 DEBUG:boto:b'yrjefh-51obj000000000000000000561000falseobj000000000000000000562024-04-16T19:56:07.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-16T19:56:29.330 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.330 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.330 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.331 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.331 DEBUG:boto:Method: GET 2024-04-16T19:56:29.331 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.331 DEBUG:boto:Data: 2024-04-16T19:56:29.331 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.331 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.331 DEBUG:boto:Params: {} 2024-04-16T19:56:29.331 DEBUG:boto:Token: None 2024-04-16T19:56:29.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo= 2024-04-16T19:56:29.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo=', 'Content-Length': '0'} 2024-04-16T19:56:29.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b83d81aebb3ad45-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.334 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.334 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.334 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.334 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.334 DEBUG:boto:Method: GET 2024-04-16T19:56:29.335 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.335 DEBUG:boto:Data: 2024-04-16T19:56:29.335 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.335 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.335 DEBUG:boto:Params: {} 2024-04-16T19:56:29.335 DEBUG:boto:Token: None 2024-04-16T19:56:29.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo= 2024-04-16T19:56:29.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo=', 'Content-Length': '0'} 2024-04-16T19:56:29.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09a465bde9ce177-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.337 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.338 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.338 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.338 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.338 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.338 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.338 DEBUG:boto:Data: 2024-04-16T19:56:29.338 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.338 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.338 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.338 DEBUG:boto:Params: {} 2024-04-16T19:56:29.338 DEBUG:boto:Token: None 2024-04-16T19:56:29.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4VUo3kybTaJE8w0vu2YFF8oNmCw= 2024-04-16T19:56:29.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4VUo3kybTaJE8w0vu2YFF8oNmCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4337d7d901cd884-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.341 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.341 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.341 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.341 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.341 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.341 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.341 DEBUG:boto:Data: 2024-04-16T19:56:29.341 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.341 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.341 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.341 DEBUG:boto:Params: {} 2024-04-16T19:56:29.342 DEBUG:boto:Token: None 2024-04-16T19:56:29.342 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4VUo3kybTaJE8w0vu2YFF8oNmCw= 2024-04-16T19:56:29.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4VUo3kybTaJE8w0vu2YFF8oNmCw=', 'Content-Length': '0'} 2024-04-16T19:56:29.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005883260c3262c680-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-16T19:56:29.344 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.344 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.344 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.344 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.344 DEBUG:boto:Method: GET 2024-04-16T19:56:29.345 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.345 DEBUG:boto:Data: 2024-04-16T19:56:29.345 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.345 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.345 DEBUG:boto:Params: {} 2024-04-16T19:56:29.345 DEBUG:boto:Token: None 2024-04-16T19:56:29.345 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.345 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo= 2024-04-16T19:56:29.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo=', 'Content-Length': '0'} 2024-04-16T19:56:29.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e3a5dd75ae529eb-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.347 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.347 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.347 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.348 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.348 DEBUG:boto:Method: GET 2024-04-16T19:56:29.348 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.348 DEBUG:boto:Data: 2024-04-16T19:56:29.348 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.348 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.348 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.348 DEBUG:boto:Params: {} 2024-04-16T19:56:29.348 DEBUG:boto:Token: None 2024-04-16T19:56:29.348 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057?versionId=null 2024-04-16T19:56:29.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo= 2024-04-16T19:56:29.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ki14YG8fdZj0+v/gmlCfnFgXJOo=', 'Content-Length': '0'} 2024-04-16T19:56:29.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b902921fce39bd-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.350 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.350 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.351 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.351 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.351 DEBUG:boto:Method: GET 2024-04-16T19:56:29.351 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.351 DEBUG:boto:Data: 2024-04-16T19:56:29.351 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.351 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.351 DEBUG:boto:Params: {} 2024-04-16T19:56:29.351 DEBUG:boto:Token: None 2024-04-16T19:56:29.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f53f218032683e57-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.398 DEBUG:boto:b'yrjefh-51obj000000000000000000571000falseobj000000000000000000572024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.399 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.399 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.399 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.399 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.399 DEBUG:boto:Method: GET 2024-04-16T19:56:29.399 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.399 DEBUG:boto:Data: 2024-04-16T19:56:29.399 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.399 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.400 DEBUG:boto:Params: {} 2024-04-16T19:56:29.400 DEBUG:boto:Token: None 2024-04-16T19:56:29.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ebd9b13095dcb1-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.404 DEBUG:boto:b'yrjefh-51obj000000000000000000571000falseobj000000000000000000572024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.405 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.405 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.405 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.405 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.405 DEBUG:boto:Data: 2024-04-16T19:56:29.405 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.405 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.405 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.405 DEBUG:boto:Params: {} 2024-04-16T19:56:29.406 DEBUG:boto:Token: None 2024-04-16T19:56:29.406 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.406 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N5sRw8RXzH+5vm1AP4J9lC5I4zk= 2024-04-16T19:56:29.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N5sRw8RXzH+5vm1AP4J9lC5I4zk=', 'Content-Length': '0'} 2024-04-16T19:56:29.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eeb0d9177e42464-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.408 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.408 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.408 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.408 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.408 DEBUG:boto:Data: 2024-04-16T19:56:29.408 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.409 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.409 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.409 DEBUG:boto:Params: {} 2024-04-16T19:56:29.409 DEBUG:boto:Token: None 2024-04-16T19:56:29.409 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N5sRw8RXzH+5vm1AP4J9lC5I4zk= 2024-04-16T19:56:29.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N5sRw8RXzH+5vm1AP4J9lC5I4zk=', 'Content-Length': '0'} 2024-04-16T19:56:29.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9804a7a256c560-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-16T19:56:29.411 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.411 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.411 DEBUG:boto:Method: GET 2024-04-16T19:56:29.412 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.412 DEBUG:boto:Data: 2024-04-16T19:56:29.412 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.412 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.412 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.412 DEBUG:boto:Params: {} 2024-04-16T19:56:29.412 DEBUG:boto:Token: None 2024-04-16T19:56:29.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O8r67SoE7IO1OCFx5nCt14cC+Wk= 2024-04-16T19:56:29.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O8r67SoE7IO1OCFx5nCt14cC+Wk=', 'Content-Length': '0'} 2024-04-16T19:56:29.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7cc7c7984d19576-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.415 DEBUG:boto:path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.415 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.415 DEBUG:boto:Method: GET 2024-04-16T19:56:29.415 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.415 DEBUG:boto:Data: 2024-04-16T19:56:29.415 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.415 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.415 DEBUG:boto:Params: {} 2024-04-16T19:56:29.415 DEBUG:boto:Token: None 2024-04-16T19:56:29.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000057 2024-04-16T19:56:29.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O8r67SoE7IO1OCFx5nCt14cC+Wk= 2024-04-16T19:56:29.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O8r67SoE7IO1OCFx5nCt14cC+Wk=', 'Content-Length': '0'} 2024-04-16T19:56:29.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed6809a6dd30d19-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.462 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.463 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.463 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.463 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.463 DEBUG:boto:Method: GET 2024-04-16T19:56:29.463 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.463 DEBUG:boto:Data: 2024-04-16T19:56:29.463 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.463 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.463 DEBUG:boto:Params: {} 2024-04-16T19:56:29.463 DEBUG:boto:Token: None 2024-04-16T19:56:29.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056203a23e3129958-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.466 DEBUG:boto:b'yrjefh-51obj000000000000000000571000falseobj000000000000000000572024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.467 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.467 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.467 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.467 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.467 DEBUG:boto:Method: GET 2024-04-16T19:56:29.467 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000057 2024-04-16T19:56:29.467 DEBUG:boto:Data: 2024-04-16T19:56:29.467 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.467 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.467 DEBUG:boto:Params: {} 2024-04-16T19:56:29.468 DEBUG:boto:Token: None 2024-04-16T19:56:29.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b46994e430dba7-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.472 DEBUG:boto:b'yrjefh-51obj000000000000000000571000falseobj000000000000000000572024-04-16T19:56:09.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-16T19:56:29.472 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.472 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.472 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.472 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.472 DEBUG:boto:Method: GET 2024-04-16T19:56:29.472 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.473 DEBUG:boto:Data: 2024-04-16T19:56:29.473 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.473 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.473 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.473 DEBUG:boto:Params: {} 2024-04-16T19:56:29.473 DEBUG:boto:Token: None 2024-04-16T19:56:29.473 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc= 2024-04-16T19:56:29.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc=', 'Content-Length': '0'} 2024-04-16T19:56:29.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dff739203fbd0ae-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.476 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.476 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.476 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.476 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.476 DEBUG:boto:Method: GET 2024-04-16T19:56:29.477 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.477 DEBUG:boto:Data: 2024-04-16T19:56:29.477 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.477 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.477 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.477 DEBUG:boto:Params: {} 2024-04-16T19:56:29.477 DEBUG:boto:Token: None 2024-04-16T19:56:29.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc= 2024-04-16T19:56:29.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc=', 'Content-Length': '0'} 2024-04-16T19:56:29.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3641d83564457a-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.479 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.480 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.480 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.480 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.480 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.480 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.480 DEBUG:boto:Data: 2024-04-16T19:56:29.480 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.480 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.480 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.480 DEBUG:boto:Params: {} 2024-04-16T19:56:29.481 DEBUG:boto:Token: None 2024-04-16T19:56:29.481 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3PyQDgqZkcmN7WqwDQpiR/OPdo= 2024-04-16T19:56:29.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3PyQDgqZkcmN7WqwDQpiR/OPdo=', 'Content-Length': '0'} 2024-04-16T19:56:29.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044755c34d81cd17d-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.484 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.484 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.484 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.484 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.484 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.484 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.484 DEBUG:boto:Data: 2024-04-16T19:56:29.484 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.484 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.484 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.484 DEBUG:boto:Params: {} 2024-04-16T19:56:29.484 DEBUG:boto:Token: None 2024-04-16T19:56:29.485 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.485 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3PyQDgqZkcmN7WqwDQpiR/OPdo= 2024-04-16T19:56:29.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3PyQDgqZkcmN7WqwDQpiR/OPdo=', 'Content-Length': '0'} 2024-04-16T19:56:29.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d3997598b1c103-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-16T19:56:29.487 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.487 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.487 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.487 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.488 DEBUG:boto:Method: GET 2024-04-16T19:56:29.488 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.488 DEBUG:boto:Data: 2024-04-16T19:56:29.488 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.488 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.488 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.488 DEBUG:boto:Params: {} 2024-04-16T19:56:29.488 DEBUG:boto:Token: None 2024-04-16T19:56:29.488 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc= 2024-04-16T19:56:29.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc=', 'Content-Length': '0'} 2024-04-16T19:56:29.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7cdd930b7a84903-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.534 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.534 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.535 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.535 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.535 DEBUG:boto:Method: GET 2024-04-16T19:56:29.535 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.535 DEBUG:boto:Data: 2024-04-16T19:56:29.535 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.535 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.535 DEBUG:boto:Params: {} 2024-04-16T19:56:29.535 DEBUG:boto:Token: None 2024-04-16T19:56:29.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058?versionId=null 2024-04-16T19:56:29.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc= 2024-04-16T19:56:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qGlwJbxwSomgbPDioUbmVYb5rnc=', 'Content-Length': '0'} 2024-04-16T19:56:29.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5ee580a2e06d93-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.538 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.538 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.538 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.538 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.538 DEBUG:boto:Method: GET 2024-04-16T19:56:29.539 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.539 DEBUG:boto:Data: 2024-04-16T19:56:29.539 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.539 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.539 DEBUG:boto:Params: {} 2024-04-16T19:56:29.539 DEBUG:boto:Token: None 2024-04-16T19:56:29.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f1493696363f74d-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.542 DEBUG:boto:b'yrjefh-51obj000000000000000000581000falseobj000000000000000000582024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.543 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.543 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.543 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.543 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.543 DEBUG:boto:Method: GET 2024-04-16T19:56:29.543 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.543 DEBUG:boto:Data: 2024-04-16T19:56:29.543 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.543 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.543 DEBUG:boto:Params: {} 2024-04-16T19:56:29.544 DEBUG:boto:Token: None 2024-04-16T19:56:29.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009fb8c491b8033fd-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.548 DEBUG:boto:b'yrjefh-51obj000000000000000000581000falseobj000000000000000000582024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.548 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.548 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.549 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.549 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.549 DEBUG:boto:Data: 2024-04-16T19:56:29.549 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.549 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.549 DEBUG:boto:Params: {} 2024-04-16T19:56:29.549 DEBUG:boto:Token: None 2024-04-16T19:56:29.549 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jQsy5K6Dm9+5NfKbWCwq3tvPH8g= 2024-04-16T19:56:29.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jQsy5K6Dm9+5NfKbWCwq3tvPH8g=', 'Content-Length': '0'} 2024-04-16T19:56:29.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097536cfb6ee51d4-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.552 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.552 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.552 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.552 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.552 DEBUG:boto:Data: 2024-04-16T19:56:29.552 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.552 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.552 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.552 DEBUG:boto:Params: {} 2024-04-16T19:56:29.553 DEBUG:boto:Token: None 2024-04-16T19:56:29.553 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jQsy5K6Dm9+5NfKbWCwq3tvPH8g= 2024-04-16T19:56:29.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jQsy5K6Dm9+5NfKbWCwq3tvPH8g=', 'Content-Length': '0'} 2024-04-16T19:56:29.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf321205e3b82cd-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-16T19:56:29.555 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.555 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.555 DEBUG:boto:Method: GET 2024-04-16T19:56:29.555 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.555 DEBUG:boto:Data: 2024-04-16T19:56:29.556 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.556 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.556 DEBUG:boto:Params: {} 2024-04-16T19:56:29.556 DEBUG:boto:Token: None 2024-04-16T19:56:29.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F7SZU/m+84ZhKWZgELX98N09UDM= 2024-04-16T19:56:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F7SZU/m+84ZhKWZgELX98N09UDM=', 'Content-Length': '0'} 2024-04-16T19:56:29.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916dbb90b0b589fb-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.603 DEBUG:boto:Method: GET 2024-04-16T19:56:29.603 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.603 DEBUG:boto:Data: 2024-04-16T19:56:29.603 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.603 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.603 DEBUG:boto:Params: {} 2024-04-16T19:56:29.603 DEBUG:boto:Token: None 2024-04-16T19:56:29.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000058 2024-04-16T19:56:29.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F7SZU/m+84ZhKWZgELX98N09UDM= 2024-04-16T19:56:29.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F7SZU/m+84ZhKWZgELX98N09UDM=', 'Content-Length': '0'} 2024-04-16T19:56:29.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444ae1120ed291d9-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.606 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.606 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.606 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.606 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.606 DEBUG:boto:Method: GET 2024-04-16T19:56:29.606 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.606 DEBUG:boto:Data: 2024-04-16T19:56:29.606 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.606 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.606 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.607 DEBUG:boto:Params: {} 2024-04-16T19:56:29.607 DEBUG:boto:Token: None 2024-04-16T19:56:29.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9aa8417cce8879-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.610 DEBUG:boto:b'yrjefh-51obj000000000000000000581000falseobj000000000000000000582024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.610 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.610 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.610 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.610 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.610 DEBUG:boto:Method: GET 2024-04-16T19:56:29.611 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000058 2024-04-16T19:56:29.611 DEBUG:boto:Data: 2024-04-16T19:56:29.611 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.611 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.611 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.611 DEBUG:boto:Params: {} 2024-04-16T19:56:29.611 DEBUG:boto:Token: None 2024-04-16T19:56:29.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70ce725e405fb86-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.615 DEBUG:boto:b'yrjefh-51obj000000000000000000581000falseobj000000000000000000582024-04-16T19:56:05.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-16T19:56:29.616 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.616 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.616 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.616 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.616 DEBUG:boto:Method: GET 2024-04-16T19:56:29.616 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.616 DEBUG:boto:Data: 2024-04-16T19:56:29.616 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.616 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.616 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.616 DEBUG:boto:Params: {} 2024-04-16T19:56:29.617 DEBUG:boto:Token: None 2024-04-16T19:56:29.617 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.617 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY= 2024-04-16T19:56:29.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY=', 'Content-Length': '0'} 2024-04-16T19:56:29.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a2e50620486214d-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.619 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.619 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.619 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.619 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.619 DEBUG:boto:Method: GET 2024-04-16T19:56:29.620 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.620 DEBUG:boto:Data: 2024-04-16T19:56:29.620 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.620 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.620 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.620 DEBUG:boto:Params: {} 2024-04-16T19:56:29.620 DEBUG:boto:Token: None 2024-04-16T19:56:29.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY= 2024-04-16T19:56:29.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY=', 'Content-Length': '0'} 2024-04-16T19:56:29.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d772950dec1872-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.622 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.623 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.623 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.623 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.623 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.623 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.623 DEBUG:boto:Data: 2024-04-16T19:56:29.623 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.623 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.623 DEBUG:boto:Params: {} 2024-04-16T19:56:29.623 DEBUG:boto:Token: None 2024-04-16T19:56:29.624 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:quhLubLOWb+6DIVNaiOypTVOrBc= 2024-04-16T19:56:29.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:quhLubLOWb+6DIVNaiOypTVOrBc=', 'Content-Length': '0'} 2024-04-16T19:56:29.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b3154b3dac93f1-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.626 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.626 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.626 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.626 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.626 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.626 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.626 DEBUG:boto:Data: 2024-04-16T19:56:29.626 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.626 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.626 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.627 DEBUG:boto:Params: {} 2024-04-16T19:56:29.627 DEBUG:boto:Token: None 2024-04-16T19:56:29.627 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:quhLubLOWb+6DIVNaiOypTVOrBc= 2024-04-16T19:56:29.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:quhLubLOWb+6DIVNaiOypTVOrBc=', 'Content-Length': '0'} 2024-04-16T19:56:29.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32a591575d685ae-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-16T19:56:29.629 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.629 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.629 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.629 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.629 DEBUG:boto:Method: GET 2024-04-16T19:56:29.629 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.630 DEBUG:boto:Data: 2024-04-16T19:56:29.630 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.630 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.630 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.630 DEBUG:boto:Params: {} 2024-04-16T19:56:29.630 DEBUG:boto:Token: None 2024-04-16T19:56:29.630 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.630 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY= 2024-04-16T19:56:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY=', 'Content-Length': '0'} 2024-04-16T19:56:29.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab59c31a76d1980-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.633 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.633 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.633 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.633 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.633 DEBUG:boto:Method: GET 2024-04-16T19:56:29.633 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.633 DEBUG:boto:Data: 2024-04-16T19:56:29.633 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.633 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.633 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.633 DEBUG:boto:Params: {} 2024-04-16T19:56:29.633 DEBUG:boto:Token: None 2024-04-16T19:56:29.634 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059?versionId=null 2024-04-16T19:56:29.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY= 2024-04-16T19:56:29.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ETT7Ub+u7hm2BIKlfIzx5G7LZMY=', 'Content-Length': '0'} 2024-04-16T19:56:29.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005c1e023dfbf498-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.678 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.678 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.678 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.679 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.679 DEBUG:boto:Method: GET 2024-04-16T19:56:29.679 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.679 DEBUG:boto:Data: 2024-04-16T19:56:29.679 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.679 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.679 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.679 DEBUG:boto:Params: {} 2024-04-16T19:56:29.679 DEBUG:boto:Token: None 2024-04-16T19:56:29.679 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd1fb48de4a5278-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.682 DEBUG:boto:b'yrjefh-51obj000000000000000000591000falseobj000000000000000000592024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.682 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.682 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.683 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.683 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.683 DEBUG:boto:Method: GET 2024-04-16T19:56:29.683 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.683 DEBUG:boto:Data: 2024-04-16T19:56:29.683 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.683 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.683 DEBUG:boto:Params: {} 2024-04-16T19:56:29.683 DEBUG:boto:Token: None 2024-04-16T19:56:29.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008732c9a911e21c12-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.687 DEBUG:boto:b'yrjefh-51obj000000000000000000591000falseobj000000000000000000592024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.688 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.688 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.688 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.688 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.688 DEBUG:boto:Data: 2024-04-16T19:56:29.688 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.688 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.688 DEBUG:boto:Params: {} 2024-04-16T19:56:29.688 DEBUG:boto:Token: None 2024-04-16T19:56:29.689 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QrTNcQyhD5lc5Hucu9JT5t5oGJc= 2024-04-16T19:56:29.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QrTNcQyhD5lc5Hucu9JT5t5oGJc=', 'Content-Length': '0'} 2024-04-16T19:56:29.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888b6940cf2ac5be-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.691 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.691 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.691 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.691 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.691 DEBUG:boto:Data: 2024-04-16T19:56:29.691 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.691 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.691 DEBUG:boto:Params: {} 2024-04-16T19:56:29.691 DEBUG:boto:Token: None 2024-04-16T19:56:29.691 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QrTNcQyhD5lc5Hucu9JT5t5oGJc= 2024-04-16T19:56:29.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QrTNcQyhD5lc5Hucu9JT5t5oGJc=', 'Content-Length': '0'} 2024-04-16T19:56:29.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ed6f2a9484f325e-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-16T19:56:29.694 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.694 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.694 DEBUG:boto:Method: GET 2024-04-16T19:56:29.694 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.694 DEBUG:boto:Data: 2024-04-16T19:56:29.694 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.694 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.694 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.694 DEBUG:boto:Params: {} 2024-04-16T19:56:29.695 DEBUG:boto:Token: None 2024-04-16T19:56:29.695 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kuFr9BcXvH9hfx9yYHuJUs0AcX0= 2024-04-16T19:56:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kuFr9BcXvH9hfx9yYHuJUs0AcX0=', 'Content-Length': '0'} 2024-04-16T19:56:29.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43f361792f70314-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.738 DEBUG:boto:path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.738 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.739 DEBUG:boto:Method: GET 2024-04-16T19:56:29.739 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.739 DEBUG:boto:Data: 2024-04-16T19:56:29.739 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.739 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.739 DEBUG:boto:Params: {} 2024-04-16T19:56:29.739 DEBUG:boto:Token: None 2024-04-16T19:56:29.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000059 2024-04-16T19:56:29.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kuFr9BcXvH9hfx9yYHuJUs0AcX0= 2024-04-16T19:56:29.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kuFr9BcXvH9hfx9yYHuJUs0AcX0=', 'Content-Length': '0'} 2024-04-16T19:56:29.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd99eb83123d158-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.742 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.742 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.742 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.742 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.742 DEBUG:boto:Method: GET 2024-04-16T19:56:29.742 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.742 DEBUG:boto:Data: 2024-04-16T19:56:29.742 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.742 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.742 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.743 DEBUG:boto:Params: {} 2024-04-16T19:56:29.743 DEBUG:boto:Token: None 2024-04-16T19:56:29.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a1a2dd8f122953-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.746 DEBUG:boto:b'yrjefh-51obj000000000000000000591000falseobj000000000000000000592024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.746 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.746 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.746 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.746 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.746 DEBUG:boto:Method: GET 2024-04-16T19:56:29.746 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000059 2024-04-16T19:56:29.746 DEBUG:boto:Data: 2024-04-16T19:56:29.747 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.747 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.747 DEBUG:boto:Params: {} 2024-04-16T19:56:29.747 DEBUG:boto:Token: None 2024-04-16T19:56:29.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e362f59d608b36-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.751 DEBUG:boto:b'yrjefh-51obj000000000000000000591000falseobj000000000000000000592024-04-16T19:56:07.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-16T19:56:29.752 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.752 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.752 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.752 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.752 DEBUG:boto:Method: GET 2024-04-16T19:56:29.752 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.752 DEBUG:boto:Data: 2024-04-16T19:56:29.752 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.752 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.753 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.753 DEBUG:boto:Params: {} 2024-04-16T19:56:29.753 DEBUG:boto:Token: None 2024-04-16T19:56:29.753 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.753 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY= 2024-04-16T19:56:29.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY=', 'Content-Length': '0'} 2024-04-16T19:56:29.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008589a57d897b634b-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.755 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.756 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.756 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.756 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.756 DEBUG:boto:Method: GET 2024-04-16T19:56:29.756 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.756 DEBUG:boto:Data: 2024-04-16T19:56:29.756 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.756 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.756 DEBUG:boto:Params: {} 2024-04-16T19:56:29.756 DEBUG:boto:Token: None 2024-04-16T19:56:29.757 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.757 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY= 2024-04-16T19:56:29.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY=', 'Content-Length': '0'} 2024-04-16T19:56:29.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed19ef8d4f186cbe-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.802 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.802 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.803 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.803 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.803 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.803 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.803 DEBUG:boto:Data: 2024-04-16T19:56:29.803 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.803 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.803 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.803 DEBUG:boto:Params: {} 2024-04-16T19:56:29.803 DEBUG:boto:Token: None 2024-04-16T19:56:29.803 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Ciu6zniO7HLGeyhN8YQ5G+wWRk= 2024-04-16T19:56:29.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Ciu6zniO7HLGeyhN8YQ5G+wWRk=', 'Content-Length': '0'} 2024-04-16T19:56:29.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3872e3211b9b5ca-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.806 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.806 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.806 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.806 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.806 DEBUG:boto:Data: 2024-04-16T19:56:29.806 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.806 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.807 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.807 DEBUG:boto:Params: {} 2024-04-16T19:56:29.807 DEBUG:boto:Token: None 2024-04-16T19:56:29.807 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Ciu6zniO7HLGeyhN8YQ5G+wWRk= 2024-04-16T19:56:29.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Ciu6zniO7HLGeyhN8YQ5G+wWRk=', 'Content-Length': '0'} 2024-04-16T19:56:29.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a0931c77b3ca2c-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-16T19:56:29.809 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.809 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.809 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.810 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.810 DEBUG:boto:Method: GET 2024-04-16T19:56:29.810 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.810 DEBUG:boto:Data: 2024-04-16T19:56:29.810 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.810 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.810 DEBUG:boto:Params: {} 2024-04-16T19:56:29.810 DEBUG:boto:Token: None 2024-04-16T19:56:29.810 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY= 2024-04-16T19:56:29.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY=', 'Content-Length': '0'} 2024-04-16T19:56:29.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c84e1dd5e0aa29-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.813 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.813 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.813 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.813 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.814 DEBUG:boto:Method: GET 2024-04-16T19:56:29.814 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.814 DEBUG:boto:Data: 2024-04-16T19:56:29.814 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.814 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.814 DEBUG:boto:Params: {} 2024-04-16T19:56:29.814 DEBUG:boto:Token: None 2024-04-16T19:56:29.814 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060?versionId=null 2024-04-16T19:56:29.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY= 2024-04-16T19:56:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vfM4BwA3L3niKduO4lXJW2xW4DY=', 'Content-Length': '0'} 2024-04-16T19:56:29.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f9d704bb42d476-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.817 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.817 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.817 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.817 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.817 DEBUG:boto:Method: GET 2024-04-16T19:56:29.817 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.818 DEBUG:boto:Data: 2024-04-16T19:56:29.818 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.818 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.818 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.818 DEBUG:boto:Params: {} 2024-04-16T19:56:29.818 DEBUG:boto:Token: None 2024-04-16T19:56:29.818 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084b042bb0d1b4eb3-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.821 DEBUG:boto:b'yrjefh-51obj000000000000000000601000falseobj000000000000000000602024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.821 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.821 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.821 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.821 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.821 DEBUG:boto:Method: GET 2024-04-16T19:56:29.822 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.822 DEBUG:boto:Data: 2024-04-16T19:56:29.822 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.822 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.822 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.822 DEBUG:boto:Params: {} 2024-04-16T19:56:29.822 DEBUG:boto:Token: None 2024-04-16T19:56:29.822 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.822 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f6990f7f9df3ca-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.827 DEBUG:boto:b'yrjefh-51obj000000000000000000601000falseobj000000000000000000602024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.827 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.827 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.827 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.827 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.827 DEBUG:boto:Data: 2024-04-16T19:56:29.827 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.828 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.828 DEBUG:boto:Params: {} 2024-04-16T19:56:29.828 DEBUG:boto:Token: None 2024-04-16T19:56:29.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zSIU/p6VPP92LdGMz7lvQlGL28= 2024-04-16T19:56:29.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zSIU/p6VPP92LdGMz7lvQlGL28=', 'Content-Length': '0'} 2024-04-16T19:56:29.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538361575ecb460d-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.830 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.830 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.831 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.831 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.831 DEBUG:boto:Data: 2024-04-16T19:56:29.831 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.831 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.831 DEBUG:boto:Params: {} 2024-04-16T19:56:29.831 DEBUG:boto:Token: None 2024-04-16T19:56:29.831 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zSIU/p6VPP92LdGMz7lvQlGL28= 2024-04-16T19:56:29.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zSIU/p6VPP92LdGMz7lvQlGL28=', 'Content-Length': '0'} 2024-04-16T19:56:29.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3dbc0d09133c728-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-16T19:56:29.834 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.834 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.834 DEBUG:boto:Method: GET 2024-04-16T19:56:29.834 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.834 DEBUG:boto:Data: 2024-04-16T19:56:29.834 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.834 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.835 DEBUG:boto:Params: {} 2024-04-16T19:56:29.835 DEBUG:boto:Token: None 2024-04-16T19:56:29.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zUtlk36igiT5c8sA3jAb6usEYt0= 2024-04-16T19:56:29.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zUtlk36igiT5c8sA3jAb6usEYt0=', 'Content-Length': '0'} 2024-04-16T19:56:29.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000275ee0776e4ab4b9-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.838 DEBUG:boto:path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.838 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.838 DEBUG:boto:Method: GET 2024-04-16T19:56:29.838 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.838 DEBUG:boto:Data: 2024-04-16T19:56:29.838 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.838 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.838 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.838 DEBUG:boto:Params: {} 2024-04-16T19:56:29.839 DEBUG:boto:Token: None 2024-04-16T19:56:29.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000060 2024-04-16T19:56:29.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zUtlk36igiT5c8sA3jAb6usEYt0= 2024-04-16T19:56:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zUtlk36igiT5c8sA3jAb6usEYt0=', 'Content-Length': '0'} 2024-04-16T19:56:29.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000856ee1b64f930a7-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.841 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.841 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.841 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.841 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.841 DEBUG:boto:Method: GET 2024-04-16T19:56:29.842 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.842 DEBUG:boto:Data: 2024-04-16T19:56:29.842 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.842 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.842 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.842 DEBUG:boto:Params: {} 2024-04-16T19:56:29.842 DEBUG:boto:Token: None 2024-04-16T19:56:29.842 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.842 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a56ec73d83df23-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.845 DEBUG:boto:b'yrjefh-51obj000000000000000000601000falseobj000000000000000000602024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.845 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.845 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.845 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.846 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.846 DEBUG:boto:Method: GET 2024-04-16T19:56:29.846 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000060 2024-04-16T19:56:29.846 DEBUG:boto:Data: 2024-04-16T19:56:29.846 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.846 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.846 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.846 DEBUG:boto:Params: {} 2024-04-16T19:56:29.846 DEBUG:boto:Token: None 2024-04-16T19:56:29.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96de52e960e7a74-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.851 DEBUG:boto:b'yrjefh-51obj000000000000000000601000falseobj000000000000000000602024-04-16T19:56:09.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-16T19:56:29.851 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.851 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.851 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.851 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.851 DEBUG:boto:Method: GET 2024-04-16T19:56:29.851 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.851 DEBUG:boto:Data: 2024-04-16T19:56:29.852 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.852 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.852 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.852 DEBUG:boto:Params: {} 2024-04-16T19:56:29.852 DEBUG:boto:Token: None 2024-04-16T19:56:29.852 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8= 2024-04-16T19:56:29.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8=', 'Content-Length': '0'} 2024-04-16T19:56:29.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007154254e516bb0ae-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.855 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.855 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.855 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.855 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.855 DEBUG:boto:Method: GET 2024-04-16T19:56:29.855 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.855 DEBUG:boto:Data: 2024-04-16T19:56:29.855 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.855 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.855 DEBUG:boto:Params: {} 2024-04-16T19:56:29.855 DEBUG:boto:Token: None 2024-04-16T19:56:29.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8= 2024-04-16T19:56:29.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8=', 'Content-Length': '0'} 2024-04-16T19:56:29.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af01cb508193830-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.898 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.898 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.899 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.899 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.899 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.899 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.899 DEBUG:boto:Data: 2024-04-16T19:56:29.899 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.899 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.899 DEBUG:boto:Params: {} 2024-04-16T19:56:29.899 DEBUG:boto:Token: None 2024-04-16T19:56:29.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndoDl5duoKxeWRVKNA8oXXXAoy4= 2024-04-16T19:56:29.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndoDl5duoKxeWRVKNA8oXXXAoy4=', 'Content-Length': '0'} 2024-04-16T19:56:29.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a10dd476a40f7c-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.902 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.902 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.902 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.902 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.902 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.902 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.902 DEBUG:boto:Data: 2024-04-16T19:56:29.902 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.903 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.903 DEBUG:boto:Params: {} 2024-04-16T19:56:29.903 DEBUG:boto:Token: None 2024-04-16T19:56:29.903 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndoDl5duoKxeWRVKNA8oXXXAoy4= 2024-04-16T19:56:29.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndoDl5duoKxeWRVKNA8oXXXAoy4=', 'Content-Length': '0'} 2024-04-16T19:56:29.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031eed8e9a80ae682-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-16T19:56:29.905 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.905 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.905 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.905 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.906 DEBUG:boto:Method: GET 2024-04-16T19:56:29.906 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.906 DEBUG:boto:Data: 2024-04-16T19:56:29.906 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.906 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.906 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.906 DEBUG:boto:Params: {} 2024-04-16T19:56:29.906 DEBUG:boto:Token: None 2024-04-16T19:56:29.906 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.906 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8= 2024-04-16T19:56:29.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8=', 'Content-Length': '0'} 2024-04-16T19:56:29.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71acc3b84453ae3-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.908 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.909 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.909 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.909 DEBUG:boto:Method: GET 2024-04-16T19:56:29.909 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.909 DEBUG:boto:Data: 2024-04-16T19:56:29.909 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.909 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.909 DEBUG:boto:Params: {} 2024-04-16T19:56:29.909 DEBUG:boto:Token: None 2024-04-16T19:56:29.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061?versionId=null 2024-04-16T19:56:29.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8= 2024-04-16T19:56:29.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVeTRD3Du4Bp31ARdSa7VnCBq8=', 'Content-Length': '0'} 2024-04-16T19:56:29.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d64c36b6f235f7-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.912 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.912 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.912 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.912 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.912 DEBUG:boto:Method: GET 2024-04-16T19:56:29.912 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.912 DEBUG:boto:Data: 2024-04-16T19:56:29.913 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.913 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.913 DEBUG:boto:Params: {} 2024-04-16T19:56:29.913 DEBUG:boto:Token: None 2024-04-16T19:56:29.913 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e516171706372ec4-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.916 DEBUG:boto:b'yrjefh-51obj000000000000000000611000falseobj000000000000000000612024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.916 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.916 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.916 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.916 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.916 DEBUG:boto:Method: GET 2024-04-16T19:56:29.917 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.917 DEBUG:boto:Data: 2024-04-16T19:56:29.917 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.917 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.917 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.917 DEBUG:boto:Params: {} 2024-04-16T19:56:29.917 DEBUG:boto:Token: None 2024-04-16T19:56:29.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d73fd6bfe35bc0c7-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.921 DEBUG:boto:b'yrjefh-51obj000000000000000000611000falseobj000000000000000000612024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.921 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.922 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.922 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.922 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.922 DEBUG:boto:Data: 2024-04-16T19:56:29.922 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.922 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.922 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.922 DEBUG:boto:Params: {} 2024-04-16T19:56:29.922 DEBUG:boto:Token: None 2024-04-16T19:56:29.922 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RxsfvmBxIwgNHtFuOF4RwjJwioY= 2024-04-16T19:56:29.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RxsfvmBxIwgNHtFuOF4RwjJwioY=', 'Content-Length': '0'} 2024-04-16T19:56:29.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028da57aa89fa43fc-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.924 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.924 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.925 DEBUG:boto:Method: HEAD 2024-04-16T19:56:29.925 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.925 DEBUG:boto:Data: 2024-04-16T19:56:29.925 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.925 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.925 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.925 DEBUG:boto:Params: {} 2024-04-16T19:56:29.925 DEBUG:boto:Token: None 2024-04-16T19:56:29.925 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RxsfvmBxIwgNHtFuOF4RwjJwioY= 2024-04-16T19:56:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RxsfvmBxIwgNHtFuOF4RwjJwioY=', 'Content-Length': '0'} 2024-04-16T19:56:29.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69397e14d661abe-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-16T19:56:29.927 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.928 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.928 DEBUG:boto:Method: GET 2024-04-16T19:56:29.928 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.928 DEBUG:boto:Data: 2024-04-16T19:56:29.928 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.928 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.928 DEBUG:boto:Params: {} 2024-04-16T19:56:29.928 DEBUG:boto:Token: None 2024-04-16T19:56:29.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6agmNLoz1mjqVP4Z+/DWGzOY61g= 2024-04-16T19:56:29.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6agmNLoz1mjqVP4Z+/DWGzOY61g=', 'Content-Length': '0'} 2024-04-16T19:56:29.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441f413baf6b9a2b-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.931 DEBUG:boto:path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.931 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.931 DEBUG:boto:Method: GET 2024-04-16T19:56:29.931 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.931 DEBUG:boto:Data: 2024-04-16T19:56:29.931 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.931 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.931 DEBUG:boto:Params: {} 2024-04-16T19:56:29.932 DEBUG:boto:Token: None 2024-04-16T19:56:29.932 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000061 2024-04-16T19:56:29.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6agmNLoz1mjqVP4Z+/DWGzOY61g= 2024-04-16T19:56:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6agmNLoz1mjqVP4Z+/DWGzOY61g=', 'Content-Length': '0'} 2024-04-16T19:56:29.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e45a2174213f687-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.974 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.974 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.975 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.975 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.975 DEBUG:boto:Method: GET 2024-04-16T19:56:29.975 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.975 DEBUG:boto:Data: 2024-04-16T19:56:29.975 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.975 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.975 DEBUG:boto:Params: {} 2024-04-16T19:56:29.975 DEBUG:boto:Token: None 2024-04-16T19:56:29.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98c8b0f0dcbaad9-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.978 DEBUG:boto:b'yrjefh-51obj000000000000000000611000falseobj000000000000000000612024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.979 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:29.979 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:29.979 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.979 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.979 DEBUG:boto:Method: GET 2024-04-16T19:56:29.979 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000061 2024-04-16T19:56:29.979 DEBUG:boto:Data: 2024-04-16T19:56:29.979 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:29.979 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.979 DEBUG:boto:Params: {} 2024-04-16T19:56:29.980 DEBUG:boto:Token: None 2024-04-16T19:56:29.980 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/ 2024-04-16T19:56:29.980 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk= 2024-04-16T19:56:29.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsYZhtRYxvbMeJRQMbFkeb2iTwk=', 'Content-Length': '0'} 2024-04-16T19:56:29.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009167258bd009df28-00661ed7ed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:29.984 DEBUG:boto:b'yrjefh-51obj000000000000000000611000falseobj000000000000000000612024-04-16T19:56:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:29.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-16T19:56:29.984 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:29.984 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:29.984 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:29.984 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:29.985 DEBUG:boto:Method: GET 2024-04-16T19:56:29.985 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:29.985 DEBUG:boto:Data: 2024-04-16T19:56:29.985 DEBUG:boto:Headers: {} 2024-04-16T19:56:29.985 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:29.985 DEBUG:boto:Port: 8000 2024-04-16T19:56:29.985 DEBUG:boto:Params: {} 2024-04-16T19:56:29.985 DEBUG:boto:Token: None 2024-04-16T19:56:29.985 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:29 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:29.985 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DJn3s0D+qHSBj2tcstiwsyav0S8= 2024-04-16T19:56:29.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DJn3s0D+qHSBj2tcstiwsyav0S8=', 'Content-Length': '0'} 2024-04-16T19:56:29.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8c73acbfcd738a-00661ed7ed-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.030 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.031 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.031 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.031 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.031 DEBUG:boto:Method: GET 2024-04-16T19:56:30.031 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.031 DEBUG:boto:Data: 2024-04-16T19:56:30.031 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.031 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.031 DEBUG:boto:Params: {} 2024-04-16T19:56:30.031 DEBUG:boto:Token: None 2024-04-16T19:56:30.032 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI= 2024-04-16T19:56:30.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI=', 'Content-Length': '0'} 2024-04-16T19:56:30.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81e4278f4c69025-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.035 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.035 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.035 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.035 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.035 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.035 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.035 DEBUG:boto:Data: 2024-04-16T19:56:30.035 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.035 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.035 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.035 DEBUG:boto:Params: {} 2024-04-16T19:56:30.036 DEBUG:boto:Token: None 2024-04-16T19:56:30.036 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TgIEr8qoez2+jRPuEVuUd6eb24w= 2024-04-16T19:56:30.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TgIEr8qoez2+jRPuEVuUd6eb24w=', 'Content-Length': '0'} 2024-04-16T19:56:30.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94ccd375d3a8a65-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.038 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.039 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.039 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.039 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.039 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.039 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.039 DEBUG:boto:Data: 2024-04-16T19:56:30.039 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.039 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.039 DEBUG:boto:Params: {} 2024-04-16T19:56:30.039 DEBUG:boto:Token: None 2024-04-16T19:56:30.040 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TgIEr8qoez2+jRPuEVuUd6eb24w= 2024-04-16T19:56:30.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TgIEr8qoez2+jRPuEVuUd6eb24w=', 'Content-Length': '0'} 2024-04-16T19:56:30.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020c9670d09f415af-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-16T19:56:30.042 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.042 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.042 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.042 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.043 DEBUG:boto:Method: GET 2024-04-16T19:56:30.043 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.043 DEBUG:boto:Data: 2024-04-16T19:56:30.043 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.043 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.043 DEBUG:boto:Params: {} 2024-04-16T19:56:30.043 DEBUG:boto:Token: None 2024-04-16T19:56:30.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI= 2024-04-16T19:56:30.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI=', 'Content-Length': '0'} 2024-04-16T19:56:30.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c484b2acb5cd456d-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.046 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.046 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.046 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.047 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.047 DEBUG:boto:Method: GET 2024-04-16T19:56:30.047 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.047 DEBUG:boto:Data: 2024-04-16T19:56:30.047 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.047 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.047 DEBUG:boto:Params: {} 2024-04-16T19:56:30.047 DEBUG:boto:Token: None 2024-04-16T19:56:30.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062?versionId=null 2024-04-16T19:56:30.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI= 2024-04-16T19:56:30.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I4tS/XWKai9sGTDoLwXmuwwRuvI=', 'Content-Length': '0'} 2024-04-16T19:56:30.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032379f047a73fb9b-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.050 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.050 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.050 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.050 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.050 DEBUG:boto:Method: GET 2024-04-16T19:56:30.051 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.051 DEBUG:boto:Data: 2024-04-16T19:56:30.051 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.051 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.051 DEBUG:boto:Params: {} 2024-04-16T19:56:30.051 DEBUG:boto:Token: None 2024-04-16T19:56:30.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c7d54ba7e97a4ef-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.054 DEBUG:boto:b'yrjefh-51obj000000000000000000621000falseobj000000000000000000622024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.055 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.055 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.055 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.055 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.055 DEBUG:boto:Method: GET 2024-04-16T19:56:30.055 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.055 DEBUG:boto:Data: 2024-04-16T19:56:30.055 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.056 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.056 DEBUG:boto:Params: {} 2024-04-16T19:56:30.056 DEBUG:boto:Token: None 2024-04-16T19:56:30.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4044c4c3c82154-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.060 DEBUG:boto:b'yrjefh-51obj000000000000000000621000falseobj000000000000000000622024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.060 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.061 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.061 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.061 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.061 DEBUG:boto:Data: 2024-04-16T19:56:30.061 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.061 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.061 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.061 DEBUG:boto:Params: {} 2024-04-16T19:56:30.062 DEBUG:boto:Token: None 2024-04-16T19:56:30.062 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.062 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IplCZnRc5b31ZhnDWgV7iWlDStM= 2024-04-16T19:56:30.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IplCZnRc5b31ZhnDWgV7iWlDStM=', 'Content-Length': '0'} 2024-04-16T19:56:30.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c837a19c57e617a-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.065 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.065 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.065 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.065 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.065 DEBUG:boto:Data: 2024-04-16T19:56:30.065 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.065 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.065 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.066 DEBUG:boto:Params: {} 2024-04-16T19:56:30.066 DEBUG:boto:Token: None 2024-04-16T19:56:30.066 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.066 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IplCZnRc5b31ZhnDWgV7iWlDStM= 2024-04-16T19:56:30.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IplCZnRc5b31ZhnDWgV7iWlDStM=', 'Content-Length': '0'} 2024-04-16T19:56:30.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009347d8685686778-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-16T19:56:30.068 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.069 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.069 DEBUG:boto:Method: GET 2024-04-16T19:56:30.069 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.069 DEBUG:boto:Data: 2024-04-16T19:56:30.069 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.069 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.069 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.069 DEBUG:boto:Params: {} 2024-04-16T19:56:30.070 DEBUG:boto:Token: None 2024-04-16T19:56:30.070 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u8n3O+5wH0H6OFvivlLS4+bYPBE= 2024-04-16T19:56:30.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u8n3O+5wH0H6OFvivlLS4+bYPBE=', 'Content-Length': '0'} 2024-04-16T19:56:30.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e8c7ae117b93bf-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.073 DEBUG:boto:Method: GET 2024-04-16T19:56:30.073 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.073 DEBUG:boto:Data: 2024-04-16T19:56:30.073 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.073 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.073 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.073 DEBUG:boto:Params: {} 2024-04-16T19:56:30.074 DEBUG:boto:Token: None 2024-04-16T19:56:30.074 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000062 2024-04-16T19:56:30.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u8n3O+5wH0H6OFvivlLS4+bYPBE= 2024-04-16T19:56:30.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u8n3O+5wH0H6OFvivlLS4+bYPBE=', 'Content-Length': '0'} 2024-04-16T19:56:30.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aeb2aca5ee0bdc4-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.118 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.118 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.119 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.119 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.119 DEBUG:boto:Method: GET 2024-04-16T19:56:30.119 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.119 DEBUG:boto:Data: 2024-04-16T19:56:30.119 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.119 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.119 DEBUG:boto:Params: {} 2024-04-16T19:56:30.119 DEBUG:boto:Token: None 2024-04-16T19:56:30.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e2ec60128e1de8-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.122 DEBUG:boto:b'yrjefh-51obj000000000000000000621000falseobj000000000000000000622024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.123 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.123 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.123 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.123 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.123 DEBUG:boto:Method: GET 2024-04-16T19:56:30.123 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000062 2024-04-16T19:56:30.123 DEBUG:boto:Data: 2024-04-16T19:56:30.123 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.124 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.124 DEBUG:boto:Params: {} 2024-04-16T19:56:30.124 DEBUG:boto:Token: None 2024-04-16T19:56:30.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b37d92f32541ec-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.128 DEBUG:boto:b'yrjefh-51obj000000000000000000621000falseobj000000000000000000622024-04-16T19:56:08.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-16T19:56:30.128 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.129 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.129 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.129 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.129 DEBUG:boto:Method: GET 2024-04-16T19:56:30.129 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.129 DEBUG:boto:Data: 2024-04-16T19:56:30.129 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.129 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.129 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.129 DEBUG:boto:Params: {} 2024-04-16T19:56:30.129 DEBUG:boto:Token: None 2024-04-16T19:56:30.130 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM= 2024-04-16T19:56:30.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM=', 'Content-Length': '0'} 2024-04-16T19:56:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000234b5949c27be56-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.174 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.174 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.175 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.175 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.175 DEBUG:boto:Method: GET 2024-04-16T19:56:30.175 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.175 DEBUG:boto:Data: 2024-04-16T19:56:30.175 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.175 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.175 DEBUG:boto:Params: {} 2024-04-16T19:56:30.175 DEBUG:boto:Token: None 2024-04-16T19:56:30.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM= 2024-04-16T19:56:30.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM=', 'Content-Length': '0'} 2024-04-16T19:56:30.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bee8de5e5937dd9-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.178 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.178 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.178 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.178 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.178 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.178 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.179 DEBUG:boto:Data: 2024-04-16T19:56:30.179 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.179 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.179 DEBUG:boto:Params: {} 2024-04-16T19:56:30.179 DEBUG:boto:Token: None 2024-04-16T19:56:30.179 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ssvQvk8fuam7MSPXZE40pv+9DU= 2024-04-16T19:56:30.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ssvQvk8fuam7MSPXZE40pv+9DU=', 'Content-Length': '0'} 2024-04-16T19:56:30.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edddd209c49d595e-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.182 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.182 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.182 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.182 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.182 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.182 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.182 DEBUG:boto:Data: 2024-04-16T19:56:30.182 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.182 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.182 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.183 DEBUG:boto:Params: {} 2024-04-16T19:56:30.183 DEBUG:boto:Token: None 2024-04-16T19:56:30.183 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0ssvQvk8fuam7MSPXZE40pv+9DU= 2024-04-16T19:56:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0ssvQvk8fuam7MSPXZE40pv+9DU=', 'Content-Length': '0'} 2024-04-16T19:56:30.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000025956d9e544ab6d-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-16T19:56:30.185 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.185 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.185 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.185 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.186 DEBUG:boto:Method: GET 2024-04-16T19:56:30.186 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.186 DEBUG:boto:Data: 2024-04-16T19:56:30.186 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.186 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.186 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.186 DEBUG:boto:Params: {} 2024-04-16T19:56:30.186 DEBUG:boto:Token: None 2024-04-16T19:56:30.186 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM= 2024-04-16T19:56:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM=', 'Content-Length': '0'} 2024-04-16T19:56:30.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c624c1e015ec007-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.189 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.189 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.189 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.189 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.189 DEBUG:boto:Method: GET 2024-04-16T19:56:30.189 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.189 DEBUG:boto:Data: 2024-04-16T19:56:30.189 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.189 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.189 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.190 DEBUG:boto:Params: {} 2024-04-16T19:56:30.190 DEBUG:boto:Token: None 2024-04-16T19:56:30.190 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065?versionId=null 2024-04-16T19:56:30.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM= 2024-04-16T19:56:30.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YXGmadrUVWB0q/MfwvWUNqPrrwM=', 'Content-Length': '0'} 2024-04-16T19:56:30.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c7f07119d585b36-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.192 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.192 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.192 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.192 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.192 DEBUG:boto:Method: GET 2024-04-16T19:56:30.193 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.193 DEBUG:boto:Data: 2024-04-16T19:56:30.193 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.193 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.193 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.193 DEBUG:boto:Params: {} 2024-04-16T19:56:30.193 DEBUG:boto:Token: None 2024-04-16T19:56:30.193 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e3389067a100b6-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.196 DEBUG:boto:b'yrjefh-51obj000000000000000000651000falseobj000000000000000000652024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.196 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.196 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.196 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.196 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.196 DEBUG:boto:Method: GET 2024-04-16T19:56:30.197 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.197 DEBUG:boto:Data: 2024-04-16T19:56:30.197 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.197 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.197 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.197 DEBUG:boto:Params: {} 2024-04-16T19:56:30.197 DEBUG:boto:Token: None 2024-04-16T19:56:30.197 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f654476e54f1140-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.201 DEBUG:boto:b'yrjefh-51obj000000000000000000651000falseobj000000000000000000652024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.202 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.202 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.202 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.202 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.202 DEBUG:boto:Data: 2024-04-16T19:56:30.202 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.202 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.202 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.202 DEBUG:boto:Params: {} 2024-04-16T19:56:30.202 DEBUG:boto:Token: None 2024-04-16T19:56:30.203 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nG4qPZbonzGUoRREA3rGwfnP8YE= 2024-04-16T19:56:30.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nG4qPZbonzGUoRREA3rGwfnP8YE=', 'Content-Length': '0'} 2024-04-16T19:56:30.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed2adf0d7e94dfd-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.205 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.205 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.205 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.205 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.205 DEBUG:boto:Data: 2024-04-16T19:56:30.205 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.205 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.205 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.206 DEBUG:boto:Params: {} 2024-04-16T19:56:30.206 DEBUG:boto:Token: None 2024-04-16T19:56:30.206 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.206 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nG4qPZbonzGUoRREA3rGwfnP8YE= 2024-04-16T19:56:30.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nG4qPZbonzGUoRREA3rGwfnP8YE=', 'Content-Length': '0'} 2024-04-16T19:56:30.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af5ac31e221a958-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2024-04-16T19:56:30.208 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.208 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.208 DEBUG:boto:Method: GET 2024-04-16T19:56:30.208 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.208 DEBUG:boto:Data: 2024-04-16T19:56:30.209 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.209 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.209 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.209 DEBUG:boto:Params: {} 2024-04-16T19:56:30.209 DEBUG:boto:Token: None 2024-04-16T19:56:30.209 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yokr/Xc9MQ3wlEhnTeTt7Dcj7UU= 2024-04-16T19:56:30.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yokr/Xc9MQ3wlEhnTeTt7Dcj7UU=', 'Content-Length': '0'} 2024-04-16T19:56:30.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c3cc6b1e363a03-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.211 DEBUG:boto:path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.211 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.211 DEBUG:boto:Method: GET 2024-04-16T19:56:30.212 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.212 DEBUG:boto:Data: 2024-04-16T19:56:30.212 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.212 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.212 DEBUG:boto:Params: {} 2024-04-16T19:56:30.212 DEBUG:boto:Token: None 2024-04-16T19:56:30.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000065 2024-04-16T19:56:30.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yokr/Xc9MQ3wlEhnTeTt7Dcj7UU= 2024-04-16T19:56:30.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yokr/Xc9MQ3wlEhnTeTt7Dcj7UU=', 'Content-Length': '0'} 2024-04-16T19:56:30.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f276b402d83657-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.214 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.215 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.215 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.215 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.215 DEBUG:boto:Method: GET 2024-04-16T19:56:30.215 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.215 DEBUG:boto:Data: 2024-04-16T19:56:30.215 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.215 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.215 DEBUG:boto:Params: {} 2024-04-16T19:56:30.215 DEBUG:boto:Token: None 2024-04-16T19:56:30.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52ce476b7e66af6-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.262 DEBUG:boto:b'yrjefh-51obj000000000000000000651000falseobj000000000000000000652024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.263 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.263 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.263 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.263 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.263 DEBUG:boto:Method: GET 2024-04-16T19:56:30.263 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000065 2024-04-16T19:56:30.263 DEBUG:boto:Data: 2024-04-16T19:56:30.263 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.263 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.263 DEBUG:boto:Params: {} 2024-04-16T19:56:30.264 DEBUG:boto:Token: None 2024-04-16T19:56:30.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bb3c0903d816266-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.268 DEBUG:boto:b'yrjefh-51obj000000000000000000651000falseobj000000000000000000652024-04-16T19:56:08.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-16T19:56:30.268 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.268 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.268 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.268 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.268 DEBUG:boto:Method: GET 2024-04-16T19:56:30.268 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.268 DEBUG:boto:Data: 2024-04-16T19:56:30.269 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.269 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.269 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.269 DEBUG:boto:Params: {} 2024-04-16T19:56:30.269 DEBUG:boto:Token: None 2024-04-16T19:56:30.269 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo= 2024-04-16T19:56:30.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo=', 'Content-Length': '0'} 2024-04-16T19:56:30.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700a1b968f0bb82b-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.272 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.272 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.272 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.272 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.272 DEBUG:boto:Method: GET 2024-04-16T19:56:30.272 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.273 DEBUG:boto:Data: 2024-04-16T19:56:30.273 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.273 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.273 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.273 DEBUG:boto:Params: {} 2024-04-16T19:56:30.273 DEBUG:boto:Token: None 2024-04-16T19:56:30.273 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo= 2024-04-16T19:56:30.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo=', 'Content-Length': '0'} 2024-04-16T19:56:30.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003082767ec859f2c4-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.318 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.318 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.319 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.319 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.319 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.319 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.319 DEBUG:boto:Data: 2024-04-16T19:56:30.319 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.319 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.319 DEBUG:boto:Params: {} 2024-04-16T19:56:30.319 DEBUG:boto:Token: None 2024-04-16T19:56:30.319 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m+I2eMbZGSyUi2gEoqr4YZVumTU= 2024-04-16T19:56:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m+I2eMbZGSyUi2gEoqr4YZVumTU=', 'Content-Length': '0'} 2024-04-16T19:56:30.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd3716a082b79a3-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.322 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.322 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.322 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.323 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.323 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.323 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.323 DEBUG:boto:Data: 2024-04-16T19:56:30.323 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.323 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.323 DEBUG:boto:Params: {} 2024-04-16T19:56:30.323 DEBUG:boto:Token: None 2024-04-16T19:56:30.323 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m+I2eMbZGSyUi2gEoqr4YZVumTU= 2024-04-16T19:56:30.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m+I2eMbZGSyUi2gEoqr4YZVumTU=', 'Content-Length': '0'} 2024-04-16T19:56:30.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb9e1dbca514cf9-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-16T19:56:30.326 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.326 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.326 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.326 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.326 DEBUG:boto:Method: GET 2024-04-16T19:56:30.326 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.326 DEBUG:boto:Data: 2024-04-16T19:56:30.326 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.327 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.327 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.327 DEBUG:boto:Params: {} 2024-04-16T19:56:30.327 DEBUG:boto:Token: None 2024-04-16T19:56:30.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo= 2024-04-16T19:56:30.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo=', 'Content-Length': '0'} 2024-04-16T19:56:30.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe443d2c6b46c9b-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.330 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.330 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.330 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.330 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.330 DEBUG:boto:Method: GET 2024-04-16T19:56:30.330 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.330 DEBUG:boto:Data: 2024-04-16T19:56:30.330 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.331 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.331 DEBUG:boto:Params: {} 2024-04-16T19:56:30.331 DEBUG:boto:Token: None 2024-04-16T19:56:30.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076?versionId=null 2024-04-16T19:56:30.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo= 2024-04-16T19:56:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srXZb0JBOiBM1XqD3zuWccdQvjo=', 'Content-Length': '0'} 2024-04-16T19:56:30.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f933c52b06dfd12e-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.333 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.333 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.334 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.334 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.334 DEBUG:boto:Method: GET 2024-04-16T19:56:30.334 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.334 DEBUG:boto:Data: 2024-04-16T19:56:30.334 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.334 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.334 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.334 DEBUG:boto:Params: {} 2024-04-16T19:56:30.334 DEBUG:boto:Token: None 2024-04-16T19:56:30.334 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b64cc0d778e0ccf9-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.337 DEBUG:boto:b'yrjefh-51obj000000000000000000761000falseobj000000000000000000762024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.337 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.338 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.338 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.338 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.338 DEBUG:boto:Method: GET 2024-04-16T19:56:30.338 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.338 DEBUG:boto:Data: 2024-04-16T19:56:30.338 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.338 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.338 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.338 DEBUG:boto:Params: {} 2024-04-16T19:56:30.339 DEBUG:boto:Token: None 2024-04-16T19:56:30.339 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b7b488347c0076a-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.343 DEBUG:boto:b'yrjefh-51obj000000000000000000761000falseobj000000000000000000762024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.343 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.344 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.344 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.344 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.344 DEBUG:boto:Data: 2024-04-16T19:56:30.344 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.344 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.344 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.344 DEBUG:boto:Params: {} 2024-04-16T19:56:30.344 DEBUG:boto:Token: None 2024-04-16T19:56:30.344 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.345 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WaQIud32gWwaNpZgBHjPVVYFYL8= 2024-04-16T19:56:30.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WaQIud32gWwaNpZgBHjPVVYFYL8=', 'Content-Length': '0'} 2024-04-16T19:56:30.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745f0a8045da8a41-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.347 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.347 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.347 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.347 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.347 DEBUG:boto:Data: 2024-04-16T19:56:30.347 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.348 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.348 DEBUG:boto:Params: {} 2024-04-16T19:56:30.348 DEBUG:boto:Token: None 2024-04-16T19:56:30.348 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WaQIud32gWwaNpZgBHjPVVYFYL8= 2024-04-16T19:56:30.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WaQIud32gWwaNpZgBHjPVVYFYL8=', 'Content-Length': '0'} 2024-04-16T19:56:30.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4a38c228804e64-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-16T19:56:30.351 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.351 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.351 DEBUG:boto:Method: GET 2024-04-16T19:56:30.351 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.351 DEBUG:boto:Data: 2024-04-16T19:56:30.351 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.351 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.351 DEBUG:boto:Params: {} 2024-04-16T19:56:30.351 DEBUG:boto:Token: None 2024-04-16T19:56:30.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W1FjwKfW+VdU/FXBma/6LhLtFMA= 2024-04-16T19:56:30.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W1FjwKfW+VdU/FXBma/6LhLtFMA=', 'Content-Length': '0'} 2024-04-16T19:56:30.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:09 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3154a3b0937475-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.354 DEBUG:boto:path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.354 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.354 DEBUG:boto:Method: GET 2024-04-16T19:56:30.355 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.355 DEBUG:boto:Data: 2024-04-16T19:56:30.355 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.355 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.355 DEBUG:boto:Params: {} 2024-04-16T19:56:30.355 DEBUG:boto:Token: None 2024-04-16T19:56:30.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000076 2024-04-16T19:56:30.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W1FjwKfW+VdU/FXBma/6LhLtFMA= 2024-04-16T19:56:30.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W1FjwKfW+VdU/FXBma/6LhLtFMA=', 'Content-Length': '0'} 2024-04-16T19:56:30.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1a984c259f9b87-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.401 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.401 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.401 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.401 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.401 DEBUG:boto:Method: GET 2024-04-16T19:56:30.401 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.401 DEBUG:boto:Data: 2024-04-16T19:56:30.401 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.402 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.402 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.402 DEBUG:boto:Params: {} 2024-04-16T19:56:30.402 DEBUG:boto:Token: None 2024-04-16T19:56:30.402 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.402 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca2fc71f64b80d7c-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.405 DEBUG:boto:b'yrjefh-51obj000000000000000000761000falseobj000000000000000000762024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.406 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.406 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.406 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.406 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.406 DEBUG:boto:Method: GET 2024-04-16T19:56:30.406 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000076 2024-04-16T19:56:30.406 DEBUG:boto:Data: 2024-04-16T19:56:30.406 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.407 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.407 DEBUG:boto:Params: {} 2024-04-16T19:56:30.407 DEBUG:boto:Token: None 2024-04-16T19:56:30.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8026e514f2a0fef-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.411 DEBUG:boto:b'yrjefh-51obj000000000000000000761000falseobj000000000000000000762024-04-16T19:56:04.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-16T19:56:30.412 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.412 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.412 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.412 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.412 DEBUG:boto:Method: GET 2024-04-16T19:56:30.412 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.412 DEBUG:boto:Data: 2024-04-16T19:56:30.413 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.413 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.413 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.413 DEBUG:boto:Params: {} 2024-04-16T19:56:30.413 DEBUG:boto:Token: None 2024-04-16T19:56:30.413 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.413 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc= 2024-04-16T19:56:30.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc=', 'Content-Length': '0'} 2024-04-16T19:56:30.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a360ba2a393c02e9-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.416 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.416 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.416 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.416 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.416 DEBUG:boto:Method: GET 2024-04-16T19:56:30.416 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.417 DEBUG:boto:Data: 2024-04-16T19:56:30.417 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.417 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.417 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.417 DEBUG:boto:Params: {} 2024-04-16T19:56:30.417 DEBUG:boto:Token: None 2024-04-16T19:56:30.417 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc= 2024-04-16T19:56:30.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc=', 'Content-Length': '0'} 2024-04-16T19:56:30.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002743ce600c357352-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.419 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.420 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.420 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.420 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.420 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.420 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.420 DEBUG:boto:Data: 2024-04-16T19:56:30.420 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.420 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.420 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.420 DEBUG:boto:Params: {} 2024-04-16T19:56:30.420 DEBUG:boto:Token: None 2024-04-16T19:56:30.421 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H624Fjs7taTQLJ2wL0Z1w8kEi6Q= 2024-04-16T19:56:30.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H624Fjs7taTQLJ2wL0Z1w8kEi6Q=', 'Content-Length': '0'} 2024-04-16T19:56:30.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b872e08e9a209f13-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.423 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.423 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.423 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.423 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.423 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.423 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.424 DEBUG:boto:Data: 2024-04-16T19:56:30.424 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.424 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.424 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.424 DEBUG:boto:Params: {} 2024-04-16T19:56:30.424 DEBUG:boto:Token: None 2024-04-16T19:56:30.424 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H624Fjs7taTQLJ2wL0Z1w8kEi6Q= 2024-04-16T19:56:30.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H624Fjs7taTQLJ2wL0Z1w8kEi6Q=', 'Content-Length': '0'} 2024-04-16T19:56:30.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383974d52765c32f-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-16T19:56:30.427 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.427 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.427 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.427 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.427 DEBUG:boto:Method: GET 2024-04-16T19:56:30.427 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.427 DEBUG:boto:Data: 2024-04-16T19:56:30.427 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.427 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.427 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.428 DEBUG:boto:Params: {} 2024-04-16T19:56:30.428 DEBUG:boto:Token: None 2024-04-16T19:56:30.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc= 2024-04-16T19:56:30.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc=', 'Content-Length': '0'} 2024-04-16T19:56:30.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c97b037a30b38f-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.474 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.474 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.475 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.475 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.475 DEBUG:boto:Method: GET 2024-04-16T19:56:30.475 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.475 DEBUG:boto:Data: 2024-04-16T19:56:30.475 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.475 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.475 DEBUG:boto:Params: {} 2024-04-16T19:56:30.475 DEBUG:boto:Token: None 2024-04-16T19:56:30.476 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133?versionId=null 2024-04-16T19:56:30.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc= 2024-04-16T19:56:30.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzaYalathvStSADSri7DpWbSdyc=', 'Content-Length': '0'} 2024-04-16T19:56:30.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3e78be57d9c6c1-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.478 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.478 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.478 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.478 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.478 DEBUG:boto:Method: GET 2024-04-16T19:56:30.479 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.479 DEBUG:boto:Data: 2024-04-16T19:56:30.479 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.479 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.479 DEBUG:boto:Params: {} 2024-04-16T19:56:30.479 DEBUG:boto:Token: None 2024-04-16T19:56:30.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826925ae00af3f9b-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.482 DEBUG:boto:b'yrjefh-51obj000000000000000001331000falseobj000000000000000001332024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.482 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.482 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.482 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.483 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.483 DEBUG:boto:Method: GET 2024-04-16T19:56:30.483 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.483 DEBUG:boto:Data: 2024-04-16T19:56:30.483 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.483 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.483 DEBUG:boto:Params: {} 2024-04-16T19:56:30.483 DEBUG:boto:Token: None 2024-04-16T19:56:30.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5e03e7052fc746-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.488 DEBUG:boto:b'yrjefh-51obj000000000000000001331000falseobj000000000000000001332024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.488 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.488 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.488 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.488 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.488 DEBUG:boto:Data: 2024-04-16T19:56:30.488 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.488 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.489 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.489 DEBUG:boto:Params: {} 2024-04-16T19:56:30.489 DEBUG:boto:Token: None 2024-04-16T19:56:30.489 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VNfgrHuoCf8ZMJdrYIxLY85Cg8Y= 2024-04-16T19:56:30.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VNfgrHuoCf8ZMJdrYIxLY85Cg8Y=', 'Content-Length': '0'} 2024-04-16T19:56:30.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002671828332996d98-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.492 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.492 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.492 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.492 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.492 DEBUG:boto:Data: 2024-04-16T19:56:30.492 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.492 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.492 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.492 DEBUG:boto:Params: {} 2024-04-16T19:56:30.492 DEBUG:boto:Token: None 2024-04-16T19:56:30.492 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.493 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VNfgrHuoCf8ZMJdrYIxLY85Cg8Y= 2024-04-16T19:56:30.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VNfgrHuoCf8ZMJdrYIxLY85Cg8Y=', 'Content-Length': '0'} 2024-04-16T19:56:30.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb2af910f7ef82b-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-16T19:56:30.495 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.495 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.495 DEBUG:boto:Method: GET 2024-04-16T19:56:30.495 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.495 DEBUG:boto:Data: 2024-04-16T19:56:30.495 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.496 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.496 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.496 DEBUG:boto:Params: {} 2024-04-16T19:56:30.496 DEBUG:boto:Token: None 2024-04-16T19:56:30.496 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sai6e0MB8YK2FWgd9FqLSDkPX7s= 2024-04-16T19:56:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sai6e0MB8YK2FWgd9FqLSDkPX7s=', 'Content-Length': '0'} 2024-04-16T19:56:30.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9a1cd9ef500988f-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.499 DEBUG:boto:path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.499 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.499 DEBUG:boto:Method: GET 2024-04-16T19:56:30.499 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.499 DEBUG:boto:Data: 2024-04-16T19:56:30.499 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.499 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.499 DEBUG:boto:Params: {} 2024-04-16T19:56:30.499 DEBUG:boto:Token: None 2024-04-16T19:56:30.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000133 2024-04-16T19:56:30.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sai6e0MB8YK2FWgd9FqLSDkPX7s= 2024-04-16T19:56:30.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sai6e0MB8YK2FWgd9FqLSDkPX7s=', 'Content-Length': '0'} 2024-04-16T19:56:30.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003648297c1e172cd1-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.502 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.502 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.502 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.502 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.502 DEBUG:boto:Method: GET 2024-04-16T19:56:30.502 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.502 DEBUG:boto:Data: 2024-04-16T19:56:30.503 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.503 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.503 DEBUG:boto:Params: {} 2024-04-16T19:56:30.503 DEBUG:boto:Token: None 2024-04-16T19:56:30.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012fc8e51b9d56ed4-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.546 DEBUG:boto:b'yrjefh-51obj000000000000000001331000falseobj000000000000000001332024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.547 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.547 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.547 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.547 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.547 DEBUG:boto:Method: GET 2024-04-16T19:56:30.547 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000133 2024-04-16T19:56:30.547 DEBUG:boto:Data: 2024-04-16T19:56:30.547 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.547 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.548 DEBUG:boto:Params: {} 2024-04-16T19:56:30.548 DEBUG:boto:Token: None 2024-04-16T19:56:30.548 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000becb31f0a14d938f-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.552 DEBUG:boto:b'yrjefh-51obj000000000000000001331000falseobj000000000000000001332024-04-16T19:56:07.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-16T19:56:30.552 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.552 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.552 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.553 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.553 DEBUG:boto:Method: GET 2024-04-16T19:56:30.553 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.553 DEBUG:boto:Data: 2024-04-16T19:56:30.553 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.553 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.553 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.553 DEBUG:boto:Params: {} 2024-04-16T19:56:30.553 DEBUG:boto:Token: None 2024-04-16T19:56:30.553 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA= 2024-04-16T19:56:30.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA=', 'Content-Length': '0'} 2024-04-16T19:56:30.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5ce0112f838057-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.556 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.556 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.556 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.556 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.557 DEBUG:boto:Method: GET 2024-04-16T19:56:30.557 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.557 DEBUG:boto:Data: 2024-04-16T19:56:30.557 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.557 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.557 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.557 DEBUG:boto:Params: {} 2024-04-16T19:56:30.557 DEBUG:boto:Token: None 2024-04-16T19:56:30.557 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.557 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA= 2024-04-16T19:56:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA=', 'Content-Length': '0'} 2024-04-16T19:56:30.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004415388c0a1348e6-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.602 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.602 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.603 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.603 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.603 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.603 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.603 DEBUG:boto:Data: 2024-04-16T19:56:30.603 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.603 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.603 DEBUG:boto:Params: {} 2024-04-16T19:56:30.603 DEBUG:boto:Token: None 2024-04-16T19:56:30.604 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIi8tD/eI1Oe1RZBycMi2KU7BKM= 2024-04-16T19:56:30.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIi8tD/eI1Oe1RZBycMi2KU7BKM=', 'Content-Length': '0'} 2024-04-16T19:56:30.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcedf3592052977a-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.606 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.606 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.606 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.606 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.606 DEBUG:boto:Data: 2024-04-16T19:56:30.607 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.607 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.607 DEBUG:boto:Params: {} 2024-04-16T19:56:30.607 DEBUG:boto:Token: None 2024-04-16T19:56:30.607 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIi8tD/eI1Oe1RZBycMi2KU7BKM= 2024-04-16T19:56:30.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIi8tD/eI1Oe1RZBycMi2KU7BKM=', 'Content-Length': '0'} 2024-04-16T19:56:30.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e1fe1a705b6790-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-16T19:56:30.609 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.609 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.610 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.610 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.610 DEBUG:boto:Method: GET 2024-04-16T19:56:30.610 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.610 DEBUG:boto:Data: 2024-04-16T19:56:30.610 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.610 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.610 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.610 DEBUG:boto:Params: {} 2024-04-16T19:56:30.610 DEBUG:boto:Token: None 2024-04-16T19:56:30.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA= 2024-04-16T19:56:30.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA=', 'Content-Length': '0'} 2024-04-16T19:56:30.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830af230fb1b671f-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.613 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.614 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.614 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.614 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.614 DEBUG:boto:Method: GET 2024-04-16T19:56:30.614 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.614 DEBUG:boto:Data: 2024-04-16T19:56:30.614 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.614 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.614 DEBUG:boto:Params: {} 2024-04-16T19:56:30.615 DEBUG:boto:Token: None 2024-04-16T19:56:30.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162?versionId=null 2024-04-16T19:56:30.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA= 2024-04-16T19:56:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F/AoCZOusAXZh11NO9QY4DTqlxA=', 'Content-Length': '0'} 2024-04-16T19:56:30.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f629eadb9b500495-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.617 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.617 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.617 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.617 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.618 DEBUG:boto:Method: GET 2024-04-16T19:56:30.618 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.618 DEBUG:boto:Data: 2024-04-16T19:56:30.618 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.618 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.618 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.618 DEBUG:boto:Params: {} 2024-04-16T19:56:30.618 DEBUG:boto:Token: None 2024-04-16T19:56:30.618 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.618 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb89ca4638846e5-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.621 DEBUG:boto:b'yrjefh-51obj000000000000000001621000falseobj000000000000000001622024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.622 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.622 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.622 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.622 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.622 DEBUG:boto:Method: GET 2024-04-16T19:56:30.622 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.622 DEBUG:boto:Data: 2024-04-16T19:56:30.622 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.622 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.622 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.623 DEBUG:boto:Params: {} 2024-04-16T19:56:30.623 DEBUG:boto:Token: None 2024-04-16T19:56:30.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efbcf3547d265bf8-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.627 DEBUG:boto:b'yrjefh-51obj000000000000000001621000falseobj000000000000000001622024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.627 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.627 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.627 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.627 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.627 DEBUG:boto:Data: 2024-04-16T19:56:30.627 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.627 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.628 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.628 DEBUG:boto:Params: {} 2024-04-16T19:56:30.628 DEBUG:boto:Token: None 2024-04-16T19:56:30.628 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oTpTOctWRCCrwS8JUmcNUMTzuro= 2024-04-16T19:56:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oTpTOctWRCCrwS8JUmcNUMTzuro=', 'Content-Length': '0'} 2024-04-16T19:56:30.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ffd059c59046fda-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.630 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.630 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.630 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.631 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.631 DEBUG:boto:Data: 2024-04-16T19:56:30.631 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.631 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.631 DEBUG:boto:Params: {} 2024-04-16T19:56:30.631 DEBUG:boto:Token: None 2024-04-16T19:56:30.631 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oTpTOctWRCCrwS8JUmcNUMTzuro= 2024-04-16T19:56:30.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oTpTOctWRCCrwS8JUmcNUMTzuro=', 'Content-Length': '0'} 2024-04-16T19:56:30.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4c60b3a1917861-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-16T19:56:30.634 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.634 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.634 DEBUG:boto:Method: GET 2024-04-16T19:56:30.634 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.634 DEBUG:boto:Data: 2024-04-16T19:56:30.634 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.634 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.635 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.635 DEBUG:boto:Params: {} 2024-04-16T19:56:30.635 DEBUG:boto:Token: None 2024-04-16T19:56:30.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbISjha31nyHwgQmtsgfbnzbcgw= 2024-04-16T19:56:30.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbISjha31nyHwgQmtsgfbnzbcgw=', 'Content-Length': '0'} 2024-04-16T19:56:30.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc32b8941b20cac8-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.637 DEBUG:boto:path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.638 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.638 DEBUG:boto:Method: GET 2024-04-16T19:56:30.638 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.638 DEBUG:boto:Data: 2024-04-16T19:56:30.638 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.638 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.638 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.638 DEBUG:boto:Params: {} 2024-04-16T19:56:30.638 DEBUG:boto:Token: None 2024-04-16T19:56:30.638 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000162 2024-04-16T19:56:30.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbISjha31nyHwgQmtsgfbnzbcgw= 2024-04-16T19:56:30.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbISjha31nyHwgQmtsgfbnzbcgw=', 'Content-Length': '0'} 2024-04-16T19:56:30.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec2c4f34f164c8f-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.641 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.641 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.641 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.641 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.641 DEBUG:boto:Method: GET 2024-04-16T19:56:30.642 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.642 DEBUG:boto:Data: 2024-04-16T19:56:30.642 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.642 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.642 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.642 DEBUG:boto:Params: {} 2024-04-16T19:56:30.642 DEBUG:boto:Token: None 2024-04-16T19:56:30.642 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e461ebc81ffa847-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.645 DEBUG:boto:b'yrjefh-51obj000000000000000001621000falseobj000000000000000001622024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.645 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.645 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.646 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.646 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.646 DEBUG:boto:Method: GET 2024-04-16T19:56:30.646 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000162 2024-04-16T19:56:30.646 DEBUG:boto:Data: 2024-04-16T19:56:30.646 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.646 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.646 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.646 DEBUG:boto:Params: {} 2024-04-16T19:56:30.646 DEBUG:boto:Token: None 2024-04-16T19:56:30.646 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef8b7fdba0c40fcb-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.651 DEBUG:boto:b'yrjefh-51obj000000000000000001621000falseobj000000000000000001622024-04-16T19:56:07.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-16T19:56:30.651 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.651 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.651 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.651 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.651 DEBUG:boto:Method: GET 2024-04-16T19:56:30.651 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.651 DEBUG:boto:Data: 2024-04-16T19:56:30.652 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.652 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.652 DEBUG:boto:Params: {} 2024-04-16T19:56:30.652 DEBUG:boto:Token: None 2024-04-16T19:56:30.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0= 2024-04-16T19:56:30.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0=', 'Content-Length': '0'} 2024-04-16T19:56:30.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d882d8678aa007-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.655 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.655 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.655 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.655 DEBUG:boto:Method: GET 2024-04-16T19:56:30.655 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.655 DEBUG:boto:Data: 2024-04-16T19:56:30.655 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.655 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.655 DEBUG:boto:Params: {} 2024-04-16T19:56:30.655 DEBUG:boto:Token: None 2024-04-16T19:56:30.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0= 2024-04-16T19:56:30.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0=', 'Content-Length': '0'} 2024-04-16T19:56:30.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777067338b4ef224-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.698 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.698 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.699 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.699 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.699 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.699 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.699 DEBUG:boto:Data: 2024-04-16T19:56:30.699 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.699 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.699 DEBUG:boto:Params: {} 2024-04-16T19:56:30.699 DEBUG:boto:Token: None 2024-04-16T19:56:30.699 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jQeqGXlNIaQM5Fpol2hst67vAl0= 2024-04-16T19:56:30.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jQeqGXlNIaQM5Fpol2hst67vAl0=', 'Content-Length': '0'} 2024-04-16T19:56:30.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de786460ea2e1bb9-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.701 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.702 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.702 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.702 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.702 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.702 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.702 DEBUG:boto:Data: 2024-04-16T19:56:30.702 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.702 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.702 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.702 DEBUG:boto:Params: {} 2024-04-16T19:56:30.703 DEBUG:boto:Token: None 2024-04-16T19:56:30.703 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jQeqGXlNIaQM5Fpol2hst67vAl0= 2024-04-16T19:56:30.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jQeqGXlNIaQM5Fpol2hst67vAl0=', 'Content-Length': '0'} 2024-04-16T19:56:30.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf7ac7612da4528-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-16T19:56:30.705 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.705 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.706 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.706 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.706 DEBUG:boto:Method: GET 2024-04-16T19:56:30.706 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.706 DEBUG:boto:Data: 2024-04-16T19:56:30.706 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.706 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.706 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.706 DEBUG:boto:Params: {} 2024-04-16T19:56:30.706 DEBUG:boto:Token: None 2024-04-16T19:56:30.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0= 2024-04-16T19:56:30.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0=', 'Content-Length': '0'} 2024-04-16T19:56:30.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077038e6538e6dc29-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.709 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.709 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.709 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.709 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.709 DEBUG:boto:Method: GET 2024-04-16T19:56:30.709 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.709 DEBUG:boto:Data: 2024-04-16T19:56:30.709 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.710 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.710 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.710 DEBUG:boto:Params: {} 2024-04-16T19:56:30.710 DEBUG:boto:Token: None 2024-04-16T19:56:30.710 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271?versionId=null 2024-04-16T19:56:30.710 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0= 2024-04-16T19:56:30.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2ChCY0m/753BaX5L8owxBij8aN0=', 'Content-Length': '0'} 2024-04-16T19:56:30.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c11066a5334b8b9-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.712 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.712 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.713 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.713 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.713 DEBUG:boto:Method: GET 2024-04-16T19:56:30.713 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.713 DEBUG:boto:Data: 2024-04-16T19:56:30.713 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.713 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.713 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.713 DEBUG:boto:Params: {} 2024-04-16T19:56:30.713 DEBUG:boto:Token: None 2024-04-16T19:56:30.713 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007beec8f983df8bfc-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.716 DEBUG:boto:b'yrjefh-51obj000000000000000002711000falseobj000000000000000002712024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.717 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.717 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.717 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.717 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.717 DEBUG:boto:Method: GET 2024-04-16T19:56:30.717 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.717 DEBUG:boto:Data: 2024-04-16T19:56:30.717 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.717 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.717 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.717 DEBUG:boto:Params: {} 2024-04-16T19:56:30.718 DEBUG:boto:Token: None 2024-04-16T19:56:30.718 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030fd172e2c05f2d7-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.722 DEBUG:boto:b'yrjefh-51obj000000000000000002711000falseobj000000000000000002712024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.722 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.722 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.722 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.722 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.722 DEBUG:boto:Data: 2024-04-16T19:56:30.722 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.723 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.723 DEBUG:boto:Params: {} 2024-04-16T19:56:30.723 DEBUG:boto:Token: None 2024-04-16T19:56:30.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDehJoNihfOjJf/4EDrwoNf/vXg= 2024-04-16T19:56:30.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDehJoNihfOjJf/4EDrwoNf/vXg=', 'Content-Length': '0'} 2024-04-16T19:56:30.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be0ee79290f870e-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.725 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.725 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.725 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.725 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.725 DEBUG:boto:Data: 2024-04-16T19:56:30.725 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.726 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.726 DEBUG:boto:Params: {} 2024-04-16T19:56:30.726 DEBUG:boto:Token: None 2024-04-16T19:56:30.726 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.726 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDehJoNihfOjJf/4EDrwoNf/vXg= 2024-04-16T19:56:30.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDehJoNihfOjJf/4EDrwoNf/vXg=', 'Content-Length': '0'} 2024-04-16T19:56:30.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e088d4d8ec82301-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-16T19:56:30.728 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.729 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.729 DEBUG:boto:Method: GET 2024-04-16T19:56:30.729 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.729 DEBUG:boto:Data: 2024-04-16T19:56:30.729 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.729 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.729 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.729 DEBUG:boto:Params: {} 2024-04-16T19:56:30.729 DEBUG:boto:Token: None 2024-04-16T19:56:30.729 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzJzzR9wrvz2v6tURZqJwV6pYVI= 2024-04-16T19:56:30.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzJzzR9wrvz2v6tURZqJwV6pYVI=', 'Content-Length': '0'} 2024-04-16T19:56:30.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c420224c22da27a1-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.732 DEBUG:boto:path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.732 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.732 DEBUG:boto:Method: GET 2024-04-16T19:56:30.732 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.732 DEBUG:boto:Data: 2024-04-16T19:56:30.732 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.732 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.732 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.732 DEBUG:boto:Params: {} 2024-04-16T19:56:30.732 DEBUG:boto:Token: None 2024-04-16T19:56:30.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000271 2024-04-16T19:56:30.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QzJzzR9wrvz2v6tURZqJwV6pYVI= 2024-04-16T19:56:30.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QzJzzR9wrvz2v6tURZqJwV6pYVI=', 'Content-Length': '0'} 2024-04-16T19:56:30.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ced8052655179549-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.735 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.735 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.735 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.735 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.735 DEBUG:boto:Method: GET 2024-04-16T19:56:30.735 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.735 DEBUG:boto:Data: 2024-04-16T19:56:30.735 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.735 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.735 DEBUG:boto:Params: {} 2024-04-16T19:56:30.735 DEBUG:boto:Token: None 2024-04-16T19:56:30.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405e2ba34b8e4ebc-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.778 DEBUG:boto:b'yrjefh-51obj000000000000000002711000falseobj000000000000000002712024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.779 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.779 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.779 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.779 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.779 DEBUG:boto:Method: GET 2024-04-16T19:56:30.779 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000271 2024-04-16T19:56:30.779 DEBUG:boto:Data: 2024-04-16T19:56:30.779 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.779 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.780 DEBUG:boto:Params: {} 2024-04-16T19:56:30.780 DEBUG:boto:Token: None 2024-04-16T19:56:30.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058face3c9009f4da-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.784 DEBUG:boto:b'yrjefh-51obj000000000000000002711000falseobj000000000000000002712024-04-16T19:56:07.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-16T19:56:30.784 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.784 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.785 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.785 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.785 DEBUG:boto:Method: GET 2024-04-16T19:56:30.785 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.785 DEBUG:boto:Data: 2024-04-16T19:56:30.785 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.785 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.785 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.785 DEBUG:boto:Params: {} 2024-04-16T19:56:30.785 DEBUG:boto:Token: None 2024-04-16T19:56:30.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.785 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ= 2024-04-16T19:56:30.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ=', 'Content-Length': '0'} 2024-04-16T19:56:30.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf7fed9511bc76b-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.788 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.788 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.788 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.788 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.788 DEBUG:boto:Method: GET 2024-04-16T19:56:30.788 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.788 DEBUG:boto:Data: 2024-04-16T19:56:30.789 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.789 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.789 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.789 DEBUG:boto:Params: {} 2024-04-16T19:56:30.789 DEBUG:boto:Token: None 2024-04-16T19:56:30.789 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.789 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ= 2024-04-16T19:56:30.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ=', 'Content-Length': '0'} 2024-04-16T19:56:30.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a361d7984bb7738-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.791 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.791 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.791 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.791 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.792 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.792 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.792 DEBUG:boto:Data: 2024-04-16T19:56:30.792 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.792 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.792 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.792 DEBUG:boto:Params: {} 2024-04-16T19:56:30.792 DEBUG:boto:Token: None 2024-04-16T19:56:30.792 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:du0ROfUvWPSiXmVYdYDt+SWTxRw= 2024-04-16T19:56:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:du0ROfUvWPSiXmVYdYDt+SWTxRw=', 'Content-Length': '0'} 2024-04-16T19:56:30.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b6889778a95b3d-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.795 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.795 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.795 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.795 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.795 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.795 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.795 DEBUG:boto:Data: 2024-04-16T19:56:30.795 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.795 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.795 DEBUG:boto:Params: {} 2024-04-16T19:56:30.795 DEBUG:boto:Token: None 2024-04-16T19:56:30.796 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:du0ROfUvWPSiXmVYdYDt+SWTxRw= 2024-04-16T19:56:30.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:du0ROfUvWPSiXmVYdYDt+SWTxRw=', 'Content-Length': '0'} 2024-04-16T19:56:30.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a435ff091be037-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-16T19:56:30.798 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.798 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.798 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.798 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.798 DEBUG:boto:Method: GET 2024-04-16T19:56:30.798 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.798 DEBUG:boto:Data: 2024-04-16T19:56:30.798 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.798 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.799 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.799 DEBUG:boto:Params: {} 2024-04-16T19:56:30.799 DEBUG:boto:Token: None 2024-04-16T19:56:30.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ= 2024-04-16T19:56:30.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ=', 'Content-Length': '0'} 2024-04-16T19:56:30.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddca3b2218bb3376-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.842 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.843 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.843 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.843 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.843 DEBUG:boto:Method: GET 2024-04-16T19:56:30.843 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.843 DEBUG:boto:Data: 2024-04-16T19:56:30.843 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.843 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.843 DEBUG:boto:Params: {} 2024-04-16T19:56:30.844 DEBUG:boto:Token: None 2024-04-16T19:56:30.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283?versionId=null 2024-04-16T19:56:30.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ= 2024-04-16T19:56:30.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fZ6GL9UQIZ0FXN1C/qHpD/YXtQQ=', 'Content-Length': '0'} 2024-04-16T19:56:30.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023dd7684b1aab7e2-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.846 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.846 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.846 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.847 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.847 DEBUG:boto:Method: GET 2024-04-16T19:56:30.847 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.847 DEBUG:boto:Data: 2024-04-16T19:56:30.847 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.847 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.847 DEBUG:boto:Params: {} 2024-04-16T19:56:30.847 DEBUG:boto:Token: None 2024-04-16T19:56:30.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c113ea0f043957-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.850 DEBUG:boto:b'yrjefh-51obj000000000000000002831000falseobj000000000000000002832024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.850 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.851 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.851 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.851 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.851 DEBUG:boto:Method: GET 2024-04-16T19:56:30.851 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.851 DEBUG:boto:Data: 2024-04-16T19:56:30.851 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.851 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.851 DEBUG:boto:Params: {} 2024-04-16T19:56:30.851 DEBUG:boto:Token: None 2024-04-16T19:56:30.851 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fbddc5f54a6f450-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.855 DEBUG:boto:b'yrjefh-51obj000000000000000002831000falseobj000000000000000002832024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.856 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.856 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.856 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.856 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.856 DEBUG:boto:Data: 2024-04-16T19:56:30.856 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.856 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.856 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.856 DEBUG:boto:Params: {} 2024-04-16T19:56:30.856 DEBUG:boto:Token: None 2024-04-16T19:56:30.857 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.857 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LNFUw/ehqHTSMbRIJR6iyvYgOu8= 2024-04-16T19:56:30.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LNFUw/ehqHTSMbRIJR6iyvYgOu8=', 'Content-Length': '0'} 2024-04-16T19:56:30.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009deabf8dc5e37692-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.859 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.859 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.859 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.859 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.859 DEBUG:boto:Data: 2024-04-16T19:56:30.859 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.859 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.859 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.860 DEBUG:boto:Params: {} 2024-04-16T19:56:30.860 DEBUG:boto:Token: None 2024-04-16T19:56:30.860 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LNFUw/ehqHTSMbRIJR6iyvYgOu8= 2024-04-16T19:56:30.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LNFUw/ehqHTSMbRIJR6iyvYgOu8=', 'Content-Length': '0'} 2024-04-16T19:56:30.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ffd064cf985ff5-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-16T19:56:30.862 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.862 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.862 DEBUG:boto:Method: GET 2024-04-16T19:56:30.863 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.863 DEBUG:boto:Data: 2024-04-16T19:56:30.863 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.863 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.863 DEBUG:boto:Params: {} 2024-04-16T19:56:30.863 DEBUG:boto:Token: None 2024-04-16T19:56:30.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L5UKjOf6MfRWFHrs27howw0Zlog= 2024-04-16T19:56:30.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5UKjOf6MfRWFHrs27howw0Zlog=', 'Content-Length': '0'} 2024-04-16T19:56:30.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c43d25de0424e68-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.865 DEBUG:boto:path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.866 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.866 DEBUG:boto:Method: GET 2024-04-16T19:56:30.866 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.866 DEBUG:boto:Data: 2024-04-16T19:56:30.866 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.866 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.866 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.866 DEBUG:boto:Params: {} 2024-04-16T19:56:30.866 DEBUG:boto:Token: None 2024-04-16T19:56:30.866 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000283 2024-04-16T19:56:30.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L5UKjOf6MfRWFHrs27howw0Zlog= 2024-04-16T19:56:30.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5UKjOf6MfRWFHrs27howw0Zlog=', 'Content-Length': '0'} 2024-04-16T19:56:30.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1574643cc329349-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.869 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.869 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.869 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.869 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.869 DEBUG:boto:Method: GET 2024-04-16T19:56:30.869 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.869 DEBUG:boto:Data: 2024-04-16T19:56:30.869 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.869 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.870 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.870 DEBUG:boto:Params: {} 2024-04-16T19:56:30.870 DEBUG:boto:Token: None 2024-04-16T19:56:30.870 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.870 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009024ce270023ebab-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.873 DEBUG:boto:b'yrjefh-51obj000000000000000002831000falseobj000000000000000002832024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.873 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.873 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.873 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.873 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.873 DEBUG:boto:Method: GET 2024-04-16T19:56:30.873 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000283 2024-04-16T19:56:30.873 DEBUG:boto:Data: 2024-04-16T19:56:30.873 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.874 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.874 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.874 DEBUG:boto:Params: {} 2024-04-16T19:56:30.874 DEBUG:boto:Token: None 2024-04-16T19:56:30.874 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.874 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009e0f688f932ffd-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.878 DEBUG:boto:b'yrjefh-51obj000000000000000002831000falseobj000000000000000002832024-04-16T19:56:07.662Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-16T19:56:30.879 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.879 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.879 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.879 DEBUG:boto:Method: GET 2024-04-16T19:56:30.879 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.879 DEBUG:boto:Data: 2024-04-16T19:56:30.879 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.879 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.879 DEBUG:boto:Params: {} 2024-04-16T19:56:30.880 DEBUG:boto:Token: None 2024-04-16T19:56:30.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0= 2024-04-16T19:56:30.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0=', 'Content-Length': '0'} 2024-04-16T19:56:30.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b064537853dd319-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.883 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.883 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.883 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.883 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.883 DEBUG:boto:Method: GET 2024-04-16T19:56:30.883 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.883 DEBUG:boto:Data: 2024-04-16T19:56:30.883 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.883 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.883 DEBUG:boto:Params: {} 2024-04-16T19:56:30.883 DEBUG:boto:Token: None 2024-04-16T19:56:30.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0= 2024-04-16T19:56:30.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0=', 'Content-Length': '0'} 2024-04-16T19:56:30.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93106d5deaeeb21-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.886 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.886 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.886 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.887 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.887 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.887 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.887 DEBUG:boto:Data: 2024-04-16T19:56:30.887 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.887 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.887 DEBUG:boto:Params: {} 2024-04-16T19:56:30.887 DEBUG:boto:Token: None 2024-04-16T19:56:30.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W6QqQejxSoD6nTXJ90Ig2JyTrRU= 2024-04-16T19:56:30.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W6QqQejxSoD6nTXJ90Ig2JyTrRU=', 'Content-Length': '0'} 2024-04-16T19:56:30.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099cb148ee03b87a1-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.890 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.890 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.890 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.890 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.891 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.891 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.891 DEBUG:boto:Data: 2024-04-16T19:56:30.891 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.891 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.891 DEBUG:boto:Params: {} 2024-04-16T19:56:30.891 DEBUG:boto:Token: None 2024-04-16T19:56:30.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W6QqQejxSoD6nTXJ90Ig2JyTrRU= 2024-04-16T19:56:30.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W6QqQejxSoD6nTXJ90Ig2JyTrRU=', 'Content-Length': '0'} 2024-04-16T19:56:30.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065adcb39d33cdba8-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-16T19:56:30.894 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.894 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.894 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.894 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.894 DEBUG:boto:Method: GET 2024-04-16T19:56:30.894 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.894 DEBUG:boto:Data: 2024-04-16T19:56:30.894 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.895 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.895 DEBUG:boto:Params: {} 2024-04-16T19:56:30.895 DEBUG:boto:Token: None 2024-04-16T19:56:30.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0= 2024-04-16T19:56:30.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0=', 'Content-Length': '0'} 2024-04-16T19:56:30.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669dd497ed6aed70-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.938 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.939 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.939 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.939 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.939 DEBUG:boto:Method: GET 2024-04-16T19:56:30.939 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.939 DEBUG:boto:Data: 2024-04-16T19:56:30.939 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.939 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.939 DEBUG:boto:Params: {} 2024-04-16T19:56:30.939 DEBUG:boto:Token: None 2024-04-16T19:56:30.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317?versionId=null 2024-04-16T19:56:30.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0= 2024-04-16T19:56:30.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+SqdoYca8URTvjva0vzK1wDIWb0=', 'Content-Length': '0'} 2024-04-16T19:56:30.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393f975396d236a7-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.942 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.942 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.943 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.943 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.943 DEBUG:boto:Method: GET 2024-04-16T19:56:30.943 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.943 DEBUG:boto:Data: 2024-04-16T19:56:30.943 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.943 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.943 DEBUG:boto:Params: {} 2024-04-16T19:56:30.943 DEBUG:boto:Token: None 2024-04-16T19:56:30.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ad6c9780b6ab18-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.946 DEBUG:boto:b'yrjefh-51obj000000000000000003171000falseobj000000000000000003172024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.946 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:30.946 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:30.947 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.947 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.947 DEBUG:boto:Method: GET 2024-04-16T19:56:30.947 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:30.947 DEBUG:boto:Data: 2024-04-16T19:56:30.947 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.947 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.947 DEBUG:boto:Params: {} 2024-04-16T19:56:30.947 DEBUG:boto:Token: None 2024-04-16T19:56:30.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/ 2024-04-16T19:56:30.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc= 2024-04-16T19:56:30.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PHeXhFrYjdCG9LJLbwQLzcxl6uc=', 'Content-Length': '0'} 2024-04-16T19:56:30.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ac199988d902f25-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.952 DEBUG:boto:b'yrjefh-51obj000000000000000003171000falseobj000000000000000003172024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:30.952 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.952 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.952 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.952 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.952 DEBUG:boto:Data: 2024-04-16T19:56:30.952 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.953 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.953 DEBUG:boto:Params: {} 2024-04-16T19:56:30.953 DEBUG:boto:Token: None 2024-04-16T19:56:30.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0PQud5gbu0zzCW6m/HXSxcpGV4M= 2024-04-16T19:56:30.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0PQud5gbu0zzCW6m/HXSxcpGV4M=', 'Content-Length': '0'} 2024-04-16T19:56:30.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dde462a8040d51-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.955 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.955 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.956 DEBUG:boto:Method: HEAD 2024-04-16T19:56:30.956 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.956 DEBUG:boto:Data: 2024-04-16T19:56:30.956 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.956 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.956 DEBUG:boto:Params: {} 2024-04-16T19:56:30.956 DEBUG:boto:Token: None 2024-04-16T19:56:30.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0PQud5gbu0zzCW6m/HXSxcpGV4M= 2024-04-16T19:56:30.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0PQud5gbu0zzCW6m/HXSxcpGV4M=', 'Content-Length': '0'} 2024-04-16T19:56:30.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfbdfd9da5c978d-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-16T19:56:30.959 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.959 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.959 DEBUG:boto:Method: GET 2024-04-16T19:56:30.959 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.959 DEBUG:boto:Data: 2024-04-16T19:56:30.959 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:30.959 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.960 DEBUG:boto:Params: {} 2024-04-16T19:56:30.960 DEBUG:boto:Token: None 2024-04-16T19:56:30.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WNnC6inz0J+MBPdQUSScMMqu+JI= 2024-04-16T19:56:30.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WNnC6inz0J+MBPdQUSScMMqu+JI=', 'Content-Length': '0'} 2024-04-16T19:56:30.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f443c12f4798566-00661ed7ee-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:30.963 DEBUG:boto:path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.963 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.963 DEBUG:boto:Method: GET 2024-04-16T19:56:30.963 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.963 DEBUG:boto:Data: 2024-04-16T19:56:30.963 DEBUG:boto:Headers: {} 2024-04-16T19:56:30.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:30.963 DEBUG:boto:Port: 8000 2024-04-16T19:56:30.963 DEBUG:boto:Params: {} 2024-04-16T19:56:30.963 DEBUG:boto:Token: None 2024-04-16T19:56:30.963 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:30 GMT /yrjefh-51/obj00000000000000000317 2024-04-16T19:56:30.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WNnC6inz0J+MBPdQUSScMMqu+JI= 2024-04-16T19:56:30.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WNnC6inz0J+MBPdQUSScMMqu+JI=', 'Content-Length': '0'} 2024-04-16T19:56:30.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5a70c26dda7d48-00661ed7ee-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.006 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.006 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.007 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.007 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.007 DEBUG:boto:Method: GET 2024-04-16T19:56:31.007 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.007 DEBUG:boto:Data: 2024-04-16T19:56:31.007 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.007 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.007 DEBUG:boto:Params: {} 2024-04-16T19:56:31.007 DEBUG:boto:Token: None 2024-04-16T19:56:31.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d443fadd25090519-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.011 DEBUG:boto:b'yrjefh-51obj000000000000000003171000falseobj000000000000000003172024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.011 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.011 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.011 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.011 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.011 DEBUG:boto:Method: GET 2024-04-16T19:56:31.011 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000317 2024-04-16T19:56:31.011 DEBUG:boto:Data: 2024-04-16T19:56:31.011 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.012 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.012 DEBUG:boto:Params: {} 2024-04-16T19:56:31.012 DEBUG:boto:Token: None 2024-04-16T19:56:31.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b37241b158484d-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.016 DEBUG:boto:b'yrjefh-51obj000000000000000003171000falseobj000000000000000003172024-04-16T19:56:08.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-16T19:56:31.016 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.016 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.017 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.017 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.017 DEBUG:boto:Method: GET 2024-04-16T19:56:31.017 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.017 DEBUG:boto:Data: 2024-04-16T19:56:31.017 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.017 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.017 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.017 DEBUG:boto:Params: {} 2024-04-16T19:56:31.017 DEBUG:boto:Token: None 2024-04-16T19:56:31.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw= 2024-04-16T19:56:31.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw=', 'Content-Length': '0'} 2024-04-16T19:56:31.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046dae5de7ee89d17-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.066 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.066 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.067 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.067 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.067 DEBUG:boto:Method: GET 2024-04-16T19:56:31.067 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.067 DEBUG:boto:Data: 2024-04-16T19:56:31.067 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.067 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.067 DEBUG:boto:Params: {} 2024-04-16T19:56:31.067 DEBUG:boto:Token: None 2024-04-16T19:56:31.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw= 2024-04-16T19:56:31.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw=', 'Content-Length': '0'} 2024-04-16T19:56:31.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fee7a79a39df2cc-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.070 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.070 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.070 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.070 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.070 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.070 DEBUG:boto:Data: 2024-04-16T19:56:31.070 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.070 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.070 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.071 DEBUG:boto:Params: {} 2024-04-16T19:56:31.071 DEBUG:boto:Token: None 2024-04-16T19:56:31.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eHBAA7ZUg7Tg2Oi/L6TPHqAoTbQ= 2024-04-16T19:56:31.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eHBAA7ZUg7Tg2Oi/L6TPHqAoTbQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b264a8df9385f347-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.073 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.073 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.074 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.074 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.074 DEBUG:boto:Data: 2024-04-16T19:56:31.074 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.074 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.074 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.074 DEBUG:boto:Params: {} 2024-04-16T19:56:31.074 DEBUG:boto:Token: None 2024-04-16T19:56:31.074 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eHBAA7ZUg7Tg2Oi/L6TPHqAoTbQ= 2024-04-16T19:56:31.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eHBAA7ZUg7Tg2Oi/L6TPHqAoTbQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4a4e3e7384c6ff7-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-16T19:56:31.076 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.077 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.077 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.077 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.077 DEBUG:boto:Method: GET 2024-04-16T19:56:31.077 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.077 DEBUG:boto:Data: 2024-04-16T19:56:31.077 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.077 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.077 DEBUG:boto:Params: {} 2024-04-16T19:56:31.077 DEBUG:boto:Token: None 2024-04-16T19:56:31.077 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.078 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw= 2024-04-16T19:56:31.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw=', 'Content-Length': '0'} 2024-04-16T19:56:31.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b18ad8e35f3dd33-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.080 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.080 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.080 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.081 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.081 DEBUG:boto:Method: GET 2024-04-16T19:56:31.081 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.081 DEBUG:boto:Data: 2024-04-16T19:56:31.081 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.081 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.081 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.081 DEBUG:boto:Params: {} 2024-04-16T19:56:31.081 DEBUG:boto:Token: None 2024-04-16T19:56:31.081 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423?versionId=null 2024-04-16T19:56:31.081 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw= 2024-04-16T19:56:31.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3npmkY+mTUExRBZIUHFESqYhYxw=', 'Content-Length': '0'} 2024-04-16T19:56:31.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b20cbbf9fd9f35-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.084 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.084 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.084 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.084 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.084 DEBUG:boto:Method: GET 2024-04-16T19:56:31.084 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.084 DEBUG:boto:Data: 2024-04-16T19:56:31.084 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.085 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.085 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.085 DEBUG:boto:Params: {} 2024-04-16T19:56:31.085 DEBUG:boto:Token: None 2024-04-16T19:56:31.085 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ae4a32605cb3b7-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.088 DEBUG:boto:b'yrjefh-51obj000000000000000004231000falseobj000000000000000004232024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.088 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.089 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.089 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.089 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.089 DEBUG:boto:Method: GET 2024-04-16T19:56:31.089 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.089 DEBUG:boto:Data: 2024-04-16T19:56:31.089 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.089 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.089 DEBUG:boto:Params: {} 2024-04-16T19:56:31.089 DEBUG:boto:Token: None 2024-04-16T19:56:31.090 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055fe761611fcc329-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.094 DEBUG:boto:b'yrjefh-51obj000000000000000004231000falseobj000000000000000004232024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.094 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.094 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.094 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.094 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.094 DEBUG:boto:Data: 2024-04-16T19:56:31.094 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.095 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.095 DEBUG:boto:Params: {} 2024-04-16T19:56:31.095 DEBUG:boto:Token: None 2024-04-16T19:56:31.095 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s7S4yH1GEjIUdCbHN21dkpImrEw= 2024-04-16T19:56:31.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s7S4yH1GEjIUdCbHN21dkpImrEw=', 'Content-Length': '0'} 2024-04-16T19:56:31.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9de1b60423b140-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.097 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.098 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.098 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.098 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.098 DEBUG:boto:Data: 2024-04-16T19:56:31.098 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.098 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.098 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.098 DEBUG:boto:Params: {} 2024-04-16T19:56:31.098 DEBUG:boto:Token: None 2024-04-16T19:56:31.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s7S4yH1GEjIUdCbHN21dkpImrEw= 2024-04-16T19:56:31.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s7S4yH1GEjIUdCbHN21dkpImrEw=', 'Content-Length': '0'} 2024-04-16T19:56:31.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380f010be1e02e44-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-16T19:56:31.101 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.101 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.101 DEBUG:boto:Method: GET 2024-04-16T19:56:31.101 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.101 DEBUG:boto:Data: 2024-04-16T19:56:31.101 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.101 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.101 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.101 DEBUG:boto:Params: {} 2024-04-16T19:56:31.102 DEBUG:boto:Token: None 2024-04-16T19:56:31.102 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bpOUrQsbxyNAxdfaichkya9T8dY= 2024-04-16T19:56:31.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bpOUrQsbxyNAxdfaichkya9T8dY=', 'Content-Length': '0'} 2024-04-16T19:56:31.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f4a9c9c99b5ac12-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.104 DEBUG:boto:path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.104 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.105 DEBUG:boto:Method: GET 2024-04-16T19:56:31.105 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.105 DEBUG:boto:Data: 2024-04-16T19:56:31.105 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.105 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.105 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.105 DEBUG:boto:Params: {} 2024-04-16T19:56:31.105 DEBUG:boto:Token: None 2024-04-16T19:56:31.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000423 2024-04-16T19:56:31.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bpOUrQsbxyNAxdfaichkya9T8dY= 2024-04-16T19:56:31.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bpOUrQsbxyNAxdfaichkya9T8dY=', 'Content-Length': '0'} 2024-04-16T19:56:31.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9df341b7231ee4b-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.108 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.108 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.108 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.108 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.108 DEBUG:boto:Method: GET 2024-04-16T19:56:31.108 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.108 DEBUG:boto:Data: 2024-04-16T19:56:31.108 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.109 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.109 DEBUG:boto:Params: {} 2024-04-16T19:56:31.109 DEBUG:boto:Token: None 2024-04-16T19:56:31.109 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab54222413cd0755-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.112 DEBUG:boto:b'yrjefh-51obj000000000000000004231000falseobj000000000000000004232024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.112 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.112 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.112 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.112 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.112 DEBUG:boto:Method: GET 2024-04-16T19:56:31.112 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000423 2024-04-16T19:56:31.112 DEBUG:boto:Data: 2024-04-16T19:56:31.113 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.113 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.113 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.113 DEBUG:boto:Params: {} 2024-04-16T19:56:31.113 DEBUG:boto:Token: None 2024-04-16T19:56:31.113 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009485806ebf5250f9-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.158 DEBUG:boto:b'yrjefh-51obj000000000000000004231000falseobj000000000000000004232024-04-16T19:56:07.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-16T19:56:31.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.159 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.159 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.159 DEBUG:boto:Method: GET 2024-04-16T19:56:31.159 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.159 DEBUG:boto:Data: 2024-04-16T19:56:31.159 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.159 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.159 DEBUG:boto:Params: {} 2024-04-16T19:56:31.160 DEBUG:boto:Token: None 2024-04-16T19:56:31.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ= 2024-04-16T19:56:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b04b1c47653b022-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.163 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.163 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.163 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.163 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.163 DEBUG:boto:Method: GET 2024-04-16T19:56:31.163 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.163 DEBUG:boto:Data: 2024-04-16T19:56:31.163 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.163 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.163 DEBUG:boto:Params: {} 2024-04-16T19:56:31.163 DEBUG:boto:Token: None 2024-04-16T19:56:31.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ= 2024-04-16T19:56:31.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197f42c77df5bff8-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.166 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.166 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.166 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.166 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.166 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.166 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.166 DEBUG:boto:Data: 2024-04-16T19:56:31.166 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.167 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.167 DEBUG:boto:Params: {} 2024-04-16T19:56:31.167 DEBUG:boto:Token: None 2024-04-16T19:56:31.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QhYF/Q4QEKmmGC0SA8iVssulTWc= 2024-04-16T19:56:31.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QhYF/Q4QEKmmGC0SA8iVssulTWc=', 'Content-Length': '0'} 2024-04-16T19:56:31.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e44bfef2e2c0f2-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.169 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.170 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.170 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.170 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.170 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.170 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.170 DEBUG:boto:Data: 2024-04-16T19:56:31.170 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.170 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.170 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.170 DEBUG:boto:Params: {} 2024-04-16T19:56:31.170 DEBUG:boto:Token: None 2024-04-16T19:56:31.171 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QhYF/Q4QEKmmGC0SA8iVssulTWc= 2024-04-16T19:56:31.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QhYF/Q4QEKmmGC0SA8iVssulTWc=', 'Content-Length': '0'} 2024-04-16T19:56:31.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d9a2c4ed3633dc-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-16T19:56:31.173 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.173 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.173 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.173 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.173 DEBUG:boto:Method: GET 2024-04-16T19:56:31.174 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.174 DEBUG:boto:Data: 2024-04-16T19:56:31.174 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.174 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.174 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.174 DEBUG:boto:Params: {} 2024-04-16T19:56:31.174 DEBUG:boto:Token: None 2024-04-16T19:56:31.174 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ= 2024-04-16T19:56:31.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c69b5882e746649-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.222 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.223 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.223 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.223 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.223 DEBUG:boto:Method: GET 2024-04-16T19:56:31.223 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.223 DEBUG:boto:Data: 2024-04-16T19:56:31.223 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.223 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.223 DEBUG:boto:Params: {} 2024-04-16T19:56:31.223 DEBUG:boto:Token: None 2024-04-16T19:56:31.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467?versionId=null 2024-04-16T19:56:31.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ= 2024-04-16T19:56:31.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MS+tsNRraO+ybhd8ccTpu2Pf1oQ=', 'Content-Length': '0'} 2024-04-16T19:56:31.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f19ea3aa05c02bc-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.226 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.227 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.227 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.227 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.227 DEBUG:boto:Method: GET 2024-04-16T19:56:31.227 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.227 DEBUG:boto:Data: 2024-04-16T19:56:31.227 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.227 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.227 DEBUG:boto:Params: {} 2024-04-16T19:56:31.227 DEBUG:boto:Token: None 2024-04-16T19:56:31.228 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcbcd3a76ee757d3-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.230 DEBUG:boto:b'yrjefh-51obj000000000000000004671000falseobj000000000000000004672024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.231 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.231 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.231 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.231 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.231 DEBUG:boto:Method: GET 2024-04-16T19:56:31.231 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.231 DEBUG:boto:Data: 2024-04-16T19:56:31.231 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.231 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.231 DEBUG:boto:Params: {} 2024-04-16T19:56:31.232 DEBUG:boto:Token: None 2024-04-16T19:56:31.232 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008618f9720adfc60b-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.236 DEBUG:boto:b'yrjefh-51obj000000000000000004671000falseobj000000000000000004672024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.236 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.236 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.236 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.236 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.237 DEBUG:boto:Data: 2024-04-16T19:56:31.237 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.237 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.237 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.237 DEBUG:boto:Params: {} 2024-04-16T19:56:31.237 DEBUG:boto:Token: None 2024-04-16T19:56:31.237 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VztpIFI3ccHmpkC7clGej4qDbL0= 2024-04-16T19:56:31.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VztpIFI3ccHmpkC7clGej4qDbL0=', 'Content-Length': '0'} 2024-04-16T19:56:31.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7eaa4399be0a524-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.239 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.239 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.240 DEBUG:boto:Method: HEAD 2024-04-16T19:56:31.240 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.240 DEBUG:boto:Data: 2024-04-16T19:56:31.240 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.240 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.240 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.240 DEBUG:boto:Params: {} 2024-04-16T19:56:31.240 DEBUG:boto:Token: None 2024-04-16T19:56:31.240 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VztpIFI3ccHmpkC7clGej4qDbL0= 2024-04-16T19:56:31.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VztpIFI3ccHmpkC7clGej4qDbL0=', 'Content-Length': '0'} 2024-04-16T19:56:31.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577ce1a9b194997a-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-16T19:56:31.243 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.243 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.243 DEBUG:boto:Method: GET 2024-04-16T19:56:31.243 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.243 DEBUG:boto:Data: 2024-04-16T19:56:31.243 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.243 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.243 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.243 DEBUG:boto:Params: {} 2024-04-16T19:56:31.244 DEBUG:boto:Token: None 2024-04-16T19:56:31.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gbf6dXt3WSU7uDnWNwgVTkI23HU= 2024-04-16T19:56:31.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gbf6dXt3WSU7uDnWNwgVTkI23HU=', 'Content-Length': '0'} 2024-04-16T19:56:31.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-51:b482a2a9-ef36-488b-9666-ada656453328.4812.20'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:56:10 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d3270f9c07fa14-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.290 DEBUG:boto:path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.290 DEBUG:boto:auth_path=/yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.291 DEBUG:boto:Method: GET 2024-04-16T19:56:31.291 DEBUG:boto:Path: /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.291 DEBUG:boto:Data: 2024-04-16T19:56:31.291 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.291 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.291 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.291 DEBUG:boto:Params: {} 2024-04-16T19:56:31.291 DEBUG:boto:Token: None 2024-04-16T19:56:31.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/obj00000000000000000467 2024-04-16T19:56:31.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gbf6dXt3WSU7uDnWNwgVTkI23HU= 2024-04-16T19:56:31.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gbf6dXt3WSU7uDnWNwgVTkI23HU=', 'Content-Length': '0'} 2024-04-16T19:56:31.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:56:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c9161a30726bd5-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.294 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.294 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.295 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.295 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.295 DEBUG:boto:Method: GET 2024-04-16T19:56:31.295 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.295 DEBUG:boto:Data: 2024-04-16T19:56:31.295 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:56:31.295 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.295 DEBUG:boto:Params: {} 2024-04-16T19:56:31.295 DEBUG:boto:Token: None 2024-04-16T19:56:31.295 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1b0e752e3ab91b-00661ed7ef-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.298 DEBUG:boto:b'yrjefh-51obj000000000000000004671000falseobj000000000000000004672024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.299 DEBUG:boto:path=/yrjefh-51/ 2024-04-16T19:56:31.299 DEBUG:boto:auth_path=/yrjefh-51/ 2024-04-16T19:56:31.299 DEBUG:boto:path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.299 DEBUG:boto:auth_path=/yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.299 DEBUG:boto:Method: GET 2024-04-16T19:56:31.299 DEBUG:boto:Path: /yrjefh-51/?prefix=obj00000000000000000467 2024-04-16T19:56:31.299 DEBUG:boto:Data: 2024-04-16T19:56:31.300 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.300 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.300 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.300 DEBUG:boto:Params: {} 2024-04-16T19:56:31.300 DEBUG:boto:Token: None 2024-04-16T19:56:31.300 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-51/ 2024-04-16T19:56:31.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k= 2024-04-16T19:56:31.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UI1a4xkMfFeyAoPgkb8cnsaDF5k=', 'Content-Length': '0'} 2024-04-16T19:56:31.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005435d18e61cbdbaa-00661ed7ef-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.304 DEBUG:boto:b'yrjefh-51obj000000000000000004671000falseobj000000000000000004672024-04-16T19:56:07.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:56:31.305 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-51 zones={test-zone2, test-zone1} 2024-04-16T19:56:31.305 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2024-04-16T19:56:31.308 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:56:31.308 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:56:31.308 DEBUG:boto:Method: PUT 2024-04-16T19:56:31.308 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:56:31.308 DEBUG:boto:Data: 2024-04-16T19:56:31.308 DEBUG:boto:Headers: {} 2024-04-16T19:56:31.308 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:56:31.308 DEBUG:boto:Port: 8000 2024-04-16T19:56:31.308 DEBUG:boto:Params: {} 2024-04-16T19:56:31.309 DEBUG:boto:Token: None 2024-04-16T19:56:31.309 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 19:56:31 GMT /yrjefh-52/ 2024-04-16T19:56:31.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BEtrNrfT+fxG/6MdFnyKNDAuPs= 2024-04-16T19:56:31.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:56:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BEtrNrfT+fxG/6MdFnyKNDAuPs=', 'Content-Length': '0'} 2024-04-16T19:56:31.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b8b0a96fc9c81502-00661ed7ef-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:56:31.332 DEBUG:rgw_multi.tests:created bucket=yrjefh-52 2024-04-16T19:56:31.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:31.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:31.333 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:31.442 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T19:56:31.465 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713297391.331007_838.1","last_update":"2024-04-16T19:56:31.331007Z"}] 2024-04-16T19:56:31.466 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:31.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T19:56:31.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:31.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:31.466 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:31.522 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:31.522 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:31.576+0000 7f7bece16800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:31.578+0000 7f7bece16800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:31.600 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:31Z"} 2024-04-16T19:56:31.600 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:31Z"} 2024-04-16T19:56:31.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:31.600 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:31.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:31.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:56:36.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:36.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:36.602 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:36.657 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:36.657 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:36.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:36.712+0000 7f43bb92d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:36.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:36.713+0000 7f43bb92d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:36.735 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:36Z"} 2024-04-16T19:56:36.736 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:36Z"} 2024-04-16T19:56:36.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:36.736 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:36.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:36.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:56:41.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:41.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:41.738 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:41.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:41.854+0000 7fd925411800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:41.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:41.855+0000 7fd925411800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:41.878 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:41Z"} 2024-04-16T19:56:41.878 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:41Z"} 2024-04-16T19:56:41.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:41.879 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:41.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:41.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:56:46.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:46.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:46.881 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:46.937 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:46.937 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:46.989+0000 7f8280f31800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:46.991+0000 7f8280f31800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:47.013 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:46Z"} 2024-04-16T19:56:47.013 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:46Z"} 2024-04-16T19:56:47.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:47.014 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:47.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:47.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:56:52.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:52.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:52.016 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:52.071 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:52.071 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:52.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:52.125+0000 7f5ee9d2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:52.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:52.127+0000 7f5ee9d2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:52.148 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:52Z"} 2024-04-16T19:56:52.149 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:52Z"} 2024-04-16T19:56:52.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:52.149 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:52.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:52.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:56:57.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:57.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:56:57.151 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:56:57.209 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:56:57.209 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:56:57.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:57.263+0000 7f255cb2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:57.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:56:57.264+0000 7f255cb2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:56:57.286 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:57Z"} 2024-04-16T19:56:57.287 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:56:57Z"} 2024-04-16T19:56:57.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:56:57.287 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:56:57.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:56:57.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:02.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:02.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:02.289 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:02.347 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:02.347 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:02.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:02.402+0000 7f6787330800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:02.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:02.403+0000 7f6787330800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:02.425 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:02Z"} 2024-04-16T19:57:02.425 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:02Z"} 2024-04-16T19:57:02.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:02.426 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:02.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:02.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:07.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:07.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:07.428 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:07.484 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:07.484 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:07.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:07.541+0000 7fbee1810800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:07.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:07.543+0000 7fbee1810800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:07.564 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:07Z"} 2024-04-16T19:57:07.565 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:07Z"} 2024-04-16T19:57:07.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:07.565 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:07.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:07.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:12.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:12.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:12.567 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:12.623 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:12.623 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:12.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:12.678+0000 7f5abf43d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:12.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:12.679+0000 7f5abf43d800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:12.700 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:12Z"} 2024-04-16T19:57:12.701 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:12Z"} 2024-04-16T19:57:12.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:12.701 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:12.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:12.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:17.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:17.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:17.703 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:17.758 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:17.758 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:17.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:17.814+0000 7f89aae10800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:17.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:17.816+0000 7f89aae10800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:17.837 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:17Z"} 2024-04-16T19:57:17.838 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:17Z"} 2024-04-16T19:57:17.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:17.838 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:17.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:17.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:22.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:22.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:22.840 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:22.896 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:22.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:22.950+0000 7feba8022800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:22.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:22.952+0000 7feba8022800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:22.973 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:22Z"} 2024-04-16T19:57:22.973 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:22Z"} 2024-04-16T19:57:22.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:22.973 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:22.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:22.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:27.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:27.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:27.975 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:28.030 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:28.030 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:28.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:28.086+0000 7f459dc36800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:28.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:28.087+0000 7f459dc36800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:28.109 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:28Z"} 2024-04-16T19:57:28.109 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:28Z"} 2024-04-16T19:57:28.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:28.109 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:28.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:28.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:33.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:33.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:33.111 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:33.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:33.223+0000 7fe382331800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:33.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:33.224+0000 7fe382331800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:33.245 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:33Z"} 2024-04-16T19:57:33.246 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:33Z"} 2024-04-16T19:57:33.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:33.246 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:33.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:33.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:38.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:38.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:38.248 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:38.304 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:38.304 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:38.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:38.360+0000 7f2175933800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:38.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:38.361+0000 7f2175933800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:38.381 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:38Z"} 2024-04-16T19:57:38.382 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:38Z"} 2024-04-16T19:57:38.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:38.382 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:38.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:38.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:43.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:43.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:43.384 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:43.440 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:43.441 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:43.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:43.503+0000 7fc2b7d33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:43.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:43.505+0000 7fc2b7d33800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:43.526 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:43Z"} 2024-04-16T19:57:43.526 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:43Z"} 2024-04-16T19:57:43.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:43.527 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:43.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:43.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:48.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:48.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:48.529 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:48.585 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:48.585 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:48.641+0000 7fec2ff2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:48.643+0000 7fec2ff2a800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:48.665 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:48Z"} 2024-04-16T19:57:48.665 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:48Z"} 2024-04-16T19:57:48.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:48.665 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:48.665 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:48.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:53.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:53.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:53.668 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:53.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:53.781+0000 7fbb65a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:53.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:53.782+0000 7fbb65a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:53.803 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:53Z"} 2024-04-16T19:57:53.804 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:53Z"} 2024-04-16T19:57:53.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:53.804 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:53.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:53.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:57:58.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:58.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:57:58.806 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:57:58.861 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:57:58.862 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:57:58.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:58.918+0000 7fb1592e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:58.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:57:58.920+0000 7fb1592e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:57:58.940 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:58Z"} 2024-04-16T19:57:58.940 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297349.599966_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:55:49.599966Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:57:58Z"} 2024-04-16T19:57:58.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297349.599966_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:55:49.599966Z', 'realm_epoch': 2}}] 2024-04-16T19:57:58.941 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:57:58.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297349.599966_705.1'} 2024-04-16T19:57:58.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713297391.331007_838.1 target=1_1713297349.599966_705.1 2024-04-16T19:58:03.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:58:03.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T19:58:03.943 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T19:58:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:58:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:58:04.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:04.054+0000 7f5764412800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:58:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:04.056+0000 7f5764412800 20 RGW-SYNC:meta: read sync status 2024-04-16T19:58:04.078 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297391.331007_838.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:56:31.331007Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:58:04Z"} 2024-04-16T19:58:04.078 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297391.331007_838.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:56:31.331007Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:58:04Z"} 2024-04-16T19:58:04.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297391.331007_838.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:56:31.331007Z', 'realm_epoch': 2}}] 2024-04-16T19:58:04.078 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:58:04.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297391.331007_838.1'} 2024-04-16T19:58:04.079 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T19:58:04.079 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:04.079 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:04.080 DEBUG:boto:Method: HEAD 2024-04-16T19:58:04.080 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:04.080 DEBUG:boto:Data: 2024-04-16T19:58:04.080 DEBUG:boto:Headers: {} 2024-04-16T19:58:04.080 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.080 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.080 DEBUG:boto:Params: {} 2024-04-16T19:58:04.081 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:58:04.081 DEBUG:boto:Token: None 2024-04-16T19:58:04.081 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/ 2024-04-16T19:58:04.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM= 2024-04-16T19:58:04.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM=', 'Content-Length': '0'} 2024-04-16T19:58:04.087 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048d3cdbe6f8345dd-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.087 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:04.087 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:04.087 DEBUG:boto:Method: PUT 2024-04-16T19:58:04.088 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T19:58:04.088 DEBUG:boto:Data: 2024-04-16T19:58:04.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:04.088 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.088 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.088 DEBUG:boto:Params: {} 2024-04-16T19:58:04.088 DEBUG:boto:Token: None 2024-04-16T19:58:04.088 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/obj11 2024-04-16T19:58:04.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PU6bUI5XpcYXyBKm7VXE1QWskVg= 2024-04-16T19:58:04.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PU6bUI5XpcYXyBKm7VXE1QWskVg='} 2024-04-16T19:58:04.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ac873a3d11eb687-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.098 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:04.099 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:04.099 DEBUG:boto:Method: HEAD 2024-04-16T19:58:04.099 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:04.099 DEBUG:boto:Data: 2024-04-16T19:58:04.099 DEBUG:boto:Headers: {} 2024-04-16T19:58:04.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.099 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.099 DEBUG:boto:Params: {} 2024-04-16T19:58:04.099 DEBUG:boto:Token: None 2024-04-16T19:58:04.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/ 2024-04-16T19:58:04.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM= 2024-04-16T19:58:04.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM=', 'Content-Length': '0'} 2024-04-16T19:58:04.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072b1b93699282b24-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.103 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:04.103 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:04.103 DEBUG:boto:Method: PUT 2024-04-16T19:58:04.103 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T19:58:04.103 DEBUG:boto:Data: 2024-04-16T19:58:04.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:04.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.103 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.103 DEBUG:boto:Params: {} 2024-04-16T19:58:04.104 DEBUG:boto:Token: None 2024-04-16T19:58:04.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/obj22 2024-04-16T19:58:04.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/RYYoL6UtJ3gM6iqHeMOsuCBNww= 2024-04-16T19:58:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/RYYoL6UtJ3gM6iqHeMOsuCBNww='} 2024-04-16T19:58:04.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004089c835a94888c2-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.147 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:04.147 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:04.147 DEBUG:boto:Method: HEAD 2024-04-16T19:58:04.147 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:04.147 DEBUG:boto:Data: 2024-04-16T19:58:04.147 DEBUG:boto:Headers: {} 2024-04-16T19:58:04.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.147 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.148 DEBUG:boto:Params: {} 2024-04-16T19:58:04.148 DEBUG:boto:Token: None 2024-04-16T19:58:04.148 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/ 2024-04-16T19:58:04.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM= 2024-04-16T19:58:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM=', 'Content-Length': '0'} 2024-04-16T19:58:04.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec5456abf8bed644-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.150 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:04.150 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:04.151 DEBUG:boto:Method: PUT 2024-04-16T19:58:04.151 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T19:58:04.151 DEBUG:boto:Data: 2024-04-16T19:58:04.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:04.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.151 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.151 DEBUG:boto:Params: {} 2024-04-16T19:58:04.151 DEBUG:boto:Token: None 2024-04-16T19:58:04.151 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/obj33 2024-04-16T19:58:04.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M62aLzO9/i5pTb+UpayG2QXP3PY= 2024-04-16T19:58:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M62aLzO9/i5pTb+UpayG2QXP3PY='} 2024-04-16T19:58:04.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076585e99edf5e826-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.158 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:04.158 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:04.158 DEBUG:boto:Method: HEAD 2024-04-16T19:58:04.158 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:04.158 DEBUG:boto:Data: 2024-04-16T19:58:04.158 DEBUG:boto:Headers: {} 2024-04-16T19:58:04.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.159 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.159 DEBUG:boto:Params: {} 2024-04-16T19:58:04.159 DEBUG:boto:Token: None 2024-04-16T19:58:04.159 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/ 2024-04-16T19:58:04.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM= 2024-04-16T19:58:04.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM=', 'Content-Length': '0'} 2024-04-16T19:58:04.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e7c5416aef8d70a-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.162 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:04.162 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:04.162 DEBUG:boto:Method: PUT 2024-04-16T19:58:04.162 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T19:58:04.162 DEBUG:boto:Data: 2024-04-16T19:58:04.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:04.162 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.163 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.163 DEBUG:boto:Params: {} 2024-04-16T19:58:04.163 DEBUG:boto:Token: None 2024-04-16T19:58:04.163 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/obj44 2024-04-16T19:58:04.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F1BvqKOTm5RqJ3Z5h91X+VsJ1lI= 2024-04-16T19:58:04.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F1BvqKOTm5RqJ3Z5h91X+VsJ1lI='} 2024-04-16T19:58:04.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df0967bbc8d0a59c-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.170 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:04.170 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:04.170 DEBUG:boto:Method: HEAD 2024-04-16T19:58:04.170 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:04.170 DEBUG:boto:Data: 2024-04-16T19:58:04.170 DEBUG:boto:Headers: {} 2024-04-16T19:58:04.170 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.170 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.171 DEBUG:boto:Params: {} 2024-04-16T19:58:04.171 DEBUG:boto:Token: None 2024-04-16T19:58:04.171 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/ 2024-04-16T19:58:04.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM= 2024-04-16T19:58:04.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEW3VfOdQh5rSFObznvnhE9WcWM=', 'Content-Length': '0'} 2024-04-16T19:58:04.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b0848466d635208-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.174 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:04.174 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:04.174 DEBUG:boto:Method: PUT 2024-04-16T19:58:04.174 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T19:58:04.174 DEBUG:boto:Data: 2024-04-16T19:58:04.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:04.174 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:04.174 DEBUG:boto:Port: 8000 2024-04-16T19:58:04.174 DEBUG:boto:Params: {} 2024-04-16T19:58:04.175 DEBUG:boto:Token: None 2024-04-16T19:58:04.175 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:04 GMT /yrjefh-52/obj55 2024-04-16T19:58:04.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SlDJt4i6aszb+NkMh3vOTPJ60XQ= 2024-04-16T19:58:04.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SlDJt4i6aszb+NkMh3vOTPJ60XQ='} 2024-04-16T19:58:04.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b12f2e0b5c35f4c2-00661ed84c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:04.219 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T19:58:04.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:04.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:04.220 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:58:04.275 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:58:04.276 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:58:04.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:04.332+0000 7fd240335800 1 waiting to reach incremental sync.. 2024-04-16T19:58:09.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:09.335+0000 7fd240335800 1 bucket sync caught up with source: 2024-04-16T19:58:09.334 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.49.6, 00000000002.104.6, 00000000002.153.6] 2024-04-16T19:58:09.334 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000001.49.6, 00000000002.104.6, 00000000002.153.6] 2024-04-16T19:58:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:09.335+0000 7fd240335800 0 bucket checkpoint complete 2024-04-16T19:58:09.361 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T19:58:09.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:09.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:09.361 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T19:58:09.424 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:58:09.424 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:58:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:58:09.489+0000 7f773fb22800 1 waiting to reach incremental sync.. 2024-04-16T19:58:14.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:58:14.491+0000 7f773fb22800 1 bucket sync caught up with source: 2024-04-16T19:58:14.490 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.157.6, 00000000002.189.6, 00000000002.262.6] 2024-04-16T19:58:14.490 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.157.6, 00000000002.189.6, 00000000002.262.6] 2024-04-16T19:58:14.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T19:58:14.491+0000 7f773fb22800 0 bucket checkpoint complete 2024-04-16T19:58:14.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T19:58:14.518 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.518 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.518 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.519 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:14.519 DEBUG:boto:Data: 2024-04-16T19:58:14.519 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.519 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.519 DEBUG:boto:Params: {} 2024-04-16T19:58:14.520 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T19:58:14.520 DEBUG:boto:Token: None 2024-04-16T19:58:14.520 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ku3darR4nAmeX/7twQTWW3iD8NU= 2024-04-16T19:58:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ku3darR4nAmeX/7twQTWW3iD8NU=', 'Content-Length': '0'} 2024-04-16T19:58:14.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d44d2ea8dd44d23-00661ed856-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.526 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.526 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.526 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.527 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:14.527 DEBUG:boto:Data: 2024-04-16T19:58:14.527 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.527 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.527 DEBUG:boto:Params: {} 2024-04-16T19:58:14.527 DEBUG:boto:Token: None 2024-04-16T19:58:14.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ku3darR4nAmeX/7twQTWW3iD8NU= 2024-04-16T19:58:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ku3darR4nAmeX/7twQTWW3iD8NU=', 'Content-Length': '0'} 2024-04-16T19:58:14.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003197fc429c05a8e3-00661ed856-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T19:58:14.531 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.531 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.531 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.531 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.531 DEBUG:boto:Method: GET 2024-04-16T19:58:14.531 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.532 DEBUG:boto:Data: 2024-04-16T19:58:14.532 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.532 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.532 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.532 DEBUG:boto:Params: {} 2024-04-16T19:58:14.532 DEBUG:boto:Token: None 2024-04-16T19:58:14.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/?versions 2024-04-16T19:58:14.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo= 2024-04-16T19:58:14.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo=', 'Content-Length': '0'} 2024-04-16T19:58:14.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e0b429052565ea-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.535 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T19:58:14.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T19:58:14.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T19:58:14.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T19:58:14.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T19:58:14.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T19:58:14.537 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.537 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.537 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.538 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.538 DEBUG:boto:Method: GET 2024-04-16T19:58:14.538 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.538 DEBUG:boto:Data: 2024-04-16T19:58:14.538 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.538 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.538 DEBUG:boto:Params: {} 2024-04-16T19:58:14.539 DEBUG:boto:Token: None 2024-04-16T19:58:14.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/?versions 2024-04-16T19:58:14.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo= 2024-04-16T19:58:14.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo=', 'Content-Length': '0'} 2024-04-16T19:58:14.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3db0c325492c7e5-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.542 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T19:58:14.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T19:58:14.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T19:58:14.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T19:58:14.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T19:58:14.543 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.543 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.544 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.544 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.544 DEBUG:boto:Method: GET 2024-04-16T19:58:14.544 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.544 DEBUG:boto:Data: 2024-04-16T19:58:14.544 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.544 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.544 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.544 DEBUG:boto:Params: {} 2024-04-16T19:58:14.545 DEBUG:boto:Token: None 2024-04-16T19:58:14.545 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/?versions 2024-04-16T19:58:14.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo= 2024-04-16T19:58:14.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo=', 'Content-Length': '0'} 2024-04-16T19:58:14.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000490439a944bb7757-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.548 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.548 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.548 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.549 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.549 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.549 DEBUG:boto:Method: GET 2024-04-16T19:58:14.549 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T19:58:14.549 DEBUG:boto:Data: 2024-04-16T19:58:14.549 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.549 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.549 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.549 DEBUG:boto:Params: {} 2024-04-16T19:58:14.550 DEBUG:boto:Token: None 2024-04-16T19:58:14.550 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/?versions 2024-04-16T19:58:14.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo= 2024-04-16T19:58:14.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IEXpFUHzZXX5G+gMyRZWqe4ioo=', 'Content-Length': '0'} 2024-04-16T19:58:14.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034cfbe84128b579f-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.553 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T19:58:14.554 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.554 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.554 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.554 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.554 DEBUG:boto:Method: GET 2024-04-16T19:58:14.555 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.555 DEBUG:boto:Data: 2024-04-16T19:58:14.555 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.555 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.555 DEBUG:boto:Params: {} 2024-04-16T19:58:14.555 DEBUG:boto:Token: None 2024-04-16T19:58:14.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o= 2024-04-16T19:58:14.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o=', 'Content-Length': '0'} 2024-04-16T19:58:14.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a8f5d62ca3db5d-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.559 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.559 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.559 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.559 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.559 DEBUG:boto:Method: GET 2024-04-16T19:58:14.559 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.559 DEBUG:boto:Data: 2024-04-16T19:58:14.560 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.560 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.560 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.560 DEBUG:boto:Params: {} 2024-04-16T19:58:14.560 DEBUG:boto:Token: None 2024-04-16T19:58:14.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o= 2024-04-16T19:58:14.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o=', 'Content-Length': '0'} 2024-04-16T19:58:14.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b6cda0de144d9a-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.563 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.563 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.563 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.563 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.563 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.563 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.563 DEBUG:boto:Data: 2024-04-16T19:58:14.564 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.564 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.564 DEBUG:boto:Params: {} 2024-04-16T19:58:14.564 DEBUG:boto:Token: None 2024-04-16T19:58:14.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OAYjGzizmYofe2XJbxuqUAkanC4= 2024-04-16T19:58:14.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OAYjGzizmYofe2XJbxuqUAkanC4=', 'Content-Length': '0'} 2024-04-16T19:58:14.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f4aaca0c222857-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.567 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.567 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.567 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.567 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.567 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.567 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.567 DEBUG:boto:Data: 2024-04-16T19:58:14.567 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.568 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.568 DEBUG:boto:Params: {} 2024-04-16T19:58:14.568 DEBUG:boto:Token: None 2024-04-16T19:58:14.568 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OAYjGzizmYofe2XJbxuqUAkanC4= 2024-04-16T19:58:14.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OAYjGzizmYofe2XJbxuqUAkanC4=', 'Content-Length': '0'} 2024-04-16T19:58:14.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934cd85266eb21fb-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T19:58:14.571 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.571 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.571 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.571 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.571 DEBUG:boto:Method: GET 2024-04-16T19:58:14.571 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.571 DEBUG:boto:Data: 2024-04-16T19:58:14.571 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.572 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.572 DEBUG:boto:Params: {} 2024-04-16T19:58:14.572 DEBUG:boto:Token: None 2024-04-16T19:58:14.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o= 2024-04-16T19:58:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o=', 'Content-Length': '0'} 2024-04-16T19:58:14.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004665ffb00b5447ae-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.575 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.575 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.575 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.575 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.575 DEBUG:boto:Method: GET 2024-04-16T19:58:14.575 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.575 DEBUG:boto:Data: 2024-04-16T19:58:14.575 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.575 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.576 DEBUG:boto:Params: {} 2024-04-16T19:58:14.576 DEBUG:boto:Token: None 2024-04-16T19:58:14.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T19:58:14.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o= 2024-04-16T19:58:14.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+/Cj7wRfTFRh+XTpMTyqGwfbP+o=', 'Content-Length': '0'} 2024-04-16T19:58:14.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006628dca670baaca9-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.579 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.579 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.579 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.579 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.579 DEBUG:boto:Method: GET 2024-04-16T19:58:14.579 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.579 DEBUG:boto:Data: 2024-04-16T19:58:14.580 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.580 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.580 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.580 DEBUG:boto:Params: {} 2024-04-16T19:58:14.580 DEBUG:boto:Token: None 2024-04-16T19:58:14.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003fd6a14d4896d2a-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.626 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.627 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.628 DEBUG:boto:Method: GET 2024-04-16T19:58:14.628 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.628 DEBUG:boto:Data: 2024-04-16T19:58:14.628 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.628 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.628 DEBUG:boto:Params: {} 2024-04-16T19:58:14.628 DEBUG:boto:Token: None 2024-04-16T19:58:14.629 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf90a0d7a25342a-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.632 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.632 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.632 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.632 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.633 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T19:58:14.633 DEBUG:boto:Data: 2024-04-16T19:58:14.633 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.633 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.633 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.633 DEBUG:boto:Params: {} 2024-04-16T19:58:14.633 DEBUG:boto:Token: None 2024-04-16T19:58:14.634 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11 2024-04-16T19:58:14.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dr5Amr8vkU54KkOGNWcBq4vzlmM= 2024-04-16T19:58:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dr5Amr8vkU54KkOGNWcBq4vzlmM=', 'Content-Length': '0'} 2024-04-16T19:58:14.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228aead912559474-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.636 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.637 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.637 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.637 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T19:58:14.637 DEBUG:boto:Data: 2024-04-16T19:58:14.637 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.637 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.637 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.637 DEBUG:boto:Params: {} 2024-04-16T19:58:14.638 DEBUG:boto:Token: None 2024-04-16T19:58:14.638 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11 2024-04-16T19:58:14.638 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Dr5Amr8vkU54KkOGNWcBq4vzlmM= 2024-04-16T19:58:14.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dr5Amr8vkU54KkOGNWcBq4vzlmM=', 'Content-Length': '0'} 2024-04-16T19:58:14.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d002e3419b4326-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T19:58:14.641 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.641 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.641 DEBUG:boto:Method: GET 2024-04-16T19:58:14.641 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T19:58:14.641 DEBUG:boto:Data: 2024-04-16T19:58:14.641 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.641 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.641 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.642 DEBUG:boto:Params: {} 2024-04-16T19:58:14.642 DEBUG:boto:Token: None 2024-04-16T19:58:14.642 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11 2024-04-16T19:58:14.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cb5neeEiDudPLUokm47TDwQ27Hk= 2024-04-16T19:58:14.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cb5neeEiDudPLUokm47TDwQ27Hk=', 'Content-Length': '0'} 2024-04-16T19:58:14.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc6cf9c4db6e0f8-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.687 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T19:58:14.687 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T19:58:14.687 DEBUG:boto:Method: GET 2024-04-16T19:58:14.687 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T19:58:14.687 DEBUG:boto:Data: 2024-04-16T19:58:14.687 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.687 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.688 DEBUG:boto:Params: {} 2024-04-16T19:58:14.688 DEBUG:boto:Token: None 2024-04-16T19:58:14.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj11 2024-04-16T19:58:14.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cb5neeEiDudPLUokm47TDwQ27Hk= 2024-04-16T19:58:14.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cb5neeEiDudPLUokm47TDwQ27Hk=', 'Content-Length': '0'} 2024-04-16T19:58:14.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006756a887374b7e43-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.691 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.691 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.691 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.691 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.691 DEBUG:boto:Method: GET 2024-04-16T19:58:14.691 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.692 DEBUG:boto:Data: 2024-04-16T19:58:14.692 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.692 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.692 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.692 DEBUG:boto:Params: {} 2024-04-16T19:58:14.692 DEBUG:boto:Token: None 2024-04-16T19:58:14.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc56721b2d79ea9-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.695 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.696 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.696 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.696 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.696 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.697 DEBUG:boto:Method: GET 2024-04-16T19:58:14.697 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T19:58:14.697 DEBUG:boto:Data: 2024-04-16T19:58:14.697 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.697 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.697 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.697 DEBUG:boto:Params: {} 2024-04-16T19:58:14.697 DEBUG:boto:Token: None 2024-04-16T19:58:14.698 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee668e496494b9b-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.742 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T19:58:14.743 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.743 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.743 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.743 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.744 DEBUG:boto:Method: GET 2024-04-16T19:58:14.744 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.744 DEBUG:boto:Data: 2024-04-16T19:58:14.744 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.744 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.744 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.744 DEBUG:boto:Params: {} 2024-04-16T19:58:14.744 DEBUG:boto:Token: None 2024-04-16T19:58:14.745 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE= 2024-04-16T19:58:14.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE=', 'Content-Length': '0'} 2024-04-16T19:58:14.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021aec9e1add56587-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.790 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.791 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.791 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.791 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.791 DEBUG:boto:Method: GET 2024-04-16T19:58:14.791 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.791 DEBUG:boto:Data: 2024-04-16T19:58:14.791 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.792 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.792 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.792 DEBUG:boto:Params: {} 2024-04-16T19:58:14.792 DEBUG:boto:Token: None 2024-04-16T19:58:14.792 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE= 2024-04-16T19:58:14.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE=', 'Content-Length': '0'} 2024-04-16T19:58:14.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31f38a761072de7-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.795 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.795 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.795 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.795 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.796 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.796 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.796 DEBUG:boto:Data: 2024-04-16T19:58:14.796 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.796 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.796 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.796 DEBUG:boto:Params: {} 2024-04-16T19:58:14.797 DEBUG:boto:Token: None 2024-04-16T19:58:14.797 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.797 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIZtJopgB/2xhvyF/C92Kcskr3g= 2024-04-16T19:58:14.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIZtJopgB/2xhvyF/C92Kcskr3g=', 'Content-Length': '0'} 2024-04-16T19:58:14.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040739e214b57e13e-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.799 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.800 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.800 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.800 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.800 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.800 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.800 DEBUG:boto:Data: 2024-04-16T19:58:14.800 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.801 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.801 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.801 DEBUG:boto:Params: {} 2024-04-16T19:58:14.801 DEBUG:boto:Token: None 2024-04-16T19:58:14.801 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QIZtJopgB/2xhvyF/C92Kcskr3g= 2024-04-16T19:58:14.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QIZtJopgB/2xhvyF/C92Kcskr3g=', 'Content-Length': '0'} 2024-04-16T19:58:14.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f7bf8b42a31c86-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T19:58:14.804 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.804 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.804 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.805 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.805 DEBUG:boto:Method: GET 2024-04-16T19:58:14.805 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.805 DEBUG:boto:Data: 2024-04-16T19:58:14.805 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.805 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.805 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.805 DEBUG:boto:Params: {} 2024-04-16T19:58:14.806 DEBUG:boto:Token: None 2024-04-16T19:58:14.806 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.806 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE= 2024-04-16T19:58:14.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE=', 'Content-Length': '0'} 2024-04-16T19:58:14.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081bdc1ee1b19296b-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.851 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.851 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.851 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.851 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.851 DEBUG:boto:Method: GET 2024-04-16T19:58:14.851 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.851 DEBUG:boto:Data: 2024-04-16T19:58:14.851 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.852 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.852 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.852 DEBUG:boto:Params: {} 2024-04-16T19:58:14.852 DEBUG:boto:Token: None 2024-04-16T19:58:14.852 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T19:58:14.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE= 2024-04-16T19:58:14.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rgishh21BCCCtZF5kSK6/o9/2aE=', 'Content-Length': '0'} 2024-04-16T19:58:14.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e13cb6952d0297f-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.855 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.855 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.855 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.856 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.856 DEBUG:boto:Method: GET 2024-04-16T19:58:14.856 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.856 DEBUG:boto:Data: 2024-04-16T19:58:14.856 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.856 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.856 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.856 DEBUG:boto:Params: {} 2024-04-16T19:58:14.857 DEBUG:boto:Token: None 2024-04-16T19:58:14.857 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.857 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000476d7241271b99a6-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.860 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.861 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.861 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.861 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.861 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.861 DEBUG:boto:Method: GET 2024-04-16T19:58:14.861 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.861 DEBUG:boto:Data: 2024-04-16T19:58:14.862 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.862 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.862 DEBUG:boto:Params: {} 2024-04-16T19:58:14.862 DEBUG:boto:Token: None 2024-04-16T19:58:14.862 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099aa66811db84f19-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.865 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.866 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.866 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.866 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.866 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T19:58:14.866 DEBUG:boto:Data: 2024-04-16T19:58:14.867 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.867 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.867 DEBUG:boto:Params: {} 2024-04-16T19:58:14.867 DEBUG:boto:Token: None 2024-04-16T19:58:14.867 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22 2024-04-16T19:58:14.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V3PXNAseZ9rC6Xo76eBulZVPdX0= 2024-04-16T19:58:14.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V3PXNAseZ9rC6Xo76eBulZVPdX0=', 'Content-Length': '0'} 2024-04-16T19:58:14.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000779574368bad3e28-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.870 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.870 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.870 DEBUG:boto:Method: HEAD 2024-04-16T19:58:14.871 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T19:58:14.871 DEBUG:boto:Data: 2024-04-16T19:58:14.871 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.871 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.871 DEBUG:boto:Params: {} 2024-04-16T19:58:14.871 DEBUG:boto:Token: None 2024-04-16T19:58:14.872 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22 2024-04-16T19:58:14.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V3PXNAseZ9rC6Xo76eBulZVPdX0= 2024-04-16T19:58:14.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V3PXNAseZ9rC6Xo76eBulZVPdX0=', 'Content-Length': '0'} 2024-04-16T19:58:14.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063056d82669cdb63-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T19:58:14.874 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.875 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.875 DEBUG:boto:Method: GET 2024-04-16T19:58:14.875 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T19:58:14.875 DEBUG:boto:Data: 2024-04-16T19:58:14.875 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.875 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.875 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.876 DEBUG:boto:Params: {} 2024-04-16T19:58:14.876 DEBUG:boto:Token: None 2024-04-16T19:58:14.876 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22 2024-04-16T19:58:14.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T+PylIB1z/Re3OsrekH/U9wkymg= 2024-04-16T19:58:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T+PylIB1z/Re3OsrekH/U9wkymg=', 'Content-Length': '0'} 2024-04-16T19:58:14.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a60074e038b918-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.918 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T19:58:14.919 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T19:58:14.919 DEBUG:boto:Method: GET 2024-04-16T19:58:14.919 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T19:58:14.919 DEBUG:boto:Data: 2024-04-16T19:58:14.919 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.919 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.920 DEBUG:boto:Params: {} 2024-04-16T19:58:14.920 DEBUG:boto:Token: None 2024-04-16T19:58:14.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj22 2024-04-16T19:58:14.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T+PylIB1z/Re3OsrekH/U9wkymg= 2024-04-16T19:58:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T+PylIB1z/Re3OsrekH/U9wkymg=', 'Content-Length': '0'} 2024-04-16T19:58:14.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e760d07a6caeb261-00661ed856-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.923 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.923 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.923 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.923 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.923 DEBUG:boto:Method: GET 2024-04-16T19:58:14.923 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.923 DEBUG:boto:Data: 2024-04-16T19:58:14.923 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.924 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.924 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.924 DEBUG:boto:Params: {} 2024-04-16T19:58:14.924 DEBUG:boto:Token: None 2024-04-16T19:58:14.924 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f114dcdfdd12422c-00661ed856-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.970 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.971 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:14.971 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:14.971 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.971 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.971 DEBUG:boto:Method: GET 2024-04-16T19:58:14.972 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T19:58:14.972 DEBUG:boto:Data: 2024-04-16T19:58:14.972 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.972 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:14.972 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.972 DEBUG:boto:Params: {} 2024-04-16T19:58:14.972 DEBUG:boto:Token: None 2024-04-16T19:58:14.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/ 2024-04-16T19:58:14.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI= 2024-04-16T19:58:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7RqTeBHSGNL2pJ1PMa6DSfilJcI=', 'Content-Length': '0'} 2024-04-16T19:58:14.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000705bfb0a8c0eac07-00661ed856-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:14.976 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:14.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T19:58:14.976 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:14.976 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:14.977 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:14.977 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:14.977 DEBUG:boto:Method: GET 2024-04-16T19:58:14.977 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:14.977 DEBUG:boto:Data: 2024-04-16T19:58:14.977 DEBUG:boto:Headers: {} 2024-04-16T19:58:14.977 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:14.977 DEBUG:boto:Port: 8000 2024-04-16T19:58:14.977 DEBUG:boto:Params: {} 2024-04-16T19:58:14.978 DEBUG:boto:Token: None 2024-04-16T19:58:14.978 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:14 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:14.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WOxYaxZ0jkoVEerijQ4Z2JYxTs= 2024-04-16T19:58:14.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WOxYaxZ0jkoVEerijQ4Z2JYxTs=', 'Content-Length': '0'} 2024-04-16T19:58:14.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b3c2aa4c964d61-00661ed856-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.022 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.023 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.023 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.023 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.023 DEBUG:boto:Method: GET 2024-04-16T19:58:15.023 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.023 DEBUG:boto:Data: 2024-04-16T19:58:15.023 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.024 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.024 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.024 DEBUG:boto:Params: {} 2024-04-16T19:58:15.024 DEBUG:boto:Token: None 2024-04-16T19:58:15.024 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4= 2024-04-16T19:58:15.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4=', 'Content-Length': '0'} 2024-04-16T19:58:15.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004867944feed71ffd-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.027 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.027 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.027 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.028 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.028 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.028 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.028 DEBUG:boto:Data: 2024-04-16T19:58:15.028 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.028 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.028 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.028 DEBUG:boto:Params: {} 2024-04-16T19:58:15.029 DEBUG:boto:Token: None 2024-04-16T19:58:15.029 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.029 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wsuyk+8qqR3M0sJUHAmVrrVUt5Y= 2024-04-16T19:58:15.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wsuyk+8qqR3M0sJUHAmVrrVUt5Y=', 'Content-Length': '0'} 2024-04-16T19:58:15.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092611c3c01873f08-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.032 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.032 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.032 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.032 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.032 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.033 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.033 DEBUG:boto:Data: 2024-04-16T19:58:15.033 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.033 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.033 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.033 DEBUG:boto:Params: {} 2024-04-16T19:58:15.033 DEBUG:boto:Token: None 2024-04-16T19:58:15.033 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.034 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wsuyk+8qqR3M0sJUHAmVrrVUt5Y= 2024-04-16T19:58:15.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wsuyk+8qqR3M0sJUHAmVrrVUt5Y=', 'Content-Length': '0'} 2024-04-16T19:58:15.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a8d78aa71dee51-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T19:58:15.037 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.037 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.037 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.037 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.037 DEBUG:boto:Method: GET 2024-04-16T19:58:15.037 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.037 DEBUG:boto:Data: 2024-04-16T19:58:15.037 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.038 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.038 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.038 DEBUG:boto:Params: {} 2024-04-16T19:58:15.038 DEBUG:boto:Token: None 2024-04-16T19:58:15.038 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4= 2024-04-16T19:58:15.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4=', 'Content-Length': '0'} 2024-04-16T19:58:15.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009776374446766e34-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.083 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.083 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.083 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.083 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.083 DEBUG:boto:Method: GET 2024-04-16T19:58:15.083 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.083 DEBUG:boto:Data: 2024-04-16T19:58:15.084 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.084 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.084 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.084 DEBUG:boto:Params: {} 2024-04-16T19:58:15.084 DEBUG:boto:Token: None 2024-04-16T19:58:15.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T19:58:15.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4= 2024-04-16T19:58:15.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MSniLusA4fdQi4N0L3kilTlb4l4=', 'Content-Length': '0'} 2024-04-16T19:58:15.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015287d13d5b5bb22-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.088 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.088 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.088 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.088 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.088 DEBUG:boto:Method: GET 2024-04-16T19:58:15.088 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.089 DEBUG:boto:Data: 2024-04-16T19:58:15.089 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.089 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.089 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.089 DEBUG:boto:Params: {} 2024-04-16T19:58:15.089 DEBUG:boto:Token: None 2024-04-16T19:58:15.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e8b2b702b759169-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.093 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.093 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.093 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.093 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.093 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.094 DEBUG:boto:Method: GET 2024-04-16T19:58:15.094 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.094 DEBUG:boto:Data: 2024-04-16T19:58:15.094 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.094 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.094 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.094 DEBUG:boto:Params: {} 2024-04-16T19:58:15.095 DEBUG:boto:Token: None 2024-04-16T19:58:15.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000819a15b64fa8b916-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.098 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.098 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.099 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.099 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.099 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T19:58:15.099 DEBUG:boto:Data: 2024-04-16T19:58:15.099 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.099 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.099 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.099 DEBUG:boto:Params: {} 2024-04-16T19:58:15.100 DEBUG:boto:Token: None 2024-04-16T19:58:15.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33 2024-04-16T19:58:15.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GxzaGI6fUjQn1J7A6h2Q0I455g4= 2024-04-16T19:58:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GxzaGI6fUjQn1J7A6h2Q0I455g4=', 'Content-Length': '0'} 2024-04-16T19:58:15.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3cabe7818152be2-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.103 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.103 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.103 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.103 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T19:58:15.103 DEBUG:boto:Data: 2024-04-16T19:58:15.103 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.104 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.104 DEBUG:boto:Params: {} 2024-04-16T19:58:15.104 DEBUG:boto:Token: None 2024-04-16T19:58:15.104 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33 2024-04-16T19:58:15.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GxzaGI6fUjQn1J7A6h2Q0I455g4= 2024-04-16T19:58:15.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GxzaGI6fUjQn1J7A6h2Q0I455g4=', 'Content-Length': '0'} 2024-04-16T19:58:15.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff7831fe5a6b048-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T19:58:15.107 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.107 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.107 DEBUG:boto:Method: GET 2024-04-16T19:58:15.107 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T19:58:15.107 DEBUG:boto:Data: 2024-04-16T19:58:15.108 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.108 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.108 DEBUG:boto:Params: {} 2024-04-16T19:58:15.108 DEBUG:boto:Token: None 2024-04-16T19:58:15.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33 2024-04-16T19:58:15.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TdewtdMeMEq1pCRekPNfuY9ZebU= 2024-04-16T19:58:15.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TdewtdMeMEq1pCRekPNfuY9ZebU=', 'Content-Length': '0'} 2024-04-16T19:58:15.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cca7042e9f71b8c-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.155 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T19:58:15.155 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T19:58:15.155 DEBUG:boto:Method: GET 2024-04-16T19:58:15.155 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T19:58:15.155 DEBUG:boto:Data: 2024-04-16T19:58:15.155 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.156 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.156 DEBUG:boto:Params: {} 2024-04-16T19:58:15.156 DEBUG:boto:Token: None 2024-04-16T19:58:15.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj33 2024-04-16T19:58:15.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TdewtdMeMEq1pCRekPNfuY9ZebU= 2024-04-16T19:58:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TdewtdMeMEq1pCRekPNfuY9ZebU=', 'Content-Length': '0'} 2024-04-16T19:58:15.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf2b81832603d86-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.160 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.160 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.160 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.160 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.160 DEBUG:boto:Method: GET 2024-04-16T19:58:15.160 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.160 DEBUG:boto:Data: 2024-04-16T19:58:15.161 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.161 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.161 DEBUG:boto:Params: {} 2024-04-16T19:58:15.161 DEBUG:boto:Token: None 2024-04-16T19:58:15.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e37a6a39685b10-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.164 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.165 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.165 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.165 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.165 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.165 DEBUG:boto:Method: GET 2024-04-16T19:58:15.165 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T19:58:15.165 DEBUG:boto:Data: 2024-04-16T19:58:15.166 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.166 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.166 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.166 DEBUG:boto:Params: {} 2024-04-16T19:58:15.166 DEBUG:boto:Token: None 2024-04-16T19:58:15.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d390d910489d7e-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.170 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T19:58:15.170 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.170 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.171 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.171 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.171 DEBUG:boto:Method: GET 2024-04-16T19:58:15.171 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.171 DEBUG:boto:Data: 2024-04-16T19:58:15.171 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.171 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.171 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.171 DEBUG:boto:Params: {} 2024-04-16T19:58:15.172 DEBUG:boto:Token: None 2024-04-16T19:58:15.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw= 2024-04-16T19:58:15.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw=', 'Content-Length': '0'} 2024-04-16T19:58:15.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8f942e5a5865ba-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.218 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.219 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.219 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.219 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.219 DEBUG:boto:Method: GET 2024-04-16T19:58:15.219 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.219 DEBUG:boto:Data: 2024-04-16T19:58:15.219 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.220 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.220 DEBUG:boto:Params: {} 2024-04-16T19:58:15.220 DEBUG:boto:Token: None 2024-04-16T19:58:15.220 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw= 2024-04-16T19:58:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw=', 'Content-Length': '0'} 2024-04-16T19:58:15.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850eaa19f74acd8a-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.223 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.223 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.223 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.224 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.224 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.224 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.224 DEBUG:boto:Data: 2024-04-16T19:58:15.224 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.224 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.224 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.224 DEBUG:boto:Params: {} 2024-04-16T19:58:15.225 DEBUG:boto:Token: None 2024-04-16T19:58:15.225 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.225 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UzUPKgomuXiO80lZZ8SVmW8U0MI= 2024-04-16T19:58:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UzUPKgomuXiO80lZZ8SVmW8U0MI=', 'Content-Length': '0'} 2024-04-16T19:58:15.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97e01bf39549d60-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.228 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.228 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.228 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.228 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.228 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.228 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.229 DEBUG:boto:Data: 2024-04-16T19:58:15.229 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.229 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.229 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.229 DEBUG:boto:Params: {} 2024-04-16T19:58:15.229 DEBUG:boto:Token: None 2024-04-16T19:58:15.229 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UzUPKgomuXiO80lZZ8SVmW8U0MI= 2024-04-16T19:58:15.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UzUPKgomuXiO80lZZ8SVmW8U0MI=', 'Content-Length': '0'} 2024-04-16T19:58:15.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006032e5f4da644117-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T19:58:15.233 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.233 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.233 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.233 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.233 DEBUG:boto:Method: GET 2024-04-16T19:58:15.233 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.233 DEBUG:boto:Data: 2024-04-16T19:58:15.233 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.234 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.234 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.234 DEBUG:boto:Params: {} 2024-04-16T19:58:15.234 DEBUG:boto:Token: None 2024-04-16T19:58:15.234 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.234 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw= 2024-04-16T19:58:15.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw=', 'Content-Length': '0'} 2024-04-16T19:58:15.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3429cc2677b9cc9-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.279 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.279 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.279 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.279 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.279 DEBUG:boto:Method: GET 2024-04-16T19:58:15.279 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.279 DEBUG:boto:Data: 2024-04-16T19:58:15.279 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.280 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.280 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.280 DEBUG:boto:Params: {} 2024-04-16T19:58:15.280 DEBUG:boto:Token: None 2024-04-16T19:58:15.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T19:58:15.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw= 2024-04-16T19:58:15.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:grZvY8Vsb3EL5uR/t8HWeQVdkiw=', 'Content-Length': '0'} 2024-04-16T19:58:15.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a378204bb94e375-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.283 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.283 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.284 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.284 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.284 DEBUG:boto:Method: GET 2024-04-16T19:58:15.284 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.284 DEBUG:boto:Data: 2024-04-16T19:58:15.284 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.284 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.284 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.285 DEBUG:boto:Params: {} 2024-04-16T19:58:15.285 DEBUG:boto:Token: None 2024-04-16T19:58:15.285 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.285 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e72a53777e5e6e-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.288 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.289 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.289 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.289 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.289 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.289 DEBUG:boto:Method: GET 2024-04-16T19:58:15.290 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.290 DEBUG:boto:Data: 2024-04-16T19:58:15.290 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.290 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.290 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.290 DEBUG:boto:Params: {} 2024-04-16T19:58:15.293 DEBUG:boto:Token: None 2024-04-16T19:58:15.293 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004884a59f57198df0-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.296 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.297 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.297 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.297 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.297 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T19:58:15.297 DEBUG:boto:Data: 2024-04-16T19:58:15.297 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.298 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.298 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.298 DEBUG:boto:Params: {} 2024-04-16T19:58:15.298 DEBUG:boto:Token: None 2024-04-16T19:58:15.298 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44 2024-04-16T19:58:15.298 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PpBCEB6/MnuJxyvWhdSIXHy2d04= 2024-04-16T19:58:15.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PpBCEB6/MnuJxyvWhdSIXHy2d04=', 'Content-Length': '0'} 2024-04-16T19:58:15.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca41caab21bd7661-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.301 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.301 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.301 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.302 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T19:58:15.302 DEBUG:boto:Data: 2024-04-16T19:58:15.302 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.302 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.302 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.302 DEBUG:boto:Params: {} 2024-04-16T19:58:15.302 DEBUG:boto:Token: None 2024-04-16T19:58:15.302 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44 2024-04-16T19:58:15.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PpBCEB6/MnuJxyvWhdSIXHy2d04= 2024-04-16T19:58:15.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PpBCEB6/MnuJxyvWhdSIXHy2d04=', 'Content-Length': '0'} 2024-04-16T19:58:15.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed0369734201205-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T19:58:15.306 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.306 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.306 DEBUG:boto:Method: GET 2024-04-16T19:58:15.306 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T19:58:15.306 DEBUG:boto:Data: 2024-04-16T19:58:15.306 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.306 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.306 DEBUG:boto:Params: {} 2024-04-16T19:58:15.307 DEBUG:boto:Token: None 2024-04-16T19:58:15.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44 2024-04-16T19:58:15.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ma1h1cGjbAawnEftvEy2WtH2iLo= 2024-04-16T19:58:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ma1h1cGjbAawnEftvEy2WtH2iLo=', 'Content-Length': '0'} 2024-04-16T19:58:15.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030cf4b0caf53cc79-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.350 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T19:58:15.351 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T19:58:15.351 DEBUG:boto:Method: GET 2024-04-16T19:58:15.351 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T19:58:15.351 DEBUG:boto:Data: 2024-04-16T19:58:15.351 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.351 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.351 DEBUG:boto:Params: {} 2024-04-16T19:58:15.352 DEBUG:boto:Token: None 2024-04-16T19:58:15.352 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj44 2024-04-16T19:58:15.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ma1h1cGjbAawnEftvEy2WtH2iLo= 2024-04-16T19:58:15.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ma1h1cGjbAawnEftvEy2WtH2iLo=', 'Content-Length': '0'} 2024-04-16T19:58:15.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815f45d22646f4f2-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.355 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.355 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.355 DEBUG:boto:Method: GET 2024-04-16T19:58:15.355 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.355 DEBUG:boto:Data: 2024-04-16T19:58:15.355 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.356 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.356 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.356 DEBUG:boto:Params: {} 2024-04-16T19:58:15.356 DEBUG:boto:Token: None 2024-04-16T19:58:15.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b996d909da1277-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.402 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.403 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.403 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.403 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.403 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.403 DEBUG:boto:Method: GET 2024-04-16T19:58:15.404 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T19:58:15.404 DEBUG:boto:Data: 2024-04-16T19:58:15.404 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.404 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.404 DEBUG:boto:Params: {} 2024-04-16T19:58:15.404 DEBUG:boto:Token: None 2024-04-16T19:58:15.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644790ef9bdc22d5-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.408 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T19:58:15.408 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.408 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.408 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.409 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.409 DEBUG:boto:Method: GET 2024-04-16T19:58:15.409 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.409 DEBUG:boto:Data: 2024-04-16T19:58:15.409 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.409 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.409 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.409 DEBUG:boto:Params: {} 2024-04-16T19:58:15.410 DEBUG:boto:Token: None 2024-04-16T19:58:15.410 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.410 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8= 2024-04-16T19:58:15.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8=', 'Content-Length': '0'} 2024-04-16T19:58:15.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09af239b6c0230c-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.454 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.455 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.455 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.455 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.455 DEBUG:boto:Method: GET 2024-04-16T19:58:15.455 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.455 DEBUG:boto:Data: 2024-04-16T19:58:15.455 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.455 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.456 DEBUG:boto:Params: {} 2024-04-16T19:58:15.456 DEBUG:boto:Token: None 2024-04-16T19:58:15.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8= 2024-04-16T19:58:15.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8=', 'Content-Length': '0'} 2024-04-16T19:58:15.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9cbb001cff566e1-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.459 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.459 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.459 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.459 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.459 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.459 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.459 DEBUG:boto:Data: 2024-04-16T19:58:15.460 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.460 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.460 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.460 DEBUG:boto:Params: {} 2024-04-16T19:58:15.460 DEBUG:boto:Token: None 2024-04-16T19:58:15.460 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhsb/T004hyU5YkgPht29vpG0TE= 2024-04-16T19:58:15.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhsb/T004hyU5YkgPht29vpG0TE=', 'Content-Length': '0'} 2024-04-16T19:58:15.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1841101e4e4ba72-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.463 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.463 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.464 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.464 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.464 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.464 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.464 DEBUG:boto:Data: 2024-04-16T19:58:15.464 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.464 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.464 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.464 DEBUG:boto:Params: {} 2024-04-16T19:58:15.464 DEBUG:boto:Token: None 2024-04-16T19:58:15.465 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhsb/T004hyU5YkgPht29vpG0TE= 2024-04-16T19:58:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhsb/T004hyU5YkgPht29vpG0TE=', 'Content-Length': '0'} 2024-04-16T19:58:15.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6435711053fbea-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T19:58:15.467 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.468 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.468 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.468 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.468 DEBUG:boto:Method: GET 2024-04-16T19:58:15.468 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.468 DEBUG:boto:Data: 2024-04-16T19:58:15.468 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.468 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.469 DEBUG:boto:Params: {} 2024-04-16T19:58:15.469 DEBUG:boto:Token: None 2024-04-16T19:58:15.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8= 2024-04-16T19:58:15.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8=', 'Content-Length': '0'} 2024-04-16T19:58:15.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17ec8348f1c761c-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.514 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.514 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.515 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.515 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.515 DEBUG:boto:Method: GET 2024-04-16T19:58:15.515 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.515 DEBUG:boto:Data: 2024-04-16T19:58:15.515 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.515 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.515 DEBUG:boto:Params: {} 2024-04-16T19:58:15.516 DEBUG:boto:Token: None 2024-04-16T19:58:15.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T19:58:15.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8= 2024-04-16T19:58:15.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ws4HzATgFUYIF0SK33JXYPNQ9c8=', 'Content-Length': '0'} 2024-04-16T19:58:15.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91dabb19173370f-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.518 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.519 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.519 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.519 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.519 DEBUG:boto:Method: GET 2024-04-16T19:58:15.519 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.519 DEBUG:boto:Data: 2024-04-16T19:58:15.519 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.519 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.520 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.520 DEBUG:boto:Params: {} 2024-04-16T19:58:15.520 DEBUG:boto:Token: None 2024-04-16T19:58:15.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa0942b9e412bef-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.523 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.524 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.524 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.524 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.524 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.524 DEBUG:boto:Method: GET 2024-04-16T19:58:15.524 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.524 DEBUG:boto:Data: 2024-04-16T19:58:15.524 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.525 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.525 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.525 DEBUG:boto:Params: {} 2024-04-16T19:58:15.525 DEBUG:boto:Token: None 2024-04-16T19:58:15.525 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.525 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000959cad006bd3a8a6-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.570 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.571 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.571 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.571 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.571 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T19:58:15.571 DEBUG:boto:Data: 2024-04-16T19:58:15.571 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.571 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.572 DEBUG:boto:Params: {} 2024-04-16T19:58:15.572 DEBUG:boto:Token: None 2024-04-16T19:58:15.572 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55 2024-04-16T19:58:15.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aKPNZlzabesm8q/yPxEcXYqZOj4= 2024-04-16T19:58:15.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aKPNZlzabesm8q/yPxEcXYqZOj4=', 'Content-Length': '0'} 2024-04-16T19:58:15.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc1b396aee6fd74-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.575 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.575 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.575 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.575 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T19:58:15.575 DEBUG:boto:Data: 2024-04-16T19:58:15.575 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.575 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.575 DEBUG:boto:Params: {} 2024-04-16T19:58:15.576 DEBUG:boto:Token: None 2024-04-16T19:58:15.576 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55 2024-04-16T19:58:15.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aKPNZlzabesm8q/yPxEcXYqZOj4= 2024-04-16T19:58:15.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aKPNZlzabesm8q/yPxEcXYqZOj4=', 'Content-Length': '0'} 2024-04-16T19:58:15.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b91d3599779573-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T19:58:15.579 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.579 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.579 DEBUG:boto:Method: GET 2024-04-16T19:58:15.579 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T19:58:15.579 DEBUG:boto:Data: 2024-04-16T19:58:15.579 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.580 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.580 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.580 DEBUG:boto:Params: {} 2024-04-16T19:58:15.580 DEBUG:boto:Token: None 2024-04-16T19:58:15.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55 2024-04-16T19:58:15.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:phmFH16pZ6AyqCOvVf98Aee1/o4= 2024-04-16T19:58:15.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:phmFH16pZ6AyqCOvVf98Aee1/o4=', 'Content-Length': '0'} 2024-04-16T19:58:15.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e22ce3e6599e9b7-00661ed857-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.626 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T19:58:15.627 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T19:58:15.627 DEBUG:boto:Method: GET 2024-04-16T19:58:15.627 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T19:58:15.627 DEBUG:boto:Data: 2024-04-16T19:58:15.627 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.627 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.627 DEBUG:boto:Params: {} 2024-04-16T19:58:15.628 DEBUG:boto:Token: None 2024-04-16T19:58:15.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj55 2024-04-16T19:58:15.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:phmFH16pZ6AyqCOvVf98Aee1/o4= 2024-04-16T19:58:15.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:phmFH16pZ6AyqCOvVf98Aee1/o4=', 'Content-Length': '0'} 2024-04-16T19:58:15.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee4f311f690ca21-00661ed857-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.631 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.631 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.631 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.631 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.631 DEBUG:boto:Method: GET 2024-04-16T19:58:15.631 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.632 DEBUG:boto:Data: 2024-04-16T19:58:15.632 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.632 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T19:58:15.632 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.632 DEBUG:boto:Params: {} 2024-04-16T19:58:15.632 DEBUG:boto:Token: None 2024-04-16T19:58:15.632 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ed49a8c34b2b2e-00661ed857-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.635 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.636 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.636 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.636 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.636 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.636 DEBUG:boto:Method: GET 2024-04-16T19:58:15.636 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T19:58:15.636 DEBUG:boto:Data: 2024-04-16T19:58:15.636 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.637 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.637 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.637 DEBUG:boto:Params: {} 2024-04-16T19:58:15.637 DEBUG:boto:Token: None 2024-04-16T19:58:15.637 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI= 2024-04-16T19:58:15.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VEHGan7EdDsIX3IBp/rixjvDsCI=', 'Content-Length': '0'} 2024-04-16T19:58:15.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000711d0a160e8fe8a0-00661ed857-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.640 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T19:58:15.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T19:58:15.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:58:15.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T19:58:15.641 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T19:58:15.700 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:58:15.700 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:58:15.776 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T19:58:15.776 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T19:58:15.781 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 5 2024-04-16T19:58:15.832 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.832 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.833 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.833 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:15.833 DEBUG:boto:Data: 2024-04-16T19:58:15.833 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.833 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.833 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.833 DEBUG:boto:Params: {} 2024-04-16T19:58:15.833 DEBUG:boto:Token: None 2024-04-16T19:58:15.834 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk= 2024-04-16T19:58:15.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk=', 'Content-Length': '0'} 2024-04-16T19:58:15.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399aa04346b59c65-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.837 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T19:58:15.837 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T19:58:15.837 DEBUG:boto:Method: PUT 2024-04-16T19:58:15.837 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T19:58:15.837 DEBUG:boto:Data: 2024-04-16T19:58:15.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:15.838 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.838 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.838 DEBUG:boto:Params: {} 2024-04-16T19:58:15.838 DEBUG:boto:Token: None 2024-04-16T19:58:15.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj17 2024-04-16T19:58:15.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DO5Hh5ind8Pj6R31dvvhymzvuc8= 2024-04-16T19:58:15.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DO5Hh5ind8Pj6R31dvvhymzvuc8='} 2024-04-16T19:58:15.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000025d7ca605cf3e0f-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.845 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.845 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.845 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.845 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:15.845 DEBUG:boto:Data: 2024-04-16T19:58:15.845 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.846 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.846 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.846 DEBUG:boto:Params: {} 2024-04-16T19:58:15.846 DEBUG:boto:Token: None 2024-04-16T19:58:15.846 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk= 2024-04-16T19:58:15.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk=', 'Content-Length': '0'} 2024-04-16T19:58:15.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082ba9492c3f83b7b-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.849 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T19:58:15.849 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T19:58:15.849 DEBUG:boto:Method: PUT 2024-04-16T19:58:15.849 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T19:58:15.850 DEBUG:boto:Data: 2024-04-16T19:58:15.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:15.850 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.850 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.850 DEBUG:boto:Params: {} 2024-04-16T19:58:15.850 DEBUG:boto:Token: None 2024-04-16T19:58:15.850 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj34 2024-04-16T19:58:15.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KulCp7fS4OCbRWchGPzBSWXIKZk= 2024-04-16T19:58:15.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KulCp7fS4OCbRWchGPzBSWXIKZk='} 2024-04-16T19:58:15.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000295fe91189fe014b-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.895 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.895 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.895 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.895 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:15.896 DEBUG:boto:Data: 2024-04-16T19:58:15.896 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.896 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.896 DEBUG:boto:Params: {} 2024-04-16T19:58:15.896 DEBUG:boto:Token: None 2024-04-16T19:58:15.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk= 2024-04-16T19:58:15.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk=', 'Content-Length': '0'} 2024-04-16T19:58:15.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ade3dee2deef562c-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.899 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T19:58:15.900 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T19:58:15.900 DEBUG:boto:Method: PUT 2024-04-16T19:58:15.900 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T19:58:15.900 DEBUG:boto:Data: 2024-04-16T19:58:15.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:15.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.900 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.901 DEBUG:boto:Params: {} 2024-04-16T19:58:15.901 DEBUG:boto:Token: None 2024-04-16T19:58:15.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj51 2024-04-16T19:58:15.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KXaAkTeFl4b5998/Vd57dAY8occ= 2024-04-16T19:58:15.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KXaAkTeFl4b5998/Vd57dAY8occ='} 2024-04-16T19:58:15.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5d36224ba561564-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.907 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.907 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.907 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.907 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:15.908 DEBUG:boto:Data: 2024-04-16T19:58:15.908 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.908 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.908 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.908 DEBUG:boto:Params: {} 2024-04-16T19:58:15.908 DEBUG:boto:Token: None 2024-04-16T19:58:15.908 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.909 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk= 2024-04-16T19:58:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk=', 'Content-Length': '0'} 2024-04-16T19:58:15.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9dd61c463099252-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.911 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T19:58:15.911 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T19:58:15.912 DEBUG:boto:Method: PUT 2024-04-16T19:58:15.912 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T19:58:15.912 DEBUG:boto:Data: 2024-04-16T19:58:15.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:15.912 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.912 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.912 DEBUG:boto:Params: {} 2024-04-16T19:58:15.912 DEBUG:boto:Token: None 2024-04-16T19:58:15.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj68 2024-04-16T19:58:15.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DagH8T340X6XF+sRO5LwMt9LDSI= 2024-04-16T19:58:15.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DagH8T340X6XF+sRO5LwMt9LDSI='} 2024-04-16T19:58:15.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f4182942ee48b7-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.959 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T19:58:15.959 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T19:58:15.959 DEBUG:boto:Method: HEAD 2024-04-16T19:58:15.959 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T19:58:15.959 DEBUG:boto:Data: 2024-04-16T19:58:15.959 DEBUG:boto:Headers: {} 2024-04-16T19:58:15.960 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.960 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.960 DEBUG:boto:Params: {} 2024-04-16T19:58:15.960 DEBUG:boto:Token: None 2024-04-16T19:58:15.960 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/ 2024-04-16T19:58:15.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk= 2024-04-16T19:58:15.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXKgcZT6ndAMq/89txS4qVEnEKk=', 'Content-Length': '0'} 2024-04-16T19:58:15.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b263ebb6b2a7f62-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.963 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T19:58:15.963 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T19:58:15.963 DEBUG:boto:Method: PUT 2024-04-16T19:58:15.964 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T19:58:15.964 DEBUG:boto:Data: 2024-04-16T19:58:15.964 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T19:58:15.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T19:58:15.964 DEBUG:boto:Port: 8000 2024-04-16T19:58:15.964 DEBUG:boto:Params: {} 2024-04-16T19:58:15.964 DEBUG:boto:Token: None 2024-04-16T19:58:15.965 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 19:58:15 GMT /yrjefh-52/obj85 2024-04-16T19:58:15.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgZkikYD/dQbOL/yznmbM3n390Y= 2024-04-16T19:58:15.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:58:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgZkikYD/dQbOL/yznmbM3n390Y='} 2024-04-16T19:58:15.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea482523a3dfa935-00661ed857-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T19:58:15.971 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T19:58:15.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:15.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T19:58:15.972 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T19:58:16.034 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T19:58:16.034 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T19:58:16.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:16.098+0000 7ff20d724800 1 waiting to reach latest gen 1, on 0.. 2024-04-16T19:58:21.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:21.100+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:21.098 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-16T19:58:21.098 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:26.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:26.101+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:26.099 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-16T19:58:26.100 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:31.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:31.103+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:31.101 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-16T19:58:31.101 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:36.104+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:36.102 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-16T19:58:36.102 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:41.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:41.105+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:41.104 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-16T19:58:41.104 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:46.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:46.106+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:46.105 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:58:46.105 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:51.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:51.108+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:51.107 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:58:51.107 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:58:56.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:58:56.110+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:58:56.108 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:58:56.108 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:01.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:01.111+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:01.109 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:59:01.109 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:06.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:06.113+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:06.110 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:59:06.110 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:11.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:11.114+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:11.112 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:59:11.112 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:16.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:16.115+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:16.113 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, , , , ] 2024-04-16T19:59:16.113 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:21.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:21.116+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:21.115 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:21.115 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:26.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:26.119+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:26.116 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:26.116 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:31.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:31.120+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:31.118 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:31.118 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:36.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:36.122+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:36.119 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:36.119 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:41.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:41.124+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:41.121 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , , , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:41.121 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:46.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:46.125+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:46.122 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:46.122 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:51.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:51.127+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:51.124 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:51.124 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:56.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T19:59:56.128+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T19:59:56.125 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T19:59:56.125 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:01.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:01.129+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:01.127 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:01.127 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:06.131+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:06.128 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:06.128 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:11.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:11.132+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:11.129 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:11.130 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:16.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:16.134+0000 7ff20d724800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:16.131 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , , , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:16.131 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:21.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:21.135+0000 7ff20d724800 1 bucket sync caught up with source: 2024-04-16T20:00:21.132 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:21.132 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000001.70.6, , , , , , 00000000001.111.6, , 00000000001.59.6, , , , 00000000001.54.6, 00000000001.42.6, , , ] 2024-04-16T20:00:21.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:21.135+0000 7ff20d724800 0 bucket checkpoint complete 2024-04-16T20:00:21.159 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:00:21.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:00:21.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:00:21.159 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:00:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:00:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:00:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:00:21.291+0000 7fc660417800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:21.288 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.159.6, 00000000001.162.6, , , 00000000001.201.6, , , 00000000001.157.6, 00000000003.199.6, 00000000002.262.6] 2024-04-16T20:00:21.288 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.159.6, 00000000001.162.6, , , 00000000001.201.6, , , 00000000001.157.6, 00000000003.199.6, 00000000003.272.6] 2024-04-16T20:00:26.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:00:26.292+0000 7fc660417800 1 bucket sync caught up with source: 2024-04-16T20:00:26.290 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.159.6, 00000000001.162.6, , , 00000000001.201.6, , , 00000000001.157.6, 00000000003.199.6, 00000000003.272.6] 2024-04-16T20:00:26.290 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.159.6, 00000000001.162.6, , , 00000000001.201.6, , , 00000000001.157.6, 00000000003.199.6, 00000000003.272.6] 2024-04-16T20:00:26.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:00:26.292+0000 7fc660417800 0 bucket checkpoint complete 2024-04-16T20:00:26.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:00:26.318 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.318 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.318 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.319 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:26.319 DEBUG:boto:Data: 2024-04-16T20:00:26.319 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.319 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.319 DEBUG:boto:Params: {} 2024-04-16T20:00:26.319 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:00:26.320 DEBUG:boto:Token: None 2024-04-16T20:00:26.320 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s2bkUWmuEc9ZUoR6H1rtto5IXck= 2024-04-16T20:00:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s2bkUWmuEc9ZUoR6H1rtto5IXck=', 'Content-Length': '0'} 2024-04-16T20:00:26.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f67d76d1f4159dac-00661ed8da-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.325 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.325 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.325 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.325 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:26.325 DEBUG:boto:Data: 2024-04-16T20:00:26.325 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.325 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.326 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.326 DEBUG:boto:Params: {} 2024-04-16T20:00:26.326 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:00:26.326 DEBUG:boto:Token: None 2024-04-16T20:00:26.326 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.326 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s2bkUWmuEc9ZUoR6H1rtto5IXck= 2024-04-16T20:00:26.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s2bkUWmuEc9ZUoR6H1rtto5IXck=', 'Content-Length': '0'} 2024-04-16T20:00:26.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be87d34314616ca3-00661ed8da-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:00:26.332 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.332 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.333 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.333 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.333 DEBUG:boto:Method: GET 2024-04-16T20:00:26.333 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.333 DEBUG:boto:Data: 2024-04-16T20:00:26.333 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.333 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.333 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.333 DEBUG:boto:Params: {} 2024-04-16T20:00:26.333 DEBUG:boto:Token: None 2024-04-16T20:00:26.333 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/?versions 2024-04-16T20:00:26.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E= 2024-04-16T20:00:26.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E=', 'Content-Length': '0'} 2024-04-16T20:00:26.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059fb1cc6825147c5-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.336 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:00:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:00:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:00:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:00:26.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:00:26.338 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.338 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.338 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.338 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.339 DEBUG:boto:Method: GET 2024-04-16T20:00:26.339 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.339 DEBUG:boto:Data: 2024-04-16T20:00:26.339 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.339 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.339 DEBUG:boto:Params: {} 2024-04-16T20:00:26.339 DEBUG:boto:Token: None 2024-04-16T20:00:26.339 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/?versions 2024-04-16T20:00:26.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E= 2024-04-16T20:00:26.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E=', 'Content-Length': '0'} 2024-04-16T20:00:26.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647c6d31b7d9941f-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.342 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:00:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:00:26.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:00:26.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:00:26.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:00:26.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:00:26.344 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.344 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.344 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.344 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.344 DEBUG:boto:Method: GET 2024-04-16T20:00:26.344 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.344 DEBUG:boto:Data: 2024-04-16T20:00:26.344 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.345 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.345 DEBUG:boto:Params: {} 2024-04-16T20:00:26.345 DEBUG:boto:Token: None 2024-04-16T20:00:26.345 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/?versions 2024-04-16T20:00:26.345 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E= 2024-04-16T20:00:26.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E=', 'Content-Length': '0'} 2024-04-16T20:00:26.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3659f857c286b2-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.348 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.348 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.349 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.349 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.349 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.349 DEBUG:boto:Method: GET 2024-04-16T20:00:26.349 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:00:26.349 DEBUG:boto:Data: 2024-04-16T20:00:26.349 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.349 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.349 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.349 DEBUG:boto:Params: {} 2024-04-16T20:00:26.349 DEBUG:boto:Token: None 2024-04-16T20:00:26.350 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/?versions 2024-04-16T20:00:26.350 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E= 2024-04-16T20:00:26.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7eg9xcDvYK0EmIbGlsERmZxA98E=', 'Content-Length': '0'} 2024-04-16T20:00:26.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4a898447eaade64-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.352 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:00:26.353 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.353 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.353 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.354 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.354 DEBUG:boto:Method: GET 2024-04-16T20:00:26.354 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.354 DEBUG:boto:Data: 2024-04-16T20:00:26.354 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.354 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.354 DEBUG:boto:Params: {} 2024-04-16T20:00:26.354 DEBUG:boto:Token: None 2024-04-16T20:00:26.354 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM= 2024-04-16T20:00:26.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM=', 'Content-Length': '0'} 2024-04-16T20:00:26.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a30ba9781cdd48-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.357 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.357 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.357 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.357 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.357 DEBUG:boto:Method: GET 2024-04-16T20:00:26.358 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.358 DEBUG:boto:Data: 2024-04-16T20:00:26.358 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.358 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.358 DEBUG:boto:Params: {} 2024-04-16T20:00:26.358 DEBUG:boto:Token: None 2024-04-16T20:00:26.358 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM= 2024-04-16T20:00:26.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM=', 'Content-Length': '0'} 2024-04-16T20:00:26.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f29f0ac38489b73-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.360 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.361 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.361 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.361 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.361 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.361 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.361 DEBUG:boto:Data: 2024-04-16T20:00:26.361 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.361 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.361 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.361 DEBUG:boto:Params: {} 2024-04-16T20:00:26.361 DEBUG:boto:Token: None 2024-04-16T20:00:26.362 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.362 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t+gTO+fvI8ioBBdKCgWsjbiwvu4= 2024-04-16T20:00:26.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t+gTO+fvI8ioBBdKCgWsjbiwvu4=', 'Content-Length': '0'} 2024-04-16T20:00:26.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e37bc6261ad19b-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.364 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.364 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.364 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.364 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.364 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.364 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.364 DEBUG:boto:Data: 2024-04-16T20:00:26.365 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.365 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.365 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.365 DEBUG:boto:Params: {} 2024-04-16T20:00:26.365 DEBUG:boto:Token: None 2024-04-16T20:00:26.365 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t+gTO+fvI8ioBBdKCgWsjbiwvu4= 2024-04-16T20:00:26.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t+gTO+fvI8ioBBdKCgWsjbiwvu4=', 'Content-Length': '0'} 2024-04-16T20:00:26.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b48fbcedc0af79-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:00:26.367 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.367 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.367 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.368 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.368 DEBUG:boto:Method: GET 2024-04-16T20:00:26.368 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.368 DEBUG:boto:Data: 2024-04-16T20:00:26.368 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.368 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.368 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.368 DEBUG:boto:Params: {} 2024-04-16T20:00:26.368 DEBUG:boto:Token: None 2024-04-16T20:00:26.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM= 2024-04-16T20:00:26.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM=', 'Content-Length': '0'} 2024-04-16T20:00:26.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851eb0da2f61011a-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.414 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.414 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.415 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.415 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.415 DEBUG:boto:Method: GET 2024-04-16T20:00:26.415 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.415 DEBUG:boto:Data: 2024-04-16T20:00:26.415 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.415 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.415 DEBUG:boto:Params: {} 2024-04-16T20:00:26.415 DEBUG:boto:Token: None 2024-04-16T20:00:26.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:00:26.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM= 2024-04-16T20:00:26.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0VuzsP9goK4uAVayiBCwLgppRM=', 'Content-Length': '0'} 2024-04-16T20:00:26.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ecd2e9f462b6970-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.418 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.418 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.418 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.418 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.418 DEBUG:boto:Method: GET 2024-04-16T20:00:26.418 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.419 DEBUG:boto:Data: 2024-04-16T20:00:26.419 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.419 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.419 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.419 DEBUG:boto:Params: {} 2024-04-16T20:00:26.419 DEBUG:boto:Token: None 2024-04-16T20:00:26.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d747774bcccddb0-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.422 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.423 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.423 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.423 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.423 DEBUG:boto:Method: GET 2024-04-16T20:00:26.423 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.423 DEBUG:boto:Data: 2024-04-16T20:00:26.423 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.423 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.423 DEBUG:boto:Params: {} 2024-04-16T20:00:26.424 DEBUG:boto:Token: None 2024-04-16T20:00:26.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bcf65c28b1f610-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.427 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.427 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.427 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.427 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.427 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:00:26.427 DEBUG:boto:Data: 2024-04-16T20:00:26.427 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.428 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.428 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.428 DEBUG:boto:Params: {} 2024-04-16T20:00:26.428 DEBUG:boto:Token: None 2024-04-16T20:00:26.428 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11 2024-04-16T20:00:26.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m2pPgBrIZqHwOqJbtOTVud+AyEA= 2024-04-16T20:00:26.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m2pPgBrIZqHwOqJbtOTVud+AyEA=', 'Content-Length': '0'} 2024-04-16T20:00:26.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f90ab5c07c0806-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.430 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.430 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.430 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.431 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:00:26.431 DEBUG:boto:Data: 2024-04-16T20:00:26.431 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.431 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.431 DEBUG:boto:Params: {} 2024-04-16T20:00:26.431 DEBUG:boto:Token: None 2024-04-16T20:00:26.431 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11 2024-04-16T20:00:26.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m2pPgBrIZqHwOqJbtOTVud+AyEA= 2024-04-16T20:00:26.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m2pPgBrIZqHwOqJbtOTVud+AyEA=', 'Content-Length': '0'} 2024-04-16T20:00:26.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc10e00bbf8a59b0-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:00:26.434 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.434 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.434 DEBUG:boto:Method: GET 2024-04-16T20:00:26.434 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:00:26.434 DEBUG:boto:Data: 2024-04-16T20:00:26.434 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.434 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.434 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.434 DEBUG:boto:Params: {} 2024-04-16T20:00:26.434 DEBUG:boto:Token: None 2024-04-16T20:00:26.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11 2024-04-16T20:00:26.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlOxpqLOGak7jJfC2WkWN2t1+s4= 2024-04-16T20:00:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlOxpqLOGak7jJfC2WkWN2t1+s4=', 'Content-Length': '0'} 2024-04-16T20:00:26.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a732dc675c949b-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.478 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:00:26.478 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:00:26.479 DEBUG:boto:Method: GET 2024-04-16T20:00:26.479 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:00:26.479 DEBUG:boto:Data: 2024-04-16T20:00:26.479 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.479 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.479 DEBUG:boto:Params: {} 2024-04-16T20:00:26.479 DEBUG:boto:Token: None 2024-04-16T20:00:26.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj11 2024-04-16T20:00:26.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlOxpqLOGak7jJfC2WkWN2t1+s4= 2024-04-16T20:00:26.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlOxpqLOGak7jJfC2WkWN2t1+s4=', 'Content-Length': '0'} 2024-04-16T20:00:26.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088fbaf096354cd4c-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.482 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.482 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.482 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.482 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.482 DEBUG:boto:Method: GET 2024-04-16T20:00:26.482 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.482 DEBUG:boto:Data: 2024-04-16T20:00:26.482 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.483 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.483 DEBUG:boto:Params: {} 2024-04-16T20:00:26.483 DEBUG:boto:Token: None 2024-04-16T20:00:26.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab2a6e0230a8206-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.526 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.527 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.527 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.527 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.527 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.527 DEBUG:boto:Method: GET 2024-04-16T20:00:26.527 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:00:26.527 DEBUG:boto:Data: 2024-04-16T20:00:26.527 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.527 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.527 DEBUG:boto:Params: {} 2024-04-16T20:00:26.528 DEBUG:boto:Token: None 2024-04-16T20:00:26.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c52b30e51884abed-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.530 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:00:26.531 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.531 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.531 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.531 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.531 DEBUG:boto:Method: GET 2024-04-16T20:00:26.531 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.532 DEBUG:boto:Data: 2024-04-16T20:00:26.532 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.532 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.532 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.532 DEBUG:boto:Params: {} 2024-04-16T20:00:26.532 DEBUG:boto:Token: None 2024-04-16T20:00:26.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA= 2024-04-16T20:00:26.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA=', 'Content-Length': '0'} 2024-04-16T20:00:26.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008650782180be337f-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.578 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.578 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.579 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.579 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.579 DEBUG:boto:Method: GET 2024-04-16T20:00:26.579 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.579 DEBUG:boto:Data: 2024-04-16T20:00:26.579 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.579 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.579 DEBUG:boto:Params: {} 2024-04-16T20:00:26.579 DEBUG:boto:Token: None 2024-04-16T20:00:26.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA= 2024-04-16T20:00:26.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA=', 'Content-Length': '0'} 2024-04-16T20:00:26.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f98e6d7008ff84-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.582 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.582 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.582 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.582 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.582 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.582 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.582 DEBUG:boto:Data: 2024-04-16T20:00:26.582 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.583 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.583 DEBUG:boto:Params: {} 2024-04-16T20:00:26.583 DEBUG:boto:Token: None 2024-04-16T20:00:26.583 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81Li0Q1miz7EdnefXndN6t/Fero= 2024-04-16T20:00:26.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81Li0Q1miz7EdnefXndN6t/Fero=', 'Content-Length': '0'} 2024-04-16T20:00:26.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1c3af672578a03-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.585 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.585 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.585 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.586 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.586 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.586 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.586 DEBUG:boto:Data: 2024-04-16T20:00:26.586 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.586 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.586 DEBUG:boto:Params: {} 2024-04-16T20:00:26.586 DEBUG:boto:Token: None 2024-04-16T20:00:26.586 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81Li0Q1miz7EdnefXndN6t/Fero= 2024-04-16T20:00:26.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81Li0Q1miz7EdnefXndN6t/Fero=', 'Content-Length': '0'} 2024-04-16T20:00:26.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086184455adfdd64d-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:00:26.589 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.589 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.589 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.589 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.589 DEBUG:boto:Method: GET 2024-04-16T20:00:26.589 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.589 DEBUG:boto:Data: 2024-04-16T20:00:26.590 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.590 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.590 DEBUG:boto:Params: {} 2024-04-16T20:00:26.590 DEBUG:boto:Token: None 2024-04-16T20:00:26.590 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.590 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA= 2024-04-16T20:00:26.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA=', 'Content-Length': '0'} 2024-04-16T20:00:26.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fe183eefdc98e2-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.634 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.634 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.634 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.635 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.635 DEBUG:boto:Method: GET 2024-04-16T20:00:26.635 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.635 DEBUG:boto:Data: 2024-04-16T20:00:26.635 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.635 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.635 DEBUG:boto:Params: {} 2024-04-16T20:00:26.635 DEBUG:boto:Token: None 2024-04-16T20:00:26.636 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:00:26.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA= 2024-04-16T20:00:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A8xmDr54hH88KUYNiyAe5V2NgaA=', 'Content-Length': '0'} 2024-04-16T20:00:26.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a83b5738feceabb-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.638 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.638 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.638 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.638 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.638 DEBUG:boto:Method: GET 2024-04-16T20:00:26.639 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.639 DEBUG:boto:Data: 2024-04-16T20:00:26.639 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.639 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.639 DEBUG:boto:Params: {} 2024-04-16T20:00:26.639 DEBUG:boto:Token: None 2024-04-16T20:00:26.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505ad2b33413d9c2-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.683 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.683 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.683 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.683 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.683 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.684 DEBUG:boto:Method: GET 2024-04-16T20:00:26.684 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.684 DEBUG:boto:Data: 2024-04-16T20:00:26.684 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.684 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.684 DEBUG:boto:Params: {} 2024-04-16T20:00:26.684 DEBUG:boto:Token: None 2024-04-16T20:00:26.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053068ac458e8060f-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.688 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.688 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.688 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.688 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.688 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:00:26.688 DEBUG:boto:Data: 2024-04-16T20:00:26.689 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.689 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.689 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.689 DEBUG:boto:Params: {} 2024-04-16T20:00:26.689 DEBUG:boto:Token: None 2024-04-16T20:00:26.689 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17 2024-04-16T20:00:26.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TAf8WoQWsc1glDB0XiUxsFjo/T4= 2024-04-16T20:00:26.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TAf8WoQWsc1glDB0XiUxsFjo/T4=', 'Content-Length': '0'} 2024-04-16T20:00:26.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b555833b2cf8fb9-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.691 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.692 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.692 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.692 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:00:26.692 DEBUG:boto:Data: 2024-04-16T20:00:26.692 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.692 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.692 DEBUG:boto:Params: {} 2024-04-16T20:00:26.692 DEBUG:boto:Token: None 2024-04-16T20:00:26.692 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17 2024-04-16T20:00:26.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TAf8WoQWsc1glDB0XiUxsFjo/T4= 2024-04-16T20:00:26.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TAf8WoQWsc1glDB0XiUxsFjo/T4=', 'Content-Length': '0'} 2024-04-16T20:00:26.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3427905753c671-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:00:26.695 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.695 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.695 DEBUG:boto:Method: GET 2024-04-16T20:00:26.695 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:00:26.695 DEBUG:boto:Data: 2024-04-16T20:00:26.695 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.696 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.696 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.696 DEBUG:boto:Params: {} 2024-04-16T20:00:26.696 DEBUG:boto:Token: None 2024-04-16T20:00:26.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17 2024-04-16T20:00:26.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UcZmy9A/0QoViG9Cs0ClTYfyzC4= 2024-04-16T20:00:26.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UcZmy9A/0QoViG9Cs0ClTYfyzC4=', 'Content-Length': '0'} 2024-04-16T20:00:26.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3aca1d346c4514-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.738 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:00:26.739 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:00:26.739 DEBUG:boto:Method: GET 2024-04-16T20:00:26.739 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:00:26.739 DEBUG:boto:Data: 2024-04-16T20:00:26.739 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.739 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.739 DEBUG:boto:Params: {} 2024-04-16T20:00:26.740 DEBUG:boto:Token: None 2024-04-16T20:00:26.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj17 2024-04-16T20:00:26.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UcZmy9A/0QoViG9Cs0ClTYfyzC4= 2024-04-16T20:00:26.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UcZmy9A/0QoViG9Cs0ClTYfyzC4=', 'Content-Length': '0'} 2024-04-16T20:00:26.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064b0699f610c947b-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.742 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.742 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.743 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.743 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.743 DEBUG:boto:Method: GET 2024-04-16T20:00:26.743 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.743 DEBUG:boto:Data: 2024-04-16T20:00:26.743 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.743 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.743 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.743 DEBUG:boto:Params: {} 2024-04-16T20:00:26.743 DEBUG:boto:Token: None 2024-04-16T20:00:26.744 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004689102c06e67faa-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.791 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.792 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.792 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.792 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.792 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.792 DEBUG:boto:Method: GET 2024-04-16T20:00:26.792 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:00:26.792 DEBUG:boto:Data: 2024-04-16T20:00:26.792 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.792 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.792 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.793 DEBUG:boto:Params: {} 2024-04-16T20:00:26.793 DEBUG:boto:Token: None 2024-04-16T20:00:26.793 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f96206e53fe3d14-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.796 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:00:26.797 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.797 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.797 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.797 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.797 DEBUG:boto:Method: GET 2024-04-16T20:00:26.797 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.797 DEBUG:boto:Data: 2024-04-16T20:00:26.797 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.797 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.797 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.797 DEBUG:boto:Params: {} 2024-04-16T20:00:26.798 DEBUG:boto:Token: None 2024-04-16T20:00:26.798 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU= 2024-04-16T20:00:26.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU=', 'Content-Length': '0'} 2024-04-16T20:00:26.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb12019a2a0a07c5-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.842 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.843 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.843 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.843 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.843 DEBUG:boto:Method: GET 2024-04-16T20:00:26.843 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.843 DEBUG:boto:Data: 2024-04-16T20:00:26.843 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.843 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.843 DEBUG:boto:Params: {} 2024-04-16T20:00:26.844 DEBUG:boto:Token: None 2024-04-16T20:00:26.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU= 2024-04-16T20:00:26.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU=', 'Content-Length': '0'} 2024-04-16T20:00:26.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4d3c23201bf694-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.846 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.846 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.846 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.847 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.847 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.847 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.847 DEBUG:boto:Data: 2024-04-16T20:00:26.847 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.847 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.847 DEBUG:boto:Params: {} 2024-04-16T20:00:26.847 DEBUG:boto:Token: None 2024-04-16T20:00:26.847 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+chratZCv170t+TNtc5gGKx6bE= 2024-04-16T20:00:26.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+chratZCv170t+TNtc5gGKx6bE=', 'Content-Length': '0'} 2024-04-16T20:00:26.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002707a684b26c9b21-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.850 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.850 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.850 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.850 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.850 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.850 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.851 DEBUG:boto:Data: 2024-04-16T20:00:26.851 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.851 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.851 DEBUG:boto:Params: {} 2024-04-16T20:00:26.851 DEBUG:boto:Token: None 2024-04-16T20:00:26.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+chratZCv170t+TNtc5gGKx6bE= 2024-04-16T20:00:26.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+chratZCv170t+TNtc5gGKx6bE=', 'Content-Length': '0'} 2024-04-16T20:00:26.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095235e7635925646-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:00:26.854 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.854 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.854 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.854 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.854 DEBUG:boto:Method: GET 2024-04-16T20:00:26.854 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.854 DEBUG:boto:Data: 2024-04-16T20:00:26.854 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.854 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.855 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.855 DEBUG:boto:Params: {} 2024-04-16T20:00:26.855 DEBUG:boto:Token: None 2024-04-16T20:00:26.855 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU= 2024-04-16T20:00:26.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU=', 'Content-Length': '0'} 2024-04-16T20:00:26.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e84c1660194e53-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.898 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.899 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.899 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.899 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.899 DEBUG:boto:Method: GET 2024-04-16T20:00:26.899 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.899 DEBUG:boto:Data: 2024-04-16T20:00:26.899 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.899 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.899 DEBUG:boto:Params: {} 2024-04-16T20:00:26.900 DEBUG:boto:Token: None 2024-04-16T20:00:26.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:00:26.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU= 2024-04-16T20:00:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM2JSfvIQdp5t1sAzxOt92jeRAU=', 'Content-Length': '0'} 2024-04-16T20:00:26.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da28b6cde553b1cf-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.902 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.902 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.902 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.902 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.902 DEBUG:boto:Method: GET 2024-04-16T20:00:26.903 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.903 DEBUG:boto:Data: 2024-04-16T20:00:26.903 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.903 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.903 DEBUG:boto:Params: {} 2024-04-16T20:00:26.903 DEBUG:boto:Token: None 2024-04-16T20:00:26.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f45fb5511d76a936-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.946 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.947 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:26.947 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:26.947 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.947 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.947 DEBUG:boto:Method: GET 2024-04-16T20:00:26.947 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:00:26.947 DEBUG:boto:Data: 2024-04-16T20:00:26.947 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.948 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.948 DEBUG:boto:Params: {} 2024-04-16T20:00:26.948 DEBUG:boto:Token: None 2024-04-16T20:00:26.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/ 2024-04-16T20:00:26.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY= 2024-04-16T20:00:26.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiyCxElVq8XSIoEJQu9s6w/h8HY=', 'Content-Length': '0'} 2024-04-16T20:00:26.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c4479ac3631b03-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.951 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:26.951 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.951 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.951 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.952 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:00:26.952 DEBUG:boto:Data: 2024-04-16T20:00:26.952 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.952 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.952 DEBUG:boto:Params: {} 2024-04-16T20:00:26.952 DEBUG:boto:Token: None 2024-04-16T20:00:26.952 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22 2024-04-16T20:00:26.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BdhS6EuoYnNf/y6+2qHWefINwXc= 2024-04-16T20:00:26.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BdhS6EuoYnNf/y6+2qHWefINwXc=', 'Content-Length': '0'} 2024-04-16T20:00:26.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6d0790d86412c1-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.955 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.955 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.955 DEBUG:boto:Method: HEAD 2024-04-16T20:00:26.955 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:00:26.955 DEBUG:boto:Data: 2024-04-16T20:00:26.955 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:26.955 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.955 DEBUG:boto:Params: {} 2024-04-16T20:00:26.956 DEBUG:boto:Token: None 2024-04-16T20:00:26.956 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22 2024-04-16T20:00:26.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BdhS6EuoYnNf/y6+2qHWefINwXc= 2024-04-16T20:00:26.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BdhS6EuoYnNf/y6+2qHWefINwXc=', 'Content-Length': '0'} 2024-04-16T20:00:26.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf59f8099a1ee54-00661ed8da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:26.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:00:26.958 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:26.958 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:26.958 DEBUG:boto:Method: GET 2024-04-16T20:00:26.958 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:00:26.959 DEBUG:boto:Data: 2024-04-16T20:00:26.959 DEBUG:boto:Headers: {} 2024-04-16T20:00:26.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:26.959 DEBUG:boto:Port: 8000 2024-04-16T20:00:26.959 DEBUG:boto:Params: {} 2024-04-16T20:00:26.959 DEBUG:boto:Token: None 2024-04-16T20:00:26.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:26 GMT /yrjefh-52/obj22 2024-04-16T20:00:26.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgV5tGhj6KbwMyfQBWkHgp8ccnA= 2024-04-16T20:00:26.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgV5tGhj6KbwMyfQBWkHgp8ccnA=', 'Content-Length': '0'} 2024-04-16T20:00:26.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c202ddcb0c56552c-00661ed8da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.006 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:00:27.006 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:00:27.007 DEBUG:boto:Method: GET 2024-04-16T20:00:27.007 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:00:27.007 DEBUG:boto:Data: 2024-04-16T20:00:27.007 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.007 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.007 DEBUG:boto:Params: {} 2024-04-16T20:00:27.007 DEBUG:boto:Token: None 2024-04-16T20:00:27.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj22 2024-04-16T20:00:27.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xK+XESfY2ysx2MlWB/Wg9VTBPRQ= 2024-04-16T20:00:27.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xK+XESfY2ysx2MlWB/Wg9VTBPRQ=', 'Content-Length': '0'} 2024-04-16T20:00:27.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092734415d12b521f-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.010 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.010 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.010 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.010 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.010 DEBUG:boto:Method: GET 2024-04-16T20:00:27.010 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.011 DEBUG:boto:Data: 2024-04-16T20:00:27.011 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.011 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.011 DEBUG:boto:Params: {} 2024-04-16T20:00:27.011 DEBUG:boto:Token: None 2024-04-16T20:00:27.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ab4aeaccdaa033-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.054 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.055 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.055 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.055 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.055 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.055 DEBUG:boto:Method: GET 2024-04-16T20:00:27.055 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:00:27.055 DEBUG:boto:Data: 2024-04-16T20:00:27.055 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.055 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.056 DEBUG:boto:Params: {} 2024-04-16T20:00:27.056 DEBUG:boto:Token: None 2024-04-16T20:00:27.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb940186f593539e-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.059 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:00:27.059 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.059 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.060 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.060 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.060 DEBUG:boto:Method: GET 2024-04-16T20:00:27.060 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.060 DEBUG:boto:Data: 2024-04-16T20:00:27.060 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.060 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.060 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.060 DEBUG:boto:Params: {} 2024-04-16T20:00:27.060 DEBUG:boto:Token: None 2024-04-16T20:00:27.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4= 2024-04-16T20:00:27.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4=', 'Content-Length': '0'} 2024-04-16T20:00:27.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924e443157876c6c-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.106 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.107 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.107 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.107 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.107 DEBUG:boto:Method: GET 2024-04-16T20:00:27.107 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.107 DEBUG:boto:Data: 2024-04-16T20:00:27.107 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.107 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.107 DEBUG:boto:Params: {} 2024-04-16T20:00:27.107 DEBUG:boto:Token: None 2024-04-16T20:00:27.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4= 2024-04-16T20:00:27.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4=', 'Content-Length': '0'} 2024-04-16T20:00:27.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047c7bd4af59b68e7-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.110 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.110 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.111 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.111 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.111 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.111 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.111 DEBUG:boto:Data: 2024-04-16T20:00:27.111 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.111 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.111 DEBUG:boto:Params: {} 2024-04-16T20:00:27.111 DEBUG:boto:Token: None 2024-04-16T20:00:27.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0VokxfgYNG2ZwHrKq8EuK+DPchs= 2024-04-16T20:00:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0VokxfgYNG2ZwHrKq8EuK+DPchs=', 'Content-Length': '0'} 2024-04-16T20:00:27.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3bf42c9548b23b-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.114 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.114 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.114 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.114 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.114 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.114 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.114 DEBUG:boto:Data: 2024-04-16T20:00:27.114 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.114 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.115 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.115 DEBUG:boto:Params: {} 2024-04-16T20:00:27.115 DEBUG:boto:Token: None 2024-04-16T20:00:27.115 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0VokxfgYNG2ZwHrKq8EuK+DPchs= 2024-04-16T20:00:27.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0VokxfgYNG2ZwHrKq8EuK+DPchs=', 'Content-Length': '0'} 2024-04-16T20:00:27.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc254ef1ba321a41-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:00:27.117 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.118 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.118 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.118 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.118 DEBUG:boto:Method: GET 2024-04-16T20:00:27.118 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.118 DEBUG:boto:Data: 2024-04-16T20:00:27.118 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.118 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.118 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.118 DEBUG:boto:Params: {} 2024-04-16T20:00:27.118 DEBUG:boto:Token: None 2024-04-16T20:00:27.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4= 2024-04-16T20:00:27.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4=', 'Content-Length': '0'} 2024-04-16T20:00:27.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a132e47fa3beddc-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.162 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.162 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.163 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.163 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.163 DEBUG:boto:Method: GET 2024-04-16T20:00:27.163 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.163 DEBUG:boto:Data: 2024-04-16T20:00:27.163 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.163 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.163 DEBUG:boto:Params: {} 2024-04-16T20:00:27.163 DEBUG:boto:Token: None 2024-04-16T20:00:27.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:00:27.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4= 2024-04-16T20:00:27.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AFRvebbqdYlywgw8cV+nyVfeDm4=', 'Content-Length': '0'} 2024-04-16T20:00:27.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bebe83f5a09bd7f4-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.166 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.166 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.166 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.167 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.167 DEBUG:boto:Method: GET 2024-04-16T20:00:27.167 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.167 DEBUG:boto:Data: 2024-04-16T20:00:27.167 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.167 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.167 DEBUG:boto:Params: {} 2024-04-16T20:00:27.167 DEBUG:boto:Token: None 2024-04-16T20:00:27.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c30cf15c01593ee-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.210 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.211 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.211 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.211 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.211 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.211 DEBUG:boto:Method: GET 2024-04-16T20:00:27.211 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.211 DEBUG:boto:Data: 2024-04-16T20:00:27.211 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.211 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.212 DEBUG:boto:Params: {} 2024-04-16T20:00:27.212 DEBUG:boto:Token: None 2024-04-16T20:00:27.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000acd1a8893ddd1af-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.215 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.215 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.215 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.216 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.216 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:00:27.216 DEBUG:boto:Data: 2024-04-16T20:00:27.216 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.216 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.216 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.216 DEBUG:boto:Params: {} 2024-04-16T20:00:27.216 DEBUG:boto:Token: None 2024-04-16T20:00:27.216 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33 2024-04-16T20:00:27.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zvbGIoMrKgQN3Zt69sDKN2bFoOw= 2024-04-16T20:00:27.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zvbGIoMrKgQN3Zt69sDKN2bFoOw=', 'Content-Length': '0'} 2024-04-16T20:00:27.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9acfc9131645d7-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.219 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.219 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.219 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.219 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:00:27.219 DEBUG:boto:Data: 2024-04-16T20:00:27.219 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.219 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.220 DEBUG:boto:Params: {} 2024-04-16T20:00:27.220 DEBUG:boto:Token: None 2024-04-16T20:00:27.220 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33 2024-04-16T20:00:27.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zvbGIoMrKgQN3Zt69sDKN2bFoOw= 2024-04-16T20:00:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zvbGIoMrKgQN3Zt69sDKN2bFoOw=', 'Content-Length': '0'} 2024-04-16T20:00:27.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b884eb5b5c5133-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:00:27.222 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.222 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.222 DEBUG:boto:Method: GET 2024-04-16T20:00:27.223 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:00:27.223 DEBUG:boto:Data: 2024-04-16T20:00:27.223 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.223 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.223 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.223 DEBUG:boto:Params: {} 2024-04-16T20:00:27.223 DEBUG:boto:Token: None 2024-04-16T20:00:27.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33 2024-04-16T20:00:27.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6iijQKwEK8uzq91hMmkeKJVVt8w= 2024-04-16T20:00:27.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6iijQKwEK8uzq91hMmkeKJVVt8w=', 'Content-Length': '0'} 2024-04-16T20:00:27.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af68870d3165581-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.266 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:00:27.266 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:00:27.267 DEBUG:boto:Method: GET 2024-04-16T20:00:27.267 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:00:27.267 DEBUG:boto:Data: 2024-04-16T20:00:27.267 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.267 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.267 DEBUG:boto:Params: {} 2024-04-16T20:00:27.267 DEBUG:boto:Token: None 2024-04-16T20:00:27.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj33 2024-04-16T20:00:27.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6iijQKwEK8uzq91hMmkeKJVVt8w= 2024-04-16T20:00:27.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6iijQKwEK8uzq91hMmkeKJVVt8w=', 'Content-Length': '0'} 2024-04-16T20:00:27.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d909a1c2aa99220-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.270 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.270 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.270 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.270 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.270 DEBUG:boto:Method: GET 2024-04-16T20:00:27.270 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.271 DEBUG:boto:Data: 2024-04-16T20:00:27.271 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.271 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.271 DEBUG:boto:Params: {} 2024-04-16T20:00:27.271 DEBUG:boto:Token: None 2024-04-16T20:00:27.271 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ece3537ecd87207-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.314 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.315 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.315 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.315 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.315 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.315 DEBUG:boto:Method: GET 2024-04-16T20:00:27.315 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:00:27.315 DEBUG:boto:Data: 2024-04-16T20:00:27.315 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.315 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.316 DEBUG:boto:Params: {} 2024-04-16T20:00:27.316 DEBUG:boto:Token: None 2024-04-16T20:00:27.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44e9fb23a75fbc5-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.319 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:00:27.319 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.319 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.319 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.320 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.320 DEBUG:boto:Method: GET 2024-04-16T20:00:27.320 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.320 DEBUG:boto:Data: 2024-04-16T20:00:27.320 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.320 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.320 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.320 DEBUG:boto:Params: {} 2024-04-16T20:00:27.320 DEBUG:boto:Token: None 2024-04-16T20:00:27.320 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E= 2024-04-16T20:00:27.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E=', 'Content-Length': '0'} 2024-04-16T20:00:27.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a76afceb245c15d8-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.366 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.366 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.367 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.367 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.367 DEBUG:boto:Method: GET 2024-04-16T20:00:27.367 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.367 DEBUG:boto:Data: 2024-04-16T20:00:27.367 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.367 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.367 DEBUG:boto:Params: {} 2024-04-16T20:00:27.367 DEBUG:boto:Token: None 2024-04-16T20:00:27.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E= 2024-04-16T20:00:27.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E=', 'Content-Length': '0'} 2024-04-16T20:00:27.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3159810714a6b2-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.370 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.370 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.370 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.370 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.370 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.371 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.371 DEBUG:boto:Data: 2024-04-16T20:00:27.371 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.371 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.371 DEBUG:boto:Params: {} 2024-04-16T20:00:27.371 DEBUG:boto:Token: None 2024-04-16T20:00:27.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1H9OUxEMHwD4SZ5CFWK0rw4Ztb4= 2024-04-16T20:00:27.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1H9OUxEMHwD4SZ5CFWK0rw4Ztb4=', 'Content-Length': '0'} 2024-04-16T20:00:27.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aabd0b1a175c676-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.374 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.374 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.374 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.374 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.374 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.374 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.374 DEBUG:boto:Data: 2024-04-16T20:00:27.374 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.374 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.375 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.375 DEBUG:boto:Params: {} 2024-04-16T20:00:27.375 DEBUG:boto:Token: None 2024-04-16T20:00:27.375 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1H9OUxEMHwD4SZ5CFWK0rw4Ztb4= 2024-04-16T20:00:27.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1H9OUxEMHwD4SZ5CFWK0rw4Ztb4=', 'Content-Length': '0'} 2024-04-16T20:00:27.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fedbcfd426eabc4-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:00:27.377 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.377 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.377 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.377 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.378 DEBUG:boto:Method: GET 2024-04-16T20:00:27.378 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.378 DEBUG:boto:Data: 2024-04-16T20:00:27.378 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.378 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.378 DEBUG:boto:Params: {} 2024-04-16T20:00:27.378 DEBUG:boto:Token: None 2024-04-16T20:00:27.378 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E= 2024-04-16T20:00:27.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E=', 'Content-Length': '0'} 2024-04-16T20:00:27.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000955e4831ec34d935-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.422 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.422 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.423 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.423 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.423 DEBUG:boto:Method: GET 2024-04-16T20:00:27.423 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.423 DEBUG:boto:Data: 2024-04-16T20:00:27.423 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.423 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.423 DEBUG:boto:Params: {} 2024-04-16T20:00:27.423 DEBUG:boto:Token: None 2024-04-16T20:00:27.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:00:27.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E= 2024-04-16T20:00:27.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+sYvTVqhZMSpMH5HlQLe0Hcud4E=', 'Content-Length': '0'} 2024-04-16T20:00:27.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e1d270a1f92dc5-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.426 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.426 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.426 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.426 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.426 DEBUG:boto:Method: GET 2024-04-16T20:00:27.427 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.427 DEBUG:boto:Data: 2024-04-16T20:00:27.427 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.427 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.427 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.427 DEBUG:boto:Params: {} 2024-04-16T20:00:27.427 DEBUG:boto:Token: None 2024-04-16T20:00:27.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c1eebebd3b1727-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.470 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.471 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.471 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.471 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.471 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.471 DEBUG:boto:Method: GET 2024-04-16T20:00:27.471 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.471 DEBUG:boto:Data: 2024-04-16T20:00:27.471 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.472 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.472 DEBUG:boto:Params: {} 2024-04-16T20:00:27.472 DEBUG:boto:Token: None 2024-04-16T20:00:27.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1b0b13d47cd156-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.475 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.476 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.476 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.476 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.476 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:00:27.476 DEBUG:boto:Data: 2024-04-16T20:00:27.476 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.476 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.476 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.476 DEBUG:boto:Params: {} 2024-04-16T20:00:27.476 DEBUG:boto:Token: None 2024-04-16T20:00:27.476 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34 2024-04-16T20:00:27.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e8xTFDh+6CRds9QlzMd0+sm/UG8= 2024-04-16T20:00:27.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e8xTFDh+6CRds9QlzMd0+sm/UG8=', 'Content-Length': '0'} 2024-04-16T20:00:27.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e3b0dc0d531143-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.479 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.479 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.479 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.479 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:00:27.479 DEBUG:boto:Data: 2024-04-16T20:00:27.479 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.479 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.479 DEBUG:boto:Params: {} 2024-04-16T20:00:27.480 DEBUG:boto:Token: None 2024-04-16T20:00:27.480 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34 2024-04-16T20:00:27.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e8xTFDh+6CRds9QlzMd0+sm/UG8= 2024-04-16T20:00:27.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e8xTFDh+6CRds9QlzMd0+sm/UG8=', 'Content-Length': '0'} 2024-04-16T20:00:27.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab7b7085bc5f7aa-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:00:27.482 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.482 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.482 DEBUG:boto:Method: GET 2024-04-16T20:00:27.482 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:00:27.482 DEBUG:boto:Data: 2024-04-16T20:00:27.482 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.482 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.482 DEBUG:boto:Params: {} 2024-04-16T20:00:27.482 DEBUG:boto:Token: None 2024-04-16T20:00:27.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34 2024-04-16T20:00:27.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wbx3scxhhzhoFb6CwHNKoc81Hiw= 2024-04-16T20:00:27.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wbx3scxhhzhoFb6CwHNKoc81Hiw=', 'Content-Length': '0'} 2024-04-16T20:00:27.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615024cba4007162-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.526 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:00:27.526 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:00:27.527 DEBUG:boto:Method: GET 2024-04-16T20:00:27.527 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:00:27.527 DEBUG:boto:Data: 2024-04-16T20:00:27.527 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.527 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.527 DEBUG:boto:Params: {} 2024-04-16T20:00:27.527 DEBUG:boto:Token: None 2024-04-16T20:00:27.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj34 2024-04-16T20:00:27.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wbx3scxhhzhoFb6CwHNKoc81Hiw= 2024-04-16T20:00:27.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wbx3scxhhzhoFb6CwHNKoc81Hiw=', 'Content-Length': '0'} 2024-04-16T20:00:27.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b47023bd8c31d6-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.529 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.530 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.530 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.530 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.530 DEBUG:boto:Method: GET 2024-04-16T20:00:27.530 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.530 DEBUG:boto:Data: 2024-04-16T20:00:27.530 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.530 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.530 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.530 DEBUG:boto:Params: {} 2024-04-16T20:00:27.530 DEBUG:boto:Token: None 2024-04-16T20:00:27.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ead3457a7da387ea-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.574 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.575 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.575 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.575 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.575 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.575 DEBUG:boto:Method: GET 2024-04-16T20:00:27.575 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:00:27.575 DEBUG:boto:Data: 2024-04-16T20:00:27.575 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.575 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.576 DEBUG:boto:Params: {} 2024-04-16T20:00:27.576 DEBUG:boto:Token: None 2024-04-16T20:00:27.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad9f6174bef867c-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.579 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:00:27.579 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.579 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.579 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.580 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.580 DEBUG:boto:Method: GET 2024-04-16T20:00:27.580 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.580 DEBUG:boto:Data: 2024-04-16T20:00:27.580 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.580 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.580 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.580 DEBUG:boto:Params: {} 2024-04-16T20:00:27.580 DEBUG:boto:Token: None 2024-04-16T20:00:27.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU= 2024-04-16T20:00:27.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU=', 'Content-Length': '0'} 2024-04-16T20:00:27.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f5fa382626456a-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.626 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.626 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.627 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.627 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.627 DEBUG:boto:Method: GET 2024-04-16T20:00:27.627 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.627 DEBUG:boto:Data: 2024-04-16T20:00:27.627 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.627 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.627 DEBUG:boto:Params: {} 2024-04-16T20:00:27.627 DEBUG:boto:Token: None 2024-04-16T20:00:27.627 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU= 2024-04-16T20:00:27.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU=', 'Content-Length': '0'} 2024-04-16T20:00:27.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e129fecd950cbb-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.630 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.630 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.630 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.630 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.630 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.631 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.631 DEBUG:boto:Data: 2024-04-16T20:00:27.631 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.631 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.631 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.631 DEBUG:boto:Params: {} 2024-04-16T20:00:27.631 DEBUG:boto:Token: None 2024-04-16T20:00:27.631 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MeeM5vfEvwMf1jgA7pcYr/dFr/M= 2024-04-16T20:00:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MeeM5vfEvwMf1jgA7pcYr/dFr/M=', 'Content-Length': '0'} 2024-04-16T20:00:27.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005198714c180bfe42-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.634 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.634 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.634 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.634 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.634 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.634 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.634 DEBUG:boto:Data: 2024-04-16T20:00:27.634 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.635 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.635 DEBUG:boto:Params: {} 2024-04-16T20:00:27.635 DEBUG:boto:Token: None 2024-04-16T20:00:27.635 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MeeM5vfEvwMf1jgA7pcYr/dFr/M= 2024-04-16T20:00:27.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MeeM5vfEvwMf1jgA7pcYr/dFr/M=', 'Content-Length': '0'} 2024-04-16T20:00:27.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cdcc3788fd82db4-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:00:27.637 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.637 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.638 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.638 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.638 DEBUG:boto:Method: GET 2024-04-16T20:00:27.638 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.638 DEBUG:boto:Data: 2024-04-16T20:00:27.638 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.638 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.638 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.638 DEBUG:boto:Params: {} 2024-04-16T20:00:27.638 DEBUG:boto:Token: None 2024-04-16T20:00:27.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU= 2024-04-16T20:00:27.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU=', 'Content-Length': '0'} 2024-04-16T20:00:27.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f38bdde4f9081f6-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.682 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.682 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.683 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.683 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.683 DEBUG:boto:Method: GET 2024-04-16T20:00:27.683 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.683 DEBUG:boto:Data: 2024-04-16T20:00:27.683 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.683 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.683 DEBUG:boto:Params: {} 2024-04-16T20:00:27.683 DEBUG:boto:Token: None 2024-04-16T20:00:27.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:00:27.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU= 2024-04-16T20:00:27.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kfzhhEu3TdDhlaLOws7et6JdSU=', 'Content-Length': '0'} 2024-04-16T20:00:27.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81de4d024bdf1e1-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.686 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.687 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.687 DEBUG:boto:Method: GET 2024-04-16T20:00:27.687 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.687 DEBUG:boto:Data: 2024-04-16T20:00:27.687 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.687 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.687 DEBUG:boto:Params: {} 2024-04-16T20:00:27.687 DEBUG:boto:Token: None 2024-04-16T20:00:27.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba150c688f9580e-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.730 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.731 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.731 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.731 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.731 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.731 DEBUG:boto:Method: GET 2024-04-16T20:00:27.731 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.731 DEBUG:boto:Data: 2024-04-16T20:00:27.731 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.731 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.732 DEBUG:boto:Params: {} 2024-04-16T20:00:27.732 DEBUG:boto:Token: None 2024-04-16T20:00:27.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef76a2dfeda77c83-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.735 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.735 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.735 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.735 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.736 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:00:27.736 DEBUG:boto:Data: 2024-04-16T20:00:27.736 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.736 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.736 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.736 DEBUG:boto:Params: {} 2024-04-16T20:00:27.736 DEBUG:boto:Token: None 2024-04-16T20:00:27.736 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44 2024-04-16T20:00:27.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5T9g/XFeP+pTdGPMha6Tr+CrEmI= 2024-04-16T20:00:27.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5T9g/XFeP+pTdGPMha6Tr+CrEmI=', 'Content-Length': '0'} 2024-04-16T20:00:27.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069acd7227e0e8c1c-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.739 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.739 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.739 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.739 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:00:27.739 DEBUG:boto:Data: 2024-04-16T20:00:27.739 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.739 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.739 DEBUG:boto:Params: {} 2024-04-16T20:00:27.739 DEBUG:boto:Token: None 2024-04-16T20:00:27.740 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44 2024-04-16T20:00:27.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5T9g/XFeP+pTdGPMha6Tr+CrEmI= 2024-04-16T20:00:27.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5T9g/XFeP+pTdGPMha6Tr+CrEmI=', 'Content-Length': '0'} 2024-04-16T20:00:27.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6e7a9b2cae66ce-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:00:27.742 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.742 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.742 DEBUG:boto:Method: GET 2024-04-16T20:00:27.742 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:00:27.742 DEBUG:boto:Data: 2024-04-16T20:00:27.742 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.742 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.743 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.743 DEBUG:boto:Params: {} 2024-04-16T20:00:27.743 DEBUG:boto:Token: None 2024-04-16T20:00:27.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44 2024-04-16T20:00:27.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dmH6urx7ZyImbeQ0RsvM4hUWwrs= 2024-04-16T20:00:27.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dmH6urx7ZyImbeQ0RsvM4hUWwrs=', 'Content-Length': '0'} 2024-04-16T20:00:27.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c394f89ac44a4543-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.786 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:00:27.787 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:00:27.787 DEBUG:boto:Method: GET 2024-04-16T20:00:27.787 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:00:27.787 DEBUG:boto:Data: 2024-04-16T20:00:27.787 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.787 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.787 DEBUG:boto:Params: {} 2024-04-16T20:00:27.787 DEBUG:boto:Token: None 2024-04-16T20:00:27.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj44 2024-04-16T20:00:27.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dmH6urx7ZyImbeQ0RsvM4hUWwrs= 2024-04-16T20:00:27.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dmH6urx7ZyImbeQ0RsvM4hUWwrs=', 'Content-Length': '0'} 2024-04-16T20:00:27.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5e32dd6543c701-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.790 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.790 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.790 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.790 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.790 DEBUG:boto:Method: GET 2024-04-16T20:00:27.790 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.790 DEBUG:boto:Data: 2024-04-16T20:00:27.791 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.791 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.791 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.791 DEBUG:boto:Params: {} 2024-04-16T20:00:27.791 DEBUG:boto:Token: None 2024-04-16T20:00:27.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a10478bf5517568f-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.834 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.835 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.835 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.835 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.835 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.835 DEBUG:boto:Method: GET 2024-04-16T20:00:27.835 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:00:27.835 DEBUG:boto:Data: 2024-04-16T20:00:27.835 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.835 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.836 DEBUG:boto:Params: {} 2024-04-16T20:00:27.836 DEBUG:boto:Token: None 2024-04-16T20:00:27.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be20664d09ba0b1d-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.839 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:00:27.839 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.839 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.839 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.840 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.840 DEBUG:boto:Method: GET 2024-04-16T20:00:27.840 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.840 DEBUG:boto:Data: 2024-04-16T20:00:27.840 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.840 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.840 DEBUG:boto:Params: {} 2024-04-16T20:00:27.840 DEBUG:boto:Token: None 2024-04-16T20:00:27.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o= 2024-04-16T20:00:27.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o=', 'Content-Length': '0'} 2024-04-16T20:00:27.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702f187f87280eab-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.886 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.886 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.887 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.887 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.887 DEBUG:boto:Method: GET 2024-04-16T20:00:27.887 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.887 DEBUG:boto:Data: 2024-04-16T20:00:27.887 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.887 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.887 DEBUG:boto:Params: {} 2024-04-16T20:00:27.887 DEBUG:boto:Token: None 2024-04-16T20:00:27.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o= 2024-04-16T20:00:27.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o=', 'Content-Length': '0'} 2024-04-16T20:00:27.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067753c4f65c1cfd8-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.890 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.890 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.890 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.891 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.891 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.891 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.891 DEBUG:boto:Data: 2024-04-16T20:00:27.891 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.891 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.891 DEBUG:boto:Params: {} 2024-04-16T20:00:27.891 DEBUG:boto:Token: None 2024-04-16T20:00:27.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0R0gAaDi18qhCeufhoDt7H+qSes= 2024-04-16T20:00:27.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0R0gAaDi18qhCeufhoDt7H+qSes=', 'Content-Length': '0'} 2024-04-16T20:00:27.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032657d7a808cabe6-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.894 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.894 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.894 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.894 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.894 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.894 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.894 DEBUG:boto:Data: 2024-04-16T20:00:27.895 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.895 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.895 DEBUG:boto:Params: {} 2024-04-16T20:00:27.895 DEBUG:boto:Token: None 2024-04-16T20:00:27.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0R0gAaDi18qhCeufhoDt7H+qSes= 2024-04-16T20:00:27.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0R0gAaDi18qhCeufhoDt7H+qSes=', 'Content-Length': '0'} 2024-04-16T20:00:27.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8c122059c74f8e-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:00:27.897 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.898 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.898 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.898 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.898 DEBUG:boto:Method: GET 2024-04-16T20:00:27.898 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.898 DEBUG:boto:Data: 2024-04-16T20:00:27.898 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.898 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.898 DEBUG:boto:Params: {} 2024-04-16T20:00:27.899 DEBUG:boto:Token: None 2024-04-16T20:00:27.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o= 2024-04-16T20:00:27.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o=', 'Content-Length': '0'} 2024-04-16T20:00:27.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005add505f041b3c42-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.942 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.942 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.943 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.943 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.943 DEBUG:boto:Method: GET 2024-04-16T20:00:27.943 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.943 DEBUG:boto:Data: 2024-04-16T20:00:27.943 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.943 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.943 DEBUG:boto:Params: {} 2024-04-16T20:00:27.943 DEBUG:boto:Token: None 2024-04-16T20:00:27.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:00:27.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o= 2024-04-16T20:00:27.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pByxlS85YxDfHElaFMT03vEPq4o=', 'Content-Length': '0'} 2024-04-16T20:00:27.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098896d4eeaab6dd7-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.946 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.946 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.946 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.946 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.946 DEBUG:boto:Method: GET 2024-04-16T20:00:27.947 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.947 DEBUG:boto:Data: 2024-04-16T20:00:27.947 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.947 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.947 DEBUG:boto:Params: {} 2024-04-16T20:00:27.947 DEBUG:boto:Token: None 2024-04-16T20:00:27.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c5b50e64ccc6966-00661ed8db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.990 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.991 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:27.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:27.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.991 DEBUG:boto:Method: GET 2024-04-16T20:00:27.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:00:27.991 DEBUG:boto:Data: 2024-04-16T20:00:27.991 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:27.992 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.992 DEBUG:boto:Params: {} 2024-04-16T20:00:27.992 DEBUG:boto:Token: None 2024-04-16T20:00:27.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/ 2024-04-16T20:00:27.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM= 2024-04-16T20:00:27.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HVZ4UlGV7g9Xrd24r7s8TxEmfGM=', 'Content-Length': '0'} 2024-04-16T20:00:27.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011bd958ea4557a89-00661ed8db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.995 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:27.995 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.996 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.996 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.996 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:00:27.996 DEBUG:boto:Data: 2024-04-16T20:00:27.996 DEBUG:boto:Headers: {} 2024-04-16T20:00:27.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:27.996 DEBUG:boto:Port: 8000 2024-04-16T20:00:27.996 DEBUG:boto:Params: {} 2024-04-16T20:00:27.996 DEBUG:boto:Token: None 2024-04-16T20:00:27.996 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:27 GMT /yrjefh-52/obj51 2024-04-16T20:00:27.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XipaP8oe+pWJoNkqzeM2y5CfI5I= 2024-04-16T20:00:27.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XipaP8oe+pWJoNkqzeM2y5CfI5I=', 'Content-Length': '0'} 2024-04-16T20:00:27.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da5166e64333228e-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:27.999 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:27.999 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:27.999 DEBUG:boto:Method: HEAD 2024-04-16T20:00:27.999 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:00:27.999 DEBUG:boto:Data: 2024-04-16T20:00:28.000 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.000 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.000 DEBUG:boto:Params: {} 2024-04-16T20:00:28.000 DEBUG:boto:Token: None 2024-04-16T20:00:28.000 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj51 2024-04-16T20:00:28.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zZpT9C45z+9ZShfuElQ/k+RvswE= 2024-04-16T20:00:28.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zZpT9C45z+9ZShfuElQ/k+RvswE=', 'Content-Length': '0'} 2024-04-16T20:00:28.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d548a6ffd623eb-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:00:28.003 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:28.003 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:28.003 DEBUG:boto:Method: GET 2024-04-16T20:00:28.003 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:00:28.003 DEBUG:boto:Data: 2024-04-16T20:00:28.003 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.003 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.003 DEBUG:boto:Params: {} 2024-04-16T20:00:28.003 DEBUG:boto:Token: None 2024-04-16T20:00:28.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj51 2024-04-16T20:00:28.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MhMLNThVpr1EjiysZBZl2VVjaYE= 2024-04-16T20:00:28.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MhMLNThVpr1EjiysZBZl2VVjaYE=', 'Content-Length': '0'} 2024-04-16T20:00:28.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056fb7831533372a0-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.046 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:00:28.046 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:00:28.047 DEBUG:boto:Method: GET 2024-04-16T20:00:28.047 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:00:28.047 DEBUG:boto:Data: 2024-04-16T20:00:28.047 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.047 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.047 DEBUG:boto:Params: {} 2024-04-16T20:00:28.047 DEBUG:boto:Token: None 2024-04-16T20:00:28.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj51 2024-04-16T20:00:28.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MhMLNThVpr1EjiysZBZl2VVjaYE= 2024-04-16T20:00:28.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MhMLNThVpr1EjiysZBZl2VVjaYE=', 'Content-Length': '0'} 2024-04-16T20:00:28.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d01e67b3b7fecbb-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.050 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.050 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.050 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.050 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.051 DEBUG:boto:Method: GET 2024-04-16T20:00:28.051 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.051 DEBUG:boto:Data: 2024-04-16T20:00:28.051 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.051 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.051 DEBUG:boto:Params: {} 2024-04-16T20:00:28.051 DEBUG:boto:Token: None 2024-04-16T20:00:28.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074321e9bbc286c86-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.094 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.095 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.095 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.095 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.095 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.095 DEBUG:boto:Method: GET 2024-04-16T20:00:28.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:00:28.095 DEBUG:boto:Data: 2024-04-16T20:00:28.095 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.095 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.096 DEBUG:boto:Params: {} 2024-04-16T20:00:28.096 DEBUG:boto:Token: None 2024-04-16T20:00:28.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000686a8bccb457c1f5-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.099 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:00:28.099 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.100 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.100 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.100 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.100 DEBUG:boto:Method: GET 2024-04-16T20:00:28.100 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.100 DEBUG:boto:Data: 2024-04-16T20:00:28.100 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.100 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.100 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.100 DEBUG:boto:Params: {} 2024-04-16T20:00:28.100 DEBUG:boto:Token: None 2024-04-16T20:00:28.101 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.101 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o= 2024-04-16T20:00:28.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o=', 'Content-Length': '0'} 2024-04-16T20:00:28.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7916b53f9f8d25b-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.146 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.146 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.147 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.147 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.147 DEBUG:boto:Method: GET 2024-04-16T20:00:28.147 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.147 DEBUG:boto:Data: 2024-04-16T20:00:28.147 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.147 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.147 DEBUG:boto:Params: {} 2024-04-16T20:00:28.147 DEBUG:boto:Token: None 2024-04-16T20:00:28.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o= 2024-04-16T20:00:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o=', 'Content-Length': '0'} 2024-04-16T20:00:28.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f230e5e887cc3944-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.150 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.150 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.150 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.150 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.150 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.150 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.150 DEBUG:boto:Data: 2024-04-16T20:00:28.150 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.151 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.151 DEBUG:boto:Params: {} 2024-04-16T20:00:28.151 DEBUG:boto:Token: None 2024-04-16T20:00:28.151 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y5QfQn7xNpdjFFJldkz24CkQwbk= 2024-04-16T20:00:28.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y5QfQn7xNpdjFFJldkz24CkQwbk=', 'Content-Length': '0'} 2024-04-16T20:00:28.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f26fd34f516027-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.153 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.153 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.154 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.154 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.154 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.154 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.154 DEBUG:boto:Data: 2024-04-16T20:00:28.154 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.154 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.154 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.154 DEBUG:boto:Params: {} 2024-04-16T20:00:28.154 DEBUG:boto:Token: None 2024-04-16T20:00:28.154 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:y5QfQn7xNpdjFFJldkz24CkQwbk= 2024-04-16T20:00:28.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:y5QfQn7xNpdjFFJldkz24CkQwbk=', 'Content-Length': '0'} 2024-04-16T20:00:28.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036534f88d5c8d39f-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:00:28.157 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.157 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.157 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.157 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.157 DEBUG:boto:Method: GET 2024-04-16T20:00:28.157 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.158 DEBUG:boto:Data: 2024-04-16T20:00:28.158 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.158 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.158 DEBUG:boto:Params: {} 2024-04-16T20:00:28.158 DEBUG:boto:Token: None 2024-04-16T20:00:28.158 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o= 2024-04-16T20:00:28.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o=', 'Content-Length': '0'} 2024-04-16T20:00:28.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38820080dc01fa7-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.202 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.202 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.203 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.203 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.203 DEBUG:boto:Method: GET 2024-04-16T20:00:28.203 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.203 DEBUG:boto:Data: 2024-04-16T20:00:28.203 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.203 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.203 DEBUG:boto:Params: {} 2024-04-16T20:00:28.203 DEBUG:boto:Token: None 2024-04-16T20:00:28.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:00:28.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o= 2024-04-16T20:00:28.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sPbA6hXedBi15HjDEd7pcazH20o=', 'Content-Length': '0'} 2024-04-16T20:00:28.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3ecc92f8d927b8-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.206 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.206 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.206 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.206 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.206 DEBUG:boto:Method: GET 2024-04-16T20:00:28.206 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.207 DEBUG:boto:Data: 2024-04-16T20:00:28.207 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.207 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.207 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.207 DEBUG:boto:Params: {} 2024-04-16T20:00:28.207 DEBUG:boto:Token: None 2024-04-16T20:00:28.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385eb6f64301c3df-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.250 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.251 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.251 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.251 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.251 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.251 DEBUG:boto:Method: GET 2024-04-16T20:00:28.251 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.251 DEBUG:boto:Data: 2024-04-16T20:00:28.251 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.251 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.252 DEBUG:boto:Params: {} 2024-04-16T20:00:28.252 DEBUG:boto:Token: None 2024-04-16T20:00:28.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0cd5bf3f16dd732-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.255 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.255 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.255 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.255 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.255 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:00:28.256 DEBUG:boto:Data: 2024-04-16T20:00:28.256 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.256 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.256 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.256 DEBUG:boto:Params: {} 2024-04-16T20:00:28.256 DEBUG:boto:Token: None 2024-04-16T20:00:28.256 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55 2024-04-16T20:00:28.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0RMMbe1wm6yjDIjZbrIsIuoZXPg= 2024-04-16T20:00:28.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0RMMbe1wm6yjDIjZbrIsIuoZXPg=', 'Content-Length': '0'} 2024-04-16T20:00:28.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5bddc8610d7b35-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.259 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.259 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.259 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.259 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:00:28.259 DEBUG:boto:Data: 2024-04-16T20:00:28.259 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.259 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.259 DEBUG:boto:Params: {} 2024-04-16T20:00:28.260 DEBUG:boto:Token: None 2024-04-16T20:00:28.260 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55 2024-04-16T20:00:28.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0RMMbe1wm6yjDIjZbrIsIuoZXPg= 2024-04-16T20:00:28.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0RMMbe1wm6yjDIjZbrIsIuoZXPg=', 'Content-Length': '0'} 2024-04-16T20:00:28.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203f61e5416b1f91-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:00:28.262 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.262 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.262 DEBUG:boto:Method: GET 2024-04-16T20:00:28.262 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:00:28.262 DEBUG:boto:Data: 2024-04-16T20:00:28.262 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.263 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.263 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.263 DEBUG:boto:Params: {} 2024-04-16T20:00:28.263 DEBUG:boto:Token: None 2024-04-16T20:00:28.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55 2024-04-16T20:00:28.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qVTwJsfyqZt6H3FBJe6Zkw/Peik= 2024-04-16T20:00:28.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qVTwJsfyqZt6H3FBJe6Zkw/Peik=', 'Content-Length': '0'} 2024-04-16T20:00:28.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d1317464c8ab4b-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.306 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:00:28.307 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:00:28.307 DEBUG:boto:Method: GET 2024-04-16T20:00:28.307 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:00:28.307 DEBUG:boto:Data: 2024-04-16T20:00:28.307 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.307 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.307 DEBUG:boto:Params: {} 2024-04-16T20:00:28.307 DEBUG:boto:Token: None 2024-04-16T20:00:28.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj55 2024-04-16T20:00:28.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qVTwJsfyqZt6H3FBJe6Zkw/Peik= 2024-04-16T20:00:28.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qVTwJsfyqZt6H3FBJe6Zkw/Peik=', 'Content-Length': '0'} 2024-04-16T20:00:28.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc8e86f37075e9a-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.310 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.310 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.310 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.310 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.310 DEBUG:boto:Method: GET 2024-04-16T20:00:28.311 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.311 DEBUG:boto:Data: 2024-04-16T20:00:28.311 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.311 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.311 DEBUG:boto:Params: {} 2024-04-16T20:00:28.311 DEBUG:boto:Token: None 2024-04-16T20:00:28.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003659a49f1527ae01-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.354 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.355 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.355 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.355 DEBUG:boto:Method: GET 2024-04-16T20:00:28.355 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:00:28.355 DEBUG:boto:Data: 2024-04-16T20:00:28.355 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.355 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.355 DEBUG:boto:Params: {} 2024-04-16T20:00:28.356 DEBUG:boto:Token: None 2024-04-16T20:00:28.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043bb4670e92f3a16-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.359 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:00:28.359 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.359 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.359 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.360 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.360 DEBUG:boto:Method: GET 2024-04-16T20:00:28.360 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.360 DEBUG:boto:Data: 2024-04-16T20:00:28.360 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.360 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.360 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.360 DEBUG:boto:Params: {} 2024-04-16T20:00:28.360 DEBUG:boto:Token: None 2024-04-16T20:00:28.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk= 2024-04-16T20:00:28.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk=', 'Content-Length': '0'} 2024-04-16T20:00:28.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643e6ea76cad2056-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.406 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.406 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.406 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.407 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.407 DEBUG:boto:Method: GET 2024-04-16T20:00:28.407 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.407 DEBUG:boto:Data: 2024-04-16T20:00:28.407 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.407 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.407 DEBUG:boto:Params: {} 2024-04-16T20:00:28.407 DEBUG:boto:Token: None 2024-04-16T20:00:28.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk= 2024-04-16T20:00:28.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk=', 'Content-Length': '0'} 2024-04-16T20:00:28.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041994d4468db6b89-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.410 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.410 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.410 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.410 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.410 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.410 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.410 DEBUG:boto:Data: 2024-04-16T20:00:28.410 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.410 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.410 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.411 DEBUG:boto:Params: {} 2024-04-16T20:00:28.411 DEBUG:boto:Token: None 2024-04-16T20:00:28.411 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v9rFQjA/gzEEEZQ+EpsiI5jJU/s= 2024-04-16T20:00:28.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v9rFQjA/gzEEEZQ+EpsiI5jJU/s=', 'Content-Length': '0'} 2024-04-16T20:00:28.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007def1bdfc263782b-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.413 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.413 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.413 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.413 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.413 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.413 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.413 DEBUG:boto:Data: 2024-04-16T20:00:28.414 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.414 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.414 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.414 DEBUG:boto:Params: {} 2024-04-16T20:00:28.414 DEBUG:boto:Token: None 2024-04-16T20:00:28.414 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.414 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v9rFQjA/gzEEEZQ+EpsiI5jJU/s= 2024-04-16T20:00:28.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v9rFQjA/gzEEEZQ+EpsiI5jJU/s=', 'Content-Length': '0'} 2024-04-16T20:00:28.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183bed45abbdc2be-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:00:28.416 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.416 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.416 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.417 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.417 DEBUG:boto:Method: GET 2024-04-16T20:00:28.417 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.417 DEBUG:boto:Data: 2024-04-16T20:00:28.417 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.417 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.417 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.417 DEBUG:boto:Params: {} 2024-04-16T20:00:28.417 DEBUG:boto:Token: None 2024-04-16T20:00:28.417 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk= 2024-04-16T20:00:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk=', 'Content-Length': '0'} 2024-04-16T20:00:28.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015a1026024c22795-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.462 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.462 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.462 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.463 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.463 DEBUG:boto:Method: GET 2024-04-16T20:00:28.463 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.463 DEBUG:boto:Data: 2024-04-16T20:00:28.463 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.463 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.463 DEBUG:boto:Params: {} 2024-04-16T20:00:28.463 DEBUG:boto:Token: None 2024-04-16T20:00:28.463 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:00:28.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk= 2024-04-16T20:00:28.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D2mIctjByhVxuoYaL4zSMyo+Sk=', 'Content-Length': '0'} 2024-04-16T20:00:28.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde1609a57985fd4-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.465 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.466 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.466 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.466 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.466 DEBUG:boto:Method: GET 2024-04-16T20:00:28.466 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.466 DEBUG:boto:Data: 2024-04-16T20:00:28.466 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.466 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.466 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.466 DEBUG:boto:Params: {} 2024-04-16T20:00:28.467 DEBUG:boto:Token: None 2024-04-16T20:00:28.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05286f272ec7fc5-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.510 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.511 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.511 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.511 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.511 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.511 DEBUG:boto:Method: GET 2024-04-16T20:00:28.511 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.511 DEBUG:boto:Data: 2024-04-16T20:00:28.511 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.511 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.511 DEBUG:boto:Params: {} 2024-04-16T20:00:28.511 DEBUG:boto:Token: None 2024-04-16T20:00:28.512 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f997dedbdcf4d1bf-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.515 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.515 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.515 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.515 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.515 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:00:28.515 DEBUG:boto:Data: 2024-04-16T20:00:28.515 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.515 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.515 DEBUG:boto:Params: {} 2024-04-16T20:00:28.516 DEBUG:boto:Token: None 2024-04-16T20:00:28.516 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68 2024-04-16T20:00:28.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mbBeuEb1TJ4h5zDG7cGS6ZfBcvo= 2024-04-16T20:00:28.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mbBeuEb1TJ4h5zDG7cGS6ZfBcvo=', 'Content-Length': '0'} 2024-04-16T20:00:28.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ef8b51918917bd3-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.518 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.518 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.518 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.519 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:00:28.519 DEBUG:boto:Data: 2024-04-16T20:00:28.519 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.519 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.519 DEBUG:boto:Params: {} 2024-04-16T20:00:28.519 DEBUG:boto:Token: None 2024-04-16T20:00:28.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68 2024-04-16T20:00:28.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mbBeuEb1TJ4h5zDG7cGS6ZfBcvo= 2024-04-16T20:00:28.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mbBeuEb1TJ4h5zDG7cGS6ZfBcvo=', 'Content-Length': '0'} 2024-04-16T20:00:28.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d94cfb4134fc47-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:00:28.522 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.522 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.522 DEBUG:boto:Method: GET 2024-04-16T20:00:28.522 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:00:28.522 DEBUG:boto:Data: 2024-04-16T20:00:28.522 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.522 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.522 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.522 DEBUG:boto:Params: {} 2024-04-16T20:00:28.522 DEBUG:boto:Token: None 2024-04-16T20:00:28.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68 2024-04-16T20:00:28.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouQ1c+uW9eDpISvVJ+kskOALVaw= 2024-04-16T20:00:28.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouQ1c+uW9eDpISvVJ+kskOALVaw=', 'Content-Length': '0'} 2024-04-16T20:00:28.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089187dcc29761922-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.566 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:00:28.566 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:00:28.567 DEBUG:boto:Method: GET 2024-04-16T20:00:28.567 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:00:28.567 DEBUG:boto:Data: 2024-04-16T20:00:28.567 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.567 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.567 DEBUG:boto:Params: {} 2024-04-16T20:00:28.567 DEBUG:boto:Token: None 2024-04-16T20:00:28.567 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj68 2024-04-16T20:00:28.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouQ1c+uW9eDpISvVJ+kskOALVaw= 2024-04-16T20:00:28.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouQ1c+uW9eDpISvVJ+kskOALVaw=', 'Content-Length': '0'} 2024-04-16T20:00:28.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000281217c4d76660f-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.570 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.570 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.570 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.570 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.570 DEBUG:boto:Method: GET 2024-04-16T20:00:28.570 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.570 DEBUG:boto:Data: 2024-04-16T20:00:28.570 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.570 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.570 DEBUG:boto:Params: {} 2024-04-16T20:00:28.571 DEBUG:boto:Token: None 2024-04-16T20:00:28.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d24547f63bf0114-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.573 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.574 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.574 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.574 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.574 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.574 DEBUG:boto:Method: GET 2024-04-16T20:00:28.574 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:00:28.574 DEBUG:boto:Data: 2024-04-16T20:00:28.574 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.575 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.575 DEBUG:boto:Params: {} 2024-04-16T20:00:28.575 DEBUG:boto:Token: None 2024-04-16T20:00:28.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002049e1f923fe6164-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.578 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:00:28.579 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.579 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.579 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.579 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.579 DEBUG:boto:Method: GET 2024-04-16T20:00:28.579 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.579 DEBUG:boto:Data: 2024-04-16T20:00:28.579 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.579 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.579 DEBUG:boto:Params: {} 2024-04-16T20:00:28.580 DEBUG:boto:Token: None 2024-04-16T20:00:28.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI= 2024-04-16T20:00:28.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI=', 'Content-Length': '0'} 2024-04-16T20:00:28.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd755cef927f2106-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.622 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.622 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.623 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.623 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.623 DEBUG:boto:Method: GET 2024-04-16T20:00:28.623 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.623 DEBUG:boto:Data: 2024-04-16T20:00:28.623 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.623 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.623 DEBUG:boto:Params: {} 2024-04-16T20:00:28.623 DEBUG:boto:Token: None 2024-04-16T20:00:28.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI= 2024-04-16T20:00:28.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI=', 'Content-Length': '0'} 2024-04-16T20:00:28.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005526e696053ec74c-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.626 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.626 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.626 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.626 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.626 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.626 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.626 DEBUG:boto:Data: 2024-04-16T20:00:28.626 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.626 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.626 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.626 DEBUG:boto:Params: {} 2024-04-16T20:00:28.627 DEBUG:boto:Token: None 2024-04-16T20:00:28.627 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OtzO9mgg18YC4QUryhEuDxxIRkI= 2024-04-16T20:00:28.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OtzO9mgg18YC4QUryhEuDxxIRkI=', 'Content-Length': '0'} 2024-04-16T20:00:28.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f8a54434749e39-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.629 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.629 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.629 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.629 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.629 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.629 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.629 DEBUG:boto:Data: 2024-04-16T20:00:28.629 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.629 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.630 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.630 DEBUG:boto:Params: {} 2024-04-16T20:00:28.630 DEBUG:boto:Token: None 2024-04-16T20:00:28.630 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.630 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OtzO9mgg18YC4QUryhEuDxxIRkI= 2024-04-16T20:00:28.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OtzO9mgg18YC4QUryhEuDxxIRkI=', 'Content-Length': '0'} 2024-04-16T20:00:28.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d804aba8d5358e8-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:00:28.632 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.632 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.632 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.633 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.633 DEBUG:boto:Method: GET 2024-04-16T20:00:28.633 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.633 DEBUG:boto:Data: 2024-04-16T20:00:28.633 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.633 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.633 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.633 DEBUG:boto:Params: {} 2024-04-16T20:00:28.633 DEBUG:boto:Token: None 2024-04-16T20:00:28.633 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.633 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI= 2024-04-16T20:00:28.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI=', 'Content-Length': '0'} 2024-04-16T20:00:28.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc593c763bbf3e34-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.678 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.678 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.678 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.679 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.679 DEBUG:boto:Method: GET 2024-04-16T20:00:28.679 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.679 DEBUG:boto:Data: 2024-04-16T20:00:28.679 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.679 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.679 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.679 DEBUG:boto:Params: {} 2024-04-16T20:00:28.679 DEBUG:boto:Token: None 2024-04-16T20:00:28.679 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:00:28.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI= 2024-04-16T20:00:28.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6K3HHhBrI8GV1Kt5nSva1bbomJI=', 'Content-Length': '0'} 2024-04-16T20:00:28.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067fd2cd81d233158-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.682 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.682 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.682 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.682 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.682 DEBUG:boto:Method: GET 2024-04-16T20:00:28.682 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.682 DEBUG:boto:Data: 2024-04-16T20:00:28.683 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.683 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.683 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.683 DEBUG:boto:Params: {} 2024-04-16T20:00:28.683 DEBUG:boto:Token: None 2024-04-16T20:00:28.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a33d24fb79b2ea8-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.686 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.686 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.686 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.686 DEBUG:boto:Method: GET 2024-04-16T20:00:28.687 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.687 DEBUG:boto:Data: 2024-04-16T20:00:28.687 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.687 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.687 DEBUG:boto:Params: {} 2024-04-16T20:00:28.687 DEBUG:boto:Token: None 2024-04-16T20:00:28.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da4b4ac3a1193f4-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.690 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.690 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.691 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.691 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.691 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:00:28.691 DEBUG:boto:Data: 2024-04-16T20:00:28.691 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.691 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.691 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.691 DEBUG:boto:Params: {} 2024-04-16T20:00:28.691 DEBUG:boto:Token: None 2024-04-16T20:00:28.691 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85 2024-04-16T20:00:28.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dE3n/dHeZhrSX5GgHvydwFojP4= 2024-04-16T20:00:28.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dE3n/dHeZhrSX5GgHvydwFojP4=', 'Content-Length': '0'} 2024-04-16T20:00:28.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8ab6c5c5a3fd70-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.694 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.694 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.694 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.694 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:00:28.694 DEBUG:boto:Data: 2024-04-16T20:00:28.694 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.694 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.694 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.694 DEBUG:boto:Params: {} 2024-04-16T20:00:28.695 DEBUG:boto:Token: None 2024-04-16T20:00:28.695 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85 2024-04-16T20:00:28.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dE3n/dHeZhrSX5GgHvydwFojP4= 2024-04-16T20:00:28.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dE3n/dHeZhrSX5GgHvydwFojP4=', 'Content-Length': '0'} 2024-04-16T20:00:28.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ff4bbae98f4d8a-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:00:28.697 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.697 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.697 DEBUG:boto:Method: GET 2024-04-16T20:00:28.697 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:00:28.697 DEBUG:boto:Data: 2024-04-16T20:00:28.697 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.697 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.698 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.698 DEBUG:boto:Params: {} 2024-04-16T20:00:28.698 DEBUG:boto:Token: None 2024-04-16T20:00:28.698 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85 2024-04-16T20:00:28.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:78dstHVa1gg7yO7d2OGbGPAPZVo= 2024-04-16T20:00:28.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:78dstHVa1gg7yO7d2OGbGPAPZVo=', 'Content-Length': '0'} 2024-04-16T20:00:28.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9981f911e50d44-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.742 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:00:28.742 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:00:28.743 DEBUG:boto:Method: GET 2024-04-16T20:00:28.743 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:00:28.743 DEBUG:boto:Data: 2024-04-16T20:00:28.743 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.743 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.743 DEBUG:boto:Params: {} 2024-04-16T20:00:28.743 DEBUG:boto:Token: None 2024-04-16T20:00:28.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj85 2024-04-16T20:00:28.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:78dstHVa1gg7yO7d2OGbGPAPZVo= 2024-04-16T20:00:28.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:78dstHVa1gg7yO7d2OGbGPAPZVo=', 'Content-Length': '0'} 2024-04-16T20:00:28.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf98b1a3850d1720-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.745 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.746 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.746 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.746 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.746 DEBUG:boto:Method: GET 2024-04-16T20:00:28.746 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.746 DEBUG:boto:Data: 2024-04-16T20:00:28.746 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.746 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:00:28.746 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.746 DEBUG:boto:Params: {} 2024-04-16T20:00:28.746 DEBUG:boto:Token: None 2024-04-16T20:00:28.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e076381360e1a6dd-00661ed8dc-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.749 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.749 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.750 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.750 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.750 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.750 DEBUG:boto:Method: GET 2024-04-16T20:00:28.750 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:00:28.750 DEBUG:boto:Data: 2024-04-16T20:00:28.750 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.750 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.750 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.750 DEBUG:boto:Params: {} 2024-04-16T20:00:28.750 DEBUG:boto:Token: None 2024-04-16T20:00:28.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8= 2024-04-16T20:00:28.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9AcoDFMMl4IiZlvWqn0y5DLP/8=', 'Content-Length': '0'} 2024-04-16T20:00:28.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98c1dd415c78aea-00661ed8dc-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.753 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:00:28.754 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:00:28.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:00:28.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:00:28.754 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:00:28.811 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:00:28.811 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:00:28.881 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:00:28.881 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T20:00:28.889 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 10 2024-04-16T20:00:28.946 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.946 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.946 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.946 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:28.946 DEBUG:boto:Data: 2024-04-16T20:00:28.947 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.947 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.947 DEBUG:boto:Params: {} 2024-04-16T20:00:28.947 DEBUG:boto:Token: None 2024-04-16T20:00:28.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fh0VMI+ex6HBKIu1fUQfiH13lM4= 2024-04-16T20:00:28.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fh0VMI+ex6HBKIu1fUQfiH13lM4=', 'Content-Length': '0'} 2024-04-16T20:00:28.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d885621e57c766e6-00661ed8dc-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.950 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:00:28.950 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:00:28.950 DEBUG:boto:Method: PUT 2024-04-16T20:00:28.950 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:00:28.950 DEBUG:boto:Data: 2024-04-16T20:00:28.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:00:28.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.951 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.951 DEBUG:boto:Params: {} 2024-04-16T20:00:28.951 DEBUG:boto:Token: None 2024-04-16T20:00:28.951 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj19 2024-04-16T20:00:28.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5Jc40z2GM7kdbnwwEd+g872xo2c= 2024-04-16T20:00:28.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5Jc40z2GM7kdbnwwEd+g872xo2c='} 2024-04-16T20:00:28.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002d44ab173966eb6-00661ed8dc-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.956 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:28.957 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:28.957 DEBUG:boto:Method: HEAD 2024-04-16T20:00:28.957 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:28.957 DEBUG:boto:Data: 2024-04-16T20:00:28.957 DEBUG:boto:Headers: {} 2024-04-16T20:00:28.957 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.957 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.957 DEBUG:boto:Params: {} 2024-04-16T20:00:28.957 DEBUG:boto:Token: None 2024-04-16T20:00:28.957 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/ 2024-04-16T20:00:28.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fh0VMI+ex6HBKIu1fUQfiH13lM4= 2024-04-16T20:00:28.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fh0VMI+ex6HBKIu1fUQfiH13lM4=', 'Content-Length': '0'} 2024-04-16T20:00:28.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0878709214a09fc-00661ed8dc-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:28.960 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:00:28.960 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:00:28.960 DEBUG:boto:Method: PUT 2024-04-16T20:00:28.960 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:00:28.960 DEBUG:boto:Data: 2024-04-16T20:00:28.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:00:28.960 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:28.961 DEBUG:boto:Port: 8000 2024-04-16T20:00:28.961 DEBUG:boto:Params: {} 2024-04-16T20:00:28.961 DEBUG:boto:Token: None 2024-04-16T20:00:28.961 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:00:28 GMT /yrjefh-52/obj38 2024-04-16T20:00:28.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6/4kBsfSYdxW+BAmNZY1reJFxc= 2024-04-16T20:00:28.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:00:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6/4kBsfSYdxW+BAmNZY1reJFxc='} 2024-04-16T20:00:29.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f87214edb1b3a7c6-00661ed8dc-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.003 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:29.003 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:29.003 DEBUG:boto:Method: HEAD 2024-04-16T20:00:29.003 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:29.003 DEBUG:boto:Data: 2024-04-16T20:00:29.003 DEBUG:boto:Headers: {} 2024-04-16T20:00:29.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.003 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.003 DEBUG:boto:Params: {} 2024-04-16T20:00:29.003 DEBUG:boto:Token: None 2024-04-16T20:00:29.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/ 2024-04-16T20:00:29.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM= 2024-04-16T20:00:29.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM=', 'Content-Length': '0'} 2024-04-16T20:00:29.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000300b7f8ec599c0ea-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.006 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:00:29.006 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:00:29.006 DEBUG:boto:Method: PUT 2024-04-16T20:00:29.006 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:00:29.007 DEBUG:boto:Data: 2024-04-16T20:00:29.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:00:29.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.007 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.007 DEBUG:boto:Params: {} 2024-04-16T20:00:29.007 DEBUG:boto:Token: None 2024-04-16T20:00:29.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/obj57 2024-04-16T20:00:29.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SfFmqubNqCb3oFL1s0JEJBhPXg8= 2024-04-16T20:00:29.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SfFmqubNqCb3oFL1s0JEJBhPXg8='} 2024-04-16T20:00:29.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb269d0fdb8c6a60-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.011 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:29.011 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:29.012 DEBUG:boto:Method: HEAD 2024-04-16T20:00:29.012 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:29.012 DEBUG:boto:Data: 2024-04-16T20:00:29.012 DEBUG:boto:Headers: {} 2024-04-16T20:00:29.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.012 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.012 DEBUG:boto:Params: {} 2024-04-16T20:00:29.012 DEBUG:boto:Token: None 2024-04-16T20:00:29.012 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/ 2024-04-16T20:00:29.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM= 2024-04-16T20:00:29.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM=', 'Content-Length': '0'} 2024-04-16T20:00:29.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a53662ce9653eb8-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.015 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:00:29.015 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:00:29.015 DEBUG:boto:Method: PUT 2024-04-16T20:00:29.015 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:00:29.015 DEBUG:boto:Data: 2024-04-16T20:00:29.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:00:29.015 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.015 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.015 DEBUG:boto:Params: {} 2024-04-16T20:00:29.016 DEBUG:boto:Token: None 2024-04-16T20:00:29.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/obj76 2024-04-16T20:00:29.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OZXZjQJpyyzkMHcRyQ0UZGZIWcs= 2024-04-16T20:00:29.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OZXZjQJpyyzkMHcRyQ0UZGZIWcs='} 2024-04-16T20:00:29.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fff499f2277f7047-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.059 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:00:29.059 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:00:29.059 DEBUG:boto:Method: HEAD 2024-04-16T20:00:29.059 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:00:29.059 DEBUG:boto:Data: 2024-04-16T20:00:29.059 DEBUG:boto:Headers: {} 2024-04-16T20:00:29.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.059 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.059 DEBUG:boto:Params: {} 2024-04-16T20:00:29.059 DEBUG:boto:Token: None 2024-04-16T20:00:29.059 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/ 2024-04-16T20:00:29.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM= 2024-04-16T20:00:29.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RlTcRXoXn9fF+06u+sHii62BlmM=', 'Content-Length': '0'} 2024-04-16T20:00:29.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fda0b1900b541b90-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.062 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:00:29.062 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:00:29.062 DEBUG:boto:Method: PUT 2024-04-16T20:00:29.063 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:00:29.063 DEBUG:boto:Data: 2024-04-16T20:00:29.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:00:29.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:00:29.063 DEBUG:boto:Port: 8000 2024-04-16T20:00:29.063 DEBUG:boto:Params: {} 2024-04-16T20:00:29.063 DEBUG:boto:Token: None 2024-04-16T20:00:29.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:00:29 GMT /yrjefh-52/obj95 2024-04-16T20:00:29.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ynt4dV0IGv6Frqx1zp+SZPAjayE= 2024-04-16T20:00:29.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:00:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ynt4dV0IGv6Frqx1zp+SZPAjayE='} 2024-04-16T20:00:29.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001aff87cc105b024f-00661ed8dd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:00:29.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:00:29.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:00:29.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:00:29.068 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:00:29.128 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:00:29.128 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:00:29.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:29.190+0000 7fc01412a800 1 waiting to reach latest gen 2, on 1.. 2024-04-16T20:00:34.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:34.193+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:34.190 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-16T20:00:34.190 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:39.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:39.193+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:39.191 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-16T20:00:39.191 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:44.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:44.196+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:44.192 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:44.192 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:49.197+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:49.194 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:49.194 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:54.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:54.199+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:54.195 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:54.195 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:00:59.200+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:00:59.197 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:00:59.197 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:04.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:04.202+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:04.198 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:04.198 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:09.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:09.203+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:09.200 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:09.200 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:14.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:14.205+0000 7fc01412a800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:14.201 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:14.201 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:19.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:19.205+0000 7fc01412a800 1 bucket sync caught up with source: 2024-04-16T20:01:19.203 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:19.203 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.68.6, 00000000004.153.6, , , , , , , , , , , ] 2024-04-16T20:01:19.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:19.205+0000 7fc01412a800 0 bucket checkpoint complete 2024-04-16T20:01:19.228 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:01:19.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:01:19.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:01:19.229 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:01:19.293 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:01:19.293 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:01:19.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:01:19.359+0000 7f5f36d28800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:19.355 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000001.162.6, 00000000001.91.6, , 00000000001.201.6, , , 00000000002.165.6, 00000000004.207.6, 00000000003.272.6] 2024-04-16T20:01:19.355 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000001.162.6, 00000000001.91.6, 00000000001.267.6, 00000000001.201.6, , , 00000000002.165.6, 00000000004.207.6, 00000000003.272.6] 2024-04-16T20:01:21.568 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T20:01:21.572+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T20:01:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:01:24.361+0000 7f5f36d28800 1 bucket sync caught up with source: 2024-04-16T20:01:24.357 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000001.162.6, 00000000001.91.6, 00000000001.267.6, 00000000001.201.6, , , 00000000002.165.6, 00000000004.207.6, 00000000003.272.6] 2024-04-16T20:01:24.357 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000001.162.6, 00000000001.91.6, 00000000001.267.6, 00000000001.201.6, , , 00000000002.165.6, 00000000004.207.6, 00000000003.272.6] 2024-04-16T20:01:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:01:24.361+0000 7f5f36d28800 0 bucket checkpoint complete 2024-04-16T20:01:24.383 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:01:24.383 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.383 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.384 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.384 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:24.384 DEBUG:boto:Data: 2024-04-16T20:01:24.384 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.384 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.384 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.384 DEBUG:boto:Params: {} 2024-04-16T20:01:24.384 DEBUG:boto:Token: None 2024-04-16T20:01:24.385 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.385 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVuo3MXmEzytbCSw4hQx4jv0Mr0= 2024-04-16T20:01:24.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVuo3MXmEzytbCSw4hQx4jv0Mr0=', 'Content-Length': '0'} 2024-04-16T20:01:24.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b056d9f4eaa621e3-00661ed914-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.388 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.388 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.388 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.388 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:24.388 DEBUG:boto:Data: 2024-04-16T20:01:24.388 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.388 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.388 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.388 DEBUG:boto:Params: {} 2024-04-16T20:01:24.389 DEBUG:boto:Token: None 2024-04-16T20:01:24.389 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVuo3MXmEzytbCSw4hQx4jv0Mr0= 2024-04-16T20:01:24.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVuo3MXmEzytbCSw4hQx4jv0Mr0=', 'Content-Length': '0'} 2024-04-16T20:01:24.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e94cbfb9b6662de-00661ed914-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:01:24.392 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.392 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.392 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.392 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.392 DEBUG:boto:Method: GET 2024-04-16T20:01:24.392 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.393 DEBUG:boto:Data: 2024-04-16T20:01:24.393 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.393 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.393 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.393 DEBUG:boto:Params: {} 2024-04-16T20:01:24.393 DEBUG:boto:Token: None 2024-04-16T20:01:24.393 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/?versions 2024-04-16T20:01:24.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70= 2024-04-16T20:01:24.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70=', 'Content-Length': '0'} 2024-04-16T20:01:24.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c8ae7b334cffc3-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.396 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:01:24.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:01:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:01:24.399 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:01:24.399 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.399 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.399 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.399 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.399 DEBUG:boto:Method: GET 2024-04-16T20:01:24.399 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.399 DEBUG:boto:Data: 2024-04-16T20:01:24.399 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.399 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.400 DEBUG:boto:Params: {} 2024-04-16T20:01:24.400 DEBUG:boto:Token: None 2024-04-16T20:01:24.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/?versions 2024-04-16T20:01:24.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70= 2024-04-16T20:01:24.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70=', 'Content-Length': '0'} 2024-04-16T20:01:24.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2a60dbfdd0e2bf-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.403 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:01:24.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:01:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:01:24.405 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.405 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.405 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.405 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.405 DEBUG:boto:Method: GET 2024-04-16T20:01:24.405 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.405 DEBUG:boto:Data: 2024-04-16T20:01:24.405 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.405 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.405 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.405 DEBUG:boto:Params: {} 2024-04-16T20:01:24.406 DEBUG:boto:Token: None 2024-04-16T20:01:24.406 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/?versions 2024-04-16T20:01:24.406 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70= 2024-04-16T20:01:24.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70=', 'Content-Length': '0'} 2024-04-16T20:01:24.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000efa2b3ed8720e39-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.409 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.410 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.410 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.410 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.410 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.410 DEBUG:boto:Method: GET 2024-04-16T20:01:24.410 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:01:24.410 DEBUG:boto:Data: 2024-04-16T20:01:24.410 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.410 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.410 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.411 DEBUG:boto:Params: {} 2024-04-16T20:01:24.411 DEBUG:boto:Token: None 2024-04-16T20:01:24.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/?versions 2024-04-16T20:01:24.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70= 2024-04-16T20:01:24.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9CDVWiOjXDltP4QxUB702hWGc70=', 'Content-Length': '0'} 2024-04-16T20:01:24.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b024dfa2c91bcffc-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.413 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:01:24.414 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.415 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.415 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.415 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.415 DEBUG:boto:Method: GET 2024-04-16T20:01:24.415 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.415 DEBUG:boto:Data: 2024-04-16T20:01:24.415 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.415 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.415 DEBUG:boto:Params: {} 2024-04-16T20:01:24.415 DEBUG:boto:Token: None 2024-04-16T20:01:24.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ= 2024-04-16T20:01:24.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ=', 'Content-Length': '0'} 2024-04-16T20:01:24.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa75e8513bfaaf1a-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.458 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.458 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.459 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.459 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.459 DEBUG:boto:Method: GET 2024-04-16T20:01:24.459 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.459 DEBUG:boto:Data: 2024-04-16T20:01:24.459 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.459 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.459 DEBUG:boto:Params: {} 2024-04-16T20:01:24.459 DEBUG:boto:Token: None 2024-04-16T20:01:24.460 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ= 2024-04-16T20:01:24.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ=', 'Content-Length': '0'} 2024-04-16T20:01:24.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d3e48bd1351527-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.462 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.462 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.462 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.462 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.462 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.462 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.462 DEBUG:boto:Data: 2024-04-16T20:01:24.463 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.463 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.463 DEBUG:boto:Params: {} 2024-04-16T20:01:24.463 DEBUG:boto:Token: None 2024-04-16T20:01:24.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bK7pEkSEtaN6Zy70R1sHhAe8XV8= 2024-04-16T20:01:24.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bK7pEkSEtaN6Zy70R1sHhAe8XV8=', 'Content-Length': '0'} 2024-04-16T20:01:24.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622cf0c860359ec1-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.466 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.466 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.466 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.466 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.466 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.466 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.466 DEBUG:boto:Data: 2024-04-16T20:01:24.466 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.466 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.466 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.467 DEBUG:boto:Params: {} 2024-04-16T20:01:24.467 DEBUG:boto:Token: None 2024-04-16T20:01:24.467 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bK7pEkSEtaN6Zy70R1sHhAe8XV8= 2024-04-16T20:01:24.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bK7pEkSEtaN6Zy70R1sHhAe8XV8=', 'Content-Length': '0'} 2024-04-16T20:01:24.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2de526558703471-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:01:24.469 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.469 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.469 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.469 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.470 DEBUG:boto:Method: GET 2024-04-16T20:01:24.470 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.470 DEBUG:boto:Data: 2024-04-16T20:01:24.470 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.470 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.470 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.470 DEBUG:boto:Params: {} 2024-04-16T20:01:24.470 DEBUG:boto:Token: None 2024-04-16T20:01:24.470 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.470 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ= 2024-04-16T20:01:24.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ=', 'Content-Length': '0'} 2024-04-16T20:01:24.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c95441fe50236c6b-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.473 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.473 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.473 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.473 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.473 DEBUG:boto:Method: GET 2024-04-16T20:01:24.473 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.473 DEBUG:boto:Data: 2024-04-16T20:01:24.474 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.474 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.474 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.474 DEBUG:boto:Params: {} 2024-04-16T20:01:24.474 DEBUG:boto:Token: None 2024-04-16T20:01:24.474 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:01:24.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ= 2024-04-16T20:01:24.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1/FQ3YPaU5TQtZIbQOuZswwWOQ=', 'Content-Length': '0'} 2024-04-16T20:01:24.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac143d009913a157-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.476 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.476 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.476 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.477 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.477 DEBUG:boto:Method: GET 2024-04-16T20:01:24.477 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.477 DEBUG:boto:Data: 2024-04-16T20:01:24.477 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.477 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.477 DEBUG:boto:Params: {} 2024-04-16T20:01:24.477 DEBUG:boto:Token: None 2024-04-16T20:01:24.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa601a226d50aa5-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.480 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.480 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.481 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.481 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.481 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.481 DEBUG:boto:Method: GET 2024-04-16T20:01:24.481 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.481 DEBUG:boto:Data: 2024-04-16T20:01:24.481 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.481 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.481 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.481 DEBUG:boto:Params: {} 2024-04-16T20:01:24.481 DEBUG:boto:Token: None 2024-04-16T20:01:24.481 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.482 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf4436f331f020f-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.484 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.484 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.485 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.485 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.485 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:01:24.485 DEBUG:boto:Data: 2024-04-16T20:01:24.485 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.485 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.485 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.485 DEBUG:boto:Params: {} 2024-04-16T20:01:24.485 DEBUG:boto:Token: None 2024-04-16T20:01:24.485 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11 2024-04-16T20:01:24.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ErYQ8n1rXru8zOXTJObCxAil1VE= 2024-04-16T20:01:24.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ErYQ8n1rXru8zOXTJObCxAil1VE=', 'Content-Length': '0'} 2024-04-16T20:01:24.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8658d3bc4a45845-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.488 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.488 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.488 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.488 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:01:24.488 DEBUG:boto:Data: 2024-04-16T20:01:24.488 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.488 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.488 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.488 DEBUG:boto:Params: {} 2024-04-16T20:01:24.488 DEBUG:boto:Token: None 2024-04-16T20:01:24.489 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11 2024-04-16T20:01:24.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ErYQ8n1rXru8zOXTJObCxAil1VE= 2024-04-16T20:01:24.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ErYQ8n1rXru8zOXTJObCxAil1VE=', 'Content-Length': '0'} 2024-04-16T20:01:24.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff5587aa93294ed-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:01:24.491 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.491 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.491 DEBUG:boto:Method: GET 2024-04-16T20:01:24.491 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:01:24.491 DEBUG:boto:Data: 2024-04-16T20:01:24.491 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.491 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.491 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.492 DEBUG:boto:Params: {} 2024-04-16T20:01:24.492 DEBUG:boto:Token: None 2024-04-16T20:01:24.492 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11 2024-04-16T20:01:24.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lG/JpRbcZbXOt/JBk7cJqK5B6q0= 2024-04-16T20:01:24.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lG/JpRbcZbXOt/JBk7cJqK5B6q0=', 'Content-Length': '0'} 2024-04-16T20:01:24.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c88eb22b01b3e4-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.534 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:01:24.534 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:01:24.535 DEBUG:boto:Method: GET 2024-04-16T20:01:24.535 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:01:24.535 DEBUG:boto:Data: 2024-04-16T20:01:24.535 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.535 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.535 DEBUG:boto:Params: {} 2024-04-16T20:01:24.535 DEBUG:boto:Token: None 2024-04-16T20:01:24.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj11 2024-04-16T20:01:24.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lG/JpRbcZbXOt/JBk7cJqK5B6q0= 2024-04-16T20:01:24.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lG/JpRbcZbXOt/JBk7cJqK5B6q0=', 'Content-Length': '0'} 2024-04-16T20:01:24.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0d2dade62fecba-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.538 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.538 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.538 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.538 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.538 DEBUG:boto:Method: GET 2024-04-16T20:01:24.538 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.538 DEBUG:boto:Data: 2024-04-16T20:01:24.539 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.539 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.539 DEBUG:boto:Params: {} 2024-04-16T20:01:24.539 DEBUG:boto:Token: None 2024-04-16T20:01:24.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4df386b0273e41-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.582 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.583 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.583 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.583 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.583 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.583 DEBUG:boto:Method: GET 2024-04-16T20:01:24.583 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:01:24.583 DEBUG:boto:Data: 2024-04-16T20:01:24.583 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.584 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.584 DEBUG:boto:Params: {} 2024-04-16T20:01:24.584 DEBUG:boto:Token: None 2024-04-16T20:01:24.584 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d4507e689256ef-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.587 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:01:24.587 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.587 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.588 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.588 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.588 DEBUG:boto:Method: GET 2024-04-16T20:01:24.588 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.588 DEBUG:boto:Data: 2024-04-16T20:01:24.588 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.588 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.588 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.588 DEBUG:boto:Params: {} 2024-04-16T20:01:24.588 DEBUG:boto:Token: None 2024-04-16T20:01:24.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk= 2024-04-16T20:01:24.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk=', 'Content-Length': '0'} 2024-04-16T20:01:24.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6bd5da3b9d1c09-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.634 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.634 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.635 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.635 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.635 DEBUG:boto:Method: GET 2024-04-16T20:01:24.635 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.635 DEBUG:boto:Data: 2024-04-16T20:01:24.635 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.635 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.635 DEBUG:boto:Params: {} 2024-04-16T20:01:24.635 DEBUG:boto:Token: None 2024-04-16T20:01:24.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk= 2024-04-16T20:01:24.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk=', 'Content-Length': '0'} 2024-04-16T20:01:24.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52d0630e932aac9-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.638 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.638 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.638 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.638 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.638 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.638 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.638 DEBUG:boto:Data: 2024-04-16T20:01:24.638 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.639 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.639 DEBUG:boto:Params: {} 2024-04-16T20:01:24.639 DEBUG:boto:Token: None 2024-04-16T20:01:24.639 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QKmSOaEe0qmIJLEQamdKEqH3EkA= 2024-04-16T20:01:24.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QKmSOaEe0qmIJLEQamdKEqH3EkA=', 'Content-Length': '0'} 2024-04-16T20:01:24.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e3833947eb6bdc-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.641 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.641 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.641 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.641 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.642 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.642 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.642 DEBUG:boto:Data: 2024-04-16T20:01:24.642 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.642 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.642 DEBUG:boto:Params: {} 2024-04-16T20:01:24.642 DEBUG:boto:Token: None 2024-04-16T20:01:24.642 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QKmSOaEe0qmIJLEQamdKEqH3EkA= 2024-04-16T20:01:24.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QKmSOaEe0qmIJLEQamdKEqH3EkA=', 'Content-Length': '0'} 2024-04-16T20:01:24.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfea8eac00d25569-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:01:24.645 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.645 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.645 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.645 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.645 DEBUG:boto:Method: GET 2024-04-16T20:01:24.645 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.645 DEBUG:boto:Data: 2024-04-16T20:01:24.645 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.646 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.646 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.646 DEBUG:boto:Params: {} 2024-04-16T20:01:24.646 DEBUG:boto:Token: None 2024-04-16T20:01:24.646 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.646 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk= 2024-04-16T20:01:24.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk=', 'Content-Length': '0'} 2024-04-16T20:01:24.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3211d2039b01b8-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.690 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.691 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.691 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.691 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.691 DEBUG:boto:Method: GET 2024-04-16T20:01:24.691 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.691 DEBUG:boto:Data: 2024-04-16T20:01:24.691 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.691 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.691 DEBUG:boto:Params: {} 2024-04-16T20:01:24.691 DEBUG:boto:Token: None 2024-04-16T20:01:24.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:01:24.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk= 2024-04-16T20:01:24.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:or2GBz6eAsVRxSEwbvZCtKUnvKk=', 'Content-Length': '0'} 2024-04-16T20:01:24.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000626f35efdcc0d2f0-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.694 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.694 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.694 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.694 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.694 DEBUG:boto:Method: GET 2024-04-16T20:01:24.694 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.694 DEBUG:boto:Data: 2024-04-16T20:01:24.695 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.695 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.695 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.695 DEBUG:boto:Params: {} 2024-04-16T20:01:24.695 DEBUG:boto:Token: None 2024-04-16T20:01:24.695 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005715f7e36a34722f-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.738 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.739 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.739 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.739 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.739 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.739 DEBUG:boto:Method: GET 2024-04-16T20:01:24.739 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.739 DEBUG:boto:Data: 2024-04-16T20:01:24.739 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.740 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.740 DEBUG:boto:Params: {} 2024-04-16T20:01:24.740 DEBUG:boto:Token: None 2024-04-16T20:01:24.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba2b6a780835092-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.743 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.743 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.743 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.743 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.743 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:01:24.744 DEBUG:boto:Data: 2024-04-16T20:01:24.744 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.744 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.744 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.744 DEBUG:boto:Params: {} 2024-04-16T20:01:24.744 DEBUG:boto:Token: None 2024-04-16T20:01:24.744 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17 2024-04-16T20:01:24.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+rkTRB3Dsrxvmu7zgb/DtFAQBM= 2024-04-16T20:01:24.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+rkTRB3Dsrxvmu7zgb/DtFAQBM=', 'Content-Length': '0'} 2024-04-16T20:01:24.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f900eaf373eefe-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.746 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.747 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.747 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.747 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:01:24.747 DEBUG:boto:Data: 2024-04-16T20:01:24.747 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.747 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.747 DEBUG:boto:Params: {} 2024-04-16T20:01:24.747 DEBUG:boto:Token: None 2024-04-16T20:01:24.747 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17 2024-04-16T20:01:24.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+rkTRB3Dsrxvmu7zgb/DtFAQBM= 2024-04-16T20:01:24.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+rkTRB3Dsrxvmu7zgb/DtFAQBM=', 'Content-Length': '0'} 2024-04-16T20:01:24.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c081d2e91477884d-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:01:24.749 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.750 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.750 DEBUG:boto:Method: GET 2024-04-16T20:01:24.750 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:01:24.750 DEBUG:boto:Data: 2024-04-16T20:01:24.750 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.750 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.750 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.750 DEBUG:boto:Params: {} 2024-04-16T20:01:24.750 DEBUG:boto:Token: None 2024-04-16T20:01:24.750 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17 2024-04-16T20:01:24.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wgsWeY1ot/lmbPXbHYn9It4nuSA= 2024-04-16T20:01:24.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wgsWeY1ot/lmbPXbHYn9It4nuSA=', 'Content-Length': '0'} 2024-04-16T20:01:24.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c26d9746482cde2-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.794 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:01:24.794 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:01:24.795 DEBUG:boto:Method: GET 2024-04-16T20:01:24.795 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:01:24.795 DEBUG:boto:Data: 2024-04-16T20:01:24.795 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.795 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.795 DEBUG:boto:Params: {} 2024-04-16T20:01:24.795 DEBUG:boto:Token: None 2024-04-16T20:01:24.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj17 2024-04-16T20:01:24.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wgsWeY1ot/lmbPXbHYn9It4nuSA= 2024-04-16T20:01:24.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wgsWeY1ot/lmbPXbHYn9It4nuSA=', 'Content-Length': '0'} 2024-04-16T20:01:24.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005435ce01dff9ef8d-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.798 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.798 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.798 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.798 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.798 DEBUG:boto:Method: GET 2024-04-16T20:01:24.798 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.798 DEBUG:boto:Data: 2024-04-16T20:01:24.798 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.799 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.799 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.799 DEBUG:boto:Params: {} 2024-04-16T20:01:24.799 DEBUG:boto:Token: None 2024-04-16T20:01:24.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7568c4828f9670-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.802 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.802 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.803 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.803 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.803 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.803 DEBUG:boto:Method: GET 2024-04-16T20:01:24.803 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:01:24.803 DEBUG:boto:Data: 2024-04-16T20:01:24.803 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.803 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.804 DEBUG:boto:Params: {} 2024-04-16T20:01:24.804 DEBUG:boto:Token: None 2024-04-16T20:01:24.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051cfd8e28211f37c-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.807 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:01:24.807 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.807 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.807 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.807 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.807 DEBUG:boto:Method: GET 2024-04-16T20:01:24.808 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.808 DEBUG:boto:Data: 2024-04-16T20:01:24.808 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.808 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.808 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.808 DEBUG:boto:Params: {} 2024-04-16T20:01:24.808 DEBUG:boto:Token: None 2024-04-16T20:01:24.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE= 2024-04-16T20:01:24.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE=', 'Content-Length': '0'} 2024-04-16T20:01:24.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df0316d90b995c9d-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.854 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.855 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.855 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.855 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.855 DEBUG:boto:Method: GET 2024-04-16T20:01:24.855 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.855 DEBUG:boto:Data: 2024-04-16T20:01:24.855 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.855 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.855 DEBUG:boto:Params: {} 2024-04-16T20:01:24.855 DEBUG:boto:Token: None 2024-04-16T20:01:24.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE= 2024-04-16T20:01:24.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE=', 'Content-Length': '0'} 2024-04-16T20:01:24.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e5467b7158f1ed-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.858 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.859 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.859 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.859 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.859 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.859 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.859 DEBUG:boto:Data: 2024-04-16T20:01:24.859 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.859 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.859 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.859 DEBUG:boto:Params: {} 2024-04-16T20:01:24.859 DEBUG:boto:Token: None 2024-04-16T20:01:24.860 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ju9gtBNpRw67Z+V4Pa5ueOyegAI= 2024-04-16T20:01:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ju9gtBNpRw67Z+V4Pa5ueOyegAI=', 'Content-Length': '0'} 2024-04-16T20:01:24.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4a06864cc01b7e-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.862 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.862 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.862 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.862 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.862 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.863 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.863 DEBUG:boto:Data: 2024-04-16T20:01:24.863 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.863 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.863 DEBUG:boto:Params: {} 2024-04-16T20:01:24.863 DEBUG:boto:Token: None 2024-04-16T20:01:24.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ju9gtBNpRw67Z+V4Pa5ueOyegAI= 2024-04-16T20:01:24.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ju9gtBNpRw67Z+V4Pa5ueOyegAI=', 'Content-Length': '0'} 2024-04-16T20:01:24.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba7d758a1b891cf-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:01:24.866 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.866 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.866 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.866 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.866 DEBUG:boto:Method: GET 2024-04-16T20:01:24.866 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.866 DEBUG:boto:Data: 2024-04-16T20:01:24.866 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.866 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.867 DEBUG:boto:Params: {} 2024-04-16T20:01:24.867 DEBUG:boto:Token: None 2024-04-16T20:01:24.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE= 2024-04-16T20:01:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE=', 'Content-Length': '0'} 2024-04-16T20:01:24.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba402fa8af0c290-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.910 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.910 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.911 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.911 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.911 DEBUG:boto:Method: GET 2024-04-16T20:01:24.911 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.911 DEBUG:boto:Data: 2024-04-16T20:01:24.911 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.911 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.911 DEBUG:boto:Params: {} 2024-04-16T20:01:24.911 DEBUG:boto:Token: None 2024-04-16T20:01:24.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:01:24.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE= 2024-04-16T20:01:24.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kOuFyAAn3tncl3w/M6R+s8W/IE=', 'Content-Length': '0'} 2024-04-16T20:01:24.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828ea9223dcb9b49-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.914 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.914 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.914 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.915 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.915 DEBUG:boto:Method: GET 2024-04-16T20:01:24.915 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.915 DEBUG:boto:Data: 2024-04-16T20:01:24.915 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.915 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.915 DEBUG:boto:Params: {} 2024-04-16T20:01:24.915 DEBUG:boto:Token: None 2024-04-16T20:01:24.915 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0af870064d50a11-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.918 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.918 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.918 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.919 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.919 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.919 DEBUG:boto:Method: GET 2024-04-16T20:01:24.919 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.919 DEBUG:boto:Data: 2024-04-16T20:01:24.919 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.919 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.919 DEBUG:boto:Params: {} 2024-04-16T20:01:24.919 DEBUG:boto:Token: None 2024-04-16T20:01:24.919 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b1d5c25dcd152e-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.923 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.923 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.923 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.923 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.923 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:01:24.923 DEBUG:boto:Data: 2024-04-16T20:01:24.923 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.924 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.924 DEBUG:boto:Params: {} 2024-04-16T20:01:24.924 DEBUG:boto:Token: None 2024-04-16T20:01:24.924 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19 2024-04-16T20:01:24.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATLldj/oy0VfXIs+v7wAzmXFU4= 2024-04-16T20:01:24.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATLldj/oy0VfXIs+v7wAzmXFU4=', 'Content-Length': '0'} 2024-04-16T20:01:24.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad84b55d112d3fc-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.926 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.926 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.926 DEBUG:boto:Method: HEAD 2024-04-16T20:01:24.927 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:01:24.927 DEBUG:boto:Data: 2024-04-16T20:01:24.927 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.927 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.927 DEBUG:boto:Params: {} 2024-04-16T20:01:24.927 DEBUG:boto:Token: None 2024-04-16T20:01:24.927 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19 2024-04-16T20:01:24.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATLldj/oy0VfXIs+v7wAzmXFU4= 2024-04-16T20:01:24.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATLldj/oy0VfXIs+v7wAzmXFU4=', 'Content-Length': '0'} 2024-04-16T20:01:24.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000546de20e2271df3e-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:01:24.930 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.930 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.930 DEBUG:boto:Method: GET 2024-04-16T20:01:24.930 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:01:24.930 DEBUG:boto:Data: 2024-04-16T20:01:24.930 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.930 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.931 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.931 DEBUG:boto:Params: {} 2024-04-16T20:01:24.931 DEBUG:boto:Token: None 2024-04-16T20:01:24.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19 2024-04-16T20:01:24.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J026WXJ6KgNDg/cdbE7mDdPIjh0= 2024-04-16T20:01:24.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J026WXJ6KgNDg/cdbE7mDdPIjh0=', 'Content-Length': '0'} 2024-04-16T20:01:24.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7ac7fd99d58bc8-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.974 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:01:24.975 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:01:24.975 DEBUG:boto:Method: GET 2024-04-16T20:01:24.975 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:01:24.975 DEBUG:boto:Data: 2024-04-16T20:01:24.975 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.975 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.975 DEBUG:boto:Params: {} 2024-04-16T20:01:24.975 DEBUG:boto:Token: None 2024-04-16T20:01:24.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj19 2024-04-16T20:01:24.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J026WXJ6KgNDg/cdbE7mDdPIjh0= 2024-04-16T20:01:24.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J026WXJ6KgNDg/cdbE7mDdPIjh0=', 'Content-Length': '0'} 2024-04-16T20:01:24.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ccad05d5b626d2-00661ed914-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.978 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.978 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.978 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.978 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.979 DEBUG:boto:Method: GET 2024-04-16T20:01:24.979 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.979 DEBUG:boto:Data: 2024-04-16T20:01:24.979 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.979 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.979 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.979 DEBUG:boto:Params: {} 2024-04-16T20:01:24.979 DEBUG:boto:Token: None 2024-04-16T20:01:24.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc6a27d6093f037-00661ed914-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.982 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.982 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:24.982 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:24.983 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.983 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.983 DEBUG:boto:Method: GET 2024-04-16T20:01:24.983 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:01:24.983 DEBUG:boto:Data: 2024-04-16T20:01:24.983 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:24.983 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.983 DEBUG:boto:Params: {} 2024-04-16T20:01:24.983 DEBUG:boto:Token: None 2024-04-16T20:01:24.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/ 2024-04-16T20:01:24.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc= 2024-04-16T20:01:24.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjFsoWp9JNVw1XN+6EnwP1bFinc=', 'Content-Length': '0'} 2024-04-16T20:01:24.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c0721772067851-00661ed914-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:24.987 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:24.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:01:24.987 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:24.987 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:24.987 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:24.987 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:24.987 DEBUG:boto:Method: GET 2024-04-16T20:01:24.987 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:24.988 DEBUG:boto:Data: 2024-04-16T20:01:24.988 DEBUG:boto:Headers: {} 2024-04-16T20:01:24.988 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:24.988 DEBUG:boto:Port: 8000 2024-04-16T20:01:24.988 DEBUG:boto:Params: {} 2024-04-16T20:01:24.988 DEBUG:boto:Token: None 2024-04-16T20:01:24.988 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:24 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:24.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IVIWkqhb5ziGjpBHqXMxf1MYIs8= 2024-04-16T20:01:24.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:24 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVIWkqhb5ziGjpBHqXMxf1MYIs8=', 'Content-Length': '0'} 2024-04-16T20:01:24.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001abb302583c07f0-00661ed914-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.030 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.030 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.031 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.031 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.031 DEBUG:boto:Method: GET 2024-04-16T20:01:25.031 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.031 DEBUG:boto:Data: 2024-04-16T20:01:25.031 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.031 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.031 DEBUG:boto:Params: {} 2024-04-16T20:01:25.031 DEBUG:boto:Token: None 2024-04-16T20:01:25.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U= 2024-04-16T20:01:25.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U=', 'Content-Length': '0'} 2024-04-16T20:01:25.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b8dabd3f5e3926-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.034 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.034 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.034 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.034 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.034 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.034 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.034 DEBUG:boto:Data: 2024-04-16T20:01:25.034 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.035 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.035 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.035 DEBUG:boto:Params: {} 2024-04-16T20:01:25.035 DEBUG:boto:Token: None 2024-04-16T20:01:25.035 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ae4v7AH/ivE1caeo1qD5RnB5GXw= 2024-04-16T20:01:25.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ae4v7AH/ivE1caeo1qD5RnB5GXw=', 'Content-Length': '0'} 2024-04-16T20:01:25.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4f945e3deeb37c-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.037 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.037 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.037 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.037 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.037 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.037 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.037 DEBUG:boto:Data: 2024-04-16T20:01:25.038 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.038 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.038 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.038 DEBUG:boto:Params: {} 2024-04-16T20:01:25.038 DEBUG:boto:Token: None 2024-04-16T20:01:25.038 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ae4v7AH/ivE1caeo1qD5RnB5GXw= 2024-04-16T20:01:25.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ae4v7AH/ivE1caeo1qD5RnB5GXw=', 'Content-Length': '0'} 2024-04-16T20:01:25.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3d609dfc307ff7-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:01:25.040 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.040 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.041 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.041 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.041 DEBUG:boto:Method: GET 2024-04-16T20:01:25.041 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.041 DEBUG:boto:Data: 2024-04-16T20:01:25.041 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.041 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.041 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.041 DEBUG:boto:Params: {} 2024-04-16T20:01:25.041 DEBUG:boto:Token: None 2024-04-16T20:01:25.041 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U= 2024-04-16T20:01:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U=', 'Content-Length': '0'} 2024-04-16T20:01:25.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7357b16d37522c3-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.086 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.086 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.087 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.087 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.087 DEBUG:boto:Method: GET 2024-04-16T20:01:25.087 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.087 DEBUG:boto:Data: 2024-04-16T20:01:25.087 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.087 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.088 DEBUG:boto:Params: {} 2024-04-16T20:01:25.088 DEBUG:boto:Token: None 2024-04-16T20:01:25.088 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:01:25.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U= 2024-04-16T20:01:25.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WTu7TDEt0pH2Qc60Y7yHfULKS6U=', 'Content-Length': '0'} 2024-04-16T20:01:25.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd051dcf01befe08-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.091 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.091 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.091 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.091 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.091 DEBUG:boto:Method: GET 2024-04-16T20:01:25.091 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.091 DEBUG:boto:Data: 2024-04-16T20:01:25.091 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.091 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.091 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.091 DEBUG:boto:Params: {} 2024-04-16T20:01:25.092 DEBUG:boto:Token: None 2024-04-16T20:01:25.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008134ada14e15e39c-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.095 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.095 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.095 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.095 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.095 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.095 DEBUG:boto:Method: GET 2024-04-16T20:01:25.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.096 DEBUG:boto:Data: 2024-04-16T20:01:25.096 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.096 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.096 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.096 DEBUG:boto:Params: {} 2024-04-16T20:01:25.096 DEBUG:boto:Token: None 2024-04-16T20:01:25.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005434bfa45551f005-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.099 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.099 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.099 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.099 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.099 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:01:25.099 DEBUG:boto:Data: 2024-04-16T20:01:25.100 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.100 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.100 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.100 DEBUG:boto:Params: {} 2024-04-16T20:01:25.100 DEBUG:boto:Token: None 2024-04-16T20:01:25.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22 2024-04-16T20:01:25.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lD56Y6uSrK/2ibp19FiZXnV8rz0= 2024-04-16T20:01:25.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lD56Y6uSrK/2ibp19FiZXnV8rz0=', 'Content-Length': '0'} 2024-04-16T20:01:25.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef6e9d08d0753e4-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.102 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.102 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.102 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.103 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:01:25.103 DEBUG:boto:Data: 2024-04-16T20:01:25.103 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.103 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.103 DEBUG:boto:Params: {} 2024-04-16T20:01:25.103 DEBUG:boto:Token: None 2024-04-16T20:01:25.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22 2024-04-16T20:01:25.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lD56Y6uSrK/2ibp19FiZXnV8rz0= 2024-04-16T20:01:25.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lD56Y6uSrK/2ibp19FiZXnV8rz0=', 'Content-Length': '0'} 2024-04-16T20:01:25.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93631e34535a806-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:01:25.105 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.105 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.106 DEBUG:boto:Method: GET 2024-04-16T20:01:25.106 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:01:25.106 DEBUG:boto:Data: 2024-04-16T20:01:25.106 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.106 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.106 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.106 DEBUG:boto:Params: {} 2024-04-16T20:01:25.106 DEBUG:boto:Token: None 2024-04-16T20:01:25.106 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22 2024-04-16T20:01:25.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LT/usMYD3XuluDEJ3SKwud1oFWg= 2024-04-16T20:01:25.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LT/usMYD3XuluDEJ3SKwud1oFWg=', 'Content-Length': '0'} 2024-04-16T20:01:25.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ca4f5a2b185f19-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.150 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:01:25.150 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:01:25.151 DEBUG:boto:Method: GET 2024-04-16T20:01:25.151 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:01:25.151 DEBUG:boto:Data: 2024-04-16T20:01:25.151 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.151 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.151 DEBUG:boto:Params: {} 2024-04-16T20:01:25.151 DEBUG:boto:Token: None 2024-04-16T20:01:25.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj22 2024-04-16T20:01:25.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LT/usMYD3XuluDEJ3SKwud1oFWg= 2024-04-16T20:01:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LT/usMYD3XuluDEJ3SKwud1oFWg=', 'Content-Length': '0'} 2024-04-16T20:01:25.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a54bd2f798a1af-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.154 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.154 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.154 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.154 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.154 DEBUG:boto:Method: GET 2024-04-16T20:01:25.154 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.154 DEBUG:boto:Data: 2024-04-16T20:01:25.154 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.154 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.154 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.154 DEBUG:boto:Params: {} 2024-04-16T20:01:25.155 DEBUG:boto:Token: None 2024-04-16T20:01:25.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc72c72335178940-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.198 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.198 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.199 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.199 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.199 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.199 DEBUG:boto:Method: GET 2024-04-16T20:01:25.199 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:01:25.199 DEBUG:boto:Data: 2024-04-16T20:01:25.199 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.199 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.199 DEBUG:boto:Params: {} 2024-04-16T20:01:25.200 DEBUG:boto:Token: None 2024-04-16T20:01:25.200 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006207cf4dd55841b2-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.202 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:01:25.203 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.203 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.203 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.203 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.203 DEBUG:boto:Method: GET 2024-04-16T20:01:25.203 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.204 DEBUG:boto:Data: 2024-04-16T20:01:25.204 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.204 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.204 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.204 DEBUG:boto:Params: {} 2024-04-16T20:01:25.204 DEBUG:boto:Token: None 2024-04-16T20:01:25.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA= 2024-04-16T20:01:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA=', 'Content-Length': '0'} 2024-04-16T20:01:25.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000498b96d7b08e884-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.250 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.251 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.251 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.251 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.251 DEBUG:boto:Method: GET 2024-04-16T20:01:25.251 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.251 DEBUG:boto:Data: 2024-04-16T20:01:25.251 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.251 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.251 DEBUG:boto:Params: {} 2024-04-16T20:01:25.252 DEBUG:boto:Token: None 2024-04-16T20:01:25.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA= 2024-04-16T20:01:25.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA=', 'Content-Length': '0'} 2024-04-16T20:01:25.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d883553eca994fc9-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.254 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.255 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.255 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.255 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.255 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.255 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.255 DEBUG:boto:Data: 2024-04-16T20:01:25.255 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.255 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.255 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.255 DEBUG:boto:Params: {} 2024-04-16T20:01:25.255 DEBUG:boto:Token: None 2024-04-16T20:01:25.256 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3fGvBdMKmbcn4aLRv+x5PdhWUQ= 2024-04-16T20:01:25.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3fGvBdMKmbcn4aLRv+x5PdhWUQ=', 'Content-Length': '0'} 2024-04-16T20:01:25.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6f153dadfdaf79-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.258 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.258 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.258 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.258 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.259 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.259 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.259 DEBUG:boto:Data: 2024-04-16T20:01:25.259 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.259 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.259 DEBUG:boto:Params: {} 2024-04-16T20:01:25.259 DEBUG:boto:Token: None 2024-04-16T20:01:25.259 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u3fGvBdMKmbcn4aLRv+x5PdhWUQ= 2024-04-16T20:01:25.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u3fGvBdMKmbcn4aLRv+x5PdhWUQ=', 'Content-Length': '0'} 2024-04-16T20:01:25.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dff3378431df7f0-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:01:25.262 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.262 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.262 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.262 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.262 DEBUG:boto:Method: GET 2024-04-16T20:01:25.263 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.263 DEBUG:boto:Data: 2024-04-16T20:01:25.263 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.263 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.263 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.263 DEBUG:boto:Params: {} 2024-04-16T20:01:25.263 DEBUG:boto:Token: None 2024-04-16T20:01:25.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA= 2024-04-16T20:01:25.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA=', 'Content-Length': '0'} 2024-04-16T20:01:25.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ba6c63c35c92f2-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.306 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.306 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.307 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.307 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.307 DEBUG:boto:Method: GET 2024-04-16T20:01:25.307 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.307 DEBUG:boto:Data: 2024-04-16T20:01:25.307 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.307 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.307 DEBUG:boto:Params: {} 2024-04-16T20:01:25.307 DEBUG:boto:Token: None 2024-04-16T20:01:25.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:01:25.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA= 2024-04-16T20:01:25.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n96E3NtZpMj2HxKUW1pvSEWz1AA=', 'Content-Length': '0'} 2024-04-16T20:01:25.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1fdd4ff976ab1e-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.310 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.310 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.310 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.310 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.310 DEBUG:boto:Method: GET 2024-04-16T20:01:25.311 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.311 DEBUG:boto:Data: 2024-04-16T20:01:25.311 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.311 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.311 DEBUG:boto:Params: {} 2024-04-16T20:01:25.311 DEBUG:boto:Token: None 2024-04-16T20:01:25.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007253b6c362ba9c86-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.354 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.355 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.355 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.355 DEBUG:boto:Method: GET 2024-04-16T20:01:25.355 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.355 DEBUG:boto:Data: 2024-04-16T20:01:25.355 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.355 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.355 DEBUG:boto:Params: {} 2024-04-16T20:01:25.356 DEBUG:boto:Token: None 2024-04-16T20:01:25.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071dca010cfb0b4f3-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.359 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.359 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.359 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.359 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.359 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:01:25.359 DEBUG:boto:Data: 2024-04-16T20:01:25.359 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.360 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.360 DEBUG:boto:Params: {} 2024-04-16T20:01:25.360 DEBUG:boto:Token: None 2024-04-16T20:01:25.360 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33 2024-04-16T20:01:25.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qb/M4xts2OvyIbtPSQW7NeQ0XUc= 2024-04-16T20:01:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qb/M4xts2OvyIbtPSQW7NeQ0XUc=', 'Content-Length': '0'} 2024-04-16T20:01:25.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c297fc8a062cf5fb-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.362 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.362 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.362 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.363 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:01:25.363 DEBUG:boto:Data: 2024-04-16T20:01:25.363 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.363 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.363 DEBUG:boto:Params: {} 2024-04-16T20:01:25.363 DEBUG:boto:Token: None 2024-04-16T20:01:25.363 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33 2024-04-16T20:01:25.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qb/M4xts2OvyIbtPSQW7NeQ0XUc= 2024-04-16T20:01:25.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qb/M4xts2OvyIbtPSQW7NeQ0XUc=', 'Content-Length': '0'} 2024-04-16T20:01:25.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3b2e9781f333de9-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:01:25.366 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.366 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.366 DEBUG:boto:Method: GET 2024-04-16T20:01:25.366 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:01:25.366 DEBUG:boto:Data: 2024-04-16T20:01:25.366 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.366 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.367 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.367 DEBUG:boto:Params: {} 2024-04-16T20:01:25.367 DEBUG:boto:Token: None 2024-04-16T20:01:25.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33 2024-04-16T20:01:25.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6gFBJSdBf1jlCrH0SKoHmVexnl4= 2024-04-16T20:01:25.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6gFBJSdBf1jlCrH0SKoHmVexnl4=', 'Content-Length': '0'} 2024-04-16T20:01:25.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b4d327fc55f88a-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.410 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:01:25.410 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:01:25.411 DEBUG:boto:Method: GET 2024-04-16T20:01:25.411 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:01:25.411 DEBUG:boto:Data: 2024-04-16T20:01:25.411 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.411 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.411 DEBUG:boto:Params: {} 2024-04-16T20:01:25.411 DEBUG:boto:Token: None 2024-04-16T20:01:25.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj33 2024-04-16T20:01:25.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6gFBJSdBf1jlCrH0SKoHmVexnl4= 2024-04-16T20:01:25.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6gFBJSdBf1jlCrH0SKoHmVexnl4=', 'Content-Length': '0'} 2024-04-16T20:01:25.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063bbd7797391242a-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.414 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.414 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.414 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.415 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.415 DEBUG:boto:Method: GET 2024-04-16T20:01:25.415 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.415 DEBUG:boto:Data: 2024-04-16T20:01:25.415 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.415 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.415 DEBUG:boto:Params: {} 2024-04-16T20:01:25.415 DEBUG:boto:Token: None 2024-04-16T20:01:25.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc16dd78b20cf00-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.418 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.418 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.419 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.419 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.419 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.419 DEBUG:boto:Method: GET 2024-04-16T20:01:25.419 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:01:25.419 DEBUG:boto:Data: 2024-04-16T20:01:25.419 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.419 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.419 DEBUG:boto:Params: {} 2024-04-16T20:01:25.419 DEBUG:boto:Token: None 2024-04-16T20:01:25.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f23b1c8247100f-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.422 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:01:25.423 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.423 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.423 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.423 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.423 DEBUG:boto:Method: GET 2024-04-16T20:01:25.423 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.423 DEBUG:boto:Data: 2024-04-16T20:01:25.423 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.424 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.424 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.424 DEBUG:boto:Params: {} 2024-04-16T20:01:25.424 DEBUG:boto:Token: None 2024-04-16T20:01:25.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw= 2024-04-16T20:01:25.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw=', 'Content-Length': '0'} 2024-04-16T20:01:25.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50444f2e286bc0c-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.466 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.466 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.467 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.467 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.467 DEBUG:boto:Method: GET 2024-04-16T20:01:25.467 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.467 DEBUG:boto:Data: 2024-04-16T20:01:25.467 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.467 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.467 DEBUG:boto:Params: {} 2024-04-16T20:01:25.467 DEBUG:boto:Token: None 2024-04-16T20:01:25.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw= 2024-04-16T20:01:25.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw=', 'Content-Length': '0'} 2024-04-16T20:01:25.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3de2665ebea73e-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.470 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.470 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.470 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.470 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.470 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.470 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.470 DEBUG:boto:Data: 2024-04-16T20:01:25.470 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.470 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.471 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.471 DEBUG:boto:Params: {} 2024-04-16T20:01:25.471 DEBUG:boto:Token: None 2024-04-16T20:01:25.471 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jaZFEVLXBMwHy4XLuGWJepYDcJs= 2024-04-16T20:01:25.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jaZFEVLXBMwHy4XLuGWJepYDcJs=', 'Content-Length': '0'} 2024-04-16T20:01:25.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c775247e198c6380-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.473 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.473 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.473 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.473 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.473 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.473 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.473 DEBUG:boto:Data: 2024-04-16T20:01:25.474 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.474 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.474 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.474 DEBUG:boto:Params: {} 2024-04-16T20:01:25.474 DEBUG:boto:Token: None 2024-04-16T20:01:25.474 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.474 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jaZFEVLXBMwHy4XLuGWJepYDcJs= 2024-04-16T20:01:25.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jaZFEVLXBMwHy4XLuGWJepYDcJs=', 'Content-Length': '0'} 2024-04-16T20:01:25.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095e4c9076705e87-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:01:25.476 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.476 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.476 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.476 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.477 DEBUG:boto:Method: GET 2024-04-16T20:01:25.477 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.477 DEBUG:boto:Data: 2024-04-16T20:01:25.477 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.477 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.477 DEBUG:boto:Params: {} 2024-04-16T20:01:25.477 DEBUG:boto:Token: None 2024-04-16T20:01:25.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw= 2024-04-16T20:01:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw=', 'Content-Length': '0'} 2024-04-16T20:01:25.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000955a6d89f233aca1-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.522 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.522 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.523 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.523 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.523 DEBUG:boto:Method: GET 2024-04-16T20:01:25.523 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.523 DEBUG:boto:Data: 2024-04-16T20:01:25.523 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.523 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.523 DEBUG:boto:Params: {} 2024-04-16T20:01:25.523 DEBUG:boto:Token: None 2024-04-16T20:01:25.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:01:25.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw= 2024-04-16T20:01:25.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3yaZjQCKx1hQloos31lNiPh9jw=', 'Content-Length': '0'} 2024-04-16T20:01:25.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15c481efd977913-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.526 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.526 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.526 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.526 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.526 DEBUG:boto:Method: GET 2024-04-16T20:01:25.526 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.526 DEBUG:boto:Data: 2024-04-16T20:01:25.526 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.527 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.527 DEBUG:boto:Params: {} 2024-04-16T20:01:25.527 DEBUG:boto:Token: None 2024-04-16T20:01:25.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ecec7d5b809adb4-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.530 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.530 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.530 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.530 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.530 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.530 DEBUG:boto:Method: GET 2024-04-16T20:01:25.531 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.531 DEBUG:boto:Data: 2024-04-16T20:01:25.531 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.531 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.531 DEBUG:boto:Params: {} 2024-04-16T20:01:25.531 DEBUG:boto:Token: None 2024-04-16T20:01:25.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a601d6056d0f658f-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.534 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.534 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.534 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.534 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.534 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:01:25.535 DEBUG:boto:Data: 2024-04-16T20:01:25.535 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.535 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.535 DEBUG:boto:Params: {} 2024-04-16T20:01:25.535 DEBUG:boto:Token: None 2024-04-16T20:01:25.535 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34 2024-04-16T20:01:25.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hOETycH4pLvD263HK/m9tJdbjAw= 2024-04-16T20:01:25.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hOETycH4pLvD263HK/m9tJdbjAw=', 'Content-Length': '0'} 2024-04-16T20:01:25.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e0325e3898b71b-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.537 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.538 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.538 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.538 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:01:25.538 DEBUG:boto:Data: 2024-04-16T20:01:25.538 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.538 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.538 DEBUG:boto:Params: {} 2024-04-16T20:01:25.538 DEBUG:boto:Token: None 2024-04-16T20:01:25.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34 2024-04-16T20:01:25.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hOETycH4pLvD263HK/m9tJdbjAw= 2024-04-16T20:01:25.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hOETycH4pLvD263HK/m9tJdbjAw=', 'Content-Length': '0'} 2024-04-16T20:01:25.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a895acfc52cbaef4-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:01:25.541 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.541 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.541 DEBUG:boto:Method: GET 2024-04-16T20:01:25.541 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:01:25.541 DEBUG:boto:Data: 2024-04-16T20:01:25.541 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.541 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.541 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.542 DEBUG:boto:Params: {} 2024-04-16T20:01:25.542 DEBUG:boto:Token: None 2024-04-16T20:01:25.542 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34 2024-04-16T20:01:25.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MReLxihrNsa5vcxrwBCaxY56e8o= 2024-04-16T20:01:25.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MReLxihrNsa5vcxrwBCaxY56e8o=', 'Content-Length': '0'} 2024-04-16T20:01:25.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da3bf246c251cddd-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.586 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:01:25.586 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:01:25.587 DEBUG:boto:Method: GET 2024-04-16T20:01:25.587 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:01:25.587 DEBUG:boto:Data: 2024-04-16T20:01:25.587 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.587 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.587 DEBUG:boto:Params: {} 2024-04-16T20:01:25.587 DEBUG:boto:Token: None 2024-04-16T20:01:25.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj34 2024-04-16T20:01:25.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MReLxihrNsa5vcxrwBCaxY56e8o= 2024-04-16T20:01:25.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MReLxihrNsa5vcxrwBCaxY56e8o=', 'Content-Length': '0'} 2024-04-16T20:01:25.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d78244adf4d051d7-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.589 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.589 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.589 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.590 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.590 DEBUG:boto:Method: GET 2024-04-16T20:01:25.590 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.590 DEBUG:boto:Data: 2024-04-16T20:01:25.590 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.590 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.590 DEBUG:boto:Params: {} 2024-04-16T20:01:25.590 DEBUG:boto:Token: None 2024-04-16T20:01:25.590 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf70ddc7382055ca-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.593 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.594 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.594 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.594 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.594 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.594 DEBUG:boto:Method: GET 2024-04-16T20:01:25.594 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:01:25.594 DEBUG:boto:Data: 2024-04-16T20:01:25.594 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.594 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.594 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.594 DEBUG:boto:Params: {} 2024-04-16T20:01:25.595 DEBUG:boto:Token: None 2024-04-16T20:01:25.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3e9f5d01a7faa2-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.598 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:01:25.598 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.598 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.598 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.599 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.599 DEBUG:boto:Method: GET 2024-04-16T20:01:25.599 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.599 DEBUG:boto:Data: 2024-04-16T20:01:25.599 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.599 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.599 DEBUG:boto:Params: {} 2024-04-16T20:01:25.599 DEBUG:boto:Token: None 2024-04-16T20:01:25.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4= 2024-04-16T20:01:25.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4=', 'Content-Length': '0'} 2024-04-16T20:01:25.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009785e0c24eb8e6b0-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.642 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.642 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.642 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.643 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.643 DEBUG:boto:Method: GET 2024-04-16T20:01:25.643 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.643 DEBUG:boto:Data: 2024-04-16T20:01:25.643 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.643 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.643 DEBUG:boto:Params: {} 2024-04-16T20:01:25.643 DEBUG:boto:Token: None 2024-04-16T20:01:25.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4= 2024-04-16T20:01:25.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4=', 'Content-Length': '0'} 2024-04-16T20:01:25.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe9e47121822da28-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.646 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.646 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.646 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.646 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.646 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.646 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.646 DEBUG:boto:Data: 2024-04-16T20:01:25.646 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.646 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.646 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.647 DEBUG:boto:Params: {} 2024-04-16T20:01:25.647 DEBUG:boto:Token: None 2024-04-16T20:01:25.647 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gq7hwwiDNULSsIqEirGgrvEGdXI= 2024-04-16T20:01:25.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gq7hwwiDNULSsIqEirGgrvEGdXI=', 'Content-Length': '0'} 2024-04-16T20:01:25.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027924a32cc41f5fb-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.649 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.649 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.649 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.649 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.649 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.649 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.650 DEBUG:boto:Data: 2024-04-16T20:01:25.650 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.650 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.650 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.650 DEBUG:boto:Params: {} 2024-04-16T20:01:25.650 DEBUG:boto:Token: None 2024-04-16T20:01:25.650 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.650 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gq7hwwiDNULSsIqEirGgrvEGdXI= 2024-04-16T20:01:25.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gq7hwwiDNULSsIqEirGgrvEGdXI=', 'Content-Length': '0'} 2024-04-16T20:01:25.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009711586d0b45bd01-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:01:25.652 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.653 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.653 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.653 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.653 DEBUG:boto:Method: GET 2024-04-16T20:01:25.653 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.653 DEBUG:boto:Data: 2024-04-16T20:01:25.653 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.653 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.653 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.653 DEBUG:boto:Params: {} 2024-04-16T20:01:25.653 DEBUG:boto:Token: None 2024-04-16T20:01:25.654 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.654 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4= 2024-04-16T20:01:25.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4=', 'Content-Length': '0'} 2024-04-16T20:01:25.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd3436e86935738-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.698 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.698 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.698 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.698 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.699 DEBUG:boto:Method: GET 2024-04-16T20:01:25.699 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.699 DEBUG:boto:Data: 2024-04-16T20:01:25.699 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.699 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.699 DEBUG:boto:Params: {} 2024-04-16T20:01:25.699 DEBUG:boto:Token: None 2024-04-16T20:01:25.699 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:01:25.699 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4= 2024-04-16T20:01:25.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EpjDPDvca/06AOE+GGmSvZZ4lF4=', 'Content-Length': '0'} 2024-04-16T20:01:25.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028a26dc6dd213d8-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.702 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.702 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.702 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.702 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.702 DEBUG:boto:Method: GET 2024-04-16T20:01:25.702 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.702 DEBUG:boto:Data: 2024-04-16T20:01:25.702 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.702 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.702 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.703 DEBUG:boto:Params: {} 2024-04-16T20:01:25.703 DEBUG:boto:Token: None 2024-04-16T20:01:25.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031a22779d3138d79-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.705 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.706 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.706 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.706 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.706 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.706 DEBUG:boto:Method: GET 2024-04-16T20:01:25.706 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.706 DEBUG:boto:Data: 2024-04-16T20:01:25.706 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.706 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.706 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.706 DEBUG:boto:Params: {} 2024-04-16T20:01:25.707 DEBUG:boto:Token: None 2024-04-16T20:01:25.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b92c3a5f2a2652-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.710 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.710 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.710 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.710 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.710 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:01:25.710 DEBUG:boto:Data: 2024-04-16T20:01:25.711 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.711 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.711 DEBUG:boto:Params: {} 2024-04-16T20:01:25.711 DEBUG:boto:Token: None 2024-04-16T20:01:25.711 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38 2024-04-16T20:01:25.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o7/yoW+kuVPo3DdYHuqdif06EDc= 2024-04-16T20:01:25.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o7/yoW+kuVPo3DdYHuqdif06EDc=', 'Content-Length': '0'} 2024-04-16T20:01:25.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b604c94c1066dbdc-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.713 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.713 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.713 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.713 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:01:25.713 DEBUG:boto:Data: 2024-04-16T20:01:25.714 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.714 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.714 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.714 DEBUG:boto:Params: {} 2024-04-16T20:01:25.714 DEBUG:boto:Token: None 2024-04-16T20:01:25.714 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38 2024-04-16T20:01:25.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o7/yoW+kuVPo3DdYHuqdif06EDc= 2024-04-16T20:01:25.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o7/yoW+kuVPo3DdYHuqdif06EDc=', 'Content-Length': '0'} 2024-04-16T20:01:25.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce6bc340d01b462-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:01:25.717 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.717 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.717 DEBUG:boto:Method: GET 2024-04-16T20:01:25.717 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:01:25.717 DEBUG:boto:Data: 2024-04-16T20:01:25.717 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.717 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.717 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.717 DEBUG:boto:Params: {} 2024-04-16T20:01:25.717 DEBUG:boto:Token: None 2024-04-16T20:01:25.717 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38 2024-04-16T20:01:25.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B/NLxIsXtrPLekTJNBT1wN/D60E= 2024-04-16T20:01:25.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B/NLxIsXtrPLekTJNBT1wN/D60E=', 'Content-Length': '0'} 2024-04-16T20:01:25.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7ed4014f29e003-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.762 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:01:25.762 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:01:25.762 DEBUG:boto:Method: GET 2024-04-16T20:01:25.763 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:01:25.763 DEBUG:boto:Data: 2024-04-16T20:01:25.763 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.763 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.763 DEBUG:boto:Params: {} 2024-04-16T20:01:25.763 DEBUG:boto:Token: None 2024-04-16T20:01:25.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj38 2024-04-16T20:01:25.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B/NLxIsXtrPLekTJNBT1wN/D60E= 2024-04-16T20:01:25.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B/NLxIsXtrPLekTJNBT1wN/D60E=', 'Content-Length': '0'} 2024-04-16T20:01:25.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c23165ff697e5d9-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.765 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.765 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.765 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.765 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.765 DEBUG:boto:Method: GET 2024-04-16T20:01:25.766 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.766 DEBUG:boto:Data: 2024-04-16T20:01:25.766 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.766 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.766 DEBUG:boto:Params: {} 2024-04-16T20:01:25.766 DEBUG:boto:Token: None 2024-04-16T20:01:25.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000257591edfdaa25d-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.769 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.769 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.770 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.770 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.770 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.770 DEBUG:boto:Method: GET 2024-04-16T20:01:25.770 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:01:25.770 DEBUG:boto:Data: 2024-04-16T20:01:25.770 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.770 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.770 DEBUG:boto:Params: {} 2024-04-16T20:01:25.770 DEBUG:boto:Token: None 2024-04-16T20:01:25.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2f8684819685a3-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.773 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:01:25.774 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.774 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.774 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.774 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.774 DEBUG:boto:Method: GET 2024-04-16T20:01:25.774 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.774 DEBUG:boto:Data: 2024-04-16T20:01:25.774 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.775 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.775 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.775 DEBUG:boto:Params: {} 2024-04-16T20:01:25.775 DEBUG:boto:Token: None 2024-04-16T20:01:25.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0= 2024-04-16T20:01:25.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940f36901553d909-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.818 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.818 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.818 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.819 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.819 DEBUG:boto:Method: GET 2024-04-16T20:01:25.819 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.819 DEBUG:boto:Data: 2024-04-16T20:01:25.819 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.819 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.819 DEBUG:boto:Params: {} 2024-04-16T20:01:25.819 DEBUG:boto:Token: None 2024-04-16T20:01:25.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0= 2024-04-16T20:01:25.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2ca95389928276-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.822 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.822 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.822 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.822 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.822 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.822 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.822 DEBUG:boto:Data: 2024-04-16T20:01:25.822 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.823 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.823 DEBUG:boto:Params: {} 2024-04-16T20:01:25.823 DEBUG:boto:Token: None 2024-04-16T20:01:25.823 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hjzZwaD/72uFDLh+lWk6ZiqSsDw= 2024-04-16T20:01:25.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hjzZwaD/72uFDLh+lWk6ZiqSsDw=', 'Content-Length': '0'} 2024-04-16T20:01:25.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018c5480b9ce510e1-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.825 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.826 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.826 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.826 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.826 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.826 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.826 DEBUG:boto:Data: 2024-04-16T20:01:25.826 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.826 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.826 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.826 DEBUG:boto:Params: {} 2024-04-16T20:01:25.826 DEBUG:boto:Token: None 2024-04-16T20:01:25.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hjzZwaD/72uFDLh+lWk6ZiqSsDw= 2024-04-16T20:01:25.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hjzZwaD/72uFDLh+lWk6ZiqSsDw=', 'Content-Length': '0'} 2024-04-16T20:01:25.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d375acc4584584-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:01:25.829 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.829 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.829 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.829 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.829 DEBUG:boto:Method: GET 2024-04-16T20:01:25.829 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.829 DEBUG:boto:Data: 2024-04-16T20:01:25.830 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.830 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.830 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.830 DEBUG:boto:Params: {} 2024-04-16T20:01:25.830 DEBUG:boto:Token: None 2024-04-16T20:01:25.830 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0= 2024-04-16T20:01:25.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80c5bd8c69605d3-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.874 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.874 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.874 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.875 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.875 DEBUG:boto:Method: GET 2024-04-16T20:01:25.875 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.875 DEBUG:boto:Data: 2024-04-16T20:01:25.875 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.875 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.875 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.875 DEBUG:boto:Params: {} 2024-04-16T20:01:25.875 DEBUG:boto:Token: None 2024-04-16T20:01:25.875 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:01:25.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0= 2024-04-16T20:01:25.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YtKSY8+Ms8iVm+GmijHNXx4KZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b2c9dbb0fbfb80-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.878 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.878 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.878 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.878 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.879 DEBUG:boto:Method: GET 2024-04-16T20:01:25.879 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.879 DEBUG:boto:Data: 2024-04-16T20:01:25.879 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.879 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.879 DEBUG:boto:Params: {} 2024-04-16T20:01:25.879 DEBUG:boto:Token: None 2024-04-16T20:01:25.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92f5a6722f59180-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.882 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.883 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.883 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.883 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.883 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.883 DEBUG:boto:Method: GET 2024-04-16T20:01:25.883 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.883 DEBUG:boto:Data: 2024-04-16T20:01:25.883 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.883 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.883 DEBUG:boto:Params: {} 2024-04-16T20:01:25.883 DEBUG:boto:Token: None 2024-04-16T20:01:25.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccaf1900c46bde15-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.887 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.887 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.887 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.887 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.887 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:01:25.887 DEBUG:boto:Data: 2024-04-16T20:01:25.887 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.887 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.888 DEBUG:boto:Params: {} 2024-04-16T20:01:25.888 DEBUG:boto:Token: None 2024-04-16T20:01:25.888 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44 2024-04-16T20:01:25.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjA66zHgz5tS2cL4B4FcW/uWlZ0= 2024-04-16T20:01:25.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjA66zHgz5tS2cL4B4FcW/uWlZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b967dc6fbdd8e78-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.890 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.890 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.890 DEBUG:boto:Method: HEAD 2024-04-16T20:01:25.891 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:01:25.891 DEBUG:boto:Data: 2024-04-16T20:01:25.891 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.891 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.891 DEBUG:boto:Params: {} 2024-04-16T20:01:25.891 DEBUG:boto:Token: None 2024-04-16T20:01:25.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44 2024-04-16T20:01:25.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UjA66zHgz5tS2cL4B4FcW/uWlZ0= 2024-04-16T20:01:25.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UjA66zHgz5tS2cL4B4FcW/uWlZ0=', 'Content-Length': '0'} 2024-04-16T20:01:25.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba31c37ec96f4be3-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:01:25.894 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.894 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.894 DEBUG:boto:Method: GET 2024-04-16T20:01:25.894 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:01:25.894 DEBUG:boto:Data: 2024-04-16T20:01:25.894 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.894 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.894 DEBUG:boto:Params: {} 2024-04-16T20:01:25.894 DEBUG:boto:Token: None 2024-04-16T20:01:25.894 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44 2024-04-16T20:01:25.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:loqqkafu7guHtSz9JPKJAJX6ayI= 2024-04-16T20:01:25.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:loqqkafu7guHtSz9JPKJAJX6ayI=', 'Content-Length': '0'} 2024-04-16T20:01:25.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447e3aca6493a93d-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.938 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:01:25.939 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:01:25.939 DEBUG:boto:Method: GET 2024-04-16T20:01:25.939 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:01:25.939 DEBUG:boto:Data: 2024-04-16T20:01:25.939 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.939 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.939 DEBUG:boto:Params: {} 2024-04-16T20:01:25.939 DEBUG:boto:Token: None 2024-04-16T20:01:25.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj44 2024-04-16T20:01:25.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:loqqkafu7guHtSz9JPKJAJX6ayI= 2024-04-16T20:01:25.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:loqqkafu7guHtSz9JPKJAJX6ayI=', 'Content-Length': '0'} 2024-04-16T20:01:25.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039dbfa684aba9a5a-00661ed915-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.942 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.942 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.942 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.942 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.942 DEBUG:boto:Method: GET 2024-04-16T20:01:25.943 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.943 DEBUG:boto:Data: 2024-04-16T20:01:25.943 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.943 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.943 DEBUG:boto:Params: {} 2024-04-16T20:01:25.943 DEBUG:boto:Token: None 2024-04-16T20:01:25.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36825e32780e1d4-00661ed915-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.946 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.947 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:25.947 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:25.947 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.947 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.947 DEBUG:boto:Method: GET 2024-04-16T20:01:25.947 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:01:25.947 DEBUG:boto:Data: 2024-04-16T20:01:25.947 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.947 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.947 DEBUG:boto:Params: {} 2024-04-16T20:01:25.948 DEBUG:boto:Token: None 2024-04-16T20:01:25.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/ 2024-04-16T20:01:25.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg= 2024-04-16T20:01:25.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sL6rU7P1YhDqJYzYndVvOdWOjfg=', 'Content-Length': '0'} 2024-04-16T20:01:25.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006538c4cacf1e321a-00661ed915-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.951 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:25.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:01:25.951 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:25.951 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:25.951 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.951 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.951 DEBUG:boto:Method: GET 2024-04-16T20:01:25.951 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.951 DEBUG:boto:Data: 2024-04-16T20:01:25.952 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:25.952 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.952 DEBUG:boto:Params: {} 2024-04-16T20:01:25.952 DEBUG:boto:Token: None 2024-04-16T20:01:25.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PYVvEtRYfDdeBPlYjFHYxfi+Jec= 2024-04-16T20:01:25.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PYVvEtRYfDdeBPlYjFHYxfi+Jec=', 'Content-Length': '0'} 2024-04-16T20:01:25.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0c8c51876ea27a-00661ed915-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:25.998 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:25.998 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:25.998 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.999 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.999 DEBUG:boto:Method: GET 2024-04-16T20:01:25.999 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.999 DEBUG:boto:Data: 2024-04-16T20:01:25.999 DEBUG:boto:Headers: {} 2024-04-16T20:01:25.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:25.999 DEBUG:boto:Port: 8000 2024-04-16T20:01:25.999 DEBUG:boto:Params: {} 2024-04-16T20:01:25.999 DEBUG:boto:Token: None 2024-04-16T20:01:25.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:25 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:25.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PYVvEtRYfDdeBPlYjFHYxfi+Jec= 2024-04-16T20:01:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:25 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PYVvEtRYfDdeBPlYjFHYxfi+Jec=', 'Content-Length': '0'} 2024-04-16T20:01:26.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001207da623b05dc1e-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.002 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.002 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.002 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.002 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.002 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.002 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.003 DEBUG:boto:Data: 2024-04-16T20:01:26.003 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.003 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.003 DEBUG:boto:Params: {} 2024-04-16T20:01:26.003 DEBUG:boto:Token: None 2024-04-16T20:01:26.003 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g16XkMRWq7XwnBQ3c6IsF7BumeY= 2024-04-16T20:01:26.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g16XkMRWq7XwnBQ3c6IsF7BumeY=', 'Content-Length': '0'} 2024-04-16T20:01:26.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be2413266c3d0cda-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.006 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.006 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.006 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.006 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.006 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.006 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.006 DEBUG:boto:Data: 2024-04-16T20:01:26.006 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.006 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.007 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.007 DEBUG:boto:Params: {} 2024-04-16T20:01:26.007 DEBUG:boto:Token: None 2024-04-16T20:01:26.007 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g16XkMRWq7XwnBQ3c6IsF7BumeY= 2024-04-16T20:01:26.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g16XkMRWq7XwnBQ3c6IsF7BumeY=', 'Content-Length': '0'} 2024-04-16T20:01:26.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf2be01156716df-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:01:26.009 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.009 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.009 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.009 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.009 DEBUG:boto:Method: GET 2024-04-16T20:01:26.010 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.010 DEBUG:boto:Data: 2024-04-16T20:01:26.010 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.010 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.010 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.010 DEBUG:boto:Params: {} 2024-04-16T20:01:26.010 DEBUG:boto:Token: None 2024-04-16T20:01:26.010 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.010 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cPMkoixWgDF9BR2topkThHHBg4= 2024-04-16T20:01:26.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cPMkoixWgDF9BR2topkThHHBg4=', 'Content-Length': '0'} 2024-04-16T20:01:26.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef1ef5d1f6c18dc-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.054 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.054 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.054 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.055 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.055 DEBUG:boto:Method: GET 2024-04-16T20:01:26.055 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.055 DEBUG:boto:Data: 2024-04-16T20:01:26.055 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.055 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.055 DEBUG:boto:Params: {} 2024-04-16T20:01:26.055 DEBUG:boto:Token: None 2024-04-16T20:01:26.055 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:01:26.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cPMkoixWgDF9BR2topkThHHBg4= 2024-04-16T20:01:26.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cPMkoixWgDF9BR2topkThHHBg4=', 'Content-Length': '0'} 2024-04-16T20:01:26.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e35785c380d97e1-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.058 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.058 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.058 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.059 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.059 DEBUG:boto:Method: GET 2024-04-16T20:01:26.059 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.059 DEBUG:boto:Data: 2024-04-16T20:01:26.059 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.059 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.059 DEBUG:boto:Params: {} 2024-04-16T20:01:26.059 DEBUG:boto:Token: None 2024-04-16T20:01:26.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc65c99e0b9bf6b3-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.062 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.062 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.063 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.063 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.063 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.063 DEBUG:boto:Method: GET 2024-04-16T20:01:26.063 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.063 DEBUG:boto:Data: 2024-04-16T20:01:26.063 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.063 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.063 DEBUG:boto:Params: {} 2024-04-16T20:01:26.063 DEBUG:boto:Token: None 2024-04-16T20:01:26.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8f799516b21cb1-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.066 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.067 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.067 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.067 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.067 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:01:26.067 DEBUG:boto:Data: 2024-04-16T20:01:26.067 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.067 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.067 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.067 DEBUG:boto:Params: {} 2024-04-16T20:01:26.067 DEBUG:boto:Token: None 2024-04-16T20:01:26.068 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51 2024-04-16T20:01:26.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yLURGeJy2JcRt8UnaiaDALJAbL8= 2024-04-16T20:01:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yLURGeJy2JcRt8UnaiaDALJAbL8=', 'Content-Length': '0'} 2024-04-16T20:01:26.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000385c8498a2995c56-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.071 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.071 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.071 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.071 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:01:26.071 DEBUG:boto:Data: 2024-04-16T20:01:26.071 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.071 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.071 DEBUG:boto:Params: {} 2024-04-16T20:01:26.071 DEBUG:boto:Token: None 2024-04-16T20:01:26.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51 2024-04-16T20:01:26.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yLURGeJy2JcRt8UnaiaDALJAbL8= 2024-04-16T20:01:26.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yLURGeJy2JcRt8UnaiaDALJAbL8=', 'Content-Length': '0'} 2024-04-16T20:01:26.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf0b2d56117181b8-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:01:26.074 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.074 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.074 DEBUG:boto:Method: GET 2024-04-16T20:01:26.074 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:01:26.074 DEBUG:boto:Data: 2024-04-16T20:01:26.075 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.075 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.075 DEBUG:boto:Params: {} 2024-04-16T20:01:26.075 DEBUG:boto:Token: None 2024-04-16T20:01:26.075 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51 2024-04-16T20:01:26.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8pnyGS5JBXU8dNYlK3k25s/Ch4= 2024-04-16T20:01:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8pnyGS5JBXU8dNYlK3k25s/Ch4=', 'Content-Length': '0'} 2024-04-16T20:01:26.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eac7361dce543e9-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.118 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:01:26.119 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:01:26.119 DEBUG:boto:Method: GET 2024-04-16T20:01:26.119 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:01:26.119 DEBUG:boto:Data: 2024-04-16T20:01:26.119 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.119 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.119 DEBUG:boto:Params: {} 2024-04-16T20:01:26.119 DEBUG:boto:Token: None 2024-04-16T20:01:26.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj51 2024-04-16T20:01:26.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t8pnyGS5JBXU8dNYlK3k25s/Ch4= 2024-04-16T20:01:26.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t8pnyGS5JBXU8dNYlK3k25s/Ch4=', 'Content-Length': '0'} 2024-04-16T20:01:26.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a960089d1a2d0c0-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.122 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.123 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.123 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.123 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.123 DEBUG:boto:Method: GET 2024-04-16T20:01:26.123 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.123 DEBUG:boto:Data: 2024-04-16T20:01:26.123 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.123 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.123 DEBUG:boto:Params: {} 2024-04-16T20:01:26.123 DEBUG:boto:Token: None 2024-04-16T20:01:26.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e6bbdf44d91c14-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.170 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.171 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.171 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.171 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.171 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.171 DEBUG:boto:Method: GET 2024-04-16T20:01:26.171 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:01:26.171 DEBUG:boto:Data: 2024-04-16T20:01:26.171 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.171 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.171 DEBUG:boto:Params: {} 2024-04-16T20:01:26.171 DEBUG:boto:Token: None 2024-04-16T20:01:26.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000828d10e1ec6c9722-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.174 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:01:26.175 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.175 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.175 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.175 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.175 DEBUG:boto:Method: GET 2024-04-16T20:01:26.175 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.175 DEBUG:boto:Data: 2024-04-16T20:01:26.176 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.176 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.176 DEBUG:boto:Params: {} 2024-04-16T20:01:26.176 DEBUG:boto:Token: None 2024-04-16T20:01:26.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o= 2024-04-16T20:01:26.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o=', 'Content-Length': '0'} 2024-04-16T20:01:26.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a67467d7aedfc1a-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.222 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.222 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.223 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.223 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.223 DEBUG:boto:Method: GET 2024-04-16T20:01:26.223 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.223 DEBUG:boto:Data: 2024-04-16T20:01:26.223 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.223 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.223 DEBUG:boto:Params: {} 2024-04-16T20:01:26.223 DEBUG:boto:Token: None 2024-04-16T20:01:26.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o= 2024-04-16T20:01:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o=', 'Content-Length': '0'} 2024-04-16T20:01:26.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8ff6722ab9e490-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.226 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.227 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.227 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.227 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.227 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.227 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.227 DEBUG:boto:Data: 2024-04-16T20:01:26.227 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.227 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.227 DEBUG:boto:Params: {} 2024-04-16T20:01:26.227 DEBUG:boto:Token: None 2024-04-16T20:01:26.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv0329Sw92dC13z0r38xSFIoAuA= 2024-04-16T20:01:26.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv0329Sw92dC13z0r38xSFIoAuA=', 'Content-Length': '0'} 2024-04-16T20:01:26.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7dbb9cd959aa7ae-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.230 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.230 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.230 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.231 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.231 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.231 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.231 DEBUG:boto:Data: 2024-04-16T20:01:26.231 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.231 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.231 DEBUG:boto:Params: {} 2024-04-16T20:01:26.231 DEBUG:boto:Token: None 2024-04-16T20:01:26.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv0329Sw92dC13z0r38xSFIoAuA= 2024-04-16T20:01:26.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv0329Sw92dC13z0r38xSFIoAuA=', 'Content-Length': '0'} 2024-04-16T20:01:26.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca029073b4d1ede-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:01:26.234 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.234 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.234 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.234 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.234 DEBUG:boto:Method: GET 2024-04-16T20:01:26.235 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.235 DEBUG:boto:Data: 2024-04-16T20:01:26.235 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.235 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.235 DEBUG:boto:Params: {} 2024-04-16T20:01:26.235 DEBUG:boto:Token: None 2024-04-16T20:01:26.235 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o= 2024-04-16T20:01:26.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o=', 'Content-Length': '0'} 2024-04-16T20:01:26.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40944a128ceeefc-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.278 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.278 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.278 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.279 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.279 DEBUG:boto:Method: GET 2024-04-16T20:01:26.279 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.279 DEBUG:boto:Data: 2024-04-16T20:01:26.279 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.279 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.279 DEBUG:boto:Params: {} 2024-04-16T20:01:26.279 DEBUG:boto:Token: None 2024-04-16T20:01:26.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:01:26.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o= 2024-04-16T20:01:26.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l1nX5wX8SniT9KM/fslpTsQKe8o=', 'Content-Length': '0'} 2024-04-16T20:01:26.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d7cea2071f4f0b-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.282 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.282 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.282 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.282 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.282 DEBUG:boto:Method: GET 2024-04-16T20:01:26.282 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.282 DEBUG:boto:Data: 2024-04-16T20:01:26.282 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.283 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.283 DEBUG:boto:Params: {} 2024-04-16T20:01:26.283 DEBUG:boto:Token: None 2024-04-16T20:01:26.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045fddc9723445709-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.326 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.327 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.327 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.327 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.327 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.327 DEBUG:boto:Method: GET 2024-04-16T20:01:26.327 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.327 DEBUG:boto:Data: 2024-04-16T20:01:26.327 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.328 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.328 DEBUG:boto:Params: {} 2024-04-16T20:01:26.328 DEBUG:boto:Token: None 2024-04-16T20:01:26.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000927564b2e8791683-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.331 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.331 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.331 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.332 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.332 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:01:26.332 DEBUG:boto:Data: 2024-04-16T20:01:26.332 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.332 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.332 DEBUG:boto:Params: {} 2024-04-16T20:01:26.332 DEBUG:boto:Token: None 2024-04-16T20:01:26.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55 2024-04-16T20:01:26.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MqooTqye8YejqvLRnkL4Nn9YiAU= 2024-04-16T20:01:26.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MqooTqye8YejqvLRnkL4Nn9YiAU=', 'Content-Length': '0'} 2024-04-16T20:01:26.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6192763827056e-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.335 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.335 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.335 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.335 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:01:26.335 DEBUG:boto:Data: 2024-04-16T20:01:26.335 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.335 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.335 DEBUG:boto:Params: {} 2024-04-16T20:01:26.335 DEBUG:boto:Token: None 2024-04-16T20:01:26.336 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55 2024-04-16T20:01:26.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MqooTqye8YejqvLRnkL4Nn9YiAU= 2024-04-16T20:01:26.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MqooTqye8YejqvLRnkL4Nn9YiAU=', 'Content-Length': '0'} 2024-04-16T20:01:26.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab3907c42d784779-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:01:26.338 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.339 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.339 DEBUG:boto:Method: GET 2024-04-16T20:01:26.339 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:01:26.339 DEBUG:boto:Data: 2024-04-16T20:01:26.339 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.339 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.339 DEBUG:boto:Params: {} 2024-04-16T20:01:26.339 DEBUG:boto:Token: None 2024-04-16T20:01:26.339 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55 2024-04-16T20:01:26.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JQPoTX/TRPUax4sLW5a8FDc2eIY= 2024-04-16T20:01:26.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JQPoTX/TRPUax4sLW5a8FDc2eIY=', 'Content-Length': '0'} 2024-04-16T20:01:26.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbb01e536086187-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.382 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:01:26.382 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:01:26.382 DEBUG:boto:Method: GET 2024-04-16T20:01:26.383 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:01:26.383 DEBUG:boto:Data: 2024-04-16T20:01:26.383 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.383 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.383 DEBUG:boto:Params: {} 2024-04-16T20:01:26.383 DEBUG:boto:Token: None 2024-04-16T20:01:26.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj55 2024-04-16T20:01:26.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JQPoTX/TRPUax4sLW5a8FDc2eIY= 2024-04-16T20:01:26.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JQPoTX/TRPUax4sLW5a8FDc2eIY=', 'Content-Length': '0'} 2024-04-16T20:01:26.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072561bf09d31e28a-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.386 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.386 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.386 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.386 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.386 DEBUG:boto:Method: GET 2024-04-16T20:01:26.386 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.386 DEBUG:boto:Data: 2024-04-16T20:01:26.386 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.387 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.387 DEBUG:boto:Params: {} 2024-04-16T20:01:26.387 DEBUG:boto:Token: None 2024-04-16T20:01:26.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2717319e2b4fbdd-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.430 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.431 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.431 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.431 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.431 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.431 DEBUG:boto:Method: GET 2024-04-16T20:01:26.431 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:01:26.431 DEBUG:boto:Data: 2024-04-16T20:01:26.431 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.431 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.431 DEBUG:boto:Params: {} 2024-04-16T20:01:26.432 DEBUG:boto:Token: None 2024-04-16T20:01:26.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e624312b0ba8db8-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.434 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:01:26.435 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.435 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.435 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.435 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.435 DEBUG:boto:Method: GET 2024-04-16T20:01:26.435 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.435 DEBUG:boto:Data: 2024-04-16T20:01:26.435 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.436 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.436 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.436 DEBUG:boto:Params: {} 2024-04-16T20:01:26.436 DEBUG:boto:Token: None 2024-04-16T20:01:26.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc= 2024-04-16T20:01:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc=', 'Content-Length': '0'} 2024-04-16T20:01:26.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005135a7626e2b23de-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.482 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.482 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.482 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.483 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.483 DEBUG:boto:Method: GET 2024-04-16T20:01:26.483 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.483 DEBUG:boto:Data: 2024-04-16T20:01:26.483 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.483 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.483 DEBUG:boto:Params: {} 2024-04-16T20:01:26.483 DEBUG:boto:Token: None 2024-04-16T20:01:26.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc= 2024-04-16T20:01:26.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc=', 'Content-Length': '0'} 2024-04-16T20:01:26.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b70e134121d52f-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.486 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.486 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.486 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.486 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.487 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.487 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.487 DEBUG:boto:Data: 2024-04-16T20:01:26.487 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.487 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.487 DEBUG:boto:Params: {} 2024-04-16T20:01:26.487 DEBUG:boto:Token: None 2024-04-16T20:01:26.487 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rBAedGLdw3Bjj+jnIQL9sjy5YH0= 2024-04-16T20:01:26.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rBAedGLdw3Bjj+jnIQL9sjy5YH0=', 'Content-Length': '0'} 2024-04-16T20:01:26.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3fc8669ffca8292-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.490 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.490 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.490 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.490 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.491 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.491 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.491 DEBUG:boto:Data: 2024-04-16T20:01:26.491 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.491 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.491 DEBUG:boto:Params: {} 2024-04-16T20:01:26.491 DEBUG:boto:Token: None 2024-04-16T20:01:26.491 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rBAedGLdw3Bjj+jnIQL9sjy5YH0= 2024-04-16T20:01:26.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rBAedGLdw3Bjj+jnIQL9sjy5YH0=', 'Content-Length': '0'} 2024-04-16T20:01:26.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f7d89e63801da9-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:01:26.494 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.494 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.494 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.494 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.494 DEBUG:boto:Method: GET 2024-04-16T20:01:26.494 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.494 DEBUG:boto:Data: 2024-04-16T20:01:26.495 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.495 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.495 DEBUG:boto:Params: {} 2024-04-16T20:01:26.495 DEBUG:boto:Token: None 2024-04-16T20:01:26.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc= 2024-04-16T20:01:26.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc=', 'Content-Length': '0'} 2024-04-16T20:01:26.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec218d116cf76cb-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.538 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.538 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.538 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.539 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.539 DEBUG:boto:Method: GET 2024-04-16T20:01:26.539 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.539 DEBUG:boto:Data: 2024-04-16T20:01:26.539 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.539 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.539 DEBUG:boto:Params: {} 2024-04-16T20:01:26.539 DEBUG:boto:Token: None 2024-04-16T20:01:26.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:01:26.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc= 2024-04-16T20:01:26.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KD9ItqGRYHJU7755/e9GFUgsWLc=', 'Content-Length': '0'} 2024-04-16T20:01:26.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec197ae74969253-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.542 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.542 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.542 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.542 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.542 DEBUG:boto:Method: GET 2024-04-16T20:01:26.542 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.543 DEBUG:boto:Data: 2024-04-16T20:01:26.543 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.543 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.543 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.543 DEBUG:boto:Params: {} 2024-04-16T20:01:26.543 DEBUG:boto:Token: None 2024-04-16T20:01:26.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056af3089b47c0567-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.586 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.587 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.587 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.587 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.587 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.587 DEBUG:boto:Method: GET 2024-04-16T20:01:26.587 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.587 DEBUG:boto:Data: 2024-04-16T20:01:26.587 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.587 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.587 DEBUG:boto:Params: {} 2024-04-16T20:01:26.588 DEBUG:boto:Token: None 2024-04-16T20:01:26.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e81bd1b1579985-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.590 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.591 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.591 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.591 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.591 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:01:26.591 DEBUG:boto:Data: 2024-04-16T20:01:26.591 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.591 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.591 DEBUG:boto:Params: {} 2024-04-16T20:01:26.591 DEBUG:boto:Token: None 2024-04-16T20:01:26.592 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57 2024-04-16T20:01:26.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FaH7mBTlT3eMQiSbTIAmzFURLfA= 2024-04-16T20:01:26.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FaH7mBTlT3eMQiSbTIAmzFURLfA=', 'Content-Length': '0'} 2024-04-16T20:01:26.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001872084a6b34dfc2-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.594 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.594 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.594 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.594 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:01:26.595 DEBUG:boto:Data: 2024-04-16T20:01:26.595 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.595 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.595 DEBUG:boto:Params: {} 2024-04-16T20:01:26.595 DEBUG:boto:Token: None 2024-04-16T20:01:26.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57 2024-04-16T20:01:26.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FaH7mBTlT3eMQiSbTIAmzFURLfA= 2024-04-16T20:01:26.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FaH7mBTlT3eMQiSbTIAmzFURLfA=', 'Content-Length': '0'} 2024-04-16T20:01:26.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f6e4f5268a54a1-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:01:26.598 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.598 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.598 DEBUG:boto:Method: GET 2024-04-16T20:01:26.598 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:01:26.598 DEBUG:boto:Data: 2024-04-16T20:01:26.598 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.598 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.598 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.598 DEBUG:boto:Params: {} 2024-04-16T20:01:26.599 DEBUG:boto:Token: None 2024-04-16T20:01:26.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57 2024-04-16T20:01:26.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/jTnsZhoL6UoOEUV97Oo2ZQBuCw= 2024-04-16T20:01:26.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/jTnsZhoL6UoOEUV97Oo2ZQBuCw=', 'Content-Length': '0'} 2024-04-16T20:01:26.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2dc57ff14ef8ec-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.646 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:01:26.646 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:01:26.646 DEBUG:boto:Method: GET 2024-04-16T20:01:26.647 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:01:26.647 DEBUG:boto:Data: 2024-04-16T20:01:26.647 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.647 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.647 DEBUG:boto:Params: {} 2024-04-16T20:01:26.647 DEBUG:boto:Token: None 2024-04-16T20:01:26.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj57 2024-04-16T20:01:26.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/jTnsZhoL6UoOEUV97Oo2ZQBuCw= 2024-04-16T20:01:26.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/jTnsZhoL6UoOEUV97Oo2ZQBuCw=', 'Content-Length': '0'} 2024-04-16T20:01:26.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e212c890f8679c-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.650 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.650 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.650 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.650 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.650 DEBUG:boto:Method: GET 2024-04-16T20:01:26.650 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.650 DEBUG:boto:Data: 2024-04-16T20:01:26.650 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.650 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.651 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.651 DEBUG:boto:Params: {} 2024-04-16T20:01:26.651 DEBUG:boto:Token: None 2024-04-16T20:01:26.651 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a1911aa1f73a4b-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.653 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.653 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.654 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.654 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.654 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.654 DEBUG:boto:Method: GET 2024-04-16T20:01:26.654 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:01:26.654 DEBUG:boto:Data: 2024-04-16T20:01:26.654 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.654 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.654 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.654 DEBUG:boto:Params: {} 2024-04-16T20:01:26.654 DEBUG:boto:Token: None 2024-04-16T20:01:26.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a8bca97151527b5-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.657 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:01:26.658 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.658 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.658 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.658 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.658 DEBUG:boto:Method: GET 2024-04-16T20:01:26.658 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.658 DEBUG:boto:Data: 2024-04-16T20:01:26.659 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.659 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.659 DEBUG:boto:Params: {} 2024-04-16T20:01:26.659 DEBUG:boto:Token: None 2024-04-16T20:01:26.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc= 2024-04-16T20:01:26.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc=', 'Content-Length': '0'} 2024-04-16T20:01:26.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca6c8e66106d4d3-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.702 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.702 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.703 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.703 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.703 DEBUG:boto:Method: GET 2024-04-16T20:01:26.703 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.703 DEBUG:boto:Data: 2024-04-16T20:01:26.703 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.703 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.703 DEBUG:boto:Params: {} 2024-04-16T20:01:26.703 DEBUG:boto:Token: None 2024-04-16T20:01:26.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc= 2024-04-16T20:01:26.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc=', 'Content-Length': '0'} 2024-04-16T20:01:26.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003219ef16c0d89856-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.706 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.706 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.706 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.706 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.706 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.706 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.707 DEBUG:boto:Data: 2024-04-16T20:01:26.707 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.707 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.707 DEBUG:boto:Params: {} 2024-04-16T20:01:26.707 DEBUG:boto:Token: None 2024-04-16T20:01:26.707 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qc48pjn+BfzpeosEufnwP1eE6Ug= 2024-04-16T20:01:26.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qc48pjn+BfzpeosEufnwP1eE6Ug=', 'Content-Length': '0'} 2024-04-16T20:01:26.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21e88639ca221bf-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.710 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.710 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.710 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.710 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.710 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.710 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.710 DEBUG:boto:Data: 2024-04-16T20:01:26.710 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.710 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.711 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.711 DEBUG:boto:Params: {} 2024-04-16T20:01:26.711 DEBUG:boto:Token: None 2024-04-16T20:01:26.711 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qc48pjn+BfzpeosEufnwP1eE6Ug= 2024-04-16T20:01:26.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qc48pjn+BfzpeosEufnwP1eE6Ug=', 'Content-Length': '0'} 2024-04-16T20:01:26.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f1b7dccaf52b63a-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:01:26.713 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.713 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.713 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.714 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.714 DEBUG:boto:Method: GET 2024-04-16T20:01:26.714 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.714 DEBUG:boto:Data: 2024-04-16T20:01:26.714 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.714 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.714 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.714 DEBUG:boto:Params: {} 2024-04-16T20:01:26.714 DEBUG:boto:Token: None 2024-04-16T20:01:26.714 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc= 2024-04-16T20:01:26.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc=', 'Content-Length': '0'} 2024-04-16T20:01:26.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075c2f5fcce3d94d-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.758 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.758 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.758 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.759 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.759 DEBUG:boto:Method: GET 2024-04-16T20:01:26.759 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.759 DEBUG:boto:Data: 2024-04-16T20:01:26.759 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.759 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.759 DEBUG:boto:Params: {} 2024-04-16T20:01:26.759 DEBUG:boto:Token: None 2024-04-16T20:01:26.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:01:26.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc= 2024-04-16T20:01:26.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1XFbbQ21n6iwCaAHQ9N/HN/QnSc=', 'Content-Length': '0'} 2024-04-16T20:01:26.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff69a30ee3214c9-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.762 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.762 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.762 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.762 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.762 DEBUG:boto:Method: GET 2024-04-16T20:01:26.762 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.762 DEBUG:boto:Data: 2024-04-16T20:01:26.762 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.762 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.762 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.762 DEBUG:boto:Params: {} 2024-04-16T20:01:26.763 DEBUG:boto:Token: None 2024-04-16T20:01:26.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba1c9f8a720e619-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.765 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.766 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.766 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.766 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.766 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.766 DEBUG:boto:Method: GET 2024-04-16T20:01:26.766 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.766 DEBUG:boto:Data: 2024-04-16T20:01:26.766 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.766 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.767 DEBUG:boto:Params: {} 2024-04-16T20:01:26.767 DEBUG:boto:Token: None 2024-04-16T20:01:26.767 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018006fd9a95d3d49-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.770 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.770 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.770 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.770 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.770 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:01:26.771 DEBUG:boto:Data: 2024-04-16T20:01:26.771 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.771 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.771 DEBUG:boto:Params: {} 2024-04-16T20:01:26.771 DEBUG:boto:Token: None 2024-04-16T20:01:26.771 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68 2024-04-16T20:01:26.771 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hFHcd7XHYaQzLSnU+NwyQJEjzZs= 2024-04-16T20:01:26.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hFHcd7XHYaQzLSnU+NwyQJEjzZs=', 'Content-Length': '0'} 2024-04-16T20:01:26.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4bc69a8088b3f52-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.774 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.774 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.774 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.774 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:01:26.774 DEBUG:boto:Data: 2024-04-16T20:01:26.774 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.774 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.774 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.774 DEBUG:boto:Params: {} 2024-04-16T20:01:26.774 DEBUG:boto:Token: None 2024-04-16T20:01:26.775 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68 2024-04-16T20:01:26.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hFHcd7XHYaQzLSnU+NwyQJEjzZs= 2024-04-16T20:01:26.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hFHcd7XHYaQzLSnU+NwyQJEjzZs=', 'Content-Length': '0'} 2024-04-16T20:01:26.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ca8b05d3b30651-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:01:26.777 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.777 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.777 DEBUG:boto:Method: GET 2024-04-16T20:01:26.777 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:01:26.777 DEBUG:boto:Data: 2024-04-16T20:01:26.777 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.777 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.777 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.777 DEBUG:boto:Params: {} 2024-04-16T20:01:26.778 DEBUG:boto:Token: None 2024-04-16T20:01:26.778 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68 2024-04-16T20:01:26.778 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f284ScK5hdTIdeLJE98IluyOIlc= 2024-04-16T20:01:26.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f284ScK5hdTIdeLJE98IluyOIlc=', 'Content-Length': '0'} 2024-04-16T20:01:26.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954363be21877391-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.822 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:01:26.823 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:01:26.823 DEBUG:boto:Method: GET 2024-04-16T20:01:26.823 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:01:26.823 DEBUG:boto:Data: 2024-04-16T20:01:26.823 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.823 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.823 DEBUG:boto:Params: {} 2024-04-16T20:01:26.823 DEBUG:boto:Token: None 2024-04-16T20:01:26.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj68 2024-04-16T20:01:26.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f284ScK5hdTIdeLJE98IluyOIlc= 2024-04-16T20:01:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f284ScK5hdTIdeLJE98IluyOIlc=', 'Content-Length': '0'} 2024-04-16T20:01:26.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001709b3b89fc5b61c-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.826 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.826 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.826 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.826 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.826 DEBUG:boto:Method: GET 2024-04-16T20:01:26.826 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.826 DEBUG:boto:Data: 2024-04-16T20:01:26.826 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.826 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.826 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.827 DEBUG:boto:Params: {} 2024-04-16T20:01:26.827 DEBUG:boto:Token: None 2024-04-16T20:01:26.827 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1ee6a50ef62062-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.829 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.830 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.830 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.830 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.830 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.830 DEBUG:boto:Method: GET 2024-04-16T20:01:26.830 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:01:26.830 DEBUG:boto:Data: 2024-04-16T20:01:26.830 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.830 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.830 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.831 DEBUG:boto:Params: {} 2024-04-16T20:01:26.831 DEBUG:boto:Token: None 2024-04-16T20:01:26.831 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a856e50b2071c55-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.834 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:01:26.834 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.834 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.834 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.835 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.835 DEBUG:boto:Method: GET 2024-04-16T20:01:26.835 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.835 DEBUG:boto:Data: 2024-04-16T20:01:26.835 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.835 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.835 DEBUG:boto:Params: {} 2024-04-16T20:01:26.835 DEBUG:boto:Token: None 2024-04-16T20:01:26.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE= 2024-04-16T20:01:26.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE=', 'Content-Length': '0'} 2024-04-16T20:01:26.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9750fd962f59c61-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.878 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.878 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.878 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.879 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.879 DEBUG:boto:Method: GET 2024-04-16T20:01:26.879 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.879 DEBUG:boto:Data: 2024-04-16T20:01:26.879 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.879 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.879 DEBUG:boto:Params: {} 2024-04-16T20:01:26.879 DEBUG:boto:Token: None 2024-04-16T20:01:26.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE= 2024-04-16T20:01:26.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE=', 'Content-Length': '0'} 2024-04-16T20:01:26.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2480035c1080a1-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.882 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.882 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.882 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.882 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.882 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.882 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.882 DEBUG:boto:Data: 2024-04-16T20:01:26.882 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.883 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.883 DEBUG:boto:Params: {} 2024-04-16T20:01:26.883 DEBUG:boto:Token: None 2024-04-16T20:01:26.883 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKLJW4A+9bseezjSlDPXchbhZO4= 2024-04-16T20:01:26.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKLJW4A+9bseezjSlDPXchbhZO4=', 'Content-Length': '0'} 2024-04-16T20:01:26.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e460d990935791a9-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.885 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.885 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.885 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.885 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.885 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.886 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.886 DEBUG:boto:Data: 2024-04-16T20:01:26.886 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.886 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.886 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.886 DEBUG:boto:Params: {} 2024-04-16T20:01:26.886 DEBUG:boto:Token: None 2024-04-16T20:01:26.886 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.886 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKLJW4A+9bseezjSlDPXchbhZO4= 2024-04-16T20:01:26.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKLJW4A+9bseezjSlDPXchbhZO4=', 'Content-Length': '0'} 2024-04-16T20:01:26.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daff32da64cdf3c8-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:01:26.889 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.889 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.889 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.889 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.889 DEBUG:boto:Method: GET 2024-04-16T20:01:26.889 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.889 DEBUG:boto:Data: 2024-04-16T20:01:26.889 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.889 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.889 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.890 DEBUG:boto:Params: {} 2024-04-16T20:01:26.890 DEBUG:boto:Token: None 2024-04-16T20:01:26.890 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.890 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE= 2024-04-16T20:01:26.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE=', 'Content-Length': '0'} 2024-04-16T20:01:26.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfdadbb713cf326-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.934 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.935 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.935 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.935 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.935 DEBUG:boto:Method: GET 2024-04-16T20:01:26.935 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.935 DEBUG:boto:Data: 2024-04-16T20:01:26.935 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.935 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.935 DEBUG:boto:Params: {} 2024-04-16T20:01:26.935 DEBUG:boto:Token: None 2024-04-16T20:01:26.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:01:26.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE= 2024-04-16T20:01:26.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P9aohZBPkXg4QEbiIl1ukPFDOhE=', 'Content-Length': '0'} 2024-04-16T20:01:26.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e139c5462aa99164-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.938 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.938 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.938 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.938 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.938 DEBUG:boto:Method: GET 2024-04-16T20:01:26.938 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.938 DEBUG:boto:Data: 2024-04-16T20:01:26.938 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.938 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.939 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.939 DEBUG:boto:Params: {} 2024-04-16T20:01:26.939 DEBUG:boto:Token: None 2024-04-16T20:01:26.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011736c4c9bc8c729-00661ed916-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.941 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.942 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:26.942 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:26.942 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.942 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.942 DEBUG:boto:Method: GET 2024-04-16T20:01:26.942 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:01:26.942 DEBUG:boto:Data: 2024-04-16T20:01:26.942 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.942 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.943 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.943 DEBUG:boto:Params: {} 2024-04-16T20:01:26.943 DEBUG:boto:Token: None 2024-04-16T20:01:26.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/ 2024-04-16T20:01:26.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0= 2024-04-16T20:01:26.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5/elPDnbk1BGPZwels48lJaElK0=', 'Content-Length': '0'} 2024-04-16T20:01:26.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a954550319e3d6be-00661ed916-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.946 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:26.946 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.947 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.947 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.947 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:01:26.947 DEBUG:boto:Data: 2024-04-16T20:01:26.947 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.947 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.947 DEBUG:boto:Params: {} 2024-04-16T20:01:26.947 DEBUG:boto:Token: None 2024-04-16T20:01:26.947 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76 2024-04-16T20:01:26.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1+petYC/QSPimMPBpeH8/r0NErk= 2024-04-16T20:01:26.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1+petYC/QSPimMPBpeH8/r0NErk=', 'Content-Length': '0'} 2024-04-16T20:01:26.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13d253e25bed49f-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.949 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.950 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.950 DEBUG:boto:Method: HEAD 2024-04-16T20:01:26.950 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:01:26.950 DEBUG:boto:Data: 2024-04-16T20:01:26.950 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.950 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.950 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.950 DEBUG:boto:Params: {} 2024-04-16T20:01:26.950 DEBUG:boto:Token: None 2024-04-16T20:01:26.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76 2024-04-16T20:01:26.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1+petYC/QSPimMPBpeH8/r0NErk= 2024-04-16T20:01:26.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1+petYC/QSPimMPBpeH8/r0NErk=', 'Content-Length': '0'} 2024-04-16T20:01:26.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7dbe14d09dc40a0-00661ed916-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:01:26.953 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.953 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.953 DEBUG:boto:Method: GET 2024-04-16T20:01:26.953 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:01:26.953 DEBUG:boto:Data: 2024-04-16T20:01:26.953 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.953 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:26.953 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.953 DEBUG:boto:Params: {} 2024-04-16T20:01:26.954 DEBUG:boto:Token: None 2024-04-16T20:01:26.954 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76 2024-04-16T20:01:26.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4QC/VvnpW3od3PCploCbw/0zxo4= 2024-04-16T20:01:26.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4QC/VvnpW3od3PCploCbw/0zxo4=', 'Content-Length': '0'} 2024-04-16T20:01:26.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5b7f8302b5a5ed-00661ed916-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:26.998 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:01:26.998 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:01:26.999 DEBUG:boto:Method: GET 2024-04-16T20:01:26.999 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:01:26.999 DEBUG:boto:Data: 2024-04-16T20:01:26.999 DEBUG:boto:Headers: {} 2024-04-16T20:01:26.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:26.999 DEBUG:boto:Port: 8000 2024-04-16T20:01:26.999 DEBUG:boto:Params: {} 2024-04-16T20:01:26.999 DEBUG:boto:Token: None 2024-04-16T20:01:26.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:26 GMT /yrjefh-52/obj76 2024-04-16T20:01:26.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4QC/VvnpW3od3PCploCbw/0zxo4= 2024-04-16T20:01:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:26 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4QC/VvnpW3od3PCploCbw/0zxo4=', 'Content-Length': '0'} 2024-04-16T20:01:27.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c32821aff2fd890-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.001 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.001 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.002 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.002 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.002 DEBUG:boto:Method: GET 2024-04-16T20:01:27.002 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.002 DEBUG:boto:Data: 2024-04-16T20:01:27.002 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.002 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.002 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.002 DEBUG:boto:Params: {} 2024-04-16T20:01:27.002 DEBUG:boto:Token: None 2024-04-16T20:01:27.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c09c104909756628-00661ed917-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.005 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.006 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.006 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.006 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.006 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.006 DEBUG:boto:Method: GET 2024-04-16T20:01:27.006 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:01:27.006 DEBUG:boto:Data: 2024-04-16T20:01:27.006 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.006 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.007 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.007 DEBUG:boto:Params: {} 2024-04-16T20:01:27.007 DEBUG:boto:Token: None 2024-04-16T20:01:27.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e19dd9b27e0afea8-00661ed917-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.010 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:01:27.010 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.010 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.010 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.011 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.011 DEBUG:boto:Method: GET 2024-04-16T20:01:27.011 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.011 DEBUG:boto:Data: 2024-04-16T20:01:27.011 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.011 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.011 DEBUG:boto:Params: {} 2024-04-16T20:01:27.011 DEBUG:boto:Token: None 2024-04-16T20:01:27.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE= 2024-04-16T20:01:27.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE=', 'Content-Length': '0'} 2024-04-16T20:01:27.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003aaea9b05756737-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.054 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.054 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.055 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.055 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.055 DEBUG:boto:Method: GET 2024-04-16T20:01:27.055 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.055 DEBUG:boto:Data: 2024-04-16T20:01:27.055 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.055 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.055 DEBUG:boto:Params: {} 2024-04-16T20:01:27.055 DEBUG:boto:Token: None 2024-04-16T20:01:27.055 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE= 2024-04-16T20:01:27.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE=', 'Content-Length': '0'} 2024-04-16T20:01:27.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89d5cc663f92f6f-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.058 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.058 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.058 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.058 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.058 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.058 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.058 DEBUG:boto:Data: 2024-04-16T20:01:27.058 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.059 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.059 DEBUG:boto:Params: {} 2024-04-16T20:01:27.059 DEBUG:boto:Token: None 2024-04-16T20:01:27.059 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I++w1bMskACEOLZ0pI8mJd6WDXA= 2024-04-16T20:01:27.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I++w1bMskACEOLZ0pI8mJd6WDXA=', 'Content-Length': '0'} 2024-04-16T20:01:27.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b2b5181a53c8a1a-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.061 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.061 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.061 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.061 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.061 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.062 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.062 DEBUG:boto:Data: 2024-04-16T20:01:27.062 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.062 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.062 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.062 DEBUG:boto:Params: {} 2024-04-16T20:01:27.062 DEBUG:boto:Token: None 2024-04-16T20:01:27.062 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.062 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I++w1bMskACEOLZ0pI8mJd6WDXA= 2024-04-16T20:01:27.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I++w1bMskACEOLZ0pI8mJd6WDXA=', 'Content-Length': '0'} 2024-04-16T20:01:27.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ec5f2a1f09d094-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:01:27.065 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.065 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.065 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.065 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.065 DEBUG:boto:Method: GET 2024-04-16T20:01:27.065 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.065 DEBUG:boto:Data: 2024-04-16T20:01:27.065 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.065 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.065 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.065 DEBUG:boto:Params: {} 2024-04-16T20:01:27.066 DEBUG:boto:Token: None 2024-04-16T20:01:27.066 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.066 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE= 2024-04-16T20:01:27.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE=', 'Content-Length': '0'} 2024-04-16T20:01:27.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc36d2a875a7ec74-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.110 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.111 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.111 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.111 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.111 DEBUG:boto:Method: GET 2024-04-16T20:01:27.111 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.111 DEBUG:boto:Data: 2024-04-16T20:01:27.111 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.111 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.111 DEBUG:boto:Params: {} 2024-04-16T20:01:27.112 DEBUG:boto:Token: None 2024-04-16T20:01:27.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:01:27.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE= 2024-04-16T20:01:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R/KP/UTUag8epoJLB9K9ErTA0FE=', 'Content-Length': '0'} 2024-04-16T20:01:27.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371f834f8f4e5f51-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.114 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.114 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.114 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.114 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.114 DEBUG:boto:Method: GET 2024-04-16T20:01:27.114 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.115 DEBUG:boto:Data: 2024-04-16T20:01:27.115 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.115 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.115 DEBUG:boto:Params: {} 2024-04-16T20:01:27.115 DEBUG:boto:Token: None 2024-04-16T20:01:27.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a5aec2c055b7fa-00661ed917-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.118 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.118 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.118 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.118 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.118 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.119 DEBUG:boto:Method: GET 2024-04-16T20:01:27.119 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.119 DEBUG:boto:Data: 2024-04-16T20:01:27.119 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.119 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.119 DEBUG:boto:Params: {} 2024-04-16T20:01:27.119 DEBUG:boto:Token: None 2024-04-16T20:01:27.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd36dc4c64b523a-00661ed917-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.123 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.123 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.123 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.123 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.123 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:01:27.123 DEBUG:boto:Data: 2024-04-16T20:01:27.123 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.123 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.124 DEBUG:boto:Params: {} 2024-04-16T20:01:27.124 DEBUG:boto:Token: None 2024-04-16T20:01:27.124 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85 2024-04-16T20:01:27.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8RwOUb153ARnOpwyP1+Fw0qUvZg= 2024-04-16T20:01:27.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8RwOUb153ARnOpwyP1+Fw0qUvZg=', 'Content-Length': '0'} 2024-04-16T20:01:27.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af72e1d380c742ca-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.126 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.126 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.126 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.126 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:01:27.126 DEBUG:boto:Data: 2024-04-16T20:01:27.126 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.127 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.127 DEBUG:boto:Params: {} 2024-04-16T20:01:27.127 DEBUG:boto:Token: None 2024-04-16T20:01:27.127 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85 2024-04-16T20:01:27.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8RwOUb153ARnOpwyP1+Fw0qUvZg= 2024-04-16T20:01:27.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8RwOUb153ARnOpwyP1+Fw0qUvZg=', 'Content-Length': '0'} 2024-04-16T20:01:27.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004000378a9c7f015-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:01:27.129 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.129 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.129 DEBUG:boto:Method: GET 2024-04-16T20:01:27.129 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:01:27.130 DEBUG:boto:Data: 2024-04-16T20:01:27.130 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.130 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.130 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.130 DEBUG:boto:Params: {} 2024-04-16T20:01:27.130 DEBUG:boto:Token: None 2024-04-16T20:01:27.130 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85 2024-04-16T20:01:27.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ghWfrfEDAJb/SUDWDmAOdcLpOL4= 2024-04-16T20:01:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ghWfrfEDAJb/SUDWDmAOdcLpOL4=', 'Content-Length': '0'} 2024-04-16T20:01:27.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f130e17d3b883050-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.174 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:01:27.174 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:01:27.174 DEBUG:boto:Method: GET 2024-04-16T20:01:27.175 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:01:27.175 DEBUG:boto:Data: 2024-04-16T20:01:27.175 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.175 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.175 DEBUG:boto:Params: {} 2024-04-16T20:01:27.175 DEBUG:boto:Token: None 2024-04-16T20:01:27.175 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj85 2024-04-16T20:01:27.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ghWfrfEDAJb/SUDWDmAOdcLpOL4= 2024-04-16T20:01:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ghWfrfEDAJb/SUDWDmAOdcLpOL4=', 'Content-Length': '0'} 2024-04-16T20:01:27.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b29e7886b997d1f-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.177 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.177 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.178 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.178 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.178 DEBUG:boto:Method: GET 2024-04-16T20:01:27.178 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.178 DEBUG:boto:Data: 2024-04-16T20:01:27.178 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.178 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.178 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.178 DEBUG:boto:Params: {} 2024-04-16T20:01:27.178 DEBUG:boto:Token: None 2024-04-16T20:01:27.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ccfd81a0faa144e-00661ed917-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.181 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.181 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.181 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.182 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.182 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.182 DEBUG:boto:Method: GET 2024-04-16T20:01:27.182 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:01:27.182 DEBUG:boto:Data: 2024-04-16T20:01:27.182 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.182 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.182 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.182 DEBUG:boto:Params: {} 2024-04-16T20:01:27.182 DEBUG:boto:Token: None 2024-04-16T20:01:27.182 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.182 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fec3132377c7c9-00661ed917-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.185 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:01:27.186 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.186 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.186 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.186 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.186 DEBUG:boto:Method: GET 2024-04-16T20:01:27.186 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.186 DEBUG:boto:Data: 2024-04-16T20:01:27.186 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.186 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.187 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.187 DEBUG:boto:Params: {} 2024-04-16T20:01:27.187 DEBUG:boto:Token: None 2024-04-16T20:01:27.187 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark= 2024-04-16T20:01:27.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark=', 'Content-Length': '0'} 2024-04-16T20:01:27.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f126dfa699c6206-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.230 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.230 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.231 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.231 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.231 DEBUG:boto:Method: GET 2024-04-16T20:01:27.231 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.231 DEBUG:boto:Data: 2024-04-16T20:01:27.231 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.231 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.231 DEBUG:boto:Params: {} 2024-04-16T20:01:27.231 DEBUG:boto:Token: None 2024-04-16T20:01:27.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark= 2024-04-16T20:01:27.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark=', 'Content-Length': '0'} 2024-04-16T20:01:27.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc88021b9867beab-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.234 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.234 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.234 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.234 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.234 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.235 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.235 DEBUG:boto:Data: 2024-04-16T20:01:27.235 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.235 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.235 DEBUG:boto:Params: {} 2024-04-16T20:01:27.235 DEBUG:boto:Token: None 2024-04-16T20:01:27.235 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UskuzPss54cwDL+hYSalSmgfDaU= 2024-04-16T20:01:27.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UskuzPss54cwDL+hYSalSmgfDaU=', 'Content-Length': '0'} 2024-04-16T20:01:27.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004858cb2c59a38f85-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.238 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.238 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.238 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.238 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.238 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.238 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.238 DEBUG:boto:Data: 2024-04-16T20:01:27.238 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.238 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.238 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.239 DEBUG:boto:Params: {} 2024-04-16T20:01:27.239 DEBUG:boto:Token: None 2024-04-16T20:01:27.239 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UskuzPss54cwDL+hYSalSmgfDaU= 2024-04-16T20:01:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UskuzPss54cwDL+hYSalSmgfDaU=', 'Content-Length': '0'} 2024-04-16T20:01:27.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef5d5bf2770d7342-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:01:27.241 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.241 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.241 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.241 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.241 DEBUG:boto:Method: GET 2024-04-16T20:01:27.241 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.242 DEBUG:boto:Data: 2024-04-16T20:01:27.242 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.242 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.242 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.242 DEBUG:boto:Params: {} 2024-04-16T20:01:27.242 DEBUG:boto:Token: None 2024-04-16T20:01:27.242 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark= 2024-04-16T20:01:27.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark=', 'Content-Length': '0'} 2024-04-16T20:01:27.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e222fb5944bfb1c-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.286 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.286 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.287 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.287 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.287 DEBUG:boto:Method: GET 2024-04-16T20:01:27.287 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.287 DEBUG:boto:Data: 2024-04-16T20:01:27.287 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.287 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.287 DEBUG:boto:Params: {} 2024-04-16T20:01:27.287 DEBUG:boto:Token: None 2024-04-16T20:01:27.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:01:27.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark= 2024-04-16T20:01:27.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVXfE9WKGweuH30mD+BRrwSpark=', 'Content-Length': '0'} 2024-04-16T20:01:27.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a665c78eae8c528f-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.290 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.290 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.290 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.290 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.290 DEBUG:boto:Method: GET 2024-04-16T20:01:27.290 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.290 DEBUG:boto:Data: 2024-04-16T20:01:27.290 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.291 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.291 DEBUG:boto:Params: {} 2024-04-16T20:01:27.291 DEBUG:boto:Token: None 2024-04-16T20:01:27.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb54eaaf3aa6c4a3-00661ed917-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.294 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.294 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.294 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.294 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.294 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.294 DEBUG:boto:Method: GET 2024-04-16T20:01:27.295 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.295 DEBUG:boto:Data: 2024-04-16T20:01:27.295 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.295 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.295 DEBUG:boto:Params: {} 2024-04-16T20:01:27.295 DEBUG:boto:Token: None 2024-04-16T20:01:27.295 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80d40b4fbf523e4-00661ed917-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.299 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.299 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.299 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.299 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.299 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:01:27.299 DEBUG:boto:Data: 2024-04-16T20:01:27.299 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.300 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.300 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.300 DEBUG:boto:Params: {} 2024-04-16T20:01:27.300 DEBUG:boto:Token: None 2024-04-16T20:01:27.300 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95 2024-04-16T20:01:27.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:57HVT8GPP/SxMxq47ace6D1vdFE= 2024-04-16T20:01:27.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:57HVT8GPP/SxMxq47ace6D1vdFE=', 'Content-Length': '0'} 2024-04-16T20:01:27.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35805e9227b599f-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.302 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.302 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.302 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.302 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:01:27.303 DEBUG:boto:Data: 2024-04-16T20:01:27.303 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.303 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.303 DEBUG:boto:Params: {} 2024-04-16T20:01:27.303 DEBUG:boto:Token: None 2024-04-16T20:01:27.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95 2024-04-16T20:01:27.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:57HVT8GPP/SxMxq47ace6D1vdFE= 2024-04-16T20:01:27.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:57HVT8GPP/SxMxq47ace6D1vdFE=', 'Content-Length': '0'} 2024-04-16T20:01:27.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e33916e3e7abf6-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:01:27.305 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.305 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.306 DEBUG:boto:Method: GET 2024-04-16T20:01:27.306 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:01:27.306 DEBUG:boto:Data: 2024-04-16T20:01:27.306 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.306 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.306 DEBUG:boto:Params: {} 2024-04-16T20:01:27.306 DEBUG:boto:Token: None 2024-04-16T20:01:27.306 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95 2024-04-16T20:01:27.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Q0oawUucm4bvVCbE98CBpBHkEs= 2024-04-16T20:01:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Q0oawUucm4bvVCbE98CBpBHkEs=', 'Content-Length': '0'} 2024-04-16T20:01:27.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c519d31f321c85-00661ed917-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.350 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:01:27.350 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:01:27.350 DEBUG:boto:Method: GET 2024-04-16T20:01:27.351 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:01:27.351 DEBUG:boto:Data: 2024-04-16T20:01:27.351 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.351 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.351 DEBUG:boto:Params: {} 2024-04-16T20:01:27.351 DEBUG:boto:Token: None 2024-04-16T20:01:27.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj95 2024-04-16T20:01:27.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Q0oawUucm4bvVCbE98CBpBHkEs= 2024-04-16T20:01:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Q0oawUucm4bvVCbE98CBpBHkEs=', 'Content-Length': '0'} 2024-04-16T20:01:27.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e875e0cd843a0c9-00661ed917-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.353 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.354 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.354 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.354 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.354 DEBUG:boto:Method: GET 2024-04-16T20:01:27.354 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.354 DEBUG:boto:Data: 2024-04-16T20:01:27.354 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:01:27.354 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.354 DEBUG:boto:Params: {} 2024-04-16T20:01:27.355 DEBUG:boto:Token: None 2024-04-16T20:01:27.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60e965d372c6986-00661ed917-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.357 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.358 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.358 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.358 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.358 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.358 DEBUG:boto:Method: GET 2024-04-16T20:01:27.358 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:01:27.358 DEBUG:boto:Data: 2024-04-16T20:01:27.358 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.358 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.358 DEBUG:boto:Params: {} 2024-04-16T20:01:27.359 DEBUG:boto:Token: None 2024-04-16T20:01:27.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw= 2024-04-16T20:01:27.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AeZ4qrSFQVihTJVFOmV6Mf0/tUw=', 'Content-Length': '0'} 2024-04-16T20:01:27.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0ce336e1feeea57-00661ed917-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.362 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:01:27.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:01:27.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:01:27.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:01:27.363 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:01:27.420 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:01:27.421 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:01:27.491 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:01:27.491 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T20:01:27.500 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 15 2024-04-16T20:01:27.560 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.561 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.561 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.561 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:27.561 DEBUG:boto:Data: 2024-04-16T20:01:27.561 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.561 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.561 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.561 DEBUG:boto:Params: {} 2024-04-16T20:01:27.561 DEBUG:boto:Token: None 2024-04-16T20:01:27.561 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.562 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk= 2024-04-16T20:01:27.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk=', 'Content-Length': '0'} 2024-04-16T20:01:27.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000911471ef6638de16-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.565 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:01:27.565 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:01:27.565 DEBUG:boto:Method: PUT 2024-04-16T20:01:27.565 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:01:27.565 DEBUG:boto:Data: 2024-04-16T20:01:27.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:01:27.565 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.565 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.566 DEBUG:boto:Params: {} 2024-04-16T20:01:27.566 DEBUG:boto:Token: None 2024-04-16T20:01:27.566 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj23 2024-04-16T20:01:27.566 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sFUcamGJvTuxsCN3I6qoem9usC0= 2024-04-16T20:01:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFUcamGJvTuxsCN3I6qoem9usC0='} 2024-04-16T20:01:27.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc1d1c4805e67714-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.573 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.573 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.573 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.573 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:27.573 DEBUG:boto:Data: 2024-04-16T20:01:27.573 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.573 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.573 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.574 DEBUG:boto:Params: {} 2024-04-16T20:01:27.574 DEBUG:boto:Token: None 2024-04-16T20:01:27.574 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk= 2024-04-16T20:01:27.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk=', 'Content-Length': '0'} 2024-04-16T20:01:27.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d53911625667d92c-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.577 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:01:27.577 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:01:27.577 DEBUG:boto:Method: PUT 2024-04-16T20:01:27.577 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:01:27.577 DEBUG:boto:Data: 2024-04-16T20:01:27.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:01:27.577 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.577 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.577 DEBUG:boto:Params: {} 2024-04-16T20:01:27.577 DEBUG:boto:Token: None 2024-04-16T20:01:27.578 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj46 2024-04-16T20:01:27.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mDVJpvVoC1dtqVOfd2XGsDLllyQ= 2024-04-16T20:01:27.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mDVJpvVoC1dtqVOfd2XGsDLllyQ='} 2024-04-16T20:01:27.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000561c9ff80a23849c-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.583 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.583 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.583 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.584 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:27.584 DEBUG:boto:Data: 2024-04-16T20:01:27.584 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.584 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.584 DEBUG:boto:Params: {} 2024-04-16T20:01:27.584 DEBUG:boto:Token: None 2024-04-16T20:01:27.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk= 2024-04-16T20:01:27.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk=', 'Content-Length': '0'} 2024-04-16T20:01:27.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924429e92928b9ec-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.587 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:01:27.587 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:01:27.587 DEBUG:boto:Method: PUT 2024-04-16T20:01:27.587 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:01:27.587 DEBUG:boto:Data: 2024-04-16T20:01:27.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:01:27.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.587 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.588 DEBUG:boto:Params: {} 2024-04-16T20:01:27.588 DEBUG:boto:Token: None 2024-04-16T20:01:27.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj69 2024-04-16T20:01:27.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CDQgkMt3FYqINTctoMT2Sk6V7N8= 2024-04-16T20:01:27.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CDQgkMt3FYqINTctoMT2Sk6V7N8='} 2024-04-16T20:01:27.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b39a21fb49f44a9d-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.630 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.631 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.631 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.631 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:27.631 DEBUG:boto:Data: 2024-04-16T20:01:27.631 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.631 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.631 DEBUG:boto:Params: {} 2024-04-16T20:01:27.631 DEBUG:boto:Token: None 2024-04-16T20:01:27.631 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk= 2024-04-16T20:01:27.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk=', 'Content-Length': '0'} 2024-04-16T20:01:27.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbcba1b97ca179f0-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.635 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:01:27.635 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:01:27.635 DEBUG:boto:Method: PUT 2024-04-16T20:01:27.635 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:01:27.635 DEBUG:boto:Data: 2024-04-16T20:01:27.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:01:27.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.635 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.635 DEBUG:boto:Params: {} 2024-04-16T20:01:27.635 DEBUG:boto:Token: None 2024-04-16T20:01:27.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj92 2024-04-16T20:01:27.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyjh4ZqPEZcO37f0Kup+lt9Kttg= 2024-04-16T20:01:27.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyjh4ZqPEZcO37f0Kup+lt9Kttg='} 2024-04-16T20:01:27.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065f1969d97e9d2d6-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.642 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:01:27.642 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:01:27.642 DEBUG:boto:Method: HEAD 2024-04-16T20:01:27.642 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:01:27.642 DEBUG:boto:Data: 2024-04-16T20:01:27.642 DEBUG:boto:Headers: {} 2024-04-16T20:01:27.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.642 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.642 DEBUG:boto:Params: {} 2024-04-16T20:01:27.643 DEBUG:boto:Token: None 2024-04-16T20:01:27.643 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/ 2024-04-16T20:01:27.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk= 2024-04-16T20:01:27.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0Q7aP2t/q6YiB6Ni3SKXPLIKOwk=', 'Content-Length': '0'} 2024-04-16T20:01:27.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000453c3857d43e5d67-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.645 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:01:27.645 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:01:27.645 DEBUG:boto:Method: PUT 2024-04-16T20:01:27.646 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:01:27.646 DEBUG:boto:Data: 2024-04-16T20:01:27.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:01:27.646 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:01:27.646 DEBUG:boto:Port: 8000 2024-04-16T20:01:27.646 DEBUG:boto:Params: {} 2024-04-16T20:01:27.646 DEBUG:boto:Token: None 2024-04-16T20:01:27.646 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:01:27 GMT /yrjefh-52/obj115 2024-04-16T20:01:27.646 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fmbo7iK/OQ/w/0hcGIsgRpRLraM= 2024-04-16T20:01:27.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:01:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fmbo7iK/OQ/w/0hcGIsgRpRLraM='} 2024-04-16T20:01:27.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004258f53db336a6ed-00661ed917-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:01:27.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:01:27.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:01:27.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:01:27.691 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:01:27.754 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:01:27.754 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:01:27.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:27.818+0000 7f1987213800 1 waiting to reach latest gen 3, on 2.. 2024-04-16T20:01:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:32.821+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:32.817 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:01:32.817 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:01:37.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:37.823+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:37.818 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:01:37.819 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:01:42.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:42.824+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:42.820 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:01:42.820 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:01:47.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:47.826+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:47.822 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:01:47.822 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:01:52.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:52.828+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:52.824 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:01:52.824 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:01:57.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:01:57.829+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:01:57.825 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:01:57.825 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:02.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:02.831+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:02.827 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:02:02.827 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:07.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:07.833+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:07.828 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:02:07.828 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:12.834+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:12.830 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:02:12.830 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:17.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:17.836+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:17.831 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , , , , , ] 2024-04-16T20:02:17.832 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:22.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:22.837+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:22.833 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:22.833 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:27.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:27.839+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:27.834 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:27.834 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:32.841+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:32.836 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:32.836 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:37.841+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:37.838 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:37.838 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:42.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:42.844+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:42.839 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:42.839 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:47.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:47.846+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:47.841 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:47.841 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:52.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:52.847+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:52.842 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:52.842 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:57.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:02:57.849+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:02:57.844 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:02:57.844 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:02.850+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:02.845 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:02.846 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:07.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:07.852+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:07.847 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:07.847 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:12.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:12.853+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:12.848 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:12.848 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:17.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:17.854+0000 7f1987213800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:17.850 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , , , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:17.850 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:22.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:22.857+0000 7f1987213800 1 bucket sync caught up with source: 2024-04-16T20:03:22.852 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:22.852 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.77.6, , , 00000000002.163.6, , 00000000002.64.6, , , , , , 00000000001.96.6, , 00000000001.165.6, , , , ] 2024-04-16T20:03:22.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:22.857+0000 7f1987213800 0 bucket checkpoint complete 2024-04-16T20:03:22.879 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:03:22.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:03:22.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:03:22.880 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:03:22.942 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:03:22.943 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:03:23.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:03:23.010+0000 7f311d728800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:23.005 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000002.271.6, 00000000002.213.6, , , 00000000002.165.6, 00000000005.215.6, 00000000003.272.6] 2024-04-16T20:03:23.005 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000002.271.6, 00000000002.213.6, , , 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:03:28.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:03:28.012+0000 7f311d728800 1 bucket sync caught up with source: 2024-04-16T20:03:28.006 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000002.271.6, 00000000002.213.6, , , 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:03:28.007 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000002.271.6, 00000000002.213.6, , , 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:03:28.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:03:28.012+0000 7f311d728800 0 bucket checkpoint complete 2024-04-16T20:03:28.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:03:28.034 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.034 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.035 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.035 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:28.035 DEBUG:boto:Data: 2024-04-16T20:03:28.035 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.035 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.035 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.035 DEBUG:boto:Params: {} 2024-04-16T20:03:28.036 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:03:28.036 DEBUG:boto:Token: None 2024-04-16T20:03:28.036 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BQVHfWoQr1GeYl/29C04lAT34Rc= 2024-04-16T20:03:28.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BQVHfWoQr1GeYl/29C04lAT34Rc=', 'Content-Length': '0'} 2024-04-16T20:03:28.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089339b4911639660-00661ed990-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.042 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.042 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.042 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.042 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:28.042 DEBUG:boto:Data: 2024-04-16T20:03:28.042 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.042 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.043 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.043 DEBUG:boto:Params: {} 2024-04-16T20:03:28.043 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:03:28.043 DEBUG:boto:Token: None 2024-04-16T20:03:28.043 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BQVHfWoQr1GeYl/29C04lAT34Rc= 2024-04-16T20:03:28.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BQVHfWoQr1GeYl/29C04lAT34Rc=', 'Content-Length': '0'} 2024-04-16T20:03:28.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093723c9a940af6f1-00661ed990-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:03:28.048 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.048 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.048 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.048 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.048 DEBUG:boto:Method: GET 2024-04-16T20:03:28.049 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.049 DEBUG:boto:Data: 2024-04-16T20:03:28.049 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.049 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.049 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.049 DEBUG:boto:Params: {} 2024-04-16T20:03:28.049 DEBUG:boto:Token: None 2024-04-16T20:03:28.049 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/?versions 2024-04-16T20:03:28.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI= 2024-04-16T20:03:28.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI=', 'Content-Length': '0'} 2024-04-16T20:03:28.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006986faf493715ca9-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.052 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:03:28.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:03:28.055 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:03:28.056 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.056 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.056 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.056 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.056 DEBUG:boto:Method: GET 2024-04-16T20:03:28.056 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.056 DEBUG:boto:Data: 2024-04-16T20:03:28.056 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.056 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.056 DEBUG:boto:Params: {} 2024-04-16T20:03:28.057 DEBUG:boto:Token: None 2024-04-16T20:03:28.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/?versions 2024-04-16T20:03:28.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI= 2024-04-16T20:03:28.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI=', 'Content-Length': '0'} 2024-04-16T20:03:28.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dee08546d11d50a0-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.060 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:03:28.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:03:28.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:03:28.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:03:28.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:03:28.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:03:28.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:03:28.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:03:28.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:03:28.063 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.063 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.063 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.063 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.063 DEBUG:boto:Method: GET 2024-04-16T20:03:28.063 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.064 DEBUG:boto:Data: 2024-04-16T20:03:28.064 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.064 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.064 DEBUG:boto:Params: {} 2024-04-16T20:03:28.064 DEBUG:boto:Token: None 2024-04-16T20:03:28.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/?versions 2024-04-16T20:03:28.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI= 2024-04-16T20:03:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI=', 'Content-Length': '0'} 2024-04-16T20:03:28.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000687f472756388f2-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.067 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.068 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.068 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.068 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.069 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.069 DEBUG:boto:Method: GET 2024-04-16T20:03:28.069 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:03:28.069 DEBUG:boto:Data: 2024-04-16T20:03:28.069 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.069 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.069 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.069 DEBUG:boto:Params: {} 2024-04-16T20:03:28.069 DEBUG:boto:Token: None 2024-04-16T20:03:28.069 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/?versions 2024-04-16T20:03:28.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI= 2024-04-16T20:03:28.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eg9xBMuqeW7LF6nT8bRVfr9raxI=', 'Content-Length': '0'} 2024-04-16T20:03:28.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7aa3fb895dc811-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.073 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:03:28.074 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.074 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.074 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.074 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.074 DEBUG:boto:Method: GET 2024-04-16T20:03:28.075 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.075 DEBUG:boto:Data: 2024-04-16T20:03:28.075 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.075 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.075 DEBUG:boto:Params: {} 2024-04-16T20:03:28.075 DEBUG:boto:Token: None 2024-04-16T20:03:28.075 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U= 2024-04-16T20:03:28.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U=', 'Content-Length': '0'} 2024-04-16T20:03:28.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9a414053fea2f9-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.078 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.078 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.078 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.078 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.078 DEBUG:boto:Method: GET 2024-04-16T20:03:28.078 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.079 DEBUG:boto:Data: 2024-04-16T20:03:28.079 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.079 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.079 DEBUG:boto:Params: {} 2024-04-16T20:03:28.079 DEBUG:boto:Token: None 2024-04-16T20:03:28.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U= 2024-04-16T20:03:28.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U=', 'Content-Length': '0'} 2024-04-16T20:03:28.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367d943d0328cbd2-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.122 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.122 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.123 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.123 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.123 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.123 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.123 DEBUG:boto:Data: 2024-04-16T20:03:28.123 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.123 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.123 DEBUG:boto:Params: {} 2024-04-16T20:03:28.123 DEBUG:boto:Token: None 2024-04-16T20:03:28.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KKQnUWMcVe62OEJhWPyJaktaNyY= 2024-04-16T20:03:28.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KKQnUWMcVe62OEJhWPyJaktaNyY=', 'Content-Length': '0'} 2024-04-16T20:03:28.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e16131fee2b16861-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.126 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.126 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.126 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.126 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.126 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.126 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.126 DEBUG:boto:Data: 2024-04-16T20:03:28.126 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.126 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.127 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.127 DEBUG:boto:Params: {} 2024-04-16T20:03:28.127 DEBUG:boto:Token: None 2024-04-16T20:03:28.127 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KKQnUWMcVe62OEJhWPyJaktaNyY= 2024-04-16T20:03:28.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KKQnUWMcVe62OEJhWPyJaktaNyY=', 'Content-Length': '0'} 2024-04-16T20:03:28.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ad6afb35e31d54-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:03:28.129 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.129 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.129 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.129 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.130 DEBUG:boto:Method: GET 2024-04-16T20:03:28.130 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.130 DEBUG:boto:Data: 2024-04-16T20:03:28.130 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.130 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.130 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.130 DEBUG:boto:Params: {} 2024-04-16T20:03:28.130 DEBUG:boto:Token: None 2024-04-16T20:03:28.130 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.130 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U= 2024-04-16T20:03:28.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U=', 'Content-Length': '0'} 2024-04-16T20:03:28.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adead2fb6fdfd16-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.174 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.174 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.175 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.175 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.175 DEBUG:boto:Method: GET 2024-04-16T20:03:28.175 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.175 DEBUG:boto:Data: 2024-04-16T20:03:28.175 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.175 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.175 DEBUG:boto:Params: {} 2024-04-16T20:03:28.175 DEBUG:boto:Token: None 2024-04-16T20:03:28.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:03:28.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U= 2024-04-16T20:03:28.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Tl19yHGQoTXd7vKMKVdftBWOs7U=', 'Content-Length': '0'} 2024-04-16T20:03:28.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1fb46c1a7dacfe5-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.178 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.178 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.178 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.178 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.178 DEBUG:boto:Method: GET 2024-04-16T20:03:28.178 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.178 DEBUG:boto:Data: 2024-04-16T20:03:28.178 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.179 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.179 DEBUG:boto:Params: {} 2024-04-16T20:03:28.179 DEBUG:boto:Token: None 2024-04-16T20:03:28.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f33da6fedcec5e-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.222 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.223 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.223 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.223 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.223 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.223 DEBUG:boto:Method: GET 2024-04-16T20:03:28.223 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.223 DEBUG:boto:Data: 2024-04-16T20:03:28.223 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.224 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.224 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.224 DEBUG:boto:Params: {} 2024-04-16T20:03:28.224 DEBUG:boto:Token: None 2024-04-16T20:03:28.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904e0c17a6c1e4da-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.227 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.227 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.227 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.228 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.228 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:03:28.228 DEBUG:boto:Data: 2024-04-16T20:03:28.228 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.228 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.228 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.228 DEBUG:boto:Params: {} 2024-04-16T20:03:28.228 DEBUG:boto:Token: None 2024-04-16T20:03:28.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11 2024-04-16T20:03:28.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eoibd4zH9UgPMsPXak2iwChlbYM= 2024-04-16T20:03:28.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eoibd4zH9UgPMsPXak2iwChlbYM=', 'Content-Length': '0'} 2024-04-16T20:03:28.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04b155fa1a7723f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.231 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.231 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.231 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.231 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:03:28.231 DEBUG:boto:Data: 2024-04-16T20:03:28.231 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.231 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.231 DEBUG:boto:Params: {} 2024-04-16T20:03:28.232 DEBUG:boto:Token: None 2024-04-16T20:03:28.232 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11 2024-04-16T20:03:28.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Eoibd4zH9UgPMsPXak2iwChlbYM= 2024-04-16T20:03:28.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Eoibd4zH9UgPMsPXak2iwChlbYM=', 'Content-Length': '0'} 2024-04-16T20:03:28.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073bfee862c8b43b4-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:03:28.234 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.234 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.234 DEBUG:boto:Method: GET 2024-04-16T20:03:28.234 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:03:28.234 DEBUG:boto:Data: 2024-04-16T20:03:28.235 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.235 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.235 DEBUG:boto:Params: {} 2024-04-16T20:03:28.235 DEBUG:boto:Token: None 2024-04-16T20:03:28.235 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11 2024-04-16T20:03:28.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QbpsRk6WnrSEdxe8PqMHOHEsoiY= 2024-04-16T20:03:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QbpsRk6WnrSEdxe8PqMHOHEsoiY=', 'Content-Length': '0'} 2024-04-16T20:03:28.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568b28a7ba0917ee-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.237 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:03:28.238 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:03:28.238 DEBUG:boto:Method: GET 2024-04-16T20:03:28.238 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:03:28.238 DEBUG:boto:Data: 2024-04-16T20:03:28.238 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.238 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.238 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.238 DEBUG:boto:Params: {} 2024-04-16T20:03:28.238 DEBUG:boto:Token: None 2024-04-16T20:03:28.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj11 2024-04-16T20:03:28.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QbpsRk6WnrSEdxe8PqMHOHEsoiY= 2024-04-16T20:03:28.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QbpsRk6WnrSEdxe8PqMHOHEsoiY=', 'Content-Length': '0'} 2024-04-16T20:03:28.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000652f77bb09fefd16-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.241 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.241 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.241 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.241 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.241 DEBUG:boto:Method: GET 2024-04-16T20:03:28.241 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.242 DEBUG:boto:Data: 2024-04-16T20:03:28.242 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.242 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.242 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.242 DEBUG:boto:Params: {} 2024-04-16T20:03:28.242 DEBUG:boto:Token: None 2024-04-16T20:03:28.242 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043ca0e5f7dc4523c-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.245 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.245 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.245 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.245 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.245 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.246 DEBUG:boto:Method: GET 2024-04-16T20:03:28.246 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:03:28.246 DEBUG:boto:Data: 2024-04-16T20:03:28.246 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.246 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.246 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.246 DEBUG:boto:Params: {} 2024-04-16T20:03:28.246 DEBUG:boto:Token: None 2024-04-16T20:03:28.246 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78c28fd478161d5-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.249 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:03:28.250 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.250 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.250 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.250 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.250 DEBUG:boto:Method: GET 2024-04-16T20:03:28.250 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.250 DEBUG:boto:Data: 2024-04-16T20:03:28.251 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.251 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.251 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.251 DEBUG:boto:Params: {} 2024-04-16T20:03:28.251 DEBUG:boto:Token: None 2024-04-16T20:03:28.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw= 2024-04-16T20:03:28.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw=', 'Content-Length': '0'} 2024-04-16T20:03:28.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6181f4b1affc85c-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.254 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.254 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.254 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.254 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.254 DEBUG:boto:Method: GET 2024-04-16T20:03:28.254 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.254 DEBUG:boto:Data: 2024-04-16T20:03:28.255 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.255 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.255 DEBUG:boto:Params: {} 2024-04-16T20:03:28.255 DEBUG:boto:Token: None 2024-04-16T20:03:28.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw= 2024-04-16T20:03:28.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw=', 'Content-Length': '0'} 2024-04-16T20:03:28.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c8b76bf7e51670-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.258 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.258 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.258 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.258 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.258 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.258 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.258 DEBUG:boto:Data: 2024-04-16T20:03:28.259 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.259 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.259 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.259 DEBUG:boto:Params: {} 2024-04-16T20:03:28.259 DEBUG:boto:Token: None 2024-04-16T20:03:28.259 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FORtMJs8DBJiWrqMIrK+O/9woE8= 2024-04-16T20:03:28.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FORtMJs8DBJiWrqMIrK+O/9woE8=', 'Content-Length': '0'} 2024-04-16T20:03:28.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8c50b823ded146-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.262 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.262 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.262 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.262 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.262 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.262 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.262 DEBUG:boto:Data: 2024-04-16T20:03:28.262 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.262 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.262 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.262 DEBUG:boto:Params: {} 2024-04-16T20:03:28.263 DEBUG:boto:Token: None 2024-04-16T20:03:28.263 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FORtMJs8DBJiWrqMIrK+O/9woE8= 2024-04-16T20:03:28.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FORtMJs8DBJiWrqMIrK+O/9woE8=', 'Content-Length': '0'} 2024-04-16T20:03:28.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b65237d6c674197-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:03:28.265 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.265 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.265 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.265 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.266 DEBUG:boto:Method: GET 2024-04-16T20:03:28.266 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.266 DEBUG:boto:Data: 2024-04-16T20:03:28.266 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.266 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.266 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.266 DEBUG:boto:Params: {} 2024-04-16T20:03:28.266 DEBUG:boto:Token: None 2024-04-16T20:03:28.266 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.266 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw= 2024-04-16T20:03:28.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw=', 'Content-Length': '0'} 2024-04-16T20:03:28.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957fd5b621524d7f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.269 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.269 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.269 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.269 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.269 DEBUG:boto:Method: GET 2024-04-16T20:03:28.270 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.270 DEBUG:boto:Data: 2024-04-16T20:03:28.270 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.270 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.270 DEBUG:boto:Params: {} 2024-04-16T20:03:28.270 DEBUG:boto:Token: None 2024-04-16T20:03:28.270 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:03:28.270 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw= 2024-04-16T20:03:28.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yC4PnSqHw8KQ1MMueCh8mFYthcw=', 'Content-Length': '0'} 2024-04-16T20:03:28.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008396d150b6b70549-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.273 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.273 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.273 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.273 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.273 DEBUG:boto:Method: GET 2024-04-16T20:03:28.273 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.273 DEBUG:boto:Data: 2024-04-16T20:03:28.273 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.273 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.273 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.274 DEBUG:boto:Params: {} 2024-04-16T20:03:28.274 DEBUG:boto:Token: None 2024-04-16T20:03:28.274 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ba78ecd1c9ff96-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.277 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.277 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.277 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.277 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.277 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.278 DEBUG:boto:Method: GET 2024-04-16T20:03:28.278 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.278 DEBUG:boto:Data: 2024-04-16T20:03:28.278 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.278 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.278 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.278 DEBUG:boto:Params: {} 2024-04-16T20:03:28.278 DEBUG:boto:Token: None 2024-04-16T20:03:28.278 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.278 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c678cf8734e6aa-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.281 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.282 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.282 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.282 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.282 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:03:28.282 DEBUG:boto:Data: 2024-04-16T20:03:28.282 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.282 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.282 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.283 DEBUG:boto:Params: {} 2024-04-16T20:03:28.283 DEBUG:boto:Token: None 2024-04-16T20:03:28.283 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115 2024-04-16T20:03:28.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IevV/pU25PXKNzsyakLAupxZcq0= 2024-04-16T20:03:28.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IevV/pU25PXKNzsyakLAupxZcq0=', 'Content-Length': '0'} 2024-04-16T20:03:28.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f905f7c189857400-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.285 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.286 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.286 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.286 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:03:28.286 DEBUG:boto:Data: 2024-04-16T20:03:28.286 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.286 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.286 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.286 DEBUG:boto:Params: {} 2024-04-16T20:03:28.286 DEBUG:boto:Token: None 2024-04-16T20:03:28.286 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115 2024-04-16T20:03:28.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IevV/pU25PXKNzsyakLAupxZcq0= 2024-04-16T20:03:28.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IevV/pU25PXKNzsyakLAupxZcq0=', 'Content-Length': '0'} 2024-04-16T20:03:28.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a530455a7374124a-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:03:28.289 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.289 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.289 DEBUG:boto:Method: GET 2024-04-16T20:03:28.289 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:03:28.289 DEBUG:boto:Data: 2024-04-16T20:03:28.289 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.290 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.290 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.290 DEBUG:boto:Params: {} 2024-04-16T20:03:28.290 DEBUG:boto:Token: None 2024-04-16T20:03:28.290 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115 2024-04-16T20:03:28.290 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:taGAty2ZUe74M8N6SFkfYRM2Y4c= 2024-04-16T20:03:28.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:taGAty2ZUe74M8N6SFkfYRM2Y4c=', 'Content-Length': '0'} 2024-04-16T20:03:28.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c367f1b3acbe114f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.334 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:03:28.334 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:03:28.335 DEBUG:boto:Method: GET 2024-04-16T20:03:28.335 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:03:28.335 DEBUG:boto:Data: 2024-04-16T20:03:28.335 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.335 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.335 DEBUG:boto:Params: {} 2024-04-16T20:03:28.335 DEBUG:boto:Token: None 2024-04-16T20:03:28.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj115 2024-04-16T20:03:28.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:taGAty2ZUe74M8N6SFkfYRM2Y4c= 2024-04-16T20:03:28.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:taGAty2ZUe74M8N6SFkfYRM2Y4c=', 'Content-Length': '0'} 2024-04-16T20:03:28.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2835c71726bbd4-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.338 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.338 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.339 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.339 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.339 DEBUG:boto:Method: GET 2024-04-16T20:03:28.339 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.339 DEBUG:boto:Data: 2024-04-16T20:03:28.339 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.339 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.339 DEBUG:boto:Params: {} 2024-04-16T20:03:28.339 DEBUG:boto:Token: None 2024-04-16T20:03:28.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ff7a311762ca90-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.382 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.383 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.383 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.383 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.383 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.383 DEBUG:boto:Method: GET 2024-04-16T20:03:28.383 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:03:28.383 DEBUG:boto:Data: 2024-04-16T20:03:28.383 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.384 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.384 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.384 DEBUG:boto:Params: {} 2024-04-16T20:03:28.384 DEBUG:boto:Token: None 2024-04-16T20:03:28.384 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b4eaaf3c677052-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.388 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:03:28.388 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.388 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.388 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.388 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.388 DEBUG:boto:Method: GET 2024-04-16T20:03:28.389 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.389 DEBUG:boto:Data: 2024-04-16T20:03:28.389 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.389 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.389 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.389 DEBUG:boto:Params: {} 2024-04-16T20:03:28.389 DEBUG:boto:Token: None 2024-04-16T20:03:28.389 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0= 2024-04-16T20:03:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0=', 'Content-Length': '0'} 2024-04-16T20:03:28.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001868736a814136f8-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.434 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.435 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.435 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.435 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.435 DEBUG:boto:Method: GET 2024-04-16T20:03:28.435 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.435 DEBUG:boto:Data: 2024-04-16T20:03:28.435 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.435 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.435 DEBUG:boto:Params: {} 2024-04-16T20:03:28.435 DEBUG:boto:Token: None 2024-04-16T20:03:28.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0= 2024-04-16T20:03:28.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0=', 'Content-Length': '0'} 2024-04-16T20:03:28.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db788e85a1cb3658-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.438 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.438 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.439 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.439 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.439 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.439 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.439 DEBUG:boto:Data: 2024-04-16T20:03:28.439 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.439 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.439 DEBUG:boto:Params: {} 2024-04-16T20:03:28.439 DEBUG:boto:Token: None 2024-04-16T20:03:28.440 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H9Sp/sCQqelty2XenyMmMKa+BJk= 2024-04-16T20:03:28.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H9Sp/sCQqelty2XenyMmMKa+BJk=', 'Content-Length': '0'} 2024-04-16T20:03:28.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9d8d201ac4d80f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.442 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.442 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.442 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.442 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.442 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.442 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.442 DEBUG:boto:Data: 2024-04-16T20:03:28.443 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.443 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.443 DEBUG:boto:Params: {} 2024-04-16T20:03:28.443 DEBUG:boto:Token: None 2024-04-16T20:03:28.443 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H9Sp/sCQqelty2XenyMmMKa+BJk= 2024-04-16T20:03:28.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H9Sp/sCQqelty2XenyMmMKa+BJk=', 'Content-Length': '0'} 2024-04-16T20:03:28.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce2b484dbeaa9e9-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:03:28.445 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.446 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.446 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.446 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.446 DEBUG:boto:Method: GET 2024-04-16T20:03:28.446 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.446 DEBUG:boto:Data: 2024-04-16T20:03:28.446 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.446 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.446 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.446 DEBUG:boto:Params: {} 2024-04-16T20:03:28.446 DEBUG:boto:Token: None 2024-04-16T20:03:28.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0= 2024-04-16T20:03:28.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0=', 'Content-Length': '0'} 2024-04-16T20:03:28.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e3b2274bfed1cb-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.490 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.491 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.491 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.491 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.491 DEBUG:boto:Method: GET 2024-04-16T20:03:28.491 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.491 DEBUG:boto:Data: 2024-04-16T20:03:28.491 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.491 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.491 DEBUG:boto:Params: {} 2024-04-16T20:03:28.491 DEBUG:boto:Token: None 2024-04-16T20:03:28.492 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:03:28.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0= 2024-04-16T20:03:28.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFJV5U+0MVFrguorqzq7nG9jSf0=', 'Content-Length': '0'} 2024-04-16T20:03:28.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f3ffdd4079c8d6-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.494 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.494 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.494 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.494 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.494 DEBUG:boto:Method: GET 2024-04-16T20:03:28.494 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.494 DEBUG:boto:Data: 2024-04-16T20:03:28.494 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.495 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.495 DEBUG:boto:Params: {} 2024-04-16T20:03:28.495 DEBUG:boto:Token: None 2024-04-16T20:03:28.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd5250ce59624e6-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.538 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.539 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.539 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.539 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.539 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.539 DEBUG:boto:Method: GET 2024-04-16T20:03:28.539 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.539 DEBUG:boto:Data: 2024-04-16T20:03:28.539 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.540 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.540 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.540 DEBUG:boto:Params: {} 2024-04-16T20:03:28.540 DEBUG:boto:Token: None 2024-04-16T20:03:28.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab95d26f44342b4-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.543 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.543 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.544 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.544 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.544 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:03:28.544 DEBUG:boto:Data: 2024-04-16T20:03:28.544 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.544 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.544 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.544 DEBUG:boto:Params: {} 2024-04-16T20:03:28.544 DEBUG:boto:Token: None 2024-04-16T20:03:28.544 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17 2024-04-16T20:03:28.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hqF7D2buueiC+WjfZ/CYjMzkcuY= 2024-04-16T20:03:28.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hqF7D2buueiC+WjfZ/CYjMzkcuY=', 'Content-Length': '0'} 2024-04-16T20:03:28.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e7872be318f0e3-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.547 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.547 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.547 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.547 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:03:28.547 DEBUG:boto:Data: 2024-04-16T20:03:28.547 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.547 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.547 DEBUG:boto:Params: {} 2024-04-16T20:03:28.548 DEBUG:boto:Token: None 2024-04-16T20:03:28.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17 2024-04-16T20:03:28.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hqF7D2buueiC+WjfZ/CYjMzkcuY= 2024-04-16T20:03:28.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hqF7D2buueiC+WjfZ/CYjMzkcuY=', 'Content-Length': '0'} 2024-04-16T20:03:28.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cecfeb2e2d7fa88-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:03:28.550 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.550 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.550 DEBUG:boto:Method: GET 2024-04-16T20:03:28.550 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:03:28.550 DEBUG:boto:Data: 2024-04-16T20:03:28.550 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.551 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.551 DEBUG:boto:Params: {} 2024-04-16T20:03:28.551 DEBUG:boto:Token: None 2024-04-16T20:03:28.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17 2024-04-16T20:03:28.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NgwwTyDZ//YAKCLSj8iO3gSkoJQ= 2024-04-16T20:03:28.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NgwwTyDZ//YAKCLSj8iO3gSkoJQ=', 'Content-Length': '0'} 2024-04-16T20:03:28.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da22c67cd9a2c75f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.594 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:03:28.595 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:03:28.595 DEBUG:boto:Method: GET 2024-04-16T20:03:28.595 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:03:28.595 DEBUG:boto:Data: 2024-04-16T20:03:28.595 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.595 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.595 DEBUG:boto:Params: {} 2024-04-16T20:03:28.595 DEBUG:boto:Token: None 2024-04-16T20:03:28.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj17 2024-04-16T20:03:28.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NgwwTyDZ//YAKCLSj8iO3gSkoJQ= 2024-04-16T20:03:28.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NgwwTyDZ//YAKCLSj8iO3gSkoJQ=', 'Content-Length': '0'} 2024-04-16T20:03:28.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45f045942b4db2b-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.598 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.598 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.598 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.598 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.598 DEBUG:boto:Method: GET 2024-04-16T20:03:28.599 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.599 DEBUG:boto:Data: 2024-04-16T20:03:28.599 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.599 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.599 DEBUG:boto:Params: {} 2024-04-16T20:03:28.599 DEBUG:boto:Token: None 2024-04-16T20:03:28.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325f38d3c99ba03f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.642 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.643 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.643 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.643 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.643 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.643 DEBUG:boto:Method: GET 2024-04-16T20:03:28.643 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:03:28.643 DEBUG:boto:Data: 2024-04-16T20:03:28.643 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.643 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.644 DEBUG:boto:Params: {} 2024-04-16T20:03:28.644 DEBUG:boto:Token: None 2024-04-16T20:03:28.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b8e4d7448c55f17-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.647 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:03:28.648 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.648 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.648 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.648 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.648 DEBUG:boto:Method: GET 2024-04-16T20:03:28.648 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.648 DEBUG:boto:Data: 2024-04-16T20:03:28.648 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.648 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.648 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.648 DEBUG:boto:Params: {} 2024-04-16T20:03:28.649 DEBUG:boto:Token: None 2024-04-16T20:03:28.649 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4= 2024-04-16T20:03:28.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4=', 'Content-Length': '0'} 2024-04-16T20:03:28.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c68066c715bbfa6-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.694 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.695 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.695 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.695 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.695 DEBUG:boto:Method: GET 2024-04-16T20:03:28.695 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.695 DEBUG:boto:Data: 2024-04-16T20:03:28.695 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.695 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.695 DEBUG:boto:Params: {} 2024-04-16T20:03:28.695 DEBUG:boto:Token: None 2024-04-16T20:03:28.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4= 2024-04-16T20:03:28.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4=', 'Content-Length': '0'} 2024-04-16T20:03:28.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008486ba08c026736f-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.698 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.699 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.699 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.699 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.699 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.699 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.699 DEBUG:boto:Data: 2024-04-16T20:03:28.699 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.699 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.699 DEBUG:boto:Params: {} 2024-04-16T20:03:28.699 DEBUG:boto:Token: None 2024-04-16T20:03:28.700 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KtVZE9/TTWz3l5xhiZfR2svxqs8= 2024-04-16T20:03:28.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KtVZE9/TTWz3l5xhiZfR2svxqs8=', 'Content-Length': '0'} 2024-04-16T20:03:28.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dbca637c19724c0-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.702 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.702 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.703 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.703 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.703 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.703 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.703 DEBUG:boto:Data: 2024-04-16T20:03:28.703 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.703 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.703 DEBUG:boto:Params: {} 2024-04-16T20:03:28.703 DEBUG:boto:Token: None 2024-04-16T20:03:28.703 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KtVZE9/TTWz3l5xhiZfR2svxqs8= 2024-04-16T20:03:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KtVZE9/TTWz3l5xhiZfR2svxqs8=', 'Content-Length': '0'} 2024-04-16T20:03:28.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98fa66805b113c5-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:03:28.706 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.706 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.706 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.706 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.706 DEBUG:boto:Method: GET 2024-04-16T20:03:28.706 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.706 DEBUG:boto:Data: 2024-04-16T20:03:28.707 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.707 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.707 DEBUG:boto:Params: {} 2024-04-16T20:03:28.707 DEBUG:boto:Token: None 2024-04-16T20:03:28.707 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.707 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4= 2024-04-16T20:03:28.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4=', 'Content-Length': '0'} 2024-04-16T20:03:28.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003591281e437dee8f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.750 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.750 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.751 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.751 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.751 DEBUG:boto:Method: GET 2024-04-16T20:03:28.751 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.751 DEBUG:boto:Data: 2024-04-16T20:03:28.751 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.751 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.751 DEBUG:boto:Params: {} 2024-04-16T20:03:28.751 DEBUG:boto:Token: None 2024-04-16T20:03:28.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:03:28.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4= 2024-04-16T20:03:28.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OFwbcSqmXAIo6yfAsenn9OftfE4=', 'Content-Length': '0'} 2024-04-16T20:03:28.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ab9fd38268d73e-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.754 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.754 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.754 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.754 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.754 DEBUG:boto:Method: GET 2024-04-16T20:03:28.754 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.754 DEBUG:boto:Data: 2024-04-16T20:03:28.755 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.755 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.755 DEBUG:boto:Params: {} 2024-04-16T20:03:28.755 DEBUG:boto:Token: None 2024-04-16T20:03:28.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003742e24db3241087-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.798 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.799 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.799 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.799 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.799 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.799 DEBUG:boto:Method: GET 2024-04-16T20:03:28.799 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.799 DEBUG:boto:Data: 2024-04-16T20:03:28.799 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.799 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.799 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.799 DEBUG:boto:Params: {} 2024-04-16T20:03:28.800 DEBUG:boto:Token: None 2024-04-16T20:03:28.800 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005684a2c761a48382-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.803 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.803 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.803 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.803 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.803 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:03:28.803 DEBUG:boto:Data: 2024-04-16T20:03:28.804 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.804 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.804 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.804 DEBUG:boto:Params: {} 2024-04-16T20:03:28.804 DEBUG:boto:Token: None 2024-04-16T20:03:28.804 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19 2024-04-16T20:03:28.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7g82Cu2iASLAtrfZypsnt6x6rb8= 2024-04-16T20:03:28.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7g82Cu2iASLAtrfZypsnt6x6rb8=', 'Content-Length': '0'} 2024-04-16T20:03:28.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e223d859b7e21d-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.806 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.807 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.807 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.807 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:03:28.807 DEBUG:boto:Data: 2024-04-16T20:03:28.807 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.807 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.807 DEBUG:boto:Params: {} 2024-04-16T20:03:28.807 DEBUG:boto:Token: None 2024-04-16T20:03:28.807 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19 2024-04-16T20:03:28.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7g82Cu2iASLAtrfZypsnt6x6rb8= 2024-04-16T20:03:28.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7g82Cu2iASLAtrfZypsnt6x6rb8=', 'Content-Length': '0'} 2024-04-16T20:03:28.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c18beb2c8e13fa-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:03:28.810 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.810 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.810 DEBUG:boto:Method: GET 2024-04-16T20:03:28.810 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:03:28.810 DEBUG:boto:Data: 2024-04-16T20:03:28.810 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.810 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.810 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.810 DEBUG:boto:Params: {} 2024-04-16T20:03:28.811 DEBUG:boto:Token: None 2024-04-16T20:03:28.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19 2024-04-16T20:03:28.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2HCfyIRaxzqy7Cyw6ANvHbcSnA= 2024-04-16T20:03:28.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2HCfyIRaxzqy7Cyw6ANvHbcSnA=', 'Content-Length': '0'} 2024-04-16T20:03:28.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a91f4fcfe8ffdb-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.854 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:03:28.854 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:03:28.855 DEBUG:boto:Method: GET 2024-04-16T20:03:28.855 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:03:28.855 DEBUG:boto:Data: 2024-04-16T20:03:28.855 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.855 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.855 DEBUG:boto:Params: {} 2024-04-16T20:03:28.855 DEBUG:boto:Token: None 2024-04-16T20:03:28.855 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj19 2024-04-16T20:03:28.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+2HCfyIRaxzqy7Cyw6ANvHbcSnA= 2024-04-16T20:03:28.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+2HCfyIRaxzqy7Cyw6ANvHbcSnA=', 'Content-Length': '0'} 2024-04-16T20:03:28.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d738179e6a505c90-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.858 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.858 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.858 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.858 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.859 DEBUG:boto:Method: GET 2024-04-16T20:03:28.859 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.859 DEBUG:boto:Data: 2024-04-16T20:03:28.859 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.859 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.859 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.859 DEBUG:boto:Params: {} 2024-04-16T20:03:28.859 DEBUG:boto:Token: None 2024-04-16T20:03:28.859 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007d22a20ce4f1890-00661ed990-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.902 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.903 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:28.903 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:28.903 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.903 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.903 DEBUG:boto:Method: GET 2024-04-16T20:03:28.903 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:03:28.903 DEBUG:boto:Data: 2024-04-16T20:03:28.903 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.903 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.903 DEBUG:boto:Params: {} 2024-04-16T20:03:28.904 DEBUG:boto:Token: None 2024-04-16T20:03:28.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/ 2024-04-16T20:03:28.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M= 2024-04-16T20:03:28.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nvgmPHA9avgAS8DjePuEtYjZT1M=', 'Content-Length': '0'} 2024-04-16T20:03:28.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009853afe05b33202f-00661ed990-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.907 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:28.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:03:28.908 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:28.908 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:28.908 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.908 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.908 DEBUG:boto:Method: GET 2024-04-16T20:03:28.908 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.908 DEBUG:boto:Data: 2024-04-16T20:03:28.908 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.908 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.908 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.908 DEBUG:boto:Params: {} 2024-04-16T20:03:28.909 DEBUG:boto:Token: None 2024-04-16T20:03:28.909 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.909 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA= 2024-04-16T20:03:28.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA=', 'Content-Length': '0'} 2024-04-16T20:03:28.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ccecf7748292dc-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.954 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:28.954 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:28.954 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.955 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.955 DEBUG:boto:Method: GET 2024-04-16T20:03:28.955 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.955 DEBUG:boto:Data: 2024-04-16T20:03:28.955 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.955 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.955 DEBUG:boto:Params: {} 2024-04-16T20:03:28.955 DEBUG:boto:Token: None 2024-04-16T20:03:28.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA= 2024-04-16T20:03:28.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA=', 'Content-Length': '0'} 2024-04-16T20:03:28.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ff597d749e2b95-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.958 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:28.958 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:28.958 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.958 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.958 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.958 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.958 DEBUG:boto:Data: 2024-04-16T20:03:28.958 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.959 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.959 DEBUG:boto:Params: {} 2024-04-16T20:03:28.959 DEBUG:boto:Token: None 2024-04-16T20:03:28.959 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jiekaec0oitJTbCVqZNTlOQPYDY= 2024-04-16T20:03:28.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jiekaec0oitJTbCVqZNTlOQPYDY=', 'Content-Length': '0'} 2024-04-16T20:03:28.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d39279bd1cbff5f-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.961 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:28.961 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:28.961 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.961 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.961 DEBUG:boto:Method: HEAD 2024-04-16T20:03:28.962 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.962 DEBUG:boto:Data: 2024-04-16T20:03:28.962 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.962 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:28.962 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.962 DEBUG:boto:Params: {} 2024-04-16T20:03:28.962 DEBUG:boto:Token: None 2024-04-16T20:03:28.962 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.962 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jiekaec0oitJTbCVqZNTlOQPYDY= 2024-04-16T20:03:28.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jiekaec0oitJTbCVqZNTlOQPYDY=', 'Content-Length': '0'} 2024-04-16T20:03:28.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c740fe43dae966e1-00661ed990-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:28.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:03:28.964 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:28.965 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:28.965 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.965 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.965 DEBUG:boto:Method: GET 2024-04-16T20:03:28.965 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.965 DEBUG:boto:Data: 2024-04-16T20:03:28.965 DEBUG:boto:Headers: {} 2024-04-16T20:03:28.965 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:28.965 DEBUG:boto:Port: 8000 2024-04-16T20:03:28.965 DEBUG:boto:Params: {} 2024-04-16T20:03:28.965 DEBUG:boto:Token: None 2024-04-16T20:03:28.966 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:28 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:28.966 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA= 2024-04-16T20:03:28.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPk5flqQlthP8mgWXOi+bRR3ydA=', 'Content-Length': '0'} 2024-04-16T20:03:28.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613029b2bd441107-00661ed990-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.010 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:29.010 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:29.011 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:29.011 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:03:29.011 DEBUG:boto:Method: GET 2024-04-16T20:03:29.011 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:29.011 DEBUG:boto:Data: 2024-04-16T20:03:29.011 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.011 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.011 DEBUG:boto:Params: {} 2024-04-16T20:03:29.011 DEBUG:boto:Token: None 2024-04-16T20:03:29.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:03:29.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LGi7fpkLNwMpsBnFYBNaUentEE4= 2024-04-16T20:03:29.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LGi7fpkLNwMpsBnFYBNaUentEE4=', 'Content-Length': '0'} 2024-04-16T20:03:29.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3577db97ca9a3b-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.014 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.014 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.014 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.014 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.014 DEBUG:boto:Method: GET 2024-04-16T20:03:29.015 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.015 DEBUG:boto:Data: 2024-04-16T20:03:29.015 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.015 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.015 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.015 DEBUG:boto:Params: {} 2024-04-16T20:03:29.015 DEBUG:boto:Token: None 2024-04-16T20:03:29.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f546b2b613e69855-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.058 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.059 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.059 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.059 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.059 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.059 DEBUG:boto:Method: GET 2024-04-16T20:03:29.059 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.059 DEBUG:boto:Data: 2024-04-16T20:03:29.059 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.059 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.060 DEBUG:boto:Params: {} 2024-04-16T20:03:29.060 DEBUG:boto:Token: None 2024-04-16T20:03:29.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19e24bdc4869c4d-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.063 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.064 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:29.064 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:29.064 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.064 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:03:29.064 DEBUG:boto:Data: 2024-04-16T20:03:29.064 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.064 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.064 DEBUG:boto:Params: {} 2024-04-16T20:03:29.065 DEBUG:boto:Token: None 2024-04-16T20:03:29.065 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj22 2024-04-16T20:03:29.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3K8y/tK+pOc+/1MJDdz21MI/Ehg= 2024-04-16T20:03:29.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3K8y/tK+pOc+/1MJDdz21MI/Ehg=', 'Content-Length': '0'} 2024-04-16T20:03:29.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000237288b5db11d19-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.067 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:29.067 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:29.067 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.067 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:03:29.067 DEBUG:boto:Data: 2024-04-16T20:03:29.067 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.068 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.068 DEBUG:boto:Params: {} 2024-04-16T20:03:29.068 DEBUG:boto:Token: None 2024-04-16T20:03:29.068 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj22 2024-04-16T20:03:29.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3K8y/tK+pOc+/1MJDdz21MI/Ehg= 2024-04-16T20:03:29.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3K8y/tK+pOc+/1MJDdz21MI/Ehg=', 'Content-Length': '0'} 2024-04-16T20:03:29.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000565a17228fe2b6a4-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:03:29.070 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:29.070 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:29.070 DEBUG:boto:Method: GET 2024-04-16T20:03:29.070 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:03:29.070 DEBUG:boto:Data: 2024-04-16T20:03:29.070 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.070 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.071 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.071 DEBUG:boto:Params: {} 2024-04-16T20:03:29.071 DEBUG:boto:Token: None 2024-04-16T20:03:29.071 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj22 2024-04-16T20:03:29.071 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oKfQ6pu2gUAKBIb/pS2Haao7jRU= 2024-04-16T20:03:29.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oKfQ6pu2gUAKBIb/pS2Haao7jRU=', 'Content-Length': '0'} 2024-04-16T20:03:29.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008caba92981317e54-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.114 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:03:29.114 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:03:29.115 DEBUG:boto:Method: GET 2024-04-16T20:03:29.115 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:03:29.115 DEBUG:boto:Data: 2024-04-16T20:03:29.115 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.115 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.115 DEBUG:boto:Params: {} 2024-04-16T20:03:29.115 DEBUG:boto:Token: None 2024-04-16T20:03:29.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj22 2024-04-16T20:03:29.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oKfQ6pu2gUAKBIb/pS2Haao7jRU= 2024-04-16T20:03:29.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oKfQ6pu2gUAKBIb/pS2Haao7jRU=', 'Content-Length': '0'} 2024-04-16T20:03:29.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04301c51d490385-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.118 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.118 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.118 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.118 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.118 DEBUG:boto:Method: GET 2024-04-16T20:03:29.118 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.118 DEBUG:boto:Data: 2024-04-16T20:03:29.118 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.118 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.119 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.119 DEBUG:boto:Params: {} 2024-04-16T20:03:29.119 DEBUG:boto:Token: None 2024-04-16T20:03:29.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d107bdef6279cfb-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.162 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.163 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.163 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.163 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.163 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.163 DEBUG:boto:Method: GET 2024-04-16T20:03:29.163 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:03:29.163 DEBUG:boto:Data: 2024-04-16T20:03:29.163 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.163 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.164 DEBUG:boto:Params: {} 2024-04-16T20:03:29.164 DEBUG:boto:Token: None 2024-04-16T20:03:29.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092607280aa7b3c54-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.167 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:03:29.168 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.168 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.168 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.168 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.168 DEBUG:boto:Method: GET 2024-04-16T20:03:29.168 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.168 DEBUG:boto:Data: 2024-04-16T20:03:29.168 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.168 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.168 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.168 DEBUG:boto:Params: {} 2024-04-16T20:03:29.169 DEBUG:boto:Token: None 2024-04-16T20:03:29.169 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE= 2024-04-16T20:03:29.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE=', 'Content-Length': '0'} 2024-04-16T20:03:29.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66724f601bd5965-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.218 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.218 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.219 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.219 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.219 DEBUG:boto:Method: GET 2024-04-16T20:03:29.219 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.219 DEBUG:boto:Data: 2024-04-16T20:03:29.219 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.219 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.219 DEBUG:boto:Params: {} 2024-04-16T20:03:29.219 DEBUG:boto:Token: None 2024-04-16T20:03:29.220 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE= 2024-04-16T20:03:29.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE=', 'Content-Length': '0'} 2024-04-16T20:03:29.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8213e6abc8f95fd-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.222 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.222 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.223 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.223 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.223 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.223 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.223 DEBUG:boto:Data: 2024-04-16T20:03:29.223 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.223 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.223 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.223 DEBUG:boto:Params: {} 2024-04-16T20:03:29.223 DEBUG:boto:Token: None 2024-04-16T20:03:29.224 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ju5USQVNkiVUHdoAzE9CH6/0ASM= 2024-04-16T20:03:29.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ju5USQVNkiVUHdoAzE9CH6/0ASM=', 'Content-Length': '0'} 2024-04-16T20:03:29.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9911b1e44d5f262-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.226 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.226 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.227 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.227 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.227 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.227 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.227 DEBUG:boto:Data: 2024-04-16T20:03:29.227 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.227 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.227 DEBUG:boto:Params: {} 2024-04-16T20:03:29.227 DEBUG:boto:Token: None 2024-04-16T20:03:29.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ju5USQVNkiVUHdoAzE9CH6/0ASM= 2024-04-16T20:03:29.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ju5USQVNkiVUHdoAzE9CH6/0ASM=', 'Content-Length': '0'} 2024-04-16T20:03:29.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7a76c3ba2f713a-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:03:29.230 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.230 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.230 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.230 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.230 DEBUG:boto:Method: GET 2024-04-16T20:03:29.230 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.230 DEBUG:boto:Data: 2024-04-16T20:03:29.230 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.231 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.231 DEBUG:boto:Params: {} 2024-04-16T20:03:29.231 DEBUG:boto:Token: None 2024-04-16T20:03:29.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE= 2024-04-16T20:03:29.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE=', 'Content-Length': '0'} 2024-04-16T20:03:29.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314f0882b233f318-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.274 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.274 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.275 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.275 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.275 DEBUG:boto:Method: GET 2024-04-16T20:03:29.275 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.275 DEBUG:boto:Data: 2024-04-16T20:03:29.275 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.275 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.275 DEBUG:boto:Params: {} 2024-04-16T20:03:29.275 DEBUG:boto:Token: None 2024-04-16T20:03:29.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:03:29.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE= 2024-04-16T20:03:29.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6rt6zylBb8+5aqdbTcFcdqjg+BE=', 'Content-Length': '0'} 2024-04-16T20:03:29.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4a90e5cbb55b31-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.278 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.278 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.278 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.278 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.278 DEBUG:boto:Method: GET 2024-04-16T20:03:29.278 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.279 DEBUG:boto:Data: 2024-04-16T20:03:29.279 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.279 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.279 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.279 DEBUG:boto:Params: {} 2024-04-16T20:03:29.279 DEBUG:boto:Token: None 2024-04-16T20:03:29.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a3c458c6d15501-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.322 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.323 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.323 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.323 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.323 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.323 DEBUG:boto:Method: GET 2024-04-16T20:03:29.323 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.323 DEBUG:boto:Data: 2024-04-16T20:03:29.323 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.323 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.323 DEBUG:boto:Params: {} 2024-04-16T20:03:29.324 DEBUG:boto:Token: None 2024-04-16T20:03:29.324 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f7423fa86366fa-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.327 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.327 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.328 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.328 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.328 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:03:29.328 DEBUG:boto:Data: 2024-04-16T20:03:29.328 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.328 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.328 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.328 DEBUG:boto:Params: {} 2024-04-16T20:03:29.328 DEBUG:boto:Token: None 2024-04-16T20:03:29.328 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23 2024-04-16T20:03:29.329 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5U3qXrDPjkMrOhHk/e2Ri15cRZE= 2024-04-16T20:03:29.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5U3qXrDPjkMrOhHk/e2Ri15cRZE=', 'Content-Length': '0'} 2024-04-16T20:03:29.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bd38dcaaceabd6-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.331 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.331 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.331 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.331 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:03:29.331 DEBUG:boto:Data: 2024-04-16T20:03:29.331 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.331 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.331 DEBUG:boto:Params: {} 2024-04-16T20:03:29.332 DEBUG:boto:Token: None 2024-04-16T20:03:29.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23 2024-04-16T20:03:29.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5U3qXrDPjkMrOhHk/e2Ri15cRZE= 2024-04-16T20:03:29.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5U3qXrDPjkMrOhHk/e2Ri15cRZE=', 'Content-Length': '0'} 2024-04-16T20:03:29.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd08f0d4d7d5ff5f-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:03:29.334 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.334 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.334 DEBUG:boto:Method: GET 2024-04-16T20:03:29.334 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:03:29.334 DEBUG:boto:Data: 2024-04-16T20:03:29.334 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.334 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.334 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.335 DEBUG:boto:Params: {} 2024-04-16T20:03:29.335 DEBUG:boto:Token: None 2024-04-16T20:03:29.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23 2024-04-16T20:03:29.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bkhgdlLEjKvyWOWVlkQrCgTtaoQ= 2024-04-16T20:03:29.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bkhgdlLEjKvyWOWVlkQrCgTtaoQ=', 'Content-Length': '0'} 2024-04-16T20:03:29.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b064bcf48d3f42-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.378 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:03:29.378 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:03:29.379 DEBUG:boto:Method: GET 2024-04-16T20:03:29.379 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:03:29.379 DEBUG:boto:Data: 2024-04-16T20:03:29.379 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.379 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.379 DEBUG:boto:Params: {} 2024-04-16T20:03:29.379 DEBUG:boto:Token: None 2024-04-16T20:03:29.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj23 2024-04-16T20:03:29.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bkhgdlLEjKvyWOWVlkQrCgTtaoQ= 2024-04-16T20:03:29.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bkhgdlLEjKvyWOWVlkQrCgTtaoQ=', 'Content-Length': '0'} 2024-04-16T20:03:29.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e59a6870e2177faa-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.382 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.383 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.383 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.383 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.383 DEBUG:boto:Method: GET 2024-04-16T20:03:29.383 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.383 DEBUG:boto:Data: 2024-04-16T20:03:29.383 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.383 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.383 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.383 DEBUG:boto:Params: {} 2024-04-16T20:03:29.383 DEBUG:boto:Token: None 2024-04-16T20:03:29.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a33ff0cce8cd44a-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.426 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.427 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.427 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.427 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.427 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.427 DEBUG:boto:Method: GET 2024-04-16T20:03:29.427 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:03:29.427 DEBUG:boto:Data: 2024-04-16T20:03:29.427 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.427 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.427 DEBUG:boto:Params: {} 2024-04-16T20:03:29.428 DEBUG:boto:Token: None 2024-04-16T20:03:29.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fbbf9735cfd9eae-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.431 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:03:29.431 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.431 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.431 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.431 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.431 DEBUG:boto:Method: GET 2024-04-16T20:03:29.432 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.432 DEBUG:boto:Data: 2024-04-16T20:03:29.432 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.432 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.432 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.432 DEBUG:boto:Params: {} 2024-04-16T20:03:29.432 DEBUG:boto:Token: None 2024-04-16T20:03:29.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM= 2024-04-16T20:03:29.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM=', 'Content-Length': '0'} 2024-04-16T20:03:29.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb3c384d30769d0-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.474 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.475 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.475 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.475 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.475 DEBUG:boto:Method: GET 2024-04-16T20:03:29.475 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.475 DEBUG:boto:Data: 2024-04-16T20:03:29.475 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.475 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.475 DEBUG:boto:Params: {} 2024-04-16T20:03:29.475 DEBUG:boto:Token: None 2024-04-16T20:03:29.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM= 2024-04-16T20:03:29.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM=', 'Content-Length': '0'} 2024-04-16T20:03:29.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f2f4789956a424-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.478 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.478 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.478 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.478 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.479 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.479 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.479 DEBUG:boto:Data: 2024-04-16T20:03:29.479 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.479 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.479 DEBUG:boto:Params: {} 2024-04-16T20:03:29.479 DEBUG:boto:Token: None 2024-04-16T20:03:29.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:snXcOt1hKeh0mFx/MDJs5oledUY= 2024-04-16T20:03:29.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:snXcOt1hKeh0mFx/MDJs5oledUY=', 'Content-Length': '0'} 2024-04-16T20:03:29.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf480a7a2832884e-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.482 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.482 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.482 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.482 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.482 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.482 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.482 DEBUG:boto:Data: 2024-04-16T20:03:29.482 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.482 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.482 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.483 DEBUG:boto:Params: {} 2024-04-16T20:03:29.483 DEBUG:boto:Token: None 2024-04-16T20:03:29.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:snXcOt1hKeh0mFx/MDJs5oledUY= 2024-04-16T20:03:29.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:snXcOt1hKeh0mFx/MDJs5oledUY=', 'Content-Length': '0'} 2024-04-16T20:03:29.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075fbcfd8f627e0fd-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:03:29.485 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.485 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.485 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.486 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.486 DEBUG:boto:Method: GET 2024-04-16T20:03:29.486 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.486 DEBUG:boto:Data: 2024-04-16T20:03:29.486 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.486 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.486 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.486 DEBUG:boto:Params: {} 2024-04-16T20:03:29.486 DEBUG:boto:Token: None 2024-04-16T20:03:29.486 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM= 2024-04-16T20:03:29.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM=', 'Content-Length': '0'} 2024-04-16T20:03:29.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067db682d4bbcd6c7-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.530 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.530 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.530 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.531 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.531 DEBUG:boto:Method: GET 2024-04-16T20:03:29.531 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.531 DEBUG:boto:Data: 2024-04-16T20:03:29.531 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.531 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.531 DEBUG:boto:Params: {} 2024-04-16T20:03:29.531 DEBUG:boto:Token: None 2024-04-16T20:03:29.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:03:29.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM= 2024-04-16T20:03:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZBNwcwezF8lvaPiOqtheTzrp2IM=', 'Content-Length': '0'} 2024-04-16T20:03:29.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a17509a78fcb892-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.534 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.534 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.535 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.535 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.535 DEBUG:boto:Method: GET 2024-04-16T20:03:29.535 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.535 DEBUG:boto:Data: 2024-04-16T20:03:29.535 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.535 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.535 DEBUG:boto:Params: {} 2024-04-16T20:03:29.535 DEBUG:boto:Token: None 2024-04-16T20:03:29.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf83b8e7c3b709e-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.578 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.579 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.579 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.579 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.579 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.579 DEBUG:boto:Method: GET 2024-04-16T20:03:29.579 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.579 DEBUG:boto:Data: 2024-04-16T20:03:29.579 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.579 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.579 DEBUG:boto:Params: {} 2024-04-16T20:03:29.579 DEBUG:boto:Token: None 2024-04-16T20:03:29.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f92447d715439ad6-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.583 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.583 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.584 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.584 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.584 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:03:29.584 DEBUG:boto:Data: 2024-04-16T20:03:29.584 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.584 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.584 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.584 DEBUG:boto:Params: {} 2024-04-16T20:03:29.584 DEBUG:boto:Token: None 2024-04-16T20:03:29.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33 2024-04-16T20:03:29.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mtE1l7V2rO0dbkv4fb78DMPeTiU= 2024-04-16T20:03:29.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mtE1l7V2rO0dbkv4fb78DMPeTiU=', 'Content-Length': '0'} 2024-04-16T20:03:29.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de90177f91b0bfc8-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.587 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.587 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.587 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.587 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:03:29.587 DEBUG:boto:Data: 2024-04-16T20:03:29.587 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.587 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.587 DEBUG:boto:Params: {} 2024-04-16T20:03:29.588 DEBUG:boto:Token: None 2024-04-16T20:03:29.588 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33 2024-04-16T20:03:29.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mtE1l7V2rO0dbkv4fb78DMPeTiU= 2024-04-16T20:03:29.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mtE1l7V2rO0dbkv4fb78DMPeTiU=', 'Content-Length': '0'} 2024-04-16T20:03:29.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fd9cb9cc838988-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:03:29.591 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.591 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.591 DEBUG:boto:Method: GET 2024-04-16T20:03:29.591 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:03:29.591 DEBUG:boto:Data: 2024-04-16T20:03:29.591 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.591 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.591 DEBUG:boto:Params: {} 2024-04-16T20:03:29.591 DEBUG:boto:Token: None 2024-04-16T20:03:29.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33 2024-04-16T20:03:29.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yaHTiIL/gjaC0jxzg9qBtDw33ms= 2024-04-16T20:03:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yaHTiIL/gjaC0jxzg9qBtDw33ms=', 'Content-Length': '0'} 2024-04-16T20:03:29.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4988819e1bfbe03-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.634 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:03:29.634 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:03:29.635 DEBUG:boto:Method: GET 2024-04-16T20:03:29.635 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:03:29.635 DEBUG:boto:Data: 2024-04-16T20:03:29.635 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.635 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.635 DEBUG:boto:Params: {} 2024-04-16T20:03:29.635 DEBUG:boto:Token: None 2024-04-16T20:03:29.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj33 2024-04-16T20:03:29.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yaHTiIL/gjaC0jxzg9qBtDw33ms= 2024-04-16T20:03:29.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yaHTiIL/gjaC0jxzg9qBtDw33ms=', 'Content-Length': '0'} 2024-04-16T20:03:29.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007803381156a0ece7-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.638 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.638 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.638 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.638 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.638 DEBUG:boto:Method: GET 2024-04-16T20:03:29.638 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.638 DEBUG:boto:Data: 2024-04-16T20:03:29.638 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.639 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.639 DEBUG:boto:Params: {} 2024-04-16T20:03:29.639 DEBUG:boto:Token: None 2024-04-16T20:03:29.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e967102538561e-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.682 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.683 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.683 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.683 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.683 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.683 DEBUG:boto:Method: GET 2024-04-16T20:03:29.683 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:03:29.683 DEBUG:boto:Data: 2024-04-16T20:03:29.683 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.683 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.683 DEBUG:boto:Params: {} 2024-04-16T20:03:29.684 DEBUG:boto:Token: None 2024-04-16T20:03:29.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3a97c8589c2c2c-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.687 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:03:29.687 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.688 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.688 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.688 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.688 DEBUG:boto:Method: GET 2024-04-16T20:03:29.688 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.688 DEBUG:boto:Data: 2024-04-16T20:03:29.688 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.688 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.688 DEBUG:boto:Params: {} 2024-04-16T20:03:29.688 DEBUG:boto:Token: None 2024-04-16T20:03:29.689 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs= 2024-04-16T20:03:29.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs=', 'Content-Length': '0'} 2024-04-16T20:03:29.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dacb62445216e637-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.734 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.734 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.735 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.735 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.735 DEBUG:boto:Method: GET 2024-04-16T20:03:29.735 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.735 DEBUG:boto:Data: 2024-04-16T20:03:29.735 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.735 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.735 DEBUG:boto:Params: {} 2024-04-16T20:03:29.735 DEBUG:boto:Token: None 2024-04-16T20:03:29.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs= 2024-04-16T20:03:29.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs=', 'Content-Length': '0'} 2024-04-16T20:03:29.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ad0b10e8b2aff2-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.738 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.738 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.738 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.738 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.738 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.738 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.739 DEBUG:boto:Data: 2024-04-16T20:03:29.739 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.739 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.739 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.739 DEBUG:boto:Params: {} 2024-04-16T20:03:29.739 DEBUG:boto:Token: None 2024-04-16T20:03:29.739 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fe/rnY62UtxGwt2qfSuU+fuswps= 2024-04-16T20:03:29.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fe/rnY62UtxGwt2qfSuU+fuswps=', 'Content-Length': '0'} 2024-04-16T20:03:29.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7f3bf5d87695fd-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.741 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.742 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.742 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.742 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.742 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.742 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.742 DEBUG:boto:Data: 2024-04-16T20:03:29.742 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.742 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.742 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.820 DEBUG:boto:Params: {} 2024-04-16T20:03:29.820 DEBUG:boto:Token: None 2024-04-16T20:03:29.820 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fe/rnY62UtxGwt2qfSuU+fuswps= 2024-04-16T20:03:29.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fe/rnY62UtxGwt2qfSuU+fuswps=', 'Content-Length': '0'} 2024-04-16T20:03:29.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e1321785be4e9c-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:03:29.823 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.823 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.823 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.823 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.823 DEBUG:boto:Method: GET 2024-04-16T20:03:29.823 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.823 DEBUG:boto:Data: 2024-04-16T20:03:29.823 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.824 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.824 DEBUG:boto:Params: {} 2024-04-16T20:03:29.824 DEBUG:boto:Token: None 2024-04-16T20:03:29.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs= 2024-04-16T20:03:29.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs=', 'Content-Length': '0'} 2024-04-16T20:03:29.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa1ffd48ac16873-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.866 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.867 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.867 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.867 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.867 DEBUG:boto:Method: GET 2024-04-16T20:03:29.867 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.867 DEBUG:boto:Data: 2024-04-16T20:03:29.867 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.867 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.867 DEBUG:boto:Params: {} 2024-04-16T20:03:29.868 DEBUG:boto:Token: None 2024-04-16T20:03:29.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:03:29.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs= 2024-04-16T20:03:29.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W5Zz1LUnDA2z771cR0iGNreQSfs=', 'Content-Length': '0'} 2024-04-16T20:03:29.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f42940dcb021a377-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.870 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.870 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.870 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.870 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.870 DEBUG:boto:Method: GET 2024-04-16T20:03:29.871 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.871 DEBUG:boto:Data: 2024-04-16T20:03:29.871 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.871 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.871 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.871 DEBUG:boto:Params: {} 2024-04-16T20:03:29.871 DEBUG:boto:Token: None 2024-04-16T20:03:29.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eaca8fcb862b13f-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.914 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.915 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.915 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.915 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.915 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.915 DEBUG:boto:Method: GET 2024-04-16T20:03:29.915 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.916 DEBUG:boto:Data: 2024-04-16T20:03:29.916 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.916 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.916 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.916 DEBUG:boto:Params: {} 2024-04-16T20:03:29.916 DEBUG:boto:Token: None 2024-04-16T20:03:29.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05b4aa3f198e59d-00661ed991-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.920 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:29.920 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.920 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.920 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.920 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:03:29.920 DEBUG:boto:Data: 2024-04-16T20:03:29.920 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.921 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.921 DEBUG:boto:Params: {} 2024-04-16T20:03:29.921 DEBUG:boto:Token: None 2024-04-16T20:03:29.921 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34 2024-04-16T20:03:29.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qPmn485GLVeLHGRrRqi6zS0wJL8= 2024-04-16T20:03:29.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qPmn485GLVeLHGRrRqi6zS0wJL8=', 'Content-Length': '0'} 2024-04-16T20:03:29.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001baf2ec825947d81-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.923 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.923 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.924 DEBUG:boto:Method: HEAD 2024-04-16T20:03:29.924 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:03:29.924 DEBUG:boto:Data: 2024-04-16T20:03:29.924 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.924 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.924 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.924 DEBUG:boto:Params: {} 2024-04-16T20:03:29.925 DEBUG:boto:Token: None 2024-04-16T20:03:29.925 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34 2024-04-16T20:03:29.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qPmn485GLVeLHGRrRqi6zS0wJL8= 2024-04-16T20:03:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qPmn485GLVeLHGRrRqi6zS0wJL8=', 'Content-Length': '0'} 2024-04-16T20:03:29.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7806126fd0ac81-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:03:29.927 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.927 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.927 DEBUG:boto:Method: GET 2024-04-16T20:03:29.927 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:03:29.927 DEBUG:boto:Data: 2024-04-16T20:03:29.927 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.928 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.928 DEBUG:boto:Params: {} 2024-04-16T20:03:29.928 DEBUG:boto:Token: None 2024-04-16T20:03:29.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34 2024-04-16T20:03:29.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZoCDVcvQgeTMb/2Xbj92JSay2jg= 2024-04-16T20:03:29.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZoCDVcvQgeTMb/2Xbj92JSay2jg=', 'Content-Length': '0'} 2024-04-16T20:03:29.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000827e6faa6bb29688-00661ed991-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.970 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:03:29.971 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:03:29.971 DEBUG:boto:Method: GET 2024-04-16T20:03:29.971 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:03:29.971 DEBUG:boto:Data: 2024-04-16T20:03:29.971 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:29.972 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.972 DEBUG:boto:Params: {} 2024-04-16T20:03:29.972 DEBUG:boto:Token: None 2024-04-16T20:03:29.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/obj34 2024-04-16T20:03:29.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZoCDVcvQgeTMb/2Xbj92JSay2jg= 2024-04-16T20:03:29.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZoCDVcvQgeTMb/2Xbj92JSay2jg=', 'Content-Length': '0'} 2024-04-16T20:03:29.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022896fcb4fefd866-00661ed991-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:29.975 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:29.975 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:29.975 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.975 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.975 DEBUG:boto:Method: GET 2024-04-16T20:03:29.975 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:03:29.975 DEBUG:boto:Data: 2024-04-16T20:03:29.975 DEBUG:boto:Headers: {} 2024-04-16T20:03:29.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:29.975 DEBUG:boto:Port: 8000 2024-04-16T20:03:29.975 DEBUG:boto:Params: {} 2024-04-16T20:03:29.976 DEBUG:boto:Token: None 2024-04-16T20:03:29.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:29 GMT /yrjefh-52/ 2024-04-16T20:03:29.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo= 2024-04-16T20:03:29.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GGqoU76VgnK9oehOBhCdW3HNGwo=', 'Content-Length': '0'} 2024-04-16T20:03:29.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071de984d52496bf2-00661ed991-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.018 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.019 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.019 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.019 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:30.019 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:03:30.019 DEBUG:boto:Method: GET 2024-04-16T20:03:30.019 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:03:30.020 DEBUG:boto:Data: 2024-04-16T20:03:30.020 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.020 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.020 DEBUG:boto:Params: {} 2024-04-16T20:03:30.020 DEBUG:boto:Token: None 2024-04-16T20:03:30.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003fba7306d4c299b-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.024 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:03:30.024 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.024 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.024 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.024 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.024 DEBUG:boto:Method: GET 2024-04-16T20:03:30.025 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.025 DEBUG:boto:Data: 2024-04-16T20:03:30.025 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.025 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.025 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.025 DEBUG:boto:Params: {} 2024-04-16T20:03:30.025 DEBUG:boto:Token: None 2024-04-16T20:03:30.025 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw= 2024-04-16T20:03:30.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw=', 'Content-Length': '0'} 2024-04-16T20:03:30.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607744f4eac4b75d-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.070 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.071 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.071 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.071 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.071 DEBUG:boto:Method: GET 2024-04-16T20:03:30.071 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.071 DEBUG:boto:Data: 2024-04-16T20:03:30.071 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.072 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.072 DEBUG:boto:Params: {} 2024-04-16T20:03:30.072 DEBUG:boto:Token: None 2024-04-16T20:03:30.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw= 2024-04-16T20:03:30.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw=', 'Content-Length': '0'} 2024-04-16T20:03:30.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064193f92dad606c3-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.075 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.075 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.075 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.075 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.075 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.075 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.075 DEBUG:boto:Data: 2024-04-16T20:03:30.076 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.076 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.076 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.076 DEBUG:boto:Params: {} 2024-04-16T20:03:30.076 DEBUG:boto:Token: None 2024-04-16T20:03:30.076 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AOWylLkesmPkCx0ty19OuK23ALY= 2024-04-16T20:03:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AOWylLkesmPkCx0ty19OuK23ALY=', 'Content-Length': '0'} 2024-04-16T20:03:30.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f17428263fd9f8-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.079 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.079 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.079 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.079 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.079 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.079 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.079 DEBUG:boto:Data: 2024-04-16T20:03:30.079 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.079 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.079 DEBUG:boto:Params: {} 2024-04-16T20:03:30.080 DEBUG:boto:Token: None 2024-04-16T20:03:30.080 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AOWylLkesmPkCx0ty19OuK23ALY= 2024-04-16T20:03:30.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AOWylLkesmPkCx0ty19OuK23ALY=', 'Content-Length': '0'} 2024-04-16T20:03:30.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d419976a67eb178-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:03:30.082 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.082 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.082 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.082 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.083 DEBUG:boto:Method: GET 2024-04-16T20:03:30.083 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.083 DEBUG:boto:Data: 2024-04-16T20:03:30.083 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.083 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.083 DEBUG:boto:Params: {} 2024-04-16T20:03:30.083 DEBUG:boto:Token: None 2024-04-16T20:03:30.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw= 2024-04-16T20:03:30.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw=', 'Content-Length': '0'} 2024-04-16T20:03:30.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2989c50b81e6d64-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.126 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.127 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.127 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.127 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.127 DEBUG:boto:Method: GET 2024-04-16T20:03:30.127 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.127 DEBUG:boto:Data: 2024-04-16T20:03:30.127 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.127 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.127 DEBUG:boto:Params: {} 2024-04-16T20:03:30.128 DEBUG:boto:Token: None 2024-04-16T20:03:30.128 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:03:30.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw= 2024-04-16T20:03:30.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1Nl+yd/y7MiAbQt0BzV8iw+SLw=', 'Content-Length': '0'} 2024-04-16T20:03:30.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046b43cc02b311e9e-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.131 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.131 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.131 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.131 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.131 DEBUG:boto:Method: GET 2024-04-16T20:03:30.131 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.131 DEBUG:boto:Data: 2024-04-16T20:03:30.131 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.131 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.131 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.131 DEBUG:boto:Params: {} 2024-04-16T20:03:30.132 DEBUG:boto:Token: None 2024-04-16T20:03:30.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e6b7605e875e2a5-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.134 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.135 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.135 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.135 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.135 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.135 DEBUG:boto:Method: GET 2024-04-16T20:03:30.135 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.136 DEBUG:boto:Data: 2024-04-16T20:03:30.136 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.136 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.136 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.136 DEBUG:boto:Params: {} 2024-04-16T20:03:30.136 DEBUG:boto:Token: None 2024-04-16T20:03:30.136 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bdf0f57935690ce-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.139 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.140 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.140 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.140 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.140 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:03:30.140 DEBUG:boto:Data: 2024-04-16T20:03:30.140 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.140 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.140 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.140 DEBUG:boto:Params: {} 2024-04-16T20:03:30.140 DEBUG:boto:Token: None 2024-04-16T20:03:30.141 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38 2024-04-16T20:03:30.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTOOX1jIz/U1Rm/TSOXM+2ttWjE= 2024-04-16T20:03:30.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTOOX1jIz/U1Rm/TSOXM+2ttWjE=', 'Content-Length': '0'} 2024-04-16T20:03:30.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91f6a3e3675ceb5-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.143 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.143 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.143 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.143 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:03:30.143 DEBUG:boto:Data: 2024-04-16T20:03:30.143 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.143 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.143 DEBUG:boto:Params: {} 2024-04-16T20:03:30.144 DEBUG:boto:Token: None 2024-04-16T20:03:30.144 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38 2024-04-16T20:03:30.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTOOX1jIz/U1Rm/TSOXM+2ttWjE= 2024-04-16T20:03:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTOOX1jIz/U1Rm/TSOXM+2ttWjE=', 'Content-Length': '0'} 2024-04-16T20:03:30.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad30cb020831b68-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:03:30.146 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.146 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.146 DEBUG:boto:Method: GET 2024-04-16T20:03:30.146 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:03:30.146 DEBUG:boto:Data: 2024-04-16T20:03:30.147 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.147 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.147 DEBUG:boto:Params: {} 2024-04-16T20:03:30.147 DEBUG:boto:Token: None 2024-04-16T20:03:30.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38 2024-04-16T20:03:30.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V80BOb8Syc3fNj4P0kb+G6rHL8A= 2024-04-16T20:03:30.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V80BOb8Syc3fNj4P0kb+G6rHL8A=', 'Content-Length': '0'} 2024-04-16T20:03:30.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000396f4b120d96a769-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.194 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:03:30.195 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:03:30.195 DEBUG:boto:Method: GET 2024-04-16T20:03:30.195 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:03:30.195 DEBUG:boto:Data: 2024-04-16T20:03:30.195 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.195 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.195 DEBUG:boto:Params: {} 2024-04-16T20:03:30.195 DEBUG:boto:Token: None 2024-04-16T20:03:30.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj38 2024-04-16T20:03:30.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V80BOb8Syc3fNj4P0kb+G6rHL8A= 2024-04-16T20:03:30.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V80BOb8Syc3fNj4P0kb+G6rHL8A=', 'Content-Length': '0'} 2024-04-16T20:03:30.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24cb1d3292d13a8-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.198 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.198 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.198 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.199 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.199 DEBUG:boto:Method: GET 2024-04-16T20:03:30.199 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.199 DEBUG:boto:Data: 2024-04-16T20:03:30.199 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.199 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.199 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.199 DEBUG:boto:Params: {} 2024-04-16T20:03:30.199 DEBUG:boto:Token: None 2024-04-16T20:03:30.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c2a1c8910d025b-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.202 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.202 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.202 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.203 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.203 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.203 DEBUG:boto:Method: GET 2024-04-16T20:03:30.203 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:03:30.203 DEBUG:boto:Data: 2024-04-16T20:03:30.203 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.203 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.203 DEBUG:boto:Params: {} 2024-04-16T20:03:30.203 DEBUG:boto:Token: None 2024-04-16T20:03:30.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c018c838371eef-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.207 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:03:30.207 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.207 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.208 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.208 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.208 DEBUG:boto:Method: GET 2024-04-16T20:03:30.208 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.208 DEBUG:boto:Data: 2024-04-16T20:03:30.208 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.208 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.208 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.208 DEBUG:boto:Params: {} 2024-04-16T20:03:30.208 DEBUG:boto:Token: None 2024-04-16T20:03:30.208 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI= 2024-04-16T20:03:30.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI=', 'Content-Length': '0'} 2024-04-16T20:03:30.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e3487a77c9c4a53-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.254 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.254 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.255 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.255 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.255 DEBUG:boto:Method: GET 2024-04-16T20:03:30.255 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.255 DEBUG:boto:Data: 2024-04-16T20:03:30.255 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.255 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.255 DEBUG:boto:Params: {} 2024-04-16T20:03:30.255 DEBUG:boto:Token: None 2024-04-16T20:03:30.256 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI= 2024-04-16T20:03:30.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI=', 'Content-Length': '0'} 2024-04-16T20:03:30.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002edb6d9fd48adab-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.258 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.258 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.258 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.258 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.258 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.259 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.259 DEBUG:boto:Data: 2024-04-16T20:03:30.259 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.259 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.259 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.259 DEBUG:boto:Params: {} 2024-04-16T20:03:30.259 DEBUG:boto:Token: None 2024-04-16T20:03:30.259 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xXpp4EG2vIWSmTlSkOdZzH4zMe8= 2024-04-16T20:03:30.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xXpp4EG2vIWSmTlSkOdZzH4zMe8=', 'Content-Length': '0'} 2024-04-16T20:03:30.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62168045de578df-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.262 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.262 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.262 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.262 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.262 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.262 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.262 DEBUG:boto:Data: 2024-04-16T20:03:30.263 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.263 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.263 DEBUG:boto:Params: {} 2024-04-16T20:03:30.263 DEBUG:boto:Token: None 2024-04-16T20:03:30.263 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xXpp4EG2vIWSmTlSkOdZzH4zMe8= 2024-04-16T20:03:30.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xXpp4EG2vIWSmTlSkOdZzH4zMe8=', 'Content-Length': '0'} 2024-04-16T20:03:30.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f0ae6d0cd7ee73-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:03:30.266 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.266 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.266 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.266 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.266 DEBUG:boto:Method: GET 2024-04-16T20:03:30.266 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.266 DEBUG:boto:Data: 2024-04-16T20:03:30.266 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.267 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.267 DEBUG:boto:Params: {} 2024-04-16T20:03:30.267 DEBUG:boto:Token: None 2024-04-16T20:03:30.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI= 2024-04-16T20:03:30.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI=', 'Content-Length': '0'} 2024-04-16T20:03:30.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010d811dc6f99522a-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.310 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.311 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.311 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.311 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.311 DEBUG:boto:Method: GET 2024-04-16T20:03:30.311 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.311 DEBUG:boto:Data: 2024-04-16T20:03:30.311 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.311 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.311 DEBUG:boto:Params: {} 2024-04-16T20:03:30.311 DEBUG:boto:Token: None 2024-04-16T20:03:30.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:03:30.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI= 2024-04-16T20:03:30.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CE0UWgm0Xtib0LB9bRclt4aXYzI=', 'Content-Length': '0'} 2024-04-16T20:03:30.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f012e92f16736b-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.314 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.314 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.314 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.315 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.315 DEBUG:boto:Method: GET 2024-04-16T20:03:30.315 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.315 DEBUG:boto:Data: 2024-04-16T20:03:30.315 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.315 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.315 DEBUG:boto:Params: {} 2024-04-16T20:03:30.315 DEBUG:boto:Token: None 2024-04-16T20:03:30.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051485a9605464b54-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.362 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.363 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.363 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.363 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.363 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.363 DEBUG:boto:Method: GET 2024-04-16T20:03:30.363 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.363 DEBUG:boto:Data: 2024-04-16T20:03:30.363 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.364 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.364 DEBUG:boto:Params: {} 2024-04-16T20:03:30.364 DEBUG:boto:Token: None 2024-04-16T20:03:30.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19e8134d3f1e9a4-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.367 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.367 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.368 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.368 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.368 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:03:30.368 DEBUG:boto:Data: 2024-04-16T20:03:30.368 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.368 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.368 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.368 DEBUG:boto:Params: {} 2024-04-16T20:03:30.368 DEBUG:boto:Token: None 2024-04-16T20:03:30.368 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44 2024-04-16T20:03:30.369 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IBHmyI+g65LF6Nc/4A7XrXmPYgk= 2024-04-16T20:03:30.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IBHmyI+g65LF6Nc/4A7XrXmPYgk=', 'Content-Length': '0'} 2024-04-16T20:03:30.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fe68a40b1890cd-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.371 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.371 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.371 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.371 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:03:30.371 DEBUG:boto:Data: 2024-04-16T20:03:30.372 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.372 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.372 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.372 DEBUG:boto:Params: {} 2024-04-16T20:03:30.372 DEBUG:boto:Token: None 2024-04-16T20:03:30.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44 2024-04-16T20:03:30.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IBHmyI+g65LF6Nc/4A7XrXmPYgk= 2024-04-16T20:03:30.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IBHmyI+g65LF6Nc/4A7XrXmPYgk=', 'Content-Length': '0'} 2024-04-16T20:03:30.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a441f71dbc883cb3-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:03:30.374 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.375 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.375 DEBUG:boto:Method: GET 2024-04-16T20:03:30.375 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:03:30.375 DEBUG:boto:Data: 2024-04-16T20:03:30.375 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.375 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.375 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.375 DEBUG:boto:Params: {} 2024-04-16T20:03:30.375 DEBUG:boto:Token: None 2024-04-16T20:03:30.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44 2024-04-16T20:03:30.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WpfDjZzLB0Askv86udxSbFYn05k= 2024-04-16T20:03:30.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WpfDjZzLB0Askv86udxSbFYn05k=', 'Content-Length': '0'} 2024-04-16T20:03:30.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e80bfb4b383ef0-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.418 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:03:30.419 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:03:30.419 DEBUG:boto:Method: GET 2024-04-16T20:03:30.419 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:03:30.419 DEBUG:boto:Data: 2024-04-16T20:03:30.419 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.419 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.419 DEBUG:boto:Params: {} 2024-04-16T20:03:30.419 DEBUG:boto:Token: None 2024-04-16T20:03:30.420 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj44 2024-04-16T20:03:30.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WpfDjZzLB0Askv86udxSbFYn05k= 2024-04-16T20:03:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WpfDjZzLB0Askv86udxSbFYn05k=', 'Content-Length': '0'} 2024-04-16T20:03:30.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050110632db193c77-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.422 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.423 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.423 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.423 DEBUG:boto:Method: GET 2024-04-16T20:03:30.423 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.423 DEBUG:boto:Data: 2024-04-16T20:03:30.423 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.423 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.423 DEBUG:boto:Params: {} 2024-04-16T20:03:30.423 DEBUG:boto:Token: None 2024-04-16T20:03:30.424 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000befef7cb2e83946d-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.466 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.467 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.467 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.467 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.467 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.467 DEBUG:boto:Method: GET 2024-04-16T20:03:30.467 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:03:30.467 DEBUG:boto:Data: 2024-04-16T20:03:30.468 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.468 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.468 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.468 DEBUG:boto:Params: {} 2024-04-16T20:03:30.468 DEBUG:boto:Token: None 2024-04-16T20:03:30.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b93a09d63de5a25-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.471 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:03:30.472 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.472 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.472 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.472 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.472 DEBUG:boto:Method: GET 2024-04-16T20:03:30.473 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.473 DEBUG:boto:Data: 2024-04-16T20:03:30.473 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.473 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.473 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.473 DEBUG:boto:Params: {} 2024-04-16T20:03:30.473 DEBUG:boto:Token: None 2024-04-16T20:03:30.473 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg= 2024-04-16T20:03:30.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg=', 'Content-Length': '0'} 2024-04-16T20:03:30.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e15c64c36c58ca8-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.518 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.519 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.519 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.519 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.519 DEBUG:boto:Method: GET 2024-04-16T20:03:30.519 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.519 DEBUG:boto:Data: 2024-04-16T20:03:30.519 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.519 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.520 DEBUG:boto:Params: {} 2024-04-16T20:03:30.520 DEBUG:boto:Token: None 2024-04-16T20:03:30.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg= 2024-04-16T20:03:30.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg=', 'Content-Length': '0'} 2024-04-16T20:03:30.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe9585261a89163-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.523 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.523 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.523 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.523 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.523 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.523 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.523 DEBUG:boto:Data: 2024-04-16T20:03:30.524 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.524 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.524 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.524 DEBUG:boto:Params: {} 2024-04-16T20:03:30.524 DEBUG:boto:Token: None 2024-04-16T20:03:30.524 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ltje6dR3cBCINJ9aSpocXegvYfg= 2024-04-16T20:03:30.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ltje6dR3cBCINJ9aSpocXegvYfg=', 'Content-Length': '0'} 2024-04-16T20:03:30.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084768827f1f226fa-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.526 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.526 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.526 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.527 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.527 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.527 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.527 DEBUG:boto:Data: 2024-04-16T20:03:30.527 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.527 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.527 DEBUG:boto:Params: {} 2024-04-16T20:03:30.527 DEBUG:boto:Token: None 2024-04-16T20:03:30.528 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ltje6dR3cBCINJ9aSpocXegvYfg= 2024-04-16T20:03:30.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ltje6dR3cBCINJ9aSpocXegvYfg=', 'Content-Length': '0'} 2024-04-16T20:03:30.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c700b64ce76471-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:03:30.530 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.530 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.530 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.530 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.531 DEBUG:boto:Method: GET 2024-04-16T20:03:30.531 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.531 DEBUG:boto:Data: 2024-04-16T20:03:30.531 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.531 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.531 DEBUG:boto:Params: {} 2024-04-16T20:03:30.531 DEBUG:boto:Token: None 2024-04-16T20:03:30.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg= 2024-04-16T20:03:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg=', 'Content-Length': '0'} 2024-04-16T20:03:30.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddadf215172c6e97-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.574 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.575 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.575 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.575 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.575 DEBUG:boto:Method: GET 2024-04-16T20:03:30.575 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.575 DEBUG:boto:Data: 2024-04-16T20:03:30.575 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.575 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.575 DEBUG:boto:Params: {} 2024-04-16T20:03:30.576 DEBUG:boto:Token: None 2024-04-16T20:03:30.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:03:30.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg= 2024-04-16T20:03:30.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lNoDeaNahbVYOTCErXFDnM+AXCg=', 'Content-Length': '0'} 2024-04-16T20:03:30.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edecb716e82177d5-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.579 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.579 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.579 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.579 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.579 DEBUG:boto:Method: GET 2024-04-16T20:03:30.579 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.579 DEBUG:boto:Data: 2024-04-16T20:03:30.579 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.580 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.580 DEBUG:boto:Params: {} 2024-04-16T20:03:30.580 DEBUG:boto:Token: None 2024-04-16T20:03:30.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d1581c389a04525-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.626 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.627 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.627 DEBUG:boto:Method: GET 2024-04-16T20:03:30.627 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.627 DEBUG:boto:Data: 2024-04-16T20:03:30.627 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.628 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.628 DEBUG:boto:Params: {} 2024-04-16T20:03:30.628 DEBUG:boto:Token: None 2024-04-16T20:03:30.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2cffe791f5b452c-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.631 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.632 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.632 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.632 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.632 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:03:30.632 DEBUG:boto:Data: 2024-04-16T20:03:30.632 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.632 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.632 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.633 DEBUG:boto:Params: {} 2024-04-16T20:03:30.633 DEBUG:boto:Token: None 2024-04-16T20:03:30.633 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46 2024-04-16T20:03:30.633 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aHbezhctkJc1CNSeZ7GbMFcUUhs= 2024-04-16T20:03:30.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aHbezhctkJc1CNSeZ7GbMFcUUhs=', 'Content-Length': '0'} 2024-04-16T20:03:30.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a70aefcae202408-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.635 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.635 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.635 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.635 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:03:30.635 DEBUG:boto:Data: 2024-04-16T20:03:30.636 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.636 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.636 DEBUG:boto:Params: {} 2024-04-16T20:03:30.636 DEBUG:boto:Token: None 2024-04-16T20:03:30.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46 2024-04-16T20:03:30.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aHbezhctkJc1CNSeZ7GbMFcUUhs= 2024-04-16T20:03:30.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aHbezhctkJc1CNSeZ7GbMFcUUhs=', 'Content-Length': '0'} 2024-04-16T20:03:30.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb5f6f707faceb1-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:03:30.639 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.639 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.639 DEBUG:boto:Method: GET 2024-04-16T20:03:30.639 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:03:30.639 DEBUG:boto:Data: 2024-04-16T20:03:30.639 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.639 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.640 DEBUG:boto:Params: {} 2024-04-16T20:03:30.640 DEBUG:boto:Token: None 2024-04-16T20:03:30.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46 2024-04-16T20:03:30.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3C6dOxCsZYSC6YMwz/8CznS871M= 2024-04-16T20:03:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3C6dOxCsZYSC6YMwz/8CznS871M=', 'Content-Length': '0'} 2024-04-16T20:03:30.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51c76248ab22eb9-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.682 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:03:30.683 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:03:30.683 DEBUG:boto:Method: GET 2024-04-16T20:03:30.683 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:03:30.683 DEBUG:boto:Data: 2024-04-16T20:03:30.683 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.683 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.683 DEBUG:boto:Params: {} 2024-04-16T20:03:30.683 DEBUG:boto:Token: None 2024-04-16T20:03:30.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj46 2024-04-16T20:03:30.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3C6dOxCsZYSC6YMwz/8CznS871M= 2024-04-16T20:03:30.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3C6dOxCsZYSC6YMwz/8CznS871M=', 'Content-Length': '0'} 2024-04-16T20:03:30.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617dd06210020e92-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.687 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.687 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.687 DEBUG:boto:Method: GET 2024-04-16T20:03:30.687 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.687 DEBUG:boto:Data: 2024-04-16T20:03:30.687 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.687 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.687 DEBUG:boto:Params: {} 2024-04-16T20:03:30.687 DEBUG:boto:Token: None 2024-04-16T20:03:30.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007382b879c8317ffe-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.730 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.731 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.731 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.731 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.731 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.731 DEBUG:boto:Method: GET 2024-04-16T20:03:30.731 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:03:30.731 DEBUG:boto:Data: 2024-04-16T20:03:30.731 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.732 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.732 DEBUG:boto:Params: {} 2024-04-16T20:03:30.732 DEBUG:boto:Token: None 2024-04-16T20:03:30.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7c06ffe96f63d7-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.735 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:03:30.736 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.736 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.736 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.736 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.736 DEBUG:boto:Method: GET 2024-04-16T20:03:30.736 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.736 DEBUG:boto:Data: 2024-04-16T20:03:30.736 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.736 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.736 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.737 DEBUG:boto:Params: {} 2024-04-16T20:03:30.737 DEBUG:boto:Token: None 2024-04-16T20:03:30.737 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.737 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U= 2024-04-16T20:03:30.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U=', 'Content-Length': '0'} 2024-04-16T20:03:30.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000006539dbe9e05b3-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.782 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.782 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.782 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.783 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.783 DEBUG:boto:Method: GET 2024-04-16T20:03:30.783 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.783 DEBUG:boto:Data: 2024-04-16T20:03:30.783 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.783 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.783 DEBUG:boto:Params: {} 2024-04-16T20:03:30.783 DEBUG:boto:Token: None 2024-04-16T20:03:30.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U= 2024-04-16T20:03:30.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U=', 'Content-Length': '0'} 2024-04-16T20:03:30.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063cc4580609e459b-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.786 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.786 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.787 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.787 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.787 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.787 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.787 DEBUG:boto:Data: 2024-04-16T20:03:30.787 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.787 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.787 DEBUG:boto:Params: {} 2024-04-16T20:03:30.787 DEBUG:boto:Token: None 2024-04-16T20:03:30.788 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/oCJ5t4cafFr7Ux3dyek+EkYevw= 2024-04-16T20:03:30.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/oCJ5t4cafFr7Ux3dyek+EkYevw=', 'Content-Length': '0'} 2024-04-16T20:03:30.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8a2073ee8a230c-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.790 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.790 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.790 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.790 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.790 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.791 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.791 DEBUG:boto:Data: 2024-04-16T20:03:30.791 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.791 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.791 DEBUG:boto:Params: {} 2024-04-16T20:03:30.791 DEBUG:boto:Token: None 2024-04-16T20:03:30.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/oCJ5t4cafFr7Ux3dyek+EkYevw= 2024-04-16T20:03:30.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/oCJ5t4cafFr7Ux3dyek+EkYevw=', 'Content-Length': '0'} 2024-04-16T20:03:30.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0f5441348fb41e-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:03:30.794 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.794 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.794 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.794 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.794 DEBUG:boto:Method: GET 2024-04-16T20:03:30.794 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.794 DEBUG:boto:Data: 2024-04-16T20:03:30.794 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.795 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.795 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.795 DEBUG:boto:Params: {} 2024-04-16T20:03:30.795 DEBUG:boto:Token: None 2024-04-16T20:03:30.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U= 2024-04-16T20:03:30.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U=', 'Content-Length': '0'} 2024-04-16T20:03:30.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e52f5fb3fdae8340-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.838 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.838 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.838 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.839 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.839 DEBUG:boto:Method: GET 2024-04-16T20:03:30.839 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.839 DEBUG:boto:Data: 2024-04-16T20:03:30.839 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.839 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.839 DEBUG:boto:Params: {} 2024-04-16T20:03:30.839 DEBUG:boto:Token: None 2024-04-16T20:03:30.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:03:30.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U= 2024-04-16T20:03:30.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+WSRq2U6AX+dAc/BIjwzIW7kJ0U=', 'Content-Length': '0'} 2024-04-16T20:03:30.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086dc549cbafee09c-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.842 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.842 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.842 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.842 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.842 DEBUG:boto:Method: GET 2024-04-16T20:03:30.842 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.842 DEBUG:boto:Data: 2024-04-16T20:03:30.842 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.843 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.843 DEBUG:boto:Params: {} 2024-04-16T20:03:30.843 DEBUG:boto:Token: None 2024-04-16T20:03:30.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002118584e063a15ca-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.886 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.887 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.887 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.887 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.887 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.887 DEBUG:boto:Method: GET 2024-04-16T20:03:30.887 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.887 DEBUG:boto:Data: 2024-04-16T20:03:30.887 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.887 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.887 DEBUG:boto:Params: {} 2024-04-16T20:03:30.888 DEBUG:boto:Token: None 2024-04-16T20:03:30.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feed62d42a35268d-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.891 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.892 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.892 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.892 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.892 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:03:30.892 DEBUG:boto:Data: 2024-04-16T20:03:30.892 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.892 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.892 DEBUG:boto:Params: {} 2024-04-16T20:03:30.892 DEBUG:boto:Token: None 2024-04-16T20:03:30.893 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51 2024-04-16T20:03:30.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:394SzyBOVSWXQAt6U9x1OzxPeSA= 2024-04-16T20:03:30.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:394SzyBOVSWXQAt6U9x1OzxPeSA=', 'Content-Length': '0'} 2024-04-16T20:03:30.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9c0a72bc76aff9-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.895 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.895 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.895 DEBUG:boto:Method: HEAD 2024-04-16T20:03:30.895 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:03:30.895 DEBUG:boto:Data: 2024-04-16T20:03:30.896 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.896 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.896 DEBUG:boto:Params: {} 2024-04-16T20:03:30.896 DEBUG:boto:Token: None 2024-04-16T20:03:30.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51 2024-04-16T20:03:30.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:394SzyBOVSWXQAt6U9x1OzxPeSA= 2024-04-16T20:03:30.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:394SzyBOVSWXQAt6U9x1OzxPeSA=', 'Content-Length': '0'} 2024-04-16T20:03:30.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7f51f381bf15e8-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:03:30.899 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.899 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.899 DEBUG:boto:Method: GET 2024-04-16T20:03:30.899 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:03:30.899 DEBUG:boto:Data: 2024-04-16T20:03:30.899 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.899 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.899 DEBUG:boto:Params: {} 2024-04-16T20:03:30.899 DEBUG:boto:Token: None 2024-04-16T20:03:30.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51 2024-04-16T20:03:30.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4mYeM7CnVywvx9x9J9n7Dey+giI= 2024-04-16T20:03:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4mYeM7CnVywvx9x9J9n7Dey+giI=', 'Content-Length': '0'} 2024-04-16T20:03:30.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238f247a870be2ae-00661ed992-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.942 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:03:30.942 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:03:30.943 DEBUG:boto:Method: GET 2024-04-16T20:03:30.943 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:03:30.943 DEBUG:boto:Data: 2024-04-16T20:03:30.943 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.943 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.943 DEBUG:boto:Params: {} 2024-04-16T20:03:30.943 DEBUG:boto:Token: None 2024-04-16T20:03:30.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj51 2024-04-16T20:03:30.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4mYeM7CnVywvx9x9J9n7Dey+giI= 2024-04-16T20:03:30.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4mYeM7CnVywvx9x9J9n7Dey+giI=', 'Content-Length': '0'} 2024-04-16T20:03:30.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a668ceffbc3d9b1-00661ed992-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.946 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.946 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.946 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.946 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.946 DEBUG:boto:Method: GET 2024-04-16T20:03:30.946 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.947 DEBUG:boto:Data: 2024-04-16T20:03:30.947 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.947 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.947 DEBUG:boto:Params: {} 2024-04-16T20:03:30.947 DEBUG:boto:Token: None 2024-04-16T20:03:30.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030f816d29b6505df-00661ed992-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.990 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.991 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:30.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:30.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.991 DEBUG:boto:Method: GET 2024-04-16T20:03:30.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:03:30.991 DEBUG:boto:Data: 2024-04-16T20:03:30.992 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:30.992 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.992 DEBUG:boto:Params: {} 2024-04-16T20:03:30.992 DEBUG:boto:Token: None 2024-04-16T20:03:30.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/ 2024-04-16T20:03:30.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c= 2024-04-16T20:03:30.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eRuRA96fOtqiB+9H4A/s6tQpD3c=', 'Content-Length': '0'} 2024-04-16T20:03:30.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0f7c035ef366c2-00661ed992-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:30.996 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:30.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:03:30.996 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:30.996 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:30.996 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:30.997 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:30.997 DEBUG:boto:Method: GET 2024-04-16T20:03:30.997 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:30.997 DEBUG:boto:Data: 2024-04-16T20:03:30.997 DEBUG:boto:Headers: {} 2024-04-16T20:03:30.997 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:30.997 DEBUG:boto:Port: 8000 2024-04-16T20:03:30.997 DEBUG:boto:Params: {} 2024-04-16T20:03:30.997 DEBUG:boto:Token: None 2024-04-16T20:03:30.997 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:30 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:30.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YCuiXhQNkkfWb2+Qv2c6sMaPJF8= 2024-04-16T20:03:30.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YCuiXhQNkkfWb2+Qv2c6sMaPJF8=', 'Content-Length': '0'} 2024-04-16T20:03:31.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088595c9ba43aaed9-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.042 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.043 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.043 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.043 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.043 DEBUG:boto:Method: GET 2024-04-16T20:03:31.043 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.043 DEBUG:boto:Data: 2024-04-16T20:03:31.043 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.043 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.043 DEBUG:boto:Params: {} 2024-04-16T20:03:31.044 DEBUG:boto:Token: None 2024-04-16T20:03:31.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU= 2024-04-16T20:03:31.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU=', 'Content-Length': '0'} 2024-04-16T20:03:31.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f2f6daca913dcc-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.046 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.047 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.047 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.047 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.047 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.047 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.047 DEBUG:boto:Data: 2024-04-16T20:03:31.047 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.047 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.047 DEBUG:boto:Params: {} 2024-04-16T20:03:31.047 DEBUG:boto:Token: None 2024-04-16T20:03:31.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yOSEbAhbcPNrxJS0wK4PGnoyfY8= 2024-04-16T20:03:31.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yOSEbAhbcPNrxJS0wK4PGnoyfY8=', 'Content-Length': '0'} 2024-04-16T20:03:31.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b1fe9430bd0f7f-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.050 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.050 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.050 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.050 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.050 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.051 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.051 DEBUG:boto:Data: 2024-04-16T20:03:31.051 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.051 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.051 DEBUG:boto:Params: {} 2024-04-16T20:03:31.051 DEBUG:boto:Token: None 2024-04-16T20:03:31.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yOSEbAhbcPNrxJS0wK4PGnoyfY8= 2024-04-16T20:03:31.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yOSEbAhbcPNrxJS0wK4PGnoyfY8=', 'Content-Length': '0'} 2024-04-16T20:03:31.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0ad9ccb29ab1f3-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:03:31.054 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.054 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.054 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.054 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.054 DEBUG:boto:Method: GET 2024-04-16T20:03:31.054 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.054 DEBUG:boto:Data: 2024-04-16T20:03:31.054 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.055 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.055 DEBUG:boto:Params: {} 2024-04-16T20:03:31.055 DEBUG:boto:Token: None 2024-04-16T20:03:31.055 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU= 2024-04-16T20:03:31.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU=', 'Content-Length': '0'} 2024-04-16T20:03:31.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc0e473e2abd4f0-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.098 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.099 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.099 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.099 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.099 DEBUG:boto:Method: GET 2024-04-16T20:03:31.099 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.099 DEBUG:boto:Data: 2024-04-16T20:03:31.099 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.099 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.100 DEBUG:boto:Params: {} 2024-04-16T20:03:31.100 DEBUG:boto:Token: None 2024-04-16T20:03:31.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:03:31.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU= 2024-04-16T20:03:31.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vXTeeWBHslYE78y9OiIvD0RdNLU=', 'Content-Length': '0'} 2024-04-16T20:03:31.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085ea585ea0f83bb8-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.103 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.103 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.103 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.103 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.103 DEBUG:boto:Method: GET 2024-04-16T20:03:31.103 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.103 DEBUG:boto:Data: 2024-04-16T20:03:31.103 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.103 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.104 DEBUG:boto:Params: {} 2024-04-16T20:03:31.104 DEBUG:boto:Token: None 2024-04-16T20:03:31.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009922d3581639880e-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.150 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.151 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.151 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.151 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.151 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.151 DEBUG:boto:Method: GET 2024-04-16T20:03:31.151 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.152 DEBUG:boto:Data: 2024-04-16T20:03:31.152 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.152 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.152 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.152 DEBUG:boto:Params: {} 2024-04-16T20:03:31.152 DEBUG:boto:Token: None 2024-04-16T20:03:31.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f021abbea4b45fb-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.156 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.156 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.156 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.156 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.156 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:03:31.156 DEBUG:boto:Data: 2024-04-16T20:03:31.156 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.157 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.157 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.157 DEBUG:boto:Params: {} 2024-04-16T20:03:31.157 DEBUG:boto:Token: None 2024-04-16T20:03:31.157 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55 2024-04-16T20:03:31.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gKOFBecItCocX1AHEBBdewJQLUQ= 2024-04-16T20:03:31.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gKOFBecItCocX1AHEBBdewJQLUQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfdc1b041775eeca-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.159 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.160 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.160 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.160 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:03:31.160 DEBUG:boto:Data: 2024-04-16T20:03:31.160 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.160 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.160 DEBUG:boto:Params: {} 2024-04-16T20:03:31.160 DEBUG:boto:Token: None 2024-04-16T20:03:31.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55 2024-04-16T20:03:31.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gKOFBecItCocX1AHEBBdewJQLUQ= 2024-04-16T20:03:31.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gKOFBecItCocX1AHEBBdewJQLUQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28fb65927dd0512-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:03:31.163 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.163 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.163 DEBUG:boto:Method: GET 2024-04-16T20:03:31.163 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:03:31.163 DEBUG:boto:Data: 2024-04-16T20:03:31.163 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.163 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.164 DEBUG:boto:Params: {} 2024-04-16T20:03:31.164 DEBUG:boto:Token: None 2024-04-16T20:03:31.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55 2024-04-16T20:03:31.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jfhk/VtHswTnyHus/QGHhdara/g= 2024-04-16T20:03:31.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jfhk/VtHswTnyHus/QGHhdara/g=', 'Content-Length': '0'} 2024-04-16T20:03:31.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f76dc212b018a94-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.206 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:03:31.206 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:03:31.207 DEBUG:boto:Method: GET 2024-04-16T20:03:31.207 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:03:31.207 DEBUG:boto:Data: 2024-04-16T20:03:31.207 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.207 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.207 DEBUG:boto:Params: {} 2024-04-16T20:03:31.207 DEBUG:boto:Token: None 2024-04-16T20:03:31.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj55 2024-04-16T20:03:31.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jfhk/VtHswTnyHus/QGHhdara/g= 2024-04-16T20:03:31.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jfhk/VtHswTnyHus/QGHhdara/g=', 'Content-Length': '0'} 2024-04-16T20:03:31.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854fdd79676d4442-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.210 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.210 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.210 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.210 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.210 DEBUG:boto:Method: GET 2024-04-16T20:03:31.210 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.211 DEBUG:boto:Data: 2024-04-16T20:03:31.211 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.211 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.211 DEBUG:boto:Params: {} 2024-04-16T20:03:31.211 DEBUG:boto:Token: None 2024-04-16T20:03:31.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c799d9e06cfa399f-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.254 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.255 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.255 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.255 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.255 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.255 DEBUG:boto:Method: GET 2024-04-16T20:03:31.255 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:03:31.256 DEBUG:boto:Data: 2024-04-16T20:03:31.256 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.256 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.256 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.256 DEBUG:boto:Params: {} 2024-04-16T20:03:31.256 DEBUG:boto:Token: None 2024-04-16T20:03:31.256 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8e1ef3e4ee8dbe-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.260 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:03:31.260 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.260 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.260 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.260 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.261 DEBUG:boto:Method: GET 2024-04-16T20:03:31.261 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.261 DEBUG:boto:Data: 2024-04-16T20:03:31.261 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.261 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.261 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.261 DEBUG:boto:Params: {} 2024-04-16T20:03:31.261 DEBUG:boto:Token: None 2024-04-16T20:03:31.261 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ= 2024-04-16T20:03:31.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3f5cd702d544e8-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.306 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.306 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.306 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.307 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.307 DEBUG:boto:Method: GET 2024-04-16T20:03:31.307 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.307 DEBUG:boto:Data: 2024-04-16T20:03:31.307 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.307 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.307 DEBUG:boto:Params: {} 2024-04-16T20:03:31.307 DEBUG:boto:Token: None 2024-04-16T20:03:31.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ= 2024-04-16T20:03:31.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36cdd13929cadf6-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.310 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.310 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.310 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.310 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.310 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.310 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.310 DEBUG:boto:Data: 2024-04-16T20:03:31.310 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.311 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.311 DEBUG:boto:Params: {} 2024-04-16T20:03:31.311 DEBUG:boto:Token: None 2024-04-16T20:03:31.311 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2goLqZaVg1EL9k93Uz/CizFXxOE= 2024-04-16T20:03:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2goLqZaVg1EL9k93Uz/CizFXxOE=', 'Content-Length': '0'} 2024-04-16T20:03:31.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072be6d6e1d0ca9bc-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.313 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.314 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.314 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.314 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.314 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.314 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.314 DEBUG:boto:Data: 2024-04-16T20:03:31.314 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.314 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.314 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.314 DEBUG:boto:Params: {} 2024-04-16T20:03:31.314 DEBUG:boto:Token: None 2024-04-16T20:03:31.315 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2goLqZaVg1EL9k93Uz/CizFXxOE= 2024-04-16T20:03:31.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2goLqZaVg1EL9k93Uz/CizFXxOE=', 'Content-Length': '0'} 2024-04-16T20:03:31.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094aa92d0427185ce-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:03:31.318 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.318 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.318 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.318 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.318 DEBUG:boto:Method: GET 2024-04-16T20:03:31.318 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.318 DEBUG:boto:Data: 2024-04-16T20:03:31.318 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.318 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.318 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.318 DEBUG:boto:Params: {} 2024-04-16T20:03:31.318 DEBUG:boto:Token: None 2024-04-16T20:03:31.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ= 2024-04-16T20:03:31.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659071e7c60c13d6-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.362 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.362 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.362 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.362 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.363 DEBUG:boto:Method: GET 2024-04-16T20:03:31.363 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.363 DEBUG:boto:Data: 2024-04-16T20:03:31.363 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.363 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.363 DEBUG:boto:Params: {} 2024-04-16T20:03:31.363 DEBUG:boto:Token: None 2024-04-16T20:03:31.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:03:31.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ= 2024-04-16T20:03:31.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EEeEMnlb78R/YuCSMBrAqqdTvJQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d398237804dc31-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.366 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.366 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.366 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.366 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.366 DEBUG:boto:Method: GET 2024-04-16T20:03:31.366 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.367 DEBUG:boto:Data: 2024-04-16T20:03:31.367 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.367 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.367 DEBUG:boto:Params: {} 2024-04-16T20:03:31.367 DEBUG:boto:Token: None 2024-04-16T20:03:31.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d44cb23a1d629d04-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.410 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.410 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.411 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.411 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.411 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.411 DEBUG:boto:Method: GET 2024-04-16T20:03:31.411 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.411 DEBUG:boto:Data: 2024-04-16T20:03:31.411 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.411 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.411 DEBUG:boto:Params: {} 2024-04-16T20:03:31.411 DEBUG:boto:Token: None 2024-04-16T20:03:31.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195fcc55db416e89-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.415 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.415 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.415 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.415 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.415 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:03:31.415 DEBUG:boto:Data: 2024-04-16T20:03:31.415 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.416 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.416 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.416 DEBUG:boto:Params: {} 2024-04-16T20:03:31.416 DEBUG:boto:Token: None 2024-04-16T20:03:31.416 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57 2024-04-16T20:03:31.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ul81mq7aU/EIEsqp5Qsyqr2H6QA= 2024-04-16T20:03:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ul81mq7aU/EIEsqp5Qsyqr2H6QA=', 'Content-Length': '0'} 2024-04-16T20:03:31.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb754c1fb613c0a-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.419 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.419 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.419 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.419 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:03:31.419 DEBUG:boto:Data: 2024-04-16T20:03:31.419 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.419 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.419 DEBUG:boto:Params: {} 2024-04-16T20:03:31.419 DEBUG:boto:Token: None 2024-04-16T20:03:31.419 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57 2024-04-16T20:03:31.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ul81mq7aU/EIEsqp5Qsyqr2H6QA= 2024-04-16T20:03:31.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ul81mq7aU/EIEsqp5Qsyqr2H6QA=', 'Content-Length': '0'} 2024-04-16T20:03:31.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c41ccf7d70e38a-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:03:31.422 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.422 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.422 DEBUG:boto:Method: GET 2024-04-16T20:03:31.422 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:03:31.422 DEBUG:boto:Data: 2024-04-16T20:03:31.422 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.422 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.422 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.422 DEBUG:boto:Params: {} 2024-04-16T20:03:31.423 DEBUG:boto:Token: None 2024-04-16T20:03:31.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57 2024-04-16T20:03:31.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e9r2IJ2RPa9ga39HFoEqagn2HeQ= 2024-04-16T20:03:31.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e9r2IJ2RPa9ga39HFoEqagn2HeQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b424814840b83e6d-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.466 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:03:31.467 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:03:31.467 DEBUG:boto:Method: GET 2024-04-16T20:03:31.467 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:03:31.467 DEBUG:boto:Data: 2024-04-16T20:03:31.467 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.467 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.467 DEBUG:boto:Params: {} 2024-04-16T20:03:31.467 DEBUG:boto:Token: None 2024-04-16T20:03:31.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj57 2024-04-16T20:03:31.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e9r2IJ2RPa9ga39HFoEqagn2HeQ= 2024-04-16T20:03:31.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e9r2IJ2RPa9ga39HFoEqagn2HeQ=', 'Content-Length': '0'} 2024-04-16T20:03:31.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfca491b13e5586e-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.470 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.470 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.471 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.471 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.471 DEBUG:boto:Method: GET 2024-04-16T20:03:31.471 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.471 DEBUG:boto:Data: 2024-04-16T20:03:31.471 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.471 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.471 DEBUG:boto:Params: {} 2024-04-16T20:03:31.471 DEBUG:boto:Token: None 2024-04-16T20:03:31.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ba1866244aa81f-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.475 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.476 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.476 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.476 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.476 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.476 DEBUG:boto:Method: GET 2024-04-16T20:03:31.476 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:03:31.476 DEBUG:boto:Data: 2024-04-16T20:03:31.476 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.476 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.476 DEBUG:boto:Params: {} 2024-04-16T20:03:31.477 DEBUG:boto:Token: None 2024-04-16T20:03:31.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb7111b1eac0f717-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.482 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:03:31.483 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.483 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.483 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.483 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.483 DEBUG:boto:Method: GET 2024-04-16T20:03:31.483 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.484 DEBUG:boto:Data: 2024-04-16T20:03:31.484 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.484 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.484 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.484 DEBUG:boto:Params: {} 2024-04-16T20:03:31.484 DEBUG:boto:Token: None 2024-04-16T20:03:31.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY= 2024-04-16T20:03:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY=', 'Content-Length': '0'} 2024-04-16T20:03:31.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015254970315c235c-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.535 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.535 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.535 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.536 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.536 DEBUG:boto:Method: GET 2024-04-16T20:03:31.536 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.536 DEBUG:boto:Data: 2024-04-16T20:03:31.536 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.536 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.536 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.536 DEBUG:boto:Params: {} 2024-04-16T20:03:31.536 DEBUG:boto:Token: None 2024-04-16T20:03:31.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY= 2024-04-16T20:03:31.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY=', 'Content-Length': '0'} 2024-04-16T20:03:31.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d385d740a46e7b-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.554 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.554 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.554 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.554 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.554 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.554 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.554 DEBUG:boto:Data: 2024-04-16T20:03:31.555 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.555 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.555 DEBUG:boto:Params: {} 2024-04-16T20:03:31.555 DEBUG:boto:Token: None 2024-04-16T20:03:31.555 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SkI7MM8tjbsrmd1XF6zK3CpcZD0= 2024-04-16T20:03:31.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SkI7MM8tjbsrmd1XF6zK3CpcZD0=', 'Content-Length': '0'} 2024-04-16T20:03:31.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d93e227d9af5d3-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.559 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.559 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.559 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.559 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.559 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.559 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.559 DEBUG:boto:Data: 2024-04-16T20:03:31.559 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.559 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.560 DEBUG:boto:Params: {} 2024-04-16T20:03:31.560 DEBUG:boto:Token: None 2024-04-16T20:03:31.560 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SkI7MM8tjbsrmd1XF6zK3CpcZD0= 2024-04-16T20:03:31.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SkI7MM8tjbsrmd1XF6zK3CpcZD0=', 'Content-Length': '0'} 2024-04-16T20:03:31.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a935b3e76701f5-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:03:31.568 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.568 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.569 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.569 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.569 DEBUG:boto:Method: GET 2024-04-16T20:03:31.569 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.569 DEBUG:boto:Data: 2024-04-16T20:03:31.569 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.569 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.569 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.569 DEBUG:boto:Params: {} 2024-04-16T20:03:31.570 DEBUG:boto:Token: None 2024-04-16T20:03:31.570 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.570 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY= 2024-04-16T20:03:31.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY=', 'Content-Length': '0'} 2024-04-16T20:03:31.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c93ce8da56c63beb-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.620 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.620 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.621 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.621 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.621 DEBUG:boto:Method: GET 2024-04-16T20:03:31.621 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.621 DEBUG:boto:Data: 2024-04-16T20:03:31.621 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.621 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.621 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.621 DEBUG:boto:Params: {} 2024-04-16T20:03:31.622 DEBUG:boto:Token: None 2024-04-16T20:03:31.622 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:03:31.622 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY= 2024-04-16T20:03:31.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxM1gtj+qS9//QvxPt7/oz5Q4GY=', 'Content-Length': '0'} 2024-04-16T20:03:31.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a4acac93f63c54e-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.628 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.628 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.628 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.628 DEBUG:boto:Method: GET 2024-04-16T20:03:31.628 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.628 DEBUG:boto:Data: 2024-04-16T20:03:31.628 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.628 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.628 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.628 DEBUG:boto:Params: {} 2024-04-16T20:03:31.629 DEBUG:boto:Token: None 2024-04-16T20:03:31.629 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.629 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc8ad3fcaa9edec8-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.679 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.680 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.680 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.680 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.680 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.680 DEBUG:boto:Method: GET 2024-04-16T20:03:31.681 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.681 DEBUG:boto:Data: 2024-04-16T20:03:31.681 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.681 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.681 DEBUG:boto:Params: {} 2024-04-16T20:03:31.681 DEBUG:boto:Token: None 2024-04-16T20:03:31.681 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.682 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000815e7ea8c02533a-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.696 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.697 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.697 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.698 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.698 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:03:31.698 DEBUG:boto:Data: 2024-04-16T20:03:31.698 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.698 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.698 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.698 DEBUG:boto:Params: {} 2024-04-16T20:03:31.701 DEBUG:boto:Token: None 2024-04-16T20:03:31.701 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68 2024-04-16T20:03:31.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0bRNINqzLFFhVOyGDCaln+p24E4= 2024-04-16T20:03:31.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0bRNINqzLFFhVOyGDCaln+p24E4=', 'Content-Length': '0'} 2024-04-16T20:03:31.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada4828fe0602ced-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.706 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.706 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.706 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.707 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:03:31.707 DEBUG:boto:Data: 2024-04-16T20:03:31.707 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.707 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.707 DEBUG:boto:Params: {} 2024-04-16T20:03:31.707 DEBUG:boto:Token: None 2024-04-16T20:03:31.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68 2024-04-16T20:03:31.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0bRNINqzLFFhVOyGDCaln+p24E4= 2024-04-16T20:03:31.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0bRNINqzLFFhVOyGDCaln+p24E4=', 'Content-Length': '0'} 2024-04-16T20:03:31.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa0d7c83aea15cc8-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:03:31.712 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.712 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.712 DEBUG:boto:Method: GET 2024-04-16T20:03:31.712 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:03:31.712 DEBUG:boto:Data: 2024-04-16T20:03:31.712 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.712 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.712 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.712 DEBUG:boto:Params: {} 2024-04-16T20:03:31.713 DEBUG:boto:Token: None 2024-04-16T20:03:31.713 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68 2024-04-16T20:03:31.713 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sH9nxZliP4dXkAIOf+JxwmANDRI= 2024-04-16T20:03:31.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sH9nxZliP4dXkAIOf+JxwmANDRI=', 'Content-Length': '0'} 2024-04-16T20:03:31.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778b2cc14623b74c-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.764 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:03:31.764 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:03:31.764 DEBUG:boto:Method: GET 2024-04-16T20:03:31.764 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:03:31.764 DEBUG:boto:Data: 2024-04-16T20:03:31.764 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.765 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.765 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.765 DEBUG:boto:Params: {} 2024-04-16T20:03:31.765 DEBUG:boto:Token: None 2024-04-16T20:03:31.765 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj68 2024-04-16T20:03:31.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sH9nxZliP4dXkAIOf+JxwmANDRI= 2024-04-16T20:03:31.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sH9nxZliP4dXkAIOf+JxwmANDRI=', 'Content-Length': '0'} 2024-04-16T20:03:31.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b749f052515c788b-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.773 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.773 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.773 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.773 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.773 DEBUG:boto:Method: GET 2024-04-16T20:03:31.773 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.773 DEBUG:boto:Data: 2024-04-16T20:03:31.773 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.773 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.774 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.774 DEBUG:boto:Params: {} 2024-04-16T20:03:31.774 DEBUG:boto:Token: None 2024-04-16T20:03:31.774 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272117dc3ec72506-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.781 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.781 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.781 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.782 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.782 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.782 DEBUG:boto:Method: GET 2024-04-16T20:03:31.782 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:03:31.782 DEBUG:boto:Data: 2024-04-16T20:03:31.782 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.782 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.782 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.783 DEBUG:boto:Params: {} 2024-04-16T20:03:31.783 DEBUG:boto:Token: None 2024-04-16T20:03:31.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771028efafa067bf-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.832 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:03:31.833 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.833 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.833 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.833 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.833 DEBUG:boto:Method: GET 2024-04-16T20:03:31.833 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.833 DEBUG:boto:Data: 2024-04-16T20:03:31.833 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.834 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.834 DEBUG:boto:Params: {} 2024-04-16T20:03:31.834 DEBUG:boto:Token: None 2024-04-16T20:03:31.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y= 2024-04-16T20:03:31.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y=', 'Content-Length': '0'} 2024-04-16T20:03:31.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859b6eb228956a0a-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.883 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.883 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.883 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.883 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.883 DEBUG:boto:Method: GET 2024-04-16T20:03:31.883 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.883 DEBUG:boto:Data: 2024-04-16T20:03:31.883 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.884 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.884 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.884 DEBUG:boto:Params: {} 2024-04-16T20:03:31.884 DEBUG:boto:Token: None 2024-04-16T20:03:31.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y= 2024-04-16T20:03:31.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y=', 'Content-Length': '0'} 2024-04-16T20:03:31.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b006b8fbd29d2e97-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.890 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.890 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.891 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.891 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.891 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.891 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.891 DEBUG:boto:Data: 2024-04-16T20:03:31.891 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.891 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.891 DEBUG:boto:Params: {} 2024-04-16T20:03:31.891 DEBUG:boto:Token: None 2024-04-16T20:03:31.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uQgN8otCB4Sdz7gG2RrIGhyTteE= 2024-04-16T20:03:31.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uQgN8otCB4Sdz7gG2RrIGhyTteE=', 'Content-Length': '0'} 2024-04-16T20:03:31.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064dcd9602b14a573-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.895 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.896 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.896 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.896 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.896 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.896 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.896 DEBUG:boto:Data: 2024-04-16T20:03:31.897 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.897 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.897 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.897 DEBUG:boto:Params: {} 2024-04-16T20:03:31.897 DEBUG:boto:Token: None 2024-04-16T20:03:31.897 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uQgN8otCB4Sdz7gG2RrIGhyTteE= 2024-04-16T20:03:31.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uQgN8otCB4Sdz7gG2RrIGhyTteE=', 'Content-Length': '0'} 2024-04-16T20:03:31.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9bf875413dc5a9-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:03:31.901 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.901 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.902 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.902 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.902 DEBUG:boto:Method: GET 2024-04-16T20:03:31.902 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.902 DEBUG:boto:Data: 2024-04-16T20:03:31.902 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.902 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.903 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.903 DEBUG:boto:Params: {} 2024-04-16T20:03:31.903 DEBUG:boto:Token: None 2024-04-16T20:03:31.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y= 2024-04-16T20:03:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y=', 'Content-Length': '0'} 2024-04-16T20:03:31.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007618971b387081d4-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.951 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.951 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.951 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.951 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.951 DEBUG:boto:Method: GET 2024-04-16T20:03:31.951 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.952 DEBUG:boto:Data: 2024-04-16T20:03:31.952 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.952 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.952 DEBUG:boto:Params: {} 2024-04-16T20:03:31.952 DEBUG:boto:Token: None 2024-04-16T20:03:31.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:03:31.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y= 2024-04-16T20:03:31.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UW/A+Eq+dLlaizpNVVwGiKaDJ/Y=', 'Content-Length': '0'} 2024-04-16T20:03:31.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005626dc5bca00258f-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.956 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.956 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.956 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.956 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.956 DEBUG:boto:Method: GET 2024-04-16T20:03:31.956 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.956 DEBUG:boto:Data: 2024-04-16T20:03:31.956 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.957 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.957 DEBUG:boto:Params: {} 2024-04-16T20:03:31.957 DEBUG:boto:Token: None 2024-04-16T20:03:31.957 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.957 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddbfa3612a0392a1-00661ed993-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.962 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.963 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:31.963 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:31.963 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.963 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.964 DEBUG:boto:Method: GET 2024-04-16T20:03:31.964 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:03:31.964 DEBUG:boto:Data: 2024-04-16T20:03:31.964 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.964 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.964 DEBUG:boto:Params: {} 2024-04-16T20:03:31.964 DEBUG:boto:Token: None 2024-04-16T20:03:31.965 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/ 2024-04-16T20:03:31.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A= 2024-04-16T20:03:31.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nKd3PiALNPHQYVuvvR3PkTBxe0A=', 'Content-Length': '0'} 2024-04-16T20:03:31.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ac2ca2569c209a0-00661ed993-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.970 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:31.971 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.971 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.971 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.971 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:03:31.971 DEBUG:boto:Data: 2024-04-16T20:03:31.972 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.972 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.972 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.972 DEBUG:boto:Params: {} 2024-04-16T20:03:31.972 DEBUG:boto:Token: None 2024-04-16T20:03:31.973 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69 2024-04-16T20:03:31.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f41g89JBAFuqJJSwAy9IH+LZlPU= 2024-04-16T20:03:31.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f41g89JBAFuqJJSwAy9IH+LZlPU=', 'Content-Length': '0'} 2024-04-16T20:03:31.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b686253767edd0d-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.978 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.978 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.978 DEBUG:boto:Method: HEAD 2024-04-16T20:03:31.978 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:03:31.978 DEBUG:boto:Data: 2024-04-16T20:03:31.978 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:31.979 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.979 DEBUG:boto:Params: {} 2024-04-16T20:03:31.979 DEBUG:boto:Token: None 2024-04-16T20:03:31.979 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69 2024-04-16T20:03:31.980 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f41g89JBAFuqJJSwAy9IH+LZlPU= 2024-04-16T20:03:31.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f41g89JBAFuqJJSwAy9IH+LZlPU=', 'Content-Length': '0'} 2024-04-16T20:03:31.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1301412522e07c9-00661ed993-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:31.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:03:31.982 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:31.983 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:31.983 DEBUG:boto:Method: GET 2024-04-16T20:03:31.983 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:03:31.983 DEBUG:boto:Data: 2024-04-16T20:03:31.983 DEBUG:boto:Headers: {} 2024-04-16T20:03:31.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:31.983 DEBUG:boto:Port: 8000 2024-04-16T20:03:31.983 DEBUG:boto:Params: {} 2024-04-16T20:03:31.984 DEBUG:boto:Token: None 2024-04-16T20:03:31.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:31 GMT /yrjefh-52/obj69 2024-04-16T20:03:31.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J8M2K55lvN7ngONkWAlrMBMQXoo= 2024-04-16T20:03:31.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J8M2K55lvN7ngONkWAlrMBMQXoo=', 'Content-Length': '0'} 2024-04-16T20:03:31.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fbfb626c7a30bad-00661ed993-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.035 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:03:32.035 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:03:32.036 DEBUG:boto:Method: GET 2024-04-16T20:03:32.036 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:03:32.036 DEBUG:boto:Data: 2024-04-16T20:03:32.036 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.036 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.036 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.036 DEBUG:boto:Params: {} 2024-04-16T20:03:32.036 DEBUG:boto:Token: None 2024-04-16T20:03:32.037 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj69 2024-04-16T20:03:32.037 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SZzJVkktIhrzcvVt/dNoo/4oRbo= 2024-04-16T20:03:32.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SZzJVkktIhrzcvVt/dNoo/4oRbo=', 'Content-Length': '0'} 2024-04-16T20:03:32.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffbc3cb7bf7f8da4-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.040 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.040 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.040 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.040 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.041 DEBUG:boto:Method: GET 2024-04-16T20:03:32.041 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.041 DEBUG:boto:Data: 2024-04-16T20:03:32.041 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.041 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.041 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.041 DEBUG:boto:Params: {} 2024-04-16T20:03:32.042 DEBUG:boto:Token: None 2024-04-16T20:03:32.042 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe40cfd829447b1-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.046 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.047 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.047 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.047 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.047 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.047 DEBUG:boto:Method: GET 2024-04-16T20:03:32.047 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:03:32.047 DEBUG:boto:Data: 2024-04-16T20:03:32.047 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.048 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.048 DEBUG:boto:Params: {} 2024-04-16T20:03:32.048 DEBUG:boto:Token: None 2024-04-16T20:03:32.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d074b6ecfb5b8dd1-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.052 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:03:32.052 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.052 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.052 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.052 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.053 DEBUG:boto:Method: GET 2024-04-16T20:03:32.053 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.053 DEBUG:boto:Data: 2024-04-16T20:03:32.053 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.053 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.053 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.053 DEBUG:boto:Params: {} 2024-04-16T20:03:32.053 DEBUG:boto:Token: None 2024-04-16T20:03:32.053 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI= 2024-04-16T20:03:32.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI=', 'Content-Length': '0'} 2024-04-16T20:03:32.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b270101cecaae14-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.099 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.099 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.099 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.099 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.099 DEBUG:boto:Method: GET 2024-04-16T20:03:32.099 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.099 DEBUG:boto:Data: 2024-04-16T20:03:32.099 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.100 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.100 DEBUG:boto:Params: {} 2024-04-16T20:03:32.100 DEBUG:boto:Token: None 2024-04-16T20:03:32.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI= 2024-04-16T20:03:32.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI=', 'Content-Length': '0'} 2024-04-16T20:03:32.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f236f820267b79-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.103 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.103 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.103 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.103 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.103 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.103 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.103 DEBUG:boto:Data: 2024-04-16T20:03:32.103 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.104 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.104 DEBUG:boto:Params: {} 2024-04-16T20:03:32.104 DEBUG:boto:Token: None 2024-04-16T20:03:32.104 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qTid9TC9RwDq2+Mrws8Bo1BwQvI= 2024-04-16T20:03:32.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qTid9TC9RwDq2+Mrws8Bo1BwQvI=', 'Content-Length': '0'} 2024-04-16T20:03:32.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe992c22f9e08267-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.106 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.107 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.107 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.107 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.107 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.107 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.107 DEBUG:boto:Data: 2024-04-16T20:03:32.107 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.107 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.107 DEBUG:boto:Params: {} 2024-04-16T20:03:32.107 DEBUG:boto:Token: None 2024-04-16T20:03:32.108 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qTid9TC9RwDq2+Mrws8Bo1BwQvI= 2024-04-16T20:03:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qTid9TC9RwDq2+Mrws8Bo1BwQvI=', 'Content-Length': '0'} 2024-04-16T20:03:32.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ca884a044953969-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:03:32.110 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.110 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.110 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.110 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.110 DEBUG:boto:Method: GET 2024-04-16T20:03:32.110 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.110 DEBUG:boto:Data: 2024-04-16T20:03:32.111 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.111 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.111 DEBUG:boto:Params: {} 2024-04-16T20:03:32.111 DEBUG:boto:Token: None 2024-04-16T20:03:32.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI= 2024-04-16T20:03:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI=', 'Content-Length': '0'} 2024-04-16T20:03:32.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a98d6c9a62c760-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.154 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.155 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.155 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.155 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.155 DEBUG:boto:Method: GET 2024-04-16T20:03:32.155 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.155 DEBUG:boto:Data: 2024-04-16T20:03:32.155 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.156 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.156 DEBUG:boto:Params: {} 2024-04-16T20:03:32.156 DEBUG:boto:Token: None 2024-04-16T20:03:32.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:03:32.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI= 2024-04-16T20:03:32.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDHOIU6E5QGiv1uU9HDVSxdf/eI=', 'Content-Length': '0'} 2024-04-16T20:03:32.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca4a1e6266fc8ff5-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.159 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.159 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.159 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.159 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.159 DEBUG:boto:Method: GET 2024-04-16T20:03:32.159 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.159 DEBUG:boto:Data: 2024-04-16T20:03:32.159 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.159 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.160 DEBUG:boto:Params: {} 2024-04-16T20:03:32.160 DEBUG:boto:Token: None 2024-04-16T20:03:32.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a5cfd9fb4aa8cf1-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.163 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.163 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.163 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.163 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.163 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.163 DEBUG:boto:Method: GET 2024-04-16T20:03:32.164 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.164 DEBUG:boto:Data: 2024-04-16T20:03:32.164 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.164 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.164 DEBUG:boto:Params: {} 2024-04-16T20:03:32.164 DEBUG:boto:Token: None 2024-04-16T20:03:32.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081970a6abfffd2c0-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.210 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.211 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.211 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.211 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.212 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:03:32.212 DEBUG:boto:Data: 2024-04-16T20:03:32.212 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.212 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.212 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.212 DEBUG:boto:Params: {} 2024-04-16T20:03:32.212 DEBUG:boto:Token: None 2024-04-16T20:03:32.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76 2024-04-16T20:03:32.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lTUWEIxePEMaE/bTymaJQ5YGGyg= 2024-04-16T20:03:32.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lTUWEIxePEMaE/bTymaJQ5YGGyg=', 'Content-Length': '0'} 2024-04-16T20:03:32.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003625684c4b300673-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.215 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.215 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.215 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.215 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:03:32.215 DEBUG:boto:Data: 2024-04-16T20:03:32.215 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.216 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.216 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.216 DEBUG:boto:Params: {} 2024-04-16T20:03:32.216 DEBUG:boto:Token: None 2024-04-16T20:03:32.216 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76 2024-04-16T20:03:32.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lTUWEIxePEMaE/bTymaJQ5YGGyg= 2024-04-16T20:03:32.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lTUWEIxePEMaE/bTymaJQ5YGGyg=', 'Content-Length': '0'} 2024-04-16T20:03:32.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed90acedb46644f-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:03:32.218 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.218 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.219 DEBUG:boto:Method: GET 2024-04-16T20:03:32.219 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:03:32.219 DEBUG:boto:Data: 2024-04-16T20:03:32.219 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.219 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.219 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.219 DEBUG:boto:Params: {} 2024-04-16T20:03:32.219 DEBUG:boto:Token: None 2024-04-16T20:03:32.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76 2024-04-16T20:03:32.220 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SRkKwqhWh8m8io5TgZ/kRXGghho= 2024-04-16T20:03:32.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SRkKwqhWh8m8io5TgZ/kRXGghho=', 'Content-Length': '0'} 2024-04-16T20:03:32.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7ad3b7a9dbcfe2-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.262 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:03:32.262 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:03:32.262 DEBUG:boto:Method: GET 2024-04-16T20:03:32.263 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:03:32.263 DEBUG:boto:Data: 2024-04-16T20:03:32.263 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.263 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.263 DEBUG:boto:Params: {} 2024-04-16T20:03:32.263 DEBUG:boto:Token: None 2024-04-16T20:03:32.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj76 2024-04-16T20:03:32.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SRkKwqhWh8m8io5TgZ/kRXGghho= 2024-04-16T20:03:32.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SRkKwqhWh8m8io5TgZ/kRXGghho=', 'Content-Length': '0'} 2024-04-16T20:03:32.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2abc6baac5c3994-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.265 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.265 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.266 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.266 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.266 DEBUG:boto:Method: GET 2024-04-16T20:03:32.266 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.266 DEBUG:boto:Data: 2024-04-16T20:03:32.266 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.266 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.266 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.266 DEBUG:boto:Params: {} 2024-04-16T20:03:32.266 DEBUG:boto:Token: None 2024-04-16T20:03:32.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896eb67e6165d494-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.310 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.311 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.311 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.311 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.311 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.311 DEBUG:boto:Method: GET 2024-04-16T20:03:32.311 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:03:32.311 DEBUG:boto:Data: 2024-04-16T20:03:32.311 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.312 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.312 DEBUG:boto:Params: {} 2024-04-16T20:03:32.312 DEBUG:boto:Token: None 2024-04-16T20:03:32.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062af6e8f4c4acfcd-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.315 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:03:32.316 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.316 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.316 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.316 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.316 DEBUG:boto:Method: GET 2024-04-16T20:03:32.316 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.316 DEBUG:boto:Data: 2024-04-16T20:03:32.317 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.317 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.317 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.317 DEBUG:boto:Params: {} 2024-04-16T20:03:32.317 DEBUG:boto:Token: None 2024-04-16T20:03:32.317 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4= 2024-04-16T20:03:32.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4=', 'Content-Length': '0'} 2024-04-16T20:03:32.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d72191045fb1b3-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.362 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.362 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.362 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.363 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.363 DEBUG:boto:Method: GET 2024-04-16T20:03:32.363 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.363 DEBUG:boto:Data: 2024-04-16T20:03:32.363 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.363 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.363 DEBUG:boto:Params: {} 2024-04-16T20:03:32.363 DEBUG:boto:Token: None 2024-04-16T20:03:32.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4= 2024-04-16T20:03:32.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4=', 'Content-Length': '0'} 2024-04-16T20:03:32.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd21836ed1f5a0de-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.365 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.366 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.366 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.366 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.366 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.366 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.366 DEBUG:boto:Data: 2024-04-16T20:03:32.366 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.366 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.366 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.366 DEBUG:boto:Params: {} 2024-04-16T20:03:32.366 DEBUG:boto:Token: None 2024-04-16T20:03:32.367 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V4s+8QbhyGM9nxX/SdBV2Tyrtp4= 2024-04-16T20:03:32.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V4s+8QbhyGM9nxX/SdBV2Tyrtp4=', 'Content-Length': '0'} 2024-04-16T20:03:32.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe2f6c29be2e8ca-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.369 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.369 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.369 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.369 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.369 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.369 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.369 DEBUG:boto:Data: 2024-04-16T20:03:32.369 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.369 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.369 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.369 DEBUG:boto:Params: {} 2024-04-16T20:03:32.370 DEBUG:boto:Token: None 2024-04-16T20:03:32.370 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.370 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V4s+8QbhyGM9nxX/SdBV2Tyrtp4= 2024-04-16T20:03:32.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V4s+8QbhyGM9nxX/SdBV2Tyrtp4=', 'Content-Length': '0'} 2024-04-16T20:03:32.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5c06771debd57e-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:03:32.372 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.372 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.372 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.372 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.372 DEBUG:boto:Method: GET 2024-04-16T20:03:32.372 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.372 DEBUG:boto:Data: 2024-04-16T20:03:32.373 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.373 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.373 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.373 DEBUG:boto:Params: {} 2024-04-16T20:03:32.373 DEBUG:boto:Token: None 2024-04-16T20:03:32.373 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4= 2024-04-16T20:03:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4=', 'Content-Length': '0'} 2024-04-16T20:03:32.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c18851e8b35645f-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.418 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.418 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.418 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.419 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.419 DEBUG:boto:Method: GET 2024-04-16T20:03:32.419 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.419 DEBUG:boto:Data: 2024-04-16T20:03:32.419 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.419 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.419 DEBUG:boto:Params: {} 2024-04-16T20:03:32.419 DEBUG:boto:Token: None 2024-04-16T20:03:32.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:03:32.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4= 2024-04-16T20:03:32.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cMG1KLcS0yFukv5HskEfaVd2Vh4=', 'Content-Length': '0'} 2024-04-16T20:03:32.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec316a9d9004b97-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.422 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.422 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.422 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.423 DEBUG:boto:Method: GET 2024-04-16T20:03:32.423 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.423 DEBUG:boto:Data: 2024-04-16T20:03:32.423 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.423 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.423 DEBUG:boto:Params: {} 2024-04-16T20:03:32.423 DEBUG:boto:Token: None 2024-04-16T20:03:32.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f3a3292ceed68c-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.466 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.467 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.467 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.467 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.467 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.467 DEBUG:boto:Method: GET 2024-04-16T20:03:32.467 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.467 DEBUG:boto:Data: 2024-04-16T20:03:32.467 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.467 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.467 DEBUG:boto:Params: {} 2024-04-16T20:03:32.468 DEBUG:boto:Token: None 2024-04-16T20:03:32.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3ac00d3743e517-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.471 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.471 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.471 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.472 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.472 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:03:32.472 DEBUG:boto:Data: 2024-04-16T20:03:32.472 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.472 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.472 DEBUG:boto:Params: {} 2024-04-16T20:03:32.472 DEBUG:boto:Token: None 2024-04-16T20:03:32.472 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85 2024-04-16T20:03:32.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eMazsqUNDL6K9ciUeeg2vq75gRI= 2024-04-16T20:03:32.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eMazsqUNDL6K9ciUeeg2vq75gRI=', 'Content-Length': '0'} 2024-04-16T20:03:32.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005983c70ccef4932c-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.474 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.475 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.475 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.475 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:03:32.475 DEBUG:boto:Data: 2024-04-16T20:03:32.475 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.475 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.475 DEBUG:boto:Params: {} 2024-04-16T20:03:32.475 DEBUG:boto:Token: None 2024-04-16T20:03:32.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85 2024-04-16T20:03:32.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eMazsqUNDL6K9ciUeeg2vq75gRI= 2024-04-16T20:03:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eMazsqUNDL6K9ciUeeg2vq75gRI=', 'Content-Length': '0'} 2024-04-16T20:03:32.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c4b77e6cc430f2-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:03:32.478 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.478 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.478 DEBUG:boto:Method: GET 2024-04-16T20:03:32.478 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:03:32.478 DEBUG:boto:Data: 2024-04-16T20:03:32.478 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.478 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.478 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.478 DEBUG:boto:Params: {} 2024-04-16T20:03:32.478 DEBUG:boto:Token: None 2024-04-16T20:03:32.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85 2024-04-16T20:03:32.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z0HXnUfimo49CaJBtdCXoBxdp7Q= 2024-04-16T20:03:32.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z0HXnUfimo49CaJBtdCXoBxdp7Q=', 'Content-Length': '0'} 2024-04-16T20:03:32.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b778235902d38c1b-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.522 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:03:32.522 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:03:32.522 DEBUG:boto:Method: GET 2024-04-16T20:03:32.523 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:03:32.523 DEBUG:boto:Data: 2024-04-16T20:03:32.523 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.523 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.523 DEBUG:boto:Params: {} 2024-04-16T20:03:32.523 DEBUG:boto:Token: None 2024-04-16T20:03:32.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj85 2024-04-16T20:03:32.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:z0HXnUfimo49CaJBtdCXoBxdp7Q= 2024-04-16T20:03:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:z0HXnUfimo49CaJBtdCXoBxdp7Q=', 'Content-Length': '0'} 2024-04-16T20:03:32.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c05220d3a11619-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.526 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.526 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.526 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.526 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.527 DEBUG:boto:Method: GET 2024-04-16T20:03:32.527 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.527 DEBUG:boto:Data: 2024-04-16T20:03:32.527 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.527 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.527 DEBUG:boto:Params: {} 2024-04-16T20:03:32.527 DEBUG:boto:Token: None 2024-04-16T20:03:32.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b07c47d49e85c6-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.531 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.531 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.531 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.531 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.531 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.531 DEBUG:boto:Method: GET 2024-04-16T20:03:32.531 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:03:32.532 DEBUG:boto:Data: 2024-04-16T20:03:32.532 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.532 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.532 DEBUG:boto:Params: {} 2024-04-16T20:03:32.532 DEBUG:boto:Token: None 2024-04-16T20:03:32.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061db5ce689f47abf-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.535 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:03:32.536 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.536 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.536 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.536 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.536 DEBUG:boto:Method: GET 2024-04-16T20:03:32.536 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.536 DEBUG:boto:Data: 2024-04-16T20:03:32.536 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.537 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.537 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.537 DEBUG:boto:Params: {} 2024-04-16T20:03:32.537 DEBUG:boto:Token: None 2024-04-16T20:03:32.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE= 2024-04-16T20:03:32.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE=', 'Content-Length': '0'} 2024-04-16T20:03:32.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006622191cf8b6ca63-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.582 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.582 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.582 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.582 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.583 DEBUG:boto:Method: GET 2024-04-16T20:03:32.583 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.583 DEBUG:boto:Data: 2024-04-16T20:03:32.583 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.583 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.583 DEBUG:boto:Params: {} 2024-04-16T20:03:32.583 DEBUG:boto:Token: None 2024-04-16T20:03:32.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE= 2024-04-16T20:03:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE=', 'Content-Length': '0'} 2024-04-16T20:03:32.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000184c6ad4a0c23b25-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.586 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.586 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.586 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.586 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.586 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.586 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.586 DEBUG:boto:Data: 2024-04-16T20:03:32.587 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.587 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.587 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.587 DEBUG:boto:Params: {} 2024-04-16T20:03:32.587 DEBUG:boto:Token: None 2024-04-16T20:03:32.587 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x9hhOEIQIo1mdCUZN1g4huBFhzQ= 2024-04-16T20:03:32.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x9hhOEIQIo1mdCUZN1g4huBFhzQ=', 'Content-Length': '0'} 2024-04-16T20:03:32.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ee7572b707c2f13-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.589 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.589 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.590 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.590 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.590 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.590 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.590 DEBUG:boto:Data: 2024-04-16T20:03:32.590 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.590 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.590 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.590 DEBUG:boto:Params: {} 2024-04-16T20:03:32.590 DEBUG:boto:Token: None 2024-04-16T20:03:32.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x9hhOEIQIo1mdCUZN1g4huBFhzQ= 2024-04-16T20:03:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x9hhOEIQIo1mdCUZN1g4huBFhzQ=', 'Content-Length': '0'} 2024-04-16T20:03:32.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003040cf481cce779a-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:03:32.593 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.593 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.593 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.593 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.593 DEBUG:boto:Method: GET 2024-04-16T20:03:32.593 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.593 DEBUG:boto:Data: 2024-04-16T20:03:32.593 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.594 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.594 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.594 DEBUG:boto:Params: {} 2024-04-16T20:03:32.594 DEBUG:boto:Token: None 2024-04-16T20:03:32.594 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.594 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE= 2024-04-16T20:03:32.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE=', 'Content-Length': '0'} 2024-04-16T20:03:32.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c323816f93e0ee-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.638 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.638 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.638 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.639 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.639 DEBUG:boto:Method: GET 2024-04-16T20:03:32.639 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.639 DEBUG:boto:Data: 2024-04-16T20:03:32.639 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.639 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.639 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.639 DEBUG:boto:Params: {} 2024-04-16T20:03:32.639 DEBUG:boto:Token: None 2024-04-16T20:03:32.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:03:32.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE= 2024-04-16T20:03:32.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dYspMFlw5u4awEdGi3ePBYGUJyE=', 'Content-Length': '0'} 2024-04-16T20:03:32.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45899bb584add69-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.642 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.642 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.642 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.642 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.642 DEBUG:boto:Method: GET 2024-04-16T20:03:32.642 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.642 DEBUG:boto:Data: 2024-04-16T20:03:32.642 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.643 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.643 DEBUG:boto:Params: {} 2024-04-16T20:03:32.643 DEBUG:boto:Token: None 2024-04-16T20:03:32.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e19046ca07f18de5-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.646 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.646 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.646 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.646 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.646 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.646 DEBUG:boto:Method: GET 2024-04-16T20:03:32.647 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.647 DEBUG:boto:Data: 2024-04-16T20:03:32.647 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.647 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.647 DEBUG:boto:Params: {} 2024-04-16T20:03:32.647 DEBUG:boto:Token: None 2024-04-16T20:03:32.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007323e6cb277c5606-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.650 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.650 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.650 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.651 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.651 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:03:32.651 DEBUG:boto:Data: 2024-04-16T20:03:32.651 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.651 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.651 DEBUG:boto:Params: {} 2024-04-16T20:03:32.651 DEBUG:boto:Token: None 2024-04-16T20:03:32.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92 2024-04-16T20:03:32.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V2MN7NnTPhMr7sdqXikLZk363B8= 2024-04-16T20:03:32.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V2MN7NnTPhMr7sdqXikLZk363B8=', 'Content-Length': '0'} 2024-04-16T20:03:32.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe54ae48d9c3e28-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.654 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.654 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.654 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.654 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:03:32.654 DEBUG:boto:Data: 2024-04-16T20:03:32.654 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.655 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.655 DEBUG:boto:Params: {} 2024-04-16T20:03:32.655 DEBUG:boto:Token: None 2024-04-16T20:03:32.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92 2024-04-16T20:03:32.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V2MN7NnTPhMr7sdqXikLZk363B8= 2024-04-16T20:03:32.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V2MN7NnTPhMr7sdqXikLZk363B8=', 'Content-Length': '0'} 2024-04-16T20:03:32.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037aa7c89550318a4-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:03:32.657 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.657 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.657 DEBUG:boto:Method: GET 2024-04-16T20:03:32.658 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:03:32.658 DEBUG:boto:Data: 2024-04-16T20:03:32.658 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.658 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.658 DEBUG:boto:Params: {} 2024-04-16T20:03:32.658 DEBUG:boto:Token: None 2024-04-16T20:03:32.658 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92 2024-04-16T20:03:32.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:20e+RfDXSjUTRA77gUWx7x+iQps= 2024-04-16T20:03:32.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:20e+RfDXSjUTRA77gUWx7x+iQps=', 'Content-Length': '0'} 2024-04-16T20:03:32.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8e2178e837fb75-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.702 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:03:32.702 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:03:32.703 DEBUG:boto:Method: GET 2024-04-16T20:03:32.703 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:03:32.703 DEBUG:boto:Data: 2024-04-16T20:03:32.703 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.703 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.703 DEBUG:boto:Params: {} 2024-04-16T20:03:32.703 DEBUG:boto:Token: None 2024-04-16T20:03:32.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj92 2024-04-16T20:03:32.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:20e+RfDXSjUTRA77gUWx7x+iQps= 2024-04-16T20:03:32.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:20e+RfDXSjUTRA77gUWx7x+iQps=', 'Content-Length': '0'} 2024-04-16T20:03:32.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389d126fbdd761be-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.706 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.706 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.707 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.707 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.707 DEBUG:boto:Method: GET 2024-04-16T20:03:32.707 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.707 DEBUG:boto:Data: 2024-04-16T20:03:32.707 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.707 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.707 DEBUG:boto:Params: {} 2024-04-16T20:03:32.708 DEBUG:boto:Token: None 2024-04-16T20:03:32.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b790e8f43a7cb9-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.754 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.755 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.755 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.755 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.755 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.755 DEBUG:boto:Method: GET 2024-04-16T20:03:32.755 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:03:32.755 DEBUG:boto:Data: 2024-04-16T20:03:32.755 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.756 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.756 DEBUG:boto:Params: {} 2024-04-16T20:03:32.756 DEBUG:boto:Token: None 2024-04-16T20:03:32.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92c41ec4c8a310e-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.759 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:03:32.760 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.760 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.760 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.760 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.760 DEBUG:boto:Method: GET 2024-04-16T20:03:32.760 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.760 DEBUG:boto:Data: 2024-04-16T20:03:32.760 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.761 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.761 DEBUG:boto:Params: {} 2024-04-16T20:03:32.761 DEBUG:boto:Token: None 2024-04-16T20:03:32.761 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs= 2024-04-16T20:03:32.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs=', 'Content-Length': '0'} 2024-04-16T20:03:32.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d63905cf1a7b1ad-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.806 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.807 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.807 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.807 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.807 DEBUG:boto:Method: GET 2024-04-16T20:03:32.807 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.807 DEBUG:boto:Data: 2024-04-16T20:03:32.807 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.808 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.808 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.808 DEBUG:boto:Params: {} 2024-04-16T20:03:32.808 DEBUG:boto:Token: None 2024-04-16T20:03:32.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs= 2024-04-16T20:03:32.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs=', 'Content-Length': '0'} 2024-04-16T20:03:32.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bccc918f42e5b6-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.811 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.811 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.811 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.811 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.811 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.812 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.812 DEBUG:boto:Data: 2024-04-16T20:03:32.812 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.812 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.812 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.812 DEBUG:boto:Params: {} 2024-04-16T20:03:32.812 DEBUG:boto:Token: None 2024-04-16T20:03:32.812 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A7pc0n/7oVsKe38UDL0Zxnp1Das= 2024-04-16T20:03:32.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A7pc0n/7oVsKe38UDL0Zxnp1Das=', 'Content-Length': '0'} 2024-04-16T20:03:32.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e4837c9b570b7d-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.814 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.815 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.815 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.815 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.815 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.815 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.815 DEBUG:boto:Data: 2024-04-16T20:03:32.815 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.815 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.815 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.815 DEBUG:boto:Params: {} 2024-04-16T20:03:32.815 DEBUG:boto:Token: None 2024-04-16T20:03:32.816 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:A7pc0n/7oVsKe38UDL0Zxnp1Das= 2024-04-16T20:03:32.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A7pc0n/7oVsKe38UDL0Zxnp1Das=', 'Content-Length': '0'} 2024-04-16T20:03:32.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3709d143ef24ecc-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:03:32.818 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.818 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.818 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.818 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.818 DEBUG:boto:Method: GET 2024-04-16T20:03:32.818 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.818 DEBUG:boto:Data: 2024-04-16T20:03:32.819 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.819 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.819 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.819 DEBUG:boto:Params: {} 2024-04-16T20:03:32.819 DEBUG:boto:Token: None 2024-04-16T20:03:32.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs= 2024-04-16T20:03:32.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs=', 'Content-Length': '0'} 2024-04-16T20:03:32.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da054b59bb3053e-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.862 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.862 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.862 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.863 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.863 DEBUG:boto:Method: GET 2024-04-16T20:03:32.863 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.863 DEBUG:boto:Data: 2024-04-16T20:03:32.863 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.863 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.863 DEBUG:boto:Params: {} 2024-04-16T20:03:32.863 DEBUG:boto:Token: None 2024-04-16T20:03:32.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:03:32.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs= 2024-04-16T20:03:32.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kZ5b4+12FcHM7u7l8yMFKwIgkqs=', 'Content-Length': '0'} 2024-04-16T20:03:32.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7797dddb48acc27-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.866 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.866 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.866 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.866 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.866 DEBUG:boto:Method: GET 2024-04-16T20:03:32.866 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.867 DEBUG:boto:Data: 2024-04-16T20:03:32.867 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.867 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.867 DEBUG:boto:Params: {} 2024-04-16T20:03:32.867 DEBUG:boto:Token: None 2024-04-16T20:03:32.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd2f120266075d5-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.910 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.911 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.911 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.911 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.911 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.911 DEBUG:boto:Method: GET 2024-04-16T20:03:32.911 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.911 DEBUG:boto:Data: 2024-04-16T20:03:32.911 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.911 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.912 DEBUG:boto:Params: {} 2024-04-16T20:03:32.912 DEBUG:boto:Token: None 2024-04-16T20:03:32.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000874550be125ce737-00661ed994-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.915 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:32.916 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.916 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.916 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.916 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:03:32.916 DEBUG:boto:Data: 2024-04-16T20:03:32.916 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.916 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.916 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.916 DEBUG:boto:Params: {} 2024-04-16T20:03:32.916 DEBUG:boto:Token: None 2024-04-16T20:03:32.916 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95 2024-04-16T20:03:32.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ylm5xvV5aAZ1ZwwxjqHvkzaI6lE= 2024-04-16T20:03:32.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ylm5xvV5aAZ1ZwwxjqHvkzaI6lE=', 'Content-Length': '0'} 2024-04-16T20:03:32.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b522b89920c7ab-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.919 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.919 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.919 DEBUG:boto:Method: HEAD 2024-04-16T20:03:32.919 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:03:32.919 DEBUG:boto:Data: 2024-04-16T20:03:32.919 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.919 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.920 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.920 DEBUG:boto:Params: {} 2024-04-16T20:03:32.920 DEBUG:boto:Token: None 2024-04-16T20:03:32.920 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95 2024-04-16T20:03:32.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ylm5xvV5aAZ1ZwwxjqHvkzaI6lE= 2024-04-16T20:03:32.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ylm5xvV5aAZ1ZwwxjqHvkzaI6lE=', 'Content-Length': '0'} 2024-04-16T20:03:32.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b5342a7a12eb13f-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:03:32.922 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.922 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.922 DEBUG:boto:Method: GET 2024-04-16T20:03:32.923 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:03:32.923 DEBUG:boto:Data: 2024-04-16T20:03:32.923 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.923 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.923 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.923 DEBUG:boto:Params: {} 2024-04-16T20:03:32.923 DEBUG:boto:Token: None 2024-04-16T20:03:32.923 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95 2024-04-16T20:03:32.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4P7/uuy4EB/RUp1s0hVsXQkrYBg= 2024-04-16T20:03:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4P7/uuy4EB/RUp1s0hVsXQkrYBg=', 'Content-Length': '0'} 2024-04-16T20:03:32.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075078bd0b0bc1db3-00661ed994-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.966 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:03:32.966 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:03:32.966 DEBUG:boto:Method: GET 2024-04-16T20:03:32.967 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:03:32.967 DEBUG:boto:Data: 2024-04-16T20:03:32.967 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:32.967 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.967 DEBUG:boto:Params: {} 2024-04-16T20:03:32.967 DEBUG:boto:Token: None 2024-04-16T20:03:32.967 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/obj95 2024-04-16T20:03:32.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4P7/uuy4EB/RUp1s0hVsXQkrYBg= 2024-04-16T20:03:32.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4P7/uuy4EB/RUp1s0hVsXQkrYBg=', 'Content-Length': '0'} 2024-04-16T20:03:32.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af4d261f26e49cf-00661ed994-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:32.970 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:32.970 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:32.970 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.970 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.970 DEBUG:boto:Method: GET 2024-04-16T20:03:32.970 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:03:32.970 DEBUG:boto:Data: 2024-04-16T20:03:32.970 DEBUG:boto:Headers: {} 2024-04-16T20:03:32.970 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:03:32.971 DEBUG:boto:Port: 8000 2024-04-16T20:03:32.971 DEBUG:boto:Params: {} 2024-04-16T20:03:32.971 DEBUG:boto:Token: None 2024-04-16T20:03:32.971 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:32 GMT /yrjefh-52/ 2024-04-16T20:03:32.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM= 2024-04-16T20:03:32.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jgoxljzJf3vzc7G9AnJ2cVIUVKM=', 'Content-Length': '0'} 2024-04-16T20:03:32.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5978f3f6c1f9de6-00661ed994-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.014 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:33.015 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.015 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.015 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:33.015 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:03:33.015 DEBUG:boto:Method: GET 2024-04-16T20:03:33.015 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:03:33.015 DEBUG:boto:Data: 2024-04-16T20:03:33.015 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.016 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.016 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.016 DEBUG:boto:Params: {} 2024-04-16T20:03:33.016 DEBUG:boto:Token: None 2024-04-16T20:03:33.016 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HkSjAHAFjTVBZdSFDqyw0bm7Q9s= 2024-04-16T20:03:33.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HkSjAHAFjTVBZdSFDqyw0bm7Q9s=', 'Content-Length': '0'} 2024-04-16T20:03:33.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072274c241b542bbd-00661ed995-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.019 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:03:33.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:03:33.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:03:33.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:03:33.020 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:03:33.078 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:03:33.078 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:03:33.153 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:03:33.153 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T20:03:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 20 2024-04-16T20:03:33.229 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.229 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.230 DEBUG:boto:Method: HEAD 2024-04-16T20:03:33.230 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:33.230 DEBUG:boto:Data: 2024-04-16T20:03:33.230 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.230 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.230 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.230 DEBUG:boto:Params: {} 2024-04-16T20:03:33.230 DEBUG:boto:Token: None 2024-04-16T20:03:33.230 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q= 2024-04-16T20:03:33.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q=', 'Content-Length': '0'} 2024-04-16T20:03:33.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edc6a71e62f4b059-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.234 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:03:33.234 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:03:33.234 DEBUG:boto:Method: PUT 2024-04-16T20:03:33.234 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:03:33.234 DEBUG:boto:Data: 2024-04-16T20:03:33.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:03:33.234 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.234 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.234 DEBUG:boto:Params: {} 2024-04-16T20:03:33.235 DEBUG:boto:Token: None 2024-04-16T20:03:33.235 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/obj29 2024-04-16T20:03:33.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djk3UR3P0let/MAHfLho8zTrR2I= 2024-04-16T20:03:33.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djk3UR3P0let/MAHfLho8zTrR2I='} 2024-04-16T20:03:33.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007222296d7914bf77-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.240 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.241 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.241 DEBUG:boto:Method: HEAD 2024-04-16T20:03:33.241 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:33.241 DEBUG:boto:Data: 2024-04-16T20:03:33.241 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.241 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.241 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.241 DEBUG:boto:Params: {} 2024-04-16T20:03:33.241 DEBUG:boto:Token: None 2024-04-16T20:03:33.241 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.242 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q= 2024-04-16T20:03:33.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q=', 'Content-Length': '0'} 2024-04-16T20:03:33.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093ccacacf588d427-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.245 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:03:33.245 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:03:33.245 DEBUG:boto:Method: PUT 2024-04-16T20:03:33.245 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:03:33.245 DEBUG:boto:Data: 2024-04-16T20:03:33.245 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:03:33.245 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.245 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.245 DEBUG:boto:Params: {} 2024-04-16T20:03:33.246 DEBUG:boto:Token: None 2024-04-16T20:03:33.246 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/obj58 2024-04-16T20:03:33.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yGd323RyJPF4O6pR2JqScuXvl8U= 2024-04-16T20:03:33.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yGd323RyJPF4O6pR2JqScuXvl8U='} 2024-04-16T20:03:33.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000112342b57f547c31-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.291 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.291 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.291 DEBUG:boto:Method: HEAD 2024-04-16T20:03:33.291 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:33.291 DEBUG:boto:Data: 2024-04-16T20:03:33.291 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.291 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.291 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.291 DEBUG:boto:Params: {} 2024-04-16T20:03:33.292 DEBUG:boto:Token: None 2024-04-16T20:03:33.292 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q= 2024-04-16T20:03:33.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q=', 'Content-Length': '0'} 2024-04-16T20:03:33.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036de9b71c5e50088-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.295 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:03:33.295 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:03:33.295 DEBUG:boto:Method: PUT 2024-04-16T20:03:33.295 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:03:33.295 DEBUG:boto:Data: 2024-04-16T20:03:33.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:03:33.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.295 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.295 DEBUG:boto:Params: {} 2024-04-16T20:03:33.295 DEBUG:boto:Token: None 2024-04-16T20:03:33.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/obj87 2024-04-16T20:03:33.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:G7eKQ77rRVnAgVVxVqIjde4okuA= 2024-04-16T20:03:33.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:G7eKQ77rRVnAgVVxVqIjde4okuA='} 2024-04-16T20:03:33.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d42b01d0d65fe388-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.302 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.302 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.302 DEBUG:boto:Method: HEAD 2024-04-16T20:03:33.302 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:33.302 DEBUG:boto:Data: 2024-04-16T20:03:33.302 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.302 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.302 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.303 DEBUG:boto:Params: {} 2024-04-16T20:03:33.303 DEBUG:boto:Token: None 2024-04-16T20:03:33.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q= 2024-04-16T20:03:33.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q=', 'Content-Length': '0'} 2024-04-16T20:03:33.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e63a0e9b42454817-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.305 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:03:33.306 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:03:33.306 DEBUG:boto:Method: PUT 2024-04-16T20:03:33.306 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:03:33.306 DEBUG:boto:Data: 2024-04-16T20:03:33.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:03:33.306 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.306 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.306 DEBUG:boto:Params: {} 2024-04-16T20:03:33.306 DEBUG:boto:Token: None 2024-04-16T20:03:33.306 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/obj116 2024-04-16T20:03:33.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FUrdLLHxJtrWdz9J9jgm855LTCU= 2024-04-16T20:03:33.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FUrdLLHxJtrWdz9J9jgm855LTCU='} 2024-04-16T20:03:33.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc70230dda5d1a86-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.351 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:03:33.351 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:03:33.351 DEBUG:boto:Method: HEAD 2024-04-16T20:03:33.351 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:03:33.351 DEBUG:boto:Data: 2024-04-16T20:03:33.351 DEBUG:boto:Headers: {} 2024-04-16T20:03:33.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.351 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.351 DEBUG:boto:Params: {} 2024-04-16T20:03:33.352 DEBUG:boto:Token: None 2024-04-16T20:03:33.352 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/ 2024-04-16T20:03:33.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q= 2024-04-16T20:03:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQkkYKv+LhbvZtLC+3Rxi+0VC4Q=', 'Content-Length': '0'} 2024-04-16T20:03:33.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000110276132d222d96-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.355 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:03:33.355 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:03:33.355 DEBUG:boto:Method: PUT 2024-04-16T20:03:33.355 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:03:33.355 DEBUG:boto:Data: 2024-04-16T20:03:33.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:03:33.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:03:33.356 DEBUG:boto:Port: 8000 2024-04-16T20:03:33.356 DEBUG:boto:Params: {} 2024-04-16T20:03:33.356 DEBUG:boto:Token: None 2024-04-16T20:03:33.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:03:33 GMT /yrjefh-52/obj145 2024-04-16T20:03:33.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zj6RQb67wPqIFeya/YDmvCqPYjU= 2024-04-16T20:03:33.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:03:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zj6RQb67wPqIFeya/YDmvCqPYjU='} 2024-04-16T20:03:33.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc24455791fb33ad-00661ed995-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:03:33.361 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:03:33.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:03:33.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:03:33.362 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:03:33.424 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:03:33.424 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:03:33.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:33.490+0000 7f5546033800 1 waiting to reach latest gen 4, on 3.. 2024-04-16T20:03:38.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:38.493+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:38.487 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:03:38.488 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:43.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:43.494+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:43.490 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:43.490 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:48.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:48.497+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:48.492 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:48.492 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:53.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:53.498+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:53.493 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:53.493 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:03:58.499+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:03:58.495 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:03:58.495 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:03.502+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:03.497 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:03.497 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:08.504+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:08.499 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:08.499 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:13.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:13.506+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:13.501 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:13.501 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:18.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:18.508+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:18.503 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:18.503 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:23.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:23.510+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:23.505 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:23.505 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:28.512+0000 7f5546033800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:28.506 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:28.506 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:33.514+0000 7f5546033800 1 bucket sync caught up with source: 2024-04-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [00000000002.177.6, 00000000001.96.6, , , , , 00000000001.65.6, , , , , , , , , , , 00000000002.61.6, 00000000001.62.6, , , , , , , , , , ] 2024-04-16T20:04:33.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:33.514+0000 7f5546033800 0 bucket checkpoint complete 2024-04-16T20:04:33.535 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:04:33.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:04:33.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:04:33.544 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:04:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:04:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:04:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:04:33.676+0000 7fcf79b33800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:33.670 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000003.281.6, 00000000003.233.6, 00000000001.188.6, 00000000001.300.6, 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:04:33.670 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000003.281.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:04:38.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:04:38.677+0000 7fcf79b33800 1 bucket sync caught up with source: 2024-04-16T20:04:38.672 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000003.281.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:04:38.672 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.163.6, 00000000002.172.6, 00000000001.91.6, 00000000003.281.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000002.165.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:04:38.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:04:38.677+0000 7fcf79b33800 0 bucket checkpoint complete 2024-04-16T20:04:38.698 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:04:38.699 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.699 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.699 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.699 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:38.699 DEBUG:boto:Data: 2024-04-16T20:04:38.699 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.700 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.700 DEBUG:boto:Params: {} 2024-04-16T20:04:38.700 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:04:38.700 DEBUG:boto:Token: None 2024-04-16T20:04:38.701 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.701 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fIxrbp1/B0RqjzknGNEaCtSY92k= 2024-04-16T20:04:38.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fIxrbp1/B0RqjzknGNEaCtSY92k=', 'Content-Length': '0'} 2024-04-16T20:04:38.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002430225e1780b196-00661ed9d6-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.706 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.706 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.707 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.707 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:38.707 DEBUG:boto:Data: 2024-04-16T20:04:38.707 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.707 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.707 DEBUG:boto:Params: {} 2024-04-16T20:04:38.707 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:04:38.707 DEBUG:boto:Token: None 2024-04-16T20:04:38.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fIxrbp1/B0RqjzknGNEaCtSY92k= 2024-04-16T20:04:38.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fIxrbp1/B0RqjzknGNEaCtSY92k=', 'Content-Length': '0'} 2024-04-16T20:04:38.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c155eac459fb7f74-00661ed9d6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.713 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:04:38.713 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.713 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.713 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.713 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.713 DEBUG:boto:Method: GET 2024-04-16T20:04:38.713 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.713 DEBUG:boto:Data: 2024-04-16T20:04:38.713 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.713 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.714 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.714 DEBUG:boto:Params: {} 2024-04-16T20:04:38.714 DEBUG:boto:Token: None 2024-04-16T20:04:38.714 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/?versions 2024-04-16T20:04:38.714 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE= 2024-04-16T20:04:38.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE=', 'Content-Length': '0'} 2024-04-16T20:04:38.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000471ebdc5ca003b22-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.717 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:04:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:04:38.720 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:04:38.721 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.721 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.721 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.721 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.721 DEBUG:boto:Method: GET 2024-04-16T20:04:38.721 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.721 DEBUG:boto:Data: 2024-04-16T20:04:38.721 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.721 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.721 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.721 DEBUG:boto:Params: {} 2024-04-16T20:04:38.722 DEBUG:boto:Token: None 2024-04-16T20:04:38.722 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/?versions 2024-04-16T20:04:38.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE= 2024-04-16T20:04:38.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE=', 'Content-Length': '0'} 2024-04-16T20:04:38.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb6e43a57ab4cef-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.725 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:04:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:04:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:04:38.729 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.729 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.729 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.729 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.729 DEBUG:boto:Method: GET 2024-04-16T20:04:38.729 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.729 DEBUG:boto:Data: 2024-04-16T20:04:38.729 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.729 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.729 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.729 DEBUG:boto:Params: {} 2024-04-16T20:04:38.730 DEBUG:boto:Token: None 2024-04-16T20:04:38.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/?versions 2024-04-16T20:04:38.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE= 2024-04-16T20:04:38.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE=', 'Content-Length': '0'} 2024-04-16T20:04:38.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009770facf1702d3b7-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.733 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.734 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.734 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.734 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.734 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.734 DEBUG:boto:Method: GET 2024-04-16T20:04:38.734 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:04:38.735 DEBUG:boto:Data: 2024-04-16T20:04:38.735 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.735 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.735 DEBUG:boto:Params: {} 2024-04-16T20:04:38.735 DEBUG:boto:Token: None 2024-04-16T20:04:38.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/?versions 2024-04-16T20:04:38.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE= 2024-04-16T20:04:38.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ouC4DUq61elt0vevaNOg8LWnrFE=', 'Content-Length': '0'} 2024-04-16T20:04:38.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa9c23625778d93-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.739 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:04:38.740 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.740 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.741 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.741 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.741 DEBUG:boto:Method: GET 2024-04-16T20:04:38.741 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.741 DEBUG:boto:Data: 2024-04-16T20:04:38.741 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.741 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.741 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.741 DEBUG:boto:Params: {} 2024-04-16T20:04:38.741 DEBUG:boto:Token: None 2024-04-16T20:04:38.742 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.742 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA= 2024-04-16T20:04:38.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA=', 'Content-Length': '0'} 2024-04-16T20:04:38.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdbbf4cc069bf6c6-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.744 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.744 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.744 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.744 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.744 DEBUG:boto:Method: GET 2024-04-16T20:04:38.745 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.745 DEBUG:boto:Data: 2024-04-16T20:04:38.745 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.745 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.745 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.745 DEBUG:boto:Params: {} 2024-04-16T20:04:38.745 DEBUG:boto:Token: None 2024-04-16T20:04:38.745 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA= 2024-04-16T20:04:38.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA=', 'Content-Length': '0'} 2024-04-16T20:04:38.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0643662c4f5dca-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.748 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.748 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.748 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.748 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.748 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.748 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.748 DEBUG:boto:Data: 2024-04-16T20:04:38.748 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.748 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.748 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.748 DEBUG:boto:Params: {} 2024-04-16T20:04:38.749 DEBUG:boto:Token: None 2024-04-16T20:04:38.749 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.749 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zn+7xc3jc3/KAJEtFenKomvk7E= 2024-04-16T20:04:38.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zn+7xc3jc3/KAJEtFenKomvk7E=', 'Content-Length': '0'} 2024-04-16T20:04:38.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d7e98658ebc7fb-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.751 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.751 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.751 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.751 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.751 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.751 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.752 DEBUG:boto:Data: 2024-04-16T20:04:38.752 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.752 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.752 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.752 DEBUG:boto:Params: {} 2024-04-16T20:04:38.752 DEBUG:boto:Token: None 2024-04-16T20:04:38.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9zn+7xc3jc3/KAJEtFenKomvk7E= 2024-04-16T20:04:38.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9zn+7xc3jc3/KAJEtFenKomvk7E=', 'Content-Length': '0'} 2024-04-16T20:04:38.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3678c8fd8e8522-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:04:38.754 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.754 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.755 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.755 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.755 DEBUG:boto:Method: GET 2024-04-16T20:04:38.755 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.755 DEBUG:boto:Data: 2024-04-16T20:04:38.755 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.755 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.755 DEBUG:boto:Params: {} 2024-04-16T20:04:38.755 DEBUG:boto:Token: None 2024-04-16T20:04:38.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA= 2024-04-16T20:04:38.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA=', 'Content-Length': '0'} 2024-04-16T20:04:38.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d9939a8f6e7f63-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.798 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.798 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.799 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.799 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.799 DEBUG:boto:Method: GET 2024-04-16T20:04:38.799 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.799 DEBUG:boto:Data: 2024-04-16T20:04:38.799 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.799 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.799 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.799 DEBUG:boto:Params: {} 2024-04-16T20:04:38.799 DEBUG:boto:Token: None 2024-04-16T20:04:38.800 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:04:38.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA= 2024-04-16T20:04:38.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4toYcp0GR5bbdEsOaPeYN6y2jiA=', 'Content-Length': '0'} 2024-04-16T20:04:38.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748015c9b82e6a22-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.802 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.802 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.802 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.802 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.802 DEBUG:boto:Method: GET 2024-04-16T20:04:38.802 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.802 DEBUG:boto:Data: 2024-04-16T20:04:38.803 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.803 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.803 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.803 DEBUG:boto:Params: {} 2024-04-16T20:04:38.803 DEBUG:boto:Token: None 2024-04-16T20:04:38.803 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b9d6bb0c9b8200-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.805 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.806 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.806 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.806 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.806 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.806 DEBUG:boto:Method: GET 2024-04-16T20:04:38.806 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.806 DEBUG:boto:Data: 2024-04-16T20:04:38.807 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.807 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.807 DEBUG:boto:Params: {} 2024-04-16T20:04:38.807 DEBUG:boto:Token: None 2024-04-16T20:04:38.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8046d117531183-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.810 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.811 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.811 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.811 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.811 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:04:38.811 DEBUG:boto:Data: 2024-04-16T20:04:38.811 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.811 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.811 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.811 DEBUG:boto:Params: {} 2024-04-16T20:04:38.811 DEBUG:boto:Token: None 2024-04-16T20:04:38.812 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11 2024-04-16T20:04:38.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XLylKdjpRfFTIdwOiXDEoWnp3T4= 2024-04-16T20:04:38.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XLylKdjpRfFTIdwOiXDEoWnp3T4=', 'Content-Length': '0'} 2024-04-16T20:04:38.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23be67bec457bfa-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.813 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.814 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.814 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.814 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:04:38.814 DEBUG:boto:Data: 2024-04-16T20:04:38.814 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.814 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.814 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.814 DEBUG:boto:Params: {} 2024-04-16T20:04:38.814 DEBUG:boto:Token: None 2024-04-16T20:04:38.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11 2024-04-16T20:04:38.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XLylKdjpRfFTIdwOiXDEoWnp3T4= 2024-04-16T20:04:38.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XLylKdjpRfFTIdwOiXDEoWnp3T4=', 'Content-Length': '0'} 2024-04-16T20:04:38.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033397a41382d9346-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:04:38.816 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.817 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.817 DEBUG:boto:Method: GET 2024-04-16T20:04:38.817 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:04:38.817 DEBUG:boto:Data: 2024-04-16T20:04:38.817 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.817 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.817 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.817 DEBUG:boto:Params: {} 2024-04-16T20:04:38.817 DEBUG:boto:Token: None 2024-04-16T20:04:38.817 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11 2024-04-16T20:04:38.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TI0QNwR2Bxm4OEKQxzMrROvvV84= 2024-04-16T20:04:38.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TI0QNwR2Bxm4OEKQxzMrROvvV84=', 'Content-Length': '0'} 2024-04-16T20:04:38.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed34dfe81fe03682-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.820 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:04:38.820 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:04:38.820 DEBUG:boto:Method: GET 2024-04-16T20:04:38.820 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:04:38.820 DEBUG:boto:Data: 2024-04-16T20:04:38.820 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.820 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.820 DEBUG:boto:Params: {} 2024-04-16T20:04:38.820 DEBUG:boto:Token: None 2024-04-16T20:04:38.821 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj11 2024-04-16T20:04:38.821 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TI0QNwR2Bxm4OEKQxzMrROvvV84= 2024-04-16T20:04:38.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TI0QNwR2Bxm4OEKQxzMrROvvV84=', 'Content-Length': '0'} 2024-04-16T20:04:38.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a108321cbe05c41a-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.823 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.823 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.823 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.823 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.823 DEBUG:boto:Method: GET 2024-04-16T20:04:38.823 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.823 DEBUG:boto:Data: 2024-04-16T20:04:38.823 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.823 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.823 DEBUG:boto:Params: {} 2024-04-16T20:04:38.824 DEBUG:boto:Token: None 2024-04-16T20:04:38.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953d810c9cd317bc-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.826 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.827 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.827 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.827 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.827 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.827 DEBUG:boto:Method: GET 2024-04-16T20:04:38.827 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:04:38.828 DEBUG:boto:Data: 2024-04-16T20:04:38.828 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.828 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.828 DEBUG:boto:Params: {} 2024-04-16T20:04:38.828 DEBUG:boto:Token: None 2024-04-16T20:04:38.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f99b4bc81eaf53a0-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.832 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:04:38.832 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.833 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.833 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.833 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.833 DEBUG:boto:Method: GET 2024-04-16T20:04:38.833 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.833 DEBUG:boto:Data: 2024-04-16T20:04:38.833 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.833 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.833 DEBUG:boto:Params: {} 2024-04-16T20:04:38.834 DEBUG:boto:Token: None 2024-04-16T20:04:38.834 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA= 2024-04-16T20:04:38.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA=', 'Content-Length': '0'} 2024-04-16T20:04:38.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c096d606db228769-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.836 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.836 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.836 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.837 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.837 DEBUG:boto:Method: GET 2024-04-16T20:04:38.837 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.837 DEBUG:boto:Data: 2024-04-16T20:04:38.837 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.837 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.837 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.837 DEBUG:boto:Params: {} 2024-04-16T20:04:38.837 DEBUG:boto:Token: None 2024-04-16T20:04:38.837 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.837 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA= 2024-04-16T20:04:38.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA=', 'Content-Length': '0'} 2024-04-16T20:04:38.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000776a3b85e12c47d-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.882 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.882 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.882 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.882 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.883 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.883 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.883 DEBUG:boto:Data: 2024-04-16T20:04:38.883 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.883 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.883 DEBUG:boto:Params: {} 2024-04-16T20:04:38.883 DEBUG:boto:Token: None 2024-04-16T20:04:38.883 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QZdO4FaWe6XLFPI0xRms5hw6Kz8= 2024-04-16T20:04:38.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QZdO4FaWe6XLFPI0xRms5hw6Kz8=', 'Content-Length': '0'} 2024-04-16T20:04:38.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b926b015d14cf377-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.886 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.886 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.886 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.886 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.886 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.886 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.886 DEBUG:boto:Data: 2024-04-16T20:04:38.886 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.886 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.886 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.886 DEBUG:boto:Params: {} 2024-04-16T20:04:38.887 DEBUG:boto:Token: None 2024-04-16T20:04:38.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QZdO4FaWe6XLFPI0xRms5hw6Kz8= 2024-04-16T20:04:38.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QZdO4FaWe6XLFPI0xRms5hw6Kz8=', 'Content-Length': '0'} 2024-04-16T20:04:38.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f92dfb3092cdb1-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:04:38.889 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.889 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.889 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.889 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.890 DEBUG:boto:Method: GET 2024-04-16T20:04:38.890 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.890 DEBUG:boto:Data: 2024-04-16T20:04:38.890 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.890 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.890 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.890 DEBUG:boto:Params: {} 2024-04-16T20:04:38.890 DEBUG:boto:Token: None 2024-04-16T20:04:38.890 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.890 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA= 2024-04-16T20:04:38.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA=', 'Content-Length': '0'} 2024-04-16T20:04:38.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900dec52c3f7284c-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.893 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.893 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.893 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.893 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.893 DEBUG:boto:Method: GET 2024-04-16T20:04:38.893 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.893 DEBUG:boto:Data: 2024-04-16T20:04:38.894 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.894 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.894 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.894 DEBUG:boto:Params: {} 2024-04-16T20:04:38.894 DEBUG:boto:Token: None 2024-04-16T20:04:38.894 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:04:38.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA= 2024-04-16T20:04:38.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Au391SlcvuUsECqJMcZIKDw07uA=', 'Content-Length': '0'} 2024-04-16T20:04:38.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f087649ad035a2-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.897 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.897 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.897 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.897 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.897 DEBUG:boto:Method: GET 2024-04-16T20:04:38.897 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.897 DEBUG:boto:Data: 2024-04-16T20:04:38.897 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.897 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.897 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.898 DEBUG:boto:Params: {} 2024-04-16T20:04:38.898 DEBUG:boto:Token: None 2024-04-16T20:04:38.898 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.898 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd96df25a1d014b5-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.900 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.901 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.901 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.901 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.901 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.901 DEBUG:boto:Method: GET 2024-04-16T20:04:38.901 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.901 DEBUG:boto:Data: 2024-04-16T20:04:38.901 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.901 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.902 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.902 DEBUG:boto:Params: {} 2024-04-16T20:04:38.902 DEBUG:boto:Token: None 2024-04-16T20:04:38.902 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063421c22058335e2-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.905 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.906 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.906 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.906 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.906 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:04:38.906 DEBUG:boto:Data: 2024-04-16T20:04:38.906 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.906 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.906 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.906 DEBUG:boto:Params: {} 2024-04-16T20:04:38.907 DEBUG:boto:Token: None 2024-04-16T20:04:38.907 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115 2024-04-16T20:04:38.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZVq1cR08d62EPDMmy/j+cdx1F48= 2024-04-16T20:04:38.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZVq1cR08d62EPDMmy/j+cdx1F48=', 'Content-Length': '0'} 2024-04-16T20:04:38.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c65e5d63a22b05-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.909 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.909 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.909 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.909 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:04:38.909 DEBUG:boto:Data: 2024-04-16T20:04:38.909 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.909 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.910 DEBUG:boto:Params: {} 2024-04-16T20:04:38.910 DEBUG:boto:Token: None 2024-04-16T20:04:38.910 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115 2024-04-16T20:04:38.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZVq1cR08d62EPDMmy/j+cdx1F48= 2024-04-16T20:04:38.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZVq1cR08d62EPDMmy/j+cdx1F48=', 'Content-Length': '0'} 2024-04-16T20:04:38.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1f88ecda2f2891-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:04:38.912 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.912 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.913 DEBUG:boto:Method: GET 2024-04-16T20:04:38.913 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:04:38.913 DEBUG:boto:Data: 2024-04-16T20:04:38.913 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.913 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.913 DEBUG:boto:Params: {} 2024-04-16T20:04:38.913 DEBUG:boto:Token: None 2024-04-16T20:04:38.913 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115 2024-04-16T20:04:38.913 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4lSfRHhuMqKi0t1KU5Y3RwSf12w= 2024-04-16T20:04:38.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4lSfRHhuMqKi0t1KU5Y3RwSf12w=', 'Content-Length': '0'} 2024-04-16T20:04:38.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93ca3b51350c31b-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.916 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:04:38.916 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:04:38.916 DEBUG:boto:Method: GET 2024-04-16T20:04:38.916 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:04:38.916 DEBUG:boto:Data: 2024-04-16T20:04:38.916 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.916 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.917 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.917 DEBUG:boto:Params: {} 2024-04-16T20:04:38.917 DEBUG:boto:Token: None 2024-04-16T20:04:38.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj115 2024-04-16T20:04:38.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4lSfRHhuMqKi0t1KU5Y3RwSf12w= 2024-04-16T20:04:38.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4lSfRHhuMqKi0t1KU5Y3RwSf12w=', 'Content-Length': '0'} 2024-04-16T20:04:38.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f59429c6ce38b68f-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.919 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.919 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.920 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.920 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.920 DEBUG:boto:Method: GET 2024-04-16T20:04:38.920 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.920 DEBUG:boto:Data: 2024-04-16T20:04:38.920 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.920 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.920 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.920 DEBUG:boto:Params: {} 2024-04-16T20:04:38.920 DEBUG:boto:Token: None 2024-04-16T20:04:38.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a74be7b23672a2-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.966 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.966 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.967 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.967 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.967 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.967 DEBUG:boto:Method: GET 2024-04-16T20:04:38.967 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:04:38.967 DEBUG:boto:Data: 2024-04-16T20:04:38.967 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.967 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.967 DEBUG:boto:Params: {} 2024-04-16T20:04:38.967 DEBUG:boto:Token: None 2024-04-16T20:04:38.967 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7077b799c893b8-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.971 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:04:38.971 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.971 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.972 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.972 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.972 DEBUG:boto:Method: GET 2024-04-16T20:04:38.972 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.972 DEBUG:boto:Data: 2024-04-16T20:04:38.972 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.972 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.972 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.972 DEBUG:boto:Params: {} 2024-04-16T20:04:38.972 DEBUG:boto:Token: None 2024-04-16T20:04:38.973 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA= 2024-04-16T20:04:38.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA=', 'Content-Length': '0'} 2024-04-16T20:04:38.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a5b78930d27d60-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.975 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.975 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.975 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.975 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.975 DEBUG:boto:Method: GET 2024-04-16T20:04:38.975 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.975 DEBUG:boto:Data: 2024-04-16T20:04:38.976 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.976 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.976 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.976 DEBUG:boto:Params: {} 2024-04-16T20:04:38.976 DEBUG:boto:Token: None 2024-04-16T20:04:38.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA= 2024-04-16T20:04:38.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA=', 'Content-Length': '0'} 2024-04-16T20:04:38.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f2c9f3a188b03e-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.978 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.978 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.978 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.978 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.979 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.979 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.979 DEBUG:boto:Data: 2024-04-16T20:04:38.979 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.979 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.979 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.979 DEBUG:boto:Params: {} 2024-04-16T20:04:38.979 DEBUG:boto:Token: None 2024-04-16T20:04:38.979 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jMwrLeBk4mTqHXGSy1/C4pCiHVU= 2024-04-16T20:04:38.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMwrLeBk4mTqHXGSy1/C4pCiHVU=', 'Content-Length': '0'} 2024-04-16T20:04:38.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b695053308d04f0-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.982 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.982 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.982 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.982 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.982 DEBUG:boto:Method: HEAD 2024-04-16T20:04:38.982 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.982 DEBUG:boto:Data: 2024-04-16T20:04:38.982 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.982 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.983 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.983 DEBUG:boto:Params: {} 2024-04-16T20:04:38.983 DEBUG:boto:Token: None 2024-04-16T20:04:38.983 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jMwrLeBk4mTqHXGSy1/C4pCiHVU= 2024-04-16T20:04:38.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMwrLeBk4mTqHXGSy1/C4pCiHVU=', 'Content-Length': '0'} 2024-04-16T20:04:38.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c1cd2117fd564b2-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:04:38.985 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.985 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.985 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.985 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.986 DEBUG:boto:Method: GET 2024-04-16T20:04:38.986 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.986 DEBUG:boto:Data: 2024-04-16T20:04:38.986 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.986 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.986 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.986 DEBUG:boto:Params: {} 2024-04-16T20:04:38.986 DEBUG:boto:Token: None 2024-04-16T20:04:38.986 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA= 2024-04-16T20:04:38.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA=', 'Content-Length': '0'} 2024-04-16T20:04:38.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc3ed72f124b571b-00661ed9d6-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.989 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:38.989 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:38.989 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.989 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.989 DEBUG:boto:Method: GET 2024-04-16T20:04:38.989 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.989 DEBUG:boto:Data: 2024-04-16T20:04:38.989 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.989 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.989 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.989 DEBUG:boto:Params: {} 2024-04-16T20:04:38.989 DEBUG:boto:Token: None 2024-04-16T20:04:38.990 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:04:38.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA= 2024-04-16T20:04:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rC8eN96PcJ3Y8kNtfY6HXxdAEoA=', 'Content-Length': '0'} 2024-04-16T20:04:38.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882c78d29eb0fb91-00661ed9d6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.992 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.992 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.992 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.992 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.992 DEBUG:boto:Method: GET 2024-04-16T20:04:38.992 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.992 DEBUG:boto:Data: 2024-04-16T20:04:38.993 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.993 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:38.993 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.993 DEBUG:boto:Params: {} 2024-04-16T20:04:38.993 DEBUG:boto:Token: None 2024-04-16T20:04:38.993 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.993 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:38.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e7d0ea34ec2478-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:38.995 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:38.996 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:38.996 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:38.996 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.996 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.996 DEBUG:boto:Method: GET 2024-04-16T20:04:38.996 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:04:38.996 DEBUG:boto:Data: 2024-04-16T20:04:38.996 DEBUG:boto:Headers: {} 2024-04-16T20:04:38.996 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:38.996 DEBUG:boto:Port: 8000 2024-04-16T20:04:38.997 DEBUG:boto:Params: {} 2024-04-16T20:04:38.997 DEBUG:boto:Token: None 2024-04-16T20:04:38.997 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:38 GMT /yrjefh-52/ 2024-04-16T20:04:38.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4= 2024-04-16T20:04:38.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORiGqG+0Oa7FzPUSEGNQsVBE5a4=', 'Content-Length': '0'} 2024-04-16T20:04:39.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ed4a1dcd889617-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.000 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.001 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:39.001 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:39.001 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.001 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:04:39.001 DEBUG:boto:Data: 2024-04-16T20:04:39.001 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.001 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.001 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.001 DEBUG:boto:Params: {} 2024-04-16T20:04:39.001 DEBUG:boto:Token: None 2024-04-16T20:04:39.002 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj116 2024-04-16T20:04:39.002 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIhbvJG6dqcbsbs0oZlXE2cbylo= 2024-04-16T20:04:39.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIhbvJG6dqcbsbs0oZlXE2cbylo=', 'Content-Length': '0'} 2024-04-16T20:04:39.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7dd0869e16ceae-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.004 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:39.004 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:39.004 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.004 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:04:39.004 DEBUG:boto:Data: 2024-04-16T20:04:39.004 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.004 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.005 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.005 DEBUG:boto:Params: {} 2024-04-16T20:04:39.005 DEBUG:boto:Token: None 2024-04-16T20:04:39.005 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj116 2024-04-16T20:04:39.005 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AIhbvJG6dqcbsbs0oZlXE2cbylo= 2024-04-16T20:04:39.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIhbvJG6dqcbsbs0oZlXE2cbylo=', 'Content-Length': '0'} 2024-04-16T20:04:39.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acac215d7ed73bd5-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:04:39.007 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:39.007 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:39.007 DEBUG:boto:Method: GET 2024-04-16T20:04:39.007 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:04:39.007 DEBUG:boto:Data: 2024-04-16T20:04:39.007 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.007 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.008 DEBUG:boto:Params: {} 2024-04-16T20:04:39.008 DEBUG:boto:Token: None 2024-04-16T20:04:39.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj116 2024-04-16T20:04:39.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0xYBn3+tKRCxhQV6fEOpA0L8z7k= 2024-04-16T20:04:39.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0xYBn3+tKRCxhQV6fEOpA0L8z7k=', 'Content-Length': '0'} 2024-04-16T20:04:39.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000340fb08a93e0a03b-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.050 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:04:39.050 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:04:39.050 DEBUG:boto:Method: GET 2024-04-16T20:04:39.051 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:04:39.051 DEBUG:boto:Data: 2024-04-16T20:04:39.051 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.051 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.051 DEBUG:boto:Params: {} 2024-04-16T20:04:39.051 DEBUG:boto:Token: None 2024-04-16T20:04:39.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj116 2024-04-16T20:04:39.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0xYBn3+tKRCxhQV6fEOpA0L8z7k= 2024-04-16T20:04:39.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0xYBn3+tKRCxhQV6fEOpA0L8z7k=', 'Content-Length': '0'} 2024-04-16T20:04:39.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee11638bb6faef27-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.053 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.053 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.053 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.054 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.054 DEBUG:boto:Method: GET 2024-04-16T20:04:39.054 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.054 DEBUG:boto:Data: 2024-04-16T20:04:39.054 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.054 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.054 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.054 DEBUG:boto:Params: {} 2024-04-16T20:04:39.054 DEBUG:boto:Token: None 2024-04-16T20:04:39.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86682b602885911-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.057 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.058 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.058 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.058 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.058 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.058 DEBUG:boto:Method: GET 2024-04-16T20:04:39.058 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:04:39.058 DEBUG:boto:Data: 2024-04-16T20:04:39.058 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.059 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.059 DEBUG:boto:Params: {} 2024-04-16T20:04:39.059 DEBUG:boto:Token: None 2024-04-16T20:04:39.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000102b6edce649751e-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.062 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:04:39.063 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.063 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.063 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.063 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.063 DEBUG:boto:Method: GET 2024-04-16T20:04:39.063 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.063 DEBUG:boto:Data: 2024-04-16T20:04:39.063 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.064 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.064 DEBUG:boto:Params: {} 2024-04-16T20:04:39.064 DEBUG:boto:Token: None 2024-04-16T20:04:39.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U= 2024-04-16T20:04:39.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U=', 'Content-Length': '0'} 2024-04-16T20:04:39.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598981e8bf3b2aca-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.067 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.067 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.067 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.067 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.067 DEBUG:boto:Method: GET 2024-04-16T20:04:39.067 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.067 DEBUG:boto:Data: 2024-04-16T20:04:39.067 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.068 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.068 DEBUG:boto:Params: {} 2024-04-16T20:04:39.068 DEBUG:boto:Token: None 2024-04-16T20:04:39.068 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U= 2024-04-16T20:04:39.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U=', 'Content-Length': '0'} 2024-04-16T20:04:39.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1626927e26ecfd-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.071 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.071 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.071 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.071 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.071 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.071 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.071 DEBUG:boto:Data: 2024-04-16T20:04:39.071 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.071 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.071 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.071 DEBUG:boto:Params: {} 2024-04-16T20:04:39.072 DEBUG:boto:Token: None 2024-04-16T20:04:39.072 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zCRHrsXteTWCLuIbiSjyVM4nwQ0= 2024-04-16T20:04:39.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zCRHrsXteTWCLuIbiSjyVM4nwQ0=', 'Content-Length': '0'} 2024-04-16T20:04:39.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7f9e5bb6f0adad-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.074 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.074 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.074 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.074 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.074 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.075 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.075 DEBUG:boto:Data: 2024-04-16T20:04:39.075 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.075 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.075 DEBUG:boto:Params: {} 2024-04-16T20:04:39.075 DEBUG:boto:Token: None 2024-04-16T20:04:39.075 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zCRHrsXteTWCLuIbiSjyVM4nwQ0= 2024-04-16T20:04:39.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zCRHrsXteTWCLuIbiSjyVM4nwQ0=', 'Content-Length': '0'} 2024-04-16T20:04:39.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007054347b8f3419a9-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:04:39.078 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.078 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.078 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.078 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.078 DEBUG:boto:Method: GET 2024-04-16T20:04:39.078 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.078 DEBUG:boto:Data: 2024-04-16T20:04:39.078 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.079 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.079 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.079 DEBUG:boto:Params: {} 2024-04-16T20:04:39.079 DEBUG:boto:Token: None 2024-04-16T20:04:39.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U= 2024-04-16T20:04:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U=', 'Content-Length': '0'} 2024-04-16T20:04:39.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb252dc8bf63c1a-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.082 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.082 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.082 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.082 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.082 DEBUG:boto:Method: GET 2024-04-16T20:04:39.082 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.082 DEBUG:boto:Data: 2024-04-16T20:04:39.082 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.082 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.083 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.083 DEBUG:boto:Params: {} 2024-04-16T20:04:39.083 DEBUG:boto:Token: None 2024-04-16T20:04:39.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:04:39.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U= 2024-04-16T20:04:39.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Gb64GixGYMZzfCkdjPAGco1E6U=', 'Content-Length': '0'} 2024-04-16T20:04:39.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33d11c50fc2bd94-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.086 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.086 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.086 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.086 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.086 DEBUG:boto:Method: GET 2024-04-16T20:04:39.086 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.086 DEBUG:boto:Data: 2024-04-16T20:04:39.086 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.086 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.086 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.086 DEBUG:boto:Params: {} 2024-04-16T20:04:39.087 DEBUG:boto:Token: None 2024-04-16T20:04:39.087 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b64993dfd7f326eb-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.090 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.090 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.090 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.090 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.090 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.090 DEBUG:boto:Method: GET 2024-04-16T20:04:39.090 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.090 DEBUG:boto:Data: 2024-04-16T20:04:39.091 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.091 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.091 DEBUG:boto:Params: {} 2024-04-16T20:04:39.091 DEBUG:boto:Token: None 2024-04-16T20:04:39.091 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5a03898dd7c24b-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.094 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.095 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.095 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.095 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.095 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:04:39.095 DEBUG:boto:Data: 2024-04-16T20:04:39.095 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.095 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.095 DEBUG:boto:Params: {} 2024-04-16T20:04:39.095 DEBUG:boto:Token: None 2024-04-16T20:04:39.096 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145 2024-04-16T20:04:39.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gVkYytm/BRObVXhZos1BDxYAJeI= 2024-04-16T20:04:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gVkYytm/BRObVXhZos1BDxYAJeI=', 'Content-Length': '0'} 2024-04-16T20:04:39.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe00493391bf134-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.098 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.098 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.098 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.098 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:04:39.099 DEBUG:boto:Data: 2024-04-16T20:04:39.099 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.099 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.099 DEBUG:boto:Params: {} 2024-04-16T20:04:39.099 DEBUG:boto:Token: None 2024-04-16T20:04:39.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145 2024-04-16T20:04:39.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gVkYytm/BRObVXhZos1BDxYAJeI= 2024-04-16T20:04:39.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gVkYytm/BRObVXhZos1BDxYAJeI=', 'Content-Length': '0'} 2024-04-16T20:04:39.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080973b138ef45d94-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:04:39.101 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.102 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.102 DEBUG:boto:Method: GET 2024-04-16T20:04:39.102 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:04:39.102 DEBUG:boto:Data: 2024-04-16T20:04:39.102 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.102 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.102 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.102 DEBUG:boto:Params: {} 2024-04-16T20:04:39.102 DEBUG:boto:Token: None 2024-04-16T20:04:39.102 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145 2024-04-16T20:04:39.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gb+sff4z3r9TmfyWQz5oiTbO5jM= 2024-04-16T20:04:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gb+sff4z3r9TmfyWQz5oiTbO5jM=', 'Content-Length': '0'} 2024-04-16T20:04:39.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd2a5a2c9a9adba-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.105 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:04:39.105 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:04:39.105 DEBUG:boto:Method: GET 2024-04-16T20:04:39.105 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:04:39.105 DEBUG:boto:Data: 2024-04-16T20:04:39.105 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.105 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.105 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.105 DEBUG:boto:Params: {} 2024-04-16T20:04:39.106 DEBUG:boto:Token: None 2024-04-16T20:04:39.106 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj145 2024-04-16T20:04:39.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gb+sff4z3r9TmfyWQz5oiTbO5jM= 2024-04-16T20:04:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gb+sff4z3r9TmfyWQz5oiTbO5jM=', 'Content-Length': '0'} 2024-04-16T20:04:39.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e12d882f672a1b-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.150 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.150 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.150 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.151 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.151 DEBUG:boto:Method: GET 2024-04-16T20:04:39.151 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.151 DEBUG:boto:Data: 2024-04-16T20:04:39.151 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.151 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.151 DEBUG:boto:Params: {} 2024-04-16T20:04:39.151 DEBUG:boto:Token: None 2024-04-16T20:04:39.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073092c6c72641b1a-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.154 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.154 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.155 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.155 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.155 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.155 DEBUG:boto:Method: GET 2024-04-16T20:04:39.155 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:04:39.155 DEBUG:boto:Data: 2024-04-16T20:04:39.155 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.155 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.155 DEBUG:boto:Params: {} 2024-04-16T20:04:39.155 DEBUG:boto:Token: None 2024-04-16T20:04:39.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2afbdf395245c0-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.159 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:04:39.160 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.160 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.160 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.160 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.160 DEBUG:boto:Method: GET 2024-04-16T20:04:39.160 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.160 DEBUG:boto:Data: 2024-04-16T20:04:39.160 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.160 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.160 DEBUG:boto:Params: {} 2024-04-16T20:04:39.161 DEBUG:boto:Token: None 2024-04-16T20:04:39.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE= 2024-04-16T20:04:39.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE=', 'Content-Length': '0'} 2024-04-16T20:04:39.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7fd90f1d0acfa3-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.163 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.163 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.163 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.163 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.163 DEBUG:boto:Method: GET 2024-04-16T20:04:39.164 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.164 DEBUG:boto:Data: 2024-04-16T20:04:39.164 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.164 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.164 DEBUG:boto:Params: {} 2024-04-16T20:04:39.164 DEBUG:boto:Token: None 2024-04-16T20:04:39.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE= 2024-04-16T20:04:39.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE=', 'Content-Length': '0'} 2024-04-16T20:04:39.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb61a65b0274ba4-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.166 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.166 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.167 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.167 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.167 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.167 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.167 DEBUG:boto:Data: 2024-04-16T20:04:39.167 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.167 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.167 DEBUG:boto:Params: {} 2024-04-16T20:04:39.167 DEBUG:boto:Token: None 2024-04-16T20:04:39.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sUXJQozRZ/kgcf24kPL1qPsaN3Q= 2024-04-16T20:04:39.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sUXJQozRZ/kgcf24kPL1qPsaN3Q=', 'Content-Length': '0'} 2024-04-16T20:04:39.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b7d7f784aa69c5-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.169 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.169 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.170 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.170 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.170 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.170 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.170 DEBUG:boto:Data: 2024-04-16T20:04:39.170 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.170 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.170 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.170 DEBUG:boto:Params: {} 2024-04-16T20:04:39.170 DEBUG:boto:Token: None 2024-04-16T20:04:39.170 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sUXJQozRZ/kgcf24kPL1qPsaN3Q= 2024-04-16T20:04:39.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sUXJQozRZ/kgcf24kPL1qPsaN3Q=', 'Content-Length': '0'} 2024-04-16T20:04:39.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a7d08120370279-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:04:39.173 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.173 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.173 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.173 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.173 DEBUG:boto:Method: GET 2024-04-16T20:04:39.173 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.173 DEBUG:boto:Data: 2024-04-16T20:04:39.173 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.173 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.173 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.173 DEBUG:boto:Params: {} 2024-04-16T20:04:39.173 DEBUG:boto:Token: None 2024-04-16T20:04:39.174 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE= 2024-04-16T20:04:39.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE=', 'Content-Length': '0'} 2024-04-16T20:04:39.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b9896900f52ea2-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.218 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.218 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.218 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.218 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.219 DEBUG:boto:Method: GET 2024-04-16T20:04:39.219 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.219 DEBUG:boto:Data: 2024-04-16T20:04:39.219 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.219 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.219 DEBUG:boto:Params: {} 2024-04-16T20:04:39.219 DEBUG:boto:Token: None 2024-04-16T20:04:39.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:04:39.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE= 2024-04-16T20:04:39.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80Mv6a/ROqWf/0XszAlm1HLoBvE=', 'Content-Length': '0'} 2024-04-16T20:04:39.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f5ab699bbab6164-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.221 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.222 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.222 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.222 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.222 DEBUG:boto:Method: GET 2024-04-16T20:04:39.222 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.222 DEBUG:boto:Data: 2024-04-16T20:04:39.222 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.222 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.222 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.222 DEBUG:boto:Params: {} 2024-04-16T20:04:39.222 DEBUG:boto:Token: None 2024-04-16T20:04:39.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1ecac1b6a8a518-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.225 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.225 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.225 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.226 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.226 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.226 DEBUG:boto:Method: GET 2024-04-16T20:04:39.226 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.226 DEBUG:boto:Data: 2024-04-16T20:04:39.226 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.226 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.226 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.226 DEBUG:boto:Params: {} 2024-04-16T20:04:39.226 DEBUG:boto:Token: None 2024-04-16T20:04:39.226 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647325e1f5ad489b-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.230 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.231 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.231 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.231 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.231 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:04:39.231 DEBUG:boto:Data: 2024-04-16T20:04:39.231 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.231 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.231 DEBUG:boto:Params: {} 2024-04-16T20:04:39.231 DEBUG:boto:Token: None 2024-04-16T20:04:39.232 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17 2024-04-16T20:04:39.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8gDKuXU481D++4LNnBYp1LNmfNU= 2024-04-16T20:04:39.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8gDKuXU481D++4LNnBYp1LNmfNU=', 'Content-Length': '0'} 2024-04-16T20:04:39.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd04644fe29ecb6-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.234 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.234 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.234 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.234 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:04:39.234 DEBUG:boto:Data: 2024-04-16T20:04:39.234 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.235 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.235 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.235 DEBUG:boto:Params: {} 2024-04-16T20:04:39.235 DEBUG:boto:Token: None 2024-04-16T20:04:39.235 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17 2024-04-16T20:04:39.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8gDKuXU481D++4LNnBYp1LNmfNU= 2024-04-16T20:04:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8gDKuXU481D++4LNnBYp1LNmfNU=', 'Content-Length': '0'} 2024-04-16T20:04:39.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d236ae00bef2c87-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:04:39.237 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.237 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.237 DEBUG:boto:Method: GET 2024-04-16T20:04:39.238 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:04:39.238 DEBUG:boto:Data: 2024-04-16T20:04:39.238 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.238 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.238 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.238 DEBUG:boto:Params: {} 2024-04-16T20:04:39.238 DEBUG:boto:Token: None 2024-04-16T20:04:39.238 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17 2024-04-16T20:04:39.238 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TE0eiPkydO1y7fTKwK5wjvWGYJg= 2024-04-16T20:04:39.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TE0eiPkydO1y7fTKwK5wjvWGYJg=', 'Content-Length': '0'} 2024-04-16T20:04:39.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fadc449b3ae148a-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.240 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:04:39.240 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:04:39.241 DEBUG:boto:Method: GET 2024-04-16T20:04:39.241 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:04:39.241 DEBUG:boto:Data: 2024-04-16T20:04:39.241 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.241 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.241 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.241 DEBUG:boto:Params: {} 2024-04-16T20:04:39.241 DEBUG:boto:Token: None 2024-04-16T20:04:39.241 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj17 2024-04-16T20:04:39.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TE0eiPkydO1y7fTKwK5wjvWGYJg= 2024-04-16T20:04:39.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TE0eiPkydO1y7fTKwK5wjvWGYJg=', 'Content-Length': '0'} 2024-04-16T20:04:39.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbaffdf3cff726e-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.243 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.243 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.244 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.244 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.244 DEBUG:boto:Method: GET 2024-04-16T20:04:39.244 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.244 DEBUG:boto:Data: 2024-04-16T20:04:39.244 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.244 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.244 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.244 DEBUG:boto:Params: {} 2024-04-16T20:04:39.244 DEBUG:boto:Token: None 2024-04-16T20:04:39.245 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e05374c38ce2fff-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.247 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.248 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.248 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.248 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.248 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.248 DEBUG:boto:Method: GET 2024-04-16T20:04:39.248 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:04:39.248 DEBUG:boto:Data: 2024-04-16T20:04:39.248 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.248 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.248 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.249 DEBUG:boto:Params: {} 2024-04-16T20:04:39.249 DEBUG:boto:Token: None 2024-04-16T20:04:39.249 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.249 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075113f3a2a58b516-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.252 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:04:39.253 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.253 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.253 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.253 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.253 DEBUG:boto:Method: GET 2024-04-16T20:04:39.253 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.253 DEBUG:boto:Data: 2024-04-16T20:04:39.253 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.254 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.254 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.254 DEBUG:boto:Params: {} 2024-04-16T20:04:39.254 DEBUG:boto:Token: None 2024-04-16T20:04:39.254 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.254 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw= 2024-04-16T20:04:39.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw=', 'Content-Length': '0'} 2024-04-16T20:04:39.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005338fb52b7157af4-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.298 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.298 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.299 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.299 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.299 DEBUG:boto:Method: GET 2024-04-16T20:04:39.299 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.299 DEBUG:boto:Data: 2024-04-16T20:04:39.299 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.299 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.299 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.299 DEBUG:boto:Params: {} 2024-04-16T20:04:39.299 DEBUG:boto:Token: None 2024-04-16T20:04:39.299 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw= 2024-04-16T20:04:39.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw=', 'Content-Length': '0'} 2024-04-16T20:04:39.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3677b71302f516-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.302 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.302 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.302 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.302 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.302 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.302 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.302 DEBUG:boto:Data: 2024-04-16T20:04:39.303 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.303 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.303 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.303 DEBUG:boto:Params: {} 2024-04-16T20:04:39.303 DEBUG:boto:Token: None 2024-04-16T20:04:39.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3f9jUoqNhkj4SVedsgw1uEgN1E= 2024-04-16T20:04:39.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3f9jUoqNhkj4SVedsgw1uEgN1E=', 'Content-Length': '0'} 2024-04-16T20:04:39.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475f87ff04361c84-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.306 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.306 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.306 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.306 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.306 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.306 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.306 DEBUG:boto:Data: 2024-04-16T20:04:39.307 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.307 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.307 DEBUG:boto:Params: {} 2024-04-16T20:04:39.307 DEBUG:boto:Token: None 2024-04-16T20:04:39.307 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r3f9jUoqNhkj4SVedsgw1uEgN1E= 2024-04-16T20:04:39.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3f9jUoqNhkj4SVedsgw1uEgN1E=', 'Content-Length': '0'} 2024-04-16T20:04:39.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c158a70a412361f-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:04:39.309 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.309 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.310 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.310 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.310 DEBUG:boto:Method: GET 2024-04-16T20:04:39.310 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.310 DEBUG:boto:Data: 2024-04-16T20:04:39.310 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.310 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.310 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.310 DEBUG:boto:Params: {} 2024-04-16T20:04:39.310 DEBUG:boto:Token: None 2024-04-16T20:04:39.310 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw= 2024-04-16T20:04:39.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw=', 'Content-Length': '0'} 2024-04-16T20:04:39.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040da31d96695bcd7-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.354 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.354 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.354 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.355 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.355 DEBUG:boto:Method: GET 2024-04-16T20:04:39.355 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.355 DEBUG:boto:Data: 2024-04-16T20:04:39.355 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.355 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.355 DEBUG:boto:Params: {} 2024-04-16T20:04:39.355 DEBUG:boto:Token: None 2024-04-16T20:04:39.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:04:39.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw= 2024-04-16T20:04:39.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uPIRkXMbbsFaq8JESv1CRYz7avw=', 'Content-Length': '0'} 2024-04-16T20:04:39.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7022f3084cb4505-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.358 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.358 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.358 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.358 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.358 DEBUG:boto:Method: GET 2024-04-16T20:04:39.358 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.358 DEBUG:boto:Data: 2024-04-16T20:04:39.358 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.358 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.358 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.358 DEBUG:boto:Params: {} 2024-04-16T20:04:39.359 DEBUG:boto:Token: None 2024-04-16T20:04:39.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9870654b9481f10-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.361 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.362 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.362 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.362 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.362 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.362 DEBUG:boto:Method: GET 2024-04-16T20:04:39.362 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.362 DEBUG:boto:Data: 2024-04-16T20:04:39.362 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.362 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.362 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.362 DEBUG:boto:Params: {} 2024-04-16T20:04:39.363 DEBUG:boto:Token: None 2024-04-16T20:04:39.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f5252e5ce385c52-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.366 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.366 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.367 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.367 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.367 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:04:39.367 DEBUG:boto:Data: 2024-04-16T20:04:39.367 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.367 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.367 DEBUG:boto:Params: {} 2024-04-16T20:04:39.367 DEBUG:boto:Token: None 2024-04-16T20:04:39.367 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19 2024-04-16T20:04:39.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RfBIZDrAEMC+qPXVTpzB4LVruzA= 2024-04-16T20:04:39.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RfBIZDrAEMC+qPXVTpzB4LVruzA=', 'Content-Length': '0'} 2024-04-16T20:04:39.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32b3a74c96b37bb-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.370 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.370 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.370 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.370 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:04:39.371 DEBUG:boto:Data: 2024-04-16T20:04:39.371 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.371 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.371 DEBUG:boto:Params: {} 2024-04-16T20:04:39.371 DEBUG:boto:Token: None 2024-04-16T20:04:39.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19 2024-04-16T20:04:39.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RfBIZDrAEMC+qPXVTpzB4LVruzA= 2024-04-16T20:04:39.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RfBIZDrAEMC+qPXVTpzB4LVruzA=', 'Content-Length': '0'} 2024-04-16T20:04:39.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac7d206fe3ca824-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:04:39.373 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.373 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.374 DEBUG:boto:Method: GET 2024-04-16T20:04:39.374 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:04:39.374 DEBUG:boto:Data: 2024-04-16T20:04:39.374 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.374 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.374 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.374 DEBUG:boto:Params: {} 2024-04-16T20:04:39.374 DEBUG:boto:Token: None 2024-04-16T20:04:39.374 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19 2024-04-16T20:04:39.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hrM5JZNn5rR33yRbQ9Sr+vnfqt0= 2024-04-16T20:04:39.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hrM5JZNn5rR33yRbQ9Sr+vnfqt0=', 'Content-Length': '0'} 2024-04-16T20:04:39.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5455709a198707e-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.418 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:04:39.418 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:04:39.419 DEBUG:boto:Method: GET 2024-04-16T20:04:39.419 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:04:39.419 DEBUG:boto:Data: 2024-04-16T20:04:39.419 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.419 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.419 DEBUG:boto:Params: {} 2024-04-16T20:04:39.419 DEBUG:boto:Token: None 2024-04-16T20:04:39.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj19 2024-04-16T20:04:39.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hrM5JZNn5rR33yRbQ9Sr+vnfqt0= 2024-04-16T20:04:39.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hrM5JZNn5rR33yRbQ9Sr+vnfqt0=', 'Content-Length': '0'} 2024-04-16T20:04:39.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f19a89836b0dd478-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.422 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.422 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.422 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.422 DEBUG:boto:Method: GET 2024-04-16T20:04:39.422 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.422 DEBUG:boto:Data: 2024-04-16T20:04:39.422 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.422 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.423 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.423 DEBUG:boto:Params: {} 2024-04-16T20:04:39.423 DEBUG:boto:Token: None 2024-04-16T20:04:39.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e0b7dcc30b37a16-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.426 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.426 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.426 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.426 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.426 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.426 DEBUG:boto:Method: GET 2024-04-16T20:04:39.427 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:04:39.427 DEBUG:boto:Data: 2024-04-16T20:04:39.427 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.427 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.427 DEBUG:boto:Params: {} 2024-04-16T20:04:39.427 DEBUG:boto:Token: None 2024-04-16T20:04:39.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c295a05ad073166-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.431 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:04:39.431 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.432 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.432 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.432 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.432 DEBUG:boto:Method: GET 2024-04-16T20:04:39.432 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.432 DEBUG:boto:Data: 2024-04-16T20:04:39.432 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.432 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.432 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.432 DEBUG:boto:Params: {} 2024-04-16T20:04:39.432 DEBUG:boto:Token: None 2024-04-16T20:04:39.433 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.433 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg= 2024-04-16T20:04:39.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg=', 'Content-Length': '0'} 2024-04-16T20:04:39.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df64139748f7a5ec-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.478 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.478 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.478 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.479 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.479 DEBUG:boto:Method: GET 2024-04-16T20:04:39.479 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.479 DEBUG:boto:Data: 2024-04-16T20:04:39.479 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.479 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.479 DEBUG:boto:Params: {} 2024-04-16T20:04:39.479 DEBUG:boto:Token: None 2024-04-16T20:04:39.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg= 2024-04-16T20:04:39.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg=', 'Content-Length': '0'} 2024-04-16T20:04:39.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000292ef50e6ef9ef3b-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.482 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.482 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.482 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.482 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.482 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.482 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.482 DEBUG:boto:Data: 2024-04-16T20:04:39.482 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.482 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.482 DEBUG:boto:Params: {} 2024-04-16T20:04:39.483 DEBUG:boto:Token: None 2024-04-16T20:04:39.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LakyAZyKwiG04GAEPeM4PY5BVFs= 2024-04-16T20:04:39.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LakyAZyKwiG04GAEPeM4PY5BVFs=', 'Content-Length': '0'} 2024-04-16T20:04:39.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e62c422b30f13f-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.485 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.485 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.486 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.486 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.486 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.486 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.486 DEBUG:boto:Data: 2024-04-16T20:04:39.486 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.486 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.486 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.486 DEBUG:boto:Params: {} 2024-04-16T20:04:39.486 DEBUG:boto:Token: None 2024-04-16T20:04:39.487 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LakyAZyKwiG04GAEPeM4PY5BVFs= 2024-04-16T20:04:39.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LakyAZyKwiG04GAEPeM4PY5BVFs=', 'Content-Length': '0'} 2024-04-16T20:04:39.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff86f9b7ef09dbb-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:04:39.489 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.489 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.489 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.489 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.490 DEBUG:boto:Method: GET 2024-04-16T20:04:39.490 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.490 DEBUG:boto:Data: 2024-04-16T20:04:39.490 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.490 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.490 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.490 DEBUG:boto:Params: {} 2024-04-16T20:04:39.490 DEBUG:boto:Token: None 2024-04-16T20:04:39.490 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.490 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg= 2024-04-16T20:04:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg=', 'Content-Length': '0'} 2024-04-16T20:04:39.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acace00a60a0e0d8-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.534 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.534 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.535 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.535 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.535 DEBUG:boto:Method: GET 2024-04-16T20:04:39.535 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.535 DEBUG:boto:Data: 2024-04-16T20:04:39.535 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.535 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.535 DEBUG:boto:Params: {} 2024-04-16T20:04:39.535 DEBUG:boto:Token: None 2024-04-16T20:04:39.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:04:39.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg= 2024-04-16T20:04:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:82Vz4gsFewZTEKw6RZcYWwSqUZg=', 'Content-Length': '0'} 2024-04-16T20:04:39.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c4c83aa95a95fe-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.538 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.539 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.539 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.539 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.539 DEBUG:boto:Method: GET 2024-04-16T20:04:39.539 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.539 DEBUG:boto:Data: 2024-04-16T20:04:39.539 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.539 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.539 DEBUG:boto:Params: {} 2024-04-16T20:04:39.539 DEBUG:boto:Token: None 2024-04-16T20:04:39.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1f1934a0e68783-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.542 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.542 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.542 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.542 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.543 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.543 DEBUG:boto:Method: GET 2024-04-16T20:04:39.543 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.543 DEBUG:boto:Data: 2024-04-16T20:04:39.543 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.543 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.543 DEBUG:boto:Params: {} 2024-04-16T20:04:39.543 DEBUG:boto:Token: None 2024-04-16T20:04:39.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002837ac7c9d5c3e09-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.547 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.547 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.548 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.548 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.548 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:04:39.548 DEBUG:boto:Data: 2024-04-16T20:04:39.548 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.548 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.548 DEBUG:boto:Params: {} 2024-04-16T20:04:39.548 DEBUG:boto:Token: None 2024-04-16T20:04:39.548 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22 2024-04-16T20:04:39.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rawM2vxBTWh+klTnhRvvuyLdmZw= 2024-04-16T20:04:39.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rawM2vxBTWh+klTnhRvvuyLdmZw=', 'Content-Length': '0'} 2024-04-16T20:04:39.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342c3990b63e8fba-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.551 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.551 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.551 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.551 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:04:39.551 DEBUG:boto:Data: 2024-04-16T20:04:39.551 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.551 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.551 DEBUG:boto:Params: {} 2024-04-16T20:04:39.552 DEBUG:boto:Token: None 2024-04-16T20:04:39.552 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22 2024-04-16T20:04:39.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rawM2vxBTWh+klTnhRvvuyLdmZw= 2024-04-16T20:04:39.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rawM2vxBTWh+klTnhRvvuyLdmZw=', 'Content-Length': '0'} 2024-04-16T20:04:39.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58d88ca27ec53ed-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:04:39.554 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.554 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.554 DEBUG:boto:Method: GET 2024-04-16T20:04:39.555 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:04:39.555 DEBUG:boto:Data: 2024-04-16T20:04:39.555 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.555 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.555 DEBUG:boto:Params: {} 2024-04-16T20:04:39.555 DEBUG:boto:Token: None 2024-04-16T20:04:39.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22 2024-04-16T20:04:39.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sG71qvzm8xteoc14bWxBwd2siX4= 2024-04-16T20:04:39.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sG71qvzm8xteoc14bWxBwd2siX4=', 'Content-Length': '0'} 2024-04-16T20:04:39.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46557c621390b2c-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.598 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:04:39.598 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:04:39.598 DEBUG:boto:Method: GET 2024-04-16T20:04:39.599 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:04:39.599 DEBUG:boto:Data: 2024-04-16T20:04:39.599 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.599 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.599 DEBUG:boto:Params: {} 2024-04-16T20:04:39.599 DEBUG:boto:Token: None 2024-04-16T20:04:39.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj22 2024-04-16T20:04:39.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sG71qvzm8xteoc14bWxBwd2siX4= 2024-04-16T20:04:39.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sG71qvzm8xteoc14bWxBwd2siX4=', 'Content-Length': '0'} 2024-04-16T20:04:39.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9a4e174890a94c9-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.602 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.602 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.602 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.602 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.602 DEBUG:boto:Method: GET 2024-04-16T20:04:39.602 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.602 DEBUG:boto:Data: 2024-04-16T20:04:39.602 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.602 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.603 DEBUG:boto:Params: {} 2024-04-16T20:04:39.603 DEBUG:boto:Token: None 2024-04-16T20:04:39.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001019bc8452e89df7-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.646 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.647 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.647 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.647 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.647 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.647 DEBUG:boto:Method: GET 2024-04-16T20:04:39.647 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:04:39.647 DEBUG:boto:Data: 2024-04-16T20:04:39.647 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.647 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.647 DEBUG:boto:Params: {} 2024-04-16T20:04:39.648 DEBUG:boto:Token: None 2024-04-16T20:04:39.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007173595bd213a535-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.651 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:04:39.651 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.651 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.651 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.651 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.652 DEBUG:boto:Method: GET 2024-04-16T20:04:39.652 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.652 DEBUG:boto:Data: 2024-04-16T20:04:39.652 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.652 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.652 DEBUG:boto:Params: {} 2024-04-16T20:04:39.652 DEBUG:boto:Token: None 2024-04-16T20:04:39.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE= 2024-04-16T20:04:39.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE=', 'Content-Length': '0'} 2024-04-16T20:04:39.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d2ab1910769dae-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.698 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.698 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.698 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.698 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.699 DEBUG:boto:Method: GET 2024-04-16T20:04:39.699 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.699 DEBUG:boto:Data: 2024-04-16T20:04:39.699 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.699 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.699 DEBUG:boto:Params: {} 2024-04-16T20:04:39.699 DEBUG:boto:Token: None 2024-04-16T20:04:39.699 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.699 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE= 2024-04-16T20:04:39.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE=', 'Content-Length': '0'} 2024-04-16T20:04:39.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b130146ad26215b-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.702 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.702 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.702 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.702 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.702 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.702 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.702 DEBUG:boto:Data: 2024-04-16T20:04:39.702 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.702 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.702 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.703 DEBUG:boto:Params: {} 2024-04-16T20:04:39.703 DEBUG:boto:Token: None 2024-04-16T20:04:39.703 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrU6zxB+LDiNTX9y74TRaU3Wpl0= 2024-04-16T20:04:39.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrU6zxB+LDiNTX9y74TRaU3Wpl0=', 'Content-Length': '0'} 2024-04-16T20:04:39.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ba30a7d0ba7b05-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.705 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.705 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.705 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.705 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.705 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.705 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.706 DEBUG:boto:Data: 2024-04-16T20:04:39.706 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.706 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.706 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.706 DEBUG:boto:Params: {} 2024-04-16T20:04:39.706 DEBUG:boto:Token: None 2024-04-16T20:04:39.706 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.706 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GrU6zxB+LDiNTX9y74TRaU3Wpl0= 2024-04-16T20:04:39.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GrU6zxB+LDiNTX9y74TRaU3Wpl0=', 'Content-Length': '0'} 2024-04-16T20:04:39.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1a24d65e69d1a6-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:04:39.709 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.709 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.709 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.709 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.709 DEBUG:boto:Method: GET 2024-04-16T20:04:39.709 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.709 DEBUG:boto:Data: 2024-04-16T20:04:39.709 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.709 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.709 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.709 DEBUG:boto:Params: {} 2024-04-16T20:04:39.710 DEBUG:boto:Token: None 2024-04-16T20:04:39.710 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.710 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE= 2024-04-16T20:04:39.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE=', 'Content-Length': '0'} 2024-04-16T20:04:39.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000484c922414ebfb29-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.754 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.755 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.755 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.755 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.755 DEBUG:boto:Method: GET 2024-04-16T20:04:39.755 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.756 DEBUG:boto:Data: 2024-04-16T20:04:39.756 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.756 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.756 DEBUG:boto:Params: {} 2024-04-16T20:04:39.756 DEBUG:boto:Token: None 2024-04-16T20:04:39.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:04:39.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE= 2024-04-16T20:04:39.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4O/NH22X5vYkPKM0bUGIHf+vaHE=', 'Content-Length': '0'} 2024-04-16T20:04:39.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b366275321af407-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.759 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.759 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.759 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.759 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.760 DEBUG:boto:Method: GET 2024-04-16T20:04:39.760 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.760 DEBUG:boto:Data: 2024-04-16T20:04:39.760 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.760 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.760 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.760 DEBUG:boto:Params: {} 2024-04-16T20:04:39.760 DEBUG:boto:Token: None 2024-04-16T20:04:39.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c017f533cb5191-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.806 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.807 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.807 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.807 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.807 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.807 DEBUG:boto:Method: GET 2024-04-16T20:04:39.807 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.807 DEBUG:boto:Data: 2024-04-16T20:04:39.808 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.808 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.808 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.808 DEBUG:boto:Params: {} 2024-04-16T20:04:39.808 DEBUG:boto:Token: None 2024-04-16T20:04:39.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090999ee6e8ba83a3-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.813 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.813 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.813 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.813 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.813 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:04:39.813 DEBUG:boto:Data: 2024-04-16T20:04:39.814 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.814 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.814 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.814 DEBUG:boto:Params: {} 2024-04-16T20:04:39.814 DEBUG:boto:Token: None 2024-04-16T20:04:39.814 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23 2024-04-16T20:04:39.814 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7TlIMdXvQl7jFHW724LQxV4Uaxs= 2024-04-16T20:04:39.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7TlIMdXvQl7jFHW724LQxV4Uaxs=', 'Content-Length': '0'} 2024-04-16T20:04:39.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ee2ab306b21bc09-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.817 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.817 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.817 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.817 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:04:39.817 DEBUG:boto:Data: 2024-04-16T20:04:39.818 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.818 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.818 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.818 DEBUG:boto:Params: {} 2024-04-16T20:04:39.818 DEBUG:boto:Token: None 2024-04-16T20:04:39.818 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23 2024-04-16T20:04:39.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7TlIMdXvQl7jFHW724LQxV4Uaxs= 2024-04-16T20:04:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7TlIMdXvQl7jFHW724LQxV4Uaxs=', 'Content-Length': '0'} 2024-04-16T20:04:39.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811a060e22a13a03-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:04:39.821 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.821 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.821 DEBUG:boto:Method: GET 2024-04-16T20:04:39.821 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:04:39.822 DEBUG:boto:Data: 2024-04-16T20:04:39.822 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.822 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.822 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.822 DEBUG:boto:Params: {} 2024-04-16T20:04:39.822 DEBUG:boto:Token: None 2024-04-16T20:04:39.822 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23 2024-04-16T20:04:39.822 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:65wKs+b1YGvizWPvEyDJ5fn3P8s= 2024-04-16T20:04:39.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:65wKs+b1YGvizWPvEyDJ5fn3P8s=', 'Content-Length': '0'} 2024-04-16T20:04:39.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe863f86c554493c-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.866 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:04:39.866 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:04:39.867 DEBUG:boto:Method: GET 2024-04-16T20:04:39.867 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:04:39.867 DEBUG:boto:Data: 2024-04-16T20:04:39.867 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.867 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.867 DEBUG:boto:Params: {} 2024-04-16T20:04:39.867 DEBUG:boto:Token: None 2024-04-16T20:04:39.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj23 2024-04-16T20:04:39.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:65wKs+b1YGvizWPvEyDJ5fn3P8s= 2024-04-16T20:04:39.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:65wKs+b1YGvizWPvEyDJ5fn3P8s=', 'Content-Length': '0'} 2024-04-16T20:04:39.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c257e61d2b75d2-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.870 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.870 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.871 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.871 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.871 DEBUG:boto:Method: GET 2024-04-16T20:04:39.871 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.871 DEBUG:boto:Data: 2024-04-16T20:04:39.871 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.871 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.871 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.871 DEBUG:boto:Params: {} 2024-04-16T20:04:39.871 DEBUG:boto:Token: None 2024-04-16T20:04:39.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aaa041fb541ee40-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.874 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.874 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.875 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.875 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.875 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.875 DEBUG:boto:Method: GET 2024-04-16T20:04:39.875 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:04:39.875 DEBUG:boto:Data: 2024-04-16T20:04:39.875 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.875 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.875 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.875 DEBUG:boto:Params: {} 2024-04-16T20:04:39.876 DEBUG:boto:Token: None 2024-04-16T20:04:39.876 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c95330bbc48899-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.880 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:04:39.880 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.880 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.880 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.880 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.881 DEBUG:boto:Method: GET 2024-04-16T20:04:39.881 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.881 DEBUG:boto:Data: 2024-04-16T20:04:39.881 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.881 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.881 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.881 DEBUG:boto:Params: {} 2024-04-16T20:04:39.881 DEBUG:boto:Token: None 2024-04-16T20:04:39.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.882 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI= 2024-04-16T20:04:39.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI=', 'Content-Length': '0'} 2024-04-16T20:04:39.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc13e6602ca71991-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.926 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.927 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.927 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.927 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.927 DEBUG:boto:Method: GET 2024-04-16T20:04:39.927 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.927 DEBUG:boto:Data: 2024-04-16T20:04:39.927 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.927 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.927 DEBUG:boto:Params: {} 2024-04-16T20:04:39.928 DEBUG:boto:Token: None 2024-04-16T20:04:39.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI= 2024-04-16T20:04:39.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI=', 'Content-Length': '0'} 2024-04-16T20:04:39.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1253c03e28780f-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.930 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.930 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.930 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.931 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.931 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.931 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.931 DEBUG:boto:Data: 2024-04-16T20:04:39.931 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.931 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.931 DEBUG:boto:Params: {} 2024-04-16T20:04:39.931 DEBUG:boto:Token: None 2024-04-16T20:04:39.932 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j9w0V7+Kt7TF07pAch09HfuJVsY= 2024-04-16T20:04:39.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j9w0V7+Kt7TF07pAch09HfuJVsY=', 'Content-Length': '0'} 2024-04-16T20:04:39.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bcb3405647442c-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.934 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.934 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.934 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.934 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.934 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.934 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.934 DEBUG:boto:Data: 2024-04-16T20:04:39.935 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.935 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.935 DEBUG:boto:Params: {} 2024-04-16T20:04:39.935 DEBUG:boto:Token: None 2024-04-16T20:04:39.935 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j9w0V7+Kt7TF07pAch09HfuJVsY= 2024-04-16T20:04:39.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j9w0V7+Kt7TF07pAch09HfuJVsY=', 'Content-Length': '0'} 2024-04-16T20:04:39.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d6377dac539aad-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:04:39.937 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.938 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.938 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.938 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.938 DEBUG:boto:Method: GET 2024-04-16T20:04:39.938 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.938 DEBUG:boto:Data: 2024-04-16T20:04:39.938 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.938 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.938 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.939 DEBUG:boto:Params: {} 2024-04-16T20:04:39.939 DEBUG:boto:Token: None 2024-04-16T20:04:39.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI= 2024-04-16T20:04:39.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI=', 'Content-Length': '0'} 2024-04-16T20:04:39.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f595beba9e5d5c-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.982 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.982 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.983 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.983 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.983 DEBUG:boto:Method: GET 2024-04-16T20:04:39.983 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.983 DEBUG:boto:Data: 2024-04-16T20:04:39.983 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.983 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.983 DEBUG:boto:Params: {} 2024-04-16T20:04:39.983 DEBUG:boto:Token: None 2024-04-16T20:04:39.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:04:39.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI= 2024-04-16T20:04:39.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jEWBfB74/6xR3zYwvr3/MdXnbuI=', 'Content-Length': '0'} 2024-04-16T20:04:39.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd62a57159304a78-00661ed9d7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.986 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.986 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.987 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.987 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.987 DEBUG:boto:Method: GET 2024-04-16T20:04:39.987 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.987 DEBUG:boto:Data: 2024-04-16T20:04:39.987 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.987 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.987 DEBUG:boto:Params: {} 2024-04-16T20:04:39.988 DEBUG:boto:Token: None 2024-04-16T20:04:39.988 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.988 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b7aeae117ddaf2-00661ed9d7-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.990 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.991 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:39.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:39.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.991 DEBUG:boto:Method: GET 2024-04-16T20:04:39.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:04:39.991 DEBUG:boto:Data: 2024-04-16T20:04:39.992 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:39.992 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.992 DEBUG:boto:Params: {} 2024-04-16T20:04:39.992 DEBUG:boto:Token: None 2024-04-16T20:04:39.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/ 2024-04-16T20:04:39.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ= 2024-04-16T20:04:39.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1LGH/CCOT+FqY03Oq1sv7AAlOQ=', 'Content-Length': '0'} 2024-04-16T20:04:39.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539f348da3eaf5c5-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.995 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:39.996 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.996 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:39.996 DEBUG:boto:Method: HEAD 2024-04-16T20:04:39.996 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:04:39.996 DEBUG:boto:Data: 2024-04-16T20:04:39.996 DEBUG:boto:Headers: {} 2024-04-16T20:04:39.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:39.997 DEBUG:boto:Port: 8000 2024-04-16T20:04:39.997 DEBUG:boto:Params: {} 2024-04-16T20:04:39.997 DEBUG:boto:Token: None 2024-04-16T20:04:39.997 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:39 GMT /yrjefh-52/obj29 2024-04-16T20:04:39.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zTwuzqAvUL6jrnBdMDw3m5AND5c= 2024-04-16T20:04:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zTwuzqAvUL6jrnBdMDw3m5AND5c=', 'Content-Length': '0'} 2024-04-16T20:04:39.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213454296031f434-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:39.999 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:39.999 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:40.000 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.000 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:04:40.000 DEBUG:boto:Data: 2024-04-16T20:04:40.000 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.000 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.000 DEBUG:boto:Params: {} 2024-04-16T20:04:40.000 DEBUG:boto:Token: None 2024-04-16T20:04:40.000 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj29 2024-04-16T20:04:40.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aoBWRStK6nnnECTuNWg+E4S3mgg= 2024-04-16T20:04:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aoBWRStK6nnnECTuNWg+E4S3mgg=', 'Content-Length': '0'} 2024-04-16T20:04:40.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1acb1815acff88-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:04:40.003 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:40.003 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:40.003 DEBUG:boto:Method: GET 2024-04-16T20:04:40.003 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:04:40.003 DEBUG:boto:Data: 2024-04-16T20:04:40.003 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.003 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.003 DEBUG:boto:Params: {} 2024-04-16T20:04:40.004 DEBUG:boto:Token: None 2024-04-16T20:04:40.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj29 2024-04-16T20:04:40.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKgKsmWmzlpjydCuwnneWuOjwV4= 2024-04-16T20:04:40.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKgKsmWmzlpjydCuwnneWuOjwV4=', 'Content-Length': '0'} 2024-04-16T20:04:40.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22d333b624ba9c6-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.046 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:04:40.047 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:04:40.047 DEBUG:boto:Method: GET 2024-04-16T20:04:40.047 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:04:40.047 DEBUG:boto:Data: 2024-04-16T20:04:40.047 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.047 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.047 DEBUG:boto:Params: {} 2024-04-16T20:04:40.047 DEBUG:boto:Token: None 2024-04-16T20:04:40.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj29 2024-04-16T20:04:40.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CKgKsmWmzlpjydCuwnneWuOjwV4= 2024-04-16T20:04:40.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CKgKsmWmzlpjydCuwnneWuOjwV4=', 'Content-Length': '0'} 2024-04-16T20:04:40.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007658ac0195a83e91-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.050 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.050 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.050 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.050 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.050 DEBUG:boto:Method: GET 2024-04-16T20:04:40.051 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.051 DEBUG:boto:Data: 2024-04-16T20:04:40.051 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.051 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.051 DEBUG:boto:Params: {} 2024-04-16T20:04:40.051 DEBUG:boto:Token: None 2024-04-16T20:04:40.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceac87600728a63b-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.094 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.095 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.095 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.095 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.095 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.095 DEBUG:boto:Method: GET 2024-04-16T20:04:40.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:04:40.095 DEBUG:boto:Data: 2024-04-16T20:04:40.095 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.095 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.096 DEBUG:boto:Params: {} 2024-04-16T20:04:40.096 DEBUG:boto:Token: None 2024-04-16T20:04:40.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000850ed7550e7ccff-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.100 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:04:40.100 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.100 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.100 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.100 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.100 DEBUG:boto:Method: GET 2024-04-16T20:04:40.100 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.101 DEBUG:boto:Data: 2024-04-16T20:04:40.101 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.101 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.101 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.101 DEBUG:boto:Params: {} 2024-04-16T20:04:40.101 DEBUG:boto:Token: None 2024-04-16T20:04:40.101 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.101 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA= 2024-04-16T20:04:40.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA=', 'Content-Length': '0'} 2024-04-16T20:04:40.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4528fd59f7b3e7d-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.146 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.146 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.146 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.147 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.147 DEBUG:boto:Method: GET 2024-04-16T20:04:40.147 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.147 DEBUG:boto:Data: 2024-04-16T20:04:40.147 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.147 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.147 DEBUG:boto:Params: {} 2024-04-16T20:04:40.147 DEBUG:boto:Token: None 2024-04-16T20:04:40.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA= 2024-04-16T20:04:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA=', 'Content-Length': '0'} 2024-04-16T20:04:40.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002596c53684808f1b-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.150 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.151 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.151 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.151 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.151 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.151 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.151 DEBUG:boto:Data: 2024-04-16T20:04:40.151 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.151 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.151 DEBUG:boto:Params: {} 2024-04-16T20:04:40.152 DEBUG:boto:Token: None 2024-04-16T20:04:40.152 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lAhlcpldYN/yd1Nx1u/iiNwQ948= 2024-04-16T20:04:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lAhlcpldYN/yd1Nx1u/iiNwQ948=', 'Content-Length': '0'} 2024-04-16T20:04:40.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e390ad22d08a210-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.154 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.155 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.155 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.155 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.155 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.155 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.155 DEBUG:boto:Data: 2024-04-16T20:04:40.155 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.155 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.155 DEBUG:boto:Params: {} 2024-04-16T20:04:40.155 DEBUG:boto:Token: None 2024-04-16T20:04:40.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lAhlcpldYN/yd1Nx1u/iiNwQ948= 2024-04-16T20:04:40.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lAhlcpldYN/yd1Nx1u/iiNwQ948=', 'Content-Length': '0'} 2024-04-16T20:04:40.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b730ef097e79be57-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:04:40.158 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.159 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.159 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.159 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.159 DEBUG:boto:Method: GET 2024-04-16T20:04:40.159 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.159 DEBUG:boto:Data: 2024-04-16T20:04:40.159 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.159 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.159 DEBUG:boto:Params: {} 2024-04-16T20:04:40.159 DEBUG:boto:Token: None 2024-04-16T20:04:40.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA= 2024-04-16T20:04:40.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA=', 'Content-Length': '0'} 2024-04-16T20:04:40.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be22bd733cffae1b-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.202 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.202 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.203 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.203 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.203 DEBUG:boto:Method: GET 2024-04-16T20:04:40.203 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.203 DEBUG:boto:Data: 2024-04-16T20:04:40.203 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.203 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.203 DEBUG:boto:Params: {} 2024-04-16T20:04:40.203 DEBUG:boto:Token: None 2024-04-16T20:04:40.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:04:40.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA= 2024-04-16T20:04:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xJIVw6jIphemrt+/Lj7yXIm3cUA=', 'Content-Length': '0'} 2024-04-16T20:04:40.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2945841681a8dc-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.206 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.206 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.206 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.207 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.207 DEBUG:boto:Method: GET 2024-04-16T20:04:40.207 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.207 DEBUG:boto:Data: 2024-04-16T20:04:40.207 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.207 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.207 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.207 DEBUG:boto:Params: {} 2024-04-16T20:04:40.207 DEBUG:boto:Token: None 2024-04-16T20:04:40.208 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd927fdd63cab1e1-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.254 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.255 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.255 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.255 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.255 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.255 DEBUG:boto:Method: GET 2024-04-16T20:04:40.255 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.255 DEBUG:boto:Data: 2024-04-16T20:04:40.255 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.256 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.256 DEBUG:boto:Params: {} 2024-04-16T20:04:40.256 DEBUG:boto:Token: None 2024-04-16T20:04:40.256 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eafb46e2820cfb2-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.260 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.260 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.260 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.260 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.260 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:04:40.260 DEBUG:boto:Data: 2024-04-16T20:04:40.260 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.260 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.260 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.261 DEBUG:boto:Params: {} 2024-04-16T20:04:40.261 DEBUG:boto:Token: None 2024-04-16T20:04:40.261 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33 2024-04-16T20:04:40.261 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T0rb94PLRL0yu/OA1gp4/RQmxys= 2024-04-16T20:04:40.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T0rb94PLRL0yu/OA1gp4/RQmxys=', 'Content-Length': '0'} 2024-04-16T20:04:40.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08bd34a558d9970-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.264 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.264 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.264 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.264 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:04:40.264 DEBUG:boto:Data: 2024-04-16T20:04:40.264 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.264 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.264 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.264 DEBUG:boto:Params: {} 2024-04-16T20:04:40.265 DEBUG:boto:Token: None 2024-04-16T20:04:40.265 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33 2024-04-16T20:04:40.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T0rb94PLRL0yu/OA1gp4/RQmxys= 2024-04-16T20:04:40.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T0rb94PLRL0yu/OA1gp4/RQmxys=', 'Content-Length': '0'} 2024-04-16T20:04:40.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e01435db9d6a9bee-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:04:40.267 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.267 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.268 DEBUG:boto:Method: GET 2024-04-16T20:04:40.268 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:04:40.268 DEBUG:boto:Data: 2024-04-16T20:04:40.268 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.268 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.268 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.268 DEBUG:boto:Params: {} 2024-04-16T20:04:40.268 DEBUG:boto:Token: None 2024-04-16T20:04:40.268 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33 2024-04-16T20:04:40.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/Nq8/XNbEAQfZTIf7M2xIVuwc= 2024-04-16T20:04:40.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/Nq8/XNbEAQfZTIf7M2xIVuwc=', 'Content-Length': '0'} 2024-04-16T20:04:40.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143fb09471c55ed8-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.314 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:04:40.314 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:04:40.314 DEBUG:boto:Method: GET 2024-04-16T20:04:40.315 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:04:40.315 DEBUG:boto:Data: 2024-04-16T20:04:40.315 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.315 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.315 DEBUG:boto:Params: {} 2024-04-16T20:04:40.315 DEBUG:boto:Token: None 2024-04-16T20:04:40.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj33 2024-04-16T20:04:40.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hp/Nq8/XNbEAQfZTIf7M2xIVuwc= 2024-04-16T20:04:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hp/Nq8/XNbEAQfZTIf7M2xIVuwc=', 'Content-Length': '0'} 2024-04-16T20:04:40.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072b6cd84b404770-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.318 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.318 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.318 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.318 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.319 DEBUG:boto:Method: GET 2024-04-16T20:04:40.319 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.319 DEBUG:boto:Data: 2024-04-16T20:04:40.319 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.319 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.319 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.319 DEBUG:boto:Params: {} 2024-04-16T20:04:40.319 DEBUG:boto:Token: None 2024-04-16T20:04:40.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e40b877051a7326c-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.362 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.363 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.363 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.363 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.363 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.363 DEBUG:boto:Method: GET 2024-04-16T20:04:40.363 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:04:40.363 DEBUG:boto:Data: 2024-04-16T20:04:40.363 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.364 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.364 DEBUG:boto:Params: {} 2024-04-16T20:04:40.364 DEBUG:boto:Token: None 2024-04-16T20:04:40.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9cd562728772eb-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.368 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:04:40.368 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.368 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.369 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.369 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.369 DEBUG:boto:Method: GET 2024-04-16T20:04:40.369 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.369 DEBUG:boto:Data: 2024-04-16T20:04:40.369 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.369 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.369 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.369 DEBUG:boto:Params: {} 2024-04-16T20:04:40.369 DEBUG:boto:Token: None 2024-04-16T20:04:40.370 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.370 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE= 2024-04-16T20:04:40.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE=', 'Content-Length': '0'} 2024-04-16T20:04:40.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1392317f78feed-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.414 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.415 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.415 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.415 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.415 DEBUG:boto:Method: GET 2024-04-16T20:04:40.415 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.415 DEBUG:boto:Data: 2024-04-16T20:04:40.415 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.415 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.415 DEBUG:boto:Params: {} 2024-04-16T20:04:40.416 DEBUG:boto:Token: None 2024-04-16T20:04:40.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE= 2024-04-16T20:04:40.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE=', 'Content-Length': '0'} 2024-04-16T20:04:40.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb0d67706e048ec-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.418 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.419 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.419 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.419 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.419 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.419 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.419 DEBUG:boto:Data: 2024-04-16T20:04:40.419 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.419 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.419 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.419 DEBUG:boto:Params: {} 2024-04-16T20:04:40.420 DEBUG:boto:Token: None 2024-04-16T20:04:40.420 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Ab8XzlAafuJYojO5vZ4TDHZJcQ= 2024-04-16T20:04:40.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Ab8XzlAafuJYojO5vZ4TDHZJcQ=', 'Content-Length': '0'} 2024-04-16T20:04:40.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3a4b3b33598f47-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.422 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.422 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.422 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.423 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.423 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.423 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.423 DEBUG:boto:Data: 2024-04-16T20:04:40.423 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.425 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.426 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.426 DEBUG:boto:Params: {} 2024-04-16T20:04:40.426 DEBUG:boto:Token: None 2024-04-16T20:04:40.426 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3Ab8XzlAafuJYojO5vZ4TDHZJcQ= 2024-04-16T20:04:40.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3Ab8XzlAafuJYojO5vZ4TDHZJcQ=', 'Content-Length': '0'} 2024-04-16T20:04:40.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d455af9939e831-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:04:40.428 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.429 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.429 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.429 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.429 DEBUG:boto:Method: GET 2024-04-16T20:04:40.429 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.429 DEBUG:boto:Data: 2024-04-16T20:04:40.429 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.429 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.429 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.429 DEBUG:boto:Params: {} 2024-04-16T20:04:40.430 DEBUG:boto:Token: None 2024-04-16T20:04:40.430 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.430 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE= 2024-04-16T20:04:40.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE=', 'Content-Length': '0'} 2024-04-16T20:04:40.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1bd716f51d19c3c-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.474 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.474 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.475 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.475 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.475 DEBUG:boto:Method: GET 2024-04-16T20:04:40.475 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.475 DEBUG:boto:Data: 2024-04-16T20:04:40.475 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.475 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.475 DEBUG:boto:Params: {} 2024-04-16T20:04:40.475 DEBUG:boto:Token: None 2024-04-16T20:04:40.476 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:04:40.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE= 2024-04-16T20:04:40.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ar1ODv0NTsWTfBads8PcmwCP0zE=', 'Content-Length': '0'} 2024-04-16T20:04:40.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0f014f14047c20-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.478 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.478 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.478 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.479 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.479 DEBUG:boto:Method: GET 2024-04-16T20:04:40.479 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.479 DEBUG:boto:Data: 2024-04-16T20:04:40.479 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.479 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.479 DEBUG:boto:Params: {} 2024-04-16T20:04:40.479 DEBUG:boto:Token: None 2024-04-16T20:04:40.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c696ccae9271b89-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.522 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.523 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.523 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.523 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.523 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.523 DEBUG:boto:Method: GET 2024-04-16T20:04:40.523 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.523 DEBUG:boto:Data: 2024-04-16T20:04:40.524 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.524 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.524 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.524 DEBUG:boto:Params: {} 2024-04-16T20:04:40.524 DEBUG:boto:Token: None 2024-04-16T20:04:40.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dce2952bcbee238-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.528 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.528 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.528 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.529 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.529 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:04:40.529 DEBUG:boto:Data: 2024-04-16T20:04:40.529 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.529 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.529 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.529 DEBUG:boto:Params: {} 2024-04-16T20:04:40.529 DEBUG:boto:Token: None 2024-04-16T20:04:40.529 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34 2024-04-16T20:04:40.530 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pounXgnEbrzyLIGQ6guq+I2yhg0= 2024-04-16T20:04:40.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pounXgnEbrzyLIGQ6guq+I2yhg0=', 'Content-Length': '0'} 2024-04-16T20:04:40.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09ab556778714f7-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.532 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.532 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.532 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.532 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:04:40.532 DEBUG:boto:Data: 2024-04-16T20:04:40.532 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.532 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.532 DEBUG:boto:Params: {} 2024-04-16T20:04:40.533 DEBUG:boto:Token: None 2024-04-16T20:04:40.533 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34 2024-04-16T20:04:40.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pounXgnEbrzyLIGQ6guq+I2yhg0= 2024-04-16T20:04:40.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pounXgnEbrzyLIGQ6guq+I2yhg0=', 'Content-Length': '0'} 2024-04-16T20:04:40.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4e17c230bee68c-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:04:40.535 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.535 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.535 DEBUG:boto:Method: GET 2024-04-16T20:04:40.535 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:04:40.535 DEBUG:boto:Data: 2024-04-16T20:04:40.536 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.536 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.536 DEBUG:boto:Params: {} 2024-04-16T20:04:40.536 DEBUG:boto:Token: None 2024-04-16T20:04:40.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34 2024-04-16T20:04:40.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tyd00MXVZGfNE8Yh4e2lfxdwr6c= 2024-04-16T20:04:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tyd00MXVZGfNE8Yh4e2lfxdwr6c=', 'Content-Length': '0'} 2024-04-16T20:04:40.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea3b6cea1e1331a-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.582 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:04:40.583 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:04:40.583 DEBUG:boto:Method: GET 2024-04-16T20:04:40.583 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:04:40.583 DEBUG:boto:Data: 2024-04-16T20:04:40.583 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.583 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.583 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.583 DEBUG:boto:Params: {} 2024-04-16T20:04:40.583 DEBUG:boto:Token: None 2024-04-16T20:04:40.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj34 2024-04-16T20:04:40.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tyd00MXVZGfNE8Yh4e2lfxdwr6c= 2024-04-16T20:04:40.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tyd00MXVZGfNE8Yh4e2lfxdwr6c=', 'Content-Length': '0'} 2024-04-16T20:04:40.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfe9d2bcc09b15b-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.586 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.586 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.586 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.586 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.586 DEBUG:boto:Method: GET 2024-04-16T20:04:40.586 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.586 DEBUG:boto:Data: 2024-04-16T20:04:40.586 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.587 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.587 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.587 DEBUG:boto:Params: {} 2024-04-16T20:04:40.587 DEBUG:boto:Token: None 2024-04-16T20:04:40.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c75deb4fb0931077-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.590 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.590 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.590 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.590 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.591 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.591 DEBUG:boto:Method: GET 2024-04-16T20:04:40.591 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:04:40.591 DEBUG:boto:Data: 2024-04-16T20:04:40.591 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.591 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.591 DEBUG:boto:Params: {} 2024-04-16T20:04:40.591 DEBUG:boto:Token: None 2024-04-16T20:04:40.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d4641b2e9f0039b-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.595 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:04:40.596 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.596 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.596 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.596 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.596 DEBUG:boto:Method: GET 2024-04-16T20:04:40.596 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.596 DEBUG:boto:Data: 2024-04-16T20:04:40.596 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.596 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.597 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.597 DEBUG:boto:Params: {} 2024-04-16T20:04:40.597 DEBUG:boto:Token: None 2024-04-16T20:04:40.597 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw= 2024-04-16T20:04:40.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw=', 'Content-Length': '0'} 2024-04-16T20:04:40.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e0764b8f6b1c00-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.642 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.642 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.643 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.643 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.643 DEBUG:boto:Method: GET 2024-04-16T20:04:40.643 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.643 DEBUG:boto:Data: 2024-04-16T20:04:40.643 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.643 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.643 DEBUG:boto:Params: {} 2024-04-16T20:04:40.644 DEBUG:boto:Token: None 2024-04-16T20:04:40.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw= 2024-04-16T20:04:40.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw=', 'Content-Length': '0'} 2024-04-16T20:04:40.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3821a30d7061f04-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.646 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.647 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.647 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.647 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.647 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.647 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.647 DEBUG:boto:Data: 2024-04-16T20:04:40.647 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.647 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.647 DEBUG:boto:Params: {} 2024-04-16T20:04:40.648 DEBUG:boto:Token: None 2024-04-16T20:04:40.648 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9bq+7HH2y+RqZ1NhdV7ZpOsVI7Q= 2024-04-16T20:04:40.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9bq+7HH2y+RqZ1NhdV7ZpOsVI7Q=', 'Content-Length': '0'} 2024-04-16T20:04:40.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562845df8366f013-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.650 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.650 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.650 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.650 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.650 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.651 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.651 DEBUG:boto:Data: 2024-04-16T20:04:40.651 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.651 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.651 DEBUG:boto:Params: {} 2024-04-16T20:04:40.651 DEBUG:boto:Token: None 2024-04-16T20:04:40.651 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.651 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9bq+7HH2y+RqZ1NhdV7ZpOsVI7Q= 2024-04-16T20:04:40.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9bq+7HH2y+RqZ1NhdV7ZpOsVI7Q=', 'Content-Length': '0'} 2024-04-16T20:04:40.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc01ae0fb4a50d9-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:04:40.654 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.654 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.654 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.654 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.654 DEBUG:boto:Method: GET 2024-04-16T20:04:40.654 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.654 DEBUG:boto:Data: 2024-04-16T20:04:40.654 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.654 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.655 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.655 DEBUG:boto:Params: {} 2024-04-16T20:04:40.655 DEBUG:boto:Token: None 2024-04-16T20:04:40.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw= 2024-04-16T20:04:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw=', 'Content-Length': '0'} 2024-04-16T20:04:40.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717c8e1c0abb24be-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.698 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.698 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.699 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.699 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.699 DEBUG:boto:Method: GET 2024-04-16T20:04:40.699 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.699 DEBUG:boto:Data: 2024-04-16T20:04:40.699 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.699 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.699 DEBUG:boto:Params: {} 2024-04-16T20:04:40.699 DEBUG:boto:Token: None 2024-04-16T20:04:40.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:04:40.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw= 2024-04-16T20:04:40.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bG+9p8W1Tp1fY4WY/1+mGoxoTZw=', 'Content-Length': '0'} 2024-04-16T20:04:40.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4735d8f125657f-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.702 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.702 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.702 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.702 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.702 DEBUG:boto:Method: GET 2024-04-16T20:04:40.703 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.703 DEBUG:boto:Data: 2024-04-16T20:04:40.703 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.703 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.703 DEBUG:boto:Params: {} 2024-04-16T20:04:40.703 DEBUG:boto:Token: None 2024-04-16T20:04:40.703 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.703 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c345d1f8aca4341e-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.706 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.707 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.707 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.707 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.707 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.707 DEBUG:boto:Method: GET 2024-04-16T20:04:40.707 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.707 DEBUG:boto:Data: 2024-04-16T20:04:40.707 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.708 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.708 DEBUG:boto:Params: {} 2024-04-16T20:04:40.708 DEBUG:boto:Token: None 2024-04-16T20:04:40.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c323096f5600147-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.712 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.712 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.712 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.712 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.712 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:04:40.712 DEBUG:boto:Data: 2024-04-16T20:04:40.713 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.713 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.713 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.713 DEBUG:boto:Params: {} 2024-04-16T20:04:40.713 DEBUG:boto:Token: None 2024-04-16T20:04:40.713 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38 2024-04-16T20:04:40.713 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oc/saoesk7XUVK8H+SjdcwVdEY0= 2024-04-16T20:04:40.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oc/saoesk7XUVK8H+SjdcwVdEY0=', 'Content-Length': '0'} 2024-04-16T20:04:40.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c305a0f541013e59-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.715 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.716 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.716 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.716 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:04:40.716 DEBUG:boto:Data: 2024-04-16T20:04:40.716 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.716 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.716 DEBUG:boto:Params: {} 2024-04-16T20:04:40.716 DEBUG:boto:Token: None 2024-04-16T20:04:40.716 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38 2024-04-16T20:04:40.717 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oc/saoesk7XUVK8H+SjdcwVdEY0= 2024-04-16T20:04:40.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oc/saoesk7XUVK8H+SjdcwVdEY0=', 'Content-Length': '0'} 2024-04-16T20:04:40.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b282fe0ca32061-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:04:40.719 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.719 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.719 DEBUG:boto:Method: GET 2024-04-16T20:04:40.719 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:04:40.719 DEBUG:boto:Data: 2024-04-16T20:04:40.719 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.719 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.719 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.720 DEBUG:boto:Params: {} 2024-04-16T20:04:40.720 DEBUG:boto:Token: None 2024-04-16T20:04:40.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38 2024-04-16T20:04:40.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1z6Q4HDBrAHcb2/lSR3uwgiaCg= 2024-04-16T20:04:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1z6Q4HDBrAHcb2/lSR3uwgiaCg=', 'Content-Length': '0'} 2024-04-16T20:04:40.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb8af33650a4d0f-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.762 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:04:40.763 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:04:40.763 DEBUG:boto:Method: GET 2024-04-16T20:04:40.763 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:04:40.763 DEBUG:boto:Data: 2024-04-16T20:04:40.763 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.763 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.763 DEBUG:boto:Params: {} 2024-04-16T20:04:40.763 DEBUG:boto:Token: None 2024-04-16T20:04:40.764 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj38 2024-04-16T20:04:40.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e1z6Q4HDBrAHcb2/lSR3uwgiaCg= 2024-04-16T20:04:40.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e1z6Q4HDBrAHcb2/lSR3uwgiaCg=', 'Content-Length': '0'} 2024-04-16T20:04:40.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a520357f36c5244-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.766 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.766 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.766 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.766 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.767 DEBUG:boto:Method: GET 2024-04-16T20:04:40.767 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.767 DEBUG:boto:Data: 2024-04-16T20:04:40.767 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.767 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.767 DEBUG:boto:Params: {} 2024-04-16T20:04:40.767 DEBUG:boto:Token: None 2024-04-16T20:04:40.767 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30397b9f5b0be2a-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.770 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.771 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.771 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.771 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.771 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.771 DEBUG:boto:Method: GET 2024-04-16T20:04:40.771 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:04:40.771 DEBUG:boto:Data: 2024-04-16T20:04:40.771 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.772 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.772 DEBUG:boto:Params: {} 2024-04-16T20:04:40.772 DEBUG:boto:Token: None 2024-04-16T20:04:40.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce327f5771c5f9c9-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.775 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:04:40.776 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.776 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.776 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.776 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.776 DEBUG:boto:Method: GET 2024-04-16T20:04:40.776 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.776 DEBUG:boto:Data: 2024-04-16T20:04:40.776 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.777 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.777 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.777 DEBUG:boto:Params: {} 2024-04-16T20:04:40.777 DEBUG:boto:Token: None 2024-04-16T20:04:40.777 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs= 2024-04-16T20:04:40.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs=', 'Content-Length': '0'} 2024-04-16T20:04:40.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef20bca8e021423-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.822 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.823 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.823 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.823 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.823 DEBUG:boto:Method: GET 2024-04-16T20:04:40.823 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.823 DEBUG:boto:Data: 2024-04-16T20:04:40.823 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.823 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.824 DEBUG:boto:Params: {} 2024-04-16T20:04:40.824 DEBUG:boto:Token: None 2024-04-16T20:04:40.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs= 2024-04-16T20:04:40.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs=', 'Content-Length': '0'} 2024-04-16T20:04:40.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf16989bcca2c24a-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.827 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.827 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.827 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.827 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.827 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.827 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.827 DEBUG:boto:Data: 2024-04-16T20:04:40.828 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.828 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.828 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.828 DEBUG:boto:Params: {} 2024-04-16T20:04:40.828 DEBUG:boto:Token: None 2024-04-16T20:04:40.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOolGPLQ7U4sGSr5rbbh/Rhvutg= 2024-04-16T20:04:40.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOolGPLQ7U4sGSr5rbbh/Rhvutg=', 'Content-Length': '0'} 2024-04-16T20:04:40.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000946225811d94099-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.831 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.831 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.831 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.831 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.831 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.831 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.831 DEBUG:boto:Data: 2024-04-16T20:04:40.831 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.831 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.832 DEBUG:boto:Params: {} 2024-04-16T20:04:40.832 DEBUG:boto:Token: None 2024-04-16T20:04:40.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XOolGPLQ7U4sGSr5rbbh/Rhvutg= 2024-04-16T20:04:40.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XOolGPLQ7U4sGSr5rbbh/Rhvutg=', 'Content-Length': '0'} 2024-04-16T20:04:40.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9373b02cd382ed-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:04:40.834 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.834 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.834 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.835 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.835 DEBUG:boto:Method: GET 2024-04-16T20:04:40.835 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.835 DEBUG:boto:Data: 2024-04-16T20:04:40.835 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.835 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.835 DEBUG:boto:Params: {} 2024-04-16T20:04:40.835 DEBUG:boto:Token: None 2024-04-16T20:04:40.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs= 2024-04-16T20:04:40.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs=', 'Content-Length': '0'} 2024-04-16T20:04:40.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b08392505a0de42-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.878 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.878 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.878 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.879 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.879 DEBUG:boto:Method: GET 2024-04-16T20:04:40.879 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.879 DEBUG:boto:Data: 2024-04-16T20:04:40.879 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.879 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.879 DEBUG:boto:Params: {} 2024-04-16T20:04:40.879 DEBUG:boto:Token: None 2024-04-16T20:04:40.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:04:40.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs= 2024-04-16T20:04:40.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rbFAEPOS+9eCXvtBf2u/ufAXkZs=', 'Content-Length': '0'} 2024-04-16T20:04:40.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e726fba5c43297a-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.882 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.883 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.883 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.883 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.883 DEBUG:boto:Method: GET 2024-04-16T20:04:40.883 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.883 DEBUG:boto:Data: 2024-04-16T20:04:40.883 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.883 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.883 DEBUG:boto:Params: {} 2024-04-16T20:04:40.884 DEBUG:boto:Token: None 2024-04-16T20:04:40.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096bdd8383887cfff-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.886 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.887 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.887 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.887 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.887 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.887 DEBUG:boto:Method: GET 2024-04-16T20:04:40.887 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.887 DEBUG:boto:Data: 2024-04-16T20:04:40.887 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.888 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.888 DEBUG:boto:Params: {} 2024-04-16T20:04:40.888 DEBUG:boto:Token: None 2024-04-16T20:04:40.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002cf3aad5aedc189-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.891 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.892 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.892 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.892 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.892 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:04:40.892 DEBUG:boto:Data: 2024-04-16T20:04:40.892 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.892 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.892 DEBUG:boto:Params: {} 2024-04-16T20:04:40.893 DEBUG:boto:Token: None 2024-04-16T20:04:40.893 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44 2024-04-16T20:04:40.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MooFgXfNkg9fcDP49Odq79Z3XwY= 2024-04-16T20:04:40.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MooFgXfNkg9fcDP49Odq79Z3XwY=', 'Content-Length': '0'} 2024-04-16T20:04:40.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb69f4c5e4ef643-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.895 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.895 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.895 DEBUG:boto:Method: HEAD 2024-04-16T20:04:40.895 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:04:40.896 DEBUG:boto:Data: 2024-04-16T20:04:40.896 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.896 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.896 DEBUG:boto:Params: {} 2024-04-16T20:04:40.896 DEBUG:boto:Token: None 2024-04-16T20:04:40.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44 2024-04-16T20:04:40.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MooFgXfNkg9fcDP49Odq79Z3XwY= 2024-04-16T20:04:40.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MooFgXfNkg9fcDP49Odq79Z3XwY=', 'Content-Length': '0'} 2024-04-16T20:04:40.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2f4c20d7a29ed4-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:04:40.899 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.899 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.899 DEBUG:boto:Method: GET 2024-04-16T20:04:40.899 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:04:40.899 DEBUG:boto:Data: 2024-04-16T20:04:40.899 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.899 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.900 DEBUG:boto:Params: {} 2024-04-16T20:04:40.900 DEBUG:boto:Token: None 2024-04-16T20:04:40.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44 2024-04-16T20:04:40.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gyB25kPnhZCXglI+T4Bv7ZrAZv4= 2024-04-16T20:04:40.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gyB25kPnhZCXglI+T4Bv7ZrAZv4=', 'Content-Length': '0'} 2024-04-16T20:04:40.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa89e921de64f09-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.942 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:04:40.942 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:04:40.942 DEBUG:boto:Method: GET 2024-04-16T20:04:40.943 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:04:40.943 DEBUG:boto:Data: 2024-04-16T20:04:40.943 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.943 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.943 DEBUG:boto:Params: {} 2024-04-16T20:04:40.943 DEBUG:boto:Token: None 2024-04-16T20:04:40.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj44 2024-04-16T20:04:40.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gyB25kPnhZCXglI+T4Bv7ZrAZv4= 2024-04-16T20:04:40.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gyB25kPnhZCXglI+T4Bv7ZrAZv4=', 'Content-Length': '0'} 2024-04-16T20:04:40.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb20508a20a264c-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.946 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.946 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.946 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.946 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.946 DEBUG:boto:Method: GET 2024-04-16T20:04:40.946 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.947 DEBUG:boto:Data: 2024-04-16T20:04:40.947 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.947 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.947 DEBUG:boto:Params: {} 2024-04-16T20:04:40.947 DEBUG:boto:Token: None 2024-04-16T20:04:40.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b2156c8670c0cef-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.950 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.950 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:40.950 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:40.950 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.950 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.950 DEBUG:boto:Method: GET 2024-04-16T20:04:40.950 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:04:40.951 DEBUG:boto:Data: 2024-04-16T20:04:40.951 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:40.951 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.951 DEBUG:boto:Params: {} 2024-04-16T20:04:40.951 DEBUG:boto:Token: None 2024-04-16T20:04:40.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/ 2024-04-16T20:04:40.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU= 2024-04-16T20:04:40.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZp8FqTkT/ipxAIuvXoZeNCJYNU=', 'Content-Length': '0'} 2024-04-16T20:04:40.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099d6530e2476d708-00661ed9d8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:40.955 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:40.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:04:40.955 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:40.955 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:40.955 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:40.955 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:40.955 DEBUG:boto:Method: GET 2024-04-16T20:04:40.956 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:40.956 DEBUG:boto:Data: 2024-04-16T20:04:40.956 DEBUG:boto:Headers: {} 2024-04-16T20:04:40.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:40.956 DEBUG:boto:Port: 8000 2024-04-16T20:04:40.956 DEBUG:boto:Params: {} 2024-04-16T20:04:40.956 DEBUG:boto:Token: None 2024-04-16T20:04:40.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:40 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:40.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bPXa7407Ld68Rx8/QobuF0wlVMI= 2024-04-16T20:04:40.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bPXa7407Ld68Rx8/QobuF0wlVMI=', 'Content-Length': '0'} 2024-04-16T20:04:40.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c49a077da672ff2-00661ed9d8-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.002 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.002 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.002 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.003 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.003 DEBUG:boto:Method: GET 2024-04-16T20:04:41.003 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.003 DEBUG:boto:Data: 2024-04-16T20:04:41.003 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.003 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.003 DEBUG:boto:Params: {} 2024-04-16T20:04:41.003 DEBUG:boto:Token: None 2024-04-16T20:04:41.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k= 2024-04-16T20:04:41.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k=', 'Content-Length': '0'} 2024-04-16T20:04:41.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2cc9fcbe4ef6027-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.006 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.006 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.006 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.006 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.006 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.006 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.006 DEBUG:boto:Data: 2024-04-16T20:04:41.006 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.006 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.006 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.007 DEBUG:boto:Params: {} 2024-04-16T20:04:41.007 DEBUG:boto:Token: None 2024-04-16T20:04:41.007 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jT8dTPfZSnV3nhQGSYtmAcKBuaQ= 2024-04-16T20:04:41.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jT8dTPfZSnV3nhQGSYtmAcKBuaQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7388596c2da471-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.009 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.009 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.009 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.009 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.010 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.010 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.010 DEBUG:boto:Data: 2024-04-16T20:04:41.010 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.010 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.010 DEBUG:boto:Params: {} 2024-04-16T20:04:41.010 DEBUG:boto:Token: None 2024-04-16T20:04:41.010 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.010 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jT8dTPfZSnV3nhQGSYtmAcKBuaQ= 2024-04-16T20:04:41.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jT8dTPfZSnV3nhQGSYtmAcKBuaQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c8a44f0c14108a7-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:04:41.013 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.013 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.013 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.013 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.013 DEBUG:boto:Method: GET 2024-04-16T20:04:41.013 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.013 DEBUG:boto:Data: 2024-04-16T20:04:41.013 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.013 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.013 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.014 DEBUG:boto:Params: {} 2024-04-16T20:04:41.014 DEBUG:boto:Token: None 2024-04-16T20:04:41.014 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.014 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k= 2024-04-16T20:04:41.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k=', 'Content-Length': '0'} 2024-04-16T20:04:41.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e1c2f54e06038d-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.058 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.058 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.058 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.059 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.059 DEBUG:boto:Method: GET 2024-04-16T20:04:41.059 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.059 DEBUG:boto:Data: 2024-04-16T20:04:41.059 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.059 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.059 DEBUG:boto:Params: {} 2024-04-16T20:04:41.059 DEBUG:boto:Token: None 2024-04-16T20:04:41.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:04:41.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k= 2024-04-16T20:04:41.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rmjORKRdFU9OqkX4E1ejX1ibq1k=', 'Content-Length': '0'} 2024-04-16T20:04:41.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb47b297d338d37-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.062 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.062 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.062 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.062 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.063 DEBUG:boto:Method: GET 2024-04-16T20:04:41.063 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.063 DEBUG:boto:Data: 2024-04-16T20:04:41.063 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.063 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.063 DEBUG:boto:Params: {} 2024-04-16T20:04:41.063 DEBUG:boto:Token: None 2024-04-16T20:04:41.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080731d55094b48de-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.106 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.107 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.107 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.107 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.107 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.107 DEBUG:boto:Method: GET 2024-04-16T20:04:41.107 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.107 DEBUG:boto:Data: 2024-04-16T20:04:41.107 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.107 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.107 DEBUG:boto:Params: {} 2024-04-16T20:04:41.108 DEBUG:boto:Token: None 2024-04-16T20:04:41.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecd8837e41e3417f-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.111 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.112 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.112 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.112 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.112 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:04:41.112 DEBUG:boto:Data: 2024-04-16T20:04:41.112 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.112 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.112 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.112 DEBUG:boto:Params: {} 2024-04-16T20:04:41.112 DEBUG:boto:Token: None 2024-04-16T20:04:41.113 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46 2024-04-16T20:04:41.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XmMemvuezKNfFiBiQyQvJmGYBJ0= 2024-04-16T20:04:41.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XmMemvuezKNfFiBiQyQvJmGYBJ0=', 'Content-Length': '0'} 2024-04-16T20:04:41.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021bd1fd187f40cf6-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.115 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.115 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.115 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.115 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:04:41.115 DEBUG:boto:Data: 2024-04-16T20:04:41.115 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.115 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.115 DEBUG:boto:Params: {} 2024-04-16T20:04:41.116 DEBUG:boto:Token: None 2024-04-16T20:04:41.116 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46 2024-04-16T20:04:41.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XmMemvuezKNfFiBiQyQvJmGYBJ0= 2024-04-16T20:04:41.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XmMemvuezKNfFiBiQyQvJmGYBJ0=', 'Content-Length': '0'} 2024-04-16T20:04:41.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fee91a54f7e2cbe-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:04:41.118 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.118 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.119 DEBUG:boto:Method: GET 2024-04-16T20:04:41.119 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:04:41.119 DEBUG:boto:Data: 2024-04-16T20:04:41.119 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.119 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.119 DEBUG:boto:Params: {} 2024-04-16T20:04:41.119 DEBUG:boto:Token: None 2024-04-16T20:04:41.119 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46 2024-04-16T20:04:41.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b37Qq3dGKreyjxFCudeF6ZJQAr8= 2024-04-16T20:04:41.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b37Qq3dGKreyjxFCudeF6ZJQAr8=', 'Content-Length': '0'} 2024-04-16T20:04:41.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003333bfbdc8145ebd-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.162 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:04:41.162 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:04:41.163 DEBUG:boto:Method: GET 2024-04-16T20:04:41.163 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:04:41.163 DEBUG:boto:Data: 2024-04-16T20:04:41.163 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.163 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.163 DEBUG:boto:Params: {} 2024-04-16T20:04:41.163 DEBUG:boto:Token: None 2024-04-16T20:04:41.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj46 2024-04-16T20:04:41.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b37Qq3dGKreyjxFCudeF6ZJQAr8= 2024-04-16T20:04:41.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b37Qq3dGKreyjxFCudeF6ZJQAr8=', 'Content-Length': '0'} 2024-04-16T20:04:41.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007870e33494e74e75-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.165 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.166 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.166 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.166 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.166 DEBUG:boto:Method: GET 2024-04-16T20:04:41.166 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.166 DEBUG:boto:Data: 2024-04-16T20:04:41.166 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.166 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.166 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.166 DEBUG:boto:Params: {} 2024-04-16T20:04:41.166 DEBUG:boto:Token: None 2024-04-16T20:04:41.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c6839ae4ba5b933-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.210 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.211 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.211 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.211 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.211 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.211 DEBUG:boto:Method: GET 2024-04-16T20:04:41.211 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:04:41.211 DEBUG:boto:Data: 2024-04-16T20:04:41.211 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.212 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.212 DEBUG:boto:Params: {} 2024-04-16T20:04:41.212 DEBUG:boto:Token: None 2024-04-16T20:04:41.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872d5be249238b86-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.215 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:04:41.216 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.216 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.216 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.216 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.216 DEBUG:boto:Method: GET 2024-04-16T20:04:41.216 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.216 DEBUG:boto:Data: 2024-04-16T20:04:41.216 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.216 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.217 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.217 DEBUG:boto:Params: {} 2024-04-16T20:04:41.217 DEBUG:boto:Token: None 2024-04-16T20:04:41.217 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4= 2024-04-16T20:04:41.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4=', 'Content-Length': '0'} 2024-04-16T20:04:41.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000390cb7247f655bdd-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.262 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.263 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.263 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.263 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.263 DEBUG:boto:Method: GET 2024-04-16T20:04:41.263 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.263 DEBUG:boto:Data: 2024-04-16T20:04:41.263 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.263 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.263 DEBUG:boto:Params: {} 2024-04-16T20:04:41.263 DEBUG:boto:Token: None 2024-04-16T20:04:41.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4= 2024-04-16T20:04:41.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4=', 'Content-Length': '0'} 2024-04-16T20:04:41.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020435cfb4c0df5f1-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.266 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.266 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.266 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.266 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.266 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.267 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.267 DEBUG:boto:Data: 2024-04-16T20:04:41.267 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.267 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.267 DEBUG:boto:Params: {} 2024-04-16T20:04:41.267 DEBUG:boto:Token: None 2024-04-16T20:04:41.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rThrFQIDlq4S1SsqI8PUmphSlp0= 2024-04-16T20:04:41.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rThrFQIDlq4S1SsqI8PUmphSlp0=', 'Content-Length': '0'} 2024-04-16T20:04:41.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080043368a787a48f-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.270 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.270 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.270 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.270 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.270 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.270 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.270 DEBUG:boto:Data: 2024-04-16T20:04:41.270 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.270 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.271 DEBUG:boto:Params: {} 2024-04-16T20:04:41.271 DEBUG:boto:Token: None 2024-04-16T20:04:41.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rThrFQIDlq4S1SsqI8PUmphSlp0= 2024-04-16T20:04:41.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rThrFQIDlq4S1SsqI8PUmphSlp0=', 'Content-Length': '0'} 2024-04-16T20:04:41.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf5b61a2739b4a8-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:04:41.273 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.274 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.274 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.274 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.274 DEBUG:boto:Method: GET 2024-04-16T20:04:41.274 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.274 DEBUG:boto:Data: 2024-04-16T20:04:41.274 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.274 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.274 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.274 DEBUG:boto:Params: {} 2024-04-16T20:04:41.274 DEBUG:boto:Token: None 2024-04-16T20:04:41.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4= 2024-04-16T20:04:41.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4=', 'Content-Length': '0'} 2024-04-16T20:04:41.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008810ec6d7de2b743-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.318 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.318 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.318 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.318 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.319 DEBUG:boto:Method: GET 2024-04-16T20:04:41.319 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.319 DEBUG:boto:Data: 2024-04-16T20:04:41.319 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.319 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.319 DEBUG:boto:Params: {} 2024-04-16T20:04:41.319 DEBUG:boto:Token: None 2024-04-16T20:04:41.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:04:41.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4= 2024-04-16T20:04:41.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6cZ3LQk4tkwZsYX8IQHtUUZF2b4=', 'Content-Length': '0'} 2024-04-16T20:04:41.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e1fdcb0e328456-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.322 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.322 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.322 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.322 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.322 DEBUG:boto:Method: GET 2024-04-16T20:04:41.322 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.322 DEBUG:boto:Data: 2024-04-16T20:04:41.322 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.322 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.323 DEBUG:boto:Params: {} 2024-04-16T20:04:41.323 DEBUG:boto:Token: None 2024-04-16T20:04:41.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72a8149b2750676-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.366 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.366 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.367 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.367 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.367 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.367 DEBUG:boto:Method: GET 2024-04-16T20:04:41.367 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.367 DEBUG:boto:Data: 2024-04-16T20:04:41.367 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.367 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.367 DEBUG:boto:Params: {} 2024-04-16T20:04:41.367 DEBUG:boto:Token: None 2024-04-16T20:04:41.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b48799c792e65088-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.371 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.372 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.372 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.372 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.372 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:04:41.372 DEBUG:boto:Data: 2024-04-16T20:04:41.372 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.372 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.372 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.372 DEBUG:boto:Params: {} 2024-04-16T20:04:41.372 DEBUG:boto:Token: None 2024-04-16T20:04:41.373 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51 2024-04-16T20:04:41.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/7vPIOnYI+3532BWGMaK7WRld44= 2024-04-16T20:04:41.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/7vPIOnYI+3532BWGMaK7WRld44=', 'Content-Length': '0'} 2024-04-16T20:04:41.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e9a08de4b11609-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.375 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.375 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.375 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.375 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:04:41.375 DEBUG:boto:Data: 2024-04-16T20:04:41.375 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.376 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.376 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.376 DEBUG:boto:Params: {} 2024-04-16T20:04:41.376 DEBUG:boto:Token: None 2024-04-16T20:04:41.376 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51 2024-04-16T20:04:41.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/7vPIOnYI+3532BWGMaK7WRld44= 2024-04-16T20:04:41.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/7vPIOnYI+3532BWGMaK7WRld44=', 'Content-Length': '0'} 2024-04-16T20:04:41.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f6fd41f4fdfd180-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:04:41.379 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.379 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.379 DEBUG:boto:Method: GET 2024-04-16T20:04:41.379 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:04:41.379 DEBUG:boto:Data: 2024-04-16T20:04:41.379 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.379 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.379 DEBUG:boto:Params: {} 2024-04-16T20:04:41.380 DEBUG:boto:Token: None 2024-04-16T20:04:41.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51 2024-04-16T20:04:41.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/FkeK3INRqAMha4qC3Qd4Y+aCXQ= 2024-04-16T20:04:41.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/FkeK3INRqAMha4qC3Qd4Y+aCXQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eed1c812620913f-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.422 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:04:41.422 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:04:41.422 DEBUG:boto:Method: GET 2024-04-16T20:04:41.423 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:04:41.423 DEBUG:boto:Data: 2024-04-16T20:04:41.423 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.423 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.423 DEBUG:boto:Params: {} 2024-04-16T20:04:41.423 DEBUG:boto:Token: None 2024-04-16T20:04:41.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj51 2024-04-16T20:04:41.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/FkeK3INRqAMha4qC3Qd4Y+aCXQ= 2024-04-16T20:04:41.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/FkeK3INRqAMha4qC3Qd4Y+aCXQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069a25e879f505e43-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.425 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.426 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.426 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.426 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.426 DEBUG:boto:Method: GET 2024-04-16T20:04:41.426 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.426 DEBUG:boto:Data: 2024-04-16T20:04:41.426 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.426 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.426 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.426 DEBUG:boto:Params: {} 2024-04-16T20:04:41.427 DEBUG:boto:Token: None 2024-04-16T20:04:41.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c0b8a8c035364e-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.470 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.471 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.471 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.471 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.471 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.471 DEBUG:boto:Method: GET 2024-04-16T20:04:41.471 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:04:41.471 DEBUG:boto:Data: 2024-04-16T20:04:41.471 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.471 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.471 DEBUG:boto:Params: {} 2024-04-16T20:04:41.472 DEBUG:boto:Token: None 2024-04-16T20:04:41.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ba0cb3b448d066-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.475 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:04:41.475 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.475 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.476 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.476 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.476 DEBUG:boto:Method: GET 2024-04-16T20:04:41.476 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.476 DEBUG:boto:Data: 2024-04-16T20:04:41.476 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.476 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.476 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.476 DEBUG:boto:Params: {} 2024-04-16T20:04:41.476 DEBUG:boto:Token: None 2024-04-16T20:04:41.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4= 2024-04-16T20:04:41.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4=', 'Content-Length': '0'} 2024-04-16T20:04:41.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a8006698c6f941-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.522 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.522 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.522 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.523 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.523 DEBUG:boto:Method: GET 2024-04-16T20:04:41.523 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.523 DEBUG:boto:Data: 2024-04-16T20:04:41.523 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.523 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.523 DEBUG:boto:Params: {} 2024-04-16T20:04:41.523 DEBUG:boto:Token: None 2024-04-16T20:04:41.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4= 2024-04-16T20:04:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4=', 'Content-Length': '0'} 2024-04-16T20:04:41.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19e90ac9be16e74-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.526 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.526 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.526 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.527 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.527 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.527 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.527 DEBUG:boto:Data: 2024-04-16T20:04:41.527 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.527 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.527 DEBUG:boto:Params: {} 2024-04-16T20:04:41.527 DEBUG:boto:Token: None 2024-04-16T20:04:41.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+bmLsH19a76hdDzp39+0HkyhYEQ= 2024-04-16T20:04:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+bmLsH19a76hdDzp39+0HkyhYEQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a993df7fb3ea33-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.530 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.530 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.530 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.530 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.530 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.530 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.530 DEBUG:boto:Data: 2024-04-16T20:04:41.530 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.531 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.531 DEBUG:boto:Params: {} 2024-04-16T20:04:41.531 DEBUG:boto:Token: None 2024-04-16T20:04:41.531 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+bmLsH19a76hdDzp39+0HkyhYEQ= 2024-04-16T20:04:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+bmLsH19a76hdDzp39+0HkyhYEQ=', 'Content-Length': '0'} 2024-04-16T20:04:41.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6411eaf8189755d-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:04:41.533 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.533 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.534 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.534 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.534 DEBUG:boto:Method: GET 2024-04-16T20:04:41.534 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.534 DEBUG:boto:Data: 2024-04-16T20:04:41.534 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.534 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.534 DEBUG:boto:Params: {} 2024-04-16T20:04:41.534 DEBUG:boto:Token: None 2024-04-16T20:04:41.534 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4= 2024-04-16T20:04:41.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4=', 'Content-Length': '0'} 2024-04-16T20:04:41.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054717652f85a0a34-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.578 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.578 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.579 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.579 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.579 DEBUG:boto:Method: GET 2024-04-16T20:04:41.579 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.579 DEBUG:boto:Data: 2024-04-16T20:04:41.579 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.579 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.579 DEBUG:boto:Params: {} 2024-04-16T20:04:41.579 DEBUG:boto:Token: None 2024-04-16T20:04:41.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:04:41.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4= 2024-04-16T20:04:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:psRCKWSghNoq3j0kGVmeSqnLBY4=', 'Content-Length': '0'} 2024-04-16T20:04:41.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446f4df43d11b152-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.582 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.582 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.582 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.582 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.582 DEBUG:boto:Method: GET 2024-04-16T20:04:41.582 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.582 DEBUG:boto:Data: 2024-04-16T20:04:41.582 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.583 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.583 DEBUG:boto:Params: {} 2024-04-16T20:04:41.583 DEBUG:boto:Token: None 2024-04-16T20:04:41.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd50dc415a18a2ef-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.626 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.627 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.627 DEBUG:boto:Method: GET 2024-04-16T20:04:41.627 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.627 DEBUG:boto:Data: 2024-04-16T20:04:41.627 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.627 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.627 DEBUG:boto:Params: {} 2024-04-16T20:04:41.627 DEBUG:boto:Token: None 2024-04-16T20:04:41.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d10ceb7c0dfe7e-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.631 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.631 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.632 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.632 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.632 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:04:41.632 DEBUG:boto:Data: 2024-04-16T20:04:41.632 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.632 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.632 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.632 DEBUG:boto:Params: {} 2024-04-16T20:04:41.632 DEBUG:boto:Token: None 2024-04-16T20:04:41.632 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55 2024-04-16T20:04:41.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5Ebj76E/VbvMKRoZQ1nFnNUPvqU= 2024-04-16T20:04:41.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5Ebj76E/VbvMKRoZQ1nFnNUPvqU=', 'Content-Length': '0'} 2024-04-16T20:04:41.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f9a027451ce564-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.635 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.635 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.635 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.635 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:04:41.635 DEBUG:boto:Data: 2024-04-16T20:04:41.635 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.635 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.635 DEBUG:boto:Params: {} 2024-04-16T20:04:41.635 DEBUG:boto:Token: None 2024-04-16T20:04:41.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55 2024-04-16T20:04:41.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5Ebj76E/VbvMKRoZQ1nFnNUPvqU= 2024-04-16T20:04:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5Ebj76E/VbvMKRoZQ1nFnNUPvqU=', 'Content-Length': '0'} 2024-04-16T20:04:41.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a0d9116795147a-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:04:41.638 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.638 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.638 DEBUG:boto:Method: GET 2024-04-16T20:04:41.639 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:04:41.639 DEBUG:boto:Data: 2024-04-16T20:04:41.639 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.639 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.639 DEBUG:boto:Params: {} 2024-04-16T20:04:41.639 DEBUG:boto:Token: None 2024-04-16T20:04:41.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55 2024-04-16T20:04:41.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:phuZmu4tL9uyCW2i9dqGcch71Qg= 2024-04-16T20:04:41.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:phuZmu4tL9uyCW2i9dqGcch71Qg=', 'Content-Length': '0'} 2024-04-16T20:04:41.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2e5f8b1e4d16fe-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.682 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:04:41.682 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:04:41.682 DEBUG:boto:Method: GET 2024-04-16T20:04:41.683 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:04:41.683 DEBUG:boto:Data: 2024-04-16T20:04:41.683 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.683 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.683 DEBUG:boto:Params: {} 2024-04-16T20:04:41.683 DEBUG:boto:Token: None 2024-04-16T20:04:41.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj55 2024-04-16T20:04:41.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:phuZmu4tL9uyCW2i9dqGcch71Qg= 2024-04-16T20:04:41.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:phuZmu4tL9uyCW2i9dqGcch71Qg=', 'Content-Length': '0'} 2024-04-16T20:04:41.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002093636056757e75-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.686 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.686 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.686 DEBUG:boto:Method: GET 2024-04-16T20:04:41.686 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.687 DEBUG:boto:Data: 2024-04-16T20:04:41.687 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.687 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.687 DEBUG:boto:Params: {} 2024-04-16T20:04:41.687 DEBUG:boto:Token: None 2024-04-16T20:04:41.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000563156d729b4bfcd-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.730 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.730 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.731 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.731 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.731 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.731 DEBUG:boto:Method: GET 2024-04-16T20:04:41.731 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:04:41.731 DEBUG:boto:Data: 2024-04-16T20:04:41.731 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.731 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.731 DEBUG:boto:Params: {} 2024-04-16T20:04:41.731 DEBUG:boto:Token: None 2024-04-16T20:04:41.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b977301b9d6d89-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.735 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:04:41.735 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.735 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.735 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.735 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.736 DEBUG:boto:Method: GET 2024-04-16T20:04:41.736 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.736 DEBUG:boto:Data: 2024-04-16T20:04:41.736 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.736 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.736 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.736 DEBUG:boto:Params: {} 2024-04-16T20:04:41.736 DEBUG:boto:Token: None 2024-04-16T20:04:41.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk= 2024-04-16T20:04:41.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk=', 'Content-Length': '0'} 2024-04-16T20:04:41.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf8ced547c523af-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.782 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.782 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.782 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.783 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.783 DEBUG:boto:Method: GET 2024-04-16T20:04:41.783 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.783 DEBUG:boto:Data: 2024-04-16T20:04:41.783 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.783 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.783 DEBUG:boto:Params: {} 2024-04-16T20:04:41.783 DEBUG:boto:Token: None 2024-04-16T20:04:41.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk= 2024-04-16T20:04:41.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk=', 'Content-Length': '0'} 2024-04-16T20:04:41.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e316f6d464f59059-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.786 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.786 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.786 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.786 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.786 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.786 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.787 DEBUG:boto:Data: 2024-04-16T20:04:41.787 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.787 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.787 DEBUG:boto:Params: {} 2024-04-16T20:04:41.787 DEBUG:boto:Token: None 2024-04-16T20:04:41.787 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3SX6qezux9IxyXq/XpCLllsS9Q= 2024-04-16T20:04:41.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3SX6qezux9IxyXq/XpCLllsS9Q=', 'Content-Length': '0'} 2024-04-16T20:04:41.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df4eaa4dcdf9941-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.789 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.789 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.789 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.790 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.790 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.790 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.790 DEBUG:boto:Data: 2024-04-16T20:04:41.790 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.790 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.790 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.790 DEBUG:boto:Params: {} 2024-04-16T20:04:41.790 DEBUG:boto:Token: None 2024-04-16T20:04:41.790 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T3SX6qezux9IxyXq/XpCLllsS9Q= 2024-04-16T20:04:41.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T3SX6qezux9IxyXq/XpCLllsS9Q=', 'Content-Length': '0'} 2024-04-16T20:04:41.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10a3964e85608fb-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:04:41.793 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.793 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.793 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.793 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.793 DEBUG:boto:Method: GET 2024-04-16T20:04:41.793 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.794 DEBUG:boto:Data: 2024-04-16T20:04:41.794 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.794 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.794 DEBUG:boto:Params: {} 2024-04-16T20:04:41.794 DEBUG:boto:Token: None 2024-04-16T20:04:41.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.794 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk= 2024-04-16T20:04:41.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk=', 'Content-Length': '0'} 2024-04-16T20:04:41.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee77f4bed0ee42f4-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.838 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.838 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.838 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.839 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.839 DEBUG:boto:Method: GET 2024-04-16T20:04:41.839 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.839 DEBUG:boto:Data: 2024-04-16T20:04:41.839 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.839 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.839 DEBUG:boto:Params: {} 2024-04-16T20:04:41.839 DEBUG:boto:Token: None 2024-04-16T20:04:41.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:04:41.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk= 2024-04-16T20:04:41.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4J4vxehut0nY85yFYKCnF8P3Rk=', 'Content-Length': '0'} 2024-04-16T20:04:41.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000802418ff833ed7d-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.842 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.842 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.842 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.842 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.842 DEBUG:boto:Method: GET 2024-04-16T20:04:41.842 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.843 DEBUG:boto:Data: 2024-04-16T20:04:41.843 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.843 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.843 DEBUG:boto:Params: {} 2024-04-16T20:04:41.843 DEBUG:boto:Token: None 2024-04-16T20:04:41.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001889a99089905016-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.886 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.887 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.887 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.887 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.887 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.887 DEBUG:boto:Method: GET 2024-04-16T20:04:41.887 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.887 DEBUG:boto:Data: 2024-04-16T20:04:41.887 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.887 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.887 DEBUG:boto:Params: {} 2024-04-16T20:04:41.888 DEBUG:boto:Token: None 2024-04-16T20:04:41.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efde31c789b74258-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.891 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.891 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.891 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.891 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.891 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:04:41.892 DEBUG:boto:Data: 2024-04-16T20:04:41.892 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.892 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.892 DEBUG:boto:Params: {} 2024-04-16T20:04:41.892 DEBUG:boto:Token: None 2024-04-16T20:04:41.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57 2024-04-16T20:04:41.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CUQw3V3PUXbkmhzGEOacCOd4Ha8= 2024-04-16T20:04:41.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CUQw3V3PUXbkmhzGEOacCOd4Ha8=', 'Content-Length': '0'} 2024-04-16T20:04:41.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24a339601db6f6d-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.895 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.895 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.895 DEBUG:boto:Method: HEAD 2024-04-16T20:04:41.895 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:04:41.895 DEBUG:boto:Data: 2024-04-16T20:04:41.895 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.895 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.895 DEBUG:boto:Params: {} 2024-04-16T20:04:41.895 DEBUG:boto:Token: None 2024-04-16T20:04:41.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57 2024-04-16T20:04:41.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CUQw3V3PUXbkmhzGEOacCOd4Ha8= 2024-04-16T20:04:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CUQw3V3PUXbkmhzGEOacCOd4Ha8=', 'Content-Length': '0'} 2024-04-16T20:04:41.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000660c5046cca4f8cb-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:04:41.898 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.898 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.899 DEBUG:boto:Method: GET 2024-04-16T20:04:41.899 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:04:41.899 DEBUG:boto:Data: 2024-04-16T20:04:41.899 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.899 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.899 DEBUG:boto:Params: {} 2024-04-16T20:04:41.899 DEBUG:boto:Token: None 2024-04-16T20:04:41.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57 2024-04-16T20:04:41.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0MWgE0ZlrGGpo0KYcdjaSk+89Qw= 2024-04-16T20:04:41.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0MWgE0ZlrGGpo0KYcdjaSk+89Qw=', 'Content-Length': '0'} 2024-04-16T20:04:41.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e508dc95c0cbff-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.942 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:04:41.942 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:04:41.942 DEBUG:boto:Method: GET 2024-04-16T20:04:41.943 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:04:41.943 DEBUG:boto:Data: 2024-04-16T20:04:41.943 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.943 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.943 DEBUG:boto:Params: {} 2024-04-16T20:04:41.943 DEBUG:boto:Token: None 2024-04-16T20:04:41.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj57 2024-04-16T20:04:41.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0MWgE0ZlrGGpo0KYcdjaSk+89Qw= 2024-04-16T20:04:41.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0MWgE0ZlrGGpo0KYcdjaSk+89Qw=', 'Content-Length': '0'} 2024-04-16T20:04:41.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f88fcf5170acf67-00661ed9d9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.945 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.945 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.946 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.946 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.946 DEBUG:boto:Method: GET 2024-04-16T20:04:41.946 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.946 DEBUG:boto:Data: 2024-04-16T20:04:41.946 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.946 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.946 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.946 DEBUG:boto:Params: {} 2024-04-16T20:04:41.946 DEBUG:boto:Token: None 2024-04-16T20:04:41.946 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047754bffadcd063f-00661ed9d9-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.990 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.991 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:41.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:41.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.991 DEBUG:boto:Method: GET 2024-04-16T20:04:41.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:04:41.991 DEBUG:boto:Data: 2024-04-16T20:04:41.992 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:41.992 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.992 DEBUG:boto:Params: {} 2024-04-16T20:04:41.992 DEBUG:boto:Token: None 2024-04-16T20:04:41.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/ 2024-04-16T20:04:41.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs= 2024-04-16T20:04:41.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yoZksywOiTJL2gnV0Rb4IFr+lLs=', 'Content-Length': '0'} 2024-04-16T20:04:41.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f602ccfa8bd8116-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:41.996 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:41.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:04:41.996 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:41.996 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:41.996 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:41.996 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:41.996 DEBUG:boto:Method: GET 2024-04-16T20:04:41.997 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:41.997 DEBUG:boto:Data: 2024-04-16T20:04:41.997 DEBUG:boto:Headers: {} 2024-04-16T20:04:41.997 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:41.997 DEBUG:boto:Port: 8000 2024-04-16T20:04:41.997 DEBUG:boto:Params: {} 2024-04-16T20:04:41.997 DEBUG:boto:Token: None 2024-04-16T20:04:41.997 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:41 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:41.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QOFSXBpgiHYR3zTH+DptB0wBExw= 2024-04-16T20:04:41.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QOFSXBpgiHYR3zTH+DptB0wBExw=', 'Content-Length': '0'} 2024-04-16T20:04:42.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac76384a58fde81-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.042 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.042 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.043 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.043 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.043 DEBUG:boto:Method: GET 2024-04-16T20:04:42.043 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.043 DEBUG:boto:Data: 2024-04-16T20:04:42.043 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.043 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.043 DEBUG:boto:Params: {} 2024-04-16T20:04:42.043 DEBUG:boto:Token: None 2024-04-16T20:04:42.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE= 2024-04-16T20:04:42.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE=', 'Content-Length': '0'} 2024-04-16T20:04:42.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c85c883b449ee40-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.046 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.046 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.046 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.046 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.046 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.046 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.046 DEBUG:boto:Data: 2024-04-16T20:04:42.047 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.047 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.047 DEBUG:boto:Params: {} 2024-04-16T20:04:42.047 DEBUG:boto:Token: None 2024-04-16T20:04:42.047 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8ydV+4yAnb6E0S3oNjklh/gu0z8= 2024-04-16T20:04:42.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8ydV+4yAnb6E0S3oNjklh/gu0z8=', 'Content-Length': '0'} 2024-04-16T20:04:42.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8233b6de1d0a92-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.049 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.049 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.049 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.050 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.050 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.050 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.050 DEBUG:boto:Data: 2024-04-16T20:04:42.050 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.050 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.050 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.050 DEBUG:boto:Params: {} 2024-04-16T20:04:42.050 DEBUG:boto:Token: None 2024-04-16T20:04:42.050 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8ydV+4yAnb6E0S3oNjklh/gu0z8= 2024-04-16T20:04:42.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8ydV+4yAnb6E0S3oNjklh/gu0z8=', 'Content-Length': '0'} 2024-04-16T20:04:42.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da6f64612494593b-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:04:42.053 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.053 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.053 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.053 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.053 DEBUG:boto:Method: GET 2024-04-16T20:04:42.053 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.053 DEBUG:boto:Data: 2024-04-16T20:04:42.053 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.053 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.053 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.054 DEBUG:boto:Params: {} 2024-04-16T20:04:42.054 DEBUG:boto:Token: None 2024-04-16T20:04:42.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE= 2024-04-16T20:04:42.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE=', 'Content-Length': '0'} 2024-04-16T20:04:42.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28d4b78ca7a34d9-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.098 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.098 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.098 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.099 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.099 DEBUG:boto:Method: GET 2024-04-16T20:04:42.099 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.099 DEBUG:boto:Data: 2024-04-16T20:04:42.099 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.099 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.099 DEBUG:boto:Params: {} 2024-04-16T20:04:42.099 DEBUG:boto:Token: None 2024-04-16T20:04:42.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:04:42.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE= 2024-04-16T20:04:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T5XVLdi2uzZJdJJSmmE06EuRDzE=', 'Content-Length': '0'} 2024-04-16T20:04:42.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051df3dc6e7260b19-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.102 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.102 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.102 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.102 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.102 DEBUG:boto:Method: GET 2024-04-16T20:04:42.102 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.102 DEBUG:boto:Data: 2024-04-16T20:04:42.103 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.103 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.103 DEBUG:boto:Params: {} 2024-04-16T20:04:42.103 DEBUG:boto:Token: None 2024-04-16T20:04:42.103 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005186ee979d1d2df-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.146 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.146 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.147 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.147 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.147 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.147 DEBUG:boto:Method: GET 2024-04-16T20:04:42.147 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.147 DEBUG:boto:Data: 2024-04-16T20:04:42.147 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.147 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.147 DEBUG:boto:Params: {} 2024-04-16T20:04:42.147 DEBUG:boto:Token: None 2024-04-16T20:04:42.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a190bbf9c7c5f8-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.151 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.151 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.151 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.151 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.151 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:04:42.152 DEBUG:boto:Data: 2024-04-16T20:04:42.152 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.152 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.152 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.152 DEBUG:boto:Params: {} 2024-04-16T20:04:42.152 DEBUG:boto:Token: None 2024-04-16T20:04:42.152 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58 2024-04-16T20:04:42.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lw91EgMf55X/KsGiXweDLuwQp3k= 2024-04-16T20:04:42.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lw91EgMf55X/KsGiXweDLuwQp3k=', 'Content-Length': '0'} 2024-04-16T20:04:42.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008438fc8661b3e3e6-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.154 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.154 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.154 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.155 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:04:42.155 DEBUG:boto:Data: 2024-04-16T20:04:42.155 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.155 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.155 DEBUG:boto:Params: {} 2024-04-16T20:04:42.155 DEBUG:boto:Token: None 2024-04-16T20:04:42.155 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58 2024-04-16T20:04:42.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lw91EgMf55X/KsGiXweDLuwQp3k= 2024-04-16T20:04:42.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lw91EgMf55X/KsGiXweDLuwQp3k=', 'Content-Length': '0'} 2024-04-16T20:04:42.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956299a923244a9f-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:04:42.158 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.158 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.158 DEBUG:boto:Method: GET 2024-04-16T20:04:42.158 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:04:42.158 DEBUG:boto:Data: 2024-04-16T20:04:42.158 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.158 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.158 DEBUG:boto:Params: {} 2024-04-16T20:04:42.158 DEBUG:boto:Token: None 2024-04-16T20:04:42.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58 2024-04-16T20:04:42.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7gzGUu2uamzvFmSlUs5bheO1icc= 2024-04-16T20:04:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7gzGUu2uamzvFmSlUs5bheO1icc=', 'Content-Length': '0'} 2024-04-16T20:04:42.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e34fa065741c0923-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.202 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:04:42.202 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:04:42.203 DEBUG:boto:Method: GET 2024-04-16T20:04:42.203 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:04:42.203 DEBUG:boto:Data: 2024-04-16T20:04:42.203 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.203 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.203 DEBUG:boto:Params: {} 2024-04-16T20:04:42.203 DEBUG:boto:Token: None 2024-04-16T20:04:42.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj58 2024-04-16T20:04:42.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7gzGUu2uamzvFmSlUs5bheO1icc= 2024-04-16T20:04:42.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7gzGUu2uamzvFmSlUs5bheO1icc=', 'Content-Length': '0'} 2024-04-16T20:04:42.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8309c817e37ed6-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.206 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.206 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.206 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.206 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.206 DEBUG:boto:Method: GET 2024-04-16T20:04:42.206 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.207 DEBUG:boto:Data: 2024-04-16T20:04:42.207 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.207 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.207 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.207 DEBUG:boto:Params: {} 2024-04-16T20:04:42.207 DEBUG:boto:Token: None 2024-04-16T20:04:42.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac813b6bfecc56e-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.210 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.210 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.210 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.210 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.210 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.210 DEBUG:boto:Method: GET 2024-04-16T20:04:42.210 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:04:42.210 DEBUG:boto:Data: 2024-04-16T20:04:42.211 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.211 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.211 DEBUG:boto:Params: {} 2024-04-16T20:04:42.211 DEBUG:boto:Token: None 2024-04-16T20:04:42.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f1048c6f01a8ca-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.215 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:04:42.215 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.215 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.215 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.215 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.215 DEBUG:boto:Method: GET 2024-04-16T20:04:42.215 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.216 DEBUG:boto:Data: 2024-04-16T20:04:42.216 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.216 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.216 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.216 DEBUG:boto:Params: {} 2024-04-16T20:04:42.216 DEBUG:boto:Token: None 2024-04-16T20:04:42.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM= 2024-04-16T20:04:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM=', 'Content-Length': '0'} 2024-04-16T20:04:42.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7172fc62a28d9d0-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.262 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.262 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.263 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.263 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.263 DEBUG:boto:Method: GET 2024-04-16T20:04:42.263 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.263 DEBUG:boto:Data: 2024-04-16T20:04:42.263 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.263 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.263 DEBUG:boto:Params: {} 2024-04-16T20:04:42.263 DEBUG:boto:Token: None 2024-04-16T20:04:42.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM= 2024-04-16T20:04:42.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM=', 'Content-Length': '0'} 2024-04-16T20:04:42.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad0703829af200d-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.266 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.266 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.266 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.266 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.266 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.266 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.266 DEBUG:boto:Data: 2024-04-16T20:04:42.266 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.267 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.267 DEBUG:boto:Params: {} 2024-04-16T20:04:42.267 DEBUG:boto:Token: None 2024-04-16T20:04:42.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T8sWuJ/kv+4yhlF+Nus7v+R6VOM= 2024-04-16T20:04:42.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T8sWuJ/kv+4yhlF+Nus7v+R6VOM=', 'Content-Length': '0'} 2024-04-16T20:04:42.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bb2dc4dc891af6-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.269 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.270 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.270 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.270 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.270 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.270 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.270 DEBUG:boto:Data: 2024-04-16T20:04:42.270 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.270 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.270 DEBUG:boto:Params: {} 2024-04-16T20:04:42.270 DEBUG:boto:Token: None 2024-04-16T20:04:42.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T8sWuJ/kv+4yhlF+Nus7v+R6VOM= 2024-04-16T20:04:42.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T8sWuJ/kv+4yhlF+Nus7v+R6VOM=', 'Content-Length': '0'} 2024-04-16T20:04:42.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829e5480c704802a-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:04:42.273 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.273 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.273 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.273 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.273 DEBUG:boto:Method: GET 2024-04-16T20:04:42.273 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.273 DEBUG:boto:Data: 2024-04-16T20:04:42.273 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.273 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.274 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.274 DEBUG:boto:Params: {} 2024-04-16T20:04:42.274 DEBUG:boto:Token: None 2024-04-16T20:04:42.274 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM= 2024-04-16T20:04:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM=', 'Content-Length': '0'} 2024-04-16T20:04:42.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0620fd10a743db5-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.318 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.318 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.318 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.319 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.319 DEBUG:boto:Method: GET 2024-04-16T20:04:42.319 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.319 DEBUG:boto:Data: 2024-04-16T20:04:42.319 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.319 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.319 DEBUG:boto:Params: {} 2024-04-16T20:04:42.319 DEBUG:boto:Token: None 2024-04-16T20:04:42.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:04:42.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM= 2024-04-16T20:04:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SeRKJrH4N7M6Rl7JWQudnsWn7nM=', 'Content-Length': '0'} 2024-04-16T20:04:42.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6007867d9c3d488-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.322 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.322 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.322 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.322 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.322 DEBUG:boto:Method: GET 2024-04-16T20:04:42.322 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.322 DEBUG:boto:Data: 2024-04-16T20:04:42.322 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.323 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.323 DEBUG:boto:Params: {} 2024-04-16T20:04:42.323 DEBUG:boto:Token: None 2024-04-16T20:04:42.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ea9aaadaf69ade-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.326 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.326 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.326 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.326 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.327 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.327 DEBUG:boto:Method: GET 2024-04-16T20:04:42.327 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.327 DEBUG:boto:Data: 2024-04-16T20:04:42.327 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.327 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.327 DEBUG:boto:Params: {} 2024-04-16T20:04:42.327 DEBUG:boto:Token: None 2024-04-16T20:04:42.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2c139f32cbbcef-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.331 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.331 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.331 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.331 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.331 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:04:42.331 DEBUG:boto:Data: 2024-04-16T20:04:42.331 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.331 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.331 DEBUG:boto:Params: {} 2024-04-16T20:04:42.332 DEBUG:boto:Token: None 2024-04-16T20:04:42.332 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68 2024-04-16T20:04:42.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Osb3vheAiJirSQkDj499f20o/nw= 2024-04-16T20:04:42.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Osb3vheAiJirSQkDj499f20o/nw=', 'Content-Length': '0'} 2024-04-16T20:04:42.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b6a9bfddac1590-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.334 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.334 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.334 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.334 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:04:42.335 DEBUG:boto:Data: 2024-04-16T20:04:42.335 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.335 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.335 DEBUG:boto:Params: {} 2024-04-16T20:04:42.335 DEBUG:boto:Token: None 2024-04-16T20:04:42.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68 2024-04-16T20:04:42.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Osb3vheAiJirSQkDj499f20o/nw= 2024-04-16T20:04:42.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Osb3vheAiJirSQkDj499f20o/nw=', 'Content-Length': '0'} 2024-04-16T20:04:42.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008337a66ae840ec6b-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:04:42.337 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.337 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.337 DEBUG:boto:Method: GET 2024-04-16T20:04:42.338 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:04:42.338 DEBUG:boto:Data: 2024-04-16T20:04:42.338 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.338 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.338 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.338 DEBUG:boto:Params: {} 2024-04-16T20:04:42.338 DEBUG:boto:Token: None 2024-04-16T20:04:42.338 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68 2024-04-16T20:04:42.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yZ7cmXE4Q6Lbm7bP1VVQEB8TdQM= 2024-04-16T20:04:42.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yZ7cmXE4Q6Lbm7bP1VVQEB8TdQM=', 'Content-Length': '0'} 2024-04-16T20:04:42.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10ff342381ad011-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.382 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:04:42.382 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:04:42.382 DEBUG:boto:Method: GET 2024-04-16T20:04:42.382 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:04:42.383 DEBUG:boto:Data: 2024-04-16T20:04:42.383 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.383 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.383 DEBUG:boto:Params: {} 2024-04-16T20:04:42.383 DEBUG:boto:Token: None 2024-04-16T20:04:42.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj68 2024-04-16T20:04:42.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yZ7cmXE4Q6Lbm7bP1VVQEB8TdQM= 2024-04-16T20:04:42.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yZ7cmXE4Q6Lbm7bP1VVQEB8TdQM=', 'Content-Length': '0'} 2024-04-16T20:04:42.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fc2e0e287827d2-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.385 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.385 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.385 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.385 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.386 DEBUG:boto:Method: GET 2024-04-16T20:04:42.386 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.386 DEBUG:boto:Data: 2024-04-16T20:04:42.386 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.386 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.386 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.386 DEBUG:boto:Params: {} 2024-04-16T20:04:42.386 DEBUG:boto:Token: None 2024-04-16T20:04:42.386 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.386 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56e796d5a149762-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.389 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.389 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.389 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.389 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.390 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.390 DEBUG:boto:Method: GET 2024-04-16T20:04:42.390 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:04:42.390 DEBUG:boto:Data: 2024-04-16T20:04:42.390 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.390 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.390 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.390 DEBUG:boto:Params: {} 2024-04-16T20:04:42.390 DEBUG:boto:Token: None 2024-04-16T20:04:42.390 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.390 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6db8f6335f0f8b-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.394 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:04:42.394 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.395 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.395 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.395 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.395 DEBUG:boto:Method: GET 2024-04-16T20:04:42.395 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.395 DEBUG:boto:Data: 2024-04-16T20:04:42.395 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.395 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.395 DEBUG:boto:Params: {} 2024-04-16T20:04:42.395 DEBUG:boto:Token: None 2024-04-16T20:04:42.396 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY= 2024-04-16T20:04:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY=', 'Content-Length': '0'} 2024-04-16T20:04:42.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ca8aee45a4f0f7-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.438 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.438 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.438 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.439 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.439 DEBUG:boto:Method: GET 2024-04-16T20:04:42.439 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.439 DEBUG:boto:Data: 2024-04-16T20:04:42.439 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.439 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.439 DEBUG:boto:Params: {} 2024-04-16T20:04:42.439 DEBUG:boto:Token: None 2024-04-16T20:04:42.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY= 2024-04-16T20:04:42.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY=', 'Content-Length': '0'} 2024-04-16T20:04:42.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ec12451b9a50d1-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.442 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.442 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.442 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.442 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.442 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.442 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.442 DEBUG:boto:Data: 2024-04-16T20:04:42.442 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.442 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.442 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.443 DEBUG:boto:Params: {} 2024-04-16T20:04:42.443 DEBUG:boto:Token: None 2024-04-16T20:04:42.443 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XR5nNx9pPMI4vwtNm1BuYJswP2g= 2024-04-16T20:04:42.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XR5nNx9pPMI4vwtNm1BuYJswP2g=', 'Content-Length': '0'} 2024-04-16T20:04:42.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438261b1bbaf5da6-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.445 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.445 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.445 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.446 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.446 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.446 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.446 DEBUG:boto:Data: 2024-04-16T20:04:42.446 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.446 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.446 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.446 DEBUG:boto:Params: {} 2024-04-16T20:04:42.446 DEBUG:boto:Token: None 2024-04-16T20:04:42.446 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.446 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XR5nNx9pPMI4vwtNm1BuYJswP2g= 2024-04-16T20:04:42.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XR5nNx9pPMI4vwtNm1BuYJswP2g=', 'Content-Length': '0'} 2024-04-16T20:04:42.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394d20f1082796bc-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:04:42.449 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.449 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.449 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.449 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.449 DEBUG:boto:Method: GET 2024-04-16T20:04:42.449 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.449 DEBUG:boto:Data: 2024-04-16T20:04:42.450 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.450 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.450 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.450 DEBUG:boto:Params: {} 2024-04-16T20:04:42.450 DEBUG:boto:Token: None 2024-04-16T20:04:42.450 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.450 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY= 2024-04-16T20:04:42.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY=', 'Content-Length': '0'} 2024-04-16T20:04:42.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a988ac31f1a2178a-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.494 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.494 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.494 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.495 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.495 DEBUG:boto:Method: GET 2024-04-16T20:04:42.495 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.495 DEBUG:boto:Data: 2024-04-16T20:04:42.495 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.495 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.495 DEBUG:boto:Params: {} 2024-04-16T20:04:42.495 DEBUG:boto:Token: None 2024-04-16T20:04:42.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:04:42.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY= 2024-04-16T20:04:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sbMZCt700jKVQ8ngk3QxWBDfEoY=', 'Content-Length': '0'} 2024-04-16T20:04:42.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000813b694ec215001-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.498 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.498 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.498 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.498 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.498 DEBUG:boto:Method: GET 2024-04-16T20:04:42.498 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.498 DEBUG:boto:Data: 2024-04-16T20:04:42.498 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.498 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.499 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.499 DEBUG:boto:Params: {} 2024-04-16T20:04:42.499 DEBUG:boto:Token: None 2024-04-16T20:04:42.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000849d7e0bed08a564-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.501 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.502 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.502 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.502 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.502 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.502 DEBUG:boto:Method: GET 2024-04-16T20:04:42.502 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.502 DEBUG:boto:Data: 2024-04-16T20:04:42.502 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.503 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.503 DEBUG:boto:Params: {} 2024-04-16T20:04:42.503 DEBUG:boto:Token: None 2024-04-16T20:04:42.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd5bc1cf50b6979d-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.507 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.507 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.507 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.507 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.507 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:04:42.507 DEBUG:boto:Data: 2024-04-16T20:04:42.507 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.507 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.507 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.507 DEBUG:boto:Params: {} 2024-04-16T20:04:42.508 DEBUG:boto:Token: None 2024-04-16T20:04:42.508 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69 2024-04-16T20:04:42.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IMj2PODo0icuXj5qqfS0WaQwcLo= 2024-04-16T20:04:42.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IMj2PODo0icuXj5qqfS0WaQwcLo=', 'Content-Length': '0'} 2024-04-16T20:04:42.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991c5d5cc5fbfc7a-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.510 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.510 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.510 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.510 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:04:42.510 DEBUG:boto:Data: 2024-04-16T20:04:42.510 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.510 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.510 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.510 DEBUG:boto:Params: {} 2024-04-16T20:04:42.511 DEBUG:boto:Token: None 2024-04-16T20:04:42.511 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69 2024-04-16T20:04:42.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IMj2PODo0icuXj5qqfS0WaQwcLo= 2024-04-16T20:04:42.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IMj2PODo0icuXj5qqfS0WaQwcLo=', 'Content-Length': '0'} 2024-04-16T20:04:42.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d7bb93a35b02d6-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:04:42.513 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.513 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.513 DEBUG:boto:Method: GET 2024-04-16T20:04:42.513 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:04:42.513 DEBUG:boto:Data: 2024-04-16T20:04:42.513 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.513 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.513 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.514 DEBUG:boto:Params: {} 2024-04-16T20:04:42.514 DEBUG:boto:Token: None 2024-04-16T20:04:42.514 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69 2024-04-16T20:04:42.514 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rKVmqqD7oZpEZUd3MyIbiB4aHmE= 2024-04-16T20:04:42.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rKVmqqD7oZpEZUd3MyIbiB4aHmE=', 'Content-Length': '0'} 2024-04-16T20:04:42.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005065e6f862a74665-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.558 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:04:42.558 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:04:42.558 DEBUG:boto:Method: GET 2024-04-16T20:04:42.559 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:04:42.559 DEBUG:boto:Data: 2024-04-16T20:04:42.559 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.559 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.559 DEBUG:boto:Params: {} 2024-04-16T20:04:42.559 DEBUG:boto:Token: None 2024-04-16T20:04:42.559 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj69 2024-04-16T20:04:42.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rKVmqqD7oZpEZUd3MyIbiB4aHmE= 2024-04-16T20:04:42.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rKVmqqD7oZpEZUd3MyIbiB4aHmE=', 'Content-Length': '0'} 2024-04-16T20:04:42.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816bd922194b7a1f-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.561 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.562 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.562 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.562 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.562 DEBUG:boto:Method: GET 2024-04-16T20:04:42.562 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.562 DEBUG:boto:Data: 2024-04-16T20:04:42.562 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.562 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.562 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.562 DEBUG:boto:Params: {} 2024-04-16T20:04:42.562 DEBUG:boto:Token: None 2024-04-16T20:04:42.563 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032143b115413be39-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.565 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.566 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.566 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.566 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.566 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.566 DEBUG:boto:Method: GET 2024-04-16T20:04:42.566 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:04:42.566 DEBUG:boto:Data: 2024-04-16T20:04:42.566 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.566 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.566 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.566 DEBUG:boto:Params: {} 2024-04-16T20:04:42.567 DEBUG:boto:Token: None 2024-04-16T20:04:42.567 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c97b2601c20f2016-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.570 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:04:42.570 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.571 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.571 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.571 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.571 DEBUG:boto:Method: GET 2024-04-16T20:04:42.571 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.571 DEBUG:boto:Data: 2024-04-16T20:04:42.571 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.571 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.571 DEBUG:boto:Params: {} 2024-04-16T20:04:42.571 DEBUG:boto:Token: None 2024-04-16T20:04:42.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8= 2024-04-16T20:04:42.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8=', 'Content-Length': '0'} 2024-04-16T20:04:42.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000152b5daa4f77aa26-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.614 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.614 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.615 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.615 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.615 DEBUG:boto:Method: GET 2024-04-16T20:04:42.615 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.615 DEBUG:boto:Data: 2024-04-16T20:04:42.615 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.615 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.615 DEBUG:boto:Params: {} 2024-04-16T20:04:42.615 DEBUG:boto:Token: None 2024-04-16T20:04:42.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8= 2024-04-16T20:04:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8=', 'Content-Length': '0'} 2024-04-16T20:04:42.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6490379cde5db8e-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.617 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.618 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.618 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.618 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.618 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.618 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.618 DEBUG:boto:Data: 2024-04-16T20:04:42.618 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.618 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.618 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.618 DEBUG:boto:Params: {} 2024-04-16T20:04:42.618 DEBUG:boto:Token: None 2024-04-16T20:04:42.619 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.619 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wquxGQKSRCf7WKsdyrUXUK2TIdA= 2024-04-16T20:04:42.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wquxGQKSRCf7WKsdyrUXUK2TIdA=', 'Content-Length': '0'} 2024-04-16T20:04:42.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004446397b2bb7db2e-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.621 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.621 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.621 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.621 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.621 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.621 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.621 DEBUG:boto:Data: 2024-04-16T20:04:42.621 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.621 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.621 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.621 DEBUG:boto:Params: {} 2024-04-16T20:04:42.622 DEBUG:boto:Token: None 2024-04-16T20:04:42.622 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.622 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wquxGQKSRCf7WKsdyrUXUK2TIdA= 2024-04-16T20:04:42.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wquxGQKSRCf7WKsdyrUXUK2TIdA=', 'Content-Length': '0'} 2024-04-16T20:04:42.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b7ada346f2de8d-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:04:42.624 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.624 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.624 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.624 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.624 DEBUG:boto:Method: GET 2024-04-16T20:04:42.624 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.624 DEBUG:boto:Data: 2024-04-16T20:04:42.624 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.624 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.624 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.625 DEBUG:boto:Params: {} 2024-04-16T20:04:42.625 DEBUG:boto:Token: None 2024-04-16T20:04:42.625 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.625 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8= 2024-04-16T20:04:42.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8=', 'Content-Length': '0'} 2024-04-16T20:04:42.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea4ffa92a85b285-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.670 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.670 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.670 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.671 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.671 DEBUG:boto:Method: GET 2024-04-16T20:04:42.671 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.671 DEBUG:boto:Data: 2024-04-16T20:04:42.671 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.671 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.671 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.671 DEBUG:boto:Params: {} 2024-04-16T20:04:42.671 DEBUG:boto:Token: None 2024-04-16T20:04:42.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:04:42.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8= 2024-04-16T20:04:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ihQ10XgSwolFt2ot1l9QtwhfN8=', 'Content-Length': '0'} 2024-04-16T20:04:42.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c1e81cafd6734f-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.674 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.674 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.674 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.674 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.674 DEBUG:boto:Method: GET 2024-04-16T20:04:42.674 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.674 DEBUG:boto:Data: 2024-04-16T20:04:42.674 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.674 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.675 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.675 DEBUG:boto:Params: {} 2024-04-16T20:04:42.675 DEBUG:boto:Token: None 2024-04-16T20:04:42.675 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b152c6169d67f2d-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.677 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.678 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.678 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.678 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.678 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.678 DEBUG:boto:Method: GET 2024-04-16T20:04:42.678 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.678 DEBUG:boto:Data: 2024-04-16T20:04:42.678 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.678 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.678 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.678 DEBUG:boto:Params: {} 2024-04-16T20:04:42.679 DEBUG:boto:Token: None 2024-04-16T20:04:42.679 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f84fed56adf949-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.682 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.683 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.683 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.683 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.683 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:04:42.683 DEBUG:boto:Data: 2024-04-16T20:04:42.683 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.683 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.683 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.683 DEBUG:boto:Params: {} 2024-04-16T20:04:42.683 DEBUG:boto:Token: None 2024-04-16T20:04:42.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76 2024-04-16T20:04:42.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R81ds6iw0/81Yz5+IHK0iK5drrQ= 2024-04-16T20:04:42.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R81ds6iw0/81Yz5+IHK0iK5drrQ=', 'Content-Length': '0'} 2024-04-16T20:04:42.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078aac68ac4f84932-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.686 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.686 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.686 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.686 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:04:42.686 DEBUG:boto:Data: 2024-04-16T20:04:42.686 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.686 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.686 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.686 DEBUG:boto:Params: {} 2024-04-16T20:04:42.686 DEBUG:boto:Token: None 2024-04-16T20:04:42.687 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76 2024-04-16T20:04:42.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R81ds6iw0/81Yz5+IHK0iK5drrQ= 2024-04-16T20:04:42.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R81ds6iw0/81Yz5+IHK0iK5drrQ=', 'Content-Length': '0'} 2024-04-16T20:04:42.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f3c55f2e7a74c6-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:04:42.689 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.689 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.689 DEBUG:boto:Method: GET 2024-04-16T20:04:42.689 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:04:42.689 DEBUG:boto:Data: 2024-04-16T20:04:42.689 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.689 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.689 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.689 DEBUG:boto:Params: {} 2024-04-16T20:04:42.689 DEBUG:boto:Token: None 2024-04-16T20:04:42.690 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76 2024-04-16T20:04:42.690 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e2hkO/Z8V03lKLNULIk37vtLRWM= 2024-04-16T20:04:42.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e2hkO/Z8V03lKLNULIk37vtLRWM=', 'Content-Length': '0'} 2024-04-16T20:04:42.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf4dd02e1e2d761-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.734 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:04:42.734 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:04:42.735 DEBUG:boto:Method: GET 2024-04-16T20:04:42.735 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:04:42.735 DEBUG:boto:Data: 2024-04-16T20:04:42.735 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.735 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.735 DEBUG:boto:Params: {} 2024-04-16T20:04:42.735 DEBUG:boto:Token: None 2024-04-16T20:04:42.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj76 2024-04-16T20:04:42.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e2hkO/Z8V03lKLNULIk37vtLRWM= 2024-04-16T20:04:42.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e2hkO/Z8V03lKLNULIk37vtLRWM=', 'Content-Length': '0'} 2024-04-16T20:04:42.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001207cf4bb555e277-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.737 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.737 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.738 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.738 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.738 DEBUG:boto:Method: GET 2024-04-16T20:04:42.738 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.738 DEBUG:boto:Data: 2024-04-16T20:04:42.738 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.738 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.738 DEBUG:boto:Params: {} 2024-04-16T20:04:42.738 DEBUG:boto:Token: None 2024-04-16T20:04:42.738 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002092c1c00391e635-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.741 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.741 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.741 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.741 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.742 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.742 DEBUG:boto:Method: GET 2024-04-16T20:04:42.742 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:04:42.742 DEBUG:boto:Data: 2024-04-16T20:04:42.742 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.742 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.742 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.742 DEBUG:boto:Params: {} 2024-04-16T20:04:42.742 DEBUG:boto:Token: None 2024-04-16T20:04:42.742 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b09164cb737bb3-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.746 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:04:42.746 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.746 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.747 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.747 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.747 DEBUG:boto:Method: GET 2024-04-16T20:04:42.747 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.747 DEBUG:boto:Data: 2024-04-16T20:04:42.747 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.747 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.747 DEBUG:boto:Params: {} 2024-04-16T20:04:42.747 DEBUG:boto:Token: None 2024-04-16T20:04:42.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY= 2024-04-16T20:04:42.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY=', 'Content-Length': '0'} 2024-04-16T20:04:42.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82926e4cad03de1-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.794 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.795 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.795 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.795 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.795 DEBUG:boto:Method: GET 2024-04-16T20:04:42.795 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.795 DEBUG:boto:Data: 2024-04-16T20:04:42.795 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.795 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.795 DEBUG:boto:Params: {} 2024-04-16T20:04:42.795 DEBUG:boto:Token: None 2024-04-16T20:04:42.796 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY= 2024-04-16T20:04:42.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY=', 'Content-Length': '0'} 2024-04-16T20:04:42.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ed6d18777519b02-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.798 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.798 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.798 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.798 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.798 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.799 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.799 DEBUG:boto:Data: 2024-04-16T20:04:42.799 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.799 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.799 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.799 DEBUG:boto:Params: {} 2024-04-16T20:04:42.799 DEBUG:boto:Token: None 2024-04-16T20:04:42.799 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1fUGfSeecuVVe8VW3Qg5F4KYwk= 2024-04-16T20:04:42.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1fUGfSeecuVVe8VW3Qg5F4KYwk=', 'Content-Length': '0'} 2024-04-16T20:04:42.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002585b32d43567bca-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.801 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.801 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.801 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.802 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.802 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.802 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.802 DEBUG:boto:Data: 2024-04-16T20:04:42.802 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.802 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.802 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.802 DEBUG:boto:Params: {} 2024-04-16T20:04:42.802 DEBUG:boto:Token: None 2024-04-16T20:04:42.802 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.802 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c1fUGfSeecuVVe8VW3Qg5F4KYwk= 2024-04-16T20:04:42.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c1fUGfSeecuVVe8VW3Qg5F4KYwk=', 'Content-Length': '0'} 2024-04-16T20:04:42.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91f3d67ff76c323-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:04:42.805 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.805 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.805 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.805 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.805 DEBUG:boto:Method: GET 2024-04-16T20:04:42.805 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.805 DEBUG:boto:Data: 2024-04-16T20:04:42.805 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.805 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.805 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.805 DEBUG:boto:Params: {} 2024-04-16T20:04:42.805 DEBUG:boto:Token: None 2024-04-16T20:04:42.806 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.806 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY= 2024-04-16T20:04:42.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY=', 'Content-Length': '0'} 2024-04-16T20:04:42.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd19d1b95fe3763-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.850 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.850 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.850 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.850 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.851 DEBUG:boto:Method: GET 2024-04-16T20:04:42.851 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.851 DEBUG:boto:Data: 2024-04-16T20:04:42.851 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.851 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.851 DEBUG:boto:Params: {} 2024-04-16T20:04:42.851 DEBUG:boto:Token: None 2024-04-16T20:04:42.851 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:04:42.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY= 2024-04-16T20:04:42.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RHdWJEh+zEI+fS7tvxh5qFlRmjY=', 'Content-Length': '0'} 2024-04-16T20:04:42.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335a7f7ba765c42b-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.853 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.853 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.854 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.854 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.854 DEBUG:boto:Method: GET 2024-04-16T20:04:42.854 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.854 DEBUG:boto:Data: 2024-04-16T20:04:42.854 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.854 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.854 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.854 DEBUG:boto:Params: {} 2024-04-16T20:04:42.854 DEBUG:boto:Token: None 2024-04-16T20:04:42.854 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e0a183db9854f1-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.857 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.857 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.857 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.857 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.857 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.857 DEBUG:boto:Method: GET 2024-04-16T20:04:42.858 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.858 DEBUG:boto:Data: 2024-04-16T20:04:42.858 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.858 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.858 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.858 DEBUG:boto:Params: {} 2024-04-16T20:04:42.858 DEBUG:boto:Token: None 2024-04-16T20:04:42.858 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019926dead29362cf-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.862 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.862 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.862 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.862 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.862 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:04:42.862 DEBUG:boto:Data: 2024-04-16T20:04:42.862 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.863 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.863 DEBUG:boto:Params: {} 2024-04-16T20:04:42.863 DEBUG:boto:Token: None 2024-04-16T20:04:42.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85 2024-04-16T20:04:42.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WXgSaoymtzDSykoLIgwnrdu51Xw= 2024-04-16T20:04:42.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WXgSaoymtzDSykoLIgwnrdu51Xw=', 'Content-Length': '0'} 2024-04-16T20:04:42.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d6f9d399ff33a1-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.865 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.865 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.865 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.865 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:04:42.865 DEBUG:boto:Data: 2024-04-16T20:04:42.866 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.866 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.866 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.866 DEBUG:boto:Params: {} 2024-04-16T20:04:42.866 DEBUG:boto:Token: None 2024-04-16T20:04:42.866 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85 2024-04-16T20:04:42.866 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WXgSaoymtzDSykoLIgwnrdu51Xw= 2024-04-16T20:04:42.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WXgSaoymtzDSykoLIgwnrdu51Xw=', 'Content-Length': '0'} 2024-04-16T20:04:42.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be5839e42b3eea5-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:04:42.868 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.868 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.868 DEBUG:boto:Method: GET 2024-04-16T20:04:42.869 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:04:42.869 DEBUG:boto:Data: 2024-04-16T20:04:42.869 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.869 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.869 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.869 DEBUG:boto:Params: {} 2024-04-16T20:04:42.869 DEBUG:boto:Token: None 2024-04-16T20:04:42.869 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85 2024-04-16T20:04:42.869 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bc1dek1gPVoXCTzEiDIZx0Ss2C4= 2024-04-16T20:04:42.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bc1dek1gPVoXCTzEiDIZx0Ss2C4=', 'Content-Length': '0'} 2024-04-16T20:04:42.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf15892b41848cf-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.914 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:04:42.914 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:04:42.914 DEBUG:boto:Method: GET 2024-04-16T20:04:42.914 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:04:42.915 DEBUG:boto:Data: 2024-04-16T20:04:42.915 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.915 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.915 DEBUG:boto:Params: {} 2024-04-16T20:04:42.915 DEBUG:boto:Token: None 2024-04-16T20:04:42.915 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj85 2024-04-16T20:04:42.915 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bc1dek1gPVoXCTzEiDIZx0Ss2C4= 2024-04-16T20:04:42.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bc1dek1gPVoXCTzEiDIZx0Ss2C4=', 'Content-Length': '0'} 2024-04-16T20:04:42.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a8a66383e759f4-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.917 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.917 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.918 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.918 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.918 DEBUG:boto:Method: GET 2024-04-16T20:04:42.918 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.918 DEBUG:boto:Data: 2024-04-16T20:04:42.918 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.918 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.918 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.918 DEBUG:boto:Params: {} 2024-04-16T20:04:42.918 DEBUG:boto:Token: None 2024-04-16T20:04:42.918 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d53ce315cbe5522-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.921 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.922 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:42.922 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:42.922 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.922 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.922 DEBUG:boto:Method: GET 2024-04-16T20:04:42.922 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:04:42.922 DEBUG:boto:Data: 2024-04-16T20:04:42.922 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.922 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.922 DEBUG:boto:Params: {} 2024-04-16T20:04:42.923 DEBUG:boto:Token: None 2024-04-16T20:04:42.923 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/ 2024-04-16T20:04:42.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A= 2024-04-16T20:04:42.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8HVDP8Kh3QDALcU+mBdkU/M165A=', 'Content-Length': '0'} 2024-04-16T20:04:42.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dcefac78828f464-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.926 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:42.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:04:42.926 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:42.927 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:42.927 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.927 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.927 DEBUG:boto:Method: GET 2024-04-16T20:04:42.927 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.927 DEBUG:boto:Data: 2024-04-16T20:04:42.927 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.927 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.927 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.927 DEBUG:boto:Params: {} 2024-04-16T20:04:42.927 DEBUG:boto:Token: None 2024-04-16T20:04:42.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els= 2024-04-16T20:04:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els=', 'Content-Length': '0'} 2024-04-16T20:04:42.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030212166ddac1170-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.970 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:42.971 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:42.971 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.971 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.971 DEBUG:boto:Method: GET 2024-04-16T20:04:42.971 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.971 DEBUG:boto:Data: 2024-04-16T20:04:42.971 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.971 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.971 DEBUG:boto:Params: {} 2024-04-16T20:04:42.971 DEBUG:boto:Token: None 2024-04-16T20:04:42.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els= 2024-04-16T20:04:42.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els=', 'Content-Length': '0'} 2024-04-16T20:04:42.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa8ebac776ebdab-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.974 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:42.974 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:42.974 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.974 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.974 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.974 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.974 DEBUG:boto:Data: 2024-04-16T20:04:42.975 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.975 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.975 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.975 DEBUG:boto:Params: {} 2024-04-16T20:04:42.975 DEBUG:boto:Token: None 2024-04-16T20:04:42.975 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N0rpCtM7kz74VPkBgn7tuC3dCic= 2024-04-16T20:04:42.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N0rpCtM7kz74VPkBgn7tuC3dCic=', 'Content-Length': '0'} 2024-04-16T20:04:42.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001caa0215c8363b69-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.978 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:42.978 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:42.978 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.978 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.978 DEBUG:boto:Method: HEAD 2024-04-16T20:04:42.978 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.978 DEBUG:boto:Data: 2024-04-16T20:04:42.978 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.978 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:42.978 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.979 DEBUG:boto:Params: {} 2024-04-16T20:04:42.979 DEBUG:boto:Token: None 2024-04-16T20:04:42.979 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N0rpCtM7kz74VPkBgn7tuC3dCic= 2024-04-16T20:04:42.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N0rpCtM7kz74VPkBgn7tuC3dCic=', 'Content-Length': '0'} 2024-04-16T20:04:42.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde6002e9697f128-00661ed9da-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:42.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:04:42.981 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:42.981 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:42.982 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.982 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.982 DEBUG:boto:Method: GET 2024-04-16T20:04:42.982 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.982 DEBUG:boto:Data: 2024-04-16T20:04:42.982 DEBUG:boto:Headers: {} 2024-04-16T20:04:42.982 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:42.982 DEBUG:boto:Port: 8000 2024-04-16T20:04:42.982 DEBUG:boto:Params: {} 2024-04-16T20:04:42.982 DEBUG:boto:Token: None 2024-04-16T20:04:42.982 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:42 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:42.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els= 2024-04-16T20:04:42.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iQSE1JOHK1AEC0XmwkRP3Ce3Els=', 'Content-Length': '0'} 2024-04-16T20:04:42.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007729ee29e700189d-00661ed9da-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.026 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:43.027 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:43.027 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:43.027 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:04:43.027 DEBUG:boto:Method: GET 2024-04-16T20:04:43.027 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:43.027 DEBUG:boto:Data: 2024-04-16T20:04:43.027 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.027 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.027 DEBUG:boto:Params: {} 2024-04-16T20:04:43.027 DEBUG:boto:Token: None 2024-04-16T20:04:43.028 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:04:43.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w0jv+izc9O43ImpJfI/DUILlPjA= 2024-04-16T20:04:43.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w0jv+izc9O43ImpJfI/DUILlPjA=', 'Content-Length': '0'} 2024-04-16T20:04:43.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d54ea55ae75cc897-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.030 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.030 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.030 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.030 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.030 DEBUG:boto:Method: GET 2024-04-16T20:04:43.030 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.031 DEBUG:boto:Data: 2024-04-16T20:04:43.031 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.031 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.031 DEBUG:boto:Params: {} 2024-04-16T20:04:43.031 DEBUG:boto:Token: None 2024-04-16T20:04:43.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b664971a3269ca6-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.034 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.034 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.034 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.034 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.035 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.035 DEBUG:boto:Method: GET 2024-04-16T20:04:43.035 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.035 DEBUG:boto:Data: 2024-04-16T20:04:43.035 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.035 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.035 DEBUG:boto:Params: {} 2024-04-16T20:04:43.035 DEBUG:boto:Token: None 2024-04-16T20:04:43.035 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1350dddbf7213aa-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.039 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.039 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:43.039 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:43.039 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.039 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:04:43.040 DEBUG:boto:Data: 2024-04-16T20:04:43.040 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.040 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.040 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.040 DEBUG:boto:Params: {} 2024-04-16T20:04:43.040 DEBUG:boto:Token: None 2024-04-16T20:04:43.040 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj87 2024-04-16T20:04:43.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TJOdbd9M015rtvMDOQWjOPJiy0A= 2024-04-16T20:04:43.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TJOdbd9M015rtvMDOQWjOPJiy0A=', 'Content-Length': '0'} 2024-04-16T20:04:43.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac3f9c1c57744d7-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.043 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:43.043 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:43.043 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.043 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:04:43.043 DEBUG:boto:Data: 2024-04-16T20:04:43.043 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.043 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.043 DEBUG:boto:Params: {} 2024-04-16T20:04:43.043 DEBUG:boto:Token: None 2024-04-16T20:04:43.043 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj87 2024-04-16T20:04:43.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TJOdbd9M015rtvMDOQWjOPJiy0A= 2024-04-16T20:04:43.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TJOdbd9M015rtvMDOQWjOPJiy0A=', 'Content-Length': '0'} 2024-04-16T20:04:43.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35933c7bd4bb917-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:04:43.046 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:43.046 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:43.046 DEBUG:boto:Method: GET 2024-04-16T20:04:43.046 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:04:43.047 DEBUG:boto:Data: 2024-04-16T20:04:43.047 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.047 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.047 DEBUG:boto:Params: {} 2024-04-16T20:04:43.047 DEBUG:boto:Token: None 2024-04-16T20:04:43.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj87 2024-04-16T20:04:43.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4nHPSFmVwmXyn97K8FyrToI6C6k= 2024-04-16T20:04:43.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4nHPSFmVwmXyn97K8FyrToI6C6k=', 'Content-Length': '0'} 2024-04-16T20:04:43.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019537e4e0bc4c82c-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.090 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:04:43.090 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:04:43.091 DEBUG:boto:Method: GET 2024-04-16T20:04:43.091 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:04:43.091 DEBUG:boto:Data: 2024-04-16T20:04:43.091 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.091 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.091 DEBUG:boto:Params: {} 2024-04-16T20:04:43.091 DEBUG:boto:Token: None 2024-04-16T20:04:43.091 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj87 2024-04-16T20:04:43.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4nHPSFmVwmXyn97K8FyrToI6C6k= 2024-04-16T20:04:43.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4nHPSFmVwmXyn97K8FyrToI6C6k=', 'Content-Length': '0'} 2024-04-16T20:04:43.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fc25865616b492-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.094 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.094 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.094 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.094 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.094 DEBUG:boto:Method: GET 2024-04-16T20:04:43.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.095 DEBUG:boto:Data: 2024-04-16T20:04:43.095 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.095 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.095 DEBUG:boto:Params: {} 2024-04-16T20:04:43.095 DEBUG:boto:Token: None 2024-04-16T20:04:43.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359cf7acd21dd94c-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.098 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.098 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.098 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.098 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.098 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.099 DEBUG:boto:Method: GET 2024-04-16T20:04:43.099 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:04:43.099 DEBUG:boto:Data: 2024-04-16T20:04:43.099 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.099 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.099 DEBUG:boto:Params: {} 2024-04-16T20:04:43.099 DEBUG:boto:Token: None 2024-04-16T20:04:43.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804972672ac815e7-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.103 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:04:43.103 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.103 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.103 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.103 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.103 DEBUG:boto:Method: GET 2024-04-16T20:04:43.103 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.104 DEBUG:boto:Data: 2024-04-16T20:04:43.104 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.104 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.104 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.104 DEBUG:boto:Params: {} 2024-04-16T20:04:43.104 DEBUG:boto:Token: None 2024-04-16T20:04:43.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8= 2024-04-16T20:04:43.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8=', 'Content-Length': '0'} 2024-04-16T20:04:43.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005a5761b4d443c3e-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.146 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.146 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.146 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.147 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.147 DEBUG:boto:Method: GET 2024-04-16T20:04:43.147 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.147 DEBUG:boto:Data: 2024-04-16T20:04:43.147 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.147 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.147 DEBUG:boto:Params: {} 2024-04-16T20:04:43.147 DEBUG:boto:Token: None 2024-04-16T20:04:43.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8= 2024-04-16T20:04:43.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8=', 'Content-Length': '0'} 2024-04-16T20:04:43.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4546ca5f42e457-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.149 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.150 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.150 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.150 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.150 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.150 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.150 DEBUG:boto:Data: 2024-04-16T20:04:43.150 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.150 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.150 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.150 DEBUG:boto:Params: {} 2024-04-16T20:04:43.151 DEBUG:boto:Token: None 2024-04-16T20:04:43.151 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5Erw+M2PFPIjtssNqWZBND6GMmE= 2024-04-16T20:04:43.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5Erw+M2PFPIjtssNqWZBND6GMmE=', 'Content-Length': '0'} 2024-04-16T20:04:43.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007727902bb038b82-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.153 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.153 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.153 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.153 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.153 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.153 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.154 DEBUG:boto:Data: 2024-04-16T20:04:43.154 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.154 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.154 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.154 DEBUG:boto:Params: {} 2024-04-16T20:04:43.154 DEBUG:boto:Token: None 2024-04-16T20:04:43.154 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.154 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5Erw+M2PFPIjtssNqWZBND6GMmE= 2024-04-16T20:04:43.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5Erw+M2PFPIjtssNqWZBND6GMmE=', 'Content-Length': '0'} 2024-04-16T20:04:43.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3caad708f5a6fcf-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:04:43.156 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.156 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.156 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.156 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.157 DEBUG:boto:Method: GET 2024-04-16T20:04:43.157 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.157 DEBUG:boto:Data: 2024-04-16T20:04:43.157 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.157 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.157 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.157 DEBUG:boto:Params: {} 2024-04-16T20:04:43.157 DEBUG:boto:Token: None 2024-04-16T20:04:43.157 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8= 2024-04-16T20:04:43.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8=', 'Content-Length': '0'} 2024-04-16T20:04:43.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0005ca6efb1c1d-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.202 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.202 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.203 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.203 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.203 DEBUG:boto:Method: GET 2024-04-16T20:04:43.203 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.203 DEBUG:boto:Data: 2024-04-16T20:04:43.203 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.203 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.203 DEBUG:boto:Params: {} 2024-04-16T20:04:43.203 DEBUG:boto:Token: None 2024-04-16T20:04:43.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:04:43.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8= 2024-04-16T20:04:43.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2eFjK2i26Xc4rg+yNvlXhAzo+q8=', 'Content-Length': '0'} 2024-04-16T20:04:43.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475a853af25903d7-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.206 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.206 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.206 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.206 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.206 DEBUG:boto:Method: GET 2024-04-16T20:04:43.206 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.206 DEBUG:boto:Data: 2024-04-16T20:04:43.206 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.206 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.206 DEBUG:boto:Params: {} 2024-04-16T20:04:43.206 DEBUG:boto:Token: None 2024-04-16T20:04:43.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a4c783f5bbf5fe-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.210 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.210 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.210 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.210 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.210 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.210 DEBUG:boto:Method: GET 2024-04-16T20:04:43.210 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.210 DEBUG:boto:Data: 2024-04-16T20:04:43.210 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.211 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.211 DEBUG:boto:Params: {} 2024-04-16T20:04:43.211 DEBUG:boto:Token: None 2024-04-16T20:04:43.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ea2d9799090aa9-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.214 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.215 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.215 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.215 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.215 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:04:43.215 DEBUG:boto:Data: 2024-04-16T20:04:43.215 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.215 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.215 DEBUG:boto:Params: {} 2024-04-16T20:04:43.215 DEBUG:boto:Token: None 2024-04-16T20:04:43.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92 2024-04-16T20:04:43.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yJIExsrvI8f+8wu62AcG+3RKBqk= 2024-04-16T20:04:43.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yJIExsrvI8f+8wu62AcG+3RKBqk=', 'Content-Length': '0'} 2024-04-16T20:04:43.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001053a271044d1a0f-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.218 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.218 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.218 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.218 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:04:43.218 DEBUG:boto:Data: 2024-04-16T20:04:43.218 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.218 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.218 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.218 DEBUG:boto:Params: {} 2024-04-16T20:04:43.218 DEBUG:boto:Token: None 2024-04-16T20:04:43.219 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92 2024-04-16T20:04:43.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yJIExsrvI8f+8wu62AcG+3RKBqk= 2024-04-16T20:04:43.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yJIExsrvI8f+8wu62AcG+3RKBqk=', 'Content-Length': '0'} 2024-04-16T20:04:43.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9d551bbe09b522-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:04:43.221 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.221 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.221 DEBUG:boto:Method: GET 2024-04-16T20:04:43.221 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:04:43.221 DEBUG:boto:Data: 2024-04-16T20:04:43.221 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.221 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.222 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.222 DEBUG:boto:Params: {} 2024-04-16T20:04:43.222 DEBUG:boto:Token: None 2024-04-16T20:04:43.222 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92 2024-04-16T20:04:43.222 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZbD7r3/f0pGbI3V2oRyt0QM/Kj4= 2024-04-16T20:04:43.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZbD7r3/f0pGbI3V2oRyt0QM/Kj4=', 'Content-Length': '0'} 2024-04-16T20:04:43.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000524f0e1e3f14c9b-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.266 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:04:43.266 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:04:43.266 DEBUG:boto:Method: GET 2024-04-16T20:04:43.266 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:04:43.267 DEBUG:boto:Data: 2024-04-16T20:04:43.267 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.267 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.267 DEBUG:boto:Params: {} 2024-04-16T20:04:43.267 DEBUG:boto:Token: None 2024-04-16T20:04:43.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj92 2024-04-16T20:04:43.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZbD7r3/f0pGbI3V2oRyt0QM/Kj4= 2024-04-16T20:04:43.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZbD7r3/f0pGbI3V2oRyt0QM/Kj4=', 'Content-Length': '0'} 2024-04-16T20:04:43.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0d02bc9a2af609-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.269 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.269 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.269 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.270 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.270 DEBUG:boto:Method: GET 2024-04-16T20:04:43.270 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.270 DEBUG:boto:Data: 2024-04-16T20:04:43.270 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.270 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.270 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.270 DEBUG:boto:Params: {} 2024-04-16T20:04:43.270 DEBUG:boto:Token: None 2024-04-16T20:04:43.270 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419c67918036ae74-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.273 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.273 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.274 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.274 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.274 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.274 DEBUG:boto:Method: GET 2024-04-16T20:04:43.274 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:04:43.274 DEBUG:boto:Data: 2024-04-16T20:04:43.274 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.274 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.274 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.274 DEBUG:boto:Params: {} 2024-04-16T20:04:43.274 DEBUG:boto:Token: None 2024-04-16T20:04:43.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5240cb26f0f727-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.278 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:04:43.278 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.278 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.278 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.279 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.279 DEBUG:boto:Method: GET 2024-04-16T20:04:43.279 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.279 DEBUG:boto:Data: 2024-04-16T20:04:43.279 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.279 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.279 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.279 DEBUG:boto:Params: {} 2024-04-16T20:04:43.279 DEBUG:boto:Token: None 2024-04-16T20:04:43.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU= 2024-04-16T20:04:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU=', 'Content-Length': '0'} 2024-04-16T20:04:43.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304ac969baafe9d3-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.322 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.322 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.322 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.323 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.323 DEBUG:boto:Method: GET 2024-04-16T20:04:43.323 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.323 DEBUG:boto:Data: 2024-04-16T20:04:43.323 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.323 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.323 DEBUG:boto:Params: {} 2024-04-16T20:04:43.323 DEBUG:boto:Token: None 2024-04-16T20:04:43.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU= 2024-04-16T20:04:43.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU=', 'Content-Length': '0'} 2024-04-16T20:04:43.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2f386f058987ce-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.326 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.326 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.326 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.326 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.326 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.326 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.326 DEBUG:boto:Data: 2024-04-16T20:04:43.327 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.327 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.327 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.327 DEBUG:boto:Params: {} 2024-04-16T20:04:43.327 DEBUG:boto:Token: None 2024-04-16T20:04:43.327 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7to9nATvgIHaA2agqvZ5/R7kKms= 2024-04-16T20:04:43.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7to9nATvgIHaA2agqvZ5/R7kKms=', 'Content-Length': '0'} 2024-04-16T20:04:43.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ace31ac631322f-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.329 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.329 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.330 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.330 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.330 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.330 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.330 DEBUG:boto:Data: 2024-04-16T20:04:43.330 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.330 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.330 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.330 DEBUG:boto:Params: {} 2024-04-16T20:04:43.330 DEBUG:boto:Token: None 2024-04-16T20:04:43.330 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7to9nATvgIHaA2agqvZ5/R7kKms= 2024-04-16T20:04:43.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7to9nATvgIHaA2agqvZ5/R7kKms=', 'Content-Length': '0'} 2024-04-16T20:04:43.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c5e6c93c519eee-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:04:43.333 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.333 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.333 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.333 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.333 DEBUG:boto:Method: GET 2024-04-16T20:04:43.333 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.333 DEBUG:boto:Data: 2024-04-16T20:04:43.333 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.333 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.333 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.334 DEBUG:boto:Params: {} 2024-04-16T20:04:43.334 DEBUG:boto:Token: None 2024-04-16T20:04:43.334 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU= 2024-04-16T20:04:43.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU=', 'Content-Length': '0'} 2024-04-16T20:04:43.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089163e8a02710a0c-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.378 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.378 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.378 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.378 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.379 DEBUG:boto:Method: GET 2024-04-16T20:04:43.379 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.379 DEBUG:boto:Data: 2024-04-16T20:04:43.379 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.379 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.379 DEBUG:boto:Params: {} 2024-04-16T20:04:43.379 DEBUG:boto:Token: None 2024-04-16T20:04:43.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:04:43.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU= 2024-04-16T20:04:43.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B6ne5HKg0yU9bMYvSw8/GGMebgU=', 'Content-Length': '0'} 2024-04-16T20:04:43.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f335109474fb816-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.381 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.382 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.382 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.382 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.382 DEBUG:boto:Method: GET 2024-04-16T20:04:43.382 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.382 DEBUG:boto:Data: 2024-04-16T20:04:43.382 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.382 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.382 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.382 DEBUG:boto:Params: {} 2024-04-16T20:04:43.382 DEBUG:boto:Token: None 2024-04-16T20:04:43.382 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a2111b519a0726c-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.385 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.385 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.385 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.385 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.385 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.385 DEBUG:boto:Method: GET 2024-04-16T20:04:43.386 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.386 DEBUG:boto:Data: 2024-04-16T20:04:43.386 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.386 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.386 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.386 DEBUG:boto:Params: {} 2024-04-16T20:04:43.386 DEBUG:boto:Token: None 2024-04-16T20:04:43.386 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.386 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a6255407249463-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.390 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.390 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.390 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.390 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.390 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:04:43.390 DEBUG:boto:Data: 2024-04-16T20:04:43.390 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.390 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.391 DEBUG:boto:Params: {} 2024-04-16T20:04:43.391 DEBUG:boto:Token: None 2024-04-16T20:04:43.391 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95 2024-04-16T20:04:43.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuifaOTtALJUa8+/OTfNxpTy/+M= 2024-04-16T20:04:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuifaOTtALJUa8+/OTfNxpTy/+M=', 'Content-Length': '0'} 2024-04-16T20:04:43.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e44f6974a236a3-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.393 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.393 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.393 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.393 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:04:43.393 DEBUG:boto:Data: 2024-04-16T20:04:43.394 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.394 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.394 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.394 DEBUG:boto:Params: {} 2024-04-16T20:04:43.394 DEBUG:boto:Token: None 2024-04-16T20:04:43.394 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95 2024-04-16T20:04:43.394 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AuifaOTtALJUa8+/OTfNxpTy/+M= 2024-04-16T20:04:43.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AuifaOTtALJUa8+/OTfNxpTy/+M=', 'Content-Length': '0'} 2024-04-16T20:04:43.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a45c3cd4b74b62-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:04:43.396 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.396 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.396 DEBUG:boto:Method: GET 2024-04-16T20:04:43.396 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:04:43.396 DEBUG:boto:Data: 2024-04-16T20:04:43.397 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.397 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.397 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.397 DEBUG:boto:Params: {} 2024-04-16T20:04:43.397 DEBUG:boto:Token: None 2024-04-16T20:04:43.397 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95 2024-04-16T20:04:43.397 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Lf5JyNGVNp8v2Lr6JQd5fJepIq8= 2024-04-16T20:04:43.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Lf5JyNGVNp8v2Lr6JQd5fJepIq8=', 'Content-Length': '0'} 2024-04-16T20:04:43.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf8b67a4547ebe0-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.442 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:04:43.442 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:04:43.442 DEBUG:boto:Method: GET 2024-04-16T20:04:43.442 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:04:43.443 DEBUG:boto:Data: 2024-04-16T20:04:43.443 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.443 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.443 DEBUG:boto:Params: {} 2024-04-16T20:04:43.443 DEBUG:boto:Token: None 2024-04-16T20:04:43.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj95 2024-04-16T20:04:43.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Lf5JyNGVNp8v2Lr6JQd5fJepIq8= 2024-04-16T20:04:43.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Lf5JyNGVNp8v2Lr6JQd5fJepIq8=', 'Content-Length': '0'} 2024-04-16T20:04:43.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000927fe861cce5b424-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.445 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.446 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.446 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.446 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.446 DEBUG:boto:Method: GET 2024-04-16T20:04:43.446 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.446 DEBUG:boto:Data: 2024-04-16T20:04:43.446 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.446 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:04:43.446 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.446 DEBUG:boto:Params: {} 2024-04-16T20:04:43.447 DEBUG:boto:Token: None 2024-04-16T20:04:43.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb50804ef6f3f8a-00661ed9db-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.449 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.449 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.450 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.450 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.450 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.450 DEBUG:boto:Method: GET 2024-04-16T20:04:43.450 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:04:43.450 DEBUG:boto:Data: 2024-04-16T20:04:43.450 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.450 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.450 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.450 DEBUG:boto:Params: {} 2024-04-16T20:04:43.450 DEBUG:boto:Token: None 2024-04-16T20:04:43.451 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.451 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ= 2024-04-16T20:04:43.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ih48x89ielcOeSCTnpQOu3BuwgQ=', 'Content-Length': '0'} 2024-04-16T20:04:43.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e32e9c6d8588fb51-00661ed9db-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.454 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:04:43.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:04:43.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:04:43.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:04:43.455 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:04:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:04:43.514 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:04:43.593 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:04:43.593 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T20:04:43.606 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 25 2024-04-16T20:04:43.683 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.683 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.683 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.683 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:43.683 DEBUG:boto:Data: 2024-04-16T20:04:43.683 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.684 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.684 DEBUG:boto:Params: {} 2024-04-16T20:04:43.684 DEBUG:boto:Token: None 2024-04-16T20:04:43.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w= 2024-04-16T20:04:43.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w=', 'Content-Length': '0'} 2024-04-16T20:04:43.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042e0b0f3cf810c14-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.687 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:04:43.687 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:04:43.687 DEBUG:boto:Method: PUT 2024-04-16T20:04:43.688 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:04:43.688 DEBUG:boto:Data: 2024-04-16T20:04:43.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:04:43.688 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.688 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.688 DEBUG:boto:Params: {} 2024-04-16T20:04:43.688 DEBUG:boto:Token: None 2024-04-16T20:04:43.688 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj31 2024-04-16T20:04:43.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fj0I+UMVpbTdydCBd8v3dxUBLYo= 2024-04-16T20:04:43.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fj0I+UMVpbTdydCBd8v3dxUBLYo='} 2024-04-16T20:04:43.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ccecdcda7194d57-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.693 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.693 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.694 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.694 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:43.694 DEBUG:boto:Data: 2024-04-16T20:04:43.694 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.694 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.694 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.694 DEBUG:boto:Params: {} 2024-04-16T20:04:43.694 DEBUG:boto:Token: None 2024-04-16T20:04:43.694 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.694 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w= 2024-04-16T20:04:43.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w=', 'Content-Length': '0'} 2024-04-16T20:04:43.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000621c16213b72f56a-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.697 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:04:43.697 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:04:43.697 DEBUG:boto:Method: PUT 2024-04-16T20:04:43.697 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:04:43.697 DEBUG:boto:Data: 2024-04-16T20:04:43.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:04:43.698 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.698 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.698 DEBUG:boto:Params: {} 2024-04-16T20:04:43.698 DEBUG:boto:Token: None 2024-04-16T20:04:43.698 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj62 2024-04-16T20:04:43.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Cv8GYUWrWWHPHv2CgwRmc9qjqZA= 2024-04-16T20:04:43.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Cv8GYUWrWWHPHv2CgwRmc9qjqZA='} 2024-04-16T20:04:43.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f7bbd00eb7e946b-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.703 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.703 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.703 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.703 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:43.704 DEBUG:boto:Data: 2024-04-16T20:04:43.704 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.704 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.704 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.704 DEBUG:boto:Params: {} 2024-04-16T20:04:43.704 DEBUG:boto:Token: None 2024-04-16T20:04:43.704 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w= 2024-04-16T20:04:43.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w=', 'Content-Length': '0'} 2024-04-16T20:04:43.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad6293f51037e2db-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.707 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:04:43.707 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:04:43.707 DEBUG:boto:Method: PUT 2024-04-16T20:04:43.708 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:04:43.708 DEBUG:boto:Data: 2024-04-16T20:04:43.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:04:43.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.708 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.708 DEBUG:boto:Params: {} 2024-04-16T20:04:43.708 DEBUG:boto:Token: None 2024-04-16T20:04:43.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj93 2024-04-16T20:04:43.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RGyocY6po7NSpiATUAb+oaHQmvQ= 2024-04-16T20:04:43.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RGyocY6po7NSpiATUAb+oaHQmvQ='} 2024-04-16T20:04:43.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d317858d8e276c6a-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.750 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.751 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.751 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.751 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:43.751 DEBUG:boto:Data: 2024-04-16T20:04:43.751 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.751 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.751 DEBUG:boto:Params: {} 2024-04-16T20:04:43.751 DEBUG:boto:Token: None 2024-04-16T20:04:43.751 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w= 2024-04-16T20:04:43.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w=', 'Content-Length': '0'} 2024-04-16T20:04:43.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b2dfcc4e3a64aa9-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.754 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:04:43.755 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:04:43.755 DEBUG:boto:Method: PUT 2024-04-16T20:04:43.755 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:04:43.755 DEBUG:boto:Data: 2024-04-16T20:04:43.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:04:43.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.755 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.755 DEBUG:boto:Params: {} 2024-04-16T20:04:43.755 DEBUG:boto:Token: None 2024-04-16T20:04:43.755 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj124 2024-04-16T20:04:43.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3R3d4Yk3IksfUIxKETVU1TyhINA= 2024-04-16T20:04:43.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3R3d4Yk3IksfUIxKETVU1TyhINA='} 2024-04-16T20:04:43.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000786f39ad3aac386e-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.761 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:04:43.761 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:04:43.761 DEBUG:boto:Method: HEAD 2024-04-16T20:04:43.761 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:04:43.761 DEBUG:boto:Data: 2024-04-16T20:04:43.761 DEBUG:boto:Headers: {} 2024-04-16T20:04:43.761 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.762 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.762 DEBUG:boto:Params: {} 2024-04-16T20:04:43.762 DEBUG:boto:Token: None 2024-04-16T20:04:43.762 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/ 2024-04-16T20:04:43.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w= 2024-04-16T20:04:43.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:80SjqjKmJK+JsS9axAV++e1JY7w=', 'Content-Length': '0'} 2024-04-16T20:04:43.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b504d2f7cb5caa8e-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.765 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:04:43.765 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:04:43.765 DEBUG:boto:Method: PUT 2024-04-16T20:04:43.765 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:04:43.765 DEBUG:boto:Data: 2024-04-16T20:04:43.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:04:43.765 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:04:43.765 DEBUG:boto:Port: 8000 2024-04-16T20:04:43.765 DEBUG:boto:Params: {} 2024-04-16T20:04:43.766 DEBUG:boto:Token: None 2024-04-16T20:04:43.766 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:04:43 GMT /yrjefh-52/obj155 2024-04-16T20:04:43.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IzgzhBML+1GousiyKTPn54RjZ70= 2024-04-16T20:04:43.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:04:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IzgzhBML+1GousiyKTPn54RjZ70='} 2024-04-16T20:04:43.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ead6f30e608670b6-00661ed9db-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:04:43.771 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:04:43.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:04:43.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:04:43.771 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:04:43.835 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:04:43.835 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:04:43.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:43.905+0000 7f3509da2800 1 waiting to reach latest gen 5, on 4.. 2024-04-16T20:04:48.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:48.907+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:48.901 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:04:48.902 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:04:53.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:53.909+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:53.903 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:04:53.904 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:04:58.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:04:58.911+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:04:58.905 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:04:58.905 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:03.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:03.913+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:03.922 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:03.923 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:08.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:08.915+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:08.909 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:08.909 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:13.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:13.916+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:13.911 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:13.911 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:18.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:18.919+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:18.914 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:18.914 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:23.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:23.922+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:23.916 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:23.916 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:28.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:28.924+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:28.917 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:05:28.917 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:33.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:33.925+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:33.919 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , 00000000001.66.6, , , , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , ] 2024-04-16T20:05:33.919 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:38.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:38.927+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:38.921 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , 00000000001.66.6, , , , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , ] 2024-04-16T20:05:38.921 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:43.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:43.929+0000 7f3509da2800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:43.923 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , 00000000001.66.6, , , , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , ] 2024-04-16T20:05:43.923 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:48.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:48.931+0000 7f3509da2800 1 bucket sync caught up with source: 2024-04-16T20:05:48.925 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:48.926 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , 00000000002.76.6, , 00000000001.66.6, , 00000000001.61.6, , , , , , , , , , 00000000001.68.6, , , , , , , , , , , , , , 00000000001.82.6] 2024-04-16T20:05:48.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:05:48.931+0000 7f3509da2800 0 bucket checkpoint complete 2024-04-16T20:05:48.950 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:05:48.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:05:48.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:05:48.951 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:05:49.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:05:49.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:05:49.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:05:49.084+0000 7fb501526800 1 waiting for incremental sync to catch up: 2024-04-16T20:05:49.077 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.163.6, 00000000002.172.6, 00000000002.108.6, 00000000003.281.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000003.182.6, 00000000005.215.6, 00000000004.282.6] 2024-04-16T20:05:49.077 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000004.288.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000003.182.6, 00000000006.247.6, 00000000004.282.6] 2024-04-16T20:05:54.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:05:54.085+0000 7fb501526800 1 bucket sync caught up with source: 2024-04-16T20:05:54.079 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000004.288.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000003.182.6, 00000000006.247.6, 00000000004.282.6] 2024-04-16T20:05:54.079 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000004.288.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000003.182.6, 00000000006.247.6, 00000000004.282.6] 2024-04-16T20:05:54.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:05:54.085+0000 7fb501526800 0 bucket checkpoint complete 2024-04-16T20:05:54.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:05:54.108 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.108 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.108 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.108 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:05:54.108 DEBUG:boto:Data: 2024-04-16T20:05:54.109 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.109 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.109 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.109 DEBUG:boto:Params: {} 2024-04-16T20:05:54.110 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:05:54.110 DEBUG:boto:Token: None 2024-04-16T20:05:54.110 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.110 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DUFMAZ8Av/ONc2ozPW/yklGBa9A= 2024-04-16T20:05:54.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DUFMAZ8Av/ONc2ozPW/yklGBa9A=', 'Content-Length': '0'} 2024-04-16T20:05:54.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6e14348428ea74c-00661eda22-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.119 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.119 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.119 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.119 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:05:54.119 DEBUG:boto:Data: 2024-04-16T20:05:54.119 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.120 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.120 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.120 DEBUG:boto:Params: {} 2024-04-16T20:05:54.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:05:54.120 DEBUG:boto:Token: None 2024-04-16T20:05:54.120 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DUFMAZ8Av/ONc2ozPW/yklGBa9A= 2024-04-16T20:05:54.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DUFMAZ8Av/ONc2ozPW/yklGBa9A=', 'Content-Length': '0'} 2024-04-16T20:05:54.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de287290ea6b565-00661eda22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:05:54.128 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.128 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.128 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.128 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.128 DEBUG:boto:Method: GET 2024-04-16T20:05:54.128 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.128 DEBUG:boto:Data: 2024-04-16T20:05:54.128 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.128 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.129 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.129 DEBUG:boto:Params: {} 2024-04-16T20:05:54.129 DEBUG:boto:Token: None 2024-04-16T20:05:54.129 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/?versions 2024-04-16T20:05:54.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8= 2024-04-16T20:05:54.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8=', 'Content-Length': '0'} 2024-04-16T20:05:54.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099906e23cc1eab77-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.133 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:05:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:05:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:05:54.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:05:54.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:05:54.139 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.139 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.139 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.139 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.139 DEBUG:boto:Method: GET 2024-04-16T20:05:54.139 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.139 DEBUG:boto:Data: 2024-04-16T20:05:54.140 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.140 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.140 DEBUG:boto:Params: {} 2024-04-16T20:05:54.140 DEBUG:boto:Token: None 2024-04-16T20:05:54.140 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/?versions 2024-04-16T20:05:54.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8= 2024-04-16T20:05:54.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8=', 'Content-Length': '0'} 2024-04-16T20:05:54.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004817edde95a4ee5c-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.145 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:05:54.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:05:54.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:05:54.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:05:54.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:05:54.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:05:54.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:05:54.150 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.150 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.150 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.150 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.150 DEBUG:boto:Method: GET 2024-04-16T20:05:54.151 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.151 DEBUG:boto:Data: 2024-04-16T20:05:54.151 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.151 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.151 DEBUG:boto:Params: {} 2024-04-16T20:05:54.151 DEBUG:boto:Token: None 2024-04-16T20:05:54.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/?versions 2024-04-16T20:05:54.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8= 2024-04-16T20:05:54.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8=', 'Content-Length': '0'} 2024-04-16T20:05:54.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7da1239bc60c32-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.155 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.160 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.160 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.160 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.160 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.161 DEBUG:boto:Method: GET 2024-04-16T20:05:54.161 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:05:54.161 DEBUG:boto:Data: 2024-04-16T20:05:54.161 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.161 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.162 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.162 DEBUG:boto:Params: {} 2024-04-16T20:05:54.162 DEBUG:boto:Token: None 2024-04-16T20:05:54.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/?versions 2024-04-16T20:05:54.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8= 2024-04-16T20:05:54.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rNzHckepE/jRG+nmlITDqofY/Q8=', 'Content-Length': '0'} 2024-04-16T20:05:54.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000948365732ae7901e-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.167 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:05:54.171 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.171 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.171 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.171 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.172 DEBUG:boto:Method: GET 2024-04-16T20:05:54.172 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.172 DEBUG:boto:Data: 2024-04-16T20:05:54.172 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.172 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.173 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.173 DEBUG:boto:Params: {} 2024-04-16T20:05:54.173 DEBUG:boto:Token: None 2024-04-16T20:05:54.174 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus= 2024-04-16T20:05:54.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus=', 'Content-Length': '0'} 2024-04-16T20:05:54.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8773d4e76da424e-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.178 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.178 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.178 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.178 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.179 DEBUG:boto:Method: GET 2024-04-16T20:05:54.179 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.179 DEBUG:boto:Data: 2024-04-16T20:05:54.179 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.180 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.180 DEBUG:boto:Params: {} 2024-04-16T20:05:54.180 DEBUG:boto:Token: None 2024-04-16T20:05:54.181 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus= 2024-04-16T20:05:54.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus=', 'Content-Length': '0'} 2024-04-16T20:05:54.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d94354da4201b9a-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.227 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.227 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.227 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.227 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.227 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.227 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.227 DEBUG:boto:Data: 2024-04-16T20:05:54.228 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.228 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.228 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.228 DEBUG:boto:Params: {} 2024-04-16T20:05:54.228 DEBUG:boto:Token: None 2024-04-16T20:05:54.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.229 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZEfUVBIJEm6SDyoq7oZCzxPq9g= 2024-04-16T20:05:54.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZEfUVBIJEm6SDyoq7oZCzxPq9g=', 'Content-Length': '0'} 2024-04-16T20:05:54.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b80a9de9a1e32d9c-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.231 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.231 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.231 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.232 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.232 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.232 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.232 DEBUG:boto:Data: 2024-04-16T20:05:54.232 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.232 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.232 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.232 DEBUG:boto:Params: {} 2024-04-16T20:05:54.233 DEBUG:boto:Token: None 2024-04-16T20:05:54.233 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.233 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZEfUVBIJEm6SDyoq7oZCzxPq9g= 2024-04-16T20:05:54.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZEfUVBIJEm6SDyoq7oZCzxPq9g=', 'Content-Length': '0'} 2024-04-16T20:05:54.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4bb39d6acdefe3-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:05:54.235 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.235 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.236 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.236 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.236 DEBUG:boto:Method: GET 2024-04-16T20:05:54.236 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.236 DEBUG:boto:Data: 2024-04-16T20:05:54.236 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.236 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.236 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.236 DEBUG:boto:Params: {} 2024-04-16T20:05:54.236 DEBUG:boto:Token: None 2024-04-16T20:05:54.237 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus= 2024-04-16T20:05:54.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus=', 'Content-Length': '0'} 2024-04-16T20:05:54.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db453425cbd8607-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.283 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.283 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.283 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.283 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.283 DEBUG:boto:Method: GET 2024-04-16T20:05:54.283 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.283 DEBUG:boto:Data: 2024-04-16T20:05:54.283 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.284 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.284 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.284 DEBUG:boto:Params: {} 2024-04-16T20:05:54.284 DEBUG:boto:Token: None 2024-04-16T20:05:54.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:05:54.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus= 2024-04-16T20:05:54.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nD0n95/p6p73fMOcZ8wcNySkgus=', 'Content-Length': '0'} 2024-04-16T20:05:54.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003671dc070081ea2a-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.287 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.287 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.287 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.287 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.287 DEBUG:boto:Method: GET 2024-04-16T20:05:54.287 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.287 DEBUG:boto:Data: 2024-04-16T20:05:54.287 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.287 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.287 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.288 DEBUG:boto:Params: {} 2024-04-16T20:05:54.288 DEBUG:boto:Token: None 2024-04-16T20:05:54.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e8a2d93e96c7c2-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.291 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.291 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.291 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.291 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.292 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.292 DEBUG:boto:Method: GET 2024-04-16T20:05:54.292 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.292 DEBUG:boto:Data: 2024-04-16T20:05:54.292 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.292 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.292 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.292 DEBUG:boto:Params: {} 2024-04-16T20:05:54.292 DEBUG:boto:Token: None 2024-04-16T20:05:54.292 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a02c682106295f7-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.296 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.297 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.297 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.297 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.297 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:05:54.297 DEBUG:boto:Data: 2024-04-16T20:05:54.297 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.297 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.297 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.297 DEBUG:boto:Params: {} 2024-04-16T20:05:54.297 DEBUG:boto:Token: None 2024-04-16T20:05:54.298 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11 2024-04-16T20:05:54.298 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qCn4AUnhUYO9OQILIkk5tKctpZk= 2024-04-16T20:05:54.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qCn4AUnhUYO9OQILIkk5tKctpZk=', 'Content-Length': '0'} 2024-04-16T20:05:54.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011af887b6f94f9fd-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.300 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.300 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.300 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.300 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:05:54.300 DEBUG:boto:Data: 2024-04-16T20:05:54.300 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.300 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.301 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.301 DEBUG:boto:Params: {} 2024-04-16T20:05:54.301 DEBUG:boto:Token: None 2024-04-16T20:05:54.301 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11 2024-04-16T20:05:54.301 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qCn4AUnhUYO9OQILIkk5tKctpZk= 2024-04-16T20:05:54.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qCn4AUnhUYO9OQILIkk5tKctpZk=', 'Content-Length': '0'} 2024-04-16T20:05:54.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b074b6cebc7168-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:05:54.303 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.303 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.304 DEBUG:boto:Method: GET 2024-04-16T20:05:54.304 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:05:54.304 DEBUG:boto:Data: 2024-04-16T20:05:54.304 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.304 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.304 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.304 DEBUG:boto:Params: {} 2024-04-16T20:05:54.304 DEBUG:boto:Token: None 2024-04-16T20:05:54.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11 2024-04-16T20:05:54.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yPAzRe/7RH2KpowbwhuvzadKkL0= 2024-04-16T20:05:54.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yPAzRe/7RH2KpowbwhuvzadKkL0=', 'Content-Length': '0'} 2024-04-16T20:05:54.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7c7fa3fd4f86e0-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.307 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:05:54.307 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:05:54.307 DEBUG:boto:Method: GET 2024-04-16T20:05:54.307 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:05:54.307 DEBUG:boto:Data: 2024-04-16T20:05:54.307 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.307 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.307 DEBUG:boto:Params: {} 2024-04-16T20:05:54.307 DEBUG:boto:Token: None 2024-04-16T20:05:54.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj11 2024-04-16T20:05:54.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yPAzRe/7RH2KpowbwhuvzadKkL0= 2024-04-16T20:05:54.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yPAzRe/7RH2KpowbwhuvzadKkL0=', 'Content-Length': '0'} 2024-04-16T20:05:54.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a2e6971318b31f-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.350 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.351 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.351 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.351 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.351 DEBUG:boto:Method: GET 2024-04-16T20:05:54.351 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.351 DEBUG:boto:Data: 2024-04-16T20:05:54.351 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.351 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.351 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.352 DEBUG:boto:Params: {} 2024-04-16T20:05:54.352 DEBUG:boto:Token: None 2024-04-16T20:05:54.352 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080890ef24a467756-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.355 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.355 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.355 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.356 DEBUG:boto:Method: GET 2024-04-16T20:05:54.356 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:05:54.356 DEBUG:boto:Data: 2024-04-16T20:05:54.356 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.356 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.356 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.356 DEBUG:boto:Params: {} 2024-04-16T20:05:54.356 DEBUG:boto:Token: None 2024-04-16T20:05:54.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5ebf4abb4926fa-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.362 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:05:54.362 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.362 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.363 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.363 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.363 DEBUG:boto:Method: GET 2024-04-16T20:05:54.363 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.363 DEBUG:boto:Data: 2024-04-16T20:05:54.363 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.363 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.363 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.363 DEBUG:boto:Params: {} 2024-04-16T20:05:54.363 DEBUG:boto:Token: None 2024-04-16T20:05:54.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4= 2024-04-16T20:05:54.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4=', 'Content-Length': '0'} 2024-04-16T20:05:54.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008079798a4c510f36-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.406 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.406 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.407 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.407 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.407 DEBUG:boto:Method: GET 2024-04-16T20:05:54.407 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.407 DEBUG:boto:Data: 2024-04-16T20:05:54.407 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.407 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.407 DEBUG:boto:Params: {} 2024-04-16T20:05:54.407 DEBUG:boto:Token: None 2024-04-16T20:05:54.408 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4= 2024-04-16T20:05:54.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4=', 'Content-Length': '0'} 2024-04-16T20:05:54.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa2e5afe69f944d1-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.410 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.410 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.411 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.411 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.411 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.411 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.411 DEBUG:boto:Data: 2024-04-16T20:05:54.411 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.411 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.411 DEBUG:boto:Params: {} 2024-04-16T20:05:54.411 DEBUG:boto:Token: None 2024-04-16T20:05:54.411 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p96kmRgtYRUmPjGJ1dBH6a6zEl4= 2024-04-16T20:05:54.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p96kmRgtYRUmPjGJ1dBH6a6zEl4=', 'Content-Length': '0'} 2024-04-16T20:05:54.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c275231ca6d713b0-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.414 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.414 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.414 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.414 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.414 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.414 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.414 DEBUG:boto:Data: 2024-04-16T20:05:54.415 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.415 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.415 DEBUG:boto:Params: {} 2024-04-16T20:05:54.415 DEBUG:boto:Token: None 2024-04-16T20:05:54.415 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.415 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p96kmRgtYRUmPjGJ1dBH6a6zEl4= 2024-04-16T20:05:54.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p96kmRgtYRUmPjGJ1dBH6a6zEl4=', 'Content-Length': '0'} 2024-04-16T20:05:54.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7347d00f127021c-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:05:54.418 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.418 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.418 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.418 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.418 DEBUG:boto:Method: GET 2024-04-16T20:05:54.418 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.418 DEBUG:boto:Data: 2024-04-16T20:05:54.418 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.419 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.419 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.419 DEBUG:boto:Params: {} 2024-04-16T20:05:54.419 DEBUG:boto:Token: None 2024-04-16T20:05:54.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4= 2024-04-16T20:05:54.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4=', 'Content-Length': '0'} 2024-04-16T20:05:54.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b978080008cd1b6e-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.421 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.422 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.422 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.422 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.422 DEBUG:boto:Method: GET 2024-04-16T20:05:54.422 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.422 DEBUG:boto:Data: 2024-04-16T20:05:54.422 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.422 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.422 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.422 DEBUG:boto:Params: {} 2024-04-16T20:05:54.422 DEBUG:boto:Token: None 2024-04-16T20:05:54.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:05:54.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4= 2024-04-16T20:05:54.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vYRlHQuiBpwrdS1js7PUhtrYfE4=', 'Content-Length': '0'} 2024-04-16T20:05:54.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda68e978d20d641-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.425 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.425 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.425 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.425 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.425 DEBUG:boto:Method: GET 2024-04-16T20:05:54.426 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.426 DEBUG:boto:Data: 2024-04-16T20:05:54.426 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.426 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.426 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.426 DEBUG:boto:Params: {} 2024-04-16T20:05:54.426 DEBUG:boto:Token: None 2024-04-16T20:05:54.426 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.426 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc5e58cefabb0b5d-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.429 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.429 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.429 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.429 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.429 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.429 DEBUG:boto:Method: GET 2024-04-16T20:05:54.430 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.430 DEBUG:boto:Data: 2024-04-16T20:05:54.430 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.430 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.430 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.430 DEBUG:boto:Params: {} 2024-04-16T20:05:54.430 DEBUG:boto:Token: None 2024-04-16T20:05:54.430 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.430 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005798ec2b930fe785-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.434 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.434 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.434 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.434 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.434 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:05:54.435 DEBUG:boto:Data: 2024-04-16T20:05:54.435 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.435 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.435 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.435 DEBUG:boto:Params: {} 2024-04-16T20:05:54.435 DEBUG:boto:Token: None 2024-04-16T20:05:54.435 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115 2024-04-16T20:05:54.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxxcYUHToBmif7ghBRhd4207r+g= 2024-04-16T20:05:54.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxxcYUHToBmif7ghBRhd4207r+g=', 'Content-Length': '0'} 2024-04-16T20:05:54.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e0fb7af5b8a99c6-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.438 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.438 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.438 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.438 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:05:54.438 DEBUG:boto:Data: 2024-04-16T20:05:54.438 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.438 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.438 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.439 DEBUG:boto:Params: {} 2024-04-16T20:05:54.439 DEBUG:boto:Token: None 2024-04-16T20:05:54.439 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115 2024-04-16T20:05:54.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxxcYUHToBmif7ghBRhd4207r+g= 2024-04-16T20:05:54.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxxcYUHToBmif7ghBRhd4207r+g=', 'Content-Length': '0'} 2024-04-16T20:05:54.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fdc12d21340f3be-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:05:54.441 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.441 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.442 DEBUG:boto:Method: GET 2024-04-16T20:05:54.442 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:05:54.442 DEBUG:boto:Data: 2024-04-16T20:05:54.442 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.442 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.442 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.442 DEBUG:boto:Params: {} 2024-04-16T20:05:54.442 DEBUG:boto:Token: None 2024-04-16T20:05:54.442 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115 2024-04-16T20:05:54.442 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F41YqXUvOirnXXiDttcI4SHQGmQ= 2024-04-16T20:05:54.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F41YqXUvOirnXXiDttcI4SHQGmQ=', 'Content-Length': '0'} 2024-04-16T20:05:54.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e25175251dc335-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.445 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:05:54.445 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:05:54.445 DEBUG:boto:Method: GET 2024-04-16T20:05:54.445 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:05:54.445 DEBUG:boto:Data: 2024-04-16T20:05:54.445 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.445 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.446 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.446 DEBUG:boto:Params: {} 2024-04-16T20:05:54.446 DEBUG:boto:Token: None 2024-04-16T20:05:54.446 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj115 2024-04-16T20:05:54.446 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F41YqXUvOirnXXiDttcI4SHQGmQ= 2024-04-16T20:05:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F41YqXUvOirnXXiDttcI4SHQGmQ=', 'Content-Length': '0'} 2024-04-16T20:05:54.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1eb5d743edf1966-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.449 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.449 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.449 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.449 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.449 DEBUG:boto:Method: GET 2024-04-16T20:05:54.449 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.449 DEBUG:boto:Data: 2024-04-16T20:05:54.449 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.449 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.449 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.449 DEBUG:boto:Params: {} 2024-04-16T20:05:54.450 DEBUG:boto:Token: None 2024-04-16T20:05:54.450 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.450 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ab46d8d2c702d2-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.452 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.453 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.453 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.453 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.453 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.453 DEBUG:boto:Method: GET 2024-04-16T20:05:54.453 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:05:54.453 DEBUG:boto:Data: 2024-04-16T20:05:54.453 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.453 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.453 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.454 DEBUG:boto:Params: {} 2024-04-16T20:05:54.454 DEBUG:boto:Token: None 2024-04-16T20:05:54.454 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.454 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108b9d2e39ca4bea-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.457 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:05:54.458 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.458 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.458 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.458 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.458 DEBUG:boto:Method: GET 2024-04-16T20:05:54.458 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.458 DEBUG:boto:Data: 2024-04-16T20:05:54.458 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.458 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.459 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.459 DEBUG:boto:Params: {} 2024-04-16T20:05:54.459 DEBUG:boto:Token: None 2024-04-16T20:05:54.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM= 2024-04-16T20:05:54.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM=', 'Content-Length': '0'} 2024-04-16T20:05:54.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a409cabaed94bedb-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.461 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.461 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.461 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.461 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.461 DEBUG:boto:Method: GET 2024-04-16T20:05:54.462 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.462 DEBUG:boto:Data: 2024-04-16T20:05:54.462 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.462 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.462 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.462 DEBUG:boto:Params: {} 2024-04-16T20:05:54.462 DEBUG:boto:Token: None 2024-04-16T20:05:54.462 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.462 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM= 2024-04-16T20:05:54.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM=', 'Content-Length': '0'} 2024-04-16T20:05:54.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743b04204c2d925c-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.465 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.465 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.465 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.465 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.465 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.465 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.465 DEBUG:boto:Data: 2024-04-16T20:05:54.465 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.466 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.466 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.466 DEBUG:boto:Params: {} 2024-04-16T20:05:54.466 DEBUG:boto:Token: None 2024-04-16T20:05:54.466 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.466 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TWBpPQwuntcfWQai6qUmEmFNZ/o= 2024-04-16T20:05:54.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TWBpPQwuntcfWQai6qUmEmFNZ/o=', 'Content-Length': '0'} 2024-04-16T20:05:54.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee3b2b6524870f5-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.468 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.468 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.468 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.468 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.468 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.468 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.468 DEBUG:boto:Data: 2024-04-16T20:05:54.469 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.469 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.469 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.469 DEBUG:boto:Params: {} 2024-04-16T20:05:54.469 DEBUG:boto:Token: None 2024-04-16T20:05:54.469 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TWBpPQwuntcfWQai6qUmEmFNZ/o= 2024-04-16T20:05:54.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TWBpPQwuntcfWQai6qUmEmFNZ/o=', 'Content-Length': '0'} 2024-04-16T20:05:54.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de071eb7d2ca7d6b-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:05:54.471 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.471 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.471 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.471 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.471 DEBUG:boto:Method: GET 2024-04-16T20:05:54.471 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.472 DEBUG:boto:Data: 2024-04-16T20:05:54.472 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.472 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.472 DEBUG:boto:Params: {} 2024-04-16T20:05:54.472 DEBUG:boto:Token: None 2024-04-16T20:05:54.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM= 2024-04-16T20:05:54.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM=', 'Content-Length': '0'} 2024-04-16T20:05:54.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a76969435558f29-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.474 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.474 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.475 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.475 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.475 DEBUG:boto:Method: GET 2024-04-16T20:05:54.475 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.475 DEBUG:boto:Data: 2024-04-16T20:05:54.475 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.475 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.475 DEBUG:boto:Params: {} 2024-04-16T20:05:54.475 DEBUG:boto:Token: None 2024-04-16T20:05:54.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:05:54.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM= 2024-04-16T20:05:54.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mv8QFJzPcFqyD5IHzBmptduGmSM=', 'Content-Length': '0'} 2024-04-16T20:05:54.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3eba35616e0552-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.478 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.478 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.478 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.478 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.478 DEBUG:boto:Method: GET 2024-04-16T20:05:54.478 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.478 DEBUG:boto:Data: 2024-04-16T20:05:54.478 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.479 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.479 DEBUG:boto:Params: {} 2024-04-16T20:05:54.479 DEBUG:boto:Token: None 2024-04-16T20:05:54.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000892c8b46f227efc5-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.482 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.482 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.482 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.482 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.482 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.482 DEBUG:boto:Method: GET 2024-04-16T20:05:54.482 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.483 DEBUG:boto:Data: 2024-04-16T20:05:54.483 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.483 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.483 DEBUG:boto:Params: {} 2024-04-16T20:05:54.483 DEBUG:boto:Token: None 2024-04-16T20:05:54.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009167f4873a3024b6-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.486 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.487 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.487 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.487 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.487 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:05:54.487 DEBUG:boto:Data: 2024-04-16T20:05:54.487 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.487 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.487 DEBUG:boto:Params: {} 2024-04-16T20:05:54.488 DEBUG:boto:Token: None 2024-04-16T20:05:54.488 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116 2024-04-16T20:05:54.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IkNTmV5+e4UUbT52rrZ7hJLu+Ck= 2024-04-16T20:05:54.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IkNTmV5+e4UUbT52rrZ7hJLu+Ck=', 'Content-Length': '0'} 2024-04-16T20:05:54.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000666c386b7c2902e0-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.490 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.490 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.490 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.490 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:05:54.490 DEBUG:boto:Data: 2024-04-16T20:05:54.490 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.490 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.490 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.490 DEBUG:boto:Params: {} 2024-04-16T20:05:54.490 DEBUG:boto:Token: None 2024-04-16T20:05:54.491 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116 2024-04-16T20:05:54.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IkNTmV5+e4UUbT52rrZ7hJLu+Ck= 2024-04-16T20:05:54.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IkNTmV5+e4UUbT52rrZ7hJLu+Ck=', 'Content-Length': '0'} 2024-04-16T20:05:54.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e4686b297539ac-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:05:54.493 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.493 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.493 DEBUG:boto:Method: GET 2024-04-16T20:05:54.493 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:05:54.493 DEBUG:boto:Data: 2024-04-16T20:05:54.494 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.494 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.494 DEBUG:boto:Params: {} 2024-04-16T20:05:54.494 DEBUG:boto:Token: None 2024-04-16T20:05:54.494 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116 2024-04-16T20:05:54.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3W8lY7DLkxwiG6WW5qteu6193FM= 2024-04-16T20:05:54.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3W8lY7DLkxwiG6WW5qteu6193FM=', 'Content-Length': '0'} 2024-04-16T20:05:54.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b140fdc78185da-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.496 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:05:54.497 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:05:54.497 DEBUG:boto:Method: GET 2024-04-16T20:05:54.497 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:05:54.497 DEBUG:boto:Data: 2024-04-16T20:05:54.497 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.497 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.497 DEBUG:boto:Params: {} 2024-04-16T20:05:54.497 DEBUG:boto:Token: None 2024-04-16T20:05:54.497 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj116 2024-04-16T20:05:54.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3W8lY7DLkxwiG6WW5qteu6193FM= 2024-04-16T20:05:54.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3W8lY7DLkxwiG6WW5qteu6193FM=', 'Content-Length': '0'} 2024-04-16T20:05:54.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f597de8db4f0f58-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.546 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.546 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.547 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.547 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.547 DEBUG:boto:Method: GET 2024-04-16T20:05:54.547 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.547 DEBUG:boto:Data: 2024-04-16T20:05:54.547 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.547 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.547 DEBUG:boto:Params: {} 2024-04-16T20:05:54.547 DEBUG:boto:Token: None 2024-04-16T20:05:54.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f180517c78f2fab-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.550 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.551 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.551 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.551 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.551 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.551 DEBUG:boto:Method: GET 2024-04-16T20:05:54.551 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:05:54.551 DEBUG:boto:Data: 2024-04-16T20:05:54.551 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.551 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.551 DEBUG:boto:Params: {} 2024-04-16T20:05:54.552 DEBUG:boto:Token: None 2024-04-16T20:05:54.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009650651996fbab98-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.555 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:05:54.556 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.556 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.556 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.556 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.556 DEBUG:boto:Method: GET 2024-04-16T20:05:54.556 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.556 DEBUG:boto:Data: 2024-04-16T20:05:54.556 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.556 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.557 DEBUG:boto:Params: {} 2024-04-16T20:05:54.557 DEBUG:boto:Token: None 2024-04-16T20:05:54.557 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.557 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU= 2024-04-16T20:05:54.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU=', 'Content-Length': '0'} 2024-04-16T20:05:54.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecdb7ed4495cea5e-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.559 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.559 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.559 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.559 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.560 DEBUG:boto:Method: GET 2024-04-16T20:05:54.560 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.560 DEBUG:boto:Data: 2024-04-16T20:05:54.560 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.560 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.560 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.560 DEBUG:boto:Params: {} 2024-04-16T20:05:54.560 DEBUG:boto:Token: None 2024-04-16T20:05:54.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU= 2024-04-16T20:05:54.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU=', 'Content-Length': '0'} 2024-04-16T20:05:54.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9d83defe060d32-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.562 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.563 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.563 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.563 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.563 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.563 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.563 DEBUG:boto:Data: 2024-04-16T20:05:54.563 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.563 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.563 DEBUG:boto:Params: {} 2024-04-16T20:05:54.564 DEBUG:boto:Token: None 2024-04-16T20:05:54.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AoV6p/2FRbO6rsI+f6NlPrlWrGI= 2024-04-16T20:05:54.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AoV6p/2FRbO6rsI+f6NlPrlWrGI=', 'Content-Length': '0'} 2024-04-16T20:05:54.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001317e950c14e39a6-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.566 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.566 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.566 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.566 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.566 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.566 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.567 DEBUG:boto:Data: 2024-04-16T20:05:54.567 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.567 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.567 DEBUG:boto:Params: {} 2024-04-16T20:05:54.567 DEBUG:boto:Token: None 2024-04-16T20:05:54.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AoV6p/2FRbO6rsI+f6NlPrlWrGI= 2024-04-16T20:05:54.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AoV6p/2FRbO6rsI+f6NlPrlWrGI=', 'Content-Length': '0'} 2024-04-16T20:05:54.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f3a983f62dadf50-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:05:54.569 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.570 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.570 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.570 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.570 DEBUG:boto:Method: GET 2024-04-16T20:05:54.570 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.570 DEBUG:boto:Data: 2024-04-16T20:05:54.570 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.570 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.570 DEBUG:boto:Params: {} 2024-04-16T20:05:54.571 DEBUG:boto:Token: None 2024-04-16T20:05:54.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU= 2024-04-16T20:05:54.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU=', 'Content-Length': '0'} 2024-04-16T20:05:54.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bcf87e17c8772ec-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.614 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.614 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.615 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.615 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.615 DEBUG:boto:Method: GET 2024-04-16T20:05:54.615 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.615 DEBUG:boto:Data: 2024-04-16T20:05:54.615 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.615 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.615 DEBUG:boto:Params: {} 2024-04-16T20:05:54.615 DEBUG:boto:Token: None 2024-04-16T20:05:54.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:05:54.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU= 2024-04-16T20:05:54.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+04HzYrSg2s2ThYviUOSYV6fWU=', 'Content-Length': '0'} 2024-04-16T20:05:54.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591017f3faea2a7b-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.618 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.618 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.618 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.618 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.618 DEBUG:boto:Method: GET 2024-04-16T20:05:54.618 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.618 DEBUG:boto:Data: 2024-04-16T20:05:54.618 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.619 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.619 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.619 DEBUG:boto:Params: {} 2024-04-16T20:05:54.619 DEBUG:boto:Token: None 2024-04-16T20:05:54.619 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.619 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041247551fc94291e-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.622 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.622 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.622 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.622 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.622 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.622 DEBUG:boto:Method: GET 2024-04-16T20:05:54.622 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.623 DEBUG:boto:Data: 2024-04-16T20:05:54.623 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.623 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.623 DEBUG:boto:Params: {} 2024-04-16T20:05:54.623 DEBUG:boto:Token: None 2024-04-16T20:05:54.623 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.623 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deffbbd7530f0805-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.627 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.627 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.627 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.628 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.628 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:05:54.628 DEBUG:boto:Data: 2024-04-16T20:05:54.628 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.628 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.628 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.628 DEBUG:boto:Params: {} 2024-04-16T20:05:54.628 DEBUG:boto:Token: None 2024-04-16T20:05:54.628 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124 2024-04-16T20:05:54.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RmaY/JWx4cCPDuoOZomjTEAUfVA= 2024-04-16T20:05:54.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RmaY/JWx4cCPDuoOZomjTEAUfVA=', 'Content-Length': '0'} 2024-04-16T20:05:54.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3c28b568396505-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.631 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.631 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.631 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.631 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:05:54.631 DEBUG:boto:Data: 2024-04-16T20:05:54.631 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.631 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.631 DEBUG:boto:Params: {} 2024-04-16T20:05:54.631 DEBUG:boto:Token: None 2024-04-16T20:05:54.632 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124 2024-04-16T20:05:54.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RmaY/JWx4cCPDuoOZomjTEAUfVA= 2024-04-16T20:05:54.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RmaY/JWx4cCPDuoOZomjTEAUfVA=', 'Content-Length': '0'} 2024-04-16T20:05:54.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bbba8b8eb95be29-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:05:54.634 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.634 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.634 DEBUG:boto:Method: GET 2024-04-16T20:05:54.634 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:05:54.634 DEBUG:boto:Data: 2024-04-16T20:05:54.634 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.635 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.635 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.635 DEBUG:boto:Params: {} 2024-04-16T20:05:54.635 DEBUG:boto:Token: None 2024-04-16T20:05:54.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124 2024-04-16T20:05:54.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xRox8yRFKOsCzXn7vlGZs6Hl7dg= 2024-04-16T20:05:54.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xRox8yRFKOsCzXn7vlGZs6Hl7dg=', 'Content-Length': '0'} 2024-04-16T20:05:54.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052bc839261d35b4f-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.637 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:05:54.637 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:05:54.637 DEBUG:boto:Method: GET 2024-04-16T20:05:54.638 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:05:54.638 DEBUG:boto:Data: 2024-04-16T20:05:54.638 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.638 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.638 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.638 DEBUG:boto:Params: {} 2024-04-16T20:05:54.638 DEBUG:boto:Token: None 2024-04-16T20:05:54.638 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj124 2024-04-16T20:05:54.638 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xRox8yRFKOsCzXn7vlGZs6Hl7dg= 2024-04-16T20:05:54.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xRox8yRFKOsCzXn7vlGZs6Hl7dg=', 'Content-Length': '0'} 2024-04-16T20:05:54.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9489c0f96dccc67-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.641 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.641 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.641 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.641 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.641 DEBUG:boto:Method: GET 2024-04-16T20:05:54.641 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.641 DEBUG:boto:Data: 2024-04-16T20:05:54.641 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.641 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.641 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.642 DEBUG:boto:Params: {} 2024-04-16T20:05:54.642 DEBUG:boto:Token: None 2024-04-16T20:05:54.642 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.642 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001186efc91db57eba-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.645 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.645 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.645 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.645 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.645 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.645 DEBUG:boto:Method: GET 2024-04-16T20:05:54.645 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:05:54.646 DEBUG:boto:Data: 2024-04-16T20:05:54.646 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.646 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.646 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.646 DEBUG:boto:Params: {} 2024-04-16T20:05:54.646 DEBUG:boto:Token: None 2024-04-16T20:05:54.646 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.646 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832b23b87dc20eeb-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.650 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:05:54.650 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.650 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.650 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.651 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.651 DEBUG:boto:Method: GET 2024-04-16T20:05:54.651 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.651 DEBUG:boto:Data: 2024-04-16T20:05:54.651 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.651 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.651 DEBUG:boto:Params: {} 2024-04-16T20:05:54.651 DEBUG:boto:Token: None 2024-04-16T20:05:54.651 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU= 2024-04-16T20:05:54.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU=', 'Content-Length': '0'} 2024-04-16T20:05:54.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b3abe904eaaeca-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.654 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.654 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.654 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.654 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.654 DEBUG:boto:Method: GET 2024-04-16T20:05:54.654 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.654 DEBUG:boto:Data: 2024-04-16T20:05:54.655 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.655 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.655 DEBUG:boto:Params: {} 2024-04-16T20:05:54.655 DEBUG:boto:Token: None 2024-04-16T20:05:54.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU= 2024-04-16T20:05:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU=', 'Content-Length': '0'} 2024-04-16T20:05:54.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef004a40bc183f6e-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.658 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.658 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.658 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.658 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.658 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.658 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.658 DEBUG:boto:Data: 2024-04-16T20:05:54.658 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.659 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.659 DEBUG:boto:Params: {} 2024-04-16T20:05:54.659 DEBUG:boto:Token: None 2024-04-16T20:05:54.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8ehkdihOibaY8PC20U6muEA6Cpw= 2024-04-16T20:05:54.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8ehkdihOibaY8PC20U6muEA6Cpw=', 'Content-Length': '0'} 2024-04-16T20:05:54.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a92939ff015e3d-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.661 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.661 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.661 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.662 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.662 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.662 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.662 DEBUG:boto:Data: 2024-04-16T20:05:54.662 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.662 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.662 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.662 DEBUG:boto:Params: {} 2024-04-16T20:05:54.662 DEBUG:boto:Token: None 2024-04-16T20:05:54.662 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8ehkdihOibaY8PC20U6muEA6Cpw= 2024-04-16T20:05:54.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8ehkdihOibaY8PC20U6muEA6Cpw=', 'Content-Length': '0'} 2024-04-16T20:05:54.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8a64fde7afdb61-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:05:54.665 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.665 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.665 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.665 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.665 DEBUG:boto:Method: GET 2024-04-16T20:05:54.665 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.665 DEBUG:boto:Data: 2024-04-16T20:05:54.665 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.665 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.665 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.666 DEBUG:boto:Params: {} 2024-04-16T20:05:54.666 DEBUG:boto:Token: None 2024-04-16T20:05:54.666 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU= 2024-04-16T20:05:54.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU=', 'Content-Length': '0'} 2024-04-16T20:05:54.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dc6cf557792680f-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.668 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.668 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.668 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.669 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.669 DEBUG:boto:Method: GET 2024-04-16T20:05:54.669 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.669 DEBUG:boto:Data: 2024-04-16T20:05:54.669 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.669 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.669 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.669 DEBUG:boto:Params: {} 2024-04-16T20:05:54.669 DEBUG:boto:Token: None 2024-04-16T20:05:54.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:05:54.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU= 2024-04-16T20:05:54.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wR6Nqvw7pKBVmCa3tFIbQcoo2mU=', 'Content-Length': '0'} 2024-04-16T20:05:54.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000372a91883ac7a6d7-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.672 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.672 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.672 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.672 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.672 DEBUG:boto:Method: GET 2024-04-16T20:05:54.672 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.672 DEBUG:boto:Data: 2024-04-16T20:05:54.672 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.673 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.673 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.673 DEBUG:boto:Params: {} 2024-04-16T20:05:54.673 DEBUG:boto:Token: None 2024-04-16T20:05:54.673 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a08f0e359dbd1ae-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.675 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.676 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.676 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.676 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.676 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.676 DEBUG:boto:Method: GET 2024-04-16T20:05:54.676 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.676 DEBUG:boto:Data: 2024-04-16T20:05:54.676 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.676 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.676 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.677 DEBUG:boto:Params: {} 2024-04-16T20:05:54.677 DEBUG:boto:Token: None 2024-04-16T20:05:54.677 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070174d69535b1837-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.680 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.680 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.681 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.681 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.681 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:05:54.681 DEBUG:boto:Data: 2024-04-16T20:05:54.681 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.681 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.681 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.681 DEBUG:boto:Params: {} 2024-04-16T20:05:54.681 DEBUG:boto:Token: None 2024-04-16T20:05:54.681 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145 2024-04-16T20:05:54.682 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nF+Lq5Jn8MkYQZ/yPdaqrZz2P7s= 2024-04-16T20:05:54.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nF+Lq5Jn8MkYQZ/yPdaqrZz2P7s=', 'Content-Length': '0'} 2024-04-16T20:05:54.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c449bb427847973-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.684 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.684 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.684 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.684 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:05:54.684 DEBUG:boto:Data: 2024-04-16T20:05:54.684 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.684 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.684 DEBUG:boto:Params: {} 2024-04-16T20:05:54.684 DEBUG:boto:Token: None 2024-04-16T20:05:54.684 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145 2024-04-16T20:05:54.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nF+Lq5Jn8MkYQZ/yPdaqrZz2P7s= 2024-04-16T20:05:54.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nF+Lq5Jn8MkYQZ/yPdaqrZz2P7s=', 'Content-Length': '0'} 2024-04-16T20:05:54.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e43976916296af-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:05:54.687 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.687 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.687 DEBUG:boto:Method: GET 2024-04-16T20:05:54.687 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:05:54.688 DEBUG:boto:Data: 2024-04-16T20:05:54.688 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.688 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.688 DEBUG:boto:Params: {} 2024-04-16T20:05:54.688 DEBUG:boto:Token: None 2024-04-16T20:05:54.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145 2024-04-16T20:05:54.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+uRWCXctzkdb0avGH0wmAe4VrOA= 2024-04-16T20:05:54.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+uRWCXctzkdb0avGH0wmAe4VrOA=', 'Content-Length': '0'} 2024-04-16T20:05:54.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965a73ddbf727df0-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.734 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:05:54.734 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:05:54.735 DEBUG:boto:Method: GET 2024-04-16T20:05:54.735 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:05:54.735 DEBUG:boto:Data: 2024-04-16T20:05:54.735 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.735 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.735 DEBUG:boto:Params: {} 2024-04-16T20:05:54.735 DEBUG:boto:Token: None 2024-04-16T20:05:54.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj145 2024-04-16T20:05:54.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+uRWCXctzkdb0avGH0wmAe4VrOA= 2024-04-16T20:05:54.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+uRWCXctzkdb0avGH0wmAe4VrOA=', 'Content-Length': '0'} 2024-04-16T20:05:54.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4d5d8bef3f118b-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.738 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.738 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.738 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.738 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.738 DEBUG:boto:Method: GET 2024-04-16T20:05:54.738 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.738 DEBUG:boto:Data: 2024-04-16T20:05:54.738 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.738 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.739 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.739 DEBUG:boto:Params: {} 2024-04-16T20:05:54.739 DEBUG:boto:Token: None 2024-04-16T20:05:54.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039ed30c79e873087-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.742 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.742 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.742 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.742 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.742 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.742 DEBUG:boto:Method: GET 2024-04-16T20:05:54.742 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:05:54.743 DEBUG:boto:Data: 2024-04-16T20:05:54.743 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.743 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.743 DEBUG:boto:Params: {} 2024-04-16T20:05:54.743 DEBUG:boto:Token: None 2024-04-16T20:05:54.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4fe5a18d9a34b02-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.747 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:05:54.747 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.747 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.747 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.747 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.748 DEBUG:boto:Method: GET 2024-04-16T20:05:54.748 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.748 DEBUG:boto:Data: 2024-04-16T20:05:54.748 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.748 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.748 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.748 DEBUG:boto:Params: {} 2024-04-16T20:05:54.748 DEBUG:boto:Token: None 2024-04-16T20:05:54.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4= 2024-04-16T20:05:54.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4=', 'Content-Length': '0'} 2024-04-16T20:05:54.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972f093b45548451-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.751 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.751 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.751 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.751 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.751 DEBUG:boto:Method: GET 2024-04-16T20:05:54.752 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.752 DEBUG:boto:Data: 2024-04-16T20:05:54.752 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.752 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.752 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.752 DEBUG:boto:Params: {} 2024-04-16T20:05:54.752 DEBUG:boto:Token: None 2024-04-16T20:05:54.752 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4= 2024-04-16T20:05:54.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4=', 'Content-Length': '0'} 2024-04-16T20:05:54.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea16b254c7d48c2c-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.755 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.755 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.755 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.755 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.755 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.755 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.755 DEBUG:boto:Data: 2024-04-16T20:05:54.755 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.755 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.755 DEBUG:boto:Params: {} 2024-04-16T20:05:54.756 DEBUG:boto:Token: None 2024-04-16T20:05:54.756 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WmyxeAxz2/pGT7fUw4ftlPPiqwM= 2024-04-16T20:05:54.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WmyxeAxz2/pGT7fUw4ftlPPiqwM=', 'Content-Length': '0'} 2024-04-16T20:05:54.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006742508748620e2e-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.758 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.758 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.758 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.758 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.758 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.758 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.759 DEBUG:boto:Data: 2024-04-16T20:05:54.759 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.759 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.759 DEBUG:boto:Params: {} 2024-04-16T20:05:54.759 DEBUG:boto:Token: None 2024-04-16T20:05:54.759 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WmyxeAxz2/pGT7fUw4ftlPPiqwM= 2024-04-16T20:05:54.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WmyxeAxz2/pGT7fUw4ftlPPiqwM=', 'Content-Length': '0'} 2024-04-16T20:05:54.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc4cec03cb1a9df-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:05:54.761 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.762 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.762 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.762 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.762 DEBUG:boto:Method: GET 2024-04-16T20:05:54.762 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.762 DEBUG:boto:Data: 2024-04-16T20:05:54.762 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.762 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.762 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.762 DEBUG:boto:Params: {} 2024-04-16T20:05:54.762 DEBUG:boto:Token: None 2024-04-16T20:05:54.762 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.763 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4= 2024-04-16T20:05:54.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4=', 'Content-Length': '0'} 2024-04-16T20:05:54.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000148ab400ea34e202-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.765 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.765 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.765 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.765 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.765 DEBUG:boto:Method: GET 2024-04-16T20:05:54.766 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.766 DEBUG:boto:Data: 2024-04-16T20:05:54.766 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.766 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.766 DEBUG:boto:Params: {} 2024-04-16T20:05:54.766 DEBUG:boto:Token: None 2024-04-16T20:05:54.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:05:54.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4= 2024-04-16T20:05:54.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5q0ev44Fg7EoKPGgv/79lKsRJ4=', 'Content-Length': '0'} 2024-04-16T20:05:54.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef97bf3168b760b4-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.771 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.771 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.771 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.771 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.772 DEBUG:boto:Method: GET 2024-04-16T20:05:54.772 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.772 DEBUG:boto:Data: 2024-04-16T20:05:54.772 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.772 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.772 DEBUG:boto:Params: {} 2024-04-16T20:05:54.772 DEBUG:boto:Token: None 2024-04-16T20:05:54.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041dca69ef30c8f77-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.775 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.775 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.775 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.775 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.775 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.775 DEBUG:boto:Method: GET 2024-04-16T20:05:54.775 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.775 DEBUG:boto:Data: 2024-04-16T20:05:54.776 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.776 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.776 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.776 DEBUG:boto:Params: {} 2024-04-16T20:05:54.776 DEBUG:boto:Token: None 2024-04-16T20:05:54.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009119bf221e434e7-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.822 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.822 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.823 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.823 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.823 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:05:54.823 DEBUG:boto:Data: 2024-04-16T20:05:54.823 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.823 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.823 DEBUG:boto:Params: {} 2024-04-16T20:05:54.823 DEBUG:boto:Token: None 2024-04-16T20:05:54.823 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155 2024-04-16T20:05:54.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l8m9UmVc3+JUttuX/Xcu0zkwtDA= 2024-04-16T20:05:54.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l8m9UmVc3+JUttuX/Xcu0zkwtDA=', 'Content-Length': '0'} 2024-04-16T20:05:54.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bda2166cc25cb67-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.826 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.826 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.826 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.827 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:05:54.827 DEBUG:boto:Data: 2024-04-16T20:05:54.827 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.827 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.827 DEBUG:boto:Params: {} 2024-04-16T20:05:54.827 DEBUG:boto:Token: None 2024-04-16T20:05:54.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155 2024-04-16T20:05:54.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l8m9UmVc3+JUttuX/Xcu0zkwtDA= 2024-04-16T20:05:54.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l8m9UmVc3+JUttuX/Xcu0zkwtDA=', 'Content-Length': '0'} 2024-04-16T20:05:54.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4d9c1a3951725d-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:05:54.830 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.830 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.830 DEBUG:boto:Method: GET 2024-04-16T20:05:54.830 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:05:54.830 DEBUG:boto:Data: 2024-04-16T20:05:54.830 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.830 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.830 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.830 DEBUG:boto:Params: {} 2024-04-16T20:05:54.831 DEBUG:boto:Token: None 2024-04-16T20:05:54.831 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155 2024-04-16T20:05:54.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KZFS28wyxhIK7JrI4+qI1bhyqZQ= 2024-04-16T20:05:54.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KZFS28wyxhIK7JrI4+qI1bhyqZQ=', 'Content-Length': '0'} 2024-04-16T20:05:54.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092eea5269780d2e2-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.833 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:05:54.833 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:05:54.833 DEBUG:boto:Method: GET 2024-04-16T20:05:54.834 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:05:54.834 DEBUG:boto:Data: 2024-04-16T20:05:54.834 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.834 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.834 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.834 DEBUG:boto:Params: {} 2024-04-16T20:05:54.834 DEBUG:boto:Token: None 2024-04-16T20:05:54.834 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj155 2024-04-16T20:05:54.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KZFS28wyxhIK7JrI4+qI1bhyqZQ= 2024-04-16T20:05:54.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KZFS28wyxhIK7JrI4+qI1bhyqZQ=', 'Content-Length': '0'} 2024-04-16T20:05:54.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762ad1fd91d72d3a-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.837 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.837 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.837 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.837 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.837 DEBUG:boto:Method: GET 2024-04-16T20:05:54.837 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.837 DEBUG:boto:Data: 2024-04-16T20:05:54.837 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.838 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.838 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.838 DEBUG:boto:Params: {} 2024-04-16T20:05:54.838 DEBUG:boto:Token: None 2024-04-16T20:05:54.838 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.838 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b517ee459bc6a9-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.841 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.841 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.841 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.841 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.841 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.841 DEBUG:boto:Method: GET 2024-04-16T20:05:54.841 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:05:54.842 DEBUG:boto:Data: 2024-04-16T20:05:54.842 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.842 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.842 DEBUG:boto:Params: {} 2024-04-16T20:05:54.842 DEBUG:boto:Token: None 2024-04-16T20:05:54.842 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.842 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e500b3c50e3582fa-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.846 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:05:54.846 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.846 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.846 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.846 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.846 DEBUG:boto:Method: GET 2024-04-16T20:05:54.847 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.847 DEBUG:boto:Data: 2024-04-16T20:05:54.847 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.847 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.847 DEBUG:boto:Params: {} 2024-04-16T20:05:54.847 DEBUG:boto:Token: None 2024-04-16T20:05:54.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw= 2024-04-16T20:05:54.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw=', 'Content-Length': '0'} 2024-04-16T20:05:54.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032e00f275c8a282-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.850 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.850 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.850 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.850 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.850 DEBUG:boto:Method: GET 2024-04-16T20:05:54.850 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.850 DEBUG:boto:Data: 2024-04-16T20:05:54.850 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.851 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.851 DEBUG:boto:Params: {} 2024-04-16T20:05:54.851 DEBUG:boto:Token: None 2024-04-16T20:05:54.851 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.851 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw= 2024-04-16T20:05:54.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw=', 'Content-Length': '0'} 2024-04-16T20:05:54.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7a3dd21cf959bb-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.854 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.854 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.854 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.854 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.854 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.854 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.854 DEBUG:boto:Data: 2024-04-16T20:05:54.854 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.855 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.855 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.855 DEBUG:boto:Params: {} 2024-04-16T20:05:54.855 DEBUG:boto:Token: None 2024-04-16T20:05:54.855 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.855 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g4RUBGtNO7Rq4ELr8jLQkl27uHs= 2024-04-16T20:05:54.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g4RUBGtNO7Rq4ELr8jLQkl27uHs=', 'Content-Length': '0'} 2024-04-16T20:05:54.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de709d1c6cf67464-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.857 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.857 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.858 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.858 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.858 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.858 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.858 DEBUG:boto:Data: 2024-04-16T20:05:54.858 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.858 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.858 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.858 DEBUG:boto:Params: {} 2024-04-16T20:05:54.858 DEBUG:boto:Token: None 2024-04-16T20:05:54.859 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g4RUBGtNO7Rq4ELr8jLQkl27uHs= 2024-04-16T20:05:54.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g4RUBGtNO7Rq4ELr8jLQkl27uHs=', 'Content-Length': '0'} 2024-04-16T20:05:54.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c315e04a97f361-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:05:54.861 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.861 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.861 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.861 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.861 DEBUG:boto:Method: GET 2024-04-16T20:05:54.861 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.861 DEBUG:boto:Data: 2024-04-16T20:05:54.861 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.861 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.861 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.862 DEBUG:boto:Params: {} 2024-04-16T20:05:54.862 DEBUG:boto:Token: None 2024-04-16T20:05:54.862 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw= 2024-04-16T20:05:54.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw=', 'Content-Length': '0'} 2024-04-16T20:05:54.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021aa025bba5bf25-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.864 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.864 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.864 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.864 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.864 DEBUG:boto:Method: GET 2024-04-16T20:05:54.865 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.865 DEBUG:boto:Data: 2024-04-16T20:05:54.865 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.865 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.865 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.865 DEBUG:boto:Params: {} 2024-04-16T20:05:54.865 DEBUG:boto:Token: None 2024-04-16T20:05:54.865 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:05:54.865 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw= 2024-04-16T20:05:54.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:INuEcUzsRP99+BFW+U/VIXl3JPw=', 'Content-Length': '0'} 2024-04-16T20:05:54.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063718bc8595b6e0e-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.867 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.867 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.868 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.868 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.868 DEBUG:boto:Method: GET 2024-04-16T20:05:54.868 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.868 DEBUG:boto:Data: 2024-04-16T20:05:54.868 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.868 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.868 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.868 DEBUG:boto:Params: {} 2024-04-16T20:05:54.868 DEBUG:boto:Token: None 2024-04-16T20:05:54.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082721c7a99913c41-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.871 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.871 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.871 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.871 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.872 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.872 DEBUG:boto:Method: GET 2024-04-16T20:05:54.872 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.872 DEBUG:boto:Data: 2024-04-16T20:05:54.872 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.872 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.872 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.872 DEBUG:boto:Params: {} 2024-04-16T20:05:54.872 DEBUG:boto:Token: None 2024-04-16T20:05:54.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b5594c791e06c09-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.876 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.876 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.876 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.876 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.876 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:05:54.876 DEBUG:boto:Data: 2024-04-16T20:05:54.876 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.877 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.877 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.877 DEBUG:boto:Params: {} 2024-04-16T20:05:54.877 DEBUG:boto:Token: None 2024-04-16T20:05:54.877 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17 2024-04-16T20:05:54.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ngma5O4T3egCUKLZ57ySCXpBAzY= 2024-04-16T20:05:54.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ngma5O4T3egCUKLZ57ySCXpBAzY=', 'Content-Length': '0'} 2024-04-16T20:05:54.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a2f40e59fe4e82-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.879 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.879 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.879 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.879 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:05:54.879 DEBUG:boto:Data: 2024-04-16T20:05:54.879 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.879 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.880 DEBUG:boto:Params: {} 2024-04-16T20:05:54.881 DEBUG:boto:Token: None 2024-04-16T20:05:54.881 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17 2024-04-16T20:05:54.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ngma5O4T3egCUKLZ57ySCXpBAzY= 2024-04-16T20:05:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ngma5O4T3egCUKLZ57ySCXpBAzY=', 'Content-Length': '0'} 2024-04-16T20:05:54.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010dfbd24de76eea7-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:05:54.884 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.884 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.884 DEBUG:boto:Method: GET 2024-04-16T20:05:54.884 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:05:54.884 DEBUG:boto:Data: 2024-04-16T20:05:54.884 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.884 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.884 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.884 DEBUG:boto:Params: {} 2024-04-16T20:05:54.884 DEBUG:boto:Token: None 2024-04-16T20:05:54.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17 2024-04-16T20:05:54.885 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IWdN1curdIS4erLcM09N7YYlIA= 2024-04-16T20:05:54.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IWdN1curdIS4erLcM09N7YYlIA=', 'Content-Length': '0'} 2024-04-16T20:05:54.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15f41c804362588-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.930 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:05:54.930 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:05:54.931 DEBUG:boto:Method: GET 2024-04-16T20:05:54.931 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:05:54.931 DEBUG:boto:Data: 2024-04-16T20:05:54.931 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.931 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.931 DEBUG:boto:Params: {} 2024-04-16T20:05:54.931 DEBUG:boto:Token: None 2024-04-16T20:05:54.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj17 2024-04-16T20:05:54.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7IWdN1curdIS4erLcM09N7YYlIA= 2024-04-16T20:05:54.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7IWdN1curdIS4erLcM09N7YYlIA=', 'Content-Length': '0'} 2024-04-16T20:05:54.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b25e923bf453b17-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.934 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.934 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.934 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.934 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.934 DEBUG:boto:Method: GET 2024-04-16T20:05:54.934 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.934 DEBUG:boto:Data: 2024-04-16T20:05:54.934 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.934 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.934 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.935 DEBUG:boto:Params: {} 2024-04-16T20:05:54.935 DEBUG:boto:Token: None 2024-04-16T20:05:54.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c46ce85a83e1a3-00661eda22-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.938 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.938 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:54.938 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:54.938 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.938 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.938 DEBUG:boto:Method: GET 2024-04-16T20:05:54.938 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:05:54.939 DEBUG:boto:Data: 2024-04-16T20:05:54.939 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.939 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.939 DEBUG:boto:Params: {} 2024-04-16T20:05:54.939 DEBUG:boto:Token: None 2024-04-16T20:05:54.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/ 2024-04-16T20:05:54.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w= 2024-04-16T20:05:54.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3jdrN9GPogUmtNBjb4GNUz0w9w=', 'Content-Length': '0'} 2024-04-16T20:05:54.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c46d4aaf34d27d8e-00661eda22-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.943 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:54.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:05:54.943 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:54.943 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:54.944 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.944 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.944 DEBUG:boto:Method: GET 2024-04-16T20:05:54.944 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.944 DEBUG:boto:Data: 2024-04-16T20:05:54.944 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.944 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.944 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.944 DEBUG:boto:Params: {} 2024-04-16T20:05:54.944 DEBUG:boto:Token: None 2024-04-16T20:05:54.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.945 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64= 2024-04-16T20:05:54.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64=', 'Content-Length': '0'} 2024-04-16T20:05:54.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d1a3b7d760822ee-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.947 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:54.947 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:54.948 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.948 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.948 DEBUG:boto:Method: GET 2024-04-16T20:05:54.948 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.948 DEBUG:boto:Data: 2024-04-16T20:05:54.948 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.948 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.948 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.948 DEBUG:boto:Params: {} 2024-04-16T20:05:54.949 DEBUG:boto:Token: None 2024-04-16T20:05:54.949 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.949 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64= 2024-04-16T20:05:54.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64=', 'Content-Length': '0'} 2024-04-16T20:05:54.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003630f754b8b8b874-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.951 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:54.951 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:54.952 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.952 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.952 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.952 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.952 DEBUG:boto:Data: 2024-04-16T20:05:54.952 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.952 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.952 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.952 DEBUG:boto:Params: {} 2024-04-16T20:05:54.953 DEBUG:boto:Token: None 2024-04-16T20:05:54.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aOUxwfH6eOBsMp1CKmZjlKsYyT0= 2024-04-16T20:05:54.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aOUxwfH6eOBsMp1CKmZjlKsYyT0=', 'Content-Length': '0'} 2024-04-16T20:05:54.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321bcf9e3748945d-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.955 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:54.956 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:54.956 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.956 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.956 DEBUG:boto:Method: HEAD 2024-04-16T20:05:54.956 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.956 DEBUG:boto:Data: 2024-04-16T20:05:54.956 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:54.956 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.956 DEBUG:boto:Params: {} 2024-04-16T20:05:54.957 DEBUG:boto:Token: None 2024-04-16T20:05:54.957 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.957 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aOUxwfH6eOBsMp1CKmZjlKsYyT0= 2024-04-16T20:05:54.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aOUxwfH6eOBsMp1CKmZjlKsYyT0=', 'Content-Length': '0'} 2024-04-16T20:05:54.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844623622063b38c-00661eda22-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:54.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:05:54.960 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:54.960 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:54.960 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.960 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.960 DEBUG:boto:Method: GET 2024-04-16T20:05:54.960 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.960 DEBUG:boto:Data: 2024-04-16T20:05:54.960 DEBUG:boto:Headers: {} 2024-04-16T20:05:54.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:54.960 DEBUG:boto:Port: 8000 2024-04-16T20:05:54.960 DEBUG:boto:Params: {} 2024-04-16T20:05:54.961 DEBUG:boto:Token: None 2024-04-16T20:05:54.961 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:54 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:54.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64= 2024-04-16T20:05:54.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:54 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+MNTEuY1dIdfOqrjzbFDl1JAT64=', 'Content-Length': '0'} 2024-04-16T20:05:54.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a15dbfa39e32251-00661eda22-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.006 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:55.006 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:55.007 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:55.007 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:05:55.007 DEBUG:boto:Method: GET 2024-04-16T20:05:55.007 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:55.007 DEBUG:boto:Data: 2024-04-16T20:05:55.007 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.007 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.007 DEBUG:boto:Params: {} 2024-04-16T20:05:55.007 DEBUG:boto:Token: None 2024-04-16T20:05:55.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:05:55.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M/Ppgaxtp/7pUdlQDI3mlI85kuU= 2024-04-16T20:05:55.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M/Ppgaxtp/7pUdlQDI3mlI85kuU=', 'Content-Length': '0'} 2024-04-16T20:05:55.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7546fffb3bc2fd5-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.010 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.010 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.010 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.010 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.010 DEBUG:boto:Method: GET 2024-04-16T20:05:55.010 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.010 DEBUG:boto:Data: 2024-04-16T20:05:55.011 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.011 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.011 DEBUG:boto:Params: {} 2024-04-16T20:05:55.011 DEBUG:boto:Token: None 2024-04-16T20:05:55.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ecc9531779fee1-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.014 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.014 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.014 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.014 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.014 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.014 DEBUG:boto:Method: GET 2024-04-16T20:05:55.014 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.014 DEBUG:boto:Data: 2024-04-16T20:05:55.015 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.015 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.015 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.015 DEBUG:boto:Params: {} 2024-04-16T20:05:55.015 DEBUG:boto:Token: None 2024-04-16T20:05:55.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7f459a136a9934-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.019 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.019 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:55.019 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:55.019 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.019 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:05:55.019 DEBUG:boto:Data: 2024-04-16T20:05:55.019 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.020 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.020 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.020 DEBUG:boto:Params: {} 2024-04-16T20:05:55.020 DEBUG:boto:Token: None 2024-04-16T20:05:55.020 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj19 2024-04-16T20:05:55.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QTCYyraKtycCr4VF1cD71vyAeOY= 2024-04-16T20:05:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QTCYyraKtycCr4VF1cD71vyAeOY=', 'Content-Length': '0'} 2024-04-16T20:05:55.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba8c08a794f379d-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.022 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:55.023 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:55.023 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.023 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:05:55.023 DEBUG:boto:Data: 2024-04-16T20:05:55.023 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.023 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.023 DEBUG:boto:Params: {} 2024-04-16T20:05:55.023 DEBUG:boto:Token: None 2024-04-16T20:05:55.023 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj19 2024-04-16T20:05:55.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QTCYyraKtycCr4VF1cD71vyAeOY= 2024-04-16T20:05:55.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QTCYyraKtycCr4VF1cD71vyAeOY=', 'Content-Length': '0'} 2024-04-16T20:05:55.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa96fbfb9299455e-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:05:55.026 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:55.026 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:55.026 DEBUG:boto:Method: GET 2024-04-16T20:05:55.026 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:05:55.026 DEBUG:boto:Data: 2024-04-16T20:05:55.026 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.026 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.026 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.026 DEBUG:boto:Params: {} 2024-04-16T20:05:55.027 DEBUG:boto:Token: None 2024-04-16T20:05:55.027 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj19 2024-04-16T20:05:55.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BJIi+VtF5YVMjZ3fAF4tmda1/hk= 2024-04-16T20:05:55.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BJIi+VtF5YVMjZ3fAF4tmda1/hk=', 'Content-Length': '0'} 2024-04-16T20:05:55.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097712c7a307bf1cb-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.029 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:05:55.029 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:05:55.030 DEBUG:boto:Method: GET 2024-04-16T20:05:55.030 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:05:55.030 DEBUG:boto:Data: 2024-04-16T20:05:55.030 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.030 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.030 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.030 DEBUG:boto:Params: {} 2024-04-16T20:05:55.030 DEBUG:boto:Token: None 2024-04-16T20:05:55.030 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj19 2024-04-16T20:05:55.030 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BJIi+VtF5YVMjZ3fAF4tmda1/hk= 2024-04-16T20:05:55.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BJIi+VtF5YVMjZ3fAF4tmda1/hk=', 'Content-Length': '0'} 2024-04-16T20:05:55.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e62799a8cb28d28-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.074 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.074 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.074 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.075 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.075 DEBUG:boto:Method: GET 2024-04-16T20:05:55.075 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.075 DEBUG:boto:Data: 2024-04-16T20:05:55.075 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.075 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.075 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.075 DEBUG:boto:Params: {} 2024-04-16T20:05:55.075 DEBUG:boto:Token: None 2024-04-16T20:05:55.075 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b393a98378700fe6-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.078 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.078 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.078 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.079 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.079 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.079 DEBUG:boto:Method: GET 2024-04-16T20:05:55.079 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:05:55.079 DEBUG:boto:Data: 2024-04-16T20:05:55.079 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.079 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.079 DEBUG:boto:Params: {} 2024-04-16T20:05:55.080 DEBUG:boto:Token: None 2024-04-16T20:05:55.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b361d990ecc43f9-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.084 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:05:55.084 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.084 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.084 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.084 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.084 DEBUG:boto:Method: GET 2024-04-16T20:05:55.085 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.085 DEBUG:boto:Data: 2024-04-16T20:05:55.085 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.085 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.085 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.085 DEBUG:boto:Params: {} 2024-04-16T20:05:55.085 DEBUG:boto:Token: None 2024-04-16T20:05:55.085 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8= 2024-04-16T20:05:55.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8=', 'Content-Length': '0'} 2024-04-16T20:05:55.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f95eef09ab1eeb-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.088 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.088 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.088 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.088 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.088 DEBUG:boto:Method: GET 2024-04-16T20:05:55.088 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.088 DEBUG:boto:Data: 2024-04-16T20:05:55.088 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.088 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.088 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.088 DEBUG:boto:Params: {} 2024-04-16T20:05:55.089 DEBUG:boto:Token: None 2024-04-16T20:05:55.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8= 2024-04-16T20:05:55.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8=', 'Content-Length': '0'} 2024-04-16T20:05:55.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad9e31ec236ee57-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.091 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.091 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.091 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.091 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.091 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.092 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.092 DEBUG:boto:Data: 2024-04-16T20:05:55.092 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.092 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.092 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.092 DEBUG:boto:Params: {} 2024-04-16T20:05:55.092 DEBUG:boto:Token: None 2024-04-16T20:05:55.092 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AGkQiPq2xfFJqIqrd5htapDVk2A= 2024-04-16T20:05:55.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AGkQiPq2xfFJqIqrd5htapDVk2A=', 'Content-Length': '0'} 2024-04-16T20:05:55.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c95adf53348f6604-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.094 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.094 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.095 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.095 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.095 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.095 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.095 DEBUG:boto:Data: 2024-04-16T20:05:55.095 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.095 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.095 DEBUG:boto:Params: {} 2024-04-16T20:05:55.095 DEBUG:boto:Token: None 2024-04-16T20:05:55.095 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AGkQiPq2xfFJqIqrd5htapDVk2A= 2024-04-16T20:05:55.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AGkQiPq2xfFJqIqrd5htapDVk2A=', 'Content-Length': '0'} 2024-04-16T20:05:55.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049da7e6327d683f7-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:05:55.098 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.098 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.098 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.098 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.098 DEBUG:boto:Method: GET 2024-04-16T20:05:55.098 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.098 DEBUG:boto:Data: 2024-04-16T20:05:55.098 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.098 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.098 DEBUG:boto:Params: {} 2024-04-16T20:05:55.099 DEBUG:boto:Token: None 2024-04-16T20:05:55.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8= 2024-04-16T20:05:55.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8=', 'Content-Length': '0'} 2024-04-16T20:05:55.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ac5f5f10a1e4b5-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.142 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.143 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.143 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.143 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.143 DEBUG:boto:Method: GET 2024-04-16T20:05:55.143 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.143 DEBUG:boto:Data: 2024-04-16T20:05:55.143 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.143 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.143 DEBUG:boto:Params: {} 2024-04-16T20:05:55.143 DEBUG:boto:Token: None 2024-04-16T20:05:55.144 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:05:55.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8= 2024-04-16T20:05:55.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4DBF9wB+IeY2oRfw+DiTKEz9vv8=', 'Content-Length': '0'} 2024-04-16T20:05:55.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e662fca5bfe976ca-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.146 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.146 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.146 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.146 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.146 DEBUG:boto:Method: GET 2024-04-16T20:05:55.146 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.146 DEBUG:boto:Data: 2024-04-16T20:05:55.147 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.147 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.147 DEBUG:boto:Params: {} 2024-04-16T20:05:55.147 DEBUG:boto:Token: None 2024-04-16T20:05:55.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bfde51ee3a46c81-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.150 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.150 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.150 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.150 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.150 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.151 DEBUG:boto:Method: GET 2024-04-16T20:05:55.151 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.151 DEBUG:boto:Data: 2024-04-16T20:05:55.151 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.151 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.151 DEBUG:boto:Params: {} 2024-04-16T20:05:55.151 DEBUG:boto:Token: None 2024-04-16T20:05:55.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa55e813b299cc0b-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.155 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.155 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.155 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.155 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.155 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:05:55.156 DEBUG:boto:Data: 2024-04-16T20:05:55.156 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.156 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.156 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.156 DEBUG:boto:Params: {} 2024-04-16T20:05:55.156 DEBUG:boto:Token: None 2024-04-16T20:05:55.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22 2024-04-16T20:05:55.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2TrIq1HAPKuhvHgiiLC0TcrX1LE= 2024-04-16T20:05:55.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2TrIq1HAPKuhvHgiiLC0TcrX1LE=', 'Content-Length': '0'} 2024-04-16T20:05:55.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f3ba98eef09678-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.158 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.158 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.158 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.158 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:05:55.159 DEBUG:boto:Data: 2024-04-16T20:05:55.159 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.159 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.159 DEBUG:boto:Params: {} 2024-04-16T20:05:55.159 DEBUG:boto:Token: None 2024-04-16T20:05:55.159 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22 2024-04-16T20:05:55.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2TrIq1HAPKuhvHgiiLC0TcrX1LE= 2024-04-16T20:05:55.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2TrIq1HAPKuhvHgiiLC0TcrX1LE=', 'Content-Length': '0'} 2024-04-16T20:05:55.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ccc463e9b3c138-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:05:55.161 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.162 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.162 DEBUG:boto:Method: GET 2024-04-16T20:05:55.162 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:05:55.162 DEBUG:boto:Data: 2024-04-16T20:05:55.162 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.162 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.162 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.162 DEBUG:boto:Params: {} 2024-04-16T20:05:55.162 DEBUG:boto:Token: None 2024-04-16T20:05:55.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22 2024-04-16T20:05:55.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MHB1c3XzARhwJY+JBE1WixH4OpI= 2024-04-16T20:05:55.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MHB1c3XzARhwJY+JBE1WixH4OpI=', 'Content-Length': '0'} 2024-04-16T20:05:55.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939ee26079c7345b-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.165 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:05:55.165 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:05:55.165 DEBUG:boto:Method: GET 2024-04-16T20:05:55.165 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:05:55.165 DEBUG:boto:Data: 2024-04-16T20:05:55.165 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.165 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.166 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.166 DEBUG:boto:Params: {} 2024-04-16T20:05:55.166 DEBUG:boto:Token: None 2024-04-16T20:05:55.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj22 2024-04-16T20:05:55.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MHB1c3XzARhwJY+JBE1WixH4OpI= 2024-04-16T20:05:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MHB1c3XzARhwJY+JBE1WixH4OpI=', 'Content-Length': '0'} 2024-04-16T20:05:55.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2810f920449f0ac-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.168 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.168 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.168 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.168 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.168 DEBUG:boto:Method: GET 2024-04-16T20:05:55.168 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.168 DEBUG:boto:Data: 2024-04-16T20:05:55.168 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.169 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.169 DEBUG:boto:Params: {} 2024-04-16T20:05:55.169 DEBUG:boto:Token: None 2024-04-16T20:05:55.169 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfcbfb8c52b26e09-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.171 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.172 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.172 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.172 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.172 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.172 DEBUG:boto:Method: GET 2024-04-16T20:05:55.172 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:05:55.172 DEBUG:boto:Data: 2024-04-16T20:05:55.172 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.172 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.172 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.172 DEBUG:boto:Params: {} 2024-04-16T20:05:55.173 DEBUG:boto:Token: None 2024-04-16T20:05:55.173 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.173 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000bd214a7eb50171-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.176 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:05:55.177 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.177 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.177 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.177 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.177 DEBUG:boto:Method: GET 2024-04-16T20:05:55.177 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.177 DEBUG:boto:Data: 2024-04-16T20:05:55.178 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.178 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.178 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.178 DEBUG:boto:Params: {} 2024-04-16T20:05:55.178 DEBUG:boto:Token: None 2024-04-16T20:05:55.178 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.178 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M= 2024-04-16T20:05:55.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M=', 'Content-Length': '0'} 2024-04-16T20:05:55.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000149dfdc74e88c8f7-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.226 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.226 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.227 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.227 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.227 DEBUG:boto:Method: GET 2024-04-16T20:05:55.227 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.227 DEBUG:boto:Data: 2024-04-16T20:05:55.227 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.227 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.227 DEBUG:boto:Params: {} 2024-04-16T20:05:55.227 DEBUG:boto:Token: None 2024-04-16T20:05:55.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M= 2024-04-16T20:05:55.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M=', 'Content-Length': '0'} 2024-04-16T20:05:55.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a619b3cbaa71d2e3-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.230 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.230 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.231 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.231 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.231 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.231 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.231 DEBUG:boto:Data: 2024-04-16T20:05:55.231 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.231 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.231 DEBUG:boto:Params: {} 2024-04-16T20:05:55.231 DEBUG:boto:Token: None 2024-04-16T20:05:55.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:31SJpCqAQJI/vJxUV8FrVMgdYzE= 2024-04-16T20:05:55.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:31SJpCqAQJI/vJxUV8FrVMgdYzE=', 'Content-Length': '0'} 2024-04-16T20:05:55.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4e1e5ca41e6874c-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.234 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.234 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.234 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.234 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.234 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.234 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.234 DEBUG:boto:Data: 2024-04-16T20:05:55.234 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.234 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.235 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.235 DEBUG:boto:Params: {} 2024-04-16T20:05:55.235 DEBUG:boto:Token: None 2024-04-16T20:05:55.235 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:31SJpCqAQJI/vJxUV8FrVMgdYzE= 2024-04-16T20:05:55.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:31SJpCqAQJI/vJxUV8FrVMgdYzE=', 'Content-Length': '0'} 2024-04-16T20:05:55.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61ca45cac669668-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:05:55.238 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.238 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.238 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.238 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.238 DEBUG:boto:Method: GET 2024-04-16T20:05:55.238 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.238 DEBUG:boto:Data: 2024-04-16T20:05:55.239 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.239 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.239 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.239 DEBUG:boto:Params: {} 2024-04-16T20:05:55.239 DEBUG:boto:Token: None 2024-04-16T20:05:55.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.239 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M= 2024-04-16T20:05:55.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M=', 'Content-Length': '0'} 2024-04-16T20:05:55.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bc9aafa9af2f16-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.282 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.282 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.283 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.283 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.283 DEBUG:boto:Method: GET 2024-04-16T20:05:55.283 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.283 DEBUG:boto:Data: 2024-04-16T20:05:55.283 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.283 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.283 DEBUG:boto:Params: {} 2024-04-16T20:05:55.283 DEBUG:boto:Token: None 2024-04-16T20:05:55.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:05:55.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M= 2024-04-16T20:05:55.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eK5BwpOanNzJ1ESNlS1iHSVjO5M=', 'Content-Length': '0'} 2024-04-16T20:05:55.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f616bdd745bbf55a-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.286 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.286 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.286 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.286 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.287 DEBUG:boto:Method: GET 2024-04-16T20:05:55.287 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.287 DEBUG:boto:Data: 2024-04-16T20:05:55.287 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.287 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.287 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.287 DEBUG:boto:Params: {} 2024-04-16T20:05:55.288 DEBUG:boto:Token: None 2024-04-16T20:05:55.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701f89940c863c08-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.291 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.292 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.292 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.292 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.292 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.292 DEBUG:boto:Method: GET 2024-04-16T20:05:55.292 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.292 DEBUG:boto:Data: 2024-04-16T20:05:55.292 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.292 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.292 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.292 DEBUG:boto:Params: {} 2024-04-16T20:05:55.293 DEBUG:boto:Token: None 2024-04-16T20:05:55.293 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076144f1f065d7d52-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.297 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.297 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.297 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.297 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.297 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:05:55.297 DEBUG:boto:Data: 2024-04-16T20:05:55.297 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.297 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.297 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.298 DEBUG:boto:Params: {} 2024-04-16T20:05:55.298 DEBUG:boto:Token: None 2024-04-16T20:05:55.298 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23 2024-04-16T20:05:55.298 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FG+7/KrpI/BcqDr0X8p9nbAbP34= 2024-04-16T20:05:55.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FG+7/KrpI/BcqDr0X8p9nbAbP34=', 'Content-Length': '0'} 2024-04-16T20:05:55.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c23afe5658b8eb-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.300 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.300 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.300 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.300 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:05:55.301 DEBUG:boto:Data: 2024-04-16T20:05:55.301 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.301 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.301 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.301 DEBUG:boto:Params: {} 2024-04-16T20:05:55.301 DEBUG:boto:Token: None 2024-04-16T20:05:55.301 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23 2024-04-16T20:05:55.301 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FG+7/KrpI/BcqDr0X8p9nbAbP34= 2024-04-16T20:05:55.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FG+7/KrpI/BcqDr0X8p9nbAbP34=', 'Content-Length': '0'} 2024-04-16T20:05:55.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d102a075c2e22afd-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:05:55.304 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.304 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.304 DEBUG:boto:Method: GET 2024-04-16T20:05:55.304 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:05:55.304 DEBUG:boto:Data: 2024-04-16T20:05:55.304 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.304 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.304 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.304 DEBUG:boto:Params: {} 2024-04-16T20:05:55.305 DEBUG:boto:Token: None 2024-04-16T20:05:55.305 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23 2024-04-16T20:05:55.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L0duVInFs4FNZqW/wnJNUNovAw8= 2024-04-16T20:05:55.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L0duVInFs4FNZqW/wnJNUNovAw8=', 'Content-Length': '0'} 2024-04-16T20:05:55.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453527c034b60b15-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.350 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:05:55.351 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:05:55.351 DEBUG:boto:Method: GET 2024-04-16T20:05:55.351 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:05:55.351 DEBUG:boto:Data: 2024-04-16T20:05:55.351 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.351 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.351 DEBUG:boto:Params: {} 2024-04-16T20:05:55.351 DEBUG:boto:Token: None 2024-04-16T20:05:55.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj23 2024-04-16T20:05:55.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L0duVInFs4FNZqW/wnJNUNovAw8= 2024-04-16T20:05:55.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L0duVInFs4FNZqW/wnJNUNovAw8=', 'Content-Length': '0'} 2024-04-16T20:05:55.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85ff6f9a722c0b3-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.354 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.354 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.355 DEBUG:boto:Method: GET 2024-04-16T20:05:55.355 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.355 DEBUG:boto:Data: 2024-04-16T20:05:55.355 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.355 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.355 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.355 DEBUG:boto:Params: {} 2024-04-16T20:05:55.355 DEBUG:boto:Token: None 2024-04-16T20:05:55.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f9f1c740fb5f8c-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.398 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.399 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.399 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.399 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.399 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.399 DEBUG:boto:Method: GET 2024-04-16T20:05:55.399 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:05:55.399 DEBUG:boto:Data: 2024-04-16T20:05:55.399 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.400 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.400 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.400 DEBUG:boto:Params: {} 2024-04-16T20:05:55.400 DEBUG:boto:Token: None 2024-04-16T20:05:55.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9b1defb928d86b-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.404 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:05:55.404 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.404 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.404 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.404 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.404 DEBUG:boto:Method: GET 2024-04-16T20:05:55.405 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.405 DEBUG:boto:Data: 2024-04-16T20:05:55.405 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.405 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.405 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.405 DEBUG:boto:Params: {} 2024-04-16T20:05:55.405 DEBUG:boto:Token: None 2024-04-16T20:05:55.405 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w= 2024-04-16T20:05:55.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w=', 'Content-Length': '0'} 2024-04-16T20:05:55.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64ecb6cc4e300d2-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.450 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.450 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.451 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.451 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.451 DEBUG:boto:Method: GET 2024-04-16T20:05:55.451 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.451 DEBUG:boto:Data: 2024-04-16T20:05:55.451 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.451 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.451 DEBUG:boto:Params: {} 2024-04-16T20:05:55.451 DEBUG:boto:Token: None 2024-04-16T20:05:55.451 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w= 2024-04-16T20:05:55.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w=', 'Content-Length': '0'} 2024-04-16T20:05:55.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca7d2892941a47a-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.454 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.454 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.454 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.454 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.454 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.454 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.454 DEBUG:boto:Data: 2024-04-16T20:05:55.454 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.455 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.455 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.455 DEBUG:boto:Params: {} 2024-04-16T20:05:55.455 DEBUG:boto:Token: None 2024-04-16T20:05:55.455 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ca2Ik3xKFrRMw3wDVFmM9S5sB0E= 2024-04-16T20:05:55.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ca2Ik3xKFrRMw3wDVFmM9S5sB0E=', 'Content-Length': '0'} 2024-04-16T20:05:55.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffa2d0de35fe48e-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.457 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.457 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.457 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.457 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.458 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.458 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.458 DEBUG:boto:Data: 2024-04-16T20:05:55.458 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.458 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.458 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.458 DEBUG:boto:Params: {} 2024-04-16T20:05:55.458 DEBUG:boto:Token: None 2024-04-16T20:05:55.458 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ca2Ik3xKFrRMw3wDVFmM9S5sB0E= 2024-04-16T20:05:55.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ca2Ik3xKFrRMw3wDVFmM9S5sB0E=', 'Content-Length': '0'} 2024-04-16T20:05:55.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8831f33af17e9c-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:05:55.461 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.461 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.461 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.461 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.461 DEBUG:boto:Method: GET 2024-04-16T20:05:55.461 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.461 DEBUG:boto:Data: 2024-04-16T20:05:55.461 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.461 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.462 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.462 DEBUG:boto:Params: {} 2024-04-16T20:05:55.462 DEBUG:boto:Token: None 2024-04-16T20:05:55.462 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.462 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w= 2024-04-16T20:05:55.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w=', 'Content-Length': '0'} 2024-04-16T20:05:55.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093640e15ce23b716-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.510 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.510 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.511 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.511 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.511 DEBUG:boto:Method: GET 2024-04-16T20:05:55.511 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.511 DEBUG:boto:Data: 2024-04-16T20:05:55.511 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.511 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.511 DEBUG:boto:Params: {} 2024-04-16T20:05:55.511 DEBUG:boto:Token: None 2024-04-16T20:05:55.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:05:55.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w= 2024-04-16T20:05:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CmOTm+3jZtBKAxcfimcRjDxit/w=', 'Content-Length': '0'} 2024-04-16T20:05:55.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c59d9490c2a7f03-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.514 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.514 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.514 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.514 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.514 DEBUG:boto:Method: GET 2024-04-16T20:05:55.514 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.514 DEBUG:boto:Data: 2024-04-16T20:05:55.514 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.515 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.515 DEBUG:boto:Params: {} 2024-04-16T20:05:55.515 DEBUG:boto:Token: None 2024-04-16T20:05:55.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e720a38284df0b-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.558 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.559 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.559 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.559 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.559 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.559 DEBUG:boto:Method: GET 2024-04-16T20:05:55.559 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.559 DEBUG:boto:Data: 2024-04-16T20:05:55.559 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.559 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.559 DEBUG:boto:Params: {} 2024-04-16T20:05:55.559 DEBUG:boto:Token: None 2024-04-16T20:05:55.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014d8a7dde4877734-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.564 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.564 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.564 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.564 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.564 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:05:55.564 DEBUG:boto:Data: 2024-04-16T20:05:55.564 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.564 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.565 DEBUG:boto:Params: {} 2024-04-16T20:05:55.565 DEBUG:boto:Token: None 2024-04-16T20:05:55.565 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29 2024-04-16T20:05:55.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bjw04BB9YpCrdQ2j0/x/P4ijcck= 2024-04-16T20:05:55.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bjw04BB9YpCrdQ2j0/x/P4ijcck=', 'Content-Length': '0'} 2024-04-16T20:05:55.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52c4aeb0927b68c-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.567 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.567 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.567 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.567 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:05:55.567 DEBUG:boto:Data: 2024-04-16T20:05:55.567 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.568 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.568 DEBUG:boto:Params: {} 2024-04-16T20:05:55.568 DEBUG:boto:Token: None 2024-04-16T20:05:55.568 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29 2024-04-16T20:05:55.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bjw04BB9YpCrdQ2j0/x/P4ijcck= 2024-04-16T20:05:55.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bjw04BB9YpCrdQ2j0/x/P4ijcck=', 'Content-Length': '0'} 2024-04-16T20:05:55.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c62b317d643069-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:05:55.570 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.570 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.570 DEBUG:boto:Method: GET 2024-04-16T20:05:55.571 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:05:55.571 DEBUG:boto:Data: 2024-04-16T20:05:55.571 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.571 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.571 DEBUG:boto:Params: {} 2024-04-16T20:05:55.571 DEBUG:boto:Token: None 2024-04-16T20:05:55.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29 2024-04-16T20:05:55.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wgB6J/t5vweHRYFvDIUb5dvIov8= 2024-04-16T20:05:55.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wgB6J/t5vweHRYFvDIUb5dvIov8=', 'Content-Length': '0'} 2024-04-16T20:05:55.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dbe575adfba27a5-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.614 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:05:55.614 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:05:55.614 DEBUG:boto:Method: GET 2024-04-16T20:05:55.615 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:05:55.615 DEBUG:boto:Data: 2024-04-16T20:05:55.615 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.615 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.615 DEBUG:boto:Params: {} 2024-04-16T20:05:55.615 DEBUG:boto:Token: None 2024-04-16T20:05:55.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj29 2024-04-16T20:05:55.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wgB6J/t5vweHRYFvDIUb5dvIov8= 2024-04-16T20:05:55.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wgB6J/t5vweHRYFvDIUb5dvIov8=', 'Content-Length': '0'} 2024-04-16T20:05:55.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904318dc892064c1-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.618 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.618 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.618 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.618 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.618 DEBUG:boto:Method: GET 2024-04-16T20:05:55.618 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.618 DEBUG:boto:Data: 2024-04-16T20:05:55.618 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.618 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.618 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.618 DEBUG:boto:Params: {} 2024-04-16T20:05:55.619 DEBUG:boto:Token: None 2024-04-16T20:05:55.619 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.619 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ae7098880a9f833-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.662 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.663 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.663 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.663 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.663 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.663 DEBUG:boto:Method: GET 2024-04-16T20:05:55.663 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:05:55.663 DEBUG:boto:Data: 2024-04-16T20:05:55.663 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.664 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.664 DEBUG:boto:Params: {} 2024-04-16T20:05:55.664 DEBUG:boto:Token: None 2024-04-16T20:05:55.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d083d135bdcd54-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.668 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:05:55.668 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.668 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.668 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.668 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.669 DEBUG:boto:Method: GET 2024-04-16T20:05:55.669 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.669 DEBUG:boto:Data: 2024-04-16T20:05:55.669 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.669 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.669 DEBUG:boto:Params: {} 2024-04-16T20:05:55.669 DEBUG:boto:Token: None 2024-04-16T20:05:55.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA= 2024-04-16T20:05:55.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA=', 'Content-Length': '0'} 2024-04-16T20:05:55.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d30438522823b46-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.714 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.715 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.715 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.715 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.715 DEBUG:boto:Method: GET 2024-04-16T20:05:55.715 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.715 DEBUG:boto:Data: 2024-04-16T20:05:55.715 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.715 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.715 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.715 DEBUG:boto:Params: {} 2024-04-16T20:05:55.716 DEBUG:boto:Token: None 2024-04-16T20:05:55.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA= 2024-04-16T20:05:55.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA=', 'Content-Length': '0'} 2024-04-16T20:05:55.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087644bc575c64e02-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.718 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.718 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.718 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.718 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.718 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.719 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.719 DEBUG:boto:Data: 2024-04-16T20:05:55.719 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.719 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.719 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.719 DEBUG:boto:Params: {} 2024-04-16T20:05:55.719 DEBUG:boto:Token: None 2024-04-16T20:05:55.719 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mVP5EHHzJCpIsHe9RcoIpy5YqNY= 2024-04-16T20:05:55.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mVP5EHHzJCpIsHe9RcoIpy5YqNY=', 'Content-Length': '0'} 2024-04-16T20:05:55.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcad68afc6c8e08-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.721 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.721 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.722 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.722 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.722 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.722 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.722 DEBUG:boto:Data: 2024-04-16T20:05:55.722 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.722 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.722 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.722 DEBUG:boto:Params: {} 2024-04-16T20:05:55.723 DEBUG:boto:Token: None 2024-04-16T20:05:55.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mVP5EHHzJCpIsHe9RcoIpy5YqNY= 2024-04-16T20:05:55.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mVP5EHHzJCpIsHe9RcoIpy5YqNY=', 'Content-Length': '0'} 2024-04-16T20:05:55.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097308e2c8ccc44fc-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:05:55.725 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.725 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.726 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.726 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.726 DEBUG:boto:Method: GET 2024-04-16T20:05:55.726 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.726 DEBUG:boto:Data: 2024-04-16T20:05:55.726 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.726 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.726 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.726 DEBUG:boto:Params: {} 2024-04-16T20:05:55.726 DEBUG:boto:Token: None 2024-04-16T20:05:55.727 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA= 2024-04-16T20:05:55.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA=', 'Content-Length': '0'} 2024-04-16T20:05:55.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e731234af95c9cc-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.770 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.770 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.770 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.771 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.771 DEBUG:boto:Method: GET 2024-04-16T20:05:55.771 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.771 DEBUG:boto:Data: 2024-04-16T20:05:55.771 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.771 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.771 DEBUG:boto:Params: {} 2024-04-16T20:05:55.771 DEBUG:boto:Token: None 2024-04-16T20:05:55.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:05:55.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA= 2024-04-16T20:05:55.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09uuoRJPcQgzFod7cw2yTNVgUmA=', 'Content-Length': '0'} 2024-04-16T20:05:55.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce478033b748cd76-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.774 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.774 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.774 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.774 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.774 DEBUG:boto:Method: GET 2024-04-16T20:05:55.774 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.774 DEBUG:boto:Data: 2024-04-16T20:05:55.774 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.774 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.774 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.775 DEBUG:boto:Params: {} 2024-04-16T20:05:55.775 DEBUG:boto:Token: None 2024-04-16T20:05:55.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006144a8e4404729eb-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.818 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.819 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.819 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.819 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.819 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.819 DEBUG:boto:Method: GET 2024-04-16T20:05:55.819 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.819 DEBUG:boto:Data: 2024-04-16T20:05:55.819 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.819 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.819 DEBUG:boto:Params: {} 2024-04-16T20:05:55.820 DEBUG:boto:Token: None 2024-04-16T20:05:55.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072211ce06f5af7b4-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.823 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.824 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.824 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.824 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.824 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:05:55.824 DEBUG:boto:Data: 2024-04-16T20:05:55.824 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.824 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.825 DEBUG:boto:Params: {} 2024-04-16T20:05:55.825 DEBUG:boto:Token: None 2024-04-16T20:05:55.825 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31 2024-04-16T20:05:55.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lGYIRZ//6vE6bZ8dmcplo5yw/mo= 2024-04-16T20:05:55.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lGYIRZ//6vE6bZ8dmcplo5yw/mo=', 'Content-Length': '0'} 2024-04-16T20:05:55.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c81586c4db5986-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.827 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.827 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.827 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.827 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:05:55.827 DEBUG:boto:Data: 2024-04-16T20:05:55.827 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.828 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.828 DEBUG:boto:Params: {} 2024-04-16T20:05:55.828 DEBUG:boto:Token: None 2024-04-16T20:05:55.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31 2024-04-16T20:05:55.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lGYIRZ//6vE6bZ8dmcplo5yw/mo= 2024-04-16T20:05:55.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lGYIRZ//6vE6bZ8dmcplo5yw/mo=', 'Content-Length': '0'} 2024-04-16T20:05:55.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005174a7843d7293f3-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:05:55.830 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.830 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.830 DEBUG:boto:Method: GET 2024-04-16T20:05:55.831 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:05:55.831 DEBUG:boto:Data: 2024-04-16T20:05:55.831 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.831 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.831 DEBUG:boto:Params: {} 2024-04-16T20:05:55.831 DEBUG:boto:Token: None 2024-04-16T20:05:55.831 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31 2024-04-16T20:05:55.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mJiIJ2lLacctNbu3uaEq+Zl9nLE= 2024-04-16T20:05:55.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mJiIJ2lLacctNbu3uaEq+Zl9nLE=', 'Content-Length': '0'} 2024-04-16T20:05:55.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021036c19cb4ef107-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.878 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:05:55.878 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:05:55.878 DEBUG:boto:Method: GET 2024-04-16T20:05:55.879 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:05:55.879 DEBUG:boto:Data: 2024-04-16T20:05:55.879 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.879 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.879 DEBUG:boto:Params: {} 2024-04-16T20:05:55.879 DEBUG:boto:Token: None 2024-04-16T20:05:55.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj31 2024-04-16T20:05:55.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mJiIJ2lLacctNbu3uaEq+Zl9nLE= 2024-04-16T20:05:55.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mJiIJ2lLacctNbu3uaEq+Zl9nLE=', 'Content-Length': '0'} 2024-04-16T20:05:55.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab7536288abec36-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.882 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.882 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.882 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.882 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.882 DEBUG:boto:Method: GET 2024-04-16T20:05:55.882 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.882 DEBUG:boto:Data: 2024-04-16T20:05:55.882 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.882 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.882 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.882 DEBUG:boto:Params: {} 2024-04-16T20:05:55.883 DEBUG:boto:Token: None 2024-04-16T20:05:55.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002838956c14d221c5-00661eda23-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.926 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.927 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:55.927 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:55.927 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.927 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.927 DEBUG:boto:Method: GET 2024-04-16T20:05:55.927 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:05:55.927 DEBUG:boto:Data: 2024-04-16T20:05:55.927 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.927 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.927 DEBUG:boto:Params: {} 2024-04-16T20:05:55.928 DEBUG:boto:Token: None 2024-04-16T20:05:55.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/ 2024-04-16T20:05:55.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4= 2024-04-16T20:05:55.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:01rKf5MpKWMX2jm/yLZhfb3neJ4=', 'Content-Length': '0'} 2024-04-16T20:05:55.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4d80f3b76742ef-00661eda23-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.931 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:55.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:05:55.932 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:55.932 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:55.932 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.932 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.932 DEBUG:boto:Method: GET 2024-04-16T20:05:55.932 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.932 DEBUG:boto:Data: 2024-04-16T20:05:55.932 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.932 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.933 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.933 DEBUG:boto:Params: {} 2024-04-16T20:05:55.933 DEBUG:boto:Token: None 2024-04-16T20:05:55.933 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.933 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY= 2024-04-16T20:05:55.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY=', 'Content-Length': '0'} 2024-04-16T20:05:55.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007399d3ff21f48387-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.978 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:55.978 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:55.978 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.978 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.979 DEBUG:boto:Method: GET 2024-04-16T20:05:55.979 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.979 DEBUG:boto:Data: 2024-04-16T20:05:55.979 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.979 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.979 DEBUG:boto:Params: {} 2024-04-16T20:05:55.979 DEBUG:boto:Token: None 2024-04-16T20:05:55.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY= 2024-04-16T20:05:55.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY=', 'Content-Length': '0'} 2024-04-16T20:05:55.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006df3b48b7dd8527-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.982 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:55.982 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:55.982 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.982 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.982 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.982 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.982 DEBUG:boto:Data: 2024-04-16T20:05:55.983 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.983 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.983 DEBUG:boto:Params: {} 2024-04-16T20:05:55.983 DEBUG:boto:Token: None 2024-04-16T20:05:55.983 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wWkw5hSfO/9UQopecr/Kw0S6rlg= 2024-04-16T20:05:55.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wWkw5hSfO/9UQopecr/Kw0S6rlg=', 'Content-Length': '0'} 2024-04-16T20:05:55.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9e60432f0bc6b0-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.985 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:55.985 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:55.986 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.986 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.986 DEBUG:boto:Method: HEAD 2024-04-16T20:05:55.986 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.986 DEBUG:boto:Data: 2024-04-16T20:05:55.986 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:55.986 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.986 DEBUG:boto:Params: {} 2024-04-16T20:05:55.986 DEBUG:boto:Token: None 2024-04-16T20:05:55.986 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wWkw5hSfO/9UQopecr/Kw0S6rlg= 2024-04-16T20:05:55.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wWkw5hSfO/9UQopecr/Kw0S6rlg=', 'Content-Length': '0'} 2024-04-16T20:05:55.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f335027fed9e049a-00661eda23-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:55.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:05:55.989 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:55.989 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:55.989 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.989 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.989 DEBUG:boto:Method: GET 2024-04-16T20:05:55.989 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.989 DEBUG:boto:Data: 2024-04-16T20:05:55.989 DEBUG:boto:Headers: {} 2024-04-16T20:05:55.989 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:55.989 DEBUG:boto:Port: 8000 2024-04-16T20:05:55.990 DEBUG:boto:Params: {} 2024-04-16T20:05:55.990 DEBUG:boto:Token: None 2024-04-16T20:05:55.990 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:55 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:55.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY= 2024-04-16T20:05:55.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:55 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eENhhkG9DPuDljjZAdIXmxPHstY=', 'Content-Length': '0'} 2024-04-16T20:05:55.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a558f1f0cf86b7cf-00661eda23-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.034 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:56.034 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:56.034 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:56.035 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:05:56.035 DEBUG:boto:Method: GET 2024-04-16T20:05:56.035 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:56.035 DEBUG:boto:Data: 2024-04-16T20:05:56.035 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.035 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.035 DEBUG:boto:Params: {} 2024-04-16T20:05:56.035 DEBUG:boto:Token: None 2024-04-16T20:05:56.035 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:05:56.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JhPjloSVcaGbZWN7AQFeYF7EVcM= 2024-04-16T20:05:56.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JhPjloSVcaGbZWN7AQFeYF7EVcM=', 'Content-Length': '0'} 2024-04-16T20:05:56.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccd977907f3205b-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.038 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.038 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.038 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.038 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.039 DEBUG:boto:Method: GET 2024-04-16T20:05:56.039 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.039 DEBUG:boto:Data: 2024-04-16T20:05:56.039 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.039 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.039 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.039 DEBUG:boto:Params: {} 2024-04-16T20:05:56.039 DEBUG:boto:Token: None 2024-04-16T20:05:56.039 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b499a252d5f917-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.082 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.083 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.083 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.083 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.083 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.083 DEBUG:boto:Method: GET 2024-04-16T20:05:56.083 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.083 DEBUG:boto:Data: 2024-04-16T20:05:56.083 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.083 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.084 DEBUG:boto:Params: {} 2024-04-16T20:05:56.084 DEBUG:boto:Token: None 2024-04-16T20:05:56.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2f90533a9a8237-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.087 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.088 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:56.088 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:56.088 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.088 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:05:56.088 DEBUG:boto:Data: 2024-04-16T20:05:56.088 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.088 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.088 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.088 DEBUG:boto:Params: {} 2024-04-16T20:05:56.088 DEBUG:boto:Token: None 2024-04-16T20:05:56.088 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj33 2024-04-16T20:05:56.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XpFt9r4t0pnySx00mdrfiRfUhgs= 2024-04-16T20:05:56.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XpFt9r4t0pnySx00mdrfiRfUhgs=', 'Content-Length': '0'} 2024-04-16T20:05:56.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c0c274453a2709-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.091 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:56.091 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:56.091 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.091 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:05:56.091 DEBUG:boto:Data: 2024-04-16T20:05:56.091 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.091 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.092 DEBUG:boto:Params: {} 2024-04-16T20:05:56.092 DEBUG:boto:Token: None 2024-04-16T20:05:56.092 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj33 2024-04-16T20:05:56.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XpFt9r4t0pnySx00mdrfiRfUhgs= 2024-04-16T20:05:56.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XpFt9r4t0pnySx00mdrfiRfUhgs=', 'Content-Length': '0'} 2024-04-16T20:05:56.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc62b9f9f9b77eb0-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:05:56.094 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:56.094 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:56.094 DEBUG:boto:Method: GET 2024-04-16T20:05:56.095 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:05:56.095 DEBUG:boto:Data: 2024-04-16T20:05:56.095 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.095 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.095 DEBUG:boto:Params: {} 2024-04-16T20:05:56.095 DEBUG:boto:Token: None 2024-04-16T20:05:56.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj33 2024-04-16T20:05:56.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gsu+tjae2NWgw9KNGjLMIuPMdH4= 2024-04-16T20:05:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gsu+tjae2NWgw9KNGjLMIuPMdH4=', 'Content-Length': '0'} 2024-04-16T20:05:56.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a6c3f0027a24e9-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.138 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:05:56.138 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:05:56.138 DEBUG:boto:Method: GET 2024-04-16T20:05:56.139 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:05:56.139 DEBUG:boto:Data: 2024-04-16T20:05:56.139 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.139 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.139 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.139 DEBUG:boto:Params: {} 2024-04-16T20:05:56.139 DEBUG:boto:Token: None 2024-04-16T20:05:56.139 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj33 2024-04-16T20:05:56.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gsu+tjae2NWgw9KNGjLMIuPMdH4= 2024-04-16T20:05:56.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gsu+tjae2NWgw9KNGjLMIuPMdH4=', 'Content-Length': '0'} 2024-04-16T20:05:56.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc5629dd286c16d-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.142 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.142 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.142 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.142 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.142 DEBUG:boto:Method: GET 2024-04-16T20:05:56.142 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.142 DEBUG:boto:Data: 2024-04-16T20:05:56.142 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.143 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.143 DEBUG:boto:Params: {} 2024-04-16T20:05:56.143 DEBUG:boto:Token: None 2024-04-16T20:05:56.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a3ff5f0516c8cc-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.186 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.187 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.187 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.187 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.187 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.187 DEBUG:boto:Method: GET 2024-04-16T20:05:56.187 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:05:56.187 DEBUG:boto:Data: 2024-04-16T20:05:56.187 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.187 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.187 DEBUG:boto:Params: {} 2024-04-16T20:05:56.188 DEBUG:boto:Token: None 2024-04-16T20:05:56.188 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009447553a5d3df60c-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.191 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:05:56.192 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.192 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.192 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.192 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.192 DEBUG:boto:Method: GET 2024-04-16T20:05:56.192 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.192 DEBUG:boto:Data: 2024-04-16T20:05:56.192 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.192 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.193 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.193 DEBUG:boto:Params: {} 2024-04-16T20:05:56.193 DEBUG:boto:Token: None 2024-04-16T20:05:56.193 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y= 2024-04-16T20:05:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y=', 'Content-Length': '0'} 2024-04-16T20:05:56.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281993f52e1bd03f-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.238 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.238 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.238 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.239 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.239 DEBUG:boto:Method: GET 2024-04-16T20:05:56.239 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.239 DEBUG:boto:Data: 2024-04-16T20:05:56.239 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.239 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.239 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.239 DEBUG:boto:Params: {} 2024-04-16T20:05:56.239 DEBUG:boto:Token: None 2024-04-16T20:05:56.239 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y= 2024-04-16T20:05:56.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y=', 'Content-Length': '0'} 2024-04-16T20:05:56.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c7b381c696c930-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.242 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.242 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.242 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.242 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.242 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.242 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.242 DEBUG:boto:Data: 2024-04-16T20:05:56.242 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.243 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.243 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.243 DEBUG:boto:Params: {} 2024-04-16T20:05:56.243 DEBUG:boto:Token: None 2024-04-16T20:05:56.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cVZXFq7hPG8pSn+2xFDIebC2XHI= 2024-04-16T20:05:56.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cVZXFq7hPG8pSn+2xFDIebC2XHI=', 'Content-Length': '0'} 2024-04-16T20:05:56.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2efcb38ee796db-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.245 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.245 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.245 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.245 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.246 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.246 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.246 DEBUG:boto:Data: 2024-04-16T20:05:56.246 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.246 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.246 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.246 DEBUG:boto:Params: {} 2024-04-16T20:05:56.246 DEBUG:boto:Token: None 2024-04-16T20:05:56.246 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cVZXFq7hPG8pSn+2xFDIebC2XHI= 2024-04-16T20:05:56.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cVZXFq7hPG8pSn+2xFDIebC2XHI=', 'Content-Length': '0'} 2024-04-16T20:05:56.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64f3c9abeeea497-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:05:56.249 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.249 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.249 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.249 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.249 DEBUG:boto:Method: GET 2024-04-16T20:05:56.249 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.249 DEBUG:boto:Data: 2024-04-16T20:05:56.249 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.249 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.249 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.250 DEBUG:boto:Params: {} 2024-04-16T20:05:56.250 DEBUG:boto:Token: None 2024-04-16T20:05:56.250 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y= 2024-04-16T20:05:56.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y=', 'Content-Length': '0'} 2024-04-16T20:05:56.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097051371406d19c2-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.298 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.298 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.298 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.299 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.299 DEBUG:boto:Method: GET 2024-04-16T20:05:56.299 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.299 DEBUG:boto:Data: 2024-04-16T20:05:56.299 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.299 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.299 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.299 DEBUG:boto:Params: {} 2024-04-16T20:05:56.299 DEBUG:boto:Token: None 2024-04-16T20:05:56.299 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:05:56.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y= 2024-04-16T20:05:56.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lfG7rjKPP7TELg4wqNNiBAmjJ1Y=', 'Content-Length': '0'} 2024-04-16T20:05:56.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284a32169d14aaa0-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.302 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.302 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.302 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.302 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.302 DEBUG:boto:Method: GET 2024-04-16T20:05:56.302 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.303 DEBUG:boto:Data: 2024-04-16T20:05:56.303 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.303 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.303 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.303 DEBUG:boto:Params: {} 2024-04-16T20:05:56.303 DEBUG:boto:Token: None 2024-04-16T20:05:56.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f626aaee70a94310-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.346 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.347 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.347 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.347 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.347 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.347 DEBUG:boto:Method: GET 2024-04-16T20:05:56.347 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.347 DEBUG:boto:Data: 2024-04-16T20:05:56.347 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.347 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.348 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.348 DEBUG:boto:Params: {} 2024-04-16T20:05:56.348 DEBUG:boto:Token: None 2024-04-16T20:05:56.348 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba91d63f8ecd396a-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.352 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.352 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.352 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.352 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.352 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:05:56.352 DEBUG:boto:Data: 2024-04-16T20:05:56.352 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.352 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.352 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.352 DEBUG:boto:Params: {} 2024-04-16T20:05:56.353 DEBUG:boto:Token: None 2024-04-16T20:05:56.353 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34 2024-04-16T20:05:56.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wHMdiBevRgZ0dtCFQBDU5X2tvkQ= 2024-04-16T20:05:56.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wHMdiBevRgZ0dtCFQBDU5X2tvkQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c6adc8b45f33e3-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.355 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.355 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.355 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.355 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:05:56.355 DEBUG:boto:Data: 2024-04-16T20:05:56.355 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.355 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.356 DEBUG:boto:Params: {} 2024-04-16T20:05:56.356 DEBUG:boto:Token: None 2024-04-16T20:05:56.356 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34 2024-04-16T20:05:56.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wHMdiBevRgZ0dtCFQBDU5X2tvkQ= 2024-04-16T20:05:56.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wHMdiBevRgZ0dtCFQBDU5X2tvkQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa138f6ecc1115bc-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:05:56.358 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.358 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.358 DEBUG:boto:Method: GET 2024-04-16T20:05:56.358 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:05:56.358 DEBUG:boto:Data: 2024-04-16T20:05:56.358 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.358 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.358 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.359 DEBUG:boto:Params: {} 2024-04-16T20:05:56.359 DEBUG:boto:Token: None 2024-04-16T20:05:56.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34 2024-04-16T20:05:56.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NmWK9UxSnVDlQLPrglUyZBvHQZ8= 2024-04-16T20:05:56.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmWK9UxSnVDlQLPrglUyZBvHQZ8=', 'Content-Length': '0'} 2024-04-16T20:05:56.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8f7276e90abcd0-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.402 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:05:56.403 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:05:56.403 DEBUG:boto:Method: GET 2024-04-16T20:05:56.403 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:05:56.403 DEBUG:boto:Data: 2024-04-16T20:05:56.403 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.403 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.403 DEBUG:boto:Params: {} 2024-04-16T20:05:56.404 DEBUG:boto:Token: None 2024-04-16T20:05:56.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj34 2024-04-16T20:05:56.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NmWK9UxSnVDlQLPrglUyZBvHQZ8= 2024-04-16T20:05:56.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmWK9UxSnVDlQLPrglUyZBvHQZ8=', 'Content-Length': '0'} 2024-04-16T20:05:56.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ab796a6eb3f4b4-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.406 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.407 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.407 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.407 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.407 DEBUG:boto:Method: GET 2024-04-16T20:05:56.407 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.407 DEBUG:boto:Data: 2024-04-16T20:05:56.407 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.407 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.407 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.407 DEBUG:boto:Params: {} 2024-04-16T20:05:56.408 DEBUG:boto:Token: None 2024-04-16T20:05:56.408 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7223d0e95276e70-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.454 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.455 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.455 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.455 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.455 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.455 DEBUG:boto:Method: GET 2024-04-16T20:05:56.456 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:05:56.456 DEBUG:boto:Data: 2024-04-16T20:05:56.456 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.456 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.456 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.456 DEBUG:boto:Params: {} 2024-04-16T20:05:56.456 DEBUG:boto:Token: None 2024-04-16T20:05:56.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.457 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff5542bc7004eee-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.460 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:05:56.461 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.461 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.461 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.461 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.461 DEBUG:boto:Method: GET 2024-04-16T20:05:56.461 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.461 DEBUG:boto:Data: 2024-04-16T20:05:56.461 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.462 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.462 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.462 DEBUG:boto:Params: {} 2024-04-16T20:05:56.462 DEBUG:boto:Token: None 2024-04-16T20:05:56.462 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.462 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY= 2024-04-16T20:05:56.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY=', 'Content-Length': '0'} 2024-04-16T20:05:56.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d9bff4e42be0cbf-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.506 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.506 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.506 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.507 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.507 DEBUG:boto:Method: GET 2024-04-16T20:05:56.507 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.507 DEBUG:boto:Data: 2024-04-16T20:05:56.507 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.507 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.507 DEBUG:boto:Params: {} 2024-04-16T20:05:56.507 DEBUG:boto:Token: None 2024-04-16T20:05:56.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY= 2024-04-16T20:05:56.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY=', 'Content-Length': '0'} 2024-04-16T20:05:56.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69e36a45fb5139f-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.510 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.510 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.510 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.510 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.510 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.510 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.510 DEBUG:boto:Data: 2024-04-16T20:05:56.511 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.511 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.511 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.511 DEBUG:boto:Params: {} 2024-04-16T20:05:56.511 DEBUG:boto:Token: None 2024-04-16T20:05:56.511 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:En4A3hxDnOyA/4rrntaeZ74sMUg= 2024-04-16T20:05:56.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:En4A3hxDnOyA/4rrntaeZ74sMUg=', 'Content-Length': '0'} 2024-04-16T20:05:56.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087011b89976ba844-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.513 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.513 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.513 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.514 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.514 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.514 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.514 DEBUG:boto:Data: 2024-04-16T20:05:56.514 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.514 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.514 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.514 DEBUG:boto:Params: {} 2024-04-16T20:05:56.515 DEBUG:boto:Token: None 2024-04-16T20:05:56.515 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:En4A3hxDnOyA/4rrntaeZ74sMUg= 2024-04-16T20:05:56.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:En4A3hxDnOyA/4rrntaeZ74sMUg=', 'Content-Length': '0'} 2024-04-16T20:05:56.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a743de9c1520a8-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:05:56.517 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.517 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.517 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.517 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.518 DEBUG:boto:Method: GET 2024-04-16T20:05:56.518 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.518 DEBUG:boto:Data: 2024-04-16T20:05:56.518 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.518 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.518 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.518 DEBUG:boto:Params: {} 2024-04-16T20:05:56.518 DEBUG:boto:Token: None 2024-04-16T20:05:56.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY= 2024-04-16T20:05:56.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY=', 'Content-Length': '0'} 2024-04-16T20:05:56.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc2388e611575ae5-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.562 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.562 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.563 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.563 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.563 DEBUG:boto:Method: GET 2024-04-16T20:05:56.563 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.563 DEBUG:boto:Data: 2024-04-16T20:05:56.563 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.563 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.563 DEBUG:boto:Params: {} 2024-04-16T20:05:56.563 DEBUG:boto:Token: None 2024-04-16T20:05:56.564 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:05:56.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY= 2024-04-16T20:05:56.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wAr/tP8iQbQP7lv3S5VWeg8bmgY=', 'Content-Length': '0'} 2024-04-16T20:05:56.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5977e3ae3652373-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.566 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.566 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.566 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.566 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.566 DEBUG:boto:Method: GET 2024-04-16T20:05:56.566 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.566 DEBUG:boto:Data: 2024-04-16T20:05:56.567 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.567 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.567 DEBUG:boto:Params: {} 2024-04-16T20:05:56.567 DEBUG:boto:Token: None 2024-04-16T20:05:56.567 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da03e5d503bfdeaf-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.610 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.611 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.611 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.611 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.611 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.611 DEBUG:boto:Method: GET 2024-04-16T20:05:56.611 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.611 DEBUG:boto:Data: 2024-04-16T20:05:56.611 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.612 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.612 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.612 DEBUG:boto:Params: {} 2024-04-16T20:05:56.612 DEBUG:boto:Token: None 2024-04-16T20:05:56.612 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3314222a13932d-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.616 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.616 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.616 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.616 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.616 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:05:56.617 DEBUG:boto:Data: 2024-04-16T20:05:56.617 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.617 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.617 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.617 DEBUG:boto:Params: {} 2024-04-16T20:05:56.617 DEBUG:boto:Token: None 2024-04-16T20:05:56.617 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38 2024-04-16T20:05:56.617 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VLZLqR/OqqqMycIHyMwKaq8/CGw= 2024-04-16T20:05:56.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VLZLqR/OqqqMycIHyMwKaq8/CGw=', 'Content-Length': '0'} 2024-04-16T20:05:56.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f55e765c43fe057-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.619 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.619 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.620 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.620 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:05:56.620 DEBUG:boto:Data: 2024-04-16T20:05:56.620 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.620 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.620 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.620 DEBUG:boto:Params: {} 2024-04-16T20:05:56.620 DEBUG:boto:Token: None 2024-04-16T20:05:56.620 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38 2024-04-16T20:05:56.621 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VLZLqR/OqqqMycIHyMwKaq8/CGw= 2024-04-16T20:05:56.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VLZLqR/OqqqMycIHyMwKaq8/CGw=', 'Content-Length': '0'} 2024-04-16T20:05:56.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001921d88415947d95-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:05:56.623 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.623 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.623 DEBUG:boto:Method: GET 2024-04-16T20:05:56.623 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:05:56.623 DEBUG:boto:Data: 2024-04-16T20:05:56.623 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.624 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.624 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.624 DEBUG:boto:Params: {} 2024-04-16T20:05:56.624 DEBUG:boto:Token: None 2024-04-16T20:05:56.624 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38 2024-04-16T20:05:56.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qyIbMn+sv22/WkLtekzdLcOa3VI= 2024-04-16T20:05:56.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qyIbMn+sv22/WkLtekzdLcOa3VI=', 'Content-Length': '0'} 2024-04-16T20:05:56.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002babe097939d2262-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.666 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:05:56.666 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:05:56.667 DEBUG:boto:Method: GET 2024-04-16T20:05:56.667 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:05:56.667 DEBUG:boto:Data: 2024-04-16T20:05:56.667 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.667 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.667 DEBUG:boto:Params: {} 2024-04-16T20:05:56.667 DEBUG:boto:Token: None 2024-04-16T20:05:56.667 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj38 2024-04-16T20:05:56.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qyIbMn+sv22/WkLtekzdLcOa3VI= 2024-04-16T20:05:56.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qyIbMn+sv22/WkLtekzdLcOa3VI=', 'Content-Length': '0'} 2024-04-16T20:05:56.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a46aed44408421-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.670 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.670 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.670 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.670 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.671 DEBUG:boto:Method: GET 2024-04-16T20:05:56.671 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.671 DEBUG:boto:Data: 2024-04-16T20:05:56.671 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.671 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.671 DEBUG:boto:Params: {} 2024-04-16T20:05:56.671 DEBUG:boto:Token: None 2024-04-16T20:05:56.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca23d642c5246b3-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.714 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.715 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.715 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.715 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.715 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.715 DEBUG:boto:Method: GET 2024-04-16T20:05:56.715 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:05:56.715 DEBUG:boto:Data: 2024-04-16T20:05:56.716 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.716 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.716 DEBUG:boto:Params: {} 2024-04-16T20:05:56.716 DEBUG:boto:Token: None 2024-04-16T20:05:56.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb2b92600682d0a3-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.720 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:05:56.721 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.721 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.721 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.721 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.721 DEBUG:boto:Method: GET 2024-04-16T20:05:56.721 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.721 DEBUG:boto:Data: 2024-04-16T20:05:56.721 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.721 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.721 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.721 DEBUG:boto:Params: {} 2024-04-16T20:05:56.722 DEBUG:boto:Token: None 2024-04-16T20:05:56.722 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ= 2024-04-16T20:05:56.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b77dba1863ef6b-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.766 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.766 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.767 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.767 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.767 DEBUG:boto:Method: GET 2024-04-16T20:05:56.767 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.767 DEBUG:boto:Data: 2024-04-16T20:05:56.767 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.767 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.767 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.767 DEBUG:boto:Params: {} 2024-04-16T20:05:56.767 DEBUG:boto:Token: None 2024-04-16T20:05:56.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ= 2024-04-16T20:05:56.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000807214a1818aea7a-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.770 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.770 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.770 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.770 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.771 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.771 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.771 DEBUG:boto:Data: 2024-04-16T20:05:56.771 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.771 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.771 DEBUG:boto:Params: {} 2024-04-16T20:05:56.772 DEBUG:boto:Token: None 2024-04-16T20:05:56.772 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I4ByOK/DVex8mkF/UrUMKuOAaCQ= 2024-04-16T20:05:56.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I4ByOK/DVex8mkF/UrUMKuOAaCQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a01749143149012-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.774 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.774 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.774 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.774 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.775 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.775 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.775 DEBUG:boto:Data: 2024-04-16T20:05:56.775 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.775 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.775 DEBUG:boto:Params: {} 2024-04-16T20:05:56.775 DEBUG:boto:Token: None 2024-04-16T20:05:56.775 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I4ByOK/DVex8mkF/UrUMKuOAaCQ= 2024-04-16T20:05:56.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I4ByOK/DVex8mkF/UrUMKuOAaCQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f883b6ab3c90611f-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:05:56.778 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.778 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.778 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.778 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.778 DEBUG:boto:Method: GET 2024-04-16T20:05:56.778 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.778 DEBUG:boto:Data: 2024-04-16T20:05:56.778 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.779 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.779 DEBUG:boto:Params: {} 2024-04-16T20:05:56.779 DEBUG:boto:Token: None 2024-04-16T20:05:56.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ= 2024-04-16T20:05:56.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4970415fac136c9-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.822 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.822 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.823 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.823 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.823 DEBUG:boto:Method: GET 2024-04-16T20:05:56.823 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.823 DEBUG:boto:Data: 2024-04-16T20:05:56.823 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.823 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.823 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.823 DEBUG:boto:Params: {} 2024-04-16T20:05:56.823 DEBUG:boto:Token: None 2024-04-16T20:05:56.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:05:56.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ= 2024-04-16T20:05:56.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BLVGzIbNgCSdSqBJjPVBPkcdsUQ=', 'Content-Length': '0'} 2024-04-16T20:05:56.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b92f6843ca11bacc-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.826 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.826 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.826 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.826 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.826 DEBUG:boto:Method: GET 2024-04-16T20:05:56.826 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.827 DEBUG:boto:Data: 2024-04-16T20:05:56.827 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.827 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.827 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.827 DEBUG:boto:Params: {} 2024-04-16T20:05:56.827 DEBUG:boto:Token: None 2024-04-16T20:05:56.827 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065af79fc3026e763-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.870 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.871 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.871 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.871 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.871 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.871 DEBUG:boto:Method: GET 2024-04-16T20:05:56.871 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.871 DEBUG:boto:Data: 2024-04-16T20:05:56.871 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.872 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.872 DEBUG:boto:Params: {} 2024-04-16T20:05:56.872 DEBUG:boto:Token: None 2024-04-16T20:05:56.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782b0dfac49b3f0f-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.875 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.876 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.876 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.876 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.876 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:05:56.876 DEBUG:boto:Data: 2024-04-16T20:05:56.876 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.876 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.876 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.876 DEBUG:boto:Params: {} 2024-04-16T20:05:56.876 DEBUG:boto:Token: None 2024-04-16T20:05:56.877 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44 2024-04-16T20:05:56.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We7HvU2fjc0gzCNC+A5Eafs/2gA= 2024-04-16T20:05:56.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We7HvU2fjc0gzCNC+A5Eafs/2gA=', 'Content-Length': '0'} 2024-04-16T20:05:56.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e9283f72ffdcea-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.879 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.879 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.879 DEBUG:boto:Method: HEAD 2024-04-16T20:05:56.879 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:05:56.880 DEBUG:boto:Data: 2024-04-16T20:05:56.880 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.880 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.880 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.880 DEBUG:boto:Params: {} 2024-04-16T20:05:56.880 DEBUG:boto:Token: None 2024-04-16T20:05:56.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44 2024-04-16T20:05:56.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We7HvU2fjc0gzCNC+A5Eafs/2gA= 2024-04-16T20:05:56.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We7HvU2fjc0gzCNC+A5Eafs/2gA=', 'Content-Length': '0'} 2024-04-16T20:05:56.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2e376be9b8799d7-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:05:56.883 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.883 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.883 DEBUG:boto:Method: GET 2024-04-16T20:05:56.883 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:05:56.883 DEBUG:boto:Data: 2024-04-16T20:05:56.883 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.883 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.883 DEBUG:boto:Params: {} 2024-04-16T20:05:56.883 DEBUG:boto:Token: None 2024-04-16T20:05:56.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44 2024-04-16T20:05:56.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vu8ZlMy8Nz849gTAKVm3QeJhTAk= 2024-04-16T20:05:56.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vu8ZlMy8Nz849gTAKVm3QeJhTAk=', 'Content-Length': '0'} 2024-04-16T20:05:56.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d99f49276a85beb-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.926 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:05:56.926 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:05:56.927 DEBUG:boto:Method: GET 2024-04-16T20:05:56.927 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:05:56.927 DEBUG:boto:Data: 2024-04-16T20:05:56.927 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.927 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.927 DEBUG:boto:Params: {} 2024-04-16T20:05:56.927 DEBUG:boto:Token: None 2024-04-16T20:05:56.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj44 2024-04-16T20:05:56.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vu8ZlMy8Nz849gTAKVm3QeJhTAk= 2024-04-16T20:05:56.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vu8ZlMy8Nz849gTAKVm3QeJhTAk=', 'Content-Length': '0'} 2024-04-16T20:05:56.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f939fff6b8c3342d-00661eda24-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.930 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.930 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.930 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.930 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.930 DEBUG:boto:Method: GET 2024-04-16T20:05:56.931 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.931 DEBUG:boto:Data: 2024-04-16T20:05:56.931 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.931 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.931 DEBUG:boto:Params: {} 2024-04-16T20:05:56.931 DEBUG:boto:Token: None 2024-04-16T20:05:56.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b65261fcdbfd369-00661eda24-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.974 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.975 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:56.975 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:56.975 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.975 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.975 DEBUG:boto:Method: GET 2024-04-16T20:05:56.975 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:05:56.976 DEBUG:boto:Data: 2024-04-16T20:05:56.976 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.976 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:56.976 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.976 DEBUG:boto:Params: {} 2024-04-16T20:05:56.976 DEBUG:boto:Token: None 2024-04-16T20:05:56.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/ 2024-04-16T20:05:56.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI= 2024-04-16T20:05:56.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mIZcAnObUW+fYXf43vBafaEUJKI=', 'Content-Length': '0'} 2024-04-16T20:05:56.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000770fb355f3509b3a-00661eda24-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:56.980 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:56.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:05:56.981 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:56.981 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:56.981 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:56.981 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:56.981 DEBUG:boto:Method: GET 2024-04-16T20:05:56.981 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:56.981 DEBUG:boto:Data: 2024-04-16T20:05:56.981 DEBUG:boto:Headers: {} 2024-04-16T20:05:56.981 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:56.982 DEBUG:boto:Port: 8000 2024-04-16T20:05:56.982 DEBUG:boto:Params: {} 2024-04-16T20:05:56.982 DEBUG:boto:Token: None 2024-04-16T20:05:56.982 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:56 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:56.982 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LpWve2FE/ky72ShfQnv12QjAG4U= 2024-04-16T20:05:56.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:56 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LpWve2FE/ky72ShfQnv12QjAG4U=', 'Content-Length': '0'} 2024-04-16T20:05:56.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fffc8b283a5ba58-00661eda24-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.026 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.026 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.027 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.027 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.027 DEBUG:boto:Method: GET 2024-04-16T20:05:57.027 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.027 DEBUG:boto:Data: 2024-04-16T20:05:57.027 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.027 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.027 DEBUG:boto:Params: {} 2024-04-16T20:05:57.027 DEBUG:boto:Token: None 2024-04-16T20:05:57.027 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4= 2024-04-16T20:05:57.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4=', 'Content-Length': '0'} 2024-04-16T20:05:57.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128b901fc7185dec-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.030 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.030 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.030 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.030 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.030 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.031 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.031 DEBUG:boto:Data: 2024-04-16T20:05:57.031 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.031 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.031 DEBUG:boto:Params: {} 2024-04-16T20:05:57.031 DEBUG:boto:Token: None 2024-04-16T20:05:57.031 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XEs3SBIW5ttz/rIAjhY0GksLEUs= 2024-04-16T20:05:57.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XEs3SBIW5ttz/rIAjhY0GksLEUs=', 'Content-Length': '0'} 2024-04-16T20:05:57.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3eb25d07655f938-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.033 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.033 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.033 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.034 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.034 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.034 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.034 DEBUG:boto:Data: 2024-04-16T20:05:57.034 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.034 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.034 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.034 DEBUG:boto:Params: {} 2024-04-16T20:05:57.034 DEBUG:boto:Token: None 2024-04-16T20:05:57.034 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XEs3SBIW5ttz/rIAjhY0GksLEUs= 2024-04-16T20:05:57.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XEs3SBIW5ttz/rIAjhY0GksLEUs=', 'Content-Length': '0'} 2024-04-16T20:05:57.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e7909ca9f1927b-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:05:57.037 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.037 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.037 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.037 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.037 DEBUG:boto:Method: GET 2024-04-16T20:05:57.037 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.037 DEBUG:boto:Data: 2024-04-16T20:05:57.037 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.038 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.038 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.038 DEBUG:boto:Params: {} 2024-04-16T20:05:57.038 DEBUG:boto:Token: None 2024-04-16T20:05:57.038 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4= 2024-04-16T20:05:57.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4=', 'Content-Length': '0'} 2024-04-16T20:05:57.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f4695b3291ba3a-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.082 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.082 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.083 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.083 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.083 DEBUG:boto:Method: GET 2024-04-16T20:05:57.083 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.083 DEBUG:boto:Data: 2024-04-16T20:05:57.083 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.083 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.083 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.083 DEBUG:boto:Params: {} 2024-04-16T20:05:57.083 DEBUG:boto:Token: None 2024-04-16T20:05:57.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:05:57.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4= 2024-04-16T20:05:57.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MJRBxleV26XYt32WlLgPMAqBjd4=', 'Content-Length': '0'} 2024-04-16T20:05:57.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3e60b5bf873162-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.086 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.086 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.086 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.087 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.087 DEBUG:boto:Method: GET 2024-04-16T20:05:57.087 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.087 DEBUG:boto:Data: 2024-04-16T20:05:57.087 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.087 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.087 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.087 DEBUG:boto:Params: {} 2024-04-16T20:05:57.087 DEBUG:boto:Token: None 2024-04-16T20:05:57.087 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1784d227c78e98-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.134 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.135 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.135 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.135 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.135 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.135 DEBUG:boto:Method: GET 2024-04-16T20:05:57.135 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.135 DEBUG:boto:Data: 2024-04-16T20:05:57.135 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.136 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.136 DEBUG:boto:Params: {} 2024-04-16T20:05:57.136 DEBUG:boto:Token: None 2024-04-16T20:05:57.136 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e357ee5ab666f0fd-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.140 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.140 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.140 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.140 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.140 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:05:57.141 DEBUG:boto:Data: 2024-04-16T20:05:57.141 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.141 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.141 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.141 DEBUG:boto:Params: {} 2024-04-16T20:05:57.141 DEBUG:boto:Token: None 2024-04-16T20:05:57.141 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46 2024-04-16T20:05:57.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AtAXI8C27T6nZPrU5UVxezxlJOQ= 2024-04-16T20:05:57.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AtAXI8C27T6nZPrU5UVxezxlJOQ=', 'Content-Length': '0'} 2024-04-16T20:05:57.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d743da70f26864f5-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.143 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.143 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.144 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.144 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:05:57.144 DEBUG:boto:Data: 2024-04-16T20:05:57.144 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.144 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.144 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.144 DEBUG:boto:Params: {} 2024-04-16T20:05:57.144 DEBUG:boto:Token: None 2024-04-16T20:05:57.144 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46 2024-04-16T20:05:57.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AtAXI8C27T6nZPrU5UVxezxlJOQ= 2024-04-16T20:05:57.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AtAXI8C27T6nZPrU5UVxezxlJOQ=', 'Content-Length': '0'} 2024-04-16T20:05:57.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fd7560ae548efad-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:05:57.147 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.147 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.147 DEBUG:boto:Method: GET 2024-04-16T20:05:57.147 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:05:57.147 DEBUG:boto:Data: 2024-04-16T20:05:57.147 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.147 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.147 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.147 DEBUG:boto:Params: {} 2024-04-16T20:05:57.147 DEBUG:boto:Token: None 2024-04-16T20:05:57.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46 2024-04-16T20:05:57.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZM7BpKH7eNG6SEKY9JWpEEwj3Q= 2024-04-16T20:05:57.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZM7BpKH7eNG6SEKY9JWpEEwj3Q=', 'Content-Length': '0'} 2024-04-16T20:05:57.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030835d0b3e967ce2-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.190 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:05:57.190 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:05:57.190 DEBUG:boto:Method: GET 2024-04-16T20:05:57.191 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:05:57.191 DEBUG:boto:Data: 2024-04-16T20:05:57.191 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.191 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.191 DEBUG:boto:Params: {} 2024-04-16T20:05:57.191 DEBUG:boto:Token: None 2024-04-16T20:05:57.191 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj46 2024-04-16T20:05:57.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZM7BpKH7eNG6SEKY9JWpEEwj3Q= 2024-04-16T20:05:57.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZM7BpKH7eNG6SEKY9JWpEEwj3Q=', 'Content-Length': '0'} 2024-04-16T20:05:57.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030dd1b6544ef9142-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.194 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.194 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.194 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.194 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.194 DEBUG:boto:Method: GET 2024-04-16T20:05:57.194 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.194 DEBUG:boto:Data: 2024-04-16T20:05:57.194 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.195 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.195 DEBUG:boto:Params: {} 2024-04-16T20:05:57.195 DEBUG:boto:Token: None 2024-04-16T20:05:57.195 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b1f6a3a0a85548-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.242 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.243 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.243 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.243 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.243 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.243 DEBUG:boto:Method: GET 2024-04-16T20:05:57.243 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:05:57.243 DEBUG:boto:Data: 2024-04-16T20:05:57.243 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.244 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.244 DEBUG:boto:Params: {} 2024-04-16T20:05:57.244 DEBUG:boto:Token: None 2024-04-16T20:05:57.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004533a8cb70503ae7-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.248 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:05:57.248 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.248 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.248 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.248 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.248 DEBUG:boto:Method: GET 2024-04-16T20:05:57.249 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.249 DEBUG:boto:Data: 2024-04-16T20:05:57.249 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.249 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.249 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.249 DEBUG:boto:Params: {} 2024-04-16T20:05:57.249 DEBUG:boto:Token: None 2024-04-16T20:05:57.249 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.249 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA= 2024-04-16T20:05:57.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA=', 'Content-Length': '0'} 2024-04-16T20:05:57.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fadf807ee368eed-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.294 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.294 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.295 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.295 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.295 DEBUG:boto:Method: GET 2024-04-16T20:05:57.295 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.295 DEBUG:boto:Data: 2024-04-16T20:05:57.295 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.295 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.295 DEBUG:boto:Params: {} 2024-04-16T20:05:57.295 DEBUG:boto:Token: None 2024-04-16T20:05:57.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA= 2024-04-16T20:05:57.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA=', 'Content-Length': '0'} 2024-04-16T20:05:57.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000925d9dc036cd8a0-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.298 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.298 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.298 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.299 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.299 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.299 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.299 DEBUG:boto:Data: 2024-04-16T20:05:57.299 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.299 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.299 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.299 DEBUG:boto:Params: {} 2024-04-16T20:05:57.299 DEBUG:boto:Token: None 2024-04-16T20:05:57.299 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q0amMm8maIieJywJAT8lWgGHBbY= 2024-04-16T20:05:57.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q0amMm8maIieJywJAT8lWgGHBbY=', 'Content-Length': '0'} 2024-04-16T20:05:57.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af29b924fbcfb14-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.302 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.302 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.302 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.302 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.302 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.302 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.302 DEBUG:boto:Data: 2024-04-16T20:05:57.302 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.303 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.303 DEBUG:boto:Params: {} 2024-04-16T20:05:57.303 DEBUG:boto:Token: None 2024-04-16T20:05:57.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q0amMm8maIieJywJAT8lWgGHBbY= 2024-04-16T20:05:57.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q0amMm8maIieJywJAT8lWgGHBbY=', 'Content-Length': '0'} 2024-04-16T20:05:57.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429ee77b1264257c-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:05:57.305 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.305 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.305 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.306 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.306 DEBUG:boto:Method: GET 2024-04-16T20:05:57.306 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.306 DEBUG:boto:Data: 2024-04-16T20:05:57.306 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.306 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.306 DEBUG:boto:Params: {} 2024-04-16T20:05:57.307 DEBUG:boto:Token: None 2024-04-16T20:05:57.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA= 2024-04-16T20:05:57.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA=', 'Content-Length': '0'} 2024-04-16T20:05:57.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040425de7093da248-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.350 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.351 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.351 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.351 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.351 DEBUG:boto:Method: GET 2024-04-16T20:05:57.351 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.351 DEBUG:boto:Data: 2024-04-16T20:05:57.351 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.351 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.351 DEBUG:boto:Params: {} 2024-04-16T20:05:57.351 DEBUG:boto:Token: None 2024-04-16T20:05:57.352 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:05:57.352 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA= 2024-04-16T20:05:57.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J7MwvED0/ns2gRXc4dKwV/gg4jA=', 'Content-Length': '0'} 2024-04-16T20:05:57.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2f7fa5ea0d97fd-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.354 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.354 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.355 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.355 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.355 DEBUG:boto:Method: GET 2024-04-16T20:05:57.355 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.355 DEBUG:boto:Data: 2024-04-16T20:05:57.355 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.355 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.355 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.355 DEBUG:boto:Params: {} 2024-04-16T20:05:57.355 DEBUG:boto:Token: None 2024-04-16T20:05:57.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4cf4e3ef91d15c-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.402 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.403 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.403 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.403 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.403 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.403 DEBUG:boto:Method: GET 2024-04-16T20:05:57.403 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.403 DEBUG:boto:Data: 2024-04-16T20:05:57.403 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.403 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.404 DEBUG:boto:Params: {} 2024-04-16T20:05:57.404 DEBUG:boto:Token: None 2024-04-16T20:05:57.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b455ed7e19756a7-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.407 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.408 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.408 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.408 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.408 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:05:57.408 DEBUG:boto:Data: 2024-04-16T20:05:57.408 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.408 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.408 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.408 DEBUG:boto:Params: {} 2024-04-16T20:05:57.408 DEBUG:boto:Token: None 2024-04-16T20:05:57.409 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51 2024-04-16T20:05:57.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1/NquGXnNOTY3Edx4xY6Btdue8o= 2024-04-16T20:05:57.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1/NquGXnNOTY3Edx4xY6Btdue8o=', 'Content-Length': '0'} 2024-04-16T20:05:57.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d648cf9934960b-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.411 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.411 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.411 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.411 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:05:57.411 DEBUG:boto:Data: 2024-04-16T20:05:57.411 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.412 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.412 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.412 DEBUG:boto:Params: {} 2024-04-16T20:05:57.412 DEBUG:boto:Token: None 2024-04-16T20:05:57.412 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51 2024-04-16T20:05:57.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1/NquGXnNOTY3Edx4xY6Btdue8o= 2024-04-16T20:05:57.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1/NquGXnNOTY3Edx4xY6Btdue8o=', 'Content-Length': '0'} 2024-04-16T20:05:57.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40462d707dc011c-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:05:57.414 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.415 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.415 DEBUG:boto:Method: GET 2024-04-16T20:05:57.415 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:05:57.415 DEBUG:boto:Data: 2024-04-16T20:05:57.415 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.415 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.415 DEBUG:boto:Params: {} 2024-04-16T20:05:57.415 DEBUG:boto:Token: None 2024-04-16T20:05:57.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51 2024-04-16T20:05:57.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P5N3ON6YbSYUToLW+HIFyIJHWtw= 2024-04-16T20:05:57.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P5N3ON6YbSYUToLW+HIFyIJHWtw=', 'Content-Length': '0'} 2024-04-16T20:05:57.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d03f0f74787629-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.458 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:05:57.459 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:05:57.459 DEBUG:boto:Method: GET 2024-04-16T20:05:57.459 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:05:57.459 DEBUG:boto:Data: 2024-04-16T20:05:57.459 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.459 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.459 DEBUG:boto:Params: {} 2024-04-16T20:05:57.459 DEBUG:boto:Token: None 2024-04-16T20:05:57.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj51 2024-04-16T20:05:57.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P5N3ON6YbSYUToLW+HIFyIJHWtw= 2024-04-16T20:05:57.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P5N3ON6YbSYUToLW+HIFyIJHWtw=', 'Content-Length': '0'} 2024-04-16T20:05:57.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f1a143655e21cce-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.462 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.462 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.462 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.462 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.463 DEBUG:boto:Method: GET 2024-04-16T20:05:57.463 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.463 DEBUG:boto:Data: 2024-04-16T20:05:57.463 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.463 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.463 DEBUG:boto:Params: {} 2024-04-16T20:05:57.463 DEBUG:boto:Token: None 2024-04-16T20:05:57.463 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000567b36270c54fb93-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.506 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.507 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.507 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.507 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.507 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.507 DEBUG:boto:Method: GET 2024-04-16T20:05:57.507 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:05:57.507 DEBUG:boto:Data: 2024-04-16T20:05:57.507 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.508 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.508 DEBUG:boto:Params: {} 2024-04-16T20:05:57.508 DEBUG:boto:Token: None 2024-04-16T20:05:57.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087937e60aa504000-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.512 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:05:57.512 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.512 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.512 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.512 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.512 DEBUG:boto:Method: GET 2024-04-16T20:05:57.513 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.513 DEBUG:boto:Data: 2024-04-16T20:05:57.513 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.513 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.513 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.513 DEBUG:boto:Params: {} 2024-04-16T20:05:57.513 DEBUG:boto:Token: None 2024-04-16T20:05:57.513 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.513 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk= 2024-04-16T20:05:57.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk=', 'Content-Length': '0'} 2024-04-16T20:05:57.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069b47bb7697c5e36-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.558 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.558 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.559 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.559 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.559 DEBUG:boto:Method: GET 2024-04-16T20:05:57.559 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.559 DEBUG:boto:Data: 2024-04-16T20:05:57.559 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.559 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.559 DEBUG:boto:Params: {} 2024-04-16T20:05:57.559 DEBUG:boto:Token: None 2024-04-16T20:05:57.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk= 2024-04-16T20:05:57.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk=', 'Content-Length': '0'} 2024-04-16T20:05:57.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002709e498d8d6c8eb-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.562 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.562 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.562 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.562 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.563 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.563 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.563 DEBUG:boto:Data: 2024-04-16T20:05:57.563 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.563 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.563 DEBUG:boto:Params: {} 2024-04-16T20:05:57.563 DEBUG:boto:Token: None 2024-04-16T20:05:57.563 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CEqJkW2EPLqXRtiJfPRQOvTeomI= 2024-04-16T20:05:57.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CEqJkW2EPLqXRtiJfPRQOvTeomI=', 'Content-Length': '0'} 2024-04-16T20:05:57.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56e76fbd8d3ce13-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.566 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.566 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.566 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.566 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.566 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.567 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.567 DEBUG:boto:Data: 2024-04-16T20:05:57.567 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.567 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.567 DEBUG:boto:Params: {} 2024-04-16T20:05:57.567 DEBUG:boto:Token: None 2024-04-16T20:05:57.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.567 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CEqJkW2EPLqXRtiJfPRQOvTeomI= 2024-04-16T20:05:57.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CEqJkW2EPLqXRtiJfPRQOvTeomI=', 'Content-Length': '0'} 2024-04-16T20:05:57.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88e57c2d01143ed-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:05:57.570 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.570 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.570 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.570 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.570 DEBUG:boto:Method: GET 2024-04-16T20:05:57.570 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.570 DEBUG:boto:Data: 2024-04-16T20:05:57.570 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.570 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.570 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.571 DEBUG:boto:Params: {} 2024-04-16T20:05:57.571 DEBUG:boto:Token: None 2024-04-16T20:05:57.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk= 2024-04-16T20:05:57.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk=', 'Content-Length': '0'} 2024-04-16T20:05:57.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3f340e17e8833b-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.614 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.614 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.615 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.615 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.615 DEBUG:boto:Method: GET 2024-04-16T20:05:57.615 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.615 DEBUG:boto:Data: 2024-04-16T20:05:57.615 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.615 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.615 DEBUG:boto:Params: {} 2024-04-16T20:05:57.615 DEBUG:boto:Token: None 2024-04-16T20:05:57.616 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:05:57.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk= 2024-04-16T20:05:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KU9Z7k10FmrOXDKLcRb38azWyNk=', 'Content-Length': '0'} 2024-04-16T20:05:57.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad462b4cfe83906-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.618 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.618 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.618 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.618 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.618 DEBUG:boto:Method: GET 2024-04-16T20:05:57.619 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.619 DEBUG:boto:Data: 2024-04-16T20:05:57.619 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.619 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.619 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.619 DEBUG:boto:Params: {} 2024-04-16T20:05:57.619 DEBUG:boto:Token: None 2024-04-16T20:05:57.619 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.619 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea236e894fd1d0e6-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.662 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.663 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.663 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.663 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.663 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.663 DEBUG:boto:Method: GET 2024-04-16T20:05:57.663 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.663 DEBUG:boto:Data: 2024-04-16T20:05:57.663 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.664 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.664 DEBUG:boto:Params: {} 2024-04-16T20:05:57.664 DEBUG:boto:Token: None 2024-04-16T20:05:57.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000366395d82e77d52b-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.668 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.668 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.668 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.668 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.668 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:05:57.668 DEBUG:boto:Data: 2024-04-16T20:05:57.668 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.669 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.669 DEBUG:boto:Params: {} 2024-04-16T20:05:57.669 DEBUG:boto:Token: None 2024-04-16T20:05:57.669 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55 2024-04-16T20:05:57.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qZclCoyOiNAuhCyUoLWxXZt5OPY= 2024-04-16T20:05:57.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qZclCoyOiNAuhCyUoLWxXZt5OPY=', 'Content-Length': '0'} 2024-04-16T20:05:57.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065524d39e7506f4a-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.671 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.672 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.672 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.672 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:05:57.672 DEBUG:boto:Data: 2024-04-16T20:05:57.672 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.672 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.672 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.672 DEBUG:boto:Params: {} 2024-04-16T20:05:57.672 DEBUG:boto:Token: None 2024-04-16T20:05:57.672 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55 2024-04-16T20:05:57.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qZclCoyOiNAuhCyUoLWxXZt5OPY= 2024-04-16T20:05:57.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qZclCoyOiNAuhCyUoLWxXZt5OPY=', 'Content-Length': '0'} 2024-04-16T20:05:57.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c62461b7e49bbce-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:05:57.675 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.675 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.676 DEBUG:boto:Method: GET 2024-04-16T20:05:57.676 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:05:57.676 DEBUG:boto:Data: 2024-04-16T20:05:57.676 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.676 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.676 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.676 DEBUG:boto:Params: {} 2024-04-16T20:05:57.676 DEBUG:boto:Token: None 2024-04-16T20:05:57.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55 2024-04-16T20:05:57.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rdVkvvLmd4U0Ap7R4nMFW7qdKx0= 2024-04-16T20:05:57.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rdVkvvLmd4U0Ap7R4nMFW7qdKx0=', 'Content-Length': '0'} 2024-04-16T20:05:57.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e31b85d2bb78426-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.726 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:05:57.726 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:05:57.727 DEBUG:boto:Method: GET 2024-04-16T20:05:57.727 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:05:57.727 DEBUG:boto:Data: 2024-04-16T20:05:57.727 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.727 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.727 DEBUG:boto:Params: {} 2024-04-16T20:05:57.727 DEBUG:boto:Token: None 2024-04-16T20:05:57.727 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj55 2024-04-16T20:05:57.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rdVkvvLmd4U0Ap7R4nMFW7qdKx0= 2024-04-16T20:05:57.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rdVkvvLmd4U0Ap7R4nMFW7qdKx0=', 'Content-Length': '0'} 2024-04-16T20:05:57.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd35d361fcbdb3b8-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.730 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.730 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.730 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.730 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.731 DEBUG:boto:Method: GET 2024-04-16T20:05:57.731 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.731 DEBUG:boto:Data: 2024-04-16T20:05:57.731 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.731 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.731 DEBUG:boto:Params: {} 2024-04-16T20:05:57.731 DEBUG:boto:Token: None 2024-04-16T20:05:57.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc24a149212979b-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.778 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.779 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.779 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.779 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.779 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.779 DEBUG:boto:Method: GET 2024-04-16T20:05:57.779 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:05:57.779 DEBUG:boto:Data: 2024-04-16T20:05:57.779 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.780 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.780 DEBUG:boto:Params: {} 2024-04-16T20:05:57.780 DEBUG:boto:Token: None 2024-04-16T20:05:57.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000425a54b26774dc1b-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.784 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:05:57.784 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.784 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.784 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.785 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.785 DEBUG:boto:Method: GET 2024-04-16T20:05:57.785 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.785 DEBUG:boto:Data: 2024-04-16T20:05:57.785 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.785 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.785 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.785 DEBUG:boto:Params: {} 2024-04-16T20:05:57.785 DEBUG:boto:Token: None 2024-04-16T20:05:57.785 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.785 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE= 2024-04-16T20:05:57.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE=', 'Content-Length': '0'} 2024-04-16T20:05:57.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ab547fa3c76b1f-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.834 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.834 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.835 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.835 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.835 DEBUG:boto:Method: GET 2024-04-16T20:05:57.835 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.835 DEBUG:boto:Data: 2024-04-16T20:05:57.835 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.835 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.835 DEBUG:boto:Params: {} 2024-04-16T20:05:57.835 DEBUG:boto:Token: None 2024-04-16T20:05:57.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE= 2024-04-16T20:05:57.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE=', 'Content-Length': '0'} 2024-04-16T20:05:57.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9bafc71bf6b2c5-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.838 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.838 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.838 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.838 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.839 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.839 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.839 DEBUG:boto:Data: 2024-04-16T20:05:57.839 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.839 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.839 DEBUG:boto:Params: {} 2024-04-16T20:05:57.839 DEBUG:boto:Token: None 2024-04-16T20:05:57.839 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B3F3gyVvEdTKdwuR0NQIDYfaw5E= 2024-04-16T20:05:57.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B3F3gyVvEdTKdwuR0NQIDYfaw5E=', 'Content-Length': '0'} 2024-04-16T20:05:57.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c5469d1c2a2208-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.842 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.842 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.842 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.842 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.842 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.842 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.842 DEBUG:boto:Data: 2024-04-16T20:05:57.842 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.843 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.843 DEBUG:boto:Params: {} 2024-04-16T20:05:57.843 DEBUG:boto:Token: None 2024-04-16T20:05:57.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B3F3gyVvEdTKdwuR0NQIDYfaw5E= 2024-04-16T20:05:57.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B3F3gyVvEdTKdwuR0NQIDYfaw5E=', 'Content-Length': '0'} 2024-04-16T20:05:57.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f2276b9a2bbba1-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:05:57.845 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.845 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.845 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.846 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.846 DEBUG:boto:Method: GET 2024-04-16T20:05:57.846 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.846 DEBUG:boto:Data: 2024-04-16T20:05:57.846 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.846 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.846 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.846 DEBUG:boto:Params: {} 2024-04-16T20:05:57.846 DEBUG:boto:Token: None 2024-04-16T20:05:57.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE= 2024-04-16T20:05:57.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE=', 'Content-Length': '0'} 2024-04-16T20:05:57.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f574eff6ca9c904f-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.890 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.890 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.891 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.891 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.891 DEBUG:boto:Method: GET 2024-04-16T20:05:57.891 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.891 DEBUG:boto:Data: 2024-04-16T20:05:57.891 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.891 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.891 DEBUG:boto:Params: {} 2024-04-16T20:05:57.891 DEBUG:boto:Token: None 2024-04-16T20:05:57.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:05:57.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE= 2024-04-16T20:05:57.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ywlAqb+nC4WC8squvE3/uGzxVlE=', 'Content-Length': '0'} 2024-04-16T20:05:57.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2e327e15315e83-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.894 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.894 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.894 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.894 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.894 DEBUG:boto:Method: GET 2024-04-16T20:05:57.895 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.895 DEBUG:boto:Data: 2024-04-16T20:05:57.895 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.895 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.895 DEBUG:boto:Params: {} 2024-04-16T20:05:57.895 DEBUG:boto:Token: None 2024-04-16T20:05:57.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b6e7626d48a9f4-00661eda25-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.942 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.943 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:57.943 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:57.943 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.943 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.943 DEBUG:boto:Method: GET 2024-04-16T20:05:57.943 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:05:57.943 DEBUG:boto:Data: 2024-04-16T20:05:57.944 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.944 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.944 DEBUG:boto:Params: {} 2024-04-16T20:05:57.944 DEBUG:boto:Token: None 2024-04-16T20:05:57.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/ 2024-04-16T20:05:57.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc= 2024-04-16T20:05:57.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1PPu/Cqgk58dl77FI7AO/S5SJLc=', 'Content-Length': '0'} 2024-04-16T20:05:57.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2cca329e6a97353-00661eda25-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.948 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:57.948 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.948 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.948 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.948 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:05:57.948 DEBUG:boto:Data: 2024-04-16T20:05:57.949 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.949 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.949 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.949 DEBUG:boto:Params: {} 2024-04-16T20:05:57.949 DEBUG:boto:Token: None 2024-04-16T20:05:57.949 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57 2024-04-16T20:05:57.949 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vN4hl1O4UnW1lo0UC/rRAZ7BFXI= 2024-04-16T20:05:57.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vN4hl1O4UnW1lo0UC/rRAZ7BFXI=', 'Content-Length': '0'} 2024-04-16T20:05:57.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49efcf07167eba9-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.952 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.952 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.952 DEBUG:boto:Method: HEAD 2024-04-16T20:05:57.952 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:05:57.952 DEBUG:boto:Data: 2024-04-16T20:05:57.952 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.952 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.952 DEBUG:boto:Params: {} 2024-04-16T20:05:57.952 DEBUG:boto:Token: None 2024-04-16T20:05:57.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57 2024-04-16T20:05:57.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vN4hl1O4UnW1lo0UC/rRAZ7BFXI= 2024-04-16T20:05:57.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vN4hl1O4UnW1lo0UC/rRAZ7BFXI=', 'Content-Length': '0'} 2024-04-16T20:05:57.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ce13f1dfc85e1c-00661eda25-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:05:57.955 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.955 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.955 DEBUG:boto:Method: GET 2024-04-16T20:05:57.955 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:05:57.955 DEBUG:boto:Data: 2024-04-16T20:05:57.955 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:57.956 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.956 DEBUG:boto:Params: {} 2024-04-16T20:05:57.956 DEBUG:boto:Token: None 2024-04-16T20:05:57.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57 2024-04-16T20:05:57.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rWU2iYkRSuft7NTbk1BZgXb0MRw= 2024-04-16T20:05:57.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rWU2iYkRSuft7NTbk1BZgXb0MRw=', 'Content-Length': '0'} 2024-04-16T20:05:57.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d8cca66367c464-00661eda25-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:57.998 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:05:57.998 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:05:57.999 DEBUG:boto:Method: GET 2024-04-16T20:05:57.999 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:05:57.999 DEBUG:boto:Data: 2024-04-16T20:05:57.999 DEBUG:boto:Headers: {} 2024-04-16T20:05:57.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:57.999 DEBUG:boto:Port: 8000 2024-04-16T20:05:57.999 DEBUG:boto:Params: {} 2024-04-16T20:05:57.999 DEBUG:boto:Token: None 2024-04-16T20:05:57.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:57 GMT /yrjefh-52/obj57 2024-04-16T20:05:58.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rWU2iYkRSuft7NTbk1BZgXb0MRw= 2024-04-16T20:05:58.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:57 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rWU2iYkRSuft7NTbk1BZgXb0MRw=', 'Content-Length': '0'} 2024-04-16T20:05:58.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cb16e3eb6b48ab4-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.002 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.002 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.002 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.002 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.003 DEBUG:boto:Method: GET 2024-04-16T20:05:58.003 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.003 DEBUG:boto:Data: 2024-04-16T20:05:58.003 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.003 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.003 DEBUG:boto:Params: {} 2024-04-16T20:05:58.003 DEBUG:boto:Token: None 2024-04-16T20:05:58.003 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0864de640dc48b-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.046 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.047 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.047 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.047 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.047 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.047 DEBUG:boto:Method: GET 2024-04-16T20:05:58.047 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:05:58.047 DEBUG:boto:Data: 2024-04-16T20:05:58.048 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.048 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.048 DEBUG:boto:Params: {} 2024-04-16T20:05:58.048 DEBUG:boto:Token: None 2024-04-16T20:05:58.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6686134c3ed52ad-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.052 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:05:58.052 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.052 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.052 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.053 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.053 DEBUG:boto:Method: GET 2024-04-16T20:05:58.053 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.053 DEBUG:boto:Data: 2024-04-16T20:05:58.053 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.053 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.053 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.053 DEBUG:boto:Params: {} 2024-04-16T20:05:58.053 DEBUG:boto:Token: None 2024-04-16T20:05:58.053 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o= 2024-04-16T20:05:58.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o=', 'Content-Length': '0'} 2024-04-16T20:05:58.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2f8ce9c6dc9147-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.098 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.098 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.099 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.099 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.099 DEBUG:boto:Method: GET 2024-04-16T20:05:58.099 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.099 DEBUG:boto:Data: 2024-04-16T20:05:58.099 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.099 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.099 DEBUG:boto:Params: {} 2024-04-16T20:05:58.099 DEBUG:boto:Token: None 2024-04-16T20:05:58.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o= 2024-04-16T20:05:58.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o=', 'Content-Length': '0'} 2024-04-16T20:05:58.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052dd8fe9d4b1b33c-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.102 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.102 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.102 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.102 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.102 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.103 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.103 DEBUG:boto:Data: 2024-04-16T20:05:58.103 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.103 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.103 DEBUG:boto:Params: {} 2024-04-16T20:05:58.103 DEBUG:boto:Token: None 2024-04-16T20:05:58.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fm8g4Tv5q7ClCeTddKaP3zzYXP4= 2024-04-16T20:05:58.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fm8g4Tv5q7ClCeTddKaP3zzYXP4=', 'Content-Length': '0'} 2024-04-16T20:05:58.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f72afe42728d566-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.105 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.106 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.106 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.106 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.106 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.106 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.106 DEBUG:boto:Data: 2024-04-16T20:05:58.107 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.107 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.107 DEBUG:boto:Params: {} 2024-04-16T20:05:58.107 DEBUG:boto:Token: None 2024-04-16T20:05:58.107 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fm8g4Tv5q7ClCeTddKaP3zzYXP4= 2024-04-16T20:05:58.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fm8g4Tv5q7ClCeTddKaP3zzYXP4=', 'Content-Length': '0'} 2024-04-16T20:05:58.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf2e3fe9ee32c18-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:05:58.110 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.110 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.110 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.110 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.110 DEBUG:boto:Method: GET 2024-04-16T20:05:58.110 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.110 DEBUG:boto:Data: 2024-04-16T20:05:58.111 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.111 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.111 DEBUG:boto:Params: {} 2024-04-16T20:05:58.111 DEBUG:boto:Token: None 2024-04-16T20:05:58.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o= 2024-04-16T20:05:58.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o=', 'Content-Length': '0'} 2024-04-16T20:05:58.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3fdf85edda9c99-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.154 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.154 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.154 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.155 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.155 DEBUG:boto:Method: GET 2024-04-16T20:05:58.155 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.155 DEBUG:boto:Data: 2024-04-16T20:05:58.155 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.155 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.155 DEBUG:boto:Params: {} 2024-04-16T20:05:58.155 DEBUG:boto:Token: None 2024-04-16T20:05:58.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:05:58.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o= 2024-04-16T20:05:58.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAQrTG0M5DpIjecC9331q/+Aj9o=', 'Content-Length': '0'} 2024-04-16T20:05:58.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c3d971c8e229889-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.158 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.158 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.158 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.158 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.159 DEBUG:boto:Method: GET 2024-04-16T20:05:58.159 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.159 DEBUG:boto:Data: 2024-04-16T20:05:58.159 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.159 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.159 DEBUG:boto:Params: {} 2024-04-16T20:05:58.159 DEBUG:boto:Token: None 2024-04-16T20:05:58.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6cef49e9f1606e-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.202 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.203 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.203 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.203 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.203 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.203 DEBUG:boto:Method: GET 2024-04-16T20:05:58.203 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.203 DEBUG:boto:Data: 2024-04-16T20:05:58.203 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.203 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.204 DEBUG:boto:Params: {} 2024-04-16T20:05:58.204 DEBUG:boto:Token: None 2024-04-16T20:05:58.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec303516f11de20d-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.207 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.207 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.208 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.208 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.208 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:05:58.208 DEBUG:boto:Data: 2024-04-16T20:05:58.208 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.208 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.208 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.208 DEBUG:boto:Params: {} 2024-04-16T20:05:58.208 DEBUG:boto:Token: None 2024-04-16T20:05:58.208 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58 2024-04-16T20:05:58.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t+Bb3vewgtBxz3EMj7BxNASWof4= 2024-04-16T20:05:58.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t+Bb3vewgtBxz3EMj7BxNASWof4=', 'Content-Length': '0'} 2024-04-16T20:05:58.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002678788ac484b12-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.211 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.211 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.211 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.211 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:05:58.211 DEBUG:boto:Data: 2024-04-16T20:05:58.211 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.211 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.211 DEBUG:boto:Params: {} 2024-04-16T20:05:58.212 DEBUG:boto:Token: None 2024-04-16T20:05:58.212 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58 2024-04-16T20:05:58.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t+Bb3vewgtBxz3EMj7BxNASWof4= 2024-04-16T20:05:58.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t+Bb3vewgtBxz3EMj7BxNASWof4=', 'Content-Length': '0'} 2024-04-16T20:05:58.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfedeb047a0cb2fb-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:05:58.214 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.215 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.215 DEBUG:boto:Method: GET 2024-04-16T20:05:58.215 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:05:58.215 DEBUG:boto:Data: 2024-04-16T20:05:58.215 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.215 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.215 DEBUG:boto:Params: {} 2024-04-16T20:05:58.215 DEBUG:boto:Token: None 2024-04-16T20:05:58.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58 2024-04-16T20:05:58.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFxOxABbr3zZXAsMgqxiCBh6qqU= 2024-04-16T20:05:58.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFxOxABbr3zZXAsMgqxiCBh6qqU=', 'Content-Length': '0'} 2024-04-16T20:05:58.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040184cd0f6562e1b-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.258 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:05:58.258 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:05:58.258 DEBUG:boto:Method: GET 2024-04-16T20:05:58.259 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:05:58.259 DEBUG:boto:Data: 2024-04-16T20:05:58.259 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.259 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.259 DEBUG:boto:Params: {} 2024-04-16T20:05:58.259 DEBUG:boto:Token: None 2024-04-16T20:05:58.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj58 2024-04-16T20:05:58.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BFxOxABbr3zZXAsMgqxiCBh6qqU= 2024-04-16T20:05:58.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BFxOxABbr3zZXAsMgqxiCBh6qqU=', 'Content-Length': '0'} 2024-04-16T20:05:58.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043346985bf271c9e-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.262 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.262 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.262 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.262 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.262 DEBUG:boto:Method: GET 2024-04-16T20:05:58.262 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.262 DEBUG:boto:Data: 2024-04-16T20:05:58.262 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.262 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.263 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.263 DEBUG:boto:Params: {} 2024-04-16T20:05:58.263 DEBUG:boto:Token: None 2024-04-16T20:05:58.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ed985c73a5ac12-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.306 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.306 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.307 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.307 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.307 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.307 DEBUG:boto:Method: GET 2024-04-16T20:05:58.307 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:05:58.307 DEBUG:boto:Data: 2024-04-16T20:05:58.307 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.307 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.307 DEBUG:boto:Params: {} 2024-04-16T20:05:58.307 DEBUG:boto:Token: None 2024-04-16T20:05:58.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003170f85699bcc283-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.311 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:05:58.312 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.312 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.312 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.312 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.312 DEBUG:boto:Method: GET 2024-04-16T20:05:58.312 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.312 DEBUG:boto:Data: 2024-04-16T20:05:58.312 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.312 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.313 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.313 DEBUG:boto:Params: {} 2024-04-16T20:05:58.313 DEBUG:boto:Token: None 2024-04-16T20:05:58.313 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.313 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg= 2024-04-16T20:05:58.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg=', 'Content-Length': '0'} 2024-04-16T20:05:58.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb948d809d5d6c6c-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.362 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.362 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.362 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.362 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.363 DEBUG:boto:Method: GET 2024-04-16T20:05:58.363 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.363 DEBUG:boto:Data: 2024-04-16T20:05:58.363 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.363 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.363 DEBUG:boto:Params: {} 2024-04-16T20:05:58.363 DEBUG:boto:Token: None 2024-04-16T20:05:58.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg= 2024-04-16T20:05:58.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg=', 'Content-Length': '0'} 2024-04-16T20:05:58.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6370fc138fe0f10-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.366 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.366 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.366 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.366 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.367 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.367 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.367 DEBUG:boto:Data: 2024-04-16T20:05:58.367 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.367 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.367 DEBUG:boto:Params: {} 2024-04-16T20:05:58.367 DEBUG:boto:Token: None 2024-04-16T20:05:58.367 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OwgnZjg2a0Teut0jCbryUuPZ4AI= 2024-04-16T20:05:58.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OwgnZjg2a0Teut0jCbryUuPZ4AI=', 'Content-Length': '0'} 2024-04-16T20:05:58.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022885771c0290206-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.370 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.370 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.370 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.370 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.370 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.370 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.370 DEBUG:boto:Data: 2024-04-16T20:05:58.370 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.371 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.371 DEBUG:boto:Params: {} 2024-04-16T20:05:58.371 DEBUG:boto:Token: None 2024-04-16T20:05:58.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OwgnZjg2a0Teut0jCbryUuPZ4AI= 2024-04-16T20:05:58.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OwgnZjg2a0Teut0jCbryUuPZ4AI=', 'Content-Length': '0'} 2024-04-16T20:05:58.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a01ec73008f9bc1-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:05:58.373 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.374 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.374 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.374 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.374 DEBUG:boto:Method: GET 2024-04-16T20:05:58.374 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.374 DEBUG:boto:Data: 2024-04-16T20:05:58.374 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.374 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.374 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.374 DEBUG:boto:Params: {} 2024-04-16T20:05:58.374 DEBUG:boto:Token: None 2024-04-16T20:05:58.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg= 2024-04-16T20:05:58.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg=', 'Content-Length': '0'} 2024-04-16T20:05:58.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067253a8dddbb077-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.418 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.418 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.418 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.419 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.419 DEBUG:boto:Method: GET 2024-04-16T20:05:58.419 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.419 DEBUG:boto:Data: 2024-04-16T20:05:58.419 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.419 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.419 DEBUG:boto:Params: {} 2024-04-16T20:05:58.419 DEBUG:boto:Token: None 2024-04-16T20:05:58.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:05:58.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg= 2024-04-16T20:05:58.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oxYRXHQdbwE99miN4I4sGfBqnSg=', 'Content-Length': '0'} 2024-04-16T20:05:58.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca87a9fff738843b-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.422 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.422 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.422 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.422 DEBUG:boto:Method: GET 2024-04-16T20:05:58.422 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.422 DEBUG:boto:Data: 2024-04-16T20:05:58.422 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.422 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.423 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.423 DEBUG:boto:Params: {} 2024-04-16T20:05:58.423 DEBUG:boto:Token: None 2024-04-16T20:05:58.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98acc68a8bca374-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.466 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.467 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.467 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.467 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.467 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.467 DEBUG:boto:Method: GET 2024-04-16T20:05:58.467 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.467 DEBUG:boto:Data: 2024-04-16T20:05:58.467 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.467 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.467 DEBUG:boto:Params: {} 2024-04-16T20:05:58.468 DEBUG:boto:Token: None 2024-04-16T20:05:58.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef5c2fc25d1a1d61-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.472 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.472 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.472 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.472 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.472 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:05:58.472 DEBUG:boto:Data: 2024-04-16T20:05:58.472 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.472 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.473 DEBUG:boto:Params: {} 2024-04-16T20:05:58.473 DEBUG:boto:Token: None 2024-04-16T20:05:58.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62 2024-04-16T20:05:58.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9h1vKYrbSPc/SMQcVmL+jqjF0BA= 2024-04-16T20:05:58.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9h1vKYrbSPc/SMQcVmL+jqjF0BA=', 'Content-Length': '0'} 2024-04-16T20:05:58.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db6c67fbe6a5822-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.475 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.475 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.476 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.476 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:05:58.476 DEBUG:boto:Data: 2024-04-16T20:05:58.476 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.476 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.476 DEBUG:boto:Params: {} 2024-04-16T20:05:58.476 DEBUG:boto:Token: None 2024-04-16T20:05:58.476 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62 2024-04-16T20:05:58.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9h1vKYrbSPc/SMQcVmL+jqjF0BA= 2024-04-16T20:05:58.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9h1vKYrbSPc/SMQcVmL+jqjF0BA=', 'Content-Length': '0'} 2024-04-16T20:05:58.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9e651ee7897f9c-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:05:58.479 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.479 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.479 DEBUG:boto:Method: GET 2024-04-16T20:05:58.479 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:05:58.479 DEBUG:boto:Data: 2024-04-16T20:05:58.479 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.479 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.479 DEBUG:boto:Params: {} 2024-04-16T20:05:58.480 DEBUG:boto:Token: None 2024-04-16T20:05:58.480 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62 2024-04-16T20:05:58.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNnwn6/2jmCVgJC8gaqT5nXKawM= 2024-04-16T20:05:58.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNnwn6/2jmCVgJC8gaqT5nXKawM=', 'Content-Length': '0'} 2024-04-16T20:05:58.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001886ef3ba36d3627-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.522 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:05:58.522 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:05:58.523 DEBUG:boto:Method: GET 2024-04-16T20:05:58.523 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:05:58.523 DEBUG:boto:Data: 2024-04-16T20:05:58.523 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.523 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.523 DEBUG:boto:Params: {} 2024-04-16T20:05:58.523 DEBUG:boto:Token: None 2024-04-16T20:05:58.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj62 2024-04-16T20:05:58.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNnwn6/2jmCVgJC8gaqT5nXKawM= 2024-04-16T20:05:58.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNnwn6/2jmCVgJC8gaqT5nXKawM=', 'Content-Length': '0'} 2024-04-16T20:05:58.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b288f7bfb443bc-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.526 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.526 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.526 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.526 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.526 DEBUG:boto:Method: GET 2024-04-16T20:05:58.526 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.526 DEBUG:boto:Data: 2024-04-16T20:05:58.526 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.527 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.527 DEBUG:boto:Params: {} 2024-04-16T20:05:58.527 DEBUG:boto:Token: None 2024-04-16T20:05:58.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb8d2253ef65187-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.570 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.571 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.571 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.571 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.571 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.571 DEBUG:boto:Method: GET 2024-04-16T20:05:58.571 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:05:58.571 DEBUG:boto:Data: 2024-04-16T20:05:58.571 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.571 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.571 DEBUG:boto:Params: {} 2024-04-16T20:05:58.571 DEBUG:boto:Token: None 2024-04-16T20:05:58.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb4676d4113098bf-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.575 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:05:58.576 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.576 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.576 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.576 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.576 DEBUG:boto:Method: GET 2024-04-16T20:05:58.576 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.576 DEBUG:boto:Data: 2024-04-16T20:05:58.576 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.576 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.576 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.576 DEBUG:boto:Params: {} 2024-04-16T20:05:58.576 DEBUG:boto:Token: None 2024-04-16T20:05:58.577 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.577 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0= 2024-04-16T20:05:58.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0=', 'Content-Length': '0'} 2024-04-16T20:05:58.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6e49c7f1a30ba6-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.622 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.622 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.623 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.623 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.623 DEBUG:boto:Method: GET 2024-04-16T20:05:58.623 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.623 DEBUG:boto:Data: 2024-04-16T20:05:58.623 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.623 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.623 DEBUG:boto:Params: {} 2024-04-16T20:05:58.623 DEBUG:boto:Token: None 2024-04-16T20:05:58.624 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0= 2024-04-16T20:05:58.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0=', 'Content-Length': '0'} 2024-04-16T20:05:58.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000660c8203e627106d-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.626 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.626 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.626 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.626 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.626 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.626 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.627 DEBUG:boto:Data: 2024-04-16T20:05:58.627 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.627 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.627 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.627 DEBUG:boto:Params: {} 2024-04-16T20:05:58.627 DEBUG:boto:Token: None 2024-04-16T20:05:58.627 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FxokyVWfkUYHZNzmhQ9AplEtW1Y= 2024-04-16T20:05:58.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FxokyVWfkUYHZNzmhQ9AplEtW1Y=', 'Content-Length': '0'} 2024-04-16T20:05:58.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c762ea6940a04a7-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.630 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.630 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.630 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.630 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.630 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.630 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.630 DEBUG:boto:Data: 2024-04-16T20:05:58.631 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.631 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.631 DEBUG:boto:Params: {} 2024-04-16T20:05:58.631 DEBUG:boto:Token: None 2024-04-16T20:05:58.631 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FxokyVWfkUYHZNzmhQ9AplEtW1Y= 2024-04-16T20:05:58.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FxokyVWfkUYHZNzmhQ9AplEtW1Y=', 'Content-Length': '0'} 2024-04-16T20:05:58.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ec602e738cfbc6-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:05:58.633 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.633 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.634 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.634 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.634 DEBUG:boto:Method: GET 2024-04-16T20:05:58.634 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.634 DEBUG:boto:Data: 2024-04-16T20:05:58.634 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.634 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.634 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.634 DEBUG:boto:Params: {} 2024-04-16T20:05:58.634 DEBUG:boto:Token: None 2024-04-16T20:05:58.634 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0= 2024-04-16T20:05:58.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0=', 'Content-Length': '0'} 2024-04-16T20:05:58.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002baf8392f110f199-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.682 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.682 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.682 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.683 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.683 DEBUG:boto:Method: GET 2024-04-16T20:05:58.683 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.683 DEBUG:boto:Data: 2024-04-16T20:05:58.683 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.683 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.683 DEBUG:boto:Params: {} 2024-04-16T20:05:58.683 DEBUG:boto:Token: None 2024-04-16T20:05:58.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:05:58.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0= 2024-04-16T20:05:58.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PR1kBsfRqj+BMWWgSeMSzLp7Hm0=', 'Content-Length': '0'} 2024-04-16T20:05:58.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3f5147eee6bf1b-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.686 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.686 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.686 DEBUG:boto:Method: GET 2024-04-16T20:05:58.686 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.686 DEBUG:boto:Data: 2024-04-16T20:05:58.686 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.687 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.687 DEBUG:boto:Params: {} 2024-04-16T20:05:58.687 DEBUG:boto:Token: None 2024-04-16T20:05:58.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751674a1d62e5d2c-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.734 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.735 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.735 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.735 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.735 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.735 DEBUG:boto:Method: GET 2024-04-16T20:05:58.735 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.735 DEBUG:boto:Data: 2024-04-16T20:05:58.735 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.735 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.736 DEBUG:boto:Params: {} 2024-04-16T20:05:58.736 DEBUG:boto:Token: None 2024-04-16T20:05:58.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be13cc6b1356e9bc-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.740 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.740 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.740 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.740 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.740 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:05:58.740 DEBUG:boto:Data: 2024-04-16T20:05:58.741 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.741 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.741 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.741 DEBUG:boto:Params: {} 2024-04-16T20:05:58.741 DEBUG:boto:Token: None 2024-04-16T20:05:58.741 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68 2024-04-16T20:05:58.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QOoRqTs4lE30icr0ZL+0gvih4Ec= 2024-04-16T20:05:58.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QOoRqTs4lE30icr0ZL+0gvih4Ec=', 'Content-Length': '0'} 2024-04-16T20:05:58.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bcc049ecf8d9fd-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.743 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.744 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.744 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.744 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:05:58.744 DEBUG:boto:Data: 2024-04-16T20:05:58.744 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.744 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.744 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.744 DEBUG:boto:Params: {} 2024-04-16T20:05:58.744 DEBUG:boto:Token: None 2024-04-16T20:05:58.744 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68 2024-04-16T20:05:58.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QOoRqTs4lE30icr0ZL+0gvih4Ec= 2024-04-16T20:05:58.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QOoRqTs4lE30icr0ZL+0gvih4Ec=', 'Content-Length': '0'} 2024-04-16T20:05:58.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009497bf709ad64bfa-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:05:58.747 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.747 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.747 DEBUG:boto:Method: GET 2024-04-16T20:05:58.747 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:05:58.747 DEBUG:boto:Data: 2024-04-16T20:05:58.747 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.747 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.747 DEBUG:boto:Params: {} 2024-04-16T20:05:58.748 DEBUG:boto:Token: None 2024-04-16T20:05:58.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68 2024-04-16T20:05:58.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2gUYQ+x1BFl+q/Tp3CZb8dcgjUk= 2024-04-16T20:05:58.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2gUYQ+x1BFl+q/Tp3CZb8dcgjUk=', 'Content-Length': '0'} 2024-04-16T20:05:58.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b23ad27f9ebadb5c-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.790 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:05:58.790 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:05:58.790 DEBUG:boto:Method: GET 2024-04-16T20:05:58.791 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:05:58.791 DEBUG:boto:Data: 2024-04-16T20:05:58.791 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.791 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.791 DEBUG:boto:Params: {} 2024-04-16T20:05:58.791 DEBUG:boto:Token: None 2024-04-16T20:05:58.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj68 2024-04-16T20:05:58.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2gUYQ+x1BFl+q/Tp3CZb8dcgjUk= 2024-04-16T20:05:58.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2gUYQ+x1BFl+q/Tp3CZb8dcgjUk=', 'Content-Length': '0'} 2024-04-16T20:05:58.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ff0a073bdd2525-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.793 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.794 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.794 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.794 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.794 DEBUG:boto:Method: GET 2024-04-16T20:05:58.794 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.794 DEBUG:boto:Data: 2024-04-16T20:05:58.794 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.794 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.794 DEBUG:boto:Params: {} 2024-04-16T20:05:58.794 DEBUG:boto:Token: None 2024-04-16T20:05:58.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042432fc688dfad71-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.838 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.839 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.839 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.839 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.839 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.839 DEBUG:boto:Method: GET 2024-04-16T20:05:58.839 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:05:58.839 DEBUG:boto:Data: 2024-04-16T20:05:58.839 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.839 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.839 DEBUG:boto:Params: {} 2024-04-16T20:05:58.840 DEBUG:boto:Token: None 2024-04-16T20:05:58.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c9a30a3ab765a65-00661eda26-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.844 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:05:58.844 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.844 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.844 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.844 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.845 DEBUG:boto:Method: GET 2024-04-16T20:05:58.845 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.845 DEBUG:boto:Data: 2024-04-16T20:05:58.845 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.845 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.845 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.845 DEBUG:boto:Params: {} 2024-04-16T20:05:58.845 DEBUG:boto:Token: None 2024-04-16T20:05:58.845 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.845 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k= 2024-04-16T20:05:58.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k=', 'Content-Length': '0'} 2024-04-16T20:05:58.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a75e8b23c295a5c-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.890 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.890 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.890 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.891 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.891 DEBUG:boto:Method: GET 2024-04-16T20:05:58.891 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.891 DEBUG:boto:Data: 2024-04-16T20:05:58.891 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.891 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.891 DEBUG:boto:Params: {} 2024-04-16T20:05:58.891 DEBUG:boto:Token: None 2024-04-16T20:05:58.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k= 2024-04-16T20:05:58.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k=', 'Content-Length': '0'} 2024-04-16T20:05:58.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8eda2b26f16b18-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.894 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.894 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.894 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.895 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.895 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.895 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.895 DEBUG:boto:Data: 2024-04-16T20:05:58.895 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.895 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.895 DEBUG:boto:Params: {} 2024-04-16T20:05:58.895 DEBUG:boto:Token: None 2024-04-16T20:05:58.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9V/9g7kQoqAPU1E7nALrlT/sJPo= 2024-04-16T20:05:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9V/9g7kQoqAPU1E7nALrlT/sJPo=', 'Content-Length': '0'} 2024-04-16T20:05:58.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd590861eb40d63-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.898 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.898 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.898 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.898 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.898 DEBUG:boto:Method: HEAD 2024-04-16T20:05:58.898 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.898 DEBUG:boto:Data: 2024-04-16T20:05:58.898 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.898 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.898 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.899 DEBUG:boto:Params: {} 2024-04-16T20:05:58.899 DEBUG:boto:Token: None 2024-04-16T20:05:58.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9V/9g7kQoqAPU1E7nALrlT/sJPo= 2024-04-16T20:05:58.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9V/9g7kQoqAPU1E7nALrlT/sJPo=', 'Content-Length': '0'} 2024-04-16T20:05:58.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081197ac65b331f32-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:05:58.901 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.901 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.901 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.901 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.902 DEBUG:boto:Method: GET 2024-04-16T20:05:58.902 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.902 DEBUG:boto:Data: 2024-04-16T20:05:58.902 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.902 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.902 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.902 DEBUG:boto:Params: {} 2024-04-16T20:05:58.902 DEBUG:boto:Token: None 2024-04-16T20:05:58.902 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k= 2024-04-16T20:05:58.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k=', 'Content-Length': '0'} 2024-04-16T20:05:58.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005eadd57e5280982-00661eda26-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.946 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:58.946 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:58.946 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.946 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.947 DEBUG:boto:Method: GET 2024-04-16T20:05:58.947 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.947 DEBUG:boto:Data: 2024-04-16T20:05:58.947 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.947 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.947 DEBUG:boto:Params: {} 2024-04-16T20:05:58.947 DEBUG:boto:Token: None 2024-04-16T20:05:58.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:05:58.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k= 2024-04-16T20:05:58.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/kZL68vpMAbhs6p2HCmZ6Jtb43k=', 'Content-Length': '0'} 2024-04-16T20:05:58.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066adb0478f6b3cbb-00661eda26-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.950 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.950 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.950 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.950 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.951 DEBUG:boto:Method: GET 2024-04-16T20:05:58.951 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.951 DEBUG:boto:Data: 2024-04-16T20:05:58.951 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:58.951 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.951 DEBUG:boto:Params: {} 2024-04-16T20:05:58.951 DEBUG:boto:Token: None 2024-04-16T20:05:58.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076993a28310f4b1a-00661eda26-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.994 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.995 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:58.995 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:58.995 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.995 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.995 DEBUG:boto:Method: GET 2024-04-16T20:05:58.995 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:05:58.995 DEBUG:boto:Data: 2024-04-16T20:05:58.995 DEBUG:boto:Headers: {} 2024-04-16T20:05:58.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:58.995 DEBUG:boto:Port: 8000 2024-04-16T20:05:58.995 DEBUG:boto:Params: {} 2024-04-16T20:05:58.996 DEBUG:boto:Token: None 2024-04-16T20:05:58.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:58 GMT /yrjefh-52/ 2024-04-16T20:05:58.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io= 2024-04-16T20:05:58.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/665mHWg24bwQa3+UFydMbU7io=', 'Content-Length': '0'} 2024-04-16T20:05:58.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd26f08a5a5bdade-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:58.999 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:58.999 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:59.000 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:59.000 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.000 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:05:59.000 DEBUG:boto:Data: 2024-04-16T20:05:59.000 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.000 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.000 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.000 DEBUG:boto:Params: {} 2024-04-16T20:05:59.000 DEBUG:boto:Token: None 2024-04-16T20:05:59.000 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj69 2024-04-16T20:05:59.001 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j9Slx3lJbPskKVnkHvxSuQNzBsw= 2024-04-16T20:05:59.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j9Slx3lJbPskKVnkHvxSuQNzBsw=', 'Content-Length': '0'} 2024-04-16T20:05:59.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000397d3ac93348d903-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.003 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:59.003 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:59.003 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.003 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:05:59.003 DEBUG:boto:Data: 2024-04-16T20:05:59.004 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.004 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.004 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.004 DEBUG:boto:Params: {} 2024-04-16T20:05:59.004 DEBUG:boto:Token: None 2024-04-16T20:05:59.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj69 2024-04-16T20:05:59.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j9Slx3lJbPskKVnkHvxSuQNzBsw= 2024-04-16T20:05:59.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j9Slx3lJbPskKVnkHvxSuQNzBsw=', 'Content-Length': '0'} 2024-04-16T20:05:59.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000322244d90723a872-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:05:59.006 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:59.006 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:59.007 DEBUG:boto:Method: GET 2024-04-16T20:05:59.007 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:05:59.007 DEBUG:boto:Data: 2024-04-16T20:05:59.007 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.007 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.007 DEBUG:boto:Params: {} 2024-04-16T20:05:59.007 DEBUG:boto:Token: None 2024-04-16T20:05:59.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj69 2024-04-16T20:05:59.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vDmZWsEAf/+HsQePZ12JysEfSzU= 2024-04-16T20:05:59.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vDmZWsEAf/+HsQePZ12JysEfSzU=', 'Content-Length': '0'} 2024-04-16T20:05:59.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eeb806b72cc9020-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.050 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:05:59.051 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:05:59.051 DEBUG:boto:Method: GET 2024-04-16T20:05:59.051 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:05:59.051 DEBUG:boto:Data: 2024-04-16T20:05:59.051 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.051 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.051 DEBUG:boto:Params: {} 2024-04-16T20:05:59.051 DEBUG:boto:Token: None 2024-04-16T20:05:59.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj69 2024-04-16T20:05:59.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vDmZWsEAf/+HsQePZ12JysEfSzU= 2024-04-16T20:05:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vDmZWsEAf/+HsQePZ12JysEfSzU=', 'Content-Length': '0'} 2024-04-16T20:05:59.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1552b25d7dfbc6-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.054 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.054 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.054 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.055 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.055 DEBUG:boto:Method: GET 2024-04-16T20:05:59.055 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.055 DEBUG:boto:Data: 2024-04-16T20:05:59.055 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.055 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.055 DEBUG:boto:Params: {} 2024-04-16T20:05:59.057 DEBUG:boto:Token: None 2024-04-16T20:05:59.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c398d9f9c379f3-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.102 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.103 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.103 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.103 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.103 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.103 DEBUG:boto:Method: GET 2024-04-16T20:05:59.103 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:05:59.103 DEBUG:boto:Data: 2024-04-16T20:05:59.103 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.103 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.103 DEBUG:boto:Params: {} 2024-04-16T20:05:59.104 DEBUG:boto:Token: None 2024-04-16T20:05:59.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2da3dd9e0c03a41-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.107 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:05:59.108 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.108 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.108 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.108 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.108 DEBUG:boto:Method: GET 2024-04-16T20:05:59.108 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.108 DEBUG:boto:Data: 2024-04-16T20:05:59.108 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.109 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.109 DEBUG:boto:Params: {} 2024-04-16T20:05:59.109 DEBUG:boto:Token: None 2024-04-16T20:05:59.109 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c= 2024-04-16T20:05:59.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c=', 'Content-Length': '0'} 2024-04-16T20:05:59.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c41784c8b7d406d-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.154 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.155 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.155 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.155 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.155 DEBUG:boto:Method: GET 2024-04-16T20:05:59.155 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.155 DEBUG:boto:Data: 2024-04-16T20:05:59.155 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.155 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.155 DEBUG:boto:Params: {} 2024-04-16T20:05:59.156 DEBUG:boto:Token: None 2024-04-16T20:05:59.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c= 2024-04-16T20:05:59.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c=', 'Content-Length': '0'} 2024-04-16T20:05:59.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1f23bb1049a532-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.158 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.158 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.158 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.159 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.159 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.159 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.159 DEBUG:boto:Data: 2024-04-16T20:05:59.159 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.159 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.159 DEBUG:boto:Params: {} 2024-04-16T20:05:59.159 DEBUG:boto:Token: None 2024-04-16T20:05:59.159 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iJFkW0/BXaqu1oEcDmR/1cJbIp4= 2024-04-16T20:05:59.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iJFkW0/BXaqu1oEcDmR/1cJbIp4=', 'Content-Length': '0'} 2024-04-16T20:05:59.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e631de8a0ff103-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.162 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.162 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.162 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.162 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.162 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.162 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.162 DEBUG:boto:Data: 2024-04-16T20:05:59.163 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.163 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.163 DEBUG:boto:Params: {} 2024-04-16T20:05:59.163 DEBUG:boto:Token: None 2024-04-16T20:05:59.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iJFkW0/BXaqu1oEcDmR/1cJbIp4= 2024-04-16T20:05:59.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iJFkW0/BXaqu1oEcDmR/1cJbIp4=', 'Content-Length': '0'} 2024-04-16T20:05:59.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007278db1aef0fe477-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:05:59.165 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.165 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.165 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.166 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.166 DEBUG:boto:Method: GET 2024-04-16T20:05:59.166 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.166 DEBUG:boto:Data: 2024-04-16T20:05:59.166 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.166 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.166 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.166 DEBUG:boto:Params: {} 2024-04-16T20:05:59.166 DEBUG:boto:Token: None 2024-04-16T20:05:59.166 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c= 2024-04-16T20:05:59.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c=', 'Content-Length': '0'} 2024-04-16T20:05:59.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888835f2aafd072f-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.210 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.210 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.210 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.210 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.211 DEBUG:boto:Method: GET 2024-04-16T20:05:59.211 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.211 DEBUG:boto:Data: 2024-04-16T20:05:59.211 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.211 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.211 DEBUG:boto:Params: {} 2024-04-16T20:05:59.211 DEBUG:boto:Token: None 2024-04-16T20:05:59.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:05:59.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c= 2024-04-16T20:05:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8lM46GvKWI5wF7XvDvwpDGRH44c=', 'Content-Length': '0'} 2024-04-16T20:05:59.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbbb8e673e2ba68-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.214 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.214 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.214 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.214 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.214 DEBUG:boto:Method: GET 2024-04-16T20:05:59.214 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.214 DEBUG:boto:Data: 2024-04-16T20:05:59.215 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.215 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.215 DEBUG:boto:Params: {} 2024-04-16T20:05:59.215 DEBUG:boto:Token: None 2024-04-16T20:05:59.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb765b3d20ea5a4a-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.258 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.259 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.259 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.259 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.259 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.259 DEBUG:boto:Method: GET 2024-04-16T20:05:59.259 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.259 DEBUG:boto:Data: 2024-04-16T20:05:59.259 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.259 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.259 DEBUG:boto:Params: {} 2024-04-16T20:05:59.260 DEBUG:boto:Token: None 2024-04-16T20:05:59.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f538b23d659ccbf2-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.263 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.263 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.263 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.264 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.264 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:05:59.264 DEBUG:boto:Data: 2024-04-16T20:05:59.264 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.264 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.264 DEBUG:boto:Params: {} 2024-04-16T20:05:59.264 DEBUG:boto:Token: None 2024-04-16T20:05:59.264 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76 2024-04-16T20:05:59.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NH68d0VVOrxIgvpRxr2ELwnoPVo= 2024-04-16T20:05:59.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NH68d0VVOrxIgvpRxr2ELwnoPVo=', 'Content-Length': '0'} 2024-04-16T20:05:59.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7f084f53f962ec-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.267 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.267 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.267 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.267 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:05:59.267 DEBUG:boto:Data: 2024-04-16T20:05:59.267 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.267 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.267 DEBUG:boto:Params: {} 2024-04-16T20:05:59.267 DEBUG:boto:Token: None 2024-04-16T20:05:59.268 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76 2024-04-16T20:05:59.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NH68d0VVOrxIgvpRxr2ELwnoPVo= 2024-04-16T20:05:59.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NH68d0VVOrxIgvpRxr2ELwnoPVo=', 'Content-Length': '0'} 2024-04-16T20:05:59.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4ae8e70cd898c6-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:05:59.270 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.270 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.270 DEBUG:boto:Method: GET 2024-04-16T20:05:59.271 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:05:59.271 DEBUG:boto:Data: 2024-04-16T20:05:59.271 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.271 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.271 DEBUG:boto:Params: {} 2024-04-16T20:05:59.271 DEBUG:boto:Token: None 2024-04-16T20:05:59.271 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76 2024-04-16T20:05:59.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TGmYQyeqLhYDoY2O81a9IbdbydQ= 2024-04-16T20:05:59.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGmYQyeqLhYDoY2O81a9IbdbydQ=', 'Content-Length': '0'} 2024-04-16T20:05:59.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba6abb3200a77edb-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.318 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:05:59.318 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:05:59.319 DEBUG:boto:Method: GET 2024-04-16T20:05:59.319 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:05:59.319 DEBUG:boto:Data: 2024-04-16T20:05:59.319 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.319 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.319 DEBUG:boto:Params: {} 2024-04-16T20:05:59.319 DEBUG:boto:Token: None 2024-04-16T20:05:59.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj76 2024-04-16T20:05:59.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TGmYQyeqLhYDoY2O81a9IbdbydQ= 2024-04-16T20:05:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGmYQyeqLhYDoY2O81a9IbdbydQ=', 'Content-Length': '0'} 2024-04-16T20:05:59.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002f14a8cd2e5b6c7-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.322 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.322 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.322 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.322 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.322 DEBUG:boto:Method: GET 2024-04-16T20:05:59.322 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.322 DEBUG:boto:Data: 2024-04-16T20:05:59.322 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.323 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.323 DEBUG:boto:Params: {} 2024-04-16T20:05:59.323 DEBUG:boto:Token: None 2024-04-16T20:05:59.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903e6d3223ee85f3-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.366 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.367 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.367 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.367 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.367 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.367 DEBUG:boto:Method: GET 2024-04-16T20:05:59.367 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:05:59.367 DEBUG:boto:Data: 2024-04-16T20:05:59.367 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.367 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.367 DEBUG:boto:Params: {} 2024-04-16T20:05:59.368 DEBUG:boto:Token: None 2024-04-16T20:05:59.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007159060b51d55e55-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.371 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:05:59.372 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.372 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.372 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.372 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.372 DEBUG:boto:Method: GET 2024-04-16T20:05:59.372 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.372 DEBUG:boto:Data: 2024-04-16T20:05:59.372 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.373 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.373 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.373 DEBUG:boto:Params: {} 2024-04-16T20:05:59.373 DEBUG:boto:Token: None 2024-04-16T20:05:59.373 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo= 2024-04-16T20:05:59.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo=', 'Content-Length': '0'} 2024-04-16T20:05:59.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007013ab34f64d5786-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.418 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.418 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.418 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.419 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.419 DEBUG:boto:Method: GET 2024-04-16T20:05:59.419 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.419 DEBUG:boto:Data: 2024-04-16T20:05:59.419 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.419 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.419 DEBUG:boto:Params: {} 2024-04-16T20:05:59.419 DEBUG:boto:Token: None 2024-04-16T20:05:59.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo= 2024-04-16T20:05:59.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo=', 'Content-Length': '0'} 2024-04-16T20:05:59.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d18473b557a7661-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.422 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.422 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.422 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.422 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.422 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.422 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.422 DEBUG:boto:Data: 2024-04-16T20:05:59.423 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.423 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.423 DEBUG:boto:Params: {} 2024-04-16T20:05:59.423 DEBUG:boto:Token: None 2024-04-16T20:05:59.423 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ikzsCsxMVEg0mre0XCwk9fNqeeU= 2024-04-16T20:05:59.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ikzsCsxMVEg0mre0XCwk9fNqeeU=', 'Content-Length': '0'} 2024-04-16T20:05:59.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081df74ea636dda65-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.426 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.426 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.426 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.426 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.426 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.426 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.426 DEBUG:boto:Data: 2024-04-16T20:05:59.426 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.426 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.426 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.427 DEBUG:boto:Params: {} 2024-04-16T20:05:59.427 DEBUG:boto:Token: None 2024-04-16T20:05:59.427 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ikzsCsxMVEg0mre0XCwk9fNqeeU= 2024-04-16T20:05:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ikzsCsxMVEg0mre0XCwk9fNqeeU=', 'Content-Length': '0'} 2024-04-16T20:05:59.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04b71ea511d536e-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:05:59.429 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.429 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.429 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.429 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.430 DEBUG:boto:Method: GET 2024-04-16T20:05:59.430 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.430 DEBUG:boto:Data: 2024-04-16T20:05:59.430 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.430 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.430 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.430 DEBUG:boto:Params: {} 2024-04-16T20:05:59.430 DEBUG:boto:Token: None 2024-04-16T20:05:59.430 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo= 2024-04-16T20:05:59.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo=', 'Content-Length': '0'} 2024-04-16T20:05:59.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e7a376d39dec26-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.478 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.478 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.479 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.479 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.479 DEBUG:boto:Method: GET 2024-04-16T20:05:59.479 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.479 DEBUG:boto:Data: 2024-04-16T20:05:59.479 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.479 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.479 DEBUG:boto:Params: {} 2024-04-16T20:05:59.480 DEBUG:boto:Token: None 2024-04-16T20:05:59.480 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:05:59.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo= 2024-04-16T20:05:59.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ht8TC1jybaOoPwAPypO9ahmHAQo=', 'Content-Length': '0'} 2024-04-16T20:05:59.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbcd0dbd776ac66-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.482 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.483 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.483 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.483 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.483 DEBUG:boto:Method: GET 2024-04-16T20:05:59.483 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.483 DEBUG:boto:Data: 2024-04-16T20:05:59.483 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.483 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.483 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.483 DEBUG:boto:Params: {} 2024-04-16T20:05:59.484 DEBUG:boto:Token: None 2024-04-16T20:05:59.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100d10b95356143c-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.530 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.531 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.531 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.531 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.531 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.531 DEBUG:boto:Method: GET 2024-04-16T20:05:59.531 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.531 DEBUG:boto:Data: 2024-04-16T20:05:59.532 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.532 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.532 DEBUG:boto:Params: {} 2024-04-16T20:05:59.532 DEBUG:boto:Token: None 2024-04-16T20:05:59.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1d67169841f4ae-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.536 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.537 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.537 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.537 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.537 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:05:59.537 DEBUG:boto:Data: 2024-04-16T20:05:59.537 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.537 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.537 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.538 DEBUG:boto:Params: {} 2024-04-16T20:05:59.538 DEBUG:boto:Token: None 2024-04-16T20:05:59.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85 2024-04-16T20:05:59.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a90BaZ4eMdLSDqxpeP34Sk8ypo4= 2024-04-16T20:05:59.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a90BaZ4eMdLSDqxpeP34Sk8ypo4=', 'Content-Length': '0'} 2024-04-16T20:05:59.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804511b76c0a1302-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.540 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.540 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.540 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.540 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:05:59.540 DEBUG:boto:Data: 2024-04-16T20:05:59.540 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.541 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.541 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.541 DEBUG:boto:Params: {} 2024-04-16T20:05:59.541 DEBUG:boto:Token: None 2024-04-16T20:05:59.541 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85 2024-04-16T20:05:59.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a90BaZ4eMdLSDqxpeP34Sk8ypo4= 2024-04-16T20:05:59.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a90BaZ4eMdLSDqxpeP34Sk8ypo4=', 'Content-Length': '0'} 2024-04-16T20:05:59.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cdcff89eedb320-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:05:59.543 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.543 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.543 DEBUG:boto:Method: GET 2024-04-16T20:05:59.543 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:05:59.543 DEBUG:boto:Data: 2024-04-16T20:05:59.544 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.544 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.544 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.544 DEBUG:boto:Params: {} 2024-04-16T20:05:59.544 DEBUG:boto:Token: None 2024-04-16T20:05:59.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85 2024-04-16T20:05:59.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sLaWL+/NlpC2W/GKW0MPovh7M34= 2024-04-16T20:05:59.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sLaWL+/NlpC2W/GKW0MPovh7M34=', 'Content-Length': '0'} 2024-04-16T20:05:59.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047dc3867768694b9-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.586 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:05:59.586 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:05:59.587 DEBUG:boto:Method: GET 2024-04-16T20:05:59.587 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:05:59.587 DEBUG:boto:Data: 2024-04-16T20:05:59.587 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.587 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.587 DEBUG:boto:Params: {} 2024-04-16T20:05:59.587 DEBUG:boto:Token: None 2024-04-16T20:05:59.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj85 2024-04-16T20:05:59.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sLaWL+/NlpC2W/GKW0MPovh7M34= 2024-04-16T20:05:59.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sLaWL+/NlpC2W/GKW0MPovh7M34=', 'Content-Length': '0'} 2024-04-16T20:05:59.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c0e6ea49ecbe021-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.590 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.590 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.590 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.590 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.590 DEBUG:boto:Method: GET 2024-04-16T20:05:59.591 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.591 DEBUG:boto:Data: 2024-04-16T20:05:59.591 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.591 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.591 DEBUG:boto:Params: {} 2024-04-16T20:05:59.591 DEBUG:boto:Token: None 2024-04-16T20:05:59.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001890c6d0af96c643-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.594 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.594 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.595 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.595 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.595 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.595 DEBUG:boto:Method: GET 2024-04-16T20:05:59.595 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:05:59.595 DEBUG:boto:Data: 2024-04-16T20:05:59.595 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.595 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.595 DEBUG:boto:Params: {} 2024-04-16T20:05:59.596 DEBUG:boto:Token: None 2024-04-16T20:05:59.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001142483ce8206bfc-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.599 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:05:59.600 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.600 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.600 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.600 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.600 DEBUG:boto:Method: GET 2024-04-16T20:05:59.600 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.601 DEBUG:boto:Data: 2024-04-16T20:05:59.601 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.601 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.601 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.601 DEBUG:boto:Params: {} 2024-04-16T20:05:59.601 DEBUG:boto:Token: None 2024-04-16T20:05:59.601 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.601 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI= 2024-04-16T20:05:59.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI=', 'Content-Length': '0'} 2024-04-16T20:05:59.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c4ac856d223c300-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.646 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.646 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.647 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.647 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.647 DEBUG:boto:Method: GET 2024-04-16T20:05:59.647 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.647 DEBUG:boto:Data: 2024-04-16T20:05:59.647 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.647 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.647 DEBUG:boto:Params: {} 2024-04-16T20:05:59.647 DEBUG:boto:Token: None 2024-04-16T20:05:59.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI= 2024-04-16T20:05:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI=', 'Content-Length': '0'} 2024-04-16T20:05:59.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ab9977012490ec-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.650 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.650 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.651 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.651 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.651 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.651 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.651 DEBUG:boto:Data: 2024-04-16T20:05:59.651 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.651 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.651 DEBUG:boto:Params: {} 2024-04-16T20:05:59.651 DEBUG:boto:Token: None 2024-04-16T20:05:59.652 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B2UAPp3QNNk7NsUmVegrlFyCSfA= 2024-04-16T20:05:59.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B2UAPp3QNNk7NsUmVegrlFyCSfA=', 'Content-Length': '0'} 2024-04-16T20:05:59.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085041020076f5d3b-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.654 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.655 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.655 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.655 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.655 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.655 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.655 DEBUG:boto:Data: 2024-04-16T20:05:59.655 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.655 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.656 DEBUG:boto:Params: {} 2024-04-16T20:05:59.656 DEBUG:boto:Token: None 2024-04-16T20:05:59.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B2UAPp3QNNk7NsUmVegrlFyCSfA= 2024-04-16T20:05:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B2UAPp3QNNk7NsUmVegrlFyCSfA=', 'Content-Length': '0'} 2024-04-16T20:05:59.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7e3ca29386069b-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:05:59.659 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.659 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.659 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.659 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.659 DEBUG:boto:Method: GET 2024-04-16T20:05:59.659 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.659 DEBUG:boto:Data: 2024-04-16T20:05:59.659 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.660 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.660 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.660 DEBUG:boto:Params: {} 2024-04-16T20:05:59.660 DEBUG:boto:Token: None 2024-04-16T20:05:59.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI= 2024-04-16T20:05:59.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI=', 'Content-Length': '0'} 2024-04-16T20:05:59.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089de730d2f2f41b7-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.706 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.707 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.707 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.707 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.707 DEBUG:boto:Method: GET 2024-04-16T20:05:59.707 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.707 DEBUG:boto:Data: 2024-04-16T20:05:59.707 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.707 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.707 DEBUG:boto:Params: {} 2024-04-16T20:05:59.708 DEBUG:boto:Token: None 2024-04-16T20:05:59.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:05:59.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI= 2024-04-16T20:05:59.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4/DsDUlYwish3zslsV81D31eXBI=', 'Content-Length': '0'} 2024-04-16T20:05:59.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0191c8b60e0b6b5-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.710 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.710 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.711 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.711 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.711 DEBUG:boto:Method: GET 2024-04-16T20:05:59.711 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.711 DEBUG:boto:Data: 2024-04-16T20:05:59.711 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.711 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.711 DEBUG:boto:Params: {} 2024-04-16T20:05:59.711 DEBUG:boto:Token: None 2024-04-16T20:05:59.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c09398fb3206691-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.715 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.715 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.715 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.715 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.715 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.716 DEBUG:boto:Method: GET 2024-04-16T20:05:59.716 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.716 DEBUG:boto:Data: 2024-04-16T20:05:59.716 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.716 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.716 DEBUG:boto:Params: {} 2024-04-16T20:05:59.716 DEBUG:boto:Token: None 2024-04-16T20:05:59.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc42f04bec53cc63-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.720 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.720 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.721 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.721 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.721 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:05:59.721 DEBUG:boto:Data: 2024-04-16T20:05:59.721 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.721 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.721 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.721 DEBUG:boto:Params: {} 2024-04-16T20:05:59.721 DEBUG:boto:Token: None 2024-04-16T20:05:59.722 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87 2024-04-16T20:05:59.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PhIzfSgHJPkau3U1Tu3nxbNhtSA= 2024-04-16T20:05:59.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PhIzfSgHJPkau3U1Tu3nxbNhtSA=', 'Content-Length': '0'} 2024-04-16T20:05:59.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763196910a637a64-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.724 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.724 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.724 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.724 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:05:59.725 DEBUG:boto:Data: 2024-04-16T20:05:59.725 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.725 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.725 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.725 DEBUG:boto:Params: {} 2024-04-16T20:05:59.725 DEBUG:boto:Token: None 2024-04-16T20:05:59.725 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87 2024-04-16T20:05:59.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PhIzfSgHJPkau3U1Tu3nxbNhtSA= 2024-04-16T20:05:59.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PhIzfSgHJPkau3U1Tu3nxbNhtSA=', 'Content-Length': '0'} 2024-04-16T20:05:59.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a028ec913cfe67e-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:05:59.728 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.728 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.728 DEBUG:boto:Method: GET 2024-04-16T20:05:59.728 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:05:59.728 DEBUG:boto:Data: 2024-04-16T20:05:59.728 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.728 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.729 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.729 DEBUG:boto:Params: {} 2024-04-16T20:05:59.729 DEBUG:boto:Token: None 2024-04-16T20:05:59.729 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87 2024-04-16T20:05:59.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORfVO8eYE6PktkrNrpuk2rCsFLI= 2024-04-16T20:05:59.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORfVO8eYE6PktkrNrpuk2rCsFLI=', 'Content-Length': '0'} 2024-04-16T20:05:59.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f99581f1a9128b-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.774 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:05:59.774 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:05:59.775 DEBUG:boto:Method: GET 2024-04-16T20:05:59.775 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:05:59.775 DEBUG:boto:Data: 2024-04-16T20:05:59.775 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.775 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.775 DEBUG:boto:Params: {} 2024-04-16T20:05:59.775 DEBUG:boto:Token: None 2024-04-16T20:05:59.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj87 2024-04-16T20:05:59.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ORfVO8eYE6PktkrNrpuk2rCsFLI= 2024-04-16T20:05:59.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ORfVO8eYE6PktkrNrpuk2rCsFLI=', 'Content-Length': '0'} 2024-04-16T20:05:59.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06ef5c3c7d918d2-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.778 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.778 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.778 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.778 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.779 DEBUG:boto:Method: GET 2024-04-16T20:05:59.779 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.779 DEBUG:boto:Data: 2024-04-16T20:05:59.779 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.779 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.779 DEBUG:boto:Params: {} 2024-04-16T20:05:59.779 DEBUG:boto:Token: None 2024-04-16T20:05:59.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62508c39a3461a0-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.826 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.827 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.827 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.827 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.827 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.827 DEBUG:boto:Method: GET 2024-04-16T20:05:59.827 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:05:59.827 DEBUG:boto:Data: 2024-04-16T20:05:59.828 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.828 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.828 DEBUG:boto:Params: {} 2024-04-16T20:05:59.828 DEBUG:boto:Token: None 2024-04-16T20:05:59.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004394fe81c74f21d9-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.832 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:05:59.832 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.833 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.833 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.833 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.833 DEBUG:boto:Method: GET 2024-04-16T20:05:59.833 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.833 DEBUG:boto:Data: 2024-04-16T20:05:59.833 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.833 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.833 DEBUG:boto:Params: {} 2024-04-16T20:05:59.834 DEBUG:boto:Token: None 2024-04-16T20:05:59.834 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA= 2024-04-16T20:05:59.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA=', 'Content-Length': '0'} 2024-04-16T20:05:59.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32b1ba819a05fa7-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.878 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.879 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.879 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.879 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.879 DEBUG:boto:Method: GET 2024-04-16T20:05:59.879 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.879 DEBUG:boto:Data: 2024-04-16T20:05:59.879 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.879 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.879 DEBUG:boto:Params: {} 2024-04-16T20:05:59.880 DEBUG:boto:Token: None 2024-04-16T20:05:59.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA= 2024-04-16T20:05:59.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA=', 'Content-Length': '0'} 2024-04-16T20:05:59.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0bd251c80b3088-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.882 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.882 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.883 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.883 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.883 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.883 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.883 DEBUG:boto:Data: 2024-04-16T20:05:59.883 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.883 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.883 DEBUG:boto:Params: {} 2024-04-16T20:05:59.883 DEBUG:boto:Token: None 2024-04-16T20:05:59.884 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0dhlJBIux+JT8vL8l9pKpJ88Oig= 2024-04-16T20:05:59.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0dhlJBIux+JT8vL8l9pKpJ88Oig=', 'Content-Length': '0'} 2024-04-16T20:05:59.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ce4a767bbdede9-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.886 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.886 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.886 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.886 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.886 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.886 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.886 DEBUG:boto:Data: 2024-04-16T20:05:59.887 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.887 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.887 DEBUG:boto:Params: {} 2024-04-16T20:05:59.887 DEBUG:boto:Token: None 2024-04-16T20:05:59.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0dhlJBIux+JT8vL8l9pKpJ88Oig= 2024-04-16T20:05:59.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0dhlJBIux+JT8vL8l9pKpJ88Oig=', 'Content-Length': '0'} 2024-04-16T20:05:59.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a98ff91bb062e5-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:05:59.890 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.890 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.890 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.890 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.890 DEBUG:boto:Method: GET 2024-04-16T20:05:59.890 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.890 DEBUG:boto:Data: 2024-04-16T20:05:59.890 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.890 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.890 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.891 DEBUG:boto:Params: {} 2024-04-16T20:05:59.891 DEBUG:boto:Token: None 2024-04-16T20:05:59.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA= 2024-04-16T20:05:59.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA=', 'Content-Length': '0'} 2024-04-16T20:05:59.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae3f53c8c648edf-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.934 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.934 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.935 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.935 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.935 DEBUG:boto:Method: GET 2024-04-16T20:05:59.935 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.935 DEBUG:boto:Data: 2024-04-16T20:05:59.935 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.935 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.935 DEBUG:boto:Params: {} 2024-04-16T20:05:59.935 DEBUG:boto:Token: None 2024-04-16T20:05:59.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:05:59.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA= 2024-04-16T20:05:59.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fDa/5+FPONsGni0IERN5T8rRFrA=', 'Content-Length': '0'} 2024-04-16T20:05:59.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da142df05d5ac933-00661eda27-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.938 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.938 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.938 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.939 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.939 DEBUG:boto:Method: GET 2024-04-16T20:05:59.939 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.939 DEBUG:boto:Data: 2024-04-16T20:05:59.939 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.939 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.939 DEBUG:boto:Params: {} 2024-04-16T20:05:59.939 DEBUG:boto:Token: None 2024-04-16T20:05:59.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f919672004d615-00661eda27-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.982 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.983 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:05:59.983 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:05:59.983 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.983 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.983 DEBUG:boto:Method: GET 2024-04-16T20:05:59.983 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:05:59.983 DEBUG:boto:Data: 2024-04-16T20:05:59.983 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.984 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.984 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.984 DEBUG:boto:Params: {} 2024-04-16T20:05:59.984 DEBUG:boto:Token: None 2024-04-16T20:05:59.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/ 2024-04-16T20:05:59.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4= 2024-04-16T20:05:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cFmLFhVqCxXXtBdU/J6ZvdhPbk4=', 'Content-Length': '0'} 2024-04-16T20:05:59.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b5be28e8d801f0e-00661eda27-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.988 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:05:59.988 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.988 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.989 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.989 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:05:59.989 DEBUG:boto:Data: 2024-04-16T20:05:59.989 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.989 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.989 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.989 DEBUG:boto:Params: {} 2024-04-16T20:05:59.989 DEBUG:boto:Token: None 2024-04-16T20:05:59.989 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92 2024-04-16T20:05:59.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gK/KLx7cm+I2wZhWFp3/tZB68+0= 2024-04-16T20:05:59.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gK/KLx7cm+I2wZhWFp3/tZB68+0=', 'Content-Length': '0'} 2024-04-16T20:05:59.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c285e60dbf70c06b-00661eda27-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:05:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.992 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.992 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.992 DEBUG:boto:Method: HEAD 2024-04-16T20:05:59.992 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:05:59.992 DEBUG:boto:Data: 2024-04-16T20:05:59.992 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:05:59.992 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.993 DEBUG:boto:Params: {} 2024-04-16T20:05:59.993 DEBUG:boto:Token: None 2024-04-16T20:05:59.993 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92 2024-04-16T20:05:59.993 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gK/KLx7cm+I2wZhWFp3/tZB68+0= 2024-04-16T20:05:59.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gK/KLx7cm+I2wZhWFp3/tZB68+0=', 'Content-Length': '0'} 2024-04-16T20:05:59.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec4a5569d735b326-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:05:59.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:05:59.995 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:05:59.995 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:05:59.995 DEBUG:boto:Method: GET 2024-04-16T20:05:59.995 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:05:59.996 DEBUG:boto:Data: 2024-04-16T20:05:59.996 DEBUG:boto:Headers: {} 2024-04-16T20:05:59.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:05:59.996 DEBUG:boto:Port: 8000 2024-04-16T20:05:59.996 DEBUG:boto:Params: {} 2024-04-16T20:05:59.996 DEBUG:boto:Token: None 2024-04-16T20:05:59.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:05:59 GMT /yrjefh-52/obj92 2024-04-16T20:05:59.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QkUOhg6/Sf9Zc68blinjPm0g4bc= 2024-04-16T20:05:59.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:05:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QkUOhg6/Sf9Zc68blinjPm0g4bc=', 'Content-Length': '0'} 2024-04-16T20:05:59.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d001bab303183f1-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.042 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:06:00.043 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:06:00.043 DEBUG:boto:Method: GET 2024-04-16T20:06:00.043 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:06:00.043 DEBUG:boto:Data: 2024-04-16T20:06:00.043 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.043 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.043 DEBUG:boto:Params: {} 2024-04-16T20:06:00.044 DEBUG:boto:Token: None 2024-04-16T20:06:00.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj92 2024-04-16T20:06:00.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HyMltppY+23rRaCUFIq8wm6MDfQ= 2024-04-16T20:06:00.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HyMltppY+23rRaCUFIq8wm6MDfQ=', 'Content-Length': '0'} 2024-04-16T20:06:00.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433e9e51fb1f0546-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.046 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.046 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.046 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.047 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.047 DEBUG:boto:Method: GET 2024-04-16T20:06:00.047 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.047 DEBUG:boto:Data: 2024-04-16T20:06:00.047 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.047 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.047 DEBUG:boto:Params: {} 2024-04-16T20:06:00.047 DEBUG:boto:Token: None 2024-04-16T20:06:00.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c272aeab648b565e-00661eda28-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.050 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.051 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.051 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.051 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.051 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.051 DEBUG:boto:Method: GET 2024-04-16T20:06:00.051 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:06:00.051 DEBUG:boto:Data: 2024-04-16T20:06:00.051 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.051 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.052 DEBUG:boto:Params: {} 2024-04-16T20:06:00.052 DEBUG:boto:Token: None 2024-04-16T20:06:00.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c1902840badbcc-00661eda28-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.056 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:06:00.056 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.056 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.056 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.056 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.056 DEBUG:boto:Method: GET 2024-04-16T20:06:00.056 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.057 DEBUG:boto:Data: 2024-04-16T20:06:00.057 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.057 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.057 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.057 DEBUG:boto:Params: {} 2024-04-16T20:06:00.057 DEBUG:boto:Token: None 2024-04-16T20:06:00.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw= 2024-04-16T20:06:00.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw=', 'Content-Length': '0'} 2024-04-16T20:06:00.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008642aa9f6c9745dd-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.102 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.103 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.103 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.103 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.103 DEBUG:boto:Method: GET 2024-04-16T20:06:00.103 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.103 DEBUG:boto:Data: 2024-04-16T20:06:00.103 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.103 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.104 DEBUG:boto:Params: {} 2024-04-16T20:06:00.104 DEBUG:boto:Token: None 2024-04-16T20:06:00.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw= 2024-04-16T20:06:00.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw=', 'Content-Length': '0'} 2024-04-16T20:06:00.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005efeb3ef4bb4f9c4-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.107 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.107 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.107 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.107 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.107 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.107 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.107 DEBUG:boto:Data: 2024-04-16T20:06:00.108 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.108 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.108 DEBUG:boto:Params: {} 2024-04-16T20:06:00.108 DEBUG:boto:Token: None 2024-04-16T20:06:00.108 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3GUwPxZGh+v/8uhVyBqguJO7A1Q= 2024-04-16T20:06:00.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3GUwPxZGh+v/8uhVyBqguJO7A1Q=', 'Content-Length': '0'} 2024-04-16T20:06:00.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a07a3cb6f2abe7-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.110 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.110 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.110 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.111 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.111 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.111 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.111 DEBUG:boto:Data: 2024-04-16T20:06:00.111 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.111 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.111 DEBUG:boto:Params: {} 2024-04-16T20:06:00.111 DEBUG:boto:Token: None 2024-04-16T20:06:00.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3GUwPxZGh+v/8uhVyBqguJO7A1Q= 2024-04-16T20:06:00.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3GUwPxZGh+v/8uhVyBqguJO7A1Q=', 'Content-Length': '0'} 2024-04-16T20:06:00.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce56ce7210d053e-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:06:00.114 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.114 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.114 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.115 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.115 DEBUG:boto:Method: GET 2024-04-16T20:06:00.115 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.115 DEBUG:boto:Data: 2024-04-16T20:06:00.115 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.115 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.115 DEBUG:boto:Params: {} 2024-04-16T20:06:00.115 DEBUG:boto:Token: None 2024-04-16T20:06:00.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw= 2024-04-16T20:06:00.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw=', 'Content-Length': '0'} 2024-04-16T20:06:00.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000202f1e476a1db9d4-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.158 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.158 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.158 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.159 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.159 DEBUG:boto:Method: GET 2024-04-16T20:06:00.159 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.159 DEBUG:boto:Data: 2024-04-16T20:06:00.159 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.159 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.159 DEBUG:boto:Params: {} 2024-04-16T20:06:00.159 DEBUG:boto:Token: None 2024-04-16T20:06:00.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:06:00.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw= 2024-04-16T20:06:00.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gt5mXu+X7RQFBaz0RLsI26IIkUw=', 'Content-Length': '0'} 2024-04-16T20:06:00.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c45b4a7b7983d27-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.162 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.162 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.162 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.162 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.162 DEBUG:boto:Method: GET 2024-04-16T20:06:00.162 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.162 DEBUG:boto:Data: 2024-04-16T20:06:00.162 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.163 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.163 DEBUG:boto:Params: {} 2024-04-16T20:06:00.163 DEBUG:boto:Token: None 2024-04-16T20:06:00.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008350743820bee14-00661eda28-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.166 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.166 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.166 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.166 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.166 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.167 DEBUG:boto:Method: GET 2024-04-16T20:06:00.167 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.167 DEBUG:boto:Data: 2024-04-16T20:06:00.167 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.167 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.167 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.167 DEBUG:boto:Params: {} 2024-04-16T20:06:00.167 DEBUG:boto:Token: None 2024-04-16T20:06:00.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd93c2d6296d8946-00661eda28-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.171 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.171 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.171 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.172 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.172 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:06:00.172 DEBUG:boto:Data: 2024-04-16T20:06:00.172 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.172 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.172 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.172 DEBUG:boto:Params: {} 2024-04-16T20:06:00.172 DEBUG:boto:Token: None 2024-04-16T20:06:00.172 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93 2024-04-16T20:06:00.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UAR0Rlli+AZBjLxdKDdqfYx9zXc= 2024-04-16T20:06:00.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UAR0Rlli+AZBjLxdKDdqfYx9zXc=', 'Content-Length': '0'} 2024-04-16T20:06:00.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690eefe48314f98e-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.175 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.175 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.175 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.175 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:06:00.175 DEBUG:boto:Data: 2024-04-16T20:06:00.175 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.176 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.176 DEBUG:boto:Params: {} 2024-04-16T20:06:00.176 DEBUG:boto:Token: None 2024-04-16T20:06:00.176 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93 2024-04-16T20:06:00.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UAR0Rlli+AZBjLxdKDdqfYx9zXc= 2024-04-16T20:06:00.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UAR0Rlli+AZBjLxdKDdqfYx9zXc=', 'Content-Length': '0'} 2024-04-16T20:06:00.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ffef4af56ea445a-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:06:00.179 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.179 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.179 DEBUG:boto:Method: GET 2024-04-16T20:06:00.179 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:06:00.179 DEBUG:boto:Data: 2024-04-16T20:06:00.179 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.179 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.179 DEBUG:boto:Params: {} 2024-04-16T20:06:00.179 DEBUG:boto:Token: None 2024-04-16T20:06:00.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93 2024-04-16T20:06:00.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ff3oxsbSp048tj/Yvr5aBDiNyCw= 2024-04-16T20:06:00.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ff3oxsbSp048tj/Yvr5aBDiNyCw=', 'Content-Length': '0'} 2024-04-16T20:06:00.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005055d36723fe29c5-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.222 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:06:00.223 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:06:00.223 DEBUG:boto:Method: GET 2024-04-16T20:06:00.223 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:06:00.223 DEBUG:boto:Data: 2024-04-16T20:06:00.223 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.223 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.223 DEBUG:boto:Params: {} 2024-04-16T20:06:00.223 DEBUG:boto:Token: None 2024-04-16T20:06:00.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj93 2024-04-16T20:06:00.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ff3oxsbSp048tj/Yvr5aBDiNyCw= 2024-04-16T20:06:00.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ff3oxsbSp048tj/Yvr5aBDiNyCw=', 'Content-Length': '0'} 2024-04-16T20:06:00.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071ebb3c48b21c4b-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.226 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.226 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.227 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.227 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.227 DEBUG:boto:Method: GET 2024-04-16T20:06:00.227 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.227 DEBUG:boto:Data: 2024-04-16T20:06:00.227 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.227 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.227 DEBUG:boto:Params: {} 2024-04-16T20:06:00.227 DEBUG:boto:Token: None 2024-04-16T20:06:00.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c08ea51627b9505c-00661eda28-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.274 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.275 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.275 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.275 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.275 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.275 DEBUG:boto:Method: GET 2024-04-16T20:06:00.275 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:06:00.275 DEBUG:boto:Data: 2024-04-16T20:06:00.275 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.275 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.275 DEBUG:boto:Params: {} 2024-04-16T20:06:00.276 DEBUG:boto:Token: None 2024-04-16T20:06:00.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059fb9b88435331da-00661eda28-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.282 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:06:00.283 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.283 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.283 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.283 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.283 DEBUG:boto:Method: GET 2024-04-16T20:06:00.284 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.284 DEBUG:boto:Data: 2024-04-16T20:06:00.284 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.284 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.284 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.284 DEBUG:boto:Params: {} 2024-04-16T20:06:00.284 DEBUG:boto:Token: None 2024-04-16T20:06:00.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM= 2024-04-16T20:06:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM=', 'Content-Length': '0'} 2024-04-16T20:06:00.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9c7775745f80ca-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.330 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.331 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.331 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.331 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.331 DEBUG:boto:Method: GET 2024-04-16T20:06:00.331 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.331 DEBUG:boto:Data: 2024-04-16T20:06:00.331 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.332 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.332 DEBUG:boto:Params: {} 2024-04-16T20:06:00.332 DEBUG:boto:Token: None 2024-04-16T20:06:00.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM= 2024-04-16T20:06:00.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM=', 'Content-Length': '0'} 2024-04-16T20:06:00.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555b76ef7a4e4de1-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.334 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.335 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.335 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.335 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.335 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.335 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.335 DEBUG:boto:Data: 2024-04-16T20:06:00.335 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.335 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.335 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.335 DEBUG:boto:Params: {} 2024-04-16T20:06:00.336 DEBUG:boto:Token: None 2024-04-16T20:06:00.336 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmXGMNa8C13g/VT5gZ3tAUU8QRU= 2024-04-16T20:06:00.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmXGMNa8C13g/VT5gZ3tAUU8QRU=', 'Content-Length': '0'} 2024-04-16T20:06:00.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc763a511a07b0a0-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.338 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.338 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.338 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.338 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.338 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.339 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.339 DEBUG:boto:Data: 2024-04-16T20:06:00.339 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.339 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.339 DEBUG:boto:Params: {} 2024-04-16T20:06:00.339 DEBUG:boto:Token: None 2024-04-16T20:06:00.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wmXGMNa8C13g/VT5gZ3tAUU8QRU= 2024-04-16T20:06:00.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wmXGMNa8C13g/VT5gZ3tAUU8QRU=', 'Content-Length': '0'} 2024-04-16T20:06:00.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792f17ba20c30ff5-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:06:00.342 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.342 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.342 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.342 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.342 DEBUG:boto:Method: GET 2024-04-16T20:06:00.342 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.342 DEBUG:boto:Data: 2024-04-16T20:06:00.342 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.343 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.343 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.343 DEBUG:boto:Params: {} 2024-04-16T20:06:00.343 DEBUG:boto:Token: None 2024-04-16T20:06:00.343 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM= 2024-04-16T20:06:00.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM=', 'Content-Length': '0'} 2024-04-16T20:06:00.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9406855621fd2b-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.386 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.386 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.386 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.387 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.387 DEBUG:boto:Method: GET 2024-04-16T20:06:00.387 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.387 DEBUG:boto:Data: 2024-04-16T20:06:00.387 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.387 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.387 DEBUG:boto:Params: {} 2024-04-16T20:06:00.387 DEBUG:boto:Token: None 2024-04-16T20:06:00.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:06:00.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM= 2024-04-16T20:06:00.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:st3VharSlGrUb3By/ThVKfwqQYM=', 'Content-Length': '0'} 2024-04-16T20:06:00.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9da65d7bb7c5ce-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.390 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.390 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.390 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.390 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.390 DEBUG:boto:Method: GET 2024-04-16T20:06:00.390 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.390 DEBUG:boto:Data: 2024-04-16T20:06:00.390 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.390 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.391 DEBUG:boto:Params: {} 2024-04-16T20:06:00.391 DEBUG:boto:Token: None 2024-04-16T20:06:00.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1c49eb9001a080-00661eda28-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.434 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.435 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.435 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.435 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.435 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.435 DEBUG:boto:Method: GET 2024-04-16T20:06:00.435 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.435 DEBUG:boto:Data: 2024-04-16T20:06:00.435 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.436 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.436 DEBUG:boto:Params: {} 2024-04-16T20:06:00.436 DEBUG:boto:Token: None 2024-04-16T20:06:00.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dfdcf5b6cb3c5d-00661eda28-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.440 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.440 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.440 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.440 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.440 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:06:00.440 DEBUG:boto:Data: 2024-04-16T20:06:00.440 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.441 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.441 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.441 DEBUG:boto:Params: {} 2024-04-16T20:06:00.441 DEBUG:boto:Token: None 2024-04-16T20:06:00.441 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95 2024-04-16T20:06:00.441 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fO3dWbJLIjHM+QOd/0yCGIFA06Y= 2024-04-16T20:06:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fO3dWbJLIjHM+QOd/0yCGIFA06Y=', 'Content-Length': '0'} 2024-04-16T20:06:00.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f3e8315e52c9f3-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.443 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.443 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.443 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.443 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:06:00.443 DEBUG:boto:Data: 2024-04-16T20:06:00.444 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.444 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.444 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.444 DEBUG:boto:Params: {} 2024-04-16T20:06:00.444 DEBUG:boto:Token: None 2024-04-16T20:06:00.444 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95 2024-04-16T20:06:00.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fO3dWbJLIjHM+QOd/0yCGIFA06Y= 2024-04-16T20:06:00.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fO3dWbJLIjHM+QOd/0yCGIFA06Y=', 'Content-Length': '0'} 2024-04-16T20:06:00.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94ab821abd51905-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:06:00.446 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.446 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.447 DEBUG:boto:Method: GET 2024-04-16T20:06:00.447 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:06:00.447 DEBUG:boto:Data: 2024-04-16T20:06:00.447 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.447 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.447 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.447 DEBUG:boto:Params: {} 2024-04-16T20:06:00.447 DEBUG:boto:Token: None 2024-04-16T20:06:00.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95 2024-04-16T20:06:00.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iCNmO5MbImVpqDuqpuip6FBmh/s= 2024-04-16T20:06:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iCNmO5MbImVpqDuqpuip6FBmh/s=', 'Content-Length': '0'} 2024-04-16T20:06:00.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a29ce1ee48f8ccc8-00661eda28-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.490 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:06:00.490 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:06:00.491 DEBUG:boto:Method: GET 2024-04-16T20:06:00.491 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:06:00.491 DEBUG:boto:Data: 2024-04-16T20:06:00.491 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.491 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.491 DEBUG:boto:Params: {} 2024-04-16T20:06:00.491 DEBUG:boto:Token: None 2024-04-16T20:06:00.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj95 2024-04-16T20:06:00.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iCNmO5MbImVpqDuqpuip6FBmh/s= 2024-04-16T20:06:00.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iCNmO5MbImVpqDuqpuip6FBmh/s=', 'Content-Length': '0'} 2024-04-16T20:06:00.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad4154348300d2e-00661eda28-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.494 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.494 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.494 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.494 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.494 DEBUG:boto:Method: GET 2024-04-16T20:06:00.494 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.494 DEBUG:boto:Data: 2024-04-16T20:06:00.494 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:06:00.494 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.495 DEBUG:boto:Params: {} 2024-04-16T20:06:00.495 DEBUG:boto:Token: None 2024-04-16T20:06:00.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aabe22bca6bf3c50-00661eda28-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.538 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.539 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.539 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.539 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.539 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.539 DEBUG:boto:Method: GET 2024-04-16T20:06:00.539 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:06:00.539 DEBUG:boto:Data: 2024-04-16T20:06:00.539 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.539 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.539 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.539 DEBUG:boto:Params: {} 2024-04-16T20:06:00.540 DEBUG:boto:Token: None 2024-04-16T20:06:00.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY= 2024-04-16T20:06:00.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YTuxUfMvu0uZJFnwrCvhnG6wogY=', 'Content-Length': '0'} 2024-04-16T20:06:00.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30679186fda03f0-00661eda28-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.543 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:06:00.544 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:06:00.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:06:00.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:06:00.544 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:06:00.602 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:06:00.602 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:06:00.676 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:06:00.676 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-52 2024-04-16T20:06:00.690 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 30 2024-04-16T20:06:00.772 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.772 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.772 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.772 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:06:00.772 DEBUG:boto:Data: 2024-04-16T20:06:00.773 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.773 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.773 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.773 DEBUG:boto:Params: {} 2024-04-16T20:06:00.773 DEBUG:boto:Token: None 2024-04-16T20:06:00.773 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.773 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c= 2024-04-16T20:06:00.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c=', 'Content-Length': '0'} 2024-04-16T20:06:00.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd516058fdd6c7d1-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.776 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:06:00.776 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:06:00.776 DEBUG:boto:Method: PUT 2024-04-16T20:06:00.777 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:06:00.777 DEBUG:boto:Data: 2024-04-16T20:06:00.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:06:00.777 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.777 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.777 DEBUG:boto:Params: {} 2024-04-16T20:06:00.777 DEBUG:boto:Token: None 2024-04-16T20:06:00.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj37 2024-04-16T20:06:00.777 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bBWvyKIFoPPgLDd3xubEoSf7EgE= 2024-04-16T20:06:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bBWvyKIFoPPgLDd3xubEoSf7EgE='} 2024-04-16T20:06:00.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daab638c697f1ce6-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.782 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.783 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.783 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.783 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:06:00.783 DEBUG:boto:Data: 2024-04-16T20:06:00.783 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.783 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.783 DEBUG:boto:Params: {} 2024-04-16T20:06:00.783 DEBUG:boto:Token: None 2024-04-16T20:06:00.784 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c= 2024-04-16T20:06:00.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c=', 'Content-Length': '0'} 2024-04-16T20:06:00.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2b27bd7fcfb1f84-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.787 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:06:00.787 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:06:00.787 DEBUG:boto:Method: PUT 2024-04-16T20:06:00.787 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:06:00.787 DEBUG:boto:Data: 2024-04-16T20:06:00.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:06:00.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.787 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.787 DEBUG:boto:Params: {} 2024-04-16T20:06:00.788 DEBUG:boto:Token: None 2024-04-16T20:06:00.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj74 2024-04-16T20:06:00.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KLo42q0ghVkvAVZrA2v9+DcltpY= 2024-04-16T20:06:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KLo42q0ghVkvAVZrA2v9+DcltpY='} 2024-04-16T20:06:00.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009dd6752a5df611d-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.831 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.831 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.831 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.831 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:06:00.831 DEBUG:boto:Data: 2024-04-16T20:06:00.831 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.831 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.831 DEBUG:boto:Params: {} 2024-04-16T20:06:00.831 DEBUG:boto:Token: None 2024-04-16T20:06:00.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c= 2024-04-16T20:06:00.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c=', 'Content-Length': '0'} 2024-04-16T20:06:00.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d1bec95942332cf-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.834 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:06:00.834 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:06:00.835 DEBUG:boto:Method: PUT 2024-04-16T20:06:00.835 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:06:00.835 DEBUG:boto:Data: 2024-04-16T20:06:00.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:06:00.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.835 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.835 DEBUG:boto:Params: {} 2024-04-16T20:06:00.835 DEBUG:boto:Token: None 2024-04-16T20:06:00.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj111 2024-04-16T20:06:00.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9QtiCQW6nJfN4ULYA5sHFaMo67U= 2024-04-16T20:06:00.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9QtiCQW6nJfN4ULYA5sHFaMo67U='} 2024-04-16T20:06:00.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000355bbe3b5ede29ba-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.841 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.842 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.842 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.842 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:06:00.842 DEBUG:boto:Data: 2024-04-16T20:06:00.842 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.842 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.842 DEBUG:boto:Params: {} 2024-04-16T20:06:00.842 DEBUG:boto:Token: None 2024-04-16T20:06:00.842 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c= 2024-04-16T20:06:00.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c=', 'Content-Length': '0'} 2024-04-16T20:06:00.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004304aadbc2c90149-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.846 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:06:00.846 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:06:00.846 DEBUG:boto:Method: PUT 2024-04-16T20:06:00.846 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:06:00.846 DEBUG:boto:Data: 2024-04-16T20:06:00.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:06:00.846 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.846 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.846 DEBUG:boto:Params: {} 2024-04-16T20:06:00.846 DEBUG:boto:Token: None 2024-04-16T20:06:00.847 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj148 2024-04-16T20:06:00.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xpQvBECi4VZUVbJsFDbCT5m/MKc= 2024-04-16T20:06:00.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xpQvBECi4VZUVbJsFDbCT5m/MKc='} 2024-04-16T20:06:00.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f2df146e5c06b56-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.852 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:06:00.852 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:06:00.852 DEBUG:boto:Method: HEAD 2024-04-16T20:06:00.852 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:06:00.853 DEBUG:boto:Data: 2024-04-16T20:06:00.853 DEBUG:boto:Headers: {} 2024-04-16T20:06:00.853 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.853 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.853 DEBUG:boto:Params: {} 2024-04-16T20:06:00.853 DEBUG:boto:Token: None 2024-04-16T20:06:00.853 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/ 2024-04-16T20:06:00.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c= 2024-04-16T20:06:00.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yg8R59OMb6KjC1ESUxiOWgPXh3c=', 'Content-Length': '0'} 2024-04-16T20:06:00.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c936718a05d9cec5-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.856 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:06:00.857 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:06:00.857 DEBUG:boto:Method: PUT 2024-04-16T20:06:00.857 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:06:00.857 DEBUG:boto:Data: 2024-04-16T20:06:00.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:06:00.857 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:06:00.857 DEBUG:boto:Port: 8000 2024-04-16T20:06:00.857 DEBUG:boto:Params: {} 2024-04-16T20:06:00.857 DEBUG:boto:Token: None 2024-04-16T20:06:00.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:06:00 GMT /yrjefh-52/obj185 2024-04-16T20:06:00.858 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:olIEoGRnWzcI89Fn0kY/0NMrzP4= 2024-04-16T20:06:00.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:06:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:olIEoGRnWzcI89Fn0kY/0NMrzP4='} 2024-04-16T20:06:00.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034a6d5f817b87afd-00661eda28-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:06:00.903 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:06:00.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:06:00.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:06:00.903 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:06:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:06:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:06:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:01.033+0000 7faa56f33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:06:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:06.037+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:06.030 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:06.030 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:11.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:11.039+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:11.033 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:11.033 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:16.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:16.041+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:16.035 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:16.035 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:21.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:21.043+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:21.037 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:21.037 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:26.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:26.046+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:26.040 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:26.040 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:31.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:31.048+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:31.042 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:31.042 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:36.049+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:36.044 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:36.044 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:41.053+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:41.046 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:41.047 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:46.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:46.055+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:46.049 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , , , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:46.049 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:51.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:51.058+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:51.051 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:51.051 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:51.525 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T20:06:51.532+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T20:06:56.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:06:56.060+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:06:56.054 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:06:56.054 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:01.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:01.062+0000 7faa56f33800 1 waiting for incremental sync to catch up: 2024-04-16T20:07:01.056 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:01.056 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:06.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:06.065+0000 7faa56f33800 1 bucket sync caught up with source: 2024-04-16T20:07:06.058 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:06.058 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:06.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:06.065+0000 7faa56f33800 0 bucket checkpoint complete 2024-04-16T20:07:06.084 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:07:06.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:06.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:06.085 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:07:06.149 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:06.149 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:07:06.219+0000 7fcbee60f800 1 waiting for incremental sync to catch up: 2024-04-16T20:07:06.212 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000005.292.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000004.186.6, 00000000007.253.6, 00000000005.316.6] 2024-04-16T20:07:06.213 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000005.292.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000004.186.6, 00000000007.253.6, 00000000006.318.6] 2024-04-16T20:07:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:07:11.221+0000 7fcbee60f800 1 bucket sync caught up with source: 2024-04-16T20:07:11.214 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000005.292.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000004.186.6, 00000000007.253.6, 00000000006.318.6] 2024-04-16T20:07:11.214 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.180.6, 00000000002.172.6, 00000000002.108.6, 00000000005.292.6, 00000000004.235.6, 00000000001.188.6, 00000000001.300.6, 00000000004.186.6, 00000000007.253.6, 00000000006.318.6] 2024-04-16T20:07:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:07:11.221+0000 7fcbee60f800 0 bucket checkpoint complete 2024-04-16T20:07:11.241 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:07:11.241 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.241 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.241 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.242 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:07:11.242 DEBUG:boto:Data: 2024-04-16T20:07:11.242 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.242 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.242 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.242 DEBUG:boto:Params: {} 2024-04-16T20:07:11.243 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:07:11.243 DEBUG:boto:Token: None 2024-04-16T20:07:11.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JuE3FobnCUZmxP+KdRnEFU5FQHU= 2024-04-16T20:07:11.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JuE3FobnCUZmxP+KdRnEFU5FQHU=', 'Content-Length': '0'} 2024-04-16T20:07:11.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af3803d0503fe94-00661eda6f-5480-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.249 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.249 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.250 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.250 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:07:11.250 DEBUG:boto:Data: 2024-04-16T20:07:11.250 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.250 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.250 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.250 DEBUG:boto:Params: {} 2024-04-16T20:07:11.250 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:07:11.251 DEBUG:boto:Token: None 2024-04-16T20:07:11.251 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JuE3FobnCUZmxP+KdRnEFU5FQHU= 2024-04-16T20:07:11.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JuE3FobnCUZmxP+KdRnEFU5FQHU=', 'Content-Length': '0'} 2024-04-16T20:07:11.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000028fb9b1c991c615-00661eda6f-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.256 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:07:11.256 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.256 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.256 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.256 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.256 DEBUG:boto:Method: GET 2024-04-16T20:07:11.256 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.257 DEBUG:boto:Data: 2024-04-16T20:07:11.257 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.257 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.257 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.257 DEBUG:boto:Params: {} 2024-04-16T20:07:11.257 DEBUG:boto:Token: None 2024-04-16T20:07:11.257 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/?versions 2024-04-16T20:07:11.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c= 2024-04-16T20:07:11.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c=', 'Content-Length': '0'} 2024-04-16T20:07:11.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039561605a0282dd8-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.261 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:07:11.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:07:11.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:07:11.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:07:11.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:07:11.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:07:11.266 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.266 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.266 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.266 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.266 DEBUG:boto:Method: GET 2024-04-16T20:07:11.266 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.266 DEBUG:boto:Data: 2024-04-16T20:07:11.266 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.267 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.267 DEBUG:boto:Params: {} 2024-04-16T20:07:11.267 DEBUG:boto:Token: None 2024-04-16T20:07:11.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/?versions 2024-04-16T20:07:11.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c= 2024-04-16T20:07:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c=', 'Content-Length': '0'} 2024-04-16T20:07:11.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e4ae2ea73cda63-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.271 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:07:11.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:07:11.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:07:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:07:11.276 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.276 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.276 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.276 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.276 DEBUG:boto:Method: GET 2024-04-16T20:07:11.276 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.276 DEBUG:boto:Data: 2024-04-16T20:07:11.276 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.276 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.276 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.276 DEBUG:boto:Params: {} 2024-04-16T20:07:11.277 DEBUG:boto:Token: None 2024-04-16T20:07:11.277 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/?versions 2024-04-16T20:07:11.277 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c= 2024-04-16T20:07:11.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c=', 'Content-Length': '0'} 2024-04-16T20:07:11.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d125fdf6e5f4356b-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.280 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.281 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.281 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.281 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.282 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.282 DEBUG:boto:Method: GET 2024-04-16T20:07:11.282 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:11.282 DEBUG:boto:Data: 2024-04-16T20:07:11.282 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.282 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.282 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.282 DEBUG:boto:Params: {} 2024-04-16T20:07:11.282 DEBUG:boto:Token: None 2024-04-16T20:07:11.282 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/?versions 2024-04-16T20:07:11.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c= 2024-04-16T20:07:11.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Va3oTWk64hZaoLq/AsWYHDyot7c=', 'Content-Length': '0'} 2024-04-16T20:07:11.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8e773d0832fd29-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.287 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:11.288 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.288 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.288 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.288 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.289 DEBUG:boto:Method: GET 2024-04-16T20:07:11.289 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.289 DEBUG:boto:Data: 2024-04-16T20:07:11.289 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.289 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.289 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.289 DEBUG:boto:Params: {} 2024-04-16T20:07:11.289 DEBUG:boto:Token: None 2024-04-16T20:07:11.289 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.289 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU= 2024-04-16T20:07:11.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU=', 'Content-Length': '0'} 2024-04-16T20:07:11.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64b6d4aaabd6ecc-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.292 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.292 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.292 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.292 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.293 DEBUG:boto:Method: GET 2024-04-16T20:07:11.293 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.293 DEBUG:boto:Data: 2024-04-16T20:07:11.293 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.293 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.293 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.293 DEBUG:boto:Params: {} 2024-04-16T20:07:11.293 DEBUG:boto:Token: None 2024-04-16T20:07:11.293 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU= 2024-04-16T20:07:11.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU=', 'Content-Length': '0'} 2024-04-16T20:07:11.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c8c14df0f68968-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.338 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.339 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.339 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.339 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.339 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.339 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.339 DEBUG:boto:Data: 2024-04-16T20:07:11.339 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.339 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.339 DEBUG:boto:Params: {} 2024-04-16T20:07:11.339 DEBUG:boto:Token: None 2024-04-16T20:07:11.340 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NvDlGmDGgXt9CmdP+2/f8zfukF4= 2024-04-16T20:07:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NvDlGmDGgXt9CmdP+2/f8zfukF4=', 'Content-Length': '0'} 2024-04-16T20:07:11.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000806f98493b31af62-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.342 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.342 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.342 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.342 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.342 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.342 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.342 DEBUG:boto:Data: 2024-04-16T20:07:11.343 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.343 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.343 DEBUG:boto:Params: {} 2024-04-16T20:07:11.343 DEBUG:boto:Token: None 2024-04-16T20:07:11.343 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NvDlGmDGgXt9CmdP+2/f8zfukF4= 2024-04-16T20:07:11.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NvDlGmDGgXt9CmdP+2/f8zfukF4=', 'Content-Length': '0'} 2024-04-16T20:07:11.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2eecdc3ec765e6-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:11.345 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.345 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.346 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.346 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.346 DEBUG:boto:Method: GET 2024-04-16T20:07:11.346 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.346 DEBUG:boto:Data: 2024-04-16T20:07:11.346 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.346 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.346 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.346 DEBUG:boto:Params: {} 2024-04-16T20:07:11.346 DEBUG:boto:Token: None 2024-04-16T20:07:11.347 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.347 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU= 2024-04-16T20:07:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU=', 'Content-Length': '0'} 2024-04-16T20:07:11.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9aea6654bca255-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.390 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.391 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.391 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.391 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.391 DEBUG:boto:Method: GET 2024-04-16T20:07:11.391 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.391 DEBUG:boto:Data: 2024-04-16T20:07:11.391 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.391 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.391 DEBUG:boto:Params: {} 2024-04-16T20:07:11.392 DEBUG:boto:Token: None 2024-04-16T20:07:11.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:11.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU= 2024-04-16T20:07:11.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QPtDcnqau53NpCi9rkSgNSEUytU=', 'Content-Length': '0'} 2024-04-16T20:07:11.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c40bd0beb7580c0-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.394 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.394 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.394 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.394 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.394 DEBUG:boto:Method: GET 2024-04-16T20:07:11.394 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.394 DEBUG:boto:Data: 2024-04-16T20:07:11.395 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.395 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.395 DEBUG:boto:Params: {} 2024-04-16T20:07:11.395 DEBUG:boto:Token: None 2024-04-16T20:07:11.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003af5ffeeed1407dc-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.438 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.439 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.439 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.439 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.439 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.439 DEBUG:boto:Method: GET 2024-04-16T20:07:11.439 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.439 DEBUG:boto:Data: 2024-04-16T20:07:11.439 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.440 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.440 DEBUG:boto:Params: {} 2024-04-16T20:07:11.440 DEBUG:boto:Token: None 2024-04-16T20:07:11.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff485384cda90b6f-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.443 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.444 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.444 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.444 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.444 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:11.444 DEBUG:boto:Data: 2024-04-16T20:07:11.444 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.444 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.444 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.445 DEBUG:boto:Params: {} 2024-04-16T20:07:11.445 DEBUG:boto:Token: None 2024-04-16T20:07:11.445 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11 2024-04-16T20:07:11.445 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sSu0/6mmm7IJhuKim3COYqlkPW8= 2024-04-16T20:07:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sSu0/6mmm7IJhuKim3COYqlkPW8=', 'Content-Length': '0'} 2024-04-16T20:07:11.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2244b52910cc7db-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.447 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.447 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.448 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.448 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:11.448 DEBUG:boto:Data: 2024-04-16T20:07:11.448 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.448 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.448 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.448 DEBUG:boto:Params: {} 2024-04-16T20:07:11.448 DEBUG:boto:Token: None 2024-04-16T20:07:11.448 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11 2024-04-16T20:07:11.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sSu0/6mmm7IJhuKim3COYqlkPW8= 2024-04-16T20:07:11.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sSu0/6mmm7IJhuKim3COYqlkPW8=', 'Content-Length': '0'} 2024-04-16T20:07:11.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5af0a5d171a16a-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:11.451 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.451 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.451 DEBUG:boto:Method: GET 2024-04-16T20:07:11.451 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:11.451 DEBUG:boto:Data: 2024-04-16T20:07:11.451 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.451 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.451 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.451 DEBUG:boto:Params: {} 2024-04-16T20:07:11.452 DEBUG:boto:Token: None 2024-04-16T20:07:11.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11 2024-04-16T20:07:11.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HqOdGLlb+M9Zh6DrvN5peBYoAfs= 2024-04-16T20:07:11.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HqOdGLlb+M9Zh6DrvN5peBYoAfs=', 'Content-Length': '0'} 2024-04-16T20:07:11.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748e6b0a91f31055-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.455 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:11.455 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:11.455 DEBUG:boto:Method: GET 2024-04-16T20:07:11.455 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:11.455 DEBUG:boto:Data: 2024-04-16T20:07:11.455 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.455 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.455 DEBUG:boto:Params: {} 2024-04-16T20:07:11.455 DEBUG:boto:Token: None 2024-04-16T20:07:11.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj11 2024-04-16T20:07:11.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HqOdGLlb+M9Zh6DrvN5peBYoAfs= 2024-04-16T20:07:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HqOdGLlb+M9Zh6DrvN5peBYoAfs=', 'Content-Length': '0'} 2024-04-16T20:07:11.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a3864376f9a58c-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.499 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.499 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.499 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.499 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.499 DEBUG:boto:Method: GET 2024-04-16T20:07:11.499 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.499 DEBUG:boto:Data: 2024-04-16T20:07:11.499 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.499 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.500 DEBUG:boto:Params: {} 2024-04-16T20:07:11.500 DEBUG:boto:Token: None 2024-04-16T20:07:11.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006647e98a5d4cd10b-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.503 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.503 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.503 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.504 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.504 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.504 DEBUG:boto:Method: GET 2024-04-16T20:07:11.504 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:11.504 DEBUG:boto:Data: 2024-04-16T20:07:11.504 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.504 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.504 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.504 DEBUG:boto:Params: {} 2024-04-16T20:07:11.504 DEBUG:boto:Token: None 2024-04-16T20:07:11.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.505 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6be99647b9a20e6-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.509 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:11.510 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.510 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.510 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.510 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.510 DEBUG:boto:Method: GET 2024-04-16T20:07:11.510 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.510 DEBUG:boto:Data: 2024-04-16T20:07:11.510 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.510 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.510 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.511 DEBUG:boto:Params: {} 2024-04-16T20:07:11.511 DEBUG:boto:Token: None 2024-04-16T20:07:11.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc= 2024-04-16T20:07:11.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc=', 'Content-Length': '0'} 2024-04-16T20:07:11.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009890bf362bb2914a-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.554 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.554 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.555 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.555 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.555 DEBUG:boto:Method: GET 2024-04-16T20:07:11.555 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.555 DEBUG:boto:Data: 2024-04-16T20:07:11.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.555 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.555 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.555 DEBUG:boto:Params: {} 2024-04-16T20:07:11.555 DEBUG:boto:Token: None 2024-04-16T20:07:11.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc= 2024-04-16T20:07:11.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc=', 'Content-Length': '0'} 2024-04-16T20:07:11.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cbabb5a9bda7ae5-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.559 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.559 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.559 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.559 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.559 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.559 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.559 DEBUG:boto:Data: 2024-04-16T20:07:11.559 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.559 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.560 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.560 DEBUG:boto:Params: {} 2024-04-16T20:07:11.560 DEBUG:boto:Token: None 2024-04-16T20:07:11.560 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yFvo/GYRDRat4vSQzSlBgDRTNXE= 2024-04-16T20:07:11.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yFvo/GYRDRat4vSQzSlBgDRTNXE=', 'Content-Length': '0'} 2024-04-16T20:07:11.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7c93fe65beb505-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.562 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.563 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.563 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.563 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.563 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.563 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.563 DEBUG:boto:Data: 2024-04-16T20:07:11.563 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.563 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.563 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.563 DEBUG:boto:Params: {} 2024-04-16T20:07:11.564 DEBUG:boto:Token: None 2024-04-16T20:07:11.564 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.564 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yFvo/GYRDRat4vSQzSlBgDRTNXE= 2024-04-16T20:07:11.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yFvo/GYRDRat4vSQzSlBgDRTNXE=', 'Content-Length': '0'} 2024-04-16T20:07:11.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bab60a671ffe421-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:11.566 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.566 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.567 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.567 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.567 DEBUG:boto:Method: GET 2024-04-16T20:07:11.567 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.567 DEBUG:boto:Data: 2024-04-16T20:07:11.567 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.567 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.567 DEBUG:boto:Params: {} 2024-04-16T20:07:11.567 DEBUG:boto:Token: None 2024-04-16T20:07:11.568 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc= 2024-04-16T20:07:11.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc=', 'Content-Length': '0'} 2024-04-16T20:07:11.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d24931be408832-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.570 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.570 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.570 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.570 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.570 DEBUG:boto:Method: GET 2024-04-16T20:07:11.570 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.570 DEBUG:boto:Data: 2024-04-16T20:07:11.571 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.571 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.571 DEBUG:boto:Params: {} 2024-04-16T20:07:11.571 DEBUG:boto:Token: None 2024-04-16T20:07:11.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:11.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc= 2024-04-16T20:07:11.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JUbcVns97qwNzjskf1jA6rM8IWc=', 'Content-Length': '0'} 2024-04-16T20:07:11.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d833e4fc0d309319-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.573 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.574 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.574 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.574 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.574 DEBUG:boto:Method: GET 2024-04-16T20:07:11.574 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.574 DEBUG:boto:Data: 2024-04-16T20:07:11.574 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.574 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.574 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.574 DEBUG:boto:Params: {} 2024-04-16T20:07:11.575 DEBUG:boto:Token: None 2024-04-16T20:07:11.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df223b690476574-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.578 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.578 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.578 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.578 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.578 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.578 DEBUG:boto:Method: GET 2024-04-16T20:07:11.579 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.579 DEBUG:boto:Data: 2024-04-16T20:07:11.579 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.579 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.579 DEBUG:boto:Params: {} 2024-04-16T20:07:11.579 DEBUG:boto:Token: None 2024-04-16T20:07:11.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ff01a97b9299d1-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.583 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.583 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.583 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.584 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.584 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:11.584 DEBUG:boto:Data: 2024-04-16T20:07:11.584 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.584 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.584 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.584 DEBUG:boto:Params: {} 2024-04-16T20:07:11.584 DEBUG:boto:Token: None 2024-04-16T20:07:11.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111 2024-04-16T20:07:11.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1A2eSYSHBQ/1uuXj1YvRn0n8+Wk= 2024-04-16T20:07:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1A2eSYSHBQ/1uuXj1YvRn0n8+Wk=', 'Content-Length': '0'} 2024-04-16T20:07:11.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84a2797b7b50429-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.587 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.588 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.588 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.588 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:11.588 DEBUG:boto:Data: 2024-04-16T20:07:11.588 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.588 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.588 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.588 DEBUG:boto:Params: {} 2024-04-16T20:07:11.588 DEBUG:boto:Token: None 2024-04-16T20:07:11.589 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111 2024-04-16T20:07:11.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1A2eSYSHBQ/1uuXj1YvRn0n8+Wk= 2024-04-16T20:07:11.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1A2eSYSHBQ/1uuXj1YvRn0n8+Wk=', 'Content-Length': '0'} 2024-04-16T20:07:11.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45ce2c6eae8ccd7-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:11.591 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.591 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.591 DEBUG:boto:Method: GET 2024-04-16T20:07:11.591 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:11.591 DEBUG:boto:Data: 2024-04-16T20:07:11.592 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.592 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.592 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.592 DEBUG:boto:Params: {} 2024-04-16T20:07:11.592 DEBUG:boto:Token: None 2024-04-16T20:07:11.592 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111 2024-04-16T20:07:11.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SRJtqYuftXubuYTjMZb5zd9CkYw= 2024-04-16T20:07:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SRJtqYuftXubuYTjMZb5zd9CkYw=', 'Content-Length': '0'} 2024-04-16T20:07:11.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f66831948b8b633-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.595 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:11.595 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:11.595 DEBUG:boto:Method: GET 2024-04-16T20:07:11.595 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:11.595 DEBUG:boto:Data: 2024-04-16T20:07:11.595 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.596 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.596 DEBUG:boto:Params: {} 2024-04-16T20:07:11.596 DEBUG:boto:Token: None 2024-04-16T20:07:11.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj111 2024-04-16T20:07:11.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SRJtqYuftXubuYTjMZb5zd9CkYw= 2024-04-16T20:07:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SRJtqYuftXubuYTjMZb5zd9CkYw=', 'Content-Length': '0'} 2024-04-16T20:07:11.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c615f4727fd5c7-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.638 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.639 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.639 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.639 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.639 DEBUG:boto:Method: GET 2024-04-16T20:07:11.639 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.639 DEBUG:boto:Data: 2024-04-16T20:07:11.639 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.639 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.639 DEBUG:boto:Params: {} 2024-04-16T20:07:11.639 DEBUG:boto:Token: None 2024-04-16T20:07:11.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3fc4b080156844d-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.643 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.643 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.643 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.643 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.643 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.643 DEBUG:boto:Method: GET 2024-04-16T20:07:11.643 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:11.643 DEBUG:boto:Data: 2024-04-16T20:07:11.644 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.644 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.644 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.644 DEBUG:boto:Params: {} 2024-04-16T20:07:11.644 DEBUG:boto:Token: None 2024-04-16T20:07:11.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081c137a46155520a-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.648 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:11.648 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.648 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.648 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.648 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.648 DEBUG:boto:Method: GET 2024-04-16T20:07:11.649 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.649 DEBUG:boto:Data: 2024-04-16T20:07:11.649 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.649 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.649 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.649 DEBUG:boto:Params: {} 2024-04-16T20:07:11.649 DEBUG:boto:Token: None 2024-04-16T20:07:11.649 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y= 2024-04-16T20:07:11.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y=', 'Content-Length': '0'} 2024-04-16T20:07:11.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c98a5b1c8ea14af4-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.694 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.695 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.695 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.695 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.695 DEBUG:boto:Method: GET 2024-04-16T20:07:11.695 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.695 DEBUG:boto:Data: 2024-04-16T20:07:11.695 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.695 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.695 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.695 DEBUG:boto:Params: {} 2024-04-16T20:07:11.696 DEBUG:boto:Token: None 2024-04-16T20:07:11.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y= 2024-04-16T20:07:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y=', 'Content-Length': '0'} 2024-04-16T20:07:11.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2cc9881a131c2a5-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.699 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.699 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.699 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.699 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.699 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.699 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.699 DEBUG:boto:Data: 2024-04-16T20:07:11.700 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.700 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.700 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.700 DEBUG:boto:Params: {} 2024-04-16T20:07:11.700 DEBUG:boto:Token: None 2024-04-16T20:07:11.700 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RdQyzzAu7qPCh301ym3eJBQna7A= 2024-04-16T20:07:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RdQyzzAu7qPCh301ym3eJBQna7A=', 'Content-Length': '0'} 2024-04-16T20:07:11.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e336cb56b7e9d2e-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.703 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.703 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.703 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.703 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.703 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.703 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.703 DEBUG:boto:Data: 2024-04-16T20:07:11.703 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.704 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.704 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.704 DEBUG:boto:Params: {} 2024-04-16T20:07:11.704 DEBUG:boto:Token: None 2024-04-16T20:07:11.704 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RdQyzzAu7qPCh301ym3eJBQna7A= 2024-04-16T20:07:11.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RdQyzzAu7qPCh301ym3eJBQna7A=', 'Content-Length': '0'} 2024-04-16T20:07:11.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d2aecffe5b0e829-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:11.707 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.707 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.707 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.707 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.707 DEBUG:boto:Method: GET 2024-04-16T20:07:11.707 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.707 DEBUG:boto:Data: 2024-04-16T20:07:11.707 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.707 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.708 DEBUG:boto:Params: {} 2024-04-16T20:07:11.708 DEBUG:boto:Token: None 2024-04-16T20:07:11.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y= 2024-04-16T20:07:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y=', 'Content-Length': '0'} 2024-04-16T20:07:11.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007143dd9e7807e7bd-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.711 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.711 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.711 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.711 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.711 DEBUG:boto:Method: GET 2024-04-16T20:07:11.711 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.711 DEBUG:boto:Data: 2024-04-16T20:07:11.711 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.712 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.712 DEBUG:boto:Params: {} 2024-04-16T20:07:11.712 DEBUG:boto:Token: None 2024-04-16T20:07:11.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:11.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y= 2024-04-16T20:07:11.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2GymVLwgb6zAyB21wji35PZME4Y=', 'Content-Length': '0'} 2024-04-16T20:07:11.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000fb3d30d8efc42c-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.715 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.715 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.715 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.715 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.715 DEBUG:boto:Method: GET 2024-04-16T20:07:11.715 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.715 DEBUG:boto:Data: 2024-04-16T20:07:11.715 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.716 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.716 DEBUG:boto:Params: {} 2024-04-16T20:07:11.716 DEBUG:boto:Token: None 2024-04-16T20:07:11.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3485d16da9072bc-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.758 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.759 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.759 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.759 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.759 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.759 DEBUG:boto:Method: GET 2024-04-16T20:07:11.759 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.759 DEBUG:boto:Data: 2024-04-16T20:07:11.759 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.760 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.760 DEBUG:boto:Params: {} 2024-04-16T20:07:11.760 DEBUG:boto:Token: None 2024-04-16T20:07:11.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b15177a3f1ded58-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.764 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.764 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.764 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.764 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.764 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:11.764 DEBUG:boto:Data: 2024-04-16T20:07:11.765 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.765 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.765 DEBUG:boto:Params: {} 2024-04-16T20:07:11.765 DEBUG:boto:Token: None 2024-04-16T20:07:11.765 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115 2024-04-16T20:07:11.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09+t/H4Z+tE96Ik42ev6EBWf1vQ= 2024-04-16T20:07:11.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09+t/H4Z+tE96Ik42ev6EBWf1vQ=', 'Content-Length': '0'} 2024-04-16T20:07:11.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0285b9d528f7a08-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.768 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.768 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.768 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.768 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:11.768 DEBUG:boto:Data: 2024-04-16T20:07:11.768 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.768 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.768 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.768 DEBUG:boto:Params: {} 2024-04-16T20:07:11.768 DEBUG:boto:Token: None 2024-04-16T20:07:11.769 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115 2024-04-16T20:07:11.769 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:09+t/H4Z+tE96Ik42ev6EBWf1vQ= 2024-04-16T20:07:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:09+t/H4Z+tE96Ik42ev6EBWf1vQ=', 'Content-Length': '0'} 2024-04-16T20:07:11.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a700700bff793ff5-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:11.771 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.771 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.771 DEBUG:boto:Method: GET 2024-04-16T20:07:11.771 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:11.771 DEBUG:boto:Data: 2024-04-16T20:07:11.771 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.772 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.772 DEBUG:boto:Params: {} 2024-04-16T20:07:11.772 DEBUG:boto:Token: None 2024-04-16T20:07:11.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115 2024-04-16T20:07:11.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uUGQmOlhltjGdvVq1yWpAvoR0yg= 2024-04-16T20:07:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uUGQmOlhltjGdvVq1yWpAvoR0yg=', 'Content-Length': '0'} 2024-04-16T20:07:11.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6ffda6252e572f-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.775 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:11.775 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:11.775 DEBUG:boto:Method: GET 2024-04-16T20:07:11.775 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:11.775 DEBUG:boto:Data: 2024-04-16T20:07:11.775 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.775 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.776 DEBUG:boto:Params: {} 2024-04-16T20:07:11.776 DEBUG:boto:Token: None 2024-04-16T20:07:11.776 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj115 2024-04-16T20:07:11.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uUGQmOlhltjGdvVq1yWpAvoR0yg= 2024-04-16T20:07:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uUGQmOlhltjGdvVq1yWpAvoR0yg=', 'Content-Length': '0'} 2024-04-16T20:07:11.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000187e7717cb83e87-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.822 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.822 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.823 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.823 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.823 DEBUG:boto:Method: GET 2024-04-16T20:07:11.823 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.823 DEBUG:boto:Data: 2024-04-16T20:07:11.823 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.823 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.823 DEBUG:boto:Params: {} 2024-04-16T20:07:11.823 DEBUG:boto:Token: None 2024-04-16T20:07:11.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f84354cc39baec-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.826 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.826 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.827 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.827 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.827 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.827 DEBUG:boto:Method: GET 2024-04-16T20:07:11.827 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:11.827 DEBUG:boto:Data: 2024-04-16T20:07:11.827 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.827 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.827 DEBUG:boto:Params: {} 2024-04-16T20:07:11.827 DEBUG:boto:Token: None 2024-04-16T20:07:11.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024dbfccc39f42012-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.831 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:11.832 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.832 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.832 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.832 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.832 DEBUG:boto:Method: GET 2024-04-16T20:07:11.832 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.832 DEBUG:boto:Data: 2024-04-16T20:07:11.832 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.832 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.832 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.833 DEBUG:boto:Params: {} 2024-04-16T20:07:11.833 DEBUG:boto:Token: None 2024-04-16T20:07:11.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM= 2024-04-16T20:07:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM=', 'Content-Length': '0'} 2024-04-16T20:07:11.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007926e7096d7a69c2-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.835 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.835 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.835 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.835 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.835 DEBUG:boto:Method: GET 2024-04-16T20:07:11.835 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.835 DEBUG:boto:Data: 2024-04-16T20:07:11.835 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.836 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.836 DEBUG:boto:Params: {} 2024-04-16T20:07:11.836 DEBUG:boto:Token: None 2024-04-16T20:07:11.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM= 2024-04-16T20:07:11.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM=', 'Content-Length': '0'} 2024-04-16T20:07:11.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac2cc9eeb07f3d7-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.838 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.838 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.838 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.838 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.839 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.839 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.839 DEBUG:boto:Data: 2024-04-16T20:07:11.839 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.839 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.839 DEBUG:boto:Params: {} 2024-04-16T20:07:11.839 DEBUG:boto:Token: None 2024-04-16T20:07:11.839 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uN3QKuXc8RtBINxG/kIV7fZBocY= 2024-04-16T20:07:11.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uN3QKuXc8RtBINxG/kIV7fZBocY=', 'Content-Length': '0'} 2024-04-16T20:07:11.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d187d148915dffde-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.841 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.842 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.842 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.842 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.842 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.842 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.842 DEBUG:boto:Data: 2024-04-16T20:07:11.842 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.842 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.842 DEBUG:boto:Params: {} 2024-04-16T20:07:11.842 DEBUG:boto:Token: None 2024-04-16T20:07:11.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uN3QKuXc8RtBINxG/kIV7fZBocY= 2024-04-16T20:07:11.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uN3QKuXc8RtBINxG/kIV7fZBocY=', 'Content-Length': '0'} 2024-04-16T20:07:11.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fed9920900bf201-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:11.845 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.845 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.845 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.845 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.845 DEBUG:boto:Method: GET 2024-04-16T20:07:11.845 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.845 DEBUG:boto:Data: 2024-04-16T20:07:11.845 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.846 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.846 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.846 DEBUG:boto:Params: {} 2024-04-16T20:07:11.846 DEBUG:boto:Token: None 2024-04-16T20:07:11.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM= 2024-04-16T20:07:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM=', 'Content-Length': '0'} 2024-04-16T20:07:11.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d58e1f5776f4b7-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.890 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.891 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.891 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.891 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.891 DEBUG:boto:Method: GET 2024-04-16T20:07:11.891 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.891 DEBUG:boto:Data: 2024-04-16T20:07:11.891 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.891 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.891 DEBUG:boto:Params: {} 2024-04-16T20:07:11.891 DEBUG:boto:Token: None 2024-04-16T20:07:11.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:11.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM= 2024-04-16T20:07:11.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTMbYswwobGudS2vNHQ74w/hfCM=', 'Content-Length': '0'} 2024-04-16T20:07:11.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975d509c813bc98a-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.894 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.895 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.895 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.895 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.895 DEBUG:boto:Method: GET 2024-04-16T20:07:11.895 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.895 DEBUG:boto:Data: 2024-04-16T20:07:11.895 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.895 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.895 DEBUG:boto:Params: {} 2024-04-16T20:07:11.895 DEBUG:boto:Token: None 2024-04-16T20:07:11.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000545c7af3b5a7c3a9-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.898 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.898 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.899 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.899 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.899 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.899 DEBUG:boto:Method: GET 2024-04-16T20:07:11.899 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.899 DEBUG:boto:Data: 2024-04-16T20:07:11.899 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.899 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.899 DEBUG:boto:Params: {} 2024-04-16T20:07:11.899 DEBUG:boto:Token: None 2024-04-16T20:07:11.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012791bc1d036a0ad-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.904 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.904 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.904 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.904 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.904 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:11.904 DEBUG:boto:Data: 2024-04-16T20:07:11.904 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.904 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.905 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.905 DEBUG:boto:Params: {} 2024-04-16T20:07:11.905 DEBUG:boto:Token: None 2024-04-16T20:07:11.905 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116 2024-04-16T20:07:11.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4U1eS/TuEFOojDRDf9lYiQfa9NY= 2024-04-16T20:07:11.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4U1eS/TuEFOojDRDf9lYiQfa9NY=', 'Content-Length': '0'} 2024-04-16T20:07:11.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef2e43bbd010ebd-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.907 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.907 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.907 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.907 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:11.907 DEBUG:boto:Data: 2024-04-16T20:07:11.908 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.908 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.908 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.908 DEBUG:boto:Params: {} 2024-04-16T20:07:11.908 DEBUG:boto:Token: None 2024-04-16T20:07:11.908 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116 2024-04-16T20:07:11.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4U1eS/TuEFOojDRDf9lYiQfa9NY= 2024-04-16T20:07:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4U1eS/TuEFOojDRDf9lYiQfa9NY=', 'Content-Length': '0'} 2024-04-16T20:07:11.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbab284d7530f28-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:11.910 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.910 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.910 DEBUG:boto:Method: GET 2024-04-16T20:07:11.910 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:11.910 DEBUG:boto:Data: 2024-04-16T20:07:11.911 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.911 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.911 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.911 DEBUG:boto:Params: {} 2024-04-16T20:07:11.911 DEBUG:boto:Token: None 2024-04-16T20:07:11.911 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116 2024-04-16T20:07:11.911 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zWE2nKsV2EWpNHxRRHxFYsxaR20= 2024-04-16T20:07:11.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zWE2nKsV2EWpNHxRRHxFYsxaR20=', 'Content-Length': '0'} 2024-04-16T20:07:11.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:04:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9bb19136dee0c6e-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.913 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:11.913 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:11.913 DEBUG:boto:Method: GET 2024-04-16T20:07:11.913 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:11.913 DEBUG:boto:Data: 2024-04-16T20:07:11.914 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.914 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.914 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.914 DEBUG:boto:Params: {} 2024-04-16T20:07:11.914 DEBUG:boto:Token: None 2024-04-16T20:07:11.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj116 2024-04-16T20:07:11.914 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zWE2nKsV2EWpNHxRRHxFYsxaR20= 2024-04-16T20:07:11.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zWE2nKsV2EWpNHxRRHxFYsxaR20=', 'Content-Length': '0'} 2024-04-16T20:07:11.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f47e57acd366abee-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.958 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.958 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.959 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.959 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.959 DEBUG:boto:Method: GET 2024-04-16T20:07:11.959 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.959 DEBUG:boto:Data: 2024-04-16T20:07:11.959 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.959 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.959 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.959 DEBUG:boto:Params: {} 2024-04-16T20:07:11.959 DEBUG:boto:Token: None 2024-04-16T20:07:11.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e69e8de9cc8e550-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.962 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.962 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:11.962 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:11.962 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.962 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.963 DEBUG:boto:Method: GET 2024-04-16T20:07:11.963 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:11.963 DEBUG:boto:Data: 2024-04-16T20:07:11.963 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.963 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.963 DEBUG:boto:Params: {} 2024-04-16T20:07:11.963 DEBUG:boto:Token: None 2024-04-16T20:07:11.963 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/ 2024-04-16T20:07:11.963 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis= 2024-04-16T20:07:11.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nMfIywzlUw76odNMejb5PWNziis=', 'Content-Length': '0'} 2024-04-16T20:07:11.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdcf118673d647f9-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.967 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:11.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:11.967 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:11.967 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:11.967 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.968 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.968 DEBUG:boto:Method: GET 2024-04-16T20:07:11.968 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.968 DEBUG:boto:Data: 2024-04-16T20:07:11.968 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.968 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.968 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.968 DEBUG:boto:Params: {} 2024-04-16T20:07:11.968 DEBUG:boto:Token: None 2024-04-16T20:07:11.968 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso= 2024-04-16T20:07:11.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso=', 'Content-Length': '0'} 2024-04-16T20:07:11.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5a1873f8903346-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.970 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:11.971 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:11.971 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.971 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.971 DEBUG:boto:Method: GET 2024-04-16T20:07:11.971 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.971 DEBUG:boto:Data: 2024-04-16T20:07:11.971 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.971 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.971 DEBUG:boto:Params: {} 2024-04-16T20:07:11.971 DEBUG:boto:Token: None 2024-04-16T20:07:11.971 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso= 2024-04-16T20:07:11.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso=', 'Content-Length': '0'} 2024-04-16T20:07:11.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047bc8dfc392a4f05-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.974 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:11.974 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:11.974 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.974 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.974 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.974 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.974 DEBUG:boto:Data: 2024-04-16T20:07:11.974 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.974 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.975 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.975 DEBUG:boto:Params: {} 2024-04-16T20:07:11.975 DEBUG:boto:Token: None 2024-04-16T20:07:11.975 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hkRviDiK6SPcobcsXXkyLtqCZ2w= 2024-04-16T20:07:11.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hkRviDiK6SPcobcsXXkyLtqCZ2w=', 'Content-Length': '0'} 2024-04-16T20:07:11.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11ae640f59c94a4-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.977 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:11.977 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:11.977 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.977 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.978 DEBUG:boto:Method: HEAD 2024-04-16T20:07:11.978 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.978 DEBUG:boto:Data: 2024-04-16T20:07:11.978 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.978 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:11.978 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.978 DEBUG:boto:Params: {} 2024-04-16T20:07:11.978 DEBUG:boto:Token: None 2024-04-16T20:07:11.978 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hkRviDiK6SPcobcsXXkyLtqCZ2w= 2024-04-16T20:07:11.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hkRviDiK6SPcobcsXXkyLtqCZ2w=', 'Content-Length': '0'} 2024-04-16T20:07:11.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018139e317f9f6927-00661eda6f-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:11.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:11.981 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:11.981 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:11.981 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.981 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.981 DEBUG:boto:Method: GET 2024-04-16T20:07:11.981 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.981 DEBUG:boto:Data: 2024-04-16T20:07:11.981 DEBUG:boto:Headers: {} 2024-04-16T20:07:11.981 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:11.982 DEBUG:boto:Port: 8000 2024-04-16T20:07:11.982 DEBUG:boto:Params: {} 2024-04-16T20:07:11.982 DEBUG:boto:Token: None 2024-04-16T20:07:11.982 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:11 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:11.982 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso= 2024-04-16T20:07:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:11 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qw2fWzuHEcOA/H3lYRwd0f+/tso=', 'Content-Length': '0'} 2024-04-16T20:07:11.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab6857d9f147aa9-00661eda6f-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.026 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:12.026 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:12.027 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:12.027 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:12.027 DEBUG:boto:Method: GET 2024-04-16T20:07:12.027 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:12.027 DEBUG:boto:Data: 2024-04-16T20:07:12.027 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.027 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.027 DEBUG:boto:Params: {} 2024-04-16T20:07:12.027 DEBUG:boto:Token: None 2024-04-16T20:07:12.027 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:12.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aAlhOpSd4IFmuP5dNvWx71ARG/c= 2024-04-16T20:07:12.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aAlhOpSd4IFmuP5dNvWx71ARG/c=', 'Content-Length': '0'} 2024-04-16T20:07:12.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87e0e8515e1afff-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.030 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.030 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.030 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.030 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.030 DEBUG:boto:Method: GET 2024-04-16T20:07:12.030 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.030 DEBUG:boto:Data: 2024-04-16T20:07:12.030 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.030 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.030 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.030 DEBUG:boto:Params: {} 2024-04-16T20:07:12.031 DEBUG:boto:Token: None 2024-04-16T20:07:12.031 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f56df33a354933d-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.033 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.033 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.034 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.034 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.034 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.034 DEBUG:boto:Method: GET 2024-04-16T20:07:12.034 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.034 DEBUG:boto:Data: 2024-04-16T20:07:12.034 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.034 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.034 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.034 DEBUG:boto:Params: {} 2024-04-16T20:07:12.034 DEBUG:boto:Token: None 2024-04-16T20:07:12.035 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a837ec98be3f2751-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.038 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.038 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:12.038 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:12.038 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.038 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:12.038 DEBUG:boto:Data: 2024-04-16T20:07:12.038 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.039 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.039 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.039 DEBUG:boto:Params: {} 2024-04-16T20:07:12.039 DEBUG:boto:Token: None 2024-04-16T20:07:12.039 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj124 2024-04-16T20:07:12.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Irz7e+6YK0p11b0c4BTH+oaOcy4= 2024-04-16T20:07:12.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Irz7e+6YK0p11b0c4BTH+oaOcy4=', 'Content-Length': '0'} 2024-04-16T20:07:12.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008076ef34fecb93a2-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.041 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:12.041 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:12.041 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.041 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:12.042 DEBUG:boto:Data: 2024-04-16T20:07:12.042 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.042 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.042 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.042 DEBUG:boto:Params: {} 2024-04-16T20:07:12.042 DEBUG:boto:Token: None 2024-04-16T20:07:12.042 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj124 2024-04-16T20:07:12.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Irz7e+6YK0p11b0c4BTH+oaOcy4= 2024-04-16T20:07:12.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Irz7e+6YK0p11b0c4BTH+oaOcy4=', 'Content-Length': '0'} 2024-04-16T20:07:12.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d275658aecaf26-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:12.044 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:12.045 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:12.045 DEBUG:boto:Method: GET 2024-04-16T20:07:12.045 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:12.045 DEBUG:boto:Data: 2024-04-16T20:07:12.045 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.045 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.045 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.045 DEBUG:boto:Params: {} 2024-04-16T20:07:12.045 DEBUG:boto:Token: None 2024-04-16T20:07:12.045 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj124 2024-04-16T20:07:12.045 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tyzaOJ2Yd/q//DWvfl3pjh46Fgs= 2024-04-16T20:07:12.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tyzaOJ2Yd/q//DWvfl3pjh46Fgs=', 'Content-Length': '0'} 2024-04-16T20:07:12.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976119617210a180-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.047 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:12.048 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:12.048 DEBUG:boto:Method: GET 2024-04-16T20:07:12.048 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:12.048 DEBUG:boto:Data: 2024-04-16T20:07:12.048 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.048 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.048 DEBUG:boto:Params: {} 2024-04-16T20:07:12.048 DEBUG:boto:Token: None 2024-04-16T20:07:12.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj124 2024-04-16T20:07:12.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tyzaOJ2Yd/q//DWvfl3pjh46Fgs= 2024-04-16T20:07:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tyzaOJ2Yd/q//DWvfl3pjh46Fgs=', 'Content-Length': '0'} 2024-04-16T20:07:12.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da21fcd07247ee5-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.051 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.051 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.051 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.051 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.051 DEBUG:boto:Method: GET 2024-04-16T20:07:12.051 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.051 DEBUG:boto:Data: 2024-04-16T20:07:12.051 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.051 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.051 DEBUG:boto:Params: {} 2024-04-16T20:07:12.052 DEBUG:boto:Token: None 2024-04-16T20:07:12.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076262b0d2f121bd6-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.054 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.055 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.055 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.055 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.055 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.055 DEBUG:boto:Method: GET 2024-04-16T20:07:12.055 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:12.055 DEBUG:boto:Data: 2024-04-16T20:07:12.055 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.055 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.055 DEBUG:boto:Params: {} 2024-04-16T20:07:12.056 DEBUG:boto:Token: None 2024-04-16T20:07:12.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7a69c215848b57-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.102 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:12.103 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.103 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.103 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.103 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.103 DEBUG:boto:Method: GET 2024-04-16T20:07:12.103 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.103 DEBUG:boto:Data: 2024-04-16T20:07:12.103 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.103 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.103 DEBUG:boto:Params: {} 2024-04-16T20:07:12.104 DEBUG:boto:Token: None 2024-04-16T20:07:12.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c= 2024-04-16T20:07:12.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c=', 'Content-Length': '0'} 2024-04-16T20:07:12.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d773fb66a1a0a3-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.106 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.107 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.107 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.107 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.107 DEBUG:boto:Method: GET 2024-04-16T20:07:12.107 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.107 DEBUG:boto:Data: 2024-04-16T20:07:12.107 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.107 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.107 DEBUG:boto:Params: {} 2024-04-16T20:07:12.107 DEBUG:boto:Token: None 2024-04-16T20:07:12.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c= 2024-04-16T20:07:12.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c=', 'Content-Length': '0'} 2024-04-16T20:07:12.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab27c9d155c947cf-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.110 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.110 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.110 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.110 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.110 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.110 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.111 DEBUG:boto:Data: 2024-04-16T20:07:12.111 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.111 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.111 DEBUG:boto:Params: {} 2024-04-16T20:07:12.111 DEBUG:boto:Token: None 2024-04-16T20:07:12.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VnmsHPSqXX1z3QcwmxU0ao36CYE= 2024-04-16T20:07:12.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VnmsHPSqXX1z3QcwmxU0ao36CYE=', 'Content-Length': '0'} 2024-04-16T20:07:12.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fba33c8745e3c2-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.114 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.114 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.114 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.114 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.114 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.114 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.114 DEBUG:boto:Data: 2024-04-16T20:07:12.114 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.114 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.114 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.114 DEBUG:boto:Params: {} 2024-04-16T20:07:12.114 DEBUG:boto:Token: None 2024-04-16T20:07:12.115 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VnmsHPSqXX1z3QcwmxU0ao36CYE= 2024-04-16T20:07:12.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VnmsHPSqXX1z3QcwmxU0ao36CYE=', 'Content-Length': '0'} 2024-04-16T20:07:12.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ded6498212f086-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:12.117 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.117 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.117 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.117 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.118 DEBUG:boto:Method: GET 2024-04-16T20:07:12.118 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.118 DEBUG:boto:Data: 2024-04-16T20:07:12.118 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.118 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.118 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.118 DEBUG:boto:Params: {} 2024-04-16T20:07:12.118 DEBUG:boto:Token: None 2024-04-16T20:07:12.118 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.118 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c= 2024-04-16T20:07:12.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c=', 'Content-Length': '0'} 2024-04-16T20:07:12.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4386acf16fa2698-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.162 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.162 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.163 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.163 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.163 DEBUG:boto:Method: GET 2024-04-16T20:07:12.163 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.163 DEBUG:boto:Data: 2024-04-16T20:07:12.163 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.163 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.163 DEBUG:boto:Params: {} 2024-04-16T20:07:12.163 DEBUG:boto:Token: None 2024-04-16T20:07:12.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:12.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c= 2024-04-16T20:07:12.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V08GqXojqqOlgzJ1nLdDY9T7v+c=', 'Content-Length': '0'} 2024-04-16T20:07:12.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e7d6ecb80128b1-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.166 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.166 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.166 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.166 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.166 DEBUG:boto:Method: GET 2024-04-16T20:07:12.167 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.167 DEBUG:boto:Data: 2024-04-16T20:07:12.167 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.167 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.167 DEBUG:boto:Params: {} 2024-04-16T20:07:12.167 DEBUG:boto:Token: None 2024-04-16T20:07:12.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a45452f4a6e0d4c-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.170 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.170 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.170 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.170 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.170 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.170 DEBUG:boto:Method: GET 2024-04-16T20:07:12.170 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.171 DEBUG:boto:Data: 2024-04-16T20:07:12.171 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.171 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.171 DEBUG:boto:Params: {} 2024-04-16T20:07:12.171 DEBUG:boto:Token: None 2024-04-16T20:07:12.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb4ee0dee1e2e0f6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.175 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.175 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.175 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.175 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.175 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:12.175 DEBUG:boto:Data: 2024-04-16T20:07:12.175 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.176 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.176 DEBUG:boto:Params: {} 2024-04-16T20:07:12.176 DEBUG:boto:Token: None 2024-04-16T20:07:12.176 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145 2024-04-16T20:07:12.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kYmGkxd7YKUcHo4nSD8jMS7EYHU= 2024-04-16T20:07:12.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kYmGkxd7YKUcHo4nSD8jMS7EYHU=', 'Content-Length': '0'} 2024-04-16T20:07:12.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ac2b21d5d94f34-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.178 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.178 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.178 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.178 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:12.178 DEBUG:boto:Data: 2024-04-16T20:07:12.178 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.179 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.179 DEBUG:boto:Params: {} 2024-04-16T20:07:12.179 DEBUG:boto:Token: None 2024-04-16T20:07:12.179 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145 2024-04-16T20:07:12.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kYmGkxd7YKUcHo4nSD8jMS7EYHU= 2024-04-16T20:07:12.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kYmGkxd7YKUcHo4nSD8jMS7EYHU=', 'Content-Length': '0'} 2024-04-16T20:07:12.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093190e3a57886ed6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:12.182 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.182 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.182 DEBUG:boto:Method: GET 2024-04-16T20:07:12.182 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:12.182 DEBUG:boto:Data: 2024-04-16T20:07:12.182 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.182 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.182 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.182 DEBUG:boto:Params: {} 2024-04-16T20:07:12.182 DEBUG:boto:Token: None 2024-04-16T20:07:12.182 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145 2024-04-16T20:07:12.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b2fW01u67om98TtNgdAYvJ/UHn8= 2024-04-16T20:07:12.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b2fW01u67om98TtNgdAYvJ/UHn8=', 'Content-Length': '0'} 2024-04-16T20:07:12.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044fe6b4d2274b89f-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.185 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:12.185 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:12.185 DEBUG:boto:Method: GET 2024-04-16T20:07:12.185 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:12.185 DEBUG:boto:Data: 2024-04-16T20:07:12.186 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.186 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.186 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.186 DEBUG:boto:Params: {} 2024-04-16T20:07:12.186 DEBUG:boto:Token: None 2024-04-16T20:07:12.186 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj145 2024-04-16T20:07:12.186 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b2fW01u67om98TtNgdAYvJ/UHn8= 2024-04-16T20:07:12.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b2fW01u67om98TtNgdAYvJ/UHn8=', 'Content-Length': '0'} 2024-04-16T20:07:12.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c5c140d10e27c35-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.189 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.189 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.189 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.189 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.189 DEBUG:boto:Method: GET 2024-04-16T20:07:12.189 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.189 DEBUG:boto:Data: 2024-04-16T20:07:12.189 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.189 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.189 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.189 DEBUG:boto:Params: {} 2024-04-16T20:07:12.190 DEBUG:boto:Token: None 2024-04-16T20:07:12.190 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002edd171aaebbf086-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.192 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.193 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.193 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.193 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.193 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.193 DEBUG:boto:Method: GET 2024-04-16T20:07:12.193 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:12.193 DEBUG:boto:Data: 2024-04-16T20:07:12.193 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.193 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.193 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.193 DEBUG:boto:Params: {} 2024-04-16T20:07:12.194 DEBUG:boto:Token: None 2024-04-16T20:07:12.194 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.194 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eaa3ebafb513ac0-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.197 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:12.197 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.198 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.198 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.198 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.198 DEBUG:boto:Method: GET 2024-04-16T20:07:12.198 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.198 DEBUG:boto:Data: 2024-04-16T20:07:12.198 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.198 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.198 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.198 DEBUG:boto:Params: {} 2024-04-16T20:07:12.199 DEBUG:boto:Token: None 2024-04-16T20:07:12.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA= 2024-04-16T20:07:12.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA=', 'Content-Length': '0'} 2024-04-16T20:07:12.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f2d1e3059b3e82-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.201 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.201 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.201 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.202 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.202 DEBUG:boto:Method: GET 2024-04-16T20:07:12.202 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.202 DEBUG:boto:Data: 2024-04-16T20:07:12.202 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.202 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.202 DEBUG:boto:Params: {} 2024-04-16T20:07:12.202 DEBUG:boto:Token: None 2024-04-16T20:07:12.202 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA= 2024-04-16T20:07:12.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA=', 'Content-Length': '0'} 2024-04-16T20:07:12.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b017a2d0ff4c28b-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.205 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.205 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.205 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.205 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.205 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.205 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.206 DEBUG:boto:Data: 2024-04-16T20:07:12.206 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.206 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.206 DEBUG:boto:Params: {} 2024-04-16T20:07:12.206 DEBUG:boto:Token: None 2024-04-16T20:07:12.206 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.206 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xiLTbXh/ggb2MQpKA73C2gg6VAk= 2024-04-16T20:07:12.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xiLTbXh/ggb2MQpKA73C2gg6VAk=', 'Content-Length': '0'} 2024-04-16T20:07:12.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b4248734c5a6a5-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.209 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.209 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.209 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.209 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.209 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.209 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.209 DEBUG:boto:Data: 2024-04-16T20:07:12.209 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.209 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.210 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.210 DEBUG:boto:Params: {} 2024-04-16T20:07:12.210 DEBUG:boto:Token: None 2024-04-16T20:07:12.210 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.210 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xiLTbXh/ggb2MQpKA73C2gg6VAk= 2024-04-16T20:07:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xiLTbXh/ggb2MQpKA73C2gg6VAk=', 'Content-Length': '0'} 2024-04-16T20:07:12.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089243f2530cc464-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:12.212 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.213 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.213 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.213 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.213 DEBUG:boto:Method: GET 2024-04-16T20:07:12.213 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.213 DEBUG:boto:Data: 2024-04-16T20:07:12.213 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.213 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.213 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.213 DEBUG:boto:Params: {} 2024-04-16T20:07:12.213 DEBUG:boto:Token: None 2024-04-16T20:07:12.214 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.214 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA= 2024-04-16T20:07:12.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA=', 'Content-Length': '0'} 2024-04-16T20:07:12.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ac43f374940773-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.216 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.216 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.216 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.217 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.217 DEBUG:boto:Method: GET 2024-04-16T20:07:12.217 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.217 DEBUG:boto:Data: 2024-04-16T20:07:12.217 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.217 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.217 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.217 DEBUG:boto:Params: {} 2024-04-16T20:07:12.217 DEBUG:boto:Token: None 2024-04-16T20:07:12.217 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:12.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA= 2024-04-16T20:07:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vR6cQBrtk5X8qbpnyKd8UZFgMEA=', 'Content-Length': '0'} 2024-04-16T20:07:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d578686cd24161f6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.220 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.220 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.220 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.220 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.220 DEBUG:boto:Method: GET 2024-04-16T20:07:12.220 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.220 DEBUG:boto:Data: 2024-04-16T20:07:12.221 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.221 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.221 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.221 DEBUG:boto:Params: {} 2024-04-16T20:07:12.221 DEBUG:boto:Token: None 2024-04-16T20:07:12.221 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b32ab6aba5d5550a-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.223 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.224 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.224 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.224 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.224 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.224 DEBUG:boto:Method: GET 2024-04-16T20:07:12.224 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.224 DEBUG:boto:Data: 2024-04-16T20:07:12.224 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.224 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.224 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.224 DEBUG:boto:Params: {} 2024-04-16T20:07:12.225 DEBUG:boto:Token: None 2024-04-16T20:07:12.225 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.225 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255813fcddb0a616-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.270 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.271 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.271 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.271 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.271 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:12.271 DEBUG:boto:Data: 2024-04-16T20:07:12.271 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.271 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.271 DEBUG:boto:Params: {} 2024-04-16T20:07:12.271 DEBUG:boto:Token: None 2024-04-16T20:07:12.272 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148 2024-04-16T20:07:12.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:90VgA1/yHZ36IYLH5mFX7L6z8os= 2024-04-16T20:07:12.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:90VgA1/yHZ36IYLH5mFX7L6z8os=', 'Content-Length': '0'} 2024-04-16T20:07:12.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daec3446270cefc-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.274 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.274 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.274 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.274 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:12.274 DEBUG:boto:Data: 2024-04-16T20:07:12.274 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.274 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.275 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.275 DEBUG:boto:Params: {} 2024-04-16T20:07:12.275 DEBUG:boto:Token: None 2024-04-16T20:07:12.275 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148 2024-04-16T20:07:12.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:90VgA1/yHZ36IYLH5mFX7L6z8os= 2024-04-16T20:07:12.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:90VgA1/yHZ36IYLH5mFX7L6z8os=', 'Content-Length': '0'} 2024-04-16T20:07:12.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6df60a86b0b9719-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:12.278 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.278 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.278 DEBUG:boto:Method: GET 2024-04-16T20:07:12.278 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:12.278 DEBUG:boto:Data: 2024-04-16T20:07:12.278 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.278 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.278 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.278 DEBUG:boto:Params: {} 2024-04-16T20:07:12.278 DEBUG:boto:Token: None 2024-04-16T20:07:12.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148 2024-04-16T20:07:12.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wPl9m4hH+unKqIbBY6cOgOSbmCA= 2024-04-16T20:07:12.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wPl9m4hH+unKqIbBY6cOgOSbmCA=', 'Content-Length': '0'} 2024-04-16T20:07:12.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb790a51a16efb6-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.281 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:12.281 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:12.281 DEBUG:boto:Method: GET 2024-04-16T20:07:12.281 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:12.281 DEBUG:boto:Data: 2024-04-16T20:07:12.281 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.281 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.281 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.282 DEBUG:boto:Params: {} 2024-04-16T20:07:12.282 DEBUG:boto:Token: None 2024-04-16T20:07:12.282 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj148 2024-04-16T20:07:12.282 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wPl9m4hH+unKqIbBY6cOgOSbmCA= 2024-04-16T20:07:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wPl9m4hH+unKqIbBY6cOgOSbmCA=', 'Content-Length': '0'} 2024-04-16T20:07:12.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000220618a7f4617d1c-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.284 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.285 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.285 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.285 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.285 DEBUG:boto:Method: GET 2024-04-16T20:07:12.285 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.285 DEBUG:boto:Data: 2024-04-16T20:07:12.285 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.285 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.285 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.285 DEBUG:boto:Params: {} 2024-04-16T20:07:12.285 DEBUG:boto:Token: None 2024-04-16T20:07:12.286 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.286 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007813546299c68f61-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.288 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.288 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.289 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.289 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.289 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.289 DEBUG:boto:Method: GET 2024-04-16T20:07:12.289 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:12.289 DEBUG:boto:Data: 2024-04-16T20:07:12.289 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.289 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.289 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.289 DEBUG:boto:Params: {} 2024-04-16T20:07:12.289 DEBUG:boto:Token: None 2024-04-16T20:07:12.290 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.290 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e87e6477e23f49-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.293 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:12.294 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.294 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.294 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.294 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.294 DEBUG:boto:Method: GET 2024-04-16T20:07:12.294 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.294 DEBUG:boto:Data: 2024-04-16T20:07:12.294 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.295 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.295 DEBUG:boto:Params: {} 2024-04-16T20:07:12.295 DEBUG:boto:Token: None 2024-04-16T20:07:12.295 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo= 2024-04-16T20:07:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo=', 'Content-Length': '0'} 2024-04-16T20:07:12.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0a98b34736cc75-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.297 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.298 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.298 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.298 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.298 DEBUG:boto:Method: GET 2024-04-16T20:07:12.298 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.298 DEBUG:boto:Data: 2024-04-16T20:07:12.298 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.298 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.298 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.298 DEBUG:boto:Params: {} 2024-04-16T20:07:12.299 DEBUG:boto:Token: None 2024-04-16T20:07:12.299 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo= 2024-04-16T20:07:12.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo=', 'Content-Length': '0'} 2024-04-16T20:07:12.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b013ec41187f27-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.301 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.301 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.301 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.301 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.301 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.302 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.302 DEBUG:boto:Data: 2024-04-16T20:07:12.302 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.302 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.302 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.302 DEBUG:boto:Params: {} 2024-04-16T20:07:12.302 DEBUG:boto:Token: None 2024-04-16T20:07:12.302 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:luIkycdKKtIFtn36pRFGfdv4XVE= 2024-04-16T20:07:12.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:luIkycdKKtIFtn36pRFGfdv4XVE=', 'Content-Length': '0'} 2024-04-16T20:07:12.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007af758051d8c0877-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.305 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.305 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.305 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.305 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.305 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.305 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.305 DEBUG:boto:Data: 2024-04-16T20:07:12.305 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.305 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.306 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.306 DEBUG:boto:Params: {} 2024-04-16T20:07:12.306 DEBUG:boto:Token: None 2024-04-16T20:07:12.306 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.306 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:luIkycdKKtIFtn36pRFGfdv4XVE= 2024-04-16T20:07:12.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:luIkycdKKtIFtn36pRFGfdv4XVE=', 'Content-Length': '0'} 2024-04-16T20:07:12.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e49b2d917d37080-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:12.309 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.309 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.309 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.309 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.309 DEBUG:boto:Method: GET 2024-04-16T20:07:12.309 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.309 DEBUG:boto:Data: 2024-04-16T20:07:12.309 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.309 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.309 DEBUG:boto:Params: {} 2024-04-16T20:07:12.309 DEBUG:boto:Token: None 2024-04-16T20:07:12.310 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.310 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo= 2024-04-16T20:07:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo=', 'Content-Length': '0'} 2024-04-16T20:07:12.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b40f22c9cd5c28-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.312 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.312 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.312 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.312 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.312 DEBUG:boto:Method: GET 2024-04-16T20:07:12.313 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.313 DEBUG:boto:Data: 2024-04-16T20:07:12.313 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.313 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.313 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.313 DEBUG:boto:Params: {} 2024-04-16T20:07:12.313 DEBUG:boto:Token: None 2024-04-16T20:07:12.313 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:12.313 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo= 2024-04-16T20:07:12.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t9b6/+ch6LytZzONtgEXgb92dHo=', 'Content-Length': '0'} 2024-04-16T20:07:12.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e85df900c198fc20-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.316 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.316 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.316 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.316 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.316 DEBUG:boto:Method: GET 2024-04-16T20:07:12.316 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.316 DEBUG:boto:Data: 2024-04-16T20:07:12.316 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.316 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.316 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.316 DEBUG:boto:Params: {} 2024-04-16T20:07:12.317 DEBUG:boto:Token: None 2024-04-16T20:07:12.317 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872eaa85f4c232f0-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.319 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.320 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.320 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.320 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.320 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.320 DEBUG:boto:Method: GET 2024-04-16T20:07:12.320 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.320 DEBUG:boto:Data: 2024-04-16T20:07:12.320 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.320 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.320 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.320 DEBUG:boto:Params: {} 2024-04-16T20:07:12.321 DEBUG:boto:Token: None 2024-04-16T20:07:12.321 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001602b8bec1537c70-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.324 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.324 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.324 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.325 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.325 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:12.325 DEBUG:boto:Data: 2024-04-16T20:07:12.325 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.325 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.325 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.325 DEBUG:boto:Params: {} 2024-04-16T20:07:12.325 DEBUG:boto:Token: None 2024-04-16T20:07:12.325 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155 2024-04-16T20:07:12.325 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GVyu6SBTajYCaKfCg18Y1/z7tZQ= 2024-04-16T20:07:12.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GVyu6SBTajYCaKfCg18Y1/z7tZQ=', 'Content-Length': '0'} 2024-04-16T20:07:12.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043aa626c3f9f0338-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.328 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.328 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.328 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.328 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:12.328 DEBUG:boto:Data: 2024-04-16T20:07:12.328 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.328 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.328 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.328 DEBUG:boto:Params: {} 2024-04-16T20:07:12.329 DEBUG:boto:Token: None 2024-04-16T20:07:12.329 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155 2024-04-16T20:07:12.329 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GVyu6SBTajYCaKfCg18Y1/z7tZQ= 2024-04-16T20:07:12.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GVyu6SBTajYCaKfCg18Y1/z7tZQ=', 'Content-Length': '0'} 2024-04-16T20:07:12.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035688ca8f033ddee-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:12.331 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.331 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.331 DEBUG:boto:Method: GET 2024-04-16T20:07:12.331 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:12.331 DEBUG:boto:Data: 2024-04-16T20:07:12.331 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.332 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.332 DEBUG:boto:Params: {} 2024-04-16T20:07:12.332 DEBUG:boto:Token: None 2024-04-16T20:07:12.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155 2024-04-16T20:07:12.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDRoHkMPSeEs3aFA7LDFSBkXUo0= 2024-04-16T20:07:12.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDRoHkMPSeEs3aFA7LDFSBkXUo0=', 'Content-Length': '0'} 2024-04-16T20:07:12.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee781c57653cbe89-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.374 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:12.374 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:12.375 DEBUG:boto:Method: GET 2024-04-16T20:07:12.375 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:12.375 DEBUG:boto:Data: 2024-04-16T20:07:12.375 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.375 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.375 DEBUG:boto:Params: {} 2024-04-16T20:07:12.375 DEBUG:boto:Token: None 2024-04-16T20:07:12.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj155 2024-04-16T20:07:12.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDRoHkMPSeEs3aFA7LDFSBkXUo0= 2024-04-16T20:07:12.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDRoHkMPSeEs3aFA7LDFSBkXUo0=', 'Content-Length': '0'} 2024-04-16T20:07:12.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002970b844a8e0ff1a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.378 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.378 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.378 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.378 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.378 DEBUG:boto:Method: GET 2024-04-16T20:07:12.378 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.378 DEBUG:boto:Data: 2024-04-16T20:07:12.378 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.378 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.378 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.378 DEBUG:boto:Params: {} 2024-04-16T20:07:12.379 DEBUG:boto:Token: None 2024-04-16T20:07:12.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c8e524f155862e-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.381 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.382 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.382 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.382 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.382 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.382 DEBUG:boto:Method: GET 2024-04-16T20:07:12.382 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:12.382 DEBUG:boto:Data: 2024-04-16T20:07:12.382 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.382 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.382 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.383 DEBUG:boto:Params: {} 2024-04-16T20:07:12.383 DEBUG:boto:Token: None 2024-04-16T20:07:12.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f105d7f4a9365389-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.386 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:12.387 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.387 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.387 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.387 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.387 DEBUG:boto:Method: GET 2024-04-16T20:07:12.387 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.387 DEBUG:boto:Data: 2024-04-16T20:07:12.387 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.387 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.387 DEBUG:boto:Params: {} 2024-04-16T20:07:12.387 DEBUG:boto:Token: None 2024-04-16T20:07:12.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI= 2024-04-16T20:07:12.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI=', 'Content-Length': '0'} 2024-04-16T20:07:12.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e285faf60a3de68-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.390 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.390 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.390 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.391 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.391 DEBUG:boto:Method: GET 2024-04-16T20:07:12.391 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.391 DEBUG:boto:Data: 2024-04-16T20:07:12.391 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.391 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.391 DEBUG:boto:Params: {} 2024-04-16T20:07:12.391 DEBUG:boto:Token: None 2024-04-16T20:07:12.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI= 2024-04-16T20:07:12.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI=', 'Content-Length': '0'} 2024-04-16T20:07:12.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4ab4ce9e1f9513-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.394 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.394 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.394 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.394 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.394 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.394 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.394 DEBUG:boto:Data: 2024-04-16T20:07:12.394 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.394 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.394 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.394 DEBUG:boto:Params: {} 2024-04-16T20:07:12.395 DEBUG:boto:Token: None 2024-04-16T20:07:12.395 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Li9raZsEuQ9+RCDywqQVl7CC+5A= 2024-04-16T20:07:12.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Li9raZsEuQ9+RCDywqQVl7CC+5A=', 'Content-Length': '0'} 2024-04-16T20:07:12.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003061a4c5738b6683-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.397 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.397 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.397 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.397 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.397 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.397 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.397 DEBUG:boto:Data: 2024-04-16T20:07:12.398 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.398 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.398 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.398 DEBUG:boto:Params: {} 2024-04-16T20:07:12.398 DEBUG:boto:Token: None 2024-04-16T20:07:12.398 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.398 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Li9raZsEuQ9+RCDywqQVl7CC+5A= 2024-04-16T20:07:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Li9raZsEuQ9+RCDywqQVl7CC+5A=', 'Content-Length': '0'} 2024-04-16T20:07:12.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ffa72e64747cb9-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:12.400 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.400 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.400 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.400 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.401 DEBUG:boto:Method: GET 2024-04-16T20:07:12.401 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.401 DEBUG:boto:Data: 2024-04-16T20:07:12.401 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.401 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.401 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.401 DEBUG:boto:Params: {} 2024-04-16T20:07:12.401 DEBUG:boto:Token: None 2024-04-16T20:07:12.401 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI= 2024-04-16T20:07:12.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI=', 'Content-Length': '0'} 2024-04-16T20:07:12.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a185aa7c16ff20b-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.404 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.404 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.404 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.404 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.404 DEBUG:boto:Method: GET 2024-04-16T20:07:12.404 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.404 DEBUG:boto:Data: 2024-04-16T20:07:12.404 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.404 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.404 DEBUG:boto:Params: {} 2024-04-16T20:07:12.405 DEBUG:boto:Token: None 2024-04-16T20:07:12.405 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:12.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI= 2024-04-16T20:07:12.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vzdafLEppDR1ihNOLkzTZP8LnCI=', 'Content-Length': '0'} 2024-04-16T20:07:12.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040eacbd16a2c2698-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.407 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.407 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.407 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.407 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.407 DEBUG:boto:Method: GET 2024-04-16T20:07:12.407 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.407 DEBUG:boto:Data: 2024-04-16T20:07:12.407 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.407 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.407 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.408 DEBUG:boto:Params: {} 2024-04-16T20:07:12.408 DEBUG:boto:Token: None 2024-04-16T20:07:12.408 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6dd88deb92fd96-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.410 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.411 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.411 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.411 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.411 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.411 DEBUG:boto:Method: GET 2024-04-16T20:07:12.411 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.411 DEBUG:boto:Data: 2024-04-16T20:07:12.411 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.411 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.411 DEBUG:boto:Params: {} 2024-04-16T20:07:12.412 DEBUG:boto:Token: None 2024-04-16T20:07:12.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005331081ba353c660-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.415 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.416 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.416 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.416 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.416 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:12.416 DEBUG:boto:Data: 2024-04-16T20:07:12.416 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.416 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.416 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.416 DEBUG:boto:Params: {} 2024-04-16T20:07:12.417 DEBUG:boto:Token: None 2024-04-16T20:07:12.417 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17 2024-04-16T20:07:12.417 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bOsn3BpX8uULMTG7ujzwcfdN6QM= 2024-04-16T20:07:12.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bOsn3BpX8uULMTG7ujzwcfdN6QM=', 'Content-Length': '0'} 2024-04-16T20:07:12.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d07ec60d24014d1b-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.419 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.419 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.419 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.419 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:12.419 DEBUG:boto:Data: 2024-04-16T20:07:12.419 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.419 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.419 DEBUG:boto:Params: {} 2024-04-16T20:07:12.420 DEBUG:boto:Token: None 2024-04-16T20:07:12.420 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17 2024-04-16T20:07:12.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bOsn3BpX8uULMTG7ujzwcfdN6QM= 2024-04-16T20:07:12.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bOsn3BpX8uULMTG7ujzwcfdN6QM=', 'Content-Length': '0'} 2024-04-16T20:07:12.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e193de9edfad2ed5-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:12.422 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.422 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.422 DEBUG:boto:Method: GET 2024-04-16T20:07:12.422 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:12.422 DEBUG:boto:Data: 2024-04-16T20:07:12.422 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.422 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.422 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.423 DEBUG:boto:Params: {} 2024-04-16T20:07:12.423 DEBUG:boto:Token: None 2024-04-16T20:07:12.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17 2024-04-16T20:07:12.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aDuKeEkA2FC8gM87CeIBX3jtDmk= 2024-04-16T20:07:12.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aDuKeEkA2FC8gM87CeIBX3jtDmk=', 'Content-Length': '0'} 2024-04-16T20:07:12.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a52b1d584e7d3a0b-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.466 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:12.466 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:12.467 DEBUG:boto:Method: GET 2024-04-16T20:07:12.467 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:12.467 DEBUG:boto:Data: 2024-04-16T20:07:12.467 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.467 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.467 DEBUG:boto:Params: {} 2024-04-16T20:07:12.467 DEBUG:boto:Token: None 2024-04-16T20:07:12.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj17 2024-04-16T20:07:12.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aDuKeEkA2FC8gM87CeIBX3jtDmk= 2024-04-16T20:07:12.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aDuKeEkA2FC8gM87CeIBX3jtDmk=', 'Content-Length': '0'} 2024-04-16T20:07:12.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f713c4208a5d53-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.469 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.469 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.470 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.470 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.470 DEBUG:boto:Method: GET 2024-04-16T20:07:12.470 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.470 DEBUG:boto:Data: 2024-04-16T20:07:12.470 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.470 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.470 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.470 DEBUG:boto:Params: {} 2024-04-16T20:07:12.470 DEBUG:boto:Token: None 2024-04-16T20:07:12.470 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ab27aa0ec6d295-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.473 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.474 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.474 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.474 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.474 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.474 DEBUG:boto:Method: GET 2024-04-16T20:07:12.474 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:12.474 DEBUG:boto:Data: 2024-04-16T20:07:12.474 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.474 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.474 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.475 DEBUG:boto:Params: {} 2024-04-16T20:07:12.475 DEBUG:boto:Token: None 2024-04-16T20:07:12.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051e64e1606892ef6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.478 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:12.479 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.479 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.479 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.479 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.479 DEBUG:boto:Method: GET 2024-04-16T20:07:12.479 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.479 DEBUG:boto:Data: 2024-04-16T20:07:12.479 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.479 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.480 DEBUG:boto:Params: {} 2024-04-16T20:07:12.480 DEBUG:boto:Token: None 2024-04-16T20:07:12.480 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE= 2024-04-16T20:07:12.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE=', 'Content-Length': '0'} 2024-04-16T20:07:12.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e34e0c0d24bdde0c-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.482 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.483 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.483 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.483 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.483 DEBUG:boto:Method: GET 2024-04-16T20:07:12.483 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.483 DEBUG:boto:Data: 2024-04-16T20:07:12.483 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.483 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.484 DEBUG:boto:Params: {} 2024-04-16T20:07:12.484 DEBUG:boto:Token: None 2024-04-16T20:07:12.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE= 2024-04-16T20:07:12.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE=', 'Content-Length': '0'} 2024-04-16T20:07:12.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6c34ace23d13f1-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.526 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.526 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.526 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.527 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.527 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.527 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.527 DEBUG:boto:Data: 2024-04-16T20:07:12.527 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.527 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.527 DEBUG:boto:Params: {} 2024-04-16T20:07:12.527 DEBUG:boto:Token: None 2024-04-16T20:07:12.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iBdINR0RcCHIRbA41ljTakZYsO4= 2024-04-16T20:07:12.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iBdINR0RcCHIRbA41ljTakZYsO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc94261a125605e8-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.530 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.530 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.530 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.530 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.530 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.530 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.530 DEBUG:boto:Data: 2024-04-16T20:07:12.530 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.530 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.530 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.531 DEBUG:boto:Params: {} 2024-04-16T20:07:12.531 DEBUG:boto:Token: None 2024-04-16T20:07:12.531 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iBdINR0RcCHIRbA41ljTakZYsO4= 2024-04-16T20:07:12.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iBdINR0RcCHIRbA41ljTakZYsO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003091dcebae3713b0-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:12.533 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.533 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.533 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.533 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.533 DEBUG:boto:Method: GET 2024-04-16T20:07:12.533 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.534 DEBUG:boto:Data: 2024-04-16T20:07:12.534 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.534 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.534 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.534 DEBUG:boto:Params: {} 2024-04-16T20:07:12.534 DEBUG:boto:Token: None 2024-04-16T20:07:12.534 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.534 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE= 2024-04-16T20:07:12.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE=', 'Content-Length': '0'} 2024-04-16T20:07:12.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4852a4933244a0b-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.536 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.536 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.537 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.537 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.537 DEBUG:boto:Method: GET 2024-04-16T20:07:12.537 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.537 DEBUG:boto:Data: 2024-04-16T20:07:12.537 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.537 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.537 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.537 DEBUG:boto:Params: {} 2024-04-16T20:07:12.537 DEBUG:boto:Token: None 2024-04-16T20:07:12.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:12.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE= 2024-04-16T20:07:12.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bZfbrixLJqlj8XJXG30JuNuLveE=', 'Content-Length': '0'} 2024-04-16T20:07:12.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0844b9db8f23646-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.540 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.540 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.540 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.540 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.540 DEBUG:boto:Method: GET 2024-04-16T20:07:12.540 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.540 DEBUG:boto:Data: 2024-04-16T20:07:12.540 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.540 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.540 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.540 DEBUG:boto:Params: {} 2024-04-16T20:07:12.541 DEBUG:boto:Token: None 2024-04-16T20:07:12.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a0fbf547777e515-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.543 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.544 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.544 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.544 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.544 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.544 DEBUG:boto:Method: GET 2024-04-16T20:07:12.544 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.544 DEBUG:boto:Data: 2024-04-16T20:07:12.544 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.545 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.545 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.545 DEBUG:boto:Params: {} 2024-04-16T20:07:12.545 DEBUG:boto:Token: None 2024-04-16T20:07:12.545 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a954dbba5efaae4a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.548 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.549 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.549 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.549 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.549 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:12.549 DEBUG:boto:Data: 2024-04-16T20:07:12.549 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.549 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.549 DEBUG:boto:Params: {} 2024-04-16T20:07:12.550 DEBUG:boto:Token: None 2024-04-16T20:07:12.550 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185 2024-04-16T20:07:12.550 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9KOh0HK3z5eW2CHxNtZEry2moA= 2024-04-16T20:07:12.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9KOh0HK3z5eW2CHxNtZEry2moA=', 'Content-Length': '0'} 2024-04-16T20:07:12.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009284a3edba3c8ea1-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.552 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.552 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.552 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.552 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:12.552 DEBUG:boto:Data: 2024-04-16T20:07:12.552 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.552 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.552 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.553 DEBUG:boto:Params: {} 2024-04-16T20:07:12.553 DEBUG:boto:Token: None 2024-04-16T20:07:12.553 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185 2024-04-16T20:07:12.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q9KOh0HK3z5eW2CHxNtZEry2moA= 2024-04-16T20:07:12.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q9KOh0HK3z5eW2CHxNtZEry2moA=', 'Content-Length': '0'} 2024-04-16T20:07:12.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b53b9b9b63de5489-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:12.555 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.555 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.555 DEBUG:boto:Method: GET 2024-04-16T20:07:12.555 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:12.556 DEBUG:boto:Data: 2024-04-16T20:07:12.556 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.556 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.556 DEBUG:boto:Params: {} 2024-04-16T20:07:12.556 DEBUG:boto:Token: None 2024-04-16T20:07:12.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185 2024-04-16T20:07:12.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7abhDvDwiVRB3C9S7EJ/vVsPGT0= 2024-04-16T20:07:12.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7abhDvDwiVRB3C9S7EJ/vVsPGT0=', 'Content-Length': '0'} 2024-04-16T20:07:12.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b29cf6bff3bffcb-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.558 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:12.558 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:12.559 DEBUG:boto:Method: GET 2024-04-16T20:07:12.559 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:12.559 DEBUG:boto:Data: 2024-04-16T20:07:12.559 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.559 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.559 DEBUG:boto:Params: {} 2024-04-16T20:07:12.559 DEBUG:boto:Token: None 2024-04-16T20:07:12.559 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj185 2024-04-16T20:07:12.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7abhDvDwiVRB3C9S7EJ/vVsPGT0= 2024-04-16T20:07:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7abhDvDwiVRB3C9S7EJ/vVsPGT0=', 'Content-Length': '0'} 2024-04-16T20:07:12.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000151caa514571475d-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.602 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.602 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.602 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.603 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.603 DEBUG:boto:Method: GET 2024-04-16T20:07:12.603 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.603 DEBUG:boto:Data: 2024-04-16T20:07:12.603 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.603 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.603 DEBUG:boto:Params: {} 2024-04-16T20:07:12.603 DEBUG:boto:Token: None 2024-04-16T20:07:12.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea48b37cece09b95-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.607 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.607 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.607 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.607 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.607 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.607 DEBUG:boto:Method: GET 2024-04-16T20:07:12.607 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:12.607 DEBUG:boto:Data: 2024-04-16T20:07:12.607 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.608 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.608 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.608 DEBUG:boto:Params: {} 2024-04-16T20:07:12.608 DEBUG:boto:Token: None 2024-04-16T20:07:12.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d4bf4385602e8b-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.612 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:12.612 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.612 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.612 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.612 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.612 DEBUG:boto:Method: GET 2024-04-16T20:07:12.612 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.612 DEBUG:boto:Data: 2024-04-16T20:07:12.613 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.613 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.613 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.613 DEBUG:boto:Params: {} 2024-04-16T20:07:12.613 DEBUG:boto:Token: None 2024-04-16T20:07:12.613 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.613 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw= 2024-04-16T20:07:12.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw=', 'Content-Length': '0'} 2024-04-16T20:07:12.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058927ba68f9f5313-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.658 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.659 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.659 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.659 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.659 DEBUG:boto:Method: GET 2024-04-16T20:07:12.659 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.659 DEBUG:boto:Data: 2024-04-16T20:07:12.659 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.659 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.659 DEBUG:boto:Params: {} 2024-04-16T20:07:12.659 DEBUG:boto:Token: None 2024-04-16T20:07:12.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw= 2024-04-16T20:07:12.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw=', 'Content-Length': '0'} 2024-04-16T20:07:12.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a03a9d544f86915-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.662 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.662 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.662 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.662 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.662 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.662 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.663 DEBUG:boto:Data: 2024-04-16T20:07:12.663 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.663 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.663 DEBUG:boto:Params: {} 2024-04-16T20:07:12.663 DEBUG:boto:Token: None 2024-04-16T20:07:12.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rjbqr8z7B/QW3f9lITEecmv9iqE= 2024-04-16T20:07:12.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rjbqr8z7B/QW3f9lITEecmv9iqE=', 'Content-Length': '0'} 2024-04-16T20:07:12.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87923a69045f029-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.666 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.666 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.666 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.666 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.666 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.666 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.666 DEBUG:boto:Data: 2024-04-16T20:07:12.666 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.666 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.666 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.666 DEBUG:boto:Params: {} 2024-04-16T20:07:12.667 DEBUG:boto:Token: None 2024-04-16T20:07:12.667 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rjbqr8z7B/QW3f9lITEecmv9iqE= 2024-04-16T20:07:12.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rjbqr8z7B/QW3f9lITEecmv9iqE=', 'Content-Length': '0'} 2024-04-16T20:07:12.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df395be7e09fbe85-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:12.669 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.669 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.669 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.669 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.669 DEBUG:boto:Method: GET 2024-04-16T20:07:12.669 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.669 DEBUG:boto:Data: 2024-04-16T20:07:12.670 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.670 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.670 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.670 DEBUG:boto:Params: {} 2024-04-16T20:07:12.670 DEBUG:boto:Token: None 2024-04-16T20:07:12.670 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.670 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw= 2024-04-16T20:07:12.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw=', 'Content-Length': '0'} 2024-04-16T20:07:12.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359f7167171f3712-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.672 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.673 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.673 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.673 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.673 DEBUG:boto:Method: GET 2024-04-16T20:07:12.673 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.673 DEBUG:boto:Data: 2024-04-16T20:07:12.673 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.673 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.673 DEBUG:boto:Params: {} 2024-04-16T20:07:12.673 DEBUG:boto:Token: None 2024-04-16T20:07:12.674 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:12.674 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw= 2024-04-16T20:07:12.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOvwPxMCTYMJSS4a1VJJyNG+rjw=', 'Content-Length': '0'} 2024-04-16T20:07:12.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc858a1174443fd5-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.676 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.676 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.676 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.676 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.677 DEBUG:boto:Method: GET 2024-04-16T20:07:12.677 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.677 DEBUG:boto:Data: 2024-04-16T20:07:12.677 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.677 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.677 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.677 DEBUG:boto:Params: {} 2024-04-16T20:07:12.677 DEBUG:boto:Token: None 2024-04-16T20:07:12.677 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008486d246cc952f5b-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.680 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.680 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.680 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.680 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.680 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.680 DEBUG:boto:Method: GET 2024-04-16T20:07:12.681 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.681 DEBUG:boto:Data: 2024-04-16T20:07:12.681 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.681 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.681 DEBUG:boto:Params: {} 2024-04-16T20:07:12.681 DEBUG:boto:Token: None 2024-04-16T20:07:12.681 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007eb155e95e3d296-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.684 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.685 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.685 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.685 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.685 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:12.685 DEBUG:boto:Data: 2024-04-16T20:07:12.685 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.685 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.685 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.685 DEBUG:boto:Params: {} 2024-04-16T20:07:12.686 DEBUG:boto:Token: None 2024-04-16T20:07:12.686 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19 2024-04-16T20:07:12.686 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e+epzV4nOPtK/Oqqs3nskwlDid4= 2024-04-16T20:07:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e+epzV4nOPtK/Oqqs3nskwlDid4=', 'Content-Length': '0'} 2024-04-16T20:07:12.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb41e368b8848a3-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.688 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.688 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.688 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.688 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:12.689 DEBUG:boto:Data: 2024-04-16T20:07:12.689 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.689 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.689 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.689 DEBUG:boto:Params: {} 2024-04-16T20:07:12.689 DEBUG:boto:Token: None 2024-04-16T20:07:12.689 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19 2024-04-16T20:07:12.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e+epzV4nOPtK/Oqqs3nskwlDid4= 2024-04-16T20:07:12.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e+epzV4nOPtK/Oqqs3nskwlDid4=', 'Content-Length': '0'} 2024-04-16T20:07:12.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fed5c2850dff82-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:12.692 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.692 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.692 DEBUG:boto:Method: GET 2024-04-16T20:07:12.692 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:12.692 DEBUG:boto:Data: 2024-04-16T20:07:12.692 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.692 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.692 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.692 DEBUG:boto:Params: {} 2024-04-16T20:07:12.692 DEBUG:boto:Token: None 2024-04-16T20:07:12.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19 2024-04-16T20:07:12.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0E9PkuPimiHGRSVL5RNEl6BegPM= 2024-04-16T20:07:12.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0E9PkuPimiHGRSVL5RNEl6BegPM=', 'Content-Length': '0'} 2024-04-16T20:07:12.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e16007154978c00-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.695 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:12.695 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:12.695 DEBUG:boto:Method: GET 2024-04-16T20:07:12.695 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:12.695 DEBUG:boto:Data: 2024-04-16T20:07:12.696 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.696 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.696 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.696 DEBUG:boto:Params: {} 2024-04-16T20:07:12.696 DEBUG:boto:Token: None 2024-04-16T20:07:12.696 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj19 2024-04-16T20:07:12.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0E9PkuPimiHGRSVL5RNEl6BegPM= 2024-04-16T20:07:12.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0E9PkuPimiHGRSVL5RNEl6BegPM=', 'Content-Length': '0'} 2024-04-16T20:07:12.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70a563db3227775-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.738 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.738 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.738 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.739 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.739 DEBUG:boto:Method: GET 2024-04-16T20:07:12.739 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.739 DEBUG:boto:Data: 2024-04-16T20:07:12.739 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.739 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.739 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.739 DEBUG:boto:Params: {} 2024-04-16T20:07:12.739 DEBUG:boto:Token: None 2024-04-16T20:07:12.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055df7aeb2e816941-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.742 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.742 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.742 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.743 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.743 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.743 DEBUG:boto:Method: GET 2024-04-16T20:07:12.743 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:12.743 DEBUG:boto:Data: 2024-04-16T20:07:12.743 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.743 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.743 DEBUG:boto:Params: {} 2024-04-16T20:07:12.743 DEBUG:boto:Token: None 2024-04-16T20:07:12.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1ce76c94c7b9f6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.747 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:12.747 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.747 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.748 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.748 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.748 DEBUG:boto:Method: GET 2024-04-16T20:07:12.748 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.748 DEBUG:boto:Data: 2024-04-16T20:07:12.748 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.748 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.748 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.748 DEBUG:boto:Params: {} 2024-04-16T20:07:12.748 DEBUG:boto:Token: None 2024-04-16T20:07:12.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY= 2024-04-16T20:07:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY=', 'Content-Length': '0'} 2024-04-16T20:07:12.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9539076b51720a0-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.753 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.753 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.753 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.753 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.753 DEBUG:boto:Method: GET 2024-04-16T20:07:12.754 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.754 DEBUG:boto:Data: 2024-04-16T20:07:12.754 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.754 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.754 DEBUG:boto:Params: {} 2024-04-16T20:07:12.754 DEBUG:boto:Token: None 2024-04-16T20:07:12.754 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.754 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY= 2024-04-16T20:07:12.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY=', 'Content-Length': '0'} 2024-04-16T20:07:12.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679689bb7a007532-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.757 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.757 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.757 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.757 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.757 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.757 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.757 DEBUG:boto:Data: 2024-04-16T20:07:12.757 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.757 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.757 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.757 DEBUG:boto:Params: {} 2024-04-16T20:07:12.757 DEBUG:boto:Token: None 2024-04-16T20:07:12.758 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.758 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:scmNtOOjHJAWdC2uiqWyhd67msg= 2024-04-16T20:07:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:scmNtOOjHJAWdC2uiqWyhd67msg=', 'Content-Length': '0'} 2024-04-16T20:07:12.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aff69b260eb3207-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.760 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.761 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.761 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.761 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.761 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.761 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.761 DEBUG:boto:Data: 2024-04-16T20:07:12.761 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.761 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.761 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.761 DEBUG:boto:Params: {} 2024-04-16T20:07:12.761 DEBUG:boto:Token: None 2024-04-16T20:07:12.762 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:scmNtOOjHJAWdC2uiqWyhd67msg= 2024-04-16T20:07:12.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:scmNtOOjHJAWdC2uiqWyhd67msg=', 'Content-Length': '0'} 2024-04-16T20:07:12.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007876683fe994a64e-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:12.764 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.764 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.764 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.764 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.764 DEBUG:boto:Method: GET 2024-04-16T20:07:12.764 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.764 DEBUG:boto:Data: 2024-04-16T20:07:12.765 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.765 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.765 DEBUG:boto:Params: {} 2024-04-16T20:07:12.765 DEBUG:boto:Token: None 2024-04-16T20:07:12.765 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY= 2024-04-16T20:07:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY=', 'Content-Length': '0'} 2024-04-16T20:07:12.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c11f987301e85c-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.810 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.810 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.811 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.811 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.811 DEBUG:boto:Method: GET 2024-04-16T20:07:12.811 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.811 DEBUG:boto:Data: 2024-04-16T20:07:12.811 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.811 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.811 DEBUG:boto:Params: {} 2024-04-16T20:07:12.811 DEBUG:boto:Token: None 2024-04-16T20:07:12.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:12.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY= 2024-04-16T20:07:12.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/roYOAkPatgukQd9JZGgIjPGbkY=', 'Content-Length': '0'} 2024-04-16T20:07:12.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094efba4639a05031-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.814 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.814 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.814 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.814 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.814 DEBUG:boto:Method: GET 2024-04-16T20:07:12.814 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.814 DEBUG:boto:Data: 2024-04-16T20:07:12.815 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.815 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.815 DEBUG:boto:Params: {} 2024-04-16T20:07:12.815 DEBUG:boto:Token: None 2024-04-16T20:07:12.815 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa51507ed7df8f3-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.818 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.818 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.818 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.818 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.818 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.818 DEBUG:boto:Method: GET 2024-04-16T20:07:12.819 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.819 DEBUG:boto:Data: 2024-04-16T20:07:12.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.819 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.819 DEBUG:boto:Params: {} 2024-04-16T20:07:12.819 DEBUG:boto:Token: None 2024-04-16T20:07:12.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be95c4ddbb0e66a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.823 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.823 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.824 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.824 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.824 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:12.824 DEBUG:boto:Data: 2024-04-16T20:07:12.824 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.824 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.824 DEBUG:boto:Params: {} 2024-04-16T20:07:12.824 DEBUG:boto:Token: None 2024-04-16T20:07:12.824 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22 2024-04-16T20:07:12.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jwte3yrfAYh4RujwHx4pgoV9AIo= 2024-04-16T20:07:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwte3yrfAYh4RujwHx4pgoV9AIo=', 'Content-Length': '0'} 2024-04-16T20:07:12.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f2ed9ea3d0ec164-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.827 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.827 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.827 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.827 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:12.827 DEBUG:boto:Data: 2024-04-16T20:07:12.827 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.827 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.827 DEBUG:boto:Params: {} 2024-04-16T20:07:12.827 DEBUG:boto:Token: None 2024-04-16T20:07:12.827 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22 2024-04-16T20:07:12.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jwte3yrfAYh4RujwHx4pgoV9AIo= 2024-04-16T20:07:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwte3yrfAYh4RujwHx4pgoV9AIo=', 'Content-Length': '0'} 2024-04-16T20:07:12.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008859619d26cbe5c8-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:12.830 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.830 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.830 DEBUG:boto:Method: GET 2024-04-16T20:07:12.830 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:12.830 DEBUG:boto:Data: 2024-04-16T20:07:12.830 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.830 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.830 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.830 DEBUG:boto:Params: {} 2024-04-16T20:07:12.831 DEBUG:boto:Token: None 2024-04-16T20:07:12.831 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22 2024-04-16T20:07:12.831 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFvxIEYhNK+j4NGCvNyMTg15QPQ= 2024-04-16T20:07:12.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFvxIEYhNK+j4NGCvNyMTg15QPQ=', 'Content-Length': '0'} 2024-04-16T20:07:12.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f4e12277fdbd7ad-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.833 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:12.833 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:12.833 DEBUG:boto:Method: GET 2024-04-16T20:07:12.833 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:12.833 DEBUG:boto:Data: 2024-04-16T20:07:12.833 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.833 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.833 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.834 DEBUG:boto:Params: {} 2024-04-16T20:07:12.834 DEBUG:boto:Token: None 2024-04-16T20:07:12.834 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj22 2024-04-16T20:07:12.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mFvxIEYhNK+j4NGCvNyMTg15QPQ= 2024-04-16T20:07:12.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mFvxIEYhNK+j4NGCvNyMTg15QPQ=', 'Content-Length': '0'} 2024-04-16T20:07:12.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e6825bf35a1f1a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.878 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.879 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.879 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.879 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.879 DEBUG:boto:Method: GET 2024-04-16T20:07:12.879 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.879 DEBUG:boto:Data: 2024-04-16T20:07:12.879 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.879 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.879 DEBUG:boto:Params: {} 2024-04-16T20:07:12.879 DEBUG:boto:Token: None 2024-04-16T20:07:12.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d0150e32eceb27-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.882 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.883 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.883 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.883 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.883 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.883 DEBUG:boto:Method: GET 2024-04-16T20:07:12.883 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:12.883 DEBUG:boto:Data: 2024-04-16T20:07:12.883 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.883 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.883 DEBUG:boto:Params: {} 2024-04-16T20:07:12.883 DEBUG:boto:Token: None 2024-04-16T20:07:12.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040794a78342c376a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.887 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:12.888 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.888 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.888 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.888 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.888 DEBUG:boto:Method: GET 2024-04-16T20:07:12.888 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.888 DEBUG:boto:Data: 2024-04-16T20:07:12.888 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.888 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.889 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.889 DEBUG:boto:Params: {} 2024-04-16T20:07:12.889 DEBUG:boto:Token: None 2024-04-16T20:07:12.889 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4= 2024-04-16T20:07:12.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ff4c02ddb1dd28-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.891 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.892 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.892 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.892 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.892 DEBUG:boto:Method: GET 2024-04-16T20:07:12.892 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.892 DEBUG:boto:Data: 2024-04-16T20:07:12.892 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.892 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.892 DEBUG:boto:Params: {} 2024-04-16T20:07:12.892 DEBUG:boto:Token: None 2024-04-16T20:07:12.893 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4= 2024-04-16T20:07:12.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db220508c06c80dd-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.895 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.895 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.895 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.895 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.896 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.896 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.896 DEBUG:boto:Data: 2024-04-16T20:07:12.896 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.896 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.896 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.896 DEBUG:boto:Params: {} 2024-04-16T20:07:12.896 DEBUG:boto:Token: None 2024-04-16T20:07:12.896 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l+2uaPQUyzfb4Epi0ZmpGCi2pqs= 2024-04-16T20:07:12.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l+2uaPQUyzfb4Epi0ZmpGCi2pqs=', 'Content-Length': '0'} 2024-04-16T20:07:12.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bada0e6f186e0795-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.899 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.899 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.899 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.899 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.899 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.899 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.899 DEBUG:boto:Data: 2024-04-16T20:07:12.899 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.900 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.900 DEBUG:boto:Params: {} 2024-04-16T20:07:12.900 DEBUG:boto:Token: None 2024-04-16T20:07:12.900 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l+2uaPQUyzfb4Epi0ZmpGCi2pqs= 2024-04-16T20:07:12.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l+2uaPQUyzfb4Epi0ZmpGCi2pqs=', 'Content-Length': '0'} 2024-04-16T20:07:12.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690f63cfbe87bc7c-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:12.903 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.903 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.903 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.903 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.903 DEBUG:boto:Method: GET 2024-04-16T20:07:12.903 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.903 DEBUG:boto:Data: 2024-04-16T20:07:12.903 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.903 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.903 DEBUG:boto:Params: {} 2024-04-16T20:07:12.904 DEBUG:boto:Token: None 2024-04-16T20:07:12.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4= 2024-04-16T20:07:12.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000955fff484d644938-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.946 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.946 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.947 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.947 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.947 DEBUG:boto:Method: GET 2024-04-16T20:07:12.947 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.947 DEBUG:boto:Data: 2024-04-16T20:07:12.947 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.947 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.947 DEBUG:boto:Params: {} 2024-04-16T20:07:12.947 DEBUG:boto:Token: None 2024-04-16T20:07:12.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:12.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4= 2024-04-16T20:07:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BXqxy3Sr6doWcm16EHY1REn1QO4=', 'Content-Length': '0'} 2024-04-16T20:07:12.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006555ef2e443d5d7b-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.950 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.950 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.950 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.950 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.950 DEBUG:boto:Method: GET 2024-04-16T20:07:12.951 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.951 DEBUG:boto:Data: 2024-04-16T20:07:12.951 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.951 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.951 DEBUG:boto:Params: {} 2024-04-16T20:07:12.951 DEBUG:boto:Token: None 2024-04-16T20:07:12.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a3bab7e9e275cf-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.954 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.954 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.955 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.955 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.955 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.955 DEBUG:boto:Method: GET 2024-04-16T20:07:12.955 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.955 DEBUG:boto:Data: 2024-04-16T20:07:12.955 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.955 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.955 DEBUG:boto:Params: {} 2024-04-16T20:07:12.955 DEBUG:boto:Token: None 2024-04-16T20:07:12.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0bd1c757c3d6f01-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.959 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.959 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.959 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.960 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.960 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:12.960 DEBUG:boto:Data: 2024-04-16T20:07:12.960 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.960 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.960 DEBUG:boto:Params: {} 2024-04-16T20:07:12.960 DEBUG:boto:Token: None 2024-04-16T20:07:12.960 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23 2024-04-16T20:07:12.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oVgOF4FZwIA5nSdMvE8fS2RAxHI= 2024-04-16T20:07:12.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oVgOF4FZwIA5nSdMvE8fS2RAxHI=', 'Content-Length': '0'} 2024-04-16T20:07:12.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f9d501e63844e1-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.963 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.963 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.963 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.963 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:12.963 DEBUG:boto:Data: 2024-04-16T20:07:12.963 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.963 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.963 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.963 DEBUG:boto:Params: {} 2024-04-16T20:07:12.964 DEBUG:boto:Token: None 2024-04-16T20:07:12.964 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23 2024-04-16T20:07:12.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oVgOF4FZwIA5nSdMvE8fS2RAxHI= 2024-04-16T20:07:12.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oVgOF4FZwIA5nSdMvE8fS2RAxHI=', 'Content-Length': '0'} 2024-04-16T20:07:12.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085157f42785824c6-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:12.966 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.966 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.967 DEBUG:boto:Method: GET 2024-04-16T20:07:12.967 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:12.967 DEBUG:boto:Data: 2024-04-16T20:07:12.967 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.967 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.967 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.967 DEBUG:boto:Params: {} 2024-04-16T20:07:12.967 DEBUG:boto:Token: None 2024-04-16T20:07:12.967 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23 2024-04-16T20:07:12.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XeIs3VCc+oBASElkX2zDucaOlLM= 2024-04-16T20:07:12.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XeIs3VCc+oBASElkX2zDucaOlLM=', 'Content-Length': '0'} 2024-04-16T20:07:12.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead26dbe8efff791-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.969 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:12.970 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:12.970 DEBUG:boto:Method: GET 2024-04-16T20:07:12.970 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:12.970 DEBUG:boto:Data: 2024-04-16T20:07:12.970 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.970 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.970 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.970 DEBUG:boto:Params: {} 2024-04-16T20:07:12.970 DEBUG:boto:Token: None 2024-04-16T20:07:12.970 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj23 2024-04-16T20:07:12.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XeIs3VCc+oBASElkX2zDucaOlLM= 2024-04-16T20:07:12.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XeIs3VCc+oBASElkX2zDucaOlLM=', 'Content-Length': '0'} 2024-04-16T20:07:12.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51243000f941b1a-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.973 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.973 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.973 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.973 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.973 DEBUG:boto:Method: GET 2024-04-16T20:07:12.973 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.974 DEBUG:boto:Data: 2024-04-16T20:07:12.974 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.974 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.974 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.974 DEBUG:boto:Params: {} 2024-04-16T20:07:12.974 DEBUG:boto:Token: None 2024-04-16T20:07:12.974 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.974 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ad7e7929e07fa6-00661eda70-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.977 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.977 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:12.977 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:12.977 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.977 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.977 DEBUG:boto:Method: GET 2024-04-16T20:07:12.978 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:12.978 DEBUG:boto:Data: 2024-04-16T20:07:12.978 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.978 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.978 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.978 DEBUG:boto:Params: {} 2024-04-16T20:07:12.978 DEBUG:boto:Token: None 2024-04-16T20:07:12.978 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/ 2024-04-16T20:07:12.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4= 2024-04-16T20:07:12.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eNEoxrAcnL4zPa5jY3lOt7/IDW4=', 'Content-Length': '0'} 2024-04-16T20:07:12.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5a51591dbfd559-00661eda70-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.982 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:12.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:12.982 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.982 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.982 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.982 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.983 DEBUG:boto:Method: GET 2024-04-16T20:07:12.983 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.983 DEBUG:boto:Data: 2024-04-16T20:07:12.983 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.983 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.983 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.983 DEBUG:boto:Params: {} 2024-04-16T20:07:12.983 DEBUG:boto:Token: None 2024-04-16T20:07:12.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs= 2024-04-16T20:07:12.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs=', 'Content-Length': '0'} 2024-04-16T20:07:12.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b497922ac39f07dc-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.986 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.986 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.986 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.986 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.986 DEBUG:boto:Method: GET 2024-04-16T20:07:12.986 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.986 DEBUG:boto:Data: 2024-04-16T20:07:12.986 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.986 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.986 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.986 DEBUG:boto:Params: {} 2024-04-16T20:07:12.987 DEBUG:boto:Token: None 2024-04-16T20:07:12.987 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs= 2024-04-16T20:07:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs=', 'Content-Length': '0'} 2024-04-16T20:07:12.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d98268349b30af-00661eda70-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.989 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.989 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.989 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.989 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.990 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.990 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.990 DEBUG:boto:Data: 2024-04-16T20:07:12.990 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.990 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.990 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.990 DEBUG:boto:Params: {} 2024-04-16T20:07:12.990 DEBUG:boto:Token: None 2024-04-16T20:07:12.990 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VCjPrvSLbtJTcRWBLm/bQnXf6Pg= 2024-04-16T20:07:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VCjPrvSLbtJTcRWBLm/bQnXf6Pg=', 'Content-Length': '0'} 2024-04-16T20:07:12.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3c61cf2d854c6c-00661eda70-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.992 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.993 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.993 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.993 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.993 DEBUG:boto:Method: HEAD 2024-04-16T20:07:12.993 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.993 DEBUG:boto:Data: 2024-04-16T20:07:12.993 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.993 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:12.993 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.993 DEBUG:boto:Params: {} 2024-04-16T20:07:12.993 DEBUG:boto:Token: None 2024-04-16T20:07:12.994 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.994 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VCjPrvSLbtJTcRWBLm/bQnXf6Pg= 2024-04-16T20:07:12.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VCjPrvSLbtJTcRWBLm/bQnXf6Pg=', 'Content-Length': '0'} 2024-04-16T20:07:12.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751299f55c2ae1cb-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:12.996 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.996 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.996 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.996 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.996 DEBUG:boto:Method: GET 2024-04-16T20:07:12.996 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.996 DEBUG:boto:Data: 2024-04-16T20:07:12.996 DEBUG:boto:Headers: {} 2024-04-16T20:07:12.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:12.996 DEBUG:boto:Port: 8000 2024-04-16T20:07:12.997 DEBUG:boto:Params: {} 2024-04-16T20:07:12.997 DEBUG:boto:Token: None 2024-04-16T20:07:12.997 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:12 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs= 2024-04-16T20:07:12.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vuFE7NswaGIqKsbfOKLYvBSsoJs=', 'Content-Length': '0'} 2024-04-16T20:07:12.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2eb7f063a3f9cc-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:12.999 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:12.999 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:12.999 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.999 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:12.999 DEBUG:boto:Method: GET 2024-04-16T20:07:13.000 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:13.000 DEBUG:boto:Data: 2024-04-16T20:07:13.000 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.000 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.000 DEBUG:boto:Params: {} 2024-04-16T20:07:13.000 DEBUG:boto:Token: None 2024-04-16T20:07:13.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:13.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L+EwrmsCZXp4WecZZVghxP5tpW0= 2024-04-16T20:07:13.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L+EwrmsCZXp4WecZZVghxP5tpW0=', 'Content-Length': '0'} 2024-04-16T20:07:13.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1233b14db9e5f9-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.003 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.003 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.003 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.003 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.003 DEBUG:boto:Method: GET 2024-04-16T20:07:13.003 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.003 DEBUG:boto:Data: 2024-04-16T20:07:13.003 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.003 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.003 DEBUG:boto:Params: {} 2024-04-16T20:07:13.004 DEBUG:boto:Token: None 2024-04-16T20:07:13.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004901f077189db8e1-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.050 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.051 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.051 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.051 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.051 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.051 DEBUG:boto:Method: GET 2024-04-16T20:07:13.051 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.051 DEBUG:boto:Data: 2024-04-16T20:07:13.051 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.051 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.052 DEBUG:boto:Params: {} 2024-04-16T20:07:13.052 DEBUG:boto:Token: None 2024-04-16T20:07:13.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075dbfa5a22b42d26-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.056 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.056 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:13.056 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:13.056 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.056 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:13.056 DEBUG:boto:Data: 2024-04-16T20:07:13.056 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.056 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.057 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.057 DEBUG:boto:Params: {} 2024-04-16T20:07:13.057 DEBUG:boto:Token: None 2024-04-16T20:07:13.057 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj29 2024-04-16T20:07:13.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lI3nZRUT1Tae2VKmPGQXl4rCvQI= 2024-04-16T20:07:13.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lI3nZRUT1Tae2VKmPGQXl4rCvQI=', 'Content-Length': '0'} 2024-04-16T20:07:13.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ce18496c00fddd-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.059 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:13.059 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:13.059 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.059 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:13.059 DEBUG:boto:Data: 2024-04-16T20:07:13.059 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.060 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.060 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.060 DEBUG:boto:Params: {} 2024-04-16T20:07:13.060 DEBUG:boto:Token: None 2024-04-16T20:07:13.060 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj29 2024-04-16T20:07:13.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lI3nZRUT1Tae2VKmPGQXl4rCvQI= 2024-04-16T20:07:13.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lI3nZRUT1Tae2VKmPGQXl4rCvQI=', 'Content-Length': '0'} 2024-04-16T20:07:13.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35b1f5cf96773a3-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:13.062 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:13.062 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:13.062 DEBUG:boto:Method: GET 2024-04-16T20:07:13.062 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:13.063 DEBUG:boto:Data: 2024-04-16T20:07:13.063 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.063 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.063 DEBUG:boto:Params: {} 2024-04-16T20:07:13.063 DEBUG:boto:Token: None 2024-04-16T20:07:13.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj29 2024-04-16T20:07:13.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QP+Z5sv+0B/4cmut+DYZOtZfbWY= 2024-04-16T20:07:13.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QP+Z5sv+0B/4cmut+DYZOtZfbWY=', 'Content-Length': '0'} 2024-04-16T20:07:13.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94269a4e945ba18-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.106 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:13.106 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:13.107 DEBUG:boto:Method: GET 2024-04-16T20:07:13.107 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:13.107 DEBUG:boto:Data: 2024-04-16T20:07:13.107 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.107 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.107 DEBUG:boto:Params: {} 2024-04-16T20:07:13.107 DEBUG:boto:Token: None 2024-04-16T20:07:13.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj29 2024-04-16T20:07:13.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QP+Z5sv+0B/4cmut+DYZOtZfbWY= 2024-04-16T20:07:13.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QP+Z5sv+0B/4cmut+DYZOtZfbWY=', 'Content-Length': '0'} 2024-04-16T20:07:13.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007446f46b6b976916-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.110 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.110 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.110 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.110 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.110 DEBUG:boto:Method: GET 2024-04-16T20:07:13.110 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.110 DEBUG:boto:Data: 2024-04-16T20:07:13.110 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.110 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.110 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.111 DEBUG:boto:Params: {} 2024-04-16T20:07:13.111 DEBUG:boto:Token: None 2024-04-16T20:07:13.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3eb57550fbb1705-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.154 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.155 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.155 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.155 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.155 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.155 DEBUG:boto:Method: GET 2024-04-16T20:07:13.155 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:13.155 DEBUG:boto:Data: 2024-04-16T20:07:13.155 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.155 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.155 DEBUG:boto:Params: {} 2024-04-16T20:07:13.156 DEBUG:boto:Token: None 2024-04-16T20:07:13.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bc30a9a54d08b79-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.160 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:13.160 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.160 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.160 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.160 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.161 DEBUG:boto:Method: GET 2024-04-16T20:07:13.161 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.161 DEBUG:boto:Data: 2024-04-16T20:07:13.161 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.161 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.161 DEBUG:boto:Params: {} 2024-04-16T20:07:13.161 DEBUG:boto:Token: None 2024-04-16T20:07:13.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs= 2024-04-16T20:07:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs=', 'Content-Length': '0'} 2024-04-16T20:07:13.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000445b83f396fb0b99-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.206 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.206 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.206 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.206 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.207 DEBUG:boto:Method: GET 2024-04-16T20:07:13.207 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.207 DEBUG:boto:Data: 2024-04-16T20:07:13.207 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.207 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.207 DEBUG:boto:Params: {} 2024-04-16T20:07:13.207 DEBUG:boto:Token: None 2024-04-16T20:07:13.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs= 2024-04-16T20:07:13.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs=', 'Content-Length': '0'} 2024-04-16T20:07:13.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29a3f45eefa9528-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.209 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.210 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.210 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.210 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.210 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.210 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.210 DEBUG:boto:Data: 2024-04-16T20:07:13.210 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.210 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.210 DEBUG:boto:Params: {} 2024-04-16T20:07:13.210 DEBUG:boto:Token: None 2024-04-16T20:07:13.211 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M7GbChF/Y9ACp/8QcxXOiXFI3kI= 2024-04-16T20:07:13.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M7GbChF/Y9ACp/8QcxXOiXFI3kI=', 'Content-Length': '0'} 2024-04-16T20:07:13.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ed799343dc5dda-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.212 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.213 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.213 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.213 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.213 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.213 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.213 DEBUG:boto:Data: 2024-04-16T20:07:13.213 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.213 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.213 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.213 DEBUG:boto:Params: {} 2024-04-16T20:07:13.213 DEBUG:boto:Token: None 2024-04-16T20:07:13.214 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.214 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M7GbChF/Y9ACp/8QcxXOiXFI3kI= 2024-04-16T20:07:13.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M7GbChF/Y9ACp/8QcxXOiXFI3kI=', 'Content-Length': '0'} 2024-04-16T20:07:13.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261c8caf203ee1f9-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:13.216 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.216 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.216 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.216 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.216 DEBUG:boto:Method: GET 2024-04-16T20:07:13.216 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.216 DEBUG:boto:Data: 2024-04-16T20:07:13.216 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.217 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.217 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.217 DEBUG:boto:Params: {} 2024-04-16T20:07:13.217 DEBUG:boto:Token: None 2024-04-16T20:07:13.217 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs= 2024-04-16T20:07:13.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs=', 'Content-Length': '0'} 2024-04-16T20:07:13.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003698442532e39ea3-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.266 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.266 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.267 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.267 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.267 DEBUG:boto:Method: GET 2024-04-16T20:07:13.267 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.267 DEBUG:boto:Data: 2024-04-16T20:07:13.267 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.267 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.267 DEBUG:boto:Params: {} 2024-04-16T20:07:13.267 DEBUG:boto:Token: None 2024-04-16T20:07:13.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:13.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs= 2024-04-16T20:07:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJi2H+RZ3p+O4X3KGlFEKj9FRAs=', 'Content-Length': '0'} 2024-04-16T20:07:13.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1bc03c05647fec4-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.270 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.270 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.270 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.270 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.270 DEBUG:boto:Method: GET 2024-04-16T20:07:13.270 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.270 DEBUG:boto:Data: 2024-04-16T20:07:13.270 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.270 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.270 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.270 DEBUG:boto:Params: {} 2024-04-16T20:07:13.271 DEBUG:boto:Token: None 2024-04-16T20:07:13.271 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23081cef47588dd-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.314 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.315 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.315 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.315 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.315 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.315 DEBUG:boto:Method: GET 2024-04-16T20:07:13.315 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.315 DEBUG:boto:Data: 2024-04-16T20:07:13.315 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.315 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.315 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.315 DEBUG:boto:Params: {} 2024-04-16T20:07:13.316 DEBUG:boto:Token: None 2024-04-16T20:07:13.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026dee633cc3b908d-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.320 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.320 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.320 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.320 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.320 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:13.320 DEBUG:boto:Data: 2024-04-16T20:07:13.320 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.320 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.320 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.321 DEBUG:boto:Params: {} 2024-04-16T20:07:13.321 DEBUG:boto:Token: None 2024-04-16T20:07:13.321 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31 2024-04-16T20:07:13.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLr4917ORIIzLIPzyfJ6iPldPZM= 2024-04-16T20:07:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLr4917ORIIzLIPzyfJ6iPldPZM=', 'Content-Length': '0'} 2024-04-16T20:07:13.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51b706724769dda-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.323 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.323 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.323 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.323 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:13.323 DEBUG:boto:Data: 2024-04-16T20:07:13.324 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.324 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.324 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.324 DEBUG:boto:Params: {} 2024-04-16T20:07:13.324 DEBUG:boto:Token: None 2024-04-16T20:07:13.324 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31 2024-04-16T20:07:13.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hLr4917ORIIzLIPzyfJ6iPldPZM= 2024-04-16T20:07:13.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hLr4917ORIIzLIPzyfJ6iPldPZM=', 'Content-Length': '0'} 2024-04-16T20:07:13.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdfb60f68d4c430e-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:13.327 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.327 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.327 DEBUG:boto:Method: GET 2024-04-16T20:07:13.327 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:13.327 DEBUG:boto:Data: 2024-04-16T20:07:13.327 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.327 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.327 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.327 DEBUG:boto:Params: {} 2024-04-16T20:07:13.327 DEBUG:boto:Token: None 2024-04-16T20:07:13.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31 2024-04-16T20:07:13.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Sg4p877PTcf3GZc6E2VmFgi7Pyo= 2024-04-16T20:07:13.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Sg4p877PTcf3GZc6E2VmFgi7Pyo=', 'Content-Length': '0'} 2024-04-16T20:07:13.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8ca8f4bb1e94c1-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.370 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:13.370 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:13.371 DEBUG:boto:Method: GET 2024-04-16T20:07:13.371 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:13.371 DEBUG:boto:Data: 2024-04-16T20:07:13.371 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.371 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.371 DEBUG:boto:Params: {} 2024-04-16T20:07:13.371 DEBUG:boto:Token: None 2024-04-16T20:07:13.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj31 2024-04-16T20:07:13.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Sg4p877PTcf3GZc6E2VmFgi7Pyo= 2024-04-16T20:07:13.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Sg4p877PTcf3GZc6E2VmFgi7Pyo=', 'Content-Length': '0'} 2024-04-16T20:07:13.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006698fc60716006e0-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.374 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.374 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.374 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.374 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.374 DEBUG:boto:Method: GET 2024-04-16T20:07:13.374 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.374 DEBUG:boto:Data: 2024-04-16T20:07:13.374 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.374 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.374 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.375 DEBUG:boto:Params: {} 2024-04-16T20:07:13.375 DEBUG:boto:Token: None 2024-04-16T20:07:13.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c76e0d7ef6de20f-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.418 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.419 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.419 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.419 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.419 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.419 DEBUG:boto:Method: GET 2024-04-16T20:07:13.419 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:13.419 DEBUG:boto:Data: 2024-04-16T20:07:13.419 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.419 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.419 DEBUG:boto:Params: {} 2024-04-16T20:07:13.419 DEBUG:boto:Token: None 2024-04-16T20:07:13.420 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000543e7e66d4c09bc8-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.423 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:13.424 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.424 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.424 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.424 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.424 DEBUG:boto:Method: GET 2024-04-16T20:07:13.424 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.424 DEBUG:boto:Data: 2024-04-16T20:07:13.424 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.424 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.424 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.424 DEBUG:boto:Params: {} 2024-04-16T20:07:13.425 DEBUG:boto:Token: None 2024-04-16T20:07:13.425 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.425 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo= 2024-04-16T20:07:13.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo=', 'Content-Length': '0'} 2024-04-16T20:07:13.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e4c6628da8295e-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.470 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.470 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.470 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.471 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.471 DEBUG:boto:Method: GET 2024-04-16T20:07:13.471 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.471 DEBUG:boto:Data: 2024-04-16T20:07:13.471 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.471 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.471 DEBUG:boto:Params: {} 2024-04-16T20:07:13.471 DEBUG:boto:Token: None 2024-04-16T20:07:13.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo= 2024-04-16T20:07:13.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo=', 'Content-Length': '0'} 2024-04-16T20:07:13.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1c1a346188bebf-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.474 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.474 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.474 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.474 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.474 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.474 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.474 DEBUG:boto:Data: 2024-04-16T20:07:13.474 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.474 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.474 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.475 DEBUG:boto:Params: {} 2024-04-16T20:07:13.475 DEBUG:boto:Token: None 2024-04-16T20:07:13.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0INbJJdpkDuEBlrLBSJ1n7UNGew= 2024-04-16T20:07:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0INbJJdpkDuEBlrLBSJ1n7UNGew=', 'Content-Length': '0'} 2024-04-16T20:07:13.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cea906b43b4ced3-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.477 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.478 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.478 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.478 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.478 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.478 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.478 DEBUG:boto:Data: 2024-04-16T20:07:13.478 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.478 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.478 DEBUG:boto:Params: {} 2024-04-16T20:07:13.478 DEBUG:boto:Token: None 2024-04-16T20:07:13.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0INbJJdpkDuEBlrLBSJ1n7UNGew= 2024-04-16T20:07:13.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0INbJJdpkDuEBlrLBSJ1n7UNGew=', 'Content-Length': '0'} 2024-04-16T20:07:13.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004818bba9ff2f8db0-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:13.481 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.482 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.482 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.482 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.482 DEBUG:boto:Method: GET 2024-04-16T20:07:13.482 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.482 DEBUG:boto:Data: 2024-04-16T20:07:13.482 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.482 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.482 DEBUG:boto:Params: {} 2024-04-16T20:07:13.482 DEBUG:boto:Token: None 2024-04-16T20:07:13.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo= 2024-04-16T20:07:13.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo=', 'Content-Length': '0'} 2024-04-16T20:07:13.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3edf44426cf3df-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.526 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.526 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.526 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.527 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.527 DEBUG:boto:Method: GET 2024-04-16T20:07:13.527 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.527 DEBUG:boto:Data: 2024-04-16T20:07:13.527 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.527 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.527 DEBUG:boto:Params: {} 2024-04-16T20:07:13.527 DEBUG:boto:Token: None 2024-04-16T20:07:13.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:13.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo= 2024-04-16T20:07:13.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QMMpQIkftbbxESF+Hvtb8SM8Ryo=', 'Content-Length': '0'} 2024-04-16T20:07:13.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b03f92b57a7a66f-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.530 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.530 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.530 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.530 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.530 DEBUG:boto:Method: GET 2024-04-16T20:07:13.530 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.531 DEBUG:boto:Data: 2024-04-16T20:07:13.531 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.531 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.531 DEBUG:boto:Params: {} 2024-04-16T20:07:13.531 DEBUG:boto:Token: None 2024-04-16T20:07:13.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097f989651f72cd9f-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.574 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.574 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.575 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.575 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.575 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.575 DEBUG:boto:Method: GET 2024-04-16T20:07:13.575 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.575 DEBUG:boto:Data: 2024-04-16T20:07:13.575 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.575 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.575 DEBUG:boto:Params: {} 2024-04-16T20:07:13.575 DEBUG:boto:Token: None 2024-04-16T20:07:13.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7ad96b02c75910-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.579 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.580 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.580 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.580 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.580 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:13.580 DEBUG:boto:Data: 2024-04-16T20:07:13.580 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.580 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.580 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.580 DEBUG:boto:Params: {} 2024-04-16T20:07:13.581 DEBUG:boto:Token: None 2024-04-16T20:07:13.581 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33 2024-04-16T20:07:13.581 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AjirJ4HfbkJros5X9FHQrwBpqBQ= 2024-04-16T20:07:13.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AjirJ4HfbkJros5X9FHQrwBpqBQ=', 'Content-Length': '0'} 2024-04-16T20:07:13.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752e89c42ff520be-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.583 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.583 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.583 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.584 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:13.584 DEBUG:boto:Data: 2024-04-16T20:07:13.584 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.584 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.584 DEBUG:boto:Params: {} 2024-04-16T20:07:13.584 DEBUG:boto:Token: None 2024-04-16T20:07:13.584 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33 2024-04-16T20:07:13.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AjirJ4HfbkJros5X9FHQrwBpqBQ= 2024-04-16T20:07:13.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AjirJ4HfbkJros5X9FHQrwBpqBQ=', 'Content-Length': '0'} 2024-04-16T20:07:13.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a6f4add3781914-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:13.587 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.587 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.587 DEBUG:boto:Method: GET 2024-04-16T20:07:13.587 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:13.587 DEBUG:boto:Data: 2024-04-16T20:07:13.587 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.587 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.587 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.587 DEBUG:boto:Params: {} 2024-04-16T20:07:13.588 DEBUG:boto:Token: None 2024-04-16T20:07:13.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33 2024-04-16T20:07:13.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TYyFj5o1U/NUyaMnKFpZgZ/f+Ag= 2024-04-16T20:07:13.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TYyFj5o1U/NUyaMnKFpZgZ/f+Ag=', 'Content-Length': '0'} 2024-04-16T20:07:13.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000317b5299e8eb6491-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.630 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:13.630 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:13.631 DEBUG:boto:Method: GET 2024-04-16T20:07:13.631 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:13.631 DEBUG:boto:Data: 2024-04-16T20:07:13.631 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.631 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.631 DEBUG:boto:Params: {} 2024-04-16T20:07:13.631 DEBUG:boto:Token: None 2024-04-16T20:07:13.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj33 2024-04-16T20:07:13.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TYyFj5o1U/NUyaMnKFpZgZ/f+Ag= 2024-04-16T20:07:13.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TYyFj5o1U/NUyaMnKFpZgZ/f+Ag=', 'Content-Length': '0'} 2024-04-16T20:07:13.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c0eb5358312e94-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.634 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.634 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.634 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.634 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.634 DEBUG:boto:Method: GET 2024-04-16T20:07:13.634 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.634 DEBUG:boto:Data: 2024-04-16T20:07:13.634 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.634 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.635 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.635 DEBUG:boto:Params: {} 2024-04-16T20:07:13.635 DEBUG:boto:Token: None 2024-04-16T20:07:13.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda4c978a4befe49-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.678 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.679 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.679 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.679 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.679 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.679 DEBUG:boto:Method: GET 2024-04-16T20:07:13.679 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:13.679 DEBUG:boto:Data: 2024-04-16T20:07:13.679 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.679 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.679 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.679 DEBUG:boto:Params: {} 2024-04-16T20:07:13.680 DEBUG:boto:Token: None 2024-04-16T20:07:13.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7b5ebe9128a552-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.683 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:13.684 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.684 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.684 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.684 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.684 DEBUG:boto:Method: GET 2024-04-16T20:07:13.684 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.684 DEBUG:boto:Data: 2024-04-16T20:07:13.684 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.684 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.684 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.685 DEBUG:boto:Params: {} 2024-04-16T20:07:13.685 DEBUG:boto:Token: None 2024-04-16T20:07:13.685 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48= 2024-04-16T20:07:13.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48=', 'Content-Length': '0'} 2024-04-16T20:07:13.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212aea810c68719b-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.730 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.730 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.730 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.731 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.731 DEBUG:boto:Method: GET 2024-04-16T20:07:13.731 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.731 DEBUG:boto:Data: 2024-04-16T20:07:13.731 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.731 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.731 DEBUG:boto:Params: {} 2024-04-16T20:07:13.731 DEBUG:boto:Token: None 2024-04-16T20:07:13.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.731 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48= 2024-04-16T20:07:13.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48=', 'Content-Length': '0'} 2024-04-16T20:07:13.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26c55f3034ffc0a-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.734 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.734 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.734 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.734 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.734 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.734 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.734 DEBUG:boto:Data: 2024-04-16T20:07:13.735 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.735 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.735 DEBUG:boto:Params: {} 2024-04-16T20:07:13.735 DEBUG:boto:Token: None 2024-04-16T20:07:13.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Pq4NDY0Vs00/Qic1rjCkb7UFHg= 2024-04-16T20:07:13.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Pq4NDY0Vs00/Qic1rjCkb7UFHg=', 'Content-Length': '0'} 2024-04-16T20:07:13.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58ab655b56a2263-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.737 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.737 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.737 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.737 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.737 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.737 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.738 DEBUG:boto:Data: 2024-04-16T20:07:13.738 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.738 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.738 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.738 DEBUG:boto:Params: {} 2024-04-16T20:07:13.738 DEBUG:boto:Token: None 2024-04-16T20:07:13.738 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.738 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9Pq4NDY0Vs00/Qic1rjCkb7UFHg= 2024-04-16T20:07:13.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9Pq4NDY0Vs00/Qic1rjCkb7UFHg=', 'Content-Length': '0'} 2024-04-16T20:07:13.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7fafc6dddc046e-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:13.740 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.740 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.741 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.741 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.741 DEBUG:boto:Method: GET 2024-04-16T20:07:13.741 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.741 DEBUG:boto:Data: 2024-04-16T20:07:13.741 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.741 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.741 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.741 DEBUG:boto:Params: {} 2024-04-16T20:07:13.741 DEBUG:boto:Token: None 2024-04-16T20:07:13.741 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48= 2024-04-16T20:07:13.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48=', 'Content-Length': '0'} 2024-04-16T20:07:13.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dd9eb8e64a4b91-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.786 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.786 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.786 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.787 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.787 DEBUG:boto:Method: GET 2024-04-16T20:07:13.787 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.787 DEBUG:boto:Data: 2024-04-16T20:07:13.787 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.787 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.787 DEBUG:boto:Params: {} 2024-04-16T20:07:13.787 DEBUG:boto:Token: None 2024-04-16T20:07:13.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:13.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48= 2024-04-16T20:07:13.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I9B9DAMC9vaUgdiMhKAsnjnyk48=', 'Content-Length': '0'} 2024-04-16T20:07:13.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc47ce04cb313cd3-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.790 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.790 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.790 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.790 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.790 DEBUG:boto:Method: GET 2024-04-16T20:07:13.790 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.790 DEBUG:boto:Data: 2024-04-16T20:07:13.790 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.791 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.791 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.791 DEBUG:boto:Params: {} 2024-04-16T20:07:13.791 DEBUG:boto:Token: None 2024-04-16T20:07:13.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a07d9a74aab80b-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.834 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.834 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.835 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.835 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.835 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.835 DEBUG:boto:Method: GET 2024-04-16T20:07:13.835 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.835 DEBUG:boto:Data: 2024-04-16T20:07:13.835 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.835 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.835 DEBUG:boto:Params: {} 2024-04-16T20:07:13.836 DEBUG:boto:Token: None 2024-04-16T20:07:13.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18823915b902dca-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.839 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.840 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.840 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.840 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.840 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:13.840 DEBUG:boto:Data: 2024-04-16T20:07:13.840 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.840 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.840 DEBUG:boto:Params: {} 2024-04-16T20:07:13.840 DEBUG:boto:Token: None 2024-04-16T20:07:13.841 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34 2024-04-16T20:07:13.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KG04gk2sxBzWMgs7u0WEsr4zRvs= 2024-04-16T20:07:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KG04gk2sxBzWMgs7u0WEsr4zRvs=', 'Content-Length': '0'} 2024-04-16T20:07:13.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229b4227f3bbfcb5-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.843 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.843 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.843 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.843 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:13.843 DEBUG:boto:Data: 2024-04-16T20:07:13.843 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.843 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.843 DEBUG:boto:Params: {} 2024-04-16T20:07:13.844 DEBUG:boto:Token: None 2024-04-16T20:07:13.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34 2024-04-16T20:07:13.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KG04gk2sxBzWMgs7u0WEsr4zRvs= 2024-04-16T20:07:13.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KG04gk2sxBzWMgs7u0WEsr4zRvs=', 'Content-Length': '0'} 2024-04-16T20:07:13.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015bf5efcb6ed4417-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:13.846 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.846 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.846 DEBUG:boto:Method: GET 2024-04-16T20:07:13.846 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:13.846 DEBUG:boto:Data: 2024-04-16T20:07:13.847 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.847 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.847 DEBUG:boto:Params: {} 2024-04-16T20:07:13.847 DEBUG:boto:Token: None 2024-04-16T20:07:13.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34 2024-04-16T20:07:13.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gfO6QQDLeSnJ6neMNEowl3WwSmo= 2024-04-16T20:07:13.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gfO6QQDLeSnJ6neMNEowl3WwSmo=', 'Content-Length': '0'} 2024-04-16T20:07:13.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b760c00b868712-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.890 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:13.890 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:13.891 DEBUG:boto:Method: GET 2024-04-16T20:07:13.891 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:13.891 DEBUG:boto:Data: 2024-04-16T20:07:13.891 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.891 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.891 DEBUG:boto:Params: {} 2024-04-16T20:07:13.891 DEBUG:boto:Token: None 2024-04-16T20:07:13.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj34 2024-04-16T20:07:13.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gfO6QQDLeSnJ6neMNEowl3WwSmo= 2024-04-16T20:07:13.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gfO6QQDLeSnJ6neMNEowl3WwSmo=', 'Content-Length': '0'} 2024-04-16T20:07:13.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e195c540d93ebfab-00661eda71-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.894 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.894 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.894 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.894 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.894 DEBUG:boto:Method: GET 2024-04-16T20:07:13.894 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.894 DEBUG:boto:Data: 2024-04-16T20:07:13.894 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.894 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.894 DEBUG:boto:Params: {} 2024-04-16T20:07:13.895 DEBUG:boto:Token: None 2024-04-16T20:07:13.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012cb02b7d113461d-00661eda71-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.938 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.938 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:13.939 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:13.939 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.939 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.939 DEBUG:boto:Method: GET 2024-04-16T20:07:13.939 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:13.939 DEBUG:boto:Data: 2024-04-16T20:07:13.939 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.939 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.939 DEBUG:boto:Params: {} 2024-04-16T20:07:13.939 DEBUG:boto:Token: None 2024-04-16T20:07:13.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/ 2024-04-16T20:07:13.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE= 2024-04-16T20:07:13.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vUoylzCr/3iMg3HtXVBxCXLg+HE=', 'Content-Length': '0'} 2024-04-16T20:07:13.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e75dc6abdddefd4-00661eda71-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.943 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:13.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:13.944 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:13.944 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:13.944 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.944 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.944 DEBUG:boto:Method: GET 2024-04-16T20:07:13.944 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.944 DEBUG:boto:Data: 2024-04-16T20:07:13.944 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.944 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.944 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.944 DEBUG:boto:Params: {} 2024-04-16T20:07:13.944 DEBUG:boto:Token: None 2024-04-16T20:07:13.945 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.945 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DnbJ0hI6HflKZqXSt5aNN7P/0FA= 2024-04-16T20:07:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DnbJ0hI6HflKZqXSt5aNN7P/0FA=', 'Content-Length': '0'} 2024-04-16T20:07:13.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000557c3b66484ffd69-00661eda71-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.990 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:13.990 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:13.990 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.990 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.991 DEBUG:boto:Method: GET 2024-04-16T20:07:13.991 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.991 DEBUG:boto:Data: 2024-04-16T20:07:13.991 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.991 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.991 DEBUG:boto:Params: {} 2024-04-16T20:07:13.991 DEBUG:boto:Token: None 2024-04-16T20:07:13.991 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DnbJ0hI6HflKZqXSt5aNN7P/0FA= 2024-04-16T20:07:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DnbJ0hI6HflKZqXSt5aNN7P/0FA=', 'Content-Length': '0'} 2024-04-16T20:07:13.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000390b59a5cc2899cb-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.994 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:13.994 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:13.994 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.994 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.994 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.994 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.994 DEBUG:boto:Data: 2024-04-16T20:07:13.994 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:13.995 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.995 DEBUG:boto:Params: {} 2024-04-16T20:07:13.995 DEBUG:boto:Token: None 2024-04-16T20:07:13.995 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ru4Qnwk6Vu3VoAoueBK7xr7qNnA= 2024-04-16T20:07:13.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ru4Qnwk6Vu3VoAoueBK7xr7qNnA=', 'Content-Length': '0'} 2024-04-16T20:07:13.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930c0394a75068cc-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:13.998 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:13.998 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:13.998 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.998 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.998 DEBUG:boto:Method: HEAD 2024-04-16T20:07:13.998 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.998 DEBUG:boto:Data: 2024-04-16T20:07:13.998 DEBUG:boto:Headers: {} 2024-04-16T20:07:13.998 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:13.998 DEBUG:boto:Port: 8000 2024-04-16T20:07:13.998 DEBUG:boto:Params: {} 2024-04-16T20:07:13.998 DEBUG:boto:Token: None 2024-04-16T20:07:13.999 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:13 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:13.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ru4Qnwk6Vu3VoAoueBK7xr7qNnA= 2024-04-16T20:07:13.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ru4Qnwk6Vu3VoAoueBK7xr7qNnA=', 'Content-Length': '0'} 2024-04-16T20:07:14.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3e938c199a095c-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:14.001 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.001 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.001 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.002 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.002 DEBUG:boto:Method: GET 2024-04-16T20:07:14.002 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.002 DEBUG:boto:Data: 2024-04-16T20:07:14.002 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.002 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.002 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.002 DEBUG:boto:Params: {} 2024-04-16T20:07:14.002 DEBUG:boto:Token: None 2024-04-16T20:07:14.002 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NPN8rZWEyiGo0mmzFDHUT0cYvyA= 2024-04-16T20:07:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NPN8rZWEyiGo0mmzFDHUT0cYvyA=', 'Content-Length': '0'} 2024-04-16T20:07:14.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd26253bb2184d25-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.046 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.046 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.047 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.047 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.047 DEBUG:boto:Method: GET 2024-04-16T20:07:14.047 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.047 DEBUG:boto:Data: 2024-04-16T20:07:14.047 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.047 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.047 DEBUG:boto:Params: {} 2024-04-16T20:07:14.047 DEBUG:boto:Token: None 2024-04-16T20:07:14.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:14.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NPN8rZWEyiGo0mmzFDHUT0cYvyA= 2024-04-16T20:07:14.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NPN8rZWEyiGo0mmzFDHUT0cYvyA=', 'Content-Length': '0'} 2024-04-16T20:07:14.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325b16bfaaf948cd-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.050 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.050 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.050 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.050 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.050 DEBUG:boto:Method: GET 2024-04-16T20:07:14.050 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.050 DEBUG:boto:Data: 2024-04-16T20:07:14.051 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.051 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.051 DEBUG:boto:Params: {} 2024-04-16T20:07:14.051 DEBUG:boto:Token: None 2024-04-16T20:07:14.051 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002199e35475d0580a-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.094 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.094 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.095 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.095 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.095 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.095 DEBUG:boto:Method: GET 2024-04-16T20:07:14.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.095 DEBUG:boto:Data: 2024-04-16T20:07:14.095 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.095 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.095 DEBUG:boto:Params: {} 2024-04-16T20:07:14.095 DEBUG:boto:Token: None 2024-04-16T20:07:14.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000570b7d798a88d180-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.099 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.099 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.099 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.099 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.100 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:14.100 DEBUG:boto:Data: 2024-04-16T20:07:14.100 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.100 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.100 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.100 DEBUG:boto:Params: {} 2024-04-16T20:07:14.100 DEBUG:boto:Token: None 2024-04-16T20:07:14.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37 2024-04-16T20:07:14.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:94yLa6riNEr9nss8FPayDt6NLJI= 2024-04-16T20:07:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:94yLa6riNEr9nss8FPayDt6NLJI=', 'Content-Length': '0'} 2024-04-16T20:07:14.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001410a804d2cb76ad-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.103 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.103 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.103 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.103 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:14.103 DEBUG:boto:Data: 2024-04-16T20:07:14.103 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.103 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.103 DEBUG:boto:Params: {} 2024-04-16T20:07:14.103 DEBUG:boto:Token: None 2024-04-16T20:07:14.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37 2024-04-16T20:07:14.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:94yLa6riNEr9nss8FPayDt6NLJI= 2024-04-16T20:07:14.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:94yLa6riNEr9nss8FPayDt6NLJI=', 'Content-Length': '0'} 2024-04-16T20:07:14.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b02f4ee1242851-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:14.106 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.106 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.106 DEBUG:boto:Method: GET 2024-04-16T20:07:14.106 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:14.106 DEBUG:boto:Data: 2024-04-16T20:07:14.106 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.106 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.106 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.107 DEBUG:boto:Params: {} 2024-04-16T20:07:14.107 DEBUG:boto:Token: None 2024-04-16T20:07:14.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37 2024-04-16T20:07:14.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UAjyOKoctXstoCp3h0s8sgq/UsY= 2024-04-16T20:07:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UAjyOKoctXstoCp3h0s8sgq/UsY=', 'Content-Length': '0'} 2024-04-16T20:07:14.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e684d6aa9c68ff5-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.150 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:14.150 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:14.150 DEBUG:boto:Method: GET 2024-04-16T20:07:14.151 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:14.151 DEBUG:boto:Data: 2024-04-16T20:07:14.151 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.151 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.151 DEBUG:boto:Params: {} 2024-04-16T20:07:14.151 DEBUG:boto:Token: None 2024-04-16T20:07:14.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj37 2024-04-16T20:07:14.151 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UAjyOKoctXstoCp3h0s8sgq/UsY= 2024-04-16T20:07:14.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UAjyOKoctXstoCp3h0s8sgq/UsY=', 'Content-Length': '0'} 2024-04-16T20:07:14.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3223569493c3886-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.153 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.154 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.154 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.154 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.154 DEBUG:boto:Method: GET 2024-04-16T20:07:14.154 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.154 DEBUG:boto:Data: 2024-04-16T20:07:14.154 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.154 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.154 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.154 DEBUG:boto:Params: {} 2024-04-16T20:07:14.155 DEBUG:boto:Token: None 2024-04-16T20:07:14.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c01d3131789fa0b5-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.198 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.199 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.199 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.199 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.199 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.199 DEBUG:boto:Method: GET 2024-04-16T20:07:14.199 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:14.199 DEBUG:boto:Data: 2024-04-16T20:07:14.199 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.199 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.199 DEBUG:boto:Params: {} 2024-04-16T20:07:14.199 DEBUG:boto:Token: None 2024-04-16T20:07:14.200 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611da69d3ed62c3d-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.203 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:14.204 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.204 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.204 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.204 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.204 DEBUG:boto:Method: GET 2024-04-16T20:07:14.204 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.204 DEBUG:boto:Data: 2024-04-16T20:07:14.204 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.204 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.204 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.204 DEBUG:boto:Params: {} 2024-04-16T20:07:14.205 DEBUG:boto:Token: None 2024-04-16T20:07:14.205 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.205 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0= 2024-04-16T20:07:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0=', 'Content-Length': '0'} 2024-04-16T20:07:14.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0637d152602cbb-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.250 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.250 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.251 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.251 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.251 DEBUG:boto:Method: GET 2024-04-16T20:07:14.251 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.251 DEBUG:boto:Data: 2024-04-16T20:07:14.251 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.251 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.251 DEBUG:boto:Params: {} 2024-04-16T20:07:14.251 DEBUG:boto:Token: None 2024-04-16T20:07:14.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0= 2024-04-16T20:07:14.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0=', 'Content-Length': '0'} 2024-04-16T20:07:14.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000403c47c86a29604b-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.254 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.254 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.254 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.254 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.254 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.254 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.254 DEBUG:boto:Data: 2024-04-16T20:07:14.254 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.255 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.255 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.255 DEBUG:boto:Params: {} 2024-04-16T20:07:14.255 DEBUG:boto:Token: None 2024-04-16T20:07:14.255 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ACCyS1NNvo958uCSy+jUVBt3nRs= 2024-04-16T20:07:14.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ACCyS1NNvo958uCSy+jUVBt3nRs=', 'Content-Length': '0'} 2024-04-16T20:07:14.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c036c7be180f4c92-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.257 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.257 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.257 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.257 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.257 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.258 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.258 DEBUG:boto:Data: 2024-04-16T20:07:14.258 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.258 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.258 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.258 DEBUG:boto:Params: {} 2024-04-16T20:07:14.258 DEBUG:boto:Token: None 2024-04-16T20:07:14.258 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.258 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ACCyS1NNvo958uCSy+jUVBt3nRs= 2024-04-16T20:07:14.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ACCyS1NNvo958uCSy+jUVBt3nRs=', 'Content-Length': '0'} 2024-04-16T20:07:14.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010d22838e7551e61-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:14.260 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.261 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.261 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.261 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.261 DEBUG:boto:Method: GET 2024-04-16T20:07:14.261 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.261 DEBUG:boto:Data: 2024-04-16T20:07:14.261 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.261 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.261 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.261 DEBUG:boto:Params: {} 2024-04-16T20:07:14.261 DEBUG:boto:Token: None 2024-04-16T20:07:14.262 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.262 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0= 2024-04-16T20:07:14.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0=', 'Content-Length': '0'} 2024-04-16T20:07:14.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b7d22d35b16b40-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.306 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.306 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.307 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.307 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.307 DEBUG:boto:Method: GET 2024-04-16T20:07:14.307 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.307 DEBUG:boto:Data: 2024-04-16T20:07:14.307 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.307 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.307 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.307 DEBUG:boto:Params: {} 2024-04-16T20:07:14.307 DEBUG:boto:Token: None 2024-04-16T20:07:14.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:14.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0= 2024-04-16T20:07:14.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yxEwQ3aCiAqQyzafQSDW+7AIPU0=', 'Content-Length': '0'} 2024-04-16T20:07:14.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001735f1925036254e-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.310 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.310 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.310 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.310 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.310 DEBUG:boto:Method: GET 2024-04-16T20:07:14.310 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.310 DEBUG:boto:Data: 2024-04-16T20:07:14.310 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.310 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.311 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.311 DEBUG:boto:Params: {} 2024-04-16T20:07:14.311 DEBUG:boto:Token: None 2024-04-16T20:07:14.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231f75e7c06e32bd-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.358 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.359 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.359 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.359 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.359 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.359 DEBUG:boto:Method: GET 2024-04-16T20:07:14.359 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.359 DEBUG:boto:Data: 2024-04-16T20:07:14.359 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.359 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.359 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.359 DEBUG:boto:Params: {} 2024-04-16T20:07:14.360 DEBUG:boto:Token: None 2024-04-16T20:07:14.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008801cb57bb94e4d7-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.363 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.364 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.364 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.364 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.364 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:14.364 DEBUG:boto:Data: 2024-04-16T20:07:14.364 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.364 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.364 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.364 DEBUG:boto:Params: {} 2024-04-16T20:07:14.365 DEBUG:boto:Token: None 2024-04-16T20:07:14.365 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38 2024-04-16T20:07:14.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W1VbeyhBdBxJc+yA/6EGvnmvkQo= 2024-04-16T20:07:14.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W1VbeyhBdBxJc+yA/6EGvnmvkQo=', 'Content-Length': '0'} 2024-04-16T20:07:14.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870d6ece69980943-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.367 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.367 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.367 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.367 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:14.367 DEBUG:boto:Data: 2024-04-16T20:07:14.367 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.367 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.368 DEBUG:boto:Params: {} 2024-04-16T20:07:14.368 DEBUG:boto:Token: None 2024-04-16T20:07:14.368 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38 2024-04-16T20:07:14.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:W1VbeyhBdBxJc+yA/6EGvnmvkQo= 2024-04-16T20:07:14.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W1VbeyhBdBxJc+yA/6EGvnmvkQo=', 'Content-Length': '0'} 2024-04-16T20:07:14.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b7c16d371bbdd7-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:14.370 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.370 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.370 DEBUG:boto:Method: GET 2024-04-16T20:07:14.370 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:14.371 DEBUG:boto:Data: 2024-04-16T20:07:14.371 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.371 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.371 DEBUG:boto:Params: {} 2024-04-16T20:07:14.371 DEBUG:boto:Token: None 2024-04-16T20:07:14.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38 2024-04-16T20:07:14.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQjEz/u1j8YiA40E5Cq66+vvJEc= 2024-04-16T20:07:14.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQjEz/u1j8YiA40E5Cq66+vvJEc=', 'Content-Length': '0'} 2024-04-16T20:07:14.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eee9da363a467cce-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.414 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:14.415 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:14.415 DEBUG:boto:Method: GET 2024-04-16T20:07:14.415 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:14.415 DEBUG:boto:Data: 2024-04-16T20:07:14.415 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.415 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.415 DEBUG:boto:Params: {} 2024-04-16T20:07:14.415 DEBUG:boto:Token: None 2024-04-16T20:07:14.415 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj38 2024-04-16T20:07:14.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vQjEz/u1j8YiA40E5Cq66+vvJEc= 2024-04-16T20:07:14.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vQjEz/u1j8YiA40E5Cq66+vvJEc=', 'Content-Length': '0'} 2024-04-16T20:07:14.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb6fe273af721d4-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.418 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.418 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.418 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.418 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.418 DEBUG:boto:Method: GET 2024-04-16T20:07:14.418 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.418 DEBUG:boto:Data: 2024-04-16T20:07:14.418 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.418 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.419 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.419 DEBUG:boto:Params: {} 2024-04-16T20:07:14.419 DEBUG:boto:Token: None 2024-04-16T20:07:14.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000394907dd6e235c95-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.462 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.463 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.463 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.463 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.463 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.463 DEBUG:boto:Method: GET 2024-04-16T20:07:14.463 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:14.463 DEBUG:boto:Data: 2024-04-16T20:07:14.463 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.463 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.463 DEBUG:boto:Params: {} 2024-04-16T20:07:14.464 DEBUG:boto:Token: None 2024-04-16T20:07:14.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a276ba25b92d2e-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.467 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:14.468 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.468 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.468 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.468 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.468 DEBUG:boto:Method: GET 2024-04-16T20:07:14.468 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.468 DEBUG:boto:Data: 2024-04-16T20:07:14.468 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.468 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.468 DEBUG:boto:Params: {} 2024-04-16T20:07:14.469 DEBUG:boto:Token: None 2024-04-16T20:07:14.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw= 2024-04-16T20:07:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw=', 'Content-Length': '0'} 2024-04-16T20:07:14.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7deeac7d89751ab-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.514 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.514 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.515 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.515 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.515 DEBUG:boto:Method: GET 2024-04-16T20:07:14.515 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.515 DEBUG:boto:Data: 2024-04-16T20:07:14.515 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.515 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.515 DEBUG:boto:Params: {} 2024-04-16T20:07:14.515 DEBUG:boto:Token: None 2024-04-16T20:07:14.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw= 2024-04-16T20:07:14.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw=', 'Content-Length': '0'} 2024-04-16T20:07:14.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079cf2b2038f5de15-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.518 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.518 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.518 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.518 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.518 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.518 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.518 DEBUG:boto:Data: 2024-04-16T20:07:14.518 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.518 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.518 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.519 DEBUG:boto:Params: {} 2024-04-16T20:07:14.519 DEBUG:boto:Token: None 2024-04-16T20:07:14.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:18+kTEETXJtwmLHxspHY/bmCW30= 2024-04-16T20:07:14.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:18+kTEETXJtwmLHxspHY/bmCW30=', 'Content-Length': '0'} 2024-04-16T20:07:14.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e7aa3da9187056-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.521 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.521 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.521 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.521 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.521 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.522 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.522 DEBUG:boto:Data: 2024-04-16T20:07:14.522 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.522 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.522 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.522 DEBUG:boto:Params: {} 2024-04-16T20:07:14.522 DEBUG:boto:Token: None 2024-04-16T20:07:14.522 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.522 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:18+kTEETXJtwmLHxspHY/bmCW30= 2024-04-16T20:07:14.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:18+kTEETXJtwmLHxspHY/bmCW30=', 'Content-Length': '0'} 2024-04-16T20:07:14.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaae4d5a87fb6e84-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:14.525 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.525 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.525 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.525 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.525 DEBUG:boto:Method: GET 2024-04-16T20:07:14.525 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.525 DEBUG:boto:Data: 2024-04-16T20:07:14.525 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.525 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.526 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.526 DEBUG:boto:Params: {} 2024-04-16T20:07:14.526 DEBUG:boto:Token: None 2024-04-16T20:07:14.526 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw= 2024-04-16T20:07:14.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw=', 'Content-Length': '0'} 2024-04-16T20:07:14.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9786c63dc49e00d-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.574 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.574 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.574 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.575 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.575 DEBUG:boto:Method: GET 2024-04-16T20:07:14.575 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.575 DEBUG:boto:Data: 2024-04-16T20:07:14.575 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.575 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.575 DEBUG:boto:Params: {} 2024-04-16T20:07:14.575 DEBUG:boto:Token: None 2024-04-16T20:07:14.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:14.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw= 2024-04-16T20:07:14.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ShfgSwi11UKPvlkuoIOTriKaUw=', 'Content-Length': '0'} 2024-04-16T20:07:14.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9fbaaf88b4067e2-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.578 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.578 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.578 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.578 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.578 DEBUG:boto:Method: GET 2024-04-16T20:07:14.578 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.578 DEBUG:boto:Data: 2024-04-16T20:07:14.578 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.578 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.578 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.579 DEBUG:boto:Params: {} 2024-04-16T20:07:14.579 DEBUG:boto:Token: None 2024-04-16T20:07:14.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000578a1614f73904c-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.626 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.627 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.627 DEBUG:boto:Method: GET 2024-04-16T20:07:14.627 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.627 DEBUG:boto:Data: 2024-04-16T20:07:14.627 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.628 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.628 DEBUG:boto:Params: {} 2024-04-16T20:07:14.628 DEBUG:boto:Token: None 2024-04-16T20:07:14.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006305b3488a682ac-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.632 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.632 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.632 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.632 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.632 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:14.632 DEBUG:boto:Data: 2024-04-16T20:07:14.632 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.632 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.632 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.633 DEBUG:boto:Params: {} 2024-04-16T20:07:14.633 DEBUG:boto:Token: None 2024-04-16T20:07:14.633 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44 2024-04-16T20:07:14.633 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XcB0zOJR+Eu9qHUMuhsboZVS1D8= 2024-04-16T20:07:14.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XcB0zOJR+Eu9qHUMuhsboZVS1D8=', 'Content-Length': '0'} 2024-04-16T20:07:14.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb4d5d789b6ddc7-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.635 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.636 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.636 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.636 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:14.636 DEBUG:boto:Data: 2024-04-16T20:07:14.636 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.636 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.636 DEBUG:boto:Params: {} 2024-04-16T20:07:14.636 DEBUG:boto:Token: None 2024-04-16T20:07:14.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44 2024-04-16T20:07:14.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XcB0zOJR+Eu9qHUMuhsboZVS1D8= 2024-04-16T20:07:14.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XcB0zOJR+Eu9qHUMuhsboZVS1D8=', 'Content-Length': '0'} 2024-04-16T20:07:14.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d03933537647b8-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:14.639 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.639 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.639 DEBUG:boto:Method: GET 2024-04-16T20:07:14.639 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:14.639 DEBUG:boto:Data: 2024-04-16T20:07:14.639 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.639 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.640 DEBUG:boto:Params: {} 2024-04-16T20:07:14.640 DEBUG:boto:Token: None 2024-04-16T20:07:14.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44 2024-04-16T20:07:14.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNYjmxmuBxejd5Fa337tJkoGG7k= 2024-04-16T20:07:14.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNYjmxmuBxejd5Fa337tJkoGG7k=', 'Content-Length': '0'} 2024-04-16T20:07:14.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ddfbc78803a263-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.682 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:14.682 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:14.683 DEBUG:boto:Method: GET 2024-04-16T20:07:14.683 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:14.683 DEBUG:boto:Data: 2024-04-16T20:07:14.683 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.683 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.683 DEBUG:boto:Params: {} 2024-04-16T20:07:14.683 DEBUG:boto:Token: None 2024-04-16T20:07:14.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj44 2024-04-16T20:07:14.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iNYjmxmuBxejd5Fa337tJkoGG7k= 2024-04-16T20:07:14.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iNYjmxmuBxejd5Fa337tJkoGG7k=', 'Content-Length': '0'} 2024-04-16T20:07:14.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab988604a6b3ade-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.686 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.686 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.686 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.686 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.686 DEBUG:boto:Method: GET 2024-04-16T20:07:14.686 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.686 DEBUG:boto:Data: 2024-04-16T20:07:14.687 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.687 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.687 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.687 DEBUG:boto:Params: {} 2024-04-16T20:07:14.687 DEBUG:boto:Token: None 2024-04-16T20:07:14.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.687 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000390afffc1e2cf24c-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.730 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.730 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.731 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.731 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.731 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.731 DEBUG:boto:Method: GET 2024-04-16T20:07:14.731 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:14.731 DEBUG:boto:Data: 2024-04-16T20:07:14.731 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.731 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.731 DEBUG:boto:Params: {} 2024-04-16T20:07:14.731 DEBUG:boto:Token: None 2024-04-16T20:07:14.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ee091ddb7b60fa-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.735 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:14.736 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.736 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.736 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.736 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.736 DEBUG:boto:Method: GET 2024-04-16T20:07:14.736 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.736 DEBUG:boto:Data: 2024-04-16T20:07:14.736 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.736 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.737 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.737 DEBUG:boto:Params: {} 2024-04-16T20:07:14.737 DEBUG:boto:Token: None 2024-04-16T20:07:14.737 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.737 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc= 2024-04-16T20:07:14.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc=', 'Content-Length': '0'} 2024-04-16T20:07:14.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f0c179329c89a9-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.782 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.782 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.782 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.783 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.783 DEBUG:boto:Method: GET 2024-04-16T20:07:14.783 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.783 DEBUG:boto:Data: 2024-04-16T20:07:14.783 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.783 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.783 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.783 DEBUG:boto:Params: {} 2024-04-16T20:07:14.783 DEBUG:boto:Token: None 2024-04-16T20:07:14.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc= 2024-04-16T20:07:14.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc=', 'Content-Length': '0'} 2024-04-16T20:07:14.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000835c0f4d9473672f-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.786 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.786 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.786 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.786 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.787 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.787 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.787 DEBUG:boto:Data: 2024-04-16T20:07:14.787 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.787 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.787 DEBUG:boto:Params: {} 2024-04-16T20:07:14.787 DEBUG:boto:Token: None 2024-04-16T20:07:14.787 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGtmZNNmSERLDY+bopUf6Kbo614= 2024-04-16T20:07:14.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGtmZNNmSERLDY+bopUf6Kbo614=', 'Content-Length': '0'} 2024-04-16T20:07:14.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af60fd6c53ba87f2-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.789 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.790 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.790 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.790 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.790 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.790 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.790 DEBUG:boto:Data: 2024-04-16T20:07:14.790 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.790 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.790 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.790 DEBUG:boto:Params: {} 2024-04-16T20:07:14.790 DEBUG:boto:Token: None 2024-04-16T20:07:14.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGtmZNNmSERLDY+bopUf6Kbo614= 2024-04-16T20:07:14.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGtmZNNmSERLDY+bopUf6Kbo614=', 'Content-Length': '0'} 2024-04-16T20:07:14.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9a235dcea9aff6-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:14.793 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.793 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.793 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.794 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.794 DEBUG:boto:Method: GET 2024-04-16T20:07:14.794 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.794 DEBUG:boto:Data: 2024-04-16T20:07:14.794 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.794 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.794 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.794 DEBUG:boto:Params: {} 2024-04-16T20:07:14.794 DEBUG:boto:Token: None 2024-04-16T20:07:14.794 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc= 2024-04-16T20:07:14.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc=', 'Content-Length': '0'} 2024-04-16T20:07:14.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000878eb354b5066a83-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.838 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.838 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.838 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.839 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.839 DEBUG:boto:Method: GET 2024-04-16T20:07:14.839 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.839 DEBUG:boto:Data: 2024-04-16T20:07:14.839 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.839 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.839 DEBUG:boto:Params: {} 2024-04-16T20:07:14.839 DEBUG:boto:Token: None 2024-04-16T20:07:14.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:14.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc= 2024-04-16T20:07:14.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GNmxzLGaYmig6BlV59cRwD2lCNc=', 'Content-Length': '0'} 2024-04-16T20:07:14.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20f82654838139d-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.842 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.842 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.842 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.842 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.842 DEBUG:boto:Method: GET 2024-04-16T20:07:14.842 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.842 DEBUG:boto:Data: 2024-04-16T20:07:14.843 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.843 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.843 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.843 DEBUG:boto:Params: {} 2024-04-16T20:07:14.843 DEBUG:boto:Token: None 2024-04-16T20:07:14.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006561e180e4b1a0e0-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.886 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.886 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.887 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.887 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.887 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.887 DEBUG:boto:Method: GET 2024-04-16T20:07:14.887 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.887 DEBUG:boto:Data: 2024-04-16T20:07:14.887 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.887 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.887 DEBUG:boto:Params: {} 2024-04-16T20:07:14.887 DEBUG:boto:Token: None 2024-04-16T20:07:14.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a8aa802000ee9a2-00661eda72-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.891 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.892 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.892 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.892 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.892 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:14.892 DEBUG:boto:Data: 2024-04-16T20:07:14.892 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.892 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.892 DEBUG:boto:Params: {} 2024-04-16T20:07:14.892 DEBUG:boto:Token: None 2024-04-16T20:07:14.893 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46 2024-04-16T20:07:14.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hpwwx2opQUjzzRpOJ4y0aH1NbHM= 2024-04-16T20:07:14.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hpwwx2opQUjzzRpOJ4y0aH1NbHM=', 'Content-Length': '0'} 2024-04-16T20:07:14.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07d1bef6979187c-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.895 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.895 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.895 DEBUG:boto:Method: HEAD 2024-04-16T20:07:14.895 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:14.895 DEBUG:boto:Data: 2024-04-16T20:07:14.895 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.895 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.895 DEBUG:boto:Params: {} 2024-04-16T20:07:14.895 DEBUG:boto:Token: None 2024-04-16T20:07:14.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46 2024-04-16T20:07:14.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hpwwx2opQUjzzRpOJ4y0aH1NbHM= 2024-04-16T20:07:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hpwwx2opQUjzzRpOJ4y0aH1NbHM=', 'Content-Length': '0'} 2024-04-16T20:07:14.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d03716b860b27c2-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:14.898 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.898 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.898 DEBUG:boto:Method: GET 2024-04-16T20:07:14.898 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:14.898 DEBUG:boto:Data: 2024-04-16T20:07:14.898 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.899 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.899 DEBUG:boto:Params: {} 2024-04-16T20:07:14.899 DEBUG:boto:Token: None 2024-04-16T20:07:14.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46 2024-04-16T20:07:14.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81kgz953R+tQk00yB2SKO8G8AAw= 2024-04-16T20:07:14.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81kgz953R+tQk00yB2SKO8G8AAw=', 'Content-Length': '0'} 2024-04-16T20:07:14.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:02:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849e39c74d4fdc18-00661eda72-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.942 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:14.942 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:14.943 DEBUG:boto:Method: GET 2024-04-16T20:07:14.943 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:14.943 DEBUG:boto:Data: 2024-04-16T20:07:14.943 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.943 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.943 DEBUG:boto:Params: {} 2024-04-16T20:07:14.943 DEBUG:boto:Token: None 2024-04-16T20:07:14.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj46 2024-04-16T20:07:14.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81kgz953R+tQk00yB2SKO8G8AAw= 2024-04-16T20:07:14.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81kgz953R+tQk00yB2SKO8G8AAw=', 'Content-Length': '0'} 2024-04-16T20:07:14.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621bd6561abba3bc-00661eda72-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.946 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.946 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.946 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.946 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.946 DEBUG:boto:Method: GET 2024-04-16T20:07:14.946 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.947 DEBUG:boto:Data: 2024-04-16T20:07:14.947 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.947 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.947 DEBUG:boto:Params: {} 2024-04-16T20:07:14.947 DEBUG:boto:Token: None 2024-04-16T20:07:14.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096f9595f3cd21a82-00661eda72-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.990 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.990 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:14.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:14.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.991 DEBUG:boto:Method: GET 2024-04-16T20:07:14.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:14.991 DEBUG:boto:Data: 2024-04-16T20:07:14.991 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:14.991 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.991 DEBUG:boto:Params: {} 2024-04-16T20:07:14.991 DEBUG:boto:Token: None 2024-04-16T20:07:14.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/ 2024-04-16T20:07:14.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg= 2024-04-16T20:07:14.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv8Q4Oa6YSNj/OwuRHlsBVqFvtg=', 'Content-Length': '0'} 2024-04-16T20:07:14.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4684235800850cc-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:14.995 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:14.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:14.996 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:14.996 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:14.996 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:14.996 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:14.996 DEBUG:boto:Method: GET 2024-04-16T20:07:14.996 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:14.996 DEBUG:boto:Data: 2024-04-16T20:07:14.996 DEBUG:boto:Headers: {} 2024-04-16T20:07:14.996 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:14.996 DEBUG:boto:Port: 8000 2024-04-16T20:07:14.996 DEBUG:boto:Params: {} 2024-04-16T20:07:14.996 DEBUG:boto:Token: None 2024-04-16T20:07:14.997 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:14 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:14.997 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1nV1oB85rWarNcUxdqmsjd+RwBk= 2024-04-16T20:07:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:14 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1nV1oB85rWarNcUxdqmsjd+RwBk=', 'Content-Length': '0'} 2024-04-16T20:07:14.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45a84d8b428dc09-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.042 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.042 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.043 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.043 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.043 DEBUG:boto:Method: GET 2024-04-16T20:07:15.043 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.043 DEBUG:boto:Data: 2024-04-16T20:07:15.043 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.043 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.043 DEBUG:boto:Params: {} 2024-04-16T20:07:15.043 DEBUG:boto:Token: None 2024-04-16T20:07:15.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4= 2024-04-16T20:07:15.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4=', 'Content-Length': '0'} 2024-04-16T20:07:15.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000165843259fe0f88-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.046 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.046 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.046 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.046 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.046 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.046 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.046 DEBUG:boto:Data: 2024-04-16T20:07:15.047 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.047 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.047 DEBUG:boto:Params: {} 2024-04-16T20:07:15.047 DEBUG:boto:Token: None 2024-04-16T20:07:15.047 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.047 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kroSw/lyyV2MawtGxbWzhr3HynM= 2024-04-16T20:07:15.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kroSw/lyyV2MawtGxbWzhr3HynM=', 'Content-Length': '0'} 2024-04-16T20:07:15.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bb7ccf791410ac-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.049 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.050 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.050 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.050 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.050 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.050 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.050 DEBUG:boto:Data: 2024-04-16T20:07:15.050 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.050 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.050 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.050 DEBUG:boto:Params: {} 2024-04-16T20:07:15.050 DEBUG:boto:Token: None 2024-04-16T20:07:15.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kroSw/lyyV2MawtGxbWzhr3HynM= 2024-04-16T20:07:15.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kroSw/lyyV2MawtGxbWzhr3HynM=', 'Content-Length': '0'} 2024-04-16T20:07:15.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000871d7fc626eb975e-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:15.053 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.053 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.053 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.053 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.054 DEBUG:boto:Method: GET 2024-04-16T20:07:15.054 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.054 DEBUG:boto:Data: 2024-04-16T20:07:15.054 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.054 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.054 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.054 DEBUG:boto:Params: {} 2024-04-16T20:07:15.054 DEBUG:boto:Token: None 2024-04-16T20:07:15.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4= 2024-04-16T20:07:15.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4=', 'Content-Length': '0'} 2024-04-16T20:07:15.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a54efab13332bae-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.098 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.098 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.098 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.099 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.099 DEBUG:boto:Method: GET 2024-04-16T20:07:15.099 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.099 DEBUG:boto:Data: 2024-04-16T20:07:15.099 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.099 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.099 DEBUG:boto:Params: {} 2024-04-16T20:07:15.099 DEBUG:boto:Token: None 2024-04-16T20:07:15.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:15.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4= 2024-04-16T20:07:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/YQayGY7TrSohiQwAKP5tKLbFL4=', 'Content-Length': '0'} 2024-04-16T20:07:15.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3501a42529a63d7-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.102 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.102 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.102 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.102 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.103 DEBUG:boto:Method: GET 2024-04-16T20:07:15.103 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.103 DEBUG:boto:Data: 2024-04-16T20:07:15.103 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.103 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.103 DEBUG:boto:Params: {} 2024-04-16T20:07:15.103 DEBUG:boto:Token: None 2024-04-16T20:07:15.103 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13bd82d3aee3694-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.146 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.147 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.147 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.147 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.147 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.147 DEBUG:boto:Method: GET 2024-04-16T20:07:15.147 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.147 DEBUG:boto:Data: 2024-04-16T20:07:15.147 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.147 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.147 DEBUG:boto:Params: {} 2024-04-16T20:07:15.147 DEBUG:boto:Token: None 2024-04-16T20:07:15.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097494f163eabc6ba-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.152 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.152 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.152 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.152 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.152 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:15.152 DEBUG:boto:Data: 2024-04-16T20:07:15.152 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.152 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.152 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.152 DEBUG:boto:Params: {} 2024-04-16T20:07:15.153 DEBUG:boto:Token: None 2024-04-16T20:07:15.153 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51 2024-04-16T20:07:15.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LnAEgx9AfHYhaisgYdja4ZoKVqg= 2024-04-16T20:07:15.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LnAEgx9AfHYhaisgYdja4ZoKVqg=', 'Content-Length': '0'} 2024-04-16T20:07:15.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058def871c99788b5-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.155 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.155 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.155 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.155 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:15.156 DEBUG:boto:Data: 2024-04-16T20:07:15.156 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.156 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.156 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.156 DEBUG:boto:Params: {} 2024-04-16T20:07:15.156 DEBUG:boto:Token: None 2024-04-16T20:07:15.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51 2024-04-16T20:07:15.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LnAEgx9AfHYhaisgYdja4ZoKVqg= 2024-04-16T20:07:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LnAEgx9AfHYhaisgYdja4ZoKVqg=', 'Content-Length': '0'} 2024-04-16T20:07:15.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe4b1d1971d4ebf-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:15.159 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.159 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.159 DEBUG:boto:Method: GET 2024-04-16T20:07:15.159 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:15.159 DEBUG:boto:Data: 2024-04-16T20:07:15.159 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.159 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.159 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.160 DEBUG:boto:Params: {} 2024-04-16T20:07:15.160 DEBUG:boto:Token: None 2024-04-16T20:07:15.160 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51 2024-04-16T20:07:15.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mxUK/CDtKi15adrrLZxs0ZzFpgw= 2024-04-16T20:07:15.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mxUK/CDtKi15adrrLZxs0ZzFpgw=', 'Content-Length': '0'} 2024-04-16T20:07:15.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e187c927ad3bcd6-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.202 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:15.202 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:15.203 DEBUG:boto:Method: GET 2024-04-16T20:07:15.203 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:15.203 DEBUG:boto:Data: 2024-04-16T20:07:15.203 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.203 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.203 DEBUG:boto:Params: {} 2024-04-16T20:07:15.203 DEBUG:boto:Token: None 2024-04-16T20:07:15.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj51 2024-04-16T20:07:15.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mxUK/CDtKi15adrrLZxs0ZzFpgw= 2024-04-16T20:07:15.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mxUK/CDtKi15adrrLZxs0ZzFpgw=', 'Content-Length': '0'} 2024-04-16T20:07:15.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a822de8871f6e69c-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.205 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.206 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.206 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.206 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.206 DEBUG:boto:Method: GET 2024-04-16T20:07:15.206 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.206 DEBUG:boto:Data: 2024-04-16T20:07:15.206 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.206 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.206 DEBUG:boto:Params: {} 2024-04-16T20:07:15.206 DEBUG:boto:Token: None 2024-04-16T20:07:15.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422753b2bf33d38c-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.250 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.250 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.251 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.251 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.251 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.251 DEBUG:boto:Method: GET 2024-04-16T20:07:15.251 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:15.251 DEBUG:boto:Data: 2024-04-16T20:07:15.251 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.251 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.251 DEBUG:boto:Params: {} 2024-04-16T20:07:15.251 DEBUG:boto:Token: None 2024-04-16T20:07:15.252 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832b1297f4f97a06-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.255 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:15.256 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.256 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.256 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.256 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.256 DEBUG:boto:Method: GET 2024-04-16T20:07:15.256 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.256 DEBUG:boto:Data: 2024-04-16T20:07:15.256 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.256 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.256 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.256 DEBUG:boto:Params: {} 2024-04-16T20:07:15.256 DEBUG:boto:Token: None 2024-04-16T20:07:15.257 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.257 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU= 2024-04-16T20:07:15.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU=', 'Content-Length': '0'} 2024-04-16T20:07:15.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004578a0bdcf20bd21-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.302 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.303 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.303 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.303 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.303 DEBUG:boto:Method: GET 2024-04-16T20:07:15.303 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.303 DEBUG:boto:Data: 2024-04-16T20:07:15.303 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.303 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.303 DEBUG:boto:Params: {} 2024-04-16T20:07:15.303 DEBUG:boto:Token: None 2024-04-16T20:07:15.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU= 2024-04-16T20:07:15.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU=', 'Content-Length': '0'} 2024-04-16T20:07:15.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f94522b6716630-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.307 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.307 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.307 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.307 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.307 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.307 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.307 DEBUG:boto:Data: 2024-04-16T20:07:15.307 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.307 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.307 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.307 DEBUG:boto:Params: {} 2024-04-16T20:07:15.308 DEBUG:boto:Token: None 2024-04-16T20:07:15.308 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cJ/PfBtxYtdyZ0vsj9xCXVy9lDc= 2024-04-16T20:07:15.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cJ/PfBtxYtdyZ0vsj9xCXVy9lDc=', 'Content-Length': '0'} 2024-04-16T20:07:15.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2070dc9721b9114-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.310 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.311 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.311 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.311 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.311 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.311 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.311 DEBUG:boto:Data: 2024-04-16T20:07:15.311 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.311 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.311 DEBUG:boto:Params: {} 2024-04-16T20:07:15.311 DEBUG:boto:Token: None 2024-04-16T20:07:15.312 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cJ/PfBtxYtdyZ0vsj9xCXVy9lDc= 2024-04-16T20:07:15.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cJ/PfBtxYtdyZ0vsj9xCXVy9lDc=', 'Content-Length': '0'} 2024-04-16T20:07:15.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b763d2ff1b2f49-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:15.314 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.314 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.314 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.314 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.314 DEBUG:boto:Method: GET 2024-04-16T20:07:15.314 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.315 DEBUG:boto:Data: 2024-04-16T20:07:15.315 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.315 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.315 DEBUG:boto:Params: {} 2024-04-16T20:07:15.315 DEBUG:boto:Token: None 2024-04-16T20:07:15.315 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.315 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU= 2024-04-16T20:07:15.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU=', 'Content-Length': '0'} 2024-04-16T20:07:15.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c25488216e3d6df-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.358 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.358 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.359 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.359 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.359 DEBUG:boto:Method: GET 2024-04-16T20:07:15.359 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.359 DEBUG:boto:Data: 2024-04-16T20:07:15.359 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.359 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.359 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.359 DEBUG:boto:Params: {} 2024-04-16T20:07:15.359 DEBUG:boto:Token: None 2024-04-16T20:07:15.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:15.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU= 2024-04-16T20:07:15.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjZEa4zWwDbb74fb0A+KU2H6WzU=', 'Content-Length': '0'} 2024-04-16T20:07:15.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7185183bec5f5b-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.362 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.362 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.363 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.363 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.363 DEBUG:boto:Method: GET 2024-04-16T20:07:15.363 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.363 DEBUG:boto:Data: 2024-04-16T20:07:15.363 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.363 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.363 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.363 DEBUG:boto:Params: {} 2024-04-16T20:07:15.363 DEBUG:boto:Token: None 2024-04-16T20:07:15.363 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b620cbc3714bb2-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.366 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.366 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.367 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.367 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.367 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.367 DEBUG:boto:Method: GET 2024-04-16T20:07:15.367 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.367 DEBUG:boto:Data: 2024-04-16T20:07:15.367 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.367 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.367 DEBUG:boto:Params: {} 2024-04-16T20:07:15.367 DEBUG:boto:Token: None 2024-04-16T20:07:15.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58a5f22c8b13085-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.371 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.372 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.372 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.372 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.372 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:15.372 DEBUG:boto:Data: 2024-04-16T20:07:15.372 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.372 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.372 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.372 DEBUG:boto:Params: {} 2024-04-16T20:07:15.372 DEBUG:boto:Token: None 2024-04-16T20:07:15.373 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55 2024-04-16T20:07:15.373 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibUGu1cnHnfK00PFPrrYrM5ML5g= 2024-04-16T20:07:15.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibUGu1cnHnfK00PFPrrYrM5ML5g=', 'Content-Length': '0'} 2024-04-16T20:07:15.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000533e88490fd4ac9-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.375 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.375 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.375 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.376 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:15.376 DEBUG:boto:Data: 2024-04-16T20:07:15.376 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.376 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.376 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.376 DEBUG:boto:Params: {} 2024-04-16T20:07:15.376 DEBUG:boto:Token: None 2024-04-16T20:07:15.376 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55 2024-04-16T20:07:15.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibUGu1cnHnfK00PFPrrYrM5ML5g= 2024-04-16T20:07:15.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibUGu1cnHnfK00PFPrrYrM5ML5g=', 'Content-Length': '0'} 2024-04-16T20:07:15.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11116d8eda4688a-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:15.379 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.379 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.379 DEBUG:boto:Method: GET 2024-04-16T20:07:15.379 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:15.379 DEBUG:boto:Data: 2024-04-16T20:07:15.379 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.379 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.379 DEBUG:boto:Params: {} 2024-04-16T20:07:15.380 DEBUG:boto:Token: None 2024-04-16T20:07:15.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55 2024-04-16T20:07:15.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GE6c2w3QK7NzQRSk6HyoHvfHKCI= 2024-04-16T20:07:15.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GE6c2w3QK7NzQRSk6HyoHvfHKCI=', 'Content-Length': '0'} 2024-04-16T20:07:15.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:58:07 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df867dafef34daa4-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.422 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:15.422 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:15.423 DEBUG:boto:Method: GET 2024-04-16T20:07:15.423 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:15.423 DEBUG:boto:Data: 2024-04-16T20:07:15.423 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.423 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.423 DEBUG:boto:Params: {} 2024-04-16T20:07:15.423 DEBUG:boto:Token: None 2024-04-16T20:07:15.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj55 2024-04-16T20:07:15.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GE6c2w3QK7NzQRSk6HyoHvfHKCI= 2024-04-16T20:07:15.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GE6c2w3QK7NzQRSk6HyoHvfHKCI=', 'Content-Length': '0'} 2024-04-16T20:07:15.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c667ed1a8a063f-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.426 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.426 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.426 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.427 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.427 DEBUG:boto:Method: GET 2024-04-16T20:07:15.427 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.427 DEBUG:boto:Data: 2024-04-16T20:07:15.427 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.427 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.427 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.427 DEBUG:boto:Params: {} 2024-04-16T20:07:15.427 DEBUG:boto:Token: None 2024-04-16T20:07:15.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01c07083fb623ca-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.470 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.471 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.471 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.471 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.471 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.471 DEBUG:boto:Method: GET 2024-04-16T20:07:15.471 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:15.471 DEBUG:boto:Data: 2024-04-16T20:07:15.471 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.471 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.472 DEBUG:boto:Params: {} 2024-04-16T20:07:15.472 DEBUG:boto:Token: None 2024-04-16T20:07:15.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed2f6eecd57d1cef-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.477 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:15.478 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.478 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.478 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.478 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.478 DEBUG:boto:Method: GET 2024-04-16T20:07:15.478 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.478 DEBUG:boto:Data: 2024-04-16T20:07:15.478 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.478 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.478 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.479 DEBUG:boto:Params: {} 2024-04-16T20:07:15.479 DEBUG:boto:Token: None 2024-04-16T20:07:15.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8= 2024-04-16T20:07:15.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8=', 'Content-Length': '0'} 2024-04-16T20:07:15.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad771c4297e6f1d7-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.522 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.522 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.523 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.523 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.523 DEBUG:boto:Method: GET 2024-04-16T20:07:15.523 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.523 DEBUG:boto:Data: 2024-04-16T20:07:15.523 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.523 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.523 DEBUG:boto:Params: {} 2024-04-16T20:07:15.523 DEBUG:boto:Token: None 2024-04-16T20:07:15.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8= 2024-04-16T20:07:15.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8=', 'Content-Length': '0'} 2024-04-16T20:07:15.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56e95bf489bb345-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.526 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.526 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.526 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.527 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.527 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.527 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.527 DEBUG:boto:Data: 2024-04-16T20:07:15.527 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.527 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.527 DEBUG:boto:Params: {} 2024-04-16T20:07:15.527 DEBUG:boto:Token: None 2024-04-16T20:07:15.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Mw2NdRO8Y8EqIsePR350y1zQOVE= 2024-04-16T20:07:15.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Mw2NdRO8Y8EqIsePR350y1zQOVE=', 'Content-Length': '0'} 2024-04-16T20:07:15.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe9dfd3bf27a158-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.530 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.530 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.530 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.530 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.530 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.530 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.531 DEBUG:boto:Data: 2024-04-16T20:07:15.531 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.531 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.531 DEBUG:boto:Params: {} 2024-04-16T20:07:15.531 DEBUG:boto:Token: None 2024-04-16T20:07:15.531 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Mw2NdRO8Y8EqIsePR350y1zQOVE= 2024-04-16T20:07:15.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Mw2NdRO8Y8EqIsePR350y1zQOVE=', 'Content-Length': '0'} 2024-04-16T20:07:15.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe881178b47c959-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:15.534 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.534 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.534 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.534 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.534 DEBUG:boto:Method: GET 2024-04-16T20:07:15.534 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.535 DEBUG:boto:Data: 2024-04-16T20:07:15.535 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.535 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.535 DEBUG:boto:Params: {} 2024-04-16T20:07:15.535 DEBUG:boto:Token: None 2024-04-16T20:07:15.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8= 2024-04-16T20:07:15.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8=', 'Content-Length': '0'} 2024-04-16T20:07:15.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003fffe0d27005c82-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.578 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.578 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.579 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.579 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.579 DEBUG:boto:Method: GET 2024-04-16T20:07:15.579 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.579 DEBUG:boto:Data: 2024-04-16T20:07:15.579 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.579 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.579 DEBUG:boto:Params: {} 2024-04-16T20:07:15.579 DEBUG:boto:Token: None 2024-04-16T20:07:15.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:15.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8= 2024-04-16T20:07:15.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NW8QD1D0PvIBi0ZTYvi6omY1Mv8=', 'Content-Length': '0'} 2024-04-16T20:07:15.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000759b233161dc41af-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.582 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.582 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.583 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.583 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.583 DEBUG:boto:Method: GET 2024-04-16T20:07:15.583 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.583 DEBUG:boto:Data: 2024-04-16T20:07:15.583 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.583 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.583 DEBUG:boto:Params: {} 2024-04-16T20:07:15.583 DEBUG:boto:Token: None 2024-04-16T20:07:15.584 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.584 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009599881bcf0a1116-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.626 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.627 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.627 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.627 DEBUG:boto:Method: GET 2024-04-16T20:07:15.627 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.627 DEBUG:boto:Data: 2024-04-16T20:07:15.627 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.627 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.628 DEBUG:boto:Params: {} 2024-04-16T20:07:15.628 DEBUG:boto:Token: None 2024-04-16T20:07:15.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe178da5805837b-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.632 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.632 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.632 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.632 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.632 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:15.632 DEBUG:boto:Data: 2024-04-16T20:07:15.632 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.633 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.633 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.633 DEBUG:boto:Params: {} 2024-04-16T20:07:15.633 DEBUG:boto:Token: None 2024-04-16T20:07:15.633 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57 2024-04-16T20:07:15.633 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mYr/4J8OzgvJjXvvtOofzaAPQ7E= 2024-04-16T20:07:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mYr/4J8OzgvJjXvvtOofzaAPQ7E=', 'Content-Length': '0'} 2024-04-16T20:07:15.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c76cd2f855803ab-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.636 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.636 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.636 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.636 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:15.636 DEBUG:boto:Data: 2024-04-16T20:07:15.636 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.636 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.636 DEBUG:boto:Params: {} 2024-04-16T20:07:15.637 DEBUG:boto:Token: None 2024-04-16T20:07:15.637 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57 2024-04-16T20:07:15.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mYr/4J8OzgvJjXvvtOofzaAPQ7E= 2024-04-16T20:07:15.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mYr/4J8OzgvJjXvvtOofzaAPQ7E=', 'Content-Length': '0'} 2024-04-16T20:07:15.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1fb0999c87e0ccd-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:15.639 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.639 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.639 DEBUG:boto:Method: GET 2024-04-16T20:07:15.640 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:15.640 DEBUG:boto:Data: 2024-04-16T20:07:15.640 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.640 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.640 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.640 DEBUG:boto:Params: {} 2024-04-16T20:07:15.640 DEBUG:boto:Token: None 2024-04-16T20:07:15.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57 2024-04-16T20:07:15.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ngz//CE3TCczLnRf2iPnZOYKoL0= 2024-04-16T20:07:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ngz//CE3TCczLnRf2iPnZOYKoL0=', 'Content-Length': '0'} 2024-04-16T20:07:15.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007208696a4461b57f-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.686 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:15.687 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:15.687 DEBUG:boto:Method: GET 2024-04-16T20:07:15.687 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:15.687 DEBUG:boto:Data: 2024-04-16T20:07:15.687 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.687 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.687 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.687 DEBUG:boto:Params: {} 2024-04-16T20:07:15.687 DEBUG:boto:Token: None 2024-04-16T20:07:15.687 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj57 2024-04-16T20:07:15.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ngz//CE3TCczLnRf2iPnZOYKoL0= 2024-04-16T20:07:15.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ngz//CE3TCczLnRf2iPnZOYKoL0=', 'Content-Length': '0'} 2024-04-16T20:07:15.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707b480799806449-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.690 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.691 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.691 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.691 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.691 DEBUG:boto:Method: GET 2024-04-16T20:07:15.691 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.691 DEBUG:boto:Data: 2024-04-16T20:07:15.691 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.691 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.691 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.691 DEBUG:boto:Params: {} 2024-04-16T20:07:15.692 DEBUG:boto:Token: None 2024-04-16T20:07:15.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5570ea9f7d4fce-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.738 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.739 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.739 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.739 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.739 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.739 DEBUG:boto:Method: GET 2024-04-16T20:07:15.739 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:15.739 DEBUG:boto:Data: 2024-04-16T20:07:15.739 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.740 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.740 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.740 DEBUG:boto:Params: {} 2024-04-16T20:07:15.740 DEBUG:boto:Token: None 2024-04-16T20:07:15.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef102b7f60f298b5-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.744 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:15.744 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.745 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.745 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.745 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.745 DEBUG:boto:Method: GET 2024-04-16T20:07:15.745 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.745 DEBUG:boto:Data: 2024-04-16T20:07:15.745 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.745 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.745 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.745 DEBUG:boto:Params: {} 2024-04-16T20:07:15.745 DEBUG:boto:Token: None 2024-04-16T20:07:15.746 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.746 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24= 2024-04-16T20:07:15.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24=', 'Content-Length': '0'} 2024-04-16T20:07:15.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aff9f671c7874f3-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.790 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.791 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.791 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.791 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.791 DEBUG:boto:Method: GET 2024-04-16T20:07:15.791 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.791 DEBUG:boto:Data: 2024-04-16T20:07:15.791 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.791 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.791 DEBUG:boto:Params: {} 2024-04-16T20:07:15.792 DEBUG:boto:Token: None 2024-04-16T20:07:15.792 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24= 2024-04-16T20:07:15.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24=', 'Content-Length': '0'} 2024-04-16T20:07:15.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d31e44a583e6c38-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.795 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.795 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.795 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.795 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.795 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.795 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.795 DEBUG:boto:Data: 2024-04-16T20:07:15.795 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.795 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.796 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.796 DEBUG:boto:Params: {} 2024-04-16T20:07:15.796 DEBUG:boto:Token: None 2024-04-16T20:07:15.796 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7UvkPq2gIC1+8XcsqiZ32fOXZso= 2024-04-16T20:07:15.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7UvkPq2gIC1+8XcsqiZ32fOXZso=', 'Content-Length': '0'} 2024-04-16T20:07:15.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157e861ff5357cf2-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.799 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.799 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.799 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.799 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.799 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.799 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.799 DEBUG:boto:Data: 2024-04-16T20:07:15.800 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.800 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.800 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.800 DEBUG:boto:Params: {} 2024-04-16T20:07:15.800 DEBUG:boto:Token: None 2024-04-16T20:07:15.800 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7UvkPq2gIC1+8XcsqiZ32fOXZso= 2024-04-16T20:07:15.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7UvkPq2gIC1+8XcsqiZ32fOXZso=', 'Content-Length': '0'} 2024-04-16T20:07:15.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd50c2f38d1f0c81-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:15.803 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.803 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.803 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.803 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.804 DEBUG:boto:Method: GET 2024-04-16T20:07:15.804 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.804 DEBUG:boto:Data: 2024-04-16T20:07:15.804 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.804 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.804 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.804 DEBUG:boto:Params: {} 2024-04-16T20:07:15.804 DEBUG:boto:Token: None 2024-04-16T20:07:15.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24= 2024-04-16T20:07:15.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24=', 'Content-Length': '0'} 2024-04-16T20:07:15.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001338a1b7891ffae8-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.850 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.851 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.851 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.851 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.851 DEBUG:boto:Method: GET 2024-04-16T20:07:15.851 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.851 DEBUG:boto:Data: 2024-04-16T20:07:15.851 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.851 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.851 DEBUG:boto:Params: {} 2024-04-16T20:07:15.852 DEBUG:boto:Token: None 2024-04-16T20:07:15.852 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:15.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24= 2024-04-16T20:07:15.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3u5r8wB/W8qpXePazyKRN900j24=', 'Content-Length': '0'} 2024-04-16T20:07:15.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018057f5b41c2fd3d-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.855 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.855 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.855 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.855 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.855 DEBUG:boto:Method: GET 2024-04-16T20:07:15.855 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.855 DEBUG:boto:Data: 2024-04-16T20:07:15.855 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.855 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.855 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.856 DEBUG:boto:Params: {} 2024-04-16T20:07:15.856 DEBUG:boto:Token: None 2024-04-16T20:07:15.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d5b2db8a4e5ac5-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.859 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.859 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.859 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.859 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.859 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.859 DEBUG:boto:Method: GET 2024-04-16T20:07:15.859 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.860 DEBUG:boto:Data: 2024-04-16T20:07:15.860 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.860 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.860 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.860 DEBUG:boto:Params: {} 2024-04-16T20:07:15.860 DEBUG:boto:Token: None 2024-04-16T20:07:15.860 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a76361d83ee798-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.906 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.907 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.907 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.907 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.907 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:15.907 DEBUG:boto:Data: 2024-04-16T20:07:15.907 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.907 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.908 DEBUG:boto:Params: {} 2024-04-16T20:07:15.908 DEBUG:boto:Token: None 2024-04-16T20:07:15.908 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58 2024-04-16T20:07:15.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kWhTT5TZanQml5/2H9FB4Etr62I= 2024-04-16T20:07:15.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kWhTT5TZanQml5/2H9FB4Etr62I=', 'Content-Length': '0'} 2024-04-16T20:07:15.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e596ca1887a2a9-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.910 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.911 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.911 DEBUG:boto:Method: HEAD 2024-04-16T20:07:15.911 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:15.911 DEBUG:boto:Data: 2024-04-16T20:07:15.911 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.911 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.911 DEBUG:boto:Params: {} 2024-04-16T20:07:15.911 DEBUG:boto:Token: None 2024-04-16T20:07:15.912 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58 2024-04-16T20:07:15.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kWhTT5TZanQml5/2H9FB4Etr62I= 2024-04-16T20:07:15.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kWhTT5TZanQml5/2H9FB4Etr62I=', 'Content-Length': '0'} 2024-04-16T20:07:15.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d25225bad848b0-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:15.915 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.915 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.915 DEBUG:boto:Method: GET 2024-04-16T20:07:15.915 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:15.915 DEBUG:boto:Data: 2024-04-16T20:07:15.915 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.915 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.915 DEBUG:boto:Params: {} 2024-04-16T20:07:15.916 DEBUG:boto:Token: None 2024-04-16T20:07:15.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58 2024-04-16T20:07:15.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fyGBwqUXDHcFaE+j54HJ6iFTz/k= 2024-04-16T20:07:15.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fyGBwqUXDHcFaE+j54HJ6iFTz/k=', 'Content-Length': '0'} 2024-04-16T20:07:15.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160792a53e3b6c96-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.958 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:15.959 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:15.959 DEBUG:boto:Method: GET 2024-04-16T20:07:15.959 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:15.959 DEBUG:boto:Data: 2024-04-16T20:07:15.959 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.959 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.959 DEBUG:boto:Params: {} 2024-04-16T20:07:15.959 DEBUG:boto:Token: None 2024-04-16T20:07:15.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj58 2024-04-16T20:07:15.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fyGBwqUXDHcFaE+j54HJ6iFTz/k= 2024-04-16T20:07:15.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fyGBwqUXDHcFaE+j54HJ6iFTz/k=', 'Content-Length': '0'} 2024-04-16T20:07:15.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e605a898e2ec83a8-00661eda73-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.963 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.963 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.963 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.963 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.963 DEBUG:boto:Method: GET 2024-04-16T20:07:15.963 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.963 DEBUG:boto:Data: 2024-04-16T20:07:15.964 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.964 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.964 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.964 DEBUG:boto:Params: {} 2024-04-16T20:07:15.964 DEBUG:boto:Token: None 2024-04-16T20:07:15.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004ce909697b31595-00661eda73-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.967 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.968 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:15.968 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:15.968 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.968 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.968 DEBUG:boto:Method: GET 2024-04-16T20:07:15.968 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:15.968 DEBUG:boto:Data: 2024-04-16T20:07:15.968 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.968 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:15.968 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.969 DEBUG:boto:Params: {} 2024-04-16T20:07:15.969 DEBUG:boto:Token: None 2024-04-16T20:07:15.969 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/ 2024-04-16T20:07:15.969 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc= 2024-04-16T20:07:15.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SOgc0OVw2ASjPCdHfls18CYLARc=', 'Content-Length': '0'} 2024-04-16T20:07:15.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b88ca0cf52d39e-00661eda73-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:15.973 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:15.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:15.973 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:15.973 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:15.974 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:15.974 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:15.974 DEBUG:boto:Method: GET 2024-04-16T20:07:15.974 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:15.974 DEBUG:boto:Data: 2024-04-16T20:07:15.974 DEBUG:boto:Headers: {} 2024-04-16T20:07:15.974 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:15.974 DEBUG:boto:Port: 8000 2024-04-16T20:07:15.974 DEBUG:boto:Params: {} 2024-04-16T20:07:15.975 DEBUG:boto:Token: None 2024-04-16T20:07:15.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:15 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:15.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Slji+ax4bp/ZzqWD8hCtihyEFJU= 2024-04-16T20:07:15.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Slji+ax4bp/ZzqWD8hCtihyEFJU=', 'Content-Length': '0'} 2024-04-16T20:07:15.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d90a6dd7135f147-00661eda73-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.018 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.019 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.019 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.019 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.019 DEBUG:boto:Method: GET 2024-04-16T20:07:16.019 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.019 DEBUG:boto:Data: 2024-04-16T20:07:16.019 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.019 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.019 DEBUG:boto:Params: {} 2024-04-16T20:07:16.020 DEBUG:boto:Token: None 2024-04-16T20:07:16.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY= 2024-04-16T20:07:16.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY=', 'Content-Length': '0'} 2024-04-16T20:07:16.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec42e4cc8672a77-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.022 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.023 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.023 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.023 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.023 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.023 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.023 DEBUG:boto:Data: 2024-04-16T20:07:16.023 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.023 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.024 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.024 DEBUG:boto:Params: {} 2024-04-16T20:07:16.024 DEBUG:boto:Token: None 2024-04-16T20:07:16.024 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bD1ZUS7DAysfwU5kS9G6tLpv2xs= 2024-04-16T20:07:16.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bD1ZUS7DAysfwU5kS9G6tLpv2xs=', 'Content-Length': '0'} 2024-04-16T20:07:16.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589889e22aa82472-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.027 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.027 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.027 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.027 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.027 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.028 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.028 DEBUG:boto:Data: 2024-04-16T20:07:16.028 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.028 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.028 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.028 DEBUG:boto:Params: {} 2024-04-16T20:07:16.028 DEBUG:boto:Token: None 2024-04-16T20:07:16.028 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bD1ZUS7DAysfwU5kS9G6tLpv2xs= 2024-04-16T20:07:16.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bD1ZUS7DAysfwU5kS9G6tLpv2xs=', 'Content-Length': '0'} 2024-04-16T20:07:16.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8c9e2e8199de99-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:16.032 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.032 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.032 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.032 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.032 DEBUG:boto:Method: GET 2024-04-16T20:07:16.032 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.032 DEBUG:boto:Data: 2024-04-16T20:07:16.032 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.032 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.033 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.033 DEBUG:boto:Params: {} 2024-04-16T20:07:16.033 DEBUG:boto:Token: None 2024-04-16T20:07:16.033 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.033 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY= 2024-04-16T20:07:16.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY=', 'Content-Length': '0'} 2024-04-16T20:07:16.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007266bb8ca3acd9b3-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.078 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.078 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.079 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.079 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.079 DEBUG:boto:Method: GET 2024-04-16T20:07:16.079 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.079 DEBUG:boto:Data: 2024-04-16T20:07:16.079 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.079 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.079 DEBUG:boto:Params: {} 2024-04-16T20:07:16.080 DEBUG:boto:Token: None 2024-04-16T20:07:16.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:16.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY= 2024-04-16T20:07:16.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:enMIDUgJkFw8DzVXrCtv0d/UQOY=', 'Content-Length': '0'} 2024-04-16T20:07:16.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac61bda3584e39c5-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.083 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.083 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.083 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.083 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.083 DEBUG:boto:Method: GET 2024-04-16T20:07:16.083 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.083 DEBUG:boto:Data: 2024-04-16T20:07:16.084 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.084 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.084 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.084 DEBUG:boto:Params: {} 2024-04-16T20:07:16.084 DEBUG:boto:Token: None 2024-04-16T20:07:16.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc0893e43e2c8235-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.130 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.131 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.131 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.131 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.131 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.131 DEBUG:boto:Method: GET 2024-04-16T20:07:16.131 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.131 DEBUG:boto:Data: 2024-04-16T20:07:16.131 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.132 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.132 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.132 DEBUG:boto:Params: {} 2024-04-16T20:07:16.132 DEBUG:boto:Token: None 2024-04-16T20:07:16.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ac10ed1066e5db-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.136 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.136 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.136 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.136 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.137 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:16.137 DEBUG:boto:Data: 2024-04-16T20:07:16.137 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.137 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.137 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.137 DEBUG:boto:Params: {} 2024-04-16T20:07:16.137 DEBUG:boto:Token: None 2024-04-16T20:07:16.137 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62 2024-04-16T20:07:16.137 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z1VWaGWyYCWbdshf4gv0/oJAwPE= 2024-04-16T20:07:16.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z1VWaGWyYCWbdshf4gv0/oJAwPE=', 'Content-Length': '0'} 2024-04-16T20:07:16.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8af2514629df63c-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.140 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.140 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.141 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.141 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:16.141 DEBUG:boto:Data: 2024-04-16T20:07:16.141 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.141 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.141 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.141 DEBUG:boto:Params: {} 2024-04-16T20:07:16.141 DEBUG:boto:Token: None 2024-04-16T20:07:16.141 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62 2024-04-16T20:07:16.142 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z1VWaGWyYCWbdshf4gv0/oJAwPE= 2024-04-16T20:07:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z1VWaGWyYCWbdshf4gv0/oJAwPE=', 'Content-Length': '0'} 2024-04-16T20:07:16.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c898510d6d84b4a6-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:16.144 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.144 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.144 DEBUG:boto:Method: GET 2024-04-16T20:07:16.145 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:16.145 DEBUG:boto:Data: 2024-04-16T20:07:16.145 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.145 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.145 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.145 DEBUG:boto:Params: {} 2024-04-16T20:07:16.145 DEBUG:boto:Token: None 2024-04-16T20:07:16.145 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62 2024-04-16T20:07:16.146 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ysWbpGEMKlZKNxF2RKJ6TpXggdg= 2024-04-16T20:07:16.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ysWbpGEMKlZKNxF2RKJ6TpXggdg=', 'Content-Length': '0'} 2024-04-16T20:07:16.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09fb3b00720675b-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.190 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:16.191 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:16.191 DEBUG:boto:Method: GET 2024-04-16T20:07:16.191 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:16.191 DEBUG:boto:Data: 2024-04-16T20:07:16.191 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.191 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.191 DEBUG:boto:Params: {} 2024-04-16T20:07:16.191 DEBUG:boto:Token: None 2024-04-16T20:07:16.192 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj62 2024-04-16T20:07:16.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ysWbpGEMKlZKNxF2RKJ6TpXggdg= 2024-04-16T20:07:16.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ysWbpGEMKlZKNxF2RKJ6TpXggdg=', 'Content-Length': '0'} 2024-04-16T20:07:16.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1b246b4d7efcdc-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.194 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.195 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.195 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.195 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.195 DEBUG:boto:Method: GET 2024-04-16T20:07:16.195 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.195 DEBUG:boto:Data: 2024-04-16T20:07:16.195 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.195 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.196 DEBUG:boto:Params: {} 2024-04-16T20:07:16.196 DEBUG:boto:Token: None 2024-04-16T20:07:16.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000745d22fbcbd4d072-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.242 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.243 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.243 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.243 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.243 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.243 DEBUG:boto:Method: GET 2024-04-16T20:07:16.243 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:16.243 DEBUG:boto:Data: 2024-04-16T20:07:16.243 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.244 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.244 DEBUG:boto:Params: {} 2024-04-16T20:07:16.244 DEBUG:boto:Token: None 2024-04-16T20:07:16.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e019f30d8ee3e268-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.248 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:16.249 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.249 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.249 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.249 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.249 DEBUG:boto:Method: GET 2024-04-16T20:07:16.249 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.249 DEBUG:boto:Data: 2024-04-16T20:07:16.249 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.249 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.249 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.250 DEBUG:boto:Params: {} 2024-04-16T20:07:16.250 DEBUG:boto:Token: None 2024-04-16T20:07:16.250 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.250 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA= 2024-04-16T20:07:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA=', 'Content-Length': '0'} 2024-04-16T20:07:16.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8e59eaef2af57f-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.294 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.294 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.295 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.295 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.295 DEBUG:boto:Method: GET 2024-04-16T20:07:16.295 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.295 DEBUG:boto:Data: 2024-04-16T20:07:16.295 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.295 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.295 DEBUG:boto:Params: {} 2024-04-16T20:07:16.295 DEBUG:boto:Token: None 2024-04-16T20:07:16.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA= 2024-04-16T20:07:16.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA=', 'Content-Length': '0'} 2024-04-16T20:07:16.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007449b5bd555040cd-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.298 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.298 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.298 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.298 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.298 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.298 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.298 DEBUG:boto:Data: 2024-04-16T20:07:16.298 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.299 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.299 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.299 DEBUG:boto:Params: {} 2024-04-16T20:07:16.299 DEBUG:boto:Token: None 2024-04-16T20:07:16.299 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lGasT86omsFAKQ70k/l7SqgXSe4= 2024-04-16T20:07:16.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lGasT86omsFAKQ70k/l7SqgXSe4=', 'Content-Length': '0'} 2024-04-16T20:07:16.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f4345f1d6475f8-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.301 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.302 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.302 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.302 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.302 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.302 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.302 DEBUG:boto:Data: 2024-04-16T20:07:16.302 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.302 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.302 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.302 DEBUG:boto:Params: {} 2024-04-16T20:07:16.303 DEBUG:boto:Token: None 2024-04-16T20:07:16.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lGasT86omsFAKQ70k/l7SqgXSe4= 2024-04-16T20:07:16.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lGasT86omsFAKQ70k/l7SqgXSe4=', 'Content-Length': '0'} 2024-04-16T20:07:16.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f65985c106cd67-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:16.305 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.305 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.305 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.306 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.306 DEBUG:boto:Method: GET 2024-04-16T20:07:16.306 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.306 DEBUG:boto:Data: 2024-04-16T20:07:16.306 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.306 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.306 DEBUG:boto:Params: {} 2024-04-16T20:07:16.306 DEBUG:boto:Token: None 2024-04-16T20:07:16.306 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA= 2024-04-16T20:07:16.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA=', 'Content-Length': '0'} 2024-04-16T20:07:16.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd0d8a9933dac4a-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.350 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.350 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.350 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.350 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.351 DEBUG:boto:Method: GET 2024-04-16T20:07:16.351 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.351 DEBUG:boto:Data: 2024-04-16T20:07:16.351 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.351 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.351 DEBUG:boto:Params: {} 2024-04-16T20:07:16.351 DEBUG:boto:Token: None 2024-04-16T20:07:16.351 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:16.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA= 2024-04-16T20:07:16.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k98q3f4Nh+I+NWWYE1mYaJTSeaA=', 'Content-Length': '0'} 2024-04-16T20:07:16.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ae0f954a4d1d60-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.353 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.354 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.354 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.354 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.354 DEBUG:boto:Method: GET 2024-04-16T20:07:16.354 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.354 DEBUG:boto:Data: 2024-04-16T20:07:16.354 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.354 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.354 DEBUG:boto:Params: {} 2024-04-16T20:07:16.355 DEBUG:boto:Token: None 2024-04-16T20:07:16.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ce88fcbfccfa654-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.398 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.399 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.399 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.399 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.399 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.399 DEBUG:boto:Method: GET 2024-04-16T20:07:16.399 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.399 DEBUG:boto:Data: 2024-04-16T20:07:16.399 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.399 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.399 DEBUG:boto:Params: {} 2024-04-16T20:07:16.399 DEBUG:boto:Token: None 2024-04-16T20:07:16.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823a3d7ff2505679-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.403 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.404 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.404 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.404 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.404 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:16.404 DEBUG:boto:Data: 2024-04-16T20:07:16.404 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.404 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.404 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.404 DEBUG:boto:Params: {} 2024-04-16T20:07:16.404 DEBUG:boto:Token: None 2024-04-16T20:07:16.405 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68 2024-04-16T20:07:16.405 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IhlSU3GFeMQCdLHREq7QirYeH1Q= 2024-04-16T20:07:16.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IhlSU3GFeMQCdLHREq7QirYeH1Q=', 'Content-Length': '0'} 2024-04-16T20:07:16.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d6dfb60e9a3977-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.407 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.407 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.407 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.407 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:16.407 DEBUG:boto:Data: 2024-04-16T20:07:16.407 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.408 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.408 DEBUG:boto:Params: {} 2024-04-16T20:07:16.408 DEBUG:boto:Token: None 2024-04-16T20:07:16.408 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68 2024-04-16T20:07:16.408 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IhlSU3GFeMQCdLHREq7QirYeH1Q= 2024-04-16T20:07:16.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IhlSU3GFeMQCdLHREq7QirYeH1Q=', 'Content-Length': '0'} 2024-04-16T20:07:16.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945ce429eb9dd5dd-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:16.410 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.410 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.410 DEBUG:boto:Method: GET 2024-04-16T20:07:16.410 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:16.411 DEBUG:boto:Data: 2024-04-16T20:07:16.411 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.411 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.411 DEBUG:boto:Params: {} 2024-04-16T20:07:16.411 DEBUG:boto:Token: None 2024-04-16T20:07:16.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68 2024-04-16T20:07:16.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nDvexnRfKZ1cYA1CQxlx+TCCWmI= 2024-04-16T20:07:16.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nDvexnRfKZ1cYA1CQxlx+TCCWmI=', 'Content-Length': '0'} 2024-04-16T20:07:16.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:59:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2d635d9cf8e956-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.454 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:16.454 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:16.454 DEBUG:boto:Method: GET 2024-04-16T20:07:16.455 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:16.455 DEBUG:boto:Data: 2024-04-16T20:07:16.455 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.455 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.455 DEBUG:boto:Params: {} 2024-04-16T20:07:16.455 DEBUG:boto:Token: None 2024-04-16T20:07:16.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj68 2024-04-16T20:07:16.455 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nDvexnRfKZ1cYA1CQxlx+TCCWmI= 2024-04-16T20:07:16.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nDvexnRfKZ1cYA1CQxlx+TCCWmI=', 'Content-Length': '0'} 2024-04-16T20:07:16.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47a5718e5f2109a-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.457 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.457 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.457 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.457 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.458 DEBUG:boto:Method: GET 2024-04-16T20:07:16.458 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.458 DEBUG:boto:Data: 2024-04-16T20:07:16.458 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.458 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.458 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.458 DEBUG:boto:Params: {} 2024-04-16T20:07:16.458 DEBUG:boto:Token: None 2024-04-16T20:07:16.458 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.458 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000527c93691772b28e-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.502 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.503 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.503 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.503 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.503 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.503 DEBUG:boto:Method: GET 2024-04-16T20:07:16.503 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:16.503 DEBUG:boto:Data: 2024-04-16T20:07:16.503 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.503 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.503 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.503 DEBUG:boto:Params: {} 2024-04-16T20:07:16.503 DEBUG:boto:Token: None 2024-04-16T20:07:16.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077142c29be700487-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.507 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:16.507 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.508 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.508 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.508 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.508 DEBUG:boto:Method: GET 2024-04-16T20:07:16.508 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.508 DEBUG:boto:Data: 2024-04-16T20:07:16.508 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.508 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.508 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.508 DEBUG:boto:Params: {} 2024-04-16T20:07:16.508 DEBUG:boto:Token: None 2024-04-16T20:07:16.508 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.509 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8= 2024-04-16T20:07:16.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8=', 'Content-Length': '0'} 2024-04-16T20:07:16.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005866a5c7148221cb-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.554 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.554 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.554 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.555 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.555 DEBUG:boto:Method: GET 2024-04-16T20:07:16.555 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.555 DEBUG:boto:Data: 2024-04-16T20:07:16.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.555 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.555 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.555 DEBUG:boto:Params: {} 2024-04-16T20:07:16.555 DEBUG:boto:Token: None 2024-04-16T20:07:16.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8= 2024-04-16T20:07:16.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8=', 'Content-Length': '0'} 2024-04-16T20:07:16.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d465fc491597b274-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.558 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.558 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.558 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.558 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.558 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.558 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.558 DEBUG:boto:Data: 2024-04-16T20:07:16.559 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.559 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.559 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.559 DEBUG:boto:Params: {} 2024-04-16T20:07:16.559 DEBUG:boto:Token: None 2024-04-16T20:07:16.559 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x50w/Z7rd/AtLNuuDaLs3XABONE= 2024-04-16T20:07:16.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x50w/Z7rd/AtLNuuDaLs3XABONE=', 'Content-Length': '0'} 2024-04-16T20:07:16.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008065974e19ccf5ca-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.561 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.562 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.562 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.562 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.562 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.562 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.562 DEBUG:boto:Data: 2024-04-16T20:07:16.562 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.562 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.562 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.562 DEBUG:boto:Params: {} 2024-04-16T20:07:16.562 DEBUG:boto:Token: None 2024-04-16T20:07:16.563 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x50w/Z7rd/AtLNuuDaLs3XABONE= 2024-04-16T20:07:16.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x50w/Z7rd/AtLNuuDaLs3XABONE=', 'Content-Length': '0'} 2024-04-16T20:07:16.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbcc245c5fa139c5-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:16.565 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.565 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.565 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.565 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.565 DEBUG:boto:Method: GET 2024-04-16T20:07:16.565 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.565 DEBUG:boto:Data: 2024-04-16T20:07:16.566 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.566 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.566 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.566 DEBUG:boto:Params: {} 2024-04-16T20:07:16.566 DEBUG:boto:Token: None 2024-04-16T20:07:16.566 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.566 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8= 2024-04-16T20:07:16.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8=', 'Content-Length': '0'} 2024-04-16T20:07:16.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97984c4ac14afeb-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.610 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.610 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.610 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.611 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.611 DEBUG:boto:Method: GET 2024-04-16T20:07:16.611 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.611 DEBUG:boto:Data: 2024-04-16T20:07:16.611 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.611 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.611 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.611 DEBUG:boto:Params: {} 2024-04-16T20:07:16.611 DEBUG:boto:Token: None 2024-04-16T20:07:16.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:16.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8= 2024-04-16T20:07:16.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iC3tD3HoB3vThzOz72oYgJ0Drs8=', 'Content-Length': '0'} 2024-04-16T20:07:16.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914f691523bc7fa9-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.614 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.614 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.614 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.614 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.615 DEBUG:boto:Method: GET 2024-04-16T20:07:16.615 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.615 DEBUG:boto:Data: 2024-04-16T20:07:16.615 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.615 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.615 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.615 DEBUG:boto:Params: {} 2024-04-16T20:07:16.615 DEBUG:boto:Token: None 2024-04-16T20:07:16.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed08bca7fd158e1-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.662 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.663 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.663 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.663 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.663 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.663 DEBUG:boto:Method: GET 2024-04-16T20:07:16.663 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.663 DEBUG:boto:Data: 2024-04-16T20:07:16.663 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.663 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.664 DEBUG:boto:Params: {} 2024-04-16T20:07:16.664 DEBUG:boto:Token: None 2024-04-16T20:07:16.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf76343754e6501f-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.667 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.668 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.668 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.668 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.668 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:16.668 DEBUG:boto:Data: 2024-04-16T20:07:16.668 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.668 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.668 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.668 DEBUG:boto:Params: {} 2024-04-16T20:07:16.668 DEBUG:boto:Token: None 2024-04-16T20:07:16.669 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69 2024-04-16T20:07:16.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GDwnDPodZmgagnovIYVlC/trZko= 2024-04-16T20:07:16.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GDwnDPodZmgagnovIYVlC/trZko=', 'Content-Length': '0'} 2024-04-16T20:07:16.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c714b97f28ad39-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.671 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.671 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.671 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.671 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:16.671 DEBUG:boto:Data: 2024-04-16T20:07:16.671 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.671 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.671 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.672 DEBUG:boto:Params: {} 2024-04-16T20:07:16.672 DEBUG:boto:Token: None 2024-04-16T20:07:16.672 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69 2024-04-16T20:07:16.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GDwnDPodZmgagnovIYVlC/trZko= 2024-04-16T20:07:16.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GDwnDPodZmgagnovIYVlC/trZko=', 'Content-Length': '0'} 2024-04-16T20:07:16.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ff3641728bedf8-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:16.674 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.674 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.674 DEBUG:boto:Method: GET 2024-04-16T20:07:16.675 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:16.675 DEBUG:boto:Data: 2024-04-16T20:07:16.675 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.675 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.675 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.675 DEBUG:boto:Params: {} 2024-04-16T20:07:16.675 DEBUG:boto:Token: None 2024-04-16T20:07:16.675 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69 2024-04-16T20:07:16.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WaqiDAFFIrM/nBQtWz/CHHNokl0= 2024-04-16T20:07:16.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WaqiDAFFIrM/nBQtWz/CHHNokl0=', 'Content-Length': '0'} 2024-04-16T20:07:16.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:33 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe819370dfeb48db-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.718 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:16.718 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:16.718 DEBUG:boto:Method: GET 2024-04-16T20:07:16.719 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:16.719 DEBUG:boto:Data: 2024-04-16T20:07:16.719 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.719 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.719 DEBUG:boto:Params: {} 2024-04-16T20:07:16.719 DEBUG:boto:Token: None 2024-04-16T20:07:16.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj69 2024-04-16T20:07:16.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WaqiDAFFIrM/nBQtWz/CHHNokl0= 2024-04-16T20:07:16.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WaqiDAFFIrM/nBQtWz/CHHNokl0=', 'Content-Length': '0'} 2024-04-16T20:07:16.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac88b3d5538cfa61-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.722 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.722 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.722 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.722 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.722 DEBUG:boto:Method: GET 2024-04-16T20:07:16.722 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.722 DEBUG:boto:Data: 2024-04-16T20:07:16.723 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.723 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.723 DEBUG:boto:Params: {} 2024-04-16T20:07:16.723 DEBUG:boto:Token: None 2024-04-16T20:07:16.723 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b182efcf509cc0b-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.766 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.766 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.767 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.767 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.767 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.767 DEBUG:boto:Method: GET 2024-04-16T20:07:16.767 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:16.767 DEBUG:boto:Data: 2024-04-16T20:07:16.767 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.767 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.767 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.767 DEBUG:boto:Params: {} 2024-04-16T20:07:16.767 DEBUG:boto:Token: None 2024-04-16T20:07:16.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a86f797dbe2e2b-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.771 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:16.772 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.772 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.772 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.772 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.772 DEBUG:boto:Method: GET 2024-04-16T20:07:16.772 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.772 DEBUG:boto:Data: 2024-04-16T20:07:16.772 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.772 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.772 DEBUG:boto:Params: {} 2024-04-16T20:07:16.773 DEBUG:boto:Token: None 2024-04-16T20:07:16.773 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.773 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw= 2024-04-16T20:07:16.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw=', 'Content-Length': '0'} 2024-04-16T20:07:16.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeede8334e9a3a02-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.818 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.818 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.818 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.819 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.819 DEBUG:boto:Method: GET 2024-04-16T20:07:16.819 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.819 DEBUG:boto:Data: 2024-04-16T20:07:16.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.819 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.819 DEBUG:boto:Params: {} 2024-04-16T20:07:16.819 DEBUG:boto:Token: None 2024-04-16T20:07:16.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw= 2024-04-16T20:07:16.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw=', 'Content-Length': '0'} 2024-04-16T20:07:16.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332f42867ab4c62d-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.822 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.822 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.822 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.822 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.822 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.822 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.822 DEBUG:boto:Data: 2024-04-16T20:07:16.822 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.822 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.822 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.823 DEBUG:boto:Params: {} 2024-04-16T20:07:16.823 DEBUG:boto:Token: None 2024-04-16T20:07:16.823 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0/PLvM6pNEntqwujh2TRuBlGluk= 2024-04-16T20:07:16.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0/PLvM6pNEntqwujh2TRuBlGluk=', 'Content-Length': '0'} 2024-04-16T20:07:16.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a7bc145fbe9d3b-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.825 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.825 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.825 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.825 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.826 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.826 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.826 DEBUG:boto:Data: 2024-04-16T20:07:16.826 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.826 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.826 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.826 DEBUG:boto:Params: {} 2024-04-16T20:07:16.826 DEBUG:boto:Token: None 2024-04-16T20:07:16.826 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.826 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0/PLvM6pNEntqwujh2TRuBlGluk= 2024-04-16T20:07:16.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0/PLvM6pNEntqwujh2TRuBlGluk=', 'Content-Length': '0'} 2024-04-16T20:07:16.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eb69bef52cbe3a-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:16.829 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.829 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.829 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.829 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.829 DEBUG:boto:Method: GET 2024-04-16T20:07:16.829 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.829 DEBUG:boto:Data: 2024-04-16T20:07:16.829 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.829 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.830 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.830 DEBUG:boto:Params: {} 2024-04-16T20:07:16.830 DEBUG:boto:Token: None 2024-04-16T20:07:16.830 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw= 2024-04-16T20:07:16.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw=', 'Content-Length': '0'} 2024-04-16T20:07:16.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30eec564543074d-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.874 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.874 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.875 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.875 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.875 DEBUG:boto:Method: GET 2024-04-16T20:07:16.875 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.875 DEBUG:boto:Data: 2024-04-16T20:07:16.875 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.875 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.875 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.875 DEBUG:boto:Params: {} 2024-04-16T20:07:16.875 DEBUG:boto:Token: None 2024-04-16T20:07:16.875 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:16.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw= 2024-04-16T20:07:16.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TfUZQt9V5uTJ2kfWXJ21yZ0womw=', 'Content-Length': '0'} 2024-04-16T20:07:16.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ae1b69f05b8494-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.878 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.878 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.878 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.878 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.878 DEBUG:boto:Method: GET 2024-04-16T20:07:16.878 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.878 DEBUG:boto:Data: 2024-04-16T20:07:16.878 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.878 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.878 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.878 DEBUG:boto:Params: {} 2024-04-16T20:07:16.879 DEBUG:boto:Token: None 2024-04-16T20:07:16.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4878be22bf82f3-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.922 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.923 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.923 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.923 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.923 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.923 DEBUG:boto:Method: GET 2024-04-16T20:07:16.923 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.923 DEBUG:boto:Data: 2024-04-16T20:07:16.923 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.923 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.923 DEBUG:boto:Params: {} 2024-04-16T20:07:16.924 DEBUG:boto:Token: None 2024-04-16T20:07:16.924 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.924 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ffe2d7450e4a81-00661eda74-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.927 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:16.928 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.928 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.928 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.928 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:16.928 DEBUG:boto:Data: 2024-04-16T20:07:16.928 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.928 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.928 DEBUG:boto:Params: {} 2024-04-16T20:07:16.928 DEBUG:boto:Token: None 2024-04-16T20:07:16.929 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74 2024-04-16T20:07:16.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGD+XEqR9oDWFF8kgWWCyOiEoPA= 2024-04-16T20:07:16.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGD+XEqR9oDWFF8kgWWCyOiEoPA=', 'Content-Length': '0'} 2024-04-16T20:07:16.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b6b6c4c65b0fa6-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.931 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.931 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.931 DEBUG:boto:Method: HEAD 2024-04-16T20:07:16.931 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:16.931 DEBUG:boto:Data: 2024-04-16T20:07:16.932 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.932 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.932 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.932 DEBUG:boto:Params: {} 2024-04-16T20:07:16.932 DEBUG:boto:Token: None 2024-04-16T20:07:16.932 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74 2024-04-16T20:07:16.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGD+XEqR9oDWFF8kgWWCyOiEoPA= 2024-04-16T20:07:16.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGD+XEqR9oDWFF8kgWWCyOiEoPA=', 'Content-Length': '0'} 2024-04-16T20:07:16.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ddc67aeeaf07e1-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:16.935 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.935 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.935 DEBUG:boto:Method: GET 2024-04-16T20:07:16.935 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:16.935 DEBUG:boto:Data: 2024-04-16T20:07:16.935 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.935 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.935 DEBUG:boto:Params: {} 2024-04-16T20:07:16.935 DEBUG:boto:Token: None 2024-04-16T20:07:16.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74 2024-04-16T20:07:16.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fp8Y2XUhN2wVG/SRiZua8eaaBfQ= 2024-04-16T20:07:16.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fp8Y2XUhN2wVG/SRiZua8eaaBfQ=', 'Content-Length': '0'} 2024-04-16T20:07:16.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:06:04 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e366fa896dfc41fd-00661eda74-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.978 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:16.978 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:16.978 DEBUG:boto:Method: GET 2024-04-16T20:07:16.978 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:16.979 DEBUG:boto:Data: 2024-04-16T20:07:16.979 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.979 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:16.979 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.979 DEBUG:boto:Params: {} 2024-04-16T20:07:16.979 DEBUG:boto:Token: None 2024-04-16T20:07:16.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/obj74 2024-04-16T20:07:16.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fp8Y2XUhN2wVG/SRiZua8eaaBfQ= 2024-04-16T20:07:16.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fp8Y2XUhN2wVG/SRiZua8eaaBfQ=', 'Content-Length': '0'} 2024-04-16T20:07:16.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fd4904eb980812-00661eda74-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:16.981 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:16.982 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:16.982 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.982 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.982 DEBUG:boto:Method: GET 2024-04-16T20:07:16.982 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:16.982 DEBUG:boto:Data: 2024-04-16T20:07:16.982 DEBUG:boto:Headers: {} 2024-04-16T20:07:16.982 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:16.982 DEBUG:boto:Port: 8000 2024-04-16T20:07:16.982 DEBUG:boto:Params: {} 2024-04-16T20:07:16.983 DEBUG:boto:Token: None 2024-04-16T20:07:16.983 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:16 GMT /yrjefh-52/ 2024-04-16T20:07:16.983 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA= 2024-04-16T20:07:16.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:16 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z/Y2keW4oww5h7g3UUH3/ZFOIiA=', 'Content-Length': '0'} 2024-04-16T20:07:16.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90f3c2aaa33de62-00661eda74-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.026 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.027 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.027 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.027 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:17.027 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:17.027 DEBUG:boto:Method: GET 2024-04-16T20:07:17.027 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:17.027 DEBUG:boto:Data: 2024-04-16T20:07:17.027 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.027 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.027 DEBUG:boto:Params: {} 2024-04-16T20:07:17.028 DEBUG:boto:Token: None 2024-04-16T20:07:17.028 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038f11ff3d90f602-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.031 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:17.032 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.032 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.032 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.032 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.032 DEBUG:boto:Method: GET 2024-04-16T20:07:17.032 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.032 DEBUG:boto:Data: 2024-04-16T20:07:17.032 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.032 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.032 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.033 DEBUG:boto:Params: {} 2024-04-16T20:07:17.033 DEBUG:boto:Token: None 2024-04-16T20:07:17.033 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.033 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E= 2024-04-16T20:07:17.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E=', 'Content-Length': '0'} 2024-04-16T20:07:17.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b01af6dab6a1ca-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.078 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.078 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.079 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.079 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.079 DEBUG:boto:Method: GET 2024-04-16T20:07:17.079 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.079 DEBUG:boto:Data: 2024-04-16T20:07:17.079 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.079 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.079 DEBUG:boto:Params: {} 2024-04-16T20:07:17.079 DEBUG:boto:Token: None 2024-04-16T20:07:17.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E= 2024-04-16T20:07:17.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E=', 'Content-Length': '0'} 2024-04-16T20:07:17.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb010bb1974c4476-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.082 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.082 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.082 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.082 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.082 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.082 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.082 DEBUG:boto:Data: 2024-04-16T20:07:17.082 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.083 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.083 DEBUG:boto:Params: {} 2024-04-16T20:07:17.083 DEBUG:boto:Token: None 2024-04-16T20:07:17.083 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.083 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dxSr5NRplOVjVQeOJpmGO0CKSEQ= 2024-04-16T20:07:17.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dxSr5NRplOVjVQeOJpmGO0CKSEQ=', 'Content-Length': '0'} 2024-04-16T20:07:17.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82d6bf43b62c068-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.085 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.085 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.085 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.085 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.085 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.085 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.086 DEBUG:boto:Data: 2024-04-16T20:07:17.086 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.086 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.086 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.086 DEBUG:boto:Params: {} 2024-04-16T20:07:17.086 DEBUG:boto:Token: None 2024-04-16T20:07:17.086 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.086 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dxSr5NRplOVjVQeOJpmGO0CKSEQ= 2024-04-16T20:07:17.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dxSr5NRplOVjVQeOJpmGO0CKSEQ=', 'Content-Length': '0'} 2024-04-16T20:07:17.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6ab3d02edf216d-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:17.088 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.089 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.089 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.089 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.089 DEBUG:boto:Method: GET 2024-04-16T20:07:17.089 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.089 DEBUG:boto:Data: 2024-04-16T20:07:17.089 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.089 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.089 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.089 DEBUG:boto:Params: {} 2024-04-16T20:07:17.089 DEBUG:boto:Token: None 2024-04-16T20:07:17.090 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E= 2024-04-16T20:07:17.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E=', 'Content-Length': '0'} 2024-04-16T20:07:17.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d4ad20c214b951-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.134 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.134 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.135 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.135 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.135 DEBUG:boto:Method: GET 2024-04-16T20:07:17.135 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.135 DEBUG:boto:Data: 2024-04-16T20:07:17.135 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.135 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.135 DEBUG:boto:Params: {} 2024-04-16T20:07:17.135 DEBUG:boto:Token: None 2024-04-16T20:07:17.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:17.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E= 2024-04-16T20:07:17.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zZxIMnAQFeF613b/rm8guxDDN5E=', 'Content-Length': '0'} 2024-04-16T20:07:17.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009379c7eef2857ae6-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.138 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.138 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.138 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.138 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.138 DEBUG:boto:Method: GET 2024-04-16T20:07:17.138 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.138 DEBUG:boto:Data: 2024-04-16T20:07:17.138 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.138 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.138 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.138 DEBUG:boto:Params: {} 2024-04-16T20:07:17.138 DEBUG:boto:Token: None 2024-04-16T20:07:17.139 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b400d00a72b4485c-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.186 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.187 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.187 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.187 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.187 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.187 DEBUG:boto:Method: GET 2024-04-16T20:07:17.187 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.187 DEBUG:boto:Data: 2024-04-16T20:07:17.187 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.187 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.187 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.188 DEBUG:boto:Params: {} 2024-04-16T20:07:17.188 DEBUG:boto:Token: None 2024-04-16T20:07:17.188 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001934681b9a6c85f3-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.192 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.192 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.192 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.192 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.192 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:17.192 DEBUG:boto:Data: 2024-04-16T20:07:17.192 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.192 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.192 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.192 DEBUG:boto:Params: {} 2024-04-16T20:07:17.193 DEBUG:boto:Token: None 2024-04-16T20:07:17.193 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76 2024-04-16T20:07:17.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lcT6qIvEUA7O51XwkCu/Ou75oeE= 2024-04-16T20:07:17.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcT6qIvEUA7O51XwkCu/Ou75oeE=', 'Content-Length': '0'} 2024-04-16T20:07:17.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d1798b89c24d89-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.195 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.195 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.195 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.195 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:17.195 DEBUG:boto:Data: 2024-04-16T20:07:17.195 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.195 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.195 DEBUG:boto:Params: {} 2024-04-16T20:07:17.196 DEBUG:boto:Token: None 2024-04-16T20:07:17.196 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76 2024-04-16T20:07:17.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lcT6qIvEUA7O51XwkCu/Ou75oeE= 2024-04-16T20:07:17.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcT6qIvEUA7O51XwkCu/Ou75oeE=', 'Content-Length': '0'} 2024-04-16T20:07:17.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8c6b0a71611448-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:17.198 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.198 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.198 DEBUG:boto:Method: GET 2024-04-16T20:07:17.198 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:17.198 DEBUG:boto:Data: 2024-04-16T20:07:17.199 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.199 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.199 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.199 DEBUG:boto:Params: {} 2024-04-16T20:07:17.199 DEBUG:boto:Token: None 2024-04-16T20:07:17.199 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76 2024-04-16T20:07:17.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9uFc5kekrOF7EDvWo3Lzdb7whQE= 2024-04-16T20:07:17.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9uFc5kekrOF7EDvWo3Lzdb7whQE=', 'Content-Length': '0'} 2024-04-16T20:07:17.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:01:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92bf399845fc897-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.242 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:17.242 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:17.243 DEBUG:boto:Method: GET 2024-04-16T20:07:17.243 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:17.243 DEBUG:boto:Data: 2024-04-16T20:07:17.243 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.243 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.243 DEBUG:boto:Params: {} 2024-04-16T20:07:17.243 DEBUG:boto:Token: None 2024-04-16T20:07:17.243 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj76 2024-04-16T20:07:17.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9uFc5kekrOF7EDvWo3Lzdb7whQE= 2024-04-16T20:07:17.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9uFc5kekrOF7EDvWo3Lzdb7whQE=', 'Content-Length': '0'} 2024-04-16T20:07:17.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997d9ca6a57b2642-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.246 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.246 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.246 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.246 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.246 DEBUG:boto:Method: GET 2024-04-16T20:07:17.246 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.246 DEBUG:boto:Data: 2024-04-16T20:07:17.246 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.246 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.247 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.247 DEBUG:boto:Params: {} 2024-04-16T20:07:17.247 DEBUG:boto:Token: None 2024-04-16T20:07:17.247 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.247 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac65280318b9d7e3-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.290 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.291 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.291 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.291 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.291 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.291 DEBUG:boto:Method: GET 2024-04-16T20:07:17.291 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:17.291 DEBUG:boto:Data: 2024-04-16T20:07:17.291 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.291 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.291 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.292 DEBUG:boto:Params: {} 2024-04-16T20:07:17.292 DEBUG:boto:Token: None 2024-04-16T20:07:17.292 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477dcb2d89565394-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.296 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:17.296 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.296 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.296 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.296 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.296 DEBUG:boto:Method: GET 2024-04-16T20:07:17.297 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.297 DEBUG:boto:Data: 2024-04-16T20:07:17.297 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.297 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.297 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.297 DEBUG:boto:Params: {} 2024-04-16T20:07:17.297 DEBUG:boto:Token: None 2024-04-16T20:07:17.297 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.297 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU= 2024-04-16T20:07:17.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU=', 'Content-Length': '0'} 2024-04-16T20:07:17.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d00a10274939417-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.342 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.343 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.343 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.343 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.343 DEBUG:boto:Method: GET 2024-04-16T20:07:17.343 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.343 DEBUG:boto:Data: 2024-04-16T20:07:17.343 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.343 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.343 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.343 DEBUG:boto:Params: {} 2024-04-16T20:07:17.344 DEBUG:boto:Token: None 2024-04-16T20:07:17.344 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.344 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU= 2024-04-16T20:07:17.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU=', 'Content-Length': '0'} 2024-04-16T20:07:17.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed546cc4ce3ed8c-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.346 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.346 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.346 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.347 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.347 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.347 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.347 DEBUG:boto:Data: 2024-04-16T20:07:17.347 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.347 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.347 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.347 DEBUG:boto:Params: {} 2024-04-16T20:07:17.347 DEBUG:boto:Token: None 2024-04-16T20:07:17.347 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.348 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YN+jk274WGrIsYRmA/kEkhBWy/k= 2024-04-16T20:07:17.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YN+jk274WGrIsYRmA/kEkhBWy/k=', 'Content-Length': '0'} 2024-04-16T20:07:17.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9995a0c9f6c80eb-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.350 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.350 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.350 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.350 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.350 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.350 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.350 DEBUG:boto:Data: 2024-04-16T20:07:17.351 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.351 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.351 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.351 DEBUG:boto:Params: {} 2024-04-16T20:07:17.351 DEBUG:boto:Token: None 2024-04-16T20:07:17.351 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.351 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YN+jk274WGrIsYRmA/kEkhBWy/k= 2024-04-16T20:07:17.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YN+jk274WGrIsYRmA/kEkhBWy/k=', 'Content-Length': '0'} 2024-04-16T20:07:17.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fafc0e6b8bb097-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:17.353 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.353 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.353 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.354 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.354 DEBUG:boto:Method: GET 2024-04-16T20:07:17.354 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.354 DEBUG:boto:Data: 2024-04-16T20:07:17.354 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.354 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.354 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.354 DEBUG:boto:Params: {} 2024-04-16T20:07:17.354 DEBUG:boto:Token: None 2024-04-16T20:07:17.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.355 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU= 2024-04-16T20:07:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU=', 'Content-Length': '0'} 2024-04-16T20:07:17.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba483ff9d1a23888-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.398 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.399 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.399 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.399 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.399 DEBUG:boto:Method: GET 2024-04-16T20:07:17.399 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.399 DEBUG:boto:Data: 2024-04-16T20:07:17.399 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.399 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.399 DEBUG:boto:Params: {} 2024-04-16T20:07:17.400 DEBUG:boto:Token: None 2024-04-16T20:07:17.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:17.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU= 2024-04-16T20:07:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MTcF/0tBlh27duNKTOehsamCHFU=', 'Content-Length': '0'} 2024-04-16T20:07:17.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac9b32ff501c630-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.402 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.403 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.403 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.403 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.403 DEBUG:boto:Method: GET 2024-04-16T20:07:17.403 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.403 DEBUG:boto:Data: 2024-04-16T20:07:17.403 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.403 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.403 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.403 DEBUG:boto:Params: {} 2024-04-16T20:07:17.404 DEBUG:boto:Token: None 2024-04-16T20:07:17.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269c6969b3281fe5-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.450 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.451 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.451 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.451 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.451 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.451 DEBUG:boto:Method: GET 2024-04-16T20:07:17.451 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.451 DEBUG:boto:Data: 2024-04-16T20:07:17.451 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.451 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.451 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.452 DEBUG:boto:Params: {} 2024-04-16T20:07:17.452 DEBUG:boto:Token: None 2024-04-16T20:07:17.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a31e5e41b2334c-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.456 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.456 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.456 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.457 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.457 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:17.457 DEBUG:boto:Data: 2024-04-16T20:07:17.457 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.457 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.457 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.457 DEBUG:boto:Params: {} 2024-04-16T20:07:17.457 DEBUG:boto:Token: None 2024-04-16T20:07:17.457 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85 2024-04-16T20:07:17.458 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2gepx6jbJBoyP1S4KCmux0KFRrY= 2024-04-16T20:07:17.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2gepx6jbJBoyP1S4KCmux0KFRrY=', 'Content-Length': '0'} 2024-04-16T20:07:17.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e842213e7a2457f0-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.460 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.460 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.460 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.460 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:17.461 DEBUG:boto:Data: 2024-04-16T20:07:17.461 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.461 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.461 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.461 DEBUG:boto:Params: {} 2024-04-16T20:07:17.461 DEBUG:boto:Token: None 2024-04-16T20:07:17.461 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85 2024-04-16T20:07:17.461 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2gepx6jbJBoyP1S4KCmux0KFRrY= 2024-04-16T20:07:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2gepx6jbJBoyP1S4KCmux0KFRrY=', 'Content-Length': '0'} 2024-04-16T20:07:17.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bbd30f434e5930-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:17.464 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.464 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.464 DEBUG:boto:Method: GET 2024-04-16T20:07:17.464 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:17.464 DEBUG:boto:Data: 2024-04-16T20:07:17.464 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.465 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.465 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.465 DEBUG:boto:Params: {} 2024-04-16T20:07:17.465 DEBUG:boto:Token: None 2024-04-16T20:07:17.465 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85 2024-04-16T20:07:17.465 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WiNWS8LajfBXby91rAFbleHIdIs= 2024-04-16T20:07:17.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WiNWS8LajfBXby91rAFbleHIdIs=', 'Content-Length': '0'} 2024-04-16T20:07:17.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1421cbff1f96a8-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.510 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:17.510 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:17.511 DEBUG:boto:Method: GET 2024-04-16T20:07:17.511 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:17.511 DEBUG:boto:Data: 2024-04-16T20:07:17.511 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.511 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.511 DEBUG:boto:Params: {} 2024-04-16T20:07:17.511 DEBUG:boto:Token: None 2024-04-16T20:07:17.512 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj85 2024-04-16T20:07:17.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WiNWS8LajfBXby91rAFbleHIdIs= 2024-04-16T20:07:17.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WiNWS8LajfBXby91rAFbleHIdIs=', 'Content-Length': '0'} 2024-04-16T20:07:17.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e39c62e0d237c5-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.514 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.514 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.515 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.515 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.515 DEBUG:boto:Method: GET 2024-04-16T20:07:17.515 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.515 DEBUG:boto:Data: 2024-04-16T20:07:17.515 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.515 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.515 DEBUG:boto:Params: {} 2024-04-16T20:07:17.516 DEBUG:boto:Token: None 2024-04-16T20:07:17.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ce55b671838a54-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.519 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.519 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.519 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.519 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.520 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.520 DEBUG:boto:Method: GET 2024-04-16T20:07:17.520 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:17.520 DEBUG:boto:Data: 2024-04-16T20:07:17.520 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.520 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.520 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.520 DEBUG:boto:Params: {} 2024-04-16T20:07:17.520 DEBUG:boto:Token: None 2024-04-16T20:07:17.521 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb67b4901d63a539-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.525 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:17.525 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.525 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.525 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.525 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.526 DEBUG:boto:Method: GET 2024-04-16T20:07:17.526 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.526 DEBUG:boto:Data: 2024-04-16T20:07:17.526 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.526 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.526 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.526 DEBUG:boto:Params: {} 2024-04-16T20:07:17.526 DEBUG:boto:Token: None 2024-04-16T20:07:17.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg= 2024-04-16T20:07:17.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg=', 'Content-Length': '0'} 2024-04-16T20:07:17.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c595b9283b0a58b8-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.570 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.570 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.571 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.571 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.571 DEBUG:boto:Method: GET 2024-04-16T20:07:17.571 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.571 DEBUG:boto:Data: 2024-04-16T20:07:17.571 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.571 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.571 DEBUG:boto:Params: {} 2024-04-16T20:07:17.572 DEBUG:boto:Token: None 2024-04-16T20:07:17.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg= 2024-04-16T20:07:17.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg=', 'Content-Length': '0'} 2024-04-16T20:07:17.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcbf7b1e8b5d011c-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.575 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.575 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.575 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.575 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.575 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.575 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.575 DEBUG:boto:Data: 2024-04-16T20:07:17.576 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.576 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.576 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.576 DEBUG:boto:Params: {} 2024-04-16T20:07:17.576 DEBUG:boto:Token: None 2024-04-16T20:07:17.576 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hyIrfgulNTOl5S2HhDXGkEXV/z0= 2024-04-16T20:07:17.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hyIrfgulNTOl5S2HhDXGkEXV/z0=', 'Content-Length': '0'} 2024-04-16T20:07:17.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c0e9a0e152fea1-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.579 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.579 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.579 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.579 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.579 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.579 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.579 DEBUG:boto:Data: 2024-04-16T20:07:17.580 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.580 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.580 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.580 DEBUG:boto:Params: {} 2024-04-16T20:07:17.580 DEBUG:boto:Token: None 2024-04-16T20:07:17.580 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hyIrfgulNTOl5S2HhDXGkEXV/z0= 2024-04-16T20:07:17.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hyIrfgulNTOl5S2HhDXGkEXV/z0=', 'Content-Length': '0'} 2024-04-16T20:07:17.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad178b1894fe7d51-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:17.583 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.583 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.583 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.583 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.584 DEBUG:boto:Method: GET 2024-04-16T20:07:17.584 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.584 DEBUG:boto:Data: 2024-04-16T20:07:17.584 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.584 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.584 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.584 DEBUG:boto:Params: {} 2024-04-16T20:07:17.584 DEBUG:boto:Token: None 2024-04-16T20:07:17.585 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg= 2024-04-16T20:07:17.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg=', 'Content-Length': '0'} 2024-04-16T20:07:17.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22a0eaac5ad267e-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.630 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.630 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.631 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.631 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.631 DEBUG:boto:Method: GET 2024-04-16T20:07:17.631 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.631 DEBUG:boto:Data: 2024-04-16T20:07:17.631 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.631 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.632 DEBUG:boto:Params: {} 2024-04-16T20:07:17.632 DEBUG:boto:Token: None 2024-04-16T20:07:17.632 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:17.632 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg= 2024-04-16T20:07:17.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nsMC+8fqplhyxoUtdiTh6AeJqRg=', 'Content-Length': '0'} 2024-04-16T20:07:17.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f63b17436c9218-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.635 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.635 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.635 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.635 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.635 DEBUG:boto:Method: GET 2024-04-16T20:07:17.636 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.636 DEBUG:boto:Data: 2024-04-16T20:07:17.636 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.636 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.636 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.636 DEBUG:boto:Params: {} 2024-04-16T20:07:17.636 DEBUG:boto:Token: None 2024-04-16T20:07:17.636 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981d0cf4d6098564-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.682 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.683 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.683 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.683 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.683 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.683 DEBUG:boto:Method: GET 2024-04-16T20:07:17.683 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.683 DEBUG:boto:Data: 2024-04-16T20:07:17.684 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.684 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.684 DEBUG:boto:Params: {} 2024-04-16T20:07:17.684 DEBUG:boto:Token: None 2024-04-16T20:07:17.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.684 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbfa9c0d344f2479-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.688 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.688 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.688 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.689 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.689 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:17.689 DEBUG:boto:Data: 2024-04-16T20:07:17.689 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.689 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.689 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.689 DEBUG:boto:Params: {} 2024-04-16T20:07:17.689 DEBUG:boto:Token: None 2024-04-16T20:07:17.689 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87 2024-04-16T20:07:17.690 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BwHMte88rx834rUo+Q7D5OE/7kE= 2024-04-16T20:07:17.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BwHMte88rx834rUo+Q7D5OE/7kE=', 'Content-Length': '0'} 2024-04-16T20:07:17.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e12908dd1582c32-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.692 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.692 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.692 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.692 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:17.693 DEBUG:boto:Data: 2024-04-16T20:07:17.693 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.693 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.693 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.693 DEBUG:boto:Params: {} 2024-04-16T20:07:17.693 DEBUG:boto:Token: None 2024-04-16T20:07:17.693 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87 2024-04-16T20:07:17.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BwHMte88rx834rUo+Q7D5OE/7kE= 2024-04-16T20:07:17.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BwHMte88rx834rUo+Q7D5OE/7kE=', 'Content-Length': '0'} 2024-04-16T20:07:17.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009647cafbada20689-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:17.696 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.696 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.696 DEBUG:boto:Method: GET 2024-04-16T20:07:17.696 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:17.696 DEBUG:boto:Data: 2024-04-16T20:07:17.696 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.696 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.697 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.697 DEBUG:boto:Params: {} 2024-04-16T20:07:17.697 DEBUG:boto:Token: None 2024-04-16T20:07:17.697 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87 2024-04-16T20:07:17.697 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pChOg4wSCX272wX2cIobKVbWtHY= 2024-04-16T20:07:17.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pChOg4wSCX272wX2cIobKVbWtHY=', 'Content-Length': '0'} 2024-04-16T20:07:17.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:38 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ae1cb14906896d-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.742 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:17.743 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:17.743 DEBUG:boto:Method: GET 2024-04-16T20:07:17.743 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:17.743 DEBUG:boto:Data: 2024-04-16T20:07:17.743 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.743 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.743 DEBUG:boto:Params: {} 2024-04-16T20:07:17.743 DEBUG:boto:Token: None 2024-04-16T20:07:17.744 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj87 2024-04-16T20:07:17.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pChOg4wSCX272wX2cIobKVbWtHY= 2024-04-16T20:07:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pChOg4wSCX272wX2cIobKVbWtHY=', 'Content-Length': '0'} 2024-04-16T20:07:17.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661a4844f695e3eb-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.746 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.747 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.747 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.747 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.747 DEBUG:boto:Method: GET 2024-04-16T20:07:17.747 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.747 DEBUG:boto:Data: 2024-04-16T20:07:17.747 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.747 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.747 DEBUG:boto:Params: {} 2024-04-16T20:07:17.748 DEBUG:boto:Token: None 2024-04-16T20:07:17.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a1ea9defde6ea0-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.751 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.751 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.752 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.752 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.752 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.752 DEBUG:boto:Method: GET 2024-04-16T20:07:17.752 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:17.752 DEBUG:boto:Data: 2024-04-16T20:07:17.752 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.752 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.752 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.752 DEBUG:boto:Params: {} 2024-04-16T20:07:17.753 DEBUG:boto:Token: None 2024-04-16T20:07:17.753 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.753 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf6239e016056de5-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.757 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:17.757 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.757 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.757 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.757 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.758 DEBUG:boto:Method: GET 2024-04-16T20:07:17.758 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.758 DEBUG:boto:Data: 2024-04-16T20:07:17.758 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.758 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.758 DEBUG:boto:Params: {} 2024-04-16T20:07:17.758 DEBUG:boto:Token: None 2024-04-16T20:07:17.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI= 2024-04-16T20:07:17.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI=', 'Content-Length': '0'} 2024-04-16T20:07:17.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000149090076b7dcf69-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.802 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.803 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.803 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.803 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.803 DEBUG:boto:Method: GET 2024-04-16T20:07:17.803 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.803 DEBUG:boto:Data: 2024-04-16T20:07:17.803 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.803 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.804 DEBUG:boto:Params: {} 2024-04-16T20:07:17.804 DEBUG:boto:Token: None 2024-04-16T20:07:17.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI= 2024-04-16T20:07:17.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI=', 'Content-Length': '0'} 2024-04-16T20:07:17.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8dc3c78a29ffc5-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.807 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.807 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.807 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.807 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.807 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.807 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.808 DEBUG:boto:Data: 2024-04-16T20:07:17.808 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.808 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.808 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.808 DEBUG:boto:Params: {} 2024-04-16T20:07:17.808 DEBUG:boto:Token: None 2024-04-16T20:07:17.808 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZL5uwrqQKSefOeIQ3VNasiSAjkw= 2024-04-16T20:07:17.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZL5uwrqQKSefOeIQ3VNasiSAjkw=', 'Content-Length': '0'} 2024-04-16T20:07:17.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1b77a6cc2ddc7d-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.811 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.811 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.811 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.811 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.811 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.811 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.812 DEBUG:boto:Data: 2024-04-16T20:07:17.812 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.812 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.812 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.812 DEBUG:boto:Params: {} 2024-04-16T20:07:17.812 DEBUG:boto:Token: None 2024-04-16T20:07:17.812 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZL5uwrqQKSefOeIQ3VNasiSAjkw= 2024-04-16T20:07:17.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZL5uwrqQKSefOeIQ3VNasiSAjkw=', 'Content-Length': '0'} 2024-04-16T20:07:17.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b92244240e643d-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:17.815 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.815 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.815 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.815 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.816 DEBUG:boto:Method: GET 2024-04-16T20:07:17.816 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.816 DEBUG:boto:Data: 2024-04-16T20:07:17.816 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.816 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.816 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.816 DEBUG:boto:Params: {} 2024-04-16T20:07:17.816 DEBUG:boto:Token: None 2024-04-16T20:07:17.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.817 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI= 2024-04-16T20:07:17.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI=', 'Content-Length': '0'} 2024-04-16T20:07:17.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d57efee5b6adec-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.862 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.863 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.863 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.863 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.863 DEBUG:boto:Method: GET 2024-04-16T20:07:17.863 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.863 DEBUG:boto:Data: 2024-04-16T20:07:17.863 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.864 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.864 DEBUG:boto:Params: {} 2024-04-16T20:07:17.864 DEBUG:boto:Token: None 2024-04-16T20:07:17.864 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:17.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI= 2024-04-16T20:07:17.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afQAooJ4WTgPbRnAoqw9GXLsdhI=', 'Content-Length': '0'} 2024-04-16T20:07:17.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0b1adce7f7ac08-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.867 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.867 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.867 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.867 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.867 DEBUG:boto:Method: GET 2024-04-16T20:07:17.868 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.868 DEBUG:boto:Data: 2024-04-16T20:07:17.868 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.868 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.868 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.868 DEBUG:boto:Params: {} 2024-04-16T20:07:17.868 DEBUG:boto:Token: None 2024-04-16T20:07:17.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.869 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000671b984deceba28d-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.914 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.915 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.915 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.915 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.915 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.915 DEBUG:boto:Method: GET 2024-04-16T20:07:17.915 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.916 DEBUG:boto:Data: 2024-04-16T20:07:17.916 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.916 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.916 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.916 DEBUG:boto:Params: {} 2024-04-16T20:07:17.916 DEBUG:boto:Token: None 2024-04-16T20:07:17.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19648083c4fef4e-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.920 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.921 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.921 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.921 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.921 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:17.921 DEBUG:boto:Data: 2024-04-16T20:07:17.922 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.922 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.922 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.922 DEBUG:boto:Params: {} 2024-04-16T20:07:17.922 DEBUG:boto:Token: None 2024-04-16T20:07:17.922 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92 2024-04-16T20:07:17.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZP0fjC1uoBPrMUcnYk6dcLT8AVc= 2024-04-16T20:07:17.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZP0fjC1uoBPrMUcnYk6dcLT8AVc=', 'Content-Length': '0'} 2024-04-16T20:07:17.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008177229463240b1e-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.925 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.925 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.925 DEBUG:boto:Method: HEAD 2024-04-16T20:07:17.926 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:17.926 DEBUG:boto:Data: 2024-04-16T20:07:17.926 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.926 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.926 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.926 DEBUG:boto:Params: {} 2024-04-16T20:07:17.926 DEBUG:boto:Token: None 2024-04-16T20:07:17.927 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92 2024-04-16T20:07:17.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZP0fjC1uoBPrMUcnYk6dcLT8AVc= 2024-04-16T20:07:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZP0fjC1uoBPrMUcnYk6dcLT8AVc=', 'Content-Length': '0'} 2024-04-16T20:07:17.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b99d3efb1ed7e86-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:17.929 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.930 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.930 DEBUG:boto:Method: GET 2024-04-16T20:07:17.930 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:17.930 DEBUG:boto:Data: 2024-04-16T20:07:17.930 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.930 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.930 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.930 DEBUG:boto:Params: {} 2024-04-16T20:07:17.931 DEBUG:boto:Token: None 2024-04-16T20:07:17.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92 2024-04-16T20:07:17.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+Q1k+j2I+JTk2QpTfM8n0EkcaY= 2024-04-16T20:07:17.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+Q1k+j2I+JTk2QpTfM8n0EkcaY=', 'Content-Length': '0'} 2024-04-16T20:07:17.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:03:22 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e34c510dc54161-00661eda75-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.974 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:17.975 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:17.975 DEBUG:boto:Method: GET 2024-04-16T20:07:17.975 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:17.975 DEBUG:boto:Data: 2024-04-16T20:07:17.975 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.975 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.976 DEBUG:boto:Params: {} 2024-04-16T20:07:17.976 DEBUG:boto:Token: None 2024-04-16T20:07:17.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj92 2024-04-16T20:07:17.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8+Q1k+j2I+JTk2QpTfM8n0EkcaY= 2024-04-16T20:07:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8+Q1k+j2I+JTk2QpTfM8n0EkcaY=', 'Content-Length': '0'} 2024-04-16T20:07:17.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84d42b7cf7c3b6c-00661eda75-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.979 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.979 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.979 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.979 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.979 DEBUG:boto:Method: GET 2024-04-16T20:07:17.980 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.980 DEBUG:boto:Data: 2024-04-16T20:07:17.980 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.980 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.980 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.980 DEBUG:boto:Params: {} 2024-04-16T20:07:17.980 DEBUG:boto:Token: None 2024-04-16T20:07:17.980 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e75a5bf01d40db2-00661eda75-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.984 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.984 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:17.984 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:17.985 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.985 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.985 DEBUG:boto:Method: GET 2024-04-16T20:07:17.985 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:17.985 DEBUG:boto:Data: 2024-04-16T20:07:17.985 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.985 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:17.985 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.986 DEBUG:boto:Params: {} 2024-04-16T20:07:17.986 DEBUG:boto:Token: None 2024-04-16T20:07:17.986 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/ 2024-04-16T20:07:17.986 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE= 2024-04-16T20:07:17.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:diL5JtwnWzo+YenxM2xGQ23UiWE=', 'Content-Length': '0'} 2024-04-16T20:07:17.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b26d963ba790a3e8-00661eda75-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:17.990 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:17.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:17.991 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:17.991 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:17.991 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:17.991 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:17.991 DEBUG:boto:Method: GET 2024-04-16T20:07:17.991 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:17.991 DEBUG:boto:Data: 2024-04-16T20:07:17.991 DEBUG:boto:Headers: {} 2024-04-16T20:07:17.992 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:17.992 DEBUG:boto:Port: 8000 2024-04-16T20:07:17.992 DEBUG:boto:Params: {} 2024-04-16T20:07:17.992 DEBUG:boto:Token: None 2024-04-16T20:07:17.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:17 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:17.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dmVoI8vjiDoAk9dYQw0KhDBC9a4= 2024-04-16T20:07:17.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:17 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dmVoI8vjiDoAk9dYQw0KhDBC9a4=', 'Content-Length': '0'} 2024-04-16T20:07:17.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7233c1637f24f2b-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.038 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.039 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.039 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.039 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.039 DEBUG:boto:Method: GET 2024-04-16T20:07:18.039 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.039 DEBUG:boto:Data: 2024-04-16T20:07:18.039 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.040 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.040 DEBUG:boto:Params: {} 2024-04-16T20:07:18.040 DEBUG:boto:Token: None 2024-04-16T20:07:18.040 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc= 2024-04-16T20:07:18.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc=', 'Content-Length': '0'} 2024-04-16T20:07:18.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005466f23713b58611-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.043 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.044 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.044 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.044 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.044 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.044 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.044 DEBUG:boto:Data: 2024-04-16T20:07:18.044 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.045 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.045 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.045 DEBUG:boto:Params: {} 2024-04-16T20:07:18.045 DEBUG:boto:Token: None 2024-04-16T20:07:18.045 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.045 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cC2QsOuciiAFakGV3MhU5e0xeLY= 2024-04-16T20:07:18.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cC2QsOuciiAFakGV3MhU5e0xeLY=', 'Content-Length': '0'} 2024-04-16T20:07:18.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0971d27bfb535b-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.048 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.048 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.049 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.049 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.049 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.049 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.049 DEBUG:boto:Data: 2024-04-16T20:07:18.049 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.049 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.049 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.050 DEBUG:boto:Params: {} 2024-04-16T20:07:18.050 DEBUG:boto:Token: None 2024-04-16T20:07:18.050 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.050 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cC2QsOuciiAFakGV3MhU5e0xeLY= 2024-04-16T20:07:18.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cC2QsOuciiAFakGV3MhU5e0xeLY=', 'Content-Length': '0'} 2024-04-16T20:07:18.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a6f60a302dfdaa-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:18.053 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.053 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.053 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.054 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.054 DEBUG:boto:Method: GET 2024-04-16T20:07:18.054 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.054 DEBUG:boto:Data: 2024-04-16T20:07:18.054 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.054 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.054 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.054 DEBUG:boto:Params: {} 2024-04-16T20:07:18.055 DEBUG:boto:Token: None 2024-04-16T20:07:18.055 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc= 2024-04-16T20:07:18.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc=', 'Content-Length': '0'} 2024-04-16T20:07:18.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab3962e1dc59f2b-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.098 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.099 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.099 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.099 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.099 DEBUG:boto:Method: GET 2024-04-16T20:07:18.099 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.099 DEBUG:boto:Data: 2024-04-16T20:07:18.099 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.100 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.100 DEBUG:boto:Params: {} 2024-04-16T20:07:18.100 DEBUG:boto:Token: None 2024-04-16T20:07:18.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:18.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc= 2024-04-16T20:07:18.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3/e+CshGEvNvccDDiOYFphkhYZc=', 'Content-Length': '0'} 2024-04-16T20:07:18.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38bbac4e6f9914d-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.103 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.103 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.104 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.104 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.104 DEBUG:boto:Method: GET 2024-04-16T20:07:18.104 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.104 DEBUG:boto:Data: 2024-04-16T20:07:18.104 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.104 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.104 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.104 DEBUG:boto:Params: {} 2024-04-16T20:07:18.105 DEBUG:boto:Token: None 2024-04-16T20:07:18.105 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7555a658bf1d143-00661eda76-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.150 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.151 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.151 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.151 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.151 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.152 DEBUG:boto:Method: GET 2024-04-16T20:07:18.152 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.152 DEBUG:boto:Data: 2024-04-16T20:07:18.152 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.152 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.152 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.152 DEBUG:boto:Params: {} 2024-04-16T20:07:18.152 DEBUG:boto:Token: None 2024-04-16T20:07:18.153 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d4e61b894d51d7-00661eda76-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.157 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.157 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.158 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.158 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.158 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:18.158 DEBUG:boto:Data: 2024-04-16T20:07:18.158 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.158 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.158 DEBUG:boto:Params: {} 2024-04-16T20:07:18.159 DEBUG:boto:Token: None 2024-04-16T20:07:18.159 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93 2024-04-16T20:07:18.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t/PEuckMTJx9FDJ9gUTFBzSpPoQ= 2024-04-16T20:07:18.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t/PEuckMTJx9FDJ9gUTFBzSpPoQ=', 'Content-Length': '0'} 2024-04-16T20:07:18.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eda3e7b591005f7-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.162 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.162 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.162 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.162 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:18.162 DEBUG:boto:Data: 2024-04-16T20:07:18.162 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.163 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.163 DEBUG:boto:Params: {} 2024-04-16T20:07:18.163 DEBUG:boto:Token: None 2024-04-16T20:07:18.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93 2024-04-16T20:07:18.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t/PEuckMTJx9FDJ9gUTFBzSpPoQ= 2024-04-16T20:07:18.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t/PEuckMTJx9FDJ9gUTFBzSpPoQ=', 'Content-Length': '0'} 2024-04-16T20:07:18.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035929f5003e8b75f-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:18.166 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.166 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.167 DEBUG:boto:Method: GET 2024-04-16T20:07:18.167 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:18.167 DEBUG:boto:Data: 2024-04-16T20:07:18.167 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.167 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.167 DEBUG:boto:Params: {} 2024-04-16T20:07:18.167 DEBUG:boto:Token: None 2024-04-16T20:07:18.168 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93 2024-04-16T20:07:18.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fou1jbw4F0yWTfKZfl5Sk43Eb4E= 2024-04-16T20:07:18.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fou1jbw4F0yWTfKZfl5Sk43Eb4E=', 'Content-Length': '0'} 2024-04-16T20:07:18.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:05:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006035326088299c52-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.210 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:18.211 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:18.211 DEBUG:boto:Method: GET 2024-04-16T20:07:18.211 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:18.211 DEBUG:boto:Data: 2024-04-16T20:07:18.211 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.211 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.211 DEBUG:boto:Params: {} 2024-04-16T20:07:18.211 DEBUG:boto:Token: None 2024-04-16T20:07:18.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj93 2024-04-16T20:07:18.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Fou1jbw4F0yWTfKZfl5Sk43Eb4E= 2024-04-16T20:07:18.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Fou1jbw4F0yWTfKZfl5Sk43Eb4E=', 'Content-Length': '0'} 2024-04-16T20:07:18.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d48f4d14c6065ef-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.215 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.215 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.215 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.215 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.215 DEBUG:boto:Method: GET 2024-04-16T20:07:18.215 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.215 DEBUG:boto:Data: 2024-04-16T20:07:18.215 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.216 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.216 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.216 DEBUG:boto:Params: {} 2024-04-16T20:07:18.216 DEBUG:boto:Token: None 2024-04-16T20:07:18.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ed80fa35a7df70-00661eda76-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.219 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.220 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.220 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.220 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.220 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.220 DEBUG:boto:Method: GET 2024-04-16T20:07:18.220 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:18.221 DEBUG:boto:Data: 2024-04-16T20:07:18.221 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.221 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.221 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.221 DEBUG:boto:Params: {} 2024-04-16T20:07:18.221 DEBUG:boto:Token: None 2024-04-16T20:07:18.221 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e078b511278fc24c-00661eda76-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.226 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:18.226 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.226 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.226 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.226 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.227 DEBUG:boto:Method: GET 2024-04-16T20:07:18.227 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.227 DEBUG:boto:Data: 2024-04-16T20:07:18.227 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.227 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.227 DEBUG:boto:Params: {} 2024-04-16T20:07:18.227 DEBUG:boto:Token: None 2024-04-16T20:07:18.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY= 2024-04-16T20:07:18.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY=', 'Content-Length': '0'} 2024-04-16T20:07:18.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8aa1f3f2f2df596-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.270 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.271 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.271 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.271 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.271 DEBUG:boto:Method: GET 2024-04-16T20:07:18.271 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.271 DEBUG:boto:Data: 2024-04-16T20:07:18.271 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.271 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.271 DEBUG:boto:Params: {} 2024-04-16T20:07:18.272 DEBUG:boto:Token: None 2024-04-16T20:07:18.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY= 2024-04-16T20:07:18.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY=', 'Content-Length': '0'} 2024-04-16T20:07:18.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048462464c8e74347-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.274 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.274 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.275 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.275 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.275 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.275 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.275 DEBUG:boto:Data: 2024-04-16T20:07:18.275 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.275 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.275 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.275 DEBUG:boto:Params: {} 2024-04-16T20:07:18.275 DEBUG:boto:Token: None 2024-04-16T20:07:18.276 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eQIVE7c4KpMnE/A6xSqLP2jD1bo= 2024-04-16T20:07:18.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eQIVE7c4KpMnE/A6xSqLP2jD1bo=', 'Content-Length': '0'} 2024-04-16T20:07:18.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b1dbfcc0ff8620-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.278 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.278 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.279 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.279 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.279 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.279 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.279 DEBUG:boto:Data: 2024-04-16T20:07:18.279 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.279 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.279 DEBUG:boto:Params: {} 2024-04-16T20:07:18.280 DEBUG:boto:Token: None 2024-04-16T20:07:18.280 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eQIVE7c4KpMnE/A6xSqLP2jD1bo= 2024-04-16T20:07:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eQIVE7c4KpMnE/A6xSqLP2jD1bo=', 'Content-Length': '0'} 2024-04-16T20:07:18.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008347d05cb263e270-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:18.283 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.283 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.283 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.283 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.283 DEBUG:boto:Method: GET 2024-04-16T20:07:18.283 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.283 DEBUG:boto:Data: 2024-04-16T20:07:18.283 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.283 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.284 DEBUG:boto:Params: {} 2024-04-16T20:07:18.284 DEBUG:boto:Token: None 2024-04-16T20:07:18.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY= 2024-04-16T20:07:18.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY=', 'Content-Length': '0'} 2024-04-16T20:07:18.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004655172e5f768c08-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.326 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.327 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.327 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.327 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.327 DEBUG:boto:Method: GET 2024-04-16T20:07:18.327 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.327 DEBUG:boto:Data: 2024-04-16T20:07:18.327 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.327 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.327 DEBUG:boto:Params: {} 2024-04-16T20:07:18.328 DEBUG:boto:Token: None 2024-04-16T20:07:18.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:18.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY= 2024-04-16T20:07:18.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNcKfeZvbmsxC1Ujfh39j2DziWY=', 'Content-Length': '0'} 2024-04-16T20:07:18.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c008926a6569e61a-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.331 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.331 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.331 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.331 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.331 DEBUG:boto:Method: GET 2024-04-16T20:07:18.331 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.331 DEBUG:boto:Data: 2024-04-16T20:07:18.331 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.332 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.332 DEBUG:boto:Params: {} 2024-04-16T20:07:18.332 DEBUG:boto:Token: None 2024-04-16T20:07:18.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e70f033f7b9219-00661eda76-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.335 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.335 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.335 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.335 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.336 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.336 DEBUG:boto:Method: GET 2024-04-16T20:07:18.336 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.336 DEBUG:boto:Data: 2024-04-16T20:07:18.336 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.336 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.336 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.336 DEBUG:boto:Params: {} 2024-04-16T20:07:18.336 DEBUG:boto:Token: None 2024-04-16T20:07:18.337 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000922a945a0a3b517c-00661eda76-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.382 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.383 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.383 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.383 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.383 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:18.383 DEBUG:boto:Data: 2024-04-16T20:07:18.384 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.384 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.384 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.384 DEBUG:boto:Params: {} 2024-04-16T20:07:18.384 DEBUG:boto:Token: None 2024-04-16T20:07:18.384 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95 2024-04-16T20:07:18.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwzxVnX9pUwT1RU+yky8X4vA0FI= 2024-04-16T20:07:18.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwzxVnX9pUwT1RU+yky8X4vA0FI=', 'Content-Length': '0'} 2024-04-16T20:07:18.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da34fcc8b0af5d6a-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.387 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.387 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.387 DEBUG:boto:Method: HEAD 2024-04-16T20:07:18.387 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:18.387 DEBUG:boto:Data: 2024-04-16T20:07:18.388 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.388 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.388 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.388 DEBUG:boto:Params: {} 2024-04-16T20:07:18.388 DEBUG:boto:Token: None 2024-04-16T20:07:18.388 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95 2024-04-16T20:07:18.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AwzxVnX9pUwT1RU+yky8X4vA0FI= 2024-04-16T20:07:18.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AwzxVnX9pUwT1RU+yky8X4vA0FI=', 'Content-Length': '0'} 2024-04-16T20:07:18.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca8b2881c1fd401-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:18.391 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.391 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.391 DEBUG:boto:Method: GET 2024-04-16T20:07:18.391 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:18.392 DEBUG:boto:Data: 2024-04-16T20:07:18.392 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.392 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.392 DEBUG:boto:Params: {} 2024-04-16T20:07:18.392 DEBUG:boto:Token: None 2024-04-16T20:07:18.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95 2024-04-16T20:07:18.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D8hFp3k1nJkw3DntqssPnr+/AlA= 2024-04-16T20:07:18.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D8hFp3k1nJkw3DntqssPnr+/AlA=', 'Content-Length': '0'} 2024-04-16T20:07:18.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:00:42 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6a4454e3b0927f-00661eda76-5480-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.439 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:18.439 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:18.439 DEBUG:boto:Method: GET 2024-04-16T20:07:18.439 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:18.439 DEBUG:boto:Data: 2024-04-16T20:07:18.439 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.440 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.440 DEBUG:boto:Params: {} 2024-04-16T20:07:18.440 DEBUG:boto:Token: None 2024-04-16T20:07:18.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/obj95 2024-04-16T20:07:18.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D8hFp3k1nJkw3DntqssPnr+/AlA= 2024-04-16T20:07:18.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D8hFp3k1nJkw3DntqssPnr+/AlA=', 'Content-Length': '0'} 2024-04-16T20:07:18.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc27b271b3ccb924-00661eda76-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.443 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.443 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.443 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.444 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.444 DEBUG:boto:Method: GET 2024-04-16T20:07:18.444 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.444 DEBUG:boto:Data: 2024-04-16T20:07:18.444 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.444 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:18.444 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.444 DEBUG:boto:Params: {} 2024-04-16T20:07:18.445 DEBUG:boto:Token: None 2024-04-16T20:07:18.445 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.445 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a658c6c183613040-00661eda76-5480-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.448 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.448 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:18.449 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:18.449 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.449 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.449 DEBUG:boto:Method: GET 2024-04-16T20:07:18.449 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:18.449 DEBUG:boto:Data: 2024-04-16T20:07:18.449 DEBUG:boto:Headers: {} 2024-04-16T20:07:18.449 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:18.450 DEBUG:boto:Port: 8000 2024-04-16T20:07:18.450 DEBUG:boto:Params: {} 2024-04-16T20:07:18.450 DEBUG:boto:Token: None 2024-04-16T20:07:18.450 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:18 GMT /yrjefh-52/ 2024-04-16T20:07:18.450 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I= 2024-04-16T20:07:18.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:18 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V6+oASTEgrih94FEf96sMLp873I=', 'Content-Length': '0'} 2024-04-16T20:07:18.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f49ea7f8e5d4c607-00661eda76-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:18.454 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:18.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:07:18.455 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:07:18.456 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:07:18.457 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:07:18.463+0000 7f2aaad7a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 138688) UID: 0 2024-04-16T20:07:18.457 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:07:18.463+0000 7f2ab0e32880 -1 shutting down 2024-04-16T20:07:23.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:07:24.555 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:07:24.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:24.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:24.556 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:24.616 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:24.616 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:24.694 INFO:teuthology.orchestra.run.smithi119.stdout:[{"name":"obj11","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.093523Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_y44i5_v7mW-wcLmrPdmTzSb_ulmdazw","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:06:00.845397Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_eR4tM5TQfC9U-ItAjqeS6d-xV50skLF","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.653967Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_VOJXty8r-bgPeEg2VtU3PhEPhmxoILr","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:03:33.314765Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Hv2recBpx3hjU1it6KCAB3BkF4OXYDZ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:04:43.764313Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uFYUasBrSUjKteB5lZzmdDdAdo6n5Gd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:03:33.364158Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_qaBIqQlt2pMVGzBPhvyQ_AiCM4h5rNt","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:06:00.856007Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_DbXVKVPoy2i12mZeboVcpu9P-F1vNLV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:04:43.774637Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_V-1GwvBYhO7rT8vc4PG4qaqV_W5spwd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:15.843569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_07LN-LUc3YveosiS6HFw6WxNy6C3YWq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:06:00.867226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_G6NLoGLZjkAjpOBYnvXBwF5Dv0HAhyk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:00:28.957569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_fds1GTR9p1QL8hpswA3ftZe9Ob4emDm","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.108841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_BbQexjEZ1hC7CKiuAnNTuZAdsKrk9e4","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:01:27.573996Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ZqwVlXsMA6sphOsrEVTj4V2-CW4vG3w","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.243147Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Ssir4wYWhHvJ_rn0fGlnSNV3CDk-blq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:04:43.697087Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_hyJ2Q697cVV6PylDnKikuRjflqASLJT","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.156310Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_idM2RXcf2hj_x-SVF8GCL9iVrtNLEOM","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:15.855500Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_hcs85JbHgYkqjIqxTS58rX31WfEkccD","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:06:00.786729Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uozJUHsZn8gXRdNx-dqn5Kf2UOabyLZ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:00:28.967396Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_FjiSwxK2pDsjKaga9OdztdPTm3pQoPT","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.167792Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_x99dKSsloEaSSSDvMivxvAPQhWARNCd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:01:27.584946Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_TWr4MYY33NoT0j9bwMKr17oEepozIG5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":9,"epoch":35},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:15.906077Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_c2RZdKIJffNcG1cAHk8gwCAk1MAaozi","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.179488Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_BpQdQf5tG1xbVUugrYykR1NWshJdRea","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.013592Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ambKvmKwCz1kGXk-9on4a1OOK30zs4m","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":9,"epoch":3},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.254425Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_zRZi62uLMr5q6ex5hHQXpuHWRlYr-ON","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:04:43.706887Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ZnxoJVJKCj5ZmAP_2BMt_T_mmfSorvF","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:58:15.918036Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_uqzvIsKYdlyk-hciTVYgYVyQZH8dNRf","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.595012Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_3IUMgJgaBGNpsN3ZK4MUrtrNmslerqV","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:06:00.797185Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_jz1DFh1Y1w1Ch8cbbsSbgBUjdgEWs5M","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:00:29.022199Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_keH5cTI0774hgxRbnN5ya5WeadjCZm_","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":9,"epoch":4},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:15.969668Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"__Io6EUvoygKlSX8v9PDhXZBqBgZdNSz","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.303990Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_k68NED0yqWluiLdmbzQla5AOOfLFYxd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:01:27.643268Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_FHfBiJaHstCzcIR4bJYx_065iRjk33k","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:04:43.717103Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_UXmsCMfggnAC85LAggXLY16iYzIipGv","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.069841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_ZpcGeREIUyGiTWzaEFKj0bsKccvruHT","flags":0,"pending_map":[],"versioned_epoch":0}] 2024-04-16T20:07:24.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:24.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:24.695 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj11 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:24.746 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:24.746 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:24.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:24.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:24.868 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj111 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:24.919 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:24.919 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.038 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj115 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.205 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj116 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.255 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.373 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj124 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.423 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.423 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.536 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj145 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.585 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.585 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.701 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj148 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.751 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.751 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:25.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:25.870 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj155 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.034 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj17 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.084 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.084 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.198 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj185 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.248 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.248 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.364 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj19 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.527 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj22 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.696 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj23 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.745 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.745 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:26.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:26.863 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj29 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:26.913 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:26.913 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.034 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj31 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.084 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.084 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.202 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj33 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.252 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.252 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.372 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj34 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.421 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.422 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.540 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj37 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.590 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.590 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.707 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj38 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.758 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.758 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:27.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:27.876 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj44 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:27.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:27.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.042 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj46 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.092 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.092 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.208 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj51 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.259 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.259 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.376 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj55 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.426 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.426 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.546 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj57 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.597 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.597 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.716 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj58 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.768 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.768 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:28.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:28.884 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj62 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.053 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj68 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.103 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.104 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:07:29.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.223 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj69 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.273 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.274 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.393 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj74 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.563 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj76 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.613 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.614 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.737 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj85 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:29.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:29.906 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj87 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.078 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj92 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.128 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.248 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj93 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.297 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.297 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.416 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --object obj95 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.583 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog trim --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.631 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.631 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.709 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:07:30.832 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:07:30.882 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:30.882 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:30.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:30.983+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-04-16T20:07:30.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:30.983+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:30.989+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: init 2024-04-16T20:07:30.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:30.998+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:30.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj116[0] 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj124[0] 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj145[0] 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj148[0] 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj17[0] 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.003+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.004+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj55[0] 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj57[0] 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.005+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.007+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.007+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.007+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.007+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj62[0] 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: success 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: success 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj68[0] 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.018+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: success 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj76[0] 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.022+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj85[0] 2024-04-16T20:07:31.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj87[0] 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: success 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.025+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: success 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj93[0] 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.028+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.031+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.035+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.035+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.036+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.036+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj11 2024-04-16T20:07:31.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.064+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.064+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: success 2024-04-16T20:07:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.068+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.068+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: success 2024-04-16T20:07:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.069+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: success 2024-04-16T20:07:31.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.071+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.071+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: success 2024-04-16T20:07:31.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.074+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.074+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: success 2024-04-16T20:07:31.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.074+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: backing out with sync_status=-5 2024-04-16T20:07:31.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.074+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: ERROR: full sync failed. error: -5 2024-04-16T20:07:31.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.076+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.076+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:31.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.081+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: full 2024-04-16T20:07:31.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.094+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj111[0] 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj116[0] 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj124[0] 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj145[0] 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj148[0] 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj155[0] 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj17[0] 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.139+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj22[0] 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj29[0] 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj34[0] 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.140+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: success 2024-04-16T20:07:31.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.143+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj55[0] 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj57[0] 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: success 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj62[0] 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: success 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.144+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: success 2024-04-16T20:07:31.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.145+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj85[0] 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj93[0] 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.146+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.147+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.148+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj19 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.149+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.151+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.153+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: success 2024-04-16T20:07:31.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.153+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj62 2024-04-16T20:07:31.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.154+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: backing out with sync_status=-5 2024-04-16T20:07:31.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.154+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: ERROR: full sync failed. error: -5 2024-04-16T20:07:31.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.158+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.158+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:31.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.203+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: full 2024-04-16T20:07:31.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.213+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj111[0] 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj124[0] 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj145[0] 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj148[0] 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.218+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj17[0] 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj22[0] 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj29[0] 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj34[0] 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.219+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.220+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj55[0] 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.221+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: success 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj57[0] 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: success 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.222+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj68[0] 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: success 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.223+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj76[0] 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: success 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.224+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: success 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.225+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.226+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.226+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.226+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.227+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.227+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.227+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.227+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.227+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj115 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.229+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: backing out with sync_status=-5 2024-04-16T20:07:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.229+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: ERROR: full sync failed. error: -5 2024-04-16T20:07:31.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.230+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.230+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:31.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.276+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: full 2024-04-16T20:07:31.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.284+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj116[0] 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj145[0] 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj155[0] 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj22[0] 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj29[0] 2024-04-16T20:07:31.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.289+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj34[0] 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.290+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.291+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj62[0] 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj68[0] 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: success 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.292+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: success 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj76[0] 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: success 2024-04-16T20:07:31.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj85[0] 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj87[0] 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.293+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj93[0] 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.294+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.295+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.295+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.295+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.296+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.296+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.296+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.296+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: success 2024-04-16T20:07:31.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.297+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: success 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.298+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.298+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj11 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.299+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: backing out with sync_status=-5 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.299+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: ERROR: full sync failed. error: -5 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.301+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.301+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:31.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.347+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: full 2024-04-16T20:07:31.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.359+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:31.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:31.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:31.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:31.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj111[0] 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj116[0] 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj124[0] 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj148[0] 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj155[0] 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.405+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj17[0] 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj22[0] 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj34[0] 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.406+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.407+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj55[0] 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.409+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: success 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.409+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.409+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.409+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj57[0] 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: success 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj62[0] 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: success 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj68[0] 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj76[0] 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: success 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.410+0000 7f92b3439800 0 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: ERROR: failed to sync object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: success 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj87[0] 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj93[0] 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: success 2024-04-16T20:07:31.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.411+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.412+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.412+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.412+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: a sync operation returned error 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.412+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.412+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.413+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.413+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.413+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.413+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.414+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.415+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.415+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.415+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.415+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: success 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.415+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj124 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.416+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: backing out with sync_status=-5 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.416+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: ERROR: full sync failed. error: -5 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.418+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.418+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: waiting on bucket lease 2024-04-16T20:07:31.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.463+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: status after acquiring the lock is: full 2024-04-16T20:07:31.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.475+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bucket for full sync 2024-04-16T20:07:31.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj11 2024-04-16T20:07:31.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj11 op=0 op_state=1 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj11[0] 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj111 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj111 op=0 op_state=1 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj111[0] 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj115 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj115 op=0 op_state=1 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj115[0] 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj116 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj116 op=0 op_state=1 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj116[0] 2024-04-16T20:07:31.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj124 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj124 op=0 op_state=1 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.480+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj124[0] 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj145 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj145 op=0 op_state=1 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj145[0] 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj148 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj148 op=0 op_state=1 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj148[0] 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj155 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj155 op=0 op_state=1 2024-04-16T20:07:31.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj155[0] 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj17 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj17 op=0 op_state=1 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj17[0] 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj185 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj185 op=0 op_state=1 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj185[0] 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj19 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj19 op=0 op_state=1 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj19[0] 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj22 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj22 op=0 op_state=1 2024-04-16T20:07:31.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj22[0] 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj23 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj23 op=0 op_state=1 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj23[0] 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj29 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj29 op=0 op_state=1 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj29[0] 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj31 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj31 op=0 op_state=1 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj31[0] 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj33 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj33 op=0 op_state=1 2024-04-16T20:07:31.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj33[0] 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj34 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj34 op=0 op_state=1 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj34[0] 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj37 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj37 op=0 op_state=1 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.481+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj37[0] 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj38 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj38 op=0 op_state=1 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj38[0] 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj44 2024-04-16T20:07:31.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj44 op=0 op_state=1 2024-04-16T20:07:31.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj44[0] 2024-04-16T20:07:31.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj46 2024-04-16T20:07:31.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj46 op=0 op_state=1 2024-04-16T20:07:31.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.482+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj46[0] 2024-04-16T20:07:31.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj148]: success 2024-04-16T20:07:31.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj51 2024-04-16T20:07:31.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj51 op=0 op_state=1 2024-04-16T20:07:31.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj51[0] 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj116]: success 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj55 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj55 op=0 op_state=1 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.486+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj55[0] 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj31]: success 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj57 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj57 op=0 op_state=1 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj57[0] 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj17]: success 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj58 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj58 op=0 op_state=1 2024-04-16T20:07:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj58[0] 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj155]: success 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj33]: success 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj62 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj62 op=0 op_state=1 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj62[0] 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj68 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj68 op=0 op_state=1 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.487+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj68[0] 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj37]: success 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj69 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj69 op=0 op_state=1 2024-04-16T20:07:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj69[0] 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj111]: success 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj38]: success 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj74 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj74 op=0 op_state=1 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj74[0] 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj76 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj76 op=0 op_state=1 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj76[0] 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj46]: success 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj85 2024-04-16T20:07:31.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj85 op=0 op_state=1 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj85[0] 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj29]: success 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj87 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj87 op=0 op_state=1 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj87[0] 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj124]: success 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj92 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj92 op=0 op_state=1 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj92[0] 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj145]: success 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj185]: success 2024-04-16T20:07:31.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj93 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj93 op=0 op_state=1 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj93[0] 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: [full sync] syncing object: yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0/obj95 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b=bucket sync single entry (source_zone=b482a2a9-ef36-488b-9666-ada656453328) b= log_entry=obj95 op=0 op_state=1 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: bucket sync: sync obj: b482a2a9-ef36-488b-9666-ada656453328/:yrjefh-52[b482a2a9-ef36-488b-9666-ada656453328.4812.21])/obj95[0] 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj11]: success 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj111 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj34]: success 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.488+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj22]: success 2024-04-16T20:07:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.489+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj115]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.489+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj23]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.489+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj19]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.489+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj44]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj58]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj51]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj74]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj62]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj76]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj57]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj69]: success 2024-04-16T20:07:31.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj68]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj55]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.491+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj76 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj95]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj93]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj87]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj92]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:entry[obj85]: success 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.492+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: updating marker oid=bucket.full-sync-status.b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21 marker=obj95 2024-04-16T20:07:31.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.494+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: set bucket state=incremental 2024-04-16T20:07:31.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.495+0000 7f92b3439800 5 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:full_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: bucket status objv={r=_3ZzgULhSRB8Cr51lNIC3ZfH:150,w=:0} 2024-04-16T20:07:31.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.498+0000 7f92b3439800 10 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-04-16T20:07:31.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.500+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.500+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.500+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.500+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: sync status for source bucket shard: 2 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:0]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:2]: sync status for source bucket shard: 2 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:3]: sync status for source bucket shard: 2 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:2]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:2]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:3]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:3]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:5]: sync status for source bucket shard: 2 2024-04-16T20:07:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:1]: sync status for source bucket shard: 2 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:5]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:5]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:1]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:1]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.501+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:4]: sync status for source bucket shard: 2 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:4]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:4]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:6]: sync status for source bucket shard: 2 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:6]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:6]: listing bilog for incremental sync; position=00000000002.106.6 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:8]: sync status for source bucket shard: 2 2024-04-16T20:07:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:8]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:8]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:9]: sync status for source bucket shard: 2 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:9]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:9]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:10]: sync status for source bucket shard: 2 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.502+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:10]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:10]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:11]: sync status for source bucket shard: 2 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:11]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:11]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:13]: sync status for source bucket shard: 2 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:13]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:13]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:14]: sync status for source bucket shard: 2 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:14]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:14]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:15]: sync status for source bucket shard: 2 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.503+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:15]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:15]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:16]: sync status for source bucket shard: 2 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:16]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:16]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:17]: sync status for source bucket shard: 2 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:17]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:17]: listing bilog for incremental sync; position=00000000002.71.6 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:18]: sync status for source bucket shard: 2 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:18]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:18]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:19]: sync status for source bucket shard: 2 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:19]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:19]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:20]: sync status for source bucket shard: 2 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:20]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:20]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.504+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:21]: sync status for source bucket shard: 2 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:21]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:21]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:22]: sync status for source bucket shard: 2 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:22]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:22]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:24]: sync status for source bucket shard: 2 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:24]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:24]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:23]: sync status for source bucket shard: 2 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:23]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:23]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:25]: sync status for source bucket shard: 2 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:25]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:25]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.505+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:26]: sync status for source bucket shard: 2 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:26]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:26]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:27]: sync status for source bucket shard: 2 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:27]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:27]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:28]: sync status for source bucket shard: 2 2024-04-16T20:07:31.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:28]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:28]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:29]: sync status for source bucket shard: 2 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:29]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:29]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:30]: sync status for source bucket shard: 2 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:30]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:30]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:31]: sync status for source bucket shard: 2 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.506+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:31]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:31]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-16T20:07:31.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:32]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:32]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:32]: listing bilog for incremental sync; position=00000000001.85.6 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:7]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:7]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:7]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:33]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:33]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:33]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:12]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:12]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:12]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:34]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:35]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:36]: sync status for source bucket shard: 2 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:34]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:34]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:35]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:35]: listing bilog for incremental sync; position= 2024-04-16T20:07:31.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:31.507+0000 7f92b3439800 20 RGW-SYNC:bucket[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21<-yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:36]:inc_sync[yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21:36]: listing bilog for incremental sync; position=00000000001.75.6 2024-04-16T20:07:31.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:31.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:31.575 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:31.631 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:31.631 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:31.712 INFO:teuthology.orchestra.run.smithi026.stdout:[{"name":"obj11","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:04.093523Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1929919587753571975","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:06:00.845397Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.3844875869094816186","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:01:27.653967Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.4780840649802295021","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:03:33.314765Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.13578391418927127174","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:04:43.764313Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8678218423142725742","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:03:33.364158Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.14709958525312512941","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:06:00.856007Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5705481589752884054","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:04:43.774637Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16921979892965732534","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:58:15.843569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.170429413413830159","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:06:00.867226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.3793954997632793341","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:00:28.957569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.203870009147420342","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:04.108841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.4650468223035672770","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:01:27.573996Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15860864558330967828","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:03:33.243147Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8224181419691523959","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:04:43.697087Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6687508779847208279","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:04.156310Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.8527669909641357350","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:15.855500Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.2981357739881398603","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:06:00.786729Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15756797176002321638","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:00:28.967396Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.17902394479872288710","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:58:04.167792Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16071490801473987996","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.584946Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.6205010274280506524","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T19:58:15.906077Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.13101923650835453284","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:04.179488Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.12767474094945203394","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.013592Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18097324844068399712","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:03:33.254425Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1234904070122404913","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:04:43.706887Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.11491986685814543467","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:15.918036Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.9436193684743669943","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.595012Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.12941693842363796125","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:06:00.797185Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.710837920862265629","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.022199Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18443535639312822343","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:15.969668Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16881784038201731381","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.303990Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15288315356500714376","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.643268Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.7345818070820704982","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:04:43.717103Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.15210773109488970858","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":9,"epoch":0},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:00:29.069841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.1945422874564821583","flags":0,"pending_map":[],"versioned_epoch":0}] 2024-04-16T20:07:31.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:31.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-52', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:31.713 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-52 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:31.838 INFO:teuthology.orchestra.run.smithi119.stdout:[{"name":"obj11","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:04.093523Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_8PbMv0G58L6Wtevw8YrPOJ8jCqya04s","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj111","instance":"","ver":{"pool":9,"epoch":13},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:06:00.845397Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_rQR-XxqYn6_m8PVolgZlc4bBrxvUGO2","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj115","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.653967Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_EIpkwcx70xbx_AWo_kAc5yQBeLIVwbc","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj116","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:03:33.314765Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_k6_exH5B9JoKlZ8rXUsEtVF9x1oym-4","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj124","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:04:43.764313Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Od5Q1kt5J7oWToQ6s63n9GfuBMF0w6N","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj145","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:03:33.364158Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_osvxAMndsL-tcFwdzBMJAV8kHNphxZg","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj148","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:06:00.856007Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Cv-x064N6kUElirlr_eyBMVFfKdHgyd","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj155","instance":"","ver":{"pool":9,"epoch":11},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:04:43.774637Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_3RVv-J8s_RJ2LosIbK9wK3Ud9OWVGcE","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj17","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:15.843569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_s2y1m9nHOPe_3G8eCU0dKEFQVrNiCCZ","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj185","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:06:00.867226Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_i0k9M4P7t8SiVoUGVg1xAEIid90qNB5","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj19","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:28.957569Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_RHwj_l-6vFecmhwHLrPqUj10LZlhXK8","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj22","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T19:58:04.108841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_tzAjs4TRCY74UsjKQJfgkIFjLgoJHsn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj23","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":3,"mtime":"2024-04-16T20:01:27.573996Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_VsEVnhcgtv4hsUXZIjB3x6pWeVs2vY2","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj29","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.243147Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Gtz8R2_Pw2Hx0h3-XCQOZ-GH9-O4Z2a","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj31","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:04:43.697087Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_agih76fLpIi0h_1vPXas0Sg-xFi_aaW","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj33","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T19:58:04.156310Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_rotb0OMnJQlJwarpAu0cIbg5tT7ejm3","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj34","instance":"","ver":{"pool":9,"epoch":13},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:15.855500Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_emAxkGnpdcEHsjfw3HAiYdqBxWM3QiM","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj37","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:06:00.786729Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_qSA0EUCMYi0WQsqNAg7UyVDtWArR99l","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj38","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:00:28.967396Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_hVqJN9L0MsmhLwcH26kModgBQFArS3t","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj44","instance":"","ver":{"pool":9,"epoch":7},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.167792Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_aMiPW7GQxEd47d9eM2YkUrHuoj_evzL","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj46","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.584946Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_wEDfEKUv4HRD2z-BUFe8Y_jrkJRUAuN","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj51","instance":"","ver":{"pool":9,"epoch":37},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:15.906077Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_yymGTonGu8HiZQpGnD9hglzQNKxurja","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj55","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:04.179488Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_cC2FVIUIWyj0-mx4rM6k6YAk2ygnWZB","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj57","instance":"","ver":{"pool":9,"epoch":11},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.013592Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_pEPot36JxsPUzn2ODPqDtXO8a19pafN","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj58","instance":"","ver":{"pool":9,"epoch":5},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:03:33.254425Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_gWEMVP8s4cEHRGpRfy2Fx2F1s41zp3o","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj62","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:04:43.706887Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_0ULIijCCr-AdJq-BK0NETY9mprSvOVq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj68","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:15.918036Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_NX9zuMWIXpmBmVrmVvgvwr37q4KzmoD","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj69","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:01:27.595012Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_Mw87gcPl1xjlCmpQMju596Na0SvqZ_B","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj74","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:06:00.797185Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_fvq-eXbjlMtBVsVz9mtO3evpO4ZmvPq","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj76","instance":"","ver":{"pool":9,"epoch":8},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:00:29.022199Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_-irvTqvndjuohEup8qLjokWDbYtQsuk","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj85","instance":"","ver":{"pool":9,"epoch":6},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T19:58:15.969668Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_WDtps9kVWF0-b-4tk1EksHXA6h6gvuS","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj87","instance":"","ver":{"pool":9,"epoch":9},"locator":"","exists":true,"meta":{"category":1,"size":5,"mtime":"2024-04-16T20:03:33.303990Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_heJrE8LzGNNukkc0NMSMuq8STjTRY2B","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj92","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:01:27.643268Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_EpPnbcO-jb_IxowOw41cPhu85v2BCNn","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj93","instance":"","ver":{"pool":9,"epoch":10},"locator":"","exists":true,"meta":{"category":1,"size":16,"mtime":"2024-04-16T20:04:43.717103Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_IQ0jNnHeu7FsLnBcOo02r3tvde6IsMB","flags":0,"pending_map":[],"versioned_epoch":0},{"name":"obj95","instance":"","ver":{"pool":9,"epoch":12},"locator":"","exists":true,"meta":{"category":1,"size":6,"mtime":"2024-04-16T20:00:29.069841Z","etag":"acbd18db4cc2f85cedef654fccc4a4d8","storage_class":"","owner":"RGW11111111111111111","owner_display_name":"","content_type":"application/octet-stream","accounted_size":3,"user_data":"","appendable":false},"tag":"_HaKjPSeinWgvPWxy0HNsRP0o1uETyu-","flags":0,"pending_map":[],"versioned_epoch":0}] 2024-04-16T20:07:31.838 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:07:31.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:07:31.840 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:07:31.841 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:07:31.860 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:07:31.860 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:07:31.860 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:07:31.860 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:07:32.029 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:07:32.037+0000 7fede9c13880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:07:32.861 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:32.880 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:07:32.880 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:07:32.880 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:07:32.881 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:07:32.881 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-52 2024-04-16T20:07:32.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:32.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:32.882 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:07:32.976 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:32.977 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:33.042+0000 7f5385333800 1 bucket sync caught up with source: 2024-04-16T20:07:33.035 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:33.035 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.106.6, , , , , , 00000000002.72.6, , , , , 00000000002.71.6, , , , , , , , , , , , , , , 00000000001.85.6, , , , 00000000001.75.6] 2024-04-16T20:07:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:33.042+0000 7f5385333800 0 bucket checkpoint complete 2024-04-16T20:07:33.056 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-52 2024-04-16T20:07:33.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:33.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-52', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:07:33.056 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-52 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:07:33.112 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:33.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:07:33.176+0000 7f3c80b2c800 1 bucket sync caught up with source: 2024-04-16T20:07:33.169 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000009.196.6, 00000000006.213.6, 00000000006.120.6, 00000000015.314.6, 00000000012.299.6, 00000000003.201.6, 00000000003.350.6, 00000000012.204.6, 00000000021.301.6, 00000000018.349.6] 2024-04-16T20:07:33.169 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000009.196.6, 00000000006.213.6, 00000000006.120.6, 00000000015.314.6, 00000000012.299.6, 00000000003.201.6, 00000000003.350.6, 00000000012.204.6, 00000000021.301.6, 00000000018.349.6] 2024-04-16T20:07:33.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:07:33.176+0000 7f3c80b2c800 0 bucket checkpoint complete 2024-04-16T20:07:33.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:07:33.190 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.190 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.190 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.191 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:07:33.191 DEBUG:boto:Data: 2024-04-16T20:07:33.191 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.191 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.191 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.191 DEBUG:boto:Params: {} 2024-04-16T20:07:33.191 DEBUG:boto:Token: None 2024-04-16T20:07:33.191 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/ 2024-04-16T20:07:33.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg= 2024-04-16T20:07:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg=', 'Content-Length': '0'} 2024-04-16T20:07:33.192 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-16T20:07:33.192 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:07:33.755 DEBUG:boto:Token: None 2024-04-16T20:07:33.755 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/ 2024-04-16T20:07:33.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg= 2024-04-16T20:07:33.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg=', 'Content-Length': '0'} 2024-04-16T20:07:33.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a138c7e5a8b6eb66-00661eda85-5958-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.763 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.764 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.764 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.764 DEBUG:boto:Path: /yrjefh-52/ 2024-04-16T20:07:33.764 DEBUG:boto:Data: 2024-04-16T20:07:33.764 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.764 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.764 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.764 DEBUG:boto:Params: {} 2024-04-16T20:07:33.764 DEBUG:boto:Token: None 2024-04-16T20:07:33.765 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/ 2024-04-16T20:07:33.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg= 2024-04-16T20:07:33.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ObiuVyRAkkj8wB/5Wbj6Gv+LtOg=', 'Content-Length': '0'} 2024-04-16T20:07:33.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca9d8bda301e5b55-00661eda85-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:07:33.768 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.768 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.768 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.769 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.769 DEBUG:boto:Method: GET 2024-04-16T20:07:33.769 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.769 DEBUG:boto:Data: 2024-04-16T20:07:33.769 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.769 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.769 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.769 DEBUG:boto:Params: {} 2024-04-16T20:07:33.769 DEBUG:boto:Token: None 2024-04-16T20:07:33.769 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/?versions 2024-04-16T20:07:33.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY= 2024-04-16T20:07:33.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY=', 'Content-Length': '0'} 2024-04-16T20:07:33.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb1fecbb6d62346c-00661eda85-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.773 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:07:33.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:07:33.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:07:33.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:07:33.777 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:07:33.777 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.778 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.778 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.778 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.778 DEBUG:boto:Method: GET 2024-04-16T20:07:33.778 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.778 DEBUG:boto:Data: 2024-04-16T20:07:33.778 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.778 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.778 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.778 DEBUG:boto:Params: {} 2024-04-16T20:07:33.778 DEBUG:boto:Token: None 2024-04-16T20:07:33.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/?versions 2024-04-16T20:07:33.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY= 2024-04-16T20:07:33.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY=', 'Content-Length': '0'} 2024-04-16T20:07:33.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cadeb58d1b4b0256-00661eda85-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.783 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:07:33.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:07:33.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:07:33.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:07:33.788 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.788 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.788 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.788 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.788 DEBUG:boto:Method: GET 2024-04-16T20:07:33.788 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.788 DEBUG:boto:Data: 2024-04-16T20:07:33.788 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.788 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.788 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.788 DEBUG:boto:Params: {} 2024-04-16T20:07:33.789 DEBUG:boto:Token: None 2024-04-16T20:07:33.789 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/?versions 2024-04-16T20:07:33.789 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY= 2024-04-16T20:07:33.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY=', 'Content-Length': '0'} 2024-04-16T20:07:33.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f76db140a8e99c-00661eda85-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.793 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.795 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.795 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.795 DEBUG:boto:path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.795 DEBUG:boto:auth_path=/yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.795 DEBUG:boto:Method: GET 2024-04-16T20:07:33.795 DEBUG:boto:Path: /yrjefh-52/?versions&max-keys=999 2024-04-16T20:07:33.795 DEBUG:boto:Data: 2024-04-16T20:07:33.795 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.795 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.795 DEBUG:boto:Params: {} 2024-04-16T20:07:33.796 DEBUG:boto:Token: None 2024-04-16T20:07:33.796 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/?versions 2024-04-16T20:07:33.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY= 2024-04-16T20:07:33.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X4JQKiz9kbCwi8vAHRkyLeuKwXY=', 'Content-Length': '0'} 2024-04-16T20:07:33.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea3384740fcf75f-00661eda85-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.800 DEBUG:boto:b'yrjefh-52999falseobj11nulltrue2024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:33.801 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.801 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.802 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.802 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.802 DEBUG:boto:Method: GET 2024-04-16T20:07:33.802 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.802 DEBUG:boto:Data: 2024-04-16T20:07:33.802 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.802 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.802 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.802 DEBUG:boto:Params: {} 2024-04-16T20:07:33.802 DEBUG:boto:Token: None 2024-04-16T20:07:33.803 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo= 2024-04-16T20:07:33.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo=', 'Content-Length': '0'} 2024-04-16T20:07:33.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9688f9299b549cf-00661eda85-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.807 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.807 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.807 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.807 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.807 DEBUG:boto:Method: GET 2024-04-16T20:07:33.807 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.807 DEBUG:boto:Data: 2024-04-16T20:07:33.807 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.807 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.807 DEBUG:boto:Params: {} 2024-04-16T20:07:33.808 DEBUG:boto:Token: None 2024-04-16T20:07:33.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo= 2024-04-16T20:07:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo=', 'Content-Length': '0'} 2024-04-16T20:07:33.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ca2ca27949588d-00661eda85-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.850 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.851 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.851 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.851 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.851 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.851 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.851 DEBUG:boto:Data: 2024-04-16T20:07:33.851 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.851 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.851 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.851 DEBUG:boto:Params: {} 2024-04-16T20:07:33.852 DEBUG:boto:Token: None 2024-04-16T20:07:33.852 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n2/SH24o47/oZar0lzr7PhU/kv4= 2024-04-16T20:07:33.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n2/SH24o47/oZar0lzr7PhU/kv4=', 'Content-Length': '0'} 2024-04-16T20:07:33.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82529411edce973-00661eda85-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.854 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.854 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.855 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.855 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.855 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.855 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.855 DEBUG:boto:Data: 2024-04-16T20:07:33.855 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.855 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.855 DEBUG:boto:Params: {} 2024-04-16T20:07:33.855 DEBUG:boto:Token: None 2024-04-16T20:07:33.856 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:n2/SH24o47/oZar0lzr7PhU/kv4= 2024-04-16T20:07:33.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n2/SH24o47/oZar0lzr7PhU/kv4=', 'Content-Length': '0'} 2024-04-16T20:07:33.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad67f8b0d4533539-00661eda85-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:33.858 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.858 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.858 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.858 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.858 DEBUG:boto:Method: GET 2024-04-16T20:07:33.858 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.858 DEBUG:boto:Data: 2024-04-16T20:07:33.859 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.859 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.859 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.859 DEBUG:boto:Params: {} 2024-04-16T20:07:33.859 DEBUG:boto:Token: None 2024-04-16T20:07:33.859 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo= 2024-04-16T20:07:33.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo=', 'Content-Length': '0'} 2024-04-16T20:07:33.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3da696ea153328-00661eda85-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.902 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.903 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.903 DEBUG:boto:path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.903 DEBUG:boto:auth_path=/yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.903 DEBUG:boto:Method: GET 2024-04-16T20:07:33.903 DEBUG:boto:Path: /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.903 DEBUG:boto:Data: 2024-04-16T20:07:33.903 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.903 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.903 DEBUG:boto:Params: {} 2024-04-16T20:07:33.903 DEBUG:boto:Token: None 2024-04-16T20:07:33.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11?versionId=null 2024-04-16T20:07:33.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo= 2024-04-16T20:07:33.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:azfBx0L6PK+cK/ImyMhLwA0CTjo=', 'Content-Length': '0'} 2024-04-16T20:07:33.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5166dd6b2c7bde-00661eda85-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.906 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.906 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.906 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.906 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.907 DEBUG:boto:Method: GET 2024-04-16T20:07:33.907 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.907 DEBUG:boto:Data: 2024-04-16T20:07:33.907 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.907 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.907 DEBUG:boto:Params: {} 2024-04-16T20:07:33.907 DEBUG:boto:Token: None 2024-04-16T20:07:33.907 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/ 2024-04-16T20:07:33.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+I7V9iNK6rwG8qjmgzS3qZyoUU= 2024-04-16T20:07:33.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+I7V9iNK6rwG8qjmgzS3qZyoUU=', 'Content-Length': '0'} 2024-04-16T20:07:33.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2ef447569488ef-00661eda85-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.950 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.951 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:33.951 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:33.951 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.951 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.951 DEBUG:boto:Method: GET 2024-04-16T20:07:33.951 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:33.951 DEBUG:boto:Data: 2024-04-16T20:07:33.952 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.952 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.952 DEBUG:boto:Params: {} 2024-04-16T20:07:33.952 DEBUG:boto:Token: None 2024-04-16T20:07:33.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/ 2024-04-16T20:07:33.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D+I7V9iNK6rwG8qjmgzS3qZyoUU= 2024-04-16T20:07:33.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D+I7V9iNK6rwG8qjmgzS3qZyoUU=', 'Content-Length': '0'} 2024-04-16T20:07:33.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e258e7d2ef30c4-00661eda85-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.956 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:33.956 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.956 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.957 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.957 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:33.957 DEBUG:boto:Data: 2024-04-16T20:07:33.957 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.957 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.957 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.957 DEBUG:boto:Params: {} 2024-04-16T20:07:33.957 DEBUG:boto:Token: None 2024-04-16T20:07:33.957 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11 2024-04-16T20:07:33.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VMVMI8mYHEgCX3T8CrDAPlh/J4U= 2024-04-16T20:07:33.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VMVMI8mYHEgCX3T8CrDAPlh/J4U=', 'Content-Length': '0'} 2024-04-16T20:07:33.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9120450f76e218f-00661eda85-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.960 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.960 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.960 DEBUG:boto:Method: HEAD 2024-04-16T20:07:33.960 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:33.960 DEBUG:boto:Data: 2024-04-16T20:07:33.960 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.960 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.960 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.961 DEBUG:boto:Params: {} 2024-04-16T20:07:33.961 DEBUG:boto:Token: None 2024-04-16T20:07:33.961 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11 2024-04-16T20:07:33.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VMVMI8mYHEgCX3T8CrDAPlh/J4U= 2024-04-16T20:07:33.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VMVMI8mYHEgCX3T8CrDAPlh/J4U=', 'Content-Length': '0'} 2024-04-16T20:07:33.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ac0c0ce1e889e0-00661eda85-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:07:33.963 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.964 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.964 DEBUG:boto:Method: GET 2024-04-16T20:07:33.964 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:33.964 DEBUG:boto:Data: 2024-04-16T20:07:33.964 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.964 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:33.964 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.964 DEBUG:boto:Params: {} 2024-04-16T20:07:33.964 DEBUG:boto:Token: None 2024-04-16T20:07:33.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11 2024-04-16T20:07:33.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x/m/ItGR6n8fuGyTwhI2ZfFxos8= 2024-04-16T20:07:33.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x/m/ItGR6n8fuGyTwhI2ZfFxos8=', 'Content-Length': '0'} 2024-04-16T20:07:33.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036080efe3e014bc9-00661eda85-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:33.967 DEBUG:boto:path=/yrjefh-52/obj11 2024-04-16T20:07:33.967 DEBUG:boto:auth_path=/yrjefh-52/obj11 2024-04-16T20:07:33.967 DEBUG:boto:Method: GET 2024-04-16T20:07:33.967 DEBUG:boto:Path: /yrjefh-52/obj11 2024-04-16T20:07:33.967 DEBUG:boto:Data: 2024-04-16T20:07:33.967 DEBUG:boto:Headers: {} 2024-04-16T20:07:33.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:33.968 DEBUG:boto:Port: 8000 2024-04-16T20:07:33.968 DEBUG:boto:Params: {} 2024-04-16T20:07:33.968 DEBUG:boto:Token: None 2024-04-16T20:07:33.968 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:33 GMT /yrjefh-52/obj11 2024-04-16T20:07:33.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x/m/ItGR6n8fuGyTwhI2ZfFxos8= 2024-04-16T20:07:33.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x/m/ItGR6n8fuGyTwhI2ZfFxos8=', 'Content-Length': '0'} 2024-04-16T20:07:33.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2794f524ab7077-00661eda85-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.010 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.011 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.011 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.011 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.011 DEBUG:boto:Method: GET 2024-04-16T20:07:34.011 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.011 DEBUG:boto:Data: 2024-04-16T20:07:34.011 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.011 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.011 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.012 DEBUG:boto:Params: {} 2024-04-16T20:07:34.012 DEBUG:boto:Token: None 2024-04-16T20:07:34.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000130cb5118254a0bf-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.015 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.016 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.016 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.016 DEBUG:boto:path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.016 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.016 DEBUG:boto:Method: GET 2024-04-16T20:07:34.016 DEBUG:boto:Path: /yrjefh-52/?prefix=obj11 2024-04-16T20:07:34.017 DEBUG:boto:Data: 2024-04-16T20:07:34.017 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.017 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.017 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.017 DEBUG:boto:Params: {} 2024-04-16T20:07:34.017 DEBUG:boto:Token: None 2024-04-16T20:07:34.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e961ee5713492bac-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.021 DEBUG:boto:b'yrjefh-52obj111000falseobj112024-04-16T19:58:04.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:34.021 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.022 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.022 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.022 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.022 DEBUG:boto:Method: GET 2024-04-16T20:07:34.022 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.022 DEBUG:boto:Data: 2024-04-16T20:07:34.022 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.022 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.022 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.022 DEBUG:boto:Params: {} 2024-04-16T20:07:34.023 DEBUG:boto:Token: None 2024-04-16T20:07:34.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.023 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ= 2024-04-16T20:07:34.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ=', 'Content-Length': '0'} 2024-04-16T20:07:34.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15f65755681dbb7-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.026 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.026 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.027 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.027 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.027 DEBUG:boto:Method: GET 2024-04-16T20:07:34.027 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.027 DEBUG:boto:Data: 2024-04-16T20:07:34.027 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.028 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.028 DEBUG:boto:Params: {} 2024-04-16T20:07:34.028 DEBUG:boto:Token: None 2024-04-16T20:07:34.028 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ= 2024-04-16T20:07:34.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ=', 'Content-Length': '0'} 2024-04-16T20:07:34.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bf6679e4c51ac8-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.031 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.031 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.031 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.031 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.031 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.032 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.032 DEBUG:boto:Data: 2024-04-16T20:07:34.032 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.032 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.032 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.032 DEBUG:boto:Params: {} 2024-04-16T20:07:34.032 DEBUG:boto:Token: None 2024-04-16T20:07:34.032 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.033 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4nKqMXermwZBSshw/JU0SlotvuA= 2024-04-16T20:07:34.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4nKqMXermwZBSshw/JU0SlotvuA=', 'Content-Length': '0'} 2024-04-16T20:07:34.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee52c1303ccb2e77-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.035 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.036 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.036 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.036 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.036 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.036 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.036 DEBUG:boto:Data: 2024-04-16T20:07:34.036 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.036 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.036 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.037 DEBUG:boto:Params: {} 2024-04-16T20:07:34.037 DEBUG:boto:Token: None 2024-04-16T20:07:34.037 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.037 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4nKqMXermwZBSshw/JU0SlotvuA= 2024-04-16T20:07:34.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4nKqMXermwZBSshw/JU0SlotvuA=', 'Content-Length': '0'} 2024-04-16T20:07:34.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5613d69d0be196-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:34.040 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.040 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.040 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.040 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.040 DEBUG:boto:Method: GET 2024-04-16T20:07:34.040 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.040 DEBUG:boto:Data: 2024-04-16T20:07:34.041 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.041 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.041 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.041 DEBUG:boto:Params: {} 2024-04-16T20:07:34.041 DEBUG:boto:Token: None 2024-04-16T20:07:34.041 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.041 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ= 2024-04-16T20:07:34.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ=', 'Content-Length': '0'} 2024-04-16T20:07:34.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5f91122edc6948d-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.086 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.087 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.087 DEBUG:boto:path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.087 DEBUG:boto:auth_path=/yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.087 DEBUG:boto:Method: GET 2024-04-16T20:07:34.087 DEBUG:boto:Path: /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.087 DEBUG:boto:Data: 2024-04-16T20:07:34.087 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.088 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.088 DEBUG:boto:Params: {} 2024-04-16T20:07:34.088 DEBUG:boto:Token: None 2024-04-16T20:07:34.088 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111?versionId=null 2024-04-16T20:07:34.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ= 2024-04-16T20:07:34.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dkiqoQ4mKi7MCL1GFIN+krFP4jQ=', 'Content-Length': '0'} 2024-04-16T20:07:34.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed66718e0bc5bfa-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.091 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.091 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.091 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.091 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.091 DEBUG:boto:Method: GET 2024-04-16T20:07:34.091 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.092 DEBUG:boto:Data: 2024-04-16T20:07:34.092 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.092 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.092 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.092 DEBUG:boto:Params: {} 2024-04-16T20:07:34.092 DEBUG:boto:Token: None 2024-04-16T20:07:34.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0593760014bdb2-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.095 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.096 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.096 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.096 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.096 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.096 DEBUG:boto:Method: GET 2024-04-16T20:07:34.096 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.096 DEBUG:boto:Data: 2024-04-16T20:07:34.096 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.097 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.097 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.097 DEBUG:boto:Params: {} 2024-04-16T20:07:34.097 DEBUG:boto:Token: None 2024-04-16T20:07:34.097 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.097 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b28685ab714c2a62-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.101 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.102 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.102 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.102 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.102 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:34.102 DEBUG:boto:Data: 2024-04-16T20:07:34.102 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.102 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.103 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.103 DEBUG:boto:Params: {} 2024-04-16T20:07:34.103 DEBUG:boto:Token: None 2024-04-16T20:07:34.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111 2024-04-16T20:07:34.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MPq1uI5XnJ+EFxS/3x852SOoQ+Y= 2024-04-16T20:07:34.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MPq1uI5XnJ+EFxS/3x852SOoQ+Y=', 'Content-Length': '0'} 2024-04-16T20:07:34.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078340ed062a2bd4f-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.106 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.106 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.106 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.107 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:34.107 DEBUG:boto:Data: 2024-04-16T20:07:34.107 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.107 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.107 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.107 DEBUG:boto:Params: {} 2024-04-16T20:07:34.107 DEBUG:boto:Token: None 2024-04-16T20:07:34.107 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111 2024-04-16T20:07:34.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:MPq1uI5XnJ+EFxS/3x852SOoQ+Y= 2024-04-16T20:07:34.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MPq1uI5XnJ+EFxS/3x852SOoQ+Y=', 'Content-Length': '0'} 2024-04-16T20:07:34.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc28db64cc16e863-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:07:34.110 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.110 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.111 DEBUG:boto:Method: GET 2024-04-16T20:07:34.111 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:34.111 DEBUG:boto:Data: 2024-04-16T20:07:34.111 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.111 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.111 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.111 DEBUG:boto:Params: {} 2024-04-16T20:07:34.111 DEBUG:boto:Token: None 2024-04-16T20:07:34.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111 2024-04-16T20:07:34.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Evf005ysjVVfmYJtJg+9z1ejlNA= 2024-04-16T20:07:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Evf005ysjVVfmYJtJg+9z1ejlNA=', 'Content-Length': '0'} 2024-04-16T20:07:34.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d071302e2ce085-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.115 DEBUG:boto:path=/yrjefh-52/obj111 2024-04-16T20:07:34.115 DEBUG:boto:auth_path=/yrjefh-52/obj111 2024-04-16T20:07:34.115 DEBUG:boto:Method: GET 2024-04-16T20:07:34.115 DEBUG:boto:Path: /yrjefh-52/obj111 2024-04-16T20:07:34.115 DEBUG:boto:Data: 2024-04-16T20:07:34.115 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.115 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.116 DEBUG:boto:Params: {} 2024-04-16T20:07:34.116 DEBUG:boto:Token: None 2024-04-16T20:07:34.116 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj111 2024-04-16T20:07:34.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Evf005ysjVVfmYJtJg+9z1ejlNA= 2024-04-16T20:07:34.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Evf005ysjVVfmYJtJg+9z1ejlNA=', 'Content-Length': '0'} 2024-04-16T20:07:34.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000077df836a06dec96-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.119 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.119 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.119 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.119 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.119 DEBUG:boto:Method: GET 2024-04-16T20:07:34.119 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.119 DEBUG:boto:Data: 2024-04-16T20:07:34.119 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.120 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.120 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.120 DEBUG:boto:Params: {} 2024-04-16T20:07:34.120 DEBUG:boto:Token: None 2024-04-16T20:07:34.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760ed33cb66307f9-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.166 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.167 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.167 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.167 DEBUG:boto:path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.167 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.167 DEBUG:boto:Method: GET 2024-04-16T20:07:34.167 DEBUG:boto:Path: /yrjefh-52/?prefix=obj111 2024-04-16T20:07:34.168 DEBUG:boto:Data: 2024-04-16T20:07:34.168 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.168 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.168 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.168 DEBUG:boto:Params: {} 2024-04-16T20:07:34.168 DEBUG:boto:Token: None 2024-04-16T20:07:34.168 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e86c7265ade6c261-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.172 DEBUG:boto:b'yrjefh-52obj1111000falseobj1112024-04-16T20:06:00.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:34.173 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.173 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.173 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.173 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.173 DEBUG:boto:Method: GET 2024-04-16T20:07:34.173 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.173 DEBUG:boto:Data: 2024-04-16T20:07:34.173 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.174 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.174 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.174 DEBUG:boto:Params: {} 2024-04-16T20:07:34.174 DEBUG:boto:Token: None 2024-04-16T20:07:34.174 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms= 2024-04-16T20:07:34.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms=', 'Content-Length': '0'} 2024-04-16T20:07:34.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac8fdd878028a93-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.178 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.178 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.178 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.178 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.178 DEBUG:boto:Method: GET 2024-04-16T20:07:34.178 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.179 DEBUG:boto:Data: 2024-04-16T20:07:34.179 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.179 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.179 DEBUG:boto:Params: {} 2024-04-16T20:07:34.179 DEBUG:boto:Token: None 2024-04-16T20:07:34.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms= 2024-04-16T20:07:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms=', 'Content-Length': '0'} 2024-04-16T20:07:34.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002068b625737a3a3b-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.222 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.223 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.223 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.223 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.223 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.223 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.223 DEBUG:boto:Data: 2024-04-16T20:07:34.223 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.223 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.223 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.223 DEBUG:boto:Params: {} 2024-04-16T20:07:34.224 DEBUG:boto:Token: None 2024-04-16T20:07:34.224 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ATzBenUD8dNB01BKIXzbn/suRx8= 2024-04-16T20:07:34.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ATzBenUD8dNB01BKIXzbn/suRx8=', 'Content-Length': '0'} 2024-04-16T20:07:34.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9dd611f62b7427-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.227 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.227 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.227 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.227 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.227 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.227 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.227 DEBUG:boto:Data: 2024-04-16T20:07:34.227 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.227 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.228 DEBUG:boto:Params: {} 2024-04-16T20:07:34.228 DEBUG:boto:Token: None 2024-04-16T20:07:34.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ATzBenUD8dNB01BKIXzbn/suRx8= 2024-04-16T20:07:34.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ATzBenUD8dNB01BKIXzbn/suRx8=', 'Content-Length': '0'} 2024-04-16T20:07:34.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000123aab084a7b1e4e-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:34.231 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.231 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.231 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.231 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.231 DEBUG:boto:Method: GET 2024-04-16T20:07:34.231 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.231 DEBUG:boto:Data: 2024-04-16T20:07:34.231 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.232 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.232 DEBUG:boto:Params: {} 2024-04-16T20:07:34.232 DEBUG:boto:Token: None 2024-04-16T20:07:34.232 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms= 2024-04-16T20:07:34.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms=', 'Content-Length': '0'} 2024-04-16T20:07:34.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0c2846e22025ae3-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.235 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.235 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.235 DEBUG:boto:path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.235 DEBUG:boto:auth_path=/yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.236 DEBUG:boto:Method: GET 2024-04-16T20:07:34.236 DEBUG:boto:Path: /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.236 DEBUG:boto:Data: 2024-04-16T20:07:34.236 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.236 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.236 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.236 DEBUG:boto:Params: {} 2024-04-16T20:07:34.236 DEBUG:boto:Token: None 2024-04-16T20:07:34.236 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115?versionId=null 2024-04-16T20:07:34.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms= 2024-04-16T20:07:34.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wsLRndequ1GDf0D/uy8XoxHpzms=', 'Content-Length': '0'} 2024-04-16T20:07:34.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61aebb7ca69c11c-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.239 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.239 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.239 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.239 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.239 DEBUG:boto:Method: GET 2024-04-16T20:07:34.240 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.240 DEBUG:boto:Data: 2024-04-16T20:07:34.240 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.240 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.240 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.240 DEBUG:boto:Params: {} 2024-04-16T20:07:34.240 DEBUG:boto:Token: None 2024-04-16T20:07:34.240 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cf774f66083980-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.243 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.244 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.244 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.244 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.244 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.244 DEBUG:boto:Method: GET 2024-04-16T20:07:34.244 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.244 DEBUG:boto:Data: 2024-04-16T20:07:34.245 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.245 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.245 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.245 DEBUG:boto:Params: {} 2024-04-16T20:07:34.245 DEBUG:boto:Token: None 2024-04-16T20:07:34.245 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d0c4b4b9018043-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.290 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.291 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.291 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.291 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.291 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:34.291 DEBUG:boto:Data: 2024-04-16T20:07:34.291 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.291 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.291 DEBUG:boto:Params: {} 2024-04-16T20:07:34.291 DEBUG:boto:Token: None 2024-04-16T20:07:34.292 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115 2024-04-16T20:07:34.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UMXvm1+TdC1XvFxwLdaxR3iLYWY= 2024-04-16T20:07:34.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UMXvm1+TdC1XvFxwLdaxR3iLYWY=', 'Content-Length': '0'} 2024-04-16T20:07:34.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040bdd173590dad3e-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.294 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.295 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.295 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.295 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:34.295 DEBUG:boto:Data: 2024-04-16T20:07:34.295 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.295 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.295 DEBUG:boto:Params: {} 2024-04-16T20:07:34.295 DEBUG:boto:Token: None 2024-04-16T20:07:34.295 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115 2024-04-16T20:07:34.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UMXvm1+TdC1XvFxwLdaxR3iLYWY= 2024-04-16T20:07:34.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UMXvm1+TdC1XvFxwLdaxR3iLYWY=', 'Content-Length': '0'} 2024-04-16T20:07:34.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003626e1b88cbf63ac-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:07:34.298 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.298 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.299 DEBUG:boto:Method: GET 2024-04-16T20:07:34.299 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:34.299 DEBUG:boto:Data: 2024-04-16T20:07:34.299 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.299 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.299 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.299 DEBUG:boto:Params: {} 2024-04-16T20:07:34.299 DEBUG:boto:Token: None 2024-04-16T20:07:34.299 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115 2024-04-16T20:07:34.299 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PS8deyLh7zBe4oQZnw8/XxFsrgE= 2024-04-16T20:07:34.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PS8deyLh7zBe4oQZnw8/XxFsrgE=', 'Content-Length': '0'} 2024-04-16T20:07:34.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f345220b326b43d8-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.302 DEBUG:boto:path=/yrjefh-52/obj115 2024-04-16T20:07:34.302 DEBUG:boto:auth_path=/yrjefh-52/obj115 2024-04-16T20:07:34.302 DEBUG:boto:Method: GET 2024-04-16T20:07:34.302 DEBUG:boto:Path: /yrjefh-52/obj115 2024-04-16T20:07:34.302 DEBUG:boto:Data: 2024-04-16T20:07:34.303 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.303 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.303 DEBUG:boto:Params: {} 2024-04-16T20:07:34.303 DEBUG:boto:Token: None 2024-04-16T20:07:34.303 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj115 2024-04-16T20:07:34.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PS8deyLh7zBe4oQZnw8/XxFsrgE= 2024-04-16T20:07:34.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PS8deyLh7zBe4oQZnw8/XxFsrgE=', 'Content-Length': '0'} 2024-04-16T20:07:34.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d3c452aeae427d-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.306 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.306 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.306 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.306 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.306 DEBUG:boto:Method: GET 2024-04-16T20:07:34.306 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.306 DEBUG:boto:Data: 2024-04-16T20:07:34.306 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.306 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.307 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.307 DEBUG:boto:Params: {} 2024-04-16T20:07:34.307 DEBUG:boto:Token: None 2024-04-16T20:07:34.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b04c1a57677d25d1-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.310 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.310 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.310 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.310 DEBUG:boto:path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.310 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.311 DEBUG:boto:Method: GET 2024-04-16T20:07:34.311 DEBUG:boto:Path: /yrjefh-52/?prefix=obj115 2024-04-16T20:07:34.311 DEBUG:boto:Data: 2024-04-16T20:07:34.311 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.311 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.311 DEBUG:boto:Params: {} 2024-04-16T20:07:34.311 DEBUG:boto:Token: None 2024-04-16T20:07:34.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001241904253c60f19-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.315 DEBUG:boto:b'yrjefh-52obj1151000falseobj1152024-04-16T20:01:27.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:34.316 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.316 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.316 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.316 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.316 DEBUG:boto:Method: GET 2024-04-16T20:07:34.316 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.316 DEBUG:boto:Data: 2024-04-16T20:07:34.316 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.316 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.317 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.317 DEBUG:boto:Params: {} 2024-04-16T20:07:34.317 DEBUG:boto:Token: None 2024-04-16T20:07:34.317 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE= 2024-04-16T20:07:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE=', 'Content-Length': '0'} 2024-04-16T20:07:34.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e9816854baf87a-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.320 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.320 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.320 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.320 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.320 DEBUG:boto:Method: GET 2024-04-16T20:07:34.320 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.320 DEBUG:boto:Data: 2024-04-16T20:07:34.320 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.321 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.321 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.321 DEBUG:boto:Params: {} 2024-04-16T20:07:34.321 DEBUG:boto:Token: None 2024-04-16T20:07:34.321 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE= 2024-04-16T20:07:34.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE=', 'Content-Length': '0'} 2024-04-16T20:07:34.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a794e1da209e9019-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.324 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.324 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.324 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.324 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.324 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.324 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.324 DEBUG:boto:Data: 2024-04-16T20:07:34.324 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.324 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.324 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.324 DEBUG:boto:Params: {} 2024-04-16T20:07:34.325 DEBUG:boto:Token: None 2024-04-16T20:07:34.325 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.325 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mlSDEj05H0vgZOTVI1HcpugC7BY= 2024-04-16T20:07:34.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mlSDEj05H0vgZOTVI1HcpugC7BY=', 'Content-Length': '0'} 2024-04-16T20:07:34.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4898b00f1e76dd8-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.327 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.327 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.327 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.327 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.327 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.327 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.327 DEBUG:boto:Data: 2024-04-16T20:07:34.328 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.328 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.328 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.328 DEBUG:boto:Params: {} 2024-04-16T20:07:34.328 DEBUG:boto:Token: None 2024-04-16T20:07:34.328 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mlSDEj05H0vgZOTVI1HcpugC7BY= 2024-04-16T20:07:34.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mlSDEj05H0vgZOTVI1HcpugC7BY=', 'Content-Length': '0'} 2024-04-16T20:07:34.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004373766394f66f0b-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:34.330 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.330 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.331 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.331 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.331 DEBUG:boto:Method: GET 2024-04-16T20:07:34.331 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.331 DEBUG:boto:Data: 2024-04-16T20:07:34.331 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.331 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.331 DEBUG:boto:Params: {} 2024-04-16T20:07:34.331 DEBUG:boto:Token: None 2024-04-16T20:07:34.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE= 2024-04-16T20:07:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE=', 'Content-Length': '0'} 2024-04-16T20:07:34.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f39101599c0a2d-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.374 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.374 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.375 DEBUG:boto:path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.375 DEBUG:boto:auth_path=/yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.375 DEBUG:boto:Method: GET 2024-04-16T20:07:34.375 DEBUG:boto:Path: /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.375 DEBUG:boto:Data: 2024-04-16T20:07:34.375 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.375 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.375 DEBUG:boto:Params: {} 2024-04-16T20:07:34.375 DEBUG:boto:Token: None 2024-04-16T20:07:34.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116?versionId=null 2024-04-16T20:07:34.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE= 2024-04-16T20:07:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+eGdvnWb+jeNsF5izLY70/8++sE=', 'Content-Length': '0'} 2024-04-16T20:07:34.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8eca3948c3a81a-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.378 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.378 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.378 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.378 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.379 DEBUG:boto:Method: GET 2024-04-16T20:07:34.379 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.379 DEBUG:boto:Data: 2024-04-16T20:07:34.379 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.379 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.379 DEBUG:boto:Params: {} 2024-04-16T20:07:34.379 DEBUG:boto:Token: None 2024-04-16T20:07:34.379 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f40f00063ab5c9-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.382 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.383 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.383 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.383 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.383 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.383 DEBUG:boto:Method: GET 2024-04-16T20:07:34.383 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.383 DEBUG:boto:Data: 2024-04-16T20:07:34.383 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.383 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.384 DEBUG:boto:Params: {} 2024-04-16T20:07:34.384 DEBUG:boto:Token: None 2024-04-16T20:07:34.384 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf020902a16cc55-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.388 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.388 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.388 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.388 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.388 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:34.388 DEBUG:boto:Data: 2024-04-16T20:07:34.389 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.389 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.389 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.389 DEBUG:boto:Params: {} 2024-04-16T20:07:34.389 DEBUG:boto:Token: None 2024-04-16T20:07:34.389 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116 2024-04-16T20:07:34.389 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JqMe0/ZPOP//RYTlO91/uXUB71g= 2024-04-16T20:07:34.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JqMe0/ZPOP//RYTlO91/uXUB71g=', 'Content-Length': '0'} 2024-04-16T20:07:34.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c288eee44df761a-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.391 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.391 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.392 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.392 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:34.392 DEBUG:boto:Data: 2024-04-16T20:07:34.392 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.392 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.392 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.392 DEBUG:boto:Params: {} 2024-04-16T20:07:34.392 DEBUG:boto:Token: None 2024-04-16T20:07:34.392 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116 2024-04-16T20:07:34.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JqMe0/ZPOP//RYTlO91/uXUB71g= 2024-04-16T20:07:34.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JqMe0/ZPOP//RYTlO91/uXUB71g=', 'Content-Length': '0'} 2024-04-16T20:07:34.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f89e6500edc807-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:07:34.395 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.395 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.395 DEBUG:boto:Method: GET 2024-04-16T20:07:34.395 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:34.395 DEBUG:boto:Data: 2024-04-16T20:07:34.395 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.395 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.395 DEBUG:boto:Params: {} 2024-04-16T20:07:34.395 DEBUG:boto:Token: None 2024-04-16T20:07:34.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116 2024-04-16T20:07:34.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+g8ftuT6HJEVmlz6yD209JSDLA= 2024-04-16T20:07:34.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+g8ftuT6HJEVmlz6yD209JSDLA=', 'Content-Length': '0'} 2024-04-16T20:07:34.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002722b57992e64d0f-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.438 DEBUG:boto:path=/yrjefh-52/obj116 2024-04-16T20:07:34.438 DEBUG:boto:auth_path=/yrjefh-52/obj116 2024-04-16T20:07:34.439 DEBUG:boto:Method: GET 2024-04-16T20:07:34.439 DEBUG:boto:Path: /yrjefh-52/obj116 2024-04-16T20:07:34.439 DEBUG:boto:Data: 2024-04-16T20:07:34.439 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.439 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.439 DEBUG:boto:Params: {} 2024-04-16T20:07:34.439 DEBUG:boto:Token: None 2024-04-16T20:07:34.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj116 2024-04-16T20:07:34.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7+g8ftuT6HJEVmlz6yD209JSDLA= 2024-04-16T20:07:34.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+g8ftuT6HJEVmlz6yD209JSDLA=', 'Content-Length': '0'} 2024-04-16T20:07:34.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8641e2a2863942d-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.442 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.442 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.442 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.442 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.442 DEBUG:boto:Method: GET 2024-04-16T20:07:34.442 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.442 DEBUG:boto:Data: 2024-04-16T20:07:34.442 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.442 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.443 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.443 DEBUG:boto:Params: {} 2024-04-16T20:07:34.443 DEBUG:boto:Token: None 2024-04-16T20:07:34.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a1f6d1f6219840-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.445 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.446 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.446 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.446 DEBUG:boto:path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.446 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.446 DEBUG:boto:Method: GET 2024-04-16T20:07:34.446 DEBUG:boto:Path: /yrjefh-52/?prefix=obj116 2024-04-16T20:07:34.446 DEBUG:boto:Data: 2024-04-16T20:07:34.446 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.447 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.447 DEBUG:boto:Params: {} 2024-04-16T20:07:34.447 DEBUG:boto:Token: None 2024-04-16T20:07:34.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002902ab5fb8781577-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.451 DEBUG:boto:b'yrjefh-52obj1161000falseobj1162024-04-16T20:03:33.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:34.451 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.451 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.451 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.451 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.452 DEBUG:boto:Method: GET 2024-04-16T20:07:34.452 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.452 DEBUG:boto:Data: 2024-04-16T20:07:34.452 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.452 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.452 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.452 DEBUG:boto:Params: {} 2024-04-16T20:07:34.452 DEBUG:boto:Token: None 2024-04-16T20:07:34.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM= 2024-04-16T20:07:34.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM=', 'Content-Length': '0'} 2024-04-16T20:07:34.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4663686f376a4d9-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.455 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.455 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.455 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.455 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.455 DEBUG:boto:Method: GET 2024-04-16T20:07:34.455 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.455 DEBUG:boto:Data: 2024-04-16T20:07:34.455 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.456 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.456 DEBUG:boto:Params: {} 2024-04-16T20:07:34.456 DEBUG:boto:Token: None 2024-04-16T20:07:34.456 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM= 2024-04-16T20:07:34.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM=', 'Content-Length': '0'} 2024-04-16T20:07:34.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a585520b13d34e-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.458 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.458 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.458 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.458 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.458 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.459 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.459 DEBUG:boto:Data: 2024-04-16T20:07:34.459 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.459 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.459 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.459 DEBUG:boto:Params: {} 2024-04-16T20:07:34.459 DEBUG:boto:Token: None 2024-04-16T20:07:34.459 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.459 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t961KRUvEoHP20zEY+neL5aHyzk= 2024-04-16T20:07:34.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t961KRUvEoHP20zEY+neL5aHyzk=', 'Content-Length': '0'} 2024-04-16T20:07:34.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c279a4fc790922-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.461 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.462 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.462 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.462 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.462 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.462 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.462 DEBUG:boto:Data: 2024-04-16T20:07:34.462 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.462 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.462 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.462 DEBUG:boto:Params: {} 2024-04-16T20:07:34.462 DEBUG:boto:Token: None 2024-04-16T20:07:34.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t961KRUvEoHP20zEY+neL5aHyzk= 2024-04-16T20:07:34.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t961KRUvEoHP20zEY+neL5aHyzk=', 'Content-Length': '0'} 2024-04-16T20:07:34.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000737b64798616f651-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:34.465 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.465 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.465 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.465 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.465 DEBUG:boto:Method: GET 2024-04-16T20:07:34.465 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.466 DEBUG:boto:Data: 2024-04-16T20:07:34.466 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.466 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.466 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.466 DEBUG:boto:Params: {} 2024-04-16T20:07:34.466 DEBUG:boto:Token: None 2024-04-16T20:07:34.466 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.466 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM= 2024-04-16T20:07:34.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM=', 'Content-Length': '0'} 2024-04-16T20:07:34.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2629c162663b83c-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.468 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.468 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.469 DEBUG:boto:path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.469 DEBUG:boto:auth_path=/yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.469 DEBUG:boto:Method: GET 2024-04-16T20:07:34.469 DEBUG:boto:Path: /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.469 DEBUG:boto:Data: 2024-04-16T20:07:34.469 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.469 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.469 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.469 DEBUG:boto:Params: {} 2024-04-16T20:07:34.469 DEBUG:boto:Token: None 2024-04-16T20:07:34.469 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124?versionId=null 2024-04-16T20:07:34.470 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM= 2024-04-16T20:07:34.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ndvWSOhllBDTf4UNKDO2cd7pUDM=', 'Content-Length': '0'} 2024-04-16T20:07:34.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5c4edfe50e3a1f-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.472 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.472 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.472 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.472 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.472 DEBUG:boto:Method: GET 2024-04-16T20:07:34.472 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.472 DEBUG:boto:Data: 2024-04-16T20:07:34.472 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.472 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.472 DEBUG:boto:Params: {} 2024-04-16T20:07:34.472 DEBUG:boto:Token: None 2024-04-16T20:07:34.473 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003328518eeb64781b-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.475 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.475 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.476 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.476 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.476 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.476 DEBUG:boto:Method: GET 2024-04-16T20:07:34.476 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.476 DEBUG:boto:Data: 2024-04-16T20:07:34.476 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.476 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.476 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.476 DEBUG:boto:Params: {} 2024-04-16T20:07:34.476 DEBUG:boto:Token: None 2024-04-16T20:07:34.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a301817a63b38fa7-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.522 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.523 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.523 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.523 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.523 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:34.523 DEBUG:boto:Data: 2024-04-16T20:07:34.523 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.523 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.523 DEBUG:boto:Params: {} 2024-04-16T20:07:34.524 DEBUG:boto:Token: None 2024-04-16T20:07:34.524 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124 2024-04-16T20:07:34.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d76tYSpIyKAPai0QJRhYxJmAplc= 2024-04-16T20:07:34.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d76tYSpIyKAPai0QJRhYxJmAplc=', 'Content-Length': '0'} 2024-04-16T20:07:34.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac26b831937698d-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.526 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.526 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.526 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.526 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:34.526 DEBUG:boto:Data: 2024-04-16T20:07:34.526 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.526 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.526 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.526 DEBUG:boto:Params: {} 2024-04-16T20:07:34.527 DEBUG:boto:Token: None 2024-04-16T20:07:34.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124 2024-04-16T20:07:34.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d76tYSpIyKAPai0QJRhYxJmAplc= 2024-04-16T20:07:34.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d76tYSpIyKAPai0QJRhYxJmAplc=', 'Content-Length': '0'} 2024-04-16T20:07:34.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07650688d5c807e-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:07:34.529 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.529 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.529 DEBUG:boto:Method: GET 2024-04-16T20:07:34.529 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:34.529 DEBUG:boto:Data: 2024-04-16T20:07:34.530 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.530 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.530 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.530 DEBUG:boto:Params: {} 2024-04-16T20:07:34.530 DEBUG:boto:Token: None 2024-04-16T20:07:34.530 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124 2024-04-16T20:07:34.530 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OEIyfxdzqySnEgmkyE3ZX5xzG6I= 2024-04-16T20:07:34.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OEIyfxdzqySnEgmkyE3ZX5xzG6I=', 'Content-Length': '0'} 2024-04-16T20:07:34.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4886b287e7ac34-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.532 DEBUG:boto:path=/yrjefh-52/obj124 2024-04-16T20:07:34.532 DEBUG:boto:auth_path=/yrjefh-52/obj124 2024-04-16T20:07:34.533 DEBUG:boto:Method: GET 2024-04-16T20:07:34.533 DEBUG:boto:Path: /yrjefh-52/obj124 2024-04-16T20:07:34.533 DEBUG:boto:Data: 2024-04-16T20:07:34.533 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.533 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.533 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.533 DEBUG:boto:Params: {} 2024-04-16T20:07:34.533 DEBUG:boto:Token: None 2024-04-16T20:07:34.533 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj124 2024-04-16T20:07:34.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OEIyfxdzqySnEgmkyE3ZX5xzG6I= 2024-04-16T20:07:34.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OEIyfxdzqySnEgmkyE3ZX5xzG6I=', 'Content-Length': '0'} 2024-04-16T20:07:34.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931e8b7328328ab0-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.536 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.536 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.536 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.536 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.536 DEBUG:boto:Method: GET 2024-04-16T20:07:34.536 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.536 DEBUG:boto:Data: 2024-04-16T20:07:34.536 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.536 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.536 DEBUG:boto:Params: {} 2024-04-16T20:07:34.537 DEBUG:boto:Token: None 2024-04-16T20:07:34.537 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065da302217b5008b-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.539 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.539 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.540 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.540 DEBUG:boto:path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.540 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.540 DEBUG:boto:Method: GET 2024-04-16T20:07:34.540 DEBUG:boto:Path: /yrjefh-52/?prefix=obj124 2024-04-16T20:07:34.540 DEBUG:boto:Data: 2024-04-16T20:07:34.540 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.540 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.540 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.540 DEBUG:boto:Params: {} 2024-04-16T20:07:34.540 DEBUG:boto:Token: None 2024-04-16T20:07:34.540 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008261ca33babf1eaa-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.544 DEBUG:boto:b'yrjefh-52obj1241000falseobj1242024-04-16T20:04:43.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:34.545 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.545 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.545 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.545 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.545 DEBUG:boto:Method: GET 2024-04-16T20:07:34.545 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.545 DEBUG:boto:Data: 2024-04-16T20:07:34.545 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.545 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.545 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.545 DEBUG:boto:Params: {} 2024-04-16T20:07:34.546 DEBUG:boto:Token: None 2024-04-16T20:07:34.546 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8= 2024-04-16T20:07:34.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8=', 'Content-Length': '0'} 2024-04-16T20:07:34.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5248e8711830f04-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.590 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.590 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.591 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.591 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.591 DEBUG:boto:Method: GET 2024-04-16T20:07:34.591 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.591 DEBUG:boto:Data: 2024-04-16T20:07:34.591 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.591 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.591 DEBUG:boto:Params: {} 2024-04-16T20:07:34.591 DEBUG:boto:Token: None 2024-04-16T20:07:34.592 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8= 2024-04-16T20:07:34.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8=', 'Content-Length': '0'} 2024-04-16T20:07:34.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ab3610ec2ac756-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.595 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.595 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.595 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.595 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.595 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.595 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.595 DEBUG:boto:Data: 2024-04-16T20:07:34.595 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.595 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.595 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.595 DEBUG:boto:Params: {} 2024-04-16T20:07:34.596 DEBUG:boto:Token: None 2024-04-16T20:07:34.596 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PZWASDfQBUPnrF/SfZudndSSD1I= 2024-04-16T20:07:34.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PZWASDfQBUPnrF/SfZudndSSD1I=', 'Content-Length': '0'} 2024-04-16T20:07:34.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b03d754866a77a-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.598 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.598 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.598 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.598 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.598 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.599 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.599 DEBUG:boto:Data: 2024-04-16T20:07:34.599 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.599 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.599 DEBUG:boto:Params: {} 2024-04-16T20:07:34.599 DEBUG:boto:Token: None 2024-04-16T20:07:34.599 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PZWASDfQBUPnrF/SfZudndSSD1I= 2024-04-16T20:07:34.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PZWASDfQBUPnrF/SfZudndSSD1I=', 'Content-Length': '0'} 2024-04-16T20:07:34.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000016a116daf945540-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:34.602 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.602 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.602 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.602 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.602 DEBUG:boto:Method: GET 2024-04-16T20:07:34.602 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.603 DEBUG:boto:Data: 2024-04-16T20:07:34.603 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.603 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.603 DEBUG:boto:Params: {} 2024-04-16T20:07:34.603 DEBUG:boto:Token: None 2024-04-16T20:07:34.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8= 2024-04-16T20:07:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8=', 'Content-Length': '0'} 2024-04-16T20:07:34.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8dd72d421cd88a-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.606 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.606 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.606 DEBUG:boto:path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.606 DEBUG:boto:auth_path=/yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.606 DEBUG:boto:Method: GET 2024-04-16T20:07:34.607 DEBUG:boto:Path: /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.607 DEBUG:boto:Data: 2024-04-16T20:07:34.607 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.607 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.607 DEBUG:boto:Params: {} 2024-04-16T20:07:34.607 DEBUG:boto:Token: None 2024-04-16T20:07:34.609 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145?versionId=null 2024-04-16T20:07:34.609 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8= 2024-04-16T20:07:34.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JvSD++UB/5inWuhiXftxCKNe6w8=', 'Content-Length': '0'} 2024-04-16T20:07:34.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517083e11e039c2b-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.611 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.612 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.612 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.612 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.612 DEBUG:boto:Method: GET 2024-04-16T20:07:34.612 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.612 DEBUG:boto:Data: 2024-04-16T20:07:34.612 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.612 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.612 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.612 DEBUG:boto:Params: {} 2024-04-16T20:07:34.612 DEBUG:boto:Token: None 2024-04-16T20:07:34.613 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.613 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fea448f1af93a9-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.615 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.615 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.616 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.616 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.616 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.616 DEBUG:boto:Method: GET 2024-04-16T20:07:34.616 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.616 DEBUG:boto:Data: 2024-04-16T20:07:34.616 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.616 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.616 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.616 DEBUG:boto:Params: {} 2024-04-16T20:07:34.616 DEBUG:boto:Token: None 2024-04-16T20:07:34.617 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.617 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac4166cb4284558e-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.662 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.663 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.663 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.663 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.663 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:34.663 DEBUG:boto:Data: 2024-04-16T20:07:34.663 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.663 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.663 DEBUG:boto:Params: {} 2024-04-16T20:07:34.663 DEBUG:boto:Token: None 2024-04-16T20:07:34.663 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145 2024-04-16T20:07:34.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M2jiAbckoRkfqCB5A3JMl5Vt0oE= 2024-04-16T20:07:34.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M2jiAbckoRkfqCB5A3JMl5Vt0oE=', 'Content-Length': '0'} 2024-04-16T20:07:34.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed682eb37007200a-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.666 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.666 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.666 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.666 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:34.666 DEBUG:boto:Data: 2024-04-16T20:07:34.667 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.667 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.667 DEBUG:boto:Params: {} 2024-04-16T20:07:34.667 DEBUG:boto:Token: None 2024-04-16T20:07:34.667 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145 2024-04-16T20:07:34.667 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M2jiAbckoRkfqCB5A3JMl5Vt0oE= 2024-04-16T20:07:34.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M2jiAbckoRkfqCB5A3JMl5Vt0oE=', 'Content-Length': '0'} 2024-04-16T20:07:34.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4bbf6966d871194-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:07:34.670 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.670 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.670 DEBUG:boto:Method: GET 2024-04-16T20:07:34.671 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:34.671 DEBUG:boto:Data: 2024-04-16T20:07:34.671 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.671 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.671 DEBUG:boto:Params: {} 2024-04-16T20:07:34.671 DEBUG:boto:Token: None 2024-04-16T20:07:34.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145 2024-04-16T20:07:34.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GnZlbsMd1IRJPEx7TSSNOc64gaw= 2024-04-16T20:07:34.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GnZlbsMd1IRJPEx7TSSNOc64gaw=', 'Content-Length': '0'} 2024-04-16T20:07:34.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd847abdb31dd692-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.674 DEBUG:boto:path=/yrjefh-52/obj145 2024-04-16T20:07:34.674 DEBUG:boto:auth_path=/yrjefh-52/obj145 2024-04-16T20:07:34.674 DEBUG:boto:Method: GET 2024-04-16T20:07:34.674 DEBUG:boto:Path: /yrjefh-52/obj145 2024-04-16T20:07:34.674 DEBUG:boto:Data: 2024-04-16T20:07:34.674 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.674 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.674 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.675 DEBUG:boto:Params: {} 2024-04-16T20:07:34.675 DEBUG:boto:Token: None 2024-04-16T20:07:34.675 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj145 2024-04-16T20:07:34.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GnZlbsMd1IRJPEx7TSSNOc64gaw= 2024-04-16T20:07:34.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GnZlbsMd1IRJPEx7TSSNOc64gaw=', 'Content-Length': '0'} 2024-04-16T20:07:34.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a5b8692345a4e7-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.677 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.677 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.677 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.678 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.678 DEBUG:boto:Method: GET 2024-04-16T20:07:34.678 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.678 DEBUG:boto:Data: 2024-04-16T20:07:34.678 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.678 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.678 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.678 DEBUG:boto:Params: {} 2024-04-16T20:07:34.678 DEBUG:boto:Token: None 2024-04-16T20:07:34.678 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1be6041ac84f87-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.681 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.682 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.682 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.682 DEBUG:boto:path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.682 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.682 DEBUG:boto:Method: GET 2024-04-16T20:07:34.682 DEBUG:boto:Path: /yrjefh-52/?prefix=obj145 2024-04-16T20:07:34.682 DEBUG:boto:Data: 2024-04-16T20:07:34.682 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.683 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.683 DEBUG:boto:Params: {} 2024-04-16T20:07:34.683 DEBUG:boto:Token: None 2024-04-16T20:07:34.683 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b2d46c7e6543be-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.687 DEBUG:boto:b'yrjefh-52obj1451000falseobj1452024-04-16T20:03:33.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:34.687 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.688 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.688 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.688 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.688 DEBUG:boto:Method: GET 2024-04-16T20:07:34.688 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.688 DEBUG:boto:Data: 2024-04-16T20:07:34.688 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.688 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.688 DEBUG:boto:Params: {} 2024-04-16T20:07:34.688 DEBUG:boto:Token: None 2024-04-16T20:07:34.689 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U= 2024-04-16T20:07:34.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U=', 'Content-Length': '0'} 2024-04-16T20:07:34.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bf449c9168b674-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.691 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.692 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.692 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.692 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.692 DEBUG:boto:Method: GET 2024-04-16T20:07:34.692 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.692 DEBUG:boto:Data: 2024-04-16T20:07:34.692 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.692 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.692 DEBUG:boto:Params: {} 2024-04-16T20:07:34.692 DEBUG:boto:Token: None 2024-04-16T20:07:34.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U= 2024-04-16T20:07:34.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U=', 'Content-Length': '0'} 2024-04-16T20:07:34.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c73a42486aadfd-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.695 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.695 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.695 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.695 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.696 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.696 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.696 DEBUG:boto:Data: 2024-04-16T20:07:34.696 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.696 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.696 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.696 DEBUG:boto:Params: {} 2024-04-16T20:07:34.696 DEBUG:boto:Token: None 2024-04-16T20:07:34.696 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.696 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pmmcoGf89nFxmjUdMNblqQ+kz+4= 2024-04-16T20:07:34.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmmcoGf89nFxmjUdMNblqQ+kz+4=', 'Content-Length': '0'} 2024-04-16T20:07:34.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432ed3d961f2933b-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.699 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.699 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.699 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.699 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.699 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.699 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.699 DEBUG:boto:Data: 2024-04-16T20:07:34.700 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.700 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.700 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.700 DEBUG:boto:Params: {} 2024-04-16T20:07:34.700 DEBUG:boto:Token: None 2024-04-16T20:07:34.700 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pmmcoGf89nFxmjUdMNblqQ+kz+4= 2024-04-16T20:07:34.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmmcoGf89nFxmjUdMNblqQ+kz+4=', 'Content-Length': '0'} 2024-04-16T20:07:34.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b67aaf35872523c-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:34.703 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.703 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.703 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.703 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.703 DEBUG:boto:Method: GET 2024-04-16T20:07:34.703 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.703 DEBUG:boto:Data: 2024-04-16T20:07:34.703 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.703 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.703 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.703 DEBUG:boto:Params: {} 2024-04-16T20:07:34.704 DEBUG:boto:Token: None 2024-04-16T20:07:34.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U= 2024-04-16T20:07:34.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U=', 'Content-Length': '0'} 2024-04-16T20:07:34.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130a60a850767f63-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.746 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.746 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.747 DEBUG:boto:path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.747 DEBUG:boto:auth_path=/yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.747 DEBUG:boto:Method: GET 2024-04-16T20:07:34.747 DEBUG:boto:Path: /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.747 DEBUG:boto:Data: 2024-04-16T20:07:34.747 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.747 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.747 DEBUG:boto:Params: {} 2024-04-16T20:07:34.747 DEBUG:boto:Token: None 2024-04-16T20:07:34.748 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148?versionId=null 2024-04-16T20:07:34.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U= 2024-04-16T20:07:34.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tO/YxpowlB7U7sIklMRYNqqr71U=', 'Content-Length': '0'} 2024-04-16T20:07:34.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e27128bf5f247f-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.750 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.751 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.751 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.751 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.751 DEBUG:boto:Method: GET 2024-04-16T20:07:34.751 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.751 DEBUG:boto:Data: 2024-04-16T20:07:34.751 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.751 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.751 DEBUG:boto:Params: {} 2024-04-16T20:07:34.751 DEBUG:boto:Token: None 2024-04-16T20:07:34.752 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332fb1073360ba78-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.754 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.754 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.754 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.755 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.755 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.755 DEBUG:boto:Method: GET 2024-04-16T20:07:34.755 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.755 DEBUG:boto:Data: 2024-04-16T20:07:34.755 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.755 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.755 DEBUG:boto:Params: {} 2024-04-16T20:07:34.755 DEBUG:boto:Token: None 2024-04-16T20:07:34.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e819b176bd40aff6-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.760 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.760 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.760 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.760 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.760 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:34.760 DEBUG:boto:Data: 2024-04-16T20:07:34.760 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.760 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.760 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.761 DEBUG:boto:Params: {} 2024-04-16T20:07:34.761 DEBUG:boto:Token: None 2024-04-16T20:07:34.761 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148 2024-04-16T20:07:34.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f53YCpncxSkPPvvIPr/WOPEEDlI= 2024-04-16T20:07:34.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f53YCpncxSkPPvvIPr/WOPEEDlI=', 'Content-Length': '0'} 2024-04-16T20:07:34.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b056c1c55e3ed5a8-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.763 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.764 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.764 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.764 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:34.764 DEBUG:boto:Data: 2024-04-16T20:07:34.764 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.764 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.764 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.764 DEBUG:boto:Params: {} 2024-04-16T20:07:34.764 DEBUG:boto:Token: None 2024-04-16T20:07:34.764 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148 2024-04-16T20:07:34.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f53YCpncxSkPPvvIPr/WOPEEDlI= 2024-04-16T20:07:34.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f53YCpncxSkPPvvIPr/WOPEEDlI=', 'Content-Length': '0'} 2024-04-16T20:07:34.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002024322e5024af8f-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:07:34.767 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.767 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.767 DEBUG:boto:Method: GET 2024-04-16T20:07:34.767 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:34.768 DEBUG:boto:Data: 2024-04-16T20:07:34.768 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.768 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.768 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.768 DEBUG:boto:Params: {} 2024-04-16T20:07:34.768 DEBUG:boto:Token: None 2024-04-16T20:07:34.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148 2024-04-16T20:07:34.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8skj4QQvDQcfX9lcEhRBjT7ygZU= 2024-04-16T20:07:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8skj4QQvDQcfX9lcEhRBjT7ygZU=', 'Content-Length': '0'} 2024-04-16T20:07:34.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000010dbf5a56e772e-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.771 DEBUG:boto:path=/yrjefh-52/obj148 2024-04-16T20:07:34.771 DEBUG:boto:auth_path=/yrjefh-52/obj148 2024-04-16T20:07:34.771 DEBUG:boto:Method: GET 2024-04-16T20:07:34.771 DEBUG:boto:Path: /yrjefh-52/obj148 2024-04-16T20:07:34.771 DEBUG:boto:Data: 2024-04-16T20:07:34.771 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.771 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.771 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.771 DEBUG:boto:Params: {} 2024-04-16T20:07:34.771 DEBUG:boto:Token: None 2024-04-16T20:07:34.772 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj148 2024-04-16T20:07:34.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8skj4QQvDQcfX9lcEhRBjT7ygZU= 2024-04-16T20:07:34.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8skj4QQvDQcfX9lcEhRBjT7ygZU=', 'Content-Length': '0'} 2024-04-16T20:07:34.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfc22e1c6bedc6e-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.814 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.815 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.815 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.815 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.815 DEBUG:boto:Method: GET 2024-04-16T20:07:34.815 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.815 DEBUG:boto:Data: 2024-04-16T20:07:34.815 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.815 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.815 DEBUG:boto:Params: {} 2024-04-16T20:07:34.815 DEBUG:boto:Token: None 2024-04-16T20:07:34.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc16fb7361543c21-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.818 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.818 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.819 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.819 DEBUG:boto:path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.819 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.819 DEBUG:boto:Method: GET 2024-04-16T20:07:34.819 DEBUG:boto:Path: /yrjefh-52/?prefix=obj148 2024-04-16T20:07:34.819 DEBUG:boto:Data: 2024-04-16T20:07:34.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.819 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.819 DEBUG:boto:Params: {} 2024-04-16T20:07:34.819 DEBUG:boto:Token: None 2024-04-16T20:07:34.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7f1f6f2fd52851c-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.823 DEBUG:boto:b'yrjefh-52obj1481000falseobj1482024-04-16T20:06:00.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:34.824 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.824 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.824 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.824 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.824 DEBUG:boto:Method: GET 2024-04-16T20:07:34.824 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.824 DEBUG:boto:Data: 2024-04-16T20:07:34.824 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.824 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.824 DEBUG:boto:Params: {} 2024-04-16T20:07:34.825 DEBUG:boto:Token: None 2024-04-16T20:07:34.825 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.825 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc= 2024-04-16T20:07:34.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc=', 'Content-Length': '0'} 2024-04-16T20:07:34.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002107d2908d037472-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.827 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.827 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.827 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.827 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.827 DEBUG:boto:Method: GET 2024-04-16T20:07:34.828 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.828 DEBUG:boto:Data: 2024-04-16T20:07:34.828 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.828 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.828 DEBUG:boto:Params: {} 2024-04-16T20:07:34.828 DEBUG:boto:Token: None 2024-04-16T20:07:34.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc= 2024-04-16T20:07:34.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc=', 'Content-Length': '0'} 2024-04-16T20:07:34.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c47836744eb57ab-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.831 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.831 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.831 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.831 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.831 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.831 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.831 DEBUG:boto:Data: 2024-04-16T20:07:34.831 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.832 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.832 DEBUG:boto:Params: {} 2024-04-16T20:07:34.832 DEBUG:boto:Token: None 2024-04-16T20:07:34.832 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:08ks8ueXE445CHCW7AedJiORTL8= 2024-04-16T20:07:34.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:08ks8ueXE445CHCW7AedJiORTL8=', 'Content-Length': '0'} 2024-04-16T20:07:34.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f213c74618d7588-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.834 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.834 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.835 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.835 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.835 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.835 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.835 DEBUG:boto:Data: 2024-04-16T20:07:34.835 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.835 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.835 DEBUG:boto:Params: {} 2024-04-16T20:07:34.835 DEBUG:boto:Token: None 2024-04-16T20:07:34.835 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:08ks8ueXE445CHCW7AedJiORTL8= 2024-04-16T20:07:34.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:08ks8ueXE445CHCW7AedJiORTL8=', 'Content-Length': '0'} 2024-04-16T20:07:34.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d734779a0e99b5-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:34.838 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.838 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.838 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.838 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.838 DEBUG:boto:Method: GET 2024-04-16T20:07:34.839 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.839 DEBUG:boto:Data: 2024-04-16T20:07:34.839 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.839 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.839 DEBUG:boto:Params: {} 2024-04-16T20:07:34.839 DEBUG:boto:Token: None 2024-04-16T20:07:34.839 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc= 2024-04-16T20:07:34.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc=', 'Content-Length': '0'} 2024-04-16T20:07:34.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ae750968520c36-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.882 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.882 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.883 DEBUG:boto:path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.883 DEBUG:boto:auth_path=/yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.883 DEBUG:boto:Method: GET 2024-04-16T20:07:34.883 DEBUG:boto:Path: /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.883 DEBUG:boto:Data: 2024-04-16T20:07:34.883 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.883 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.883 DEBUG:boto:Params: {} 2024-04-16T20:07:34.883 DEBUG:boto:Token: None 2024-04-16T20:07:34.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155?versionId=null 2024-04-16T20:07:34.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc= 2024-04-16T20:07:34.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fmWuXNhddvSblQfXEbMIQoCtBDc=', 'Content-Length': '0'} 2024-04-16T20:07:34.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008965f1e5e242f6f6-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.886 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.886 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.886 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.887 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.887 DEBUG:boto:Method: GET 2024-04-16T20:07:34.887 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.887 DEBUG:boto:Data: 2024-04-16T20:07:34.887 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.887 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.887 DEBUG:boto:Params: {} 2024-04-16T20:07:34.887 DEBUG:boto:Token: None 2024-04-16T20:07:34.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001830a3ec61f1c253-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.890 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.891 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.891 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.891 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.891 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.891 DEBUG:boto:Method: GET 2024-04-16T20:07:34.891 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.891 DEBUG:boto:Data: 2024-04-16T20:07:34.891 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.891 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.892 DEBUG:boto:Params: {} 2024-04-16T20:07:34.892 DEBUG:boto:Token: None 2024-04-16T20:07:34.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d1347a1c555a5b-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.895 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.896 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.896 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.896 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.896 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:34.896 DEBUG:boto:Data: 2024-04-16T20:07:34.896 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.896 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.896 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.896 DEBUG:boto:Params: {} 2024-04-16T20:07:34.896 DEBUG:boto:Token: None 2024-04-16T20:07:34.897 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155 2024-04-16T20:07:34.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3vKVdTj0eXoFJEA//kZVYZ6U2Ec= 2024-04-16T20:07:34.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3vKVdTj0eXoFJEA//kZVYZ6U2Ec=', 'Content-Length': '0'} 2024-04-16T20:07:34.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009349633920ff3c18-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.899 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.899 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.899 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.899 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:34.899 DEBUG:boto:Data: 2024-04-16T20:07:34.900 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.900 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.900 DEBUG:boto:Params: {} 2024-04-16T20:07:34.900 DEBUG:boto:Token: None 2024-04-16T20:07:34.900 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155 2024-04-16T20:07:34.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3vKVdTj0eXoFJEA//kZVYZ6U2Ec= 2024-04-16T20:07:34.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3vKVdTj0eXoFJEA//kZVYZ6U2Ec=', 'Content-Length': '0'} 2024-04-16T20:07:34.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c85642c00d3e4cf-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:07:34.902 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.903 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.903 DEBUG:boto:Method: GET 2024-04-16T20:07:34.903 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:34.903 DEBUG:boto:Data: 2024-04-16T20:07:34.903 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.903 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.903 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.903 DEBUG:boto:Params: {} 2024-04-16T20:07:34.903 DEBUG:boto:Token: None 2024-04-16T20:07:34.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155 2024-04-16T20:07:34.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/b+eYnM+BxluemINQzvuWYx3t6o= 2024-04-16T20:07:34.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/b+eYnM+BxluemINQzvuWYx3t6o=', 'Content-Length': '0'} 2024-04-16T20:07:34.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad0d00ebad5457f-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.946 DEBUG:boto:path=/yrjefh-52/obj155 2024-04-16T20:07:34.946 DEBUG:boto:auth_path=/yrjefh-52/obj155 2024-04-16T20:07:34.947 DEBUG:boto:Method: GET 2024-04-16T20:07:34.947 DEBUG:boto:Path: /yrjefh-52/obj155 2024-04-16T20:07:34.947 DEBUG:boto:Data: 2024-04-16T20:07:34.947 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.947 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.947 DEBUG:boto:Params: {} 2024-04-16T20:07:34.947 DEBUG:boto:Token: None 2024-04-16T20:07:34.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj155 2024-04-16T20:07:34.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/b+eYnM+BxluemINQzvuWYx3t6o= 2024-04-16T20:07:34.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/b+eYnM+BxluemINQzvuWYx3t6o=', 'Content-Length': '0'} 2024-04-16T20:07:34.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ce2c7c3c5e9f81-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.950 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.950 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.950 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.950 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.951 DEBUG:boto:Method: GET 2024-04-16T20:07:34.951 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.951 DEBUG:boto:Data: 2024-04-16T20:07:34.951 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.951 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.951 DEBUG:boto:Params: {} 2024-04-16T20:07:34.951 DEBUG:boto:Token: None 2024-04-16T20:07:34.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ba1279fd41861e-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.954 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.954 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.954 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.955 DEBUG:boto:path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.955 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.955 DEBUG:boto:Method: GET 2024-04-16T20:07:34.955 DEBUG:boto:Path: /yrjefh-52/?prefix=obj155 2024-04-16T20:07:34.955 DEBUG:boto:Data: 2024-04-16T20:07:34.955 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.955 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.955 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.955 DEBUG:boto:Params: {} 2024-04-16T20:07:34.955 DEBUG:boto:Token: None 2024-04-16T20:07:34.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e82bb7707b85584d-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.959 DEBUG:boto:b'yrjefh-52obj1551000falseobj1552024-04-16T20:04:43.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:34.959 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.960 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.960 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.960 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.960 DEBUG:boto:Method: GET 2024-04-16T20:07:34.960 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.960 DEBUG:boto:Data: 2024-04-16T20:07:34.960 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.960 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.960 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.960 DEBUG:boto:Params: {} 2024-04-16T20:07:34.960 DEBUG:boto:Token: None 2024-04-16T20:07:34.961 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.961 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E= 2024-04-16T20:07:34.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E=', 'Content-Length': '0'} 2024-04-16T20:07:34.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027efb522b4fbafbb-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.963 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.963 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.963 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.963 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.963 DEBUG:boto:Method: GET 2024-04-16T20:07:34.963 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.963 DEBUG:boto:Data: 2024-04-16T20:07:34.964 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.964 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.964 DEBUG:boto:Params: {} 2024-04-16T20:07:34.964 DEBUG:boto:Token: None 2024-04-16T20:07:34.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E= 2024-04-16T20:07:34.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E=', 'Content-Length': '0'} 2024-04-16T20:07:34.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a2d2823acd10f3-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.966 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.966 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.966 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.966 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.966 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.967 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.967 DEBUG:boto:Data: 2024-04-16T20:07:34.967 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.967 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.967 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.967 DEBUG:boto:Params: {} 2024-04-16T20:07:34.967 DEBUG:boto:Token: None 2024-04-16T20:07:34.967 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.967 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tbMhIONhLERpyE2I3EIa3o1z6Tw= 2024-04-16T20:07:34.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tbMhIONhLERpyE2I3EIa3o1z6Tw=', 'Content-Length': '0'} 2024-04-16T20:07:34.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de586f418698716-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.969 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.969 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.969 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.970 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.970 DEBUG:boto:Method: HEAD 2024-04-16T20:07:34.970 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.970 DEBUG:boto:Data: 2024-04-16T20:07:34.970 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.970 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.970 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.970 DEBUG:boto:Params: {} 2024-04-16T20:07:34.970 DEBUG:boto:Token: None 2024-04-16T20:07:34.970 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tbMhIONhLERpyE2I3EIa3o1z6Tw= 2024-04-16T20:07:34.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tbMhIONhLERpyE2I3EIa3o1z6Tw=', 'Content-Length': '0'} 2024-04-16T20:07:34.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b89a5a525c5b6c-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:34.973 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.973 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.973 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.973 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.973 DEBUG:boto:Method: GET 2024-04-16T20:07:34.973 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.974 DEBUG:boto:Data: 2024-04-16T20:07:34.974 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.974 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.974 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.974 DEBUG:boto:Params: {} 2024-04-16T20:07:34.974 DEBUG:boto:Token: None 2024-04-16T20:07:34.974 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.974 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E= 2024-04-16T20:07:34.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E=', 'Content-Length': '0'} 2024-04-16T20:07:34.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bedcb58c1e69e1-00661eda86-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.976 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:34.976 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:34.976 DEBUG:boto:path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.977 DEBUG:boto:auth_path=/yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.977 DEBUG:boto:Method: GET 2024-04-16T20:07:34.977 DEBUG:boto:Path: /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.977 DEBUG:boto:Data: 2024-04-16T20:07:34.977 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.977 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.977 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.977 DEBUG:boto:Params: {} 2024-04-16T20:07:34.977 DEBUG:boto:Token: None 2024-04-16T20:07:34.977 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/obj17?versionId=null 2024-04-16T20:07:34.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E= 2024-04-16T20:07:34.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQuj6QosxAqhXN98CRFsc33kY9E=', 'Content-Length': '0'} 2024-04-16T20:07:34.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa12940c4b2503c2-00661eda86-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.980 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.980 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.980 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.980 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.980 DEBUG:boto:Method: GET 2024-04-16T20:07:34.980 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.980 DEBUG:boto:Data: 2024-04-16T20:07:34.980 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.980 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:34.980 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.980 DEBUG:boto:Params: {} 2024-04-16T20:07:34.981 DEBUG:boto:Token: None 2024-04-16T20:07:34.981 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd4b9cada90e584-00661eda86-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:34.983 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:34.984 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:34.984 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:34.984 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.984 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.984 DEBUG:boto:Method: GET 2024-04-16T20:07:34.984 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:34.984 DEBUG:boto:Data: 2024-04-16T20:07:34.984 DEBUG:boto:Headers: {} 2024-04-16T20:07:34.984 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:34.985 DEBUG:boto:Port: 8000 2024-04-16T20:07:34.985 DEBUG:boto:Params: {} 2024-04-16T20:07:34.985 DEBUG:boto:Token: None 2024-04-16T20:07:34.985 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:34 GMT /yrjefh-52/ 2024-04-16T20:07:34.985 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs= 2024-04-16T20:07:34.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:srwyNYX9zmwZtJZzHiOr/8y6sDs=', 'Content-Length': '0'} 2024-04-16T20:07:34.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e7f9a021ccc554-00661eda86-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.030 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.031 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:35.031 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:35.031 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.031 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:35.031 DEBUG:boto:Data: 2024-04-16T20:07:35.031 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.031 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.031 DEBUG:boto:Params: {} 2024-04-16T20:07:35.032 DEBUG:boto:Token: None 2024-04-16T20:07:35.032 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj17 2024-04-16T20:07:35.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hYwm4S2xUvN2SmEFOOjFe7lCoKk= 2024-04-16T20:07:35.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hYwm4S2xUvN2SmEFOOjFe7lCoKk=', 'Content-Length': '0'} 2024-04-16T20:07:35.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7e118a8b6b3df2-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.034 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:35.034 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:35.034 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.034 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:35.034 DEBUG:boto:Data: 2024-04-16T20:07:35.034 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.034 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.034 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.034 DEBUG:boto:Params: {} 2024-04-16T20:07:35.035 DEBUG:boto:Token: None 2024-04-16T20:07:35.035 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj17 2024-04-16T20:07:35.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hYwm4S2xUvN2SmEFOOjFe7lCoKk= 2024-04-16T20:07:35.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hYwm4S2xUvN2SmEFOOjFe7lCoKk=', 'Content-Length': '0'} 2024-04-16T20:07:35.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002952833bab020f61-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:07:35.037 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:35.037 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:35.037 DEBUG:boto:Method: GET 2024-04-16T20:07:35.037 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:35.038 DEBUG:boto:Data: 2024-04-16T20:07:35.038 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.038 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.038 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.038 DEBUG:boto:Params: {} 2024-04-16T20:07:35.038 DEBUG:boto:Token: None 2024-04-16T20:07:35.038 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj17 2024-04-16T20:07:35.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mecMHd17DVBBqJu4B94QZyg8lwE= 2024-04-16T20:07:35.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mecMHd17DVBBqJu4B94QZyg8lwE=', 'Content-Length': '0'} 2024-04-16T20:07:35.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c50fc6ae0bd80d-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.040 DEBUG:boto:path=/yrjefh-52/obj17 2024-04-16T20:07:35.040 DEBUG:boto:auth_path=/yrjefh-52/obj17 2024-04-16T20:07:35.041 DEBUG:boto:Method: GET 2024-04-16T20:07:35.041 DEBUG:boto:Path: /yrjefh-52/obj17 2024-04-16T20:07:35.041 DEBUG:boto:Data: 2024-04-16T20:07:35.041 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.041 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.041 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.041 DEBUG:boto:Params: {} 2024-04-16T20:07:35.041 DEBUG:boto:Token: None 2024-04-16T20:07:35.041 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj17 2024-04-16T20:07:35.041 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mecMHd17DVBBqJu4B94QZyg8lwE= 2024-04-16T20:07:35.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mecMHd17DVBBqJu4B94QZyg8lwE=', 'Content-Length': '0'} 2024-04-16T20:07:35.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087642f52672f3bc4-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.044 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.044 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.044 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.044 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.044 DEBUG:boto:Method: GET 2024-04-16T20:07:35.044 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.044 DEBUG:boto:Data: 2024-04-16T20:07:35.044 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.044 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.044 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.044 DEBUG:boto:Params: {} 2024-04-16T20:07:35.045 DEBUG:boto:Token: None 2024-04-16T20:07:35.045 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.045 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca1cb4931ccd5c5-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.048 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.048 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.048 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.048 DEBUG:boto:path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.048 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.048 DEBUG:boto:Method: GET 2024-04-16T20:07:35.048 DEBUG:boto:Path: /yrjefh-52/?prefix=obj17 2024-04-16T20:07:35.048 DEBUG:boto:Data: 2024-04-16T20:07:35.049 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.049 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.049 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.049 DEBUG:boto:Params: {} 2024-04-16T20:07:35.049 DEBUG:boto:Token: None 2024-04-16T20:07:35.049 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd5c24922363669-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.053 DEBUG:boto:b'yrjefh-52obj171000falseobj172024-04-16T19:58:15.843Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:35.053 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.053 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.053 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.054 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.054 DEBUG:boto:Method: GET 2024-04-16T20:07:35.054 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.054 DEBUG:boto:Data: 2024-04-16T20:07:35.054 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.054 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.054 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.054 DEBUG:boto:Params: {} 2024-04-16T20:07:35.054 DEBUG:boto:Token: None 2024-04-16T20:07:35.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM= 2024-04-16T20:07:35.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM=', 'Content-Length': '0'} 2024-04-16T20:07:35.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5bbe74d0fbabfa-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.098 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.098 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.099 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.099 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.099 DEBUG:boto:Method: GET 2024-04-16T20:07:35.099 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.099 DEBUG:boto:Data: 2024-04-16T20:07:35.099 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.099 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.099 DEBUG:boto:Params: {} 2024-04-16T20:07:35.099 DEBUG:boto:Token: None 2024-04-16T20:07:35.099 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM= 2024-04-16T20:07:35.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM=', 'Content-Length': '0'} 2024-04-16T20:07:35.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245e3d083c7964e9-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.102 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.102 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.102 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.102 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.102 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.102 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.102 DEBUG:boto:Data: 2024-04-16T20:07:35.102 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.102 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.102 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.102 DEBUG:boto:Params: {} 2024-04-16T20:07:35.103 DEBUG:boto:Token: None 2024-04-16T20:07:35.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WEyVeBpis6vC1uWY49QHu3fTm7Q= 2024-04-16T20:07:35.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WEyVeBpis6vC1uWY49QHu3fTm7Q=', 'Content-Length': '0'} 2024-04-16T20:07:35.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052280789bb6f26c2-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.105 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.105 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.105 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.105 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.106 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.106 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.106 DEBUG:boto:Data: 2024-04-16T20:07:35.106 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.106 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.106 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.106 DEBUG:boto:Params: {} 2024-04-16T20:07:35.106 DEBUG:boto:Token: None 2024-04-16T20:07:35.106 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WEyVeBpis6vC1uWY49QHu3fTm7Q= 2024-04-16T20:07:35.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WEyVeBpis6vC1uWY49QHu3fTm7Q=', 'Content-Length': '0'} 2024-04-16T20:07:35.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005430ab5dbdc40020-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:35.109 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.109 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.109 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.109 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.109 DEBUG:boto:Method: GET 2024-04-16T20:07:35.109 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.109 DEBUG:boto:Data: 2024-04-16T20:07:35.110 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.110 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.110 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.110 DEBUG:boto:Params: {} 2024-04-16T20:07:35.110 DEBUG:boto:Token: None 2024-04-16T20:07:35.110 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.110 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM= 2024-04-16T20:07:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM=', 'Content-Length': '0'} 2024-04-16T20:07:35.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011450b8f52e01958-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.112 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.113 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.113 DEBUG:boto:path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.113 DEBUG:boto:auth_path=/yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.113 DEBUG:boto:Method: GET 2024-04-16T20:07:35.113 DEBUG:boto:Path: /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.113 DEBUG:boto:Data: 2024-04-16T20:07:35.113 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.113 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.113 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.113 DEBUG:boto:Params: {} 2024-04-16T20:07:35.114 DEBUG:boto:Token: None 2024-04-16T20:07:35.114 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185?versionId=null 2024-04-16T20:07:35.114 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM= 2024-04-16T20:07:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dGOJ79FVEIOpR1ZSDbqh5waXqHM=', 'Content-Length': '0'} 2024-04-16T20:07:35.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000946bfbbe290eb82-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.116 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.116 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.116 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.116 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.116 DEBUG:boto:Method: GET 2024-04-16T20:07:35.116 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.116 DEBUG:boto:Data: 2024-04-16T20:07:35.117 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.117 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.117 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.117 DEBUG:boto:Params: {} 2024-04-16T20:07:35.117 DEBUG:boto:Token: None 2024-04-16T20:07:35.117 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aadf15298c64c8b-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.120 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.120 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.120 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.120 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.120 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.120 DEBUG:boto:Method: GET 2024-04-16T20:07:35.120 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.121 DEBUG:boto:Data: 2024-04-16T20:07:35.121 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.121 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.121 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.121 DEBUG:boto:Params: {} 2024-04-16T20:07:35.121 DEBUG:boto:Token: None 2024-04-16T20:07:35.121 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a732f0af65f8d24-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.125 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.125 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.125 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.125 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.125 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:35.125 DEBUG:boto:Data: 2024-04-16T20:07:35.125 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.126 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.126 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.126 DEBUG:boto:Params: {} 2024-04-16T20:07:35.126 DEBUG:boto:Token: None 2024-04-16T20:07:35.126 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185 2024-04-16T20:07:35.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KXtQaBK+4njoiIQz8LKOXQr41VQ= 2024-04-16T20:07:35.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KXtQaBK+4njoiIQz8LKOXQr41VQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001972880d76ba9827-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.128 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.129 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.129 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.129 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:35.129 DEBUG:boto:Data: 2024-04-16T20:07:35.129 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.130 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.130 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.130 DEBUG:boto:Params: {} 2024-04-16T20:07:35.130 DEBUG:boto:Token: None 2024-04-16T20:07:35.130 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185 2024-04-16T20:07:35.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KXtQaBK+4njoiIQz8LKOXQr41VQ= 2024-04-16T20:07:35.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KXtQaBK+4njoiIQz8LKOXQr41VQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2a21871f566428-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:07:35.134 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.134 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.134 DEBUG:boto:Method: GET 2024-04-16T20:07:35.134 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:35.134 DEBUG:boto:Data: 2024-04-16T20:07:35.135 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.135 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.135 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.135 DEBUG:boto:Params: {} 2024-04-16T20:07:35.135 DEBUG:boto:Token: None 2024-04-16T20:07:35.136 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185 2024-04-16T20:07:35.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kWTRCia8uFsFbk0hckdhcn1SvB0= 2024-04-16T20:07:35.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kWTRCia8uFsFbk0hckdhcn1SvB0=', 'Content-Length': '0'} 2024-04-16T20:07:35.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d171697813c44c02-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.139 DEBUG:boto:path=/yrjefh-52/obj185 2024-04-16T20:07:35.139 DEBUG:boto:auth_path=/yrjefh-52/obj185 2024-04-16T20:07:35.139 DEBUG:boto:Method: GET 2024-04-16T20:07:35.139 DEBUG:boto:Path: /yrjefh-52/obj185 2024-04-16T20:07:35.139 DEBUG:boto:Data: 2024-04-16T20:07:35.140 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.140 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.140 DEBUG:boto:Params: {} 2024-04-16T20:07:35.140 DEBUG:boto:Token: None 2024-04-16T20:07:35.141 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj185 2024-04-16T20:07:35.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kWTRCia8uFsFbk0hckdhcn1SvB0= 2024-04-16T20:07:35.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kWTRCia8uFsFbk0hckdhcn1SvB0=', 'Content-Length': '0'} 2024-04-16T20:07:35.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c54ee60668a112b-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.186 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.186 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.187 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.187 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.187 DEBUG:boto:Method: GET 2024-04-16T20:07:35.187 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.187 DEBUG:boto:Data: 2024-04-16T20:07:35.187 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.187 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.187 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.187 DEBUG:boto:Params: {} 2024-04-16T20:07:35.187 DEBUG:boto:Token: None 2024-04-16T20:07:35.187 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a094a00c78e43a5-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.190 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.190 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.191 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.191 DEBUG:boto:path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.191 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.191 DEBUG:boto:Method: GET 2024-04-16T20:07:35.191 DEBUG:boto:Path: /yrjefh-52/?prefix=obj185 2024-04-16T20:07:35.191 DEBUG:boto:Data: 2024-04-16T20:07:35.191 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.191 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.191 DEBUG:boto:Params: {} 2024-04-16T20:07:35.191 DEBUG:boto:Token: None 2024-04-16T20:07:35.191 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069612f3db145d49c-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.196 DEBUG:boto:b'yrjefh-52obj1851000falseobj1852024-04-16T20:06:00.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:35.196 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.196 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.196 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.196 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.196 DEBUG:boto:Method: GET 2024-04-16T20:07:35.197 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.197 DEBUG:boto:Data: 2024-04-16T20:07:35.197 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.197 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.197 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.197 DEBUG:boto:Params: {} 2024-04-16T20:07:35.197 DEBUG:boto:Token: None 2024-04-16T20:07:35.197 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8= 2024-04-16T20:07:35.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8=', 'Content-Length': '0'} 2024-04-16T20:07:35.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042016f1f50484642-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.200 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.200 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.200 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.200 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.200 DEBUG:boto:Method: GET 2024-04-16T20:07:35.201 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.201 DEBUG:boto:Data: 2024-04-16T20:07:35.201 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.201 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.201 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.201 DEBUG:boto:Params: {} 2024-04-16T20:07:35.201 DEBUG:boto:Token: None 2024-04-16T20:07:35.201 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.201 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8= 2024-04-16T20:07:35.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8=', 'Content-Length': '0'} 2024-04-16T20:07:35.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d8a0f018fa9e7c-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.204 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.204 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.204 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.204 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.204 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.204 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.204 DEBUG:boto:Data: 2024-04-16T20:07:35.204 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.204 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.205 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.205 DEBUG:boto:Params: {} 2024-04-16T20:07:35.205 DEBUG:boto:Token: None 2024-04-16T20:07:35.205 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.205 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVWqszWm9w54vyMpuQV2emLOK+c= 2024-04-16T20:07:35.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVWqszWm9w54vyMpuQV2emLOK+c=', 'Content-Length': '0'} 2024-04-16T20:07:35.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8266858e28408b-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.207 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.208 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.208 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.208 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.208 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.208 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.208 DEBUG:boto:Data: 2024-04-16T20:07:35.208 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.208 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.208 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.208 DEBUG:boto:Params: {} 2024-04-16T20:07:35.208 DEBUG:boto:Token: None 2024-04-16T20:07:35.209 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zVWqszWm9w54vyMpuQV2emLOK+c= 2024-04-16T20:07:35.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zVWqszWm9w54vyMpuQV2emLOK+c=', 'Content-Length': '0'} 2024-04-16T20:07:35.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003631dbedadf517af-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:35.211 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.211 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.211 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.211 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.211 DEBUG:boto:Method: GET 2024-04-16T20:07:35.211 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.212 DEBUG:boto:Data: 2024-04-16T20:07:35.212 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.212 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.212 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.212 DEBUG:boto:Params: {} 2024-04-16T20:07:35.212 DEBUG:boto:Token: None 2024-04-16T20:07:35.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8= 2024-04-16T20:07:35.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8=', 'Content-Length': '0'} 2024-04-16T20:07:35.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b52f89703315130f-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.258 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.259 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.259 DEBUG:boto:path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.259 DEBUG:boto:auth_path=/yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.259 DEBUG:boto:Method: GET 2024-04-16T20:07:35.259 DEBUG:boto:Path: /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.259 DEBUG:boto:Data: 2024-04-16T20:07:35.259 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.259 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.259 DEBUG:boto:Params: {} 2024-04-16T20:07:35.259 DEBUG:boto:Token: None 2024-04-16T20:07:35.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19?versionId=null 2024-04-16T20:07:35.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8= 2024-04-16T20:07:35.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KLROa7iPc3f4FjEheRhGiFtFlT8=', 'Content-Length': '0'} 2024-04-16T20:07:35.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8df447b3098cf9d-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.262 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.262 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.262 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.262 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.263 DEBUG:boto:Method: GET 2024-04-16T20:07:35.263 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.263 DEBUG:boto:Data: 2024-04-16T20:07:35.263 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.263 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.263 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.263 DEBUG:boto:Params: {} 2024-04-16T20:07:35.263 DEBUG:boto:Token: None 2024-04-16T20:07:35.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf874bb99900bd0-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.266 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.266 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.266 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.266 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.266 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.267 DEBUG:boto:Method: GET 2024-04-16T20:07:35.267 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.267 DEBUG:boto:Data: 2024-04-16T20:07:35.267 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.267 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.267 DEBUG:boto:Params: {} 2024-04-16T20:07:35.267 DEBUG:boto:Token: None 2024-04-16T20:07:35.267 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ba33529c17bbe2-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.271 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.271 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.271 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.271 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.272 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:35.272 DEBUG:boto:Data: 2024-04-16T20:07:35.272 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.272 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.272 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.272 DEBUG:boto:Params: {} 2024-04-16T20:07:35.272 DEBUG:boto:Token: None 2024-04-16T20:07:35.272 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19 2024-04-16T20:07:35.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VvypGQTJf3byn384Fmz6QtXs9sQ= 2024-04-16T20:07:35.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VvypGQTJf3byn384Fmz6QtXs9sQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b09a43d1ffd406-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.275 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.275 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.275 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.275 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:35.275 DEBUG:boto:Data: 2024-04-16T20:07:35.275 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.275 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.275 DEBUG:boto:Params: {} 2024-04-16T20:07:35.276 DEBUG:boto:Token: None 2024-04-16T20:07:35.276 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19 2024-04-16T20:07:35.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VvypGQTJf3byn384Fmz6QtXs9sQ= 2024-04-16T20:07:35.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VvypGQTJf3byn384Fmz6QtXs9sQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb8f0278659e3480-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:07:35.279 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.279 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.279 DEBUG:boto:Method: GET 2024-04-16T20:07:35.279 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:35.279 DEBUG:boto:Data: 2024-04-16T20:07:35.279 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.279 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.279 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.279 DEBUG:boto:Params: {} 2024-04-16T20:07:35.279 DEBUG:boto:Token: None 2024-04-16T20:07:35.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19 2024-04-16T20:07:35.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hsunv4Y2L8yZUQ1iwsoWrzn8M6U= 2024-04-16T20:07:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hsunv4Y2L8yZUQ1iwsoWrzn8M6U=', 'Content-Length': '0'} 2024-04-16T20:07:35.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279cbc8fa7cc9be8-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.282 DEBUG:boto:path=/yrjefh-52/obj19 2024-04-16T20:07:35.282 DEBUG:boto:auth_path=/yrjefh-52/obj19 2024-04-16T20:07:35.282 DEBUG:boto:Method: GET 2024-04-16T20:07:35.282 DEBUG:boto:Path: /yrjefh-52/obj19 2024-04-16T20:07:35.282 DEBUG:boto:Data: 2024-04-16T20:07:35.282 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.282 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.283 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.283 DEBUG:boto:Params: {} 2024-04-16T20:07:35.283 DEBUG:boto:Token: None 2024-04-16T20:07:35.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj19 2024-04-16T20:07:35.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hsunv4Y2L8yZUQ1iwsoWrzn8M6U= 2024-04-16T20:07:35.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hsunv4Y2L8yZUQ1iwsoWrzn8M6U=', 'Content-Length': '0'} 2024-04-16T20:07:35.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007369c1287a6b4dd5-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.286 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.286 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.286 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.286 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.286 DEBUG:boto:Method: GET 2024-04-16T20:07:35.286 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.286 DEBUG:boto:Data: 2024-04-16T20:07:35.286 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.286 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.286 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.287 DEBUG:boto:Params: {} 2024-04-16T20:07:35.287 DEBUG:boto:Token: None 2024-04-16T20:07:35.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e90e6dffc44846-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.289 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.290 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.290 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.290 DEBUG:boto:path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.290 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.290 DEBUG:boto:Method: GET 2024-04-16T20:07:35.290 DEBUG:boto:Path: /yrjefh-52/?prefix=obj19 2024-04-16T20:07:35.290 DEBUG:boto:Data: 2024-04-16T20:07:35.290 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.290 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.290 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.290 DEBUG:boto:Params: {} 2024-04-16T20:07:35.291 DEBUG:boto:Token: None 2024-04-16T20:07:35.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ebca6a26c0724e-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.294 DEBUG:boto:b'yrjefh-52obj191000falseobj192024-04-16T20:00:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:35.295 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.295 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.295 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.295 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.295 DEBUG:boto:Method: GET 2024-04-16T20:07:35.295 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.295 DEBUG:boto:Data: 2024-04-16T20:07:35.296 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.296 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.296 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.296 DEBUG:boto:Params: {} 2024-04-16T20:07:35.296 DEBUG:boto:Token: None 2024-04-16T20:07:35.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE= 2024-04-16T20:07:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE=', 'Content-Length': '0'} 2024-04-16T20:07:35.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e620d9e655036cab-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.299 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.299 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.299 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.299 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.299 DEBUG:boto:Method: GET 2024-04-16T20:07:35.299 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.299 DEBUG:boto:Data: 2024-04-16T20:07:35.299 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.299 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.299 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.299 DEBUG:boto:Params: {} 2024-04-16T20:07:35.300 DEBUG:boto:Token: None 2024-04-16T20:07:35.300 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.300 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE= 2024-04-16T20:07:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE=', 'Content-Length': '0'} 2024-04-16T20:07:35.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9933bdf9c360c5b-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.302 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.302 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.302 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.302 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.302 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.302 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.302 DEBUG:boto:Data: 2024-04-16T20:07:35.302 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.303 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.303 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.303 DEBUG:boto:Params: {} 2024-04-16T20:07:35.303 DEBUG:boto:Token: None 2024-04-16T20:07:35.303 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.303 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0VQWFAWGoCLws7/4SufqP/K5IAI= 2024-04-16T20:07:35.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0VQWFAWGoCLws7/4SufqP/K5IAI=', 'Content-Length': '0'} 2024-04-16T20:07:35.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f0e085588646db-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.305 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.305 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.305 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.305 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.306 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.306 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.306 DEBUG:boto:Data: 2024-04-16T20:07:35.306 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.306 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.306 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.306 DEBUG:boto:Params: {} 2024-04-16T20:07:35.306 DEBUG:boto:Token: None 2024-04-16T20:07:35.306 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.306 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0VQWFAWGoCLws7/4SufqP/K5IAI= 2024-04-16T20:07:35.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0VQWFAWGoCLws7/4SufqP/K5IAI=', 'Content-Length': '0'} 2024-04-16T20:07:35.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d38da950a4bdae1-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:35.308 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.309 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.309 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.309 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.309 DEBUG:boto:Method: GET 2024-04-16T20:07:35.309 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.309 DEBUG:boto:Data: 2024-04-16T20:07:35.309 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.309 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.309 DEBUG:boto:Params: {} 2024-04-16T20:07:35.309 DEBUG:boto:Token: None 2024-04-16T20:07:35.310 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.310 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE= 2024-04-16T20:07:35.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE=', 'Content-Length': '0'} 2024-04-16T20:07:35.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c885e52a7d975b-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.354 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.354 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.355 DEBUG:boto:path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.355 DEBUG:boto:auth_path=/yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.355 DEBUG:boto:Method: GET 2024-04-16T20:07:35.355 DEBUG:boto:Path: /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.355 DEBUG:boto:Data: 2024-04-16T20:07:35.355 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.355 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.355 DEBUG:boto:Params: {} 2024-04-16T20:07:35.355 DEBUG:boto:Token: None 2024-04-16T20:07:35.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22?versionId=null 2024-04-16T20:07:35.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE= 2024-04-16T20:07:35.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ux9tqXmY/wIOu5gi4s6DqaZzjCE=', 'Content-Length': '0'} 2024-04-16T20:07:35.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c9c526743982fb-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.358 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.358 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.358 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.358 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.358 DEBUG:boto:Method: GET 2024-04-16T20:07:35.358 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.358 DEBUG:boto:Data: 2024-04-16T20:07:35.358 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.359 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.359 DEBUG:boto:Params: {} 2024-04-16T20:07:35.359 DEBUG:boto:Token: None 2024-04-16T20:07:35.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000857565a4a6f4e37c-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.362 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.362 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.362 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.362 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.362 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.363 DEBUG:boto:Method: GET 2024-04-16T20:07:35.363 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.363 DEBUG:boto:Data: 2024-04-16T20:07:35.363 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.363 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.364 DEBUG:boto:Params: {} 2024-04-16T20:07:35.364 DEBUG:boto:Token: None 2024-04-16T20:07:35.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f362fba581663a20-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.367 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.368 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.368 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.368 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.368 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:35.368 DEBUG:boto:Data: 2024-04-16T20:07:35.368 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.368 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.368 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.368 DEBUG:boto:Params: {} 2024-04-16T20:07:35.369 DEBUG:boto:Token: None 2024-04-16T20:07:35.369 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22 2024-04-16T20:07:35.369 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LyOspyUa1ugbNb6iEfnJkDxvZqc= 2024-04-16T20:07:35.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LyOspyUa1ugbNb6iEfnJkDxvZqc=', 'Content-Length': '0'} 2024-04-16T20:07:35.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a3689b4bc6326e-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.371 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.371 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.371 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.371 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:35.371 DEBUG:boto:Data: 2024-04-16T20:07:35.371 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.371 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.371 DEBUG:boto:Params: {} 2024-04-16T20:07:35.372 DEBUG:boto:Token: None 2024-04-16T20:07:35.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22 2024-04-16T20:07:35.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LyOspyUa1ugbNb6iEfnJkDxvZqc= 2024-04-16T20:07:35.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LyOspyUa1ugbNb6iEfnJkDxvZqc=', 'Content-Length': '0'} 2024-04-16T20:07:35.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023619b4e2cb07622-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:07:35.374 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.374 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.374 DEBUG:boto:Method: GET 2024-04-16T20:07:35.374 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:35.374 DEBUG:boto:Data: 2024-04-16T20:07:35.375 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.375 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.375 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.375 DEBUG:boto:Params: {} 2024-04-16T20:07:35.375 DEBUG:boto:Token: None 2024-04-16T20:07:35.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22 2024-04-16T20:07:35.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bX1tGA7XN6vhMJchcxbCEXKU2LU= 2024-04-16T20:07:35.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bX1tGA7XN6vhMJchcxbCEXKU2LU=', 'Content-Length': '0'} 2024-04-16T20:07:35.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e8a866e16f16a0-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.377 DEBUG:boto:path=/yrjefh-52/obj22 2024-04-16T20:07:35.377 DEBUG:boto:auth_path=/yrjefh-52/obj22 2024-04-16T20:07:35.377 DEBUG:boto:Method: GET 2024-04-16T20:07:35.378 DEBUG:boto:Path: /yrjefh-52/obj22 2024-04-16T20:07:35.378 DEBUG:boto:Data: 2024-04-16T20:07:35.378 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.378 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.378 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.378 DEBUG:boto:Params: {} 2024-04-16T20:07:35.378 DEBUG:boto:Token: None 2024-04-16T20:07:35.378 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj22 2024-04-16T20:07:35.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bX1tGA7XN6vhMJchcxbCEXKU2LU= 2024-04-16T20:07:35.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bX1tGA7XN6vhMJchcxbCEXKU2LU=', 'Content-Length': '0'} 2024-04-16T20:07:35.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c12e128a6af2a6d-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.422 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.423 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.423 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.423 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.423 DEBUG:boto:Method: GET 2024-04-16T20:07:35.423 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.423 DEBUG:boto:Data: 2024-04-16T20:07:35.423 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.423 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.423 DEBUG:boto:Params: {} 2024-04-16T20:07:35.423 DEBUG:boto:Token: None 2024-04-16T20:07:35.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.424 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fd1668f3a053165-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.429 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.430 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.430 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.430 DEBUG:boto:path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.430 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.430 DEBUG:boto:Method: GET 2024-04-16T20:07:35.430 DEBUG:boto:Path: /yrjefh-52/?prefix=obj22 2024-04-16T20:07:35.431 DEBUG:boto:Data: 2024-04-16T20:07:35.431 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.431 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.431 DEBUG:boto:Params: {} 2024-04-16T20:07:35.431 DEBUG:boto:Token: None 2024-04-16T20:07:35.431 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be05a8af52f76155-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.435 DEBUG:boto:b'yrjefh-52obj221000falseobj222024-04-16T19:58:04.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:35.435 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.435 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.435 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.435 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.436 DEBUG:boto:Method: GET 2024-04-16T20:07:35.436 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.436 DEBUG:boto:Data: 2024-04-16T20:07:35.436 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.436 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.436 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.436 DEBUG:boto:Params: {} 2024-04-16T20:07:35.436 DEBUG:boto:Token: None 2024-04-16T20:07:35.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w= 2024-04-16T20:07:35.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w=', 'Content-Length': '0'} 2024-04-16T20:07:35.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d4a537cc07366e-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.482 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.482 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.483 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.483 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.483 DEBUG:boto:Method: GET 2024-04-16T20:07:35.483 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.483 DEBUG:boto:Data: 2024-04-16T20:07:35.483 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.483 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.483 DEBUG:boto:Params: {} 2024-04-16T20:07:35.483 DEBUG:boto:Token: None 2024-04-16T20:07:35.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w= 2024-04-16T20:07:35.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w=', 'Content-Length': '0'} 2024-04-16T20:07:35.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9599b5c2e6aadac-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.486 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.487 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.487 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.487 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.487 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.487 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.487 DEBUG:boto:Data: 2024-04-16T20:07:35.487 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.487 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.487 DEBUG:boto:Params: {} 2024-04-16T20:07:35.487 DEBUG:boto:Token: None 2024-04-16T20:07:35.488 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VDzSKsodUcqUTSMel3cmJJ/z4QA= 2024-04-16T20:07:35.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VDzSKsodUcqUTSMel3cmJJ/z4QA=', 'Content-Length': '0'} 2024-04-16T20:07:35.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057661f73059e507c-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.490 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.490 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.490 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.490 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.490 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.490 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.491 DEBUG:boto:Data: 2024-04-16T20:07:35.491 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.491 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.491 DEBUG:boto:Params: {} 2024-04-16T20:07:35.491 DEBUG:boto:Token: None 2024-04-16T20:07:35.491 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VDzSKsodUcqUTSMel3cmJJ/z4QA= 2024-04-16T20:07:35.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VDzSKsodUcqUTSMel3cmJJ/z4QA=', 'Content-Length': '0'} 2024-04-16T20:07:35.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005575012093675325-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:35.494 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.494 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.494 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.494 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.494 DEBUG:boto:Method: GET 2024-04-16T20:07:35.494 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.494 DEBUG:boto:Data: 2024-04-16T20:07:35.494 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.494 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.494 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.494 DEBUG:boto:Params: {} 2024-04-16T20:07:35.494 DEBUG:boto:Token: None 2024-04-16T20:07:35.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w= 2024-04-16T20:07:35.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w=', 'Content-Length': '0'} 2024-04-16T20:07:35.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6633a1b7f93dc5-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.497 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.497 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.498 DEBUG:boto:path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.498 DEBUG:boto:auth_path=/yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.498 DEBUG:boto:Method: GET 2024-04-16T20:07:35.498 DEBUG:boto:Path: /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.498 DEBUG:boto:Data: 2024-04-16T20:07:35.498 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.498 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.498 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.498 DEBUG:boto:Params: {} 2024-04-16T20:07:35.498 DEBUG:boto:Token: None 2024-04-16T20:07:35.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23?versionId=null 2024-04-16T20:07:35.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w= 2024-04-16T20:07:35.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:07o8eGDwpyefyjUQvbk5toTpg+w=', 'Content-Length': '0'} 2024-04-16T20:07:35.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000057dd63d7b01ad8f-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.501 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.501 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.502 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.502 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.502 DEBUG:boto:Method: GET 2024-04-16T20:07:35.502 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.502 DEBUG:boto:Data: 2024-04-16T20:07:35.502 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.502 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.502 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.502 DEBUG:boto:Params: {} 2024-04-16T20:07:35.502 DEBUG:boto:Token: None 2024-04-16T20:07:35.502 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074be174d04e0e33f-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.505 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.505 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.506 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.506 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.506 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.506 DEBUG:boto:Method: GET 2024-04-16T20:07:35.506 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.506 DEBUG:boto:Data: 2024-04-16T20:07:35.506 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.506 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.506 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.506 DEBUG:boto:Params: {} 2024-04-16T20:07:35.506 DEBUG:boto:Token: None 2024-04-16T20:07:35.506 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ea83492afa52fe-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.510 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.510 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.510 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.510 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.511 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:35.511 DEBUG:boto:Data: 2024-04-16T20:07:35.511 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.511 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.511 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.511 DEBUG:boto:Params: {} 2024-04-16T20:07:35.511 DEBUG:boto:Token: None 2024-04-16T20:07:35.511 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23 2024-04-16T20:07:35.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xsH6TcRt5VCiMG8+qgNkNCO9SxY= 2024-04-16T20:07:35.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xsH6TcRt5VCiMG8+qgNkNCO9SxY=', 'Content-Length': '0'} 2024-04-16T20:07:35.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b88772eca6fde7a-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.514 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.514 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.514 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.514 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:35.514 DEBUG:boto:Data: 2024-04-16T20:07:35.514 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.514 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.514 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.514 DEBUG:boto:Params: {} 2024-04-16T20:07:35.514 DEBUG:boto:Token: None 2024-04-16T20:07:35.515 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23 2024-04-16T20:07:35.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xsH6TcRt5VCiMG8+qgNkNCO9SxY= 2024-04-16T20:07:35.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xsH6TcRt5VCiMG8+qgNkNCO9SxY=', 'Content-Length': '0'} 2024-04-16T20:07:35.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d205efd5bf753dc7-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:07:35.517 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.517 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.517 DEBUG:boto:Method: GET 2024-04-16T20:07:35.517 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:35.518 DEBUG:boto:Data: 2024-04-16T20:07:35.518 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.518 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.518 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.518 DEBUG:boto:Params: {} 2024-04-16T20:07:35.518 DEBUG:boto:Token: None 2024-04-16T20:07:35.518 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23 2024-04-16T20:07:35.518 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3jDVW0M2o0NJv2F4TYUCvNaSv2Y= 2024-04-16T20:07:35.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3jDVW0M2o0NJv2F4TYUCvNaSv2Y=', 'Content-Length': '0'} 2024-04-16T20:07:35.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006283a24a561a74ae-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.521 DEBUG:boto:path=/yrjefh-52/obj23 2024-04-16T20:07:35.521 DEBUG:boto:auth_path=/yrjefh-52/obj23 2024-04-16T20:07:35.521 DEBUG:boto:Method: GET 2024-04-16T20:07:35.521 DEBUG:boto:Path: /yrjefh-52/obj23 2024-04-16T20:07:35.521 DEBUG:boto:Data: 2024-04-16T20:07:35.521 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.521 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.521 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.522 DEBUG:boto:Params: {} 2024-04-16T20:07:35.522 DEBUG:boto:Token: None 2024-04-16T20:07:35.522 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj23 2024-04-16T20:07:35.522 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3jDVW0M2o0NJv2F4TYUCvNaSv2Y= 2024-04-16T20:07:35.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3jDVW0M2o0NJv2F4TYUCvNaSv2Y=', 'Content-Length': '0'} 2024-04-16T20:07:35.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c544598c7f9822f2-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.525 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.525 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.525 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.525 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.525 DEBUG:boto:Method: GET 2024-04-16T20:07:35.525 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.525 DEBUG:boto:Data: 2024-04-16T20:07:35.525 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.525 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.525 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.525 DEBUG:boto:Params: {} 2024-04-16T20:07:35.526 DEBUG:boto:Token: None 2024-04-16T20:07:35.526 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0527fc7cf3f5bed-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.528 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.529 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.529 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.529 DEBUG:boto:path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.529 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.529 DEBUG:boto:Method: GET 2024-04-16T20:07:35.529 DEBUG:boto:Path: /yrjefh-52/?prefix=obj23 2024-04-16T20:07:35.529 DEBUG:boto:Data: 2024-04-16T20:07:35.529 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.529 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.529 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.530 DEBUG:boto:Params: {} 2024-04-16T20:07:35.530 DEBUG:boto:Token: None 2024-04-16T20:07:35.530 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.530 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c0551bb125d5390-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.533 DEBUG:boto:b'yrjefh-52obj231000falseobj232024-04-16T20:01:27.573Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:35.534 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.534 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.534 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.534 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.534 DEBUG:boto:Method: GET 2024-04-16T20:07:35.534 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.535 DEBUG:boto:Data: 2024-04-16T20:07:35.535 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.535 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.535 DEBUG:boto:Params: {} 2024-04-16T20:07:35.535 DEBUG:boto:Token: None 2024-04-16T20:07:35.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE= 2024-04-16T20:07:35.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE=', 'Content-Length': '0'} 2024-04-16T20:07:35.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c54885c1238b94-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.538 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.538 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.538 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.538 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.538 DEBUG:boto:Method: GET 2024-04-16T20:07:35.538 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.538 DEBUG:boto:Data: 2024-04-16T20:07:35.538 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.539 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.539 DEBUG:boto:Params: {} 2024-04-16T20:07:35.539 DEBUG:boto:Token: None 2024-04-16T20:07:35.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE= 2024-04-16T20:07:35.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE=', 'Content-Length': '0'} 2024-04-16T20:07:35.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002500f689b070497f-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.541 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.541 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.541 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.541 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.541 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.542 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.542 DEBUG:boto:Data: 2024-04-16T20:07:35.542 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.542 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.542 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.542 DEBUG:boto:Params: {} 2024-04-16T20:07:35.542 DEBUG:boto:Token: None 2024-04-16T20:07:35.542 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZQd4OprVawLmXpjn/REuQU7nG4= 2024-04-16T20:07:35.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZQd4OprVawLmXpjn/REuQU7nG4=', 'Content-Length': '0'} 2024-04-16T20:07:35.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a45b649c49a00e-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.544 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.544 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.545 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.545 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.545 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.545 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.545 DEBUG:boto:Data: 2024-04-16T20:07:35.545 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.545 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.545 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.545 DEBUG:boto:Params: {} 2024-04-16T20:07:35.545 DEBUG:boto:Token: None 2024-04-16T20:07:35.546 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oZQd4OprVawLmXpjn/REuQU7nG4= 2024-04-16T20:07:35.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oZQd4OprVawLmXpjn/REuQU7nG4=', 'Content-Length': '0'} 2024-04-16T20:07:35.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c866a6d1750ccb1b-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:35.548 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.548 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.548 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.548 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.548 DEBUG:boto:Method: GET 2024-04-16T20:07:35.548 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.548 DEBUG:boto:Data: 2024-04-16T20:07:35.549 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.549 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.549 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.549 DEBUG:boto:Params: {} 2024-04-16T20:07:35.549 DEBUG:boto:Token: None 2024-04-16T20:07:35.549 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE= 2024-04-16T20:07:35.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE=', 'Content-Length': '0'} 2024-04-16T20:07:35.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b67104b89345e0-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.551 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.551 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.551 DEBUG:boto:path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.552 DEBUG:boto:auth_path=/yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.552 DEBUG:boto:Method: GET 2024-04-16T20:07:35.552 DEBUG:boto:Path: /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.552 DEBUG:boto:Data: 2024-04-16T20:07:35.552 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.552 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.552 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.552 DEBUG:boto:Params: {} 2024-04-16T20:07:35.552 DEBUG:boto:Token: None 2024-04-16T20:07:35.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29?versionId=null 2024-04-16T20:07:35.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE= 2024-04-16T20:07:35.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i0lmFhQzlO9fBIoAjk5vS0LJEzE=', 'Content-Length': '0'} 2024-04-16T20:07:35.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d66ab566982147-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.555 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.555 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.555 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.555 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.555 DEBUG:boto:Method: GET 2024-04-16T20:07:35.555 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.555 DEBUG:boto:Data: 2024-04-16T20:07:35.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.556 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.556 DEBUG:boto:Params: {} 2024-04-16T20:07:35.556 DEBUG:boto:Token: None 2024-04-16T20:07:35.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003013030b200e36c9-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.559 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.559 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.559 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.559 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.559 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.559 DEBUG:boto:Method: GET 2024-04-16T20:07:35.559 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.559 DEBUG:boto:Data: 2024-04-16T20:07:35.559 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.560 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.560 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.560 DEBUG:boto:Params: {} 2024-04-16T20:07:35.560 DEBUG:boto:Token: None 2024-04-16T20:07:35.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf0e9c8c547ec09-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.606 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.607 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.607 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.607 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.607 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:35.607 DEBUG:boto:Data: 2024-04-16T20:07:35.607 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.607 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.607 DEBUG:boto:Params: {} 2024-04-16T20:07:35.607 DEBUG:boto:Token: None 2024-04-16T20:07:35.607 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29 2024-04-16T20:07:35.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/zjl1I7Y4SjT5wy5+H3oaPO7lso= 2024-04-16T20:07:35.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/zjl1I7Y4SjT5wy5+H3oaPO7lso=', 'Content-Length': '0'} 2024-04-16T20:07:35.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbb31da42a153ef-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.610 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.610 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.610 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.610 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:35.610 DEBUG:boto:Data: 2024-04-16T20:07:35.610 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.610 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.610 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.610 DEBUG:boto:Params: {} 2024-04-16T20:07:35.611 DEBUG:boto:Token: None 2024-04-16T20:07:35.611 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29 2024-04-16T20:07:35.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/zjl1I7Y4SjT5wy5+H3oaPO7lso= 2024-04-16T20:07:35.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/zjl1I7Y4SjT5wy5+H3oaPO7lso=', 'Content-Length': '0'} 2024-04-16T20:07:35.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6a481ad7a20e0c-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:07:35.613 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.613 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.613 DEBUG:boto:Method: GET 2024-04-16T20:07:35.613 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:35.613 DEBUG:boto:Data: 2024-04-16T20:07:35.613 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.613 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.613 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.613 DEBUG:boto:Params: {} 2024-04-16T20:07:35.614 DEBUG:boto:Token: None 2024-04-16T20:07:35.614 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29 2024-04-16T20:07:35.614 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1A2dNbn2za+alujDjWxxBM9NuFU= 2024-04-16T20:07:35.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1A2dNbn2za+alujDjWxxBM9NuFU=', 'Content-Length': '0'} 2024-04-16T20:07:35.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db071f07f50b9c6-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.658 DEBUG:boto:path=/yrjefh-52/obj29 2024-04-16T20:07:35.659 DEBUG:boto:auth_path=/yrjefh-52/obj29 2024-04-16T20:07:35.659 DEBUG:boto:Method: GET 2024-04-16T20:07:35.659 DEBUG:boto:Path: /yrjefh-52/obj29 2024-04-16T20:07:35.659 DEBUG:boto:Data: 2024-04-16T20:07:35.659 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.659 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.659 DEBUG:boto:Params: {} 2024-04-16T20:07:35.659 DEBUG:boto:Token: None 2024-04-16T20:07:35.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj29 2024-04-16T20:07:35.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1A2dNbn2za+alujDjWxxBM9NuFU= 2024-04-16T20:07:35.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1A2dNbn2za+alujDjWxxBM9NuFU=', 'Content-Length': '0'} 2024-04-16T20:07:35.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf28e872d0c96f39-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.662 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.662 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.662 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.662 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.662 DEBUG:boto:Method: GET 2024-04-16T20:07:35.662 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.662 DEBUG:boto:Data: 2024-04-16T20:07:35.662 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.662 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.662 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.662 DEBUG:boto:Params: {} 2024-04-16T20:07:35.663 DEBUG:boto:Token: None 2024-04-16T20:07:35.663 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009010610706650056-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.710 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.711 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.711 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.711 DEBUG:boto:path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.711 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.711 DEBUG:boto:Method: GET 2024-04-16T20:07:35.711 DEBUG:boto:Path: /yrjefh-52/?prefix=obj29 2024-04-16T20:07:35.711 DEBUG:boto:Data: 2024-04-16T20:07:35.711 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.711 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.712 DEBUG:boto:Params: {} 2024-04-16T20:07:35.712 DEBUG:boto:Token: None 2024-04-16T20:07:35.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e35de19cc3bef57-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.716 DEBUG:boto:b'yrjefh-52obj291000falseobj292024-04-16T20:03:33.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:35.716 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.716 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.716 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.716 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.716 DEBUG:boto:Method: GET 2024-04-16T20:07:35.716 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.717 DEBUG:boto:Data: 2024-04-16T20:07:35.717 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.717 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.717 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.717 DEBUG:boto:Params: {} 2024-04-16T20:07:35.717 DEBUG:boto:Token: None 2024-04-16T20:07:35.717 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.717 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI= 2024-04-16T20:07:35.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI=', 'Content-Length': '0'} 2024-04-16T20:07:35.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776ba8839c5bc33f-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.762 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.762 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.763 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.763 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.763 DEBUG:boto:Method: GET 2024-04-16T20:07:35.763 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.763 DEBUG:boto:Data: 2024-04-16T20:07:35.763 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.763 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.763 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.763 DEBUG:boto:Params: {} 2024-04-16T20:07:35.763 DEBUG:boto:Token: None 2024-04-16T20:07:35.763 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI= 2024-04-16T20:07:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI=', 'Content-Length': '0'} 2024-04-16T20:07:35.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb84a0109242cc0b-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.766 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.766 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.766 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.766 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.766 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.766 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.766 DEBUG:boto:Data: 2024-04-16T20:07:35.766 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.766 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.766 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.766 DEBUG:boto:Params: {} 2024-04-16T20:07:35.767 DEBUG:boto:Token: None 2024-04-16T20:07:35.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.767 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7zU/K8qSqtjzSZjYQPLnnEEt+yQ= 2024-04-16T20:07:35.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7zU/K8qSqtjzSZjYQPLnnEEt+yQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f484562c667826b-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.769 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.769 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.769 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.769 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.769 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.769 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.769 DEBUG:boto:Data: 2024-04-16T20:07:35.769 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.770 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.770 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.770 DEBUG:boto:Params: {} 2024-04-16T20:07:35.770 DEBUG:boto:Token: None 2024-04-16T20:07:35.770 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7zU/K8qSqtjzSZjYQPLnnEEt+yQ= 2024-04-16T20:07:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7zU/K8qSqtjzSZjYQPLnnEEt+yQ=', 'Content-Length': '0'} 2024-04-16T20:07:35.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb6f9208461ffbf-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:35.772 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.773 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.773 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.773 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.773 DEBUG:boto:Method: GET 2024-04-16T20:07:35.773 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.773 DEBUG:boto:Data: 2024-04-16T20:07:35.773 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.773 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.773 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.773 DEBUG:boto:Params: {} 2024-04-16T20:07:35.773 DEBUG:boto:Token: None 2024-04-16T20:07:35.774 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI= 2024-04-16T20:07:35.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI=', 'Content-Length': '0'} 2024-04-16T20:07:35.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7fa72aed74c92b-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.818 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.818 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.819 DEBUG:boto:path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.819 DEBUG:boto:auth_path=/yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.819 DEBUG:boto:Method: GET 2024-04-16T20:07:35.819 DEBUG:boto:Path: /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.819 DEBUG:boto:Data: 2024-04-16T20:07:35.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.819 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.819 DEBUG:boto:Params: {} 2024-04-16T20:07:35.819 DEBUG:boto:Token: None 2024-04-16T20:07:35.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31?versionId=null 2024-04-16T20:07:35.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI= 2024-04-16T20:07:35.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QSeN3wQEGODSqlBmAiavlQnu4hI=', 'Content-Length': '0'} 2024-04-16T20:07:35.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2698f3b9709439-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.822 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.822 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.822 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.822 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.822 DEBUG:boto:Method: GET 2024-04-16T20:07:35.822 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.822 DEBUG:boto:Data: 2024-04-16T20:07:35.822 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.822 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.822 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.822 DEBUG:boto:Params: {} 2024-04-16T20:07:35.823 DEBUG:boto:Token: None 2024-04-16T20:07:35.823 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91d1ffb09cc0e55-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.866 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.867 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.867 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.867 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.867 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.867 DEBUG:boto:Method: GET 2024-04-16T20:07:35.867 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.867 DEBUG:boto:Data: 2024-04-16T20:07:35.867 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.867 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.867 DEBUG:boto:Params: {} 2024-04-16T20:07:35.868 DEBUG:boto:Token: None 2024-04-16T20:07:35.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8efb4275e86b29-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.872 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.872 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.872 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.873 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.873 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:35.873 DEBUG:boto:Data: 2024-04-16T20:07:35.873 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.873 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.873 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.873 DEBUG:boto:Params: {} 2024-04-16T20:07:35.873 DEBUG:boto:Token: None 2024-04-16T20:07:35.873 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31 2024-04-16T20:07:35.873 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k9Y12cvubw3wGN2W86M+MY7kY+g= 2024-04-16T20:07:35.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k9Y12cvubw3wGN2W86M+MY7kY+g=', 'Content-Length': '0'} 2024-04-16T20:07:35.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34b7b3473e1465e-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.876 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.876 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.876 DEBUG:boto:Method: HEAD 2024-04-16T20:07:35.876 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:35.876 DEBUG:boto:Data: 2024-04-16T20:07:35.876 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.876 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.876 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.876 DEBUG:boto:Params: {} 2024-04-16T20:07:35.876 DEBUG:boto:Token: None 2024-04-16T20:07:35.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31 2024-04-16T20:07:35.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k9Y12cvubw3wGN2W86M+MY7kY+g= 2024-04-16T20:07:35.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k9Y12cvubw3wGN2W86M+MY7kY+g=', 'Content-Length': '0'} 2024-04-16T20:07:35.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eff5ee583623677-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:07:35.879 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.879 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.879 DEBUG:boto:Method: GET 2024-04-16T20:07:35.879 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:35.879 DEBUG:boto:Data: 2024-04-16T20:07:35.879 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.879 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.880 DEBUG:boto:Params: {} 2024-04-16T20:07:35.880 DEBUG:boto:Token: None 2024-04-16T20:07:35.880 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31 2024-04-16T20:07:35.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhKvvxGDgcN8hhqh2eV/zH/Svcc= 2024-04-16T20:07:35.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhKvvxGDgcN8hhqh2eV/zH/Svcc=', 'Content-Length': '0'} 2024-04-16T20:07:35.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9315cf94adb57c5-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.922 DEBUG:boto:path=/yrjefh-52/obj31 2024-04-16T20:07:35.922 DEBUG:boto:auth_path=/yrjefh-52/obj31 2024-04-16T20:07:35.923 DEBUG:boto:Method: GET 2024-04-16T20:07:35.923 DEBUG:boto:Path: /yrjefh-52/obj31 2024-04-16T20:07:35.923 DEBUG:boto:Data: 2024-04-16T20:07:35.923 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.923 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.923 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.923 DEBUG:boto:Params: {} 2024-04-16T20:07:35.923 DEBUG:boto:Token: None 2024-04-16T20:07:35.923 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj31 2024-04-16T20:07:35.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mhKvvxGDgcN8hhqh2eV/zH/Svcc= 2024-04-16T20:07:35.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mhKvvxGDgcN8hhqh2eV/zH/Svcc=', 'Content-Length': '0'} 2024-04-16T20:07:35.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4daf9ce5524ae7c-00661eda87-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.926 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.926 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.926 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.926 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.926 DEBUG:boto:Method: GET 2024-04-16T20:07:35.926 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.926 DEBUG:boto:Data: 2024-04-16T20:07:35.926 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.926 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.927 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.927 DEBUG:boto:Params: {} 2024-04-16T20:07:35.927 DEBUG:boto:Token: None 2024-04-16T20:07:35.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095098054716cba57-00661eda87-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.970 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.971 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:35.971 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:35.971 DEBUG:boto:path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.971 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.971 DEBUG:boto:Method: GET 2024-04-16T20:07:35.971 DEBUG:boto:Path: /yrjefh-52/?prefix=obj31 2024-04-16T20:07:35.971 DEBUG:boto:Data: 2024-04-16T20:07:35.971 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:35.971 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.971 DEBUG:boto:Params: {} 2024-04-16T20:07:35.972 DEBUG:boto:Token: None 2024-04-16T20:07:35.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/ 2024-04-16T20:07:35.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g= 2024-04-16T20:07:35.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tQEymM9iHY+6SBYNPGACuGLbi3g=', 'Content-Length': '0'} 2024-04-16T20:07:35.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b60bc1d6a3be3f2-00661eda87-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:35.975 DEBUG:boto:b'yrjefh-52obj311000falseobj312024-04-16T20:04:43.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:35.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:35.976 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:35.976 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:35.976 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:35.976 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:35.976 DEBUG:boto:Method: GET 2024-04-16T20:07:35.976 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:35.976 DEBUG:boto:Data: 2024-04-16T20:07:35.976 DEBUG:boto:Headers: {} 2024-04-16T20:07:35.976 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:35.976 DEBUG:boto:Port: 8000 2024-04-16T20:07:35.976 DEBUG:boto:Params: {} 2024-04-16T20:07:35.977 DEBUG:boto:Token: None 2024-04-16T20:07:35.977 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:35 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:35.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/CInMjyx8Mhq+mL3DFqdckKh3F4= 2024-04-16T20:07:35.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/CInMjyx8Mhq+mL3DFqdckKh3F4=', 'Content-Length': '0'} 2024-04-16T20:07:35.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8f6e7893000269-00661eda87-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.022 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.022 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.023 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.023 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.023 DEBUG:boto:Method: GET 2024-04-16T20:07:36.023 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.023 DEBUG:boto:Data: 2024-04-16T20:07:36.023 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.023 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.023 DEBUG:boto:Params: {} 2024-04-16T20:07:36.023 DEBUG:boto:Token: None 2024-04-16T20:07:36.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo= 2024-04-16T20:07:36.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo=', 'Content-Length': '0'} 2024-04-16T20:07:36.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14945a57dd53980-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.026 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.026 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.026 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.027 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.027 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.027 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.027 DEBUG:boto:Data: 2024-04-16T20:07:36.027 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.027 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.027 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.027 DEBUG:boto:Params: {} 2024-04-16T20:07:36.027 DEBUG:boto:Token: None 2024-04-16T20:07:36.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.027 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qondJpJfbb8pC485pS8zlX/b4nc= 2024-04-16T20:07:36.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qondJpJfbb8pC485pS8zlX/b4nc=', 'Content-Length': '0'} 2024-04-16T20:07:36.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018efc07264c893d3-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.030 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.030 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.030 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.030 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.030 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.030 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.030 DEBUG:boto:Data: 2024-04-16T20:07:36.030 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.030 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.030 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.031 DEBUG:boto:Params: {} 2024-04-16T20:07:36.031 DEBUG:boto:Token: None 2024-04-16T20:07:36.031 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qondJpJfbb8pC485pS8zlX/b4nc= 2024-04-16T20:07:36.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qondJpJfbb8pC485pS8zlX/b4nc=', 'Content-Length': '0'} 2024-04-16T20:07:36.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be437fec89fc11dc-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:36.034 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.034 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.034 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.034 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.034 DEBUG:boto:Method: GET 2024-04-16T20:07:36.034 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.034 DEBUG:boto:Data: 2024-04-16T20:07:36.034 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.034 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.034 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.034 DEBUG:boto:Params: {} 2024-04-16T20:07:36.035 DEBUG:boto:Token: None 2024-04-16T20:07:36.035 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo= 2024-04-16T20:07:36.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo=', 'Content-Length': '0'} 2024-04-16T20:07:36.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae778aeb9aaccd5-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.078 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.079 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.079 DEBUG:boto:path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.079 DEBUG:boto:auth_path=/yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.079 DEBUG:boto:Method: GET 2024-04-16T20:07:36.079 DEBUG:boto:Path: /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.079 DEBUG:boto:Data: 2024-04-16T20:07:36.079 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.079 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.079 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.079 DEBUG:boto:Params: {} 2024-04-16T20:07:36.080 DEBUG:boto:Token: None 2024-04-16T20:07:36.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33?versionId=null 2024-04-16T20:07:36.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo= 2024-04-16T20:07:36.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8YodXpkSlQwWqwKAjQxnd1OmmFo=', 'Content-Length': '0'} 2024-04-16T20:07:36.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1cda63941848ec1-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.082 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.083 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.083 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.083 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.083 DEBUG:boto:Method: GET 2024-04-16T20:07:36.083 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.083 DEBUG:boto:Data: 2024-04-16T20:07:36.083 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.083 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.083 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.083 DEBUG:boto:Params: {} 2024-04-16T20:07:36.083 DEBUG:boto:Token: None 2024-04-16T20:07:36.083 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50ce133acfa200e-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.126 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.127 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.127 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.127 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.127 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.127 DEBUG:boto:Method: GET 2024-04-16T20:07:36.127 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.127 DEBUG:boto:Data: 2024-04-16T20:07:36.127 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.127 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.128 DEBUG:boto:Params: {} 2024-04-16T20:07:36.128 DEBUG:boto:Token: None 2024-04-16T20:07:36.128 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000389391195de8bb33-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.131 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.132 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.132 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.132 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.132 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:36.132 DEBUG:boto:Data: 2024-04-16T20:07:36.132 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.132 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.132 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.132 DEBUG:boto:Params: {} 2024-04-16T20:07:36.133 DEBUG:boto:Token: None 2024-04-16T20:07:36.133 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33 2024-04-16T20:07:36.133 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DNL/sWgTEvd6V2aWn2t+JNWtdH0= 2024-04-16T20:07:36.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DNL/sWgTEvd6V2aWn2t+JNWtdH0=', 'Content-Length': '0'} 2024-04-16T20:07:36.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b61406fce21a70-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.135 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.135 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.136 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.136 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:36.136 DEBUG:boto:Data: 2024-04-16T20:07:36.136 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.136 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.136 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.136 DEBUG:boto:Params: {} 2024-04-16T20:07:36.136 DEBUG:boto:Token: None 2024-04-16T20:07:36.136 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33 2024-04-16T20:07:36.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DNL/sWgTEvd6V2aWn2t+JNWtdH0= 2024-04-16T20:07:36.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DNL/sWgTEvd6V2aWn2t+JNWtdH0=', 'Content-Length': '0'} 2024-04-16T20:07:36.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8d0c3f8e0bc071-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:07:36.139 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.139 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.139 DEBUG:boto:Method: GET 2024-04-16T20:07:36.139 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:36.139 DEBUG:boto:Data: 2024-04-16T20:07:36.139 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.139 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.139 DEBUG:boto:Params: {} 2024-04-16T20:07:36.140 DEBUG:boto:Token: None 2024-04-16T20:07:36.140 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33 2024-04-16T20:07:36.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:smD+GS1x4fsm9mf6cczzQsYt0Go= 2024-04-16T20:07:36.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:smD+GS1x4fsm9mf6cczzQsYt0Go=', 'Content-Length': '0'} 2024-04-16T20:07:36.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006747f306ac2a458f-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.182 DEBUG:boto:path=/yrjefh-52/obj33 2024-04-16T20:07:36.183 DEBUG:boto:auth_path=/yrjefh-52/obj33 2024-04-16T20:07:36.183 DEBUG:boto:Method: GET 2024-04-16T20:07:36.183 DEBUG:boto:Path: /yrjefh-52/obj33 2024-04-16T20:07:36.183 DEBUG:boto:Data: 2024-04-16T20:07:36.183 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.183 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.183 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.183 DEBUG:boto:Params: {} 2024-04-16T20:07:36.183 DEBUG:boto:Token: None 2024-04-16T20:07:36.183 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj33 2024-04-16T20:07:36.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:smD+GS1x4fsm9mf6cczzQsYt0Go= 2024-04-16T20:07:36.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:smD+GS1x4fsm9mf6cczzQsYt0Go=', 'Content-Length': '0'} 2024-04-16T20:07:36.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea7c8ddf84c7aec-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.186 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.186 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.186 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.186 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.186 DEBUG:boto:Method: GET 2024-04-16T20:07:36.186 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.187 DEBUG:boto:Data: 2024-04-16T20:07:36.187 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.187 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.187 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.187 DEBUG:boto:Params: {} 2024-04-16T20:07:36.187 DEBUG:boto:Token: None 2024-04-16T20:07:36.187 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea99843f5f107ba1-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.230 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.231 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.231 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.231 DEBUG:boto:path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.231 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.231 DEBUG:boto:Method: GET 2024-04-16T20:07:36.231 DEBUG:boto:Path: /yrjefh-52/?prefix=obj33 2024-04-16T20:07:36.231 DEBUG:boto:Data: 2024-04-16T20:07:36.231 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.232 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.232 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.232 DEBUG:boto:Params: {} 2024-04-16T20:07:36.232 DEBUG:boto:Token: None 2024-04-16T20:07:36.232 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006914385f38b29c7c-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.236 DEBUG:boto:b'yrjefh-52obj331000falseobj332024-04-16T19:58:04.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:36.236 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.236 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.236 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.237 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.237 DEBUG:boto:Method: GET 2024-04-16T20:07:36.237 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.237 DEBUG:boto:Data: 2024-04-16T20:07:36.237 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.237 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.237 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.237 DEBUG:boto:Params: {} 2024-04-16T20:07:36.237 DEBUG:boto:Token: None 2024-04-16T20:07:36.237 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU= 2024-04-16T20:07:36.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU=', 'Content-Length': '0'} 2024-04-16T20:07:36.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d35ac994902584-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.282 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.282 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.283 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.283 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.283 DEBUG:boto:Method: GET 2024-04-16T20:07:36.283 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.283 DEBUG:boto:Data: 2024-04-16T20:07:36.283 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.283 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.283 DEBUG:boto:Params: {} 2024-04-16T20:07:36.283 DEBUG:boto:Token: None 2024-04-16T20:07:36.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU= 2024-04-16T20:07:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU=', 'Content-Length': '0'} 2024-04-16T20:07:36.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997ca640271d7920-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.286 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.286 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.286 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.286 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.286 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.286 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.286 DEBUG:boto:Data: 2024-04-16T20:07:36.286 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.287 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.287 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.287 DEBUG:boto:Params: {} 2024-04-16T20:07:36.287 DEBUG:boto:Token: None 2024-04-16T20:07:36.287 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6QwMcT9wsfH72Eg2xtrRgXk6LCw= 2024-04-16T20:07:36.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6QwMcT9wsfH72Eg2xtrRgXk6LCw=', 'Content-Length': '0'} 2024-04-16T20:07:36.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68d502554dfc7e7-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.289 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.289 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.289 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.289 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.289 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.290 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.290 DEBUG:boto:Data: 2024-04-16T20:07:36.290 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.290 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.290 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.290 DEBUG:boto:Params: {} 2024-04-16T20:07:36.290 DEBUG:boto:Token: None 2024-04-16T20:07:36.290 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.290 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6QwMcT9wsfH72Eg2xtrRgXk6LCw= 2024-04-16T20:07:36.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6QwMcT9wsfH72Eg2xtrRgXk6LCw=', 'Content-Length': '0'} 2024-04-16T20:07:36.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d890f6f4c84a3073-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:36.293 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.293 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.293 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.293 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.293 DEBUG:boto:Method: GET 2024-04-16T20:07:36.293 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.293 DEBUG:boto:Data: 2024-04-16T20:07:36.293 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.293 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.293 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.293 DEBUG:boto:Params: {} 2024-04-16T20:07:36.293 DEBUG:boto:Token: None 2024-04-16T20:07:36.294 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.294 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU= 2024-04-16T20:07:36.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU=', 'Content-Length': '0'} 2024-04-16T20:07:36.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8f9ebc9fbea685-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.338 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.338 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.339 DEBUG:boto:path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.339 DEBUG:boto:auth_path=/yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.339 DEBUG:boto:Method: GET 2024-04-16T20:07:36.339 DEBUG:boto:Path: /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.339 DEBUG:boto:Data: 2024-04-16T20:07:36.339 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.339 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.339 DEBUG:boto:Params: {} 2024-04-16T20:07:36.339 DEBUG:boto:Token: None 2024-04-16T20:07:36.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34?versionId=null 2024-04-16T20:07:36.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU= 2024-04-16T20:07:36.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2dReCypJY2sAEbrlNkkyJslZUVU=', 'Content-Length': '0'} 2024-04-16T20:07:36.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf5bdb57f1f1318-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.342 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.342 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.342 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.343 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.343 DEBUG:boto:Method: GET 2024-04-16T20:07:36.343 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.343 DEBUG:boto:Data: 2024-04-16T20:07:36.343 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.343 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.343 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.343 DEBUG:boto:Params: {} 2024-04-16T20:07:36.343 DEBUG:boto:Token: None 2024-04-16T20:07:36.343 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2102e25204cf91e-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.386 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.387 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.387 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.387 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.387 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.387 DEBUG:boto:Method: GET 2024-04-16T20:07:36.387 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.387 DEBUG:boto:Data: 2024-04-16T20:07:36.387 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.387 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.388 DEBUG:boto:Params: {} 2024-04-16T20:07:36.388 DEBUG:boto:Token: None 2024-04-16T20:07:36.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec700bcfaeac51a-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.391 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.392 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.392 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.392 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.392 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:36.392 DEBUG:boto:Data: 2024-04-16T20:07:36.392 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.392 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.392 DEBUG:boto:Params: {} 2024-04-16T20:07:36.393 DEBUG:boto:Token: None 2024-04-16T20:07:36.393 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34 2024-04-16T20:07:36.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mmzjdHK5h0WV4nj5TNPrj2ztN24= 2024-04-16T20:07:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mmzjdHK5h0WV4nj5TNPrj2ztN24=', 'Content-Length': '0'} 2024-04-16T20:07:36.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65a7d32043c9548-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.395 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.395 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.395 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.396 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:36.396 DEBUG:boto:Data: 2024-04-16T20:07:36.396 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.396 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.396 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.396 DEBUG:boto:Params: {} 2024-04-16T20:07:36.396 DEBUG:boto:Token: None 2024-04-16T20:07:36.396 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34 2024-04-16T20:07:36.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mmzjdHK5h0WV4nj5TNPrj2ztN24= 2024-04-16T20:07:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mmzjdHK5h0WV4nj5TNPrj2ztN24=', 'Content-Length': '0'} 2024-04-16T20:07:36.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ce256808bc5956-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:07:36.399 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.399 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.399 DEBUG:boto:Method: GET 2024-04-16T20:07:36.399 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:36.399 DEBUG:boto:Data: 2024-04-16T20:07:36.399 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.399 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.399 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.399 DEBUG:boto:Params: {} 2024-04-16T20:07:36.400 DEBUG:boto:Token: None 2024-04-16T20:07:36.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34 2024-04-16T20:07:36.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlQsRUtmiNPp+rrNQe6MjoIyUNQ= 2024-04-16T20:07:36.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlQsRUtmiNPp+rrNQe6MjoIyUNQ=', 'Content-Length': '0'} 2024-04-16T20:07:36.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000467dd368c38ee49b-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.442 DEBUG:boto:path=/yrjefh-52/obj34 2024-04-16T20:07:36.442 DEBUG:boto:auth_path=/yrjefh-52/obj34 2024-04-16T20:07:36.443 DEBUG:boto:Method: GET 2024-04-16T20:07:36.443 DEBUG:boto:Path: /yrjefh-52/obj34 2024-04-16T20:07:36.443 DEBUG:boto:Data: 2024-04-16T20:07:36.443 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.443 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.443 DEBUG:boto:Params: {} 2024-04-16T20:07:36.443 DEBUG:boto:Token: None 2024-04-16T20:07:36.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj34 2024-04-16T20:07:36.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IlQsRUtmiNPp+rrNQe6MjoIyUNQ= 2024-04-16T20:07:36.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IlQsRUtmiNPp+rrNQe6MjoIyUNQ=', 'Content-Length': '0'} 2024-04-16T20:07:36.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa6c53d2b26fd75-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.446 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.446 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.446 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.446 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.446 DEBUG:boto:Method: GET 2024-04-16T20:07:36.446 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.447 DEBUG:boto:Data: 2024-04-16T20:07:36.447 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.447 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.447 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.447 DEBUG:boto:Params: {} 2024-04-16T20:07:36.447 DEBUG:boto:Token: None 2024-04-16T20:07:36.447 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c89530bc543da4-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.490 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.491 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.491 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.491 DEBUG:boto:path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.491 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.491 DEBUG:boto:Method: GET 2024-04-16T20:07:36.491 DEBUG:boto:Path: /yrjefh-52/?prefix=obj34 2024-04-16T20:07:36.491 DEBUG:boto:Data: 2024-04-16T20:07:36.491 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.491 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.491 DEBUG:boto:Params: {} 2024-04-16T20:07:36.492 DEBUG:boto:Token: None 2024-04-16T20:07:36.492 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165c500733a35141-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.496 DEBUG:boto:b'yrjefh-52obj341000falseobj342024-04-16T19:58:15.855Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:36.496 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.496 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.496 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.497 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.497 DEBUG:boto:Method: GET 2024-04-16T20:07:36.497 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.497 DEBUG:boto:Data: 2024-04-16T20:07:36.497 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.497 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.497 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.497 DEBUG:boto:Params: {} 2024-04-16T20:07:36.497 DEBUG:boto:Token: None 2024-04-16T20:07:36.497 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0= 2024-04-16T20:07:36.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0=', 'Content-Length': '0'} 2024-04-16T20:07:36.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289188f4b3276115-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.542 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.542 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.543 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.543 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.543 DEBUG:boto:Method: GET 2024-04-16T20:07:36.543 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.543 DEBUG:boto:Data: 2024-04-16T20:07:36.543 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.543 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.543 DEBUG:boto:Params: {} 2024-04-16T20:07:36.543 DEBUG:boto:Token: None 2024-04-16T20:07:36.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0= 2024-04-16T20:07:36.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0=', 'Content-Length': '0'} 2024-04-16T20:07:36.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000544666086294072-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.546 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.546 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.547 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.547 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.547 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.547 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.547 DEBUG:boto:Data: 2024-04-16T20:07:36.547 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.547 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.547 DEBUG:boto:Params: {} 2024-04-16T20:07:36.547 DEBUG:boto:Token: None 2024-04-16T20:07:36.547 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B9rQOTmVPefTflwkPjBqk5P2KD0= 2024-04-16T20:07:36.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B9rQOTmVPefTflwkPjBqk5P2KD0=', 'Content-Length': '0'} 2024-04-16T20:07:36.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c3d19fcae1894d-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.550 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.550 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.551 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.551 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.551 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.551 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.551 DEBUG:boto:Data: 2024-04-16T20:07:36.551 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.551 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.551 DEBUG:boto:Params: {} 2024-04-16T20:07:36.551 DEBUG:boto:Token: None 2024-04-16T20:07:36.552 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B9rQOTmVPefTflwkPjBqk5P2KD0= 2024-04-16T20:07:36.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B9rQOTmVPefTflwkPjBqk5P2KD0=', 'Content-Length': '0'} 2024-04-16T20:07:36.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c99f9a8ef57f50-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:36.554 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.554 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.554 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.554 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.555 DEBUG:boto:Method: GET 2024-04-16T20:07:36.555 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.555 DEBUG:boto:Data: 2024-04-16T20:07:36.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.555 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.555 DEBUG:boto:Params: {} 2024-04-16T20:07:36.555 DEBUG:boto:Token: None 2024-04-16T20:07:36.555 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0= 2024-04-16T20:07:36.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0=', 'Content-Length': '0'} 2024-04-16T20:07:36.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cb061506d03de3-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.598 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.599 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.599 DEBUG:boto:path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.599 DEBUG:boto:auth_path=/yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.599 DEBUG:boto:Method: GET 2024-04-16T20:07:36.599 DEBUG:boto:Path: /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.599 DEBUG:boto:Data: 2024-04-16T20:07:36.599 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.599 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.599 DEBUG:boto:Params: {} 2024-04-16T20:07:36.599 DEBUG:boto:Token: None 2024-04-16T20:07:36.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37?versionId=null 2024-04-16T20:07:36.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0= 2024-04-16T20:07:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M+gDOOxu1R8ltUJgbeerU027Nj0=', 'Content-Length': '0'} 2024-04-16T20:07:36.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8c961b4b7394c2d-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.602 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.602 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.603 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.603 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.603 DEBUG:boto:Method: GET 2024-04-16T20:07:36.603 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.603 DEBUG:boto:Data: 2024-04-16T20:07:36.603 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.603 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.603 DEBUG:boto:Params: {} 2024-04-16T20:07:36.603 DEBUG:boto:Token: None 2024-04-16T20:07:36.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c7493e0c160fc01-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.646 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.647 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.647 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.647 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.647 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.647 DEBUG:boto:Method: GET 2024-04-16T20:07:36.647 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.647 DEBUG:boto:Data: 2024-04-16T20:07:36.647 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.647 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.648 DEBUG:boto:Params: {} 2024-04-16T20:07:36.648 DEBUG:boto:Token: None 2024-04-16T20:07:36.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4894985107696e-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.651 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.652 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.652 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.652 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.652 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:36.652 DEBUG:boto:Data: 2024-04-16T20:07:36.652 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.652 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.652 DEBUG:boto:Params: {} 2024-04-16T20:07:36.653 DEBUG:boto:Token: None 2024-04-16T20:07:36.653 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37 2024-04-16T20:07:36.653 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58lRvaJN5/xm0//UTL3MOn9akA= 2024-04-16T20:07:36.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58lRvaJN5/xm0//UTL3MOn9akA=', 'Content-Length': '0'} 2024-04-16T20:07:36.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50f7cd5635e6f30-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.655 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.656 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.656 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.656 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:36.656 DEBUG:boto:Data: 2024-04-16T20:07:36.656 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.656 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.656 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.656 DEBUG:boto:Params: {} 2024-04-16T20:07:36.656 DEBUG:boto:Token: None 2024-04-16T20:07:36.657 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37 2024-04-16T20:07:36.657 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P58lRvaJN5/xm0//UTL3MOn9akA= 2024-04-16T20:07:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P58lRvaJN5/xm0//UTL3MOn9akA=', 'Content-Length': '0'} 2024-04-16T20:07:36.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c4e4a903f04d91-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:07:36.659 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.659 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.659 DEBUG:boto:Method: GET 2024-04-16T20:07:36.660 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:36.660 DEBUG:boto:Data: 2024-04-16T20:07:36.660 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.660 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.660 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.660 DEBUG:boto:Params: {} 2024-04-16T20:07:36.660 DEBUG:boto:Token: None 2024-04-16T20:07:36.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37 2024-04-16T20:07:36.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2nECW9ZUULWVRXTZIkfW3f9ZoLk= 2024-04-16T20:07:36.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2nECW9ZUULWVRXTZIkfW3f9ZoLk=', 'Content-Length': '0'} 2024-04-16T20:07:36.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a22069bf738198c-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.706 DEBUG:boto:path=/yrjefh-52/obj37 2024-04-16T20:07:36.707 DEBUG:boto:auth_path=/yrjefh-52/obj37 2024-04-16T20:07:36.707 DEBUG:boto:Method: GET 2024-04-16T20:07:36.707 DEBUG:boto:Path: /yrjefh-52/obj37 2024-04-16T20:07:36.707 DEBUG:boto:Data: 2024-04-16T20:07:36.707 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.707 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.707 DEBUG:boto:Params: {} 2024-04-16T20:07:36.707 DEBUG:boto:Token: None 2024-04-16T20:07:36.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj37 2024-04-16T20:07:36.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2nECW9ZUULWVRXTZIkfW3f9ZoLk= 2024-04-16T20:07:36.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2nECW9ZUULWVRXTZIkfW3f9ZoLk=', 'Content-Length': '0'} 2024-04-16T20:07:36.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a474a99f3fe019-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.710 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.710 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.710 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.710 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.710 DEBUG:boto:Method: GET 2024-04-16T20:07:36.711 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.711 DEBUG:boto:Data: 2024-04-16T20:07:36.711 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.711 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.711 DEBUG:boto:Params: {} 2024-04-16T20:07:36.711 DEBUG:boto:Token: None 2024-04-16T20:07:36.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea6d3811a7476f7-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.758 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.759 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.759 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.759 DEBUG:boto:path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.759 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.759 DEBUG:boto:Method: GET 2024-04-16T20:07:36.759 DEBUG:boto:Path: /yrjefh-52/?prefix=obj37 2024-04-16T20:07:36.759 DEBUG:boto:Data: 2024-04-16T20:07:36.760 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.760 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.760 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.760 DEBUG:boto:Params: {} 2024-04-16T20:07:36.760 DEBUG:boto:Token: None 2024-04-16T20:07:36.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046dd24752595b81a-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.764 DEBUG:boto:b'yrjefh-52obj371000falseobj372024-04-16T20:06:00.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:36.765 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.765 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.765 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.765 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.765 DEBUG:boto:Method: GET 2024-04-16T20:07:36.765 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.765 DEBUG:boto:Data: 2024-04-16T20:07:36.765 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.765 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.766 DEBUG:boto:Params: {} 2024-04-16T20:07:36.766 DEBUG:boto:Token: None 2024-04-16T20:07:36.766 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU= 2024-04-16T20:07:36.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU=', 'Content-Length': '0'} 2024-04-16T20:07:36.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebebc6ead8f5850f-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.811 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.811 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.811 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.811 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.811 DEBUG:boto:Method: GET 2024-04-16T20:07:36.811 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.811 DEBUG:boto:Data: 2024-04-16T20:07:36.811 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.812 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.812 DEBUG:boto:Params: {} 2024-04-16T20:07:36.812 DEBUG:boto:Token: None 2024-04-16T20:07:36.812 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU= 2024-04-16T20:07:36.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU=', 'Content-Length': '0'} 2024-04-16T20:07:36.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002428561397b58485-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.815 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.815 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.815 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.815 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.815 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.815 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.815 DEBUG:boto:Data: 2024-04-16T20:07:36.815 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.816 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.816 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.816 DEBUG:boto:Params: {} 2024-04-16T20:07:36.816 DEBUG:boto:Token: None 2024-04-16T20:07:36.816 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VzZBcsybgIvqShoNAHFyzyt5rVY= 2024-04-16T20:07:36.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VzZBcsybgIvqShoNAHFyzyt5rVY=', 'Content-Length': '0'} 2024-04-16T20:07:36.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df5b626653dc19a-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.818 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.819 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.819 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.819 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.819 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.819 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.819 DEBUG:boto:Data: 2024-04-16T20:07:36.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.819 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.819 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.819 DEBUG:boto:Params: {} 2024-04-16T20:07:36.820 DEBUG:boto:Token: None 2024-04-16T20:07:36.820 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VzZBcsybgIvqShoNAHFyzyt5rVY= 2024-04-16T20:07:36.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VzZBcsybgIvqShoNAHFyzyt5rVY=', 'Content-Length': '0'} 2024-04-16T20:07:36.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e48be953ab61a6-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:36.822 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.822 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.822 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.823 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.823 DEBUG:boto:Method: GET 2024-04-16T20:07:36.823 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.823 DEBUG:boto:Data: 2024-04-16T20:07:36.823 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.823 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.823 DEBUG:boto:Params: {} 2024-04-16T20:07:36.823 DEBUG:boto:Token: None 2024-04-16T20:07:36.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU= 2024-04-16T20:07:36.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU=', 'Content-Length': '0'} 2024-04-16T20:07:36.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009910db005ef2f831-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.866 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.866 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.867 DEBUG:boto:path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.867 DEBUG:boto:auth_path=/yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.867 DEBUG:boto:Method: GET 2024-04-16T20:07:36.867 DEBUG:boto:Path: /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.867 DEBUG:boto:Data: 2024-04-16T20:07:36.867 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.867 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.867 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.867 DEBUG:boto:Params: {} 2024-04-16T20:07:36.867 DEBUG:boto:Token: None 2024-04-16T20:07:36.868 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38?versionId=null 2024-04-16T20:07:36.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU= 2024-04-16T20:07:36.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:34qZ+DBNdgkvLRgJgzKxPHFXyyU=', 'Content-Length': '0'} 2024-04-16T20:07:36.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2d05c1d8f84c6e-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.870 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.870 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.870 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.871 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.871 DEBUG:boto:Method: GET 2024-04-16T20:07:36.871 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.871 DEBUG:boto:Data: 2024-04-16T20:07:36.871 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.871 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.871 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.871 DEBUG:boto:Params: {} 2024-04-16T20:07:36.871 DEBUG:boto:Token: None 2024-04-16T20:07:36.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841a81db401eed59-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.914 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.915 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.915 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.915 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.915 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.915 DEBUG:boto:Method: GET 2024-04-16T20:07:36.915 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.915 DEBUG:boto:Data: 2024-04-16T20:07:36.915 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.916 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.916 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.916 DEBUG:boto:Params: {} 2024-04-16T20:07:36.916 DEBUG:boto:Token: None 2024-04-16T20:07:36.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0ec2c2e7f17a3e-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.920 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.920 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.920 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.921 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.921 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:36.921 DEBUG:boto:Data: 2024-04-16T20:07:36.921 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.921 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.921 DEBUG:boto:Params: {} 2024-04-16T20:07:36.921 DEBUG:boto:Token: None 2024-04-16T20:07:36.921 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38 2024-04-16T20:07:36.922 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TRKtwr6I2WqE7aAVDtUfYtgRGcU= 2024-04-16T20:07:36.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TRKtwr6I2WqE7aAVDtUfYtgRGcU=', 'Content-Length': '0'} 2024-04-16T20:07:36.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678fa732f2172036-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.924 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.924 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.924 DEBUG:boto:Method: HEAD 2024-04-16T20:07:36.924 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:36.924 DEBUG:boto:Data: 2024-04-16T20:07:36.924 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.924 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.925 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.925 DEBUG:boto:Params: {} 2024-04-16T20:07:36.925 DEBUG:boto:Token: None 2024-04-16T20:07:36.925 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38 2024-04-16T20:07:36.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TRKtwr6I2WqE7aAVDtUfYtgRGcU= 2024-04-16T20:07:36.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TRKtwr6I2WqE7aAVDtUfYtgRGcU=', 'Content-Length': '0'} 2024-04-16T20:07:36.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d387d8d9a95bdbc-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:07:36.927 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.928 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.928 DEBUG:boto:Method: GET 2024-04-16T20:07:36.928 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:36.928 DEBUG:boto:Data: 2024-04-16T20:07:36.928 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.928 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.928 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.928 DEBUG:boto:Params: {} 2024-04-16T20:07:36.928 DEBUG:boto:Token: None 2024-04-16T20:07:36.928 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38 2024-04-16T20:07:36.929 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Z/MxLwuuRvdt9F9UqkgOwIf4PQ= 2024-04-16T20:07:36.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Z/MxLwuuRvdt9F9UqkgOwIf4PQ=', 'Content-Length': '0'} 2024-04-16T20:07:36.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ec868a1738cc1b-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.974 DEBUG:boto:path=/yrjefh-52/obj38 2024-04-16T20:07:36.975 DEBUG:boto:auth_path=/yrjefh-52/obj38 2024-04-16T20:07:36.975 DEBUG:boto:Method: GET 2024-04-16T20:07:36.975 DEBUG:boto:Path: /yrjefh-52/obj38 2024-04-16T20:07:36.975 DEBUG:boto:Data: 2024-04-16T20:07:36.975 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.975 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.975 DEBUG:boto:Params: {} 2024-04-16T20:07:36.975 DEBUG:boto:Token: None 2024-04-16T20:07:36.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj38 2024-04-16T20:07:36.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4Z/MxLwuuRvdt9F9UqkgOwIf4PQ= 2024-04-16T20:07:36.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4Z/MxLwuuRvdt9F9UqkgOwIf4PQ=', 'Content-Length': '0'} 2024-04-16T20:07:36.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509ec4030c338b28-00661eda88-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.978 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.978 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.978 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.978 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.978 DEBUG:boto:Method: GET 2024-04-16T20:07:36.978 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.979 DEBUG:boto:Data: 2024-04-16T20:07:36.979 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.979 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.979 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.979 DEBUG:boto:Params: {} 2024-04-16T20:07:36.979 DEBUG:boto:Token: None 2024-04-16T20:07:36.979 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.979 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9574ea8a07fb118-00661eda88-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.982 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.983 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:36.983 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:36.983 DEBUG:boto:path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.983 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.983 DEBUG:boto:Method: GET 2024-04-16T20:07:36.983 DEBUG:boto:Path: /yrjefh-52/?prefix=obj38 2024-04-16T20:07:36.983 DEBUG:boto:Data: 2024-04-16T20:07:36.983 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:36.983 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.984 DEBUG:boto:Params: {} 2024-04-16T20:07:36.984 DEBUG:boto:Token: None 2024-04-16T20:07:36.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/ 2024-04-16T20:07:36.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY= 2024-04-16T20:07:36.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5ZEE3p4+iOc5E+HdpFEiG/f6/HY=', 'Content-Length': '0'} 2024-04-16T20:07:36.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d3b88eb0a74f0f-00661eda88-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:36.987 DEBUG:boto:b'yrjefh-52obj381000falseobj382024-04-16T20:00:28.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:36.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:36.988 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:36.988 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:36.988 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:36.988 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:36.988 DEBUG:boto:Method: GET 2024-04-16T20:07:36.988 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:36.988 DEBUG:boto:Data: 2024-04-16T20:07:36.988 DEBUG:boto:Headers: {} 2024-04-16T20:07:36.988 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:36.988 DEBUG:boto:Port: 8000 2024-04-16T20:07:36.988 DEBUG:boto:Params: {} 2024-04-16T20:07:36.989 DEBUG:boto:Token: None 2024-04-16T20:07:36.989 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:36 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:36.989 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d0wuY5+Oj9lJb7fgj5P2U5zL8IA= 2024-04-16T20:07:36.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d0wuY5+Oj9lJb7fgj5P2U5zL8IA=', 'Content-Length': '0'} 2024-04-16T20:07:36.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed593f2c47238c4f-00661eda88-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.034 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.034 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.035 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.035 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.035 DEBUG:boto:Method: GET 2024-04-16T20:07:37.035 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.035 DEBUG:boto:Data: 2024-04-16T20:07:37.035 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.035 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.035 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.035 DEBUG:boto:Params: {} 2024-04-16T20:07:37.036 DEBUG:boto:Token: None 2024-04-16T20:07:37.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4= 2024-04-16T20:07:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4=', 'Content-Length': '0'} 2024-04-16T20:07:37.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac9de96ab420a30-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.038 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.038 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.039 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.039 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.039 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.039 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.039 DEBUG:boto:Data: 2024-04-16T20:07:37.039 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.039 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.039 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.039 DEBUG:boto:Params: {} 2024-04-16T20:07:37.039 DEBUG:boto:Token: None 2024-04-16T20:07:37.040 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JmHGEn+mNg/zWV8u4NrZne3z+Yk= 2024-04-16T20:07:37.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JmHGEn+mNg/zWV8u4NrZne3z+Yk=', 'Content-Length': '0'} 2024-04-16T20:07:37.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c70f9d72897ba2-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.042 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.042 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.042 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.042 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.043 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.043 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.043 DEBUG:boto:Data: 2024-04-16T20:07:37.043 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.043 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.043 DEBUG:boto:Params: {} 2024-04-16T20:07:37.043 DEBUG:boto:Token: None 2024-04-16T20:07:37.043 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JmHGEn+mNg/zWV8u4NrZne3z+Yk= 2024-04-16T20:07:37.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JmHGEn+mNg/zWV8u4NrZne3z+Yk=', 'Content-Length': '0'} 2024-04-16T20:07:37.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd0226633c5ce01-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:37.046 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.046 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.046 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.047 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.047 DEBUG:boto:Method: GET 2024-04-16T20:07:37.047 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.047 DEBUG:boto:Data: 2024-04-16T20:07:37.047 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.047 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.047 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.047 DEBUG:boto:Params: {} 2024-04-16T20:07:37.047 DEBUG:boto:Token: None 2024-04-16T20:07:37.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4= 2024-04-16T20:07:37.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4=', 'Content-Length': '0'} 2024-04-16T20:07:37.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121f7e485ab9eeb2-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.090 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.090 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.090 DEBUG:boto:path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.091 DEBUG:boto:auth_path=/yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.091 DEBUG:boto:Method: GET 2024-04-16T20:07:37.091 DEBUG:boto:Path: /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.091 DEBUG:boto:Data: 2024-04-16T20:07:37.091 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.091 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.091 DEBUG:boto:Params: {} 2024-04-16T20:07:37.091 DEBUG:boto:Token: None 2024-04-16T20:07:37.091 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44?versionId=null 2024-04-16T20:07:37.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4= 2024-04-16T20:07:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bc7//xI8kkOcxxRR96nY0xExNA4=', 'Content-Length': '0'} 2024-04-16T20:07:37.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036cfe77c3b8fa34-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.094 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.095 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.095 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.095 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.095 DEBUG:boto:Method: GET 2024-04-16T20:07:37.095 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.095 DEBUG:boto:Data: 2024-04-16T20:07:37.095 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.095 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.095 DEBUG:boto:Params: {} 2024-04-16T20:07:37.095 DEBUG:boto:Token: None 2024-04-16T20:07:37.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20d41372036b24f-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.099 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.099 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.099 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.099 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.099 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.099 DEBUG:boto:Method: GET 2024-04-16T20:07:37.100 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.100 DEBUG:boto:Data: 2024-04-16T20:07:37.100 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.100 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.100 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.100 DEBUG:boto:Params: {} 2024-04-16T20:07:37.100 DEBUG:boto:Token: None 2024-04-16T20:07:37.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088031520e4b8212-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.104 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.104 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.105 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.105 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.105 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:37.105 DEBUG:boto:Data: 2024-04-16T20:07:37.105 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.105 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.105 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.105 DEBUG:boto:Params: {} 2024-04-16T20:07:37.105 DEBUG:boto:Token: None 2024-04-16T20:07:37.105 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44 2024-04-16T20:07:37.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2EXXatJDNPWozg9TZw6bWw9OncU= 2024-04-16T20:07:37.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2EXXatJDNPWozg9TZw6bWw9OncU=', 'Content-Length': '0'} 2024-04-16T20:07:37.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007250acdcd3c8cc62-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.108 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.108 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.108 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.108 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:37.109 DEBUG:boto:Data: 2024-04-16T20:07:37.109 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.109 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.109 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.109 DEBUG:boto:Params: {} 2024-04-16T20:07:37.109 DEBUG:boto:Token: None 2024-04-16T20:07:37.109 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44 2024-04-16T20:07:37.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2EXXatJDNPWozg9TZw6bWw9OncU= 2024-04-16T20:07:37.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2EXXatJDNPWozg9TZw6bWw9OncU=', 'Content-Length': '0'} 2024-04-16T20:07:37.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791e5321874ff0a9-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:07:37.112 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.112 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.112 DEBUG:boto:Method: GET 2024-04-16T20:07:37.112 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:37.112 DEBUG:boto:Data: 2024-04-16T20:07:37.112 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.113 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.113 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.113 DEBUG:boto:Params: {} 2024-04-16T20:07:37.113 DEBUG:boto:Token: None 2024-04-16T20:07:37.113 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44 2024-04-16T20:07:37.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qcVPOflUgL2wlWOydXk1MqNRW6Y= 2024-04-16T20:07:37.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qcVPOflUgL2wlWOydXk1MqNRW6Y=', 'Content-Length': '0'} 2024-04-16T20:07:37.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2c59bc22c08435-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.162 DEBUG:boto:path=/yrjefh-52/obj44 2024-04-16T20:07:37.163 DEBUG:boto:auth_path=/yrjefh-52/obj44 2024-04-16T20:07:37.163 DEBUG:boto:Method: GET 2024-04-16T20:07:37.163 DEBUG:boto:Path: /yrjefh-52/obj44 2024-04-16T20:07:37.163 DEBUG:boto:Data: 2024-04-16T20:07:37.163 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.163 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.163 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.163 DEBUG:boto:Params: {} 2024-04-16T20:07:37.163 DEBUG:boto:Token: None 2024-04-16T20:07:37.163 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj44 2024-04-16T20:07:37.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qcVPOflUgL2wlWOydXk1MqNRW6Y= 2024-04-16T20:07:37.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qcVPOflUgL2wlWOydXk1MqNRW6Y=', 'Content-Length': '0'} 2024-04-16T20:07:37.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001787695f18106c3e-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.166 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.166 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.166 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.166 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.166 DEBUG:boto:Method: GET 2024-04-16T20:07:37.167 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.167 DEBUG:boto:Data: 2024-04-16T20:07:37.167 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.167 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.167 DEBUG:boto:Params: {} 2024-04-16T20:07:37.167 DEBUG:boto:Token: None 2024-04-16T20:07:37.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ac124ca51e8171-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.170 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.170 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.170 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.171 DEBUG:boto:path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.171 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.171 DEBUG:boto:Method: GET 2024-04-16T20:07:37.171 DEBUG:boto:Path: /yrjefh-52/?prefix=obj44 2024-04-16T20:07:37.171 DEBUG:boto:Data: 2024-04-16T20:07:37.171 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.171 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.171 DEBUG:boto:Params: {} 2024-04-16T20:07:37.171 DEBUG:boto:Token: None 2024-04-16T20:07:37.171 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed38af869462642-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.175 DEBUG:boto:b'yrjefh-52obj441000falseobj442024-04-16T19:58:04.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:37.176 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.176 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.176 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.176 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.176 DEBUG:boto:Method: GET 2024-04-16T20:07:37.176 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.176 DEBUG:boto:Data: 2024-04-16T20:07:37.176 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.177 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.177 DEBUG:boto:Params: {} 2024-04-16T20:07:37.177 DEBUG:boto:Token: None 2024-04-16T20:07:37.177 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.177 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw= 2024-04-16T20:07:37.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw=', 'Content-Length': '0'} 2024-04-16T20:07:37.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb88df6dbfe9b3f1-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.222 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.223 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.223 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.223 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.223 DEBUG:boto:Method: GET 2024-04-16T20:07:37.223 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.223 DEBUG:boto:Data: 2024-04-16T20:07:37.223 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.223 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.223 DEBUG:boto:Params: {} 2024-04-16T20:07:37.223 DEBUG:boto:Token: None 2024-04-16T20:07:37.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw= 2024-04-16T20:07:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw=', 'Content-Length': '0'} 2024-04-16T20:07:37.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dacaaf58f65bd9c-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.226 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.227 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.227 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.227 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.227 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.227 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.227 DEBUG:boto:Data: 2024-04-16T20:07:37.227 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.227 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.227 DEBUG:boto:Params: {} 2024-04-16T20:07:37.228 DEBUG:boto:Token: None 2024-04-16T20:07:37.228 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cG6OJbCI7OMG5lXkORQTRh9qjlI= 2024-04-16T20:07:37.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cG6OJbCI7OMG5lXkORQTRh9qjlI=', 'Content-Length': '0'} 2024-04-16T20:07:37.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7606a5bd88b628-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.230 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.230 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.230 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.230 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.231 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.231 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.231 DEBUG:boto:Data: 2024-04-16T20:07:37.231 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.231 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.231 DEBUG:boto:Params: {} 2024-04-16T20:07:37.231 DEBUG:boto:Token: None 2024-04-16T20:07:37.231 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cG6OJbCI7OMG5lXkORQTRh9qjlI= 2024-04-16T20:07:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cG6OJbCI7OMG5lXkORQTRh9qjlI=', 'Content-Length': '0'} 2024-04-16T20:07:37.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d8409ea5968efa-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:37.235 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.235 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.235 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.235 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.235 DEBUG:boto:Method: GET 2024-04-16T20:07:37.235 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.235 DEBUG:boto:Data: 2024-04-16T20:07:37.235 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.235 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.236 DEBUG:boto:Params: {} 2024-04-16T20:07:37.236 DEBUG:boto:Token: None 2024-04-16T20:07:37.236 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.236 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw= 2024-04-16T20:07:37.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw=', 'Content-Length': '0'} 2024-04-16T20:07:37.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abe760b91764e7e-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.278 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.279 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.279 DEBUG:boto:path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.279 DEBUG:boto:auth_path=/yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.279 DEBUG:boto:Method: GET 2024-04-16T20:07:37.279 DEBUG:boto:Path: /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.279 DEBUG:boto:Data: 2024-04-16T20:07:37.279 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.279 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.279 DEBUG:boto:Params: {} 2024-04-16T20:07:37.279 DEBUG:boto:Token: None 2024-04-16T20:07:37.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46?versionId=null 2024-04-16T20:07:37.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw= 2024-04-16T20:07:37.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UgIVVuFAGwgWP4oZhPvKawSvQGw=', 'Content-Length': '0'} 2024-04-16T20:07:37.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008970d0c7f5750a55-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.282 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.282 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.282 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.282 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.283 DEBUG:boto:Method: GET 2024-04-16T20:07:37.283 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.283 DEBUG:boto:Data: 2024-04-16T20:07:37.283 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.283 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.283 DEBUG:boto:Params: {} 2024-04-16T20:07:37.283 DEBUG:boto:Token: None 2024-04-16T20:07:37.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001873cafbd257fb93-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.330 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.331 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.331 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.331 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.331 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.331 DEBUG:boto:Method: GET 2024-04-16T20:07:37.331 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.331 DEBUG:boto:Data: 2024-04-16T20:07:37.331 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.332 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.332 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.332 DEBUG:boto:Params: {} 2024-04-16T20:07:37.332 DEBUG:boto:Token: None 2024-04-16T20:07:37.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000307785f5ac1a0e79-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.336 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.336 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.336 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.336 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.337 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:37.337 DEBUG:boto:Data: 2024-04-16T20:07:37.337 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.337 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.337 DEBUG:boto:Params: {} 2024-04-16T20:07:37.337 DEBUG:boto:Token: None 2024-04-16T20:07:37.337 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46 2024-04-16T20:07:37.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0/sX75RE3sBx+Xq13x28Tl6grBo= 2024-04-16T20:07:37.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0/sX75RE3sBx+Xq13x28Tl6grBo=', 'Content-Length': '0'} 2024-04-16T20:07:37.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078291f0946c698be-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.340 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.340 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.340 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.340 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:37.340 DEBUG:boto:Data: 2024-04-16T20:07:37.340 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.340 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.340 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.340 DEBUG:boto:Params: {} 2024-04-16T20:07:37.340 DEBUG:boto:Token: None 2024-04-16T20:07:37.341 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46 2024-04-16T20:07:37.341 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0/sX75RE3sBx+Xq13x28Tl6grBo= 2024-04-16T20:07:37.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0/sX75RE3sBx+Xq13x28Tl6grBo=', 'Content-Length': '0'} 2024-04-16T20:07:37.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd1b49c640e2033-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:07:37.343 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.344 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.344 DEBUG:boto:Method: GET 2024-04-16T20:07:37.344 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:37.344 DEBUG:boto:Data: 2024-04-16T20:07:37.344 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.344 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.344 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.344 DEBUG:boto:Params: {} 2024-04-16T20:07:37.344 DEBUG:boto:Token: None 2024-04-16T20:07:37.344 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46 2024-04-16T20:07:37.345 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nj2X+AJ3DdWTo9oQ9Lhb4zbp11g= 2024-04-16T20:07:37.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nj2X+AJ3DdWTo9oQ9Lhb4zbp11g=', 'Content-Length': '0'} 2024-04-16T20:07:37.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff656716b71469c1-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.390 DEBUG:boto:path=/yrjefh-52/obj46 2024-04-16T20:07:37.391 DEBUG:boto:auth_path=/yrjefh-52/obj46 2024-04-16T20:07:37.391 DEBUG:boto:Method: GET 2024-04-16T20:07:37.391 DEBUG:boto:Path: /yrjefh-52/obj46 2024-04-16T20:07:37.391 DEBUG:boto:Data: 2024-04-16T20:07:37.391 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.391 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.391 DEBUG:boto:Params: {} 2024-04-16T20:07:37.391 DEBUG:boto:Token: None 2024-04-16T20:07:37.391 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj46 2024-04-16T20:07:37.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nj2X+AJ3DdWTo9oQ9Lhb4zbp11g= 2024-04-16T20:07:37.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nj2X+AJ3DdWTo9oQ9Lhb4zbp11g=', 'Content-Length': '0'} 2024-04-16T20:07:37.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239eeb711e6b897f-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.394 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.394 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.395 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.395 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.395 DEBUG:boto:Method: GET 2024-04-16T20:07:37.395 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.395 DEBUG:boto:Data: 2024-04-16T20:07:37.395 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.395 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.395 DEBUG:boto:Params: {} 2024-04-16T20:07:37.395 DEBUG:boto:Token: None 2024-04-16T20:07:37.396 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981eddb5c462a030-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.438 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.439 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.439 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.439 DEBUG:boto:path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.439 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.439 DEBUG:boto:Method: GET 2024-04-16T20:07:37.439 DEBUG:boto:Path: /yrjefh-52/?prefix=obj46 2024-04-16T20:07:37.439 DEBUG:boto:Data: 2024-04-16T20:07:37.439 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.440 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.440 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.440 DEBUG:boto:Params: {} 2024-04-16T20:07:37.440 DEBUG:boto:Token: None 2024-04-16T20:07:37.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2f9bd4c281cb9a-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.444 DEBUG:boto:b'yrjefh-52obj461000falseobj462024-04-16T20:01:27.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:37.444 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.444 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.444 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.444 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.445 DEBUG:boto:Method: GET 2024-04-16T20:07:37.445 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.445 DEBUG:boto:Data: 2024-04-16T20:07:37.445 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.445 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.445 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.445 DEBUG:boto:Params: {} 2024-04-16T20:07:37.445 DEBUG:boto:Token: None 2024-04-16T20:07:37.445 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.445 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0= 2024-04-16T20:07:37.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0=', 'Content-Length': '0'} 2024-04-16T20:07:37.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237b7d7972e81950-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.490 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.491 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.491 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.491 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.491 DEBUG:boto:Method: GET 2024-04-16T20:07:37.491 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.491 DEBUG:boto:Data: 2024-04-16T20:07:37.491 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.491 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.491 DEBUG:boto:Params: {} 2024-04-16T20:07:37.492 DEBUG:boto:Token: None 2024-04-16T20:07:37.492 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0= 2024-04-16T20:07:37.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0=', 'Content-Length': '0'} 2024-04-16T20:07:37.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d081537073181661-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.495 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.495 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.495 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.495 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.495 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.495 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.495 DEBUG:boto:Data: 2024-04-16T20:07:37.495 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.495 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.496 DEBUG:boto:Params: {} 2024-04-16T20:07:37.496 DEBUG:boto:Token: None 2024-04-16T20:07:37.496 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c76iSUR+XyZ/l86Yp3NXWkdKj/c= 2024-04-16T20:07:37.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c76iSUR+XyZ/l86Yp3NXWkdKj/c=', 'Content-Length': '0'} 2024-04-16T20:07:37.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ac83f5536845a4-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.499 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.499 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.499 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.499 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.499 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.499 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.499 DEBUG:boto:Data: 2024-04-16T20:07:37.499 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.500 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.500 DEBUG:boto:Params: {} 2024-04-16T20:07:37.500 DEBUG:boto:Token: None 2024-04-16T20:07:37.500 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c76iSUR+XyZ/l86Yp3NXWkdKj/c= 2024-04-16T20:07:37.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c76iSUR+XyZ/l86Yp3NXWkdKj/c=', 'Content-Length': '0'} 2024-04-16T20:07:37.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec22ec12210a4c3-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:37.503 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.503 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.503 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.503 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.503 DEBUG:boto:Method: GET 2024-04-16T20:07:37.503 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.503 DEBUG:boto:Data: 2024-04-16T20:07:37.503 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.504 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.504 DEBUG:boto:Params: {} 2024-04-16T20:07:37.504 DEBUG:boto:Token: None 2024-04-16T20:07:37.504 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.504 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0= 2024-04-16T20:07:37.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0=', 'Content-Length': '0'} 2024-04-16T20:07:37.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008671f7062e5bcf2e-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.550 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.550 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.551 DEBUG:boto:path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.551 DEBUG:boto:auth_path=/yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.551 DEBUG:boto:Method: GET 2024-04-16T20:07:37.551 DEBUG:boto:Path: /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.551 DEBUG:boto:Data: 2024-04-16T20:07:37.551 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.551 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.551 DEBUG:boto:Params: {} 2024-04-16T20:07:37.551 DEBUG:boto:Token: None 2024-04-16T20:07:37.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51?versionId=null 2024-04-16T20:07:37.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0= 2024-04-16T20:07:37.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mt8gTG0/vdEr55WsAqM1DjgdNK0=', 'Content-Length': '0'} 2024-04-16T20:07:37.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f3b54b58386384-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.555 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.555 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.555 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.555 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.555 DEBUG:boto:Method: GET 2024-04-16T20:07:37.555 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.555 DEBUG:boto:Data: 2024-04-16T20:07:37.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.556 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.556 DEBUG:boto:Params: {} 2024-04-16T20:07:37.556 DEBUG:boto:Token: None 2024-04-16T20:07:37.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a0fbc3155edf6f-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.559 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.559 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.559 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.559 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.559 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.560 DEBUG:boto:Method: GET 2024-04-16T20:07:37.560 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.560 DEBUG:boto:Data: 2024-04-16T20:07:37.560 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.560 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.560 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.560 DEBUG:boto:Params: {} 2024-04-16T20:07:37.560 DEBUG:boto:Token: None 2024-04-16T20:07:37.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.561 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a3142b0c772fc3-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.606 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.607 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.607 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.607 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.607 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:37.607 DEBUG:boto:Data: 2024-04-16T20:07:37.607 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.607 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.607 DEBUG:boto:Params: {} 2024-04-16T20:07:37.608 DEBUG:boto:Token: None 2024-04-16T20:07:37.608 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51 2024-04-16T20:07:37.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AHbOrls2lA6l5XL38JVjqIkIiLM= 2024-04-16T20:07:37.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AHbOrls2lA6l5XL38JVjqIkIiLM=', 'Content-Length': '0'} 2024-04-16T20:07:37.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058481dc1ec14413a-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.610 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.610 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.611 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.611 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:37.611 DEBUG:boto:Data: 2024-04-16T20:07:37.611 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.611 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.611 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.611 DEBUG:boto:Params: {} 2024-04-16T20:07:37.611 DEBUG:boto:Token: None 2024-04-16T20:07:37.611 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51 2024-04-16T20:07:37.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AHbOrls2lA6l5XL38JVjqIkIiLM= 2024-04-16T20:07:37.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AHbOrls2lA6l5XL38JVjqIkIiLM=', 'Content-Length': '0'} 2024-04-16T20:07:37.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6605dc327d93f56-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:07:37.614 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.614 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.614 DEBUG:boto:Method: GET 2024-04-16T20:07:37.615 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:37.615 DEBUG:boto:Data: 2024-04-16T20:07:37.615 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.615 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.615 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.615 DEBUG:boto:Params: {} 2024-04-16T20:07:37.615 DEBUG:boto:Token: None 2024-04-16T20:07:37.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51 2024-04-16T20:07:37.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QtibRls2DIizGxH1vYO4pd/Vw9w= 2024-04-16T20:07:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QtibRls2DIizGxH1vYO4pd/Vw9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007518065aa433e602-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.658 DEBUG:boto:path=/yrjefh-52/obj51 2024-04-16T20:07:37.659 DEBUG:boto:auth_path=/yrjefh-52/obj51 2024-04-16T20:07:37.659 DEBUG:boto:Method: GET 2024-04-16T20:07:37.659 DEBUG:boto:Path: /yrjefh-52/obj51 2024-04-16T20:07:37.659 DEBUG:boto:Data: 2024-04-16T20:07:37.659 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.659 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.659 DEBUG:boto:Params: {} 2024-04-16T20:07:37.659 DEBUG:boto:Token: None 2024-04-16T20:07:37.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj51 2024-04-16T20:07:37.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QtibRls2DIizGxH1vYO4pd/Vw9w= 2024-04-16T20:07:37.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QtibRls2DIizGxH1vYO4pd/Vw9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cbd68003b24cc4-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.662 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.662 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.663 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.663 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.663 DEBUG:boto:Method: GET 2024-04-16T20:07:37.663 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.663 DEBUG:boto:Data: 2024-04-16T20:07:37.663 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.663 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.663 DEBUG:boto:Params: {} 2024-04-16T20:07:37.664 DEBUG:boto:Token: None 2024-04-16T20:07:37.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c853f6cce22311-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.667 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.667 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.667 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.667 DEBUG:boto:path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.668 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.668 DEBUG:boto:Method: GET 2024-04-16T20:07:37.668 DEBUG:boto:Path: /yrjefh-52/?prefix=obj51 2024-04-16T20:07:37.668 DEBUG:boto:Data: 2024-04-16T20:07:37.668 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.668 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.668 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.668 DEBUG:boto:Params: {} 2024-04-16T20:07:37.668 DEBUG:boto:Token: None 2024-04-16T20:07:37.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b308b95d376caf5-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.673 DEBUG:boto:b'yrjefh-52obj511000falseobj512024-04-16T19:58:15.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:37.673 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.673 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.673 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.673 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.673 DEBUG:boto:Method: GET 2024-04-16T20:07:37.674 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.674 DEBUG:boto:Data: 2024-04-16T20:07:37.674 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.674 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.674 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.674 DEBUG:boto:Params: {} 2024-04-16T20:07:37.674 DEBUG:boto:Token: None 2024-04-16T20:07:37.674 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.675 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w= 2024-04-16T20:07:37.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ed41d4d745088c4-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.718 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.718 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.719 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.719 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.719 DEBUG:boto:Method: GET 2024-04-16T20:07:37.719 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.719 DEBUG:boto:Data: 2024-04-16T20:07:37.719 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.719 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.719 DEBUG:boto:Params: {} 2024-04-16T20:07:37.719 DEBUG:boto:Token: None 2024-04-16T20:07:37.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w= 2024-04-16T20:07:37.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773c8f2ef4c83322-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.722 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.722 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.723 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.723 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.723 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.723 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.723 DEBUG:boto:Data: 2024-04-16T20:07:37.723 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.723 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.723 DEBUG:boto:Params: {} 2024-04-16T20:07:37.723 DEBUG:boto:Token: None 2024-04-16T20:07:37.724 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.724 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:G/5iFpB5w+wNtPx5pl4wMKXhU2Y= 2024-04-16T20:07:37.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:G/5iFpB5w+wNtPx5pl4wMKXhU2Y=', 'Content-Length': '0'} 2024-04-16T20:07:37.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a5791346c9f39f0-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.726 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.727 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.727 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.727 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.727 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.727 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.727 DEBUG:boto:Data: 2024-04-16T20:07:37.727 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.727 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.727 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.727 DEBUG:boto:Params: {} 2024-04-16T20:07:37.727 DEBUG:boto:Token: None 2024-04-16T20:07:37.728 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:G/5iFpB5w+wNtPx5pl4wMKXhU2Y= 2024-04-16T20:07:37.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:G/5iFpB5w+wNtPx5pl4wMKXhU2Y=', 'Content-Length': '0'} 2024-04-16T20:07:37.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c7a21d71aba8bf-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:37.731 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.731 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.731 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.731 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.731 DEBUG:boto:Method: GET 2024-04-16T20:07:37.731 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.731 DEBUG:boto:Data: 2024-04-16T20:07:37.731 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.731 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.731 DEBUG:boto:Params: {} 2024-04-16T20:07:37.732 DEBUG:boto:Token: None 2024-04-16T20:07:37.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w= 2024-04-16T20:07:37.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab08367aa45a818-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.778 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.779 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.779 DEBUG:boto:path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.779 DEBUG:boto:auth_path=/yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.779 DEBUG:boto:Method: GET 2024-04-16T20:07:37.779 DEBUG:boto:Path: /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.779 DEBUG:boto:Data: 2024-04-16T20:07:37.779 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.779 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.779 DEBUG:boto:Params: {} 2024-04-16T20:07:37.779 DEBUG:boto:Token: None 2024-04-16T20:07:37.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55?versionId=null 2024-04-16T20:07:37.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w= 2024-04-16T20:07:37.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ibzx4wYAkY5x9ovLeAgVpSKbB9w=', 'Content-Length': '0'} 2024-04-16T20:07:37.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b60b83fc8401b5c-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.782 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.782 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.782 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.782 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.782 DEBUG:boto:Method: GET 2024-04-16T20:07:37.782 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.782 DEBUG:boto:Data: 2024-04-16T20:07:37.783 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.783 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.783 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.783 DEBUG:boto:Params: {} 2024-04-16T20:07:37.783 DEBUG:boto:Token: None 2024-04-16T20:07:37.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db76a61f2754675-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.786 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.786 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.786 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.786 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.786 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.786 DEBUG:boto:Method: GET 2024-04-16T20:07:37.786 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.787 DEBUG:boto:Data: 2024-04-16T20:07:37.787 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.787 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.787 DEBUG:boto:Params: {} 2024-04-16T20:07:37.787 DEBUG:boto:Token: None 2024-04-16T20:07:37.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec61e75a6dc45dc-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.791 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.792 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.792 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.792 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.792 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:37.792 DEBUG:boto:Data: 2024-04-16T20:07:37.792 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.792 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.792 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.792 DEBUG:boto:Params: {} 2024-04-16T20:07:37.792 DEBUG:boto:Token: None 2024-04-16T20:07:37.792 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55 2024-04-16T20:07:37.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i7DKTXWiYVxB3OgK2yy1CfEZMQs= 2024-04-16T20:07:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i7DKTXWiYVxB3OgK2yy1CfEZMQs=', 'Content-Length': '0'} 2024-04-16T20:07:37.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab17e247adb4bbda-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.795 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.795 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.795 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.795 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:37.795 DEBUG:boto:Data: 2024-04-16T20:07:37.795 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.796 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.796 DEBUG:boto:Params: {} 2024-04-16T20:07:37.796 DEBUG:boto:Token: None 2024-04-16T20:07:37.796 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55 2024-04-16T20:07:37.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:i7DKTXWiYVxB3OgK2yy1CfEZMQs= 2024-04-16T20:07:37.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:i7DKTXWiYVxB3OgK2yy1CfEZMQs=', 'Content-Length': '0'} 2024-04-16T20:07:37.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff140fb27fd8b8e0-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:07:37.798 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.798 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.798 DEBUG:boto:Method: GET 2024-04-16T20:07:37.798 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:37.799 DEBUG:boto:Data: 2024-04-16T20:07:37.799 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.799 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.799 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.799 DEBUG:boto:Params: {} 2024-04-16T20:07:37.799 DEBUG:boto:Token: None 2024-04-16T20:07:37.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55 2024-04-16T20:07:37.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X18kncSu6EQYOVNqd3lkaWQUjkA= 2024-04-16T20:07:37.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X18kncSu6EQYOVNqd3lkaWQUjkA=', 'Content-Length': '0'} 2024-04-16T20:07:37.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013663ba4ec13023e-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.842 DEBUG:boto:path=/yrjefh-52/obj55 2024-04-16T20:07:37.842 DEBUG:boto:auth_path=/yrjefh-52/obj55 2024-04-16T20:07:37.842 DEBUG:boto:Method: GET 2024-04-16T20:07:37.843 DEBUG:boto:Path: /yrjefh-52/obj55 2024-04-16T20:07:37.843 DEBUG:boto:Data: 2024-04-16T20:07:37.843 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.843 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.843 DEBUG:boto:Params: {} 2024-04-16T20:07:37.843 DEBUG:boto:Token: None 2024-04-16T20:07:37.843 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj55 2024-04-16T20:07:37.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X18kncSu6EQYOVNqd3lkaWQUjkA= 2024-04-16T20:07:37.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X18kncSu6EQYOVNqd3lkaWQUjkA=', 'Content-Length': '0'} 2024-04-16T20:07:37.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7e25263379e0f1-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.846 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.846 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.846 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.846 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.847 DEBUG:boto:Method: GET 2024-04-16T20:07:37.847 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.847 DEBUG:boto:Data: 2024-04-16T20:07:37.847 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.847 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.847 DEBUG:boto:Params: {} 2024-04-16T20:07:37.847 DEBUG:boto:Token: None 2024-04-16T20:07:37.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fbddcd434e3a9ab-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.850 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.850 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.850 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.851 DEBUG:boto:path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.851 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.851 DEBUG:boto:Method: GET 2024-04-16T20:07:37.851 DEBUG:boto:Path: /yrjefh-52/?prefix=obj55 2024-04-16T20:07:37.851 DEBUG:boto:Data: 2024-04-16T20:07:37.851 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.851 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.851 DEBUG:boto:Params: {} 2024-04-16T20:07:37.851 DEBUG:boto:Token: None 2024-04-16T20:07:37.851 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c41ddb55f7f5c31-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.855 DEBUG:boto:b'yrjefh-52obj551000falseobj552024-04-16T19:58:04.179Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:37.855 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.855 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.855 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.856 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.856 DEBUG:boto:Method: GET 2024-04-16T20:07:37.856 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.856 DEBUG:boto:Data: 2024-04-16T20:07:37.856 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.856 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.856 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.856 DEBUG:boto:Params: {} 2024-04-16T20:07:37.856 DEBUG:boto:Token: None 2024-04-16T20:07:37.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY= 2024-04-16T20:07:37.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY=', 'Content-Length': '0'} 2024-04-16T20:07:37.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fe259c92b618a1b-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.902 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.902 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.903 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.903 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.903 DEBUG:boto:Method: GET 2024-04-16T20:07:37.903 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.903 DEBUG:boto:Data: 2024-04-16T20:07:37.903 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.903 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.903 DEBUG:boto:Params: {} 2024-04-16T20:07:37.903 DEBUG:boto:Token: None 2024-04-16T20:07:37.903 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY= 2024-04-16T20:07:37.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY=', 'Content-Length': '0'} 2024-04-16T20:07:37.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000275c10497cf2fd6a-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.906 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.906 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.906 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.906 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.906 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.906 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.906 DEBUG:boto:Data: 2024-04-16T20:07:37.907 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.907 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.907 DEBUG:boto:Params: {} 2024-04-16T20:07:37.907 DEBUG:boto:Token: None 2024-04-16T20:07:37.907 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.907 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PIMQ1Q7pTbaJfq88u8EA797XskM= 2024-04-16T20:07:37.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PIMQ1Q7pTbaJfq88u8EA797XskM=', 'Content-Length': '0'} 2024-04-16T20:07:37.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c39599fc170dcff3-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.909 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.909 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.910 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.910 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.910 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.910 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.910 DEBUG:boto:Data: 2024-04-16T20:07:37.910 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.910 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.910 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.910 DEBUG:boto:Params: {} 2024-04-16T20:07:37.910 DEBUG:boto:Token: None 2024-04-16T20:07:37.910 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.911 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PIMQ1Q7pTbaJfq88u8EA797XskM= 2024-04-16T20:07:37.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PIMQ1Q7pTbaJfq88u8EA797XskM=', 'Content-Length': '0'} 2024-04-16T20:07:37.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a288f65b13c898-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:37.913 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.913 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.913 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.913 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.913 DEBUG:boto:Method: GET 2024-04-16T20:07:37.913 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.913 DEBUG:boto:Data: 2024-04-16T20:07:37.914 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.914 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.914 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.914 DEBUG:boto:Params: {} 2024-04-16T20:07:37.914 DEBUG:boto:Token: None 2024-04-16T20:07:37.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.914 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY= 2024-04-16T20:07:37.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY=', 'Content-Length': '0'} 2024-04-16T20:07:37.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b442f5bddf1e0903-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.916 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.916 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.917 DEBUG:boto:path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.917 DEBUG:boto:auth_path=/yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.917 DEBUG:boto:Method: GET 2024-04-16T20:07:37.917 DEBUG:boto:Path: /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.917 DEBUG:boto:Data: 2024-04-16T20:07:37.917 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.917 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.917 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.917 DEBUG:boto:Params: {} 2024-04-16T20:07:37.917 DEBUG:boto:Token: None 2024-04-16T20:07:37.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57?versionId=null 2024-04-16T20:07:37.918 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY= 2024-04-16T20:07:37.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WKoy+mQ3P6FU/9yjVFeWFOJtGWY=', 'Content-Length': '0'} 2024-04-16T20:07:37.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc86f11e8c671ce2-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.920 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.920 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.920 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.920 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.921 DEBUG:boto:Method: GET 2024-04-16T20:07:37.921 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.921 DEBUG:boto:Data: 2024-04-16T20:07:37.921 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.921 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.921 DEBUG:boto:Params: {} 2024-04-16T20:07:37.921 DEBUG:boto:Token: None 2024-04-16T20:07:37.921 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.921 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9ec195eef72818-00661eda89-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.924 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.924 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:37.925 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:37.925 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.925 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.925 DEBUG:boto:Method: GET 2024-04-16T20:07:37.925 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:37.925 DEBUG:boto:Data: 2024-04-16T20:07:37.925 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.925 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.925 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.925 DEBUG:boto:Params: {} 2024-04-16T20:07:37.925 DEBUG:boto:Token: None 2024-04-16T20:07:37.926 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/ 2024-04-16T20:07:37.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU= 2024-04-16T20:07:37.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kyaonyENgpfNj6sAw+oFPn9A/DU=', 'Content-Length': '0'} 2024-04-16T20:07:37.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f86a25d5200f0129-00661eda89-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.970 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:37.971 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.971 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.971 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.971 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:37.971 DEBUG:boto:Data: 2024-04-16T20:07:37.971 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.971 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.971 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.971 DEBUG:boto:Params: {} 2024-04-16T20:07:37.971 DEBUG:boto:Token: None 2024-04-16T20:07:37.971 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57 2024-04-16T20:07:37.972 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YJ5oL7KObkfTMOWSiANRTy2nDYo= 2024-04-16T20:07:37.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YJ5oL7KObkfTMOWSiANRTy2nDYo=', 'Content-Length': '0'} 2024-04-16T20:07:37.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf26779c90771bc-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.974 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.974 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.974 DEBUG:boto:Method: HEAD 2024-04-16T20:07:37.974 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:37.974 DEBUG:boto:Data: 2024-04-16T20:07:37.974 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.974 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:37.975 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.975 DEBUG:boto:Params: {} 2024-04-16T20:07:37.975 DEBUG:boto:Token: None 2024-04-16T20:07:37.975 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57 2024-04-16T20:07:37.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YJ5oL7KObkfTMOWSiANRTy2nDYo= 2024-04-16T20:07:37.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YJ5oL7KObkfTMOWSiANRTy2nDYo=', 'Content-Length': '0'} 2024-04-16T20:07:37.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a37a8f1cfb537a-00661eda89-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:37.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:07:37.977 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:37.978 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:37.978 DEBUG:boto:Method: GET 2024-04-16T20:07:37.978 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:37.978 DEBUG:boto:Data: 2024-04-16T20:07:37.978 DEBUG:boto:Headers: {} 2024-04-16T20:07:37.978 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:37.978 DEBUG:boto:Port: 8000 2024-04-16T20:07:37.978 DEBUG:boto:Params: {} 2024-04-16T20:07:37.978 DEBUG:boto:Token: None 2024-04-16T20:07:37.978 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:37 GMT /yrjefh-52/obj57 2024-04-16T20:07:37.978 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V1c0z828TZRTeRhITiGuvdBCENY= 2024-04-16T20:07:37.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V1c0z828TZRTeRhITiGuvdBCENY=', 'Content-Length': '0'} 2024-04-16T20:07:37.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e745891b71845d-00661eda89-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.022 DEBUG:boto:path=/yrjefh-52/obj57 2024-04-16T20:07:38.022 DEBUG:boto:auth_path=/yrjefh-52/obj57 2024-04-16T20:07:38.022 DEBUG:boto:Method: GET 2024-04-16T20:07:38.023 DEBUG:boto:Path: /yrjefh-52/obj57 2024-04-16T20:07:38.023 DEBUG:boto:Data: 2024-04-16T20:07:38.023 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.023 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.023 DEBUG:boto:Params: {} 2024-04-16T20:07:38.023 DEBUG:boto:Token: None 2024-04-16T20:07:38.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj57 2024-04-16T20:07:38.023 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8Dhj/TR6fsz1X1S++u+tWPmTVZw= 2024-04-16T20:07:38.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8Dhj/TR6fsz1X1S++u+tWPmTVZw=', 'Content-Length': '0'} 2024-04-16T20:07:38.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd731eaa4bfb8003-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.025 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.025 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.025 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.025 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.026 DEBUG:boto:Method: GET 2024-04-16T20:07:38.026 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.026 DEBUG:boto:Data: 2024-04-16T20:07:38.026 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.026 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.026 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.026 DEBUG:boto:Params: {} 2024-04-16T20:07:38.026 DEBUG:boto:Token: None 2024-04-16T20:07:38.026 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.026 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10d2c020b571516-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.070 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.071 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.071 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.071 DEBUG:boto:path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.071 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.071 DEBUG:boto:Method: GET 2024-04-16T20:07:38.071 DEBUG:boto:Path: /yrjefh-52/?prefix=obj57 2024-04-16T20:07:38.071 DEBUG:boto:Data: 2024-04-16T20:07:38.071 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.071 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.071 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.072 DEBUG:boto:Params: {} 2024-04-16T20:07:38.072 DEBUG:boto:Token: None 2024-04-16T20:07:38.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ce039c956649914-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.075 DEBUG:boto:b'yrjefh-52obj571000falseobj572024-04-16T20:00:29.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:38.076 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.076 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.076 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.076 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.076 DEBUG:boto:Method: GET 2024-04-16T20:07:38.076 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.076 DEBUG:boto:Data: 2024-04-16T20:07:38.076 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.077 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.077 DEBUG:boto:Params: {} 2024-04-16T20:07:38.077 DEBUG:boto:Token: None 2024-04-16T20:07:38.077 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.077 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw= 2024-04-16T20:07:38.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw=', 'Content-Length': '0'} 2024-04-16T20:07:38.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004caa55cbd3258634-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.123 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.123 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.123 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.123 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.123 DEBUG:boto:Method: GET 2024-04-16T20:07:38.123 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.123 DEBUG:boto:Data: 2024-04-16T20:07:38.123 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.123 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.123 DEBUG:boto:Params: {} 2024-04-16T20:07:38.124 DEBUG:boto:Token: None 2024-04-16T20:07:38.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw= 2024-04-16T20:07:38.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw=', 'Content-Length': '0'} 2024-04-16T20:07:38.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5abb0b91f132c22-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.126 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.127 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.127 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.127 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.127 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.127 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.127 DEBUG:boto:Data: 2024-04-16T20:07:38.127 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.127 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.127 DEBUG:boto:Params: {} 2024-04-16T20:07:38.127 DEBUG:boto:Token: None 2024-04-16T20:07:38.128 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RzuqhY1RCNexAwxGe8xm2G8iCok= 2024-04-16T20:07:38.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RzuqhY1RCNexAwxGe8xm2G8iCok=', 'Content-Length': '0'} 2024-04-16T20:07:38.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d551e5c89247bf9-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.130 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.130 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.130 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.130 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.130 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.131 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.131 DEBUG:boto:Data: 2024-04-16T20:07:38.131 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.131 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.131 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.131 DEBUG:boto:Params: {} 2024-04-16T20:07:38.131 DEBUG:boto:Token: None 2024-04-16T20:07:38.131 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.131 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RzuqhY1RCNexAwxGe8xm2G8iCok= 2024-04-16T20:07:38.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RzuqhY1RCNexAwxGe8xm2G8iCok=', 'Content-Length': '0'} 2024-04-16T20:07:38.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4006067f9748002-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:38.134 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.134 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.134 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.134 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.134 DEBUG:boto:Method: GET 2024-04-16T20:07:38.134 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.134 DEBUG:boto:Data: 2024-04-16T20:07:38.134 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.134 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.134 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.135 DEBUG:boto:Params: {} 2024-04-16T20:07:38.135 DEBUG:boto:Token: None 2024-04-16T20:07:38.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw= 2024-04-16T20:07:38.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw=', 'Content-Length': '0'} 2024-04-16T20:07:38.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c917575b129c240-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.179 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.179 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.179 DEBUG:boto:path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.179 DEBUG:boto:auth_path=/yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.179 DEBUG:boto:Method: GET 2024-04-16T20:07:38.179 DEBUG:boto:Path: /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.179 DEBUG:boto:Data: 2024-04-16T20:07:38.179 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.179 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.180 DEBUG:boto:Params: {} 2024-04-16T20:07:38.180 DEBUG:boto:Token: None 2024-04-16T20:07:38.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58?versionId=null 2024-04-16T20:07:38.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw= 2024-04-16T20:07:38.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0UCnNhziSM45REDm6ke2LOoYRqw=', 'Content-Length': '0'} 2024-04-16T20:07:38.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c90681387005eac-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.182 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.183 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.183 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.183 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.183 DEBUG:boto:Method: GET 2024-04-16T20:07:38.183 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.183 DEBUG:boto:Data: 2024-04-16T20:07:38.183 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.183 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.183 DEBUG:boto:Params: {} 2024-04-16T20:07:38.183 DEBUG:boto:Token: None 2024-04-16T20:07:38.184 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de37839f246a1f08-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.226 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.227 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.227 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.227 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.227 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.227 DEBUG:boto:Method: GET 2024-04-16T20:07:38.227 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.227 DEBUG:boto:Data: 2024-04-16T20:07:38.227 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.227 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.227 DEBUG:boto:Params: {} 2024-04-16T20:07:38.228 DEBUG:boto:Token: None 2024-04-16T20:07:38.228 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.228 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388a07e8f9f341a4-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.231 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.232 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.232 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.232 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.232 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:38.232 DEBUG:boto:Data: 2024-04-16T20:07:38.232 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.232 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.232 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.233 DEBUG:boto:Params: {} 2024-04-16T20:07:38.233 DEBUG:boto:Token: None 2024-04-16T20:07:38.233 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58 2024-04-16T20:07:38.233 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q0D9kA+BrFl0hTnmQDIfeTUcv4U= 2024-04-16T20:07:38.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q0D9kA+BrFl0hTnmQDIfeTUcv4U=', 'Content-Length': '0'} 2024-04-16T20:07:38.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3988543f64c1da8-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.235 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.235 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.236 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.236 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:38.236 DEBUG:boto:Data: 2024-04-16T20:07:38.236 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.236 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.236 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.236 DEBUG:boto:Params: {} 2024-04-16T20:07:38.236 DEBUG:boto:Token: None 2024-04-16T20:07:38.236 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58 2024-04-16T20:07:38.236 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q0D9kA+BrFl0hTnmQDIfeTUcv4U= 2024-04-16T20:07:38.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q0D9kA+BrFl0hTnmQDIfeTUcv4U=', 'Content-Length': '0'} 2024-04-16T20:07:38.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3241a6f21fa2c35-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:07:38.239 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.239 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.239 DEBUG:boto:Method: GET 2024-04-16T20:07:38.239 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:38.239 DEBUG:boto:Data: 2024-04-16T20:07:38.239 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.240 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.240 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.240 DEBUG:boto:Params: {} 2024-04-16T20:07:38.240 DEBUG:boto:Token: None 2024-04-16T20:07:38.240 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58 2024-04-16T20:07:38.240 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMHVS1gFSIMwZVp1oeBgLVNYxkU= 2024-04-16T20:07:38.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMHVS1gFSIMwZVp1oeBgLVNYxkU=', 'Content-Length': '0'} 2024-04-16T20:07:38.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d18974a7e873a87-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.286 DEBUG:boto:path=/yrjefh-52/obj58 2024-04-16T20:07:38.286 DEBUG:boto:auth_path=/yrjefh-52/obj58 2024-04-16T20:07:38.287 DEBUG:boto:Method: GET 2024-04-16T20:07:38.287 DEBUG:boto:Path: /yrjefh-52/obj58 2024-04-16T20:07:38.287 DEBUG:boto:Data: 2024-04-16T20:07:38.287 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.287 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.287 DEBUG:boto:Params: {} 2024-04-16T20:07:38.287 DEBUG:boto:Token: None 2024-04-16T20:07:38.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj58 2024-04-16T20:07:38.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rMHVS1gFSIMwZVp1oeBgLVNYxkU= 2024-04-16T20:07:38.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMHVS1gFSIMwZVp1oeBgLVNYxkU=', 'Content-Length': '0'} 2024-04-16T20:07:38.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0691f50a983882-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.290 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.290 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.290 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.290 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.290 DEBUG:boto:Method: GET 2024-04-16T20:07:38.291 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.291 DEBUG:boto:Data: 2024-04-16T20:07:38.291 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.291 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.291 DEBUG:boto:Params: {} 2024-04-16T20:07:38.291 DEBUG:boto:Token: None 2024-04-16T20:07:38.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce06b944fec340c3-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.334 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.335 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.335 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.335 DEBUG:boto:path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.335 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.335 DEBUG:boto:Method: GET 2024-04-16T20:07:38.335 DEBUG:boto:Path: /yrjefh-52/?prefix=obj58 2024-04-16T20:07:38.335 DEBUG:boto:Data: 2024-04-16T20:07:38.335 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.335 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.335 DEBUG:boto:Params: {} 2024-04-16T20:07:38.336 DEBUG:boto:Token: None 2024-04-16T20:07:38.336 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009415c575725897e7-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.339 DEBUG:boto:b'yrjefh-52obj581000falseobj582024-04-16T20:03:33.254Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:38.340 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.340 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.340 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.340 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.340 DEBUG:boto:Method: GET 2024-04-16T20:07:38.340 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.340 DEBUG:boto:Data: 2024-04-16T20:07:38.341 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.341 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.341 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.341 DEBUG:boto:Params: {} 2024-04-16T20:07:38.341 DEBUG:boto:Token: None 2024-04-16T20:07:38.341 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.341 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o= 2024-04-16T20:07:38.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o=', 'Content-Length': '0'} 2024-04-16T20:07:38.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ebef38139cf2020-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.386 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.386 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.387 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.387 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.387 DEBUG:boto:Method: GET 2024-04-16T20:07:38.387 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.387 DEBUG:boto:Data: 2024-04-16T20:07:38.387 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.387 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.387 DEBUG:boto:Params: {} 2024-04-16T20:07:38.387 DEBUG:boto:Token: None 2024-04-16T20:07:38.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o= 2024-04-16T20:07:38.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o=', 'Content-Length': '0'} 2024-04-16T20:07:38.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ec5471798699f37-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.390 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.390 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.390 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.390 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.391 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.391 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.391 DEBUG:boto:Data: 2024-04-16T20:07:38.391 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.391 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.391 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.391 DEBUG:boto:Params: {} 2024-04-16T20:07:38.391 DEBUG:boto:Token: None 2024-04-16T20:07:38.391 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9A8ZmUPbpSU/Jnz6AkOiVtH9DOA= 2024-04-16T20:07:38.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9A8ZmUPbpSU/Jnz6AkOiVtH9DOA=', 'Content-Length': '0'} 2024-04-16T20:07:38.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089eb8a65e5ea8c7f-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.394 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.394 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.394 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.394 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.394 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.394 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.394 DEBUG:boto:Data: 2024-04-16T20:07:38.394 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.394 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.395 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.395 DEBUG:boto:Params: {} 2024-04-16T20:07:38.395 DEBUG:boto:Token: None 2024-04-16T20:07:38.395 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9A8ZmUPbpSU/Jnz6AkOiVtH9DOA= 2024-04-16T20:07:38.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9A8ZmUPbpSU/Jnz6AkOiVtH9DOA=', 'Content-Length': '0'} 2024-04-16T20:07:38.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460ba501b4cec560-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:38.398 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.398 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.398 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.398 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.398 DEBUG:boto:Method: GET 2024-04-16T20:07:38.398 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.398 DEBUG:boto:Data: 2024-04-16T20:07:38.398 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.398 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.399 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.399 DEBUG:boto:Params: {} 2024-04-16T20:07:38.399 DEBUG:boto:Token: None 2024-04-16T20:07:38.399 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.399 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o= 2024-04-16T20:07:38.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o=', 'Content-Length': '0'} 2024-04-16T20:07:38.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e737350e162294c-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.402 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.402 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.402 DEBUG:boto:path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.402 DEBUG:boto:auth_path=/yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.402 DEBUG:boto:Method: GET 2024-04-16T20:07:38.402 DEBUG:boto:Path: /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.402 DEBUG:boto:Data: 2024-04-16T20:07:38.402 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.403 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.403 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.403 DEBUG:boto:Params: {} 2024-04-16T20:07:38.403 DEBUG:boto:Token: None 2024-04-16T20:07:38.403 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62?versionId=null 2024-04-16T20:07:38.403 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o= 2024-04-16T20:07:38.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:D6zuDqXP/2QYI7Jj9z3lcorll7o=', 'Content-Length': '0'} 2024-04-16T20:07:38.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa64c5f0d26e04fc-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.406 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.406 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.406 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.406 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.406 DEBUG:boto:Method: GET 2024-04-16T20:07:38.406 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.406 DEBUG:boto:Data: 2024-04-16T20:07:38.406 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.406 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.406 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.406 DEBUG:boto:Params: {} 2024-04-16T20:07:38.407 DEBUG:boto:Token: None 2024-04-16T20:07:38.407 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000696369f255f52a4b-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.410 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.410 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.410 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.410 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.410 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.410 DEBUG:boto:Method: GET 2024-04-16T20:07:38.410 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.410 DEBUG:boto:Data: 2024-04-16T20:07:38.411 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.411 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.411 DEBUG:boto:Params: {} 2024-04-16T20:07:38.411 DEBUG:boto:Token: None 2024-04-16T20:07:38.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a86c51d41dfcf6-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.415 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.415 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.415 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.415 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.415 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:38.415 DEBUG:boto:Data: 2024-04-16T20:07:38.416 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.416 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.416 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.416 DEBUG:boto:Params: {} 2024-04-16T20:07:38.416 DEBUG:boto:Token: None 2024-04-16T20:07:38.416 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62 2024-04-16T20:07:38.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0DiPO+dzKxV0hH1U/j+awkSXrs= 2024-04-16T20:07:38.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0DiPO+dzKxV0hH1U/j+awkSXrs=', 'Content-Length': '0'} 2024-04-16T20:07:38.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b31e32e23c23e2-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.418 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.419 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.419 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.419 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:38.419 DEBUG:boto:Data: 2024-04-16T20:07:38.419 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.419 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.419 DEBUG:boto:Params: {} 2024-04-16T20:07:38.419 DEBUG:boto:Token: None 2024-04-16T20:07:38.419 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62 2024-04-16T20:07:38.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R0DiPO+dzKxV0hH1U/j+awkSXrs= 2024-04-16T20:07:38.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R0DiPO+dzKxV0hH1U/j+awkSXrs=', 'Content-Length': '0'} 2024-04-16T20:07:38.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015219db84f6d5828-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:07:38.422 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.422 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.422 DEBUG:boto:Method: GET 2024-04-16T20:07:38.423 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:38.423 DEBUG:boto:Data: 2024-04-16T20:07:38.423 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.423 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.423 DEBUG:boto:Params: {} 2024-04-16T20:07:38.423 DEBUG:boto:Token: None 2024-04-16T20:07:38.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62 2024-04-16T20:07:38.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+I4/l4WY0RX3M9OLaBk6AtJ5bRw= 2024-04-16T20:07:38.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+I4/l4WY0RX3M9OLaBk6AtJ5bRw=', 'Content-Length': '0'} 2024-04-16T20:07:38.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99c5d274ae1418f-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.466 DEBUG:boto:path=/yrjefh-52/obj62 2024-04-16T20:07:38.466 DEBUG:boto:auth_path=/yrjefh-52/obj62 2024-04-16T20:07:38.467 DEBUG:boto:Method: GET 2024-04-16T20:07:38.467 DEBUG:boto:Path: /yrjefh-52/obj62 2024-04-16T20:07:38.467 DEBUG:boto:Data: 2024-04-16T20:07:38.467 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.467 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.467 DEBUG:boto:Params: {} 2024-04-16T20:07:38.467 DEBUG:boto:Token: None 2024-04-16T20:07:38.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj62 2024-04-16T20:07:38.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+I4/l4WY0RX3M9OLaBk6AtJ5bRw= 2024-04-16T20:07:38.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+I4/l4WY0RX3M9OLaBk6AtJ5bRw=', 'Content-Length': '0'} 2024-04-16T20:07:38.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124fff2a3ab2d638-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.470 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.470 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.470 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.470 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.471 DEBUG:boto:Method: GET 2024-04-16T20:07:38.471 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.471 DEBUG:boto:Data: 2024-04-16T20:07:38.471 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.471 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.471 DEBUG:boto:Params: {} 2024-04-16T20:07:38.471 DEBUG:boto:Token: None 2024-04-16T20:07:38.471 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002849ac55d87c3dd5-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.514 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.515 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.515 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.515 DEBUG:boto:path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.515 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.515 DEBUG:boto:Method: GET 2024-04-16T20:07:38.515 DEBUG:boto:Path: /yrjefh-52/?prefix=obj62 2024-04-16T20:07:38.515 DEBUG:boto:Data: 2024-04-16T20:07:38.515 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.515 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.515 DEBUG:boto:Params: {} 2024-04-16T20:07:38.516 DEBUG:boto:Token: None 2024-04-16T20:07:38.516 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d480c1d550905e6-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.520 DEBUG:boto:b'yrjefh-52obj621000falseobj622024-04-16T20:04:43.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:38.520 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.520 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.520 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.520 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.521 DEBUG:boto:Method: GET 2024-04-16T20:07:38.521 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.521 DEBUG:boto:Data: 2024-04-16T20:07:38.521 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.521 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.521 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.521 DEBUG:boto:Params: {} 2024-04-16T20:07:38.521 DEBUG:boto:Token: None 2024-04-16T20:07:38.521 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE= 2024-04-16T20:07:38.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE=', 'Content-Length': '0'} 2024-04-16T20:07:38.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b978b51de195801b-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.566 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.567 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.567 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.567 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.567 DEBUG:boto:Method: GET 2024-04-16T20:07:38.567 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.567 DEBUG:boto:Data: 2024-04-16T20:07:38.567 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.567 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.567 DEBUG:boto:Params: {} 2024-04-16T20:07:38.567 DEBUG:boto:Token: None 2024-04-16T20:07:38.568 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE= 2024-04-16T20:07:38.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE=', 'Content-Length': '0'} 2024-04-16T20:07:38.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452f9e95ddeba546-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.570 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.570 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.570 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.570 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.571 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.571 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.571 DEBUG:boto:Data: 2024-04-16T20:07:38.571 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.571 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.571 DEBUG:boto:Params: {} 2024-04-16T20:07:38.571 DEBUG:boto:Token: None 2024-04-16T20:07:38.571 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/uw2kpD4d+P5m0C3DTtHgXUkM1k= 2024-04-16T20:07:38.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/uw2kpD4d+P5m0C3DTtHgXUkM1k=', 'Content-Length': '0'} 2024-04-16T20:07:38.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a63dcf0baf0e2d-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.574 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.574 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.574 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.574 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.574 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.574 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.574 DEBUG:boto:Data: 2024-04-16T20:07:38.575 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.575 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.575 DEBUG:boto:Params: {} 2024-04-16T20:07:38.575 DEBUG:boto:Token: None 2024-04-16T20:07:38.575 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/uw2kpD4d+P5m0C3DTtHgXUkM1k= 2024-04-16T20:07:38.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/uw2kpD4d+P5m0C3DTtHgXUkM1k=', 'Content-Length': '0'} 2024-04-16T20:07:38.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b09afb745b7c0f84-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:38.577 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.578 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.578 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.578 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.578 DEBUG:boto:Method: GET 2024-04-16T20:07:38.578 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.578 DEBUG:boto:Data: 2024-04-16T20:07:38.578 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.578 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.578 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.578 DEBUG:boto:Params: {} 2024-04-16T20:07:38.578 DEBUG:boto:Token: None 2024-04-16T20:07:38.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE= 2024-04-16T20:07:38.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE=', 'Content-Length': '0'} 2024-04-16T20:07:38.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9e5e44a4075fa9-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.622 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.623 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.623 DEBUG:boto:path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.623 DEBUG:boto:auth_path=/yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.623 DEBUG:boto:Method: GET 2024-04-16T20:07:38.623 DEBUG:boto:Path: /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.623 DEBUG:boto:Data: 2024-04-16T20:07:38.623 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.623 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.623 DEBUG:boto:Params: {} 2024-04-16T20:07:38.624 DEBUG:boto:Token: None 2024-04-16T20:07:38.624 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68?versionId=null 2024-04-16T20:07:38.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE= 2024-04-16T20:07:38.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XA7a9yl4dMo8oTZPNMrhueFd+zE=', 'Content-Length': '0'} 2024-04-16T20:07:38.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010fa98512461cd7a-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.626 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.626 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.627 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.627 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.627 DEBUG:boto:Method: GET 2024-04-16T20:07:38.627 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.627 DEBUG:boto:Data: 2024-04-16T20:07:38.627 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.627 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.627 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.627 DEBUG:boto:Params: {} 2024-04-16T20:07:38.627 DEBUG:boto:Token: None 2024-04-16T20:07:38.627 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2476f6ff6aef85f-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.670 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.671 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.671 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.671 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.671 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.671 DEBUG:boto:Method: GET 2024-04-16T20:07:38.671 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.671 DEBUG:boto:Data: 2024-04-16T20:07:38.671 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.671 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.671 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.672 DEBUG:boto:Params: {} 2024-04-16T20:07:38.672 DEBUG:boto:Token: None 2024-04-16T20:07:38.672 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac5b57e290630cf4-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.676 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.676 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.676 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.676 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.677 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:38.677 DEBUG:boto:Data: 2024-04-16T20:07:38.677 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.677 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.677 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.677 DEBUG:boto:Params: {} 2024-04-16T20:07:38.677 DEBUG:boto:Token: None 2024-04-16T20:07:38.677 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68 2024-04-16T20:07:38.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qfb6kaQVj0Po+aI3c5GaebyfjHk= 2024-04-16T20:07:38.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qfb6kaQVj0Po+aI3c5GaebyfjHk=', 'Content-Length': '0'} 2024-04-16T20:07:38.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005271fb92a8b00b02-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.680 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.680 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.680 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.680 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:38.680 DEBUG:boto:Data: 2024-04-16T20:07:38.680 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.681 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.681 DEBUG:boto:Params: {} 2024-04-16T20:07:38.681 DEBUG:boto:Token: None 2024-04-16T20:07:38.681 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68 2024-04-16T20:07:38.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Qfb6kaQVj0Po+aI3c5GaebyfjHk= 2024-04-16T20:07:38.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Qfb6kaQVj0Po+aI3c5GaebyfjHk=', 'Content-Length': '0'} 2024-04-16T20:07:38.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038701d39af606e7-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:07:38.683 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.684 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.684 DEBUG:boto:Method: GET 2024-04-16T20:07:38.684 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:38.684 DEBUG:boto:Data: 2024-04-16T20:07:38.684 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.684 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.684 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.684 DEBUG:boto:Params: {} 2024-04-16T20:07:38.684 DEBUG:boto:Token: None 2024-04-16T20:07:38.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68 2024-04-16T20:07:38.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dLZFer+QUILoOqyMlHr9qlNDnas= 2024-04-16T20:07:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dLZFer+QUILoOqyMlHr9qlNDnas=', 'Content-Length': '0'} 2024-04-16T20:07:38.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359bc44b347a1a9c-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.730 DEBUG:boto:path=/yrjefh-52/obj68 2024-04-16T20:07:38.731 DEBUG:boto:auth_path=/yrjefh-52/obj68 2024-04-16T20:07:38.731 DEBUG:boto:Method: GET 2024-04-16T20:07:38.731 DEBUG:boto:Path: /yrjefh-52/obj68 2024-04-16T20:07:38.731 DEBUG:boto:Data: 2024-04-16T20:07:38.731 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.731 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.731 DEBUG:boto:Params: {} 2024-04-16T20:07:38.731 DEBUG:boto:Token: None 2024-04-16T20:07:38.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj68 2024-04-16T20:07:38.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dLZFer+QUILoOqyMlHr9qlNDnas= 2024-04-16T20:07:38.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dLZFer+QUILoOqyMlHr9qlNDnas=', 'Content-Length': '0'} 2024-04-16T20:07:38.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac868314b41dd1e-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.734 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.734 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.734 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.734 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.734 DEBUG:boto:Method: GET 2024-04-16T20:07:38.735 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.735 DEBUG:boto:Data: 2024-04-16T20:07:38.735 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.735 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.735 DEBUG:boto:Params: {} 2024-04-16T20:07:38.735 DEBUG:boto:Token: None 2024-04-16T20:07:38.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd5769d2766ee2b-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.778 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.779 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.779 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.779 DEBUG:boto:path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.779 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.779 DEBUG:boto:Method: GET 2024-04-16T20:07:38.779 DEBUG:boto:Path: /yrjefh-52/?prefix=obj68 2024-04-16T20:07:38.779 DEBUG:boto:Data: 2024-04-16T20:07:38.779 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.779 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.780 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.780 DEBUG:boto:Params: {} 2024-04-16T20:07:38.780 DEBUG:boto:Token: None 2024-04-16T20:07:38.780 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a26d5ebeeb2fd49-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.784 DEBUG:boto:b'yrjefh-52obj681000falseobj682024-04-16T19:58:15.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:38.784 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.785 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.785 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.785 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.785 DEBUG:boto:Method: GET 2024-04-16T20:07:38.785 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.785 DEBUG:boto:Data: 2024-04-16T20:07:38.785 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.785 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.785 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.785 DEBUG:boto:Params: {} 2024-04-16T20:07:38.786 DEBUG:boto:Token: None 2024-04-16T20:07:38.786 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.786 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc= 2024-04-16T20:07:38.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc=', 'Content-Length': '0'} 2024-04-16T20:07:38.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf865f4f1fb9f39-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.830 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.831 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.831 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.831 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.831 DEBUG:boto:Method: GET 2024-04-16T20:07:38.831 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.831 DEBUG:boto:Data: 2024-04-16T20:07:38.831 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.831 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.831 DEBUG:boto:Params: {} 2024-04-16T20:07:38.832 DEBUG:boto:Token: None 2024-04-16T20:07:38.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc= 2024-04-16T20:07:38.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc=', 'Content-Length': '0'} 2024-04-16T20:07:38.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000528977deb414dd12-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.835 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.835 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.835 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.835 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.835 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.835 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.835 DEBUG:boto:Data: 2024-04-16T20:07:38.835 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.835 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.836 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.836 DEBUG:boto:Params: {} 2024-04-16T20:07:38.836 DEBUG:boto:Token: None 2024-04-16T20:07:38.836 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cVpHN3YBAlrqhfKVa2w3KjA8kV8= 2024-04-16T20:07:38.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cVpHN3YBAlrqhfKVa2w3KjA8kV8=', 'Content-Length': '0'} 2024-04-16T20:07:38.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cde7ae9514f5a2-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.839 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.839 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.839 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.839 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.839 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.839 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.839 DEBUG:boto:Data: 2024-04-16T20:07:38.839 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.839 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.840 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.840 DEBUG:boto:Params: {} 2024-04-16T20:07:38.840 DEBUG:boto:Token: None 2024-04-16T20:07:38.840 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cVpHN3YBAlrqhfKVa2w3KjA8kV8= 2024-04-16T20:07:38.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cVpHN3YBAlrqhfKVa2w3KjA8kV8=', 'Content-Length': '0'} 2024-04-16T20:07:38.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a473fb2a630fce03-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:38.843 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.843 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.843 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.843 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.844 DEBUG:boto:Method: GET 2024-04-16T20:07:38.844 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.844 DEBUG:boto:Data: 2024-04-16T20:07:38.844 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.844 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.844 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.844 DEBUG:boto:Params: {} 2024-04-16T20:07:38.844 DEBUG:boto:Token: None 2024-04-16T20:07:38.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.845 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc= 2024-04-16T20:07:38.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc=', 'Content-Length': '0'} 2024-04-16T20:07:38.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721197c7f9c2f50f-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.890 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.891 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.891 DEBUG:boto:path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.891 DEBUG:boto:auth_path=/yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.891 DEBUG:boto:Method: GET 2024-04-16T20:07:38.891 DEBUG:boto:Path: /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.891 DEBUG:boto:Data: 2024-04-16T20:07:38.891 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.891 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.892 DEBUG:boto:Params: {} 2024-04-16T20:07:38.892 DEBUG:boto:Token: None 2024-04-16T20:07:38.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69?versionId=null 2024-04-16T20:07:38.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc= 2024-04-16T20:07:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4y97lzda831BAe+UxM+zDuBK7fc=', 'Content-Length': '0'} 2024-04-16T20:07:38.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46eb187ccdaaf94-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.895 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.895 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.895 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.895 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.895 DEBUG:boto:Method: GET 2024-04-16T20:07:38.895 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.895 DEBUG:boto:Data: 2024-04-16T20:07:38.895 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.896 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.896 DEBUG:boto:Params: {} 2024-04-16T20:07:38.896 DEBUG:boto:Token: None 2024-04-16T20:07:38.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006997ebcf439a20d-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.899 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.899 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.899 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.900 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.900 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.900 DEBUG:boto:Method: GET 2024-04-16T20:07:38.900 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.900 DEBUG:boto:Data: 2024-04-16T20:07:38.900 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.900 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.900 DEBUG:boto:Params: {} 2024-04-16T20:07:38.901 DEBUG:boto:Token: None 2024-04-16T20:07:38.901 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e2205111e92363-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.905 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:38.905 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.905 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.905 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.905 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:38.905 DEBUG:boto:Data: 2024-04-16T20:07:38.905 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.906 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.906 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.906 DEBUG:boto:Params: {} 2024-04-16T20:07:38.906 DEBUG:boto:Token: None 2024-04-16T20:07:38.906 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69 2024-04-16T20:07:38.906 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qXgmcvw7nKwG/dBkvdFz+D+xQdA= 2024-04-16T20:07:38.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qXgmcvw7nKwG/dBkvdFz+D+xQdA=', 'Content-Length': '0'} 2024-04-16T20:07:38.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc08d0595d2cd46f-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.909 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.909 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.909 DEBUG:boto:Method: HEAD 2024-04-16T20:07:38.909 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:38.909 DEBUG:boto:Data: 2024-04-16T20:07:38.909 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.909 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.909 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.910 DEBUG:boto:Params: {} 2024-04-16T20:07:38.910 DEBUG:boto:Token: None 2024-04-16T20:07:38.910 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69 2024-04-16T20:07:38.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qXgmcvw7nKwG/dBkvdFz+D+xQdA= 2024-04-16T20:07:38.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qXgmcvw7nKwG/dBkvdFz+D+xQdA=', 'Content-Length': '0'} 2024-04-16T20:07:38.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c451ccd69d9e56-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:07:38.913 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.913 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.913 DEBUG:boto:Method: GET 2024-04-16T20:07:38.913 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:38.913 DEBUG:boto:Data: 2024-04-16T20:07:38.913 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.913 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.913 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.913 DEBUG:boto:Params: {} 2024-04-16T20:07:38.913 DEBUG:boto:Token: None 2024-04-16T20:07:38.914 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69 2024-04-16T20:07:38.914 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Cr+19wRHD9JoR4tazn7x2uc+L7E= 2024-04-16T20:07:38.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Cr+19wRHD9JoR4tazn7x2uc+L7E=', 'Content-Length': '0'} 2024-04-16T20:07:38.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff79651af814265-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.958 DEBUG:boto:path=/yrjefh-52/obj69 2024-04-16T20:07:38.959 DEBUG:boto:auth_path=/yrjefh-52/obj69 2024-04-16T20:07:38.959 DEBUG:boto:Method: GET 2024-04-16T20:07:38.959 DEBUG:boto:Path: /yrjefh-52/obj69 2024-04-16T20:07:38.959 DEBUG:boto:Data: 2024-04-16T20:07:38.959 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:38.959 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.959 DEBUG:boto:Params: {} 2024-04-16T20:07:38.960 DEBUG:boto:Token: None 2024-04-16T20:07:38.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/obj69 2024-04-16T20:07:38.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Cr+19wRHD9JoR4tazn7x2uc+L7E= 2024-04-16T20:07:38.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Cr+19wRHD9JoR4tazn7x2uc+L7E=', 'Content-Length': '0'} 2024-04-16T20:07:38.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb168e25ff464c42-00661eda8a-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:38.962 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:38.963 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:38.963 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.963 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.963 DEBUG:boto:Method: GET 2024-04-16T20:07:38.963 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:38.963 DEBUG:boto:Data: 2024-04-16T20:07:38.963 DEBUG:boto:Headers: {} 2024-04-16T20:07:38.963 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:38.963 DEBUG:boto:Port: 8000 2024-04-16T20:07:38.963 DEBUG:boto:Params: {} 2024-04-16T20:07:38.964 DEBUG:boto:Token: None 2024-04-16T20:07:38.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:38 GMT /yrjefh-52/ 2024-04-16T20:07:38.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0= 2024-04-16T20:07:38.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:38 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k+jpVX3srSNJ5j9jgC9gu9WFvF0=', 'Content-Length': '0'} 2024-04-16T20:07:38.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3c43ac2c10a82d-00661eda8a-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.010 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.011 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.011 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.011 DEBUG:boto:path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:39.011 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj69 2024-04-16T20:07:39.011 DEBUG:boto:Method: GET 2024-04-16T20:07:39.011 DEBUG:boto:Path: /yrjefh-52/?prefix=obj69 2024-04-16T20:07:39.011 DEBUG:boto:Data: 2024-04-16T20:07:39.011 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.012 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.012 DEBUG:boto:Params: {} 2024-04-16T20:07:39.012 DEBUG:boto:Token: None 2024-04-16T20:07:39.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000468ece8478f6cec5-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.016 DEBUG:boto:b'yrjefh-52obj691000falseobj692024-04-16T20:01:27.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:39.016 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.016 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.016 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.017 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.017 DEBUG:boto:Method: GET 2024-04-16T20:07:39.017 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.017 DEBUG:boto:Data: 2024-04-16T20:07:39.017 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.017 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.017 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.017 DEBUG:boto:Params: {} 2024-04-16T20:07:39.017 DEBUG:boto:Token: None 2024-04-16T20:07:39.017 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0= 2024-04-16T20:07:39.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0=', 'Content-Length': '0'} 2024-04-16T20:07:39.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072bbd5df48c202d-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.066 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.066 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.067 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.067 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.067 DEBUG:boto:Method: GET 2024-04-16T20:07:39.067 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.067 DEBUG:boto:Data: 2024-04-16T20:07:39.067 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.067 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.067 DEBUG:boto:Params: {} 2024-04-16T20:07:39.067 DEBUG:boto:Token: None 2024-04-16T20:07:39.067 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0= 2024-04-16T20:07:39.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0=', 'Content-Length': '0'} 2024-04-16T20:07:39.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007139d91f54d59dfd-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.070 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.070 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.071 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.071 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.071 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.071 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.071 DEBUG:boto:Data: 2024-04-16T20:07:39.071 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.071 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.071 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.071 DEBUG:boto:Params: {} 2024-04-16T20:07:39.071 DEBUG:boto:Token: None 2024-04-16T20:07:39.071 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:scpzy4ATb8u3FzUv6ViZHrhAxbI= 2024-04-16T20:07:39.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:scpzy4ATb8u3FzUv6ViZHrhAxbI=', 'Content-Length': '0'} 2024-04-16T20:07:39.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad09972bed23f566-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.074 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.074 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.074 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.074 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.074 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.074 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.074 DEBUG:boto:Data: 2024-04-16T20:07:39.075 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.075 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.075 DEBUG:boto:Params: {} 2024-04-16T20:07:39.075 DEBUG:boto:Token: None 2024-04-16T20:07:39.075 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:scpzy4ATb8u3FzUv6ViZHrhAxbI= 2024-04-16T20:07:39.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:scpzy4ATb8u3FzUv6ViZHrhAxbI=', 'Content-Length': '0'} 2024-04-16T20:07:39.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1b17f653187e9e-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:39.077 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.078 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.078 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.078 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.078 DEBUG:boto:Method: GET 2024-04-16T20:07:39.078 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.078 DEBUG:boto:Data: 2024-04-16T20:07:39.078 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.078 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.078 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.078 DEBUG:boto:Params: {} 2024-04-16T20:07:39.078 DEBUG:boto:Token: None 2024-04-16T20:07:39.079 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.079 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0= 2024-04-16T20:07:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0=', 'Content-Length': '0'} 2024-04-16T20:07:39.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e173b8f0e31a4068-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.122 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.123 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.123 DEBUG:boto:path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.123 DEBUG:boto:auth_path=/yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.123 DEBUG:boto:Method: GET 2024-04-16T20:07:39.123 DEBUG:boto:Path: /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.123 DEBUG:boto:Data: 2024-04-16T20:07:39.123 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.123 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.123 DEBUG:boto:Params: {} 2024-04-16T20:07:39.123 DEBUG:boto:Token: None 2024-04-16T20:07:39.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74?versionId=null 2024-04-16T20:07:39.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0= 2024-04-16T20:07:39.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRt4aBmIipobkvEPQ5zQvvowqR0=', 'Content-Length': '0'} 2024-04-16T20:07:39.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004905a7f100943029-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.126 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.126 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.126 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.126 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.126 DEBUG:boto:Method: GET 2024-04-16T20:07:39.127 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.127 DEBUG:boto:Data: 2024-04-16T20:07:39.127 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.127 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.127 DEBUG:boto:Params: {} 2024-04-16T20:07:39.127 DEBUG:boto:Token: None 2024-04-16T20:07:39.127 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003024dde84836d363-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.170 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.171 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.171 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.171 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.171 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.171 DEBUG:boto:Method: GET 2024-04-16T20:07:39.171 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.171 DEBUG:boto:Data: 2024-04-16T20:07:39.171 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.171 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.172 DEBUG:boto:Params: {} 2024-04-16T20:07:39.172 DEBUG:boto:Token: None 2024-04-16T20:07:39.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a3034e13d09cf6-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.176 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.176 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.176 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.176 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.176 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:39.176 DEBUG:boto:Data: 2024-04-16T20:07:39.177 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.177 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.177 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.177 DEBUG:boto:Params: {} 2024-04-16T20:07:39.177 DEBUG:boto:Token: None 2024-04-16T20:07:39.177 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74 2024-04-16T20:07:39.177 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mmXd6xMfIs9gbDHjV+tVSWAu3LI= 2024-04-16T20:07:39.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mmXd6xMfIs9gbDHjV+tVSWAu3LI=', 'Content-Length': '0'} 2024-04-16T20:07:39.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4998ea00884e1b-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.179 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.179 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.179 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.180 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:39.180 DEBUG:boto:Data: 2024-04-16T20:07:39.180 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.180 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.180 DEBUG:boto:Params: {} 2024-04-16T20:07:39.180 DEBUG:boto:Token: None 2024-04-16T20:07:39.180 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74 2024-04-16T20:07:39.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mmXd6xMfIs9gbDHjV+tVSWAu3LI= 2024-04-16T20:07:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mmXd6xMfIs9gbDHjV+tVSWAu3LI=', 'Content-Length': '0'} 2024-04-16T20:07:39.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005712620c937e1d05-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:07:39.182 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.182 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.183 DEBUG:boto:Method: GET 2024-04-16T20:07:39.183 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:39.183 DEBUG:boto:Data: 2024-04-16T20:07:39.183 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.183 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.183 DEBUG:boto:Params: {} 2024-04-16T20:07:39.183 DEBUG:boto:Token: None 2024-04-16T20:07:39.183 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74 2024-04-16T20:07:39.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xm6l6sJodCGt6WkJyuMy1F8QsuU= 2024-04-16T20:07:39.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xm6l6sJodCGt6WkJyuMy1F8QsuU=', 'Content-Length': '0'} 2024-04-16T20:07:39.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada456e3c9fa8935-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.226 DEBUG:boto:path=/yrjefh-52/obj74 2024-04-16T20:07:39.226 DEBUG:boto:auth_path=/yrjefh-52/obj74 2024-04-16T20:07:39.227 DEBUG:boto:Method: GET 2024-04-16T20:07:39.227 DEBUG:boto:Path: /yrjefh-52/obj74 2024-04-16T20:07:39.227 DEBUG:boto:Data: 2024-04-16T20:07:39.227 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.227 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.227 DEBUG:boto:Params: {} 2024-04-16T20:07:39.227 DEBUG:boto:Token: None 2024-04-16T20:07:39.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj74 2024-04-16T20:07:39.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xm6l6sJodCGt6WkJyuMy1F8QsuU= 2024-04-16T20:07:39.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xm6l6sJodCGt6WkJyuMy1F8QsuU=', 'Content-Length': '0'} 2024-04-16T20:07:39.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:06:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b907a97d67f501c2-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.230 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.230 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.230 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.230 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.230 DEBUG:boto:Method: GET 2024-04-16T20:07:39.230 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.230 DEBUG:boto:Data: 2024-04-16T20:07:39.230 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.230 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.230 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.230 DEBUG:boto:Params: {} 2024-04-16T20:07:39.231 DEBUG:boto:Token: None 2024-04-16T20:07:39.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f981049d26ecbd-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.274 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.275 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.275 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.275 DEBUG:boto:path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.275 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.275 DEBUG:boto:Method: GET 2024-04-16T20:07:39.275 DEBUG:boto:Path: /yrjefh-52/?prefix=obj74 2024-04-16T20:07:39.275 DEBUG:boto:Data: 2024-04-16T20:07:39.275 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.275 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.275 DEBUG:boto:Params: {} 2024-04-16T20:07:39.275 DEBUG:boto:Token: None 2024-04-16T20:07:39.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5f8f5c97fd80b8-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.279 DEBUG:boto:b'yrjefh-52obj741000falseobj742024-04-16T20:06:00.797Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:39.279 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.280 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.280 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.280 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.280 DEBUG:boto:Method: GET 2024-04-16T20:07:39.280 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.280 DEBUG:boto:Data: 2024-04-16T20:07:39.280 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.280 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.280 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.280 DEBUG:boto:Params: {} 2024-04-16T20:07:39.280 DEBUG:boto:Token: None 2024-04-16T20:07:39.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI= 2024-04-16T20:07:39.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI=', 'Content-Length': '0'} 2024-04-16T20:07:39.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f6a3e35d2c9390-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.326 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.326 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.326 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.327 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.327 DEBUG:boto:Method: GET 2024-04-16T20:07:39.327 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.327 DEBUG:boto:Data: 2024-04-16T20:07:39.327 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.327 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.327 DEBUG:boto:Params: {} 2024-04-16T20:07:39.327 DEBUG:boto:Token: None 2024-04-16T20:07:39.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI= 2024-04-16T20:07:39.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI=', 'Content-Length': '0'} 2024-04-16T20:07:39.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb905d3e9774fecb-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.330 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.330 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.330 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.330 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.330 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.330 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.330 DEBUG:boto:Data: 2024-04-16T20:07:39.330 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.330 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.331 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.331 DEBUG:boto:Params: {} 2024-04-16T20:07:39.331 DEBUG:boto:Token: None 2024-04-16T20:07:39.331 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:keg3Vo5Pk3qRKgDtvHvL3mZYYtw= 2024-04-16T20:07:39.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:keg3Vo5Pk3qRKgDtvHvL3mZYYtw=', 'Content-Length': '0'} 2024-04-16T20:07:39.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3c5b3d4c25a68c-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.333 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.333 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.333 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.333 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.333 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.334 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.334 DEBUG:boto:Data: 2024-04-16T20:07:39.334 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.334 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.334 DEBUG:boto:Params: {} 2024-04-16T20:07:39.334 DEBUG:boto:Token: None 2024-04-16T20:07:39.334 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:keg3Vo5Pk3qRKgDtvHvL3mZYYtw= 2024-04-16T20:07:39.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:keg3Vo5Pk3qRKgDtvHvL3mZYYtw=', 'Content-Length': '0'} 2024-04-16T20:07:39.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e87406e694f079-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:39.337 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.337 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.337 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.337 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.337 DEBUG:boto:Method: GET 2024-04-16T20:07:39.337 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.337 DEBUG:boto:Data: 2024-04-16T20:07:39.337 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.337 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.337 DEBUG:boto:Params: {} 2024-04-16T20:07:39.337 DEBUG:boto:Token: None 2024-04-16T20:07:39.338 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI= 2024-04-16T20:07:39.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI=', 'Content-Length': '0'} 2024-04-16T20:07:39.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7147590841649f6-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.382 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.383 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.383 DEBUG:boto:path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.383 DEBUG:boto:auth_path=/yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.383 DEBUG:boto:Method: GET 2024-04-16T20:07:39.383 DEBUG:boto:Path: /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.383 DEBUG:boto:Data: 2024-04-16T20:07:39.383 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.383 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.383 DEBUG:boto:Params: {} 2024-04-16T20:07:39.384 DEBUG:boto:Token: None 2024-04-16T20:07:39.384 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76?versionId=null 2024-04-16T20:07:39.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI= 2024-04-16T20:07:39.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mGnH9ACnb5B5A7FKs62uVAmybeI=', 'Content-Length': '0'} 2024-04-16T20:07:39.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b334eda0e2e8d88c-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.386 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.386 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.386 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.387 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.387 DEBUG:boto:Method: GET 2024-04-16T20:07:39.387 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.387 DEBUG:boto:Data: 2024-04-16T20:07:39.387 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.387 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.387 DEBUG:boto:Params: {} 2024-04-16T20:07:39.387 DEBUG:boto:Token: None 2024-04-16T20:07:39.387 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322b3d962104461d-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.430 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.431 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.431 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.431 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.431 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.431 DEBUG:boto:Method: GET 2024-04-16T20:07:39.431 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.431 DEBUG:boto:Data: 2024-04-16T20:07:39.431 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.431 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.431 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.431 DEBUG:boto:Params: {} 2024-04-16T20:07:39.432 DEBUG:boto:Token: None 2024-04-16T20:07:39.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed62b54221742ce-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.435 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.435 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.436 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.436 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.436 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:39.436 DEBUG:boto:Data: 2024-04-16T20:07:39.436 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.436 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.436 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.436 DEBUG:boto:Params: {} 2024-04-16T20:07:39.436 DEBUG:boto:Token: None 2024-04-16T20:07:39.436 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76 2024-04-16T20:07:39.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8xtbm0uCQDqGzSV3eiY2xiA3fg= 2024-04-16T20:07:39.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8xtbm0uCQDqGzSV3eiY2xiA3fg=', 'Content-Length': '0'} 2024-04-16T20:07:39.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc38ce9ffa33e57-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.438 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.439 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.439 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.439 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:39.439 DEBUG:boto:Data: 2024-04-16T20:07:39.439 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.439 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.439 DEBUG:boto:Params: {} 2024-04-16T20:07:39.439 DEBUG:boto:Token: None 2024-04-16T20:07:39.439 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76 2024-04-16T20:07:39.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8xtbm0uCQDqGzSV3eiY2xiA3fg= 2024-04-16T20:07:39.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8xtbm0uCQDqGzSV3eiY2xiA3fg=', 'Content-Length': '0'} 2024-04-16T20:07:39.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009682d096728fb4ee-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:07:39.442 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.442 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.442 DEBUG:boto:Method: GET 2024-04-16T20:07:39.442 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:39.442 DEBUG:boto:Data: 2024-04-16T20:07:39.442 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.442 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.442 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.442 DEBUG:boto:Params: {} 2024-04-16T20:07:39.442 DEBUG:boto:Token: None 2024-04-16T20:07:39.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76 2024-04-16T20:07:39.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gJ1JLxYybBmt/WYap8o8NQMmESk= 2024-04-16T20:07:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gJ1JLxYybBmt/WYap8o8NQMmESk=', 'Content-Length': '0'} 2024-04-16T20:07:39.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b8cd72268d6e39-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.486 DEBUG:boto:path=/yrjefh-52/obj76 2024-04-16T20:07:39.486 DEBUG:boto:auth_path=/yrjefh-52/obj76 2024-04-16T20:07:39.487 DEBUG:boto:Method: GET 2024-04-16T20:07:39.487 DEBUG:boto:Path: /yrjefh-52/obj76 2024-04-16T20:07:39.487 DEBUG:boto:Data: 2024-04-16T20:07:39.487 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.487 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.487 DEBUG:boto:Params: {} 2024-04-16T20:07:39.487 DEBUG:boto:Token: None 2024-04-16T20:07:39.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj76 2024-04-16T20:07:39.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gJ1JLxYybBmt/WYap8o8NQMmESk= 2024-04-16T20:07:39.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gJ1JLxYybBmt/WYap8o8NQMmESk=', 'Content-Length': '0'} 2024-04-16T20:07:39.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db6de272a8cdb3c-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.490 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.490 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.490 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.490 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.490 DEBUG:boto:Method: GET 2024-04-16T20:07:39.490 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.490 DEBUG:boto:Data: 2024-04-16T20:07:39.490 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.490 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.491 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.491 DEBUG:boto:Params: {} 2024-04-16T20:07:39.491 DEBUG:boto:Token: None 2024-04-16T20:07:39.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f677743e5f0971b-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.534 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.535 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.535 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.535 DEBUG:boto:path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.535 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.535 DEBUG:boto:Method: GET 2024-04-16T20:07:39.535 DEBUG:boto:Path: /yrjefh-52/?prefix=obj76 2024-04-16T20:07:39.535 DEBUG:boto:Data: 2024-04-16T20:07:39.535 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.535 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.535 DEBUG:boto:Params: {} 2024-04-16T20:07:39.536 DEBUG:boto:Token: None 2024-04-16T20:07:39.536 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530c76d36269ef6b-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.540 DEBUG:boto:b'yrjefh-52obj761000falseobj762024-04-16T20:00:29.022Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:39.540 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.540 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.540 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.540 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.541 DEBUG:boto:Method: GET 2024-04-16T20:07:39.541 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.541 DEBUG:boto:Data: 2024-04-16T20:07:39.541 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.541 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.541 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.541 DEBUG:boto:Params: {} 2024-04-16T20:07:39.541 DEBUG:boto:Token: None 2024-04-16T20:07:39.541 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.541 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo= 2024-04-16T20:07:39.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo=', 'Content-Length': '0'} 2024-04-16T20:07:39.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814b785efe6bb8a0-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.586 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.586 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.586 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.587 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.587 DEBUG:boto:Method: GET 2024-04-16T20:07:39.587 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.587 DEBUG:boto:Data: 2024-04-16T20:07:39.587 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.587 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.587 DEBUG:boto:Params: {} 2024-04-16T20:07:39.587 DEBUG:boto:Token: None 2024-04-16T20:07:39.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo= 2024-04-16T20:07:39.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo=', 'Content-Length': '0'} 2024-04-16T20:07:39.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a085d15ce0e8d4e-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.590 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.590 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.590 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.590 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.590 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.591 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.591 DEBUG:boto:Data: 2024-04-16T20:07:39.591 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.591 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.591 DEBUG:boto:Params: {} 2024-04-16T20:07:39.591 DEBUG:boto:Token: None 2024-04-16T20:07:39.591 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xb6AWsKKgMYaF/HrIEw/sXNYyMI= 2024-04-16T20:07:39.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xb6AWsKKgMYaF/HrIEw/sXNYyMI=', 'Content-Length': '0'} 2024-04-16T20:07:39.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010bf99b5630aa1cb-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.594 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.594 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.594 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.594 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.594 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.594 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.594 DEBUG:boto:Data: 2024-04-16T20:07:39.595 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.595 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.595 DEBUG:boto:Params: {} 2024-04-16T20:07:39.595 DEBUG:boto:Token: None 2024-04-16T20:07:39.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xb6AWsKKgMYaF/HrIEw/sXNYyMI= 2024-04-16T20:07:39.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xb6AWsKKgMYaF/HrIEw/sXNYyMI=', 'Content-Length': '0'} 2024-04-16T20:07:39.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662afacc86c4d5a7-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:39.597 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.598 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.598 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.598 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.598 DEBUG:boto:Method: GET 2024-04-16T20:07:39.598 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.598 DEBUG:boto:Data: 2024-04-16T20:07:39.598 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.598 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.598 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.598 DEBUG:boto:Params: {} 2024-04-16T20:07:39.598 DEBUG:boto:Token: None 2024-04-16T20:07:39.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo= 2024-04-16T20:07:39.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo=', 'Content-Length': '0'} 2024-04-16T20:07:39.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d4a901b9f4e68fa-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.642 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.642 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.643 DEBUG:boto:path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.643 DEBUG:boto:auth_path=/yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.643 DEBUG:boto:Method: GET 2024-04-16T20:07:39.643 DEBUG:boto:Path: /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.643 DEBUG:boto:Data: 2024-04-16T20:07:39.643 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.643 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.643 DEBUG:boto:Params: {} 2024-04-16T20:07:39.643 DEBUG:boto:Token: None 2024-04-16T20:07:39.644 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85?versionId=null 2024-04-16T20:07:39.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo= 2024-04-16T20:07:39.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B0shFTpRuCapTVnFMx6dhB3C3lo=', 'Content-Length': '0'} 2024-04-16T20:07:39.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328dd99c335c723e-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.646 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.646 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.646 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.646 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.646 DEBUG:boto:Method: GET 2024-04-16T20:07:39.646 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.647 DEBUG:boto:Data: 2024-04-16T20:07:39.647 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.647 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.647 DEBUG:boto:Params: {} 2024-04-16T20:07:39.647 DEBUG:boto:Token: None 2024-04-16T20:07:39.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120addd546a8b45f-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.690 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.691 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.691 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.691 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.691 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.691 DEBUG:boto:Method: GET 2024-04-16T20:07:39.691 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.691 DEBUG:boto:Data: 2024-04-16T20:07:39.691 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.692 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.692 DEBUG:boto:Params: {} 2024-04-16T20:07:39.692 DEBUG:boto:Token: None 2024-04-16T20:07:39.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eefd381a28eb4cca-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.696 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.696 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.697 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.697 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.697 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:39.697 DEBUG:boto:Data: 2024-04-16T20:07:39.697 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.697 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.697 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.697 DEBUG:boto:Params: {} 2024-04-16T20:07:39.697 DEBUG:boto:Token: None 2024-04-16T20:07:39.697 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85 2024-04-16T20:07:39.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ssVsf2rlfYceyryQ3rg9Qy6V3W4= 2024-04-16T20:07:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ssVsf2rlfYceyryQ3rg9Qy6V3W4=', 'Content-Length': '0'} 2024-04-16T20:07:39.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe38f619226a06c6-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.700 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.700 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.700 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.700 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:39.701 DEBUG:boto:Data: 2024-04-16T20:07:39.701 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.701 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.701 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.701 DEBUG:boto:Params: {} 2024-04-16T20:07:39.701 DEBUG:boto:Token: None 2024-04-16T20:07:39.701 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85 2024-04-16T20:07:39.701 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ssVsf2rlfYceyryQ3rg9Qy6V3W4= 2024-04-16T20:07:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ssVsf2rlfYceyryQ3rg9Qy6V3W4=', 'Content-Length': '0'} 2024-04-16T20:07:39.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5099cdb804530c-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:07:39.704 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.704 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.704 DEBUG:boto:Method: GET 2024-04-16T20:07:39.704 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:39.704 DEBUG:boto:Data: 2024-04-16T20:07:39.704 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.704 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.704 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.705 DEBUG:boto:Params: {} 2024-04-16T20:07:39.705 DEBUG:boto:Token: None 2024-04-16T20:07:39.705 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85 2024-04-16T20:07:39.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2NnQACi+7t5rjjyzM5axgcqFMzE= 2024-04-16T20:07:39.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2NnQACi+7t5rjjyzM5axgcqFMzE=', 'Content-Length': '0'} 2024-04-16T20:07:39.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064a2fcff7e958fd3-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.750 DEBUG:boto:path=/yrjefh-52/obj85 2024-04-16T20:07:39.751 DEBUG:boto:auth_path=/yrjefh-52/obj85 2024-04-16T20:07:39.751 DEBUG:boto:Method: GET 2024-04-16T20:07:39.751 DEBUG:boto:Path: /yrjefh-52/obj85 2024-04-16T20:07:39.751 DEBUG:boto:Data: 2024-04-16T20:07:39.751 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.751 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.751 DEBUG:boto:Params: {} 2024-04-16T20:07:39.751 DEBUG:boto:Token: None 2024-04-16T20:07:39.752 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj85 2024-04-16T20:07:39.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2NnQACi+7t5rjjyzM5axgcqFMzE= 2024-04-16T20:07:39.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2NnQACi+7t5rjjyzM5axgcqFMzE=', 'Content-Length': '0'} 2024-04-16T20:07:39.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:58:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a054a3c3a9ce24-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.754 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.754 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.754 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.755 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.755 DEBUG:boto:Method: GET 2024-04-16T20:07:39.755 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.755 DEBUG:boto:Data: 2024-04-16T20:07:39.755 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.755 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.755 DEBUG:boto:Params: {} 2024-04-16T20:07:39.755 DEBUG:boto:Token: None 2024-04-16T20:07:39.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c5038647c39df7-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.758 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.758 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.759 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.759 DEBUG:boto:path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.759 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.759 DEBUG:boto:Method: GET 2024-04-16T20:07:39.759 DEBUG:boto:Path: /yrjefh-52/?prefix=obj85 2024-04-16T20:07:39.759 DEBUG:boto:Data: 2024-04-16T20:07:39.759 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.759 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.759 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.759 DEBUG:boto:Params: {} 2024-04-16T20:07:39.760 DEBUG:boto:Token: None 2024-04-16T20:07:39.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.760 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4b651b008a6954-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.764 DEBUG:boto:b'yrjefh-52obj851000falseobj852024-04-16T19:58:15.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:39.764 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.764 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.764 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.764 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.764 DEBUG:boto:Method: GET 2024-04-16T20:07:39.765 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.765 DEBUG:boto:Data: 2024-04-16T20:07:39.765 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.765 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.765 DEBUG:boto:Params: {} 2024-04-16T20:07:39.765 DEBUG:boto:Token: None 2024-04-16T20:07:39.765 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g= 2024-04-16T20:07:39.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g=', 'Content-Length': '0'} 2024-04-16T20:07:39.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc81abc0d89b254-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.810 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.811 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.811 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.811 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.811 DEBUG:boto:Method: GET 2024-04-16T20:07:39.811 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.811 DEBUG:boto:Data: 2024-04-16T20:07:39.811 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.811 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.811 DEBUG:boto:Params: {} 2024-04-16T20:07:39.812 DEBUG:boto:Token: None 2024-04-16T20:07:39.812 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g= 2024-04-16T20:07:39.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g=', 'Content-Length': '0'} 2024-04-16T20:07:39.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f81e68af4bf465-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.815 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.815 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.815 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.815 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.815 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.815 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.815 DEBUG:boto:Data: 2024-04-16T20:07:39.815 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.815 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.816 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.816 DEBUG:boto:Params: {} 2024-04-16T20:07:39.816 DEBUG:boto:Token: None 2024-04-16T20:07:39.816 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yTGaqSlK1VdW7bKkrtGHvGqYt48= 2024-04-16T20:07:39.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yTGaqSlK1VdW7bKkrtGHvGqYt48=', 'Content-Length': '0'} 2024-04-16T20:07:39.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006006414b7539eb6a-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.819 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.819 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.819 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.819 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.819 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.819 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.819 DEBUG:boto:Data: 2024-04-16T20:07:39.819 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.820 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.820 DEBUG:boto:Params: {} 2024-04-16T20:07:39.820 DEBUG:boto:Token: None 2024-04-16T20:07:39.820 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yTGaqSlK1VdW7bKkrtGHvGqYt48= 2024-04-16T20:07:39.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yTGaqSlK1VdW7bKkrtGHvGqYt48=', 'Content-Length': '0'} 2024-04-16T20:07:39.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026d532d7d2a5a3eb-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:39.823 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.823 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.823 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.823 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.823 DEBUG:boto:Method: GET 2024-04-16T20:07:39.823 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.823 DEBUG:boto:Data: 2024-04-16T20:07:39.824 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.824 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.824 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.824 DEBUG:boto:Params: {} 2024-04-16T20:07:39.824 DEBUG:boto:Token: None 2024-04-16T20:07:39.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g= 2024-04-16T20:07:39.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g=', 'Content-Length': '0'} 2024-04-16T20:07:39.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c376822426cc50-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.870 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.871 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.871 DEBUG:boto:path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.871 DEBUG:boto:auth_path=/yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.871 DEBUG:boto:Method: GET 2024-04-16T20:07:39.871 DEBUG:boto:Path: /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.871 DEBUG:boto:Data: 2024-04-16T20:07:39.871 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.871 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.872 DEBUG:boto:Params: {} 2024-04-16T20:07:39.872 DEBUG:boto:Token: None 2024-04-16T20:07:39.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87?versionId=null 2024-04-16T20:07:39.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g= 2024-04-16T20:07:39.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Hc8Ss3Nef9A5Rq/5DqjXUcN1M6g=', 'Content-Length': '0'} 2024-04-16T20:07:39.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcfa387fab40a408-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.875 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.875 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.875 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.875 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.875 DEBUG:boto:Method: GET 2024-04-16T20:07:39.875 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.875 DEBUG:boto:Data: 2024-04-16T20:07:39.875 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.876 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.876 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.876 DEBUG:boto:Params: {} 2024-04-16T20:07:39.876 DEBUG:boto:Token: None 2024-04-16T20:07:39.876 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008340293ac5a72c94-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.879 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.880 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.880 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.880 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.880 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.880 DEBUG:boto:Method: GET 2024-04-16T20:07:39.880 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.880 DEBUG:boto:Data: 2024-04-16T20:07:39.880 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.881 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.881 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.881 DEBUG:boto:Params: {} 2024-04-16T20:07:39.881 DEBUG:boto:Token: None 2024-04-16T20:07:39.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4476273c406e9b4-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.885 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.885 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.886 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.886 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.886 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:39.886 DEBUG:boto:Data: 2024-04-16T20:07:39.886 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.886 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.886 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.886 DEBUG:boto:Params: {} 2024-04-16T20:07:39.886 DEBUG:boto:Token: None 2024-04-16T20:07:39.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87 2024-04-16T20:07:39.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vvreXNqCFq6eqy1IY1PpQd2p3IA= 2024-04-16T20:07:39.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vvreXNqCFq6eqy1IY1PpQd2p3IA=', 'Content-Length': '0'} 2024-04-16T20:07:39.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d731223bd8bc0690-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.889 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.889 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.890 DEBUG:boto:Method: HEAD 2024-04-16T20:07:39.890 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:39.890 DEBUG:boto:Data: 2024-04-16T20:07:39.890 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.890 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.890 DEBUG:boto:Params: {} 2024-04-16T20:07:39.890 DEBUG:boto:Token: None 2024-04-16T20:07:39.890 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87 2024-04-16T20:07:39.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vvreXNqCFq6eqy1IY1PpQd2p3IA= 2024-04-16T20:07:39.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vvreXNqCFq6eqy1IY1PpQd2p3IA=', 'Content-Length': '0'} 2024-04-16T20:07:39.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfdd7a5a4ba47fa7-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:07:39.893 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.893 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.894 DEBUG:boto:Method: GET 2024-04-16T20:07:39.894 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:39.894 DEBUG:boto:Data: 2024-04-16T20:07:39.894 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.894 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.894 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.894 DEBUG:boto:Params: {} 2024-04-16T20:07:39.894 DEBUG:boto:Token: None 2024-04-16T20:07:39.894 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87 2024-04-16T20:07:39.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiKy/kYyr7RGe4PsBvsJFNH5eAU= 2024-04-16T20:07:39.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiKy/kYyr7RGe4PsBvsJFNH5eAU=', 'Content-Length': '0'} 2024-04-16T20:07:39.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aac69b9cff0c8b7-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.939 DEBUG:boto:path=/yrjefh-52/obj87 2024-04-16T20:07:39.939 DEBUG:boto:auth_path=/yrjefh-52/obj87 2024-04-16T20:07:39.939 DEBUG:boto:Method: GET 2024-04-16T20:07:39.939 DEBUG:boto:Path: /yrjefh-52/obj87 2024-04-16T20:07:39.939 DEBUG:boto:Data: 2024-04-16T20:07:39.939 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.939 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.940 DEBUG:boto:Params: {} 2024-04-16T20:07:39.940 DEBUG:boto:Token: None 2024-04-16T20:07:39.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj87 2024-04-16T20:07:39.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AiKy/kYyr7RGe4PsBvsJFNH5eAU= 2024-04-16T20:07:39.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AiKy/kYyr7RGe4PsBvsJFNH5eAU=', 'Content-Length': '0'} 2024-04-16T20:07:39.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:03:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e2ef4bb20d0cc70-00661eda8b-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.943 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.943 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.943 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.943 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.943 DEBUG:boto:Method: GET 2024-04-16T20:07:39.943 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.943 DEBUG:boto:Data: 2024-04-16T20:07:39.943 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.944 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.944 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.944 DEBUG:boto:Params: {} 2024-04-16T20:07:39.944 DEBUG:boto:Token: None 2024-04-16T20:07:39.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd84c6805a1ba7c-00661eda8b-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.990 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.991 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:39.991 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:39.991 DEBUG:boto:path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.991 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.991 DEBUG:boto:Method: GET 2024-04-16T20:07:39.991 DEBUG:boto:Path: /yrjefh-52/?prefix=obj87 2024-04-16T20:07:39.991 DEBUG:boto:Data: 2024-04-16T20:07:39.992 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:39.992 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.992 DEBUG:boto:Params: {} 2024-04-16T20:07:39.992 DEBUG:boto:Token: None 2024-04-16T20:07:39.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/ 2024-04-16T20:07:39.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY= 2024-04-16T20:07:39.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NuNez3kJlA/D8kGdbrYg3V0k+oY=', 'Content-Length': '0'} 2024-04-16T20:07:39.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566f1326d2e9dd87-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:39.996 DEBUG:boto:b'yrjefh-52obj871000falseobj872024-04-16T20:03:33.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:39.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:39.997 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:39.997 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:39.997 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:39.997 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:39.997 DEBUG:boto:Method: GET 2024-04-16T20:07:39.997 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:39.998 DEBUG:boto:Data: 2024-04-16T20:07:39.998 DEBUG:boto:Headers: {} 2024-04-16T20:07:39.998 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:39.998 DEBUG:boto:Port: 8000 2024-04-16T20:07:39.998 DEBUG:boto:Params: {} 2024-04-16T20:07:39.998 DEBUG:boto:Token: None 2024-04-16T20:07:39.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:39 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:39.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fduFVhOdZLNqOn8EE0kwzympybY= 2024-04-16T20:07:39.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:39 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fduFVhOdZLNqOn8EE0kwzympybY=', 'Content-Length': '0'} 2024-04-16T20:07:40.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c9436433ab5894-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.042 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.043 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.043 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.043 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.043 DEBUG:boto:Method: GET 2024-04-16T20:07:40.043 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.043 DEBUG:boto:Data: 2024-04-16T20:07:40.043 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.044 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.044 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.044 DEBUG:boto:Params: {} 2024-04-16T20:07:40.044 DEBUG:boto:Token: None 2024-04-16T20:07:40.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo= 2024-04-16T20:07:40.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo=', 'Content-Length': '0'} 2024-04-16T20:07:40.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90e29f81d94c448-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.047 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.047 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.047 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.047 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.047 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.047 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.048 DEBUG:boto:Data: 2024-04-16T20:07:40.048 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.048 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.048 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.048 DEBUG:boto:Params: {} 2024-04-16T20:07:40.048 DEBUG:boto:Token: None 2024-04-16T20:07:40.048 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y85PbgIVer48yA95c+8dZUG3qh0= 2024-04-16T20:07:40.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y85PbgIVer48yA95c+8dZUG3qh0=', 'Content-Length': '0'} 2024-04-16T20:07:40.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008809de87dfbc00bf-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.051 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.051 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.051 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.051 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.052 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.052 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.052 DEBUG:boto:Data: 2024-04-16T20:07:40.052 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.052 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.052 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.052 DEBUG:boto:Params: {} 2024-04-16T20:07:40.052 DEBUG:boto:Token: None 2024-04-16T20:07:40.053 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.053 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y85PbgIVer48yA95c+8dZUG3qh0= 2024-04-16T20:07:40.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y85PbgIVer48yA95c+8dZUG3qh0=', 'Content-Length': '0'} 2024-04-16T20:07:40.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c384e501859d4ff9-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:40.055 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.056 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.056 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.056 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.056 DEBUG:boto:Method: GET 2024-04-16T20:07:40.056 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.056 DEBUG:boto:Data: 2024-04-16T20:07:40.056 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.056 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.057 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.057 DEBUG:boto:Params: {} 2024-04-16T20:07:40.057 DEBUG:boto:Token: None 2024-04-16T20:07:40.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo= 2024-04-16T20:07:40.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo=', 'Content-Length': '0'} 2024-04-16T20:07:40.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025068c578db60111-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.102 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.103 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.103 DEBUG:boto:path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.103 DEBUG:boto:auth_path=/yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.103 DEBUG:boto:Method: GET 2024-04-16T20:07:40.103 DEBUG:boto:Path: /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.103 DEBUG:boto:Data: 2024-04-16T20:07:40.103 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.104 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.104 DEBUG:boto:Params: {} 2024-04-16T20:07:40.104 DEBUG:boto:Token: None 2024-04-16T20:07:40.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92?versionId=null 2024-04-16T20:07:40.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo= 2024-04-16T20:07:40.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4BmzdiTdIt0yTGOQglJcgokyEMo=', 'Content-Length': '0'} 2024-04-16T20:07:40.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e1fd0ea7d1bca6d-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.107 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.107 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.107 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.107 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.107 DEBUG:boto:Method: GET 2024-04-16T20:07:40.107 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.108 DEBUG:boto:Data: 2024-04-16T20:07:40.108 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.108 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.108 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.108 DEBUG:boto:Params: {} 2024-04-16T20:07:40.108 DEBUG:boto:Token: None 2024-04-16T20:07:40.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000501535278055eca8-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.111 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.112 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.112 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.112 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.112 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.112 DEBUG:boto:Method: GET 2024-04-16T20:07:40.113 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.113 DEBUG:boto:Data: 2024-04-16T20:07:40.113 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.113 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.113 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.113 DEBUG:boto:Params: {} 2024-04-16T20:07:40.113 DEBUG:boto:Token: None 2024-04-16T20:07:40.114 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.114 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb833f405d5a115-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.118 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.118 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.118 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.118 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.119 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:40.119 DEBUG:boto:Data: 2024-04-16T20:07:40.119 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.119 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.119 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.119 DEBUG:boto:Params: {} 2024-04-16T20:07:40.119 DEBUG:boto:Token: None 2024-04-16T20:07:40.119 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92 2024-04-16T20:07:40.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlpK71Vup7Qny3fIwxafJ39rKvI= 2024-04-16T20:07:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlpK71Vup7Qny3fIwxafJ39rKvI=', 'Content-Length': '0'} 2024-04-16T20:07:40.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003255a3f05fa9816-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.122 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.122 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.122 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.123 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:40.123 DEBUG:boto:Data: 2024-04-16T20:07:40.123 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.123 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.123 DEBUG:boto:Params: {} 2024-04-16T20:07:40.123 DEBUG:boto:Token: None 2024-04-16T20:07:40.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92 2024-04-16T20:07:40.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LlpK71Vup7Qny3fIwxafJ39rKvI= 2024-04-16T20:07:40.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LlpK71Vup7Qny3fIwxafJ39rKvI=', 'Content-Length': '0'} 2024-04-16T20:07:40.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda5e65b56d8cc61-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:07:40.127 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.127 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.127 DEBUG:boto:Method: GET 2024-04-16T20:07:40.127 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:40.127 DEBUG:boto:Data: 2024-04-16T20:07:40.127 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.127 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.127 DEBUG:boto:Params: {} 2024-04-16T20:07:40.128 DEBUG:boto:Token: None 2024-04-16T20:07:40.128 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92 2024-04-16T20:07:40.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NaEHa8s+h9f06mM/optY0GJUq4U= 2024-04-16T20:07:40.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NaEHa8s+h9f06mM/optY0GJUq4U=', 'Content-Length': '0'} 2024-04-16T20:07:40.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b96b5db58d9d3f-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.171 DEBUG:boto:path=/yrjefh-52/obj92 2024-04-16T20:07:40.171 DEBUG:boto:auth_path=/yrjefh-52/obj92 2024-04-16T20:07:40.171 DEBUG:boto:Method: GET 2024-04-16T20:07:40.171 DEBUG:boto:Path: /yrjefh-52/obj92 2024-04-16T20:07:40.171 DEBUG:boto:Data: 2024-04-16T20:07:40.171 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.171 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.171 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.172 DEBUG:boto:Params: {} 2024-04-16T20:07:40.172 DEBUG:boto:Token: None 2024-04-16T20:07:40.172 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj92 2024-04-16T20:07:40.172 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NaEHa8s+h9f06mM/optY0GJUq4U= 2024-04-16T20:07:40.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NaEHa8s+h9f06mM/optY0GJUq4U=', 'Content-Length': '0'} 2024-04-16T20:07:40.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:01:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0caf845d7ba06b5-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.175 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.175 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.175 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.176 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.176 DEBUG:boto:Method: GET 2024-04-16T20:07:40.176 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.176 DEBUG:boto:Data: 2024-04-16T20:07:40.176 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.176 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.176 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.176 DEBUG:boto:Params: {} 2024-04-16T20:07:40.177 DEBUG:boto:Token: None 2024-04-16T20:07:40.177 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.177 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b288055c3612a68e-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.222 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.223 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.223 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.223 DEBUG:boto:path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.223 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.223 DEBUG:boto:Method: GET 2024-04-16T20:07:40.223 DEBUG:boto:Path: /yrjefh-52/?prefix=obj92 2024-04-16T20:07:40.224 DEBUG:boto:Data: 2024-04-16T20:07:40.224 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.224 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.224 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.224 DEBUG:boto:Params: {} 2024-04-16T20:07:40.224 DEBUG:boto:Token: None 2024-04-16T20:07:40.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.225 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f45470ebd6a231-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.228 DEBUG:boto:b'yrjefh-52obj921000falseobj922024-04-16T20:01:27.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:40.229 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.229 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.229 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.229 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.230 DEBUG:boto:Method: GET 2024-04-16T20:07:40.230 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.230 DEBUG:boto:Data: 2024-04-16T20:07:40.230 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.230 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.230 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.230 DEBUG:boto:Params: {} 2024-04-16T20:07:40.230 DEBUG:boto:Token: None 2024-04-16T20:07:40.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE= 2024-04-16T20:07:40.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE=', 'Content-Length': '0'} 2024-04-16T20:07:40.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004959659df2d88b8-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.274 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.275 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.275 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.275 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.275 DEBUG:boto:Method: GET 2024-04-16T20:07:40.275 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.275 DEBUG:boto:Data: 2024-04-16T20:07:40.275 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.276 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.276 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.276 DEBUG:boto:Params: {} 2024-04-16T20:07:40.276 DEBUG:boto:Token: None 2024-04-16T20:07:40.276 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE= 2024-04-16T20:07:40.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE=', 'Content-Length': '0'} 2024-04-16T20:07:40.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012457ac2d5e799c2-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.279 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.279 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.279 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.280 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.280 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.280 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.280 DEBUG:boto:Data: 2024-04-16T20:07:40.280 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.280 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.280 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.280 DEBUG:boto:Params: {} 2024-04-16T20:07:40.281 DEBUG:boto:Token: None 2024-04-16T20:07:40.281 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QrT9qkIbIGc7g01N1eUOsvu2M5o= 2024-04-16T20:07:40.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QrT9qkIbIGc7g01N1eUOsvu2M5o=', 'Content-Length': '0'} 2024-04-16T20:07:40.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d29df30bb51ab43-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.284 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.284 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.284 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.284 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.284 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.285 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.285 DEBUG:boto:Data: 2024-04-16T20:07:40.285 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.285 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.285 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.285 DEBUG:boto:Params: {} 2024-04-16T20:07:40.285 DEBUG:boto:Token: None 2024-04-16T20:07:40.285 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.286 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QrT9qkIbIGc7g01N1eUOsvu2M5o= 2024-04-16T20:07:40.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QrT9qkIbIGc7g01N1eUOsvu2M5o=', 'Content-Length': '0'} 2024-04-16T20:07:40.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b23563aed5988ba-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:40.289 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.289 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.289 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.289 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.289 DEBUG:boto:Method: GET 2024-04-16T20:07:40.289 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.289 DEBUG:boto:Data: 2024-04-16T20:07:40.290 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.290 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.290 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.290 DEBUG:boto:Params: {} 2024-04-16T20:07:40.290 DEBUG:boto:Token: None 2024-04-16T20:07:40.290 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE= 2024-04-16T20:07:40.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE=', 'Content-Length': '0'} 2024-04-16T20:07:40.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3465abeb2c86b3b-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.334 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.335 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.335 DEBUG:boto:path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.335 DEBUG:boto:auth_path=/yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.335 DEBUG:boto:Method: GET 2024-04-16T20:07:40.335 DEBUG:boto:Path: /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.335 DEBUG:boto:Data: 2024-04-16T20:07:40.335 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.336 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.336 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.336 DEBUG:boto:Params: {} 2024-04-16T20:07:40.336 DEBUG:boto:Token: None 2024-04-16T20:07:40.336 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93?versionId=null 2024-04-16T20:07:40.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE= 2024-04-16T20:07:40.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qfyG2t1SjZO9+eo6ekOCSRYoAE=', 'Content-Length': '0'} 2024-04-16T20:07:40.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004464272e85016bb0-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.339 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.339 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.339 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.339 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.340 DEBUG:boto:Method: GET 2024-04-16T20:07:40.340 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.340 DEBUG:boto:Data: 2024-04-16T20:07:40.340 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.340 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.340 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.340 DEBUG:boto:Params: {} 2024-04-16T20:07:40.340 DEBUG:boto:Token: None 2024-04-16T20:07:40.341 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.341 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e43b5edf5ab945b3-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.343 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.344 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.344 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.344 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.344 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.344 DEBUG:boto:Method: GET 2024-04-16T20:07:40.345 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.345 DEBUG:boto:Data: 2024-04-16T20:07:40.345 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.345 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.345 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.345 DEBUG:boto:Params: {} 2024-04-16T20:07:40.345 DEBUG:boto:Token: None 2024-04-16T20:07:40.345 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64200cfd5812234-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.390 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.391 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.391 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.391 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.391 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:40.391 DEBUG:boto:Data: 2024-04-16T20:07:40.392 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.392 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.392 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.392 DEBUG:boto:Params: {} 2024-04-16T20:07:40.392 DEBUG:boto:Token: None 2024-04-16T20:07:40.392 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93 2024-04-16T20:07:40.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CErjFhSJRLrZ2euEQ2YHB+jT3i4= 2024-04-16T20:07:40.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CErjFhSJRLrZ2euEQ2YHB+jT3i4=', 'Content-Length': '0'} 2024-04-16T20:07:40.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e88a462bfa1e40-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.395 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.395 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.395 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.395 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:40.395 DEBUG:boto:Data: 2024-04-16T20:07:40.395 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.396 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.396 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.396 DEBUG:boto:Params: {} 2024-04-16T20:07:40.396 DEBUG:boto:Token: None 2024-04-16T20:07:40.396 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93 2024-04-16T20:07:40.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CErjFhSJRLrZ2euEQ2YHB+jT3i4= 2024-04-16T20:07:40.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CErjFhSJRLrZ2euEQ2YHB+jT3i4=', 'Content-Length': '0'} 2024-04-16T20:07:40.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000783d61a3aab6e8fc-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:07:40.399 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.400 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.400 DEBUG:boto:Method: GET 2024-04-16T20:07:40.400 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:40.400 DEBUG:boto:Data: 2024-04-16T20:07:40.400 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.400 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.400 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.400 DEBUG:boto:Params: {} 2024-04-16T20:07:40.400 DEBUG:boto:Token: None 2024-04-16T20:07:40.401 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93 2024-04-16T20:07:40.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vdZEr8KXwRR6XIn/gl0jYj5YTlI= 2024-04-16T20:07:40.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vdZEr8KXwRR6XIn/gl0jYj5YTlI=', 'Content-Length': '0'} 2024-04-16T20:07:40.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d80f14682d2bc4-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.446 DEBUG:boto:path=/yrjefh-52/obj93 2024-04-16T20:07:40.447 DEBUG:boto:auth_path=/yrjefh-52/obj93 2024-04-16T20:07:40.447 DEBUG:boto:Method: GET 2024-04-16T20:07:40.447 DEBUG:boto:Path: /yrjefh-52/obj93 2024-04-16T20:07:40.447 DEBUG:boto:Data: 2024-04-16T20:07:40.447 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.447 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.447 DEBUG:boto:Params: {} 2024-04-16T20:07:40.447 DEBUG:boto:Token: None 2024-04-16T20:07:40.448 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj93 2024-04-16T20:07:40.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vdZEr8KXwRR6XIn/gl0jYj5YTlI= 2024-04-16T20:07:40.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vdZEr8KXwRR6XIn/gl0jYj5YTlI=', 'Content-Length': '0'} 2024-04-16T20:07:40.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:04:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173f75afb1916e5f-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.451 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.451 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.451 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.451 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.451 DEBUG:boto:Method: GET 2024-04-16T20:07:40.451 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.451 DEBUG:boto:Data: 2024-04-16T20:07:40.451 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.451 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.452 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.452 DEBUG:boto:Params: {} 2024-04-16T20:07:40.452 DEBUG:boto:Token: None 2024-04-16T20:07:40.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009afdafbb9e446ade-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.498 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.499 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.499 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.499 DEBUG:boto:path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.499 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.499 DEBUG:boto:Method: GET 2024-04-16T20:07:40.499 DEBUG:boto:Path: /yrjefh-52/?prefix=obj93 2024-04-16T20:07:40.500 DEBUG:boto:Data: 2024-04-16T20:07:40.500 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.500 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.500 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.500 DEBUG:boto:Params: {} 2024-04-16T20:07:40.500 DEBUG:boto:Token: None 2024-04-16T20:07:40.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b459a1cdffdbdc8-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.504 DEBUG:boto:b'yrjefh-52obj931000falseobj932024-04-16T20:04:43.717Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:40.505 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.505 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.505 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.505 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.505 DEBUG:boto:Method: GET 2024-04-16T20:07:40.505 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.505 DEBUG:boto:Data: 2024-04-16T20:07:40.505 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.506 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.506 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.506 DEBUG:boto:Params: {} 2024-04-16T20:07:40.506 DEBUG:boto:Token: None 2024-04-16T20:07:40.506 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.506 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U= 2024-04-16T20:07:40.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U=', 'Content-Length': '0'} 2024-04-16T20:07:40.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1964df29f6fa88-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.550 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.550 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.551 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.551 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.551 DEBUG:boto:Method: GET 2024-04-16T20:07:40.551 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.551 DEBUG:boto:Data: 2024-04-16T20:07:40.551 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.551 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.551 DEBUG:boto:Params: {} 2024-04-16T20:07:40.552 DEBUG:boto:Token: None 2024-04-16T20:07:40.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U= 2024-04-16T20:07:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U=', 'Content-Length': '0'} 2024-04-16T20:07:40.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008458b23fa4dc5826-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.554 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.554 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.555 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.555 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.555 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.555 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.555 DEBUG:boto:Data: 2024-04-16T20:07:40.555 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.555 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.555 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.555 DEBUG:boto:Params: {} 2024-04-16T20:07:40.555 DEBUG:boto:Token: None 2024-04-16T20:07:40.556 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyQOZKfPY5GEgX3dq+I59l0x054= 2024-04-16T20:07:40.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyQOZKfPY5GEgX3dq+I59l0x054=', 'Content-Length': '0'} 2024-04-16T20:07:40.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096f427b292e73e1-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.558 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.558 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.558 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.558 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.559 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.559 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.559 DEBUG:boto:Data: 2024-04-16T20:07:40.559 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.559 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.559 DEBUG:boto:Params: {} 2024-04-16T20:07:40.559 DEBUG:boto:Token: None 2024-04-16T20:07:40.559 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.559 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyQOZKfPY5GEgX3dq+I59l0x054= 2024-04-16T20:07:40.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyQOZKfPY5GEgX3dq+I59l0x054=', 'Content-Length': '0'} 2024-04-16T20:07:40.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5384fdc332d983-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:40.562 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.562 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.562 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.562 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.562 DEBUG:boto:Method: GET 2024-04-16T20:07:40.562 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.562 DEBUG:boto:Data: 2024-04-16T20:07:40.562 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.563 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.563 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.563 DEBUG:boto:Params: {} 2024-04-16T20:07:40.563 DEBUG:boto:Token: None 2024-04-16T20:07:40.563 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U= 2024-04-16T20:07:40.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U=', 'Content-Length': '0'} 2024-04-16T20:07:40.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd99f5a6b7389902-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.606 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.606 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.607 DEBUG:boto:path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.607 DEBUG:boto:auth_path=/yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.607 DEBUG:boto:Method: GET 2024-04-16T20:07:40.607 DEBUG:boto:Path: /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.607 DEBUG:boto:Data: 2024-04-16T20:07:40.607 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.607 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.607 DEBUG:boto:Params: {} 2024-04-16T20:07:40.607 DEBUG:boto:Token: None 2024-04-16T20:07:40.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95?versionId=null 2024-04-16T20:07:40.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U= 2024-04-16T20:07:40.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wQIC5P99Ts6NBnElFuG96VfSQ5U=', 'Content-Length': '0'} 2024-04-16T20:07:40.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032137d63f2cd2191-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.610 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.610 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.610 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.610 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.610 DEBUG:boto:Method: GET 2024-04-16T20:07:40.610 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.610 DEBUG:boto:Data: 2024-04-16T20:07:40.611 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.611 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.611 DEBUG:boto:Params: {} 2024-04-16T20:07:40.611 DEBUG:boto:Token: None 2024-04-16T20:07:40.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb657079b3e410bd-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.654 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.655 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.655 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.655 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.655 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.655 DEBUG:boto:Method: GET 2024-04-16T20:07:40.655 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.655 DEBUG:boto:Data: 2024-04-16T20:07:40.655 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.656 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.656 DEBUG:boto:Params: {} 2024-04-16T20:07:40.656 DEBUG:boto:Token: None 2024-04-16T20:07:40.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9a1048f0dfc6b7-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.660 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.660 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.660 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.660 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.660 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:40.661 DEBUG:boto:Data: 2024-04-16T20:07:40.661 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.661 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.661 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.661 DEBUG:boto:Params: {} 2024-04-16T20:07:40.661 DEBUG:boto:Token: None 2024-04-16T20:07:40.661 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95 2024-04-16T20:07:40.661 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UQp5JHQQG+ynRfU19z59c+IlBqY= 2024-04-16T20:07:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UQp5JHQQG+ynRfU19z59c+IlBqY=', 'Content-Length': '0'} 2024-04-16T20:07:40.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066f6b9688433c390-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.664 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.664 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.664 DEBUG:boto:Method: HEAD 2024-04-16T20:07:40.664 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:40.664 DEBUG:boto:Data: 2024-04-16T20:07:40.664 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.664 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.665 DEBUG:boto:Params: {} 2024-04-16T20:07:40.665 DEBUG:boto:Token: None 2024-04-16T20:07:40.665 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95 2024-04-16T20:07:40.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UQp5JHQQG+ynRfU19z59c+IlBqY= 2024-04-16T20:07:40.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UQp5JHQQG+ynRfU19z59c+IlBqY=', 'Content-Length': '0'} 2024-04-16T20:07:40.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5015f398622b9f9-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:07:40.667 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.667 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.667 DEBUG:boto:Method: GET 2024-04-16T20:07:40.668 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:40.668 DEBUG:boto:Data: 2024-04-16T20:07:40.668 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.668 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.668 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.668 DEBUG:boto:Params: {} 2024-04-16T20:07:40.668 DEBUG:boto:Token: None 2024-04-16T20:07:40.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95 2024-04-16T20:07:40.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XBlEF6aqNWzrfSxJ4P8xQSJZb5Y= 2024-04-16T20:07:40.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XBlEF6aqNWzrfSxJ4P8xQSJZb5Y=', 'Content-Length': '0'} 2024-04-16T20:07:40.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-52:b482a2a9-ef36-488b-9666-ada656453328.4812.21'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:07:31 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000730e0112ec9ac6c0-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.714 DEBUG:boto:path=/yrjefh-52/obj95 2024-04-16T20:07:40.714 DEBUG:boto:auth_path=/yrjefh-52/obj95 2024-04-16T20:07:40.715 DEBUG:boto:Method: GET 2024-04-16T20:07:40.715 DEBUG:boto:Path: /yrjefh-52/obj95 2024-04-16T20:07:40.715 DEBUG:boto:Data: 2024-04-16T20:07:40.715 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.715 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.715 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.715 DEBUG:boto:Params: {} 2024-04-16T20:07:40.715 DEBUG:boto:Token: None 2024-04-16T20:07:40.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/obj95 2024-04-16T20:07:40.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XBlEF6aqNWzrfSxJ4P8xQSJZb5Y= 2024-04-16T20:07:40.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XBlEF6aqNWzrfSxJ4P8xQSJZb5Y=', 'Content-Length': '0'} 2024-04-16T20:07:40.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:00:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d5b0b7bf3342261-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.718 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.718 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.718 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.719 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.719 DEBUG:boto:Method: GET 2024-04-16T20:07:40.719 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.719 DEBUG:boto:Data: 2024-04-16T20:07:40.719 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.719 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:07:40.719 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.719 DEBUG:boto:Params: {} 2024-04-16T20:07:40.719 DEBUG:boto:Token: None 2024-04-16T20:07:40.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84e8006f2376143-00661eda8c-5958-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.766 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.767 DEBUG:boto:path=/yrjefh-52/ 2024-04-16T20:07:40.767 DEBUG:boto:auth_path=/yrjefh-52/ 2024-04-16T20:07:40.767 DEBUG:boto:path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.767 DEBUG:boto:auth_path=/yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.767 DEBUG:boto:Method: GET 2024-04-16T20:07:40.767 DEBUG:boto:Path: /yrjefh-52/?prefix=obj95 2024-04-16T20:07:40.767 DEBUG:boto:Data: 2024-04-16T20:07:40.767 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.768 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.768 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.768 DEBUG:boto:Params: {} 2024-04-16T20:07:40.768 DEBUG:boto:Token: None 2024-04-16T20:07:40.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-52/ 2024-04-16T20:07:40.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc= 2024-04-16T20:07:40.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LEMEDhl5JGOM/pYv4MnhBfkNnsc=', 'Content-Length': '0'} 2024-04-16T20:07:40.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c36b7ff6804b64-00661eda8c-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.772 DEBUG:boto:b'yrjefh-52obj951000falseobj952024-04-16T20:00:29.069Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:07:40.773 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-52 zones={test-zone2, test-zone1} 2024-04-16T20:07:40.773 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2024-04-16T20:07:40.779 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:07:40.779 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:07:40.779 DEBUG:boto:Method: PUT 2024-04-16T20:07:40.779 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:07:40.780 DEBUG:boto:Data: 2024-04-16T20:07:40.780 DEBUG:boto:Headers: {} 2024-04-16T20:07:40.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:07:40.780 DEBUG:boto:Port: 8000 2024-04-16T20:07:40.780 DEBUG:boto:Params: {} 2024-04-16T20:07:40.780 DEBUG:boto:Token: None 2024-04-16T20:07:40.780 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:07:40 GMT /yrjefh-53/ 2024-04-16T20:07:40.780 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/HJNSNGb1AKi6JhhhljTKYCFG2Q= 2024-04-16T20:07:40.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:07:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/HJNSNGb1AKi6JhhhljTKYCFG2Q=', 'Content-Length': '0'} 2024-04-16T20:07:40.798 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d64d81079b83c40-00661eda8c-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:07:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:07:40.798 DEBUG:rgw_multi.tests:created bucket=yrjefh-53 2024-04-16T20:07:40.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:40.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:40.799 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:07:40.932 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713298060.804277_974.1","last_update":"2024-04-16T20:07:40.804277Z"}] 2024-04-16T20:07:40.932 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:07:40.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:07:40.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:40.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:40.933 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:41.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:41.050+0000 7fe02f12c800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:41.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:41.052+0000 7fe02f12c800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:41.067 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:41Z"} 2024-04-16T20:07:41.067 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:41Z"} 2024-04-16T20:07:41.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:07:41.068 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:07:41.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:07:41.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:07:46.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:46.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:46.069 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:46.124 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:46.124 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:46.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:46.186+0000 7fd98753b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:46.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:46.187+0000 7fd98753b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:46.203 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:46Z"} 2024-04-16T20:07:46.203 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:46Z"} 2024-04-16T20:07:46.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:07:46.203 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:07:46.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:07:46.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:07:51.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:51.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:51.205 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:51.260 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:51.260 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:51.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:51.320+0000 7f7a2e813800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:51.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:51.322+0000 7f7a2e813800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:51.339 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:51Z"} 2024-04-16T20:07:51.339 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:51Z"} 2024-04-16T20:07:51.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:07:51.340 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:07:51.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:07:51.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:07:56.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:56.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:07:56.341 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:07:56.397 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:07:56.397 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:07:56.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:56.513+0000 7f2dc313b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:56.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:07:56.515+0000 7f2dc313b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:07:56.532 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:56Z"} 2024-04-16T20:07:56.532 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:07:56Z"} 2024-04-16T20:07:56.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:07:56.533 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:07:56.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:07:56.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:01.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:01.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:01.534 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:01.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:01.652+0000 7ff58c89d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:01.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:01.654+0000 7ff58c89d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:01.669 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:01Z"} 2024-04-16T20:08:01.669 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:01Z"} 2024-04-16T20:08:01.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:01.669 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:01.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:01.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:06.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:06.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:06.671 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:06.729 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:06.730 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:06.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:06.792+0000 7fb57d044800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:06.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:06.794+0000 7fb57d044800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:06.809 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:06Z"} 2024-04-16T20:08:06.809 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:06Z"} 2024-04-16T20:08:06.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:06.810 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:06.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:06.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:11.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:11.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:11.812 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:11.869 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:11.869 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:11.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:11.931+0000 7fafb7e0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:11.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:11.932+0000 7fafb7e0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:11.947 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:11Z"} 2024-04-16T20:08:11.947 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:11Z"} 2024-04-16T20:08:11.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:11.948 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:11.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:11.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:16.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:16.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:16.949 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:17.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:17.068+0000 7f9e1a34b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:17.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:17.069+0000 7f9e1a34b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:17.086 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:17Z"} 2024-04-16T20:08:17.086 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:17Z"} 2024-04-16T20:08:17.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:17.087 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:17.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:17.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:22.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:22.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:22.089 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:22.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:22.202+0000 7efdc4d21800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:22.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:22.203+0000 7efdc4d21800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:22.217 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:22Z"} 2024-04-16T20:08:22.218 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:22Z"} 2024-04-16T20:08:22.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:22.218 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:22.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:22.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:27.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:27.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:27.220 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:27.275 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:27.275 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:27.338+0000 7fa43bf1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:27.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:27.339+0000 7fa43bf1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:27.354 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:27Z"} 2024-04-16T20:08:27.354 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:27Z"} 2024-04-16T20:08:27.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:27.355 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:27.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:27.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:32.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:32.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:32.357 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:32.412 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:32.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:32.473+0000 7fc71e52a800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:32.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:32.474+0000 7fc71e52a800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:32.490 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:32Z"} 2024-04-16T20:08:32.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:32Z"} 2024-04-16T20:08:32.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:32.491 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:32.491 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:32.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:37.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:37.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:37.492 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:37.610+0000 7f3fad91e800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:37.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:37.611+0000 7f3fad91e800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:37.626 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:37Z"} 2024-04-16T20:08:37.626 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:37Z"} 2024-04-16T20:08:37.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:37.627 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:37.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:37.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:42.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:42.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:42.628 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:42.684 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:42.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:42.745+0000 7f95d1f24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:42.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:42.747+0000 7f95d1f24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:42.763 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:42Z"} 2024-04-16T20:08:42.763 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:42Z"} 2024-04-16T20:08:42.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:42.764 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:42.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:42.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:47.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:47.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:47.765 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:47.823 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:47.823 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:47.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:47.888+0000 7fd8d5a24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:47.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:47.889+0000 7fd8d5a24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:47.905 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:47Z"} 2024-04-16T20:08:47.905 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:47Z"} 2024-04-16T20:08:47.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:47.905 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:47.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:47.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:52.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:52.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:52.907 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:52.962 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:52.962 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:53.024+0000 7f2328fa2800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:53.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:53.026+0000 7f2328fa2800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:53.041 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:53Z"} 2024-04-16T20:08:53.042 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:53Z"} 2024-04-16T20:08:53.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:53.042 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:53.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:53.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:08:58.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:58.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:08:58.044 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:08:58.098 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:08:58.098 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:08:58.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:58.161+0000 7f928713b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:58.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:08:58.163+0000 7f928713b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:08:58.178 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:58Z"} 2024-04-16T20:08:58.179 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:08:58Z"} 2024-04-16T20:08:58.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:08:58.179 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:08:58.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:08:58.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:03.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:03.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:03.181 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:03.235 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:03.235 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:03.297+0000 7f7935b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:03.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:03.299+0000 7f7935b2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:03.314 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:03Z"} 2024-04-16T20:09:03.314 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:03Z"} 2024-04-16T20:09:03.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:09:03.314 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:03.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:09:03.314 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:08.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:08.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:08.316 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:08.372 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:08.372 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:08.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:08.432+0000 7f2c6df28800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:08.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:08.434+0000 7f2c6df28800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:08.452 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:08Z"} 2024-04-16T20:09:08.452 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:08Z"} 2024-04-16T20:09:08.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:09:08.453 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:08.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:09:08.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:13.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:13.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:13.454 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:13.573+0000 7f2b9d414800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:13.574+0000 7f2b9d414800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:13.589 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:13Z"} 2024-04-16T20:09:13.589 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:13Z"} 2024-04-16T20:09:13.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:09:13.590 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:13.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:09:13.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:18.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:18.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:18.592 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:18.647 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:18.648 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:18.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:18.711+0000 7f867aa15800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:18.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:18.712+0000 7f867aa15800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:18.727 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:18Z"} 2024-04-16T20:09:18.727 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:18Z"} 2024-04-16T20:09:18.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:09:18.728 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:18.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:09:18.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:23.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:23.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:23.730 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:23.786 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:23.786 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:23.848+0000 7f1a42c0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:23.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:23.849+0000 7f1a42c0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:23.865 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:23Z"} 2024-04-16T20:09:23.866 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297960.705794_926.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.705794Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:23Z"} 2024-04-16T20:09:23.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297960.705794_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.705794Z', 'realm_epoch': 2}}] 2024-04-16T20:09:23.866 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:23.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297960.705794_926.1'} 2024-04-16T20:09:23.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713297960.705794_926.1 2024-04-16T20:09:28.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:28.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:28.869 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:28.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:28.987+0000 7fc8a774b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:28.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:28.988+0000 7fc8a774b800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:29.004 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298060.795124_971.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:40.795124Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:28Z"} 2024-04-16T20:09:29.004 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298060.795124_971.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:40.795124Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:28Z"} 2024-04-16T20:09:29.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298060.795124_971.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:07:40.795124Z', 'realm_epoch': 2}}] 2024-04-16T20:09:29.005 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:29.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.795124_971.1'} 2024-04-16T20:09:29.005 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298060.804277_974.1 target=1_1713298060.795124_971.1 2024-04-16T20:09:34.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:34.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:09:34.006 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:09:34.062 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:34.062 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:34.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:34.124+0000 7fbca8733800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:34.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:34.126+0000 7fbca8733800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:09:34.141 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298060.804277_974.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:40.804277Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:34Z"} 2024-04-16T20:09:34.141 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298060.804277_974.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:40.804277Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:09:34Z"} 2024-04-16T20:09:34.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298060.804277_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:07:40.804277Z', 'realm_epoch': 2}}] 2024-04-16T20:09:34.142 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:34.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298060.804277_974.1'} 2024-04-16T20:09:34.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:09:34.143 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:09:34.143 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:09:34.144 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:09:34.152+0000 7fede3f59640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 180015) UID: 0 2024-04-16T20:09:34.144 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:09:34.152+0000 7fede9c13880 -1 shutting down 2024-04-16T20:09:35.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:09:40.245 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:09:40.246 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:09:40.246 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:09:40.246 DEBUG:boto:Method: HEAD 2024-04-16T20:09:40.246 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:09:40.246 DEBUG:boto:Data: 2024-04-16T20:09:40.246 DEBUG:boto:Headers: {} 2024-04-16T20:09:40.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.247 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.247 DEBUG:boto:Params: {} 2024-04-16T20:09:40.247 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:09:40.248 DEBUG:boto:Token: None 2024-04-16T20:09:40.248 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/ 2024-04-16T20:09:40.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918= 2024-04-16T20:09:40.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918=', 'Content-Length': '0'} 2024-04-16T20:09:40.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d5242db31ff1cae-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.254 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:09:40.254 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:09:40.254 DEBUG:boto:Method: PUT 2024-04-16T20:09:40.254 DEBUG:boto:Path: /yrjefh-53/obj11 2024-04-16T20:09:40.254 DEBUG:boto:Data: 2024-04-16T20:09:40.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:09:40.254 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.254 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.255 DEBUG:boto:Params: {} 2024-04-16T20:09:40.255 DEBUG:boto:Token: None 2024-04-16T20:09:40.255 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/obj11 2024-04-16T20:09:40.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GuwmXVe6S8h4iT+SCWvqXSvgivg= 2024-04-16T20:09:40.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GuwmXVe6S8h4iT+SCWvqXSvgivg='} 2024-04-16T20:09:40.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bb375f1173c7370-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.263 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:09:40.263 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:09:40.263 DEBUG:boto:Method: HEAD 2024-04-16T20:09:40.263 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:09:40.263 DEBUG:boto:Data: 2024-04-16T20:09:40.263 DEBUG:boto:Headers: {} 2024-04-16T20:09:40.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.263 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.264 DEBUG:boto:Params: {} 2024-04-16T20:09:40.264 DEBUG:boto:Token: None 2024-04-16T20:09:40.264 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/ 2024-04-16T20:09:40.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918= 2024-04-16T20:09:40.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918=', 'Content-Length': '0'} 2024-04-16T20:09:40.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc7f0f7201a5a215-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.267 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:09:40.267 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:09:40.267 DEBUG:boto:Method: PUT 2024-04-16T20:09:40.267 DEBUG:boto:Path: /yrjefh-53/obj22 2024-04-16T20:09:40.267 DEBUG:boto:Data: 2024-04-16T20:09:40.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:09:40.267 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.267 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.267 DEBUG:boto:Params: {} 2024-04-16T20:09:40.267 DEBUG:boto:Token: None 2024-04-16T20:09:40.267 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/obj22 2024-04-16T20:09:40.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NyFFMXocZspkc1+hToHK9a0hwp0= 2024-04-16T20:09:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NyFFMXocZspkc1+hToHK9a0hwp0='} 2024-04-16T20:09:40.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c54de56fe5125a2-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.273 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:09:40.273 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:09:40.273 DEBUG:boto:Method: HEAD 2024-04-16T20:09:40.273 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:09:40.273 DEBUG:boto:Data: 2024-04-16T20:09:40.273 DEBUG:boto:Headers: {} 2024-04-16T20:09:40.273 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.274 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.274 DEBUG:boto:Params: {} 2024-04-16T20:09:40.274 DEBUG:boto:Token: None 2024-04-16T20:09:40.274 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/ 2024-04-16T20:09:40.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918= 2024-04-16T20:09:40.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918=', 'Content-Length': '0'} 2024-04-16T20:09:40.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc1fa44a923aacd6-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.276 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:09:40.276 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:09:40.276 DEBUG:boto:Method: PUT 2024-04-16T20:09:40.276 DEBUG:boto:Path: /yrjefh-53/obj33 2024-04-16T20:09:40.276 DEBUG:boto:Data: 2024-04-16T20:09:40.276 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:09:40.277 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.277 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.277 DEBUG:boto:Params: {} 2024-04-16T20:09:40.277 DEBUG:boto:Token: None 2024-04-16T20:09:40.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/obj33 2024-04-16T20:09:40.277 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yKHjq1Ngp4xLktO/LJRvecdc9AE= 2024-04-16T20:09:40.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yKHjq1Ngp4xLktO/LJRvecdc9AE='} 2024-04-16T20:09:40.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf074bc213e390ec-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.318 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:09:40.319 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:09:40.319 DEBUG:boto:Method: HEAD 2024-04-16T20:09:40.319 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:09:40.319 DEBUG:boto:Data: 2024-04-16T20:09:40.319 DEBUG:boto:Headers: {} 2024-04-16T20:09:40.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.319 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.319 DEBUG:boto:Params: {} 2024-04-16T20:09:40.319 DEBUG:boto:Token: None 2024-04-16T20:09:40.319 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/ 2024-04-16T20:09:40.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918= 2024-04-16T20:09:40.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918=', 'Content-Length': '0'} 2024-04-16T20:09:40.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f92b06b10ae118d-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.322 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:09:40.322 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:09:40.322 DEBUG:boto:Method: PUT 2024-04-16T20:09:40.322 DEBUG:boto:Path: /yrjefh-53/obj44 2024-04-16T20:09:40.322 DEBUG:boto:Data: 2024-04-16T20:09:40.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:09:40.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.323 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.323 DEBUG:boto:Params: {} 2024-04-16T20:09:40.323 DEBUG:boto:Token: None 2024-04-16T20:09:40.323 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/obj44 2024-04-16T20:09:40.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+wE7ufd4STqcP+r/nSZiKC44vP8= 2024-04-16T20:09:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+wE7ufd4STqcP+r/nSZiKC44vP8='} 2024-04-16T20:09:40.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005cdacde6cea1b6b-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.330 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:09:40.330 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:09:40.331 DEBUG:boto:Method: HEAD 2024-04-16T20:09:40.331 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:09:40.331 DEBUG:boto:Data: 2024-04-16T20:09:40.331 DEBUG:boto:Headers: {} 2024-04-16T20:09:40.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.331 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.331 DEBUG:boto:Params: {} 2024-04-16T20:09:40.331 DEBUG:boto:Token: None 2024-04-16T20:09:40.331 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/ 2024-04-16T20:09:40.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918= 2024-04-16T20:09:40.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CvcYAJEcjyB8XAKd0EJHR6a1918=', 'Content-Length': '0'} 2024-04-16T20:09:40.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003847cfcb8417f059-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.334 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:09:40.334 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:09:40.334 DEBUG:boto:Method: PUT 2024-04-16T20:09:40.334 DEBUG:boto:Path: /yrjefh-53/obj55 2024-04-16T20:09:40.334 DEBUG:boto:Data: 2024-04-16T20:09:40.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:09:40.334 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:09:40.334 DEBUG:boto:Port: 8000 2024-04-16T20:09:40.335 DEBUG:boto:Params: {} 2024-04-16T20:09:40.335 DEBUG:boto:Token: None 2024-04-16T20:09:40.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:09:40 GMT /yrjefh-53/obj55 2024-04-16T20:09:40.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:39zSbsE+ag3S4ENW09mmx88HlJw= 2024-04-16T20:09:40.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:09:40 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:39zSbsE+ag3S4ENW09mmx88HlJw='} 2024-04-16T20:09:40.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009973178740518c9c-00661edb04-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:09:40.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:09:40.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:09:40.341 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:09:40.396 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:40.397 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:40.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:40.461+0000 7f0c14339800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T20:09:40.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:40.462+0000 7f0c14339800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T20:09:40.487 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:09:40.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:09:40.489 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:09:40.489 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:09:40.507 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:09:40.507 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:09:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:09:40.509 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:09:40.509 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:09:40.674 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:09:40.682+0000 7efd7002d880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:09:41.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:09:41.510 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:09:41.531 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:09:41.531 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:09:41.531 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:09:41.532 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:09:41.532 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-53 2024-04-16T20:09:41.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-53', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:09:41.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-53', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:09:41.532 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-53 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:09:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:09:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:09:41.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:41.688+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:09:46.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:46.689+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:09:51.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:51.690+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:09:56.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:09:56.691+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:01.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:01.691+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:06.692+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:11.693+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:16.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:16.694+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:21.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:21.694+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:26.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:26.694+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:31.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:31.696+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:36.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:36.696+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:41.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:41.697+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:46.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:46.698+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:51.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:51.699+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:10:56.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:10:56.700+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:01.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:01.701+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:06.700+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:11.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:11.702+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:13.980 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:11:13.988+0000 7efc55551640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-16T20:11:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:16.702+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:21.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:21.704+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:26.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:26.705+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:31.706+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:36.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:36.707+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:41.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:41.707+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:46.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:46.707+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:51.709+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:11:56.531 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T20:11:56.540+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T20:11:56.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:11:56.710+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:01.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:01.711+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:06.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:06.712+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:11.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:11.713+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:16.714+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:21.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:21.715+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:26.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:26.714+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:31.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:31.717+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:36.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:36.717+0000 7f479e32d800 1 waiting to reach incremental sync.. 2024-04-16T20:12:41.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:41.719+0000 7f479e32d800 1 bucket sync caught up with source: 2024-04-16T20:12:41.712 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.72.6, 00000000002.118.6, 00000000002.79.6] 2024-04-16T20:12:41.712 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000001.72.6, 00000000002.118.6, 00000000002.79.6] 2024-04-16T20:12:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:41.719+0000 7f479e32d800 0 bucket checkpoint complete 2024-04-16T20:12:41.740 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-53 2024-04-16T20:12:41.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-53', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:12:41.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-53', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:12:41.741 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-53 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:12:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:41.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:12:41.876+0000 7f014d033800 1 bucket sync caught up with source: 2024-04-16T20:12:41.868 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.101.6, 00000000002.327.6, 00000000002.102.6] 2024-04-16T20:12:41.869 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.101.6, 00000000002.327.6, 00000000002.102.6] 2024-04-16T20:12:41.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:12:41.876+0000 7f014d033800 0 bucket checkpoint complete 2024-04-16T20:12:41.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-53 zones={test-zone2, test-zone1} 2024-04-16T20:12:41.891 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.892 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.892 DEBUG:boto:Method: HEAD 2024-04-16T20:12:41.892 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:12:41.892 DEBUG:boto:Data: 2024-04-16T20:12:41.892 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.893 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.893 DEBUG:boto:Params: {} 2024-04-16T20:12:41.893 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:12:41.893 DEBUG:boto:Token: None 2024-04-16T20:12:41.894 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/ 2024-04-16T20:12:41.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mTEbii8JWlshHt5v9IDAheaf/Dg= 2024-04-16T20:12:41.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mTEbii8JWlshHt5v9IDAheaf/Dg=', 'Content-Length': '0'} 2024-04-16T20:12:41.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026c49cf35126624d-00661edbb9-6142-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.902 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.902 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.902 DEBUG:boto:Method: HEAD 2024-04-16T20:12:41.902 DEBUG:boto:Path: /yrjefh-53/ 2024-04-16T20:12:41.902 DEBUG:boto:Data: 2024-04-16T20:12:41.903 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.903 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.903 DEBUG:boto:Params: {} 2024-04-16T20:12:41.903 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:12:41.903 DEBUG:boto:Token: None 2024-04-16T20:12:41.904 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/ 2024-04-16T20:12:41.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mTEbii8JWlshHt5v9IDAheaf/Dg= 2024-04-16T20:12:41.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mTEbii8JWlshHt5v9IDAheaf/Dg=', 'Content-Length': '0'} 2024-04-16T20:12:41.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019998aeb7879067b-00661edbb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:12:41.908 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.909 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.909 DEBUG:boto:path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.909 DEBUG:boto:auth_path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.909 DEBUG:boto:Method: GET 2024-04-16T20:12:41.909 DEBUG:boto:Path: /yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.909 DEBUG:boto:Data: 2024-04-16T20:12:41.909 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.909 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.909 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.909 DEBUG:boto:Params: {} 2024-04-16T20:12:41.910 DEBUG:boto:Token: None 2024-04-16T20:12:41.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/?versions 2024-04-16T20:12:41.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM= 2024-04-16T20:12:41.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM=', 'Content-Length': '0'} 2024-04-16T20:12:41.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efaf71af8fe77338-00661edbb9-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.913 DEBUG:boto:b'yrjefh-53999falseobj11nulltrue2024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:41.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:12:41.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:12:41.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:12:41.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:12:41.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:12:41.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:12:41.915 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.915 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.915 DEBUG:boto:path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.916 DEBUG:boto:auth_path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.916 DEBUG:boto:Method: GET 2024-04-16T20:12:41.916 DEBUG:boto:Path: /yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.916 DEBUG:boto:Data: 2024-04-16T20:12:41.916 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.916 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.916 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.916 DEBUG:boto:Params: {} 2024-04-16T20:12:41.916 DEBUG:boto:Token: None 2024-04-16T20:12:41.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/?versions 2024-04-16T20:12:41.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM= 2024-04-16T20:12:41.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM=', 'Content-Length': '0'} 2024-04-16T20:12:41.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afaca0261ddc7e5e-00661edbb9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.919 DEBUG:boto:b'yrjefh-53999falseobj11nulltrue2024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:12:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:12:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:12:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:12:41.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:12:41.920 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.921 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.921 DEBUG:boto:path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.921 DEBUG:boto:auth_path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.921 DEBUG:boto:Method: GET 2024-04-16T20:12:41.921 DEBUG:boto:Path: /yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.921 DEBUG:boto:Data: 2024-04-16T20:12:41.921 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.921 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.921 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.921 DEBUG:boto:Params: {} 2024-04-16T20:12:41.922 DEBUG:boto:Token: None 2024-04-16T20:12:41.922 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/?versions 2024-04-16T20:12:41.922 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM= 2024-04-16T20:12:41.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM=', 'Content-Length': '0'} 2024-04-16T20:12:41.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe12df4d19bd0ba7-00661edbb9-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.925 DEBUG:boto:b'yrjefh-53999falseobj11nulltrue2024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:41.925 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.925 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.925 DEBUG:boto:path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.926 DEBUG:boto:auth_path=/yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.926 DEBUG:boto:Method: GET 2024-04-16T20:12:41.926 DEBUG:boto:Path: /yrjefh-53/?versions&max-keys=999 2024-04-16T20:12:41.926 DEBUG:boto:Data: 2024-04-16T20:12:41.926 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.926 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.926 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.926 DEBUG:boto:Params: {} 2024-04-16T20:12:41.926 DEBUG:boto:Token: None 2024-04-16T20:12:41.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/?versions 2024-04-16T20:12:41.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM= 2024-04-16T20:12:41.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxsCWp2nt6Q4Oh6Ru5MHTv2RGpM=', 'Content-Length': '0'} 2024-04-16T20:12:41.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d37a01da931453-00661edbb9-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.930 DEBUG:boto:b'yrjefh-53999falseobj11nulltrue2024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:41.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:12:41.931 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.931 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.931 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.931 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.931 DEBUG:boto:Method: GET 2024-04-16T20:12:41.931 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.931 DEBUG:boto:Data: 2024-04-16T20:12:41.931 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.932 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.932 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.932 DEBUG:boto:Params: {} 2024-04-16T20:12:41.932 DEBUG:boto:Token: None 2024-04-16T20:12:41.932 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg= 2024-04-16T20:12:41.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg=', 'Content-Length': '0'} 2024-04-16T20:12:41.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:40 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a480645b41e6f1-00661edbb9-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.936 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.936 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.936 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.936 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.936 DEBUG:boto:Method: GET 2024-04-16T20:12:41.936 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.936 DEBUG:boto:Data: 2024-04-16T20:12:41.936 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.936 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.936 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.937 DEBUG:boto:Params: {} 2024-04-16T20:12:41.937 DEBUG:boto:Token: None 2024-04-16T20:12:41.937 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.937 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg= 2024-04-16T20:12:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg=', 'Content-Length': '0'} 2024-04-16T20:12:41.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b95adfa915f552-00661edbb9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.939 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.939 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.939 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.940 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.940 DEBUG:boto:Method: HEAD 2024-04-16T20:12:41.940 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.940 DEBUG:boto:Data: 2024-04-16T20:12:41.940 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.940 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.940 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.940 DEBUG:boto:Params: {} 2024-04-16T20:12:41.940 DEBUG:boto:Token: None 2024-04-16T20:12:41.941 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.941 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y/eo/ccBJmbFTgyssjo49RcxKgc= 2024-04-16T20:12:41.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y/eo/ccBJmbFTgyssjo49RcxKgc=', 'Content-Length': '0'} 2024-04-16T20:12:41.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:40 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce8ff9466230863f-00661edbb9-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.943 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.943 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.943 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.944 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.944 DEBUG:boto:Method: HEAD 2024-04-16T20:12:41.944 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.944 DEBUG:boto:Data: 2024-04-16T20:12:41.944 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.944 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.944 DEBUG:boto:Params: {} 2024-04-16T20:12:41.944 DEBUG:boto:Token: None 2024-04-16T20:12:41.945 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.945 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y/eo/ccBJmbFTgyssjo49RcxKgc= 2024-04-16T20:12:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y/eo/ccBJmbFTgyssjo49RcxKgc=', 'Content-Length': '0'} 2024-04-16T20:12:41.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005baf50c73704d4cf-00661edbb9-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:12:41.947 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.947 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.947 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.947 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.947 DEBUG:boto:Method: GET 2024-04-16T20:12:41.947 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.947 DEBUG:boto:Data: 2024-04-16T20:12:41.947 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.948 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.948 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.948 DEBUG:boto:Params: {} 2024-04-16T20:12:41.948 DEBUG:boto:Token: None 2024-04-16T20:12:41.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg= 2024-04-16T20:12:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg=', 'Content-Length': '0'} 2024-04-16T20:12:41.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:40 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c967821430bba7-00661edbb9-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.990 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:41.991 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:41.991 DEBUG:boto:path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.991 DEBUG:boto:auth_path=/yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.991 DEBUG:boto:Method: GET 2024-04-16T20:12:41.991 DEBUG:boto:Path: /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.991 DEBUG:boto:Data: 2024-04-16T20:12:41.991 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.991 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:41.991 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.991 DEBUG:boto:Params: {} 2024-04-16T20:12:41.992 DEBUG:boto:Token: None 2024-04-16T20:12:41.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/obj11?versionId=null 2024-04-16T20:12:41.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg= 2024-04-16T20:12:41.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8i2Md0OrCEOyYrM6/anYA8uE3Xg=', 'Content-Length': '0'} 2024-04-16T20:12:41.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000466b90dd663ac90-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.994 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.994 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.994 DEBUG:boto:path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:41.994 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:41.994 DEBUG:boto:Method: GET 2024-04-16T20:12:41.995 DEBUG:boto:Path: /yrjefh-53/?prefix=obj11 2024-04-16T20:12:41.995 DEBUG:boto:Data: 2024-04-16T20:12:41.995 DEBUG:boto:Headers: {} 2024-04-16T20:12:41.995 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:41.995 DEBUG:boto:Port: 8000 2024-04-16T20:12:41.995 DEBUG:boto:Params: {} 2024-04-16T20:12:41.995 DEBUG:boto:Token: None 2024-04-16T20:12:41.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:41 GMT /yrjefh-53/ 2024-04-16T20:12:41.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:uhFiGJl/ZFF8d67/9ZF4lh5jev0= 2024-04-16T20:12:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uhFiGJl/ZFF8d67/9ZF4lh5jev0=', 'Content-Length': '0'} 2024-04-16T20:12:41.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dfb020edc9e0ecd-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:41.998 DEBUG:boto:b'yrjefh-53obj111000falseobj112024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:41.999 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:41.999 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:41.999 DEBUG:boto:path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:41.999 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:41.999 DEBUG:boto:Method: GET 2024-04-16T20:12:41.999 DEBUG:boto:Path: /yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.000 DEBUG:boto:Data: 2024-04-16T20:12:42.000 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.000 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.000 DEBUG:boto:Params: {} 2024-04-16T20:12:42.000 DEBUG:boto:Token: None 2024-04-16T20:12:42.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d580714a71341d3-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.003 DEBUG:boto:b'yrjefh-53obj111000falseobj112024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.003 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:42.003 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:42.003 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.003 DEBUG:boto:Path: /yrjefh-53/obj11 2024-04-16T20:12:42.003 DEBUG:boto:Data: 2024-04-16T20:12:42.003 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.004 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.004 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.004 DEBUG:boto:Params: {} 2024-04-16T20:12:42.004 DEBUG:boto:Token: None 2024-04-16T20:12:42.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj11 2024-04-16T20:12:42.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WindY1MjOgE8iP6DXVnR4v++42s= 2024-04-16T20:12:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WindY1MjOgE8iP6DXVnR4v++42s=', 'Content-Length': '0'} 2024-04-16T20:12:42.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:40 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006524aa2c7fafa65a-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.007 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:42.007 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:42.007 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.007 DEBUG:boto:Path: /yrjefh-53/obj11 2024-04-16T20:12:42.007 DEBUG:boto:Data: 2024-04-16T20:12:42.007 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.007 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.007 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.007 DEBUG:boto:Params: {} 2024-04-16T20:12:42.007 DEBUG:boto:Token: None 2024-04-16T20:12:42.008 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj11 2024-04-16T20:12:42.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WindY1MjOgE8iP6DXVnR4v++42s= 2024-04-16T20:12:42.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WindY1MjOgE8iP6DXVnR4v++42s=', 'Content-Length': '0'} 2024-04-16T20:12:42.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041367a4cba55c919-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:12:42.010 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:42.010 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:42.010 DEBUG:boto:Method: GET 2024-04-16T20:12:42.010 DEBUG:boto:Path: /yrjefh-53/obj11 2024-04-16T20:12:42.010 DEBUG:boto:Data: 2024-04-16T20:12:42.010 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.010 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.010 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.010 DEBUG:boto:Params: {} 2024-04-16T20:12:42.011 DEBUG:boto:Token: None 2024-04-16T20:12:42.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj11 2024-04-16T20:12:42.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c8E5Jf6EJZxxJiJk/3kLqPY6Vf8= 2024-04-16T20:12:42.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c8E5Jf6EJZxxJiJk/3kLqPY6Vf8=', 'Content-Length': '0'} 2024-04-16T20:12:42.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:40 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b8936978cbf0c7-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.054 DEBUG:boto:path=/yrjefh-53/obj11 2024-04-16T20:12:42.055 DEBUG:boto:auth_path=/yrjefh-53/obj11 2024-04-16T20:12:42.055 DEBUG:boto:Method: GET 2024-04-16T20:12:42.055 DEBUG:boto:Path: /yrjefh-53/obj11 2024-04-16T20:12:42.055 DEBUG:boto:Data: 2024-04-16T20:12:42.055 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.055 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.055 DEBUG:boto:Params: {} 2024-04-16T20:12:42.057 DEBUG:boto:Token: None 2024-04-16T20:12:42.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj11 2024-04-16T20:12:42.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c8E5Jf6EJZxxJiJk/3kLqPY6Vf8= 2024-04-16T20:12:42.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c8E5Jf6EJZxxJiJk/3kLqPY6Vf8=', 'Content-Length': '0'} 2024-04-16T20:12:42.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec02ae6618085fb4-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.059 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.060 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.060 DEBUG:boto:path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.060 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.060 DEBUG:boto:Method: GET 2024-04-16T20:12:42.060 DEBUG:boto:Path: /yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.060 DEBUG:boto:Data: 2024-04-16T20:12:42.060 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.060 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.060 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.060 DEBUG:boto:Params: {} 2024-04-16T20:12:42.061 DEBUG:boto:Token: None 2024-04-16T20:12:42.061 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.061 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622e03aa6d2eba16-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.106 DEBUG:boto:b'yrjefh-53obj111000falseobj112024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.107 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.107 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.107 DEBUG:boto:path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.107 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.107 DEBUG:boto:Method: GET 2024-04-16T20:12:42.108 DEBUG:boto:Path: /yrjefh-53/?prefix=obj11 2024-04-16T20:12:42.108 DEBUG:boto:Data: 2024-04-16T20:12:42.108 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.108 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.108 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.108 DEBUG:boto:Params: {} 2024-04-16T20:12:42.108 DEBUG:boto:Token: None 2024-04-16T20:12:42.108 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca49a9882b1189c-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.111 DEBUG:boto:b'yrjefh-53obj111000falseobj112024-04-16T20:09:40.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:12:42.112 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.112 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.112 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.112 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.112 DEBUG:boto:Method: GET 2024-04-16T20:12:42.112 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.113 DEBUG:boto:Data: 2024-04-16T20:12:42.113 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.113 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.113 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.113 DEBUG:boto:Params: {} 2024-04-16T20:12:42.113 DEBUG:boto:Token: None 2024-04-16T20:12:42.113 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8= 2024-04-16T20:12:42.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8=', 'Content-Length': '0'} 2024-04-16T20:12:42.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3a3a5bdeb4b178-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.158 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.184 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.184 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.184 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.184 DEBUG:boto:Method: GET 2024-04-16T20:12:42.184 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.184 DEBUG:boto:Data: 2024-04-16T20:12:42.185 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.185 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.185 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.185 DEBUG:boto:Params: {} 2024-04-16T20:12:42.185 DEBUG:boto:Token: None 2024-04-16T20:12:42.185 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.185 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8= 2024-04-16T20:12:42.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8=', 'Content-Length': '0'} 2024-04-16T20:12:42.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f1b5709bb75e3a-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.188 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.188 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.188 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.188 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.189 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.189 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.189 DEBUG:boto:Data: 2024-04-16T20:12:42.189 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.189 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.189 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.189 DEBUG:boto:Params: {} 2024-04-16T20:12:42.189 DEBUG:boto:Token: None 2024-04-16T20:12:42.189 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DQ5pzXHDHQ7/GHd9jQT5fdfANGs= 2024-04-16T20:12:42.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DQ5pzXHDHQ7/GHd9jQT5fdfANGs=', 'Content-Length': '0'} 2024-04-16T20:12:42.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aab3842534ab4ea-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.192 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.192 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.192 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.192 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.192 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.193 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.193 DEBUG:boto:Data: 2024-04-16T20:12:42.193 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.193 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.193 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.193 DEBUG:boto:Params: {} 2024-04-16T20:12:42.193 DEBUG:boto:Token: None 2024-04-16T20:12:42.193 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DQ5pzXHDHQ7/GHd9jQT5fdfANGs= 2024-04-16T20:12:42.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DQ5pzXHDHQ7/GHd9jQT5fdfANGs=', 'Content-Length': '0'} 2024-04-16T20:12:42.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba69ad7bf68bc3b-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:12:42.196 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.196 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.196 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.196 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.196 DEBUG:boto:Method: GET 2024-04-16T20:12:42.196 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.196 DEBUG:boto:Data: 2024-04-16T20:12:42.197 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.197 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.197 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.197 DEBUG:boto:Params: {} 2024-04-16T20:12:42.197 DEBUG:boto:Token: None 2024-04-16T20:12:42.197 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8= 2024-04-16T20:12:42.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8=', 'Content-Length': '0'} 2024-04-16T20:12:42.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f02fcc77e68c62-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.242 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.243 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.243 DEBUG:boto:path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.243 DEBUG:boto:auth_path=/yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.243 DEBUG:boto:Method: GET 2024-04-16T20:12:42.243 DEBUG:boto:Path: /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.243 DEBUG:boto:Data: 2024-04-16T20:12:42.243 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.243 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.243 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.243 DEBUG:boto:Params: {} 2024-04-16T20:12:42.244 DEBUG:boto:Token: None 2024-04-16T20:12:42.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22?versionId=null 2024-04-16T20:12:42.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8= 2024-04-16T20:12:42.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AQeXs9hPkYfHiBnNVfXNg1Cn4q8=', 'Content-Length': '0'} 2024-04-16T20:12:42.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c74d8a81ca4daada-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.246 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.247 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.247 DEBUG:boto:path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.247 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.247 DEBUG:boto:Method: GET 2024-04-16T20:12:42.247 DEBUG:boto:Path: /yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.247 DEBUG:boto:Data: 2024-04-16T20:12:42.247 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.247 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.247 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.247 DEBUG:boto:Params: {} 2024-04-16T20:12:42.248 DEBUG:boto:Token: None 2024-04-16T20:12:42.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe8242a439cf8a2-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.294 DEBUG:boto:b'yrjefh-53obj221000falseobj222024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.295 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.295 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.295 DEBUG:boto:path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.295 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.295 DEBUG:boto:Method: GET 2024-04-16T20:12:42.296 DEBUG:boto:Path: /yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.296 DEBUG:boto:Data: 2024-04-16T20:12:42.296 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.296 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.296 DEBUG:boto:Params: {} 2024-04-16T20:12:42.296 DEBUG:boto:Token: None 2024-04-16T20:12:42.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.297 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cbc30e63b6c85aa-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.299 DEBUG:boto:b'yrjefh-53obj221000falseobj222024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.300 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.300 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.300 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.300 DEBUG:boto:Path: /yrjefh-53/obj22 2024-04-16T20:12:42.300 DEBUG:boto:Data: 2024-04-16T20:12:42.300 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.300 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.301 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.301 DEBUG:boto:Params: {} 2024-04-16T20:12:42.301 DEBUG:boto:Token: None 2024-04-16T20:12:42.301 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22 2024-04-16T20:12:42.301 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LMmwrjdqBkkqa8JqXOmwWXH8VVU= 2024-04-16T20:12:42.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LMmwrjdqBkkqa8JqXOmwWXH8VVU=', 'Content-Length': '0'} 2024-04-16T20:12:42.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002115e8564639d017-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.304 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.304 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.304 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.304 DEBUG:boto:Path: /yrjefh-53/obj22 2024-04-16T20:12:42.305 DEBUG:boto:Data: 2024-04-16T20:12:42.305 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.305 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.305 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.305 DEBUG:boto:Params: {} 2024-04-16T20:12:42.305 DEBUG:boto:Token: None 2024-04-16T20:12:42.305 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22 2024-04-16T20:12:42.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LMmwrjdqBkkqa8JqXOmwWXH8VVU= 2024-04-16T20:12:42.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LMmwrjdqBkkqa8JqXOmwWXH8VVU=', 'Content-Length': '0'} 2024-04-16T20:12:42.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b394e0711807d5f-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:12:42.308 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.308 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.308 DEBUG:boto:Method: GET 2024-04-16T20:12:42.308 DEBUG:boto:Path: /yrjefh-53/obj22 2024-04-16T20:12:42.308 DEBUG:boto:Data: 2024-04-16T20:12:42.308 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.309 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.309 DEBUG:boto:Params: {} 2024-04-16T20:12:42.309 DEBUG:boto:Token: None 2024-04-16T20:12:42.309 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22 2024-04-16T20:12:42.309 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aM0fekPH4y6F44Woh+Eih155R7o= 2024-04-16T20:12:42.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aM0fekPH4y6F44Woh+Eih155R7o=', 'Content-Length': '0'} 2024-04-16T20:12:42.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcae4c6e7b82164a-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.354 DEBUG:boto:path=/yrjefh-53/obj22 2024-04-16T20:12:42.355 DEBUG:boto:auth_path=/yrjefh-53/obj22 2024-04-16T20:12:42.355 DEBUG:boto:Method: GET 2024-04-16T20:12:42.355 DEBUG:boto:Path: /yrjefh-53/obj22 2024-04-16T20:12:42.355 DEBUG:boto:Data: 2024-04-16T20:12:42.355 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.355 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.355 DEBUG:boto:Params: {} 2024-04-16T20:12:42.356 DEBUG:boto:Token: None 2024-04-16T20:12:42.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj22 2024-04-16T20:12:42.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aM0fekPH4y6F44Woh+Eih155R7o= 2024-04-16T20:12:42.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aM0fekPH4y6F44Woh+Eih155R7o=', 'Content-Length': '0'} 2024-04-16T20:12:42.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2583338afcab79-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.358 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.359 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.359 DEBUG:boto:path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.359 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.359 DEBUG:boto:Method: GET 2024-04-16T20:12:42.359 DEBUG:boto:Path: /yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.359 DEBUG:boto:Data: 2024-04-16T20:12:42.359 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.360 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.360 DEBUG:boto:Params: {} 2024-04-16T20:12:42.360 DEBUG:boto:Token: None 2024-04-16T20:12:42.360 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031bb1aa69eb99def-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.406 DEBUG:boto:b'yrjefh-53obj221000falseobj222024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.407 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.407 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.407 DEBUG:boto:path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.407 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.407 DEBUG:boto:Method: GET 2024-04-16T20:12:42.408 DEBUG:boto:Path: /yrjefh-53/?prefix=obj22 2024-04-16T20:12:42.408 DEBUG:boto:Data: 2024-04-16T20:12:42.408 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.408 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.408 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.408 DEBUG:boto:Params: {} 2024-04-16T20:12:42.408 DEBUG:boto:Token: None 2024-04-16T20:12:42.408 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984fb8e3d2aa85ae-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.411 DEBUG:boto:b'yrjefh-53obj221000falseobj222024-04-16T20:09:40.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:12:42.412 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.412 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.412 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.412 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.413 DEBUG:boto:Method: GET 2024-04-16T20:12:42.413 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.413 DEBUG:boto:Data: 2024-04-16T20:12:42.413 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.413 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.413 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.413 DEBUG:boto:Params: {} 2024-04-16T20:12:42.413 DEBUG:boto:Token: None 2024-04-16T20:12:42.414 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.414 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps= 2024-04-16T20:12:42.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps=', 'Content-Length': '0'} 2024-04-16T20:12:42.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe81c567f98b1f41-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.459 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.459 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.459 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.459 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.459 DEBUG:boto:Method: GET 2024-04-16T20:12:42.459 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.459 DEBUG:boto:Data: 2024-04-16T20:12:42.459 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.460 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.460 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.460 DEBUG:boto:Params: {} 2024-04-16T20:12:42.460 DEBUG:boto:Token: None 2024-04-16T20:12:42.460 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps= 2024-04-16T20:12:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps=', 'Content-Length': '0'} 2024-04-16T20:12:42.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f7d69a0a78458e-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.466 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.466 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.466 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.466 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.466 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.467 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.467 DEBUG:boto:Data: 2024-04-16T20:12:42.467 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.467 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.467 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.467 DEBUG:boto:Params: {} 2024-04-16T20:12:42.467 DEBUG:boto:Token: None 2024-04-16T20:12:42.468 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OG6zLYM7Qwt9ZgO27MyNrBMasD8= 2024-04-16T20:12:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OG6zLYM7Qwt9ZgO27MyNrBMasD8=', 'Content-Length': '0'} 2024-04-16T20:12:42.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb9c0ad314befbf-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.471 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.471 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.471 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.471 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.471 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.471 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.471 DEBUG:boto:Data: 2024-04-16T20:12:42.472 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.472 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.472 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.472 DEBUG:boto:Params: {} 2024-04-16T20:12:42.472 DEBUG:boto:Token: None 2024-04-16T20:12:42.472 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OG6zLYM7Qwt9ZgO27MyNrBMasD8= 2024-04-16T20:12:42.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OG6zLYM7Qwt9ZgO27MyNrBMasD8=', 'Content-Length': '0'} 2024-04-16T20:12:42.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4023e990ed8d82-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:12:42.475 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.475 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.475 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.476 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.476 DEBUG:boto:Method: GET 2024-04-16T20:12:42.476 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.476 DEBUG:boto:Data: 2024-04-16T20:12:42.476 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.476 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.476 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.476 DEBUG:boto:Params: {} 2024-04-16T20:12:42.476 DEBUG:boto:Token: None 2024-04-16T20:12:42.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps= 2024-04-16T20:12:42.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps=', 'Content-Length': '0'} 2024-04-16T20:12:42.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca68a14a0ce1f1d-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.522 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.523 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.523 DEBUG:boto:path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.523 DEBUG:boto:auth_path=/yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.523 DEBUG:boto:Method: GET 2024-04-16T20:12:42.523 DEBUG:boto:Path: /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.523 DEBUG:boto:Data: 2024-04-16T20:12:42.523 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.524 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.524 DEBUG:boto:Params: {} 2024-04-16T20:12:42.524 DEBUG:boto:Token: None 2024-04-16T20:12:42.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33?versionId=null 2024-04-16T20:12:42.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps= 2024-04-16T20:12:42.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tNBbaX3ie9aWdfydQhwt8Budyps=', 'Content-Length': '0'} 2024-04-16T20:12:42.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f0e1de29363732-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.527 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.527 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.527 DEBUG:boto:path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.527 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.527 DEBUG:boto:Method: GET 2024-04-16T20:12:42.528 DEBUG:boto:Path: /yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.528 DEBUG:boto:Data: 2024-04-16T20:12:42.528 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.528 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.528 DEBUG:boto:Params: {} 2024-04-16T20:12:42.528 DEBUG:boto:Token: None 2024-04-16T20:12:42.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971449930d85e400-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.574 DEBUG:boto:b'yrjefh-53obj331000falseobj332024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.575 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.575 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.575 DEBUG:boto:path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.575 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.576 DEBUG:boto:Method: GET 2024-04-16T20:12:42.576 DEBUG:boto:Path: /yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.576 DEBUG:boto:Data: 2024-04-16T20:12:42.576 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.576 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.576 DEBUG:boto:Params: {} 2024-04-16T20:12:42.576 DEBUG:boto:Token: None 2024-04-16T20:12:42.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.577 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001765ac144a729513-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.580 DEBUG:boto:b'yrjefh-53obj331000falseobj332024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.580 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.580 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.580 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.580 DEBUG:boto:Path: /yrjefh-53/obj33 2024-04-16T20:12:42.581 DEBUG:boto:Data: 2024-04-16T20:12:42.581 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.581 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.581 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.581 DEBUG:boto:Params: {} 2024-04-16T20:12:42.581 DEBUG:boto:Token: None 2024-04-16T20:12:42.581 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33 2024-04-16T20:12:42.581 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j0oqC4eHziWvYNsVNNdlRD0kcnk= 2024-04-16T20:12:42.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j0oqC4eHziWvYNsVNNdlRD0kcnk=', 'Content-Length': '0'} 2024-04-16T20:12:42.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008574d9cb5f3c2a1c-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.584 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.585 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.585 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.585 DEBUG:boto:Path: /yrjefh-53/obj33 2024-04-16T20:12:42.585 DEBUG:boto:Data: 2024-04-16T20:12:42.585 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.585 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.585 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.585 DEBUG:boto:Params: {} 2024-04-16T20:12:42.586 DEBUG:boto:Token: None 2024-04-16T20:12:42.586 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33 2024-04-16T20:12:42.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:j0oqC4eHziWvYNsVNNdlRD0kcnk= 2024-04-16T20:12:42.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j0oqC4eHziWvYNsVNNdlRD0kcnk=', 'Content-Length': '0'} 2024-04-16T20:12:42.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595aba4ac18f9cee-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:12:42.589 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.589 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.589 DEBUG:boto:Method: GET 2024-04-16T20:12:42.589 DEBUG:boto:Path: /yrjefh-53/obj33 2024-04-16T20:12:42.590 DEBUG:boto:Data: 2024-04-16T20:12:42.590 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.590 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.590 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.590 DEBUG:boto:Params: {} 2024-04-16T20:12:42.590 DEBUG:boto:Token: None 2024-04-16T20:12:42.590 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33 2024-04-16T20:12:42.590 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KQFFeSui1GqSufDb9F0tmVHTiIY= 2024-04-16T20:12:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KQFFeSui1GqSufDb9F0tmVHTiIY=', 'Content-Length': '0'} 2024-04-16T20:12:42.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003975cb4ce315e97e-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.634 DEBUG:boto:path=/yrjefh-53/obj33 2024-04-16T20:12:42.635 DEBUG:boto:auth_path=/yrjefh-53/obj33 2024-04-16T20:12:42.635 DEBUG:boto:Method: GET 2024-04-16T20:12:42.635 DEBUG:boto:Path: /yrjefh-53/obj33 2024-04-16T20:12:42.635 DEBUG:boto:Data: 2024-04-16T20:12:42.635 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.635 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.635 DEBUG:boto:Params: {} 2024-04-16T20:12:42.636 DEBUG:boto:Token: None 2024-04-16T20:12:42.636 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj33 2024-04-16T20:12:42.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KQFFeSui1GqSufDb9F0tmVHTiIY= 2024-04-16T20:12:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KQFFeSui1GqSufDb9F0tmVHTiIY=', 'Content-Length': '0'} 2024-04-16T20:12:42.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279455f8d8dfaf39-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.639 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.639 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.639 DEBUG:boto:path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.639 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.639 DEBUG:boto:Method: GET 2024-04-16T20:12:42.640 DEBUG:boto:Path: /yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.640 DEBUG:boto:Data: 2024-04-16T20:12:42.640 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.640 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.640 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.640 DEBUG:boto:Params: {} 2024-04-16T20:12:42.640 DEBUG:boto:Token: None 2024-04-16T20:12:42.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.641 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c99fe9f91cf76c47-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.686 DEBUG:boto:b'yrjefh-53obj331000falseobj332024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.687 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.687 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.687 DEBUG:boto:path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.687 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.687 DEBUG:boto:Method: GET 2024-04-16T20:12:42.687 DEBUG:boto:Path: /yrjefh-53/?prefix=obj33 2024-04-16T20:12:42.688 DEBUG:boto:Data: 2024-04-16T20:12:42.688 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.688 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.688 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.688 DEBUG:boto:Params: {} 2024-04-16T20:12:42.688 DEBUG:boto:Token: None 2024-04-16T20:12:42.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1188a069fe700b-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.692 DEBUG:boto:b'yrjefh-53obj331000falseobj332024-04-16T20:09:40.287Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:12:42.692 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.692 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.692 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.693 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.693 DEBUG:boto:Method: GET 2024-04-16T20:12:42.693 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.693 DEBUG:boto:Data: 2024-04-16T20:12:42.693 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.693 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.693 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.693 DEBUG:boto:Params: {} 2024-04-16T20:12:42.694 DEBUG:boto:Token: None 2024-04-16T20:12:42.694 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.694 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA= 2024-04-16T20:12:42.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA=', 'Content-Length': '0'} 2024-04-16T20:12:42.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000edf91a43427564f-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.739 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.739 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.739 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.739 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.739 DEBUG:boto:Method: GET 2024-04-16T20:12:42.739 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.739 DEBUG:boto:Data: 2024-04-16T20:12:42.740 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.740 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.740 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.740 DEBUG:boto:Params: {} 2024-04-16T20:12:42.740 DEBUG:boto:Token: None 2024-04-16T20:12:42.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA= 2024-04-16T20:12:42.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA=', 'Content-Length': '0'} 2024-04-16T20:12:42.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082499ce454925716-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.743 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.743 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.743 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.743 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.743 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.744 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.744 DEBUG:boto:Data: 2024-04-16T20:12:42.744 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.744 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.744 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.744 DEBUG:boto:Params: {} 2024-04-16T20:12:42.744 DEBUG:boto:Token: None 2024-04-16T20:12:42.744 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.745 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6+sSYqtqQktAxF5RKOvESUfQRiw= 2024-04-16T20:12:42.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6+sSYqtqQktAxF5RKOvESUfQRiw=', 'Content-Length': '0'} 2024-04-16T20:12:42.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003761540c50256d37-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.747 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.747 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.748 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.748 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.748 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.748 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.748 DEBUG:boto:Data: 2024-04-16T20:12:42.748 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.748 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.748 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.748 DEBUG:boto:Params: {} 2024-04-16T20:12:42.749 DEBUG:boto:Token: None 2024-04-16T20:12:42.749 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.749 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6+sSYqtqQktAxF5RKOvESUfQRiw= 2024-04-16T20:12:42.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6+sSYqtqQktAxF5RKOvESUfQRiw=', 'Content-Length': '0'} 2024-04-16T20:12:42.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ad3c7dcc0abae3-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:12:42.752 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.752 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.752 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.752 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.752 DEBUG:boto:Method: GET 2024-04-16T20:12:42.752 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.752 DEBUG:boto:Data: 2024-04-16T20:12:42.753 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.753 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.753 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.753 DEBUG:boto:Params: {} 2024-04-16T20:12:42.753 DEBUG:boto:Token: None 2024-04-16T20:12:42.753 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.753 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA= 2024-04-16T20:12:42.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA=', 'Content-Length': '0'} 2024-04-16T20:12:42.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a79e23ae600e90b-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.798 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.798 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.799 DEBUG:boto:path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.799 DEBUG:boto:auth_path=/yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.799 DEBUG:boto:Method: GET 2024-04-16T20:12:42.799 DEBUG:boto:Path: /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.799 DEBUG:boto:Data: 2024-04-16T20:12:42.799 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.799 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.799 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.799 DEBUG:boto:Params: {} 2024-04-16T20:12:42.800 DEBUG:boto:Token: None 2024-04-16T20:12:42.800 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44?versionId=null 2024-04-16T20:12:42.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA= 2024-04-16T20:12:42.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyH48YMlhgPy+3T7zPc7brdcVIA=', 'Content-Length': '0'} 2024-04-16T20:12:42.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b5247236043020-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.803 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.803 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.803 DEBUG:boto:path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.803 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.803 DEBUG:boto:Method: GET 2024-04-16T20:12:42.803 DEBUG:boto:Path: /yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.804 DEBUG:boto:Data: 2024-04-16T20:12:42.804 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.804 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.804 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.804 DEBUG:boto:Params: {} 2024-04-16T20:12:42.804 DEBUG:boto:Token: None 2024-04-16T20:12:42.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ce8361e8ed2128-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.808 DEBUG:boto:b'yrjefh-53obj441000falseobj442024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.808 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.808 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.808 DEBUG:boto:path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.809 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.809 DEBUG:boto:Method: GET 2024-04-16T20:12:42.809 DEBUG:boto:Path: /yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.809 DEBUG:boto:Data: 2024-04-16T20:12:42.809 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.809 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.809 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.809 DEBUG:boto:Params: {} 2024-04-16T20:12:42.809 DEBUG:boto:Token: None 2024-04-16T20:12:42.810 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.810 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806ad42c497627fb-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.854 DEBUG:boto:b'yrjefh-53obj441000falseobj442024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.855 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.855 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.855 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.855 DEBUG:boto:Path: /yrjefh-53/obj44 2024-04-16T20:12:42.855 DEBUG:boto:Data: 2024-04-16T20:12:42.855 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.856 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.856 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.856 DEBUG:boto:Params: {} 2024-04-16T20:12:42.856 DEBUG:boto:Token: None 2024-04-16T20:12:42.856 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44 2024-04-16T20:12:42.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/USeCkFcYHHI/tvE8gxjZyujf0A= 2024-04-16T20:12:42.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/USeCkFcYHHI/tvE8gxjZyujf0A=', 'Content-Length': '0'} 2024-04-16T20:12:42.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49d148a36ede80d-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.859 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.859 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.859 DEBUG:boto:Method: HEAD 2024-04-16T20:12:42.859 DEBUG:boto:Path: /yrjefh-53/obj44 2024-04-16T20:12:42.859 DEBUG:boto:Data: 2024-04-16T20:12:42.859 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.860 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.860 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.860 DEBUG:boto:Params: {} 2024-04-16T20:12:42.860 DEBUG:boto:Token: None 2024-04-16T20:12:42.860 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44 2024-04-16T20:12:42.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/USeCkFcYHHI/tvE8gxjZyujf0A= 2024-04-16T20:12:42.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/USeCkFcYHHI/tvE8gxjZyujf0A=', 'Content-Length': '0'} 2024-04-16T20:12:42.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c07fa562f74cab-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:12:42.863 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.863 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.863 DEBUG:boto:Method: GET 2024-04-16T20:12:42.863 DEBUG:boto:Path: /yrjefh-53/obj44 2024-04-16T20:12:42.863 DEBUG:boto:Data: 2024-04-16T20:12:42.864 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.864 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.864 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.864 DEBUG:boto:Params: {} 2024-04-16T20:12:42.864 DEBUG:boto:Token: None 2024-04-16T20:12:42.864 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44 2024-04-16T20:12:42.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OlyaCzBBYapb6OX1cmQ3f+LkuGs= 2024-04-16T20:12:42.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OlyaCzBBYapb6OX1cmQ3f+LkuGs=', 'Content-Length': '0'} 2024-04-16T20:12:42.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313a68321e346a3e-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.910 DEBUG:boto:path=/yrjefh-53/obj44 2024-04-16T20:12:42.911 DEBUG:boto:auth_path=/yrjefh-53/obj44 2024-04-16T20:12:42.911 DEBUG:boto:Method: GET 2024-04-16T20:12:42.911 DEBUG:boto:Path: /yrjefh-53/obj44 2024-04-16T20:12:42.911 DEBUG:boto:Data: 2024-04-16T20:12:42.911 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.911 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.912 DEBUG:boto:Params: {} 2024-04-16T20:12:42.912 DEBUG:boto:Token: None 2024-04-16T20:12:42.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj44 2024-04-16T20:12:42.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OlyaCzBBYapb6OX1cmQ3f+LkuGs= 2024-04-16T20:12:42.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OlyaCzBBYapb6OX1cmQ3f+LkuGs=', 'Content-Length': '0'} 2024-04-16T20:12:42.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa13e4127779377f-00661edbba-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.915 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.915 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.915 DEBUG:boto:path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.915 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.915 DEBUG:boto:Method: GET 2024-04-16T20:12:42.915 DEBUG:boto:Path: /yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.915 DEBUG:boto:Data: 2024-04-16T20:12:42.915 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.916 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.916 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.916 DEBUG:boto:Params: {} 2024-04-16T20:12:42.916 DEBUG:boto:Token: None 2024-04-16T20:12:42.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b9b4f0ab4583a9e-00661edbba-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.962 DEBUG:boto:b'yrjefh-53obj441000falseobj442024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.963 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:42.963 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:42.963 DEBUG:boto:path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.963 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.963 DEBUG:boto:Method: GET 2024-04-16T20:12:42.963 DEBUG:boto:Path: /yrjefh-53/?prefix=obj44 2024-04-16T20:12:42.964 DEBUG:boto:Data: 2024-04-16T20:12:42.964 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:42.964 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.964 DEBUG:boto:Params: {} 2024-04-16T20:12:42.964 DEBUG:boto:Token: None 2024-04-16T20:12:42.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/ 2024-04-16T20:12:42.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU= 2024-04-16T20:12:42.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RDylyths4QYaP1iTkUDisz6PznU=', 'Content-Length': '0'} 2024-04-16T20:12:42.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462827fa5411cad2-00661edbba-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:42.968 DEBUG:boto:b'yrjefh-53obj441000falseobj442024-04-16T20:09:40.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:42.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:12:42.968 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:42.968 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:42.968 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:42.969 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:42.969 DEBUG:boto:Method: GET 2024-04-16T20:12:42.969 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:42.969 DEBUG:boto:Data: 2024-04-16T20:12:42.969 DEBUG:boto:Headers: {} 2024-04-16T20:12:42.969 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:42.969 DEBUG:boto:Port: 8000 2024-04-16T20:12:42.969 DEBUG:boto:Params: {} 2024-04-16T20:12:42.970 DEBUG:boto:Token: None 2024-04-16T20:12:42.970 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:42 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:42.970 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DCIj65NEfISD4+7S5W9BbZhFO24= 2024-04-16T20:12:42.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DCIj65NEfISD4+7S5W9BbZhFO24=', 'Content-Length': '0'} 2024-04-16T20:12:42.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7904413e2bdefc-00661edbba-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.018 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.019 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.019 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.019 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.019 DEBUG:boto:Method: GET 2024-04-16T20:12:43.019 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.019 DEBUG:boto:Data: 2024-04-16T20:12:43.019 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.020 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.020 DEBUG:boto:Params: {} 2024-04-16T20:12:43.020 DEBUG:boto:Token: None 2024-04-16T20:12:43.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU= 2024-04-16T20:12:43.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU=', 'Content-Length': '0'} 2024-04-16T20:12:43.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000347474f6c184ed8-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.023 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.023 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.024 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.024 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.024 DEBUG:boto:Method: HEAD 2024-04-16T20:12:43.024 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.024 DEBUG:boto:Data: 2024-04-16T20:12:43.024 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.024 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.024 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.025 DEBUG:boto:Params: {} 2024-04-16T20:12:43.025 DEBUG:boto:Token: None 2024-04-16T20:12:43.025 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.025 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oMeTFOm3jO8dXqDvDSexo55ydTA= 2024-04-16T20:12:43.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oMeTFOm3jO8dXqDvDSexo55ydTA=', 'Content-Length': '0'} 2024-04-16T20:12:43.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066fd2ec9d42254a0-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.028 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.028 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.028 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.028 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.028 DEBUG:boto:Method: HEAD 2024-04-16T20:12:43.028 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.029 DEBUG:boto:Data: 2024-04-16T20:12:43.029 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.029 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.029 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.029 DEBUG:boto:Params: {} 2024-04-16T20:12:43.029 DEBUG:boto:Token: None 2024-04-16T20:12:43.029 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.029 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oMeTFOm3jO8dXqDvDSexo55ydTA= 2024-04-16T20:12:43.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oMeTFOm3jO8dXqDvDSexo55ydTA=', 'Content-Length': '0'} 2024-04-16T20:12:43.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef6b5da526327e9-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:12:43.032 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.032 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.033 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.033 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.033 DEBUG:boto:Method: GET 2024-04-16T20:12:43.033 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.033 DEBUG:boto:Data: 2024-04-16T20:12:43.033 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.033 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.033 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.033 DEBUG:boto:Params: {} 2024-04-16T20:12:43.034 DEBUG:boto:Token: None 2024-04-16T20:12:43.034 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.034 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU= 2024-04-16T20:12:43.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU=', 'Content-Length': '0'} 2024-04-16T20:12:43.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b92b82794a5aaebf-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.078 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.079 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.079 DEBUG:boto:path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.079 DEBUG:boto:auth_path=/yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.079 DEBUG:boto:Method: GET 2024-04-16T20:12:43.079 DEBUG:boto:Path: /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.079 DEBUG:boto:Data: 2024-04-16T20:12:43.079 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.080 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.080 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.080 DEBUG:boto:Params: {} 2024-04-16T20:12:43.080 DEBUG:boto:Token: None 2024-04-16T20:12:43.080 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55?versionId=null 2024-04-16T20:12:43.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU= 2024-04-16T20:12:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lOqm/M6vVNawEchtYH1my3GqUdU=', 'Content-Length': '0'} 2024-04-16T20:12:43.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b565df3991396f-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.083 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:43.083 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:43.083 DEBUG:boto:path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.083 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.084 DEBUG:boto:Method: GET 2024-04-16T20:12:43.084 DEBUG:boto:Path: /yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.084 DEBUG:boto:Data: 2024-04-16T20:12:43.084 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.084 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.084 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.084 DEBUG:boto:Params: {} 2024-04-16T20:12:43.084 DEBUG:boto:Token: None 2024-04-16T20:12:43.085 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/ 2024-04-16T20:12:43.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo= 2024-04-16T20:12:43.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo=', 'Content-Length': '0'} 2024-04-16T20:12:43.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000899292eec590dd18-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.088 DEBUG:boto:b'yrjefh-53obj551000falseobj552024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:43.088 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:43.089 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:43.089 DEBUG:boto:path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.089 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.089 DEBUG:boto:Method: GET 2024-04-16T20:12:43.089 DEBUG:boto:Path: /yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.089 DEBUG:boto:Data: 2024-04-16T20:12:43.089 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.089 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.090 DEBUG:boto:Params: {} 2024-04-16T20:12:43.090 DEBUG:boto:Token: None 2024-04-16T20:12:43.090 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/ 2024-04-16T20:12:43.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo= 2024-04-16T20:12:43.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo=', 'Content-Length': '0'} 2024-04-16T20:12:43.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a18a0b685883fc-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.134 DEBUG:boto:b'yrjefh-53obj551000falseobj552024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:43.135 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.135 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.135 DEBUG:boto:Method: HEAD 2024-04-16T20:12:43.135 DEBUG:boto:Path: /yrjefh-53/obj55 2024-04-16T20:12:43.135 DEBUG:boto:Data: 2024-04-16T20:12:43.135 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.135 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.136 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.136 DEBUG:boto:Params: {} 2024-04-16T20:12:43.136 DEBUG:boto:Token: None 2024-04-16T20:12:43.136 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55 2024-04-16T20:12:43.136 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:54HuI2MjRuzIVR/Z9VW/CQLI/UA= 2024-04-16T20:12:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:54HuI2MjRuzIVR/Z9VW/CQLI/UA=', 'Content-Length': '0'} 2024-04-16T20:12:43.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370e242526965f1c-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.139 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.139 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.139 DEBUG:boto:Method: HEAD 2024-04-16T20:12:43.139 DEBUG:boto:Path: /yrjefh-53/obj55 2024-04-16T20:12:43.139 DEBUG:boto:Data: 2024-04-16T20:12:43.140 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.140 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.140 DEBUG:boto:Params: {} 2024-04-16T20:12:43.140 DEBUG:boto:Token: None 2024-04-16T20:12:43.140 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55 2024-04-16T20:12:43.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:54HuI2MjRuzIVR/Z9VW/CQLI/UA= 2024-04-16T20:12:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:54HuI2MjRuzIVR/Z9VW/CQLI/UA=', 'Content-Length': '0'} 2024-04-16T20:12:43.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008726de4ff501894c-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:12:43.143 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.143 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.143 DEBUG:boto:Method: GET 2024-04-16T20:12:43.144 DEBUG:boto:Path: /yrjefh-53/obj55 2024-04-16T20:12:43.144 DEBUG:boto:Data: 2024-04-16T20:12:43.144 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.144 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.144 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.144 DEBUG:boto:Params: {} 2024-04-16T20:12:43.144 DEBUG:boto:Token: None 2024-04-16T20:12:43.145 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55 2024-04-16T20:12:43.145 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qclco3MUKfwCny14lDMig6d9Q28= 2024-04-16T20:12:43.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qclco3MUKfwCny14lDMig6d9Q28=', 'Content-Length': '0'} 2024-04-16T20:12:43.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-53:b482a2a9-ef36-488b-9666-ada656453328.4812.22'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:11:14 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f806582170f26eba-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.190 DEBUG:boto:path=/yrjefh-53/obj55 2024-04-16T20:12:43.191 DEBUG:boto:auth_path=/yrjefh-53/obj55 2024-04-16T20:12:43.191 DEBUG:boto:Method: GET 2024-04-16T20:12:43.191 DEBUG:boto:Path: /yrjefh-53/obj55 2024-04-16T20:12:43.191 DEBUG:boto:Data: 2024-04-16T20:12:43.191 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.191 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.192 DEBUG:boto:Params: {} 2024-04-16T20:12:43.192 DEBUG:boto:Token: None 2024-04-16T20:12:43.192 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/obj55 2024-04-16T20:12:43.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qclco3MUKfwCny14lDMig6d9Q28= 2024-04-16T20:12:43.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qclco3MUKfwCny14lDMig6d9Q28=', 'Content-Length': '0'} 2024-04-16T20:12:43.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:09:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007974f311f434099-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.195 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:43.195 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:43.195 DEBUG:boto:path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.195 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.196 DEBUG:boto:Method: GET 2024-04-16T20:12:43.196 DEBUG:boto:Path: /yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.196 DEBUG:boto:Data: 2024-04-16T20:12:43.196 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.196 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:43.196 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.196 DEBUG:boto:Params: {} 2024-04-16T20:12:43.196 DEBUG:boto:Token: None 2024-04-16T20:12:43.197 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/ 2024-04-16T20:12:43.197 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo= 2024-04-16T20:12:43.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo=', 'Content-Length': '0'} 2024-04-16T20:12:43.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f31a111c1c4e1196-00661edbbb-6142-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.242 DEBUG:boto:b'yrjefh-53obj551000falseobj552024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:43.243 DEBUG:boto:path=/yrjefh-53/ 2024-04-16T20:12:43.243 DEBUG:boto:auth_path=/yrjefh-53/ 2024-04-16T20:12:43.243 DEBUG:boto:path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.243 DEBUG:boto:auth_path=/yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.243 DEBUG:boto:Method: GET 2024-04-16T20:12:43.243 DEBUG:boto:Path: /yrjefh-53/?prefix=obj55 2024-04-16T20:12:43.244 DEBUG:boto:Data: 2024-04-16T20:12:43.244 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.244 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.244 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.244 DEBUG:boto:Params: {} 2024-04-16T20:12:43.244 DEBUG:boto:Token: None 2024-04-16T20:12:43.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-53/ 2024-04-16T20:12:43.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo= 2024-04-16T20:12:43.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sKe2umMUjIBRtRb3glgv9rYu5bo=', 'Content-Length': '0'} 2024-04-16T20:12:43.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064a10a55fed8ae52-00661edbbb-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.247 DEBUG:boto:b'yrjefh-53obj551000falseobj552024-04-16T20:09:40.345Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:12:43.248 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-53 zones={test-zone2, test-zone1} 2024-04-16T20:12:43.248 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T20:12:43.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.249 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.303 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.303 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:43.390 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000028347","last_update":"2024-04-16T20:09:47.496972Z"},{"marker":"00000000000000000000:00000000000000021977","last_update":"2024-04-16T20:06:00.751107Z"},{"marker":"00000000000000000000:00000000000000032117","last_update":"2024-04-16T20:09:40.338040Z"},{"marker":"00000000000000000000:00000000000000029159","last_update":"2024-04-16T20:09:47.498879Z"}] 2024-04-16T20:12:43.390 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000028347', 1: '00000000000000000000:00000000000000021977', 2: '00000000000000000000:00000000000000032117', 3: '00000000000000000000:00000000000000029159'} 2024-04-16T20:12:43.390 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:12:43.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.391 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.448 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.448 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:43.527 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9333116933636540806},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000028347","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:09:47.496972Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000021977","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.751107Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000032117","next_step_marker":"","total_entries":44,"pos":22,"timestamp":"2024-04-16T20:09:40.338040Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000029159","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:09:47.498879Z"}}]},"full_sync":{"total":53,"complete":53},"current_time":"2024-04-16T20:12:43Z"} 2024-04-16T20:12:43.527 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9333116933636540806},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000028347","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:09:47.496972Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000021977","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:06:00.751107Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000032117","next_step_marker":"","total_entries":44,"pos":22,"timestamp":"2024-04-16T20:09:40.338040Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000029159","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:09:47.498879Z"}}]},"full_sync":{"total":53,"complete":53},"current_time":"2024-04-16T20:12:43Z"} 2024-04-16T20:12:43.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028347', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:09:47.496972Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021977', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:06:00.751107Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032117', 'next_step_marker': '', 'total_entries': 44, 'pos': 22, 'timestamp': '2024-04-16T20:09:40.338040Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000029159', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:09:47.498879Z'}}] 2024-04-16T20:12:43.528 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028347', 1: '00000000000000000000:00000000000000021977', 2: '00000000000000000000:00000000000000032117', 3: '00000000000000000000:00000000000000029159'} 2024-04-16T20:12:43.528 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028347', 1: '00000000000000000000:00000000000000021977', 2: '00000000000000000000:00000000000000032117', 3: '00000000000000000000:00000000000000029159'} 2024-04-16T20:12:43.528 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:12:43.528 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T20:12:43.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.528 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:43.655 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000013214","last_update":"2024-04-16T20:11:52.632306Z"},{"marker":"00000000000000000000:00000000000000007546","last_update":"2024-04-16T20:07:31.557627Z"},{"marker":"00000000000000000000:00000000000000014870","last_update":"2024-04-16T20:11:14.404214Z"},{"marker":"00000000000000000000:00000000000000015584","last_update":"2024-04-16T20:11:30.629998Z"}] 2024-04-16T20:12:43.656 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000013214', 1: '00000000000000000000:00000000000000007546', 2: '00000000000000000000:00000000000000014870', 3: '00000000000000000000:00000000000000015584'} 2024-04-16T20:12:43.656 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:12:43.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.656 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:43.785 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000013214","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:11:52.632306Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007546","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:31.557627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:11:14.404214Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015584","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:11:30.629998Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:12:43Z"} 2024-04-16T20:12:43.786 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000013214","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:11:52.632306Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007546","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:07:31.557627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014870","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:11:14.404214Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015584","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:11:30.629998Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:12:43Z"} 2024-04-16T20:12:43.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013214', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:11:52.632306Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007546', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:07:31.557627Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:11:14.404214Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015584', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T20:11:30.629998Z'}}] 2024-04-16T20:12:43.786 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013214', 1: '00000000000000000000:00000000000000007546', 2: '00000000000000000000:00000000000000014870', 3: '00000000000000000000:00000000000000015584'} 2024-04-16T20:12:43.786 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013214', 1: '00000000000000000000:00000000000000007546', 2: '00000000000000000000:00000000000000014870', 3: '00000000000000000000:00000000000000015584'} 2024-04-16T20:12:43.786 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:12:43.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2024-04-16T20:12:43.788 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:43.789 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:43.789 DEBUG:boto:Method: PUT 2024-04-16T20:12:43.789 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:43.789 DEBUG:boto:Data: 2024-04-16T20:12:43.789 DEBUG:boto:Headers: {} 2024-04-16T20:12:43.789 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:43.789 DEBUG:boto:Port: 8000 2024-04-16T20:12:43.789 DEBUG:boto:Params: {} 2024-04-16T20:12:43.789 DEBUG:boto:Token: None 2024-04-16T20:12:43.790 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:12:43 GMT /yrjefh-54/ 2024-04-16T20:12:43.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jhyFJmYu7cWjKnYEd7v7geZSRGs= 2024-04-16T20:12:43.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jhyFJmYu7cWjKnYEd7v7geZSRGs=', 'Content-Length': '0'} 2024-04-16T20:12:43.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086a322b2e5ebf663-00661edbbb-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:43.809 DEBUG:rgw_multi.tests:created bucket=yrjefh-54 2024-04-16T20:12:43.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.810 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.856 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.857 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:43.912 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:12:43.934 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713298363.816594_1000.1","last_update":"2024-04-16T20:12:43.816594Z"}] 2024-04-16T20:12:43.934 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298363.816594_1000.1'} 2024-04-16T20:12:43.934 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:12:43.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:12:43.935 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:12:43.983 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:43.983 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:44.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:44.043+0000 7fbf38331800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:12:44.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:44.044+0000 7fbf38331800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:12:44.058 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298363.816594_1000.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:12:43.816594Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:12:44Z"} 2024-04-16T20:12:44.058 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298363.816594_1000.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:12:43.816594Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:12:44Z"} 2024-04-16T20:12:44.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298363.816594_1000.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:12:43.816594Z', 'realm_epoch': 2}}] 2024-04-16T20:12:44.058 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298363.816594_1000.1'} 2024-04-16T20:12:44.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298363.816594_1000.1'} 2024-04-16T20:12:44.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:12:44.059 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:12:44.059 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:12:44.060 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:12:44.069+0000 7efd69f7a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 195927) UID: 0 2024-04-16T20:12:44.060 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:12:44.069+0000 7efd7002d880 -1 shutting down 2024-04-16T20:12:47.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:12:50.160 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:12:50.161 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.161 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.161 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.161 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.161 DEBUG:boto:Data: 2024-04-16T20:12:50.161 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.161 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.161 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.161 DEBUG:boto:Params: {} 2024-04-16T20:12:50.162 DEBUG:boto:Token: None 2024-04-16T20:12:50.162 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.162 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9ea653bef92675b-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.165 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:12:50.165 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:12:50.165 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.165 DEBUG:boto:Path: /yrjefh-54/obj11 2024-04-16T20:12:50.165 DEBUG:boto:Data: 2024-04-16T20:12:50.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.165 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.165 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.165 DEBUG:boto:Params: {} 2024-04-16T20:12:50.166 DEBUG:boto:Token: None 2024-04-16T20:12:50.166 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj11 2024-04-16T20:12:50.166 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oWlAV6FNgO5fgK73GbErrjKd3vU= 2024-04-16T20:12:50.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oWlAV6FNgO5fgK73GbErrjKd3vU='} 2024-04-16T20:12:50.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000413ae9760080279d-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.174 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.174 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.174 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.174 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.174 DEBUG:boto:Data: 2024-04-16T20:12:50.174 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.174 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.175 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.175 DEBUG:boto:Params: {} 2024-04-16T20:12:50.175 DEBUG:boto:Token: None 2024-04-16T20:12:50.175 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.175 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073093216754bbc3e-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.178 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:12:50.178 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:12:50.178 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.178 DEBUG:boto:Path: /yrjefh-54/obj22 2024-04-16T20:12:50.178 DEBUG:boto:Data: 2024-04-16T20:12:50.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.179 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.179 DEBUG:boto:Params: {} 2024-04-16T20:12:50.179 DEBUG:boto:Token: None 2024-04-16T20:12:50.179 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj22 2024-04-16T20:12:50.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4c6i7BGx5gkdehVaFvUSjr3PTnI= 2024-04-16T20:12:50.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4c6i7BGx5gkdehVaFvUSjr3PTnI='} 2024-04-16T20:12:50.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deb6f9fb82d36a22-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.184 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.184 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.184 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.184 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.184 DEBUG:boto:Data: 2024-04-16T20:12:50.184 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.184 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.184 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.185 DEBUG:boto:Params: {} 2024-04-16T20:12:50.185 DEBUG:boto:Token: None 2024-04-16T20:12:50.185 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.185 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b68241376922ef6-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.187 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:12:50.188 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:12:50.188 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.188 DEBUG:boto:Path: /yrjefh-54/obj33 2024-04-16T20:12:50.188 DEBUG:boto:Data: 2024-04-16T20:12:50.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.188 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.188 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.188 DEBUG:boto:Params: {} 2024-04-16T20:12:50.188 DEBUG:boto:Token: None 2024-04-16T20:12:50.188 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj33 2024-04-16T20:12:50.189 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:p3iZhDnXvcI3KjAD9i4YRL81+ic= 2024-04-16T20:12:50.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p3iZhDnXvcI3KjAD9i4YRL81+ic='} 2024-04-16T20:12:50.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be0683eac3663309-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.231 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.231 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.231 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.231 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.231 DEBUG:boto:Data: 2024-04-16T20:12:50.231 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.231 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.231 DEBUG:boto:Params: {} 2024-04-16T20:12:50.232 DEBUG:boto:Token: None 2024-04-16T20:12:50.232 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.232 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cba06510701809a0-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.234 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:12:50.235 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:12:50.235 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.235 DEBUG:boto:Path: /yrjefh-54/obj44 2024-04-16T20:12:50.235 DEBUG:boto:Data: 2024-04-16T20:12:50.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.235 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.235 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.235 DEBUG:boto:Params: {} 2024-04-16T20:12:50.235 DEBUG:boto:Token: None 2024-04-16T20:12:50.235 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj44 2024-04-16T20:12:50.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:81S0u5UZTfbAc2P88fwT6qhlnaE= 2024-04-16T20:12:50.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:81S0u5UZTfbAc2P88fwT6qhlnaE='} 2024-04-16T20:12:50.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4887eb828502344-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.242 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.242 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.242 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.242 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.242 DEBUG:boto:Data: 2024-04-16T20:12:50.242 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.242 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.242 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.242 DEBUG:boto:Params: {} 2024-04-16T20:12:50.243 DEBUG:boto:Token: None 2024-04-16T20:12:50.243 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.243 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7139b86f443d300-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.245 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:12:50.245 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:12:50.245 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.245 DEBUG:boto:Path: /yrjefh-54/obj55 2024-04-16T20:12:50.245 DEBUG:boto:Data: 2024-04-16T20:12:50.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.246 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.246 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.246 DEBUG:boto:Params: {} 2024-04-16T20:12:50.246 DEBUG:boto:Token: None 2024-04-16T20:12:50.246 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj55 2024-04-16T20:12:50.246 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8n7sXcdZO1Q8qT2zBXDXyCCvx8g= 2024-04-16T20:12:50.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8n7sXcdZO1Q8qT2zBXDXyCCvx8g='} 2024-04-16T20:12:50.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000168a6fe5db476341-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.251 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:50.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:50.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:50.373 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:50.373 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:50.377 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 5 2024-04-16T20:12:50.426 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.426 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.426 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.426 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.426 DEBUG:boto:Data: 2024-04-16T20:12:50.426 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.426 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.426 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.426 DEBUG:boto:Params: {} 2024-04-16T20:12:50.427 DEBUG:boto:Token: None 2024-04-16T20:12:50.427 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd105601c2bf55d2-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.429 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:12:50.429 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:12:50.430 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.430 DEBUG:boto:Path: /yrjefh-54/obj17 2024-04-16T20:12:50.430 DEBUG:boto:Data: 2024-04-16T20:12:50.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.430 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.430 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.430 DEBUG:boto:Params: {} 2024-04-16T20:12:50.430 DEBUG:boto:Token: None 2024-04-16T20:12:50.431 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj17 2024-04-16T20:12:50.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:X24sD+WeXrouYx6hjSgyIaYgDDU= 2024-04-16T20:12:50.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X24sD+WeXrouYx6hjSgyIaYgDDU='} 2024-04-16T20:12:50.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f85f545393b8eb89-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.437 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.437 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.437 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.437 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.437 DEBUG:boto:Data: 2024-04-16T20:12:50.437 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.437 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.437 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.437 DEBUG:boto:Params: {} 2024-04-16T20:12:50.437 DEBUG:boto:Token: None 2024-04-16T20:12:50.438 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.438 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026b6775c8fd3813e-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.440 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:12:50.440 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:12:50.440 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.440 DEBUG:boto:Path: /yrjefh-54/obj34 2024-04-16T20:12:50.441 DEBUG:boto:Data: 2024-04-16T20:12:50.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.441 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.441 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.441 DEBUG:boto:Params: {} 2024-04-16T20:12:50.441 DEBUG:boto:Token: None 2024-04-16T20:12:50.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj34 2024-04-16T20:12:50.441 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O2EgNJek7ufrSpY+/CCPmcO9PwQ= 2024-04-16T20:12:50.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O2EgNJek7ufrSpY+/CCPmcO9PwQ='} 2024-04-16T20:12:50.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c62208eeb07fbb1-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.486 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.487 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.487 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.487 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.487 DEBUG:boto:Data: 2024-04-16T20:12:50.487 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.487 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.487 DEBUG:boto:Params: {} 2024-04-16T20:12:50.487 DEBUG:boto:Token: None 2024-04-16T20:12:50.487 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b5bc66a9f10dc1d-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.490 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:12:50.490 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:12:50.490 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.490 DEBUG:boto:Path: /yrjefh-54/obj51 2024-04-16T20:12:50.490 DEBUG:boto:Data: 2024-04-16T20:12:50.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.491 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.491 DEBUG:boto:Params: {} 2024-04-16T20:12:50.491 DEBUG:boto:Token: None 2024-04-16T20:12:50.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj51 2024-04-16T20:12:50.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ghpSWTatK/IJyUWnFxm9Pl9JZlY= 2024-04-16T20:12:50.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ghpSWTatK/IJyUWnFxm9Pl9JZlY='} 2024-04-16T20:12:50.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eb47e709ce06ec0-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.496 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.497 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.497 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.497 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.497 DEBUG:boto:Data: 2024-04-16T20:12:50.497 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.497 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.497 DEBUG:boto:Params: {} 2024-04-16T20:12:50.497 DEBUG:boto:Token: None 2024-04-16T20:12:50.497 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ce869fdc99e3fd-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.500 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:12:50.500 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:12:50.500 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.500 DEBUG:boto:Path: /yrjefh-54/obj68 2024-04-16T20:12:50.500 DEBUG:boto:Data: 2024-04-16T20:12:50.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.500 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.500 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.500 DEBUG:boto:Params: {} 2024-04-16T20:12:50.500 DEBUG:boto:Token: None 2024-04-16T20:12:50.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj68 2024-04-16T20:12:50.501 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6po4H/QytqtkgwmrJQAgPhjFeVI= 2024-04-16T20:12:50.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6po4H/QytqtkgwmrJQAgPhjFeVI='} 2024-04-16T20:12:50.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005058c02e2400704e-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.543 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.543 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.543 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.543 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.543 DEBUG:boto:Data: 2024-04-16T20:12:50.543 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.543 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.543 DEBUG:boto:Params: {} 2024-04-16T20:12:50.543 DEBUG:boto:Token: None 2024-04-16T20:12:50.544 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb0b6ef8eb1a4c06-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.546 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:12:50.546 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:12:50.546 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.546 DEBUG:boto:Path: /yrjefh-54/obj85 2024-04-16T20:12:50.546 DEBUG:boto:Data: 2024-04-16T20:12:50.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.547 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.547 DEBUG:boto:Params: {} 2024-04-16T20:12:50.547 DEBUG:boto:Token: None 2024-04-16T20:12:50.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj85 2024-04-16T20:12:50.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+mZqJRcDpDTHF64F8HJSGdYDQT0= 2024-04-16T20:12:50.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+mZqJRcDpDTHF64F8HJSGdYDQT0='} 2024-04-16T20:12:50.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbaae460324739ad-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.553 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:50.605 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:50.605 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:50.668 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:50.668 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:50.675 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 10 2024-04-16T20:12:50.728 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.728 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.729 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.729 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.729 DEBUG:boto:Data: 2024-04-16T20:12:50.729 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.729 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.729 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.729 DEBUG:boto:Params: {} 2024-04-16T20:12:50.729 DEBUG:boto:Token: None 2024-04-16T20:12:50.729 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000192752b564b1246a-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.732 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:12:50.732 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:12:50.732 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.732 DEBUG:boto:Path: /yrjefh-54/obj19 2024-04-16T20:12:50.732 DEBUG:boto:Data: 2024-04-16T20:12:50.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.733 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.733 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.733 DEBUG:boto:Params: {} 2024-04-16T20:12:50.733 DEBUG:boto:Token: None 2024-04-16T20:12:50.733 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj19 2024-04-16T20:12:50.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QRfUlCDCM/M8Dhh9Jvyx8E64iGc= 2024-04-16T20:12:50.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QRfUlCDCM/M8Dhh9Jvyx8E64iGc='} 2024-04-16T20:12:50.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ddf2abdf5634ddd-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.738 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.738 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.738 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.739 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.739 DEBUG:boto:Data: 2024-04-16T20:12:50.739 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.739 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.739 DEBUG:boto:Params: {} 2024-04-16T20:12:50.739 DEBUG:boto:Token: None 2024-04-16T20:12:50.739 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a87340b4ab6fcba6-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.742 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:12:50.742 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:12:50.742 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.742 DEBUG:boto:Path: /yrjefh-54/obj38 2024-04-16T20:12:50.742 DEBUG:boto:Data: 2024-04-16T20:12:50.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.742 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.742 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.742 DEBUG:boto:Params: {} 2024-04-16T20:12:50.742 DEBUG:boto:Token: None 2024-04-16T20:12:50.743 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj38 2024-04-16T20:12:50.743 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NSPDPw1f18oVmxsWyFKn2XMSHxI= 2024-04-16T20:12:50.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NSPDPw1f18oVmxsWyFKn2XMSHxI='} 2024-04-16T20:12:50.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003655689769367d52-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.787 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.787 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.787 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.787 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.787 DEBUG:boto:Data: 2024-04-16T20:12:50.787 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.787 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.787 DEBUG:boto:Params: {} 2024-04-16T20:12:50.787 DEBUG:boto:Token: None 2024-04-16T20:12:50.787 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056173df0476b57ec-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.790 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:12:50.790 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:12:50.790 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.790 DEBUG:boto:Path: /yrjefh-54/obj57 2024-04-16T20:12:50.790 DEBUG:boto:Data: 2024-04-16T20:12:50.790 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.791 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.791 DEBUG:boto:Params: {} 2024-04-16T20:12:50.791 DEBUG:boto:Token: None 2024-04-16T20:12:50.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj57 2024-04-16T20:12:50.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SR8Lz5rayqRjpeFKtsjM6JxMbkw= 2024-04-16T20:12:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SR8Lz5rayqRjpeFKtsjM6JxMbkw='} 2024-04-16T20:12:50.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022ea1a48631f17eb-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.795 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.795 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.795 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.795 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.795 DEBUG:boto:Data: 2024-04-16T20:12:50.795 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.795 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.795 DEBUG:boto:Params: {} 2024-04-16T20:12:50.795 DEBUG:boto:Token: None 2024-04-16T20:12:50.796 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.796 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009432543097a5984-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.798 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:12:50.798 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:12:50.798 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.798 DEBUG:boto:Path: /yrjefh-54/obj76 2024-04-16T20:12:50.798 DEBUG:boto:Data: 2024-04-16T20:12:50.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.798 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.799 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.799 DEBUG:boto:Params: {} 2024-04-16T20:12:50.799 DEBUG:boto:Token: None 2024-04-16T20:12:50.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj76 2024-04-16T20:12:50.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ompkYJRAbW+WjD6tJjfoWQglBTw= 2024-04-16T20:12:50.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ompkYJRAbW+WjD6tJjfoWQglBTw='} 2024-04-16T20:12:50.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f50d5c2aac25e34-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.804 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:50.804 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:50.804 DEBUG:boto:Method: HEAD 2024-04-16T20:12:50.804 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:50.804 DEBUG:boto:Data: 2024-04-16T20:12:50.804 DEBUG:boto:Headers: {} 2024-04-16T20:12:50.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.804 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.805 DEBUG:boto:Params: {} 2024-04-16T20:12:50.805 DEBUG:boto:Token: None 2024-04-16T20:12:50.805 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/ 2024-04-16T20:12:50.805 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo= 2024-04-16T20:12:50.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h0bFELhlfnnr60aQ4VZfYNZPCAo=', 'Content-Length': '0'} 2024-04-16T20:12:50.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000578d564222005720-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.807 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:12:50.807 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:12:50.807 DEBUG:boto:Method: PUT 2024-04-16T20:12:50.807 DEBUG:boto:Path: /yrjefh-54/obj95 2024-04-16T20:12:50.807 DEBUG:boto:Data: 2024-04-16T20:12:50.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:50.808 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:50.808 DEBUG:boto:Port: 8000 2024-04-16T20:12:50.808 DEBUG:boto:Params: {} 2024-04-16T20:12:50.808 DEBUG:boto:Token: None 2024-04-16T20:12:50.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:50 GMT /yrjefh-54/obj95 2024-04-16T20:12:50.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bSXiK3AePiuxZLiXFn6fjpTce6k= 2024-04-16T20:12:50.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bSXiK3AePiuxZLiXFn6fjpTce6k='} 2024-04-16T20:12:50.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000001aab8c2d7e7c39-00661edbc2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:50.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:50.851 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:50.901 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:50.902 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:50.964 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:50.965 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:50.973 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 15 2024-04-16T20:12:51.032 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.033 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.033 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.033 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.033 DEBUG:boto:Data: 2024-04-16T20:12:51.033 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.033 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.033 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.033 DEBUG:boto:Params: {} 2024-04-16T20:12:51.034 DEBUG:boto:Token: None 2024-04-16T20:12:51.034 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.034 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e77849a57678b9e7-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.038 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:12:51.038 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:12:51.038 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.038 DEBUG:boto:Path: /yrjefh-54/obj23 2024-04-16T20:12:51.038 DEBUG:boto:Data: 2024-04-16T20:12:51.038 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.038 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.039 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.039 DEBUG:boto:Params: {} 2024-04-16T20:12:51.039 DEBUG:boto:Token: None 2024-04-16T20:12:51.039 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj23 2024-04-16T20:12:51.039 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8br7Q5XdVYfgMW7mz4sZvWoL1v0= 2024-04-16T20:12:51.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8br7Q5XdVYfgMW7mz4sZvWoL1v0='} 2024-04-16T20:12:51.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004427433dcac334c7-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.045 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.045 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.045 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.045 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.045 DEBUG:boto:Data: 2024-04-16T20:12:51.045 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.045 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.045 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.045 DEBUG:boto:Params: {} 2024-04-16T20:12:51.045 DEBUG:boto:Token: None 2024-04-16T20:12:51.046 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.046 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ae9b598ff7c788a-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.048 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:12:51.048 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:12:51.048 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.049 DEBUG:boto:Path: /yrjefh-54/obj46 2024-04-16T20:12:51.049 DEBUG:boto:Data: 2024-04-16T20:12:51.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.049 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.049 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.049 DEBUG:boto:Params: {} 2024-04-16T20:12:51.049 DEBUG:boto:Token: None 2024-04-16T20:12:51.049 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj46 2024-04-16T20:12:51.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JwZZUYPnAY/70aL/tcdlWLYvdkk= 2024-04-16T20:12:51.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JwZZUYPnAY/70aL/tcdlWLYvdkk='} 2024-04-16T20:12:51.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4b45c97c787f69b-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.054 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.055 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.055 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.055 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.055 DEBUG:boto:Data: 2024-04-16T20:12:51.055 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.055 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.055 DEBUG:boto:Params: {} 2024-04-16T20:12:51.055 DEBUG:boto:Token: None 2024-04-16T20:12:51.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076f643da67ea2eed-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.058 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:12:51.058 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:12:51.058 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.058 DEBUG:boto:Path: /yrjefh-54/obj69 2024-04-16T20:12:51.058 DEBUG:boto:Data: 2024-04-16T20:12:51.058 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.059 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.059 DEBUG:boto:Params: {} 2024-04-16T20:12:51.059 DEBUG:boto:Token: None 2024-04-16T20:12:51.059 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj69 2024-04-16T20:12:51.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hZWA5dBGgombrSi8vCXeRqbMZaQ= 2024-04-16T20:12:51.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hZWA5dBGgombrSi8vCXeRqbMZaQ='} 2024-04-16T20:12:51.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053d03498451a9a26-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.064 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.065 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.065 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.065 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.065 DEBUG:boto:Data: 2024-04-16T20:12:51.065 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.065 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.065 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.065 DEBUG:boto:Params: {} 2024-04-16T20:12:51.065 DEBUG:boto:Token: None 2024-04-16T20:12:51.065 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.065 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d25dfaa4e684ced-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.068 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:12:51.068 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:12:51.068 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.068 DEBUG:boto:Path: /yrjefh-54/obj92 2024-04-16T20:12:51.068 DEBUG:boto:Data: 2024-04-16T20:12:51.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.069 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.069 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.069 DEBUG:boto:Params: {} 2024-04-16T20:12:51.069 DEBUG:boto:Token: None 2024-04-16T20:12:51.069 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj92 2024-04-16T20:12:51.069 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RO1/MISH5BUH6iXAxv/DePRtaUk= 2024-04-16T20:12:51.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RO1/MISH5BUH6iXAxv/DePRtaUk='} 2024-04-16T20:12:51.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b0ea33bafc3f548-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.110 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.111 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.111 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.111 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.111 DEBUG:boto:Data: 2024-04-16T20:12:51.111 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.111 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.111 DEBUG:boto:Params: {} 2024-04-16T20:12:51.111 DEBUG:boto:Token: None 2024-04-16T20:12:51.111 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bd99553644f42da-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.114 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:12:51.114 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:12:51.114 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.114 DEBUG:boto:Path: /yrjefh-54/obj115 2024-04-16T20:12:51.114 DEBUG:boto:Data: 2024-04-16T20:12:51.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.115 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.115 DEBUG:boto:Params: {} 2024-04-16T20:12:51.115 DEBUG:boto:Token: None 2024-04-16T20:12:51.115 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj115 2024-04-16T20:12:51.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LJdFYx/3bAg14+N1aLfG4IaDE6g= 2024-04-16T20:12:51.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LJdFYx/3bAg14+N1aLfG4IaDE6g='} 2024-04-16T20:12:51.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073bed2949aec9b64-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.121 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:51.171 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:51.171 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:51.237 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:51.237 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:51.246 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 20 2024-04-16T20:12:51.307 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.307 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.307 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.307 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.307 DEBUG:boto:Data: 2024-04-16T20:12:51.307 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.308 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.308 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.308 DEBUG:boto:Params: {} 2024-04-16T20:12:51.308 DEBUG:boto:Token: None 2024-04-16T20:12:51.308 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5e3239daf09057-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.311 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:12:51.311 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:12:51.311 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.311 DEBUG:boto:Path: /yrjefh-54/obj29 2024-04-16T20:12:51.311 DEBUG:boto:Data: 2024-04-16T20:12:51.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.312 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.312 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.312 DEBUG:boto:Params: {} 2024-04-16T20:12:51.312 DEBUG:boto:Token: None 2024-04-16T20:12:51.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj29 2024-04-16T20:12:51.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RwFJ4Pb0fhntRdn4nB6Nr+AOsV4= 2024-04-16T20:12:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RwFJ4Pb0fhntRdn4nB6Nr+AOsV4='} 2024-04-16T20:12:51.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064a019c08376a29a-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.317 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.317 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.317 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.317 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.318 DEBUG:boto:Data: 2024-04-16T20:12:51.318 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.318 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.318 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.318 DEBUG:boto:Params: {} 2024-04-16T20:12:51.318 DEBUG:boto:Token: None 2024-04-16T20:12:51.318 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.318 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad1ae92ecf4096d-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.321 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:12:51.321 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:12:51.321 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.321 DEBUG:boto:Path: /yrjefh-54/obj58 2024-04-16T20:12:51.322 DEBUG:boto:Data: 2024-04-16T20:12:51.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.322 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.322 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.322 DEBUG:boto:Params: {} 2024-04-16T20:12:51.322 DEBUG:boto:Token: None 2024-04-16T20:12:51.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj58 2024-04-16T20:12:51.322 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vLSEGCQqEVvM0UXkMyvzhQ6Ccn0= 2024-04-16T20:12:51.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vLSEGCQqEVvM0UXkMyvzhQ6Ccn0='} 2024-04-16T20:12:51.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5b03ba053777783-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.367 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.367 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.367 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.367 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.367 DEBUG:boto:Data: 2024-04-16T20:12:51.367 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.367 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.367 DEBUG:boto:Params: {} 2024-04-16T20:12:51.367 DEBUG:boto:Token: None 2024-04-16T20:12:51.368 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000563143059f8a8f18-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.370 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:12:51.370 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:12:51.371 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.371 DEBUG:boto:Path: /yrjefh-54/obj87 2024-04-16T20:12:51.371 DEBUG:boto:Data: 2024-04-16T20:12:51.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.371 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.371 DEBUG:boto:Params: {} 2024-04-16T20:12:51.371 DEBUG:boto:Token: None 2024-04-16T20:12:51.371 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj87 2024-04-16T20:12:51.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/6fSrbycicqwHnAJnahRB908/s0= 2024-04-16T20:12:51.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/6fSrbycicqwHnAJnahRB908/s0='} 2024-04-16T20:12:51.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007df248c463475fbb-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.377 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.377 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.377 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.377 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.377 DEBUG:boto:Data: 2024-04-16T20:12:51.377 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.377 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.377 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.377 DEBUG:boto:Params: {} 2024-04-16T20:12:51.377 DEBUG:boto:Token: None 2024-04-16T20:12:51.377 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050b36612a1185238-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.381 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:12:51.381 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:12:51.381 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.381 DEBUG:boto:Path: /yrjefh-54/obj116 2024-04-16T20:12:51.381 DEBUG:boto:Data: 2024-04-16T20:12:51.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.381 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.381 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.381 DEBUG:boto:Params: {} 2024-04-16T20:12:51.381 DEBUG:boto:Token: None 2024-04-16T20:12:51.382 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj116 2024-04-16T20:12:51.382 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lQRe/CUyQePUTdrpW2B8roREe2s= 2024-04-16T20:12:51.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lQRe/CUyQePUTdrpW2B8roREe2s='} 2024-04-16T20:12:51.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9194859d1577102-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.387 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.387 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.387 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.387 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.387 DEBUG:boto:Data: 2024-04-16T20:12:51.387 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.387 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.387 DEBUG:boto:Params: {} 2024-04-16T20:12:51.387 DEBUG:boto:Token: None 2024-04-16T20:12:51.388 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e385de29c5653002-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.390 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:12:51.390 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:12:51.390 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.390 DEBUG:boto:Path: /yrjefh-54/obj145 2024-04-16T20:12:51.391 DEBUG:boto:Data: 2024-04-16T20:12:51.391 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.391 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.391 DEBUG:boto:Params: {} 2024-04-16T20:12:51.391 DEBUG:boto:Token: None 2024-04-16T20:12:51.391 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj145 2024-04-16T20:12:51.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4+CWBZb1txHIRj8A73l0Ty7YrBI= 2024-04-16T20:12:51.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4+CWBZb1txHIRj8A73l0Ty7YrBI='} 2024-04-16T20:12:51.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d18619f6b0100c28-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.435 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:51.552 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:51.552 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:51.565 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 25 2024-04-16T20:12:51.637 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.637 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.638 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.638 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.638 DEBUG:boto:Data: 2024-04-16T20:12:51.638 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.638 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.638 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.638 DEBUG:boto:Params: {} 2024-04-16T20:12:51.638 DEBUG:boto:Token: None 2024-04-16T20:12:51.638 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.638 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8dbf4047d75a19b-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.642 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:12:51.642 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:12:51.642 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.642 DEBUG:boto:Path: /yrjefh-54/obj31 2024-04-16T20:12:51.642 DEBUG:boto:Data: 2024-04-16T20:12:51.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.642 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.642 DEBUG:boto:Params: {} 2024-04-16T20:12:51.642 DEBUG:boto:Token: None 2024-04-16T20:12:51.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj31 2024-04-16T20:12:51.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ch8nvZEJdJD88dKQA5uk1SmajwQ= 2024-04-16T20:12:51.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ch8nvZEJdJD88dKQA5uk1SmajwQ='} 2024-04-16T20:12:51.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4164693549054fd-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.648 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.648 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.648 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.649 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.649 DEBUG:boto:Data: 2024-04-16T20:12:51.649 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.649 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.649 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.649 DEBUG:boto:Params: {} 2024-04-16T20:12:51.649 DEBUG:boto:Token: None 2024-04-16T20:12:51.649 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.649 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000984a2af3796d6ca3-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.652 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:12:51.652 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:12:51.652 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.652 DEBUG:boto:Path: /yrjefh-54/obj62 2024-04-16T20:12:51.652 DEBUG:boto:Data: 2024-04-16T20:12:51.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.653 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.653 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.653 DEBUG:boto:Params: {} 2024-04-16T20:12:51.653 DEBUG:boto:Token: None 2024-04-16T20:12:51.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj62 2024-04-16T20:12:51.653 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4j8ilbTj3rmeI0riJ8st7fFyheQ= 2024-04-16T20:12:51.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4j8ilbTj3rmeI0riJ8st7fFyheQ='} 2024-04-16T20:12:51.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d5881d24606ab49-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.659 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.659 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.659 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.659 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.659 DEBUG:boto:Data: 2024-04-16T20:12:51.659 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.659 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.660 DEBUG:boto:Params: {} 2024-04-16T20:12:51.660 DEBUG:boto:Token: None 2024-04-16T20:12:51.660 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004dffd5787634871-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.663 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:12:51.663 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:12:51.663 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.663 DEBUG:boto:Path: /yrjefh-54/obj93 2024-04-16T20:12:51.663 DEBUG:boto:Data: 2024-04-16T20:12:51.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.664 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.664 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.664 DEBUG:boto:Params: {} 2024-04-16T20:12:51.664 DEBUG:boto:Token: None 2024-04-16T20:12:51.664 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj93 2024-04-16T20:12:51.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qM9A2PD8Ghr6MqOP8LAw+in6/3Y= 2024-04-16T20:12:51.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qM9A2PD8Ghr6MqOP8LAw+in6/3Y='} 2024-04-16T20:12:51.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8834da626b3c225-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.707 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.707 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.707 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.707 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.707 DEBUG:boto:Data: 2024-04-16T20:12:51.707 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.707 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.707 DEBUG:boto:Params: {} 2024-04-16T20:12:51.707 DEBUG:boto:Token: None 2024-04-16T20:12:51.707 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa245f5e6b54bf88-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.711 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:12:51.711 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:12:51.711 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.711 DEBUG:boto:Path: /yrjefh-54/obj124 2024-04-16T20:12:51.711 DEBUG:boto:Data: 2024-04-16T20:12:51.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.711 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.711 DEBUG:boto:Params: {} 2024-04-16T20:12:51.711 DEBUG:boto:Token: None 2024-04-16T20:12:51.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj124 2024-04-16T20:12:51.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3L/lNgA/bh6KApx48cX66A4IX7Q= 2024-04-16T20:12:51.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3L/lNgA/bh6KApx48cX66A4IX7Q='} 2024-04-16T20:12:51.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057a95cc2016d7697-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.717 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.717 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.717 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.717 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.718 DEBUG:boto:Data: 2024-04-16T20:12:51.718 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.718 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.718 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.718 DEBUG:boto:Params: {} 2024-04-16T20:12:51.718 DEBUG:boto:Token: None 2024-04-16T20:12:51.718 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fabd2b2047c3ec39-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.721 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:12:51.721 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:12:51.721 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.722 DEBUG:boto:Path: /yrjefh-54/obj155 2024-04-16T20:12:51.722 DEBUG:boto:Data: 2024-04-16T20:12:51.722 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.722 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.722 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.722 DEBUG:boto:Params: {} 2024-04-16T20:12:51.722 DEBUG:boto:Token: None 2024-04-16T20:12:51.722 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj155 2024-04-16T20:12:51.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DfloWYL6zQBx7qj5r9+3sn9z/e8= 2024-04-16T20:12:51.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DfloWYL6zQBx7qj5r9+3sn9z/e8='} 2024-04-16T20:12:51.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022374e7d31274d86-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-54', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:51.767 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-54 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:51.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:51.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:51.885 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:12:51.885 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-54 2024-04-16T20:12:51.898 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 30 2024-04-16T20:12:51.970 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.971 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.971 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.971 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.971 DEBUG:boto:Data: 2024-04-16T20:12:51.971 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.971 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.971 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.971 DEBUG:boto:Params: {} 2024-04-16T20:12:51.971 DEBUG:boto:Token: None 2024-04-16T20:12:51.971 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000439091340218e0ee-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.974 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:12:51.975 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:12:51.975 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.975 DEBUG:boto:Path: /yrjefh-54/obj37 2024-04-16T20:12:51.975 DEBUG:boto:Data: 2024-04-16T20:12:51.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.975 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.975 DEBUG:boto:Params: {} 2024-04-16T20:12:51.975 DEBUG:boto:Token: None 2024-04-16T20:12:51.975 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj37 2024-04-16T20:12:51.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e0N12MaYrZSgFX7IcpHZ+Kg2LNw= 2024-04-16T20:12:51.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e0N12MaYrZSgFX7IcpHZ+Kg2LNw='} 2024-04-16T20:12:51.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010e91305281b6170-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.980 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:51.981 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:51.981 DEBUG:boto:Method: HEAD 2024-04-16T20:12:51.981 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:51.981 DEBUG:boto:Data: 2024-04-16T20:12:51.981 DEBUG:boto:Headers: {} 2024-04-16T20:12:51.981 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.981 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.981 DEBUG:boto:Params: {} 2024-04-16T20:12:51.981 DEBUG:boto:Token: None 2024-04-16T20:12:51.981 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/ 2024-04-16T20:12:51.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98= 2024-04-16T20:12:51.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3j9tTMYk8bTCqp0UvBgdbH2Rq98=', 'Content-Length': '0'} 2024-04-16T20:12:51.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000032fa101cb2893d9-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:51.984 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:12:51.984 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:12:51.984 DEBUG:boto:Method: PUT 2024-04-16T20:12:51.984 DEBUG:boto:Path: /yrjefh-54/obj74 2024-04-16T20:12:51.984 DEBUG:boto:Data: 2024-04-16T20:12:51.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:51.985 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:51.985 DEBUG:boto:Port: 8000 2024-04-16T20:12:51.985 DEBUG:boto:Params: {} 2024-04-16T20:12:51.985 DEBUG:boto:Token: None 2024-04-16T20:12:51.985 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:51 GMT /yrjefh-54/obj74 2024-04-16T20:12:51.985 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0D52vXr4y6c5K5b8sqFvVED0p1c= 2024-04-16T20:12:51.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D52vXr4y6c5K5b8sqFvVED0p1c='} 2024-04-16T20:12:52.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2a9b7485ddc8d71-00661edbc3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.026 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:52.027 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:52.027 DEBUG:boto:Method: HEAD 2024-04-16T20:12:52.027 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:52.027 DEBUG:boto:Data: 2024-04-16T20:12:52.027 DEBUG:boto:Headers: {} 2024-04-16T20:12:52.027 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.027 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.027 DEBUG:boto:Params: {} 2024-04-16T20:12:52.027 DEBUG:boto:Token: None 2024-04-16T20:12:52.027 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/ 2024-04-16T20:12:52.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI= 2024-04-16T20:12:52.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI=', 'Content-Length': '0'} 2024-04-16T20:12:52.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9f4e6e5f2f63299-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.031 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:12:52.031 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:12:52.031 DEBUG:boto:Method: PUT 2024-04-16T20:12:52.031 DEBUG:boto:Path: /yrjefh-54/obj111 2024-04-16T20:12:52.031 DEBUG:boto:Data: 2024-04-16T20:12:52.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:52.031 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.031 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.031 DEBUG:boto:Params: {} 2024-04-16T20:12:52.031 DEBUG:boto:Token: None 2024-04-16T20:12:52.031 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/obj111 2024-04-16T20:12:52.032 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FUbwK0Q3ImsQ4uKebOpVqFXKBqE= 2024-04-16T20:12:52.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FUbwK0Q3ImsQ4uKebOpVqFXKBqE='} 2024-04-16T20:12:52.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fe426c2b7c9c233-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.037 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:52.037 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:52.037 DEBUG:boto:Method: HEAD 2024-04-16T20:12:52.037 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:52.037 DEBUG:boto:Data: 2024-04-16T20:12:52.037 DEBUG:boto:Headers: {} 2024-04-16T20:12:52.037 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.037 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.037 DEBUG:boto:Params: {} 2024-04-16T20:12:52.037 DEBUG:boto:Token: None 2024-04-16T20:12:52.037 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/ 2024-04-16T20:12:52.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI= 2024-04-16T20:12:52.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI=', 'Content-Length': '0'} 2024-04-16T20:12:52.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000888ccbbf04f7f9d1-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.041 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:12:52.041 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:12:52.041 DEBUG:boto:Method: PUT 2024-04-16T20:12:52.041 DEBUG:boto:Path: /yrjefh-54/obj148 2024-04-16T20:12:52.041 DEBUG:boto:Data: 2024-04-16T20:12:52.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:52.041 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.041 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.041 DEBUG:boto:Params: {} 2024-04-16T20:12:52.042 DEBUG:boto:Token: None 2024-04-16T20:12:52.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/obj148 2024-04-16T20:12:52.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HaAFXoBBg+7nxyEav1Nr3sGLgVs= 2024-04-16T20:12:52.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HaAFXoBBg+7nxyEav1Nr3sGLgVs='} 2024-04-16T20:12:52.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000255376b9ffe22959-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.087 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:12:52.087 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:12:52.087 DEBUG:boto:Method: HEAD 2024-04-16T20:12:52.087 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:12:52.087 DEBUG:boto:Data: 2024-04-16T20:12:52.087 DEBUG:boto:Headers: {} 2024-04-16T20:12:52.087 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.087 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.087 DEBUG:boto:Params: {} 2024-04-16T20:12:52.087 DEBUG:boto:Token: None 2024-04-16T20:12:52.087 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/ 2024-04-16T20:12:52.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI= 2024-04-16T20:12:52.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Pt8JGkumIWIK0/UmbzxjfDy/lkI=', 'Content-Length': '0'} 2024-04-16T20:12:52.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c04fd456f0befa40-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.091 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:12:52.091 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:12:52.091 DEBUG:boto:Method: PUT 2024-04-16T20:12:52.091 DEBUG:boto:Path: /yrjefh-54/obj185 2024-04-16T20:12:52.091 DEBUG:boto:Data: 2024-04-16T20:12:52.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:12:52.092 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:12:52.092 DEBUG:boto:Port: 8000 2024-04-16T20:12:52.092 DEBUG:boto:Params: {} 2024-04-16T20:12:52.092 DEBUG:boto:Token: None 2024-04-16T20:12:52.092 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:12:52 GMT /yrjefh-54/obj185 2024-04-16T20:12:52.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cdvqPPRYL09hxmYv2CgTNHIPVn0= 2024-04-16T20:12:52.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:12:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cdvqPPRYL09hxmYv2CgTNHIPVn0='} 2024-04-16T20:12:52.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b4a5eaad22c26c-00661edbc4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:12:52.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:52.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:12:52.098 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:12:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:52.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:52.221+0000 7f1ce022f800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T20:12:52.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:52.223+0000 7f1ce022f800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T20:12:52.245 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:12:52.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:12:52.248 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:12:52.248 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:12:52.267 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:12:52.267 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:12:52.267 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:12:52.267 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:12:52.436 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:12:52.445+0000 7f905cc34880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:12:53.268 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:12:53.288 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:12:53.289 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:12:53.290 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:12:53.290 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:12:53.290 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-54 2024-04-16T20:12:53.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-54', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:12:53.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-54', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:12:53.291 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-54 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:12:53.385 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:12:53.385 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:12:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:12:53.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:53.451+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:12:58.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:12:58.452+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:03.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:03.452+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:08.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:08.453+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:13.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:13.453+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:18.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:18.454+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:23.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:23.454+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:28.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:28.454+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:33.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:33.455+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:38.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:38.455+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:43.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:43.455+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:48.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:48.456+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:53.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:53.457+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:13:58.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:13:58.458+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:03.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:03.458+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:08.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:08.459+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:13.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:13.460+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:18.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:18.460+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:23.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:23.460+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:28.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:28.462+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:33.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:33.462+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:38.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:38.463+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:43.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:43.464+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:48.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:48.464+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:53.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:53.465+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:14:58.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:14:58.466+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:03.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:03.466+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:08.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:08.467+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:13.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:13.467+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:18.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:18.468+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:23.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:23.469+0000 7f63ff733800 1 waiting to reach incremental sync.. 2024-04-16T20:15:28.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:28.472+0000 7f63ff733800 1 bucket sync caught up with source: 2024-04-16T20:15:28.464 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.96.6, , , , , , 00000000002.67.6, , , , , 00000000002.89.6, , , , , , , , , , , , , , , 00000000001.117.6, , , , 00000000001.91.6] 2024-04-16T20:15:28.464 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.96.6, , , , , , 00000000002.67.6, , , , , 00000000002.89.6, , , , , , , , , , , , , , , 00000000001.117.6, , , , 00000000001.91.6] 2024-04-16T20:15:28.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:28.472+0000 7f63ff733800 0 bucket checkpoint complete 2024-04-16T20:15:28.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-54 2024-04-16T20:15:28.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-54', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:28.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-54', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:28.492 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-54 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:15:28.555 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:28.555 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:28.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:28.628+0000 7fd57512f800 1 bucket sync caught up with source: 2024-04-16T20:15:28.619 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.98.6, 00000000002.91.6, 00000000002.179.6, 00000000005.208.6, 00000000004.174.6, 00000000001.247.6, 00000000001.176.6, 00000000004.98.6, 00000000007.228.6, 00000000006.169.6] 2024-04-16T20:15:28.619 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.98.6, 00000000002.91.6, 00000000002.179.6, 00000000005.208.6, 00000000004.174.6, 00000000001.247.6, 00000000001.176.6, 00000000004.98.6, 00000000007.228.6, 00000000006.169.6] 2024-04-16T20:15:28.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:28.628+0000 7fd57512f800 0 bucket checkpoint complete 2024-04-16T20:15:28.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-54 zones={test-zone2, test-zone1} 2024-04-16T20:15:28.642 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.643 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.643 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.643 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:15:28.643 DEBUG:boto:Data: 2024-04-16T20:15:28.643 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.643 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.644 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.644 DEBUG:boto:Params: {} 2024-04-16T20:15:28.644 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:15:28.645 DEBUG:boto:Token: None 2024-04-16T20:15:28.645 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.645 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lSaDyev/o9y7MMRNKMvApRjvdXk= 2024-04-16T20:15:28.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lSaDyev/o9y7MMRNKMvApRjvdXk=', 'Content-Length': '0'} 2024-04-16T20:15:28.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f790d4fbacde54eb-00661edc60-6186-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.654 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.654 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.654 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.655 DEBUG:boto:Path: /yrjefh-54/ 2024-04-16T20:15:28.655 DEBUG:boto:Data: 2024-04-16T20:15:28.655 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.655 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.655 DEBUG:boto:Params: {} 2024-04-16T20:15:28.655 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:15:28.656 DEBUG:boto:Token: None 2024-04-16T20:15:28.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lSaDyev/o9y7MMRNKMvApRjvdXk= 2024-04-16T20:15:28.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lSaDyev/o9y7MMRNKMvApRjvdXk=', 'Content-Length': '0'} 2024-04-16T20:15:28.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fc9f6c569a8b76f-00661edc60-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:15:28.663 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.663 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.663 DEBUG:boto:path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.663 DEBUG:boto:auth_path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.663 DEBUG:boto:Method: GET 2024-04-16T20:15:28.664 DEBUG:boto:Path: /yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.664 DEBUG:boto:Data: 2024-04-16T20:15:28.664 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.664 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.664 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.664 DEBUG:boto:Params: {} 2024-04-16T20:15:28.664 DEBUG:boto:Token: None 2024-04-16T20:15:28.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/?versions 2024-04-16T20:15:28.665 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI= 2024-04-16T20:15:28.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI=', 'Content-Length': '0'} 2024-04-16T20:15:28.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f44af71cfab78c-00661edc60-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.668 DEBUG:boto:b'yrjefh-54999falseobj11nulltrue2024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:15:28.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:15:28.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:15:28.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:15:28.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:15:28.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:15:28.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:15:28.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:15:28.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:15:28.674 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:15:28.674 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.674 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.674 DEBUG:boto:path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.674 DEBUG:boto:auth_path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.674 DEBUG:boto:Method: GET 2024-04-16T20:15:28.675 DEBUG:boto:Path: /yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.675 DEBUG:boto:Data: 2024-04-16T20:15:28.675 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.675 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.675 DEBUG:boto:Params: {} 2024-04-16T20:15:28.675 DEBUG:boto:Token: None 2024-04-16T20:15:28.675 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/?versions 2024-04-16T20:15:28.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI= 2024-04-16T20:15:28.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI=', 'Content-Length': '0'} 2024-04-16T20:15:28.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76aeba1bb7ecc28-00661edc60-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.679 DEBUG:boto:b'yrjefh-54999falseobj11nulltrue2024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:15:28.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:15:28.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:15:28.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:15:28.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:15:28.684 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.685 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.685 DEBUG:boto:path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.685 DEBUG:boto:auth_path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.685 DEBUG:boto:Method: GET 2024-04-16T20:15:28.685 DEBUG:boto:Path: /yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.685 DEBUG:boto:Data: 2024-04-16T20:15:28.685 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.685 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.686 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.686 DEBUG:boto:Params: {} 2024-04-16T20:15:28.686 DEBUG:boto:Token: None 2024-04-16T20:15:28.686 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/?versions 2024-04-16T20:15:28.686 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI= 2024-04-16T20:15:28.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI=', 'Content-Length': '0'} 2024-04-16T20:15:28.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466583ce4bcbd259-00661edc60-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.689 DEBUG:boto:b'yrjefh-54999falseobj11nulltrue2024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.691 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.691 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.691 DEBUG:boto:path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.691 DEBUG:boto:auth_path=/yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.691 DEBUG:boto:Method: GET 2024-04-16T20:15:28.692 DEBUG:boto:Path: /yrjefh-54/?versions&max-keys=999 2024-04-16T20:15:28.692 DEBUG:boto:Data: 2024-04-16T20:15:28.692 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.692 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.692 DEBUG:boto:Params: {} 2024-04-16T20:15:28.692 DEBUG:boto:Token: None 2024-04-16T20:15:28.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/?versions 2024-04-16T20:15:28.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI= 2024-04-16T20:15:28.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:866WA02RI5x79kw7oSJMPq/zhJI=', 'Content-Length': '0'} 2024-04-16T20:15:28.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085452602e8729876-00661edc60-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.696 DEBUG:boto:b'yrjefh-54999falseobj11nulltrue2024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:15:28.698 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.698 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.698 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.699 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.699 DEBUG:boto:Method: GET 2024-04-16T20:15:28.699 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.699 DEBUG:boto:Data: 2024-04-16T20:15:28.699 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.699 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.699 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.699 DEBUG:boto:Params: {} 2024-04-16T20:15:28.699 DEBUG:boto:Token: None 2024-04-16T20:15:28.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo= 2024-04-16T20:15:28.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo=', 'Content-Length': '0'} 2024-04-16T20:15:28.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f610e031c39e32fd-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.703 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.703 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.703 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.703 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.704 DEBUG:boto:Method: GET 2024-04-16T20:15:28.704 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.704 DEBUG:boto:Data: 2024-04-16T20:15:28.704 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.704 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.704 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.704 DEBUG:boto:Params: {} 2024-04-16T20:15:28.704 DEBUG:boto:Token: None 2024-04-16T20:15:28.705 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo= 2024-04-16T20:15:28.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo=', 'Content-Length': '0'} 2024-04-16T20:15:28.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003072796115346e07-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.750 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.751 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.751 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.751 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.751 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.751 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.751 DEBUG:boto:Data: 2024-04-16T20:15:28.751 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.751 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.752 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.752 DEBUG:boto:Params: {} 2024-04-16T20:15:28.752 DEBUG:boto:Token: None 2024-04-16T20:15:28.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jwGAnRNtnb3lNSw0dzJu9jv7DEY= 2024-04-16T20:15:28.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwGAnRNtnb3lNSw0dzJu9jv7DEY=', 'Content-Length': '0'} 2024-04-16T20:15:28.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b389b2dd5c5c2de-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.755 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.755 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.755 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.755 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.755 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.755 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.755 DEBUG:boto:Data: 2024-04-16T20:15:28.756 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.756 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.756 DEBUG:boto:Params: {} 2024-04-16T20:15:28.756 DEBUG:boto:Token: None 2024-04-16T20:15:28.756 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jwGAnRNtnb3lNSw0dzJu9jv7DEY= 2024-04-16T20:15:28.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwGAnRNtnb3lNSw0dzJu9jv7DEY=', 'Content-Length': '0'} 2024-04-16T20:15:28.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589a4eae92df6b6b-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:15:28.759 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.759 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.759 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.759 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.759 DEBUG:boto:Method: GET 2024-04-16T20:15:28.759 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.760 DEBUG:boto:Data: 2024-04-16T20:15:28.760 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.760 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.760 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.760 DEBUG:boto:Params: {} 2024-04-16T20:15:28.760 DEBUG:boto:Token: None 2024-04-16T20:15:28.760 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo= 2024-04-16T20:15:28.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo=', 'Content-Length': '0'} 2024-04-16T20:15:28.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063bc3e3c28445fe3-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.806 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.807 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.807 DEBUG:boto:path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.807 DEBUG:boto:auth_path=/yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.807 DEBUG:boto:Method: GET 2024-04-16T20:15:28.807 DEBUG:boto:Path: /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.807 DEBUG:boto:Data: 2024-04-16T20:15:28.807 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.808 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.808 DEBUG:boto:Params: {} 2024-04-16T20:15:28.808 DEBUG:boto:Token: None 2024-04-16T20:15:28.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11?versionId=null 2024-04-16T20:15:28.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo= 2024-04-16T20:15:28.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CDcjRpJLQmlHrCPonbZyHZZz7zo=', 'Content-Length': '0'} 2024-04-16T20:15:28.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924aa08679b99bfa-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.811 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.811 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.811 DEBUG:boto:path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.811 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.812 DEBUG:boto:Method: GET 2024-04-16T20:15:28.812 DEBUG:boto:Path: /yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.812 DEBUG:boto:Data: 2024-04-16T20:15:28.812 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.812 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.812 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.812 DEBUG:boto:Params: {} 2024-04-16T20:15:28.812 DEBUG:boto:Token: None 2024-04-16T20:15:28.813 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.813 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc925c7f87a75dae-00661edc60-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.816 DEBUG:boto:b'yrjefh-54obj111000falseobj112024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.816 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.817 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.817 DEBUG:boto:path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.817 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.817 DEBUG:boto:Method: GET 2024-04-16T20:15:28.817 DEBUG:boto:Path: /yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.817 DEBUG:boto:Data: 2024-04-16T20:15:28.817 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.817 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.818 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.818 DEBUG:boto:Params: {} 2024-04-16T20:15:28.818 DEBUG:boto:Token: None 2024-04-16T20:15:28.818 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.818 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002167d5ab4a5ae9fd-00661edc60-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.822 DEBUG:boto:b'yrjefh-54obj111000falseobj112024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.822 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.823 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.823 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.823 DEBUG:boto:Path: /yrjefh-54/obj11 2024-04-16T20:15:28.823 DEBUG:boto:Data: 2024-04-16T20:15:28.823 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.823 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.823 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.823 DEBUG:boto:Params: {} 2024-04-16T20:15:28.824 DEBUG:boto:Token: None 2024-04-16T20:15:28.824 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11 2024-04-16T20:15:28.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kqdxA+YymEdwXnJLJWipbKVsoq8= 2024-04-16T20:15:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kqdxA+YymEdwXnJLJWipbKVsoq8=', 'Content-Length': '0'} 2024-04-16T20:15:28.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af24af07e62e7127-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.826 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.827 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.827 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.827 DEBUG:boto:Path: /yrjefh-54/obj11 2024-04-16T20:15:28.827 DEBUG:boto:Data: 2024-04-16T20:15:28.827 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.827 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.827 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.827 DEBUG:boto:Params: {} 2024-04-16T20:15:28.828 DEBUG:boto:Token: None 2024-04-16T20:15:28.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11 2024-04-16T20:15:28.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kqdxA+YymEdwXnJLJWipbKVsoq8= 2024-04-16T20:15:28.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kqdxA+YymEdwXnJLJWipbKVsoq8=', 'Content-Length': '0'} 2024-04-16T20:15:28.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d6b5f3a72ddf7c-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:15:28.830 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.830 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.831 DEBUG:boto:Method: GET 2024-04-16T20:15:28.831 DEBUG:boto:Path: /yrjefh-54/obj11 2024-04-16T20:15:28.831 DEBUG:boto:Data: 2024-04-16T20:15:28.831 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.831 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.831 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.831 DEBUG:boto:Params: {} 2024-04-16T20:15:28.831 DEBUG:boto:Token: None 2024-04-16T20:15:28.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11 2024-04-16T20:15:28.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rk+nNKx2GQHr6Cq+8tIOfzWpX5g= 2024-04-16T20:15:28.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rk+nNKx2GQHr6Cq+8tIOfzWpX5g=', 'Content-Length': '0'} 2024-04-16T20:15:28.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498d36c005d7658a-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.834 DEBUG:boto:path=/yrjefh-54/obj11 2024-04-16T20:15:28.835 DEBUG:boto:auth_path=/yrjefh-54/obj11 2024-04-16T20:15:28.835 DEBUG:boto:Method: GET 2024-04-16T20:15:28.835 DEBUG:boto:Path: /yrjefh-54/obj11 2024-04-16T20:15:28.835 DEBUG:boto:Data: 2024-04-16T20:15:28.835 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.835 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.835 DEBUG:boto:Params: {} 2024-04-16T20:15:28.836 DEBUG:boto:Token: None 2024-04-16T20:15:28.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj11 2024-04-16T20:15:28.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rk+nNKx2GQHr6Cq+8tIOfzWpX5g= 2024-04-16T20:15:28.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rk+nNKx2GQHr6Cq+8tIOfzWpX5g=', 'Content-Length': '0'} 2024-04-16T20:15:28.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005a1e3ed786eb294-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.838 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.839 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.839 DEBUG:boto:path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.839 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.839 DEBUG:boto:Method: GET 2024-04-16T20:15:28.839 DEBUG:boto:Path: /yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.839 DEBUG:boto:Data: 2024-04-16T20:15:28.839 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.840 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.840 DEBUG:boto:Params: {} 2024-04-16T20:15:28.840 DEBUG:boto:Token: None 2024-04-16T20:15:28.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eac87ffa778b7b5-00661edc60-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.843 DEBUG:boto:b'yrjefh-54obj111000falseobj112024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.843 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.843 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.843 DEBUG:boto:path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.843 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.844 DEBUG:boto:Method: GET 2024-04-16T20:15:28.844 DEBUG:boto:Path: /yrjefh-54/?prefix=obj11 2024-04-16T20:15:28.844 DEBUG:boto:Data: 2024-04-16T20:15:28.844 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.844 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.844 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.844 DEBUG:boto:Params: {} 2024-04-16T20:15:28.844 DEBUG:boto:Token: None 2024-04-16T20:15:28.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.845 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c523b70a4352f4-00661edc60-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.890 DEBUG:boto:b'yrjefh-54obj111000falseobj112024-04-16T20:12:50.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:15:28.891 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.891 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.891 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.891 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.892 DEBUG:boto:Method: GET 2024-04-16T20:15:28.892 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.892 DEBUG:boto:Data: 2024-04-16T20:15:28.892 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.892 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.892 DEBUG:boto:Params: {} 2024-04-16T20:15:28.892 DEBUG:boto:Token: None 2024-04-16T20:15:28.893 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE= 2024-04-16T20:15:28.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE=', 'Content-Length': '0'} 2024-04-16T20:15:28.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae4f6b6706f117d-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.895 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.895 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.895 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.895 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.896 DEBUG:boto:Method: GET 2024-04-16T20:15:28.896 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.896 DEBUG:boto:Data: 2024-04-16T20:15:28.896 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.896 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.896 DEBUG:boto:Params: {} 2024-04-16T20:15:28.896 DEBUG:boto:Token: None 2024-04-16T20:15:28.897 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE= 2024-04-16T20:15:28.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE=', 'Content-Length': '0'} 2024-04-16T20:15:28.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023596f65fbdc0d0a-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.899 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.899 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.899 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.899 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.899 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.899 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.899 DEBUG:boto:Data: 2024-04-16T20:15:28.899 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.900 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.900 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.900 DEBUG:boto:Params: {} 2024-04-16T20:15:28.900 DEBUG:boto:Token: None 2024-04-16T20:15:28.900 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dBZpdDm97lBV2nRfMsvLbijHwBE= 2024-04-16T20:15:28.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dBZpdDm97lBV2nRfMsvLbijHwBE=', 'Content-Length': '0'} 2024-04-16T20:15:28.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620e6d87ba39ac7d-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.902 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.903 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.903 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.903 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.903 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.903 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.903 DEBUG:boto:Data: 2024-04-16T20:15:28.903 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.903 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.903 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.904 DEBUG:boto:Params: {} 2024-04-16T20:15:28.904 DEBUG:boto:Token: None 2024-04-16T20:15:28.904 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.904 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dBZpdDm97lBV2nRfMsvLbijHwBE= 2024-04-16T20:15:28.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dBZpdDm97lBV2nRfMsvLbijHwBE=', 'Content-Length': '0'} 2024-04-16T20:15:28.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f82b1e65cd655f-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:15:28.906 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.906 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.906 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.907 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.907 DEBUG:boto:Method: GET 2024-04-16T20:15:28.907 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.907 DEBUG:boto:Data: 2024-04-16T20:15:28.907 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.907 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.907 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.907 DEBUG:boto:Params: {} 2024-04-16T20:15:28.907 DEBUG:boto:Token: None 2024-04-16T20:15:28.907 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.908 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE= 2024-04-16T20:15:28.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE=', 'Content-Length': '0'} 2024-04-16T20:15:28.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6878bda9fd99cb-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.950 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.950 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.951 DEBUG:boto:path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.951 DEBUG:boto:auth_path=/yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.951 DEBUG:boto:Method: GET 2024-04-16T20:15:28.951 DEBUG:boto:Path: /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.951 DEBUG:boto:Data: 2024-04-16T20:15:28.951 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.951 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.951 DEBUG:boto:Params: {} 2024-04-16T20:15:28.952 DEBUG:boto:Token: None 2024-04-16T20:15:28.952 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111?versionId=null 2024-04-16T20:15:28.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE= 2024-04-16T20:15:28.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w1xGK4nPGep9YEjC3mpwaQDLQXE=', 'Content-Length': '0'} 2024-04-16T20:15:28.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b039f37a185a9137-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.954 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.954 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.954 DEBUG:boto:path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.954 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.954 DEBUG:boto:Method: GET 2024-04-16T20:15:28.954 DEBUG:boto:Path: /yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.955 DEBUG:boto:Data: 2024-04-16T20:15:28.955 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.955 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.955 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.955 DEBUG:boto:Params: {} 2024-04-16T20:15:28.955 DEBUG:boto:Token: None 2024-04-16T20:15:28.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731ab79c71afe70c-00661edc60-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.958 DEBUG:boto:b'yrjefh-54obj1111000falseobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.958 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:28.958 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:28.959 DEBUG:boto:path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.959 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.959 DEBUG:boto:Method: GET 2024-04-16T20:15:28.959 DEBUG:boto:Path: /yrjefh-54/?prefix=obj111 2024-04-16T20:15:28.959 DEBUG:boto:Data: 2024-04-16T20:15:28.959 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.959 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.959 DEBUG:boto:Params: {} 2024-04-16T20:15:28.959 DEBUG:boto:Token: None 2024-04-16T20:15:28.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/ 2024-04-16T20:15:28.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50= 2024-04-16T20:15:28.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pn4rKlPsb6ztxKoLaQ143QfZv50=', 'Content-Length': '0'} 2024-04-16T20:15:28.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1894b381330297b-00661edc60-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.963 DEBUG:boto:b'yrjefh-54obj1111000falseobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:28.963 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.963 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.964 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.964 DEBUG:boto:Path: /yrjefh-54/obj111 2024-04-16T20:15:28.964 DEBUG:boto:Data: 2024-04-16T20:15:28.964 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.964 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.964 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.964 DEBUG:boto:Params: {} 2024-04-16T20:15:28.964 DEBUG:boto:Token: None 2024-04-16T20:15:28.964 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111 2024-04-16T20:15:28.965 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BAdlNXpyLLZLo4Qn4xPqea5rfCQ= 2024-04-16T20:15:28.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BAdlNXpyLLZLo4Qn4xPqea5rfCQ=', 'Content-Length': '0'} 2024-04-16T20:15:28.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6109566fd42a2e5-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.967 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.967 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.967 DEBUG:boto:Method: HEAD 2024-04-16T20:15:28.967 DEBUG:boto:Path: /yrjefh-54/obj111 2024-04-16T20:15:28.967 DEBUG:boto:Data: 2024-04-16T20:15:28.967 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.967 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.967 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.968 DEBUG:boto:Params: {} 2024-04-16T20:15:28.968 DEBUG:boto:Token: None 2024-04-16T20:15:28.968 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111 2024-04-16T20:15:28.968 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BAdlNXpyLLZLo4Qn4xPqea5rfCQ= 2024-04-16T20:15:28.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BAdlNXpyLLZLo4Qn4xPqea5rfCQ=', 'Content-Length': '0'} 2024-04-16T20:15:28.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609a187847122604-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:15:28.970 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.970 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.970 DEBUG:boto:Method: GET 2024-04-16T20:15:28.970 DEBUG:boto:Path: /yrjefh-54/obj111 2024-04-16T20:15:28.970 DEBUG:boto:Data: 2024-04-16T20:15:28.970 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.971 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:28.971 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.971 DEBUG:boto:Params: {} 2024-04-16T20:15:28.971 DEBUG:boto:Token: None 2024-04-16T20:15:28.971 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111 2024-04-16T20:15:28.971 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TCKHH3LOpjS/WHhO63XsG3544AQ= 2024-04-16T20:15:28.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TCKHH3LOpjS/WHhO63XsG3544AQ=', 'Content-Length': '0'} 2024-04-16T20:15:28.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf552a608389b2d-00661edc60-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:28.974 DEBUG:boto:path=/yrjefh-54/obj111 2024-04-16T20:15:28.974 DEBUG:boto:auth_path=/yrjefh-54/obj111 2024-04-16T20:15:28.974 DEBUG:boto:Method: GET 2024-04-16T20:15:28.974 DEBUG:boto:Path: /yrjefh-54/obj111 2024-04-16T20:15:28.974 DEBUG:boto:Data: 2024-04-16T20:15:28.974 DEBUG:boto:Headers: {} 2024-04-16T20:15:28.974 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:28.974 DEBUG:boto:Port: 8000 2024-04-16T20:15:28.974 DEBUG:boto:Params: {} 2024-04-16T20:15:28.975 DEBUG:boto:Token: None 2024-04-16T20:15:28.975 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:28 GMT /yrjefh-54/obj111 2024-04-16T20:15:28.975 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TCKHH3LOpjS/WHhO63XsG3544AQ= 2024-04-16T20:15:28.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TCKHH3LOpjS/WHhO63XsG3544AQ=', 'Content-Length': '0'} 2024-04-16T20:15:28.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002203bcd13eb42d99-00661edc60-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.018 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.018 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.019 DEBUG:boto:path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.019 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.019 DEBUG:boto:Method: GET 2024-04-16T20:15:29.019 DEBUG:boto:Path: /yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.019 DEBUG:boto:Data: 2024-04-16T20:15:29.019 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.019 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.019 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.019 DEBUG:boto:Params: {} 2024-04-16T20:15:29.020 DEBUG:boto:Token: None 2024-04-16T20:15:29.020 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4e7a73e358eceda-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.022 DEBUG:boto:b'yrjefh-54obj1111000falseobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.023 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.023 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.023 DEBUG:boto:path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.023 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.023 DEBUG:boto:Method: GET 2024-04-16T20:15:29.023 DEBUG:boto:Path: /yrjefh-54/?prefix=obj111 2024-04-16T20:15:29.023 DEBUG:boto:Data: 2024-04-16T20:15:29.024 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.024 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.024 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.024 DEBUG:boto:Params: {} 2024-04-16T20:15:29.024 DEBUG:boto:Token: None 2024-04-16T20:15:29.024 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b55ed075f5750e-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.028 DEBUG:boto:b'yrjefh-54obj1111000falseobj1112024-04-16T20:12:52.043Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:15:29.028 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.028 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.028 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.028 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.028 DEBUG:boto:Method: GET 2024-04-16T20:15:29.029 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.029 DEBUG:boto:Data: 2024-04-16T20:15:29.029 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.029 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.029 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.029 DEBUG:boto:Params: {} 2024-04-16T20:15:29.029 DEBUG:boto:Token: None 2024-04-16T20:15:29.029 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.029 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ= 2024-04-16T20:15:29.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3546b708971bad-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.074 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.075 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.075 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.075 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.075 DEBUG:boto:Method: GET 2024-04-16T20:15:29.075 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.075 DEBUG:boto:Data: 2024-04-16T20:15:29.075 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.075 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.076 DEBUG:boto:Params: {} 2024-04-16T20:15:29.076 DEBUG:boto:Token: None 2024-04-16T20:15:29.076 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ= 2024-04-16T20:15:29.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aa3209f6cd9cc0c-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.078 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.079 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.079 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.079 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.079 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.079 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.079 DEBUG:boto:Data: 2024-04-16T20:15:29.079 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.079 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.079 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.080 DEBUG:boto:Params: {} 2024-04-16T20:15:29.080 DEBUG:boto:Token: None 2024-04-16T20:15:29.080 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.080 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mKTSynNfFgCLjnRQNNe+XYBEUNI= 2024-04-16T20:15:29.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mKTSynNfFgCLjnRQNNe+XYBEUNI=', 'Content-Length': '0'} 2024-04-16T20:15:29.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004242c58c52650907-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.083 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.083 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.083 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.083 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.083 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.083 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.084 DEBUG:boto:Data: 2024-04-16T20:15:29.084 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.084 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.084 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.084 DEBUG:boto:Params: {} 2024-04-16T20:15:29.084 DEBUG:boto:Token: None 2024-04-16T20:15:29.084 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.084 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mKTSynNfFgCLjnRQNNe+XYBEUNI= 2024-04-16T20:15:29.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mKTSynNfFgCLjnRQNNe+XYBEUNI=', 'Content-Length': '0'} 2024-04-16T20:15:29.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d172db67d17d0b64-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:15:29.087 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.087 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.088 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.088 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.088 DEBUG:boto:Method: GET 2024-04-16T20:15:29.088 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.088 DEBUG:boto:Data: 2024-04-16T20:15:29.088 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.088 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.088 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.088 DEBUG:boto:Params: {} 2024-04-16T20:15:29.088 DEBUG:boto:Token: None 2024-04-16T20:15:29.089 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ= 2024-04-16T20:15:29.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1736b439b43f357-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.092 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.092 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.092 DEBUG:boto:path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.092 DEBUG:boto:auth_path=/yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.092 DEBUG:boto:Method: GET 2024-04-16T20:15:29.092 DEBUG:boto:Path: /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.092 DEBUG:boto:Data: 2024-04-16T20:15:29.092 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.092 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.093 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.093 DEBUG:boto:Params: {} 2024-04-16T20:15:29.093 DEBUG:boto:Token: None 2024-04-16T20:15:29.093 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115?versionId=null 2024-04-16T20:15:29.093 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ= 2024-04-16T20:15:29.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zpvxxPTxRNBW/Xx2NDLbucLpmxQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1f210e24ea051a-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.138 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.139 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.139 DEBUG:boto:path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.139 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.139 DEBUG:boto:Method: GET 2024-04-16T20:15:29.139 DEBUG:boto:Path: /yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.139 DEBUG:boto:Data: 2024-04-16T20:15:29.139 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.139 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.140 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.140 DEBUG:boto:Params: {} 2024-04-16T20:15:29.140 DEBUG:boto:Token: None 2024-04-16T20:15:29.140 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903f87ceb4de4cb1-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.143 DEBUG:boto:b'yrjefh-54obj1151000falseobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.143 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.143 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.143 DEBUG:boto:path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.144 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.144 DEBUG:boto:Method: GET 2024-04-16T20:15:29.144 DEBUG:boto:Path: /yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.144 DEBUG:boto:Data: 2024-04-16T20:15:29.144 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.144 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.144 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.144 DEBUG:boto:Params: {} 2024-04-16T20:15:29.144 DEBUG:boto:Token: None 2024-04-16T20:15:29.145 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.145 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a24065835a92147b-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.148 DEBUG:boto:b'yrjefh-54obj1151000falseobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.149 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.149 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.149 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.149 DEBUG:boto:Path: /yrjefh-54/obj115 2024-04-16T20:15:29.149 DEBUG:boto:Data: 2024-04-16T20:15:29.149 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.149 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.150 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.150 DEBUG:boto:Params: {} 2024-04-16T20:15:29.150 DEBUG:boto:Token: None 2024-04-16T20:15:29.150 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115 2024-04-16T20:15:29.150 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KPo0qTJ8yDuEfJlKABfrLR1x/JE= 2024-04-16T20:15:29.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KPo0qTJ8yDuEfJlKABfrLR1x/JE=', 'Content-Length': '0'} 2024-04-16T20:15:29.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef98ccda7318590-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.153 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.153 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.153 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.153 DEBUG:boto:Path: /yrjefh-54/obj115 2024-04-16T20:15:29.153 DEBUG:boto:Data: 2024-04-16T20:15:29.154 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.154 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.154 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.154 DEBUG:boto:Params: {} 2024-04-16T20:15:29.154 DEBUG:boto:Token: None 2024-04-16T20:15:29.154 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115 2024-04-16T20:15:29.154 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KPo0qTJ8yDuEfJlKABfrLR1x/JE= 2024-04-16T20:15:29.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KPo0qTJ8yDuEfJlKABfrLR1x/JE=', 'Content-Length': '0'} 2024-04-16T20:15:29.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4554dd927824812-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:15:29.157 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.157 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.157 DEBUG:boto:Method: GET 2024-04-16T20:15:29.157 DEBUG:boto:Path: /yrjefh-54/obj115 2024-04-16T20:15:29.157 DEBUG:boto:Data: 2024-04-16T20:15:29.158 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.158 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.158 DEBUG:boto:Params: {} 2024-04-16T20:15:29.158 DEBUG:boto:Token: None 2024-04-16T20:15:29.158 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115 2024-04-16T20:15:29.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VTFT5iTnGKX6lrAfREozbmAN6Rc= 2024-04-16T20:15:29.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VTFT5iTnGKX6lrAfREozbmAN6Rc=', 'Content-Length': '0'} 2024-04-16T20:15:29.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026188bf992b9ccfe-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.203 DEBUG:boto:path=/yrjefh-54/obj115 2024-04-16T20:15:29.203 DEBUG:boto:auth_path=/yrjefh-54/obj115 2024-04-16T20:15:29.203 DEBUG:boto:Method: GET 2024-04-16T20:15:29.203 DEBUG:boto:Path: /yrjefh-54/obj115 2024-04-16T20:15:29.203 DEBUG:boto:Data: 2024-04-16T20:15:29.203 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.204 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.204 DEBUG:boto:Params: {} 2024-04-16T20:15:29.204 DEBUG:boto:Token: None 2024-04-16T20:15:29.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj115 2024-04-16T20:15:29.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VTFT5iTnGKX6lrAfREozbmAN6Rc= 2024-04-16T20:15:29.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VTFT5iTnGKX6lrAfREozbmAN6Rc=', 'Content-Length': '0'} 2024-04-16T20:15:29.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d7b5edb476777f-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.210 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.210 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.210 DEBUG:boto:path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.210 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.211 DEBUG:boto:Method: GET 2024-04-16T20:15:29.211 DEBUG:boto:Path: /yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.211 DEBUG:boto:Data: 2024-04-16T20:15:29.211 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.211 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.211 DEBUG:boto:Params: {} 2024-04-16T20:15:29.211 DEBUG:boto:Token: None 2024-04-16T20:15:29.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f99456a90fe83f-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.215 DEBUG:boto:b'yrjefh-54obj1151000falseobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.215 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.215 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.216 DEBUG:boto:path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.216 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.216 DEBUG:boto:Method: GET 2024-04-16T20:15:29.216 DEBUG:boto:Path: /yrjefh-54/?prefix=obj115 2024-04-16T20:15:29.216 DEBUG:boto:Data: 2024-04-16T20:15:29.216 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.216 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.216 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.216 DEBUG:boto:Params: {} 2024-04-16T20:15:29.217 DEBUG:boto:Token: None 2024-04-16T20:15:29.217 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008234596aece5a961-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.262 DEBUG:boto:b'yrjefh-54obj1151000falseobj1152024-04-16T20:12:51.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:15:29.263 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.263 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.263 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.263 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.264 DEBUG:boto:Method: GET 2024-04-16T20:15:29.264 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.264 DEBUG:boto:Data: 2024-04-16T20:15:29.264 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.264 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.264 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.264 DEBUG:boto:Params: {} 2024-04-16T20:15:29.264 DEBUG:boto:Token: None 2024-04-16T20:15:29.265 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.265 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY= 2024-04-16T20:15:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY=', 'Content-Length': '0'} 2024-04-16T20:15:29.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8beba6ca08ac743-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.268 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.268 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.268 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.268 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.268 DEBUG:boto:Method: GET 2024-04-16T20:15:29.268 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.268 DEBUG:boto:Data: 2024-04-16T20:15:29.269 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.269 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.269 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.269 DEBUG:boto:Params: {} 2024-04-16T20:15:29.269 DEBUG:boto:Token: None 2024-04-16T20:15:29.269 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.269 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY= 2024-04-16T20:15:29.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY=', 'Content-Length': '0'} 2024-04-16T20:15:29.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc09cfed605a5e4-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.272 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.272 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.273 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.273 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.273 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.273 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.273 DEBUG:boto:Data: 2024-04-16T20:15:29.273 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.273 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.273 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.274 DEBUG:boto:Params: {} 2024-04-16T20:15:29.274 DEBUG:boto:Token: None 2024-04-16T20:15:29.274 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.274 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:65SpHPlXHvDVKrajptau3Oc1s5s= 2024-04-16T20:15:29.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:65SpHPlXHvDVKrajptau3Oc1s5s=', 'Content-Length': '0'} 2024-04-16T20:15:29.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0bbf9b7275cc2b-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.277 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.277 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.277 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.277 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.277 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.278 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.278 DEBUG:boto:Data: 2024-04-16T20:15:29.278 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.278 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.278 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.278 DEBUG:boto:Params: {} 2024-04-16T20:15:29.278 DEBUG:boto:Token: None 2024-04-16T20:15:29.279 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:65SpHPlXHvDVKrajptau3Oc1s5s= 2024-04-16T20:15:29.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:65SpHPlXHvDVKrajptau3Oc1s5s=', 'Content-Length': '0'} 2024-04-16T20:15:29.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000381712c10544eb9e-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:15:29.282 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.282 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.282 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.282 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.282 DEBUG:boto:Method: GET 2024-04-16T20:15:29.282 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.282 DEBUG:boto:Data: 2024-04-16T20:15:29.282 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.283 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.283 DEBUG:boto:Params: {} 2024-04-16T20:15:29.283 DEBUG:boto:Token: None 2024-04-16T20:15:29.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY= 2024-04-16T20:15:29.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY=', 'Content-Length': '0'} 2024-04-16T20:15:29.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035539be132f2ffd-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.286 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.286 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.287 DEBUG:boto:path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.287 DEBUG:boto:auth_path=/yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.287 DEBUG:boto:Method: GET 2024-04-16T20:15:29.287 DEBUG:boto:Path: /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.287 DEBUG:boto:Data: 2024-04-16T20:15:29.287 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.287 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.288 DEBUG:boto:Params: {} 2024-04-16T20:15:29.288 DEBUG:boto:Token: None 2024-04-16T20:15:29.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116?versionId=null 2024-04-16T20:15:29.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY= 2024-04-16T20:15:29.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KVNf98VfDhQCDHrXrUWOUdrA6ZY=', 'Content-Length': '0'} 2024-04-16T20:15:29.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed838bacf883acc5-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.291 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.291 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.291 DEBUG:boto:path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.291 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.291 DEBUG:boto:Method: GET 2024-04-16T20:15:29.292 DEBUG:boto:Path: /yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.292 DEBUG:boto:Data: 2024-04-16T20:15:29.292 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.292 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.292 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.292 DEBUG:boto:Params: {} 2024-04-16T20:15:29.292 DEBUG:boto:Token: None 2024-04-16T20:15:29.292 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.293 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a868972ab3992bb7-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.295 DEBUG:boto:b'yrjefh-54obj1161000falseobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.295 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.296 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.296 DEBUG:boto:path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.296 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.296 DEBUG:boto:Method: GET 2024-04-16T20:15:29.296 DEBUG:boto:Path: /yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.296 DEBUG:boto:Data: 2024-04-16T20:15:29.296 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.297 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.297 DEBUG:boto:Params: {} 2024-04-16T20:15:29.297 DEBUG:boto:Token: None 2024-04-16T20:15:29.297 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.297 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e771943105ada8b3-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.342 DEBUG:boto:b'yrjefh-54obj1161000falseobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.343 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.343 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.343 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.343 DEBUG:boto:Path: /yrjefh-54/obj116 2024-04-16T20:15:29.343 DEBUG:boto:Data: 2024-04-16T20:15:29.343 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.344 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.344 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.344 DEBUG:boto:Params: {} 2024-04-16T20:15:29.344 DEBUG:boto:Token: None 2024-04-16T20:15:29.344 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116 2024-04-16T20:15:29.344 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CSSG6EnR3I8JmcYWxPczuSTh2l8= 2024-04-16T20:15:29.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CSSG6EnR3I8JmcYWxPczuSTh2l8=', 'Content-Length': '0'} 2024-04-16T20:15:29.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22ee99f831ff128-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.347 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.347 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.347 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.347 DEBUG:boto:Path: /yrjefh-54/obj116 2024-04-16T20:15:29.348 DEBUG:boto:Data: 2024-04-16T20:15:29.348 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.348 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.348 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.348 DEBUG:boto:Params: {} 2024-04-16T20:15:29.348 DEBUG:boto:Token: None 2024-04-16T20:15:29.348 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116 2024-04-16T20:15:29.349 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CSSG6EnR3I8JmcYWxPczuSTh2l8= 2024-04-16T20:15:29.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CSSG6EnR3I8JmcYWxPczuSTh2l8=', 'Content-Length': '0'} 2024-04-16T20:15:29.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c22fd4a09ba73e6-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:15:29.351 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.352 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.352 DEBUG:boto:Method: GET 2024-04-16T20:15:29.352 DEBUG:boto:Path: /yrjefh-54/obj116 2024-04-16T20:15:29.352 DEBUG:boto:Data: 2024-04-16T20:15:29.352 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.352 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.352 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.352 DEBUG:boto:Params: {} 2024-04-16T20:15:29.353 DEBUG:boto:Token: None 2024-04-16T20:15:29.353 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116 2024-04-16T20:15:29.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sb8IcUcAWEV2fV0l89DYFLgmE54= 2024-04-16T20:15:29.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sb8IcUcAWEV2fV0l89DYFLgmE54=', 'Content-Length': '0'} 2024-04-16T20:15:29.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d56e85c7434c70-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.398 DEBUG:boto:path=/yrjefh-54/obj116 2024-04-16T20:15:29.399 DEBUG:boto:auth_path=/yrjefh-54/obj116 2024-04-16T20:15:29.399 DEBUG:boto:Method: GET 2024-04-16T20:15:29.399 DEBUG:boto:Path: /yrjefh-54/obj116 2024-04-16T20:15:29.399 DEBUG:boto:Data: 2024-04-16T20:15:29.399 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.399 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.399 DEBUG:boto:Params: {} 2024-04-16T20:15:29.400 DEBUG:boto:Token: None 2024-04-16T20:15:29.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj116 2024-04-16T20:15:29.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sb8IcUcAWEV2fV0l89DYFLgmE54= 2024-04-16T20:15:29.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sb8IcUcAWEV2fV0l89DYFLgmE54=', 'Content-Length': '0'} 2024-04-16T20:15:29.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d15ac077f81ca5-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.403 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.403 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.403 DEBUG:boto:path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.403 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.403 DEBUG:boto:Method: GET 2024-04-16T20:15:29.403 DEBUG:boto:Path: /yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.403 DEBUG:boto:Data: 2024-04-16T20:15:29.403 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.404 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.404 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.404 DEBUG:boto:Params: {} 2024-04-16T20:15:29.404 DEBUG:boto:Token: None 2024-04-16T20:15:29.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007473b74f60426b2c-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.407 DEBUG:boto:b'yrjefh-54obj1161000falseobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.408 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.408 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.408 DEBUG:boto:path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.408 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.408 DEBUG:boto:Method: GET 2024-04-16T20:15:29.408 DEBUG:boto:Path: /yrjefh-54/?prefix=obj116 2024-04-16T20:15:29.408 DEBUG:boto:Data: 2024-04-16T20:15:29.408 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.409 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.409 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.409 DEBUG:boto:Params: {} 2024-04-16T20:15:29.409 DEBUG:boto:Token: None 2024-04-16T20:15:29.409 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0372209c68ae95-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.414 DEBUG:boto:b'yrjefh-54obj1161000falseobj1162024-04-16T20:12:51.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:15:29.415 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.415 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.415 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.415 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.415 DEBUG:boto:Method: GET 2024-04-16T20:15:29.415 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.415 DEBUG:boto:Data: 2024-04-16T20:15:29.415 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.416 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.416 DEBUG:boto:Params: {} 2024-04-16T20:15:29.416 DEBUG:boto:Token: None 2024-04-16T20:15:29.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY= 2024-04-16T20:15:29.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY=', 'Content-Length': '0'} 2024-04-16T20:15:29.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380feeacdbbe41cf-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.419 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.419 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.419 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.419 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.419 DEBUG:boto:Method: GET 2024-04-16T20:15:29.420 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.420 DEBUG:boto:Data: 2024-04-16T20:15:29.420 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.420 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.420 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.420 DEBUG:boto:Params: {} 2024-04-16T20:15:29.420 DEBUG:boto:Token: None 2024-04-16T20:15:29.421 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY= 2024-04-16T20:15:29.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY=', 'Content-Length': '0'} 2024-04-16T20:15:29.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d1fe75cbeb00c7-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.467 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.467 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.467 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.467 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.467 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.467 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.467 DEBUG:boto:Data: 2024-04-16T20:15:29.467 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.468 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.468 DEBUG:boto:Params: {} 2024-04-16T20:15:29.468 DEBUG:boto:Token: None 2024-04-16T20:15:29.468 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jfybT0bKvO/ko12hHbXV7oDxj0E= 2024-04-16T20:15:29.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jfybT0bKvO/ko12hHbXV7oDxj0E=', 'Content-Length': '0'} 2024-04-16T20:15:29.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009114812cee09be5d-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.471 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.471 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.472 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.472 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.472 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.472 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.472 DEBUG:boto:Data: 2024-04-16T20:15:29.472 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.472 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.472 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.473 DEBUG:boto:Params: {} 2024-04-16T20:15:29.473 DEBUG:boto:Token: None 2024-04-16T20:15:29.473 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jfybT0bKvO/ko12hHbXV7oDxj0E= 2024-04-16T20:15:29.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jfybT0bKvO/ko12hHbXV7oDxj0E=', 'Content-Length': '0'} 2024-04-16T20:15:29.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002741bdfdfd3a90fc-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:15:29.476 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.476 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.476 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.476 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.476 DEBUG:boto:Method: GET 2024-04-16T20:15:29.476 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.477 DEBUG:boto:Data: 2024-04-16T20:15:29.477 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.477 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.477 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.477 DEBUG:boto:Params: {} 2024-04-16T20:15:29.477 DEBUG:boto:Token: None 2024-04-16T20:15:29.477 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.477 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY= 2024-04-16T20:15:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY=', 'Content-Length': '0'} 2024-04-16T20:15:29.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d00c93a293534f0-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.522 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.523 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.523 DEBUG:boto:path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.523 DEBUG:boto:auth_path=/yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.523 DEBUG:boto:Method: GET 2024-04-16T20:15:29.523 DEBUG:boto:Path: /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.523 DEBUG:boto:Data: 2024-04-16T20:15:29.523 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.524 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.524 DEBUG:boto:Params: {} 2024-04-16T20:15:29.524 DEBUG:boto:Token: None 2024-04-16T20:15:29.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124?versionId=null 2024-04-16T20:15:29.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY= 2024-04-16T20:15:29.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rIeNHPJogYUGfXpb4nPjOUJZtsY=', 'Content-Length': '0'} 2024-04-16T20:15:29.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097a2bc5c1b702f5-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.527 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.527 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.527 DEBUG:boto:path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.528 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.528 DEBUG:boto:Method: GET 2024-04-16T20:15:29.528 DEBUG:boto:Path: /yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.528 DEBUG:boto:Data: 2024-04-16T20:15:29.528 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.528 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.528 DEBUG:boto:Params: {} 2024-04-16T20:15:29.529 DEBUG:boto:Token: None 2024-04-16T20:15:29.529 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.529 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535c8846727cd3fb-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.532 DEBUG:boto:b'yrjefh-54obj1241000falseobj1242024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.532 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.532 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.532 DEBUG:boto:path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.533 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.533 DEBUG:boto:Method: GET 2024-04-16T20:15:29.533 DEBUG:boto:Path: /yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.533 DEBUG:boto:Data: 2024-04-16T20:15:29.533 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.533 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.533 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.533 DEBUG:boto:Params: {} 2024-04-16T20:15:29.534 DEBUG:boto:Token: None 2024-04-16T20:15:29.534 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.534 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e92bcbfcd0b620-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.538 DEBUG:boto:b'yrjefh-54obj1241000falseobj1242024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.538 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.538 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.539 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.539 DEBUG:boto:Path: /yrjefh-54/obj124 2024-04-16T20:15:29.539 DEBUG:boto:Data: 2024-04-16T20:15:29.539 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.539 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.539 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.539 DEBUG:boto:Params: {} 2024-04-16T20:15:29.539 DEBUG:boto:Token: None 2024-04-16T20:15:29.540 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124 2024-04-16T20:15:29.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fkt9ShAIWP2V0c3uMmbL1Fbpo6M= 2024-04-16T20:15:29.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fkt9ShAIWP2V0c3uMmbL1Fbpo6M=', 'Content-Length': '0'} 2024-04-16T20:15:29.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093df0910e01eb522-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.542 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.543 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.543 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.543 DEBUG:boto:Path: /yrjefh-54/obj124 2024-04-16T20:15:29.543 DEBUG:boto:Data: 2024-04-16T20:15:29.543 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.543 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.543 DEBUG:boto:Params: {} 2024-04-16T20:15:29.544 DEBUG:boto:Token: None 2024-04-16T20:15:29.544 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124 2024-04-16T20:15:29.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fkt9ShAIWP2V0c3uMmbL1Fbpo6M= 2024-04-16T20:15:29.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fkt9ShAIWP2V0c3uMmbL1Fbpo6M=', 'Content-Length': '0'} 2024-04-16T20:15:29.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004247bc9cde41e551-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:15:29.547 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.547 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.547 DEBUG:boto:Method: GET 2024-04-16T20:15:29.547 DEBUG:boto:Path: /yrjefh-54/obj124 2024-04-16T20:15:29.547 DEBUG:boto:Data: 2024-04-16T20:15:29.547 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.547 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.548 DEBUG:boto:Params: {} 2024-04-16T20:15:29.548 DEBUG:boto:Token: None 2024-04-16T20:15:29.548 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124 2024-04-16T20:15:29.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DwJNwrfMwVP8hH9XBPg4NVqMJMM= 2024-04-16T20:15:29.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DwJNwrfMwVP8hH9XBPg4NVqMJMM=', 'Content-Length': '0'} 2024-04-16T20:15:29.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac23f2e0a7afd427-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.551 DEBUG:boto:path=/yrjefh-54/obj124 2024-04-16T20:15:29.551 DEBUG:boto:auth_path=/yrjefh-54/obj124 2024-04-16T20:15:29.551 DEBUG:boto:Method: GET 2024-04-16T20:15:29.552 DEBUG:boto:Path: /yrjefh-54/obj124 2024-04-16T20:15:29.552 DEBUG:boto:Data: 2024-04-16T20:15:29.552 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.552 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.552 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.552 DEBUG:boto:Params: {} 2024-04-16T20:15:29.552 DEBUG:boto:Token: None 2024-04-16T20:15:29.552 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj124 2024-04-16T20:15:29.553 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DwJNwrfMwVP8hH9XBPg4NVqMJMM= 2024-04-16T20:15:29.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DwJNwrfMwVP8hH9XBPg4NVqMJMM=', 'Content-Length': '0'} 2024-04-16T20:15:29.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030976e53893d1f5-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.555 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.555 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.556 DEBUG:boto:path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.556 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.556 DEBUG:boto:Method: GET 2024-04-16T20:15:29.556 DEBUG:boto:Path: /yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.556 DEBUG:boto:Data: 2024-04-16T20:15:29.556 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.556 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.556 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.556 DEBUG:boto:Params: {} 2024-04-16T20:15:29.557 DEBUG:boto:Token: None 2024-04-16T20:15:29.557 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.557 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003be8f7e38831d302-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.602 DEBUG:boto:b'yrjefh-54obj1241000falseobj1242024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.603 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.603 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.603 DEBUG:boto:path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.603 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.603 DEBUG:boto:Method: GET 2024-04-16T20:15:29.604 DEBUG:boto:Path: /yrjefh-54/?prefix=obj124 2024-04-16T20:15:29.604 DEBUG:boto:Data: 2024-04-16T20:15:29.604 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.604 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.604 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.604 DEBUG:boto:Params: {} 2024-04-16T20:15:29.604 DEBUG:boto:Token: None 2024-04-16T20:15:29.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.605 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019d2886c32ab5100-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.609 DEBUG:boto:b'yrjefh-54obj1241000falseobj1242024-04-16T20:12:51.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:15:29.609 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.609 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.609 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.610 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.610 DEBUG:boto:Method: GET 2024-04-16T20:15:29.610 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.610 DEBUG:boto:Data: 2024-04-16T20:15:29.610 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.610 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.610 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.610 DEBUG:boto:Params: {} 2024-04-16T20:15:29.611 DEBUG:boto:Token: None 2024-04-16T20:15:29.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg= 2024-04-16T20:15:29.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg=', 'Content-Length': '0'} 2024-04-16T20:15:29.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fa013604e31c68c-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.614 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.614 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.614 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.614 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.614 DEBUG:boto:Method: GET 2024-04-16T20:15:29.615 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.615 DEBUG:boto:Data: 2024-04-16T20:15:29.615 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.615 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.615 DEBUG:boto:Params: {} 2024-04-16T20:15:29.615 DEBUG:boto:Token: None 2024-04-16T20:15:29.616 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg= 2024-04-16T20:15:29.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg=', 'Content-Length': '0'} 2024-04-16T20:15:29.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000834269fd6d716c4f-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.658 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.659 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.659 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.659 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.659 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.659 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.659 DEBUG:boto:Data: 2024-04-16T20:15:29.659 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.659 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.659 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.659 DEBUG:boto:Params: {} 2024-04-16T20:15:29.660 DEBUG:boto:Token: None 2024-04-16T20:15:29.660 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WikNdyqLeyRYcWDkZkV/EdbMLiY= 2024-04-16T20:15:29.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WikNdyqLeyRYcWDkZkV/EdbMLiY=', 'Content-Length': '0'} 2024-04-16T20:15:29.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006336a311a340d2c-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.662 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.663 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.663 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.663 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.663 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.663 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.663 DEBUG:boto:Data: 2024-04-16T20:15:29.663 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.663 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.663 DEBUG:boto:Params: {} 2024-04-16T20:15:29.664 DEBUG:boto:Token: None 2024-04-16T20:15:29.664 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WikNdyqLeyRYcWDkZkV/EdbMLiY= 2024-04-16T20:15:29.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WikNdyqLeyRYcWDkZkV/EdbMLiY=', 'Content-Length': '0'} 2024-04-16T20:15:29.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a390b50a56c6ea-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:15:29.667 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.667 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.667 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.667 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.667 DEBUG:boto:Method: GET 2024-04-16T20:15:29.667 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.667 DEBUG:boto:Data: 2024-04-16T20:15:29.667 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.667 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.668 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.668 DEBUG:boto:Params: {} 2024-04-16T20:15:29.668 DEBUG:boto:Token: None 2024-04-16T20:15:29.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg= 2024-04-16T20:15:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg=', 'Content-Length': '0'} 2024-04-16T20:15:29.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007739f2859ddc2da0-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.671 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.671 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.671 DEBUG:boto:path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.671 DEBUG:boto:auth_path=/yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.671 DEBUG:boto:Method: GET 2024-04-16T20:15:29.671 DEBUG:boto:Path: /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.671 DEBUG:boto:Data: 2024-04-16T20:15:29.672 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.672 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.672 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.672 DEBUG:boto:Params: {} 2024-04-16T20:15:29.672 DEBUG:boto:Token: None 2024-04-16T20:15:29.672 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145?versionId=null 2024-04-16T20:15:29.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg= 2024-04-16T20:15:29.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Phc8osQqBnToajweEeFoopFclgg=', 'Content-Length': '0'} 2024-04-16T20:15:29.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd7d5a4f2ba9453-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.675 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.675 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.675 DEBUG:boto:path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.675 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.675 DEBUG:boto:Method: GET 2024-04-16T20:15:29.676 DEBUG:boto:Path: /yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.676 DEBUG:boto:Data: 2024-04-16T20:15:29.676 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.676 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.676 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.676 DEBUG:boto:Params: {} 2024-04-16T20:15:29.676 DEBUG:boto:Token: None 2024-04-16T20:15:29.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009974b4729415298b-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.679 DEBUG:boto:b'yrjefh-54obj1451000falseobj1452024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.679 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.679 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.679 DEBUG:boto:path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.680 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.680 DEBUG:boto:Method: GET 2024-04-16T20:15:29.680 DEBUG:boto:Path: /yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.680 DEBUG:boto:Data: 2024-04-16T20:15:29.680 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.680 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.680 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.680 DEBUG:boto:Params: {} 2024-04-16T20:15:29.680 DEBUG:boto:Token: None 2024-04-16T20:15:29.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b4a5af8f01070a-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.684 DEBUG:boto:b'yrjefh-54obj1451000falseobj1452024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.685 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.685 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.685 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.685 DEBUG:boto:Path: /yrjefh-54/obj145 2024-04-16T20:15:29.685 DEBUG:boto:Data: 2024-04-16T20:15:29.685 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.685 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.685 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.685 DEBUG:boto:Params: {} 2024-04-16T20:15:29.686 DEBUG:boto:Token: None 2024-04-16T20:15:29.686 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145 2024-04-16T20:15:29.686 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7PzA2h99RZ8RBfnBa8FxLRY7OWI= 2024-04-16T20:15:29.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7PzA2h99RZ8RBfnBa8FxLRY7OWI=', 'Content-Length': '0'} 2024-04-16T20:15:29.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae71d371ac1cf36e-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.689 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.689 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.689 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.689 DEBUG:boto:Path: /yrjefh-54/obj145 2024-04-16T20:15:29.689 DEBUG:boto:Data: 2024-04-16T20:15:29.689 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.689 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.689 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.689 DEBUG:boto:Params: {} 2024-04-16T20:15:29.690 DEBUG:boto:Token: None 2024-04-16T20:15:29.690 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145 2024-04-16T20:15:29.690 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7PzA2h99RZ8RBfnBa8FxLRY7OWI= 2024-04-16T20:15:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7PzA2h99RZ8RBfnBa8FxLRY7OWI=', 'Content-Length': '0'} 2024-04-16T20:15:29.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0fe2da21ad5c927-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:15:29.693 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.693 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.693 DEBUG:boto:Method: GET 2024-04-16T20:15:29.693 DEBUG:boto:Path: /yrjefh-54/obj145 2024-04-16T20:15:29.693 DEBUG:boto:Data: 2024-04-16T20:15:29.693 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.693 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.693 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.693 DEBUG:boto:Params: {} 2024-04-16T20:15:29.694 DEBUG:boto:Token: None 2024-04-16T20:15:29.694 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145 2024-04-16T20:15:29.694 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IDLFVfTTUeAcY5J+HpUCNRGC5FQ= 2024-04-16T20:15:29.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IDLFVfTTUeAcY5J+HpUCNRGC5FQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f8e27b73da08d5-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.739 DEBUG:boto:path=/yrjefh-54/obj145 2024-04-16T20:15:29.739 DEBUG:boto:auth_path=/yrjefh-54/obj145 2024-04-16T20:15:29.739 DEBUG:boto:Method: GET 2024-04-16T20:15:29.739 DEBUG:boto:Path: /yrjefh-54/obj145 2024-04-16T20:15:29.739 DEBUG:boto:Data: 2024-04-16T20:15:29.739 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.739 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.739 DEBUG:boto:Params: {} 2024-04-16T20:15:29.740 DEBUG:boto:Token: None 2024-04-16T20:15:29.740 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj145 2024-04-16T20:15:29.740 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IDLFVfTTUeAcY5J+HpUCNRGC5FQ= 2024-04-16T20:15:29.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IDLFVfTTUeAcY5J+HpUCNRGC5FQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ed1b09015f70a79-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.742 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.742 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.742 DEBUG:boto:path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.743 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.743 DEBUG:boto:Method: GET 2024-04-16T20:15:29.743 DEBUG:boto:Path: /yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.743 DEBUG:boto:Data: 2024-04-16T20:15:29.743 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.743 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.743 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.743 DEBUG:boto:Params: {} 2024-04-16T20:15:29.743 DEBUG:boto:Token: None 2024-04-16T20:15:29.743 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16a3de3e664f733-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.746 DEBUG:boto:b'yrjefh-54obj1451000falseobj1452024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.746 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.746 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.746 DEBUG:boto:path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.747 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.747 DEBUG:boto:Method: GET 2024-04-16T20:15:29.747 DEBUG:boto:Path: /yrjefh-54/?prefix=obj145 2024-04-16T20:15:29.747 DEBUG:boto:Data: 2024-04-16T20:15:29.747 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.747 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.747 DEBUG:boto:Params: {} 2024-04-16T20:15:29.747 DEBUG:boto:Token: None 2024-04-16T20:15:29.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009656e4d5ec3a0d9c-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.751 DEBUG:boto:b'yrjefh-54obj1451000falseobj1452024-04-16T20:12:51.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:15:29.752 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.752 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.752 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.752 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.752 DEBUG:boto:Method: GET 2024-04-16T20:15:29.752 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.753 DEBUG:boto:Data: 2024-04-16T20:15:29.753 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.753 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.753 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.753 DEBUG:boto:Params: {} 2024-04-16T20:15:29.753 DEBUG:boto:Token: None 2024-04-16T20:15:29.753 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.753 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU= 2024-04-16T20:15:29.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU=', 'Content-Length': '0'} 2024-04-16T20:15:29.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc643097b9ea1fa-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.756 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.756 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.757 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.757 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.757 DEBUG:boto:Method: GET 2024-04-16T20:15:29.757 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.757 DEBUG:boto:Data: 2024-04-16T20:15:29.757 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.757 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.757 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.757 DEBUG:boto:Params: {} 2024-04-16T20:15:29.757 DEBUG:boto:Token: None 2024-04-16T20:15:29.758 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.758 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU= 2024-04-16T20:15:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU=', 'Content-Length': '0'} 2024-04-16T20:15:29.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073bc7e70e9321e9f-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.760 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.760 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.761 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.761 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.761 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.761 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.761 DEBUG:boto:Data: 2024-04-16T20:15:29.761 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.761 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.761 DEBUG:boto:Params: {} 2024-04-16T20:15:29.761 DEBUG:boto:Token: None 2024-04-16T20:15:29.762 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bCmEkyzfuqliGut8DKcSTwzM7oQ= 2024-04-16T20:15:29.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bCmEkyzfuqliGut8DKcSTwzM7oQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ba12d4336d9feb-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.765 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.765 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.765 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.765 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.765 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.765 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.765 DEBUG:boto:Data: 2024-04-16T20:15:29.765 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.766 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.766 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.766 DEBUG:boto:Params: {} 2024-04-16T20:15:29.766 DEBUG:boto:Token: None 2024-04-16T20:15:29.766 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.766 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bCmEkyzfuqliGut8DKcSTwzM7oQ= 2024-04-16T20:15:29.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bCmEkyzfuqliGut8DKcSTwzM7oQ=', 'Content-Length': '0'} 2024-04-16T20:15:29.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6c6da79d751ace-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:15:29.769 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.769 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.769 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.769 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.769 DEBUG:boto:Method: GET 2024-04-16T20:15:29.769 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.769 DEBUG:boto:Data: 2024-04-16T20:15:29.769 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.769 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.769 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.770 DEBUG:boto:Params: {} 2024-04-16T20:15:29.770 DEBUG:boto:Token: None 2024-04-16T20:15:29.770 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.770 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU= 2024-04-16T20:15:29.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU=', 'Content-Length': '0'} 2024-04-16T20:15:29.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48b52aa5b969fb3-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.773 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.773 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.773 DEBUG:boto:path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.773 DEBUG:boto:auth_path=/yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.773 DEBUG:boto:Method: GET 2024-04-16T20:15:29.773 DEBUG:boto:Path: /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.774 DEBUG:boto:Data: 2024-04-16T20:15:29.774 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.774 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.774 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.774 DEBUG:boto:Params: {} 2024-04-16T20:15:29.774 DEBUG:boto:Token: None 2024-04-16T20:15:29.774 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148?versionId=null 2024-04-16T20:15:29.774 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU= 2024-04-16T20:15:29.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s5v/I1vl+NOZvVaCNfAdqDY/QlU=', 'Content-Length': '0'} 2024-04-16T20:15:29.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009548d0a3e9d2a013-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.777 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.777 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.777 DEBUG:boto:path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.777 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.777 DEBUG:boto:Method: GET 2024-04-16T20:15:29.778 DEBUG:boto:Path: /yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.778 DEBUG:boto:Data: 2024-04-16T20:15:29.778 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.778 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.778 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.778 DEBUG:boto:Params: {} 2024-04-16T20:15:29.778 DEBUG:boto:Token: None 2024-04-16T20:15:29.778 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.778 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5b5d1d5f10664e-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.781 DEBUG:boto:b'yrjefh-54obj1481000falseobj1482024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.781 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.782 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.782 DEBUG:boto:path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.782 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.782 DEBUG:boto:Method: GET 2024-04-16T20:15:29.782 DEBUG:boto:Path: /yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.782 DEBUG:boto:Data: 2024-04-16T20:15:29.782 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.782 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.782 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.782 DEBUG:boto:Params: {} 2024-04-16T20:15:29.783 DEBUG:boto:Token: None 2024-04-16T20:15:29.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f55acb1465d890-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.786 DEBUG:boto:b'yrjefh-54obj1481000falseobj1482024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.787 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.787 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.787 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.787 DEBUG:boto:Path: /yrjefh-54/obj148 2024-04-16T20:15:29.787 DEBUG:boto:Data: 2024-04-16T20:15:29.787 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.787 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.788 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.788 DEBUG:boto:Params: {} 2024-04-16T20:15:29.788 DEBUG:boto:Token: None 2024-04-16T20:15:29.788 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148 2024-04-16T20:15:29.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l3nalLbZKXjNO4xenWQGemVEUjo= 2024-04-16T20:15:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l3nalLbZKXjNO4xenWQGemVEUjo=', 'Content-Length': '0'} 2024-04-16T20:15:29.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04ab76f3179951a-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.790 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.791 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.791 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.791 DEBUG:boto:Path: /yrjefh-54/obj148 2024-04-16T20:15:29.791 DEBUG:boto:Data: 2024-04-16T20:15:29.791 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.791 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.791 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.791 DEBUG:boto:Params: {} 2024-04-16T20:15:29.791 DEBUG:boto:Token: None 2024-04-16T20:15:29.792 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148 2024-04-16T20:15:29.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l3nalLbZKXjNO4xenWQGemVEUjo= 2024-04-16T20:15:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l3nalLbZKXjNO4xenWQGemVEUjo=', 'Content-Length': '0'} 2024-04-16T20:15:29.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb53f0d7ba8545d7-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:15:29.794 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.794 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.794 DEBUG:boto:Method: GET 2024-04-16T20:15:29.794 DEBUG:boto:Path: /yrjefh-54/obj148 2024-04-16T20:15:29.794 DEBUG:boto:Data: 2024-04-16T20:15:29.794 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.795 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.795 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.795 DEBUG:boto:Params: {} 2024-04-16T20:15:29.795 DEBUG:boto:Token: None 2024-04-16T20:15:29.795 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148 2024-04-16T20:15:29.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yK/usu70UNzotuR/Z3an/QoWMMg= 2024-04-16T20:15:29.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yK/usu70UNzotuR/Z3an/QoWMMg=', 'Content-Length': '0'} 2024-04-16T20:15:29.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9dbf9166a0fe38c-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.798 DEBUG:boto:path=/yrjefh-54/obj148 2024-04-16T20:15:29.798 DEBUG:boto:auth_path=/yrjefh-54/obj148 2024-04-16T20:15:29.798 DEBUG:boto:Method: GET 2024-04-16T20:15:29.798 DEBUG:boto:Path: /yrjefh-54/obj148 2024-04-16T20:15:29.798 DEBUG:boto:Data: 2024-04-16T20:15:29.798 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.798 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.799 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.799 DEBUG:boto:Params: {} 2024-04-16T20:15:29.799 DEBUG:boto:Token: None 2024-04-16T20:15:29.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj148 2024-04-16T20:15:29.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yK/usu70UNzotuR/Z3an/QoWMMg= 2024-04-16T20:15:29.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yK/usu70UNzotuR/Z3an/QoWMMg=', 'Content-Length': '0'} 2024-04-16T20:15:29.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a92cac9ec341bc08-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.802 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.802 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.802 DEBUG:boto:path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.802 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.802 DEBUG:boto:Method: GET 2024-04-16T20:15:29.802 DEBUG:boto:Path: /yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.802 DEBUG:boto:Data: 2024-04-16T20:15:29.802 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.802 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.802 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.803 DEBUG:boto:Params: {} 2024-04-16T20:15:29.803 DEBUG:boto:Token: None 2024-04-16T20:15:29.803 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.803 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea72992b69854bb1-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.805 DEBUG:boto:b'yrjefh-54obj1481000falseobj1482024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.806 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.806 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.806 DEBUG:boto:path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.806 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.806 DEBUG:boto:Method: GET 2024-04-16T20:15:29.806 DEBUG:boto:Path: /yrjefh-54/?prefix=obj148 2024-04-16T20:15:29.806 DEBUG:boto:Data: 2024-04-16T20:15:29.807 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.807 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.807 DEBUG:boto:Params: {} 2024-04-16T20:15:29.807 DEBUG:boto:Token: None 2024-04-16T20:15:29.807 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.807 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e08f31522fa531d-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.811 DEBUG:boto:b'yrjefh-54obj1481000falseobj1482024-04-16T20:12:52.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:15:29.811 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.811 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.811 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.812 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.812 DEBUG:boto:Method: GET 2024-04-16T20:15:29.812 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.812 DEBUG:boto:Data: 2024-04-16T20:15:29.812 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.812 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.812 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.812 DEBUG:boto:Params: {} 2024-04-16T20:15:29.812 DEBUG:boto:Token: None 2024-04-16T20:15:29.812 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.813 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q= 2024-04-16T20:15:29.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q=', 'Content-Length': '0'} 2024-04-16T20:15:29.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c98ba170b5fc9b3d-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.815 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.815 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.816 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.816 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.816 DEBUG:boto:Method: GET 2024-04-16T20:15:29.816 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.816 DEBUG:boto:Data: 2024-04-16T20:15:29.816 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.816 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.816 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.816 DEBUG:boto:Params: {} 2024-04-16T20:15:29.816 DEBUG:boto:Token: None 2024-04-16T20:15:29.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.817 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q= 2024-04-16T20:15:29.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q=', 'Content-Length': '0'} 2024-04-16T20:15:29.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c517f85c3d2888f-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.866 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.866 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.867 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.867 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.867 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.867 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.867 DEBUG:boto:Data: 2024-04-16T20:15:29.867 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.867 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.867 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.867 DEBUG:boto:Params: {} 2024-04-16T20:15:29.867 DEBUG:boto:Token: None 2024-04-16T20:15:29.867 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.868 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HlGYzOeManzihAfz+JVAOj+R3lM= 2024-04-16T20:15:29.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HlGYzOeManzihAfz+JVAOj+R3lM=', 'Content-Length': '0'} 2024-04-16T20:15:29.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ca61f632a3e954-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.870 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.870 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.870 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.870 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.870 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.871 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.871 DEBUG:boto:Data: 2024-04-16T20:15:29.871 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.871 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.871 DEBUG:boto:Params: {} 2024-04-16T20:15:29.871 DEBUG:boto:Token: None 2024-04-16T20:15:29.871 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HlGYzOeManzihAfz+JVAOj+R3lM= 2024-04-16T20:15:29.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HlGYzOeManzihAfz+JVAOj+R3lM=', 'Content-Length': '0'} 2024-04-16T20:15:29.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c92eda3522141a-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:15:29.874 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.874 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.874 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.874 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.875 DEBUG:boto:Method: GET 2024-04-16T20:15:29.875 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.875 DEBUG:boto:Data: 2024-04-16T20:15:29.875 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.875 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.875 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.875 DEBUG:boto:Params: {} 2024-04-16T20:15:29.875 DEBUG:boto:Token: None 2024-04-16T20:15:29.875 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q= 2024-04-16T20:15:29.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q=', 'Content-Length': '0'} 2024-04-16T20:15:29.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d54dc444740d42a-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.878 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.878 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.878 DEBUG:boto:path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.878 DEBUG:boto:auth_path=/yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.878 DEBUG:boto:Method: GET 2024-04-16T20:15:29.878 DEBUG:boto:Path: /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.878 DEBUG:boto:Data: 2024-04-16T20:15:29.879 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.879 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.879 DEBUG:boto:Params: {} 2024-04-16T20:15:29.879 DEBUG:boto:Token: None 2024-04-16T20:15:29.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155?versionId=null 2024-04-16T20:15:29.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q= 2024-04-16T20:15:29.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:44bElD+qcYheoyxuvWV6G3p2X0Q=', 'Content-Length': '0'} 2024-04-16T20:15:29.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df50db4f6d9c8908-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.882 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.882 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.882 DEBUG:boto:path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.882 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.882 DEBUG:boto:Method: GET 2024-04-16T20:15:29.882 DEBUG:boto:Path: /yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.882 DEBUG:boto:Data: 2024-04-16T20:15:29.882 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.882 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.882 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.883 DEBUG:boto:Params: {} 2024-04-16T20:15:29.883 DEBUG:boto:Token: None 2024-04-16T20:15:29.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070af43edc26e28a0-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.885 DEBUG:boto:b'yrjefh-54obj1551000falseobj1552024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.886 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.886 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.886 DEBUG:boto:path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.886 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.886 DEBUG:boto:Method: GET 2024-04-16T20:15:29.886 DEBUG:boto:Path: /yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.886 DEBUG:boto:Data: 2024-04-16T20:15:29.886 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.886 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.886 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.887 DEBUG:boto:Params: {} 2024-04-16T20:15:29.887 DEBUG:boto:Token: None 2024-04-16T20:15:29.887 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017167a9fad3fab95-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.891 DEBUG:boto:b'yrjefh-54obj1551000falseobj1552024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.891 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.891 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.891 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.891 DEBUG:boto:Path: /yrjefh-54/obj155 2024-04-16T20:15:29.891 DEBUG:boto:Data: 2024-04-16T20:15:29.891 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.892 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.892 DEBUG:boto:Params: {} 2024-04-16T20:15:29.892 DEBUG:boto:Token: None 2024-04-16T20:15:29.892 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155 2024-04-16T20:15:29.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ZzJ+0P47XxcX3mfVQm51Teola4= 2024-04-16T20:15:29.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ZzJ+0P47XxcX3mfVQm51Teola4=', 'Content-Length': '0'} 2024-04-16T20:15:29.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0607a305aa6153-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.894 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.895 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.895 DEBUG:boto:Method: HEAD 2024-04-16T20:15:29.895 DEBUG:boto:Path: /yrjefh-54/obj155 2024-04-16T20:15:29.895 DEBUG:boto:Data: 2024-04-16T20:15:29.895 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.895 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.895 DEBUG:boto:Params: {} 2024-04-16T20:15:29.895 DEBUG:boto:Token: None 2024-04-16T20:15:29.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155 2024-04-16T20:15:29.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3ZzJ+0P47XxcX3mfVQm51Teola4= 2024-04-16T20:15:29.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ZzJ+0P47XxcX3mfVQm51Teola4=', 'Content-Length': '0'} 2024-04-16T20:15:29.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c9d8fe14667456-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:15:29.898 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.898 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.898 DEBUG:boto:Method: GET 2024-04-16T20:15:29.898 DEBUG:boto:Path: /yrjefh-54/obj155 2024-04-16T20:15:29.898 DEBUG:boto:Data: 2024-04-16T20:15:29.898 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.898 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.898 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.899 DEBUG:boto:Params: {} 2024-04-16T20:15:29.899 DEBUG:boto:Token: None 2024-04-16T20:15:29.899 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155 2024-04-16T20:15:29.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:seQNZuLZHpqBjOZ3G6jcqMKkAGs= 2024-04-16T20:15:29.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:seQNZuLZHpqBjOZ3G6jcqMKkAGs=', 'Content-Length': '0'} 2024-04-16T20:15:29.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352fb246c8049760-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.942 DEBUG:boto:path=/yrjefh-54/obj155 2024-04-16T20:15:29.942 DEBUG:boto:auth_path=/yrjefh-54/obj155 2024-04-16T20:15:29.943 DEBUG:boto:Method: GET 2024-04-16T20:15:29.943 DEBUG:boto:Path: /yrjefh-54/obj155 2024-04-16T20:15:29.943 DEBUG:boto:Data: 2024-04-16T20:15:29.943 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.943 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.943 DEBUG:boto:Params: {} 2024-04-16T20:15:29.943 DEBUG:boto:Token: None 2024-04-16T20:15:29.943 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj155 2024-04-16T20:15:29.943 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:seQNZuLZHpqBjOZ3G6jcqMKkAGs= 2024-04-16T20:15:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:seQNZuLZHpqBjOZ3G6jcqMKkAGs=', 'Content-Length': '0'} 2024-04-16T20:15:29.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000badf52a34bda5aa2-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.946 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.946 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.946 DEBUG:boto:path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.946 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.946 DEBUG:boto:Method: GET 2024-04-16T20:15:29.946 DEBUG:boto:Path: /yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.946 DEBUG:boto:Data: 2024-04-16T20:15:29.946 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.947 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.947 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.947 DEBUG:boto:Params: {} 2024-04-16T20:15:29.947 DEBUG:boto:Token: None 2024-04-16T20:15:29.947 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fbad792c854f8e0-00661edc61-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.949 DEBUG:boto:b'yrjefh-54obj1551000falseobj1552024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.950 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:29.950 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:29.950 DEBUG:boto:path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.950 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.950 DEBUG:boto:Method: GET 2024-04-16T20:15:29.950 DEBUG:boto:Path: /yrjefh-54/?prefix=obj155 2024-04-16T20:15:29.950 DEBUG:boto:Data: 2024-04-16T20:15:29.950 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.950 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.951 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.951 DEBUG:boto:Params: {} 2024-04-16T20:15:29.951 DEBUG:boto:Token: None 2024-04-16T20:15:29.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/ 2024-04-16T20:15:29.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8= 2024-04-16T20:15:29.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:O7QjTNoytaum9MgMP8aqkoL34s8=', 'Content-Length': '0'} 2024-04-16T20:15:29.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee103db1204d146-00661edc61-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.954 DEBUG:boto:b'yrjefh-54obj1551000falseobj1552024-04-16T20:12:51.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:29.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:15:29.955 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:29.955 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:29.955 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.955 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.955 DEBUG:boto:Method: GET 2024-04-16T20:15:29.955 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.955 DEBUG:boto:Data: 2024-04-16T20:15:29.955 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.956 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:29.956 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.956 DEBUG:boto:Params: {} 2024-04-16T20:15:29.956 DEBUG:boto:Token: None 2024-04-16T20:15:29.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8zI2PBnrixK+7aiSDpbDXkqcP4g= 2024-04-16T20:15:29.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8zI2PBnrixK+7aiSDpbDXkqcP4g=', 'Content-Length': '0'} 2024-04-16T20:15:29.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019feaffcd7adb272-00661edc61-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:29.959 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:29.959 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:29.959 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.959 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.959 DEBUG:boto:Method: GET 2024-04-16T20:15:29.959 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.959 DEBUG:boto:Data: 2024-04-16T20:15:29.959 DEBUG:boto:Headers: {} 2024-04-16T20:15:29.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:29.960 DEBUG:boto:Port: 8000 2024-04-16T20:15:29.960 DEBUG:boto:Params: {} 2024-04-16T20:15:29.960 DEBUG:boto:Token: None 2024-04-16T20:15:29.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:29 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:29.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8zI2PBnrixK+7aiSDpbDXkqcP4g= 2024-04-16T20:15:29.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8zI2PBnrixK+7aiSDpbDXkqcP4g=', 'Content-Length': '0'} 2024-04-16T20:15:29.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b94db1c45275a17-00661edc61-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.006 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.006 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.007 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.007 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.007 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.007 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.007 DEBUG:boto:Data: 2024-04-16T20:15:30.007 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.007 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.007 DEBUG:boto:Params: {} 2024-04-16T20:15:30.007 DEBUG:boto:Token: None 2024-04-16T20:15:30.007 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QBx2LTOL6OuQz64wWEpF7bTVcA8= 2024-04-16T20:15:30.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QBx2LTOL6OuQz64wWEpF7bTVcA8=', 'Content-Length': '0'} 2024-04-16T20:15:30.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2c133568935214-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.010 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.010 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.010 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.010 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.010 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.010 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.011 DEBUG:boto:Data: 2024-04-16T20:15:30.011 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.011 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.011 DEBUG:boto:Params: {} 2024-04-16T20:15:30.011 DEBUG:boto:Token: None 2024-04-16T20:15:30.011 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QBx2LTOL6OuQz64wWEpF7bTVcA8= 2024-04-16T20:15:30.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QBx2LTOL6OuQz64wWEpF7bTVcA8=', 'Content-Length': '0'} 2024-04-16T20:15:30.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccf98164ed848dd-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:15:30.014 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.014 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.014 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.014 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.014 DEBUG:boto:Method: GET 2024-04-16T20:15:30.014 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.014 DEBUG:boto:Data: 2024-04-16T20:15:30.015 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.015 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.015 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.015 DEBUG:boto:Params: {} 2024-04-16T20:15:30.015 DEBUG:boto:Token: None 2024-04-16T20:15:30.015 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.015 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9kDpyy34lubGNkBMnCPzxqaSAMs= 2024-04-16T20:15:30.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9kDpyy34lubGNkBMnCPzxqaSAMs=', 'Content-Length': '0'} 2024-04-16T20:15:30.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0d1d7665de883a-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.017 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.017 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.018 DEBUG:boto:path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.018 DEBUG:boto:auth_path=/yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.018 DEBUG:boto:Method: GET 2024-04-16T20:15:30.018 DEBUG:boto:Path: /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.018 DEBUG:boto:Data: 2024-04-16T20:15:30.018 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.018 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.018 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.018 DEBUG:boto:Params: {} 2024-04-16T20:15:30.018 DEBUG:boto:Token: None 2024-04-16T20:15:30.018 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17?versionId=null 2024-04-16T20:15:30.019 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9kDpyy34lubGNkBMnCPzxqaSAMs= 2024-04-16T20:15:30.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9kDpyy34lubGNkBMnCPzxqaSAMs=', 'Content-Length': '0'} 2024-04-16T20:15:30.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007275b9975401e9f1-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.021 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.021 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.021 DEBUG:boto:path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.021 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.021 DEBUG:boto:Method: GET 2024-04-16T20:15:30.021 DEBUG:boto:Path: /yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.021 DEBUG:boto:Data: 2024-04-16T20:15:30.021 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.021 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.021 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.022 DEBUG:boto:Params: {} 2024-04-16T20:15:30.022 DEBUG:boto:Token: None 2024-04-16T20:15:30.022 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.022 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000133bb2631b8a8efe-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.025 DEBUG:boto:b'yrjefh-54obj171000falseobj172024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.025 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.025 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.025 DEBUG:boto:path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.025 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.025 DEBUG:boto:Method: GET 2024-04-16T20:15:30.025 DEBUG:boto:Path: /yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.025 DEBUG:boto:Data: 2024-04-16T20:15:30.025 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.026 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.026 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.026 DEBUG:boto:Params: {} 2024-04-16T20:15:30.026 DEBUG:boto:Token: None 2024-04-16T20:15:30.026 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.026 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116817c94c435040-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.030 DEBUG:boto:b'yrjefh-54obj171000falseobj172024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.030 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.030 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.030 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.030 DEBUG:boto:Path: /yrjefh-54/obj17 2024-04-16T20:15:30.030 DEBUG:boto:Data: 2024-04-16T20:15:30.031 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.031 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.031 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.031 DEBUG:boto:Params: {} 2024-04-16T20:15:30.031 DEBUG:boto:Token: None 2024-04-16T20:15:30.031 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17 2024-04-16T20:15:30.031 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RGJfGQV9NEmNxGSea1DydwD89p4= 2024-04-16T20:15:30.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RGJfGQV9NEmNxGSea1DydwD89p4=', 'Content-Length': '0'} 2024-04-16T20:15:30.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cba81b99e6fe34-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.033 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.034 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.034 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.034 DEBUG:boto:Path: /yrjefh-54/obj17 2024-04-16T20:15:30.034 DEBUG:boto:Data: 2024-04-16T20:15:30.034 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.034 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.034 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.034 DEBUG:boto:Params: {} 2024-04-16T20:15:30.034 DEBUG:boto:Token: None 2024-04-16T20:15:30.034 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17 2024-04-16T20:15:30.035 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RGJfGQV9NEmNxGSea1DydwD89p4= 2024-04-16T20:15:30.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RGJfGQV9NEmNxGSea1DydwD89p4=', 'Content-Length': '0'} 2024-04-16T20:15:30.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007628e2c213dc9385-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:15:30.037 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.037 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.037 DEBUG:boto:Method: GET 2024-04-16T20:15:30.037 DEBUG:boto:Path: /yrjefh-54/obj17 2024-04-16T20:15:30.037 DEBUG:boto:Data: 2024-04-16T20:15:30.038 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.038 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.038 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.038 DEBUG:boto:Params: {} 2024-04-16T20:15:30.038 DEBUG:boto:Token: None 2024-04-16T20:15:30.038 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17 2024-04-16T20:15:30.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SvemNkczsreMVocH/Lnr0ARNC8Y= 2024-04-16T20:15:30.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SvemNkczsreMVocH/Lnr0ARNC8Y=', 'Content-Length': '0'} 2024-04-16T20:15:30.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010fc97bb2aa8e020-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.040 DEBUG:boto:path=/yrjefh-54/obj17 2024-04-16T20:15:30.041 DEBUG:boto:auth_path=/yrjefh-54/obj17 2024-04-16T20:15:30.041 DEBUG:boto:Method: GET 2024-04-16T20:15:30.041 DEBUG:boto:Path: /yrjefh-54/obj17 2024-04-16T20:15:30.041 DEBUG:boto:Data: 2024-04-16T20:15:30.041 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.041 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.041 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.041 DEBUG:boto:Params: {} 2024-04-16T20:15:30.041 DEBUG:boto:Token: None 2024-04-16T20:15:30.041 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj17 2024-04-16T20:15:30.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SvemNkczsreMVocH/Lnr0ARNC8Y= 2024-04-16T20:15:30.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SvemNkczsreMVocH/Lnr0ARNC8Y=', 'Content-Length': '0'} 2024-04-16T20:15:30.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6445f082c6fced-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.044 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.044 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.044 DEBUG:boto:path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.044 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.044 DEBUG:boto:Method: GET 2024-04-16T20:15:30.044 DEBUG:boto:Path: /yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.044 DEBUG:boto:Data: 2024-04-16T20:15:30.044 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.044 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.044 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.044 DEBUG:boto:Params: {} 2024-04-16T20:15:30.045 DEBUG:boto:Token: None 2024-04-16T20:15:30.045 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.045 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5b588d74b824a4f-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.047 DEBUG:boto:b'yrjefh-54obj171000falseobj172024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.047 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.047 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.048 DEBUG:boto:path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.048 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.048 DEBUG:boto:Method: GET 2024-04-16T20:15:30.048 DEBUG:boto:Path: /yrjefh-54/?prefix=obj17 2024-04-16T20:15:30.048 DEBUG:boto:Data: 2024-04-16T20:15:30.048 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.048 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.048 DEBUG:boto:Params: {} 2024-04-16T20:15:30.048 DEBUG:boto:Token: None 2024-04-16T20:15:30.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f24370928dcaf1-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.052 DEBUG:boto:b'yrjefh-54obj171000falseobj172024-04-16T20:12:50.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:15:30.053 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.053 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.053 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.053 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.053 DEBUG:boto:Method: GET 2024-04-16T20:15:30.053 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.053 DEBUG:boto:Data: 2024-04-16T20:15:30.053 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.053 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.053 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.053 DEBUG:boto:Params: {} 2024-04-16T20:15:30.054 DEBUG:boto:Token: None 2024-04-16T20:15:30.054 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.054 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg= 2024-04-16T20:15:30.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg=', 'Content-Length': '0'} 2024-04-16T20:15:30.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c571b50a4285cde7-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.056 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.057 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.057 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.057 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.057 DEBUG:boto:Method: GET 2024-04-16T20:15:30.057 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.057 DEBUG:boto:Data: 2024-04-16T20:15:30.057 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.057 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.057 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.057 DEBUG:boto:Params: {} 2024-04-16T20:15:30.057 DEBUG:boto:Token: None 2024-04-16T20:15:30.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg= 2024-04-16T20:15:30.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg=', 'Content-Length': '0'} 2024-04-16T20:15:30.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e8f24fb354dd49b-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.102 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.102 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.102 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.103 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.103 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.103 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.103 DEBUG:boto:Data: 2024-04-16T20:15:30.103 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.103 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.103 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.103 DEBUG:boto:Params: {} 2024-04-16T20:15:30.103 DEBUG:boto:Token: None 2024-04-16T20:15:30.103 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3WNWEG2KNr0lTao/rfMxBEllgE= 2024-04-16T20:15:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3WNWEG2KNr0lTao/rfMxBEllgE=', 'Content-Length': '0'} 2024-04-16T20:15:30.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabf48225d3854cf-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.105 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.105 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.105 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.106 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.106 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.106 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.106 DEBUG:boto:Data: 2024-04-16T20:15:30.106 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.106 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.106 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.106 DEBUG:boto:Params: {} 2024-04-16T20:15:30.106 DEBUG:boto:Token: None 2024-04-16T20:15:30.106 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J3WNWEG2KNr0lTao/rfMxBEllgE= 2024-04-16T20:15:30.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J3WNWEG2KNr0lTao/rfMxBEllgE=', 'Content-Length': '0'} 2024-04-16T20:15:30.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032636498ed361635-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:15:30.109 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.109 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.109 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.109 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.109 DEBUG:boto:Method: GET 2024-04-16T20:15:30.109 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.109 DEBUG:boto:Data: 2024-04-16T20:15:30.109 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.109 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.109 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.109 DEBUG:boto:Params: {} 2024-04-16T20:15:30.109 DEBUG:boto:Token: None 2024-04-16T20:15:30.110 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.110 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg= 2024-04-16T20:15:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg=', 'Content-Length': '0'} 2024-04-16T20:15:30.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c96ae8bcee035cb9-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.112 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.112 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.112 DEBUG:boto:path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.112 DEBUG:boto:auth_path=/yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.112 DEBUG:boto:Method: GET 2024-04-16T20:15:30.112 DEBUG:boto:Path: /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.112 DEBUG:boto:Data: 2024-04-16T20:15:30.112 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.113 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.113 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.113 DEBUG:boto:Params: {} 2024-04-16T20:15:30.113 DEBUG:boto:Token: None 2024-04-16T20:15:30.113 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185?versionId=null 2024-04-16T20:15:30.113 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg= 2024-04-16T20:15:30.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LOdOxwM77DVWhEWWAN9Pt/lttlg=', 'Content-Length': '0'} 2024-04-16T20:15:30.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e988a8abefd48766-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.115 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.116 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.116 DEBUG:boto:path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.116 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.116 DEBUG:boto:Method: GET 2024-04-16T20:15:30.116 DEBUG:boto:Path: /yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.116 DEBUG:boto:Data: 2024-04-16T20:15:30.116 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.116 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.116 DEBUG:boto:Params: {} 2024-04-16T20:15:30.116 DEBUG:boto:Token: None 2024-04-16T20:15:30.117 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673a4e76ad0303e1-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.119 DEBUG:boto:b'yrjefh-54obj1851000falseobj1852024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.120 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.120 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.120 DEBUG:boto:path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.120 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.120 DEBUG:boto:Method: GET 2024-04-16T20:15:30.120 DEBUG:boto:Path: /yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.120 DEBUG:boto:Data: 2024-04-16T20:15:30.120 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.120 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.120 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.120 DEBUG:boto:Params: {} 2024-04-16T20:15:30.121 DEBUG:boto:Token: None 2024-04-16T20:15:30.121 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.121 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef522cc63ad2334-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.124 DEBUG:boto:b'yrjefh-54obj1851000falseobj1852024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.125 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.125 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.125 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.125 DEBUG:boto:Path: /yrjefh-54/obj185 2024-04-16T20:15:30.125 DEBUG:boto:Data: 2024-04-16T20:15:30.125 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.125 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.125 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.126 DEBUG:boto:Params: {} 2024-04-16T20:15:30.126 DEBUG:boto:Token: None 2024-04-16T20:15:30.126 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185 2024-04-16T20:15:30.126 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rjv7IHTqi89n2nNnW4xirKE4L/I= 2024-04-16T20:15:30.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rjv7IHTqi89n2nNnW4xirKE4L/I=', 'Content-Length': '0'} 2024-04-16T20:15:30.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007752f387709a7e4e-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.128 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.128 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.128 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.128 DEBUG:boto:Path: /yrjefh-54/obj185 2024-04-16T20:15:30.128 DEBUG:boto:Data: 2024-04-16T20:15:30.129 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.129 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.129 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.129 DEBUG:boto:Params: {} 2024-04-16T20:15:30.129 DEBUG:boto:Token: None 2024-04-16T20:15:30.129 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185 2024-04-16T20:15:30.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rjv7IHTqi89n2nNnW4xirKE4L/I= 2024-04-16T20:15:30.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rjv7IHTqi89n2nNnW4xirKE4L/I=', 'Content-Length': '0'} 2024-04-16T20:15:30.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c9b3f93966b36e7-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:15:30.131 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.132 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.132 DEBUG:boto:Method: GET 2024-04-16T20:15:30.132 DEBUG:boto:Path: /yrjefh-54/obj185 2024-04-16T20:15:30.132 DEBUG:boto:Data: 2024-04-16T20:15:30.132 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.132 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.132 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.132 DEBUG:boto:Params: {} 2024-04-16T20:15:30.132 DEBUG:boto:Token: None 2024-04-16T20:15:30.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185 2024-04-16T20:15:30.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jbPsRAUDvc6GlE9X4LPbfmbWmbQ= 2024-04-16T20:15:30.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jbPsRAUDvc6GlE9X4LPbfmbWmbQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114425785cdfd1b1-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.178 DEBUG:boto:path=/yrjefh-54/obj185 2024-04-16T20:15:30.178 DEBUG:boto:auth_path=/yrjefh-54/obj185 2024-04-16T20:15:30.179 DEBUG:boto:Method: GET 2024-04-16T20:15:30.179 DEBUG:boto:Path: /yrjefh-54/obj185 2024-04-16T20:15:30.179 DEBUG:boto:Data: 2024-04-16T20:15:30.179 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.179 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.179 DEBUG:boto:Params: {} 2024-04-16T20:15:30.179 DEBUG:boto:Token: None 2024-04-16T20:15:30.179 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj185 2024-04-16T20:15:30.179 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jbPsRAUDvc6GlE9X4LPbfmbWmbQ= 2024-04-16T20:15:30.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jbPsRAUDvc6GlE9X4LPbfmbWmbQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f12fd4ff31d5d83d-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.182 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.182 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.182 DEBUG:boto:path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.182 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.182 DEBUG:boto:Method: GET 2024-04-16T20:15:30.182 DEBUG:boto:Path: /yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.182 DEBUG:boto:Data: 2024-04-16T20:15:30.182 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.182 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.182 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.183 DEBUG:boto:Params: {} 2024-04-16T20:15:30.183 DEBUG:boto:Token: None 2024-04-16T20:15:30.183 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081c9f77cde089028-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.185 DEBUG:boto:b'yrjefh-54obj1851000falseobj1852024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.186 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.186 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.186 DEBUG:boto:path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.186 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.186 DEBUG:boto:Method: GET 2024-04-16T20:15:30.186 DEBUG:boto:Path: /yrjefh-54/?prefix=obj185 2024-04-16T20:15:30.186 DEBUG:boto:Data: 2024-04-16T20:15:30.186 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.186 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.187 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.187 DEBUG:boto:Params: {} 2024-04-16T20:15:30.187 DEBUG:boto:Token: None 2024-04-16T20:15:30.187 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.187 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f236a669fba9d9-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.190 DEBUG:boto:b'yrjefh-54obj1851000falseobj1852024-04-16T20:12:52.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:15:30.191 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.191 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.191 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.191 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.191 DEBUG:boto:Method: GET 2024-04-16T20:15:30.191 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.192 DEBUG:boto:Data: 2024-04-16T20:15:30.192 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.192 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.192 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.192 DEBUG:boto:Params: {} 2024-04-16T20:15:30.192 DEBUG:boto:Token: None 2024-04-16T20:15:30.192 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84= 2024-04-16T20:15:30.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84=', 'Content-Length': '0'} 2024-04-16T20:15:30.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391cb929b50ec205-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.195 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.195 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.195 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.195 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.195 DEBUG:boto:Method: GET 2024-04-16T20:15:30.195 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.195 DEBUG:boto:Data: 2024-04-16T20:15:30.195 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.195 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.195 DEBUG:boto:Params: {} 2024-04-16T20:15:30.196 DEBUG:boto:Token: None 2024-04-16T20:15:30.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84= 2024-04-16T20:15:30.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84=', 'Content-Length': '0'} 2024-04-16T20:15:30.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084271c82b6e75851-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.198 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.198 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.198 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.198 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.198 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.198 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.199 DEBUG:boto:Data: 2024-04-16T20:15:30.199 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.199 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.199 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.199 DEBUG:boto:Params: {} 2024-04-16T20:15:30.199 DEBUG:boto:Token: None 2024-04-16T20:15:30.199 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:THCcMqtSm6lfRmouars+oW/RgY0= 2024-04-16T20:15:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:THCcMqtSm6lfRmouars+oW/RgY0=', 'Content-Length': '0'} 2024-04-16T20:15:30.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1aedf1138ce027-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.201 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.201 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.201 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.202 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.202 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.202 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.202 DEBUG:boto:Data: 2024-04-16T20:15:30.202 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.202 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.202 DEBUG:boto:Params: {} 2024-04-16T20:15:30.202 DEBUG:boto:Token: None 2024-04-16T20:15:30.203 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:THCcMqtSm6lfRmouars+oW/RgY0= 2024-04-16T20:15:30.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:THCcMqtSm6lfRmouars+oW/RgY0=', 'Content-Length': '0'} 2024-04-16T20:15:30.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb787c68b881335f-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:15:30.205 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.205 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.205 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.205 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.205 DEBUG:boto:Method: GET 2024-04-16T20:15:30.205 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.205 DEBUG:boto:Data: 2024-04-16T20:15:30.206 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.206 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.206 DEBUG:boto:Params: {} 2024-04-16T20:15:30.206 DEBUG:boto:Token: None 2024-04-16T20:15:30.206 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.206 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84= 2024-04-16T20:15:30.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84=', 'Content-Length': '0'} 2024-04-16T20:15:30.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c544b3e8adef558a-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.208 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.209 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.209 DEBUG:boto:path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.209 DEBUG:boto:auth_path=/yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.209 DEBUG:boto:Method: GET 2024-04-16T20:15:30.209 DEBUG:boto:Path: /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.209 DEBUG:boto:Data: 2024-04-16T20:15:30.209 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.209 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.209 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.209 DEBUG:boto:Params: {} 2024-04-16T20:15:30.209 DEBUG:boto:Token: None 2024-04-16T20:15:30.210 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19?versionId=null 2024-04-16T20:15:30.210 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84= 2024-04-16T20:15:30.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwVo6HRR75dYgzwsK+LwZ74TZ84=', 'Content-Length': '0'} 2024-04-16T20:15:30.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f352340d6672782-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.212 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.212 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.212 DEBUG:boto:path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.212 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.212 DEBUG:boto:Method: GET 2024-04-16T20:15:30.212 DEBUG:boto:Path: /yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.213 DEBUG:boto:Data: 2024-04-16T20:15:30.213 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.213 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.213 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.213 DEBUG:boto:Params: {} 2024-04-16T20:15:30.213 DEBUG:boto:Token: None 2024-04-16T20:15:30.213 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.213 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f1080ef0cc66d3-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.216 DEBUG:boto:b'yrjefh-54obj191000falseobj192024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.216 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.216 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.216 DEBUG:boto:path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.216 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.216 DEBUG:boto:Method: GET 2024-04-16T20:15:30.216 DEBUG:boto:Path: /yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.216 DEBUG:boto:Data: 2024-04-16T20:15:30.217 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.217 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.217 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.217 DEBUG:boto:Params: {} 2024-04-16T20:15:30.217 DEBUG:boto:Token: None 2024-04-16T20:15:30.217 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b228cfc8628fed-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.221 DEBUG:boto:b'yrjefh-54obj191000falseobj192024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.222 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.222 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.222 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.222 DEBUG:boto:Path: /yrjefh-54/obj19 2024-04-16T20:15:30.222 DEBUG:boto:Data: 2024-04-16T20:15:30.222 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.222 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.222 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.222 DEBUG:boto:Params: {} 2024-04-16T20:15:30.222 DEBUG:boto:Token: None 2024-04-16T20:15:30.223 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19 2024-04-16T20:15:30.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xb3699H1SYQXEXCVWY6oWBU8MSQ= 2024-04-16T20:15:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xb3699H1SYQXEXCVWY6oWBU8MSQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014149fc25bb70bf8-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.225 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.225 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.225 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.225 DEBUG:boto:Path: /yrjefh-54/obj19 2024-04-16T20:15:30.225 DEBUG:boto:Data: 2024-04-16T20:15:30.225 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.225 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.225 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.225 DEBUG:boto:Params: {} 2024-04-16T20:15:30.226 DEBUG:boto:Token: None 2024-04-16T20:15:30.226 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19 2024-04-16T20:15:30.226 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xb3699H1SYQXEXCVWY6oWBU8MSQ= 2024-04-16T20:15:30.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xb3699H1SYQXEXCVWY6oWBU8MSQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af09f94ab381fd3-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:15:30.228 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.228 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.229 DEBUG:boto:Method: GET 2024-04-16T20:15:30.229 DEBUG:boto:Path: /yrjefh-54/obj19 2024-04-16T20:15:30.229 DEBUG:boto:Data: 2024-04-16T20:15:30.229 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.229 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.229 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.229 DEBUG:boto:Params: {} 2024-04-16T20:15:30.229 DEBUG:boto:Token: None 2024-04-16T20:15:30.229 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19 2024-04-16T20:15:30.229 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h1XmZWPrHgpY+gHwjkiuU1ynVws= 2024-04-16T20:15:30.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h1XmZWPrHgpY+gHwjkiuU1ynVws=', 'Content-Length': '0'} 2024-04-16T20:15:30.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c604dd88bac83e8f-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.274 DEBUG:boto:path=/yrjefh-54/obj19 2024-04-16T20:15:30.274 DEBUG:boto:auth_path=/yrjefh-54/obj19 2024-04-16T20:15:30.275 DEBUG:boto:Method: GET 2024-04-16T20:15:30.275 DEBUG:boto:Path: /yrjefh-54/obj19 2024-04-16T20:15:30.275 DEBUG:boto:Data: 2024-04-16T20:15:30.275 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.275 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.275 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.275 DEBUG:boto:Params: {} 2024-04-16T20:15:30.275 DEBUG:boto:Token: None 2024-04-16T20:15:30.275 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj19 2024-04-16T20:15:30.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h1XmZWPrHgpY+gHwjkiuU1ynVws= 2024-04-16T20:15:30.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h1XmZWPrHgpY+gHwjkiuU1ynVws=', 'Content-Length': '0'} 2024-04-16T20:15:30.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e82b4d1f4507e1e-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.278 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.278 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.278 DEBUG:boto:path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.278 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.278 DEBUG:boto:Method: GET 2024-04-16T20:15:30.278 DEBUG:boto:Path: /yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.278 DEBUG:boto:Data: 2024-04-16T20:15:30.278 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.278 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.278 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.278 DEBUG:boto:Params: {} 2024-04-16T20:15:30.279 DEBUG:boto:Token: None 2024-04-16T20:15:30.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007275273a88f2040e-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.281 DEBUG:boto:b'yrjefh-54obj191000falseobj192024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.282 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.282 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.282 DEBUG:boto:path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.282 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.282 DEBUG:boto:Method: GET 2024-04-16T20:15:30.282 DEBUG:boto:Path: /yrjefh-54/?prefix=obj19 2024-04-16T20:15:30.282 DEBUG:boto:Data: 2024-04-16T20:15:30.282 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.282 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.283 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.283 DEBUG:boto:Params: {} 2024-04-16T20:15:30.283 DEBUG:boto:Token: None 2024-04-16T20:15:30.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7eb8366ae547a9e-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.287 DEBUG:boto:b'yrjefh-54obj191000falseobj192024-04-16T20:12:50.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:15:30.287 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.287 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.287 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.288 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.288 DEBUG:boto:Method: GET 2024-04-16T20:15:30.288 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.288 DEBUG:boto:Data: 2024-04-16T20:15:30.288 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.288 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.288 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.288 DEBUG:boto:Params: {} 2024-04-16T20:15:30.288 DEBUG:boto:Token: None 2024-04-16T20:15:30.288 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY= 2024-04-16T20:15:30.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY=', 'Content-Length': '0'} 2024-04-16T20:15:30.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b408bfe704bd73a1-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.291 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.291 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.291 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.291 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.291 DEBUG:boto:Method: GET 2024-04-16T20:15:30.291 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.292 DEBUG:boto:Data: 2024-04-16T20:15:30.292 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.292 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.292 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.292 DEBUG:boto:Params: {} 2024-04-16T20:15:30.292 DEBUG:boto:Token: None 2024-04-16T20:15:30.292 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.292 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY= 2024-04-16T20:15:30.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY=', 'Content-Length': '0'} 2024-04-16T20:15:30.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007151799bf2c9b792-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.334 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.335 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.335 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.335 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.335 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.335 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.335 DEBUG:boto:Data: 2024-04-16T20:15:30.335 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.335 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.335 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.335 DEBUG:boto:Params: {} 2024-04-16T20:15:30.335 DEBUG:boto:Token: None 2024-04-16T20:15:30.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.336 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H9pLS7JEslkHs4lH9mDh7dwOiTM= 2024-04-16T20:15:30.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H9pLS7JEslkHs4lH9mDh7dwOiTM=', 'Content-Length': '0'} 2024-04-16T20:15:30.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ae1abe6806b098-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.338 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.338 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.338 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.338 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.338 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.338 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.338 DEBUG:boto:Data: 2024-04-16T20:15:30.338 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.338 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.338 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.339 DEBUG:boto:Params: {} 2024-04-16T20:15:30.339 DEBUG:boto:Token: None 2024-04-16T20:15:30.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:H9pLS7JEslkHs4lH9mDh7dwOiTM= 2024-04-16T20:15:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H9pLS7JEslkHs4lH9mDh7dwOiTM=', 'Content-Length': '0'} 2024-04-16T20:15:30.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b76063535831f1-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:15:30.341 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.341 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.341 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.342 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.342 DEBUG:boto:Method: GET 2024-04-16T20:15:30.342 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.342 DEBUG:boto:Data: 2024-04-16T20:15:30.342 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.342 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.342 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.342 DEBUG:boto:Params: {} 2024-04-16T20:15:30.342 DEBUG:boto:Token: None 2024-04-16T20:15:30.342 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY= 2024-04-16T20:15:30.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY=', 'Content-Length': '0'} 2024-04-16T20:15:30.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb144a98160cb8bb-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.345 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.345 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.345 DEBUG:boto:path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.345 DEBUG:boto:auth_path=/yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.345 DEBUG:boto:Method: GET 2024-04-16T20:15:30.345 DEBUG:boto:Path: /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.345 DEBUG:boto:Data: 2024-04-16T20:15:30.345 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.345 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.345 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.345 DEBUG:boto:Params: {} 2024-04-16T20:15:30.346 DEBUG:boto:Token: None 2024-04-16T20:15:30.346 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22?versionId=null 2024-04-16T20:15:30.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY= 2024-04-16T20:15:30.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hv+Nvhe6IUj0BDpKLPsXUej7eJY=', 'Content-Length': '0'} 2024-04-16T20:15:30.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e80db8d819aa967-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.348 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.348 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.348 DEBUG:boto:path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.348 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.348 DEBUG:boto:Method: GET 2024-04-16T20:15:30.348 DEBUG:boto:Path: /yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.349 DEBUG:boto:Data: 2024-04-16T20:15:30.349 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.349 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.349 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.349 DEBUG:boto:Params: {} 2024-04-16T20:15:30.349 DEBUG:boto:Token: None 2024-04-16T20:15:30.349 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.349 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b11b3ede2312a36-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.352 DEBUG:boto:b'yrjefh-54obj221000falseobj222024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.352 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.352 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.352 DEBUG:boto:path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.352 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.353 DEBUG:boto:Method: GET 2024-04-16T20:15:30.353 DEBUG:boto:Path: /yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.353 DEBUG:boto:Data: 2024-04-16T20:15:30.353 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.353 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.353 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.353 DEBUG:boto:Params: {} 2024-04-16T20:15:30.353 DEBUG:boto:Token: None 2024-04-16T20:15:30.353 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.353 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e16e32648dcb9eb-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.357 DEBUG:boto:b'yrjefh-54obj221000falseobj222024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.357 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.357 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.358 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.358 DEBUG:boto:Path: /yrjefh-54/obj22 2024-04-16T20:15:30.358 DEBUG:boto:Data: 2024-04-16T20:15:30.358 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.358 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.358 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.358 DEBUG:boto:Params: {} 2024-04-16T20:15:30.358 DEBUG:boto:Token: None 2024-04-16T20:15:30.358 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22 2024-04-16T20:15:30.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8j2iS7vcY5FdYYgnENV7xKP/UjM= 2024-04-16T20:15:30.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8j2iS7vcY5FdYYgnENV7xKP/UjM=', 'Content-Length': '0'} 2024-04-16T20:15:30.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50c561765a25450-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.361 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.361 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.361 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.361 DEBUG:boto:Path: /yrjefh-54/obj22 2024-04-16T20:15:30.361 DEBUG:boto:Data: 2024-04-16T20:15:30.361 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.361 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.361 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.361 DEBUG:boto:Params: {} 2024-04-16T20:15:30.362 DEBUG:boto:Token: None 2024-04-16T20:15:30.362 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22 2024-04-16T20:15:30.362 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8j2iS7vcY5FdYYgnENV7xKP/UjM= 2024-04-16T20:15:30.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8j2iS7vcY5FdYYgnENV7xKP/UjM=', 'Content-Length': '0'} 2024-04-16T20:15:30.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7e8fffac497314-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:15:30.364 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.364 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.364 DEBUG:boto:Method: GET 2024-04-16T20:15:30.364 DEBUG:boto:Path: /yrjefh-54/obj22 2024-04-16T20:15:30.364 DEBUG:boto:Data: 2024-04-16T20:15:30.364 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.364 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.364 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.364 DEBUG:boto:Params: {} 2024-04-16T20:15:30.364 DEBUG:boto:Token: None 2024-04-16T20:15:30.365 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22 2024-04-16T20:15:30.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyo8iiM9vgEHYeDeRWrlxmA6AN0= 2024-04-16T20:15:30.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyo8iiM9vgEHYeDeRWrlxmA6AN0=', 'Content-Length': '0'} 2024-04-16T20:15:30.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021dcb186b8a3ca68-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.367 DEBUG:boto:path=/yrjefh-54/obj22 2024-04-16T20:15:30.367 DEBUG:boto:auth_path=/yrjefh-54/obj22 2024-04-16T20:15:30.367 DEBUG:boto:Method: GET 2024-04-16T20:15:30.367 DEBUG:boto:Path: /yrjefh-54/obj22 2024-04-16T20:15:30.367 DEBUG:boto:Data: 2024-04-16T20:15:30.367 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.367 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.367 DEBUG:boto:Params: {} 2024-04-16T20:15:30.367 DEBUG:boto:Token: None 2024-04-16T20:15:30.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj22 2024-04-16T20:15:30.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vyo8iiM9vgEHYeDeRWrlxmA6AN0= 2024-04-16T20:15:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vyo8iiM9vgEHYeDeRWrlxmA6AN0=', 'Content-Length': '0'} 2024-04-16T20:15:30.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c991a8f4cd93536b-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.370 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.370 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.370 DEBUG:boto:path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.370 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.370 DEBUG:boto:Method: GET 2024-04-16T20:15:30.370 DEBUG:boto:Path: /yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.370 DEBUG:boto:Data: 2024-04-16T20:15:30.370 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.370 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.370 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.370 DEBUG:boto:Params: {} 2024-04-16T20:15:30.371 DEBUG:boto:Token: None 2024-04-16T20:15:30.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b891a861e6c405-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.414 DEBUG:boto:b'yrjefh-54obj221000falseobj222024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.415 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.415 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.415 DEBUG:boto:path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.415 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.415 DEBUG:boto:Method: GET 2024-04-16T20:15:30.415 DEBUG:boto:Path: /yrjefh-54/?prefix=obj22 2024-04-16T20:15:30.415 DEBUG:boto:Data: 2024-04-16T20:15:30.415 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.415 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.415 DEBUG:boto:Params: {} 2024-04-16T20:15:30.416 DEBUG:boto:Token: None 2024-04-16T20:15:30.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ee930f623a37c2-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.419 DEBUG:boto:b'yrjefh-54obj221000falseobj222024-04-16T20:12:50.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:15:30.420 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.420 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.420 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.420 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.420 DEBUG:boto:Method: GET 2024-04-16T20:15:30.420 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.420 DEBUG:boto:Data: 2024-04-16T20:15:30.420 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.421 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.421 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.421 DEBUG:boto:Params: {} 2024-04-16T20:15:30.421 DEBUG:boto:Token: None 2024-04-16T20:15:30.421 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40= 2024-04-16T20:15:30.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40=', 'Content-Length': '0'} 2024-04-16T20:15:30.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e683a419ff20483-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.424 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.424 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.424 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.424 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.424 DEBUG:boto:Method: GET 2024-04-16T20:15:30.424 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.424 DEBUG:boto:Data: 2024-04-16T20:15:30.424 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.424 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.424 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.424 DEBUG:boto:Params: {} 2024-04-16T20:15:30.425 DEBUG:boto:Token: None 2024-04-16T20:15:30.425 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.425 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40= 2024-04-16T20:15:30.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40=', 'Content-Length': '0'} 2024-04-16T20:15:30.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e36aafa954df953-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.470 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.470 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.470 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.471 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.471 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.471 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.471 DEBUG:boto:Data: 2024-04-16T20:15:30.471 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.471 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.471 DEBUG:boto:Params: {} 2024-04-16T20:15:30.471 DEBUG:boto:Token: None 2024-04-16T20:15:30.471 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LeZFIMzzaTlw1jrxBiG03it4fhY= 2024-04-16T20:15:30.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LeZFIMzzaTlw1jrxBiG03it4fhY=', 'Content-Length': '0'} 2024-04-16T20:15:30.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4d11753c3a0479-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.474 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.474 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.474 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.474 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.474 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.475 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.475 DEBUG:boto:Data: 2024-04-16T20:15:30.475 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.475 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.475 DEBUG:boto:Params: {} 2024-04-16T20:15:30.475 DEBUG:boto:Token: None 2024-04-16T20:15:30.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LeZFIMzzaTlw1jrxBiG03it4fhY= 2024-04-16T20:15:30.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LeZFIMzzaTlw1jrxBiG03it4fhY=', 'Content-Length': '0'} 2024-04-16T20:15:30.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd4bc715bede3bd-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:15:30.478 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.478 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.478 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.478 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.478 DEBUG:boto:Method: GET 2024-04-16T20:15:30.478 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.478 DEBUG:boto:Data: 2024-04-16T20:15:30.478 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.479 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.479 DEBUG:boto:Params: {} 2024-04-16T20:15:30.479 DEBUG:boto:Token: None 2024-04-16T20:15:30.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40= 2024-04-16T20:15:30.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40=', 'Content-Length': '0'} 2024-04-16T20:15:30.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1bdcdd781358a4c-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.481 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.482 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.482 DEBUG:boto:path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.482 DEBUG:boto:auth_path=/yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.482 DEBUG:boto:Method: GET 2024-04-16T20:15:30.482 DEBUG:boto:Path: /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.482 DEBUG:boto:Data: 2024-04-16T20:15:30.482 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.482 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.482 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.482 DEBUG:boto:Params: {} 2024-04-16T20:15:30.482 DEBUG:boto:Token: None 2024-04-16T20:15:30.483 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23?versionId=null 2024-04-16T20:15:30.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40= 2024-04-16T20:15:30.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Oh++cSDiYFG4pi7FNV+O1CH2x40=', 'Content-Length': '0'} 2024-04-16T20:15:30.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003455d0078a42b1a5-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.485 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.485 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.486 DEBUG:boto:path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.486 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.486 DEBUG:boto:Method: GET 2024-04-16T20:15:30.486 DEBUG:boto:Path: /yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.486 DEBUG:boto:Data: 2024-04-16T20:15:30.486 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.486 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.486 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.486 DEBUG:boto:Params: {} 2024-04-16T20:15:30.486 DEBUG:boto:Token: None 2024-04-16T20:15:30.486 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbdd854e294388ad-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.489 DEBUG:boto:b'yrjefh-54obj231000falseobj232024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.489 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.490 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.490 DEBUG:boto:path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.490 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.490 DEBUG:boto:Method: GET 2024-04-16T20:15:30.490 DEBUG:boto:Path: /yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.490 DEBUG:boto:Data: 2024-04-16T20:15:30.490 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.490 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.490 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.490 DEBUG:boto:Params: {} 2024-04-16T20:15:30.491 DEBUG:boto:Token: None 2024-04-16T20:15:30.491 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.491 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cebeb36b58f92a0-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.494 DEBUG:boto:b'yrjefh-54obj231000falseobj232024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.495 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.495 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.495 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.495 DEBUG:boto:Path: /yrjefh-54/obj23 2024-04-16T20:15:30.495 DEBUG:boto:Data: 2024-04-16T20:15:30.495 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.495 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.495 DEBUG:boto:Params: {} 2024-04-16T20:15:30.496 DEBUG:boto:Token: None 2024-04-16T20:15:30.496 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23 2024-04-16T20:15:30.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:97/9E7js3842qKiCKBHAiEEuHJQ= 2024-04-16T20:15:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:97/9E7js3842qKiCKBHAiEEuHJQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d33d74c5840487-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.499 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.499 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.499 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.499 DEBUG:boto:Path: /yrjefh-54/obj23 2024-04-16T20:15:30.499 DEBUG:boto:Data: 2024-04-16T20:15:30.499 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.499 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.499 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.499 DEBUG:boto:Params: {} 2024-04-16T20:15:30.500 DEBUG:boto:Token: None 2024-04-16T20:15:30.500 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23 2024-04-16T20:15:30.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:97/9E7js3842qKiCKBHAiEEuHJQ= 2024-04-16T20:15:30.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:97/9E7js3842qKiCKBHAiEEuHJQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083341f7f9931ab7a-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:15:30.502 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.502 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.502 DEBUG:boto:Method: GET 2024-04-16T20:15:30.502 DEBUG:boto:Path: /yrjefh-54/obj23 2024-04-16T20:15:30.503 DEBUG:boto:Data: 2024-04-16T20:15:30.503 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.503 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.503 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.503 DEBUG:boto:Params: {} 2024-04-16T20:15:30.503 DEBUG:boto:Token: None 2024-04-16T20:15:30.503 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23 2024-04-16T20:15:30.503 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gn0zNNlGonhyJlBWAkCd/w+XOdM= 2024-04-16T20:15:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gn0zNNlGonhyJlBWAkCd/w+XOdM=', 'Content-Length': '0'} 2024-04-16T20:15:30.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000403c87d00afffa9c-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.506 DEBUG:boto:path=/yrjefh-54/obj23 2024-04-16T20:15:30.506 DEBUG:boto:auth_path=/yrjefh-54/obj23 2024-04-16T20:15:30.507 DEBUG:boto:Method: GET 2024-04-16T20:15:30.507 DEBUG:boto:Path: /yrjefh-54/obj23 2024-04-16T20:15:30.507 DEBUG:boto:Data: 2024-04-16T20:15:30.507 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.507 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.507 DEBUG:boto:Params: {} 2024-04-16T20:15:30.507 DEBUG:boto:Token: None 2024-04-16T20:15:30.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj23 2024-04-16T20:15:30.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gn0zNNlGonhyJlBWAkCd/w+XOdM= 2024-04-16T20:15:30.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gn0zNNlGonhyJlBWAkCd/w+XOdM=', 'Content-Length': '0'} 2024-04-16T20:15:30.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca828fd7595c3b01-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.510 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.510 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.510 DEBUG:boto:path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.510 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.510 DEBUG:boto:Method: GET 2024-04-16T20:15:30.510 DEBUG:boto:Path: /yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.510 DEBUG:boto:Data: 2024-04-16T20:15:30.511 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.511 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.511 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.511 DEBUG:boto:Params: {} 2024-04-16T20:15:30.511 DEBUG:boto:Token: None 2024-04-16T20:15:30.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113c630b6223d727-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.513 DEBUG:boto:b'yrjefh-54obj231000falseobj232024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.514 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.514 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.514 DEBUG:boto:path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.514 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.514 DEBUG:boto:Method: GET 2024-04-16T20:15:30.514 DEBUG:boto:Path: /yrjefh-54/?prefix=obj23 2024-04-16T20:15:30.515 DEBUG:boto:Data: 2024-04-16T20:15:30.515 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.515 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.515 DEBUG:boto:Params: {} 2024-04-16T20:15:30.515 DEBUG:boto:Token: None 2024-04-16T20:15:30.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039ab7c056c09688a-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.519 DEBUG:boto:b'yrjefh-54obj231000falseobj232024-04-16T20:12:51.050Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:15:30.519 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.519 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.519 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.520 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.520 DEBUG:boto:Method: GET 2024-04-16T20:15:30.520 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.520 DEBUG:boto:Data: 2024-04-16T20:15:30.520 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.520 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.520 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.520 DEBUG:boto:Params: {} 2024-04-16T20:15:30.520 DEBUG:boto:Token: None 2024-04-16T20:15:30.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.521 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU= 2024-04-16T20:15:30.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU=', 'Content-Length': '0'} 2024-04-16T20:15:30.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007684d85f3d212beb-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.523 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.523 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.523 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.523 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.523 DEBUG:boto:Method: GET 2024-04-16T20:15:30.523 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.523 DEBUG:boto:Data: 2024-04-16T20:15:30.523 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.524 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.524 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.524 DEBUG:boto:Params: {} 2024-04-16T20:15:30.524 DEBUG:boto:Token: None 2024-04-16T20:15:30.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU= 2024-04-16T20:15:30.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU=', 'Content-Length': '0'} 2024-04-16T20:15:30.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064876046556713f-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.566 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.566 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.567 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.567 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.567 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.567 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.567 DEBUG:boto:Data: 2024-04-16T20:15:30.567 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.567 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.567 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.567 DEBUG:boto:Params: {} 2024-04-16T20:15:30.567 DEBUG:boto:Token: None 2024-04-16T20:15:30.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wm0hMKhfnx1bIrdhtTtsdjO2UBQ= 2024-04-16T20:15:30.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wm0hMKhfnx1bIrdhtTtsdjO2UBQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a393cc41ea6259-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.570 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.570 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.570 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.570 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.570 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.571 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.571 DEBUG:boto:Data: 2024-04-16T20:15:30.571 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.571 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.571 DEBUG:boto:Params: {} 2024-04-16T20:15:30.571 DEBUG:boto:Token: None 2024-04-16T20:15:30.571 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wm0hMKhfnx1bIrdhtTtsdjO2UBQ= 2024-04-16T20:15:30.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wm0hMKhfnx1bIrdhtTtsdjO2UBQ=', 'Content-Length': '0'} 2024-04-16T20:15:30.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb23057a03f0d700-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:15:30.574 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.574 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.574 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.574 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.574 DEBUG:boto:Method: GET 2024-04-16T20:15:30.574 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.574 DEBUG:boto:Data: 2024-04-16T20:15:30.574 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.574 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.575 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.575 DEBUG:boto:Params: {} 2024-04-16T20:15:30.575 DEBUG:boto:Token: None 2024-04-16T20:15:30.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU= 2024-04-16T20:15:30.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU=', 'Content-Length': '0'} 2024-04-16T20:15:30.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9265fb48814c8f6-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.577 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.578 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.578 DEBUG:boto:path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.578 DEBUG:boto:auth_path=/yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.578 DEBUG:boto:Method: GET 2024-04-16T20:15:30.578 DEBUG:boto:Path: /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.578 DEBUG:boto:Data: 2024-04-16T20:15:30.578 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.578 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.578 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.578 DEBUG:boto:Params: {} 2024-04-16T20:15:30.578 DEBUG:boto:Token: None 2024-04-16T20:15:30.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29?versionId=null 2024-04-16T20:15:30.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU= 2024-04-16T20:15:30.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/HApDf+IqwDyouxeWDtoyW2BPXU=', 'Content-Length': '0'} 2024-04-16T20:15:30.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d988d7c5e805df7-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.581 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.581 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.581 DEBUG:boto:path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.582 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.582 DEBUG:boto:Method: GET 2024-04-16T20:15:30.582 DEBUG:boto:Path: /yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.582 DEBUG:boto:Data: 2024-04-16T20:15:30.582 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.582 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.582 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.582 DEBUG:boto:Params: {} 2024-04-16T20:15:30.582 DEBUG:boto:Token: None 2024-04-16T20:15:30.582 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe5f637c5792cbd-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.585 DEBUG:boto:b'yrjefh-54obj291000falseobj292024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.586 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.586 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.586 DEBUG:boto:path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.586 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.586 DEBUG:boto:Method: GET 2024-04-16T20:15:30.586 DEBUG:boto:Path: /yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.586 DEBUG:boto:Data: 2024-04-16T20:15:30.586 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.587 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.587 DEBUG:boto:Params: {} 2024-04-16T20:15:30.587 DEBUG:boto:Token: None 2024-04-16T20:15:30.587 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.587 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0cf541613f94917-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.590 DEBUG:boto:b'yrjefh-54obj291000falseobj292024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.591 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.591 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.591 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.591 DEBUG:boto:Path: /yrjefh-54/obj29 2024-04-16T20:15:30.591 DEBUG:boto:Data: 2024-04-16T20:15:30.591 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.591 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.591 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.591 DEBUG:boto:Params: {} 2024-04-16T20:15:30.592 DEBUG:boto:Token: None 2024-04-16T20:15:30.592 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29 2024-04-16T20:15:30.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kUs49aMMLwuYzNT4XNL4BOKBSNo= 2024-04-16T20:15:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kUs49aMMLwuYzNT4XNL4BOKBSNo=', 'Content-Length': '0'} 2024-04-16T20:15:30.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310579c178e3ec03-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.594 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.595 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.595 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.595 DEBUG:boto:Path: /yrjefh-54/obj29 2024-04-16T20:15:30.595 DEBUG:boto:Data: 2024-04-16T20:15:30.595 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.595 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.595 DEBUG:boto:Params: {} 2024-04-16T20:15:30.595 DEBUG:boto:Token: None 2024-04-16T20:15:30.595 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29 2024-04-16T20:15:30.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kUs49aMMLwuYzNT4XNL4BOKBSNo= 2024-04-16T20:15:30.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kUs49aMMLwuYzNT4XNL4BOKBSNo=', 'Content-Length': '0'} 2024-04-16T20:15:30.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b071106cdc292ee-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:15:30.598 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.598 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.598 DEBUG:boto:Method: GET 2024-04-16T20:15:30.598 DEBUG:boto:Path: /yrjefh-54/obj29 2024-04-16T20:15:30.598 DEBUG:boto:Data: 2024-04-16T20:15:30.599 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.599 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.599 DEBUG:boto:Params: {} 2024-04-16T20:15:30.599 DEBUG:boto:Token: None 2024-04-16T20:15:30.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29 2024-04-16T20:15:30.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fpGdlw8C8xgMqftjWz/OHLSxD3M= 2024-04-16T20:15:30.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fpGdlw8C8xgMqftjWz/OHLSxD3M=', 'Content-Length': '0'} 2024-04-16T20:15:30.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd28d08e90c23a1-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.642 DEBUG:boto:path=/yrjefh-54/obj29 2024-04-16T20:15:30.643 DEBUG:boto:auth_path=/yrjefh-54/obj29 2024-04-16T20:15:30.643 DEBUG:boto:Method: GET 2024-04-16T20:15:30.643 DEBUG:boto:Path: /yrjefh-54/obj29 2024-04-16T20:15:30.643 DEBUG:boto:Data: 2024-04-16T20:15:30.643 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.643 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.643 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.643 DEBUG:boto:Params: {} 2024-04-16T20:15:30.643 DEBUG:boto:Token: None 2024-04-16T20:15:30.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj29 2024-04-16T20:15:30.644 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fpGdlw8C8xgMqftjWz/OHLSxD3M= 2024-04-16T20:15:30.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fpGdlw8C8xgMqftjWz/OHLSxD3M=', 'Content-Length': '0'} 2024-04-16T20:15:30.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f2653f37e12a67-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.646 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.646 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.646 DEBUG:boto:path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.646 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.647 DEBUG:boto:Method: GET 2024-04-16T20:15:30.647 DEBUG:boto:Path: /yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.647 DEBUG:boto:Data: 2024-04-16T20:15:30.647 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.647 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.647 DEBUG:boto:Params: {} 2024-04-16T20:15:30.647 DEBUG:boto:Token: None 2024-04-16T20:15:30.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbaeb162a57e9e1-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.690 DEBUG:boto:b'yrjefh-54obj291000falseobj292024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.691 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.691 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.691 DEBUG:boto:path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.691 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.691 DEBUG:boto:Method: GET 2024-04-16T20:15:30.691 DEBUG:boto:Path: /yrjefh-54/?prefix=obj29 2024-04-16T20:15:30.691 DEBUG:boto:Data: 2024-04-16T20:15:30.691 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.692 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.692 DEBUG:boto:Params: {} 2024-04-16T20:15:30.692 DEBUG:boto:Token: None 2024-04-16T20:15:30.692 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.692 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000924d3ba4cfdb0e4e-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.696 DEBUG:boto:b'yrjefh-54obj291000falseobj292024-04-16T20:12:51.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:15:30.696 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.697 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.697 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.697 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.697 DEBUG:boto:Method: GET 2024-04-16T20:15:30.697 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.697 DEBUG:boto:Data: 2024-04-16T20:15:30.697 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.697 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.697 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.697 DEBUG:boto:Params: {} 2024-04-16T20:15:30.698 DEBUG:boto:Token: None 2024-04-16T20:15:30.698 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU= 2024-04-16T20:15:30.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU=', 'Content-Length': '0'} 2024-04-16T20:15:30.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04af98ab361b9bc-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.742 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.742 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.743 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.743 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.743 DEBUG:boto:Method: GET 2024-04-16T20:15:30.743 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.743 DEBUG:boto:Data: 2024-04-16T20:15:30.743 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.743 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.743 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.743 DEBUG:boto:Params: {} 2024-04-16T20:15:30.743 DEBUG:boto:Token: None 2024-04-16T20:15:30.744 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.744 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU= 2024-04-16T20:15:30.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU=', 'Content-Length': '0'} 2024-04-16T20:15:30.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ccbbd4dd4e5ba8-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.746 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.747 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.747 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.747 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.747 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.747 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.747 DEBUG:boto:Data: 2024-04-16T20:15:30.747 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.747 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.747 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.747 DEBUG:boto:Params: {} 2024-04-16T20:15:30.748 DEBUG:boto:Token: None 2024-04-16T20:15:30.748 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.748 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afzAurgEBRGP7PvUPscZNyHjThI= 2024-04-16T20:15:30.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afzAurgEBRGP7PvUPscZNyHjThI=', 'Content-Length': '0'} 2024-04-16T20:15:30.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d992d3bf86cfa384-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.750 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.750 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.750 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.751 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.751 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.751 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.751 DEBUG:boto:Data: 2024-04-16T20:15:30.751 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.751 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.751 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.751 DEBUG:boto:Params: {} 2024-04-16T20:15:30.752 DEBUG:boto:Token: None 2024-04-16T20:15:30.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:afzAurgEBRGP7PvUPscZNyHjThI= 2024-04-16T20:15:30.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:afzAurgEBRGP7PvUPscZNyHjThI=', 'Content-Length': '0'} 2024-04-16T20:15:30.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c93a770d15d56fa7-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:15:30.755 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.755 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.755 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.755 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.755 DEBUG:boto:Method: GET 2024-04-16T20:15:30.755 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.755 DEBUG:boto:Data: 2024-04-16T20:15:30.755 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.755 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.755 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.756 DEBUG:boto:Params: {} 2024-04-16T20:15:30.756 DEBUG:boto:Token: None 2024-04-16T20:15:30.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU= 2024-04-16T20:15:30.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU=', 'Content-Length': '0'} 2024-04-16T20:15:30.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b93e3706623106d-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.803 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.803 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.803 DEBUG:boto:path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.803 DEBUG:boto:auth_path=/yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.803 DEBUG:boto:Method: GET 2024-04-16T20:15:30.803 DEBUG:boto:Path: /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.803 DEBUG:boto:Data: 2024-04-16T20:15:30.803 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.804 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.804 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.804 DEBUG:boto:Params: {} 2024-04-16T20:15:30.804 DEBUG:boto:Token: None 2024-04-16T20:15:30.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31?versionId=null 2024-04-16T20:15:30.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU= 2024-04-16T20:15:30.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VV6s6yBQLgwZOJUyI8lFo/hssyU=', 'Content-Length': '0'} 2024-04-16T20:15:30.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851c2e0865b5ba63-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.807 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.807 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.807 DEBUG:boto:path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.807 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.807 DEBUG:boto:Method: GET 2024-04-16T20:15:30.808 DEBUG:boto:Path: /yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.808 DEBUG:boto:Data: 2024-04-16T20:15:30.808 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.808 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.808 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.808 DEBUG:boto:Params: {} 2024-04-16T20:15:30.808 DEBUG:boto:Token: None 2024-04-16T20:15:30.808 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e75d3a3195c38a8-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.854 DEBUG:boto:b'yrjefh-54obj311000falseobj312024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.855 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.855 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.855 DEBUG:boto:path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.855 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.855 DEBUG:boto:Method: GET 2024-04-16T20:15:30.855 DEBUG:boto:Path: /yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.855 DEBUG:boto:Data: 2024-04-16T20:15:30.856 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.856 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.856 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.856 DEBUG:boto:Params: {} 2024-04-16T20:15:30.856 DEBUG:boto:Token: None 2024-04-16T20:15:30.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35716ffc7f51fae-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.860 DEBUG:boto:b'yrjefh-54obj311000falseobj312024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.860 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.860 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.861 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.861 DEBUG:boto:Path: /yrjefh-54/obj31 2024-04-16T20:15:30.861 DEBUG:boto:Data: 2024-04-16T20:15:30.861 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.861 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.861 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.861 DEBUG:boto:Params: {} 2024-04-16T20:15:30.861 DEBUG:boto:Token: None 2024-04-16T20:15:30.862 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31 2024-04-16T20:15:30.862 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwYOhFOnTphNqROvmnI6bKQDC98= 2024-04-16T20:15:30.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwYOhFOnTphNqROvmnI6bKQDC98=', 'Content-Length': '0'} 2024-04-16T20:15:30.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c444d002f25903ad-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.865 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.865 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.865 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.865 DEBUG:boto:Path: /yrjefh-54/obj31 2024-04-16T20:15:30.865 DEBUG:boto:Data: 2024-04-16T20:15:30.865 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.865 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.865 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.865 DEBUG:boto:Params: {} 2024-04-16T20:15:30.866 DEBUG:boto:Token: None 2024-04-16T20:15:30.866 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31 2024-04-16T20:15:30.866 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qwYOhFOnTphNqROvmnI6bKQDC98= 2024-04-16T20:15:30.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qwYOhFOnTphNqROvmnI6bKQDC98=', 'Content-Length': '0'} 2024-04-16T20:15:30.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25ea5535df7f6ba-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:15:30.869 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.869 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.869 DEBUG:boto:Method: GET 2024-04-16T20:15:30.869 DEBUG:boto:Path: /yrjefh-54/obj31 2024-04-16T20:15:30.869 DEBUG:boto:Data: 2024-04-16T20:15:30.869 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.869 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.870 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.870 DEBUG:boto:Params: {} 2024-04-16T20:15:30.870 DEBUG:boto:Token: None 2024-04-16T20:15:30.870 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31 2024-04-16T20:15:30.870 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1SpGNdIZ5NzVZtIsrCLELFisQ1c= 2024-04-16T20:15:30.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1SpGNdIZ5NzVZtIsrCLELFisQ1c=', 'Content-Length': '0'} 2024-04-16T20:15:30.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a311ee5e502bb5a0-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.915 DEBUG:boto:path=/yrjefh-54/obj31 2024-04-16T20:15:30.915 DEBUG:boto:auth_path=/yrjefh-54/obj31 2024-04-16T20:15:30.915 DEBUG:boto:Method: GET 2024-04-16T20:15:30.915 DEBUG:boto:Path: /yrjefh-54/obj31 2024-04-16T20:15:30.915 DEBUG:boto:Data: 2024-04-16T20:15:30.915 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.915 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.915 DEBUG:boto:Params: {} 2024-04-16T20:15:30.916 DEBUG:boto:Token: None 2024-04-16T20:15:30.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj31 2024-04-16T20:15:30.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1SpGNdIZ5NzVZtIsrCLELFisQ1c= 2024-04-16T20:15:30.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1SpGNdIZ5NzVZtIsrCLELFisQ1c=', 'Content-Length': '0'} 2024-04-16T20:15:30.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b614450e0134e800-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.919 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.919 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.919 DEBUG:boto:path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.919 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.919 DEBUG:boto:Method: GET 2024-04-16T20:15:30.919 DEBUG:boto:Path: /yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.919 DEBUG:boto:Data: 2024-04-16T20:15:30.920 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.920 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.920 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.920 DEBUG:boto:Params: {} 2024-04-16T20:15:30.920 DEBUG:boto:Token: None 2024-04-16T20:15:30.920 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.920 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002081de3e5a449c8f-00661edc62-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.923 DEBUG:boto:b'yrjefh-54obj311000falseobj312024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.923 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:30.924 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:30.924 DEBUG:boto:path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.924 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.924 DEBUG:boto:Method: GET 2024-04-16T20:15:30.924 DEBUG:boto:Path: /yrjefh-54/?prefix=obj31 2024-04-16T20:15:30.924 DEBUG:boto:Data: 2024-04-16T20:15:30.924 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.924 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.925 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.925 DEBUG:boto:Params: {} 2024-04-16T20:15:30.925 DEBUG:boto:Token: None 2024-04-16T20:15:30.925 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/ 2024-04-16T20:15:30.925 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA= 2024-04-16T20:15:30.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o6wZvLnj2naDThShRFE2TfZckfA=', 'Content-Length': '0'} 2024-04-16T20:15:30.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc3262382efee22-00661edc62-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.929 DEBUG:boto:b'yrjefh-54obj311000falseobj312024-04-16T20:12:51.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:30.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:15:30.930 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:30.930 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:30.930 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.930 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.930 DEBUG:boto:Method: GET 2024-04-16T20:15:30.930 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.930 DEBUG:boto:Data: 2024-04-16T20:15:30.930 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.931 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.931 DEBUG:boto:Params: {} 2024-04-16T20:15:30.931 DEBUG:boto:Token: None 2024-04-16T20:15:30.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU= 2024-04-16T20:15:30.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU=', 'Content-Length': '0'} 2024-04-16T20:15:30.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f7444a436d9b34-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.974 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:30.975 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:30.975 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.975 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.975 DEBUG:boto:Method: GET 2024-04-16T20:15:30.975 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.975 DEBUG:boto:Data: 2024-04-16T20:15:30.975 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.976 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.976 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.976 DEBUG:boto:Params: {} 2024-04-16T20:15:30.976 DEBUG:boto:Token: None 2024-04-16T20:15:30.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU= 2024-04-16T20:15:30.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU=', 'Content-Length': '0'} 2024-04-16T20:15:30.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f319ac9a68ce082-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.979 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:30.979 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:30.979 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.979 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.980 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.980 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.980 DEBUG:boto:Data: 2024-04-16T20:15:30.980 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.980 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.980 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.980 DEBUG:boto:Params: {} 2024-04-16T20:15:30.980 DEBUG:boto:Token: None 2024-04-16T20:15:30.981 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.981 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sd72yu4u9K/V7wx0KBcD/fi2LtE= 2024-04-16T20:15:30.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sd72yu4u9K/V7wx0KBcD/fi2LtE=', 'Content-Length': '0'} 2024-04-16T20:15:30.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fece9cc161fad6c-00661edc62-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.983 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:30.984 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:30.984 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.984 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.984 DEBUG:boto:Method: HEAD 2024-04-16T20:15:30.984 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.984 DEBUG:boto:Data: 2024-04-16T20:15:30.984 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.984 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:30.984 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.985 DEBUG:boto:Params: {} 2024-04-16T20:15:30.985 DEBUG:boto:Token: None 2024-04-16T20:15:30.985 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.985 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sd72yu4u9K/V7wx0KBcD/fi2LtE= 2024-04-16T20:15:30.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sd72yu4u9K/V7wx0KBcD/fi2LtE=', 'Content-Length': '0'} 2024-04-16T20:15:30.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfa3586ffb9e925-00661edc62-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:30.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:15:30.988 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:30.988 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:30.988 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.989 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.989 DEBUG:boto:Method: GET 2024-04-16T20:15:30.989 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.989 DEBUG:boto:Data: 2024-04-16T20:15:30.989 DEBUG:boto:Headers: {} 2024-04-16T20:15:30.989 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:30.989 DEBUG:boto:Port: 8000 2024-04-16T20:15:30.989 DEBUG:boto:Params: {} 2024-04-16T20:15:30.990 DEBUG:boto:Token: None 2024-04-16T20:15:30.990 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:30 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:30.990 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU= 2024-04-16T20:15:30.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:30 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VtGrdaYejTvFiNvWFyToKmwUpSU=', 'Content-Length': '0'} 2024-04-16T20:15:30.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48b245143d1aef9-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.034 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:31.035 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:31.035 DEBUG:boto:path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:31.035 DEBUG:boto:auth_path=/yrjefh-54/obj33?versionId=null 2024-04-16T20:15:31.035 DEBUG:boto:Method: GET 2024-04-16T20:15:31.035 DEBUG:boto:Path: /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:31.035 DEBUG:boto:Data: 2024-04-16T20:15:31.035 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.036 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.036 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.036 DEBUG:boto:Params: {} 2024-04-16T20:15:31.036 DEBUG:boto:Token: None 2024-04-16T20:15:31.036 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj33?versionId=null 2024-04-16T20:15:31.036 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CjnhGw7E7zMiNXvHJM4XGvD9q8g= 2024-04-16T20:15:31.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CjnhGw7E7zMiNXvHJM4XGvD9q8g=', 'Content-Length': '0'} 2024-04-16T20:15:31.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071fc155cf1cec2b-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.039 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.039 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.039 DEBUG:boto:path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.040 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.040 DEBUG:boto:Method: GET 2024-04-16T20:15:31.040 DEBUG:boto:Path: /yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.040 DEBUG:boto:Data: 2024-04-16T20:15:31.040 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.040 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.040 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.040 DEBUG:boto:Params: {} 2024-04-16T20:15:31.041 DEBUG:boto:Token: None 2024-04-16T20:15:31.041 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.041 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cddeaf2c7605071-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.086 DEBUG:boto:b'yrjefh-54obj331000falseobj332024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.087 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.087 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.087 DEBUG:boto:path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.087 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.087 DEBUG:boto:Method: GET 2024-04-16T20:15:31.087 DEBUG:boto:Path: /yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.087 DEBUG:boto:Data: 2024-04-16T20:15:31.088 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.088 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.088 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.088 DEBUG:boto:Params: {} 2024-04-16T20:15:31.088 DEBUG:boto:Token: None 2024-04-16T20:15:31.088 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7e48c147be7672-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.092 DEBUG:boto:b'yrjefh-54obj331000falseobj332024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.092 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:31.093 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:31.093 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.093 DEBUG:boto:Path: /yrjefh-54/obj33 2024-04-16T20:15:31.093 DEBUG:boto:Data: 2024-04-16T20:15:31.093 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.093 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.093 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.093 DEBUG:boto:Params: {} 2024-04-16T20:15:31.093 DEBUG:boto:Token: None 2024-04-16T20:15:31.094 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj33 2024-04-16T20:15:31.094 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BjHsTrv4z2hWAPLGIDZfZN2DILY= 2024-04-16T20:15:31.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BjHsTrv4z2hWAPLGIDZfZN2DILY=', 'Content-Length': '0'} 2024-04-16T20:15:31.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042863b6feb38878f-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.097 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:31.097 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:31.097 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.097 DEBUG:boto:Path: /yrjefh-54/obj33 2024-04-16T20:15:31.097 DEBUG:boto:Data: 2024-04-16T20:15:31.097 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.097 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.097 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.098 DEBUG:boto:Params: {} 2024-04-16T20:15:31.098 DEBUG:boto:Token: None 2024-04-16T20:15:31.098 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj33 2024-04-16T20:15:31.098 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BjHsTrv4z2hWAPLGIDZfZN2DILY= 2024-04-16T20:15:31.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BjHsTrv4z2hWAPLGIDZfZN2DILY=', 'Content-Length': '0'} 2024-04-16T20:15:31.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000324205a64fface2-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:15:31.101 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:31.101 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:31.101 DEBUG:boto:Method: GET 2024-04-16T20:15:31.101 DEBUG:boto:Path: /yrjefh-54/obj33 2024-04-16T20:15:31.101 DEBUG:boto:Data: 2024-04-16T20:15:31.101 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.101 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.102 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.102 DEBUG:boto:Params: {} 2024-04-16T20:15:31.102 DEBUG:boto:Token: None 2024-04-16T20:15:31.102 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj33 2024-04-16T20:15:31.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XuIaQiRW2bXVX3BgGaRL+B+a2WM= 2024-04-16T20:15:31.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XuIaQiRW2bXVX3BgGaRL+B+a2WM=', 'Content-Length': '0'} 2024-04-16T20:15:31.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf2c78186486101-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.146 DEBUG:boto:path=/yrjefh-54/obj33 2024-04-16T20:15:31.146 DEBUG:boto:auth_path=/yrjefh-54/obj33 2024-04-16T20:15:31.147 DEBUG:boto:Method: GET 2024-04-16T20:15:31.147 DEBUG:boto:Path: /yrjefh-54/obj33 2024-04-16T20:15:31.147 DEBUG:boto:Data: 2024-04-16T20:15:31.147 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.147 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.147 DEBUG:boto:Params: {} 2024-04-16T20:15:31.147 DEBUG:boto:Token: None 2024-04-16T20:15:31.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj33 2024-04-16T20:15:31.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XuIaQiRW2bXVX3BgGaRL+B+a2WM= 2024-04-16T20:15:31.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XuIaQiRW2bXVX3BgGaRL+B+a2WM=', 'Content-Length': '0'} 2024-04-16T20:15:31.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a855a3c47e1589-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.150 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.150 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.150 DEBUG:boto:path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.150 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.151 DEBUG:boto:Method: GET 2024-04-16T20:15:31.151 DEBUG:boto:Path: /yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.151 DEBUG:boto:Data: 2024-04-16T20:15:31.151 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.151 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.151 DEBUG:boto:Params: {} 2024-04-16T20:15:31.151 DEBUG:boto:Token: None 2024-04-16T20:15:31.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4ff8636f38da755-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.154 DEBUG:boto:b'yrjefh-54obj331000falseobj332024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.154 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.154 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.155 DEBUG:boto:path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.155 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.155 DEBUG:boto:Method: GET 2024-04-16T20:15:31.155 DEBUG:boto:Path: /yrjefh-54/?prefix=obj33 2024-04-16T20:15:31.155 DEBUG:boto:Data: 2024-04-16T20:15:31.155 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.155 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.155 DEBUG:boto:Params: {} 2024-04-16T20:15:31.155 DEBUG:boto:Token: None 2024-04-16T20:15:31.156 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815404a2e0aaceab-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.159 DEBUG:boto:b'yrjefh-54obj331000falseobj332024-04-16T20:12:50.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:15:31.160 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.160 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.160 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.160 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.160 DEBUG:boto:Method: GET 2024-04-16T20:15:31.160 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.160 DEBUG:boto:Data: 2024-04-16T20:15:31.161 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.161 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.161 DEBUG:boto:Params: {} 2024-04-16T20:15:31.161 DEBUG:boto:Token: None 2024-04-16T20:15:31.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI= 2024-04-16T20:15:31.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI=', 'Content-Length': '0'} 2024-04-16T20:15:31.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b893c9dcfb6b3e-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.206 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.207 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.207 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.207 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.207 DEBUG:boto:Method: GET 2024-04-16T20:15:31.207 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.207 DEBUG:boto:Data: 2024-04-16T20:15:31.207 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.207 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.207 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.207 DEBUG:boto:Params: {} 2024-04-16T20:15:31.207 DEBUG:boto:Token: None 2024-04-16T20:15:31.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI= 2024-04-16T20:15:31.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI=', 'Content-Length': '0'} 2024-04-16T20:15:31.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f8ef463a67b026-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.210 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.210 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.210 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.210 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.210 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.210 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.210 DEBUG:boto:Data: 2024-04-16T20:15:31.210 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.210 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.211 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.211 DEBUG:boto:Params: {} 2024-04-16T20:15:31.211 DEBUG:boto:Token: None 2024-04-16T20:15:31.211 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gdatqm2dQb3Mo/rfiwt7vk61aKg= 2024-04-16T20:15:31.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gdatqm2dQb3Mo/rfiwt7vk61aKg=', 'Content-Length': '0'} 2024-04-16T20:15:31.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002325ac3d7007a7fa-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.213 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.213 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.214 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.214 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.214 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.214 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.214 DEBUG:boto:Data: 2024-04-16T20:15:31.214 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.214 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.214 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.214 DEBUG:boto:Params: {} 2024-04-16T20:15:31.214 DEBUG:boto:Token: None 2024-04-16T20:15:31.215 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Gdatqm2dQb3Mo/rfiwt7vk61aKg= 2024-04-16T20:15:31.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gdatqm2dQb3Mo/rfiwt7vk61aKg=', 'Content-Length': '0'} 2024-04-16T20:15:31.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bdfff960e89f130-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:15:31.217 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.217 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.218 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.218 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.218 DEBUG:boto:Method: GET 2024-04-16T20:15:31.218 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.218 DEBUG:boto:Data: 2024-04-16T20:15:31.218 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.218 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.218 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.218 DEBUG:boto:Params: {} 2024-04-16T20:15:31.218 DEBUG:boto:Token: None 2024-04-16T20:15:31.218 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI= 2024-04-16T20:15:31.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI=', 'Content-Length': '0'} 2024-04-16T20:15:31.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db714a0c67bc043-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.262 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.262 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.263 DEBUG:boto:path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.263 DEBUG:boto:auth_path=/yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.263 DEBUG:boto:Method: GET 2024-04-16T20:15:31.263 DEBUG:boto:Path: /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.263 DEBUG:boto:Data: 2024-04-16T20:15:31.263 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.263 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.263 DEBUG:boto:Params: {} 2024-04-16T20:15:31.263 DEBUG:boto:Token: None 2024-04-16T20:15:31.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34?versionId=null 2024-04-16T20:15:31.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI= 2024-04-16T20:15:31.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZKn1Nsj99sWw5DQ7XtUnWZwHVWI=', 'Content-Length': '0'} 2024-04-16T20:15:31.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a27dcaa95713c3b6-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.266 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.267 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.267 DEBUG:boto:path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.267 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.267 DEBUG:boto:Method: GET 2024-04-16T20:15:31.267 DEBUG:boto:Path: /yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.267 DEBUG:boto:Data: 2024-04-16T20:15:31.267 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.267 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.267 DEBUG:boto:Params: {} 2024-04-16T20:15:31.267 DEBUG:boto:Token: None 2024-04-16T20:15:31.268 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d5e08f8bbebe3d-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.270 DEBUG:boto:b'yrjefh-54obj341000falseobj342024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.271 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.271 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.271 DEBUG:boto:path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.271 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.271 DEBUG:boto:Method: GET 2024-04-16T20:15:31.271 DEBUG:boto:Path: /yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.271 DEBUG:boto:Data: 2024-04-16T20:15:31.271 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.271 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.271 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.272 DEBUG:boto:Params: {} 2024-04-16T20:15:31.272 DEBUG:boto:Token: None 2024-04-16T20:15:31.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad26518e28d1fbd7-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.275 DEBUG:boto:b'yrjefh-54obj341000falseobj342024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.276 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.276 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.276 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.276 DEBUG:boto:Path: /yrjefh-54/obj34 2024-04-16T20:15:31.276 DEBUG:boto:Data: 2024-04-16T20:15:31.276 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.276 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.276 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.276 DEBUG:boto:Params: {} 2024-04-16T20:15:31.277 DEBUG:boto:Token: None 2024-04-16T20:15:31.277 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34 2024-04-16T20:15:31.277 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RnD8NwYXuO8KCv5HYwLZQ7o1Nj4= 2024-04-16T20:15:31.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RnD8NwYXuO8KCv5HYwLZQ7o1Nj4=', 'Content-Length': '0'} 2024-04-16T20:15:31.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160b4320a29b6127-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.280 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.280 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.280 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.280 DEBUG:boto:Path: /yrjefh-54/obj34 2024-04-16T20:15:31.280 DEBUG:boto:Data: 2024-04-16T20:15:31.280 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.280 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.280 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.280 DEBUG:boto:Params: {} 2024-04-16T20:15:31.281 DEBUG:boto:Token: None 2024-04-16T20:15:31.281 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34 2024-04-16T20:15:31.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RnD8NwYXuO8KCv5HYwLZQ7o1Nj4= 2024-04-16T20:15:31.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RnD8NwYXuO8KCv5HYwLZQ7o1Nj4=', 'Content-Length': '0'} 2024-04-16T20:15:31.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb252d617efbc604-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:15:31.283 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.283 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.283 DEBUG:boto:Method: GET 2024-04-16T20:15:31.283 DEBUG:boto:Path: /yrjefh-54/obj34 2024-04-16T20:15:31.283 DEBUG:boto:Data: 2024-04-16T20:15:31.283 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.283 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.283 DEBUG:boto:Params: {} 2024-04-16T20:15:31.283 DEBUG:boto:Token: None 2024-04-16T20:15:31.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34 2024-04-16T20:15:31.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4blXHD60KY+RyfmpLIKLYJSz1xs= 2024-04-16T20:15:31.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4blXHD60KY+RyfmpLIKLYJSz1xs=', 'Content-Length': '0'} 2024-04-16T20:15:31.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:14:25 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e284f039816a636-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.326 DEBUG:boto:path=/yrjefh-54/obj34 2024-04-16T20:15:31.326 DEBUG:boto:auth_path=/yrjefh-54/obj34 2024-04-16T20:15:31.327 DEBUG:boto:Method: GET 2024-04-16T20:15:31.327 DEBUG:boto:Path: /yrjefh-54/obj34 2024-04-16T20:15:31.327 DEBUG:boto:Data: 2024-04-16T20:15:31.327 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.327 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.327 DEBUG:boto:Params: {} 2024-04-16T20:15:31.327 DEBUG:boto:Token: None 2024-04-16T20:15:31.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj34 2024-04-16T20:15:31.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4blXHD60KY+RyfmpLIKLYJSz1xs= 2024-04-16T20:15:31.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4blXHD60KY+RyfmpLIKLYJSz1xs=', 'Content-Length': '0'} 2024-04-16T20:15:31.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004121b21291de3a40-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.330 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.330 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.330 DEBUG:boto:path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.330 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.330 DEBUG:boto:Method: GET 2024-04-16T20:15:31.330 DEBUG:boto:Path: /yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.331 DEBUG:boto:Data: 2024-04-16T20:15:31.331 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.331 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.331 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.331 DEBUG:boto:Params: {} 2024-04-16T20:15:31.331 DEBUG:boto:Token: None 2024-04-16T20:15:31.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe1a938e8d67718-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.334 DEBUG:boto:b'yrjefh-54obj341000falseobj342024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.334 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.334 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.334 DEBUG:boto:path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.334 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.334 DEBUG:boto:Method: GET 2024-04-16T20:15:31.334 DEBUG:boto:Path: /yrjefh-54/?prefix=obj34 2024-04-16T20:15:31.335 DEBUG:boto:Data: 2024-04-16T20:15:31.335 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.335 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.335 DEBUG:boto:Params: {} 2024-04-16T20:15:31.335 DEBUG:boto:Token: None 2024-04-16T20:15:31.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bfe23e423d4abea-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.339 DEBUG:boto:b'yrjefh-54obj341000falseobj342024-04-16T20:12:50.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:15:31.339 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.339 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.339 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.339 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.339 DEBUG:boto:Method: GET 2024-04-16T20:15:31.339 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.339 DEBUG:boto:Data: 2024-04-16T20:15:31.340 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.340 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.340 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.340 DEBUG:boto:Params: {} 2024-04-16T20:15:31.340 DEBUG:boto:Token: None 2024-04-16T20:15:31.340 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.340 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs= 2024-04-16T20:15:31.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs=', 'Content-Length': '0'} 2024-04-16T20:15:31.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531d9c0b2d6a1807-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.382 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.382 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.383 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.383 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.383 DEBUG:boto:Method: GET 2024-04-16T20:15:31.383 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.383 DEBUG:boto:Data: 2024-04-16T20:15:31.383 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.383 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.383 DEBUG:boto:Params: {} 2024-04-16T20:15:31.383 DEBUG:boto:Token: None 2024-04-16T20:15:31.383 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs= 2024-04-16T20:15:31.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs=', 'Content-Length': '0'} 2024-04-16T20:15:31.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d968b078ea945d78-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.386 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.386 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.386 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.386 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.386 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.386 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.387 DEBUG:boto:Data: 2024-04-16T20:15:31.387 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.387 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.387 DEBUG:boto:Params: {} 2024-04-16T20:15:31.387 DEBUG:boto:Token: None 2024-04-16T20:15:31.387 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/GPeohJ+OOQPUY+pQk/z6b4Sl+I= 2024-04-16T20:15:31.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GPeohJ+OOQPUY+pQk/z6b4Sl+I=', 'Content-Length': '0'} 2024-04-16T20:15:31.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002914450d3f324a30-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.389 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.389 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.389 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.389 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.390 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.390 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.390 DEBUG:boto:Data: 2024-04-16T20:15:31.390 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.390 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.390 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.390 DEBUG:boto:Params: {} 2024-04-16T20:15:31.390 DEBUG:boto:Token: None 2024-04-16T20:15:31.390 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/GPeohJ+OOQPUY+pQk/z6b4Sl+I= 2024-04-16T20:15:31.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GPeohJ+OOQPUY+pQk/z6b4Sl+I=', 'Content-Length': '0'} 2024-04-16T20:15:31.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa3969cdb5e4fac-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:15:31.393 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.393 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.393 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.393 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.393 DEBUG:boto:Method: GET 2024-04-16T20:15:31.393 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.393 DEBUG:boto:Data: 2024-04-16T20:15:31.394 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.394 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.394 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.394 DEBUG:boto:Params: {} 2024-04-16T20:15:31.394 DEBUG:boto:Token: None 2024-04-16T20:15:31.394 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.394 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs= 2024-04-16T20:15:31.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs=', 'Content-Length': '0'} 2024-04-16T20:15:31.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352c083866f003ab-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.438 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.439 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.439 DEBUG:boto:path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.439 DEBUG:boto:auth_path=/yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.439 DEBUG:boto:Method: GET 2024-04-16T20:15:31.439 DEBUG:boto:Path: /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.439 DEBUG:boto:Data: 2024-04-16T20:15:31.439 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.439 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.439 DEBUG:boto:Params: {} 2024-04-16T20:15:31.439 DEBUG:boto:Token: None 2024-04-16T20:15:31.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37?versionId=null 2024-04-16T20:15:31.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs= 2024-04-16T20:15:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zz5nLclc8QalXi98PYuC4+4bbcs=', 'Content-Length': '0'} 2024-04-16T20:15:31.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0a18980f818505-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.442 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.442 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.442 DEBUG:boto:path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.442 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.443 DEBUG:boto:Method: GET 2024-04-16T20:15:31.443 DEBUG:boto:Path: /yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.443 DEBUG:boto:Data: 2024-04-16T20:15:31.443 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.443 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.443 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.443 DEBUG:boto:Params: {} 2024-04-16T20:15:31.443 DEBUG:boto:Token: None 2024-04-16T20:15:31.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a43c50b456e419a-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.486 DEBUG:boto:b'yrjefh-54obj371000falseobj372024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.487 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.487 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.487 DEBUG:boto:path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.487 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.487 DEBUG:boto:Method: GET 2024-04-16T20:15:31.487 DEBUG:boto:Path: /yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.487 DEBUG:boto:Data: 2024-04-16T20:15:31.487 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.487 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.487 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.488 DEBUG:boto:Params: {} 2024-04-16T20:15:31.488 DEBUG:boto:Token: None 2024-04-16T20:15:31.488 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb072a3ddab350aa-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.492 DEBUG:boto:b'yrjefh-54obj371000falseobj372024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.493 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.493 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.493 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.493 DEBUG:boto:Path: /yrjefh-54/obj37 2024-04-16T20:15:31.493 DEBUG:boto:Data: 2024-04-16T20:15:31.493 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.493 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.493 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.493 DEBUG:boto:Params: {} 2024-04-16T20:15:31.493 DEBUG:boto:Token: None 2024-04-16T20:15:31.494 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37 2024-04-16T20:15:31.494 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K+X7fz3cWdDwQy0mJakFS/GNn0E= 2024-04-16T20:15:31.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K+X7fz3cWdDwQy0mJakFS/GNn0E=', 'Content-Length': '0'} 2024-04-16T20:15:31.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2d43d11acae018-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.496 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.496 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.496 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.496 DEBUG:boto:Path: /yrjefh-54/obj37 2024-04-16T20:15:31.497 DEBUG:boto:Data: 2024-04-16T20:15:31.497 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.497 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.497 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.497 DEBUG:boto:Params: {} 2024-04-16T20:15:31.497 DEBUG:boto:Token: None 2024-04-16T20:15:31.497 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37 2024-04-16T20:15:31.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K+X7fz3cWdDwQy0mJakFS/GNn0E= 2024-04-16T20:15:31.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K+X7fz3cWdDwQy0mJakFS/GNn0E=', 'Content-Length': '0'} 2024-04-16T20:15:31.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001534c3e7a1806ed1-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:15:31.499 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.499 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.499 DEBUG:boto:Method: GET 2024-04-16T20:15:31.500 DEBUG:boto:Path: /yrjefh-54/obj37 2024-04-16T20:15:31.500 DEBUG:boto:Data: 2024-04-16T20:15:31.500 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.500 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.500 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.500 DEBUG:boto:Params: {} 2024-04-16T20:15:31.500 DEBUG:boto:Token: None 2024-04-16T20:15:31.500 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37 2024-04-16T20:15:31.500 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAdBJA5dyfq+CPNObVhfQGdLJJc= 2024-04-16T20:15:31.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAdBJA5dyfq+CPNObVhfQGdLJJc=', 'Content-Length': '0'} 2024-04-16T20:15:31.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f8b2f4fb787bad-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.546 DEBUG:boto:path=/yrjefh-54/obj37 2024-04-16T20:15:31.546 DEBUG:boto:auth_path=/yrjefh-54/obj37 2024-04-16T20:15:31.547 DEBUG:boto:Method: GET 2024-04-16T20:15:31.547 DEBUG:boto:Path: /yrjefh-54/obj37 2024-04-16T20:15:31.547 DEBUG:boto:Data: 2024-04-16T20:15:31.547 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.547 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.547 DEBUG:boto:Params: {} 2024-04-16T20:15:31.547 DEBUG:boto:Token: None 2024-04-16T20:15:31.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj37 2024-04-16T20:15:31.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FAdBJA5dyfq+CPNObVhfQGdLJJc= 2024-04-16T20:15:31.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FAdBJA5dyfq+CPNObVhfQGdLJJc=', 'Content-Length': '0'} 2024-04-16T20:15:31.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd85e25a6b8b20f6-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.550 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.550 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.550 DEBUG:boto:path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.550 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.551 DEBUG:boto:Method: GET 2024-04-16T20:15:31.551 DEBUG:boto:Path: /yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.551 DEBUG:boto:Data: 2024-04-16T20:15:31.551 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.551 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.551 DEBUG:boto:Params: {} 2024-04-16T20:15:31.551 DEBUG:boto:Token: None 2024-04-16T20:15:31.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edda5e50ff667c86-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.554 DEBUG:boto:b'yrjefh-54obj371000falseobj372024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.555 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.555 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.555 DEBUG:boto:path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.555 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.555 DEBUG:boto:Method: GET 2024-04-16T20:15:31.555 DEBUG:boto:Path: /yrjefh-54/?prefix=obj37 2024-04-16T20:15:31.555 DEBUG:boto:Data: 2024-04-16T20:15:31.555 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.555 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.555 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.556 DEBUG:boto:Params: {} 2024-04-16T20:15:31.556 DEBUG:boto:Token: None 2024-04-16T20:15:31.556 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.556 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a5c1410bcfcc31-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.560 DEBUG:boto:b'yrjefh-54obj371000falseobj372024-04-16T20:12:51.987Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:15:31.560 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.560 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.560 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.561 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.561 DEBUG:boto:Method: GET 2024-04-16T20:15:31.561 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.561 DEBUG:boto:Data: 2024-04-16T20:15:31.561 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.561 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.561 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.561 DEBUG:boto:Params: {} 2024-04-16T20:15:31.561 DEBUG:boto:Token: None 2024-04-16T20:15:31.561 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.562 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI= 2024-04-16T20:15:31.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI=', 'Content-Length': '0'} 2024-04-16T20:15:31.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080641a28d6549871-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.606 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.606 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.607 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.607 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.607 DEBUG:boto:Method: GET 2024-04-16T20:15:31.607 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.607 DEBUG:boto:Data: 2024-04-16T20:15:31.607 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.607 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.607 DEBUG:boto:Params: {} 2024-04-16T20:15:31.607 DEBUG:boto:Token: None 2024-04-16T20:15:31.608 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.608 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI= 2024-04-16T20:15:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI=', 'Content-Length': '0'} 2024-04-16T20:15:31.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002696141af3ac4bae-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.611 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.611 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.611 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.611 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.611 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.611 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.611 DEBUG:boto:Data: 2024-04-16T20:15:31.611 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.611 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.612 DEBUG:boto:Params: {} 2024-04-16T20:15:31.612 DEBUG:boto:Token: None 2024-04-16T20:15:31.612 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.612 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:23H1zv8/N9X4hh2XyGLb4bkGcOw= 2024-04-16T20:15:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:23H1zv8/N9X4hh2XyGLb4bkGcOw=', 'Content-Length': '0'} 2024-04-16T20:15:31.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a499628c11dc60f-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.615 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.615 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.615 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.615 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.615 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.615 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.615 DEBUG:boto:Data: 2024-04-16T20:15:31.615 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.616 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.616 DEBUG:boto:Params: {} 2024-04-16T20:15:31.616 DEBUG:boto:Token: None 2024-04-16T20:15:31.616 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.616 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:23H1zv8/N9X4hh2XyGLb4bkGcOw= 2024-04-16T20:15:31.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:23H1zv8/N9X4hh2XyGLb4bkGcOw=', 'Content-Length': '0'} 2024-04-16T20:15:31.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae5c7fa1bc1bd9f-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:15:31.619 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.619 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.619 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.619 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.619 DEBUG:boto:Method: GET 2024-04-16T20:15:31.619 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.619 DEBUG:boto:Data: 2024-04-16T20:15:31.619 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.619 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.619 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.619 DEBUG:boto:Params: {} 2024-04-16T20:15:31.620 DEBUG:boto:Token: None 2024-04-16T20:15:31.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI= 2024-04-16T20:15:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI=', 'Content-Length': '0'} 2024-04-16T20:15:31.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d467a86c739362-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.666 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.667 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.667 DEBUG:boto:path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.667 DEBUG:boto:auth_path=/yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.667 DEBUG:boto:Method: GET 2024-04-16T20:15:31.667 DEBUG:boto:Path: /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.667 DEBUG:boto:Data: 2024-04-16T20:15:31.667 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.667 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.667 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.667 DEBUG:boto:Params: {} 2024-04-16T20:15:31.667 DEBUG:boto:Token: None 2024-04-16T20:15:31.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38?versionId=null 2024-04-16T20:15:31.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI= 2024-04-16T20:15:31.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:e/8uSlWbIYeiHtExm0yxia+dsdI=', 'Content-Length': '0'} 2024-04-16T20:15:31.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f81beb3e03a2c2-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.670 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.670 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.670 DEBUG:boto:path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.671 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.671 DEBUG:boto:Method: GET 2024-04-16T20:15:31.671 DEBUG:boto:Path: /yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.671 DEBUG:boto:Data: 2024-04-16T20:15:31.671 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.671 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.671 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.671 DEBUG:boto:Params: {} 2024-04-16T20:15:31.671 DEBUG:boto:Token: None 2024-04-16T20:15:31.671 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.672 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e11c3f228973af-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.674 DEBUG:boto:b'yrjefh-54obj381000falseobj382024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.675 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.675 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.675 DEBUG:boto:path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.675 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.675 DEBUG:boto:Method: GET 2024-04-16T20:15:31.675 DEBUG:boto:Path: /yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.675 DEBUG:boto:Data: 2024-04-16T20:15:31.675 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.675 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.675 DEBUG:boto:Params: {} 2024-04-16T20:15:31.676 DEBUG:boto:Token: None 2024-04-16T20:15:31.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f50286a11f955d-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.680 DEBUG:boto:b'yrjefh-54obj381000falseobj382024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.680 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.680 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.680 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.680 DEBUG:boto:Path: /yrjefh-54/obj38 2024-04-16T20:15:31.680 DEBUG:boto:Data: 2024-04-16T20:15:31.681 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.681 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.681 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.681 DEBUG:boto:Params: {} 2024-04-16T20:15:31.681 DEBUG:boto:Token: None 2024-04-16T20:15:31.681 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38 2024-04-16T20:15:31.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TFC1H62R96x05KO0u8iCUsdRaJI= 2024-04-16T20:15:31.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TFC1H62R96x05KO0u8iCUsdRaJI=', 'Content-Length': '0'} 2024-04-16T20:15:31.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f729e2aa611c9472-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.684 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.684 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.684 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.684 DEBUG:boto:Path: /yrjefh-54/obj38 2024-04-16T20:15:31.684 DEBUG:boto:Data: 2024-04-16T20:15:31.684 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.684 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.684 DEBUG:boto:Params: {} 2024-04-16T20:15:31.684 DEBUG:boto:Token: None 2024-04-16T20:15:31.685 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38 2024-04-16T20:15:31.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TFC1H62R96x05KO0u8iCUsdRaJI= 2024-04-16T20:15:31.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TFC1H62R96x05KO0u8iCUsdRaJI=', 'Content-Length': '0'} 2024-04-16T20:15:31.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5cdc8061aed498-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:15:31.687 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.687 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.687 DEBUG:boto:Method: GET 2024-04-16T20:15:31.687 DEBUG:boto:Path: /yrjefh-54/obj38 2024-04-16T20:15:31.688 DEBUG:boto:Data: 2024-04-16T20:15:31.688 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.688 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.688 DEBUG:boto:Params: {} 2024-04-16T20:15:31.688 DEBUG:boto:Token: None 2024-04-16T20:15:31.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38 2024-04-16T20:15:31.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Czz6pOvohMh8mqgu/g6Z6rNQHBQ= 2024-04-16T20:15:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Czz6pOvohMh8mqgu/g6Z6rNQHBQ=', 'Content-Length': '0'} 2024-04-16T20:15:31.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8538f3f0a787610-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.734 DEBUG:boto:path=/yrjefh-54/obj38 2024-04-16T20:15:31.734 DEBUG:boto:auth_path=/yrjefh-54/obj38 2024-04-16T20:15:31.735 DEBUG:boto:Method: GET 2024-04-16T20:15:31.735 DEBUG:boto:Path: /yrjefh-54/obj38 2024-04-16T20:15:31.735 DEBUG:boto:Data: 2024-04-16T20:15:31.735 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.735 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.735 DEBUG:boto:Params: {} 2024-04-16T20:15:31.735 DEBUG:boto:Token: None 2024-04-16T20:15:31.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj38 2024-04-16T20:15:31.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Czz6pOvohMh8mqgu/g6Z6rNQHBQ= 2024-04-16T20:15:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Czz6pOvohMh8mqgu/g6Z6rNQHBQ=', 'Content-Length': '0'} 2024-04-16T20:15:31.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006678b803e510acdc-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.738 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.738 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.738 DEBUG:boto:path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.738 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.739 DEBUG:boto:Method: GET 2024-04-16T20:15:31.739 DEBUG:boto:Path: /yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.739 DEBUG:boto:Data: 2024-04-16T20:15:31.739 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.739 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.739 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.739 DEBUG:boto:Params: {} 2024-04-16T20:15:31.739 DEBUG:boto:Token: None 2024-04-16T20:15:31.739 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a749ec6b2fb698-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.782 DEBUG:boto:b'yrjefh-54obj381000falseobj382024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.783 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.783 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.783 DEBUG:boto:path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.783 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.783 DEBUG:boto:Method: GET 2024-04-16T20:15:31.783 DEBUG:boto:Path: /yrjefh-54/?prefix=obj38 2024-04-16T20:15:31.783 DEBUG:boto:Data: 2024-04-16T20:15:31.783 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.784 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.784 DEBUG:boto:Params: {} 2024-04-16T20:15:31.784 DEBUG:boto:Token: None 2024-04-16T20:15:31.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b38d16e519820f4-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.788 DEBUG:boto:b'yrjefh-54obj381000falseobj382024-04-16T20:12:50.754Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:15:31.788 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.788 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.789 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.789 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.789 DEBUG:boto:Method: GET 2024-04-16T20:15:31.789 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.789 DEBUG:boto:Data: 2024-04-16T20:15:31.789 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.789 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.789 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.789 DEBUG:boto:Params: {} 2024-04-16T20:15:31.789 DEBUG:boto:Token: None 2024-04-16T20:15:31.790 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s= 2024-04-16T20:15:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s=', 'Content-Length': '0'} 2024-04-16T20:15:31.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7c1897b474f0bf0-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.834 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.835 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.835 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.835 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.835 DEBUG:boto:Method: GET 2024-04-16T20:15:31.835 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.835 DEBUG:boto:Data: 2024-04-16T20:15:31.835 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.835 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.835 DEBUG:boto:Params: {} 2024-04-16T20:15:31.836 DEBUG:boto:Token: None 2024-04-16T20:15:31.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s= 2024-04-16T20:15:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s=', 'Content-Length': '0'} 2024-04-16T20:15:31.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d25cf2e45f51bad-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.838 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.839 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.839 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.839 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.839 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.839 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.839 DEBUG:boto:Data: 2024-04-16T20:15:31.839 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.839 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.839 DEBUG:boto:Params: {} 2024-04-16T20:15:31.840 DEBUG:boto:Token: None 2024-04-16T20:15:31.840 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kAYv3OQHuY28JReJLjsx2JYytOo= 2024-04-16T20:15:31.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kAYv3OQHuY28JReJLjsx2JYytOo=', 'Content-Length': '0'} 2024-04-16T20:15:31.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe814d773c30a50-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.842 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.843 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.843 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.843 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.843 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.843 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.843 DEBUG:boto:Data: 2024-04-16T20:15:31.843 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.843 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.844 DEBUG:boto:Params: {} 2024-04-16T20:15:31.844 DEBUG:boto:Token: None 2024-04-16T20:15:31.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kAYv3OQHuY28JReJLjsx2JYytOo= 2024-04-16T20:15:31.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kAYv3OQHuY28JReJLjsx2JYytOo=', 'Content-Length': '0'} 2024-04-16T20:15:31.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b49dba94efd27f-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:15:31.847 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.847 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.847 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.847 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.847 DEBUG:boto:Method: GET 2024-04-16T20:15:31.847 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.847 DEBUG:boto:Data: 2024-04-16T20:15:31.847 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.847 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.848 DEBUG:boto:Params: {} 2024-04-16T20:15:31.848 DEBUG:boto:Token: None 2024-04-16T20:15:31.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s= 2024-04-16T20:15:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s=', 'Content-Length': '0'} 2024-04-16T20:15:31.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f97423f48d5c04-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.890 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.891 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.891 DEBUG:boto:path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.891 DEBUG:boto:auth_path=/yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.891 DEBUG:boto:Method: GET 2024-04-16T20:15:31.891 DEBUG:boto:Path: /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.891 DEBUG:boto:Data: 2024-04-16T20:15:31.891 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.891 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.891 DEBUG:boto:Params: {} 2024-04-16T20:15:31.892 DEBUG:boto:Token: None 2024-04-16T20:15:31.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44?versionId=null 2024-04-16T20:15:31.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s= 2024-04-16T20:15:31.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5WEBWB8izm8NwvYMX8oyzAJI59s=', 'Content-Length': '0'} 2024-04-16T20:15:31.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa29a39bfec4dad3-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.894 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.894 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.895 DEBUG:boto:path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.895 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.895 DEBUG:boto:Method: GET 2024-04-16T20:15:31.895 DEBUG:boto:Path: /yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.895 DEBUG:boto:Data: 2024-04-16T20:15:31.895 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.895 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.895 DEBUG:boto:Params: {} 2024-04-16T20:15:31.895 DEBUG:boto:Token: None 2024-04-16T20:15:31.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d58fc8544f21150-00661edc63-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.942 DEBUG:boto:b'yrjefh-54obj441000falseobj442024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.943 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:31.943 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:31.943 DEBUG:boto:path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.943 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.943 DEBUG:boto:Method: GET 2024-04-16T20:15:31.943 DEBUG:boto:Path: /yrjefh-54/?prefix=obj44 2024-04-16T20:15:31.943 DEBUG:boto:Data: 2024-04-16T20:15:31.943 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.944 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.944 DEBUG:boto:Params: {} 2024-04-16T20:15:31.944 DEBUG:boto:Token: None 2024-04-16T20:15:31.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/ 2024-04-16T20:15:31.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w= 2024-04-16T20:15:31.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PKPPaAxsv1IKjCf9EcD0ViQBa+w=', 'Content-Length': '0'} 2024-04-16T20:15:31.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5283841f293caa-00661edc63-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.948 DEBUG:boto:b'yrjefh-54obj441000falseobj442024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:31.949 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.949 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.949 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.949 DEBUG:boto:Path: /yrjefh-54/obj44 2024-04-16T20:15:31.949 DEBUG:boto:Data: 2024-04-16T20:15:31.949 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.949 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.949 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.949 DEBUG:boto:Params: {} 2024-04-16T20:15:31.950 DEBUG:boto:Token: None 2024-04-16T20:15:31.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44 2024-04-16T20:15:31.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YAMujBScFiO8Kw3O/llsd63/ZYo= 2024-04-16T20:15:31.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YAMujBScFiO8Kw3O/llsd63/ZYo=', 'Content-Length': '0'} 2024-04-16T20:15:31.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679119292fd2f353-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.952 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.953 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.953 DEBUG:boto:Method: HEAD 2024-04-16T20:15:31.953 DEBUG:boto:Path: /yrjefh-54/obj44 2024-04-16T20:15:31.953 DEBUG:boto:Data: 2024-04-16T20:15:31.953 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.953 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:31.953 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.953 DEBUG:boto:Params: {} 2024-04-16T20:15:31.954 DEBUG:boto:Token: None 2024-04-16T20:15:31.954 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44 2024-04-16T20:15:31.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YAMujBScFiO8Kw3O/llsd63/ZYo= 2024-04-16T20:15:31.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YAMujBScFiO8Kw3O/llsd63/ZYo=', 'Content-Length': '0'} 2024-04-16T20:15:31.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6208e3aae555bf-00661edc63-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:31.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:15:31.956 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:31.957 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:31.957 DEBUG:boto:Method: GET 2024-04-16T20:15:31.957 DEBUG:boto:Path: /yrjefh-54/obj44 2024-04-16T20:15:31.957 DEBUG:boto:Data: 2024-04-16T20:15:31.957 DEBUG:boto:Headers: {} 2024-04-16T20:15:31.957 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:31.957 DEBUG:boto:Port: 8000 2024-04-16T20:15:31.957 DEBUG:boto:Params: {} 2024-04-16T20:15:31.957 DEBUG:boto:Token: None 2024-04-16T20:15:31.958 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:31 GMT /yrjefh-54/obj44 2024-04-16T20:15:31.958 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LiXxiVMitngJEDp/JKEA+oIAErA= 2024-04-16T20:15:31.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:31 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LiXxiVMitngJEDp/JKEA+oIAErA=', 'Content-Length': '0'} 2024-04-16T20:15:31.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a86d7322b1b7625-00661edc63-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.002 DEBUG:boto:path=/yrjefh-54/obj44 2024-04-16T20:15:32.003 DEBUG:boto:auth_path=/yrjefh-54/obj44 2024-04-16T20:15:32.003 DEBUG:boto:Method: GET 2024-04-16T20:15:32.003 DEBUG:boto:Path: /yrjefh-54/obj44 2024-04-16T20:15:32.003 DEBUG:boto:Data: 2024-04-16T20:15:32.003 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.003 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.003 DEBUG:boto:Params: {} 2024-04-16T20:15:32.003 DEBUG:boto:Token: None 2024-04-16T20:15:32.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj44 2024-04-16T20:15:32.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o4nL+inDuLrQTuuQPy5Govzuedk= 2024-04-16T20:15:32.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o4nL+inDuLrQTuuQPy5Govzuedk=', 'Content-Length': '0'} 2024-04-16T20:15:32.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631f3cd6a32b076b-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.006 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.006 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.006 DEBUG:boto:path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.007 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.007 DEBUG:boto:Method: GET 2024-04-16T20:15:32.007 DEBUG:boto:Path: /yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.007 DEBUG:boto:Data: 2024-04-16T20:15:32.007 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.007 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.007 DEBUG:boto:Params: {} 2024-04-16T20:15:32.007 DEBUG:boto:Token: None 2024-04-16T20:15:32.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.008 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093cd1e8aafee16ae-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.010 DEBUG:boto:b'yrjefh-54obj441000falseobj442024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.011 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.011 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.011 DEBUG:boto:path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.011 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.011 DEBUG:boto:Method: GET 2024-04-16T20:15:32.011 DEBUG:boto:Path: /yrjefh-54/?prefix=obj44 2024-04-16T20:15:32.012 DEBUG:boto:Data: 2024-04-16T20:15:32.012 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.012 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.012 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.012 DEBUG:boto:Params: {} 2024-04-16T20:15:32.012 DEBUG:boto:Token: None 2024-04-16T20:15:32.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc5de462ff42c5f-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.016 DEBUG:boto:b'yrjefh-54obj441000falseobj442024-04-16T20:12:50.247Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:15:32.017 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.017 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.017 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.017 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.017 DEBUG:boto:Method: GET 2024-04-16T20:15:32.017 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.017 DEBUG:boto:Data: 2024-04-16T20:15:32.018 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.018 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.018 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.018 DEBUG:boto:Params: {} 2024-04-16T20:15:32.018 DEBUG:boto:Token: None 2024-04-16T20:15:32.018 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY= 2024-04-16T20:15:32.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY=', 'Content-Length': '0'} 2024-04-16T20:15:32.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006abc8eea57a50503-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.062 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.063 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.063 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.063 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.063 DEBUG:boto:Method: GET 2024-04-16T20:15:32.063 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.063 DEBUG:boto:Data: 2024-04-16T20:15:32.063 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.064 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.064 DEBUG:boto:Params: {} 2024-04-16T20:15:32.064 DEBUG:boto:Token: None 2024-04-16T20:15:32.064 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY= 2024-04-16T20:15:32.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY=', 'Content-Length': '0'} 2024-04-16T20:15:32.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002210559cb4887af9-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.067 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.067 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.067 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.067 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.068 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.068 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.068 DEBUG:boto:Data: 2024-04-16T20:15:32.068 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.068 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.068 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.068 DEBUG:boto:Params: {} 2024-04-16T20:15:32.068 DEBUG:boto:Token: None 2024-04-16T20:15:32.069 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.069 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m+e1hYjGMU8BMXfr/9WCpdB3+Lc= 2024-04-16T20:15:32.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m+e1hYjGMU8BMXfr/9WCpdB3+Lc=', 'Content-Length': '0'} 2024-04-16T20:15:32.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5eeef789b9f4e42-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.072 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.072 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.072 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.072 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.072 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.072 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.072 DEBUG:boto:Data: 2024-04-16T20:15:32.072 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.072 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.073 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.073 DEBUG:boto:Params: {} 2024-04-16T20:15:32.073 DEBUG:boto:Token: None 2024-04-16T20:15:32.073 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m+e1hYjGMU8BMXfr/9WCpdB3+Lc= 2024-04-16T20:15:32.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m+e1hYjGMU8BMXfr/9WCpdB3+Lc=', 'Content-Length': '0'} 2024-04-16T20:15:32.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc015a04297772e6-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:15:32.076 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.076 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.076 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.076 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.077 DEBUG:boto:Method: GET 2024-04-16T20:15:32.077 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.077 DEBUG:boto:Data: 2024-04-16T20:15:32.077 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.077 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.077 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.077 DEBUG:boto:Params: {} 2024-04-16T20:15:32.077 DEBUG:boto:Token: None 2024-04-16T20:15:32.078 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.078 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY= 2024-04-16T20:15:32.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY=', 'Content-Length': '0'} 2024-04-16T20:15:32.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fc5114d66f9789-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.122 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.122 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.123 DEBUG:boto:path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.123 DEBUG:boto:auth_path=/yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.123 DEBUG:boto:Method: GET 2024-04-16T20:15:32.123 DEBUG:boto:Path: /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.123 DEBUG:boto:Data: 2024-04-16T20:15:32.123 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.123 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.123 DEBUG:boto:Params: {} 2024-04-16T20:15:32.123 DEBUG:boto:Token: None 2024-04-16T20:15:32.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46?versionId=null 2024-04-16T20:15:32.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY= 2024-04-16T20:15:32.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v471KfwpttsDhCtP+inZHwcBAJY=', 'Content-Length': '0'} 2024-04-16T20:15:32.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c3f6a1a7dd4980-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.126 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.126 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.127 DEBUG:boto:path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.127 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.127 DEBUG:boto:Method: GET 2024-04-16T20:15:32.127 DEBUG:boto:Path: /yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.127 DEBUG:boto:Data: 2024-04-16T20:15:32.127 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.127 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.127 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.127 DEBUG:boto:Params: {} 2024-04-16T20:15:32.127 DEBUG:boto:Token: None 2024-04-16T20:15:32.128 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf381338f80e8ace-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.174 DEBUG:boto:b'yrjefh-54obj461000falseobj462024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.175 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.175 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.175 DEBUG:boto:path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.175 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.175 DEBUG:boto:Method: GET 2024-04-16T20:15:32.175 DEBUG:boto:Path: /yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.175 DEBUG:boto:Data: 2024-04-16T20:15:32.175 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.175 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.176 DEBUG:boto:Params: {} 2024-04-16T20:15:32.176 DEBUG:boto:Token: None 2024-04-16T20:15:32.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a0021ccb5680c6-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.179 DEBUG:boto:b'yrjefh-54obj461000falseobj462024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.179 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.180 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.180 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.180 DEBUG:boto:Path: /yrjefh-54/obj46 2024-04-16T20:15:32.180 DEBUG:boto:Data: 2024-04-16T20:15:32.180 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.180 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.180 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.180 DEBUG:boto:Params: {} 2024-04-16T20:15:32.180 DEBUG:boto:Token: None 2024-04-16T20:15:32.180 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46 2024-04-16T20:15:32.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv4AaSi8b+SnP42CjggiNRMa84g= 2024-04-16T20:15:32.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv4AaSi8b+SnP42CjggiNRMa84g=', 'Content-Length': '0'} 2024-04-16T20:15:32.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ab9b6ba8259856-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.183 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.183 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.183 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.183 DEBUG:boto:Path: /yrjefh-54/obj46 2024-04-16T20:15:32.184 DEBUG:boto:Data: 2024-04-16T20:15:32.184 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.184 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.184 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.184 DEBUG:boto:Params: {} 2024-04-16T20:15:32.184 DEBUG:boto:Token: None 2024-04-16T20:15:32.184 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46 2024-04-16T20:15:32.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wv4AaSi8b+SnP42CjggiNRMa84g= 2024-04-16T20:15:32.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wv4AaSi8b+SnP42CjggiNRMa84g=', 'Content-Length': '0'} 2024-04-16T20:15:32.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1338a13d8a6940-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:15:32.186 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.187 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.187 DEBUG:boto:Method: GET 2024-04-16T20:15:32.187 DEBUG:boto:Path: /yrjefh-54/obj46 2024-04-16T20:15:32.187 DEBUG:boto:Data: 2024-04-16T20:15:32.187 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.187 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.187 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.187 DEBUG:boto:Params: {} 2024-04-16T20:15:32.187 DEBUG:boto:Token: None 2024-04-16T20:15:32.188 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46 2024-04-16T20:15:32.188 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ozZdrM4G6VswS46M49Y0F97+yao= 2024-04-16T20:15:32.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ozZdrM4G6VswS46M49Y0F97+yao=', 'Content-Length': '0'} 2024-04-16T20:15:32.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c3e508d2d3c206f-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.230 DEBUG:boto:path=/yrjefh-54/obj46 2024-04-16T20:15:32.230 DEBUG:boto:auth_path=/yrjefh-54/obj46 2024-04-16T20:15:32.231 DEBUG:boto:Method: GET 2024-04-16T20:15:32.231 DEBUG:boto:Path: /yrjefh-54/obj46 2024-04-16T20:15:32.231 DEBUG:boto:Data: 2024-04-16T20:15:32.231 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.231 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.231 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.231 DEBUG:boto:Params: {} 2024-04-16T20:15:32.231 DEBUG:boto:Token: None 2024-04-16T20:15:32.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj46 2024-04-16T20:15:32.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ozZdrM4G6VswS46M49Y0F97+yao= 2024-04-16T20:15:32.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ozZdrM4G6VswS46M49Y0F97+yao=', 'Content-Length': '0'} 2024-04-16T20:15:32.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c43aab56bebf042-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.234 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.234 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.234 DEBUG:boto:path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.234 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.235 DEBUG:boto:Method: GET 2024-04-16T20:15:32.235 DEBUG:boto:Path: /yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.235 DEBUG:boto:Data: 2024-04-16T20:15:32.235 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.235 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.235 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.235 DEBUG:boto:Params: {} 2024-04-16T20:15:32.235 DEBUG:boto:Token: None 2024-04-16T20:15:32.235 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.235 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f23dd137aa89d8-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.278 DEBUG:boto:b'yrjefh-54obj461000falseobj462024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.279 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.279 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.279 DEBUG:boto:path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.279 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.279 DEBUG:boto:Method: GET 2024-04-16T20:15:32.279 DEBUG:boto:Path: /yrjefh-54/?prefix=obj46 2024-04-16T20:15:32.279 DEBUG:boto:Data: 2024-04-16T20:15:32.279 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.279 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.279 DEBUG:boto:Params: {} 2024-04-16T20:15:32.280 DEBUG:boto:Token: None 2024-04-16T20:15:32.280 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.280 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f390939b637c7a1-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.284 DEBUG:boto:b'yrjefh-54obj461000falseobj462024-04-16T20:12:51.060Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:15:32.284 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.284 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.284 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.284 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.284 DEBUG:boto:Method: GET 2024-04-16T20:15:32.285 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.285 DEBUG:boto:Data: 2024-04-16T20:15:32.285 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.285 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.285 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.285 DEBUG:boto:Params: {} 2024-04-16T20:15:32.285 DEBUG:boto:Token: None 2024-04-16T20:15:32.285 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.285 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ= 2024-04-16T20:15:32.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1e0c3c34f4fd21-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.330 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.330 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.331 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.331 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.331 DEBUG:boto:Method: GET 2024-04-16T20:15:32.331 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.331 DEBUG:boto:Data: 2024-04-16T20:15:32.331 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.331 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.331 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.331 DEBUG:boto:Params: {} 2024-04-16T20:15:32.331 DEBUG:boto:Token: None 2024-04-16T20:15:32.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ= 2024-04-16T20:15:32.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d865bb7c9c60b1f-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.334 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.334 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.334 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.334 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.334 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.335 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.335 DEBUG:boto:Data: 2024-04-16T20:15:32.335 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.335 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.335 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.335 DEBUG:boto:Params: {} 2024-04-16T20:15:32.335 DEBUG:boto:Token: None 2024-04-16T20:15:32.335 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jsvcowPMq8aOdTLcLtGE/6DqD6w= 2024-04-16T20:15:32.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jsvcowPMq8aOdTLcLtGE/6DqD6w=', 'Content-Length': '0'} 2024-04-16T20:15:32.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888482c1c5392801-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.338 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.338 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.338 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.338 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.338 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.338 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.338 DEBUG:boto:Data: 2024-04-16T20:15:32.338 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.339 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.339 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.339 DEBUG:boto:Params: {} 2024-04-16T20:15:32.339 DEBUG:boto:Token: None 2024-04-16T20:15:32.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jsvcowPMq8aOdTLcLtGE/6DqD6w= 2024-04-16T20:15:32.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jsvcowPMq8aOdTLcLtGE/6DqD6w=', 'Content-Length': '0'} 2024-04-16T20:15:32.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec9aa28f85e7bab-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:15:32.341 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.342 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.342 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.342 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.342 DEBUG:boto:Method: GET 2024-04-16T20:15:32.342 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.342 DEBUG:boto:Data: 2024-04-16T20:15:32.342 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.342 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.342 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.342 DEBUG:boto:Params: {} 2024-04-16T20:15:32.342 DEBUG:boto:Token: None 2024-04-16T20:15:32.343 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.343 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ= 2024-04-16T20:15:32.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a37ae821265541e-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.386 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.387 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.387 DEBUG:boto:path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.387 DEBUG:boto:auth_path=/yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.387 DEBUG:boto:Method: GET 2024-04-16T20:15:32.387 DEBUG:boto:Path: /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.387 DEBUG:boto:Data: 2024-04-16T20:15:32.387 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.387 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.387 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.388 DEBUG:boto:Params: {} 2024-04-16T20:15:32.388 DEBUG:boto:Token: None 2024-04-16T20:15:32.388 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51?versionId=null 2024-04-16T20:15:32.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ= 2024-04-16T20:15:32.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gqa2R95Fm4U1+1L8/fYd1ZtoaZQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035912d2b46150b51-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.391 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.391 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.391 DEBUG:boto:path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.391 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.391 DEBUG:boto:Method: GET 2024-04-16T20:15:32.391 DEBUG:boto:Path: /yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.391 DEBUG:boto:Data: 2024-04-16T20:15:32.391 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.391 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.391 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.391 DEBUG:boto:Params: {} 2024-04-16T20:15:32.392 DEBUG:boto:Token: None 2024-04-16T20:15:32.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bd167c9d2a6153a-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.438 DEBUG:boto:b'yrjefh-54obj511000falseobj512024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.439 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.439 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.439 DEBUG:boto:path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.439 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.439 DEBUG:boto:Method: GET 2024-04-16T20:15:32.439 DEBUG:boto:Path: /yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.439 DEBUG:boto:Data: 2024-04-16T20:15:32.439 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.439 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.439 DEBUG:boto:Params: {} 2024-04-16T20:15:32.440 DEBUG:boto:Token: None 2024-04-16T20:15:32.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d1683dd8e199f0-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.444 DEBUG:boto:b'yrjefh-54obj511000falseobj512024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.444 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.444 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.444 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.444 DEBUG:boto:Path: /yrjefh-54/obj51 2024-04-16T20:15:32.444 DEBUG:boto:Data: 2024-04-16T20:15:32.444 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.444 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.444 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.445 DEBUG:boto:Params: {} 2024-04-16T20:15:32.445 DEBUG:boto:Token: None 2024-04-16T20:15:32.445 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51 2024-04-16T20:15:32.445 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f3/f2WlZUtnzfrqWXYbkwMuIKaA= 2024-04-16T20:15:32.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f3/f2WlZUtnzfrqWXYbkwMuIKaA=', 'Content-Length': '0'} 2024-04-16T20:15:32.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c63dd6d0f652f7-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.447 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.447 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.448 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.448 DEBUG:boto:Path: /yrjefh-54/obj51 2024-04-16T20:15:32.448 DEBUG:boto:Data: 2024-04-16T20:15:32.448 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.448 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.448 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.448 DEBUG:boto:Params: {} 2024-04-16T20:15:32.448 DEBUG:boto:Token: None 2024-04-16T20:15:32.448 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51 2024-04-16T20:15:32.448 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:f3/f2WlZUtnzfrqWXYbkwMuIKaA= 2024-04-16T20:15:32.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:f3/f2WlZUtnzfrqWXYbkwMuIKaA=', 'Content-Length': '0'} 2024-04-16T20:15:32.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b964dac8338d29-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:15:32.450 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.451 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.451 DEBUG:boto:Method: GET 2024-04-16T20:15:32.451 DEBUG:boto:Path: /yrjefh-54/obj51 2024-04-16T20:15:32.451 DEBUG:boto:Data: 2024-04-16T20:15:32.451 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.451 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.451 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.451 DEBUG:boto:Params: {} 2024-04-16T20:15:32.451 DEBUG:boto:Token: None 2024-04-16T20:15:32.452 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51 2024-04-16T20:15:32.452 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qf6rAyDKkF4fql2sAjyKtMnscDw= 2024-04-16T20:15:32.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qf6rAyDKkF4fql2sAjyKtMnscDw=', 'Content-Length': '0'} 2024-04-16T20:15:32.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde2d14174a8d7fd-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.494 DEBUG:boto:path=/yrjefh-54/obj51 2024-04-16T20:15:32.495 DEBUG:boto:auth_path=/yrjefh-54/obj51 2024-04-16T20:15:32.495 DEBUG:boto:Method: GET 2024-04-16T20:15:32.495 DEBUG:boto:Path: /yrjefh-54/obj51 2024-04-16T20:15:32.495 DEBUG:boto:Data: 2024-04-16T20:15:32.495 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.495 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.495 DEBUG:boto:Params: {} 2024-04-16T20:15:32.495 DEBUG:boto:Token: None 2024-04-16T20:15:32.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj51 2024-04-16T20:15:32.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qf6rAyDKkF4fql2sAjyKtMnscDw= 2024-04-16T20:15:32.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qf6rAyDKkF4fql2sAjyKtMnscDw=', 'Content-Length': '0'} 2024-04-16T20:15:32.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebcb96cd570b146-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.498 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.498 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.498 DEBUG:boto:path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.498 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.498 DEBUG:boto:Method: GET 2024-04-16T20:15:32.499 DEBUG:boto:Path: /yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.499 DEBUG:boto:Data: 2024-04-16T20:15:32.499 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.499 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.499 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.499 DEBUG:boto:Params: {} 2024-04-16T20:15:32.499 DEBUG:boto:Token: None 2024-04-16T20:15:32.499 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.499 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d7264d76cc146b-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.542 DEBUG:boto:b'yrjefh-54obj511000falseobj512024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.543 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.543 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.543 DEBUG:boto:path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.543 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.543 DEBUG:boto:Method: GET 2024-04-16T20:15:32.543 DEBUG:boto:Path: /yrjefh-54/?prefix=obj51 2024-04-16T20:15:32.543 DEBUG:boto:Data: 2024-04-16T20:15:32.543 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.544 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.544 DEBUG:boto:Params: {} 2024-04-16T20:15:32.544 DEBUG:boto:Token: None 2024-04-16T20:15:32.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fab5d1df830ff75-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.547 DEBUG:boto:b'yrjefh-54obj511000falseobj512024-04-16T20:12:50.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:15:32.548 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.548 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.548 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.548 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.548 DEBUG:boto:Method: GET 2024-04-16T20:15:32.548 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.548 DEBUG:boto:Data: 2024-04-16T20:15:32.548 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.548 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.548 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.549 DEBUG:boto:Params: {} 2024-04-16T20:15:32.549 DEBUG:boto:Token: None 2024-04-16T20:15:32.549 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU= 2024-04-16T20:15:32.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU=', 'Content-Length': '0'} 2024-04-16T20:15:32.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63a6a8ad16b5969-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.594 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.595 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.595 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.595 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.595 DEBUG:boto:Method: GET 2024-04-16T20:15:32.595 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.595 DEBUG:boto:Data: 2024-04-16T20:15:32.595 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.595 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.595 DEBUG:boto:Params: {} 2024-04-16T20:15:32.596 DEBUG:boto:Token: None 2024-04-16T20:15:32.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU= 2024-04-16T20:15:32.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU=', 'Content-Length': '0'} 2024-04-16T20:15:32.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ff4ea70f0c845b-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.598 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.598 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.599 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.599 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.599 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.599 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.599 DEBUG:boto:Data: 2024-04-16T20:15:32.599 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.599 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.599 DEBUG:boto:Params: {} 2024-04-16T20:15:32.599 DEBUG:boto:Token: None 2024-04-16T20:15:32.600 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dQL8wtPCVRzETQYNTFd15tMLx8g= 2024-04-16T20:15:32.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dQL8wtPCVRzETQYNTFd15tMLx8g=', 'Content-Length': '0'} 2024-04-16T20:15:32.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a9fe814a6197e5-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.602 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.602 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.602 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.602 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.602 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.602 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.603 DEBUG:boto:Data: 2024-04-16T20:15:32.603 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.603 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.603 DEBUG:boto:Params: {} 2024-04-16T20:15:32.603 DEBUG:boto:Token: None 2024-04-16T20:15:32.603 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dQL8wtPCVRzETQYNTFd15tMLx8g= 2024-04-16T20:15:32.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dQL8wtPCVRzETQYNTFd15tMLx8g=', 'Content-Length': '0'} 2024-04-16T20:15:32.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e787ba6b61bfd3e-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:15:32.605 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.606 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.606 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.606 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.606 DEBUG:boto:Method: GET 2024-04-16T20:15:32.606 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.606 DEBUG:boto:Data: 2024-04-16T20:15:32.606 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.606 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.606 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.606 DEBUG:boto:Params: {} 2024-04-16T20:15:32.606 DEBUG:boto:Token: None 2024-04-16T20:15:32.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU= 2024-04-16T20:15:32.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU=', 'Content-Length': '0'} 2024-04-16T20:15:32.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6ce55fe6db2e00-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.650 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.650 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.651 DEBUG:boto:path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.651 DEBUG:boto:auth_path=/yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.651 DEBUG:boto:Method: GET 2024-04-16T20:15:32.651 DEBUG:boto:Path: /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.651 DEBUG:boto:Data: 2024-04-16T20:15:32.651 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.651 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.651 DEBUG:boto:Params: {} 2024-04-16T20:15:32.652 DEBUG:boto:Token: None 2024-04-16T20:15:32.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55?versionId=null 2024-04-16T20:15:32.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU= 2024-04-16T20:15:32.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5EwvChX6gUYmKV8gIzN9JW2GHvU=', 'Content-Length': '0'} 2024-04-16T20:15:32.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096644345c76aaa0a-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.654 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.654 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.655 DEBUG:boto:path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.655 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.655 DEBUG:boto:Method: GET 2024-04-16T20:15:32.655 DEBUG:boto:Path: /yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.655 DEBUG:boto:Data: 2024-04-16T20:15:32.655 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.655 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.655 DEBUG:boto:Params: {} 2024-04-16T20:15:32.655 DEBUG:boto:Token: None 2024-04-16T20:15:32.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000626153953de91539-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.698 DEBUG:boto:b'yrjefh-54obj551000falseobj552024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.699 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.699 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.699 DEBUG:boto:path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.699 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.699 DEBUG:boto:Method: GET 2024-04-16T20:15:32.699 DEBUG:boto:Path: /yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.699 DEBUG:boto:Data: 2024-04-16T20:15:32.699 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.699 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.699 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.700 DEBUG:boto:Params: {} 2024-04-16T20:15:32.700 DEBUG:boto:Token: None 2024-04-16T20:15:32.700 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.700 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec8c0b340a5069e-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.703 DEBUG:boto:b'yrjefh-54obj551000falseobj552024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.704 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.704 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.704 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.704 DEBUG:boto:Path: /yrjefh-54/obj55 2024-04-16T20:15:32.704 DEBUG:boto:Data: 2024-04-16T20:15:32.704 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.704 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.704 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.704 DEBUG:boto:Params: {} 2024-04-16T20:15:32.704 DEBUG:boto:Token: None 2024-04-16T20:15:32.705 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55 2024-04-16T20:15:32.705 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EVGF5P4vXaXf9YTaOG7JN4k9dZc= 2024-04-16T20:15:32.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EVGF5P4vXaXf9YTaOG7JN4k9dZc=', 'Content-Length': '0'} 2024-04-16T20:15:32.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b32b3284e55e0c1-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.707 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.707 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.707 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.707 DEBUG:boto:Path: /yrjefh-54/obj55 2024-04-16T20:15:32.707 DEBUG:boto:Data: 2024-04-16T20:15:32.707 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.708 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.708 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.708 DEBUG:boto:Params: {} 2024-04-16T20:15:32.708 DEBUG:boto:Token: None 2024-04-16T20:15:32.708 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55 2024-04-16T20:15:32.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EVGF5P4vXaXf9YTaOG7JN4k9dZc= 2024-04-16T20:15:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EVGF5P4vXaXf9YTaOG7JN4k9dZc=', 'Content-Length': '0'} 2024-04-16T20:15:32.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dcf25ba3e69bc9-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:15:32.710 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.711 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.711 DEBUG:boto:Method: GET 2024-04-16T20:15:32.711 DEBUG:boto:Path: /yrjefh-54/obj55 2024-04-16T20:15:32.711 DEBUG:boto:Data: 2024-04-16T20:15:32.711 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.711 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.711 DEBUG:boto:Params: {} 2024-04-16T20:15:32.711 DEBUG:boto:Token: None 2024-04-16T20:15:32.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55 2024-04-16T20:15:32.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hkwnb/Yk8z7biecnXQKAjhhbkBc= 2024-04-16T20:15:32.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hkwnb/Yk8z7biecnXQKAjhhbkBc=', 'Content-Length': '0'} 2024-04-16T20:15:32.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8e3ba2f5a516c9-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.754 DEBUG:boto:path=/yrjefh-54/obj55 2024-04-16T20:15:32.754 DEBUG:boto:auth_path=/yrjefh-54/obj55 2024-04-16T20:15:32.755 DEBUG:boto:Method: GET 2024-04-16T20:15:32.755 DEBUG:boto:Path: /yrjefh-54/obj55 2024-04-16T20:15:32.755 DEBUG:boto:Data: 2024-04-16T20:15:32.755 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.755 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.755 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.755 DEBUG:boto:Params: {} 2024-04-16T20:15:32.755 DEBUG:boto:Token: None 2024-04-16T20:15:32.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj55 2024-04-16T20:15:32.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hkwnb/Yk8z7biecnXQKAjhhbkBc= 2024-04-16T20:15:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hkwnb/Yk8z7biecnXQKAjhhbkBc=', 'Content-Length': '0'} 2024-04-16T20:15:32.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cdc33dae632eda-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.758 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.758 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.758 DEBUG:boto:path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.758 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.758 DEBUG:boto:Method: GET 2024-04-16T20:15:32.758 DEBUG:boto:Path: /yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.758 DEBUG:boto:Data: 2024-04-16T20:15:32.759 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.759 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.759 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.759 DEBUG:boto:Params: {} 2024-04-16T20:15:32.759 DEBUG:boto:Token: None 2024-04-16T20:15:32.759 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2a423fe421aae1-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.802 DEBUG:boto:b'yrjefh-54obj551000falseobj552024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.803 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.803 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.803 DEBUG:boto:path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.803 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.803 DEBUG:boto:Method: GET 2024-04-16T20:15:32.803 DEBUG:boto:Path: /yrjefh-54/?prefix=obj55 2024-04-16T20:15:32.803 DEBUG:boto:Data: 2024-04-16T20:15:32.803 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.803 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.804 DEBUG:boto:Params: {} 2024-04-16T20:15:32.804 DEBUG:boto:Token: None 2024-04-16T20:15:32.804 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df3f57d3f24f19f-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.808 DEBUG:boto:b'yrjefh-54obj551000falseobj552024-04-16T20:12:50.257Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:15:32.808 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.808 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.808 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.808 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.809 DEBUG:boto:Method: GET 2024-04-16T20:15:32.809 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.809 DEBUG:boto:Data: 2024-04-16T20:15:32.809 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.809 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.809 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.809 DEBUG:boto:Params: {} 2024-04-16T20:15:32.809 DEBUG:boto:Token: None 2024-04-16T20:15:32.809 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.809 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ= 2024-04-16T20:15:32.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c43950680f7c0b-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.854 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.855 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.855 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.855 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.855 DEBUG:boto:Method: GET 2024-04-16T20:15:32.855 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.855 DEBUG:boto:Data: 2024-04-16T20:15:32.855 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.855 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.855 DEBUG:boto:Params: {} 2024-04-16T20:15:32.855 DEBUG:boto:Token: None 2024-04-16T20:15:32.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ= 2024-04-16T20:15:32.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5d28048a6bb1be-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.858 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.858 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.858 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.858 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.858 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.859 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.859 DEBUG:boto:Data: 2024-04-16T20:15:32.859 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.859 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.859 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.859 DEBUG:boto:Params: {} 2024-04-16T20:15:32.859 DEBUG:boto:Token: None 2024-04-16T20:15:32.859 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a/yLWULTmkXKGPwhiKYCu431JYk= 2024-04-16T20:15:32.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a/yLWULTmkXKGPwhiKYCu431JYk=', 'Content-Length': '0'} 2024-04-16T20:15:32.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ed8c174d5a832b4-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.862 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.862 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.862 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.862 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.862 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.862 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.862 DEBUG:boto:Data: 2024-04-16T20:15:32.862 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.862 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.863 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.863 DEBUG:boto:Params: {} 2024-04-16T20:15:32.863 DEBUG:boto:Token: None 2024-04-16T20:15:32.863 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:a/yLWULTmkXKGPwhiKYCu431JYk= 2024-04-16T20:15:32.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:a/yLWULTmkXKGPwhiKYCu431JYk=', 'Content-Length': '0'} 2024-04-16T20:15:32.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000179195e44fc99b60-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:15:32.865 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.865 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.865 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.866 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.866 DEBUG:boto:Method: GET 2024-04-16T20:15:32.866 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.866 DEBUG:boto:Data: 2024-04-16T20:15:32.866 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.866 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.866 DEBUG:boto:Params: {} 2024-04-16T20:15:32.866 DEBUG:boto:Token: None 2024-04-16T20:15:32.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ= 2024-04-16T20:15:32.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac1865b03b32d76-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.910 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.911 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.911 DEBUG:boto:path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.911 DEBUG:boto:auth_path=/yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.911 DEBUG:boto:Method: GET 2024-04-16T20:15:32.911 DEBUG:boto:Path: /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.911 DEBUG:boto:Data: 2024-04-16T20:15:32.911 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.911 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.911 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.911 DEBUG:boto:Params: {} 2024-04-16T20:15:32.911 DEBUG:boto:Token: None 2024-04-16T20:15:32.912 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57?versionId=null 2024-04-16T20:15:32.912 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ= 2024-04-16T20:15:32.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6rOVygvZnz9BSYGateZAgYj6pQ=', 'Content-Length': '0'} 2024-04-16T20:15:32.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196fe0be1a58ce4f-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.914 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.915 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.915 DEBUG:boto:path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.915 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.915 DEBUG:boto:Method: GET 2024-04-16T20:15:32.915 DEBUG:boto:Path: /yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.915 DEBUG:boto:Data: 2024-04-16T20:15:32.915 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.915 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.915 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.915 DEBUG:boto:Params: {} 2024-04-16T20:15:32.915 DEBUG:boto:Token: None 2024-04-16T20:15:32.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2166a4272de892d-00661edc64-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.958 DEBUG:boto:b'yrjefh-54obj571000falseobj572024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.959 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:32.959 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:32.959 DEBUG:boto:path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.959 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.959 DEBUG:boto:Method: GET 2024-04-16T20:15:32.959 DEBUG:boto:Path: /yrjefh-54/?prefix=obj57 2024-04-16T20:15:32.960 DEBUG:boto:Data: 2024-04-16T20:15:32.960 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.960 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.960 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.960 DEBUG:boto:Params: {} 2024-04-16T20:15:32.960 DEBUG:boto:Token: None 2024-04-16T20:15:32.960 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/ 2024-04-16T20:15:32.960 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo= 2024-04-16T20:15:32.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:djIG2Pseg31+Wu5BiZcuq0YBhFo=', 'Content-Length': '0'} 2024-04-16T20:15:32.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080265c6a2c6343a1-00661edc64-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.964 DEBUG:boto:b'yrjefh-54obj571000falseobj572024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:32.964 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.964 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.965 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.965 DEBUG:boto:Path: /yrjefh-54/obj57 2024-04-16T20:15:32.965 DEBUG:boto:Data: 2024-04-16T20:15:32.965 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.965 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.965 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.965 DEBUG:boto:Params: {} 2024-04-16T20:15:32.965 DEBUG:boto:Token: None 2024-04-16T20:15:32.965 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57 2024-04-16T20:15:32.966 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zf7wf+lSpozBtFHqAz0pNybOq9k= 2024-04-16T20:15:32.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zf7wf+lSpozBtFHqAz0pNybOq9k=', 'Content-Length': '0'} 2024-04-16T20:15:32.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b003b77cc81b129-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.968 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.968 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.968 DEBUG:boto:Method: HEAD 2024-04-16T20:15:32.968 DEBUG:boto:Path: /yrjefh-54/obj57 2024-04-16T20:15:32.968 DEBUG:boto:Data: 2024-04-16T20:15:32.968 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.969 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:32.969 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.969 DEBUG:boto:Params: {} 2024-04-16T20:15:32.969 DEBUG:boto:Token: None 2024-04-16T20:15:32.969 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57 2024-04-16T20:15:32.969 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zf7wf+lSpozBtFHqAz0pNybOq9k= 2024-04-16T20:15:32.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zf7wf+lSpozBtFHqAz0pNybOq9k=', 'Content-Length': '0'} 2024-04-16T20:15:32.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840213ba38b434b6-00661edc64-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:32.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:15:32.971 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:32.972 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:32.972 DEBUG:boto:Method: GET 2024-04-16T20:15:32.972 DEBUG:boto:Path: /yrjefh-54/obj57 2024-04-16T20:15:32.972 DEBUG:boto:Data: 2024-04-16T20:15:32.972 DEBUG:boto:Headers: {} 2024-04-16T20:15:32.972 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:32.972 DEBUG:boto:Port: 8000 2024-04-16T20:15:32.972 DEBUG:boto:Params: {} 2024-04-16T20:15:32.972 DEBUG:boto:Token: None 2024-04-16T20:15:32.972 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:32 GMT /yrjefh-54/obj57 2024-04-16T20:15:32.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qIKAVd3s7b+HSpbmYWi5BXA9n0c= 2024-04-16T20:15:32.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:32 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qIKAVd3s7b+HSpbmYWi5BXA9n0c=', 'Content-Length': '0'} 2024-04-16T20:15:32.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f86080ffadec1ae8-00661edc64-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.018 DEBUG:boto:path=/yrjefh-54/obj57 2024-04-16T20:15:33.018 DEBUG:boto:auth_path=/yrjefh-54/obj57 2024-04-16T20:15:33.019 DEBUG:boto:Method: GET 2024-04-16T20:15:33.019 DEBUG:boto:Path: /yrjefh-54/obj57 2024-04-16T20:15:33.019 DEBUG:boto:Data: 2024-04-16T20:15:33.019 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.019 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.019 DEBUG:boto:Params: {} 2024-04-16T20:15:33.019 DEBUG:boto:Token: None 2024-04-16T20:15:33.019 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj57 2024-04-16T20:15:33.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pF12DkvephOc8Q4R1EK5J1/xKDM= 2024-04-16T20:15:33.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pF12DkvephOc8Q4R1EK5J1/xKDM=', 'Content-Length': '0'} 2024-04-16T20:15:33.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001535dc6ab15ccf56-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.022 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.022 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.022 DEBUG:boto:path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.022 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.022 DEBUG:boto:Method: GET 2024-04-16T20:15:33.022 DEBUG:boto:Path: /yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.022 DEBUG:boto:Data: 2024-04-16T20:15:33.022 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.022 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.023 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.023 DEBUG:boto:Params: {} 2024-04-16T20:15:33.023 DEBUG:boto:Token: None 2024-04-16T20:15:33.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.023 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017509fd5ff190f6c-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.066 DEBUG:boto:b'yrjefh-54obj571000falseobj572024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.067 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.067 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.067 DEBUG:boto:path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.067 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.067 DEBUG:boto:Method: GET 2024-04-16T20:15:33.067 DEBUG:boto:Path: /yrjefh-54/?prefix=obj57 2024-04-16T20:15:33.067 DEBUG:boto:Data: 2024-04-16T20:15:33.067 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.068 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.068 DEBUG:boto:Params: {} 2024-04-16T20:15:33.068 DEBUG:boto:Token: None 2024-04-16T20:15:33.068 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000164eb19f37a44f23-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.072 DEBUG:boto:b'yrjefh-54obj571000falseobj572024-04-16T20:12:50.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:15:33.072 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.073 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.073 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.073 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.073 DEBUG:boto:Method: GET 2024-04-16T20:15:33.073 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.073 DEBUG:boto:Data: 2024-04-16T20:15:33.073 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.073 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.073 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.073 DEBUG:boto:Params: {} 2024-04-16T20:15:33.073 DEBUG:boto:Token: None 2024-04-16T20:15:33.074 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.074 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s= 2024-04-16T20:15:33.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s=', 'Content-Length': '0'} 2024-04-16T20:15:33.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d71840a197c542-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.118 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.119 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.119 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.119 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.119 DEBUG:boto:Method: GET 2024-04-16T20:15:33.119 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.119 DEBUG:boto:Data: 2024-04-16T20:15:33.119 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.119 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.119 DEBUG:boto:Params: {} 2024-04-16T20:15:33.120 DEBUG:boto:Token: None 2024-04-16T20:15:33.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s= 2024-04-16T20:15:33.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s=', 'Content-Length': '0'} 2024-04-16T20:15:33.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002951147c6eec4f2a-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.123 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.123 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.123 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.123 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.123 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.123 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.123 DEBUG:boto:Data: 2024-04-16T20:15:33.123 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.123 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.124 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.124 DEBUG:boto:Params: {} 2024-04-16T20:15:33.124 DEBUG:boto:Token: None 2024-04-16T20:15:33.124 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+15Akvr9pkt7ZzEKmb1kl5YC6tI= 2024-04-16T20:15:33.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+15Akvr9pkt7ZzEKmb1kl5YC6tI=', 'Content-Length': '0'} 2024-04-16T20:15:33.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b4873f5f64989b-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.127 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.127 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.127 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.127 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.127 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.127 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.127 DEBUG:boto:Data: 2024-04-16T20:15:33.127 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.128 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.128 DEBUG:boto:Params: {} 2024-04-16T20:15:33.128 DEBUG:boto:Token: None 2024-04-16T20:15:33.128 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.128 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+15Akvr9pkt7ZzEKmb1kl5YC6tI= 2024-04-16T20:15:33.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+15Akvr9pkt7ZzEKmb1kl5YC6tI=', 'Content-Length': '0'} 2024-04-16T20:15:33.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ed46bb0fcae5b9-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:15:33.131 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.131 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.131 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.131 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.131 DEBUG:boto:Method: GET 2024-04-16T20:15:33.131 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.131 DEBUG:boto:Data: 2024-04-16T20:15:33.132 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.132 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.132 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.132 DEBUG:boto:Params: {} 2024-04-16T20:15:33.132 DEBUG:boto:Token: None 2024-04-16T20:15:33.132 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s= 2024-04-16T20:15:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s=', 'Content-Length': '0'} 2024-04-16T20:15:33.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290aab7752d422ca-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.174 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.174 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.175 DEBUG:boto:path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.175 DEBUG:boto:auth_path=/yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.175 DEBUG:boto:Method: GET 2024-04-16T20:15:33.175 DEBUG:boto:Path: /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.175 DEBUG:boto:Data: 2024-04-16T20:15:33.175 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.175 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.175 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.175 DEBUG:boto:Params: {} 2024-04-16T20:15:33.175 DEBUG:boto:Token: None 2024-04-16T20:15:33.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58?versionId=null 2024-04-16T20:15:33.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s= 2024-04-16T20:15:33.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h8HZH60ujciy/eYviQWj3CQGr5s=', 'Content-Length': '0'} 2024-04-16T20:15:33.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000129a06f23216fed-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.178 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.179 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.179 DEBUG:boto:path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.179 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.179 DEBUG:boto:Method: GET 2024-04-16T20:15:33.179 DEBUG:boto:Path: /yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.179 DEBUG:boto:Data: 2024-04-16T20:15:33.179 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.179 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.179 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.179 DEBUG:boto:Params: {} 2024-04-16T20:15:33.180 DEBUG:boto:Token: None 2024-04-16T20:15:33.180 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ab7266011b950f-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.222 DEBUG:boto:b'yrjefh-54obj581000falseobj582024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.223 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.223 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.223 DEBUG:boto:path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.223 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.223 DEBUG:boto:Method: GET 2024-04-16T20:15:33.223 DEBUG:boto:Path: /yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.223 DEBUG:boto:Data: 2024-04-16T20:15:33.223 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.224 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.224 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.224 DEBUG:boto:Params: {} 2024-04-16T20:15:33.224 DEBUG:boto:Token: None 2024-04-16T20:15:33.224 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e009dd85d08c14-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.228 DEBUG:boto:b'yrjefh-54obj581000falseobj582024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.228 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.228 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.228 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.228 DEBUG:boto:Path: /yrjefh-54/obj58 2024-04-16T20:15:33.229 DEBUG:boto:Data: 2024-04-16T20:15:33.229 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.229 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.229 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.229 DEBUG:boto:Params: {} 2024-04-16T20:15:33.229 DEBUG:boto:Token: None 2024-04-16T20:15:33.229 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58 2024-04-16T20:15:33.229 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6SbcqrhIZPhYFNiU1/bTLonzR7s= 2024-04-16T20:15:33.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6SbcqrhIZPhYFNiU1/bTLonzR7s=', 'Content-Length': '0'} 2024-04-16T20:15:33.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003117e56e6b509318-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.232 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.232 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.232 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.232 DEBUG:boto:Path: /yrjefh-54/obj58 2024-04-16T20:15:33.232 DEBUG:boto:Data: 2024-04-16T20:15:33.232 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.232 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.232 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.232 DEBUG:boto:Params: {} 2024-04-16T20:15:33.233 DEBUG:boto:Token: None 2024-04-16T20:15:33.233 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58 2024-04-16T20:15:33.233 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6SbcqrhIZPhYFNiU1/bTLonzR7s= 2024-04-16T20:15:33.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6SbcqrhIZPhYFNiU1/bTLonzR7s=', 'Content-Length': '0'} 2024-04-16T20:15:33.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9be95b477b61383-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:15:33.235 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.235 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.236 DEBUG:boto:Method: GET 2024-04-16T20:15:33.236 DEBUG:boto:Path: /yrjefh-54/obj58 2024-04-16T20:15:33.236 DEBUG:boto:Data: 2024-04-16T20:15:33.236 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.236 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.236 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.236 DEBUG:boto:Params: {} 2024-04-16T20:15:33.236 DEBUG:boto:Token: None 2024-04-16T20:15:33.236 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58 2024-04-16T20:15:33.237 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJOHv0DyfECmYGU0IymuQnMnGEQ= 2024-04-16T20:15:33.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJOHv0DyfECmYGU0IymuQnMnGEQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d584b2119f7dd90f-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.282 DEBUG:boto:path=/yrjefh-54/obj58 2024-04-16T20:15:33.283 DEBUG:boto:auth_path=/yrjefh-54/obj58 2024-04-16T20:15:33.283 DEBUG:boto:Method: GET 2024-04-16T20:15:33.283 DEBUG:boto:Path: /yrjefh-54/obj58 2024-04-16T20:15:33.283 DEBUG:boto:Data: 2024-04-16T20:15:33.283 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.283 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.283 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.283 DEBUG:boto:Params: {} 2024-04-16T20:15:33.283 DEBUG:boto:Token: None 2024-04-16T20:15:33.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj58 2024-04-16T20:15:33.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PJOHv0DyfECmYGU0IymuQnMnGEQ= 2024-04-16T20:15:33.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PJOHv0DyfECmYGU0IymuQnMnGEQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870914d6ae4c1f8e-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.286 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.286 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.286 DEBUG:boto:path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.287 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.287 DEBUG:boto:Method: GET 2024-04-16T20:15:33.287 DEBUG:boto:Path: /yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.287 DEBUG:boto:Data: 2024-04-16T20:15:33.287 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.287 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.287 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.287 DEBUG:boto:Params: {} 2024-04-16T20:15:33.287 DEBUG:boto:Token: None 2024-04-16T20:15:33.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.288 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b60300e228d2b4-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.330 DEBUG:boto:b'yrjefh-54obj581000falseobj582024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.331 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.331 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.331 DEBUG:boto:path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.331 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.331 DEBUG:boto:Method: GET 2024-04-16T20:15:33.331 DEBUG:boto:Path: /yrjefh-54/?prefix=obj58 2024-04-16T20:15:33.331 DEBUG:boto:Data: 2024-04-16T20:15:33.331 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.332 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.332 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.332 DEBUG:boto:Params: {} 2024-04-16T20:15:33.332 DEBUG:boto:Token: None 2024-04-16T20:15:33.332 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.332 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081078abc282edfe-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.336 DEBUG:boto:b'yrjefh-54obj581000falseobj582024-04-16T20:12:51.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:15:33.337 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.337 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.337 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.337 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.337 DEBUG:boto:Method: GET 2024-04-16T20:15:33.337 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.337 DEBUG:boto:Data: 2024-04-16T20:15:33.337 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.337 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.338 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.338 DEBUG:boto:Params: {} 2024-04-16T20:15:33.338 DEBUG:boto:Token: None 2024-04-16T20:15:33.338 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.338 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk= 2024-04-16T20:15:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk=', 'Content-Length': '0'} 2024-04-16T20:15:33.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d617cd981121537-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.382 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.383 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.383 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.383 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.383 DEBUG:boto:Method: GET 2024-04-16T20:15:33.383 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.383 DEBUG:boto:Data: 2024-04-16T20:15:33.383 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.383 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.383 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.383 DEBUG:boto:Params: {} 2024-04-16T20:15:33.384 DEBUG:boto:Token: None 2024-04-16T20:15:33.384 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.384 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk= 2024-04-16T20:15:33.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk=', 'Content-Length': '0'} 2024-04-16T20:15:33.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001971983d7bbdd606-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.386 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.386 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.387 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.387 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.387 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.387 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.387 DEBUG:boto:Data: 2024-04-16T20:15:33.387 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.387 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.387 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.387 DEBUG:boto:Params: {} 2024-04-16T20:15:33.387 DEBUG:boto:Token: None 2024-04-16T20:15:33.388 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.388 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qtTqTOZbQz/LcOcyffPVxxnvJYo= 2024-04-16T20:15:33.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qtTqTOZbQz/LcOcyffPVxxnvJYo=', 'Content-Length': '0'} 2024-04-16T20:15:33.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f669229b871d6256-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.390 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.390 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.390 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.391 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.391 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.391 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.391 DEBUG:boto:Data: 2024-04-16T20:15:33.391 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.391 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.391 DEBUG:boto:Params: {} 2024-04-16T20:15:33.391 DEBUG:boto:Token: None 2024-04-16T20:15:33.392 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qtTqTOZbQz/LcOcyffPVxxnvJYo= 2024-04-16T20:15:33.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qtTqTOZbQz/LcOcyffPVxxnvJYo=', 'Content-Length': '0'} 2024-04-16T20:15:33.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e339c6543f396e1-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:15:33.394 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.394 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.394 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.394 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.395 DEBUG:boto:Method: GET 2024-04-16T20:15:33.395 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.395 DEBUG:boto:Data: 2024-04-16T20:15:33.395 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.395 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.395 DEBUG:boto:Params: {} 2024-04-16T20:15:33.395 DEBUG:boto:Token: None 2024-04-16T20:15:33.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk= 2024-04-16T20:15:33.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk=', 'Content-Length': '0'} 2024-04-16T20:15:33.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53e24e190d9caf6-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.438 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.438 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.439 DEBUG:boto:path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.439 DEBUG:boto:auth_path=/yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.439 DEBUG:boto:Method: GET 2024-04-16T20:15:33.439 DEBUG:boto:Path: /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.439 DEBUG:boto:Data: 2024-04-16T20:15:33.439 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.439 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.439 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.439 DEBUG:boto:Params: {} 2024-04-16T20:15:33.439 DEBUG:boto:Token: None 2024-04-16T20:15:33.440 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62?versionId=null 2024-04-16T20:15:33.440 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk= 2024-04-16T20:15:33.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zff0yKrCXGB+37HxLs7QQbP/Txk=', 'Content-Length': '0'} 2024-04-16T20:15:33.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a82c2a93383139-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.442 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.442 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.442 DEBUG:boto:path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.442 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.442 DEBUG:boto:Method: GET 2024-04-16T20:15:33.442 DEBUG:boto:Path: /yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.443 DEBUG:boto:Data: 2024-04-16T20:15:33.443 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.443 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.443 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.443 DEBUG:boto:Params: {} 2024-04-16T20:15:33.443 DEBUG:boto:Token: None 2024-04-16T20:15:33.443 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.443 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcdf590ae32433b2-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.486 DEBUG:boto:b'yrjefh-54obj621000falseobj622024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.487 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.487 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.487 DEBUG:boto:path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.487 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.487 DEBUG:boto:Method: GET 2024-04-16T20:15:33.487 DEBUG:boto:Path: /yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.487 DEBUG:boto:Data: 2024-04-16T20:15:33.487 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.488 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.488 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.488 DEBUG:boto:Params: {} 2024-04-16T20:15:33.488 DEBUG:boto:Token: None 2024-04-16T20:15:33.488 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e3fccabe45bb71-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.492 DEBUG:boto:b'yrjefh-54obj621000falseobj622024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.492 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.492 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.492 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.492 DEBUG:boto:Path: /yrjefh-54/obj62 2024-04-16T20:15:33.492 DEBUG:boto:Data: 2024-04-16T20:15:33.493 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.493 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.493 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.493 DEBUG:boto:Params: {} 2024-04-16T20:15:33.493 DEBUG:boto:Token: None 2024-04-16T20:15:33.493 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62 2024-04-16T20:15:33.493 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7CQi2CQGNLlWAN1N9pga6G8bIk= 2024-04-16T20:15:33.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7CQi2CQGNLlWAN1N9pga6G8bIk=', 'Content-Length': '0'} 2024-04-16T20:15:33.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e08a659f3a05756a-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.496 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.496 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.496 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.496 DEBUG:boto:Path: /yrjefh-54/obj62 2024-04-16T20:15:33.496 DEBUG:boto:Data: 2024-04-16T20:15:33.496 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.496 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.496 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.497 DEBUG:boto:Params: {} 2024-04-16T20:15:33.497 DEBUG:boto:Token: None 2024-04-16T20:15:33.497 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62 2024-04-16T20:15:33.497 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T7CQi2CQGNLlWAN1N9pga6G8bIk= 2024-04-16T20:15:33.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T7CQi2CQGNLlWAN1N9pga6G8bIk=', 'Content-Length': '0'} 2024-04-16T20:15:33.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b8a14aedeac637-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:15:33.500 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.500 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.500 DEBUG:boto:Method: GET 2024-04-16T20:15:33.500 DEBUG:boto:Path: /yrjefh-54/obj62 2024-04-16T20:15:33.500 DEBUG:boto:Data: 2024-04-16T20:15:33.500 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.500 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.500 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.500 DEBUG:boto:Params: {} 2024-04-16T20:15:33.500 DEBUG:boto:Token: None 2024-04-16T20:15:33.501 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62 2024-04-16T20:15:33.501 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BioUJDKWzWGmT3aDrFxypHIptM8= 2024-04-16T20:15:33.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BioUJDKWzWGmT3aDrFxypHIptM8=', 'Content-Length': '0'} 2024-04-16T20:15:33.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18a72eb8ad66d76-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.546 DEBUG:boto:path=/yrjefh-54/obj62 2024-04-16T20:15:33.546 DEBUG:boto:auth_path=/yrjefh-54/obj62 2024-04-16T20:15:33.547 DEBUG:boto:Method: GET 2024-04-16T20:15:33.547 DEBUG:boto:Path: /yrjefh-54/obj62 2024-04-16T20:15:33.547 DEBUG:boto:Data: 2024-04-16T20:15:33.547 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.547 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.547 DEBUG:boto:Params: {} 2024-04-16T20:15:33.547 DEBUG:boto:Token: None 2024-04-16T20:15:33.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj62 2024-04-16T20:15:33.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BioUJDKWzWGmT3aDrFxypHIptM8= 2024-04-16T20:15:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BioUJDKWzWGmT3aDrFxypHIptM8=', 'Content-Length': '0'} 2024-04-16T20:15:33.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8ab02a8e3e6256-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.550 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.550 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.550 DEBUG:boto:path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.550 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.550 DEBUG:boto:Method: GET 2024-04-16T20:15:33.550 DEBUG:boto:Path: /yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.550 DEBUG:boto:Data: 2024-04-16T20:15:33.551 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.551 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.551 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.551 DEBUG:boto:Params: {} 2024-04-16T20:15:33.551 DEBUG:boto:Token: None 2024-04-16T20:15:33.551 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259dff59b9574640-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.594 DEBUG:boto:b'yrjefh-54obj621000falseobj622024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.595 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.595 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.595 DEBUG:boto:path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.595 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.595 DEBUG:boto:Method: GET 2024-04-16T20:15:33.595 DEBUG:boto:Path: /yrjefh-54/?prefix=obj62 2024-04-16T20:15:33.595 DEBUG:boto:Data: 2024-04-16T20:15:33.595 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.595 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.596 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.596 DEBUG:boto:Params: {} 2024-04-16T20:15:33.596 DEBUG:boto:Token: None 2024-04-16T20:15:33.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180d764dd76e76de-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.600 DEBUG:boto:b'yrjefh-54obj621000falseobj622024-04-16T20:12:51.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:15:33.600 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.601 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.601 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.601 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.601 DEBUG:boto:Method: GET 2024-04-16T20:15:33.601 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.601 DEBUG:boto:Data: 2024-04-16T20:15:33.601 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.601 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.601 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.601 DEBUG:boto:Params: {} 2024-04-16T20:15:33.602 DEBUG:boto:Token: None 2024-04-16T20:15:33.602 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.602 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38= 2024-04-16T20:15:33.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38=', 'Content-Length': '0'} 2024-04-16T20:15:33.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050cee1a0b943b360-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.646 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.647 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.647 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.647 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.647 DEBUG:boto:Method: GET 2024-04-16T20:15:33.647 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.647 DEBUG:boto:Data: 2024-04-16T20:15:33.647 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.647 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.647 DEBUG:boto:Params: {} 2024-04-16T20:15:33.648 DEBUG:boto:Token: None 2024-04-16T20:15:33.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38= 2024-04-16T20:15:33.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38=', 'Content-Length': '0'} 2024-04-16T20:15:33.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aefea03ee1c0b36-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.651 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.651 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.651 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.651 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.651 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.651 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.652 DEBUG:boto:Data: 2024-04-16T20:15:33.652 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.652 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.652 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.652 DEBUG:boto:Params: {} 2024-04-16T20:15:33.652 DEBUG:boto:Token: None 2024-04-16T20:15:33.652 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMyWmMkjakAMUCkW4a4MNdtMkXM= 2024-04-16T20:15:33.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMyWmMkjakAMUCkW4a4MNdtMkXM=', 'Content-Length': '0'} 2024-04-16T20:15:33.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005649e2294017fb9a-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.655 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.655 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.655 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.655 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.655 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.655 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.655 DEBUG:boto:Data: 2024-04-16T20:15:33.655 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.655 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.655 DEBUG:boto:Params: {} 2024-04-16T20:15:33.656 DEBUG:boto:Token: None 2024-04-16T20:15:33.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMyWmMkjakAMUCkW4a4MNdtMkXM= 2024-04-16T20:15:33.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMyWmMkjakAMUCkW4a4MNdtMkXM=', 'Content-Length': '0'} 2024-04-16T20:15:33.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5521c856e5aabb-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:15:33.659 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.659 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.659 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.659 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.659 DEBUG:boto:Method: GET 2024-04-16T20:15:33.659 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.659 DEBUG:boto:Data: 2024-04-16T20:15:33.659 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.660 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.660 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.660 DEBUG:boto:Params: {} 2024-04-16T20:15:33.660 DEBUG:boto:Token: None 2024-04-16T20:15:33.660 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.660 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38= 2024-04-16T20:15:33.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38=', 'Content-Length': '0'} 2024-04-16T20:15:33.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace02e03b05bd606-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.706 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.707 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.707 DEBUG:boto:path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.707 DEBUG:boto:auth_path=/yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.707 DEBUG:boto:Method: GET 2024-04-16T20:15:33.707 DEBUG:boto:Path: /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.707 DEBUG:boto:Data: 2024-04-16T20:15:33.707 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.707 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.707 DEBUG:boto:Params: {} 2024-04-16T20:15:33.707 DEBUG:boto:Token: None 2024-04-16T20:15:33.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68?versionId=null 2024-04-16T20:15:33.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38= 2024-04-16T20:15:33.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BEejgxdo6LYqI4CqwEafKIqZU38=', 'Content-Length': '0'} 2024-04-16T20:15:33.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fb9bd6877c32d6-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.711 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.711 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.711 DEBUG:boto:path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.711 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.711 DEBUG:boto:Method: GET 2024-04-16T20:15:33.711 DEBUG:boto:Path: /yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.711 DEBUG:boto:Data: 2024-04-16T20:15:33.711 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.712 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.712 DEBUG:boto:Params: {} 2024-04-16T20:15:33.712 DEBUG:boto:Token: None 2024-04-16T20:15:33.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005edf7cd7c9811a2-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.715 DEBUG:boto:b'yrjefh-54obj681000falseobj682024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.715 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.715 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.715 DEBUG:boto:path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.715 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.715 DEBUG:boto:Method: GET 2024-04-16T20:15:33.716 DEBUG:boto:Path: /yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.716 DEBUG:boto:Data: 2024-04-16T20:15:33.716 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.716 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.716 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.716 DEBUG:boto:Params: {} 2024-04-16T20:15:33.716 DEBUG:boto:Token: None 2024-04-16T20:15:33.716 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.716 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628d563a4ffc4818-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.762 DEBUG:boto:b'yrjefh-54obj681000falseobj682024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.763 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.763 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.763 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.763 DEBUG:boto:Path: /yrjefh-54/obj68 2024-04-16T20:15:33.763 DEBUG:boto:Data: 2024-04-16T20:15:33.763 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.763 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.763 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.763 DEBUG:boto:Params: {} 2024-04-16T20:15:33.764 DEBUG:boto:Token: None 2024-04-16T20:15:33.764 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68 2024-04-16T20:15:33.764 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CjTfMBOHWxwIOPExtRjbe9yoBTM= 2024-04-16T20:15:33.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CjTfMBOHWxwIOPExtRjbe9yoBTM=', 'Content-Length': '0'} 2024-04-16T20:15:33.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ad81f86baa164b-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.766 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.767 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.767 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.767 DEBUG:boto:Path: /yrjefh-54/obj68 2024-04-16T20:15:33.767 DEBUG:boto:Data: 2024-04-16T20:15:33.767 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.767 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.767 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.767 DEBUG:boto:Params: {} 2024-04-16T20:15:33.767 DEBUG:boto:Token: None 2024-04-16T20:15:33.767 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68 2024-04-16T20:15:33.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CjTfMBOHWxwIOPExtRjbe9yoBTM= 2024-04-16T20:15:33.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CjTfMBOHWxwIOPExtRjbe9yoBTM=', 'Content-Length': '0'} 2024-04-16T20:15:33.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a2a35bc69e641b-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:15:33.770 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.771 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.771 DEBUG:boto:Method: GET 2024-04-16T20:15:33.771 DEBUG:boto:Path: /yrjefh-54/obj68 2024-04-16T20:15:33.771 DEBUG:boto:Data: 2024-04-16T20:15:33.771 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.771 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.771 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.771 DEBUG:boto:Params: {} 2024-04-16T20:15:33.771 DEBUG:boto:Token: None 2024-04-16T20:15:33.771 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68 2024-04-16T20:15:33.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yRXDaVSIlayLbAM5E5QcOAbfkwg= 2024-04-16T20:15:33.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yRXDaVSIlayLbAM5E5QcOAbfkwg=', 'Content-Length': '0'} 2024-04-16T20:15:33.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd506f7e02511782-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.814 DEBUG:boto:path=/yrjefh-54/obj68 2024-04-16T20:15:33.814 DEBUG:boto:auth_path=/yrjefh-54/obj68 2024-04-16T20:15:33.815 DEBUG:boto:Method: GET 2024-04-16T20:15:33.815 DEBUG:boto:Path: /yrjefh-54/obj68 2024-04-16T20:15:33.815 DEBUG:boto:Data: 2024-04-16T20:15:33.815 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.815 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.815 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.815 DEBUG:boto:Params: {} 2024-04-16T20:15:33.815 DEBUG:boto:Token: None 2024-04-16T20:15:33.815 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj68 2024-04-16T20:15:33.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yRXDaVSIlayLbAM5E5QcOAbfkwg= 2024-04-16T20:15:33.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yRXDaVSIlayLbAM5E5QcOAbfkwg=', 'Content-Length': '0'} 2024-04-16T20:15:33.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bc82bdf5676907-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.818 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.819 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.819 DEBUG:boto:path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.819 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.819 DEBUG:boto:Method: GET 2024-04-16T20:15:33.819 DEBUG:boto:Path: /yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.819 DEBUG:boto:Data: 2024-04-16T20:15:33.819 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.819 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.819 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.819 DEBUG:boto:Params: {} 2024-04-16T20:15:33.820 DEBUG:boto:Token: None 2024-04-16T20:15:33.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c76fcd77dd3770-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.823 DEBUG:boto:b'yrjefh-54obj681000falseobj682024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.823 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.823 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.823 DEBUG:boto:path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.823 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.823 DEBUG:boto:Method: GET 2024-04-16T20:15:33.823 DEBUG:boto:Path: /yrjefh-54/?prefix=obj68 2024-04-16T20:15:33.824 DEBUG:boto:Data: 2024-04-16T20:15:33.824 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.824 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.824 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.824 DEBUG:boto:Params: {} 2024-04-16T20:15:33.824 DEBUG:boto:Token: None 2024-04-16T20:15:33.824 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.824 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341b52788b1a3913-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.828 DEBUG:boto:b'yrjefh-54obj681000falseobj682024-04-16T20:12:50.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:15:33.829 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.829 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.829 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.829 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.829 DEBUG:boto:Method: GET 2024-04-16T20:15:33.829 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.829 DEBUG:boto:Data: 2024-04-16T20:15:33.829 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.829 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.829 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.830 DEBUG:boto:Params: {} 2024-04-16T20:15:33.830 DEBUG:boto:Token: None 2024-04-16T20:15:33.830 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.830 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU= 2024-04-16T20:15:33.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU=', 'Content-Length': '0'} 2024-04-16T20:15:33.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f937dc769ffa712b-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.874 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.875 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.875 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.875 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.875 DEBUG:boto:Method: GET 2024-04-16T20:15:33.875 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.875 DEBUG:boto:Data: 2024-04-16T20:15:33.875 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.875 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.876 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.876 DEBUG:boto:Params: {} 2024-04-16T20:15:33.876 DEBUG:boto:Token: None 2024-04-16T20:15:33.876 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU= 2024-04-16T20:15:33.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU=', 'Content-Length': '0'} 2024-04-16T20:15:33.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d6cfab045e3d3c-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.878 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.879 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.879 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.879 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.879 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.879 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.879 DEBUG:boto:Data: 2024-04-16T20:15:33.879 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.879 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.879 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.879 DEBUG:boto:Params: {} 2024-04-16T20:15:33.880 DEBUG:boto:Token: None 2024-04-16T20:15:33.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bzC+UKVoLERcEva6byMyrwPcuQI= 2024-04-16T20:15:33.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bzC+UKVoLERcEva6byMyrwPcuQI=', 'Content-Length': '0'} 2024-04-16T20:15:33.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f20ea1cf7513705-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.882 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.882 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.883 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.883 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.883 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.883 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.883 DEBUG:boto:Data: 2024-04-16T20:15:33.883 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.883 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.883 DEBUG:boto:Params: {} 2024-04-16T20:15:33.883 DEBUG:boto:Token: None 2024-04-16T20:15:33.884 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bzC+UKVoLERcEva6byMyrwPcuQI= 2024-04-16T20:15:33.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bzC+UKVoLERcEva6byMyrwPcuQI=', 'Content-Length': '0'} 2024-04-16T20:15:33.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f143d8f9786d7d-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:15:33.886 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.886 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.886 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.887 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.887 DEBUG:boto:Method: GET 2024-04-16T20:15:33.887 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.887 DEBUG:boto:Data: 2024-04-16T20:15:33.887 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.887 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.887 DEBUG:boto:Params: {} 2024-04-16T20:15:33.887 DEBUG:boto:Token: None 2024-04-16T20:15:33.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU= 2024-04-16T20:15:33.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU=', 'Content-Length': '0'} 2024-04-16T20:15:33.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f817b205d70a174-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.930 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.931 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.931 DEBUG:boto:path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.931 DEBUG:boto:auth_path=/yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.931 DEBUG:boto:Method: GET 2024-04-16T20:15:33.931 DEBUG:boto:Path: /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.931 DEBUG:boto:Data: 2024-04-16T20:15:33.931 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.931 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.931 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.932 DEBUG:boto:Params: {} 2024-04-16T20:15:33.932 DEBUG:boto:Token: None 2024-04-16T20:15:33.932 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69?versionId=null 2024-04-16T20:15:33.932 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU= 2024-04-16T20:15:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zP24x3y+b4PXcqV9blIeNfaYoLU=', 'Content-Length': '0'} 2024-04-16T20:15:33.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d668b63c8f890f67-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.935 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.935 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.935 DEBUG:boto:path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.935 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.935 DEBUG:boto:Method: GET 2024-04-16T20:15:33.935 DEBUG:boto:Path: /yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.935 DEBUG:boto:Data: 2024-04-16T20:15:33.935 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.935 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.936 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.936 DEBUG:boto:Params: {} 2024-04-16T20:15:33.936 DEBUG:boto:Token: None 2024-04-16T20:15:33.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3e1126c2bb6c0f-00661edc65-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.939 DEBUG:boto:b'yrjefh-54obj691000falseobj692024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.939 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:33.939 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:33.940 DEBUG:boto:path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.940 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.940 DEBUG:boto:Method: GET 2024-04-16T20:15:33.940 DEBUG:boto:Path: /yrjefh-54/?prefix=obj69 2024-04-16T20:15:33.940 DEBUG:boto:Data: 2024-04-16T20:15:33.940 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.940 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.940 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.940 DEBUG:boto:Params: {} 2024-04-16T20:15:33.940 DEBUG:boto:Token: None 2024-04-16T20:15:33.941 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/ 2024-04-16T20:15:33.941 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ= 2024-04-16T20:15:33.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S8RcZZHEcVdK+SOOAzIGZRnkRQQ=', 'Content-Length': '0'} 2024-04-16T20:15:33.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c106fa065e0ba83-00661edc65-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.945 DEBUG:boto:b'yrjefh-54obj691000falseobj692024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:33.945 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.945 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.945 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.945 DEBUG:boto:Path: /yrjefh-54/obj69 2024-04-16T20:15:33.946 DEBUG:boto:Data: 2024-04-16T20:15:33.946 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.946 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.946 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.946 DEBUG:boto:Params: {} 2024-04-16T20:15:33.946 DEBUG:boto:Token: None 2024-04-16T20:15:33.946 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69 2024-04-16T20:15:33.946 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hzheKKACdGa+FtscrA+HWV3YKHk= 2024-04-16T20:15:33.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hzheKKACdGa+FtscrA+HWV3YKHk=', 'Content-Length': '0'} 2024-04-16T20:15:33.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e219909c71a285-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.948 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.949 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.949 DEBUG:boto:Method: HEAD 2024-04-16T20:15:33.949 DEBUG:boto:Path: /yrjefh-54/obj69 2024-04-16T20:15:33.949 DEBUG:boto:Data: 2024-04-16T20:15:33.949 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.949 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.949 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.949 DEBUG:boto:Params: {} 2024-04-16T20:15:33.950 DEBUG:boto:Token: None 2024-04-16T20:15:33.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69 2024-04-16T20:15:33.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hzheKKACdGa+FtscrA+HWV3YKHk= 2024-04-16T20:15:33.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hzheKKACdGa+FtscrA+HWV3YKHk=', 'Content-Length': '0'} 2024-04-16T20:15:33.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a38b9027b47239-00661edc65-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:15:33.952 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.952 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.953 DEBUG:boto:Method: GET 2024-04-16T20:15:33.953 DEBUG:boto:Path: /yrjefh-54/obj69 2024-04-16T20:15:33.953 DEBUG:boto:Data: 2024-04-16T20:15:33.953 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.953 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:33.953 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.953 DEBUG:boto:Params: {} 2024-04-16T20:15:33.953 DEBUG:boto:Token: None 2024-04-16T20:15:33.953 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69 2024-04-16T20:15:33.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:20q1azSSuLjLKKR9iAt9Vj7WsXg= 2024-04-16T20:15:33.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:20q1azSSuLjLKKR9iAt9Vj7WsXg=', 'Content-Length': '0'} 2024-04-16T20:15:33.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c332de3b749099c-00661edc65-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:33.998 DEBUG:boto:path=/yrjefh-54/obj69 2024-04-16T20:15:33.998 DEBUG:boto:auth_path=/yrjefh-54/obj69 2024-04-16T20:15:33.999 DEBUG:boto:Method: GET 2024-04-16T20:15:33.999 DEBUG:boto:Path: /yrjefh-54/obj69 2024-04-16T20:15:33.999 DEBUG:boto:Data: 2024-04-16T20:15:33.999 DEBUG:boto:Headers: {} 2024-04-16T20:15:33.999 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:33.999 DEBUG:boto:Port: 8000 2024-04-16T20:15:33.999 DEBUG:boto:Params: {} 2024-04-16T20:15:33.999 DEBUG:boto:Token: None 2024-04-16T20:15:34.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:33 GMT /yrjefh-54/obj69 2024-04-16T20:15:34.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:20q1azSSuLjLKKR9iAt9Vj7WsXg= 2024-04-16T20:15:34.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:20q1azSSuLjLKKR9iAt9Vj7WsXg=', 'Content-Length': '0'} 2024-04-16T20:15:34.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df9c1e4786b608a-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.002 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.002 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.003 DEBUG:boto:path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.003 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.003 DEBUG:boto:Method: GET 2024-04-16T20:15:34.003 DEBUG:boto:Path: /yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.003 DEBUG:boto:Data: 2024-04-16T20:15:34.003 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.003 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.003 DEBUG:boto:Params: {} 2024-04-16T20:15:34.004 DEBUG:boto:Token: None 2024-04-16T20:15:34.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292c616e9cb062f8-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.007 DEBUG:boto:b'yrjefh-54obj691000falseobj692024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.007 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.007 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.007 DEBUG:boto:path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.007 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.007 DEBUG:boto:Method: GET 2024-04-16T20:15:34.008 DEBUG:boto:Path: /yrjefh-54/?prefix=obj69 2024-04-16T20:15:34.008 DEBUG:boto:Data: 2024-04-16T20:15:34.008 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.008 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.008 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.008 DEBUG:boto:Params: {} 2024-04-16T20:15:34.008 DEBUG:boto:Token: None 2024-04-16T20:15:34.008 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.009 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000295ebd5ff04b8b7b-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.012 DEBUG:boto:b'yrjefh-54obj691000falseobj692024-04-16T20:12:51.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:15:34.012 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.013 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.013 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.013 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.013 DEBUG:boto:Method: GET 2024-04-16T20:15:34.013 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.013 DEBUG:boto:Data: 2024-04-16T20:15:34.013 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.013 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.013 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.014 DEBUG:boto:Params: {} 2024-04-16T20:15:34.014 DEBUG:boto:Token: None 2024-04-16T20:15:34.014 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.014 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E= 2024-04-16T20:15:34.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E=', 'Content-Length': '0'} 2024-04-16T20:15:34.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003727c63a4beff848-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.058 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.059 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.059 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.059 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.059 DEBUG:boto:Method: GET 2024-04-16T20:15:34.059 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.059 DEBUG:boto:Data: 2024-04-16T20:15:34.059 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.059 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.060 DEBUG:boto:Params: {} 2024-04-16T20:15:34.060 DEBUG:boto:Token: None 2024-04-16T20:15:34.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E= 2024-04-16T20:15:34.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E=', 'Content-Length': '0'} 2024-04-16T20:15:34.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b52a724516c11df6-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.063 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.063 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.063 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.063 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.063 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.063 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.063 DEBUG:boto:Data: 2024-04-16T20:15:34.063 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.064 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.064 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.064 DEBUG:boto:Params: {} 2024-04-16T20:15:34.064 DEBUG:boto:Token: None 2024-04-16T20:15:34.064 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5j9EPlF79jpE39TIe6z0iu3hW/Y= 2024-04-16T20:15:34.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5j9EPlF79jpE39TIe6z0iu3hW/Y=', 'Content-Length': '0'} 2024-04-16T20:15:34.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6f436447a96b8f-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.066 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.067 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.067 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.067 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.067 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.067 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.067 DEBUG:boto:Data: 2024-04-16T20:15:34.067 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.068 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.068 DEBUG:boto:Params: {} 2024-04-16T20:15:34.068 DEBUG:boto:Token: None 2024-04-16T20:15:34.068 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5j9EPlF79jpE39TIe6z0iu3hW/Y= 2024-04-16T20:15:34.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5j9EPlF79jpE39TIe6z0iu3hW/Y=', 'Content-Length': '0'} 2024-04-16T20:15:34.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec31a67eae90a079-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:15:34.071 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.071 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.071 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.071 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.071 DEBUG:boto:Method: GET 2024-04-16T20:15:34.072 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.072 DEBUG:boto:Data: 2024-04-16T20:15:34.072 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.072 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.072 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.072 DEBUG:boto:Params: {} 2024-04-16T20:15:34.072 DEBUG:boto:Token: None 2024-04-16T20:15:34.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E= 2024-04-16T20:15:34.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E=', 'Content-Length': '0'} 2024-04-16T20:15:34.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000112ea7e957695945-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.118 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.119 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.119 DEBUG:boto:path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.119 DEBUG:boto:auth_path=/yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.119 DEBUG:boto:Method: GET 2024-04-16T20:15:34.119 DEBUG:boto:Path: /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.119 DEBUG:boto:Data: 2024-04-16T20:15:34.119 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.119 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.120 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.120 DEBUG:boto:Params: {} 2024-04-16T20:15:34.120 DEBUG:boto:Token: None 2024-04-16T20:15:34.120 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74?versionId=null 2024-04-16T20:15:34.120 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E= 2024-04-16T20:15:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3VxL/bCgKYTQ5KI1rKzzHivWZ8E=', 'Content-Length': '0'} 2024-04-16T20:15:34.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60587d2121d144f-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.123 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.123 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.123 DEBUG:boto:path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.123 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.123 DEBUG:boto:Method: GET 2024-04-16T20:15:34.123 DEBUG:boto:Path: /yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.123 DEBUG:boto:Data: 2024-04-16T20:15:34.124 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.124 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.124 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.124 DEBUG:boto:Params: {} 2024-04-16T20:15:34.124 DEBUG:boto:Token: None 2024-04-16T20:15:34.124 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018dd60b078789fac-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.127 DEBUG:boto:b'yrjefh-54obj741000falseobj742024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.128 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.128 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.128 DEBUG:boto:path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.128 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.128 DEBUG:boto:Method: GET 2024-04-16T20:15:34.128 DEBUG:boto:Path: /yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.128 DEBUG:boto:Data: 2024-04-16T20:15:34.128 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.129 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.129 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.129 DEBUG:boto:Params: {} 2024-04-16T20:15:34.129 DEBUG:boto:Token: None 2024-04-16T20:15:34.129 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000df8d5d2df5917f9-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.133 DEBUG:boto:b'yrjefh-54obj741000falseobj742024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.134 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.134 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.134 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.134 DEBUG:boto:Path: /yrjefh-54/obj74 2024-04-16T20:15:34.134 DEBUG:boto:Data: 2024-04-16T20:15:34.134 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.134 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.134 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.135 DEBUG:boto:Params: {} 2024-04-16T20:15:34.135 DEBUG:boto:Token: None 2024-04-16T20:15:34.135 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74 2024-04-16T20:15:34.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d5bhyMVBWTsSP3jijGiWwWOn9uw= 2024-04-16T20:15:34.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d5bhyMVBWTsSP3jijGiWwWOn9uw=', 'Content-Length': '0'} 2024-04-16T20:15:34.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a820a32b503359-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.138 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.138 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.138 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.138 DEBUG:boto:Path: /yrjefh-54/obj74 2024-04-16T20:15:34.138 DEBUG:boto:Data: 2024-04-16T20:15:34.138 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.138 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.139 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.139 DEBUG:boto:Params: {} 2024-04-16T20:15:34.139 DEBUG:boto:Token: None 2024-04-16T20:15:34.139 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74 2024-04-16T20:15:34.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:d5bhyMVBWTsSP3jijGiWwWOn9uw= 2024-04-16T20:15:34.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:d5bhyMVBWTsSP3jijGiWwWOn9uw=', 'Content-Length': '0'} 2024-04-16T20:15:34.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a0ca83161ef95a-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:15:34.141 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.142 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.142 DEBUG:boto:Method: GET 2024-04-16T20:15:34.142 DEBUG:boto:Path: /yrjefh-54/obj74 2024-04-16T20:15:34.142 DEBUG:boto:Data: 2024-04-16T20:15:34.142 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.142 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.142 DEBUG:boto:Params: {} 2024-04-16T20:15:34.142 DEBUG:boto:Token: None 2024-04-16T20:15:34.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74 2024-04-16T20:15:34.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kTPJxMFeLJAviAY5IImPfFkOV2M= 2024-04-16T20:15:34.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kTPJxMFeLJAviAY5IImPfFkOV2M=', 'Content-Length': '0'} 2024-04-16T20:15:34.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007932a11b71156902-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.190 DEBUG:boto:path=/yrjefh-54/obj74 2024-04-16T20:15:34.191 DEBUG:boto:auth_path=/yrjefh-54/obj74 2024-04-16T20:15:34.191 DEBUG:boto:Method: GET 2024-04-16T20:15:34.191 DEBUG:boto:Path: /yrjefh-54/obj74 2024-04-16T20:15:34.191 DEBUG:boto:Data: 2024-04-16T20:15:34.191 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.191 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.191 DEBUG:boto:Params: {} 2024-04-16T20:15:34.192 DEBUG:boto:Token: None 2024-04-16T20:15:34.192 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj74 2024-04-16T20:15:34.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kTPJxMFeLJAviAY5IImPfFkOV2M= 2024-04-16T20:15:34.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kTPJxMFeLJAviAY5IImPfFkOV2M=', 'Content-Length': '0'} 2024-04-16T20:15:34.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d655c2985761b5ec-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.194 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.194 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.195 DEBUG:boto:path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.195 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.195 DEBUG:boto:Method: GET 2024-04-16T20:15:34.195 DEBUG:boto:Path: /yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.195 DEBUG:boto:Data: 2024-04-16T20:15:34.195 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.195 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.195 DEBUG:boto:Params: {} 2024-04-16T20:15:34.196 DEBUG:boto:Token: None 2024-04-16T20:15:34.196 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.196 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd0c080a4dd2ec4-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.242 DEBUG:boto:b'yrjefh-54obj741000falseobj742024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.243 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.243 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.243 DEBUG:boto:path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.243 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.243 DEBUG:boto:Method: GET 2024-04-16T20:15:34.243 DEBUG:boto:Path: /yrjefh-54/?prefix=obj74 2024-04-16T20:15:34.243 DEBUG:boto:Data: 2024-04-16T20:15:34.244 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.244 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.244 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.244 DEBUG:boto:Params: {} 2024-04-16T20:15:34.244 DEBUG:boto:Token: None 2024-04-16T20:15:34.244 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.244 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a819ce164bed8df9-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.248 DEBUG:boto:b'yrjefh-54obj741000falseobj742024-04-16T20:12:51.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:15:34.249 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.249 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.249 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.249 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.249 DEBUG:boto:Method: GET 2024-04-16T20:15:34.249 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.250 DEBUG:boto:Data: 2024-04-16T20:15:34.250 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.250 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.250 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.250 DEBUG:boto:Params: {} 2024-04-16T20:15:34.250 DEBUG:boto:Token: None 2024-04-16T20:15:34.250 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc= 2024-04-16T20:15:34.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc=', 'Content-Length': '0'} 2024-04-16T20:15:34.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6eefa992b54cc4c-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.294 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.295 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.295 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.295 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.295 DEBUG:boto:Method: GET 2024-04-16T20:15:34.295 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.295 DEBUG:boto:Data: 2024-04-16T20:15:34.295 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.296 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.296 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.296 DEBUG:boto:Params: {} 2024-04-16T20:15:34.296 DEBUG:boto:Token: None 2024-04-16T20:15:34.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc= 2024-04-16T20:15:34.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc=', 'Content-Length': '0'} 2024-04-16T20:15:34.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000077756fb3a01dedc-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.299 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.299 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.299 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.299 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.300 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.300 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.300 DEBUG:boto:Data: 2024-04-16T20:15:34.300 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.300 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.300 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.300 DEBUG:boto:Params: {} 2024-04-16T20:15:34.300 DEBUG:boto:Token: None 2024-04-16T20:15:34.301 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.301 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrHnh6DmgHZAcd9oWK2kHWVK3xM= 2024-04-16T20:15:34.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrHnh6DmgHZAcd9oWK2kHWVK3xM=', 'Content-Length': '0'} 2024-04-16T20:15:34.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7b626e4b095c41-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.303 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.304 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.304 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.304 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.304 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.304 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.304 DEBUG:boto:Data: 2024-04-16T20:15:34.304 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.304 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.304 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.304 DEBUG:boto:Params: {} 2024-04-16T20:15:34.305 DEBUG:boto:Token: None 2024-04-16T20:15:34.305 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xrHnh6DmgHZAcd9oWK2kHWVK3xM= 2024-04-16T20:15:34.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xrHnh6DmgHZAcd9oWK2kHWVK3xM=', 'Content-Length': '0'} 2024-04-16T20:15:34.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ebf97034daacf7a-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:15:34.310 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.310 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.310 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.310 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.310 DEBUG:boto:Method: GET 2024-04-16T20:15:34.311 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.311 DEBUG:boto:Data: 2024-04-16T20:15:34.311 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.311 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.311 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.311 DEBUG:boto:Params: {} 2024-04-16T20:15:34.311 DEBUG:boto:Token: None 2024-04-16T20:15:34.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc= 2024-04-16T20:15:34.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc=', 'Content-Length': '0'} 2024-04-16T20:15:34.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db2acb144058264-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.354 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.355 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.355 DEBUG:boto:path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.355 DEBUG:boto:auth_path=/yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.355 DEBUG:boto:Method: GET 2024-04-16T20:15:34.355 DEBUG:boto:Path: /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.355 DEBUG:boto:Data: 2024-04-16T20:15:34.355 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.356 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.356 DEBUG:boto:Params: {} 2024-04-16T20:15:34.356 DEBUG:boto:Token: None 2024-04-16T20:15:34.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76?versionId=null 2024-04-16T20:15:34.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc= 2024-04-16T20:15:34.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QaIxDEXd0i0kPcqmlejQGc2IOAc=', 'Content-Length': '0'} 2024-04-16T20:15:34.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e507033acacd4d7-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.359 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.359 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.359 DEBUG:boto:path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.359 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.360 DEBUG:boto:Method: GET 2024-04-16T20:15:34.360 DEBUG:boto:Path: /yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.360 DEBUG:boto:Data: 2024-04-16T20:15:34.360 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.360 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.360 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.360 DEBUG:boto:Params: {} 2024-04-16T20:15:34.360 DEBUG:boto:Token: None 2024-04-16T20:15:34.361 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.361 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de9827603447537-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.364 DEBUG:boto:b'yrjefh-54obj761000falseobj762024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.364 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.364 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.364 DEBUG:boto:path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.364 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.365 DEBUG:boto:Method: GET 2024-04-16T20:15:34.365 DEBUG:boto:Path: /yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.365 DEBUG:boto:Data: 2024-04-16T20:15:34.365 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.365 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.365 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.365 DEBUG:boto:Params: {} 2024-04-16T20:15:34.365 DEBUG:boto:Token: None 2024-04-16T20:15:34.366 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.366 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f29f29ad947d032-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.370 DEBUG:boto:b'yrjefh-54obj761000falseobj762024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.370 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.370 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.371 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.371 DEBUG:boto:Path: /yrjefh-54/obj76 2024-04-16T20:15:34.371 DEBUG:boto:Data: 2024-04-16T20:15:34.371 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.371 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.371 DEBUG:boto:Params: {} 2024-04-16T20:15:34.372 DEBUG:boto:Token: None 2024-04-16T20:15:34.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76 2024-04-16T20:15:34.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9q+j66MZGG/2Pu7j8Kfb+R/NgT8= 2024-04-16T20:15:34.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9q+j66MZGG/2Pu7j8Kfb+R/NgT8=', 'Content-Length': '0'} 2024-04-16T20:15:34.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099645f3833bfd8b7-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.375 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.375 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.375 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.375 DEBUG:boto:Path: /yrjefh-54/obj76 2024-04-16T20:15:34.375 DEBUG:boto:Data: 2024-04-16T20:15:34.375 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.375 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.376 DEBUG:boto:Params: {} 2024-04-16T20:15:34.376 DEBUG:boto:Token: None 2024-04-16T20:15:34.376 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76 2024-04-16T20:15:34.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9q+j66MZGG/2Pu7j8Kfb+R/NgT8= 2024-04-16T20:15:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9q+j66MZGG/2Pu7j8Kfb+R/NgT8=', 'Content-Length': '0'} 2024-04-16T20:15:34.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c58ce1c034c793-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:15:34.379 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.379 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.379 DEBUG:boto:Method: GET 2024-04-16T20:15:34.379 DEBUG:boto:Path: /yrjefh-54/obj76 2024-04-16T20:15:34.379 DEBUG:boto:Data: 2024-04-16T20:15:34.379 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.379 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.379 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.379 DEBUG:boto:Params: {} 2024-04-16T20:15:34.380 DEBUG:boto:Token: None 2024-04-16T20:15:34.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76 2024-04-16T20:15:34.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hWuqkjNKdfjsAZMbuiKaaz20nGQ= 2024-04-16T20:15:34.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hWuqkjNKdfjsAZMbuiKaaz20nGQ=', 'Content-Length': '0'} 2024-04-16T20:15:34.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1d60a65cf6ea02-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.426 DEBUG:boto:path=/yrjefh-54/obj76 2024-04-16T20:15:34.427 DEBUG:boto:auth_path=/yrjefh-54/obj76 2024-04-16T20:15:34.427 DEBUG:boto:Method: GET 2024-04-16T20:15:34.427 DEBUG:boto:Path: /yrjefh-54/obj76 2024-04-16T20:15:34.427 DEBUG:boto:Data: 2024-04-16T20:15:34.427 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.427 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.427 DEBUG:boto:Params: {} 2024-04-16T20:15:34.428 DEBUG:boto:Token: None 2024-04-16T20:15:34.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj76 2024-04-16T20:15:34.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hWuqkjNKdfjsAZMbuiKaaz20nGQ= 2024-04-16T20:15:34.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hWuqkjNKdfjsAZMbuiKaaz20nGQ=', 'Content-Length': '0'} 2024-04-16T20:15:34.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d08cd1dc582b8a3-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.431 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.431 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.431 DEBUG:boto:path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.431 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.431 DEBUG:boto:Method: GET 2024-04-16T20:15:34.431 DEBUG:boto:Path: /yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.431 DEBUG:boto:Data: 2024-04-16T20:15:34.431 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.432 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.432 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.432 DEBUG:boto:Params: {} 2024-04-16T20:15:34.432 DEBUG:boto:Token: None 2024-04-16T20:15:34.432 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.432 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b501104f5daa2c55-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.478 DEBUG:boto:b'yrjefh-54obj761000falseobj762024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.479 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.479 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.479 DEBUG:boto:path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.479 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.480 DEBUG:boto:Method: GET 2024-04-16T20:15:34.480 DEBUG:boto:Path: /yrjefh-54/?prefix=obj76 2024-04-16T20:15:34.480 DEBUG:boto:Data: 2024-04-16T20:15:34.480 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.480 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.480 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.480 DEBUG:boto:Params: {} 2024-04-16T20:15:34.480 DEBUG:boto:Token: None 2024-04-16T20:15:34.481 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.481 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc57e97b6e0dbac-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.485 DEBUG:boto:b'yrjefh-54obj761000falseobj762024-04-16T20:12:50.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:15:34.486 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.486 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.486 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.486 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.486 DEBUG:boto:Method: GET 2024-04-16T20:15:34.486 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.486 DEBUG:boto:Data: 2024-04-16T20:15:34.487 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.487 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.487 DEBUG:boto:Params: {} 2024-04-16T20:15:34.487 DEBUG:boto:Token: None 2024-04-16T20:15:34.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.487 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc= 2024-04-16T20:15:34.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc=', 'Content-Length': '0'} 2024-04-16T20:15:34.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000660848a4bb92ddf3-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.531 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.531 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.531 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.531 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.531 DEBUG:boto:Method: GET 2024-04-16T20:15:34.531 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.531 DEBUG:boto:Data: 2024-04-16T20:15:34.532 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.532 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.532 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.532 DEBUG:boto:Params: {} 2024-04-16T20:15:34.532 DEBUG:boto:Token: None 2024-04-16T20:15:34.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.533 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc= 2024-04-16T20:15:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc=', 'Content-Length': '0'} 2024-04-16T20:15:34.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73d8a1b7579ce30-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.536 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.536 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.536 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.536 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.536 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.536 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.536 DEBUG:boto:Data: 2024-04-16T20:15:34.536 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.537 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.537 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.537 DEBUG:boto:Params: {} 2024-04-16T20:15:34.537 DEBUG:boto:Token: None 2024-04-16T20:15:34.537 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UB0bI1T9hsyvy4hvQFU0/IJReSU= 2024-04-16T20:15:34.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UB0bI1T9hsyvy4hvQFU0/IJReSU=', 'Content-Length': '0'} 2024-04-16T20:15:34.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0efcb953ab7ab1-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.540 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.540 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.540 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.540 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.541 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.541 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.541 DEBUG:boto:Data: 2024-04-16T20:15:34.541 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.541 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.541 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.541 DEBUG:boto:Params: {} 2024-04-16T20:15:34.541 DEBUG:boto:Token: None 2024-04-16T20:15:34.542 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UB0bI1T9hsyvy4hvQFU0/IJReSU= 2024-04-16T20:15:34.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UB0bI1T9hsyvy4hvQFU0/IJReSU=', 'Content-Length': '0'} 2024-04-16T20:15:34.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ff9944db51079f-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:15:34.545 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.545 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.545 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.545 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.545 DEBUG:boto:Method: GET 2024-04-16T20:15:34.545 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.545 DEBUG:boto:Data: 2024-04-16T20:15:34.546 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.546 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.546 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.546 DEBUG:boto:Params: {} 2024-04-16T20:15:34.546 DEBUG:boto:Token: None 2024-04-16T20:15:34.546 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.546 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc= 2024-04-16T20:15:34.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc=', 'Content-Length': '0'} 2024-04-16T20:15:34.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f7b1028f313fa1-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.590 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.591 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.591 DEBUG:boto:path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.591 DEBUG:boto:auth_path=/yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.591 DEBUG:boto:Method: GET 2024-04-16T20:15:34.591 DEBUG:boto:Path: /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.591 DEBUG:boto:Data: 2024-04-16T20:15:34.591 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.592 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.592 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.592 DEBUG:boto:Params: {} 2024-04-16T20:15:34.592 DEBUG:boto:Token: None 2024-04-16T20:15:34.592 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85?versionId=null 2024-04-16T20:15:34.592 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc= 2024-04-16T20:15:34.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zx+lOFkW4nYPuafO0OwS4CHF4nc=', 'Content-Length': '0'} 2024-04-16T20:15:34.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b47710c718070be-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.595 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.596 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.596 DEBUG:boto:path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.596 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.596 DEBUG:boto:Method: GET 2024-04-16T20:15:34.596 DEBUG:boto:Path: /yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.596 DEBUG:boto:Data: 2024-04-16T20:15:34.596 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.597 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.597 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.597 DEBUG:boto:Params: {} 2024-04-16T20:15:34.597 DEBUG:boto:Token: None 2024-04-16T20:15:34.597 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f220fd48df295c99-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.600 DEBUG:boto:b'yrjefh-54obj851000falseobj852024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.601 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.601 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.601 DEBUG:boto:path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.601 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.601 DEBUG:boto:Method: GET 2024-04-16T20:15:34.601 DEBUG:boto:Path: /yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.601 DEBUG:boto:Data: 2024-04-16T20:15:34.601 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.602 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.602 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.602 DEBUG:boto:Params: {} 2024-04-16T20:15:34.602 DEBUG:boto:Token: None 2024-04-16T20:15:34.602 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.602 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067baa0c4fc38fa96-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.607 DEBUG:boto:b'yrjefh-54obj851000falseobj852024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.607 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.607 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.607 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.608 DEBUG:boto:Path: /yrjefh-54/obj85 2024-04-16T20:15:34.608 DEBUG:boto:Data: 2024-04-16T20:15:34.608 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.608 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.608 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.608 DEBUG:boto:Params: {} 2024-04-16T20:15:34.608 DEBUG:boto:Token: None 2024-04-16T20:15:34.608 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85 2024-04-16T20:15:34.609 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AL6h0tUV13oOYFfvSaviVt3pI+o= 2024-04-16T20:15:34.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AL6h0tUV13oOYFfvSaviVt3pI+o=', 'Content-Length': '0'} 2024-04-16T20:15:34.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880ae878e81f7510-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.611 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.612 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.612 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.612 DEBUG:boto:Path: /yrjefh-54/obj85 2024-04-16T20:15:34.612 DEBUG:boto:Data: 2024-04-16T20:15:34.612 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.612 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.612 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.613 DEBUG:boto:Params: {} 2024-04-16T20:15:34.613 DEBUG:boto:Token: None 2024-04-16T20:15:34.613 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85 2024-04-16T20:15:34.613 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AL6h0tUV13oOYFfvSaviVt3pI+o= 2024-04-16T20:15:34.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AL6h0tUV13oOYFfvSaviVt3pI+o=', 'Content-Length': '0'} 2024-04-16T20:15:34.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe149d7ffec5691-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:15:34.616 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.616 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.616 DEBUG:boto:Method: GET 2024-04-16T20:15:34.617 DEBUG:boto:Path: /yrjefh-54/obj85 2024-04-16T20:15:34.617 DEBUG:boto:Data: 2024-04-16T20:15:34.617 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.617 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.617 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.617 DEBUG:boto:Params: {} 2024-04-16T20:15:34.617 DEBUG:boto:Token: None 2024-04-16T20:15:34.617 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85 2024-04-16T20:15:34.618 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r+D5hDrRD1NC+udsqJR3YLDuu2c= 2024-04-16T20:15:34.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r+D5hDrRD1NC+udsqJR3YLDuu2c=', 'Content-Length': '0'} 2024-04-16T20:15:34.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c930e83e2efe3ac7-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.662 DEBUG:boto:path=/yrjefh-54/obj85 2024-04-16T20:15:34.663 DEBUG:boto:auth_path=/yrjefh-54/obj85 2024-04-16T20:15:34.663 DEBUG:boto:Method: GET 2024-04-16T20:15:34.663 DEBUG:boto:Path: /yrjefh-54/obj85 2024-04-16T20:15:34.663 DEBUG:boto:Data: 2024-04-16T20:15:34.663 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.663 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.663 DEBUG:boto:Params: {} 2024-04-16T20:15:34.664 DEBUG:boto:Token: None 2024-04-16T20:15:34.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj85 2024-04-16T20:15:34.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r+D5hDrRD1NC+udsqJR3YLDuu2c= 2024-04-16T20:15:34.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r+D5hDrRD1NC+udsqJR3YLDuu2c=', 'Content-Length': '0'} 2024-04-16T20:15:34.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af02875161fc92c3-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.667 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.667 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.667 DEBUG:boto:path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.667 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.667 DEBUG:boto:Method: GET 2024-04-16T20:15:34.667 DEBUG:boto:Path: /yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.667 DEBUG:boto:Data: 2024-04-16T20:15:34.668 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.668 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.668 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.668 DEBUG:boto:Params: {} 2024-04-16T20:15:34.668 DEBUG:boto:Token: None 2024-04-16T20:15:34.668 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.668 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a15e617df27bba-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.671 DEBUG:boto:b'yrjefh-54obj851000falseobj852024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.672 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.672 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.672 DEBUG:boto:path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.672 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.672 DEBUG:boto:Method: GET 2024-04-16T20:15:34.673 DEBUG:boto:Path: /yrjefh-54/?prefix=obj85 2024-04-16T20:15:34.673 DEBUG:boto:Data: 2024-04-16T20:15:34.673 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.673 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.673 DEBUG:boto:Params: {} 2024-04-16T20:15:34.673 DEBUG:boto:Token: None 2024-04-16T20:15:34.673 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.674 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037e5141ce1836403-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.718 DEBUG:boto:b'yrjefh-54obj851000falseobj852024-04-16T20:12:50.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:15:34.719 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.719 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.719 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.719 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.720 DEBUG:boto:Method: GET 2024-04-16T20:15:34.720 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.720 DEBUG:boto:Data: 2024-04-16T20:15:34.720 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.720 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.720 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.720 DEBUG:boto:Params: {} 2024-04-16T20:15:34.720 DEBUG:boto:Token: None 2024-04-16T20:15:34.721 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.721 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs= 2024-04-16T20:15:34.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs=', 'Content-Length': '0'} 2024-04-16T20:15:34.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b7a0a2f647ea4a8-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.766 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.767 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.767 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.767 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.767 DEBUG:boto:Method: GET 2024-04-16T20:15:34.767 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.767 DEBUG:boto:Data: 2024-04-16T20:15:34.767 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.768 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.768 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.768 DEBUG:boto:Params: {} 2024-04-16T20:15:34.768 DEBUG:boto:Token: None 2024-04-16T20:15:34.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs= 2024-04-16T20:15:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs=', 'Content-Length': '0'} 2024-04-16T20:15:34.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5c5bf941c35da9a-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.771 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.771 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.772 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.772 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.772 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.772 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.772 DEBUG:boto:Data: 2024-04-16T20:15:34.772 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.772 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.772 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.773 DEBUG:boto:Params: {} 2024-04-16T20:15:34.773 DEBUG:boto:Token: None 2024-04-16T20:15:34.773 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.773 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IRq8DunLOtSMe9Zocr0w6UsK5YI= 2024-04-16T20:15:34.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IRq8DunLOtSMe9Zocr0w6UsK5YI=', 'Content-Length': '0'} 2024-04-16T20:15:34.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d456e9c3dadd77a-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.776 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.776 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.776 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.776 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.776 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.777 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.777 DEBUG:boto:Data: 2024-04-16T20:15:34.777 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.777 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.777 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.777 DEBUG:boto:Params: {} 2024-04-16T20:15:34.777 DEBUG:boto:Token: None 2024-04-16T20:15:34.778 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.778 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IRq8DunLOtSMe9Zocr0w6UsK5YI= 2024-04-16T20:15:34.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IRq8DunLOtSMe9Zocr0w6UsK5YI=', 'Content-Length': '0'} 2024-04-16T20:15:34.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ed27c627e88ad1-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:15:34.781 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.781 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.781 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.781 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.781 DEBUG:boto:Method: GET 2024-04-16T20:15:34.781 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.782 DEBUG:boto:Data: 2024-04-16T20:15:34.782 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.782 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.782 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.782 DEBUG:boto:Params: {} 2024-04-16T20:15:34.782 DEBUG:boto:Token: None 2024-04-16T20:15:34.782 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs= 2024-04-16T20:15:34.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs=', 'Content-Length': '0'} 2024-04-16T20:15:34.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819f4765f99625fc-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.826 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.827 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.827 DEBUG:boto:path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.827 DEBUG:boto:auth_path=/yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.827 DEBUG:boto:Method: GET 2024-04-16T20:15:34.827 DEBUG:boto:Path: /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.827 DEBUG:boto:Data: 2024-04-16T20:15:34.827 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.828 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.828 DEBUG:boto:Params: {} 2024-04-16T20:15:34.828 DEBUG:boto:Token: None 2024-04-16T20:15:34.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87?versionId=null 2024-04-16T20:15:34.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs= 2024-04-16T20:15:34.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wFI/Qc0jJ2wwfMSctt3m+xyQbAs=', 'Content-Length': '0'} 2024-04-16T20:15:34.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6388f82fdcc3c8e-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.831 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.831 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.832 DEBUG:boto:path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.832 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.832 DEBUG:boto:Method: GET 2024-04-16T20:15:34.832 DEBUG:boto:Path: /yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.832 DEBUG:boto:Data: 2024-04-16T20:15:34.832 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.832 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.832 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.833 DEBUG:boto:Params: {} 2024-04-16T20:15:34.833 DEBUG:boto:Token: None 2024-04-16T20:15:34.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007976e2dff277f995-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.836 DEBUG:boto:b'yrjefh-54obj871000falseobj872024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.837 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.837 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.837 DEBUG:boto:path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.837 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.837 DEBUG:boto:Method: GET 2024-04-16T20:15:34.837 DEBUG:boto:Path: /yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.838 DEBUG:boto:Data: 2024-04-16T20:15:34.838 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.838 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.838 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.838 DEBUG:boto:Params: {} 2024-04-16T20:15:34.838 DEBUG:boto:Token: None 2024-04-16T20:15:34.838 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005524524a329f128-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.843 DEBUG:boto:b'yrjefh-54obj871000falseobj872024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.843 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.843 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.843 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.843 DEBUG:boto:Path: /yrjefh-54/obj87 2024-04-16T20:15:34.844 DEBUG:boto:Data: 2024-04-16T20:15:34.844 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.844 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.844 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.844 DEBUG:boto:Params: {} 2024-04-16T20:15:34.844 DEBUG:boto:Token: None 2024-04-16T20:15:34.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87 2024-04-16T20:15:34.845 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rHlW+IN06jg2qhE8sdbqT3swcRo= 2024-04-16T20:15:34.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rHlW+IN06jg2qhE8sdbqT3swcRo=', 'Content-Length': '0'} 2024-04-16T20:15:34.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2fb65cce1c61909-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.847 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.848 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.848 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.848 DEBUG:boto:Path: /yrjefh-54/obj87 2024-04-16T20:15:34.848 DEBUG:boto:Data: 2024-04-16T20:15:34.848 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.848 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.848 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.848 DEBUG:boto:Params: {} 2024-04-16T20:15:34.849 DEBUG:boto:Token: None 2024-04-16T20:15:34.849 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87 2024-04-16T20:15:34.849 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rHlW+IN06jg2qhE8sdbqT3swcRo= 2024-04-16T20:15:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rHlW+IN06jg2qhE8sdbqT3swcRo=', 'Content-Length': '0'} 2024-04-16T20:15:34.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ee2dad6c3cbcb0-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:15:34.852 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.852 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.852 DEBUG:boto:Method: GET 2024-04-16T20:15:34.852 DEBUG:boto:Path: /yrjefh-54/obj87 2024-04-16T20:15:34.852 DEBUG:boto:Data: 2024-04-16T20:15:34.852 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.852 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.853 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.853 DEBUG:boto:Params: {} 2024-04-16T20:15:34.853 DEBUG:boto:Token: None 2024-04-16T20:15:34.853 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87 2024-04-16T20:15:34.853 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNcjZOBQrO8TT2YeMjevcwW94Qs= 2024-04-16T20:15:34.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNcjZOBQrO8TT2YeMjevcwW94Qs=', 'Content-Length': '0'} 2024-04-16T20:15:34.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a48529d5b303fb3-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.899 DEBUG:boto:path=/yrjefh-54/obj87 2024-04-16T20:15:34.899 DEBUG:boto:auth_path=/yrjefh-54/obj87 2024-04-16T20:15:34.899 DEBUG:boto:Method: GET 2024-04-16T20:15:34.899 DEBUG:boto:Path: /yrjefh-54/obj87 2024-04-16T20:15:34.899 DEBUG:boto:Data: 2024-04-16T20:15:34.899 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.899 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.900 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.900 DEBUG:boto:Params: {} 2024-04-16T20:15:34.900 DEBUG:boto:Token: None 2024-04-16T20:15:34.900 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj87 2024-04-16T20:15:34.900 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oNcjZOBQrO8TT2YeMjevcwW94Qs= 2024-04-16T20:15:34.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oNcjZOBQrO8TT2YeMjevcwW94Qs=', 'Content-Length': '0'} 2024-04-16T20:15:34.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf890db51fe0791-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.903 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.903 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.903 DEBUG:boto:path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.904 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.904 DEBUG:boto:Method: GET 2024-04-16T20:15:34.904 DEBUG:boto:Path: /yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.904 DEBUG:boto:Data: 2024-04-16T20:15:34.904 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.904 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.904 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.904 DEBUG:boto:Params: {} 2024-04-16T20:15:34.905 DEBUG:boto:Token: None 2024-04-16T20:15:34.905 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e34180063575cef9-00661edc66-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.908 DEBUG:boto:b'yrjefh-54obj871000falseobj872024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.909 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:34.909 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:34.909 DEBUG:boto:path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.909 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.909 DEBUG:boto:Method: GET 2024-04-16T20:15:34.909 DEBUG:boto:Path: /yrjefh-54/?prefix=obj87 2024-04-16T20:15:34.909 DEBUG:boto:Data: 2024-04-16T20:15:34.909 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.910 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.910 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.910 DEBUG:boto:Params: {} 2024-04-16T20:15:34.910 DEBUG:boto:Token: None 2024-04-16T20:15:34.910 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/ 2024-04-16T20:15:34.910 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U= 2024-04-16T20:15:34.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Q/rjeRJSmj43461dkwwNP0n23U=', 'Content-Length': '0'} 2024-04-16T20:15:34.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9b6f7684fb4d80-00661edc66-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.915 DEBUG:boto:b'yrjefh-54obj871000falseobj872024-04-16T20:12:51.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:34.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:15:34.915 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:34.915 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:34.915 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.915 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.916 DEBUG:boto:Method: GET 2024-04-16T20:15:34.916 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.916 DEBUG:boto:Data: 2024-04-16T20:15:34.916 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.916 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.916 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.916 DEBUG:boto:Params: {} 2024-04-16T20:15:34.916 DEBUG:boto:Token: None 2024-04-16T20:15:34.917 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.917 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE= 2024-04-16T20:15:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE=', 'Content-Length': '0'} 2024-04-16T20:15:34.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006faa3a6a631b1d11-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.963 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:34.963 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:34.963 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.963 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.963 DEBUG:boto:Method: GET 2024-04-16T20:15:34.963 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.963 DEBUG:boto:Data: 2024-04-16T20:15:34.963 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.964 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.964 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.964 DEBUG:boto:Params: {} 2024-04-16T20:15:34.964 DEBUG:boto:Token: None 2024-04-16T20:15:34.964 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.964 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE= 2024-04-16T20:15:34.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE=', 'Content-Length': '0'} 2024-04-16T20:15:34.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae90c27a095f6583-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.967 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:34.967 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:34.968 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.968 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.968 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.968 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.968 DEBUG:boto:Data: 2024-04-16T20:15:34.968 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.968 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.968 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.968 DEBUG:boto:Params: {} 2024-04-16T20:15:34.969 DEBUG:boto:Token: None 2024-04-16T20:15:34.969 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.969 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6FzWpT03rvsKUZALvLtcl618RI= 2024-04-16T20:15:34.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6FzWpT03rvsKUZALvLtcl618RI=', 'Content-Length': '0'} 2024-04-16T20:15:34.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004948d973765647a7-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.971 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:34.971 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:34.972 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.972 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.972 DEBUG:boto:Method: HEAD 2024-04-16T20:15:34.972 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.972 DEBUG:boto:Data: 2024-04-16T20:15:34.972 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.972 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:34.972 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.972 DEBUG:boto:Params: {} 2024-04-16T20:15:34.973 DEBUG:boto:Token: None 2024-04-16T20:15:34.973 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.973 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:m6FzWpT03rvsKUZALvLtcl618RI= 2024-04-16T20:15:34.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:m6FzWpT03rvsKUZALvLtcl618RI=', 'Content-Length': '0'} 2024-04-16T20:15:34.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005319553dfbf2b9c-00661edc66-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:34.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:15:34.976 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:34.976 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:34.976 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.976 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.976 DEBUG:boto:Method: GET 2024-04-16T20:15:34.976 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.976 DEBUG:boto:Data: 2024-04-16T20:15:34.976 DEBUG:boto:Headers: {} 2024-04-16T20:15:34.977 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:34.977 DEBUG:boto:Port: 8000 2024-04-16T20:15:34.977 DEBUG:boto:Params: {} 2024-04-16T20:15:34.977 DEBUG:boto:Token: None 2024-04-16T20:15:34.977 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:34 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:34.977 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE= 2024-04-16T20:15:34.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jQbOYrbTfINwNYukfTW5zUHdyE=', 'Content-Length': '0'} 2024-04-16T20:15:34.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f0c2d34d4bf7b8-00661edc66-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.022 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:35.023 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:35.023 DEBUG:boto:path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:35.023 DEBUG:boto:auth_path=/yrjefh-54/obj92?versionId=null 2024-04-16T20:15:35.023 DEBUG:boto:Method: GET 2024-04-16T20:15:35.023 DEBUG:boto:Path: /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:35.023 DEBUG:boto:Data: 2024-04-16T20:15:35.023 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.023 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.023 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.024 DEBUG:boto:Params: {} 2024-04-16T20:15:35.024 DEBUG:boto:Token: None 2024-04-16T20:15:35.024 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj92?versionId=null 2024-04-16T20:15:35.024 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:UNSYCFrGvTHh+DIFGrBxpUhcB78= 2024-04-16T20:15:35.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UNSYCFrGvTHh+DIFGrBxpUhcB78=', 'Content-Length': '0'} 2024-04-16T20:15:35.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22a321b77da4892-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.026 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.027 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.027 DEBUG:boto:path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.027 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.027 DEBUG:boto:Method: GET 2024-04-16T20:15:35.027 DEBUG:boto:Path: /yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.027 DEBUG:boto:Data: 2024-04-16T20:15:35.027 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.027 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.027 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.028 DEBUG:boto:Params: {} 2024-04-16T20:15:35.028 DEBUG:boto:Token: None 2024-04-16T20:15:35.028 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.028 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000099b5c1dc70c0487-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.031 DEBUG:boto:b'yrjefh-54obj921000falseobj922024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.031 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.031 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.031 DEBUG:boto:path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.031 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.032 DEBUG:boto:Method: GET 2024-04-16T20:15:35.032 DEBUG:boto:Path: /yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.032 DEBUG:boto:Data: 2024-04-16T20:15:35.032 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.032 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.032 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.032 DEBUG:boto:Params: {} 2024-04-16T20:15:35.032 DEBUG:boto:Token: None 2024-04-16T20:15:35.033 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.033 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e5acd867d7268d-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.037 DEBUG:boto:b'yrjefh-54obj921000falseobj922024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.037 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:35.037 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:35.037 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.037 DEBUG:boto:Path: /yrjefh-54/obj92 2024-04-16T20:15:35.037 DEBUG:boto:Data: 2024-04-16T20:15:35.037 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.038 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.038 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.038 DEBUG:boto:Params: {} 2024-04-16T20:15:35.038 DEBUG:boto:Token: None 2024-04-16T20:15:35.038 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj92 2024-04-16T20:15:35.038 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gj01RETRgVxev5fzBt1pypsRSJA= 2024-04-16T20:15:35.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gj01RETRgVxev5fzBt1pypsRSJA=', 'Content-Length': '0'} 2024-04-16T20:15:35.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d52aa52c70f1b50-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.041 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:35.041 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:35.041 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.041 DEBUG:boto:Path: /yrjefh-54/obj92 2024-04-16T20:15:35.041 DEBUG:boto:Data: 2024-04-16T20:15:35.041 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.041 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.041 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.041 DEBUG:boto:Params: {} 2024-04-16T20:15:35.042 DEBUG:boto:Token: None 2024-04-16T20:15:35.042 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj92 2024-04-16T20:15:35.042 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gj01RETRgVxev5fzBt1pypsRSJA= 2024-04-16T20:15:35.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gj01RETRgVxev5fzBt1pypsRSJA=', 'Content-Length': '0'} 2024-04-16T20:15:35.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e13370766b91bf1-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:15:35.044 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:35.044 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:35.045 DEBUG:boto:Method: GET 2024-04-16T20:15:35.045 DEBUG:boto:Path: /yrjefh-54/obj92 2024-04-16T20:15:35.045 DEBUG:boto:Data: 2024-04-16T20:15:35.045 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.045 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.045 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.045 DEBUG:boto:Params: {} 2024-04-16T20:15:35.045 DEBUG:boto:Token: None 2024-04-16T20:15:35.045 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj92 2024-04-16T20:15:35.046 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WSB1Yf+oS55T7zumTyB9M8r9Uik= 2024-04-16T20:15:35.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WSB1Yf+oS55T7zumTyB9M8r9Uik=', 'Content-Length': '0'} 2024-04-16T20:15:35.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062bed02cafb09fe4-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.090 DEBUG:boto:path=/yrjefh-54/obj92 2024-04-16T20:15:35.091 DEBUG:boto:auth_path=/yrjefh-54/obj92 2024-04-16T20:15:35.091 DEBUG:boto:Method: GET 2024-04-16T20:15:35.091 DEBUG:boto:Path: /yrjefh-54/obj92 2024-04-16T20:15:35.091 DEBUG:boto:Data: 2024-04-16T20:15:35.091 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.091 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.091 DEBUG:boto:Params: {} 2024-04-16T20:15:35.091 DEBUG:boto:Token: None 2024-04-16T20:15:35.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj92 2024-04-16T20:15:35.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WSB1Yf+oS55T7zumTyB9M8r9Uik= 2024-04-16T20:15:35.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WSB1Yf+oS55T7zumTyB9M8r9Uik=', 'Content-Length': '0'} 2024-04-16T20:15:35.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be97c6a0c299983-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.094 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.094 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.095 DEBUG:boto:path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.095 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.095 DEBUG:boto:Method: GET 2024-04-16T20:15:35.095 DEBUG:boto:Path: /yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.095 DEBUG:boto:Data: 2024-04-16T20:15:35.095 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.095 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.095 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.095 DEBUG:boto:Params: {} 2024-04-16T20:15:35.095 DEBUG:boto:Token: None 2024-04-16T20:15:35.096 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.096 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e27bed98e2a2d6-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.138 DEBUG:boto:b'yrjefh-54obj921000falseobj922024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.139 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.139 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.139 DEBUG:boto:path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.139 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.139 DEBUG:boto:Method: GET 2024-04-16T20:15:35.139 DEBUG:boto:Path: /yrjefh-54/?prefix=obj92 2024-04-16T20:15:35.139 DEBUG:boto:Data: 2024-04-16T20:15:35.139 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.140 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.140 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.140 DEBUG:boto:Params: {} 2024-04-16T20:15:35.140 DEBUG:boto:Token: None 2024-04-16T20:15:35.140 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0bef6582f46fcb-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.144 DEBUG:boto:b'yrjefh-54obj921000falseobj922024-04-16T20:12:51.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:15:35.144 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.144 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.144 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.145 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.145 DEBUG:boto:Method: GET 2024-04-16T20:15:35.145 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.145 DEBUG:boto:Data: 2024-04-16T20:15:35.145 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.145 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.145 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.145 DEBUG:boto:Params: {} 2024-04-16T20:15:35.145 DEBUG:boto:Token: None 2024-04-16T20:15:35.146 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.146 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY= 2024-04-16T20:15:35.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY=', 'Content-Length': '0'} 2024-04-16T20:15:35.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d816a2d58be1c05a-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.190 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.191 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.191 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.191 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.191 DEBUG:boto:Method: GET 2024-04-16T20:15:35.191 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.191 DEBUG:boto:Data: 2024-04-16T20:15:35.191 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.191 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.191 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.191 DEBUG:boto:Params: {} 2024-04-16T20:15:35.192 DEBUG:boto:Token: None 2024-04-16T20:15:35.192 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.192 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY= 2024-04-16T20:15:35.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY=', 'Content-Length': '0'} 2024-04-16T20:15:35.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046888b4642be632d-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.194 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.194 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.194 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.194 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.195 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.195 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.195 DEBUG:boto:Data: 2024-04-16T20:15:35.195 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.195 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.195 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.195 DEBUG:boto:Params: {} 2024-04-16T20:15:35.195 DEBUG:boto:Token: None 2024-04-16T20:15:35.195 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1HXLeycFqcFz4kMTdGQoHwf7W7U= 2024-04-16T20:15:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1HXLeycFqcFz4kMTdGQoHwf7W7U=', 'Content-Length': '0'} 2024-04-16T20:15:35.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a760bdd8631f62f5-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.197 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.198 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.198 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.198 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.198 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.198 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.198 DEBUG:boto:Data: 2024-04-16T20:15:35.198 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.198 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.198 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.198 DEBUG:boto:Params: {} 2024-04-16T20:15:35.199 DEBUG:boto:Token: None 2024-04-16T20:15:35.199 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.199 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1HXLeycFqcFz4kMTdGQoHwf7W7U= 2024-04-16T20:15:35.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1HXLeycFqcFz4kMTdGQoHwf7W7U=', 'Content-Length': '0'} 2024-04-16T20:15:35.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7035c4c1f73218-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:15:35.201 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.201 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.201 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.202 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.202 DEBUG:boto:Method: GET 2024-04-16T20:15:35.202 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.202 DEBUG:boto:Data: 2024-04-16T20:15:35.202 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.202 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.202 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.202 DEBUG:boto:Params: {} 2024-04-16T20:15:35.202 DEBUG:boto:Token: None 2024-04-16T20:15:35.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.203 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY= 2024-04-16T20:15:35.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY=', 'Content-Length': '0'} 2024-04-16T20:15:35.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763fd2d7d7593bf9-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.246 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.247 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.247 DEBUG:boto:path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.247 DEBUG:boto:auth_path=/yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.247 DEBUG:boto:Method: GET 2024-04-16T20:15:35.247 DEBUG:boto:Path: /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.247 DEBUG:boto:Data: 2024-04-16T20:15:35.247 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.247 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.247 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.247 DEBUG:boto:Params: {} 2024-04-16T20:15:35.248 DEBUG:boto:Token: None 2024-04-16T20:15:35.248 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93?versionId=null 2024-04-16T20:15:35.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY= 2024-04-16T20:15:35.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ChmwlMNHy/khfhGePHZ1JqvpYhY=', 'Content-Length': '0'} 2024-04-16T20:15:35.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72a5fd3214c7ba7-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.250 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.250 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.250 DEBUG:boto:path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.250 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.251 DEBUG:boto:Method: GET 2024-04-16T20:15:35.251 DEBUG:boto:Path: /yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.251 DEBUG:boto:Data: 2024-04-16T20:15:35.251 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.251 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.251 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.251 DEBUG:boto:Params: {} 2024-04-16T20:15:35.251 DEBUG:boto:Token: None 2024-04-16T20:15:35.251 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce1ba1ebb7ebdc5-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.294 DEBUG:boto:b'yrjefh-54obj931000falseobj932024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.295 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.295 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.295 DEBUG:boto:path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.295 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.295 DEBUG:boto:Method: GET 2024-04-16T20:15:35.295 DEBUG:boto:Path: /yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.295 DEBUG:boto:Data: 2024-04-16T20:15:35.295 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.295 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.296 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.296 DEBUG:boto:Params: {} 2024-04-16T20:15:35.296 DEBUG:boto:Token: None 2024-04-16T20:15:35.296 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.296 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa349a901e68be3-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.300 DEBUG:boto:b'yrjefh-54obj931000falseobj932024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.300 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.300 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.301 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.301 DEBUG:boto:Path: /yrjefh-54/obj93 2024-04-16T20:15:35.301 DEBUG:boto:Data: 2024-04-16T20:15:35.301 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.301 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.301 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.301 DEBUG:boto:Params: {} 2024-04-16T20:15:35.301 DEBUG:boto:Token: None 2024-04-16T20:15:35.301 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93 2024-04-16T20:15:35.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNkidXHlWYMy6I+emGfOZj399rE= 2024-04-16T20:15:35.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNkidXHlWYMy6I+emGfOZj399rE=', 'Content-Length': '0'} 2024-04-16T20:15:35.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006687153442e76e70-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.304 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.304 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.304 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.304 DEBUG:boto:Path: /yrjefh-54/obj93 2024-04-16T20:15:35.304 DEBUG:boto:Data: 2024-04-16T20:15:35.304 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.304 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.304 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.304 DEBUG:boto:Params: {} 2024-04-16T20:15:35.305 DEBUG:boto:Token: None 2024-04-16T20:15:35.305 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93 2024-04-16T20:15:35.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dNkidXHlWYMy6I+emGfOZj399rE= 2024-04-16T20:15:35.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dNkidXHlWYMy6I+emGfOZj399rE=', 'Content-Length': '0'} 2024-04-16T20:15:35.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86bac7fa868cb7b-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:15:35.307 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.307 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.308 DEBUG:boto:Method: GET 2024-04-16T20:15:35.308 DEBUG:boto:Path: /yrjefh-54/obj93 2024-04-16T20:15:35.308 DEBUG:boto:Data: 2024-04-16T20:15:35.308 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.308 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.308 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.308 DEBUG:boto:Params: {} 2024-04-16T20:15:35.308 DEBUG:boto:Token: None 2024-04-16T20:15:35.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93 2024-04-16T20:15:35.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDWwBIHpfXxVdKiAdu6WbUEuHHY= 2024-04-16T20:15:35.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDWwBIHpfXxVdKiAdu6WbUEuHHY=', 'Content-Length': '0'} 2024-04-16T20:15:35.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b61540c7863803-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.354 DEBUG:boto:path=/yrjefh-54/obj93 2024-04-16T20:15:35.354 DEBUG:boto:auth_path=/yrjefh-54/obj93 2024-04-16T20:15:35.355 DEBUG:boto:Method: GET 2024-04-16T20:15:35.355 DEBUG:boto:Path: /yrjefh-54/obj93 2024-04-16T20:15:35.355 DEBUG:boto:Data: 2024-04-16T20:15:35.355 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.355 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.355 DEBUG:boto:Params: {} 2024-04-16T20:15:35.355 DEBUG:boto:Token: None 2024-04-16T20:15:35.355 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj93 2024-04-16T20:15:35.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sDWwBIHpfXxVdKiAdu6WbUEuHHY= 2024-04-16T20:15:35.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sDWwBIHpfXxVdKiAdu6WbUEuHHY=', 'Content-Length': '0'} 2024-04-16T20:15:35.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cabd14ff5c6a30cd-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.358 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.358 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.358 DEBUG:boto:path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.358 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.358 DEBUG:boto:Method: GET 2024-04-16T20:15:35.358 DEBUG:boto:Path: /yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.359 DEBUG:boto:Data: 2024-04-16T20:15:35.359 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.359 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.359 DEBUG:boto:Params: {} 2024-04-16T20:15:35.359 DEBUG:boto:Token: None 2024-04-16T20:15:35.359 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.359 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0fd857b2499ed69-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.402 DEBUG:boto:b'yrjefh-54obj931000falseobj932024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.403 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.403 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.403 DEBUG:boto:path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.403 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.403 DEBUG:boto:Method: GET 2024-04-16T20:15:35.403 DEBUG:boto:Path: /yrjefh-54/?prefix=obj93 2024-04-16T20:15:35.403 DEBUG:boto:Data: 2024-04-16T20:15:35.404 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.404 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.404 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.404 DEBUG:boto:Params: {} 2024-04-16T20:15:35.404 DEBUG:boto:Token: None 2024-04-16T20:15:35.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07d9867094e291e-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.408 DEBUG:boto:b'yrjefh-54obj931000falseobj932024-04-16T20:12:51.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:15:35.408 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.408 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.408 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.408 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.408 DEBUG:boto:Method: GET 2024-04-16T20:15:35.409 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.409 DEBUG:boto:Data: 2024-04-16T20:15:35.409 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.409 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.409 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.409 DEBUG:boto:Params: {} 2024-04-16T20:15:35.409 DEBUG:boto:Token: None 2024-04-16T20:15:35.409 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg= 2024-04-16T20:15:35.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg=', 'Content-Length': '0'} 2024-04-16T20:15:35.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da3b930911798bf0-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.454 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.454 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.455 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.455 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.455 DEBUG:boto:Method: GET 2024-04-16T20:15:35.455 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.455 DEBUG:boto:Data: 2024-04-16T20:15:35.455 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.455 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.455 DEBUG:boto:Params: {} 2024-04-16T20:15:35.455 DEBUG:boto:Token: None 2024-04-16T20:15:35.455 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg= 2024-04-16T20:15:35.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg=', 'Content-Length': '0'} 2024-04-16T20:15:35.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3f8bea38f83368-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.458 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.458 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.459 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.459 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.459 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.459 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.459 DEBUG:boto:Data: 2024-04-16T20:15:35.459 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.459 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.459 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.459 DEBUG:boto:Params: {} 2024-04-16T20:15:35.459 DEBUG:boto:Token: None 2024-04-16T20:15:35.459 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l/3/+BBbmBEYAMCxerNzZnt2yzI= 2024-04-16T20:15:35.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l/3/+BBbmBEYAMCxerNzZnt2yzI=', 'Content-Length': '0'} 2024-04-16T20:15:35.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003346db3014a5a3c4-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.462 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.462 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.462 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.462 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.462 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.462 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.462 DEBUG:boto:Data: 2024-04-16T20:15:35.463 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.463 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.463 DEBUG:boto:Params: {} 2024-04-16T20:15:35.463 DEBUG:boto:Token: None 2024-04-16T20:15:35.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:l/3/+BBbmBEYAMCxerNzZnt2yzI= 2024-04-16T20:15:35.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:l/3/+BBbmBEYAMCxerNzZnt2yzI=', 'Content-Length': '0'} 2024-04-16T20:15:35.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8941b4fc4ccf1b5-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:15:35.466 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.466 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.466 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.466 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.466 DEBUG:boto:Method: GET 2024-04-16T20:15:35.466 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.466 DEBUG:boto:Data: 2024-04-16T20:15:35.467 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.467 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.467 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.467 DEBUG:boto:Params: {} 2024-04-16T20:15:35.467 DEBUG:boto:Token: None 2024-04-16T20:15:35.467 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg= 2024-04-16T20:15:35.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg=', 'Content-Length': '0'} 2024-04-16T20:15:35.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c8dc9c22f3ebf5-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.510 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.510 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.511 DEBUG:boto:path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.511 DEBUG:boto:auth_path=/yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.511 DEBUG:boto:Method: GET 2024-04-16T20:15:35.511 DEBUG:boto:Path: /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.511 DEBUG:boto:Data: 2024-04-16T20:15:35.511 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.511 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.511 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.511 DEBUG:boto:Params: {} 2024-04-16T20:15:35.511 DEBUG:boto:Token: None 2024-04-16T20:15:35.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95?versionId=null 2024-04-16T20:15:35.512 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg= 2024-04-16T20:15:35.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CCUkt7ijeQevGYIKcP+V4UdmLUg=', 'Content-Length': '0'} 2024-04-16T20:15:35.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cccb00b1140c28b7-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.514 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.514 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.514 DEBUG:boto:path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.514 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.514 DEBUG:boto:Method: GET 2024-04-16T20:15:35.514 DEBUG:boto:Path: /yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.514 DEBUG:boto:Data: 2024-04-16T20:15:35.514 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.515 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.515 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.515 DEBUG:boto:Params: {} 2024-04-16T20:15:35.515 DEBUG:boto:Token: None 2024-04-16T20:15:35.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d973c61f1d28c4-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.558 DEBUG:boto:b'yrjefh-54obj951000falseobj952024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.559 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.559 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.559 DEBUG:boto:path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.559 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.559 DEBUG:boto:Method: GET 2024-04-16T20:15:35.559 DEBUG:boto:Path: /yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.559 DEBUG:boto:Data: 2024-04-16T20:15:35.559 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.559 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.559 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.559 DEBUG:boto:Params: {} 2024-04-16T20:15:35.559 DEBUG:boto:Token: None 2024-04-16T20:15:35.560 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.560 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002245e66fad20512e-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.563 DEBUG:boto:b'yrjefh-54obj951000falseobj952024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.564 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.564 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.564 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.564 DEBUG:boto:Path: /yrjefh-54/obj95 2024-04-16T20:15:35.564 DEBUG:boto:Data: 2024-04-16T20:15:35.564 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.564 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.564 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.564 DEBUG:boto:Params: {} 2024-04-16T20:15:35.565 DEBUG:boto:Token: None 2024-04-16T20:15:35.565 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95 2024-04-16T20:15:35.565 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dqbI310Ixt/isVEFdu8QOCalf3c= 2024-04-16T20:15:35.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dqbI310Ixt/isVEFdu8QOCalf3c=', 'Content-Length': '0'} 2024-04-16T20:15:35.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b253e4937a3a513e-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.567 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.567 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.567 DEBUG:boto:Method: HEAD 2024-04-16T20:15:35.567 DEBUG:boto:Path: /yrjefh-54/obj95 2024-04-16T20:15:35.567 DEBUG:boto:Data: 2024-04-16T20:15:35.568 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.568 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.568 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.568 DEBUG:boto:Params: {} 2024-04-16T20:15:35.568 DEBUG:boto:Token: None 2024-04-16T20:15:35.568 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95 2024-04-16T20:15:35.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dqbI310Ixt/isVEFdu8QOCalf3c= 2024-04-16T20:15:35.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dqbI310Ixt/isVEFdu8QOCalf3c=', 'Content-Length': '0'} 2024-04-16T20:15:35.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa5344d6f858b7b-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:15:35.571 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.571 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.571 DEBUG:boto:Method: GET 2024-04-16T20:15:35.571 DEBUG:boto:Path: /yrjefh-54/obj95 2024-04-16T20:15:35.571 DEBUG:boto:Data: 2024-04-16T20:15:35.571 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.571 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.571 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.571 DEBUG:boto:Params: {} 2024-04-16T20:15:35.572 DEBUG:boto:Token: None 2024-04-16T20:15:35.572 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95 2024-04-16T20:15:35.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/5CSRgKaM3SscHxICSniuYfvQ7I= 2024-04-16T20:15:35.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/5CSRgKaM3SscHxICSniuYfvQ7I=', 'Content-Length': '0'} 2024-04-16T20:15:35.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-54:b482a2a9-ef36-488b-9666-ada656453328.4812.23'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:13:56 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f52fdbdc5fdd50-00661edc67-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.614 DEBUG:boto:path=/yrjefh-54/obj95 2024-04-16T20:15:35.614 DEBUG:boto:auth_path=/yrjefh-54/obj95 2024-04-16T20:15:35.615 DEBUG:boto:Method: GET 2024-04-16T20:15:35.615 DEBUG:boto:Path: /yrjefh-54/obj95 2024-04-16T20:15:35.615 DEBUG:boto:Data: 2024-04-16T20:15:35.615 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.615 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.615 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.615 DEBUG:boto:Params: {} 2024-04-16T20:15:35.615 DEBUG:boto:Token: None 2024-04-16T20:15:35.615 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/obj95 2024-04-16T20:15:35.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/5CSRgKaM3SscHxICSniuYfvQ7I= 2024-04-16T20:15:35.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/5CSRgKaM3SscHxICSniuYfvQ7I=', 'Content-Length': '0'} 2024-04-16T20:15:35.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:12:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f974cb3edfee88-00661edc67-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.618 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.618 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.618 DEBUG:boto:path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.618 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.618 DEBUG:boto:Method: GET 2024-04-16T20:15:35.618 DEBUG:boto:Path: /yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.618 DEBUG:boto:Data: 2024-04-16T20:15:35.618 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.618 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:15:35.619 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.619 DEBUG:boto:Params: {} 2024-04-16T20:15:35.619 DEBUG:boto:Token: None 2024-04-16T20:15:35.619 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.619 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004463b8b54c595010-00661edc67-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.662 DEBUG:boto:b'yrjefh-54obj951000falseobj952024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.663 DEBUG:boto:path=/yrjefh-54/ 2024-04-16T20:15:35.663 DEBUG:boto:auth_path=/yrjefh-54/ 2024-04-16T20:15:35.663 DEBUG:boto:path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.663 DEBUG:boto:auth_path=/yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.663 DEBUG:boto:Method: GET 2024-04-16T20:15:35.663 DEBUG:boto:Path: /yrjefh-54/?prefix=obj95 2024-04-16T20:15:35.663 DEBUG:boto:Data: 2024-04-16T20:15:35.663 DEBUG:boto:Headers: {} 2024-04-16T20:15:35.663 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:35.663 DEBUG:boto:Port: 8000 2024-04-16T20:15:35.663 DEBUG:boto:Params: {} 2024-04-16T20:15:35.664 DEBUG:boto:Token: None 2024-04-16T20:15:35.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:15:35 GMT /yrjefh-54/ 2024-04-16T20:15:35.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0= 2024-04-16T20:15:35.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2SMB0YqPyf4xu1a5UFThIkcDvr0=', 'Content-Length': '0'} 2024-04-16T20:15:35.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dfe2b80bbbcfd4c-00661edc67-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:35.668 DEBUG:boto:b'yrjefh-54obj951000falseobj952024-04-16T20:12:50.819Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:15:35.668 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-54 zones={test-zone2, test-zone1} 2024-04-16T20:15:35.668 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T20:15:35.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.669 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:35.811 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000034719","last_update":"2024-04-16T20:13:05.502787Z"},{"marker":"00000000000000000000:00000000000000026366","last_update":"2024-04-16T20:13:05.504529Z"},{"marker":"00000000000000000000:00000000000000038766","last_update":"2024-04-16T20:12:52.106951Z"},{"marker":"00000000000000000000:00000000000000034817","last_update":"2024-04-16T20:13:05.506292Z"}] 2024-04-16T20:15:35.811 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000034719', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038766', 3: '00000000000000000000:00000000000000034817'} 2024-04-16T20:15:35.812 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:15:35.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.812 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:35.870 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:35.870 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:35.952 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7377314411986567608},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000034719","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.502787Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026366","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038766","next_step_marker":"","total_entries":45,"pos":0,"timestamp":"2024-04-16T20:12:52.106951Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000034817","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:13:05.506292Z"}}]},"full_sync":{"total":54,"complete":54},"current_time":"2024-04-16T20:15:35Z"} 2024-04-16T20:15:35.952 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7377314411986567608},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000034719","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.502787Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026366","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038766","next_step_marker":"","total_entries":45,"pos":0,"timestamp":"2024-04-16T20:12:52.106951Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000034817","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:13:05.506292Z"}}]},"full_sync":{"total":54,"complete":54},"current_time":"2024-04-16T20:15:35Z"} 2024-04-16T20:15:35.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034719', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.502787Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026366', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038766', 'next_step_marker': '', 'total_entries': 45, 'pos': 0, 'timestamp': '2024-04-16T20:12:52.106951Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034817', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.506292Z'}}] 2024-04-16T20:15:35.952 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034719', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038766', 3: '00000000000000000000:00000000000000034817'} 2024-04-16T20:15:35.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034719', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038766', 3: '00000000000000000000:00000000000000034817'} 2024-04-16T20:15:35.953 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:15:35.953 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T20:15:35.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:35.953 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:36.080 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000014063","last_update":"2024-04-16T20:14:42.399397Z"},{"marker":"00000000000000000000:00000000000000007969","last_update":"2024-04-16T20:14:25.875693Z"},{"marker":"00000000000000000000:00000000000000015434","last_update":"2024-04-16T20:13:58.395039Z"},{"marker":"00000000000000000000:00000000000000016712","last_update":"2024-04-16T20:14:42.401486Z"}] 2024-04-16T20:15:36.080 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000014063', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015434', 3: '00000000000000000000:00000000000000016712'} 2024-04-16T20:15:36.081 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:15:36.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.081 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:36.134 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:36.134 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:36.211 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014063","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:42.399397Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007969","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:25.875693Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015434","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:58.395039Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000016712","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:14:42.401486Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:15:36Z"} 2024-04-16T20:15:36.211 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014063","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:42.399397Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007969","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:25.875693Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015434","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:58.395039Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000016712","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:14:42.401486Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:15:36Z"} 2024-04-16T20:15:36.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:14:42.399397Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007969', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:14:25.875693Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:58.395039Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016712', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T20:14:42.401486Z'}}] 2024-04-16T20:15:36.212 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014063', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015434', 3: '00000000000000000000:00000000000000016712'} 2024-04-16T20:15:36.212 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014063', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015434', 3: '00000000000000000000:00000000000000016712'} 2024-04-16T20:15:36.212 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:15:36.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2024-04-16T20:15:36.214 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:15:36.215 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:15:36.215 DEBUG:boto:Method: PUT 2024-04-16T20:15:36.215 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:15:36.215 DEBUG:boto:Data: 2024-04-16T20:15:36.215 DEBUG:boto:Headers: {} 2024-04-16T20:15:36.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:36.215 DEBUG:boto:Port: 8000 2024-04-16T20:15:36.215 DEBUG:boto:Params: {} 2024-04-16T20:15:36.215 DEBUG:boto:Token: None 2024-04-16T20:15:36.216 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:15:36 GMT /yrjefh-55/ 2024-04-16T20:15:36.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qhpOOyjZrf4DB7OlUgmlj3Jbfns= 2024-04-16T20:15:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qhpOOyjZrf4DB7OlUgmlj3Jbfns=', 'Content-Length': '0'} 2024-04-16T20:15:36.231 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033b3fb320b91eac8-00661edc68-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:36.232 DEBUG:rgw_multi.tests:created bucket=yrjefh-55 2024-04-16T20:15:36.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.232 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:36.279 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:36.279 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:36.332 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:15:36.355 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713298536.239829_1112.1","last_update":"2024-04-16T20:15:36.239829Z"}] 2024-04-16T20:15:36.355 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298536.239829_1112.1'} 2024-04-16T20:15:36.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:15:36.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:15:36.356 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:15:36.404 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:36.404 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:36.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:36.464+0000 7ff9c7811800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:15:36.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:36.465+0000 7ff9c7811800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:15:36.477 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298536.239829_1112.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:15:36.239829Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:15:36Z"} 2024-04-16T20:15:36.478 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298536.239829_1112.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:15:36.239829Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:15:36Z"} 2024-04-16T20:15:36.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298536.239829_1112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:15:36.239829Z', 'realm_epoch': 2}}] 2024-04-16T20:15:36.478 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298536.239829_1112.1'} 2024-04-16T20:15:36.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298536.239829_1112.1'} 2024-04-16T20:15:36.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:15:36.478 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:15:36.479 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:15:36.479 DEBUG:boto:Method: HEAD 2024-04-16T20:15:36.479 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:15:36.479 DEBUG:boto:Data: 2024-04-16T20:15:36.479 DEBUG:boto:Headers: {} 2024-04-16T20:15:36.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:36.479 DEBUG:boto:Port: 8000 2024-04-16T20:15:36.479 DEBUG:boto:Params: {} 2024-04-16T20:15:36.479 DEBUG:boto:Token: None 2024-04-16T20:15:36.479 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:15:36 GMT /yrjefh-55/ 2024-04-16T20:15:36.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:N2w3TQSVjDSqo9YpXvGdJWSvx0w= 2024-04-16T20:15:36.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:15:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N2w3TQSVjDSqo9YpXvGdJWSvx0w=', 'Content-Length': '0'} 2024-04-16T20:15:36.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a999adcaaa7f2f2-00661edc68-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:36.483 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:15:36.483 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:15:36.483 DEBUG:boto:Method: PUT 2024-04-16T20:15:36.483 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:15:36.483 DEBUG:boto:Data: 2024-04-16T20:15:36.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:15:36.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:15:36.483 DEBUG:boto:Port: 8000 2024-04-16T20:15:36.483 DEBUG:boto:Params: {} 2024-04-16T20:15:36.484 DEBUG:boto:Token: None 2024-04-16T20:15:36.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:15:36 GMT /yrjefh-55/dummy 2024-04-16T20:15:36.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1urGW5WitLdE08bXRgtZPakHFuA= 2024-04-16T20:15:36.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:15:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1urGW5WitLdE08bXRgtZPakHFuA='} 2024-04-16T20:15:36.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b95f417d54186f1-00661edc68-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:15:36.490 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-55 2024-04-16T20:15:36.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:36.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:36.491 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:15:36.545 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:36.545 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:36.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:36.612+0000 7f3e57131800 1 waiting to reach incremental sync.. 2024-04-16T20:15:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:41.613+0000 7f3e57131800 1 bucket sync caught up with source: 2024-04-16T20:15:41.605 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:15:41.605 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:15:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:15:41.613+0000 7f3e57131800 0 bucket checkpoint complete 2024-04-16T20:15:41.631 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-55 2024-04-16T20:15:41.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:41.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:15:41.632 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-55 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:15:41.695 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:15:41.695 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:15:41.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:41.766+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:15:46.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:46.767+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:15:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:51.768+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:15:56.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:15:56.768+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:01.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:01.769+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:06.770+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:11.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:11.770+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:16.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:16.771+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:21.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:21.771+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:26.772+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:31.772+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:36.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:36.773+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:41.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:41.774+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:46.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:46.775+0000 7f835c215800 1 waiting to reach incremental sync.. 2024-04-16T20:16:51.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:51.777+0000 7f835c215800 1 bucket sync caught up with source: 2024-04-16T20:16:51.768 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.228.6, , , , , ] 2024-04-16T20:16:51.768 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.228.6, , , , , ] 2024-04-16T20:16:51.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:16:51.777+0000 7f835c215800 0 bucket checkpoint complete 2024-04-16T20:16:51.796 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-55 zones={test-zone2, test-zone1} 2024-04-16T20:16:51.796 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.796 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.796 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.797 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:51.797 DEBUG:boto:Data: 2024-04-16T20:16:51.797 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.797 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.797 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.797 DEBUG:boto:Params: {} 2024-04-16T20:16:51.798 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:16:51.798 DEBUG:boto:Token: None 2024-04-16T20:16:51.799 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxc9h5MVpv08FP1IV1cPG2kafxY= 2024-04-16T20:16:51.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxc9h5MVpv08FP1IV1cPG2kafxY=', 'Content-Length': '0'} 2024-04-16T20:16:51.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ea4604a58cc52e4-00661edcb3-6186-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.805 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.806 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.806 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.806 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:51.806 DEBUG:boto:Data: 2024-04-16T20:16:51.806 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.806 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.807 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.807 DEBUG:boto:Params: {} 2024-04-16T20:16:51.807 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:16:51.807 DEBUG:boto:Token: None 2024-04-16T20:16:51.807 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qxc9h5MVpv08FP1IV1cPG2kafxY= 2024-04-16T20:16:51.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qxc9h5MVpv08FP1IV1cPG2kafxY=', 'Content-Length': '0'} 2024-04-16T20:16:51.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4f44df772878a8e-00661edcb3-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:16:51.813 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.813 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.813 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.813 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.813 DEBUG:boto:Method: GET 2024-04-16T20:16:51.814 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.814 DEBUG:boto:Data: 2024-04-16T20:16:51.814 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.814 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.814 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.814 DEBUG:boto:Params: {} 2024-04-16T20:16:51.814 DEBUG:boto:Token: None 2024-04-16T20:16:51.815 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/?versions 2024-04-16T20:16:51.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM= 2024-04-16T20:16:51.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM=', 'Content-Length': '0'} 2024-04-16T20:16:51.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000607999039c74b0b6-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.818 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.819 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T20:16:51.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:16:51.819 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.819 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.820 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.820 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.820 DEBUG:boto:Method: GET 2024-04-16T20:16:51.820 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.820 DEBUG:boto:Data: 2024-04-16T20:16:51.820 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.820 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.820 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.821 DEBUG:boto:Params: {} 2024-04-16T20:16:51.821 DEBUG:boto:Token: None 2024-04-16T20:16:51.821 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/?versions 2024-04-16T20:16:51.821 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM= 2024-04-16T20:16:51.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM=', 'Content-Length': '0'} 2024-04-16T20:16:51.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e35a4912eba7462-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.824 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.824 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T20:16:51.825 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.825 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.825 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.825 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.825 DEBUG:boto:Method: GET 2024-04-16T20:16:51.825 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.825 DEBUG:boto:Data: 2024-04-16T20:16:51.826 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.826 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.826 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.826 DEBUG:boto:Params: {} 2024-04-16T20:16:51.826 DEBUG:boto:Token: None 2024-04-16T20:16:51.826 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/?versions 2024-04-16T20:16:51.827 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM= 2024-04-16T20:16:51.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM=', 'Content-Length': '0'} 2024-04-16T20:16:51.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a6d02448bcec10-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.830 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.830 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.830 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.831 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.831 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.831 DEBUG:boto:Method: GET 2024-04-16T20:16:51.831 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:16:51.831 DEBUG:boto:Data: 2024-04-16T20:16:51.831 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.832 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.832 DEBUG:boto:Params: {} 2024-04-16T20:16:51.832 DEBUG:boto:Token: None 2024-04-16T20:16:51.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/?versions 2024-04-16T20:16:51.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM= 2024-04-16T20:16:51.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C5i9LI1TgWgrku8s7R8CCY0gygM=', 'Content-Length': '0'} 2024-04-16T20:16:51.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010b48499340c477-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.835 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:16:51.836 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.836 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.836 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.836 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.836 DEBUG:boto:Method: GET 2024-04-16T20:16:51.837 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.837 DEBUG:boto:Data: 2024-04-16T20:16:51.837 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.837 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.837 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.837 DEBUG:boto:Params: {} 2024-04-16T20:16:51.837 DEBUG:boto:Token: None 2024-04-16T20:16:51.837 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.838 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY= 2024-04-16T20:16:51.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY=', 'Content-Length': '0'} 2024-04-16T20:16:51.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e1aca6f895a5fb-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.840 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.841 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.841 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.841 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.841 DEBUG:boto:Method: GET 2024-04-16T20:16:51.841 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.841 DEBUG:boto:Data: 2024-04-16T20:16:51.841 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.841 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.842 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.842 DEBUG:boto:Params: {} 2024-04-16T20:16:51.842 DEBUG:boto:Token: None 2024-04-16T20:16:51.842 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.842 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY= 2024-04-16T20:16:51.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY=', 'Content-Length': '0'} 2024-04-16T20:16:51.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a998c5bab35a528f-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.845 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.846 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.846 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.846 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.846 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.846 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.846 DEBUG:boto:Data: 2024-04-16T20:16:51.846 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.847 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.847 DEBUG:boto:Params: {} 2024-04-16T20:16:51.847 DEBUG:boto:Token: None 2024-04-16T20:16:51.847 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9NMGRbqDYb0dcxmdQY0yjU78nXw= 2024-04-16T20:16:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9NMGRbqDYb0dcxmdQY0yjU78nXw=', 'Content-Length': '0'} 2024-04-16T20:16:51.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd5ec9b77ea474a-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.850 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.850 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.850 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.850 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.850 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.850 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.851 DEBUG:boto:Data: 2024-04-16T20:16:51.851 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.851 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.851 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.851 DEBUG:boto:Params: {} 2024-04-16T20:16:51.851 DEBUG:boto:Token: None 2024-04-16T20:16:51.851 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9NMGRbqDYb0dcxmdQY0yjU78nXw= 2024-04-16T20:16:51.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9NMGRbqDYb0dcxmdQY0yjU78nXw=', 'Content-Length': '0'} 2024-04-16T20:16:51.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092baba84f76e5b7e-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:16:51.854 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.854 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.854 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.854 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.855 DEBUG:boto:Method: GET 2024-04-16T20:16:51.855 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.855 DEBUG:boto:Data: 2024-04-16T20:16:51.855 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.855 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.855 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.855 DEBUG:boto:Params: {} 2024-04-16T20:16:51.855 DEBUG:boto:Token: None 2024-04-16T20:16:51.856 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY= 2024-04-16T20:16:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY=', 'Content-Length': '0'} 2024-04-16T20:16:51.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf94e20830b6d3b9-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.858 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.858 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.858 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.859 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.859 DEBUG:boto:Method: GET 2024-04-16T20:16:51.859 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.859 DEBUG:boto:Data: 2024-04-16T20:16:51.859 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.859 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.859 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.859 DEBUG:boto:Params: {} 2024-04-16T20:16:51.859 DEBUG:boto:Token: None 2024-04-16T20:16:51.860 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:16:51.860 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY= 2024-04-16T20:16:51.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OPmcyW7thADYO9nG43SeSvbY3SY=', 'Content-Length': '0'} 2024-04-16T20:16:51.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce2cac54752c167-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.862 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.862 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.862 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.863 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.863 DEBUG:boto:Method: GET 2024-04-16T20:16:51.863 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.863 DEBUG:boto:Data: 2024-04-16T20:16:51.863 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.863 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.863 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.863 DEBUG:boto:Params: {} 2024-04-16T20:16:51.864 DEBUG:boto:Token: None 2024-04-16T20:16:51.864 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks= 2024-04-16T20:16:51.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks=', 'Content-Length': '0'} 2024-04-16T20:16:51.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c5522fce1c2122-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.867 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.867 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.867 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.867 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.868 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.868 DEBUG:boto:Method: GET 2024-04-16T20:16:51.868 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.868 DEBUG:boto:Data: 2024-04-16T20:16:51.868 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.868 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.868 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.868 DEBUG:boto:Params: {} 2024-04-16T20:16:51.868 DEBUG:boto:Token: None 2024-04-16T20:16:51.869 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.869 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks= 2024-04-16T20:16:51.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks=', 'Content-Length': '0'} 2024-04-16T20:16:51.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f051da995e3ac1-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.871 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.872 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.872 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.872 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.872 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:16:51.872 DEBUG:boto:Data: 2024-04-16T20:16:51.872 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.873 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.873 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.873 DEBUG:boto:Params: {} 2024-04-16T20:16:51.873 DEBUG:boto:Token: None 2024-04-16T20:16:51.873 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy 2024-04-16T20:16:51.873 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qZpKWoxjgIxmEB8gCzyjwxw15W0= 2024-04-16T20:16:51.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qZpKWoxjgIxmEB8gCzyjwxw15W0=', 'Content-Length': '0'} 2024-04-16T20:16:51.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003365d5527799995d-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.876 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.876 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.876 DEBUG:boto:Method: HEAD 2024-04-16T20:16:51.876 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:16:51.876 DEBUG:boto:Data: 2024-04-16T20:16:51.876 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.877 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.877 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.877 DEBUG:boto:Params: {} 2024-04-16T20:16:51.877 DEBUG:boto:Token: None 2024-04-16T20:16:51.877 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy 2024-04-16T20:16:51.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qZpKWoxjgIxmEB8gCzyjwxw15W0= 2024-04-16T20:16:51.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qZpKWoxjgIxmEB8gCzyjwxw15W0=', 'Content-Length': '0'} 2024-04-16T20:16:51.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009964ae0ddc2de2ee-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:16:51.879 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.879 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.880 DEBUG:boto:Method: GET 2024-04-16T20:16:51.880 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:16:51.880 DEBUG:boto:Data: 2024-04-16T20:16:51.880 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.880 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.880 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.880 DEBUG:boto:Params: {} 2024-04-16T20:16:51.880 DEBUG:boto:Token: None 2024-04-16T20:16:51.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy 2024-04-16T20:16:51.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1lRBzmPhQl46cyJmXKtn/7X3NE= 2024-04-16T20:16:51.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1lRBzmPhQl46cyJmXKtn/7X3NE=', 'Content-Length': '0'} 2024-04-16T20:16:51.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007393d31d0dc49c2d-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.926 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:16:51.927 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:16:51.927 DEBUG:boto:Method: GET 2024-04-16T20:16:51.927 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:16:51.927 DEBUG:boto:Data: 2024-04-16T20:16:51.927 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.927 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.927 DEBUG:boto:Params: {} 2024-04-16T20:16:51.927 DEBUG:boto:Token: None 2024-04-16T20:16:51.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/dummy 2024-04-16T20:16:51.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1lRBzmPhQl46cyJmXKtn/7X3NE= 2024-04-16T20:16:51.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1lRBzmPhQl46cyJmXKtn/7X3NE=', 'Content-Length': '0'} 2024-04-16T20:16:51.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715d9198268e9a76-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.930 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.930 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.930 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.930 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.930 DEBUG:boto:Method: GET 2024-04-16T20:16:51.930 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.930 DEBUG:boto:Data: 2024-04-16T20:16:51.931 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:51.931 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.931 DEBUG:boto:Params: {} 2024-04-16T20:16:51.931 DEBUG:boto:Token: None 2024-04-16T20:16:51.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks= 2024-04-16T20:16:51.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks=', 'Content-Length': '0'} 2024-04-16T20:16:51.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a952a1dc3bd748-00661edcb3-6186-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.974 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.975 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:51.975 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:51.975 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.975 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.975 DEBUG:boto:Method: GET 2024-04-16T20:16:51.975 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:16:51.975 DEBUG:boto:Data: 2024-04-16T20:16:51.975 DEBUG:boto:Headers: {} 2024-04-16T20:16:51.975 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:51.975 DEBUG:boto:Port: 8000 2024-04-16T20:16:51.976 DEBUG:boto:Params: {} 2024-04-16T20:16:51.976 DEBUG:boto:Token: None 2024-04-16T20:16:51.976 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:16:51 GMT /yrjefh-55/ 2024-04-16T20:16:51.976 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks= 2024-04-16T20:16:51.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vxXi7GCfsrSvnqIgMVB+W0VCqks=', 'Content-Length': '0'} 2024-04-16T20:16:51.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b1938ca1d5a156c-00661edcb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:51.978 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:16:51.979 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-55 zones={test-zone2, test-zone1} 2024-04-16T20:16:51.979 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:16:51.979 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:16:51.980 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:16:51.990+0000 7f9056b69640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 199680) UID: 0 2024-04-16T20:16:51.980 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:16:51.990+0000 7f905cc34880 -1 shutting down 2024-04-16T20:16:53.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:16:58.080 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:16:58.080 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:58.080 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:58.080 DEBUG:boto:Method: HEAD 2024-04-16T20:16:58.081 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:58.081 DEBUG:boto:Data: 2024-04-16T20:16:58.081 DEBUG:boto:Headers: {} 2024-04-16T20:16:58.081 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.081 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.081 DEBUG:boto:Params: {} 2024-04-16T20:16:58.081 DEBUG:boto:Token: None 2024-04-16T20:16:58.082 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/ 2024-04-16T20:16:58.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg= 2024-04-16T20:16:58.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg=', 'Content-Length': '0'} 2024-04-16T20:16:58.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052d3e73ae095639c-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.085 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:16:58.086 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:16:58.086 DEBUG:boto:Method: PUT 2024-04-16T20:16:58.086 DEBUG:boto:Path: /yrjefh-55/obj11 2024-04-16T20:16:58.086 DEBUG:boto:Data: 2024-04-16T20:16:58.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:16:58.086 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.086 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.087 DEBUG:boto:Params: {} 2024-04-16T20:16:58.087 DEBUG:boto:Token: None 2024-04-16T20:16:58.087 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/obj11 2024-04-16T20:16:58.087 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:66YaEhdVISxwSBB6V8BTeYR30V0= 2024-04-16T20:16:58.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:66YaEhdVISxwSBB6V8BTeYR30V0='} 2024-04-16T20:16:58.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a63bd099a90b9092-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.098 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:58.099 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:58.099 DEBUG:boto:Method: HEAD 2024-04-16T20:16:58.099 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:58.099 DEBUG:boto:Data: 2024-04-16T20:16:58.099 DEBUG:boto:Headers: {} 2024-04-16T20:16:58.099 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.099 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.099 DEBUG:boto:Params: {} 2024-04-16T20:16:58.100 DEBUG:boto:Token: None 2024-04-16T20:16:58.100 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/ 2024-04-16T20:16:58.100 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg= 2024-04-16T20:16:58.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg=', 'Content-Length': '0'} 2024-04-16T20:16:58.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dcf4564ae6d1e6c-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.103 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:16:58.103 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:16:58.103 DEBUG:boto:Method: PUT 2024-04-16T20:16:58.104 DEBUG:boto:Path: /yrjefh-55/obj22 2024-04-16T20:16:58.104 DEBUG:boto:Data: 2024-04-16T20:16:58.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:16:58.104 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.104 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.104 DEBUG:boto:Params: {} 2024-04-16T20:16:58.104 DEBUG:boto:Token: None 2024-04-16T20:16:58.105 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/obj22 2024-04-16T20:16:58.105 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3wFXuq20SvWCHvO+EQcoLxEa+jI= 2024-04-16T20:16:58.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3wFXuq20SvWCHvO+EQcoLxEa+jI='} 2024-04-16T20:16:58.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025a9d37a4a4f3931-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.110 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:58.111 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:58.111 DEBUG:boto:Method: HEAD 2024-04-16T20:16:58.111 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:58.111 DEBUG:boto:Data: 2024-04-16T20:16:58.111 DEBUG:boto:Headers: {} 2024-04-16T20:16:58.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.111 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.111 DEBUG:boto:Params: {} 2024-04-16T20:16:58.112 DEBUG:boto:Token: None 2024-04-16T20:16:58.112 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/ 2024-04-16T20:16:58.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg= 2024-04-16T20:16:58.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg=', 'Content-Length': '0'} 2024-04-16T20:16:58.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000421ee2876ad9da70-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.115 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:16:58.115 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:16:58.115 DEBUG:boto:Method: PUT 2024-04-16T20:16:58.115 DEBUG:boto:Path: /yrjefh-55/obj33 2024-04-16T20:16:58.116 DEBUG:boto:Data: 2024-04-16T20:16:58.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:16:58.116 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.116 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.116 DEBUG:boto:Params: {} 2024-04-16T20:16:58.116 DEBUG:boto:Token: None 2024-04-16T20:16:58.116 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/obj33 2024-04-16T20:16:58.117 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xZbPS7QNLiwOFda/mc1NCw4AHcA= 2024-04-16T20:16:58.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZbPS7QNLiwOFda/mc1NCw4AHcA='} 2024-04-16T20:16:58.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d809b57c6ab09993-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.159 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:58.159 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:58.159 DEBUG:boto:Method: HEAD 2024-04-16T20:16:58.159 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:58.159 DEBUG:boto:Data: 2024-04-16T20:16:58.160 DEBUG:boto:Headers: {} 2024-04-16T20:16:58.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.160 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.160 DEBUG:boto:Params: {} 2024-04-16T20:16:58.160 DEBUG:boto:Token: None 2024-04-16T20:16:58.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/ 2024-04-16T20:16:58.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg= 2024-04-16T20:16:58.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg=', 'Content-Length': '0'} 2024-04-16T20:16:58.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010b20c46efc0344c-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.164 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:16:58.164 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:16:58.164 DEBUG:boto:Method: PUT 2024-04-16T20:16:58.164 DEBUG:boto:Path: /yrjefh-55/obj44 2024-04-16T20:16:58.164 DEBUG:boto:Data: 2024-04-16T20:16:58.164 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:16:58.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.164 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.164 DEBUG:boto:Params: {} 2024-04-16T20:16:58.165 DEBUG:boto:Token: None 2024-04-16T20:16:58.165 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/obj44 2024-04-16T20:16:58.165 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9sgCJNeqcbENPAiUIJdxaGXrN4U= 2024-04-16T20:16:58.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9sgCJNeqcbENPAiUIJdxaGXrN4U='} 2024-04-16T20:16:58.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000862620144971d6ec-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.172 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:16:58.172 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:16:58.173 DEBUG:boto:Method: HEAD 2024-04-16T20:16:58.173 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:16:58.173 DEBUG:boto:Data: 2024-04-16T20:16:58.173 DEBUG:boto:Headers: {} 2024-04-16T20:16:58.173 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.173 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.173 DEBUG:boto:Params: {} 2024-04-16T20:16:58.173 DEBUG:boto:Token: None 2024-04-16T20:16:58.174 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/ 2024-04-16T20:16:58.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg= 2024-04-16T20:16:58.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:We9vppwfhy+meduUhusEEX6Nbkg=', 'Content-Length': '0'} 2024-04-16T20:16:58.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099fa0a38f58488f5-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.176 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:16:58.176 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:16:58.177 DEBUG:boto:Method: PUT 2024-04-16T20:16:58.177 DEBUG:boto:Path: /yrjefh-55/obj55 2024-04-16T20:16:58.177 DEBUG:boto:Data: 2024-04-16T20:16:58.177 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:16:58.177 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:16:58.177 DEBUG:boto:Port: 8000 2024-04-16T20:16:58.177 DEBUG:boto:Params: {} 2024-04-16T20:16:58.177 DEBUG:boto:Token: None 2024-04-16T20:16:58.178 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:16:58 GMT /yrjefh-55/obj55 2024-04-16T20:16:58.178 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3314XheYpusNWblEkA+sk6s+6E= 2024-04-16T20:16:58.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:16:58 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3314XheYpusNWblEkA+sk6s+6E='} 2024-04-16T20:16:58.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b61ffdc5041df1f2-00661edcba-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:16:58.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:16:58.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:16:58.223 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:16:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:16:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:16:58.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:16:58.346+0000 7f35664c0800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T20:16:58.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:16:58.347+0000 7f35664c0800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T20:16:58.369 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:16:58.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:16:58.372 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:16:58.372 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:16:58.392 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:16:58.392 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:16:58.393 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:16:58.393 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:16:58.567 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:16:58.577+0000 7f9a35322880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:16:59.394 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:16:59.415 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:16:59.415 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:16:59.415 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:16:59.416 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:16:59.416 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-55 2024-04-16T20:16:59.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:16:59.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:16:59.417 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:16:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:16:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:16:59.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:16:59.575+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:16:59.566 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:16:59.567 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:16:59.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:17:04.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:04.578+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:04.568 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:04.569 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:09.579+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:09.570 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:09.570 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:14.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:14.580+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:14.571 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:14.571 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:19.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:19.581+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:19.572 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:19.572 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:24.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:24.583+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:24.574 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:24.574 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:29.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:29.584+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:29.575 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:29.575 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:34.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:34.586+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:34.576 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:34.576 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:39.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:39.586+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:39.578 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:39.578 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:44.588+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:44.579 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:44.579 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:49.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:49.590+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:49.580 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:49.580 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:54.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:54.591+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:54.582 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , ] 2024-04-16T20:17:54.582 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:17:59.006 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T20:17:59.016+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T20:17:59.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:17:59.593+0000 7f74ba11f800 1 waiting for incremental sync to catch up: 2024-04-16T20:17:59.583 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , , , 00000000001.101.6] 2024-04-16T20:17:59.583 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:18:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:04.594+0000 7f74ba11f800 1 bucket sync caught up with source: 2024-04-16T20:18:04.584 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:18:04.584 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , 00000000001.131.6, , , 00000000001.83.6, 00000000002.85.6, 00000000002.103.6] 2024-04-16T20:18:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:04.594+0000 7f74ba11f800 0 bucket checkpoint complete 2024-04-16T20:18:04.612 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-55 2024-04-16T20:18:04.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:18:04.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:18:04.613 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-55 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:18:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:04.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:18:04.750+0000 7f5dc2b30800 1 waiting for incremental sync to catch up: 2024-04-16T20:18:04.740 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.228.6, , , 00000000001.136.6, 00000000002.138.6, 00000000001.105.6] 2024-04-16T20:18:04.740 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.228.6, , , 00000000001.136.6, 00000000002.138.6, 00000000002.107.6] 2024-04-16T20:18:09.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:18:09.751+0000 7f5dc2b30800 1 bucket sync caught up with source: 2024-04-16T20:18:09.742 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.228.6, , , 00000000001.136.6, 00000000002.138.6, 00000000002.107.6] 2024-04-16T20:18:09.742 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.228.6, , , 00000000001.136.6, 00000000002.138.6, 00000000002.107.6] 2024-04-16T20:18:09.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:18:09.751+0000 7f5dc2b30800 0 bucket checkpoint complete 2024-04-16T20:18:09.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-55 zones={test-zone2, test-zone1} 2024-04-16T20:18:09.770 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.770 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.770 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.770 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:18:09.770 DEBUG:boto:Data: 2024-04-16T20:18:09.770 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.770 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.771 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.771 DEBUG:boto:Params: {} 2024-04-16T20:18:09.771 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:18:09.771 DEBUG:boto:Token: None 2024-04-16T20:18:09.772 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.772 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PqSCaABF+uCAb+yNbmbf09/NVhI= 2024-04-16T20:18:09.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PqSCaABF+uCAb+yNbmbf09/NVhI=', 'Content-Length': '0'} 2024-04-16T20:18:09.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfaf35d611352550-00661edd01-6237-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.780 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.780 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.780 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.780 DEBUG:boto:Path: /yrjefh-55/ 2024-04-16T20:18:09.780 DEBUG:boto:Data: 2024-04-16T20:18:09.780 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.780 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.780 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.780 DEBUG:boto:Params: {} 2024-04-16T20:18:09.781 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:18:09.781 DEBUG:boto:Token: None 2024-04-16T20:18:09.781 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.781 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PqSCaABF+uCAb+yNbmbf09/NVhI= 2024-04-16T20:18:09.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PqSCaABF+uCAb+yNbmbf09/NVhI=', 'Content-Length': '0'} 2024-04-16T20:18:09.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a62726fa1045e9e2-00661edd01-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:18:09.785 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.785 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.785 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.785 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.786 DEBUG:boto:Method: GET 2024-04-16T20:18:09.786 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.786 DEBUG:boto:Data: 2024-04-16T20:18:09.786 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.786 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.786 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.786 DEBUG:boto:Params: {} 2024-04-16T20:18:09.786 DEBUG:boto:Token: None 2024-04-16T20:18:09.786 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/?versions 2024-04-16T20:18:09.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4= 2024-04-16T20:18:09.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4=', 'Content-Length': '0'} 2024-04-16T20:18:09.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fd62b8bac3357e7-00661edd01-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.789 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj11nulltrue2024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.790 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T20:18:09.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:18:09.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:18:09.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:18:09.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:18:09.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:18:09.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:18:09.791 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.791 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.791 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.791 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.791 DEBUG:boto:Method: GET 2024-04-16T20:18:09.792 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.792 DEBUG:boto:Data: 2024-04-16T20:18:09.792 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.792 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.792 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.792 DEBUG:boto:Params: {} 2024-04-16T20:18:09.792 DEBUG:boto:Token: None 2024-04-16T20:18:09.792 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/?versions 2024-04-16T20:18:09.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4= 2024-04-16T20:18:09.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4=', 'Content-Length': '0'} 2024-04-16T20:18:09.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a9129325613415-00661edd01-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.795 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj11nulltrue2024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.795 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-16T20:18:09.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:18:09.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:18:09.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:18:09.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:18:09.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:18:09.796 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.796 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.796 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.796 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.796 DEBUG:boto:Method: GET 2024-04-16T20:18:09.796 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.796 DEBUG:boto:Data: 2024-04-16T20:18:09.796 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.797 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.797 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.797 DEBUG:boto:Params: {} 2024-04-16T20:18:09.797 DEBUG:boto:Token: None 2024-04-16T20:18:09.797 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/?versions 2024-04-16T20:18:09.797 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4= 2024-04-16T20:18:09.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4=', 'Content-Length': '0'} 2024-04-16T20:18:09.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095e359e5aabb13c6-00661edd01-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.800 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj11nulltrue2024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.800 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.800 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.800 DEBUG:boto:path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.800 DEBUG:boto:auth_path=/yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.800 DEBUG:boto:Method: GET 2024-04-16T20:18:09.801 DEBUG:boto:Path: /yrjefh-55/?versions&max-keys=999 2024-04-16T20:18:09.801 DEBUG:boto:Data: 2024-04-16T20:18:09.801 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.801 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.801 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.801 DEBUG:boto:Params: {} 2024-04-16T20:18:09.801 DEBUG:boto:Token: None 2024-04-16T20:18:09.801 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/?versions 2024-04-16T20:18:09.801 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4= 2024-04-16T20:18:09.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NlpbbvmGTlzx84ozYr9e0WrHJZ4=', 'Content-Length': '0'} 2024-04-16T20:18:09.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeaac067ddd49d02-00661edd01-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.804 DEBUG:boto:b'yrjefh-55999falsedummynulltrue2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj11nulltrue2024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:18:09.804 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.805 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.805 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.805 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.805 DEBUG:boto:Method: GET 2024-04-16T20:18:09.805 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.805 DEBUG:boto:Data: 2024-04-16T20:18:09.805 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.805 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.805 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.805 DEBUG:boto:Params: {} 2024-04-16T20:18:09.805 DEBUG:boto:Token: None 2024-04-16T20:18:09.806 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.806 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc= 2024-04-16T20:18:09.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc=', 'Content-Length': '0'} 2024-04-16T20:18:09.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e3aa3f95f06920-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.808 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.808 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.809 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.809 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.809 DEBUG:boto:Method: GET 2024-04-16T20:18:09.809 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.809 DEBUG:boto:Data: 2024-04-16T20:18:09.809 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.809 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.809 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.809 DEBUG:boto:Params: {} 2024-04-16T20:18:09.809 DEBUG:boto:Token: None 2024-04-16T20:18:09.809 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.810 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc= 2024-04-16T20:18:09.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc=', 'Content-Length': '0'} 2024-04-16T20:18:09.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d61113fca19488-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.812 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.812 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.812 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.812 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.812 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.812 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.812 DEBUG:boto:Data: 2024-04-16T20:18:09.812 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.812 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.813 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.813 DEBUG:boto:Params: {} 2024-04-16T20:18:09.813 DEBUG:boto:Token: None 2024-04-16T20:18:09.813 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.813 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:axWGw3L9OHyTQYJ1hjDA5SFe+yk= 2024-04-16T20:18:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:axWGw3L9OHyTQYJ1hjDA5SFe+yk=', 'Content-Length': '0'} 2024-04-16T20:18:09.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087010df89076895a-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.815 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.815 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.815 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.815 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.815 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.815 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.816 DEBUG:boto:Data: 2024-04-16T20:18:09.816 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.816 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.816 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.816 DEBUG:boto:Params: {} 2024-04-16T20:18:09.816 DEBUG:boto:Token: None 2024-04-16T20:18:09.816 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:axWGw3L9OHyTQYJ1hjDA5SFe+yk= 2024-04-16T20:18:09.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:axWGw3L9OHyTQYJ1hjDA5SFe+yk=', 'Content-Length': '0'} 2024-04-16T20:18:09.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e93b9c06ec29ac0-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:18:09.818 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.819 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.819 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.819 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.819 DEBUG:boto:Method: GET 2024-04-16T20:18:09.819 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.819 DEBUG:boto:Data: 2024-04-16T20:18:09.819 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.819 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.819 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.819 DEBUG:boto:Params: {} 2024-04-16T20:18:09.819 DEBUG:boto:Token: None 2024-04-16T20:18:09.820 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.820 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc= 2024-04-16T20:18:09.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc=', 'Content-Length': '0'} 2024-04-16T20:18:09.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004286ac29a5084b54-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.862 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.862 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.862 DEBUG:boto:path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.862 DEBUG:boto:auth_path=/yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.863 DEBUG:boto:Method: GET 2024-04-16T20:18:09.863 DEBUG:boto:Path: /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.863 DEBUG:boto:Data: 2024-04-16T20:18:09.863 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.863 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.863 DEBUG:boto:Params: {} 2024-04-16T20:18:09.863 DEBUG:boto:Token: None 2024-04-16T20:18:09.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy?versionId=null 2024-04-16T20:18:09.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc= 2024-04-16T20:18:09.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:msxqDa842/X9a2Y4tAu3kjzlAFc=', 'Content-Length': '0'} 2024-04-16T20:18:09.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb7888935ef5a9f-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.865 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.865 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.866 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.866 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.866 DEBUG:boto:Method: GET 2024-04-16T20:18:09.866 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.866 DEBUG:boto:Data: 2024-04-16T20:18:09.866 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.866 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.866 DEBUG:boto:Params: {} 2024-04-16T20:18:09.866 DEBUG:boto:Token: None 2024-04-16T20:18:09.866 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc= 2024-04-16T20:18:09.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc=', 'Content-Length': '0'} 2024-04-16T20:18:09.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037e8cea24b099f1e-00661edd01-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.869 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.870 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.870 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.870 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.870 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.870 DEBUG:boto:Method: GET 2024-04-16T20:18:09.870 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.870 DEBUG:boto:Data: 2024-04-16T20:18:09.870 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.870 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.870 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.871 DEBUG:boto:Params: {} 2024-04-16T20:18:09.871 DEBUG:boto:Token: None 2024-04-16T20:18:09.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.871 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc= 2024-04-16T20:18:09.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc=', 'Content-Length': '0'} 2024-04-16T20:18:09.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000406bfbab6100891a-00661edd01-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.873 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.874 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.874 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.874 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.874 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:18:09.874 DEBUG:boto:Data: 2024-04-16T20:18:09.874 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.874 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.874 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.874 DEBUG:boto:Params: {} 2024-04-16T20:18:09.874 DEBUG:boto:Token: None 2024-04-16T20:18:09.875 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy 2024-04-16T20:18:09.875 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JgzeO8kicvidvbKd4F+gD0VPiyU= 2024-04-16T20:18:09.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JgzeO8kicvidvbKd4F+gD0VPiyU=', 'Content-Length': '0'} 2024-04-16T20:18:09.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cc52618f3a6dfd-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.877 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.877 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.877 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.877 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:18:09.877 DEBUG:boto:Data: 2024-04-16T20:18:09.877 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.877 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.878 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.878 DEBUG:boto:Params: {} 2024-04-16T20:18:09.878 DEBUG:boto:Token: None 2024-04-16T20:18:09.878 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy 2024-04-16T20:18:09.878 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JgzeO8kicvidvbKd4F+gD0VPiyU= 2024-04-16T20:18:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JgzeO8kicvidvbKd4F+gD0VPiyU=', 'Content-Length': '0'} 2024-04-16T20:18:09.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3d392c31f6caa1-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-16T20:18:09.880 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.881 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.881 DEBUG:boto:Method: GET 2024-04-16T20:18:09.881 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:18:09.881 DEBUG:boto:Data: 2024-04-16T20:18:09.881 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.881 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.881 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.881 DEBUG:boto:Params: {} 2024-04-16T20:18:09.881 DEBUG:boto:Token: None 2024-04-16T20:18:09.881 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy 2024-04-16T20:18:09.881 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QuCM7C9eI2eM1qpGqn0PgxyRboo= 2024-04-16T20:18:09.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QuCM7C9eI2eM1qpGqn0PgxyRboo=', 'Content-Length': '0'} 2024-04-16T20:18:09.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:15:41 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b2f5107b23a3b1-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.926 DEBUG:boto:path=/yrjefh-55/dummy 2024-04-16T20:18:09.926 DEBUG:boto:auth_path=/yrjefh-55/dummy 2024-04-16T20:18:09.927 DEBUG:boto:Method: GET 2024-04-16T20:18:09.927 DEBUG:boto:Path: /yrjefh-55/dummy 2024-04-16T20:18:09.927 DEBUG:boto:Data: 2024-04-16T20:18:09.927 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.927 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.927 DEBUG:boto:Params: {} 2024-04-16T20:18:09.927 DEBUG:boto:Token: None 2024-04-16T20:18:09.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/dummy 2024-04-16T20:18:09.927 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QuCM7C9eI2eM1qpGqn0PgxyRboo= 2024-04-16T20:18:09.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QuCM7C9eI2eM1qpGqn0PgxyRboo=', 'Content-Length': '0'} 2024-04-16T20:18:09.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:15:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd3d8affd1f7490-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.929 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.930 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.930 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.930 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.930 DEBUG:boto:Method: GET 2024-04-16T20:18:09.930 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.930 DEBUG:boto:Data: 2024-04-16T20:18:09.930 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.930 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.930 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.930 DEBUG:boto:Params: {} 2024-04-16T20:18:09.930 DEBUG:boto:Token: None 2024-04-16T20:18:09.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc= 2024-04-16T20:18:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc=', 'Content-Length': '0'} 2024-04-16T20:18:09.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a7faa579d3d85f-00661edd01-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.933 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.934 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:09.934 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:09.934 DEBUG:boto:path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.934 DEBUG:boto:auth_path=/yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.934 DEBUG:boto:Method: GET 2024-04-16T20:18:09.934 DEBUG:boto:Path: /yrjefh-55/?prefix=dummy 2024-04-16T20:18:09.934 DEBUG:boto:Data: 2024-04-16T20:18:09.934 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.934 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.934 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.935 DEBUG:boto:Params: {} 2024-04-16T20:18:09.935 DEBUG:boto:Token: None 2024-04-16T20:18:09.935 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/ 2024-04-16T20:18:09.935 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc= 2024-04-16T20:18:09.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9jVsEwVD9F2PY8QlwXd2P8KWyDc=', 'Content-Length': '0'} 2024-04-16T20:18:09.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e83090cff94a98-00661edd01-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.938 DEBUG:boto:b'yrjefh-55dummy1000falsedummy2024-04-16T20:15:36.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:09.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:18:09.938 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:09.938 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:09.938 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.938 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.938 DEBUG:boto:Method: GET 2024-04-16T20:18:09.938 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.938 DEBUG:boto:Data: 2024-04-16T20:18:09.939 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.939 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.939 DEBUG:boto:Params: {} 2024-04-16T20:18:09.939 DEBUG:boto:Token: None 2024-04-16T20:18:09.939 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.939 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc= 2024-04-16T20:18:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc=', 'Content-Length': '0'} 2024-04-16T20:18:09.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:17:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054dfcef98f9cde46-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.982 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:09.982 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:09.983 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.983 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.983 DEBUG:boto:Method: GET 2024-04-16T20:18:09.983 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.983 DEBUG:boto:Data: 2024-04-16T20:18:09.983 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.983 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.983 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.983 DEBUG:boto:Params: {} 2024-04-16T20:18:09.983 DEBUG:boto:Token: None 2024-04-16T20:18:09.984 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.984 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc= 2024-04-16T20:18:09.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc=', 'Content-Length': '0'} 2024-04-16T20:18:09.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0cda1b4353392c1-00661edd01-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.986 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:09.986 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:09.986 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.986 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.987 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.987 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.987 DEBUG:boto:Data: 2024-04-16T20:18:09.987 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.987 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.987 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.987 DEBUG:boto:Params: {} 2024-04-16T20:18:09.987 DEBUG:boto:Token: None 2024-04-16T20:18:09.987 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.987 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wevJEswqwr8TEarNO5B1bDmWEMA= 2024-04-16T20:18:09.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wevJEswqwr8TEarNO5B1bDmWEMA=', 'Content-Length': '0'} 2024-04-16T20:18:09.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:17:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6e0adc9d615a64-00661edd01-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.989 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:09.990 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:09.990 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.990 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.990 DEBUG:boto:Method: HEAD 2024-04-16T20:18:09.990 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.990 DEBUG:boto:Data: 2024-04-16T20:18:09.990 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.990 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:09.990 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.990 DEBUG:boto:Params: {} 2024-04-16T20:18:09.990 DEBUG:boto:Token: None 2024-04-16T20:18:09.991 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.991 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wevJEswqwr8TEarNO5B1bDmWEMA= 2024-04-16T20:18:09.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wevJEswqwr8TEarNO5B1bDmWEMA=', 'Content-Length': '0'} 2024-04-16T20:18:09.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc86f2520d37550-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:09.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:18:09.993 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:09.993 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:09.993 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.993 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.993 DEBUG:boto:Method: GET 2024-04-16T20:18:09.994 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.994 DEBUG:boto:Data: 2024-04-16T20:18:09.994 DEBUG:boto:Headers: {} 2024-04-16T20:18:09.994 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:09.994 DEBUG:boto:Port: 8000 2024-04-16T20:18:09.994 DEBUG:boto:Params: {} 2024-04-16T20:18:09.994 DEBUG:boto:Token: None 2024-04-16T20:18:09.994 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:09 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:09.994 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc= 2024-04-16T20:18:09.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:09 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AlP64KiAnkrd/GJ10pGeQ9fdhSc=', 'Content-Length': '0'} 2024-04-16T20:18:09.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:17:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a89585fd4297d3-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.038 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:10.039 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:10.039 DEBUG:boto:path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:10.039 DEBUG:boto:auth_path=/yrjefh-55/obj11?versionId=null 2024-04-16T20:18:10.039 DEBUG:boto:Method: GET 2024-04-16T20:18:10.039 DEBUG:boto:Path: /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:10.039 DEBUG:boto:Data: 2024-04-16T20:18:10.039 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.039 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.039 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.039 DEBUG:boto:Params: {} 2024-04-16T20:18:10.039 DEBUG:boto:Token: None 2024-04-16T20:18:10.040 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj11?versionId=null 2024-04-16T20:18:10.040 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TxMLzy+yq/rWCBzPvyEZG1VpWfM= 2024-04-16T20:18:10.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TxMLzy+yq/rWCBzPvyEZG1VpWfM=', 'Content-Length': '0'} 2024-04-16T20:18:10.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e6c0dd7f6665cc-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.042 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.042 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.042 DEBUG:boto:path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.042 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.043 DEBUG:boto:Method: GET 2024-04-16T20:18:10.043 DEBUG:boto:Path: /yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.043 DEBUG:boto:Data: 2024-04-16T20:18:10.043 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.043 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.043 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.043 DEBUG:boto:Params: {} 2024-04-16T20:18:10.043 DEBUG:boto:Token: None 2024-04-16T20:18:10.043 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.043 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665740f93b79e431-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.046 DEBUG:boto:b'yrjefh-55obj111000falseobj112024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.046 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.046 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.047 DEBUG:boto:path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.047 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.047 DEBUG:boto:Method: GET 2024-04-16T20:18:10.047 DEBUG:boto:Path: /yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.047 DEBUG:boto:Data: 2024-04-16T20:18:10.047 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.047 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.047 DEBUG:boto:Params: {} 2024-04-16T20:18:10.047 DEBUG:boto:Token: None 2024-04-16T20:18:10.047 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edc6129316c65d85-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.050 DEBUG:boto:b'yrjefh-55obj111000falseobj112024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.050 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:10.050 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:10.050 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.051 DEBUG:boto:Path: /yrjefh-55/obj11 2024-04-16T20:18:10.051 DEBUG:boto:Data: 2024-04-16T20:18:10.051 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.051 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.051 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.051 DEBUG:boto:Params: {} 2024-04-16T20:18:10.051 DEBUG:boto:Token: None 2024-04-16T20:18:10.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj11 2024-04-16T20:18:10.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AA/a8CzsqKYjnpTzLHyTqENyvQM= 2024-04-16T20:18:10.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AA/a8CzsqKYjnpTzLHyTqENyvQM=', 'Content-Length': '0'} 2024-04-16T20:18:10.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:17:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008203a05cf01e0123-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.054 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:10.054 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:10.054 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.054 DEBUG:boto:Path: /yrjefh-55/obj11 2024-04-16T20:18:10.054 DEBUG:boto:Data: 2024-04-16T20:18:10.054 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.054 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.054 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.055 DEBUG:boto:Params: {} 2024-04-16T20:18:10.055 DEBUG:boto:Token: None 2024-04-16T20:18:10.055 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj11 2024-04-16T20:18:10.055 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AA/a8CzsqKYjnpTzLHyTqENyvQM= 2024-04-16T20:18:10.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AA/a8CzsqKYjnpTzLHyTqENyvQM=', 'Content-Length': '0'} 2024-04-16T20:18:10.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e277105c63cd8905-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:18:10.057 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:10.057 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:10.057 DEBUG:boto:Method: GET 2024-04-16T20:18:10.057 DEBUG:boto:Path: /yrjefh-55/obj11 2024-04-16T20:18:10.057 DEBUG:boto:Data: 2024-04-16T20:18:10.057 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.057 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.057 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.057 DEBUG:boto:Params: {} 2024-04-16T20:18:10.058 DEBUG:boto:Token: None 2024-04-16T20:18:10.058 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj11 2024-04-16T20:18:10.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:16Jq/OYtbre6GKC5mZQ8HHTLqdY= 2024-04-16T20:18:10.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:16Jq/OYtbre6GKC5mZQ8HHTLqdY=', 'Content-Length': '0'} 2024-04-16T20:18:10.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:17:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d0209f62db66ae-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.102 DEBUG:boto:path=/yrjefh-55/obj11 2024-04-16T20:18:10.102 DEBUG:boto:auth_path=/yrjefh-55/obj11 2024-04-16T20:18:10.103 DEBUG:boto:Method: GET 2024-04-16T20:18:10.103 DEBUG:boto:Path: /yrjefh-55/obj11 2024-04-16T20:18:10.103 DEBUG:boto:Data: 2024-04-16T20:18:10.103 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.103 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.103 DEBUG:boto:Params: {} 2024-04-16T20:18:10.103 DEBUG:boto:Token: None 2024-04-16T20:18:10.103 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj11 2024-04-16T20:18:10.103 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:16Jq/OYtbre6GKC5mZQ8HHTLqdY= 2024-04-16T20:18:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:16Jq/OYtbre6GKC5mZQ8HHTLqdY=', 'Content-Length': '0'} 2024-04-16T20:18:10.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf41525fb60bc11e-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.106 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.106 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.106 DEBUG:boto:path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.106 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.106 DEBUG:boto:Method: GET 2024-04-16T20:18:10.106 DEBUG:boto:Path: /yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.106 DEBUG:boto:Data: 2024-04-16T20:18:10.106 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.106 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.106 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.107 DEBUG:boto:Params: {} 2024-04-16T20:18:10.107 DEBUG:boto:Token: None 2024-04-16T20:18:10.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.107 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f15a0664d6c67d-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.109 DEBUG:boto:b'yrjefh-55obj111000falseobj112024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.110 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.110 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.110 DEBUG:boto:path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.110 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.110 DEBUG:boto:Method: GET 2024-04-16T20:18:10.110 DEBUG:boto:Path: /yrjefh-55/?prefix=obj11 2024-04-16T20:18:10.110 DEBUG:boto:Data: 2024-04-16T20:18:10.110 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.110 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.111 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.111 DEBUG:boto:Params: {} 2024-04-16T20:18:10.111 DEBUG:boto:Token: None 2024-04-16T20:18:10.111 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.111 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fdceb60024f3033-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.113 DEBUG:boto:b'yrjefh-55obj111000falseobj112024-04-16T20:16:58.101Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:18:10.114 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.114 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.114 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.114 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.114 DEBUG:boto:Method: GET 2024-04-16T20:18:10.114 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.114 DEBUG:boto:Data: 2024-04-16T20:18:10.114 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.115 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.115 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.115 DEBUG:boto:Params: {} 2024-04-16T20:18:10.115 DEBUG:boto:Token: None 2024-04-16T20:18:10.115 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU= 2024-04-16T20:18:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU=', 'Content-Length': '0'} 2024-04-16T20:18:10.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b668056cf4db20cc-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.158 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.158 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.158 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.159 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.159 DEBUG:boto:Method: GET 2024-04-16T20:18:10.159 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.159 DEBUG:boto:Data: 2024-04-16T20:18:10.159 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.159 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.159 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.159 DEBUG:boto:Params: {} 2024-04-16T20:18:10.159 DEBUG:boto:Token: None 2024-04-16T20:18:10.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU= 2024-04-16T20:18:10.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU=', 'Content-Length': '0'} 2024-04-16T20:18:10.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235d53917abe4421-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.162 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.162 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.162 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.162 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.162 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.163 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.163 DEBUG:boto:Data: 2024-04-16T20:18:10.163 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.163 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.163 DEBUG:boto:Params: {} 2024-04-16T20:18:10.163 DEBUG:boto:Token: None 2024-04-16T20:18:10.163 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.163 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NujHYwtxWkhOVEnzOQzrNZTJNwk= 2024-04-16T20:18:10.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NujHYwtxWkhOVEnzOQzrNZTJNwk=', 'Content-Length': '0'} 2024-04-16T20:18:10.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb2effc5c88fa2b-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.165 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.166 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.166 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.166 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.166 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.166 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.166 DEBUG:boto:Data: 2024-04-16T20:18:10.166 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.166 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.166 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.166 DEBUG:boto:Params: {} 2024-04-16T20:18:10.166 DEBUG:boto:Token: None 2024-04-16T20:18:10.167 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.167 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NujHYwtxWkhOVEnzOQzrNZTJNwk= 2024-04-16T20:18:10.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NujHYwtxWkhOVEnzOQzrNZTJNwk=', 'Content-Length': '0'} 2024-04-16T20:18:10.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fc08d6bf60e94c-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:18:10.169 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.169 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.169 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.169 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.169 DEBUG:boto:Method: GET 2024-04-16T20:18:10.169 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.169 DEBUG:boto:Data: 2024-04-16T20:18:10.169 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.169 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.169 DEBUG:boto:Params: {} 2024-04-16T20:18:10.170 DEBUG:boto:Token: None 2024-04-16T20:18:10.170 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.170 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU= 2024-04-16T20:18:10.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU=', 'Content-Length': '0'} 2024-04-16T20:18:10.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10143430fcc213d-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.214 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.214 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.214 DEBUG:boto:path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.215 DEBUG:boto:auth_path=/yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.215 DEBUG:boto:Method: GET 2024-04-16T20:18:10.215 DEBUG:boto:Path: /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.215 DEBUG:boto:Data: 2024-04-16T20:18:10.215 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.215 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.215 DEBUG:boto:Params: {} 2024-04-16T20:18:10.215 DEBUG:boto:Token: None 2024-04-16T20:18:10.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22?versionId=null 2024-04-16T20:18:10.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU= 2024-04-16T20:18:10.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ajczc2Tu8zr9RbK1/hnAscX2PnU=', 'Content-Length': '0'} 2024-04-16T20:18:10.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40e333b9055abb3-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.218 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.218 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.218 DEBUG:boto:path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.218 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.218 DEBUG:boto:Method: GET 2024-04-16T20:18:10.218 DEBUG:boto:Path: /yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.218 DEBUG:boto:Data: 2024-04-16T20:18:10.218 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.218 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.218 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.218 DEBUG:boto:Params: {} 2024-04-16T20:18:10.218 DEBUG:boto:Token: None 2024-04-16T20:18:10.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eae171d94277a34-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.221 DEBUG:boto:b'yrjefh-55obj221000falseobj222024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.222 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.222 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.222 DEBUG:boto:path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.222 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.222 DEBUG:boto:Method: GET 2024-04-16T20:18:10.222 DEBUG:boto:Path: /yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.222 DEBUG:boto:Data: 2024-04-16T20:18:10.222 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.222 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.222 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.222 DEBUG:boto:Params: {} 2024-04-16T20:18:10.223 DEBUG:boto:Token: None 2024-04-16T20:18:10.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858baa219270a6b5-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.225 DEBUG:boto:b'yrjefh-55obj221000falseobj222024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.226 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.226 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.226 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.226 DEBUG:boto:Path: /yrjefh-55/obj22 2024-04-16T20:18:10.226 DEBUG:boto:Data: 2024-04-16T20:18:10.226 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.226 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.226 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.226 DEBUG:boto:Params: {} 2024-04-16T20:18:10.226 DEBUG:boto:Token: None 2024-04-16T20:18:10.227 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22 2024-04-16T20:18:10.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:caX2tWGf3r4cTZe0deqYtnIKG8s= 2024-04-16T20:18:10.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:caX2tWGf3r4cTZe0deqYtnIKG8s=', 'Content-Length': '0'} 2024-04-16T20:18:10.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e690a4f2aa52154-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.229 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.229 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.229 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.229 DEBUG:boto:Path: /yrjefh-55/obj22 2024-04-16T20:18:10.229 DEBUG:boto:Data: 2024-04-16T20:18:10.229 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.230 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.230 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.230 DEBUG:boto:Params: {} 2024-04-16T20:18:10.230 DEBUG:boto:Token: None 2024-04-16T20:18:10.230 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22 2024-04-16T20:18:10.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:caX2tWGf3r4cTZe0deqYtnIKG8s= 2024-04-16T20:18:10.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:caX2tWGf3r4cTZe0deqYtnIKG8s=', 'Content-Length': '0'} 2024-04-16T20:18:10.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e9cb7780b301362-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:18:10.233 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.233 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.233 DEBUG:boto:Method: GET 2024-04-16T20:18:10.233 DEBUG:boto:Path: /yrjefh-55/obj22 2024-04-16T20:18:10.233 DEBUG:boto:Data: 2024-04-16T20:18:10.233 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.233 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.233 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.233 DEBUG:boto:Params: {} 2024-04-16T20:18:10.233 DEBUG:boto:Token: None 2024-04-16T20:18:10.234 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22 2024-04-16T20:18:10.234 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s/i31v7EOtF/FCwGnzGHp7XNsmo= 2024-04-16T20:18:10.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s/i31v7EOtF/FCwGnzGHp7XNsmo=', 'Content-Length': '0'} 2024-04-16T20:18:10.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead115815e753eee-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.278 DEBUG:boto:path=/yrjefh-55/obj22 2024-04-16T20:18:10.279 DEBUG:boto:auth_path=/yrjefh-55/obj22 2024-04-16T20:18:10.279 DEBUG:boto:Method: GET 2024-04-16T20:18:10.279 DEBUG:boto:Path: /yrjefh-55/obj22 2024-04-16T20:18:10.279 DEBUG:boto:Data: 2024-04-16T20:18:10.279 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.279 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.279 DEBUG:boto:Params: {} 2024-04-16T20:18:10.279 DEBUG:boto:Token: None 2024-04-16T20:18:10.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj22 2024-04-16T20:18:10.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s/i31v7EOtF/FCwGnzGHp7XNsmo= 2024-04-16T20:18:10.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s/i31v7EOtF/FCwGnzGHp7XNsmo=', 'Content-Length': '0'} 2024-04-16T20:18:10.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1f74e585ec9aa7-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.282 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.282 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.282 DEBUG:boto:path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.282 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.282 DEBUG:boto:Method: GET 2024-04-16T20:18:10.282 DEBUG:boto:Path: /yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.282 DEBUG:boto:Data: 2024-04-16T20:18:10.282 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.282 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.282 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.283 DEBUG:boto:Params: {} 2024-04-16T20:18:10.283 DEBUG:boto:Token: None 2024-04-16T20:18:10.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c63a4c69236979-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.286 DEBUG:boto:b'yrjefh-55obj221000falseobj222024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.286 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.286 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.286 DEBUG:boto:path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.286 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.286 DEBUG:boto:Method: GET 2024-04-16T20:18:10.286 DEBUG:boto:Path: /yrjefh-55/?prefix=obj22 2024-04-16T20:18:10.286 DEBUG:boto:Data: 2024-04-16T20:18:10.287 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.287 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.287 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.287 DEBUG:boto:Params: {} 2024-04-16T20:18:10.287 DEBUG:boto:Token: None 2024-04-16T20:18:10.287 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.287 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f7e11d67da88c7-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.290 DEBUG:boto:b'yrjefh-55obj221000falseobj222024-04-16T20:16:58.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:18:10.290 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.290 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.290 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.290 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.291 DEBUG:boto:Method: GET 2024-04-16T20:18:10.291 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.291 DEBUG:boto:Data: 2024-04-16T20:18:10.291 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.291 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.291 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.291 DEBUG:boto:Params: {} 2024-04-16T20:18:10.291 DEBUG:boto:Token: None 2024-04-16T20:18:10.291 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.291 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA= 2024-04-16T20:18:10.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA=', 'Content-Length': '0'} 2024-04-16T20:18:10.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713db7bdd0df1352-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.334 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.334 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.335 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.335 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.335 DEBUG:boto:Method: GET 2024-04-16T20:18:10.335 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.335 DEBUG:boto:Data: 2024-04-16T20:18:10.335 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.335 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.335 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.335 DEBUG:boto:Params: {} 2024-04-16T20:18:10.335 DEBUG:boto:Token: None 2024-04-16T20:18:10.335 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.335 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA= 2024-04-16T20:18:10.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA=', 'Content-Length': '0'} 2024-04-16T20:18:10.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19ddd9dc00cff1d-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.338 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.338 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.338 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.338 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.338 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.338 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.339 DEBUG:boto:Data: 2024-04-16T20:18:10.339 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.339 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.339 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.339 DEBUG:boto:Params: {} 2024-04-16T20:18:10.339 DEBUG:boto:Token: None 2024-04-16T20:18:10.339 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.339 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+xZDAB+ReLk/uLPwgD8WfEAkm3A= 2024-04-16T20:18:10.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+xZDAB+ReLk/uLPwgD8WfEAkm3A=', 'Content-Length': '0'} 2024-04-16T20:18:10.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006929758d9d251783-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.341 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.341 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.341 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.341 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.342 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.342 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.342 DEBUG:boto:Data: 2024-04-16T20:18:10.342 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.342 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.342 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.342 DEBUG:boto:Params: {} 2024-04-16T20:18:10.342 DEBUG:boto:Token: None 2024-04-16T20:18:10.342 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.342 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+xZDAB+ReLk/uLPwgD8WfEAkm3A= 2024-04-16T20:18:10.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+xZDAB+ReLk/uLPwgD8WfEAkm3A=', 'Content-Length': '0'} 2024-04-16T20:18:10.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a069bb4379661c18-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:18:10.345 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.345 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.345 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.345 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.345 DEBUG:boto:Method: GET 2024-04-16T20:18:10.345 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.345 DEBUG:boto:Data: 2024-04-16T20:18:10.345 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.345 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.346 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.346 DEBUG:boto:Params: {} 2024-04-16T20:18:10.346 DEBUG:boto:Token: None 2024-04-16T20:18:10.346 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.346 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA= 2024-04-16T20:18:10.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA=', 'Content-Length': '0'} 2024-04-16T20:18:10.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ba021fe4dd0b2f-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.390 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.390 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.391 DEBUG:boto:path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.391 DEBUG:boto:auth_path=/yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.391 DEBUG:boto:Method: GET 2024-04-16T20:18:10.391 DEBUG:boto:Path: /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.391 DEBUG:boto:Data: 2024-04-16T20:18:10.391 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.391 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.391 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.391 DEBUG:boto:Params: {} 2024-04-16T20:18:10.391 DEBUG:boto:Token: None 2024-04-16T20:18:10.392 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33?versionId=null 2024-04-16T20:18:10.392 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA= 2024-04-16T20:18:10.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GZGjNXbEycshEN2DQscdhx50KWA=', 'Content-Length': '0'} 2024-04-16T20:18:10.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e4bf3403f7360f-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.394 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.394 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.394 DEBUG:boto:path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.395 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.395 DEBUG:boto:Method: GET 2024-04-16T20:18:10.395 DEBUG:boto:Path: /yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.395 DEBUG:boto:Data: 2024-04-16T20:18:10.395 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.395 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.395 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.395 DEBUG:boto:Params: {} 2024-04-16T20:18:10.395 DEBUG:boto:Token: None 2024-04-16T20:18:10.395 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e9be224fdf05dd-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.398 DEBUG:boto:b'yrjefh-55obj331000falseobj332024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.398 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.399 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.399 DEBUG:boto:path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.399 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.399 DEBUG:boto:Method: GET 2024-04-16T20:18:10.399 DEBUG:boto:Path: /yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.399 DEBUG:boto:Data: 2024-04-16T20:18:10.399 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.399 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.399 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.399 DEBUG:boto:Params: {} 2024-04-16T20:18:10.399 DEBUG:boto:Token: None 2024-04-16T20:18:10.400 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.400 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846ae25078b6f023-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.403 DEBUG:boto:b'yrjefh-55obj331000falseobj332024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.403 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.403 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.403 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.403 DEBUG:boto:Path: /yrjefh-55/obj33 2024-04-16T20:18:10.403 DEBUG:boto:Data: 2024-04-16T20:18:10.403 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.403 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.404 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.404 DEBUG:boto:Params: {} 2024-04-16T20:18:10.404 DEBUG:boto:Token: None 2024-04-16T20:18:10.404 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33 2024-04-16T20:18:10.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5qNtLieKokkq2XR1gmZnpFpUiM= 2024-04-16T20:18:10.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5qNtLieKokkq2XR1gmZnpFpUiM=', 'Content-Length': '0'} 2024-04-16T20:18:10.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040dd65fb0b3034af-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.406 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.406 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.407 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.407 DEBUG:boto:Path: /yrjefh-55/obj33 2024-04-16T20:18:10.407 DEBUG:boto:Data: 2024-04-16T20:18:10.407 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.407 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.407 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.407 DEBUG:boto:Params: {} 2024-04-16T20:18:10.407 DEBUG:boto:Token: None 2024-04-16T20:18:10.407 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33 2024-04-16T20:18:10.407 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I5qNtLieKokkq2XR1gmZnpFpUiM= 2024-04-16T20:18:10.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I5qNtLieKokkq2XR1gmZnpFpUiM=', 'Content-Length': '0'} 2024-04-16T20:18:10.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07e8ebbc7a33f2a-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:18:10.410 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.410 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.410 DEBUG:boto:Method: GET 2024-04-16T20:18:10.410 DEBUG:boto:Path: /yrjefh-55/obj33 2024-04-16T20:18:10.410 DEBUG:boto:Data: 2024-04-16T20:18:10.411 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.411 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.411 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.411 DEBUG:boto:Params: {} 2024-04-16T20:18:10.411 DEBUG:boto:Token: None 2024-04-16T20:18:10.411 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33 2024-04-16T20:18:10.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHjw3FOjDBE/Dok6HfjleTR/jo= 2024-04-16T20:18:10.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHjw3FOjDBE/Dok6HfjleTR/jo=', 'Content-Length': '0'} 2024-04-16T20:18:10.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000251816480c5df228-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.458 DEBUG:boto:path=/yrjefh-55/obj33 2024-04-16T20:18:10.458 DEBUG:boto:auth_path=/yrjefh-55/obj33 2024-04-16T20:18:10.459 DEBUG:boto:Method: GET 2024-04-16T20:18:10.459 DEBUG:boto:Path: /yrjefh-55/obj33 2024-04-16T20:18:10.459 DEBUG:boto:Data: 2024-04-16T20:18:10.459 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.459 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.459 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.459 DEBUG:boto:Params: {} 2024-04-16T20:18:10.459 DEBUG:boto:Token: None 2024-04-16T20:18:10.459 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj33 2024-04-16T20:18:10.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaHjw3FOjDBE/Dok6HfjleTR/jo= 2024-04-16T20:18:10.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHjw3FOjDBE/Dok6HfjleTR/jo=', 'Content-Length': '0'} 2024-04-16T20:18:10.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84334f32789b9ad-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.462 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.462 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.463 DEBUG:boto:path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.463 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.463 DEBUG:boto:Method: GET 2024-04-16T20:18:10.463 DEBUG:boto:Path: /yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.463 DEBUG:boto:Data: 2024-04-16T20:18:10.463 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.463 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.463 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.463 DEBUG:boto:Params: {} 2024-04-16T20:18:10.463 DEBUG:boto:Token: None 2024-04-16T20:18:10.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005741e6c0f222d6e3-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.466 DEBUG:boto:b'yrjefh-55obj331000falseobj332024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.466 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.467 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.467 DEBUG:boto:path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.467 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.467 DEBUG:boto:Method: GET 2024-04-16T20:18:10.467 DEBUG:boto:Path: /yrjefh-55/?prefix=obj33 2024-04-16T20:18:10.467 DEBUG:boto:Data: 2024-04-16T20:18:10.467 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.467 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.467 DEBUG:boto:Params: {} 2024-04-16T20:18:10.467 DEBUG:boto:Token: None 2024-04-16T20:18:10.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058aa1e2d2e9f6332-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.470 DEBUG:boto:b'yrjefh-55obj331000falseobj332024-04-16T20:16:58.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:18:10.471 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.471 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.471 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.471 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.471 DEBUG:boto:Method: GET 2024-04-16T20:18:10.471 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.471 DEBUG:boto:Data: 2024-04-16T20:18:10.471 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.471 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.472 DEBUG:boto:Params: {} 2024-04-16T20:18:10.472 DEBUG:boto:Token: None 2024-04-16T20:18:10.472 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk= 2024-04-16T20:18:10.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk=', 'Content-Length': '0'} 2024-04-16T20:18:10.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b142ea00fb18ea4d-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.514 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.514 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.514 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.515 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.515 DEBUG:boto:Method: GET 2024-04-16T20:18:10.515 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.515 DEBUG:boto:Data: 2024-04-16T20:18:10.515 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.515 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.515 DEBUG:boto:Params: {} 2024-04-16T20:18:10.515 DEBUG:boto:Token: None 2024-04-16T20:18:10.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.516 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk= 2024-04-16T20:18:10.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk=', 'Content-Length': '0'} 2024-04-16T20:18:10.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6d389d8f8d53a9-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.517 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.518 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.518 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.518 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.518 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.518 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.518 DEBUG:boto:Data: 2024-04-16T20:18:10.518 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.518 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.518 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.518 DEBUG:boto:Params: {} 2024-04-16T20:18:10.518 DEBUG:boto:Token: None 2024-04-16T20:18:10.519 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x409ku0IEj+B6rCmkFDKRtKOr58= 2024-04-16T20:18:10.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x409ku0IEj+B6rCmkFDKRtKOr58=', 'Content-Length': '0'} 2024-04-16T20:18:10.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692a5e519484ac2c-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.521 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.521 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.521 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.521 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.521 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.522 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.522 DEBUG:boto:Data: 2024-04-16T20:18:10.522 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.522 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.522 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.522 DEBUG:boto:Params: {} 2024-04-16T20:18:10.522 DEBUG:boto:Token: None 2024-04-16T20:18:10.522 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x409ku0IEj+B6rCmkFDKRtKOr58= 2024-04-16T20:18:10.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x409ku0IEj+B6rCmkFDKRtKOr58=', 'Content-Length': '0'} 2024-04-16T20:18:10.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69d4de1685643e6-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:18:10.525 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.525 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.525 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.525 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.525 DEBUG:boto:Method: GET 2024-04-16T20:18:10.525 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.525 DEBUG:boto:Data: 2024-04-16T20:18:10.525 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.525 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.525 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.525 DEBUG:boto:Params: {} 2024-04-16T20:18:10.526 DEBUG:boto:Token: None 2024-04-16T20:18:10.526 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.526 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk= 2024-04-16T20:18:10.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk=', 'Content-Length': '0'} 2024-04-16T20:18:10.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfd95fbf50d8118b-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.570 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.570 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.570 DEBUG:boto:path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.571 DEBUG:boto:auth_path=/yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.571 DEBUG:boto:Method: GET 2024-04-16T20:18:10.571 DEBUG:boto:Path: /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.571 DEBUG:boto:Data: 2024-04-16T20:18:10.571 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.571 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.571 DEBUG:boto:Params: {} 2024-04-16T20:18:10.571 DEBUG:boto:Token: None 2024-04-16T20:18:10.571 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44?versionId=null 2024-04-16T20:18:10.572 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk= 2024-04-16T20:18:10.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqvT5DEtwXGt9LLwdvnqJPNmPnk=', 'Content-Length': '0'} 2024-04-16T20:18:10.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b654a95609a7f0-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.574 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.574 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.574 DEBUG:boto:path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.574 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.574 DEBUG:boto:Method: GET 2024-04-16T20:18:10.574 DEBUG:boto:Path: /yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.575 DEBUG:boto:Data: 2024-04-16T20:18:10.575 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.575 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.575 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.575 DEBUG:boto:Params: {} 2024-04-16T20:18:10.575 DEBUG:boto:Token: None 2024-04-16T20:18:10.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.575 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098440714d34c32cc-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.578 DEBUG:boto:b'yrjefh-55obj441000falseobj442024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.578 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.578 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.578 DEBUG:boto:path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.578 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.579 DEBUG:boto:Method: GET 2024-04-16T20:18:10.579 DEBUG:boto:Path: /yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.579 DEBUG:boto:Data: 2024-04-16T20:18:10.579 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.579 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.579 DEBUG:boto:Params: {} 2024-04-16T20:18:10.579 DEBUG:boto:Token: None 2024-04-16T20:18:10.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7c9caf9e341f4b-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.582 DEBUG:boto:b'yrjefh-55obj441000falseobj442024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.582 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.582 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.582 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.582 DEBUG:boto:Path: /yrjefh-55/obj44 2024-04-16T20:18:10.583 DEBUG:boto:Data: 2024-04-16T20:18:10.583 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.583 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.583 DEBUG:boto:Params: {} 2024-04-16T20:18:10.583 DEBUG:boto:Token: None 2024-04-16T20:18:10.583 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44 2024-04-16T20:18:10.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q57VJo8Q6P91Uj0O44DAUVG9nsQ= 2024-04-16T20:18:10.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q57VJo8Q6P91Uj0O44DAUVG9nsQ=', 'Content-Length': '0'} 2024-04-16T20:18:10.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53853751500bc13-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.585 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.585 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.586 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.586 DEBUG:boto:Path: /yrjefh-55/obj44 2024-04-16T20:18:10.586 DEBUG:boto:Data: 2024-04-16T20:18:10.586 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.586 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.586 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.586 DEBUG:boto:Params: {} 2024-04-16T20:18:10.586 DEBUG:boto:Token: None 2024-04-16T20:18:10.586 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44 2024-04-16T20:18:10.586 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:q57VJo8Q6P91Uj0O44DAUVG9nsQ= 2024-04-16T20:18:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:q57VJo8Q6P91Uj0O44DAUVG9nsQ=', 'Content-Length': '0'} 2024-04-16T20:18:10.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e62c4e4534e385e-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:18:10.588 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.588 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.589 DEBUG:boto:Method: GET 2024-04-16T20:18:10.589 DEBUG:boto:Path: /yrjefh-55/obj44 2024-04-16T20:18:10.589 DEBUG:boto:Data: 2024-04-16T20:18:10.589 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.589 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.589 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.589 DEBUG:boto:Params: {} 2024-04-16T20:18:10.589 DEBUG:boto:Token: None 2024-04-16T20:18:10.589 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44 2024-04-16T20:18:10.589 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beZNWdA/8dwWy8fNV7tUegs9BbU= 2024-04-16T20:18:10.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beZNWdA/8dwWy8fNV7tUegs9BbU=', 'Content-Length': '0'} 2024-04-16T20:18:10.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bdcdec573d8ca00-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.634 DEBUG:boto:path=/yrjefh-55/obj44 2024-04-16T20:18:10.634 DEBUG:boto:auth_path=/yrjefh-55/obj44 2024-04-16T20:18:10.635 DEBUG:boto:Method: GET 2024-04-16T20:18:10.635 DEBUG:boto:Path: /yrjefh-55/obj44 2024-04-16T20:18:10.635 DEBUG:boto:Data: 2024-04-16T20:18:10.635 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.635 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.635 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.635 DEBUG:boto:Params: {} 2024-04-16T20:18:10.635 DEBUG:boto:Token: None 2024-04-16T20:18:10.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj44 2024-04-16T20:18:10.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:beZNWdA/8dwWy8fNV7tUegs9BbU= 2024-04-16T20:18:10.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beZNWdA/8dwWy8fNV7tUegs9BbU=', 'Content-Length': '0'} 2024-04-16T20:18:10.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058251e8f795fa4fd-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.638 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.638 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.638 DEBUG:boto:path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.638 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.638 DEBUG:boto:Method: GET 2024-04-16T20:18:10.638 DEBUG:boto:Path: /yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.638 DEBUG:boto:Data: 2024-04-16T20:18:10.638 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.638 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.638 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.638 DEBUG:boto:Params: {} 2024-04-16T20:18:10.639 DEBUG:boto:Token: None 2024-04-16T20:18:10.639 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.639 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5df39e82d2c32fc-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.641 DEBUG:boto:b'yrjefh-55obj441000falseobj442024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.642 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.642 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.642 DEBUG:boto:path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.642 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.642 DEBUG:boto:Method: GET 2024-04-16T20:18:10.642 DEBUG:boto:Path: /yrjefh-55/?prefix=obj44 2024-04-16T20:18:10.642 DEBUG:boto:Data: 2024-04-16T20:18:10.642 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.642 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.643 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.643 DEBUG:boto:Params: {} 2024-04-16T20:18:10.643 DEBUG:boto:Token: None 2024-04-16T20:18:10.643 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.643 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef670ea8f054657-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.646 DEBUG:boto:b'yrjefh-55obj441000falseobj442024-04-16T20:16:58.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:18:10.646 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.646 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.646 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.647 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.647 DEBUG:boto:Method: GET 2024-04-16T20:18:10.647 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.647 DEBUG:boto:Data: 2024-04-16T20:18:10.647 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.647 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.647 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.647 DEBUG:boto:Params: {} 2024-04-16T20:18:10.647 DEBUG:boto:Token: None 2024-04-16T20:18:10.647 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.647 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo= 2024-04-16T20:18:10.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo=', 'Content-Length': '0'} 2024-04-16T20:18:10.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009afa45425dcb773b-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.690 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.690 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.690 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.691 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.691 DEBUG:boto:Method: GET 2024-04-16T20:18:10.691 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.691 DEBUG:boto:Data: 2024-04-16T20:18:10.691 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.691 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.691 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.691 DEBUG:boto:Params: {} 2024-04-16T20:18:10.691 DEBUG:boto:Token: None 2024-04-16T20:18:10.691 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.691 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo= 2024-04-16T20:18:10.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo=', 'Content-Length': '0'} 2024-04-16T20:18:10.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034041b6a2337e34a-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.694 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.694 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.694 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.694 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.694 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.694 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.694 DEBUG:boto:Data: 2024-04-16T20:18:10.694 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.695 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.695 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.695 DEBUG:boto:Params: {} 2024-04-16T20:18:10.695 DEBUG:boto:Token: None 2024-04-16T20:18:10.695 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.695 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g0j2/s02x3gS20qID/NW6KJXda4= 2024-04-16T20:18:10.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g0j2/s02x3gS20qID/NW6KJXda4=', 'Content-Length': '0'} 2024-04-16T20:18:10.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d77611dbb6c410-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.697 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.697 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.697 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.697 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.697 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.697 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.698 DEBUG:boto:Data: 2024-04-16T20:18:10.698 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.698 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.698 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.698 DEBUG:boto:Params: {} 2024-04-16T20:18:10.698 DEBUG:boto:Token: None 2024-04-16T20:18:10.698 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.698 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:g0j2/s02x3gS20qID/NW6KJXda4= 2024-04-16T20:18:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g0j2/s02x3gS20qID/NW6KJXda4=', 'Content-Length': '0'} 2024-04-16T20:18:10.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047709a4745803d65-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:18:10.701 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.701 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.701 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.701 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.701 DEBUG:boto:Method: GET 2024-04-16T20:18:10.701 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.701 DEBUG:boto:Data: 2024-04-16T20:18:10.701 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.701 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.701 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.701 DEBUG:boto:Params: {} 2024-04-16T20:18:10.701 DEBUG:boto:Token: None 2024-04-16T20:18:10.702 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.702 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo= 2024-04-16T20:18:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo=', 'Content-Length': '0'} 2024-04-16T20:18:10.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e27cf0b167d512-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.746 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.746 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.746 DEBUG:boto:path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.746 DEBUG:boto:auth_path=/yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.747 DEBUG:boto:Method: GET 2024-04-16T20:18:10.747 DEBUG:boto:Path: /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.747 DEBUG:boto:Data: 2024-04-16T20:18:10.747 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.747 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.747 DEBUG:boto:Params: {} 2024-04-16T20:18:10.747 DEBUG:boto:Token: None 2024-04-16T20:18:10.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55?versionId=null 2024-04-16T20:18:10.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo= 2024-04-16T20:18:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YNXCUeqeqe8uOq/4Om7ngASFNNo=', 'Content-Length': '0'} 2024-04-16T20:18:10.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7b555f8c5a52dc-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.750 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.750 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.750 DEBUG:boto:path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.750 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.750 DEBUG:boto:Method: GET 2024-04-16T20:18:10.750 DEBUG:boto:Path: /yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.750 DEBUG:boto:Data: 2024-04-16T20:18:10.750 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.750 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.750 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.751 DEBUG:boto:Params: {} 2024-04-16T20:18:10.751 DEBUG:boto:Token: None 2024-04-16T20:18:10.751 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38d6612221d6c27-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.753 DEBUG:boto:b'yrjefh-55obj551000falseobj552024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.754 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.754 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.754 DEBUG:boto:path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.754 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.754 DEBUG:boto:Method: GET 2024-04-16T20:18:10.754 DEBUG:boto:Path: /yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.754 DEBUG:boto:Data: 2024-04-16T20:18:10.754 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.754 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.754 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.755 DEBUG:boto:Params: {} 2024-04-16T20:18:10.755 DEBUG:boto:Token: None 2024-04-16T20:18:10.755 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.755 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b71ec689b0d8dc-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.757 DEBUG:boto:b'yrjefh-55obj551000falseobj552024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.758 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.758 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.758 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.758 DEBUG:boto:Path: /yrjefh-55/obj55 2024-04-16T20:18:10.758 DEBUG:boto:Data: 2024-04-16T20:18:10.758 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.758 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.758 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.758 DEBUG:boto:Params: {} 2024-04-16T20:18:10.759 DEBUG:boto:Token: None 2024-04-16T20:18:10.759 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55 2024-04-16T20:18:10.759 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Frrek03ivoYeDy7zAtFttrZe+1o= 2024-04-16T20:18:10.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Frrek03ivoYeDy7zAtFttrZe+1o=', 'Content-Length': '0'} 2024-04-16T20:18:10.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f352b351a7ae55c7-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.761 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.761 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.761 DEBUG:boto:Method: HEAD 2024-04-16T20:18:10.761 DEBUG:boto:Path: /yrjefh-55/obj55 2024-04-16T20:18:10.761 DEBUG:boto:Data: 2024-04-16T20:18:10.761 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.761 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.762 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.762 DEBUG:boto:Params: {} 2024-04-16T20:18:10.762 DEBUG:boto:Token: None 2024-04-16T20:18:10.762 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55 2024-04-16T20:18:10.762 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Frrek03ivoYeDy7zAtFttrZe+1o= 2024-04-16T20:18:10.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Frrek03ivoYeDy7zAtFttrZe+1o=', 'Content-Length': '0'} 2024-04-16T20:18:10.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b851827d7d58bb07-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:18:10.764 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.764 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.764 DEBUG:boto:Method: GET 2024-04-16T20:18:10.764 DEBUG:boto:Path: /yrjefh-55/obj55 2024-04-16T20:18:10.764 DEBUG:boto:Data: 2024-04-16T20:18:10.764 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.765 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.765 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.765 DEBUG:boto:Params: {} 2024-04-16T20:18:10.765 DEBUG:boto:Token: None 2024-04-16T20:18:10.765 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55 2024-04-16T20:18:10.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bLU4JdCQh7voCQ1VvsLvFTE/SHI= 2024-04-16T20:18:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bLU4JdCQh7voCQ1VvsLvFTE/SHI=', 'Content-Length': '0'} 2024-04-16T20:18:10.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-55:b482a2a9-ef36-488b-9666-ada656453328.4812.24'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:02 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f983749264c10cf-00661edd02-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.810 DEBUG:boto:path=/yrjefh-55/obj55 2024-04-16T20:18:10.810 DEBUG:boto:auth_path=/yrjefh-55/obj55 2024-04-16T20:18:10.810 DEBUG:boto:Method: GET 2024-04-16T20:18:10.811 DEBUG:boto:Path: /yrjefh-55/obj55 2024-04-16T20:18:10.811 DEBUG:boto:Data: 2024-04-16T20:18:10.811 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.811 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.811 DEBUG:boto:Params: {} 2024-04-16T20:18:10.811 DEBUG:boto:Token: None 2024-04-16T20:18:10.811 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/obj55 2024-04-16T20:18:10.811 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:bLU4JdCQh7voCQ1VvsLvFTE/SHI= 2024-04-16T20:18:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bLU4JdCQh7voCQ1VvsLvFTE/SHI=', 'Content-Length': '0'} 2024-04-16T20:18:10.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:16:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bed94db45dacf40-00661edd02-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.813 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.814 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.814 DEBUG:boto:path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.814 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.814 DEBUG:boto:Method: GET 2024-04-16T20:18:10.814 DEBUG:boto:Path: /yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.814 DEBUG:boto:Data: 2024-04-16T20:18:10.814 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.814 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:18:10.814 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.814 DEBUG:boto:Params: {} 2024-04-16T20:18:10.814 DEBUG:boto:Token: None 2024-04-16T20:18:10.815 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.815 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb501d8b244ac0be-00661edd02-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.817 DEBUG:boto:b'yrjefh-55obj551000falseobj552024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.818 DEBUG:boto:path=/yrjefh-55/ 2024-04-16T20:18:10.818 DEBUG:boto:auth_path=/yrjefh-55/ 2024-04-16T20:18:10.818 DEBUG:boto:path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.818 DEBUG:boto:auth_path=/yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.818 DEBUG:boto:Method: GET 2024-04-16T20:18:10.818 DEBUG:boto:Path: /yrjefh-55/?prefix=obj55 2024-04-16T20:18:10.818 DEBUG:boto:Data: 2024-04-16T20:18:10.818 DEBUG:boto:Headers: {} 2024-04-16T20:18:10.818 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:10.818 DEBUG:boto:Port: 8000 2024-04-16T20:18:10.819 DEBUG:boto:Params: {} 2024-04-16T20:18:10.819 DEBUG:boto:Token: None 2024-04-16T20:18:10.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:18:10 GMT /yrjefh-55/ 2024-04-16T20:18:10.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM= 2024-04-16T20:18:10.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DgUCc5yEROlPl26hbnt0+F7k5LM=', 'Content-Length': '0'} 2024-04-16T20:18:10.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054fb22141309738b-00661edd02-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:10.821 DEBUG:boto:b'yrjefh-55obj551000falseobj552024-04-16T20:16:58.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:18:10.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-55 zones={test-zone2, test-zone1} 2024-04-16T20:18:10.822 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-16T20:18:10.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:10.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:10.822 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:10.876 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:10.876 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:10.963 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"00000000000000000000:00000000000000035003","last_update":"2024-04-16T20:17:07.509371Z"},{"marker":"00000000000000000000:00000000000000026366","last_update":"2024-04-16T20:13:05.504529Z"},{"marker":"00000000000000000000:00000000000000038908","last_update":"2024-04-16T20:16:58.181960Z"},{"marker":"00000000000000000000:00000000000000035240","last_update":"2024-04-16T20:17:07.511074Z"}] 2024-04-16T20:18:10.963 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:10.963 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:18:10.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:10.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:10.964 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:11.022 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:11.022 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:11.102 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:11Z"} 2024-04-16T20:18:11.103 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:11Z"} 2024-04-16T20:18:11.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000026366', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000035099', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.132668Z'}}] 2024-04-16T20:18:11.103 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:11.103 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '', 2: '00000000000000000000:00000000000000038908', 3: ''} 2024-04-16T20:18:11.103 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026366 target=, shard=3 master=00000000000000000000:00000000000000035240 target= 2024-04-16T20:18:16.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:16.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:16.105 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:16.164 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:16.164 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:16.243 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:16Z"} 2024-04-16T20:18:16.244 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:16Z"} 2024-04-16T20:18:16.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000026366', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000035099', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.132668Z'}}] 2024-04-16T20:18:16.244 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:16.244 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '', 2: '00000000000000000000:00000000000000038908', 3: ''} 2024-04-16T20:18:16.244 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026366 target=, shard=3 master=00000000000000000000:00000000000000035240 target= 2024-04-16T20:18:21.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:21.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:21.245 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:21.384 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:21Z"} 2024-04-16T20:18:21.384 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:21Z"} 2024-04-16T20:18:21.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000026366', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000035099', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.132668Z'}}] 2024-04-16T20:18:21.385 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:21.385 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '', 2: '00000000000000000000:00000000000000038908', 3: ''} 2024-04-16T20:18:21.385 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026366 target=, shard=3 master=00000000000000000000:00000000000000035240 target= 2024-04-16T20:18:26.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:26.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:26.386 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:26.526 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:26Z"} 2024-04-16T20:18:26.527 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:26Z"} 2024-04-16T20:18:26.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000026366', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000035099', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.132668Z'}}] 2024-04-16T20:18:26.527 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:26.527 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '', 2: '00000000000000000000:00000000000000038908', 3: ''} 2024-04-16T20:18:26.527 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026366 target=, shard=3 master=00000000000000000000:00000000000000035240 target= 2024-04-16T20:18:31.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:31.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:31.529 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:31.586 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:31.586 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:31.668 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:31Z"} 2024-04-16T20:18:31.668 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000026366","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"full-sync","marker":"","next_step_marker":"00000000000000000000:00000000000000035099","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:16:58.132668Z"}}]},"full_sync":{"total":55,"complete":46},"current_time":"2024-04-16T20:18:31Z"} 2024-04-16T20:18:31.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000026366', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000035099', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.132668Z'}}] 2024-04-16T20:18:31.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:31.668 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '', 2: '00000000000000000000:00000000000000038908', 3: ''} 2024-04-16T20:18:31.668 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026366 target=, shard=3 master=00000000000000000000:00000000000000035240 target= 2024-04-16T20:18:36.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.670 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:36.730 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:36.730 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:36.813 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026366","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035240","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:17:07.511074Z"}}]},"full_sync":{"total":55,"complete":55},"current_time":"2024-04-16T20:18:36Z"} 2024-04-16T20:18:36.813 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13537744503167961095},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035003","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:17:07.509371Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000026366","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:13:05.504529Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000038908","next_step_marker":"","total_entries":46,"pos":0,"timestamp":"2024-04-16T20:16:58.181960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000035240","next_step_marker":"","total_entries":9,"pos":0,"timestamp":"2024-04-16T20:17:07.511074Z"}}]},"full_sync":{"total":55,"complete":55},"current_time":"2024-04-16T20:18:36Z"} 2024-04-16T20:18:36.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.509371Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026366', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:13:05.504529Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000038908', 'next_step_marker': '', 'total_entries': 46, 'pos': 0, 'timestamp': '2024-04-16T20:16:58.181960Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035240', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-16T20:17:07.511074Z'}}] 2024-04-16T20:18:36.813 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:36.814 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000035003', 1: '00000000000000000000:00000000000000026366', 2: '00000000000000000000:00000000000000038908', 3: '00000000000000000000:00000000000000035240'} 2024-04-16T20:18:36.814 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-16T20:18:36.814 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-16T20:18:36.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.814 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:36.861 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:36.861 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:36.942 INFO:teuthology.orchestra.run.smithi119.stdout:[{"marker":"00000000000000000000:00000000000000014347","last_update":"2024-04-16T20:18:04.517422Z"},{"marker":"00000000000000000000:00000000000000007969","last_update":"2024-04-16T20:14:25.875693Z"},{"marker":"00000000000000000000:00000000000000015575","last_update":"2024-04-16T20:18:02.308895Z"},{"marker":"00000000000000000000:00000000000000017135","last_update":"2024-04-16T20:18:04.519488Z"}] 2024-04-16T20:18:36.942 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000014347', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015575', 3: '00000000000000000000:00000000000000017135'} 2024-04-16T20:18:36.942 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:18:36.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:36.943 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:37.082 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014347","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:04.517422Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007969","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:25.875693Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015575","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:02.308895Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000017135","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:18:04.519488Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T20:18:37.083 DEBUG:rgw_multi.tests:current data sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":133780877208329192},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014347","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:04.517422Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007969","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:14:25.875693Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000015575","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:02.308895Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000017135","next_step_marker":"","total_entries":2,"pos":0,"timestamp":"2024-04-16T20:18:04.519488Z"}}]},"full_sync":{"total":2,"complete":2},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T20:18:37.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014347', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:04.517422Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007969', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:14:25.875693Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015575', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:02.308895Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017135', 'next_step_marker': '', 'total_entries': 2, 'pos': 0, 'timestamp': '2024-04-16T20:18:04.519488Z'}}] 2024-04-16T20:18:37.084 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014347', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015575', 3: '00000000000000000000:00000000000000017135'} 2024-04-16T20:18:37.084 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014347', 1: '00000000000000000000:00000000000000007969', 2: '00000000000000000000:00000000000000015575', 3: '00000000000000000000:00000000000000017135'} 2024-04-16T20:18:37.084 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-16T20:18:37.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2024-04-16T20:18:37.087 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:37.088 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:37.088 DEBUG:boto:Method: PUT 2024-04-16T20:18:37.088 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:37.088 DEBUG:boto:Data: 2024-04-16T20:18:37.089 DEBUG:boto:Headers: {} 2024-04-16T20:18:37.089 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:37.089 DEBUG:boto:Port: 8000 2024-04-16T20:18:37.089 DEBUG:boto:Params: {} 2024-04-16T20:18:37.089 DEBUG:boto:Token: None 2024-04-16T20:18:37.090 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:18:37 GMT /yrjefh-56/ 2024-04-16T20:18:37.090 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2HbgxF/3u73RrekhVsZJvkisZOM= 2024-04-16T20:18:37.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2HbgxF/3u73RrekhVsZJvkisZOM=', 'Content-Length': '0'} 2024-04-16T20:18:37.105 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6df4b68802604a8-00661edd1d-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:37.105 DEBUG:rgw_multi.tests:created bucket=yrjefh-56 2024-04-16T20:18:37.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:37.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:37.106 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:18:37.229 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713298717.114405_1133.1","last_update":"2024-04-16T20:18:37.114405Z"}] 2024-04-16T20:18:37.229 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T20:18:37.230 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:18:37.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:37.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:37.230 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:37.279 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:37.279 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:37.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:37.339+0000 7fa597375800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:18:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:37.341+0000 7fa597375800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:18:37.353 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.107221_1130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.107221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T20:18:37.353 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.107221_1130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.107221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T20:18:37.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298717.107221_1130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:37.107221Z', 'realm_epoch': 2}}] 2024-04-16T20:18:37.353 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T20:18:37.353 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.107221_1130.1'} 2024-04-16T20:18:37.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713298717.114405_1133.1 target=1_1713298717.107221_1130.1 2024-04-16T20:18:42.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:42.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:18:42.355 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:18:42.409 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:42.409 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:42.475+0000 7f226ed1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:18:42.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:42.476+0000 7f226ed1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:18:42.490 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.114405_1133.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.114405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:42Z"} 2024-04-16T20:18:42.490 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.114405_1133.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.114405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:42Z"} 2024-04-16T20:18:42.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298717.114405_1133.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:37.114405Z', 'realm_epoch': 2}}] 2024-04-16T20:18:42.490 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T20:18:42.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T20:18:42.491 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:18:42.491 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.491 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.491 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.491 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.491 DEBUG:boto:Data: 2024-04-16T20:18:42.491 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.491 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.491 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.491 DEBUG:boto:Params: {} 2024-04-16T20:18:42.491 DEBUG:boto:Token: None 2024-04-16T20:18:42.492 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030cead5e5e762e2d-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.495 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:18:42.495 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:18:42.495 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.495 DEBUG:boto:Path: /yrjefh-56/obj11 2024-04-16T20:18:42.495 DEBUG:boto:Data: 2024-04-16T20:18:42.495 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.495 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.495 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.495 DEBUG:boto:Params: {} 2024-04-16T20:18:42.495 DEBUG:boto:Token: None 2024-04-16T20:18:42.496 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj11 2024-04-16T20:18:42.496 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ei850mX93LfhWqd7UR29TexZcDg= 2024-04-16T20:18:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ei850mX93LfhWqd7UR29TexZcDg='} 2024-04-16T20:18:42.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c8e7c218ab554b6-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.503 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.504 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.504 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.504 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.504 DEBUG:boto:Data: 2024-04-16T20:18:42.504 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.504 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.504 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.504 DEBUG:boto:Params: {} 2024-04-16T20:18:42.504 DEBUG:boto:Token: None 2024-04-16T20:18:42.504 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.505 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccb9503e81a4900c-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.507 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:18:42.507 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:18:42.507 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.507 DEBUG:boto:Path: /yrjefh-56/obj22 2024-04-16T20:18:42.507 DEBUG:boto:Data: 2024-04-16T20:18:42.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.508 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.508 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.508 DEBUG:boto:Params: {} 2024-04-16T20:18:42.508 DEBUG:boto:Token: None 2024-04-16T20:18:42.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj22 2024-04-16T20:18:42.508 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tHnsVjnXaxG18QiwdGVRYgHWPLk= 2024-04-16T20:18:42.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tHnsVjnXaxG18QiwdGVRYgHWPLk='} 2024-04-16T20:18:42.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deacefb2dca957a3-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.551 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.551 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.551 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.551 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.551 DEBUG:boto:Data: 2024-04-16T20:18:42.551 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.551 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.551 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.551 DEBUG:boto:Params: {} 2024-04-16T20:18:42.551 DEBUG:boto:Token: None 2024-04-16T20:18:42.552 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.552 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b1d50a916c1d406-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.554 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:18:42.554 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:18:42.554 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.554 DEBUG:boto:Path: /yrjefh-56/obj33 2024-04-16T20:18:42.554 DEBUG:boto:Data: 2024-04-16T20:18:42.554 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.555 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.555 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.555 DEBUG:boto:Params: {} 2024-04-16T20:18:42.555 DEBUG:boto:Token: None 2024-04-16T20:18:42.555 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj33 2024-04-16T20:18:42.555 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XkM2LZ2vEpAb+5dtiXtLXRGqEko= 2024-04-16T20:18:42.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XkM2LZ2vEpAb+5dtiXtLXRGqEko='} 2024-04-16T20:18:42.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c56242465654552f-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.562 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.562 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.562 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.562 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.562 DEBUG:boto:Data: 2024-04-16T20:18:42.562 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.562 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.562 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.562 DEBUG:boto:Params: {} 2024-04-16T20:18:42.562 DEBUG:boto:Token: None 2024-04-16T20:18:42.563 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.563 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca78c572d3f60ac4-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.565 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:18:42.565 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:18:42.565 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.565 DEBUG:boto:Path: /yrjefh-56/obj44 2024-04-16T20:18:42.565 DEBUG:boto:Data: 2024-04-16T20:18:42.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.565 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.565 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.565 DEBUG:boto:Params: {} 2024-04-16T20:18:42.566 DEBUG:boto:Token: None 2024-04-16T20:18:42.566 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj44 2024-04-16T20:18:42.566 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PsOgwh8go30mz2R9+Qm/hhh9BRk= 2024-04-16T20:18:42.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PsOgwh8go30mz2R9+Qm/hhh9BRk='} 2024-04-16T20:18:42.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088e35268e4e3e54e-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.573 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.573 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.574 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.574 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.574 DEBUG:boto:Data: 2024-04-16T20:18:42.574 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.574 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.574 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.574 DEBUG:boto:Params: {} 2024-04-16T20:18:42.574 DEBUG:boto:Token: None 2024-04-16T20:18:42.574 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000785fb29d4411fdbe-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.576 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:18:42.577 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:18:42.577 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.577 DEBUG:boto:Path: /yrjefh-56/obj55 2024-04-16T20:18:42.577 DEBUG:boto:Data: 2024-04-16T20:18:42.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.577 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.577 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.577 DEBUG:boto:Params: {} 2024-04-16T20:18:42.577 DEBUG:boto:Token: None 2024-04-16T20:18:42.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj55 2024-04-16T20:18:42.578 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RvGyZLpjpkyFihLmaZ27pNUs2LQ= 2024-04-16T20:18:42.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RvGyZLpjpkyFihLmaZ27pNUs2LQ='} 2024-04-16T20:18:42.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a585ef2b621d5db-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:42.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:42.623 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:42.681 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:42.681 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:42.752 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:42.752 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:42.757 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 5 2024-04-16T20:18:42.807 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.807 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.807 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.807 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.807 DEBUG:boto:Data: 2024-04-16T20:18:42.807 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.807 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.807 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.808 DEBUG:boto:Params: {} 2024-04-16T20:18:42.808 DEBUG:boto:Token: None 2024-04-16T20:18:42.808 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.808 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f438a5599526dc4-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.811 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:18:42.811 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:18:42.811 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.811 DEBUG:boto:Path: /yrjefh-56/obj17 2024-04-16T20:18:42.811 DEBUG:boto:Data: 2024-04-16T20:18:42.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.811 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.812 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.812 DEBUG:boto:Params: {} 2024-04-16T20:18:42.812 DEBUG:boto:Token: None 2024-04-16T20:18:42.812 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj17 2024-04-16T20:18:42.812 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h3H4NmHEnZJQ+Lo7IKmv9chwWso= 2024-04-16T20:18:42.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h3H4NmHEnZJQ+Lo7IKmv9chwWso='} 2024-04-16T20:18:42.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000699946e8f5996ee9-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.818 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.818 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.818 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.818 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.818 DEBUG:boto:Data: 2024-04-16T20:18:42.818 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.818 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.818 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.818 DEBUG:boto:Params: {} 2024-04-16T20:18:42.819 DEBUG:boto:Token: None 2024-04-16T20:18:42.819 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2a765d48d87f81-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.822 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:18:42.822 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:18:42.822 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.822 DEBUG:boto:Path: /yrjefh-56/obj34 2024-04-16T20:18:42.822 DEBUG:boto:Data: 2024-04-16T20:18:42.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.822 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.822 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.822 DEBUG:boto:Params: {} 2024-04-16T20:18:42.823 DEBUG:boto:Token: None 2024-04-16T20:18:42.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj34 2024-04-16T20:18:42.823 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QE3RsfnFohD4l00gvKrbQlMBB0c= 2024-04-16T20:18:42.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QE3RsfnFohD4l00gvKrbQlMBB0c='} 2024-04-16T20:18:42.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a90fdb3bfbc7cdf2-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.828 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.828 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.828 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.828 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.828 DEBUG:boto:Data: 2024-04-16T20:18:42.828 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.829 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.829 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.829 DEBUG:boto:Params: {} 2024-04-16T20:18:42.829 DEBUG:boto:Token: None 2024-04-16T20:18:42.829 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.829 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005645cb0a27146fa0-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.832 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:18:42.832 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:18:42.832 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.832 DEBUG:boto:Path: /yrjefh-56/obj51 2024-04-16T20:18:42.832 DEBUG:boto:Data: 2024-04-16T20:18:42.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.833 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.833 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.833 DEBUG:boto:Params: {} 2024-04-16T20:18:42.833 DEBUG:boto:Token: None 2024-04-16T20:18:42.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj51 2024-04-16T20:18:42.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv4lY1r6mFajmdgBwnDWW7zKqmM= 2024-04-16T20:18:42.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv4lY1r6mFajmdgBwnDWW7zKqmM='} 2024-04-16T20:18:42.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3a7d184aebd1973-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.879 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.879 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.879 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.879 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.879 DEBUG:boto:Data: 2024-04-16T20:18:42.879 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.879 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.879 DEBUG:boto:Params: {} 2024-04-16T20:18:42.879 DEBUG:boto:Token: None 2024-04-16T20:18:42.879 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000281eec47ebd9d143-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.882 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:18:42.882 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:18:42.882 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.882 DEBUG:boto:Path: /yrjefh-56/obj68 2024-04-16T20:18:42.882 DEBUG:boto:Data: 2024-04-16T20:18:42.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.883 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.883 DEBUG:boto:Params: {} 2024-04-16T20:18:42.883 DEBUG:boto:Token: None 2024-04-16T20:18:42.883 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj68 2024-04-16T20:18:42.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IcX+COQxaNuNMVXYDl/r6H65zTc= 2024-04-16T20:18:42.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IcX+COQxaNuNMVXYDl/r6H65zTc='} 2024-04-16T20:18:42.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed4f6aecbcafb36a-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.888 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:42.888 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:42.888 DEBUG:boto:Method: HEAD 2024-04-16T20:18:42.888 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:42.888 DEBUG:boto:Data: 2024-04-16T20:18:42.889 DEBUG:boto:Headers: {} 2024-04-16T20:18:42.889 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.889 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.889 DEBUG:boto:Params: {} 2024-04-16T20:18:42.889 DEBUG:boto:Token: None 2024-04-16T20:18:42.889 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/ 2024-04-16T20:18:42.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T20:18:42.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T20:18:42.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e57a24cbd62c271-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.892 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:18:42.892 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:18:42.892 DEBUG:boto:Method: PUT 2024-04-16T20:18:42.892 DEBUG:boto:Path: /yrjefh-56/obj85 2024-04-16T20:18:42.892 DEBUG:boto:Data: 2024-04-16T20:18:42.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:42.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:42.892 DEBUG:boto:Port: 8000 2024-04-16T20:18:42.892 DEBUG:boto:Params: {} 2024-04-16T20:18:42.892 DEBUG:boto:Token: None 2024-04-16T20:18:42.893 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:42 GMT /yrjefh-56/obj85 2024-04-16T20:18:42.893 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iQLl1c6qddkjDMjiavLvcnV4iy0= 2024-04-16T20:18:42.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iQLl1c6qddkjDMjiavLvcnV4iy0='} 2024-04-16T20:18:42.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b93086a85cf1f15a-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:42.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:42.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:42.935 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:42.986 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:42.986 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:43.052 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:43.052 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:43.059 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 10 2024-04-16T20:18:43.114 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.114 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.114 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.114 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.114 DEBUG:boto:Data: 2024-04-16T20:18:43.115 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.115 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.115 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.115 DEBUG:boto:Params: {} 2024-04-16T20:18:43.115 DEBUG:boto:Token: None 2024-04-16T20:18:43.115 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.115 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c84c5aef789e35ac-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.118 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:18:43.118 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:18:43.118 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.118 DEBUG:boto:Path: /yrjefh-56/obj19 2024-04-16T20:18:43.118 DEBUG:boto:Data: 2024-04-16T20:18:43.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.118 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.118 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.118 DEBUG:boto:Params: {} 2024-04-16T20:18:43.118 DEBUG:boto:Token: None 2024-04-16T20:18:43.119 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj19 2024-04-16T20:18:43.119 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xPXNrAJ1p7kQlpc4IanzEqmFlbg= 2024-04-16T20:18:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xPXNrAJ1p7kQlpc4IanzEqmFlbg='} 2024-04-16T20:18:43.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a16f8228b9d90bfc-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.124 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.124 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.124 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.124 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.124 DEBUG:boto:Data: 2024-04-16T20:18:43.124 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.124 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.124 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.124 DEBUG:boto:Params: {} 2024-04-16T20:18:43.125 DEBUG:boto:Token: None 2024-04-16T20:18:43.125 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.125 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fa42bed07f051cb-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.128 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:18:43.128 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:18:43.128 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.128 DEBUG:boto:Path: /yrjefh-56/obj38 2024-04-16T20:18:43.128 DEBUG:boto:Data: 2024-04-16T20:18:43.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.128 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.128 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.128 DEBUG:boto:Params: {} 2024-04-16T20:18:43.128 DEBUG:boto:Token: None 2024-04-16T20:18:43.128 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj38 2024-04-16T20:18:43.129 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vsbUOb0ksf7H8Q2455UWNwl090E= 2024-04-16T20:18:43.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsbUOb0ksf7H8Q2455UWNwl090E='} 2024-04-16T20:18:43.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cdd63e0821d1c6d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.132 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.132 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.133 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.133 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.133 DEBUG:boto:Data: 2024-04-16T20:18:43.133 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.133 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.133 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.133 DEBUG:boto:Params: {} 2024-04-16T20:18:43.133 DEBUG:boto:Token: None 2024-04-16T20:18:43.133 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.133 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af4f3ea239f7c1d6-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.136 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:18:43.136 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:18:43.136 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.136 DEBUG:boto:Path: /yrjefh-56/obj57 2024-04-16T20:18:43.136 DEBUG:boto:Data: 2024-04-16T20:18:43.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.136 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.136 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.136 DEBUG:boto:Params: {} 2024-04-16T20:18:43.137 DEBUG:boto:Token: None 2024-04-16T20:18:43.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj57 2024-04-16T20:18:43.137 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2AObidFTwdNql4PenePDHcZSazw= 2024-04-16T20:18:43.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2AObidFTwdNql4PenePDHcZSazw='} 2024-04-16T20:18:43.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053eb45d226be0160-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.178 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.179 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.179 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.179 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.179 DEBUG:boto:Data: 2024-04-16T20:18:43.179 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.179 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.179 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.179 DEBUG:boto:Params: {} 2024-04-16T20:18:43.179 DEBUG:boto:Token: None 2024-04-16T20:18:43.179 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.180 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b362681b35d1c25-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.182 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:18:43.182 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:18:43.182 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.183 DEBUG:boto:Path: /yrjefh-56/obj76 2024-04-16T20:18:43.183 DEBUG:boto:Data: 2024-04-16T20:18:43.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.183 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.183 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.183 DEBUG:boto:Params: {} 2024-04-16T20:18:43.183 DEBUG:boto:Token: None 2024-04-16T20:18:43.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj76 2024-04-16T20:18:43.183 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hazeifjdopvqaEmYcCFK+akK+/U= 2024-04-16T20:18:43.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hazeifjdopvqaEmYcCFK+akK+/U='} 2024-04-16T20:18:43.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095811c13c8b2bead-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.189 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.189 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.189 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.189 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.189 DEBUG:boto:Data: 2024-04-16T20:18:43.189 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.189 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.189 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.189 DEBUG:boto:Params: {} 2024-04-16T20:18:43.189 DEBUG:boto:Token: None 2024-04-16T20:18:43.189 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.190 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000930c9f8a774c5fe9-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.192 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:18:43.192 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:18:43.192 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.192 DEBUG:boto:Path: /yrjefh-56/obj95 2024-04-16T20:18:43.193 DEBUG:boto:Data: 2024-04-16T20:18:43.193 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.193 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.193 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.193 DEBUG:boto:Params: {} 2024-04-16T20:18:43.193 DEBUG:boto:Token: None 2024-04-16T20:18:43.193 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj95 2024-04-16T20:18:43.193 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tDXpehHrOR5aVIIHeQxxDbDqD1s= 2024-04-16T20:18:43.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tDXpehHrOR5aVIIHeQxxDbDqD1s='} 2024-04-16T20:18:43.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deb4e1826fb26c26-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.197 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:43.247 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:43.248 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:43.313 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:43.313 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:43.320 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 15 2024-04-16T20:18:43.378 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.378 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.378 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.378 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.378 DEBUG:boto:Data: 2024-04-16T20:18:43.378 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.378 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.379 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.379 DEBUG:boto:Params: {} 2024-04-16T20:18:43.379 DEBUG:boto:Token: None 2024-04-16T20:18:43.379 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.379 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f806f1ccbfc19cce-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.382 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:18:43.382 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:18:43.382 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.382 DEBUG:boto:Path: /yrjefh-56/obj23 2024-04-16T20:18:43.382 DEBUG:boto:Data: 2024-04-16T20:18:43.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.382 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.382 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.382 DEBUG:boto:Params: {} 2024-04-16T20:18:43.383 DEBUG:boto:Token: None 2024-04-16T20:18:43.383 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj23 2024-04-16T20:18:43.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jApS8M0VSiXyQI4YKNp2mUyvjMs= 2024-04-16T20:18:43.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jApS8M0VSiXyQI4YKNp2mUyvjMs='} 2024-04-16T20:18:43.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3d488b13d651d75-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.388 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.389 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.389 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.389 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.389 DEBUG:boto:Data: 2024-04-16T20:18:43.389 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.389 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.389 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.389 DEBUG:boto:Params: {} 2024-04-16T20:18:43.389 DEBUG:boto:Token: None 2024-04-16T20:18:43.389 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.390 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0a9525436158aad-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.392 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:18:43.392 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:18:43.392 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.393 DEBUG:boto:Path: /yrjefh-56/obj46 2024-04-16T20:18:43.393 DEBUG:boto:Data: 2024-04-16T20:18:43.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.393 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.393 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.393 DEBUG:boto:Params: {} 2024-04-16T20:18:43.393 DEBUG:boto:Token: None 2024-04-16T20:18:43.393 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj46 2024-04-16T20:18:43.393 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:yjgBwnue/IFIYAH9iZGvrd3PNeM= 2024-04-16T20:18:43.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yjgBwnue/IFIYAH9iZGvrd3PNeM='} 2024-04-16T20:18:43.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bec248258f171dd8-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.443 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.443 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.443 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.443 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.443 DEBUG:boto:Data: 2024-04-16T20:18:43.443 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.443 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.443 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.443 DEBUG:boto:Params: {} 2024-04-16T20:18:43.443 DEBUG:boto:Token: None 2024-04-16T20:18:43.444 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e17bc47d84028e-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.446 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:18:43.446 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:18:43.447 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.447 DEBUG:boto:Path: /yrjefh-56/obj69 2024-04-16T20:18:43.447 DEBUG:boto:Data: 2024-04-16T20:18:43.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.447 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.447 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.447 DEBUG:boto:Params: {} 2024-04-16T20:18:43.447 DEBUG:boto:Token: None 2024-04-16T20:18:43.447 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj69 2024-04-16T20:18:43.447 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eXhOyuSpEvfm8WUCC0vdESQ7g1o= 2024-04-16T20:18:43.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eXhOyuSpEvfm8WUCC0vdESQ7g1o='} 2024-04-16T20:18:43.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d8e6042c66c3776-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.453 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.453 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.453 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.453 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.454 DEBUG:boto:Data: 2024-04-16T20:18:43.454 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.454 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.454 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.454 DEBUG:boto:Params: {} 2024-04-16T20:18:43.454 DEBUG:boto:Token: None 2024-04-16T20:18:43.454 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.454 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069246b9a8798de40-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.457 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:18:43.457 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:18:43.457 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.457 DEBUG:boto:Path: /yrjefh-56/obj92 2024-04-16T20:18:43.457 DEBUG:boto:Data: 2024-04-16T20:18:43.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.457 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.457 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.457 DEBUG:boto:Params: {} 2024-04-16T20:18:43.457 DEBUG:boto:Token: None 2024-04-16T20:18:43.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj92 2024-04-16T20:18:43.458 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:epkCsVBSqaLrcVejRWfR4xTcOiQ= 2024-04-16T20:18:43.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:epkCsVBSqaLrcVejRWfR4xTcOiQ='} 2024-04-16T20:18:43.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bc6f2e9f5d1ac6f-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.462 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.462 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.463 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.463 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.463 DEBUG:boto:Data: 2024-04-16T20:18:43.463 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.463 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.463 DEBUG:boto:Params: {} 2024-04-16T20:18:43.463 DEBUG:boto:Token: None 2024-04-16T20:18:43.463 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.463 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006013df2c70f3b182-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.466 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:18:43.466 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:18:43.466 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.467 DEBUG:boto:Path: /yrjefh-56/obj115 2024-04-16T20:18:43.467 DEBUG:boto:Data: 2024-04-16T20:18:43.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.467 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.467 DEBUG:boto:Params: {} 2024-04-16T20:18:43.467 DEBUG:boto:Token: None 2024-04-16T20:18:43.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj115 2024-04-16T20:18:43.467 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rUggxpUMk4jppP3tpfrZ0ofBGAo= 2024-04-16T20:18:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rUggxpUMk4jppP3tpfrZ0ofBGAo='} 2024-04-16T20:18:43.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b093cc8b2c67037d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.472 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:43.595 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:43.595 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:43.605 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 20 2024-04-16T20:18:43.676 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.676 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.676 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.676 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.677 DEBUG:boto:Data: 2024-04-16T20:18:43.677 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.677 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.677 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.677 DEBUG:boto:Params: {} 2024-04-16T20:18:43.677 DEBUG:boto:Token: None 2024-04-16T20:18:43.677 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.677 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000735f85113ab62a25-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.680 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:18:43.680 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:18:43.681 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.681 DEBUG:boto:Path: /yrjefh-56/obj29 2024-04-16T20:18:43.681 DEBUG:boto:Data: 2024-04-16T20:18:43.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.681 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.681 DEBUG:boto:Params: {} 2024-04-16T20:18:43.681 DEBUG:boto:Token: None 2024-04-16T20:18:43.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj29 2024-04-16T20:18:43.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rnfcJPQpXSjxQCvNlsGBMuhivN4= 2024-04-16T20:18:43.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rnfcJPQpXSjxQCvNlsGBMuhivN4='} 2024-04-16T20:18:43.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f1281cc73c692ba-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.731 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.731 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.731 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.731 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.731 DEBUG:boto:Data: 2024-04-16T20:18:43.731 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.731 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.731 DEBUG:boto:Params: {} 2024-04-16T20:18:43.731 DEBUG:boto:Token: None 2024-04-16T20:18:43.731 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090ba74043f4636cc-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.734 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:18:43.734 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:18:43.735 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.735 DEBUG:boto:Path: /yrjefh-56/obj58 2024-04-16T20:18:43.735 DEBUG:boto:Data: 2024-04-16T20:18:43.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.735 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.735 DEBUG:boto:Params: {} 2024-04-16T20:18:43.735 DEBUG:boto:Token: None 2024-04-16T20:18:43.735 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj58 2024-04-16T20:18:43.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oC5OtZIDJSEMPEFQFcp7R7KyYmE= 2024-04-16T20:18:43.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oC5OtZIDJSEMPEFQFcp7R7KyYmE='} 2024-04-16T20:18:43.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d01179759637a260-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.741 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.741 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.741 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.741 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.741 DEBUG:boto:Data: 2024-04-16T20:18:43.741 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.741 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.741 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.741 DEBUG:boto:Params: {} 2024-04-16T20:18:43.742 DEBUG:boto:Token: None 2024-04-16T20:18:43.742 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.742 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009120d9eec306ac7e-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.745 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:18:43.745 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:18:43.745 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.745 DEBUG:boto:Path: /yrjefh-56/obj87 2024-04-16T20:18:43.745 DEBUG:boto:Data: 2024-04-16T20:18:43.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.745 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.746 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.746 DEBUG:boto:Params: {} 2024-04-16T20:18:43.746 DEBUG:boto:Token: None 2024-04-16T20:18:43.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj87 2024-04-16T20:18:43.746 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M8j3bP7TLEVKhmamVj+6f9XqNTA= 2024-04-16T20:18:43.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M8j3bP7TLEVKhmamVj+6f9XqNTA='} 2024-04-16T20:18:43.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000387886e07fdc9b9d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.751 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.751 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.752 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.752 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.752 DEBUG:boto:Data: 2024-04-16T20:18:43.752 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.752 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.752 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.752 DEBUG:boto:Params: {} 2024-04-16T20:18:43.752 DEBUG:boto:Token: None 2024-04-16T20:18:43.752 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.752 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b93bf8bf3c2431-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.755 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:18:43.755 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:18:43.755 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.755 DEBUG:boto:Path: /yrjefh-56/obj116 2024-04-16T20:18:43.755 DEBUG:boto:Data: 2024-04-16T20:18:43.756 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.756 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.756 DEBUG:boto:Params: {} 2024-04-16T20:18:43.756 DEBUG:boto:Token: None 2024-04-16T20:18:43.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj116 2024-04-16T20:18:43.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XxhrP8MnRx0vA/EecBY4z+eru4o= 2024-04-16T20:18:43.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XxhrP8MnRx0vA/EecBY4z+eru4o='} 2024-04-16T20:18:43.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007bdc600592d8596-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.799 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:43.799 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:43.799 DEBUG:boto:Method: HEAD 2024-04-16T20:18:43.799 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:43.799 DEBUG:boto:Data: 2024-04-16T20:18:43.799 DEBUG:boto:Headers: {} 2024-04-16T20:18:43.799 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.799 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.799 DEBUG:boto:Params: {} 2024-04-16T20:18:43.799 DEBUG:boto:Token: None 2024-04-16T20:18:43.799 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/ 2024-04-16T20:18:43.800 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T20:18:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T20:18:43.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eb20e907ced8eff-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.802 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:18:43.803 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:18:43.803 DEBUG:boto:Method: PUT 2024-04-16T20:18:43.803 DEBUG:boto:Path: /yrjefh-56/obj145 2024-04-16T20:18:43.803 DEBUG:boto:Data: 2024-04-16T20:18:43.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:43.803 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:43.803 DEBUG:boto:Port: 8000 2024-04-16T20:18:43.803 DEBUG:boto:Params: {} 2024-04-16T20:18:43.803 DEBUG:boto:Token: None 2024-04-16T20:18:43.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:43 GMT /yrjefh-56/obj145 2024-04-16T20:18:43.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4R/NZYv/V+vrd53gHrUlOcKv+ac= 2024-04-16T20:18:43.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4R/NZYv/V+vrd53gHrUlOcKv+ac='} 2024-04-16T20:18:43.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc6d6b7fb5a70597-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:43.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:43.809 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:43.860 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:43.860 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:43.928 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:43.929 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:43.941 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 25 2024-04-16T20:18:44.012 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.012 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.012 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.013 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.013 DEBUG:boto:Data: 2024-04-16T20:18:44.013 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.013 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.013 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.013 DEBUG:boto:Params: {} 2024-04-16T20:18:44.013 DEBUG:boto:Token: None 2024-04-16T20:18:44.013 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.013 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009da0c9f957d77bd9-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.017 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:18:44.017 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:18:44.017 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.017 DEBUG:boto:Path: /yrjefh-56/obj31 2024-04-16T20:18:44.017 DEBUG:boto:Data: 2024-04-16T20:18:44.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.017 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.017 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.017 DEBUG:boto:Params: {} 2024-04-16T20:18:44.017 DEBUG:boto:Token: None 2024-04-16T20:18:44.017 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj31 2024-04-16T20:18:44.018 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Nkh0PFsRE81z/Dc8Iu/75ABx02o= 2024-04-16T20:18:44.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nkh0PFsRE81z/Dc8Iu/75ABx02o='} 2024-04-16T20:18:44.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074d9e4d4c716e4ac-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.063 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.063 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.063 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.063 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.063 DEBUG:boto:Data: 2024-04-16T20:18:44.063 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.063 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.063 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.063 DEBUG:boto:Params: {} 2024-04-16T20:18:44.063 DEBUG:boto:Token: None 2024-04-16T20:18:44.064 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.064 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000710009642ed29f0e-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.067 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:18:44.067 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:18:44.067 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.067 DEBUG:boto:Path: /yrjefh-56/obj62 2024-04-16T20:18:44.067 DEBUG:boto:Data: 2024-04-16T20:18:44.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.067 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.067 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.067 DEBUG:boto:Params: {} 2024-04-16T20:18:44.067 DEBUG:boto:Token: None 2024-04-16T20:18:44.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj62 2024-04-16T20:18:44.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mcmnJaw9MdZNIfwzi8RTLLyry1I= 2024-04-16T20:18:44.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mcmnJaw9MdZNIfwzi8RTLLyry1I='} 2024-04-16T20:18:44.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2ef7c52a5dadbaa-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.073 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.074 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.074 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.074 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.074 DEBUG:boto:Data: 2024-04-16T20:18:44.074 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.074 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.074 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.074 DEBUG:boto:Params: {} 2024-04-16T20:18:44.074 DEBUG:boto:Token: None 2024-04-16T20:18:44.074 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.075 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077bf90e624879934-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.077 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:18:44.077 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:18:44.077 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.078 DEBUG:boto:Path: /yrjefh-56/obj93 2024-04-16T20:18:44.078 DEBUG:boto:Data: 2024-04-16T20:18:44.078 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.078 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.078 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.078 DEBUG:boto:Params: {} 2024-04-16T20:18:44.078 DEBUG:boto:Token: None 2024-04-16T20:18:44.078 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj93 2024-04-16T20:18:44.078 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IsPZT0IN0bZm/6AshlUYFsnm1/A= 2024-04-16T20:18:44.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IsPZT0IN0bZm/6AshlUYFsnm1/A='} 2024-04-16T20:18:44.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de3da05a46e21e1c-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.123 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.123 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.123 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.123 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.123 DEBUG:boto:Data: 2024-04-16T20:18:44.123 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.123 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.123 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.123 DEBUG:boto:Params: {} 2024-04-16T20:18:44.123 DEBUG:boto:Token: None 2024-04-16T20:18:44.123 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.124 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c80e769b3b08eb7-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.126 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:18:44.126 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:18:44.127 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.127 DEBUG:boto:Path: /yrjefh-56/obj124 2024-04-16T20:18:44.127 DEBUG:boto:Data: 2024-04-16T20:18:44.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.127 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.127 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.127 DEBUG:boto:Params: {} 2024-04-16T20:18:44.127 DEBUG:boto:Token: None 2024-04-16T20:18:44.127 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj124 2024-04-16T20:18:44.127 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9asGkxEKuQc8FFwE5zGyyso9yq4= 2024-04-16T20:18:44.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9asGkxEKuQc8FFwE5zGyyso9yq4='} 2024-04-16T20:18:44.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebcbfc030af025de-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.133 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.133 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.133 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.133 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.133 DEBUG:boto:Data: 2024-04-16T20:18:44.133 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.133 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.133 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.133 DEBUG:boto:Params: {} 2024-04-16T20:18:44.134 DEBUG:boto:Token: None 2024-04-16T20:18:44.134 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.134 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f49ae84bfb33fc-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.136 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:18:44.137 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:18:44.137 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.137 DEBUG:boto:Path: /yrjefh-56/obj155 2024-04-16T20:18:44.137 DEBUG:boto:Data: 2024-04-16T20:18:44.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.137 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.137 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.137 DEBUG:boto:Params: {} 2024-04-16T20:18:44.137 DEBUG:boto:Token: None 2024-04-16T20:18:44.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj155 2024-04-16T20:18:44.137 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WCDEHdTqoozDHS2uvqgoNcaTZdI= 2024-04-16T20:18:44.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WCDEHdTqoozDHS2uvqgoNcaTZdI='} 2024-04-16T20:18:44.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a99b9902a6493d62-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:44.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:18:44.143 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:18:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:44.194 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:44.263 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:18:44.263 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-56 2024-04-16T20:18:44.277 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 30 2024-04-16T20:18:44.352 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.353 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.353 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.353 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.353 DEBUG:boto:Data: 2024-04-16T20:18:44.353 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.353 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.353 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.353 DEBUG:boto:Params: {} 2024-04-16T20:18:44.353 DEBUG:boto:Token: None 2024-04-16T20:18:44.354 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.354 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000691a263d4321f8ae-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.357 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:18:44.357 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:18:44.357 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.358 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:18:44.358 DEBUG:boto:Data: 2024-04-16T20:18:44.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.358 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.358 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.358 DEBUG:boto:Params: {} 2024-04-16T20:18:44.358 DEBUG:boto:Token: None 2024-04-16T20:18:44.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj37 2024-04-16T20:18:44.358 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:w4O+Si8ThIDwPZgL4+BIU6eS+II= 2024-04-16T20:18:44.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w4O+Si8ThIDwPZgL4+BIU6eS+II='} 2024-04-16T20:18:44.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000532fee7b92e57e1f-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.364 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.364 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.364 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.364 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.364 DEBUG:boto:Data: 2024-04-16T20:18:44.364 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.364 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.364 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.364 DEBUG:boto:Params: {} 2024-04-16T20:18:44.365 DEBUG:boto:Token: None 2024-04-16T20:18:44.365 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.365 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7d30a06a12f450-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.368 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:18:44.368 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:18:44.368 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.368 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:18:44.368 DEBUG:boto:Data: 2024-04-16T20:18:44.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.368 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.368 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.369 DEBUG:boto:Params: {} 2024-04-16T20:18:44.369 DEBUG:boto:Token: None 2024-04-16T20:18:44.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj74 2024-04-16T20:18:44.369 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T9heMy18v8ettoUFH2U+C3dEsqc= 2024-04-16T20:18:44.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T9heMy18v8ettoUFH2U+C3dEsqc='} 2024-04-16T20:18:44.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd2ab59009788cc-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.415 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.415 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.415 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.415 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.415 DEBUG:boto:Data: 2024-04-16T20:18:44.415 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.415 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.415 DEBUG:boto:Params: {} 2024-04-16T20:18:44.416 DEBUG:boto:Token: None 2024-04-16T20:18:44.416 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f87f2c16e86fb7d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.419 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:18:44.419 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:18:44.419 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.419 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:18:44.419 DEBUG:boto:Data: 2024-04-16T20:18:44.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.420 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.420 DEBUG:boto:Params: {} 2024-04-16T20:18:44.420 DEBUG:boto:Token: None 2024-04-16T20:18:44.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj111 2024-04-16T20:18:44.420 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9SEB/Lajb7oetLDvlravJm/zD/4= 2024-04-16T20:18:44.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9SEB/Lajb7oetLDvlravJm/zD/4='} 2024-04-16T20:18:44.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f474251818941ca-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.426 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.426 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.426 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.426 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.426 DEBUG:boto:Data: 2024-04-16T20:18:44.426 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.426 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.427 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.427 DEBUG:boto:Params: {} 2024-04-16T20:18:44.427 DEBUG:boto:Token: None 2024-04-16T20:18:44.427 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082e54a71ab61eb1d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.430 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:18:44.430 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:18:44.430 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.431 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:18:44.431 DEBUG:boto:Data: 2024-04-16T20:18:44.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.433 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.433 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.433 DEBUG:boto:Params: {} 2024-04-16T20:18:44.433 DEBUG:boto:Token: None 2024-04-16T20:18:44.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj148 2024-04-16T20:18:44.434 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:r1qkG2JA59hykTvntpxT1sgEJn0= 2024-04-16T20:18:44.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r1qkG2JA59hykTvntpxT1sgEJn0='} 2024-04-16T20:18:44.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005332d2778e6558fa-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.479 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:18:44.479 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:18:44.479 DEBUG:boto:Method: HEAD 2024-04-16T20:18:44.479 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:18:44.479 DEBUG:boto:Data: 2024-04-16T20:18:44.479 DEBUG:boto:Headers: {} 2024-04-16T20:18:44.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.479 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.479 DEBUG:boto:Params: {} 2024-04-16T20:18:44.479 DEBUG:boto:Token: None 2024-04-16T20:18:44.480 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/ 2024-04-16T20:18:44.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T20:18:44.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T20:18:44.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eac3eb8e9e2fc5a2-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.483 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:18:44.483 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:18:44.483 DEBUG:boto:Method: PUT 2024-04-16T20:18:44.483 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:18:44.483 DEBUG:boto:Data: 2024-04-16T20:18:44.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:18:44.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:18:44.483 DEBUG:boto:Port: 8000 2024-04-16T20:18:44.483 DEBUG:boto:Params: {} 2024-04-16T20:18:44.484 DEBUG:boto:Token: None 2024-04-16T20:18:44.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:18:44 GMT /yrjefh-56/obj185 2024-04-16T20:18:44.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S3iCC41UyffVRidPoKTnNEO4ESE= 2024-04-16T20:18:44.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S3iCC41UyffVRidPoKTnNEO4ESE='} 2024-04-16T20:18:44.489 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024cf8c0058a3210d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:18:44.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-56 2024-04-16T20:18:44.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:18:44.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:18:44.490 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:18:44.552 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:18:44.552 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:18:44.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:44.626+0000 7f13ced33800 1 waiting to reach latest gen 6, on 3.. 2024-04-16T20:18:49.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:49.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T20:18:54.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:54.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T20:18:59.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:18:59.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T20:19:04.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:04.629+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T20:19:09.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:09.630+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:14.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:14.631+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:19.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:19.632+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:24.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:24.633+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:29.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:29.634+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:34.635+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:39.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:39.636+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:44.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:44.637+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:49.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:49.637+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:54.638+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:19:59.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:19:59.638+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:20:04.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:04.639+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T20:20:09.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:09.642+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:09.632 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:09.632 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:14.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:14.645+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:14.635 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:14.635 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:19.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:19.646+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:19.637 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:19.637 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:24.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:24.649+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:24.639 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:24.640 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:29.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:29.652+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:29.642 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:29.642 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:34.654+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:34.644 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:34.644 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:39.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:39.657+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:39.647 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:39.647 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:44.659+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:44.649 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:44.649 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:49.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:49.661+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:49.651 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:49.651 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:54.663+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T20:20:54.654 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.112.6, , , , , , , , , , , , , , , , , , , ] 2024-04-16T20:20:54.654 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:59.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:59.666+0000 7f13ced33800 1 bucket sync caught up with source: 2024-04-16T20:20:59.656 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:59.656 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:20:59.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:20:59.666+0000 7f13ced33800 0 bucket checkpoint complete 2024-04-16T20:20:59.683 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-56 2024-04-16T20:20:59.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:20:59.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:20:59.684 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:20:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:20:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:20:59.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:20:59.820+0000 7f8da1d2d800 1 bucket sync caught up with source: 2024-04-16T20:20:59.810 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.283.6, 00000000002.207.6, 00000000002.428.6, 00000000005.319.6, 00000000004.245.6, 00000000001.402.6, 00000000001.292.6, 00000000004.221.6, 00000000007.375.6, 00000000006.397.6] 2024-04-16T20:20:59.810 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.283.6, 00000000002.207.6, 00000000002.428.6, 00000000005.319.6, 00000000004.245.6, 00000000001.402.6, 00000000001.292.6, 00000000004.221.6, 00000000007.375.6, 00000000006.397.6] 2024-04-16T20:20:59.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:20:59.820+0000 7f8da1d2d800 0 bucket checkpoint complete 2024-04-16T20:20:59.832 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-56 zones={test-zone2, test-zone1} 2024-04-16T20:20:59.832 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.832 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.832 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.833 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:20:59.833 DEBUG:boto:Data: 2024-04-16T20:20:59.833 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.833 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.833 DEBUG:boto:Params: {} 2024-04-16T20:20:59.834 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:20:59.834 DEBUG:boto:Token: None 2024-04-16T20:20:59.834 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/ 2024-04-16T20:20:59.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18= 2024-04-16T20:20:59.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18=', 'Content-Length': '0'} 2024-04-16T20:20:59.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c98b166c6eb16015-00661eddab-6237-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.840 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.840 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.840 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.840 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:20:59.840 DEBUG:boto:Data: 2024-04-16T20:20:59.840 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.840 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.840 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.840 DEBUG:boto:Params: {} 2024-04-16T20:20:59.841 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:20:59.841 DEBUG:boto:Token: None 2024-04-16T20:20:59.841 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/ 2024-04-16T20:20:59.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18= 2024-04-16T20:20:59.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18=', 'Content-Length': '0'} 2024-04-16T20:20:59.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce973ed79ce3d530-00661eddab-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.846 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:20:59.846 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.846 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.846 DEBUG:boto:path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.846 DEBUG:boto:auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.846 DEBUG:boto:Method: GET 2024-04-16T20:20:59.846 DEBUG:boto:Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.846 DEBUG:boto:Data: 2024-04-16T20:20:59.846 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.847 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.847 DEBUG:boto:Params: {} 2024-04-16T20:20:59.847 DEBUG:boto:Token: None 2024-04-16T20:20:59.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/?versions 2024-04-16T20:20:59.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T20:20:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T20:20:59.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000407ad93180aab46-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.851 DEBUG:boto:b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:20:59.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:20:59.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:20:59.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:20:59.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:20:59.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:20:59.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:20:59.856 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:20:59.856 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.856 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.856 DEBUG:boto:path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.856 DEBUG:boto:auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.856 DEBUG:boto:Method: GET 2024-04-16T20:20:59.857 DEBUG:boto:Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.857 DEBUG:boto:Data: 2024-04-16T20:20:59.857 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.857 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.857 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.857 DEBUG:boto:Params: {} 2024-04-16T20:20:59.857 DEBUG:boto:Token: None 2024-04-16T20:20:59.857 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/?versions 2024-04-16T20:20:59.857 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T20:20:59.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T20:20:59.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f39166cbcfaa42f-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.861 DEBUG:boto:b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:20:59.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-16T20:20:59.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-16T20:20:59.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-16T20:20:59.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-16T20:20:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-16T20:20:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-16T20:20:59.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-16T20:20:59.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-16T20:20:59.865 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.865 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.865 DEBUG:boto:path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.865 DEBUG:boto:auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.865 DEBUG:boto:Method: GET 2024-04-16T20:20:59.865 DEBUG:boto:Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.865 DEBUG:boto:Data: 2024-04-16T20:20:59.865 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.865 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.866 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.866 DEBUG:boto:Params: {} 2024-04-16T20:20:59.866 DEBUG:boto:Token: None 2024-04-16T20:20:59.866 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/?versions 2024-04-16T20:20:59.866 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T20:20:59.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T20:20:59.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4cad3158ca7cb59-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.869 DEBUG:boto:b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.871 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.871 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.871 DEBUG:boto:path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.871 DEBUG:boto:auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.871 DEBUG:boto:Method: GET 2024-04-16T20:20:59.871 DEBUG:boto:Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T20:20:59.871 DEBUG:boto:Data: 2024-04-16T20:20:59.871 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.872 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.872 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.872 DEBUG:boto:Params: {} 2024-04-16T20:20:59.872 DEBUG:boto:Token: None 2024-04-16T20:20:59.872 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/?versions 2024-04-16T20:20:59.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T20:20:59.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T20:20:59.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ffb063212df57a-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.876 DEBUG:boto:b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:20:59.877 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.877 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.878 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.878 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.878 DEBUG:boto:Method: GET 2024-04-16T20:20:59.878 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.878 DEBUG:boto:Data: 2024-04-16T20:20:59.878 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.878 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.878 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.878 DEBUG:boto:Params: {} 2024-04-16T20:20:59.878 DEBUG:boto:Token: None 2024-04-16T20:20:59.879 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.879 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T20:20:59.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T20:20:59.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047aa17df41635390-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.881 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.881 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.881 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.881 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.881 DEBUG:boto:Method: GET 2024-04-16T20:20:59.881 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.882 DEBUG:boto:Data: 2024-04-16T20:20:59.882 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.882 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.882 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.882 DEBUG:boto:Params: {} 2024-04-16T20:20:59.882 DEBUG:boto:Token: None 2024-04-16T20:20:59.882 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.882 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T20:20:59.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T20:20:59.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c709c72d8afacb-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.884 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.884 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.884 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.885 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.885 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.885 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.885 DEBUG:boto:Data: 2024-04-16T20:20:59.885 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.885 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.885 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.885 DEBUG:boto:Params: {} 2024-04-16T20:20:59.885 DEBUG:boto:Token: None 2024-04-16T20:20:59.885 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.886 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs= 2024-04-16T20:20:59.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs=', 'Content-Length': '0'} 2024-04-16T20:20:59.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023ac602c050af89b-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.888 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.888 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.888 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.888 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.888 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.888 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.888 DEBUG:boto:Data: 2024-04-16T20:20:59.888 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.889 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.889 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.889 DEBUG:boto:Params: {} 2024-04-16T20:20:59.889 DEBUG:boto:Token: None 2024-04-16T20:20:59.889 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.889 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs= 2024-04-16T20:20:59.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs=', 'Content-Length': '0'} 2024-04-16T20:20:59.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b6e6a576e71939-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:20:59.891 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.891 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.891 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.891 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.891 DEBUG:boto:Method: GET 2024-04-16T20:20:59.891 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.891 DEBUG:boto:Data: 2024-04-16T20:20:59.892 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.892 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.892 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.892 DEBUG:boto:Params: {} 2024-04-16T20:20:59.892 DEBUG:boto:Token: None 2024-04-16T20:20:59.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T20:20:59.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T20:20:59.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc936e6f7424cff1-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.935 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.935 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.935 DEBUG:boto:path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.935 DEBUG:boto:auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.935 DEBUG:boto:Method: GET 2024-04-16T20:20:59.936 DEBUG:boto:Path: /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.936 DEBUG:boto:Data: 2024-04-16T20:20:59.936 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.936 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.936 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.936 DEBUG:boto:Params: {} 2024-04-16T20:20:59.936 DEBUG:boto:Token: None 2024-04-16T20:20:59.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11?versionId=null 2024-04-16T20:20:59.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T20:20:59.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T20:20:59.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b6095a8b68c120-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.938 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.939 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.939 DEBUG:boto:path=/yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.939 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.939 DEBUG:boto:Method: GET 2024-04-16T20:20:59.939 DEBUG:boto:Path: /yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.939 DEBUG:boto:Data: 2024-04-16T20:20:59.939 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.939 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.939 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.939 DEBUG:boto:Params: {} 2024-04-16T20:20:59.939 DEBUG:boto:Token: None 2024-04-16T20:20:59.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/ 2024-04-16T20:20:59.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU= 2024-04-16T20:20:59.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU=', 'Content-Length': '0'} 2024-04-16T20:20:59.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015278e1adb273c27-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.942 DEBUG:boto:b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.943 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:20:59.943 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:20:59.943 DEBUG:boto:path=/yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.943 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.943 DEBUG:boto:Method: GET 2024-04-16T20:20:59.944 DEBUG:boto:Path: /yrjefh-56/?prefix=obj11 2024-04-16T20:20:59.944 DEBUG:boto:Data: 2024-04-16T20:20:59.944 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.944 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.944 DEBUG:boto:Params: {} 2024-04-16T20:20:59.944 DEBUG:boto:Token: None 2024-04-16T20:20:59.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/ 2024-04-16T20:20:59.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU= 2024-04-16T20:20:59.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU=', 'Content-Length': '0'} 2024-04-16T20:20:59.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f58084f1bb37748-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.948 DEBUG:boto:b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:20:59.948 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.949 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.949 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.949 DEBUG:boto:Path: /yrjefh-56/obj11 2024-04-16T20:20:59.949 DEBUG:boto:Data: 2024-04-16T20:20:59.949 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.949 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.949 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.949 DEBUG:boto:Params: {} 2024-04-16T20:20:59.949 DEBUG:boto:Token: None 2024-04-16T20:20:59.949 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11 2024-04-16T20:20:59.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk= 2024-04-16T20:20:59.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk=', 'Content-Length': '0'} 2024-04-16T20:20:59.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc17037ac582bde-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.952 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.952 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.952 DEBUG:boto:Method: HEAD 2024-04-16T20:20:59.952 DEBUG:boto:Path: /yrjefh-56/obj11 2024-04-16T20:20:59.952 DEBUG:boto:Data: 2024-04-16T20:20:59.952 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.953 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.953 DEBUG:boto:Params: {} 2024-04-16T20:20:59.953 DEBUG:boto:Token: None 2024-04-16T20:20:59.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11 2024-04-16T20:20:59.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk= 2024-04-16T20:20:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk=', 'Content-Length': '0'} 2024-04-16T20:20:59.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f8f0d149344b63-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:20:59.955 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.955 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.955 DEBUG:boto:Method: GET 2024-04-16T20:20:59.955 DEBUG:boto:Path: /yrjefh-56/obj11 2024-04-16T20:20:59.955 DEBUG:boto:Data: 2024-04-16T20:20:59.955 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.955 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:20:59.955 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.956 DEBUG:boto:Params: {} 2024-04-16T20:20:59.956 DEBUG:boto:Token: None 2024-04-16T20:20:59.956 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11 2024-04-16T20:20:59.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc= 2024-04-16T20:20:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc=', 'Content-Length': '0'} 2024-04-16T20:20:59.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002efc2ba11f38b307-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:20:59.958 DEBUG:boto:path=/yrjefh-56/obj11 2024-04-16T20:20:59.958 DEBUG:boto:auth_path=/yrjefh-56/obj11 2024-04-16T20:20:59.958 DEBUG:boto:Method: GET 2024-04-16T20:20:59.958 DEBUG:boto:Path: /yrjefh-56/obj11 2024-04-16T20:20:59.958 DEBUG:boto:Data: 2024-04-16T20:20:59.959 DEBUG:boto:Headers: {} 2024-04-16T20:20:59.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:20:59.959 DEBUG:boto:Port: 8000 2024-04-16T20:20:59.959 DEBUG:boto:Params: {} 2024-04-16T20:20:59.959 DEBUG:boto:Token: None 2024-04-16T20:20:59.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:20:59 GMT /yrjefh-56/obj11 2024-04-16T20:20:59.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc= 2024-04-16T20:20:59.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc=', 'Content-Length': '0'} 2024-04-16T20:20:59.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fac03ceadae7629-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.002 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.002 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.003 DEBUG:boto:path=/yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.003 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.003 DEBUG:boto:Method: GET 2024-04-16T20:21:00.003 DEBUG:boto:Path: /yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.003 DEBUG:boto:Data: 2024-04-16T20:21:00.003 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.003 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.004 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.004 DEBUG:boto:Params: {} 2024-04-16T20:21:00.004 DEBUG:boto:Token: None 2024-04-16T20:21:00.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058c10e37f21625d2-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.007 DEBUG:boto:b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.008 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.008 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.009 DEBUG:boto:path=/yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.009 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.009 DEBUG:boto:Method: GET 2024-04-16T20:21:00.009 DEBUG:boto:Path: /yrjefh-56/?prefix=obj11 2024-04-16T20:21:00.009 DEBUG:boto:Data: 2024-04-16T20:21:00.010 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.010 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.010 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.010 DEBUG:boto:Params: {} 2024-04-16T20:21:00.010 DEBUG:boto:Token: None 2024-04-16T20:21:00.010 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.010 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1304e5a5033402-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.014 DEBUG:boto:b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:21:00.014 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.014 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.015 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.015 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.015 DEBUG:boto:Method: GET 2024-04-16T20:21:00.015 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.015 DEBUG:boto:Data: 2024-04-16T20:21:00.015 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.015 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.015 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.015 DEBUG:boto:Params: {} 2024-04-16T20:21:00.015 DEBUG:boto:Token: None 2024-04-16T20:21:00.016 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.016 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T20:21:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T20:21:00.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432d3e4a11f52ae4-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.058 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.059 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.059 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.059 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.059 DEBUG:boto:Method: GET 2024-04-16T20:21:00.059 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.059 DEBUG:boto:Data: 2024-04-16T20:21:00.059 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.059 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.059 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.059 DEBUG:boto:Params: {} 2024-04-16T20:21:00.059 DEBUG:boto:Token: None 2024-04-16T20:21:00.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T20:21:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T20:21:00.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1697422beed44e1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.062 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.062 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.062 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.062 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.062 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.062 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.062 DEBUG:boto:Data: 2024-04-16T20:21:00.062 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.063 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.063 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.063 DEBUG:boto:Params: {} 2024-04-16T20:21:00.063 DEBUG:boto:Token: None 2024-04-16T20:21:00.063 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs= 2024-04-16T20:21:00.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs=', 'Content-Length': '0'} 2024-04-16T20:21:00.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066aa57bfd8260d67-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.065 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.065 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.065 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.065 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.066 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.066 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.066 DEBUG:boto:Data: 2024-04-16T20:21:00.066 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.066 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.066 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.066 DEBUG:boto:Params: {} 2024-04-16T20:21:00.066 DEBUG:boto:Token: None 2024-04-16T20:21:00.066 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.066 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs= 2024-04-16T20:21:00.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs=', 'Content-Length': '0'} 2024-04-16T20:21:00.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b64d73d454b707f-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:21:00.069 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.069 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.069 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.069 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.069 DEBUG:boto:Method: GET 2024-04-16T20:21:00.069 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.069 DEBUG:boto:Data: 2024-04-16T20:21:00.069 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.069 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.069 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.069 DEBUG:boto:Params: {} 2024-04-16T20:21:00.070 DEBUG:boto:Token: None 2024-04-16T20:21:00.070 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.070 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T20:21:00.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T20:21:00.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7045700a0019b3-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.072 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.072 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.072 DEBUG:boto:path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.072 DEBUG:boto:auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.072 DEBUG:boto:Method: GET 2024-04-16T20:21:00.072 DEBUG:boto:Path: /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.072 DEBUG:boto:Data: 2024-04-16T20:21:00.073 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.073 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.073 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.073 DEBUG:boto:Params: {} 2024-04-16T20:21:00.073 DEBUG:boto:Token: None 2024-04-16T20:21:00.073 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111?versionId=null 2024-04-16T20:21:00.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T20:21:00.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T20:21:00.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003036531dbc3cd3c6-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.075 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.075 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.075 DEBUG:boto:path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.075 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.075 DEBUG:boto:Method: GET 2024-04-16T20:21:00.076 DEBUG:boto:Path: /yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.076 DEBUG:boto:Data: 2024-04-16T20:21:00.076 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.076 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.076 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.076 DEBUG:boto:Params: {} 2024-04-16T20:21:00.076 DEBUG:boto:Token: None 2024-04-16T20:21:00.076 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb93b50f6e5e71d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.079 DEBUG:boto:b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.079 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.080 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.080 DEBUG:boto:path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.080 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.080 DEBUG:boto:Method: GET 2024-04-16T20:21:00.080 DEBUG:boto:Path: /yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.080 DEBUG:boto:Data: 2024-04-16T20:21:00.080 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.080 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.080 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.080 DEBUG:boto:Params: {} 2024-04-16T20:21:00.081 DEBUG:boto:Token: None 2024-04-16T20:21:00.081 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.081 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ad515112b9c5a0-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.084 DEBUG:boto:b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.084 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.085 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.085 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.085 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:21:00.085 DEBUG:boto:Data: 2024-04-16T20:21:00.085 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.085 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.085 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.085 DEBUG:boto:Params: {} 2024-04-16T20:21:00.085 DEBUG:boto:Token: None 2024-04-16T20:21:00.085 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111 2024-04-16T20:21:00.086 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE= 2024-04-16T20:21:00.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE=', 'Content-Length': '0'} 2024-04-16T20:21:00.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fe14668de6120c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.088 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.088 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.088 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.088 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:21:00.088 DEBUG:boto:Data: 2024-04-16T20:21:00.088 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.088 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.088 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.088 DEBUG:boto:Params: {} 2024-04-16T20:21:00.089 DEBUG:boto:Token: None 2024-04-16T20:21:00.089 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111 2024-04-16T20:21:00.089 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE= 2024-04-16T20:21:00.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE=', 'Content-Length': '0'} 2024-04-16T20:21:00.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3a189b976468c1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-16T20:21:00.091 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.091 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.091 DEBUG:boto:Method: GET 2024-04-16T20:21:00.091 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:21:00.091 DEBUG:boto:Data: 2024-04-16T20:21:00.091 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.091 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.091 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.091 DEBUG:boto:Params: {} 2024-04-16T20:21:00.092 DEBUG:boto:Token: None 2024-04-16T20:21:00.092 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111 2024-04-16T20:21:00.092 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0= 2024-04-16T20:21:00.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0=', 'Content-Length': '0'} 2024-04-16T20:21:00.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096537f26ae361c03-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.134 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:00.134 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:00.135 DEBUG:boto:Method: GET 2024-04-16T20:21:00.135 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:21:00.135 DEBUG:boto:Data: 2024-04-16T20:21:00.135 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.135 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.135 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.135 DEBUG:boto:Params: {} 2024-04-16T20:21:00.135 DEBUG:boto:Token: None 2024-04-16T20:21:00.135 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj111 2024-04-16T20:21:00.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0= 2024-04-16T20:21:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0=', 'Content-Length': '0'} 2024-04-16T20:21:00.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9060f59959e0758-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.137 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.137 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.138 DEBUG:boto:path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.138 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.138 DEBUG:boto:Method: GET 2024-04-16T20:21:00.138 DEBUG:boto:Path: /yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.138 DEBUG:boto:Data: 2024-04-16T20:21:00.138 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.138 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.138 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.138 DEBUG:boto:Params: {} 2024-04-16T20:21:00.138 DEBUG:boto:Token: None 2024-04-16T20:21:00.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.139 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d2a60619da6e5c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.141 DEBUG:boto:b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.142 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.142 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.142 DEBUG:boto:path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.142 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.142 DEBUG:boto:Method: GET 2024-04-16T20:21:00.142 DEBUG:boto:Path: /yrjefh-56/?prefix=obj111 2024-04-16T20:21:00.142 DEBUG:boto:Data: 2024-04-16T20:21:00.142 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.143 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.143 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.143 DEBUG:boto:Params: {} 2024-04-16T20:21:00.143 DEBUG:boto:Token: None 2024-04-16T20:21:00.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a44b4474048665-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.147 DEBUG:boto:b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:21:00.147 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.147 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.147 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.147 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.148 DEBUG:boto:Method: GET 2024-04-16T20:21:00.148 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.148 DEBUG:boto:Data: 2024-04-16T20:21:00.148 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.148 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.148 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.148 DEBUG:boto:Params: {} 2024-04-16T20:21:00.148 DEBUG:boto:Token: None 2024-04-16T20:21:00.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T20:21:00.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T20:21:00.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066bf54cd9bb55d89-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.151 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.151 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.151 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.151 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.151 DEBUG:boto:Method: GET 2024-04-16T20:21:00.151 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.151 DEBUG:boto:Data: 2024-04-16T20:21:00.151 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.151 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.152 DEBUG:boto:Params: {} 2024-04-16T20:21:00.152 DEBUG:boto:Token: None 2024-04-16T20:21:00.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T20:21:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T20:21:00.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906469f51f415f78-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.154 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.154 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.154 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.154 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.154 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.154 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.154 DEBUG:boto:Data: 2024-04-16T20:21:00.154 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.155 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.155 DEBUG:boto:Params: {} 2024-04-16T20:21:00.155 DEBUG:boto:Token: None 2024-04-16T20:21:00.155 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg= 2024-04-16T20:21:00.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg=', 'Content-Length': '0'} 2024-04-16T20:21:00.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027208c0e7714931f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.157 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.157 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.157 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.157 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.157 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.158 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.158 DEBUG:boto:Data: 2024-04-16T20:21:00.158 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.158 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.158 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.158 DEBUG:boto:Params: {} 2024-04-16T20:21:00.158 DEBUG:boto:Token: None 2024-04-16T20:21:00.158 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg= 2024-04-16T20:21:00.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg=', 'Content-Length': '0'} 2024-04-16T20:21:00.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f985e673a86097-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:21:00.160 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.160 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.161 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.161 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.161 DEBUG:boto:Method: GET 2024-04-16T20:21:00.161 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.161 DEBUG:boto:Data: 2024-04-16T20:21:00.161 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.161 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.161 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.161 DEBUG:boto:Params: {} 2024-04-16T20:21:00.161 DEBUG:boto:Token: None 2024-04-16T20:21:00.162 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.162 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T20:21:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T20:21:00.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce1f9329bd89806-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.164 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.164 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.164 DEBUG:boto:path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.164 DEBUG:boto:auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.165 DEBUG:boto:Method: GET 2024-04-16T20:21:00.165 DEBUG:boto:Path: /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.165 DEBUG:boto:Data: 2024-04-16T20:21:00.165 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.165 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.165 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.165 DEBUG:boto:Params: {} 2024-04-16T20:21:00.165 DEBUG:boto:Token: None 2024-04-16T20:21:00.165 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115?versionId=null 2024-04-16T20:21:00.165 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T20:21:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T20:21:00.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728f152c5533a21d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.167 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.167 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.168 DEBUG:boto:path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.168 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.168 DEBUG:boto:Method: GET 2024-04-16T20:21:00.168 DEBUG:boto:Path: /yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.168 DEBUG:boto:Data: 2024-04-16T20:21:00.168 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.168 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.168 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.168 DEBUG:boto:Params: {} 2024-04-16T20:21:00.168 DEBUG:boto:Token: None 2024-04-16T20:21:00.169 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000225bfa26ff98ead-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.172 DEBUG:boto:b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.172 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.172 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.172 DEBUG:boto:path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.172 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.172 DEBUG:boto:Method: GET 2024-04-16T20:21:00.172 DEBUG:boto:Path: /yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.172 DEBUG:boto:Data: 2024-04-16T20:21:00.173 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.173 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.173 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.173 DEBUG:boto:Params: {} 2024-04-16T20:21:00.173 DEBUG:boto:Token: None 2024-04-16T20:21:00.173 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.173 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f247904ca78aa643-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.176 DEBUG:boto:b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.176 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.177 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.177 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.177 DEBUG:boto:Path: /yrjefh-56/obj115 2024-04-16T20:21:00.177 DEBUG:boto:Data: 2024-04-16T20:21:00.177 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.177 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.177 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.177 DEBUG:boto:Params: {} 2024-04-16T20:21:00.177 DEBUG:boto:Token: None 2024-04-16T20:21:00.178 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115 2024-04-16T20:21:00.178 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA= 2024-04-16T20:21:00.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA=', 'Content-Length': '0'} 2024-04-16T20:21:00.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d71921e209c455b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.180 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.180 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.180 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.180 DEBUG:boto:Path: /yrjefh-56/obj115 2024-04-16T20:21:00.180 DEBUG:boto:Data: 2024-04-16T20:21:00.180 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.180 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.180 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.180 DEBUG:boto:Params: {} 2024-04-16T20:21:00.181 DEBUG:boto:Token: None 2024-04-16T20:21:00.181 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115 2024-04-16T20:21:00.181 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA= 2024-04-16T20:21:00.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA=', 'Content-Length': '0'} 2024-04-16T20:21:00.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b08dace58c3cd9-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-16T20:21:00.183 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.183 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.183 DEBUG:boto:Method: GET 2024-04-16T20:21:00.183 DEBUG:boto:Path: /yrjefh-56/obj115 2024-04-16T20:21:00.183 DEBUG:boto:Data: 2024-04-16T20:21:00.183 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.183 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.183 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.184 DEBUG:boto:Params: {} 2024-04-16T20:21:00.184 DEBUG:boto:Token: None 2024-04-16T20:21:00.184 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115 2024-04-16T20:21:00.184 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk= 2024-04-16T20:21:00.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk=', 'Content-Length': '0'} 2024-04-16T20:21:00.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aaa5e25836e1cfe-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.226 DEBUG:boto:path=/yrjefh-56/obj115 2024-04-16T20:21:00.226 DEBUG:boto:auth_path=/yrjefh-56/obj115 2024-04-16T20:21:00.227 DEBUG:boto:Method: GET 2024-04-16T20:21:00.227 DEBUG:boto:Path: /yrjefh-56/obj115 2024-04-16T20:21:00.227 DEBUG:boto:Data: 2024-04-16T20:21:00.227 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.227 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.227 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.227 DEBUG:boto:Params: {} 2024-04-16T20:21:00.227 DEBUG:boto:Token: None 2024-04-16T20:21:00.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj115 2024-04-16T20:21:00.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk= 2024-04-16T20:21:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk=', 'Content-Length': '0'} 2024-04-16T20:21:00.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c16dec2329afeb8-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.230 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.230 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.230 DEBUG:boto:path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.230 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.230 DEBUG:boto:Method: GET 2024-04-16T20:21:00.230 DEBUG:boto:Path: /yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.231 DEBUG:boto:Data: 2024-04-16T20:21:00.231 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.231 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.231 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.231 DEBUG:boto:Params: {} 2024-04-16T20:21:00.231 DEBUG:boto:Token: None 2024-04-16T20:21:00.231 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.231 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb07fc29c671d0a-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.234 DEBUG:boto:b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.235 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.235 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.235 DEBUG:boto:path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.235 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.235 DEBUG:boto:Method: GET 2024-04-16T20:21:00.235 DEBUG:boto:Path: /yrjefh-56/?prefix=obj115 2024-04-16T20:21:00.235 DEBUG:boto:Data: 2024-04-16T20:21:00.235 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.235 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.235 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.235 DEBUG:boto:Params: {} 2024-04-16T20:21:00.236 DEBUG:boto:Token: None 2024-04-16T20:21:00.236 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.236 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ddfea5b17a69f2f-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.239 DEBUG:boto:b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:21:00.240 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.240 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.240 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.240 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.240 DEBUG:boto:Method: GET 2024-04-16T20:21:00.240 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.240 DEBUG:boto:Data: 2024-04-16T20:21:00.240 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.240 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.240 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.241 DEBUG:boto:Params: {} 2024-04-16T20:21:00.241 DEBUG:boto:Token: None 2024-04-16T20:21:00.241 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.241 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T20:21:00.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T20:21:00.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b210d8fa97e9abbf-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.243 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.244 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.244 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.244 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.244 DEBUG:boto:Method: GET 2024-04-16T20:21:00.244 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.244 DEBUG:boto:Data: 2024-04-16T20:21:00.244 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.244 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.244 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.244 DEBUG:boto:Params: {} 2024-04-16T20:21:00.244 DEBUG:boto:Token: None 2024-04-16T20:21:00.245 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.245 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T20:21:00.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T20:21:00.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b841d71af4c892-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.247 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.247 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.247 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.247 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.247 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.247 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.247 DEBUG:boto:Data: 2024-04-16T20:21:00.248 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.248 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.248 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.248 DEBUG:boto:Params: {} 2024-04-16T20:21:00.248 DEBUG:boto:Token: None 2024-04-16T20:21:00.248 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.248 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU= 2024-04-16T20:21:00.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU=', 'Content-Length': '0'} 2024-04-16T20:21:00.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a54daace8701671-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.250 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.250 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.250 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.250 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.251 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.251 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.251 DEBUG:boto:Data: 2024-04-16T20:21:00.251 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.251 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.251 DEBUG:boto:Params: {} 2024-04-16T20:21:00.251 DEBUG:boto:Token: None 2024-04-16T20:21:00.251 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.251 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU= 2024-04-16T20:21:00.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU=', 'Content-Length': '0'} 2024-04-16T20:21:00.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2937bef9733899-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:21:00.253 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.254 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.254 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.254 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.254 DEBUG:boto:Method: GET 2024-04-16T20:21:00.254 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.254 DEBUG:boto:Data: 2024-04-16T20:21:00.254 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.254 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.254 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.254 DEBUG:boto:Params: {} 2024-04-16T20:21:00.255 DEBUG:boto:Token: None 2024-04-16T20:21:00.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.255 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T20:21:00.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T20:21:00.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103bbe72c3ba388b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.302 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.302 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.303 DEBUG:boto:path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.303 DEBUG:boto:auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.303 DEBUG:boto:Method: GET 2024-04-16T20:21:00.303 DEBUG:boto:Path: /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.303 DEBUG:boto:Data: 2024-04-16T20:21:00.303 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.303 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.303 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.303 DEBUG:boto:Params: {} 2024-04-16T20:21:00.303 DEBUG:boto:Token: None 2024-04-16T20:21:00.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116?versionId=null 2024-04-16T20:21:00.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T20:21:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T20:21:00.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b782fa2111f614e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.306 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.306 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.306 DEBUG:boto:path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.306 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.306 DEBUG:boto:Method: GET 2024-04-16T20:21:00.306 DEBUG:boto:Path: /yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.306 DEBUG:boto:Data: 2024-04-16T20:21:00.307 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.307 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.307 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.307 DEBUG:boto:Params: {} 2024-04-16T20:21:00.307 DEBUG:boto:Token: None 2024-04-16T20:21:00.307 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.307 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc0a5414242b8d7-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.310 DEBUG:boto:b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.310 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.310 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.310 DEBUG:boto:path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.311 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.311 DEBUG:boto:Method: GET 2024-04-16T20:21:00.311 DEBUG:boto:Path: /yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.311 DEBUG:boto:Data: 2024-04-16T20:21:00.311 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.311 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.311 DEBUG:boto:Params: {} 2024-04-16T20:21:00.311 DEBUG:boto:Token: None 2024-04-16T20:21:00.311 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.311 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d6d9874d6eb1dc-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.315 DEBUG:boto:b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.316 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.316 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.316 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.316 DEBUG:boto:Path: /yrjefh-56/obj116 2024-04-16T20:21:00.316 DEBUG:boto:Data: 2024-04-16T20:21:00.316 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.316 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.316 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.316 DEBUG:boto:Params: {} 2024-04-16T20:21:00.316 DEBUG:boto:Token: None 2024-04-16T20:21:00.316 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116 2024-04-16T20:21:00.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y= 2024-04-16T20:21:00.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y=', 'Content-Length': '0'} 2024-04-16T20:21:00.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586143ed51e18c67-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.319 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.319 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.319 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.319 DEBUG:boto:Path: /yrjefh-56/obj116 2024-04-16T20:21:00.319 DEBUG:boto:Data: 2024-04-16T20:21:00.319 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.320 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.320 DEBUG:boto:Params: {} 2024-04-16T20:21:00.320 DEBUG:boto:Token: None 2024-04-16T20:21:00.320 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116 2024-04-16T20:21:00.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y= 2024-04-16T20:21:00.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y=', 'Content-Length': '0'} 2024-04-16T20:21:00.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9fc5de1c70149d3-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-16T20:21:00.322 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.323 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.323 DEBUG:boto:Method: GET 2024-04-16T20:21:00.323 DEBUG:boto:Path: /yrjefh-56/obj116 2024-04-16T20:21:00.323 DEBUG:boto:Data: 2024-04-16T20:21:00.323 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.323 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.323 DEBUG:boto:Params: {} 2024-04-16T20:21:00.323 DEBUG:boto:Token: None 2024-04-16T20:21:00.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116 2024-04-16T20:21:00.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw= 2024-04-16T20:21:00.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw=', 'Content-Length': '0'} 2024-04-16T20:21:00.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a52eb2c4a518614-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.326 DEBUG:boto:path=/yrjefh-56/obj116 2024-04-16T20:21:00.326 DEBUG:boto:auth_path=/yrjefh-56/obj116 2024-04-16T20:21:00.326 DEBUG:boto:Method: GET 2024-04-16T20:21:00.326 DEBUG:boto:Path: /yrjefh-56/obj116 2024-04-16T20:21:00.326 DEBUG:boto:Data: 2024-04-16T20:21:00.326 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.326 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.326 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.327 DEBUG:boto:Params: {} 2024-04-16T20:21:00.327 DEBUG:boto:Token: None 2024-04-16T20:21:00.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj116 2024-04-16T20:21:00.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw= 2024-04-16T20:21:00.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw=', 'Content-Length': '0'} 2024-04-16T20:21:00.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f434af938d21ccf-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.370 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.371 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.371 DEBUG:boto:path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.371 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.371 DEBUG:boto:Method: GET 2024-04-16T20:21:00.371 DEBUG:boto:Path: /yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.371 DEBUG:boto:Data: 2024-04-16T20:21:00.371 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.371 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.371 DEBUG:boto:Params: {} 2024-04-16T20:21:00.371 DEBUG:boto:Token: None 2024-04-16T20:21:00.372 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008096e9ece555f34d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.374 DEBUG:boto:b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.375 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.375 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.375 DEBUG:boto:path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.375 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.375 DEBUG:boto:Method: GET 2024-04-16T20:21:00.375 DEBUG:boto:Path: /yrjefh-56/?prefix=obj116 2024-04-16T20:21:00.375 DEBUG:boto:Data: 2024-04-16T20:21:00.375 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.375 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.375 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.376 DEBUG:boto:Params: {} 2024-04-16T20:21:00.376 DEBUG:boto:Token: None 2024-04-16T20:21:00.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548a581ebb16b346-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.380 DEBUG:boto:b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:21:00.380 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.380 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.380 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.380 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.381 DEBUG:boto:Method: GET 2024-04-16T20:21:00.381 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.381 DEBUG:boto:Data: 2024-04-16T20:21:00.381 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.381 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.381 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.381 DEBUG:boto:Params: {} 2024-04-16T20:21:00.381 DEBUG:boto:Token: None 2024-04-16T20:21:00.381 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T20:21:00.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T20:21:00.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab60ba2b893cd00d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.426 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.426 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.427 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.427 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.427 DEBUG:boto:Method: GET 2024-04-16T20:21:00.427 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.427 DEBUG:boto:Data: 2024-04-16T20:21:00.427 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.427 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.427 DEBUG:boto:Params: {} 2024-04-16T20:21:00.427 DEBUG:boto:Token: None 2024-04-16T20:21:00.428 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T20:21:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T20:21:00.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5045f78d028168-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.430 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.430 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.430 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.430 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.430 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.430 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.430 DEBUG:boto:Data: 2024-04-16T20:21:00.430 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.430 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.431 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.431 DEBUG:boto:Params: {} 2024-04-16T20:21:00.431 DEBUG:boto:Token: None 2024-04-16T20:21:00.431 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo= 2024-04-16T20:21:00.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo=', 'Content-Length': '0'} 2024-04-16T20:21:00.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4bc835095ba0b1-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.433 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.433 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.433 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.433 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.434 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.434 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.434 DEBUG:boto:Data: 2024-04-16T20:21:00.434 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.434 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.434 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.434 DEBUG:boto:Params: {} 2024-04-16T20:21:00.434 DEBUG:boto:Token: None 2024-04-16T20:21:00.434 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.434 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo= 2024-04-16T20:21:00.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo=', 'Content-Length': '0'} 2024-04-16T20:21:00.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ef7c3261c86b62-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:21:00.436 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.436 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.437 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.437 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.437 DEBUG:boto:Method: GET 2024-04-16T20:21:00.437 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.437 DEBUG:boto:Data: 2024-04-16T20:21:00.437 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.437 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.437 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.437 DEBUG:boto:Params: {} 2024-04-16T20:21:00.437 DEBUG:boto:Token: None 2024-04-16T20:21:00.437 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.438 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T20:21:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T20:21:00.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004971eaa2860217f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.440 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.440 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.440 DEBUG:boto:path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.440 DEBUG:boto:auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.440 DEBUG:boto:Method: GET 2024-04-16T20:21:00.440 DEBUG:boto:Path: /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.440 DEBUG:boto:Data: 2024-04-16T20:21:00.440 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.441 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.441 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.441 DEBUG:boto:Params: {} 2024-04-16T20:21:00.441 DEBUG:boto:Token: None 2024-04-16T20:21:00.441 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124?versionId=null 2024-04-16T20:21:00.441 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T20:21:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T20:21:00.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091668cb981ab84b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.443 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.443 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.443 DEBUG:boto:path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.443 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.443 DEBUG:boto:Method: GET 2024-04-16T20:21:00.444 DEBUG:boto:Path: /yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.444 DEBUG:boto:Data: 2024-04-16T20:21:00.444 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.444 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.444 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.444 DEBUG:boto:Params: {} 2024-04-16T20:21:00.444 DEBUG:boto:Token: None 2024-04-16T20:21:00.444 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.444 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a2bc74dd7ac00c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.447 DEBUG:boto:b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.447 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.448 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.448 DEBUG:boto:path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.448 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.448 DEBUG:boto:Method: GET 2024-04-16T20:21:00.448 DEBUG:boto:Path: /yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.448 DEBUG:boto:Data: 2024-04-16T20:21:00.448 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.448 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.448 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.448 DEBUG:boto:Params: {} 2024-04-16T20:21:00.448 DEBUG:boto:Token: None 2024-04-16T20:21:00.449 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.449 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d059ffb847e2bb09-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.452 DEBUG:boto:b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.452 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.452 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.453 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.453 DEBUG:boto:Path: /yrjefh-56/obj124 2024-04-16T20:21:00.453 DEBUG:boto:Data: 2024-04-16T20:21:00.453 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.453 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.453 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.453 DEBUG:boto:Params: {} 2024-04-16T20:21:00.453 DEBUG:boto:Token: None 2024-04-16T20:21:00.453 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124 2024-04-16T20:21:00.453 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0= 2024-04-16T20:21:00.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0=', 'Content-Length': '0'} 2024-04-16T20:21:00.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002805035aa22af401-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.455 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.456 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.456 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.456 DEBUG:boto:Path: /yrjefh-56/obj124 2024-04-16T20:21:00.456 DEBUG:boto:Data: 2024-04-16T20:21:00.456 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.456 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.456 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.456 DEBUG:boto:Params: {} 2024-04-16T20:21:00.456 DEBUG:boto:Token: None 2024-04-16T20:21:00.456 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124 2024-04-16T20:21:00.457 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0= 2024-04-16T20:21:00.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0=', 'Content-Length': '0'} 2024-04-16T20:21:00.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b2a4e99e073d39e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-16T20:21:00.459 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.459 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.459 DEBUG:boto:Method: GET 2024-04-16T20:21:00.459 DEBUG:boto:Path: /yrjefh-56/obj124 2024-04-16T20:21:00.459 DEBUG:boto:Data: 2024-04-16T20:21:00.459 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.459 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.459 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.459 DEBUG:boto:Params: {} 2024-04-16T20:21:00.459 DEBUG:boto:Token: None 2024-04-16T20:21:00.460 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124 2024-04-16T20:21:00.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg= 2024-04-16T20:21:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg=', 'Content-Length': '0'} 2024-04-16T20:21:00.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf094270f5b162b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.506 DEBUG:boto:path=/yrjefh-56/obj124 2024-04-16T20:21:00.506 DEBUG:boto:auth_path=/yrjefh-56/obj124 2024-04-16T20:21:00.507 DEBUG:boto:Method: GET 2024-04-16T20:21:00.507 DEBUG:boto:Path: /yrjefh-56/obj124 2024-04-16T20:21:00.507 DEBUG:boto:Data: 2024-04-16T20:21:00.507 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.507 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.507 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.507 DEBUG:boto:Params: {} 2024-04-16T20:21:00.507 DEBUG:boto:Token: None 2024-04-16T20:21:00.507 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj124 2024-04-16T20:21:00.507 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg= 2024-04-16T20:21:00.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg=', 'Content-Length': '0'} 2024-04-16T20:21:00.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004626ea5131287cf7-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.510 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.510 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.510 DEBUG:boto:path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.510 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.510 DEBUG:boto:Method: GET 2024-04-16T20:21:00.510 DEBUG:boto:Path: /yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.510 DEBUG:boto:Data: 2024-04-16T20:21:00.510 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.510 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.510 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.511 DEBUG:boto:Params: {} 2024-04-16T20:21:00.511 DEBUG:boto:Token: None 2024-04-16T20:21:00.511 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.511 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a575be4d16978c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.514 DEBUG:boto:b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.514 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.514 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.514 DEBUG:boto:path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.514 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.514 DEBUG:boto:Method: GET 2024-04-16T20:21:00.514 DEBUG:boto:Path: /yrjefh-56/?prefix=obj124 2024-04-16T20:21:00.515 DEBUG:boto:Data: 2024-04-16T20:21:00.515 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.515 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.515 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.515 DEBUG:boto:Params: {} 2024-04-16T20:21:00.515 DEBUG:boto:Token: None 2024-04-16T20:21:00.515 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.515 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994e5e1fead3fe0b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.519 DEBUG:boto:b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:21:00.519 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.519 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.519 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.519 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.519 DEBUG:boto:Method: GET 2024-04-16T20:21:00.520 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.520 DEBUG:boto:Data: 2024-04-16T20:21:00.520 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.520 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.520 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.520 DEBUG:boto:Params: {} 2024-04-16T20:21:00.520 DEBUG:boto:Token: None 2024-04-16T20:21:00.520 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T20:21:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T20:21:00.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4001dd8b751c39d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.523 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.523 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.523 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.524 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.524 DEBUG:boto:Method: GET 2024-04-16T20:21:00.524 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.524 DEBUG:boto:Data: 2024-04-16T20:21:00.524 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.524 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.524 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.524 DEBUG:boto:Params: {} 2024-04-16T20:21:00.524 DEBUG:boto:Token: None 2024-04-16T20:21:00.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T20:21:00.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T20:21:00.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f05d64a9458c490-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.527 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.527 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.527 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.527 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.527 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.527 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.527 DEBUG:boto:Data: 2024-04-16T20:21:00.528 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.528 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.528 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.528 DEBUG:boto:Params: {} 2024-04-16T20:21:00.528 DEBUG:boto:Token: None 2024-04-16T20:21:00.528 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak= 2024-04-16T20:21:00.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak=', 'Content-Length': '0'} 2024-04-16T20:21:00.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0da91b797d490e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.531 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.531 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.531 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.531 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.531 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.531 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.531 DEBUG:boto:Data: 2024-04-16T20:21:00.531 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.532 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.532 DEBUG:boto:Params: {} 2024-04-16T20:21:00.532 DEBUG:boto:Token: None 2024-04-16T20:21:00.532 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak= 2024-04-16T20:21:00.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak=', 'Content-Length': '0'} 2024-04-16T20:21:00.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71907109bdc34db-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:21:00.534 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.534 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.534 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.534 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.534 DEBUG:boto:Method: GET 2024-04-16T20:21:00.535 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.535 DEBUG:boto:Data: 2024-04-16T20:21:00.535 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.535 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.535 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.535 DEBUG:boto:Params: {} 2024-04-16T20:21:00.535 DEBUG:boto:Token: None 2024-04-16T20:21:00.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.535 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T20:21:00.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T20:21:00.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f11a298994d527b0-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.578 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.578 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.579 DEBUG:boto:path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.579 DEBUG:boto:auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.579 DEBUG:boto:Method: GET 2024-04-16T20:21:00.579 DEBUG:boto:Path: /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.579 DEBUG:boto:Data: 2024-04-16T20:21:00.579 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.579 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.579 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.579 DEBUG:boto:Params: {} 2024-04-16T20:21:00.579 DEBUG:boto:Token: None 2024-04-16T20:21:00.580 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145?versionId=null 2024-04-16T20:21:00.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T20:21:00.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T20:21:00.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d837e5e91aeed73-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.582 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.582 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.582 DEBUG:boto:path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.582 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.583 DEBUG:boto:Method: GET 2024-04-16T20:21:00.583 DEBUG:boto:Path: /yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.583 DEBUG:boto:Data: 2024-04-16T20:21:00.583 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.583 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.583 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.583 DEBUG:boto:Params: {} 2024-04-16T20:21:00.583 DEBUG:boto:Token: None 2024-04-16T20:21:00.583 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.583 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106c4244a53487cb-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.586 DEBUG:boto:b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.587 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.587 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.587 DEBUG:boto:path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.587 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.587 DEBUG:boto:Method: GET 2024-04-16T20:21:00.587 DEBUG:boto:Path: /yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.587 DEBUG:boto:Data: 2024-04-16T20:21:00.587 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.587 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.587 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.587 DEBUG:boto:Params: {} 2024-04-16T20:21:00.588 DEBUG:boto:Token: None 2024-04-16T20:21:00.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1b9e180de246f4-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.591 DEBUG:boto:b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.592 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.592 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.592 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.592 DEBUG:boto:Path: /yrjefh-56/obj145 2024-04-16T20:21:00.592 DEBUG:boto:Data: 2024-04-16T20:21:00.592 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.592 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.592 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.592 DEBUG:boto:Params: {} 2024-04-16T20:21:00.592 DEBUG:boto:Token: None 2024-04-16T20:21:00.593 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145 2024-04-16T20:21:00.593 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs= 2024-04-16T20:21:00.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs=', 'Content-Length': '0'} 2024-04-16T20:21:00.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb12fcbf4ade92f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.595 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.595 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.595 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.595 DEBUG:boto:Path: /yrjefh-56/obj145 2024-04-16T20:21:00.596 DEBUG:boto:Data: 2024-04-16T20:21:00.596 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.596 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.596 DEBUG:boto:Params: {} 2024-04-16T20:21:00.596 DEBUG:boto:Token: None 2024-04-16T20:21:00.596 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145 2024-04-16T20:21:00.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs= 2024-04-16T20:21:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs=', 'Content-Length': '0'} 2024-04-16T20:21:00.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047081e682fe95184-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-16T20:21:00.599 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.599 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.599 DEBUG:boto:Method: GET 2024-04-16T20:21:00.599 DEBUG:boto:Path: /yrjefh-56/obj145 2024-04-16T20:21:00.599 DEBUG:boto:Data: 2024-04-16T20:21:00.599 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.599 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.599 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.599 DEBUG:boto:Params: {} 2024-04-16T20:21:00.600 DEBUG:boto:Token: None 2024-04-16T20:21:00.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145 2024-04-16T20:21:00.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0= 2024-04-16T20:21:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0=', 'Content-Length': '0'} 2024-04-16T20:21:00.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095bb06f9fca59fa-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.602 DEBUG:boto:path=/yrjefh-56/obj145 2024-04-16T20:21:00.602 DEBUG:boto:auth_path=/yrjefh-56/obj145 2024-04-16T20:21:00.602 DEBUG:boto:Method: GET 2024-04-16T20:21:00.602 DEBUG:boto:Path: /yrjefh-56/obj145 2024-04-16T20:21:00.603 DEBUG:boto:Data: 2024-04-16T20:21:00.603 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.603 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.603 DEBUG:boto:Params: {} 2024-04-16T20:21:00.603 DEBUG:boto:Token: None 2024-04-16T20:21:00.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj145 2024-04-16T20:21:00.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0= 2024-04-16T20:21:00.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0=', 'Content-Length': '0'} 2024-04-16T20:21:00.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e380acac9bb9ec-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.650 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.650 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.651 DEBUG:boto:path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.651 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.651 DEBUG:boto:Method: GET 2024-04-16T20:21:00.651 DEBUG:boto:Path: /yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.651 DEBUG:boto:Data: 2024-04-16T20:21:00.651 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.651 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.651 DEBUG:boto:Params: {} 2024-04-16T20:21:00.651 DEBUG:boto:Token: None 2024-04-16T20:21:00.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078e54ce948a57c3f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.654 DEBUG:boto:b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.655 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.655 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.655 DEBUG:boto:path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.655 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.655 DEBUG:boto:Method: GET 2024-04-16T20:21:00.655 DEBUG:boto:Path: /yrjefh-56/?prefix=obj145 2024-04-16T20:21:00.655 DEBUG:boto:Data: 2024-04-16T20:21:00.655 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.656 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.656 DEBUG:boto:Params: {} 2024-04-16T20:21:00.656 DEBUG:boto:Token: None 2024-04-16T20:21:00.656 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa64d15b584912eb-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.660 DEBUG:boto:b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:21:00.660 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.660 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.660 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.660 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.660 DEBUG:boto:Method: GET 2024-04-16T20:21:00.661 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.661 DEBUG:boto:Data: 2024-04-16T20:21:00.661 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.661 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.661 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.661 DEBUG:boto:Params: {} 2024-04-16T20:21:00.661 DEBUG:boto:Token: None 2024-04-16T20:21:00.661 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.661 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T20:21:00.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T20:21:00.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c20d638af1dd873-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.706 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.706 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.707 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.707 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.707 DEBUG:boto:Method: GET 2024-04-16T20:21:00.707 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.707 DEBUG:boto:Data: 2024-04-16T20:21:00.707 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.707 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.707 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.707 DEBUG:boto:Params: {} 2024-04-16T20:21:00.707 DEBUG:boto:Token: None 2024-04-16T20:21:00.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T20:21:00.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T20:21:00.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a128102e16bf8d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.710 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.710 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.710 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.710 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.710 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.711 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.711 DEBUG:boto:Data: 2024-04-16T20:21:00.711 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.711 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.711 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.711 DEBUG:boto:Params: {} 2024-04-16T20:21:00.711 DEBUG:boto:Token: None 2024-04-16T20:21:00.711 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.711 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98= 2024-04-16T20:21:00.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98=', 'Content-Length': '0'} 2024-04-16T20:21:00.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c6343efa07fe2d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.714 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.714 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.714 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.714 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.714 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.714 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.714 DEBUG:boto:Data: 2024-04-16T20:21:00.714 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.714 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.714 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.714 DEBUG:boto:Params: {} 2024-04-16T20:21:00.715 DEBUG:boto:Token: None 2024-04-16T20:21:00.715 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98= 2024-04-16T20:21:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98=', 'Content-Length': '0'} 2024-04-16T20:21:00.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fb9cf6900dcc97-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:21:00.717 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.717 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.717 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.717 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.717 DEBUG:boto:Method: GET 2024-04-16T20:21:00.717 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.717 DEBUG:boto:Data: 2024-04-16T20:21:00.717 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.717 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.718 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.718 DEBUG:boto:Params: {} 2024-04-16T20:21:00.718 DEBUG:boto:Token: None 2024-04-16T20:21:00.718 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.718 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T20:21:00.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T20:21:00.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc84ba7e69e2ae4-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.720 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.720 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.720 DEBUG:boto:path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.721 DEBUG:boto:auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.721 DEBUG:boto:Method: GET 2024-04-16T20:21:00.721 DEBUG:boto:Path: /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.721 DEBUG:boto:Data: 2024-04-16T20:21:00.721 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.721 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.721 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.721 DEBUG:boto:Params: {} 2024-04-16T20:21:00.721 DEBUG:boto:Token: None 2024-04-16T20:21:00.721 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148?versionId=null 2024-04-16T20:21:00.722 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T20:21:00.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T20:21:00.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007140d7b56b21f760-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.724 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.724 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.724 DEBUG:boto:path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.724 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.724 DEBUG:boto:Method: GET 2024-04-16T20:21:00.724 DEBUG:boto:Path: /yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.724 DEBUG:boto:Data: 2024-04-16T20:21:00.724 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.725 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.725 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.725 DEBUG:boto:Params: {} 2024-04-16T20:21:00.725 DEBUG:boto:Token: None 2024-04-16T20:21:00.725 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b167b8655b6ec68-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.728 DEBUG:boto:b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.728 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.728 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.728 DEBUG:boto:path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.728 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.728 DEBUG:boto:Method: GET 2024-04-16T20:21:00.728 DEBUG:boto:Path: /yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.729 DEBUG:boto:Data: 2024-04-16T20:21:00.729 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.729 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.729 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.729 DEBUG:boto:Params: {} 2024-04-16T20:21:00.729 DEBUG:boto:Token: None 2024-04-16T20:21:00.729 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.729 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c83b0a9a97817e08-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.733 DEBUG:boto:b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.733 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.733 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.733 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.733 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:21:00.733 DEBUG:boto:Data: 2024-04-16T20:21:00.733 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.733 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.734 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.734 DEBUG:boto:Params: {} 2024-04-16T20:21:00.734 DEBUG:boto:Token: None 2024-04-16T20:21:00.734 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148 2024-04-16T20:21:00.734 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4= 2024-04-16T20:21:00.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4=', 'Content-Length': '0'} 2024-04-16T20:21:00.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426f596c2ef367f5-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.736 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.736 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.736 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.736 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:21:00.737 DEBUG:boto:Data: 2024-04-16T20:21:00.737 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.737 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.737 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.737 DEBUG:boto:Params: {} 2024-04-16T20:21:00.737 DEBUG:boto:Token: None 2024-04-16T20:21:00.737 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148 2024-04-16T20:21:00.737 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4= 2024-04-16T20:21:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4=', 'Content-Length': '0'} 2024-04-16T20:21:00.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701199626695792b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-16T20:21:00.740 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.740 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.740 DEBUG:boto:Method: GET 2024-04-16T20:21:00.740 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:21:00.740 DEBUG:boto:Data: 2024-04-16T20:21:00.740 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.740 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.740 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.740 DEBUG:boto:Params: {} 2024-04-16T20:21:00.740 DEBUG:boto:Token: None 2024-04-16T20:21:00.741 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148 2024-04-16T20:21:00.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U= 2024-04-16T20:21:00.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U=', 'Content-Length': '0'} 2024-04-16T20:21:00.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088db3ae03e81fcab-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.746 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:00.746 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:00.746 DEBUG:boto:Method: GET 2024-04-16T20:21:00.746 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:21:00.746 DEBUG:boto:Data: 2024-04-16T20:21:00.746 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.746 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.746 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.747 DEBUG:boto:Params: {} 2024-04-16T20:21:00.747 DEBUG:boto:Token: None 2024-04-16T20:21:00.747 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj148 2024-04-16T20:21:00.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U= 2024-04-16T20:21:00.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U=', 'Content-Length': '0'} 2024-04-16T20:21:00.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c158dde45e57b2-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.790 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.791 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.791 DEBUG:boto:path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.791 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.791 DEBUG:boto:Method: GET 2024-04-16T20:21:00.791 DEBUG:boto:Path: /yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.791 DEBUG:boto:Data: 2024-04-16T20:21:00.791 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.792 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.792 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.792 DEBUG:boto:Params: {} 2024-04-16T20:21:00.792 DEBUG:boto:Token: None 2024-04-16T20:21:00.793 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.793 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411ba7197881619a-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.796 DEBUG:boto:b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.796 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.796 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.797 DEBUG:boto:path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.797 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.797 DEBUG:boto:Method: GET 2024-04-16T20:21:00.797 DEBUG:boto:Path: /yrjefh-56/?prefix=obj148 2024-04-16T20:21:00.797 DEBUG:boto:Data: 2024-04-16T20:21:00.797 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.797 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.797 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.797 DEBUG:boto:Params: {} 2024-04-16T20:21:00.797 DEBUG:boto:Token: None 2024-04-16T20:21:00.798 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.798 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516d4a37f693764e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.802 DEBUG:boto:b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:21:00.802 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.802 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.802 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.802 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.803 DEBUG:boto:Method: GET 2024-04-16T20:21:00.803 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.803 DEBUG:boto:Data: 2024-04-16T20:21:00.803 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.803 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.803 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.803 DEBUG:boto:Params: {} 2024-04-16T20:21:00.803 DEBUG:boto:Token: None 2024-04-16T20:21:00.803 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.804 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T20:21:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T20:21:00.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787e0fefd69a2f36-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.846 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.847 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.847 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.847 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.847 DEBUG:boto:Method: GET 2024-04-16T20:21:00.847 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.847 DEBUG:boto:Data: 2024-04-16T20:21:00.847 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.847 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.847 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.848 DEBUG:boto:Params: {} 2024-04-16T20:21:00.848 DEBUG:boto:Token: None 2024-04-16T20:21:00.848 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T20:21:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T20:21:00.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd523088089b12d1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.851 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.851 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.851 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.851 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.851 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.851 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.851 DEBUG:boto:Data: 2024-04-16T20:21:00.851 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.852 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.852 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.852 DEBUG:boto:Params: {} 2024-04-16T20:21:00.852 DEBUG:boto:Token: None 2024-04-16T20:21:00.852 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.852 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU= 2024-04-16T20:21:00.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU=', 'Content-Length': '0'} 2024-04-16T20:21:00.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2a6fce117e2cd7-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.855 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.855 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.855 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.855 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.855 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.855 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.855 DEBUG:boto:Data: 2024-04-16T20:21:00.855 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.855 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.855 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.856 DEBUG:boto:Params: {} 2024-04-16T20:21:00.856 DEBUG:boto:Token: None 2024-04-16T20:21:00.856 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.856 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU= 2024-04-16T20:21:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU=', 'Content-Length': '0'} 2024-04-16T20:21:00.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020782ba4e1126fa6-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:21:00.858 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.858 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.858 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.858 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.858 DEBUG:boto:Method: GET 2024-04-16T20:21:00.859 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.859 DEBUG:boto:Data: 2024-04-16T20:21:00.859 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.859 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.859 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.859 DEBUG:boto:Params: {} 2024-04-16T20:21:00.859 DEBUG:boto:Token: None 2024-04-16T20:21:00.859 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.859 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T20:21:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T20:21:00.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d36a01325bd36f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.862 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.862 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.862 DEBUG:boto:path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.862 DEBUG:boto:auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.862 DEBUG:boto:Method: GET 2024-04-16T20:21:00.863 DEBUG:boto:Path: /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.863 DEBUG:boto:Data: 2024-04-16T20:21:00.863 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.863 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.863 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.863 DEBUG:boto:Params: {} 2024-04-16T20:21:00.863 DEBUG:boto:Token: None 2024-04-16T20:21:00.863 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155?versionId=null 2024-04-16T20:21:00.863 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T20:21:00.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T20:21:00.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021acb53d9cab821d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.866 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.866 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.866 DEBUG:boto:path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.866 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.866 DEBUG:boto:Method: GET 2024-04-16T20:21:00.866 DEBUG:boto:Path: /yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.866 DEBUG:boto:Data: 2024-04-16T20:21:00.866 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.866 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.867 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.867 DEBUG:boto:Params: {} 2024-04-16T20:21:00.867 DEBUG:boto:Token: None 2024-04-16T20:21:00.867 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.867 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8d74299956d91e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.870 DEBUG:boto:b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.870 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.870 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.871 DEBUG:boto:path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.871 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.871 DEBUG:boto:Method: GET 2024-04-16T20:21:00.871 DEBUG:boto:Path: /yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.871 DEBUG:boto:Data: 2024-04-16T20:21:00.871 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.871 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.871 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.871 DEBUG:boto:Params: {} 2024-04-16T20:21:00.871 DEBUG:boto:Token: None 2024-04-16T20:21:00.871 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.872 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ca602e1bb9185c-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.875 DEBUG:boto:b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.875 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.876 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.876 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.876 DEBUG:boto:Path: /yrjefh-56/obj155 2024-04-16T20:21:00.876 DEBUG:boto:Data: 2024-04-16T20:21:00.876 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.876 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.876 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.876 DEBUG:boto:Params: {} 2024-04-16T20:21:00.876 DEBUG:boto:Token: None 2024-04-16T20:21:00.877 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155 2024-04-16T20:21:00.877 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY= 2024-04-16T20:21:00.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b75061432297ac-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.879 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.879 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.879 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.879 DEBUG:boto:Path: /yrjefh-56/obj155 2024-04-16T20:21:00.879 DEBUG:boto:Data: 2024-04-16T20:21:00.879 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.879 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.880 DEBUG:boto:Params: {} 2024-04-16T20:21:00.880 DEBUG:boto:Token: None 2024-04-16T20:21:00.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155 2024-04-16T20:21:00.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY= 2024-04-16T20:21:00.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c514d407208839aa-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-16T20:21:00.882 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.882 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.882 DEBUG:boto:Method: GET 2024-04-16T20:21:00.883 DEBUG:boto:Path: /yrjefh-56/obj155 2024-04-16T20:21:00.883 DEBUG:boto:Data: 2024-04-16T20:21:00.883 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.883 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.883 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.883 DEBUG:boto:Params: {} 2024-04-16T20:21:00.883 DEBUG:boto:Token: None 2024-04-16T20:21:00.883 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155 2024-04-16T20:21:00.883 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY= 2024-04-16T20:21:00.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY=', 'Content-Length': '0'} 2024-04-16T20:21:00.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924b32864499cffb-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.926 DEBUG:boto:path=/yrjefh-56/obj155 2024-04-16T20:21:00.927 DEBUG:boto:auth_path=/yrjefh-56/obj155 2024-04-16T20:21:00.927 DEBUG:boto:Method: GET 2024-04-16T20:21:00.927 DEBUG:boto:Path: /yrjefh-56/obj155 2024-04-16T20:21:00.927 DEBUG:boto:Data: 2024-04-16T20:21:00.927 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.927 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.927 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.927 DEBUG:boto:Params: {} 2024-04-16T20:21:00.927 DEBUG:boto:Token: None 2024-04-16T20:21:00.927 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj155 2024-04-16T20:21:00.928 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY= 2024-04-16T20:21:00.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY=', 'Content-Length': '0'} 2024-04-16T20:21:00.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a083ae3bc8afac-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.930 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.930 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.930 DEBUG:boto:path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.930 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.931 DEBUG:boto:Method: GET 2024-04-16T20:21:00.931 DEBUG:boto:Path: /yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.931 DEBUG:boto:Data: 2024-04-16T20:21:00.931 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.931 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.931 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.931 DEBUG:boto:Params: {} 2024-04-16T20:21:00.931 DEBUG:boto:Token: None 2024-04-16T20:21:00.931 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.931 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672f9c85b0dd412c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.934 DEBUG:boto:b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.935 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:00.935 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:00.935 DEBUG:boto:path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.935 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.935 DEBUG:boto:Method: GET 2024-04-16T20:21:00.935 DEBUG:boto:Path: /yrjefh-56/?prefix=obj155 2024-04-16T20:21:00.935 DEBUG:boto:Data: 2024-04-16T20:21:00.935 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.935 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.935 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.935 DEBUG:boto:Params: {} 2024-04-16T20:21:00.936 DEBUG:boto:Token: None 2024-04-16T20:21:00.936 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/ 2024-04-16T20:21:00.936 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T20:21:00.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T20:21:00.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5fe3818b233ca18-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.939 DEBUG:boto:b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:00.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:21:00.940 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:00.940 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:00.940 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.940 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.940 DEBUG:boto:Method: GET 2024-04-16T20:21:00.940 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.940 DEBUG:boto:Data: 2024-04-16T20:21:00.940 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.940 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.940 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.941 DEBUG:boto:Params: {} 2024-04-16T20:21:00.941 DEBUG:boto:Token: None 2024-04-16T20:21:00.941 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.941 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T20:21:00.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T20:21:00.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d9840e537715ff-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.943 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:00.943 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:00.944 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.944 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.944 DEBUG:boto:Method: GET 2024-04-16T20:21:00.944 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.944 DEBUG:boto:Data: 2024-04-16T20:21:00.944 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.944 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.944 DEBUG:boto:Params: {} 2024-04-16T20:21:00.944 DEBUG:boto:Token: None 2024-04-16T20:21:00.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.945 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T20:21:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T20:21:00.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a96be26d318eee-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.949 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:00.949 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:00.949 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.949 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.949 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.949 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.950 DEBUG:boto:Data: 2024-04-16T20:21:00.950 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.950 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.950 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.950 DEBUG:boto:Params: {} 2024-04-16T20:21:00.950 DEBUG:boto:Token: None 2024-04-16T20:21:00.950 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE= 2024-04-16T20:21:00.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE=', 'Content-Length': '0'} 2024-04-16T20:21:00.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa79d7e940b8b90e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.953 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:00.953 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:00.953 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.953 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.953 DEBUG:boto:Method: HEAD 2024-04-16T20:21:00.953 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.953 DEBUG:boto:Data: 2024-04-16T20:21:00.953 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.953 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:00.953 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.953 DEBUG:boto:Params: {} 2024-04-16T20:21:00.954 DEBUG:boto:Token: None 2024-04-16T20:21:00.954 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.954 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE= 2024-04-16T20:21:00.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE=', 'Content-Length': '0'} 2024-04-16T20:21:00.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019225495d53590ca-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:00.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:21:00.956 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:00.956 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:00.956 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.956 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.956 DEBUG:boto:Method: GET 2024-04-16T20:21:00.957 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.957 DEBUG:boto:Data: 2024-04-16T20:21:00.957 DEBUG:boto:Headers: {} 2024-04-16T20:21:00.957 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:00.957 DEBUG:boto:Port: 8000 2024-04-16T20:21:00.957 DEBUG:boto:Params: {} 2024-04-16T20:21:00.957 DEBUG:boto:Token: None 2024-04-16T20:21:00.957 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:00 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:00.957 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T20:21:00.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T20:21:00.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f451244c250b2eb5-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.002 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:01.003 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:01.003 DEBUG:boto:path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:01.003 DEBUG:boto:auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T20:21:01.003 DEBUG:boto:Method: GET 2024-04-16T20:21:01.003 DEBUG:boto:Path: /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:01.003 DEBUG:boto:Data: 2024-04-16T20:21:01.003 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.003 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.003 DEBUG:boto:Params: {} 2024-04-16T20:21:01.003 DEBUG:boto:Token: None 2024-04-16T20:21:01.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj17?versionId=null 2024-04-16T20:21:01.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aX/VePocZHScP3Zm35yWNmQfRBY= 2024-04-16T20:21:01.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aX/VePocZHScP3Zm35yWNmQfRBY=', 'Content-Length': '0'} 2024-04-16T20:21:01.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d2574f137d0bfa-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.006 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.006 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.006 DEBUG:boto:path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.006 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.006 DEBUG:boto:Method: GET 2024-04-16T20:21:01.007 DEBUG:boto:Path: /yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.007 DEBUG:boto:Data: 2024-04-16T20:21:01.007 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.007 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.007 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.007 DEBUG:boto:Params: {} 2024-04-16T20:21:01.007 DEBUG:boto:Token: None 2024-04-16T20:21:01.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da3c1df2d8b77195-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.010 DEBUG:boto:b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.010 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.011 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.011 DEBUG:boto:path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.011 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.011 DEBUG:boto:Method: GET 2024-04-16T20:21:01.011 DEBUG:boto:Path: /yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.011 DEBUG:boto:Data: 2024-04-16T20:21:01.011 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.011 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.011 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.011 DEBUG:boto:Params: {} 2024-04-16T20:21:01.011 DEBUG:boto:Token: None 2024-04-16T20:21:01.012 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.012 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bae7b22a9860a6-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.015 DEBUG:boto:b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.016 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:01.016 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:01.016 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.016 DEBUG:boto:Path: /yrjefh-56/obj17 2024-04-16T20:21:01.016 DEBUG:boto:Data: 2024-04-16T20:21:01.016 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.016 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.016 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.016 DEBUG:boto:Params: {} 2024-04-16T20:21:01.016 DEBUG:boto:Token: None 2024-04-16T20:21:01.017 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj17 2024-04-16T20:21:01.017 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY= 2024-04-16T20:21:01.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY=', 'Content-Length': '0'} 2024-04-16T20:21:01.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf151775871cfb4d-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.019 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:01.019 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:01.019 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.019 DEBUG:boto:Path: /yrjefh-56/obj17 2024-04-16T20:21:01.019 DEBUG:boto:Data: 2024-04-16T20:21:01.019 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.019 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.019 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.019 DEBUG:boto:Params: {} 2024-04-16T20:21:01.019 DEBUG:boto:Token: None 2024-04-16T20:21:01.020 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj17 2024-04-16T20:21:01.020 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY= 2024-04-16T20:21:01.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY=', 'Content-Length': '0'} 2024-04-16T20:21:01.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307c02dff95229c7-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-16T20:21:01.022 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:01.022 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:01.022 DEBUG:boto:Method: GET 2024-04-16T20:21:01.022 DEBUG:boto:Path: /yrjefh-56/obj17 2024-04-16T20:21:01.022 DEBUG:boto:Data: 2024-04-16T20:21:01.022 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.022 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.023 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.023 DEBUG:boto:Params: {} 2024-04-16T20:21:01.023 DEBUG:boto:Token: None 2024-04-16T20:21:01.023 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj17 2024-04-16T20:21:01.023 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0= 2024-04-16T20:21:01.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0=', 'Content-Length': '0'} 2024-04-16T20:21:01.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62492ea5dc33f4e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.025 DEBUG:boto:path=/yrjefh-56/obj17 2024-04-16T20:21:01.025 DEBUG:boto:auth_path=/yrjefh-56/obj17 2024-04-16T20:21:01.025 DEBUG:boto:Method: GET 2024-04-16T20:21:01.025 DEBUG:boto:Path: /yrjefh-56/obj17 2024-04-16T20:21:01.025 DEBUG:boto:Data: 2024-04-16T20:21:01.026 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.026 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.026 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.026 DEBUG:boto:Params: {} 2024-04-16T20:21:01.026 DEBUG:boto:Token: None 2024-04-16T20:21:01.026 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj17 2024-04-16T20:21:01.026 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0= 2024-04-16T20:21:01.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0=', 'Content-Length': '0'} 2024-04-16T20:21:01.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009742645b328e3ba5-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.070 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.071 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.071 DEBUG:boto:path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.071 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.071 DEBUG:boto:Method: GET 2024-04-16T20:21:01.071 DEBUG:boto:Path: /yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.071 DEBUG:boto:Data: 2024-04-16T20:21:01.071 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.071 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.071 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.071 DEBUG:boto:Params: {} 2024-04-16T20:21:01.071 DEBUG:boto:Token: None 2024-04-16T20:21:01.072 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.072 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba564a4a052f87c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.074 DEBUG:boto:b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.074 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.075 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.075 DEBUG:boto:path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.075 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.075 DEBUG:boto:Method: GET 2024-04-16T20:21:01.075 DEBUG:boto:Path: /yrjefh-56/?prefix=obj17 2024-04-16T20:21:01.075 DEBUG:boto:Data: 2024-04-16T20:21:01.075 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.075 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.075 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.075 DEBUG:boto:Params: {} 2024-04-16T20:21:01.075 DEBUG:boto:Token: None 2024-04-16T20:21:01.076 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.076 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9adcb879ba0b0c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.079 DEBUG:boto:b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:21:01.080 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.080 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.080 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.080 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.080 DEBUG:boto:Method: GET 2024-04-16T20:21:01.080 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.080 DEBUG:boto:Data: 2024-04-16T20:21:01.080 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.081 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.081 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.081 DEBUG:boto:Params: {} 2024-04-16T20:21:01.081 DEBUG:boto:Token: None 2024-04-16T20:21:01.081 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.081 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T20:21:01.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T20:21:01.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d46a6eb7e0fbe2-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.083 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.083 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.084 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.084 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.084 DEBUG:boto:Method: GET 2024-04-16T20:21:01.084 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.084 DEBUG:boto:Data: 2024-04-16T20:21:01.084 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.084 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.084 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.084 DEBUG:boto:Params: {} 2024-04-16T20:21:01.084 DEBUG:boto:Token: None 2024-04-16T20:21:01.084 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.085 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T20:21:01.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T20:21:01.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ccf62a853711a2f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.087 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.087 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.087 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.087 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.087 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.087 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.087 DEBUG:boto:Data: 2024-04-16T20:21:01.087 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.087 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.088 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.088 DEBUG:boto:Params: {} 2024-04-16T20:21:01.088 DEBUG:boto:Token: None 2024-04-16T20:21:01.088 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.088 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI= 2024-04-16T20:21:01.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI=', 'Content-Length': '0'} 2024-04-16T20:21:01.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3510b4b2e385ffb-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.090 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.090 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.090 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.090 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.090 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.090 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.090 DEBUG:boto:Data: 2024-04-16T20:21:01.091 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.091 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.091 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.091 DEBUG:boto:Params: {} 2024-04-16T20:21:01.091 DEBUG:boto:Token: None 2024-04-16T20:21:01.091 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.091 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI= 2024-04-16T20:21:01.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI=', 'Content-Length': '0'} 2024-04-16T20:21:01.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b97f6eabd49057a-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:21:01.093 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.093 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.094 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.094 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.094 DEBUG:boto:Method: GET 2024-04-16T20:21:01.094 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.094 DEBUG:boto:Data: 2024-04-16T20:21:01.094 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.094 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.094 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.094 DEBUG:boto:Params: {} 2024-04-16T20:21:01.094 DEBUG:boto:Token: None 2024-04-16T20:21:01.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T20:21:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T20:21:01.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca8b0dd7b44c729-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.138 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.138 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.138 DEBUG:boto:path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.139 DEBUG:boto:auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.139 DEBUG:boto:Method: GET 2024-04-16T20:21:01.139 DEBUG:boto:Path: /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.139 DEBUG:boto:Data: 2024-04-16T20:21:01.139 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.139 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.139 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.139 DEBUG:boto:Params: {} 2024-04-16T20:21:01.139 DEBUG:boto:Token: None 2024-04-16T20:21:01.139 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185?versionId=null 2024-04-16T20:21:01.140 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T20:21:01.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T20:21:01.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6277a2b1af0bd95-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.142 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.142 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.142 DEBUG:boto:path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.142 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.142 DEBUG:boto:Method: GET 2024-04-16T20:21:01.142 DEBUG:boto:Path: /yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.142 DEBUG:boto:Data: 2024-04-16T20:21:01.142 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.142 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.143 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.143 DEBUG:boto:Params: {} 2024-04-16T20:21:01.143 DEBUG:boto:Token: None 2024-04-16T20:21:01.143 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.143 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003109239bd9ddd4b0-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.146 DEBUG:boto:b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.146 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.146 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.146 DEBUG:boto:path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.146 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.147 DEBUG:boto:Method: GET 2024-04-16T20:21:01.147 DEBUG:boto:Path: /yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.147 DEBUG:boto:Data: 2024-04-16T20:21:01.147 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.147 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.147 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.147 DEBUG:boto:Params: {} 2024-04-16T20:21:01.147 DEBUG:boto:Token: None 2024-04-16T20:21:01.147 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.147 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28cc3a5e4d1e871-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.151 DEBUG:boto:b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.151 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.151 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.152 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.152 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:21:01.152 DEBUG:boto:Data: 2024-04-16T20:21:01.152 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.152 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.152 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.152 DEBUG:boto:Params: {} 2024-04-16T20:21:01.152 DEBUG:boto:Token: None 2024-04-16T20:21:01.152 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185 2024-04-16T20:21:01.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI= 2024-04-16T20:21:01.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI=', 'Content-Length': '0'} 2024-04-16T20:21:01.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002020850ced7c7f5c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.155 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.155 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.155 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.155 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:21:01.155 DEBUG:boto:Data: 2024-04-16T20:21:01.155 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.157 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.157 DEBUG:boto:Params: {} 2024-04-16T20:21:01.157 DEBUG:boto:Token: None 2024-04-16T20:21:01.157 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185 2024-04-16T20:21:01.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI= 2024-04-16T20:21:01.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI=', 'Content-Length': '0'} 2024-04-16T20:21:01.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c646013f2305d54-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-16T20:21:01.160 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.160 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.160 DEBUG:boto:Method: GET 2024-04-16T20:21:01.160 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:21:01.160 DEBUG:boto:Data: 2024-04-16T20:21:01.160 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.160 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.160 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.161 DEBUG:boto:Params: {} 2024-04-16T20:21:01.161 DEBUG:boto:Token: None 2024-04-16T20:21:01.161 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185 2024-04-16T20:21:01.161 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A= 2024-04-16T20:21:01.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A=', 'Content-Length': '0'} 2024-04-16T20:21:01.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc41fae62e7a4f04-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.163 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:01.163 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:01.163 DEBUG:boto:Method: GET 2024-04-16T20:21:01.163 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:21:01.163 DEBUG:boto:Data: 2024-04-16T20:21:01.164 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.164 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.164 DEBUG:boto:Params: {} 2024-04-16T20:21:01.164 DEBUG:boto:Token: None 2024-04-16T20:21:01.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj185 2024-04-16T20:21:01.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A= 2024-04-16T20:21:01.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A=', 'Content-Length': '0'} 2024-04-16T20:21:01.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012148962bc3db3d0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.166 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.167 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.167 DEBUG:boto:path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.167 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.167 DEBUG:boto:Method: GET 2024-04-16T20:21:01.167 DEBUG:boto:Path: /yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.167 DEBUG:boto:Data: 2024-04-16T20:21:01.167 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.167 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.167 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.167 DEBUG:boto:Params: {} 2024-04-16T20:21:01.167 DEBUG:boto:Token: None 2024-04-16T20:21:01.168 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606617719b83866e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.214 DEBUG:boto:b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.215 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.215 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.215 DEBUG:boto:path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.215 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.215 DEBUG:boto:Method: GET 2024-04-16T20:21:01.215 DEBUG:boto:Path: /yrjefh-56/?prefix=obj185 2024-04-16T20:21:01.215 DEBUG:boto:Data: 2024-04-16T20:21:01.215 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.215 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.215 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.215 DEBUG:boto:Params: {} 2024-04-16T20:21:01.216 DEBUG:boto:Token: None 2024-04-16T20:21:01.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5684da1efe3b1ce-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.220 DEBUG:boto:b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:21:01.220 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.220 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.220 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.220 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.221 DEBUG:boto:Method: GET 2024-04-16T20:21:01.221 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.221 DEBUG:boto:Data: 2024-04-16T20:21:01.221 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.221 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.221 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.221 DEBUG:boto:Params: {} 2024-04-16T20:21:01.221 DEBUG:boto:Token: None 2024-04-16T20:21:01.221 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T20:21:01.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T20:21:01.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7aaf4ba92a8f6fe-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.224 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.224 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.224 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.224 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.224 DEBUG:boto:Method: GET 2024-04-16T20:21:01.224 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.224 DEBUG:boto:Data: 2024-04-16T20:21:01.224 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.224 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.224 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.224 DEBUG:boto:Params: {} 2024-04-16T20:21:01.225 DEBUG:boto:Token: None 2024-04-16T20:21:01.225 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.225 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T20:21:01.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T20:21:01.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a79a5ac46f3414c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.270 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.270 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.271 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.271 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.271 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.271 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.271 DEBUG:boto:Data: 2024-04-16T20:21:01.271 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.271 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.271 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.271 DEBUG:boto:Params: {} 2024-04-16T20:21:01.271 DEBUG:boto:Token: None 2024-04-16T20:21:01.272 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw= 2024-04-16T20:21:01.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw=', 'Content-Length': '0'} 2024-04-16T20:21:01.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2a44fd9188030a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.274 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.274 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.274 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.274 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.274 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.274 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.274 DEBUG:boto:Data: 2024-04-16T20:21:01.274 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.274 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.275 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.275 DEBUG:boto:Params: {} 2024-04-16T20:21:01.275 DEBUG:boto:Token: None 2024-04-16T20:21:01.275 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw= 2024-04-16T20:21:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw=', 'Content-Length': '0'} 2024-04-16T20:21:01.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06e4ceb71428754-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:21:01.277 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.277 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.277 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.277 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.278 DEBUG:boto:Method: GET 2024-04-16T20:21:01.278 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.278 DEBUG:boto:Data: 2024-04-16T20:21:01.278 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.278 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.278 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.278 DEBUG:boto:Params: {} 2024-04-16T20:21:01.278 DEBUG:boto:Token: None 2024-04-16T20:21:01.278 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.278 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T20:21:01.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T20:21:01.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f025922ea4b710b-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.281 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.281 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.281 DEBUG:boto:path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.281 DEBUG:boto:auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.281 DEBUG:boto:Method: GET 2024-04-16T20:21:01.281 DEBUG:boto:Path: /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.281 DEBUG:boto:Data: 2024-04-16T20:21:01.281 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.281 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.281 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.281 DEBUG:boto:Params: {} 2024-04-16T20:21:01.282 DEBUG:boto:Token: None 2024-04-16T20:21:01.282 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19?versionId=null 2024-04-16T20:21:01.282 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T20:21:01.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T20:21:01.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e64dd0a7e447c548-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.284 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.284 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.285 DEBUG:boto:path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.285 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.285 DEBUG:boto:Method: GET 2024-04-16T20:21:01.285 DEBUG:boto:Path: /yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.285 DEBUG:boto:Data: 2024-04-16T20:21:01.285 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.285 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.285 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.285 DEBUG:boto:Params: {} 2024-04-16T20:21:01.285 DEBUG:boto:Token: None 2024-04-16T20:21:01.286 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.286 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce69d8c454889a84-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.288 DEBUG:boto:b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.289 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.289 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.289 DEBUG:boto:path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.289 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.289 DEBUG:boto:Method: GET 2024-04-16T20:21:01.289 DEBUG:boto:Path: /yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.289 DEBUG:boto:Data: 2024-04-16T20:21:01.289 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.290 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.290 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.290 DEBUG:boto:Params: {} 2024-04-16T20:21:01.290 DEBUG:boto:Token: None 2024-04-16T20:21:01.290 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.290 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081edfef05842d230-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.294 DEBUG:boto:b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.294 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.294 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.294 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.294 DEBUG:boto:Path: /yrjefh-56/obj19 2024-04-16T20:21:01.294 DEBUG:boto:Data: 2024-04-16T20:21:01.294 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.295 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.295 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.295 DEBUG:boto:Params: {} 2024-04-16T20:21:01.295 DEBUG:boto:Token: None 2024-04-16T20:21:01.295 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19 2024-04-16T20:21:01.295 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE= 2024-04-16T20:21:01.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE=', 'Content-Length': '0'} 2024-04-16T20:21:01.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da0fb4d7122b389-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.297 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.297 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.297 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.297 DEBUG:boto:Path: /yrjefh-56/obj19 2024-04-16T20:21:01.298 DEBUG:boto:Data: 2024-04-16T20:21:01.298 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.298 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.298 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.298 DEBUG:boto:Params: {} 2024-04-16T20:21:01.298 DEBUG:boto:Token: None 2024-04-16T20:21:01.298 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19 2024-04-16T20:21:01.298 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE= 2024-04-16T20:21:01.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE=', 'Content-Length': '0'} 2024-04-16T20:21:01.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62e7adad2500a4e-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-16T20:21:01.301 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.301 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.301 DEBUG:boto:Method: GET 2024-04-16T20:21:01.301 DEBUG:boto:Path: /yrjefh-56/obj19 2024-04-16T20:21:01.301 DEBUG:boto:Data: 2024-04-16T20:21:01.301 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.301 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.301 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.301 DEBUG:boto:Params: {} 2024-04-16T20:21:01.301 DEBUG:boto:Token: None 2024-04-16T20:21:01.302 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19 2024-04-16T20:21:01.302 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4= 2024-04-16T20:21:01.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4=', 'Content-Length': '0'} 2024-04-16T20:21:01.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ef34206cd1d0932-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.304 DEBUG:boto:path=/yrjefh-56/obj19 2024-04-16T20:21:01.304 DEBUG:boto:auth_path=/yrjefh-56/obj19 2024-04-16T20:21:01.304 DEBUG:boto:Method: GET 2024-04-16T20:21:01.304 DEBUG:boto:Path: /yrjefh-56/obj19 2024-04-16T20:21:01.304 DEBUG:boto:Data: 2024-04-16T20:21:01.304 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.304 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.305 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.305 DEBUG:boto:Params: {} 2024-04-16T20:21:01.305 DEBUG:boto:Token: None 2024-04-16T20:21:01.305 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj19 2024-04-16T20:21:01.305 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4= 2024-04-16T20:21:01.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4=', 'Content-Length': '0'} 2024-04-16T20:21:01.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e4c5c879592ccd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.307 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.307 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.307 DEBUG:boto:path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.307 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.307 DEBUG:boto:Method: GET 2024-04-16T20:21:01.308 DEBUG:boto:Path: /yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.308 DEBUG:boto:Data: 2024-04-16T20:21:01.308 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.308 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.308 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.308 DEBUG:boto:Params: {} 2024-04-16T20:21:01.308 DEBUG:boto:Token: None 2024-04-16T20:21:01.308 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.308 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613950e46ef7b9df-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.311 DEBUG:boto:b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.311 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.312 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.312 DEBUG:boto:path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.312 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.312 DEBUG:boto:Method: GET 2024-04-16T20:21:01.312 DEBUG:boto:Path: /yrjefh-56/?prefix=obj19 2024-04-16T20:21:01.312 DEBUG:boto:Data: 2024-04-16T20:21:01.312 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.312 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.312 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.312 DEBUG:boto:Params: {} 2024-04-16T20:21:01.312 DEBUG:boto:Token: None 2024-04-16T20:21:01.313 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.313 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5d541b2907e770-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.316 DEBUG:boto:b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:21:01.316 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.317 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.317 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.317 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.317 DEBUG:boto:Method: GET 2024-04-16T20:21:01.317 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.317 DEBUG:boto:Data: 2024-04-16T20:21:01.317 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.317 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.317 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.317 DEBUG:boto:Params: {} 2024-04-16T20:21:01.317 DEBUG:boto:Token: None 2024-04-16T20:21:01.318 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.318 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T20:21:01.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T20:21:01.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f270244f77265bc-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.320 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.320 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.320 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.320 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.320 DEBUG:boto:Method: GET 2024-04-16T20:21:01.320 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.320 DEBUG:boto:Data: 2024-04-16T20:21:01.320 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.320 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.320 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.320 DEBUG:boto:Params: {} 2024-04-16T20:21:01.321 DEBUG:boto:Token: None 2024-04-16T20:21:01.321 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.321 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T20:21:01.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T20:21:01.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd3d7fb26c5a5a3-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.366 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.366 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.367 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.367 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.367 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.367 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.367 DEBUG:boto:Data: 2024-04-16T20:21:01.367 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.367 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.367 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.367 DEBUG:boto:Params: {} 2024-04-16T20:21:01.367 DEBUG:boto:Token: None 2024-04-16T20:21:01.368 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE= 2024-04-16T20:21:01.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE=', 'Content-Length': '0'} 2024-04-16T20:21:01.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda75938150ad59f-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.370 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.370 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.370 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.370 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.370 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.371 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.371 DEBUG:boto:Data: 2024-04-16T20:21:01.371 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.371 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.371 DEBUG:boto:Params: {} 2024-04-16T20:21:01.371 DEBUG:boto:Token: None 2024-04-16T20:21:01.371 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE= 2024-04-16T20:21:01.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE=', 'Content-Length': '0'} 2024-04-16T20:21:01.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9ceee69d9f0cdd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:21:01.374 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.374 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.374 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.374 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.374 DEBUG:boto:Method: GET 2024-04-16T20:21:01.374 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.374 DEBUG:boto:Data: 2024-04-16T20:21:01.374 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.374 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.374 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.375 DEBUG:boto:Params: {} 2024-04-16T20:21:01.375 DEBUG:boto:Token: None 2024-04-16T20:21:01.375 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T20:21:01.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T20:21:01.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840e55b4820276ac-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.377 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.377 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.377 DEBUG:boto:path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.377 DEBUG:boto:auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.377 DEBUG:boto:Method: GET 2024-04-16T20:21:01.378 DEBUG:boto:Path: /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.378 DEBUG:boto:Data: 2024-04-16T20:21:01.378 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.378 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.378 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.378 DEBUG:boto:Params: {} 2024-04-16T20:21:01.378 DEBUG:boto:Token: None 2024-04-16T20:21:01.378 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22?versionId=null 2024-04-16T20:21:01.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T20:21:01.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T20:21:01.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daf28d79688647a-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.381 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.381 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.381 DEBUG:boto:path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.381 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.381 DEBUG:boto:Method: GET 2024-04-16T20:21:01.381 DEBUG:boto:Path: /yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.381 DEBUG:boto:Data: 2024-04-16T20:21:01.381 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.381 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.381 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.382 DEBUG:boto:Params: {} 2024-04-16T20:21:01.382 DEBUG:boto:Token: None 2024-04-16T20:21:01.382 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.382 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb348b808095e1db-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.385 DEBUG:boto:b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.385 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.385 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.385 DEBUG:boto:path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.385 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.385 DEBUG:boto:Method: GET 2024-04-16T20:21:01.386 DEBUG:boto:Path: /yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.386 DEBUG:boto:Data: 2024-04-16T20:21:01.386 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.386 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.386 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.386 DEBUG:boto:Params: {} 2024-04-16T20:21:01.386 DEBUG:boto:Token: None 2024-04-16T20:21:01.386 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.386 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ee1016710ee57f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.390 DEBUG:boto:b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.390 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.390 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.391 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.391 DEBUG:boto:Path: /yrjefh-56/obj22 2024-04-16T20:21:01.391 DEBUG:boto:Data: 2024-04-16T20:21:01.391 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.391 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.391 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.391 DEBUG:boto:Params: {} 2024-04-16T20:21:01.391 DEBUG:boto:Token: None 2024-04-16T20:21:01.391 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22 2024-04-16T20:21:01.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q= 2024-04-16T20:21:01.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q=', 'Content-Length': '0'} 2024-04-16T20:21:01.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111e68ca826a3d3e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.394 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.394 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.394 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.394 DEBUG:boto:Path: /yrjefh-56/obj22 2024-04-16T20:21:01.394 DEBUG:boto:Data: 2024-04-16T20:21:01.394 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.394 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.394 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.394 DEBUG:boto:Params: {} 2024-04-16T20:21:01.394 DEBUG:boto:Token: None 2024-04-16T20:21:01.395 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22 2024-04-16T20:21:01.395 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q= 2024-04-16T20:21:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q=', 'Content-Length': '0'} 2024-04-16T20:21:01.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a7a84dab9a61bd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:21:01.397 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.397 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.397 DEBUG:boto:Method: GET 2024-04-16T20:21:01.397 DEBUG:boto:Path: /yrjefh-56/obj22 2024-04-16T20:21:01.397 DEBUG:boto:Data: 2024-04-16T20:21:01.397 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.398 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.398 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.398 DEBUG:boto:Params: {} 2024-04-16T20:21:01.398 DEBUG:boto:Token: None 2024-04-16T20:21:01.398 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22 2024-04-16T20:21:01.398 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c= 2024-04-16T20:21:01.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c=', 'Content-Length': '0'} 2024-04-16T20:21:01.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809406deea50273e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.400 DEBUG:boto:path=/yrjefh-56/obj22 2024-04-16T20:21:01.400 DEBUG:boto:auth_path=/yrjefh-56/obj22 2024-04-16T20:21:01.400 DEBUG:boto:Method: GET 2024-04-16T20:21:01.400 DEBUG:boto:Path: /yrjefh-56/obj22 2024-04-16T20:21:01.400 DEBUG:boto:Data: 2024-04-16T20:21:01.401 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.401 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.401 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.401 DEBUG:boto:Params: {} 2024-04-16T20:21:01.401 DEBUG:boto:Token: None 2024-04-16T20:21:01.401 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj22 2024-04-16T20:21:01.401 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c= 2024-04-16T20:21:01.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c=', 'Content-Length': '0'} 2024-04-16T20:21:01.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c8e3c18ca03141-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.403 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.403 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.403 DEBUG:boto:path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.403 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.404 DEBUG:boto:Method: GET 2024-04-16T20:21:01.404 DEBUG:boto:Path: /yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.404 DEBUG:boto:Data: 2024-04-16T20:21:01.404 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.404 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.404 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.404 DEBUG:boto:Params: {} 2024-04-16T20:21:01.404 DEBUG:boto:Token: None 2024-04-16T20:21:01.404 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.404 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064dde3b2b3ab62b-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.407 DEBUG:boto:b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.407 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.408 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.408 DEBUG:boto:path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.408 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.408 DEBUG:boto:Method: GET 2024-04-16T20:21:01.408 DEBUG:boto:Path: /yrjefh-56/?prefix=obj22 2024-04-16T20:21:01.408 DEBUG:boto:Data: 2024-04-16T20:21:01.408 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.408 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.408 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.408 DEBUG:boto:Params: {} 2024-04-16T20:21:01.408 DEBUG:boto:Token: None 2024-04-16T20:21:01.409 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.409 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e62579d1b54bcf-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.412 DEBUG:boto:b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:21:01.412 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.412 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.412 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.413 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.413 DEBUG:boto:Method: GET 2024-04-16T20:21:01.413 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.413 DEBUG:boto:Data: 2024-04-16T20:21:01.413 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.413 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.413 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.413 DEBUG:boto:Params: {} 2024-04-16T20:21:01.413 DEBUG:boto:Token: None 2024-04-16T20:21:01.413 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.414 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T20:21:01.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T20:21:01.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b97f7b0bdaf974e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.462 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.463 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.463 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.463 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.463 DEBUG:boto:Method: GET 2024-04-16T20:21:01.463 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.463 DEBUG:boto:Data: 2024-04-16T20:21:01.463 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.463 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.463 DEBUG:boto:Params: {} 2024-04-16T20:21:01.463 DEBUG:boto:Token: None 2024-04-16T20:21:01.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T20:21:01.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T20:21:01.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082fa0dfe5173574f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.466 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.467 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.467 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.467 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.467 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.467 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.467 DEBUG:boto:Data: 2024-04-16T20:21:01.467 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.467 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.467 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.467 DEBUG:boto:Params: {} 2024-04-16T20:21:01.467 DEBUG:boto:Token: None 2024-04-16T20:21:01.468 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk= 2024-04-16T20:21:01.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk=', 'Content-Length': '0'} 2024-04-16T20:21:01.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1efcd2d057de32-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.470 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.470 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.471 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.471 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.471 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.471 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.471 DEBUG:boto:Data: 2024-04-16T20:21:01.471 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.471 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.471 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.471 DEBUG:boto:Params: {} 2024-04-16T20:21:01.471 DEBUG:boto:Token: None 2024-04-16T20:21:01.471 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk= 2024-04-16T20:21:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk=', 'Content-Length': '0'} 2024-04-16T20:21:01.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcd9c5428724673-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:21:01.474 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.474 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.474 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.474 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.474 DEBUG:boto:Method: GET 2024-04-16T20:21:01.474 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.474 DEBUG:boto:Data: 2024-04-16T20:21:01.474 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.475 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.475 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.475 DEBUG:boto:Params: {} 2024-04-16T20:21:01.475 DEBUG:boto:Token: None 2024-04-16T20:21:01.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T20:21:01.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T20:21:01.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028db988c2e8a69ac-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.478 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.478 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.478 DEBUG:boto:path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.478 DEBUG:boto:auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.478 DEBUG:boto:Method: GET 2024-04-16T20:21:01.478 DEBUG:boto:Path: /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.478 DEBUG:boto:Data: 2024-04-16T20:21:01.478 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.478 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.478 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.479 DEBUG:boto:Params: {} 2024-04-16T20:21:01.479 DEBUG:boto:Token: None 2024-04-16T20:21:01.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23?versionId=null 2024-04-16T20:21:01.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T20:21:01.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T20:21:01.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbbf2a1bb80de34-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.526 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.526 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.527 DEBUG:boto:path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.527 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.527 DEBUG:boto:Method: GET 2024-04-16T20:21:01.527 DEBUG:boto:Path: /yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.527 DEBUG:boto:Data: 2024-04-16T20:21:01.527 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.527 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.527 DEBUG:boto:Params: {} 2024-04-16T20:21:01.527 DEBUG:boto:Token: None 2024-04-16T20:21:01.527 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005923d7c11bc5c477-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.530 DEBUG:boto:b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.531 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.531 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.531 DEBUG:boto:path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.531 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.531 DEBUG:boto:Method: GET 2024-04-16T20:21:01.531 DEBUG:boto:Path: /yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.531 DEBUG:boto:Data: 2024-04-16T20:21:01.531 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.532 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.532 DEBUG:boto:Params: {} 2024-04-16T20:21:01.532 DEBUG:boto:Token: None 2024-04-16T20:21:01.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec572b92ec82960d-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.535 DEBUG:boto:b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.536 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.536 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.536 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.536 DEBUG:boto:Path: /yrjefh-56/obj23 2024-04-16T20:21:01.536 DEBUG:boto:Data: 2024-04-16T20:21:01.536 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.536 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.536 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.536 DEBUG:boto:Params: {} 2024-04-16T20:21:01.537 DEBUG:boto:Token: None 2024-04-16T20:21:01.537 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23 2024-04-16T20:21:01.537 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ= 2024-04-16T20:21:01.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005359a31b7661948a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.539 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.539 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.539 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.539 DEBUG:boto:Path: /yrjefh-56/obj23 2024-04-16T20:21:01.539 DEBUG:boto:Data: 2024-04-16T20:21:01.539 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.540 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.540 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.540 DEBUG:boto:Params: {} 2024-04-16T20:21:01.540 DEBUG:boto:Token: None 2024-04-16T20:21:01.540 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23 2024-04-16T20:21:01.540 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ= 2024-04-16T20:21:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e700ecf2425762-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-16T20:21:01.543 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.543 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.543 DEBUG:boto:Method: GET 2024-04-16T20:21:01.543 DEBUG:boto:Path: /yrjefh-56/obj23 2024-04-16T20:21:01.543 DEBUG:boto:Data: 2024-04-16T20:21:01.543 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.543 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.543 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.543 DEBUG:boto:Params: {} 2024-04-16T20:21:01.543 DEBUG:boto:Token: None 2024-04-16T20:21:01.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23 2024-04-16T20:21:01.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8= 2024-04-16T20:21:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8=', 'Content-Length': '0'} 2024-04-16T20:21:01.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004746a1403eefac34-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.546 DEBUG:boto:path=/yrjefh-56/obj23 2024-04-16T20:21:01.546 DEBUG:boto:auth_path=/yrjefh-56/obj23 2024-04-16T20:21:01.546 DEBUG:boto:Method: GET 2024-04-16T20:21:01.546 DEBUG:boto:Path: /yrjefh-56/obj23 2024-04-16T20:21:01.546 DEBUG:boto:Data: 2024-04-16T20:21:01.546 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.547 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.547 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.547 DEBUG:boto:Params: {} 2024-04-16T20:21:01.547 DEBUG:boto:Token: None 2024-04-16T20:21:01.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj23 2024-04-16T20:21:01.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8= 2024-04-16T20:21:01.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8=', 'Content-Length': '0'} 2024-04-16T20:21:01.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7dde34d7177cad0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.594 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.595 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.595 DEBUG:boto:path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.595 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.595 DEBUG:boto:Method: GET 2024-04-16T20:21:01.595 DEBUG:boto:Path: /yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.595 DEBUG:boto:Data: 2024-04-16T20:21:01.595 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.595 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.595 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.595 DEBUG:boto:Params: {} 2024-04-16T20:21:01.595 DEBUG:boto:Token: None 2024-04-16T20:21:01.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.596 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11a94880bb3ee49-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.599 DEBUG:boto:b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.599 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.599 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.599 DEBUG:boto:path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.599 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.599 DEBUG:boto:Method: GET 2024-04-16T20:21:01.599 DEBUG:boto:Path: /yrjefh-56/?prefix=obj23 2024-04-16T20:21:01.600 DEBUG:boto:Data: 2024-04-16T20:21:01.600 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.600 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.600 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.600 DEBUG:boto:Params: {} 2024-04-16T20:21:01.600 DEBUG:boto:Token: None 2024-04-16T20:21:01.600 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ddbe62548572e9f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.606 DEBUG:boto:b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:21:01.606 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.606 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.606 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.606 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.607 DEBUG:boto:Method: GET 2024-04-16T20:21:01.607 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.607 DEBUG:boto:Data: 2024-04-16T20:21:01.607 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.607 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.607 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.607 DEBUG:boto:Params: {} 2024-04-16T20:21:01.607 DEBUG:boto:Token: None 2024-04-16T20:21:01.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T20:21:01.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T20:21:01.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4eb94837957d65-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.650 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.650 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.651 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.651 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.651 DEBUG:boto:Method: GET 2024-04-16T20:21:01.651 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.651 DEBUG:boto:Data: 2024-04-16T20:21:01.651 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.651 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.651 DEBUG:boto:Params: {} 2024-04-16T20:21:01.651 DEBUG:boto:Token: None 2024-04-16T20:21:01.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T20:21:01.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T20:21:01.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f8c4c6149d89a9-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.654 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.654 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.654 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.654 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.654 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.654 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.654 DEBUG:boto:Data: 2024-04-16T20:21:01.654 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.655 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.655 DEBUG:boto:Params: {} 2024-04-16T20:21:01.655 DEBUG:boto:Token: None 2024-04-16T20:21:01.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8= 2024-04-16T20:21:01.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8=', 'Content-Length': '0'} 2024-04-16T20:21:01.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020679f466476041-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.657 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.657 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.658 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.658 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.658 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.658 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.658 DEBUG:boto:Data: 2024-04-16T20:21:01.658 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.658 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.658 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.658 DEBUG:boto:Params: {} 2024-04-16T20:21:01.658 DEBUG:boto:Token: None 2024-04-16T20:21:01.658 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8= 2024-04-16T20:21:01.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8=', 'Content-Length': '0'} 2024-04-16T20:21:01.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c130cbfbd09027b-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:21:01.661 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.661 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.661 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.661 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.661 DEBUG:boto:Method: GET 2024-04-16T20:21:01.661 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.661 DEBUG:boto:Data: 2024-04-16T20:21:01.661 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.662 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.662 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.662 DEBUG:boto:Params: {} 2024-04-16T20:21:01.662 DEBUG:boto:Token: None 2024-04-16T20:21:01.662 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.662 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T20:21:01.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T20:21:01.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d247c20b010d205c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.664 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.665 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.665 DEBUG:boto:path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.665 DEBUG:boto:auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.665 DEBUG:boto:Method: GET 2024-04-16T20:21:01.665 DEBUG:boto:Path: /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.665 DEBUG:boto:Data: 2024-04-16T20:21:01.665 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.665 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.665 DEBUG:boto:Params: {} 2024-04-16T20:21:01.665 DEBUG:boto:Token: None 2024-04-16T20:21:01.666 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29?versionId=null 2024-04-16T20:21:01.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T20:21:01.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T20:21:01.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebcbae2fe306ade0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.668 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.668 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.668 DEBUG:boto:path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.668 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.668 DEBUG:boto:Method: GET 2024-04-16T20:21:01.668 DEBUG:boto:Path: /yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.668 DEBUG:boto:Data: 2024-04-16T20:21:01.668 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.669 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.669 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.669 DEBUG:boto:Params: {} 2024-04-16T20:21:01.669 DEBUG:boto:Token: None 2024-04-16T20:21:01.669 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.669 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ea8ac53cd2f8e5-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.672 DEBUG:boto:b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.672 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.672 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.672 DEBUG:boto:path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.672 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.673 DEBUG:boto:Method: GET 2024-04-16T20:21:01.673 DEBUG:boto:Path: /yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.673 DEBUG:boto:Data: 2024-04-16T20:21:01.673 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.673 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.673 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.673 DEBUG:boto:Params: {} 2024-04-16T20:21:01.673 DEBUG:boto:Token: None 2024-04-16T20:21:01.673 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.673 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056bdf5d8c7b87b4c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.677 DEBUG:boto:b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.677 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.677 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.677 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.678 DEBUG:boto:Path: /yrjefh-56/obj29 2024-04-16T20:21:01.678 DEBUG:boto:Data: 2024-04-16T20:21:01.678 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.678 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.678 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.678 DEBUG:boto:Params: {} 2024-04-16T20:21:01.678 DEBUG:boto:Token: None 2024-04-16T20:21:01.678 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29 2024-04-16T20:21:01.678 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0= 2024-04-16T20:21:01.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0=', 'Content-Length': '0'} 2024-04-16T20:21:01.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b76576ddef3e3cc-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.680 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.681 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.681 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.681 DEBUG:boto:Path: /yrjefh-56/obj29 2024-04-16T20:21:01.681 DEBUG:boto:Data: 2024-04-16T20:21:01.681 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.681 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.681 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.681 DEBUG:boto:Params: {} 2024-04-16T20:21:01.681 DEBUG:boto:Token: None 2024-04-16T20:21:01.681 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29 2024-04-16T20:21:01.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0= 2024-04-16T20:21:01.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0=', 'Content-Length': '0'} 2024-04-16T20:21:01.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370101d0ea393521-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-16T20:21:01.684 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.684 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.684 DEBUG:boto:Method: GET 2024-04-16T20:21:01.684 DEBUG:boto:Path: /yrjefh-56/obj29 2024-04-16T20:21:01.684 DEBUG:boto:Data: 2024-04-16T20:21:01.684 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.684 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.684 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.684 DEBUG:boto:Params: {} 2024-04-16T20:21:01.685 DEBUG:boto:Token: None 2024-04-16T20:21:01.685 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29 2024-04-16T20:21:01.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg= 2024-04-16T20:21:01.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg=', 'Content-Length': '0'} 2024-04-16T20:21:01.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f13acecad603a3-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.730 DEBUG:boto:path=/yrjefh-56/obj29 2024-04-16T20:21:01.730 DEBUG:boto:auth_path=/yrjefh-56/obj29 2024-04-16T20:21:01.731 DEBUG:boto:Method: GET 2024-04-16T20:21:01.731 DEBUG:boto:Path: /yrjefh-56/obj29 2024-04-16T20:21:01.731 DEBUG:boto:Data: 2024-04-16T20:21:01.731 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.731 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.731 DEBUG:boto:Params: {} 2024-04-16T20:21:01.731 DEBUG:boto:Token: None 2024-04-16T20:21:01.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj29 2024-04-16T20:21:01.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg= 2024-04-16T20:21:01.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg=', 'Content-Length': '0'} 2024-04-16T20:21:01.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4c64e739bac3de-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.734 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.734 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.734 DEBUG:boto:path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.734 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.734 DEBUG:boto:Method: GET 2024-04-16T20:21:01.734 DEBUG:boto:Path: /yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.734 DEBUG:boto:Data: 2024-04-16T20:21:01.734 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.734 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.735 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.735 DEBUG:boto:Params: {} 2024-04-16T20:21:01.735 DEBUG:boto:Token: None 2024-04-16T20:21:01.735 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d0cdc81dab9a78-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.782 DEBUG:boto:b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.783 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.783 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.783 DEBUG:boto:path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.783 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.783 DEBUG:boto:Method: GET 2024-04-16T20:21:01.784 DEBUG:boto:Path: /yrjefh-56/?prefix=obj29 2024-04-16T20:21:01.784 DEBUG:boto:Data: 2024-04-16T20:21:01.784 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.784 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.784 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.784 DEBUG:boto:Params: {} 2024-04-16T20:21:01.784 DEBUG:boto:Token: None 2024-04-16T20:21:01.784 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.784 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b581794fb8f5a4-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.788 DEBUG:boto:b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:21:01.788 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.789 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.789 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.789 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.789 DEBUG:boto:Method: GET 2024-04-16T20:21:01.789 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.789 DEBUG:boto:Data: 2024-04-16T20:21:01.789 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.789 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.789 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.789 DEBUG:boto:Params: {} 2024-04-16T20:21:01.789 DEBUG:boto:Token: None 2024-04-16T20:21:01.789 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.790 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T20:21:01.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dab15723f6e3b17-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.834 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.834 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.835 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.835 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.835 DEBUG:boto:Method: GET 2024-04-16T20:21:01.835 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.835 DEBUG:boto:Data: 2024-04-16T20:21:01.835 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.835 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.835 DEBUG:boto:Params: {} 2024-04-16T20:21:01.835 DEBUG:boto:Token: None 2024-04-16T20:21:01.835 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T20:21:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547712eb591b57cf-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.838 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.838 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.838 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.838 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.839 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.839 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.839 DEBUG:boto:Data: 2024-04-16T20:21:01.839 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.839 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.839 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.839 DEBUG:boto:Params: {} 2024-04-16T20:21:01.839 DEBUG:boto:Token: None 2024-04-16T20:21:01.839 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.839 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY= 2024-04-16T20:21:01.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY=', 'Content-Length': '0'} 2024-04-16T20:21:01.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1ad6a32797c021-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.842 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.842 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.842 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.842 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.842 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.842 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.842 DEBUG:boto:Data: 2024-04-16T20:21:01.842 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.842 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.842 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.842 DEBUG:boto:Params: {} 2024-04-16T20:21:01.843 DEBUG:boto:Token: None 2024-04-16T20:21:01.843 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.843 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY= 2024-04-16T20:21:01.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY=', 'Content-Length': '0'} 2024-04-16T20:21:01.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a814d0f1bcf2b3e7-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:21:01.845 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.845 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.845 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.846 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.846 DEBUG:boto:Method: GET 2024-04-16T20:21:01.846 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.846 DEBUG:boto:Data: 2024-04-16T20:21:01.846 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.846 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.846 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.846 DEBUG:boto:Params: {} 2024-04-16T20:21:01.846 DEBUG:boto:Token: None 2024-04-16T20:21:01.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T20:21:01.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b451368bb0c25a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.890 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.890 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.890 DEBUG:boto:path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.891 DEBUG:boto:auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.891 DEBUG:boto:Method: GET 2024-04-16T20:21:01.891 DEBUG:boto:Path: /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.891 DEBUG:boto:Data: 2024-04-16T20:21:01.891 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.891 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.891 DEBUG:boto:Params: {} 2024-04-16T20:21:01.891 DEBUG:boto:Token: None 2024-04-16T20:21:01.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31?versionId=null 2024-04-16T20:21:01.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T20:21:01.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T20:21:01.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f000ce0cc9afff13-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.894 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.894 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.894 DEBUG:boto:path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.894 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.894 DEBUG:boto:Method: GET 2024-04-16T20:21:01.895 DEBUG:boto:Path: /yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.895 DEBUG:boto:Data: 2024-04-16T20:21:01.895 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.895 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.895 DEBUG:boto:Params: {} 2024-04-16T20:21:01.895 DEBUG:boto:Token: None 2024-04-16T20:21:01.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137d9436652510e5-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.938 DEBUG:boto:b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.939 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.939 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.939 DEBUG:boto:path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.939 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.939 DEBUG:boto:Method: GET 2024-04-16T20:21:01.939 DEBUG:boto:Path: /yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.939 DEBUG:boto:Data: 2024-04-16T20:21:01.939 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.939 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.940 DEBUG:boto:Params: {} 2024-04-16T20:21:01.940 DEBUG:boto:Token: None 2024-04-16T20:21:01.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:01.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba0ecba55a7f5a60-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.943 DEBUG:boto:b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:01.944 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.944 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.944 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.944 DEBUG:boto:Path: /yrjefh-56/obj31 2024-04-16T20:21:01.944 DEBUG:boto:Data: 2024-04-16T20:21:01.944 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.944 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.944 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.944 DEBUG:boto:Params: {} 2024-04-16T20:21:01.944 DEBUG:boto:Token: None 2024-04-16T20:21:01.945 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31 2024-04-16T20:21:01.945 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU= 2024-04-16T20:21:01.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU=', 'Content-Length': '0'} 2024-04-16T20:21:01.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b799be90bef8126e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.947 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.947 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.947 DEBUG:boto:Method: HEAD 2024-04-16T20:21:01.947 DEBUG:boto:Path: /yrjefh-56/obj31 2024-04-16T20:21:01.947 DEBUG:boto:Data: 2024-04-16T20:21:01.947 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.947 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.947 DEBUG:boto:Params: {} 2024-04-16T20:21:01.948 DEBUG:boto:Token: None 2024-04-16T20:21:01.948 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31 2024-04-16T20:21:01.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU= 2024-04-16T20:21:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU=', 'Content-Length': '0'} 2024-04-16T20:21:01.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afee03c026bb1725-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-16T20:21:01.950 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.950 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.950 DEBUG:boto:Method: GET 2024-04-16T20:21:01.951 DEBUG:boto:Path: /yrjefh-56/obj31 2024-04-16T20:21:01.951 DEBUG:boto:Data: 2024-04-16T20:21:01.951 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.951 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.951 DEBUG:boto:Params: {} 2024-04-16T20:21:01.951 DEBUG:boto:Token: None 2024-04-16T20:21:01.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31 2024-04-16T20:21:01.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw= 2024-04-16T20:21:01.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw=', 'Content-Length': '0'} 2024-04-16T20:21:01.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008865ff3b9f199126-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.994 DEBUG:boto:path=/yrjefh-56/obj31 2024-04-16T20:21:01.994 DEBUG:boto:auth_path=/yrjefh-56/obj31 2024-04-16T20:21:01.995 DEBUG:boto:Method: GET 2024-04-16T20:21:01.995 DEBUG:boto:Path: /yrjefh-56/obj31 2024-04-16T20:21:01.995 DEBUG:boto:Data: 2024-04-16T20:21:01.995 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:01.995 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.995 DEBUG:boto:Params: {} 2024-04-16T20:21:01.995 DEBUG:boto:Token: None 2024-04-16T20:21:01.995 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/obj31 2024-04-16T20:21:01.995 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw= 2024-04-16T20:21:01.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw=', 'Content-Length': '0'} 2024-04-16T20:21:01.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba0267687f9f29b1-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:01.998 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:01.998 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:01.998 DEBUG:boto:path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.999 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.999 DEBUG:boto:Method: GET 2024-04-16T20:21:01.999 DEBUG:boto:Path: /yrjefh-56/?prefix=obj31 2024-04-16T20:21:01.999 DEBUG:boto:Data: 2024-04-16T20:21:01.999 DEBUG:boto:Headers: {} 2024-04-16T20:21:01.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:01.999 DEBUG:boto:Port: 8000 2024-04-16T20:21:01.999 DEBUG:boto:Params: {} 2024-04-16T20:21:01.999 DEBUG:boto:Token: None 2024-04-16T20:21:01.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:01 GMT /yrjefh-56/ 2024-04-16T20:21:01.999 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T20:21:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T20:21:02.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c979fa09a8448fd-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.042 DEBUG:boto:b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.043 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.043 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.043 DEBUG:boto:path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:02.043 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T20:21:02.043 DEBUG:boto:Method: GET 2024-04-16T20:21:02.043 DEBUG:boto:Path: /yrjefh-56/?prefix=obj31 2024-04-16T20:21:02.043 DEBUG:boto:Data: 2024-04-16T20:21:02.043 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.043 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.043 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.044 DEBUG:boto:Params: {} 2024-04-16T20:21:02.044 DEBUG:boto:Token: None 2024-04-16T20:21:02.044 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.044 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022fa8fc1d3fe399b-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.047 DEBUG:boto:b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:21:02.048 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.048 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.048 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.048 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.048 DEBUG:boto:Method: GET 2024-04-16T20:21:02.048 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.048 DEBUG:boto:Data: 2024-04-16T20:21:02.048 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.048 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.049 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.049 DEBUG:boto:Params: {} 2024-04-16T20:21:02.049 DEBUG:boto:Token: None 2024-04-16T20:21:02.049 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.049 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T20:21:02.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T20:21:02.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d7169ac8bf9f06-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.094 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.094 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.094 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.095 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.095 DEBUG:boto:Method: GET 2024-04-16T20:21:02.095 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.095 DEBUG:boto:Data: 2024-04-16T20:21:02.095 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.095 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.095 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.095 DEBUG:boto:Params: {} 2024-04-16T20:21:02.095 DEBUG:boto:Token: None 2024-04-16T20:21:02.095 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.095 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T20:21:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T20:21:02.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e24ded074ad69ed-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.097 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.098 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.098 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.098 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.098 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.098 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.098 DEBUG:boto:Data: 2024-04-16T20:21:02.098 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.098 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.098 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.098 DEBUG:boto:Params: {} 2024-04-16T20:21:02.099 DEBUG:boto:Token: None 2024-04-16T20:21:02.099 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.099 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30= 2024-04-16T20:21:02.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30=', 'Content-Length': '0'} 2024-04-16T20:21:02.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3eafbd020caef73-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.101 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.101 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.101 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.101 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.101 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.101 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.101 DEBUG:boto:Data: 2024-04-16T20:21:02.101 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.101 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.102 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.102 DEBUG:boto:Params: {} 2024-04-16T20:21:02.102 DEBUG:boto:Token: None 2024-04-16T20:21:02.102 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.102 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30= 2024-04-16T20:21:02.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30=', 'Content-Length': '0'} 2024-04-16T20:21:02.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014e2ca76172a042-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:21:02.105 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.105 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.105 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.105 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.105 DEBUG:boto:Method: GET 2024-04-16T20:21:02.105 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.105 DEBUG:boto:Data: 2024-04-16T20:21:02.105 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.105 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.105 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.105 DEBUG:boto:Params: {} 2024-04-16T20:21:02.106 DEBUG:boto:Token: None 2024-04-16T20:21:02.106 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T20:21:02.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T20:21:02.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df73711eaa4e40f-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.150 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.150 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.151 DEBUG:boto:path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.151 DEBUG:boto:auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.151 DEBUG:boto:Method: GET 2024-04-16T20:21:02.151 DEBUG:boto:Path: /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.151 DEBUG:boto:Data: 2024-04-16T20:21:02.151 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.151 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.151 DEBUG:boto:Params: {} 2024-04-16T20:21:02.151 DEBUG:boto:Token: None 2024-04-16T20:21:02.151 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33?versionId=null 2024-04-16T20:21:02.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T20:21:02.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T20:21:02.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfc4f52b389d921-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.154 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.154 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.154 DEBUG:boto:path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.154 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.154 DEBUG:boto:Method: GET 2024-04-16T20:21:02.154 DEBUG:boto:Path: /yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.154 DEBUG:boto:Data: 2024-04-16T20:21:02.154 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.155 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.155 DEBUG:boto:Params: {} 2024-04-16T20:21:02.155 DEBUG:boto:Token: None 2024-04-16T20:21:02.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe1d3f286bf6f41-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.198 DEBUG:boto:b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.199 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.199 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.199 DEBUG:boto:path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.199 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.199 DEBUG:boto:Method: GET 2024-04-16T20:21:02.199 DEBUG:boto:Path: /yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.199 DEBUG:boto:Data: 2024-04-16T20:21:02.199 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.199 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.200 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.200 DEBUG:boto:Params: {} 2024-04-16T20:21:02.200 DEBUG:boto:Token: None 2024-04-16T20:21:02.200 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.200 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068450862f740c8d0-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.204 DEBUG:boto:b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.204 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.204 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.204 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.204 DEBUG:boto:Path: /yrjefh-56/obj33 2024-04-16T20:21:02.205 DEBUG:boto:Data: 2024-04-16T20:21:02.205 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.205 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.205 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.205 DEBUG:boto:Params: {} 2024-04-16T20:21:02.205 DEBUG:boto:Token: None 2024-04-16T20:21:02.205 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33 2024-04-16T20:21:02.205 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E= 2024-04-16T20:21:02.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E=', 'Content-Length': '0'} 2024-04-16T20:21:02.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568c950274e4b998-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.207 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.207 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.207 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.207 DEBUG:boto:Path: /yrjefh-56/obj33 2024-04-16T20:21:02.208 DEBUG:boto:Data: 2024-04-16T20:21:02.208 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.208 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.208 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.208 DEBUG:boto:Params: {} 2024-04-16T20:21:02.208 DEBUG:boto:Token: None 2024-04-16T20:21:02.208 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33 2024-04-16T20:21:02.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E= 2024-04-16T20:21:02.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E=', 'Content-Length': '0'} 2024-04-16T20:21:02.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033846bbb27688d31-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:21:02.211 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.211 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.211 DEBUG:boto:Method: GET 2024-04-16T20:21:02.211 DEBUG:boto:Path: /yrjefh-56/obj33 2024-04-16T20:21:02.211 DEBUG:boto:Data: 2024-04-16T20:21:02.211 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.211 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.211 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.211 DEBUG:boto:Params: {} 2024-04-16T20:21:02.211 DEBUG:boto:Token: None 2024-04-16T20:21:02.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33 2024-04-16T20:21:02.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY= 2024-04-16T20:21:02.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY=', 'Content-Length': '0'} 2024-04-16T20:21:02.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007f26dafda34cd07-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.254 DEBUG:boto:path=/yrjefh-56/obj33 2024-04-16T20:21:02.254 DEBUG:boto:auth_path=/yrjefh-56/obj33 2024-04-16T20:21:02.255 DEBUG:boto:Method: GET 2024-04-16T20:21:02.255 DEBUG:boto:Path: /yrjefh-56/obj33 2024-04-16T20:21:02.255 DEBUG:boto:Data: 2024-04-16T20:21:02.255 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.255 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.255 DEBUG:boto:Params: {} 2024-04-16T20:21:02.255 DEBUG:boto:Token: None 2024-04-16T20:21:02.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj33 2024-04-16T20:21:02.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY= 2024-04-16T20:21:02.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY=', 'Content-Length': '0'} 2024-04-16T20:21:02.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a372574ec550db-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.258 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.258 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.258 DEBUG:boto:path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.258 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.258 DEBUG:boto:Method: GET 2024-04-16T20:21:02.258 DEBUG:boto:Path: /yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.258 DEBUG:boto:Data: 2024-04-16T20:21:02.258 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.258 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.258 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.259 DEBUG:boto:Params: {} 2024-04-16T20:21:02.259 DEBUG:boto:Token: None 2024-04-16T20:21:02.259 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a49bb2bf440d1d-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.302 DEBUG:boto:b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.303 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.303 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.303 DEBUG:boto:path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.303 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.303 DEBUG:boto:Method: GET 2024-04-16T20:21:02.303 DEBUG:boto:Path: /yrjefh-56/?prefix=obj33 2024-04-16T20:21:02.303 DEBUG:boto:Data: 2024-04-16T20:21:02.304 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.304 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.304 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.304 DEBUG:boto:Params: {} 2024-04-16T20:21:02.304 DEBUG:boto:Token: None 2024-04-16T20:21:02.304 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.304 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3ebcc1fabc40dc-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.308 DEBUG:boto:b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:21:02.309 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.309 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.309 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.309 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.309 DEBUG:boto:Method: GET 2024-04-16T20:21:02.309 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.309 DEBUG:boto:Data: 2024-04-16T20:21:02.309 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.309 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.309 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.309 DEBUG:boto:Params: {} 2024-04-16T20:21:02.310 DEBUG:boto:Token: None 2024-04-16T20:21:02.310 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.310 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T20:21:02.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T20:21:02.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd42ee313571a3b-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.354 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.355 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.355 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.355 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.355 DEBUG:boto:Method: GET 2024-04-16T20:21:02.355 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.355 DEBUG:boto:Data: 2024-04-16T20:21:02.355 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.355 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.355 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.355 DEBUG:boto:Params: {} 2024-04-16T20:21:02.355 DEBUG:boto:Token: None 2024-04-16T20:21:02.356 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.356 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T20:21:02.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T20:21:02.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999e6adcf87c63ff-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.358 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.358 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.358 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.358 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.359 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.359 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.359 DEBUG:boto:Data: 2024-04-16T20:21:02.359 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.359 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.359 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.359 DEBUG:boto:Params: {} 2024-04-16T20:21:02.359 DEBUG:boto:Token: None 2024-04-16T20:21:02.359 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.360 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM= 2024-04-16T20:21:02.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM=', 'Content-Length': '0'} 2024-04-16T20:21:02.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93a5bc467d6df7c-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.362 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.362 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.362 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.362 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.362 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.362 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.362 DEBUG:boto:Data: 2024-04-16T20:21:02.362 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.362 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.362 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.362 DEBUG:boto:Params: {} 2024-04-16T20:21:02.363 DEBUG:boto:Token: None 2024-04-16T20:21:02.363 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.363 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM= 2024-04-16T20:21:02.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM=', 'Content-Length': '0'} 2024-04-16T20:21:02.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbb443242090439-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:21:02.365 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.365 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.365 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.365 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.365 DEBUG:boto:Method: GET 2024-04-16T20:21:02.365 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.365 DEBUG:boto:Data: 2024-04-16T20:21:02.365 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.365 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.366 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.366 DEBUG:boto:Params: {} 2024-04-16T20:21:02.366 DEBUG:boto:Token: None 2024-04-16T20:21:02.366 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.366 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T20:21:02.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T20:21:02.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6c40628640c9a4-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.410 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.411 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.411 DEBUG:boto:path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.411 DEBUG:boto:auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.411 DEBUG:boto:Method: GET 2024-04-16T20:21:02.411 DEBUG:boto:Path: /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.411 DEBUG:boto:Data: 2024-04-16T20:21:02.411 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.411 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.411 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.412 DEBUG:boto:Params: {} 2024-04-16T20:21:02.412 DEBUG:boto:Token: None 2024-04-16T20:21:02.412 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34?versionId=null 2024-04-16T20:21:02.412 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T20:21:02.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T20:21:02.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089acc7eeb6432f33-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.415 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.415 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.415 DEBUG:boto:path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.415 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.415 DEBUG:boto:Method: GET 2024-04-16T20:21:02.415 DEBUG:boto:Path: /yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.415 DEBUG:boto:Data: 2024-04-16T20:21:02.415 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.415 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.415 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.416 DEBUG:boto:Params: {} 2024-04-16T20:21:02.416 DEBUG:boto:Token: None 2024-04-16T20:21:02.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46bfb6aa8e196a9-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.462 DEBUG:boto:b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.463 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.463 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.463 DEBUG:boto:path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.463 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.463 DEBUG:boto:Method: GET 2024-04-16T20:21:02.463 DEBUG:boto:Path: /yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.463 DEBUG:boto:Data: 2024-04-16T20:21:02.463 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.463 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.463 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.464 DEBUG:boto:Params: {} 2024-04-16T20:21:02.464 DEBUG:boto:Token: None 2024-04-16T20:21:02.464 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.464 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6eb2646f492d276-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.467 DEBUG:boto:b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.468 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.468 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.468 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.468 DEBUG:boto:Path: /yrjefh-56/obj34 2024-04-16T20:21:02.468 DEBUG:boto:Data: 2024-04-16T20:21:02.468 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.468 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.468 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.468 DEBUG:boto:Params: {} 2024-04-16T20:21:02.469 DEBUG:boto:Token: None 2024-04-16T20:21:02.469 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34 2024-04-16T20:21:02.469 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8= 2024-04-16T20:21:02.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8=', 'Content-Length': '0'} 2024-04-16T20:21:02.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb88b397896e3406-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.471 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.471 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.471 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.471 DEBUG:boto:Path: /yrjefh-56/obj34 2024-04-16T20:21:02.471 DEBUG:boto:Data: 2024-04-16T20:21:02.471 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.472 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.472 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.472 DEBUG:boto:Params: {} 2024-04-16T20:21:02.472 DEBUG:boto:Token: None 2024-04-16T20:21:02.472 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34 2024-04-16T20:21:02.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8= 2024-04-16T20:21:02.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8=', 'Content-Length': '0'} 2024-04-16T20:21:02.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002545ce037e9a010a-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-16T20:21:02.474 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.474 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.475 DEBUG:boto:Method: GET 2024-04-16T20:21:02.475 DEBUG:boto:Path: /yrjefh-56/obj34 2024-04-16T20:21:02.475 DEBUG:boto:Data: 2024-04-16T20:21:02.475 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.475 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.475 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.475 DEBUG:boto:Params: {} 2024-04-16T20:21:02.475 DEBUG:boto:Token: None 2024-04-16T20:21:02.475 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34 2024-04-16T20:21:02.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg= 2024-04-16T20:21:02.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg=', 'Content-Length': '0'} 2024-04-16T20:21:02.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cba898e224fbbc0-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.518 DEBUG:boto:path=/yrjefh-56/obj34 2024-04-16T20:21:02.518 DEBUG:boto:auth_path=/yrjefh-56/obj34 2024-04-16T20:21:02.519 DEBUG:boto:Method: GET 2024-04-16T20:21:02.519 DEBUG:boto:Path: /yrjefh-56/obj34 2024-04-16T20:21:02.519 DEBUG:boto:Data: 2024-04-16T20:21:02.519 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.519 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.519 DEBUG:boto:Params: {} 2024-04-16T20:21:02.519 DEBUG:boto:Token: None 2024-04-16T20:21:02.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj34 2024-04-16T20:21:02.519 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg= 2024-04-16T20:21:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg=', 'Content-Length': '0'} 2024-04-16T20:21:02.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a9cf2e101c3255-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.522 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.522 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.522 DEBUG:boto:path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.522 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.522 DEBUG:boto:Method: GET 2024-04-16T20:21:02.522 DEBUG:boto:Path: /yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.522 DEBUG:boto:Data: 2024-04-16T20:21:02.523 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.523 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.523 DEBUG:boto:Params: {} 2024-04-16T20:21:02.523 DEBUG:boto:Token: None 2024-04-16T20:21:02.523 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.523 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa81c84b53065b6a-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.566 DEBUG:boto:b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.567 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.567 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.567 DEBUG:boto:path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.567 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.567 DEBUG:boto:Method: GET 2024-04-16T20:21:02.567 DEBUG:boto:Path: /yrjefh-56/?prefix=obj34 2024-04-16T20:21:02.567 DEBUG:boto:Data: 2024-04-16T20:21:02.567 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.568 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.568 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.568 DEBUG:boto:Params: {} 2024-04-16T20:21:02.568 DEBUG:boto:Token: None 2024-04-16T20:21:02.568 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adf5dd3d196618f-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.572 DEBUG:boto:b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:21:02.572 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.573 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.573 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.573 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.573 DEBUG:boto:Method: GET 2024-04-16T20:21:02.573 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.573 DEBUG:boto:Data: 2024-04-16T20:21:02.573 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.573 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.573 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.573 DEBUG:boto:Params: {} 2024-04-16T20:21:02.573 DEBUG:boto:Token: None 2024-04-16T20:21:02.574 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.574 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T20:21:02.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T20:21:02.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007946bc1a40b5b3cd-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.618 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.618 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.619 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.619 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.619 DEBUG:boto:Method: GET 2024-04-16T20:21:02.619 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.619 DEBUG:boto:Data: 2024-04-16T20:21:02.619 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.619 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.619 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.619 DEBUG:boto:Params: {} 2024-04-16T20:21:02.619 DEBUG:boto:Token: None 2024-04-16T20:21:02.620 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.620 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T20:21:02.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T20:21:02.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d2f831a3fd5a6e-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.622 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.623 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.623 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.623 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.623 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.623 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.623 DEBUG:boto:Data: 2024-04-16T20:21:02.623 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.623 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.623 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.623 DEBUG:boto:Params: {} 2024-04-16T20:21:02.624 DEBUG:boto:Token: None 2024-04-16T20:21:02.624 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U= 2024-04-16T20:21:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U=', 'Content-Length': '0'} 2024-04-16T20:21:02.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a311924d20f465b6-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.626 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.626 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.626 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.626 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.627 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.627 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.627 DEBUG:boto:Data: 2024-04-16T20:21:02.627 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.627 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.627 DEBUG:boto:Params: {} 2024-04-16T20:21:02.627 DEBUG:boto:Token: None 2024-04-16T20:21:02.627 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.627 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U= 2024-04-16T20:21:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U=', 'Content-Length': '0'} 2024-04-16T20:21:02.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f627540749ab21-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:21:02.630 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.630 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.630 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.630 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.630 DEBUG:boto:Method: GET 2024-04-16T20:21:02.630 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.630 DEBUG:boto:Data: 2024-04-16T20:21:02.630 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.631 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.631 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.631 DEBUG:boto:Params: {} 2024-04-16T20:21:02.631 DEBUG:boto:Token: None 2024-04-16T20:21:02.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T20:21:02.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T20:21:02.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9167564fe946fd4-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.674 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.675 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.675 DEBUG:boto:path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.675 DEBUG:boto:auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.675 DEBUG:boto:Method: GET 2024-04-16T20:21:02.675 DEBUG:boto:Path: /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.675 DEBUG:boto:Data: 2024-04-16T20:21:02.675 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.675 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.675 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.675 DEBUG:boto:Params: {} 2024-04-16T20:21:02.676 DEBUG:boto:Token: None 2024-04-16T20:21:02.676 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37?versionId=null 2024-04-16T20:21:02.676 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T20:21:02.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T20:21:02.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a839c64c312eefb6-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.679 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.679 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.679 DEBUG:boto:path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.679 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.679 DEBUG:boto:Method: GET 2024-04-16T20:21:02.679 DEBUG:boto:Path: /yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.679 DEBUG:boto:Data: 2024-04-16T20:21:02.679 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.679 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.679 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.679 DEBUG:boto:Params: {} 2024-04-16T20:21:02.680 DEBUG:boto:Token: None 2024-04-16T20:21:02.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.680 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7aec7d3e8caff10-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.726 DEBUG:boto:b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.727 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.727 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.727 DEBUG:boto:path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.727 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.727 DEBUG:boto:Method: GET 2024-04-16T20:21:02.728 DEBUG:boto:Path: /yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.728 DEBUG:boto:Data: 2024-04-16T20:21:02.728 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.728 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.728 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.728 DEBUG:boto:Params: {} 2024-04-16T20:21:02.728 DEBUG:boto:Token: None 2024-04-16T20:21:02.728 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dcf38c267f3b903-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.732 DEBUG:boto:b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.732 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.732 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.733 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.733 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:21:02.733 DEBUG:boto:Data: 2024-04-16T20:21:02.733 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.733 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.733 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.733 DEBUG:boto:Params: {} 2024-04-16T20:21:02.733 DEBUG:boto:Token: None 2024-04-16T20:21:02.733 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37 2024-04-16T20:21:02.733 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs= 2024-04-16T20:21:02.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs=', 'Content-Length': '0'} 2024-04-16T20:21:02.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801bbee0bab45d57-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.736 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.736 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.736 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.736 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:21:02.736 DEBUG:boto:Data: 2024-04-16T20:21:02.736 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.737 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.737 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.737 DEBUG:boto:Params: {} 2024-04-16T20:21:02.737 DEBUG:boto:Token: None 2024-04-16T20:21:02.737 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37 2024-04-16T20:21:02.737 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs= 2024-04-16T20:21:02.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs=', 'Content-Length': '0'} 2024-04-16T20:21:02.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ea05425c569e12d-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-16T20:21:02.740 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.740 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.740 DEBUG:boto:Method: GET 2024-04-16T20:21:02.740 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:21:02.740 DEBUG:boto:Data: 2024-04-16T20:21:02.740 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.740 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.740 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.740 DEBUG:boto:Params: {} 2024-04-16T20:21:02.740 DEBUG:boto:Token: None 2024-04-16T20:21:02.741 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37 2024-04-16T20:21:02.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag= 2024-04-16T20:21:02.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag=', 'Content-Length': '0'} 2024-04-16T20:21:02.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d881de548a52f07e-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.786 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:02.787 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:02.787 DEBUG:boto:Method: GET 2024-04-16T20:21:02.787 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:21:02.787 DEBUG:boto:Data: 2024-04-16T20:21:02.787 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.787 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.787 DEBUG:boto:Params: {} 2024-04-16T20:21:02.788 DEBUG:boto:Token: None 2024-04-16T20:21:02.788 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj37 2024-04-16T20:21:02.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag= 2024-04-16T20:21:02.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag=', 'Content-Length': '0'} 2024-04-16T20:21:02.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5b06f884e28553-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.791 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.791 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.791 DEBUG:boto:path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.791 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.791 DEBUG:boto:Method: GET 2024-04-16T20:21:02.791 DEBUG:boto:Path: /yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.791 DEBUG:boto:Data: 2024-04-16T20:21:02.791 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.791 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.792 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.792 DEBUG:boto:Params: {} 2024-04-16T20:21:02.792 DEBUG:boto:Token: None 2024-04-16T20:21:02.792 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b4980a080371e3-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.838 DEBUG:boto:b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.839 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.839 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.839 DEBUG:boto:path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.839 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.839 DEBUG:boto:Method: GET 2024-04-16T20:21:02.839 DEBUG:boto:Path: /yrjefh-56/?prefix=obj37 2024-04-16T20:21:02.840 DEBUG:boto:Data: 2024-04-16T20:21:02.840 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.840 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.840 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.840 DEBUG:boto:Params: {} 2024-04-16T20:21:02.840 DEBUG:boto:Token: None 2024-04-16T20:21:02.840 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.840 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b1b222a9853ecd3-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.844 DEBUG:boto:b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:21:02.845 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.845 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.845 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.845 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.845 DEBUG:boto:Method: GET 2024-04-16T20:21:02.845 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.845 DEBUG:boto:Data: 2024-04-16T20:21:02.845 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.845 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.845 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.846 DEBUG:boto:Params: {} 2024-04-16T20:21:02.846 DEBUG:boto:Token: None 2024-04-16T20:21:02.846 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.846 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T20:21:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T20:21:02.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2931094c01706d-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.890 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.890 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.890 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.891 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.891 DEBUG:boto:Method: GET 2024-04-16T20:21:02.891 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.891 DEBUG:boto:Data: 2024-04-16T20:21:02.891 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.891 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.891 DEBUG:boto:Params: {} 2024-04-16T20:21:02.891 DEBUG:boto:Token: None 2024-04-16T20:21:02.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T20:21:02.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T20:21:02.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df401ede586af5a-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.894 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.894 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.894 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.894 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.895 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.895 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.895 DEBUG:boto:Data: 2024-04-16T20:21:02.895 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.895 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.895 DEBUG:boto:Params: {} 2024-04-16T20:21:02.895 DEBUG:boto:Token: None 2024-04-16T20:21:02.895 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA= 2024-04-16T20:21:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA=', 'Content-Length': '0'} 2024-04-16T20:21:02.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290b059d810fe56c-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.898 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.898 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.898 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.898 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.898 DEBUG:boto:Method: HEAD 2024-04-16T20:21:02.898 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.898 DEBUG:boto:Data: 2024-04-16T20:21:02.898 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.898 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.898 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.898 DEBUG:boto:Params: {} 2024-04-16T20:21:02.899 DEBUG:boto:Token: None 2024-04-16T20:21:02.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA= 2024-04-16T20:21:02.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA=', 'Content-Length': '0'} 2024-04-16T20:21:02.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000dcb97ab2b78f71-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:21:02.901 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.901 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.901 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.901 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.902 DEBUG:boto:Method: GET 2024-04-16T20:21:02.902 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.902 DEBUG:boto:Data: 2024-04-16T20:21:02.902 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.902 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.902 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.902 DEBUG:boto:Params: {} 2024-04-16T20:21:02.902 DEBUG:boto:Token: None 2024-04-16T20:21:02.902 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.903 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T20:21:02.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T20:21:02.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6408d5116fd0aba-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.946 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:02.947 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:02.947 DEBUG:boto:path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.947 DEBUG:boto:auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.947 DEBUG:boto:Method: GET 2024-04-16T20:21:02.947 DEBUG:boto:Path: /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.947 DEBUG:boto:Data: 2024-04-16T20:21:02.947 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.947 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.947 DEBUG:boto:Params: {} 2024-04-16T20:21:02.947 DEBUG:boto:Token: None 2024-04-16T20:21:02.948 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/obj38?versionId=null 2024-04-16T20:21:02.948 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T20:21:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T20:21:02.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdeb0030c7338c9-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.950 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.950 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.950 DEBUG:boto:path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.950 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.951 DEBUG:boto:Method: GET 2024-04-16T20:21:02.951 DEBUG:boto:Path: /yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.951 DEBUG:boto:Data: 2024-04-16T20:21:02.951 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.951 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:02.951 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.951 DEBUG:boto:Params: {} 2024-04-16T20:21:02.951 DEBUG:boto:Token: None 2024-04-16T20:21:02.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:02.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a977f95961b591-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:02.994 DEBUG:boto:b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:02.995 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:02.995 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:02.995 DEBUG:boto:path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.995 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.995 DEBUG:boto:Method: GET 2024-04-16T20:21:02.995 DEBUG:boto:Path: /yrjefh-56/?prefix=obj38 2024-04-16T20:21:02.995 DEBUG:boto:Data: 2024-04-16T20:21:02.995 DEBUG:boto:Headers: {} 2024-04-16T20:21:02.996 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:02.996 DEBUG:boto:Port: 8000 2024-04-16T20:21:02.996 DEBUG:boto:Params: {} 2024-04-16T20:21:02.996 DEBUG:boto:Token: None 2024-04-16T20:21:02.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:02 GMT /yrjefh-56/ 2024-04-16T20:21:02.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T20:21:02.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T20:21:03.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c69ef6b2c23f8d7-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.000 DEBUG:boto:b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.000 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:03.000 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:03.000 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.000 DEBUG:boto:Path: /yrjefh-56/obj38 2024-04-16T20:21:03.000 DEBUG:boto:Data: 2024-04-16T20:21:03.000 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.001 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.001 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.001 DEBUG:boto:Params: {} 2024-04-16T20:21:03.001 DEBUG:boto:Token: None 2024-04-16T20:21:03.001 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj38 2024-04-16T20:21:03.001 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY= 2024-04-16T20:21:03.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY=', 'Content-Length': '0'} 2024-04-16T20:21:03.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a387a20cad23f20a-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.003 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:03.003 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:03.004 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.004 DEBUG:boto:Path: /yrjefh-56/obj38 2024-04-16T20:21:03.004 DEBUG:boto:Data: 2024-04-16T20:21:03.004 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.004 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.004 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.004 DEBUG:boto:Params: {} 2024-04-16T20:21:03.004 DEBUG:boto:Token: None 2024-04-16T20:21:03.004 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj38 2024-04-16T20:21:03.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY= 2024-04-16T20:21:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY=', 'Content-Length': '0'} 2024-04-16T20:21:03.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7aab66c52b5befb-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-16T20:21:03.010 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:03.010 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:03.010 DEBUG:boto:Method: GET 2024-04-16T20:21:03.010 DEBUG:boto:Path: /yrjefh-56/obj38 2024-04-16T20:21:03.010 DEBUG:boto:Data: 2024-04-16T20:21:03.010 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.010 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.011 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.011 DEBUG:boto:Params: {} 2024-04-16T20:21:03.011 DEBUG:boto:Token: None 2024-04-16T20:21:03.011 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj38 2024-04-16T20:21:03.011 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA= 2024-04-16T20:21:03.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA=', 'Content-Length': '0'} 2024-04-16T20:21:03.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436962b1e9f8a78c-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.054 DEBUG:boto:path=/yrjefh-56/obj38 2024-04-16T20:21:03.054 DEBUG:boto:auth_path=/yrjefh-56/obj38 2024-04-16T20:21:03.055 DEBUG:boto:Method: GET 2024-04-16T20:21:03.055 DEBUG:boto:Path: /yrjefh-56/obj38 2024-04-16T20:21:03.055 DEBUG:boto:Data: 2024-04-16T20:21:03.055 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.055 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.055 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.055 DEBUG:boto:Params: {} 2024-04-16T20:21:03.055 DEBUG:boto:Token: None 2024-04-16T20:21:03.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj38 2024-04-16T20:21:03.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA= 2024-04-16T20:21:03.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA=', 'Content-Length': '0'} 2024-04-16T20:21:03.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547df91362e05766-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.058 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.058 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.059 DEBUG:boto:path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.059 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.059 DEBUG:boto:Method: GET 2024-04-16T20:21:03.059 DEBUG:boto:Path: /yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.059 DEBUG:boto:Data: 2024-04-16T20:21:03.059 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.059 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.059 DEBUG:boto:Params: {} 2024-04-16T20:21:03.059 DEBUG:boto:Token: None 2024-04-16T20:21:03.060 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016fe9dcaf058d6ea-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.110 DEBUG:boto:b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.111 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.111 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.111 DEBUG:boto:path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.111 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.111 DEBUG:boto:Method: GET 2024-04-16T20:21:03.111 DEBUG:boto:Path: /yrjefh-56/?prefix=obj38 2024-04-16T20:21:03.111 DEBUG:boto:Data: 2024-04-16T20:21:03.111 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.112 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.112 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.112 DEBUG:boto:Params: {} 2024-04-16T20:21:03.112 DEBUG:boto:Token: None 2024-04-16T20:21:03.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3dc03423357fd4-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.116 DEBUG:boto:b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:21:03.116 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.117 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.117 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.117 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.117 DEBUG:boto:Method: GET 2024-04-16T20:21:03.117 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.117 DEBUG:boto:Data: 2024-04-16T20:21:03.117 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.117 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.117 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.117 DEBUG:boto:Params: {} 2024-04-16T20:21:03.118 DEBUG:boto:Token: None 2024-04-16T20:21:03.118 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.118 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T20:21:03.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T20:21:03.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110e44d6bb59fca5-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.166 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.166 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.167 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.167 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.167 DEBUG:boto:Method: GET 2024-04-16T20:21:03.167 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.167 DEBUG:boto:Data: 2024-04-16T20:21:03.167 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.167 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.167 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.167 DEBUG:boto:Params: {} 2024-04-16T20:21:03.167 DEBUG:boto:Token: None 2024-04-16T20:21:03.167 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.168 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T20:21:03.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T20:21:03.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c747b2d04b3d7d05-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.170 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.170 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.170 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.170 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.170 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.170 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.170 DEBUG:boto:Data: 2024-04-16T20:21:03.170 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.171 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.171 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.171 DEBUG:boto:Params: {} 2024-04-16T20:21:03.171 DEBUG:boto:Token: None 2024-04-16T20:21:03.171 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.171 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc= 2024-04-16T20:21:03.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc=', 'Content-Length': '0'} 2024-04-16T20:21:03.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bca885d0c545638-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.173 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.173 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.173 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.173 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.173 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.174 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.174 DEBUG:boto:Data: 2024-04-16T20:21:03.174 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.174 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.174 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.174 DEBUG:boto:Params: {} 2024-04-16T20:21:03.174 DEBUG:boto:Token: None 2024-04-16T20:21:03.174 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.174 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc= 2024-04-16T20:21:03.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc=', 'Content-Length': '0'} 2024-04-16T20:21:03.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba30c4ba46c71995-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:21:03.176 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.177 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.177 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.177 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.177 DEBUG:boto:Method: GET 2024-04-16T20:21:03.177 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.177 DEBUG:boto:Data: 2024-04-16T20:21:03.177 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.177 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.177 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.177 DEBUG:boto:Params: {} 2024-04-16T20:21:03.177 DEBUG:boto:Token: None 2024-04-16T20:21:03.178 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.178 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T20:21:03.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T20:21:03.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7171e24211ddafc-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.222 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.222 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.223 DEBUG:boto:path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.223 DEBUG:boto:auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.223 DEBUG:boto:Method: GET 2024-04-16T20:21:03.223 DEBUG:boto:Path: /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.223 DEBUG:boto:Data: 2024-04-16T20:21:03.223 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.223 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.223 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.223 DEBUG:boto:Params: {} 2024-04-16T20:21:03.223 DEBUG:boto:Token: None 2024-04-16T20:21:03.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44?versionId=null 2024-04-16T20:21:03.224 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T20:21:03.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T20:21:03.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a6cdc50985acf0-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.226 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.226 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.226 DEBUG:boto:path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.226 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.226 DEBUG:boto:Method: GET 2024-04-16T20:21:03.226 DEBUG:boto:Path: /yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.227 DEBUG:boto:Data: 2024-04-16T20:21:03.227 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.227 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.227 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.227 DEBUG:boto:Params: {} 2024-04-16T20:21:03.227 DEBUG:boto:Token: None 2024-04-16T20:21:03.227 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ff33b9893f3415-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.270 DEBUG:boto:b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.271 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.271 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.271 DEBUG:boto:path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.271 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.271 DEBUG:boto:Method: GET 2024-04-16T20:21:03.271 DEBUG:boto:Path: /yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.271 DEBUG:boto:Data: 2024-04-16T20:21:03.271 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.272 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.272 DEBUG:boto:Params: {} 2024-04-16T20:21:03.272 DEBUG:boto:Token: None 2024-04-16T20:21:03.272 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.272 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa89f23d02d90fc1-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.276 DEBUG:boto:b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.276 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.276 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.276 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.277 DEBUG:boto:Path: /yrjefh-56/obj44 2024-04-16T20:21:03.277 DEBUG:boto:Data: 2024-04-16T20:21:03.277 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.277 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.277 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.277 DEBUG:boto:Params: {} 2024-04-16T20:21:03.277 DEBUG:boto:Token: None 2024-04-16T20:21:03.277 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44 2024-04-16T20:21:03.277 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI= 2024-04-16T20:21:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI=', 'Content-Length': '0'} 2024-04-16T20:21:03.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b3c6890690257e-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.280 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.280 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.280 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.280 DEBUG:boto:Path: /yrjefh-56/obj44 2024-04-16T20:21:03.280 DEBUG:boto:Data: 2024-04-16T20:21:03.280 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.280 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.280 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.280 DEBUG:boto:Params: {} 2024-04-16T20:21:03.280 DEBUG:boto:Token: None 2024-04-16T20:21:03.281 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44 2024-04-16T20:21:03.281 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI= 2024-04-16T20:21:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI=', 'Content-Length': '0'} 2024-04-16T20:21:03.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da025dd63ada757-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:21:03.283 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.283 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.283 DEBUG:boto:Method: GET 2024-04-16T20:21:03.283 DEBUG:boto:Path: /yrjefh-56/obj44 2024-04-16T20:21:03.283 DEBUG:boto:Data: 2024-04-16T20:21:03.284 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.284 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.284 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.284 DEBUG:boto:Params: {} 2024-04-16T20:21:03.284 DEBUG:boto:Token: None 2024-04-16T20:21:03.284 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44 2024-04-16T20:21:03.284 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58= 2024-04-16T20:21:03.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58=', 'Content-Length': '0'} 2024-04-16T20:21:03.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3b77c7557f1b39-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.326 DEBUG:boto:path=/yrjefh-56/obj44 2024-04-16T20:21:03.327 DEBUG:boto:auth_path=/yrjefh-56/obj44 2024-04-16T20:21:03.327 DEBUG:boto:Method: GET 2024-04-16T20:21:03.327 DEBUG:boto:Path: /yrjefh-56/obj44 2024-04-16T20:21:03.327 DEBUG:boto:Data: 2024-04-16T20:21:03.327 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.327 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.327 DEBUG:boto:Params: {} 2024-04-16T20:21:03.327 DEBUG:boto:Token: None 2024-04-16T20:21:03.327 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj44 2024-04-16T20:21:03.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58= 2024-04-16T20:21:03.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58=', 'Content-Length': '0'} 2024-04-16T20:21:03.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e404c44659d44a41-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.330 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.330 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.330 DEBUG:boto:path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.330 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.330 DEBUG:boto:Method: GET 2024-04-16T20:21:03.330 DEBUG:boto:Path: /yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.330 DEBUG:boto:Data: 2024-04-16T20:21:03.330 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.330 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.330 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.330 DEBUG:boto:Params: {} 2024-04-16T20:21:03.331 DEBUG:boto:Token: None 2024-04-16T20:21:03.331 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.331 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bb6beb5c6a80b7-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.374 DEBUG:boto:b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.375 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.375 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.375 DEBUG:boto:path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.375 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.375 DEBUG:boto:Method: GET 2024-04-16T20:21:03.375 DEBUG:boto:Path: /yrjefh-56/?prefix=obj44 2024-04-16T20:21:03.375 DEBUG:boto:Data: 2024-04-16T20:21:03.375 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.376 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.376 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.376 DEBUG:boto:Params: {} 2024-04-16T20:21:03.376 DEBUG:boto:Token: None 2024-04-16T20:21:03.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d15b12dd9efe33c2-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.380 DEBUG:boto:b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:21:03.380 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.380 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.381 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.381 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.381 DEBUG:boto:Method: GET 2024-04-16T20:21:03.381 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.381 DEBUG:boto:Data: 2024-04-16T20:21:03.381 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.381 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.381 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.381 DEBUG:boto:Params: {} 2024-04-16T20:21:03.381 DEBUG:boto:Token: None 2024-04-16T20:21:03.381 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.381 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T20:21:03.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T20:21:03.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004c6f59b3133e53-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.426 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.426 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.427 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.427 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.427 DEBUG:boto:Method: GET 2024-04-16T20:21:03.427 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.427 DEBUG:boto:Data: 2024-04-16T20:21:03.427 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.427 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.427 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.427 DEBUG:boto:Params: {} 2024-04-16T20:21:03.427 DEBUG:boto:Token: None 2024-04-16T20:21:03.427 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.428 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T20:21:03.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T20:21:03.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007883e6af552e4117-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.430 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.430 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.430 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.430 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.430 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.431 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.431 DEBUG:boto:Data: 2024-04-16T20:21:03.431 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.431 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.431 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.431 DEBUG:boto:Params: {} 2024-04-16T20:21:03.431 DEBUG:boto:Token: None 2024-04-16T20:21:03.431 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio= 2024-04-16T20:21:03.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio=', 'Content-Length': '0'} 2024-04-16T20:21:03.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065482a503484db41-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.434 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.434 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.434 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.434 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.434 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.434 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.434 DEBUG:boto:Data: 2024-04-16T20:21:03.434 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.434 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.434 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.434 DEBUG:boto:Params: {} 2024-04-16T20:21:03.435 DEBUG:boto:Token: None 2024-04-16T20:21:03.435 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio= 2024-04-16T20:21:03.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio=', 'Content-Length': '0'} 2024-04-16T20:21:03.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db137302479e0c1-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:21:03.437 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.437 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.438 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.438 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.438 DEBUG:boto:Method: GET 2024-04-16T20:21:03.438 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.438 DEBUG:boto:Data: 2024-04-16T20:21:03.438 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.438 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.438 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.438 DEBUG:boto:Params: {} 2024-04-16T20:21:03.438 DEBUG:boto:Token: None 2024-04-16T20:21:03.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T20:21:03.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T20:21:03.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1370c816552a4e6-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.482 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.482 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.483 DEBUG:boto:path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.483 DEBUG:boto:auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.483 DEBUG:boto:Method: GET 2024-04-16T20:21:03.483 DEBUG:boto:Path: /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.483 DEBUG:boto:Data: 2024-04-16T20:21:03.483 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.483 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.483 DEBUG:boto:Params: {} 2024-04-16T20:21:03.483 DEBUG:boto:Token: None 2024-04-16T20:21:03.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46?versionId=null 2024-04-16T20:21:03.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T20:21:03.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T20:21:03.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e163650105cb5f-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.486 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.486 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.487 DEBUG:boto:path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.487 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.487 DEBUG:boto:Method: GET 2024-04-16T20:21:03.487 DEBUG:boto:Path: /yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.487 DEBUG:boto:Data: 2024-04-16T20:21:03.487 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.487 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.487 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.487 DEBUG:boto:Params: {} 2024-04-16T20:21:03.487 DEBUG:boto:Token: None 2024-04-16T20:21:03.487 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.488 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002320a21bda6395c6-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.530 DEBUG:boto:b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.531 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.531 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.531 DEBUG:boto:path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.531 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.531 DEBUG:boto:Method: GET 2024-04-16T20:21:03.531 DEBUG:boto:Path: /yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.531 DEBUG:boto:Data: 2024-04-16T20:21:03.531 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.532 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.532 DEBUG:boto:Params: {} 2024-04-16T20:21:03.532 DEBUG:boto:Token: None 2024-04-16T20:21:03.532 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023def55d76026d9e-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.536 DEBUG:boto:b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.537 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.537 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.537 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.537 DEBUG:boto:Path: /yrjefh-56/obj46 2024-04-16T20:21:03.537 DEBUG:boto:Data: 2024-04-16T20:21:03.537 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.537 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.538 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.538 DEBUG:boto:Params: {} 2024-04-16T20:21:03.538 DEBUG:boto:Token: None 2024-04-16T20:21:03.538 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46 2024-04-16T20:21:03.538 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE= 2024-04-16T20:21:03.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE=', 'Content-Length': '0'} 2024-04-16T20:21:03.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdcd1fbb54c1cf5-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.541 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.541 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.541 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.541 DEBUG:boto:Path: /yrjefh-56/obj46 2024-04-16T20:21:03.541 DEBUG:boto:Data: 2024-04-16T20:21:03.541 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.541 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.541 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.541 DEBUG:boto:Params: {} 2024-04-16T20:21:03.541 DEBUG:boto:Token: None 2024-04-16T20:21:03.542 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46 2024-04-16T20:21:03.542 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE= 2024-04-16T20:21:03.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE=', 'Content-Length': '0'} 2024-04-16T20:21:03.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdb155e5343d7a7-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-16T20:21:03.544 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.544 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.544 DEBUG:boto:Method: GET 2024-04-16T20:21:03.544 DEBUG:boto:Path: /yrjefh-56/obj46 2024-04-16T20:21:03.544 DEBUG:boto:Data: 2024-04-16T20:21:03.545 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.545 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.545 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.545 DEBUG:boto:Params: {} 2024-04-16T20:21:03.545 DEBUG:boto:Token: None 2024-04-16T20:21:03.545 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46 2024-04-16T20:21:03.545 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys= 2024-04-16T20:21:03.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys=', 'Content-Length': '0'} 2024-04-16T20:21:03.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7164dd5d664b7e-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.590 DEBUG:boto:path=/yrjefh-56/obj46 2024-04-16T20:21:03.590 DEBUG:boto:auth_path=/yrjefh-56/obj46 2024-04-16T20:21:03.590 DEBUG:boto:Method: GET 2024-04-16T20:21:03.591 DEBUG:boto:Path: /yrjefh-56/obj46 2024-04-16T20:21:03.591 DEBUG:boto:Data: 2024-04-16T20:21:03.591 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.591 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.591 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.591 DEBUG:boto:Params: {} 2024-04-16T20:21:03.591 DEBUG:boto:Token: None 2024-04-16T20:21:03.591 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj46 2024-04-16T20:21:03.591 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys= 2024-04-16T20:21:03.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys=', 'Content-Length': '0'} 2024-04-16T20:21:03.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb272c3d0513272-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.594 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.594 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.594 DEBUG:boto:path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.594 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.594 DEBUG:boto:Method: GET 2024-04-16T20:21:03.594 DEBUG:boto:Path: /yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.594 DEBUG:boto:Data: 2024-04-16T20:21:03.595 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.595 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.595 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.595 DEBUG:boto:Params: {} 2024-04-16T20:21:03.595 DEBUG:boto:Token: None 2024-04-16T20:21:03.595 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.595 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65551e3dd94201f-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.598 DEBUG:boto:b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.598 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.598 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.598 DEBUG:boto:path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.598 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.598 DEBUG:boto:Method: GET 2024-04-16T20:21:03.598 DEBUG:boto:Path: /yrjefh-56/?prefix=obj46 2024-04-16T20:21:03.598 DEBUG:boto:Data: 2024-04-16T20:21:03.599 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.599 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.599 DEBUG:boto:Params: {} 2024-04-16T20:21:03.599 DEBUG:boto:Token: None 2024-04-16T20:21:03.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d9e9bbdd9b8d17-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.603 DEBUG:boto:b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:21:03.603 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.603 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.603 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.604 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.604 DEBUG:boto:Method: GET 2024-04-16T20:21:03.604 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.604 DEBUG:boto:Data: 2024-04-16T20:21:03.604 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.604 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.604 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.604 DEBUG:boto:Params: {} 2024-04-16T20:21:03.604 DEBUG:boto:Token: None 2024-04-16T20:21:03.604 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T20:21:03.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T20:21:03.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed10a23889d59322-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.650 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.651 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.651 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.651 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.651 DEBUG:boto:Method: GET 2024-04-16T20:21:03.651 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.651 DEBUG:boto:Data: 2024-04-16T20:21:03.651 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.651 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.651 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.651 DEBUG:boto:Params: {} 2024-04-16T20:21:03.652 DEBUG:boto:Token: None 2024-04-16T20:21:03.652 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T20:21:03.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T20:21:03.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065728fe6f2c5d38-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.655 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.655 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.655 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.655 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.655 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.655 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.655 DEBUG:boto:Data: 2024-04-16T20:21:03.655 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.655 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.655 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.655 DEBUG:boto:Params: {} 2024-04-16T20:21:03.656 DEBUG:boto:Token: None 2024-04-16T20:21:03.656 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ= 2024-04-16T20:21:03.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ=', 'Content-Length': '0'} 2024-04-16T20:21:03.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e69087ee9d2391-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.658 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.658 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.658 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.659 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.659 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.659 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.659 DEBUG:boto:Data: 2024-04-16T20:21:03.659 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.659 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.659 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.659 DEBUG:boto:Params: {} 2024-04-16T20:21:03.659 DEBUG:boto:Token: None 2024-04-16T20:21:03.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ= 2024-04-16T20:21:03.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ=', 'Content-Length': '0'} 2024-04-16T20:21:03.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc4f48ddc1381ca4-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:21:03.662 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.662 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.662 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.663 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.663 DEBUG:boto:Method: GET 2024-04-16T20:21:03.663 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.663 DEBUG:boto:Data: 2024-04-16T20:21:03.663 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.663 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.663 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.663 DEBUG:boto:Params: {} 2024-04-16T20:21:03.663 DEBUG:boto:Token: None 2024-04-16T20:21:03.664 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.664 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T20:21:03.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T20:21:03.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa51c6a6144d19e0-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.710 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.710 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.710 DEBUG:boto:path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.711 DEBUG:boto:auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.711 DEBUG:boto:Method: GET 2024-04-16T20:21:03.711 DEBUG:boto:Path: /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.711 DEBUG:boto:Data: 2024-04-16T20:21:03.711 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.711 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.711 DEBUG:boto:Params: {} 2024-04-16T20:21:03.711 DEBUG:boto:Token: None 2024-04-16T20:21:03.712 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51?versionId=null 2024-04-16T20:21:03.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T20:21:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T20:21:03.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eada67e5a8182e-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.714 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.714 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.714 DEBUG:boto:path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.714 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.715 DEBUG:boto:Method: GET 2024-04-16T20:21:03.715 DEBUG:boto:Path: /yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.715 DEBUG:boto:Data: 2024-04-16T20:21:03.715 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.715 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.715 DEBUG:boto:Params: {} 2024-04-16T20:21:03.715 DEBUG:boto:Token: None 2024-04-16T20:21:03.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ebba50387b03756-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.718 DEBUG:boto:b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.718 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.719 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.719 DEBUG:boto:path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.719 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.719 DEBUG:boto:Method: GET 2024-04-16T20:21:03.719 DEBUG:boto:Path: /yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.719 DEBUG:boto:Data: 2024-04-16T20:21:03.719 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.719 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.719 DEBUG:boto:Params: {} 2024-04-16T20:21:03.720 DEBUG:boto:Token: None 2024-04-16T20:21:03.720 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.720 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f413d984b633611-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.723 DEBUG:boto:b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.724 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.724 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.724 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.724 DEBUG:boto:Path: /yrjefh-56/obj51 2024-04-16T20:21:03.724 DEBUG:boto:Data: 2024-04-16T20:21:03.724 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.724 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.724 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.724 DEBUG:boto:Params: {} 2024-04-16T20:21:03.725 DEBUG:boto:Token: None 2024-04-16T20:21:03.725 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51 2024-04-16T20:21:03.725 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc= 2024-04-16T20:21:03.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc=', 'Content-Length': '0'} 2024-04-16T20:21:03.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5f3cdccfae03d1-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.727 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.727 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.727 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.728 DEBUG:boto:Path: /yrjefh-56/obj51 2024-04-16T20:21:03.728 DEBUG:boto:Data: 2024-04-16T20:21:03.728 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.728 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.728 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.728 DEBUG:boto:Params: {} 2024-04-16T20:21:03.728 DEBUG:boto:Token: None 2024-04-16T20:21:03.728 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51 2024-04-16T20:21:03.728 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc= 2024-04-16T20:21:03.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc=', 'Content-Length': '0'} 2024-04-16T20:21:03.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fce46ae7fcd8547-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-16T20:21:03.731 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.731 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.731 DEBUG:boto:Method: GET 2024-04-16T20:21:03.731 DEBUG:boto:Path: /yrjefh-56/obj51 2024-04-16T20:21:03.731 DEBUG:boto:Data: 2024-04-16T20:21:03.731 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.731 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.732 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.732 DEBUG:boto:Params: {} 2024-04-16T20:21:03.732 DEBUG:boto:Token: None 2024-04-16T20:21:03.732 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51 2024-04-16T20:21:03.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY= 2024-04-16T20:21:03.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY=', 'Content-Length': '0'} 2024-04-16T20:21:03.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ebfe90937b9aad9-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.774 DEBUG:boto:path=/yrjefh-56/obj51 2024-04-16T20:21:03.775 DEBUG:boto:auth_path=/yrjefh-56/obj51 2024-04-16T20:21:03.775 DEBUG:boto:Method: GET 2024-04-16T20:21:03.775 DEBUG:boto:Path: /yrjefh-56/obj51 2024-04-16T20:21:03.775 DEBUG:boto:Data: 2024-04-16T20:21:03.775 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.775 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.775 DEBUG:boto:Params: {} 2024-04-16T20:21:03.775 DEBUG:boto:Token: None 2024-04-16T20:21:03.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj51 2024-04-16T20:21:03.776 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY= 2024-04-16T20:21:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY=', 'Content-Length': '0'} 2024-04-16T20:21:03.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c07bfd7d184271-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.778 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.778 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.778 DEBUG:boto:path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.778 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.778 DEBUG:boto:Method: GET 2024-04-16T20:21:03.778 DEBUG:boto:Path: /yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.778 DEBUG:boto:Data: 2024-04-16T20:21:03.779 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.779 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.779 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.779 DEBUG:boto:Params: {} 2024-04-16T20:21:03.779 DEBUG:boto:Token: None 2024-04-16T20:21:03.779 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6cfdd53365b83f0-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.826 DEBUG:boto:b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.827 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.827 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.827 DEBUG:boto:path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.827 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.827 DEBUG:boto:Method: GET 2024-04-16T20:21:03.827 DEBUG:boto:Path: /yrjefh-56/?prefix=obj51 2024-04-16T20:21:03.828 DEBUG:boto:Data: 2024-04-16T20:21:03.828 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.828 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.828 DEBUG:boto:Params: {} 2024-04-16T20:21:03.828 DEBUG:boto:Token: None 2024-04-16T20:21:03.828 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd26c36dc3393b21-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.832 DEBUG:boto:b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:21:03.832 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.832 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.833 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.833 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.833 DEBUG:boto:Method: GET 2024-04-16T20:21:03.833 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.833 DEBUG:boto:Data: 2024-04-16T20:21:03.833 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.833 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.833 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.833 DEBUG:boto:Params: {} 2024-04-16T20:21:03.833 DEBUG:boto:Token: None 2024-04-16T20:21:03.833 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.834 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T20:21:03.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T20:21:03.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b31b09c09cab52-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.882 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.883 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.883 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.883 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.883 DEBUG:boto:Method: GET 2024-04-16T20:21:03.883 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.883 DEBUG:boto:Data: 2024-04-16T20:21:03.883 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.884 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.884 DEBUG:boto:Params: {} 2024-04-16T20:21:03.884 DEBUG:boto:Token: None 2024-04-16T20:21:03.884 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T20:21:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T20:21:03.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a5591f0666fbea-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.887 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.887 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.887 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.887 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.887 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.887 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.887 DEBUG:boto:Data: 2024-04-16T20:21:03.887 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.887 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.888 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.888 DEBUG:boto:Params: {} 2024-04-16T20:21:03.888 DEBUG:boto:Token: None 2024-04-16T20:21:03.888 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM= 2024-04-16T20:21:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM=', 'Content-Length': '0'} 2024-04-16T20:21:03.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000153a6f5b09e2679-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.890 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.890 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.890 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.890 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.890 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.891 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.891 DEBUG:boto:Data: 2024-04-16T20:21:03.891 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.891 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.891 DEBUG:boto:Params: {} 2024-04-16T20:21:03.891 DEBUG:boto:Token: None 2024-04-16T20:21:03.891 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM= 2024-04-16T20:21:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM=', 'Content-Length': '0'} 2024-04-16T20:21:03.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdfc65f99328809-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:21:03.894 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.894 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.894 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.894 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.894 DEBUG:boto:Method: GET 2024-04-16T20:21:03.894 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.894 DEBUG:boto:Data: 2024-04-16T20:21:03.894 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.895 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.895 DEBUG:boto:Params: {} 2024-04-16T20:21:03.895 DEBUG:boto:Token: None 2024-04-16T20:21:03.895 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.895 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T20:21:03.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T20:21:03.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346c915035beebd7-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.938 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.939 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.939 DEBUG:boto:path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.939 DEBUG:boto:auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.939 DEBUG:boto:Method: GET 2024-04-16T20:21:03.939 DEBUG:boto:Path: /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.939 DEBUG:boto:Data: 2024-04-16T20:21:03.939 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.940 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.940 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.940 DEBUG:boto:Params: {} 2024-04-16T20:21:03.940 DEBUG:boto:Token: None 2024-04-16T20:21:03.940 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55?versionId=null 2024-04-16T20:21:03.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T20:21:03.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T20:21:03.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008152cd0ac0ddf4ba-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.943 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.943 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.943 DEBUG:boto:path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.943 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.943 DEBUG:boto:Method: GET 2024-04-16T20:21:03.943 DEBUG:boto:Path: /yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.943 DEBUG:boto:Data: 2024-04-16T20:21:03.943 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.943 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.944 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.944 DEBUG:boto:Params: {} 2024-04-16T20:21:03.944 DEBUG:boto:Token: None 2024-04-16T20:21:03.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbac1fa5d3f5e5ad-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.990 DEBUG:boto:b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.991 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:03.991 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:03.991 DEBUG:boto:path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.991 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.991 DEBUG:boto:Method: GET 2024-04-16T20:21:03.992 DEBUG:boto:Path: /yrjefh-56/?prefix=obj55 2024-04-16T20:21:03.992 DEBUG:boto:Data: 2024-04-16T20:21:03.992 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.992 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:03.992 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.992 DEBUG:boto:Params: {} 2024-04-16T20:21:03.992 DEBUG:boto:Token: None 2024-04-16T20:21:03.992 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/ 2024-04-16T20:21:03.992 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T20:21:03.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T20:21:03.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012021375c48f563b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:03.996 DEBUG:boto:b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:03.997 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:03.997 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:03.997 DEBUG:boto:Method: HEAD 2024-04-16T20:21:03.997 DEBUG:boto:Path: /yrjefh-56/obj55 2024-04-16T20:21:03.997 DEBUG:boto:Data: 2024-04-16T20:21:03.997 DEBUG:boto:Headers: {} 2024-04-16T20:21:03.997 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:03.997 DEBUG:boto:Port: 8000 2024-04-16T20:21:03.997 DEBUG:boto:Params: {} 2024-04-16T20:21:03.997 DEBUG:boto:Token: None 2024-04-16T20:21:03.998 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:03 GMT /yrjefh-56/obj55 2024-04-16T20:21:03.998 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WcH6ul89ytNpujcF2IFnBCplQTg= 2024-04-16T20:21:03.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WcH6ul89ytNpujcF2IFnBCplQTg=', 'Content-Length': '0'} 2024-04-16T20:21:04.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff54756c8684908c-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.000 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:04.000 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:04.000 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.000 DEBUG:boto:Path: /yrjefh-56/obj55 2024-04-16T20:21:04.000 DEBUG:boto:Data: 2024-04-16T20:21:04.000 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.001 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.001 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.001 DEBUG:boto:Params: {} 2024-04-16T20:21:04.001 DEBUG:boto:Token: None 2024-04-16T20:21:04.001 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj55 2024-04-16T20:21:04.001 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t7YntjF1nN0YOWo9gZu5KcQC9r4= 2024-04-16T20:21:04.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t7YntjF1nN0YOWo9gZu5KcQC9r4=', 'Content-Length': '0'} 2024-04-16T20:21:04.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65a89d96ab9e3ca-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:21:04.003 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:04.003 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:04.003 DEBUG:boto:Method: GET 2024-04-16T20:21:04.003 DEBUG:boto:Path: /yrjefh-56/obj55 2024-04-16T20:21:04.004 DEBUG:boto:Data: 2024-04-16T20:21:04.004 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.004 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.004 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.004 DEBUG:boto:Params: {} 2024-04-16T20:21:04.004 DEBUG:boto:Token: None 2024-04-16T20:21:04.004 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj55 2024-04-16T20:21:04.004 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY= 2024-04-16T20:21:04.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY=', 'Content-Length': '0'} 2024-04-16T20:21:04.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908b5e411986e198-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.047 DEBUG:boto:path=/yrjefh-56/obj55 2024-04-16T20:21:04.047 DEBUG:boto:auth_path=/yrjefh-56/obj55 2024-04-16T20:21:04.047 DEBUG:boto:Method: GET 2024-04-16T20:21:04.047 DEBUG:boto:Path: /yrjefh-56/obj55 2024-04-16T20:21:04.048 DEBUG:boto:Data: 2024-04-16T20:21:04.048 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.048 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.048 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.048 DEBUG:boto:Params: {} 2024-04-16T20:21:04.048 DEBUG:boto:Token: None 2024-04-16T20:21:04.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj55 2024-04-16T20:21:04.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY= 2024-04-16T20:21:04.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY=', 'Content-Length': '0'} 2024-04-16T20:21:04.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5312bc1c9c5d51-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.051 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.051 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.051 DEBUG:boto:path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.051 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.051 DEBUG:boto:Method: GET 2024-04-16T20:21:04.051 DEBUG:boto:Path: /yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.051 DEBUG:boto:Data: 2024-04-16T20:21:04.052 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.052 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.052 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.052 DEBUG:boto:Params: {} 2024-04-16T20:21:04.052 DEBUG:boto:Token: None 2024-04-16T20:21:04.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1325b4efd4b9b60-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.055 DEBUG:boto:b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.056 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.056 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.056 DEBUG:boto:path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.056 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.056 DEBUG:boto:Method: GET 2024-04-16T20:21:04.056 DEBUG:boto:Path: /yrjefh-56/?prefix=obj55 2024-04-16T20:21:04.056 DEBUG:boto:Data: 2024-04-16T20:21:04.056 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.056 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.057 DEBUG:boto:Params: {} 2024-04-16T20:21:04.057 DEBUG:boto:Token: None 2024-04-16T20:21:04.057 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.057 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003963da7b2fab185d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.102 DEBUG:boto:b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:21:04.103 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.103 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.103 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.103 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.103 DEBUG:boto:Method: GET 2024-04-16T20:21:04.104 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.104 DEBUG:boto:Data: 2024-04-16T20:21:04.104 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.104 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.104 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.104 DEBUG:boto:Params: {} 2024-04-16T20:21:04.104 DEBUG:boto:Token: None 2024-04-16T20:21:04.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T20:21:04.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T20:21:04.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e531678c08484fb2-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.150 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.151 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.151 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.151 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.151 DEBUG:boto:Method: GET 2024-04-16T20:21:04.151 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.151 DEBUG:boto:Data: 2024-04-16T20:21:04.151 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.151 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.151 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.152 DEBUG:boto:Params: {} 2024-04-16T20:21:04.152 DEBUG:boto:Token: None 2024-04-16T20:21:04.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T20:21:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T20:21:04.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeaa17ba8753125a-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.154 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.155 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.155 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.155 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.155 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.155 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.155 DEBUG:boto:Data: 2024-04-16T20:21:04.155 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.155 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.155 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.156 DEBUG:boto:Params: {} 2024-04-16T20:21:04.156 DEBUG:boto:Token: None 2024-04-16T20:21:04.156 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI= 2024-04-16T20:21:04.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI=', 'Content-Length': '0'} 2024-04-16T20:21:04.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008122c3139de1ae23-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.159 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.159 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.159 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.159 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.159 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.159 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.159 DEBUG:boto:Data: 2024-04-16T20:21:04.159 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.160 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.160 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.160 DEBUG:boto:Params: {} 2024-04-16T20:21:04.160 DEBUG:boto:Token: None 2024-04-16T20:21:04.160 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.160 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI= 2024-04-16T20:21:04.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI=', 'Content-Length': '0'} 2024-04-16T20:21:04.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a61d37def826232-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:21:04.163 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.163 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.163 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.163 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.163 DEBUG:boto:Method: GET 2024-04-16T20:21:04.163 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.163 DEBUG:boto:Data: 2024-04-16T20:21:04.163 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.163 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.163 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.163 DEBUG:boto:Params: {} 2024-04-16T20:21:04.164 DEBUG:boto:Token: None 2024-04-16T20:21:04.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T20:21:04.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T20:21:04.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a5462362aac37c-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.210 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.210 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.211 DEBUG:boto:path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.211 DEBUG:boto:auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.211 DEBUG:boto:Method: GET 2024-04-16T20:21:04.211 DEBUG:boto:Path: /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.211 DEBUG:boto:Data: 2024-04-16T20:21:04.211 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.211 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.211 DEBUG:boto:Params: {} 2024-04-16T20:21:04.211 DEBUG:boto:Token: None 2024-04-16T20:21:04.212 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57?versionId=null 2024-04-16T20:21:04.212 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T20:21:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T20:21:04.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98b9adf929dee9d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.214 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.215 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.215 DEBUG:boto:path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.215 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.215 DEBUG:boto:Method: GET 2024-04-16T20:21:04.215 DEBUG:boto:Path: /yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.215 DEBUG:boto:Data: 2024-04-16T20:21:04.215 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.215 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.215 DEBUG:boto:Params: {} 2024-04-16T20:21:04.216 DEBUG:boto:Token: None 2024-04-16T20:21:04.216 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.216 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862369cae561892a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.219 DEBUG:boto:b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.219 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.220 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.220 DEBUG:boto:path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.220 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.220 DEBUG:boto:Method: GET 2024-04-16T20:21:04.220 DEBUG:boto:Path: /yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.220 DEBUG:boto:Data: 2024-04-16T20:21:04.220 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.220 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.220 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.220 DEBUG:boto:Params: {} 2024-04-16T20:21:04.221 DEBUG:boto:Token: None 2024-04-16T20:21:04.221 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.221 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6b99a926be416d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.224 DEBUG:boto:b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.225 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.225 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.225 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.225 DEBUG:boto:Path: /yrjefh-56/obj57 2024-04-16T20:21:04.225 DEBUG:boto:Data: 2024-04-16T20:21:04.225 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.225 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.225 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.225 DEBUG:boto:Params: {} 2024-04-16T20:21:04.226 DEBUG:boto:Token: None 2024-04-16T20:21:04.226 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57 2024-04-16T20:21:04.226 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU= 2024-04-16T20:21:04.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU=', 'Content-Length': '0'} 2024-04-16T20:21:04.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ecf33fd2f808380-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.228 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.229 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.229 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.229 DEBUG:boto:Path: /yrjefh-56/obj57 2024-04-16T20:21:04.229 DEBUG:boto:Data: 2024-04-16T20:21:04.229 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.229 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.229 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.229 DEBUG:boto:Params: {} 2024-04-16T20:21:04.229 DEBUG:boto:Token: None 2024-04-16T20:21:04.229 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57 2024-04-16T20:21:04.230 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU= 2024-04-16T20:21:04.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU=', 'Content-Length': '0'} 2024-04-16T20:21:04.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019847c69abf69077-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-16T20:21:04.232 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.232 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.233 DEBUG:boto:Method: GET 2024-04-16T20:21:04.233 DEBUG:boto:Path: /yrjefh-56/obj57 2024-04-16T20:21:04.233 DEBUG:boto:Data: 2024-04-16T20:21:04.233 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.233 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.233 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.233 DEBUG:boto:Params: {} 2024-04-16T20:21:04.233 DEBUG:boto:Token: None 2024-04-16T20:21:04.233 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57 2024-04-16T20:21:04.233 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA= 2024-04-16T20:21:04.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA=', 'Content-Length': '0'} 2024-04-16T20:21:04.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000348932a4e493cbe1-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.278 DEBUG:boto:path=/yrjefh-56/obj57 2024-04-16T20:21:04.278 DEBUG:boto:auth_path=/yrjefh-56/obj57 2024-04-16T20:21:04.279 DEBUG:boto:Method: GET 2024-04-16T20:21:04.279 DEBUG:boto:Path: /yrjefh-56/obj57 2024-04-16T20:21:04.279 DEBUG:boto:Data: 2024-04-16T20:21:04.279 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.279 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.279 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.279 DEBUG:boto:Params: {} 2024-04-16T20:21:04.279 DEBUG:boto:Token: None 2024-04-16T20:21:04.279 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj57 2024-04-16T20:21:04.279 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA= 2024-04-16T20:21:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA=', 'Content-Length': '0'} 2024-04-16T20:21:04.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89bc20a1153ff70-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.282 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.282 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.282 DEBUG:boto:path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.282 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.282 DEBUG:boto:Method: GET 2024-04-16T20:21:04.283 DEBUG:boto:Path: /yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.283 DEBUG:boto:Data: 2024-04-16T20:21:04.283 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.283 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.283 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.283 DEBUG:boto:Params: {} 2024-04-16T20:21:04.283 DEBUG:boto:Token: None 2024-04-16T20:21:04.283 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.283 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc96d44e32415f6b-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.326 DEBUG:boto:b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.327 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.327 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.327 DEBUG:boto:path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.327 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.327 DEBUG:boto:Method: GET 2024-04-16T20:21:04.327 DEBUG:boto:Path: /yrjefh-56/?prefix=obj57 2024-04-16T20:21:04.327 DEBUG:boto:Data: 2024-04-16T20:21:04.327 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.328 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.328 DEBUG:boto:Params: {} 2024-04-16T20:21:04.328 DEBUG:boto:Token: None 2024-04-16T20:21:04.328 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.328 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889cef224bbddd7f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.331 DEBUG:boto:b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:21:04.332 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.332 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.332 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.332 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.332 DEBUG:boto:Method: GET 2024-04-16T20:21:04.332 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.332 DEBUG:boto:Data: 2024-04-16T20:21:04.332 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.332 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.333 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.333 DEBUG:boto:Params: {} 2024-04-16T20:21:04.333 DEBUG:boto:Token: None 2024-04-16T20:21:04.333 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.333 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T20:21:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T20:21:04.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004634d2f5d339ea12-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.378 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.379 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.379 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.379 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.379 DEBUG:boto:Method: GET 2024-04-16T20:21:04.379 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.379 DEBUG:boto:Data: 2024-04-16T20:21:04.379 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.379 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.379 DEBUG:boto:Params: {} 2024-04-16T20:21:04.380 DEBUG:boto:Token: None 2024-04-16T20:21:04.380 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T20:21:04.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T20:21:04.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4456ebee3c12a6f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.382 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.382 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.382 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.382 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.383 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.383 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.383 DEBUG:boto:Data: 2024-04-16T20:21:04.383 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.383 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.383 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.383 DEBUG:boto:Params: {} 2024-04-16T20:21:04.383 DEBUG:boto:Token: None 2024-04-16T20:21:04.383 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.383 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk= 2024-04-16T20:21:04.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk=', 'Content-Length': '0'} 2024-04-16T20:21:04.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9d469bec9b09b3-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.385 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.386 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.386 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.386 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.386 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.386 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.386 DEBUG:boto:Data: 2024-04-16T20:21:04.386 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.386 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.386 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.386 DEBUG:boto:Params: {} 2024-04-16T20:21:04.387 DEBUG:boto:Token: None 2024-04-16T20:21:04.387 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.387 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk= 2024-04-16T20:21:04.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk=', 'Content-Length': '0'} 2024-04-16T20:21:04.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3cd88a4c67e356-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:21:04.389 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.389 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.389 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.390 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.390 DEBUG:boto:Method: GET 2024-04-16T20:21:04.390 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.390 DEBUG:boto:Data: 2024-04-16T20:21:04.390 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.390 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.390 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.390 DEBUG:boto:Params: {} 2024-04-16T20:21:04.390 DEBUG:boto:Token: None 2024-04-16T20:21:04.390 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.391 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T20:21:04.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T20:21:04.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002631b6757abd88f0-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.434 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.434 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.435 DEBUG:boto:path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.435 DEBUG:boto:auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.435 DEBUG:boto:Method: GET 2024-04-16T20:21:04.435 DEBUG:boto:Path: /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.435 DEBUG:boto:Data: 2024-04-16T20:21:04.435 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.435 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.435 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.435 DEBUG:boto:Params: {} 2024-04-16T20:21:04.435 DEBUG:boto:Token: None 2024-04-16T20:21:04.436 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58?versionId=null 2024-04-16T20:21:04.436 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T20:21:04.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T20:21:04.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023db64ec2210984b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.438 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.438 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.438 DEBUG:boto:path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.438 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.438 DEBUG:boto:Method: GET 2024-04-16T20:21:04.439 DEBUG:boto:Path: /yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.439 DEBUG:boto:Data: 2024-04-16T20:21:04.439 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.439 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.439 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.439 DEBUG:boto:Params: {} 2024-04-16T20:21:04.439 DEBUG:boto:Token: None 2024-04-16T20:21:04.439 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.439 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb53db5def8bc3a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.482 DEBUG:boto:b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.483 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.483 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.483 DEBUG:boto:path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.483 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.483 DEBUG:boto:Method: GET 2024-04-16T20:21:04.483 DEBUG:boto:Path: /yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.483 DEBUG:boto:Data: 2024-04-16T20:21:04.483 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.483 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.484 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.484 DEBUG:boto:Params: {} 2024-04-16T20:21:04.484 DEBUG:boto:Token: None 2024-04-16T20:21:04.484 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.484 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c013cc35273b7bd4-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.488 DEBUG:boto:b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.488 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.488 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.488 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.488 DEBUG:boto:Path: /yrjefh-56/obj58 2024-04-16T20:21:04.488 DEBUG:boto:Data: 2024-04-16T20:21:04.488 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.488 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.489 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.489 DEBUG:boto:Params: {} 2024-04-16T20:21:04.489 DEBUG:boto:Token: None 2024-04-16T20:21:04.489 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58 2024-04-16T20:21:04.489 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4= 2024-04-16T20:21:04.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4=', 'Content-Length': '0'} 2024-04-16T20:21:04.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c0b2be992ee66b-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.491 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.491 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.491 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.491 DEBUG:boto:Path: /yrjefh-56/obj58 2024-04-16T20:21:04.491 DEBUG:boto:Data: 2024-04-16T20:21:04.492 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.492 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.492 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.492 DEBUG:boto:Params: {} 2024-04-16T20:21:04.492 DEBUG:boto:Token: None 2024-04-16T20:21:04.492 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58 2024-04-16T20:21:04.492 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4= 2024-04-16T20:21:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4=', 'Content-Length': '0'} 2024-04-16T20:21:04.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d5e3d7ad6b46863-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-16T20:21:04.494 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.494 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.495 DEBUG:boto:Method: GET 2024-04-16T20:21:04.495 DEBUG:boto:Path: /yrjefh-56/obj58 2024-04-16T20:21:04.495 DEBUG:boto:Data: 2024-04-16T20:21:04.495 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.495 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.495 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.495 DEBUG:boto:Params: {} 2024-04-16T20:21:04.495 DEBUG:boto:Token: None 2024-04-16T20:21:04.495 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58 2024-04-16T20:21:04.495 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k= 2024-04-16T20:21:04.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k=', 'Content-Length': '0'} 2024-04-16T20:21:04.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04d8e32c63247d6-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.542 DEBUG:boto:path=/yrjefh-56/obj58 2024-04-16T20:21:04.543 DEBUG:boto:auth_path=/yrjefh-56/obj58 2024-04-16T20:21:04.543 DEBUG:boto:Method: GET 2024-04-16T20:21:04.543 DEBUG:boto:Path: /yrjefh-56/obj58 2024-04-16T20:21:04.543 DEBUG:boto:Data: 2024-04-16T20:21:04.543 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.543 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.543 DEBUG:boto:Params: {} 2024-04-16T20:21:04.543 DEBUG:boto:Token: None 2024-04-16T20:21:04.544 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj58 2024-04-16T20:21:04.544 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k= 2024-04-16T20:21:04.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k=', 'Content-Length': '0'} 2024-04-16T20:21:04.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47cd5f5ad95d37f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.546 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.546 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.546 DEBUG:boto:path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.547 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.547 DEBUG:boto:Method: GET 2024-04-16T20:21:04.547 DEBUG:boto:Path: /yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.547 DEBUG:boto:Data: 2024-04-16T20:21:04.547 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.547 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.547 DEBUG:boto:Params: {} 2024-04-16T20:21:04.547 DEBUG:boto:Token: None 2024-04-16T20:21:04.547 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.548 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4c34e46d19ff4a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.594 DEBUG:boto:b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.595 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.595 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.595 DEBUG:boto:path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.595 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.596 DEBUG:boto:Method: GET 2024-04-16T20:21:04.596 DEBUG:boto:Path: /yrjefh-56/?prefix=obj58 2024-04-16T20:21:04.596 DEBUG:boto:Data: 2024-04-16T20:21:04.596 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.596 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.596 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.596 DEBUG:boto:Params: {} 2024-04-16T20:21:04.596 DEBUG:boto:Token: None 2024-04-16T20:21:04.596 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.597 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27667b00d4cc34d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.600 DEBUG:boto:b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:21:04.601 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.601 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.601 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.601 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.602 DEBUG:boto:Method: GET 2024-04-16T20:21:04.602 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.602 DEBUG:boto:Data: 2024-04-16T20:21:04.602 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.602 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.602 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.602 DEBUG:boto:Params: {} 2024-04-16T20:21:04.602 DEBUG:boto:Token: None 2024-04-16T20:21:04.602 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T20:21:04.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T20:21:04.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2149174a06400bd-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.646 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.647 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.647 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.647 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.647 DEBUG:boto:Method: GET 2024-04-16T20:21:04.647 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.647 DEBUG:boto:Data: 2024-04-16T20:21:04.647 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.647 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.647 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.647 DEBUG:boto:Params: {} 2024-04-16T20:21:04.647 DEBUG:boto:Token: None 2024-04-16T20:21:04.648 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.648 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T20:21:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T20:21:04.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c46e48a537a377-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.650 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.651 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.651 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.651 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.651 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.651 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.651 DEBUG:boto:Data: 2024-04-16T20:21:04.651 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.651 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.651 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.651 DEBUG:boto:Params: {} 2024-04-16T20:21:04.651 DEBUG:boto:Token: None 2024-04-16T20:21:04.652 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.652 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw= 2024-04-16T20:21:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw=', 'Content-Length': '0'} 2024-04-16T20:21:04.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0bce9b0c233e2a4-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.654 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.654 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.654 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.654 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.654 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.654 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.654 DEBUG:boto:Data: 2024-04-16T20:21:04.655 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.655 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.655 DEBUG:boto:Params: {} 2024-04-16T20:21:04.655 DEBUG:boto:Token: None 2024-04-16T20:21:04.655 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.655 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw= 2024-04-16T20:21:04.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw=', 'Content-Length': '0'} 2024-04-16T20:21:04.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f25cf3359f5a0fc-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:21:04.658 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.658 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.658 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.658 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.658 DEBUG:boto:Method: GET 2024-04-16T20:21:04.658 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.658 DEBUG:boto:Data: 2024-04-16T20:21:04.658 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.659 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.659 DEBUG:boto:Params: {} 2024-04-16T20:21:04.659 DEBUG:boto:Token: None 2024-04-16T20:21:04.659 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T20:21:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T20:21:04.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ecb13d131d904d4-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.702 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.703 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.703 DEBUG:boto:path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.703 DEBUG:boto:auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.703 DEBUG:boto:Method: GET 2024-04-16T20:21:04.703 DEBUG:boto:Path: /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.703 DEBUG:boto:Data: 2024-04-16T20:21:04.703 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.703 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.703 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.704 DEBUG:boto:Params: {} 2024-04-16T20:21:04.704 DEBUG:boto:Token: None 2024-04-16T20:21:04.704 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62?versionId=null 2024-04-16T20:21:04.704 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T20:21:04.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T20:21:04.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69e3b19c1fe2c06-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.707 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.707 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.707 DEBUG:boto:path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.707 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.707 DEBUG:boto:Method: GET 2024-04-16T20:21:04.707 DEBUG:boto:Path: /yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.707 DEBUG:boto:Data: 2024-04-16T20:21:04.707 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.707 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.707 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.708 DEBUG:boto:Params: {} 2024-04-16T20:21:04.708 DEBUG:boto:Token: None 2024-04-16T20:21:04.708 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.708 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3da4d07328a70d-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.754 DEBUG:boto:b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.755 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.755 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.755 DEBUG:boto:path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.755 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.755 DEBUG:boto:Method: GET 2024-04-16T20:21:04.755 DEBUG:boto:Path: /yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.755 DEBUG:boto:Data: 2024-04-16T20:21:04.756 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.756 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.756 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.756 DEBUG:boto:Params: {} 2024-04-16T20:21:04.756 DEBUG:boto:Token: None 2024-04-16T20:21:04.756 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.756 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50d9e516df6d2ac-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.760 DEBUG:boto:b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.760 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.760 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.760 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.760 DEBUG:boto:Path: /yrjefh-56/obj62 2024-04-16T20:21:04.761 DEBUG:boto:Data: 2024-04-16T20:21:04.761 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.761 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.761 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.761 DEBUG:boto:Params: {} 2024-04-16T20:21:04.761 DEBUG:boto:Token: None 2024-04-16T20:21:04.761 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62 2024-04-16T20:21:04.761 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94= 2024-04-16T20:21:04.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94=', 'Content-Length': '0'} 2024-04-16T20:21:04.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7fc16a10e98459-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.764 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.764 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.764 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.764 DEBUG:boto:Path: /yrjefh-56/obj62 2024-04-16T20:21:04.764 DEBUG:boto:Data: 2024-04-16T20:21:04.764 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.764 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.764 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.764 DEBUG:boto:Params: {} 2024-04-16T20:21:04.764 DEBUG:boto:Token: None 2024-04-16T20:21:04.765 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62 2024-04-16T20:21:04.765 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94= 2024-04-16T20:21:04.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94=', 'Content-Length': '0'} 2024-04-16T20:21:04.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd39cc31e64c0d3-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-16T20:21:04.767 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.767 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.767 DEBUG:boto:Method: GET 2024-04-16T20:21:04.767 DEBUG:boto:Path: /yrjefh-56/obj62 2024-04-16T20:21:04.767 DEBUG:boto:Data: 2024-04-16T20:21:04.767 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.767 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.767 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.767 DEBUG:boto:Params: {} 2024-04-16T20:21:04.768 DEBUG:boto:Token: None 2024-04-16T20:21:04.768 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62 2024-04-16T20:21:04.768 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg= 2024-04-16T20:21:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg=', 'Content-Length': '0'} 2024-04-16T20:21:04.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aacf6f57860c98a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.814 DEBUG:boto:path=/yrjefh-56/obj62 2024-04-16T20:21:04.815 DEBUG:boto:auth_path=/yrjefh-56/obj62 2024-04-16T20:21:04.815 DEBUG:boto:Method: GET 2024-04-16T20:21:04.815 DEBUG:boto:Path: /yrjefh-56/obj62 2024-04-16T20:21:04.815 DEBUG:boto:Data: 2024-04-16T20:21:04.815 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.815 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.815 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.815 DEBUG:boto:Params: {} 2024-04-16T20:21:04.815 DEBUG:boto:Token: None 2024-04-16T20:21:04.816 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj62 2024-04-16T20:21:04.816 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg= 2024-04-16T20:21:04.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg=', 'Content-Length': '0'} 2024-04-16T20:21:04.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfc59f018f923e9-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.818 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.818 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.818 DEBUG:boto:path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.818 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.819 DEBUG:boto:Method: GET 2024-04-16T20:21:04.819 DEBUG:boto:Path: /yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.819 DEBUG:boto:Data: 2024-04-16T20:21:04.819 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.819 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.819 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.819 DEBUG:boto:Params: {} 2024-04-16T20:21:04.819 DEBUG:boto:Token: None 2024-04-16T20:21:04.819 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.819 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17b2b2ef4b69de3-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.862 DEBUG:boto:b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.863 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.863 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.863 DEBUG:boto:path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.863 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.863 DEBUG:boto:Method: GET 2024-04-16T20:21:04.864 DEBUG:boto:Path: /yrjefh-56/?prefix=obj62 2024-04-16T20:21:04.864 DEBUG:boto:Data: 2024-04-16T20:21:04.864 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.864 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.864 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.864 DEBUG:boto:Params: {} 2024-04-16T20:21:04.864 DEBUG:boto:Token: None 2024-04-16T20:21:04.864 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:04.864 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:04.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:04.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064895f8b5b9b6e8b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.868 DEBUG:boto:b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:04.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:21:04.869 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.869 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.869 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.869 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.869 DEBUG:boto:Method: GET 2024-04-16T20:21:04.869 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.869 DEBUG:boto:Data: 2024-04-16T20:21:04.869 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.869 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.869 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.870 DEBUG:boto:Params: {} 2024-04-16T20:21:04.870 DEBUG:boto:Token: None 2024-04-16T20:21:04.870 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.870 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T20:21:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T20:21:04.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6c351b3674dbaa-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.914 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.915 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.915 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.915 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.915 DEBUG:boto:Method: GET 2024-04-16T20:21:04.915 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.915 DEBUG:boto:Data: 2024-04-16T20:21:04.915 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.915 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.915 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.915 DEBUG:boto:Params: {} 2024-04-16T20:21:04.916 DEBUG:boto:Token: None 2024-04-16T20:21:04.916 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.916 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T20:21:04.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T20:21:04.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c7a95af762ea7b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.918 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.918 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.918 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.918 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.918 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.919 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.919 DEBUG:boto:Data: 2024-04-16T20:21:04.919 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.919 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.919 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.919 DEBUG:boto:Params: {} 2024-04-16T20:21:04.919 DEBUG:boto:Token: None 2024-04-16T20:21:04.919 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.919 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88= 2024-04-16T20:21:04.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88=', 'Content-Length': '0'} 2024-04-16T20:21:04.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd658f65b7c07b8a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.921 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.921 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.921 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.922 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.922 DEBUG:boto:Method: HEAD 2024-04-16T20:21:04.922 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.922 DEBUG:boto:Data: 2024-04-16T20:21:04.922 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.922 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.922 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.922 DEBUG:boto:Params: {} 2024-04-16T20:21:04.922 DEBUG:boto:Token: None 2024-04-16T20:21:04.922 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.923 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88= 2024-04-16T20:21:04.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88=', 'Content-Length': '0'} 2024-04-16T20:21:04.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5d1e994d95c271-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:21:04.925 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.925 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.925 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.925 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.925 DEBUG:boto:Method: GET 2024-04-16T20:21:04.925 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.925 DEBUG:boto:Data: 2024-04-16T20:21:04.925 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.925 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.925 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.926 DEBUG:boto:Params: {} 2024-04-16T20:21:04.926 DEBUG:boto:Token: None 2024-04-16T20:21:04.926 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.926 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T20:21:04.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T20:21:04.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14cdec17637d96d-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.994 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:04.995 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:04.995 DEBUG:boto:path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.995 DEBUG:boto:auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.995 DEBUG:boto:Method: GET 2024-04-16T20:21:04.995 DEBUG:boto:Path: /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.995 DEBUG:boto:Data: 2024-04-16T20:21:04.995 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:04.995 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.995 DEBUG:boto:Params: {} 2024-04-16T20:21:04.996 DEBUG:boto:Token: None 2024-04-16T20:21:04.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/obj68?versionId=null 2024-04-16T20:21:04.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T20:21:04.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T20:21:04.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee051b1bea5cf060-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:04.998 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:04.998 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:04.999 DEBUG:boto:path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:04.999 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:04.999 DEBUG:boto:Method: GET 2024-04-16T20:21:04.999 DEBUG:boto:Path: /yrjefh-56/?prefix=obj68 2024-04-16T20:21:04.999 DEBUG:boto:Data: 2024-04-16T20:21:04.999 DEBUG:boto:Headers: {} 2024-04-16T20:21:04.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:04.999 DEBUG:boto:Port: 8000 2024-04-16T20:21:04.999 DEBUG:boto:Params: {} 2024-04-16T20:21:04.999 DEBUG:boto:Token: None 2024-04-16T20:21:04.999 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:04 GMT /yrjefh-56/ 2024-04-16T20:21:05.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T20:21:05.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T20:21:05.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb033baf56a1151-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.045 DEBUG:boto:b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.046 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.046 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.047 DEBUG:boto:path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.047 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.047 DEBUG:boto:Method: GET 2024-04-16T20:21:05.047 DEBUG:boto:Path: /yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.047 DEBUG:boto:Data: 2024-04-16T20:21:05.047 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.047 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.047 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.047 DEBUG:boto:Params: {} 2024-04-16T20:21:05.047 DEBUG:boto:Token: None 2024-04-16T20:21:05.048 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.048 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccd5f2e4b146943-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.051 DEBUG:boto:b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.052 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:05.052 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:05.052 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.052 DEBUG:boto:Path: /yrjefh-56/obj68 2024-04-16T20:21:05.052 DEBUG:boto:Data: 2024-04-16T20:21:05.052 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.052 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.052 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.053 DEBUG:boto:Params: {} 2024-04-16T20:21:05.053 DEBUG:boto:Token: None 2024-04-16T20:21:05.053 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj68 2024-04-16T20:21:05.053 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM= 2024-04-16T20:21:05.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM=', 'Content-Length': '0'} 2024-04-16T20:21:05.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003851c4d99f5f3215-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.055 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:05.055 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:05.055 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.055 DEBUG:boto:Path: /yrjefh-56/obj68 2024-04-16T20:21:05.055 DEBUG:boto:Data: 2024-04-16T20:21:05.056 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.056 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.056 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.056 DEBUG:boto:Params: {} 2024-04-16T20:21:05.056 DEBUG:boto:Token: None 2024-04-16T20:21:05.056 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj68 2024-04-16T20:21:05.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM= 2024-04-16T20:21:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM=', 'Content-Length': '0'} 2024-04-16T20:21:05.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ac1b50b7bb0fb0-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-16T20:21:05.058 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:05.059 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:05.059 DEBUG:boto:Method: GET 2024-04-16T20:21:05.059 DEBUG:boto:Path: /yrjefh-56/obj68 2024-04-16T20:21:05.059 DEBUG:boto:Data: 2024-04-16T20:21:05.059 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.059 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.059 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.059 DEBUG:boto:Params: {} 2024-04-16T20:21:05.059 DEBUG:boto:Token: None 2024-04-16T20:21:05.059 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj68 2024-04-16T20:21:05.060 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU= 2024-04-16T20:21:05.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU=', 'Content-Length': '0'} 2024-04-16T20:21:05.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3ee234e1ef4470-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.102 DEBUG:boto:path=/yrjefh-56/obj68 2024-04-16T20:21:05.103 DEBUG:boto:auth_path=/yrjefh-56/obj68 2024-04-16T20:21:05.103 DEBUG:boto:Method: GET 2024-04-16T20:21:05.103 DEBUG:boto:Path: /yrjefh-56/obj68 2024-04-16T20:21:05.103 DEBUG:boto:Data: 2024-04-16T20:21:05.103 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.103 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.103 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.103 DEBUG:boto:Params: {} 2024-04-16T20:21:05.104 DEBUG:boto:Token: None 2024-04-16T20:21:05.104 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj68 2024-04-16T20:21:05.104 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU= 2024-04-16T20:21:05.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU=', 'Content-Length': '0'} 2024-04-16T20:21:05.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2df58775ed9bc2-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.106 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.107 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.107 DEBUG:boto:path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.107 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.107 DEBUG:boto:Method: GET 2024-04-16T20:21:05.107 DEBUG:boto:Path: /yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.107 DEBUG:boto:Data: 2024-04-16T20:21:05.107 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.107 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.107 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.107 DEBUG:boto:Params: {} 2024-04-16T20:21:05.107 DEBUG:boto:Token: None 2024-04-16T20:21:05.107 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.108 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a06bfd0da30b88-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.150 DEBUG:boto:b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.151 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.151 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.151 DEBUG:boto:path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.151 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.151 DEBUG:boto:Method: GET 2024-04-16T20:21:05.152 DEBUG:boto:Path: /yrjefh-56/?prefix=obj68 2024-04-16T20:21:05.152 DEBUG:boto:Data: 2024-04-16T20:21:05.152 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.152 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.152 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.152 DEBUG:boto:Params: {} 2024-04-16T20:21:05.152 DEBUG:boto:Token: None 2024-04-16T20:21:05.152 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.153 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b52ef8296c47cd-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.156 DEBUG:boto:b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:21:05.157 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.157 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.157 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.157 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.157 DEBUG:boto:Method: GET 2024-04-16T20:21:05.157 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.157 DEBUG:boto:Data: 2024-04-16T20:21:05.158 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.158 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.158 DEBUG:boto:Params: {} 2024-04-16T20:21:05.158 DEBUG:boto:Token: None 2024-04-16T20:21:05.158 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.158 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T20:21:05.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T20:21:05.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8dd6237f9566e8-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.202 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.203 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.203 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.203 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.203 DEBUG:boto:Method: GET 2024-04-16T20:21:05.203 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.203 DEBUG:boto:Data: 2024-04-16T20:21:05.203 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.204 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.204 DEBUG:boto:Params: {} 2024-04-16T20:21:05.204 DEBUG:boto:Token: None 2024-04-16T20:21:05.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T20:21:05.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T20:21:05.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068aec31d305df100-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.207 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.207 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.207 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.207 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.207 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.207 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.207 DEBUG:boto:Data: 2024-04-16T20:21:05.207 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.207 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.207 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.207 DEBUG:boto:Params: {} 2024-04-16T20:21:05.208 DEBUG:boto:Token: None 2024-04-16T20:21:05.208 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.208 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc= 2024-04-16T20:21:05.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc=', 'Content-Length': '0'} 2024-04-16T20:21:05.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545b7f98b4bf8a5a-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.210 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.210 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.210 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.210 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.210 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.210 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.210 DEBUG:boto:Data: 2024-04-16T20:21:05.211 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.211 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.211 DEBUG:boto:Params: {} 2024-04-16T20:21:05.211 DEBUG:boto:Token: None 2024-04-16T20:21:05.211 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc= 2024-04-16T20:21:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc=', 'Content-Length': '0'} 2024-04-16T20:21:05.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78906baac106c81-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:21:05.214 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.214 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.214 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.214 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.214 DEBUG:boto:Method: GET 2024-04-16T20:21:05.214 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.214 DEBUG:boto:Data: 2024-04-16T20:21:05.214 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.214 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.214 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.214 DEBUG:boto:Params: {} 2024-04-16T20:21:05.215 DEBUG:boto:Token: None 2024-04-16T20:21:05.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T20:21:05.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T20:21:05.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b29b54e11d59fa5-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.258 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.259 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.259 DEBUG:boto:path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.259 DEBUG:boto:auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.259 DEBUG:boto:Method: GET 2024-04-16T20:21:05.259 DEBUG:boto:Path: /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.259 DEBUG:boto:Data: 2024-04-16T20:21:05.259 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.259 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.259 DEBUG:boto:Params: {} 2024-04-16T20:21:05.259 DEBUG:boto:Token: None 2024-04-16T20:21:05.260 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69?versionId=null 2024-04-16T20:21:05.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T20:21:05.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T20:21:05.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce260ad80e8f0e61-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.262 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.262 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.262 DEBUG:boto:path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.262 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.263 DEBUG:boto:Method: GET 2024-04-16T20:21:05.263 DEBUG:boto:Path: /yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.263 DEBUG:boto:Data: 2024-04-16T20:21:05.263 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.263 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.263 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.263 DEBUG:boto:Params: {} 2024-04-16T20:21:05.263 DEBUG:boto:Token: None 2024-04-16T20:21:05.263 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a6ce1993b1665c-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.310 DEBUG:boto:b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.311 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.311 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.311 DEBUG:boto:path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.311 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.311 DEBUG:boto:Method: GET 2024-04-16T20:21:05.311 DEBUG:boto:Path: /yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.311 DEBUG:boto:Data: 2024-04-16T20:21:05.311 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.311 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.312 DEBUG:boto:Params: {} 2024-04-16T20:21:05.312 DEBUG:boto:Token: None 2024-04-16T20:21:05.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf499c2f163697b2-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.315 DEBUG:boto:b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.316 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.316 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.316 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.316 DEBUG:boto:Path: /yrjefh-56/obj69 2024-04-16T20:21:05.316 DEBUG:boto:Data: 2024-04-16T20:21:05.316 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.316 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.316 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.317 DEBUG:boto:Params: {} 2024-04-16T20:21:05.317 DEBUG:boto:Token: None 2024-04-16T20:21:05.317 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69 2024-04-16T20:21:05.317 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4= 2024-04-16T20:21:05.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4=', 'Content-Length': '0'} 2024-04-16T20:21:05.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056be71f6875f7453-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.319 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.319 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.319 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.319 DEBUG:boto:Path: /yrjefh-56/obj69 2024-04-16T20:21:05.319 DEBUG:boto:Data: 2024-04-16T20:21:05.319 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.319 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.319 DEBUG:boto:Params: {} 2024-04-16T20:21:05.320 DEBUG:boto:Token: None 2024-04-16T20:21:05.320 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69 2024-04-16T20:21:05.320 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4= 2024-04-16T20:21:05.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4=', 'Content-Length': '0'} 2024-04-16T20:21:05.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229bacbc2a64c923-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-16T20:21:05.322 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.322 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.322 DEBUG:boto:Method: GET 2024-04-16T20:21:05.322 DEBUG:boto:Path: /yrjefh-56/obj69 2024-04-16T20:21:05.322 DEBUG:boto:Data: 2024-04-16T20:21:05.323 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.323 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.323 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.323 DEBUG:boto:Params: {} 2024-04-16T20:21:05.323 DEBUG:boto:Token: None 2024-04-16T20:21:05.323 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69 2024-04-16T20:21:05.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo= 2024-04-16T20:21:05.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo=', 'Content-Length': '0'} 2024-04-16T20:21:05.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000754e46db76e48e84-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.366 DEBUG:boto:path=/yrjefh-56/obj69 2024-04-16T20:21:05.366 DEBUG:boto:auth_path=/yrjefh-56/obj69 2024-04-16T20:21:05.367 DEBUG:boto:Method: GET 2024-04-16T20:21:05.367 DEBUG:boto:Path: /yrjefh-56/obj69 2024-04-16T20:21:05.367 DEBUG:boto:Data: 2024-04-16T20:21:05.367 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.367 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.367 DEBUG:boto:Params: {} 2024-04-16T20:21:05.367 DEBUG:boto:Token: None 2024-04-16T20:21:05.367 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj69 2024-04-16T20:21:05.367 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo= 2024-04-16T20:21:05.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo=', 'Content-Length': '0'} 2024-04-16T20:21:05.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc53c687eedb156-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.370 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.370 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.370 DEBUG:boto:path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.370 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.370 DEBUG:boto:Method: GET 2024-04-16T20:21:05.370 DEBUG:boto:Path: /yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.370 DEBUG:boto:Data: 2024-04-16T20:21:05.370 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.370 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.370 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.371 DEBUG:boto:Params: {} 2024-04-16T20:21:05.371 DEBUG:boto:Token: None 2024-04-16T20:21:05.371 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.371 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071da58e5249c9e38-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.414 DEBUG:boto:b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.415 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.415 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.415 DEBUG:boto:path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.415 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.415 DEBUG:boto:Method: GET 2024-04-16T20:21:05.415 DEBUG:boto:Path: /yrjefh-56/?prefix=obj69 2024-04-16T20:21:05.415 DEBUG:boto:Data: 2024-04-16T20:21:05.415 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.415 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.416 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.416 DEBUG:boto:Params: {} 2024-04-16T20:21:05.416 DEBUG:boto:Token: None 2024-04-16T20:21:05.416 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.416 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0a4f26677bb4782-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.419 DEBUG:boto:b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:21:05.420 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.420 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.420 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.420 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.420 DEBUG:boto:Method: GET 2024-04-16T20:21:05.420 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.420 DEBUG:boto:Data: 2024-04-16T20:21:05.420 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.421 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.421 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.421 DEBUG:boto:Params: {} 2024-04-16T20:21:05.421 DEBUG:boto:Token: None 2024-04-16T20:21:05.421 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.421 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T20:21:05.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T20:21:05.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f3d71a2d90dbbd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.466 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.467 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.467 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.467 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.467 DEBUG:boto:Method: GET 2024-04-16T20:21:05.467 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.467 DEBUG:boto:Data: 2024-04-16T20:21:05.467 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.467 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.468 DEBUG:boto:Params: {} 2024-04-16T20:21:05.468 DEBUG:boto:Token: None 2024-04-16T20:21:05.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T20:21:05.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T20:21:05.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033916ba0d21d37bb-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.471 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.471 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.471 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.471 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.471 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.471 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.471 DEBUG:boto:Data: 2024-04-16T20:21:05.471 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.471 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.472 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.472 DEBUG:boto:Params: {} 2024-04-16T20:21:05.472 DEBUG:boto:Token: None 2024-04-16T20:21:05.472 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.472 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8= 2024-04-16T20:21:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8=', 'Content-Length': '0'} 2024-04-16T20:21:05.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6df2f3191ead07-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.474 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.474 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.475 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.475 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.475 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.475 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.475 DEBUG:boto:Data: 2024-04-16T20:21:05.475 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.475 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.475 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.475 DEBUG:boto:Params: {} 2024-04-16T20:21:05.475 DEBUG:boto:Token: None 2024-04-16T20:21:05.475 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.476 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8= 2024-04-16T20:21:05.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8=', 'Content-Length': '0'} 2024-04-16T20:21:05.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee929ef1fdbb8ecd-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:21:05.478 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.478 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.478 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.478 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.478 DEBUG:boto:Method: GET 2024-04-16T20:21:05.478 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.479 DEBUG:boto:Data: 2024-04-16T20:21:05.479 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.479 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.479 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.479 DEBUG:boto:Params: {} 2024-04-16T20:21:05.479 DEBUG:boto:Token: None 2024-04-16T20:21:05.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.479 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T20:21:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T20:21:05.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a539eccbe6ed4bfd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.522 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.523 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.523 DEBUG:boto:path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.523 DEBUG:boto:auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.523 DEBUG:boto:Method: GET 2024-04-16T20:21:05.523 DEBUG:boto:Path: /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.523 DEBUG:boto:Data: 2024-04-16T20:21:05.523 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.523 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.523 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.524 DEBUG:boto:Params: {} 2024-04-16T20:21:05.524 DEBUG:boto:Token: None 2024-04-16T20:21:05.524 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74?versionId=null 2024-04-16T20:21:05.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T20:21:05.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T20:21:05.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008746688aa1db6dd5-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.527 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.527 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.527 DEBUG:boto:path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.527 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.527 DEBUG:boto:Method: GET 2024-04-16T20:21:05.527 DEBUG:boto:Path: /yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.527 DEBUG:boto:Data: 2024-04-16T20:21:05.527 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.527 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.528 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.528 DEBUG:boto:Params: {} 2024-04-16T20:21:05.528 DEBUG:boto:Token: None 2024-04-16T20:21:05.528 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.528 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d34147d77ac4d9-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.574 DEBUG:boto:b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.575 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.575 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.575 DEBUG:boto:path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.575 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.575 DEBUG:boto:Method: GET 2024-04-16T20:21:05.575 DEBUG:boto:Path: /yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.575 DEBUG:boto:Data: 2024-04-16T20:21:05.576 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.576 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.576 DEBUG:boto:Params: {} 2024-04-16T20:21:05.576 DEBUG:boto:Token: None 2024-04-16T20:21:05.576 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f81b029b8d6fb9-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.580 DEBUG:boto:b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.580 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.580 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.580 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.580 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:21:05.581 DEBUG:boto:Data: 2024-04-16T20:21:05.581 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.581 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.581 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.581 DEBUG:boto:Params: {} 2024-04-16T20:21:05.581 DEBUG:boto:Token: None 2024-04-16T20:21:05.581 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74 2024-04-16T20:21:05.581 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY= 2024-04-16T20:21:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY=', 'Content-Length': '0'} 2024-04-16T20:21:05.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f15c28fdedbb839-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.584 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.584 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.584 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.584 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:21:05.584 DEBUG:boto:Data: 2024-04-16T20:21:05.584 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.584 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.584 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.584 DEBUG:boto:Params: {} 2024-04-16T20:21:05.584 DEBUG:boto:Token: None 2024-04-16T20:21:05.585 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74 2024-04-16T20:21:05.585 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY= 2024-04-16T20:21:05.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY=', 'Content-Length': '0'} 2024-04-16T20:21:05.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25fd9137bcd7479-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-16T20:21:05.587 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.587 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.587 DEBUG:boto:Method: GET 2024-04-16T20:21:05.587 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:21:05.587 DEBUG:boto:Data: 2024-04-16T20:21:05.587 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.588 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.588 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.588 DEBUG:boto:Params: {} 2024-04-16T20:21:05.588 DEBUG:boto:Token: None 2024-04-16T20:21:05.588 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74 2024-04-16T20:21:05.588 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls= 2024-04-16T20:21:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls=', 'Content-Length': '0'} 2024-04-16T20:21:05.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a178fe749731ddfe-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.630 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:05.630 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:05.631 DEBUG:boto:Method: GET 2024-04-16T20:21:05.631 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:21:05.631 DEBUG:boto:Data: 2024-04-16T20:21:05.631 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.631 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.631 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.631 DEBUG:boto:Params: {} 2024-04-16T20:21:05.631 DEBUG:boto:Token: None 2024-04-16T20:21:05.631 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj74 2024-04-16T20:21:05.631 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls= 2024-04-16T20:21:05.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls=', 'Content-Length': '0'} 2024-04-16T20:21:05.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cc97a3a8bed592-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.633 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.634 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.634 DEBUG:boto:path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.634 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.634 DEBUG:boto:Method: GET 2024-04-16T20:21:05.634 DEBUG:boto:Path: /yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.634 DEBUG:boto:Data: 2024-04-16T20:21:05.634 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.634 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.634 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.634 DEBUG:boto:Params: {} 2024-04-16T20:21:05.635 DEBUG:boto:Token: None 2024-04-16T20:21:05.635 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.635 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067dc3185b2ff53dd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.678 DEBUG:boto:b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.679 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.679 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.679 DEBUG:boto:path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.680 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.680 DEBUG:boto:Method: GET 2024-04-16T20:21:05.680 DEBUG:boto:Path: /yrjefh-56/?prefix=obj74 2024-04-16T20:21:05.680 DEBUG:boto:Data: 2024-04-16T20:21:05.680 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.680 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.680 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.680 DEBUG:boto:Params: {} 2024-04-16T20:21:05.680 DEBUG:boto:Token: None 2024-04-16T20:21:05.680 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.681 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006248aeb89a6c97e3-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.684 DEBUG:boto:b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:21:05.685 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.685 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.685 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.685 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.685 DEBUG:boto:Method: GET 2024-04-16T20:21:05.685 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.685 DEBUG:boto:Data: 2024-04-16T20:21:05.685 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.685 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.685 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.685 DEBUG:boto:Params: {} 2024-04-16T20:21:05.686 DEBUG:boto:Token: None 2024-04-16T20:21:05.686 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.686 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T20:21:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T20:21:05.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff02e0cd2fe1fb9d-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.730 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.730 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.730 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.731 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.731 DEBUG:boto:Method: GET 2024-04-16T20:21:05.731 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.731 DEBUG:boto:Data: 2024-04-16T20:21:05.731 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.731 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.731 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.731 DEBUG:boto:Params: {} 2024-04-16T20:21:05.731 DEBUG:boto:Token: None 2024-04-16T20:21:05.731 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.732 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T20:21:05.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T20:21:05.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac8d67c63d20f63-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.734 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.734 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.734 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.734 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.734 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.734 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.734 DEBUG:boto:Data: 2024-04-16T20:21:05.735 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.735 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.735 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.735 DEBUG:boto:Params: {} 2024-04-16T20:21:05.735 DEBUG:boto:Token: None 2024-04-16T20:21:05.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.735 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM= 2024-04-16T20:21:05.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM=', 'Content-Length': '0'} 2024-04-16T20:21:05.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae5e058a8443838-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.737 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.737 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.737 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.738 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.738 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.738 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.738 DEBUG:boto:Data: 2024-04-16T20:21:05.738 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.738 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.738 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.738 DEBUG:boto:Params: {} 2024-04-16T20:21:05.738 DEBUG:boto:Token: None 2024-04-16T20:21:05.738 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM= 2024-04-16T20:21:05.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM=', 'Content-Length': '0'} 2024-04-16T20:21:05.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6e37a9d259cea8-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:21:05.741 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.741 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.741 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.741 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.741 DEBUG:boto:Method: GET 2024-04-16T20:21:05.741 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.741 DEBUG:boto:Data: 2024-04-16T20:21:05.741 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.741 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.741 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.742 DEBUG:boto:Params: {} 2024-04-16T20:21:05.742 DEBUG:boto:Token: None 2024-04-16T20:21:05.742 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.742 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T20:21:05.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T20:21:05.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ae4e281e64c0fb-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.786 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.786 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.787 DEBUG:boto:path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.787 DEBUG:boto:auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.787 DEBUG:boto:Method: GET 2024-04-16T20:21:05.787 DEBUG:boto:Path: /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.787 DEBUG:boto:Data: 2024-04-16T20:21:05.787 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.787 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.787 DEBUG:boto:Params: {} 2024-04-16T20:21:05.787 DEBUG:boto:Token: None 2024-04-16T20:21:05.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76?versionId=null 2024-04-16T20:21:05.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T20:21:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T20:21:05.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b78e1a2ac8217a-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.790 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.790 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.790 DEBUG:boto:path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.790 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.790 DEBUG:boto:Method: GET 2024-04-16T20:21:05.790 DEBUG:boto:Path: /yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.790 DEBUG:boto:Data: 2024-04-16T20:21:05.790 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.791 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.791 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.791 DEBUG:boto:Params: {} 2024-04-16T20:21:05.791 DEBUG:boto:Token: None 2024-04-16T20:21:05.791 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.791 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118ea31890b1023d-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.834 DEBUG:boto:b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.835 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.835 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.835 DEBUG:boto:path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.835 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.835 DEBUG:boto:Method: GET 2024-04-16T20:21:05.835 DEBUG:boto:Path: /yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.835 DEBUG:boto:Data: 2024-04-16T20:21:05.835 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.835 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.835 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.836 DEBUG:boto:Params: {} 2024-04-16T20:21:05.836 DEBUG:boto:Token: None 2024-04-16T20:21:05.836 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.836 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbe1cd3d7b8a6aa-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.840 DEBUG:boto:b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.840 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.840 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.840 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.840 DEBUG:boto:Path: /yrjefh-56/obj76 2024-04-16T20:21:05.840 DEBUG:boto:Data: 2024-04-16T20:21:05.840 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.840 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.840 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.841 DEBUG:boto:Params: {} 2024-04-16T20:21:05.841 DEBUG:boto:Token: None 2024-04-16T20:21:05.841 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76 2024-04-16T20:21:05.841 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU= 2024-04-16T20:21:05.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU=', 'Content-Length': '0'} 2024-04-16T20:21:05.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aafff5b668d58a0-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.843 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.843 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.843 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.843 DEBUG:boto:Path: /yrjefh-56/obj76 2024-04-16T20:21:05.843 DEBUG:boto:Data: 2024-04-16T20:21:05.843 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.844 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.844 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.844 DEBUG:boto:Params: {} 2024-04-16T20:21:05.844 DEBUG:boto:Token: None 2024-04-16T20:21:05.844 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76 2024-04-16T20:21:05.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU= 2024-04-16T20:21:05.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU=', 'Content-Length': '0'} 2024-04-16T20:21:05.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3ce1ae11164907-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-16T20:21:05.846 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.846 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.846 DEBUG:boto:Method: GET 2024-04-16T20:21:05.847 DEBUG:boto:Path: /yrjefh-56/obj76 2024-04-16T20:21:05.847 DEBUG:boto:Data: 2024-04-16T20:21:05.847 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.847 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.847 DEBUG:boto:Params: {} 2024-04-16T20:21:05.847 DEBUG:boto:Token: None 2024-04-16T20:21:05.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76 2024-04-16T20:21:05.847 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY= 2024-04-16T20:21:05.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY=', 'Content-Length': '0'} 2024-04-16T20:21:05.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001341596a01a56c0a-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.891 DEBUG:boto:path=/yrjefh-56/obj76 2024-04-16T20:21:05.891 DEBUG:boto:auth_path=/yrjefh-56/obj76 2024-04-16T20:21:05.891 DEBUG:boto:Method: GET 2024-04-16T20:21:05.891 DEBUG:boto:Path: /yrjefh-56/obj76 2024-04-16T20:21:05.892 DEBUG:boto:Data: 2024-04-16T20:21:05.892 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.892 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.892 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.892 DEBUG:boto:Params: {} 2024-04-16T20:21:05.892 DEBUG:boto:Token: None 2024-04-16T20:21:05.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj76 2024-04-16T20:21:05.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY= 2024-04-16T20:21:05.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY=', 'Content-Length': '0'} 2024-04-16T20:21:05.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff546ec80349999-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.895 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.895 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.895 DEBUG:boto:path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.895 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.895 DEBUG:boto:Method: GET 2024-04-16T20:21:05.895 DEBUG:boto:Path: /yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.895 DEBUG:boto:Data: 2024-04-16T20:21:05.895 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.895 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.895 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.896 DEBUG:boto:Params: {} 2024-04-16T20:21:05.896 DEBUG:boto:Token: None 2024-04-16T20:21:05.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e3945546b21366-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.942 DEBUG:boto:b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.943 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:05.943 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:05.943 DEBUG:boto:path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.943 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.943 DEBUG:boto:Method: GET 2024-04-16T20:21:05.943 DEBUG:boto:Path: /yrjefh-56/?prefix=obj76 2024-04-16T20:21:05.944 DEBUG:boto:Data: 2024-04-16T20:21:05.944 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.944 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.944 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.944 DEBUG:boto:Params: {} 2024-04-16T20:21:05.944 DEBUG:boto:Token: None 2024-04-16T20:21:05.944 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/ 2024-04-16T20:21:05.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T20:21:05.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T20:21:05.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9ad2a6b28ed7ea-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.948 DEBUG:boto:b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:05.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:21:05.949 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:05.949 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:05.949 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.949 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.949 DEBUG:boto:Method: GET 2024-04-16T20:21:05.949 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.949 DEBUG:boto:Data: 2024-04-16T20:21:05.949 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.949 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.950 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.950 DEBUG:boto:Params: {} 2024-04-16T20:21:05.950 DEBUG:boto:Token: None 2024-04-16T20:21:05.950 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.950 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I= 2024-04-16T20:21:05.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I=', 'Content-Length': '0'} 2024-04-16T20:21:05.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667746c7a7b43d83-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.994 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:05.995 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:05.995 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.995 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.995 DEBUG:boto:Method: GET 2024-04-16T20:21:05.995 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.995 DEBUG:boto:Data: 2024-04-16T20:21:05.995 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.995 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:05.995 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.995 DEBUG:boto:Params: {} 2024-04-16T20:21:05.996 DEBUG:boto:Token: None 2024-04-16T20:21:05.996 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:05 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.996 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I= 2024-04-16T20:21:05.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I=', 'Content-Length': '0'} 2024-04-16T20:21:05.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db01a05f238cab3d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:05.998 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:05.999 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:05.999 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.999 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.999 DEBUG:boto:Method: HEAD 2024-04-16T20:21:05.999 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:05.999 DEBUG:boto:Data: 2024-04-16T20:21:05.999 DEBUG:boto:Headers: {} 2024-04-16T20:21:05.999 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:05.999 DEBUG:boto:Port: 8000 2024-04-16T20:21:05.999 DEBUG:boto:Params: {} 2024-04-16T20:21:05.999 DEBUG:boto:Token: None 2024-04-16T20:21:06.000 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw= 2024-04-16T20:21:06.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw=', 'Content-Length': '0'} 2024-04-16T20:21:06.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c3fb8436732cb-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.002 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.002 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.002 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.002 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.002 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.003 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.003 DEBUG:boto:Data: 2024-04-16T20:21:06.003 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.003 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.003 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.003 DEBUG:boto:Params: {} 2024-04-16T20:21:06.003 DEBUG:boto:Token: None 2024-04-16T20:21:06.003 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.003 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw= 2024-04-16T20:21:06.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw=', 'Content-Length': '0'} 2024-04-16T20:21:06.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035317566af605ea2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:21:06.006 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.006 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.006 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.006 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.006 DEBUG:boto:Method: GET 2024-04-16T20:21:06.006 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.006 DEBUG:boto:Data: 2024-04-16T20:21:06.006 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.006 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.007 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.007 DEBUG:boto:Params: {} 2024-04-16T20:21:06.007 DEBUG:boto:Token: None 2024-04-16T20:21:06.007 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.007 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w= 2024-04-16T20:21:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w=', 'Content-Length': '0'} 2024-04-16T20:21:06.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d6fd2593b11e74-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.050 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.051 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.051 DEBUG:boto:path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.051 DEBUG:boto:auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.051 DEBUG:boto:Method: GET 2024-04-16T20:21:06.051 DEBUG:boto:Path: /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.051 DEBUG:boto:Data: 2024-04-16T20:21:06.051 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.051 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.052 DEBUG:boto:Params: {} 2024-04-16T20:21:06.052 DEBUG:boto:Token: None 2024-04-16T20:21:06.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85?versionId=null 2024-04-16T20:21:06.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w= 2024-04-16T20:21:06.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w=', 'Content-Length': '0'} 2024-04-16T20:21:06.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c66ebb0207720d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.054 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.055 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.055 DEBUG:boto:path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.055 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.055 DEBUG:boto:Method: GET 2024-04-16T20:21:06.055 DEBUG:boto:Path: /yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.055 DEBUG:boto:Data: 2024-04-16T20:21:06.055 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.055 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.055 DEBUG:boto:Params: {} 2024-04-16T20:21:06.056 DEBUG:boto:Token: None 2024-04-16T20:21:06.056 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47dfa526385fdcc-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.098 DEBUG:boto:b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.099 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.099 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.100 DEBUG:boto:path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.100 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.100 DEBUG:boto:Method: GET 2024-04-16T20:21:06.100 DEBUG:boto:Path: /yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.100 DEBUG:boto:Data: 2024-04-16T20:21:06.100 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.100 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.100 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.100 DEBUG:boto:Params: {} 2024-04-16T20:21:06.100 DEBUG:boto:Token: None 2024-04-16T20:21:06.100 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.101 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f716ec2f278063d1-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.104 DEBUG:boto:b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.104 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.105 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.105 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.105 DEBUG:boto:Path: /yrjefh-56/obj85 2024-04-16T20:21:06.105 DEBUG:boto:Data: 2024-04-16T20:21:06.105 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.105 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.105 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.105 DEBUG:boto:Params: {} 2024-04-16T20:21:06.105 DEBUG:boto:Token: None 2024-04-16T20:21:06.105 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85 2024-04-16T20:21:06.106 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s= 2024-04-16T20:21:06.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s=', 'Content-Length': '0'} 2024-04-16T20:21:06.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b61609d56969e02d-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.108 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.108 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.108 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.108 DEBUG:boto:Path: /yrjefh-56/obj85 2024-04-16T20:21:06.108 DEBUG:boto:Data: 2024-04-16T20:21:06.108 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.108 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.108 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.109 DEBUG:boto:Params: {} 2024-04-16T20:21:06.109 DEBUG:boto:Token: None 2024-04-16T20:21:06.109 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85 2024-04-16T20:21:06.109 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s= 2024-04-16T20:21:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s=', 'Content-Length': '0'} 2024-04-16T20:21:06.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4894b46513ba63-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-16T20:21:06.111 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.111 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.111 DEBUG:boto:Method: GET 2024-04-16T20:21:06.111 DEBUG:boto:Path: /yrjefh-56/obj85 2024-04-16T20:21:06.112 DEBUG:boto:Data: 2024-04-16T20:21:06.112 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.112 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.112 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.112 DEBUG:boto:Params: {} 2024-04-16T20:21:06.112 DEBUG:boto:Token: None 2024-04-16T20:21:06.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85 2024-04-16T20:21:06.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug= 2024-04-16T20:21:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug=', 'Content-Length': '0'} 2024-04-16T20:21:06.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4757e7a09b267ae-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.154 DEBUG:boto:path=/yrjefh-56/obj85 2024-04-16T20:21:06.154 DEBUG:boto:auth_path=/yrjefh-56/obj85 2024-04-16T20:21:06.155 DEBUG:boto:Method: GET 2024-04-16T20:21:06.155 DEBUG:boto:Path: /yrjefh-56/obj85 2024-04-16T20:21:06.155 DEBUG:boto:Data: 2024-04-16T20:21:06.155 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.155 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.155 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.155 DEBUG:boto:Params: {} 2024-04-16T20:21:06.155 DEBUG:boto:Token: None 2024-04-16T20:21:06.155 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj85 2024-04-16T20:21:06.156 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug= 2024-04-16T20:21:06.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug=', 'Content-Length': '0'} 2024-04-16T20:21:06.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c674d6e127d52d2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.158 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.158 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.158 DEBUG:boto:path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.158 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.158 DEBUG:boto:Method: GET 2024-04-16T20:21:06.158 DEBUG:boto:Path: /yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.158 DEBUG:boto:Data: 2024-04-16T20:21:06.158 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.158 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.158 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.159 DEBUG:boto:Params: {} 2024-04-16T20:21:06.159 DEBUG:boto:Token: None 2024-04-16T20:21:06.159 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.159 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed024049a98ff6fd-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.202 DEBUG:boto:b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.203 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.203 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.203 DEBUG:boto:path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.203 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.203 DEBUG:boto:Method: GET 2024-04-16T20:21:06.203 DEBUG:boto:Path: /yrjefh-56/?prefix=obj85 2024-04-16T20:21:06.203 DEBUG:boto:Data: 2024-04-16T20:21:06.203 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.204 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.204 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.204 DEBUG:boto:Params: {} 2024-04-16T20:21:06.204 DEBUG:boto:Token: None 2024-04-16T20:21:06.204 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf5bf7dfe0701839-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.208 DEBUG:boto:b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:21:06.208 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.208 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.208 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.208 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.209 DEBUG:boto:Method: GET 2024-04-16T20:21:06.209 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.209 DEBUG:boto:Data: 2024-04-16T20:21:06.209 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.209 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.209 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.209 DEBUG:boto:Params: {} 2024-04-16T20:21:06.209 DEBUG:boto:Token: None 2024-04-16T20:21:06.209 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.209 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T20:21:06.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005368869f991be4c1-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.254 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.254 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.255 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.255 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.255 DEBUG:boto:Method: GET 2024-04-16T20:21:06.255 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.255 DEBUG:boto:Data: 2024-04-16T20:21:06.255 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.255 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.255 DEBUG:boto:Params: {} 2024-04-16T20:21:06.255 DEBUG:boto:Token: None 2024-04-16T20:21:06.255 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T20:21:06.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d205ff27c77404f-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.258 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.258 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.258 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.259 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.259 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.259 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.259 DEBUG:boto:Data: 2024-04-16T20:21:06.259 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.259 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.259 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.259 DEBUG:boto:Params: {} 2024-04-16T20:21:06.259 DEBUG:boto:Token: None 2024-04-16T20:21:06.259 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.259 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE= 2024-04-16T20:21:06.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE=', 'Content-Length': '0'} 2024-04-16T20:21:06.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432835f5fd3dc697-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.261 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.262 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.262 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.262 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.262 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.262 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.262 DEBUG:boto:Data: 2024-04-16T20:21:06.262 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.262 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.262 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.262 DEBUG:boto:Params: {} 2024-04-16T20:21:06.262 DEBUG:boto:Token: None 2024-04-16T20:21:06.263 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE= 2024-04-16T20:21:06.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE=', 'Content-Length': '0'} 2024-04-16T20:21:06.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e518c06b63a21d0e-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:21:06.265 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.265 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.266 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.266 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.266 DEBUG:boto:Method: GET 2024-04-16T20:21:06.266 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.266 DEBUG:boto:Data: 2024-04-16T20:21:06.266 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.266 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.266 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.266 DEBUG:boto:Params: {} 2024-04-16T20:21:06.266 DEBUG:boto:Token: None 2024-04-16T20:21:06.266 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T20:21:06.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a787b1d2dd3b9c5-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.310 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.310 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.311 DEBUG:boto:path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.311 DEBUG:boto:auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.311 DEBUG:boto:Method: GET 2024-04-16T20:21:06.311 DEBUG:boto:Path: /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.311 DEBUG:boto:Data: 2024-04-16T20:21:06.311 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.311 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.311 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.311 DEBUG:boto:Params: {} 2024-04-16T20:21:06.311 DEBUG:boto:Token: None 2024-04-16T20:21:06.312 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87?versionId=null 2024-04-16T20:21:06.312 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T20:21:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d758396e6649b7-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.314 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.314 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.315 DEBUG:boto:path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.315 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.315 DEBUG:boto:Method: GET 2024-04-16T20:21:06.315 DEBUG:boto:Path: /yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.315 DEBUG:boto:Data: 2024-04-16T20:21:06.315 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.315 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.315 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.315 DEBUG:boto:Params: {} 2024-04-16T20:21:06.315 DEBUG:boto:Token: None 2024-04-16T20:21:06.316 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.316 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3af1670891bda1c-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.362 DEBUG:boto:b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.363 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.363 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.363 DEBUG:boto:path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.363 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.363 DEBUG:boto:Method: GET 2024-04-16T20:21:06.363 DEBUG:boto:Path: /yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.363 DEBUG:boto:Data: 2024-04-16T20:21:06.363 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.363 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.364 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.364 DEBUG:boto:Params: {} 2024-04-16T20:21:06.364 DEBUG:boto:Token: None 2024-04-16T20:21:06.364 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.364 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c503899dc5ba84-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.367 DEBUG:boto:b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.368 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.368 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.368 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.368 DEBUG:boto:Path: /yrjefh-56/obj87 2024-04-16T20:21:06.368 DEBUG:boto:Data: 2024-04-16T20:21:06.368 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.368 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.368 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.368 DEBUG:boto:Params: {} 2024-04-16T20:21:06.369 DEBUG:boto:Token: None 2024-04-16T20:21:06.369 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87 2024-04-16T20:21:06.369 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4= 2024-04-16T20:21:06.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4=', 'Content-Length': '0'} 2024-04-16T20:21:06.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cce7d229ded101-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.371 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.371 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.371 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.371 DEBUG:boto:Path: /yrjefh-56/obj87 2024-04-16T20:21:06.371 DEBUG:boto:Data: 2024-04-16T20:21:06.371 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.371 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.372 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.372 DEBUG:boto:Params: {} 2024-04-16T20:21:06.372 DEBUG:boto:Token: None 2024-04-16T20:21:06.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87 2024-04-16T20:21:06.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4= 2024-04-16T20:21:06.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4=', 'Content-Length': '0'} 2024-04-16T20:21:06.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d67b1ac500de81-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-16T20:21:06.375 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.375 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.375 DEBUG:boto:Method: GET 2024-04-16T20:21:06.375 DEBUG:boto:Path: /yrjefh-56/obj87 2024-04-16T20:21:06.375 DEBUG:boto:Data: 2024-04-16T20:21:06.375 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.375 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.375 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.375 DEBUG:boto:Params: {} 2024-04-16T20:21:06.376 DEBUG:boto:Token: None 2024-04-16T20:21:06.376 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87 2024-04-16T20:21:06.376 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA= 2024-04-16T20:21:06.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA=', 'Content-Length': '0'} 2024-04-16T20:21:06.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff005fbce758781-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.418 DEBUG:boto:path=/yrjefh-56/obj87 2024-04-16T20:21:06.418 DEBUG:boto:auth_path=/yrjefh-56/obj87 2024-04-16T20:21:06.419 DEBUG:boto:Method: GET 2024-04-16T20:21:06.419 DEBUG:boto:Path: /yrjefh-56/obj87 2024-04-16T20:21:06.419 DEBUG:boto:Data: 2024-04-16T20:21:06.419 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.419 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.419 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.419 DEBUG:boto:Params: {} 2024-04-16T20:21:06.419 DEBUG:boto:Token: None 2024-04-16T20:21:06.419 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj87 2024-04-16T20:21:06.419 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA= 2024-04-16T20:21:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA=', 'Content-Length': '0'} 2024-04-16T20:21:06.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d845232564fbc7aa-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.422 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.422 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.422 DEBUG:boto:path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.422 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.423 DEBUG:boto:Method: GET 2024-04-16T20:21:06.423 DEBUG:boto:Path: /yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.423 DEBUG:boto:Data: 2024-04-16T20:21:06.423 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.423 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.423 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.423 DEBUG:boto:Params: {} 2024-04-16T20:21:06.423 DEBUG:boto:Token: None 2024-04-16T20:21:06.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000429691d94c297e9f-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.466 DEBUG:boto:b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.467 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.467 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.467 DEBUG:boto:path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.467 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.467 DEBUG:boto:Method: GET 2024-04-16T20:21:06.467 DEBUG:boto:Path: /yrjefh-56/?prefix=obj87 2024-04-16T20:21:06.467 DEBUG:boto:Data: 2024-04-16T20:21:06.467 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.467 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.467 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.467 DEBUG:boto:Params: {} 2024-04-16T20:21:06.468 DEBUG:boto:Token: None 2024-04-16T20:21:06.468 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.468 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025fe151e00411ef0-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.471 DEBUG:boto:b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:21:06.472 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.472 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.472 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.472 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.472 DEBUG:boto:Method: GET 2024-04-16T20:21:06.472 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.472 DEBUG:boto:Data: 2024-04-16T20:21:06.472 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.472 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.472 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.473 DEBUG:boto:Params: {} 2024-04-16T20:21:06.473 DEBUG:boto:Token: None 2024-04-16T20:21:06.473 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.473 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T20:21:06.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T20:21:06.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e5e4341a3beea8-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.518 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.518 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.519 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.519 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.519 DEBUG:boto:Method: GET 2024-04-16T20:21:06.519 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.519 DEBUG:boto:Data: 2024-04-16T20:21:06.519 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.519 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.519 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.519 DEBUG:boto:Params: {} 2024-04-16T20:21:06.519 DEBUG:boto:Token: None 2024-04-16T20:21:06.519 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T20:21:06.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T20:21:06.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2119cd7dc90e22-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.522 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.522 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.523 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.523 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.523 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.523 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.523 DEBUG:boto:Data: 2024-04-16T20:21:06.523 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.523 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.523 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.523 DEBUG:boto:Params: {} 2024-04-16T20:21:06.523 DEBUG:boto:Token: None 2024-04-16T20:21:06.524 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs= 2024-04-16T20:21:06.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs=', 'Content-Length': '0'} 2024-04-16T20:21:06.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e12ab679f4d0f0-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.526 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.526 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.526 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.527 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.527 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.527 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.527 DEBUG:boto:Data: 2024-04-16T20:21:06.527 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.527 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.527 DEBUG:boto:Params: {} 2024-04-16T20:21:06.527 DEBUG:boto:Token: None 2024-04-16T20:21:06.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs= 2024-04-16T20:21:06.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs=', 'Content-Length': '0'} 2024-04-16T20:21:06.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ea82438c759ce5-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:21:06.530 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.530 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.530 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.530 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.530 DEBUG:boto:Method: GET 2024-04-16T20:21:06.530 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.531 DEBUG:boto:Data: 2024-04-16T20:21:06.531 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.531 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.531 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.531 DEBUG:boto:Params: {} 2024-04-16T20:21:06.531 DEBUG:boto:Token: None 2024-04-16T20:21:06.531 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.531 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T20:21:06.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T20:21:06.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aefe54d0bd68a6-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.574 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.574 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.575 DEBUG:boto:path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.575 DEBUG:boto:auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.575 DEBUG:boto:Method: GET 2024-04-16T20:21:06.575 DEBUG:boto:Path: /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.575 DEBUG:boto:Data: 2024-04-16T20:21:06.575 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.575 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.575 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.575 DEBUG:boto:Params: {} 2024-04-16T20:21:06.575 DEBUG:boto:Token: None 2024-04-16T20:21:06.575 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92?versionId=null 2024-04-16T20:21:06.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T20:21:06.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T20:21:06.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8441928bc99cce-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.578 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.578 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.578 DEBUG:boto:path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.578 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.578 DEBUG:boto:Method: GET 2024-04-16T20:21:06.578 DEBUG:boto:Path: /yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.578 DEBUG:boto:Data: 2024-04-16T20:21:06.578 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.579 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.579 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.579 DEBUG:boto:Params: {} 2024-04-16T20:21:06.579 DEBUG:boto:Token: None 2024-04-16T20:21:06.579 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.579 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665dd6211bb479c8-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.626 DEBUG:boto:b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.627 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.627 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.627 DEBUG:boto:path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.627 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.627 DEBUG:boto:Method: GET 2024-04-16T20:21:06.627 DEBUG:boto:Path: /yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.627 DEBUG:boto:Data: 2024-04-16T20:21:06.627 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.628 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.628 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.628 DEBUG:boto:Params: {} 2024-04-16T20:21:06.628 DEBUG:boto:Token: None 2024-04-16T20:21:06.628 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f1a9f7afd673b9-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.632 DEBUG:boto:b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.632 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.632 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.632 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.632 DEBUG:boto:Path: /yrjefh-56/obj92 2024-04-16T20:21:06.632 DEBUG:boto:Data: 2024-04-16T20:21:06.632 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.632 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.633 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.633 DEBUG:boto:Params: {} 2024-04-16T20:21:06.633 DEBUG:boto:Token: None 2024-04-16T20:21:06.633 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92 2024-04-16T20:21:06.633 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc= 2024-04-16T20:21:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc=', 'Content-Length': '0'} 2024-04-16T20:21:06.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b037fb4597a436-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.635 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.635 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.635 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.635 DEBUG:boto:Path: /yrjefh-56/obj92 2024-04-16T20:21:06.636 DEBUG:boto:Data: 2024-04-16T20:21:06.636 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.636 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.636 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.636 DEBUG:boto:Params: {} 2024-04-16T20:21:06.636 DEBUG:boto:Token: None 2024-04-16T20:21:06.636 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92 2024-04-16T20:21:06.636 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc= 2024-04-16T20:21:06.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc=', 'Content-Length': '0'} 2024-04-16T20:21:06.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9cbd89444f9a0d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-16T20:21:06.639 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.639 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.639 DEBUG:boto:Method: GET 2024-04-16T20:21:06.639 DEBUG:boto:Path: /yrjefh-56/obj92 2024-04-16T20:21:06.639 DEBUG:boto:Data: 2024-04-16T20:21:06.639 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.639 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.639 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.639 DEBUG:boto:Params: {} 2024-04-16T20:21:06.639 DEBUG:boto:Token: None 2024-04-16T20:21:06.640 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92 2024-04-16T20:21:06.640 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE= 2024-04-16T20:21:06.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE=', 'Content-Length': '0'} 2024-04-16T20:21:06.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136901e9d855a133-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.683 DEBUG:boto:path=/yrjefh-56/obj92 2024-04-16T20:21:06.683 DEBUG:boto:auth_path=/yrjefh-56/obj92 2024-04-16T20:21:06.684 DEBUG:boto:Method: GET 2024-04-16T20:21:06.684 DEBUG:boto:Path: /yrjefh-56/obj92 2024-04-16T20:21:06.684 DEBUG:boto:Data: 2024-04-16T20:21:06.684 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.684 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.684 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.684 DEBUG:boto:Params: {} 2024-04-16T20:21:06.684 DEBUG:boto:Token: None 2024-04-16T20:21:06.684 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj92 2024-04-16T20:21:06.685 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE= 2024-04-16T20:21:06.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE=', 'Content-Length': '0'} 2024-04-16T20:21:06.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4011a1c78ed5c2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.687 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.687 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.687 DEBUG:boto:path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.687 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.687 DEBUG:boto:Method: GET 2024-04-16T20:21:06.687 DEBUG:boto:Path: /yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.687 DEBUG:boto:Data: 2024-04-16T20:21:06.688 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.688 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.688 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.688 DEBUG:boto:Params: {} 2024-04-16T20:21:06.688 DEBUG:boto:Token: None 2024-04-16T20:21:06.688 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.688 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30ce9d53d664b01-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.734 DEBUG:boto:b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.735 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.735 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.735 DEBUG:boto:path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.735 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.735 DEBUG:boto:Method: GET 2024-04-16T20:21:06.735 DEBUG:boto:Path: /yrjefh-56/?prefix=obj92 2024-04-16T20:21:06.735 DEBUG:boto:Data: 2024-04-16T20:21:06.735 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.736 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.736 DEBUG:boto:Params: {} 2024-04-16T20:21:06.736 DEBUG:boto:Token: None 2024-04-16T20:21:06.736 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4f45af4b936047-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.740 DEBUG:boto:b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:21:06.740 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.740 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.740 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.740 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.740 DEBUG:boto:Method: GET 2024-04-16T20:21:06.741 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.741 DEBUG:boto:Data: 2024-04-16T20:21:06.741 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.741 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.741 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.741 DEBUG:boto:Params: {} 2024-04-16T20:21:06.741 DEBUG:boto:Token: None 2024-04-16T20:21:06.741 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.741 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T20:21:06.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T20:21:06.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b258c41cfd6264-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.786 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.786 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.787 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.787 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.787 DEBUG:boto:Method: GET 2024-04-16T20:21:06.787 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.787 DEBUG:boto:Data: 2024-04-16T20:21:06.787 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.787 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.787 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.787 DEBUG:boto:Params: {} 2024-04-16T20:21:06.787 DEBUG:boto:Token: None 2024-04-16T20:21:06.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.788 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T20:21:06.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T20:21:06.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c87784171b295ef-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.790 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.790 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.791 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.791 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.791 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.791 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.791 DEBUG:boto:Data: 2024-04-16T20:21:06.791 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.791 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.791 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.791 DEBUG:boto:Params: {} 2024-04-16T20:21:06.791 DEBUG:boto:Token: None 2024-04-16T20:21:06.791 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.792 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I= 2024-04-16T20:21:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I=', 'Content-Length': '0'} 2024-04-16T20:21:06.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53d37ec9752e68b-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.794 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.794 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.794 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.794 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.794 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.794 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.795 DEBUG:boto:Data: 2024-04-16T20:21:06.795 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.795 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.795 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.795 DEBUG:boto:Params: {} 2024-04-16T20:21:06.795 DEBUG:boto:Token: None 2024-04-16T20:21:06.795 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.795 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I= 2024-04-16T20:21:06.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I=', 'Content-Length': '0'} 2024-04-16T20:21:06.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040970bc676b10214-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:21:06.798 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.798 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.798 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.798 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.798 DEBUG:boto:Method: GET 2024-04-16T20:21:06.798 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.798 DEBUG:boto:Data: 2024-04-16T20:21:06.798 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.798 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.798 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.799 DEBUG:boto:Params: {} 2024-04-16T20:21:06.799 DEBUG:boto:Token: None 2024-04-16T20:21:06.799 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.799 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T20:21:06.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T20:21:06.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c59fc77fe3a033-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.842 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.842 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.843 DEBUG:boto:path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.843 DEBUG:boto:auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.843 DEBUG:boto:Method: GET 2024-04-16T20:21:06.843 DEBUG:boto:Path: /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.843 DEBUG:boto:Data: 2024-04-16T20:21:06.843 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.843 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.843 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.843 DEBUG:boto:Params: {} 2024-04-16T20:21:06.843 DEBUG:boto:Token: None 2024-04-16T20:21:06.844 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93?versionId=null 2024-04-16T20:21:06.844 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T20:21:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T20:21:06.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd331e91196ebdf1-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.846 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.846 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.846 DEBUG:boto:path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.847 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.847 DEBUG:boto:Method: GET 2024-04-16T20:21:06.847 DEBUG:boto:Path: /yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.847 DEBUG:boto:Data: 2024-04-16T20:21:06.847 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.847 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.847 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.847 DEBUG:boto:Params: {} 2024-04-16T20:21:06.847 DEBUG:boto:Token: None 2024-04-16T20:21:06.847 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.848 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e97cb2524e6cae-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.890 DEBUG:boto:b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.891 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.891 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.891 DEBUG:boto:path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.891 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.891 DEBUG:boto:Method: GET 2024-04-16T20:21:06.891 DEBUG:boto:Path: /yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.891 DEBUG:boto:Data: 2024-04-16T20:21:06.891 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.891 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.892 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.892 DEBUG:boto:Params: {} 2024-04-16T20:21:06.892 DEBUG:boto:Token: None 2024-04-16T20:21:06.892 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.892 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4b0a39bdefc42f-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.896 DEBUG:boto:b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.896 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.896 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.896 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.897 DEBUG:boto:Path: /yrjefh-56/obj93 2024-04-16T20:21:06.897 DEBUG:boto:Data: 2024-04-16T20:21:06.897 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.897 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.897 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.897 DEBUG:boto:Params: {} 2024-04-16T20:21:06.897 DEBUG:boto:Token: None 2024-04-16T20:21:06.897 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93 2024-04-16T20:21:06.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE= 2024-04-16T20:21:06.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE=', 'Content-Length': '0'} 2024-04-16T20:21:06.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007062356cbdbbb261-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.900 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.900 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.900 DEBUG:boto:Method: HEAD 2024-04-16T20:21:06.900 DEBUG:boto:Path: /yrjefh-56/obj93 2024-04-16T20:21:06.900 DEBUG:boto:Data: 2024-04-16T20:21:06.900 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.900 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.900 DEBUG:boto:Params: {} 2024-04-16T20:21:06.901 DEBUG:boto:Token: None 2024-04-16T20:21:06.901 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93 2024-04-16T20:21:06.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE= 2024-04-16T20:21:06.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE=', 'Content-Length': '0'} 2024-04-16T20:21:06.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243957e7d08f87de-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-16T20:21:06.903 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.904 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.904 DEBUG:boto:Method: GET 2024-04-16T20:21:06.904 DEBUG:boto:Path: /yrjefh-56/obj93 2024-04-16T20:21:06.904 DEBUG:boto:Data: 2024-04-16T20:21:06.904 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.904 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.904 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.904 DEBUG:boto:Params: {} 2024-04-16T20:21:06.904 DEBUG:boto:Token: None 2024-04-16T20:21:06.904 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93 2024-04-16T20:21:06.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ= 2024-04-16T20:21:06.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068423b1907649369-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.950 DEBUG:boto:path=/yrjefh-56/obj93 2024-04-16T20:21:06.950 DEBUG:boto:auth_path=/yrjefh-56/obj93 2024-04-16T20:21:06.951 DEBUG:boto:Method: GET 2024-04-16T20:21:06.951 DEBUG:boto:Path: /yrjefh-56/obj93 2024-04-16T20:21:06.951 DEBUG:boto:Data: 2024-04-16T20:21:06.951 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:06.951 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.951 DEBUG:boto:Params: {} 2024-04-16T20:21:06.951 DEBUG:boto:Token: None 2024-04-16T20:21:06.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/obj93 2024-04-16T20:21:06.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ= 2024-04-16T20:21:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ=', 'Content-Length': '0'} 2024-04-16T20:21:06.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c9aeca4cbe29ec-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.954 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.954 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.954 DEBUG:boto:path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.954 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.954 DEBUG:boto:Method: GET 2024-04-16T20:21:06.955 DEBUG:boto:Path: /yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.955 DEBUG:boto:Data: 2024-04-16T20:21:06.955 DEBUG:boto:Headers: {} 2024-04-16T20:21:06.955 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:06.955 DEBUG:boto:Port: 8000 2024-04-16T20:21:06.955 DEBUG:boto:Params: {} 2024-04-16T20:21:06.955 DEBUG:boto:Token: None 2024-04-16T20:21:06.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:06 GMT /yrjefh-56/ 2024-04-16T20:21:06.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T20:21:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T20:21:06.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb0ba661079ca30-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:06.998 DEBUG:boto:b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:06.999 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:06.999 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:06.999 DEBUG:boto:path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.999 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.999 DEBUG:boto:Method: GET 2024-04-16T20:21:06.999 DEBUG:boto:Path: /yrjefh-56/?prefix=obj93 2024-04-16T20:21:06.999 DEBUG:boto:Data: 2024-04-16T20:21:07.000 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.000 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.000 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.000 DEBUG:boto:Params: {} 2024-04-16T20:21:07.000 DEBUG:boto:Token: None 2024-04-16T20:21:07.000 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/ 2024-04-16T20:21:07.000 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T20:21:07.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T20:21:07.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351cb8c80cdc7b96-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.004 DEBUG:boto:b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:07.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:21:07.004 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.004 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.005 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.005 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.005 DEBUG:boto:Method: GET 2024-04-16T20:21:07.005 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.005 DEBUG:boto:Data: 2024-04-16T20:21:07.005 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.005 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.005 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.005 DEBUG:boto:Params: {} 2024-04-16T20:21:07.005 DEBUG:boto:Token: None 2024-04-16T20:21:07.006 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.006 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T20:21:07.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T20:21:07.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e2b0d6e80683e4-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.050 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.051 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.051 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.051 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.051 DEBUG:boto:Method: GET 2024-04-16T20:21:07.051 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.051 DEBUG:boto:Data: 2024-04-16T20:21:07.051 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.051 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.051 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.052 DEBUG:boto:Params: {} 2024-04-16T20:21:07.052 DEBUG:boto:Token: None 2024-04-16T20:21:07.052 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.052 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T20:21:07.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T20:21:07.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075255f167c77c068-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.054 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.054 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.055 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.055 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.055 DEBUG:boto:Method: HEAD 2024-04-16T20:21:07.055 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.055 DEBUG:boto:Data: 2024-04-16T20:21:07.055 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.055 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.055 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.055 DEBUG:boto:Params: {} 2024-04-16T20:21:07.055 DEBUG:boto:Token: None 2024-04-16T20:21:07.056 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.056 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw= 2024-04-16T20:21:07.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw=', 'Content-Length': '0'} 2024-04-16T20:21:07.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc60e41b08bf95d0-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.058 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.058 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.058 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.058 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.058 DEBUG:boto:Method: HEAD 2024-04-16T20:21:07.058 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.058 DEBUG:boto:Data: 2024-04-16T20:21:07.058 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.058 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.058 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.059 DEBUG:boto:Params: {} 2024-04-16T20:21:07.059 DEBUG:boto:Token: None 2024-04-16T20:21:07.059 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.059 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw= 2024-04-16T20:21:07.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw=', 'Content-Length': '0'} 2024-04-16T20:21:07.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5e2d3afa5e915d-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:21:07.061 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.061 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.061 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.061 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.062 DEBUG:boto:Method: GET 2024-04-16T20:21:07.062 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.062 DEBUG:boto:Data: 2024-04-16T20:21:07.062 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.062 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.062 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.062 DEBUG:boto:Params: {} 2024-04-16T20:21:07.062 DEBUG:boto:Token: None 2024-04-16T20:21:07.062 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.062 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T20:21:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T20:21:07.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d3b93506faf68a-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.110 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.111 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.111 DEBUG:boto:path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.111 DEBUG:boto:auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.111 DEBUG:boto:Method: GET 2024-04-16T20:21:07.111 DEBUG:boto:Path: /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.111 DEBUG:boto:Data: 2024-04-16T20:21:07.111 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.111 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.111 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.111 DEBUG:boto:Params: {} 2024-04-16T20:21:07.112 DEBUG:boto:Token: None 2024-04-16T20:21:07.112 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95?versionId=null 2024-04-16T20:21:07.112 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T20:21:07.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T20:21:07.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009027035021fb4487-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.115 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:07.115 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:07.115 DEBUG:boto:path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.115 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.115 DEBUG:boto:Method: GET 2024-04-16T20:21:07.115 DEBUG:boto:Path: /yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.115 DEBUG:boto:Data: 2024-04-16T20:21:07.115 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.116 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.116 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.116 DEBUG:boto:Params: {} 2024-04-16T20:21:07.116 DEBUG:boto:Token: None 2024-04-16T20:21:07.116 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/ 2024-04-16T20:21:07.116 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T20:21:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T20:21:07.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91a35ad0b0e69c6-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.162 DEBUG:boto:b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:07.163 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:07.163 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:07.163 DEBUG:boto:path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.163 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.163 DEBUG:boto:Method: GET 2024-04-16T20:21:07.163 DEBUG:boto:Path: /yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.163 DEBUG:boto:Data: 2024-04-16T20:21:07.164 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.164 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.164 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.164 DEBUG:boto:Params: {} 2024-04-16T20:21:07.164 DEBUG:boto:Token: None 2024-04-16T20:21:07.164 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/ 2024-04-16T20:21:07.164 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T20:21:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T20:21:07.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fedd94b90f42d60-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.168 DEBUG:boto:b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:07.168 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.168 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.168 DEBUG:boto:Method: HEAD 2024-04-16T20:21:07.169 DEBUG:boto:Path: /yrjefh-56/obj95 2024-04-16T20:21:07.169 DEBUG:boto:Data: 2024-04-16T20:21:07.169 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.169 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.169 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.169 DEBUG:boto:Params: {} 2024-04-16T20:21:07.169 DEBUG:boto:Token: None 2024-04-16T20:21:07.169 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95 2024-04-16T20:21:07.169 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY= 2024-04-16T20:21:07.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY=', 'Content-Length': '0'} 2024-04-16T20:21:07.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3f4d2982cbee8a-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.171 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.171 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.172 DEBUG:boto:Method: HEAD 2024-04-16T20:21:07.172 DEBUG:boto:Path: /yrjefh-56/obj95 2024-04-16T20:21:07.172 DEBUG:boto:Data: 2024-04-16T20:21:07.172 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.172 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.172 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.172 DEBUG:boto:Params: {} 2024-04-16T20:21:07.172 DEBUG:boto:Token: None 2024-04-16T20:21:07.172 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95 2024-04-16T20:21:07.173 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY= 2024-04-16T20:21:07.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY=', 'Content-Length': '0'} 2024-04-16T20:21:07.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e7bcf64ab0d600-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-16T20:21:07.175 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.175 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.175 DEBUG:boto:Method: GET 2024-04-16T20:21:07.175 DEBUG:boto:Path: /yrjefh-56/obj95 2024-04-16T20:21:07.175 DEBUG:boto:Data: 2024-04-16T20:21:07.175 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.175 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.175 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.175 DEBUG:boto:Params: {} 2024-04-16T20:21:07.176 DEBUG:boto:Token: None 2024-04-16T20:21:07.176 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95 2024-04-16T20:21:07.176 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg= 2024-04-16T20:21:07.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg=', 'Content-Length': '0'} 2024-04-16T20:21:07.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad3a033cc28b143-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.218 DEBUG:boto:path=/yrjefh-56/obj95 2024-04-16T20:21:07.218 DEBUG:boto:auth_path=/yrjefh-56/obj95 2024-04-16T20:21:07.219 DEBUG:boto:Method: GET 2024-04-16T20:21:07.219 DEBUG:boto:Path: /yrjefh-56/obj95 2024-04-16T20:21:07.219 DEBUG:boto:Data: 2024-04-16T20:21:07.219 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.219 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.219 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.219 DEBUG:boto:Params: {} 2024-04-16T20:21:07.219 DEBUG:boto:Token: None 2024-04-16T20:21:07.219 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/obj95 2024-04-16T20:21:07.219 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg= 2024-04-16T20:21:07.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg=', 'Content-Length': '0'} 2024-04-16T20:21:07.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cf14607b4cb02b-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.222 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:07.222 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:07.222 DEBUG:boto:path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.222 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.222 DEBUG:boto:Method: GET 2024-04-16T20:21:07.222 DEBUG:boto:Path: /yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.222 DEBUG:boto:Data: 2024-04-16T20:21:07.222 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.222 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:07.223 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.223 DEBUG:boto:Params: {} 2024-04-16T20:21:07.223 DEBUG:boto:Token: None 2024-04-16T20:21:07.223 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/ 2024-04-16T20:21:07.223 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T20:21:07.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T20:21:07.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002da838174754adc6-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.266 DEBUG:boto:b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:07.267 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:07.267 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:07.267 DEBUG:boto:path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.267 DEBUG:boto:auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.267 DEBUG:boto:Method: GET 2024-04-16T20:21:07.267 DEBUG:boto:Path: /yrjefh-56/?prefix=obj95 2024-04-16T20:21:07.267 DEBUG:boto:Data: 2024-04-16T20:21:07.267 DEBUG:boto:Headers: {} 2024-04-16T20:21:07.268 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:07.268 DEBUG:boto:Port: 8000 2024-04-16T20:21:07.268 DEBUG:boto:Params: {} 2024-04-16T20:21:07.268 DEBUG:boto:Token: None 2024-04-16T20:21:07.268 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:21:07 GMT /yrjefh-56/ 2024-04-16T20:21:07.268 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T20:21:07.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T20:21:07.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d893e8524b1e0c-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:07.271 DEBUG:boto:b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:21:07.272 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-56 zones={test-zone2, test-zone1} 2024-04-16T20:21:07.272 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:21:07.273 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:21:07.273 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:21:07.284+0000 7f9a2f9a4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 204040) UID: 0 2024-04-16T20:21:07.274 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:21:07.284+0000 7f9a35322880 -1 shutting down 2024-04-16T20:21:11.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:21:13.372 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:21:13.373 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:13.373 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:13.373 DEBUG:boto:Method: HEAD 2024-04-16T20:21:13.373 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:21:13.373 DEBUG:boto:Data: 2024-04-16T20:21:13.373 DEBUG:boto:Headers: {} 2024-04-16T20:21:13.373 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.374 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.374 DEBUG:boto:Params: {} 2024-04-16T20:21:13.374 DEBUG:boto:Token: None 2024-04-16T20:21:13.374 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/ 2024-04-16T20:21:13.374 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T20:21:13.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T20:21:13.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f829d13ca22c8352-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.378 DEBUG:boto:path=/yrjefh-56/obj37 2024-04-16T20:21:13.379 DEBUG:boto:auth_path=/yrjefh-56/obj37 2024-04-16T20:21:13.379 DEBUG:boto:Method: PUT 2024-04-16T20:21:13.379 DEBUG:boto:Path: /yrjefh-56/obj37 2024-04-16T20:21:13.379 DEBUG:boto:Data: 2024-04-16T20:21:13.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:21:13.379 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.379 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.379 DEBUG:boto:Params: {} 2024-04-16T20:21:13.379 DEBUG:boto:Token: None 2024-04-16T20:21:13.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/obj37 2024-04-16T20:21:13.380 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:/hAg3qh+yL80JLW3SP5g1Ts4fPk= 2024-04-16T20:21:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/hAg3qh+yL80JLW3SP5g1Ts4fPk='} 2024-04-16T20:21:13.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1a342c3445dc85a-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.389 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:13.389 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:13.389 DEBUG:boto:Method: HEAD 2024-04-16T20:21:13.390 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:21:13.390 DEBUG:boto:Data: 2024-04-16T20:21:13.390 DEBUG:boto:Headers: {} 2024-04-16T20:21:13.390 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.390 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.390 DEBUG:boto:Params: {} 2024-04-16T20:21:13.390 DEBUG:boto:Token: None 2024-04-16T20:21:13.390 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/ 2024-04-16T20:21:13.390 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T20:21:13.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T20:21:13.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008361c4bc81f1a64b-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.395 DEBUG:boto:path=/yrjefh-56/obj74 2024-04-16T20:21:13.395 DEBUG:boto:auth_path=/yrjefh-56/obj74 2024-04-16T20:21:13.395 DEBUG:boto:Method: PUT 2024-04-16T20:21:13.395 DEBUG:boto:Path: /yrjefh-56/obj74 2024-04-16T20:21:13.395 DEBUG:boto:Data: 2024-04-16T20:21:13.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:21:13.395 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.395 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.395 DEBUG:boto:Params: {} 2024-04-16T20:21:13.396 DEBUG:boto:Token: None 2024-04-16T20:21:13.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/obj74 2024-04-16T20:21:13.396 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:AZAZYrYGETmNhNz0UzeY2iO7HMg= 2024-04-16T20:21:13.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AZAZYrYGETmNhNz0UzeY2iO7HMg='} 2024-04-16T20:21:13.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066fc5917e68abad1-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.405 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:13.405 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:13.405 DEBUG:boto:Method: HEAD 2024-04-16T20:21:13.405 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:21:13.405 DEBUG:boto:Data: 2024-04-16T20:21:13.405 DEBUG:boto:Headers: {} 2024-04-16T20:21:13.405 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.406 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.406 DEBUG:boto:Params: {} 2024-04-16T20:21:13.406 DEBUG:boto:Token: None 2024-04-16T20:21:13.406 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/ 2024-04-16T20:21:13.406 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T20:21:13.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T20:21:13.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3c903edb5206057-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.409 DEBUG:boto:path=/yrjefh-56/obj111 2024-04-16T20:21:13.409 DEBUG:boto:auth_path=/yrjefh-56/obj111 2024-04-16T20:21:13.410 DEBUG:boto:Method: PUT 2024-04-16T20:21:13.410 DEBUG:boto:Path: /yrjefh-56/obj111 2024-04-16T20:21:13.410 DEBUG:boto:Data: 2024-04-16T20:21:13.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:21:13.410 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.410 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.410 DEBUG:boto:Params: {} 2024-04-16T20:21:13.410 DEBUG:boto:Token: None 2024-04-16T20:21:13.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/obj111 2024-04-16T20:21:13.411 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Rot7S0mGhIVm9gTMD6tjw7J6YFM= 2024-04-16T20:21:13.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rot7S0mGhIVm9gTMD6tjw7J6YFM='} 2024-04-16T20:21:13.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a504becd2104c4b6-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.455 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:13.455 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:13.455 DEBUG:boto:Method: HEAD 2024-04-16T20:21:13.455 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:21:13.455 DEBUG:boto:Data: 2024-04-16T20:21:13.455 DEBUG:boto:Headers: {} 2024-04-16T20:21:13.455 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.455 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.455 DEBUG:boto:Params: {} 2024-04-16T20:21:13.455 DEBUG:boto:Token: None 2024-04-16T20:21:13.455 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/ 2024-04-16T20:21:13.456 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T20:21:13.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T20:21:13.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9889bff69539171-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.459 DEBUG:boto:path=/yrjefh-56/obj148 2024-04-16T20:21:13.459 DEBUG:boto:auth_path=/yrjefh-56/obj148 2024-04-16T20:21:13.459 DEBUG:boto:Method: PUT 2024-04-16T20:21:13.459 DEBUG:boto:Path: /yrjefh-56/obj148 2024-04-16T20:21:13.459 DEBUG:boto:Data: 2024-04-16T20:21:13.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:21:13.460 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.460 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.460 DEBUG:boto:Params: {} 2024-04-16T20:21:13.460 DEBUG:boto:Token: None 2024-04-16T20:21:13.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/obj148 2024-04-16T20:21:13.460 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:oPWRb+iBYqVFzF5nm2KpN8g//b0= 2024-04-16T20:21:13.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oPWRb+iBYqVFzF5nm2KpN8g//b0='} 2024-04-16T20:21:13.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071cbd9aab550e7c0-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.469 DEBUG:boto:path=/yrjefh-56/ 2024-04-16T20:21:13.469 DEBUG:boto:auth_path=/yrjefh-56/ 2024-04-16T20:21:13.469 DEBUG:boto:Method: HEAD 2024-04-16T20:21:13.470 DEBUG:boto:Path: /yrjefh-56/ 2024-04-16T20:21:13.470 DEBUG:boto:Data: 2024-04-16T20:21:13.470 DEBUG:boto:Headers: {} 2024-04-16T20:21:13.470 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.470 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.470 DEBUG:boto:Params: {} 2024-04-16T20:21:13.470 DEBUG:boto:Token: None 2024-04-16T20:21:13.470 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/ 2024-04-16T20:21:13.471 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T20:21:13.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T20:21:13.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c978e03717b888c-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.474 DEBUG:boto:path=/yrjefh-56/obj185 2024-04-16T20:21:13.474 DEBUG:boto:auth_path=/yrjefh-56/obj185 2024-04-16T20:21:13.474 DEBUG:boto:Method: PUT 2024-04-16T20:21:13.474 DEBUG:boto:Path: /yrjefh-56/obj185 2024-04-16T20:21:13.474 DEBUG:boto:Data: 2024-04-16T20:21:13.474 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:21:13.474 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:21:13.474 DEBUG:boto:Port: 8000 2024-04-16T20:21:13.475 DEBUG:boto:Params: {} 2024-04-16T20:21:13.475 DEBUG:boto:Token: None 2024-04-16T20:21:13.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:21:13 GMT /yrjefh-56/obj185 2024-04-16T20:21:13.475 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:o/LjpnajVNEki5pIW+d0JhpWM08= 2024-04-16T20:21:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o/LjpnajVNEki5pIW+d0JhpWM08='} 2024-04-16T20:21:13.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d60fa368612721c4-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:21:13.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:21:13.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:21:13.519 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:21:13.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:21:13.575 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:21:13.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:13.642+0000 7fc5bb525800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T20:21:13.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:13.643+0000 7fc5bb525800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T20:21:13.669 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:21:13.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:21:13.672 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:21:13.672 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:21:13.691 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:21:13.691 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:21:13.691 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:21:13.692 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:21:13.870 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:21:13.881+0000 7f87e5025880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:21:14.692 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:21:14.713 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:21:14.714 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:21:14.714 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:21:14.715 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:21:14.715 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-56 2024-04-16T20:21:14.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:21:14.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:21:14.716 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:21:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:21:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:21:14.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:14.883+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:14.873 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:14.873 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:17.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:21:19.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:19.886+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:19.875 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:19.875 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:24.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:24.888+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:24.878 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:24.878 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:29.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:29.890+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:29.880 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:29.880 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:34.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:34.892+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:34.882 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:34.883 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:39.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:39.895+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:39.885 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:39.885 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:44.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:44.898+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:44.887 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:44.887 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:49.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:49.900+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:49.890 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:49.890 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:54.902+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:54.892 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:54.892 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:21:59.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:21:59.905+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:21:59.894 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:21:59.894 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:04.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:04.907+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:04.897 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:04.897 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:09.910+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:09.899 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:09.899 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:14.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:14.911+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:14.902 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:14.902 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:19.914+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:24.917+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:24.911 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:24.911 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:29.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:29.919+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:29.909 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:29.909 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:34.922+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:34.911 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:34.912 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:39.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:39.924+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:39.914 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:39.914 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:44.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:44.927+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:44.916 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:44.916 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:49.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:49.929+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:49.919 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:49.919 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:54.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:54.930+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:54.921 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:54.921 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:22:59.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:22:59.934+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:22:59.923 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:22:59.923 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:04.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:04.936+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:04.926 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:23:04.926 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:09.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:09.938+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:09.928 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:23:09.928 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:14.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:14.941+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:14.930 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T20:23:14.930 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:19.037 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T20:23:19.048+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T20:23:19.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:19.943+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:19.933 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:19.933 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:24.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:24.945+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:24.935 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:24.935 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:29.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:29.948+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:29.937 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:29.937 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:34.949+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:34.940 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:34.940 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:39.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:39.953+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:39.942 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:39.942 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:44.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:44.955+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:44.944 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:44.944 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:49.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:49.957+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:49.947 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:49.947 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:54.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:54.960+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:54.949 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:54.949 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:59.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:23:59.962+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:23:59.952 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:23:59.952 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:04.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:04.965+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:04.954 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:04.954 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:09.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:09.967+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:09.956 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:09.957 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:14.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:14.968+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:14.959 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:14.959 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:19.972+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:19.961 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:19.961 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:24.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:24.974+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:24.963 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:24.964 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:29.976+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:29.966 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:29.966 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:34.979+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:34.968 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:34.968 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:39.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:39.981+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:39.971 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:39.971 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:44.984+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:44.973 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:44.973 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:49.986+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:54.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:54.988+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:54.978 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:54.978 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:59.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:24:59.991+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:24:59.980 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:24:59.980 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:04.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:04.993+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:04.982 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:04.982 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:09.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:09.995+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:09.985 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:09.986 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:14.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:14.997+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:14.987 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:14.987 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:19.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:20.000+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:19.989 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:19.989 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:24.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:25.002+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:24.991 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:24.991 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:29.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:30.004+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:29.993 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:29.993 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:35.005+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:34.995 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:34.996 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:39.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:40.009+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:39.998 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:39.998 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:45.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:45.011+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:45.000 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:45.000 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:50.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:50.014+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:50.003 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:50.003 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:55.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:25:55.016+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:25:55.006 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:25:55.006 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:26:00.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:00.019+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:26:00.008 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:26:00.008 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:26:05.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:05.021+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T20:26:05.010 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:26:05.010 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T20:26:10.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:10.023+0000 7f264892c800 0 bucket checkpoint timed out waiting for incremental sync to catch up 2024-04-16T20:26:10.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:10.023+0000 7f264892c800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T20:26:10.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:10.023+0000 7f264892c800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T20:26:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-16T20:26:10.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-16T20:26:10.225 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:10.225 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:10.225 DEBUG:boto:Method: PUT 2024-04-16T20:26:10.225 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:10.226 DEBUG:boto:Data: 2024-04-16T20:26:10.226 DEBUG:boto:Headers: {} 2024-04-16T20:26:10.226 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:10.226 DEBUG:boto:Port: 8000 2024-04-16T20:26:10.226 DEBUG:boto:Params: {} 2024-04-16T20:26:10.226 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:26:10.227 DEBUG:boto:Token: None 2024-04-16T20:26:10.227 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:26:10 GMT /yrjefh-57/ 2024-04-16T20:26:10.227 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:PUnEJvfiHZ2OTRlTftfOUbIkvpg= 2024-04-16T20:26:10.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PUnEJvfiHZ2OTRlTftfOUbIkvpg=', 'Content-Length': '0'} 2024-04-16T20:26:10.250 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f524d439354292b6-00661edee2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:10.250 DEBUG:rgw_multi.tests:created bucket=yrjefh-57 2024-04-16T20:26:10.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:10.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:10.251 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:26:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:26:10.386 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299170.258360_1249.1","last_update":"2024-04-16T20:26:10.258360Z"}] 2024-04-16T20:26:10.386 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T20:26:10.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:26:10.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:10.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:10.387 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:26:10.443 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:10.443 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:10.510+0000 7f09cd733800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:26:10.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:10.512+0000 7f09cd733800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:26:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.248736_1246.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.248736Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:10Z"} 2024-04-16T20:26:10.524 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.248736_1246.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.248736Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:10Z"} 2024-04-16T20:26:10.524 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299170.248736_1246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:10.248736Z', 'realm_epoch': 2}}] 2024-04-16T20:26:10.525 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T20:26:10.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.248736_1246.1'} 2024-04-16T20:26:10.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299170.258360_1249.1 target=1_1713299170.248736_1246.1 2024-04-16T20:26:15.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:15.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:26:15.526 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:26:15.583 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:15.584 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:15.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:15.651+0000 7fb89d40f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:26:15.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:15.652+0000 7fb89d40f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:26:15.664 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.258360_1249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.258360Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:15Z"} 2024-04-16T20:26:15.664 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.258360_1249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.258360Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:15Z"} 2024-04-16T20:26:15.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299170.258360_1249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:10.258360Z', 'realm_epoch': 2}}] 2024-04-16T20:26:15.664 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T20:26:15.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T20:26:15.664 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:26:15.665 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:15.665 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:15.665 DEBUG:boto:Method: HEAD 2024-04-16T20:26:15.665 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:15.665 DEBUG:boto:Data: 2024-04-16T20:26:15.665 DEBUG:boto:Headers: {} 2024-04-16T20:26:15.665 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.665 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.665 DEBUG:boto:Params: {} 2024-04-16T20:26:15.666 DEBUG:boto:Token: None 2024-04-16T20:26:15.666 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/ 2024-04-16T20:26:15.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T20:26:15.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T20:26:15.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047ab9968071be02c-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.669 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:15.670 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:15.670 DEBUG:boto:Method: PUT 2024-04-16T20:26:15.670 DEBUG:boto:Path: /yrjefh-57/obj11 2024-04-16T20:26:15.670 DEBUG:boto:Data: 2024-04-16T20:26:15.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:15.670 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.670 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.670 DEBUG:boto:Params: {} 2024-04-16T20:26:15.670 DEBUG:boto:Token: None 2024-04-16T20:26:15.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/obj11 2024-04-16T20:26:15.671 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:u+rvuw8ONbIoH4AZ4Z5biRAsorY= 2024-04-16T20:26:15.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u+rvuw8ONbIoH4AZ4Z5biRAsorY='} 2024-04-16T20:26:15.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8fd0d9f77226023-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.678 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:15.678 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:15.678 DEBUG:boto:Method: HEAD 2024-04-16T20:26:15.678 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:15.678 DEBUG:boto:Data: 2024-04-16T20:26:15.679 DEBUG:boto:Headers: {} 2024-04-16T20:26:15.679 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.679 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.679 DEBUG:boto:Params: {} 2024-04-16T20:26:15.679 DEBUG:boto:Token: None 2024-04-16T20:26:15.679 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/ 2024-04-16T20:26:15.679 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T20:26:15.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T20:26:15.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b01f4366e896c5-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.682 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:15.682 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:15.682 DEBUG:boto:Method: PUT 2024-04-16T20:26:15.682 DEBUG:boto:Path: /yrjefh-57/obj22 2024-04-16T20:26:15.682 DEBUG:boto:Data: 2024-04-16T20:26:15.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:15.683 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.683 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.683 DEBUG:boto:Params: {} 2024-04-16T20:26:15.683 DEBUG:boto:Token: None 2024-04-16T20:26:15.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/obj22 2024-04-16T20:26:15.683 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:NKbYH6MwN6vBn5gSnKn1NvND4b4= 2024-04-16T20:26:15.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NKbYH6MwN6vBn5gSnKn1NvND4b4='} 2024-04-16T20:26:15.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c664263e98ab876-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.688 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:15.688 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:15.688 DEBUG:boto:Method: HEAD 2024-04-16T20:26:15.688 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:15.688 DEBUG:boto:Data: 2024-04-16T20:26:15.688 DEBUG:boto:Headers: {} 2024-04-16T20:26:15.688 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.689 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.689 DEBUG:boto:Params: {} 2024-04-16T20:26:15.689 DEBUG:boto:Token: None 2024-04-16T20:26:15.689 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/ 2024-04-16T20:26:15.689 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T20:26:15.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T20:26:15.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad46bb10c82e4331-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.691 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:15.692 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:15.692 DEBUG:boto:Method: PUT 2024-04-16T20:26:15.692 DEBUG:boto:Path: /yrjefh-57/obj33 2024-04-16T20:26:15.692 DEBUG:boto:Data: 2024-04-16T20:26:15.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:15.692 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.692 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.692 DEBUG:boto:Params: {} 2024-04-16T20:26:15.692 DEBUG:boto:Token: None 2024-04-16T20:26:15.693 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/obj33 2024-04-16T20:26:15.693 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:7pDGAU0exG4cANmcmw6iRnWvqjM= 2024-04-16T20:26:15.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7pDGAU0exG4cANmcmw6iRnWvqjM='} 2024-04-16T20:26:15.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aece47cc9661271b-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.735 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:15.735 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:15.735 DEBUG:boto:Method: HEAD 2024-04-16T20:26:15.735 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:15.735 DEBUG:boto:Data: 2024-04-16T20:26:15.735 DEBUG:boto:Headers: {} 2024-04-16T20:26:15.735 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.735 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.735 DEBUG:boto:Params: {} 2024-04-16T20:26:15.735 DEBUG:boto:Token: None 2024-04-16T20:26:15.735 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/ 2024-04-16T20:26:15.736 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T20:26:15.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T20:26:15.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf67295a9dfe56d3-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.738 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:15.738 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:15.738 DEBUG:boto:Method: PUT 2024-04-16T20:26:15.738 DEBUG:boto:Path: /yrjefh-57/obj44 2024-04-16T20:26:15.739 DEBUG:boto:Data: 2024-04-16T20:26:15.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:15.739 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.739 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.739 DEBUG:boto:Params: {} 2024-04-16T20:26:15.739 DEBUG:boto:Token: None 2024-04-16T20:26:15.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/obj44 2024-04-16T20:26:15.739 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kjgOeXlKgskTO1xDij/yzSeSLtY= 2024-04-16T20:26:15.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kjgOeXlKgskTO1xDij/yzSeSLtY='} 2024-04-16T20:26:15.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed4ac8a55c4e75e0-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.746 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:15.746 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:15.746 DEBUG:boto:Method: HEAD 2024-04-16T20:26:15.746 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:15.746 DEBUG:boto:Data: 2024-04-16T20:26:15.746 DEBUG:boto:Headers: {} 2024-04-16T20:26:15.747 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.747 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.747 DEBUG:boto:Params: {} 2024-04-16T20:26:15.747 DEBUG:boto:Token: None 2024-04-16T20:26:15.747 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/ 2024-04-16T20:26:15.747 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T20:26:15.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T20:26:15.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caeed73ae130c8b3-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.749 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:15.750 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:15.750 DEBUG:boto:Method: PUT 2024-04-16T20:26:15.750 DEBUG:boto:Path: /yrjefh-57/obj55 2024-04-16T20:26:15.750 DEBUG:boto:Data: 2024-04-16T20:26:15.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:15.750 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:15.750 DEBUG:boto:Port: 8000 2024-04-16T20:26:15.750 DEBUG:boto:Params: {} 2024-04-16T20:26:15.750 DEBUG:boto:Token: None 2024-04-16T20:26:15.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:15 GMT /yrjefh-57/obj55 2024-04-16T20:26:15.751 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5+F1F4W4g1yzLVW6lSRdxcjZCnM= 2024-04-16T20:26:15.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5+F1F4W4g1yzLVW6lSRdxcjZCnM='} 2024-04-16T20:26:15.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4e8c87ba9e49113-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:15.756 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-57 2024-04-16T20:26:15.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:15.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:15.756 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:26:15.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:15.812 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:15.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:15.880+0000 7f085a71f800 1 waiting to reach incremental sync.. 2024-04-16T20:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:20.882+0000 7f085a71f800 1 bucket sync caught up with source: 2024-04-16T20:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr: local status: [, , , , , , , , 00000000001.110.6, 00000000002.147.6, 00000000002.123.6] 2024-04-16T20:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr: remote markers: [, , , , , , , , 00000000001.110.6, 00000000002.147.6, 00000000002.123.6] 2024-04-16T20:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:20.882+0000 7f085a71f800 0 bucket checkpoint complete 2024-04-16T20:26:20.899 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-57 2024-04-16T20:26:20.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:20.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:20.899 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T20:26:20.962 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:20.962 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:21.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:26:21.037+0000 7f12cb12c800 1 bucket sync caught up with source: 2024-04-16T20:26:21.027 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.394.6, 00000000002.310.6, 00000000002.165.6] 2024-04-16T20:26:21.027 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.394.6, 00000000002.310.6, 00000000002.165.6] 2024-04-16T20:26:21.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T20:26:21.037+0000 7f12cb12c800 0 bucket checkpoint complete 2024-04-16T20:26:21.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=yrjefh-57 zones={test-zone2, test-zone1} 2024-04-16T20:26:21.050 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.050 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.050 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.050 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:21.050 DEBUG:boto:Data: 2024-04-16T20:26:21.050 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.050 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.050 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.050 DEBUG:boto:Params: {} 2024-04-16T20:26:21.051 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:26:21.051 DEBUG:boto:Token: None 2024-04-16T20:26:21.051 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.051 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk= 2024-04-16T20:26:21.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk=', 'Content-Length': '0'} 2024-04-16T20:26:21.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001766dbf7e64bd757-00661edeed-6330-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.057 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.057 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.058 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.058 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:21.058 DEBUG:boto:Data: 2024-04-16T20:26:21.058 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.058 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.058 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.058 DEBUG:boto:Params: {} 2024-04-16T20:26:21.058 DEBUG:boto:Token: None 2024-04-16T20:26:21.058 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.058 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk= 2024-04-16T20:26:21.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk=', 'Content-Length': '0'} 2024-04-16T20:26:21.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000374c3c137ec81427-00661edeed-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-16T20:26:21.061 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.061 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.062 DEBUG:boto:path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.062 DEBUG:boto:auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.062 DEBUG:boto:Method: GET 2024-04-16T20:26:21.062 DEBUG:boto:Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.062 DEBUG:boto:Data: 2024-04-16T20:26:21.062 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.062 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.062 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.062 DEBUG:boto:Params: {} 2024-04-16T20:26:21.062 DEBUG:boto:Token: None 2024-04-16T20:26:21.063 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/?versions 2024-04-16T20:26:21.063 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T20:26:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T20:26:21.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018f9c92a8a25410d-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.065 DEBUG:boto:b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:26:21.067 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-16T20:26:21.067 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.067 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.067 DEBUG:boto:path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.067 DEBUG:boto:auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.068 DEBUG:boto:Method: GET 2024-04-16T20:26:21.068 DEBUG:boto:Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.068 DEBUG:boto:Data: 2024-04-16T20:26:21.068 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.068 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.068 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.068 DEBUG:boto:Params: {} 2024-04-16T20:26:21.068 DEBUG:boto:Token: None 2024-04-16T20:26:21.068 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/?versions 2024-04-16T20:26:21.068 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T20:26:21.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T20:26:21.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5b2a48fefdc0930-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.071 DEBUG:boto:b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-16T20:26:21.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-16T20:26:21.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-16T20:26:21.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-16T20:26:21.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-16T20:26:21.072 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.072 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.072 DEBUG:boto:path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.072 DEBUG:boto:auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.072 DEBUG:boto:Method: GET 2024-04-16T20:26:21.072 DEBUG:boto:Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.072 DEBUG:boto:Data: 2024-04-16T20:26:21.072 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.072 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.072 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.073 DEBUG:boto:Params: {} 2024-04-16T20:26:21.073 DEBUG:boto:Token: None 2024-04-16T20:26:21.073 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/?versions 2024-04-16T20:26:21.073 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T20:26:21.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T20:26:21.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d946f14a9bfe900a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.076 DEBUG:boto:b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.076 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.076 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.076 DEBUG:boto:path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.076 DEBUG:boto:auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.077 DEBUG:boto:Method: GET 2024-04-16T20:26:21.077 DEBUG:boto:Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T20:26:21.077 DEBUG:boto:Data: 2024-04-16T20:26:21.077 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.077 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.077 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.077 DEBUG:boto:Params: {} 2024-04-16T20:26:21.077 DEBUG:boto:Token: None 2024-04-16T20:26:21.077 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/?versions 2024-04-16T20:26:21.077 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T20:26:21.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T20:26:21.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003025006259e538a7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.080 DEBUG:boto:b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:26:21.080 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.081 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.081 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.081 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.081 DEBUG:boto:Method: GET 2024-04-16T20:26:21.081 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.081 DEBUG:boto:Data: 2024-04-16T20:26:21.081 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.081 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.081 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.081 DEBUG:boto:Params: {} 2024-04-16T20:26:21.081 DEBUG:boto:Token: None 2024-04-16T20:26:21.082 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.082 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T20:26:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T20:26:21.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d2718ebef2c16f-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.084 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.084 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.085 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.085 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.085 DEBUG:boto:Method: GET 2024-04-16T20:26:21.085 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.085 DEBUG:boto:Data: 2024-04-16T20:26:21.085 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.085 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.085 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.085 DEBUG:boto:Params: {} 2024-04-16T20:26:21.085 DEBUG:boto:Token: None 2024-04-16T20:26:21.085 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.086 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T20:26:21.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T20:26:21.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8306c6c057e6bee-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.130 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.130 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.131 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.131 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.131 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.131 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.131 DEBUG:boto:Data: 2024-04-16T20:26:21.131 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.131 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.131 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.131 DEBUG:boto:Params: {} 2024-04-16T20:26:21.131 DEBUG:boto:Token: None 2024-04-16T20:26:21.131 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.132 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4= 2024-04-16T20:26:21.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4=', 'Content-Length': '0'} 2024-04-16T20:26:21.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be480d25df94232-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.134 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.134 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.134 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.134 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.134 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.134 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.134 DEBUG:boto:Data: 2024-04-16T20:26:21.134 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.134 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.134 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.134 DEBUG:boto:Params: {} 2024-04-16T20:26:21.135 DEBUG:boto:Token: None 2024-04-16T20:26:21.135 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.135 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4= 2024-04-16T20:26:21.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4=', 'Content-Length': '0'} 2024-04-16T20:26:21.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff565c39002bab85-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:26:21.137 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.137 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.137 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.137 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.137 DEBUG:boto:Method: GET 2024-04-16T20:26:21.137 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.137 DEBUG:boto:Data: 2024-04-16T20:26:21.137 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.137 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.137 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.137 DEBUG:boto:Params: {} 2024-04-16T20:26:21.137 DEBUG:boto:Token: None 2024-04-16T20:26:21.138 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.138 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T20:26:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T20:26:21.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bffde4ec6b970ad-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.140 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.140 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.140 DEBUG:boto:path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.140 DEBUG:boto:auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.140 DEBUG:boto:Method: GET 2024-04-16T20:26:21.140 DEBUG:boto:Path: /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.141 DEBUG:boto:Data: 2024-04-16T20:26:21.141 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.141 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.141 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.141 DEBUG:boto:Params: {} 2024-04-16T20:26:21.141 DEBUG:boto:Token: None 2024-04-16T20:26:21.141 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11?versionId=null 2024-04-16T20:26:21.141 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T20:26:21.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T20:26:21.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63daf7fbea6031f-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.143 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.143 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.143 DEBUG:boto:path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.143 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.143 DEBUG:boto:Method: GET 2024-04-16T20:26:21.143 DEBUG:boto:Path: /yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.144 DEBUG:boto:Data: 2024-04-16T20:26:21.144 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.144 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.144 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.144 DEBUG:boto:Params: {} 2024-04-16T20:26:21.144 DEBUG:boto:Token: None 2024-04-16T20:26:21.144 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.144 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d346933c2fe71d3-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.147 DEBUG:boto:b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.147 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.147 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.147 DEBUG:boto:path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.147 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.148 DEBUG:boto:Method: GET 2024-04-16T20:26:21.148 DEBUG:boto:Path: /yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.148 DEBUG:boto:Data: 2024-04-16T20:26:21.148 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.148 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.148 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.148 DEBUG:boto:Params: {} 2024-04-16T20:26:21.148 DEBUG:boto:Token: None 2024-04-16T20:26:21.148 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.148 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ccaf7c24f74d6f7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.150 DEBUG:boto:b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.151 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.151 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.151 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.151 DEBUG:boto:Path: /yrjefh-57/obj11 2024-04-16T20:26:21.151 DEBUG:boto:Data: 2024-04-16T20:26:21.151 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.151 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.151 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.151 DEBUG:boto:Params: {} 2024-04-16T20:26:21.152 DEBUG:boto:Token: None 2024-04-16T20:26:21.152 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11 2024-04-16T20:26:21.152 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k= 2024-04-16T20:26:21.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k=', 'Content-Length': '0'} 2024-04-16T20:26:21.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2737f80efa546a5-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.154 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.154 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.154 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.154 DEBUG:boto:Path: /yrjefh-57/obj11 2024-04-16T20:26:21.154 DEBUG:boto:Data: 2024-04-16T20:26:21.154 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.154 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.154 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.154 DEBUG:boto:Params: {} 2024-04-16T20:26:21.154 DEBUG:boto:Token: None 2024-04-16T20:26:21.155 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11 2024-04-16T20:26:21.155 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k= 2024-04-16T20:26:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k=', 'Content-Length': '0'} 2024-04-16T20:26:21.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a673a8af4cc69e9-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-16T20:26:21.156 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.157 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.157 DEBUG:boto:Method: GET 2024-04-16T20:26:21.157 DEBUG:boto:Path: /yrjefh-57/obj11 2024-04-16T20:26:21.157 DEBUG:boto:Data: 2024-04-16T20:26:21.157 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.157 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.157 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.157 DEBUG:boto:Params: {} 2024-04-16T20:26:21.157 DEBUG:boto:Token: None 2024-04-16T20:26:21.157 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11 2024-04-16T20:26:21.157 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg= 2024-04-16T20:26:21.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg=', 'Content-Length': '0'} 2024-04-16T20:26:21.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4fe5f8e7caa87b-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.202 DEBUG:boto:path=/yrjefh-57/obj11 2024-04-16T20:26:21.202 DEBUG:boto:auth_path=/yrjefh-57/obj11 2024-04-16T20:26:21.203 DEBUG:boto:Method: GET 2024-04-16T20:26:21.203 DEBUG:boto:Path: /yrjefh-57/obj11 2024-04-16T20:26:21.203 DEBUG:boto:Data: 2024-04-16T20:26:21.203 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.203 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.203 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.203 DEBUG:boto:Params: {} 2024-04-16T20:26:21.203 DEBUG:boto:Token: None 2024-04-16T20:26:21.203 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj11 2024-04-16T20:26:21.204 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg= 2024-04-16T20:26:21.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg=', 'Content-Length': '0'} 2024-04-16T20:26:21.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ab8f854609d566-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.205 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.206 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.206 DEBUG:boto:path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.206 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.206 DEBUG:boto:Method: GET 2024-04-16T20:26:21.206 DEBUG:boto:Path: /yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.206 DEBUG:boto:Data: 2024-04-16T20:26:21.206 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.206 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.206 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.206 DEBUG:boto:Params: {} 2024-04-16T20:26:21.207 DEBUG:boto:Token: None 2024-04-16T20:26:21.207 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da616b9efa3ee973-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.209 DEBUG:boto:b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.210 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.210 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.210 DEBUG:boto:path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.210 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.210 DEBUG:boto:Method: GET 2024-04-16T20:26:21.210 DEBUG:boto:Path: /yrjefh-57/?prefix=obj11 2024-04-16T20:26:21.210 DEBUG:boto:Data: 2024-04-16T20:26:21.210 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.211 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.211 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.211 DEBUG:boto:Params: {} 2024-04-16T20:26:21.211 DEBUG:boto:Token: None 2024-04-16T20:26:21.211 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.211 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0877970da5dc3f-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.213 DEBUG:boto:b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:26:21.214 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.214 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.214 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.214 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.214 DEBUG:boto:Method: GET 2024-04-16T20:26:21.214 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.214 DEBUG:boto:Data: 2024-04-16T20:26:21.214 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.215 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.215 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.215 DEBUG:boto:Params: {} 2024-04-16T20:26:21.215 DEBUG:boto:Token: None 2024-04-16T20:26:21.215 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.215 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T20:26:21.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T20:26:21.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff377e9aabd7c1ba-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.262 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.263 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.263 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.263 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.263 DEBUG:boto:Method: GET 2024-04-16T20:26:21.263 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.263 DEBUG:boto:Data: 2024-04-16T20:26:21.263 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.263 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.263 DEBUG:boto:Params: {} 2024-04-16T20:26:21.264 DEBUG:boto:Token: None 2024-04-16T20:26:21.264 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T20:26:21.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T20:26:21.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12271cb34327826-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.266 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.266 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.266 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.266 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.266 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.266 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.266 DEBUG:boto:Data: 2024-04-16T20:26:21.267 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.267 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.267 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.267 DEBUG:boto:Params: {} 2024-04-16T20:26:21.267 DEBUG:boto:Token: None 2024-04-16T20:26:21.267 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.267 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso= 2024-04-16T20:26:21.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso=', 'Content-Length': '0'} 2024-04-16T20:26:21.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068128ee794828c3-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.269 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.269 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.269 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.269 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.269 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.270 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.270 DEBUG:boto:Data: 2024-04-16T20:26:21.270 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.270 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.270 DEBUG:boto:Params: {} 2024-04-16T20:26:21.270 DEBUG:boto:Token: None 2024-04-16T20:26:21.270 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.270 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso= 2024-04-16T20:26:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso=', 'Content-Length': '0'} 2024-04-16T20:26:21.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c15fd973f20e749-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:26:21.272 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.272 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.273 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.273 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.273 DEBUG:boto:Method: GET 2024-04-16T20:26:21.273 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.273 DEBUG:boto:Data: 2024-04-16T20:26:21.273 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.273 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.273 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.273 DEBUG:boto:Params: {} 2024-04-16T20:26:21.273 DEBUG:boto:Token: None 2024-04-16T20:26:21.273 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T20:26:21.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T20:26:21.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2aa55c84f263081-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.318 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.318 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.318 DEBUG:boto:path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.319 DEBUG:boto:auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.319 DEBUG:boto:Method: GET 2024-04-16T20:26:21.319 DEBUG:boto:Path: /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.319 DEBUG:boto:Data: 2024-04-16T20:26:21.319 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.319 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.319 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.319 DEBUG:boto:Params: {} 2024-04-16T20:26:21.319 DEBUG:boto:Token: None 2024-04-16T20:26:21.319 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22?versionId=null 2024-04-16T20:26:21.319 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T20:26:21.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T20:26:21.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a669536802e6d1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.321 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.321 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.321 DEBUG:boto:path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.322 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.322 DEBUG:boto:Method: GET 2024-04-16T20:26:21.322 DEBUG:boto:Path: /yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.322 DEBUG:boto:Data: 2024-04-16T20:26:21.322 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.322 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.322 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.322 DEBUG:boto:Params: {} 2024-04-16T20:26:21.322 DEBUG:boto:Token: None 2024-04-16T20:26:21.322 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.323 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548260cfa53b87c2-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.366 DEBUG:boto:b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.367 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.367 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.367 DEBUG:boto:path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.367 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.367 DEBUG:boto:Method: GET 2024-04-16T20:26:21.367 DEBUG:boto:Path: /yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.367 DEBUG:boto:Data: 2024-04-16T20:26:21.367 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.367 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.367 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.368 DEBUG:boto:Params: {} 2024-04-16T20:26:21.368 DEBUG:boto:Token: None 2024-04-16T20:26:21.368 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.368 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf477c3a1f70cc1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.370 DEBUG:boto:b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.371 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.371 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.371 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.371 DEBUG:boto:Path: /yrjefh-57/obj22 2024-04-16T20:26:21.371 DEBUG:boto:Data: 2024-04-16T20:26:21.371 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.371 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.371 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.371 DEBUG:boto:Params: {} 2024-04-16T20:26:21.371 DEBUG:boto:Token: None 2024-04-16T20:26:21.372 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22 2024-04-16T20:26:21.372 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4= 2024-04-16T20:26:21.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4=', 'Content-Length': '0'} 2024-04-16T20:26:21.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd3510c75929b3f-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.374 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.374 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.374 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.374 DEBUG:boto:Path: /yrjefh-57/obj22 2024-04-16T20:26:21.374 DEBUG:boto:Data: 2024-04-16T20:26:21.374 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.374 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.374 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.375 DEBUG:boto:Params: {} 2024-04-16T20:26:21.375 DEBUG:boto:Token: None 2024-04-16T20:26:21.375 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22 2024-04-16T20:26:21.375 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4= 2024-04-16T20:26:21.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4=', 'Content-Length': '0'} 2024-04-16T20:26:21.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006104b598744af1b1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-16T20:26:21.377 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.377 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.377 DEBUG:boto:Method: GET 2024-04-16T20:26:21.377 DEBUG:boto:Path: /yrjefh-57/obj22 2024-04-16T20:26:21.377 DEBUG:boto:Data: 2024-04-16T20:26:21.377 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.377 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.377 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.378 DEBUG:boto:Params: {} 2024-04-16T20:26:21.378 DEBUG:boto:Token: None 2024-04-16T20:26:21.378 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22 2024-04-16T20:26:21.378 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk= 2024-04-16T20:26:21.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk=', 'Content-Length': '0'} 2024-04-16T20:26:21.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e300ce2107d6aa7-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.422 DEBUG:boto:path=/yrjefh-57/obj22 2024-04-16T20:26:21.422 DEBUG:boto:auth_path=/yrjefh-57/obj22 2024-04-16T20:26:21.423 DEBUG:boto:Method: GET 2024-04-16T20:26:21.423 DEBUG:boto:Path: /yrjefh-57/obj22 2024-04-16T20:26:21.423 DEBUG:boto:Data: 2024-04-16T20:26:21.423 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.423 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.423 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.423 DEBUG:boto:Params: {} 2024-04-16T20:26:21.423 DEBUG:boto:Token: None 2024-04-16T20:26:21.423 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj22 2024-04-16T20:26:21.423 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk= 2024-04-16T20:26:21.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk=', 'Content-Length': '0'} 2024-04-16T20:26:21.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dfbbed4dda5a82-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.425 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.425 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.426 DEBUG:boto:path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.426 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.426 DEBUG:boto:Method: GET 2024-04-16T20:26:21.426 DEBUG:boto:Path: /yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.426 DEBUG:boto:Data: 2024-04-16T20:26:21.426 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.426 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.426 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.426 DEBUG:boto:Params: {} 2024-04-16T20:26:21.426 DEBUG:boto:Token: None 2024-04-16T20:26:21.426 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.427 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7febe438b035a86-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.429 DEBUG:boto:b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.430 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.430 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.430 DEBUG:boto:path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.430 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.430 DEBUG:boto:Method: GET 2024-04-16T20:26:21.430 DEBUG:boto:Path: /yrjefh-57/?prefix=obj22 2024-04-16T20:26:21.430 DEBUG:boto:Data: 2024-04-16T20:26:21.430 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.430 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.430 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.431 DEBUG:boto:Params: {} 2024-04-16T20:26:21.431 DEBUG:boto:Token: None 2024-04-16T20:26:21.431 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.431 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dffc4ededb60d66-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.433 DEBUG:boto:b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:26:21.434 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.434 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.434 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.434 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.434 DEBUG:boto:Method: GET 2024-04-16T20:26:21.434 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.434 DEBUG:boto:Data: 2024-04-16T20:26:21.434 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.434 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.434 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.435 DEBUG:boto:Params: {} 2024-04-16T20:26:21.435 DEBUG:boto:Token: None 2024-04-16T20:26:21.435 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.435 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T20:26:21.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T20:26:21.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3c644d44cb252a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.478 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.479 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.479 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.479 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.479 DEBUG:boto:Method: GET 2024-04-16T20:26:21.479 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.479 DEBUG:boto:Data: 2024-04-16T20:26:21.479 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.479 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.479 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.479 DEBUG:boto:Params: {} 2024-04-16T20:26:21.479 DEBUG:boto:Token: None 2024-04-16T20:26:21.479 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.480 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T20:26:21.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T20:26:21.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c5e58e6dada858-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.482 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.482 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.482 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.482 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.482 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.482 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.482 DEBUG:boto:Data: 2024-04-16T20:26:21.482 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.482 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.482 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.483 DEBUG:boto:Params: {} 2024-04-16T20:26:21.483 DEBUG:boto:Token: None 2024-04-16T20:26:21.483 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.483 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU= 2024-04-16T20:26:21.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU=', 'Content-Length': '0'} 2024-04-16T20:26:21.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a63976b374634b4-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.485 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.485 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.485 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.486 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.486 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.486 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.486 DEBUG:boto:Data: 2024-04-16T20:26:21.486 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.486 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.486 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.486 DEBUG:boto:Params: {} 2024-04-16T20:26:21.486 DEBUG:boto:Token: None 2024-04-16T20:26:21.486 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.486 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU= 2024-04-16T20:26:21.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU=', 'Content-Length': '0'} 2024-04-16T20:26:21.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c6ec7cee87bacc-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:26:21.489 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.489 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.489 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.489 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.489 DEBUG:boto:Method: GET 2024-04-16T20:26:21.489 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.489 DEBUG:boto:Data: 2024-04-16T20:26:21.489 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.489 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.489 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.489 DEBUG:boto:Params: {} 2024-04-16T20:26:21.490 DEBUG:boto:Token: None 2024-04-16T20:26:21.490 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.490 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T20:26:21.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T20:26:21.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb943c99141cabb-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.534 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.534 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.535 DEBUG:boto:path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.535 DEBUG:boto:auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.535 DEBUG:boto:Method: GET 2024-04-16T20:26:21.535 DEBUG:boto:Path: /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.535 DEBUG:boto:Data: 2024-04-16T20:26:21.535 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.535 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.535 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.535 DEBUG:boto:Params: {} 2024-04-16T20:26:21.535 DEBUG:boto:Token: None 2024-04-16T20:26:21.535 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33?versionId=null 2024-04-16T20:26:21.536 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T20:26:21.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T20:26:21.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000052142ae00bc9735-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.538 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.538 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.538 DEBUG:boto:path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.538 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.538 DEBUG:boto:Method: GET 2024-04-16T20:26:21.538 DEBUG:boto:Path: /yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.538 DEBUG:boto:Data: 2024-04-16T20:26:21.538 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.538 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.539 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.539 DEBUG:boto:Params: {} 2024-04-16T20:26:21.539 DEBUG:boto:Token: None 2024-04-16T20:26:21.539 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094d5acd75d38a100-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.542 DEBUG:boto:b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.542 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.542 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.542 DEBUG:boto:path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.543 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.543 DEBUG:boto:Method: GET 2024-04-16T20:26:21.543 DEBUG:boto:Path: /yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.543 DEBUG:boto:Data: 2024-04-16T20:26:21.543 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.543 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.543 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.543 DEBUG:boto:Params: {} 2024-04-16T20:26:21.543 DEBUG:boto:Token: None 2024-04-16T20:26:21.543 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008919ed7a6606ad15-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.546 DEBUG:boto:b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.546 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.546 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.547 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.547 DEBUG:boto:Path: /yrjefh-57/obj33 2024-04-16T20:26:21.547 DEBUG:boto:Data: 2024-04-16T20:26:21.547 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.547 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.547 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.547 DEBUG:boto:Params: {} 2024-04-16T20:26:21.547 DEBUG:boto:Token: None 2024-04-16T20:26:21.547 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33 2024-04-16T20:26:21.547 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk= 2024-04-16T20:26:21.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk=', 'Content-Length': '0'} 2024-04-16T20:26:21.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15bd151c0ab4169-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.549 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.550 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.550 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.550 DEBUG:boto:Path: /yrjefh-57/obj33 2024-04-16T20:26:21.550 DEBUG:boto:Data: 2024-04-16T20:26:21.550 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.550 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.550 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.550 DEBUG:boto:Params: {} 2024-04-16T20:26:21.550 DEBUG:boto:Token: None 2024-04-16T20:26:21.550 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33 2024-04-16T20:26:21.551 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk= 2024-04-16T20:26:21.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk=', 'Content-Length': '0'} 2024-04-16T20:26:21.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f7a0cc70dd6862-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-16T20:26:21.553 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.553 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.553 DEBUG:boto:Method: GET 2024-04-16T20:26:21.553 DEBUG:boto:Path: /yrjefh-57/obj33 2024-04-16T20:26:21.553 DEBUG:boto:Data: 2024-04-16T20:26:21.553 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.554 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.554 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.554 DEBUG:boto:Params: {} 2024-04-16T20:26:21.554 DEBUG:boto:Token: None 2024-04-16T20:26:21.554 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33 2024-04-16T20:26:21.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo= 2024-04-16T20:26:21.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo=', 'Content-Length': '0'} 2024-04-16T20:26:21.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8067133bdf333c-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.598 DEBUG:boto:path=/yrjefh-57/obj33 2024-04-16T20:26:21.598 DEBUG:boto:auth_path=/yrjefh-57/obj33 2024-04-16T20:26:21.598 DEBUG:boto:Method: GET 2024-04-16T20:26:21.599 DEBUG:boto:Path: /yrjefh-57/obj33 2024-04-16T20:26:21.599 DEBUG:boto:Data: 2024-04-16T20:26:21.599 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.599 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.599 DEBUG:boto:Params: {} 2024-04-16T20:26:21.599 DEBUG:boto:Token: None 2024-04-16T20:26:21.599 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj33 2024-04-16T20:26:21.599 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo= 2024-04-16T20:26:21.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo=', 'Content-Length': '0'} 2024-04-16T20:26:21.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68ca5b35d75dae3-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.602 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.602 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.602 DEBUG:boto:path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.602 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.602 DEBUG:boto:Method: GET 2024-04-16T20:26:21.602 DEBUG:boto:Path: /yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.602 DEBUG:boto:Data: 2024-04-16T20:26:21.602 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.603 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.603 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.603 DEBUG:boto:Params: {} 2024-04-16T20:26:21.603 DEBUG:boto:Token: None 2024-04-16T20:26:21.603 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.603 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405bf2a227be5170-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.606 DEBUG:boto:b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.606 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.606 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.606 DEBUG:boto:path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.606 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.606 DEBUG:boto:Method: GET 2024-04-16T20:26:21.606 DEBUG:boto:Path: /yrjefh-57/?prefix=obj33 2024-04-16T20:26:21.606 DEBUG:boto:Data: 2024-04-16T20:26:21.607 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.607 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.607 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.607 DEBUG:boto:Params: {} 2024-04-16T20:26:21.607 DEBUG:boto:Token: None 2024-04-16T20:26:21.607 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.607 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054cd28f5ff4516b0-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.610 DEBUG:boto:b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:26:21.610 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.610 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.610 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.610 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.611 DEBUG:boto:Method: GET 2024-04-16T20:26:21.611 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.611 DEBUG:boto:Data: 2024-04-16T20:26:21.611 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.611 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.611 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.611 DEBUG:boto:Params: {} 2024-04-16T20:26:21.611 DEBUG:boto:Token: None 2024-04-16T20:26:21.611 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.611 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T20:26:21.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T20:26:21.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be712cc2f6a189e6-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.654 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.654 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.655 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.655 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.655 DEBUG:boto:Method: GET 2024-04-16T20:26:21.655 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.655 DEBUG:boto:Data: 2024-04-16T20:26:21.655 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.655 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.655 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.655 DEBUG:boto:Params: {} 2024-04-16T20:26:21.655 DEBUG:boto:Token: None 2024-04-16T20:26:21.655 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.656 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T20:26:21.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T20:26:21.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e69fc5966c61566-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.658 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.658 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.658 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.658 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.658 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.658 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.658 DEBUG:boto:Data: 2024-04-16T20:26:21.658 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.658 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.658 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.658 DEBUG:boto:Params: {} 2024-04-16T20:26:21.659 DEBUG:boto:Token: None 2024-04-16T20:26:21.659 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.659 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0= 2024-04-16T20:26:21.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0=', 'Content-Length': '0'} 2024-04-16T20:26:21.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072d36605de524dc2-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.661 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.661 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.661 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.662 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.662 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.662 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.662 DEBUG:boto:Data: 2024-04-16T20:26:21.662 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.662 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.662 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.662 DEBUG:boto:Params: {} 2024-04-16T20:26:21.662 DEBUG:boto:Token: None 2024-04-16T20:26:21.662 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.663 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0= 2024-04-16T20:26:21.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0=', 'Content-Length': '0'} 2024-04-16T20:26:21.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7220cc15c7e77b-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:26:21.665 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.665 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.665 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.665 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.665 DEBUG:boto:Method: GET 2024-04-16T20:26:21.665 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.665 DEBUG:boto:Data: 2024-04-16T20:26:21.665 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.665 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.665 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.666 DEBUG:boto:Params: {} 2024-04-16T20:26:21.666 DEBUG:boto:Token: None 2024-04-16T20:26:21.666 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.666 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T20:26:21.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T20:26:21.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2033c207cb7bcdc-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.710 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.710 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.711 DEBUG:boto:path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.711 DEBUG:boto:auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.711 DEBUG:boto:Method: GET 2024-04-16T20:26:21.711 DEBUG:boto:Path: /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.711 DEBUG:boto:Data: 2024-04-16T20:26:21.711 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.711 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.711 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.711 DEBUG:boto:Params: {} 2024-04-16T20:26:21.711 DEBUG:boto:Token: None 2024-04-16T20:26:21.711 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44?versionId=null 2024-04-16T20:26:21.712 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T20:26:21.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T20:26:21.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d097c8a3fa3d09-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.714 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.714 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.714 DEBUG:boto:path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.714 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.714 DEBUG:boto:Method: GET 2024-04-16T20:26:21.714 DEBUG:boto:Path: /yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.714 DEBUG:boto:Data: 2024-04-16T20:26:21.714 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.715 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.715 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.715 DEBUG:boto:Params: {} 2024-04-16T20:26:21.715 DEBUG:boto:Token: None 2024-04-16T20:26:21.715 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.715 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef485c8d324aaf55-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.718 DEBUG:boto:b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.718 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.718 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.718 DEBUG:boto:path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.718 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.719 DEBUG:boto:Method: GET 2024-04-16T20:26:21.719 DEBUG:boto:Path: /yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.719 DEBUG:boto:Data: 2024-04-16T20:26:21.719 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.719 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.719 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.719 DEBUG:boto:Params: {} 2024-04-16T20:26:21.719 DEBUG:boto:Token: None 2024-04-16T20:26:21.719 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.719 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9bf7dee2ecf44c1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.722 DEBUG:boto:b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.722 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.722 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.722 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.722 DEBUG:boto:Path: /yrjefh-57/obj44 2024-04-16T20:26:21.722 DEBUG:boto:Data: 2024-04-16T20:26:21.723 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.723 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.723 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.723 DEBUG:boto:Params: {} 2024-04-16T20:26:21.723 DEBUG:boto:Token: None 2024-04-16T20:26:21.723 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44 2024-04-16T20:26:21.723 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ= 2024-04-16T20:26:21.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ=', 'Content-Length': '0'} 2024-04-16T20:26:21.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448efe797d8f088a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.726 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.726 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.726 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.726 DEBUG:boto:Path: /yrjefh-57/obj44 2024-04-16T20:26:21.726 DEBUG:boto:Data: 2024-04-16T20:26:21.726 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.726 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.726 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.726 DEBUG:boto:Params: {} 2024-04-16T20:26:21.727 DEBUG:boto:Token: None 2024-04-16T20:26:21.727 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44 2024-04-16T20:26:21.727 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ= 2024-04-16T20:26:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ=', 'Content-Length': '0'} 2024-04-16T20:26:21.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4fbf796208a09c7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-16T20:26:21.729 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.729 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.729 DEBUG:boto:Method: GET 2024-04-16T20:26:21.729 DEBUG:boto:Path: /yrjefh-57/obj44 2024-04-16T20:26:21.729 DEBUG:boto:Data: 2024-04-16T20:26:21.729 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.729 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.729 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.729 DEBUG:boto:Params: {} 2024-04-16T20:26:21.729 DEBUG:boto:Token: None 2024-04-16T20:26:21.730 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44 2024-04-16T20:26:21.730 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI= 2024-04-16T20:26:21.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI=', 'Content-Length': '0'} 2024-04-16T20:26:21.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046836c53226d17ea-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.774 DEBUG:boto:path=/yrjefh-57/obj44 2024-04-16T20:26:21.774 DEBUG:boto:auth_path=/yrjefh-57/obj44 2024-04-16T20:26:21.774 DEBUG:boto:Method: GET 2024-04-16T20:26:21.775 DEBUG:boto:Path: /yrjefh-57/obj44 2024-04-16T20:26:21.775 DEBUG:boto:Data: 2024-04-16T20:26:21.775 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.775 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.775 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.775 DEBUG:boto:Params: {} 2024-04-16T20:26:21.775 DEBUG:boto:Token: None 2024-04-16T20:26:21.775 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj44 2024-04-16T20:26:21.775 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI= 2024-04-16T20:26:21.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI=', 'Content-Length': '0'} 2024-04-16T20:26:21.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1f962f6afe75f9-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.777 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.777 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.778 DEBUG:boto:path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.778 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.778 DEBUG:boto:Method: GET 2024-04-16T20:26:21.778 DEBUG:boto:Path: /yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.778 DEBUG:boto:Data: 2024-04-16T20:26:21.778 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.778 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.778 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.778 DEBUG:boto:Params: {} 2024-04-16T20:26:21.778 DEBUG:boto:Token: None 2024-04-16T20:26:21.778 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.779 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f298c8254e98d651-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.781 DEBUG:boto:b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.782 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.782 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.782 DEBUG:boto:path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.782 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.782 DEBUG:boto:Method: GET 2024-04-16T20:26:21.782 DEBUG:boto:Path: /yrjefh-57/?prefix=obj44 2024-04-16T20:26:21.782 DEBUG:boto:Data: 2024-04-16T20:26:21.782 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.782 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.783 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.783 DEBUG:boto:Params: {} 2024-04-16T20:26:21.783 DEBUG:boto:Token: None 2024-04-16T20:26:21.783 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.783 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325d202e36638e40-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.785 DEBUG:boto:b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:26:21.786 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.786 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.786 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.786 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.786 DEBUG:boto:Method: GET 2024-04-16T20:26:21.786 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.786 DEBUG:boto:Data: 2024-04-16T20:26:21.786 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.786 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.787 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.787 DEBUG:boto:Params: {} 2024-04-16T20:26:21.787 DEBUG:boto:Token: None 2024-04-16T20:26:21.787 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.787 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T20:26:21.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T20:26:21.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004211c66231aad5e-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.830 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.830 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.831 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.831 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.831 DEBUG:boto:Method: GET 2024-04-16T20:26:21.831 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.831 DEBUG:boto:Data: 2024-04-16T20:26:21.831 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.831 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.831 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.831 DEBUG:boto:Params: {} 2024-04-16T20:26:21.831 DEBUG:boto:Token: None 2024-04-16T20:26:21.832 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.832 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T20:26:21.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T20:26:21.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003336d8195e703bb3-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.834 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.834 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.834 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.834 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.834 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.834 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.834 DEBUG:boto:Data: 2024-04-16T20:26:21.834 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.834 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.834 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.835 DEBUG:boto:Params: {} 2024-04-16T20:26:21.835 DEBUG:boto:Token: None 2024-04-16T20:26:21.835 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.835 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo= 2024-04-16T20:26:21.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo=', 'Content-Length': '0'} 2024-04-16T20:26:21.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0c0a9a5aea7a54-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.837 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.837 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.837 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.837 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.837 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.837 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.838 DEBUG:boto:Data: 2024-04-16T20:26:21.838 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.838 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.838 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.838 DEBUG:boto:Params: {} 2024-04-16T20:26:21.838 DEBUG:boto:Token: None 2024-04-16T20:26:21.838 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.838 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo= 2024-04-16T20:26:21.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo=', 'Content-Length': '0'} 2024-04-16T20:26:21.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e6a54aa944b045-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:26:21.840 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.840 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.841 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.841 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.841 DEBUG:boto:Method: GET 2024-04-16T20:26:21.841 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.841 DEBUG:boto:Data: 2024-04-16T20:26:21.841 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.841 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.841 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.841 DEBUG:boto:Params: {} 2024-04-16T20:26:21.841 DEBUG:boto:Token: None 2024-04-16T20:26:21.842 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.842 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T20:26:21.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T20:26:21.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a013baccde4d55ff-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.886 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.887 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.887 DEBUG:boto:path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.887 DEBUG:boto:auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.887 DEBUG:boto:Method: GET 2024-04-16T20:26:21.887 DEBUG:boto:Path: /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.887 DEBUG:boto:Data: 2024-04-16T20:26:21.887 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.887 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.887 DEBUG:boto:Params: {} 2024-04-16T20:26:21.887 DEBUG:boto:Token: None 2024-04-16T20:26:21.888 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55?versionId=null 2024-04-16T20:26:21.888 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T20:26:21.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T20:26:21.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16cd78ebbc7e3a4-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.890 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.890 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.890 DEBUG:boto:path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.890 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.890 DEBUG:boto:Method: GET 2024-04-16T20:26:21.890 DEBUG:boto:Path: /yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.891 DEBUG:boto:Data: 2024-04-16T20:26:21.891 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.891 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.891 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.891 DEBUG:boto:Params: {} 2024-04-16T20:26:21.891 DEBUG:boto:Token: None 2024-04-16T20:26:21.891 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001457c8454cdbcb1b-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.894 DEBUG:boto:b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.894 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.895 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.895 DEBUG:boto:path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.895 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.895 DEBUG:boto:Method: GET 2024-04-16T20:26:21.895 DEBUG:boto:Path: /yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.895 DEBUG:boto:Data: 2024-04-16T20:26:21.895 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.895 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.895 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.895 DEBUG:boto:Params: {} 2024-04-16T20:26:21.895 DEBUG:boto:Token: None 2024-04-16T20:26:21.896 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.896 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000427096d0d1b99483-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.898 DEBUG:boto:b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.898 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.898 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.898 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.899 DEBUG:boto:Path: /yrjefh-57/obj55 2024-04-16T20:26:21.899 DEBUG:boto:Data: 2024-04-16T20:26:21.899 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.899 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.899 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.899 DEBUG:boto:Params: {} 2024-04-16T20:26:21.899 DEBUG:boto:Token: None 2024-04-16T20:26:21.899 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55 2024-04-16T20:26:21.899 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA= 2024-04-16T20:26:21.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA=', 'Content-Length': '0'} 2024-04-16T20:26:21.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939df91402219ebd-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.901 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.901 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.901 DEBUG:boto:Method: HEAD 2024-04-16T20:26:21.902 DEBUG:boto:Path: /yrjefh-57/obj55 2024-04-16T20:26:21.902 DEBUG:boto:Data: 2024-04-16T20:26:21.902 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.902 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.902 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.902 DEBUG:boto:Params: {} 2024-04-16T20:26:21.902 DEBUG:boto:Token: None 2024-04-16T20:26:21.902 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55 2024-04-16T20:26:21.902 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA= 2024-04-16T20:26:21.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA=', 'Content-Length': '0'} 2024-04-16T20:26:21.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00e6c36458ef3a0-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-16T20:26:21.904 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.904 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.904 DEBUG:boto:Method: GET 2024-04-16T20:26:21.905 DEBUG:boto:Path: /yrjefh-57/obj55 2024-04-16T20:26:21.905 DEBUG:boto:Data: 2024-04-16T20:26:21.905 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.905 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.905 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.905 DEBUG:boto:Params: {} 2024-04-16T20:26:21.905 DEBUG:boto:Token: None 2024-04-16T20:26:21.905 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55 2024-04-16T20:26:21.905 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs= 2024-04-16T20:26:21.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs=', 'Content-Length': '0'} 2024-04-16T20:26:21.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088398476cf6b4506-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.950 DEBUG:boto:path=/yrjefh-57/obj55 2024-04-16T20:26:21.950 DEBUG:boto:auth_path=/yrjefh-57/obj55 2024-04-16T20:26:21.951 DEBUG:boto:Method: GET 2024-04-16T20:26:21.951 DEBUG:boto:Path: /yrjefh-57/obj55 2024-04-16T20:26:21.951 DEBUG:boto:Data: 2024-04-16T20:26:21.951 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.951 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.951 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.951 DEBUG:boto:Params: {} 2024-04-16T20:26:21.951 DEBUG:boto:Token: None 2024-04-16T20:26:21.951 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/obj55 2024-04-16T20:26:21.951 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs= 2024-04-16T20:26:21.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs=', 'Content-Length': '0'} 2024-04-16T20:26:21.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923fbcc9e40a51f2-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.953 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.954 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.954 DEBUG:boto:path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.954 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.954 DEBUG:boto:Method: GET 2024-04-16T20:26:21.954 DEBUG:boto:Path: /yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.954 DEBUG:boto:Data: 2024-04-16T20:26:21.954 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.954 DEBUG:boto:Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:21.954 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.954 DEBUG:boto:Params: {} 2024-04-16T20:26:21.955 DEBUG:boto:Token: None 2024-04-16T20:26:21.955 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.955 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001940983b5a92cf40-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.957 DEBUG:boto:b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.958 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:21.958 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:21.958 DEBUG:boto:path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.958 DEBUG:boto:auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.958 DEBUG:boto:Method: GET 2024-04-16T20:26:21.958 DEBUG:boto:Path: /yrjefh-57/?prefix=obj55 2024-04-16T20:26:21.958 DEBUG:boto:Data: 2024-04-16T20:26:21.959 DEBUG:boto:Headers: {} 2024-04-16T20:26:21.959 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:21.959 DEBUG:boto:Port: 8000 2024-04-16T20:26:21.959 DEBUG:boto:Params: {} 2024-04-16T20:26:21.959 DEBUG:boto:Token: None 2024-04-16T20:26:21.959 DEBUG:boto:StringToSign: GET Tue, 16 Apr 2024 20:26:21 GMT /yrjefh-57/ 2024-04-16T20:26:21.959 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T20:26:21.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T20:26:21.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa664e78bbca7a6-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:21.961 DEBUG:boto:b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T20:26:21.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=yrjefh-57 zones={test-zone2, test-zone1} 2024-04-16T20:26:21.962 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T20:26:21.962 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T20:26:21.963 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:26:21.974+0000 7f87e17cc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 212061) UID: 0 2024-04-16T20:26:21.963 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:26:21.974+0000 7f87e5025880 -1 shutting down 2024-04-16T20:26:24.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:26:28.062 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T20:26:28.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.063 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:28.124 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:28.124 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:28.195 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:28.195 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:28.200 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 5 2024-04-16T20:26:28.250 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.250 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.251 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.251 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.251 DEBUG:boto:Data: 2024-04-16T20:26:28.251 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.251 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.251 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.251 DEBUG:boto:Params: {} 2024-04-16T20:26:28.251 DEBUG:boto:Token: None 2024-04-16T20:26:28.251 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.252 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f273c76154825183-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.255 DEBUG:boto:path=/yrjefh-57/obj17 2024-04-16T20:26:28.255 DEBUG:boto:auth_path=/yrjefh-57/obj17 2024-04-16T20:26:28.255 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.255 DEBUG:boto:Path: /yrjefh-57/obj17 2024-04-16T20:26:28.255 DEBUG:boto:Data: 2024-04-16T20:26:28.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.255 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.255 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.255 DEBUG:boto:Params: {} 2024-04-16T20:26:28.255 DEBUG:boto:Token: None 2024-04-16T20:26:28.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj17 2024-04-16T20:26:28.256 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:kuxS56hZ5WlpX3rsfFmhwlFSxqg= 2024-04-16T20:26:28.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kuxS56hZ5WlpX3rsfFmhwlFSxqg='} 2024-04-16T20:26:28.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de5457606d1b17ef-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.261 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.262 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.262 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.262 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.262 DEBUG:boto:Data: 2024-04-16T20:26:28.262 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.262 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.262 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.262 DEBUG:boto:Params: {} 2024-04-16T20:26:28.262 DEBUG:boto:Token: None 2024-04-16T20:26:28.263 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.263 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055176342908f1699-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.265 DEBUG:boto:path=/yrjefh-57/obj34 2024-04-16T20:26:28.265 DEBUG:boto:auth_path=/yrjefh-57/obj34 2024-04-16T20:26:28.265 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.265 DEBUG:boto:Path: /yrjefh-57/obj34 2024-04-16T20:26:28.265 DEBUG:boto:Data: 2024-04-16T20:26:28.265 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.265 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.266 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.266 DEBUG:boto:Params: {} 2024-04-16T20:26:28.266 DEBUG:boto:Token: None 2024-04-16T20:26:28.266 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj34 2024-04-16T20:26:28.266 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hQ688Io42sSODN3ItDJ6QfWoi+g= 2024-04-16T20:26:28.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hQ688Io42sSODN3ItDJ6QfWoi+g='} 2024-04-16T20:26:28.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8e2e44d2c6cc3cd-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.272 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.272 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.272 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.272 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.272 DEBUG:boto:Data: 2024-04-16T20:26:28.272 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.272 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.272 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.272 DEBUG:boto:Params: {} 2024-04-16T20:26:28.272 DEBUG:boto:Token: None 2024-04-16T20:26:28.273 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.273 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef14ffe1db0664b3-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.275 DEBUG:boto:path=/yrjefh-57/obj51 2024-04-16T20:26:28.275 DEBUG:boto:auth_path=/yrjefh-57/obj51 2024-04-16T20:26:28.275 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.275 DEBUG:boto:Path: /yrjefh-57/obj51 2024-04-16T20:26:28.275 DEBUG:boto:Data: 2024-04-16T20:26:28.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.276 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.276 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.276 DEBUG:boto:Params: {} 2024-04-16T20:26:28.276 DEBUG:boto:Token: None 2024-04-16T20:26:28.276 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj51 2024-04-16T20:26:28.276 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jrPfF4PvBgvYn6tQS+4jAnX7AkY= 2024-04-16T20:26:28.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jrPfF4PvBgvYn6tQS+4jAnX7AkY='} 2024-04-16T20:26:28.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf0aa2102a041b6b-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.323 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.323 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.323 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.323 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.323 DEBUG:boto:Data: 2024-04-16T20:26:28.323 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.323 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.323 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.323 DEBUG:boto:Params: {} 2024-04-16T20:26:28.323 DEBUG:boto:Token: None 2024-04-16T20:26:28.323 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.324 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cb34c568be6c0c7-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.326 DEBUG:boto:path=/yrjefh-57/obj68 2024-04-16T20:26:28.326 DEBUG:boto:auth_path=/yrjefh-57/obj68 2024-04-16T20:26:28.326 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.327 DEBUG:boto:Path: /yrjefh-57/obj68 2024-04-16T20:26:28.327 DEBUG:boto:Data: 2024-04-16T20:26:28.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.327 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.327 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.327 DEBUG:boto:Params: {} 2024-04-16T20:26:28.327 DEBUG:boto:Token: None 2024-04-16T20:26:28.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj68 2024-04-16T20:26:28.327 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:xt3Ly3tRF11zHykV4jZCNyWBPuc= 2024-04-16T20:26:28.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xt3Ly3tRF11zHykV4jZCNyWBPuc='} 2024-04-16T20:26:28.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a2c7b94741964fb-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.333 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.333 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.333 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.333 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.333 DEBUG:boto:Data: 2024-04-16T20:26:28.333 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.333 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.333 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.334 DEBUG:boto:Params: {} 2024-04-16T20:26:28.334 DEBUG:boto:Token: None 2024-04-16T20:26:28.334 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.334 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7bf83d21dd460f6-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.336 DEBUG:boto:path=/yrjefh-57/obj85 2024-04-16T20:26:28.336 DEBUG:boto:auth_path=/yrjefh-57/obj85 2024-04-16T20:26:28.336 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.337 DEBUG:boto:Path: /yrjefh-57/obj85 2024-04-16T20:26:28.337 DEBUG:boto:Data: 2024-04-16T20:26:28.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.337 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.337 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.337 DEBUG:boto:Params: {} 2024-04-16T20:26:28.337 DEBUG:boto:Token: None 2024-04-16T20:26:28.337 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj85 2024-04-16T20:26:28.337 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2BCvO5Xr4RjiN+F8xmPUAxxJAr0= 2024-04-16T20:26:28.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BCvO5Xr4RjiN+F8xmPUAxxJAr0='} 2024-04-16T20:26:28.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070d073d147876b05-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.343 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:28.395 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:28.457 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:28.457 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:28.464 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 10 2024-04-16T20:26:28.519 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.519 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.519 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.519 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.520 DEBUG:boto:Data: 2024-04-16T20:26:28.520 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.520 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.520 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.520 DEBUG:boto:Params: {} 2024-04-16T20:26:28.520 DEBUG:boto:Token: None 2024-04-16T20:26:28.520 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.520 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eddd8c1720c0dbe-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.523 DEBUG:boto:path=/yrjefh-57/obj19 2024-04-16T20:26:28.523 DEBUG:boto:auth_path=/yrjefh-57/obj19 2024-04-16T20:26:28.523 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.523 DEBUG:boto:Path: /yrjefh-57/obj19 2024-04-16T20:26:28.524 DEBUG:boto:Data: 2024-04-16T20:26:28.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.524 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.524 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.524 DEBUG:boto:Params: {} 2024-04-16T20:26:28.524 DEBUG:boto:Token: None 2024-04-16T20:26:28.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj19 2024-04-16T20:26:28.524 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:FwRLPblWVZ8nl9TQ4QDf3AERLic= 2024-04-16T20:26:28.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwRLPblWVZ8nl9TQ4QDf3AERLic='} 2024-04-16T20:26:28.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002878716f54dd43e4-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.567 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.567 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.567 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.567 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.567 DEBUG:boto:Data: 2024-04-16T20:26:28.567 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.567 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.567 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.567 DEBUG:boto:Params: {} 2024-04-16T20:26:28.567 DEBUG:boto:Token: None 2024-04-16T20:26:28.567 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.568 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5328e389f66bc78-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.570 DEBUG:boto:path=/yrjefh-57/obj38 2024-04-16T20:26:28.571 DEBUG:boto:auth_path=/yrjefh-57/obj38 2024-04-16T20:26:28.571 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.571 DEBUG:boto:Path: /yrjefh-57/obj38 2024-04-16T20:26:28.571 DEBUG:boto:Data: 2024-04-16T20:26:28.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.571 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.571 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.571 DEBUG:boto:Params: {} 2024-04-16T20:26:28.571 DEBUG:boto:Token: None 2024-04-16T20:26:28.571 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj38 2024-04-16T20:26:28.571 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4H2pRWRceZRWwZklddO/G/W+vgk= 2024-04-16T20:26:28.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4H2pRWRceZRWwZklddO/G/W+vgk='} 2024-04-16T20:26:28.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f424e92576105be-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.575 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.576 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.576 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.576 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.576 DEBUG:boto:Data: 2024-04-16T20:26:28.576 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.576 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.576 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.576 DEBUG:boto:Params: {} 2024-04-16T20:26:28.576 DEBUG:boto:Token: None 2024-04-16T20:26:28.576 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.576 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002725d00faa66be6e-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.579 DEBUG:boto:path=/yrjefh-57/obj57 2024-04-16T20:26:28.579 DEBUG:boto:auth_path=/yrjefh-57/obj57 2024-04-16T20:26:28.579 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.579 DEBUG:boto:Path: /yrjefh-57/obj57 2024-04-16T20:26:28.579 DEBUG:boto:Data: 2024-04-16T20:26:28.579 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.580 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.580 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.580 DEBUG:boto:Params: {} 2024-04-16T20:26:28.580 DEBUG:boto:Token: None 2024-04-16T20:26:28.580 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj57 2024-04-16T20:26:28.580 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:zI9Hq57e0bmZXrS56BDiPhoGYxI= 2024-04-16T20:26:28.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zI9Hq57e0bmZXrS56BDiPhoGYxI='} 2024-04-16T20:26:28.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003711743659843709-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.623 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.623 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.623 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.623 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.623 DEBUG:boto:Data: 2024-04-16T20:26:28.623 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.623 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.623 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.623 DEBUG:boto:Params: {} 2024-04-16T20:26:28.624 DEBUG:boto:Token: None 2024-04-16T20:26:28.624 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.624 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d5402b8f7b792d-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.627 DEBUG:boto:path=/yrjefh-57/obj76 2024-04-16T20:26:28.627 DEBUG:boto:auth_path=/yrjefh-57/obj76 2024-04-16T20:26:28.627 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.627 DEBUG:boto:Path: /yrjefh-57/obj76 2024-04-16T20:26:28.627 DEBUG:boto:Data: 2024-04-16T20:26:28.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.627 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.627 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.627 DEBUG:boto:Params: {} 2024-04-16T20:26:28.627 DEBUG:boto:Token: None 2024-04-16T20:26:28.628 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj76 2024-04-16T20:26:28.628 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:CoBS5uUIhAbo/f4Y3GPofhwSShw= 2024-04-16T20:26:28.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CoBS5uUIhAbo/f4Y3GPofhwSShw='} 2024-04-16T20:26:28.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000693a4685bcf06744-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.633 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.633 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.633 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.633 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.633 DEBUG:boto:Data: 2024-04-16T20:26:28.633 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.633 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.633 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.633 DEBUG:boto:Params: {} 2024-04-16T20:26:28.633 DEBUG:boto:Token: None 2024-04-16T20:26:28.634 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.634 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e1ea901658ed9a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.636 DEBUG:boto:path=/yrjefh-57/obj95 2024-04-16T20:26:28.636 DEBUG:boto:auth_path=/yrjefh-57/obj95 2024-04-16T20:26:28.636 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.636 DEBUG:boto:Path: /yrjefh-57/obj95 2024-04-16T20:26:28.636 DEBUG:boto:Data: 2024-04-16T20:26:28.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.637 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.637 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.637 DEBUG:boto:Params: {} 2024-04-16T20:26:28.637 DEBUG:boto:Token: None 2024-04-16T20:26:28.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj95 2024-04-16T20:26:28.637 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:39LTyGOyBPsrblqJRy7I2xxalV8= 2024-04-16T20:26:28.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:39LTyGOyBPsrblqJRy7I2xxalV8='} 2024-04-16T20:26:28.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f086498fbe79bc7a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.687 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:28.738 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:28.738 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:28.807 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:28.807 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:28.815 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 15 2024-04-16T20:26:28.875 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.875 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.875 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.875 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.875 DEBUG:boto:Data: 2024-04-16T20:26:28.875 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.876 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.876 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.876 DEBUG:boto:Params: {} 2024-04-16T20:26:28.876 DEBUG:boto:Token: None 2024-04-16T20:26:28.876 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.876 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000153744cbea2e9e61-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.879 DEBUG:boto:path=/yrjefh-57/obj23 2024-04-16T20:26:28.879 DEBUG:boto:auth_path=/yrjefh-57/obj23 2024-04-16T20:26:28.879 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.879 DEBUG:boto:Path: /yrjefh-57/obj23 2024-04-16T20:26:28.879 DEBUG:boto:Data: 2024-04-16T20:26:28.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.880 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.880 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.880 DEBUG:boto:Params: {} 2024-04-16T20:26:28.880 DEBUG:boto:Token: None 2024-04-16T20:26:28.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj23 2024-04-16T20:26:28.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:vsoXkr5DTRQyCBbJYuUSOGvFKag= 2024-04-16T20:26:28.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsoXkr5DTRQyCBbJYuUSOGvFKag='} 2024-04-16T20:26:28.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052f99582b3a85aed-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.886 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.886 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.886 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.886 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.886 DEBUG:boto:Data: 2024-04-16T20:26:28.886 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.887 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.887 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.887 DEBUG:boto:Params: {} 2024-04-16T20:26:28.887 DEBUG:boto:Token: None 2024-04-16T20:26:28.887 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.887 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dbf747691f0ef90-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.889 DEBUG:boto:path=/yrjefh-57/obj46 2024-04-16T20:26:28.890 DEBUG:boto:auth_path=/yrjefh-57/obj46 2024-04-16T20:26:28.890 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.890 DEBUG:boto:Path: /yrjefh-57/obj46 2024-04-16T20:26:28.890 DEBUG:boto:Data: 2024-04-16T20:26:28.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.890 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.890 DEBUG:boto:Params: {} 2024-04-16T20:26:28.890 DEBUG:boto:Token: None 2024-04-16T20:26:28.890 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj46 2024-04-16T20:26:28.891 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:JZLJivsPtayKN/MVF9UNodBuxvA= 2024-04-16T20:26:28.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JZLJivsPtayKN/MVF9UNodBuxvA='} 2024-04-16T20:26:28.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050f8a5e0a4c32e0b-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.896 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.896 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.896 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.896 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.896 DEBUG:boto:Data: 2024-04-16T20:26:28.896 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.896 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.896 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.896 DEBUG:boto:Params: {} 2024-04-16T20:26:28.897 DEBUG:boto:Token: None 2024-04-16T20:26:28.897 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.897 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fc892a093038a43-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.900 DEBUG:boto:path=/yrjefh-57/obj69 2024-04-16T20:26:28.900 DEBUG:boto:auth_path=/yrjefh-57/obj69 2024-04-16T20:26:28.900 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.900 DEBUG:boto:Path: /yrjefh-57/obj69 2024-04-16T20:26:28.900 DEBUG:boto:Data: 2024-04-16T20:26:28.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.900 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.900 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.900 DEBUG:boto:Params: {} 2024-04-16T20:26:28.900 DEBUG:boto:Token: None 2024-04-16T20:26:28.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj69 2024-04-16T20:26:28.901 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jePztHeYSYQwfRaJS9KwhhZ4TPU= 2024-04-16T20:26:28.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jePztHeYSYQwfRaJS9KwhhZ4TPU='} 2024-04-16T20:26:28.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c78cd6c2409eb740-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.943 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.943 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.943 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.943 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.943 DEBUG:boto:Data: 2024-04-16T20:26:28.943 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.943 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.943 DEBUG:boto:Params: {} 2024-04-16T20:26:28.943 DEBUG:boto:Token: None 2024-04-16T20:26:28.944 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac6a337b2f0d9098-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.946 DEBUG:boto:path=/yrjefh-57/obj92 2024-04-16T20:26:28.946 DEBUG:boto:auth_path=/yrjefh-57/obj92 2024-04-16T20:26:28.946 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.946 DEBUG:boto:Path: /yrjefh-57/obj92 2024-04-16T20:26:28.946 DEBUG:boto:Data: 2024-04-16T20:26:28.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.947 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.947 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.947 DEBUG:boto:Params: {} 2024-04-16T20:26:28.947 DEBUG:boto:Token: None 2024-04-16T20:26:28.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj92 2024-04-16T20:26:28.947 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:4ok8V3HwSOCUc5y9dLRUlfgyuMw= 2024-04-16T20:26:28.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4ok8V3HwSOCUc5y9dLRUlfgyuMw='} 2024-04-16T20:26:28.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a72384affd2d2381-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.952 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:28.952 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:28.952 DEBUG:boto:Method: HEAD 2024-04-16T20:26:28.952 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:28.953 DEBUG:boto:Data: 2024-04-16T20:26:28.953 DEBUG:boto:Headers: {} 2024-04-16T20:26:28.953 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.953 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.953 DEBUG:boto:Params: {} 2024-04-16T20:26:28.953 DEBUG:boto:Token: None 2024-04-16T20:26:28.953 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/ 2024-04-16T20:26:28.953 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T20:26:28.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T20:26:28.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000293b541a47304b1a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.956 DEBUG:boto:path=/yrjefh-57/obj115 2024-04-16T20:26:28.956 DEBUG:boto:auth_path=/yrjefh-57/obj115 2024-04-16T20:26:28.956 DEBUG:boto:Method: PUT 2024-04-16T20:26:28.956 DEBUG:boto:Path: /yrjefh-57/obj115 2024-04-16T20:26:28.956 DEBUG:boto:Data: 2024-04-16T20:26:28.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:28.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:28.956 DEBUG:boto:Port: 8000 2024-04-16T20:26:28.956 DEBUG:boto:Params: {} 2024-04-16T20:26:28.957 DEBUG:boto:Token: None 2024-04-16T20:26:28.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:28 GMT /yrjefh-57/obj115 2024-04-16T20:26:28.957 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:6o6N8mWYtgqcyEUVOkukQ4i248o= 2024-04-16T20:26:28.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6o6N8mWYtgqcyEUVOkukQ4i248o='} 2024-04-16T20:26:28.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed6440b36c1a5406-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:28.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:28.999 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:29.051 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:29.052 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:29.116 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:29.116 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:29.126 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 20 2024-04-16T20:26:29.190 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.190 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.190 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.190 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.190 DEBUG:boto:Data: 2024-04-16T20:26:29.190 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.190 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.190 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.190 DEBUG:boto:Params: {} 2024-04-16T20:26:29.191 DEBUG:boto:Token: None 2024-04-16T20:26:29.191 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.191 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e25e4f1bb1d7b88-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.194 DEBUG:boto:path=/yrjefh-57/obj29 2024-04-16T20:26:29.194 DEBUG:boto:auth_path=/yrjefh-57/obj29 2024-04-16T20:26:29.195 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.195 DEBUG:boto:Path: /yrjefh-57/obj29 2024-04-16T20:26:29.195 DEBUG:boto:Data: 2024-04-16T20:26:29.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.195 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.195 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.195 DEBUG:boto:Params: {} 2024-04-16T20:26:29.195 DEBUG:boto:Token: None 2024-04-16T20:26:29.195 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj29 2024-04-16T20:26:29.195 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:S/25sxXNCmHWh5NVfbq50dtL7y4= 2024-04-16T20:26:29.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S/25sxXNCmHWh5NVfbq50dtL7y4='} 2024-04-16T20:26:29.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049ce41a9e13a4b71-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.201 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.201 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.201 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.202 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.202 DEBUG:boto:Data: 2024-04-16T20:26:29.202 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.202 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.202 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.202 DEBUG:boto:Params: {} 2024-04-16T20:26:29.202 DEBUG:boto:Token: None 2024-04-16T20:26:29.202 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.202 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcdbc9700fa0a186-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.205 DEBUG:boto:path=/yrjefh-57/obj58 2024-04-16T20:26:29.205 DEBUG:boto:auth_path=/yrjefh-57/obj58 2024-04-16T20:26:29.206 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.206 DEBUG:boto:Path: /yrjefh-57/obj58 2024-04-16T20:26:29.206 DEBUG:boto:Data: 2024-04-16T20:26:29.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.206 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.206 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.206 DEBUG:boto:Params: {} 2024-04-16T20:26:29.206 DEBUG:boto:Token: None 2024-04-16T20:26:29.206 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj58 2024-04-16T20:26:29.207 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:SZO1vAs6cwerzdeklfBVjRpFx8k= 2024-04-16T20:26:29.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SZO1vAs6cwerzdeklfBVjRpFx8k='} 2024-04-16T20:26:29.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6c8d4049a8e5f7a-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.212 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.212 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.212 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.212 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.212 DEBUG:boto:Data: 2024-04-16T20:26:29.212 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.212 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.212 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.212 DEBUG:boto:Params: {} 2024-04-16T20:26:29.213 DEBUG:boto:Token: None 2024-04-16T20:26:29.213 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.213 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faa89c14bc9fb051-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.216 DEBUG:boto:path=/yrjefh-57/obj87 2024-04-16T20:26:29.216 DEBUG:boto:auth_path=/yrjefh-57/obj87 2024-04-16T20:26:29.216 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.216 DEBUG:boto:Path: /yrjefh-57/obj87 2024-04-16T20:26:29.216 DEBUG:boto:Data: 2024-04-16T20:26:29.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.216 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.216 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.217 DEBUG:boto:Params: {} 2024-04-16T20:26:29.217 DEBUG:boto:Token: None 2024-04-16T20:26:29.217 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj87 2024-04-16T20:26:29.217 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:c4Zd7utH1IaziNof62fCBVUz2hw= 2024-04-16T20:26:29.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4Zd7utH1IaziNof62fCBVUz2hw='} 2024-04-16T20:26:29.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a91699b9715c52f-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.259 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.259 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.259 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.259 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.259 DEBUG:boto:Data: 2024-04-16T20:26:29.259 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.259 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.259 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.259 DEBUG:boto:Params: {} 2024-04-16T20:26:29.260 DEBUG:boto:Token: None 2024-04-16T20:26:29.260 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.260 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6bea43e812ad1ea-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.263 DEBUG:boto:path=/yrjefh-57/obj116 2024-04-16T20:26:29.263 DEBUG:boto:auth_path=/yrjefh-57/obj116 2024-04-16T20:26:29.263 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.263 DEBUG:boto:Path: /yrjefh-57/obj116 2024-04-16T20:26:29.263 DEBUG:boto:Data: 2024-04-16T20:26:29.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.263 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.263 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.264 DEBUG:boto:Params: {} 2024-04-16T20:26:29.264 DEBUG:boto:Token: None 2024-04-16T20:26:29.264 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj116 2024-04-16T20:26:29.264 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:KEcJVcPwPyS9R3I1x68RDyVyA/0= 2024-04-16T20:26:29.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KEcJVcPwPyS9R3I1x68RDyVyA/0='} 2024-04-16T20:26:29.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000947cf9ff60106635-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.269 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.270 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.270 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.270 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.270 DEBUG:boto:Data: 2024-04-16T20:26:29.270 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.270 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.270 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.270 DEBUG:boto:Params: {} 2024-04-16T20:26:29.270 DEBUG:boto:Token: None 2024-04-16T20:26:29.271 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.271 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005609d0be64a9316a-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.273 DEBUG:boto:path=/yrjefh-57/obj145 2024-04-16T20:26:29.274 DEBUG:boto:auth_path=/yrjefh-57/obj145 2024-04-16T20:26:29.274 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.274 DEBUG:boto:Path: /yrjefh-57/obj145 2024-04-16T20:26:29.274 DEBUG:boto:Data: 2024-04-16T20:26:29.274 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.274 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.274 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.274 DEBUG:boto:Params: {} 2024-04-16T20:26:29.274 DEBUG:boto:Token: None 2024-04-16T20:26:29.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj145 2024-04-16T20:26:29.275 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:jwRj9rj6E67j/e0t76uKlX6XOcs= 2024-04-16T20:26:29.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwRj9rj6E67j/e0t76uKlX6XOcs='} 2024-04-16T20:26:29.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dabdaeb54b3ed1d3-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.319 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:29.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:29.370 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:29.441 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:29.442 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:29.453 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 25 2024-04-16T20:26:29.526 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.526 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.526 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.526 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.526 DEBUG:boto:Data: 2024-04-16T20:26:29.526 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.527 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.527 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.527 DEBUG:boto:Params: {} 2024-04-16T20:26:29.527 DEBUG:boto:Token: None 2024-04-16T20:26:29.527 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.527 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bbbd752037b8163-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.531 DEBUG:boto:path=/yrjefh-57/obj31 2024-04-16T20:26:29.531 DEBUG:boto:auth_path=/yrjefh-57/obj31 2024-04-16T20:26:29.531 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.531 DEBUG:boto:Path: /yrjefh-57/obj31 2024-04-16T20:26:29.531 DEBUG:boto:Data: 2024-04-16T20:26:29.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.531 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.531 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.531 DEBUG:boto:Params: {} 2024-04-16T20:26:29.531 DEBUG:boto:Token: None 2024-04-16T20:26:29.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj31 2024-04-16T20:26:29.532 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IeTWoKMEe4p/LiSPOyim6SMizqY= 2024-04-16T20:26:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IeTWoKMEe4p/LiSPOyim6SMizqY='} 2024-04-16T20:26:29.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039bb035ae2c747d1-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.538 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.538 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.538 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.538 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.538 DEBUG:boto:Data: 2024-04-16T20:26:29.538 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.538 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.538 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.539 DEBUG:boto:Params: {} 2024-04-16T20:26:29.539 DEBUG:boto:Token: None 2024-04-16T20:26:29.539 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.539 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc774115c0ca84fd-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.542 DEBUG:boto:path=/yrjefh-57/obj62 2024-04-16T20:26:29.542 DEBUG:boto:auth_path=/yrjefh-57/obj62 2024-04-16T20:26:29.542 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.542 DEBUG:boto:Path: /yrjefh-57/obj62 2024-04-16T20:26:29.542 DEBUG:boto:Data: 2024-04-16T20:26:29.542 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.542 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.542 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.542 DEBUG:boto:Params: {} 2024-04-16T20:26:29.543 DEBUG:boto:Token: None 2024-04-16T20:26:29.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj62 2024-04-16T20:26:29.543 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:dC1cJCM9FjjM2wgJR1Exf0cLvAA= 2024-04-16T20:26:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dC1cJCM9FjjM2wgJR1Exf0cLvAA='} 2024-04-16T20:26:29.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b5b2ca690368686-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.548 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.548 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.549 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.549 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.549 DEBUG:boto:Data: 2024-04-16T20:26:29.549 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.549 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.549 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.549 DEBUG:boto:Params: {} 2024-04-16T20:26:29.549 DEBUG:boto:Token: None 2024-04-16T20:26:29.549 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.549 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbb4e74f54ab5935-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.552 DEBUG:boto:path=/yrjefh-57/obj93 2024-04-16T20:26:29.553 DEBUG:boto:auth_path=/yrjefh-57/obj93 2024-04-16T20:26:29.553 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.553 DEBUG:boto:Path: /yrjefh-57/obj93 2024-04-16T20:26:29.553 DEBUG:boto:Data: 2024-04-16T20:26:29.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.553 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.553 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.553 DEBUG:boto:Params: {} 2024-04-16T20:26:29.553 DEBUG:boto:Token: None 2024-04-16T20:26:29.553 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj93 2024-04-16T20:26:29.554 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jl+FZVEgui95+OZIhfAVLPgTCNQ= 2024-04-16T20:26:29.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jl+FZVEgui95+OZIhfAVLPgTCNQ='} 2024-04-16T20:26:29.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051c9f2b77b956a08-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.599 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.599 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.599 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.599 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.599 DEBUG:boto:Data: 2024-04-16T20:26:29.599 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.599 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.599 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.599 DEBUG:boto:Params: {} 2024-04-16T20:26:29.599 DEBUG:boto:Token: None 2024-04-16T20:26:29.599 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.600 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069728e3919e5fbe1-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.602 DEBUG:boto:path=/yrjefh-57/obj124 2024-04-16T20:26:29.603 DEBUG:boto:auth_path=/yrjefh-57/obj124 2024-04-16T20:26:29.603 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.603 DEBUG:boto:Path: /yrjefh-57/obj124 2024-04-16T20:26:29.603 DEBUG:boto:Data: 2024-04-16T20:26:29.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.603 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.603 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.603 DEBUG:boto:Params: {} 2024-04-16T20:26:29.603 DEBUG:boto:Token: None 2024-04-16T20:26:29.603 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj124 2024-04-16T20:26:29.604 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:VMW1bUMwWhFvZaMsj2B005KOZVE= 2024-04-16T20:26:29.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VMW1bUMwWhFvZaMsj2B005KOZVE='} 2024-04-16T20:26:29.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b60bff2ec5fdef-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.609 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.609 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.609 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.609 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.609 DEBUG:boto:Data: 2024-04-16T20:26:29.609 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.609 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.610 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.610 DEBUG:boto:Params: {} 2024-04-16T20:26:29.610 DEBUG:boto:Token: None 2024-04-16T20:26:29.610 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.610 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c83e1b01a9da1f4-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.613 DEBUG:boto:path=/yrjefh-57/obj155 2024-04-16T20:26:29.614 DEBUG:boto:auth_path=/yrjefh-57/obj155 2024-04-16T20:26:29.614 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.614 DEBUG:boto:Path: /yrjefh-57/obj155 2024-04-16T20:26:29.614 DEBUG:boto:Data: 2024-04-16T20:26:29.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.614 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.614 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.614 DEBUG:boto:Params: {} 2024-04-16T20:26:29.614 DEBUG:boto:Token: None 2024-04-16T20:26:29.615 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj155 2024-04-16T20:26:29.615 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:hy0PEBGY7wyrmKKg0Eq9N+g3BiY= 2024-04-16T20:26:29.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hy0PEBGY7wyrmKKg0Eq9N+g3BiY='} 2024-04-16T20:26:29.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b536d60b716af21-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.620 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:29.737 INFO:teuthology.orchestra.run.smithi026.stdout:tenant: 2024-04-16T20:26:29.737 INFO:teuthology.orchestra.run.smithi026.stdout:bucket name: yrjefh-57 2024-04-16T20:26:29.751 INFO:teuthology.orchestra.run.smithi026.stdout:total entries: 30 2024-04-16T20:26:29.827 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.828 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.828 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.828 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.828 DEBUG:boto:Data: 2024-04-16T20:26:29.828 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.828 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.828 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.828 DEBUG:boto:Params: {} 2024-04-16T20:26:29.828 DEBUG:boto:Token: None 2024-04-16T20:26:29.828 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.828 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043711d95e0d7597c-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.832 DEBUG:boto:path=/yrjefh-57/obj37 2024-04-16T20:26:29.832 DEBUG:boto:auth_path=/yrjefh-57/obj37 2024-04-16T20:26:29.832 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.832 DEBUG:boto:Path: /yrjefh-57/obj37 2024-04-16T20:26:29.832 DEBUG:boto:Data: 2024-04-16T20:26:29.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.833 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.833 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.833 DEBUG:boto:Params: {} 2024-04-16T20:26:29.833 DEBUG:boto:Token: None 2024-04-16T20:26:29.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj37 2024-04-16T20:26:29.833 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:t7JO84Zb052Azza7dYsS0SpcGyE= 2024-04-16T20:26:29.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t7JO84Zb052Azza7dYsS0SpcGyE='} 2024-04-16T20:26:29.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c4162b38068b09b-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.879 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.879 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.879 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.879 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.879 DEBUG:boto:Data: 2024-04-16T20:26:29.879 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.879 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.879 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.879 DEBUG:boto:Params: {} 2024-04-16T20:26:29.879 DEBUG:boto:Token: None 2024-04-16T20:26:29.880 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.880 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008981756cfe851118-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.883 DEBUG:boto:path=/yrjefh-57/obj74 2024-04-16T20:26:29.883 DEBUG:boto:auth_path=/yrjefh-57/obj74 2024-04-16T20:26:29.883 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.883 DEBUG:boto:Path: /yrjefh-57/obj74 2024-04-16T20:26:29.883 DEBUG:boto:Data: 2024-04-16T20:26:29.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.883 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.883 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.883 DEBUG:boto:Params: {} 2024-04-16T20:26:29.883 DEBUG:boto:Token: None 2024-04-16T20:26:29.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj74 2024-04-16T20:26:29.884 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:IrD+0k5XE4KENK0RL1UEfvr/APk= 2024-04-16T20:26:29.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IrD+0k5XE4KENK0RL1UEfvr/APk='} 2024-04-16T20:26:29.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4b655516a60d925-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.889 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.889 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.890 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.890 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.890 DEBUG:boto:Data: 2024-04-16T20:26:29.890 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.890 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.890 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.890 DEBUG:boto:Params: {} 2024-04-16T20:26:29.890 DEBUG:boto:Token: None 2024-04-16T20:26:29.890 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.890 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089b7e608e9e17268-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.893 DEBUG:boto:path=/yrjefh-57/obj111 2024-04-16T20:26:29.893 DEBUG:boto:auth_path=/yrjefh-57/obj111 2024-04-16T20:26:29.894 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.894 DEBUG:boto:Path: /yrjefh-57/obj111 2024-04-16T20:26:29.894 DEBUG:boto:Data: 2024-04-16T20:26:29.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.894 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.894 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.894 DEBUG:boto:Params: {} 2024-04-16T20:26:29.894 DEBUG:boto:Token: None 2024-04-16T20:26:29.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj111 2024-04-16T20:26:29.894 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:2tY7vaGHmyvnu4Fcdufka3z115o= 2024-04-16T20:26:29.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2tY7vaGHmyvnu4Fcdufka3z115o='} 2024-04-16T20:26:29.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fc525aa9144cd43-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.939 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.939 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.939 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.939 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.939 DEBUG:boto:Data: 2024-04-16T20:26:29.939 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.939 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.939 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.939 DEBUG:boto:Params: {} 2024-04-16T20:26:29.940 DEBUG:boto:Token: None 2024-04-16T20:26:29.940 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.940 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4ed12a37d14cab7-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.943 DEBUG:boto:path=/yrjefh-57/obj148 2024-04-16T20:26:29.943 DEBUG:boto:auth_path=/yrjefh-57/obj148 2024-04-16T20:26:29.943 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.943 DEBUG:boto:Path: /yrjefh-57/obj148 2024-04-16T20:26:29.943 DEBUG:boto:Data: 2024-04-16T20:26:29.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.943 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.943 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.943 DEBUG:boto:Params: {} 2024-04-16T20:26:29.944 DEBUG:boto:Token: None 2024-04-16T20:26:29.944 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj148 2024-04-16T20:26:29.944 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:aR2OwydOoAwNe9KBxvNH5UzjU8U= 2024-04-16T20:26:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aR2OwydOoAwNe9KBxvNH5UzjU8U='} 2024-04-16T20:26:29.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004895a2c10d8b1fd6-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.951 DEBUG:boto:path=/yrjefh-57/ 2024-04-16T20:26:29.951 DEBUG:boto:auth_path=/yrjefh-57/ 2024-04-16T20:26:29.952 DEBUG:boto:Method: HEAD 2024-04-16T20:26:29.952 DEBUG:boto:Path: /yrjefh-57/ 2024-04-16T20:26:29.952 DEBUG:boto:Data: 2024-04-16T20:26:29.952 DEBUG:boto:Headers: {} 2024-04-16T20:26:29.952 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.952 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.952 DEBUG:boto:Params: {} 2024-04-16T20:26:29.952 DEBUG:boto:Token: None 2024-04-16T20:26:29.952 DEBUG:boto:StringToSign: HEAD Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/ 2024-04-16T20:26:29.952 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T20:26:29.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T20:26:29.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065731220358b4d8d-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.955 DEBUG:boto:path=/yrjefh-57/obj185 2024-04-16T20:26:29.955 DEBUG:boto:auth_path=/yrjefh-57/obj185 2024-04-16T20:26:29.955 DEBUG:boto:Method: PUT 2024-04-16T20:26:29.956 DEBUG:boto:Path: /yrjefh-57/obj185 2024-04-16T20:26:29.956 DEBUG:boto:Data: 2024-04-16T20:26:29.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T20:26:29.956 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:26:29.956 DEBUG:boto:Port: 8000 2024-04-16T20:26:29.956 DEBUG:boto:Params: {} 2024-04-16T20:26:29.956 DEBUG:boto:Token: None 2024-04-16T20:26:29.956 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 16 Apr 2024 20:26:29 GMT /yrjefh-57/obj185 2024-04-16T20:26:29.956 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:Jnsi3LnWx//UEYKivAdH29Bt/sI= 2024-04-16T20:26:29.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jnsi3LnWx//UEYKivAdH29Bt/sI='} 2024-04-16T20:26:29.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0dd3b8b756ead7e-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:26:29.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:26:29.999 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T20:26:30.056 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:30.056 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:30.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:30.126+0000 7f4863a10800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T20:26:30.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:30.128+0000 7f4863a10800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T20:26:30.148 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-16T20:26:30.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T20:26:30.150 INFO:tasks.rgw.c2.client.0:Started 2024-04-16T20:26:30.151 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:30.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-16T20:26:30.169 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:26:30.169 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:26:30.169 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T20:26:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-16T20:26:30.171 INFO:teuthology.orchestra.run.smithi119.stderr:curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T20:26:30.349 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:26:30.361+0000 7f652f252880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T20:26:31.172 DEBUG:teuthology.orchestra.run.smithi119:> curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T20:26:31.192 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T20:26:31.193 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2024-04-16T20:26:31.193 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T20:26:31.193 INFO:teuthology.orchestra.run.smithi119.stdout:anonymous 2024-04-16T20:26:31.194 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-57 2024-04-16T20:26:31.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:31.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T20:26:31.194 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T20:26:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:26:31.289 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:26:31.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:31.357+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:26:36.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:36.358+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:26:41.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:41.359+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:26:46.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:46.360+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:26:51.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:51.360+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:26:56.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:26:56.361+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:01.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:01.363+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:06.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:06.364+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:11.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:11.365+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:16.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:16.365+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:21.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:21.366+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:26.368+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:31.368+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:36.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:36.369+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:41.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:41.370+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:46.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:46.371+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:51.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:51.372+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:27:56.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:27:56.372+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:01.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:01.374+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:06.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:06.375+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:11.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:11.374+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:16.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:16.376+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:21.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:21.377+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:26.378+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:31.379+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:34.124 INFO:tasks.rgw.c2.client.0.smithi119.stdout:*** Caught signal (Segmentation fault) ** 2024-04-16T20:28:34.124 INFO:tasks.rgw.c2.client.0.smithi119.stdout: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T20:28:34.125 INFO:tasks.rgw.c2.client.0.smithi119.stdout: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T20:28:34.125 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T20:28:34.126 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout:2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T20:28:34.127 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T20:28:34.128 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.137 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 0> 2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T20:28:34.138 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T20:28:34.139 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 0> 2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T20:28:34.146 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T20:28:34.147 INFO:tasks.rgw.c2.client.0.smithi119.stdout: 2024-04-16T20:28:34.179 INFO:tasks.rgw.c2.client.0.smithi119.stderr:daemon-helper: command crashed with signal 11 2024-04-16T20:28:36.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T20:28:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:36.380+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:41.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:41.382+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:42.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2024-04-16T20:28:46.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:46.383+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:48.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~12s 2024-04-16T20:28:51.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:51.383+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:53.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~18s 2024-04-16T20:28:56.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:28:56.384+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:28:59.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~24s 2024-04-16T20:29:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:01.385+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:05.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~30s 2024-04-16T20:29:06.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:06.386+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:11.387+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:11.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~35s 2024-04-16T20:29:16.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:16.388+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:17.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~41s 2024-04-16T20:29:21.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:21.389+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:23.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~47s 2024-04-16T20:29:26.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:26.390+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:29.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~53s 2024-04-16T20:29:31.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:31.390+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:35.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~59s 2024-04-16T20:29:36.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:36.392+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:41.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~65s 2024-04-16T20:29:41.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:41.393+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:46.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:46.394+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:47.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~71s 2024-04-16T20:29:51.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:51.395+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:53.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~77s 2024-04-16T20:29:56.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:29:56.396+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:29:58.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~83s 2024-04-16T20:30:01.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:01.397+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:04.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~89s 2024-04-16T20:30:06.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:06.397+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:10.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2024-04-16T20:30:11.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:11.398+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:16.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:16.399+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:16.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~100s 2024-04-16T20:30:21.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:21.400+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:22.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~106s 2024-04-16T20:30:26.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:26.401+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:28.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2024-04-16T20:30:31.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:31.402+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:34.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2024-04-16T20:30:36.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:36.403+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:40.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~124s 2024-04-16T20:30:41.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:41.404+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:46.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~130s 2024-04-16T20:30:46.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:46.404+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:51.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:51.406+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:52.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~136s 2024-04-16T20:30:56.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:30:56.407+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:30:58.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~142s 2024-04-16T20:31:01.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:01.408+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:31:03.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~148s 2024-04-16T20:31:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:06.409+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:31:09.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~154s 2024-04-16T20:31:11.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:11.410+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:31:15.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~160s 2024-04-16T20:31:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:16.411+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:31:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:21.412+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T20:31:21.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~165s 2024-04-16T20:31:26.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:26.412+0000 7f0ed1733800 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-16T20:31:26.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:26.412+0000 7f0ed1733800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T20:31:26.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:26.412+0000 7f0ed1733800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T20:31:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-16T20:31:26.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-16T20:31:26.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:26.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:26.435 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:26.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:26.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:26.549 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:31:26.574 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299189.772386_1320.1","last_update":"2024-04-16T20:26:29.772386Z"}] 2024-04-16T20:31:26.574 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:26.575 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:31:26.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:26.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:26.575 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:26.631 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:26.631 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:26.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:26.696+0000 7fb5a3d28800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:26.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:26.698+0000 7fb5a3d28800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:26.711 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:26Z"} 2024-04-16T20:31:26.711 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:26Z"} 2024-04-16T20:31:26.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:26.711 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:26.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:26.712 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-16T20:31:26.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.712 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:26.837 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[]} 2024-04-16T20:31:26.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.838 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:26.885 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:26.886 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:26.959 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-16T20:31:26.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:26.960 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.008 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.008 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.084 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:27.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.085 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.208 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-16T20:31:27.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.209 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.252 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.326 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:27.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.326 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.418 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.419 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.495 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:27.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.495 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~171s 2024-04-16T20:31:27.627 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:27.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.628 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.679 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.679 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.754 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-16T20:31:27.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.755 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.801 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.802 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:27.875 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-16T20:31:27.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:27.875 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:27.922 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.018 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":11,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]}],"directional":[{"source_zone":"b482a2a9-ef36-488b-9666-ada656453328","dest_zone":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]},"dest":{"bucket":"*","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T20:31:28.019 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=yrjefh-58 2024-04-16T20:31:28.019 DEBUG:boto:path=/yrjefh-58/ 2024-04-16T20:31:28.019 DEBUG:boto:auth_path=/yrjefh-58/ 2024-04-16T20:31:28.020 DEBUG:boto:Method: PUT 2024-04-16T20:31:28.020 DEBUG:boto:Path: /yrjefh-58/ 2024-04-16T20:31:28.020 DEBUG:boto:Data: 2024-04-16T20:31:28.020 DEBUG:boto:Headers: {} 2024-04-16T20:31:28.020 DEBUG:boto:Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T20:31:28.020 DEBUG:boto:Port: 8000 2024-04-16T20:31:28.020 DEBUG:boto:Params: {} 2024-04-16T20:31:28.021 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T20:31:28.021 DEBUG:boto:Token: None 2024-04-16T20:31:28.021 DEBUG:boto:StringToSign: PUT Tue, 16 Apr 2024 20:31:28 GMT /yrjefh-58/ 2024-04-16T20:31:28.021 DEBUG:boto:Signature: AWS VS71ZU627N05UVO7JCG7:37OccI8N4A4AsG8rGKr1pOYv1Ds= 2024-04-16T20:31:28.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:31:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:37OccI8N4A4AsG8rGKr1pOYv1Ds=', 'Content-Length': '0'} 2024-04-16T20:31:28.315 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e93b4744c0b1e5f-00661ee020-5877-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:31:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T20:31:28.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.316 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:28.373 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:28.373 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.460 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-16T20:31:28.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.461 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.597 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:28.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.598 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:28.645 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:28.646 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.718 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-16T20:31:28.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.719 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:28.764 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:28.764 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.836 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:28.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.837 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:28.969 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:28.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:28.970 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:29.102 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-16T20:31:29.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:29.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:29.102 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:29.154 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:29.154 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:29.231 INFO:teuthology.orchestra.run.smithi026.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-16T20:31:29.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:29.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T20:31:29.232 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T20:31:29.280 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:29.281 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:29.355 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-16T20:31:29.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:29.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:29.356 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:29.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:29.452 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:31:29.475 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:31:29.475 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:29.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:31:29.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:29.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:29.476 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:29.532 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:29.532 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:29.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:29.598+0000 7f230c111800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:29.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:29.599+0000 7f230c111800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:29.612 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:29Z"} 2024-04-16T20:31:29.612 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:29Z"} 2024-04-16T20:31:29.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:29.612 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:29.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:29.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:33.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~177s 2024-04-16T20:31:34.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:34.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:34.617 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:34.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:34.739+0000 7f5a3bd2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:34.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:34.741+0000 7f5a3bd2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:34.756 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:34Z"} 2024-04-16T20:31:34.756 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:34Z"} 2024-04-16T20:31:34.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:34.756 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:34.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:34.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:39.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~183s 2024-04-16T20:31:39.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:39.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:39.758 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:39.813 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:39.813 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:39.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:39.881+0000 7f987d528800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:39.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:39.883+0000 7f987d528800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:39.894 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:39Z"} 2024-04-16T20:31:39.895 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:39Z"} 2024-04-16T20:31:39.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:39.895 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:39.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:39.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:44.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:44.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:44.898 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:44.953 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:44.953 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:45.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:45.019+0000 7f580dd1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:45.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:45.020+0000 7f580dd1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:45.031 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:45Z"} 2024-04-16T20:31:45.031 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:45Z"} 2024-04-16T20:31:45.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:45.032 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:45.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:45.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:45.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~189s 2024-04-16T20:31:50.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:50.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:50.034 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:50.089 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:50.127 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:50.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:50.156+0000 7f1bd9e67800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:50.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:50.158+0000 7f1bd9e67800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:50.169 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:50Z"} 2024-04-16T20:31:50.170 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:50Z"} 2024-04-16T20:31:50.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:50.170 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:50.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:50.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:51.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~195s 2024-04-16T20:31:55.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:55.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:31:55.172 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:31:55.227 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:31:55.228 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:31:55.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:55.295+0000 7f68ba533800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:55.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:31:55.297+0000 7f68ba533800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:31:55.308 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:55Z"} 2024-04-16T20:31:55.308 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:55Z"} 2024-04-16T20:31:55.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:31:55.309 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:31:55.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:31:55.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:31:57.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~201s 2024-04-16T20:32:00.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:00.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:00.310 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:00.366 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:00.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:00.432+0000 7fdd70530800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:00.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:00.434+0000 7fdd70530800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:00.445 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:00Z"} 2024-04-16T20:32:00.446 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:00Z"} 2024-04-16T20:32:00.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:00.446 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:00.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:00.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:03.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2024-04-16T20:32:05.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:05.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:05.492 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:05.548 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:05.549 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:05.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:05.615+0000 7fc42c813800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:05.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:05.616+0000 7fc42c813800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:05.629 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:05Z"} 2024-04-16T20:32:05.630 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:05Z"} 2024-04-16T20:32:05.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:05.630 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:05.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:05.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:08.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2024-04-16T20:32:10.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:10.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:10.632 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:10.687 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:10.688 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:10.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:10.754+0000 7f9da0d24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:10.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:10.756+0000 7f9da0d24800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:10.767 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:10Z"} 2024-04-16T20:32:10.767 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:10Z"} 2024-04-16T20:32:10.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:10.768 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:10.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:10.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:14.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2024-04-16T20:32:15.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:15.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:15.770 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:15.826 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:15.826 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:15.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:15.893+0000 7fd853523800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:15.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:15.894+0000 7fd853523800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:15.907 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:15Z"} 2024-04-16T20:32:15.907 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:15Z"} 2024-04-16T20:32:15.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:15.908 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:15.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:15.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:20.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2024-04-16T20:32:20.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:20.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:20.909 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:21.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:21.032+0000 7fca843af800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:21.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:21.034+0000 7fca843af800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:21.046 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:21Z"} 2024-04-16T20:32:21.046 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:21Z"} 2024-04-16T20:32:21.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:21.046 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:21.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:21.047 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:26.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:26.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:26.048 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:26.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:26.168+0000 7f06f432a800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:26.170+0000 7f06f432a800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:26.181 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:26Z"} 2024-04-16T20:32:26.182 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:26Z"} 2024-04-16T20:32:26.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:26.182 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:26.182 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:26.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:26.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2024-04-16T20:32:31.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:31.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:31.184 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:31.240 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:31.240 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:31.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:31.307+0000 7f4aa832d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:31.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:31.308+0000 7f4aa832d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:31.320 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:31Z"} 2024-04-16T20:32:31.320 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:31Z"} 2024-04-16T20:32:31.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:31.321 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:31.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:31.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:32.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~236s 2024-04-16T20:32:36.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:36.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:36.323 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:36.378 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:36.378 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:36.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:36.445+0000 7f08182e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:36.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:36.446+0000 7f08182e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:36.458 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:36Z"} 2024-04-16T20:32:36.459 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:36Z"} 2024-04-16T20:32:36.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:36.459 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:36.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:36.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:38.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~242s 2024-04-16T20:32:41.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:41.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:41.461 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:41.517 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:41.517 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:41.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:41.584+0000 7f6aba467800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:41.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:41.586+0000 7f6aba467800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:41.599 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:41Z"} 2024-04-16T20:32:41.599 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:41Z"} 2024-04-16T20:32:41.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:41.599 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:41.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:41.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:44.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~248s 2024-04-16T20:32:46.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:46.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:46.601 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:46.656 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:46.656 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:46.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:46.723+0000 7f5e43f1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:46.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:46.724+0000 7f5e43f1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:46.737 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:46Z"} 2024-04-16T20:32:46.737 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:46Z"} 2024-04-16T20:32:46.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:46.738 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:46.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:46.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:50.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~254s 2024-04-16T20:32:51.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:51.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:51.740 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:51.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:51.863+0000 7fd387c2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:51.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:51.864+0000 7fd387c2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:51.875 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:51Z"} 2024-04-16T20:32:51.875 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:51Z"} 2024-04-16T20:32:51.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:51.875 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:51.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:51.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:32:56.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~260s 2024-04-16T20:32:56.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:56.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:32:56.878 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:32:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:32:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:32:56.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:57.001+0000 7fbceed31800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:56.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:32:57.002+0000 7fbceed31800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:32:57.013 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:57Z"} 2024-04-16T20:32:57.013 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:57Z"} 2024-04-16T20:32:57.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:32:57.014 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:32:57.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:32:57.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:02.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:02.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:02.016 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:02.071 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:02.072 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:02.138+0000 7fabda28c800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:02.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~266s 2024-04-16T20:33:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:02.139+0000 7fabda28c800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:02.151 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:02Z"} 2024-04-16T20:33:02.151 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:02Z"} 2024-04-16T20:33:02.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:02.152 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:02.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:02.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:07.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:07.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:07.154 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:07.277+0000 7f7571f2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:07.278+0000 7f7571f2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:07.289 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:07Z"} 2024-04-16T20:33:07.290 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:07Z"} 2024-04-16T20:33:07.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:07.291 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:07.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:07.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:08.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~272s 2024-04-16T20:33:12.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:14.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:14.136 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:14.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:14.258+0000 7f9234424800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:14.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:14.259+0000 7f9234424800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:14.270 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:14Z"} 2024-04-16T20:33:14.274 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:14Z"} 2024-04-16T20:33:14.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:14.275 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:14.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:14.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:15.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~279s 2024-04-16T20:33:19.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:19.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:19.276 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:19.332 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:19.332 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:19.400+0000 7f6cc4424800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:19.401+0000 7f6cc4424800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:19.412 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:19Z"} 2024-04-16T20:33:19.413 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:19Z"} 2024-04-16T20:33:19.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:19.413 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:19.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:19.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:20.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~285s 2024-04-16T20:33:24.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:24.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:24.415 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:24.472 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:24.472 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:24.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:24.539+0000 7f484aa12800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:24.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:24.540+0000 7f484aa12800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:24.552 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:24Z"} 2024-04-16T20:33:24.552 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:24Z"} 2024-04-16T20:33:24.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:24.553 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:24.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:24.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:26.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~291s 2024-04-16T20:33:29.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:29.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:29.555 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:29.610 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:29.611 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:29.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:29.678+0000 7f850ce0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:29.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:29.679+0000 7f850ce0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:29.690 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:29Z"} 2024-04-16T20:33:29.691 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:29Z"} 2024-04-16T20:33:29.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:29.691 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:29.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:29.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:32.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2024-04-16T20:33:34.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:34.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:34.693 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:34.749 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:34.749 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:33:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:34.815+0000 7f8f85533800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:33:34.818+0000 7f8f85533800 20 RGW-SYNC:meta: read sync status 2024-04-16T20:33:34.832 INFO:teuthology.orchestra.run.smithi119.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:34Z"} 2024-04-16T20:33:34.833 DEBUG:rgw_multi.tests:current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:34Z"} 2024-04-16T20:33:34.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T20:33:34.833 DEBUG:rgw_multi.tests:log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:33:34.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T20:33:34.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T20:33:38.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~302s 2024-04-16T20:33:38.684 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T20:33:39.590 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2024-04-16T20:33:39.591 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2024-04-16T20:33:39.591 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2024-04-16T20:33:39.591 INFO:tasks.ceph.c2.osd.3:Sent signal 15 2024-04-16T20:33:39.592 INFO:tasks.ceph.c2.osd.4:Sent signal 15 2024-04-16T20:33:39.592 INFO:tasks.ceph.c2.osd.5:Sent signal 15 2024-04-16T20:33:39.593 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2024-04-16T20:33:39.593 INFO:tasks.rgw.c2.client.1:Sent signal 15 2024-04-16T20:33:39.593 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2024-04-16T20:33:39.595 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 3 (PID: 32157) UID: 0 2024-04-16T20:33:39.595 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 osd.3 55 *** Got signal Terminated *** 2024-04-16T20:33:39.595 INFO:tasks.ceph.c2.osd.3.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 osd.3 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.595 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 32537) UID: 0 2024-04-16T20:33:39.596 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 osd.0 55 *** Got signal Terminated *** 2024-04-16T20:33:39.596 INFO:tasks.ceph.c2.osd.0.smithi119.stderr:2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 osd.0 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.596 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 4 (PID: 32158) UID: 0 2024-04-16T20:33:39.596 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 osd.4 55 *** Got signal Terminated *** 2024-04-16T20:33:39.596 INFO:tasks.ceph.c2.osd.4.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 osd.4 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T20:33:39.604+0000 7fe86e792640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 32539) UID: 0 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T20:33:39.604+0000 7fe86e792640 -1 osd.1 55 *** Got signal Terminated *** 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.1.smithi119.stderr:2024-04-16T20:33:39.604+0000 7fe86e792640 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f242c244640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 5 (PID: 32161) UID: 0 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T20:33:39.605+0000 7f242c244640 -1 osd.5 55 *** Got signal Terminated *** 2024-04-16T20:33:39.597 INFO:tasks.ceph.c2.osd.5.smithi151.stderr:2024-04-16T20:33:39.606+0000 7f242c244640 -1 osd.5 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.598 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T20:33:39.605+0000 7fe85b524640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 32540) UID: 0 2024-04-16T20:33:39.598 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T20:33:39.605+0000 7fe85b524640 -1 osd.2 55 *** Got signal Terminated *** 2024-04-16T20:33:39.598 INFO:tasks.ceph.c2.osd.2.smithi119.stderr:2024-04-16T20:33:39.605+0000 7fe85b524640 -1 osd.2 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T20:33:39.599 INFO:tasks.rgw.c2.client.1.smithi151.stdout:2024-04-16T20:33:39.606+0000 7ffbd341d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 36798) UID: 0 2024-04-16T20:33:39.599 INFO:tasks.rgw.c2.client.1.smithi151.stdout:2024-04-16T20:33:39.607+0000 7ffbd942f880 -1 shutting down 2024-04-16T20:33:39.599 INFO:tasks.ceph.c2.mon.a.smithi119.stderr:2024-04-16T20:33:39.606+0000 7f2b098e8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 32368) UID: 0 2024-04-16T20:33:39.599 INFO:tasks.ceph.c2.mon.a.smithi119.stderr:2024-04-16T20:33:39.606+0000 7f2b098e8640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T20:33:39.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:39.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:33:39.835 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:33:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:33:39.900 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:38:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T20:38:39.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:38:39.912+0000 7f0c37d11800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T20:38:39.951 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T20:38:42.671 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2024-04-16T20:38:44.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:38:44.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:38:44.773 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:38:44.880 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:38:44.880 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:38:44.938 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:38:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:38:45.021 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:38:45.021 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:38:45.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:38:45.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:38:45.021 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:38:45.104 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:38:45.105 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:43:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T20:43:45.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:43:45.119+0000 7f783c31e800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T20:43:45.123 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T20:43:45.168 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-04-16T20:43:45.537 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-16T20:43:45.542 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-16T20:43:45.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:43:45.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:43:45.572 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:43:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:43:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:43:45.685 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:43:45.709 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:43:45.710 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:43:45.710 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:43:45.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:43:45.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:43:45.711 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:43:45.769 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:43:45.769 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:48:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T20:48:45.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:48:45.783+0000 7fc40ad2a800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T20:48:45.772 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T20:48:45.855 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2024-04-16T20:48:45.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:48:45.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:48:45.886 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:48:45.941 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:48:45.941 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:48:45.999 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:48:46.037 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:48:46.038 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:48:46.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:48:46.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:48:46.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:48:46.038 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:48:46.095 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:48:46.096 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:53:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T20:53:46.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:53:46.111+0000 7ff6de0c0800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T20:53:46.100 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T20:53:46.134 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2024-04-16T20:53:46.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:53:46.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:53:46.163 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:53:46.217 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:53:46.217 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:53:46.276 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:53:46.300 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:53:46.302 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:53:46.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:53:46.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:53:46.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:53:46.303 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:53:46.357 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:53:46.358 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:58:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T20:58:46.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T20:58:46.373+0000 7f44d3f11800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T20:58:46.362 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T20:58:46.421 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-04-16T20:58:46.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:58:46.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:58:46.471 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:58:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:58:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T20:58:46.620 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T20:58:46.645 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T20:58:46.646 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T20:58:46.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-16T20:58:46.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:58:46.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T20:58:46.648 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T20:58:46.703 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T20:58:46.703 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T21:03:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:03:46.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T21:03:46.719+0000 7f2202f33800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:46.707 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:46.750 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2024-04-16T21:03:46.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create_rgw_down ... SKIP: test_bucket_create_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove_rgw_down ... SKIP: test_bucket_remove_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync_rgw_down ... SKIP: test_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete_rgw_down ... SKIP: test_object_delete_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync_rgw_down ... SKIP: test_concurrent_versioned_object_incremental_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync_rgw_down ... SKIP: test_suspended_delete_marker_full_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl_rgw_down ... SKIP: test_bucket_acl_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable_rgw_down ... SKIP: test_bucket_sync_enable_right_after_disable_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync_rgw_down ... SKIP: test_multipart_object_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_run_basic_incremental_rgw_down ... SKIP: test_bucket_sync_run_basic_incremental_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync_rgw_down ... SKIP: test_role_sync_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init_rgw_down ... SKIP: test_bucket_full_sync_after_data_sync_init_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_policy_config_zonegroup_rgw_down ... SKIP: test_sync_policy_config_zonegroup_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:46.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_sync_flow_symmetrical_zonegroup_all_rgw_down ... SKIP: test_sync_flow_symmetrical_zonegroup_all_rgw_down skipped. More than one rgw needed in any one or multiple zone(s). 2024-04-16T21:03:47.206 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.239 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:47.261 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_topic_notification_sync 2024-04-16T21:03:47.290 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:47.470 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:47.470 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 1788, in test_topic_notification_sync 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: zone_conn.create_notification(bucket.name, [topic_conf]) 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multi/zone_rados.py", line 160, in create_notification 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: return self.s3_client.put_bucket_notification_configuration( 2024-04-16T21:03:47.476 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call 2024-04-16T21:03:47.478 INFO:tasks.rgw_multisite_tests: return self._make_api_call(operation_name, kwargs) 2024-04-16T21:03:47.478 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call 2024-04-16T21:03:47.478 INFO:tasks.rgw_multisite_tests: raise error_class(parsed_response, operation_name) 2024-04-16T21:03:47.478 INFO:tasks.rgw_multisite_tests:botocore.exceptions.ClientError: An error occurred (ConcurrentModification) when calling the PutBucketNotificationConfiguration operation: None 2024-04-16T21:03:47.478 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.479 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:06.903+0000 7f0516216800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:06.905+0000 7f0516216800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:06Z"} 2024-04-16T21:03:47.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T21:03:47.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713295583.603379_140.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create topic zone=test-zone1 name=topic-yrjefh-1 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'topic-yrjefh-1', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'http://kaboom:9999', 'Attributes.entry.2.key': 'persistent', 'Attributes.entry.2.value': 'true'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler > 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.sns.CreateTopic: calling handler 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:POST 2024-04-16T21:03:47.520 INFO:tasks.rgw_multisite_tests:/ 2024-04-16T21:03:47.521 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.521 INFO:tasks.rgw_multisite_tests:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:host:smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195411Z 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:content-type;host;x-amz-date 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:6ea285337f3d2901b9784063b7470502ca9dcbf9383f4b30bb8b2af9c15ccdd1 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:20240416T195411Z 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/sns/aws4_request 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:6eff7ed354e76a14e0c72b37a921274ac52c867592b5fe52aa11b28df393a852 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.523 INFO:tasks.rgw_multisite_tests:50dae9fb41b8097f7593b436c8571628370d01a9838007e5dc6363f40e66fa01 2024-04-16T21:03:47.533 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler 2024-04-16T21:03:47.533 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.533 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000edb00c9ccb5bc253-00661ed763-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '337', 'Date': 'Tue, 16 Apr 2024 19:54:11 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:b'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1b482a2a9-ef36-488b-9666-ada656453328.4812.17127203250456216147' 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.sns.CreateTopic: calling handler 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create topic zone=test-zone2 name=topic-yrjefh-2 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'topic-yrjefh-2', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'http://kaboom:9999', 'Attributes.entry.2.key': 'persistent', 'Attributes.entry.2.value': 'true'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler > 2024-04-16T21:03:47.534 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.sns.CreateTopic: calling handler 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:POST 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:/ 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-16T21:03:47.535 INFO:tasks.rgw_multisite_tests:host:smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195411Z 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:content-type;host;x-amz-date 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:485e4ca69f91960253a643c532fe0db5b5e57cd613a06750d5157b31631406db 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:20240416T195411Z 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/sns/aws4_request 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:202e2eccc4e8e65e7bff985b385acb76f6f84766db8bcd621f49adf8a5b5ab0a 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.537 INFO:tasks.rgw_multisite_tests:3b13eeddbd39f4a1af39116185d59d17d13b7eaf863dc241e155432c094dd6b0 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000006dce9b2ff72424c0-00661ed763-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '336', 'Date': 'Tue, 16 Apr 2024 19:54:11 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:b'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2eafd0fcf-abef-4654-893b-ccd3b2eb5bea.5331.7912432225648649408' 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.sns.CreateTopic: calling handler 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: topic_arns: ['arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1', 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'] 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:47.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713297251.965282_828.1","last_update":"2024-04-16T19:54:11.965282Z"},{"marker":"1_1713297244.914928_633.1","last_update":"2024-04-16T19:54:04.914928Z"}] 2024-04-16T21:03:47.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:12.219+0000 7f2f3af33800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:12.221+0000 7f2f3af33800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T21:03:47.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297244.914928_633.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:04.914928Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297251.965282_828.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:11.965282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297244.914928_633.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.914928Z', 'realm_epoch': 2}}] 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297244.914928_633.1'} 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.sns.ListTopics: calling handler 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000'} 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.ListTopics: calling handler 2024-04-16T21:03:47.655 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.ListTopics: calling handler 2024-04-16T21:03:47.667 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=ListTopics) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListTopics', 'Version': '2010-03-31'}, 'url': 'http://smithi026.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T21:03:47.668 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.ListTopics: calling handler > 2024-04-16T21:03:47.668 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.sns.ListTopics: calling handler 2024-04-16T21:03:47.669 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.669 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:POST 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:/ 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:host:smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195412Z 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:content-type;host;x-amz-date 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:20240416T195412Z 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/sns/aws4_request 2024-04-16T21:03:47.671 INFO:tasks.rgw_multisite_tests:bddf64a8f5d2b8f90555dcedd13fa9531557a2620b0710be0044e4df0ad7e4db 2024-04-16T21:03:47.672 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.672 INFO:tasks.rgw_multisite_tests:41cf30b5719ddf0bab493b45f855c5a1baa49f64b085399451163a36ce9e725d 2024-04-16T21:03:47.672 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.ListTopics: calling handler 2024-04-16T21:03:47.672 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.672 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000003d1358660dd4c313-00661ed764-4812-test-zone1', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1414', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:b'RGW11111111111111111topic-yrjefh-1http://kaboom:9999Version=2010-03-31topic-yrjefh-1FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1RGW11111111111111111topic-yrjefh-2http://kaboom:9999Version=2010-03-31&rgwx-uid=RGW11111111111111111&rgwx-zonegroup=test-zonegrouptopic-yrjefh-2FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2b482a2a9-ef36-488b-9666-ada656453328.4812.4400958456213062419' 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.sns.ListTopics: calling handler 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: topics for zone=test-zone1 = [{'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1'}, {'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'}] 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.sns.ListTopics: calling handler 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Region': 'test-zonegroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000'} 2024-04-16T21:03:47.674 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.ListTopics: calling handler 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.sns.ListTopics: calling handler 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=ListTopics) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': {'Action': 'ListTopics', 'Version': '2010-03-31'}, 'url': 'http://smithi119.front.sepia.ceph.com:8000/', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.ListTopics: calling handler > 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.sns.ListTopics: calling handler 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:POST 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:/ 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:host:smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195412Z 2024-04-16T21:03:47.675 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:content-type;host;x-amz-date 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:20240416T195412Z 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/sns/aws4_request 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:d85286e07b540e4fa298579bcb62f7de89ca95e537225788046606c560a9ccb3 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:43703228d02333abb8a0d4ab5bf6bc2729300f8b5346ed0ce6920e292deb3a58 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.sns.ListTopics: calling handler 2024-04-16T21:03:47.676 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000eb2899fcc08c7bd5-00661ed764-5331-test-zone2', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '1415', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:b'RGW11111111111111111topic-yrjefh-1http://kaboom:9999Version=2010-03-31topic-yrjefh-1FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1RGW11111111111111111topic-yrjefh-2http://kaboom:9999Version=2010-03-31&rgwx-uid=RGW11111111111111111&rgwx-zonegroup=test-zonegrouptopic-yrjefh-2FalseTrueNoneNoneNonearn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2eafd0fcf-abef-4654-893b-ccd3b2eb5bea.5331.16944962908822731733' 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.sns.ListTopics: calling handler 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: topics for zone=test-zone2 = [{'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1'}, {'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2'}] 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-49/ 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-49/ 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-49/ 2024-04-16T21:03:47.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:47.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:12 GMT 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:/yrjefh-49/ 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6N4SwSnwJesEvhXD92ZAd8aH2oc= 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6N4SwSnwJesEvhXD92ZAd8aH2oc=', 'Content-Length': '0'} 2024-04-16T21:03:47.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008b53afeb19387562-00661ed764-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yrjefh-49 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297246.414809_116.1","last_update":"2024-04-16T19:54:06.414809Z"},{"marker":"1_1713297251.965282_828.1","last_update":"2024-04-16T19:54:11.965282Z"},{"marker":"1_1713297252.276314_637.1","last_update":"2024-04-16T19:54:12.276314Z"}] 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:47.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:47.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:47.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:47.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:12.503+0000 7f9d8272a800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:12.505+0000 7f9d8272a800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:47.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297252.276314_637.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:12.276314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T21:03:47.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297246.414809_116.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:06.414809Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297251.965282_828.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:11.965282Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297252.276314_637.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:12.276314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:12Z"} 2024-04-16T21:03:47.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297246.414809_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:06.414809Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297251.965282_828.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:11.965282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297252.276314_637.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:12.276314Z', 'realm_epoch': 2}}] 2024-04-16T21:03:47.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T21:03:47.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297246.414809_116.1', 2: '1_1713297251.965282_828.1', 3: '1_1713297252.276314_637.1'} 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: creating bucket notification for zone=test-zone1 name=bn-yrjefh-1 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler > 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Bucket': 'yrjefh-49', 'Region': 'test-zonegroup', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://smithi026.front.sepia.ceph.com:8000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True, 'UseS3ExpressControlEndpoint': True} 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi026.front.sepia.ceph.com:8000/yrjefh-49 2024-04-16T21:03:47.706 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': b'bn-yrjefh-1arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1s3:ObjectCreated:*', 'auth_path': '/yrjefh-49/', 'url': 'http://smithi026.front.sepia.ceph.com:8000/yrjefh-49?notification', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'yrjefh-49', 'params': {'Bucket': 'yrjefh-49', 'NotificationConfiguration': {'TopicConfigurations': [{'Id': 'bn-yrjefh-1', 'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-1', 'Events': ['s3:ObjectCreated:*']}]}}}, 'S3Express': {'bucket_name': 'yrjefh-49'}, 'signing': {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'test-zonegroup'}]}}} 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.707 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:/yrjefh-49 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:notification= 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:host:smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:x-amz-content-sha256:be87c4a3acfd35705d711d944633e64c1ba48da889de87610544d17cdea27e1e 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195412Z 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:host;x-amz-content-sha256;x-amz-date 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:be87c4a3acfd35705d711d944633e64c1ba48da889de87610544d17cdea27e1e 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:20240416T195412Z 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/s3/aws4_request 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:226d73d40d3e32b0a4a709841ebbbf799f1eb0a9ebc2098bdeaa46acec794bb4 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.708 INFO:tasks.rgw_multisite_tests:ad152cde17734ea01472cac197c735adeef270113bb4d56f0319f86002bbce81 2024-04-16T21:03:47.709 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.709 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000bce598ca9790daa1-00661ed764-4812-test-zone1', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '0', 'Date': 'Tue, 16 Apr 2024 19:54:12 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:b'' 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: creating bucket notification for zone=test-zone2 name=bn-yrjefh-2 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler > 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Bucket': 'yrjefh-49', 'Region': 'test-zonegroup', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://smithi119.front.sepia.ceph.com:8000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True, 'UseS3ExpressControlEndpoint': True} 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Endpoint provider result: http://smithi119.front.sepia.ceph.com:8000/yrjefh-49 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.regions: DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.717 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.718 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.718 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.8 md/Botocore#1.34.8 ua/2.0 os/linux#5.4.0-166-generic md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.8'}, 'body': b'bn-yrjefh-2arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2s3:ObjectCreated:*', 'auth_path': '/yrjefh-49/', 'url': 'http://smithi119.front.sepia.ceph.com:8000/yrjefh-49?notification', 'context': {'client_region': 'test-zonegroup', 'client_config': , 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'yrjefh-49', 'params': {'Bucket': 'yrjefh-49', 'NotificationConfiguration': {'TopicConfigurations': [{'Id': 'bn-yrjefh-2', 'TopicArn': 'arn:aws:sns:test-zonegroup:RGW11111111111111111:topic-yrjefh-2', 'Events': ['s3:ObjectCreated:*']}]}}}, 'S3Express': {'bucket_name': 'yrjefh-49'}, 'signing': {'region': 'test-zonegroup', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'test-zonegroup'}]}}} 2024-04-16T21:03:47.718 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: CanonicalRequest: 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:/yrjefh-49 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:notification= 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:host:smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:x-amz-content-sha256:09fafb09dfdc35bb179c0f67bc7cfb4d7f29d8187ab56ed97957fecd46102881 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests:x-amz-date:20240416T195412Z 2024-04-16T21:03:47.720 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:host;x-amz-content-sha256;x-amz-date 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:09fafb09dfdc35bb179c0f67bc7cfb4d7f29d8187ab56ed97957fecd46102881 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: StringToSign: 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:AWS4-HMAC-SHA256 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:20240416T195412Z 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:20240416/test-zonegroup/s3/aws4_request 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:f817c808b14a1195f9b70dd7b79eea454fa090b7c0b41e01d9c81b1b5708a649 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.auth: DEBUG: Signature: 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:f62613d11e6117884d71f4887a1653eea86debab513057db386452716b9542d9 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.endpoint: DEBUG: Sending http request: 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response headers: {'Content-Length': '261', 'x-amz-request-id': 'tx00000a0b4a01262222d3b-00661ed764-5331-test-zone2', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Date': 'Tue, 16 Apr 2024 19:54:13 GMT', 'Connection': 'Keep-Alive'} 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.parsers: DEBUG: Response body: 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:b'ConcurrentModificationyrjefh-49tx00000a0b4a01262222d3b-00661ed764-5331-test-zone25331-test-zone2-test-zonegroup' 2024-04-16T21:03:47.721 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-04-16T21:03:47.722 INFO:tasks.rgw_multisite_tests:botocore.retryhandler: DEBUG: No retry needed. 2024-04-16T21:03:47.722 INFO:tasks.rgw_multisite_tests:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-16T21:03:47.722 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:47.722 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:47.741 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:47.741 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-16T21:03:47.742 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 880, in test_multi_object_delete 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 390, in zone_bucket_checkpoint 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-16T21:03:48.060 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yrjefh-11 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QEQBaF+ahEVrwB3QSxMN1c9Ecm4= 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QEQBaF+ahEVrwB3QSxMN1c9Ecm4=', 'Content-Length': '0'} 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008cdca3691f685361-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=yrjefh-12 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.063 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Sfnj14g6QYwvujKaYBf3agh7qgY= 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Sfnj14g6QYwvujKaYBf3agh7qgY=', 'Content-Length': '0'} 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e5b0951cc15c6d79-00661ed0fa-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0881a826cebd79c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj1 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj1 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj1 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.066 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj1 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:H+PMlk07VEbYt9nkOUwTYEF0FeU= 2024-04-16T21:03:48.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:H+PMlk07VEbYt9nkOUwTYEF0FeU='} 2024-04-16T21:03:48.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004da4898423efa264-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.070 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff258eeef90375ef-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj2 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj2 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj2 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj2 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OOsLE9mFlyRMMQOXi+dwYOWDWEE= 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OOsLE9mFlyRMMQOXi+dwYOWDWEE='} 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af4f3a64b7cb0f8f-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.073 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e6f4c3e40316d07-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj3 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj3 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj3 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj3 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zIbASO3lho5H1JJySGPFFAKIIoU= 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zIbASO3lho5H1JJySGPFFAKIIoU='} 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013c9d9b502c0e344-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.076 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081202dd1cac1c30c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj4 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj4 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj4 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj4 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6SzmzIxv+qFTNGP2ANxciNGnxV8= 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6SzmzIxv+qFTNGP2ANxciNGnxV8='} 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd6c419ca6e11ddf-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.079 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021418df9289a1a62-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj5 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj5 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj5 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj5 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2+fUM5Puvz26T3V+qWA1Gli37y8= 2024-04-16T21:03:48.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2+fUM5Puvz26T3V+qWA1Gli37y8='} 2024-04-16T21:03:48.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c975a4ec6f6a10ff-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055ff5c8e082ec9da-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj6 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj6 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj6 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.086 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj6 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gUtj3p+S8NxKoDWjwkBrznYfeMY= 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gUtj3p+S8NxKoDWjwkBrznYfeMY='} 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021f24912cb6e3cf6-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6e6a8c9470e5d7c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj7 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj7 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj7 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.089 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj7 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SMnYfVAc4YT0uaSuDDMehFQR1lg= 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SMnYfVAc4YT0uaSuDDMehFQR1lg='} 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4c70df07dafd5d2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003410257b4fdb8c7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj8 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj8 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj8 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj8 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:meg2u+c5qQ9U/oL+6LQt79XhDCU= 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:meg2u+c5qQ9U/oL+6LQt79XhDCU='} 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057ff3e738cbe88be-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.094 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076a2372e4d2280df-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj9 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj9 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj9 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj9 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:itVjPyJXUGzRUbI9xzweYsBvZCc= 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:itVjPyJXUGzRUbI9xzweYsBvZCc='} 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000220a26c288c561ef-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1b9cd4f73fc5ce-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj10 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj10 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj10 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.099 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj10 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CSGlPy4oKPkBwnXIgO2xvmPJ1nA= 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CSGlPy4oKPkBwnXIgO2xvmPJ1nA='} 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4309ffdd7efad61-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.101 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bd1770dba7aff6b-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj11 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj11 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj11 2024-04-16T21:03:48.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj11 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s9qXXbuQYX67yfyQL4dgPtdACBQ= 2024-04-16T21:03:48.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s9qXXbuQYX67yfyQL4dgPtdACBQ='} 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f18f628432dfd011-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006562266b8809cdcf-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj12 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj12 2024-04-16T21:03:48.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj12 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj12 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kUBn6eAC0uVd6ulI6KJQUqryjSI= 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kUBn6eAC0uVd6ulI6KJQUqryjSI='} 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e24a70f5815601b7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e90e77e3458b91ce-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj13 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj13 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj13 2024-04-16T21:03:48.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.130 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj13 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zyB4icBzPYCmJO64sveJMfWMRV4= 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zyB4icBzPYCmJO64sveJMfWMRV4='} 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df6548f4e79feb74-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c035391ccb9f6d61-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj14 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj14 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj14 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj14 2024-04-16T21:03:48.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.134 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:35WWd0N6er8aiUr1S8ABcNEdLZs= 2024-04-16T21:03:48.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35WWd0N6er8aiUr1S8ABcNEdLZs='} 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008de1c9640d54bffb-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006552993c0e4baf84-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj15 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj15 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj15 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.138 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.139 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj15 2024-04-16T21:03:48.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.139 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:M4wlcANluCLUBZE1SbI3kgV7sf0= 2024-04-16T21:03:48.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M4wlcANluCLUBZE1SbI3kgV7sf0='} 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6636f0d7a94ba43-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd746df397ef182d-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj16 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj16 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj16 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.143 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.144 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.144 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj16 2024-04-16T21:03:48.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.144 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8RP78nYp/BgXZIb7XHfBN27SpnQ= 2024-04-16T21:03:48.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8RP78nYp/BgXZIb7XHfBN27SpnQ='} 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019a3ec9a4b75d01c-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007978e01f77a2ac72-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj17 2024-04-16T21:03:48.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj17 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj17 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.149 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.149 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.149 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.149 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.149 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj17 2024-04-16T21:03:48.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.152 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:EKUd4UGR0ydeSK4SEf6IzToMNiY= 2024-04-16T21:03:48.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EKUd4UGR0ydeSK4SEf6IzToMNiY='} 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006026850401352a06-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beebedb56ad967fd-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj18 2024-04-16T21:03:48.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj18 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj18 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.156 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj18 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GRGYeWa6pCMZrGzp+MlFpIdf1q4= 2024-04-16T21:03:48.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GRGYeWa6pCMZrGzp+MlFpIdf1q4='} 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009922147c34580a1b-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.164 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.164 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015042aa8b509270f-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj19 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj19 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj19 2024-04-16T21:03:48.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj19 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lrsmStDMJ+9s/Wj+9Bu48Qe7Xzs= 2024-04-16T21:03:48.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lrsmStDMJ+9s/Wj+9Bu48Qe7Xzs='} 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cbcda8a726c6aa9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1f21b5b73fed0ba-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj20 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj20 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj20 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj20 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ls3RTUi5Zc2pnhAJF++XdhuFIPU= 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ls3RTUi5Zc2pnhAJF++XdhuFIPU='} 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44efa74e900b8e6-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.186 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7d82dfcc409927e-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj21 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj21 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj21 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.187 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj21 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:bWlnCPY5oFlEtJ4U0wdjs7NwjAw= 2024-04-16T21:03:48.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bWlnCPY5oFlEtJ4U0wdjs7NwjAw='} 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000c29c3ece3567fe-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa89db627720a518-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj22 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj22 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj22 2024-04-16T21:03:48.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.206 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj22 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c5BQ6mUaFpKx3wsGPUXzI5JnUZo= 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c5BQ6mUaFpKx3wsGPUXzI5JnUZo='} 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c012246ace0a916-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.213 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000840b6fb7057b42e2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj23 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj23 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj23 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj23 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7ULAnS06NKgMfq0VFD7QecvlcwY= 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7ULAnS06NKgMfq0VFD7QecvlcwY='} 2024-04-16T21:03:48.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d2282ccd4e97843-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.218 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a200e2ccc852e7f9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj24 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj24 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj24 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj24 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Kr/04c1a8D9n36uZXMoOnYpjrPw= 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Kr/04c1a8D9n36uZXMoOnYpjrPw='} 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2fcd22a9ca34491-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.221 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dddecbdd24d61dc-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj25 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj25 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj25 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj25 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tZLjaP6q1RYk6Gr0UfpWrb1dyHk= 2024-04-16T21:03:48.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tZLjaP6q1RYk6Gr0UfpWrb1dyHk='} 2024-04-16T21:03:48.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3d5ba4e3a2079c9-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000681847a383af12c7-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj26 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj26 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj26 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj26 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.241 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jsFBx1EVuI1XgQm0DeIUerLQ4FQ= 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jsFBx1EVuI1XgQm0DeIUerLQ4FQ='} 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b0c3eee95c0e8ab-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000243c32de476ef6a2-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj27 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj27 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj27 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj27 2024-04-16T21:03:48.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.247 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pZ74elxSY5R6qcHB+RQ8RFC3YCk= 2024-04-16T21:03:48.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pZ74elxSY5R6qcHB+RQ8RFC3YCk='} 2024-04-16T21:03:48.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052fd208104cec402-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.249 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f50063c2052ce985-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj28 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj28 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj28 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.251 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj28 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wB/Y8+56N8Fa4RCR4jmtqIoRD3w= 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wB/Y8+56N8Fa4RCR4jmtqIoRD3w='} 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067fc803444995def-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU= 2024-04-16T21:03:48.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Ly806m8HSyVE6/7ANHoDFaA0zU=', 'Content-Length': '0'} 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3c6589116b14dc-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj29 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj29 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj29 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.254 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:50 GMT 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj29 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:YA/tqq/KO1e62BYkkAe5ulILFZ8= 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:50 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YA/tqq/KO1e62BYkkAe5ulILFZ8='} 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd09747b9b5a4100-00661ed0fa-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.256 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000820002739544a41-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj30 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj30 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj30 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj30 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jVZK/lY9PstIy9sVRnlHVUzZ7P4= 2024-04-16T21:03:48.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jVZK/lY9PstIy9sVRnlHVUzZ7P4='} 2024-04-16T21:03:48.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000568185cb48e92ea7-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.261 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b87edf8b4113cb42-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj31 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj31 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj31 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.263 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj31 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Q0/Pt+z6Xn1+osiNP6y8gSDeHAE= 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q0/Pt+z6Xn1+osiNP6y8gSDeHAE='} 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3ce578c90760054-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c7caac0276a4c6d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj32 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj32 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj32 2024-04-16T21:03:48.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj32 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.266 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:COZdk1jOfOXjyqxBW/jGES1W5/g= 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:COZdk1jOfOXjyqxBW/jGES1W5/g='} 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058fab48682da643e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.267 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f77d33ae9bba893-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj33 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj33 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj33 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj33 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2uMytdFc0mRquimE1p+ZoFPdLLk= 2024-04-16T21:03:48.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2uMytdFc0mRquimE1p+ZoFPdLLk='} 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb964f8024248a40-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c4808f06965f544-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj34 2024-04-16T21:03:48.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj34 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj34 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj34 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:t2WCYni02X4ug9QHmC+BSXlbBa0= 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t2WCYni02X4ug9QHmC+BSXlbBa0='} 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f0b89af60cafcb9-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.276 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc95099407256ac-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj35 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj35 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj35 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj35 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T2KxNjm+r9ZpY4jxOOSsHgaiDOo= 2024-04-16T21:03:48.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T2KxNjm+r9ZpY4jxOOSsHgaiDOo='} 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006025c2bc1d922935-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e23a8f315c3e1dd-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj36 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj36 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj36 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj36 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:bTUOSovdQ1fMD1uZDoex3FjrotQ= 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bTUOSovdQ1fMD1uZDoex3FjrotQ='} 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d2f8bc52446924e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.283 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007996871eaca9b0ec-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj37 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj37 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj37 2024-04-16T21:03:48.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj37 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.285 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HkxyCSOFf4MAeRkRigonR58p0pI= 2024-04-16T21:03:48.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HkxyCSOFf4MAeRkRigonR58p0pI='} 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000271203cc03272857-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caacbb34f9bd9cb3-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj38 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj38 2024-04-16T21:03:48.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj38 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.290 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj38 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:EUezPiRm1t9NbSc76fAe5INgeeA= 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EUezPiRm1t9NbSc76fAe5INgeeA='} 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f3e3a6e5fe5de53-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.292 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000138c012f4b13fbee-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj39 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj39 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj39 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj39 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tatDDQ3f2zRwQ5w7h/xYFi407cA= 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tatDDQ3f2zRwQ5w7h/xYFi407cA='} 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075391506984427dc-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.295 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f500b752301e93d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj40 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj40 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj40 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj40 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:puF0B/2TjvBup59eHmMTZmUsA9I= 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:puF0B/2TjvBup59eHmMTZmUsA9I='} 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e744b68a834962e4-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.298 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e62de0588313e152-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj41 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj41 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj41 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.300 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj41 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:N0wl1SgdvT3zXsZkgxXGZclsf0Y= 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:N0wl1SgdvT3zXsZkgxXGZclsf0Y='} 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aebd55b9e59bb631-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.302 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.303 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.303 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4fd194eb51f329d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj42 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj42 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj42 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.305 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj42 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PFsLqXTnUD/LE2UjAA7RShd0L7g= 2024-04-16T21:03:48.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PFsLqXTnUD/LE2UjAA7RShd0L7g='} 2024-04-16T21:03:48.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed363655cf7d7c50-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.310 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.310 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000223cbc4ef9eb824-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj43 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj43 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj43 2024-04-16T21:03:48.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.314 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj43 2024-04-16T21:03:48.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.315 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:euBqNB0KQH5NX3PJnIGejZ+5awM= 2024-04-16T21:03:48.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:euBqNB0KQH5NX3PJnIGejZ+5awM='} 2024-04-16T21:03:48.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2804274db182b2b-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.317 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.318 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9be922075e04ebb-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj44 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj44 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj44 2024-04-16T21:03:48.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.321 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.322 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj44 2024-04-16T21:03:48.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.322 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cf4ZP//V6evi9GrBrh5RFJu1Jws= 2024-04-16T21:03:48.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cf4ZP//V6evi9GrBrh5RFJu1Jws='} 2024-04-16T21:03:48.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9928da302da153e-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.324 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.325 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.325 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.325 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.325 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.326 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d32d1c65a0958ef0-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj45 2024-04-16T21:03:48.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj45 2024-04-16T21:03:48.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj45 2024-04-16T21:03:48.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.328 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.329 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.329 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.329 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj45 2024-04-16T21:03:48.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.329 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fkoqEq+kfCMMxJAZsIKbY1ef7AU= 2024-04-16T21:03:48.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fkoqEq+kfCMMxJAZsIKbY1ef7AU='} 2024-04-16T21:03:48.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d5b412eeebdad69-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.333 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2809a99684d07c0-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj46 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj46 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj46 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj46 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:loYLKDsZExFIYNmSoWebanfBbEk= 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:loYLKDsZExFIYNmSoWebanfBbEk='} 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c2d14c816acb523-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.336 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007388e397720a0f0b-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj47 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj47 2024-04-16T21:03:48.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj47 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.338 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj47 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5XeqYK0zi9zIYX+3Q5bYnRWz+RM= 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5XeqYK0zi9zIYX+3Q5bYnRWz+RM='} 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c54abf7947999ed5-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.340 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cce0f1b3d4b85cac-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj48 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj48 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj48 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.341 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj48 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:n80ChO8YNo9wELOlUoltOSklkwA= 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:n80ChO8YNo9wELOlUoltOSklkwA='} 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000480a27f80564c62d-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/ 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/ 2024-04-16T21:03:48.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/ 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.343 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.391 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.391 INFO:tasks.rgw_multisite_tests:/yrjefh-11/ 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks= 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jOqDvgQh26PUQ+nruLSNwunc2Ks=', 'Content-Length': '0'} 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dde1586c736688f1-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-11/obj49 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-11/obj49 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-11/obj49 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:/yrjefh-11/obj49 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8FlPUC540V6egdA26pIuWbpfXd0= 2024-04-16T21:03:48.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8FlPUC540V6egdA26pIuWbpfXd0='} 2024-04-16T21:03:48.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c18cc138f9931ad9-00661ed0fb-4361-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f79c09b56c167f15-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj1 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj1 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj1 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj1 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lB+yq2t9zQVZnELqxhfJFFc5y6U= 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lB+yq2t9zQVZnELqxhfJFFc5y6U='} 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b4dfae00c177d1-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008265a3ac3424a503-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj2 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj2 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj2 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj2 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SUzZ0fjH57HvhI0INJpiD3ER/Q8= 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SUzZ0fjH57HvhI0INJpiD3ER/Q8='} 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041e93ee1e89adbe2-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040acc17718f11a55-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj3 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj3 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj3 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.403 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj3 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E/8uE9xBtKBtFBRuAYi5VKg9dVw= 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/8uE9xBtKBtFBRuAYi5VKg9dVw='} 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec6792dfd61cd960-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c636a5f742cf30e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj4 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj4 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj4 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.406 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj4 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DVwvg1JsenrBFtxiY+pXOP2+7RQ= 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DVwvg1JsenrBFtxiY+pXOP2+7RQ='} 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000426d47799f2b205a-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5826dd855eca384-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj5 2024-04-16T21:03:48.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj5 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj5 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.409 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj5 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:UeVPE5DCpR56lTJR2bfTOdAymDg= 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:UeVPE5DCpR56lTJR2bfTOdAymDg='} 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9f055347b15eeb6-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da6fa241c4501da0-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj6 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj6 2024-04-16T21:03:48.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj6 2024-04-16T21:03:48.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.414 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj6 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jovyVKCGe6JvCwDPkKQPxLDINvo= 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jovyVKCGe6JvCwDPkKQPxLDINvo='} 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007357c4f1c1288282-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005894ded2c90fc373-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj7 2024-04-16T21:03:48.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj7 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj7 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj7 2024-04-16T21:03:48.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7aaBFKitnMndt1jhXFmXd4czAMs= 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7aaBFKitnMndt1jhXFmXd4czAMs='} 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b098a71d29ac8bf-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.422 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076cb5c42e700b676-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj8 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj8 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj8 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.423 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj8 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qENy4y2z3HUYcko3eDZWdDZPIds= 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qENy4y2z3HUYcko3eDZWdDZPIds='} 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d14b12e3ea618913-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d45ff30b7d1e8e27-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj9 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj9 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj9 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.426 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.428 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.428 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.429 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.429 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj9 2024-04-16T21:03:48.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.429 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:k1FWQCY99lexn42p27izoe8IghM= 2024-04-16T21:03:48.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:k1FWQCY99lexn42p27izoe8IghM='} 2024-04-16T21:03:48.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c624fb0133256d05-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.431 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000931977e10db72c77-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj10 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj10 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj10 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj10 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:sFb6zUI1p+tlhWygMxsjlly/7YY= 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFb6zUI1p+tlhWygMxsjlly/7YY='} 2024-04-16T21:03:48.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a14b57efadc9ac5d-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.434 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081243e58e0639f16-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj11 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj11 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj11 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj11 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xmWms5aT00S79zylPmZxnEHRUyc= 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xmWms5aT00S79zylPmZxnEHRUyc='} 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d417d2a9ffb34d2e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.437 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51a0781cb60ca98-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj12 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj12 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj12 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj12 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:YcokwFF7/PmXUgWJekSBND080tg= 2024-04-16T21:03:48.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YcokwFF7/PmXUgWJekSBND080tg='} 2024-04-16T21:03:48.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013863407d16b77fd-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.448 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e5573a89ee35b07-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj13 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj13 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj13 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.449 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj13 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:B5m5P6bipZkUFb5gh700QNFLDEA= 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:B5m5P6bipZkUFb5gh700QNFLDEA='} 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1c726728c411c43-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e6b4a78521fe33e-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj14 2024-04-16T21:03:48.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj14 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj14 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.452 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj14 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lMI7AyDbnGGY5AU2Sqdya5/zvg4= 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lMI7AyDbnGGY5AU2Sqdya5/zvg4='} 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c173659f785f13e0-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e63db72aa2957016-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj15 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj15 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj15 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj15 2024-04-16T21:03:48.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s/6+1wokHKqdt6+nU+3TOmVZVp4= 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s/6+1wokHKqdt6+nU+3TOmVZVp4='} 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dad90c8cd42d38ba-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfd34cd1d36720c7-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj16 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj16 2024-04-16T21:03:48.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj16 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj16 2024-04-16T21:03:48.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mM5ipUgKWYgPYF1+awOOx4UcQRc= 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mM5ipUgKWYgPYF1+awOOx4UcQRc='} 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e39a36a1b49d179-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b616d8d453b7e53-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj17 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj17 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj17 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj17 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.463 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J2UWDR4meVeX43/kNPdDUiXfFq4= 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J2UWDR4meVeX43/kNPdDUiXfFq4='} 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005321fc056bcb05f5-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68= 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TpqzZqj+yb84VCLZlUbx/vTRS68=', 'Content-Length': '0'} 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072f1eb38054d6455-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj18 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj18 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj18 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:51 GMT 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj18 2024-04-16T21:03:48.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:bdBL9gaVRT6ULqrEDrLyNLkvTms= 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:51 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bdBL9gaVRT6ULqrEDrLyNLkvTms='} 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de4fd4801cd4a32b-00661ed0fb-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d6349272d2476d7-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj19 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj19 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj19 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj19 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IjPDJVYeU0tgHtBrsuWlLkQSqJU= 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IjPDJVYeU0tgHtBrsuWlLkQSqJU='} 2024-04-16T21:03:48.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfd09137cc9d07d7-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.470 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9b4bea1abcfd319-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj20 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj20 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj20 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj20 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:g9/VL4U9q+2m4YtXNw6CBeBt15E= 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:g9/VL4U9q+2m4YtXNw6CBeBt15E='} 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000452938e616097205-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.473 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016cd98fb4cb7443d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj21 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj21 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj21 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj21 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jMCT0B0iFmgMCFTR29sALeVU37c= 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jMCT0B0iFmgMCFTR29sALeVU37c='} 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028ee90c5e90bd884-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.476 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bfec42b76085c08-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj22 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj22 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj22 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj22 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5dMhF6/lo+vsuXbHYv9cC6Et37I= 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5dMhF6/lo+vsuXbHYv9cC6Et37I='} 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3424ebb9f786ff1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.479 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.481 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.481 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.481 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.481 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5836c89b88bcbdc-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj23 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj23 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj23 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.482 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj23 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DRJAYwpmkO18sdmw3NuQkmhEsXk= 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRJAYwpmkO18sdmw3NuQkmhEsXk='} 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b3f43e24030f2e-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.484 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099cabeb1b4329ebf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj24 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj24 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj24 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.485 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj24 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:An4uizaRtX4pCajGku5HNU1bvcQ= 2024-04-16T21:03:48.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:An4uizaRtX4pCajGku5HNU1bvcQ='} 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000603f91a159e72681-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3842f21c878c59-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj25 2024-04-16T21:03:48.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj25 2024-04-16T21:03:48.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj25 2024-04-16T21:03:48.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj25 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jtc6641C8fejLdrHzNc62ln5Des= 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jtc6641C8fejLdrHzNc62ln5Des='} 2024-04-16T21:03:48.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c3c35e158fb4c7a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.493 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077257a2d9e5c5c1b-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj26 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj26 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj26 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj26 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5FWzFDQtKOIMNAdIs4NKv0wAckk= 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5FWzFDQtKOIMNAdIs4NKv0wAckk='} 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095d90e3d6309fb86-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007695a81adc4a2aaf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj27 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj27 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj27 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.497 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj27 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:JQfb71mnKv+A5Z0SyXSoFX00sf8= 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JQfb71mnKv+A5Z0SyXSoFX00sf8='} 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005215f1c8f515aa6-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc5605e835e72134-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj28 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj28 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj28 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.500 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj28 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fzHe+mbw4gYMSPvP73L1Lr5EenE= 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fzHe+mbw4gYMSPvP73L1Lr5EenE='} 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016bd362be56a85b1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6978081ae912898-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj29 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj29 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj29 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.503 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj29 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5vPnXkN6lZf6PdRPFjSEDfM2zC4= 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5vPnXkN6lZf6PdRPFjSEDfM2zC4='} 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b6d20662521ea0a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaa5358d5e2ca639-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj30 2024-04-16T21:03:48.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj30 2024-04-16T21:03:48.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj30 2024-04-16T21:03:48.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.514 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj30 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Feo1CcHsteWU4/JaPdKTHbxfhXQ= 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Feo1CcHsteWU4/JaPdKTHbxfhXQ='} 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008713638177885e07-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ac4691d1e918cab-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj31 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj31 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj31 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.517 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj31 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:EdGWjQXq9U8sh5AyLAXC1Uz+0no= 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EdGWjQXq9U8sh5AyLAXC1Uz+0no='} 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003818d81cedc712d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad4bc6a9de932a16-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj32 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj32 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj32 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj32 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SIzMT6xK0jddiqA0ko7tAE4W8EI= 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SIzMT6xK0jddiqA0ko7tAE4W8EI='} 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001409a20bb8b551ac-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb61bc932dc3d120-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj33 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj33 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj33 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.523 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj33 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WqLR/CJu80dd6qN78/xlaYTEA5k= 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WqLR/CJu80dd6qN78/xlaYTEA5k='} 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003aab63a42988ba87-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e847e4c9f1badf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj34 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj34 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj34 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.526 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj34 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gPqlc4xa3BlKfnJ00O9ByV2sYxA= 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gPqlc4xa3BlKfnJ00O9ByV2sYxA='} 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dce1ee357641af4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003444405d08aaeaca-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj35 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj35 2024-04-16T21:03:48.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj35 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.529 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj35 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:beFTrp4oKRZbGex5TbiPvVl1ax4= 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:beFTrp4oKRZbGex5TbiPvVl1ax4='} 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c34d2f3f73807fac-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efab9757960f4dc4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj36 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj36 2024-04-16T21:03:48.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj36 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.532 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj36 2024-04-16T21:03:48.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.533 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RkLk/GKnoEKM9An9hIMwq1wRrF4= 2024-04-16T21:03:48.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkLk/GKnoEKM9An9hIMwq1wRrF4='} 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1821f16e58884ea-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.535 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adbb1333f2e9e86c-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj37 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj37 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj37 2024-04-16T21:03:48.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj37 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:43ZBrRwTvT6SG6EdWhMfFxICVRc= 2024-04-16T21:03:48.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:43ZBrRwTvT6SG6EdWhMfFxICVRc='} 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012ef11fccf4fff34-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.538 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fcd4a743e7392a2-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj38 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj38 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj38 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj38 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:JTMVngw6lMAXGH4eUxax3WPgxUQ= 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JTMVngw6lMAXGH4eUxax3WPgxUQ='} 2024-04-16T21:03:48.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000840e028ed75dfbaf-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.541 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000449c45e94cf3bd52-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj39 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj39 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj39 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.543 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.543 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj39 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0qGH+mzCMA5Bgbp9ddQuYn+FZag= 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0qGH+mzCMA5Bgbp9ddQuYn+FZag='} 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d8330378999aca5-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dfce296130ef09a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj40 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj40 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj40 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj40 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8JE8ZuD5QsoFScnskmTOZ0Nfgs0= 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8JE8ZuD5QsoFScnskmTOZ0Nfgs0='} 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad75e194f685332f-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.549 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a0608047bf6684-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj41 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj41 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj41 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.550 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj41 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cmBkZaOzdVotsKbaxS6ndPRXbuo= 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cmBkZaOzdVotsKbaxS6ndPRXbuo='} 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b1b0a4c0c162b02-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047bba4c9c1920eeb-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj42 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj42 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj42 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.553 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj42 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:bUvo9sE7KkNsI++IpRBvx5lOIxs= 2024-04-16T21:03:48.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:bUvo9sE7KkNsI++IpRBvx5lOIxs='} 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c766a4b5a9f7d9fc-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.556 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b160af09bf9e0b1-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj43 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj43 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj43 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj43 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:X5TmOjQAsYw7c6nElH+Rcy0G8mY= 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:X5TmOjQAsYw7c6nElH+Rcy0G8mY='} 2024-04-16T21:03:48.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f611e44dc903b716-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.559 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000423b95d1a2e4cd97-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj44 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj44 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj44 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj44 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XtmSlbkW3IiN+lblprroRIwZ/bM= 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtmSlbkW3IiN+lblprroRIwZ/bM='} 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db8783374d54c636-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.562 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063b2b65ac4fab42a-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj45 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj45 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj45 2024-04-16T21:03:48.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj45 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:uFM1QVVUPkENMgwrYamRxWK/DIQ= 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uFM1QVVUPkENMgwrYamRxWK/DIQ='} 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000526bbf3175fcf3f2-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005058262dd505ec4e-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj46 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj46 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj46 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.569 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj46 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:w3+sdGwv6oHf8tZ1uWd5BIL5kN8= 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w3+sdGwv6oHf8tZ1uWd5BIL5kN8='} 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f202fb61fb839ab4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790705ef94130a1d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj47 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj47 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj47 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.572 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj47 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1d6unHf/rew/15rU/kT0zplb5W4= 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1d6unHf/rew/15rU/kT0zplb5W4='} 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000072dcba10270888d-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.576 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005092bd3be141c096-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj48 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj48 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj48 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.578 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj48 2024-04-16T21:03:48.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.579 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wglv1teydWA1vQhO4XVCRq5+6Fc= 2024-04-16T21:03:48.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wglv1teydWA1vQhO4XVCRq5+6Fc='} 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000209d47a0e91e7aa4-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/ 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/ 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/ 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:/yrjefh-12/ 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw= 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5d5SXZf7CgWBqnvCsuFRp1X5htw=', 'Content-Length': '0'} 2024-04-16T21:03:48.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000769ff5e825148efb-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-12/obj49 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-12/obj49 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-12/obj49 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.583 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:26:52 GMT 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:/yrjefh-12/obj49 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dmUHahZl++QVE+NwQ1Gk+RMKHIA= 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:26:52 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dmUHahZl++QVE+NwQ1Gk+RMKHIA='} 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000596e61dd55fe04db-00661ed0fc-4339-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:26:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713295540.367942_20.1","last_update":"2024-04-16T19:25:40.367942Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1713295583.603379_140.1","last_update":"2024-04-16T19:26:23.603379Z"},{"marker":"1_1713295610.154792_56.1","last_update":"2024-04-16T19:26:50.154792Z"}] 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:26:53.119+0000 7efeb2a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:26:53.121+0000 7efeb2a44800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295610.154792_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:50.154792Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:53Z"} 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713295540.367942_20.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:25:40.367942Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713295583.603379_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:23.603379Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713295610.154792_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:26:50.154792Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:26:53Z"} 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713295540.367942_20.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:25:40.367942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713295583.603379_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:23.603379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713295610.154792_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:26:50.154792Z', 'realm_epoch': 2}}] 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713295540.367942_20.1', 1: '', 2: '1_1713295583.603379_140.1', 3: '1_1713295610.154792_56.1'} 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-11', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-11', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-11 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:26:53.255+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:26:58.256+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:03.258+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:08.258+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:13.260+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:18.260+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:23.261+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:28.262+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:33.263+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:38.264+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:43.265+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:48.267+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:53.267+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:27:58.268+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:03.269+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:08.270+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:13.271+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:18.272+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:23.273+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:28.274+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:33.275+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:38.275+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:43.276+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:48.277+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:53.278+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:28:58.279+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:03.280+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:08.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:13.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:18.282+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:23.284+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:28.285+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:33.286+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:38.287+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:43.288+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:48.289+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:53.290+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:29:58.290+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:03.292+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:08.293+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:13.294+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:18.295+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:23.296+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:28.296+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:33.297+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:38.298+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:43.299+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:48.300+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:53.301+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:30:58.302+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:03.303+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:08.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:13.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:18.305+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:23.307+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:28.308+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:33.309+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:38.310+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:43.311+0000 7f9d44424800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:48.312+0000 7f9d44424800 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:48.312+0000 7f9d44424800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:31:48.312+0000 7f9d44424800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:48.592 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.595 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:48.595 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-16T21:03:48.595 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 1998, in test_bucket_reshard_index_log_trim 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:48.811 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yrjefh-50 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.812 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0D7KBXfPn9Efi4zAWcMj0t7gzF8= 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0D7KBXfPn9Efi4zAWcMj0t7gzF8=', 'Content-Length': '0'} 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b26772a881962264-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T21:03:48.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f83694e3ae0b2b3-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OawIn4ucckKNFHeCdrJon1ciU+Q= 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OawIn4ucckKNFHeCdrJon1ciU+Q='} 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000476b57e09300983f-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T21:03:48.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b07081aa92ee8a6-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.818 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tW/jLtJqfJ/5LMBEwax+H6Uj+mE= 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tW/jLtJqfJ/5LMBEwax+H6Uj+mE='} 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea1a48a51f046db-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T21:03:48.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006745302b41a79f99-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.821 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T2JmAG52tRnr2/Yhg7thY7GkxE4= 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T2JmAG52tRnr2/Yhg7thY7GkxE4='} 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e2e0bf6a96fef8a-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI= 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qdtKUGoF8nQyCpqVffMgglGZNMI=', 'Content-Length': '0'} 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b7eccb6f585757-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:48.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:27 GMT 2024-04-16T21:03:48.824 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ovhsf3NSedHasPFFrVL6oe/vPxE= 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:27 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ovhsf3NSedHasPFFrVL6oe/vPxE='} 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3fbb02a31ea2620-00661ed773-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713297267.375662_676.1","last_update":"2024-04-16T19:54:27.375662Z"}] 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T21:03:48.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:27.728+0000 7f7cac42e800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:27.730+0000 7f7cac42e800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:48.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297267.375662_676.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:27.375662Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T21:03:48.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713297267.375662_676.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:27.375662Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T19:54:27Z"} 2024-04-16T21:03:48.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713297267.375662_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:27.375662Z', 'realm_epoch': 2}}] 2024-04-16T21:03:48.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T21:03:48.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713297267.375662_676.1'} 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:27.870+0000 7f3ab7f49800 1 waiting to reach incremental sync.. 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:32.872+0000 7f3ab7f49800 1 bucket sync caught up with source: 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, 00000000001.33.6, , , , 00000000001.22.6, , , 00000000001.65.6, , 00000000001.30.6] 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, 00000000001.33.6, , , , 00000000001.22.6, , , 00000000001.65.6, , 00000000001.30.6] 2024-04-16T21:03:48.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:32.872+0000 7f3ab7f49800 0 bucket checkpoint complete 2024-04-16T21:03:48.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T21:03:48.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:48.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:54:33.022+0000 7f6ab8c33800 1 bucket sync caught up with source: 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, 00000000001.158.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000001.101.6] 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, 00000000001.158.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000001.101.6] 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:54:33.022+0000 7f6ab8c33800 0 bucket checkpoint complete 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.879 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo= 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo=', 'Content-Length': '0'} 2024-04-16T21:03:48.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8421e9bd8ca1b68-00661ed779-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo= 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:j/TA2LwhdKxSml1qGDUPBXB0rEo=', 'Content-Length': '0'} 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd921ecc120362d9-00661ed779-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.905 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e06f0e23b47f69c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.907 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d663544bccdcae5d-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.909 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78d5b9d44c08b6f-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.911 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps= 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BCL9QvOs0sgSDHACRcSFELeszps=', 'Content-Length': '0'} 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b92fe7790b50290-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.914 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T21:03:48.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T21:03:48.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056950200471a350d-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da628d660edb81fa-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.918 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME= 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME=', 'Content-Length': '0'} 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c486108ae93b019a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME= 2024-04-16T21:03:48.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:MG3z4s81S3LWd7LfamYMeVWxXME=', 'Content-Length': '0'} 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b38e33f5abd8934-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc21e34e595afb8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.923 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk= 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ezr/bvAOLMdOs5chdGRdsIeLNVk=', 'Content-Length': '0'} 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601812f818ac0e7f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d7d766319d9311a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d678d331d64ebe-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:48.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc= 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc=', 'Content-Length': '0'} 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4a93730105db4e-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc= 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OMvxmUJMk243YqpCyxv+W4yzpDc=', 'Content-Length': '0'} 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffd10d7fec5a3f89-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:48.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.931 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w= 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w=', 'Content-Length': '0'} 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921c4a99365c3ca3-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w= 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vqEng5q/f9Uu17+krWJt66Vc06w=', 'Content-Length': '0'} 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdc500994a5a6bd-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.934 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023969a605a8bfea-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003174bada875c88af-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574c663ad4671e32-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.946 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T21:03:48.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004924b8f7ef45b14f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A= 2024-04-16T21:03:48.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A=', 'Content-Length': '0'} 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8b22d7490ed44b-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A= 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A/WQazA3JqYm7c5LT/Q4leLFH0A=', 'Content-Length': '0'} 2024-04-16T21:03:48.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca42f12cb8ec0b5-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004851f5885856e216-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.966 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA= 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J6yVdMzyDs8fWbUwvMNByOADlgA=', 'Content-Length': '0'} 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9835b59b7b9ef6f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000322352c69192794-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.969 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f98bc2b8a9f52a-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:48.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.971 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo= 2024-04-16T21:03:48.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo=', 'Content-Length': '0'} 2024-04-16T21:03:48.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcfa702747df718-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo= 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2KC58mSS0ShHcsLnv8BhcoNErbo=', 'Content-Length': '0'} 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9aaacebe1d68209-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.995 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4= 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4=', 'Content-Length': '0'} 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099987ec66ef6a1f8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4= 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p4kXVlOL7YUJua9TK9n8G0CL0V4=', 'Content-Length': '0'} 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008432ce83365f0e02-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.998 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051af450a4cffaeeb-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:48.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019fb149afe21f0f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5c737781d1f2f8-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.009 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd796c95e7e24bc-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.011 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg= 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg=', 'Content-Length': '0'} 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005750d9b81cc97758-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.013 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg= 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GJB2ZtJOuuF0P/Ia07O8fubD1Eg=', 'Content-Length': '0'} 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099006f6e732829ec-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001834e2fe0f71f1e6-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.016 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8= 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ16gQq4Z4BlImmPeUn0iQbh1h8=', 'Content-Length': '0'} 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6cdd95aadce5c7-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c4cf763b0d92497-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aed17b9fbaf63cf-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.021 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE= 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE=', 'Content-Length': '0'} 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035204177dcd40e7a-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE= 2024-04-16T21:03:49.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yN0crDQnmxTbVWimJ/xmTMg39jE=', 'Content-Length': '0'} 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8875cf9fbb78852-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY= 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY=', 'Content-Length': '0'} 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8794e8546417b71-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.026 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY= 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DXUpBDPIhellEZtIw6wLICvebdY=', 'Content-Length': '0'} 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008902efea9ba0b15d-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c5650345c37595-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0e84e41635d187-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.031 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3717e3532674557-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T21:03:49.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662191c5b34de497-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs= 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs=', 'Content-Length': '0'} 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e534b3f43a012c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.036 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs= 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITg5tS38k2SeNCXXZ5iwlZ0dZqs=', 'Content-Length': '0'} 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a660dc23abf1e5-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.038 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970aab111ea18c62-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc= 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:faY7VQCbskk4FQsUC0DGZUZgkgc=', 'Content-Length': '0'} 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ee202a274b83ba-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.042 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4283f584c0fb121-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.045 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f034391ea856e6d6-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.048 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g= 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g=', 'Content-Length': '0'} 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d9acabd24bbf1c-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.050 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g= 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:A1EKzrBO6L8IRSHcAe10k4HEV5g=', 'Content-Length': '0'} 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6c210b002c8bcf-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII= 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII=', 'Content-Length': '0'} 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba85475a1e43de9-00661ed779-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.055 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII= 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xOjkbpvj0Kq3afzj7tw8dmUtcII=', 'Content-Length': '0'} 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef9397d1c93013-00661ed779-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a98dc64e8ffb43-00661ed779-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:33 GMT 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw= 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:33 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zf/L5iQoju/qL/IvbpMbSlsooUw=', 'Content-Length': '0'} 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be18496560aab6f-00661ed779-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"layout":{"resharding":"None","current_index":{"gen":0,"layout":{"type":"Normal","normal":{"num_shards":11,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}}]}} 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-50 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"op_id":"1#00000000001.33.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5146303621251504191","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.385768586Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"5#00000000001.22.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.16427917761036559904","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.456540920Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"8#00000000001.65.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.5633453337919614858","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.444947649Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]},{"op_id":"10#00000000001.30.6","op_tag":"b482a2a9-ef36-488b-9666-ada656453328.4812.18348127270871254747","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2024-04-16T19:54:27.397899049Z","ver":{"pool":9,"epoch":0},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19"}]}] 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-50 --num-shards 13 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-50 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 4 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"layout":{"resharding":"None","current_index":{"gen":1,"layout":{"type":"Normal","normal":{"num_shards":13,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}}]}} 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'yrjefh-50', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket yrjefh-50 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [] 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d4421f1d3120c37-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.069 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TYHKDcCS6fzNYecgHITUYmxYSvI= 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TYHKDcCS6fzNYecgHITUYmxYSvI='} 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005455716a19bd63bd-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a967f7b72bdcb97-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.072 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:p18Mzx5ZJbm933HXs+AubNP5XS8= 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:p18Mzx5ZJbm933HXs+AubNP5XS8='} 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbd1f199b95578f4-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b26f40b782495fc2-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lWRjg0u+SZY78sdpNz9scQ6S11o= 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lWRjg0u+SZY78sdpNz9scQ6S11o='} 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cc1bb4b52d8150f-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE= 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pBvt01bMA/MQ2faYrY6fZjf6XrE=', 'Content-Length': '0'} 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083dc07e2b5a581c4-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:54:34 GMT 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0gaMaRTjqfdC2bAXet/zNj/UEpM= 2024-04-16T21:03:49.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 19:54:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0gaMaRTjqfdC2bAXet/zNj/UEpM='} 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b57ce3935e914c2-00661ed77a-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:34.684+0000 7f768e52d800 1 waiting to reach latest gen 1, on 0.. 2024-04-16T21:03:49.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:39.686+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:44.688+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:49.688+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:54.690+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:54:59.692+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:04.693+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:09.695+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:14.696+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:19.698+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:24.699+0000 7f768e52d800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , 00000000002.28.6, , , , 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:29.700+0000 7f768e52d800 1 bucket sync caught up with source: 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , 00000000001.46.6, , , , , 00000000002.28.6, , , 00000000001.48.6, 00000000001.44.6, ] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:29.700+0000 7f768e52d800 0 bucket checkpoint complete 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:55:29.854+0000 7f30cec13800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, 00000000002.162.6, , , , 00000000001.141.6, , , 00000000001.100.6, , 00000000002.105.6] 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:55:34.856+0000 7f30cec13800 1 bucket sync caught up with source: 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:55:34.856+0000 7f30cec13800 0 bucket checkpoint complete 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.084 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w= 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w=', 'Content-Length': '0'} 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074164ea878b4b586-00661ed7b6-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w= 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:uqjw65v72vKLC0OwpQ/uDgUxI+w=', 'Content-Length': '0'} 2024-04-16T21:03:49.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000144b041dc0cae123-00661ed7b6-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a3c11d972812a95-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.090 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a8c0fc258a0efe-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-16T21:03:49.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec25bf2793be3c47-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.094 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE= 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RajLscnQJpBmx+euXSkWpBBZ5fE=', 'Content-Length': '0'} 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce490b0701c5291b-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T21:03:49.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e809398d4b2189e6-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.097 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b4c1e7fb7d2fbb-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.099 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo= 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo=', 'Content-Length': '0'} 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d56ef0e5777e236-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo= 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AIRIcjU+3D8QMAJJmreHBvYJPQo=', 'Content-Length': '0'} 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976c6446bca7b863-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.102 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772564f8137bda24-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M= 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fYyp2mrrYxaD0dnu7nbhy1f/D6M=', 'Content-Length': '0'} 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bb29023e1607477-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.105 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM= 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024aedd1d15b26a4e-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM= 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nxr2KAiqO2FULiXHGVxEdMjUDiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ca1fe7f2c0f2e6-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g= 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g=', 'Content-Length': '0'} 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73e797fb03f0c46-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g= 2024-04-16T21:03:49.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IqVb+9GV2M3q3IxjBBtNsr14E9g=', 'Content-Length': '0'} 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d524103f90c09bf8-00661ed7b6-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.111 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:34 GMT 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xZFqUTNvuhohpx+ys6fWhjZs3rU= 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:34 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZFqUTNvuhohpx+ys6fWhjZs3rU=', 'Content-Length': '0'} 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d7811c235d4e27-00661ed7b6-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Gprj+si39b6GW6GUi75CPi7oCac= 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Gprj+si39b6GW6GUi75CPi7oCac=', 'Content-Length': '0'} 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84acf5c0dbeced1-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.114 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40746e6436537fb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.116 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd1db27b7cf30c6-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a13cc520b709b1-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T21:03:49.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d00af871c072e32-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.121 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc= 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc=', 'Content-Length': '0'} 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9b2968210d3f55-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc= 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v3I8LRRzhIsh9fPRIOywG1yNHNc=', 'Content-Length': '0'} 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b232ebcb1163cda7-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.127 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d73298c319a7cd-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY= 2024-04-16T21:03:49.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/yVbN7EkZuWwoPD8ulTzVICSfxY=', 'Content-Length': '0'} 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50cabf251a9927f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3147e04a080b81-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b1f093b58bfe14-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ= 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001260be41d1531994-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.135 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ= 2024-04-16T21:03:49.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r3ksCZcjVp0yEonEuynrrUeSGYQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1987190385a868-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.138 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU= 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU=', 'Content-Length': '0'} 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093dc13705b92933a-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU= 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tMxm5f+Mny0RS5O5pNJngrPD7eU=', 'Content-Length': '0'} 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ef0eb75fa5a5ef-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.142 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9368bbdf3496a17-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f805452a640f615-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T21:03:49.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2367d7f46145a7-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.147 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f00af875007824e-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg= 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg=', 'Content-Length': '0'} 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbbca1dc4b72f77-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg= 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1y+jxFvJhfC5tSqYdz+xnQfFBtg=', 'Content-Length': '0'} 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b44ee43480d636-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.151 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.152 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987981d9c0947ef5-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY= 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8sbYCt4I16FdUVNg1kAb9QouZNY=', 'Content-Length': '0'} 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7acfba19cc72ac6-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c35067489ba158-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.158 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058974238029d2da7-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw= 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw=', 'Content-Length': '0'} 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e054e0ce47a9dcb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.161 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw= 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pd1XaMKf6Ms2jsmUEvjJnH6kkjw=', 'Content-Length': '0'} 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea737cd6de6b82e-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.163 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII= 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII=', 'Content-Length': '0'} 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f75ebb80fb705c2-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII= 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pmxahnAn+iIpH2N5y0c2DfFmOII=', 'Content-Length': '0'} 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2190939aaf912d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.166 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bfd65c2a2523fab-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b49d0f1a9429998-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T21:03:49.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed6d8b99b05ab4d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e7dbc778805f34-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.173 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU= 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU=', 'Content-Length': '0'} 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9a3c72bb8721ea-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.175 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU= 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Z6jDus67kUxl+E4Togq4Pvw9YtU=', 'Content-Length': '0'} 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9625b40cb45fd4-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T21:03:49.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8293d94f69d7ec-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM= 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/MW7Uya+OkhFX/c74fmOZkcpdsM=', 'Content-Length': '0'} 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b400dc46790485f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d48e1ad62a50c6-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006611b1260ec557ec-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.184 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8= 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8=', 'Content-Length': '0'} 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef722758a13d087-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.185 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8= 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GlS+cJwlQkchSXAYsfcG6wsipM8=', 'Content-Length': '0'} 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181caffd2659f229-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.187 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA= 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA=', 'Content-Length': '0'} 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ef8dac20bfaa92-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA= 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1qmrA8ficaHsiE97ATiSBz3hvGA=', 'Content-Length': '0'} 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eca006814999a923-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.190 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007458b2e485306322-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013856cd9eb455ab0-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef471b9d0db319f4-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c4f45161f50961-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.197 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w= 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w=', 'Content-Length': '0'} 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c9165a5e938b43-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w= 2024-04-16T21:03:49.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BkQOLI1QAPXmcb+cxim74eowz0w=', 'Content-Length': '0'} 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073de2cc2af2236d0-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae97145747176811-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.202 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U= 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vcFXhJpH2XsrPuLjNlNYA9osA4U=', 'Content-Length': '0'} 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064dce6e5b203de8a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.204 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce6ba8fcc2ab174-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.206 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860747e5f7e7cf81-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg= 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg=', 'Content-Length': '0'} 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096670747b28458a1-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.209 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg= 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TR4smuQR2wcGFxGC3efRQEuOArg=', 'Content-Length': '0'} 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8b5969250bf49f-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U= 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U=', 'Content-Length': '0'} 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f1122927fd986d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.212 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U= 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ME7KiqKgE9oseVNzuSN8U0WWS1U=', 'Content-Length': '0'} 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855f0c01b0a09147-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.215 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4e6e89e53df938-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.219 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31df3e1fa78e06d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.221 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.223 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084990df426643815-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.227 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.227 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.227 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.227 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.227 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8b523b975dfa6a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.231 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.231 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.231 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.231 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.231 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I= 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I=', 'Content-Length': '0'} 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b253156a1018eb5-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.236 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I= 2024-04-16T21:03:49.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VwqbVjK3ZRMsVsEwKS9Hhbk7e4I=', 'Content-Length': '0'} 2024-04-16T21:03:49.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6cd00b45eb192a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.236 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.242 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.242 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d6b8d370a5ca58f-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.244 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM= 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1ayQz6FZ7Mif4v5ZHWJN//S/RiM=', 'Content-Length': '0'} 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004754eff8955ad718-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd8fb2ad5405c1d-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19c5d60bafe125a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.252 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs= 2024-04-16T21:03:49.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs=', 'Content-Length': '0'} 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c84f08a059f510-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs= 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HEvTWVrZB9SFjml7mPrYoODEEhs=', 'Content-Length': '0'} 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8e6bc5f655507a-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.259 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI= 2024-04-16T21:03:49.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI=', 'Content-Length': '0'} 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780fe4aed69f6afb-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.263 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.263 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.263 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.263 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI= 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxGAdiITZYZrEH+VwaJK30yBBLI=', 'Content-Length': '0'} 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699891c118b19958-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.267 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05cf3242b4d0a35-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.271 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4= 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E41J9zAn0BRGsIUXUdAPg9UfmN4=', 'Content-Length': '0'} 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004239dfbac23865ef-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.272 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.275 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1de076087f74707-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.278 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008306bf94d3feb05d-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8= 2024-04-16T21:03:49.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8=', 'Content-Length': '0'} 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb548b9b69e4c162-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8= 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oAiYvs70U6y7zp6SRT5U8vmqLr8=', 'Content-Length': '0'} 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c01809b3d5d4fe-00661ed7b7-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.284 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:35 GMT 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo= 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:35 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wVms6Nc4l3RiAu8m9DufPwvDdgo=', 'Content-Length': '0'} 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d216dadaec9cbc4a-00661ed7b7-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:poxmjJJ42bAZFatclaSVqMWWgsU= 2024-04-16T21:03:49.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:poxmjJJ42bAZFatclaSVqMWWgsU=', 'Content-Length': '0'} 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055ab237f56a7354-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720a804cd6b23a7a-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.289 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632cecd4a82ee0e5-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo= 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo=', 'Content-Length': '0'} 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f626b805f760125b-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo= 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jqMlesa9MeYiRmCo1+qFmd4b3jo=', 'Content-Length': '0'} 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b39e306fdf45e6-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.294 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI= 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI=', 'Content-Length': '0'} 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f895164625a63bc-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI= 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XfBBaxEhXFEUrRG4853LvFD/MzI=', 'Content-Length': '0'} 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ce2d2e56baf8c8-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.297 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a366c567d2bbbb36-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.299 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fabad3e61473c5cd-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fbf7d828a9c95c-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d1b971f688adcc-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A= 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A=', 'Content-Length': '0'} 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24591668187355c-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.306 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A= 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:YivzstyYCmrBMdpfJx+1VGZfp3A=', 'Content-Length': '0'} 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084af76aba47463b7-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.308 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070aaaba6e72dcf15-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.310 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY= 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+ddeBoq9e5HamRafvURzxvwKQJY=', 'Content-Length': '0'} 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003117eccfef1999c7-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81c01564ccb14aa-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e64b7b3db1e5d6a2-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco= 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco=', 'Content-Length': '0'} 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f179c7d520ef1b-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.316 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco= 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7hf6aloCrOfzbiKsnTml75yTaco=', 'Content-Length': '0'} 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b586293e1fe56d4e-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4= 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4=', 'Content-Length': '0'} 2024-04-16T21:03:49.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073db62cf40fc7d90-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.319 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4= 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ZpFacIXadDYKDi3LAvX7z7QBQL4=', 'Content-Length': '0'} 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a231b47a51bb4cd2-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380fdc21523b82fc-00661ed7b8-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.322 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:36 GMT 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc= 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:36 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rMTng3T/pfjJSb2coytlEXOA3mc=', 'Content-Length': '0'} 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031028ddff9575851-00661ed7b8-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'yrjefh-50', '--num-shards', '15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket yrjefh-50 --num-shards 15 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-50 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 8 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":0,"layout":{"type":"InIndex","in_index":{"gen":0,"layout":{"num_shards":11,"hash_type":"Mod"}}}},{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-50 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:36.795+0000 7f58b3f33800 1 waiting to reach latest gen 2, on 1.. 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:41.796+0000 7f58b3f33800 1 bucket sync caught up with empty source 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T19:55:41.796+0000 7f58b3f33800 0 bucket checkpoint complete 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-50 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-50', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-50 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:55:41.949+0000 7f2750016800 1 bucket sync caught up with source: 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, 00000000002.162.6, , , 00000000001.217.6, 00000000001.141.6, , , 00000000002.104.6, , 00000000002.105.6] 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T19:55:41.949+0000 7f2750016800 0 bucket checkpoint complete 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.326 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8= 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8=', 'Content-Length': '0'} 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af355f4675315ec9-00661ed7bd-5331-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/ 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8= 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jX61Kamk8Zaq9WxOJUj5BsSmKi8=', 'Content-Length': '0'} 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec1c3949b17cf011-00661ed7bd-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.329 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dff54893410da9b-00661ed7bd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:49.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998c13f910b89a04-00661ed7bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-16T21:03:49.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T21:03:49.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f95229fb1d5ad5c-00661ed7bd-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?versions&max-keys=999 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:41 GMT 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:/yrjefh-50/?versions 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y= 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:41 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xZ8RsBLkeephmmftZxk6UU3Vk+Y=', 'Content-Length': '0'} 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462fcc87bf7bc441-00661ed7bd-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50999falseanulltrue2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalbnulltrue2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalcnulltrue2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normaldnulltrue2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalenulltrue2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalfnulltrue2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalgnulltrue2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalhnulltrue2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.337 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a0f0e339cbd03d-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T21:03:49.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630edb97bcabae24-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.340 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q= 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q=', 'Content-Length': '0'} 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057cf18e45b9b8037-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.342 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q= 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:W7wRWTQTm/BJwcPFiEp7slkMA3Q=', 'Content-Length': '0'} 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261b0c005af62261-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093dfdf0dcddd0745-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a?versionId=null 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a?versionId=null 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM= 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DRYkHYYGicZccRpMmJPuTe8KZhM=', 'Content-Length': '0'} 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca459da217e90c1-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.347 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd2d46bd5a56ffb-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a2a2e70862b244-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.350 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E= 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E=', 'Content-Length': '0'} 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb795115e5cd081-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.352 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E= 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7yfZxY0a46AbNZKpJaDIFePHC/E=', 'Content-Length': '0'} 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e175a69ee048feb6-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU= 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU=', 'Content-Length': '0'} 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff67daf7f91d7b9-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/a 2024-04-16T21:03:49.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/a 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/a 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.355 INFO:tasks.rgw_multisite_tests:/yrjefh-50/a 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU= 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8AvhOC+HjdgoYz/ox3/dnFpAdgU=', 'Content-Length': '0'} 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d91fb433fc79296-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001311fb7e36e4c347-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=a 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=a 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.359 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083444184df440ffc-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50a1000falsea2024-04-16T19:54:27.385Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.361 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab70e066517be6e-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.363 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a00f92063e9e4c-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI= 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI=', 'Content-Length': '0'} 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0db546abba284f-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.367 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI= 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3ibWEKmFF5CWKi6TkgFMuf4thNI=', 'Content-Length': '0'} 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5158020d8ed520-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ef86ca813a6055-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b?versionId=null 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.370 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b?versionId=null 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8= 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xDuVSa7cYLdyyg/6YfOYOLeSdU8=', 'Content-Length': '0'} 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5adc3e4ef7c9d55-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983113bc9b4b91a3-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005796bcd374343013-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.375 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo= 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo=', 'Content-Length': '0'} 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac41e47502f6a6b3-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo= 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LshCAakDUWxHvLsiofaoidWQPlo=', 'Content-Length': '0'} 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606df98a267d6b0b-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.378 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA= 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA=', 'Content-Length': '0'} 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b4bcbd84d534c4-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/b 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/b 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/b 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:/yrjefh-50/b 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA= 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tAiY82cpcwvbyvFD0eR8JXHPXWA=', 'Content-Length': '0'} 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490c72b0fc4b0bb0-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.381 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea57f273be80f265-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=b 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=b 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.383 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b3b736c8b794f3-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50b1000falseb2024-04-16T19:54:27.397Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T21:03:49.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc33c082827bcf2b-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee4b0ba548bdcee-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.388 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.391 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.391 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.391 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.391 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA= 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA=', 'Content-Length': '0'} 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010aeabe9b47680c1-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA= 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NmjyGsBJh6CqVrEjs0ePbS/PMhA=', 'Content-Length': '0'} 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007338fd6ef82acd84-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.394 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e5df9a27de90354-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c?versionId=null 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.396 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c?versionId=null 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74= 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kFK4WNHUln6P7b9qsHbqROQcx74=', 'Content-Length': '0'} 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e07f37d5ffc9cb-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7472325a1ba99e-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.399 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002380b4791f8ebdb0-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg= 2024-04-16T21:03:49.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg=', 'Content-Length': '0'} 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fac2b6201fb471-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.402 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg= 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nH7hyL0KK+9zeK1OoHoeNU78uJg=', 'Content-Length': '0'} 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36fb24366858bca-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA= 2024-04-16T21:03:49.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA=', 'Content-Length': '0'} 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bd12d6674c971f-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/c 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/c 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/c 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.405 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:/yrjefh-50/c 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA= 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Dkcf9zCJiDvgWyONgPqOo2X/1XA=', 'Content-Length': '0'} 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d7048e057ead37-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.407 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007875f20eacf76924-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=c 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=c 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb71f28d1e11a4ed-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50c1000falsec2024-04-16T19:54:27.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T21:03:49.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a22f4dcb1a1087-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13755c0aae3a1d2-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.414 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI= 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI=', 'Content-Length': '0'} 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef78f5261e7693b-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI= 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6VIyj1dzp3hqC/bFX1Y1AL1FbeI=', 'Content-Length': '0'} 2024-04-16T21:03:49.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d638068c9842f9-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13ee5e26bd52490-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d?versionId=null 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.419 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d?versionId=null 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY= 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xtKIaR99LofvvldI/zP7vfkYMqY=', 'Content-Length': '0'} 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9072dc18896cbb-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b07d83ab41ecc0-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.422 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007699174c1f2e61dc-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8= 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8=', 'Content-Length': '0'} 2024-04-16T21:03:49.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003391c3de7f5e3202-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.425 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8= 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hJniwc3I21lNlAIrQk9saP9Hcu8=', 'Content-Length': '0'} 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047555504c92d3090-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho= 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho=', 'Content-Length': '0'} 2024-04-16T21:03:49.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9cd03cc3e42a1a-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/d 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/d 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/d 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.428 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:/yrjefh-50/d 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho= 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tFyDbWYZ5g5qxLY2dZpd+AGOlho=', 'Content-Length': '0'} 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abb316ee4696a56-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.430 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027527f3614f2133d-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=d 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=d 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.432 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee38d3e1f11a727-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50d1000falsed2024-04-16T19:54:27.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.434 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccbda6ee3b7c686-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73739dd9895cb3c-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI= 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI=', 'Content-Length': '0'} 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61958e4bbd3fb3c-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.439 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI= 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7+if5jdX/HW2mN4krmXsb864zsI=', 'Content-Length': '0'} 2024-04-16T21:03:49.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097826f6e8abc2864-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.443 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.446 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.446 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.446 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.446 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a43a83596a45954-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e?versionId=null 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.449 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e?versionId=null 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA= 2024-04-16T21:03:49.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaHPvt0/sY9v+n80iS57j/7n1bA=', 'Content-Length': '0'} 2024-04-16T21:03:49.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001386cf55a11104e4-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.453 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.454 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.454 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.454 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599c115043fa9548-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.457 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000378c1ab6402bd887-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0= 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0=', 'Content-Length': '0'} 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd11e4cd4f6eaca-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.461 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0= 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ClzUswmZaqV4bTaWqBjkf44uCk0=', 'Content-Length': '0'} 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bed10a5e20c2047-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.463 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw= 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw=', 'Content-Length': '0'} 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb3672a2fa8aafa-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/e 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/e 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/e 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.465 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.466 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.466 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.466 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.466 INFO:tasks.rgw_multisite_tests:/yrjefh-50/e 2024-04-16T21:03:49.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.467 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw= 2024-04-16T21:03:49.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3JB2Iq7QKx1juKEYad49TZQWYYw=', 'Content-Length': '0'} 2024-04-16T21:03:49.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef7e8c4efc9df33-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.470 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.470 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.470 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.470 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.470 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14562b4bf94f0f0-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=e 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=e 2024-04-16T21:03:49.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.474 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.474 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.474 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.474 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8= 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NboLXEb0C8JpSOzSVBVbW7c8Rj8=', 'Content-Length': '0'} 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f0786831235f88-00661ed7be-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50e1000falsee2024-04-16T19:54:34.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.476 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.478 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.478 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:42 GMT 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:iF/zG/rVkqChw0QMlNv0vV22/fc= 2024-04-16T21:03:49.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iF/zG/rVkqChw0QMlNv0vV22/fc=', 'Content-Length': '0'} 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2fbdf15fffc868-00661ed7be-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.481 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T21:03:49.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a960dbeb068b8ddb-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.484 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8= 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8=', 'Content-Length': '0'} 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2677f11c7f45983-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.486 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8= 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wjO1I3XxYScUqpyoFYuVNnyU6D8=', 'Content-Length': '0'} 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000499b2f0a2fef08fc-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.488 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d9973f5d4ca7f9-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f?versionId=null 2024-04-16T21:03:49.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.491 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.492 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.492 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f?versionId=null 2024-04-16T21:03:49.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.492 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8= 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3b/6Bt9cmqp/jh6KjrZryGMxg8=', 'Content-Length': '0'} 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000560b1b4a8e2e61e3-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007090b74a17a19518-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.498 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211068b1f451bd04-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg= 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg=', 'Content-Length': '0'} 2024-04-16T21:03:49.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b908f9211658b5-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.503 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg= 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y3HTRWPV1avySO6sRDzhPbHMgSg=', 'Content-Length': '0'} 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cbec0168b96f76-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.505 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg= 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg=', 'Content-Length': '0'} 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:54:57 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032abd57693e80f06-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/f 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/f 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/f 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:/yrjefh-50/f 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg= 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dp0YyUzCTJZMM7i8lUqo4i6gkJg=', 'Content-Length': '0'} 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af2f1691bc52f7a-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.508 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506055cb07c106b8-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=f 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=f 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.510 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b566720c5bfc245f-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50f1000falsef2024-04-16T19:54:34.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T21:03:49.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000993747ab13a9a121-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.513 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040147434fd61bcff-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.515 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ= 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488f7608e2e00aa0-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ= 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GBmKXd0uhyDGTorHVR0B55PjxQQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbc8b0343a313e9b-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.518 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477004d2f8daaddd-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g?versionId=null 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g?versionId=null 2024-04-16T21:03:49.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8= 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:U+L5IySTRApVrX9K3XeGu4HXjo8=', 'Content-Length': '0'} 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b50787ff2eda8d9-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.522 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa716cb44f6c440c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd840a17f97bf9b-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.525 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE= 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE=', 'Content-Length': '0'} 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052456cbb4eb6bbaf-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE= 2024-04-16T21:03:49.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TebWd7JOTKjYLD4pyUziNQcxcPE=', 'Content-Length': '0'} 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e9d204f411b611-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.529 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE= 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE=', 'Content-Length': '0'} 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce128e93105bfae-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/g 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/g 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/g 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:/yrjefh-50/g 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE= 2024-04-16T21:03:49.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2RtTKXuhm9zN3QYi5Ea06aDgfHE=', 'Content-Length': '0'} 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c03cd118b0999f1-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d26d53e2458a8f4e-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=g 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=g 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043798b6333773f6c-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50g1000falseg2024-04-16T19:54:34.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.535 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.536 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc0a2f3ad1256b3-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.538 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa33490d9138d92d-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ= 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e1c1261919018c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ= 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZbSgrBTRDA220RP9ytDu/HCuDQ=', 'Content-Length': '0'} 2024-04-16T21:03:49.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28a987b6819b877-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000581bd69136d9db88-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h?versionId=null 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h?versionId=null 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw= 2024-04-16T21:03:49.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kI7YMUVKEEhXDzTuu5aVfm+MPxw=', 'Content-Length': '0'} 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a20adbdc65acf5f-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.545 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd6f75b34b7b542-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.547 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a164585c003b5154-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0= 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0=', 'Content-Length': '0'} 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509e26a3bb7e2f0c-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0= 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5oYhTgBBWlh/1KRrr7zPQUhx0g0=', 'Content-Length': '0'} 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000572b3cbf5296eddd-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0= 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0=', 'Content-Length': '0'} 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-50:b482a2a9-ef36-488b-9666-ada656453328.4812.19'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 19:55:27 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277968b089ab404e-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/h 2024-04-16T21:03:49.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/h 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/h 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.553 INFO:tasks.rgw_multisite_tests:/yrjefh-50/h 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0= 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ciyDSojkm5KIWuYUah+ghNen9i0=', 'Content-Length': '0'} 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 19:54:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6dc1c6f9e30a332-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:49.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f267aaa31ff802-00661ed7bf-5331-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/ 2024-04-16T21:03:49.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/ 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-50/?prefix=h 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-50/?prefix=h 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 19:55:43 GMT 2024-04-16T21:03:49.556 INFO:tasks.rgw_multisite_tests:/yrjefh-50/ 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c= 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 19:55:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kR+2paqz8IRbEKDt6fSiEKx/74c=', 'Content-Length': '0'} 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771fcb84b457a5e1-00661ed7bf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 19:55:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-50h1000falseh2024-04-16T19:54:34.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yrjefh-50 zones={test-zone2, test-zone1} 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:43.769+0000 7f992a6c6640 -1 req 16214947220498892368 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-35:b482a2a9-ef36-488b-9666-ada656453328.4812.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:43.782+0000 7f99336d8640 -1 req 12603266879409288315 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-32:b482a2a9-ef36-488b-9666-ada656453328.4812.1,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:43.826+0000 7f983d4ec640 -1 req 11052363494949724008 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-21:b482a2a9-ef36-488b-9666-ada656453328.4361.25,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.557 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:43.826+0000 7f9939ee5640 -1 req 13017457097852656348 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-43:b482a2a9-ef36-488b-9666-ada656453328.4812.12,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:44.249+0000 7f988557c640 -1 req 8480006673959211356 0.002999924s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-2:b482a2a9-ef36-488b-9666-ada656453328.4361.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.558 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:44.257+0000 7f98acdcb640 -1 req 9740684810442569463 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-4:b482a2a9-ef36-488b-9666-ada656453328.4361.4,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.559 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:44.305+0000 7f98c2df7640 -1 req 1317718920376686029 0.001999949s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-22:b482a2a9-ef36-488b-9666-ada656453328.4361.26,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.559 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:44.348+0000 7f98cd60c640 -1 req 7121165894475656509 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-33:b482a2a9-ef36-488b-9666-ada656453328.4812.2,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.559 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T19:55:44.350+0000 7f98e2636640 -1 req 17659004084310902894 0.002999923s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=yrjefh-3:b482a2a9-ef36-488b-9666-ada656453328.4361.3,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} returned ret=-2 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'yrjefh-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket yrjefh-50 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"layout":{"resharding":"None","current_index":{"gen":2,"layout":{"type":"Normal","normal":{"num_shards":15,"hash_type":"Mod"}}},"logs":[{"gen":1,"layout":{"type":"InIndex","in_index":{"gen":1,"layout":{"num_shards":13,"hash_type":"Mod"}}}},{"gen":2,"layout":{"type":"InIndex","in_index":{"gen":2,"layout":{"num_shards":15,"hash_type":"Mod"}}}}]}} 2024-04-16T21:03:49.679 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:49.752 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.754 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:49.762 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-16T21:03:49.762 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:49.855 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:49.855 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:49.855 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:49.855 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2288, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 398, in zonegroup_bucket_checkpoint 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 390, in zone_bucket_checkpoint 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:49.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:37 GMT 2024-04-16T21:03:49.858 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2HbgxF/3u73RrekhVsZJvkisZOM= 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:37 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2HbgxF/3u73RrekhVsZJvkisZOM=', 'Content-Length': '0'} 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e6df4b68802604a8-00661edd1d-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yrjefh-56 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713298717.114405_1133.1","last_update":"2024-04-16T20:18:37.114405Z"}] 2024-04-16T21:03:49.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:37.339+0000 7fa597375800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:37.341+0000 7fa597375800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.107221_1130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.107221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.107221_1130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.107221Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:37Z"} 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298717.107221_1130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:37.107221Z', 'realm_epoch': 2}}] 2024-04-16T21:03:49.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.107221_1130.1'} 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713298717.114405_1133.1 target=1_1713298717.107221_1130.1 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:42.475+0000 7f226ed1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:42.476+0000 7f226ed1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.114405_1133.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.114405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:42Z"} 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713298717.114405_1133.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:18:37.114405Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:18:42Z"} 2024-04-16T21:03:49.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713298717.114405_1133.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:18:37.114405Z', 'realm_epoch': 2}}] 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713298717.114405_1133.1'} 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030cead5e5e762e2d-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ei850mX93LfhWqd7UR29TexZcDg= 2024-04-16T21:03:49.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ei850mX93LfhWqd7UR29TexZcDg='} 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c8e7c218ab554b6-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccb9503e81a4900c-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:49.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.867 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tHnsVjnXaxG18QiwdGVRYgHWPLk= 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tHnsVjnXaxG18QiwdGVRYgHWPLk='} 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deacefb2dca957a3-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b1d50a916c1d406-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:49.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.870 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XkM2LZ2vEpAb+5dtiXtLXRGqEko= 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XkM2LZ2vEpAb+5dtiXtLXRGqEko='} 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c56242465654552f-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca78c572d3f60ac4-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PsOgwh8go30mz2R9+Qm/hhh9BRk= 2024-04-16T21:03:49.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PsOgwh8go30mz2R9+Qm/hhh9BRk='} 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088e35268e4e3e54e-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.874 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000785fb29d4411fdbe-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RvGyZLpjpkyFihLmaZ27pNUs2LQ= 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RvGyZLpjpkyFihLmaZ27pNUs2LQ='} 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a585ef2b621d5db-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 5 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.879 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f438a5599526dc4-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.882 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.882 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.882 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:h3H4NmHEnZJQ+Lo7IKmv9chwWso= 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h3H4NmHEnZJQ+Lo7IKmv9chwWso='} 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000699946e8f5996ee9-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.885 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.886 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2a765d48d87f81-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34 2024-04-16T21:03:49.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.889 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QE3RsfnFohD4l00gvKrbQlMBB0c= 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QE3RsfnFohD4l00gvKrbQlMBB0c='} 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a90fdb3bfbc7cdf2-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.892 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.893 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005645cb0a27146fa0-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:49.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:49.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51 2024-04-16T21:03:49.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.896 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Zv4lY1r6mFajmdgBwnDWW7zKqmM= 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv4lY1r6mFajmdgBwnDWW7zKqmM='} 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3a7d184aebd1973-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.899 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.900 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000281eec47ebd9d143-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:49.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:49.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68 2024-04-16T21:03:49.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.903 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IcX+COQxaNuNMVXYDl/r6H65zTc= 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IcX+COQxaNuNMVXYDl/r6H65zTc='} 2024-04-16T21:03:49.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed4f6aecbcafb36a-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.907 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.907 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.907 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.907 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM= 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TKRnimvA5U+Mz5OQM5Yy0FlPNSM=', 'Content-Length': '0'} 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e57a24cbd62c271-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:49.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.909 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:42 GMT 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:iQLl1c6qddkjDMjiavLvcnV4iy0= 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:42 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iQLl1c6qddkjDMjiavLvcnV4iy0='} 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b93086a85cf1f15a-00661edd22-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 10 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.912 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c84c5aef789e35ac-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xPXNrAJ1p7kQlpc4IanzEqmFlbg= 2024-04-16T21:03:49.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xPXNrAJ1p7kQlpc4IanzEqmFlbg='} 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a16f8228b9d90bfc-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fa42bed07f051cb-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38 2024-04-16T21:03:49.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.919 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vsbUOb0ksf7H8Q2455UWNwl090E= 2024-04-16T21:03:49.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsbUOb0ksf7H8Q2455UWNwl090E='} 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cdd63e0821d1c6d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.923 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.924 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af4f3ea239f7c1d6-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:49.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:49.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57 2024-04-16T21:03:49.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.926 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.927 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2AObidFTwdNql4PenePDHcZSazw= 2024-04-16T21:03:49.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2AObidFTwdNql4PenePDHcZSazw='} 2024-04-16T21:03:49.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053eb45d226be0160-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.930 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b362681b35d1c25-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.932 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hazeifjdopvqaEmYcCFK+akK+/U= 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hazeifjdopvqaEmYcCFK+akK+/U='} 2024-04-16T21:03:49.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095811c13c8b2bead-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000930c9f8a774c5fe9-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:49.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95 2024-04-16T21:03:49.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tDXpehHrOR5aVIIHeQxxDbDqD1s= 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tDXpehHrOR5aVIIHeQxxDbDqD1s='} 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000deb4e1826fb26c26-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 15 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.938 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f806f1ccbfc19cce-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jApS8M0VSiXyQI4YKNp2mUyvjMs= 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jApS8M0VSiXyQI4YKNp2mUyvjMs='} 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3d488b13d651d75-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0a9525436158aad-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.942 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:yjgBwnue/IFIYAH9iZGvrd3PNeM= 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:yjgBwnue/IFIYAH9iZGvrd3PNeM='} 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bec248258f171dd8-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e17bc47d84028e-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.945 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:eXhOyuSpEvfm8WUCC0vdESQ7g1o= 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eXhOyuSpEvfm8WUCC0vdESQ7g1o='} 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d8e6042c66c3776-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069246b9a8798de40-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:49.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.948 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:epkCsVBSqaLrcVejRWfR4xTcOiQ= 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:epkCsVBSqaLrcVejRWfR4xTcOiQ='} 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bc6f2e9f5d1ac6f-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006013df2c70f3b182-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:49.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.951 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rUggxpUMk4jppP3tpfrZ0ofBGAo= 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rUggxpUMk4jppP3tpfrZ0ofBGAo='} 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b093cc8b2c67037d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 20 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000735f85113ab62a25-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:49.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.955 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rnfcJPQpXSjxQCvNlsGBMuhivN4= 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rnfcJPQpXSjxQCvNlsGBMuhivN4='} 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f1281cc73c692ba-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090ba74043f4636cc-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.958 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:oC5OtZIDJSEMPEFQFcp7R7KyYmE= 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oC5OtZIDJSEMPEFQFcp7R7KyYmE='} 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d01179759637a260-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009120d9eec306ac7e-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.961 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:M8j3bP7TLEVKhmamVj+6f9XqNTA= 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M8j3bP7TLEVKhmamVj+6f9XqNTA='} 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000387886e07fdc9b9d-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b93bf8bf3c2431-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.964 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XxhrP8MnRx0vA/EecBY4z+eru4o= 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XxhrP8MnRx0vA/EecBY4z+eru4o='} 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007bdc600592d8596-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.966 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg= 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OcpNLd0szwgsji48y7be7Z75GXg=', 'Content-Length': '0'} 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eb20e907ced8eff-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:43 GMT 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145 2024-04-16T21:03:49.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:4R/NZYv/V+vrd53gHrUlOcKv+ac= 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:43 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4R/NZYv/V+vrd53gHrUlOcKv+ac='} 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc6d6b7fb5a70597-00661edd23-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 25 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.970 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009da0c9f957d77bd9-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Nkh0PFsRE81z/Dc8Iu/75ABx02o= 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Nkh0PFsRE81z/Dc8Iu/75ABx02o='} 2024-04-16T21:03:49.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074d9e4d4c716e4ac-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.973 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000710009642ed29f0e-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62 2024-04-16T21:03:49.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.975 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mcmnJaw9MdZNIfwzi8RTLLyry1I= 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mcmnJaw9MdZNIfwzi8RTLLyry1I='} 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2ef7c52a5dadbaa-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077bf90e624879934-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.978 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IsPZT0IN0bZm/6AshlUYFsnm1/A= 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IsPZT0IN0bZm/6AshlUYFsnm1/A='} 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de3da05a46e21e1c-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c80e769b3b08eb7-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:49.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.981 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:9asGkxEKuQc8FFwE5zGyyso9yq4= 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9asGkxEKuQc8FFwE5zGyyso9yq4='} 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebcbfc030af025de-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f49ae84bfb33fc-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WCDEHdTqoozDHS2uvqgoNcaTZdI= 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WCDEHdTqoozDHS2uvqgoNcaTZdI='} 2024-04-16T21:03:49.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a99b9902a6493d62-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-56', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-56 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-56 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 30 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000691a263d4321f8ae-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.987 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:w4O+Si8ThIDwPZgL4+BIU6eS+II= 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:w4O+Si8ThIDwPZgL4+BIU6eS+II='} 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000532fee7b92e57e1f-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7d30a06a12f450-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:49.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.990 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T9heMy18v8ettoUFH2U+C3dEsqc= 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T9heMy18v8ettoUFH2U+C3dEsqc='} 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd2ab59009788cc-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f87f2c16e86fb7d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:49.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.993 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:9SEB/Lajb7oetLDvlravJm/zD/4= 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9SEB/Lajb7oetLDvlravJm/zD/4='} 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f474251818941ca-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082e54a71ab61eb1d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:49.996 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:r1qkG2JA59hykTvntpxT1sgEJn0= 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:r1qkG2JA59hykTvntpxT1sgEJn0='} 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005332d2778e6558fa-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU= 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Yx8UiMTPIBW1CaL1Dab+OMhu1dU=', 'Content-Length': '0'} 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eac3eb8e9e2fc5a2-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:49.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:49.999 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:18:44 GMT 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S3iCC41UyffVRidPoKTnNEO4ESE= 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:18:44 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S3iCC41UyffVRidPoKTnNEO4ESE='} 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024cf8c0058a3210d-00661edd24-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-56 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:44.626+0000 7f13ced33800 1 waiting to reach latest gen 6, on 3.. 2024-04-16T21:03:50.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:49.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:54.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:18:59.627+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:04.629+0000 7f13ced33800 1 waiting to reach latest gen 6, on 4.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:09.630+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:14.631+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:19.632+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:24.633+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:29.634+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:34.635+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:39.636+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:44.637+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:49.637+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:54.638+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:19:59.638+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:04.639+0000 7f13ced33800 1 waiting to reach latest gen 6, on 5.. 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:09.642+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:14.645+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:19.646+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:24.649+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:29.652+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:34.654+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:39.657+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:44.659+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:49.661+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:54.663+0000 7f13ced33800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , 00000000002.112.6, , , , , , , , , , , , , , , , , , , ] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:59.666+0000 7f13ced33800 1 bucket sync caught up with source: 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:20:59.666+0000 7f13ced33800 0 bucket checkpoint complete 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-56 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T20:20:59.820+0000 7f8da1d2d800 1 bucket sync caught up with source: 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, 00000000003.283.6, 00000000002.207.6, 00000000002.428.6, 00000000005.319.6, 00000000004.245.6, 00000000001.402.6, 00000000001.292.6, 00000000004.221.6, 00000000007.375.6, 00000000006.397.6] 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, 00000000003.283.6, 00000000002.207.6, 00000000002.428.6, 00000000005.319.6, 00000000004.245.6, 00000000001.402.6, 00000000001.292.6, 00000000004.221.6, 00000000007.375.6, 00000000006.397.6] 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T20:20:59.820+0000 7f8da1d2d800 0 bucket checkpoint complete 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yrjefh-56 zones={test-zone2, test-zone1} 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18= 2024-04-16T21:03:50.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18=', 'Content-Length': '0'} 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c98b166c6eb16015-00661eddab-6237-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.007 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18= 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3xuOye4YghIXV2fPub/9t7k4p18=', 'Content-Length': '0'} 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce973ed79ce3d530-00661eddab-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:/yrjefh-56/?versions 2024-04-16T21:03:50.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.010 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T21:03:50.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T21:03:50.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000407ad93180aab46-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-16T21:03:50.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj111 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj115 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj116 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj124 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj145 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj148 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj155 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj185 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-16T21:03:50.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-16T21:03:50.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj57 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj58 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj62 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj69 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj74 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj76 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj87 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj92 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj93 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj95 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.029 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:/yrjefh-56/?versions 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f39166cbcfaa42f-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj111 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj115 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj116 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj124 2024-04-16T21:03:50.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj145 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj148 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj155 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj185 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-16T21:03:50.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj57 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj58 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj62 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj69 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj74 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj76 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj87 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj92 2024-04-16T21:03:50.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj93 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj95 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:/yrjefh-56/?versions 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4cad3158ca7cb59-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?versions&max-keys=999 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.035 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:/yrjefh-56/?versions 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc= 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DxEOey+ZHbgEidzL+6FqjCrIlhc=', 'Content-Length': '0'} 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ffb063212df57a-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56999falseobj11nulltrue2024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj111nulltrue2024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj115nulltrue2024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj116nulltrue2024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj124nulltrue2024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj145nulltrue2024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj148nulltrue2024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj155nulltrue2024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj17nulltrue2024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj185nulltrue2024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj19nulltrue2024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj23nulltrue2024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj29nulltrue2024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj31nulltrue2024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj34nulltrue2024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj37nulltrue2024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj38nulltrue2024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj46nulltrue2024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj51nulltrue2024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj57nulltrue2024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj58nulltrue2024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj62nulltrue2024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj68nulltrue2024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj69nulltrue2024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj74nulltrue2024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj76nulltrue2024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj85nulltrue2024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj87nulltrue2024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj92nulltrue2024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj93nulltrue2024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj95nulltrue2024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.037 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047aa17df41635390-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c709c72d8afacb-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.040 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs= 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs=', 'Content-Length': '0'} 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023ac602c050af89b-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs= 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:GjQG3qt4GBlqZLJ1UAmcwbItrLs=', 'Content-Length': '0'} 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b6e6a576e71939-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.043 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc936e6f7424cff1-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11?versionId=null 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg= 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ITQaW9JVsdx0SXVf9P+cVDbzRUg=', 'Content-Length': '0'} 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b6095a8b68c120-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.046 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU= 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015278e1adb273c27-00661eddab-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU= 2024-04-16T21:03:50.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:69ZtESRsBvqE+OFKZnEfIXMXNJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f58084f1bb37748-00661eddab-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.049 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk= 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk=', 'Content-Length': '0'} 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc17037ac582bde-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk= 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cSloq0TR3571v7jyGe09REATElk=', 'Content-Length': '0'} 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f8f0d149344b63-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:50.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.052 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc= 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc=', 'Content-Length': '0'} 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002efc2ba11f38b307-00661eddab-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj11 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj11 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj11 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:20:59 GMT 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj11 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc= 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:20:59 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7qEgdsZSwLIFH8x5HOFg9ZClMkc=', 'Content-Length': '0'} 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fac03ceadae7629-00661eddab-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058c10e37f21625d2-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj11 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1304e5a5033402-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj111000falseobj112024-04-16T20:18:42.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T21:03:50.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432d3e4a11f52ae4-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.062 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1697422beed44e1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs= 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs=', 'Content-Length': '0'} 2024-04-16T21:03:50.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066aa57bfd8260d67-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.065 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs= 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Q6+cas29mUVb3SuUbK11BGxY+gs=', 'Content-Length': '0'} 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b64d73d454b707f-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.067 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7045700a0019b3-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111?versionId=null 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI= 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aklq7fXkc2Ws1B9R045qVEVvcBI=', 'Content-Length': '0'} 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003036531dbc3cd3c6-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.070 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb93b50f6e5e71d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ad515112b9c5a0-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE= 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE=', 'Content-Length': '0'} 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fe14668de6120c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.075 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE= 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:I2w72F3mY/qHZ/ya+HtwOXpkLAE=', 'Content-Length': '0'} 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3a189b976468c1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj111 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:50.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.077 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0= 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0=', 'Content-Length': '0'} 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096537f26ae361c03-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0= 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:go+vwJ5T4ACDGScGFz4ATWzjbn0=', 'Content-Length': '0'} 2024-04-16T21:03:50.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9060f59959e0758-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d2a60619da6e5c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj111 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.082 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a44b4474048665-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1111000falseobj1112024-04-16T20:18:44.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066bf54cd9bb55d89-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906469f51f415f78-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.087 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg= 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg=', 'Content-Length': '0'} 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027208c0e7714931f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg= 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4JY5bsSQAXsRHywQVRgGD1umKVg=', 'Content-Length': '0'} 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f985e673a86097-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.090 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce1f9329bd89806-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.092 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115?versionId=null 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0= 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5gCqQB4+tABISb8iEdJqTyXZto0=', 'Content-Length': '0'} 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728f152c5533a21d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000225bfa26ff98ead-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.095 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f247904ca78aa643-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA= 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA=', 'Content-Length': '0'} 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d71921e209c455b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115 2024-04-16T21:03:50.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA= 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mOrHxY/k2p1W9Xy6wGD2f5iImDA=', 'Content-Length': '0'} 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b08dace58c3cd9-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj115 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.099 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk= 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk=', 'Content-Length': '0'} 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aaa5e25836e1cfe-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj115 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj115 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj115 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj115 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk= 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0u1FPqEhMPMRopda95Q8wpOkBhk=', 'Content-Length': '0'} 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c16dec2329afeb8-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.102 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb07fc29c671d0a-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj115 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.104 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ddfea5b17a69f2f-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1151000falseobj1152024-04-16T20:18:43.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.106 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b210d8fa97e9abbf-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.108 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b841d71af4c892-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU= 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU=', 'Content-Length': '0'} 2024-04-16T21:03:50.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a54daace8701671-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU= 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fPyU9D59ieXs2t1yszNwLic4CaU=', 'Content-Length': '0'} 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2937bef9733899-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.113 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103bbe72c3ba388b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116?versionId=null 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg= 2024-04-16T21:03:50.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:v0AJmo5SS1hrh2yLKTICE+Ip+kg=', 'Content-Length': '0'} 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b782fa2111f614e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc0a5414242b8d7-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.118 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d6d9874d6eb1dc-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y= 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586143ed51e18c67-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116 2024-04-16T21:03:50.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y= 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+QNvLlZXI1vkUGr3bgMcj5Fsw5Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9fc5de1c70149d3-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj116 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw= 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw=', 'Content-Length': '0'} 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a52eb2c4a518614-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj116 2024-04-16T21:03:50.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj116 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj116 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.124 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj116 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw= 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HMuNzQ+bQlG4vmaqU3XSzj8V9nw=', 'Content-Length': '0'} 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f434af938d21ccf-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.126 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008096e9ece555f34d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj116 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548a581ebb16b346-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1161000falseobj1162024-04-16T20:18:43.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2024-04-16T21:03:50.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.129 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab60ba2b893cd00d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T21:03:50.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5045f78d028168-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo= 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo=', 'Content-Length': '0'} 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4bc835095ba0b1-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo= 2024-04-16T21:03:50.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TGcLubafrJWwqDkRI3PqDhqQGNo=', 'Content-Length': '0'} 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ef7c3261c86b62-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004971eaa2860217f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124?versionId=null 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.137 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI= 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OD7gBRjdhU1h6ogQ8FFvoxa1NZI=', 'Content-Length': '0'} 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091668cb981ab84b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a2bc74dd7ac00c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.140 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d059ffb847e2bb09-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.141 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0= 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0=', 'Content-Length': '0'} 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002805035aa22af401-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0= 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5VpQqFf1BIqX98LG1iQJGRC0se0=', 'Content-Length': '0'} 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b2a4e99e073d39e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj124 2024-04-16T21:03:50.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg= 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg=', 'Content-Length': '0'} 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf094270f5b162b-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj124 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj124 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj124 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj124 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg= 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RFRJkBNnnLN+T0C/DiYhG1l0Ptg=', 'Content-Length': '0'} 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004626ea5131287cf7-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a575be4d16978c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj124 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.149 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994e5e1fead3fe0b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1241000falseobj1242024-04-16T20:18:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4001dd8b751c39d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.152 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f05d64a9458c490-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.154 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak= 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak=', 'Content-Length': '0'} 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0da91b797d490e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak= 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QAfVQcOdSTzX5eal6dmJYCJd0Ak=', 'Content-Length': '0'} 2024-04-16T21:03:50.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71907109bdc34db-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f11a298994d527b0-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145?versionId=null 2024-04-16T21:03:50.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0= 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c9opex0ty4hTBTOp7B3G9sLpko0=', 'Content-Length': '0'} 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d837e5e91aeed73-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106c4244a53487cb-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.162 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1b9e180de246f4-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs= 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs=', 'Content-Length': '0'} 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb12fcbf4ade92f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145 2024-04-16T21:03:50.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs= 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3EqRx3SpAL6WjX8wnyxu9h0umCs=', 'Content-Length': '0'} 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047081e682fe95184-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.165 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj145 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.166 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0= 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0=', 'Content-Length': '0'} 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095bb06f9fca59fa-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj145 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj145 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj145 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj145 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0= 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ncw8Vl8a4vT5QGBPWZNtE/sQam0=', 'Content-Length': '0'} 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e380acac9bb9ec-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078e54ce948a57c3f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj145 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa64d15b584912eb-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1451000falseobj1452024-04-16T20:18:43.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.172 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c20d638af1dd873-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a128102e16bf8d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.175 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98= 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98=', 'Content-Length': '0'} 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c6343efa07fe2d-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98= 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XtMjz9m+kARLtsl76cFRz78bs98=', 'Content-Length': '0'} 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007fb9cf6900dcc97-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2024-04-16T21:03:50.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.178 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc84ba7e69e2ae4-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148?versionId=null 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw= 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E9mUUhmcMNDvBsQAXrXz7U4Uagw=', 'Content-Length': '0'} 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007140d7b56b21f760-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.181 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b167b8655b6ec68-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c83b0a9a97817e08-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4= 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4=', 'Content-Length': '0'} 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426f596c2ef367f5-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.186 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4= 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gwkQJBKPWrRyhj+qbi2dtFuD1k4=', 'Content-Length': '0'} 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701199626695792b-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj148 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U= 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U=', 'Content-Length': '0'} 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088db3ae03e81fcab-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:50.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.189 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U= 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3V9JcDI7gElhEUNhrE5FoYRRt2U=', 'Content-Length': '0'} 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c158dde45e57b2-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411ba7197881619a-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj148 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.192 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516d4a37f693764e-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1481000falseobj1482024-04-16T20:18:44.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787e0fefd69a2f36-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd523088089b12d1-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.197 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU= 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2a6fce117e2cd7-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU= 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:P2IxYtbAVpuUZX+33nBFWcKQnJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020782ba4e1126fa6-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2024-04-16T21:03:50.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.200 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d36a01325bd36f-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155?versionId=null 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU= 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rQXCiue/+bdC/ymxCAWRNtddzvU=', 'Content-Length': '0'} 2024-04-16T21:03:50.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021acb53d9cab821d-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.203 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8d74299956d91e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ca602e1bb9185c-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY= 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b75061432297ac-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.208 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY= 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pjUA2zSi/07TxEw+2KoL++9iKGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c514d407208839aa-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj155 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.209 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY= 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY=', 'Content-Length': '0'} 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924b32864499cffb-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj155 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj155 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj155 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj155 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY= 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NUP89CjjPxjm27wiIHKycto3+zY=', 'Content-Length': '0'} 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a083ae3bc8afac-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.212 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672f9c85b0dd412c-00661eddac-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj155 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY= 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/nNgVmMNdFiL/8+95zUY6EQnUGY=', 'Content-Length': '0'} 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5fe3818b233ca18-00661eddac-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1551000falseobj1552024-04-16T20:18:44.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d9840e537715ff-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.217 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a96be26d318eee-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE= 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE=', 'Content-Length': '0'} 2024-04-16T21:03:50.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa79d7e940b8b90e-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.220 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE= 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KyOIaX370+0OL4xJLEjAsFiRVHE=', 'Content-Length': '0'} 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019225495d53590ca-00661eddac-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:00 GMT 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.223 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k= 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:00 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xoW14pK4usLG4C4h0Z8qYUpwS3k=', 'Content-Length': '0'} 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f451244c250b2eb5-00661eddac-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17?versionId=null 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aX/VePocZHScP3Zm35yWNmQfRBY= 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aX/VePocZHScP3Zm35yWNmQfRBY=', 'Content-Length': '0'} 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d2574f137d0bfa-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da3c1df2d8b77195-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bae7b22a9860a6-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.229 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY= 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY=', 'Content-Length': '0'} 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf151775871cfb4d-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.230 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY= 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kvrIs1oUnJzSmtJzYgtLvCk21cY=', 'Content-Length': '0'} 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307c02dff95229c7-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0= 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0=', 'Content-Length': '0'} 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62492ea5dc33f4e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj17 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj17 2024-04-16T21:03:50.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj17 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj17 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0= 2024-04-16T21:03:50.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mSFWfhE/q3gTP+Ky6Gu8LT3U7G0=', 'Content-Length': '0'} 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009742645b328e3ba5-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.234 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba564a4a052f87c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj17 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.236 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9adcb879ba0b0c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj171000falseobj172024-04-16T20:18:42.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T21:03:50.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d46a6eb7e0fbe2-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.239 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ccf62a853711a2f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI= 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI=', 'Content-Length': '0'} 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3510b4b2e385ffb-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI= 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Zv3qa32o3LXUQViiznBNkAJg+uI=', 'Content-Length': '0'} 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b97f6eabd49057a-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.244 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca8b0dd7b44c729-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185?versionId=null 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs= 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b1Bo3O0kvZFWudzJK+HKkSxNbGs=', 'Content-Length': '0'} 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6277a2b1af0bd95-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.247 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003109239bd9ddd4b0-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28cc3a5e4d1e871-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.250 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI= 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI=', 'Content-Length': '0'} 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002020850ced7c7f5c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI= 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2FB/JYeAKZx8grnqGLeBPQch3UI=', 'Content-Length': '0'} 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c646013f2305d54-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj185 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.253 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A= 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A=', 'Content-Length': '0'} 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc41fae62e7a4f04-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A= 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:rOdlGj8Qq9MWdQr33rE5Tg6NZ1A=', 'Content-Length': '0'} 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012148962bc3db3d0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.256 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606617719b83866e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj185 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.258 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5684da1efe3b1ce-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj1851000falseobj1852024-04-16T20:18:44.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.262 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7aaf4ba92a8f6fe-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a79a5ac46f3414c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.265 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw= 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw=', 'Content-Length': '0'} 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2a44fd9188030a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.267 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw= 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u4Y3QXvpNMdYHXmeERLFEy8mjuw=', 'Content-Length': '0'} 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06e4ceb71428754-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f025922ea4b710b-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.270 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19?versionId=null 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw= 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:J9NkSSSQ3lFYd/2zDB+v2iHtHxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e64dd0a7e447c548-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.272 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce69d8c454889a84-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081edfef05842d230-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.275 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE= 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE=', 'Content-Length': '0'} 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da0fb4d7122b389-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE= 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hW6dER5fx85HJKn1loMmw6t4TLE=', 'Content-Length': '0'} 2024-04-16T21:03:50.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62e7adad2500a4e-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4= 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4=', 'Content-Length': '0'} 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ef34206cd1d0932-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj19 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj19 2024-04-16T21:03:50.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj19 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.280 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj19 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4= 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L5ZorEKTr3B10R8ezTxjWgWnNf4=', 'Content-Length': '0'} 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e4c5c879592ccd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613950e46ef7b9df-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj19 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.283 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5d541b2907e770-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj191000falseobj192024-04-16T20:18:43.131Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.285 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f270244f77265bc-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd3d7fb26c5a5a3-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.288 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE= 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE=', 'Content-Length': '0'} 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda75938150ad59f-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE= 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XVAh88M1aaAIeFDqWGSxkdJT3DE=', 'Content-Length': '0'} 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9ceee69d9f0cdd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.290 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.291 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840e55b4820276ac-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22?versionId=null 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.293 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik= 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IVEuWuUCvGUJukS2jFuIOAWa7Ik=', 'Content-Length': '0'} 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daf28d79688647a-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb348b808095e1db-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.296 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ee1016710ee57f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q= 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q=', 'Content-Length': '0'} 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111e68ca826a3d3e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.299 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q= 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5MkkRcioNLFiMREi42GIenveG0Q=', 'Content-Length': '0'} 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a7a84dab9a61bd-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c= 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c=', 'Content-Length': '0'} 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809406deea50273e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj22 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj22 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj22 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj22 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.302 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c= 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7Qpp0FugQFWWJXjaUHSwQ4VNj5c=', 'Content-Length': '0'} 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c8e3c18ca03141-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064dde3b2b3ab62b-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj22 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.305 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e62579d1b54bcf-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj221000falseobj222024-04-16T20:18:42.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T21:03:50.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b97f7b0bdaf974e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082fa0dfe5173574f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.310 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk= 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk=', 'Content-Length': '0'} 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1efcd2d057de32-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk= 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ac6QPCa351Agy8HsK0S3z9BNDPk=', 'Content-Length': '0'} 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcd9c5428724673-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.313 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028db988c2e8a69ac-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23?versionId=null 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y= 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NVb/Bo2todEIGmcWOwdpvYA1k2Y=', 'Content-Length': '0'} 2024-04-16T21:03:50.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbbf2a1bb80de34-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.316 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005923d7c11bc5c477-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.318 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec572b92ec82960d-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ= 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005359a31b7661948a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23 2024-04-16T21:03:50.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ= 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CkzPUMpwa6m0sVyiKAx+zBgzYXQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9e700ecf2425762-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.322 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8= 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8=', 'Content-Length': '0'} 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004746a1403eefac34-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj23 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj23 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj23 2024-04-16T21:03:50.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj23 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.324 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8= 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:QLa+XhKbn7QArsnp0EdR78euFQ8=', 'Content-Length': '0'} 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7dde34d7177cad0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11a94880bb3ee49-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj23 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.327 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ddbe62548572e9f-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj231000falseobj232024-04-16T20:18:43.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.329 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4eb94837957d65-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f8c4c6149d89a9-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.332 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8= 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8=', 'Content-Length': '0'} 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020679f466476041-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8= 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LjVlN1iWNhhwiutE5NkLAgrwOj8=', 'Content-Length': '0'} 2024-04-16T21:03:50.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c130cbfbd09027b-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d247c20b010d205c-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.337 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29?versionId=null 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo= 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:R1ACo2C8RaoFZ/H9vgw/6ZlIKZo=', 'Content-Length': '0'} 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebcbae2fe306ade0-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ea8ac53cd2f8e5-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.340 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056bdf5d8c7b87b4c-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29 2024-04-16T21:03:50.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0= 2024-04-16T21:03:50.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0=', 'Content-Length': '0'} 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b76576ddef3e3cc-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.343 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0= 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SPuO/5wciz/cJJC7ANktWdrUFS0=', 'Content-Length': '0'} 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370101d0ea393521-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg= 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg=', 'Content-Length': '0'} 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f13acecad603a3-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj29 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj29 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj29 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.346 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj29 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg= 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:M658n4QcKtUqiEI3yZ1d7iTHfFg=', 'Content-Length': '0'} 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4c64e739bac3de-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d0cdc81dab9a78-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj29 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.349 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b581794fb8f5a4-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj291000falseobj292024-04-16T20:18:43.694Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dab15723f6e3b17-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547712eb591b57cf-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.354 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY= 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY=', 'Content-Length': '0'} 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1ad6a32797c021-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY= 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6PN3LDkj1wGdrUGzbgKHNYkBepY=', 'Content-Length': '0'} 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a814d0f1bcf2b3e7-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-16T21:03:50.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.357 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b451368bb0c25a-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31?versionId=null 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.359 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ= 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:L4zINGN0BlfD7HdcXvwGnYBy7VQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f000ce0cc9afff13-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137d9436652510e5-00661eddad-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.362 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba0ecba55a7f5a60-00661eddad-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU= 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU=', 'Content-Length': '0'} 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b799be90bef8126e-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31 2024-04-16T21:03:50.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU= 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1QjXj4OOkH68LF6agFAuduhFnZU=', 'Content-Length': '0'} 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afee03c026bb1725-00661eddad-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw= 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw=', 'Content-Length': '0'} 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008865ff3b9f199126-00661eddad-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj31 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj31 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj31 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj31 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw= 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0XbOgem5XCFORbr09rZMU/EGlAw=', 'Content-Length': '0'} 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba0267687f9f29b1-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.370 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:01 GMT 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o= 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:01 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TA5ivVnLpbB3ehkTl75XfkTfg2o=', 'Content-Length': '0'} 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c979fa09a8448fd-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj31 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.372 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022fa8fc1d3fe399b-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj311000falseobj312024-04-16T20:18:44.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T21:03:50.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d7169ac8bf9f06-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e24ded074ad69ed-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.377 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30= 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30=', 'Content-Length': '0'} 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3eafbd020caef73-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30= 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+fJznar9Ojy/58FBQaVe1Z/yV30=', 'Content-Length': '0'} 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014e2ca76172a042-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:50.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.380 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df73711eaa4e40f-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33?versionId=null 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.382 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc= 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1++/968GNK4w6uG1ELQ3cues1kc=', 'Content-Length': '0'} 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfc4f52b389d921-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe1d3f286bf6f41-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.385 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068450862f740c8d0-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E= 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E=', 'Content-Length': '0'} 2024-04-16T21:03:50.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568c950274e4b998-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.388 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E= 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WjN5rusy0/6cWN/IoQj/ypUt42E=', 'Content-Length': '0'} 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033846bbb27688d31-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY= 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY=', 'Content-Length': '0'} 2024-04-16T21:03:50.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007f26dafda34cd07-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj33 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj33 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj33 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.391 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj33 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY= 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Ch4qDUyftWtDujs2s6QioIoOEsY=', 'Content-Length': '0'} 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a372574ec550db-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.393 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a49bb2bf440d1d-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj33 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3ebcc1fabc40dc-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj331000falseobj332024-04-16T20:18:42.568Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd42ee313571a3b-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.398 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999e6adcf87c63ff-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM= 2024-04-16T21:03:50.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM=', 'Content-Length': '0'} 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93a5bc467d6df7c-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM= 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tK4xfxrZBwa7C1/SO68fnkvYJzM=', 'Content-Length': '0'} 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbb443242090439-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.403 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6c40628640c9a4-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34?versionId=null 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo= 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2K/C3Xmgfj3GVzchvINS4q03DBo=', 'Content-Length': '0'} 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089acc7eeb6432f33-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.406 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46bfb6aa8e196a9-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.408 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6eb2646f492d276-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8= 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8=', 'Content-Length': '0'} 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb88b397896e3406-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.411 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8= 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0DX4ysmrDYiim8aFfkOSZZmE7t8=', 'Content-Length': '0'} 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002545ce037e9a010a-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg= 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg=', 'Content-Length': '0'} 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cba898e224fbbc0-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj34 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj34 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj34 2024-04-16T21:03:50.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj34 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.414 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg= 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6Wsz/Y1pKMCfVM68Qpj+F79NaKg=', 'Content-Length': '0'} 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a9cf2e101c3255-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa81c84b53065b6a-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj34 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.417 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adf5dd3d196618f-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj341000falseobj342024-04-16T20:18:42.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007946bc1a40b5b3cd-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d2f831a3fd5a6e-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.422 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U= 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U=', 'Content-Length': '0'} 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a311924d20f465b6-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U= 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dN1TxTBdMQnuowDw78wQw7tFq0U=', 'Content-Length': '0'} 2024-04-16T21:03:50.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f627540749ab21-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.425 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9167564fe946fd4-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37?versionId=null 2024-04-16T21:03:50.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg= 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nQUgX3e+l0DEq9fmdWW5ktyQvZg=', 'Content-Length': '0'} 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a839c64c312eefb6-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7aec7d3e8caff10-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dcf38c267f3b903-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs= 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs=', 'Content-Length': '0'} 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801bbee0bab45d57-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:50.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs= 2024-04-16T21:03:50.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DMyud+3aHgVcRGmI3esapSm3FXs=', 'Content-Length': '0'} 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ea05425c569e12d-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.434 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag= 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag=', 'Content-Length': '0'} 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d881de548a52f07e-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag= 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:EsCq3WywNK82Ww99JmFmD8yVCag=', 'Content-Length': '0'} 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5b06f884e28553-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.437 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b4980a080371e3-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj37 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b1b222a9853ecd3-00661eddae-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj371000falseobj372024-04-16T20:18:44.371Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2931094c01706d-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.442 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df401ede586af5a-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA= 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA=', 'Content-Length': '0'} 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290b059d810fe56c-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.445 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA= 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cq4Sy6E26OT0kKSXLmmNBJuxRBA=', 'Content-Length': '0'} 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000dcb97ab2b78f71-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6408d5116fd0aba-00661eddae-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.448 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38?versionId=null 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4= 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IwHQ8SqE0RQ9YPgfoE7dna+Vce4=', 'Content-Length': '0'} 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdeb0030c7338c9-00661eddae-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.450 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a977f95961b591-00661eddae-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:02 GMT 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ= 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:02 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:s+567HwnosEiVF5XYZXO/wX2gDQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c69ef6b2c23f8d7-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38 2024-04-16T21:03:50.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY= 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY=', 'Content-Length': '0'} 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a387a20cad23f20a-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY= 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aZ9nlX+63ykk+TDomrDIZzVW+aY=', 'Content-Length': '0'} 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7aab66c52b5befb-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.456 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA= 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA=', 'Content-Length': '0'} 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436962b1e9f8a78c-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj38 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj38 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj38 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.457 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj38 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA= 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o9wZymX1c7B75V81p8eRxufsYYA=', 'Content-Length': '0'} 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547df91362e05766-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.459 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016fe9dcaf058d6ea-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj38 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3dc03423357fd4-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj381000falseobj382024-04-16T20:18:43.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:50.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.462 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110e44d6bb59fca5-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.464 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c747b2d04b3d7d05-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc= 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc=', 'Content-Length': '0'} 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bca885d0c545638-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.467 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc= 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AProntuk12xn5dyEy7Zuy7pJhdc=', 'Content-Length': '0'} 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba30c4ba46c71995-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7171e24211ddafc-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44?versionId=null 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs= 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VATePYQqkvLQWYnHDPzCfoD1tZs=', 'Content-Length': '0'} 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a6cdc50985acf0-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ff33b9893f3415-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa89f23d02d90fc1-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.475 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI= 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI=', 'Content-Length': '0'} 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b3c6890690257e-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI= 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zb/u6cdhDUrCl5npV/AWCFAwaEI=', 'Content-Length': '0'} 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da025dd63ada757-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:50.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.478 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58= 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58=', 'Content-Length': '0'} 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3b77c7557f1b39-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj44 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj44 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj44 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj44 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58= 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:staXR1iWn5JBNTz+r0NmS/9qL58=', 'Content-Length': '0'} 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e404c44659d44a41-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.481 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bb6beb5c6a80b7-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj44 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d15b12dd9efe33c2-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj441000falseobj442024-04-16T20:18:42.579Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004c6f59b3133e53-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.486 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007883e6af552e4117-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio= 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio=', 'Content-Length': '0'} 2024-04-16T21:03:50.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065482a503484db41-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio= 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:HfeCSK/ysALUFIjjybjy/JFYHio=', 'Content-Length': '0'} 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db137302479e0c1-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1370c816552a4e6-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46?versionId=null 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk= 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BOXjT6gosxk8W5ZEI0h6dNMhtwk=', 'Content-Length': '0'} 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e163650105cb5f-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.494 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002320a21bda6395c6-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023def55d76026d9e-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.497 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE= 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE=', 'Content-Length': '0'} 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdcd1fbb54c1cf5-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE= 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SifJRb4mHGun9HNxqW5LIA8GIrE=', 'Content-Length': '0'} 2024-04-16T21:03:50.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdb155e5343d7a7-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.500 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys= 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys=', 'Content-Length': '0'} 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7164dd5d664b7e-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj46 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj46 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj46 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj46 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys= 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:F3jZiextdZzYY+p6bUjcpAGRBys=', 'Content-Length': '0'} 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb272c3d0513272-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.503 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65551e3dd94201f-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj46 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.525 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d9e9bbdd9b8d17-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj461000falseobj462024-04-16T20:18:43.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed10a23889d59322-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065728fe6f2c5d38-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ= 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e69087ee9d2391-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.532 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ= 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:cXmzblCg6pHVryl1r77/O5bpchQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc4f48ddc1381ca4-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.534 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa51c6a6144d19e0-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51?versionId=null 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4= 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3IgFY1rCNBxcbecLrgkDqE/Dlw4=', 'Content-Length': '0'} 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eada67e5a8182e-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ebba50387b03756-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f413d984b633611-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.540 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc= 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc=', 'Content-Length': '0'} 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5f3cdccfae03d1-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc= 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:lcaJUAp9ndzp9VmqsXNdF8gnUrc=', 'Content-Length': '0'} 2024-04-16T21:03:50.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fce46ae7fcd8547-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.543 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY= 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY=', 'Content-Length': '0'} 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ebfe90937b9aad9-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj51 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj51 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj51 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj51 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY= 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vT848fkapj0j+h29NFimqXp4nAY=', 'Content-Length': '0'} 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c07bfd7d184271-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.546 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6cfdd53365b83f0-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj51 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd26c36dc3393b21-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj511000falseobj512024-04-16T20:18:42.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b31b09c09cab52-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.551 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a5591f0666fbea-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM= 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM=', 'Content-Length': '0'} 2024-04-16T21:03:50.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000153a6f5b09e2679-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM= 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fH5iplHpZ7w51ryZTklg4PKDGjM=', 'Content-Length': '0'} 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdfc65f99328809-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.556 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346c915035beebd7-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55?versionId=null 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0= 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:T6mzDXffzPEGCThEfMIvEnLN8w0=', 'Content-Length': '0'} 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008152cd0ac0ddf4ba-00661eddaf-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.559 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbac1fa5d3f5e5ad-00661eddaf-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM= 2024-04-16T21:03:50.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3fDsypW593k3AcfClAtZYqvyDiM=', 'Content-Length': '0'} 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012021375c48f563b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:03 GMT 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WcH6ul89ytNpujcF2IFnBCplQTg= 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:03 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WcH6ul89ytNpujcF2IFnBCplQTg=', 'Content-Length': '0'} 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff54756c8684908c-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55 2024-04-16T21:03:50.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:t7YntjF1nN0YOWo9gZu5KcQC9r4= 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t7YntjF1nN0YOWo9gZu5KcQC9r4=', 'Content-Length': '0'} 2024-04-16T21:03:50.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65a89d96ab9e3ca-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.565 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY= 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY=', 'Content-Length': '0'} 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908b5e411986e198-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj55 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj55 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj55 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj55 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY= 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:tS51Q+GgjpKsxGRYytIYJ4aumMY=', 'Content-Length': '0'} 2024-04-16T21:03:50.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5312bc1c9c5d51-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.568 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1325b4efd4b9b60-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj55 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.570 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003963da7b2fab185d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj551000falseobj552024-04-16T20:18:42.590Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T21:03:50.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e531678c08484fb2-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.573 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeaa17ba8753125a-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.575 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI= 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI=', 'Content-Length': '0'} 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008122c3139de1ae23-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI= 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dt1Ik7JYnqmzfYPh3f5842DCpTI=', 'Content-Length': '0'} 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a61d37def826232-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2024-04-16T21:03:50.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.578 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a5462362aac37c-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57?versionId=null 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g= 2024-04-16T21:03:50.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NMgUjgtEeD3TqQVXQWEVR4y8X4g=', 'Content-Length': '0'} 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98b9adf929dee9d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.581 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862369cae561892a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.583 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6b99a926be416d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.584 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU= 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU=', 'Content-Length': '0'} 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ecf33fd2f808380-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU= 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8shlYs4YFqtmcfHaG+M1+Y5l7gU=', 'Content-Length': '0'} 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019847c69abf69077-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.586 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj57 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.587 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA= 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA=', 'Content-Length': '0'} 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000348932a4e493cbe1-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj57 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj57 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj57 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj57 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA= 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:K1opoihrGxw+9gE9Xu2y7xzBmFA=', 'Content-Length': '0'} 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89bc20a1153ff70-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.590 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc96d44e32415f6b-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj57 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889cef224bbddd7f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj571000falseobj572024-04-16T20:18:43.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004634d2f5d339ea12-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.595 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4456ebee3c12a6f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk= 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk=', 'Content-Length': '0'} 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9d469bec9b09b3-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.598 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk= 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qheiWjsszVO6vBmfA5EooDXE+Uk=', 'Content-Length': '0'} 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3cd88a4c67e356-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002631b6757abd88f0-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.601 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58?versionId=null 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0= 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PPhmaQj+1M3lkSxdEILdgoMG5n0=', 'Content-Length': '0'} 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023db64ec2210984b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb53db5def8bc3a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c013cc35273b7bd4-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58 2024-04-16T21:03:50.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4= 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4=', 'Content-Length': '0'} 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c0b2be992ee66b-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4= 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2oEGI6+UUFsxRis1c10tUeqO6z4=', 'Content-Length': '0'} 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d5e3d7ad6b46863-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj58 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58 2024-04-16T21:03:50.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k= 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k=', 'Content-Length': '0'} 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04d8e32c63247d6-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj58 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj58 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj58 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj58 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k= 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xaFcNRYxb8hoIbdznilt9N2rt+k=', 'Content-Length': '0'} 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47cd5f5ad95d37f-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.612 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4c34e46d19ff4a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj58 2024-04-16T21:03:50.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.614 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27667b00d4cc34d-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj581000falseobj582024-04-16T20:18:43.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2149174a06400bd-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.617 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.618 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.618 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T21:03:50.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c46e48a537a377-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw= 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw=', 'Content-Length': '0'} 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0bce9b0c233e2a4-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.627 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw= 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:35upmCqo8d432GkTahhnBNvV5Tw=', 'Content-Length': '0'} 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f25cf3359f5a0fc-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T21:03:50.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ecb13d131d904d4-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62?versionId=null 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU= 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IoyxUC290dk3YFIwJvTzW4juPJU=', 'Content-Length': '0'} 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69e3b19c1fe2c06-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.632 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3da4d07328a70d-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50d9e516df6d2ac-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.635 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94= 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94=', 'Content-Length': '0'} 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7fc16a10e98459-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94= 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3n/IxAx7unLX02yT7qtqGyRdN94=', 'Content-Length': '0'} 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd39cc31e64c0d3-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj62 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62 2024-04-16T21:03:50.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg= 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg=', 'Content-Length': '0'} 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:53 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aacf6f57860c98a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj62 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj62 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj62 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj62 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg= 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nPLG0Ikr5XEdhWk4KqN/HBty2Yg=', 'Content-Length': '0'} 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfc59f018f923e9-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17b2b2ef4b69de3-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj62 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064895f8b5b9b6e8b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj621000falseobj622024-04-16T20:18:44.080Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.643 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6c351b3674dbaa-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.646 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c7a95af762ea7b-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88= 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88=', 'Content-Length': '0'} 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd658f65b7c07b8a-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.649 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88= 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wuz+4yA9F8noee4EHDhCc7zCl88=', 'Content-Length': '0'} 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5d1e994d95c271-00661eddb0-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.651 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14cdec17637d96d-00661eddb0-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68?versionId=null 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw= 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:x59/3s3XHxO2UNcZKUcqKwibvFw=', 'Content-Length': '0'} 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee051b1bea5cf060-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:04 GMT 2024-04-16T21:03:50.654 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo= 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:04 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Wi5Ot+x67ijMBRPgpno26q6gWHo=', 'Content-Length': '0'} 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb033baf56a1151-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccd5f2e4b146943-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68 2024-04-16T21:03:50.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM= 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM=', 'Content-Length': '0'} 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003851c4d99f5f3215-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.658 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM= 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Xv4BVVmDHL7/jCXl4MOFhnv03zM=', 'Content-Length': '0'} 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ac1b50b7bb0fb0-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68 2024-04-16T21:03:50.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU= 2024-04-16T21:03:50.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU=', 'Content-Length': '0'} 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3ee234e1ef4470-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj68 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj68 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj68 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.661 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj68 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU= 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2Q4Q3gAzP6AH9azRao2SPHp6haU=', 'Content-Length': '0'} 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2df58775ed9bc2-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a06bfd0da30b88-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj68 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b52ef8296c47cd-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj681000falseobj682024-04-16T20:18:42.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.666 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8dd6237f9566e8-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068aec31d305df100-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.669 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc= 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc=', 'Content-Length': '0'} 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545b7f98b4bf8a5a-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.671 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc= 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:nfWZLIMnLQp5BCzv93t/xkwbaGc=', 'Content-Length': '0'} 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78906baac106c81-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b29b54e11d59fa5-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69?versionId=null 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE= 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TmHFu1vTykAuCuqxHkk4HqYi5sE=', 'Content-Length': '0'} 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce260ad80e8f0e61-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.676 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a6ce1993b1665c-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf499c2f163697b2-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4= 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4=', 'Content-Length': '0'} 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056be71f6875f7453-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4= 2024-04-16T21:03:50.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:toqARsNidxtek+xa9rOitnMqnW4=', 'Content-Length': '0'} 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229bacbc2a64c923-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj69 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo= 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo=', 'Content-Length': '0'} 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000754e46db76e48e84-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj69 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj69 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj69 2024-04-16T21:03:50.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj69 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo= 2024-04-16T21:03:50.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/GyW2ncyOCGoPHWmgvHanLdbLRo=', 'Content-Length': '0'} 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc53c687eedb156-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071da58e5249c9e38-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj69 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.687 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0a4f26677bb4782-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj691000falseobj692024-04-16T20:18:43.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.689 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f3d71a2d90dbbd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T21:03:50.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033916ba0d21d37bb-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8= 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8=', 'Content-Length': '0'} 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6df2f3191ead07-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.694 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8= 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:BqBjjdVFpPQ/0RfakeJ5561v2B8=', 'Content-Length': '0'} 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee929ef1fdbb8ecd-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T21:03:50.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a539eccbe6ed4bfd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.697 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74?versionId=null 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8= 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:h5Soj35d8scd/jc/6hcQGW9L5F8=', 'Content-Length': '0'} 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008746688aa1db6dd5-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.699 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d34147d77ac4d9-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f81b029b8d6fb9-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.702 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.703 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY= 2024-04-16T21:03:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY=', 'Content-Length': '0'} 2024-04-16T21:03:50.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f15c28fdedbb839-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY= 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E/wvdr4gErfaCadDx/9bTnY24DY=', 'Content-Length': '0'} 2024-04-16T21:03:50.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25fd9137bcd7479-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj74 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.707 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls= 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls=', 'Content-Length': '0'} 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:20:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a178fe749731ddfe-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls= 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vz+9jJaQ5i+xdSRedYfNxvMzGls=', 'Content-Length': '0'} 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cc97a3a8bed592-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.710 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067dc3185b2ff53dd-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj74 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.712 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006248aeb89a6c97e3-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj741000falseobj742024-04-16T20:18:44.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.739 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff02e0cd2fe1fb9d-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac8d67c63d20f63-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.742 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM= 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM=', 'Content-Length': '0'} 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae5e058a8443838-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM= 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:0kI0/NkPv/BbPzsj/Hx7BPv0PaM=', 'Content-Length': '0'} 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6e37a9d259cea8-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T21:03:50.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ae4e281e64c0fb-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.748 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76?versionId=null 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno= 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zOy5Z1E9+6VudQO1SH3mu07Tzno=', 'Content-Length': '0'} 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b78e1a2ac8217a-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118ea31890b1023d-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbe1cd3d7b8a6aa-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.753 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU= 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU=', 'Content-Length': '0'} 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aafff5b668d58a0-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU= 2024-04-16T21:03:50.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:pKasZVbNoY/BjFsL424lNcJqwBU=', 'Content-Length': '0'} 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3ce1ae11164907-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj76 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.756 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY= 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY=', 'Content-Length': '0'} 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001341596a01a56c0a-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj76 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj76 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj76 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.758 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj76 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY= 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TVX+MTAICPQE7TKpfMX3XlyFLZY=', 'Content-Length': '0'} 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff546ec80349999-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e3945546b21366-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj76 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE= 2024-04-16T21:03:50.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8iH1l/xq9AiK1TITr4bBNI0gvpE=', 'Content-Length': '0'} 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9ad2a6b28ed7ea-00661eddb1-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj761000falseobj762024-04-16T20:18:43.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I= 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I=', 'Content-Length': '0'} 2024-04-16T21:03:50.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667746c7a7b43d83-00661eddb1-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:05 GMT 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I= 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:05 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WIAMcvZWIzT7HZ+HTv2rvLLQL5I=', 'Content-Length': '0'} 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db01a05f238cab3d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw= 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c3fb8436732cb-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.770 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw= 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ROYwHNkMKOTalcXkCnC/VRN9kxw=', 'Content-Length': '0'} 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035317566af605ea2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w= 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w=', 'Content-Length': '0'} 2024-04-16T21:03:50.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d6fd2593b11e74-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.773 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85?versionId=null 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w= 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:DWpvx6hMNjrZ6iHQ+N9pFiPSW6w=', 'Content-Length': '0'} 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c66ebb0207720d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.775 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47dfa526385fdcc-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.777 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f716ec2f278063d1-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s= 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s=', 'Content-Length': '0'} 2024-04-16T21:03:50.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b61609d56969e02d-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s= 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:LSbqfA5qAkKem2g+++GpRoGhn3s=', 'Content-Length': '0'} 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4894b46513ba63-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2024-04-16T21:03:50.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.782 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug= 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug=', 'Content-Length': '0'} 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4757e7a09b267ae-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj85 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj85 2024-04-16T21:03:50.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj85 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.784 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj85 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug= 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:V8rknd/U4XlyMfrcHDVlU4N35ug=', 'Content-Length': '0'} 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c674d6e127d52d2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed024049a98ff6fd-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj85 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.788 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf5bf7dfe0701839-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj851000falseobj852024-04-16T20:18:42.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.790 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005368869f991be4c1-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.792 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d205ff27c77404f-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.794 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE= 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE=', 'Content-Length': '0'} 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432835f5fd3dc697-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.796 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE= 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:C/K62hbinNA4biB22pvm18GSnJE=', 'Content-Length': '0'} 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e518c06b63a21d0e-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T21:03:50.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a787b1d2dd3b9c5-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87?versionId=null 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ= 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:02aTVZX6Oy6QJWny0n1nLuxt7QQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d758396e6649b7-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3af1670891bda1c-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.803 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c503899dc5ba84-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87 2024-04-16T21:03:50.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4= 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4=', 'Content-Length': '0'} 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cce7d229ded101-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4= 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:9TnSI7vxljPCIAsjXozg2803Xp4=', 'Content-Length': '0'} 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d67b1ac500de81-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj87 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.808 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA= 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA=', 'Content-Length': '0'} 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:48 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff005fbce758781-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj87 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj87 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj87 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj87 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA= 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ff8TeBFHMs7GnO3Sp/EtQkV8KwA=', 'Content-Length': '0'} 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d845232564fbc7aa-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000429691d94c297e9f-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj87 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025fe151e00411ef0-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj871000falseobj872024-04-16T20:18:43.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.815 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e5e4341a3beea8-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.819 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2119cd7dc90e22-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs= 2024-04-16T21:03:50.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs=', 'Content-Length': '0'} 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e12ab679f4d0f0-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.823 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs= 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qQDc9opZDaeZyt5xfmKClL9R6bs=', 'Content-Length': '0'} 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ea82438c759ce5-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T21:03:50.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aefe54d0bd68a6-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92?versionId=null 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g= 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CYiK5j1V7XdjCQaaZ6DwzG+qP8g=', 'Content-Length': '0'} 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8441928bc99cce-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.828 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665dd6211bb479c8-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f1a9f7afd673b9-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc= 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc=', 'Content-Length': '0'} 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b037fb4597a436-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc= 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:OQow6rfJW2EJDx0scNSfT6Z0bZc=', 'Content-Length': '0'} 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9cbd89444f9a0d-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj92 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.835 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE= 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE=', 'Content-Length': '0'} 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136901e9d855a133-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj92 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj92 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj92 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj92 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE= 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TTzrb6Mw3QE3gM7hufuaoS8GnAE=', 'Content-Length': '0'} 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4011a1c78ed5c2-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.838 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30ce9d53d664b01-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj92 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4f45af4b936047-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj921000falseobj922024-04-16T20:18:43.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b258c41cfd6264-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c87784171b295ef-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.845 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I= 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I=', 'Content-Length': '0'} 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53d37ec9752e68b-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.847 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I= 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:qU0gTs+iDZZiRQYcF5XAShRSK2I=', 'Content-Length': '0'} 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040970bc676b10214-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c59fc77fe3a033-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.850 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93?versionId=null 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg= 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XDUnoCNmi+tGAW2LyPijqO/TkHg=', 'Content-Length': '0'} 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd331e91196ebdf1-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e97cb2524e6cae-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.855 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4b0a39bdefc42f-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93 2024-04-16T21:03:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE= 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE=', 'Content-Length': '0'} 2024-04-16T21:03:50.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007062356cbdbbb261-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.858 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE= 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VpQ+ijd+Bof65GqS85244NY9SJE=', 'Content-Length': '0'} 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243957e7d08f87de-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj93 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93 2024-04-16T21:03:50.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ= 2024-04-16T21:03:50.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:19:58 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068423b1907649369-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj93 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj93 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj93 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj93 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ= 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8eLiTLYA4WdprVqdwFI/Sj1EAcQ=', 'Content-Length': '0'} 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c9aeca4cbe29ec-00661eddb2-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.863 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:06 GMT 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM= 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:06 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E+E2V12IgknULjVj9NHGBWeFScM=', 'Content-Length': '0'} 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb0ba661079ca30-00661eddb2-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj93 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.865 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351cb8c80cdc7b96-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj931000falseobj932024-04-16T20:18:44.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.868 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.869 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T21:03:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T21:03:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e2b0d6e80683e4-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.873 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.873 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.873 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.873 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.874 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.874 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T21:03:50.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T21:03:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075255f167c77c068-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw= 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw=', 'Content-Length': '0'} 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc60e41b08bf95d0-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.881 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw= 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:avkTXYvgsStbOEh6uP9R9iJzTKw=', 'Content-Length': '0'} 2024-04-16T21:03:50.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5e2d3afa5e915d-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.885 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.885 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.886 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.886 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.886 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.886 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d3b93506faf68a-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.889 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95?versionId=null 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE= 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:mZ5VBXCsI1ZCr3xzqma2JK9FZwE=', 'Content-Length': '0'} 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009027035021fb4487-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.892 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.893 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.893 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.893 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.893 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91a35ad0b0e69c6-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.895 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fedd94b90f42d60-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY= 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY=', 'Content-Length': '0'} 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3f4d2982cbee8a-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.898 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY= 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:+CQPRqDaQuBz4lHWLWBQLlWayuY=', 'Content-Length': '0'} 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e7bcf64ab0d600-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj95 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95 2024-04-16T21:03:50.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.900 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg= 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg=', 'Content-Length': '0'} 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-56:b482a2a9-ef36-488b-9666-ada656453328.4812.25'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad3a033cc28b143-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj95 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj95 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj95 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj95 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg= 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TZvawvgVpBKOZkNkmN/DnGCnEjg=', 'Content-Length': '0'} 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:18:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cf14607b4cb02b-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.904 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002da838174754adc6-00661eddb3-6237-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/?prefix=obj95 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.906 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:07 GMT 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM= 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:07 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:WsvneXukr4p8Su1XZb1EdEjZEQM=', 'Content-Length': '0'} 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d893e8524b1e0c-00661eddb3-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-56obj951000falseobj952024-04-16T20:18:43.206Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yrjefh-56 zones={test-zone2, test-zone1} 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:21:07.284+0000 7f9a2f9a4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 204040) UID: 0 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:21:07.284+0000 7f9a35322880 -1 shutting down 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2024-04-16T21:03:50.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.908 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f829d13ca22c8352-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj37 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj37 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj37 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj37 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:/hAg3qh+yL80JLW3SP5g1Ts4fPk= 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:/hAg3qh+yL80JLW3SP5g1Ts4fPk='} 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1a342c3445dc85a-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.911 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008361c4bc81f1a64b-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj74 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj74 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj74 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj74 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:AZAZYrYGETmNhNz0UzeY2iO7HMg= 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:AZAZYrYGETmNhNz0UzeY2iO7HMg='} 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066fc5917e68abad1-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.914 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3c903edb5206057-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj111 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj111 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj111 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.916 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj111 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Rot7S0mGhIVm9gTMD6tjw7J6YFM= 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Rot7S0mGhIVm9gTMD6tjw7J6YFM='} 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a504becd2104c4b6-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9889bff69539171-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj148 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj148 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj148 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.919 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj148 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:oPWRb+iBYqVFzF5nm2KpN8g//b0= 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:oPWRb+iBYqVFzF5nm2KpN8g//b0='} 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071cbd9aab550e7c0-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/ 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/ 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/ 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:/yrjefh-56/ 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY= 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S0MJhrhZxNy+AdWy9Bhb1vRw3xY=', 'Content-Length': '0'} 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c978e03717b888c-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-56/obj185 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-56/obj185 2024-04-16T21:03:50.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-56/obj185 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.922 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:21:13 GMT 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:/yrjefh-56/obj185 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:o/LjpnajVNEki5pIW+d0JhpWM08= 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:21:13 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:o/LjpnajVNEki5pIW+d0JhpWM08='} 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d60fa368612721c4-00661eddb9-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:13.642+0000 7fc5bb525800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:13.643+0000 7fc5bb525800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T21:03:50.923 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:21:13.881+0000 7f87e5025880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T21:03:50.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: anonymous 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-56 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-56', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-56 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:14.883+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:19.886+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:24.888+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:29.890+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:34.892+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:39.895+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:44.898+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:49.900+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:54.902+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:21:59.905+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:04.907+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:09.910+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:14.911+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:19.914+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:24.917+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:29.919+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:34.922+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:39.924+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:44.927+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:49.929+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:54.930+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:22:59.934+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:04.936+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:09.938+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:14.941+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000002.104.6, , , , , , 00000000002.147.6, , , , , 00000000002.112.6, , , , , , , , , , , , , , , 00000000001.114.6, , , , 00000000001.107.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi026.stdout: INFO: 2024-04-16T20:23:19.048+0000 7faa014a1640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:19.943+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:24.945+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:29.948+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:34.949+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:39.953+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:44.955+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:49.957+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:54.960+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:23:59.962+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:04.965+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:09.967+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:14.968+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:19.972+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:24.974+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:29.976+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:34.979+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:39.981+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:44.984+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:49.986+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:54.988+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:24:59.991+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:04.993+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:09.995+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:14.997+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:20.000+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:25.002+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:30.004+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:35.005+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:40.009+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:45.011+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:50.014+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:25:55.016+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:00.019+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:05.021+0000 7f264892c800 1 waiting for incremental sync to catch up: 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , 00000000003.106.6, , , , , , 00000000002.147.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , 00000000003.106.6, , , , , , 00000000003.149.6, , , , , 00000000003.114.6, , , , , , , , , , , , , , , 00000000002.116.6, , , , 00000000002.109.6] 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:10.023+0000 7f264892c800 0 bucket checkpoint timed out waiting for incremental sync to catch up 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:10.023+0000 7f264892c800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:10.023+0000 7f264892c800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-16T21:03:50.939 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:50.940 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.940 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:50.940 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-16T21:03:50.940 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:50.952 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:50.952 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:50.952 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2332, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 398, in zonegroup_bucket_checkpoint 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 390, in zone_bucket_checkpoint 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.954 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:10 GMT 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:PUnEJvfiHZ2OTRlTftfOUbIkvpg= 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:10 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:PUnEJvfiHZ2OTRlTftfOUbIkvpg=', 'Content-Length': '0'} 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f524d439354292b6-00661edee2-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=yrjefh-57 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299170.258360_1249.1","last_update":"2024-04-16T20:26:10.258360Z"}] 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:10.510+0000 7f09cd733800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:10.512+0000 7f09cd733800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.248736_1246.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.248736Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:10Z"} 2024-04-16T21:03:50.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.248736_1246.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.248736Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:10Z"} 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299170.248736_1246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:10.248736Z', 'realm_epoch': 2}}] 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.248736_1246.1'} 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299170.258360_1249.1 target=1_1713299170.248736_1246.1 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:15.651+0000 7fb89d40f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:15.652+0000 7fb89d40f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.258360_1249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.258360Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:15Z"} 2024-04-16T21:03:50.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299170.258360_1249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:10.258360Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:26:15Z"} 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299170.258360_1249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:10.258360Z', 'realm_epoch': 2}}] 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299170.258360_1249.1'} 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.959 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047ab9968071be02c-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:u+rvuw8ONbIoH4AZ4Z5biRAsorY= 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:u+rvuw8ONbIoH4AZ4Z5biRAsorY='} 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8fd0d9f77226023-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.962 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b01f4366e896c5-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:NKbYH6MwN6vBn5gSnKn1NvND4b4= 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:NKbYH6MwN6vBn5gSnKn1NvND4b4='} 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c664263e98ab876-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.965 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad46bb10c82e4331-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:7pDGAU0exG4cANmcmw6iRnWvqjM= 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:7pDGAU0exG4cANmcmw6iRnWvqjM='} 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aece47cc9661271b-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf67295a9dfe56d3-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:50.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.970 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kjgOeXlKgskTO1xDij/yzSeSLtY= 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kjgOeXlKgskTO1xDij/yzSeSLtY='} 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed4ac8a55c4e75e0-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU= 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6x4r5Kw9d7mEkR97bssnfpYhhLU=', 'Content-Length': '0'} 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caeed73ae130c8b3-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:50.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:15 GMT 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55 2024-04-16T21:03:50.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5+F1F4W4g1yzLVW6lSRdxcjZCnM= 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:15 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5+F1F4W4g1yzLVW6lSRdxcjZCnM='} 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4e8c87ba9e49113-00661edee7-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-57 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:15.880+0000 7f085a71f800 1 waiting to reach incremental sync.. 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:20.882+0000 7f085a71f800 1 bucket sync caught up with source: 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: local status: [, , , , , , , , 00000000001.110.6, 00000000002.147.6, 00000000002.123.6] 2024-04-16T21:03:50.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: remote markers: [, , , , , , , , 00000000001.110.6, 00000000002.147.6, 00000000002.123.6] 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:20.882+0000 7f085a71f800 0 bucket checkpoint complete 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=yrjefh-57 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T20:26:21.037+0000 7f12cb12c800 1 bucket sync caught up with source: 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: local status: [, , , , , , , , 00000000001.394.6, 00000000002.310.6, 00000000002.165.6] 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: remote markers: [, , , , , , , , 00000000001.394.6, 00000000002.310.6, 00000000002.165.6] 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: 2024-04-16T20:26:21.037+0000 7f12cb12c800 0 bucket checkpoint complete 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=yrjefh-57 zones={test-zone2, test-zone1} 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.976 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk= 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk=', 'Content-Length': '0'} 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001766dbf7e64bd757-00661edeed-6330-test-zone2'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk= 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sx7Q/48pMG9dq8YnUb6gmZwPyVk=', 'Content-Length': '0'} 2024-04-16T21:03:50.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000374c3c137ec81427-00661edeed-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.981 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:/yrjefh-57/?versions 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018f9c92a8a25410d-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-16T21:03:50.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.983 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:/yrjefh-57/?versions 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5b2a48fefdc0930-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-16T21:03:50.984 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:/yrjefh-57/?versions 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T21:03:50.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d946f14a9bfe900a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?versions&max-keys=999 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:/yrjefh-57/?versions 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g= 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:3uOb3H+B25EuZOdBvMYK9gQTO6g=', 'Content-Length': '0'} 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003025006259e538a7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57999falseobj11nulltrue2024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj22nulltrue2024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj33nulltrue2024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj44nulltrue2024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normalobj55nulltrue2024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:50.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.989 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d2718ebef2c16f-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T21:03:50.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8306c6c057e6bee-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:50.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.994 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.994 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.995 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.995 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.995 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:50.995 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4= 2024-04-16T21:03:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4=', 'Content-Length': '0'} 2024-04-16T21:03:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be480d25df94232-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:50.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:50.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:50.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:50.999 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:50.999 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.999 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:50.999 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:50.999 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4= 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kiInjrZYIbmGW2HbQ1qyXmpvPS4=', 'Content-Length': '0'} 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff565c39002bab85-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:51.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.003 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bffde4ec6b970ad-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11?versionId=null 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.007 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA= 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:eUL70uXmbfai5kid3lynxWQJ3hA=', 'Content-Length': '0'} 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63daf7fbea6031f-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.010 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.010 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.010 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.011 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.011 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.011 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d346933c2fe71d3-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.014 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ccaf7c24f74d6f7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.017 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.018 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11 2024-04-16T21:03:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.018 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k= 2024-04-16T21:03:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k=', 'Content-Length': '0'} 2024-04-16T21:03:51.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2737f80efa546a5-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k= 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:XdruZ4OS27ENIwIHohPOrVHOG6k=', 'Content-Length': '0'} 2024-04-16T21:03:51.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a673a8af4cc69e9-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg= 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg=', 'Content-Length': '0'} 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4fe5f8e7caa87b-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj11 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj11 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj11 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.024 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj11 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg= 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:ygchAZ+CrQ5HBPAsfiy1QTJCtMg=', 'Content-Length': '0'} 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ab8f854609d566-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da616b9efa3ee973-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj11 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0877970da5dc3f-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj111000falseobj112024-04-16T20:26:15.684Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.029 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff377e9aabd7c1ba-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.031 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12271cb34327826-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso= 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso=', 'Content-Length': '0'} 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068128ee794828c3-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso= 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:iPQD9ws79dN2S7eYC3PVOnhZYso=', 'Content-Length': '0'} 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c15fd973f20e749-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.036 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2aa55c84f263081-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.038 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22?versionId=null 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s= 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:b6RhmimN6vIzW+M+b/ZgQ7G8l3s=', 'Content-Length': '0'} 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a669536802e6d1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548260cfa53b87c2-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf477c3a1f70cc1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4= 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4=', 'Content-Length': '0'} 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd3510c75929b3f-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.045 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4= 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:wTDgCyor71Zb9bYZB54Ab/RtWz4=', 'Content-Length': '0'} 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006104b598744af1b1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk= 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk=', 'Content-Length': '0'} 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e300ce2107d6aa7-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj22 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj22 2024-04-16T21:03:51.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj22 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj22 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.048 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk= 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Uo3p1m46fcQpkWOPM8J/ix3hAAk=', 'Content-Length': '0'} 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dfbbed4dda5a82-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.049 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7febe438b035a86-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj22 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.051 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dffc4ededb60d66-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj221000falseobj222024-04-16T20:26:15.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.053 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3c644d44cb252a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T21:03:51.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c5e58e6dada858-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU= 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU=', 'Content-Length': '0'} 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a63976b374634b4-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.058 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU= 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:sFcFTs4IDXfZB+fxuCIMcU8l6FU=', 'Content-Length': '0'} 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c6ec7cee87bacc-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb943c99141cabb-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.061 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33?versionId=null 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA= 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:5kBgIZENao5p+9wa1b0Qqav2iDA=', 'Content-Length': '0'} 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000052142ae00bc9735-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.063 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094d5acd75d38a100-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.065 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008919ed7a6606ad15-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33 2024-04-16T21:03:51.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.067 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk= 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk=', 'Content-Length': '0'} 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15bd151c0ab4169-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk= 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:gvCOoatV0owePezzdaBsKUJshRk=', 'Content-Length': '0'} 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f7a0cc70dd6862-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.070 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo= 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo=', 'Content-Length': '0'} 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8067133bdf333c-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj33 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj33 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj33 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj33 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo= 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hPuQ/I6vPRsM3CwZ134Kf/HVIXo=', 'Content-Length': '0'} 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68ca5b35d75dae3-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.073 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405bf2a227be5170-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj33 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.077 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054cd28f5ff4516b0-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj331000falseobj332024-04-16T20:26:15.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.079 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be712cc2f6a189e6-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e69fc5966c61566-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0= 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0=', 'Content-Length': '0'} 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072d36605de524dc2-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0= 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:RkRD5X4pS/hDnUu9a6/tuUWq5G0=', 'Content-Length': '0'} 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7220cc15c7e77b-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.086 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T21:03:51.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2033c207cb7bcdc-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.090 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.090 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.090 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.091 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.091 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44?versionId=null 2024-04-16T21:03:51.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.091 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g= 2024-04-16T21:03:51.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Y81mFfJ3HNwFRO9S/YUgFHtQi0g=', 'Content-Length': '0'} 2024-04-16T21:03:51.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d097c8a3fa3d09-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.094 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.095 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.095 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.095 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.095 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.096 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef485c8d324aaf55-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.099 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.100 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9bf7dee2ecf44c1-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.102 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ= 2024-04-16T21:03:51.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ=', 'Content-Length': '0'} 2024-04-16T21:03:51.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448efe797d8f088a-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44 2024-04-16T21:03:51.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.106 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.106 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ= 2024-04-16T21:03:51.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:heLqG5TOYxMR2l9Zjj8a42DvMsQ=', 'Content-Length': '0'} 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4fbf796208a09c7-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI= 2024-04-16T21:03:51.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI=', 'Content-Length': '0'} 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046836c53226d17ea-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj44 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj44 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj44 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.110 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj44 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI= 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IZbqkJSwx1rGSy5R9zQmTotvwMI=', 'Content-Length': '0'} 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1f962f6afe75f9-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.112 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f298c8254e98d651-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj44 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325d202e36638e40-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj441000falseobj442024-04-16T20:26:15.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:51.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.115 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004211c66231aad5e-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T21:03:51.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003336d8195e703bb3-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo= 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo=', 'Content-Length': '0'} 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0c0a9a5aea7a54-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo= 2024-04-16T21:03:51.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:8DftwqhksIjKY1X0d9B1oCZBVjo=', 'Content-Length': '0'} 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e6a54aa944b045-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a013baccde4d55ff-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.123 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55?versionId=null 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M= 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:TDhqRUOjaD9rZ8bl4ZmYK+dmQ9M=', 'Content-Length': '0'} 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16cd78ebbc7e3a4-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001457c8454cdbcb1b-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000427096d0d1b99483-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.128 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA= 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA=', 'Content-Length': '0'} 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939df91402219ebd-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.129 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA= 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:fQAZgGksSIgzV6E+r58210DsNoA=', 'Content-Length': '0'} 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00e6c36458ef3a0-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55 2024-04-16T21:03:51.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs= 2024-04-16T21:03:51.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs=', 'Content-Length': '0'} 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'b482a2a9-ef36-488b-9666-ada656453328:yrjefh-57:b482a2a9-ef36-488b-9666-ada656453328.4812.26'), ('x-rgw-replicated-at', 'Tue, 16 Apr 2024 20:26:17 GMT'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088398476cf6b4506-00661edeed-6330-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj55 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj55 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj55 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.132 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj55 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs= 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:imn+89/acHLhVZYdepA7D9L8DNs=', 'Content-Length': '0'} 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 16 Apr 2024 20:26:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923fbcc9e40a51f2-00661edeed-4812-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001940983b5a92cf40-00661edeed-6330-test-zone2'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/?prefix=obj55 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:21 GMT 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw= 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:21 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:1UnaCbToSopvY12A20FUzYn7BTw=', 'Content-Length': '0'} 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa664e78bbca7a6-00661edeed-4812-test-zone1'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'yrjefh-57obj551000falseobj552024-04-16T20:26:15.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDRGW11111111111111111Normal' 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=yrjefh-57 zones={test-zone2, test-zone1} 2024-04-16T21:03:51.136 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:26:21.974+0000 7f87e17cc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 212061) UID: 0 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:26:21.974+0000 7f87e5025880 -1 shutting down 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 5 2024-04-16T21:03:51.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.138 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f273c76154825183-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj17 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj17 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj17 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj17 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:kuxS56hZ5WlpX3rsfFmhwlFSxqg= 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:kuxS56hZ5WlpX3rsfFmhwlFSxqg='} 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de5457606d1b17ef-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.141 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055176342908f1699-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj34 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj34 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj34 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj34 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hQ688Io42sSODN3ItDJ6QfWoi+g= 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hQ688Io42sSODN3ItDJ6QfWoi+g='} 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8e2e44d2c6cc3cd-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.144 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef14ffe1db0664b3-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj51 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj51 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj51 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj51 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jrPfF4PvBgvYn6tQS+4jAnX7AkY= 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jrPfF4PvBgvYn6tQS+4jAnX7AkY='} 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf0aa2102a041b6b-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.147 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cb34c568be6c0c7-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj68 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj68 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj68 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj68 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:xt3Ly3tRF11zHykV4jZCNyWBPuc= 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:xt3Ly3tRF11zHykV4jZCNyWBPuc='} 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a2c7b94741964fb-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.150 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7bf83d21dd460f6-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj85 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj85 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj85 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj85 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2BCvO5Xr4RjiN+F8xmPUAxxJAr0= 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2BCvO5Xr4RjiN+F8xmPUAxxJAr0='} 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070d073d147876b05-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 10 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.153 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eddd8c1720c0dbe-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj19 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj19 2024-04-16T21:03:51.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj19 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.155 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj19 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:FwRLPblWVZ8nl9TQ4QDf3AERLic= 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:FwRLPblWVZ8nl9TQ4QDf3AERLic='} 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002878716f54dd43e4-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5328e389f66bc78-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj38 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj38 2024-04-16T21:03:51.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj38 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj38 2024-04-16T21:03:51.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:4H2pRWRceZRWwZklddO/G/W+vgk= 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4H2pRWRceZRWwZklddO/G/W+vgk='} 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f424e92576105be-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002725d00faa66be6e-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj57 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj57 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj57 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj57 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:zI9Hq57e0bmZXrS56BDiPhoGYxI= 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:zI9Hq57e0bmZXrS56BDiPhoGYxI='} 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003711743659843709-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.162 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d5402b8f7b792d-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj76 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj76 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj76 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj76 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:CoBS5uUIhAbo/f4Y3GPofhwSShw= 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:CoBS5uUIhAbo/f4Y3GPofhwSShw='} 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000693a4685bcf06744-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.165 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e1ea901658ed9a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj95 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj95 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj95 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj95 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:39LTyGOyBPsrblqJRy7I2xxalV8= 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:39LTyGOyBPsrblqJRy7I2xxalV8='} 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f086498fbe79bc7a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 15 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000153744cbea2e9e61-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj23 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj23 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj23 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.170 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj23 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:vsoXkr5DTRQyCBbJYuUSOGvFKag= 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:vsoXkr5DTRQyCBbJYuUSOGvFKag='} 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052f99582b3a85aed-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dbf747691f0ef90-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj46 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj46 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj46 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj46 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:JZLJivsPtayKN/MVF9UNodBuxvA= 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:JZLJivsPtayKN/MVF9UNodBuxvA='} 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050f8a5e0a4c32e0b-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.174 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fc892a093038a43-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj69 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj69 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj69 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj69 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jePztHeYSYQwfRaJS9KwhhZ4TPU= 2024-04-16T21:03:51.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jePztHeYSYQwfRaJS9KwhhZ4TPU='} 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c78cd6c2409eb740-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.177 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac6a337b2f0d9098-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj92 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj92 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj92 2024-04-16T21:03:51.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj92 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.179 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:4ok8V3HwSOCUc5y9dLRUlfgyuMw= 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:4ok8V3HwSOCUc5y9dLRUlfgyuMw='} 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a72384affd2d2381-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY= 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jdgw73+QXEyG49UpZ50kOHBhwmY=', 'Content-Length': '0'} 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000293b541a47304b1a-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj115 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj115 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj115 2024-04-16T21:03:51.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:28 GMT 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj115 2024-04-16T21:03:51.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:6o6N8mWYtgqcyEUVOkukQ4i248o= 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:6o6N8mWYtgqcyEUVOkukQ4i248o='} 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed6440b36c1a5406-00661edef4-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 20 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.184 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e25e4f1bb1d7b88-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj29 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj29 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj29 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj29 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:S/25sxXNCmHWh5NVfbq50dtL7y4= 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:S/25sxXNCmHWh5NVfbq50dtL7y4='} 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049ce41a9e13a4b71-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.187 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcdbc9700fa0a186-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj58 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj58 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj58 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj58 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:SZO1vAs6cwerzdeklfBVjRpFx8k= 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:SZO1vAs6cwerzdeklfBVjRpFx8k='} 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6c8d4049a8e5f7a-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.190 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faa89c14bc9fb051-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj87 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj87 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj87 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj87 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:c4Zd7utH1IaziNof62fCBVUz2hw= 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:c4Zd7utH1IaziNof62fCBVUz2hw='} 2024-04-16T21:03:51.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a91699b9715c52f-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.193 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6bea43e812ad1ea-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj116 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj116 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj116 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj116 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:KEcJVcPwPyS9R3I1x68RDyVyA/0= 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:KEcJVcPwPyS9R3I1x68RDyVyA/0='} 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000947cf9ff60106635-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.196 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005609d0be64a9316a-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj145 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj145 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj145 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj145 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:jwRj9rj6E67j/e0t76uKlX6XOcs= 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:jwRj9rj6E67j/e0t76uKlX6XOcs='} 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dabdaeb54b3ed1d3-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 25 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bbbd752037b8163-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj31 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj31 2024-04-16T21:03:51.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj31 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj31 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IeTWoKMEe4p/LiSPOyim6SMizqY= 2024-04-16T21:03:51.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IeTWoKMEe4p/LiSPOyim6SMizqY='} 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039bb035ae2c747d1-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.202 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc774115c0ca84fd-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj62 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj62 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj62 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj62 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:dC1cJCM9FjjM2wgJR1Exf0cLvAA= 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:dC1cJCM9FjjM2wgJR1Exf0cLvAA='} 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b5b2ca690368686-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.205 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbb4e74f54ab5935-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj93 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj93 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj93 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj93 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Jl+FZVEgui95+OZIhfAVLPgTCNQ= 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jl+FZVEgui95+OZIhfAVLPgTCNQ='} 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051c9f2b77b956a08-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069728e3919e5fbe1-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj124 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj124 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj124 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.209 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj124 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:VMW1bUMwWhFvZaMsj2B005KOZVE= 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:VMW1bUMwWhFvZaMsj2B005KOZVE='} 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5b60bff2ec5fdef-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c83e1b01a9da1f4-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj155 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj155 2024-04-16T21:03:51.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj155 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj155 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.212 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:hy0PEBGY7wyrmKKg0Eq9N+g3BiY= 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:hy0PEBGY7wyrmKKg0Eq9N+g3BiY='} 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b536d60b716af21-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'yrjefh-57', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket yrjefh-57 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: tenant: 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: bucket name: yrjefh-57 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: total entries: 30 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.214 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043711d95e0d7597c-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj37 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj37 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj37 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj37 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:t7JO84Zb052Azza7dYsS0SpcGyE= 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:t7JO84Zb052Azza7dYsS0SpcGyE='} 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c4162b38068b09b-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008981756cfe851118-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj74 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj74 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj74 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.218 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj74 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:IrD+0k5XE4KENK0RL1UEfvr/APk= 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:IrD+0k5XE4KENK0RL1UEfvr/APk='} 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4b655516a60d925-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.220 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089b7e608e9e17268-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj111 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj111 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj111 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj111 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:2tY7vaGHmyvnu4Fcdufka3z115o= 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:2tY7vaGHmyvnu4Fcdufka3z115o='} 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fc525aa9144cd43-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4ed12a37d14cab7-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj148 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj148 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj148 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj148 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:aR2OwydOoAwNe9KBxvNH5UzjU8U= 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:aR2OwydOoAwNe9KBxvNH5UzjU8U='} 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004895a2c10d8b1fd6-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/ 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/ 2024-04-16T21:03:51.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/ 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.226 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:/yrjefh-57/ 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c= 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:E3NZo/3/CjuXbfkTi9BHbEzru+c=', 'Content-Length': '0'} 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065731220358b4d8d-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-57/obj185 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-57/obj185 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-57/obj185 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:26:29 GMT 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:/yrjefh-57/obj185 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:Jnsi3LnWx//UEYKivAdH29Bt/sI= 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 16 Apr 2024 20:26:29 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:Jnsi3LnWx//UEYKivAdH29Bt/sI='} 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0dd3b8b756ead7e-00661edef5-4812-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 16 Apr 2024 20:26:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:30.126+0000 7f4863a10800 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:30.128+0000 7f4863a10800 5 RGW-SYNC:data: acquired data sync status lease 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-16T21:03:51.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: curl: (7) Failed to connect to smithi119.front.sepia.ceph.com port 8000: Connection refused 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:26:30.361+0000 7f652f252880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > curl http://smithi119.front.sepia.ceph.com:8000 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 182k 0 --:--:-- --:--:-- --:--:-- 182k 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: anonymous 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=yrjefh-57 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'yrjefh-57', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket yrjefh-57 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:31.357+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:36.358+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:41.359+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:46.360+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:51.360+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:26:56.361+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:01.363+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:06.364+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:11.365+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:16.365+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:21.366+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:26.368+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:31.368+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:36.369+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:41.370+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:46.371+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:51.372+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:27:56.372+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:01.374+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:06.375+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:11.374+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:16.376+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:21.377+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:26.378+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:31.379+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: *** Caught signal (Segmentation fault) ** 2024-04-16T21:03:51.232 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T21:03:51.233 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T21:03:51.234 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 0> 2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T21:03:51.235 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 0> 2024-04-16T20:28:34.137+0000 7f640afb8640 -1 *** Caught signal (Segmentation fault) ** 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: in thread 7f640afb8640 thread_name:data-sync 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.236 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: ceph version 19.0.0-3089-gb0c7b917 (b0c7b917267c471208541fdeb652f5e897d00086) squid (dev) 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 1: /lib64/libc.so.6(+0x54db0) [0x7f6531454db0] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2: radosgw(+0xe3f9ed) [0x555b4a5319ed] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 3: radosgw(+0x4130e2) [0x555b49b050e2] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 4: radosgw(+0x8e01b9) [0x555b49fd21b9] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 5: radosgw(+0xac6a87) [0x555b4a1b8a87] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x130) [0x555b49d38830] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x481) [0x555b49d3a381] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xad) [0x555b49d3b15d] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 9: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x4bf) [0x555b4a1bfaef] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 10: radosgw(+0x81bba8) [0x555b49f0dba8] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 11: (RGWRadosThread::Worker::entry()+0xb3) [0x555b49f0f0b3] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 12: /lib64/libc.so.6(+0x9f802) [0x7f653149f802] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 13: /lib64/libc.so.6(+0x3f450) [0x7f653143f450] 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stdout: INFO: 2024-04-16T21:03:51.237 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi119.stderr: INFO: daemon-helper: command crashed with signal 11 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:36.380+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:41.382+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~6s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:46.383+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~12s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:51.383+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~18s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:28:56.384+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~24s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:01.385+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~30s 2024-04-16T21:03:51.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:06.386+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:11.387+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~35s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:16.388+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~41s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:21.389+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~47s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:26.390+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~53s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:31.390+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~59s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:36.392+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~65s 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:41.393+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:46.394+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~71s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:51.395+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~77s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:29:56.396+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~83s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:01.397+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~89s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:06.397+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~95s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:11.398+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:16.399+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~100s 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:21.400+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.240 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~106s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:26.401+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~112s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:31.402+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~118s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:36.403+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~124s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:41.404+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~130s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:46.404+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:51.406+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~136s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:30:56.407+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~142s 2024-04-16T21:03:51.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:01.408+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~148s 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:06.409+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~154s 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:11.410+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~160s 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:16.411+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:21.412+0000 7f0ed1733800 1 waiting to reach latest gen 6, on 0.. 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~165s 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:26.412+0000 7f0ed1733800 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:26.412+0000 7f0ed1733800 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:26.412+0000 7f0ed1733800 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-16T21:03:51.242 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.243 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.243 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.243 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2024-04-16T21:03:51.243 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2503, in test_sync_policy_config_zonegroup 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.254 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299189.772386_1320.1","last_update":"2024-04-16T20:26:29.772386Z"}] 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:26.696+0000 7fb5a3d28800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:26.698+0000 7fb5a3d28800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:26Z"} 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:26Z"} 2024-04-16T21:03:51.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[]} 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~171s 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","epoch":11,"predecessor_uuid":"05d34c3b-cf84-4753-a65b-67914d39a038","sync_status":[],"period_map":{"id":"f1f5b060-9fe8-4227-8567-18236dc68d0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":true,"endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b482a2a9-ef36-488b-9666-ada656453328","zones":[{"id":"b482a2a9-ef36-488b-9666-ada656453328","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]},{"id":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","name":"test-zone2","endpoints":["http://smithi119.front.sepia.ceph.com:8000"],"log_meta":false,"log_data":true,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","sync_policy":{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]}],"directional":[{"source_zone":"b482a2a9-ef36-488b-9666-ada656453328","dest_zone":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]},"dest":{"bucket":"*","zones":["b482a2a9-ef36-488b-9666-ada656453328","eafd0fcf-abef-4654-893b-ccd3b2eb5bea"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b482a2a9-ef36-488b-9666-ada656453328","val":207324556},{"key":"eafd0fcf-abef-4654-893b-ccd3b2eb5bea","val":1730565424}]},"master_zonegroup":"test-zonegroup","master_zone":"b482a2a9-ef36-488b-9666-ada656453328","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"82a6ab2b-1604-42ea-987b-3570b8df3977","realm_epoch":2} 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=yrjefh-58 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/yrjefh-58/ 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/yrjefh-58/ 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /yrjefh-58/ 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi026.front.sepia.ceph.com:8000 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-16T21:03:51.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:PUT 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:Tue, 16 Apr 2024 20:31:28 GMT 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:/yrjefh-58/ 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:AWS VS71ZU627N05UVO7JCG7:37OccI8N4A4AsG8rGKr1pOYv1Ds= 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 16 Apr 2024 20:31:28 GMT', 'Authorization': 'AWS VS71ZU627N05UVO7JCG7:37OccI8N4A4AsG8rGKr1pOYv1Ds=', 'Content-Length': '0'} 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004e93b4744c0b1e5f-00661ee020-5877-test-zone1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 16 Apr 2024 20:31:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-16T21:03:51.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'yrjefh-58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket yrjefh-58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: {"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:29.598+0000 7f230c111800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:29.599+0000 7f230c111800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:29Z"} 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:29Z"} 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~177s 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:34.739+0000 7f5a3bd2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:34.741+0000 7f5a3bd2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:34Z"} 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:34Z"} 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~183s 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:39.881+0000 7f987d528800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:39.883+0000 7f987d528800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:39Z"} 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:39Z"} 2024-04-16T21:03:51.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:45.019+0000 7f580dd1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:45.020+0000 7f580dd1f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:45Z"} 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:45Z"} 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~189s 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:50.156+0000 7f1bd9e67800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:50.158+0000 7f1bd9e67800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:50Z"} 2024-04-16T21:03:51.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:50Z"} 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~195s 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:55.295+0000 7f68ba533800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:31:55.297+0000 7f68ba533800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:55Z"} 2024-04-16T21:03:51.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:31:55Z"} 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~201s 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:00.432+0000 7fdd70530800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:00.434+0000 7fdd70530800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:00Z"} 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:00Z"} 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~207s 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:05.615+0000 7fc42c813800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:05.616+0000 7fc42c813800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:05Z"} 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:05Z"} 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~213s 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:10.754+0000 7f9da0d24800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:10.756+0000 7f9da0d24800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:10Z"} 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:10Z"} 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~219s 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:15.893+0000 7fd853523800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:15.894+0000 7fd853523800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:15Z"} 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:15Z"} 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~224s 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:21.032+0000 7fca843af800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:21.034+0000 7fca843af800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:21Z"} 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:21Z"} 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:26.168+0000 7f06f432a800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:26.170+0000 7f06f432a800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:26Z"} 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:26Z"} 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~230s 2024-04-16T21:03:51.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:31.307+0000 7f4aa832d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:31.308+0000 7f4aa832d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:31Z"} 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:31Z"} 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.281 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~236s 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:36.445+0000 7f08182e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:36.446+0000 7f08182e4800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:36Z"} 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:36Z"} 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~242s 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:41.584+0000 7f6aba467800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:41.586+0000 7f6aba467800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:41Z"} 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:41Z"} 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~248s 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:46.723+0000 7f5e43f1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:46.724+0000 7f5e43f1e800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:46Z"} 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:46Z"} 2024-04-16T21:03:51.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~254s 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:51.863+0000 7fd387c2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:51.864+0000 7fd387c2f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:51Z"} 2024-04-16T21:03:51.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:51Z"} 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~260s 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:57.001+0000 7fbceed31800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:32:57.002+0000 7fbceed31800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:57Z"} 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:32:57Z"} 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:02.138+0000 7fabda28c800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.287 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~266s 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:02.139+0000 7fabda28c800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:02Z"} 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:02Z"} 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:07.277+0000 7f7571f2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:07.278+0000 7f7571f2d800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:07Z"} 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:07Z"} 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~272s 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:14.258+0000 7f9234424800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:14.259+0000 7f9234424800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:14Z"} 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:14Z"} 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~279s 2024-04-16T21:03:51.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:19.400+0000 7f6cc4424800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:19.401+0000 7f6cc4424800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:19Z"} 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:19Z"} 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~285s 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:24.539+0000 7f484aa12800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:24.540+0000 7f484aa12800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:24Z"} 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:24Z"} 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~291s 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:29.678+0000 7f850ce0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:29.679+0000 7f850ce0f800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:29Z"} 2024-04-16T21:03:51.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:29Z"} 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~297s 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:34.815+0000 7f8f85533800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:33:34.818+0000 7f8f85533800 20 RGW-SYNC:meta: read sync status 2024-04-16T21:03:51.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stdout: INFO: {"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:34Z"} 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f1f5b060-9fe8-4227-8567-18236dc68d0a","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1713297244.875722_185.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2024-04-16T19:54:04.875722Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1713297261.630622_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.630622Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1713297261.044058_860.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T19:54:21.044058Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1713299189.772386_1320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2024-04-16T20:26:29.772386Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1},"current_time":"2024-04-16T20:33:34Z"} 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713297244.875722_185.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-16T19:54:04.875722Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1713297261.630622_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.630622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713297261.044058_860.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T19:54:21.044058Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713299189.772386_1320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-16T20:26:29.772386Z', 'realm_epoch': 2}}] 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299189.772386_1320.1'} 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=3 master=1_1713299489.222246_1703.1 target=1_1713299189.772386_1320.1 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~302s 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: BARK! unmounting mounts and killing all daemons 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0: INFO: Sent signal 15 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1: INFO: Sent signal 15 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2: INFO: Sent signal 15 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.3: INFO: Sent signal 15 2024-04-16T21:03:51.295 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.4: INFO: Sent signal 15 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.5: INFO: Sent signal 15 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a: INFO: Sent signal 15 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1: INFO: Sent signal 15 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mgr.x: INFO: Sent signal 15 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.3.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 3 (PID: 32157) UID: 0 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.3.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 osd.3 55 *** Got signal Terminated *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.3.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f7fa752c640 -1 osd.3 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi119.stderr: INFO: 2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 32537) UID: 0 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi119.stderr: INFO: 2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 osd.0 55 *** Got signal Terminated *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.0.smithi119.stderr: INFO: 2024-04-16T20:33:39.603+0000 7ffbb4651640 -1 osd.0 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.4.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 4 (PID: 32158) UID: 0 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.4.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 osd.4 55 *** Got signal Terminated *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.4.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f92f8a46640 -1 osd.4 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.296 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi119.stderr: INFO: 2024-04-16T20:33:39.604+0000 7fe86e792640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 32539) UID: 0 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi119.stderr: INFO: 2024-04-16T20:33:39.604+0000 7fe86e792640 -1 osd.1 55 *** Got signal Terminated *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.1.smithi119.stderr: INFO: 2024-04-16T20:33:39.604+0000 7fe86e792640 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.5.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f242c244640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 5 (PID: 32161) UID: 0 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.5.smithi151.stderr: INFO: 2024-04-16T20:33:39.605+0000 7f242c244640 -1 osd.5 55 *** Got signal Terminated *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.5.smithi151.stderr: INFO: 2024-04-16T20:33:39.606+0000 7f242c244640 -1 osd.5 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi119.stderr: INFO: 2024-04-16T20:33:39.605+0000 7fe85b524640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 32540) UID: 0 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi119.stderr: INFO: 2024-04-16T20:33:39.605+0000 7fe85b524640 -1 osd.2 55 *** Got signal Terminated *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.osd.2.smithi119.stderr: INFO: 2024-04-16T20:33:39.605+0000 7fe85b524640 -1 osd.2 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi151.stdout: INFO: 2024-04-16T20:33:39.606+0000 7ffbd341d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 36798) UID: 0 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi151.stdout: INFO: 2024-04-16T20:33:39.607+0000 7ffbd942f880 -1 shutting down 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi119.stderr: INFO: 2024-04-16T20:33:39.606+0000 7f2b098e8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 32368) UID: 0 2024-04-16T21:03:51.297 INFO:tasks.rgw_multisite_tests:tasks.ceph.c2.mon.a.smithi119.stderr: INFO: 2024-04-16T20:33:39.606+0000 7f2b098e8640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:38:39.912+0000 7f0c37d11800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.298 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2534, in test_sync_flow_symmetrical_zonegroup_all 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.299 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:43:45.119+0000 7f783c31e800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.301 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2783, in test_sync_single_bucket 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:48:45.783+0000 7fc40ad2a800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.304 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 2901, in test_sync_different_buckets 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:53:46.111+0000 7ff6de0c0800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.306 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 3048, in test_sync_multiple_buckets_to_single 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.307 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T20:58:46.373+0000 7f44d3f11800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-16T21:03:51.309 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 3170, in test_sync_single_bucket_to_multiple 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 186, in zone_meta_checkpoint 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: global_status, period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_sync_status 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2024-04-16T21:03:51.310 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stderr: INFO: No --period given, using current period=f1f5b060-9fe8-4227-8567-18236dc68d0a 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi026.stdout: INFO: [{"marker":"1_1713297244.875722_185.1","last_update":"2024-04-16T19:54:04.875722Z"},{"marker":"1_1713297261.630622_136.1","last_update":"2024-04-16T19:54:21.630622Z"},{"marker":"1_1713297261.044058_860.1","last_update":"2024-04-16T19:54:21.044058Z"},{"marker":"1_1713299489.222246_1703.1","last_update":"2024-04-16T20:31:29.222246Z"}] 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713297244.875722_185.1', 1: '1_1713297261.630622_136.1', 2: '1_1713297261.044058_860.1', 3: '1_1713299489.222246_1703.1'} 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-16T21:03:51.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 1 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: 2024-04-16T21:03:46.719+0000 7f2202f33800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi119.stderr: INFO: failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-16T21:03:51.312 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.380 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-16T21:03:51.381 INFO:tasks.rgw_multisite_tests:Ran 65 tests in 5880.682s 2024-04-16T21:03:51.381 INFO:tasks.rgw_multisite_tests: 2024-04-16T21:03:51.395 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=18, errors=1, failures=10) 2024-04-16T21:03:55.871 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multisite_tests.py", line 96, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-16T21:04:04.543 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b8fd33bc95804001836289ef42f77c90 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw_multisite_tests.py", line 96, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-16T21:04:04.601 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-16T21:04:05.459 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-16T21:04:05.479 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-16T21:04:05.503 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-16T21:04:05.545 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:04:06.205 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T21:04:05.520+0000 7fab226e3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 41685) UID: 0 2024-04-16T21:04:06.206 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2024-04-16T21:04:05.520+0000 7fab25f34880 -1 shutting down 2024-04-16T21:04:08.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-16T21:04:12.265 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-16T21:04:12.330 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-16T21:04:12.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/vault-root-token 2024-04-16T21:04:12.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-04-16T21:04:12.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['gc', 'process', '--include-all'] 2024-04-16T21:04:12.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'gc', 'process', '--include-all'] 2024-04-16T21:04:12.499 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 gc process --include-all 2024-04-16T21:04:12.585 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2024-04-16T21:04:12.585 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T21:04:12.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.634+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.635+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:12.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.635+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.635+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:12.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.635+0000 7fde3852a800 20 realm test-realm 82a6ab2b-1604-42ea-987b-3570b8df3977 2024-04-16T21:04:12.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.635+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.636+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:12.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.636+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=10 2024-04-16T21:04:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=1885 2024-04-16T21:04:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:12.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:12.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.637+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 rados_obj.operate() r=0 bl.length=1165 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 zone test-zone1 found 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 4 Realm: test-realm (82a6ab2b-1604-42ea-987b-3570b8df3977) 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 4 ZoneGroup: test-zonegroup (test-zonegroup) 2024-04-16T21:04:12.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 4 Zone: test-zone1 (b482a2a9-ef36-488b-9666-ada656453328) 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 4 using period configuration: f1f5b060-9fe8-4227-8567-18236dc68d0a:11 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 period zonegroup name test-zonegroup 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 using current period zonegroup test-zonegroup 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.638+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.639+0000 7fde3852a800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.639+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:12.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 init(): pipe_map (zone=b482a2a9-ef36-488b-9666-ada656453328 bucket=*): adding potential pipe: {id=sync-pipe,s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea}} 2024-04-16T21:04:12.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.640+0000 7fde3852a800 20 rados->read ofs=0 len=0 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 init(): pipe_map (zone=eafd0fcf-abef-4654-893b-ccd3b2eb5bea bucket=*): adding potential pipe: {id=sync-pipe,s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea}} 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:12.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 started sync module instance, tier type = 2024-04-16T21:04:12.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 generating connection object for zone test-zone2 id eafd0fcf-abef-4654-893b-ccd3b2eb5bea 2024-04-16T21:04:12.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.641+0000 7fde3852a800 20 started zone id=b482a2a9-ef36-488b-9666-ada656453328 (name=test-zone1) with tier type = 2024-04-16T21:04:12.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=0 2024-04-16T21:04:12.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=1 2024-04-16T21:04:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=2 2024-04-16T21:04:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=3 2024-04-16T21:04:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=4 2024-04-16T21:04:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=5 2024-04-16T21:04:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.652+0000 7fde3852a800 20 add_watcher() i=6 2024-04-16T21:04:12.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.653+0000 7fde3852a800 20 add_watcher() i=7 2024-04-16T21:04:12.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.653+0000 7fde3852a800 2 all 8 watchers are set, enabling cache 2024-04-16T21:04:12.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.655+0000 7fdceafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T21:04:12.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.655+0000 7fde3852a800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T21:04:12.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.655+0000 7fde3852a800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T21:04:12.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.701+0000 7fdce9fe3640 20 reqs_thread_entry: start 2024-04-16T21:04:12.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.701+0000 7fde3852a800 20 init_complete bucket index max shards: 11 2024-04-16T21:04:12.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.701+0000 7fde3852a800 20 Filter name: none 2024-04-16T21:04:12.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.701+0000 7fdce2ffd640 20 reqs_thread_entry: start 2024-04-16T21:04:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.701+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-04-16T21:04:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.703+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.704+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.704+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.705+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-04-16T21:04:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.707+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.708+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.708+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.709+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-04-16T21:04:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.711+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.711+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.711+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.712+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-04-16T21:04:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.713+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.714+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.714+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.715+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-04-16T21:04:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.717+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.717+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.717+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.719+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-04-16T21:04:12.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.720+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.720+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.721+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.722+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-04-16T21:04:12.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.723+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.724+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.724+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.725+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-04-16T21:04:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.726+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.727+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.727+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.728+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-04-16T21:04:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.729+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.730+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.730+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.731+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-04-16T21:04:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.733+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.733+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.733+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.734+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-04-16T21:04:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.735+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.736+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.736+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.737+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-04-16T21:04:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.738+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.739+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.739+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.740+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-04-16T21:04:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.742+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.742+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.742+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.743+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-04-16T21:04:12.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.745+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.745+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.745+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.746+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-04-16T21:04:12.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.748+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.748+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.748+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.749+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-04-16T21:04:12.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.750+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.751+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.751+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.752+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-04-16T21:04:12.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.754+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.754+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.754+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.755+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-04-16T21:04:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.757+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.758+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.758+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.759+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-04-16T21:04:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.761+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.761+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.762+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.763+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-04-16T21:04:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.764+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.765+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.765+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.766+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-04-16T21:04:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.767+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.768+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.768+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.769+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-04-16T21:04:12.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.771+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.771+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.771+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.772+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-04-16T21:04:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.774+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.774+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.774+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.776+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-04-16T21:04:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.777+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.778+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.778+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.779+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-04-16T21:04:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.780+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.781+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.781+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.782+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-04-16T21:04:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.783+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.784+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.784+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.785+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-04-16T21:04:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.786+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.786+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.786+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.787+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-04-16T21:04:12.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.788+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.789+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.789+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.790+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-04-16T21:04:12.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.792+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.792+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.792+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.793+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-04-16T21:04:12.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.795+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.795+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.795+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.796+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-04-16T21:04:12.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.797+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.798+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.798+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.799+0000 7fde3852a800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-04-16T21:04:12.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.800+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.800+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:12.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.801+0000 7fde3852a800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:12.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.802+0000 7fdceafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T21:04:12.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.805+0000 7fde3852a800 20 remove_watcher() i=0 2024-04-16T21:04:12.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.805+0000 7fde3852a800 2 removed watcher, disabling cache 2024-04-16T21:04:12.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.806+0000 7fde3852a800 20 remove_watcher() i=1 2024-04-16T21:04:12.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.807+0000 7fde3852a800 20 remove_watcher() i=2 2024-04-16T21:04:12.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.808+0000 7fde3852a800 20 remove_watcher() i=3 2024-04-16T21:04:12.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.810+0000 7fde3852a800 20 remove_watcher() i=4 2024-04-16T21:04:12.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.811+0000 7fde3852a800 20 remove_watcher() i=5 2024-04-16T21:04:12.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.812+0000 7fde3852a800 20 remove_watcher() i=6 2024-04-16T21:04:12.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-04-16T21:04:12.813+0000 7fde3852a800 20 remove_watcher() i=7 2024-04-16T21:04:12.810 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-16T21:04:12.810 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:04:12.816 INFO:tasks.rgw.c1.client.1.smithi069.stdout:2024-04-16T21:04:12.827+0000 7f7011647640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 36711) UID: 0 2024-04-16T21:04:12.816 INFO:tasks.rgw.c1.client.1.smithi069.stdout:2024-04-16T21:04:12.827+0000 7f7016f22880 -1 shutting down 2024-04-16T21:04:14.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-16T21:04:14.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-16T21:04:18.915 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-16T21:04:18.916 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-16T21:04:18.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/vault-root-token 2024-04-16T21:04:19.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-04-16T21:04:19.089 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['gc', 'process', '--include-all'] 2024-04-16T21:04:19.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'gc', 'process', '--include-all'] 2024-04-16T21:04:19.089 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 gc process --include-all 2024-04-16T21:04:19.176 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setuser ceph since I am not root 2024-04-16T21:04:19.177 INFO:teuthology.orchestra.run.smithi069.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T21:04:19.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.230+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.231+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:19.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.231+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:19.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 realm test-realm 82a6ab2b-1604-42ea-987b-3570b8df3977 2024-04-16T21:04:19.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:19.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=10 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.232+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=1885 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:19.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=110 2024-04-16T21:04:19.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.233+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.234+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=46 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.234+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 rados_obj.operate() r=0 bl.length=1165 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 zone test-zone1 found 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 4 Realm: test-realm (82a6ab2b-1604-42ea-987b-3570b8df3977) 2024-04-16T21:04:19.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 4 ZoneGroup: test-zonegroup (test-zonegroup) 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 4 Zone: test-zone1 (b482a2a9-ef36-488b-9666-ada656453328) 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 4 using period configuration: f1f5b060-9fe8-4227-8567-18236dc68d0a:11 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 period zonegroup name test-zonegroup 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 using current period zonegroup test-zonegroup 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:19.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.235+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 init(): pipe_map (zone=b482a2a9-ef36-488b-9666-ada656453328 bucket=*): adding potential pipe: {id=sync-pipe,s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea}} 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 rados->read ofs=0 len=0 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 rados_obj.operate() r=-2 bl.length=0 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 init(): pipe_map (zone=eafd0fcf-abef-4654-893b-ccd3b2eb5bea bucket=*): adding potential pipe: {id=sync-pipe,s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,eafd0fcf-abef-4654-893b-ccd3b2eb5bea}} 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0},d={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b482a2a9-ef36-488b-9666-ada656453328,az=0},d={b=*,z=eafd0fcf-abef-4654-893b-ccd3b2eb5bea,az=0}} 2024-04-16T21:04:19.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 started sync module instance, tier type = 2024-04-16T21:04:19.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 generating connection object for zone test-zone2 id eafd0fcf-abef-4654-893b-ccd3b2eb5bea 2024-04-16T21:04:19.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.236+0000 7fb4c6f2c800 20 started zone id=b482a2a9-ef36-488b-9666-ada656453328 (name=test-zone1) with tier type = 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=0 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=1 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=2 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=3 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=4 2024-04-16T21:04:19.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=5 2024-04-16T21:04:19.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.247+0000 7fb4c6f2c800 20 add_watcher() i=6 2024-04-16T21:04:19.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.248+0000 7fb4c6f2c800 20 add_watcher() i=7 2024-04-16T21:04:19.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.248+0000 7fb4c6f2c800 2 all 8 watchers are set, enabling cache 2024-04-16T21:04:19.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.251+0000 7fb3797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T21:04:19.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.251+0000 7fb4c6f2c800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-16T21:04:19.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.251+0000 7fb4c6f2c800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-16T21:04:19.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.296+0000 7fb2f3fff640 20 reqs_thread_entry: start 2024-04-16T21:04:19.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.296+0000 7fb4c6f2c800 20 init_complete bucket index max shards: 11 2024-04-16T21:04:19.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.296+0000 7fb4c6f2c800 20 Filter name: none 2024-04-16T21:04:19.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.296+0000 7fb2f1ffb640 20 reqs_thread_entry: start 2024-04-16T21:04:19.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.296+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-04-16T21:04:19.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.298+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.298+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.298+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.299+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-04-16T21:04:19.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.300+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.301+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.301+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.302+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-04-16T21:04:19.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.303+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.303+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.303+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.305+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-04-16T21:04:19.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.306+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.306+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.307+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.308+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-04-16T21:04:19.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.309+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.309+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.309+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.310+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-04-16T21:04:19.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.312+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.312+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.313+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.314+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-04-16T21:04:19.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.317+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.317+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.317+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.318+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-04-16T21:04:19.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.319+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.320+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.320+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.322+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-04-16T21:04:19.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.324+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.324+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.324+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.326+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-04-16T21:04:19.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.327+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.327+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.328+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.329+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-04-16T21:04:19.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.330+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.330+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.331+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.331+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-04-16T21:04:19.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.333+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.333+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.333+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.334+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-04-16T21:04:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.335+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.336+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.336+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.337+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-04-16T21:04:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.338+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.338+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.338+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.340+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-04-16T21:04:19.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.341+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.342+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.342+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.343+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-04-16T21:04:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.344+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.345+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.345+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.346+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-04-16T21:04:19.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.348+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.348+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.348+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.349+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-04-16T21:04:19.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.351+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.351+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.351+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.353+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-04-16T21:04:19.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.354+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.354+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.355+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.355+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-04-16T21:04:19.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.357+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.358+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.359+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.360+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-04-16T21:04:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.361+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.362+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.362+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.363+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-04-16T21:04:19.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.365+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.365+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.365+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.367+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-04-16T21:04:19.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.368+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.368+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.368+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.369+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-04-16T21:04:19.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.370+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.371+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.371+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.372+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-04-16T21:04:19.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.374+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.374+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.374+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.376+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-04-16T21:04:19.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.377+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.377+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.377+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.378+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-04-16T21:04:19.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.379+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.380+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.380+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.381+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-04-16T21:04:19.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.382+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.383+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.383+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.384+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-04-16T21:04:19.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.385+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.386+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.386+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.387+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-04-16T21:04:19.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.388+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.389+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.389+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.390+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-04-16T21:04:19.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.391+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.391+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.392+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.392+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-04-16T21:04:19.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.394+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.394+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-16T21:04:19.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.394+0000 7fb4c6f2c800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-16T21:04:19.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.396+0000 7fb3797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-16T21:04:19.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.398+0000 7fb4c6f2c800 20 remove_watcher() i=0 2024-04-16T21:04:19.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.398+0000 7fb4c6f2c800 2 removed watcher, disabling cache 2024-04-16T21:04:19.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.400+0000 7fb4c6f2c800 20 remove_watcher() i=1 2024-04-16T21:04:19.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.401+0000 7fb4c6f2c800 20 remove_watcher() i=2 2024-04-16T21:04:19.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.402+0000 7fb4c6f2c800 20 remove_watcher() i=3 2024-04-16T21:04:19.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.403+0000 7fb4c6f2c800 20 remove_watcher() i=4 2024-04-16T21:04:19.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.404+0000 7fb4c6f2c800 20 remove_watcher() i=5 2024-04-16T21:04:19.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.405+0000 7fb4c6f2c800 20 remove_watcher() i=6 2024-04-16T21:04:19.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-16T21:04:19.406+0000 7fb4c6f2c800 20 remove_watcher() i=7 2024-04-16T21:04:19.401 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-16T21:04:19.401 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:04:19.402 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-16T21:04:19.402 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-16T21:04:19.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/vault-root-token 2024-04-16T21:04:19.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-04-16T21:04:19.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['gc', 'process', '--include-all'] 2024-04-16T21:04:19.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'gc', 'process', '--include-all'] 2024-04-16T21:04:19.567 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all 2024-04-16T21:04:19.655 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setuser ceph since I am not root 2024-04-16T21:04:19.656 INFO:teuthology.orchestra.run.smithi119.stderr:ignoring --setgroup ceph since I am not root 2024-04-16T21:04:20.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-16T21:09:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:09:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-16T21:09:19.673+0000 7fe813e13800 0 monclient(hunting): authenticate timed out after 300 2024-04-16T21:09:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-16T21:09:19.726 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' 2024-04-16T21:09:19.952 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T21:09:20.036 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-16T21:09:20.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-16T21:11:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T21:11:20.214 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2024-04-16T21:11:20.337 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T21:11:20.368 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T21:11:20.368 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-16T21:11:20.368 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.373 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-16T21:11:20.373 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-16T21:11:20.373 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.373 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-16T21:11:20.374 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-16T21:11:20.374 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.374 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-16T21:11:20.374 DEBUG:tasks.ceph.c2.osd.3:waiting for process to exit 2024-04-16T21:11:20.374 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.375 INFO:tasks.ceph.c2.osd.3:Stopped 2024-04-16T21:11:20.375 DEBUG:tasks.ceph.c2.osd.4:waiting for process to exit 2024-04-16T21:11:20.375 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.375 INFO:tasks.ceph.c2.osd.4:Stopped 2024-04-16T21:11:20.375 DEBUG:tasks.ceph.c2.osd.5:waiting for process to exit 2024-04-16T21:11:20.376 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.376 INFO:tasks.ceph.c2.osd.5:Stopped 2024-04-16T21:11:20.421 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T21:11:20.421 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-16T21:11:20.421 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.424 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-16T21:11:20.440 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T21:11:20.441 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-16T21:11:20.441 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:11:20.441 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-16T21:11:20.459 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T21:11:20.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-16T21:11:20.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T21:11:20.606 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-16T21:11:20.715 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T21:11:20.715 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-16T21:11:20.800 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T21:11:20.800 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-16T21:11:20.865 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-3 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T21:11:20.865 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/c2-3 2024-04-16T21:11:21.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-4 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T21:11:21.022 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/c2-4 2024-04-16T21:11:21.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-5 on ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T21:11:21.084 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/c2-5 2024-04-16T21:11:21.142 INFO:tasks.ceph:Archiving mon data... 2024-04-16T21:11:21.195 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/data/c2.mon.a.tgz 2024-04-16T21:11:21.233 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-04-16T21:11:21.252 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.JNhOOUc9gW 2024-04-16T21:11:21.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.JNhOOUc9gW 2024-04-16T21:11:21.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.JNhOOUc9gW 2024-04-16T21:11:21.841 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.JNhOOUc9gW is 302KB 2024-04-16T21:11:21.985 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.JNhOOUc9gW 2024-04-16T21:11:22.002 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T21:11:22.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:11:22.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:11:22.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:11:22.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:11:22.106 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json' 2024-04-16T21:11:22.114 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T21:11:22.125 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-16T21:11:22.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:11:22.479 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:11:22.499 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:11:22.504 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3274,"stamp":"2024-04-16T21:11:21.319203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30466,"ondisk_log_size":30466,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":307840,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":296630,"kb_avail":628837760,"statfs":{"total":644245094400,"available":643929866240,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":303749793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005593"},"pg_stats":[{"pgid":"9.35","version":"52'28","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.485159+0000","last_change":"2024-04-16T19:25:09.466020+0000","last_active":"2024-04-16T21:06:39.485159+0000","last_peered":"2024-04-16T21:06:39.485159+0000","last_clean":"2024-04-16T21:06:39.485159+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:06:39.485159+0000","last_undegraded":"2024-04-16T21:06:39.485159+0000","last_fullsized":"2024-04-16T21:06:39.485159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365868+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:08:44.365868+0000","last_peered":"2024-04-16T21:08:44.365868+0000","last_clean":"2024-04-16T21:08:44.365868+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:08:44.365868+0000","last_undegraded":"2024-04-16T21:08:44.365868+0000","last_fullsized":"2024-04-16T21:08:44.365868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648735+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:07:09.648735+0000","last_peered":"2024-04-16T21:07:09.648735+0000","last_clean":"2024-04-16T21:07:09.648735+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:07:09.648735+0000","last_undegraded":"2024-04-16T21:07:09.648735+0000","last_fullsized":"2024-04-16T21:07:09.648735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"47'5","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643488+0000","last_change":"2024-04-16T19:25:09.467691+0000","last_active":"2024-04-16T21:06:39.643488+0000","last_peered":"2024-04-16T21:06:39.643488+0000","last_clean":"2024-04-16T21:06:39.643488+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:06:39.643488+0000","last_undegraded":"2024-04-16T21:06:39.643488+0000","last_fullsized":"2024-04-16T21:06:39.643488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364114+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:08:44.364114+0000","last_peered":"2024-04-16T21:08:44.364114+0000","last_clean":"2024-04-16T21:08:44.364114+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:08:44.364114+0000","last_undegraded":"2024-04-16T21:08:44.364114+0000","last_fullsized":"2024-04-16T21:08:44.364114+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649318+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:07:09.649318+0000","last_peered":"2024-04-16T21:07:09.649318+0000","last_clean":"2024-04-16T21:07:09.649318+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:07:09.649318+0000","last_undegraded":"2024-04-16T21:07:09.649318+0000","last_fullsized":"2024-04-16T21:07:09.649318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'13","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646812+0000","last_change":"2024-04-16T19:25:09.457429+0000","last_active":"2024-04-16T21:06:54.646812+0000","last_peered":"2024-04-16T21:06:54.646812+0000","last_clean":"2024-04-16T21:06:54.646812+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:06:54.646812+0000","last_undegraded":"2024-04-16T21:06:54.646812+0000","last_fullsized":"2024-04-16T21:06:54.646812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369756+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:08:44.369756+0000","last_peered":"2024-04-16T21:08:44.369756+0000","last_clean":"2024-04-16T21:08:44.369756+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:08:44.369756+0000","last_undegraded":"2024-04-16T21:08:44.369756+0000","last_fullsized":"2024-04-16T21:08:44.369756+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:11.722618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649006+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:07:09.649006+0000","last_peered":"2024-04-16T21:07:09.649006+0000","last_clean":"2024-04-16T21:07:09.649006+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:07:09.649006+0000","last_undegraded":"2024-04-16T21:07:09.649006+0000","last_fullsized":"2024-04-16T21:07:09.649006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'13","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.485201+0000","last_change":"2024-04-16T19:25:09.467050+0000","last_active":"2024-04-16T21:06:39.485201+0000","last_peered":"2024-04-16T21:06:39.485201+0000","last_clean":"2024-04-16T21:06:39.485201+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:06:39.485201+0000","last_undegraded":"2024-04-16T21:06:39.485201+0000","last_fullsized":"2024-04-16T21:06:39.485201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366441+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:08:54.366441+0000","last_peered":"2024-04-16T21:08:54.366441+0000","last_clean":"2024-04-16T21:08:54.366441+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:08:54.366441+0000","last_undegraded":"2024-04-16T21:08:54.366441+0000","last_fullsized":"2024-04-16T21:08:54.366441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648038+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:07:09.648038+0000","last_peered":"2024-04-16T21:07:09.648038+0000","last_clean":"2024-04-16T21:07:09.648038+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:07:09.648038+0000","last_undegraded":"2024-04-16T21:07:09.648038+0000","last_fullsized":"2024-04-16T21:07:09.648038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'6","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655139+0000","last_change":"2024-04-16T19:25:09.459828+0000","last_active":"2024-04-16T21:07:49.655139+0000","last_peered":"2024-04-16T21:07:49.655139+0000","last_clean":"2024-04-16T21:07:49.655139+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:07:49.655139+0000","last_undegraded":"2024-04-16T21:07:49.655139+0000","last_fullsized":"2024-04-16T21:07:49.655139+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648499+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:07:04.648499+0000","last_peered":"2024-04-16T21:07:04.648499+0000","last_clean":"2024-04-16T21:07:04.648499+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:07:04.648499+0000","last_undegraded":"2024-04-16T21:07:04.648499+0000","last_fullsized":"2024-04-16T21:07:04.648499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367192+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:08:59.367192+0000","last_peered":"2024-04-16T21:08:59.367192+0000","last_clean":"2024-04-16T21:08:59.367192+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:08:59.367192+0000","last_undegraded":"2024-04-16T21:08:59.367192+0000","last_fullsized":"2024-04-16T21:08:59.367192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'18","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654366+0000","last_change":"2024-04-16T19:25:09.461822+0000","last_active":"2024-04-16T21:07:49.654366+0000","last_peered":"2024-04-16T21:07:49.654366+0000","last_clean":"2024-04-16T21:07:49.654366+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:07:49.654366+0000","last_undegraded":"2024-04-16T21:07:49.654366+0000","last_fullsized":"2024-04-16T21:07:49.654366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647976+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:07:04.647976+0000","last_peered":"2024-04-16T21:07:04.647976+0000","last_clean":"2024-04-16T21:07:04.647976+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:07:04.647976+0000","last_undegraded":"2024-04-16T21:07:04.647976+0000","last_fullsized":"2024-04-16T21:07:04.647976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:07.917853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365518+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:08:49.365518+0000","last_peered":"2024-04-16T21:08:49.365518+0000","last_clean":"2024-04-16T21:08:49.365518+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:08:49.365518+0000","last_undegraded":"2024-04-16T21:08:49.365518+0000","last_fullsized":"2024-04-16T21:08:49.365518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649782+0000","last_change":"2024-04-16T19:25:41.508581+0000","last_active":"2024-04-16T21:07:14.649782+0000","last_peered":"2024-04-16T21:07:14.649782+0000","last_clean":"2024-04-16T21:07:14.649782+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:07:14.649782+0000","last_undegraded":"2024-04-16T21:07:14.649782+0000","last_fullsized":"2024-04-16T21:07:14.649782+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:12.396788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648409+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:07:04.648409+0000","last_peered":"2024-04-16T21:07:04.648409+0000","last_clean":"2024-04-16T21:07:04.648409+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:07:04.648409+0000","last_undegraded":"2024-04-16T21:07:04.648409+0000","last_fullsized":"2024-04-16T21:07:04.648409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:34.027154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365507+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:08:49.365507+0000","last_peered":"2024-04-16T21:08:49.365507+0000","last_clean":"2024-04-16T21:08:49.365507+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:08:49.365507+0000","last_undegraded":"2024-04-16T21:08:49.365507+0000","last_fullsized":"2024-04-16T21:08:49.365507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'15","reported_seq":238,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657926+0000","last_change":"2024-04-16T19:25:09.463558+0000","last_active":"2024-04-16T21:08:04.657926+0000","last_peered":"2024-04-16T21:08:04.657926+0000","last_clean":"2024-04-16T21:08:04.657926+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:08:04.657926+0000","last_undegraded":"2024-04-16T21:08:04.657926+0000","last_fullsized":"2024-04-16T21:08:04.657926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489018+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:07:04.489018+0000","last_peered":"2024-04-16T21:07:04.489018+0000","last_clean":"2024-04-16T21:07:04.489018+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:07:04.489018+0000","last_undegraded":"2024-04-16T21:07:04.489018+0000","last_fullsized":"2024-04-16T21:07:04.489018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506158+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:08:49.506158+0000","last_peered":"2024-04-16T21:08:49.506158+0000","last_clean":"2024-04-16T21:08:49.506158+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:08:49.506158+0000","last_undegraded":"2024-04-16T21:08:49.506158+0000","last_fullsized":"2024-04-16T21:08:49.506158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'7","reported_seq":220,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649667+0000","last_change":"2024-04-16T19:25:09.467094+0000","last_active":"2024-04-16T21:07:14.649667+0000","last_peered":"2024-04-16T21:07:14.649667+0000","last_clean":"2024-04-16T21:07:14.649667+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:07:14.649667+0000","last_undegraded":"2024-04-16T21:07:14.649667+0000","last_fullsized":"2024-04-16T21:07:14.649667+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:52.158151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650393+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:07:19.650393+0000","last_peered":"2024-04-16T21:07:19.650393+0000","last_clean":"2024-04-16T21:07:19.650393+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:07:19.650393+0000","last_undegraded":"2024-04-16T21:07:19.650393+0000","last_fullsized":"2024-04-16T21:07:19.650393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366531+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:08:49.366531+0000","last_peered":"2024-04-16T21:08:49.366531+0000","last_clean":"2024-04-16T21:08:49.366531+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:08:49.366531+0000","last_undegraded":"2024-04-16T21:08:49.366531+0000","last_fullsized":"2024-04-16T21:08:49.366531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'6","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651913+0000","last_change":"2024-04-16T19:25:09.459483+0000","last_active":"2024-04-16T21:07:24.651913+0000","last_peered":"2024-04-16T21:07:24.651913+0000","last_clean":"2024-04-16T21:07:24.651913+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:07:24.651913+0000","last_undegraded":"2024-04-16T21:07:24.651913+0000","last_fullsized":"2024-04-16T21:07:24.651913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650394+0000","last_change":"2024-04-16T19:25:41.503681+0000","last_active":"2024-04-16T21:07:19.650394+0000","last_peered":"2024-04-16T21:07:19.650394+0000","last_clean":"2024-04-16T21:07:19.650394+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:07:19.650394+0000","last_undegraded":"2024-04-16T21:07:19.650394+0000","last_fullsized":"2024-04-16T21:07:19.650394+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:33:58.945193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662645+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:08:29.662645+0000","last_peered":"2024-04-16T21:08:29.662645+0000","last_clean":"2024-04-16T21:08:29.662645+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:08:29.662645+0000","last_undegraded":"2024-04-16T21:08:29.662645+0000","last_fullsized":"2024-04-16T21:08:29.662645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'5","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490935+0000","last_change":"2024-04-16T19:25:09.464787+0000","last_active":"2024-04-16T21:07:14.490935+0000","last_peered":"2024-04-16T21:07:14.490935+0000","last_clean":"2024-04-16T21:07:14.490935+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:07:14.490935+0000","last_undegraded":"2024-04-16T21:07:14.490935+0000","last_fullsized":"2024-04-16T21:07:14.490935+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652786+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:07:34.652786+0000","last_peered":"2024-04-16T21:07:34.652786+0000","last_clean":"2024-04-16T21:07:34.652786+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:07:34.652786+0000","last_undegraded":"2024-04-16T21:07:34.652786+0000","last_fullsized":"2024-04-16T21:07:34.652786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662148+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:08:29.662148+0000","last_peered":"2024-04-16T21:08:29.662148+0000","last_clean":"2024-04-16T21:08:29.662148+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:08:29.662148+0000","last_undegraded":"2024-04-16T21:08:29.662148+0000","last_fullsized":"2024-04-16T21:08:29.662148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490887+0000","last_change":"2024-04-16T19:25:09.463733+0000","last_active":"2024-04-16T21:07:14.490887+0000","last_peered":"2024-04-16T21:07:14.490887+0000","last_clean":"2024-04-16T21:07:14.490887+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:07:14.490887+0000","last_undegraded":"2024-04-16T21:07:14.490887+0000","last_fullsized":"2024-04-16T21:07:14.490887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:40:59.948853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503495+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:08:34.503495+0000","last_peered":"2024-04-16T21:08:34.503495+0000","last_clean":"2024-04-16T21:08:34.503495+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:08:34.503495+0000","last_undegraded":"2024-04-16T21:08:34.503495+0000","last_fullsized":"2024-04-16T21:08:34.503495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:23.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661720+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:08:29.661720+0000","last_peered":"2024-04-16T21:08:29.661720+0000","last_clean":"2024-04-16T21:08:29.661720+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:08:29.661720+0000","last_undegraded":"2024-04-16T21:08:29.661720+0000","last_fullsized":"2024-04-16T21:08:29.661720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:09:50.552696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'9","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654268+0000","last_change":"2024-04-16T19:25:09.462286+0000","last_active":"2024-04-16T21:07:39.654268+0000","last_peered":"2024-04-16T21:07:39.654268+0000","last_clean":"2024-04-16T21:07:39.654268+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:07:39.654268+0000","last_undegraded":"2024-04-16T21:07:39.654268+0000","last_fullsized":"2024-04-16T21:07:39.654268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503526+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:08:34.503526+0000","last_peered":"2024-04-16T21:08:34.503526+0000","last_clean":"2024-04-16T21:08:34.503526+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:08:34.503526+0000","last_undegraded":"2024-04-16T21:08:34.503526+0000","last_fullsized":"2024-04-16T21:08:34.503526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661148+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:08:29.661148+0000","last_peered":"2024-04-16T21:08:29.661148+0000","last_clean":"2024-04-16T21:08:29.661148+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:08:29.661148+0000","last_undegraded":"2024-04-16T21:08:29.661148+0000","last_fullsized":"2024-04-16T21:08:29.661148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'15","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494683+0000","last_change":"2024-04-16T19:25:09.461082+0000","last_active":"2024-04-16T21:07:39.494683+0000","last_peered":"2024-04-16T21:07:39.494683+0000","last_clean":"2024-04-16T21:07:39.494683+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:07:39.494683+0000","last_undegraded":"2024-04-16T21:07:39.494683+0000","last_fullsized":"2024-04-16T21:07:39.494683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662702+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:08:34.662702+0000","last_peered":"2024-04-16T21:08:34.662702+0000","last_clean":"2024-04-16T21:08:34.662702+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:08:34.662702+0000","last_undegraded":"2024-04-16T21:08:34.662702+0000","last_fullsized":"2024-04-16T21:08:34.662702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651169+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:07:29.651169+0000","last_peered":"2024-04-16T21:07:29.651169+0000","last_clean":"2024-04-16T21:07:29.651169+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:07:29.651169+0000","last_undegraded":"2024-04-16T21:07:29.651169+0000","last_fullsized":"2024-04-16T21:07:29.651169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365000+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:08:49.365000+0000","last_peered":"2024-04-16T21:08:49.365000+0000","last_clean":"2024-04-16T21:08:49.365000+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:08:49.365000+0000","last_undegraded":"2024-04-16T21:08:49.365000+0000","last_fullsized":"2024-04-16T21:08:49.365000+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:32.817843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1720,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659053+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:08:04.659053+0000","last_peered":"2024-04-16T21:08:04.659053+0000","last_clean":"2024-04-16T21:08:04.659053+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:08:04.659053+0000","last_undegraded":"2024-04-16T21:08:04.659053+0000","last_fullsized":"2024-04-16T21:08:04.659053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660478+0000","last_change":"2024-04-16T19:25:41.505822+0000","last_active":"2024-04-16T21:08:14.660478+0000","last_peered":"2024-04-16T21:08:14.660478+0000","last_clean":"2024-04-16T21:08:14.660478+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:08:14.660478+0000","last_undegraded":"2024-04-16T21:08:14.660478+0000","last_fullsized":"2024-04-16T21:08:14.660478+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'6","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654478+0000","last_change":"2024-04-16T19:25:15.486584+0000","last_active":"2024-04-16T21:07:49.654478+0000","last_peered":"2024-04-16T21:07:49.654478+0000","last_clean":"2024-04-16T21:07:49.654478+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:07:49.654478+0000","last_undegraded":"2024-04-16T21:07:49.654478+0000","last_fullsized":"2024-04-16T21:07:49.654478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654387+0000","last_change":"2024-04-16T19:25:13.469257+0000","last_active":"2024-04-16T21:07:44.654387+0000","last_peered":"2024-04-16T21:07:44.654387+0000","last_clean":"2024-04-16T21:07:44.654387+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:07:44.654387+0000","last_undegraded":"2024-04-16T21:07:44.654387+0000","last_fullsized":"2024-04-16T21:07:44.654387+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'203","reported_seq":447,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655143+0000","last_change":"2024-04-16T19:25:11.464182+0000","last_active":"2024-04-16T21:07:44.655143+0000","last_peered":"2024-04-16T21:07:44.655143+0000","last_clean":"2024-04-16T21:07:44.655143+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:07:44.655143+0000","last_undegraded":"2024-04-16T21:07:44.655143+0000","last_fullsized":"2024-04-16T21:07:44.655143+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:07.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'19","reported_seq":284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487393+0000","last_change":"2024-04-16T19:25:09.464946+0000","last_active":"2024-04-16T21:06:54.487393+0000","last_peered":"2024-04-16T21:06:54.487393+0000","last_clean":"2024-04-16T21:06:54.487393+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:06:54.487393+0000","last_undegraded":"2024-04-16T21:06:54.487393+0000","last_fullsized":"2024-04-16T21:06:54.487393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:45:45.734532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365360+0000","last_change":"2024-04-16T19:24:51.436483+0000","last_active":"2024-04-16T21:08:49.365360+0000","last_peered":"2024-04-16T21:08:49.365360+0000","last_clean":"2024-04-16T21:08:49.365360+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:08:49.365360+0000","last_undegraded":"2024-04-16T21:08:49.365360+0000","last_fullsized":"2024-04-16T21:08:49.365360+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:40.603125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494096+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:07:34.494096+0000","last_peered":"2024-04-16T21:07:34.494096+0000","last_clean":"2024-04-16T21:07:34.494096+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:07:34.494096+0000","last_undegraded":"2024-04-16T21:07:34.494096+0000","last_fullsized":"2024-04-16T21:07:34.494096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:52.988517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650257+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:07:14.650257+0000","last_peered":"2024-04-16T21:07:14.650257+0000","last_clean":"2024-04-16T21:07:14.650257+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:07:14.650257+0000","last_undegraded":"2024-04-16T21:07:14.650257+0000","last_fullsized":"2024-04-16T21:07:14.650257+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:07.194576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:42:58.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369810+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:08:44.369810+0000","last_peered":"2024-04-16T21:08:44.369810+0000","last_clean":"2024-04-16T21:08:44.369810+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:08:44.369810+0000","last_undegraded":"2024-04-16T21:08:44.369810+0000","last_fullsized":"2024-04-16T21:08:44.369810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:08:44.364641+0000","last_peered":"2024-04-16T21:08:44.364641+0000","last_clean":"2024-04-16T21:08:44.364641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:08:44.364641+0000","last_undegraded":"2024-04-16T21:08:44.364641+0000","last_fullsized":"2024-04-16T21:08:44.364641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:42:28.507077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496581+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:07:49.496581+0000","last_peered":"2024-04-16T21:07:49.496581+0000","last_clean":"2024-04-16T21:07:49.496581+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:07:49.496581+0000","last_undegraded":"2024-04-16T21:07:49.496581+0000","last_fullsized":"2024-04-16T21:07:49.496581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:43.119570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1836,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365530+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:08:49.365530+0000","last_peered":"2024-04-16T21:08:49.365530+0000","last_clean":"2024-04-16T21:08:49.365530+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:08:49.365530+0000","last_undegraded":"2024-04-16T21:08:49.365530+0000","last_fullsized":"2024-04-16T21:08:49.365530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659544+0000","last_change":"2024-04-16T19:25:41.503686+0000","last_active":"2024-04-16T21:08:14.659544+0000","last_peered":"2024-04-16T21:08:14.659544+0000","last_clean":"2024-04-16T21:08:14.659544+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:08:14.659544+0000","last_undegraded":"2024-04-16T21:08:14.659544+0000","last_fullsized":"2024-04-16T21:08:14.659544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:30:44.336557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'5","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660521+0000","last_change":"2024-04-16T19:25:41.505903+0000","last_active":"2024-04-16T21:08:14.660521+0000","last_peered":"2024-04-16T21:08:14.660521+0000","last_clean":"2024-04-16T21:08:14.660521+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:08:14.660521+0000","last_undegraded":"2024-04-16T21:08:14.660521+0000","last_fullsized":"2024-04-16T21:08:14.660521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:37.409520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654445+0000","last_change":"2024-04-16T19:25:13.471998+0000","last_active":"2024-04-16T21:07:44.654445+0000","last_peered":"2024-04-16T21:07:44.654445+0000","last_clean":"2024-04-16T21:07:44.654445+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:07:44.654445+0000","last_undegraded":"2024-04-16T21:07:44.654445+0000","last_fullsized":"2024-04-16T21:07:44.654445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:15.038784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'241","reported_seq":976,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495593+0000","last_change":"2024-04-16T19:25:11.465710+0000","last_active":"2024-04-16T21:07:44.495593+0000","last_peered":"2024-04-16T21:07:44.495593+0000","last_clean":"2024-04-16T21:07:44.495593+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:07:44.495593+0000","last_undegraded":"2024-04-16T21:07:44.495593+0000","last_fullsized":"2024-04-16T21:07:44.495593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:25.422152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'5","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643326+0000","last_change":"2024-04-16T19:25:09.467837+0000","last_active":"2024-04-16T21:06:39.643326+0000","last_peered":"2024-04-16T21:06:39.643326+0000","last_clean":"2024-04-16T21:06:39.643326+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:06:39.643326+0000","last_undegraded":"2024-04-16T21:06:39.643326+0000","last_fullsized":"2024-04-16T21:06:39.643326+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643388+0000","last_change":"2024-04-16T19:24:51.438136+0000","last_active":"2024-04-16T21:06:39.643388+0000","last_peered":"2024-04-16T21:06:39.643388+0000","last_clean":"2024-04-16T21:06:39.643388+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:06:39.643388+0000","last_undegraded":"2024-04-16T21:06:39.643388+0000","last_fullsized":"2024-04-16T21:06:39.643388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663065+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:08:34.663065+0000","last_peered":"2024-04-16T21:08:34.663065+0000","last_clean":"2024-04-16T21:08:34.663065+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:08:34.663065+0000","last_undegraded":"2024-04-16T21:08:34.663065+0000","last_fullsized":"2024-04-16T21:08:34.663065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:39.643446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648771+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:07:14.648771+0000","last_peered":"2024-04-16T21:07:14.648771+0000","last_clean":"2024-04-16T21:07:14.648771+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:07:14.648771+0000","last_undegraded":"2024-04-16T21:07:14.648771+0000","last_fullsized":"2024-04-16T21:07:14.648771+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:19.314716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653671+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:07:44.653671+0000","last_peered":"2024-04-16T21:07:44.653671+0000","last_clean":"2024-04-16T21:07:44.653671+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:07:44.653671+0000","last_undegraded":"2024-04-16T21:07:44.653671+0000","last_fullsized":"2024-04-16T21:07:44.653671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:26.398044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364812+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:08:44.364812+0000","last_peered":"2024-04-16T21:08:44.364812+0000","last_clean":"2024-04-16T21:08:44.364812+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:08:44.364812+0000","last_undegraded":"2024-04-16T21:08:44.364812+0000","last_fullsized":"2024-04-16T21:08:44.364812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:34.874297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364788+0000","last_change":"2024-04-16T19:24:49.426639+0000","last_active":"2024-04-16T21:08:44.364788+0000","last_peered":"2024-04-16T21:08:44.364788+0000","last_clean":"2024-04-16T21:08:44.364788+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:08:44.364788+0000","last_undegraded":"2024-04-16T21:08:44.364788+0000","last_fullsized":"2024-04-16T21:08:44.364788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:38.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652789+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:07:34.652789+0000","last_peered":"2024-04-16T21:07:34.652789+0000","last_clean":"2024-04-16T21:07:34.652789+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:07:34.652789+0000","last_undegraded":"2024-04-16T21:07:34.652789+0000","last_fullsized":"2024-04-16T21:07:34.652789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.732507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1871,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659473+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:08:19.659473+0000","last_peered":"2024-04-16T21:08:19.659473+0000","last_clean":"2024-04-16T21:08:19.659473+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:08:19.659473+0000","last_undegraded":"2024-04-16T21:08:19.659473+0000","last_fullsized":"2024-04-16T21:08:19.659473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:55.150921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655316+0000","last_change":"2024-04-16T19:25:41.502400+0000","last_active":"2024-04-16T21:07:49.655316+0000","last_peered":"2024-04-16T21:07:49.655316+0000","last_clean":"2024-04-16T21:07:49.655316+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:07:49.655316+0000","last_undegraded":"2024-04-16T21:07:49.655316+0000","last_fullsized":"2024-04-16T21:07:49.655316+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:02.633032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'17","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652839+0000","last_change":"2024-04-16T19:25:15.480449+0000","last_active":"2024-04-16T21:07:34.652839+0000","last_peered":"2024-04-16T21:07:34.652839+0000","last_clean":"2024-04-16T21:07:34.652839+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:07:34.652839+0000","last_undegraded":"2024-04-16T21:07:34.652839+0000","last_fullsized":"2024-04-16T21:07:34.652839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.646356+0000","last_change":"2024-04-16T19:25:13.475293+0000","last_active":"2024-04-16T21:06:59.646356+0000","last_peered":"2024-04-16T21:06:59.646356+0000","last_clean":"2024-04-16T21:06:59.646356+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:06:59.646356+0000","last_undegraded":"2024-04-16T21:06:59.646356+0000","last_fullsized":"2024-04-16T21:06:59.646356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'696","reported_seq":41908,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485741+0000","last_change":"2024-04-16T19:25:11.465229+0000","last_active":"2024-04-16T21:06:44.485741+0000","last_peered":"2024-04-16T21:06:44.485741+0000","last_clean":"2024-04-16T21:06:44.485741+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:06:44.485741+0000","last_undegraded":"2024-04-16T21:06:44.485741+0000","last_fullsized":"2024-04-16T21:06:44.485741+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:43.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'25","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652851+0000","last_change":"2024-04-16T19:25:09.464644+0000","last_active":"2024-04-16T21:07:39.652851+0000","last_peered":"2024-04-16T21:07:39.652851+0000","last_clean":"2024-04-16T21:07:39.652851+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:07:39.652851+0000","last_undegraded":"2024-04-16T21:07:39.652851+0000","last_fullsized":"2024-04-16T21:07:39.652851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:46:24.560604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650411+0000","last_change":"2024-04-16T19:24:51.438116+0000","last_active":"2024-04-16T21:07:24.650411+0000","last_peered":"2024-04-16T21:07:24.650411+0000","last_clean":"2024-04-16T21:07:24.650411+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:07:24.650411+0000","last_undegraded":"2024-04-16T21:07:24.650411+0000","last_fullsized":"2024-04-16T21:07:24.650411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662083+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:08:34.662083+0000","last_peered":"2024-04-16T21:08:34.662083+0000","last_clean":"2024-04-16T21:08:34.662083+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:08:34.662083+0000","last_undegraded":"2024-04-16T21:08:34.662083+0000","last_fullsized":"2024-04-16T21:08:34.662083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650147+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:07:14.650147+0000","last_peered":"2024-04-16T21:07:14.650147+0000","last_clean":"2024-04-16T21:07:14.650147+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:07:14.650147+0000","last_undegraded":"2024-04-16T21:07:14.650147+0000","last_fullsized":"2024-04-16T21:07:14.650147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:05.587315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505369+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:08:44.505369+0000","last_peered":"2024-04-16T21:08:44.505369+0000","last_clean":"2024-04-16T21:08:44.505369+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:08:44.505369+0000","last_undegraded":"2024-04-16T21:08:44.505369+0000","last_fullsized":"2024-04-16T21:08:44.505369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:53.769511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":274,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495624+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:07:44.495624+0000","last_peered":"2024-04-16T21:07:44.495624+0000","last_clean":"2024-04-16T21:07:44.495624+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:07:44.495624+0000","last_undegraded":"2024-04-16T21:07:44.495624+0000","last_fullsized":"2024-04-16T21:07:44.495624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:44.851136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'12","reported_seq":1308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365800+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:08:44.365800+0000","last_peered":"2024-04-16T21:08:44.365800+0000","last_clean":"2024-04-16T21:08:44.365800+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:08:44.365800+0000","last_undegraded":"2024-04-16T21:08:44.365800+0000","last_fullsized":"2024-04-16T21:08:44.365800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:40.173753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":72,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494076+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:07:34.494076+0000","last_peered":"2024-04-16T21:07:34.494076+0000","last_clean":"2024-04-16T21:07:34.494076+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:07:34.494076+0000","last_undegraded":"2024-04-16T21:07:34.494076+0000","last_fullsized":"2024-04-16T21:07:34.494076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:53.541439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1472,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659510+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:08:19.659510+0000","last_peered":"2024-04-16T21:08:19.659510+0000","last_clean":"2024-04-16T21:08:19.659510+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:08:19.659510+0000","last_undegraded":"2024-04-16T21:08:19.659510+0000","last_fullsized":"2024-04-16T21:08:19.659510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:30.342275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657195+0000","last_change":"2024-04-16T19:25:41.501744+0000","last_active":"2024-04-16T21:07:59.657195+0000","last_peered":"2024-04-16T21:07:59.657195+0000","last_clean":"2024-04-16T21:07:59.657195+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:07:59.657195+0000","last_undegraded":"2024-04-16T21:07:59.657195+0000","last_fullsized":"2024-04-16T21:07:59.657195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:33.802040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'9","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.652032+0000","last_change":"2024-04-16T19:25:15.487126+0000","last_active":"2024-04-16T21:07:24.652032+0000","last_peered":"2024-04-16T21:07:24.652032+0000","last_clean":"2024-04-16T21:07:24.652032+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:07:24.652032+0000","last_undegraded":"2024-04-16T21:07:24.652032+0000","last_fullsized":"2024-04-16T21:07:24.652032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485708+0000","last_change":"2024-04-16T19:25:13.474735+0000","last_active":"2024-04-16T21:06:44.485708+0000","last_peered":"2024-04-16T21:06:44.485708+0000","last_clean":"2024-04-16T21:06:44.485708+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:06:44.485708+0000","last_undegraded":"2024-04-16T21:06:44.485708+0000","last_fullsized":"2024-04-16T21:06:44.485708+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'339","reported_seq":652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488172+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:06:59.488172+0000","last_peered":"2024-04-16T21:06:59.488172+0000","last_clean":"2024-04-16T21:06:59.488172+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:06:59.488172+0000","last_undegraded":"2024-04-16T21:06:59.488172+0000","last_fullsized":"2024-04-16T21:06:59.488172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:21.744422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'10","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494764+0000","last_change":"2024-04-16T19:25:09.468124+0000","last_active":"2024-04-16T21:07:39.494764+0000","last_peered":"2024-04-16T21:07:39.494764+0000","last_clean":"2024-04-16T21:07:39.494764+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:07:39.494764+0000","last_undegraded":"2024-04-16T21:07:39.494764+0000","last_fullsized":"2024-04-16T21:07:39.494764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:31.945014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651298+0000","last_change":"2024-04-16T19:24:51.572963+0000","last_active":"2024-04-16T21:07:24.651298+0000","last_peered":"2024-04-16T21:07:24.651298+0000","last_clean":"2024-04-16T21:07:24.651298+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:07:24.651298+0000","last_undegraded":"2024-04-16T21:07:24.651298+0000","last_fullsized":"2024-04-16T21:07:24.651298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:46.905067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662114+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:08:34.662114+0000","last_peered":"2024-04-16T21:08:34.662114+0000","last_clean":"2024-04-16T21:08:34.662114+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:08:34.662114+0000","last_undegraded":"2024-04-16T21:08:34.662114+0000","last_fullsized":"2024-04-16T21:08:34.662114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649505+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:07:14.649505+0000","last_peered":"2024-04-16T21:07:14.649505+0000","last_clean":"2024-04-16T21:07:14.649505+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:07:14.649505+0000","last_undegraded":"2024-04-16T21:07:14.649505+0000","last_fullsized":"2024-04-16T21:07:14.649505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365827+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:08:44.365827+0000","last_peered":"2024-04-16T21:08:44.365827+0000","last_clean":"2024-04-16T21:08:44.365827+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:08:44.365827+0000","last_undegraded":"2024-04-16T21:08:44.365827+0000","last_fullsized":"2024-04-16T21:08:44.365827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:41.163065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365785+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:08:44.365785+0000","last_peered":"2024-04-16T21:08:44.365785+0000","last_clean":"2024-04-16T21:08:44.365785+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:08:44.365785+0000","last_undegraded":"2024-04-16T21:08:44.365785+0000","last_fullsized":"2024-04-16T21:08:44.365785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:19.370822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495596+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:07:44.495596+0000","last_peered":"2024-04-16T21:07:44.495596+0000","last_clean":"2024-04-16T21:07:44.495596+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:07:44.495596+0000","last_undegraded":"2024-04-16T21:07:44.495596+0000","last_fullsized":"2024-04-16T21:07:44.495596+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:27.385349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365439+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:08:49.365439+0000","last_peered":"2024-04-16T21:08:49.365439+0000","last_clean":"2024-04-16T21:08:49.365439+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:08:49.365439+0000","last_undegraded":"2024-04-16T21:08:49.365439+0000","last_fullsized":"2024-04-16T21:08:49.365439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:12:02.296640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2006,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656784+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:07:54.656784+0000","last_peered":"2024-04-16T21:07:54.656784+0000","last_clean":"2024-04-16T21:07:54.656784+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:07:54.656784+0000","last_undegraded":"2024-04-16T21:07:54.656784+0000","last_fullsized":"2024-04-16T21:07:54.656784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:10:23.783937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655892+0000","last_change":"2024-04-16T19:25:41.506037+0000","last_active":"2024-04-16T21:07:49.655892+0000","last_peered":"2024-04-16T21:07:49.655892+0000","last_clean":"2024-04-16T21:07:49.655892+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:07:49.655892+0000","last_undegraded":"2024-04-16T21:07:49.655892+0000","last_fullsized":"2024-04-16T21:07:49.655892+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'33","reported_seq":183,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652615+0000","last_change":"2024-04-16T19:25:15.482185+0000","last_active":"2024-04-16T21:07:34.652615+0000","last_peered":"2024-04-16T21:07:34.652615+0000","last_clean":"2024-04-16T21:07:34.652615+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:07:34.652615+0000","last_undegraded":"2024-04-16T21:07:34.652615+0000","last_fullsized":"2024-04-16T21:07:34.652615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:07.940015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648199+0000","last_change":"2024-04-16T19:25:13.475047+0000","last_active":"2024-04-16T21:07:04.648199+0000","last_peered":"2024-04-16T21:07:04.648199+0000","last_clean":"2024-04-16T21:07:04.648199+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:07:04.648199+0000","last_undegraded":"2024-04-16T21:07:04.648199+0000","last_fullsized":"2024-04-16T21:07:04.648199+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:29:34.601553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1740","reported_seq":4600,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648166+0000","last_change":"2024-04-16T19:25:11.468809+0000","last_active":"2024-04-16T21:07:04.648166+0000","last_peered":"2024-04-16T21:07:04.648166+0000","last_clean":"2024-04-16T21:07:04.648166+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:07:04.648166+0000","last_undegraded":"2024-04-16T21:07:04.648166+0000","last_fullsized":"2024-04-16T21:07:04.648166+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1740,"log_dups_size":0,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:53.762063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'19","reported_seq":270,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651251+0000","last_change":"2024-04-16T19:25:09.468894+0000","last_active":"2024-04-16T21:07:29.651251+0000","last_peered":"2024-04-16T21:07:29.651251+0000","last_clean":"2024-04-16T21:07:29.651251+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:07:29.651251+0000","last_undegraded":"2024-04-16T21:07:29.651251+0000","last_fullsized":"2024-04-16T21:07:29.651251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:05.430878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664164+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:08:39.664164+0000","last_peered":"2024-04-16T21:08:39.664164+0000","last_clean":"2024-04-16T21:08:39.664164+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:08:39.664164+0000","last_undegraded":"2024-04-16T21:08:39.664164+0000","last_fullsized":"2024-04-16T21:08:39.664164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660546+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:08:19.660546+0000","last_peered":"2024-04-16T21:08:19.660546+0000","last_clean":"2024-04-16T21:08:19.660546+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:08:19.660546+0000","last_undegraded":"2024-04-16T21:08:19.660546+0000","last_fullsized":"2024-04-16T21:08:19.660546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365365+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:08:49.365365+0000","last_peered":"2024-04-16T21:08:49.365365+0000","last_clean":"2024-04-16T21:08:49.365365+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:08:49.365365+0000","last_undegraded":"2024-04-16T21:08:49.365365+0000","last_fullsized":"2024-04-16T21:08:49.365365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:48.947818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":394,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368921+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:09:09.368921+0000","last_peered":"2024-04-16T21:09:09.368921+0000","last_clean":"2024-04-16T21:09:09.368921+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:09:09.368921+0000","last_undegraded":"2024-04-16T21:09:09.368921+0000","last_fullsized":"2024-04-16T21:09:09.368921+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:36.588167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501183+0000","last_change":"2024-04-16T19:24:49.434497+0000","last_active":"2024-04-16T21:08:19.501183+0000","last_peered":"2024-04-16T21:08:19.501183+0000","last_clean":"2024-04-16T21:08:19.501183+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:08:19.501183+0000","last_undegraded":"2024-04-16T21:08:19.501183+0000","last_fullsized":"2024-04-16T21:08:19.501183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:08.864273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368025+0000","last_change":"2024-04-16T19:24:51.437122+0000","last_active":"2024-04-16T21:09:04.368025+0000","last_peered":"2024-04-16T21:09:04.368025+0000","last_clean":"2024-04-16T21:09:04.368025+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:09:04.368025+0000","last_undegraded":"2024-04-16T21:09:04.368025+0000","last_fullsized":"2024-04-16T21:09:04.368025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:10.861306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506044+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:08:49.506044+0000","last_peered":"2024-04-16T21:08:49.506044+0000","last_clean":"2024-04-16T21:08:49.506044+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:08:49.506044+0000","last_undegraded":"2024-04-16T21:08:49.506044+0000","last_fullsized":"2024-04-16T21:08:49.506044+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:48:31.872270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1928,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655341+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:07:54.655341+0000","last_peered":"2024-04-16T21:07:54.655341+0000","last_clean":"2024-04-16T21:07:54.655341+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:07:54.655341+0000","last_undegraded":"2024-04-16T21:07:54.655341+0000","last_fullsized":"2024-04-16T21:07:54.655341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657746+0000","last_change":"2024-04-16T19:25:41.502674+0000","last_active":"2024-04-16T21:07:59.657746+0000","last_peered":"2024-04-16T21:07:59.657746+0000","last_clean":"2024-04-16T21:07:59.657746+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:07:59.657746+0000","last_undegraded":"2024-04-16T21:07:59.657746+0000","last_fullsized":"2024-04-16T21:07:59.657746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'13","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651772+0000","last_change":"2024-04-16T19:25:15.485057+0000","last_active":"2024-04-16T21:07:24.651772+0000","last_peered":"2024-04-16T21:07:24.651772+0000","last_clean":"2024-04-16T21:07:24.651772+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:07:24.651772+0000","last_undegraded":"2024-04-16T21:07:24.651772+0000","last_fullsized":"2024-04-16T21:07:24.651772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:45.514212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647443+0000","last_change":"2024-04-16T19:25:13.472089+0000","last_active":"2024-04-16T21:07:04.647443+0000","last_peered":"2024-04-16T21:07:04.647443+0000","last_clean":"2024-04-16T21:07:04.647443+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:07:04.647443+0000","last_undegraded":"2024-04-16T21:07:04.647443+0000","last_fullsized":"2024-04-16T21:07:04.647443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'304","reported_seq":628,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648664+0000","last_change":"2024-04-16T19:25:11.462997+0000","last_active":"2024-04-16T21:07:04.648664+0000","last_peered":"2024-04-16T21:07:04.648664+0000","last_clean":"2024-04-16T21:07:04.648664+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:07:04.648664+0000","last_undegraded":"2024-04-16T21:07:04.648664+0000","last_fullsized":"2024-04-16T21:07:04.648664+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:14.402830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'8","reported_seq":279,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490828+0000","last_change":"2024-04-16T19:25:09.467337+0000","last_active":"2024-04-16T21:07:14.490828+0000","last_peered":"2024-04-16T21:07:14.490828+0000","last_clean":"2024-04-16T21:07:14.490828+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:07:14.490828+0000","last_undegraded":"2024-04-16T21:07:14.490828+0000","last_fullsized":"2024-04-16T21:07:14.490828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663257+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:08:39.663257+0000","last_peered":"2024-04-16T21:08:39.663257+0000","last_clean":"2024-04-16T21:08:39.663257+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:08:39.663257+0000","last_undegraded":"2024-04-16T21:08:39.663257+0000","last_fullsized":"2024-04-16T21:08:39.663257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658066+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:08:04.658066+0000","last_peered":"2024-04-16T21:08:04.658066+0000","last_clean":"2024-04-16T21:08:04.658066+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:08:04.658066+0000","last_undegraded":"2024-04-16T21:08:04.658066+0000","last_fullsized":"2024-04-16T21:08:04.658066+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:20.908683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":959,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.505980+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:08:49.505980+0000","last_peered":"2024-04-16T21:08:49.505980+0000","last_clean":"2024-04-16T21:08:49.505980+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:08:49.505980+0000","last_undegraded":"2024-04-16T21:08:49.505980+0000","last_fullsized":"2024-04-16T21:08:49.505980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:35.897776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501172+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:08:19.501172+0000","last_peered":"2024-04-16T21:08:19.501172+0000","last_clean":"2024-04-16T21:08:19.501172+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:08:19.501172+0000","last_undegraded":"2024-04-16T21:08:19.501172+0000","last_fullsized":"2024-04-16T21:08:19.501172+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:40.145350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508432+0000","last_change":"2024-04-16T19:24:49.429526+0000","last_active":"2024-04-16T21:09:04.508432+0000","last_peered":"2024-04-16T21:09:04.508432+0000","last_clean":"2024-04-16T21:09:04.508432+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:09:04.508432+0000","last_undegraded":"2024-04-16T21:09:04.508432+0000","last_fullsized":"2024-04-16T21:09:04.508432+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:36.881378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649882+0000","last_change":"2024-04-16T19:24:51.437305+0000","last_active":"2024-04-16T21:07:14.649882+0000","last_peered":"2024-04-16T21:07:14.649882+0000","last_clean":"2024-04-16T21:07:14.649882+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:07:14.649882+0000","last_undegraded":"2024-04-16T21:07:14.649882+0000","last_fullsized":"2024-04-16T21:07:14.649882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:24.125422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366382+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:08:49.366382+0000","last_peered":"2024-04-16T21:08:49.366382+0000","last_clean":"2024-04-16T21:08:49.366382+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:08:49.366382+0000","last_undegraded":"2024-04-16T21:08:49.366382+0000","last_fullsized":"2024-04-16T21:08:49.366382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:22:33.730359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'93","reported_seq":1378,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498742+0000","last_change":"2024-04-16T19:26:41.564908+0000","last_active":"2024-04-16T21:08:04.498742+0000","last_peered":"2024-04-16T21:08:04.498742+0000","last_clean":"2024-04-16T21:08:04.498742+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:08:04.498742+0000","last_undegraded":"2024-04-16T21:08:04.498742+0000","last_fullsized":"2024-04-16T21:08:04.498742+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:47:23.710928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654443+0000","last_change":"2024-04-16T19:25:41.508929+0000","last_active":"2024-04-16T21:07:49.654443+0000","last_peered":"2024-04-16T21:07:49.654443+0000","last_clean":"2024-04-16T21:07:49.654443+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:07:49.654443+0000","last_undegraded":"2024-04-16T21:07:49.654443+0000","last_fullsized":"2024-04-16T21:07:49.654443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'19","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651849+0000","last_change":"2024-04-16T19:25:15.482026+0000","last_active":"2024-04-16T21:07:24.651849+0000","last_peered":"2024-04-16T21:07:24.651849+0000","last_clean":"2024-04-16T21:07:24.651849+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:07:24.651849+0000","last_undegraded":"2024-04-16T21:07:24.651849+0000","last_fullsized":"2024-04-16T21:07:24.651849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652807+0000","last_change":"2024-04-16T19:25:13.475419+0000","last_active":"2024-04-16T21:07:34.652807+0000","last_peered":"2024-04-16T21:07:34.652807+0000","last_clean":"2024-04-16T21:07:34.652807+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:07:34.652807+0000","last_undegraded":"2024-04-16T21:07:34.652807+0000","last_fullsized":"2024-04-16T21:07:34.652807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2044","reported_seq":5813,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649720+0000","last_change":"2024-04-16T19:25:11.468843+0000","last_active":"2024-04-16T21:07:19.649720+0000","last_peered":"2024-04-16T21:07:19.649720+0000","last_clean":"2024-04-16T21:07:19.649720+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:07:19.649720+0000","last_undegraded":"2024-04-16T21:07:19.649720+0000","last_fullsized":"2024-04-16T21:07:19.649720+0000","mapping_epoch":33,"log_start":"47'300","ondisk_log_start":"47'300","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1744,"log_dups_size":300,"ondisk_log_size":1744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:03.761996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'14","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369114+0000","last_change":"2024-04-16T19:25:09.467702+0000","last_active":"2024-04-16T21:09:09.369114+0000","last_peered":"2024-04-16T21:09:09.369114+0000","last_clean":"2024-04-16T21:09:09.369114+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:09:09.369114+0000","last_undegraded":"2024-04-16T21:09:09.369114+0000","last_fullsized":"2024-04-16T21:09:09.369114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:31.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367115+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:08:59.367115+0000","last_peered":"2024-04-16T21:08:59.367115+0000","last_clean":"2024-04-16T21:08:59.367115+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:08:59.367115+0000","last_undegraded":"2024-04-16T21:08:59.367115+0000","last_fullsized":"2024-04-16T21:08:59.367115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:40.677584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659023+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:08:04.659023+0000","last_peered":"2024-04-16T21:08:04.659023+0000","last_clean":"2024-04-16T21:08:04.659023+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:08:04.659023+0000","last_undegraded":"2024-04-16T21:08:04.659023+0000","last_fullsized":"2024-04-16T21:08:04.659023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:49.358784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500796+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:08:19.500796+0000","last_peered":"2024-04-16T21:08:19.500796+0000","last_clean":"2024-04-16T21:08:19.500796+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:08:19.500796+0000","last_undegraded":"2024-04-16T21:08:19.500796+0000","last_fullsized":"2024-04-16T21:08:19.500796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:59.160445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":328,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366327+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:08:49.366327+0000","last_peered":"2024-04-16T21:08:49.366327+0000","last_clean":"2024-04-16T21:08:49.366327+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:08:49.366327+0000","last_undegraded":"2024-04-16T21:08:49.366327+0000","last_fullsized":"2024-04-16T21:08:49.366327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1313,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370434+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:08:49.370434+0000","last_peered":"2024-04-16T21:08:49.370434+0000","last_clean":"2024-04-16T21:08:49.370434+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:08:49.370434+0000","last_undegraded":"2024-04-16T21:08:49.370434+0000","last_fullsized":"2024-04-16T21:08:49.370434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:30.260652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371475+0000","last_change":"2024-04-16T19:24:51.437991+0000","last_active":"2024-04-16T21:08:54.371475+0000","last_peered":"2024-04-16T21:08:54.371475+0000","last_clean":"2024-04-16T21:08:54.371475+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:08:54.371475+0000","last_undegraded":"2024-04-16T21:08:54.371475+0000","last_fullsized":"2024-04-16T21:08:54.371475+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:43:50.519739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496442+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:07:49.496442+0000","last_peered":"2024-04-16T21:07:49.496442+0000","last_clean":"2024-04-16T21:07:49.496442+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:07:49.496442+0000","last_undegraded":"2024-04-16T21:07:49.496442+0000","last_fullsized":"2024-04-16T21:07:49.496442+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:16:07.995736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'136","reported_seq":1889,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497219+0000","last_change":"2024-04-16T19:25:49.523163+0000","last_active":"2024-04-16T21:07:54.497219+0000","last_peered":"2024-04-16T21:07:54.497219+0000","last_clean":"2024-04-16T21:07:54.497219+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:07:54.497219+0000","last_undegraded":"2024-04-16T21:07:54.497219+0000","last_fullsized":"2024-04-16T21:07:54.497219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:08.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655310+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:07:49.655310+0000","last_peered":"2024-04-16T21:07:49.655310+0000","last_clean":"2024-04-16T21:07:49.655310+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:07:49.655310+0000","last_undegraded":"2024-04-16T21:07:49.655310+0000","last_fullsized":"2024-04-16T21:07:49.655310+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:36.879550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'11","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650465+0000","last_change":"2024-04-16T19:25:15.480717+0000","last_active":"2024-04-16T21:07:24.650465+0000","last_peered":"2024-04-16T21:07:24.650465+0000","last_clean":"2024-04-16T21:07:24.650465+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:07:24.650465+0000","last_undegraded":"2024-04-16T21:07:24.650465+0000","last_fullsized":"2024-04-16T21:07:24.650465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:39.031938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650534+0000","last_change":"2024-04-16T19:25:13.476070+0000","last_active":"2024-04-16T21:07:19.650534+0000","last_peered":"2024-04-16T21:07:19.650534+0000","last_clean":"2024-04-16T21:07:19.650534+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:07:19.650534+0000","last_undegraded":"2024-04-16T21:07:19.650534+0000","last_fullsized":"2024-04-16T21:07:19.650534+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:42.684942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'341","reported_seq":664,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494066+0000","last_change":"2024-04-16T19:25:11.467316+0000","last_active":"2024-04-16T21:07:34.494066+0000","last_peered":"2024-04-16T21:07:34.494066+0000","last_clean":"2024-04-16T21:07:34.494066+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:07:34.494066+0000","last_undegraded":"2024-04-16T21:07:34.494066+0000","last_fullsized":"2024-04-16T21:07:34.494066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'9","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369072+0000","last_change":"2024-04-16T19:25:09.470409+0000","last_active":"2024-04-16T21:09:09.369072+0000","last_peered":"2024-04-16T21:09:09.369072+0000","last_clean":"2024-04-16T21:09:09.369072+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:09:09.369072+0000","last_undegraded":"2024-04-16T21:09:09.369072+0000","last_fullsized":"2024-04-16T21:09:09.369072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657857+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:08:09.657857+0000","last_peered":"2024-04-16T21:08:09.657857+0000","last_clean":"2024-04-16T21:08:09.657857+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:08:09.657857+0000","last_undegraded":"2024-04-16T21:08:09.657857+0000","last_fullsized":"2024-04-16T21:08:09.657857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658003+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:08:04.658003+0000","last_peered":"2024-04-16T21:08:04.658003+0000","last_clean":"2024-04-16T21:08:04.658003+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:08:04.658003+0000","last_undegraded":"2024-04-16T21:08:04.658003+0000","last_fullsized":"2024-04-16T21:08:04.658003+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:29.699119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":457,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508414+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:09:04.508414+0000","last_peered":"2024-04-16T21:09:04.508414+0000","last_clean":"2024-04-16T21:09:04.508414+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:09:04.508414+0000","last_undegraded":"2024-04-16T21:09:04.508414+0000","last_fullsized":"2024-04-16T21:09:04.508414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:12.039717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":246,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370400+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:08:49.370400+0000","last_peered":"2024-04-16T21:08:49.370400+0000","last_clean":"2024-04-16T21:08:49.370400+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:08:49.370400+0000","last_undegraded":"2024-04-16T21:08:49.370400+0000","last_fullsized":"2024-04-16T21:08:49.370400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:16.109122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506003+0000","last_change":"2024-04-16T19:24:49.432294+0000","last_active":"2024-04-16T21:08:49.506003+0000","last_peered":"2024-04-16T21:08:49.506003+0000","last_clean":"2024-04-16T21:08:49.506003+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:08:49.506003+0000","last_undegraded":"2024-04-16T21:08:49.506003+0000","last_fullsized":"2024-04-16T21:08:49.506003+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:45.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365802+0000","last_change":"2024-04-16T19:24:51.442478+0000","last_active":"2024-04-16T21:08:54.365802+0000","last_peered":"2024-04-16T21:08:54.365802+0000","last_clean":"2024-04-16T21:08:54.365802+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:08:54.365802+0000","last_undegraded":"2024-04-16T21:08:54.365802+0000","last_fullsized":"2024-04-16T21:08:54.365802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495710+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:07:44.495710+0000","last_peered":"2024-04-16T21:07:44.495710+0000","last_clean":"2024-04-16T21:07:44.495710+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:07:44.495710+0000","last_undegraded":"2024-04-16T21:07:44.495710+0000","last_fullsized":"2024-04-16T21:07:44.495710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:02:07.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2345,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652944+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:07:39.652944+0000","last_peered":"2024-04-16T21:07:39.652944+0000","last_clean":"2024-04-16T21:07:39.652944+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:07:39.652944+0000","last_undegraded":"2024-04-16T21:07:39.652944+0000","last_fullsized":"2024-04-16T21:07:39.652944+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658027+0000","last_change":"2024-04-16T19:25:41.509053+0000","last_active":"2024-04-16T21:08:04.658027+0000","last_peered":"2024-04-16T21:08:04.658027+0000","last_clean":"2024-04-16T21:08:04.658027+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:08:04.658027+0000","last_undegraded":"2024-04-16T21:08:04.658027+0000","last_fullsized":"2024-04-16T21:08:04.658027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:22.621386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"47'8","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651351+0000","last_change":"2024-04-16T19:25:15.486624+0000","last_active":"2024-04-16T21:07:24.651351+0000","last_peered":"2024-04-16T21:07:24.651351+0000","last_clean":"2024-04-16T21:07:24.651351+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:07:24.651351+0000","last_undegraded":"2024-04-16T21:07:24.651351+0000","last_fullsized":"2024-04-16T21:07:24.651351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:35.616571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'705","reported_seq":2064,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651010+0000","last_change":"2024-04-16T19:25:13.473118+0000","last_active":"2024-04-16T21:07:19.651010+0000","last_peered":"2024-04-16T21:07:19.651010+0000","last_clean":"2024-04-16T21:07:19.651010+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:07:19.651010+0000","last_undegraded":"2024-04-16T21:07:19.651010+0000","last_fullsized":"2024-04-16T21:07:19.651010+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'269","reported_seq":540,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491606+0000","last_change":"2024-04-16T19:25:11.465450+0000","last_active":"2024-04-16T21:07:19.491606+0000","last_peered":"2024-04-16T21:07:19.491606+0000","last_clean":"2024-04-16T21:07:19.491606+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:07:19.491606+0000","last_undegraded":"2024-04-16T21:07:19.491606+0000","last_fullsized":"2024-04-16T21:07:19.491606+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:53.479823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'5","reported_seq":214,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648747+0000","last_change":"2024-04-16T19:25:09.458268+0000","last_active":"2024-04-16T21:07:14.648747+0000","last_peered":"2024-04-16T21:07:14.648747+0000","last_clean":"2024-04-16T21:07:14.648747+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:07:14.648747+0000","last_undegraded":"2024-04-16T21:07:14.648747+0000","last_fullsized":"2024-04-16T21:07:14.648747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:05.252198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658783+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:08:09.658783+0000","last_peered":"2024-04-16T21:08:09.658783+0000","last_clean":"2024-04-16T21:08:09.658783+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:08:09.658783+0000","last_undegraded":"2024-04-16T21:08:09.658783+0000","last_fullsized":"2024-04-16T21:08:09.658783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:29.013265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498840+0000","last_change":"2024-04-16T19:25:41.503885+0000","last_active":"2024-04-16T21:08:04.498840+0000","last_peered":"2024-04-16T21:08:04.498840+0000","last_clean":"2024-04-16T21:08:04.498840+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:08:04.498840+0000","last_undegraded":"2024-04-16T21:08:04.498840+0000","last_fullsized":"2024-04-16T21:08:04.498840+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:54.516253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500846+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:08:19.500846+0000","last_peered":"2024-04-16T21:08:19.500846+0000","last_clean":"2024-04-16T21:08:19.500846+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:08:19.500846+0000","last_undegraded":"2024-04-16T21:08:19.500846+0000","last_fullsized":"2024-04-16T21:08:19.500846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:30.224132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660316+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:08:19.660316+0000","last_peered":"2024-04-16T21:08:19.660316+0000","last_clean":"2024-04-16T21:08:19.660316+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:08:19.660316+0000","last_undegraded":"2024-04-16T21:08:19.660316+0000","last_fullsized":"2024-04-16T21:08:19.660316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:43.606761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662124+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:08:34.662124+0000","last_peered":"2024-04-16T21:08:34.662124+0000","last_clean":"2024-04-16T21:08:34.662124+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:08:34.662124+0000","last_undegraded":"2024-04-16T21:08:34.662124+0000","last_fullsized":"2024-04-16T21:08:34.662124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368874+0000","last_change":"2024-04-16T19:24:51.434984+0000","last_active":"2024-04-16T21:09:14.368874+0000","last_peered":"2024-04-16T21:09:14.368874+0000","last_clean":"2024-04-16T21:09:14.368874+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:09:14.368874+0000","last_undegraded":"2024-04-16T21:09:14.368874+0000","last_fullsized":"2024-04-16T21:09:14.368874+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:26.835802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653730+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:07:44.653730+0000","last_peered":"2024-04-16T21:07:44.653730+0000","last_clean":"2024-04-16T21:07:44.653730+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:07:44.653730+0000","last_undegraded":"2024-04-16T21:07:44.653730+0000","last_fullsized":"2024-04-16T21:07:44.653730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:56.544746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1678,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653711+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:07:39.653711+0000","last_peered":"2024-04-16T21:07:39.653711+0000","last_clean":"2024-04-16T21:07:39.653711+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:07:39.653711+0000","last_undegraded":"2024-04-16T21:07:39.653711+0000","last_fullsized":"2024-04-16T21:07:39.653711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:01.314955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655387+0000","last_change":"2024-04-16T19:25:41.504293+0000","last_active":"2024-04-16T21:07:49.655387+0000","last_peered":"2024-04-16T21:07:49.655387+0000","last_clean":"2024-04-16T21:07:49.655387+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:07:49.655387+0000","last_undegraded":"2024-04-16T21:07:49.655387+0000","last_fullsized":"2024-04-16T21:07:49.655387+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'136","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654370+0000","last_change":"2024-04-16T19:25:15.483254+0000","last_active":"2024-04-16T21:07:39.654370+0000","last_peered":"2024-04-16T21:07:39.654370+0000","last_clean":"2024-04-16T21:07:39.654370+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:07:39.654370+0000","last_undegraded":"2024-04-16T21:07:39.654370+0000","last_fullsized":"2024-04-16T21:07:39.654370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491600+0000","last_change":"2024-04-16T19:25:13.471760+0000","last_active":"2024-04-16T21:07:19.491600+0000","last_peered":"2024-04-16T21:07:19.491600+0000","last_clean":"2024-04-16T21:07:19.491600+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:07:19.491600+0000","last_undegraded":"2024-04-16T21:07:19.491600+0000","last_fullsized":"2024-04-16T21:07:19.491600+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'347","reported_seq":724,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491639+0000","last_change":"2024-04-16T19:25:11.467463+0000","last_active":"2024-04-16T21:07:19.491639+0000","last_peered":"2024-04-16T21:07:19.491639+0000","last_clean":"2024-04-16T21:07:19.491639+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:07:19.491639+0000","last_undegraded":"2024-04-16T21:07:19.491639+0000","last_fullsized":"2024-04-16T21:07:19.491639+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492364+0000","last_change":"2024-04-16T19:25:09.461452+0000","last_active":"2024-04-16T21:07:24.492364+0000","last_peered":"2024-04-16T21:07:24.492364+0000","last_clean":"2024-04-16T21:07:24.492364+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:07:24.492364+0000","last_undegraded":"2024-04-16T21:07:24.492364+0000","last_fullsized":"2024-04-16T21:07:24.492364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:44:08.538156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659737+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:08:09.659737+0000","last_peered":"2024-04-16T21:08:09.659737+0000","last_clean":"2024-04-16T21:08:09.659737+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:08:09.659737+0000","last_undegraded":"2024-04-16T21:08:09.659737+0000","last_fullsized":"2024-04-16T21:08:09.659737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496482+0000","last_change":"2024-04-16T19:25:41.506323+0000","last_active":"2024-04-16T21:07:49.496482+0000","last_peered":"2024-04-16T21:07:49.496482+0000","last_clean":"2024-04-16T21:07:49.496482+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:07:49.496482+0000","last_undegraded":"2024-04-16T21:07:49.496482+0000","last_fullsized":"2024-04-16T21:07:49.496482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:05.815658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501233+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:08:19.501233+0000","last_peered":"2024-04-16T21:08:19.501233+0000","last_clean":"2024-04-16T21:08:19.501233+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:08:19.501233+0000","last_undegraded":"2024-04-16T21:08:19.501233+0000","last_fullsized":"2024-04-16T21:08:19.501233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:51.524577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'140","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503630+0000","last_change":"2024-04-16T19:24:47.426406+0000","last_active":"2024-04-16T21:08:34.503630+0000","last_peered":"2024-04-16T21:08:34.503630+0000","last_clean":"2024-04-16T21:08:34.503630+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:08:34.503630+0000","last_undegraded":"2024-04-16T21:08:34.503630+0000","last_fullsized":"2024-04-16T21:08:34.503630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:12.378311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500867+0000","last_change":"2024-04-16T19:24:49.430773+0000","last_active":"2024-04-16T21:08:19.500867+0000","last_peered":"2024-04-16T21:08:19.500867+0000","last_clean":"2024-04-16T21:08:19.500867+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:08:19.500867+0000","last_undegraded":"2024-04-16T21:08:19.500867+0000","last_fullsized":"2024-04-16T21:08:19.500867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:52.965094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367299+0000","last_change":"2024-04-16T19:24:51.436754+0000","last_active":"2024-04-16T21:09:04.367299+0000","last_peered":"2024-04-16T21:09:04.367299+0000","last_clean":"2024-04-16T21:09:04.367299+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:09:04.367299+0000","last_undegraded":"2024-04-16T21:09:04.367299+0000","last_fullsized":"2024-04-16T21:09:04.367299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:01.678207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368731+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:09:04.368731+0000","last_peered":"2024-04-16T21:09:04.368731+0000","last_clean":"2024-04-16T21:09:04.368731+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:09:04.368731+0000","last_undegraded":"2024-04-16T21:09:04.368731+0000","last_fullsized":"2024-04-16T21:09:04.368731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:01:32.497457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1870,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656880+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:07:54.656880+0000","last_peered":"2024-04-16T21:07:54.656880+0000","last_clean":"2024-04-16T21:07:54.656880+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:07:54.656880+0000","last_undegraded":"2024-04-16T21:07:54.656880+0000","last_fullsized":"2024-04-16T21:07:54.656880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653650+0000","last_change":"2024-04-16T19:25:41.507626+0000","last_active":"2024-04-16T21:07:34.653650+0000","last_peered":"2024-04-16T21:07:34.653650+0000","last_clean":"2024-04-16T21:07:34.653650+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:07:34.653650+0000","last_undegraded":"2024-04-16T21:07:34.653650+0000","last_fullsized":"2024-04-16T21:07:34.653650+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'7","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489918+0000","last_change":"2024-04-16T19:25:15.477722+0000","last_active":"2024-04-16T21:07:09.489918+0000","last_peered":"2024-04-16T21:07:09.489918+0000","last_clean":"2024-04-16T21:07:09.489918+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:07:09.489918+0000","last_undegraded":"2024-04-16T21:07:09.489918+0000","last_fullsized":"2024-04-16T21:07:09.489918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:11.553471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650458+0000","last_change":"2024-04-16T19:25:13.476043+0000","last_active":"2024-04-16T21:07:19.650458+0000","last_peered":"2024-04-16T21:07:19.650458+0000","last_clean":"2024-04-16T21:07:19.650458+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:07:19.650458+0000","last_undegraded":"2024-04-16T21:07:19.650458+0000","last_fullsized":"2024-04-16T21:07:19.650458+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'453","reported_seq":868,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651934+0000","last_change":"2024-04-16T19:25:11.462559+0000","last_active":"2024-04-16T21:07:29.651934+0000","last_peered":"2024-04-16T21:07:29.651934+0000","last_clean":"2024-04-16T21:07:29.651934+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:07:29.651934+0000","last_undegraded":"2024-04-16T21:07:29.651934+0000","last_fullsized":"2024-04-16T21:07:29.651934+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:33.939814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'8","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655202+0000","last_change":"2024-04-16T19:25:41.507771+0000","last_active":"2024-04-16T21:07:49.655202+0000","last_peered":"2024-04-16T21:07:49.655202+0000","last_clean":"2024-04-16T21:07:49.655202+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:07:49.655202+0000","last_undegraded":"2024-04-16T21:07:49.655202+0000","last_fullsized":"2024-04-16T21:07:49.655202+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:53.575807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658859+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:08:09.658859+0000","last_peered":"2024-04-16T21:08:09.658859+0000","last_clean":"2024-04-16T21:08:09.658859+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:08:09.658859+0000","last_undegraded":"2024-04-16T21:08:09.658859+0000","last_fullsized":"2024-04-16T21:08:09.658859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662590+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:08:34.662590+0000","last_peered":"2024-04-16T21:08:34.662590+0000","last_clean":"2024-04-16T21:08:34.662590+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:08:34.662590+0000","last_undegraded":"2024-04-16T21:08:34.662590+0000","last_fullsized":"2024-04-16T21:08:34.662590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1361,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663160+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:08:34.663160+0000","last_peered":"2024-04-16T21:08:34.663160+0000","last_clean":"2024-04-16T21:08:34.663160+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:08:34.663160+0000","last_undegraded":"2024-04-16T21:08:34.663160+0000","last_fullsized":"2024-04-16T21:08:34.663160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:46.278655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":276,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501256+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:08:19.501256+0000","last_peered":"2024-04-16T21:08:19.501256+0000","last_clean":"2024-04-16T21:08:19.501256+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:08:19.501256+0000","last_undegraded":"2024-04-16T21:08:19.501256+0000","last_fullsized":"2024-04-16T21:08:19.501256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:23.115125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660267+0000","last_change":"2024-04-16T19:24:49.428502+0000","last_active":"2024-04-16T21:08:19.660267+0000","last_peered":"2024-04-16T21:08:19.660267+0000","last_clean":"2024-04-16T21:08:19.660267+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:08:19.660267+0000","last_undegraded":"2024-04-16T21:08:19.660267+0000","last_fullsized":"2024-04-16T21:08:19.660267+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:25.122206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367884+0000","last_change":"2024-04-16T19:24:51.435114+0000","last_active":"2024-04-16T21:09:04.367884+0000","last_peered":"2024-04-16T21:09:04.367884+0000","last_clean":"2024-04-16T21:09:04.367884+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:09:04.367884+0000","last_undegraded":"2024-04-16T21:09:04.367884+0000","last_fullsized":"2024-04-16T21:09:04.367884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:56.562306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364938+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:08:49.364938+0000","last_peered":"2024-04-16T21:08:49.364938+0000","last_clean":"2024-04-16T21:08:49.364938+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:08:49.364938+0000","last_undegraded":"2024-04-16T21:08:49.364938+0000","last_fullsized":"2024-04-16T21:08:49.364938+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:08:56.543210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657898+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:08:09.657898+0000","last_peered":"2024-04-16T21:08:09.657898+0000","last_clean":"2024-04-16T21:08:09.657898+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:08:09.657898+0000","last_undegraded":"2024-04-16T21:08:09.657898+0000","last_fullsized":"2024-04-16T21:08:09.657898+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:44:18.098055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653619+0000","last_change":"2024-04-16T19:25:41.506250+0000","last_active":"2024-04-16T21:07:34.653619+0000","last_peered":"2024-04-16T21:07:34.653619+0000","last_clean":"2024-04-16T21:07:34.653619+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:07:34.653619+0000","last_undegraded":"2024-04-16T21:07:34.653619+0000","last_fullsized":"2024-04-16T21:07:34.653619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489957+0000","last_change":"2024-04-16T19:25:15.479905+0000","last_active":"2024-04-16T21:07:09.489957+0000","last_peered":"2024-04-16T21:07:09.489957+0000","last_clean":"2024-04-16T21:07:09.489957+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:07:09.489957+0000","last_undegraded":"2024-04-16T21:07:09.489957+0000","last_fullsized":"2024-04-16T21:07:09.489957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:03.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'705","reported_seq":2463,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493089+0000","last_change":"2024-04-16T19:25:13.470216+0000","last_active":"2024-04-16T21:07:29.493089+0000","last_peered":"2024-04-16T21:07:29.493089+0000","last_clean":"2024-04-16T21:07:29.493089+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:07:29.493089+0000","last_undegraded":"2024-04-16T21:07:29.493089+0000","last_fullsized":"2024-04-16T21:07:29.493089+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:40.891478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'311","reported_seq":572,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650437+0000","last_change":"2024-04-16T19:25:11.464746+0000","last_active":"2024-04-16T21:07:19.650437+0000","last_peered":"2024-04-16T21:07:19.650437+0000","last_clean":"2024-04-16T21:07:19.650437+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:07:19.650437+0000","last_undegraded":"2024-04-16T21:07:19.650437+0000","last_fullsized":"2024-04-16T21:07:19.650437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'6","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490772+0000","last_change":"2024-04-16T19:25:09.466128+0000","last_active":"2024-04-16T21:07:14.490772+0000","last_peered":"2024-04-16T21:07:14.490772+0000","last_clean":"2024-04-16T21:07:14.490772+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:07:14.490772+0000","last_undegraded":"2024-04-16T21:07:14.490772+0000","last_fullsized":"2024-04-16T21:07:14.490772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.502055+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:08:24.502055+0000","last_peered":"2024-04-16T21:08:24.502055+0000","last_clean":"2024-04-16T21:08:24.502055+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:08:24.502055+0000","last_undegraded":"2024-04-16T21:08:24.502055+0000","last_fullsized":"2024-04-16T21:08:24.502055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:20.204370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503320+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:08:34.503320+0000","last_peered":"2024-04-16T21:08:34.503320+0000","last_clean":"2024-04-16T21:08:34.503320+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:08:34.503320+0000","last_undegraded":"2024-04-16T21:08:34.503320+0000","last_fullsized":"2024-04-16T21:08:34.503320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:08:26.977348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661224+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:08:19.661224+0000","last_peered":"2024-04-16T21:08:19.661224+0000","last_clean":"2024-04-16T21:08:19.661224+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:08:19.661224+0000","last_undegraded":"2024-04-16T21:08:19.661224+0000","last_fullsized":"2024-04-16T21:08:19.661224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:45.288392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":359,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660463+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:08:19.660463+0000","last_peered":"2024-04-16T21:08:19.660463+0000","last_clean":"2024-04-16T21:08:19.660463+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:08:19.660463+0000","last_undegraded":"2024-04-16T21:08:19.660463+0000","last_fullsized":"2024-04-16T21:08:19.660463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:48.207837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661146+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:08:19.661146+0000","last_peered":"2024-04-16T21:08:19.661146+0000","last_clean":"2024-04-16T21:08:19.661146+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:08:19.661146+0000","last_undegraded":"2024-04-16T21:08:19.661146+0000","last_fullsized":"2024-04-16T21:08:19.661146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367334+0000","last_change":"2024-04-16T19:24:51.434704+0000","last_active":"2024-04-16T21:09:04.367334+0000","last_peered":"2024-04-16T21:09:04.367334+0000","last_clean":"2024-04-16T21:09:04.367334+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:09:04.367334+0000","last_undegraded":"2024-04-16T21:09:04.367334+0000","last_fullsized":"2024-04-16T21:09:04.367334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503573+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:08:34.503573+0000","last_peered":"2024-04-16T21:08:34.503573+0000","last_clean":"2024-04-16T21:08:34.503573+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:08:34.503573+0000","last_undegraded":"2024-04-16T21:08:34.503573+0000","last_fullsized":"2024-04-16T21:08:34.503573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:11.491071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'162","reported_seq":2095,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650676+0000","last_change":"2024-04-16T19:25:49.518828+0000","last_active":"2024-04-16T21:07:19.650676+0000","last_peered":"2024-04-16T21:07:19.650676+0000","last_clean":"2024-04-16T21:07:19.650676+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:07:19.650676+0000","last_undegraded":"2024-04-16T21:07:19.650676+0000","last_fullsized":"2024-04-16T21:07:19.650676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:22.736467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488302+0000","last_change":"2024-04-16T19:25:41.510329+0000","last_active":"2024-04-16T21:06:59.488302+0000","last_peered":"2024-04-16T21:06:59.488302+0000","last_clean":"2024-04-16T21:06:59.488302+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:06:59.488302+0000","last_undegraded":"2024-04-16T21:06:59.488302+0000","last_fullsized":"2024-04-16T21:06:59.488302+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:14.723662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'12","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365503+0000","last_change":"2024-04-16T19:25:15.480613+0000","last_active":"2024-04-16T21:08:49.365503+0000","last_peered":"2024-04-16T21:08:49.365503+0000","last_clean":"2024-04-16T21:08:49.365503+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:08:49.365503+0000","last_undegraded":"2024-04-16T21:08:49.365503+0000","last_fullsized":"2024-04-16T21:08:49.365503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491622+0000","last_change":"2024-04-16T19:25:13.469639+0000","last_active":"2024-04-16T21:07:19.491622+0000","last_peered":"2024-04-16T21:07:19.491622+0000","last_clean":"2024-04-16T21:07:19.491622+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:07:19.491622+0000","last_undegraded":"2024-04-16T21:07:19.491622+0000","last_fullsized":"2024-04-16T21:07:19.491622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:02.547322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'256","reported_seq":566,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364495+0000","last_change":"2024-04-16T19:25:11.463503+0000","last_active":"2024-04-16T21:08:44.364495+0000","last_peered":"2024-04-16T21:08:44.364495+0000","last_clean":"2024-04-16T21:08:44.364495+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:08:44.364495+0000","last_undegraded":"2024-04-16T21:08:44.364495+0000","last_fullsized":"2024-04-16T21:08:44.364495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:07.258240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'6","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509214+0000","last_change":"2024-04-16T19:25:09.462147+0000","last_active":"2024-04-16T21:09:09.509214+0000","last_peered":"2024-04-16T21:09:09.509214+0000","last_clean":"2024-04-16T21:09:09.509214+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:09:09.509214+0000","last_undegraded":"2024-04-16T21:09:09.509214+0000","last_fullsized":"2024-04-16T21:09:09.509214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369516+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:09:09.369516+0000","last_peered":"2024-04-16T21:09:09.369516+0000","last_clean":"2024-04-16T21:09:09.369516+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:09:09.369516+0000","last_undegraded":"2024-04-16T21:09:09.369516+0000","last_fullsized":"2024-04-16T21:09:09.369516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:22.844339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658852+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:08:04.658852+0000","last_peered":"2024-04-16T21:08:04.658852+0000","last_clean":"2024-04-16T21:08:04.658852+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:08:04.658852+0000","last_undegraded":"2024-04-16T21:08:04.658852+0000","last_fullsized":"2024-04-16T21:08:04.658852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657180+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:07:59.657180+0000","last_peered":"2024-04-16T21:07:59.657180+0000","last_clean":"2024-04-16T21:07:59.657180+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:07:59.657180+0000","last_undegraded":"2024-04-16T21:07:59.657180+0000","last_fullsized":"2024-04-16T21:07:59.657180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.363997+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:08:44.363997+0000","last_peered":"2024-04-16T21:08:44.363997+0000","last_clean":"2024-04-16T21:08:44.363997+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:08:44.363997+0000","last_undegraded":"2024-04-16T21:08:44.363997+0000","last_fullsized":"2024-04-16T21:08:44.363997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367212+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:08:59.367212+0000","last_peered":"2024-04-16T21:08:59.367212+0000","last_clean":"2024-04-16T21:08:59.367212+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:08:59.367212+0000","last_undegraded":"2024-04-16T21:08:59.367212+0000","last_fullsized":"2024-04-16T21:08:59.367212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:49.663208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365555+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:08:49.365555+0000","last_peered":"2024-04-16T21:08:49.365555+0000","last_clean":"2024-04-16T21:08:49.365555+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:08:49.365555+0000","last_undegraded":"2024-04-16T21:08:49.365555+0000","last_fullsized":"2024-04-16T21:08:49.365555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370592+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:08:49.370592+0000","last_peered":"2024-04-16T21:08:49.370592+0000","last_clean":"2024-04-16T21:08:49.370592+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:08:49.370592+0000","last_undegraded":"2024-04-16T21:08:49.370592+0000","last_fullsized":"2024-04-16T21:08:49.370592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366370+0000","last_change":"2024-04-16T19:24:51.442576+0000","last_active":"2024-04-16T21:08:54.366370+0000","last_peered":"2024-04-16T21:08:54.366370+0000","last_clean":"2024-04-16T21:08:54.366370+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:08:54.366370+0000","last_undegraded":"2024-04-16T21:08:54.366370+0000","last_fullsized":"2024-04-16T21:08:54.366370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662725+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:08:34.662725+0000","last_peered":"2024-04-16T21:08:34.662725+0000","last_clean":"2024-04-16T21:08:34.662725+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:08:34.662725+0000","last_undegraded":"2024-04-16T21:08:34.662725+0000","last_fullsized":"2024-04-16T21:08:34.662725+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:34:27.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1950,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647397+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:07:04.647397+0000","last_peered":"2024-04-16T21:07:04.647397+0000","last_clean":"2024-04-16T21:07:04.647397+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:07:04.647397+0000","last_undegraded":"2024-04-16T21:07:04.647397+0000","last_fullsized":"2024-04-16T21:07:04.647397+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:42.651587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649772+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:07:14.649772+0000","last_peered":"2024-04-16T21:07:14.649772+0000","last_clean":"2024-04-16T21:07:14.649772+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:07:14.649772+0000","last_undegraded":"2024-04-16T21:07:14.649772+0000","last_fullsized":"2024-04-16T21:07:14.649772+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'13","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370574+0000","last_change":"2024-04-16T19:25:15.485212+0000","last_active":"2024-04-16T21:08:49.370574+0000","last_peered":"2024-04-16T21:08:49.370574+0000","last_clean":"2024-04-16T21:08:49.370574+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:08:49.370574+0000","last_undegraded":"2024-04-16T21:08:49.370574+0000","last_fullsized":"2024-04-16T21:08:49.370574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:41.072415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'705","reported_seq":2076,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489124+0000","last_change":"2024-04-16T19:25:13.467216+0000","last_active":"2024-04-16T21:07:04.489124+0000","last_peered":"2024-04-16T21:07:04.489124+0000","last_clean":"2024-04-16T21:07:04.489124+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:07:04.489124+0000","last_undegraded":"2024-04-16T21:07:04.489124+0000","last_fullsized":"2024-04-16T21:07:04.489124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'306","reported_seq":614,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647255+0000","last_change":"2024-04-16T19:25:11.467288+0000","last_active":"2024-04-16T21:07:04.647255+0000","last_peered":"2024-04-16T21:07:04.647255+0000","last_clean":"2024-04-16T21:07:04.647255+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:07:04.647255+0000","last_undegraded":"2024-04-16T21:07:04.647255+0000","last_fullsized":"2024-04-16T21:07:04.647255+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'12","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662672+0000","last_change":"2024-04-16T19:25:09.464018+0000","last_active":"2024-04-16T21:08:39.662672+0000","last_peered":"2024-04-16T21:08:39.662672+0000","last_clean":"2024-04-16T21:08:39.662672+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:08:39.662672+0000","last_undegraded":"2024-04-16T21:08:39.662672+0000","last_fullsized":"2024-04-16T21:08:39.662672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:52.913431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662788+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:08:39.662788+0000","last_peered":"2024-04-16T21:08:39.662788+0000","last_clean":"2024-04-16T21:08:39.662788+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:08:39.662788+0000","last_undegraded":"2024-04-16T21:08:39.662788+0000","last_fullsized":"2024-04-16T21:08:39.662788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658875+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:08:04.658875+0000","last_peered":"2024-04-16T21:08:04.658875+0000","last_clean":"2024-04-16T21:08:04.658875+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:08:04.658875+0000","last_undegraded":"2024-04-16T21:08:04.658875+0000","last_fullsized":"2024-04-16T21:08:04.658875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:26.452728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369665+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:08:44.369665+0000","last_peered":"2024-04-16T21:08:44.369665+0000","last_clean":"2024-04-16T21:08:44.369665+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:08:44.369665+0000","last_undegraded":"2024-04-16T21:08:44.369665+0000","last_fullsized":"2024-04-16T21:08:44.369665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:23.643674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":591,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365570+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:08:49.365570+0000","last_peered":"2024-04-16T21:08:49.365570+0000","last_clean":"2024-04-16T21:08:49.365570+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:08:49.365570+0000","last_undegraded":"2024-04-16T21:08:49.365570+0000","last_fullsized":"2024-04-16T21:08:49.365570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:34.895652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366657+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:08:59.366657+0000","last_peered":"2024-04-16T21:08:59.366657+0000","last_clean":"2024-04-16T21:08:59.366657+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:08:59.366657+0000","last_undegraded":"2024-04-16T21:08:59.366657+0000","last_fullsized":"2024-04-16T21:08:59.366657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:28.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649142+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:07:09.649142+0000","last_peered":"2024-04-16T21:07:09.649142+0000","last_clean":"2024-04-16T21:07:09.649142+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:07:09.649142+0000","last_undegraded":"2024-04-16T21:07:09.649142+0000","last_fullsized":"2024-04-16T21:07:09.649142+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.152431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488247+0000","last_change":"2024-04-16T19:24:51.574513+0000","last_active":"2024-04-16T21:06:59.488247+0000","last_peered":"2024-04-16T21:06:59.488247+0000","last_clean":"2024-04-16T21:06:59.488247+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:06:59.488247+0000","last_undegraded":"2024-04-16T21:06:59.488247+0000","last_fullsized":"2024-04-16T21:06:59.488247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:32.268529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663021+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:08:34.663021+0000","last_peered":"2024-04-16T21:08:34.663021+0000","last_clean":"2024-04-16T21:08:34.663021+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:08:34.663021+0000","last_undegraded":"2024-04-16T21:08:34.663021+0000","last_fullsized":"2024-04-16T21:08:34.663021+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:37.701350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2368,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648112+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:07:04.648112+0000","last_peered":"2024-04-16T21:07:04.648112+0000","last_clean":"2024-04-16T21:07:04.648112+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:07:04.648112+0000","last_undegraded":"2024-04-16T21:07:04.648112+0000","last_fullsized":"2024-04-16T21:07:04.648112+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:11:19.505797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647141+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:06:59.647141+0000","last_peered":"2024-04-16T21:06:59.647141+0000","last_clean":"2024-04-16T21:06:59.647141+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:06:59.647141+0000","last_undegraded":"2024-04-16T21:06:59.647141+0000","last_fullsized":"2024-04-16T21:06:59.647141+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'16","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.646016+0000","last_change":"2024-04-16T19:25:15.484662+0000","last_active":"2024-04-16T21:06:49.646016+0000","last_peered":"2024-04-16T21:06:49.646016+0000","last_clean":"2024-04-16T21:06:49.646016+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:06:49.646016+0000","last_undegraded":"2024-04-16T21:06:49.646016+0000","last_fullsized":"2024-04-16T21:06:49.646016+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'20","reported_seq":971,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489057+0000","last_change":"2024-04-16T19:25:13.474689+0000","last_active":"2024-04-16T21:07:04.489057+0000","last_peered":"2024-04-16T21:07:04.489057+0000","last_clean":"2024-04-16T21:07:04.489057+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:07:04.489057+0000","last_undegraded":"2024-04-16T21:07:04.489057+0000","last_fullsized":"2024-04-16T21:07:04.489057+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:36.914573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'249","reported_seq":647,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648427+0000","last_change":"2024-04-16T19:25:11.460951+0000","last_active":"2024-04-16T21:07:04.648427+0000","last_peered":"2024-04-16T21:07:04.648427+0000","last_clean":"2024-04-16T21:07:04.648427+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:07:04.648427+0000","last_undegraded":"2024-04-16T21:07:04.648427+0000","last_fullsized":"2024-04-16T21:07:04.648427+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:42:59.386912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'7","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650231+0000","last_change":"2024-04-16T19:25:09.460368+0000","last_active":"2024-04-16T21:07:14.650231+0000","last_peered":"2024-04-16T21:07:14.650231+0000","last_clean":"2024-04-16T21:07:14.650231+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:07:14.650231+0000","last_undegraded":"2024-04-16T21:07:14.650231+0000","last_fullsized":"2024-04-16T21:07:14.650231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:19.738380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504660+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:08:39.504660+0000","last_peered":"2024-04-16T21:08:39.504660+0000","last_clean":"2024-04-16T21:08:39.504660+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:08:39.504660+0000","last_undegraded":"2024-04-16T21:08:39.504660+0000","last_fullsized":"2024-04-16T21:08:39.504660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.284891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497271+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:07:54.497271+0000","last_peered":"2024-04-16T21:07:54.497271+0000","last_clean":"2024-04-16T21:07:54.497271+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:07:54.497271+0000","last_undegraded":"2024-04-16T21:07:54.497271+0000","last_fullsized":"2024-04-16T21:07:54.497271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:27.416414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365627+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:08:54.365627+0000","last_peered":"2024-04-16T21:08:54.365627+0000","last_clean":"2024-04-16T21:08:54.365627+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:08:54.365627+0000","last_undegraded":"2024-04-16T21:08:54.365627+0000","last_fullsized":"2024-04-16T21:08:54.365627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:44.737149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506112+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:08:49.506112+0000","last_peered":"2024-04-16T21:08:49.506112+0000","last_clean":"2024-04-16T21:08:49.506112+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:08:49.506112+0000","last_undegraded":"2024-04-16T21:08:49.506112+0000","last_fullsized":"2024-04-16T21:08:49.506112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:32.935806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1512,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648886+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:07:09.648886+0000","last_peered":"2024-04-16T21:07:09.648886+0000","last_clean":"2024-04-16T21:07:09.648886+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:07:09.648886+0000","last_undegraded":"2024-04-16T21:07:09.648886+0000","last_fullsized":"2024-04-16T21:07:09.648886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372374+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:08:59.372374+0000","last_peered":"2024-04-16T21:08:59.372374+0000","last_clean":"2024-04-16T21:08:59.372374+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:08:59.372374+0000","last_undegraded":"2024-04-16T21:08:59.372374+0000","last_fullsized":"2024-04-16T21:08:59.372374+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660288+0000","last_change":"2024-04-16T19:24:51.438114+0000","last_active":"2024-04-16T21:08:24.660288+0000","last_peered":"2024-04-16T21:08:24.660288+0000","last_clean":"2024-04-16T21:08:24.660288+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:08:24.660288+0000","last_undegraded":"2024-04-16T21:08:24.660288+0000","last_fullsized":"2024-04-16T21:08:24.660288+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:59.734805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366544+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:08:49.366544+0000","last_peered":"2024-04-16T21:08:49.366544+0000","last_clean":"2024-04-16T21:08:49.366544+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:08:49.366544+0000","last_undegraded":"2024-04-16T21:08:49.366544+0000","last_fullsized":"2024-04-16T21:08:49.366544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:25:31.551960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1765,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648426+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:07:04.648426+0000","last_peered":"2024-04-16T21:07:04.648426+0000","last_clean":"2024-04-16T21:07:04.648426+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:07:04.648426+0000","last_undegraded":"2024-04-16T21:07:04.648426+0000","last_fullsized":"2024-04-16T21:07:04.648426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:10.736923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647564+0000","last_change":"2024-04-16T19:25:41.505274+0000","last_active":"2024-04-16T21:06:59.647564+0000","last_peered":"2024-04-16T21:06:59.647564+0000","last_clean":"2024-04-16T21:06:59.647564+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:06:59.647564+0000","last_undegraded":"2024-04-16T21:06:59.647564+0000","last_fullsized":"2024-04-16T21:06:59.647564+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:50:27.279669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'15","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370523+0000","last_change":"2024-04-16T19:25:15.487696+0000","last_active":"2024-04-16T21:08:49.370523+0000","last_peered":"2024-04-16T21:08:49.370523+0000","last_clean":"2024-04-16T21:08:49.370523+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:08:49.370523+0000","last_undegraded":"2024-04-16T21:08:49.370523+0000","last_fullsized":"2024-04-16T21:08:49.370523+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:34.149887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505292+0000","last_change":"2024-04-16T19:25:13.473305+0000","last_active":"2024-04-16T21:08:44.505292+0000","last_peered":"2024-04-16T21:08:44.505292+0000","last_clean":"2024-04-16T21:08:44.505292+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:08:44.505292+0000","last_undegraded":"2024-04-16T21:08:44.505292+0000","last_fullsized":"2024-04-16T21:08:44.505292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:38.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'690","reported_seq":1689,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:25:11.468818+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:21.906682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'16","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373858+0000","last_change":"2024-04-16T19:25:09.462529+0000","last_active":"2024-04-16T21:09:09.373858+0000","last_peered":"2024-04-16T21:09:09.373858+0000","last_clean":"2024-04-16T21:09:09.373858+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:09:09.373858+0000","last_undegraded":"2024-04-16T21:09:09.373858+0000","last_fullsized":"2024-04-16T21:09:09.373858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364926+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:08:49.364926+0000","last_peered":"2024-04-16T21:08:49.364926+0000","last_clean":"2024-04-16T21:08:49.364926+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:08:49.364926+0000","last_undegraded":"2024-04-16T21:08:49.364926+0000","last_fullsized":"2024-04-16T21:08:49.364926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:02.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657141+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:07:59.657141+0000","last_peered":"2024-04-16T21:07:59.657141+0000","last_clean":"2024-04-16T21:07:59.657141+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:07:59.657141+0000","last_undegraded":"2024-04-16T21:07:59.657141+0000","last_fullsized":"2024-04-16T21:07:59.657141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364646+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:08:44.364646+0000","last_peered":"2024-04-16T21:08:44.364646+0000","last_clean":"2024-04-16T21:08:44.364646+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:08:44.364646+0000","last_undegraded":"2024-04-16T21:08:44.364646+0000","last_fullsized":"2024-04-16T21:08:44.364646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:21.093051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648115+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:07:09.648115+0000","last_peered":"2024-04-16T21:07:09.648115+0000","last_clean":"2024-04-16T21:07:09.648115+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:07:09.648115+0000","last_undegraded":"2024-04-16T21:07:09.648115+0000","last_fullsized":"2024-04-16T21:07:09.648115+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:21.104930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364984+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:08:49.364984+0000","last_peered":"2024-04-16T21:08:49.364984+0000","last_clean":"2024-04-16T21:08:49.364984+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:08:49.364984+0000","last_undegraded":"2024-04-16T21:08:49.364984+0000","last_fullsized":"2024-04-16T21:08:49.364984+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:58.758206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366555+0000","last_change":"2024-04-16T19:24:49.432013+0000","last_active":"2024-04-16T21:08:49.366555+0000","last_peered":"2024-04-16T21:08:49.366555+0000","last_clean":"2024-04-16T21:08:49.366555+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:08:49.366555+0000","last_undegraded":"2024-04-16T21:08:49.366555+0000","last_fullsized":"2024-04-16T21:08:49.366555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366349+0000","last_change":"2024-04-16T19:24:51.436955+0000","last_active":"2024-04-16T21:08:54.366349+0000","last_peered":"2024-04-16T21:08:54.366349+0000","last_clean":"2024-04-16T21:08:54.366349+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:08:54.366349+0000","last_undegraded":"2024-04-16T21:08:54.366349+0000","last_fullsized":"2024-04-16T21:08:54.366349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654394+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:07:44.654394+0000","last_peered":"2024-04-16T21:07:44.654394+0000","last_clean":"2024-04-16T21:07:44.654394+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:07:44.654394+0000","last_undegraded":"2024-04-16T21:07:44.654394+0000","last_fullsized":"2024-04-16T21:07:44.654394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:57:15.042193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652729+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:07:39.652729+0000","last_peered":"2024-04-16T21:07:39.652729+0000","last_clean":"2024-04-16T21:07:39.652729+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:07:39.652729+0000","last_undegraded":"2024-04-16T21:07:39.652729+0000","last_fullsized":"2024-04-16T21:07:39.652729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:02.023550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647634+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:06:59.647634+0000","last_peered":"2024-04-16T21:06:59.647634+0000","last_clean":"2024-04-16T21:06:59.647634+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:06:59.647634+0000","last_undegraded":"2024-04-16T21:06:59.647634+0000","last_fullsized":"2024-04-16T21:06:59.647634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'3","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492346+0000","last_change":"2024-04-16T19:25:15.483924+0000","last_active":"2024-04-16T21:07:24.492346+0000","last_peered":"2024-04-16T21:07:24.492346+0000","last_clean":"2024-04-16T21:07:24.492346+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:07:24.492346+0000","last_undegraded":"2024-04-16T21:07:24.492346+0000","last_fullsized":"2024-04-16T21:07:24.492346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'705","reported_seq":2159,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505330+0000","last_change":"2024-04-16T19:25:13.475694+0000","last_active":"2024-04-16T21:08:44.505330+0000","last_peered":"2024-04-16T21:08:44.505330+0000","last_clean":"2024-04-16T21:08:44.505330+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:08:44.505330+0000","last_undegraded":"2024-04-16T21:08:44.505330+0000","last_fullsized":"2024-04-16T21:08:44.505330+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:56.172757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'247","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364687+0000","last_change":"2024-04-16T19:25:11.466002+0000","last_active":"2024-04-16T21:08:44.364687+0000","last_peered":"2024-04-16T21:08:44.364687+0000","last_clean":"2024-04-16T21:08:44.364687+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:08:44.364687+0000","last_undegraded":"2024-04-16T21:08:44.364687+0000","last_fullsized":"2024-04-16T21:08:44.364687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:38.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'27","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367274+0000","last_change":"2024-04-16T19:25:09.469480+0000","last_active":"2024-04-16T21:08:59.367274+0000","last_peered":"2024-04-16T21:08:59.367274+0000","last_clean":"2024-04-16T21:08:59.367274+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:08:59.367274+0000","last_undegraded":"2024-04-16T21:08:59.367274+0000","last_fullsized":"2024-04-16T21:08:59.367274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:00.355585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364874+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:08:49.364874+0000","last_peered":"2024-04-16T21:08:49.364874+0000","last_clean":"2024-04-16T21:08:49.364874+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:08:49.364874+0000","last_undegraded":"2024-04-16T21:08:49.364874+0000","last_fullsized":"2024-04-16T21:08:49.364874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:25.878014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368069+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:09:04.368069+0000","last_peered":"2024-04-16T21:09:04.368069+0000","last_clean":"2024-04-16T21:09:04.368069+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:09:04.368069+0000","last_undegraded":"2024-04-16T21:09:04.368069+0000","last_fullsized":"2024-04-16T21:09:04.368069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498140+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:07:59.498140+0000","last_peered":"2024-04-16T21:07:59.498140+0000","last_clean":"2024-04-16T21:07:59.498140+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:07:59.498140+0000","last_undegraded":"2024-04-16T21:07:59.498140+0000","last_fullsized":"2024-04-16T21:07:59.498140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372342+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:08:59.372342+0000","last_peered":"2024-04-16T21:08:59.372342+0000","last_clean":"2024-04-16T21:08:59.372342+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:08:59.372342+0000","last_undegraded":"2024-04-16T21:08:59.372342+0000","last_fullsized":"2024-04-16T21:08:59.372342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:38.135155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":349,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373804+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:09:09.373804+0000","last_peered":"2024-04-16T21:09:09.373804+0000","last_clean":"2024-04-16T21:09:09.373804+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:09:09.373804+0000","last_undegraded":"2024-04-16T21:09:09.373804+0000","last_fullsized":"2024-04-16T21:09:09.373804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367113+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:08:59.367113+0000","last_peered":"2024-04-16T21:08:59.367113+0000","last_clean":"2024-04-16T21:08:59.367113+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:08:59.367113+0000","last_undegraded":"2024-04-16T21:08:59.367113+0000","last_fullsized":"2024-04-16T21:08:59.367113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:50.295932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660391+0000","last_change":"2024-04-16T19:24:51.572391+0000","last_active":"2024-04-16T21:08:24.660391+0000","last_peered":"2024-04-16T21:08:24.660391+0000","last_clean":"2024-04-16T21:08:24.660391+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:08:24.660391+0000","last_undegraded":"2024-04-16T21:08:24.660391+0000","last_fullsized":"2024-04-16T21:08:24.660391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:04.095435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495516+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:07:44.495516+0000","last_peered":"2024-04-16T21:07:44.495516+0000","last_clean":"2024-04-16T21:07:44.495516+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:07:44.495516+0000","last_undegraded":"2024-04-16T21:07:44.495516+0000","last_fullsized":"2024-04-16T21:07:44.495516+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:34.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2002,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653496+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:07:39.653496+0000","last_peered":"2024-04-16T21:07:39.653496+0000","last_clean":"2024-04-16T21:07:39.653496+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:07:39.653496+0000","last_undegraded":"2024-04-16T21:07:39.653496+0000","last_fullsized":"2024-04-16T21:07:39.653496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:04:06.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496500+0000","last_change":"2024-04-16T19:25:41.510087+0000","last_active":"2024-04-16T21:07:49.496500+0000","last_peered":"2024-04-16T21:07:49.496500+0000","last_clean":"2024-04-16T21:07:49.496500+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:07:49.496500+0000","last_undegraded":"2024-04-16T21:07:49.496500+0000","last_fullsized":"2024-04-16T21:07:49.496500+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:45.367111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'23","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370491+0000","last_change":"2024-04-16T19:25:15.487252+0000","last_active":"2024-04-16T21:08:49.370491+0000","last_peered":"2024-04-16T21:08:49.370491+0000","last_clean":"2024-04-16T21:08:49.370491+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:08:49.370491+0000","last_undegraded":"2024-04-16T21:08:49.370491+0000","last_fullsized":"2024-04-16T21:08:49.370491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:14.101479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364063+0000","last_change":"2024-04-16T19:25:13.475862+0000","last_active":"2024-04-16T21:08:44.364063+0000","last_peered":"2024-04-16T21:08:44.364063+0000","last_clean":"2024-04-16T21:08:44.364063+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:08:44.364063+0000","last_undegraded":"2024-04-16T21:08:44.364063+0000","last_fullsized":"2024-04-16T21:08:44.364063+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:47.410023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'714","reported_seq":1574,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364715+0000","last_change":"2024-04-16T19:25:11.465490+0000","last_active":"2024-04-16T21:08:44.364715+0000","last_peered":"2024-04-16T21:08:44.364715+0000","last_clean":"2024-04-16T21:08:44.364715+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:08:44.364715+0000","last_undegraded":"2024-04-16T21:08:44.364715+0000","last_fullsized":"2024-04-16T21:08:44.364715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:03.668322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'6","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664395+0000","last_change":"2024-04-16T19:25:09.463391+0000","last_active":"2024-04-16T21:08:39.664395+0000","last_peered":"2024-04-16T21:08:39.664395+0000","last_clean":"2024-04-16T21:08:39.664395+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:08:39.664395+0000","last_undegraded":"2024-04-16T21:08:39.664395+0000","last_fullsized":"2024-04-16T21:08:39.664395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370504+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:08:49.370504+0000","last_peered":"2024-04-16T21:08:49.370504+0000","last_clean":"2024-04-16T21:08:49.370504+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:08:49.370504+0000","last_undegraded":"2024-04-16T21:08:49.370504+0000","last_fullsized":"2024-04-16T21:08:49.370504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364075+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:08:44.364075+0000","last_peered":"2024-04-16T21:08:44.364075+0000","last_clean":"2024-04-16T21:08:44.364075+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:08:44.364075+0000","last_undegraded":"2024-04-16T21:08:44.364075+0000","last_fullsized":"2024-04-16T21:08:44.364075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660502+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:08:14.660502+0000","last_peered":"2024-04-16T21:08:14.660502+0000","last_clean":"2024-04-16T21:08:14.660502+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:08:14.660502+0000","last_undegraded":"2024-04-16T21:08:14.660502+0000","last_fullsized":"2024-04-16T21:08:14.660502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:07.162476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367144+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:08:59.367144+0000","last_peered":"2024-04-16T21:08:59.367144+0000","last_clean":"2024-04-16T21:08:59.367144+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:08:59.367144+0000","last_undegraded":"2024-04-16T21:08:59.367144+0000","last_fullsized":"2024-04-16T21:08:59.367144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:27.238344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":346,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366566+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:08:59.366566+0000","last_peered":"2024-04-16T21:08:59.366566+0000","last_clean":"2024-04-16T21:08:59.366566+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:08:59.366566+0000","last_undegraded":"2024-04-16T21:08:59.366566+0000","last_fullsized":"2024-04-16T21:08:59.366566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:36.905847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1317,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368814+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:09:09.368814+0000","last_peered":"2024-04-16T21:09:09.368814+0000","last_clean":"2024-04-16T21:09:09.368814+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:09:09.368814+0000","last_undegraded":"2024-04-16T21:09:09.368814+0000","last_fullsized":"2024-04-16T21:09:09.368814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:34.836984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662820+0000","last_change":"2024-04-16T19:24:51.572638+0000","last_active":"2024-04-16T21:08:39.662820+0000","last_peered":"2024-04-16T21:08:39.662820+0000","last_clean":"2024-04-16T21:08:39.662820+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:08:39.662820+0000","last_undegraded":"2024-04-16T21:08:39.662820+0000","last_fullsized":"2024-04-16T21:08:39.662820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:13.926597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503462+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:08:34.503462+0000","last_peered":"2024-04-16T21:08:34.503462+0000","last_clean":"2024-04-16T21:08:34.503462+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:08:34.503462+0000","last_undegraded":"2024-04-16T21:08:34.503462+0000","last_fullsized":"2024-04-16T21:08:34.503462+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:20.743689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1526,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656812+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:07:54.656812+0000","last_peered":"2024-04-16T21:07:54.656812+0000","last_clean":"2024-04-16T21:07:54.656812+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:07:54.656812+0000","last_undegraded":"2024-04-16T21:07:54.656812+0000","last_fullsized":"2024-04-16T21:07:54.656812+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:25.577697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652730+0000","last_change":"2024-04-16T19:25:41.498506+0000","last_active":"2024-04-16T21:07:34.652730+0000","last_peered":"2024-04-16T21:07:34.652730+0000","last_clean":"2024-04-16T21:07:34.652730+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:07:34.652730+0000","last_undegraded":"2024-04-16T21:07:34.652730+0000","last_fullsized":"2024-04-16T21:07:34.652730+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:14.970337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'2","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489802+0000","last_change":"2024-04-16T19:25:15.483677+0000","last_active":"2024-04-16T21:07:09.489802+0000","last_peered":"2024-04-16T21:07:09.489802+0000","last_clean":"2024-04-16T21:07:09.489802+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:07:09.489802+0000","last_undegraded":"2024-04-16T21:07:09.489802+0000","last_fullsized":"2024-04-16T21:07:09.489802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368793+0000","last_change":"2024-04-16T19:25:13.473234+0000","last_active":"2024-04-16T21:09:04.368793+0000","last_peered":"2024-04-16T21:09:04.368793+0000","last_clean":"2024-04-16T21:09:04.368793+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:09:04.368793+0000","last_undegraded":"2024-04-16T21:09:04.368793+0000","last_fullsized":"2024-04-16T21:09:04.368793+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:10.112989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'267","reported_seq":524,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364765+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:08:44.364765+0000","last_peered":"2024-04-16T21:08:44.364765+0000","last_clean":"2024-04-16T21:08:44.364765+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:08:44.364765+0000","last_undegraded":"2024-04-16T21:08:44.364765+0000","last_fullsized":"2024-04-16T21:08:44.364765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:51.610204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'14","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664322+0000","last_change":"2024-04-16T19:25:09.461719+0000","last_active":"2024-04-16T21:08:39.664322+0000","last_peered":"2024-04-16T21:08:39.664322+0000","last_clean":"2024-04-16T21:08:39.664322+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:08:39.664322+0000","last_undegraded":"2024-04-16T21:08:39.664322+0000","last_fullsized":"2024-04-16T21:08:39.664322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:24.102149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372250+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:08:59.372250+0000","last_peered":"2024-04-16T21:08:59.372250+0000","last_clean":"2024-04-16T21:08:59.372250+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:08:59.372250+0000","last_undegraded":"2024-04-16T21:08:59.372250+0000","last_fullsized":"2024-04-16T21:08:59.372250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503593+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:08:34.503593+0000","last_peered":"2024-04-16T21:08:34.503593+0000","last_clean":"2024-04-16T21:08:34.503593+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:08:34.503593+0000","last_undegraded":"2024-04-16T21:08:34.503593+0000","last_fullsized":"2024-04-16T21:08:34.503593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495661+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:07:44.495661+0000","last_peered":"2024-04-16T21:07:44.495661+0000","last_clean":"2024-04-16T21:07:44.495661+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:07:44.495661+0000","last_undegraded":"2024-04-16T21:07:44.495661+0000","last_fullsized":"2024-04-16T21:07:44.495661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367951+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:08:59.367951+0000","last_peered":"2024-04-16T21:08:59.367951+0000","last_clean":"2024-04-16T21:08:59.367951+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:08:59.367951+0000","last_undegraded":"2024-04-16T21:08:59.367951+0000","last_fullsized":"2024-04-16T21:08:59.367951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:48.749243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367980+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:08:59.367980+0000","last_peered":"2024-04-16T21:08:59.367980+0000","last_clean":"2024-04-16T21:08:59.367980+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:08:59.367980+0000","last_undegraded":"2024-04-16T21:08:59.367980+0000","last_fullsized":"2024-04-16T21:08:59.367980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372321+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:08:59.372321+0000","last_peered":"2024-04-16T21:08:59.372321+0000","last_clean":"2024-04-16T21:08:59.372321+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:08:59.372321+0000","last_undegraded":"2024-04-16T21:08:59.372321+0000","last_fullsized":"2024-04-16T21:08:59.372321+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660333+0000","last_change":"2024-04-16T19:24:51.573460+0000","last_active":"2024-04-16T21:08:24.660333+0000","last_peered":"2024-04-16T21:08:24.660333+0000","last_clean":"2024-04-16T21:08:24.660333+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:08:24.660333+0000","last_undegraded":"2024-04-16T21:08:24.660333+0000","last_fullsized":"2024-04-16T21:08:24.660333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:31.585745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498095+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:07:59.498095+0000","last_peered":"2024-04-16T21:07:59.498095+0000","last_clean":"2024-04-16T21:07:59.498095+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:07:59.498095+0000","last_undegraded":"2024-04-16T21:07:59.498095+0000","last_fullsized":"2024-04-16T21:07:59.498095+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:33:27.173018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1521,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657201+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:07:59.657201+0000","last_peered":"2024-04-16T21:07:59.657201+0000","last_clean":"2024-04-16T21:07:59.657201+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:07:59.657201+0000","last_undegraded":"2024-04-16T21:07:59.657201+0000","last_fullsized":"2024-04-16T21:07:59.657201+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:16.733301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.493948+0000","last_change":"2024-04-16T19:25:41.502910+0000","last_active":"2024-04-16T21:07:34.493948+0000","last_peered":"2024-04-16T21:07:34.493948+0000","last_clean":"2024-04-16T21:07:34.493948+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:07:34.493948+0000","last_undegraded":"2024-04-16T21:07:34.493948+0000","last_fullsized":"2024-04-16T21:07:34.493948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:25.621893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'12","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489876+0000","last_change":"2024-04-16T19:25:15.485965+0000","last_active":"2024-04-16T21:07:09.489876+0000","last_peered":"2024-04-16T21:07:09.489876+0000","last_clean":"2024-04-16T21:07:09.489876+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:07:09.489876+0000","last_undegraded":"2024-04-16T21:07:09.489876+0000","last_fullsized":"2024-04-16T21:07:09.489876+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'705","reported_seq":2080,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364751+0000","last_change":"2024-04-16T19:25:13.471357+0000","last_active":"2024-04-16T21:08:44.364751+0000","last_peered":"2024-04-16T21:08:44.364751+0000","last_clean":"2024-04-16T21:08:44.364751+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:08:44.364751+0000","last_undegraded":"2024-04-16T21:08:44.364751+0000","last_fullsized":"2024-04-16T21:08:44.364751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'318","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368826+0000","last_change":"2024-04-16T19:25:11.461881+0000","last_active":"2024-04-16T21:09:04.368826+0000","last_peered":"2024-04-16T21:09:04.368826+0000","last_clean":"2024-04-16T21:09:04.368826+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:09:04.368826+0000","last_undegraded":"2024-04-16T21:09:04.368826+0000","last_fullsized":"2024-04-16T21:09:04.368826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:13.030219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'6","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504516+0000","last_change":"2024-04-16T19:25:09.461256+0000","last_active":"2024-04-16T21:08:39.504516+0000","last_peered":"2024-04-16T21:08:39.504516+0000","last_clean":"2024-04-16T21:08:39.504516+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:08:39.504516+0000","last_undegraded":"2024-04-16T21:08:39.504516+0000","last_fullsized":"2024-04-16T21:08:39.504516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:56.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365397+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:08:49.365397+0000","last_peered":"2024-04-16T21:08:49.365397+0000","last_clean":"2024-04-16T21:08:49.365397+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:08:49.365397+0000","last_undegraded":"2024-04-16T21:08:49.365397+0000","last_fullsized":"2024-04-16T21:08:49.365397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:21.322076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662064+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:08:34.662064+0000","last_peered":"2024-04-16T21:08:34.662064+0000","last_clean":"2024-04-16T21:08:34.662064+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:08:34.662064+0000","last_undegraded":"2024-04-16T21:08:34.662064+0000","last_fullsized":"2024-04-16T21:08:34.662064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.656111+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:07:49.656111+0000","last_peered":"2024-04-16T21:07:49.656111+0000","last_clean":"2024-04-16T21:07:49.656111+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:07:49.656111+0000","last_undegraded":"2024-04-16T21:07:49.656111+0000","last_fullsized":"2024-04-16T21:07:49.656111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:42.139799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508363+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:09:04.508363+0000","last_peered":"2024-04-16T21:09:04.508363+0000","last_clean":"2024-04-16T21:09:04.508363+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:09:04.508363+0000","last_undegraded":"2024-04-16T21:09:04.508363+0000","last_fullsized":"2024-04-16T21:09:04.508363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:28:03.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":331,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367176+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:08:59.367176+0000","last_peered":"2024-04-16T21:08:59.367176+0000","last_clean":"2024-04-16T21:08:59.367176+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:08:59.367176+0000","last_undegraded":"2024-04-16T21:08:59.367176+0000","last_fullsized":"2024-04-16T21:08:59.367176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:23.723705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507550+0000","last_change":"2024-04-16T19:24:49.435265+0000","last_active":"2024-04-16T21:08:59.507550+0000","last_peered":"2024-04-16T21:08:59.507550+0000","last_clean":"2024-04-16T21:08:59.507550+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:08:59.507550+0000","last_undegraded":"2024-04-16T21:08:59.507550+0000","last_fullsized":"2024-04-16T21:08:59.507550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:08.160472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661329+0000","last_change":"2024-04-16T19:24:51.438071+0000","last_active":"2024-04-16T21:08:24.661329+0000","last_peered":"2024-04-16T21:08:24.661329+0000","last_clean":"2024-04-16T21:08:24.661329+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:08:24.661329+0000","last_undegraded":"2024-04-16T21:08:24.661329+0000","last_fullsized":"2024-04-16T21:08:24.661329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:42.181631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649582+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:07:19.649582+0000","last_peered":"2024-04-16T21:07:19.649582+0000","last_clean":"2024-04-16T21:07:19.649582+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:07:19.649582+0000","last_undegraded":"2024-04-16T21:07:19.649582+0000","last_fullsized":"2024-04-16T21:07:19.649582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:34:01.244792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653404+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:07:34.653404+0000","last_peered":"2024-04-16T21:07:34.653404+0000","last_clean":"2024-04-16T21:07:34.653404+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:07:34.653404+0000","last_undegraded":"2024-04-16T21:07:34.653404+0000","last_fullsized":"2024-04-16T21:07:34.653404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:03.502015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.500263+0000","last_change":"2024-04-16T19:25:41.506596+0000","last_active":"2024-04-16T21:08:14.500263+0000","last_peered":"2024-04-16T21:08:14.500263+0000","last_clean":"2024-04-16T21:08:14.500263+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:08:14.500263+0000","last_undegraded":"2024-04-16T21:08:14.500263+0000","last_fullsized":"2024-04-16T21:08:14.500263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:01.639245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'4","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655215+0000","last_change":"2024-04-16T19:25:15.480477+0000","last_active":"2024-04-16T21:07:49.655215+0000","last_peered":"2024-04-16T21:07:49.655215+0000","last_clean":"2024-04-16T21:07:49.655215+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:07:49.655215+0000","last_undegraded":"2024-04-16T21:07:49.655215+0000","last_fullsized":"2024-04-16T21:07:49.655215+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654540+0000","last_change":"2024-04-16T19:25:13.470985+0000","last_active":"2024-04-16T21:07:44.654540+0000","last_peered":"2024-04-16T21:07:44.654540+0000","last_clean":"2024-04-16T21:07:44.654540+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:07:44.654540+0000","last_undegraded":"2024-04-16T21:07:44.654540+0000","last_fullsized":"2024-04-16T21:07:44.654540+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:54.589907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'292","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654437+0000","last_change":"2024-04-16T19:25:11.464321+0000","last_active":"2024-04-16T21:07:44.654437+0000","last_peered":"2024-04-16T21:07:44.654437+0000","last_clean":"2024-04-16T21:07:44.654437+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:07:44.654437+0000","last_undegraded":"2024-04-16T21:07:44.654437+0000","last_fullsized":"2024-04-16T21:07:44.654437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:17.793032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'13","reported_seq":236,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650531+0000","last_change":"2024-04-16T19:25:09.460391+0000","last_active":"2024-04-16T21:07:24.650531+0000","last_peered":"2024-04-16T21:07:24.650531+0000","last_clean":"2024-04-16T21:07:24.650531+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:07:24.650531+0000","last_undegraded":"2024-04-16T21:07:24.650531+0000","last_fullsized":"2024-04-16T21:07:24.650531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373217+0000","last_change":"2024-04-16T19:24:51.434685+0000","last_active":"2024-04-16T21:09:04.373217+0000","last_peered":"2024-04-16T21:09:04.373217+0000","last_clean":"2024-04-16T21:09:04.373217+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:09:04.373217+0000","last_undegraded":"2024-04-16T21:09:04.373217+0000","last_fullsized":"2024-04-16T21:09:04.373217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:57:30.733796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650563+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:07:19.650563+0000","last_peered":"2024-04-16T21:07:19.650563+0000","last_clean":"2024-04-16T21:07:19.650563+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:07:19.650563+0000","last_undegraded":"2024-04-16T21:07:19.650563+0000","last_fullsized":"2024-04-16T21:07:19.650563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:28.749905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504494+0000","last_change":"2024-04-16T19:25:41.504766+0000","last_active":"2024-04-16T21:08:39.504494+0000","last_peered":"2024-04-16T21:08:39.504494+0000","last_clean":"2024-04-16T21:08:39.504494+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:08:39.504494+0000","last_undegraded":"2024-04-16T21:08:39.504494+0000","last_fullsized":"2024-04-16T21:08:39.504494+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:37.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":743,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493189+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:07:29.493189+0000","last_peered":"2024-04-16T21:07:29.493189+0000","last_clean":"2024-04-16T21:07:29.493189+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:07:29.493189+0000","last_undegraded":"2024-04-16T21:07:29.493189+0000","last_fullsized":"2024-04-16T21:07:29.493189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:30.438117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493117+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:07:29.493117+0000","last_peered":"2024-04-16T21:07:29.493117+0000","last_clean":"2024-04-16T21:07:29.493117+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:07:29.493117+0000","last_undegraded":"2024-04-16T21:07:29.493117+0000","last_fullsized":"2024-04-16T21:07:29.493117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:51.673404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493130+0000","last_change":"2024-04-16T19:24:49.434342+0000","last_active":"2024-04-16T21:07:29.493130+0000","last_peered":"2024-04-16T21:07:29.493130+0000","last_clean":"2024-04-16T21:07:29.493130+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:07:29.493130+0000","last_undegraded":"2024-04-16T21:07:29.493130+0000","last_fullsized":"2024-04-16T21:07:29.493130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:07.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651977+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:07:29.651977+0000","last_peered":"2024-04-16T21:07:29.651977+0000","last_clean":"2024-04-16T21:07:29.651977+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:07:29.651977+0000","last_undegraded":"2024-04-16T21:07:29.651977+0000","last_fullsized":"2024-04-16T21:07:29.651977+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:21.199802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'118","reported_seq":1711,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491510+0000","last_change":"2024-04-16T19:25:49.520399+0000","last_active":"2024-04-16T21:07:19.491510+0000","last_peered":"2024-04-16T21:07:19.491510+0000","last_clean":"2024-04-16T21:07:19.491510+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:07:19.491510+0000","last_undegraded":"2024-04-16T21:07:19.491510+0000","last_fullsized":"2024-04-16T21:07:19.491510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:16:26.165977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659512+0000","last_change":"2024-04-16T19:25:41.505754+0000","last_active":"2024-04-16T21:08:14.659512+0000","last_peered":"2024-04-16T21:08:14.659512+0000","last_clean":"2024-04-16T21:08:14.659512+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:08:14.659512+0000","last_undegraded":"2024-04-16T21:08:14.659512+0000","last_fullsized":"2024-04-16T21:08:14.659512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:10.716421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655168+0000","last_change":"2024-04-16T19:25:15.482484+0000","last_active":"2024-04-16T21:07:49.655168+0000","last_peered":"2024-04-16T21:07:49.655168+0000","last_clean":"2024-04-16T21:07:49.655168+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:07:49.655168+0000","last_undegraded":"2024-04-16T21:07:49.655168+0000","last_fullsized":"2024-04-16T21:07:49.655168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:25.761792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655229+0000","last_change":"2024-04-16T19:25:13.472620+0000","last_active":"2024-04-16T21:07:44.655229+0000","last_peered":"2024-04-16T21:07:44.655229+0000","last_clean":"2024-04-16T21:07:44.655229+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:07:44.655229+0000","last_undegraded":"2024-04-16T21:07:44.655229+0000","last_fullsized":"2024-04-16T21:07:44.655229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:28.935744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1140","reported_seq":2898,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654522+0000","last_change":"2024-04-16T19:25:11.462713+0000","last_active":"2024-04-16T21:07:44.654522+0000","last_peered":"2024-04-16T21:07:44.654522+0000","last_clean":"2024-04-16T21:07:44.654522+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:07:44.654522+0000","last_undegraded":"2024-04-16T21:07:44.654522+0000","last_fullsized":"2024-04-16T21:07:44.654522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:45.401459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'9","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490903+0000","last_change":"2024-04-16T19:25:09.466079+0000","last_active":"2024-04-16T21:07:14.490903+0000","last_peered":"2024-04-16T21:07:14.490903+0000","last_clean":"2024-04-16T21:07:14.490903+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:07:14.490903+0000","last_undegraded":"2024-04-16T21:07:14.490903+0000","last_fullsized":"2024-04-16T21:07:14.490903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369024+0000","last_change":"2024-04-16T19:24:51.442540+0000","last_active":"2024-04-16T21:09:14.369024+0000","last_peered":"2024-04-16T21:09:14.369024+0000","last_clean":"2024-04-16T21:09:14.369024+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:09:14.369024+0000","last_undegraded":"2024-04-16T21:09:14.369024+0000","last_fullsized":"2024-04-16T21:09:14.369024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491649+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:07:19.491649+0000","last_peered":"2024-04-16T21:07:19.491649+0000","last_clean":"2024-04-16T21:07:19.491649+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:07:19.491649+0000","last_undegraded":"2024-04-16T21:07:19.491649+0000","last_fullsized":"2024-04-16T21:07:19.491649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662604+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:08:29.662604+0000","last_peered":"2024-04-16T21:08:29.662604+0000","last_clean":"2024-04-16T21:08:29.662604+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:08:29.662604+0000","last_undegraded":"2024-04-16T21:08:29.662604+0000","last_fullsized":"2024-04-16T21:08:29.662604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493228+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:07:29.493228+0000","last_peered":"2024-04-16T21:07:29.493228+0000","last_clean":"2024-04-16T21:07:29.493228+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:07:29.493228+0000","last_undegraded":"2024-04-16T21:07:29.493228+0000","last_fullsized":"2024-04-16T21:07:29.493228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:10.368115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651116+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:07:29.651116+0000","last_peered":"2024-04-16T21:07:29.651116+0000","last_clean":"2024-04-16T21:07:29.651116+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:07:29.651116+0000","last_undegraded":"2024-04-16T21:07:29.651116+0000","last_fullsized":"2024-04-16T21:07:29.651116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:05.198319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652671+0000","last_change":"2024-04-16T19:24:49.431431+0000","last_active":"2024-04-16T21:07:29.652671+0000","last_peered":"2024-04-16T21:07:29.652671+0000","last_clean":"2024-04-16T21:07:29.652671+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:07:29.652671+0000","last_undegraded":"2024-04-16T21:07:29.652671+0000","last_fullsized":"2024-04-16T21:07:29.652671+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652772+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:07:34.652772+0000","last_peered":"2024-04-16T21:07:34.652772+0000","last_clean":"2024-04-16T21:07:34.652772+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:07:34.652772+0000","last_undegraded":"2024-04-16T21:07:34.652772+0000","last_fullsized":"2024-04-16T21:07:34.652772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:31:39.104319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1703,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501136+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:08:19.501136+0000","last_peered":"2024-04-16T21:08:19.501136+0000","last_clean":"2024-04-16T21:08:19.501136+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:08:19.501136+0000","last_undegraded":"2024-04-16T21:08:19.501136+0000","last_fullsized":"2024-04-16T21:08:19.501136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:15.107893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649480+0000","last_change":"2024-04-16T19:25:41.499274+0000","last_active":"2024-04-16T21:07:14.649480+0000","last_peered":"2024-04-16T21:07:14.649480+0000","last_clean":"2024-04-16T21:07:14.649480+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:07:14.649480+0000","last_undegraded":"2024-04-16T21:07:14.649480+0000","last_fullsized":"2024-04-16T21:07:14.649480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:22:55.194832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'20","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647308+0000","last_change":"2024-04-16T19:25:15.485235+0000","last_active":"2024-04-16T21:07:04.647308+0000","last_peered":"2024-04-16T21:07:04.647308+0000","last_clean":"2024-04-16T21:07:04.647308+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:07:04.647308+0000","last_undegraded":"2024-04-16T21:07:04.647308+0000","last_fullsized":"2024-04-16T21:07:04.647308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652131+0000","last_change":"2024-04-16T19:25:13.476041+0000","last_active":"2024-04-16T21:07:29.652131+0000","last_peered":"2024-04-16T21:07:29.652131+0000","last_clean":"2024-04-16T21:07:29.652131+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:07:29.652131+0000","last_undegraded":"2024-04-16T21:07:29.652131+0000","last_fullsized":"2024-04-16T21:07:29.652131+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:17.872253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'274","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651118+0000","last_change":"2024-04-16T19:25:11.462598+0000","last_active":"2024-04-16T21:07:19.651118+0000","last_peered":"2024-04-16T21:07:19.651118+0000","last_clean":"2024-04-16T21:07:19.651118+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:07:19.651118+0000","last_undegraded":"2024-04-16T21:07:19.651118+0000","last_fullsized":"2024-04-16T21:07:19.651118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:55.581849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'8","reported_seq":245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652902+0000","last_change":"2024-04-16T19:25:09.461313+0000","last_active":"2024-04-16T21:07:39.652902+0000","last_peered":"2024-04-16T21:07:39.652902+0000","last_clean":"2024-04-16T21:07:39.652902+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:07:39.652902+0000","last_undegraded":"2024-04-16T21:07:39.652902+0000","last_fullsized":"2024-04-16T21:07:39.652902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650513+0000","last_change":"2024-04-16T19:24:51.435662+0000","last_active":"2024-04-16T21:07:24.650513+0000","last_peered":"2024-04-16T21:07:24.650513+0000","last_clean":"2024-04-16T21:07:24.650513+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:07:24.650513+0000","last_undegraded":"2024-04-16T21:07:24.650513+0000","last_fullsized":"2024-04-16T21:07:24.650513+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:57.768573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491559+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:07:19.491559+0000","last_peered":"2024-04-16T21:07:19.491559+0000","last_clean":"2024-04-16T21:07:19.491559+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:07:19.491559+0000","last_undegraded":"2024-04-16T21:07:19.491559+0000","last_fullsized":"2024-04-16T21:07:19.491559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:46:59.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662208+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:08:29.662208+0000","last_peered":"2024-04-16T21:08:29.662208+0000","last_clean":"2024-04-16T21:08:29.662208+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:08:29.662208+0000","last_undegraded":"2024-04-16T21:08:29.662208+0000","last_fullsized":"2024-04-16T21:08:29.662208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":611,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651931+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:07:29.651931+0000","last_peered":"2024-04-16T21:07:29.651931+0000","last_clean":"2024-04-16T21:07:29.651931+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:07:29.651931+0000","last_undegraded":"2024-04-16T21:07:29.651931+0000","last_fullsized":"2024-04-16T21:07:29.651931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:01.385906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493180+0000","last_change":"2024-04-16T19:24:47.424747+0000","last_active":"2024-04-16T21:07:29.493180+0000","last_peered":"2024-04-16T21:07:29.493180+0000","last_clean":"2024-04-16T21:07:29.493180+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:07:29.493180+0000","last_undegraded":"2024-04-16T21:07:29.493180+0000","last_fullsized":"2024-04-16T21:07:29.493180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:48.064472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652026+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:07:29.652026+0000","last_peered":"2024-04-16T21:07:29.652026+0000","last_clean":"2024-04-16T21:07:29.652026+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:07:29.652026+0000","last_undegraded":"2024-04-16T21:07:29.652026+0000","last_fullsized":"2024-04-16T21:07:29.652026+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:21.377903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653531+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:07:34.653531+0000","last_peered":"2024-04-16T21:07:34.653531+0000","last_clean":"2024-04-16T21:07:34.653531+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:07:34.653531+0000","last_undegraded":"2024-04-16T21:07:34.653531+0000","last_fullsized":"2024-04-16T21:07:34.653531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659404+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:08:19.659404+0000","last_peered":"2024-04-16T21:08:19.659404+0000","last_clean":"2024-04-16T21:08:19.659404+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:08:19.659404+0000","last_undegraded":"2024-04-16T21:08:19.659404+0000","last_fullsized":"2024-04-16T21:08:19.659404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493079+0000","last_change":"2024-04-16T19:25:41.508866+0000","last_active":"2024-04-16T21:07:29.493079+0000","last_peered":"2024-04-16T21:07:29.493079+0000","last_clean":"2024-04-16T21:07:29.493079+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:07:29.493079+0000","last_undegraded":"2024-04-16T21:07:29.493079+0000","last_fullsized":"2024-04-16T21:07:29.493079+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:29.426956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'22","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.486695+0000","last_change":"2024-04-16T19:25:15.480626+0000","last_active":"2024-04-16T21:06:49.486695+0000","last_peered":"2024-04-16T21:06:49.486695+0000","last_clean":"2024-04-16T21:06:49.486695+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:06:49.486695+0000","last_undegraded":"2024-04-16T21:06:49.486695+0000","last_fullsized":"2024-04-16T21:06:49.486695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:39.898263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'705","reported_seq":2140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491714+0000","last_change":"2024-04-16T19:25:13.473351+0000","last_active":"2024-04-16T21:07:19.491714+0000","last_peered":"2024-04-16T21:07:19.491714+0000","last_clean":"2024-04-16T21:07:19.491714+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:07:19.491714+0000","last_undegraded":"2024-04-16T21:07:19.491714+0000","last_fullsized":"2024-04-16T21:07:19.491714+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:04.370192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1017","reported_seq":1516,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651283+0000","last_change":"2024-04-16T19:25:11.466163+0000","last_active":"2024-04-16T21:07:29.651283+0000","last_peered":"2024-04-16T21:07:29.651283+0000","last_clean":"2024-04-16T21:07:29.651283+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:07:29.651283+0000","last_undegraded":"2024-04-16T21:07:29.651283+0000","last_fullsized":"2024-04-16T21:07:29.651283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:40.676377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'14","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652919+0000","last_change":"2024-04-16T19:25:09.467909+0000","last_active":"2024-04-16T21:07:39.652919+0000","last_peered":"2024-04-16T21:07:39.652919+0000","last_clean":"2024-04-16T21:07:39.652919+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:07:39.652919+0000","last_undegraded":"2024-04-16T21:07:39.652919+0000","last_fullsized":"2024-04-16T21:07:39.652919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:05.530059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651388+0000","last_change":"2024-04-16T19:24:51.438772+0000","last_active":"2024-04-16T21:07:24.651388+0000","last_peered":"2024-04-16T21:07:24.651388+0000","last_clean":"2024-04-16T21:07:24.651388+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:07:24.651388+0000","last_undegraded":"2024-04-16T21:07:24.651388+0000","last_fullsized":"2024-04-16T21:07:24.651388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491555+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:07:19.491555+0000","last_peered":"2024-04-16T21:07:19.491555+0000","last_clean":"2024-04-16T21:07:19.491555+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:07:19.491555+0000","last_undegraded":"2024-04-16T21:07:19.491555+0000","last_fullsized":"2024-04-16T21:07:19.491555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661253+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:08:29.661253+0000","last_peered":"2024-04-16T21:08:29.661253+0000","last_clean":"2024-04-16T21:08:29.661253+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:08:29.661253+0000","last_undegraded":"2024-04-16T21:08:29.661253+0000","last_fullsized":"2024-04-16T21:08:29.661253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652693+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:07:29.652693+0000","last_peered":"2024-04-16T21:07:29.652693+0000","last_clean":"2024-04-16T21:07:29.652693+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:07:29.652693+0000","last_undegraded":"2024-04-16T21:07:29.652693+0000","last_fullsized":"2024-04-16T21:07:29.652693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652054+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:07:29.652054+0000","last_peered":"2024-04-16T21:07:29.652054+0000","last_clean":"2024-04-16T21:07:29.652054+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:07:29.652054+0000","last_undegraded":"2024-04-16T21:07:29.652054+0000","last_fullsized":"2024-04-16T21:07:29.652054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:52.547347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651195+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:07:29.651195+0000","last_peered":"2024-04-16T21:07:29.651195+0000","last_clean":"2024-04-16T21:07:29.651195+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:07:29.651195+0000","last_undegraded":"2024-04-16T21:07:29.651195+0000","last_fullsized":"2024-04-16T21:07:29.651195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648537+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:07:04.648537+0000","last_peered":"2024-04-16T21:07:04.648537+0000","last_clean":"2024-04-16T21:07:04.648537+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:07:04.648537+0000","last_undegraded":"2024-04-16T21:07:04.648537+0000","last_fullsized":"2024-04-16T21:07:04.648537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:37.665943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2028,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651065+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:07:19.651065+0000","last_peered":"2024-04-16T21:07:19.651065+0000","last_clean":"2024-04-16T21:07:19.651065+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:07:19.651065+0000","last_undegraded":"2024-04-16T21:07:19.651065+0000","last_fullsized":"2024-04-16T21:07:19.651065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:48.277899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648877+0000","last_change":"2024-04-16T19:25:41.507140+0000","last_active":"2024-04-16T21:07:14.648877+0000","last_peered":"2024-04-16T21:07:14.648877+0000","last_clean":"2024-04-16T21:07:14.648877+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:07:14.648877+0000","last_undegraded":"2024-04-16T21:07:14.648877+0000","last_fullsized":"2024-04-16T21:07:14.648877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:14:47.746067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'5","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645995+0000","last_change":"2024-04-16T19:25:15.481140+0000","last_active":"2024-04-16T21:06:49.645995+0000","last_peered":"2024-04-16T21:06:49.645995+0000","last_clean":"2024-04-16T21:06:49.645995+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:06:49.645995+0000","last_undegraded":"2024-04-16T21:06:49.645995+0000","last_fullsized":"2024-04-16T21:06:49.645995+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:22.980049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655276+0000","last_change":"2024-04-16T19:25:13.476134+0000","last_active":"2024-04-16T21:07:54.655276+0000","last_peered":"2024-04-16T21:07:54.655276+0000","last_clean":"2024-04-16T21:07:54.655276+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:07:54.655276+0000","last_undegraded":"2024-04-16T21:07:54.655276+0000","last_fullsized":"2024-04-16T21:07:54.655276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'305","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656185+0000","last_change":"2024-04-16T19:25:11.465819+0000","last_active":"2024-04-16T21:07:54.656185+0000","last_peered":"2024-04-16T21:07:54.656185+0000","last_clean":"2024-04-16T21:07:54.656185+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:07:54.656185+0000","last_undegraded":"2024-04-16T21:07:54.656185+0000","last_fullsized":"2024-04-16T21:07:54.656185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:38.882914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'6","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655341+0000","last_change":"2024-04-16T19:25:09.469614+0000","last_active":"2024-04-16T21:07:49.655341+0000","last_peered":"2024-04-16T21:07:49.655341+0000","last_clean":"2024-04-16T21:07:49.655341+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:07:49.655341+0000","last_undegraded":"2024-04-16T21:07:49.655341+0000","last_fullsized":"2024-04-16T21:07:49.655341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366575+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:08:49.366575+0000","last_peered":"2024-04-16T21:08:49.366575+0000","last_clean":"2024-04-16T21:08:49.366575+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:08:49.366575+0000","last_undegraded":"2024-04-16T21:08:49.366575+0000","last_fullsized":"2024-04-16T21:08:49.366575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:54.123433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502246+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:08:29.502246+0000","last_peered":"2024-04-16T21:08:29.502246+0000","last_clean":"2024-04-16T21:08:29.502246+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:08:29.502246+0000","last_undegraded":"2024-04-16T21:08:29.502246+0000","last_fullsized":"2024-04-16T21:08:29.502246+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:19:16.213346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.374003+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:09:09.374003+0000","last_peered":"2024-04-16T21:09:09.374003+0000","last_clean":"2024-04-16T21:09:09.374003+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:09:09.374003+0000","last_undegraded":"2024-04-16T21:09:09.374003+0000","last_fullsized":"2024-04-16T21:09:09.374003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491759+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:07:19.491759+0000","last_peered":"2024-04-16T21:07:19.491759+0000","last_clean":"2024-04-16T21:07:19.491759+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:07:19.491759+0000","last_undegraded":"2024-04-16T21:07:19.491759+0000","last_fullsized":"2024-04-16T21:07:19.491759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:50.759439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651037+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:07:19.651037+0000","last_peered":"2024-04-16T21:07:19.651037+0000","last_clean":"2024-04-16T21:07:19.651037+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:07:19.651037+0000","last_undegraded":"2024-04-16T21:07:19.651037+0000","last_fullsized":"2024-04-16T21:07:19.651037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494045+0000","last_change":"2024-04-16T19:24:51.574221+0000","last_active":"2024-04-16T21:07:34.494045+0000","last_peered":"2024-04-16T21:07:34.494045+0000","last_clean":"2024-04-16T21:07:34.494045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:07:34.494045+0000","last_undegraded":"2024-04-16T21:07:34.494045+0000","last_fullsized":"2024-04-16T21:07:34.494045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:09.678742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650574+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:07:19.650574+0000","last_peered":"2024-04-16T21:07:19.650574+0000","last_clean":"2024-04-16T21:07:19.650574+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:07:19.650574+0000","last_undegraded":"2024-04-16T21:07:19.650574+0000","last_fullsized":"2024-04-16T21:07:19.650574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:44.374493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1826,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365476+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:08:49.365476+0000","last_peered":"2024-04-16T21:08:49.365476+0000","last_clean":"2024-04-16T21:08:49.365476+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:08:49.365476+0000","last_undegraded":"2024-04-16T21:08:49.365476+0000","last_fullsized":"2024-04-16T21:08:49.365476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:09.652050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649723+0000","last_change":"2024-04-16T19:25:41.504184+0000","last_active":"2024-04-16T21:07:14.649723+0000","last_peered":"2024-04-16T21:07:14.649723+0000","last_clean":"2024-04-16T21:07:14.649723+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:07:14.649723+0000","last_undegraded":"2024-04-16T21:07:14.649723+0000","last_fullsized":"2024-04-16T21:07:14.649723+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645440+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:06:49.645440+0000","last_peered":"2024-04-16T21:06:49.645440+0000","last_clean":"2024-04-16T21:06:49.645440+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:06:49.645440+0000","last_undegraded":"2024-04-16T21:06:49.645440+0000","last_fullsized":"2024-04-16T21:06:49.645440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:47:48.700988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497245+0000","last_change":"2024-04-16T19:25:13.472015+0000","last_active":"2024-04-16T21:07:54.497245+0000","last_peered":"2024-04-16T21:07:54.497245+0000","last_clean":"2024-04-16T21:07:54.497245+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:07:54.497245+0000","last_undegraded":"2024-04-16T21:07:54.497245+0000","last_fullsized":"2024-04-16T21:07:54.497245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'121","reported_seq":272,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655968+0000","last_change":"2024-04-16T19:25:11.465848+0000","last_active":"2024-04-16T21:07:54.655968+0000","last_peered":"2024-04-16T21:07:54.655968+0000","last_clean":"2024-04-16T21:07:54.655968+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:07:54.655968+0000","last_undegraded":"2024-04-16T21:07:54.655968+0000","last_fullsized":"2024-04-16T21:07:54.655968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'4","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655987+0000","last_change":"2024-04-16T19:25:09.462205+0000","last_active":"2024-04-16T21:07:49.655987+0000","last_peered":"2024-04-16T21:07:49.655987+0000","last_clean":"2024-04-16T21:07:49.655987+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:07:49.655987+0000","last_undegraded":"2024-04-16T21:07:49.655987+0000","last_fullsized":"2024-04-16T21:07:49.655987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:53.677676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372422+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:08:59.372422+0000","last_peered":"2024-04-16T21:08:59.372422+0000","last_clean":"2024-04-16T21:08:59.372422+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:08:59.372422+0000","last_undegraded":"2024-04-16T21:08:59.372422+0000","last_fullsized":"2024-04-16T21:08:59.372422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364818+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:08:49.364818+0000","last_peered":"2024-04-16T21:08:49.364818+0000","last_clean":"2024-04-16T21:08:49.364818+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:08:49.364818+0000","last_undegraded":"2024-04-16T21:08:49.364818+0000","last_fullsized":"2024-04-16T21:08:49.364818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.368088+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:08:59.368088+0000","last_peered":"2024-04-16T21:08:59.368088+0000","last_clean":"2024-04-16T21:08:59.368088+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:08:59.368088+0000","last_undegraded":"2024-04-16T21:08:59.368088+0000","last_fullsized":"2024-04-16T21:08:59.368088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:34.577971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651049+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:07:19.651049+0000","last_peered":"2024-04-16T21:07:19.651049+0000","last_clean":"2024-04-16T21:07:19.651049+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:07:19.651049+0000","last_undegraded":"2024-04-16T21:07:19.651049+0000","last_fullsized":"2024-04-16T21:07:19.651049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:14.554679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491744+0000","last_change":"2024-04-16T19:24:49.432195+0000","last_active":"2024-04-16T21:07:19.491744+0000","last_peered":"2024-04-16T21:07:19.491744+0000","last_clean":"2024-04-16T21:07:19.491744+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:07:19.491744+0000","last_undegraded":"2024-04-16T21:07:19.491744+0000","last_fullsized":"2024-04-16T21:07:19.491744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:18.116203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653498+0000","last_change":"2024-04-16T19:24:51.573063+0000","last_active":"2024-04-16T21:07:34.653498+0000","last_peered":"2024-04-16T21:07:34.653498+0000","last_clean":"2024-04-16T21:07:34.653498+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:07:34.653498+0000","last_undegraded":"2024-04-16T21:07:34.653498+0000","last_fullsized":"2024-04-16T21:07:34.653498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:01.882223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649776+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:07:19.649776+0000","last_peered":"2024-04-16T21:07:19.649776+0000","last_clean":"2024-04-16T21:07:19.649776+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:07:19.649776+0000","last_undegraded":"2024-04-16T21:07:19.649776+0000","last_fullsized":"2024-04-16T21:07:19.649776+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:28:47.506010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491827+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:07:19.491827+0000","last_peered":"2024-04-16T21:07:19.491827+0000","last_clean":"2024-04-16T21:07:19.491827+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:07:19.491827+0000","last_undegraded":"2024-04-16T21:07:19.491827+0000","last_fullsized":"2024-04-16T21:07:19.491827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:32:33.273769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662724+0000","last_change":"2024-04-16T19:25:41.509494+0000","last_active":"2024-04-16T21:08:39.662724+0000","last_peered":"2024-04-16T21:08:39.662724+0000","last_clean":"2024-04-16T21:08:39.662724+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:08:39.662724+0000","last_undegraded":"2024-04-16T21:08:39.662724+0000","last_fullsized":"2024-04-16T21:08:39.662724+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'8","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645398+0000","last_change":"2024-04-16T19:25:15.483611+0000","last_active":"2024-04-16T21:06:49.645398+0000","last_peered":"2024-04-16T21:06:49.645398+0000","last_clean":"2024-04-16T21:06:49.645398+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:06:49.645398+0000","last_undegraded":"2024-04-16T21:06:49.645398+0000","last_fullsized":"2024-04-16T21:06:49.645398+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656848+0000","last_change":"2024-04-16T19:25:13.472723+0000","last_active":"2024-04-16T21:07:54.656848+0000","last_peered":"2024-04-16T21:07:54.656848+0000","last_clean":"2024-04-16T21:07:54.656848+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:07:54.656848+0000","last_undegraded":"2024-04-16T21:07:54.656848+0000","last_fullsized":"2024-04-16T21:07:54.656848+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'321","reported_seq":850,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497294+0000","last_change":"2024-04-16T19:25:11.464441+0000","last_active":"2024-04-16T21:07:54.497294+0000","last_peered":"2024-04-16T21:07:54.497294+0000","last_clean":"2024-04-16T21:07:54.497294+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:07:54.497294+0000","last_undegraded":"2024-04-16T21:07:54.497294+0000","last_fullsized":"2024-04-16T21:07:54.497294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:55.683718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'12","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655101+0000","last_change":"2024-04-16T19:25:09.465881+0000","last_active":"2024-04-16T21:07:49.655101+0000","last_peered":"2024-04-16T21:07:49.655101+0000","last_clean":"2024-04-16T21:07:49.655101+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:07:49.655101+0000","last_undegraded":"2024-04-16T21:07:49.655101+0000","last_fullsized":"2024-04-16T21:07:49.655101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648098+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:07:09.648098+0000","last_peered":"2024-04-16T21:07:09.648098+0000","last_clean":"2024-04-16T21:07:09.648098+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:07:09.648098+0000","last_undegraded":"2024-04-16T21:07:09.648098+0000","last_fullsized":"2024-04-16T21:07:09.648098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648274+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:07:04.648274+0000","last_peered":"2024-04-16T21:07:04.648274+0000","last_clean":"2024-04-16T21:07:04.648274+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:07:04.648274+0000","last_undegraded":"2024-04-16T21:07:04.648274+0000","last_fullsized":"2024-04-16T21:07:04.648274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:48.449534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650601+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:07:19.650601+0000","last_peered":"2024-04-16T21:07:19.650601+0000","last_clean":"2024-04-16T21:07:19.650601+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:07:19.650601+0000","last_undegraded":"2024-04-16T21:07:19.650601+0000","last_fullsized":"2024-04-16T21:07:19.650601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:17.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507635+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:08:59.507635+0000","last_peered":"2024-04-16T21:08:59.507635+0000","last_clean":"2024-04-16T21:08:59.507635+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:08:59.507635+0000","last_undegraded":"2024-04-16T21:08:59.507635+0000","last_fullsized":"2024-04-16T21:08:59.507635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:40.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368151+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:09:09.368151+0000","last_peered":"2024-04-16T21:09:09.368151+0000","last_clean":"2024-04-16T21:09:09.368151+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:09:09.368151+0000","last_undegraded":"2024-04-16T21:09:09.368151+0000","last_fullsized":"2024-04-16T21:09:09.368151+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:22.117545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653453+0000","last_change":"2024-04-16T19:24:51.442399+0000","last_active":"2024-04-16T21:07:34.653453+0000","last_peered":"2024-04-16T21:07:34.653453+0000","last_clean":"2024-04-16T21:07:34.653453+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:07:34.653453+0000","last_undegraded":"2024-04-16T21:07:34.653453+0000","last_fullsized":"2024-04-16T21:07:34.653453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:24.851144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650701+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:07:19.650701+0000","last_peered":"2024-04-16T21:07:19.650701+0000","last_clean":"2024-04-16T21:07:19.650701+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:07:19.650701+0000","last_undegraded":"2024-04-16T21:07:19.650701+0000","last_fullsized":"2024-04-16T21:07:19.650701+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:42.705352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1306,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491787+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:07:19.491787+0000","last_peered":"2024-04-16T21:07:19.491787+0000","last_clean":"2024-04-16T21:07:19.491787+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:07:19.491787+0000","last_undegraded":"2024-04-16T21:07:19.491787+0000","last_fullsized":"2024-04-16T21:07:19.491787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:35:13.067416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649861+0000","last_change":"2024-04-16T19:25:41.509274+0000","last_active":"2024-04-16T21:07:14.649861+0000","last_peered":"2024-04-16T21:07:14.649861+0000","last_clean":"2024-04-16T21:07:14.649861+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:07:14.649861+0000","last_undegraded":"2024-04-16T21:07:14.649861+0000","last_fullsized":"2024-04-16T21:07:14.649861+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:45.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"47'5","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659574+0000","last_change":"2024-04-16T19:25:15.486177+0000","last_active":"2024-04-16T21:08:14.659574+0000","last_peered":"2024-04-16T21:08:14.659574+0000","last_clean":"2024-04-16T21:08:14.659574+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:08:14.659574+0000","last_undegraded":"2024-04-16T21:08:14.659574+0000","last_fullsized":"2024-04-16T21:08:14.659574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:20.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656265+0000","last_change":"2024-04-16T19:25:13.473925+0000","last_active":"2024-04-16T21:07:54.656265+0000","last_peered":"2024-04-16T21:07:54.656265+0000","last_clean":"2024-04-16T21:07:54.656265+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:07:54.656265+0000","last_undegraded":"2024-04-16T21:07:54.656265+0000","last_fullsized":"2024-04-16T21:07:54.656265+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:22.051385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'241","reported_seq":1104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656229+0000","last_change":"2024-04-16T19:25:11.465793+0000","last_active":"2024-04-16T21:07:54.656229+0000","last_peered":"2024-04-16T21:07:54.656229+0000","last_clean":"2024-04-16T21:07:54.656229+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:07:54.656229+0000","last_undegraded":"2024-04-16T21:07:54.656229+0000","last_fullsized":"2024-04-16T21:07:54.656229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:00.538757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'7","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655923+0000","last_change":"2024-04-16T19:25:09.458604+0000","last_active":"2024-04-16T21:07:49.655923+0000","last_peered":"2024-04-16T21:07:49.655923+0000","last_clean":"2024-04-16T21:07:49.655923+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:07:49.655923+0000","last_undegraded":"2024-04-16T21:07:49.655923+0000","last_fullsized":"2024-04-16T21:07:49.655923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489948+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:07:09.489948+0000","last_peered":"2024-04-16T21:07:09.489948+0000","last_clean":"2024-04-16T21:07:09.489948+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:07:09.489948+0000","last_undegraded":"2024-04-16T21:07:09.489948+0000","last_fullsized":"2024-04-16T21:07:09.489948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647363+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:07:04.647363+0000","last_peered":"2024-04-16T21:07:04.647363+0000","last_clean":"2024-04-16T21:07:04.647363+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:07:04.647363+0000","last_undegraded":"2024-04-16T21:07:04.647363+0000","last_fullsized":"2024-04-16T21:07:04.647363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491726+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:07:19.491726+0000","last_peered":"2024-04-16T21:07:19.491726+0000","last_clean":"2024-04-16T21:07:19.491726+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:07:19.491726+0000","last_undegraded":"2024-04-16T21:07:19.491726+0000","last_fullsized":"2024-04-16T21:07:19.491726+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:45.917687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369571+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:09:09.369571+0000","last_peered":"2024-04-16T21:09:09.369571+0000","last_clean":"2024-04-16T21:09:09.369571+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:09:09.369571+0000","last_undegraded":"2024-04-16T21:09:09.369571+0000","last_fullsized":"2024-04-16T21:09:09.369571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:32.829903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367311+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:08:59.367311+0000","last_peered":"2024-04-16T21:08:59.367311+0000","last_clean":"2024-04-16T21:08:59.367311+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:08:59.367311+0000","last_undegraded":"2024-04-16T21:08:59.367311+0000","last_fullsized":"2024-04-16T21:08:59.367311+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:02.604453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653371+0000","last_change":"2024-04-16T19:24:51.438145+0000","last_active":"2024-04-16T21:07:34.653371+0000","last_peered":"2024-04-16T21:07:34.653371+0000","last_clean":"2024-04-16T21:07:34.653371+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:07:34.653371+0000","last_undegraded":"2024-04-16T21:07:34.653371+0000","last_fullsized":"2024-04-16T21:07:34.653371+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:45.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646595+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:06:54.646595+0000","last_peered":"2024-04-16T21:06:54.646595+0000","last_clean":"2024-04-16T21:06:54.646595+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:06:54.646595+0000","last_undegraded":"2024-04-16T21:06:54.646595+0000","last_fullsized":"2024-04-16T21:06:54.646595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:31.554611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661634+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:08:29.661634+0000","last_peered":"2024-04-16T21:08:29.661634+0000","last_clean":"2024-04-16T21:08:29.661634+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:08:29.661634+0000","last_undegraded":"2024-04-16T21:08:29.661634+0000","last_fullsized":"2024-04-16T21:08:29.661634+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:20.643182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660228+0000","last_change":"2024-04-16T19:25:41.508939+0000","last_active":"2024-04-16T21:08:24.660228+0000","last_peered":"2024-04-16T21:08:24.660228+0000","last_clean":"2024-04-16T21:08:24.660228+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:08:24.660228+0000","last_undegraded":"2024-04-16T21:08:24.660228+0000","last_fullsized":"2024-04-16T21:08:24.660228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'11","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498087+0000","last_change":"2024-04-16T19:25:15.480096+0000","last_active":"2024-04-16T21:07:59.498087+0000","last_peered":"2024-04-16T21:07:59.498087+0000","last_clean":"2024-04-16T21:07:59.498087+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:07:59.498087+0000","last_undegraded":"2024-04-16T21:07:59.498087+0000","last_fullsized":"2024-04-16T21:07:59.498087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:58.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659661+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:08:09.659661+0000","last_peered":"2024-04-16T21:08:09.659661+0000","last_clean":"2024-04-16T21:08:09.659661+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:08:09.659661+0000","last_undegraded":"2024-04-16T21:08:09.659661+0000","last_fullsized":"2024-04-16T21:08:09.659661+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'540","reported_seq":1355,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.643433+0000","last_change":"2024-04-16T19:25:11.465566+0000","last_active":"2024-04-16T21:06:44.643433+0000","last_peered":"2024-04-16T21:06:44.643433+0000","last_clean":"2024-04-16T21:06:44.643433+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:06:44.643433+0000","last_undegraded":"2024-04-16T21:06:44.643433+0000","last_fullsized":"2024-04-16T21:06:44.643433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:52.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'9","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643501+0000","last_change":"2024-04-16T19:25:09.458347+0000","last_active":"2024-04-16T21:06:39.643501+0000","last_peered":"2024-04-16T21:06:39.643501+0000","last_clean":"2024-04-16T21:06:39.643501+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:06:39.643501+0000","last_undegraded":"2024-04-16T21:06:39.643501+0000","last_fullsized":"2024-04-16T21:06:39.643501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649060+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:07:09.649060+0000","last_peered":"2024-04-16T21:07:09.649060+0000","last_clean":"2024-04-16T21:07:09.649060+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:07:09.649060+0000","last_undegraded":"2024-04-16T21:07:09.649060+0000","last_fullsized":"2024-04-16T21:07:09.649060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:41.726894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648043+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:07:04.648043+0000","last_peered":"2024-04-16T21:07:04.648043+0000","last_clean":"2024-04-16T21:07:04.648043+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:07:04.648043+0000","last_undegraded":"2024-04-16T21:07:04.648043+0000","last_fullsized":"2024-04-16T21:07:04.648043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650514+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:07:19.650514+0000","last_peered":"2024-04-16T21:07:19.650514+0000","last_clean":"2024-04-16T21:07:19.650514+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:07:19.650514+0000","last_undegraded":"2024-04-16T21:07:19.650514+0000","last_fullsized":"2024-04-16T21:07:19.650514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:46.054276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.642525+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:06:34.642525+0000","last_peered":"2024-04-16T21:06:34.642525+0000","last_clean":"2024-04-16T21:06:34.642525+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:06:34.642525+0000","last_undegraded":"2024-04-16T21:06:34.642525+0000","last_fullsized":"2024-04-16T21:06:34.642525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:47.298587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369608+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:08:44.369608+0000","last_peered":"2024-04-16T21:08:44.369608+0000","last_clean":"2024-04-16T21:08:44.369608+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:08:44.369608+0000","last_undegraded":"2024-04-16T21:08:44.369608+0000","last_fullsized":"2024-04-16T21:08:44.369608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:38:59.670330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366563+0000","last_change":"2024-04-16T19:24:51.437809+0000","last_active":"2024-04-16T21:08:49.366563+0000","last_peered":"2024-04-16T21:08:49.366563+0000","last_clean":"2024-04-16T21:08:49.366563+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:08:49.366563+0000","last_undegraded":"2024-04-16T21:08:49.366563+0000","last_fullsized":"2024-04-16T21:08:49.366563+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487434+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:06:54.487434+0000","last_peered":"2024-04-16T21:06:54.487434+0000","last_clean":"2024-04-16T21:06:54.487434+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:06:54.487434+0000","last_undegraded":"2024-04-16T21:06:54.487434+0000","last_fullsized":"2024-04-16T21:06:54.487434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:55:01.357303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1337,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502628+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:08:29.502628+0000","last_peered":"2024-04-16T21:08:29.502628+0000","last_clean":"2024-04-16T21:08:29.502628+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:08:29.502628+0000","last_undegraded":"2024-04-16T21:08:29.502628+0000","last_fullsized":"2024-04-16T21:08:29.502628+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:48.136957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661216+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:08:24.661216+0000","last_peered":"2024-04-16T21:08:24.661216+0000","last_clean":"2024-04-16T21:08:24.661216+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:08:24.661216+0000","last_undegraded":"2024-04-16T21:08:24.661216+0000","last_fullsized":"2024-04-16T21:08:24.661216+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:47.003646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'8","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656179+0000","last_change":"2024-04-16T19:25:15.481698+0000","last_active":"2024-04-16T21:07:59.656179+0000","last_peered":"2024-04-16T21:07:59.656179+0000","last_clean":"2024-04-16T21:07:59.656179+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:07:59.656179+0000","last_undegraded":"2024-04-16T21:07:59.656179+0000","last_fullsized":"2024-04-16T21:07:59.656179+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'705","reported_seq":2071,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485874+0000","last_change":"2024-04-16T19:25:13.474684+0000","last_active":"2024-04-16T21:06:44.485874+0000","last_peered":"2024-04-16T21:06:44.485874+0000","last_clean":"2024-04-16T21:06:44.485874+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:06:44.485874+0000","last_undegraded":"2024-04-16T21:06:44.485874+0000","last_fullsized":"2024-04-16T21:06:44.485874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:52.004725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'519","reported_seq":41267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499619+0000","last_change":"2024-04-16T19:25:11.465897+0000","last_active":"2024-04-16T21:08:09.499619+0000","last_peered":"2024-04-16T21:08:09.499619+0000","last_clean":"2024-04-16T21:08:09.499619+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:08:09.499619+0000","last_undegraded":"2024-04-16T21:08:09.499619+0000","last_fullsized":"2024-04-16T21:08:09.499619+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:54.168091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'9","reported_seq":307,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643870+0000","last_change":"2024-04-16T19:25:09.459964+0000","last_active":"2024-04-16T21:06:39.643870+0000","last_peered":"2024-04-16T21:06:39.643870+0000","last_clean":"2024-04-16T21:06:39.643870+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:06:39.643870+0000","last_undegraded":"2024-04-16T21:06:39.643870+0000","last_fullsized":"2024-04-16T21:06:39.643870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649029+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:07:09.649029+0000","last_peered":"2024-04-16T21:07:09.649029+0000","last_clean":"2024-04-16T21:07:09.649029+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:07:09.649029+0000","last_undegraded":"2024-04-16T21:07:09.649029+0000","last_fullsized":"2024-04-16T21:07:09.649029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648095+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:07:04.648095+0000","last_peered":"2024-04-16T21:07:04.648095+0000","last_clean":"2024-04-16T21:07:04.648095+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:07:04.648095+0000","last_undegraded":"2024-04-16T21:07:04.648095+0000","last_fullsized":"2024-04-16T21:07:04.648095+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:23.870654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:04.143195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369636+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:08:44.369636+0000","last_peered":"2024-04-16T21:08:44.369636+0000","last_clean":"2024-04-16T21:08:44.369636+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:08:44.369636+0000","last_undegraded":"2024-04-16T21:08:44.369636+0000","last_fullsized":"2024-04-16T21:08:44.369636+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.642926+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:06:34.642926+0000","last_peered":"2024-04-16T21:06:34.642926+0000","last_clean":"2024-04-16T21:06:34.642926+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:06:34.642926+0000","last_undegraded":"2024-04-16T21:06:34.642926+0000","last_fullsized":"2024-04-16T21:06:34.642926+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364971+0000","last_change":"2024-04-16T19:24:51.573983+0000","last_active":"2024-04-16T21:08:49.364971+0000","last_peered":"2024-04-16T21:08:49.364971+0000","last_clean":"2024-04-16T21:08:49.364971+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:08:49.364971+0000","last_undegraded":"2024-04-16T21:08:49.364971+0000","last_fullsized":"2024-04-16T21:08:49.364971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.645515+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:06:54.645515+0000","last_peered":"2024-04-16T21:06:54.645515+0000","last_clean":"2024-04-16T21:06:54.645515+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:06:54.645515+0000","last_undegraded":"2024-04-16T21:06:54.645515+0000","last_fullsized":"2024-04-16T21:06:54.645515+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'167","reported_seq":2111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502381+0000","last_change":"2024-04-16T19:25:49.519298+0000","last_active":"2024-04-16T21:08:29.502381+0000","last_peered":"2024-04-16T21:08:29.502381+0000","last_clean":"2024-04-16T21:08:29.502381+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:08:29.502381+0000","last_undegraded":"2024-04-16T21:08:29.502381+0000","last_fullsized":"2024-04-16T21:08:29.502381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:10.645836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501890+0000","last_change":"2024-04-16T19:25:41.507268+0000","last_active":"2024-04-16T21:08:24.501890+0000","last_peered":"2024-04-16T21:08:24.501890+0000","last_clean":"2024-04-16T21:08:24.501890+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:08:24.501890+0000","last_undegraded":"2024-04-16T21:08:24.501890+0000","last_fullsized":"2024-04-16T21:08:24.501890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:46.285785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657100+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:07:59.657100+0000","last_peered":"2024-04-16T21:07:59.657100+0000","last_clean":"2024-04-16T21:07:59.657100+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:07:59.657100+0000","last_undegraded":"2024-04-16T21:07:59.657100+0000","last_fullsized":"2024-04-16T21:07:59.657100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:46.607704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644452+0000","last_change":"2024-04-16T19:25:13.472244+0000","last_active":"2024-04-16T21:06:44.644452+0000","last_peered":"2024-04-16T21:06:44.644452+0000","last_clean":"2024-04-16T21:06:44.644452+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:06:44.644452+0000","last_undegraded":"2024-04-16T21:06:44.644452+0000","last_fullsized":"2024-04-16T21:06:44.644452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:59.248190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'573","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644779+0000","last_change":"2024-04-16T19:25:11.468986+0000","last_active":"2024-04-16T21:06:44.644779+0000","last_peered":"2024-04-16T21:06:44.644779+0000","last_clean":"2024-04-16T21:06:44.644779+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:06:44.644779+0000","last_undegraded":"2024-04-16T21:06:44.644779+0000","last_fullsized":"2024-04-16T21:06:44.644779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'23","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498919+0000","last_change":"2024-04-16T19:25:09.469035+0000","last_active":"2024-04-16T21:08:04.498919+0000","last_peered":"2024-04-16T21:08:04.498919+0000","last_clean":"2024-04-16T21:08:04.498919+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:08:04.498919+0000","last_undegraded":"2024-04-16T21:08:04.498919+0000","last_fullsized":"2024-04-16T21:08:04.498919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371344+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:08:54.371344+0000","last_peered":"2024-04-16T21:08:54.371344+0000","last_clean":"2024-04-16T21:08:54.371344+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:08:54.371344+0000","last_undegraded":"2024-04-16T21:08:54.371344+0000","last_fullsized":"2024-04-16T21:08:54.371344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506100+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:08:49.506100+0000","last_peered":"2024-04-16T21:08:49.506100+0000","last_clean":"2024-04-16T21:08:49.506100+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:08:49.506100+0000","last_undegraded":"2024-04-16T21:08:49.506100+0000","last_fullsized":"2024-04-16T21:08:49.506100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:11.958520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649687+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:07:19.649687+0000","last_peered":"2024-04-16T21:07:19.649687+0000","last_clean":"2024-04-16T21:07:19.649687+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:07:19.649687+0000","last_undegraded":"2024-04-16T21:07:19.649687+0000","last_fullsized":"2024-04-16T21:07:19.649687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650982+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:07:19.650982+0000","last_peered":"2024-04-16T21:07:19.650982+0000","last_clean":"2024-04-16T21:07:19.650982+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:07:19.650982+0000","last_undegraded":"2024-04-16T21:07:19.650982+0000","last_fullsized":"2024-04-16T21:07:19.650982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366474+0000","last_change":"2024-04-16T19:24:51.444882+0000","last_active":"2024-04-16T21:08:49.366474+0000","last_peered":"2024-04-16T21:08:49.366474+0000","last_clean":"2024-04-16T21:08:49.366474+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:08:49.366474+0000","last_undegraded":"2024-04-16T21:08:49.366474+0000","last_fullsized":"2024-04-16T21:08:49.366474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:18.795884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487525+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:06:54.487525+0000","last_peered":"2024-04-16T21:06:54.487525+0000","last_clean":"2024-04-16T21:06:54.487525+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:06:54.487525+0000","last_undegraded":"2024-04-16T21:06:54.487525+0000","last_fullsized":"2024-04-16T21:06:54.487525+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:37:36.899944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'127","reported_seq":1509,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502335+0000","last_change":"2024-04-16T19:25:49.517472+0000","last_active":"2024-04-16T21:08:29.502335+0000","last_peered":"2024-04-16T21:08:29.502335+0000","last_clean":"2024-04-16T21:08:29.502335+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:08:29.502335+0000","last_undegraded":"2024-04-16T21:08:29.502335+0000","last_fullsized":"2024-04-16T21:08:29.502335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:30.634191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501937+0000","last_change":"2024-04-16T19:25:41.502982+0000","last_active":"2024-04-16T21:08:24.501937+0000","last_peered":"2024-04-16T21:08:24.501937+0000","last_clean":"2024-04-16T21:08:24.501937+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:08:24.501937+0000","last_undegraded":"2024-04-16T21:08:24.501937+0000","last_fullsized":"2024-04-16T21:08:24.501937+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:12.630377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"47'10","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656132+0000","last_change":"2024-04-16T19:25:15.481921+0000","last_active":"2024-04-16T21:07:59.656132+0000","last_peered":"2024-04-16T21:07:59.656132+0000","last_clean":"2024-04-16T21:07:59.656132+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:07:59.656132+0000","last_undegraded":"2024-04-16T21:07:59.656132+0000","last_fullsized":"2024-04-16T21:07:59.656132+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644518+0000","last_change":"2024-04-16T19:25:13.473915+0000","last_active":"2024-04-16T21:06:44.644518+0000","last_peered":"2024-04-16T21:06:44.644518+0000","last_clean":"2024-04-16T21:06:44.644518+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:06:44.644518+0000","last_undegraded":"2024-04-16T21:06:44.644518+0000","last_fullsized":"2024-04-16T21:06:44.644518+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:05.770561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'230","reported_seq":647,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644717+0000","last_change":"2024-04-16T19:25:11.461723+0000","last_active":"2024-04-16T21:06:44.644717+0000","last_peered":"2024-04-16T21:06:44.644717+0000","last_clean":"2024-04-16T21:06:44.644717+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:06:44.644717+0000","last_undegraded":"2024-04-16T21:06:44.644717+0000","last_fullsized":"2024-04-16T21:06:44.644717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:15.327068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'6","reported_seq":174,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643461+0000","last_change":"2024-04-16T19:25:09.458030+0000","last_active":"2024-04-16T21:06:39.643461+0000","last_peered":"2024-04-16T21:06:39.643461+0000","last_clean":"2024-04-16T21:06:39.643461+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:06:39.643461+0000","last_undegraded":"2024-04-16T21:06:39.643461+0000","last_fullsized":"2024-04-16T21:06:39.643461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663306+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:08:34.663306+0000","last_peered":"2024-04-16T21:08:34.663306+0000","last_clean":"2024-04-16T21:08:34.663306+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:08:34.663306+0000","last_undegraded":"2024-04-16T21:08:34.663306+0000","last_fullsized":"2024-04-16T21:08:34.663306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369834+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:08:44.369834+0000","last_peered":"2024-04-16T21:08:44.369834+0000","last_clean":"2024-04-16T21:08:44.369834+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:08:44.369834+0000","last_undegraded":"2024-04-16T21:08:44.369834+0000","last_fullsized":"2024-04-16T21:08:44.369834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:41.742389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.484254+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:06:34.484254+0000","last_peered":"2024-04-16T21:06:34.484254+0000","last_clean":"2024-04-16T21:06:34.484254+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:06:34.484254+0000","last_undegraded":"2024-04-16T21:06:34.484254+0000","last_fullsized":"2024-04-16T21:06:34.484254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:38.387958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":222,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650527+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:07:19.650527+0000","last_peered":"2024-04-16T21:07:19.650527+0000","last_clean":"2024-04-16T21:07:19.650527+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:07:19.650527+0000","last_undegraded":"2024-04-16T21:07:19.650527+0000","last_fullsized":"2024-04-16T21:07:19.650527+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:47.927883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649659+0000","last_change":"2024-04-16T19:24:49.432869+0000","last_active":"2024-04-16T21:07:19.649659+0000","last_peered":"2024-04-16T21:07:19.649659+0000","last_clean":"2024-04-16T21:07:19.649659+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:07:19.649659+0000","last_undegraded":"2024-04-16T21:07:19.649659+0000","last_fullsized":"2024-04-16T21:07:19.649659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655242+0000","last_change":"2024-04-16T19:24:51.442598+0000","last_active":"2024-04-16T21:07:49.655242+0000","last_peered":"2024-04-16T21:07:49.655242+0000","last_clean":"2024-04-16T21:07:49.655242+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:07:49.655242+0000","last_undegraded":"2024-04-16T21:07:49.655242+0000","last_fullsized":"2024-04-16T21:07:49.655242+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:52.222795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'12","reported_seq":186,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504441+0000","last_change":"2024-04-16T19:25:09.464004+0000","last_active":"2024-04-16T21:08:39.504441+0000","last_peered":"2024-04-16T21:08:39.504441+0000","last_clean":"2024-04-16T21:08:39.504441+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:08:39.504441+0000","last_undegraded":"2024-04-16T21:08:39.504441+0000","last_fullsized":"2024-04-16T21:08:39.504441+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504683+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:08:39.504683+0000","last_peered":"2024-04-16T21:08:39.504683+0000","last_clean":"2024-04-16T21:08:39.504683+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:08:39.504683+0000","last_undegraded":"2024-04-16T21:08:39.504683+0000","last_fullsized":"2024-04-16T21:08:39.504683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364532+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:08:44.364532+0000","last_peered":"2024-04-16T21:08:44.364532+0000","last_clean":"2024-04-16T21:08:44.364532+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:08:44.364532+0000","last_undegraded":"2024-04-16T21:08:44.364532+0000","last_fullsized":"2024-04-16T21:08:44.364532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'12","reported_seq":242,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663758+0000","last_change":"2024-04-16T19:25:09.466486+0000","last_active":"2024-04-16T21:08:39.663758+0000","last_peered":"2024-04-16T21:08:39.663758+0000","last_clean":"2024-04-16T21:08:39.663758+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:08:39.663758+0000","last_undegraded":"2024-04-16T21:08:39.663758+0000","last_fullsized":"2024-04-16T21:08:39.663758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504632+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:08:39.504632+0000","last_peered":"2024-04-16T21:08:39.504632+0000","last_clean":"2024-04-16T21:08:39.504632+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:08:39.504632+0000","last_undegraded":"2024-04-16T21:08:39.504632+0000","last_fullsized":"2024-04-16T21:08:39.504632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:49.140683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505241+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:08:44.505241+0000","last_peered":"2024-04-16T21:08:44.505241+0000","last_clean":"2024-04-16T21:08:44.505241+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:08:44.505241+0000","last_undegraded":"2024-04-16T21:08:44.505241+0000","last_fullsized":"2024-04-16T21:08:44.505241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:03:25.186300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'5","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649520+0000","last_change":"2024-04-16T19:25:09.467044+0000","last_active":"2024-04-16T21:07:14.649520+0000","last_peered":"2024-04-16T21:07:14.649520+0000","last_clean":"2024-04-16T21:07:14.649520+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:07:14.649520+0000","last_undegraded":"2024-04-16T21:07:14.649520+0000","last_fullsized":"2024-04-16T21:07:14.649520+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663818+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:08:39.663818+0000","last_peered":"2024-04-16T21:08:39.663818+0000","last_clean":"2024-04-16T21:08:39.663818+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:08:39.663818+0000","last_undegraded":"2024-04-16T21:08:39.663818+0000","last_fullsized":"2024-04-16T21:08:39.663818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365701+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:08:44.365701+0000","last_peered":"2024-04-16T21:08:44.365701+0000","last_clean":"2024-04-16T21:08:44.365701+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:08:44.365701+0000","last_undegraded":"2024-04-16T21:08:44.365701+0000","last_fullsized":"2024-04-16T21:08:44.365701+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'7","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664206+0000","last_change":"2024-04-16T19:25:09.458671+0000","last_active":"2024-04-16T21:08:39.664206+0000","last_peered":"2024-04-16T21:08:39.664206+0000","last_clean":"2024-04-16T21:08:39.664206+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:08:39.664206+0000","last_undegraded":"2024-04-16T21:08:39.664206+0000","last_fullsized":"2024-04-16T21:08:39.664206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:52.162049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504531+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:08:39.504531+0000","last_peered":"2024-04-16T21:08:39.504531+0000","last_clean":"2024-04-16T21:08:39.504531+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:08:39.504531+0000","last_undegraded":"2024-04-16T21:08:39.504531+0000","last_fullsized":"2024-04-16T21:08:39.504531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:22.916044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373116+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:09:04.373116+0000","last_peered":"2024-04-16T21:09:04.373116+0000","last_clean":"2024-04-16T21:09:04.373116+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:09:04.373116+0000","last_undegraded":"2024-04-16T21:09:04.373116+0000","last_fullsized":"2024-04-16T21:09:04.373116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'11","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655456+0000","last_change":"2024-04-16T19:25:09.466470+0000","last_active":"2024-04-16T21:07:49.655456+0000","last_peered":"2024-04-16T21:07:49.655456+0000","last_clean":"2024-04-16T21:07:49.655456+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:07:49.655456+0000","last_undegraded":"2024-04-16T21:07:49.655456+0000","last_fullsized":"2024-04-16T21:07:49.655456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508457+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:09:04.508457+0000","last_peered":"2024-04-16T21:09:04.508457+0000","last_clean":"2024-04-16T21:09:04.508457+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:09:04.508457+0000","last_undegraded":"2024-04-16T21:09:04.508457+0000","last_fullsized":"2024-04-16T21:09:04.508457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:58.689460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503608+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:08:34.503608+0000","last_peered":"2024-04-16T21:08:34.503608+0000","last_clean":"2024-04-16T21:08:34.503608+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:08:34.503608+0000","last_undegraded":"2024-04-16T21:08:34.503608+0000","last_fullsized":"2024-04-16T21:08:34.503608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:49.978039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'20","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496627+0000","last_change":"2024-04-16T19:25:09.460695+0000","last_active":"2024-04-16T21:07:49.496627+0000","last_peered":"2024-04-16T21:07:49.496627+0000","last_clean":"2024-04-16T21:07:49.496627+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:07:49.496627+0000","last_undegraded":"2024-04-16T21:07:49.496627+0000","last_fullsized":"2024-04-16T21:07:49.496627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:28.879723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365419+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:08:49.365419+0000","last_peered":"2024-04-16T21:08:49.365419+0000","last_clean":"2024-04-16T21:08:49.365419+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:08:49.365419+0000","last_undegraded":"2024-04-16T21:08:49.365419+0000","last_fullsized":"2024-04-16T21:08:49.365419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:16.805347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503638+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:08:34.503638+0000","last_peered":"2024-04-16T21:08:34.503638+0000","last_clean":"2024-04-16T21:08:34.503638+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:08:34.503638+0000","last_undegraded":"2024-04-16T21:08:34.503638+0000","last_fullsized":"2024-04-16T21:08:34.503638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'11","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504570+0000","last_change":"2024-04-16T19:25:09.459969+0000","last_active":"2024-04-16T21:08:39.504570+0000","last_peered":"2024-04-16T21:08:39.504570+0000","last_clean":"2024-04-16T21:08:39.504570+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:08:39.504570+0000","last_undegraded":"2024-04-16T21:08:39.504570+0000","last_fullsized":"2024-04-16T21:08:39.504570+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364844+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:08:49.364844+0000","last_peered":"2024-04-16T21:08:49.364844+0000","last_clean":"2024-04-16T21:08:49.364844+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:08:49.364844+0000","last_undegraded":"2024-04-16T21:08:49.364844+0000","last_fullsized":"2024-04-16T21:08:49.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663475+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:08:34.663475+0000","last_peered":"2024-04-16T21:08:34.663475+0000","last_clean":"2024-04-16T21:08:34.663475+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:08:34.663475+0000","last_undegraded":"2024-04-16T21:08:34.663475+0000","last_fullsized":"2024-04-16T21:08:34.663475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:40.025589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'13","reported_seq":290,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.506803+0000","last_change":"2024-04-16T19:25:09.459396+0000","last_active":"2024-04-16T21:08:54.506803+0000","last_peered":"2024-04-16T21:08:54.506803+0000","last_clean":"2024-04-16T21:08:54.506803+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:08:54.506803+0000","last_undegraded":"2024-04-16T21:08:54.506803+0000","last_fullsized":"2024-04-16T21:08:54.506803+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:44.854219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366364+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:08:49.366364+0000","last_peered":"2024-04-16T21:08:49.366364+0000","last_clean":"2024-04-16T21:08:49.366364+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:08:49.366364+0000","last_undegraded":"2024-04-16T21:08:49.366364+0000","last_fullsized":"2024-04-16T21:08:49.366364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503606+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:08:34.503606+0000","last_peered":"2024-04-16T21:08:34.503606+0000","last_clean":"2024-04-16T21:08:34.503606+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:08:34.503606+0000","last_undegraded":"2024-04-16T21:08:34.503606+0000","last_fullsized":"2024-04-16T21:08:34.503606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:56:59.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'38","reported_seq":309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650208+0000","last_change":"2024-04-16T19:25:09.462718+0000","last_active":"2024-04-16T21:07:14.650208+0000","last_peered":"2024-04-16T21:07:14.650208+0000","last_clean":"2024-04-16T21:07:14.650208+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:07:14.650208+0000","last_undegraded":"2024-04-16T21:07:14.650208+0000","last_fullsized":"2024-04-16T21:07:14.650208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:51.473781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649801+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:07:19.649801+0000","last_peered":"2024-04-16T21:07:19.649801+0000","last_clean":"2024-04-16T21:07:19.649801+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:07:19.649801+0000","last_undegraded":"2024-04-16T21:07:19.649801+0000","last_fullsized":"2024-04-16T21:07:19.649801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366306+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:08:54.366306+0000","last_peered":"2024-04-16T21:08:54.366306+0000","last_clean":"2024-04-16T21:08:54.366306+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:08:54.366306+0000","last_undegraded":"2024-04-16T21:08:54.366306+0000","last_fullsized":"2024-04-16T21:08:54.366306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'14","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493167+0000","last_change":"2024-04-16T19:25:09.467165+0000","last_active":"2024-04-16T21:07:29.493167+0000","last_peered":"2024-04-16T21:07:29.493167+0000","last_clean":"2024-04-16T21:07:29.493167+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:07:29.493167+0000","last_undegraded":"2024-04-16T21:07:29.493167+0000","last_fullsized":"2024-04-16T21:07:29.493167+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:19.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650655+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:07:19.650655+0000","last_peered":"2024-04-16T21:07:19.650655+0000","last_clean":"2024-04-16T21:07:19.650655+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:07:19.650655+0000","last_undegraded":"2024-04-16T21:07:19.650655+0000","last_fullsized":"2024-04-16T21:07:19.650655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657012+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:08:04.657012+0000","last_peered":"2024-04-16T21:08:04.657012+0000","last_clean":"2024-04-16T21:08:04.657012+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:08:04.657012+0000","last_undegraded":"2024-04-16T21:08:04.657012+0000","last_fullsized":"2024-04-16T21:08:04.657012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'9","reported_seq":179,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648706+0000","last_change":"2024-04-16T19:25:09.464008+0000","last_active":"2024-04-16T21:07:14.648706+0000","last_peered":"2024-04-16T21:07:14.648706+0000","last_clean":"2024-04-16T21:07:14.648706+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:07:14.648706+0000","last_undegraded":"2024-04-16T21:07:14.648706+0000","last_fullsized":"2024-04-16T21:07:14.648706+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501544+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:08:24.501544+0000","last_peered":"2024-04-16T21:08:24.501544+0000","last_clean":"2024-04-16T21:08:24.501544+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:08:24.501544+0000","last_undegraded":"2024-04-16T21:08:24.501544+0000","last_fullsized":"2024-04-16T21:08:24.501544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657935+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:08:04.657935+0000","last_peered":"2024-04-16T21:08:04.657935+0000","last_clean":"2024-04-16T21:08:04.657935+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:08:04.657935+0000","last_undegraded":"2024-04-16T21:08:04.657935+0000","last_fullsized":"2024-04-16T21:08:04.657935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'22","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490768+0000","last_change":"2024-04-16T19:25:09.467151+0000","last_active":"2024-04-16T21:07:14.490768+0000","last_peered":"2024-04-16T21:07:14.490768+0000","last_clean":"2024-04-16T21:07:14.490768+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:07:14.490768+0000","last_undegraded":"2024-04-16T21:07:14.490768+0000","last_fullsized":"2024-04-16T21:07:14.490768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:22.630747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659703+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:08:09.659703+0000","last_peered":"2024-04-16T21:08:09.659703+0000","last_clean":"2024-04-16T21:08:09.659703+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:08:09.659703+0000","last_undegraded":"2024-04-16T21:08:09.659703+0000","last_fullsized":"2024-04-16T21:08:09.659703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488264+0000","last_change":"2024-04-16T19:25:41.505958+0000","last_active":"2024-04-16T21:06:59.488264+0000","last_peered":"2024-04-16T21:06:59.488264+0000","last_clean":"2024-04-16T21:06:59.488264+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:06:59.488264+0000","last_undegraded":"2024-04-16T21:06:59.488264+0000","last_fullsized":"2024-04-16T21:06:59.488264+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:19.162890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'5","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369023+0000","last_change":"2024-04-16T19:25:09.467605+0000","last_active":"2024-04-16T21:09:09.369023+0000","last_peered":"2024-04-16T21:09:09.369023+0000","last_clean":"2024-04-16T21:09:09.369023+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:09:09.369023+0000","last_undegraded":"2024-04-16T21:09:09.369023+0000","last_fullsized":"2024-04-16T21:09:09.369023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658742+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:08:09.658742+0000","last_peered":"2024-04-16T21:08:09.658742+0000","last_clean":"2024-04-16T21:08:09.658742+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:08:09.658742+0000","last_undegraded":"2024-04-16T21:08:09.658742+0000","last_fullsized":"2024-04-16T21:08:09.658742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:24.447630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658987+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:08:04.658987+0000","last_peered":"2024-04-16T21:08:04.658987+0000","last_clean":"2024-04-16T21:08:04.658987+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:08:04.658987+0000","last_undegraded":"2024-04-16T21:08:04.658987+0000","last_fullsized":"2024-04-16T21:08:04.658987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'14","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509341+0000","last_change":"2024-04-16T19:25:09.464438+0000","last_active":"2024-04-16T21:09:09.509341+0000","last_peered":"2024-04-16T21:09:09.509341+0000","last_clean":"2024-04-16T21:09:09.509341+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:09:09.509341+0000","last_undegraded":"2024-04-16T21:09:09.509341+0000","last_fullsized":"2024-04-16T21:09:09.509341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657776+0000","last_change":"2024-04-16T19:25:41.508398+0000","last_active":"2024-04-16T21:08:09.657776+0000","last_peered":"2024-04-16T21:08:09.657776+0000","last_clean":"2024-04-16T21:08:09.657776+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:08:09.657776+0000","last_undegraded":"2024-04-16T21:08:09.657776+0000","last_fullsized":"2024-04-16T21:08:09.657776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:28:30.770132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661196+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:08:19.661196+0000","last_peered":"2024-04-16T21:08:19.661196+0000","last_clean":"2024-04-16T21:08:19.661196+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:08:19.661196+0000","last_undegraded":"2024-04-16T21:08:19.661196+0000","last_fullsized":"2024-04-16T21:08:19.661196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:37.636086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'3","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652749+0000","last_change":"2024-04-16T19:25:41.499612+0000","last_active":"2024-04-16T21:07:34.652749+0000","last_peered":"2024-04-16T21:07:34.652749+0000","last_clean":"2024-04-16T21:07:34.652749+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:07:34.652749+0000","last_undegraded":"2024-04-16T21:07:34.652749+0000","last_fullsized":"2024-04-16T21:07:34.652749+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499546+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:08:09.499546+0000","last_peered":"2024-04-16T21:08:09.499546+0000","last_clean":"2024-04-16T21:08:09.499546+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:08:09.499546+0000","last_undegraded":"2024-04-16T21:08:09.499546+0000","last_fullsized":"2024-04-16T21:08:09.499546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490904+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:07:14.490904+0000","last_peered":"2024-04-16T21:07:14.490904+0000","last_clean":"2024-04-16T21:07:14.490904+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:07:14.490904+0000","last_undegraded":"2024-04-16T21:07:14.490904+0000","last_fullsized":"2024-04-16T21:07:14.490904+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:50.557311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'19","reported_seq":286,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373955+0000","last_change":"2024-04-16T19:25:09.468675+0000","last_active":"2024-04-16T21:09:09.373955+0000","last_peered":"2024-04-16T21:09:09.373955+0000","last_clean":"2024-04-16T21:09:09.373955+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:09:09.373955+0000","last_undegraded":"2024-04-16T21:09:09.373955+0000","last_fullsized":"2024-04-16T21:09:09.373955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:21:12.600213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367898+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:08:59.367898+0000","last_peered":"2024-04-16T21:08:59.367898+0000","last_clean":"2024-04-16T21:08:59.367898+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:08:59.367898+0000","last_undegraded":"2024-04-16T21:08:59.367898+0000","last_fullsized":"2024-04-16T21:08:59.367898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649751+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:07:14.649751+0000","last_peered":"2024-04-16T21:07:14.649751+0000","last_clean":"2024-04-16T21:07:14.649751+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:07:14.649751+0000","last_undegraded":"2024-04-16T21:07:14.649751+0000","last_fullsized":"2024-04-16T21:07:14.649751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:50.981157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'12","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652830+0000","last_change":"2024-04-16T19:25:09.459052+0000","last_active":"2024-04-16T21:07:39.652830+0000","last_peered":"2024-04-16T21:07:39.652830+0000","last_clean":"2024-04-16T21:07:39.652830+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:07:39.652830+0000","last_undegraded":"2024-04-16T21:07:39.652830+0000","last_fullsized":"2024-04-16T21:07:39.652830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:51.756021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650094+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:07:14.650094+0000","last_peered":"2024-04-16T21:07:14.650094+0000","last_clean":"2024-04-16T21:07:14.650094+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:07:14.650094+0000","last_undegraded":"2024-04-16T21:07:14.650094+0000","last_fullsized":"2024-04-16T21:07:14.650094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371494+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:08:54.371494+0000","last_peered":"2024-04-16T21:08:54.371494+0000","last_clean":"2024-04-16T21:08:54.371494+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:08:54.371494+0000","last_undegraded":"2024-04-16T21:08:54.371494+0000","last_fullsized":"2024-04-16T21:08:54.371494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'3","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494798+0000","last_change":"2024-04-16T19:25:09.460170+0000","last_active":"2024-04-16T21:07:39.494798+0000","last_peered":"2024-04-16T21:07:39.494798+0000","last_clean":"2024-04-16T21:07:39.494798+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:07:39.494798+0000","last_undegraded":"2024-04-16T21:07:39.494798+0000","last_fullsized":"2024-04-16T21:07:39.494798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490719+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:07:14.490719+0000","last_peered":"2024-04-16T21:07:14.490719+0000","last_clean":"2024-04-16T21:07:14.490719+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:07:14.490719+0000","last_undegraded":"2024-04-16T21:07:14.490719+0000","last_fullsized":"2024-04-16T21:07:14.490719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:08:34.663111+0000","last_peered":"2024-04-16T21:08:34.663111+0000","last_clean":"2024-04-16T21:08:34.663111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:08:34.663111+0000","last_undegraded":"2024-04-16T21:08:34.663111+0000","last_fullsized":"2024-04-16T21:08:34.663111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4203,"ondisk_log_size":4203,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3420,"ondisk_log_size":3420,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5660,"ondisk_log_size":5660,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641550,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41132,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816468,"statfs":{"total":107374182400,"available":107332063232,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641551,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641550,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55568,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802032,"statfs":{"total":107374182400,"available":107317280768,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641551,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59228,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798372,"statfs":{"total":107374182400,"available":107313532928,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641551,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816412,"statfs":{"total":107374182400,"available":107332005888,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641550,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55800,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104801800,"statfs":{"total":107374182400,"available":107317043200,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:11:22.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:11:22.889 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:11:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:11:22.910 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3274,"stamp":"2024-04-16T21:11:21.319203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30466,"ondisk_log_size":30466,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":307840,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":296630,"kb_avail":628837760,"statfs":{"total":644245094400,"available":643929866240,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":303749793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005593"},"pg_stats":[{"pgid":"9.35","version":"52'28","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.485159+0000","last_change":"2024-04-16T19:25:09.466020+0000","last_active":"2024-04-16T21:06:39.485159+0000","last_peered":"2024-04-16T21:06:39.485159+0000","last_clean":"2024-04-16T21:06:39.485159+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:06:39.485159+0000","last_undegraded":"2024-04-16T21:06:39.485159+0000","last_fullsized":"2024-04-16T21:06:39.485159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365868+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:08:44.365868+0000","last_peered":"2024-04-16T21:08:44.365868+0000","last_clean":"2024-04-16T21:08:44.365868+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:08:44.365868+0000","last_undegraded":"2024-04-16T21:08:44.365868+0000","last_fullsized":"2024-04-16T21:08:44.365868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648735+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:07:09.648735+0000","last_peered":"2024-04-16T21:07:09.648735+0000","last_clean":"2024-04-16T21:07:09.648735+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:07:09.648735+0000","last_undegraded":"2024-04-16T21:07:09.648735+0000","last_fullsized":"2024-04-16T21:07:09.648735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"47'5","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643488+0000","last_change":"2024-04-16T19:25:09.467691+0000","last_active":"2024-04-16T21:06:39.643488+0000","last_peered":"2024-04-16T21:06:39.643488+0000","last_clean":"2024-04-16T21:06:39.643488+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:06:39.643488+0000","last_undegraded":"2024-04-16T21:06:39.643488+0000","last_fullsized":"2024-04-16T21:06:39.643488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364114+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:08:44.364114+0000","last_peered":"2024-04-16T21:08:44.364114+0000","last_clean":"2024-04-16T21:08:44.364114+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:08:44.364114+0000","last_undegraded":"2024-04-16T21:08:44.364114+0000","last_fullsized":"2024-04-16T21:08:44.364114+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649318+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:07:09.649318+0000","last_peered":"2024-04-16T21:07:09.649318+0000","last_clean":"2024-04-16T21:07:09.649318+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:07:09.649318+0000","last_undegraded":"2024-04-16T21:07:09.649318+0000","last_fullsized":"2024-04-16T21:07:09.649318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'13","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646812+0000","last_change":"2024-04-16T19:25:09.457429+0000","last_active":"2024-04-16T21:06:54.646812+0000","last_peered":"2024-04-16T21:06:54.646812+0000","last_clean":"2024-04-16T21:06:54.646812+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:06:54.646812+0000","last_undegraded":"2024-04-16T21:06:54.646812+0000","last_fullsized":"2024-04-16T21:06:54.646812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369756+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:08:44.369756+0000","last_peered":"2024-04-16T21:08:44.369756+0000","last_clean":"2024-04-16T21:08:44.369756+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:08:44.369756+0000","last_undegraded":"2024-04-16T21:08:44.369756+0000","last_fullsized":"2024-04-16T21:08:44.369756+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:11.722618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649006+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:07:09.649006+0000","last_peered":"2024-04-16T21:07:09.649006+0000","last_clean":"2024-04-16T21:07:09.649006+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:07:09.649006+0000","last_undegraded":"2024-04-16T21:07:09.649006+0000","last_fullsized":"2024-04-16T21:07:09.649006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'13","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.485201+0000","last_change":"2024-04-16T19:25:09.467050+0000","last_active":"2024-04-16T21:06:39.485201+0000","last_peered":"2024-04-16T21:06:39.485201+0000","last_clean":"2024-04-16T21:06:39.485201+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:06:39.485201+0000","last_undegraded":"2024-04-16T21:06:39.485201+0000","last_fullsized":"2024-04-16T21:06:39.485201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366441+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:08:54.366441+0000","last_peered":"2024-04-16T21:08:54.366441+0000","last_clean":"2024-04-16T21:08:54.366441+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:08:54.366441+0000","last_undegraded":"2024-04-16T21:08:54.366441+0000","last_fullsized":"2024-04-16T21:08:54.366441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648038+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:07:09.648038+0000","last_peered":"2024-04-16T21:07:09.648038+0000","last_clean":"2024-04-16T21:07:09.648038+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:07:09.648038+0000","last_undegraded":"2024-04-16T21:07:09.648038+0000","last_fullsized":"2024-04-16T21:07:09.648038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'6","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655139+0000","last_change":"2024-04-16T19:25:09.459828+0000","last_active":"2024-04-16T21:07:49.655139+0000","last_peered":"2024-04-16T21:07:49.655139+0000","last_clean":"2024-04-16T21:07:49.655139+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:07:49.655139+0000","last_undegraded":"2024-04-16T21:07:49.655139+0000","last_fullsized":"2024-04-16T21:07:49.655139+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648499+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:07:04.648499+0000","last_peered":"2024-04-16T21:07:04.648499+0000","last_clean":"2024-04-16T21:07:04.648499+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:07:04.648499+0000","last_undegraded":"2024-04-16T21:07:04.648499+0000","last_fullsized":"2024-04-16T21:07:04.648499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367192+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:08:59.367192+0000","last_peered":"2024-04-16T21:08:59.367192+0000","last_clean":"2024-04-16T21:08:59.367192+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:08:59.367192+0000","last_undegraded":"2024-04-16T21:08:59.367192+0000","last_fullsized":"2024-04-16T21:08:59.367192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'18","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654366+0000","last_change":"2024-04-16T19:25:09.461822+0000","last_active":"2024-04-16T21:07:49.654366+0000","last_peered":"2024-04-16T21:07:49.654366+0000","last_clean":"2024-04-16T21:07:49.654366+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:07:49.654366+0000","last_undegraded":"2024-04-16T21:07:49.654366+0000","last_fullsized":"2024-04-16T21:07:49.654366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647976+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:07:04.647976+0000","last_peered":"2024-04-16T21:07:04.647976+0000","last_clean":"2024-04-16T21:07:04.647976+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:07:04.647976+0000","last_undegraded":"2024-04-16T21:07:04.647976+0000","last_fullsized":"2024-04-16T21:07:04.647976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:07.917853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365518+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:08:49.365518+0000","last_peered":"2024-04-16T21:08:49.365518+0000","last_clean":"2024-04-16T21:08:49.365518+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:08:49.365518+0000","last_undegraded":"2024-04-16T21:08:49.365518+0000","last_fullsized":"2024-04-16T21:08:49.365518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649782+0000","last_change":"2024-04-16T19:25:41.508581+0000","last_active":"2024-04-16T21:07:14.649782+0000","last_peered":"2024-04-16T21:07:14.649782+0000","last_clean":"2024-04-16T21:07:14.649782+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:07:14.649782+0000","last_undegraded":"2024-04-16T21:07:14.649782+0000","last_fullsized":"2024-04-16T21:07:14.649782+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:12.396788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648409+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:07:04.648409+0000","last_peered":"2024-04-16T21:07:04.648409+0000","last_clean":"2024-04-16T21:07:04.648409+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:07:04.648409+0000","last_undegraded":"2024-04-16T21:07:04.648409+0000","last_fullsized":"2024-04-16T21:07:04.648409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:34.027154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365507+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:08:49.365507+0000","last_peered":"2024-04-16T21:08:49.365507+0000","last_clean":"2024-04-16T21:08:49.365507+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:08:49.365507+0000","last_undegraded":"2024-04-16T21:08:49.365507+0000","last_fullsized":"2024-04-16T21:08:49.365507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'15","reported_seq":238,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657926+0000","last_change":"2024-04-16T19:25:09.463558+0000","last_active":"2024-04-16T21:08:04.657926+0000","last_peered":"2024-04-16T21:08:04.657926+0000","last_clean":"2024-04-16T21:08:04.657926+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:08:04.657926+0000","last_undegraded":"2024-04-16T21:08:04.657926+0000","last_fullsized":"2024-04-16T21:08:04.657926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489018+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:07:04.489018+0000","last_peered":"2024-04-16T21:07:04.489018+0000","last_clean":"2024-04-16T21:07:04.489018+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:07:04.489018+0000","last_undegraded":"2024-04-16T21:07:04.489018+0000","last_fullsized":"2024-04-16T21:07:04.489018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506158+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:08:49.506158+0000","last_peered":"2024-04-16T21:08:49.506158+0000","last_clean":"2024-04-16T21:08:49.506158+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:08:49.506158+0000","last_undegraded":"2024-04-16T21:08:49.506158+0000","last_fullsized":"2024-04-16T21:08:49.506158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'7","reported_seq":220,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649667+0000","last_change":"2024-04-16T19:25:09.467094+0000","last_active":"2024-04-16T21:07:14.649667+0000","last_peered":"2024-04-16T21:07:14.649667+0000","last_clean":"2024-04-16T21:07:14.649667+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:07:14.649667+0000","last_undegraded":"2024-04-16T21:07:14.649667+0000","last_fullsized":"2024-04-16T21:07:14.649667+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:52.158151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650393+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:07:19.650393+0000","last_peered":"2024-04-16T21:07:19.650393+0000","last_clean":"2024-04-16T21:07:19.650393+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:07:19.650393+0000","last_undegraded":"2024-04-16T21:07:19.650393+0000","last_fullsized":"2024-04-16T21:07:19.650393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366531+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:08:49.366531+0000","last_peered":"2024-04-16T21:08:49.366531+0000","last_clean":"2024-04-16T21:08:49.366531+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:08:49.366531+0000","last_undegraded":"2024-04-16T21:08:49.366531+0000","last_fullsized":"2024-04-16T21:08:49.366531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'6","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651913+0000","last_change":"2024-04-16T19:25:09.459483+0000","last_active":"2024-04-16T21:07:24.651913+0000","last_peered":"2024-04-16T21:07:24.651913+0000","last_clean":"2024-04-16T21:07:24.651913+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:07:24.651913+0000","last_undegraded":"2024-04-16T21:07:24.651913+0000","last_fullsized":"2024-04-16T21:07:24.651913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650394+0000","last_change":"2024-04-16T19:25:41.503681+0000","last_active":"2024-04-16T21:07:19.650394+0000","last_peered":"2024-04-16T21:07:19.650394+0000","last_clean":"2024-04-16T21:07:19.650394+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:07:19.650394+0000","last_undegraded":"2024-04-16T21:07:19.650394+0000","last_fullsized":"2024-04-16T21:07:19.650394+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:33:58.945193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662645+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:08:29.662645+0000","last_peered":"2024-04-16T21:08:29.662645+0000","last_clean":"2024-04-16T21:08:29.662645+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:08:29.662645+0000","last_undegraded":"2024-04-16T21:08:29.662645+0000","last_fullsized":"2024-04-16T21:08:29.662645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'5","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490935+0000","last_change":"2024-04-16T19:25:09.464787+0000","last_active":"2024-04-16T21:07:14.490935+0000","last_peered":"2024-04-16T21:07:14.490935+0000","last_clean":"2024-04-16T21:07:14.490935+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:07:14.490935+0000","last_undegraded":"2024-04-16T21:07:14.490935+0000","last_fullsized":"2024-04-16T21:07:14.490935+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652786+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:07:34.652786+0000","last_peered":"2024-04-16T21:07:34.652786+0000","last_clean":"2024-04-16T21:07:34.652786+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:07:34.652786+0000","last_undegraded":"2024-04-16T21:07:34.652786+0000","last_fullsized":"2024-04-16T21:07:34.652786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662148+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:08:29.662148+0000","last_peered":"2024-04-16T21:08:29.662148+0000","last_clean":"2024-04-16T21:08:29.662148+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:08:29.662148+0000","last_undegraded":"2024-04-16T21:08:29.662148+0000","last_fullsized":"2024-04-16T21:08:29.662148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490887+0000","last_change":"2024-04-16T19:25:09.463733+0000","last_active":"2024-04-16T21:07:14.490887+0000","last_peered":"2024-04-16T21:07:14.490887+0000","last_clean":"2024-04-16T21:07:14.490887+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:07:14.490887+0000","last_undegraded":"2024-04-16T21:07:14.490887+0000","last_fullsized":"2024-04-16T21:07:14.490887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:40:59.948853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503495+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:08:34.503495+0000","last_peered":"2024-04-16T21:08:34.503495+0000","last_clean":"2024-04-16T21:08:34.503495+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:08:34.503495+0000","last_undegraded":"2024-04-16T21:08:34.503495+0000","last_fullsized":"2024-04-16T21:08:34.503495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:23.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661720+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:08:29.661720+0000","last_peered":"2024-04-16T21:08:29.661720+0000","last_clean":"2024-04-16T21:08:29.661720+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:08:29.661720+0000","last_undegraded":"2024-04-16T21:08:29.661720+0000","last_fullsized":"2024-04-16T21:08:29.661720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:09:50.552696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'9","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654268+0000","last_change":"2024-04-16T19:25:09.462286+0000","last_active":"2024-04-16T21:07:39.654268+0000","last_peered":"2024-04-16T21:07:39.654268+0000","last_clean":"2024-04-16T21:07:39.654268+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:07:39.654268+0000","last_undegraded":"2024-04-16T21:07:39.654268+0000","last_fullsized":"2024-04-16T21:07:39.654268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503526+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:08:34.503526+0000","last_peered":"2024-04-16T21:08:34.503526+0000","last_clean":"2024-04-16T21:08:34.503526+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:08:34.503526+0000","last_undegraded":"2024-04-16T21:08:34.503526+0000","last_fullsized":"2024-04-16T21:08:34.503526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661148+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:08:29.661148+0000","last_peered":"2024-04-16T21:08:29.661148+0000","last_clean":"2024-04-16T21:08:29.661148+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:08:29.661148+0000","last_undegraded":"2024-04-16T21:08:29.661148+0000","last_fullsized":"2024-04-16T21:08:29.661148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'15","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494683+0000","last_change":"2024-04-16T19:25:09.461082+0000","last_active":"2024-04-16T21:07:39.494683+0000","last_peered":"2024-04-16T21:07:39.494683+0000","last_clean":"2024-04-16T21:07:39.494683+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:07:39.494683+0000","last_undegraded":"2024-04-16T21:07:39.494683+0000","last_fullsized":"2024-04-16T21:07:39.494683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662702+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:08:34.662702+0000","last_peered":"2024-04-16T21:08:34.662702+0000","last_clean":"2024-04-16T21:08:34.662702+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:08:34.662702+0000","last_undegraded":"2024-04-16T21:08:34.662702+0000","last_fullsized":"2024-04-16T21:08:34.662702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651169+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:07:29.651169+0000","last_peered":"2024-04-16T21:07:29.651169+0000","last_clean":"2024-04-16T21:07:29.651169+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:07:29.651169+0000","last_undegraded":"2024-04-16T21:07:29.651169+0000","last_fullsized":"2024-04-16T21:07:29.651169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365000+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:08:49.365000+0000","last_peered":"2024-04-16T21:08:49.365000+0000","last_clean":"2024-04-16T21:08:49.365000+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:08:49.365000+0000","last_undegraded":"2024-04-16T21:08:49.365000+0000","last_fullsized":"2024-04-16T21:08:49.365000+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:32.817843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1720,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659053+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:08:04.659053+0000","last_peered":"2024-04-16T21:08:04.659053+0000","last_clean":"2024-04-16T21:08:04.659053+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:08:04.659053+0000","last_undegraded":"2024-04-16T21:08:04.659053+0000","last_fullsized":"2024-04-16T21:08:04.659053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660478+0000","last_change":"2024-04-16T19:25:41.505822+0000","last_active":"2024-04-16T21:08:14.660478+0000","last_peered":"2024-04-16T21:08:14.660478+0000","last_clean":"2024-04-16T21:08:14.660478+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:08:14.660478+0000","last_undegraded":"2024-04-16T21:08:14.660478+0000","last_fullsized":"2024-04-16T21:08:14.660478+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'6","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654478+0000","last_change":"2024-04-16T19:25:15.486584+0000","last_active":"2024-04-16T21:07:49.654478+0000","last_peered":"2024-04-16T21:07:49.654478+0000","last_clean":"2024-04-16T21:07:49.654478+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:07:49.654478+0000","last_undegraded":"2024-04-16T21:07:49.654478+0000","last_fullsized":"2024-04-16T21:07:49.654478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654387+0000","last_change":"2024-04-16T19:25:13.469257+0000","last_active":"2024-04-16T21:07:44.654387+0000","last_peered":"2024-04-16T21:07:44.654387+0000","last_clean":"2024-04-16T21:07:44.654387+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:07:44.654387+0000","last_undegraded":"2024-04-16T21:07:44.654387+0000","last_fullsized":"2024-04-16T21:07:44.654387+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'203","reported_seq":447,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655143+0000","last_change":"2024-04-16T19:25:11.464182+0000","last_active":"2024-04-16T21:07:44.655143+0000","last_peered":"2024-04-16T21:07:44.655143+0000","last_clean":"2024-04-16T21:07:44.655143+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:07:44.655143+0000","last_undegraded":"2024-04-16T21:07:44.655143+0000","last_fullsized":"2024-04-16T21:07:44.655143+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:07.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'19","reported_seq":284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487393+0000","last_change":"2024-04-16T19:25:09.464946+0000","last_active":"2024-04-16T21:06:54.487393+0000","last_peered":"2024-04-16T21:06:54.487393+0000","last_clean":"2024-04-16T21:06:54.487393+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:06:54.487393+0000","last_undegraded":"2024-04-16T21:06:54.487393+0000","last_fullsized":"2024-04-16T21:06:54.487393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:45:45.734532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365360+0000","last_change":"2024-04-16T19:24:51.436483+0000","last_active":"2024-04-16T21:08:49.365360+0000","last_peered":"2024-04-16T21:08:49.365360+0000","last_clean":"2024-04-16T21:08:49.365360+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:08:49.365360+0000","last_undegraded":"2024-04-16T21:08:49.365360+0000","last_fullsized":"2024-04-16T21:08:49.365360+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:40.603125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494096+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:07:34.494096+0000","last_peered":"2024-04-16T21:07:34.494096+0000","last_clean":"2024-04-16T21:07:34.494096+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:07:34.494096+0000","last_undegraded":"2024-04-16T21:07:34.494096+0000","last_fullsized":"2024-04-16T21:07:34.494096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:52.988517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650257+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:07:14.650257+0000","last_peered":"2024-04-16T21:07:14.650257+0000","last_clean":"2024-04-16T21:07:14.650257+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:07:14.650257+0000","last_undegraded":"2024-04-16T21:07:14.650257+0000","last_fullsized":"2024-04-16T21:07:14.650257+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:07.194576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:42:58.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369810+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:08:44.369810+0000","last_peered":"2024-04-16T21:08:44.369810+0000","last_clean":"2024-04-16T21:08:44.369810+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:08:44.369810+0000","last_undegraded":"2024-04-16T21:08:44.369810+0000","last_fullsized":"2024-04-16T21:08:44.369810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:08:44.364641+0000","last_peered":"2024-04-16T21:08:44.364641+0000","last_clean":"2024-04-16T21:08:44.364641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:08:44.364641+0000","last_undegraded":"2024-04-16T21:08:44.364641+0000","last_fullsized":"2024-04-16T21:08:44.364641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:42:28.507077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496581+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:07:49.496581+0000","last_peered":"2024-04-16T21:07:49.496581+0000","last_clean":"2024-04-16T21:07:49.496581+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:07:49.496581+0000","last_undegraded":"2024-04-16T21:07:49.496581+0000","last_fullsized":"2024-04-16T21:07:49.496581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:43.119570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1836,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365530+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:08:49.365530+0000","last_peered":"2024-04-16T21:08:49.365530+0000","last_clean":"2024-04-16T21:08:49.365530+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:08:49.365530+0000","last_undegraded":"2024-04-16T21:08:49.365530+0000","last_fullsized":"2024-04-16T21:08:49.365530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659544+0000","last_change":"2024-04-16T19:25:41.503686+0000","last_active":"2024-04-16T21:08:14.659544+0000","last_peered":"2024-04-16T21:08:14.659544+0000","last_clean":"2024-04-16T21:08:14.659544+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:08:14.659544+0000","last_undegraded":"2024-04-16T21:08:14.659544+0000","last_fullsized":"2024-04-16T21:08:14.659544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:30:44.336557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'5","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660521+0000","last_change":"2024-04-16T19:25:41.505903+0000","last_active":"2024-04-16T21:08:14.660521+0000","last_peered":"2024-04-16T21:08:14.660521+0000","last_clean":"2024-04-16T21:08:14.660521+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:08:14.660521+0000","last_undegraded":"2024-04-16T21:08:14.660521+0000","last_fullsized":"2024-04-16T21:08:14.660521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:37.409520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654445+0000","last_change":"2024-04-16T19:25:13.471998+0000","last_active":"2024-04-16T21:07:44.654445+0000","last_peered":"2024-04-16T21:07:44.654445+0000","last_clean":"2024-04-16T21:07:44.654445+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:07:44.654445+0000","last_undegraded":"2024-04-16T21:07:44.654445+0000","last_fullsized":"2024-04-16T21:07:44.654445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:15.038784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'241","reported_seq":976,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495593+0000","last_change":"2024-04-16T19:25:11.465710+0000","last_active":"2024-04-16T21:07:44.495593+0000","last_peered":"2024-04-16T21:07:44.495593+0000","last_clean":"2024-04-16T21:07:44.495593+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:07:44.495593+0000","last_undegraded":"2024-04-16T21:07:44.495593+0000","last_fullsized":"2024-04-16T21:07:44.495593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:25.422152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'5","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643326+0000","last_change":"2024-04-16T19:25:09.467837+0000","last_active":"2024-04-16T21:06:39.643326+0000","last_peered":"2024-04-16T21:06:39.643326+0000","last_clean":"2024-04-16T21:06:39.643326+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:06:39.643326+0000","last_undegraded":"2024-04-16T21:06:39.643326+0000","last_fullsized":"2024-04-16T21:06:39.643326+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643388+0000","last_change":"2024-04-16T19:24:51.438136+0000","last_active":"2024-04-16T21:06:39.643388+0000","last_peered":"2024-04-16T21:06:39.643388+0000","last_clean":"2024-04-16T21:06:39.643388+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:06:39.643388+0000","last_undegraded":"2024-04-16T21:06:39.643388+0000","last_fullsized":"2024-04-16T21:06:39.643388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663065+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:08:34.663065+0000","last_peered":"2024-04-16T21:08:34.663065+0000","last_clean":"2024-04-16T21:08:34.663065+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:08:34.663065+0000","last_undegraded":"2024-04-16T21:08:34.663065+0000","last_fullsized":"2024-04-16T21:08:34.663065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:39.643446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648771+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:07:14.648771+0000","last_peered":"2024-04-16T21:07:14.648771+0000","last_clean":"2024-04-16T21:07:14.648771+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:07:14.648771+0000","last_undegraded":"2024-04-16T21:07:14.648771+0000","last_fullsized":"2024-04-16T21:07:14.648771+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:19.314716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653671+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:07:44.653671+0000","last_peered":"2024-04-16T21:07:44.653671+0000","last_clean":"2024-04-16T21:07:44.653671+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:07:44.653671+0000","last_undegraded":"2024-04-16T21:07:44.653671+0000","last_fullsized":"2024-04-16T21:07:44.653671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:26.398044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364812+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:08:44.364812+0000","last_peered":"2024-04-16T21:08:44.364812+0000","last_clean":"2024-04-16T21:08:44.364812+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:08:44.364812+0000","last_undegraded":"2024-04-16T21:08:44.364812+0000","last_fullsized":"2024-04-16T21:08:44.364812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:34.874297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364788+0000","last_change":"2024-04-16T19:24:49.426639+0000","last_active":"2024-04-16T21:08:44.364788+0000","last_peered":"2024-04-16T21:08:44.364788+0000","last_clean":"2024-04-16T21:08:44.364788+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:08:44.364788+0000","last_undegraded":"2024-04-16T21:08:44.364788+0000","last_fullsized":"2024-04-16T21:08:44.364788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:38.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652789+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:07:34.652789+0000","last_peered":"2024-04-16T21:07:34.652789+0000","last_clean":"2024-04-16T21:07:34.652789+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:07:34.652789+0000","last_undegraded":"2024-04-16T21:07:34.652789+0000","last_fullsized":"2024-04-16T21:07:34.652789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.732507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1871,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659473+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:08:19.659473+0000","last_peered":"2024-04-16T21:08:19.659473+0000","last_clean":"2024-04-16T21:08:19.659473+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:08:19.659473+0000","last_undegraded":"2024-04-16T21:08:19.659473+0000","last_fullsized":"2024-04-16T21:08:19.659473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:55.150921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655316+0000","last_change":"2024-04-16T19:25:41.502400+0000","last_active":"2024-04-16T21:07:49.655316+0000","last_peered":"2024-04-16T21:07:49.655316+0000","last_clean":"2024-04-16T21:07:49.655316+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:07:49.655316+0000","last_undegraded":"2024-04-16T21:07:49.655316+0000","last_fullsized":"2024-04-16T21:07:49.655316+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:02.633032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'17","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652839+0000","last_change":"2024-04-16T19:25:15.480449+0000","last_active":"2024-04-16T21:07:34.652839+0000","last_peered":"2024-04-16T21:07:34.652839+0000","last_clean":"2024-04-16T21:07:34.652839+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:07:34.652839+0000","last_undegraded":"2024-04-16T21:07:34.652839+0000","last_fullsized":"2024-04-16T21:07:34.652839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.646356+0000","last_change":"2024-04-16T19:25:13.475293+0000","last_active":"2024-04-16T21:06:59.646356+0000","last_peered":"2024-04-16T21:06:59.646356+0000","last_clean":"2024-04-16T21:06:59.646356+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:06:59.646356+0000","last_undegraded":"2024-04-16T21:06:59.646356+0000","last_fullsized":"2024-04-16T21:06:59.646356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'696","reported_seq":41908,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485741+0000","last_change":"2024-04-16T19:25:11.465229+0000","last_active":"2024-04-16T21:06:44.485741+0000","last_peered":"2024-04-16T21:06:44.485741+0000","last_clean":"2024-04-16T21:06:44.485741+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:06:44.485741+0000","last_undegraded":"2024-04-16T21:06:44.485741+0000","last_fullsized":"2024-04-16T21:06:44.485741+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:43.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'25","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652851+0000","last_change":"2024-04-16T19:25:09.464644+0000","last_active":"2024-04-16T21:07:39.652851+0000","last_peered":"2024-04-16T21:07:39.652851+0000","last_clean":"2024-04-16T21:07:39.652851+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:07:39.652851+0000","last_undegraded":"2024-04-16T21:07:39.652851+0000","last_fullsized":"2024-04-16T21:07:39.652851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:46:24.560604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650411+0000","last_change":"2024-04-16T19:24:51.438116+0000","last_active":"2024-04-16T21:07:24.650411+0000","last_peered":"2024-04-16T21:07:24.650411+0000","last_clean":"2024-04-16T21:07:24.650411+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:07:24.650411+0000","last_undegraded":"2024-04-16T21:07:24.650411+0000","last_fullsized":"2024-04-16T21:07:24.650411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662083+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:08:34.662083+0000","last_peered":"2024-04-16T21:08:34.662083+0000","last_clean":"2024-04-16T21:08:34.662083+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:08:34.662083+0000","last_undegraded":"2024-04-16T21:08:34.662083+0000","last_fullsized":"2024-04-16T21:08:34.662083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650147+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:07:14.650147+0000","last_peered":"2024-04-16T21:07:14.650147+0000","last_clean":"2024-04-16T21:07:14.650147+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:07:14.650147+0000","last_undegraded":"2024-04-16T21:07:14.650147+0000","last_fullsized":"2024-04-16T21:07:14.650147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:05.587315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505369+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:08:44.505369+0000","last_peered":"2024-04-16T21:08:44.505369+0000","last_clean":"2024-04-16T21:08:44.505369+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:08:44.505369+0000","last_undegraded":"2024-04-16T21:08:44.505369+0000","last_fullsized":"2024-04-16T21:08:44.505369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:53.769511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":274,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495624+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:07:44.495624+0000","last_peered":"2024-04-16T21:07:44.495624+0000","last_clean":"2024-04-16T21:07:44.495624+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:07:44.495624+0000","last_undegraded":"2024-04-16T21:07:44.495624+0000","last_fullsized":"2024-04-16T21:07:44.495624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:44.851136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'12","reported_seq":1308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365800+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:08:44.365800+0000","last_peered":"2024-04-16T21:08:44.365800+0000","last_clean":"2024-04-16T21:08:44.365800+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:08:44.365800+0000","last_undegraded":"2024-04-16T21:08:44.365800+0000","last_fullsized":"2024-04-16T21:08:44.365800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:40.173753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":72,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494076+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:07:34.494076+0000","last_peered":"2024-04-16T21:07:34.494076+0000","last_clean":"2024-04-16T21:07:34.494076+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:07:34.494076+0000","last_undegraded":"2024-04-16T21:07:34.494076+0000","last_fullsized":"2024-04-16T21:07:34.494076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:53.541439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1472,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659510+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:08:19.659510+0000","last_peered":"2024-04-16T21:08:19.659510+0000","last_clean":"2024-04-16T21:08:19.659510+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:08:19.659510+0000","last_undegraded":"2024-04-16T21:08:19.659510+0000","last_fullsized":"2024-04-16T21:08:19.659510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:30.342275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657195+0000","last_change":"2024-04-16T19:25:41.501744+0000","last_active":"2024-04-16T21:07:59.657195+0000","last_peered":"2024-04-16T21:07:59.657195+0000","last_clean":"2024-04-16T21:07:59.657195+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:07:59.657195+0000","last_undegraded":"2024-04-16T21:07:59.657195+0000","last_fullsized":"2024-04-16T21:07:59.657195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:33.802040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'9","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.652032+0000","last_change":"2024-04-16T19:25:15.487126+0000","last_active":"2024-04-16T21:07:24.652032+0000","last_peered":"2024-04-16T21:07:24.652032+0000","last_clean":"2024-04-16T21:07:24.652032+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:07:24.652032+0000","last_undegraded":"2024-04-16T21:07:24.652032+0000","last_fullsized":"2024-04-16T21:07:24.652032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485708+0000","last_change":"2024-04-16T19:25:13.474735+0000","last_active":"2024-04-16T21:06:44.485708+0000","last_peered":"2024-04-16T21:06:44.485708+0000","last_clean":"2024-04-16T21:06:44.485708+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:06:44.485708+0000","last_undegraded":"2024-04-16T21:06:44.485708+0000","last_fullsized":"2024-04-16T21:06:44.485708+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'339","reported_seq":652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488172+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:06:59.488172+0000","last_peered":"2024-04-16T21:06:59.488172+0000","last_clean":"2024-04-16T21:06:59.488172+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:06:59.488172+0000","last_undegraded":"2024-04-16T21:06:59.488172+0000","last_fullsized":"2024-04-16T21:06:59.488172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:21.744422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'10","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494764+0000","last_change":"2024-04-16T19:25:09.468124+0000","last_active":"2024-04-16T21:07:39.494764+0000","last_peered":"2024-04-16T21:07:39.494764+0000","last_clean":"2024-04-16T21:07:39.494764+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:07:39.494764+0000","last_undegraded":"2024-04-16T21:07:39.494764+0000","last_fullsized":"2024-04-16T21:07:39.494764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:31.945014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651298+0000","last_change":"2024-04-16T19:24:51.572963+0000","last_active":"2024-04-16T21:07:24.651298+0000","last_peered":"2024-04-16T21:07:24.651298+0000","last_clean":"2024-04-16T21:07:24.651298+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:07:24.651298+0000","last_undegraded":"2024-04-16T21:07:24.651298+0000","last_fullsized":"2024-04-16T21:07:24.651298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:46.905067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662114+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:08:34.662114+0000","last_peered":"2024-04-16T21:08:34.662114+0000","last_clean":"2024-04-16T21:08:34.662114+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:08:34.662114+0000","last_undegraded":"2024-04-16T21:08:34.662114+0000","last_fullsized":"2024-04-16T21:08:34.662114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649505+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:07:14.649505+0000","last_peered":"2024-04-16T21:07:14.649505+0000","last_clean":"2024-04-16T21:07:14.649505+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:07:14.649505+0000","last_undegraded":"2024-04-16T21:07:14.649505+0000","last_fullsized":"2024-04-16T21:07:14.649505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365827+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:08:44.365827+0000","last_peered":"2024-04-16T21:08:44.365827+0000","last_clean":"2024-04-16T21:08:44.365827+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:08:44.365827+0000","last_undegraded":"2024-04-16T21:08:44.365827+0000","last_fullsized":"2024-04-16T21:08:44.365827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:41.163065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365785+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:08:44.365785+0000","last_peered":"2024-04-16T21:08:44.365785+0000","last_clean":"2024-04-16T21:08:44.365785+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:08:44.365785+0000","last_undegraded":"2024-04-16T21:08:44.365785+0000","last_fullsized":"2024-04-16T21:08:44.365785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:19.370822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495596+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:07:44.495596+0000","last_peered":"2024-04-16T21:07:44.495596+0000","last_clean":"2024-04-16T21:07:44.495596+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:07:44.495596+0000","last_undegraded":"2024-04-16T21:07:44.495596+0000","last_fullsized":"2024-04-16T21:07:44.495596+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:27.385349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365439+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:08:49.365439+0000","last_peered":"2024-04-16T21:08:49.365439+0000","last_clean":"2024-04-16T21:08:49.365439+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:08:49.365439+0000","last_undegraded":"2024-04-16T21:08:49.365439+0000","last_fullsized":"2024-04-16T21:08:49.365439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:12:02.296640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2006,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656784+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:07:54.656784+0000","last_peered":"2024-04-16T21:07:54.656784+0000","last_clean":"2024-04-16T21:07:54.656784+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:07:54.656784+0000","last_undegraded":"2024-04-16T21:07:54.656784+0000","last_fullsized":"2024-04-16T21:07:54.656784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:10:23.783937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655892+0000","last_change":"2024-04-16T19:25:41.506037+0000","last_active":"2024-04-16T21:07:49.655892+0000","last_peered":"2024-04-16T21:07:49.655892+0000","last_clean":"2024-04-16T21:07:49.655892+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:07:49.655892+0000","last_undegraded":"2024-04-16T21:07:49.655892+0000","last_fullsized":"2024-04-16T21:07:49.655892+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'33","reported_seq":183,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652615+0000","last_change":"2024-04-16T19:25:15.482185+0000","last_active":"2024-04-16T21:07:34.652615+0000","last_peered":"2024-04-16T21:07:34.652615+0000","last_clean":"2024-04-16T21:07:34.652615+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:07:34.652615+0000","last_undegraded":"2024-04-16T21:07:34.652615+0000","last_fullsized":"2024-04-16T21:07:34.652615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:07.940015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648199+0000","last_change":"2024-04-16T19:25:13.475047+0000","last_active":"2024-04-16T21:07:04.648199+0000","last_peered":"2024-04-16T21:07:04.648199+0000","last_clean":"2024-04-16T21:07:04.648199+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:07:04.648199+0000","last_undegraded":"2024-04-16T21:07:04.648199+0000","last_fullsized":"2024-04-16T21:07:04.648199+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:29:34.601553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1740","reported_seq":4600,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648166+0000","last_change":"2024-04-16T19:25:11.468809+0000","last_active":"2024-04-16T21:07:04.648166+0000","last_peered":"2024-04-16T21:07:04.648166+0000","last_clean":"2024-04-16T21:07:04.648166+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:07:04.648166+0000","last_undegraded":"2024-04-16T21:07:04.648166+0000","last_fullsized":"2024-04-16T21:07:04.648166+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1740,"log_dups_size":0,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:53.762063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'19","reported_seq":270,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651251+0000","last_change":"2024-04-16T19:25:09.468894+0000","last_active":"2024-04-16T21:07:29.651251+0000","last_peered":"2024-04-16T21:07:29.651251+0000","last_clean":"2024-04-16T21:07:29.651251+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:07:29.651251+0000","last_undegraded":"2024-04-16T21:07:29.651251+0000","last_fullsized":"2024-04-16T21:07:29.651251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:05.430878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664164+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:08:39.664164+0000","last_peered":"2024-04-16T21:08:39.664164+0000","last_clean":"2024-04-16T21:08:39.664164+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:08:39.664164+0000","last_undegraded":"2024-04-16T21:08:39.664164+0000","last_fullsized":"2024-04-16T21:08:39.664164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660546+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:08:19.660546+0000","last_peered":"2024-04-16T21:08:19.660546+0000","last_clean":"2024-04-16T21:08:19.660546+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:08:19.660546+0000","last_undegraded":"2024-04-16T21:08:19.660546+0000","last_fullsized":"2024-04-16T21:08:19.660546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365365+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:08:49.365365+0000","last_peered":"2024-04-16T21:08:49.365365+0000","last_clean":"2024-04-16T21:08:49.365365+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:08:49.365365+0000","last_undegraded":"2024-04-16T21:08:49.365365+0000","last_fullsized":"2024-04-16T21:08:49.365365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:48.947818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":394,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368921+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:09:09.368921+0000","last_peered":"2024-04-16T21:09:09.368921+0000","last_clean":"2024-04-16T21:09:09.368921+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:09:09.368921+0000","last_undegraded":"2024-04-16T21:09:09.368921+0000","last_fullsized":"2024-04-16T21:09:09.368921+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:36.588167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501183+0000","last_change":"2024-04-16T19:24:49.434497+0000","last_active":"2024-04-16T21:08:19.501183+0000","last_peered":"2024-04-16T21:08:19.501183+0000","last_clean":"2024-04-16T21:08:19.501183+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:08:19.501183+0000","last_undegraded":"2024-04-16T21:08:19.501183+0000","last_fullsized":"2024-04-16T21:08:19.501183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:08.864273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368025+0000","last_change":"2024-04-16T19:24:51.437122+0000","last_active":"2024-04-16T21:09:04.368025+0000","last_peered":"2024-04-16T21:09:04.368025+0000","last_clean":"2024-04-16T21:09:04.368025+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:09:04.368025+0000","last_undegraded":"2024-04-16T21:09:04.368025+0000","last_fullsized":"2024-04-16T21:09:04.368025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:10.861306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506044+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:08:49.506044+0000","last_peered":"2024-04-16T21:08:49.506044+0000","last_clean":"2024-04-16T21:08:49.506044+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:08:49.506044+0000","last_undegraded":"2024-04-16T21:08:49.506044+0000","last_fullsized":"2024-04-16T21:08:49.506044+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:48:31.872270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1928,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655341+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:07:54.655341+0000","last_peered":"2024-04-16T21:07:54.655341+0000","last_clean":"2024-04-16T21:07:54.655341+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:07:54.655341+0000","last_undegraded":"2024-04-16T21:07:54.655341+0000","last_fullsized":"2024-04-16T21:07:54.655341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657746+0000","last_change":"2024-04-16T19:25:41.502674+0000","last_active":"2024-04-16T21:07:59.657746+0000","last_peered":"2024-04-16T21:07:59.657746+0000","last_clean":"2024-04-16T21:07:59.657746+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:07:59.657746+0000","last_undegraded":"2024-04-16T21:07:59.657746+0000","last_fullsized":"2024-04-16T21:07:59.657746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'13","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651772+0000","last_change":"2024-04-16T19:25:15.485057+0000","last_active":"2024-04-16T21:07:24.651772+0000","last_peered":"2024-04-16T21:07:24.651772+0000","last_clean":"2024-04-16T21:07:24.651772+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:07:24.651772+0000","last_undegraded":"2024-04-16T21:07:24.651772+0000","last_fullsized":"2024-04-16T21:07:24.651772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:45.514212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647443+0000","last_change":"2024-04-16T19:25:13.472089+0000","last_active":"2024-04-16T21:07:04.647443+0000","last_peered":"2024-04-16T21:07:04.647443+0000","last_clean":"2024-04-16T21:07:04.647443+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:07:04.647443+0000","last_undegraded":"2024-04-16T21:07:04.647443+0000","last_fullsized":"2024-04-16T21:07:04.647443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'304","reported_seq":628,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648664+0000","last_change":"2024-04-16T19:25:11.462997+0000","last_active":"2024-04-16T21:07:04.648664+0000","last_peered":"2024-04-16T21:07:04.648664+0000","last_clean":"2024-04-16T21:07:04.648664+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:07:04.648664+0000","last_undegraded":"2024-04-16T21:07:04.648664+0000","last_fullsized":"2024-04-16T21:07:04.648664+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:14.402830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'8","reported_seq":279,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490828+0000","last_change":"2024-04-16T19:25:09.467337+0000","last_active":"2024-04-16T21:07:14.490828+0000","last_peered":"2024-04-16T21:07:14.490828+0000","last_clean":"2024-04-16T21:07:14.490828+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:07:14.490828+0000","last_undegraded":"2024-04-16T21:07:14.490828+0000","last_fullsized":"2024-04-16T21:07:14.490828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663257+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:08:39.663257+0000","last_peered":"2024-04-16T21:08:39.663257+0000","last_clean":"2024-04-16T21:08:39.663257+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:08:39.663257+0000","last_undegraded":"2024-04-16T21:08:39.663257+0000","last_fullsized":"2024-04-16T21:08:39.663257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658066+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:08:04.658066+0000","last_peered":"2024-04-16T21:08:04.658066+0000","last_clean":"2024-04-16T21:08:04.658066+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:08:04.658066+0000","last_undegraded":"2024-04-16T21:08:04.658066+0000","last_fullsized":"2024-04-16T21:08:04.658066+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:20.908683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":959,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.505980+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:08:49.505980+0000","last_peered":"2024-04-16T21:08:49.505980+0000","last_clean":"2024-04-16T21:08:49.505980+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:08:49.505980+0000","last_undegraded":"2024-04-16T21:08:49.505980+0000","last_fullsized":"2024-04-16T21:08:49.505980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:35.897776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501172+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:08:19.501172+0000","last_peered":"2024-04-16T21:08:19.501172+0000","last_clean":"2024-04-16T21:08:19.501172+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:08:19.501172+0000","last_undegraded":"2024-04-16T21:08:19.501172+0000","last_fullsized":"2024-04-16T21:08:19.501172+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:40.145350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508432+0000","last_change":"2024-04-16T19:24:49.429526+0000","last_active":"2024-04-16T21:09:04.508432+0000","last_peered":"2024-04-16T21:09:04.508432+0000","last_clean":"2024-04-16T21:09:04.508432+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:09:04.508432+0000","last_undegraded":"2024-04-16T21:09:04.508432+0000","last_fullsized":"2024-04-16T21:09:04.508432+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:36.881378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649882+0000","last_change":"2024-04-16T19:24:51.437305+0000","last_active":"2024-04-16T21:07:14.649882+0000","last_peered":"2024-04-16T21:07:14.649882+0000","last_clean":"2024-04-16T21:07:14.649882+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:07:14.649882+0000","last_undegraded":"2024-04-16T21:07:14.649882+0000","last_fullsized":"2024-04-16T21:07:14.649882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:24.125422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366382+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:08:49.366382+0000","last_peered":"2024-04-16T21:08:49.366382+0000","last_clean":"2024-04-16T21:08:49.366382+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:08:49.366382+0000","last_undegraded":"2024-04-16T21:08:49.366382+0000","last_fullsized":"2024-04-16T21:08:49.366382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:22:33.730359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'93","reported_seq":1378,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498742+0000","last_change":"2024-04-16T19:26:41.564908+0000","last_active":"2024-04-16T21:08:04.498742+0000","last_peered":"2024-04-16T21:08:04.498742+0000","last_clean":"2024-04-16T21:08:04.498742+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:08:04.498742+0000","last_undegraded":"2024-04-16T21:08:04.498742+0000","last_fullsized":"2024-04-16T21:08:04.498742+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:47:23.710928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654443+0000","last_change":"2024-04-16T19:25:41.508929+0000","last_active":"2024-04-16T21:07:49.654443+0000","last_peered":"2024-04-16T21:07:49.654443+0000","last_clean":"2024-04-16T21:07:49.654443+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:07:49.654443+0000","last_undegraded":"2024-04-16T21:07:49.654443+0000","last_fullsized":"2024-04-16T21:07:49.654443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'19","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651849+0000","last_change":"2024-04-16T19:25:15.482026+0000","last_active":"2024-04-16T21:07:24.651849+0000","last_peered":"2024-04-16T21:07:24.651849+0000","last_clean":"2024-04-16T21:07:24.651849+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:07:24.651849+0000","last_undegraded":"2024-04-16T21:07:24.651849+0000","last_fullsized":"2024-04-16T21:07:24.651849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652807+0000","last_change":"2024-04-16T19:25:13.475419+0000","last_active":"2024-04-16T21:07:34.652807+0000","last_peered":"2024-04-16T21:07:34.652807+0000","last_clean":"2024-04-16T21:07:34.652807+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:07:34.652807+0000","last_undegraded":"2024-04-16T21:07:34.652807+0000","last_fullsized":"2024-04-16T21:07:34.652807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2044","reported_seq":5813,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649720+0000","last_change":"2024-04-16T19:25:11.468843+0000","last_active":"2024-04-16T21:07:19.649720+0000","last_peered":"2024-04-16T21:07:19.649720+0000","last_clean":"2024-04-16T21:07:19.649720+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:07:19.649720+0000","last_undegraded":"2024-04-16T21:07:19.649720+0000","last_fullsized":"2024-04-16T21:07:19.649720+0000","mapping_epoch":33,"log_start":"47'300","ondisk_log_start":"47'300","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1744,"log_dups_size":300,"ondisk_log_size":1744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:03.761996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'14","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369114+0000","last_change":"2024-04-16T19:25:09.467702+0000","last_active":"2024-04-16T21:09:09.369114+0000","last_peered":"2024-04-16T21:09:09.369114+0000","last_clean":"2024-04-16T21:09:09.369114+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:09:09.369114+0000","last_undegraded":"2024-04-16T21:09:09.369114+0000","last_fullsized":"2024-04-16T21:09:09.369114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:31.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367115+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:08:59.367115+0000","last_peered":"2024-04-16T21:08:59.367115+0000","last_clean":"2024-04-16T21:08:59.367115+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:08:59.367115+0000","last_undegraded":"2024-04-16T21:08:59.367115+0000","last_fullsized":"2024-04-16T21:08:59.367115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:40.677584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659023+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:08:04.659023+0000","last_peered":"2024-04-16T21:08:04.659023+0000","last_clean":"2024-04-16T21:08:04.659023+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:08:04.659023+0000","last_undegraded":"2024-04-16T21:08:04.659023+0000","last_fullsized":"2024-04-16T21:08:04.659023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:49.358784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500796+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:08:19.500796+0000","last_peered":"2024-04-16T21:08:19.500796+0000","last_clean":"2024-04-16T21:08:19.500796+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:08:19.500796+0000","last_undegraded":"2024-04-16T21:08:19.500796+0000","last_fullsized":"2024-04-16T21:08:19.500796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:59.160445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":328,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366327+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:08:49.366327+0000","last_peered":"2024-04-16T21:08:49.366327+0000","last_clean":"2024-04-16T21:08:49.366327+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:08:49.366327+0000","last_undegraded":"2024-04-16T21:08:49.366327+0000","last_fullsized":"2024-04-16T21:08:49.366327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1313,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370434+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:08:49.370434+0000","last_peered":"2024-04-16T21:08:49.370434+0000","last_clean":"2024-04-16T21:08:49.370434+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:08:49.370434+0000","last_undegraded":"2024-04-16T21:08:49.370434+0000","last_fullsized":"2024-04-16T21:08:49.370434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:30.260652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371475+0000","last_change":"2024-04-16T19:24:51.437991+0000","last_active":"2024-04-16T21:08:54.371475+0000","last_peered":"2024-04-16T21:08:54.371475+0000","last_clean":"2024-04-16T21:08:54.371475+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:08:54.371475+0000","last_undegraded":"2024-04-16T21:08:54.371475+0000","last_fullsized":"2024-04-16T21:08:54.371475+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:43:50.519739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496442+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:07:49.496442+0000","last_peered":"2024-04-16T21:07:49.496442+0000","last_clean":"2024-04-16T21:07:49.496442+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:07:49.496442+0000","last_undegraded":"2024-04-16T21:07:49.496442+0000","last_fullsized":"2024-04-16T21:07:49.496442+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:16:07.995736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'136","reported_seq":1889,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497219+0000","last_change":"2024-04-16T19:25:49.523163+0000","last_active":"2024-04-16T21:07:54.497219+0000","last_peered":"2024-04-16T21:07:54.497219+0000","last_clean":"2024-04-16T21:07:54.497219+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:07:54.497219+0000","last_undegraded":"2024-04-16T21:07:54.497219+0000","last_fullsized":"2024-04-16T21:07:54.497219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:08.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655310+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:07:49.655310+0000","last_peered":"2024-04-16T21:07:49.655310+0000","last_clean":"2024-04-16T21:07:49.655310+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:07:49.655310+0000","last_undegraded":"2024-04-16T21:07:49.655310+0000","last_fullsized":"2024-04-16T21:07:49.655310+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:36.879550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'11","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650465+0000","last_change":"2024-04-16T19:25:15.480717+0000","last_active":"2024-04-16T21:07:24.650465+0000","last_peered":"2024-04-16T21:07:24.650465+0000","last_clean":"2024-04-16T21:07:24.650465+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:07:24.650465+0000","last_undegraded":"2024-04-16T21:07:24.650465+0000","last_fullsized":"2024-04-16T21:07:24.650465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:39.031938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650534+0000","last_change":"2024-04-16T19:25:13.476070+0000","last_active":"2024-04-16T21:07:19.650534+0000","last_peered":"2024-04-16T21:07:19.650534+0000","last_clean":"2024-04-16T21:07:19.650534+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:07:19.650534+0000","last_undegraded":"2024-04-16T21:07:19.650534+0000","last_fullsized":"2024-04-16T21:07:19.650534+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:42.684942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'341","reported_seq":664,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494066+0000","last_change":"2024-04-16T19:25:11.467316+0000","last_active":"2024-04-16T21:07:34.494066+0000","last_peered":"2024-04-16T21:07:34.494066+0000","last_clean":"2024-04-16T21:07:34.494066+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:07:34.494066+0000","last_undegraded":"2024-04-16T21:07:34.494066+0000","last_fullsized":"2024-04-16T21:07:34.494066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'9","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369072+0000","last_change":"2024-04-16T19:25:09.470409+0000","last_active":"2024-04-16T21:09:09.369072+0000","last_peered":"2024-04-16T21:09:09.369072+0000","last_clean":"2024-04-16T21:09:09.369072+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:09:09.369072+0000","last_undegraded":"2024-04-16T21:09:09.369072+0000","last_fullsized":"2024-04-16T21:09:09.369072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657857+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:08:09.657857+0000","last_peered":"2024-04-16T21:08:09.657857+0000","last_clean":"2024-04-16T21:08:09.657857+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:08:09.657857+0000","last_undegraded":"2024-04-16T21:08:09.657857+0000","last_fullsized":"2024-04-16T21:08:09.657857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658003+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:08:04.658003+0000","last_peered":"2024-04-16T21:08:04.658003+0000","last_clean":"2024-04-16T21:08:04.658003+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:08:04.658003+0000","last_undegraded":"2024-04-16T21:08:04.658003+0000","last_fullsized":"2024-04-16T21:08:04.658003+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:29.699119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":457,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508414+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:09:04.508414+0000","last_peered":"2024-04-16T21:09:04.508414+0000","last_clean":"2024-04-16T21:09:04.508414+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:09:04.508414+0000","last_undegraded":"2024-04-16T21:09:04.508414+0000","last_fullsized":"2024-04-16T21:09:04.508414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:12.039717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":246,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370400+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:08:49.370400+0000","last_peered":"2024-04-16T21:08:49.370400+0000","last_clean":"2024-04-16T21:08:49.370400+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:08:49.370400+0000","last_undegraded":"2024-04-16T21:08:49.370400+0000","last_fullsized":"2024-04-16T21:08:49.370400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:16.109122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506003+0000","last_change":"2024-04-16T19:24:49.432294+0000","last_active":"2024-04-16T21:08:49.506003+0000","last_peered":"2024-04-16T21:08:49.506003+0000","last_clean":"2024-04-16T21:08:49.506003+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:08:49.506003+0000","last_undegraded":"2024-04-16T21:08:49.506003+0000","last_fullsized":"2024-04-16T21:08:49.506003+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:45.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365802+0000","last_change":"2024-04-16T19:24:51.442478+0000","last_active":"2024-04-16T21:08:54.365802+0000","last_peered":"2024-04-16T21:08:54.365802+0000","last_clean":"2024-04-16T21:08:54.365802+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:08:54.365802+0000","last_undegraded":"2024-04-16T21:08:54.365802+0000","last_fullsized":"2024-04-16T21:08:54.365802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495710+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:07:44.495710+0000","last_peered":"2024-04-16T21:07:44.495710+0000","last_clean":"2024-04-16T21:07:44.495710+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:07:44.495710+0000","last_undegraded":"2024-04-16T21:07:44.495710+0000","last_fullsized":"2024-04-16T21:07:44.495710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:02:07.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2345,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652944+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:07:39.652944+0000","last_peered":"2024-04-16T21:07:39.652944+0000","last_clean":"2024-04-16T21:07:39.652944+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:07:39.652944+0000","last_undegraded":"2024-04-16T21:07:39.652944+0000","last_fullsized":"2024-04-16T21:07:39.652944+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658027+0000","last_change":"2024-04-16T19:25:41.509053+0000","last_active":"2024-04-16T21:08:04.658027+0000","last_peered":"2024-04-16T21:08:04.658027+0000","last_clean":"2024-04-16T21:08:04.658027+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:08:04.658027+0000","last_undegraded":"2024-04-16T21:08:04.658027+0000","last_fullsized":"2024-04-16T21:08:04.658027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:22.621386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"47'8","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651351+0000","last_change":"2024-04-16T19:25:15.486624+0000","last_active":"2024-04-16T21:07:24.651351+0000","last_peered":"2024-04-16T21:07:24.651351+0000","last_clean":"2024-04-16T21:07:24.651351+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:07:24.651351+0000","last_undegraded":"2024-04-16T21:07:24.651351+0000","last_fullsized":"2024-04-16T21:07:24.651351+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:35.616571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'705","reported_seq":2064,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651010+0000","last_change":"2024-04-16T19:25:13.473118+0000","last_active":"2024-04-16T21:07:19.651010+0000","last_peered":"2024-04-16T21:07:19.651010+0000","last_clean":"2024-04-16T21:07:19.651010+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:07:19.651010+0000","last_undegraded":"2024-04-16T21:07:19.651010+0000","last_fullsized":"2024-04-16T21:07:19.651010+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'269","reported_seq":540,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491606+0000","last_change":"2024-04-16T19:25:11.465450+0000","last_active":"2024-04-16T21:07:19.491606+0000","last_peered":"2024-04-16T21:07:19.491606+0000","last_clean":"2024-04-16T21:07:19.491606+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:07:19.491606+0000","last_undegraded":"2024-04-16T21:07:19.491606+0000","last_fullsized":"2024-04-16T21:07:19.491606+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:53.479823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'5","reported_seq":214,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648747+0000","last_change":"2024-04-16T19:25:09.458268+0000","last_active":"2024-04-16T21:07:14.648747+0000","last_peered":"2024-04-16T21:07:14.648747+0000","last_clean":"2024-04-16T21:07:14.648747+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:07:14.648747+0000","last_undegraded":"2024-04-16T21:07:14.648747+0000","last_fullsized":"2024-04-16T21:07:14.648747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:05.252198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658783+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:08:09.658783+0000","last_peered":"2024-04-16T21:08:09.658783+0000","last_clean":"2024-04-16T21:08:09.658783+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:08:09.658783+0000","last_undegraded":"2024-04-16T21:08:09.658783+0000","last_fullsized":"2024-04-16T21:08:09.658783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:29.013265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498840+0000","last_change":"2024-04-16T19:25:41.503885+0000","last_active":"2024-04-16T21:08:04.498840+0000","last_peered":"2024-04-16T21:08:04.498840+0000","last_clean":"2024-04-16T21:08:04.498840+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:08:04.498840+0000","last_undegraded":"2024-04-16T21:08:04.498840+0000","last_fullsized":"2024-04-16T21:08:04.498840+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:54.516253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500846+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:08:19.500846+0000","last_peered":"2024-04-16T21:08:19.500846+0000","last_clean":"2024-04-16T21:08:19.500846+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:08:19.500846+0000","last_undegraded":"2024-04-16T21:08:19.500846+0000","last_fullsized":"2024-04-16T21:08:19.500846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:30.224132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660316+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:08:19.660316+0000","last_peered":"2024-04-16T21:08:19.660316+0000","last_clean":"2024-04-16T21:08:19.660316+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:08:19.660316+0000","last_undegraded":"2024-04-16T21:08:19.660316+0000","last_fullsized":"2024-04-16T21:08:19.660316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:43.606761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662124+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:08:34.662124+0000","last_peered":"2024-04-16T21:08:34.662124+0000","last_clean":"2024-04-16T21:08:34.662124+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:08:34.662124+0000","last_undegraded":"2024-04-16T21:08:34.662124+0000","last_fullsized":"2024-04-16T21:08:34.662124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368874+0000","last_change":"2024-04-16T19:24:51.434984+0000","last_active":"2024-04-16T21:09:14.368874+0000","last_peered":"2024-04-16T21:09:14.368874+0000","last_clean":"2024-04-16T21:09:14.368874+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:09:14.368874+0000","last_undegraded":"2024-04-16T21:09:14.368874+0000","last_fullsized":"2024-04-16T21:09:14.368874+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:26.835802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653730+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:07:44.653730+0000","last_peered":"2024-04-16T21:07:44.653730+0000","last_clean":"2024-04-16T21:07:44.653730+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:07:44.653730+0000","last_undegraded":"2024-04-16T21:07:44.653730+0000","last_fullsized":"2024-04-16T21:07:44.653730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:56.544746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1678,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653711+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:07:39.653711+0000","last_peered":"2024-04-16T21:07:39.653711+0000","last_clean":"2024-04-16T21:07:39.653711+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:07:39.653711+0000","last_undegraded":"2024-04-16T21:07:39.653711+0000","last_fullsized":"2024-04-16T21:07:39.653711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:01.314955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655387+0000","last_change":"2024-04-16T19:25:41.504293+0000","last_active":"2024-04-16T21:07:49.655387+0000","last_peered":"2024-04-16T21:07:49.655387+0000","last_clean":"2024-04-16T21:07:49.655387+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:07:49.655387+0000","last_undegraded":"2024-04-16T21:07:49.655387+0000","last_fullsized":"2024-04-16T21:07:49.655387+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'136","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654370+0000","last_change":"2024-04-16T19:25:15.483254+0000","last_active":"2024-04-16T21:07:39.654370+0000","last_peered":"2024-04-16T21:07:39.654370+0000","last_clean":"2024-04-16T21:07:39.654370+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:07:39.654370+0000","last_undegraded":"2024-04-16T21:07:39.654370+0000","last_fullsized":"2024-04-16T21:07:39.654370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491600+0000","last_change":"2024-04-16T19:25:13.471760+0000","last_active":"2024-04-16T21:07:19.491600+0000","last_peered":"2024-04-16T21:07:19.491600+0000","last_clean":"2024-04-16T21:07:19.491600+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:07:19.491600+0000","last_undegraded":"2024-04-16T21:07:19.491600+0000","last_fullsized":"2024-04-16T21:07:19.491600+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'347","reported_seq":724,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491639+0000","last_change":"2024-04-16T19:25:11.467463+0000","last_active":"2024-04-16T21:07:19.491639+0000","last_peered":"2024-04-16T21:07:19.491639+0000","last_clean":"2024-04-16T21:07:19.491639+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:07:19.491639+0000","last_undegraded":"2024-04-16T21:07:19.491639+0000","last_fullsized":"2024-04-16T21:07:19.491639+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492364+0000","last_change":"2024-04-16T19:25:09.461452+0000","last_active":"2024-04-16T21:07:24.492364+0000","last_peered":"2024-04-16T21:07:24.492364+0000","last_clean":"2024-04-16T21:07:24.492364+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:07:24.492364+0000","last_undegraded":"2024-04-16T21:07:24.492364+0000","last_fullsized":"2024-04-16T21:07:24.492364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:44:08.538156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659737+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:08:09.659737+0000","last_peered":"2024-04-16T21:08:09.659737+0000","last_clean":"2024-04-16T21:08:09.659737+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:08:09.659737+0000","last_undegraded":"2024-04-16T21:08:09.659737+0000","last_fullsized":"2024-04-16T21:08:09.659737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496482+0000","last_change":"2024-04-16T19:25:41.506323+0000","last_active":"2024-04-16T21:07:49.496482+0000","last_peered":"2024-04-16T21:07:49.496482+0000","last_clean":"2024-04-16T21:07:49.496482+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:07:49.496482+0000","last_undegraded":"2024-04-16T21:07:49.496482+0000","last_fullsized":"2024-04-16T21:07:49.496482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:05.815658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501233+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:08:19.501233+0000","last_peered":"2024-04-16T21:08:19.501233+0000","last_clean":"2024-04-16T21:08:19.501233+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:08:19.501233+0000","last_undegraded":"2024-04-16T21:08:19.501233+0000","last_fullsized":"2024-04-16T21:08:19.501233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:51.524577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'140","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503630+0000","last_change":"2024-04-16T19:24:47.426406+0000","last_active":"2024-04-16T21:08:34.503630+0000","last_peered":"2024-04-16T21:08:34.503630+0000","last_clean":"2024-04-16T21:08:34.503630+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:08:34.503630+0000","last_undegraded":"2024-04-16T21:08:34.503630+0000","last_fullsized":"2024-04-16T21:08:34.503630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:12.378311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500867+0000","last_change":"2024-04-16T19:24:49.430773+0000","last_active":"2024-04-16T21:08:19.500867+0000","last_peered":"2024-04-16T21:08:19.500867+0000","last_clean":"2024-04-16T21:08:19.500867+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:08:19.500867+0000","last_undegraded":"2024-04-16T21:08:19.500867+0000","last_fullsized":"2024-04-16T21:08:19.500867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:52.965094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367299+0000","last_change":"2024-04-16T19:24:51.436754+0000","last_active":"2024-04-16T21:09:04.367299+0000","last_peered":"2024-04-16T21:09:04.367299+0000","last_clean":"2024-04-16T21:09:04.367299+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:09:04.367299+0000","last_undegraded":"2024-04-16T21:09:04.367299+0000","last_fullsized":"2024-04-16T21:09:04.367299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:01.678207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368731+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:09:04.368731+0000","last_peered":"2024-04-16T21:09:04.368731+0000","last_clean":"2024-04-16T21:09:04.368731+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:09:04.368731+0000","last_undegraded":"2024-04-16T21:09:04.368731+0000","last_fullsized":"2024-04-16T21:09:04.368731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:01:32.497457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1870,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656880+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:07:54.656880+0000","last_peered":"2024-04-16T21:07:54.656880+0000","last_clean":"2024-04-16T21:07:54.656880+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:07:54.656880+0000","last_undegraded":"2024-04-16T21:07:54.656880+0000","last_fullsized":"2024-04-16T21:07:54.656880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653650+0000","last_change":"2024-04-16T19:25:41.507626+0000","last_active":"2024-04-16T21:07:34.653650+0000","last_peered":"2024-04-16T21:07:34.653650+0000","last_clean":"2024-04-16T21:07:34.653650+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:07:34.653650+0000","last_undegraded":"2024-04-16T21:07:34.653650+0000","last_fullsized":"2024-04-16T21:07:34.653650+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'7","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489918+0000","last_change":"2024-04-16T19:25:15.477722+0000","last_active":"2024-04-16T21:07:09.489918+0000","last_peered":"2024-04-16T21:07:09.489918+0000","last_clean":"2024-04-16T21:07:09.489918+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:07:09.489918+0000","last_undegraded":"2024-04-16T21:07:09.489918+0000","last_fullsized":"2024-04-16T21:07:09.489918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:11.553471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650458+0000","last_change":"2024-04-16T19:25:13.476043+0000","last_active":"2024-04-16T21:07:19.650458+0000","last_peered":"2024-04-16T21:07:19.650458+0000","last_clean":"2024-04-16T21:07:19.650458+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:07:19.650458+0000","last_undegraded":"2024-04-16T21:07:19.650458+0000","last_fullsized":"2024-04-16T21:07:19.650458+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'453","reported_seq":868,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651934+0000","last_change":"2024-04-16T19:25:11.462559+0000","last_active":"2024-04-16T21:07:29.651934+0000","last_peered":"2024-04-16T21:07:29.651934+0000","last_clean":"2024-04-16T21:07:29.651934+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:07:29.651934+0000","last_undegraded":"2024-04-16T21:07:29.651934+0000","last_fullsized":"2024-04-16T21:07:29.651934+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:33.939814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'8","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655202+0000","last_change":"2024-04-16T19:25:41.507771+0000","last_active":"2024-04-16T21:07:49.655202+0000","last_peered":"2024-04-16T21:07:49.655202+0000","last_clean":"2024-04-16T21:07:49.655202+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:07:49.655202+0000","last_undegraded":"2024-04-16T21:07:49.655202+0000","last_fullsized":"2024-04-16T21:07:49.655202+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:53.575807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658859+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:08:09.658859+0000","last_peered":"2024-04-16T21:08:09.658859+0000","last_clean":"2024-04-16T21:08:09.658859+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:08:09.658859+0000","last_undegraded":"2024-04-16T21:08:09.658859+0000","last_fullsized":"2024-04-16T21:08:09.658859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662590+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:08:34.662590+0000","last_peered":"2024-04-16T21:08:34.662590+0000","last_clean":"2024-04-16T21:08:34.662590+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:08:34.662590+0000","last_undegraded":"2024-04-16T21:08:34.662590+0000","last_fullsized":"2024-04-16T21:08:34.662590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1361,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663160+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:08:34.663160+0000","last_peered":"2024-04-16T21:08:34.663160+0000","last_clean":"2024-04-16T21:08:34.663160+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:08:34.663160+0000","last_undegraded":"2024-04-16T21:08:34.663160+0000","last_fullsized":"2024-04-16T21:08:34.663160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:46.278655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":276,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501256+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:08:19.501256+0000","last_peered":"2024-04-16T21:08:19.501256+0000","last_clean":"2024-04-16T21:08:19.501256+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:08:19.501256+0000","last_undegraded":"2024-04-16T21:08:19.501256+0000","last_fullsized":"2024-04-16T21:08:19.501256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:23.115125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660267+0000","last_change":"2024-04-16T19:24:49.428502+0000","last_active":"2024-04-16T21:08:19.660267+0000","last_peered":"2024-04-16T21:08:19.660267+0000","last_clean":"2024-04-16T21:08:19.660267+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:08:19.660267+0000","last_undegraded":"2024-04-16T21:08:19.660267+0000","last_fullsized":"2024-04-16T21:08:19.660267+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:25.122206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367884+0000","last_change":"2024-04-16T19:24:51.435114+0000","last_active":"2024-04-16T21:09:04.367884+0000","last_peered":"2024-04-16T21:09:04.367884+0000","last_clean":"2024-04-16T21:09:04.367884+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:09:04.367884+0000","last_undegraded":"2024-04-16T21:09:04.367884+0000","last_fullsized":"2024-04-16T21:09:04.367884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:56.562306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364938+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:08:49.364938+0000","last_peered":"2024-04-16T21:08:49.364938+0000","last_clean":"2024-04-16T21:08:49.364938+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:08:49.364938+0000","last_undegraded":"2024-04-16T21:08:49.364938+0000","last_fullsized":"2024-04-16T21:08:49.364938+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:08:56.543210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657898+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:08:09.657898+0000","last_peered":"2024-04-16T21:08:09.657898+0000","last_clean":"2024-04-16T21:08:09.657898+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:08:09.657898+0000","last_undegraded":"2024-04-16T21:08:09.657898+0000","last_fullsized":"2024-04-16T21:08:09.657898+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:44:18.098055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653619+0000","last_change":"2024-04-16T19:25:41.506250+0000","last_active":"2024-04-16T21:07:34.653619+0000","last_peered":"2024-04-16T21:07:34.653619+0000","last_clean":"2024-04-16T21:07:34.653619+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:07:34.653619+0000","last_undegraded":"2024-04-16T21:07:34.653619+0000","last_fullsized":"2024-04-16T21:07:34.653619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489957+0000","last_change":"2024-04-16T19:25:15.479905+0000","last_active":"2024-04-16T21:07:09.489957+0000","last_peered":"2024-04-16T21:07:09.489957+0000","last_clean":"2024-04-16T21:07:09.489957+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:07:09.489957+0000","last_undegraded":"2024-04-16T21:07:09.489957+0000","last_fullsized":"2024-04-16T21:07:09.489957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:03.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'705","reported_seq":2463,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493089+0000","last_change":"2024-04-16T19:25:13.470216+0000","last_active":"2024-04-16T21:07:29.493089+0000","last_peered":"2024-04-16T21:07:29.493089+0000","last_clean":"2024-04-16T21:07:29.493089+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:07:29.493089+0000","last_undegraded":"2024-04-16T21:07:29.493089+0000","last_fullsized":"2024-04-16T21:07:29.493089+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:40.891478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'311","reported_seq":572,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650437+0000","last_change":"2024-04-16T19:25:11.464746+0000","last_active":"2024-04-16T21:07:19.650437+0000","last_peered":"2024-04-16T21:07:19.650437+0000","last_clean":"2024-04-16T21:07:19.650437+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:07:19.650437+0000","last_undegraded":"2024-04-16T21:07:19.650437+0000","last_fullsized":"2024-04-16T21:07:19.650437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'6","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490772+0000","last_change":"2024-04-16T19:25:09.466128+0000","last_active":"2024-04-16T21:07:14.490772+0000","last_peered":"2024-04-16T21:07:14.490772+0000","last_clean":"2024-04-16T21:07:14.490772+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:07:14.490772+0000","last_undegraded":"2024-04-16T21:07:14.490772+0000","last_fullsized":"2024-04-16T21:07:14.490772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.502055+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:08:24.502055+0000","last_peered":"2024-04-16T21:08:24.502055+0000","last_clean":"2024-04-16T21:08:24.502055+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:08:24.502055+0000","last_undegraded":"2024-04-16T21:08:24.502055+0000","last_fullsized":"2024-04-16T21:08:24.502055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:20.204370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503320+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:08:34.503320+0000","last_peered":"2024-04-16T21:08:34.503320+0000","last_clean":"2024-04-16T21:08:34.503320+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:08:34.503320+0000","last_undegraded":"2024-04-16T21:08:34.503320+0000","last_fullsized":"2024-04-16T21:08:34.503320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:08:26.977348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661224+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:08:19.661224+0000","last_peered":"2024-04-16T21:08:19.661224+0000","last_clean":"2024-04-16T21:08:19.661224+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:08:19.661224+0000","last_undegraded":"2024-04-16T21:08:19.661224+0000","last_fullsized":"2024-04-16T21:08:19.661224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:45.288392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":359,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660463+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:08:19.660463+0000","last_peered":"2024-04-16T21:08:19.660463+0000","last_clean":"2024-04-16T21:08:19.660463+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:08:19.660463+0000","last_undegraded":"2024-04-16T21:08:19.660463+0000","last_fullsized":"2024-04-16T21:08:19.660463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:48.207837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661146+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:08:19.661146+0000","last_peered":"2024-04-16T21:08:19.661146+0000","last_clean":"2024-04-16T21:08:19.661146+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:08:19.661146+0000","last_undegraded":"2024-04-16T21:08:19.661146+0000","last_fullsized":"2024-04-16T21:08:19.661146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367334+0000","last_change":"2024-04-16T19:24:51.434704+0000","last_active":"2024-04-16T21:09:04.367334+0000","last_peered":"2024-04-16T21:09:04.367334+0000","last_clean":"2024-04-16T21:09:04.367334+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:09:04.367334+0000","last_undegraded":"2024-04-16T21:09:04.367334+0000","last_fullsized":"2024-04-16T21:09:04.367334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503573+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:08:34.503573+0000","last_peered":"2024-04-16T21:08:34.503573+0000","last_clean":"2024-04-16T21:08:34.503573+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:08:34.503573+0000","last_undegraded":"2024-04-16T21:08:34.503573+0000","last_fullsized":"2024-04-16T21:08:34.503573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:11.491071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'162","reported_seq":2095,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650676+0000","last_change":"2024-04-16T19:25:49.518828+0000","last_active":"2024-04-16T21:07:19.650676+0000","last_peered":"2024-04-16T21:07:19.650676+0000","last_clean":"2024-04-16T21:07:19.650676+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:07:19.650676+0000","last_undegraded":"2024-04-16T21:07:19.650676+0000","last_fullsized":"2024-04-16T21:07:19.650676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:22.736467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488302+0000","last_change":"2024-04-16T19:25:41.510329+0000","last_active":"2024-04-16T21:06:59.488302+0000","last_peered":"2024-04-16T21:06:59.488302+0000","last_clean":"2024-04-16T21:06:59.488302+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:06:59.488302+0000","last_undegraded":"2024-04-16T21:06:59.488302+0000","last_fullsized":"2024-04-16T21:06:59.488302+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:14.723662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'12","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365503+0000","last_change":"2024-04-16T19:25:15.480613+0000","last_active":"2024-04-16T21:08:49.365503+0000","last_peered":"2024-04-16T21:08:49.365503+0000","last_clean":"2024-04-16T21:08:49.365503+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:08:49.365503+0000","last_undegraded":"2024-04-16T21:08:49.365503+0000","last_fullsized":"2024-04-16T21:08:49.365503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491622+0000","last_change":"2024-04-16T19:25:13.469639+0000","last_active":"2024-04-16T21:07:19.491622+0000","last_peered":"2024-04-16T21:07:19.491622+0000","last_clean":"2024-04-16T21:07:19.491622+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:07:19.491622+0000","last_undegraded":"2024-04-16T21:07:19.491622+0000","last_fullsized":"2024-04-16T21:07:19.491622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:02.547322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'256","reported_seq":566,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364495+0000","last_change":"2024-04-16T19:25:11.463503+0000","last_active":"2024-04-16T21:08:44.364495+0000","last_peered":"2024-04-16T21:08:44.364495+0000","last_clean":"2024-04-16T21:08:44.364495+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:08:44.364495+0000","last_undegraded":"2024-04-16T21:08:44.364495+0000","last_fullsized":"2024-04-16T21:08:44.364495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:07.258240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'6","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509214+0000","last_change":"2024-04-16T19:25:09.462147+0000","last_active":"2024-04-16T21:09:09.509214+0000","last_peered":"2024-04-16T21:09:09.509214+0000","last_clean":"2024-04-16T21:09:09.509214+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:09:09.509214+0000","last_undegraded":"2024-04-16T21:09:09.509214+0000","last_fullsized":"2024-04-16T21:09:09.509214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369516+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:09:09.369516+0000","last_peered":"2024-04-16T21:09:09.369516+0000","last_clean":"2024-04-16T21:09:09.369516+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:09:09.369516+0000","last_undegraded":"2024-04-16T21:09:09.369516+0000","last_fullsized":"2024-04-16T21:09:09.369516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:22.844339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658852+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:08:04.658852+0000","last_peered":"2024-04-16T21:08:04.658852+0000","last_clean":"2024-04-16T21:08:04.658852+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:08:04.658852+0000","last_undegraded":"2024-04-16T21:08:04.658852+0000","last_fullsized":"2024-04-16T21:08:04.658852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657180+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:07:59.657180+0000","last_peered":"2024-04-16T21:07:59.657180+0000","last_clean":"2024-04-16T21:07:59.657180+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:07:59.657180+0000","last_undegraded":"2024-04-16T21:07:59.657180+0000","last_fullsized":"2024-04-16T21:07:59.657180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.363997+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:08:44.363997+0000","last_peered":"2024-04-16T21:08:44.363997+0000","last_clean":"2024-04-16T21:08:44.363997+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:08:44.363997+0000","last_undegraded":"2024-04-16T21:08:44.363997+0000","last_fullsized":"2024-04-16T21:08:44.363997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367212+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:08:59.367212+0000","last_peered":"2024-04-16T21:08:59.367212+0000","last_clean":"2024-04-16T21:08:59.367212+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:08:59.367212+0000","last_undegraded":"2024-04-16T21:08:59.367212+0000","last_fullsized":"2024-04-16T21:08:59.367212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:49.663208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365555+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:08:49.365555+0000","last_peered":"2024-04-16T21:08:49.365555+0000","last_clean":"2024-04-16T21:08:49.365555+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:08:49.365555+0000","last_undegraded":"2024-04-16T21:08:49.365555+0000","last_fullsized":"2024-04-16T21:08:49.365555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370592+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:08:49.370592+0000","last_peered":"2024-04-16T21:08:49.370592+0000","last_clean":"2024-04-16T21:08:49.370592+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:08:49.370592+0000","last_undegraded":"2024-04-16T21:08:49.370592+0000","last_fullsized":"2024-04-16T21:08:49.370592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366370+0000","last_change":"2024-04-16T19:24:51.442576+0000","last_active":"2024-04-16T21:08:54.366370+0000","last_peered":"2024-04-16T21:08:54.366370+0000","last_clean":"2024-04-16T21:08:54.366370+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:08:54.366370+0000","last_undegraded":"2024-04-16T21:08:54.366370+0000","last_fullsized":"2024-04-16T21:08:54.366370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662725+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:08:34.662725+0000","last_peered":"2024-04-16T21:08:34.662725+0000","last_clean":"2024-04-16T21:08:34.662725+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:08:34.662725+0000","last_undegraded":"2024-04-16T21:08:34.662725+0000","last_fullsized":"2024-04-16T21:08:34.662725+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:34:27.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1950,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647397+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:07:04.647397+0000","last_peered":"2024-04-16T21:07:04.647397+0000","last_clean":"2024-04-16T21:07:04.647397+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:07:04.647397+0000","last_undegraded":"2024-04-16T21:07:04.647397+0000","last_fullsized":"2024-04-16T21:07:04.647397+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:42.651587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649772+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:07:14.649772+0000","last_peered":"2024-04-16T21:07:14.649772+0000","last_clean":"2024-04-16T21:07:14.649772+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:07:14.649772+0000","last_undegraded":"2024-04-16T21:07:14.649772+0000","last_fullsized":"2024-04-16T21:07:14.649772+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'13","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370574+0000","last_change":"2024-04-16T19:25:15.485212+0000","last_active":"2024-04-16T21:08:49.370574+0000","last_peered":"2024-04-16T21:08:49.370574+0000","last_clean":"2024-04-16T21:08:49.370574+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:08:49.370574+0000","last_undegraded":"2024-04-16T21:08:49.370574+0000","last_fullsized":"2024-04-16T21:08:49.370574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:41.072415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'705","reported_seq":2076,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489124+0000","last_change":"2024-04-16T19:25:13.467216+0000","last_active":"2024-04-16T21:07:04.489124+0000","last_peered":"2024-04-16T21:07:04.489124+0000","last_clean":"2024-04-16T21:07:04.489124+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:07:04.489124+0000","last_undegraded":"2024-04-16T21:07:04.489124+0000","last_fullsized":"2024-04-16T21:07:04.489124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'306","reported_seq":614,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647255+0000","last_change":"2024-04-16T19:25:11.467288+0000","last_active":"2024-04-16T21:07:04.647255+0000","last_peered":"2024-04-16T21:07:04.647255+0000","last_clean":"2024-04-16T21:07:04.647255+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:07:04.647255+0000","last_undegraded":"2024-04-16T21:07:04.647255+0000","last_fullsized":"2024-04-16T21:07:04.647255+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'12","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662672+0000","last_change":"2024-04-16T19:25:09.464018+0000","last_active":"2024-04-16T21:08:39.662672+0000","last_peered":"2024-04-16T21:08:39.662672+0000","last_clean":"2024-04-16T21:08:39.662672+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:08:39.662672+0000","last_undegraded":"2024-04-16T21:08:39.662672+0000","last_fullsized":"2024-04-16T21:08:39.662672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:52.913431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662788+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:08:39.662788+0000","last_peered":"2024-04-16T21:08:39.662788+0000","last_clean":"2024-04-16T21:08:39.662788+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:08:39.662788+0000","last_undegraded":"2024-04-16T21:08:39.662788+0000","last_fullsized":"2024-04-16T21:08:39.662788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658875+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:08:04.658875+0000","last_peered":"2024-04-16T21:08:04.658875+0000","last_clean":"2024-04-16T21:08:04.658875+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:08:04.658875+0000","last_undegraded":"2024-04-16T21:08:04.658875+0000","last_fullsized":"2024-04-16T21:08:04.658875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:26.452728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369665+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:08:44.369665+0000","last_peered":"2024-04-16T21:08:44.369665+0000","last_clean":"2024-04-16T21:08:44.369665+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:08:44.369665+0000","last_undegraded":"2024-04-16T21:08:44.369665+0000","last_fullsized":"2024-04-16T21:08:44.369665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:23.643674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":591,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365570+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:08:49.365570+0000","last_peered":"2024-04-16T21:08:49.365570+0000","last_clean":"2024-04-16T21:08:49.365570+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:08:49.365570+0000","last_undegraded":"2024-04-16T21:08:49.365570+0000","last_fullsized":"2024-04-16T21:08:49.365570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:34.895652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366657+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:08:59.366657+0000","last_peered":"2024-04-16T21:08:59.366657+0000","last_clean":"2024-04-16T21:08:59.366657+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:08:59.366657+0000","last_undegraded":"2024-04-16T21:08:59.366657+0000","last_fullsized":"2024-04-16T21:08:59.366657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:28.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649142+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:07:09.649142+0000","last_peered":"2024-04-16T21:07:09.649142+0000","last_clean":"2024-04-16T21:07:09.649142+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:07:09.649142+0000","last_undegraded":"2024-04-16T21:07:09.649142+0000","last_fullsized":"2024-04-16T21:07:09.649142+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.152431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488247+0000","last_change":"2024-04-16T19:24:51.574513+0000","last_active":"2024-04-16T21:06:59.488247+0000","last_peered":"2024-04-16T21:06:59.488247+0000","last_clean":"2024-04-16T21:06:59.488247+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:06:59.488247+0000","last_undegraded":"2024-04-16T21:06:59.488247+0000","last_fullsized":"2024-04-16T21:06:59.488247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:32.268529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663021+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:08:34.663021+0000","last_peered":"2024-04-16T21:08:34.663021+0000","last_clean":"2024-04-16T21:08:34.663021+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:08:34.663021+0000","last_undegraded":"2024-04-16T21:08:34.663021+0000","last_fullsized":"2024-04-16T21:08:34.663021+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:37.701350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2368,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648112+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:07:04.648112+0000","last_peered":"2024-04-16T21:07:04.648112+0000","last_clean":"2024-04-16T21:07:04.648112+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:07:04.648112+0000","last_undegraded":"2024-04-16T21:07:04.648112+0000","last_fullsized":"2024-04-16T21:07:04.648112+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:11:19.505797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647141+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:06:59.647141+0000","last_peered":"2024-04-16T21:06:59.647141+0000","last_clean":"2024-04-16T21:06:59.647141+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:06:59.647141+0000","last_undegraded":"2024-04-16T21:06:59.647141+0000","last_fullsized":"2024-04-16T21:06:59.647141+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'16","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.646016+0000","last_change":"2024-04-16T19:25:15.484662+0000","last_active":"2024-04-16T21:06:49.646016+0000","last_peered":"2024-04-16T21:06:49.646016+0000","last_clean":"2024-04-16T21:06:49.646016+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:06:49.646016+0000","last_undegraded":"2024-04-16T21:06:49.646016+0000","last_fullsized":"2024-04-16T21:06:49.646016+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'20","reported_seq":971,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489057+0000","last_change":"2024-04-16T19:25:13.474689+0000","last_active":"2024-04-16T21:07:04.489057+0000","last_peered":"2024-04-16T21:07:04.489057+0000","last_clean":"2024-04-16T21:07:04.489057+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:07:04.489057+0000","last_undegraded":"2024-04-16T21:07:04.489057+0000","last_fullsized":"2024-04-16T21:07:04.489057+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:36.914573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'249","reported_seq":647,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648427+0000","last_change":"2024-04-16T19:25:11.460951+0000","last_active":"2024-04-16T21:07:04.648427+0000","last_peered":"2024-04-16T21:07:04.648427+0000","last_clean":"2024-04-16T21:07:04.648427+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:07:04.648427+0000","last_undegraded":"2024-04-16T21:07:04.648427+0000","last_fullsized":"2024-04-16T21:07:04.648427+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:42:59.386912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'7","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650231+0000","last_change":"2024-04-16T19:25:09.460368+0000","last_active":"2024-04-16T21:07:14.650231+0000","last_peered":"2024-04-16T21:07:14.650231+0000","last_clean":"2024-04-16T21:07:14.650231+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:07:14.650231+0000","last_undegraded":"2024-04-16T21:07:14.650231+0000","last_fullsized":"2024-04-16T21:07:14.650231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:19.738380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504660+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:08:39.504660+0000","last_peered":"2024-04-16T21:08:39.504660+0000","last_clean":"2024-04-16T21:08:39.504660+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:08:39.504660+0000","last_undegraded":"2024-04-16T21:08:39.504660+0000","last_fullsized":"2024-04-16T21:08:39.504660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.284891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497271+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:07:54.497271+0000","last_peered":"2024-04-16T21:07:54.497271+0000","last_clean":"2024-04-16T21:07:54.497271+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:07:54.497271+0000","last_undegraded":"2024-04-16T21:07:54.497271+0000","last_fullsized":"2024-04-16T21:07:54.497271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:27.416414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365627+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:08:54.365627+0000","last_peered":"2024-04-16T21:08:54.365627+0000","last_clean":"2024-04-16T21:08:54.365627+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:08:54.365627+0000","last_undegraded":"2024-04-16T21:08:54.365627+0000","last_fullsized":"2024-04-16T21:08:54.365627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:44.737149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506112+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:08:49.506112+0000","last_peered":"2024-04-16T21:08:49.506112+0000","last_clean":"2024-04-16T21:08:49.506112+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:08:49.506112+0000","last_undegraded":"2024-04-16T21:08:49.506112+0000","last_fullsized":"2024-04-16T21:08:49.506112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:32.935806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1512,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648886+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:07:09.648886+0000","last_peered":"2024-04-16T21:07:09.648886+0000","last_clean":"2024-04-16T21:07:09.648886+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:07:09.648886+0000","last_undegraded":"2024-04-16T21:07:09.648886+0000","last_fullsized":"2024-04-16T21:07:09.648886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372374+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:08:59.372374+0000","last_peered":"2024-04-16T21:08:59.372374+0000","last_clean":"2024-04-16T21:08:59.372374+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:08:59.372374+0000","last_undegraded":"2024-04-16T21:08:59.372374+0000","last_fullsized":"2024-04-16T21:08:59.372374+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660288+0000","last_change":"2024-04-16T19:24:51.438114+0000","last_active":"2024-04-16T21:08:24.660288+0000","last_peered":"2024-04-16T21:08:24.660288+0000","last_clean":"2024-04-16T21:08:24.660288+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:08:24.660288+0000","last_undegraded":"2024-04-16T21:08:24.660288+0000","last_fullsized":"2024-04-16T21:08:24.660288+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:59.734805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366544+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:08:49.366544+0000","last_peered":"2024-04-16T21:08:49.366544+0000","last_clean":"2024-04-16T21:08:49.366544+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:08:49.366544+0000","last_undegraded":"2024-04-16T21:08:49.366544+0000","last_fullsized":"2024-04-16T21:08:49.366544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:25:31.551960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1765,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648426+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:07:04.648426+0000","last_peered":"2024-04-16T21:07:04.648426+0000","last_clean":"2024-04-16T21:07:04.648426+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:07:04.648426+0000","last_undegraded":"2024-04-16T21:07:04.648426+0000","last_fullsized":"2024-04-16T21:07:04.648426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:10.736923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647564+0000","last_change":"2024-04-16T19:25:41.505274+0000","last_active":"2024-04-16T21:06:59.647564+0000","last_peered":"2024-04-16T21:06:59.647564+0000","last_clean":"2024-04-16T21:06:59.647564+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:06:59.647564+0000","last_undegraded":"2024-04-16T21:06:59.647564+0000","last_fullsized":"2024-04-16T21:06:59.647564+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:50:27.279669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'15","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370523+0000","last_change":"2024-04-16T19:25:15.487696+0000","last_active":"2024-04-16T21:08:49.370523+0000","last_peered":"2024-04-16T21:08:49.370523+0000","last_clean":"2024-04-16T21:08:49.370523+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:08:49.370523+0000","last_undegraded":"2024-04-16T21:08:49.370523+0000","last_fullsized":"2024-04-16T21:08:49.370523+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:34.149887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505292+0000","last_change":"2024-04-16T19:25:13.473305+0000","last_active":"2024-04-16T21:08:44.505292+0000","last_peered":"2024-04-16T21:08:44.505292+0000","last_clean":"2024-04-16T21:08:44.505292+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:08:44.505292+0000","last_undegraded":"2024-04-16T21:08:44.505292+0000","last_fullsized":"2024-04-16T21:08:44.505292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:38.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'690","reported_seq":1689,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:25:11.468818+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:21.906682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'16","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373858+0000","last_change":"2024-04-16T19:25:09.462529+0000","last_active":"2024-04-16T21:09:09.373858+0000","last_peered":"2024-04-16T21:09:09.373858+0000","last_clean":"2024-04-16T21:09:09.373858+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:09:09.373858+0000","last_undegraded":"2024-04-16T21:09:09.373858+0000","last_fullsized":"2024-04-16T21:09:09.373858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364926+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:08:49.364926+0000","last_peered":"2024-04-16T21:08:49.364926+0000","last_clean":"2024-04-16T21:08:49.364926+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:08:49.364926+0000","last_undegraded":"2024-04-16T21:08:49.364926+0000","last_fullsized":"2024-04-16T21:08:49.364926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:02.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657141+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:07:59.657141+0000","last_peered":"2024-04-16T21:07:59.657141+0000","last_clean":"2024-04-16T21:07:59.657141+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:07:59.657141+0000","last_undegraded":"2024-04-16T21:07:59.657141+0000","last_fullsized":"2024-04-16T21:07:59.657141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364646+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:08:44.364646+0000","last_peered":"2024-04-16T21:08:44.364646+0000","last_clean":"2024-04-16T21:08:44.364646+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:08:44.364646+0000","last_undegraded":"2024-04-16T21:08:44.364646+0000","last_fullsized":"2024-04-16T21:08:44.364646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:21.093051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648115+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:07:09.648115+0000","last_peered":"2024-04-16T21:07:09.648115+0000","last_clean":"2024-04-16T21:07:09.648115+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:07:09.648115+0000","last_undegraded":"2024-04-16T21:07:09.648115+0000","last_fullsized":"2024-04-16T21:07:09.648115+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:21.104930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364984+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:08:49.364984+0000","last_peered":"2024-04-16T21:08:49.364984+0000","last_clean":"2024-04-16T21:08:49.364984+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:08:49.364984+0000","last_undegraded":"2024-04-16T21:08:49.364984+0000","last_fullsized":"2024-04-16T21:08:49.364984+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:58.758206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366555+0000","last_change":"2024-04-16T19:24:49.432013+0000","last_active":"2024-04-16T21:08:49.366555+0000","last_peered":"2024-04-16T21:08:49.366555+0000","last_clean":"2024-04-16T21:08:49.366555+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:08:49.366555+0000","last_undegraded":"2024-04-16T21:08:49.366555+0000","last_fullsized":"2024-04-16T21:08:49.366555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366349+0000","last_change":"2024-04-16T19:24:51.436955+0000","last_active":"2024-04-16T21:08:54.366349+0000","last_peered":"2024-04-16T21:08:54.366349+0000","last_clean":"2024-04-16T21:08:54.366349+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:08:54.366349+0000","last_undegraded":"2024-04-16T21:08:54.366349+0000","last_fullsized":"2024-04-16T21:08:54.366349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654394+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:07:44.654394+0000","last_peered":"2024-04-16T21:07:44.654394+0000","last_clean":"2024-04-16T21:07:44.654394+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:07:44.654394+0000","last_undegraded":"2024-04-16T21:07:44.654394+0000","last_fullsized":"2024-04-16T21:07:44.654394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:57:15.042193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652729+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:07:39.652729+0000","last_peered":"2024-04-16T21:07:39.652729+0000","last_clean":"2024-04-16T21:07:39.652729+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:07:39.652729+0000","last_undegraded":"2024-04-16T21:07:39.652729+0000","last_fullsized":"2024-04-16T21:07:39.652729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:02.023550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.647634+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:06:59.647634+0000","last_peered":"2024-04-16T21:06:59.647634+0000","last_clean":"2024-04-16T21:06:59.647634+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:06:59.647634+0000","last_undegraded":"2024-04-16T21:06:59.647634+0000","last_fullsized":"2024-04-16T21:06:59.647634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'3","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492346+0000","last_change":"2024-04-16T19:25:15.483924+0000","last_active":"2024-04-16T21:07:24.492346+0000","last_peered":"2024-04-16T21:07:24.492346+0000","last_clean":"2024-04-16T21:07:24.492346+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:07:24.492346+0000","last_undegraded":"2024-04-16T21:07:24.492346+0000","last_fullsized":"2024-04-16T21:07:24.492346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'705","reported_seq":2159,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505330+0000","last_change":"2024-04-16T19:25:13.475694+0000","last_active":"2024-04-16T21:08:44.505330+0000","last_peered":"2024-04-16T21:08:44.505330+0000","last_clean":"2024-04-16T21:08:44.505330+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:08:44.505330+0000","last_undegraded":"2024-04-16T21:08:44.505330+0000","last_fullsized":"2024-04-16T21:08:44.505330+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:56.172757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'247","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364687+0000","last_change":"2024-04-16T19:25:11.466002+0000","last_active":"2024-04-16T21:08:44.364687+0000","last_peered":"2024-04-16T21:08:44.364687+0000","last_clean":"2024-04-16T21:08:44.364687+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:08:44.364687+0000","last_undegraded":"2024-04-16T21:08:44.364687+0000","last_fullsized":"2024-04-16T21:08:44.364687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:38.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'27","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367274+0000","last_change":"2024-04-16T19:25:09.469480+0000","last_active":"2024-04-16T21:08:59.367274+0000","last_peered":"2024-04-16T21:08:59.367274+0000","last_clean":"2024-04-16T21:08:59.367274+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:08:59.367274+0000","last_undegraded":"2024-04-16T21:08:59.367274+0000","last_fullsized":"2024-04-16T21:08:59.367274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:00.355585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364874+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:08:49.364874+0000","last_peered":"2024-04-16T21:08:49.364874+0000","last_clean":"2024-04-16T21:08:49.364874+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:08:49.364874+0000","last_undegraded":"2024-04-16T21:08:49.364874+0000","last_fullsized":"2024-04-16T21:08:49.364874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:25.878014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368069+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:09:04.368069+0000","last_peered":"2024-04-16T21:09:04.368069+0000","last_clean":"2024-04-16T21:09:04.368069+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:09:04.368069+0000","last_undegraded":"2024-04-16T21:09:04.368069+0000","last_fullsized":"2024-04-16T21:09:04.368069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498140+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:07:59.498140+0000","last_peered":"2024-04-16T21:07:59.498140+0000","last_clean":"2024-04-16T21:07:59.498140+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:07:59.498140+0000","last_undegraded":"2024-04-16T21:07:59.498140+0000","last_fullsized":"2024-04-16T21:07:59.498140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372342+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:08:59.372342+0000","last_peered":"2024-04-16T21:08:59.372342+0000","last_clean":"2024-04-16T21:08:59.372342+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:08:59.372342+0000","last_undegraded":"2024-04-16T21:08:59.372342+0000","last_fullsized":"2024-04-16T21:08:59.372342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:38.135155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":349,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373804+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:09:09.373804+0000","last_peered":"2024-04-16T21:09:09.373804+0000","last_clean":"2024-04-16T21:09:09.373804+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:09:09.373804+0000","last_undegraded":"2024-04-16T21:09:09.373804+0000","last_fullsized":"2024-04-16T21:09:09.373804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367113+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:08:59.367113+0000","last_peered":"2024-04-16T21:08:59.367113+0000","last_clean":"2024-04-16T21:08:59.367113+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:08:59.367113+0000","last_undegraded":"2024-04-16T21:08:59.367113+0000","last_fullsized":"2024-04-16T21:08:59.367113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:50.295932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660391+0000","last_change":"2024-04-16T19:24:51.572391+0000","last_active":"2024-04-16T21:08:24.660391+0000","last_peered":"2024-04-16T21:08:24.660391+0000","last_clean":"2024-04-16T21:08:24.660391+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:08:24.660391+0000","last_undegraded":"2024-04-16T21:08:24.660391+0000","last_fullsized":"2024-04-16T21:08:24.660391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:04.095435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495516+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:07:44.495516+0000","last_peered":"2024-04-16T21:07:44.495516+0000","last_clean":"2024-04-16T21:07:44.495516+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:07:44.495516+0000","last_undegraded":"2024-04-16T21:07:44.495516+0000","last_fullsized":"2024-04-16T21:07:44.495516+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:34.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2002,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653496+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:07:39.653496+0000","last_peered":"2024-04-16T21:07:39.653496+0000","last_clean":"2024-04-16T21:07:39.653496+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:07:39.653496+0000","last_undegraded":"2024-04-16T21:07:39.653496+0000","last_fullsized":"2024-04-16T21:07:39.653496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:04:06.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496500+0000","last_change":"2024-04-16T19:25:41.510087+0000","last_active":"2024-04-16T21:07:49.496500+0000","last_peered":"2024-04-16T21:07:49.496500+0000","last_clean":"2024-04-16T21:07:49.496500+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:07:49.496500+0000","last_undegraded":"2024-04-16T21:07:49.496500+0000","last_fullsized":"2024-04-16T21:07:49.496500+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:45.367111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'23","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370491+0000","last_change":"2024-04-16T19:25:15.487252+0000","last_active":"2024-04-16T21:08:49.370491+0000","last_peered":"2024-04-16T21:08:49.370491+0000","last_clean":"2024-04-16T21:08:49.370491+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:08:49.370491+0000","last_undegraded":"2024-04-16T21:08:49.370491+0000","last_fullsized":"2024-04-16T21:08:49.370491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:14.101479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364063+0000","last_change":"2024-04-16T19:25:13.475862+0000","last_active":"2024-04-16T21:08:44.364063+0000","last_peered":"2024-04-16T21:08:44.364063+0000","last_clean":"2024-04-16T21:08:44.364063+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:08:44.364063+0000","last_undegraded":"2024-04-16T21:08:44.364063+0000","last_fullsized":"2024-04-16T21:08:44.364063+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:47.410023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'714","reported_seq":1574,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364715+0000","last_change":"2024-04-16T19:25:11.465490+0000","last_active":"2024-04-16T21:08:44.364715+0000","last_peered":"2024-04-16T21:08:44.364715+0000","last_clean":"2024-04-16T21:08:44.364715+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:08:44.364715+0000","last_undegraded":"2024-04-16T21:08:44.364715+0000","last_fullsized":"2024-04-16T21:08:44.364715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:03.668322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'6","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664395+0000","last_change":"2024-04-16T19:25:09.463391+0000","last_active":"2024-04-16T21:08:39.664395+0000","last_peered":"2024-04-16T21:08:39.664395+0000","last_clean":"2024-04-16T21:08:39.664395+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:08:39.664395+0000","last_undegraded":"2024-04-16T21:08:39.664395+0000","last_fullsized":"2024-04-16T21:08:39.664395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370504+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:08:49.370504+0000","last_peered":"2024-04-16T21:08:49.370504+0000","last_clean":"2024-04-16T21:08:49.370504+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:08:49.370504+0000","last_undegraded":"2024-04-16T21:08:49.370504+0000","last_fullsized":"2024-04-16T21:08:49.370504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364075+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:08:44.364075+0000","last_peered":"2024-04-16T21:08:44.364075+0000","last_clean":"2024-04-16T21:08:44.364075+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:08:44.364075+0000","last_undegraded":"2024-04-16T21:08:44.364075+0000","last_fullsized":"2024-04-16T21:08:44.364075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660502+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:08:14.660502+0000","last_peered":"2024-04-16T21:08:14.660502+0000","last_clean":"2024-04-16T21:08:14.660502+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:08:14.660502+0000","last_undegraded":"2024-04-16T21:08:14.660502+0000","last_fullsized":"2024-04-16T21:08:14.660502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:07.162476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367144+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:08:59.367144+0000","last_peered":"2024-04-16T21:08:59.367144+0000","last_clean":"2024-04-16T21:08:59.367144+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:08:59.367144+0000","last_undegraded":"2024-04-16T21:08:59.367144+0000","last_fullsized":"2024-04-16T21:08:59.367144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:27.238344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":346,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366566+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:08:59.366566+0000","last_peered":"2024-04-16T21:08:59.366566+0000","last_clean":"2024-04-16T21:08:59.366566+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:08:59.366566+0000","last_undegraded":"2024-04-16T21:08:59.366566+0000","last_fullsized":"2024-04-16T21:08:59.366566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:36.905847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1317,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368814+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:09:09.368814+0000","last_peered":"2024-04-16T21:09:09.368814+0000","last_clean":"2024-04-16T21:09:09.368814+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:09:09.368814+0000","last_undegraded":"2024-04-16T21:09:09.368814+0000","last_fullsized":"2024-04-16T21:09:09.368814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:34.836984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662820+0000","last_change":"2024-04-16T19:24:51.572638+0000","last_active":"2024-04-16T21:08:39.662820+0000","last_peered":"2024-04-16T21:08:39.662820+0000","last_clean":"2024-04-16T21:08:39.662820+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:08:39.662820+0000","last_undegraded":"2024-04-16T21:08:39.662820+0000","last_fullsized":"2024-04-16T21:08:39.662820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:13.926597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503462+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:08:34.503462+0000","last_peered":"2024-04-16T21:08:34.503462+0000","last_clean":"2024-04-16T21:08:34.503462+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:08:34.503462+0000","last_undegraded":"2024-04-16T21:08:34.503462+0000","last_fullsized":"2024-04-16T21:08:34.503462+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:20.743689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1526,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656812+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:07:54.656812+0000","last_peered":"2024-04-16T21:07:54.656812+0000","last_clean":"2024-04-16T21:07:54.656812+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:07:54.656812+0000","last_undegraded":"2024-04-16T21:07:54.656812+0000","last_fullsized":"2024-04-16T21:07:54.656812+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:25.577697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652730+0000","last_change":"2024-04-16T19:25:41.498506+0000","last_active":"2024-04-16T21:07:34.652730+0000","last_peered":"2024-04-16T21:07:34.652730+0000","last_clean":"2024-04-16T21:07:34.652730+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:07:34.652730+0000","last_undegraded":"2024-04-16T21:07:34.652730+0000","last_fullsized":"2024-04-16T21:07:34.652730+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:14.970337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'2","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489802+0000","last_change":"2024-04-16T19:25:15.483677+0000","last_active":"2024-04-16T21:07:09.489802+0000","last_peered":"2024-04-16T21:07:09.489802+0000","last_clean":"2024-04-16T21:07:09.489802+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:07:09.489802+0000","last_undegraded":"2024-04-16T21:07:09.489802+0000","last_fullsized":"2024-04-16T21:07:09.489802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368793+0000","last_change":"2024-04-16T19:25:13.473234+0000","last_active":"2024-04-16T21:09:04.368793+0000","last_peered":"2024-04-16T21:09:04.368793+0000","last_clean":"2024-04-16T21:09:04.368793+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:09:04.368793+0000","last_undegraded":"2024-04-16T21:09:04.368793+0000","last_fullsized":"2024-04-16T21:09:04.368793+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:10.112989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'267","reported_seq":524,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364765+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:08:44.364765+0000","last_peered":"2024-04-16T21:08:44.364765+0000","last_clean":"2024-04-16T21:08:44.364765+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:08:44.364765+0000","last_undegraded":"2024-04-16T21:08:44.364765+0000","last_fullsized":"2024-04-16T21:08:44.364765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:51.610204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'14","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664322+0000","last_change":"2024-04-16T19:25:09.461719+0000","last_active":"2024-04-16T21:08:39.664322+0000","last_peered":"2024-04-16T21:08:39.664322+0000","last_clean":"2024-04-16T21:08:39.664322+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:08:39.664322+0000","last_undegraded":"2024-04-16T21:08:39.664322+0000","last_fullsized":"2024-04-16T21:08:39.664322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:24.102149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372250+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:08:59.372250+0000","last_peered":"2024-04-16T21:08:59.372250+0000","last_clean":"2024-04-16T21:08:59.372250+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:08:59.372250+0000","last_undegraded":"2024-04-16T21:08:59.372250+0000","last_fullsized":"2024-04-16T21:08:59.372250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503593+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:08:34.503593+0000","last_peered":"2024-04-16T21:08:34.503593+0000","last_clean":"2024-04-16T21:08:34.503593+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:08:34.503593+0000","last_undegraded":"2024-04-16T21:08:34.503593+0000","last_fullsized":"2024-04-16T21:08:34.503593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495661+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:07:44.495661+0000","last_peered":"2024-04-16T21:07:44.495661+0000","last_clean":"2024-04-16T21:07:44.495661+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:07:44.495661+0000","last_undegraded":"2024-04-16T21:07:44.495661+0000","last_fullsized":"2024-04-16T21:07:44.495661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367951+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:08:59.367951+0000","last_peered":"2024-04-16T21:08:59.367951+0000","last_clean":"2024-04-16T21:08:59.367951+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:08:59.367951+0000","last_undegraded":"2024-04-16T21:08:59.367951+0000","last_fullsized":"2024-04-16T21:08:59.367951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:48.749243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367980+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:08:59.367980+0000","last_peered":"2024-04-16T21:08:59.367980+0000","last_clean":"2024-04-16T21:08:59.367980+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:08:59.367980+0000","last_undegraded":"2024-04-16T21:08:59.367980+0000","last_fullsized":"2024-04-16T21:08:59.367980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372321+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:08:59.372321+0000","last_peered":"2024-04-16T21:08:59.372321+0000","last_clean":"2024-04-16T21:08:59.372321+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:08:59.372321+0000","last_undegraded":"2024-04-16T21:08:59.372321+0000","last_fullsized":"2024-04-16T21:08:59.372321+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660333+0000","last_change":"2024-04-16T19:24:51.573460+0000","last_active":"2024-04-16T21:08:24.660333+0000","last_peered":"2024-04-16T21:08:24.660333+0000","last_clean":"2024-04-16T21:08:24.660333+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:08:24.660333+0000","last_undegraded":"2024-04-16T21:08:24.660333+0000","last_fullsized":"2024-04-16T21:08:24.660333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:31.585745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498095+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:07:59.498095+0000","last_peered":"2024-04-16T21:07:59.498095+0000","last_clean":"2024-04-16T21:07:59.498095+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:07:59.498095+0000","last_undegraded":"2024-04-16T21:07:59.498095+0000","last_fullsized":"2024-04-16T21:07:59.498095+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:33:27.173018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1521,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657201+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:07:59.657201+0000","last_peered":"2024-04-16T21:07:59.657201+0000","last_clean":"2024-04-16T21:07:59.657201+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:07:59.657201+0000","last_undegraded":"2024-04-16T21:07:59.657201+0000","last_fullsized":"2024-04-16T21:07:59.657201+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:16.733301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.493948+0000","last_change":"2024-04-16T19:25:41.502910+0000","last_active":"2024-04-16T21:07:34.493948+0000","last_peered":"2024-04-16T21:07:34.493948+0000","last_clean":"2024-04-16T21:07:34.493948+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:07:34.493948+0000","last_undegraded":"2024-04-16T21:07:34.493948+0000","last_fullsized":"2024-04-16T21:07:34.493948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:25.621893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'12","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489876+0000","last_change":"2024-04-16T19:25:15.485965+0000","last_active":"2024-04-16T21:07:09.489876+0000","last_peered":"2024-04-16T21:07:09.489876+0000","last_clean":"2024-04-16T21:07:09.489876+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:07:09.489876+0000","last_undegraded":"2024-04-16T21:07:09.489876+0000","last_fullsized":"2024-04-16T21:07:09.489876+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'705","reported_seq":2080,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364751+0000","last_change":"2024-04-16T19:25:13.471357+0000","last_active":"2024-04-16T21:08:44.364751+0000","last_peered":"2024-04-16T21:08:44.364751+0000","last_clean":"2024-04-16T21:08:44.364751+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:08:44.364751+0000","last_undegraded":"2024-04-16T21:08:44.364751+0000","last_fullsized":"2024-04-16T21:08:44.364751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'318","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368826+0000","last_change":"2024-04-16T19:25:11.461881+0000","last_active":"2024-04-16T21:09:04.368826+0000","last_peered":"2024-04-16T21:09:04.368826+0000","last_clean":"2024-04-16T21:09:04.368826+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:09:04.368826+0000","last_undegraded":"2024-04-16T21:09:04.368826+0000","last_fullsized":"2024-04-16T21:09:04.368826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:13.030219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'6","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504516+0000","last_change":"2024-04-16T19:25:09.461256+0000","last_active":"2024-04-16T21:08:39.504516+0000","last_peered":"2024-04-16T21:08:39.504516+0000","last_clean":"2024-04-16T21:08:39.504516+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:08:39.504516+0000","last_undegraded":"2024-04-16T21:08:39.504516+0000","last_fullsized":"2024-04-16T21:08:39.504516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:56.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365397+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:08:49.365397+0000","last_peered":"2024-04-16T21:08:49.365397+0000","last_clean":"2024-04-16T21:08:49.365397+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:08:49.365397+0000","last_undegraded":"2024-04-16T21:08:49.365397+0000","last_fullsized":"2024-04-16T21:08:49.365397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:21.322076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662064+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:08:34.662064+0000","last_peered":"2024-04-16T21:08:34.662064+0000","last_clean":"2024-04-16T21:08:34.662064+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:08:34.662064+0000","last_undegraded":"2024-04-16T21:08:34.662064+0000","last_fullsized":"2024-04-16T21:08:34.662064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.656111+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:07:49.656111+0000","last_peered":"2024-04-16T21:07:49.656111+0000","last_clean":"2024-04-16T21:07:49.656111+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:07:49.656111+0000","last_undegraded":"2024-04-16T21:07:49.656111+0000","last_fullsized":"2024-04-16T21:07:49.656111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:42.139799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508363+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:09:04.508363+0000","last_peered":"2024-04-16T21:09:04.508363+0000","last_clean":"2024-04-16T21:09:04.508363+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:09:04.508363+0000","last_undegraded":"2024-04-16T21:09:04.508363+0000","last_fullsized":"2024-04-16T21:09:04.508363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:28:03.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":331,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367176+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:08:59.367176+0000","last_peered":"2024-04-16T21:08:59.367176+0000","last_clean":"2024-04-16T21:08:59.367176+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:08:59.367176+0000","last_undegraded":"2024-04-16T21:08:59.367176+0000","last_fullsized":"2024-04-16T21:08:59.367176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:23.723705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507550+0000","last_change":"2024-04-16T19:24:49.435265+0000","last_active":"2024-04-16T21:08:59.507550+0000","last_peered":"2024-04-16T21:08:59.507550+0000","last_clean":"2024-04-16T21:08:59.507550+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:08:59.507550+0000","last_undegraded":"2024-04-16T21:08:59.507550+0000","last_fullsized":"2024-04-16T21:08:59.507550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:08.160472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661329+0000","last_change":"2024-04-16T19:24:51.438071+0000","last_active":"2024-04-16T21:08:24.661329+0000","last_peered":"2024-04-16T21:08:24.661329+0000","last_clean":"2024-04-16T21:08:24.661329+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:08:24.661329+0000","last_undegraded":"2024-04-16T21:08:24.661329+0000","last_fullsized":"2024-04-16T21:08:24.661329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:42.181631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649582+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:07:19.649582+0000","last_peered":"2024-04-16T21:07:19.649582+0000","last_clean":"2024-04-16T21:07:19.649582+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:07:19.649582+0000","last_undegraded":"2024-04-16T21:07:19.649582+0000","last_fullsized":"2024-04-16T21:07:19.649582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:34:01.244792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653404+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:07:34.653404+0000","last_peered":"2024-04-16T21:07:34.653404+0000","last_clean":"2024-04-16T21:07:34.653404+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:07:34.653404+0000","last_undegraded":"2024-04-16T21:07:34.653404+0000","last_fullsized":"2024-04-16T21:07:34.653404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:03.502015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.500263+0000","last_change":"2024-04-16T19:25:41.506596+0000","last_active":"2024-04-16T21:08:14.500263+0000","last_peered":"2024-04-16T21:08:14.500263+0000","last_clean":"2024-04-16T21:08:14.500263+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:08:14.500263+0000","last_undegraded":"2024-04-16T21:08:14.500263+0000","last_fullsized":"2024-04-16T21:08:14.500263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:01.639245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'4","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655215+0000","last_change":"2024-04-16T19:25:15.480477+0000","last_active":"2024-04-16T21:07:49.655215+0000","last_peered":"2024-04-16T21:07:49.655215+0000","last_clean":"2024-04-16T21:07:49.655215+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:07:49.655215+0000","last_undegraded":"2024-04-16T21:07:49.655215+0000","last_fullsized":"2024-04-16T21:07:49.655215+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654540+0000","last_change":"2024-04-16T19:25:13.470985+0000","last_active":"2024-04-16T21:07:44.654540+0000","last_peered":"2024-04-16T21:07:44.654540+0000","last_clean":"2024-04-16T21:07:44.654540+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:07:44.654540+0000","last_undegraded":"2024-04-16T21:07:44.654540+0000","last_fullsized":"2024-04-16T21:07:44.654540+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:54.589907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'292","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654437+0000","last_change":"2024-04-16T19:25:11.464321+0000","last_active":"2024-04-16T21:07:44.654437+0000","last_peered":"2024-04-16T21:07:44.654437+0000","last_clean":"2024-04-16T21:07:44.654437+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:07:44.654437+0000","last_undegraded":"2024-04-16T21:07:44.654437+0000","last_fullsized":"2024-04-16T21:07:44.654437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:17.793032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'13","reported_seq":236,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650531+0000","last_change":"2024-04-16T19:25:09.460391+0000","last_active":"2024-04-16T21:07:24.650531+0000","last_peered":"2024-04-16T21:07:24.650531+0000","last_clean":"2024-04-16T21:07:24.650531+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:07:24.650531+0000","last_undegraded":"2024-04-16T21:07:24.650531+0000","last_fullsized":"2024-04-16T21:07:24.650531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373217+0000","last_change":"2024-04-16T19:24:51.434685+0000","last_active":"2024-04-16T21:09:04.373217+0000","last_peered":"2024-04-16T21:09:04.373217+0000","last_clean":"2024-04-16T21:09:04.373217+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:09:04.373217+0000","last_undegraded":"2024-04-16T21:09:04.373217+0000","last_fullsized":"2024-04-16T21:09:04.373217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:57:30.733796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650563+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:07:19.650563+0000","last_peered":"2024-04-16T21:07:19.650563+0000","last_clean":"2024-04-16T21:07:19.650563+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:07:19.650563+0000","last_undegraded":"2024-04-16T21:07:19.650563+0000","last_fullsized":"2024-04-16T21:07:19.650563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:28.749905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504494+0000","last_change":"2024-04-16T19:25:41.504766+0000","last_active":"2024-04-16T21:08:39.504494+0000","last_peered":"2024-04-16T21:08:39.504494+0000","last_clean":"2024-04-16T21:08:39.504494+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:08:39.504494+0000","last_undegraded":"2024-04-16T21:08:39.504494+0000","last_fullsized":"2024-04-16T21:08:39.504494+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:37.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":743,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493189+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:07:29.493189+0000","last_peered":"2024-04-16T21:07:29.493189+0000","last_clean":"2024-04-16T21:07:29.493189+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:07:29.493189+0000","last_undegraded":"2024-04-16T21:07:29.493189+0000","last_fullsized":"2024-04-16T21:07:29.493189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:30.438117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493117+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:07:29.493117+0000","last_peered":"2024-04-16T21:07:29.493117+0000","last_clean":"2024-04-16T21:07:29.493117+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:07:29.493117+0000","last_undegraded":"2024-04-16T21:07:29.493117+0000","last_fullsized":"2024-04-16T21:07:29.493117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:51.673404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493130+0000","last_change":"2024-04-16T19:24:49.434342+0000","last_active":"2024-04-16T21:07:29.493130+0000","last_peered":"2024-04-16T21:07:29.493130+0000","last_clean":"2024-04-16T21:07:29.493130+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:07:29.493130+0000","last_undegraded":"2024-04-16T21:07:29.493130+0000","last_fullsized":"2024-04-16T21:07:29.493130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:07.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651977+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:07:29.651977+0000","last_peered":"2024-04-16T21:07:29.651977+0000","last_clean":"2024-04-16T21:07:29.651977+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:07:29.651977+0000","last_undegraded":"2024-04-16T21:07:29.651977+0000","last_fullsized":"2024-04-16T21:07:29.651977+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:21.199802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'118","reported_seq":1711,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491510+0000","last_change":"2024-04-16T19:25:49.520399+0000","last_active":"2024-04-16T21:07:19.491510+0000","last_peered":"2024-04-16T21:07:19.491510+0000","last_clean":"2024-04-16T21:07:19.491510+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:07:19.491510+0000","last_undegraded":"2024-04-16T21:07:19.491510+0000","last_fullsized":"2024-04-16T21:07:19.491510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:16:26.165977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659512+0000","last_change":"2024-04-16T19:25:41.505754+0000","last_active":"2024-04-16T21:08:14.659512+0000","last_peered":"2024-04-16T21:08:14.659512+0000","last_clean":"2024-04-16T21:08:14.659512+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:08:14.659512+0000","last_undegraded":"2024-04-16T21:08:14.659512+0000","last_fullsized":"2024-04-16T21:08:14.659512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:10.716421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655168+0000","last_change":"2024-04-16T19:25:15.482484+0000","last_active":"2024-04-16T21:07:49.655168+0000","last_peered":"2024-04-16T21:07:49.655168+0000","last_clean":"2024-04-16T21:07:49.655168+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:07:49.655168+0000","last_undegraded":"2024-04-16T21:07:49.655168+0000","last_fullsized":"2024-04-16T21:07:49.655168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:25.761792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655229+0000","last_change":"2024-04-16T19:25:13.472620+0000","last_active":"2024-04-16T21:07:44.655229+0000","last_peered":"2024-04-16T21:07:44.655229+0000","last_clean":"2024-04-16T21:07:44.655229+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:07:44.655229+0000","last_undegraded":"2024-04-16T21:07:44.655229+0000","last_fullsized":"2024-04-16T21:07:44.655229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:28.935744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1140","reported_seq":2898,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654522+0000","last_change":"2024-04-16T19:25:11.462713+0000","last_active":"2024-04-16T21:07:44.654522+0000","last_peered":"2024-04-16T21:07:44.654522+0000","last_clean":"2024-04-16T21:07:44.654522+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:07:44.654522+0000","last_undegraded":"2024-04-16T21:07:44.654522+0000","last_fullsized":"2024-04-16T21:07:44.654522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:45.401459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'9","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490903+0000","last_change":"2024-04-16T19:25:09.466079+0000","last_active":"2024-04-16T21:07:14.490903+0000","last_peered":"2024-04-16T21:07:14.490903+0000","last_clean":"2024-04-16T21:07:14.490903+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:07:14.490903+0000","last_undegraded":"2024-04-16T21:07:14.490903+0000","last_fullsized":"2024-04-16T21:07:14.490903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369024+0000","last_change":"2024-04-16T19:24:51.442540+0000","last_active":"2024-04-16T21:09:14.369024+0000","last_peered":"2024-04-16T21:09:14.369024+0000","last_clean":"2024-04-16T21:09:14.369024+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:09:14.369024+0000","last_undegraded":"2024-04-16T21:09:14.369024+0000","last_fullsized":"2024-04-16T21:09:14.369024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491649+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:07:19.491649+0000","last_peered":"2024-04-16T21:07:19.491649+0000","last_clean":"2024-04-16T21:07:19.491649+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:07:19.491649+0000","last_undegraded":"2024-04-16T21:07:19.491649+0000","last_fullsized":"2024-04-16T21:07:19.491649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662604+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:08:29.662604+0000","last_peered":"2024-04-16T21:08:29.662604+0000","last_clean":"2024-04-16T21:08:29.662604+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:08:29.662604+0000","last_undegraded":"2024-04-16T21:08:29.662604+0000","last_fullsized":"2024-04-16T21:08:29.662604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493228+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:07:29.493228+0000","last_peered":"2024-04-16T21:07:29.493228+0000","last_clean":"2024-04-16T21:07:29.493228+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:07:29.493228+0000","last_undegraded":"2024-04-16T21:07:29.493228+0000","last_fullsized":"2024-04-16T21:07:29.493228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:10.368115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651116+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:07:29.651116+0000","last_peered":"2024-04-16T21:07:29.651116+0000","last_clean":"2024-04-16T21:07:29.651116+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:07:29.651116+0000","last_undegraded":"2024-04-16T21:07:29.651116+0000","last_fullsized":"2024-04-16T21:07:29.651116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:05.198319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652671+0000","last_change":"2024-04-16T19:24:49.431431+0000","last_active":"2024-04-16T21:07:29.652671+0000","last_peered":"2024-04-16T21:07:29.652671+0000","last_clean":"2024-04-16T21:07:29.652671+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:07:29.652671+0000","last_undegraded":"2024-04-16T21:07:29.652671+0000","last_fullsized":"2024-04-16T21:07:29.652671+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652772+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:07:34.652772+0000","last_peered":"2024-04-16T21:07:34.652772+0000","last_clean":"2024-04-16T21:07:34.652772+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:07:34.652772+0000","last_undegraded":"2024-04-16T21:07:34.652772+0000","last_fullsized":"2024-04-16T21:07:34.652772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:31:39.104319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1703,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501136+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:08:19.501136+0000","last_peered":"2024-04-16T21:08:19.501136+0000","last_clean":"2024-04-16T21:08:19.501136+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:08:19.501136+0000","last_undegraded":"2024-04-16T21:08:19.501136+0000","last_fullsized":"2024-04-16T21:08:19.501136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:15.107893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649480+0000","last_change":"2024-04-16T19:25:41.499274+0000","last_active":"2024-04-16T21:07:14.649480+0000","last_peered":"2024-04-16T21:07:14.649480+0000","last_clean":"2024-04-16T21:07:14.649480+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:07:14.649480+0000","last_undegraded":"2024-04-16T21:07:14.649480+0000","last_fullsized":"2024-04-16T21:07:14.649480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:22:55.194832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'20","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647308+0000","last_change":"2024-04-16T19:25:15.485235+0000","last_active":"2024-04-16T21:07:04.647308+0000","last_peered":"2024-04-16T21:07:04.647308+0000","last_clean":"2024-04-16T21:07:04.647308+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:07:04.647308+0000","last_undegraded":"2024-04-16T21:07:04.647308+0000","last_fullsized":"2024-04-16T21:07:04.647308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652131+0000","last_change":"2024-04-16T19:25:13.476041+0000","last_active":"2024-04-16T21:07:29.652131+0000","last_peered":"2024-04-16T21:07:29.652131+0000","last_clean":"2024-04-16T21:07:29.652131+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:07:29.652131+0000","last_undegraded":"2024-04-16T21:07:29.652131+0000","last_fullsized":"2024-04-16T21:07:29.652131+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:17.872253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'274","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651118+0000","last_change":"2024-04-16T19:25:11.462598+0000","last_active":"2024-04-16T21:07:19.651118+0000","last_peered":"2024-04-16T21:07:19.651118+0000","last_clean":"2024-04-16T21:07:19.651118+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:07:19.651118+0000","last_undegraded":"2024-04-16T21:07:19.651118+0000","last_fullsized":"2024-04-16T21:07:19.651118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:55.581849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'8","reported_seq":245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652902+0000","last_change":"2024-04-16T19:25:09.461313+0000","last_active":"2024-04-16T21:07:39.652902+0000","last_peered":"2024-04-16T21:07:39.652902+0000","last_clean":"2024-04-16T21:07:39.652902+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:07:39.652902+0000","last_undegraded":"2024-04-16T21:07:39.652902+0000","last_fullsized":"2024-04-16T21:07:39.652902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650513+0000","last_change":"2024-04-16T19:24:51.435662+0000","last_active":"2024-04-16T21:07:24.650513+0000","last_peered":"2024-04-16T21:07:24.650513+0000","last_clean":"2024-04-16T21:07:24.650513+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:07:24.650513+0000","last_undegraded":"2024-04-16T21:07:24.650513+0000","last_fullsized":"2024-04-16T21:07:24.650513+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:57.768573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491559+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:07:19.491559+0000","last_peered":"2024-04-16T21:07:19.491559+0000","last_clean":"2024-04-16T21:07:19.491559+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:07:19.491559+0000","last_undegraded":"2024-04-16T21:07:19.491559+0000","last_fullsized":"2024-04-16T21:07:19.491559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:46:59.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662208+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:08:29.662208+0000","last_peered":"2024-04-16T21:08:29.662208+0000","last_clean":"2024-04-16T21:08:29.662208+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:08:29.662208+0000","last_undegraded":"2024-04-16T21:08:29.662208+0000","last_fullsized":"2024-04-16T21:08:29.662208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":611,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651931+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:07:29.651931+0000","last_peered":"2024-04-16T21:07:29.651931+0000","last_clean":"2024-04-16T21:07:29.651931+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:07:29.651931+0000","last_undegraded":"2024-04-16T21:07:29.651931+0000","last_fullsized":"2024-04-16T21:07:29.651931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:01.385906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493180+0000","last_change":"2024-04-16T19:24:47.424747+0000","last_active":"2024-04-16T21:07:29.493180+0000","last_peered":"2024-04-16T21:07:29.493180+0000","last_clean":"2024-04-16T21:07:29.493180+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:07:29.493180+0000","last_undegraded":"2024-04-16T21:07:29.493180+0000","last_fullsized":"2024-04-16T21:07:29.493180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:48.064472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652026+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:07:29.652026+0000","last_peered":"2024-04-16T21:07:29.652026+0000","last_clean":"2024-04-16T21:07:29.652026+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:07:29.652026+0000","last_undegraded":"2024-04-16T21:07:29.652026+0000","last_fullsized":"2024-04-16T21:07:29.652026+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:21.377903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653531+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:07:34.653531+0000","last_peered":"2024-04-16T21:07:34.653531+0000","last_clean":"2024-04-16T21:07:34.653531+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:07:34.653531+0000","last_undegraded":"2024-04-16T21:07:34.653531+0000","last_fullsized":"2024-04-16T21:07:34.653531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659404+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:08:19.659404+0000","last_peered":"2024-04-16T21:08:19.659404+0000","last_clean":"2024-04-16T21:08:19.659404+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:08:19.659404+0000","last_undegraded":"2024-04-16T21:08:19.659404+0000","last_fullsized":"2024-04-16T21:08:19.659404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493079+0000","last_change":"2024-04-16T19:25:41.508866+0000","last_active":"2024-04-16T21:07:29.493079+0000","last_peered":"2024-04-16T21:07:29.493079+0000","last_clean":"2024-04-16T21:07:29.493079+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:07:29.493079+0000","last_undegraded":"2024-04-16T21:07:29.493079+0000","last_fullsized":"2024-04-16T21:07:29.493079+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:29.426956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'22","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.486695+0000","last_change":"2024-04-16T19:25:15.480626+0000","last_active":"2024-04-16T21:06:49.486695+0000","last_peered":"2024-04-16T21:06:49.486695+0000","last_clean":"2024-04-16T21:06:49.486695+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:06:49.486695+0000","last_undegraded":"2024-04-16T21:06:49.486695+0000","last_fullsized":"2024-04-16T21:06:49.486695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:39.898263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'705","reported_seq":2140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491714+0000","last_change":"2024-04-16T19:25:13.473351+0000","last_active":"2024-04-16T21:07:19.491714+0000","last_peered":"2024-04-16T21:07:19.491714+0000","last_clean":"2024-04-16T21:07:19.491714+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:07:19.491714+0000","last_undegraded":"2024-04-16T21:07:19.491714+0000","last_fullsized":"2024-04-16T21:07:19.491714+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:04.370192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1017","reported_seq":1516,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651283+0000","last_change":"2024-04-16T19:25:11.466163+0000","last_active":"2024-04-16T21:07:29.651283+0000","last_peered":"2024-04-16T21:07:29.651283+0000","last_clean":"2024-04-16T21:07:29.651283+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:07:29.651283+0000","last_undegraded":"2024-04-16T21:07:29.651283+0000","last_fullsized":"2024-04-16T21:07:29.651283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:40.676377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'14","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652919+0000","last_change":"2024-04-16T19:25:09.467909+0000","last_active":"2024-04-16T21:07:39.652919+0000","last_peered":"2024-04-16T21:07:39.652919+0000","last_clean":"2024-04-16T21:07:39.652919+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:07:39.652919+0000","last_undegraded":"2024-04-16T21:07:39.652919+0000","last_fullsized":"2024-04-16T21:07:39.652919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:05.530059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651388+0000","last_change":"2024-04-16T19:24:51.438772+0000","last_active":"2024-04-16T21:07:24.651388+0000","last_peered":"2024-04-16T21:07:24.651388+0000","last_clean":"2024-04-16T21:07:24.651388+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:07:24.651388+0000","last_undegraded":"2024-04-16T21:07:24.651388+0000","last_fullsized":"2024-04-16T21:07:24.651388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491555+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:07:19.491555+0000","last_peered":"2024-04-16T21:07:19.491555+0000","last_clean":"2024-04-16T21:07:19.491555+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:07:19.491555+0000","last_undegraded":"2024-04-16T21:07:19.491555+0000","last_fullsized":"2024-04-16T21:07:19.491555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661253+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:08:29.661253+0000","last_peered":"2024-04-16T21:08:29.661253+0000","last_clean":"2024-04-16T21:08:29.661253+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:08:29.661253+0000","last_undegraded":"2024-04-16T21:08:29.661253+0000","last_fullsized":"2024-04-16T21:08:29.661253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652693+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:07:29.652693+0000","last_peered":"2024-04-16T21:07:29.652693+0000","last_clean":"2024-04-16T21:07:29.652693+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:07:29.652693+0000","last_undegraded":"2024-04-16T21:07:29.652693+0000","last_fullsized":"2024-04-16T21:07:29.652693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652054+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:07:29.652054+0000","last_peered":"2024-04-16T21:07:29.652054+0000","last_clean":"2024-04-16T21:07:29.652054+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:07:29.652054+0000","last_undegraded":"2024-04-16T21:07:29.652054+0000","last_fullsized":"2024-04-16T21:07:29.652054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:52.547347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651195+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:07:29.651195+0000","last_peered":"2024-04-16T21:07:29.651195+0000","last_clean":"2024-04-16T21:07:29.651195+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:07:29.651195+0000","last_undegraded":"2024-04-16T21:07:29.651195+0000","last_fullsized":"2024-04-16T21:07:29.651195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648537+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:07:04.648537+0000","last_peered":"2024-04-16T21:07:04.648537+0000","last_clean":"2024-04-16T21:07:04.648537+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:07:04.648537+0000","last_undegraded":"2024-04-16T21:07:04.648537+0000","last_fullsized":"2024-04-16T21:07:04.648537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:37.665943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2028,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651065+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:07:19.651065+0000","last_peered":"2024-04-16T21:07:19.651065+0000","last_clean":"2024-04-16T21:07:19.651065+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:07:19.651065+0000","last_undegraded":"2024-04-16T21:07:19.651065+0000","last_fullsized":"2024-04-16T21:07:19.651065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:48.277899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648877+0000","last_change":"2024-04-16T19:25:41.507140+0000","last_active":"2024-04-16T21:07:14.648877+0000","last_peered":"2024-04-16T21:07:14.648877+0000","last_clean":"2024-04-16T21:07:14.648877+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:07:14.648877+0000","last_undegraded":"2024-04-16T21:07:14.648877+0000","last_fullsized":"2024-04-16T21:07:14.648877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:14:47.746067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'5","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645995+0000","last_change":"2024-04-16T19:25:15.481140+0000","last_active":"2024-04-16T21:06:49.645995+0000","last_peered":"2024-04-16T21:06:49.645995+0000","last_clean":"2024-04-16T21:06:49.645995+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:06:49.645995+0000","last_undegraded":"2024-04-16T21:06:49.645995+0000","last_fullsized":"2024-04-16T21:06:49.645995+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:22.980049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655276+0000","last_change":"2024-04-16T19:25:13.476134+0000","last_active":"2024-04-16T21:07:54.655276+0000","last_peered":"2024-04-16T21:07:54.655276+0000","last_clean":"2024-04-16T21:07:54.655276+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:07:54.655276+0000","last_undegraded":"2024-04-16T21:07:54.655276+0000","last_fullsized":"2024-04-16T21:07:54.655276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'305","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656185+0000","last_change":"2024-04-16T19:25:11.465819+0000","last_active":"2024-04-16T21:07:54.656185+0000","last_peered":"2024-04-16T21:07:54.656185+0000","last_clean":"2024-04-16T21:07:54.656185+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:07:54.656185+0000","last_undegraded":"2024-04-16T21:07:54.656185+0000","last_fullsized":"2024-04-16T21:07:54.656185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:38.882914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'6","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655341+0000","last_change":"2024-04-16T19:25:09.469614+0000","last_active":"2024-04-16T21:07:49.655341+0000","last_peered":"2024-04-16T21:07:49.655341+0000","last_clean":"2024-04-16T21:07:49.655341+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:07:49.655341+0000","last_undegraded":"2024-04-16T21:07:49.655341+0000","last_fullsized":"2024-04-16T21:07:49.655341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366575+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:08:49.366575+0000","last_peered":"2024-04-16T21:08:49.366575+0000","last_clean":"2024-04-16T21:08:49.366575+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:08:49.366575+0000","last_undegraded":"2024-04-16T21:08:49.366575+0000","last_fullsized":"2024-04-16T21:08:49.366575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:54.123433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502246+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:08:29.502246+0000","last_peered":"2024-04-16T21:08:29.502246+0000","last_clean":"2024-04-16T21:08:29.502246+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:08:29.502246+0000","last_undegraded":"2024-04-16T21:08:29.502246+0000","last_fullsized":"2024-04-16T21:08:29.502246+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:19:16.213346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.374003+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:09:09.374003+0000","last_peered":"2024-04-16T21:09:09.374003+0000","last_clean":"2024-04-16T21:09:09.374003+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:09:09.374003+0000","last_undegraded":"2024-04-16T21:09:09.374003+0000","last_fullsized":"2024-04-16T21:09:09.374003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491759+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:07:19.491759+0000","last_peered":"2024-04-16T21:07:19.491759+0000","last_clean":"2024-04-16T21:07:19.491759+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:07:19.491759+0000","last_undegraded":"2024-04-16T21:07:19.491759+0000","last_fullsized":"2024-04-16T21:07:19.491759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:50.759439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651037+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:07:19.651037+0000","last_peered":"2024-04-16T21:07:19.651037+0000","last_clean":"2024-04-16T21:07:19.651037+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:07:19.651037+0000","last_undegraded":"2024-04-16T21:07:19.651037+0000","last_fullsized":"2024-04-16T21:07:19.651037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494045+0000","last_change":"2024-04-16T19:24:51.574221+0000","last_active":"2024-04-16T21:07:34.494045+0000","last_peered":"2024-04-16T21:07:34.494045+0000","last_clean":"2024-04-16T21:07:34.494045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:07:34.494045+0000","last_undegraded":"2024-04-16T21:07:34.494045+0000","last_fullsized":"2024-04-16T21:07:34.494045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:09.678742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650574+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:07:19.650574+0000","last_peered":"2024-04-16T21:07:19.650574+0000","last_clean":"2024-04-16T21:07:19.650574+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:07:19.650574+0000","last_undegraded":"2024-04-16T21:07:19.650574+0000","last_fullsized":"2024-04-16T21:07:19.650574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:44.374493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1826,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365476+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:08:49.365476+0000","last_peered":"2024-04-16T21:08:49.365476+0000","last_clean":"2024-04-16T21:08:49.365476+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:08:49.365476+0000","last_undegraded":"2024-04-16T21:08:49.365476+0000","last_fullsized":"2024-04-16T21:08:49.365476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:09.652050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649723+0000","last_change":"2024-04-16T19:25:41.504184+0000","last_active":"2024-04-16T21:07:14.649723+0000","last_peered":"2024-04-16T21:07:14.649723+0000","last_clean":"2024-04-16T21:07:14.649723+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:07:14.649723+0000","last_undegraded":"2024-04-16T21:07:14.649723+0000","last_fullsized":"2024-04-16T21:07:14.649723+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645440+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:06:49.645440+0000","last_peered":"2024-04-16T21:06:49.645440+0000","last_clean":"2024-04-16T21:06:49.645440+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:06:49.645440+0000","last_undegraded":"2024-04-16T21:06:49.645440+0000","last_fullsized":"2024-04-16T21:06:49.645440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:47:48.700988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497245+0000","last_change":"2024-04-16T19:25:13.472015+0000","last_active":"2024-04-16T21:07:54.497245+0000","last_peered":"2024-04-16T21:07:54.497245+0000","last_clean":"2024-04-16T21:07:54.497245+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:07:54.497245+0000","last_undegraded":"2024-04-16T21:07:54.497245+0000","last_fullsized":"2024-04-16T21:07:54.497245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'121","reported_seq":272,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655968+0000","last_change":"2024-04-16T19:25:11.465848+0000","last_active":"2024-04-16T21:07:54.655968+0000","last_peered":"2024-04-16T21:07:54.655968+0000","last_clean":"2024-04-16T21:07:54.655968+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:07:54.655968+0000","last_undegraded":"2024-04-16T21:07:54.655968+0000","last_fullsized":"2024-04-16T21:07:54.655968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'4","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655987+0000","last_change":"2024-04-16T19:25:09.462205+0000","last_active":"2024-04-16T21:07:49.655987+0000","last_peered":"2024-04-16T21:07:49.655987+0000","last_clean":"2024-04-16T21:07:49.655987+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:07:49.655987+0000","last_undegraded":"2024-04-16T21:07:49.655987+0000","last_fullsized":"2024-04-16T21:07:49.655987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:53.677676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372422+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:08:59.372422+0000","last_peered":"2024-04-16T21:08:59.372422+0000","last_clean":"2024-04-16T21:08:59.372422+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:08:59.372422+0000","last_undegraded":"2024-04-16T21:08:59.372422+0000","last_fullsized":"2024-04-16T21:08:59.372422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364818+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:08:49.364818+0000","last_peered":"2024-04-16T21:08:49.364818+0000","last_clean":"2024-04-16T21:08:49.364818+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:08:49.364818+0000","last_undegraded":"2024-04-16T21:08:49.364818+0000","last_fullsized":"2024-04-16T21:08:49.364818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.368088+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:08:59.368088+0000","last_peered":"2024-04-16T21:08:59.368088+0000","last_clean":"2024-04-16T21:08:59.368088+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:08:59.368088+0000","last_undegraded":"2024-04-16T21:08:59.368088+0000","last_fullsized":"2024-04-16T21:08:59.368088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:34.577971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651049+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:07:19.651049+0000","last_peered":"2024-04-16T21:07:19.651049+0000","last_clean":"2024-04-16T21:07:19.651049+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:07:19.651049+0000","last_undegraded":"2024-04-16T21:07:19.651049+0000","last_fullsized":"2024-04-16T21:07:19.651049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:14.554679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491744+0000","last_change":"2024-04-16T19:24:49.432195+0000","last_active":"2024-04-16T21:07:19.491744+0000","last_peered":"2024-04-16T21:07:19.491744+0000","last_clean":"2024-04-16T21:07:19.491744+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:07:19.491744+0000","last_undegraded":"2024-04-16T21:07:19.491744+0000","last_fullsized":"2024-04-16T21:07:19.491744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:18.116203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653498+0000","last_change":"2024-04-16T19:24:51.573063+0000","last_active":"2024-04-16T21:07:34.653498+0000","last_peered":"2024-04-16T21:07:34.653498+0000","last_clean":"2024-04-16T21:07:34.653498+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:07:34.653498+0000","last_undegraded":"2024-04-16T21:07:34.653498+0000","last_fullsized":"2024-04-16T21:07:34.653498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:01.882223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649776+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:07:19.649776+0000","last_peered":"2024-04-16T21:07:19.649776+0000","last_clean":"2024-04-16T21:07:19.649776+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:07:19.649776+0000","last_undegraded":"2024-04-16T21:07:19.649776+0000","last_fullsized":"2024-04-16T21:07:19.649776+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:28:47.506010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491827+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:07:19.491827+0000","last_peered":"2024-04-16T21:07:19.491827+0000","last_clean":"2024-04-16T21:07:19.491827+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:07:19.491827+0000","last_undegraded":"2024-04-16T21:07:19.491827+0000","last_fullsized":"2024-04-16T21:07:19.491827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:32:33.273769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662724+0000","last_change":"2024-04-16T19:25:41.509494+0000","last_active":"2024-04-16T21:08:39.662724+0000","last_peered":"2024-04-16T21:08:39.662724+0000","last_clean":"2024-04-16T21:08:39.662724+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:08:39.662724+0000","last_undegraded":"2024-04-16T21:08:39.662724+0000","last_fullsized":"2024-04-16T21:08:39.662724+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'8","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645398+0000","last_change":"2024-04-16T19:25:15.483611+0000","last_active":"2024-04-16T21:06:49.645398+0000","last_peered":"2024-04-16T21:06:49.645398+0000","last_clean":"2024-04-16T21:06:49.645398+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:06:49.645398+0000","last_undegraded":"2024-04-16T21:06:49.645398+0000","last_fullsized":"2024-04-16T21:06:49.645398+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656848+0000","last_change":"2024-04-16T19:25:13.472723+0000","last_active":"2024-04-16T21:07:54.656848+0000","last_peered":"2024-04-16T21:07:54.656848+0000","last_clean":"2024-04-16T21:07:54.656848+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:07:54.656848+0000","last_undegraded":"2024-04-16T21:07:54.656848+0000","last_fullsized":"2024-04-16T21:07:54.656848+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'321","reported_seq":850,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497294+0000","last_change":"2024-04-16T19:25:11.464441+0000","last_active":"2024-04-16T21:07:54.497294+0000","last_peered":"2024-04-16T21:07:54.497294+0000","last_clean":"2024-04-16T21:07:54.497294+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:07:54.497294+0000","last_undegraded":"2024-04-16T21:07:54.497294+0000","last_fullsized":"2024-04-16T21:07:54.497294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:55.683718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'12","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655101+0000","last_change":"2024-04-16T19:25:09.465881+0000","last_active":"2024-04-16T21:07:49.655101+0000","last_peered":"2024-04-16T21:07:49.655101+0000","last_clean":"2024-04-16T21:07:49.655101+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:07:49.655101+0000","last_undegraded":"2024-04-16T21:07:49.655101+0000","last_fullsized":"2024-04-16T21:07:49.655101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648098+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:07:09.648098+0000","last_peered":"2024-04-16T21:07:09.648098+0000","last_clean":"2024-04-16T21:07:09.648098+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:07:09.648098+0000","last_undegraded":"2024-04-16T21:07:09.648098+0000","last_fullsized":"2024-04-16T21:07:09.648098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648274+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:07:04.648274+0000","last_peered":"2024-04-16T21:07:04.648274+0000","last_clean":"2024-04-16T21:07:04.648274+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:07:04.648274+0000","last_undegraded":"2024-04-16T21:07:04.648274+0000","last_fullsized":"2024-04-16T21:07:04.648274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:48.449534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650601+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:07:19.650601+0000","last_peered":"2024-04-16T21:07:19.650601+0000","last_clean":"2024-04-16T21:07:19.650601+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:07:19.650601+0000","last_undegraded":"2024-04-16T21:07:19.650601+0000","last_fullsized":"2024-04-16T21:07:19.650601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:17.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507635+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:08:59.507635+0000","last_peered":"2024-04-16T21:08:59.507635+0000","last_clean":"2024-04-16T21:08:59.507635+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:08:59.507635+0000","last_undegraded":"2024-04-16T21:08:59.507635+0000","last_fullsized":"2024-04-16T21:08:59.507635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:40.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368151+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:09:09.368151+0000","last_peered":"2024-04-16T21:09:09.368151+0000","last_clean":"2024-04-16T21:09:09.368151+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:09:09.368151+0000","last_undegraded":"2024-04-16T21:09:09.368151+0000","last_fullsized":"2024-04-16T21:09:09.368151+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:22.117545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653453+0000","last_change":"2024-04-16T19:24:51.442399+0000","last_active":"2024-04-16T21:07:34.653453+0000","last_peered":"2024-04-16T21:07:34.653453+0000","last_clean":"2024-04-16T21:07:34.653453+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:07:34.653453+0000","last_undegraded":"2024-04-16T21:07:34.653453+0000","last_fullsized":"2024-04-16T21:07:34.653453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:24.851144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650701+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:07:19.650701+0000","last_peered":"2024-04-16T21:07:19.650701+0000","last_clean":"2024-04-16T21:07:19.650701+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:07:19.650701+0000","last_undegraded":"2024-04-16T21:07:19.650701+0000","last_fullsized":"2024-04-16T21:07:19.650701+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:42.705352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1306,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491787+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:07:19.491787+0000","last_peered":"2024-04-16T21:07:19.491787+0000","last_clean":"2024-04-16T21:07:19.491787+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:07:19.491787+0000","last_undegraded":"2024-04-16T21:07:19.491787+0000","last_fullsized":"2024-04-16T21:07:19.491787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:35:13.067416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649861+0000","last_change":"2024-04-16T19:25:41.509274+0000","last_active":"2024-04-16T21:07:14.649861+0000","last_peered":"2024-04-16T21:07:14.649861+0000","last_clean":"2024-04-16T21:07:14.649861+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:07:14.649861+0000","last_undegraded":"2024-04-16T21:07:14.649861+0000","last_fullsized":"2024-04-16T21:07:14.649861+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:45.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"47'5","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659574+0000","last_change":"2024-04-16T19:25:15.486177+0000","last_active":"2024-04-16T21:08:14.659574+0000","last_peered":"2024-04-16T21:08:14.659574+0000","last_clean":"2024-04-16T21:08:14.659574+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:08:14.659574+0000","last_undegraded":"2024-04-16T21:08:14.659574+0000","last_fullsized":"2024-04-16T21:08:14.659574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:20.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656265+0000","last_change":"2024-04-16T19:25:13.473925+0000","last_active":"2024-04-16T21:07:54.656265+0000","last_peered":"2024-04-16T21:07:54.656265+0000","last_clean":"2024-04-16T21:07:54.656265+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:07:54.656265+0000","last_undegraded":"2024-04-16T21:07:54.656265+0000","last_fullsized":"2024-04-16T21:07:54.656265+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:22.051385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'241","reported_seq":1104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656229+0000","last_change":"2024-04-16T19:25:11.465793+0000","last_active":"2024-04-16T21:07:54.656229+0000","last_peered":"2024-04-16T21:07:54.656229+0000","last_clean":"2024-04-16T21:07:54.656229+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:07:54.656229+0000","last_undegraded":"2024-04-16T21:07:54.656229+0000","last_fullsized":"2024-04-16T21:07:54.656229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:00.538757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'7","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655923+0000","last_change":"2024-04-16T19:25:09.458604+0000","last_active":"2024-04-16T21:07:49.655923+0000","last_peered":"2024-04-16T21:07:49.655923+0000","last_clean":"2024-04-16T21:07:49.655923+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:07:49.655923+0000","last_undegraded":"2024-04-16T21:07:49.655923+0000","last_fullsized":"2024-04-16T21:07:49.655923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489948+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:07:09.489948+0000","last_peered":"2024-04-16T21:07:09.489948+0000","last_clean":"2024-04-16T21:07:09.489948+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:07:09.489948+0000","last_undegraded":"2024-04-16T21:07:09.489948+0000","last_fullsized":"2024-04-16T21:07:09.489948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647363+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:07:04.647363+0000","last_peered":"2024-04-16T21:07:04.647363+0000","last_clean":"2024-04-16T21:07:04.647363+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:07:04.647363+0000","last_undegraded":"2024-04-16T21:07:04.647363+0000","last_fullsized":"2024-04-16T21:07:04.647363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491726+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:07:19.491726+0000","last_peered":"2024-04-16T21:07:19.491726+0000","last_clean":"2024-04-16T21:07:19.491726+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:07:19.491726+0000","last_undegraded":"2024-04-16T21:07:19.491726+0000","last_fullsized":"2024-04-16T21:07:19.491726+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:45.917687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369571+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:09:09.369571+0000","last_peered":"2024-04-16T21:09:09.369571+0000","last_clean":"2024-04-16T21:09:09.369571+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:09:09.369571+0000","last_undegraded":"2024-04-16T21:09:09.369571+0000","last_fullsized":"2024-04-16T21:09:09.369571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:32.829903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367311+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:08:59.367311+0000","last_peered":"2024-04-16T21:08:59.367311+0000","last_clean":"2024-04-16T21:08:59.367311+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:08:59.367311+0000","last_undegraded":"2024-04-16T21:08:59.367311+0000","last_fullsized":"2024-04-16T21:08:59.367311+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:02.604453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653371+0000","last_change":"2024-04-16T19:24:51.438145+0000","last_active":"2024-04-16T21:07:34.653371+0000","last_peered":"2024-04-16T21:07:34.653371+0000","last_clean":"2024-04-16T21:07:34.653371+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:07:34.653371+0000","last_undegraded":"2024-04-16T21:07:34.653371+0000","last_fullsized":"2024-04-16T21:07:34.653371+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:45.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646595+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:06:54.646595+0000","last_peered":"2024-04-16T21:06:54.646595+0000","last_clean":"2024-04-16T21:06:54.646595+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:06:54.646595+0000","last_undegraded":"2024-04-16T21:06:54.646595+0000","last_fullsized":"2024-04-16T21:06:54.646595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:31.554611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661634+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:08:29.661634+0000","last_peered":"2024-04-16T21:08:29.661634+0000","last_clean":"2024-04-16T21:08:29.661634+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:08:29.661634+0000","last_undegraded":"2024-04-16T21:08:29.661634+0000","last_fullsized":"2024-04-16T21:08:29.661634+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:20.643182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660228+0000","last_change":"2024-04-16T19:25:41.508939+0000","last_active":"2024-04-16T21:08:24.660228+0000","last_peered":"2024-04-16T21:08:24.660228+0000","last_clean":"2024-04-16T21:08:24.660228+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:08:24.660228+0000","last_undegraded":"2024-04-16T21:08:24.660228+0000","last_fullsized":"2024-04-16T21:08:24.660228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'11","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498087+0000","last_change":"2024-04-16T19:25:15.480096+0000","last_active":"2024-04-16T21:07:59.498087+0000","last_peered":"2024-04-16T21:07:59.498087+0000","last_clean":"2024-04-16T21:07:59.498087+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:07:59.498087+0000","last_undegraded":"2024-04-16T21:07:59.498087+0000","last_fullsized":"2024-04-16T21:07:59.498087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:58.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659661+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:08:09.659661+0000","last_peered":"2024-04-16T21:08:09.659661+0000","last_clean":"2024-04-16T21:08:09.659661+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:08:09.659661+0000","last_undegraded":"2024-04-16T21:08:09.659661+0000","last_fullsized":"2024-04-16T21:08:09.659661+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'540","reported_seq":1355,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.643433+0000","last_change":"2024-04-16T19:25:11.465566+0000","last_active":"2024-04-16T21:06:44.643433+0000","last_peered":"2024-04-16T21:06:44.643433+0000","last_clean":"2024-04-16T21:06:44.643433+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:06:44.643433+0000","last_undegraded":"2024-04-16T21:06:44.643433+0000","last_fullsized":"2024-04-16T21:06:44.643433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:52.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'9","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643501+0000","last_change":"2024-04-16T19:25:09.458347+0000","last_active":"2024-04-16T21:06:39.643501+0000","last_peered":"2024-04-16T21:06:39.643501+0000","last_clean":"2024-04-16T21:06:39.643501+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:06:39.643501+0000","last_undegraded":"2024-04-16T21:06:39.643501+0000","last_fullsized":"2024-04-16T21:06:39.643501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649060+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:07:09.649060+0000","last_peered":"2024-04-16T21:07:09.649060+0000","last_clean":"2024-04-16T21:07:09.649060+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:07:09.649060+0000","last_undegraded":"2024-04-16T21:07:09.649060+0000","last_fullsized":"2024-04-16T21:07:09.649060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:41.726894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648043+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:07:04.648043+0000","last_peered":"2024-04-16T21:07:04.648043+0000","last_clean":"2024-04-16T21:07:04.648043+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:07:04.648043+0000","last_undegraded":"2024-04-16T21:07:04.648043+0000","last_fullsized":"2024-04-16T21:07:04.648043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650514+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:07:19.650514+0000","last_peered":"2024-04-16T21:07:19.650514+0000","last_clean":"2024-04-16T21:07:19.650514+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:07:19.650514+0000","last_undegraded":"2024-04-16T21:07:19.650514+0000","last_fullsized":"2024-04-16T21:07:19.650514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:46.054276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.642525+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:06:34.642525+0000","last_peered":"2024-04-16T21:06:34.642525+0000","last_clean":"2024-04-16T21:06:34.642525+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:06:34.642525+0000","last_undegraded":"2024-04-16T21:06:34.642525+0000","last_fullsized":"2024-04-16T21:06:34.642525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:47.298587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369608+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:08:44.369608+0000","last_peered":"2024-04-16T21:08:44.369608+0000","last_clean":"2024-04-16T21:08:44.369608+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:08:44.369608+0000","last_undegraded":"2024-04-16T21:08:44.369608+0000","last_fullsized":"2024-04-16T21:08:44.369608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:38:59.670330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366563+0000","last_change":"2024-04-16T19:24:51.437809+0000","last_active":"2024-04-16T21:08:49.366563+0000","last_peered":"2024-04-16T21:08:49.366563+0000","last_clean":"2024-04-16T21:08:49.366563+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:08:49.366563+0000","last_undegraded":"2024-04-16T21:08:49.366563+0000","last_fullsized":"2024-04-16T21:08:49.366563+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487434+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:06:54.487434+0000","last_peered":"2024-04-16T21:06:54.487434+0000","last_clean":"2024-04-16T21:06:54.487434+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:06:54.487434+0000","last_undegraded":"2024-04-16T21:06:54.487434+0000","last_fullsized":"2024-04-16T21:06:54.487434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:55:01.357303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1337,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502628+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:08:29.502628+0000","last_peered":"2024-04-16T21:08:29.502628+0000","last_clean":"2024-04-16T21:08:29.502628+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:08:29.502628+0000","last_undegraded":"2024-04-16T21:08:29.502628+0000","last_fullsized":"2024-04-16T21:08:29.502628+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:48.136957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661216+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:08:24.661216+0000","last_peered":"2024-04-16T21:08:24.661216+0000","last_clean":"2024-04-16T21:08:24.661216+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:08:24.661216+0000","last_undegraded":"2024-04-16T21:08:24.661216+0000","last_fullsized":"2024-04-16T21:08:24.661216+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:47.003646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'8","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656179+0000","last_change":"2024-04-16T19:25:15.481698+0000","last_active":"2024-04-16T21:07:59.656179+0000","last_peered":"2024-04-16T21:07:59.656179+0000","last_clean":"2024-04-16T21:07:59.656179+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:07:59.656179+0000","last_undegraded":"2024-04-16T21:07:59.656179+0000","last_fullsized":"2024-04-16T21:07:59.656179+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'705","reported_seq":2071,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.485874+0000","last_change":"2024-04-16T19:25:13.474684+0000","last_active":"2024-04-16T21:06:44.485874+0000","last_peered":"2024-04-16T21:06:44.485874+0000","last_clean":"2024-04-16T21:06:44.485874+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:06:44.485874+0000","last_undegraded":"2024-04-16T21:06:44.485874+0000","last_fullsized":"2024-04-16T21:06:44.485874+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:52.004725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'519","reported_seq":41267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499619+0000","last_change":"2024-04-16T19:25:11.465897+0000","last_active":"2024-04-16T21:08:09.499619+0000","last_peered":"2024-04-16T21:08:09.499619+0000","last_clean":"2024-04-16T21:08:09.499619+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:08:09.499619+0000","last_undegraded":"2024-04-16T21:08:09.499619+0000","last_fullsized":"2024-04-16T21:08:09.499619+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:54.168091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'9","reported_seq":307,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643870+0000","last_change":"2024-04-16T19:25:09.459964+0000","last_active":"2024-04-16T21:06:39.643870+0000","last_peered":"2024-04-16T21:06:39.643870+0000","last_clean":"2024-04-16T21:06:39.643870+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:06:39.643870+0000","last_undegraded":"2024-04-16T21:06:39.643870+0000","last_fullsized":"2024-04-16T21:06:39.643870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649029+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:07:09.649029+0000","last_peered":"2024-04-16T21:07:09.649029+0000","last_clean":"2024-04-16T21:07:09.649029+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:07:09.649029+0000","last_undegraded":"2024-04-16T21:07:09.649029+0000","last_fullsized":"2024-04-16T21:07:09.649029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648095+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:07:04.648095+0000","last_peered":"2024-04-16T21:07:04.648095+0000","last_clean":"2024-04-16T21:07:04.648095+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:07:04.648095+0000","last_undegraded":"2024-04-16T21:07:04.648095+0000","last_fullsized":"2024-04-16T21:07:04.648095+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:23.870654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:04.143195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369636+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:08:44.369636+0000","last_peered":"2024-04-16T21:08:44.369636+0000","last_clean":"2024-04-16T21:08:44.369636+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:08:44.369636+0000","last_undegraded":"2024-04-16T21:08:44.369636+0000","last_fullsized":"2024-04-16T21:08:44.369636+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.642926+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:06:34.642926+0000","last_peered":"2024-04-16T21:06:34.642926+0000","last_clean":"2024-04-16T21:06:34.642926+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:06:34.642926+0000","last_undegraded":"2024-04-16T21:06:34.642926+0000","last_fullsized":"2024-04-16T21:06:34.642926+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364971+0000","last_change":"2024-04-16T19:24:51.573983+0000","last_active":"2024-04-16T21:08:49.364971+0000","last_peered":"2024-04-16T21:08:49.364971+0000","last_clean":"2024-04-16T21:08:49.364971+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:08:49.364971+0000","last_undegraded":"2024-04-16T21:08:49.364971+0000","last_fullsized":"2024-04-16T21:08:49.364971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.645515+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:06:54.645515+0000","last_peered":"2024-04-16T21:06:54.645515+0000","last_clean":"2024-04-16T21:06:54.645515+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:06:54.645515+0000","last_undegraded":"2024-04-16T21:06:54.645515+0000","last_fullsized":"2024-04-16T21:06:54.645515+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'167","reported_seq":2111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502381+0000","last_change":"2024-04-16T19:25:49.519298+0000","last_active":"2024-04-16T21:08:29.502381+0000","last_peered":"2024-04-16T21:08:29.502381+0000","last_clean":"2024-04-16T21:08:29.502381+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:08:29.502381+0000","last_undegraded":"2024-04-16T21:08:29.502381+0000","last_fullsized":"2024-04-16T21:08:29.502381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:10.645836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501890+0000","last_change":"2024-04-16T19:25:41.507268+0000","last_active":"2024-04-16T21:08:24.501890+0000","last_peered":"2024-04-16T21:08:24.501890+0000","last_clean":"2024-04-16T21:08:24.501890+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:08:24.501890+0000","last_undegraded":"2024-04-16T21:08:24.501890+0000","last_fullsized":"2024-04-16T21:08:24.501890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:46.285785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657100+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:07:59.657100+0000","last_peered":"2024-04-16T21:07:59.657100+0000","last_clean":"2024-04-16T21:07:59.657100+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:07:59.657100+0000","last_undegraded":"2024-04-16T21:07:59.657100+0000","last_fullsized":"2024-04-16T21:07:59.657100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:46.607704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644452+0000","last_change":"2024-04-16T19:25:13.472244+0000","last_active":"2024-04-16T21:06:44.644452+0000","last_peered":"2024-04-16T21:06:44.644452+0000","last_clean":"2024-04-16T21:06:44.644452+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:06:44.644452+0000","last_undegraded":"2024-04-16T21:06:44.644452+0000","last_fullsized":"2024-04-16T21:06:44.644452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:59.248190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'573","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644779+0000","last_change":"2024-04-16T19:25:11.468986+0000","last_active":"2024-04-16T21:06:44.644779+0000","last_peered":"2024-04-16T21:06:44.644779+0000","last_clean":"2024-04-16T21:06:44.644779+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:06:44.644779+0000","last_undegraded":"2024-04-16T21:06:44.644779+0000","last_fullsized":"2024-04-16T21:06:44.644779+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'23","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498919+0000","last_change":"2024-04-16T19:25:09.469035+0000","last_active":"2024-04-16T21:08:04.498919+0000","last_peered":"2024-04-16T21:08:04.498919+0000","last_clean":"2024-04-16T21:08:04.498919+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:08:04.498919+0000","last_undegraded":"2024-04-16T21:08:04.498919+0000","last_fullsized":"2024-04-16T21:08:04.498919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371344+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:08:54.371344+0000","last_peered":"2024-04-16T21:08:54.371344+0000","last_clean":"2024-04-16T21:08:54.371344+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:08:54.371344+0000","last_undegraded":"2024-04-16T21:08:54.371344+0000","last_fullsized":"2024-04-16T21:08:54.371344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506100+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:08:49.506100+0000","last_peered":"2024-04-16T21:08:49.506100+0000","last_clean":"2024-04-16T21:08:49.506100+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:08:49.506100+0000","last_undegraded":"2024-04-16T21:08:49.506100+0000","last_fullsized":"2024-04-16T21:08:49.506100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:11.958520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649687+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:07:19.649687+0000","last_peered":"2024-04-16T21:07:19.649687+0000","last_clean":"2024-04-16T21:07:19.649687+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:07:19.649687+0000","last_undegraded":"2024-04-16T21:07:19.649687+0000","last_fullsized":"2024-04-16T21:07:19.649687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650982+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:07:19.650982+0000","last_peered":"2024-04-16T21:07:19.650982+0000","last_clean":"2024-04-16T21:07:19.650982+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:07:19.650982+0000","last_undegraded":"2024-04-16T21:07:19.650982+0000","last_fullsized":"2024-04-16T21:07:19.650982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366474+0000","last_change":"2024-04-16T19:24:51.444882+0000","last_active":"2024-04-16T21:08:49.366474+0000","last_peered":"2024-04-16T21:08:49.366474+0000","last_clean":"2024-04-16T21:08:49.366474+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:08:49.366474+0000","last_undegraded":"2024-04-16T21:08:49.366474+0000","last_fullsized":"2024-04-16T21:08:49.366474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:18.795884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487525+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:06:54.487525+0000","last_peered":"2024-04-16T21:06:54.487525+0000","last_clean":"2024-04-16T21:06:54.487525+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:06:54.487525+0000","last_undegraded":"2024-04-16T21:06:54.487525+0000","last_fullsized":"2024-04-16T21:06:54.487525+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:37:36.899944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'127","reported_seq":1509,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502335+0000","last_change":"2024-04-16T19:25:49.517472+0000","last_active":"2024-04-16T21:08:29.502335+0000","last_peered":"2024-04-16T21:08:29.502335+0000","last_clean":"2024-04-16T21:08:29.502335+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:08:29.502335+0000","last_undegraded":"2024-04-16T21:08:29.502335+0000","last_fullsized":"2024-04-16T21:08:29.502335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:30.634191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501937+0000","last_change":"2024-04-16T19:25:41.502982+0000","last_active":"2024-04-16T21:08:24.501937+0000","last_peered":"2024-04-16T21:08:24.501937+0000","last_clean":"2024-04-16T21:08:24.501937+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:08:24.501937+0000","last_undegraded":"2024-04-16T21:08:24.501937+0000","last_fullsized":"2024-04-16T21:08:24.501937+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:12.630377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"47'10","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656132+0000","last_change":"2024-04-16T19:25:15.481921+0000","last_active":"2024-04-16T21:07:59.656132+0000","last_peered":"2024-04-16T21:07:59.656132+0000","last_clean":"2024-04-16T21:07:59.656132+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:07:59.656132+0000","last_undegraded":"2024-04-16T21:07:59.656132+0000","last_fullsized":"2024-04-16T21:07:59.656132+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644518+0000","last_change":"2024-04-16T19:25:13.473915+0000","last_active":"2024-04-16T21:06:44.644518+0000","last_peered":"2024-04-16T21:06:44.644518+0000","last_clean":"2024-04-16T21:06:44.644518+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:06:44.644518+0000","last_undegraded":"2024-04-16T21:06:44.644518+0000","last_fullsized":"2024-04-16T21:06:44.644518+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:05.770561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'230","reported_seq":647,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:44.644717+0000","last_change":"2024-04-16T19:25:11.461723+0000","last_active":"2024-04-16T21:06:44.644717+0000","last_peered":"2024-04-16T21:06:44.644717+0000","last_clean":"2024-04-16T21:06:44.644717+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:06:44.644717+0000","last_undegraded":"2024-04-16T21:06:44.644717+0000","last_fullsized":"2024-04-16T21:06:44.644717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:15.327068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'6","reported_seq":174,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:39.643461+0000","last_change":"2024-04-16T19:25:09.458030+0000","last_active":"2024-04-16T21:06:39.643461+0000","last_peered":"2024-04-16T21:06:39.643461+0000","last_clean":"2024-04-16T21:06:39.643461+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:06:39.643461+0000","last_undegraded":"2024-04-16T21:06:39.643461+0000","last_fullsized":"2024-04-16T21:06:39.643461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663306+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:08:34.663306+0000","last_peered":"2024-04-16T21:08:34.663306+0000","last_clean":"2024-04-16T21:08:34.663306+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:08:34.663306+0000","last_undegraded":"2024-04-16T21:08:34.663306+0000","last_fullsized":"2024-04-16T21:08:34.663306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369834+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:08:44.369834+0000","last_peered":"2024-04-16T21:08:44.369834+0000","last_clean":"2024-04-16T21:08:44.369834+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:08:44.369834+0000","last_undegraded":"2024-04-16T21:08:44.369834+0000","last_fullsized":"2024-04-16T21:08:44.369834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:41.742389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:34.484254+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:06:34.484254+0000","last_peered":"2024-04-16T21:06:34.484254+0000","last_clean":"2024-04-16T21:06:34.484254+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:06:34.484254+0000","last_undegraded":"2024-04-16T21:06:34.484254+0000","last_fullsized":"2024-04-16T21:06:34.484254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:38.387958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":222,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650527+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:07:19.650527+0000","last_peered":"2024-04-16T21:07:19.650527+0000","last_clean":"2024-04-16T21:07:19.650527+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:07:19.650527+0000","last_undegraded":"2024-04-16T21:07:19.650527+0000","last_fullsized":"2024-04-16T21:07:19.650527+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:47.927883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649659+0000","last_change":"2024-04-16T19:24:49.432869+0000","last_active":"2024-04-16T21:07:19.649659+0000","last_peered":"2024-04-16T21:07:19.649659+0000","last_clean":"2024-04-16T21:07:19.649659+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:07:19.649659+0000","last_undegraded":"2024-04-16T21:07:19.649659+0000","last_fullsized":"2024-04-16T21:07:19.649659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655242+0000","last_change":"2024-04-16T19:24:51.442598+0000","last_active":"2024-04-16T21:07:49.655242+0000","last_peered":"2024-04-16T21:07:49.655242+0000","last_clean":"2024-04-16T21:07:49.655242+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:07:49.655242+0000","last_undegraded":"2024-04-16T21:07:49.655242+0000","last_fullsized":"2024-04-16T21:07:49.655242+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:52.222795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'12","reported_seq":186,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504441+0000","last_change":"2024-04-16T19:25:09.464004+0000","last_active":"2024-04-16T21:08:39.504441+0000","last_peered":"2024-04-16T21:08:39.504441+0000","last_clean":"2024-04-16T21:08:39.504441+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:08:39.504441+0000","last_undegraded":"2024-04-16T21:08:39.504441+0000","last_fullsized":"2024-04-16T21:08:39.504441+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504683+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:08:39.504683+0000","last_peered":"2024-04-16T21:08:39.504683+0000","last_clean":"2024-04-16T21:08:39.504683+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:08:39.504683+0000","last_undegraded":"2024-04-16T21:08:39.504683+0000","last_fullsized":"2024-04-16T21:08:39.504683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364532+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:08:44.364532+0000","last_peered":"2024-04-16T21:08:44.364532+0000","last_clean":"2024-04-16T21:08:44.364532+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:08:44.364532+0000","last_undegraded":"2024-04-16T21:08:44.364532+0000","last_fullsized":"2024-04-16T21:08:44.364532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'12","reported_seq":242,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663758+0000","last_change":"2024-04-16T19:25:09.466486+0000","last_active":"2024-04-16T21:08:39.663758+0000","last_peered":"2024-04-16T21:08:39.663758+0000","last_clean":"2024-04-16T21:08:39.663758+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:08:39.663758+0000","last_undegraded":"2024-04-16T21:08:39.663758+0000","last_fullsized":"2024-04-16T21:08:39.663758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504632+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:08:39.504632+0000","last_peered":"2024-04-16T21:08:39.504632+0000","last_clean":"2024-04-16T21:08:39.504632+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:08:39.504632+0000","last_undegraded":"2024-04-16T21:08:39.504632+0000","last_fullsized":"2024-04-16T21:08:39.504632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:49.140683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505241+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:08:44.505241+0000","last_peered":"2024-04-16T21:08:44.505241+0000","last_clean":"2024-04-16T21:08:44.505241+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:08:44.505241+0000","last_undegraded":"2024-04-16T21:08:44.505241+0000","last_fullsized":"2024-04-16T21:08:44.505241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:03:25.186300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'5","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649520+0000","last_change":"2024-04-16T19:25:09.467044+0000","last_active":"2024-04-16T21:07:14.649520+0000","last_peered":"2024-04-16T21:07:14.649520+0000","last_clean":"2024-04-16T21:07:14.649520+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:07:14.649520+0000","last_undegraded":"2024-04-16T21:07:14.649520+0000","last_fullsized":"2024-04-16T21:07:14.649520+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663818+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:08:39.663818+0000","last_peered":"2024-04-16T21:08:39.663818+0000","last_clean":"2024-04-16T21:08:39.663818+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:08:39.663818+0000","last_undegraded":"2024-04-16T21:08:39.663818+0000","last_fullsized":"2024-04-16T21:08:39.663818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365701+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:08:44.365701+0000","last_peered":"2024-04-16T21:08:44.365701+0000","last_clean":"2024-04-16T21:08:44.365701+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:08:44.365701+0000","last_undegraded":"2024-04-16T21:08:44.365701+0000","last_fullsized":"2024-04-16T21:08:44.365701+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'7","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664206+0000","last_change":"2024-04-16T19:25:09.458671+0000","last_active":"2024-04-16T21:08:39.664206+0000","last_peered":"2024-04-16T21:08:39.664206+0000","last_clean":"2024-04-16T21:08:39.664206+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:08:39.664206+0000","last_undegraded":"2024-04-16T21:08:39.664206+0000","last_fullsized":"2024-04-16T21:08:39.664206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:52.162049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504531+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:08:39.504531+0000","last_peered":"2024-04-16T21:08:39.504531+0000","last_clean":"2024-04-16T21:08:39.504531+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:08:39.504531+0000","last_undegraded":"2024-04-16T21:08:39.504531+0000","last_fullsized":"2024-04-16T21:08:39.504531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:22.916044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373116+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:09:04.373116+0000","last_peered":"2024-04-16T21:09:04.373116+0000","last_clean":"2024-04-16T21:09:04.373116+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:09:04.373116+0000","last_undegraded":"2024-04-16T21:09:04.373116+0000","last_fullsized":"2024-04-16T21:09:04.373116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'11","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655456+0000","last_change":"2024-04-16T19:25:09.466470+0000","last_active":"2024-04-16T21:07:49.655456+0000","last_peered":"2024-04-16T21:07:49.655456+0000","last_clean":"2024-04-16T21:07:49.655456+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:07:49.655456+0000","last_undegraded":"2024-04-16T21:07:49.655456+0000","last_fullsized":"2024-04-16T21:07:49.655456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508457+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:09:04.508457+0000","last_peered":"2024-04-16T21:09:04.508457+0000","last_clean":"2024-04-16T21:09:04.508457+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:09:04.508457+0000","last_undegraded":"2024-04-16T21:09:04.508457+0000","last_fullsized":"2024-04-16T21:09:04.508457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:58.689460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503608+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:08:34.503608+0000","last_peered":"2024-04-16T21:08:34.503608+0000","last_clean":"2024-04-16T21:08:34.503608+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:08:34.503608+0000","last_undegraded":"2024-04-16T21:08:34.503608+0000","last_fullsized":"2024-04-16T21:08:34.503608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:49.978039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'20","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496627+0000","last_change":"2024-04-16T19:25:09.460695+0000","last_active":"2024-04-16T21:07:49.496627+0000","last_peered":"2024-04-16T21:07:49.496627+0000","last_clean":"2024-04-16T21:07:49.496627+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:07:49.496627+0000","last_undegraded":"2024-04-16T21:07:49.496627+0000","last_fullsized":"2024-04-16T21:07:49.496627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:28.879723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365419+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:08:49.365419+0000","last_peered":"2024-04-16T21:08:49.365419+0000","last_clean":"2024-04-16T21:08:49.365419+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:08:49.365419+0000","last_undegraded":"2024-04-16T21:08:49.365419+0000","last_fullsized":"2024-04-16T21:08:49.365419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:16.805347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503638+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:08:34.503638+0000","last_peered":"2024-04-16T21:08:34.503638+0000","last_clean":"2024-04-16T21:08:34.503638+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:08:34.503638+0000","last_undegraded":"2024-04-16T21:08:34.503638+0000","last_fullsized":"2024-04-16T21:08:34.503638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'11","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504570+0000","last_change":"2024-04-16T19:25:09.459969+0000","last_active":"2024-04-16T21:08:39.504570+0000","last_peered":"2024-04-16T21:08:39.504570+0000","last_clean":"2024-04-16T21:08:39.504570+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:08:39.504570+0000","last_undegraded":"2024-04-16T21:08:39.504570+0000","last_fullsized":"2024-04-16T21:08:39.504570+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364844+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:08:49.364844+0000","last_peered":"2024-04-16T21:08:49.364844+0000","last_clean":"2024-04-16T21:08:49.364844+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:08:49.364844+0000","last_undegraded":"2024-04-16T21:08:49.364844+0000","last_fullsized":"2024-04-16T21:08:49.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663475+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:08:34.663475+0000","last_peered":"2024-04-16T21:08:34.663475+0000","last_clean":"2024-04-16T21:08:34.663475+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:08:34.663475+0000","last_undegraded":"2024-04-16T21:08:34.663475+0000","last_fullsized":"2024-04-16T21:08:34.663475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:40.025589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'13","reported_seq":290,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.506803+0000","last_change":"2024-04-16T19:25:09.459396+0000","last_active":"2024-04-16T21:08:54.506803+0000","last_peered":"2024-04-16T21:08:54.506803+0000","last_clean":"2024-04-16T21:08:54.506803+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:08:54.506803+0000","last_undegraded":"2024-04-16T21:08:54.506803+0000","last_fullsized":"2024-04-16T21:08:54.506803+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:44.854219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366364+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:08:49.366364+0000","last_peered":"2024-04-16T21:08:49.366364+0000","last_clean":"2024-04-16T21:08:49.366364+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:08:49.366364+0000","last_undegraded":"2024-04-16T21:08:49.366364+0000","last_fullsized":"2024-04-16T21:08:49.366364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503606+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:08:34.503606+0000","last_peered":"2024-04-16T21:08:34.503606+0000","last_clean":"2024-04-16T21:08:34.503606+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:08:34.503606+0000","last_undegraded":"2024-04-16T21:08:34.503606+0000","last_fullsized":"2024-04-16T21:08:34.503606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:56:59.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'38","reported_seq":309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650208+0000","last_change":"2024-04-16T19:25:09.462718+0000","last_active":"2024-04-16T21:07:14.650208+0000","last_peered":"2024-04-16T21:07:14.650208+0000","last_clean":"2024-04-16T21:07:14.650208+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:07:14.650208+0000","last_undegraded":"2024-04-16T21:07:14.650208+0000","last_fullsized":"2024-04-16T21:07:14.650208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:51.473781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649801+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:07:19.649801+0000","last_peered":"2024-04-16T21:07:19.649801+0000","last_clean":"2024-04-16T21:07:19.649801+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:07:19.649801+0000","last_undegraded":"2024-04-16T21:07:19.649801+0000","last_fullsized":"2024-04-16T21:07:19.649801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366306+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:08:54.366306+0000","last_peered":"2024-04-16T21:08:54.366306+0000","last_clean":"2024-04-16T21:08:54.366306+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:08:54.366306+0000","last_undegraded":"2024-04-16T21:08:54.366306+0000","last_fullsized":"2024-04-16T21:08:54.366306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'14","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493167+0000","last_change":"2024-04-16T19:25:09.467165+0000","last_active":"2024-04-16T21:07:29.493167+0000","last_peered":"2024-04-16T21:07:29.493167+0000","last_clean":"2024-04-16T21:07:29.493167+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:07:29.493167+0000","last_undegraded":"2024-04-16T21:07:29.493167+0000","last_fullsized":"2024-04-16T21:07:29.493167+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:19.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650655+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:07:19.650655+0000","last_peered":"2024-04-16T21:07:19.650655+0000","last_clean":"2024-04-16T21:07:19.650655+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:07:19.650655+0000","last_undegraded":"2024-04-16T21:07:19.650655+0000","last_fullsized":"2024-04-16T21:07:19.650655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657012+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:08:04.657012+0000","last_peered":"2024-04-16T21:08:04.657012+0000","last_clean":"2024-04-16T21:08:04.657012+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:08:04.657012+0000","last_undegraded":"2024-04-16T21:08:04.657012+0000","last_fullsized":"2024-04-16T21:08:04.657012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'9","reported_seq":179,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648706+0000","last_change":"2024-04-16T19:25:09.464008+0000","last_active":"2024-04-16T21:07:14.648706+0000","last_peered":"2024-04-16T21:07:14.648706+0000","last_clean":"2024-04-16T21:07:14.648706+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:07:14.648706+0000","last_undegraded":"2024-04-16T21:07:14.648706+0000","last_fullsized":"2024-04-16T21:07:14.648706+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501544+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:08:24.501544+0000","last_peered":"2024-04-16T21:08:24.501544+0000","last_clean":"2024-04-16T21:08:24.501544+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:08:24.501544+0000","last_undegraded":"2024-04-16T21:08:24.501544+0000","last_fullsized":"2024-04-16T21:08:24.501544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657935+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:08:04.657935+0000","last_peered":"2024-04-16T21:08:04.657935+0000","last_clean":"2024-04-16T21:08:04.657935+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:08:04.657935+0000","last_undegraded":"2024-04-16T21:08:04.657935+0000","last_fullsized":"2024-04-16T21:08:04.657935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'22","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490768+0000","last_change":"2024-04-16T19:25:09.467151+0000","last_active":"2024-04-16T21:07:14.490768+0000","last_peered":"2024-04-16T21:07:14.490768+0000","last_clean":"2024-04-16T21:07:14.490768+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:07:14.490768+0000","last_undegraded":"2024-04-16T21:07:14.490768+0000","last_fullsized":"2024-04-16T21:07:14.490768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:22.630747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659703+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:08:09.659703+0000","last_peered":"2024-04-16T21:08:09.659703+0000","last_clean":"2024-04-16T21:08:09.659703+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:08:09.659703+0000","last_undegraded":"2024-04-16T21:08:09.659703+0000","last_fullsized":"2024-04-16T21:08:09.659703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488264+0000","last_change":"2024-04-16T19:25:41.505958+0000","last_active":"2024-04-16T21:06:59.488264+0000","last_peered":"2024-04-16T21:06:59.488264+0000","last_clean":"2024-04-16T21:06:59.488264+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:06:59.488264+0000","last_undegraded":"2024-04-16T21:06:59.488264+0000","last_fullsized":"2024-04-16T21:06:59.488264+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:19.162890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'5","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369023+0000","last_change":"2024-04-16T19:25:09.467605+0000","last_active":"2024-04-16T21:09:09.369023+0000","last_peered":"2024-04-16T21:09:09.369023+0000","last_clean":"2024-04-16T21:09:09.369023+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:09:09.369023+0000","last_undegraded":"2024-04-16T21:09:09.369023+0000","last_fullsized":"2024-04-16T21:09:09.369023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658742+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:08:09.658742+0000","last_peered":"2024-04-16T21:08:09.658742+0000","last_clean":"2024-04-16T21:08:09.658742+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:08:09.658742+0000","last_undegraded":"2024-04-16T21:08:09.658742+0000","last_fullsized":"2024-04-16T21:08:09.658742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:24.447630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658987+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:08:04.658987+0000","last_peered":"2024-04-16T21:08:04.658987+0000","last_clean":"2024-04-16T21:08:04.658987+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:08:04.658987+0000","last_undegraded":"2024-04-16T21:08:04.658987+0000","last_fullsized":"2024-04-16T21:08:04.658987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'14","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509341+0000","last_change":"2024-04-16T19:25:09.464438+0000","last_active":"2024-04-16T21:09:09.509341+0000","last_peered":"2024-04-16T21:09:09.509341+0000","last_clean":"2024-04-16T21:09:09.509341+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:09:09.509341+0000","last_undegraded":"2024-04-16T21:09:09.509341+0000","last_fullsized":"2024-04-16T21:09:09.509341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657776+0000","last_change":"2024-04-16T19:25:41.508398+0000","last_active":"2024-04-16T21:08:09.657776+0000","last_peered":"2024-04-16T21:08:09.657776+0000","last_clean":"2024-04-16T21:08:09.657776+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:08:09.657776+0000","last_undegraded":"2024-04-16T21:08:09.657776+0000","last_fullsized":"2024-04-16T21:08:09.657776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:28:30.770132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661196+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:08:19.661196+0000","last_peered":"2024-04-16T21:08:19.661196+0000","last_clean":"2024-04-16T21:08:19.661196+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:08:19.661196+0000","last_undegraded":"2024-04-16T21:08:19.661196+0000","last_fullsized":"2024-04-16T21:08:19.661196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:37.636086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'3","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652749+0000","last_change":"2024-04-16T19:25:41.499612+0000","last_active":"2024-04-16T21:07:34.652749+0000","last_peered":"2024-04-16T21:07:34.652749+0000","last_clean":"2024-04-16T21:07:34.652749+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:07:34.652749+0000","last_undegraded":"2024-04-16T21:07:34.652749+0000","last_fullsized":"2024-04-16T21:07:34.652749+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499546+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:08:09.499546+0000","last_peered":"2024-04-16T21:08:09.499546+0000","last_clean":"2024-04-16T21:08:09.499546+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:08:09.499546+0000","last_undegraded":"2024-04-16T21:08:09.499546+0000","last_fullsized":"2024-04-16T21:08:09.499546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490904+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:07:14.490904+0000","last_peered":"2024-04-16T21:07:14.490904+0000","last_clean":"2024-04-16T21:07:14.490904+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:07:14.490904+0000","last_undegraded":"2024-04-16T21:07:14.490904+0000","last_fullsized":"2024-04-16T21:07:14.490904+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:50.557311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'19","reported_seq":286,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373955+0000","last_change":"2024-04-16T19:25:09.468675+0000","last_active":"2024-04-16T21:09:09.373955+0000","last_peered":"2024-04-16T21:09:09.373955+0000","last_clean":"2024-04-16T21:09:09.373955+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:09:09.373955+0000","last_undegraded":"2024-04-16T21:09:09.373955+0000","last_fullsized":"2024-04-16T21:09:09.373955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:21:12.600213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367898+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:08:59.367898+0000","last_peered":"2024-04-16T21:08:59.367898+0000","last_clean":"2024-04-16T21:08:59.367898+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:08:59.367898+0000","last_undegraded":"2024-04-16T21:08:59.367898+0000","last_fullsized":"2024-04-16T21:08:59.367898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649751+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:07:14.649751+0000","last_peered":"2024-04-16T21:07:14.649751+0000","last_clean":"2024-04-16T21:07:14.649751+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:07:14.649751+0000","last_undegraded":"2024-04-16T21:07:14.649751+0000","last_fullsized":"2024-04-16T21:07:14.649751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:50.981157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'12","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652830+0000","last_change":"2024-04-16T19:25:09.459052+0000","last_active":"2024-04-16T21:07:39.652830+0000","last_peered":"2024-04-16T21:07:39.652830+0000","last_clean":"2024-04-16T21:07:39.652830+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:07:39.652830+0000","last_undegraded":"2024-04-16T21:07:39.652830+0000","last_fullsized":"2024-04-16T21:07:39.652830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:51.756021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650094+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:07:14.650094+0000","last_peered":"2024-04-16T21:07:14.650094+0000","last_clean":"2024-04-16T21:07:14.650094+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:07:14.650094+0000","last_undegraded":"2024-04-16T21:07:14.650094+0000","last_fullsized":"2024-04-16T21:07:14.650094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371494+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:08:54.371494+0000","last_peered":"2024-04-16T21:08:54.371494+0000","last_clean":"2024-04-16T21:08:54.371494+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:08:54.371494+0000","last_undegraded":"2024-04-16T21:08:54.371494+0000","last_fullsized":"2024-04-16T21:08:54.371494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'3","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494798+0000","last_change":"2024-04-16T19:25:09.460170+0000","last_active":"2024-04-16T21:07:39.494798+0000","last_peered":"2024-04-16T21:07:39.494798+0000","last_clean":"2024-04-16T21:07:39.494798+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:07:39.494798+0000","last_undegraded":"2024-04-16T21:07:39.494798+0000","last_fullsized":"2024-04-16T21:07:39.494798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490719+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:07:14.490719+0000","last_peered":"2024-04-16T21:07:14.490719+0000","last_clean":"2024-04-16T21:07:14.490719+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:07:14.490719+0000","last_undegraded":"2024-04-16T21:07:14.490719+0000","last_fullsized":"2024-04-16T21:07:14.490719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:08:34.663111+0000","last_peered":"2024-04-16T21:08:34.663111+0000","last_clean":"2024-04-16T21:08:34.663111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:08:34.663111+0000","last_undegraded":"2024-04-16T21:08:34.663111+0000","last_fullsized":"2024-04-16T21:08:34.663111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4203,"ondisk_log_size":4203,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3420,"ondisk_log_size":3420,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5660,"ondisk_log_size":5660,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641550,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41132,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816468,"statfs":{"total":107374182400,"available":107332063232,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641551,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641550,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55568,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802032,"statfs":{"total":107374182400,"available":107317280768,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641551,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59228,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798372,"statfs":{"total":107374182400,"available":107313532928,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641551,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816412,"statfs":{"total":107374182400,"available":107332005888,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641550,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55800,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104801800,"statfs":{"total":107374182400,"available":107317043200,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:11:40.101 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-16T21:11:40.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:11:40.458 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:11:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:11:40.482 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3283,"stamp":"2024-04-16T21:11:39.328224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30466,"ondisk_log_size":30466,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":307840,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":296630,"kb_avail":628837760,"statfs":{"total":644245094400,"available":643929866240,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":303749793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005488"},"pg_stats":[{"pgid":"9.35","version":"52'28","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367393+0000","last_change":"2024-04-16T19:25:09.466020+0000","last_active":"2024-04-16T21:09:04.367393+0000","last_peered":"2024-04-16T21:09:04.367393+0000","last_clean":"2024-04-16T21:09:04.367393+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:09:04.367393+0000","last_undegraded":"2024-04-16T21:09:04.367393+0000","last_fullsized":"2024-04-16T21:09:04.367393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365868+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:08:44.365868+0000","last_peered":"2024-04-16T21:08:44.365868+0000","last_clean":"2024-04-16T21:08:44.365868+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:08:44.365868+0000","last_undegraded":"2024-04-16T21:08:44.365868+0000","last_fullsized":"2024-04-16T21:08:44.365868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648735+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:07:09.648735+0000","last_peered":"2024-04-16T21:07:09.648735+0000","last_clean":"2024-04-16T21:07:09.648735+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:07:09.648735+0000","last_undegraded":"2024-04-16T21:07:09.648735+0000","last_fullsized":"2024-04-16T21:07:09.648735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"47'5","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368044+0000","last_change":"2024-04-16T19:25:09.467691+0000","last_active":"2024-04-16T21:09:04.368044+0000","last_peered":"2024-04-16T21:09:04.368044+0000","last_clean":"2024-04-16T21:09:04.368044+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:09:04.368044+0000","last_undegraded":"2024-04-16T21:09:04.368044+0000","last_fullsized":"2024-04-16T21:09:04.368044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364114+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:08:44.364114+0000","last_peered":"2024-04-16T21:08:44.364114+0000","last_clean":"2024-04-16T21:08:44.364114+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:08:44.364114+0000","last_undegraded":"2024-04-16T21:08:44.364114+0000","last_fullsized":"2024-04-16T21:08:44.364114+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649318+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:07:09.649318+0000","last_peered":"2024-04-16T21:07:09.649318+0000","last_clean":"2024-04-16T21:07:09.649318+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:07:09.649318+0000","last_undegraded":"2024-04-16T21:07:09.649318+0000","last_fullsized":"2024-04-16T21:07:09.649318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'13","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646812+0000","last_change":"2024-04-16T19:25:09.457429+0000","last_active":"2024-04-16T21:06:54.646812+0000","last_peered":"2024-04-16T21:06:54.646812+0000","last_clean":"2024-04-16T21:06:54.646812+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:06:54.646812+0000","last_undegraded":"2024-04-16T21:06:54.646812+0000","last_fullsized":"2024-04-16T21:06:54.646812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369756+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:08:44.369756+0000","last_peered":"2024-04-16T21:08:44.369756+0000","last_clean":"2024-04-16T21:08:44.369756+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:08:44.369756+0000","last_undegraded":"2024-04-16T21:08:44.369756+0000","last_fullsized":"2024-04-16T21:08:44.369756+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:11.722618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649006+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:07:09.649006+0000","last_peered":"2024-04-16T21:07:09.649006+0000","last_clean":"2024-04-16T21:07:09.649006+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:07:09.649006+0000","last_undegraded":"2024-04-16T21:07:09.649006+0000","last_fullsized":"2024-04-16T21:07:09.649006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'13","reported_seq":234,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367421+0000","last_change":"2024-04-16T19:25:09.467050+0000","last_active":"2024-04-16T21:09:04.367421+0000","last_peered":"2024-04-16T21:09:04.367421+0000","last_clean":"2024-04-16T21:09:04.367421+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:09:04.367421+0000","last_undegraded":"2024-04-16T21:09:04.367421+0000","last_fullsized":"2024-04-16T21:09:04.367421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366441+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:08:54.366441+0000","last_peered":"2024-04-16T21:08:54.366441+0000","last_clean":"2024-04-16T21:08:54.366441+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:08:54.366441+0000","last_undegraded":"2024-04-16T21:08:54.366441+0000","last_fullsized":"2024-04-16T21:08:54.366441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648038+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:07:09.648038+0000","last_peered":"2024-04-16T21:07:09.648038+0000","last_clean":"2024-04-16T21:07:09.648038+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:07:09.648038+0000","last_undegraded":"2024-04-16T21:07:09.648038+0000","last_fullsized":"2024-04-16T21:07:09.648038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'6","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655139+0000","last_change":"2024-04-16T19:25:09.459828+0000","last_active":"2024-04-16T21:07:49.655139+0000","last_peered":"2024-04-16T21:07:49.655139+0000","last_clean":"2024-04-16T21:07:49.655139+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:07:49.655139+0000","last_undegraded":"2024-04-16T21:07:49.655139+0000","last_fullsized":"2024-04-16T21:07:49.655139+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648499+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:07:04.648499+0000","last_peered":"2024-04-16T21:07:04.648499+0000","last_clean":"2024-04-16T21:07:04.648499+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:07:04.648499+0000","last_undegraded":"2024-04-16T21:07:04.648499+0000","last_fullsized":"2024-04-16T21:07:04.648499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367192+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:08:59.367192+0000","last_peered":"2024-04-16T21:08:59.367192+0000","last_clean":"2024-04-16T21:08:59.367192+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:08:59.367192+0000","last_undegraded":"2024-04-16T21:08:59.367192+0000","last_fullsized":"2024-04-16T21:08:59.367192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'18","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654366+0000","last_change":"2024-04-16T19:25:09.461822+0000","last_active":"2024-04-16T21:07:49.654366+0000","last_peered":"2024-04-16T21:07:49.654366+0000","last_clean":"2024-04-16T21:07:49.654366+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:07:49.654366+0000","last_undegraded":"2024-04-16T21:07:49.654366+0000","last_fullsized":"2024-04-16T21:07:49.654366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647976+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:07:04.647976+0000","last_peered":"2024-04-16T21:07:04.647976+0000","last_clean":"2024-04-16T21:07:04.647976+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:07:04.647976+0000","last_undegraded":"2024-04-16T21:07:04.647976+0000","last_fullsized":"2024-04-16T21:07:04.647976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:07.917853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365518+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:08:49.365518+0000","last_peered":"2024-04-16T21:08:49.365518+0000","last_clean":"2024-04-16T21:08:49.365518+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:08:49.365518+0000","last_undegraded":"2024-04-16T21:08:49.365518+0000","last_fullsized":"2024-04-16T21:08:49.365518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649782+0000","last_change":"2024-04-16T19:25:41.508581+0000","last_active":"2024-04-16T21:07:14.649782+0000","last_peered":"2024-04-16T21:07:14.649782+0000","last_clean":"2024-04-16T21:07:14.649782+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:07:14.649782+0000","last_undegraded":"2024-04-16T21:07:14.649782+0000","last_fullsized":"2024-04-16T21:07:14.649782+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:12.396788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648409+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:07:04.648409+0000","last_peered":"2024-04-16T21:07:04.648409+0000","last_clean":"2024-04-16T21:07:04.648409+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:07:04.648409+0000","last_undegraded":"2024-04-16T21:07:04.648409+0000","last_fullsized":"2024-04-16T21:07:04.648409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:34.027154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365507+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:08:49.365507+0000","last_peered":"2024-04-16T21:08:49.365507+0000","last_clean":"2024-04-16T21:08:49.365507+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:08:49.365507+0000","last_undegraded":"2024-04-16T21:08:49.365507+0000","last_fullsized":"2024-04-16T21:08:49.365507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'15","reported_seq":238,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657926+0000","last_change":"2024-04-16T19:25:09.463558+0000","last_active":"2024-04-16T21:08:04.657926+0000","last_peered":"2024-04-16T21:08:04.657926+0000","last_clean":"2024-04-16T21:08:04.657926+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:08:04.657926+0000","last_undegraded":"2024-04-16T21:08:04.657926+0000","last_fullsized":"2024-04-16T21:08:04.657926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489018+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:07:04.489018+0000","last_peered":"2024-04-16T21:07:04.489018+0000","last_clean":"2024-04-16T21:07:04.489018+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:07:04.489018+0000","last_undegraded":"2024-04-16T21:07:04.489018+0000","last_fullsized":"2024-04-16T21:07:04.489018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506158+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:08:49.506158+0000","last_peered":"2024-04-16T21:08:49.506158+0000","last_clean":"2024-04-16T21:08:49.506158+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:08:49.506158+0000","last_undegraded":"2024-04-16T21:08:49.506158+0000","last_fullsized":"2024-04-16T21:08:49.506158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'7","reported_seq":221,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370754+0000","last_change":"2024-04-16T19:25:09.467094+0000","last_active":"2024-04-16T21:09:19.370754+0000","last_peered":"2024-04-16T21:09:19.370754+0000","last_clean":"2024-04-16T21:09:19.370754+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:09:19.370754+0000","last_undegraded":"2024-04-16T21:09:19.370754+0000","last_fullsized":"2024-04-16T21:09:19.370754+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:52.158151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650393+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:07:19.650393+0000","last_peered":"2024-04-16T21:07:19.650393+0000","last_clean":"2024-04-16T21:07:19.650393+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:07:19.650393+0000","last_undegraded":"2024-04-16T21:07:19.650393+0000","last_fullsized":"2024-04-16T21:07:19.650393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366531+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:08:49.366531+0000","last_peered":"2024-04-16T21:08:49.366531+0000","last_clean":"2024-04-16T21:08:49.366531+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:08:49.366531+0000","last_undegraded":"2024-04-16T21:08:49.366531+0000","last_fullsized":"2024-04-16T21:08:49.366531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'6","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372746+0000","last_change":"2024-04-16T19:25:09.459483+0000","last_active":"2024-04-16T21:09:29.372746+0000","last_peered":"2024-04-16T21:09:29.372746+0000","last_clean":"2024-04-16T21:09:29.372746+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:09:29.372746+0000","last_undegraded":"2024-04-16T21:09:29.372746+0000","last_fullsized":"2024-04-16T21:09:29.372746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650394+0000","last_change":"2024-04-16T19:25:41.503681+0000","last_active":"2024-04-16T21:07:19.650394+0000","last_peered":"2024-04-16T21:07:19.650394+0000","last_clean":"2024-04-16T21:07:19.650394+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:07:19.650394+0000","last_undegraded":"2024-04-16T21:07:19.650394+0000","last_fullsized":"2024-04-16T21:07:19.650394+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:33:58.945193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662645+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:08:29.662645+0000","last_peered":"2024-04-16T21:08:29.662645+0000","last_clean":"2024-04-16T21:08:29.662645+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:08:29.662645+0000","last_undegraded":"2024-04-16T21:08:29.662645+0000","last_fullsized":"2024-04-16T21:08:29.662645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'5","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510833+0000","last_change":"2024-04-16T19:25:09.464787+0000","last_active":"2024-04-16T21:09:19.510833+0000","last_peered":"2024-04-16T21:09:19.510833+0000","last_clean":"2024-04-16T21:09:19.510833+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:09:19.510833+0000","last_undegraded":"2024-04-16T21:09:19.510833+0000","last_fullsized":"2024-04-16T21:09:19.510833+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652786+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:07:34.652786+0000","last_peered":"2024-04-16T21:07:34.652786+0000","last_clean":"2024-04-16T21:07:34.652786+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:07:34.652786+0000","last_undegraded":"2024-04-16T21:07:34.652786+0000","last_fullsized":"2024-04-16T21:07:34.652786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662148+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:08:29.662148+0000","last_peered":"2024-04-16T21:08:29.662148+0000","last_clean":"2024-04-16T21:08:29.662148+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:08:29.662148+0000","last_undegraded":"2024-04-16T21:08:29.662148+0000","last_fullsized":"2024-04-16T21:08:29.662148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'10","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510806+0000","last_change":"2024-04-16T19:25:09.463733+0000","last_active":"2024-04-16T21:09:19.510806+0000","last_peered":"2024-04-16T21:09:19.510806+0000","last_clean":"2024-04-16T21:09:19.510806+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:09:19.510806+0000","last_undegraded":"2024-04-16T21:09:19.510806+0000","last_fullsized":"2024-04-16T21:09:19.510806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:40:59.948853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503495+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:08:34.503495+0000","last_peered":"2024-04-16T21:08:34.503495+0000","last_clean":"2024-04-16T21:08:34.503495+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:08:34.503495+0000","last_undegraded":"2024-04-16T21:08:34.503495+0000","last_fullsized":"2024-04-16T21:08:34.503495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:23.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661720+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:08:29.661720+0000","last_peered":"2024-04-16T21:08:29.661720+0000","last_clean":"2024-04-16T21:08:29.661720+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:08:29.661720+0000","last_undegraded":"2024-04-16T21:08:29.661720+0000","last_fullsized":"2024-04-16T21:08:29.661720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:09:50.552696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'9","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654268+0000","last_change":"2024-04-16T19:25:09.462286+0000","last_active":"2024-04-16T21:07:39.654268+0000","last_peered":"2024-04-16T21:07:39.654268+0000","last_clean":"2024-04-16T21:07:39.654268+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:07:39.654268+0000","last_undegraded":"2024-04-16T21:07:39.654268+0000","last_fullsized":"2024-04-16T21:07:39.654268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503526+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:08:34.503526+0000","last_peered":"2024-04-16T21:08:34.503526+0000","last_clean":"2024-04-16T21:08:34.503526+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:08:34.503526+0000","last_undegraded":"2024-04-16T21:08:34.503526+0000","last_fullsized":"2024-04-16T21:08:34.503526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661148+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:08:29.661148+0000","last_peered":"2024-04-16T21:08:29.661148+0000","last_clean":"2024-04-16T21:08:29.661148+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:08:29.661148+0000","last_undegraded":"2024-04-16T21:08:29.661148+0000","last_fullsized":"2024-04-16T21:08:29.661148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'15","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494683+0000","last_change":"2024-04-16T19:25:09.461082+0000","last_active":"2024-04-16T21:07:39.494683+0000","last_peered":"2024-04-16T21:07:39.494683+0000","last_clean":"2024-04-16T21:07:39.494683+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:07:39.494683+0000","last_undegraded":"2024-04-16T21:07:39.494683+0000","last_fullsized":"2024-04-16T21:07:39.494683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662702+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:08:34.662702+0000","last_peered":"2024-04-16T21:08:34.662702+0000","last_clean":"2024-04-16T21:08:34.662702+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:08:34.662702+0000","last_undegraded":"2024-04-16T21:08:34.662702+0000","last_fullsized":"2024-04-16T21:08:34.662702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651169+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:07:29.651169+0000","last_peered":"2024-04-16T21:07:29.651169+0000","last_clean":"2024-04-16T21:07:29.651169+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:07:29.651169+0000","last_undegraded":"2024-04-16T21:07:29.651169+0000","last_fullsized":"2024-04-16T21:07:29.651169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365000+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:08:49.365000+0000","last_peered":"2024-04-16T21:08:49.365000+0000","last_clean":"2024-04-16T21:08:49.365000+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:08:49.365000+0000","last_undegraded":"2024-04-16T21:08:49.365000+0000","last_fullsized":"2024-04-16T21:08:49.365000+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:32.817843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1720,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659053+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:08:04.659053+0000","last_peered":"2024-04-16T21:08:04.659053+0000","last_clean":"2024-04-16T21:08:04.659053+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:08:04.659053+0000","last_undegraded":"2024-04-16T21:08:04.659053+0000","last_fullsized":"2024-04-16T21:08:04.659053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660478+0000","last_change":"2024-04-16T19:25:41.505822+0000","last_active":"2024-04-16T21:08:14.660478+0000","last_peered":"2024-04-16T21:08:14.660478+0000","last_clean":"2024-04-16T21:08:14.660478+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:08:14.660478+0000","last_undegraded":"2024-04-16T21:08:14.660478+0000","last_fullsized":"2024-04-16T21:08:14.660478+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'6","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654478+0000","last_change":"2024-04-16T19:25:15.486584+0000","last_active":"2024-04-16T21:07:49.654478+0000","last_peered":"2024-04-16T21:07:49.654478+0000","last_clean":"2024-04-16T21:07:49.654478+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:07:49.654478+0000","last_undegraded":"2024-04-16T21:07:49.654478+0000","last_fullsized":"2024-04-16T21:07:49.654478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654387+0000","last_change":"2024-04-16T19:25:13.469257+0000","last_active":"2024-04-16T21:07:44.654387+0000","last_peered":"2024-04-16T21:07:44.654387+0000","last_clean":"2024-04-16T21:07:44.654387+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:07:44.654387+0000","last_undegraded":"2024-04-16T21:07:44.654387+0000","last_fullsized":"2024-04-16T21:07:44.654387+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'203","reported_seq":447,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655143+0000","last_change":"2024-04-16T19:25:11.464182+0000","last_active":"2024-04-16T21:07:44.655143+0000","last_peered":"2024-04-16T21:07:44.655143+0000","last_clean":"2024-04-16T21:07:44.655143+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:07:44.655143+0000","last_undegraded":"2024-04-16T21:07:44.655143+0000","last_fullsized":"2024-04-16T21:07:44.655143+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:07.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'19","reported_seq":284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487393+0000","last_change":"2024-04-16T19:25:09.464946+0000","last_active":"2024-04-16T21:06:54.487393+0000","last_peered":"2024-04-16T21:06:54.487393+0000","last_clean":"2024-04-16T21:06:54.487393+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:06:54.487393+0000","last_undegraded":"2024-04-16T21:06:54.487393+0000","last_fullsized":"2024-04-16T21:06:54.487393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:45:45.734532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365360+0000","last_change":"2024-04-16T19:24:51.436483+0000","last_active":"2024-04-16T21:08:49.365360+0000","last_peered":"2024-04-16T21:08:49.365360+0000","last_clean":"2024-04-16T21:08:49.365360+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:08:49.365360+0000","last_undegraded":"2024-04-16T21:08:49.365360+0000","last_fullsized":"2024-04-16T21:08:49.365360+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:40.603125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494096+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:07:34.494096+0000","last_peered":"2024-04-16T21:07:34.494096+0000","last_clean":"2024-04-16T21:07:34.494096+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:07:34.494096+0000","last_undegraded":"2024-04-16T21:07:34.494096+0000","last_fullsized":"2024-04-16T21:07:34.494096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:52.988517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650257+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:07:14.650257+0000","last_peered":"2024-04-16T21:07:14.650257+0000","last_clean":"2024-04-16T21:07:14.650257+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:07:14.650257+0000","last_undegraded":"2024-04-16T21:07:14.650257+0000","last_fullsized":"2024-04-16T21:07:14.650257+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:07.194576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:42:58.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369810+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:08:44.369810+0000","last_peered":"2024-04-16T21:08:44.369810+0000","last_clean":"2024-04-16T21:08:44.369810+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:08:44.369810+0000","last_undegraded":"2024-04-16T21:08:44.369810+0000","last_fullsized":"2024-04-16T21:08:44.369810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:08:44.364641+0000","last_peered":"2024-04-16T21:08:44.364641+0000","last_clean":"2024-04-16T21:08:44.364641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:08:44.364641+0000","last_undegraded":"2024-04-16T21:08:44.364641+0000","last_fullsized":"2024-04-16T21:08:44.364641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:42:28.507077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496581+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:07:49.496581+0000","last_peered":"2024-04-16T21:07:49.496581+0000","last_clean":"2024-04-16T21:07:49.496581+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:07:49.496581+0000","last_undegraded":"2024-04-16T21:07:49.496581+0000","last_fullsized":"2024-04-16T21:07:49.496581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:43.119570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1836,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365530+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:08:49.365530+0000","last_peered":"2024-04-16T21:08:49.365530+0000","last_clean":"2024-04-16T21:08:49.365530+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:08:49.365530+0000","last_undegraded":"2024-04-16T21:08:49.365530+0000","last_fullsized":"2024-04-16T21:08:49.365530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659544+0000","last_change":"2024-04-16T19:25:41.503686+0000","last_active":"2024-04-16T21:08:14.659544+0000","last_peered":"2024-04-16T21:08:14.659544+0000","last_clean":"2024-04-16T21:08:14.659544+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:08:14.659544+0000","last_undegraded":"2024-04-16T21:08:14.659544+0000","last_fullsized":"2024-04-16T21:08:14.659544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:30:44.336557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'5","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660521+0000","last_change":"2024-04-16T19:25:41.505903+0000","last_active":"2024-04-16T21:08:14.660521+0000","last_peered":"2024-04-16T21:08:14.660521+0000","last_clean":"2024-04-16T21:08:14.660521+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:08:14.660521+0000","last_undegraded":"2024-04-16T21:08:14.660521+0000","last_fullsized":"2024-04-16T21:08:14.660521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:37.409520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654445+0000","last_change":"2024-04-16T19:25:13.471998+0000","last_active":"2024-04-16T21:07:44.654445+0000","last_peered":"2024-04-16T21:07:44.654445+0000","last_clean":"2024-04-16T21:07:44.654445+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:07:44.654445+0000","last_undegraded":"2024-04-16T21:07:44.654445+0000","last_fullsized":"2024-04-16T21:07:44.654445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:15.038784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'241","reported_seq":976,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495593+0000","last_change":"2024-04-16T19:25:11.465710+0000","last_active":"2024-04-16T21:07:44.495593+0000","last_peered":"2024-04-16T21:07:44.495593+0000","last_clean":"2024-04-16T21:07:44.495593+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:07:44.495593+0000","last_undegraded":"2024-04-16T21:07:44.495593+0000","last_fullsized":"2024-04-16T21:07:44.495593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:25.422152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'5","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367870+0000","last_change":"2024-04-16T19:25:09.467837+0000","last_active":"2024-04-16T21:09:04.367870+0000","last_peered":"2024-04-16T21:09:04.367870+0000","last_clean":"2024-04-16T21:09:04.367870+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:09:04.367870+0000","last_undegraded":"2024-04-16T21:09:04.367870+0000","last_fullsized":"2024-04-16T21:09:04.367870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368878+0000","last_change":"2024-04-16T19:24:51.438136+0000","last_active":"2024-04-16T21:09:09.368878+0000","last_peered":"2024-04-16T21:09:09.368878+0000","last_clean":"2024-04-16T21:09:09.368878+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:09:09.368878+0000","last_undegraded":"2024-04-16T21:09:09.368878+0000","last_fullsized":"2024-04-16T21:09:09.368878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663065+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:08:34.663065+0000","last_peered":"2024-04-16T21:08:34.663065+0000","last_clean":"2024-04-16T21:08:34.663065+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:08:34.663065+0000","last_undegraded":"2024-04-16T21:08:34.663065+0000","last_fullsized":"2024-04-16T21:08:34.663065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:39.643446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648771+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:07:14.648771+0000","last_peered":"2024-04-16T21:07:14.648771+0000","last_clean":"2024-04-16T21:07:14.648771+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:07:14.648771+0000","last_undegraded":"2024-04-16T21:07:14.648771+0000","last_fullsized":"2024-04-16T21:07:14.648771+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:19.314716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653671+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:07:44.653671+0000","last_peered":"2024-04-16T21:07:44.653671+0000","last_clean":"2024-04-16T21:07:44.653671+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:07:44.653671+0000","last_undegraded":"2024-04-16T21:07:44.653671+0000","last_fullsized":"2024-04-16T21:07:44.653671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:26.398044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364812+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:08:44.364812+0000","last_peered":"2024-04-16T21:08:44.364812+0000","last_clean":"2024-04-16T21:08:44.364812+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:08:44.364812+0000","last_undegraded":"2024-04-16T21:08:44.364812+0000","last_fullsized":"2024-04-16T21:08:44.364812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:34.874297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364788+0000","last_change":"2024-04-16T19:24:49.426639+0000","last_active":"2024-04-16T21:08:44.364788+0000","last_peered":"2024-04-16T21:08:44.364788+0000","last_clean":"2024-04-16T21:08:44.364788+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:08:44.364788+0000","last_undegraded":"2024-04-16T21:08:44.364788+0000","last_fullsized":"2024-04-16T21:08:44.364788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:38.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652789+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:07:34.652789+0000","last_peered":"2024-04-16T21:07:34.652789+0000","last_clean":"2024-04-16T21:07:34.652789+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:07:34.652789+0000","last_undegraded":"2024-04-16T21:07:34.652789+0000","last_fullsized":"2024-04-16T21:07:34.652789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.732507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1871,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659473+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:08:19.659473+0000","last_peered":"2024-04-16T21:08:19.659473+0000","last_clean":"2024-04-16T21:08:19.659473+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:08:19.659473+0000","last_undegraded":"2024-04-16T21:08:19.659473+0000","last_fullsized":"2024-04-16T21:08:19.659473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:55.150921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655316+0000","last_change":"2024-04-16T19:25:41.502400+0000","last_active":"2024-04-16T21:07:49.655316+0000","last_peered":"2024-04-16T21:07:49.655316+0000","last_clean":"2024-04-16T21:07:49.655316+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:07:49.655316+0000","last_undegraded":"2024-04-16T21:07:49.655316+0000","last_fullsized":"2024-04-16T21:07:49.655316+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:02.633032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'17","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652839+0000","last_change":"2024-04-16T19:25:15.480449+0000","last_active":"2024-04-16T21:07:34.652839+0000","last_peered":"2024-04-16T21:07:34.652839+0000","last_clean":"2024-04-16T21:07:34.652839+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:07:34.652839+0000","last_undegraded":"2024-04-16T21:07:34.652839+0000","last_fullsized":"2024-04-16T21:07:34.652839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.646356+0000","last_change":"2024-04-16T19:25:13.475293+0000","last_active":"2024-04-16T21:06:59.646356+0000","last_peered":"2024-04-16T21:06:59.646356+0000","last_clean":"2024-04-16T21:06:59.646356+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:06:59.646356+0000","last_undegraded":"2024-04-16T21:06:59.646356+0000","last_fullsized":"2024-04-16T21:06:59.646356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'696","reported_seq":41909,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509197+0000","last_change":"2024-04-16T19:25:11.465229+0000","last_active":"2024-04-16T21:09:09.509197+0000","last_peered":"2024-04-16T21:09:09.509197+0000","last_clean":"2024-04-16T21:09:09.509197+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:09:09.509197+0000","last_undegraded":"2024-04-16T21:09:09.509197+0000","last_fullsized":"2024-04-16T21:09:09.509197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:43.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'25","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652851+0000","last_change":"2024-04-16T19:25:09.464644+0000","last_active":"2024-04-16T21:07:39.652851+0000","last_peered":"2024-04-16T21:07:39.652851+0000","last_clean":"2024-04-16T21:07:39.652851+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:07:39.652851+0000","last_undegraded":"2024-04-16T21:07:39.652851+0000","last_fullsized":"2024-04-16T21:07:39.652851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:46:24.560604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650411+0000","last_change":"2024-04-16T19:24:51.438116+0000","last_active":"2024-04-16T21:07:24.650411+0000","last_peered":"2024-04-16T21:07:24.650411+0000","last_clean":"2024-04-16T21:07:24.650411+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:07:24.650411+0000","last_undegraded":"2024-04-16T21:07:24.650411+0000","last_fullsized":"2024-04-16T21:07:24.650411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662083+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:08:34.662083+0000","last_peered":"2024-04-16T21:08:34.662083+0000","last_clean":"2024-04-16T21:08:34.662083+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:08:34.662083+0000","last_undegraded":"2024-04-16T21:08:34.662083+0000","last_fullsized":"2024-04-16T21:08:34.662083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650147+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:07:14.650147+0000","last_peered":"2024-04-16T21:07:14.650147+0000","last_clean":"2024-04-16T21:07:14.650147+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:07:14.650147+0000","last_undegraded":"2024-04-16T21:07:14.650147+0000","last_fullsized":"2024-04-16T21:07:14.650147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:05.587315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505369+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:08:44.505369+0000","last_peered":"2024-04-16T21:08:44.505369+0000","last_clean":"2024-04-16T21:08:44.505369+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:08:44.505369+0000","last_undegraded":"2024-04-16T21:08:44.505369+0000","last_fullsized":"2024-04-16T21:08:44.505369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:53.769511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":274,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495624+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:07:44.495624+0000","last_peered":"2024-04-16T21:07:44.495624+0000","last_clean":"2024-04-16T21:07:44.495624+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:07:44.495624+0000","last_undegraded":"2024-04-16T21:07:44.495624+0000","last_fullsized":"2024-04-16T21:07:44.495624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:44.851136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'12","reported_seq":1308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365800+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:08:44.365800+0000","last_peered":"2024-04-16T21:08:44.365800+0000","last_clean":"2024-04-16T21:08:44.365800+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:08:44.365800+0000","last_undegraded":"2024-04-16T21:08:44.365800+0000","last_fullsized":"2024-04-16T21:08:44.365800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:40.173753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":72,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494076+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:07:34.494076+0000","last_peered":"2024-04-16T21:07:34.494076+0000","last_clean":"2024-04-16T21:07:34.494076+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:07:34.494076+0000","last_undegraded":"2024-04-16T21:07:34.494076+0000","last_fullsized":"2024-04-16T21:07:34.494076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:53.541439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1472,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659510+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:08:19.659510+0000","last_peered":"2024-04-16T21:08:19.659510+0000","last_clean":"2024-04-16T21:08:19.659510+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:08:19.659510+0000","last_undegraded":"2024-04-16T21:08:19.659510+0000","last_fullsized":"2024-04-16T21:08:19.659510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:30.342275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657195+0000","last_change":"2024-04-16T19:25:41.501744+0000","last_active":"2024-04-16T21:07:59.657195+0000","last_peered":"2024-04-16T21:07:59.657195+0000","last_clean":"2024-04-16T21:07:59.657195+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:07:59.657195+0000","last_undegraded":"2024-04-16T21:07:59.657195+0000","last_fullsized":"2024-04-16T21:07:59.657195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:33.802040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'9","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372841+0000","last_change":"2024-04-16T19:25:15.487126+0000","last_active":"2024-04-16T21:09:29.372841+0000","last_peered":"2024-04-16T21:09:29.372841+0000","last_clean":"2024-04-16T21:09:29.372841+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:09:29.372841+0000","last_undegraded":"2024-04-16T21:09:29.372841+0000","last_fullsized":"2024-04-16T21:09:29.372841+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509179+0000","last_change":"2024-04-16T19:25:13.474735+0000","last_active":"2024-04-16T21:09:09.509179+0000","last_peered":"2024-04-16T21:09:09.509179+0000","last_clean":"2024-04-16T21:09:09.509179+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:09:09.509179+0000","last_undegraded":"2024-04-16T21:09:09.509179+0000","last_fullsized":"2024-04-16T21:09:09.509179+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'339","reported_seq":652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488172+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:06:59.488172+0000","last_peered":"2024-04-16T21:06:59.488172+0000","last_clean":"2024-04-16T21:06:59.488172+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:06:59.488172+0000","last_undegraded":"2024-04-16T21:06:59.488172+0000","last_fullsized":"2024-04-16T21:06:59.488172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:21.744422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'10","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494764+0000","last_change":"2024-04-16T19:25:09.468124+0000","last_active":"2024-04-16T21:07:39.494764+0000","last_peered":"2024-04-16T21:07:39.494764+0000","last_clean":"2024-04-16T21:07:39.494764+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:07:39.494764+0000","last_undegraded":"2024-04-16T21:07:39.494764+0000","last_fullsized":"2024-04-16T21:07:39.494764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:31.945014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651298+0000","last_change":"2024-04-16T19:24:51.572963+0000","last_active":"2024-04-16T21:07:24.651298+0000","last_peered":"2024-04-16T21:07:24.651298+0000","last_clean":"2024-04-16T21:07:24.651298+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:07:24.651298+0000","last_undegraded":"2024-04-16T21:07:24.651298+0000","last_fullsized":"2024-04-16T21:07:24.651298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:46.905067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662114+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:08:34.662114+0000","last_peered":"2024-04-16T21:08:34.662114+0000","last_clean":"2024-04-16T21:08:34.662114+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:08:34.662114+0000","last_undegraded":"2024-04-16T21:08:34.662114+0000","last_fullsized":"2024-04-16T21:08:34.662114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649505+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:07:14.649505+0000","last_peered":"2024-04-16T21:07:14.649505+0000","last_clean":"2024-04-16T21:07:14.649505+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:07:14.649505+0000","last_undegraded":"2024-04-16T21:07:14.649505+0000","last_fullsized":"2024-04-16T21:07:14.649505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365827+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:08:44.365827+0000","last_peered":"2024-04-16T21:08:44.365827+0000","last_clean":"2024-04-16T21:08:44.365827+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:08:44.365827+0000","last_undegraded":"2024-04-16T21:08:44.365827+0000","last_fullsized":"2024-04-16T21:08:44.365827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:41.163065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365785+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:08:44.365785+0000","last_peered":"2024-04-16T21:08:44.365785+0000","last_clean":"2024-04-16T21:08:44.365785+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:08:44.365785+0000","last_undegraded":"2024-04-16T21:08:44.365785+0000","last_fullsized":"2024-04-16T21:08:44.365785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:19.370822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495596+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:07:44.495596+0000","last_peered":"2024-04-16T21:07:44.495596+0000","last_clean":"2024-04-16T21:07:44.495596+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:07:44.495596+0000","last_undegraded":"2024-04-16T21:07:44.495596+0000","last_fullsized":"2024-04-16T21:07:44.495596+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:27.385349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365439+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:08:49.365439+0000","last_peered":"2024-04-16T21:08:49.365439+0000","last_clean":"2024-04-16T21:08:49.365439+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:08:49.365439+0000","last_undegraded":"2024-04-16T21:08:49.365439+0000","last_fullsized":"2024-04-16T21:08:49.365439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:12:02.296640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2006,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656784+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:07:54.656784+0000","last_peered":"2024-04-16T21:07:54.656784+0000","last_clean":"2024-04-16T21:07:54.656784+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:07:54.656784+0000","last_undegraded":"2024-04-16T21:07:54.656784+0000","last_fullsized":"2024-04-16T21:07:54.656784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:10:23.783937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655892+0000","last_change":"2024-04-16T19:25:41.506037+0000","last_active":"2024-04-16T21:07:49.655892+0000","last_peered":"2024-04-16T21:07:49.655892+0000","last_clean":"2024-04-16T21:07:49.655892+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:07:49.655892+0000","last_undegraded":"2024-04-16T21:07:49.655892+0000","last_fullsized":"2024-04-16T21:07:49.655892+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'33","reported_seq":183,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652615+0000","last_change":"2024-04-16T19:25:15.482185+0000","last_active":"2024-04-16T21:07:34.652615+0000","last_peered":"2024-04-16T21:07:34.652615+0000","last_clean":"2024-04-16T21:07:34.652615+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:07:34.652615+0000","last_undegraded":"2024-04-16T21:07:34.652615+0000","last_fullsized":"2024-04-16T21:07:34.652615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:07.940015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370057+0000","last_change":"2024-04-16T19:25:13.475047+0000","last_active":"2024-04-16T21:09:14.370057+0000","last_peered":"2024-04-16T21:09:14.370057+0000","last_clean":"2024-04-16T21:09:14.370057+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:09:14.370057+0000","last_undegraded":"2024-04-16T21:09:14.370057+0000","last_fullsized":"2024-04-16T21:09:14.370057+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:29:34.601553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1740","reported_seq":4601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370026+0000","last_change":"2024-04-16T19:25:11.468809+0000","last_active":"2024-04-16T21:09:14.370026+0000","last_peered":"2024-04-16T21:09:14.370026+0000","last_clean":"2024-04-16T21:09:14.370026+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:09:14.370026+0000","last_undegraded":"2024-04-16T21:09:14.370026+0000","last_fullsized":"2024-04-16T21:09:14.370026+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1740,"log_dups_size":0,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:53.762063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'19","reported_seq":270,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651251+0000","last_change":"2024-04-16T19:25:09.468894+0000","last_active":"2024-04-16T21:07:29.651251+0000","last_peered":"2024-04-16T21:07:29.651251+0000","last_clean":"2024-04-16T21:07:29.651251+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:07:29.651251+0000","last_undegraded":"2024-04-16T21:07:29.651251+0000","last_fullsized":"2024-04-16T21:07:29.651251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:05.430878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664164+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:08:39.664164+0000","last_peered":"2024-04-16T21:08:39.664164+0000","last_clean":"2024-04-16T21:08:39.664164+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:08:39.664164+0000","last_undegraded":"2024-04-16T21:08:39.664164+0000","last_fullsized":"2024-04-16T21:08:39.664164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660546+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:08:19.660546+0000","last_peered":"2024-04-16T21:08:19.660546+0000","last_clean":"2024-04-16T21:08:19.660546+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:08:19.660546+0000","last_undegraded":"2024-04-16T21:08:19.660546+0000","last_fullsized":"2024-04-16T21:08:19.660546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365365+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:08:49.365365+0000","last_peered":"2024-04-16T21:08:49.365365+0000","last_clean":"2024-04-16T21:08:49.365365+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:08:49.365365+0000","last_undegraded":"2024-04-16T21:08:49.365365+0000","last_fullsized":"2024-04-16T21:08:49.365365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:48.947818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":394,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368921+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:09:09.368921+0000","last_peered":"2024-04-16T21:09:09.368921+0000","last_clean":"2024-04-16T21:09:09.368921+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:09:09.368921+0000","last_undegraded":"2024-04-16T21:09:09.368921+0000","last_fullsized":"2024-04-16T21:09:09.368921+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:36.588167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501183+0000","last_change":"2024-04-16T19:24:49.434497+0000","last_active":"2024-04-16T21:08:19.501183+0000","last_peered":"2024-04-16T21:08:19.501183+0000","last_clean":"2024-04-16T21:08:19.501183+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:08:19.501183+0000","last_undegraded":"2024-04-16T21:08:19.501183+0000","last_fullsized":"2024-04-16T21:08:19.501183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:08.864273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368025+0000","last_change":"2024-04-16T19:24:51.437122+0000","last_active":"2024-04-16T21:09:04.368025+0000","last_peered":"2024-04-16T21:09:04.368025+0000","last_clean":"2024-04-16T21:09:04.368025+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:09:04.368025+0000","last_undegraded":"2024-04-16T21:09:04.368025+0000","last_fullsized":"2024-04-16T21:09:04.368025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:10.861306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506044+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:08:49.506044+0000","last_peered":"2024-04-16T21:08:49.506044+0000","last_clean":"2024-04-16T21:08:49.506044+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:08:49.506044+0000","last_undegraded":"2024-04-16T21:08:49.506044+0000","last_fullsized":"2024-04-16T21:08:49.506044+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:48:31.872270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1928,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655341+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:07:54.655341+0000","last_peered":"2024-04-16T21:07:54.655341+0000","last_clean":"2024-04-16T21:07:54.655341+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:07:54.655341+0000","last_undegraded":"2024-04-16T21:07:54.655341+0000","last_fullsized":"2024-04-16T21:07:54.655341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657746+0000","last_change":"2024-04-16T19:25:41.502674+0000","last_active":"2024-04-16T21:07:59.657746+0000","last_peered":"2024-04-16T21:07:59.657746+0000","last_clean":"2024-04-16T21:07:59.657746+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:07:59.657746+0000","last_undegraded":"2024-04-16T21:07:59.657746+0000","last_fullsized":"2024-04-16T21:07:59.657746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'13","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372599+0000","last_change":"2024-04-16T19:25:15.485057+0000","last_active":"2024-04-16T21:09:29.372599+0000","last_peered":"2024-04-16T21:09:29.372599+0000","last_clean":"2024-04-16T21:09:29.372599+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:09:29.372599+0000","last_undegraded":"2024-04-16T21:09:29.372599+0000","last_fullsized":"2024-04-16T21:09:29.372599+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:45.514212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374924+0000","last_change":"2024-04-16T19:25:13.472089+0000","last_active":"2024-04-16T21:09:14.374924+0000","last_peered":"2024-04-16T21:09:14.374924+0000","last_clean":"2024-04-16T21:09:14.374924+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:09:14.374924+0000","last_undegraded":"2024-04-16T21:09:14.374924+0000","last_fullsized":"2024-04-16T21:09:14.374924+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'304","reported_seq":629,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370404+0000","last_change":"2024-04-16T19:25:11.462997+0000","last_active":"2024-04-16T21:09:14.370404+0000","last_peered":"2024-04-16T21:09:14.370404+0000","last_clean":"2024-04-16T21:09:14.370404+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:09:14.370404+0000","last_undegraded":"2024-04-16T21:09:14.370404+0000","last_fullsized":"2024-04-16T21:09:14.370404+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:14.402830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'8","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369760+0000","last_change":"2024-04-16T19:25:09.467337+0000","last_active":"2024-04-16T21:09:19.369760+0000","last_peered":"2024-04-16T21:09:19.369760+0000","last_clean":"2024-04-16T21:09:19.369760+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:09:19.369760+0000","last_undegraded":"2024-04-16T21:09:19.369760+0000","last_fullsized":"2024-04-16T21:09:19.369760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663257+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:08:39.663257+0000","last_peered":"2024-04-16T21:08:39.663257+0000","last_clean":"2024-04-16T21:08:39.663257+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:08:39.663257+0000","last_undegraded":"2024-04-16T21:08:39.663257+0000","last_fullsized":"2024-04-16T21:08:39.663257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658066+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:08:04.658066+0000","last_peered":"2024-04-16T21:08:04.658066+0000","last_clean":"2024-04-16T21:08:04.658066+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:08:04.658066+0000","last_undegraded":"2024-04-16T21:08:04.658066+0000","last_fullsized":"2024-04-16T21:08:04.658066+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:20.908683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":959,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.505980+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:08:49.505980+0000","last_peered":"2024-04-16T21:08:49.505980+0000","last_clean":"2024-04-16T21:08:49.505980+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:08:49.505980+0000","last_undegraded":"2024-04-16T21:08:49.505980+0000","last_fullsized":"2024-04-16T21:08:49.505980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:35.897776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501172+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:08:19.501172+0000","last_peered":"2024-04-16T21:08:19.501172+0000","last_clean":"2024-04-16T21:08:19.501172+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:08:19.501172+0000","last_undegraded":"2024-04-16T21:08:19.501172+0000","last_fullsized":"2024-04-16T21:08:19.501172+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:40.145350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508432+0000","last_change":"2024-04-16T19:24:49.429526+0000","last_active":"2024-04-16T21:09:04.508432+0000","last_peered":"2024-04-16T21:09:04.508432+0000","last_clean":"2024-04-16T21:09:04.508432+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:09:04.508432+0000","last_undegraded":"2024-04-16T21:09:04.508432+0000","last_fullsized":"2024-04-16T21:09:04.508432+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:36.881378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371372+0000","last_change":"2024-04-16T19:24:51.437305+0000","last_active":"2024-04-16T21:09:24.371372+0000","last_peered":"2024-04-16T21:09:24.371372+0000","last_clean":"2024-04-16T21:09:24.371372+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:09:24.371372+0000","last_undegraded":"2024-04-16T21:09:24.371372+0000","last_fullsized":"2024-04-16T21:09:24.371372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:24.125422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366382+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:08:49.366382+0000","last_peered":"2024-04-16T21:08:49.366382+0000","last_clean":"2024-04-16T21:08:49.366382+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:08:49.366382+0000","last_undegraded":"2024-04-16T21:08:49.366382+0000","last_fullsized":"2024-04-16T21:08:49.366382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:22:33.730359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'93","reported_seq":1378,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498742+0000","last_change":"2024-04-16T19:26:41.564908+0000","last_active":"2024-04-16T21:08:04.498742+0000","last_peered":"2024-04-16T21:08:04.498742+0000","last_clean":"2024-04-16T21:08:04.498742+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:08:04.498742+0000","last_undegraded":"2024-04-16T21:08:04.498742+0000","last_fullsized":"2024-04-16T21:08:04.498742+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:47:23.710928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654443+0000","last_change":"2024-04-16T19:25:41.508929+0000","last_active":"2024-04-16T21:07:49.654443+0000","last_peered":"2024-04-16T21:07:49.654443+0000","last_clean":"2024-04-16T21:07:49.654443+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:07:49.654443+0000","last_undegraded":"2024-04-16T21:07:49.654443+0000","last_fullsized":"2024-04-16T21:07:49.654443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'19","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372668+0000","last_change":"2024-04-16T19:25:15.482026+0000","last_active":"2024-04-16T21:09:29.372668+0000","last_peered":"2024-04-16T21:09:29.372668+0000","last_clean":"2024-04-16T21:09:29.372668+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:09:29.372668+0000","last_undegraded":"2024-04-16T21:09:29.372668+0000","last_fullsized":"2024-04-16T21:09:29.372668+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652807+0000","last_change":"2024-04-16T19:25:13.475419+0000","last_active":"2024-04-16T21:07:34.652807+0000","last_peered":"2024-04-16T21:07:34.652807+0000","last_clean":"2024-04-16T21:07:34.652807+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:07:34.652807+0000","last_undegraded":"2024-04-16T21:07:34.652807+0000","last_fullsized":"2024-04-16T21:07:34.652807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2044","reported_seq":5814,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.376528+0000","last_change":"2024-04-16T19:25:11.468843+0000","last_active":"2024-04-16T21:09:24.376528+0000","last_peered":"2024-04-16T21:09:24.376528+0000","last_clean":"2024-04-16T21:09:24.376528+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:09:24.376528+0000","last_undegraded":"2024-04-16T21:09:24.376528+0000","last_fullsized":"2024-04-16T21:09:24.376528+0000","mapping_epoch":33,"log_start":"47'300","ondisk_log_start":"47'300","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1744,"log_dups_size":300,"ondisk_log_size":1744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:03.761996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'14","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369114+0000","last_change":"2024-04-16T19:25:09.467702+0000","last_active":"2024-04-16T21:09:09.369114+0000","last_peered":"2024-04-16T21:09:09.369114+0000","last_clean":"2024-04-16T21:09:09.369114+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:09:09.369114+0000","last_undegraded":"2024-04-16T21:09:09.369114+0000","last_fullsized":"2024-04-16T21:09:09.369114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:31.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367115+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:08:59.367115+0000","last_peered":"2024-04-16T21:08:59.367115+0000","last_clean":"2024-04-16T21:08:59.367115+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:08:59.367115+0000","last_undegraded":"2024-04-16T21:08:59.367115+0000","last_fullsized":"2024-04-16T21:08:59.367115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:40.677584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659023+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:08:04.659023+0000","last_peered":"2024-04-16T21:08:04.659023+0000","last_clean":"2024-04-16T21:08:04.659023+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:08:04.659023+0000","last_undegraded":"2024-04-16T21:08:04.659023+0000","last_fullsized":"2024-04-16T21:08:04.659023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:49.358784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500796+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:08:19.500796+0000","last_peered":"2024-04-16T21:08:19.500796+0000","last_clean":"2024-04-16T21:08:19.500796+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:08:19.500796+0000","last_undegraded":"2024-04-16T21:08:19.500796+0000","last_fullsized":"2024-04-16T21:08:19.500796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:59.160445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":328,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366327+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:08:49.366327+0000","last_peered":"2024-04-16T21:08:49.366327+0000","last_clean":"2024-04-16T21:08:49.366327+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:08:49.366327+0000","last_undegraded":"2024-04-16T21:08:49.366327+0000","last_fullsized":"2024-04-16T21:08:49.366327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1313,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370434+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:08:49.370434+0000","last_peered":"2024-04-16T21:08:49.370434+0000","last_clean":"2024-04-16T21:08:49.370434+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:08:49.370434+0000","last_undegraded":"2024-04-16T21:08:49.370434+0000","last_fullsized":"2024-04-16T21:08:49.370434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:30.260652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371475+0000","last_change":"2024-04-16T19:24:51.437991+0000","last_active":"2024-04-16T21:08:54.371475+0000","last_peered":"2024-04-16T21:08:54.371475+0000","last_clean":"2024-04-16T21:08:54.371475+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:08:54.371475+0000","last_undegraded":"2024-04-16T21:08:54.371475+0000","last_fullsized":"2024-04-16T21:08:54.371475+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:43:50.519739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496442+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:07:49.496442+0000","last_peered":"2024-04-16T21:07:49.496442+0000","last_clean":"2024-04-16T21:07:49.496442+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:07:49.496442+0000","last_undegraded":"2024-04-16T21:07:49.496442+0000","last_fullsized":"2024-04-16T21:07:49.496442+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:16:07.995736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'136","reported_seq":1889,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497219+0000","last_change":"2024-04-16T19:25:49.523163+0000","last_active":"2024-04-16T21:07:54.497219+0000","last_peered":"2024-04-16T21:07:54.497219+0000","last_clean":"2024-04-16T21:07:54.497219+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:07:54.497219+0000","last_undegraded":"2024-04-16T21:07:54.497219+0000","last_fullsized":"2024-04-16T21:07:54.497219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:08.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655310+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:07:49.655310+0000","last_peered":"2024-04-16T21:07:49.655310+0000","last_clean":"2024-04-16T21:07:49.655310+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:07:49.655310+0000","last_undegraded":"2024-04-16T21:07:49.655310+0000","last_fullsized":"2024-04-16T21:07:49.655310+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:36.879550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'11","reported_seq":150,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.377292+0000","last_change":"2024-04-16T19:25:15.480717+0000","last_active":"2024-04-16T21:09:29.377292+0000","last_peered":"2024-04-16T21:09:29.377292+0000","last_clean":"2024-04-16T21:09:29.377292+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:09:29.377292+0000","last_undegraded":"2024-04-16T21:09:29.377292+0000","last_fullsized":"2024-04-16T21:09:29.377292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:39.031938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371460+0000","last_change":"2024-04-16T19:25:13.476070+0000","last_active":"2024-04-16T21:09:24.371460+0000","last_peered":"2024-04-16T21:09:24.371460+0000","last_clean":"2024-04-16T21:09:24.371460+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:09:24.371460+0000","last_undegraded":"2024-04-16T21:09:24.371460+0000","last_fullsized":"2024-04-16T21:09:24.371460+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:42.684942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'341","reported_seq":664,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494066+0000","last_change":"2024-04-16T19:25:11.467316+0000","last_active":"2024-04-16T21:07:34.494066+0000","last_peered":"2024-04-16T21:07:34.494066+0000","last_clean":"2024-04-16T21:07:34.494066+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:07:34.494066+0000","last_undegraded":"2024-04-16T21:07:34.494066+0000","last_fullsized":"2024-04-16T21:07:34.494066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'9","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369072+0000","last_change":"2024-04-16T19:25:09.470409+0000","last_active":"2024-04-16T21:09:09.369072+0000","last_peered":"2024-04-16T21:09:09.369072+0000","last_clean":"2024-04-16T21:09:09.369072+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:09:09.369072+0000","last_undegraded":"2024-04-16T21:09:09.369072+0000","last_fullsized":"2024-04-16T21:09:09.369072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657857+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:08:09.657857+0000","last_peered":"2024-04-16T21:08:09.657857+0000","last_clean":"2024-04-16T21:08:09.657857+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:08:09.657857+0000","last_undegraded":"2024-04-16T21:08:09.657857+0000","last_fullsized":"2024-04-16T21:08:09.657857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658003+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:08:04.658003+0000","last_peered":"2024-04-16T21:08:04.658003+0000","last_clean":"2024-04-16T21:08:04.658003+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:08:04.658003+0000","last_undegraded":"2024-04-16T21:08:04.658003+0000","last_fullsized":"2024-04-16T21:08:04.658003+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:29.699119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":457,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508414+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:09:04.508414+0000","last_peered":"2024-04-16T21:09:04.508414+0000","last_clean":"2024-04-16T21:09:04.508414+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:09:04.508414+0000","last_undegraded":"2024-04-16T21:09:04.508414+0000","last_fullsized":"2024-04-16T21:09:04.508414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:12.039717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":246,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370400+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:08:49.370400+0000","last_peered":"2024-04-16T21:08:49.370400+0000","last_clean":"2024-04-16T21:08:49.370400+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:08:49.370400+0000","last_undegraded":"2024-04-16T21:08:49.370400+0000","last_fullsized":"2024-04-16T21:08:49.370400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:16.109122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506003+0000","last_change":"2024-04-16T19:24:49.432294+0000","last_active":"2024-04-16T21:08:49.506003+0000","last_peered":"2024-04-16T21:08:49.506003+0000","last_clean":"2024-04-16T21:08:49.506003+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:08:49.506003+0000","last_undegraded":"2024-04-16T21:08:49.506003+0000","last_fullsized":"2024-04-16T21:08:49.506003+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:45.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365802+0000","last_change":"2024-04-16T19:24:51.442478+0000","last_active":"2024-04-16T21:08:54.365802+0000","last_peered":"2024-04-16T21:08:54.365802+0000","last_clean":"2024-04-16T21:08:54.365802+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:08:54.365802+0000","last_undegraded":"2024-04-16T21:08:54.365802+0000","last_fullsized":"2024-04-16T21:08:54.365802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495710+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:07:44.495710+0000","last_peered":"2024-04-16T21:07:44.495710+0000","last_clean":"2024-04-16T21:07:44.495710+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:07:44.495710+0000","last_undegraded":"2024-04-16T21:07:44.495710+0000","last_fullsized":"2024-04-16T21:07:44.495710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:02:07.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2345,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652944+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:07:39.652944+0000","last_peered":"2024-04-16T21:07:39.652944+0000","last_clean":"2024-04-16T21:07:39.652944+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:07:39.652944+0000","last_undegraded":"2024-04-16T21:07:39.652944+0000","last_fullsized":"2024-04-16T21:07:39.652944+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658027+0000","last_change":"2024-04-16T19:25:41.509053+0000","last_active":"2024-04-16T21:08:04.658027+0000","last_peered":"2024-04-16T21:08:04.658027+0000","last_clean":"2024-04-16T21:08:04.658027+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:08:04.658027+0000","last_undegraded":"2024-04-16T21:08:04.658027+0000","last_fullsized":"2024-04-16T21:08:04.658027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:22.621386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"47'8","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372308+0000","last_change":"2024-04-16T19:25:15.486624+0000","last_active":"2024-04-16T21:09:29.372308+0000","last_peered":"2024-04-16T21:09:29.372308+0000","last_clean":"2024-04-16T21:09:29.372308+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:09:29.372308+0000","last_undegraded":"2024-04-16T21:09:29.372308+0000","last_fullsized":"2024-04-16T21:09:29.372308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:35.616571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'705","reported_seq":2065,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371841+0000","last_change":"2024-04-16T19:25:13.473118+0000","last_active":"2024-04-16T21:09:24.371841+0000","last_peered":"2024-04-16T21:09:24.371841+0000","last_clean":"2024-04-16T21:09:24.371841+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:09:24.371841+0000","last_undegraded":"2024-04-16T21:09:24.371841+0000","last_fullsized":"2024-04-16T21:09:24.371841+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'269","reported_seq":541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.370624+0000","last_change":"2024-04-16T19:25:11.465450+0000","last_active":"2024-04-16T21:09:24.370624+0000","last_peered":"2024-04-16T21:09:24.370624+0000","last_clean":"2024-04-16T21:09:24.370624+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:09:24.370624+0000","last_undegraded":"2024-04-16T21:09:24.370624+0000","last_fullsized":"2024-04-16T21:09:24.370624+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:53.479823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'5","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375596+0000","last_change":"2024-04-16T19:25:09.458268+0000","last_active":"2024-04-16T21:09:19.375596+0000","last_peered":"2024-04-16T21:09:19.375596+0000","last_clean":"2024-04-16T21:09:19.375596+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:09:19.375596+0000","last_undegraded":"2024-04-16T21:09:19.375596+0000","last_fullsized":"2024-04-16T21:09:19.375596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:05.252198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658783+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:08:09.658783+0000","last_peered":"2024-04-16T21:08:09.658783+0000","last_clean":"2024-04-16T21:08:09.658783+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:08:09.658783+0000","last_undegraded":"2024-04-16T21:08:09.658783+0000","last_fullsized":"2024-04-16T21:08:09.658783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:29.013265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498840+0000","last_change":"2024-04-16T19:25:41.503885+0000","last_active":"2024-04-16T21:08:04.498840+0000","last_peered":"2024-04-16T21:08:04.498840+0000","last_clean":"2024-04-16T21:08:04.498840+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:08:04.498840+0000","last_undegraded":"2024-04-16T21:08:04.498840+0000","last_fullsized":"2024-04-16T21:08:04.498840+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:54.516253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500846+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:08:19.500846+0000","last_peered":"2024-04-16T21:08:19.500846+0000","last_clean":"2024-04-16T21:08:19.500846+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:08:19.500846+0000","last_undegraded":"2024-04-16T21:08:19.500846+0000","last_fullsized":"2024-04-16T21:08:19.500846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:30.224132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660316+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:08:19.660316+0000","last_peered":"2024-04-16T21:08:19.660316+0000","last_clean":"2024-04-16T21:08:19.660316+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:08:19.660316+0000","last_undegraded":"2024-04-16T21:08:19.660316+0000","last_fullsized":"2024-04-16T21:08:19.660316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:43.606761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662124+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:08:34.662124+0000","last_peered":"2024-04-16T21:08:34.662124+0000","last_clean":"2024-04-16T21:08:34.662124+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:08:34.662124+0000","last_undegraded":"2024-04-16T21:08:34.662124+0000","last_fullsized":"2024-04-16T21:08:34.662124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368874+0000","last_change":"2024-04-16T19:24:51.434984+0000","last_active":"2024-04-16T21:09:14.368874+0000","last_peered":"2024-04-16T21:09:14.368874+0000","last_clean":"2024-04-16T21:09:14.368874+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:09:14.368874+0000","last_undegraded":"2024-04-16T21:09:14.368874+0000","last_fullsized":"2024-04-16T21:09:14.368874+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:26.835802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653730+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:07:44.653730+0000","last_peered":"2024-04-16T21:07:44.653730+0000","last_clean":"2024-04-16T21:07:44.653730+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:07:44.653730+0000","last_undegraded":"2024-04-16T21:07:44.653730+0000","last_fullsized":"2024-04-16T21:07:44.653730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:56.544746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1678,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653711+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:07:39.653711+0000","last_peered":"2024-04-16T21:07:39.653711+0000","last_clean":"2024-04-16T21:07:39.653711+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:07:39.653711+0000","last_undegraded":"2024-04-16T21:07:39.653711+0000","last_fullsized":"2024-04-16T21:07:39.653711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:01.314955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655387+0000","last_change":"2024-04-16T19:25:41.504293+0000","last_active":"2024-04-16T21:07:49.655387+0000","last_peered":"2024-04-16T21:07:49.655387+0000","last_clean":"2024-04-16T21:07:49.655387+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:07:49.655387+0000","last_undegraded":"2024-04-16T21:07:49.655387+0000","last_fullsized":"2024-04-16T21:07:49.655387+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'136","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654370+0000","last_change":"2024-04-16T19:25:15.483254+0000","last_active":"2024-04-16T21:07:39.654370+0000","last_peered":"2024-04-16T21:07:39.654370+0000","last_clean":"2024-04-16T21:07:39.654370+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:07:39.654370+0000","last_undegraded":"2024-04-16T21:07:39.654370+0000","last_fullsized":"2024-04-16T21:07:39.654370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511597+0000","last_change":"2024-04-16T19:25:13.471760+0000","last_active":"2024-04-16T21:09:24.511597+0000","last_peered":"2024-04-16T21:09:24.511597+0000","last_clean":"2024-04-16T21:09:24.511597+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:09:24.511597+0000","last_undegraded":"2024-04-16T21:09:24.511597+0000","last_fullsized":"2024-04-16T21:09:24.511597+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'347","reported_seq":725,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511636+0000","last_change":"2024-04-16T19:25:11.467463+0000","last_active":"2024-04-16T21:09:24.511636+0000","last_peered":"2024-04-16T21:09:24.511636+0000","last_clean":"2024-04-16T21:09:24.511636+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:09:24.511636+0000","last_undegraded":"2024-04-16T21:09:24.511636+0000","last_fullsized":"2024-04-16T21:09:24.511636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'10","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.371455+0000","last_change":"2024-04-16T19:25:09.461452+0000","last_active":"2024-04-16T21:09:29.371455+0000","last_peered":"2024-04-16T21:09:29.371455+0000","last_clean":"2024-04-16T21:09:29.371455+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:09:29.371455+0000","last_undegraded":"2024-04-16T21:09:29.371455+0000","last_fullsized":"2024-04-16T21:09:29.371455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:44:08.538156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659737+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:08:09.659737+0000","last_peered":"2024-04-16T21:08:09.659737+0000","last_clean":"2024-04-16T21:08:09.659737+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:08:09.659737+0000","last_undegraded":"2024-04-16T21:08:09.659737+0000","last_fullsized":"2024-04-16T21:08:09.659737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496482+0000","last_change":"2024-04-16T19:25:41.506323+0000","last_active":"2024-04-16T21:07:49.496482+0000","last_peered":"2024-04-16T21:07:49.496482+0000","last_clean":"2024-04-16T21:07:49.496482+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:07:49.496482+0000","last_undegraded":"2024-04-16T21:07:49.496482+0000","last_fullsized":"2024-04-16T21:07:49.496482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:05.815658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501233+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:08:19.501233+0000","last_peered":"2024-04-16T21:08:19.501233+0000","last_clean":"2024-04-16T21:08:19.501233+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:08:19.501233+0000","last_undegraded":"2024-04-16T21:08:19.501233+0000","last_fullsized":"2024-04-16T21:08:19.501233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:51.524577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'140","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503630+0000","last_change":"2024-04-16T19:24:47.426406+0000","last_active":"2024-04-16T21:08:34.503630+0000","last_peered":"2024-04-16T21:08:34.503630+0000","last_clean":"2024-04-16T21:08:34.503630+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:08:34.503630+0000","last_undegraded":"2024-04-16T21:08:34.503630+0000","last_fullsized":"2024-04-16T21:08:34.503630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:12.378311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500867+0000","last_change":"2024-04-16T19:24:49.430773+0000","last_active":"2024-04-16T21:08:19.500867+0000","last_peered":"2024-04-16T21:08:19.500867+0000","last_clean":"2024-04-16T21:08:19.500867+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:08:19.500867+0000","last_undegraded":"2024-04-16T21:08:19.500867+0000","last_fullsized":"2024-04-16T21:08:19.500867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:52.965094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367299+0000","last_change":"2024-04-16T19:24:51.436754+0000","last_active":"2024-04-16T21:09:04.367299+0000","last_peered":"2024-04-16T21:09:04.367299+0000","last_clean":"2024-04-16T21:09:04.367299+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:09:04.367299+0000","last_undegraded":"2024-04-16T21:09:04.367299+0000","last_fullsized":"2024-04-16T21:09:04.367299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:01.678207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368731+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:09:04.368731+0000","last_peered":"2024-04-16T21:09:04.368731+0000","last_clean":"2024-04-16T21:09:04.368731+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:09:04.368731+0000","last_undegraded":"2024-04-16T21:09:04.368731+0000","last_fullsized":"2024-04-16T21:09:04.368731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:01:32.497457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1870,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656880+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:07:54.656880+0000","last_peered":"2024-04-16T21:07:54.656880+0000","last_clean":"2024-04-16T21:07:54.656880+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:07:54.656880+0000","last_undegraded":"2024-04-16T21:07:54.656880+0000","last_fullsized":"2024-04-16T21:07:54.656880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653650+0000","last_change":"2024-04-16T19:25:41.507626+0000","last_active":"2024-04-16T21:07:34.653650+0000","last_peered":"2024-04-16T21:07:34.653650+0000","last_clean":"2024-04-16T21:07:34.653650+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:07:34.653650+0000","last_undegraded":"2024-04-16T21:07:34.653650+0000","last_fullsized":"2024-04-16T21:07:34.653650+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'7","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369826+0000","last_change":"2024-04-16T19:25:15.477722+0000","last_active":"2024-04-16T21:09:19.369826+0000","last_peered":"2024-04-16T21:09:19.369826+0000","last_clean":"2024-04-16T21:09:19.369826+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:09:19.369826+0000","last_undegraded":"2024-04-16T21:09:19.369826+0000","last_fullsized":"2024-04-16T21:09:19.369826+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:11.553471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371260+0000","last_change":"2024-04-16T19:25:13.476043+0000","last_active":"2024-04-16T21:09:24.371260+0000","last_peered":"2024-04-16T21:09:24.371260+0000","last_clean":"2024-04-16T21:09:24.371260+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:09:24.371260+0000","last_undegraded":"2024-04-16T21:09:24.371260+0000","last_fullsized":"2024-04-16T21:09:24.371260+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'453","reported_seq":869,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.373003+0000","last_change":"2024-04-16T19:25:11.462559+0000","last_active":"2024-04-16T21:09:34.373003+0000","last_peered":"2024-04-16T21:09:34.373003+0000","last_clean":"2024-04-16T21:09:34.373003+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:09:34.373003+0000","last_undegraded":"2024-04-16T21:09:34.373003+0000","last_fullsized":"2024-04-16T21:09:34.373003+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:33.939814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'8","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655202+0000","last_change":"2024-04-16T19:25:41.507771+0000","last_active":"2024-04-16T21:07:49.655202+0000","last_peered":"2024-04-16T21:07:49.655202+0000","last_clean":"2024-04-16T21:07:49.655202+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:07:49.655202+0000","last_undegraded":"2024-04-16T21:07:49.655202+0000","last_fullsized":"2024-04-16T21:07:49.655202+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:53.575807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658859+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:08:09.658859+0000","last_peered":"2024-04-16T21:08:09.658859+0000","last_clean":"2024-04-16T21:08:09.658859+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:08:09.658859+0000","last_undegraded":"2024-04-16T21:08:09.658859+0000","last_fullsized":"2024-04-16T21:08:09.658859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662590+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:08:34.662590+0000","last_peered":"2024-04-16T21:08:34.662590+0000","last_clean":"2024-04-16T21:08:34.662590+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:08:34.662590+0000","last_undegraded":"2024-04-16T21:08:34.662590+0000","last_fullsized":"2024-04-16T21:08:34.662590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1361,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663160+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:08:34.663160+0000","last_peered":"2024-04-16T21:08:34.663160+0000","last_clean":"2024-04-16T21:08:34.663160+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:08:34.663160+0000","last_undegraded":"2024-04-16T21:08:34.663160+0000","last_fullsized":"2024-04-16T21:08:34.663160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:46.278655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":276,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501256+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:08:19.501256+0000","last_peered":"2024-04-16T21:08:19.501256+0000","last_clean":"2024-04-16T21:08:19.501256+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:08:19.501256+0000","last_undegraded":"2024-04-16T21:08:19.501256+0000","last_fullsized":"2024-04-16T21:08:19.501256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:23.115125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660267+0000","last_change":"2024-04-16T19:24:49.428502+0000","last_active":"2024-04-16T21:08:19.660267+0000","last_peered":"2024-04-16T21:08:19.660267+0000","last_clean":"2024-04-16T21:08:19.660267+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:08:19.660267+0000","last_undegraded":"2024-04-16T21:08:19.660267+0000","last_fullsized":"2024-04-16T21:08:19.660267+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:25.122206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367884+0000","last_change":"2024-04-16T19:24:51.435114+0000","last_active":"2024-04-16T21:09:04.367884+0000","last_peered":"2024-04-16T21:09:04.367884+0000","last_clean":"2024-04-16T21:09:04.367884+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:09:04.367884+0000","last_undegraded":"2024-04-16T21:09:04.367884+0000","last_fullsized":"2024-04-16T21:09:04.367884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:56.562306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364938+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:08:49.364938+0000","last_peered":"2024-04-16T21:08:49.364938+0000","last_clean":"2024-04-16T21:08:49.364938+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:08:49.364938+0000","last_undegraded":"2024-04-16T21:08:49.364938+0000","last_fullsized":"2024-04-16T21:08:49.364938+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:08:56.543210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657898+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:08:09.657898+0000","last_peered":"2024-04-16T21:08:09.657898+0000","last_clean":"2024-04-16T21:08:09.657898+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:08:09.657898+0000","last_undegraded":"2024-04-16T21:08:09.657898+0000","last_fullsized":"2024-04-16T21:08:09.657898+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:44:18.098055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653619+0000","last_change":"2024-04-16T19:25:41.506250+0000","last_active":"2024-04-16T21:07:34.653619+0000","last_peered":"2024-04-16T21:07:34.653619+0000","last_clean":"2024-04-16T21:07:34.653619+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:07:34.653619+0000","last_undegraded":"2024-04-16T21:07:34.653619+0000","last_fullsized":"2024-04-16T21:07:34.653619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369866+0000","last_change":"2024-04-16T19:25:15.479905+0000","last_active":"2024-04-16T21:09:19.369866+0000","last_peered":"2024-04-16T21:09:19.369866+0000","last_clean":"2024-04-16T21:09:19.369866+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:09:19.369866+0000","last_undegraded":"2024-04-16T21:09:19.369866+0000","last_fullsized":"2024-04-16T21:09:19.369866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:03.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'705","reported_seq":2464,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.372193+0000","last_change":"2024-04-16T19:25:13.470216+0000","last_active":"2024-04-16T21:09:34.372193+0000","last_peered":"2024-04-16T21:09:34.372193+0000","last_clean":"2024-04-16T21:09:34.372193+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:09:34.372193+0000","last_undegraded":"2024-04-16T21:09:34.372193+0000","last_fullsized":"2024-04-16T21:09:34.372193+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:40.891478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'311","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371233+0000","last_change":"2024-04-16T19:25:11.464746+0000","last_active":"2024-04-16T21:09:24.371233+0000","last_peered":"2024-04-16T21:09:24.371233+0000","last_clean":"2024-04-16T21:09:24.371233+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:09:24.371233+0000","last_undegraded":"2024-04-16T21:09:24.371233+0000","last_fullsized":"2024-04-16T21:09:24.371233+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'6","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369697+0000","last_change":"2024-04-16T19:25:09.466128+0000","last_active":"2024-04-16T21:09:19.369697+0000","last_peered":"2024-04-16T21:09:19.369697+0000","last_clean":"2024-04-16T21:09:19.369697+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:09:19.369697+0000","last_undegraded":"2024-04-16T21:09:19.369697+0000","last_fullsized":"2024-04-16T21:09:19.369697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.502055+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:08:24.502055+0000","last_peered":"2024-04-16T21:08:24.502055+0000","last_clean":"2024-04-16T21:08:24.502055+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:08:24.502055+0000","last_undegraded":"2024-04-16T21:08:24.502055+0000","last_fullsized":"2024-04-16T21:08:24.502055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:20.204370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503320+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:08:34.503320+0000","last_peered":"2024-04-16T21:08:34.503320+0000","last_clean":"2024-04-16T21:08:34.503320+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:08:34.503320+0000","last_undegraded":"2024-04-16T21:08:34.503320+0000","last_fullsized":"2024-04-16T21:08:34.503320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:08:26.977348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661224+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:08:19.661224+0000","last_peered":"2024-04-16T21:08:19.661224+0000","last_clean":"2024-04-16T21:08:19.661224+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:08:19.661224+0000","last_undegraded":"2024-04-16T21:08:19.661224+0000","last_fullsized":"2024-04-16T21:08:19.661224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:45.288392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":359,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660463+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:08:19.660463+0000","last_peered":"2024-04-16T21:08:19.660463+0000","last_clean":"2024-04-16T21:08:19.660463+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:08:19.660463+0000","last_undegraded":"2024-04-16T21:08:19.660463+0000","last_fullsized":"2024-04-16T21:08:19.660463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:48.207837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661146+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:08:19.661146+0000","last_peered":"2024-04-16T21:08:19.661146+0000","last_clean":"2024-04-16T21:08:19.661146+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:08:19.661146+0000","last_undegraded":"2024-04-16T21:08:19.661146+0000","last_fullsized":"2024-04-16T21:08:19.661146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367334+0000","last_change":"2024-04-16T19:24:51.434704+0000","last_active":"2024-04-16T21:09:04.367334+0000","last_peered":"2024-04-16T21:09:04.367334+0000","last_clean":"2024-04-16T21:09:04.367334+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:09:04.367334+0000","last_undegraded":"2024-04-16T21:09:04.367334+0000","last_fullsized":"2024-04-16T21:09:04.367334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503573+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:08:34.503573+0000","last_peered":"2024-04-16T21:08:34.503573+0000","last_clean":"2024-04-16T21:08:34.503573+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:08:34.503573+0000","last_undegraded":"2024-04-16T21:08:34.503573+0000","last_fullsized":"2024-04-16T21:08:34.503573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:11.491071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'162","reported_seq":2095,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650676+0000","last_change":"2024-04-16T19:25:49.518828+0000","last_active":"2024-04-16T21:07:19.650676+0000","last_peered":"2024-04-16T21:07:19.650676+0000","last_clean":"2024-04-16T21:07:19.650676+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:07:19.650676+0000","last_undegraded":"2024-04-16T21:07:19.650676+0000","last_fullsized":"2024-04-16T21:07:19.650676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:22.736467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368970+0000","last_change":"2024-04-16T19:25:41.510329+0000","last_active":"2024-04-16T21:09:14.368970+0000","last_peered":"2024-04-16T21:09:14.368970+0000","last_clean":"2024-04-16T21:09:14.368970+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:09:14.368970+0000","last_undegraded":"2024-04-16T21:09:14.368970+0000","last_fullsized":"2024-04-16T21:09:14.368970+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:14.723662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'12","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365503+0000","last_change":"2024-04-16T19:25:15.480613+0000","last_active":"2024-04-16T21:08:49.365503+0000","last_peered":"2024-04-16T21:08:49.365503+0000","last_clean":"2024-04-16T21:08:49.365503+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:08:49.365503+0000","last_undegraded":"2024-04-16T21:08:49.365503+0000","last_fullsized":"2024-04-16T21:08:49.365503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491622+0000","last_change":"2024-04-16T19:25:13.469639+0000","last_active":"2024-04-16T21:07:19.491622+0000","last_peered":"2024-04-16T21:07:19.491622+0000","last_clean":"2024-04-16T21:07:19.491622+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:07:19.491622+0000","last_undegraded":"2024-04-16T21:07:19.491622+0000","last_fullsized":"2024-04-16T21:07:19.491622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:02.547322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'256","reported_seq":566,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364495+0000","last_change":"2024-04-16T19:25:11.463503+0000","last_active":"2024-04-16T21:08:44.364495+0000","last_peered":"2024-04-16T21:08:44.364495+0000","last_clean":"2024-04-16T21:08:44.364495+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:08:44.364495+0000","last_undegraded":"2024-04-16T21:08:44.364495+0000","last_fullsized":"2024-04-16T21:08:44.364495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:07.258240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'6","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509214+0000","last_change":"2024-04-16T19:25:09.462147+0000","last_active":"2024-04-16T21:09:09.509214+0000","last_peered":"2024-04-16T21:09:09.509214+0000","last_clean":"2024-04-16T21:09:09.509214+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:09:09.509214+0000","last_undegraded":"2024-04-16T21:09:09.509214+0000","last_fullsized":"2024-04-16T21:09:09.509214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369516+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:09:09.369516+0000","last_peered":"2024-04-16T21:09:09.369516+0000","last_clean":"2024-04-16T21:09:09.369516+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:09:09.369516+0000","last_undegraded":"2024-04-16T21:09:09.369516+0000","last_fullsized":"2024-04-16T21:09:09.369516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:22.844339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658852+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:08:04.658852+0000","last_peered":"2024-04-16T21:08:04.658852+0000","last_clean":"2024-04-16T21:08:04.658852+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:08:04.658852+0000","last_undegraded":"2024-04-16T21:08:04.658852+0000","last_fullsized":"2024-04-16T21:08:04.658852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657180+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:07:59.657180+0000","last_peered":"2024-04-16T21:07:59.657180+0000","last_clean":"2024-04-16T21:07:59.657180+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:07:59.657180+0000","last_undegraded":"2024-04-16T21:07:59.657180+0000","last_fullsized":"2024-04-16T21:07:59.657180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.363997+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:08:44.363997+0000","last_peered":"2024-04-16T21:08:44.363997+0000","last_clean":"2024-04-16T21:08:44.363997+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:08:44.363997+0000","last_undegraded":"2024-04-16T21:08:44.363997+0000","last_fullsized":"2024-04-16T21:08:44.363997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367212+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:08:59.367212+0000","last_peered":"2024-04-16T21:08:59.367212+0000","last_clean":"2024-04-16T21:08:59.367212+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:08:59.367212+0000","last_undegraded":"2024-04-16T21:08:59.367212+0000","last_fullsized":"2024-04-16T21:08:59.367212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:49.663208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365555+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:08:49.365555+0000","last_peered":"2024-04-16T21:08:49.365555+0000","last_clean":"2024-04-16T21:08:49.365555+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:08:49.365555+0000","last_undegraded":"2024-04-16T21:08:49.365555+0000","last_fullsized":"2024-04-16T21:08:49.365555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370592+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:08:49.370592+0000","last_peered":"2024-04-16T21:08:49.370592+0000","last_clean":"2024-04-16T21:08:49.370592+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:08:49.370592+0000","last_undegraded":"2024-04-16T21:08:49.370592+0000","last_fullsized":"2024-04-16T21:08:49.370592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366370+0000","last_change":"2024-04-16T19:24:51.442576+0000","last_active":"2024-04-16T21:08:54.366370+0000","last_peered":"2024-04-16T21:08:54.366370+0000","last_clean":"2024-04-16T21:08:54.366370+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:08:54.366370+0000","last_undegraded":"2024-04-16T21:08:54.366370+0000","last_fullsized":"2024-04-16T21:08:54.366370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662725+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:08:34.662725+0000","last_peered":"2024-04-16T21:08:34.662725+0000","last_clean":"2024-04-16T21:08:34.662725+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:08:34.662725+0000","last_undegraded":"2024-04-16T21:08:34.662725+0000","last_fullsized":"2024-04-16T21:08:34.662725+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:34:27.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1951,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375667+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:09:19.375667+0000","last_peered":"2024-04-16T21:09:19.375667+0000","last_clean":"2024-04-16T21:09:19.375667+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:09:19.375667+0000","last_undegraded":"2024-04-16T21:09:19.375667+0000","last_fullsized":"2024-04-16T21:09:19.375667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:42.651587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649772+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:07:14.649772+0000","last_peered":"2024-04-16T21:07:14.649772+0000","last_clean":"2024-04-16T21:07:14.649772+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:07:14.649772+0000","last_undegraded":"2024-04-16T21:07:14.649772+0000","last_fullsized":"2024-04-16T21:07:14.649772+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'13","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370574+0000","last_change":"2024-04-16T19:25:15.485212+0000","last_active":"2024-04-16T21:08:49.370574+0000","last_peered":"2024-04-16T21:08:49.370574+0000","last_clean":"2024-04-16T21:08:49.370574+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:08:49.370574+0000","last_undegraded":"2024-04-16T21:08:49.370574+0000","last_fullsized":"2024-04-16T21:08:49.370574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:41.072415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'705","reported_seq":2077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509982+0000","last_change":"2024-04-16T19:25:13.467216+0000","last_active":"2024-04-16T21:09:14.509982+0000","last_peered":"2024-04-16T21:09:14.509982+0000","last_clean":"2024-04-16T21:09:14.509982+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:09:14.509982+0000","last_undegraded":"2024-04-16T21:09:14.509982+0000","last_fullsized":"2024-04-16T21:09:14.509982+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'306","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374758+0000","last_change":"2024-04-16T19:25:11.467288+0000","last_active":"2024-04-16T21:09:14.374758+0000","last_peered":"2024-04-16T21:09:14.374758+0000","last_clean":"2024-04-16T21:09:14.374758+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:09:14.374758+0000","last_undegraded":"2024-04-16T21:09:14.374758+0000","last_fullsized":"2024-04-16T21:09:14.374758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'12","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662672+0000","last_change":"2024-04-16T19:25:09.464018+0000","last_active":"2024-04-16T21:08:39.662672+0000","last_peered":"2024-04-16T21:08:39.662672+0000","last_clean":"2024-04-16T21:08:39.662672+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:08:39.662672+0000","last_undegraded":"2024-04-16T21:08:39.662672+0000","last_fullsized":"2024-04-16T21:08:39.662672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:52.913431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662788+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:08:39.662788+0000","last_peered":"2024-04-16T21:08:39.662788+0000","last_clean":"2024-04-16T21:08:39.662788+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:08:39.662788+0000","last_undegraded":"2024-04-16T21:08:39.662788+0000","last_fullsized":"2024-04-16T21:08:39.662788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658875+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:08:04.658875+0000","last_peered":"2024-04-16T21:08:04.658875+0000","last_clean":"2024-04-16T21:08:04.658875+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:08:04.658875+0000","last_undegraded":"2024-04-16T21:08:04.658875+0000","last_fullsized":"2024-04-16T21:08:04.658875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:26.452728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369665+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:08:44.369665+0000","last_peered":"2024-04-16T21:08:44.369665+0000","last_clean":"2024-04-16T21:08:44.369665+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:08:44.369665+0000","last_undegraded":"2024-04-16T21:08:44.369665+0000","last_fullsized":"2024-04-16T21:08:44.369665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:23.643674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":591,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365570+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:08:49.365570+0000","last_peered":"2024-04-16T21:08:49.365570+0000","last_clean":"2024-04-16T21:08:49.365570+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:08:49.365570+0000","last_undegraded":"2024-04-16T21:08:49.365570+0000","last_fullsized":"2024-04-16T21:08:49.365570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:34.895652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366657+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:08:59.366657+0000","last_peered":"2024-04-16T21:08:59.366657+0000","last_clean":"2024-04-16T21:08:59.366657+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:08:59.366657+0000","last_undegraded":"2024-04-16T21:08:59.366657+0000","last_fullsized":"2024-04-16T21:08:59.366657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:28.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370637+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:09:19.370637+0000","last_peered":"2024-04-16T21:09:19.370637+0000","last_clean":"2024-04-16T21:09:19.370637+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:09:19.370637+0000","last_undegraded":"2024-04-16T21:09:19.370637+0000","last_fullsized":"2024-04-16T21:09:19.370637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.152431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368921+0000","last_change":"2024-04-16T19:24:51.574513+0000","last_active":"2024-04-16T21:09:14.368921+0000","last_peered":"2024-04-16T21:09:14.368921+0000","last_clean":"2024-04-16T21:09:14.368921+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:09:14.368921+0000","last_undegraded":"2024-04-16T21:09:14.368921+0000","last_fullsized":"2024-04-16T21:09:14.368921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:32.268529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663021+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:08:34.663021+0000","last_peered":"2024-04-16T21:08:34.663021+0000","last_clean":"2024-04-16T21:08:34.663021+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:08:34.663021+0000","last_undegraded":"2024-04-16T21:08:34.663021+0000","last_fullsized":"2024-04-16T21:08:34.663021+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:37.701350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2369,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370711+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:09:19.370711+0000","last_peered":"2024-04-16T21:09:19.370711+0000","last_clean":"2024-04-16T21:09:19.370711+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:09:19.370711+0000","last_undegraded":"2024-04-16T21:09:19.370711+0000","last_fullsized":"2024-04-16T21:09:19.370711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:11:19.505797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370008+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:09:14.370008+0000","last_peered":"2024-04-16T21:09:14.370008+0000","last_clean":"2024-04-16T21:09:14.370008+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:09:14.370008+0000","last_undegraded":"2024-04-16T21:09:14.370008+0000","last_fullsized":"2024-04-16T21:09:14.370008+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'16","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369588+0000","last_change":"2024-04-16T19:25:15.484662+0000","last_active":"2024-04-16T21:09:09.369588+0000","last_peered":"2024-04-16T21:09:09.369588+0000","last_clean":"2024-04-16T21:09:09.369588+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:09:09.369588+0000","last_undegraded":"2024-04-16T21:09:09.369588+0000","last_fullsized":"2024-04-16T21:09:09.369588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'20","reported_seq":972,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509922+0000","last_change":"2024-04-16T19:25:13.474689+0000","last_active":"2024-04-16T21:09:14.509922+0000","last_peered":"2024-04-16T21:09:14.509922+0000","last_clean":"2024-04-16T21:09:14.509922+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:09:14.509922+0000","last_undegraded":"2024-04-16T21:09:14.509922+0000","last_fullsized":"2024-04-16T21:09:14.509922+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:36.914573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'249","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370263+0000","last_change":"2024-04-16T19:25:11.460951+0000","last_active":"2024-04-16T21:09:14.370263+0000","last_peered":"2024-04-16T21:09:14.370263+0000","last_clean":"2024-04-16T21:09:14.370263+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:09:14.370263+0000","last_undegraded":"2024-04-16T21:09:14.370263+0000","last_fullsized":"2024-04-16T21:09:14.370263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:42:59.386912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'7","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650231+0000","last_change":"2024-04-16T19:25:09.460368+0000","last_active":"2024-04-16T21:07:14.650231+0000","last_peered":"2024-04-16T21:07:14.650231+0000","last_clean":"2024-04-16T21:07:14.650231+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:07:14.650231+0000","last_undegraded":"2024-04-16T21:07:14.650231+0000","last_fullsized":"2024-04-16T21:07:14.650231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:19.738380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504660+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:08:39.504660+0000","last_peered":"2024-04-16T21:08:39.504660+0000","last_clean":"2024-04-16T21:08:39.504660+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:08:39.504660+0000","last_undegraded":"2024-04-16T21:08:39.504660+0000","last_fullsized":"2024-04-16T21:08:39.504660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.284891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497271+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:07:54.497271+0000","last_peered":"2024-04-16T21:07:54.497271+0000","last_clean":"2024-04-16T21:07:54.497271+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:07:54.497271+0000","last_undegraded":"2024-04-16T21:07:54.497271+0000","last_fullsized":"2024-04-16T21:07:54.497271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:27.416414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365627+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:08:54.365627+0000","last_peered":"2024-04-16T21:08:54.365627+0000","last_clean":"2024-04-16T21:08:54.365627+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:08:54.365627+0000","last_undegraded":"2024-04-16T21:08:54.365627+0000","last_fullsized":"2024-04-16T21:08:54.365627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:44.737149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506112+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:08:49.506112+0000","last_peered":"2024-04-16T21:08:49.506112+0000","last_clean":"2024-04-16T21:08:49.506112+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:08:49.506112+0000","last_undegraded":"2024-04-16T21:08:49.506112+0000","last_fullsized":"2024-04-16T21:08:49.506112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:32.935806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1513,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370734+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:09:19.370734+0000","last_peered":"2024-04-16T21:09:19.370734+0000","last_clean":"2024-04-16T21:09:19.370734+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:09:19.370734+0000","last_undegraded":"2024-04-16T21:09:19.370734+0000","last_fullsized":"2024-04-16T21:09:19.370734+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372374+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:08:59.372374+0000","last_peered":"2024-04-16T21:08:59.372374+0000","last_clean":"2024-04-16T21:08:59.372374+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:08:59.372374+0000","last_undegraded":"2024-04-16T21:08:59.372374+0000","last_fullsized":"2024-04-16T21:08:59.372374+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660288+0000","last_change":"2024-04-16T19:24:51.438114+0000","last_active":"2024-04-16T21:08:24.660288+0000","last_peered":"2024-04-16T21:08:24.660288+0000","last_clean":"2024-04-16T21:08:24.660288+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:08:24.660288+0000","last_undegraded":"2024-04-16T21:08:24.660288+0000","last_fullsized":"2024-04-16T21:08:24.660288+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:59.734805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366544+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:08:49.366544+0000","last_peered":"2024-04-16T21:08:49.366544+0000","last_clean":"2024-04-16T21:08:49.366544+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:08:49.366544+0000","last_undegraded":"2024-04-16T21:08:49.366544+0000","last_fullsized":"2024-04-16T21:08:49.366544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:25:31.551960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1766,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370669+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:09:19.370669+0000","last_peered":"2024-04-16T21:09:19.370669+0000","last_clean":"2024-04-16T21:09:19.370669+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:09:19.370669+0000","last_undegraded":"2024-04-16T21:09:19.370669+0000","last_fullsized":"2024-04-16T21:09:19.370669+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:10.736923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369787+0000","last_change":"2024-04-16T19:25:41.505274+0000","last_active":"2024-04-16T21:09:14.369787+0000","last_peered":"2024-04-16T21:09:14.369787+0000","last_clean":"2024-04-16T21:09:14.369787+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:09:14.369787+0000","last_undegraded":"2024-04-16T21:09:14.369787+0000","last_fullsized":"2024-04-16T21:09:14.369787+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:50:27.279669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'15","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370523+0000","last_change":"2024-04-16T19:25:15.487696+0000","last_active":"2024-04-16T21:08:49.370523+0000","last_peered":"2024-04-16T21:08:49.370523+0000","last_clean":"2024-04-16T21:08:49.370523+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:08:49.370523+0000","last_undegraded":"2024-04-16T21:08:49.370523+0000","last_fullsized":"2024-04-16T21:08:49.370523+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:34.149887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505292+0000","last_change":"2024-04-16T19:25:13.473305+0000","last_active":"2024-04-16T21:08:44.505292+0000","last_peered":"2024-04-16T21:08:44.505292+0000","last_clean":"2024-04-16T21:08:44.505292+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:08:44.505292+0000","last_undegraded":"2024-04-16T21:08:44.505292+0000","last_fullsized":"2024-04-16T21:08:44.505292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:38.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'690","reported_seq":1689,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:25:11.468818+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:21.906682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'16","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373858+0000","last_change":"2024-04-16T19:25:09.462529+0000","last_active":"2024-04-16T21:09:09.373858+0000","last_peered":"2024-04-16T21:09:09.373858+0000","last_clean":"2024-04-16T21:09:09.373858+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:09:09.373858+0000","last_undegraded":"2024-04-16T21:09:09.373858+0000","last_fullsized":"2024-04-16T21:09:09.373858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364926+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:08:49.364926+0000","last_peered":"2024-04-16T21:08:49.364926+0000","last_clean":"2024-04-16T21:08:49.364926+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:08:49.364926+0000","last_undegraded":"2024-04-16T21:08:49.364926+0000","last_fullsized":"2024-04-16T21:08:49.364926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:02.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657141+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:07:59.657141+0000","last_peered":"2024-04-16T21:07:59.657141+0000","last_clean":"2024-04-16T21:07:59.657141+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:07:59.657141+0000","last_undegraded":"2024-04-16T21:07:59.657141+0000","last_fullsized":"2024-04-16T21:07:59.657141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364646+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:08:44.364646+0000","last_peered":"2024-04-16T21:08:44.364646+0000","last_clean":"2024-04-16T21:08:44.364646+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:08:44.364646+0000","last_undegraded":"2024-04-16T21:08:44.364646+0000","last_fullsized":"2024-04-16T21:08:44.364646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:21.093051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375716+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:09:19.375716+0000","last_peered":"2024-04-16T21:09:19.375716+0000","last_clean":"2024-04-16T21:09:19.375716+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:09:19.375716+0000","last_undegraded":"2024-04-16T21:09:19.375716+0000","last_fullsized":"2024-04-16T21:09:19.375716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:21.104930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364984+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:08:49.364984+0000","last_peered":"2024-04-16T21:08:49.364984+0000","last_clean":"2024-04-16T21:08:49.364984+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:08:49.364984+0000","last_undegraded":"2024-04-16T21:08:49.364984+0000","last_fullsized":"2024-04-16T21:08:49.364984+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:58.758206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366555+0000","last_change":"2024-04-16T19:24:49.432013+0000","last_active":"2024-04-16T21:08:49.366555+0000","last_peered":"2024-04-16T21:08:49.366555+0000","last_clean":"2024-04-16T21:08:49.366555+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:08:49.366555+0000","last_undegraded":"2024-04-16T21:08:49.366555+0000","last_fullsized":"2024-04-16T21:08:49.366555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366349+0000","last_change":"2024-04-16T19:24:51.436955+0000","last_active":"2024-04-16T21:08:54.366349+0000","last_peered":"2024-04-16T21:08:54.366349+0000","last_clean":"2024-04-16T21:08:54.366349+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:08:54.366349+0000","last_undegraded":"2024-04-16T21:08:54.366349+0000","last_fullsized":"2024-04-16T21:08:54.366349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654394+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:07:44.654394+0000","last_peered":"2024-04-16T21:07:44.654394+0000","last_clean":"2024-04-16T21:07:44.654394+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:07:44.654394+0000","last_undegraded":"2024-04-16T21:07:44.654394+0000","last_fullsized":"2024-04-16T21:07:44.654394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:57:15.042193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652729+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:07:39.652729+0000","last_peered":"2024-04-16T21:07:39.652729+0000","last_clean":"2024-04-16T21:07:39.652729+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:07:39.652729+0000","last_undegraded":"2024-04-16T21:07:39.652729+0000","last_fullsized":"2024-04-16T21:07:39.652729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:02.023550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370324+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:09:14.370324+0000","last_peered":"2024-04-16T21:09:14.370324+0000","last_clean":"2024-04-16T21:09:14.370324+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:09:14.370324+0000","last_undegraded":"2024-04-16T21:09:14.370324+0000","last_fullsized":"2024-04-16T21:09:14.370324+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'3","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492346+0000","last_change":"2024-04-16T19:25:15.483924+0000","last_active":"2024-04-16T21:07:24.492346+0000","last_peered":"2024-04-16T21:07:24.492346+0000","last_clean":"2024-04-16T21:07:24.492346+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:07:24.492346+0000","last_undegraded":"2024-04-16T21:07:24.492346+0000","last_fullsized":"2024-04-16T21:07:24.492346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'705","reported_seq":2159,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505330+0000","last_change":"2024-04-16T19:25:13.475694+0000","last_active":"2024-04-16T21:08:44.505330+0000","last_peered":"2024-04-16T21:08:44.505330+0000","last_clean":"2024-04-16T21:08:44.505330+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:08:44.505330+0000","last_undegraded":"2024-04-16T21:08:44.505330+0000","last_fullsized":"2024-04-16T21:08:44.505330+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:56.172757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'247","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364687+0000","last_change":"2024-04-16T19:25:11.466002+0000","last_active":"2024-04-16T21:08:44.364687+0000","last_peered":"2024-04-16T21:08:44.364687+0000","last_clean":"2024-04-16T21:08:44.364687+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:08:44.364687+0000","last_undegraded":"2024-04-16T21:08:44.364687+0000","last_fullsized":"2024-04-16T21:08:44.364687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:38.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'27","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367274+0000","last_change":"2024-04-16T19:25:09.469480+0000","last_active":"2024-04-16T21:08:59.367274+0000","last_peered":"2024-04-16T21:08:59.367274+0000","last_clean":"2024-04-16T21:08:59.367274+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:08:59.367274+0000","last_undegraded":"2024-04-16T21:08:59.367274+0000","last_fullsized":"2024-04-16T21:08:59.367274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:00.355585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364874+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:08:49.364874+0000","last_peered":"2024-04-16T21:08:49.364874+0000","last_clean":"2024-04-16T21:08:49.364874+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:08:49.364874+0000","last_undegraded":"2024-04-16T21:08:49.364874+0000","last_fullsized":"2024-04-16T21:08:49.364874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:25.878014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368069+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:09:04.368069+0000","last_peered":"2024-04-16T21:09:04.368069+0000","last_clean":"2024-04-16T21:09:04.368069+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:09:04.368069+0000","last_undegraded":"2024-04-16T21:09:04.368069+0000","last_fullsized":"2024-04-16T21:09:04.368069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498140+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:07:59.498140+0000","last_peered":"2024-04-16T21:07:59.498140+0000","last_clean":"2024-04-16T21:07:59.498140+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:07:59.498140+0000","last_undegraded":"2024-04-16T21:07:59.498140+0000","last_fullsized":"2024-04-16T21:07:59.498140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372342+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:08:59.372342+0000","last_peered":"2024-04-16T21:08:59.372342+0000","last_clean":"2024-04-16T21:08:59.372342+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:08:59.372342+0000","last_undegraded":"2024-04-16T21:08:59.372342+0000","last_fullsized":"2024-04-16T21:08:59.372342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:38.135155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":349,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373804+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:09:09.373804+0000","last_peered":"2024-04-16T21:09:09.373804+0000","last_clean":"2024-04-16T21:09:09.373804+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:09:09.373804+0000","last_undegraded":"2024-04-16T21:09:09.373804+0000","last_fullsized":"2024-04-16T21:09:09.373804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367113+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:08:59.367113+0000","last_peered":"2024-04-16T21:08:59.367113+0000","last_clean":"2024-04-16T21:08:59.367113+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:08:59.367113+0000","last_undegraded":"2024-04-16T21:08:59.367113+0000","last_fullsized":"2024-04-16T21:08:59.367113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:50.295932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660391+0000","last_change":"2024-04-16T19:24:51.572391+0000","last_active":"2024-04-16T21:08:24.660391+0000","last_peered":"2024-04-16T21:08:24.660391+0000","last_clean":"2024-04-16T21:08:24.660391+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:08:24.660391+0000","last_undegraded":"2024-04-16T21:08:24.660391+0000","last_fullsized":"2024-04-16T21:08:24.660391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:04.095435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495516+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:07:44.495516+0000","last_peered":"2024-04-16T21:07:44.495516+0000","last_clean":"2024-04-16T21:07:44.495516+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:07:44.495516+0000","last_undegraded":"2024-04-16T21:07:44.495516+0000","last_fullsized":"2024-04-16T21:07:44.495516+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:34.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2002,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653496+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:07:39.653496+0000","last_peered":"2024-04-16T21:07:39.653496+0000","last_clean":"2024-04-16T21:07:39.653496+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:07:39.653496+0000","last_undegraded":"2024-04-16T21:07:39.653496+0000","last_fullsized":"2024-04-16T21:07:39.653496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:04:06.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496500+0000","last_change":"2024-04-16T19:25:41.510087+0000","last_active":"2024-04-16T21:07:49.496500+0000","last_peered":"2024-04-16T21:07:49.496500+0000","last_clean":"2024-04-16T21:07:49.496500+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:07:49.496500+0000","last_undegraded":"2024-04-16T21:07:49.496500+0000","last_fullsized":"2024-04-16T21:07:49.496500+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:45.367111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'23","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370491+0000","last_change":"2024-04-16T19:25:15.487252+0000","last_active":"2024-04-16T21:08:49.370491+0000","last_peered":"2024-04-16T21:08:49.370491+0000","last_clean":"2024-04-16T21:08:49.370491+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:08:49.370491+0000","last_undegraded":"2024-04-16T21:08:49.370491+0000","last_fullsized":"2024-04-16T21:08:49.370491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:14.101479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364063+0000","last_change":"2024-04-16T19:25:13.475862+0000","last_active":"2024-04-16T21:08:44.364063+0000","last_peered":"2024-04-16T21:08:44.364063+0000","last_clean":"2024-04-16T21:08:44.364063+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:08:44.364063+0000","last_undegraded":"2024-04-16T21:08:44.364063+0000","last_fullsized":"2024-04-16T21:08:44.364063+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:47.410023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'714","reported_seq":1574,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364715+0000","last_change":"2024-04-16T19:25:11.465490+0000","last_active":"2024-04-16T21:08:44.364715+0000","last_peered":"2024-04-16T21:08:44.364715+0000","last_clean":"2024-04-16T21:08:44.364715+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:08:44.364715+0000","last_undegraded":"2024-04-16T21:08:44.364715+0000","last_fullsized":"2024-04-16T21:08:44.364715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:03.668322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'6","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664395+0000","last_change":"2024-04-16T19:25:09.463391+0000","last_active":"2024-04-16T21:08:39.664395+0000","last_peered":"2024-04-16T21:08:39.664395+0000","last_clean":"2024-04-16T21:08:39.664395+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:08:39.664395+0000","last_undegraded":"2024-04-16T21:08:39.664395+0000","last_fullsized":"2024-04-16T21:08:39.664395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370504+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:08:49.370504+0000","last_peered":"2024-04-16T21:08:49.370504+0000","last_clean":"2024-04-16T21:08:49.370504+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:08:49.370504+0000","last_undegraded":"2024-04-16T21:08:49.370504+0000","last_fullsized":"2024-04-16T21:08:49.370504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364075+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:08:44.364075+0000","last_peered":"2024-04-16T21:08:44.364075+0000","last_clean":"2024-04-16T21:08:44.364075+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:08:44.364075+0000","last_undegraded":"2024-04-16T21:08:44.364075+0000","last_fullsized":"2024-04-16T21:08:44.364075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660502+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:08:14.660502+0000","last_peered":"2024-04-16T21:08:14.660502+0000","last_clean":"2024-04-16T21:08:14.660502+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:08:14.660502+0000","last_undegraded":"2024-04-16T21:08:14.660502+0000","last_fullsized":"2024-04-16T21:08:14.660502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:07.162476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367144+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:08:59.367144+0000","last_peered":"2024-04-16T21:08:59.367144+0000","last_clean":"2024-04-16T21:08:59.367144+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:08:59.367144+0000","last_undegraded":"2024-04-16T21:08:59.367144+0000","last_fullsized":"2024-04-16T21:08:59.367144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:27.238344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":346,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366566+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:08:59.366566+0000","last_peered":"2024-04-16T21:08:59.366566+0000","last_clean":"2024-04-16T21:08:59.366566+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:08:59.366566+0000","last_undegraded":"2024-04-16T21:08:59.366566+0000","last_fullsized":"2024-04-16T21:08:59.366566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:36.905847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1317,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368814+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:09:09.368814+0000","last_peered":"2024-04-16T21:09:09.368814+0000","last_clean":"2024-04-16T21:09:09.368814+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:09:09.368814+0000","last_undegraded":"2024-04-16T21:09:09.368814+0000","last_fullsized":"2024-04-16T21:09:09.368814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:34.836984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662820+0000","last_change":"2024-04-16T19:24:51.572638+0000","last_active":"2024-04-16T21:08:39.662820+0000","last_peered":"2024-04-16T21:08:39.662820+0000","last_clean":"2024-04-16T21:08:39.662820+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:08:39.662820+0000","last_undegraded":"2024-04-16T21:08:39.662820+0000","last_fullsized":"2024-04-16T21:08:39.662820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:13.926597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503462+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:08:34.503462+0000","last_peered":"2024-04-16T21:08:34.503462+0000","last_clean":"2024-04-16T21:08:34.503462+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:08:34.503462+0000","last_undegraded":"2024-04-16T21:08:34.503462+0000","last_fullsized":"2024-04-16T21:08:34.503462+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:20.743689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1526,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656812+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:07:54.656812+0000","last_peered":"2024-04-16T21:07:54.656812+0000","last_clean":"2024-04-16T21:07:54.656812+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:07:54.656812+0000","last_undegraded":"2024-04-16T21:07:54.656812+0000","last_fullsized":"2024-04-16T21:07:54.656812+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:25.577697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652730+0000","last_change":"2024-04-16T19:25:41.498506+0000","last_active":"2024-04-16T21:07:34.652730+0000","last_peered":"2024-04-16T21:07:34.652730+0000","last_clean":"2024-04-16T21:07:34.652730+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:07:34.652730+0000","last_undegraded":"2024-04-16T21:07:34.652730+0000","last_fullsized":"2024-04-16T21:07:34.652730+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:14.970337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'2","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510638+0000","last_change":"2024-04-16T19:25:15.483677+0000","last_active":"2024-04-16T21:09:19.510638+0000","last_peered":"2024-04-16T21:09:19.510638+0000","last_clean":"2024-04-16T21:09:19.510638+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:09:19.510638+0000","last_undegraded":"2024-04-16T21:09:19.510638+0000","last_fullsized":"2024-04-16T21:09:19.510638+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368793+0000","last_change":"2024-04-16T19:25:13.473234+0000","last_active":"2024-04-16T21:09:04.368793+0000","last_peered":"2024-04-16T21:09:04.368793+0000","last_clean":"2024-04-16T21:09:04.368793+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:09:04.368793+0000","last_undegraded":"2024-04-16T21:09:04.368793+0000","last_fullsized":"2024-04-16T21:09:04.368793+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:10.112989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'267","reported_seq":524,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364765+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:08:44.364765+0000","last_peered":"2024-04-16T21:08:44.364765+0000","last_clean":"2024-04-16T21:08:44.364765+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:08:44.364765+0000","last_undegraded":"2024-04-16T21:08:44.364765+0000","last_fullsized":"2024-04-16T21:08:44.364765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:51.610204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'14","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664322+0000","last_change":"2024-04-16T19:25:09.461719+0000","last_active":"2024-04-16T21:08:39.664322+0000","last_peered":"2024-04-16T21:08:39.664322+0000","last_clean":"2024-04-16T21:08:39.664322+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:08:39.664322+0000","last_undegraded":"2024-04-16T21:08:39.664322+0000","last_fullsized":"2024-04-16T21:08:39.664322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:24.102149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372250+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:08:59.372250+0000","last_peered":"2024-04-16T21:08:59.372250+0000","last_clean":"2024-04-16T21:08:59.372250+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:08:59.372250+0000","last_undegraded":"2024-04-16T21:08:59.372250+0000","last_fullsized":"2024-04-16T21:08:59.372250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503593+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:08:34.503593+0000","last_peered":"2024-04-16T21:08:34.503593+0000","last_clean":"2024-04-16T21:08:34.503593+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:08:34.503593+0000","last_undegraded":"2024-04-16T21:08:34.503593+0000","last_fullsized":"2024-04-16T21:08:34.503593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495661+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:07:44.495661+0000","last_peered":"2024-04-16T21:07:44.495661+0000","last_clean":"2024-04-16T21:07:44.495661+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:07:44.495661+0000","last_undegraded":"2024-04-16T21:07:44.495661+0000","last_fullsized":"2024-04-16T21:07:44.495661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367951+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:08:59.367951+0000","last_peered":"2024-04-16T21:08:59.367951+0000","last_clean":"2024-04-16T21:08:59.367951+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:08:59.367951+0000","last_undegraded":"2024-04-16T21:08:59.367951+0000","last_fullsized":"2024-04-16T21:08:59.367951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:48.749243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367980+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:08:59.367980+0000","last_peered":"2024-04-16T21:08:59.367980+0000","last_clean":"2024-04-16T21:08:59.367980+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:08:59.367980+0000","last_undegraded":"2024-04-16T21:08:59.367980+0000","last_fullsized":"2024-04-16T21:08:59.367980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372321+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:08:59.372321+0000","last_peered":"2024-04-16T21:08:59.372321+0000","last_clean":"2024-04-16T21:08:59.372321+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:08:59.372321+0000","last_undegraded":"2024-04-16T21:08:59.372321+0000","last_fullsized":"2024-04-16T21:08:59.372321+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660333+0000","last_change":"2024-04-16T19:24:51.573460+0000","last_active":"2024-04-16T21:08:24.660333+0000","last_peered":"2024-04-16T21:08:24.660333+0000","last_clean":"2024-04-16T21:08:24.660333+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:08:24.660333+0000","last_undegraded":"2024-04-16T21:08:24.660333+0000","last_fullsized":"2024-04-16T21:08:24.660333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:31.585745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498095+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:07:59.498095+0000","last_peered":"2024-04-16T21:07:59.498095+0000","last_clean":"2024-04-16T21:07:59.498095+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:07:59.498095+0000","last_undegraded":"2024-04-16T21:07:59.498095+0000","last_fullsized":"2024-04-16T21:07:59.498095+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:33:27.173018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1521,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657201+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:07:59.657201+0000","last_peered":"2024-04-16T21:07:59.657201+0000","last_clean":"2024-04-16T21:07:59.657201+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:07:59.657201+0000","last_undegraded":"2024-04-16T21:07:59.657201+0000","last_fullsized":"2024-04-16T21:07:59.657201+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:16.733301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.493948+0000","last_change":"2024-04-16T19:25:41.502910+0000","last_active":"2024-04-16T21:07:34.493948+0000","last_peered":"2024-04-16T21:07:34.493948+0000","last_clean":"2024-04-16T21:07:34.493948+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:07:34.493948+0000","last_undegraded":"2024-04-16T21:07:34.493948+0000","last_fullsized":"2024-04-16T21:07:34.493948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:25.621893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'12","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510709+0000","last_change":"2024-04-16T19:25:15.485965+0000","last_active":"2024-04-16T21:09:19.510709+0000","last_peered":"2024-04-16T21:09:19.510709+0000","last_clean":"2024-04-16T21:09:19.510709+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:09:19.510709+0000","last_undegraded":"2024-04-16T21:09:19.510709+0000","last_fullsized":"2024-04-16T21:09:19.510709+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'705","reported_seq":2080,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364751+0000","last_change":"2024-04-16T19:25:13.471357+0000","last_active":"2024-04-16T21:08:44.364751+0000","last_peered":"2024-04-16T21:08:44.364751+0000","last_clean":"2024-04-16T21:08:44.364751+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:08:44.364751+0000","last_undegraded":"2024-04-16T21:08:44.364751+0000","last_fullsized":"2024-04-16T21:08:44.364751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'318","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368826+0000","last_change":"2024-04-16T19:25:11.461881+0000","last_active":"2024-04-16T21:09:04.368826+0000","last_peered":"2024-04-16T21:09:04.368826+0000","last_clean":"2024-04-16T21:09:04.368826+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:09:04.368826+0000","last_undegraded":"2024-04-16T21:09:04.368826+0000","last_fullsized":"2024-04-16T21:09:04.368826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:13.030219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'6","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504516+0000","last_change":"2024-04-16T19:25:09.461256+0000","last_active":"2024-04-16T21:08:39.504516+0000","last_peered":"2024-04-16T21:08:39.504516+0000","last_clean":"2024-04-16T21:08:39.504516+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:08:39.504516+0000","last_undegraded":"2024-04-16T21:08:39.504516+0000","last_fullsized":"2024-04-16T21:08:39.504516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:56.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365397+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:08:49.365397+0000","last_peered":"2024-04-16T21:08:49.365397+0000","last_clean":"2024-04-16T21:08:49.365397+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:08:49.365397+0000","last_undegraded":"2024-04-16T21:08:49.365397+0000","last_fullsized":"2024-04-16T21:08:49.365397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:21.322076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662064+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:08:34.662064+0000","last_peered":"2024-04-16T21:08:34.662064+0000","last_clean":"2024-04-16T21:08:34.662064+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:08:34.662064+0000","last_undegraded":"2024-04-16T21:08:34.662064+0000","last_fullsized":"2024-04-16T21:08:34.662064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.656111+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:07:49.656111+0000","last_peered":"2024-04-16T21:07:49.656111+0000","last_clean":"2024-04-16T21:07:49.656111+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:07:49.656111+0000","last_undegraded":"2024-04-16T21:07:49.656111+0000","last_fullsized":"2024-04-16T21:07:49.656111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:42.139799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508363+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:09:04.508363+0000","last_peered":"2024-04-16T21:09:04.508363+0000","last_clean":"2024-04-16T21:09:04.508363+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:09:04.508363+0000","last_undegraded":"2024-04-16T21:09:04.508363+0000","last_fullsized":"2024-04-16T21:09:04.508363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:28:03.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":331,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367176+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:08:59.367176+0000","last_peered":"2024-04-16T21:08:59.367176+0000","last_clean":"2024-04-16T21:08:59.367176+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:08:59.367176+0000","last_undegraded":"2024-04-16T21:08:59.367176+0000","last_fullsized":"2024-04-16T21:08:59.367176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:23.723705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507550+0000","last_change":"2024-04-16T19:24:49.435265+0000","last_active":"2024-04-16T21:08:59.507550+0000","last_peered":"2024-04-16T21:08:59.507550+0000","last_clean":"2024-04-16T21:08:59.507550+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:08:59.507550+0000","last_undegraded":"2024-04-16T21:08:59.507550+0000","last_fullsized":"2024-04-16T21:08:59.507550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:08.160472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661329+0000","last_change":"2024-04-16T19:24:51.438071+0000","last_active":"2024-04-16T21:08:24.661329+0000","last_peered":"2024-04-16T21:08:24.661329+0000","last_clean":"2024-04-16T21:08:24.661329+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:08:24.661329+0000","last_undegraded":"2024-04-16T21:08:24.661329+0000","last_fullsized":"2024-04-16T21:08:24.661329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:42.181631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649582+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:07:19.649582+0000","last_peered":"2024-04-16T21:07:19.649582+0000","last_clean":"2024-04-16T21:07:19.649582+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:07:19.649582+0000","last_undegraded":"2024-04-16T21:07:19.649582+0000","last_fullsized":"2024-04-16T21:07:19.649582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:34:01.244792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653404+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:07:34.653404+0000","last_peered":"2024-04-16T21:07:34.653404+0000","last_clean":"2024-04-16T21:07:34.653404+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:07:34.653404+0000","last_undegraded":"2024-04-16T21:07:34.653404+0000","last_fullsized":"2024-04-16T21:07:34.653404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:03.502015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.500263+0000","last_change":"2024-04-16T19:25:41.506596+0000","last_active":"2024-04-16T21:08:14.500263+0000","last_peered":"2024-04-16T21:08:14.500263+0000","last_clean":"2024-04-16T21:08:14.500263+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:08:14.500263+0000","last_undegraded":"2024-04-16T21:08:14.500263+0000","last_fullsized":"2024-04-16T21:08:14.500263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:01.639245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'4","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655215+0000","last_change":"2024-04-16T19:25:15.480477+0000","last_active":"2024-04-16T21:07:49.655215+0000","last_peered":"2024-04-16T21:07:49.655215+0000","last_clean":"2024-04-16T21:07:49.655215+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:07:49.655215+0000","last_undegraded":"2024-04-16T21:07:49.655215+0000","last_fullsized":"2024-04-16T21:07:49.655215+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654540+0000","last_change":"2024-04-16T19:25:13.470985+0000","last_active":"2024-04-16T21:07:44.654540+0000","last_peered":"2024-04-16T21:07:44.654540+0000","last_clean":"2024-04-16T21:07:44.654540+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:07:44.654540+0000","last_undegraded":"2024-04-16T21:07:44.654540+0000","last_fullsized":"2024-04-16T21:07:44.654540+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:54.589907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'292","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654437+0000","last_change":"2024-04-16T19:25:11.464321+0000","last_active":"2024-04-16T21:07:44.654437+0000","last_peered":"2024-04-16T21:07:44.654437+0000","last_clean":"2024-04-16T21:07:44.654437+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:07:44.654437+0000","last_undegraded":"2024-04-16T21:07:44.654437+0000","last_fullsized":"2024-04-16T21:07:44.654437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:17.793032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'13","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.377356+0000","last_change":"2024-04-16T19:25:09.460391+0000","last_active":"2024-04-16T21:09:29.377356+0000","last_peered":"2024-04-16T21:09:29.377356+0000","last_clean":"2024-04-16T21:09:29.377356+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:09:29.377356+0000","last_undegraded":"2024-04-16T21:09:29.377356+0000","last_fullsized":"2024-04-16T21:09:29.377356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373217+0000","last_change":"2024-04-16T19:24:51.434685+0000","last_active":"2024-04-16T21:09:04.373217+0000","last_peered":"2024-04-16T21:09:04.373217+0000","last_clean":"2024-04-16T21:09:04.373217+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:09:04.373217+0000","last_undegraded":"2024-04-16T21:09:04.373217+0000","last_fullsized":"2024-04-16T21:09:04.373217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:57:30.733796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650563+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:07:19.650563+0000","last_peered":"2024-04-16T21:07:19.650563+0000","last_clean":"2024-04-16T21:07:19.650563+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:07:19.650563+0000","last_undegraded":"2024-04-16T21:07:19.650563+0000","last_fullsized":"2024-04-16T21:07:19.650563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:28.749905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504494+0000","last_change":"2024-04-16T19:25:41.504766+0000","last_active":"2024-04-16T21:08:39.504494+0000","last_peered":"2024-04-16T21:08:39.504494+0000","last_clean":"2024-04-16T21:08:39.504494+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:08:39.504494+0000","last_undegraded":"2024-04-16T21:08:39.504494+0000","last_fullsized":"2024-04-16T21:08:39.504494+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:37.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":743,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493189+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:07:29.493189+0000","last_peered":"2024-04-16T21:07:29.493189+0000","last_clean":"2024-04-16T21:07:29.493189+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:07:29.493189+0000","last_undegraded":"2024-04-16T21:07:29.493189+0000","last_fullsized":"2024-04-16T21:07:29.493189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:30.438117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493117+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:07:29.493117+0000","last_peered":"2024-04-16T21:07:29.493117+0000","last_clean":"2024-04-16T21:07:29.493117+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:07:29.493117+0000","last_undegraded":"2024-04-16T21:07:29.493117+0000","last_fullsized":"2024-04-16T21:07:29.493117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:51.673404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493130+0000","last_change":"2024-04-16T19:24:49.434342+0000","last_active":"2024-04-16T21:07:29.493130+0000","last_peered":"2024-04-16T21:07:29.493130+0000","last_clean":"2024-04-16T21:07:29.493130+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:07:29.493130+0000","last_undegraded":"2024-04-16T21:07:29.493130+0000","last_fullsized":"2024-04-16T21:07:29.493130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:07.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651977+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:07:29.651977+0000","last_peered":"2024-04-16T21:07:29.651977+0000","last_clean":"2024-04-16T21:07:29.651977+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:07:29.651977+0000","last_undegraded":"2024-04-16T21:07:29.651977+0000","last_fullsized":"2024-04-16T21:07:29.651977+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:21.199802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'118","reported_seq":1711,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491510+0000","last_change":"2024-04-16T19:25:49.520399+0000","last_active":"2024-04-16T21:07:19.491510+0000","last_peered":"2024-04-16T21:07:19.491510+0000","last_clean":"2024-04-16T21:07:19.491510+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:07:19.491510+0000","last_undegraded":"2024-04-16T21:07:19.491510+0000","last_fullsized":"2024-04-16T21:07:19.491510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:16:26.165977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659512+0000","last_change":"2024-04-16T19:25:41.505754+0000","last_active":"2024-04-16T21:08:14.659512+0000","last_peered":"2024-04-16T21:08:14.659512+0000","last_clean":"2024-04-16T21:08:14.659512+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:08:14.659512+0000","last_undegraded":"2024-04-16T21:08:14.659512+0000","last_fullsized":"2024-04-16T21:08:14.659512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:10.716421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655168+0000","last_change":"2024-04-16T19:25:15.482484+0000","last_active":"2024-04-16T21:07:49.655168+0000","last_peered":"2024-04-16T21:07:49.655168+0000","last_clean":"2024-04-16T21:07:49.655168+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:07:49.655168+0000","last_undegraded":"2024-04-16T21:07:49.655168+0000","last_fullsized":"2024-04-16T21:07:49.655168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:25.761792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655229+0000","last_change":"2024-04-16T19:25:13.472620+0000","last_active":"2024-04-16T21:07:44.655229+0000","last_peered":"2024-04-16T21:07:44.655229+0000","last_clean":"2024-04-16T21:07:44.655229+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:07:44.655229+0000","last_undegraded":"2024-04-16T21:07:44.655229+0000","last_fullsized":"2024-04-16T21:07:44.655229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:28.935744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1140","reported_seq":2898,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654522+0000","last_change":"2024-04-16T19:25:11.462713+0000","last_active":"2024-04-16T21:07:44.654522+0000","last_peered":"2024-04-16T21:07:44.654522+0000","last_clean":"2024-04-16T21:07:44.654522+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:07:44.654522+0000","last_undegraded":"2024-04-16T21:07:44.654522+0000","last_fullsized":"2024-04-16T21:07:44.654522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:45.401459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'9","reported_seq":192,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369883+0000","last_change":"2024-04-16T19:25:09.466079+0000","last_active":"2024-04-16T21:09:19.369883+0000","last_peered":"2024-04-16T21:09:19.369883+0000","last_clean":"2024-04-16T21:09:19.369883+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:09:19.369883+0000","last_undegraded":"2024-04-16T21:09:19.369883+0000","last_fullsized":"2024-04-16T21:09:19.369883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369024+0000","last_change":"2024-04-16T19:24:51.442540+0000","last_active":"2024-04-16T21:09:14.369024+0000","last_peered":"2024-04-16T21:09:14.369024+0000","last_clean":"2024-04-16T21:09:14.369024+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:09:14.369024+0000","last_undegraded":"2024-04-16T21:09:14.369024+0000","last_fullsized":"2024-04-16T21:09:14.369024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491649+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:07:19.491649+0000","last_peered":"2024-04-16T21:07:19.491649+0000","last_clean":"2024-04-16T21:07:19.491649+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:07:19.491649+0000","last_undegraded":"2024-04-16T21:07:19.491649+0000","last_fullsized":"2024-04-16T21:07:19.491649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662604+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:08:29.662604+0000","last_peered":"2024-04-16T21:08:29.662604+0000","last_clean":"2024-04-16T21:08:29.662604+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:08:29.662604+0000","last_undegraded":"2024-04-16T21:08:29.662604+0000","last_fullsized":"2024-04-16T21:08:29.662604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493228+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:07:29.493228+0000","last_peered":"2024-04-16T21:07:29.493228+0000","last_clean":"2024-04-16T21:07:29.493228+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:07:29.493228+0000","last_undegraded":"2024-04-16T21:07:29.493228+0000","last_fullsized":"2024-04-16T21:07:29.493228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:10.368115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651116+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:07:29.651116+0000","last_peered":"2024-04-16T21:07:29.651116+0000","last_clean":"2024-04-16T21:07:29.651116+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:07:29.651116+0000","last_undegraded":"2024-04-16T21:07:29.651116+0000","last_fullsized":"2024-04-16T21:07:29.651116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:05.198319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652671+0000","last_change":"2024-04-16T19:24:49.431431+0000","last_active":"2024-04-16T21:07:29.652671+0000","last_peered":"2024-04-16T21:07:29.652671+0000","last_clean":"2024-04-16T21:07:29.652671+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:07:29.652671+0000","last_undegraded":"2024-04-16T21:07:29.652671+0000","last_fullsized":"2024-04-16T21:07:29.652671+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652772+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:07:34.652772+0000","last_peered":"2024-04-16T21:07:34.652772+0000","last_clean":"2024-04-16T21:07:34.652772+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:07:34.652772+0000","last_undegraded":"2024-04-16T21:07:34.652772+0000","last_fullsized":"2024-04-16T21:07:34.652772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:31:39.104319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1703,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501136+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:08:19.501136+0000","last_peered":"2024-04-16T21:08:19.501136+0000","last_clean":"2024-04-16T21:08:19.501136+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:08:19.501136+0000","last_undegraded":"2024-04-16T21:08:19.501136+0000","last_fullsized":"2024-04-16T21:08:19.501136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:15.107893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649480+0000","last_change":"2024-04-16T19:25:41.499274+0000","last_active":"2024-04-16T21:07:14.649480+0000","last_peered":"2024-04-16T21:07:14.649480+0000","last_clean":"2024-04-16T21:07:14.649480+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:07:14.649480+0000","last_undegraded":"2024-04-16T21:07:14.649480+0000","last_fullsized":"2024-04-16T21:07:14.649480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:22:55.194832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'20","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647308+0000","last_change":"2024-04-16T19:25:15.485235+0000","last_active":"2024-04-16T21:07:04.647308+0000","last_peered":"2024-04-16T21:07:04.647308+0000","last_clean":"2024-04-16T21:07:04.647308+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:07:04.647308+0000","last_undegraded":"2024-04-16T21:07:04.647308+0000","last_fullsized":"2024-04-16T21:07:04.647308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.373278+0000","last_change":"2024-04-16T19:25:13.476041+0000","last_active":"2024-04-16T21:09:34.373278+0000","last_peered":"2024-04-16T21:09:34.373278+0000","last_clean":"2024-04-16T21:09:34.373278+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:09:34.373278+0000","last_undegraded":"2024-04-16T21:09:34.373278+0000","last_fullsized":"2024-04-16T21:09:34.373278+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:17.872253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'274","reported_seq":578,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371939+0000","last_change":"2024-04-16T19:25:11.462598+0000","last_active":"2024-04-16T21:09:24.371939+0000","last_peered":"2024-04-16T21:09:24.371939+0000","last_clean":"2024-04-16T21:09:24.371939+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:09:24.371939+0000","last_undegraded":"2024-04-16T21:09:24.371939+0000","last_fullsized":"2024-04-16T21:09:24.371939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:55.581849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'8","reported_seq":245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652902+0000","last_change":"2024-04-16T19:25:09.461313+0000","last_active":"2024-04-16T21:07:39.652902+0000","last_peered":"2024-04-16T21:07:39.652902+0000","last_clean":"2024-04-16T21:07:39.652902+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:07:39.652902+0000","last_undegraded":"2024-04-16T21:07:39.652902+0000","last_fullsized":"2024-04-16T21:07:39.652902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650513+0000","last_change":"2024-04-16T19:24:51.435662+0000","last_active":"2024-04-16T21:07:24.650513+0000","last_peered":"2024-04-16T21:07:24.650513+0000","last_clean":"2024-04-16T21:07:24.650513+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:07:24.650513+0000","last_undegraded":"2024-04-16T21:07:24.650513+0000","last_fullsized":"2024-04-16T21:07:24.650513+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:57.768573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491559+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:07:19.491559+0000","last_peered":"2024-04-16T21:07:19.491559+0000","last_clean":"2024-04-16T21:07:19.491559+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:07:19.491559+0000","last_undegraded":"2024-04-16T21:07:19.491559+0000","last_fullsized":"2024-04-16T21:07:19.491559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:46:59.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662208+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:08:29.662208+0000","last_peered":"2024-04-16T21:08:29.662208+0000","last_clean":"2024-04-16T21:08:29.662208+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:08:29.662208+0000","last_undegraded":"2024-04-16T21:08:29.662208+0000","last_fullsized":"2024-04-16T21:08:29.662208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":611,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651931+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:07:29.651931+0000","last_peered":"2024-04-16T21:07:29.651931+0000","last_clean":"2024-04-16T21:07:29.651931+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:07:29.651931+0000","last_undegraded":"2024-04-16T21:07:29.651931+0000","last_fullsized":"2024-04-16T21:07:29.651931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:01.385906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493180+0000","last_change":"2024-04-16T19:24:47.424747+0000","last_active":"2024-04-16T21:07:29.493180+0000","last_peered":"2024-04-16T21:07:29.493180+0000","last_clean":"2024-04-16T21:07:29.493180+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:07:29.493180+0000","last_undegraded":"2024-04-16T21:07:29.493180+0000","last_fullsized":"2024-04-16T21:07:29.493180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:48.064472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652026+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:07:29.652026+0000","last_peered":"2024-04-16T21:07:29.652026+0000","last_clean":"2024-04-16T21:07:29.652026+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:07:29.652026+0000","last_undegraded":"2024-04-16T21:07:29.652026+0000","last_fullsized":"2024-04-16T21:07:29.652026+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:21.377903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653531+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:07:34.653531+0000","last_peered":"2024-04-16T21:07:34.653531+0000","last_clean":"2024-04-16T21:07:34.653531+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:07:34.653531+0000","last_undegraded":"2024-04-16T21:07:34.653531+0000","last_fullsized":"2024-04-16T21:07:34.653531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659404+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:08:19.659404+0000","last_peered":"2024-04-16T21:08:19.659404+0000","last_clean":"2024-04-16T21:08:19.659404+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:08:19.659404+0000","last_undegraded":"2024-04-16T21:08:19.659404+0000","last_fullsized":"2024-04-16T21:08:19.659404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493079+0000","last_change":"2024-04-16T19:25:41.508866+0000","last_active":"2024-04-16T21:07:29.493079+0000","last_peered":"2024-04-16T21:07:29.493079+0000","last_clean":"2024-04-16T21:07:29.493079+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:07:29.493079+0000","last_undegraded":"2024-04-16T21:07:29.493079+0000","last_fullsized":"2024-04-16T21:07:29.493079+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:29.426956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'22","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.486695+0000","last_change":"2024-04-16T19:25:15.480626+0000","last_active":"2024-04-16T21:06:49.486695+0000","last_peered":"2024-04-16T21:06:49.486695+0000","last_clean":"2024-04-16T21:06:49.486695+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:06:49.486695+0000","last_undegraded":"2024-04-16T21:06:49.486695+0000","last_fullsized":"2024-04-16T21:06:49.486695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:39.898263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'705","reported_seq":2141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511688+0000","last_change":"2024-04-16T19:25:13.473351+0000","last_active":"2024-04-16T21:09:24.511688+0000","last_peered":"2024-04-16T21:09:24.511688+0000","last_clean":"2024-04-16T21:09:24.511688+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:09:24.511688+0000","last_undegraded":"2024-04-16T21:09:24.511688+0000","last_fullsized":"2024-04-16T21:09:24.511688+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:04.370192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1017","reported_seq":1517,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.378182+0000","last_change":"2024-04-16T19:25:11.466163+0000","last_active":"2024-04-16T21:09:34.378182+0000","last_peered":"2024-04-16T21:09:34.378182+0000","last_clean":"2024-04-16T21:09:34.378182+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:09:34.378182+0000","last_undegraded":"2024-04-16T21:09:34.378182+0000","last_fullsized":"2024-04-16T21:09:34.378182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:40.676377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'14","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652919+0000","last_change":"2024-04-16T19:25:09.467909+0000","last_active":"2024-04-16T21:07:39.652919+0000","last_peered":"2024-04-16T21:07:39.652919+0000","last_clean":"2024-04-16T21:07:39.652919+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:07:39.652919+0000","last_undegraded":"2024-04-16T21:07:39.652919+0000","last_fullsized":"2024-04-16T21:07:39.652919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:05.530059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651388+0000","last_change":"2024-04-16T19:24:51.438772+0000","last_active":"2024-04-16T21:07:24.651388+0000","last_peered":"2024-04-16T21:07:24.651388+0000","last_clean":"2024-04-16T21:07:24.651388+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:07:24.651388+0000","last_undegraded":"2024-04-16T21:07:24.651388+0000","last_fullsized":"2024-04-16T21:07:24.651388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491555+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:07:19.491555+0000","last_peered":"2024-04-16T21:07:19.491555+0000","last_clean":"2024-04-16T21:07:19.491555+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:07:19.491555+0000","last_undegraded":"2024-04-16T21:07:19.491555+0000","last_fullsized":"2024-04-16T21:07:19.491555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661253+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:08:29.661253+0000","last_peered":"2024-04-16T21:08:29.661253+0000","last_clean":"2024-04-16T21:08:29.661253+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:08:29.661253+0000","last_undegraded":"2024-04-16T21:08:29.661253+0000","last_fullsized":"2024-04-16T21:08:29.661253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652693+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:07:29.652693+0000","last_peered":"2024-04-16T21:07:29.652693+0000","last_clean":"2024-04-16T21:07:29.652693+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:07:29.652693+0000","last_undegraded":"2024-04-16T21:07:29.652693+0000","last_fullsized":"2024-04-16T21:07:29.652693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652054+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:07:29.652054+0000","last_peered":"2024-04-16T21:07:29.652054+0000","last_clean":"2024-04-16T21:07:29.652054+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:07:29.652054+0000","last_undegraded":"2024-04-16T21:07:29.652054+0000","last_fullsized":"2024-04-16T21:07:29.652054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:52.547347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651195+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:07:29.651195+0000","last_peered":"2024-04-16T21:07:29.651195+0000","last_clean":"2024-04-16T21:07:29.651195+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:07:29.651195+0000","last_undegraded":"2024-04-16T21:07:29.651195+0000","last_fullsized":"2024-04-16T21:07:29.651195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648537+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:07:04.648537+0000","last_peered":"2024-04-16T21:07:04.648537+0000","last_clean":"2024-04-16T21:07:04.648537+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:07:04.648537+0000","last_undegraded":"2024-04-16T21:07:04.648537+0000","last_fullsized":"2024-04-16T21:07:04.648537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:37.665943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2028,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651065+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:07:19.651065+0000","last_peered":"2024-04-16T21:07:19.651065+0000","last_clean":"2024-04-16T21:07:19.651065+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:07:19.651065+0000","last_undegraded":"2024-04-16T21:07:19.651065+0000","last_fullsized":"2024-04-16T21:07:19.651065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:48.277899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648877+0000","last_change":"2024-04-16T19:25:41.507140+0000","last_active":"2024-04-16T21:07:14.648877+0000","last_peered":"2024-04-16T21:07:14.648877+0000","last_clean":"2024-04-16T21:07:14.648877+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:07:14.648877+0000","last_undegraded":"2024-04-16T21:07:14.648877+0000","last_fullsized":"2024-04-16T21:07:14.648877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:14:47.746067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'5","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645995+0000","last_change":"2024-04-16T19:25:15.481140+0000","last_active":"2024-04-16T21:06:49.645995+0000","last_peered":"2024-04-16T21:06:49.645995+0000","last_clean":"2024-04-16T21:06:49.645995+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:06:49.645995+0000","last_undegraded":"2024-04-16T21:06:49.645995+0000","last_fullsized":"2024-04-16T21:06:49.645995+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:22.980049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655276+0000","last_change":"2024-04-16T19:25:13.476134+0000","last_active":"2024-04-16T21:07:54.655276+0000","last_peered":"2024-04-16T21:07:54.655276+0000","last_clean":"2024-04-16T21:07:54.655276+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:07:54.655276+0000","last_undegraded":"2024-04-16T21:07:54.655276+0000","last_fullsized":"2024-04-16T21:07:54.655276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'305","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656185+0000","last_change":"2024-04-16T19:25:11.465819+0000","last_active":"2024-04-16T21:07:54.656185+0000","last_peered":"2024-04-16T21:07:54.656185+0000","last_clean":"2024-04-16T21:07:54.656185+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:07:54.656185+0000","last_undegraded":"2024-04-16T21:07:54.656185+0000","last_fullsized":"2024-04-16T21:07:54.656185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:38.882914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'6","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655341+0000","last_change":"2024-04-16T19:25:09.469614+0000","last_active":"2024-04-16T21:07:49.655341+0000","last_peered":"2024-04-16T21:07:49.655341+0000","last_clean":"2024-04-16T21:07:49.655341+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:07:49.655341+0000","last_undegraded":"2024-04-16T21:07:49.655341+0000","last_fullsized":"2024-04-16T21:07:49.655341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366575+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:08:49.366575+0000","last_peered":"2024-04-16T21:08:49.366575+0000","last_clean":"2024-04-16T21:08:49.366575+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:08:49.366575+0000","last_undegraded":"2024-04-16T21:08:49.366575+0000","last_fullsized":"2024-04-16T21:08:49.366575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:54.123433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502246+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:08:29.502246+0000","last_peered":"2024-04-16T21:08:29.502246+0000","last_clean":"2024-04-16T21:08:29.502246+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:08:29.502246+0000","last_undegraded":"2024-04-16T21:08:29.502246+0000","last_fullsized":"2024-04-16T21:08:29.502246+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:19:16.213346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.374003+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:09:09.374003+0000","last_peered":"2024-04-16T21:09:09.374003+0000","last_clean":"2024-04-16T21:09:09.374003+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:09:09.374003+0000","last_undegraded":"2024-04-16T21:09:09.374003+0000","last_fullsized":"2024-04-16T21:09:09.374003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491759+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:07:19.491759+0000","last_peered":"2024-04-16T21:07:19.491759+0000","last_clean":"2024-04-16T21:07:19.491759+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:07:19.491759+0000","last_undegraded":"2024-04-16T21:07:19.491759+0000","last_fullsized":"2024-04-16T21:07:19.491759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:50.759439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651037+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:07:19.651037+0000","last_peered":"2024-04-16T21:07:19.651037+0000","last_clean":"2024-04-16T21:07:19.651037+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:07:19.651037+0000","last_undegraded":"2024-04-16T21:07:19.651037+0000","last_fullsized":"2024-04-16T21:07:19.651037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494045+0000","last_change":"2024-04-16T19:24:51.574221+0000","last_active":"2024-04-16T21:07:34.494045+0000","last_peered":"2024-04-16T21:07:34.494045+0000","last_clean":"2024-04-16T21:07:34.494045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:07:34.494045+0000","last_undegraded":"2024-04-16T21:07:34.494045+0000","last_fullsized":"2024-04-16T21:07:34.494045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:09.678742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650574+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:07:19.650574+0000","last_peered":"2024-04-16T21:07:19.650574+0000","last_clean":"2024-04-16T21:07:19.650574+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:07:19.650574+0000","last_undegraded":"2024-04-16T21:07:19.650574+0000","last_fullsized":"2024-04-16T21:07:19.650574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:44.374493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1826,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365476+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:08:49.365476+0000","last_peered":"2024-04-16T21:08:49.365476+0000","last_clean":"2024-04-16T21:08:49.365476+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:08:49.365476+0000","last_undegraded":"2024-04-16T21:08:49.365476+0000","last_fullsized":"2024-04-16T21:08:49.365476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:09.652050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649723+0000","last_change":"2024-04-16T19:25:41.504184+0000","last_active":"2024-04-16T21:07:14.649723+0000","last_peered":"2024-04-16T21:07:14.649723+0000","last_clean":"2024-04-16T21:07:14.649723+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:07:14.649723+0000","last_undegraded":"2024-04-16T21:07:14.649723+0000","last_fullsized":"2024-04-16T21:07:14.649723+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645440+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:06:49.645440+0000","last_peered":"2024-04-16T21:06:49.645440+0000","last_clean":"2024-04-16T21:06:49.645440+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:06:49.645440+0000","last_undegraded":"2024-04-16T21:06:49.645440+0000","last_fullsized":"2024-04-16T21:06:49.645440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:47:48.700988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497245+0000","last_change":"2024-04-16T19:25:13.472015+0000","last_active":"2024-04-16T21:07:54.497245+0000","last_peered":"2024-04-16T21:07:54.497245+0000","last_clean":"2024-04-16T21:07:54.497245+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:07:54.497245+0000","last_undegraded":"2024-04-16T21:07:54.497245+0000","last_fullsized":"2024-04-16T21:07:54.497245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'121","reported_seq":272,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655968+0000","last_change":"2024-04-16T19:25:11.465848+0000","last_active":"2024-04-16T21:07:54.655968+0000","last_peered":"2024-04-16T21:07:54.655968+0000","last_clean":"2024-04-16T21:07:54.655968+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:07:54.655968+0000","last_undegraded":"2024-04-16T21:07:54.655968+0000","last_fullsized":"2024-04-16T21:07:54.655968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'4","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655987+0000","last_change":"2024-04-16T19:25:09.462205+0000","last_active":"2024-04-16T21:07:49.655987+0000","last_peered":"2024-04-16T21:07:49.655987+0000","last_clean":"2024-04-16T21:07:49.655987+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:07:49.655987+0000","last_undegraded":"2024-04-16T21:07:49.655987+0000","last_fullsized":"2024-04-16T21:07:49.655987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:53.677676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372422+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:08:59.372422+0000","last_peered":"2024-04-16T21:08:59.372422+0000","last_clean":"2024-04-16T21:08:59.372422+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:08:59.372422+0000","last_undegraded":"2024-04-16T21:08:59.372422+0000","last_fullsized":"2024-04-16T21:08:59.372422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364818+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:08:49.364818+0000","last_peered":"2024-04-16T21:08:49.364818+0000","last_clean":"2024-04-16T21:08:49.364818+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:08:49.364818+0000","last_undegraded":"2024-04-16T21:08:49.364818+0000","last_fullsized":"2024-04-16T21:08:49.364818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.368088+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:08:59.368088+0000","last_peered":"2024-04-16T21:08:59.368088+0000","last_clean":"2024-04-16T21:08:59.368088+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:08:59.368088+0000","last_undegraded":"2024-04-16T21:08:59.368088+0000","last_fullsized":"2024-04-16T21:08:59.368088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:34.577971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651049+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:07:19.651049+0000","last_peered":"2024-04-16T21:07:19.651049+0000","last_clean":"2024-04-16T21:07:19.651049+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:07:19.651049+0000","last_undegraded":"2024-04-16T21:07:19.651049+0000","last_fullsized":"2024-04-16T21:07:19.651049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:14.554679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491744+0000","last_change":"2024-04-16T19:24:49.432195+0000","last_active":"2024-04-16T21:07:19.491744+0000","last_peered":"2024-04-16T21:07:19.491744+0000","last_clean":"2024-04-16T21:07:19.491744+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:07:19.491744+0000","last_undegraded":"2024-04-16T21:07:19.491744+0000","last_fullsized":"2024-04-16T21:07:19.491744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:18.116203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653498+0000","last_change":"2024-04-16T19:24:51.573063+0000","last_active":"2024-04-16T21:07:34.653498+0000","last_peered":"2024-04-16T21:07:34.653498+0000","last_clean":"2024-04-16T21:07:34.653498+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:07:34.653498+0000","last_undegraded":"2024-04-16T21:07:34.653498+0000","last_fullsized":"2024-04-16T21:07:34.653498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:01.882223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649776+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:07:19.649776+0000","last_peered":"2024-04-16T21:07:19.649776+0000","last_clean":"2024-04-16T21:07:19.649776+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:07:19.649776+0000","last_undegraded":"2024-04-16T21:07:19.649776+0000","last_fullsized":"2024-04-16T21:07:19.649776+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:28:47.506010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491827+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:07:19.491827+0000","last_peered":"2024-04-16T21:07:19.491827+0000","last_clean":"2024-04-16T21:07:19.491827+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:07:19.491827+0000","last_undegraded":"2024-04-16T21:07:19.491827+0000","last_fullsized":"2024-04-16T21:07:19.491827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:32:33.273769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662724+0000","last_change":"2024-04-16T19:25:41.509494+0000","last_active":"2024-04-16T21:08:39.662724+0000","last_peered":"2024-04-16T21:08:39.662724+0000","last_clean":"2024-04-16T21:08:39.662724+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:08:39.662724+0000","last_undegraded":"2024-04-16T21:08:39.662724+0000","last_fullsized":"2024-04-16T21:08:39.662724+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'8","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:49.645398+0000","last_change":"2024-04-16T19:25:15.483611+0000","last_active":"2024-04-16T21:06:49.645398+0000","last_peered":"2024-04-16T21:06:49.645398+0000","last_clean":"2024-04-16T21:06:49.645398+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:06:49.645398+0000","last_undegraded":"2024-04-16T21:06:49.645398+0000","last_fullsized":"2024-04-16T21:06:49.645398+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656848+0000","last_change":"2024-04-16T19:25:13.472723+0000","last_active":"2024-04-16T21:07:54.656848+0000","last_peered":"2024-04-16T21:07:54.656848+0000","last_clean":"2024-04-16T21:07:54.656848+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:07:54.656848+0000","last_undegraded":"2024-04-16T21:07:54.656848+0000","last_fullsized":"2024-04-16T21:07:54.656848+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'321","reported_seq":850,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497294+0000","last_change":"2024-04-16T19:25:11.464441+0000","last_active":"2024-04-16T21:07:54.497294+0000","last_peered":"2024-04-16T21:07:54.497294+0000","last_clean":"2024-04-16T21:07:54.497294+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:07:54.497294+0000","last_undegraded":"2024-04-16T21:07:54.497294+0000","last_fullsized":"2024-04-16T21:07:54.497294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:55.683718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'12","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655101+0000","last_change":"2024-04-16T19:25:09.465881+0000","last_active":"2024-04-16T21:07:49.655101+0000","last_peered":"2024-04-16T21:07:49.655101+0000","last_clean":"2024-04-16T21:07:49.655101+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:07:49.655101+0000","last_undegraded":"2024-04-16T21:07:49.655101+0000","last_fullsized":"2024-04-16T21:07:49.655101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648098+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:07:09.648098+0000","last_peered":"2024-04-16T21:07:09.648098+0000","last_clean":"2024-04-16T21:07:09.648098+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:07:09.648098+0000","last_undegraded":"2024-04-16T21:07:09.648098+0000","last_fullsized":"2024-04-16T21:07:09.648098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648274+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:07:04.648274+0000","last_peered":"2024-04-16T21:07:04.648274+0000","last_clean":"2024-04-16T21:07:04.648274+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:07:04.648274+0000","last_undegraded":"2024-04-16T21:07:04.648274+0000","last_fullsized":"2024-04-16T21:07:04.648274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:48.449534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650601+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:07:19.650601+0000","last_peered":"2024-04-16T21:07:19.650601+0000","last_clean":"2024-04-16T21:07:19.650601+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:07:19.650601+0000","last_undegraded":"2024-04-16T21:07:19.650601+0000","last_fullsized":"2024-04-16T21:07:19.650601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:17.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507635+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:08:59.507635+0000","last_peered":"2024-04-16T21:08:59.507635+0000","last_clean":"2024-04-16T21:08:59.507635+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:08:59.507635+0000","last_undegraded":"2024-04-16T21:08:59.507635+0000","last_fullsized":"2024-04-16T21:08:59.507635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:40.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368151+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:09:09.368151+0000","last_peered":"2024-04-16T21:09:09.368151+0000","last_clean":"2024-04-16T21:09:09.368151+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:09:09.368151+0000","last_undegraded":"2024-04-16T21:09:09.368151+0000","last_fullsized":"2024-04-16T21:09:09.368151+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:22.117545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653453+0000","last_change":"2024-04-16T19:24:51.442399+0000","last_active":"2024-04-16T21:07:34.653453+0000","last_peered":"2024-04-16T21:07:34.653453+0000","last_clean":"2024-04-16T21:07:34.653453+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:07:34.653453+0000","last_undegraded":"2024-04-16T21:07:34.653453+0000","last_fullsized":"2024-04-16T21:07:34.653453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:24.851144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650701+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:07:19.650701+0000","last_peered":"2024-04-16T21:07:19.650701+0000","last_clean":"2024-04-16T21:07:19.650701+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:07:19.650701+0000","last_undegraded":"2024-04-16T21:07:19.650701+0000","last_fullsized":"2024-04-16T21:07:19.650701+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:42.705352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1306,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491787+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:07:19.491787+0000","last_peered":"2024-04-16T21:07:19.491787+0000","last_clean":"2024-04-16T21:07:19.491787+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:07:19.491787+0000","last_undegraded":"2024-04-16T21:07:19.491787+0000","last_fullsized":"2024-04-16T21:07:19.491787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:35:13.067416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649861+0000","last_change":"2024-04-16T19:25:41.509274+0000","last_active":"2024-04-16T21:07:14.649861+0000","last_peered":"2024-04-16T21:07:14.649861+0000","last_clean":"2024-04-16T21:07:14.649861+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:07:14.649861+0000","last_undegraded":"2024-04-16T21:07:14.649861+0000","last_fullsized":"2024-04-16T21:07:14.649861+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:45.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"47'5","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659574+0000","last_change":"2024-04-16T19:25:15.486177+0000","last_active":"2024-04-16T21:08:14.659574+0000","last_peered":"2024-04-16T21:08:14.659574+0000","last_clean":"2024-04-16T21:08:14.659574+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:08:14.659574+0000","last_undegraded":"2024-04-16T21:08:14.659574+0000","last_fullsized":"2024-04-16T21:08:14.659574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:20.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656265+0000","last_change":"2024-04-16T19:25:13.473925+0000","last_active":"2024-04-16T21:07:54.656265+0000","last_peered":"2024-04-16T21:07:54.656265+0000","last_clean":"2024-04-16T21:07:54.656265+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:07:54.656265+0000","last_undegraded":"2024-04-16T21:07:54.656265+0000","last_fullsized":"2024-04-16T21:07:54.656265+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:22.051385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'241","reported_seq":1104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656229+0000","last_change":"2024-04-16T19:25:11.465793+0000","last_active":"2024-04-16T21:07:54.656229+0000","last_peered":"2024-04-16T21:07:54.656229+0000","last_clean":"2024-04-16T21:07:54.656229+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:07:54.656229+0000","last_undegraded":"2024-04-16T21:07:54.656229+0000","last_fullsized":"2024-04-16T21:07:54.656229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:00.538757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'7","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655923+0000","last_change":"2024-04-16T19:25:09.458604+0000","last_active":"2024-04-16T21:07:49.655923+0000","last_peered":"2024-04-16T21:07:49.655923+0000","last_clean":"2024-04-16T21:07:49.655923+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:07:49.655923+0000","last_undegraded":"2024-04-16T21:07:49.655923+0000","last_fullsized":"2024-04-16T21:07:49.655923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489948+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:07:09.489948+0000","last_peered":"2024-04-16T21:07:09.489948+0000","last_clean":"2024-04-16T21:07:09.489948+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:07:09.489948+0000","last_undegraded":"2024-04-16T21:07:09.489948+0000","last_fullsized":"2024-04-16T21:07:09.489948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647363+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:07:04.647363+0000","last_peered":"2024-04-16T21:07:04.647363+0000","last_clean":"2024-04-16T21:07:04.647363+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:07:04.647363+0000","last_undegraded":"2024-04-16T21:07:04.647363+0000","last_fullsized":"2024-04-16T21:07:04.647363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491726+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:07:19.491726+0000","last_peered":"2024-04-16T21:07:19.491726+0000","last_clean":"2024-04-16T21:07:19.491726+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:07:19.491726+0000","last_undegraded":"2024-04-16T21:07:19.491726+0000","last_fullsized":"2024-04-16T21:07:19.491726+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:45.917687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369571+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:09:09.369571+0000","last_peered":"2024-04-16T21:09:09.369571+0000","last_clean":"2024-04-16T21:09:09.369571+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:09:09.369571+0000","last_undegraded":"2024-04-16T21:09:09.369571+0000","last_fullsized":"2024-04-16T21:09:09.369571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:32.829903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367311+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:08:59.367311+0000","last_peered":"2024-04-16T21:08:59.367311+0000","last_clean":"2024-04-16T21:08:59.367311+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:08:59.367311+0000","last_undegraded":"2024-04-16T21:08:59.367311+0000","last_fullsized":"2024-04-16T21:08:59.367311+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:02.604453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653371+0000","last_change":"2024-04-16T19:24:51.438145+0000","last_active":"2024-04-16T21:07:34.653371+0000","last_peered":"2024-04-16T21:07:34.653371+0000","last_clean":"2024-04-16T21:07:34.653371+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:07:34.653371+0000","last_undegraded":"2024-04-16T21:07:34.653371+0000","last_fullsized":"2024-04-16T21:07:34.653371+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:45.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369666+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:09:14.369666+0000","last_peered":"2024-04-16T21:09:14.369666+0000","last_clean":"2024-04-16T21:09:14.369666+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:09:14.369666+0000","last_undegraded":"2024-04-16T21:09:14.369666+0000","last_fullsized":"2024-04-16T21:09:14.369666+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:31.554611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661634+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:08:29.661634+0000","last_peered":"2024-04-16T21:08:29.661634+0000","last_clean":"2024-04-16T21:08:29.661634+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:08:29.661634+0000","last_undegraded":"2024-04-16T21:08:29.661634+0000","last_fullsized":"2024-04-16T21:08:29.661634+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:20.643182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660228+0000","last_change":"2024-04-16T19:25:41.508939+0000","last_active":"2024-04-16T21:08:24.660228+0000","last_peered":"2024-04-16T21:08:24.660228+0000","last_clean":"2024-04-16T21:08:24.660228+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:08:24.660228+0000","last_undegraded":"2024-04-16T21:08:24.660228+0000","last_fullsized":"2024-04-16T21:08:24.660228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'11","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498087+0000","last_change":"2024-04-16T19:25:15.480096+0000","last_active":"2024-04-16T21:07:59.498087+0000","last_peered":"2024-04-16T21:07:59.498087+0000","last_clean":"2024-04-16T21:07:59.498087+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:07:59.498087+0000","last_undegraded":"2024-04-16T21:07:59.498087+0000","last_fullsized":"2024-04-16T21:07:59.498087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:58.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659661+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:08:09.659661+0000","last_peered":"2024-04-16T21:08:09.659661+0000","last_clean":"2024-04-16T21:08:09.659661+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:08:09.659661+0000","last_undegraded":"2024-04-16T21:08:09.659661+0000","last_fullsized":"2024-04-16T21:08:09.659661+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'540","reported_seq":1356,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373967+0000","last_change":"2024-04-16T19:25:11.465566+0000","last_active":"2024-04-16T21:09:09.373967+0000","last_peered":"2024-04-16T21:09:09.373967+0000","last_clean":"2024-04-16T21:09:09.373967+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:09:09.373967+0000","last_undegraded":"2024-04-16T21:09:09.373967+0000","last_fullsized":"2024-04-16T21:09:09.373967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:52.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'9","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368023+0000","last_change":"2024-04-16T19:25:09.458347+0000","last_active":"2024-04-16T21:09:04.368023+0000","last_peered":"2024-04-16T21:09:04.368023+0000","last_clean":"2024-04-16T21:09:04.368023+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:09:04.368023+0000","last_undegraded":"2024-04-16T21:09:04.368023+0000","last_fullsized":"2024-04-16T21:09:04.368023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649060+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:07:09.649060+0000","last_peered":"2024-04-16T21:07:09.649060+0000","last_clean":"2024-04-16T21:07:09.649060+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:07:09.649060+0000","last_undegraded":"2024-04-16T21:07:09.649060+0000","last_fullsized":"2024-04-16T21:07:09.649060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:41.726894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648043+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:07:04.648043+0000","last_peered":"2024-04-16T21:07:04.648043+0000","last_clean":"2024-04-16T21:07:04.648043+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:07:04.648043+0000","last_undegraded":"2024-04-16T21:07:04.648043+0000","last_fullsized":"2024-04-16T21:07:04.648043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650514+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:07:19.650514+0000","last_peered":"2024-04-16T21:07:19.650514+0000","last_clean":"2024-04-16T21:07:19.650514+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:07:19.650514+0000","last_undegraded":"2024-04-16T21:07:19.650514+0000","last_fullsized":"2024-04-16T21:07:19.650514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:46.054276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367931+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:09:04.367931+0000","last_peered":"2024-04-16T21:09:04.367931+0000","last_clean":"2024-04-16T21:09:04.367931+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:09:04.367931+0000","last_undegraded":"2024-04-16T21:09:04.367931+0000","last_fullsized":"2024-04-16T21:09:04.367931+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:47.298587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369608+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:08:44.369608+0000","last_peered":"2024-04-16T21:08:44.369608+0000","last_clean":"2024-04-16T21:08:44.369608+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:08:44.369608+0000","last_undegraded":"2024-04-16T21:08:44.369608+0000","last_fullsized":"2024-04-16T21:08:44.369608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:38:59.670330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366563+0000","last_change":"2024-04-16T19:24:51.437809+0000","last_active":"2024-04-16T21:08:49.366563+0000","last_peered":"2024-04-16T21:08:49.366563+0000","last_clean":"2024-04-16T21:08:49.366563+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:08:49.366563+0000","last_undegraded":"2024-04-16T21:08:49.366563+0000","last_fullsized":"2024-04-16T21:08:49.366563+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509969+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:09:14.509969+0000","last_peered":"2024-04-16T21:09:14.509969+0000","last_clean":"2024-04-16T21:09:14.509969+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:09:14.509969+0000","last_undegraded":"2024-04-16T21:09:14.509969+0000","last_fullsized":"2024-04-16T21:09:14.509969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:55:01.357303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1337,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502628+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:08:29.502628+0000","last_peered":"2024-04-16T21:08:29.502628+0000","last_clean":"2024-04-16T21:08:29.502628+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:08:29.502628+0000","last_undegraded":"2024-04-16T21:08:29.502628+0000","last_fullsized":"2024-04-16T21:08:29.502628+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:48.136957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661216+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:08:24.661216+0000","last_peered":"2024-04-16T21:08:24.661216+0000","last_clean":"2024-04-16T21:08:24.661216+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:08:24.661216+0000","last_undegraded":"2024-04-16T21:08:24.661216+0000","last_fullsized":"2024-04-16T21:08:24.661216+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:47.003646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'8","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656179+0000","last_change":"2024-04-16T19:25:15.481698+0000","last_active":"2024-04-16T21:07:59.656179+0000","last_peered":"2024-04-16T21:07:59.656179+0000","last_clean":"2024-04-16T21:07:59.656179+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:07:59.656179+0000","last_undegraded":"2024-04-16T21:07:59.656179+0000","last_fullsized":"2024-04-16T21:07:59.656179+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'705","reported_seq":2072,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509303+0000","last_change":"2024-04-16T19:25:13.474684+0000","last_active":"2024-04-16T21:09:09.509303+0000","last_peered":"2024-04-16T21:09:09.509303+0000","last_clean":"2024-04-16T21:09:09.509303+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:09:09.509303+0000","last_undegraded":"2024-04-16T21:09:09.509303+0000","last_fullsized":"2024-04-16T21:09:09.509303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:52.004725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'519","reported_seq":41267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499619+0000","last_change":"2024-04-16T19:25:11.465897+0000","last_active":"2024-04-16T21:08:09.499619+0000","last_peered":"2024-04-16T21:08:09.499619+0000","last_clean":"2024-04-16T21:08:09.499619+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:08:09.499619+0000","last_undegraded":"2024-04-16T21:08:09.499619+0000","last_fullsized":"2024-04-16T21:08:09.499619+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:54.168091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'9","reported_seq":308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368812+0000","last_change":"2024-04-16T19:25:09.459964+0000","last_active":"2024-04-16T21:09:04.368812+0000","last_peered":"2024-04-16T21:09:04.368812+0000","last_clean":"2024-04-16T21:09:04.368812+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:09:04.368812+0000","last_undegraded":"2024-04-16T21:09:04.368812+0000","last_fullsized":"2024-04-16T21:09:04.368812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649029+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:07:09.649029+0000","last_peered":"2024-04-16T21:07:09.649029+0000","last_clean":"2024-04-16T21:07:09.649029+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:07:09.649029+0000","last_undegraded":"2024-04-16T21:07:09.649029+0000","last_fullsized":"2024-04-16T21:07:09.649029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648095+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:07:04.648095+0000","last_peered":"2024-04-16T21:07:04.648095+0000","last_clean":"2024-04-16T21:07:04.648095+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:07:04.648095+0000","last_undegraded":"2024-04-16T21:07:04.648095+0000","last_fullsized":"2024-04-16T21:07:04.648095+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:23.870654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:04.143195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369636+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:08:44.369636+0000","last_peered":"2024-04-16T21:08:44.369636+0000","last_clean":"2024-04-16T21:08:44.369636+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:08:44.369636+0000","last_undegraded":"2024-04-16T21:08:44.369636+0000","last_fullsized":"2024-04-16T21:08:44.369636+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368656+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:09:04.368656+0000","last_peered":"2024-04-16T21:09:04.368656+0000","last_clean":"2024-04-16T21:09:04.368656+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:09:04.368656+0000","last_undegraded":"2024-04-16T21:09:04.368656+0000","last_fullsized":"2024-04-16T21:09:04.368656+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364971+0000","last_change":"2024-04-16T19:24:51.573983+0000","last_active":"2024-04-16T21:08:49.364971+0000","last_peered":"2024-04-16T21:08:49.364971+0000","last_clean":"2024-04-16T21:08:49.364971+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:08:49.364971+0000","last_undegraded":"2024-04-16T21:08:49.364971+0000","last_fullsized":"2024-04-16T21:08:49.364971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374829+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:09:14.374829+0000","last_peered":"2024-04-16T21:09:14.374829+0000","last_clean":"2024-04-16T21:09:14.374829+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:09:14.374829+0000","last_undegraded":"2024-04-16T21:09:14.374829+0000","last_fullsized":"2024-04-16T21:09:14.374829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'167","reported_seq":2111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502381+0000","last_change":"2024-04-16T19:25:49.519298+0000","last_active":"2024-04-16T21:08:29.502381+0000","last_peered":"2024-04-16T21:08:29.502381+0000","last_clean":"2024-04-16T21:08:29.502381+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:08:29.502381+0000","last_undegraded":"2024-04-16T21:08:29.502381+0000","last_fullsized":"2024-04-16T21:08:29.502381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:10.645836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501890+0000","last_change":"2024-04-16T19:25:41.507268+0000","last_active":"2024-04-16T21:08:24.501890+0000","last_peered":"2024-04-16T21:08:24.501890+0000","last_clean":"2024-04-16T21:08:24.501890+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:08:24.501890+0000","last_undegraded":"2024-04-16T21:08:24.501890+0000","last_fullsized":"2024-04-16T21:08:24.501890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:46.285785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657100+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:07:59.657100+0000","last_peered":"2024-04-16T21:07:59.657100+0000","last_clean":"2024-04-16T21:07:59.657100+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:07:59.657100+0000","last_undegraded":"2024-04-16T21:07:59.657100+0000","last_fullsized":"2024-04-16T21:07:59.657100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:46.607704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369049+0000","last_change":"2024-04-16T19:25:13.472244+0000","last_active":"2024-04-16T21:09:09.369049+0000","last_peered":"2024-04-16T21:09:09.369049+0000","last_clean":"2024-04-16T21:09:09.369049+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:09:09.369049+0000","last_undegraded":"2024-04-16T21:09:09.369049+0000","last_fullsized":"2024-04-16T21:09:09.369049+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:59.248190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'573","reported_seq":1335,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369613+0000","last_change":"2024-04-16T19:25:11.468986+0000","last_active":"2024-04-16T21:09:09.369613+0000","last_peered":"2024-04-16T21:09:09.369613+0000","last_clean":"2024-04-16T21:09:09.369613+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:09:09.369613+0000","last_undegraded":"2024-04-16T21:09:09.369613+0000","last_fullsized":"2024-04-16T21:09:09.369613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'23","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498919+0000","last_change":"2024-04-16T19:25:09.469035+0000","last_active":"2024-04-16T21:08:04.498919+0000","last_peered":"2024-04-16T21:08:04.498919+0000","last_clean":"2024-04-16T21:08:04.498919+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:08:04.498919+0000","last_undegraded":"2024-04-16T21:08:04.498919+0000","last_fullsized":"2024-04-16T21:08:04.498919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371344+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:08:54.371344+0000","last_peered":"2024-04-16T21:08:54.371344+0000","last_clean":"2024-04-16T21:08:54.371344+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:08:54.371344+0000","last_undegraded":"2024-04-16T21:08:54.371344+0000","last_fullsized":"2024-04-16T21:08:54.371344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506100+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:08:49.506100+0000","last_peered":"2024-04-16T21:08:49.506100+0000","last_clean":"2024-04-16T21:08:49.506100+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:08:49.506100+0000","last_undegraded":"2024-04-16T21:08:49.506100+0000","last_fullsized":"2024-04-16T21:08:49.506100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:11.958520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649687+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:07:19.649687+0000","last_peered":"2024-04-16T21:07:19.649687+0000","last_clean":"2024-04-16T21:07:19.649687+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:07:19.649687+0000","last_undegraded":"2024-04-16T21:07:19.649687+0000","last_fullsized":"2024-04-16T21:07:19.649687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650982+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:07:19.650982+0000","last_peered":"2024-04-16T21:07:19.650982+0000","last_clean":"2024-04-16T21:07:19.650982+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:07:19.650982+0000","last_undegraded":"2024-04-16T21:07:19.650982+0000","last_fullsized":"2024-04-16T21:07:19.650982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366474+0000","last_change":"2024-04-16T19:24:51.444882+0000","last_active":"2024-04-16T21:08:49.366474+0000","last_peered":"2024-04-16T21:08:49.366474+0000","last_clean":"2024-04-16T21:08:49.366474+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:08:49.366474+0000","last_undegraded":"2024-04-16T21:08:49.366474+0000","last_fullsized":"2024-04-16T21:08:49.366474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:18.795884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368933+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:09:14.368933+0000","last_peered":"2024-04-16T21:09:14.368933+0000","last_clean":"2024-04-16T21:09:14.368933+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:09:14.368933+0000","last_undegraded":"2024-04-16T21:09:14.368933+0000","last_fullsized":"2024-04-16T21:09:14.368933+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:37:36.899944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'127","reported_seq":1509,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502335+0000","last_change":"2024-04-16T19:25:49.517472+0000","last_active":"2024-04-16T21:08:29.502335+0000","last_peered":"2024-04-16T21:08:29.502335+0000","last_clean":"2024-04-16T21:08:29.502335+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:08:29.502335+0000","last_undegraded":"2024-04-16T21:08:29.502335+0000","last_fullsized":"2024-04-16T21:08:29.502335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:30.634191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501937+0000","last_change":"2024-04-16T19:25:41.502982+0000","last_active":"2024-04-16T21:08:24.501937+0000","last_peered":"2024-04-16T21:08:24.501937+0000","last_clean":"2024-04-16T21:08:24.501937+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:08:24.501937+0000","last_undegraded":"2024-04-16T21:08:24.501937+0000","last_fullsized":"2024-04-16T21:08:24.501937+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:12.630377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"47'10","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656132+0000","last_change":"2024-04-16T19:25:15.481921+0000","last_active":"2024-04-16T21:07:59.656132+0000","last_peered":"2024-04-16T21:07:59.656132+0000","last_clean":"2024-04-16T21:07:59.656132+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:07:59.656132+0000","last_undegraded":"2024-04-16T21:07:59.656132+0000","last_fullsized":"2024-04-16T21:07:59.656132+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369098+0000","last_change":"2024-04-16T19:25:13.473915+0000","last_active":"2024-04-16T21:09:09.369098+0000","last_peered":"2024-04-16T21:09:09.369098+0000","last_clean":"2024-04-16T21:09:09.369098+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:09:09.369098+0000","last_undegraded":"2024-04-16T21:09:09.369098+0000","last_fullsized":"2024-04-16T21:09:09.369098+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:05.770561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'230","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369554+0000","last_change":"2024-04-16T19:25:11.461723+0000","last_active":"2024-04-16T21:09:09.369554+0000","last_peered":"2024-04-16T21:09:09.369554+0000","last_clean":"2024-04-16T21:09:09.369554+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:09:09.369554+0000","last_undegraded":"2024-04-16T21:09:09.369554+0000","last_fullsized":"2024-04-16T21:09:09.369554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:15.327068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'6","reported_seq":175,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367981+0000","last_change":"2024-04-16T19:25:09.458030+0000","last_active":"2024-04-16T21:09:04.367981+0000","last_peered":"2024-04-16T21:09:04.367981+0000","last_clean":"2024-04-16T21:09:04.367981+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:09:04.367981+0000","last_undegraded":"2024-04-16T21:09:04.367981+0000","last_fullsized":"2024-04-16T21:09:04.367981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663306+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:08:34.663306+0000","last_peered":"2024-04-16T21:08:34.663306+0000","last_clean":"2024-04-16T21:08:34.663306+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:08:34.663306+0000","last_undegraded":"2024-04-16T21:08:34.663306+0000","last_fullsized":"2024-04-16T21:08:34.663306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369834+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:08:44.369834+0000","last_peered":"2024-04-16T21:08:44.369834+0000","last_clean":"2024-04-16T21:08:44.369834+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:08:44.369834+0000","last_undegraded":"2024-04-16T21:08:44.369834+0000","last_fullsized":"2024-04-16T21:08:44.369834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:41.742389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367273+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:09:04.367273+0000","last_peered":"2024-04-16T21:09:04.367273+0000","last_clean":"2024-04-16T21:09:04.367273+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:09:04.367273+0000","last_undegraded":"2024-04-16T21:09:04.367273+0000","last_fullsized":"2024-04-16T21:09:04.367273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:38.387958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":222,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650527+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:07:19.650527+0000","last_peered":"2024-04-16T21:07:19.650527+0000","last_clean":"2024-04-16T21:07:19.650527+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:07:19.650527+0000","last_undegraded":"2024-04-16T21:07:19.650527+0000","last_fullsized":"2024-04-16T21:07:19.650527+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:47.927883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649659+0000","last_change":"2024-04-16T19:24:49.432869+0000","last_active":"2024-04-16T21:07:19.649659+0000","last_peered":"2024-04-16T21:07:19.649659+0000","last_clean":"2024-04-16T21:07:19.649659+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:07:19.649659+0000","last_undegraded":"2024-04-16T21:07:19.649659+0000","last_fullsized":"2024-04-16T21:07:19.649659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655242+0000","last_change":"2024-04-16T19:24:51.442598+0000","last_active":"2024-04-16T21:07:49.655242+0000","last_peered":"2024-04-16T21:07:49.655242+0000","last_clean":"2024-04-16T21:07:49.655242+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:07:49.655242+0000","last_undegraded":"2024-04-16T21:07:49.655242+0000","last_fullsized":"2024-04-16T21:07:49.655242+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:52.222795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'12","reported_seq":186,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504441+0000","last_change":"2024-04-16T19:25:09.464004+0000","last_active":"2024-04-16T21:08:39.504441+0000","last_peered":"2024-04-16T21:08:39.504441+0000","last_clean":"2024-04-16T21:08:39.504441+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:08:39.504441+0000","last_undegraded":"2024-04-16T21:08:39.504441+0000","last_fullsized":"2024-04-16T21:08:39.504441+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504683+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:08:39.504683+0000","last_peered":"2024-04-16T21:08:39.504683+0000","last_clean":"2024-04-16T21:08:39.504683+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:08:39.504683+0000","last_undegraded":"2024-04-16T21:08:39.504683+0000","last_fullsized":"2024-04-16T21:08:39.504683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364532+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:08:44.364532+0000","last_peered":"2024-04-16T21:08:44.364532+0000","last_clean":"2024-04-16T21:08:44.364532+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:08:44.364532+0000","last_undegraded":"2024-04-16T21:08:44.364532+0000","last_fullsized":"2024-04-16T21:08:44.364532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'12","reported_seq":242,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663758+0000","last_change":"2024-04-16T19:25:09.466486+0000","last_active":"2024-04-16T21:08:39.663758+0000","last_peered":"2024-04-16T21:08:39.663758+0000","last_clean":"2024-04-16T21:08:39.663758+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:08:39.663758+0000","last_undegraded":"2024-04-16T21:08:39.663758+0000","last_fullsized":"2024-04-16T21:08:39.663758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504632+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:08:39.504632+0000","last_peered":"2024-04-16T21:08:39.504632+0000","last_clean":"2024-04-16T21:08:39.504632+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:08:39.504632+0000","last_undegraded":"2024-04-16T21:08:39.504632+0000","last_fullsized":"2024-04-16T21:08:39.504632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:49.140683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505241+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:08:44.505241+0000","last_peered":"2024-04-16T21:08:44.505241+0000","last_clean":"2024-04-16T21:08:44.505241+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:08:44.505241+0000","last_undegraded":"2024-04-16T21:08:44.505241+0000","last_fullsized":"2024-04-16T21:08:44.505241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:03:25.186300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'5","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649520+0000","last_change":"2024-04-16T19:25:09.467044+0000","last_active":"2024-04-16T21:07:14.649520+0000","last_peered":"2024-04-16T21:07:14.649520+0000","last_clean":"2024-04-16T21:07:14.649520+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:07:14.649520+0000","last_undegraded":"2024-04-16T21:07:14.649520+0000","last_fullsized":"2024-04-16T21:07:14.649520+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663818+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:08:39.663818+0000","last_peered":"2024-04-16T21:08:39.663818+0000","last_clean":"2024-04-16T21:08:39.663818+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:08:39.663818+0000","last_undegraded":"2024-04-16T21:08:39.663818+0000","last_fullsized":"2024-04-16T21:08:39.663818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365701+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:08:44.365701+0000","last_peered":"2024-04-16T21:08:44.365701+0000","last_clean":"2024-04-16T21:08:44.365701+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:08:44.365701+0000","last_undegraded":"2024-04-16T21:08:44.365701+0000","last_fullsized":"2024-04-16T21:08:44.365701+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'7","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664206+0000","last_change":"2024-04-16T19:25:09.458671+0000","last_active":"2024-04-16T21:08:39.664206+0000","last_peered":"2024-04-16T21:08:39.664206+0000","last_clean":"2024-04-16T21:08:39.664206+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:08:39.664206+0000","last_undegraded":"2024-04-16T21:08:39.664206+0000","last_fullsized":"2024-04-16T21:08:39.664206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:52.162049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504531+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:08:39.504531+0000","last_peered":"2024-04-16T21:08:39.504531+0000","last_clean":"2024-04-16T21:08:39.504531+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:08:39.504531+0000","last_undegraded":"2024-04-16T21:08:39.504531+0000","last_fullsized":"2024-04-16T21:08:39.504531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:22.916044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373116+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:09:04.373116+0000","last_peered":"2024-04-16T21:09:04.373116+0000","last_clean":"2024-04-16T21:09:04.373116+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:09:04.373116+0000","last_undegraded":"2024-04-16T21:09:04.373116+0000","last_fullsized":"2024-04-16T21:09:04.373116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'11","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655456+0000","last_change":"2024-04-16T19:25:09.466470+0000","last_active":"2024-04-16T21:07:49.655456+0000","last_peered":"2024-04-16T21:07:49.655456+0000","last_clean":"2024-04-16T21:07:49.655456+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:07:49.655456+0000","last_undegraded":"2024-04-16T21:07:49.655456+0000","last_fullsized":"2024-04-16T21:07:49.655456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508457+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:09:04.508457+0000","last_peered":"2024-04-16T21:09:04.508457+0000","last_clean":"2024-04-16T21:09:04.508457+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:09:04.508457+0000","last_undegraded":"2024-04-16T21:09:04.508457+0000","last_fullsized":"2024-04-16T21:09:04.508457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:58.689460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503608+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:08:34.503608+0000","last_peered":"2024-04-16T21:08:34.503608+0000","last_clean":"2024-04-16T21:08:34.503608+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:08:34.503608+0000","last_undegraded":"2024-04-16T21:08:34.503608+0000","last_fullsized":"2024-04-16T21:08:34.503608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:49.978039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'20","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496627+0000","last_change":"2024-04-16T19:25:09.460695+0000","last_active":"2024-04-16T21:07:49.496627+0000","last_peered":"2024-04-16T21:07:49.496627+0000","last_clean":"2024-04-16T21:07:49.496627+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:07:49.496627+0000","last_undegraded":"2024-04-16T21:07:49.496627+0000","last_fullsized":"2024-04-16T21:07:49.496627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:28.879723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365419+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:08:49.365419+0000","last_peered":"2024-04-16T21:08:49.365419+0000","last_clean":"2024-04-16T21:08:49.365419+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:08:49.365419+0000","last_undegraded":"2024-04-16T21:08:49.365419+0000","last_fullsized":"2024-04-16T21:08:49.365419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:16.805347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503638+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:08:34.503638+0000","last_peered":"2024-04-16T21:08:34.503638+0000","last_clean":"2024-04-16T21:08:34.503638+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:08:34.503638+0000","last_undegraded":"2024-04-16T21:08:34.503638+0000","last_fullsized":"2024-04-16T21:08:34.503638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'11","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504570+0000","last_change":"2024-04-16T19:25:09.459969+0000","last_active":"2024-04-16T21:08:39.504570+0000","last_peered":"2024-04-16T21:08:39.504570+0000","last_clean":"2024-04-16T21:08:39.504570+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:08:39.504570+0000","last_undegraded":"2024-04-16T21:08:39.504570+0000","last_fullsized":"2024-04-16T21:08:39.504570+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364844+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:08:49.364844+0000","last_peered":"2024-04-16T21:08:49.364844+0000","last_clean":"2024-04-16T21:08:49.364844+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:08:49.364844+0000","last_undegraded":"2024-04-16T21:08:49.364844+0000","last_fullsized":"2024-04-16T21:08:49.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663475+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:08:34.663475+0000","last_peered":"2024-04-16T21:08:34.663475+0000","last_clean":"2024-04-16T21:08:34.663475+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:08:34.663475+0000","last_undegraded":"2024-04-16T21:08:34.663475+0000","last_fullsized":"2024-04-16T21:08:34.663475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:40.025589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'13","reported_seq":290,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.506803+0000","last_change":"2024-04-16T19:25:09.459396+0000","last_active":"2024-04-16T21:08:54.506803+0000","last_peered":"2024-04-16T21:08:54.506803+0000","last_clean":"2024-04-16T21:08:54.506803+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:08:54.506803+0000","last_undegraded":"2024-04-16T21:08:54.506803+0000","last_fullsized":"2024-04-16T21:08:54.506803+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:44.854219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366364+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:08:49.366364+0000","last_peered":"2024-04-16T21:08:49.366364+0000","last_clean":"2024-04-16T21:08:49.366364+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:08:49.366364+0000","last_undegraded":"2024-04-16T21:08:49.366364+0000","last_fullsized":"2024-04-16T21:08:49.366364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503606+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:08:34.503606+0000","last_peered":"2024-04-16T21:08:34.503606+0000","last_clean":"2024-04-16T21:08:34.503606+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:08:34.503606+0000","last_undegraded":"2024-04-16T21:08:34.503606+0000","last_fullsized":"2024-04-16T21:08:34.503606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:56:59.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'38","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.371052+0000","last_change":"2024-04-16T19:25:09.462718+0000","last_active":"2024-04-16T21:09:19.371052+0000","last_peered":"2024-04-16T21:09:19.371052+0000","last_clean":"2024-04-16T21:09:19.371052+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:09:19.371052+0000","last_undegraded":"2024-04-16T21:09:19.371052+0000","last_fullsized":"2024-04-16T21:09:19.371052+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:51.473781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649801+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:07:19.649801+0000","last_peered":"2024-04-16T21:07:19.649801+0000","last_clean":"2024-04-16T21:07:19.649801+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:07:19.649801+0000","last_undegraded":"2024-04-16T21:07:19.649801+0000","last_fullsized":"2024-04-16T21:07:19.649801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366306+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:08:54.366306+0000","last_peered":"2024-04-16T21:08:54.366306+0000","last_clean":"2024-04-16T21:08:54.366306+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:08:54.366306+0000","last_undegraded":"2024-04-16T21:08:54.366306+0000","last_fullsized":"2024-04-16T21:08:54.366306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'14","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493167+0000","last_change":"2024-04-16T19:25:09.467165+0000","last_active":"2024-04-16T21:07:29.493167+0000","last_peered":"2024-04-16T21:07:29.493167+0000","last_clean":"2024-04-16T21:07:29.493167+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:07:29.493167+0000","last_undegraded":"2024-04-16T21:07:29.493167+0000","last_fullsized":"2024-04-16T21:07:29.493167+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:19.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650655+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:07:19.650655+0000","last_peered":"2024-04-16T21:07:19.650655+0000","last_clean":"2024-04-16T21:07:19.650655+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:07:19.650655+0000","last_undegraded":"2024-04-16T21:07:19.650655+0000","last_fullsized":"2024-04-16T21:07:19.650655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657012+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:08:04.657012+0000","last_peered":"2024-04-16T21:08:04.657012+0000","last_clean":"2024-04-16T21:08:04.657012+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:08:04.657012+0000","last_undegraded":"2024-04-16T21:08:04.657012+0000","last_fullsized":"2024-04-16T21:08:04.657012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'9","reported_seq":180,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375542+0000","last_change":"2024-04-16T19:25:09.464008+0000","last_active":"2024-04-16T21:09:19.375542+0000","last_peered":"2024-04-16T21:09:19.375542+0000","last_clean":"2024-04-16T21:09:19.375542+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:09:19.375542+0000","last_undegraded":"2024-04-16T21:09:19.375542+0000","last_fullsized":"2024-04-16T21:09:19.375542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501544+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:08:24.501544+0000","last_peered":"2024-04-16T21:08:24.501544+0000","last_clean":"2024-04-16T21:08:24.501544+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:08:24.501544+0000","last_undegraded":"2024-04-16T21:08:24.501544+0000","last_fullsized":"2024-04-16T21:08:24.501544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657935+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:08:04.657935+0000","last_peered":"2024-04-16T21:08:04.657935+0000","last_clean":"2024-04-16T21:08:04.657935+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:08:04.657935+0000","last_undegraded":"2024-04-16T21:08:04.657935+0000","last_fullsized":"2024-04-16T21:08:04.657935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'22","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510724+0000","last_change":"2024-04-16T19:25:09.467151+0000","last_active":"2024-04-16T21:09:19.510724+0000","last_peered":"2024-04-16T21:09:19.510724+0000","last_clean":"2024-04-16T21:09:19.510724+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:09:19.510724+0000","last_undegraded":"2024-04-16T21:09:19.510724+0000","last_fullsized":"2024-04-16T21:09:19.510724+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:22.630747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659703+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:08:09.659703+0000","last_peered":"2024-04-16T21:08:09.659703+0000","last_clean":"2024-04-16T21:08:09.659703+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:08:09.659703+0000","last_undegraded":"2024-04-16T21:08:09.659703+0000","last_fullsized":"2024-04-16T21:08:09.659703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368947+0000","last_change":"2024-04-16T19:25:41.505958+0000","last_active":"2024-04-16T21:09:14.368947+0000","last_peered":"2024-04-16T21:09:14.368947+0000","last_clean":"2024-04-16T21:09:14.368947+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:09:14.368947+0000","last_undegraded":"2024-04-16T21:09:14.368947+0000","last_fullsized":"2024-04-16T21:09:14.368947+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:19.162890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'5","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369023+0000","last_change":"2024-04-16T19:25:09.467605+0000","last_active":"2024-04-16T21:09:09.369023+0000","last_peered":"2024-04-16T21:09:09.369023+0000","last_clean":"2024-04-16T21:09:09.369023+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:09:09.369023+0000","last_undegraded":"2024-04-16T21:09:09.369023+0000","last_fullsized":"2024-04-16T21:09:09.369023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658742+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:08:09.658742+0000","last_peered":"2024-04-16T21:08:09.658742+0000","last_clean":"2024-04-16T21:08:09.658742+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:08:09.658742+0000","last_undegraded":"2024-04-16T21:08:09.658742+0000","last_fullsized":"2024-04-16T21:08:09.658742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:24.447630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658987+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:08:04.658987+0000","last_peered":"2024-04-16T21:08:04.658987+0000","last_clean":"2024-04-16T21:08:04.658987+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:08:04.658987+0000","last_undegraded":"2024-04-16T21:08:04.658987+0000","last_fullsized":"2024-04-16T21:08:04.658987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'14","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509341+0000","last_change":"2024-04-16T19:25:09.464438+0000","last_active":"2024-04-16T21:09:09.509341+0000","last_peered":"2024-04-16T21:09:09.509341+0000","last_clean":"2024-04-16T21:09:09.509341+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:09:09.509341+0000","last_undegraded":"2024-04-16T21:09:09.509341+0000","last_fullsized":"2024-04-16T21:09:09.509341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657776+0000","last_change":"2024-04-16T19:25:41.508398+0000","last_active":"2024-04-16T21:08:09.657776+0000","last_peered":"2024-04-16T21:08:09.657776+0000","last_clean":"2024-04-16T21:08:09.657776+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:08:09.657776+0000","last_undegraded":"2024-04-16T21:08:09.657776+0000","last_fullsized":"2024-04-16T21:08:09.657776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:28:30.770132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661196+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:08:19.661196+0000","last_peered":"2024-04-16T21:08:19.661196+0000","last_clean":"2024-04-16T21:08:19.661196+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:08:19.661196+0000","last_undegraded":"2024-04-16T21:08:19.661196+0000","last_fullsized":"2024-04-16T21:08:19.661196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:37.636086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'3","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652749+0000","last_change":"2024-04-16T19:25:41.499612+0000","last_active":"2024-04-16T21:07:34.652749+0000","last_peered":"2024-04-16T21:07:34.652749+0000","last_clean":"2024-04-16T21:07:34.652749+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:07:34.652749+0000","last_undegraded":"2024-04-16T21:07:34.652749+0000","last_fullsized":"2024-04-16T21:07:34.652749+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499546+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:08:09.499546+0000","last_peered":"2024-04-16T21:08:09.499546+0000","last_clean":"2024-04-16T21:08:09.499546+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:08:09.499546+0000","last_undegraded":"2024-04-16T21:08:09.499546+0000","last_fullsized":"2024-04-16T21:08:09.499546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490904+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:07:14.490904+0000","last_peered":"2024-04-16T21:07:14.490904+0000","last_clean":"2024-04-16T21:07:14.490904+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:07:14.490904+0000","last_undegraded":"2024-04-16T21:07:14.490904+0000","last_fullsized":"2024-04-16T21:07:14.490904+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:50.557311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'19","reported_seq":286,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373955+0000","last_change":"2024-04-16T19:25:09.468675+0000","last_active":"2024-04-16T21:09:09.373955+0000","last_peered":"2024-04-16T21:09:09.373955+0000","last_clean":"2024-04-16T21:09:09.373955+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:09:09.373955+0000","last_undegraded":"2024-04-16T21:09:09.373955+0000","last_fullsized":"2024-04-16T21:09:09.373955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:21:12.600213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367898+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:08:59.367898+0000","last_peered":"2024-04-16T21:08:59.367898+0000","last_clean":"2024-04-16T21:08:59.367898+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:08:59.367898+0000","last_undegraded":"2024-04-16T21:08:59.367898+0000","last_fullsized":"2024-04-16T21:08:59.367898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649751+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:07:14.649751+0000","last_peered":"2024-04-16T21:07:14.649751+0000","last_clean":"2024-04-16T21:07:14.649751+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:07:14.649751+0000","last_undegraded":"2024-04-16T21:07:14.649751+0000","last_fullsized":"2024-04-16T21:07:14.649751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:50.981157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'12","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652830+0000","last_change":"2024-04-16T19:25:09.459052+0000","last_active":"2024-04-16T21:07:39.652830+0000","last_peered":"2024-04-16T21:07:39.652830+0000","last_clean":"2024-04-16T21:07:39.652830+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:07:39.652830+0000","last_undegraded":"2024-04-16T21:07:39.652830+0000","last_fullsized":"2024-04-16T21:07:39.652830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:51.756021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650094+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:07:14.650094+0000","last_peered":"2024-04-16T21:07:14.650094+0000","last_clean":"2024-04-16T21:07:14.650094+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:07:14.650094+0000","last_undegraded":"2024-04-16T21:07:14.650094+0000","last_fullsized":"2024-04-16T21:07:14.650094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371494+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:08:54.371494+0000","last_peered":"2024-04-16T21:08:54.371494+0000","last_clean":"2024-04-16T21:08:54.371494+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:08:54.371494+0000","last_undegraded":"2024-04-16T21:08:54.371494+0000","last_fullsized":"2024-04-16T21:08:54.371494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'3","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494798+0000","last_change":"2024-04-16T19:25:09.460170+0000","last_active":"2024-04-16T21:07:39.494798+0000","last_peered":"2024-04-16T21:07:39.494798+0000","last_clean":"2024-04-16T21:07:39.494798+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:07:39.494798+0000","last_undegraded":"2024-04-16T21:07:39.494798+0000","last_fullsized":"2024-04-16T21:07:39.494798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490719+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:07:14.490719+0000","last_peered":"2024-04-16T21:07:14.490719+0000","last_clean":"2024-04-16T21:07:14.490719+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:07:14.490719+0000","last_undegraded":"2024-04-16T21:07:14.490719+0000","last_fullsized":"2024-04-16T21:07:14.490719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:08:34.663111+0000","last_peered":"2024-04-16T21:08:34.663111+0000","last_clean":"2024-04-16T21:08:34.663111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:08:34.663111+0000","last_undegraded":"2024-04-16T21:08:34.663111+0000","last_fullsized":"2024-04-16T21:08:34.663111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4203,"ondisk_log_size":4203,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3420,"ondisk_log_size":3420,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5660,"ondisk_log_size":5660,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641553,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41132,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816468,"statfs":{"total":107374182400,"available":107332063232,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641554,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641553,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55568,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802032,"statfs":{"total":107374182400,"available":107317280768,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641554,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59228,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798372,"statfs":{"total":107374182400,"available":107313532928,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641554,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816412,"statfs":{"total":107374182400,"available":107332005888,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641553,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55800,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104801800,"statfs":{"total":107374182400,"available":107317043200,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:11:40.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-16T21:11:40.805 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:11:40.806 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":52,"fsid":"46f1113d-f19a-4ad9-a219-ddfdf9343e7d","created":"2024-04-16T19:23:36.119697+0000","modified":"2024-04-16T19:47:42.276147+0000","last_up_change":"2024-04-16T19:23:44.107341+0000","last_in_change":"2024-04-16T19:23:37.815629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T19:23:44.317491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T19:23:46.042846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-04-16T19:24:32.240286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-04-16T19:24:33.712654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-04-16T19:24:43.785835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-04-16T19:24:45.465505+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-04-16T19:24:47.429444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-04-16T19:24:49.469769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2024-04-16T19:25:07.966159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2024-04-16T19:25:09.670661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2024-04-16T19:25:11.472380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2024-04-16T19:25:13.497005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2024-04-16T19:25:39.655025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2024-04-16T19:25:48.175100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2024-04-16T19:47:32.955705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"097ebd2b-2937-4f76-8474-06350d6fcb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6802","nonce":562010544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6806","nonce":562010544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6814","nonce":562010544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":562010544},{"type":"v1","addr":"172.21.15.26:6810","nonce":562010544}]},"public_addr":"172.21.15.26:6802/562010544","cluster_addr":"172.21.15.26:6806/562010544","heartbeat_back_addr":"172.21.15.26:6814/562010544","heartbeat_front_addr":"172.21.15.26:6810/562010544","state":["exists","up"]},{"osd":1,"uuid":"e61cc21d-8baf-43dc-8b1b-b8931c3c5b4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6801","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6803","nonce":930307441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6805","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6807","nonce":930307441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6813","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6815","nonce":930307441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6809","nonce":930307441},{"type":"v1","addr":"172.21.15.26:6811","nonce":930307441}]},"public_addr":"172.21.15.26:6803/930307441","cluster_addr":"172.21.15.26:6807/930307441","heartbeat_back_addr":"172.21.15.26:6815/930307441","heartbeat_front_addr":"172.21.15.26:6811/930307441","state":["exists","up"]},{"osd":2,"uuid":"0965bd9d-6a0a-4f22-b3a5-e78191cf4526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6817","nonce":1193744923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6819","nonce":1193744923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6823","nonce":1193744923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":1193744923},{"type":"v1","addr":"172.21.15.26:6821","nonce":1193744923}]},"public_addr":"172.21.15.26:6817/1193744923","cluster_addr":"172.21.15.26:6819/1193744923","heartbeat_back_addr":"172.21.15.26:6823/1193744923","heartbeat_front_addr":"172.21.15.26:6821/1193744923","state":["exists","up"]},{"osd":3,"uuid":"485c1994-8091-434c-b395-8a6aef81561d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6806","nonce":800096592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6809","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6811","nonce":800096592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6821","nonce":800096592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":800096592},{"type":"v1","addr":"172.21.15.69:6817","nonce":800096592}]},"public_addr":"172.21.15.69:6806/800096592","cluster_addr":"172.21.15.69:6811/800096592","heartbeat_back_addr":"172.21.15.69:6821/800096592","heartbeat_front_addr":"172.21.15.69:6817/800096592","state":["exists","up"]},{"osd":4,"uuid":"0da3d944-0b26-4daa-93f4-c5b7de1d72ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6803","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6808","nonce":3493128902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6816","nonce":3493128902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6823","nonce":3493128902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3493128902},{"type":"v1","addr":"172.21.15.69:6820","nonce":3493128902}]},"public_addr":"172.21.15.69:6808/3493128902","cluster_addr":"172.21.15.69:6816/3493128902","heartbeat_back_addr":"172.21.15.69:6823/3493128902","heartbeat_front_addr":"172.21.15.69:6820/3493128902","state":["exists","up"]},{"osd":5,"uuid":"82f781fe-193b-4e5f-813b-8645acd09ab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6801","nonce":3895908262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6805","nonce":3895908262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6813","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6815","nonce":3895908262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6807","nonce":3895908262},{"type":"v1","addr":"172.21.15.69:6810","nonce":3895908262}]},"public_addr":"172.21.15.69:6801/3895908262","cluster_addr":"172.21.15.69:6805/3895908262","heartbeat_back_addr":"172.21.15.69:6815/3895908262","heartbeat_front_addr":"172.21.15.69:6810/3895908262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.125284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.152292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.113122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.161231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.162925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T19:23:42.192596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":3,"to":1}]},{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.14","mappings":[{"from":3,"to":1}]},{"pgid":"3.24","mappings":[{"from":0,"to":1}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"9.3","mappings":[{"from":3,"to":4}]},{"pgid":"9.24","mappings":[{"from":2,"to":4}]},{"pgid":"9.33","mappings":[{"from":3,"to":4}]},{"pgid":"12.1d","mappings":[{"from":0,"to":3}]},{"pgid":"14.0","mappings":[{"from":0,"to":3}]},{"pgid":"14.b","mappings":[{"from":0,"to":5}]},{"pgid":"14.1f","mappings":[{"from":0,"to":5}]},{"pgid":"15.1b","mappings":[{"from":4,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T21:11:41.819 INFO:tasks.ceph:Scrubbing c1.osd.0 2024-04-16T21:11:41.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:41.960 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:41.960 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:41.961 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:41.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 0 2024-04-16T21:11:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2024-04-16T21:11:42.287 INFO:tasks.ceph:Scrubbing c1.osd.1 2024-04-16T21:11:42.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:42.415 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:42.415 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:42.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 1 2024-04-16T21:11:42.725 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2024-04-16T21:11:42.738 INFO:tasks.ceph:Scrubbing c1.osd.2 2024-04-16T21:11:42.739 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:42.866 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:42.867 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:42.867 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:42.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 2 2024-04-16T21:11:43.177 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2024-04-16T21:11:43.190 INFO:tasks.ceph:Scrubbing c1.osd.3 2024-04-16T21:11:43.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:43.319 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:43.320 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:43.320 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:43.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 3 2024-04-16T21:11:43.632 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 3 to deep-scrub 2024-04-16T21:11:43.645 INFO:tasks.ceph:Scrubbing c1.osd.4 2024-04-16T21:11:43.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:43.775 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:43.775 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:43.775 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:43.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 4 2024-04-16T21:11:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 4 to deep-scrub 2024-04-16T21:11:44.101 INFO:tasks.ceph:Scrubbing c1.osd.5 2024-04-16T21:11:44.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-16T21:11:44.230 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-04-16T21:11:44.231 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-16T21:11:44.231 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-04-16T21:11:44.241 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd deep-scrub 5 2024-04-16T21:11:44.542 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 5 to deep-scrub 2024-04-16T21:11:44.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:11:44.900 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:11:44.918 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:11:44.921 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3285,"stamp":"2024-04-16T21:11:43.330466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30466,"ondisk_log_size":30466,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":307840,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":296630,"kb_avail":628837760,"statfs":{"total":644245094400,"available":643929866240,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":303749793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006557"},"pg_stats":[{"pgid":"9.35","version":"52'28","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367393+0000","last_change":"2024-04-16T19:25:09.466020+0000","last_active":"2024-04-16T21:09:04.367393+0000","last_peered":"2024-04-16T21:09:04.367393+0000","last_clean":"2024-04-16T21:09:04.367393+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:09:04.367393+0000","last_undegraded":"2024-04-16T21:09:04.367393+0000","last_fullsized":"2024-04-16T21:09:04.367393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365868+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:08:44.365868+0000","last_peered":"2024-04-16T21:08:44.365868+0000","last_clean":"2024-04-16T21:08:44.365868+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:08:44.365868+0000","last_undegraded":"2024-04-16T21:08:44.365868+0000","last_fullsized":"2024-04-16T21:08:44.365868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648735+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:07:09.648735+0000","last_peered":"2024-04-16T21:07:09.648735+0000","last_clean":"2024-04-16T21:07:09.648735+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:07:09.648735+0000","last_undegraded":"2024-04-16T21:07:09.648735+0000","last_fullsized":"2024-04-16T21:07:09.648735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"47'5","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368044+0000","last_change":"2024-04-16T19:25:09.467691+0000","last_active":"2024-04-16T21:09:04.368044+0000","last_peered":"2024-04-16T21:09:04.368044+0000","last_clean":"2024-04-16T21:09:04.368044+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:09:04.368044+0000","last_undegraded":"2024-04-16T21:09:04.368044+0000","last_fullsized":"2024-04-16T21:09:04.368044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364114+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:08:44.364114+0000","last_peered":"2024-04-16T21:08:44.364114+0000","last_clean":"2024-04-16T21:08:44.364114+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:08:44.364114+0000","last_undegraded":"2024-04-16T21:08:44.364114+0000","last_fullsized":"2024-04-16T21:08:44.364114+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649318+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:07:09.649318+0000","last_peered":"2024-04-16T21:07:09.649318+0000","last_clean":"2024-04-16T21:07:09.649318+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:07:09.649318+0000","last_undegraded":"2024-04-16T21:07:09.649318+0000","last_fullsized":"2024-04-16T21:07:09.649318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'13","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.646812+0000","last_change":"2024-04-16T19:25:09.457429+0000","last_active":"2024-04-16T21:06:54.646812+0000","last_peered":"2024-04-16T21:06:54.646812+0000","last_clean":"2024-04-16T21:06:54.646812+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:06:54.646812+0000","last_undegraded":"2024-04-16T21:06:54.646812+0000","last_fullsized":"2024-04-16T21:06:54.646812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369756+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:08:44.369756+0000","last_peered":"2024-04-16T21:08:44.369756+0000","last_clean":"2024-04-16T21:08:44.369756+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:08:44.369756+0000","last_undegraded":"2024-04-16T21:08:44.369756+0000","last_fullsized":"2024-04-16T21:08:44.369756+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:11.722618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649006+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:07:09.649006+0000","last_peered":"2024-04-16T21:07:09.649006+0000","last_clean":"2024-04-16T21:07:09.649006+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:07:09.649006+0000","last_undegraded":"2024-04-16T21:07:09.649006+0000","last_fullsized":"2024-04-16T21:07:09.649006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'13","reported_seq":234,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367421+0000","last_change":"2024-04-16T19:25:09.467050+0000","last_active":"2024-04-16T21:09:04.367421+0000","last_peered":"2024-04-16T21:09:04.367421+0000","last_clean":"2024-04-16T21:09:04.367421+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:09:04.367421+0000","last_undegraded":"2024-04-16T21:09:04.367421+0000","last_fullsized":"2024-04-16T21:09:04.367421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366441+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:08:54.366441+0000","last_peered":"2024-04-16T21:08:54.366441+0000","last_clean":"2024-04-16T21:08:54.366441+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:08:54.366441+0000","last_undegraded":"2024-04-16T21:08:54.366441+0000","last_fullsized":"2024-04-16T21:08:54.366441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648038+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:07:09.648038+0000","last_peered":"2024-04-16T21:07:09.648038+0000","last_clean":"2024-04-16T21:07:09.648038+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:07:09.648038+0000","last_undegraded":"2024-04-16T21:07:09.648038+0000","last_fullsized":"2024-04-16T21:07:09.648038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:20.418355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'6","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655139+0000","last_change":"2024-04-16T19:25:09.459828+0000","last_active":"2024-04-16T21:07:49.655139+0000","last_peered":"2024-04-16T21:07:49.655139+0000","last_clean":"2024-04-16T21:07:49.655139+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:07:49.655139+0000","last_undegraded":"2024-04-16T21:07:49.655139+0000","last_fullsized":"2024-04-16T21:07:49.655139+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648499+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:07:04.648499+0000","last_peered":"2024-04-16T21:07:04.648499+0000","last_clean":"2024-04-16T21:07:04.648499+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:07:04.648499+0000","last_undegraded":"2024-04-16T21:07:04.648499+0000","last_fullsized":"2024-04-16T21:07:04.648499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:14.797987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367192+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:08:59.367192+0000","last_peered":"2024-04-16T21:08:59.367192+0000","last_clean":"2024-04-16T21:08:59.367192+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:08:59.367192+0000","last_undegraded":"2024-04-16T21:08:59.367192+0000","last_fullsized":"2024-04-16T21:08:59.367192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'18","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654366+0000","last_change":"2024-04-16T19:25:09.461822+0000","last_active":"2024-04-16T21:07:49.654366+0000","last_peered":"2024-04-16T21:07:49.654366+0000","last_clean":"2024-04-16T21:07:49.654366+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:07:49.654366+0000","last_undegraded":"2024-04-16T21:07:49.654366+0000","last_fullsized":"2024-04-16T21:07:49.654366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647976+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:07:04.647976+0000","last_peered":"2024-04-16T21:07:04.647976+0000","last_clean":"2024-04-16T21:07:04.647976+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:07:04.647976+0000","last_undegraded":"2024-04-16T21:07:04.647976+0000","last_fullsized":"2024-04-16T21:07:04.647976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:07.917853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365518+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:08:49.365518+0000","last_peered":"2024-04-16T21:08:49.365518+0000","last_clean":"2024-04-16T21:08:49.365518+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:08:49.365518+0000","last_undegraded":"2024-04-16T21:08:49.365518+0000","last_fullsized":"2024-04-16T21:08:49.365518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'10","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649782+0000","last_change":"2024-04-16T19:25:41.508581+0000","last_active":"2024-04-16T21:07:14.649782+0000","last_peered":"2024-04-16T21:07:14.649782+0000","last_clean":"2024-04-16T21:07:14.649782+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:07:14.649782+0000","last_undegraded":"2024-04-16T21:07:14.649782+0000","last_fullsized":"2024-04-16T21:07:14.649782+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:12.396788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648409+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:07:04.648409+0000","last_peered":"2024-04-16T21:07:04.648409+0000","last_clean":"2024-04-16T21:07:04.648409+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:07:04.648409+0000","last_undegraded":"2024-04-16T21:07:04.648409+0000","last_fullsized":"2024-04-16T21:07:04.648409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:34.027154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365507+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:08:49.365507+0000","last_peered":"2024-04-16T21:08:49.365507+0000","last_clean":"2024-04-16T21:08:49.365507+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:08:49.365507+0000","last_undegraded":"2024-04-16T21:08:49.365507+0000","last_fullsized":"2024-04-16T21:08:49.365507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'15","reported_seq":238,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657926+0000","last_change":"2024-04-16T19:25:09.463558+0000","last_active":"2024-04-16T21:08:04.657926+0000","last_peered":"2024-04-16T21:08:04.657926+0000","last_clean":"2024-04-16T21:08:04.657926+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:08:04.657926+0000","last_undegraded":"2024-04-16T21:08:04.657926+0000","last_fullsized":"2024-04-16T21:08:04.657926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.489018+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:07:04.489018+0000","last_peered":"2024-04-16T21:07:04.489018+0000","last_clean":"2024-04-16T21:07:04.489018+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:07:04.489018+0000","last_undegraded":"2024-04-16T21:07:04.489018+0000","last_fullsized":"2024-04-16T21:07:04.489018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506158+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:08:49.506158+0000","last_peered":"2024-04-16T21:08:49.506158+0000","last_clean":"2024-04-16T21:08:49.506158+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:08:49.506158+0000","last_undegraded":"2024-04-16T21:08:49.506158+0000","last_fullsized":"2024-04-16T21:08:49.506158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'7","reported_seq":221,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370754+0000","last_change":"2024-04-16T19:25:09.467094+0000","last_active":"2024-04-16T21:09:19.370754+0000","last_peered":"2024-04-16T21:09:19.370754+0000","last_clean":"2024-04-16T21:09:19.370754+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:09:19.370754+0000","last_undegraded":"2024-04-16T21:09:19.370754+0000","last_fullsized":"2024-04-16T21:09:19.370754+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:52.158151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650393+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:07:19.650393+0000","last_peered":"2024-04-16T21:07:19.650393+0000","last_clean":"2024-04-16T21:07:19.650393+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:07:19.650393+0000","last_undegraded":"2024-04-16T21:07:19.650393+0000","last_fullsized":"2024-04-16T21:07:19.650393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366531+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:08:49.366531+0000","last_peered":"2024-04-16T21:08:49.366531+0000","last_clean":"2024-04-16T21:08:49.366531+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:08:49.366531+0000","last_undegraded":"2024-04-16T21:08:49.366531+0000","last_fullsized":"2024-04-16T21:08:49.366531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'6","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372746+0000","last_change":"2024-04-16T19:25:09.459483+0000","last_active":"2024-04-16T21:09:29.372746+0000","last_peered":"2024-04-16T21:09:29.372746+0000","last_clean":"2024-04-16T21:09:29.372746+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:09:29.372746+0000","last_undegraded":"2024-04-16T21:09:29.372746+0000","last_fullsized":"2024-04-16T21:09:29.372746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:31.933772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650394+0000","last_change":"2024-04-16T19:25:41.503681+0000","last_active":"2024-04-16T21:07:19.650394+0000","last_peered":"2024-04-16T21:07:19.650394+0000","last_clean":"2024-04-16T21:07:19.650394+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:07:19.650394+0000","last_undegraded":"2024-04-16T21:07:19.650394+0000","last_fullsized":"2024-04-16T21:07:19.650394+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:33:58.945193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662645+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:08:29.662645+0000","last_peered":"2024-04-16T21:08:29.662645+0000","last_clean":"2024-04-16T21:08:29.662645+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:08:29.662645+0000","last_undegraded":"2024-04-16T21:08:29.662645+0000","last_fullsized":"2024-04-16T21:08:29.662645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'5","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510833+0000","last_change":"2024-04-16T19:25:09.464787+0000","last_active":"2024-04-16T21:09:19.510833+0000","last_peered":"2024-04-16T21:09:19.510833+0000","last_clean":"2024-04-16T21:09:19.510833+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:09:19.510833+0000","last_undegraded":"2024-04-16T21:09:19.510833+0000","last_fullsized":"2024-04-16T21:09:19.510833+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:27.821789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652786+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:07:34.652786+0000","last_peered":"2024-04-16T21:07:34.652786+0000","last_clean":"2024-04-16T21:07:34.652786+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:07:34.652786+0000","last_undegraded":"2024-04-16T21:07:34.652786+0000","last_fullsized":"2024-04-16T21:07:34.652786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662148+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:08:29.662148+0000","last_peered":"2024-04-16T21:08:29.662148+0000","last_clean":"2024-04-16T21:08:29.662148+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:08:29.662148+0000","last_undegraded":"2024-04-16T21:08:29.662148+0000","last_fullsized":"2024-04-16T21:08:29.662148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'10","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510806+0000","last_change":"2024-04-16T19:25:09.463733+0000","last_active":"2024-04-16T21:09:19.510806+0000","last_peered":"2024-04-16T21:09:19.510806+0000","last_clean":"2024-04-16T21:09:19.510806+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:09:19.510806+0000","last_undegraded":"2024-04-16T21:09:19.510806+0000","last_fullsized":"2024-04-16T21:09:19.510806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:40:59.948853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503495+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:08:34.503495+0000","last_peered":"2024-04-16T21:08:34.503495+0000","last_clean":"2024-04-16T21:08:34.503495+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:08:34.503495+0000","last_undegraded":"2024-04-16T21:08:34.503495+0000","last_fullsized":"2024-04-16T21:08:34.503495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:23.100412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661720+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:08:29.661720+0000","last_peered":"2024-04-16T21:08:29.661720+0000","last_clean":"2024-04-16T21:08:29.661720+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:08:29.661720+0000","last_undegraded":"2024-04-16T21:08:29.661720+0000","last_fullsized":"2024-04-16T21:08:29.661720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:09:50.552696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'9","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654268+0000","last_change":"2024-04-16T19:25:09.462286+0000","last_active":"2024-04-16T21:07:39.654268+0000","last_peered":"2024-04-16T21:07:39.654268+0000","last_clean":"2024-04-16T21:07:39.654268+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:07:39.654268+0000","last_undegraded":"2024-04-16T21:07:39.654268+0000","last_fullsized":"2024-04-16T21:07:39.654268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503526+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:08:34.503526+0000","last_peered":"2024-04-16T21:08:34.503526+0000","last_clean":"2024-04-16T21:08:34.503526+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:08:34.503526+0000","last_undegraded":"2024-04-16T21:08:34.503526+0000","last_fullsized":"2024-04-16T21:08:34.503526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:21.972819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661148+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:08:29.661148+0000","last_peered":"2024-04-16T21:08:29.661148+0000","last_clean":"2024-04-16T21:08:29.661148+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:08:29.661148+0000","last_undegraded":"2024-04-16T21:08:29.661148+0000","last_fullsized":"2024-04-16T21:08:29.661148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:27.005905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'15","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494683+0000","last_change":"2024-04-16T19:25:09.461082+0000","last_active":"2024-04-16T21:07:39.494683+0000","last_peered":"2024-04-16T21:07:39.494683+0000","last_clean":"2024-04-16T21:07:39.494683+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:07:39.494683+0000","last_undegraded":"2024-04-16T21:07:39.494683+0000","last_fullsized":"2024-04-16T21:07:39.494683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662702+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:08:34.662702+0000","last_peered":"2024-04-16T21:08:34.662702+0000","last_clean":"2024-04-16T21:08:34.662702+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:08:34.662702+0000","last_undegraded":"2024-04-16T21:08:34.662702+0000","last_fullsized":"2024-04-16T21:08:34.662702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651169+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:07:29.651169+0000","last_peered":"2024-04-16T21:07:29.651169+0000","last_clean":"2024-04-16T21:07:29.651169+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:07:29.651169+0000","last_undegraded":"2024-04-16T21:07:29.651169+0000","last_fullsized":"2024-04-16T21:07:29.651169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365000+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:08:49.365000+0000","last_peered":"2024-04-16T21:08:49.365000+0000","last_clean":"2024-04-16T21:08:49.365000+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:08:49.365000+0000","last_undegraded":"2024-04-16T21:08:49.365000+0000","last_fullsized":"2024-04-16T21:08:49.365000+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:32.817843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1720,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659053+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:08:04.659053+0000","last_peered":"2024-04-16T21:08:04.659053+0000","last_clean":"2024-04-16T21:08:04.659053+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:08:04.659053+0000","last_undegraded":"2024-04-16T21:08:04.659053+0000","last_fullsized":"2024-04-16T21:08:04.659053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660478+0000","last_change":"2024-04-16T19:25:41.505822+0000","last_active":"2024-04-16T21:08:14.660478+0000","last_peered":"2024-04-16T21:08:14.660478+0000","last_clean":"2024-04-16T21:08:14.660478+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:08:14.660478+0000","last_undegraded":"2024-04-16T21:08:14.660478+0000","last_fullsized":"2024-04-16T21:08:14.660478+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'6","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654478+0000","last_change":"2024-04-16T19:25:15.486584+0000","last_active":"2024-04-16T21:07:49.654478+0000","last_peered":"2024-04-16T21:07:49.654478+0000","last_clean":"2024-04-16T21:07:49.654478+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:07:49.654478+0000","last_undegraded":"2024-04-16T21:07:49.654478+0000","last_fullsized":"2024-04-16T21:07:49.654478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654387+0000","last_change":"2024-04-16T19:25:13.469257+0000","last_active":"2024-04-16T21:07:44.654387+0000","last_peered":"2024-04-16T21:07:44.654387+0000","last_clean":"2024-04-16T21:07:44.654387+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:07:44.654387+0000","last_undegraded":"2024-04-16T21:07:44.654387+0000","last_fullsized":"2024-04-16T21:07:44.654387+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'203","reported_seq":447,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655143+0000","last_change":"2024-04-16T19:25:11.464182+0000","last_active":"2024-04-16T21:07:44.655143+0000","last_peered":"2024-04-16T21:07:44.655143+0000","last_clean":"2024-04-16T21:07:44.655143+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:07:44.655143+0000","last_undegraded":"2024-04-16T21:07:44.655143+0000","last_fullsized":"2024-04-16T21:07:44.655143+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:07.536101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'19","reported_seq":284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:54.487393+0000","last_change":"2024-04-16T19:25:09.464946+0000","last_active":"2024-04-16T21:06:54.487393+0000","last_peered":"2024-04-16T21:06:54.487393+0000","last_clean":"2024-04-16T21:06:54.487393+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:06:54.487393+0000","last_undegraded":"2024-04-16T21:06:54.487393+0000","last_fullsized":"2024-04-16T21:06:54.487393+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:45:45.734532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365360+0000","last_change":"2024-04-16T19:24:51.436483+0000","last_active":"2024-04-16T21:08:49.365360+0000","last_peered":"2024-04-16T21:08:49.365360+0000","last_clean":"2024-04-16T21:08:49.365360+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:08:49.365360+0000","last_undegraded":"2024-04-16T21:08:49.365360+0000","last_fullsized":"2024-04-16T21:08:49.365360+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:40.603125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494096+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:07:34.494096+0000","last_peered":"2024-04-16T21:07:34.494096+0000","last_clean":"2024-04-16T21:07:34.494096+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:07:34.494096+0000","last_undegraded":"2024-04-16T21:07:34.494096+0000","last_fullsized":"2024-04-16T21:07:34.494096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:52.988517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650257+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:07:14.650257+0000","last_peered":"2024-04-16T21:07:14.650257+0000","last_clean":"2024-04-16T21:07:14.650257+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:07:14.650257+0000","last_undegraded":"2024-04-16T21:07:14.650257+0000","last_fullsized":"2024-04-16T21:07:14.650257+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:07.194576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:42:58.780798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369810+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:08:44.369810+0000","last_peered":"2024-04-16T21:08:44.369810+0000","last_clean":"2024-04-16T21:08:44.369810+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:08:44.369810+0000","last_undegraded":"2024-04-16T21:08:44.369810+0000","last_fullsized":"2024-04-16T21:08:44.369810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:08:44.364641+0000","last_peered":"2024-04-16T21:08:44.364641+0000","last_clean":"2024-04-16T21:08:44.364641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:08:44.364641+0000","last_undegraded":"2024-04-16T21:08:44.364641+0000","last_fullsized":"2024-04-16T21:08:44.364641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:42:28.507077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496581+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:07:49.496581+0000","last_peered":"2024-04-16T21:07:49.496581+0000","last_clean":"2024-04-16T21:07:49.496581+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:07:49.496581+0000","last_undegraded":"2024-04-16T21:07:49.496581+0000","last_fullsized":"2024-04-16T21:07:49.496581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:18:43.119570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1836,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365530+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:08:49.365530+0000","last_peered":"2024-04-16T21:08:49.365530+0000","last_clean":"2024-04-16T21:08:49.365530+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:08:49.365530+0000","last_undegraded":"2024-04-16T21:08:49.365530+0000","last_fullsized":"2024-04-16T21:08:49.365530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659544+0000","last_change":"2024-04-16T19:25:41.503686+0000","last_active":"2024-04-16T21:08:14.659544+0000","last_peered":"2024-04-16T21:08:14.659544+0000","last_clean":"2024-04-16T21:08:14.659544+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:08:14.659544+0000","last_undegraded":"2024-04-16T21:08:14.659544+0000","last_fullsized":"2024-04-16T21:08:14.659544+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:30:44.336557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'5","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660521+0000","last_change":"2024-04-16T19:25:41.505903+0000","last_active":"2024-04-16T21:08:14.660521+0000","last_peered":"2024-04-16T21:08:14.660521+0000","last_clean":"2024-04-16T21:08:14.660521+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:08:14.660521+0000","last_undegraded":"2024-04-16T21:08:14.660521+0000","last_fullsized":"2024-04-16T21:08:14.660521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:37.409520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654445+0000","last_change":"2024-04-16T19:25:13.471998+0000","last_active":"2024-04-16T21:07:44.654445+0000","last_peered":"2024-04-16T21:07:44.654445+0000","last_clean":"2024-04-16T21:07:44.654445+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:07:44.654445+0000","last_undegraded":"2024-04-16T21:07:44.654445+0000","last_fullsized":"2024-04-16T21:07:44.654445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:15.038784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'241","reported_seq":976,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495593+0000","last_change":"2024-04-16T19:25:11.465710+0000","last_active":"2024-04-16T21:07:44.495593+0000","last_peered":"2024-04-16T21:07:44.495593+0000","last_clean":"2024-04-16T21:07:44.495593+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:07:44.495593+0000","last_undegraded":"2024-04-16T21:07:44.495593+0000","last_fullsized":"2024-04-16T21:07:44.495593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:25.422152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'5","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367870+0000","last_change":"2024-04-16T19:25:09.467837+0000","last_active":"2024-04-16T21:09:04.367870+0000","last_peered":"2024-04-16T21:09:04.367870+0000","last_clean":"2024-04-16T21:09:04.367870+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:09:04.367870+0000","last_undegraded":"2024-04-16T21:09:04.367870+0000","last_fullsized":"2024-04-16T21:09:04.367870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368878+0000","last_change":"2024-04-16T19:24:51.438136+0000","last_active":"2024-04-16T21:09:09.368878+0000","last_peered":"2024-04-16T21:09:09.368878+0000","last_clean":"2024-04-16T21:09:09.368878+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:09:09.368878+0000","last_undegraded":"2024-04-16T21:09:09.368878+0000","last_fullsized":"2024-04-16T21:09:09.368878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663065+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:08:34.663065+0000","last_peered":"2024-04-16T21:08:34.663065+0000","last_clean":"2024-04-16T21:08:34.663065+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:08:34.663065+0000","last_undegraded":"2024-04-16T21:08:34.663065+0000","last_fullsized":"2024-04-16T21:08:34.663065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:39.643446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648771+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:07:14.648771+0000","last_peered":"2024-04-16T21:07:14.648771+0000","last_clean":"2024-04-16T21:07:14.648771+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:07:14.648771+0000","last_undegraded":"2024-04-16T21:07:14.648771+0000","last_fullsized":"2024-04-16T21:07:14.648771+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:19.314716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653671+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:07:44.653671+0000","last_peered":"2024-04-16T21:07:44.653671+0000","last_clean":"2024-04-16T21:07:44.653671+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:07:44.653671+0000","last_undegraded":"2024-04-16T21:07:44.653671+0000","last_fullsized":"2024-04-16T21:07:44.653671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:26.398044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364812+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:08:44.364812+0000","last_peered":"2024-04-16T21:08:44.364812+0000","last_clean":"2024-04-16T21:08:44.364812+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:08:44.364812+0000","last_undegraded":"2024-04-16T21:08:44.364812+0000","last_fullsized":"2024-04-16T21:08:44.364812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:34.874297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364788+0000","last_change":"2024-04-16T19:24:49.426639+0000","last_active":"2024-04-16T21:08:44.364788+0000","last_peered":"2024-04-16T21:08:44.364788+0000","last_clean":"2024-04-16T21:08:44.364788+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:08:44.364788+0000","last_undegraded":"2024-04-16T21:08:44.364788+0000","last_fullsized":"2024-04-16T21:08:44.364788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:38.912929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652789+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:07:34.652789+0000","last_peered":"2024-04-16T21:07:34.652789+0000","last_clean":"2024-04-16T21:07:34.652789+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:07:34.652789+0000","last_undegraded":"2024-04-16T21:07:34.652789+0000","last_fullsized":"2024-04-16T21:07:34.652789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.732507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1871,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659473+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:08:19.659473+0000","last_peered":"2024-04-16T21:08:19.659473+0000","last_clean":"2024-04-16T21:08:19.659473+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:08:19.659473+0000","last_undegraded":"2024-04-16T21:08:19.659473+0000","last_fullsized":"2024-04-16T21:08:19.659473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:55:55.150921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655316+0000","last_change":"2024-04-16T19:25:41.502400+0000","last_active":"2024-04-16T21:07:49.655316+0000","last_peered":"2024-04-16T21:07:49.655316+0000","last_clean":"2024-04-16T21:07:49.655316+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:07:49.655316+0000","last_undegraded":"2024-04-16T21:07:49.655316+0000","last_fullsized":"2024-04-16T21:07:49.655316+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:02.633032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'17","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373866+0000","last_change":"2024-04-16T19:25:15.480449+0000","last_active":"2024-04-16T21:09:39.373866+0000","last_peered":"2024-04-16T21:09:39.373866+0000","last_clean":"2024-04-16T21:09:39.373866+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:09:39.373866+0000","last_undegraded":"2024-04-16T21:09:39.373866+0000","last_fullsized":"2024-04-16T21:09:39.373866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.646356+0000","last_change":"2024-04-16T19:25:13.475293+0000","last_active":"2024-04-16T21:06:59.646356+0000","last_peered":"2024-04-16T21:06:59.646356+0000","last_clean":"2024-04-16T21:06:59.646356+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:06:59.646356+0000","last_undegraded":"2024-04-16T21:06:59.646356+0000","last_fullsized":"2024-04-16T21:06:59.646356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'696","reported_seq":41909,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509197+0000","last_change":"2024-04-16T19:25:11.465229+0000","last_active":"2024-04-16T21:09:09.509197+0000","last_peered":"2024-04-16T21:09:09.509197+0000","last_clean":"2024-04-16T21:09:09.509197+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:09:09.509197+0000","last_undegraded":"2024-04-16T21:09:09.509197+0000","last_fullsized":"2024-04-16T21:09:09.509197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:43.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'25","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652851+0000","last_change":"2024-04-16T19:25:09.464644+0000","last_active":"2024-04-16T21:07:39.652851+0000","last_peered":"2024-04-16T21:07:39.652851+0000","last_clean":"2024-04-16T21:07:39.652851+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:07:39.652851+0000","last_undegraded":"2024-04-16T21:07:39.652851+0000","last_fullsized":"2024-04-16T21:07:39.652851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:46:24.560604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650411+0000","last_change":"2024-04-16T19:24:51.438116+0000","last_active":"2024-04-16T21:07:24.650411+0000","last_peered":"2024-04-16T21:07:24.650411+0000","last_clean":"2024-04-16T21:07:24.650411+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:07:24.650411+0000","last_undegraded":"2024-04-16T21:07:24.650411+0000","last_fullsized":"2024-04-16T21:07:24.650411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662083+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:08:34.662083+0000","last_peered":"2024-04-16T21:08:34.662083+0000","last_clean":"2024-04-16T21:08:34.662083+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:08:34.662083+0000","last_undegraded":"2024-04-16T21:08:34.662083+0000","last_fullsized":"2024-04-16T21:08:34.662083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650147+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:07:14.650147+0000","last_peered":"2024-04-16T21:07:14.650147+0000","last_clean":"2024-04-16T21:07:14.650147+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:07:14.650147+0000","last_undegraded":"2024-04-16T21:07:14.650147+0000","last_fullsized":"2024-04-16T21:07:14.650147+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:05.587315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505369+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:08:44.505369+0000","last_peered":"2024-04-16T21:08:44.505369+0000","last_clean":"2024-04-16T21:08:44.505369+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:08:44.505369+0000","last_undegraded":"2024-04-16T21:08:44.505369+0000","last_fullsized":"2024-04-16T21:08:44.505369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:53.769511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":274,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495624+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:07:44.495624+0000","last_peered":"2024-04-16T21:07:44.495624+0000","last_clean":"2024-04-16T21:07:44.495624+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:07:44.495624+0000","last_undegraded":"2024-04-16T21:07:44.495624+0000","last_fullsized":"2024-04-16T21:07:44.495624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:44.851136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'12","reported_seq":1308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365800+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:08:44.365800+0000","last_peered":"2024-04-16T21:08:44.365800+0000","last_clean":"2024-04-16T21:08:44.365800+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:08:44.365800+0000","last_undegraded":"2024-04-16T21:08:44.365800+0000","last_fullsized":"2024-04-16T21:08:44.365800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:40.173753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":72,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494076+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:07:34.494076+0000","last_peered":"2024-04-16T21:07:34.494076+0000","last_clean":"2024-04-16T21:07:34.494076+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:07:34.494076+0000","last_undegraded":"2024-04-16T21:07:34.494076+0000","last_fullsized":"2024-04-16T21:07:34.494076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:59:53.541439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1472,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659510+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:08:19.659510+0000","last_peered":"2024-04-16T21:08:19.659510+0000","last_clean":"2024-04-16T21:08:19.659510+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:08:19.659510+0000","last_undegraded":"2024-04-16T21:08:19.659510+0000","last_fullsized":"2024-04-16T21:08:19.659510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:30.342275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657195+0000","last_change":"2024-04-16T19:25:41.501744+0000","last_active":"2024-04-16T21:07:59.657195+0000","last_peered":"2024-04-16T21:07:59.657195+0000","last_clean":"2024-04-16T21:07:59.657195+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:07:59.657195+0000","last_undegraded":"2024-04-16T21:07:59.657195+0000","last_fullsized":"2024-04-16T21:07:59.657195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:33.802040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'9","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372841+0000","last_change":"2024-04-16T19:25:15.487126+0000","last_active":"2024-04-16T21:09:29.372841+0000","last_peered":"2024-04-16T21:09:29.372841+0000","last_clean":"2024-04-16T21:09:29.372841+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:09:29.372841+0000","last_undegraded":"2024-04-16T21:09:29.372841+0000","last_fullsized":"2024-04-16T21:09:29.372841+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:11:17.010374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509179+0000","last_change":"2024-04-16T19:25:13.474735+0000","last_active":"2024-04-16T21:09:09.509179+0000","last_peered":"2024-04-16T21:09:09.509179+0000","last_clean":"2024-04-16T21:09:09.509179+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:09:09.509179+0000","last_undegraded":"2024-04-16T21:09:09.509179+0000","last_fullsized":"2024-04-16T21:09:09.509179+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'339","reported_seq":652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:06:59.488172+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:06:59.488172+0000","last_peered":"2024-04-16T21:06:59.488172+0000","last_clean":"2024-04-16T21:06:59.488172+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:06:59.488172+0000","last_undegraded":"2024-04-16T21:06:59.488172+0000","last_fullsized":"2024-04-16T21:06:59.488172+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:21.744422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'10","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494764+0000","last_change":"2024-04-16T19:25:09.468124+0000","last_active":"2024-04-16T21:07:39.494764+0000","last_peered":"2024-04-16T21:07:39.494764+0000","last_clean":"2024-04-16T21:07:39.494764+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:07:39.494764+0000","last_undegraded":"2024-04-16T21:07:39.494764+0000","last_fullsized":"2024-04-16T21:07:39.494764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:31.945014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651298+0000","last_change":"2024-04-16T19:24:51.572963+0000","last_active":"2024-04-16T21:07:24.651298+0000","last_peered":"2024-04-16T21:07:24.651298+0000","last_clean":"2024-04-16T21:07:24.651298+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:07:24.651298+0000","last_undegraded":"2024-04-16T21:07:24.651298+0000","last_fullsized":"2024-04-16T21:07:24.651298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:46.905067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662114+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:08:34.662114+0000","last_peered":"2024-04-16T21:08:34.662114+0000","last_clean":"2024-04-16T21:08:34.662114+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:08:34.662114+0000","last_undegraded":"2024-04-16T21:08:34.662114+0000","last_fullsized":"2024-04-16T21:08:34.662114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649505+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:07:14.649505+0000","last_peered":"2024-04-16T21:07:14.649505+0000","last_clean":"2024-04-16T21:07:14.649505+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:07:14.649505+0000","last_undegraded":"2024-04-16T21:07:14.649505+0000","last_fullsized":"2024-04-16T21:07:14.649505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365827+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:08:44.365827+0000","last_peered":"2024-04-16T21:08:44.365827+0000","last_clean":"2024-04-16T21:08:44.365827+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:08:44.365827+0000","last_undegraded":"2024-04-16T21:08:44.365827+0000","last_fullsized":"2024-04-16T21:08:44.365827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:41.163065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365785+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:08:44.365785+0000","last_peered":"2024-04-16T21:08:44.365785+0000","last_clean":"2024-04-16T21:08:44.365785+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:08:44.365785+0000","last_undegraded":"2024-04-16T21:08:44.365785+0000","last_fullsized":"2024-04-16T21:08:44.365785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:19.370822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495596+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:07:44.495596+0000","last_peered":"2024-04-16T21:07:44.495596+0000","last_clean":"2024-04-16T21:07:44.495596+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:07:44.495596+0000","last_undegraded":"2024-04-16T21:07:44.495596+0000","last_fullsized":"2024-04-16T21:07:44.495596+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:27.385349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365439+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:08:49.365439+0000","last_peered":"2024-04-16T21:08:49.365439+0000","last_clean":"2024-04-16T21:08:49.365439+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:08:49.365439+0000","last_undegraded":"2024-04-16T21:08:49.365439+0000","last_fullsized":"2024-04-16T21:08:49.365439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:12:02.296640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2006,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656784+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:07:54.656784+0000","last_peered":"2024-04-16T21:07:54.656784+0000","last_clean":"2024-04-16T21:07:54.656784+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:07:54.656784+0000","last_undegraded":"2024-04-16T21:07:54.656784+0000","last_fullsized":"2024-04-16T21:07:54.656784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:10:23.783937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655892+0000","last_change":"2024-04-16T19:25:41.506037+0000","last_active":"2024-04-16T21:07:49.655892+0000","last_peered":"2024-04-16T21:07:49.655892+0000","last_clean":"2024-04-16T21:07:49.655892+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:07:49.655892+0000","last_undegraded":"2024-04-16T21:07:49.655892+0000","last_fullsized":"2024-04-16T21:07:49.655892+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'33","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373669+0000","last_change":"2024-04-16T19:25:15.482185+0000","last_active":"2024-04-16T21:09:39.373669+0000","last_peered":"2024-04-16T21:09:39.373669+0000","last_clean":"2024-04-16T21:09:39.373669+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:09:39.373669+0000","last_undegraded":"2024-04-16T21:09:39.373669+0000","last_fullsized":"2024-04-16T21:09:39.373669+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:07.940015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370057+0000","last_change":"2024-04-16T19:25:13.475047+0000","last_active":"2024-04-16T21:09:14.370057+0000","last_peered":"2024-04-16T21:09:14.370057+0000","last_clean":"2024-04-16T21:09:14.370057+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:09:14.370057+0000","last_undegraded":"2024-04-16T21:09:14.370057+0000","last_fullsized":"2024-04-16T21:09:14.370057+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:29:34.601553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1740","reported_seq":4601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370026+0000","last_change":"2024-04-16T19:25:11.468809+0000","last_active":"2024-04-16T21:09:14.370026+0000","last_peered":"2024-04-16T21:09:14.370026+0000","last_clean":"2024-04-16T21:09:14.370026+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:09:14.370026+0000","last_undegraded":"2024-04-16T21:09:14.370026+0000","last_fullsized":"2024-04-16T21:09:14.370026+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1740,"log_dups_size":0,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:53.762063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'19","reported_seq":270,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651251+0000","last_change":"2024-04-16T19:25:09.468894+0000","last_active":"2024-04-16T21:07:29.651251+0000","last_peered":"2024-04-16T21:07:29.651251+0000","last_clean":"2024-04-16T21:07:29.651251+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:07:29.651251+0000","last_undegraded":"2024-04-16T21:07:29.651251+0000","last_fullsized":"2024-04-16T21:07:29.651251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:54:05.430878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664164+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:08:39.664164+0000","last_peered":"2024-04-16T21:08:39.664164+0000","last_clean":"2024-04-16T21:08:39.664164+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:08:39.664164+0000","last_undegraded":"2024-04-16T21:08:39.664164+0000","last_fullsized":"2024-04-16T21:08:39.664164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660546+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:08:19.660546+0000","last_peered":"2024-04-16T21:08:19.660546+0000","last_clean":"2024-04-16T21:08:19.660546+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:08:19.660546+0000","last_undegraded":"2024-04-16T21:08:19.660546+0000","last_fullsized":"2024-04-16T21:08:19.660546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365365+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:08:49.365365+0000","last_peered":"2024-04-16T21:08:49.365365+0000","last_clean":"2024-04-16T21:08:49.365365+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:08:49.365365+0000","last_undegraded":"2024-04-16T21:08:49.365365+0000","last_fullsized":"2024-04-16T21:08:49.365365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:48.947818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":394,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368921+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:09:09.368921+0000","last_peered":"2024-04-16T21:09:09.368921+0000","last_clean":"2024-04-16T21:09:09.368921+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:09:09.368921+0000","last_undegraded":"2024-04-16T21:09:09.368921+0000","last_fullsized":"2024-04-16T21:09:09.368921+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:36.588167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501183+0000","last_change":"2024-04-16T19:24:49.434497+0000","last_active":"2024-04-16T21:08:19.501183+0000","last_peered":"2024-04-16T21:08:19.501183+0000","last_clean":"2024-04-16T21:08:19.501183+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:08:19.501183+0000","last_undegraded":"2024-04-16T21:08:19.501183+0000","last_fullsized":"2024-04-16T21:08:19.501183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:08.864273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368025+0000","last_change":"2024-04-16T19:24:51.437122+0000","last_active":"2024-04-16T21:09:04.368025+0000","last_peered":"2024-04-16T21:09:04.368025+0000","last_clean":"2024-04-16T21:09:04.368025+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:09:04.368025+0000","last_undegraded":"2024-04-16T21:09:04.368025+0000","last_fullsized":"2024-04-16T21:09:04.368025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:10.861306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506044+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:08:49.506044+0000","last_peered":"2024-04-16T21:08:49.506044+0000","last_clean":"2024-04-16T21:08:49.506044+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:08:49.506044+0000","last_undegraded":"2024-04-16T21:08:49.506044+0000","last_fullsized":"2024-04-16T21:08:49.506044+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:48:31.872270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1928,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655341+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:07:54.655341+0000","last_peered":"2024-04-16T21:07:54.655341+0000","last_clean":"2024-04-16T21:07:54.655341+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:07:54.655341+0000","last_undegraded":"2024-04-16T21:07:54.655341+0000","last_fullsized":"2024-04-16T21:07:54.655341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657746+0000","last_change":"2024-04-16T19:25:41.502674+0000","last_active":"2024-04-16T21:07:59.657746+0000","last_peered":"2024-04-16T21:07:59.657746+0000","last_clean":"2024-04-16T21:07:59.657746+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:07:59.657746+0000","last_undegraded":"2024-04-16T21:07:59.657746+0000","last_fullsized":"2024-04-16T21:07:59.657746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'13","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372599+0000","last_change":"2024-04-16T19:25:15.485057+0000","last_active":"2024-04-16T21:09:29.372599+0000","last_peered":"2024-04-16T21:09:29.372599+0000","last_clean":"2024-04-16T21:09:29.372599+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:09:29.372599+0000","last_undegraded":"2024-04-16T21:09:29.372599+0000","last_fullsized":"2024-04-16T21:09:29.372599+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:50:45.514212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374924+0000","last_change":"2024-04-16T19:25:13.472089+0000","last_active":"2024-04-16T21:09:14.374924+0000","last_peered":"2024-04-16T21:09:14.374924+0000","last_clean":"2024-04-16T21:09:14.374924+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:09:14.374924+0000","last_undegraded":"2024-04-16T21:09:14.374924+0000","last_fullsized":"2024-04-16T21:09:14.374924+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'304","reported_seq":629,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370404+0000","last_change":"2024-04-16T19:25:11.462997+0000","last_active":"2024-04-16T21:09:14.370404+0000","last_peered":"2024-04-16T21:09:14.370404+0000","last_clean":"2024-04-16T21:09:14.370404+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:09:14.370404+0000","last_undegraded":"2024-04-16T21:09:14.370404+0000","last_fullsized":"2024-04-16T21:09:14.370404+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:18:14.402830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'8","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369760+0000","last_change":"2024-04-16T19:25:09.467337+0000","last_active":"2024-04-16T21:09:19.369760+0000","last_peered":"2024-04-16T21:09:19.369760+0000","last_clean":"2024-04-16T21:09:19.369760+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:09:19.369760+0000","last_undegraded":"2024-04-16T21:09:19.369760+0000","last_fullsized":"2024-04-16T21:09:19.369760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663257+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:08:39.663257+0000","last_peered":"2024-04-16T21:08:39.663257+0000","last_clean":"2024-04-16T21:08:39.663257+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:08:39.663257+0000","last_undegraded":"2024-04-16T21:08:39.663257+0000","last_fullsized":"2024-04-16T21:08:39.663257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658066+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:08:04.658066+0000","last_peered":"2024-04-16T21:08:04.658066+0000","last_clean":"2024-04-16T21:08:04.658066+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:08:04.658066+0000","last_undegraded":"2024-04-16T21:08:04.658066+0000","last_fullsized":"2024-04-16T21:08:04.658066+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:20.908683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":959,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.505980+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:08:49.505980+0000","last_peered":"2024-04-16T21:08:49.505980+0000","last_clean":"2024-04-16T21:08:49.505980+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:08:49.505980+0000","last_undegraded":"2024-04-16T21:08:49.505980+0000","last_fullsized":"2024-04-16T21:08:49.505980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:35.897776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501172+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:08:19.501172+0000","last_peered":"2024-04-16T21:08:19.501172+0000","last_clean":"2024-04-16T21:08:19.501172+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:08:19.501172+0000","last_undegraded":"2024-04-16T21:08:19.501172+0000","last_fullsized":"2024-04-16T21:08:19.501172+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:40.145350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508432+0000","last_change":"2024-04-16T19:24:49.429526+0000","last_active":"2024-04-16T21:09:04.508432+0000","last_peered":"2024-04-16T21:09:04.508432+0000","last_clean":"2024-04-16T21:09:04.508432+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:09:04.508432+0000","last_undegraded":"2024-04-16T21:09:04.508432+0000","last_fullsized":"2024-04-16T21:09:04.508432+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:36.881378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371372+0000","last_change":"2024-04-16T19:24:51.437305+0000","last_active":"2024-04-16T21:09:24.371372+0000","last_peered":"2024-04-16T21:09:24.371372+0000","last_clean":"2024-04-16T21:09:24.371372+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:09:24.371372+0000","last_undegraded":"2024-04-16T21:09:24.371372+0000","last_fullsized":"2024-04-16T21:09:24.371372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:24.125422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366382+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:08:49.366382+0000","last_peered":"2024-04-16T21:08:49.366382+0000","last_clean":"2024-04-16T21:08:49.366382+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:08:49.366382+0000","last_undegraded":"2024-04-16T21:08:49.366382+0000","last_fullsized":"2024-04-16T21:08:49.366382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:22:33.730359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'93","reported_seq":1378,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498742+0000","last_change":"2024-04-16T19:26:41.564908+0000","last_active":"2024-04-16T21:08:04.498742+0000","last_peered":"2024-04-16T21:08:04.498742+0000","last_clean":"2024-04-16T21:08:04.498742+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:08:04.498742+0000","last_undegraded":"2024-04-16T21:08:04.498742+0000","last_fullsized":"2024-04-16T21:08:04.498742+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:47:23.710928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.654443+0000","last_change":"2024-04-16T19:25:41.508929+0000","last_active":"2024-04-16T21:07:49.654443+0000","last_peered":"2024-04-16T21:07:49.654443+0000","last_clean":"2024-04-16T21:07:49.654443+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:07:49.654443+0000","last_undegraded":"2024-04-16T21:07:49.654443+0000","last_fullsized":"2024-04-16T21:07:49.654443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'19","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372668+0000","last_change":"2024-04-16T19:25:15.482026+0000","last_active":"2024-04-16T21:09:29.372668+0000","last_peered":"2024-04-16T21:09:29.372668+0000","last_clean":"2024-04-16T21:09:29.372668+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:09:29.372668+0000","last_undegraded":"2024-04-16T21:09:29.372668+0000","last_fullsized":"2024-04-16T21:09:29.372668+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652807+0000","last_change":"2024-04-16T19:25:13.475419+0000","last_active":"2024-04-16T21:07:34.652807+0000","last_peered":"2024-04-16T21:07:34.652807+0000","last_clean":"2024-04-16T21:07:34.652807+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:07:34.652807+0000","last_undegraded":"2024-04-16T21:07:34.652807+0000","last_fullsized":"2024-04-16T21:07:34.652807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2044","reported_seq":5814,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.376528+0000","last_change":"2024-04-16T19:25:11.468843+0000","last_active":"2024-04-16T21:09:24.376528+0000","last_peered":"2024-04-16T21:09:24.376528+0000","last_clean":"2024-04-16T21:09:24.376528+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:09:24.376528+0000","last_undegraded":"2024-04-16T21:09:24.376528+0000","last_fullsized":"2024-04-16T21:09:24.376528+0000","mapping_epoch":33,"log_start":"47'300","ondisk_log_start":"47'300","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1744,"log_dups_size":300,"ondisk_log_size":1744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:03.761996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'14","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369114+0000","last_change":"2024-04-16T19:25:09.467702+0000","last_active":"2024-04-16T21:09:09.369114+0000","last_peered":"2024-04-16T21:09:09.369114+0000","last_clean":"2024-04-16T21:09:09.369114+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:09:09.369114+0000","last_undegraded":"2024-04-16T21:09:09.369114+0000","last_fullsized":"2024-04-16T21:09:09.369114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:31.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367115+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:08:59.367115+0000","last_peered":"2024-04-16T21:08:59.367115+0000","last_clean":"2024-04-16T21:08:59.367115+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:08:59.367115+0000","last_undegraded":"2024-04-16T21:08:59.367115+0000","last_fullsized":"2024-04-16T21:08:59.367115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:40.677584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.659023+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:08:04.659023+0000","last_peered":"2024-04-16T21:08:04.659023+0000","last_clean":"2024-04-16T21:08:04.659023+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:08:04.659023+0000","last_undegraded":"2024-04-16T21:08:04.659023+0000","last_fullsized":"2024-04-16T21:08:04.659023+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:49.358784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500796+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:08:19.500796+0000","last_peered":"2024-04-16T21:08:19.500796+0000","last_clean":"2024-04-16T21:08:19.500796+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:08:19.500796+0000","last_undegraded":"2024-04-16T21:08:19.500796+0000","last_fullsized":"2024-04-16T21:08:19.500796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:59.160445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":328,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366327+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:08:49.366327+0000","last_peered":"2024-04-16T21:08:49.366327+0000","last_clean":"2024-04-16T21:08:49.366327+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:08:49.366327+0000","last_undegraded":"2024-04-16T21:08:49.366327+0000","last_fullsized":"2024-04-16T21:08:49.366327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:00:06.857191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1313,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370434+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:08:49.370434+0000","last_peered":"2024-04-16T21:08:49.370434+0000","last_clean":"2024-04-16T21:08:49.370434+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:08:49.370434+0000","last_undegraded":"2024-04-16T21:08:49.370434+0000","last_fullsized":"2024-04-16T21:08:49.370434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:30.260652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371475+0000","last_change":"2024-04-16T19:24:51.437991+0000","last_active":"2024-04-16T21:08:54.371475+0000","last_peered":"2024-04-16T21:08:54.371475+0000","last_clean":"2024-04-16T21:08:54.371475+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:08:54.371475+0000","last_undegraded":"2024-04-16T21:08:54.371475+0000","last_fullsized":"2024-04-16T21:08:54.371475+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:43:50.519739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496442+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:07:49.496442+0000","last_peered":"2024-04-16T21:07:49.496442+0000","last_clean":"2024-04-16T21:07:49.496442+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:07:49.496442+0000","last_undegraded":"2024-04-16T21:07:49.496442+0000","last_fullsized":"2024-04-16T21:07:49.496442+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:16:07.995736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'136","reported_seq":1889,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497219+0000","last_change":"2024-04-16T19:25:49.523163+0000","last_active":"2024-04-16T21:07:54.497219+0000","last_peered":"2024-04-16T21:07:54.497219+0000","last_clean":"2024-04-16T21:07:54.497219+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:07:54.497219+0000","last_undegraded":"2024-04-16T21:07:54.497219+0000","last_fullsized":"2024-04-16T21:07:54.497219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:08.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655310+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:07:49.655310+0000","last_peered":"2024-04-16T21:07:49.655310+0000","last_clean":"2024-04-16T21:07:49.655310+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:07:49.655310+0000","last_undegraded":"2024-04-16T21:07:49.655310+0000","last_fullsized":"2024-04-16T21:07:49.655310+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:29:36.879550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'11","reported_seq":150,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.377292+0000","last_change":"2024-04-16T19:25:15.480717+0000","last_active":"2024-04-16T21:09:29.377292+0000","last_peered":"2024-04-16T21:09:29.377292+0000","last_clean":"2024-04-16T21:09:29.377292+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:09:29.377292+0000","last_undegraded":"2024-04-16T21:09:29.377292+0000","last_fullsized":"2024-04-16T21:09:29.377292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:39.031938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371460+0000","last_change":"2024-04-16T19:25:13.476070+0000","last_active":"2024-04-16T21:09:24.371460+0000","last_peered":"2024-04-16T21:09:24.371460+0000","last_clean":"2024-04-16T21:09:24.371460+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:09:24.371460+0000","last_undegraded":"2024-04-16T21:09:24.371460+0000","last_fullsized":"2024-04-16T21:09:24.371460+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:42.684942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'341","reported_seq":664,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494066+0000","last_change":"2024-04-16T19:25:11.467316+0000","last_active":"2024-04-16T21:07:34.494066+0000","last_peered":"2024-04-16T21:07:34.494066+0000","last_clean":"2024-04-16T21:07:34.494066+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:07:34.494066+0000","last_undegraded":"2024-04-16T21:07:34.494066+0000","last_fullsized":"2024-04-16T21:07:34.494066+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'9","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369072+0000","last_change":"2024-04-16T19:25:09.470409+0000","last_active":"2024-04-16T21:09:09.369072+0000","last_peered":"2024-04-16T21:09:09.369072+0000","last_clean":"2024-04-16T21:09:09.369072+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:09:09.369072+0000","last_undegraded":"2024-04-16T21:09:09.369072+0000","last_fullsized":"2024-04-16T21:09:09.369072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:43:01.916603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657857+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:08:09.657857+0000","last_peered":"2024-04-16T21:08:09.657857+0000","last_clean":"2024-04-16T21:08:09.657857+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:08:09.657857+0000","last_undegraded":"2024-04-16T21:08:09.657857+0000","last_fullsized":"2024-04-16T21:08:09.657857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:07.594675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658003+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:08:04.658003+0000","last_peered":"2024-04-16T21:08:04.658003+0000","last_clean":"2024-04-16T21:08:04.658003+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:08:04.658003+0000","last_undegraded":"2024-04-16T21:08:04.658003+0000","last_fullsized":"2024-04-16T21:08:04.658003+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:29.699119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":457,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508414+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:09:04.508414+0000","last_peered":"2024-04-16T21:09:04.508414+0000","last_clean":"2024-04-16T21:09:04.508414+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:09:04.508414+0000","last_undegraded":"2024-04-16T21:09:04.508414+0000","last_fullsized":"2024-04-16T21:09:04.508414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:12.039717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":246,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370400+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:08:49.370400+0000","last_peered":"2024-04-16T21:08:49.370400+0000","last_clean":"2024-04-16T21:08:49.370400+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:08:49.370400+0000","last_undegraded":"2024-04-16T21:08:49.370400+0000","last_fullsized":"2024-04-16T21:08:49.370400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:16.109122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506003+0000","last_change":"2024-04-16T19:24:49.432294+0000","last_active":"2024-04-16T21:08:49.506003+0000","last_peered":"2024-04-16T21:08:49.506003+0000","last_clean":"2024-04-16T21:08:49.506003+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:08:49.506003+0000","last_undegraded":"2024-04-16T21:08:49.506003+0000","last_fullsized":"2024-04-16T21:08:49.506003+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:45.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365802+0000","last_change":"2024-04-16T19:24:51.442478+0000","last_active":"2024-04-16T21:08:54.365802+0000","last_peered":"2024-04-16T21:08:54.365802+0000","last_clean":"2024-04-16T21:08:54.365802+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:08:54.365802+0000","last_undegraded":"2024-04-16T21:08:54.365802+0000","last_fullsized":"2024-04-16T21:08:54.365802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:34.151780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495710+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:07:44.495710+0000","last_peered":"2024-04-16T21:07:44.495710+0000","last_clean":"2024-04-16T21:07:44.495710+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:07:44.495710+0000","last_undegraded":"2024-04-16T21:07:44.495710+0000","last_fullsized":"2024-04-16T21:07:44.495710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:02:07.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2345,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652944+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:07:39.652944+0000","last_peered":"2024-04-16T21:07:39.652944+0000","last_clean":"2024-04-16T21:07:39.652944+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:07:39.652944+0000","last_undegraded":"2024-04-16T21:07:39.652944+0000","last_fullsized":"2024-04-16T21:07:39.652944+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:04.827551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658027+0000","last_change":"2024-04-16T19:25:41.509053+0000","last_active":"2024-04-16T21:08:04.658027+0000","last_peered":"2024-04-16T21:08:04.658027+0000","last_clean":"2024-04-16T21:08:04.658027+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:08:04.658027+0000","last_undegraded":"2024-04-16T21:08:04.658027+0000","last_fullsized":"2024-04-16T21:08:04.658027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:22.621386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"47'8","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372308+0000","last_change":"2024-04-16T19:25:15.486624+0000","last_active":"2024-04-16T21:09:29.372308+0000","last_peered":"2024-04-16T21:09:29.372308+0000","last_clean":"2024-04-16T21:09:29.372308+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:09:29.372308+0000","last_undegraded":"2024-04-16T21:09:29.372308+0000","last_fullsized":"2024-04-16T21:09:29.372308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:35.616571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'705","reported_seq":2065,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371841+0000","last_change":"2024-04-16T19:25:13.473118+0000","last_active":"2024-04-16T21:09:24.371841+0000","last_peered":"2024-04-16T21:09:24.371841+0000","last_clean":"2024-04-16T21:09:24.371841+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:09:24.371841+0000","last_undegraded":"2024-04-16T21:09:24.371841+0000","last_fullsized":"2024-04-16T21:09:24.371841+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'269","reported_seq":541,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.370624+0000","last_change":"2024-04-16T19:25:11.465450+0000","last_active":"2024-04-16T21:09:24.370624+0000","last_peered":"2024-04-16T21:09:24.370624+0000","last_clean":"2024-04-16T21:09:24.370624+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:09:24.370624+0000","last_undegraded":"2024-04-16T21:09:24.370624+0000","last_fullsized":"2024-04-16T21:09:24.370624+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:53.479823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'5","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375596+0000","last_change":"2024-04-16T19:25:09.458268+0000","last_active":"2024-04-16T21:09:19.375596+0000","last_peered":"2024-04-16T21:09:19.375596+0000","last_clean":"2024-04-16T21:09:19.375596+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:09:19.375596+0000","last_undegraded":"2024-04-16T21:09:19.375596+0000","last_fullsized":"2024-04-16T21:09:19.375596+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:05.252198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658783+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:08:09.658783+0000","last_peered":"2024-04-16T21:08:09.658783+0000","last_clean":"2024-04-16T21:08:09.658783+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:08:09.658783+0000","last_undegraded":"2024-04-16T21:08:09.658783+0000","last_fullsized":"2024-04-16T21:08:09.658783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:44:29.013265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498840+0000","last_change":"2024-04-16T19:25:41.503885+0000","last_active":"2024-04-16T21:08:04.498840+0000","last_peered":"2024-04-16T21:08:04.498840+0000","last_clean":"2024-04-16T21:08:04.498840+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:08:04.498840+0000","last_undegraded":"2024-04-16T21:08:04.498840+0000","last_fullsized":"2024-04-16T21:08:04.498840+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:28:54.516253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500846+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:08:19.500846+0000","last_peered":"2024-04-16T21:08:19.500846+0000","last_clean":"2024-04-16T21:08:19.500846+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:08:19.500846+0000","last_undegraded":"2024-04-16T21:08:19.500846+0000","last_fullsized":"2024-04-16T21:08:19.500846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:30.224132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660316+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:08:19.660316+0000","last_peered":"2024-04-16T21:08:19.660316+0000","last_clean":"2024-04-16T21:08:19.660316+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:08:19.660316+0000","last_undegraded":"2024-04-16T21:08:19.660316+0000","last_fullsized":"2024-04-16T21:08:19.660316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:43.606761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662124+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:08:34.662124+0000","last_peered":"2024-04-16T21:08:34.662124+0000","last_clean":"2024-04-16T21:08:34.662124+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:08:34.662124+0000","last_undegraded":"2024-04-16T21:08:34.662124+0000","last_fullsized":"2024-04-16T21:08:34.662124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368874+0000","last_change":"2024-04-16T19:24:51.434984+0000","last_active":"2024-04-16T21:09:14.368874+0000","last_peered":"2024-04-16T21:09:14.368874+0000","last_clean":"2024-04-16T21:09:14.368874+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:09:14.368874+0000","last_undegraded":"2024-04-16T21:09:14.368874+0000","last_fullsized":"2024-04-16T21:09:14.368874+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:26.835802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.653730+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:07:44.653730+0000","last_peered":"2024-04-16T21:07:44.653730+0000","last_clean":"2024-04-16T21:07:44.653730+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:07:44.653730+0000","last_undegraded":"2024-04-16T21:07:44.653730+0000","last_fullsized":"2024-04-16T21:07:44.653730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:56.544746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1678,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653711+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:07:39.653711+0000","last_peered":"2024-04-16T21:07:39.653711+0000","last_clean":"2024-04-16T21:07:39.653711+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:07:39.653711+0000","last_undegraded":"2024-04-16T21:07:39.653711+0000","last_fullsized":"2024-04-16T21:07:39.653711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:32:01.314955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655387+0000","last_change":"2024-04-16T19:25:41.504293+0000","last_active":"2024-04-16T21:07:49.655387+0000","last_peered":"2024-04-16T21:07:49.655387+0000","last_clean":"2024-04-16T21:07:49.655387+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:07:49.655387+0000","last_undegraded":"2024-04-16T21:07:49.655387+0000","last_fullsized":"2024-04-16T21:07:49.655387+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:05:53.361713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'136","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.654370+0000","last_change":"2024-04-16T19:25:15.483254+0000","last_active":"2024-04-16T21:07:39.654370+0000","last_peered":"2024-04-16T21:07:39.654370+0000","last_clean":"2024-04-16T21:07:39.654370+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:07:39.654370+0000","last_undegraded":"2024-04-16T21:07:39.654370+0000","last_fullsized":"2024-04-16T21:07:39.654370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511597+0000","last_change":"2024-04-16T19:25:13.471760+0000","last_active":"2024-04-16T21:09:24.511597+0000","last_peered":"2024-04-16T21:09:24.511597+0000","last_clean":"2024-04-16T21:09:24.511597+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:09:24.511597+0000","last_undegraded":"2024-04-16T21:09:24.511597+0000","last_fullsized":"2024-04-16T21:09:24.511597+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:27.469479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'347","reported_seq":725,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511636+0000","last_change":"2024-04-16T19:25:11.467463+0000","last_active":"2024-04-16T21:09:24.511636+0000","last_peered":"2024-04-16T21:09:24.511636+0000","last_clean":"2024-04-16T21:09:24.511636+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:09:24.511636+0000","last_undegraded":"2024-04-16T21:09:24.511636+0000","last_fullsized":"2024-04-16T21:09:24.511636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'10","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.371455+0000","last_change":"2024-04-16T19:25:09.461452+0000","last_active":"2024-04-16T21:09:29.371455+0000","last_peered":"2024-04-16T21:09:29.371455+0000","last_clean":"2024-04-16T21:09:29.371455+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:09:29.371455+0000","last_undegraded":"2024-04-16T21:09:29.371455+0000","last_fullsized":"2024-04-16T21:09:29.371455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:44:08.538156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659737+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:08:09.659737+0000","last_peered":"2024-04-16T21:08:09.659737+0000","last_clean":"2024-04-16T21:08:09.659737+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:08:09.659737+0000","last_undegraded":"2024-04-16T21:08:09.659737+0000","last_fullsized":"2024-04-16T21:08:09.659737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:27:31.576803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496482+0000","last_change":"2024-04-16T19:25:41.506323+0000","last_active":"2024-04-16T21:07:49.496482+0000","last_peered":"2024-04-16T21:07:49.496482+0000","last_clean":"2024-04-16T21:07:49.496482+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:07:49.496482+0000","last_undegraded":"2024-04-16T21:07:49.496482+0000","last_fullsized":"2024-04-16T21:07:49.496482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:05.815658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501233+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:08:19.501233+0000","last_peered":"2024-04-16T21:08:19.501233+0000","last_clean":"2024-04-16T21:08:19.501233+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:08:19.501233+0000","last_undegraded":"2024-04-16T21:08:19.501233+0000","last_fullsized":"2024-04-16T21:08:19.501233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:51.524577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'140","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503630+0000","last_change":"2024-04-16T19:24:47.426406+0000","last_active":"2024-04-16T21:08:34.503630+0000","last_peered":"2024-04-16T21:08:34.503630+0000","last_clean":"2024-04-16T21:08:34.503630+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:08:34.503630+0000","last_undegraded":"2024-04-16T21:08:34.503630+0000","last_fullsized":"2024-04-16T21:08:34.503630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:12.378311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.500867+0000","last_change":"2024-04-16T19:24:49.430773+0000","last_active":"2024-04-16T21:08:19.500867+0000","last_peered":"2024-04-16T21:08:19.500867+0000","last_clean":"2024-04-16T21:08:19.500867+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:08:19.500867+0000","last_undegraded":"2024-04-16T21:08:19.500867+0000","last_fullsized":"2024-04-16T21:08:19.500867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:02:52.965094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367299+0000","last_change":"2024-04-16T19:24:51.436754+0000","last_active":"2024-04-16T21:09:04.367299+0000","last_peered":"2024-04-16T21:09:04.367299+0000","last_clean":"2024-04-16T21:09:04.367299+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:09:04.367299+0000","last_undegraded":"2024-04-16T21:09:04.367299+0000","last_fullsized":"2024-04-16T21:09:04.367299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:01.678207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368731+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:09:04.368731+0000","last_peered":"2024-04-16T21:09:04.368731+0000","last_clean":"2024-04-16T21:09:04.368731+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:09:04.368731+0000","last_undegraded":"2024-04-16T21:09:04.368731+0000","last_fullsized":"2024-04-16T21:09:04.368731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:01:32.497457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1870,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656880+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:07:54.656880+0000","last_peered":"2024-04-16T21:07:54.656880+0000","last_clean":"2024-04-16T21:07:54.656880+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:07:54.656880+0000","last_undegraded":"2024-04-16T21:07:54.656880+0000","last_fullsized":"2024-04-16T21:07:54.656880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:11:54.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653650+0000","last_change":"2024-04-16T19:25:41.507626+0000","last_active":"2024-04-16T21:07:34.653650+0000","last_peered":"2024-04-16T21:07:34.653650+0000","last_clean":"2024-04-16T21:07:34.653650+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:07:34.653650+0000","last_undegraded":"2024-04-16T21:07:34.653650+0000","last_fullsized":"2024-04-16T21:07:34.653650+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'7","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369826+0000","last_change":"2024-04-16T19:25:15.477722+0000","last_active":"2024-04-16T21:09:19.369826+0000","last_peered":"2024-04-16T21:09:19.369826+0000","last_clean":"2024-04-16T21:09:19.369826+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:09:19.369826+0000","last_undegraded":"2024-04-16T21:09:19.369826+0000","last_fullsized":"2024-04-16T21:09:19.369826+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:11.553471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371260+0000","last_change":"2024-04-16T19:25:13.476043+0000","last_active":"2024-04-16T21:09:24.371260+0000","last_peered":"2024-04-16T21:09:24.371260+0000","last_clean":"2024-04-16T21:09:24.371260+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:09:24.371260+0000","last_undegraded":"2024-04-16T21:09:24.371260+0000","last_fullsized":"2024-04-16T21:09:24.371260+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:32.861196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'453","reported_seq":869,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.373003+0000","last_change":"2024-04-16T19:25:11.462559+0000","last_active":"2024-04-16T21:09:34.373003+0000","last_peered":"2024-04-16T21:09:34.373003+0000","last_clean":"2024-04-16T21:09:34.373003+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:09:34.373003+0000","last_undegraded":"2024-04-16T21:09:34.373003+0000","last_fullsized":"2024-04-16T21:09:34.373003+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:33.939814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'8","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655202+0000","last_change":"2024-04-16T19:25:41.507771+0000","last_active":"2024-04-16T21:07:49.655202+0000","last_peered":"2024-04-16T21:07:49.655202+0000","last_clean":"2024-04-16T21:07:49.655202+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:07:49.655202+0000","last_undegraded":"2024-04-16T21:07:49.655202+0000","last_fullsized":"2024-04-16T21:07:49.655202+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:53.575807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658859+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:08:09.658859+0000","last_peered":"2024-04-16T21:08:09.658859+0000","last_clean":"2024-04-16T21:08:09.658859+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:08:09.658859+0000","last_undegraded":"2024-04-16T21:08:09.658859+0000","last_fullsized":"2024-04-16T21:08:09.658859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662590+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:08:34.662590+0000","last_peered":"2024-04-16T21:08:34.662590+0000","last_clean":"2024-04-16T21:08:34.662590+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:08:34.662590+0000","last_undegraded":"2024-04-16T21:08:34.662590+0000","last_fullsized":"2024-04-16T21:08:34.662590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1361,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663160+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:08:34.663160+0000","last_peered":"2024-04-16T21:08:34.663160+0000","last_clean":"2024-04-16T21:08:34.663160+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:08:34.663160+0000","last_undegraded":"2024-04-16T21:08:34.663160+0000","last_fullsized":"2024-04-16T21:08:34.663160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:46.278655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":276,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501256+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:08:19.501256+0000","last_peered":"2024-04-16T21:08:19.501256+0000","last_clean":"2024-04-16T21:08:19.501256+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:08:19.501256+0000","last_undegraded":"2024-04-16T21:08:19.501256+0000","last_fullsized":"2024-04-16T21:08:19.501256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:23.115125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'12","reported_seq":1311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660267+0000","last_change":"2024-04-16T19:24:49.428502+0000","last_active":"2024-04-16T21:08:19.660267+0000","last_peered":"2024-04-16T21:08:19.660267+0000","last_clean":"2024-04-16T21:08:19.660267+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:08:19.660267+0000","last_undegraded":"2024-04-16T21:08:19.660267+0000","last_fullsized":"2024-04-16T21:08:19.660267+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:25.122206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367884+0000","last_change":"2024-04-16T19:24:51.435114+0000","last_active":"2024-04-16T21:09:04.367884+0000","last_peered":"2024-04-16T21:09:04.367884+0000","last_clean":"2024-04-16T21:09:04.367884+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:09:04.367884+0000","last_undegraded":"2024-04-16T21:09:04.367884+0000","last_fullsized":"2024-04-16T21:09:04.367884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:56.562306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364938+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:08:49.364938+0000","last_peered":"2024-04-16T21:08:49.364938+0000","last_clean":"2024-04-16T21:08:49.364938+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:08:49.364938+0000","last_undegraded":"2024-04-16T21:08:49.364938+0000","last_fullsized":"2024-04-16T21:08:49.364938+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:08:56.543210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657898+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:08:09.657898+0000","last_peered":"2024-04-16T21:08:09.657898+0000","last_clean":"2024-04-16T21:08:09.657898+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:08:09.657898+0000","last_undegraded":"2024-04-16T21:08:09.657898+0000","last_fullsized":"2024-04-16T21:08:09.657898+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:44:18.098055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653619+0000","last_change":"2024-04-16T19:25:41.506250+0000","last_active":"2024-04-16T21:07:34.653619+0000","last_peered":"2024-04-16T21:07:34.653619+0000","last_clean":"2024-04-16T21:07:34.653619+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:07:34.653619+0000","last_undegraded":"2024-04-16T21:07:34.653619+0000","last_fullsized":"2024-04-16T21:07:34.653619+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:27.042973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369866+0000","last_change":"2024-04-16T19:25:15.479905+0000","last_active":"2024-04-16T21:09:19.369866+0000","last_peered":"2024-04-16T21:09:19.369866+0000","last_clean":"2024-04-16T21:09:19.369866+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:09:19.369866+0000","last_undegraded":"2024-04-16T21:09:19.369866+0000","last_fullsized":"2024-04-16T21:09:19.369866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:03.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'705","reported_seq":2464,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.372193+0000","last_change":"2024-04-16T19:25:13.470216+0000","last_active":"2024-04-16T21:09:34.372193+0000","last_peered":"2024-04-16T21:09:34.372193+0000","last_clean":"2024-04-16T21:09:34.372193+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:09:34.372193+0000","last_undegraded":"2024-04-16T21:09:34.372193+0000","last_fullsized":"2024-04-16T21:09:34.372193+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:40.891478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'311","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371233+0000","last_change":"2024-04-16T19:25:11.464746+0000","last_active":"2024-04-16T21:09:24.371233+0000","last_peered":"2024-04-16T21:09:24.371233+0000","last_clean":"2024-04-16T21:09:24.371233+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:09:24.371233+0000","last_undegraded":"2024-04-16T21:09:24.371233+0000","last_fullsized":"2024-04-16T21:09:24.371233+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:44.875603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'6","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369697+0000","last_change":"2024-04-16T19:25:09.466128+0000","last_active":"2024-04-16T21:09:19.369697+0000","last_peered":"2024-04-16T21:09:19.369697+0000","last_clean":"2024-04-16T21:09:19.369697+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:09:19.369697+0000","last_undegraded":"2024-04-16T21:09:19.369697+0000","last_fullsized":"2024-04-16T21:09:19.369697+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:15:28.846589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.502055+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:08:24.502055+0000","last_peered":"2024-04-16T21:08:24.502055+0000","last_clean":"2024-04-16T21:08:24.502055+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:08:24.502055+0000","last_undegraded":"2024-04-16T21:08:24.502055+0000","last_fullsized":"2024-04-16T21:08:24.502055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:20.204370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503320+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:08:34.503320+0000","last_peered":"2024-04-16T21:08:34.503320+0000","last_clean":"2024-04-16T21:08:34.503320+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:08:34.503320+0000","last_undegraded":"2024-04-16T21:08:34.503320+0000","last_fullsized":"2024-04-16T21:08:34.503320+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:08:26.977348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661224+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:08:19.661224+0000","last_peered":"2024-04-16T21:08:19.661224+0000","last_clean":"2024-04-16T21:08:19.661224+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:08:19.661224+0000","last_undegraded":"2024-04-16T21:08:19.661224+0000","last_fullsized":"2024-04-16T21:08:19.661224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:45.288392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":359,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.660463+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:08:19.660463+0000","last_peered":"2024-04-16T21:08:19.660463+0000","last_clean":"2024-04-16T21:08:19.660463+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:08:19.660463+0000","last_undegraded":"2024-04-16T21:08:19.660463+0000","last_fullsized":"2024-04-16T21:08:19.660463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:48.207837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661146+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:08:19.661146+0000","last_peered":"2024-04-16T21:08:19.661146+0000","last_clean":"2024-04-16T21:08:19.661146+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:08:19.661146+0000","last_undegraded":"2024-04-16T21:08:19.661146+0000","last_fullsized":"2024-04-16T21:08:19.661146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367334+0000","last_change":"2024-04-16T19:24:51.434704+0000","last_active":"2024-04-16T21:09:04.367334+0000","last_peered":"2024-04-16T21:09:04.367334+0000","last_clean":"2024-04-16T21:09:04.367334+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:09:04.367334+0000","last_undegraded":"2024-04-16T21:09:04.367334+0000","last_fullsized":"2024-04-16T21:09:04.367334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503573+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:08:34.503573+0000","last_peered":"2024-04-16T21:08:34.503573+0000","last_clean":"2024-04-16T21:08:34.503573+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:08:34.503573+0000","last_undegraded":"2024-04-16T21:08:34.503573+0000","last_fullsized":"2024-04-16T21:08:34.503573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:19:11.491071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'162","reported_seq":2095,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650676+0000","last_change":"2024-04-16T19:25:49.518828+0000","last_active":"2024-04-16T21:07:19.650676+0000","last_peered":"2024-04-16T21:07:19.650676+0000","last_clean":"2024-04-16T21:07:19.650676+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:07:19.650676+0000","last_undegraded":"2024-04-16T21:07:19.650676+0000","last_fullsized":"2024-04-16T21:07:19.650676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:22.736467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368970+0000","last_change":"2024-04-16T19:25:41.510329+0000","last_active":"2024-04-16T21:09:14.368970+0000","last_peered":"2024-04-16T21:09:14.368970+0000","last_clean":"2024-04-16T21:09:14.368970+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:09:14.368970+0000","last_undegraded":"2024-04-16T21:09:14.368970+0000","last_fullsized":"2024-04-16T21:09:14.368970+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:48:14.723662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'12","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365503+0000","last_change":"2024-04-16T19:25:15.480613+0000","last_active":"2024-04-16T21:08:49.365503+0000","last_peered":"2024-04-16T21:08:49.365503+0000","last_clean":"2024-04-16T21:08:49.365503+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:08:49.365503+0000","last_undegraded":"2024-04-16T21:08:49.365503+0000","last_fullsized":"2024-04-16T21:08:49.365503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491622+0000","last_change":"2024-04-16T19:25:13.469639+0000","last_active":"2024-04-16T21:07:19.491622+0000","last_peered":"2024-04-16T21:07:19.491622+0000","last_clean":"2024-04-16T21:07:19.491622+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:07:19.491622+0000","last_undegraded":"2024-04-16T21:07:19.491622+0000","last_fullsized":"2024-04-16T21:07:19.491622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:02.547322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'256","reported_seq":566,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364495+0000","last_change":"2024-04-16T19:25:11.463503+0000","last_active":"2024-04-16T21:08:44.364495+0000","last_peered":"2024-04-16T21:08:44.364495+0000","last_clean":"2024-04-16T21:08:44.364495+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:08:44.364495+0000","last_undegraded":"2024-04-16T21:08:44.364495+0000","last_fullsized":"2024-04-16T21:08:44.364495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:28:07.258240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'6","reported_seq":233,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509214+0000","last_change":"2024-04-16T19:25:09.462147+0000","last_active":"2024-04-16T21:09:09.509214+0000","last_peered":"2024-04-16T21:09:09.509214+0000","last_clean":"2024-04-16T21:09:09.509214+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:09:09.509214+0000","last_undegraded":"2024-04-16T21:09:09.509214+0000","last_fullsized":"2024-04-16T21:09:09.509214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:52.170197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369516+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:09:09.369516+0000","last_peered":"2024-04-16T21:09:09.369516+0000","last_clean":"2024-04-16T21:09:09.369516+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:09:09.369516+0000","last_undegraded":"2024-04-16T21:09:09.369516+0000","last_fullsized":"2024-04-16T21:09:09.369516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:41:22.844339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658852+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:08:04.658852+0000","last_peered":"2024-04-16T21:08:04.658852+0000","last_clean":"2024-04-16T21:08:04.658852+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:08:04.658852+0000","last_undegraded":"2024-04-16T21:08:04.658852+0000","last_fullsized":"2024-04-16T21:08:04.658852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:26.415645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657180+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:07:59.657180+0000","last_peered":"2024-04-16T21:07:59.657180+0000","last_clean":"2024-04-16T21:07:59.657180+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:07:59.657180+0000","last_undegraded":"2024-04-16T21:07:59.657180+0000","last_fullsized":"2024-04-16T21:07:59.657180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.363997+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:08:44.363997+0000","last_peered":"2024-04-16T21:08:44.363997+0000","last_clean":"2024-04-16T21:08:44.363997+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:08:44.363997+0000","last_undegraded":"2024-04-16T21:08:44.363997+0000","last_fullsized":"2024-04-16T21:08:44.363997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:01:19.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367212+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:08:59.367212+0000","last_peered":"2024-04-16T21:08:59.367212+0000","last_clean":"2024-04-16T21:08:59.367212+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:08:59.367212+0000","last_undegraded":"2024-04-16T21:08:59.367212+0000","last_fullsized":"2024-04-16T21:08:59.367212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:49.663208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365555+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:08:49.365555+0000","last_peered":"2024-04-16T21:08:49.365555+0000","last_clean":"2024-04-16T21:08:49.365555+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:08:49.365555+0000","last_undegraded":"2024-04-16T21:08:49.365555+0000","last_fullsized":"2024-04-16T21:08:49.365555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370592+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:08:49.370592+0000","last_peered":"2024-04-16T21:08:49.370592+0000","last_clean":"2024-04-16T21:08:49.370592+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:08:49.370592+0000","last_undegraded":"2024-04-16T21:08:49.370592+0000","last_fullsized":"2024-04-16T21:08:49.370592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366370+0000","last_change":"2024-04-16T19:24:51.442576+0000","last_active":"2024-04-16T21:08:54.366370+0000","last_peered":"2024-04-16T21:08:54.366370+0000","last_clean":"2024-04-16T21:08:54.366370+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:08:54.366370+0000","last_undegraded":"2024-04-16T21:08:54.366370+0000","last_fullsized":"2024-04-16T21:08:54.366370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662725+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:08:34.662725+0000","last_peered":"2024-04-16T21:08:34.662725+0000","last_clean":"2024-04-16T21:08:34.662725+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:08:34.662725+0000","last_undegraded":"2024-04-16T21:08:34.662725+0000","last_fullsized":"2024-04-16T21:08:34.662725+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:34:27.769930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1951,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375667+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:09:19.375667+0000","last_peered":"2024-04-16T21:09:19.375667+0000","last_clean":"2024-04-16T21:09:19.375667+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:09:19.375667+0000","last_undegraded":"2024-04-16T21:09:19.375667+0000","last_fullsized":"2024-04-16T21:09:19.375667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:42.651587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649772+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:07:14.649772+0000","last_peered":"2024-04-16T21:07:14.649772+0000","last_clean":"2024-04-16T21:07:14.649772+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:07:14.649772+0000","last_undegraded":"2024-04-16T21:07:14.649772+0000","last_fullsized":"2024-04-16T21:07:14.649772+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'13","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370574+0000","last_change":"2024-04-16T19:25:15.485212+0000","last_active":"2024-04-16T21:08:49.370574+0000","last_peered":"2024-04-16T21:08:49.370574+0000","last_clean":"2024-04-16T21:08:49.370574+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:08:49.370574+0000","last_undegraded":"2024-04-16T21:08:49.370574+0000","last_fullsized":"2024-04-16T21:08:49.370574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:41.072415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'705","reported_seq":2077,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509982+0000","last_change":"2024-04-16T19:25:13.467216+0000","last_active":"2024-04-16T21:09:14.509982+0000","last_peered":"2024-04-16T21:09:14.509982+0000","last_clean":"2024-04-16T21:09:14.509982+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:09:14.509982+0000","last_undegraded":"2024-04-16T21:09:14.509982+0000","last_fullsized":"2024-04-16T21:09:14.509982+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'306","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374758+0000","last_change":"2024-04-16T19:25:11.467288+0000","last_active":"2024-04-16T21:09:14.374758+0000","last_peered":"2024-04-16T21:09:14.374758+0000","last_clean":"2024-04-16T21:09:14.374758+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:09:14.374758+0000","last_undegraded":"2024-04-16T21:09:14.374758+0000","last_fullsized":"2024-04-16T21:09:14.374758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'12","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662672+0000","last_change":"2024-04-16T19:25:09.464018+0000","last_active":"2024-04-16T21:08:39.662672+0000","last_peered":"2024-04-16T21:08:39.662672+0000","last_clean":"2024-04-16T21:08:39.662672+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:08:39.662672+0000","last_undegraded":"2024-04-16T21:08:39.662672+0000","last_fullsized":"2024-04-16T21:08:39.662672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:52.913431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662788+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:08:39.662788+0000","last_peered":"2024-04-16T21:08:39.662788+0000","last_clean":"2024-04-16T21:08:39.662788+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:08:39.662788+0000","last_undegraded":"2024-04-16T21:08:39.662788+0000","last_fullsized":"2024-04-16T21:08:39.662788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658875+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:08:04.658875+0000","last_peered":"2024-04-16T21:08:04.658875+0000","last_clean":"2024-04-16T21:08:04.658875+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:08:04.658875+0000","last_undegraded":"2024-04-16T21:08:04.658875+0000","last_fullsized":"2024-04-16T21:08:04.658875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:26.452728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369665+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:08:44.369665+0000","last_peered":"2024-04-16T21:08:44.369665+0000","last_clean":"2024-04-16T21:08:44.369665+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:08:44.369665+0000","last_undegraded":"2024-04-16T21:08:44.369665+0000","last_fullsized":"2024-04-16T21:08:44.369665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:23.643674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":591,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365570+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:08:49.365570+0000","last_peered":"2024-04-16T21:08:49.365570+0000","last_clean":"2024-04-16T21:08:49.365570+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:08:49.365570+0000","last_undegraded":"2024-04-16T21:08:49.365570+0000","last_fullsized":"2024-04-16T21:08:49.365570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:34.895652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366657+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:08:59.366657+0000","last_peered":"2024-04-16T21:08:59.366657+0000","last_clean":"2024-04-16T21:08:59.366657+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:08:59.366657+0000","last_undegraded":"2024-04-16T21:08:59.366657+0000","last_fullsized":"2024-04-16T21:08:59.366657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:26:28.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370637+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:09:19.370637+0000","last_peered":"2024-04-16T21:09:19.370637+0000","last_clean":"2024-04-16T21:09:19.370637+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:09:19.370637+0000","last_undegraded":"2024-04-16T21:09:19.370637+0000","last_fullsized":"2024-04-16T21:09:19.370637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:42.152431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368921+0000","last_change":"2024-04-16T19:24:51.574513+0000","last_active":"2024-04-16T21:09:14.368921+0000","last_peered":"2024-04-16T21:09:14.368921+0000","last_clean":"2024-04-16T21:09:14.368921+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:09:14.368921+0000","last_undegraded":"2024-04-16T21:09:14.368921+0000","last_fullsized":"2024-04-16T21:09:14.368921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:32.268529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663021+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:08:34.663021+0000","last_peered":"2024-04-16T21:08:34.663021+0000","last_clean":"2024-04-16T21:08:34.663021+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:08:34.663021+0000","last_undegraded":"2024-04-16T21:08:34.663021+0000","last_fullsized":"2024-04-16T21:08:34.663021+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:37.701350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2369,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370711+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:09:19.370711+0000","last_peered":"2024-04-16T21:09:19.370711+0000","last_clean":"2024-04-16T21:09:19.370711+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:09:19.370711+0000","last_undegraded":"2024-04-16T21:09:19.370711+0000","last_fullsized":"2024-04-16T21:09:19.370711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:11:19.505797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370008+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:09:14.370008+0000","last_peered":"2024-04-16T21:09:14.370008+0000","last_clean":"2024-04-16T21:09:14.370008+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:09:14.370008+0000","last_undegraded":"2024-04-16T21:09:14.370008+0000","last_fullsized":"2024-04-16T21:09:14.370008+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:52.804371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'16","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369588+0000","last_change":"2024-04-16T19:25:15.484662+0000","last_active":"2024-04-16T21:09:09.369588+0000","last_peered":"2024-04-16T21:09:09.369588+0000","last_clean":"2024-04-16T21:09:09.369588+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:09:09.369588+0000","last_undegraded":"2024-04-16T21:09:09.369588+0000","last_fullsized":"2024-04-16T21:09:09.369588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'20","reported_seq":972,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509922+0000","last_change":"2024-04-16T19:25:13.474689+0000","last_active":"2024-04-16T21:09:14.509922+0000","last_peered":"2024-04-16T21:09:14.509922+0000","last_clean":"2024-04-16T21:09:14.509922+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:09:14.509922+0000","last_undegraded":"2024-04-16T21:09:14.509922+0000","last_fullsized":"2024-04-16T21:09:14.509922+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:36.914573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'249","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370263+0000","last_change":"2024-04-16T19:25:11.460951+0000","last_active":"2024-04-16T21:09:14.370263+0000","last_peered":"2024-04-16T21:09:14.370263+0000","last_clean":"2024-04-16T21:09:14.370263+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:09:14.370263+0000","last_undegraded":"2024-04-16T21:09:14.370263+0000","last_fullsized":"2024-04-16T21:09:14.370263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:42:59.386912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'7","reported_seq":166,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372690+0000","last_change":"2024-04-16T19:25:09.460368+0000","last_active":"2024-04-16T21:09:29.372690+0000","last_peered":"2024-04-16T21:09:29.372690+0000","last_clean":"2024-04-16T21:09:29.372690+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:09:29.372690+0000","last_undegraded":"2024-04-16T21:09:29.372690+0000","last_fullsized":"2024-04-16T21:09:29.372690+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:19.738380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504660+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:08:39.504660+0000","last_peered":"2024-04-16T21:08:39.504660+0000","last_clean":"2024-04-16T21:08:39.504660+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:08:39.504660+0000","last_undegraded":"2024-04-16T21:08:39.504660+0000","last_fullsized":"2024-04-16T21:08:39.504660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:35.284891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497271+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:07:54.497271+0000","last_peered":"2024-04-16T21:07:54.497271+0000","last_clean":"2024-04-16T21:07:54.497271+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:07:54.497271+0000","last_undegraded":"2024-04-16T21:07:54.497271+0000","last_fullsized":"2024-04-16T21:07:54.497271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:16:27.416414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.365627+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:08:54.365627+0000","last_peered":"2024-04-16T21:08:54.365627+0000","last_clean":"2024-04-16T21:08:54.365627+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:08:54.365627+0000","last_undegraded":"2024-04-16T21:08:54.365627+0000","last_fullsized":"2024-04-16T21:08:54.365627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:44.737149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506112+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:08:49.506112+0000","last_peered":"2024-04-16T21:08:49.506112+0000","last_clean":"2024-04-16T21:08:49.506112+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:08:49.506112+0000","last_undegraded":"2024-04-16T21:08:49.506112+0000","last_fullsized":"2024-04-16T21:08:49.506112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:32.935806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1513,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370734+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:09:19.370734+0000","last_peered":"2024-04-16T21:09:19.370734+0000","last_clean":"2024-04-16T21:09:19.370734+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:09:19.370734+0000","last_undegraded":"2024-04-16T21:09:19.370734+0000","last_fullsized":"2024-04-16T21:09:19.370734+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372374+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:08:59.372374+0000","last_peered":"2024-04-16T21:08:59.372374+0000","last_clean":"2024-04-16T21:08:59.372374+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:08:59.372374+0000","last_undegraded":"2024-04-16T21:08:59.372374+0000","last_fullsized":"2024-04-16T21:08:59.372374+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660288+0000","last_change":"2024-04-16T19:24:51.438114+0000","last_active":"2024-04-16T21:08:24.660288+0000","last_peered":"2024-04-16T21:08:24.660288+0000","last_clean":"2024-04-16T21:08:24.660288+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:08:24.660288+0000","last_undegraded":"2024-04-16T21:08:24.660288+0000","last_fullsized":"2024-04-16T21:08:24.660288+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:01:59.734805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366544+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:08:49.366544+0000","last_peered":"2024-04-16T21:08:49.366544+0000","last_clean":"2024-04-16T21:08:49.366544+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:08:49.366544+0000","last_undegraded":"2024-04-16T21:08:49.366544+0000","last_fullsized":"2024-04-16T21:08:49.366544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:25:31.551960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1766,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.370669+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:09:19.370669+0000","last_peered":"2024-04-16T21:09:19.370669+0000","last_clean":"2024-04-16T21:09:19.370669+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:09:19.370669+0000","last_undegraded":"2024-04-16T21:09:19.370669+0000","last_fullsized":"2024-04-16T21:09:19.370669+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:10.736923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369787+0000","last_change":"2024-04-16T19:25:41.505274+0000","last_active":"2024-04-16T21:09:14.369787+0000","last_peered":"2024-04-16T21:09:14.369787+0000","last_clean":"2024-04-16T21:09:14.369787+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:09:14.369787+0000","last_undegraded":"2024-04-16T21:09:14.369787+0000","last_fullsized":"2024-04-16T21:09:14.369787+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:50:27.279669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'15","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370523+0000","last_change":"2024-04-16T19:25:15.487696+0000","last_active":"2024-04-16T21:08:49.370523+0000","last_peered":"2024-04-16T21:08:49.370523+0000","last_clean":"2024-04-16T21:08:49.370523+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:08:49.370523+0000","last_undegraded":"2024-04-16T21:08:49.370523+0000","last_fullsized":"2024-04-16T21:08:49.370523+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:34.149887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505292+0000","last_change":"2024-04-16T19:25:13.473305+0000","last_active":"2024-04-16T21:08:44.505292+0000","last_peered":"2024-04-16T21:08:44.505292+0000","last_clean":"2024-04-16T21:08:44.505292+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:08:44.505292+0000","last_undegraded":"2024-04-16T21:08:44.505292+0000","last_fullsized":"2024-04-16T21:08:44.505292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:38.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'690","reported_seq":1689,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:25:11.468818+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:21.906682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'16","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373858+0000","last_change":"2024-04-16T19:25:09.462529+0000","last_active":"2024-04-16T21:09:09.373858+0000","last_peered":"2024-04-16T21:09:09.373858+0000","last_clean":"2024-04-16T21:09:09.373858+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:09:09.373858+0000","last_undegraded":"2024-04-16T21:09:09.373858+0000","last_fullsized":"2024-04-16T21:09:09.373858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364926+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:08:49.364926+0000","last_peered":"2024-04-16T21:08:49.364926+0000","last_clean":"2024-04-16T21:08:49.364926+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:08:49.364926+0000","last_undegraded":"2024-04-16T21:08:49.364926+0000","last_fullsized":"2024-04-16T21:08:49.364926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:02.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657141+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:07:59.657141+0000","last_peered":"2024-04-16T21:07:59.657141+0000","last_clean":"2024-04-16T21:07:59.657141+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:07:59.657141+0000","last_undegraded":"2024-04-16T21:07:59.657141+0000","last_fullsized":"2024-04-16T21:07:59.657141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:54:58.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364646+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:08:44.364646+0000","last_peered":"2024-04-16T21:08:44.364646+0000","last_clean":"2024-04-16T21:08:44.364646+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:08:44.364646+0000","last_undegraded":"2024-04-16T21:08:44.364646+0000","last_fullsized":"2024-04-16T21:08:44.364646+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:21:21.093051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375716+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:09:19.375716+0000","last_peered":"2024-04-16T21:09:19.375716+0000","last_clean":"2024-04-16T21:09:19.375716+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:09:19.375716+0000","last_undegraded":"2024-04-16T21:09:19.375716+0000","last_fullsized":"2024-04-16T21:09:19.375716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:21.104930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":311,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364984+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:08:49.364984+0000","last_peered":"2024-04-16T21:08:49.364984+0000","last_clean":"2024-04-16T21:08:49.364984+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:08:49.364984+0000","last_undegraded":"2024-04-16T21:08:49.364984+0000","last_fullsized":"2024-04-16T21:08:49.364984+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:58.758206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366555+0000","last_change":"2024-04-16T19:24:49.432013+0000","last_active":"2024-04-16T21:08:49.366555+0000","last_peered":"2024-04-16T21:08:49.366555+0000","last_clean":"2024-04-16T21:08:49.366555+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:08:49.366555+0000","last_undegraded":"2024-04-16T21:08:49.366555+0000","last_fullsized":"2024-04-16T21:08:49.366555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:07.795495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366349+0000","last_change":"2024-04-16T19:24:51.436955+0000","last_active":"2024-04-16T21:08:54.366349+0000","last_peered":"2024-04-16T21:08:54.366349+0000","last_clean":"2024-04-16T21:08:54.366349+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:08:54.366349+0000","last_undegraded":"2024-04-16T21:08:54.366349+0000","last_fullsized":"2024-04-16T21:08:54.366349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:16.063621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654394+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:07:44.654394+0000","last_peered":"2024-04-16T21:07:44.654394+0000","last_clean":"2024-04-16T21:07:44.654394+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:07:44.654394+0000","last_undegraded":"2024-04-16T21:07:44.654394+0000","last_fullsized":"2024-04-16T21:07:44.654394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:57:15.042193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1334,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652729+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:07:39.652729+0000","last_peered":"2024-04-16T21:07:39.652729+0000","last_clean":"2024-04-16T21:07:39.652729+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:07:39.652729+0000","last_undegraded":"2024-04-16T21:07:39.652729+0000","last_fullsized":"2024-04-16T21:07:39.652729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:24:02.023550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.370324+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:09:14.370324+0000","last_peered":"2024-04-16T21:09:14.370324+0000","last_clean":"2024-04-16T21:09:14.370324+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:09:14.370324+0000","last_undegraded":"2024-04-16T21:09:14.370324+0000","last_fullsized":"2024-04-16T21:09:14.370324+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'3","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.492346+0000","last_change":"2024-04-16T19:25:15.483924+0000","last_active":"2024-04-16T21:07:24.492346+0000","last_peered":"2024-04-16T21:07:24.492346+0000","last_clean":"2024-04-16T21:07:24.492346+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:07:24.492346+0000","last_undegraded":"2024-04-16T21:07:24.492346+0000","last_fullsized":"2024-04-16T21:07:24.492346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:57.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'705","reported_seq":2159,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505330+0000","last_change":"2024-04-16T19:25:13.475694+0000","last_active":"2024-04-16T21:08:44.505330+0000","last_peered":"2024-04-16T21:08:44.505330+0000","last_clean":"2024-04-16T21:08:44.505330+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:08:44.505330+0000","last_undegraded":"2024-04-16T21:08:44.505330+0000","last_fullsized":"2024-04-16T21:08:44.505330+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:31:56.172757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'247","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364687+0000","last_change":"2024-04-16T19:25:11.466002+0000","last_active":"2024-04-16T21:08:44.364687+0000","last_peered":"2024-04-16T21:08:44.364687+0000","last_clean":"2024-04-16T21:08:44.364687+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:08:44.364687+0000","last_undegraded":"2024-04-16T21:08:44.364687+0000","last_fullsized":"2024-04-16T21:08:44.364687+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:38.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'27","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367274+0000","last_change":"2024-04-16T19:25:09.469480+0000","last_active":"2024-04-16T21:08:59.367274+0000","last_peered":"2024-04-16T21:08:59.367274+0000","last_clean":"2024-04-16T21:08:59.367274+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:08:59.367274+0000","last_undegraded":"2024-04-16T21:08:59.367274+0000","last_fullsized":"2024-04-16T21:08:59.367274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:00.355585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364874+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:08:49.364874+0000","last_peered":"2024-04-16T21:08:49.364874+0000","last_clean":"2024-04-16T21:08:49.364874+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:08:49.364874+0000","last_undegraded":"2024-04-16T21:08:49.364874+0000","last_fullsized":"2024-04-16T21:08:49.364874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:25.878014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368069+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:09:04.368069+0000","last_peered":"2024-04-16T21:09:04.368069+0000","last_clean":"2024-04-16T21:09:04.368069+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:09:04.368069+0000","last_undegraded":"2024-04-16T21:09:04.368069+0000","last_fullsized":"2024-04-16T21:09:04.368069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498140+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:07:59.498140+0000","last_peered":"2024-04-16T21:07:59.498140+0000","last_clean":"2024-04-16T21:07:59.498140+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:07:59.498140+0000","last_undegraded":"2024-04-16T21:07:59.498140+0000","last_fullsized":"2024-04-16T21:07:59.498140+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:57:33.584949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372342+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:08:59.372342+0000","last_peered":"2024-04-16T21:08:59.372342+0000","last_clean":"2024-04-16T21:08:59.372342+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:08:59.372342+0000","last_undegraded":"2024-04-16T21:08:59.372342+0000","last_fullsized":"2024-04-16T21:08:59.372342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:41:38.135155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":349,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373804+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:09:09.373804+0000","last_peered":"2024-04-16T21:09:09.373804+0000","last_clean":"2024-04-16T21:09:09.373804+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:09:09.373804+0000","last_undegraded":"2024-04-16T21:09:09.373804+0000","last_fullsized":"2024-04-16T21:09:09.373804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:13:42.890109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367113+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:08:59.367113+0000","last_peered":"2024-04-16T21:08:59.367113+0000","last_clean":"2024-04-16T21:08:59.367113+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:08:59.367113+0000","last_undegraded":"2024-04-16T21:08:59.367113+0000","last_fullsized":"2024-04-16T21:08:59.367113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:50.295932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660391+0000","last_change":"2024-04-16T19:24:51.572391+0000","last_active":"2024-04-16T21:08:24.660391+0000","last_peered":"2024-04-16T21:08:24.660391+0000","last_clean":"2024-04-16T21:08:24.660391+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:08:24.660391+0000","last_undegraded":"2024-04-16T21:08:24.660391+0000","last_fullsized":"2024-04-16T21:08:24.660391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:49:04.095435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495516+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:07:44.495516+0000","last_peered":"2024-04-16T21:07:44.495516+0000","last_clean":"2024-04-16T21:07:44.495516+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:07:44.495516+0000","last_undegraded":"2024-04-16T21:07:44.495516+0000","last_fullsized":"2024-04-16T21:07:44.495516+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:34.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2002,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.653496+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:07:39.653496+0000","last_peered":"2024-04-16T21:07:39.653496+0000","last_clean":"2024-04-16T21:07:39.653496+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:07:39.653496+0000","last_undegraded":"2024-04-16T21:07:39.653496+0000","last_fullsized":"2024-04-16T21:07:39.653496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:04:06.107623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496500+0000","last_change":"2024-04-16T19:25:41.510087+0000","last_active":"2024-04-16T21:07:49.496500+0000","last_peered":"2024-04-16T21:07:49.496500+0000","last_clean":"2024-04-16T21:07:49.496500+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:07:49.496500+0000","last_undegraded":"2024-04-16T21:07:49.496500+0000","last_fullsized":"2024-04-16T21:07:49.496500+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:45.367111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'23","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370491+0000","last_change":"2024-04-16T19:25:15.487252+0000","last_active":"2024-04-16T21:08:49.370491+0000","last_peered":"2024-04-16T21:08:49.370491+0000","last_clean":"2024-04-16T21:08:49.370491+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:08:49.370491+0000","last_undegraded":"2024-04-16T21:08:49.370491+0000","last_fullsized":"2024-04-16T21:08:49.370491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:14.101479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364063+0000","last_change":"2024-04-16T19:25:13.475862+0000","last_active":"2024-04-16T21:08:44.364063+0000","last_peered":"2024-04-16T21:08:44.364063+0000","last_clean":"2024-04-16T21:08:44.364063+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:08:44.364063+0000","last_undegraded":"2024-04-16T21:08:44.364063+0000","last_fullsized":"2024-04-16T21:08:44.364063+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:47.410023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'714","reported_seq":1574,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364715+0000","last_change":"2024-04-16T19:25:11.465490+0000","last_active":"2024-04-16T21:08:44.364715+0000","last_peered":"2024-04-16T21:08:44.364715+0000","last_clean":"2024-04-16T21:08:44.364715+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:08:44.364715+0000","last_undegraded":"2024-04-16T21:08:44.364715+0000","last_fullsized":"2024-04-16T21:08:44.364715+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:03.668322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'6","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664395+0000","last_change":"2024-04-16T19:25:09.463391+0000","last_active":"2024-04-16T21:08:39.664395+0000","last_peered":"2024-04-16T21:08:39.664395+0000","last_clean":"2024-04-16T21:08:39.664395+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:08:39.664395+0000","last_undegraded":"2024-04-16T21:08:39.664395+0000","last_fullsized":"2024-04-16T21:08:39.664395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.370504+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:08:49.370504+0000","last_peered":"2024-04-16T21:08:49.370504+0000","last_clean":"2024-04-16T21:08:49.370504+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:08:49.370504+0000","last_undegraded":"2024-04-16T21:08:49.370504+0000","last_fullsized":"2024-04-16T21:08:49.370504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:36.034515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364075+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:08:44.364075+0000","last_peered":"2024-04-16T21:08:44.364075+0000","last_clean":"2024-04-16T21:08:44.364075+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:08:44.364075+0000","last_undegraded":"2024-04-16T21:08:44.364075+0000","last_fullsized":"2024-04-16T21:08:44.364075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.660502+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:08:14.660502+0000","last_peered":"2024-04-16T21:08:14.660502+0000","last_clean":"2024-04-16T21:08:14.660502+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:08:14.660502+0000","last_undegraded":"2024-04-16T21:08:14.660502+0000","last_fullsized":"2024-04-16T21:08:14.660502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:07.162476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367144+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:08:59.367144+0000","last_peered":"2024-04-16T21:08:59.367144+0000","last_clean":"2024-04-16T21:08:59.367144+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:08:59.367144+0000","last_undegraded":"2024-04-16T21:08:59.367144+0000","last_fullsized":"2024-04-16T21:08:59.367144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:27.238344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":346,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.366566+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:08:59.366566+0000","last_peered":"2024-04-16T21:08:59.366566+0000","last_clean":"2024-04-16T21:08:59.366566+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:08:59.366566+0000","last_undegraded":"2024-04-16T21:08:59.366566+0000","last_fullsized":"2024-04-16T21:08:59.366566+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:56:36.905847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1317,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368814+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:09:09.368814+0000","last_peered":"2024-04-16T21:09:09.368814+0000","last_clean":"2024-04-16T21:09:09.368814+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:09:09.368814+0000","last_undegraded":"2024-04-16T21:09:09.368814+0000","last_fullsized":"2024-04-16T21:09:09.368814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:34.836984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662820+0000","last_change":"2024-04-16T19:24:51.572638+0000","last_active":"2024-04-16T21:08:39.662820+0000","last_peered":"2024-04-16T21:08:39.662820+0000","last_clean":"2024-04-16T21:08:39.662820+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:08:39.662820+0000","last_undegraded":"2024-04-16T21:08:39.662820+0000","last_fullsized":"2024-04-16T21:08:39.662820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:13.926597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503462+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:08:34.503462+0000","last_peered":"2024-04-16T21:08:34.503462+0000","last_clean":"2024-04-16T21:08:34.503462+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:08:34.503462+0000","last_undegraded":"2024-04-16T21:08:34.503462+0000","last_fullsized":"2024-04-16T21:08:34.503462+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:20.743689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1526,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656812+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:07:54.656812+0000","last_peered":"2024-04-16T21:07:54.656812+0000","last_clean":"2024-04-16T21:07:54.656812+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:07:54.656812+0000","last_undegraded":"2024-04-16T21:07:54.656812+0000","last_fullsized":"2024-04-16T21:07:54.656812+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:25.577697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652730+0000","last_change":"2024-04-16T19:25:41.498506+0000","last_active":"2024-04-16T21:07:34.652730+0000","last_peered":"2024-04-16T21:07:34.652730+0000","last_clean":"2024-04-16T21:07:34.652730+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:07:34.652730+0000","last_undegraded":"2024-04-16T21:07:34.652730+0000","last_fullsized":"2024-04-16T21:07:34.652730+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:04:14.970337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'2","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510638+0000","last_change":"2024-04-16T19:25:15.483677+0000","last_active":"2024-04-16T21:09:19.510638+0000","last_peered":"2024-04-16T21:09:19.510638+0000","last_clean":"2024-04-16T21:09:19.510638+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:09:19.510638+0000","last_undegraded":"2024-04-16T21:09:19.510638+0000","last_fullsized":"2024-04-16T21:09:19.510638+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368793+0000","last_change":"2024-04-16T19:25:13.473234+0000","last_active":"2024-04-16T21:09:04.368793+0000","last_peered":"2024-04-16T21:09:04.368793+0000","last_clean":"2024-04-16T21:09:04.368793+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:09:04.368793+0000","last_undegraded":"2024-04-16T21:09:04.368793+0000","last_fullsized":"2024-04-16T21:09:04.368793+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:10.112989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'267","reported_seq":524,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364765+0000","last_change":"2024-04-16T19:25:11.463707+0000","last_active":"2024-04-16T21:08:44.364765+0000","last_peered":"2024-04-16T21:08:44.364765+0000","last_clean":"2024-04-16T21:08:44.364765+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:08:44.364765+0000","last_undegraded":"2024-04-16T21:08:44.364765+0000","last_fullsized":"2024-04-16T21:08:44.364765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:51.610204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'14","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664322+0000","last_change":"2024-04-16T19:25:09.461719+0000","last_active":"2024-04-16T21:08:39.664322+0000","last_peered":"2024-04-16T21:08:39.664322+0000","last_clean":"2024-04-16T21:08:39.664322+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:08:39.664322+0000","last_undegraded":"2024-04-16T21:08:39.664322+0000","last_fullsized":"2024-04-16T21:08:39.664322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:24.102149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372250+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:08:59.372250+0000","last_peered":"2024-04-16T21:08:59.372250+0000","last_clean":"2024-04-16T21:08:59.372250+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:08:59.372250+0000","last_undegraded":"2024-04-16T21:08:59.372250+0000","last_fullsized":"2024-04-16T21:08:59.372250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:01.938758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503593+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:08:34.503593+0000","last_peered":"2024-04-16T21:08:34.503593+0000","last_clean":"2024-04-16T21:08:34.503593+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:08:34.503593+0000","last_undegraded":"2024-04-16T21:08:34.503593+0000","last_fullsized":"2024-04-16T21:08:34.503593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.495661+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:07:44.495661+0000","last_peered":"2024-04-16T21:07:44.495661+0000","last_clean":"2024-04-16T21:07:44.495661+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:07:44.495661+0000","last_undegraded":"2024-04-16T21:07:44.495661+0000","last_fullsized":"2024-04-16T21:07:44.495661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:23:46.270000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367951+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:08:59.367951+0000","last_peered":"2024-04-16T21:08:59.367951+0000","last_clean":"2024-04-16T21:08:59.367951+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:08:59.367951+0000","last_undegraded":"2024-04-16T21:08:59.367951+0000","last_fullsized":"2024-04-16T21:08:59.367951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:48.749243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":280,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367980+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:08:59.367980+0000","last_peered":"2024-04-16T21:08:59.367980+0000","last_clean":"2024-04-16T21:08:59.367980+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:08:59.367980+0000","last_undegraded":"2024-04-16T21:08:59.367980+0000","last_fullsized":"2024-04-16T21:08:59.367980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372321+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:08:59.372321+0000","last_peered":"2024-04-16T21:08:59.372321+0000","last_clean":"2024-04-16T21:08:59.372321+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:08:59.372321+0000","last_undegraded":"2024-04-16T21:08:59.372321+0000","last_fullsized":"2024-04-16T21:08:59.372321+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660333+0000","last_change":"2024-04-16T19:24:51.573460+0000","last_active":"2024-04-16T21:08:24.660333+0000","last_peered":"2024-04-16T21:08:24.660333+0000","last_clean":"2024-04-16T21:08:24.660333+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:08:24.660333+0000","last_undegraded":"2024-04-16T21:08:24.660333+0000","last_fullsized":"2024-04-16T21:08:24.660333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:09:31.585745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498095+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:07:59.498095+0000","last_peered":"2024-04-16T21:07:59.498095+0000","last_clean":"2024-04-16T21:07:59.498095+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:07:59.498095+0000","last_undegraded":"2024-04-16T21:07:59.498095+0000","last_fullsized":"2024-04-16T21:07:59.498095+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:33:27.173018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1521,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657201+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:07:59.657201+0000","last_peered":"2024-04-16T21:07:59.657201+0000","last_clean":"2024-04-16T21:07:59.657201+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:07:59.657201+0000","last_undegraded":"2024-04-16T21:07:59.657201+0000","last_fullsized":"2024-04-16T21:07:59.657201+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:21:16.733301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.493948+0000","last_change":"2024-04-16T19:25:41.502910+0000","last_active":"2024-04-16T21:07:34.493948+0000","last_peered":"2024-04-16T21:07:34.493948+0000","last_clean":"2024-04-16T21:07:34.493948+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:07:34.493948+0000","last_undegraded":"2024-04-16T21:07:34.493948+0000","last_fullsized":"2024-04-16T21:07:34.493948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:58:25.621893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'12","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510709+0000","last_change":"2024-04-16T19:25:15.485965+0000","last_active":"2024-04-16T21:09:19.510709+0000","last_peered":"2024-04-16T21:09:19.510709+0000","last_clean":"2024-04-16T21:09:19.510709+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:09:19.510709+0000","last_undegraded":"2024-04-16T21:09:19.510709+0000","last_fullsized":"2024-04-16T21:09:19.510709+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:59.701088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'705","reported_seq":2080,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364751+0000","last_change":"2024-04-16T19:25:13.471357+0000","last_active":"2024-04-16T21:08:44.364751+0000","last_peered":"2024-04-16T21:08:44.364751+0000","last_clean":"2024-04-16T21:08:44.364751+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:08:44.364751+0000","last_undegraded":"2024-04-16T21:08:44.364751+0000","last_fullsized":"2024-04-16T21:08:44.364751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:10.156733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'318","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368826+0000","last_change":"2024-04-16T19:25:11.461881+0000","last_active":"2024-04-16T21:09:04.368826+0000","last_peered":"2024-04-16T21:09:04.368826+0000","last_clean":"2024-04-16T21:09:04.368826+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:09:04.368826+0000","last_undegraded":"2024-04-16T21:09:04.368826+0000","last_fullsized":"2024-04-16T21:09:04.368826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:55:13.030219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'6","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504516+0000","last_change":"2024-04-16T19:25:09.461256+0000","last_active":"2024-04-16T21:08:39.504516+0000","last_peered":"2024-04-16T21:08:39.504516+0000","last_clean":"2024-04-16T21:08:39.504516+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:08:39.504516+0000","last_undegraded":"2024-04-16T21:08:39.504516+0000","last_fullsized":"2024-04-16T21:08:39.504516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:56.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365397+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:08:49.365397+0000","last_peered":"2024-04-16T21:08:49.365397+0000","last_clean":"2024-04-16T21:08:49.365397+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:08:49.365397+0000","last_undegraded":"2024-04-16T21:08:49.365397+0000","last_fullsized":"2024-04-16T21:08:49.365397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:21.322076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.662064+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:08:34.662064+0000","last_peered":"2024-04-16T21:08:34.662064+0000","last_clean":"2024-04-16T21:08:34.662064+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:08:34.662064+0000","last_undegraded":"2024-04-16T21:08:34.662064+0000","last_fullsized":"2024-04-16T21:08:34.662064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:41:18.197010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.656111+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:07:49.656111+0000","last_peered":"2024-04-16T21:07:49.656111+0000","last_clean":"2024-04-16T21:07:49.656111+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:07:49.656111+0000","last_undegraded":"2024-04-16T21:07:49.656111+0000","last_fullsized":"2024-04-16T21:07:49.656111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:42.139799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508363+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:09:04.508363+0000","last_peered":"2024-04-16T21:09:04.508363+0000","last_clean":"2024-04-16T21:09:04.508363+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:09:04.508363+0000","last_undegraded":"2024-04-16T21:09:04.508363+0000","last_fullsized":"2024-04-16T21:09:04.508363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:28:03.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":331,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367176+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:08:59.367176+0000","last_peered":"2024-04-16T21:08:59.367176+0000","last_clean":"2024-04-16T21:08:59.367176+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:08:59.367176+0000","last_undegraded":"2024-04-16T21:08:59.367176+0000","last_fullsized":"2024-04-16T21:08:59.367176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:23.723705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507550+0000","last_change":"2024-04-16T19:24:49.435265+0000","last_active":"2024-04-16T21:08:59.507550+0000","last_peered":"2024-04-16T21:08:59.507550+0000","last_clean":"2024-04-16T21:08:59.507550+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:08:59.507550+0000","last_undegraded":"2024-04-16T21:08:59.507550+0000","last_fullsized":"2024-04-16T21:08:59.507550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:08.160472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661329+0000","last_change":"2024-04-16T19:24:51.438071+0000","last_active":"2024-04-16T21:08:24.661329+0000","last_peered":"2024-04-16T21:08:24.661329+0000","last_clean":"2024-04-16T21:08:24.661329+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:08:24.661329+0000","last_undegraded":"2024-04-16T21:08:24.661329+0000","last_fullsized":"2024-04-16T21:08:24.661329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:37:42.181631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649582+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:07:19.649582+0000","last_peered":"2024-04-16T21:07:19.649582+0000","last_clean":"2024-04-16T21:07:19.649582+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:07:19.649582+0000","last_undegraded":"2024-04-16T21:07:19.649582+0000","last_fullsized":"2024-04-16T21:07:19.649582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:34:01.244792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653404+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:07:34.653404+0000","last_peered":"2024-04-16T21:07:34.653404+0000","last_clean":"2024-04-16T21:07:34.653404+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:07:34.653404+0000","last_undegraded":"2024-04-16T21:07:34.653404+0000","last_fullsized":"2024-04-16T21:07:34.653404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:52:03.502015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.500263+0000","last_change":"2024-04-16T19:25:41.506596+0000","last_active":"2024-04-16T21:08:14.500263+0000","last_peered":"2024-04-16T21:08:14.500263+0000","last_clean":"2024-04-16T21:08:14.500263+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:08:14.500263+0000","last_undegraded":"2024-04-16T21:08:14.500263+0000","last_fullsized":"2024-04-16T21:08:14.500263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:01.639245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'4","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655215+0000","last_change":"2024-04-16T19:25:15.480477+0000","last_active":"2024-04-16T21:07:49.655215+0000","last_peered":"2024-04-16T21:07:49.655215+0000","last_clean":"2024-04-16T21:07:49.655215+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:07:49.655215+0000","last_undegraded":"2024-04-16T21:07:49.655215+0000","last_fullsized":"2024-04-16T21:07:49.655215+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:58.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654540+0000","last_change":"2024-04-16T19:25:13.470985+0000","last_active":"2024-04-16T21:07:44.654540+0000","last_peered":"2024-04-16T21:07:44.654540+0000","last_clean":"2024-04-16T21:07:44.654540+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:07:44.654540+0000","last_undegraded":"2024-04-16T21:07:44.654540+0000","last_fullsized":"2024-04-16T21:07:44.654540+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:54.589907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'292","reported_seq":577,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654437+0000","last_change":"2024-04-16T19:25:11.464321+0000","last_active":"2024-04-16T21:07:44.654437+0000","last_peered":"2024-04-16T21:07:44.654437+0000","last_clean":"2024-04-16T21:07:44.654437+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:07:44.654437+0000","last_undegraded":"2024-04-16T21:07:44.654437+0000","last_fullsized":"2024-04-16T21:07:44.654437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:17.793032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'13","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.377356+0000","last_change":"2024-04-16T19:25:09.460391+0000","last_active":"2024-04-16T21:09:29.377356+0000","last_peered":"2024-04-16T21:09:29.377356+0000","last_clean":"2024-04-16T21:09:29.377356+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:09:29.377356+0000","last_undegraded":"2024-04-16T21:09:29.377356+0000","last_fullsized":"2024-04-16T21:09:29.377356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373217+0000","last_change":"2024-04-16T19:24:51.434685+0000","last_active":"2024-04-16T21:09:04.373217+0000","last_peered":"2024-04-16T21:09:04.373217+0000","last_clean":"2024-04-16T21:09:04.373217+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:09:04.373217+0000","last_undegraded":"2024-04-16T21:09:04.373217+0000","last_fullsized":"2024-04-16T21:09:04.373217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:57:30.733796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650563+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:07:19.650563+0000","last_peered":"2024-04-16T21:07:19.650563+0000","last_clean":"2024-04-16T21:07:19.650563+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:07:19.650563+0000","last_undegraded":"2024-04-16T21:07:19.650563+0000","last_fullsized":"2024-04-16T21:07:19.650563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:28.749905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504494+0000","last_change":"2024-04-16T19:25:41.504766+0000","last_active":"2024-04-16T21:08:39.504494+0000","last_peered":"2024-04-16T21:08:39.504494+0000","last_clean":"2024-04-16T21:08:39.504494+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:08:39.504494+0000","last_undegraded":"2024-04-16T21:08:39.504494+0000","last_fullsized":"2024-04-16T21:08:39.504494+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:37.786953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":743,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493189+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:07:29.493189+0000","last_peered":"2024-04-16T21:07:29.493189+0000","last_clean":"2024-04-16T21:07:29.493189+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:07:29.493189+0000","last_undegraded":"2024-04-16T21:07:29.493189+0000","last_fullsized":"2024-04-16T21:07:29.493189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:14:30.438117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493117+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:07:29.493117+0000","last_peered":"2024-04-16T21:07:29.493117+0000","last_clean":"2024-04-16T21:07:29.493117+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:07:29.493117+0000","last_undegraded":"2024-04-16T21:07:29.493117+0000","last_fullsized":"2024-04-16T21:07:29.493117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:51.673404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493130+0000","last_change":"2024-04-16T19:24:49.434342+0000","last_active":"2024-04-16T21:07:29.493130+0000","last_peered":"2024-04-16T21:07:29.493130+0000","last_clean":"2024-04-16T21:07:29.493130+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:07:29.493130+0000","last_undegraded":"2024-04-16T21:07:29.493130+0000","last_fullsized":"2024-04-16T21:07:29.493130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:07.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651977+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:07:29.651977+0000","last_peered":"2024-04-16T21:07:29.651977+0000","last_clean":"2024-04-16T21:07:29.651977+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:07:29.651977+0000","last_undegraded":"2024-04-16T21:07:29.651977+0000","last_fullsized":"2024-04-16T21:07:29.651977+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:21.199802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'118","reported_seq":1711,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491510+0000","last_change":"2024-04-16T19:25:49.520399+0000","last_active":"2024-04-16T21:07:19.491510+0000","last_peered":"2024-04-16T21:07:19.491510+0000","last_clean":"2024-04-16T21:07:19.491510+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:07:19.491510+0000","last_undegraded":"2024-04-16T21:07:19.491510+0000","last_fullsized":"2024-04-16T21:07:19.491510+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:16:26.165977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659512+0000","last_change":"2024-04-16T19:25:41.505754+0000","last_active":"2024-04-16T21:08:14.659512+0000","last_peered":"2024-04-16T21:08:14.659512+0000","last_clean":"2024-04-16T21:08:14.659512+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:08:14.659512+0000","last_undegraded":"2024-04-16T21:08:14.659512+0000","last_fullsized":"2024-04-16T21:08:14.659512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:08:10.716421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655168+0000","last_change":"2024-04-16T19:25:15.482484+0000","last_active":"2024-04-16T21:07:49.655168+0000","last_peered":"2024-04-16T21:07:49.655168+0000","last_clean":"2024-04-16T21:07:49.655168+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:07:49.655168+0000","last_undegraded":"2024-04-16T21:07:49.655168+0000","last_fullsized":"2024-04-16T21:07:49.655168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:36:25.761792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.655229+0000","last_change":"2024-04-16T19:25:13.472620+0000","last_active":"2024-04-16T21:07:44.655229+0000","last_peered":"2024-04-16T21:07:44.655229+0000","last_clean":"2024-04-16T21:07:44.655229+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:07:44.655229+0000","last_undegraded":"2024-04-16T21:07:44.655229+0000","last_fullsized":"2024-04-16T21:07:44.655229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:01:28.935744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1140","reported_seq":2898,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:44.654522+0000","last_change":"2024-04-16T19:25:11.462713+0000","last_active":"2024-04-16T21:07:44.654522+0000","last_peered":"2024-04-16T21:07:44.654522+0000","last_clean":"2024-04-16T21:07:44.654522+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:07:44.654522+0000","last_undegraded":"2024-04-16T21:07:44.654522+0000","last_fullsized":"2024-04-16T21:07:44.654522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:45.401459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'9","reported_seq":192,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.369883+0000","last_change":"2024-04-16T19:25:09.466079+0000","last_active":"2024-04-16T21:09:19.369883+0000","last_peered":"2024-04-16T21:09:19.369883+0000","last_clean":"2024-04-16T21:09:19.369883+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:09:19.369883+0000","last_undegraded":"2024-04-16T21:09:19.369883+0000","last_fullsized":"2024-04-16T21:09:19.369883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369024+0000","last_change":"2024-04-16T19:24:51.442540+0000","last_active":"2024-04-16T21:09:14.369024+0000","last_peered":"2024-04-16T21:09:14.369024+0000","last_clean":"2024-04-16T21:09:14.369024+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:09:14.369024+0000","last_undegraded":"2024-04-16T21:09:14.369024+0000","last_fullsized":"2024-04-16T21:09:14.369024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491649+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:07:19.491649+0000","last_peered":"2024-04-16T21:07:19.491649+0000","last_clean":"2024-04-16T21:07:19.491649+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:07:19.491649+0000","last_undegraded":"2024-04-16T21:07:19.491649+0000","last_fullsized":"2024-04-16T21:07:19.491649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:09.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662604+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:08:29.662604+0000","last_peered":"2024-04-16T21:08:29.662604+0000","last_clean":"2024-04-16T21:08:29.662604+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:08:29.662604+0000","last_undegraded":"2024-04-16T21:08:29.662604+0000","last_fullsized":"2024-04-16T21:08:29.662604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493228+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:07:29.493228+0000","last_peered":"2024-04-16T21:07:29.493228+0000","last_clean":"2024-04-16T21:07:29.493228+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:07:29.493228+0000","last_undegraded":"2024-04-16T21:07:29.493228+0000","last_fullsized":"2024-04-16T21:07:29.493228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:10.368115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651116+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:07:29.651116+0000","last_peered":"2024-04-16T21:07:29.651116+0000","last_clean":"2024-04-16T21:07:29.651116+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:07:29.651116+0000","last_undegraded":"2024-04-16T21:07:29.651116+0000","last_fullsized":"2024-04-16T21:07:29.651116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:18:05.198319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652671+0000","last_change":"2024-04-16T19:24:49.431431+0000","last_active":"2024-04-16T21:07:29.652671+0000","last_peered":"2024-04-16T21:07:29.652671+0000","last_clean":"2024-04-16T21:07:29.652671+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:07:29.652671+0000","last_undegraded":"2024-04-16T21:07:29.652671+0000","last_fullsized":"2024-04-16T21:07:29.652671+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:35:58.171133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":49,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652772+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:07:34.652772+0000","last_peered":"2024-04-16T21:07:34.652772+0000","last_clean":"2024-04-16T21:07:34.652772+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:07:34.652772+0000","last_undegraded":"2024-04-16T21:07:34.652772+0000","last_fullsized":"2024-04-16T21:07:34.652772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:31:39.104319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1703,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.501136+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:08:19.501136+0000","last_peered":"2024-04-16T21:08:19.501136+0000","last_clean":"2024-04-16T21:08:19.501136+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:08:19.501136+0000","last_undegraded":"2024-04-16T21:08:19.501136+0000","last_fullsized":"2024-04-16T21:08:19.501136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:34:15.107893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649480+0000","last_change":"2024-04-16T19:25:41.499274+0000","last_active":"2024-04-16T21:07:14.649480+0000","last_peered":"2024-04-16T21:07:14.649480+0000","last_clean":"2024-04-16T21:07:14.649480+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:07:14.649480+0000","last_undegraded":"2024-04-16T21:07:14.649480+0000","last_fullsized":"2024-04-16T21:07:14.649480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:22:55.194832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'20","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647308+0000","last_change":"2024-04-16T19:25:15.485235+0000","last_active":"2024-04-16T21:07:04.647308+0000","last_peered":"2024-04-16T21:07:04.647308+0000","last_clean":"2024-04-16T21:07:04.647308+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:07:04.647308+0000","last_undegraded":"2024-04-16T21:07:04.647308+0000","last_fullsized":"2024-04-16T21:07:04.647308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:26.694643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.373278+0000","last_change":"2024-04-16T19:25:13.476041+0000","last_active":"2024-04-16T21:09:34.373278+0000","last_peered":"2024-04-16T21:09:34.373278+0000","last_clean":"2024-04-16T21:09:34.373278+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:09:34.373278+0000","last_undegraded":"2024-04-16T21:09:34.373278+0000","last_fullsized":"2024-04-16T21:09:34.373278+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:17.872253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'274","reported_seq":578,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.371939+0000","last_change":"2024-04-16T19:25:11.462598+0000","last_active":"2024-04-16T21:09:24.371939+0000","last_peered":"2024-04-16T21:09:24.371939+0000","last_clean":"2024-04-16T21:09:24.371939+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:09:24.371939+0000","last_undegraded":"2024-04-16T21:09:24.371939+0000","last_fullsized":"2024-04-16T21:09:24.371939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:55.581849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'8","reported_seq":245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652902+0000","last_change":"2024-04-16T19:25:09.461313+0000","last_active":"2024-04-16T21:07:39.652902+0000","last_peered":"2024-04-16T21:07:39.652902+0000","last_clean":"2024-04-16T21:07:39.652902+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:07:39.652902+0000","last_undegraded":"2024-04-16T21:07:39.652902+0000","last_fullsized":"2024-04-16T21:07:39.652902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.650513+0000","last_change":"2024-04-16T19:24:51.435662+0000","last_active":"2024-04-16T21:07:24.650513+0000","last_peered":"2024-04-16T21:07:24.650513+0000","last_clean":"2024-04-16T21:07:24.650513+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:07:24.650513+0000","last_undegraded":"2024-04-16T21:07:24.650513+0000","last_fullsized":"2024-04-16T21:07:24.650513+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:57.768573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491559+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:07:19.491559+0000","last_peered":"2024-04-16T21:07:19.491559+0000","last_clean":"2024-04-16T21:07:19.491559+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:07:19.491559+0000","last_undegraded":"2024-04-16T21:07:19.491559+0000","last_fullsized":"2024-04-16T21:07:19.491559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:46:59.602165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.662208+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:08:29.662208+0000","last_peered":"2024-04-16T21:08:29.662208+0000","last_clean":"2024-04-16T21:08:29.662208+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:08:29.662208+0000","last_undegraded":"2024-04-16T21:08:29.662208+0000","last_fullsized":"2024-04-16T21:08:29.662208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:56:00.012595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":611,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651931+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:07:29.651931+0000","last_peered":"2024-04-16T21:07:29.651931+0000","last_clean":"2024-04-16T21:07:29.651931+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:07:29.651931+0000","last_undegraded":"2024-04-16T21:07:29.651931+0000","last_fullsized":"2024-04-16T21:07:29.651931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:22:01.385906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'96","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493180+0000","last_change":"2024-04-16T19:24:47.424747+0000","last_active":"2024-04-16T21:07:29.493180+0000","last_peered":"2024-04-16T21:07:29.493180+0000","last_clean":"2024-04-16T21:07:29.493180+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:07:29.493180+0000","last_undegraded":"2024-04-16T21:07:29.493180+0000","last_fullsized":"2024-04-16T21:07:29.493180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:48.064472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652026+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:07:29.652026+0000","last_peered":"2024-04-16T21:07:29.652026+0000","last_clean":"2024-04-16T21:07:29.652026+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:07:29.652026+0000","last_undegraded":"2024-04-16T21:07:29.652026+0000","last_fullsized":"2024-04-16T21:07:29.652026+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:21.377903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653531+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:07:34.653531+0000","last_peered":"2024-04-16T21:07:34.653531+0000","last_clean":"2024-04-16T21:07:34.653531+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:07:34.653531+0000","last_undegraded":"2024-04-16T21:07:34.653531+0000","last_fullsized":"2024-04-16T21:07:34.653531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.659404+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:08:19.659404+0000","last_peered":"2024-04-16T21:08:19.659404+0000","last_clean":"2024-04-16T21:08:19.659404+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:08:19.659404+0000","last_undegraded":"2024-04-16T21:08:19.659404+0000","last_fullsized":"2024-04-16T21:08:19.659404+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:52.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.493079+0000","last_change":"2024-04-16T19:25:41.508866+0000","last_active":"2024-04-16T21:07:29.493079+0000","last_peered":"2024-04-16T21:07:29.493079+0000","last_clean":"2024-04-16T21:07:29.493079+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:07:29.493079+0000","last_undegraded":"2024-04-16T21:07:29.493079+0000","last_fullsized":"2024-04-16T21:07:29.493079+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:43:29.426956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'22","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368844+0000","last_change":"2024-04-16T19:25:15.480626+0000","last_active":"2024-04-16T21:09:14.368844+0000","last_peered":"2024-04-16T21:09:14.368844+0000","last_clean":"2024-04-16T21:09:14.368844+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:09:14.368844+0000","last_undegraded":"2024-04-16T21:09:14.368844+0000","last_fullsized":"2024-04-16T21:09:14.368844+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:39.898263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'705","reported_seq":2141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:24.511688+0000","last_change":"2024-04-16T19:25:13.473351+0000","last_active":"2024-04-16T21:09:24.511688+0000","last_peered":"2024-04-16T21:09:24.511688+0000","last_clean":"2024-04-16T21:09:24.511688+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:09:24.511688+0000","last_undegraded":"2024-04-16T21:09:24.511688+0000","last_fullsized":"2024-04-16T21:09:24.511688+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:47:04.370192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1017","reported_seq":1517,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.378182+0000","last_change":"2024-04-16T19:25:11.466163+0000","last_active":"2024-04-16T21:09:34.378182+0000","last_peered":"2024-04-16T21:09:34.378182+0000","last_clean":"2024-04-16T21:09:34.378182+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:09:34.378182+0000","last_undegraded":"2024-04-16T21:09:34.378182+0000","last_fullsized":"2024-04-16T21:09:34.378182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:07:40.676377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'14","reported_seq":184,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652919+0000","last_change":"2024-04-16T19:25:09.467909+0000","last_active":"2024-04-16T21:07:39.652919+0000","last_peered":"2024-04-16T21:07:39.652919+0000","last_clean":"2024-04-16T21:07:39.652919+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:07:39.652919+0000","last_undegraded":"2024-04-16T21:07:39.652919+0000","last_fullsized":"2024-04-16T21:07:39.652919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:13:05.530059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:24.651388+0000","last_change":"2024-04-16T19:24:51.438772+0000","last_active":"2024-04-16T21:07:24.651388+0000","last_peered":"2024-04-16T21:07:24.651388+0000","last_clean":"2024-04-16T21:07:24.651388+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:07:24.651388+0000","last_undegraded":"2024-04-16T21:07:24.651388+0000","last_fullsized":"2024-04-16T21:07:24.651388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:00.179010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491555+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:07:19.491555+0000","last_peered":"2024-04-16T21:07:19.491555+0000","last_clean":"2024-04-16T21:07:19.491555+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:07:19.491555+0000","last_undegraded":"2024-04-16T21:07:19.491555+0000","last_fullsized":"2024-04-16T21:07:19.491555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:36:31.706480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661253+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:08:29.661253+0000","last_peered":"2024-04-16T21:08:29.661253+0000","last_clean":"2024-04-16T21:08:29.661253+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:08:29.661253+0000","last_undegraded":"2024-04-16T21:08:29.661253+0000","last_fullsized":"2024-04-16T21:08:29.661253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:33.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652693+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:07:29.652693+0000","last_peered":"2024-04-16T21:07:29.652693+0000","last_clean":"2024-04-16T21:07:29.652693+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:07:29.652693+0000","last_undegraded":"2024-04-16T21:07:29.652693+0000","last_fullsized":"2024-04-16T21:07:29.652693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.652054+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:07:29.652054+0000","last_peered":"2024-04-16T21:07:29.652054+0000","last_clean":"2024-04-16T21:07:29.652054+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:07:29.652054+0000","last_undegraded":"2024-04-16T21:07:29.652054+0000","last_fullsized":"2024-04-16T21:07:29.652054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:20:52.547347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:29.651195+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:07:29.651195+0000","last_peered":"2024-04-16T21:07:29.651195+0000","last_clean":"2024-04-16T21:07:29.651195+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:07:29.651195+0000","last_undegraded":"2024-04-16T21:07:29.651195+0000","last_fullsized":"2024-04-16T21:07:29.651195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:14.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648537+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:07:04.648537+0000","last_peered":"2024-04-16T21:07:04.648537+0000","last_clean":"2024-04-16T21:07:04.648537+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:07:04.648537+0000","last_undegraded":"2024-04-16T21:07:04.648537+0000","last_fullsized":"2024-04-16T21:07:04.648537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:12:37.665943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2028,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651065+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:07:19.651065+0000","last_peered":"2024-04-16T21:07:19.651065+0000","last_clean":"2024-04-16T21:07:19.651065+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:07:19.651065+0000","last_undegraded":"2024-04-16T21:07:19.651065+0000","last_fullsized":"2024-04-16T21:07:19.651065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:58:48.277899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.648877+0000","last_change":"2024-04-16T19:25:41.507140+0000","last_active":"2024-04-16T21:07:14.648877+0000","last_peered":"2024-04-16T21:07:14.648877+0000","last_clean":"2024-04-16T21:07:14.648877+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:07:14.648877+0000","last_undegraded":"2024-04-16T21:07:14.648877+0000","last_fullsized":"2024-04-16T21:07:14.648877+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:14:47.746067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'5","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369849+0000","last_change":"2024-04-16T19:25:15.481140+0000","last_active":"2024-04-16T21:09:14.369849+0000","last_peered":"2024-04-16T21:09:14.369849+0000","last_clean":"2024-04-16T21:09:14.369849+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:09:14.369849+0000","last_undegraded":"2024-04-16T21:09:14.369849+0000","last_fullsized":"2024-04-16T21:09:14.369849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:40:22.980049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655276+0000","last_change":"2024-04-16T19:25:13.476134+0000","last_active":"2024-04-16T21:07:54.655276+0000","last_peered":"2024-04-16T21:07:54.655276+0000","last_clean":"2024-04-16T21:07:54.655276+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:07:54.655276+0000","last_undegraded":"2024-04-16T21:07:54.655276+0000","last_fullsized":"2024-04-16T21:07:54.655276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:45.900958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'305","reported_seq":615,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656185+0000","last_change":"2024-04-16T19:25:11.465819+0000","last_active":"2024-04-16T21:07:54.656185+0000","last_peered":"2024-04-16T21:07:54.656185+0000","last_clean":"2024-04-16T21:07:54.656185+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:07:54.656185+0000","last_undegraded":"2024-04-16T21:07:54.656185+0000","last_fullsized":"2024-04-16T21:07:54.656185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:38.882914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'6","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655341+0000","last_change":"2024-04-16T19:25:09.469614+0000","last_active":"2024-04-16T21:07:49.655341+0000","last_peered":"2024-04-16T21:07:49.655341+0000","last_clean":"2024-04-16T21:07:49.655341+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:07:49.655341+0000","last_undegraded":"2024-04-16T21:07:49.655341+0000","last_fullsized":"2024-04-16T21:07:49.655341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:56.484702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366575+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:08:49.366575+0000","last_peered":"2024-04-16T21:08:49.366575+0000","last_clean":"2024-04-16T21:08:49.366575+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:08:49.366575+0000","last_undegraded":"2024-04-16T21:08:49.366575+0000","last_fullsized":"2024-04-16T21:08:49.366575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:54.123433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502246+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:08:29.502246+0000","last_peered":"2024-04-16T21:08:29.502246+0000","last_clean":"2024-04-16T21:08:29.502246+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:08:29.502246+0000","last_undegraded":"2024-04-16T21:08:29.502246+0000","last_fullsized":"2024-04-16T21:08:29.502246+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:19:16.213346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.374003+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:09:09.374003+0000","last_peered":"2024-04-16T21:09:09.374003+0000","last_clean":"2024-04-16T21:09:09.374003+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:09:09.374003+0000","last_undegraded":"2024-04-16T21:09:09.374003+0000","last_fullsized":"2024-04-16T21:09:09.374003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:44:56.748010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491759+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:07:19.491759+0000","last_peered":"2024-04-16T21:07:19.491759+0000","last_clean":"2024-04-16T21:07:19.491759+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:07:19.491759+0000","last_undegraded":"2024-04-16T21:07:19.491759+0000","last_fullsized":"2024-04-16T21:07:19.491759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:38:50.759439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651037+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:07:19.651037+0000","last_peered":"2024-04-16T21:07:19.651037+0000","last_clean":"2024-04-16T21:07:19.651037+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:07:19.651037+0000","last_undegraded":"2024-04-16T21:07:19.651037+0000","last_fullsized":"2024-04-16T21:07:19.651037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:08.853429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.494045+0000","last_change":"2024-04-16T19:24:51.574221+0000","last_active":"2024-04-16T21:07:34.494045+0000","last_peered":"2024-04-16T21:07:34.494045+0000","last_clean":"2024-04-16T21:07:34.494045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:07:34.494045+0000","last_undegraded":"2024-04-16T21:07:34.494045+0000","last_fullsized":"2024-04-16T21:07:34.494045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:09.678742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650574+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:07:19.650574+0000","last_peered":"2024-04-16T21:07:19.650574+0000","last_clean":"2024-04-16T21:07:19.650574+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:07:19.650574+0000","last_undegraded":"2024-04-16T21:07:19.650574+0000","last_fullsized":"2024-04-16T21:07:19.650574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:44.374493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1826,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365476+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:08:49.365476+0000","last_peered":"2024-04-16T21:08:49.365476+0000","last_clean":"2024-04-16T21:08:49.365476+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:08:49.365476+0000","last_undegraded":"2024-04-16T21:08:49.365476+0000","last_fullsized":"2024-04-16T21:08:49.365476+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:20:09.652050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649723+0000","last_change":"2024-04-16T19:25:41.504184+0000","last_active":"2024-04-16T21:07:14.649723+0000","last_peered":"2024-04-16T21:07:14.649723+0000","last_clean":"2024-04-16T21:07:14.649723+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:07:14.649723+0000","last_undegraded":"2024-04-16T21:07:14.649723+0000","last_fullsized":"2024-04-16T21:07:14.649723+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:53:41.995945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369969+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:09:14.369969+0000","last_peered":"2024-04-16T21:09:14.369969+0000","last_clean":"2024-04-16T21:09:14.369969+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:09:14.369969+0000","last_undegraded":"2024-04-16T21:09:14.369969+0000","last_fullsized":"2024-04-16T21:09:14.369969+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:47:48.700988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497245+0000","last_change":"2024-04-16T19:25:13.472015+0000","last_active":"2024-04-16T21:07:54.497245+0000","last_peered":"2024-04-16T21:07:54.497245+0000","last_clean":"2024-04-16T21:07:54.497245+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:07:54.497245+0000","last_undegraded":"2024-04-16T21:07:54.497245+0000","last_fullsized":"2024-04-16T21:07:54.497245+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:10:00.499309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'121","reported_seq":272,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.655968+0000","last_change":"2024-04-16T19:25:11.465848+0000","last_active":"2024-04-16T21:07:54.655968+0000","last_peered":"2024-04-16T21:07:54.655968+0000","last_clean":"2024-04-16T21:07:54.655968+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:07:54.655968+0000","last_undegraded":"2024-04-16T21:07:54.655968+0000","last_fullsized":"2024-04-16T21:07:54.655968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:11:05.115499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'4","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655987+0000","last_change":"2024-04-16T19:25:09.462205+0000","last_active":"2024-04-16T21:07:49.655987+0000","last_peered":"2024-04-16T21:07:49.655987+0000","last_clean":"2024-04-16T21:07:49.655987+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:07:49.655987+0000","last_undegraded":"2024-04-16T21:07:49.655987+0000","last_fullsized":"2024-04-16T21:07:49.655987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:53.677676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.372422+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:08:59.372422+0000","last_peered":"2024-04-16T21:08:59.372422+0000","last_clean":"2024-04-16T21:08:59.372422+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:08:59.372422+0000","last_undegraded":"2024-04-16T21:08:59.372422+0000","last_fullsized":"2024-04-16T21:08:59.372422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364818+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:08:49.364818+0000","last_peered":"2024-04-16T21:08:49.364818+0000","last_clean":"2024-04-16T21:08:49.364818+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:08:49.364818+0000","last_undegraded":"2024-04-16T21:08:49.364818+0000","last_fullsized":"2024-04-16T21:08:49.364818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:09:17.036672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.368088+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:08:59.368088+0000","last_peered":"2024-04-16T21:08:59.368088+0000","last_clean":"2024-04-16T21:08:59.368088+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:08:59.368088+0000","last_undegraded":"2024-04-16T21:08:59.368088+0000","last_fullsized":"2024-04-16T21:08:59.368088+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:55:34.577971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.651049+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:07:19.651049+0000","last_peered":"2024-04-16T21:07:19.651049+0000","last_clean":"2024-04-16T21:07:19.651049+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:07:19.651049+0000","last_undegraded":"2024-04-16T21:07:19.651049+0000","last_fullsized":"2024-04-16T21:07:19.651049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:14.554679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491744+0000","last_change":"2024-04-16T19:24:49.432195+0000","last_active":"2024-04-16T21:07:19.491744+0000","last_peered":"2024-04-16T21:07:19.491744+0000","last_clean":"2024-04-16T21:07:19.491744+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:07:19.491744+0000","last_undegraded":"2024-04-16T21:07:19.491744+0000","last_fullsized":"2024-04-16T21:07:19.491744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:18.116203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653498+0000","last_change":"2024-04-16T19:24:51.573063+0000","last_active":"2024-04-16T21:07:34.653498+0000","last_peered":"2024-04-16T21:07:34.653498+0000","last_clean":"2024-04-16T21:07:34.653498+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:07:34.653498+0000","last_undegraded":"2024-04-16T21:07:34.653498+0000","last_fullsized":"2024-04-16T21:07:34.653498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:01.882223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649776+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:07:19.649776+0000","last_peered":"2024-04-16T21:07:19.649776+0000","last_clean":"2024-04-16T21:07:19.649776+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:07:19.649776+0000","last_undegraded":"2024-04-16T21:07:19.649776+0000","last_fullsized":"2024-04-16T21:07:19.649776+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:28:47.506010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491827+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:07:19.491827+0000","last_peered":"2024-04-16T21:07:19.491827+0000","last_clean":"2024-04-16T21:07:19.491827+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:07:19.491827+0000","last_undegraded":"2024-04-16T21:07:19.491827+0000","last_fullsized":"2024-04-16T21:07:19.491827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:32:33.273769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.662724+0000","last_change":"2024-04-16T19:25:41.509494+0000","last_active":"2024-04-16T21:08:39.662724+0000","last_peered":"2024-04-16T21:08:39.662724+0000","last_clean":"2024-04-16T21:08:39.662724+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:08:39.662724+0000","last_undegraded":"2024-04-16T21:08:39.662724+0000","last_fullsized":"2024-04-16T21:08:39.662724+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:48:03.150130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'8","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369901+0000","last_change":"2024-04-16T19:25:15.483611+0000","last_active":"2024-04-16T21:09:14.369901+0000","last_peered":"2024-04-16T21:09:14.369901+0000","last_clean":"2024-04-16T21:09:14.369901+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:09:14.369901+0000","last_undegraded":"2024-04-16T21:09:14.369901+0000","last_fullsized":"2024-04-16T21:09:14.369901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:36.701444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656848+0000","last_change":"2024-04-16T19:25:13.472723+0000","last_active":"2024-04-16T21:07:54.656848+0000","last_peered":"2024-04-16T21:07:54.656848+0000","last_clean":"2024-04-16T21:07:54.656848+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:07:54.656848+0000","last_undegraded":"2024-04-16T21:07:54.656848+0000","last_fullsized":"2024-04-16T21:07:54.656848+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:20:44.486891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'321","reported_seq":850,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.497294+0000","last_change":"2024-04-16T19:25:11.464441+0000","last_active":"2024-04-16T21:07:54.497294+0000","last_peered":"2024-04-16T21:07:54.497294+0000","last_clean":"2024-04-16T21:07:54.497294+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:07:54.497294+0000","last_undegraded":"2024-04-16T21:07:54.497294+0000","last_fullsized":"2024-04-16T21:07:54.497294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:55:55.683718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'12","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655101+0000","last_change":"2024-04-16T19:25:09.465881+0000","last_active":"2024-04-16T21:07:49.655101+0000","last_peered":"2024-04-16T21:07:49.655101+0000","last_clean":"2024-04-16T21:07:49.655101+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:07:49.655101+0000","last_undegraded":"2024-04-16T21:07:49.655101+0000","last_fullsized":"2024-04-16T21:07:49.655101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.648098+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:07:09.648098+0000","last_peered":"2024-04-16T21:07:09.648098+0000","last_clean":"2024-04-16T21:07:09.648098+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:07:09.648098+0000","last_undegraded":"2024-04-16T21:07:09.648098+0000","last_fullsized":"2024-04-16T21:07:09.648098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648274+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:07:04.648274+0000","last_peered":"2024-04-16T21:07:04.648274+0000","last_clean":"2024-04-16T21:07:04.648274+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:07:04.648274+0000","last_undegraded":"2024-04-16T21:07:04.648274+0000","last_fullsized":"2024-04-16T21:07:04.648274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:48.449534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650601+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:07:19.650601+0000","last_peered":"2024-04-16T21:07:19.650601+0000","last_clean":"2024-04-16T21:07:19.650601+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:07:19.650601+0000","last_undegraded":"2024-04-16T21:07:19.650601+0000","last_fullsized":"2024-04-16T21:07:19.650601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:17.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.507635+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:08:59.507635+0000","last_peered":"2024-04-16T21:08:59.507635+0000","last_clean":"2024-04-16T21:08:59.507635+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:08:59.507635+0000","last_undegraded":"2024-04-16T21:08:59.507635+0000","last_fullsized":"2024-04-16T21:08:59.507635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:40.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.368151+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:09:09.368151+0000","last_peered":"2024-04-16T21:09:09.368151+0000","last_clean":"2024-04-16T21:09:09.368151+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:09:09.368151+0000","last_undegraded":"2024-04-16T21:09:09.368151+0000","last_fullsized":"2024-04-16T21:09:09.368151+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:22.117545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653453+0000","last_change":"2024-04-16T19:24:51.442399+0000","last_active":"2024-04-16T21:07:34.653453+0000","last_peered":"2024-04-16T21:07:34.653453+0000","last_clean":"2024-04-16T21:07:34.653453+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:07:34.653453+0000","last_undegraded":"2024-04-16T21:07:34.653453+0000","last_fullsized":"2024-04-16T21:07:34.653453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:24.851144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650701+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:07:19.650701+0000","last_peered":"2024-04-16T21:07:19.650701+0000","last_clean":"2024-04-16T21:07:19.650701+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:07:19.650701+0000","last_undegraded":"2024-04-16T21:07:19.650701+0000","last_fullsized":"2024-04-16T21:07:19.650701+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:13:42.705352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1306,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491787+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:07:19.491787+0000","last_peered":"2024-04-16T21:07:19.491787+0000","last_clean":"2024-04-16T21:07:19.491787+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:07:19.491787+0000","last_undegraded":"2024-04-16T21:07:19.491787+0000","last_fullsized":"2024-04-16T21:07:19.491787+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:35:13.067416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649861+0000","last_change":"2024-04-16T19:25:41.509274+0000","last_active":"2024-04-16T21:07:14.649861+0000","last_peered":"2024-04-16T21:07:14.649861+0000","last_clean":"2024-04-16T21:07:14.649861+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:07:14.649861+0000","last_undegraded":"2024-04-16T21:07:14.649861+0000","last_fullsized":"2024-04-16T21:07:14.649861+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:03:45.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"47'5","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:14.659574+0000","last_change":"2024-04-16T19:25:15.486177+0000","last_active":"2024-04-16T21:08:14.659574+0000","last_peered":"2024-04-16T21:08:14.659574+0000","last_clean":"2024-04-16T21:08:14.659574+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:08:14.659574+0000","last_undegraded":"2024-04-16T21:08:14.659574+0000","last_fullsized":"2024-04-16T21:08:14.659574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:20.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656265+0000","last_change":"2024-04-16T19:25:13.473925+0000","last_active":"2024-04-16T21:07:54.656265+0000","last_peered":"2024-04-16T21:07:54.656265+0000","last_clean":"2024-04-16T21:07:54.656265+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:07:54.656265+0000","last_undegraded":"2024-04-16T21:07:54.656265+0000","last_fullsized":"2024-04-16T21:07:54.656265+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:22.051385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'241","reported_seq":1104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:54.656229+0000","last_change":"2024-04-16T19:25:11.465793+0000","last_active":"2024-04-16T21:07:54.656229+0000","last_peered":"2024-04-16T21:07:54.656229+0000","last_clean":"2024-04-16T21:07:54.656229+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:07:54.656229+0000","last_undegraded":"2024-04-16T21:07:54.656229+0000","last_fullsized":"2024-04-16T21:07:54.656229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:41:00.538757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'7","reported_seq":153,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655923+0000","last_change":"2024-04-16T19:25:09.458604+0000","last_active":"2024-04-16T21:07:49.655923+0000","last_peered":"2024-04-16T21:07:49.655923+0000","last_clean":"2024-04-16T21:07:49.655923+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:07:49.655923+0000","last_undegraded":"2024-04-16T21:07:49.655923+0000","last_fullsized":"2024-04-16T21:07:49.655923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:31:36.876872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.489948+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:07:09.489948+0000","last_peered":"2024-04-16T21:07:09.489948+0000","last_clean":"2024-04-16T21:07:09.489948+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:07:09.489948+0000","last_undegraded":"2024-04-16T21:07:09.489948+0000","last_fullsized":"2024-04-16T21:07:09.489948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:36:09.925214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.647363+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:07:04.647363+0000","last_peered":"2024-04-16T21:07:04.647363+0000","last_clean":"2024-04-16T21:07:04.647363+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:07:04.647363+0000","last_undegraded":"2024-04-16T21:07:04.647363+0000","last_fullsized":"2024-04-16T21:07:04.647363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:38.670224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.491726+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:07:19.491726+0000","last_peered":"2024-04-16T21:07:19.491726+0000","last_clean":"2024-04-16T21:07:19.491726+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:07:19.491726+0000","last_undegraded":"2024-04-16T21:07:19.491726+0000","last_fullsized":"2024-04-16T21:07:19.491726+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:45.917687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369571+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:09:09.369571+0000","last_peered":"2024-04-16T21:09:09.369571+0000","last_clean":"2024-04-16T21:09:09.369571+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:09:09.369571+0000","last_undegraded":"2024-04-16T21:09:09.369571+0000","last_fullsized":"2024-04-16T21:09:09.369571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:19:32.829903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1315,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367311+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:08:59.367311+0000","last_peered":"2024-04-16T21:08:59.367311+0000","last_clean":"2024-04-16T21:08:59.367311+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:08:59.367311+0000","last_undegraded":"2024-04-16T21:08:59.367311+0000","last_fullsized":"2024-04-16T21:08:59.367311+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:02.604453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.653371+0000","last_change":"2024-04-16T19:24:51.438145+0000","last_active":"2024-04-16T21:07:34.653371+0000","last_peered":"2024-04-16T21:07:34.653371+0000","last_clean":"2024-04-16T21:07:34.653371+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:07:34.653371+0000","last_undegraded":"2024-04-16T21:07:34.653371+0000","last_fullsized":"2024-04-16T21:07:34.653371+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:10:45.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.369666+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:09:14.369666+0000","last_peered":"2024-04-16T21:09:14.369666+0000","last_clean":"2024-04-16T21:09:14.369666+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:09:14.369666+0000","last_undegraded":"2024-04-16T21:09:14.369666+0000","last_fullsized":"2024-04-16T21:09:14.369666+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:31.554611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.661634+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:08:29.661634+0000","last_peered":"2024-04-16T21:08:29.661634+0000","last_clean":"2024-04-16T21:08:29.661634+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:08:29.661634+0000","last_undegraded":"2024-04-16T21:08:29.661634+0000","last_fullsized":"2024-04-16T21:08:29.661634+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:20.643182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.660228+0000","last_change":"2024-04-16T19:25:41.508939+0000","last_active":"2024-04-16T21:08:24.660228+0000","last_peered":"2024-04-16T21:08:24.660228+0000","last_clean":"2024-04-16T21:08:24.660228+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:08:24.660228+0000","last_undegraded":"2024-04-16T21:08:24.660228+0000","last_fullsized":"2024-04-16T21:08:24.660228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:02:25.507030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'11","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.498087+0000","last_change":"2024-04-16T19:25:15.480096+0000","last_active":"2024-04-16T21:07:59.498087+0000","last_peered":"2024-04-16T21:07:59.498087+0000","last_clean":"2024-04-16T21:07:59.498087+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:07:59.498087+0000","last_undegraded":"2024-04-16T21:07:59.498087+0000","last_fullsized":"2024-04-16T21:07:59.498087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:53:58.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659661+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:08:09.659661+0000","last_peered":"2024-04-16T21:08:09.659661+0000","last_clean":"2024-04-16T21:08:09.659661+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:08:09.659661+0000","last_undegraded":"2024-04-16T21:08:09.659661+0000","last_fullsized":"2024-04-16T21:08:09.659661+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:01.595327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'540","reported_seq":1356,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373967+0000","last_change":"2024-04-16T19:25:11.465566+0000","last_active":"2024-04-16T21:09:09.373967+0000","last_peered":"2024-04-16T21:09:09.373967+0000","last_clean":"2024-04-16T21:09:09.373967+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:09:09.373967+0000","last_undegraded":"2024-04-16T21:09:09.373967+0000","last_fullsized":"2024-04-16T21:09:09.373967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:04:52.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'9","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368023+0000","last_change":"2024-04-16T19:25:09.458347+0000","last_active":"2024-04-16T21:09:04.368023+0000","last_peered":"2024-04-16T21:09:04.368023+0000","last_clean":"2024-04-16T21:09:04.368023+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:09:04.368023+0000","last_undegraded":"2024-04-16T21:09:04.368023+0000","last_fullsized":"2024-04-16T21:09:04.368023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649060+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:07:09.649060+0000","last_peered":"2024-04-16T21:07:09.649060+0000","last_clean":"2024-04-16T21:07:09.649060+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:07:09.649060+0000","last_undegraded":"2024-04-16T21:07:09.649060+0000","last_fullsized":"2024-04-16T21:07:09.649060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:41.726894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648043+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:07:04.648043+0000","last_peered":"2024-04-16T21:07:04.648043+0000","last_clean":"2024-04-16T21:07:04.648043+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:07:04.648043+0000","last_undegraded":"2024-04-16T21:07:04.648043+0000","last_fullsized":"2024-04-16T21:07:04.648043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650514+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:07:19.650514+0000","last_peered":"2024-04-16T21:07:19.650514+0000","last_clean":"2024-04-16T21:07:19.650514+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:07:19.650514+0000","last_undegraded":"2024-04-16T21:07:19.650514+0000","last_fullsized":"2024-04-16T21:07:19.650514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:06:46.054276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367931+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:09:04.367931+0000","last_peered":"2024-04-16T21:09:04.367931+0000","last_clean":"2024-04-16T21:09:04.367931+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:09:04.367931+0000","last_undegraded":"2024-04-16T21:09:04.367931+0000","last_fullsized":"2024-04-16T21:09:04.367931+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:14:47.298587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369608+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:08:44.369608+0000","last_peered":"2024-04-16T21:08:44.369608+0000","last_clean":"2024-04-16T21:08:44.369608+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:08:44.369608+0000","last_undegraded":"2024-04-16T21:08:44.369608+0000","last_fullsized":"2024-04-16T21:08:44.369608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:38:59.670330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366563+0000","last_change":"2024-04-16T19:24:51.437809+0000","last_active":"2024-04-16T21:08:49.366563+0000","last_peered":"2024-04-16T21:08:49.366563+0000","last_clean":"2024-04-16T21:08:49.366563+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:08:49.366563+0000","last_undegraded":"2024-04-16T21:08:49.366563+0000","last_fullsized":"2024-04-16T21:08:49.366563+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:43.643863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.509969+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:09:14.509969+0000","last_peered":"2024-04-16T21:09:14.509969+0000","last_clean":"2024-04-16T21:09:14.509969+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:09:14.509969+0000","last_undegraded":"2024-04-16T21:09:14.509969+0000","last_fullsized":"2024-04-16T21:09:14.509969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:55:01.357303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1337,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502628+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:08:29.502628+0000","last_peered":"2024-04-16T21:08:29.502628+0000","last_clean":"2024-04-16T21:08:29.502628+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:08:29.502628+0000","last_undegraded":"2024-04-16T21:08:29.502628+0000","last_fullsized":"2024-04-16T21:08:29.502628+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:59:48.136957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.661216+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:08:24.661216+0000","last_peered":"2024-04-16T21:08:24.661216+0000","last_clean":"2024-04-16T21:08:24.661216+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:08:24.661216+0000","last_undegraded":"2024-04-16T21:08:24.661216+0000","last_fullsized":"2024-04-16T21:08:24.661216+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:47.003646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'8","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656179+0000","last_change":"2024-04-16T19:25:15.481698+0000","last_active":"2024-04-16T21:07:59.656179+0000","last_peered":"2024-04-16T21:07:59.656179+0000","last_clean":"2024-04-16T21:07:59.656179+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:07:59.656179+0000","last_undegraded":"2024-04-16T21:07:59.656179+0000","last_fullsized":"2024-04-16T21:07:59.656179+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:46:49.675450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'705","reported_seq":2072,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509303+0000","last_change":"2024-04-16T19:25:13.474684+0000","last_active":"2024-04-16T21:09:09.509303+0000","last_peered":"2024-04-16T21:09:09.509303+0000","last_clean":"2024-04-16T21:09:09.509303+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:09:09.509303+0000","last_undegraded":"2024-04-16T21:09:09.509303+0000","last_fullsized":"2024-04-16T21:09:09.509303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:52.004725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'519","reported_seq":41267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499619+0000","last_change":"2024-04-16T19:25:11.465897+0000","last_active":"2024-04-16T21:08:09.499619+0000","last_peered":"2024-04-16T21:08:09.499619+0000","last_clean":"2024-04-16T21:08:09.499619+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:08:09.499619+0000","last_undegraded":"2024-04-16T21:08:09.499619+0000","last_fullsized":"2024-04-16T21:08:09.499619+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:54.168091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'9","reported_seq":308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368812+0000","last_change":"2024-04-16T19:25:09.459964+0000","last_active":"2024-04-16T21:09:04.368812+0000","last_peered":"2024-04-16T21:09:04.368812+0000","last_clean":"2024-04-16T21:09:04.368812+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:09:04.368812+0000","last_undegraded":"2024-04-16T21:09:04.368812+0000","last_fullsized":"2024-04-16T21:09:04.368812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:43.395417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:09.649029+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:07:09.649029+0000","last_peered":"2024-04-16T21:07:09.649029+0000","last_clean":"2024-04-16T21:07:09.649029+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:07:09.649029+0000","last_undegraded":"2024-04-16T21:07:09.649029+0000","last_fullsized":"2024-04-16T21:07:09.649029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:25.651034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:04.648095+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:07:04.648095+0000","last_peered":"2024-04-16T21:07:04.648095+0000","last_clean":"2024-04-16T21:07:04.648095+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:07:04.648095+0000","last_undegraded":"2024-04-16T21:07:04.648095+0000","last_fullsized":"2024-04-16T21:07:04.648095+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:23.870654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650488+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:07:19.650488+0000","last_peered":"2024-04-16T21:07:19.650488+0000","last_clean":"2024-04-16T21:07:19.650488+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:07:19.650488+0000","last_undegraded":"2024-04-16T21:07:19.650488+0000","last_fullsized":"2024-04-16T21:07:19.650488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:23:04.143195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369636+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:08:44.369636+0000","last_peered":"2024-04-16T21:08:44.369636+0000","last_clean":"2024-04-16T21:08:44.369636+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:08:44.369636+0000","last_undegraded":"2024-04-16T21:08:44.369636+0000","last_fullsized":"2024-04-16T21:08:44.369636+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:35:20.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.368656+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:09:04.368656+0000","last_peered":"2024-04-16T21:09:04.368656+0000","last_clean":"2024-04-16T21:09:04.368656+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:09:04.368656+0000","last_undegraded":"2024-04-16T21:09:04.368656+0000","last_fullsized":"2024-04-16T21:09:04.368656+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:13:32.143903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364971+0000","last_change":"2024-04-16T19:24:51.573983+0000","last_active":"2024-04-16T21:08:49.364971+0000","last_peered":"2024-04-16T21:08:49.364971+0000","last_clean":"2024-04-16T21:08:49.364971+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:08:49.364971+0000","last_undegraded":"2024-04-16T21:08:49.364971+0000","last_fullsized":"2024-04-16T21:08:49.364971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:14:09.803372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.374829+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:09:14.374829+0000","last_peered":"2024-04-16T21:09:14.374829+0000","last_clean":"2024-04-16T21:09:14.374829+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:09:14.374829+0000","last_undegraded":"2024-04-16T21:09:14.374829+0000","last_fullsized":"2024-04-16T21:09:14.374829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:05.975304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'167","reported_seq":2111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502381+0000","last_change":"2024-04-16T19:25:49.519298+0000","last_active":"2024-04-16T21:08:29.502381+0000","last_peered":"2024-04-16T21:08:29.502381+0000","last_clean":"2024-04-16T21:08:29.502381+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:08:29.502381+0000","last_undegraded":"2024-04-16T21:08:29.502381+0000","last_fullsized":"2024-04-16T21:08:29.502381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:30:10.645836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501890+0000","last_change":"2024-04-16T19:25:41.507268+0000","last_active":"2024-04-16T21:08:24.501890+0000","last_peered":"2024-04-16T21:08:24.501890+0000","last_clean":"2024-04-16T21:08:24.501890+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:08:24.501890+0000","last_undegraded":"2024-04-16T21:08:24.501890+0000","last_fullsized":"2024-04-16T21:08:24.501890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:41:46.285785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.657100+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:07:59.657100+0000","last_peered":"2024-04-16T21:07:59.657100+0000","last_clean":"2024-04-16T21:07:59.657100+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:07:59.657100+0000","last_undegraded":"2024-04-16T21:07:59.657100+0000","last_fullsized":"2024-04-16T21:07:59.657100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:21:46.607704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369049+0000","last_change":"2024-04-16T19:25:13.472244+0000","last_active":"2024-04-16T21:09:09.369049+0000","last_peered":"2024-04-16T21:09:09.369049+0000","last_clean":"2024-04-16T21:09:09.369049+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:09:09.369049+0000","last_undegraded":"2024-04-16T21:09:09.369049+0000","last_fullsized":"2024-04-16T21:09:09.369049+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:59.248190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'573","reported_seq":1335,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369613+0000","last_change":"2024-04-16T19:25:11.468986+0000","last_active":"2024-04-16T21:09:09.369613+0000","last_peered":"2024-04-16T21:09:09.369613+0000","last_clean":"2024-04-16T21:09:09.369613+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:09:09.369613+0000","last_undegraded":"2024-04-16T21:09:09.369613+0000","last_fullsized":"2024-04-16T21:09:09.369613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:25:20.042820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'23","reported_seq":321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.498919+0000","last_change":"2024-04-16T19:25:09.469035+0000","last_active":"2024-04-16T21:08:04.498919+0000","last_peered":"2024-04-16T21:08:04.498919+0000","last_clean":"2024-04-16T21:08:04.498919+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:08:04.498919+0000","last_undegraded":"2024-04-16T21:08:04.498919+0000","last_fullsized":"2024-04-16T21:08:04.498919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371344+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:08:54.371344+0000","last_peered":"2024-04-16T21:08:54.371344+0000","last_clean":"2024-04-16T21:08:54.371344+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:08:54.371344+0000","last_undegraded":"2024-04-16T21:08:54.371344+0000","last_fullsized":"2024-04-16T21:08:54.371344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.506100+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:08:49.506100+0000","last_peered":"2024-04-16T21:08:49.506100+0000","last_clean":"2024-04-16T21:08:49.506100+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:08:49.506100+0000","last_undegraded":"2024-04-16T21:08:49.506100+0000","last_fullsized":"2024-04-16T21:08:49.506100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364559+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:08:44.364559+0000","last_peered":"2024-04-16T21:08:44.364559+0000","last_clean":"2024-04-16T21:08:44.364559+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:08:44.364559+0000","last_undegraded":"2024-04-16T21:08:44.364559+0000","last_fullsized":"2024-04-16T21:08:44.364559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:39:11.958520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649687+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:07:19.649687+0000","last_peered":"2024-04-16T21:07:19.649687+0000","last_clean":"2024-04-16T21:07:19.649687+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:07:19.649687+0000","last_undegraded":"2024-04-16T21:07:19.649687+0000","last_fullsized":"2024-04-16T21:07:19.649687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:21:25.378268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650982+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:07:19.650982+0000","last_peered":"2024-04-16T21:07:19.650982+0000","last_clean":"2024-04-16T21:07:19.650982+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:07:19.650982+0000","last_undegraded":"2024-04-16T21:07:19.650982+0000","last_fullsized":"2024-04-16T21:07:19.650982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366474+0000","last_change":"2024-04-16T19:24:51.444882+0000","last_active":"2024-04-16T21:08:49.366474+0000","last_peered":"2024-04-16T21:08:49.366474+0000","last_clean":"2024-04-16T21:08:49.366474+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:08:49.366474+0000","last_undegraded":"2024-04-16T21:08:49.366474+0000","last_fullsized":"2024-04-16T21:08:49.366474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:21:18.795884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368933+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:09:14.368933+0000","last_peered":"2024-04-16T21:09:14.368933+0000","last_clean":"2024-04-16T21:09:14.368933+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:09:14.368933+0000","last_undegraded":"2024-04-16T21:09:14.368933+0000","last_fullsized":"2024-04-16T21:09:14.368933+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:37:36.899944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'127","reported_seq":1509,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:29.502335+0000","last_change":"2024-04-16T19:25:49.517472+0000","last_active":"2024-04-16T21:08:29.502335+0000","last_peered":"2024-04-16T21:08:29.502335+0000","last_clean":"2024-04-16T21:08:29.502335+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:08:29.502335+0000","last_undegraded":"2024-04-16T21:08:29.502335+0000","last_fullsized":"2024-04-16T21:08:29.502335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:05:30.634191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501937+0000","last_change":"2024-04-16T19:25:41.502982+0000","last_active":"2024-04-16T21:08:24.501937+0000","last_peered":"2024-04-16T21:08:24.501937+0000","last_clean":"2024-04-16T21:08:24.501937+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:08:24.501937+0000","last_undegraded":"2024-04-16T21:08:24.501937+0000","last_fullsized":"2024-04-16T21:08:24.501937+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:12.630377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"47'10","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:59.656132+0000","last_change":"2024-04-16T19:25:15.481921+0000","last_active":"2024-04-16T21:07:59.656132+0000","last_peered":"2024-04-16T21:07:59.656132+0000","last_clean":"2024-04-16T21:07:59.656132+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:07:59.656132+0000","last_undegraded":"2024-04-16T21:07:59.656132+0000","last_fullsized":"2024-04-16T21:07:59.656132+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:42:20.018470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369098+0000","last_change":"2024-04-16T19:25:13.473915+0000","last_active":"2024-04-16T21:09:09.369098+0000","last_peered":"2024-04-16T21:09:09.369098+0000","last_clean":"2024-04-16T21:09:09.369098+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:09:09.369098+0000","last_undegraded":"2024-04-16T21:09:09.369098+0000","last_fullsized":"2024-04-16T21:09:09.369098+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:32:05.770561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'230","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369554+0000","last_change":"2024-04-16T19:25:11.461723+0000","last_active":"2024-04-16T21:09:09.369554+0000","last_peered":"2024-04-16T21:09:09.369554+0000","last_clean":"2024-04-16T21:09:09.369554+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:09:09.369554+0000","last_undegraded":"2024-04-16T21:09:09.369554+0000","last_fullsized":"2024-04-16T21:09:09.369554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:10.445017+0000","last_clean_scrub_stamp":"2024-04-16T19:25:10.445017+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:44:15.327068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'6","reported_seq":175,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367981+0000","last_change":"2024-04-16T19:25:09.458030+0000","last_active":"2024-04-16T21:09:04.367981+0000","last_peered":"2024-04-16T21:09:04.367981+0000","last_clean":"2024-04-16T21:09:04.367981+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:09:04.367981+0000","last_undegraded":"2024-04-16T21:09:04.367981+0000","last_fullsized":"2024-04-16T21:09:04.367981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:57:34.082038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663306+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:08:34.663306+0000","last_peered":"2024-04-16T21:08:34.663306+0000","last_clean":"2024-04-16T21:08:34.663306+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:08:34.663306+0000","last_undegraded":"2024-04-16T21:08:34.663306+0000","last_fullsized":"2024-04-16T21:08:34.663306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.369834+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:08:44.369834+0000","last_peered":"2024-04-16T21:08:44.369834+0000","last_clean":"2024-04-16T21:08:44.369834+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:08:44.369834+0000","last_undegraded":"2024-04-16T21:08:44.369834+0000","last_fullsized":"2024-04-16T21:08:44.369834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:41.742389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.367273+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:09:04.367273+0000","last_peered":"2024-04-16T21:09:04.367273+0000","last_clean":"2024-04-16T21:09:04.367273+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:09:04.367273+0000","last_undegraded":"2024-04-16T21:09:04.367273+0000","last_fullsized":"2024-04-16T21:09:04.367273+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:38.387958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":222,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650527+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:07:19.650527+0000","last_peered":"2024-04-16T21:07:19.650527+0000","last_clean":"2024-04-16T21:07:19.650527+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:07:19.650527+0000","last_undegraded":"2024-04-16T21:07:19.650527+0000","last_fullsized":"2024-04-16T21:07:19.650527+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:47.927883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649659+0000","last_change":"2024-04-16T19:24:49.432869+0000","last_active":"2024-04-16T21:07:19.649659+0000","last_peered":"2024-04-16T21:07:19.649659+0000","last_clean":"2024-04-16T21:07:19.649659+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:07:19.649659+0000","last_undegraded":"2024-04-16T21:07:19.649659+0000","last_fullsized":"2024-04-16T21:07:19.649659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:07.670865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655242+0000","last_change":"2024-04-16T19:24:51.442598+0000","last_active":"2024-04-16T21:07:49.655242+0000","last_peered":"2024-04-16T21:07:49.655242+0000","last_clean":"2024-04-16T21:07:49.655242+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:07:49.655242+0000","last_undegraded":"2024-04-16T21:07:49.655242+0000","last_fullsized":"2024-04-16T21:07:49.655242+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:20:52.222795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'12","reported_seq":186,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504441+0000","last_change":"2024-04-16T19:25:09.464004+0000","last_active":"2024-04-16T21:08:39.504441+0000","last_peered":"2024-04-16T21:08:39.504441+0000","last_clean":"2024-04-16T21:08:39.504441+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:08:39.504441+0000","last_undegraded":"2024-04-16T21:08:39.504441+0000","last_fullsized":"2024-04-16T21:08:39.504441+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:23:50.286946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504683+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:08:39.504683+0000","last_peered":"2024-04-16T21:08:39.504683+0000","last_clean":"2024-04-16T21:08:39.504683+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:08:39.504683+0000","last_undegraded":"2024-04-16T21:08:39.504683+0000","last_fullsized":"2024-04-16T21:08:39.504683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:43:43.749749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.364532+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:08:44.364532+0000","last_peered":"2024-04-16T21:08:44.364532+0000","last_clean":"2024-04-16T21:08:44.364532+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:08:44.364532+0000","last_undegraded":"2024-04-16T21:08:44.364532+0000","last_fullsized":"2024-04-16T21:08:44.364532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:29:11.626800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'12","reported_seq":242,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663758+0000","last_change":"2024-04-16T19:25:09.466486+0000","last_active":"2024-04-16T21:08:39.663758+0000","last_peered":"2024-04-16T21:08:39.663758+0000","last_clean":"2024-04-16T21:08:39.663758+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:08:39.663758+0000","last_undegraded":"2024-04-16T21:08:39.663758+0000","last_fullsized":"2024-04-16T21:08:39.663758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:36:06.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504632+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:08:39.504632+0000","last_peered":"2024-04-16T21:08:39.504632+0000","last_clean":"2024-04-16T21:08:39.504632+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:08:39.504632+0000","last_undegraded":"2024-04-16T21:08:39.504632+0000","last_fullsized":"2024-04-16T21:08:39.504632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:49.140683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.505241+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:08:44.505241+0000","last_peered":"2024-04-16T21:08:44.505241+0000","last_clean":"2024-04-16T21:08:44.505241+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:08:44.505241+0000","last_undegraded":"2024-04-16T21:08:44.505241+0000","last_fullsized":"2024-04-16T21:08:44.505241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:03:25.186300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'5","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:29.372179+0000","last_change":"2024-04-16T19:25:09.467044+0000","last_active":"2024-04-16T21:09:29.372179+0000","last_peered":"2024-04-16T21:09:29.372179+0000","last_clean":"2024-04-16T21:09:29.372179+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:09:29.372179+0000","last_undegraded":"2024-04-16T21:09:29.372179+0000","last_fullsized":"2024-04-16T21:09:29.372179+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:00:23.454872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.663818+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:08:39.663818+0000","last_peered":"2024-04-16T21:08:39.663818+0000","last_clean":"2024-04-16T21:08:39.663818+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:08:39.663818+0000","last_undegraded":"2024-04-16T21:08:39.663818+0000","last_fullsized":"2024-04-16T21:08:39.663818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:21:51.366144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:44.365701+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:08:44.365701+0000","last_peered":"2024-04-16T21:08:44.365701+0000","last_clean":"2024-04-16T21:08:44.365701+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:08:44.365701+0000","last_undegraded":"2024-04-16T21:08:44.365701+0000","last_fullsized":"2024-04-16T21:08:44.365701+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'7","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.664206+0000","last_change":"2024-04-16T19:25:09.458671+0000","last_active":"2024-04-16T21:08:39.664206+0000","last_peered":"2024-04-16T21:08:39.664206+0000","last_clean":"2024-04-16T21:08:39.664206+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:08:39.664206+0000","last_undegraded":"2024-04-16T21:08:39.664206+0000","last_fullsized":"2024-04-16T21:08:39.664206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:54:52.162049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504531+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:08:39.504531+0000","last_peered":"2024-04-16T21:08:39.504531+0000","last_clean":"2024-04-16T21:08:39.504531+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:08:39.504531+0000","last_undegraded":"2024-04-16T21:08:39.504531+0000","last_fullsized":"2024-04-16T21:08:39.504531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:22.916044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.373116+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:09:04.373116+0000","last_peered":"2024-04-16T21:09:04.373116+0000","last_clean":"2024-04-16T21:09:04.373116+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:09:04.373116+0000","last_undegraded":"2024-04-16T21:09:04.373116+0000","last_fullsized":"2024-04-16T21:09:04.373116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'11","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.655456+0000","last_change":"2024-04-16T19:25:09.466470+0000","last_active":"2024-04-16T21:07:49.655456+0000","last_peered":"2024-04-16T21:07:49.655456+0000","last_clean":"2024-04-16T21:07:49.655456+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:07:49.655456+0000","last_undegraded":"2024-04-16T21:07:49.655456+0000","last_fullsized":"2024-04-16T21:07:49.655456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:34:47.990118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:04.508457+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:09:04.508457+0000","last_peered":"2024-04-16T21:09:04.508457+0000","last_clean":"2024-04-16T21:09:04.508457+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:09:04.508457+0000","last_undegraded":"2024-04-16T21:09:04.508457+0000","last_fullsized":"2024-04-16T21:09:04.508457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:58.689460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503608+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:08:34.503608+0000","last_peered":"2024-04-16T21:08:34.503608+0000","last_clean":"2024-04-16T21:08:34.503608+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:08:34.503608+0000","last_undegraded":"2024-04-16T21:08:34.503608+0000","last_fullsized":"2024-04-16T21:08:34.503608+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:02:49.978039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'20","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:49.496627+0000","last_change":"2024-04-16T19:25:09.460695+0000","last_active":"2024-04-16T21:07:49.496627+0000","last_peered":"2024-04-16T21:07:49.496627+0000","last_clean":"2024-04-16T21:07:49.496627+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:07:49.496627+0000","last_undegraded":"2024-04-16T21:07:49.496627+0000","last_fullsized":"2024-04-16T21:07:49.496627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:56:28.879723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.365419+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:08:49.365419+0000","last_peered":"2024-04-16T21:08:49.365419+0000","last_clean":"2024-04-16T21:08:49.365419+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:08:49.365419+0000","last_undegraded":"2024-04-16T21:08:49.365419+0000","last_fullsized":"2024-04-16T21:08:49.365419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:49:16.805347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503638+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:08:34.503638+0000","last_peered":"2024-04-16T21:08:34.503638+0000","last_clean":"2024-04-16T21:08:34.503638+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:08:34.503638+0000","last_undegraded":"2024-04-16T21:08:34.503638+0000","last_fullsized":"2024-04-16T21:08:34.503638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:22.440269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'11","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:39.504570+0000","last_change":"2024-04-16T19:25:09.459969+0000","last_active":"2024-04-16T21:08:39.504570+0000","last_peered":"2024-04-16T21:08:39.504570+0000","last_clean":"2024-04-16T21:08:39.504570+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:08:39.504570+0000","last_undegraded":"2024-04-16T21:08:39.504570+0000","last_fullsized":"2024-04-16T21:08:39.504570+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:52:32.899966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.364844+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:08:49.364844+0000","last_peered":"2024-04-16T21:08:49.364844+0000","last_clean":"2024-04-16T21:08:49.364844+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:08:49.364844+0000","last_undegraded":"2024-04-16T21:08:49.364844+0000","last_fullsized":"2024-04-16T21:08:49.364844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:13.729979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663475+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:08:34.663475+0000","last_peered":"2024-04-16T21:08:34.663475+0000","last_clean":"2024-04-16T21:08:34.663475+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:08:34.663475+0000","last_undegraded":"2024-04-16T21:08:34.663475+0000","last_fullsized":"2024-04-16T21:08:34.663475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:40.025589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'13","reported_seq":290,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.506803+0000","last_change":"2024-04-16T19:25:09.459396+0000","last_active":"2024-04-16T21:08:54.506803+0000","last_peered":"2024-04-16T21:08:54.506803+0000","last_clean":"2024-04-16T21:08:54.506803+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:08:54.506803+0000","last_undegraded":"2024-04-16T21:08:54.506803+0000","last_fullsized":"2024-04-16T21:08:54.506803+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:07:44.854219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:49.366364+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:08:49.366364+0000","last_peered":"2024-04-16T21:08:49.366364+0000","last_clean":"2024-04-16T21:08:49.366364+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:08:49.366364+0000","last_undegraded":"2024-04-16T21:08:49.366364+0000","last_fullsized":"2024-04-16T21:08:49.366364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:00.348353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.503606+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:08:34.503606+0000","last_peered":"2024-04-16T21:08:34.503606+0000","last_clean":"2024-04-16T21:08:34.503606+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:08:34.503606+0000","last_undegraded":"2024-04-16T21:08:34.503606+0000","last_fullsized":"2024-04-16T21:08:34.503606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:56:59.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'38","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.371052+0000","last_change":"2024-04-16T19:25:09.462718+0000","last_active":"2024-04-16T21:09:19.371052+0000","last_peered":"2024-04-16T21:09:19.371052+0000","last_clean":"2024-04-16T21:09:19.371052+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:09:19.371052+0000","last_undegraded":"2024-04-16T21:09:19.371052+0000","last_fullsized":"2024-04-16T21:09:19.371052+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:51.473781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.649801+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:07:19.649801+0000","last_peered":"2024-04-16T21:07:19.649801+0000","last_clean":"2024-04-16T21:07:19.649801+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:07:19.649801+0000","last_undegraded":"2024-04-16T21:07:19.649801+0000","last_fullsized":"2024-04-16T21:07:19.649801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:55:35.897925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.366306+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:08:54.366306+0000","last_peered":"2024-04-16T21:08:54.366306+0000","last_clean":"2024-04-16T21:08:54.366306+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:08:54.366306+0000","last_undegraded":"2024-04-16T21:08:54.366306+0000","last_fullsized":"2024-04-16T21:08:54.366306+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T20:01:56.682100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'14","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:34.513064+0000","last_change":"2024-04-16T19:25:09.467165+0000","last_active":"2024-04-16T21:09:34.513064+0000","last_peered":"2024-04-16T21:09:34.513064+0000","last_clean":"2024-04-16T21:09:34.513064+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:09:34.513064+0000","last_undegraded":"2024-04-16T21:09:34.513064+0000","last_fullsized":"2024-04-16T21:09:34.513064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:39:19.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:19.650655+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:07:19.650655+0000","last_peered":"2024-04-16T21:07:19.650655+0000","last_clean":"2024-04-16T21:07:19.650655+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:07:19.650655+0000","last_undegraded":"2024-04-16T21:07:19.650655+0000","last_fullsized":"2024-04-16T21:07:19.650655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657012+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:08:04.657012+0000","last_peered":"2024-04-16T21:08:04.657012+0000","last_clean":"2024-04-16T21:08:04.657012+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:08:04.657012+0000","last_undegraded":"2024-04-16T21:08:04.657012+0000","last_fullsized":"2024-04-16T21:08:04.657012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:45.504901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'9","reported_seq":180,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.375542+0000","last_change":"2024-04-16T19:25:09.464008+0000","last_active":"2024-04-16T21:09:19.375542+0000","last_peered":"2024-04-16T21:09:19.375542+0000","last_clean":"2024-04-16T21:09:19.375542+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:09:19.375542+0000","last_undegraded":"2024-04-16T21:09:19.375542+0000","last_fullsized":"2024-04-16T21:09:19.375542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:40:58.532715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:24.501544+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:08:24.501544+0000","last_peered":"2024-04-16T21:08:24.501544+0000","last_clean":"2024-04-16T21:08:24.501544+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:08:24.501544+0000","last_undegraded":"2024-04-16T21:08:24.501544+0000","last_fullsized":"2024-04-16T21:08:24.501544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:19:47.512261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.657935+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:08:04.657935+0000","last_peered":"2024-04-16T21:08:04.657935+0000","last_clean":"2024-04-16T21:08:04.657935+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:08:04.657935+0000","last_undegraded":"2024-04-16T21:08:04.657935+0000","last_fullsized":"2024-04-16T21:08:04.657935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:59:39.719534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'22","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:19.510724+0000","last_change":"2024-04-16T19:25:09.467151+0000","last_active":"2024-04-16T21:09:19.510724+0000","last_peered":"2024-04-16T21:09:19.510724+0000","last_clean":"2024-04-16T21:09:19.510724+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:09:19.510724+0000","last_undegraded":"2024-04-16T21:09:19.510724+0000","last_fullsized":"2024-04-16T21:09:19.510724+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:07:22.630747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.659703+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:08:09.659703+0000","last_peered":"2024-04-16T21:08:09.659703+0000","last_clean":"2024-04-16T21:08:09.659703+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:08:09.659703+0000","last_undegraded":"2024-04-16T21:08:09.659703+0000","last_fullsized":"2024-04-16T21:08:09.659703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:58:11.693748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:14.368947+0000","last_change":"2024-04-16T19:25:41.505958+0000","last_active":"2024-04-16T21:09:14.368947+0000","last_peered":"2024-04-16T21:09:14.368947+0000","last_clean":"2024-04-16T21:09:14.368947+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:09:14.368947+0000","last_undegraded":"2024-04-16T21:09:14.368947+0000","last_fullsized":"2024-04-16T21:09:14.368947+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:49:19.162890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'5","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.369023+0000","last_change":"2024-04-16T19:25:09.467605+0000","last_active":"2024-04-16T21:09:09.369023+0000","last_peered":"2024-04-16T21:09:09.369023+0000","last_clean":"2024-04-16T21:09:09.369023+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:09:09.369023+0000","last_undegraded":"2024-04-16T21:09:09.369023+0000","last_fullsized":"2024-04-16T21:09:09.369023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:34.712542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.658742+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:08:09.658742+0000","last_peered":"2024-04-16T21:08:09.658742+0000","last_clean":"2024-04-16T21:08:09.658742+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:08:09.658742+0000","last_undegraded":"2024-04-16T21:08:09.658742+0000","last_fullsized":"2024-04-16T21:08:09.658742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:24.447630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:04.658987+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:08:04.658987+0000","last_peered":"2024-04-16T21:08:04.658987+0000","last_clean":"2024-04-16T21:08:04.658987+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:08:04.658987+0000","last_undegraded":"2024-04-16T21:08:04.658987+0000","last_fullsized":"2024-04-16T21:08:04.658987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:47.133322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'14","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.509341+0000","last_change":"2024-04-16T19:25:09.464438+0000","last_active":"2024-04-16T21:09:09.509341+0000","last_peered":"2024-04-16T21:09:09.509341+0000","last_clean":"2024-04-16T21:09:09.509341+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:09:09.509341+0000","last_undegraded":"2024-04-16T21:09:09.509341+0000","last_fullsized":"2024-04-16T21:09:09.509341+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:36:18.197427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.657776+0000","last_change":"2024-04-16T19:25:41.508398+0000","last_active":"2024-04-16T21:08:09.657776+0000","last_peered":"2024-04-16T21:08:09.657776+0000","last_clean":"2024-04-16T21:08:09.657776+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:08:09.657776+0000","last_undegraded":"2024-04-16T21:08:09.657776+0000","last_fullsized":"2024-04-16T21:08:09.657776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:28:30.770132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:19.661196+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:08:19.661196+0000","last_peered":"2024-04-16T21:08:19.661196+0000","last_clean":"2024-04-16T21:08:19.661196+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:08:19.661196+0000","last_undegraded":"2024-04-16T21:08:19.661196+0000","last_fullsized":"2024-04-16T21:08:19.661196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:10:37.636086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'3","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:34.652749+0000","last_change":"2024-04-16T19:25:41.499612+0000","last_active":"2024-04-16T21:07:34.652749+0000","last_peered":"2024-04-16T21:07:34.652749+0000","last_clean":"2024-04-16T21:07:34.652749+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:07:34.652749+0000","last_undegraded":"2024-04-16T21:07:34.652749+0000","last_fullsized":"2024-04-16T21:07:34.652749+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:23:33.874851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:09.499546+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:08:09.499546+0000","last_peered":"2024-04-16T21:08:09.499546+0000","last_clean":"2024-04-16T21:08:09.499546+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:08:09.499546+0000","last_undegraded":"2024-04-16T21:08:09.499546+0000","last_fullsized":"2024-04-16T21:08:09.499546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T19:59:42.032950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490904+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:07:14.490904+0000","last_peered":"2024-04-16T21:07:14.490904+0000","last_clean":"2024-04-16T21:07:14.490904+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:07:14.490904+0000","last_undegraded":"2024-04-16T21:07:14.490904+0000","last_fullsized":"2024-04-16T21:07:14.490904+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:19:50.557311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'19","reported_seq":286,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:09.373955+0000","last_change":"2024-04-16T19:25:09.468675+0000","last_active":"2024-04-16T21:09:09.373955+0000","last_peered":"2024-04-16T21:09:09.373955+0000","last_clean":"2024-04-16T21:09:09.373955+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:09:09.373955+0000","last_undegraded":"2024-04-16T21:09:09.373955+0000","last_fullsized":"2024-04-16T21:09:09.373955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:21:12.600213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:59.367898+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:08:59.367898+0000","last_peered":"2024-04-16T21:08:59.367898+0000","last_clean":"2024-04-16T21:08:59.367898+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:08:59.367898+0000","last_undegraded":"2024-04-16T21:08:59.367898+0000","last_fullsized":"2024-04-16T21:08:59.367898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:07:28.368196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.649751+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:07:14.649751+0000","last_peered":"2024-04-16T21:07:14.649751+0000","last_clean":"2024-04-16T21:07:14.649751+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:07:14.649751+0000","last_undegraded":"2024-04-16T21:07:14.649751+0000","last_fullsized":"2024-04-16T21:07:14.649751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:50.981157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'12","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.652830+0000","last_change":"2024-04-16T19:25:09.459052+0000","last_active":"2024-04-16T21:07:39.652830+0000","last_peered":"2024-04-16T21:07:39.652830+0000","last_clean":"2024-04-16T21:07:39.652830+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:07:39.652830+0000","last_undegraded":"2024-04-16T21:07:39.652830+0000","last_fullsized":"2024-04-16T21:07:39.652830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:51.756021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.650094+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:07:14.650094+0000","last_peered":"2024-04-16T21:07:14.650094+0000","last_clean":"2024-04-16T21:07:14.650094+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:07:14.650094+0000","last_undegraded":"2024-04-16T21:07:14.650094+0000","last_fullsized":"2024-04-16T21:07:14.650094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:26.742545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:54.371494+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:08:54.371494+0000","last_peered":"2024-04-16T21:08:54.371494+0000","last_clean":"2024-04-16T21:08:54.371494+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:08:54.371494+0000","last_undegraded":"2024-04-16T21:08:54.371494+0000","last_fullsized":"2024-04-16T21:08:54.371494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:02:41.504815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'3","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:39.494798+0000","last_change":"2024-04-16T19:25:09.460170+0000","last_active":"2024-04-16T21:07:39.494798+0000","last_peered":"2024-04-16T21:07:39.494798+0000","last_clean":"2024-04-16T21:07:39.494798+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:07:39.494798+0000","last_undegraded":"2024-04-16T21:07:39.494798+0000","last_fullsized":"2024-04-16T21:07:39.494798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:04:27.697159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:07:14.490719+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:07:14.490719+0000","last_peered":"2024-04-16T21:07:14.490719+0000","last_clean":"2024-04-16T21:07:14.490719+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:07:14.490719+0000","last_undegraded":"2024-04-16T21:07:14.490719+0000","last_fullsized":"2024-04-16T21:07:14.490719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:10.072949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:08:34.663111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:08:34.663111+0000","last_peered":"2024-04-16T21:08:34.663111+0000","last_clean":"2024-04-16T21:08:34.663111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:08:34.663111+0000","last_undegraded":"2024-04-16T21:08:34.663111+0000","last_fullsized":"2024-04-16T21:08:34.663111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:37:32.821179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4203,"ondisk_log_size":4203,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3420,"ondisk_log_size":3420,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5660,"ondisk_log_size":5660,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641554,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41132,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816468,"statfs":{"total":107374182400,"available":107332063232,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641555,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641554,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55568,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802032,"statfs":{"total":107374182400,"available":107317280768,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641555,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59228,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798372,"statfs":{"total":107374182400,"available":107313532928,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641555,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41188,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816412,"statfs":{"total":107374182400,"available":107332005888,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641554,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55800,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104801800,"statfs":{"total":107374182400,"available":107317043200,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:11:44.947 INFO:tasks.ceph:pgid 9.35 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.948 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.948 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.948 INFO:tasks.ceph:pgid 9.34 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.948 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.948 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.949 INFO:tasks.ceph:pgid 9.37 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.949 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.949 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.949 INFO:tasks.ceph:pgid 9.36 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.949 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.950 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.950 INFO:tasks.ceph:pgid 9.31 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.950 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.950 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.950 INFO:tasks.ceph:pgid 9.30 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.951 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.951 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.951 INFO:tasks.ceph:pgid 9.33 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.951 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.951 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.952 INFO:tasks.ceph:pgid 9.32 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.952 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.952 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.952 INFO:tasks.ceph:pgid 9.3d last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.952 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 9.3c last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 9.3f last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.953 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.954 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.954 INFO:tasks.ceph:pgid 9.3e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.954 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.954 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.955 INFO:tasks.ceph:pgid 9.39 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.955 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.955 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.955 INFO:tasks.ceph:pgid 9.38 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 13.1d last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.956 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.957 INFO:tasks.ceph:pgid 11.1b last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.957 INFO:tasks.ceph:pgid 10.1a last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.957 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.957 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.957 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.958 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.958 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.958 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.958 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.958 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 13.1c last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 11.1a last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 10.1b last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.959 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.960 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.960 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.960 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.960 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.960 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 14.1c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 13.1f last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 12.1e last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.961 INFO:tasks.ceph:pgid 11.19 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.962 INFO:tasks.ceph:pgid 10.18 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.962 INFO:tasks.ceph:pgid 9.1b last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.962 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.962 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.962 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.963 INFO:tasks.ceph:pgid 13.1e last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.964 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.964 INFO:tasks.ceph:pgid 11.18 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.964 INFO:tasks.ceph:pgid 10.19 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.964 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.964 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.965 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.966 INFO:tasks.ceph:pgid 14.2 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.966 INFO:tasks.ceph:pgid 13.1 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.966 INFO:tasks.ceph:pgid 12.0 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.966 INFO:tasks.ceph:pgid 11.7 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.966 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.967 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.967 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.967 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.967 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.967 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 13.0 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.968 INFO:tasks.ceph:pgid 12.1 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.969 INFO:tasks.ceph:pgid 11.6 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.969 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.969 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.969 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.969 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.970 INFO:tasks.ceph:pgid 14.0 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.971 INFO:tasks.ceph:pgid 13.3 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.971 INFO:tasks.ceph:pgid 12.2 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.971 INFO:tasks.ceph:pgid 11.5 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.971 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.971 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.972 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.973 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.973 INFO:tasks.ceph:pgid 14.1 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.973 INFO:tasks.ceph:pgid 13.2 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.973 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.973 INFO:tasks.ceph:pgid 11.4 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.974 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.975 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.975 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.975 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.975 INFO:tasks.ceph:pgid 14.6 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.975 INFO:tasks.ceph:pgid 13.5 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 12.4 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 11.3 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.976 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.977 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.977 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.977 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.977 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.977 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.978 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.978 INFO:tasks.ceph:pgid 13.4 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.978 INFO:tasks.ceph:pgid 12.5 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.978 INFO:tasks.ceph:pgid 11.2 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.978 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.979 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.980 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.980 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.980 INFO:tasks.ceph:pgid 14.4 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.980 INFO:tasks.ceph:pgid 13.7 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.981 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.981 INFO:tasks.ceph:pgid 11.1 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.981 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.981 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.981 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.982 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.982 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.982 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.982 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.982 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.983 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.983 INFO:tasks.ceph:pgid 14.5 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.983 INFO:tasks.ceph:pgid 13.6 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.983 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.983 INFO:tasks.ceph:pgid 11.0 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.984 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 14.f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 13.c last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.985 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.986 INFO:tasks.ceph:pgid 11.a last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.986 INFO:tasks.ceph:pgid 10.b last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.986 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.986 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.987 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T19:23:45.108082+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.987 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.987 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.987 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.987 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.988 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.988 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.988 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.988 INFO:tasks.ceph:pgid 14.c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 13.f last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 11.9 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 10.8 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.989 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.990 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.990 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.990 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.990 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.990 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 13.e last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.991 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.992 INFO:tasks.ceph:pgid 10.9 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.992 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.992 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.992 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.992 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.993 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.994 INFO:tasks.ceph:pgid 13.d last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.994 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.994 INFO:tasks.ceph:pgid 11.b last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.994 INFO:tasks.ceph:pgid 10.a last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.994 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.995 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.996 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.996 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.996 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.996 INFO:tasks.ceph:pgid 13.b last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.996 INFO:tasks.ceph:pgid 12.a last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 11.d last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 10.c last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.997 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.998 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.998 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.998 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.998 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.998 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 14.9 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 13.a last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 11.c last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 10.d last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:44.999 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.000 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.000 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.000 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.000 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.000 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 14.a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 13.9 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.001 INFO:tasks.ceph:pgid 12.8 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.002 INFO:tasks.ceph:pgid 11.f last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.002 INFO:tasks.ceph:pgid 10.e last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.002 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.002 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.002 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.003 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.003 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.003 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.003 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.003 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.004 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.004 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.004 INFO:tasks.ceph:pgid 13.8 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.004 INFO:tasks.ceph:pgid 12.9 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.004 INFO:tasks.ceph:pgid 11.e last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.005 INFO:tasks.ceph:pgid 10.f last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.005 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.005 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.005 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.005 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.006 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.006 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.006 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.006 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.006 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.007 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.007 INFO:tasks.ceph:pgid 13.1a last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.007 INFO:tasks.ceph:pgid 12.1b last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.007 INFO:tasks.ceph:pgid 11.1c last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.007 INFO:tasks.ceph:pgid 10.1d last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.008 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.009 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.009 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.009 INFO:tasks.ceph:pgid 14.18 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.009 INFO:tasks.ceph:pgid 13.1b last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.009 INFO:tasks.ceph:pgid 12.1a last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.010 INFO:tasks.ceph:pgid 11.1d last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.010 INFO:tasks.ceph:pgid 10.1c last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.010 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.010 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.010 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.011 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.011 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.011 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.011 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.011 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 13.18 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 11.1e last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 10.1f last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.012 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.013 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.013 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.013 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.013 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.013 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.014 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.014 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.014 INFO:tasks.ceph:pgid 14.1a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.014 INFO:tasks.ceph:pgid 13.19 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.014 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.015 INFO:tasks.ceph:pgid 11.1f last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.015 INFO:tasks.ceph:pgid 10.1e last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.015 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.015 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.015 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.016 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.016 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.016 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.016 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.016 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.017 INFO:tasks.ceph:pgid 14.15 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.017 INFO:tasks.ceph:pgid 13.16 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.017 INFO:tasks.ceph:pgid 12.17 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.017 INFO:tasks.ceph:pgid 11.10 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.017 INFO:tasks.ceph:pgid 10.11 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.018 INFO:tasks.ceph:pgid 9.12 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.018 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.018 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.018 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.019 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.019 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.019 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.019 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.019 INFO:tasks.ceph:pgid 14.14 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.020 INFO:tasks.ceph:pgid 13.17 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.020 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.020 INFO:tasks.ceph:pgid 11.11 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.020 INFO:tasks.ceph:pgid 10.10 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.020 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.021 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.021 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.021 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.021 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.021 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 14.17 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 13.14 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 12.15 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.022 INFO:tasks.ceph:pgid 11.12 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.023 INFO:tasks.ceph:pgid 10.13 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.023 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.023 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.023 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.023 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.024 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.024 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.024 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.024 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.024 INFO:tasks.ceph:pgid 14.16 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 13.15 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 12.14 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 11.13 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 10.12 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.025 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.026 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.026 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.026 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.026 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.026 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 14.11 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 13.12 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.027 INFO:tasks.ceph:pgid 10.15 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.028 INFO:tasks.ceph:pgid 9.16 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.028 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.028 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.028 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.028 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.029 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.029 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.029 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.029 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.029 INFO:tasks.ceph:pgid 13.13 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 11.15 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 10.14 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 9.17 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.030 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.031 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.031 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.031 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.031 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.031 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 14.13 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 13.10 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 11.16 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 10.17 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.032 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.033 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.033 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.033 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.033 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.033 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.034 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.034 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.034 INFO:tasks.ceph:pgid 14.12 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.034 INFO:tasks.ceph:pgid 13.11 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.034 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.035 INFO:tasks.ceph:pgid 11.17 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.035 INFO:tasks.ceph:pgid 10.16 last_scrub_stamp 2024-04-16T19:25:10.445017+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=10, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.035 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.035 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.035 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 9.2a last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.036 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.037 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.037 INFO:tasks.ceph:pgid 9.2b last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.037 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.037 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.037 INFO:tasks.ceph:pgid 9.28 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.038 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.038 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.038 INFO:tasks.ceph:pgid 9.29 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.038 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.038 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 9.2e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 9.2f last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.039 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.040 INFO:tasks.ceph:pgid 9.2c last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.040 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.040 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.040 INFO:tasks.ceph:pgid 9.2d last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.040 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 9.22 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 9.23 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.041 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.042 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.042 INFO:tasks.ceph:pgid 9.20 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.042 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.042 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.042 INFO:tasks.ceph:pgid 9.21 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.043 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.043 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.043 INFO:tasks.ceph:pgid 9.26 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.043 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.043 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 9.27 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 9.24 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.044 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 9.25 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 9.3a last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.045 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.046 INFO:tasks.ceph:pgid 9.3b last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.046 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.046 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:11:45.046 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T21:12:05.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:12:05.405 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:12:05.425 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:12:05.427 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3296,"stamp":"2024-04-16T21:12:05.340823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31432,"ondisk_log_size":31432,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":316160,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":304950,"kb_avail":628829440,"statfs":{"total":644245094400,"available":643921346560,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":312269473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":138030,"num_omap_keys":704,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005347"},"pg_stats":[{"pgid":"9.35","version":"52'41","reported_seq":316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.577077+0000","last_change":"2024-04-16T21:11:52.577152+0000","last_active":"2024-04-16T21:11:52.577077+0000","last_peered":"2024-04-16T21:11:52.577077+0000","last_clean":"2024-04-16T21:11:52.577077+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:11:52.577077+0000","last_undegraded":"2024-04-16T21:11:52.577077+0000","last_fullsized":"2024-04-16T21:11:52.577077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'41","last_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_deep_scrub":"52'41","last_deep_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.577012+0000","objects_scrubbed":13,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:52:59.327819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.386010+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:10:49.386010+0000","last_peered":"2024-04-16T21:10:49.386010+0000","last_clean":"2024-04-16T21:10:49.386010+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:10:49.386010+0000","last_undegraded":"2024-04-16T21:10:49.386010+0000","last_fullsized":"2024-04-16T21:10:49.386010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373983+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:09:39.373983+0000","last_peered":"2024-04-16T21:09:39.373983+0000","last_clean":"2024-04-16T21:09:39.373983+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:09:39.373983+0000","last_undegraded":"2024-04-16T21:09:39.373983+0000","last_fullsized":"2024-04-16T21:09:39.373983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"52'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.722895+0000","last_change":"2024-04-16T21:11:56.722928+0000","last_active":"2024-04-16T21:11:56.722895+0000","last_peered":"2024-04-16T21:11:56.722895+0000","last_clean":"2024-04-16T21:11:56.722895+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:11:56.722895+0000","last_undegraded":"2024-04-16T21:11:56.722895+0000","last_fullsized":"2024-04-16T21:11:56.722895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.722838+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:40.375473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384626+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:10:49.384626+0000","last_peered":"2024-04-16T21:10:49.384626+0000","last_clean":"2024-04-16T21:10:49.384626+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:10:49.384626+0000","last_undegraded":"2024-04-16T21:10:49.384626+0000","last_fullsized":"2024-04-16T21:10:49.384626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.374230+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:09:39.374230+0000","last_peered":"2024-04-16T21:09:39.374230+0000","last_clean":"2024-04-16T21:09:39.374230+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:09:39.374230+0000","last_undegraded":"2024-04-16T21:09:39.374230+0000","last_fullsized":"2024-04-16T21:09:39.374230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'13","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107060+0000","last_change":"2024-04-16T19:25:09.457429+0000","last_active":"2024-04-16T21:11:44.107060+0000","last_peered":"2024-04-16T21:11:44.107060+0000","last_clean":"2024-04-16T21:11:44.107060+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:11:44.107060+0000","last_undegraded":"2024-04-16T21:11:44.107060+0000","last_fullsized":"2024-04-16T21:11:44.107060+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390655+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:10:49.390655+0000","last_peered":"2024-04-16T21:10:49.390655+0000","last_clean":"2024-04-16T21:10:49.390655+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:10:49.390655+0000","last_undegraded":"2024-04-16T21:10:49.390655+0000","last_fullsized":"2024-04-16T21:10:49.390655+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373743+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:09:39.373743+0000","last_peered":"2024-04-16T21:09:39.373743+0000","last_clean":"2024-04-16T21:09:39.373743+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:09:39.373743+0000","last_undegraded":"2024-04-16T21:09:39.373743+0000","last_fullsized":"2024-04-16T21:09:39.373743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'22","reported_seq":252,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.610592+0000","last_change":"2024-04-16T21:11:47.610645+0000","last_active":"2024-04-16T21:11:47.610592+0000","last_peered":"2024-04-16T21:11:47.610592+0000","last_clean":"2024-04-16T21:11:47.610592+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:11:47.610592+0000","last_undegraded":"2024-04-16T21:11:47.610592+0000","last_fullsized":"2024-04-16T21:11:47.610592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.610532+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:13.030892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386247+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:10:54.386247+0000","last_peered":"2024-04-16T21:10:54.386247+0000","last_clean":"2024-04-16T21:10:54.386247+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:10:54.386247+0000","last_undegraded":"2024-04-16T21:10:54.386247+0000","last_fullsized":"2024-04-16T21:10:54.386247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.378890+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:09:39.378890+0000","last_peered":"2024-04-16T21:09:39.378890+0000","last_clean":"2024-04-16T21:09:39.378890+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:09:39.378890+0000","last_undegraded":"2024-04-16T21:09:39.378890+0000","last_fullsized":"2024-04-16T21:09:39.378890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'12","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.719672+0000","last_change":"2024-04-16T21:11:58.719715+0000","last_active":"2024-04-16T21:11:58.719672+0000","last_peered":"2024-04-16T21:11:58.719672+0000","last_clean":"2024-04-16T21:11:58.719672+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:11:58.719672+0000","last_undegraded":"2024-04-16T21:11:58.719672+0000","last_fullsized":"2024-04-16T21:11:58.719672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.719610+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:50.501313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650162+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:11:43.650162+0000","last_peered":"2024-04-16T21:11:43.650162+0000","last_clean":"2024-04-16T21:11:43.650162+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:11:43.650162+0000","last_undegraded":"2024-04-16T21:11:43.650162+0000","last_fullsized":"2024-04-16T21:11:43.650162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387029+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:10:59.387029+0000","last_peered":"2024-04-16T21:10:59.387029+0000","last_clean":"2024-04-16T21:10:59.387029+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:10:59.387029+0000","last_undegraded":"2024-04-16T21:10:59.387029+0000","last_fullsized":"2024-04-16T21:10:59.387029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'27","reported_seq":223,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.166051+0000","last_change":"2024-04-16T21:11:51.166108+0000","last_active":"2024-04-16T21:11:51.166051+0000","last_peered":"2024-04-16T21:11:51.166051+0000","last_clean":"2024-04-16T21:11:51.166051+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:11:51.166051+0000","last_undegraded":"2024-04-16T21:11:51.166051+0000","last_fullsized":"2024-04-16T21:11:51.166051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.165987+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:01.812779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560300+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:11:44.560300+0000","last_peered":"2024-04-16T21:11:44.560300+0000","last_clean":"2024-04-16T21:11:44.560300+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:11:44.560300+0000","last_undegraded":"2024-04-16T21:11:44.560300+0000","last_fullsized":"2024-04-16T21:11:44.560300+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386392+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:10:54.386392+0000","last_peered":"2024-04-16T21:10:54.386392+0000","last_clean":"2024-04-16T21:10:54.386392+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:10:54.386392+0000","last_undegraded":"2024-04-16T21:10:54.386392+0000","last_fullsized":"2024-04-16T21:10:54.386392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'10","reported_seq":199,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107045+0000","last_change":"2024-04-16T19:25:41.508581+0000","last_active":"2024-04-16T21:11:44.107045+0000","last_peered":"2024-04-16T21:11:44.107045+0000","last_clean":"2024-04-16T21:11:44.107045+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:11:44.107045+0000","last_undegraded":"2024-04-16T21:11:44.107045+0000","last_fullsized":"2024-04-16T21:11:44.107045+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650158+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:11:43.650158+0000","last_peered":"2024-04-16T21:11:43.650158+0000","last_clean":"2024-04-16T21:11:43.650158+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:11:43.650158+0000","last_undegraded":"2024-04-16T21:11:43.650158+0000","last_fullsized":"2024-04-16T21:11:43.650158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386334+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:10:54.386334+0000","last_peered":"2024-04-16T21:10:54.386334+0000","last_clean":"2024-04-16T21:10:54.386334+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:10:54.386334+0000","last_undegraded":"2024-04-16T21:10:54.386334+0000","last_fullsized":"2024-04-16T21:10:54.386334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'22","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.654742+0000","last_change":"2024-04-16T21:11:56.654830+0000","last_active":"2024-04-16T21:11:56.654742+0000","last_peered":"2024-04-16T21:11:56.654742+0000","last_clean":"2024-04-16T21:11:56.654742+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:11:56.654742+0000","last_undegraded":"2024-04-16T21:11:56.654742+0000","last_fullsized":"2024-04-16T21:11:56.654742+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.654666+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.077668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194861+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:11:43.194861+0000","last_peered":"2024-04-16T21:11:43.194861+0000","last_clean":"2024-04-16T21:11:43.194861+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:11:43.194861+0000","last_undegraded":"2024-04-16T21:11:43.194861+0000","last_fullsized":"2024-04-16T21:11:43.194861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525521+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:10:54.525521+0000","last_peered":"2024-04-16T21:10:54.525521+0000","last_clean":"2024-04-16T21:10:54.525521+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:10:54.525521+0000","last_undegraded":"2024-04-16T21:10:54.525521+0000","last_fullsized":"2024-04-16T21:10:54.525521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'12","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.701548+0000","last_change":"2024-04-16T21:12:00.701594+0000","last_active":"2024-04-16T21:12:00.701548+0000","last_peered":"2024-04-16T21:12:00.701548+0000","last_clean":"2024-04-16T21:12:00.701548+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:12:00.701548+0000","last_undegraded":"2024-04-16T21:12:00.701548+0000","last_fullsized":"2024-04-16T21:12:00.701548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.701488+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:22.203073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106424+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:11:44.106424+0000","last_peered":"2024-04-16T21:11:44.106424+0000","last_clean":"2024-04-16T21:11:44.106424+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:11:44.106424+0000","last_undegraded":"2024-04-16T21:11:44.106424+0000","last_fullsized":"2024-04-16T21:11:44.106424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390309+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:11:14.390309+0000","last_peered":"2024-04-16T21:11:14.390309+0000","last_clean":"2024-04-16T21:11:14.390309+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:11:14.390309+0000","last_undegraded":"2024-04-16T21:11:14.390309+0000","last_fullsized":"2024-04-16T21:11:14.390309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'12","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.436184+0000","last_change":"2024-04-16T21:11:55.436228+0000","last_active":"2024-04-16T21:11:55.436184+0000","last_peered":"2024-04-16T21:11:55.436184+0000","last_clean":"2024-04-16T21:11:55.436184+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:11:55.436184+0000","last_undegraded":"2024-04-16T21:11:55.436184+0000","last_fullsized":"2024-04-16T21:11:55.436184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.436126+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:58.763911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560395+0000","last_change":"2024-04-16T19:25:41.503681+0000","last_active":"2024-04-16T21:11:44.560395+0000","last_peered":"2024-04-16T21:11:44.560395+0000","last_clean":"2024-04-16T21:11:44.560395+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:11:44.560395+0000","last_undegraded":"2024-04-16T21:11:44.560395+0000","last_fullsized":"2024-04-16T21:11:44.560395+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386917+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:10:54.386917+0000","last_peered":"2024-04-16T21:10:54.386917+0000","last_clean":"2024-04-16T21:10:54.386917+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:10:54.386917+0000","last_undegraded":"2024-04-16T21:10:54.386917+0000","last_fullsized":"2024-04-16T21:10:54.386917+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'10","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.882687+0000","last_change":"2024-04-16T21:11:51.882767+0000","last_active":"2024-04-16T21:11:51.882687+0000","last_peered":"2024-04-16T21:11:51.882687+0000","last_clean":"2024-04-16T21:11:51.882687+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:11:51.882687+0000","last_undegraded":"2024-04-16T21:11:51.882687+0000","last_fullsized":"2024-04-16T21:11:51.882687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.882609+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.077377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:59.377620+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:09:59.377620+0000","last_peered":"2024-04-16T21:09:59.377620+0000","last_clean":"2024-04-16T21:09:59.377620+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:09:59.377620+0000","last_undegraded":"2024-04-16T21:09:59.377620+0000","last_fullsized":"2024-04-16T21:09:59.377620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386292+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:10:54.386292+0000","last_peered":"2024-04-16T21:10:54.386292+0000","last_clean":"2024-04-16T21:10:54.386292+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:10:54.386292+0000","last_undegraded":"2024-04-16T21:10:54.386292+0000","last_fullsized":"2024-04-16T21:10:54.386292+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'10","reported_seq":231,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.530531+0000","last_change":"2024-04-16T19:25:09.463733+0000","last_active":"2024-04-16T21:11:24.530531+0000","last_peered":"2024-04-16T21:11:24.530531+0000","last_clean":"2024-04-16T21:11:24.530531+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:11:24.530531+0000","last_undegraded":"2024-04-16T21:11:24.530531+0000","last_fullsized":"2024-04-16T21:11:24.530531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526272+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:10:59.526272+0000","last_peered":"2024-04-16T21:10:59.526272+0000","last_clean":"2024-04-16T21:10:59.526272+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:10:59.526272+0000","last_undegraded":"2024-04-16T21:10:59.526272+0000","last_fullsized":"2024-04-16T21:10:59.526272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386361+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:10:54.386361+0000","last_peered":"2024-04-16T21:10:54.386361+0000","last_clean":"2024-04-16T21:10:54.386361+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:10:54.386361+0000","last_undegraded":"2024-04-16T21:10:54.386361+0000","last_fullsized":"2024-04-16T21:10:54.386361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'16","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.428126+0000","last_change":"2024-04-16T21:11:47.428185+0000","last_active":"2024-04-16T21:11:47.428126+0000","last_peered":"2024-04-16T21:11:47.428126+0000","last_clean":"2024-04-16T21:11:47.428126+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:11:47.428126+0000","last_undegraded":"2024-04-16T21:11:47.428126+0000","last_fullsized":"2024-04-16T21:11:47.428126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.428059+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:23.278965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526292+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:10:59.526292+0000","last_peered":"2024-04-16T21:10:59.526292+0000","last_clean":"2024-04-16T21:10:59.526292+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:10:59.526292+0000","last_undegraded":"2024-04-16T21:10:59.526292+0000","last_fullsized":"2024-04-16T21:10:59.526292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391334+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:10:54.391334+0000","last_peered":"2024-04-16T21:10:54.391334+0000","last_clean":"2024-04-16T21:10:54.391334+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:10:54.391334+0000","last_undegraded":"2024-04-16T21:10:54.391334+0000","last_fullsized":"2024-04-16T21:10:54.391334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'15","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.518198+0000","last_change":"2024-04-16T19:25:09.461082+0000","last_active":"2024-04-16T21:10:09.518198+0000","last_peered":"2024-04-16T21:10:09.518198+0000","last_clean":"2024-04-16T21:10:09.518198+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:10:09.518198+0000","last_undegraded":"2024-04-16T21:10:09.518198+0000","last_fullsized":"2024-04-16T21:10:09.518198+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387165+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:10:59.387165+0000","last_peered":"2024-04-16T21:10:59.387165+0000","last_clean":"2024-04-16T21:10:59.387165+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:10:59.387165+0000","last_undegraded":"2024-04-16T21:10:59.387165+0000","last_fullsized":"2024-04-16T21:10:59.387165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.381557+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:09:54.381557+0000","last_peered":"2024-04-16T21:09:54.381557+0000","last_clean":"2024-04-16T21:09:54.381557+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:09:54.381557+0000","last_undegraded":"2024-04-16T21:09:54.381557+0000","last_fullsized":"2024-04-16T21:09:54.381557+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385608+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:10:54.385608+0000","last_peered":"2024-04-16T21:10:54.385608+0000","last_clean":"2024-04-16T21:10:54.385608+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:10:54.385608+0000","last_undegraded":"2024-04-16T21:10:54.385608+0000","last_fullsized":"2024-04-16T21:10:54.385608+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1721,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.379450+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:10:09.379450+0000","last_peered":"2024-04-16T21:10:09.379450+0000","last_clean":"2024-04-16T21:10:09.379450+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:10:09.379450+0000","last_undegraded":"2024-04-16T21:10:09.379450+0000","last_fullsized":"2024-04-16T21:10:09.379450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.385091+0000","last_change":"2024-04-16T19:25:41.505822+0000","last_active":"2024-04-16T21:10:44.385091+0000","last_peered":"2024-04-16T21:10:44.385091+0000","last_clean":"2024-04-16T21:10:44.385091+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:10:44.385091+0000","last_undegraded":"2024-04-16T21:10:44.385091+0000","last_fullsized":"2024-04-16T21:10:44.385091+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'6","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385619+0000","last_change":"2024-04-16T19:25:15.486584+0000","last_active":"2024-04-16T21:10:19.385619+0000","last_peered":"2024-04-16T21:10:19.385619+0000","last_clean":"2024-04-16T21:10:19.385619+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:10:19.385619+0000","last_undegraded":"2024-04-16T21:10:19.385619+0000","last_fullsized":"2024-04-16T21:10:19.385619+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.738443+0000","last_change":"2024-04-16T21:11:45.738465+0000","last_active":"2024-04-16T21:11:45.738443+0000","last_peered":"2024-04-16T21:11:45.738443+0000","last_clean":"2024-04-16T21:11:45.738443+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:11:45.738443+0000","last_undegraded":"2024-04-16T21:11:45.738443+0000","last_fullsized":"2024-04-16T21:11:45.738443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.738344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:26.908264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'223","reported_seq":476,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.490112+0000","last_change":"2024-04-16T21:11:58.490168+0000","last_active":"2024-04-16T21:11:58.490112+0000","last_peered":"2024-04-16T21:11:58.490112+0000","last_clean":"2024-04-16T21:11:58.490112+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:11:58.490112+0000","last_undegraded":"2024-04-16T21:11:58.490112+0000","last_fullsized":"2024-04-16T21:11:58.490112+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'223","last_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_deep_scrub":"52'223","last_deep_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.490053+0000","objects_scrubbed":20,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:48.708953+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1971,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'29","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.926237+0000","last_change":"2024-04-16T21:11:50.926284+0000","last_active":"2024-04-16T21:11:50.926237+0000","last_peered":"2024-04-16T21:11:50.926237+0000","last_clean":"2024-04-16T21:11:50.926237+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:11:50.926237+0000","last_undegraded":"2024-04-16T21:11:50.926237+0000","last_fullsized":"2024-04-16T21:11:50.926237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'29","last_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_deep_scrub":"52'29","last_deep_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.926173+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:17.509632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.389403+0000","last_change":"2024-04-16T19:24:51.436483+0000","last_active":"2024-04-16T21:11:14.389403+0000","last_peered":"2024-04-16T21:11:14.389403+0000","last_clean":"2024-04-16T21:11:14.389403+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:11:14.389403+0000","last_undegraded":"2024-04-16T21:11:14.389403+0000","last_fullsized":"2024-04-16T21:11:14.389403+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:59.376613+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:09:59.376613+0000","last_peered":"2024-04-16T21:09:59.376613+0000","last_clean":"2024-04-16T21:09:59.376613+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:09:59.376613+0000","last_undegraded":"2024-04-16T21:09:59.376613+0000","last_fullsized":"2024-04-16T21:09:59.376613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649927+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:11:43.649927+0000","last_peered":"2024-04-16T21:11:43.649927+0000","last_clean":"2024-04-16T21:11:43.649927+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:11:43.649927+0000","last_undegraded":"2024-04-16T21:11:43.649927+0000","last_fullsized":"2024-04-16T21:11:43.649927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388585+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:11:09.388585+0000","last_peered":"2024-04-16T21:11:09.388585+0000","last_clean":"2024-04-16T21:11:09.388585+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:11:09.388585+0000","last_undegraded":"2024-04-16T21:11:09.388585+0000","last_fullsized":"2024-04-16T21:11:09.388585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.394046+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:11:09.394046+0000","last_peered":"2024-04-16T21:11:09.394046+0000","last_clean":"2024-04-16T21:11:09.394046+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:11:09.394046+0000","last_undegraded":"2024-04-16T21:11:09.394046+0000","last_fullsized":"2024-04-16T21:11:09.394046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:11:09.388641+0000","last_peered":"2024-04-16T21:11:09.388641+0000","last_clean":"2024-04-16T21:11:09.388641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:11:09.388641+0000","last_undegraded":"2024-04-16T21:11:09.388641+0000","last_fullsized":"2024-04-16T21:11:09.388641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.196037+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:11:43.196037+0000","last_peered":"2024-04-16T21:11:43.196037+0000","last_clean":"2024-04-16T21:11:43.196037+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:11:43.196037+0000","last_undegraded":"2024-04-16T21:11:43.196037+0000","last_fullsized":"2024-04-16T21:11:43.196037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1837,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386348+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:10:54.386348+0000","last_peered":"2024-04-16T21:10:54.386348+0000","last_clean":"2024-04-16T21:10:54.386348+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:10:54.386348+0000","last_undegraded":"2024-04-16T21:10:54.386348+0000","last_fullsized":"2024-04-16T21:10:54.386348+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384475+0000","last_change":"2024-04-16T19:25:41.503686+0000","last_active":"2024-04-16T21:10:44.384475+0000","last_peered":"2024-04-16T21:10:44.384475+0000","last_clean":"2024-04-16T21:10:44.384475+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:10:44.384475+0000","last_undegraded":"2024-04-16T21:10:44.384475+0000","last_fullsized":"2024-04-16T21:10:44.384475+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'5","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.385140+0000","last_change":"2024-04-16T19:25:41.505903+0000","last_active":"2024-04-16T21:10:44.385140+0000","last_peered":"2024-04-16T21:10:44.385140+0000","last_clean":"2024-04-16T21:10:44.385140+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:10:44.385140+0000","last_undegraded":"2024-04-16T21:10:44.385140+0000","last_fullsized":"2024-04-16T21:10:44.385140+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.686963+0000","last_change":"2024-04-16T21:11:57.686990+0000","last_active":"2024-04-16T21:11:57.686963+0000","last_peered":"2024-04-16T21:11:57.686963+0000","last_clean":"2024-04-16T21:11:57.686963+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:11:57.686963+0000","last_undegraded":"2024-04-16T21:11:57.686963+0000","last_fullsized":"2024-04-16T21:11:57.686963+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.686888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:39.802344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'264","reported_seq":1008,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.984736+0000","last_change":"2024-04-16T21:11:43.984786+0000","last_active":"2024-04-16T21:11:43.984736+0000","last_peered":"2024-04-16T21:11:43.984736+0000","last_clean":"2024-04-16T21:11:43.984736+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:11:43.984736+0000","last_undegraded":"2024-04-16T21:11:43.984736+0000","last_fullsized":"2024-04-16T21:11:43.984736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.984688+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:21.448843+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6375,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'10","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.702465+0000","last_change":"2024-04-16T21:11:54.702515+0000","last_active":"2024-04-16T21:11:54.702465+0000","last_peered":"2024-04-16T21:11:54.702465+0000","last_clean":"2024-04-16T21:11:54.702465+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:11:54.702465+0000","last_undegraded":"2024-04-16T21:11:54.702465+0000","last_fullsized":"2024-04-16T21:11:54.702465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.702406+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.795058+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392588+0000","last_change":"2024-04-16T19:24:51.438136+0000","last_active":"2024-04-16T21:11:34.392588+0000","last_peered":"2024-04-16T21:11:34.392588+0000","last_clean":"2024-04-16T21:11:34.392588+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:11:34.392588+0000","last_undegraded":"2024-04-16T21:11:34.392588+0000","last_fullsized":"2024-04-16T21:11:34.392588+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387060+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:10:59.387060+0000","last_peered":"2024-04-16T21:10:59.387060+0000","last_clean":"2024-04-16T21:10:59.387060+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:10:59.387060+0000","last_undegraded":"2024-04-16T21:10:59.387060+0000","last_fullsized":"2024-04-16T21:10:59.387060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291858+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:11:42.291858+0000","last_peered":"2024-04-16T21:11:42.291858+0000","last_clean":"2024-04-16T21:11:42.291858+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:11:42.291858+0000","last_undegraded":"2024-04-16T21:11:42.291858+0000","last_fullsized":"2024-04-16T21:11:42.291858+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.384073+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:10:09.384073+0000","last_peered":"2024-04-16T21:10:09.384073+0000","last_clean":"2024-04-16T21:10:09.384073+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:10:09.384073+0000","last_undegraded":"2024-04-16T21:10:09.384073+0000","last_fullsized":"2024-04-16T21:10:09.384073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388791+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:11:09.388791+0000","last_peered":"2024-04-16T21:11:09.388791+0000","last_clean":"2024-04-16T21:11:09.388791+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:11:09.388791+0000","last_undegraded":"2024-04-16T21:11:09.388791+0000","last_fullsized":"2024-04-16T21:11:09.388791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388773+0000","last_change":"2024-04-16T19:24:49.426639+0000","last_active":"2024-04-16T21:11:09.388773+0000","last_peered":"2024-04-16T21:11:09.388773+0000","last_clean":"2024-04-16T21:11:09.388773+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:11:09.388773+0000","last_undegraded":"2024-04-16T21:11:09.388773+0000","last_fullsized":"2024-04-16T21:11:09.388773+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377974+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:10:04.377974+0000","last_peered":"2024-04-16T21:10:04.377974+0000","last_clean":"2024-04-16T21:10:04.377974+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:10:04.377974+0000","last_undegraded":"2024-04-16T21:10:04.377974+0000","last_fullsized":"2024-04-16T21:10:04.377974+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1872,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390599+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:10:49.390599+0000","last_peered":"2024-04-16T21:10:49.390599+0000","last_clean":"2024-04-16T21:10:49.390599+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:10:49.390599+0000","last_undegraded":"2024-04-16T21:10:49.390599+0000","last_fullsized":"2024-04-16T21:10:49.390599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.746629+0000","last_change":"2024-04-16T21:12:00.746651+0000","last_active":"2024-04-16T21:12:00.746629+0000","last_peered":"2024-04-16T21:12:00.746629+0000","last_clean":"2024-04-16T21:12:00.746629+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:12:00.746629+0000","last_undegraded":"2024-04-16T21:12:00.746629+0000","last_fullsized":"2024-04-16T21:12:00.746629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.746571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:24:45.291359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'25","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.693813+0000","last_change":"2024-04-16T21:11:57.693880+0000","last_active":"2024-04-16T21:11:57.693813+0000","last_peered":"2024-04-16T21:11:57.693813+0000","last_clean":"2024-04-16T21:11:57.693813+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:11:57.693813+0000","last_undegraded":"2024-04-16T21:11:57.693813+0000","last_fullsized":"2024-04-16T21:11:57.693813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'25","last_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_deep_scrub":"52'25","last_deep_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.693743+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.739736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.385829+0000","last_change":"2024-04-16T21:12:04.385854+0000","last_active":"2024-04-16T21:12:04.385829+0000","last_peered":"2024-04-16T21:12:04.385829+0000","last_clean":"2024-04-16T21:12:04.385829+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:12:04.385829+0000","last_undegraded":"2024-04-16T21:12:04.385829+0000","last_fullsized":"2024-04-16T21:12:04.385829+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.385746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:20.771679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'716","reported_seq":41938,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.996557+0000","last_change":"2024-04-16T21:11:45.996631+0000","last_active":"2024-04-16T21:11:45.996557+0000","last_peered":"2024-04-16T21:11:45.996557+0000","last_clean":"2024-04-16T21:11:45.996557+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:11:45.996557+0000","last_undegraded":"2024-04-16T21:11:45.996557+0000","last_fullsized":"2024-04-16T21:11:45.996557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'716","last_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_deep_scrub":"52'716","last_deep_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.996464+0000","objects_scrubbed":20,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:40.351620+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'35","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.105334+0000","last_change":"2024-04-16T21:11:44.105372+0000","last_active":"2024-04-16T21:11:44.105334+0000","last_peered":"2024-04-16T21:11:44.105334+0000","last_clean":"2024-04-16T21:11:44.105334+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:11:44.105334+0000","last_undegraded":"2024-04-16T21:11:44.105334+0000","last_fullsized":"2024-04-16T21:11:44.105334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'35","last_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_deep_scrub":"52'35","last_deep_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.105287+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:44.283094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.409213+0000","last_change":"2024-04-16T21:12:03.409254+0000","last_active":"2024-04-16T21:12:03.409213+0000","last_peered":"2024-04-16T21:12:03.409213+0000","last_clean":"2024-04-16T21:12:03.409213+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:12:03.409213+0000","last_undegraded":"2024-04-16T21:12:03.409213+0000","last_fullsized":"2024-04-16T21:12:03.409213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.409137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:49.655116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392318+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:10:59.392318+0000","last_peered":"2024-04-16T21:10:59.392318+0000","last_clean":"2024-04-16T21:10:59.392318+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:10:59.392318+0000","last_undegraded":"2024-04-16T21:10:59.392318+0000","last_fullsized":"2024-04-16T21:10:59.392318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649938+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:11:43.649938+0000","last_peered":"2024-04-16T21:11:43.649938+0000","last_clean":"2024-04-16T21:11:43.649938+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:11:43.649938+0000","last_undegraded":"2024-04-16T21:11:43.649938+0000","last_fullsized":"2024-04-16T21:11:43.649938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.528197+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:11:09.528197+0000","last_peered":"2024-04-16T21:11:09.528197+0000","last_clean":"2024-04-16T21:11:09.528197+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:11:09.528197+0000","last_undegraded":"2024-04-16T21:11:09.528197+0000","last_fullsized":"2024-04-16T21:11:09.528197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":275,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.517540+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:10:04.517540+0000","last_peered":"2024-04-16T21:10:04.517540+0000","last_clean":"2024-04-16T21:10:04.517540+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:10:04.517540+0000","last_undegraded":"2024-04-16T21:10:04.517540+0000","last_fullsized":"2024-04-16T21:10:04.517540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'12","reported_seq":1309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389479+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:11:09.389479+0000","last_peered":"2024-04-16T21:11:09.389479+0000","last_clean":"2024-04-16T21:11:09.389479+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:11:09.389479+0000","last_undegraded":"2024-04-16T21:11:09.389479+0000","last_fullsized":"2024-04-16T21:11:09.389479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377342+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:10:04.377342+0000","last_peered":"2024-04-16T21:10:04.377342+0000","last_clean":"2024-04-16T21:10:04.377342+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:10:04.377342+0000","last_undegraded":"2024-04-16T21:10:04.377342+0000","last_fullsized":"2024-04-16T21:10:04.377342+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1473,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390635+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:10:49.390635+0000","last_peered":"2024-04-16T21:10:49.390635+0000","last_clean":"2024-04-16T21:10:49.390635+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:10:49.390635+0000","last_undegraded":"2024-04-16T21:10:49.390635+0000","last_fullsized":"2024-04-16T21:10:49.390635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.703764+0000","last_change":"2024-04-16T21:12:02.703764+0000","last_active":"2024-04-16T21:12:02.703764+0000","last_peered":"2024-04-16T21:12:02.703764+0000","last_clean":"2024-04-16T21:12:02.703764+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:12:02.703764+0000","last_undegraded":"2024-04-16T21:12:02.703764+0000","last_fullsized":"2024-04-16T21:12:02.703764+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.703681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:59.311639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'9","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.393532+0000","last_change":"2024-04-16T19:25:15.487126+0000","last_active":"2024-04-16T21:11:34.393532+0000","last_peered":"2024-04-16T21:11:34.393532+0000","last_clean":"2024-04-16T21:11:34.393532+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:11:34.393532+0000","last_undegraded":"2024-04-16T21:11:34.393532+0000","last_fullsized":"2024-04-16T21:11:34.393532+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.821906+0000","last_change":"2024-04-16T21:11:59.821929+0000","last_active":"2024-04-16T21:11:59.821906+0000","last_peered":"2024-04-16T21:11:59.821906+0000","last_clean":"2024-04-16T21:11:59.821906+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:11:59.821906+0000","last_undegraded":"2024-04-16T21:11:59.821906+0000","last_fullsized":"2024-04-16T21:11:59.821906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.821840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.493143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'366","reported_seq":690,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.574732+0000","last_change":"2024-04-16T21:11:46.574732+0000","last_active":"2024-04-16T21:11:46.574732+0000","last_peered":"2024-04-16T21:11:46.574732+0000","last_clean":"2024-04-16T21:11:46.574732+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:11:46.574732+0000","last_undegraded":"2024-04-16T21:11:46.574732+0000","last_fullsized":"2024-04-16T21:11:46.574732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'366","last_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_deep_scrub":"52'366","last_deep_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.574641+0000","objects_scrubbed":27,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:24.475352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'12","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.580757+0000","last_change":"2024-04-16T21:11:44.580812+0000","last_active":"2024-04-16T21:11:44.580757+0000","last_peered":"2024-04-16T21:11:44.580757+0000","last_clean":"2024-04-16T21:11:44.580757+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:11:44.580757+0000","last_undegraded":"2024-04-16T21:11:44.580757+0000","last_fullsized":"2024-04-16T21:11:44.580757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.580695+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:57.775463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.376395+0000","last_change":"2024-04-16T19:24:51.572963+0000","last_active":"2024-04-16T21:09:54.376395+0000","last_peered":"2024-04-16T21:09:54.376395+0000","last_clean":"2024-04-16T21:09:54.376395+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:09:54.376395+0000","last_undegraded":"2024-04-16T21:09:54.376395+0000","last_fullsized":"2024-04-16T21:09:54.376395+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392386+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:10:59.392386+0000","last_peered":"2024-04-16T21:10:59.392386+0000","last_clean":"2024-04-16T21:10:59.392386+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:10:59.392386+0000","last_undegraded":"2024-04-16T21:10:59.392386+0000","last_fullsized":"2024-04-16T21:10:59.392386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560171+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:11:44.560171+0000","last_peered":"2024-04-16T21:11:44.560171+0000","last_clean":"2024-04-16T21:11:44.560171+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:11:44.560171+0000","last_undegraded":"2024-04-16T21:11:44.560171+0000","last_fullsized":"2024-04-16T21:11:44.560171+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389520+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:11:09.389520+0000","last_peered":"2024-04-16T21:11:09.389520+0000","last_clean":"2024-04-16T21:11:09.389520+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:11:09.389520+0000","last_undegraded":"2024-04-16T21:11:09.389520+0000","last_fullsized":"2024-04-16T21:11:09.389520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389466+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:11:09.389466+0000","last_peered":"2024-04-16T21:11:09.389466+0000","last_clean":"2024-04-16T21:11:09.389466+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:11:09.389466+0000","last_undegraded":"2024-04-16T21:11:09.389466+0000","last_fullsized":"2024-04-16T21:11:09.389466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.378158+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:10:09.378158+0000","last_peered":"2024-04-16T21:10:09.378158+0000","last_clean":"2024-04-16T21:10:09.378158+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:10:09.378158+0000","last_undegraded":"2024-04-16T21:10:09.378158+0000","last_fullsized":"2024-04-16T21:10:09.378158+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386322+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:10:54.386322+0000","last_peered":"2024-04-16T21:10:54.386322+0000","last_clean":"2024-04-16T21:10:54.386322+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:10:54.386322+0000","last_undegraded":"2024-04-16T21:10:54.386322+0000","last_fullsized":"2024-04-16T21:10:54.386322+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2008,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651384+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:11:43.651384+0000","last_peered":"2024-04-16T21:11:43.651384+0000","last_clean":"2024-04-16T21:11:43.651384+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:11:43.651384+0000","last_undegraded":"2024-04-16T21:11:43.651384+0000","last_fullsized":"2024-04-16T21:11:43.651384+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651269+0000","last_change":"2024-04-16T19:25:41.506037+0000","last_active":"2024-04-16T21:11:43.651269+0000","last_peered":"2024-04-16T21:11:43.651269+0000","last_clean":"2024-04-16T21:11:43.651269+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:11:43.651269+0000","last_undegraded":"2024-04-16T21:11:43.651269+0000","last_fullsized":"2024-04-16T21:11:43.651269+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'33","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.393624+0000","last_change":"2024-04-16T19:25:15.482185+0000","last_active":"2024-04-16T21:11:39.393624+0000","last_peered":"2024-04-16T21:11:39.393624+0000","last_clean":"2024-04-16T21:11:39.393624+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:11:39.393624+0000","last_undegraded":"2024-04-16T21:11:39.393624+0000","last_fullsized":"2024-04-16T21:11:39.393624+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.391079+0000","last_change":"2024-04-16T19:25:13.475047+0000","last_active":"2024-04-16T21:11:24.391079+0000","last_peered":"2024-04-16T21:11:24.391079+0000","last_clean":"2024-04-16T21:11:24.391079+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:11:24.391079+0000","last_undegraded":"2024-04-16T21:11:24.391079+0000","last_fullsized":"2024-04-16T21:11:24.391079+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1762","reported_seq":4632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.707742+0000","last_change":"2024-04-16T21:11:51.707808+0000","last_active":"2024-04-16T21:11:51.707742+0000","last_peered":"2024-04-16T21:11:51.707742+0000","last_clean":"2024-04-16T21:11:51.707742+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:11:51.707742+0000","last_undegraded":"2024-04-16T21:11:51.707742+0000","last_fullsized":"2024-04-16T21:11:51.707742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1762","last_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_deep_scrub":"52'1762","last_deep_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.707684+0000","objects_scrubbed":22,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:25.851168+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":89480,"num_omap_keys":462,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'27","reported_seq":288,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.189093+0000","last_change":"2024-04-16T21:11:48.189147+0000","last_active":"2024-04-16T21:11:48.189093+0000","last_peered":"2024-04-16T21:11:48.189093+0000","last_clean":"2024-04-16T21:11:48.189093+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:11:48.189093+0000","last_undegraded":"2024-04-16T21:11:48.189093+0000","last_fullsized":"2024-04-16T21:11:48.189093+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.189029+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:30.304485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385804+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:10:49.385804+0000","last_peered":"2024-04-16T21:10:49.385804+0000","last_clean":"2024-04-16T21:10:49.385804+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:10:49.385804+0000","last_undegraded":"2024-04-16T21:10:49.385804+0000","last_fullsized":"2024-04-16T21:10:49.385804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.383849+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:10:39.383849+0000","last_peered":"2024-04-16T21:10:39.383849+0000","last_clean":"2024-04-16T21:10:39.383849+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:10:39.383849+0000","last_undegraded":"2024-04-16T21:10:39.383849+0000","last_fullsized":"2024-04-16T21:10:39.383849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.386931+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:10:59.386931+0000","last_peered":"2024-04-16T21:10:59.386931+0000","last_clean":"2024-04-16T21:10:59.386931+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:10:59.386931+0000","last_undegraded":"2024-04-16T21:10:59.386931+0000","last_fullsized":"2024-04-16T21:10:59.386931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":395,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.390148+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:11:19.390148+0000","last_peered":"2024-04-16T21:11:19.390148+0000","last_clean":"2024-04-16T21:11:19.390148+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:11:19.390148+0000","last_undegraded":"2024-04-16T21:11:19.390148+0000","last_fullsized":"2024-04-16T21:11:19.390148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382059+0000","last_change":"2024-04-16T19:24:49.434497+0000","last_active":"2024-04-16T21:10:34.382059+0000","last_peered":"2024-04-16T21:10:34.382059+0000","last_clean":"2024-04-16T21:10:34.382059+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:10:34.382059+0000","last_undegraded":"2024-04-16T21:10:34.382059+0000","last_fullsized":"2024-04-16T21:10:34.382059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388546+0000","last_change":"2024-04-16T19:24:51.437122+0000","last_active":"2024-04-16T21:11:09.388546+0000","last_peered":"2024-04-16T21:11:09.388546+0000","last_clean":"2024-04-16T21:11:09.388546+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:11:09.388546+0000","last_undegraded":"2024-04-16T21:11:09.388546+0000","last_fullsized":"2024-04-16T21:11:09.388546+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525424+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:10:54.525424+0000","last_peered":"2024-04-16T21:10:54.525424+0000","last_clean":"2024-04-16T21:10:54.525424+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:10:54.525424+0000","last_undegraded":"2024-04-16T21:10:54.525424+0000","last_fullsized":"2024-04-16T21:10:54.525424+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293176+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:11:42.293176+0000","last_peered":"2024-04-16T21:11:42.293176+0000","last_clean":"2024-04-16T21:11:42.293176+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:11:42.293176+0000","last_undegraded":"2024-04-16T21:11:42.293176+0000","last_fullsized":"2024-04-16T21:11:42.293176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651262+0000","last_change":"2024-04-16T19:25:41.502674+0000","last_active":"2024-04-16T21:11:43.651262+0000","last_peered":"2024-04-16T21:11:43.651262+0000","last_clean":"2024-04-16T21:11:43.651262+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:11:43.651262+0000","last_undegraded":"2024-04-16T21:11:43.651262+0000","last_fullsized":"2024-04-16T21:11:43.651262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'13","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.393250+0000","last_change":"2024-04-16T19:25:15.485057+0000","last_active":"2024-04-16T21:11:34.393250+0000","last_peered":"2024-04-16T21:11:34.393250+0000","last_clean":"2024-04-16T21:11:34.393250+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:11:34.393250+0000","last_undegraded":"2024-04-16T21:11:34.393250+0000","last_fullsized":"2024-04-16T21:11:34.393250+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.377316+0000","last_change":"2024-04-16T21:12:02.377338+0000","last_active":"2024-04-16T21:12:02.377316+0000","last_peered":"2024-04-16T21:12:02.377316+0000","last_clean":"2024-04-16T21:12:02.377316+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:12:02.377316+0000","last_undegraded":"2024-04-16T21:12:02.377316+0000","last_fullsized":"2024-04-16T21:12:02.377316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.377250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:49.251281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'325","reported_seq":659,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.419682+0000","last_change":"2024-04-16T21:11:56.419734+0000","last_active":"2024-04-16T21:11:56.419682+0000","last_peered":"2024-04-16T21:11:56.419682+0000","last_clean":"2024-04-16T21:11:56.419682+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:11:56.419682+0000","last_undegraded":"2024-04-16T21:11:56.419682+0000","last_fullsized":"2024-04-16T21:11:56.419682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'325","last_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_deep_scrub":"52'325","last_deep_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.419626+0000","objects_scrubbed":21,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:24.477776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6361,"num_omap_keys":29,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'16","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.568997+0000","last_change":"2024-04-16T21:11:45.569097+0000","last_active":"2024-04-16T21:11:45.568997+0000","last_peered":"2024-04-16T21:11:45.568997+0000","last_clean":"2024-04-16T21:11:45.568997+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:11:45.568997+0000","last_undegraded":"2024-04-16T21:11:45.568997+0000","last_fullsized":"2024-04-16T21:11:45.568997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.568923+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:31.758184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385305+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:10:49.385305+0000","last_peered":"2024-04-16T21:10:49.385305+0000","last_clean":"2024-04-16T21:10:49.385305+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:10:49.385305+0000","last_undegraded":"2024-04-16T21:10:49.385305+0000","last_fullsized":"2024-04-16T21:10:49.385305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380988+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:10:19.380988+0000","last_peered":"2024-04-16T21:10:19.380988+0000","last_clean":"2024-04-16T21:10:19.380988+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:10:19.380988+0000","last_undegraded":"2024-04-16T21:10:19.380988+0000","last_fullsized":"2024-04-16T21:10:19.380988+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":960,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526178+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:10:59.526178+0000","last_peered":"2024-04-16T21:10:59.526178+0000","last_clean":"2024-04-16T21:10:59.526178+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:10:59.526178+0000","last_undegraded":"2024-04-16T21:10:59.526178+0000","last_fullsized":"2024-04-16T21:10:59.526178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382034+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:10:34.382034+0000","last_peered":"2024-04-16T21:10:34.382034+0000","last_clean":"2024-04-16T21:10:34.382034+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:10:34.382034+0000","last_undegraded":"2024-04-16T21:10:34.382034+0000","last_fullsized":"2024-04-16T21:10:34.382034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.528852+0000","last_change":"2024-04-16T19:24:49.429526+0000","last_active":"2024-04-16T21:11:14.528852+0000","last_peered":"2024-04-16T21:11:14.528852+0000","last_clean":"2024-04-16T21:11:14.528852+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:11:14.528852+0000","last_undegraded":"2024-04-16T21:11:14.528852+0000","last_fullsized":"2024-04-16T21:11:14.528852+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391803+0000","last_change":"2024-04-16T19:24:51.437305+0000","last_active":"2024-04-16T21:11:29.391803+0000","last_peered":"2024-04-16T21:11:29.391803+0000","last_clean":"2024-04-16T21:11:29.391803+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:11:29.391803+0000","last_undegraded":"2024-04-16T21:11:29.391803+0000","last_fullsized":"2024-04-16T21:11:29.391803+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386715+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:10:54.386715+0000","last_peered":"2024-04-16T21:10:54.386715+0000","last_clean":"2024-04-16T21:10:54.386715+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:10:54.386715+0000","last_undegraded":"2024-04-16T21:10:54.386715+0000","last_fullsized":"2024-04-16T21:10:54.386715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'93","reported_seq":1380,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195890+0000","last_change":"2024-04-16T19:26:41.564908+0000","last_active":"2024-04-16T21:11:43.195890+0000","last_peered":"2024-04-16T21:11:43.195890+0000","last_clean":"2024-04-16T21:11:43.195890+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:11:43.195890+0000","last_undegraded":"2024-04-16T21:11:43.195890+0000","last_fullsized":"2024-04-16T21:11:43.195890+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293084+0000","last_change":"2024-04-16T19:25:41.508929+0000","last_active":"2024-04-16T21:11:42.293084+0000","last_peered":"2024-04-16T21:11:42.293084+0000","last_clean":"2024-04-16T21:11:42.293084+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:11:42.293084+0000","last_undegraded":"2024-04-16T21:11:42.293084+0000","last_fullsized":"2024-04-16T21:11:42.293084+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'19","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.393326+0000","last_change":"2024-04-16T19:25:15.482026+0000","last_active":"2024-04-16T21:11:34.393326+0000","last_peered":"2024-04-16T21:11:34.393326+0000","last_clean":"2024-04-16T21:11:34.393326+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:11:34.393326+0000","last_undegraded":"2024-04-16T21:11:34.393326+0000","last_fullsized":"2024-04-16T21:11:34.393326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.647510+0000","last_change":"2024-04-16T21:11:51.647550+0000","last_active":"2024-04-16T21:11:51.647510+0000","last_peered":"2024-04-16T21:11:51.647510+0000","last_clean":"2024-04-16T21:11:51.647510+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:11:51.647510+0000","last_undegraded":"2024-04-16T21:11:51.647510+0000","last_fullsized":"2024-04-16T21:11:51.647510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.647441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:25.536915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2068","reported_seq":5847,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.355619+0000","last_change":"2024-04-16T21:11:58.355713+0000","last_active":"2024-04-16T21:11:58.355619+0000","last_peered":"2024-04-16T21:11:58.355619+0000","last_clean":"2024-04-16T21:11:58.355619+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:11:58.355619+0000","last_undegraded":"2024-04-16T21:11:58.355619+0000","last_fullsized":"2024-04-16T21:11:58.355619+0000","mapping_epoch":33,"log_start":"47'400","ondisk_log_start":"47'400","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2068","last_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_deep_scrub":"52'2068","last_deep_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.355545+0000","objects_scrubbed":24,"log_size":1668,"log_dups_size":400,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:46.390834+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16267,"num_omap_keys":93,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'26","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.715965+0000","last_change":"2024-04-16T21:11:55.716041+0000","last_active":"2024-04-16T21:11:55.715965+0000","last_peered":"2024-04-16T21:11:55.715965+0000","last_clean":"2024-04-16T21:11:55.715965+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:11:55.715965+0000","last_undegraded":"2024-04-16T21:11:55.715965+0000","last_fullsized":"2024-04-16T21:11:55.715965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.715893+0000","objects_scrubbed":12,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:15.543756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388499+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:11:09.388499+0000","last_peered":"2024-04-16T21:11:09.388499+0000","last_clean":"2024-04-16T21:11:09.388499+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:11:09.388499+0000","last_undegraded":"2024-04-16T21:11:09.388499+0000","last_fullsized":"2024-04-16T21:11:09.388499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380998+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:10:19.380998+0000","last_peered":"2024-04-16T21:10:19.380998+0000","last_clean":"2024-04-16T21:10:19.380998+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:10:19.380998+0000","last_undegraded":"2024-04-16T21:10:19.380998+0000","last_fullsized":"2024-04-16T21:10:19.380998+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.522362+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:10:34.522362+0000","last_peered":"2024-04-16T21:10:34.522362+0000","last_clean":"2024-04-16T21:10:34.522362+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:10:34.522362+0000","last_undegraded":"2024-04-16T21:10:34.522362+0000","last_fullsized":"2024-04-16T21:10:34.522362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387605+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:10:59.387605+0000","last_peered":"2024-04-16T21:10:59.387605+0000","last_clean":"2024-04-16T21:10:59.387605+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:10:59.387605+0000","last_undegraded":"2024-04-16T21:10:59.387605+0000","last_fullsized":"2024-04-16T21:10:59.387605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1314,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392208+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:10:59.392208+0000","last_peered":"2024-04-16T21:10:59.392208+0000","last_clean":"2024-04-16T21:10:59.392208+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:10:59.392208+0000","last_undegraded":"2024-04-16T21:10:59.392208+0000","last_fullsized":"2024-04-16T21:10:59.392208+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393320+0000","last_change":"2024-04-16T19:24:51.437991+0000","last_active":"2024-04-16T21:11:04.393320+0000","last_peered":"2024-04-16T21:11:04.393320+0000","last_clean":"2024-04-16T21:11:04.393320+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:11:04.393320+0000","last_undegraded":"2024-04-16T21:11:04.393320+0000","last_fullsized":"2024-04-16T21:11:04.393320+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195903+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:11:43.195903+0000","last_peered":"2024-04-16T21:11:43.195903+0000","last_clean":"2024-04-16T21:11:43.195903+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:11:43.195903+0000","last_undegraded":"2024-04-16T21:11:43.195903+0000","last_fullsized":"2024-04-16T21:11:43.195903+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'136","reported_seq":1891,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744051+0000","last_change":"2024-04-16T19:25:49.523163+0000","last_active":"2024-04-16T21:11:42.744051+0000","last_peered":"2024-04-16T21:11:42.744051+0000","last_clean":"2024-04-16T21:11:42.744051+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:11:42.744051+0000","last_undegraded":"2024-04-16T21:11:42.744051+0000","last_fullsized":"2024-04-16T21:11:42.744051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561358+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:11:44.561358+0000","last_peered":"2024-04-16T21:11:44.561358+0000","last_clean":"2024-04-16T21:11:44.561358+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:11:44.561358+0000","last_undegraded":"2024-04-16T21:11:44.561358+0000","last_fullsized":"2024-04-16T21:11:44.561358+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'11","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398079+0000","last_change":"2024-04-16T19:25:15.480717+0000","last_active":"2024-04-16T21:11:34.398079+0000","last_peered":"2024-04-16T21:11:34.398079+0000","last_clean":"2024-04-16T21:11:34.398079+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:11:34.398079+0000","last_undegraded":"2024-04-16T21:11:34.398079+0000","last_fullsized":"2024-04-16T21:11:34.398079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391759+0000","last_change":"2024-04-16T19:25:13.476070+0000","last_active":"2024-04-16T21:11:29.391759+0000","last_peered":"2024-04-16T21:11:29.391759+0000","last_clean":"2024-04-16T21:11:29.391759+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:11:29.391759+0000","last_undegraded":"2024-04-16T21:11:29.391759+0000","last_fullsized":"2024-04-16T21:11:29.391759+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'362","reported_seq":695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.975586+0000","last_change":"2024-04-16T21:11:44.975641+0000","last_active":"2024-04-16T21:11:44.975586+0000","last_peered":"2024-04-16T21:11:44.975586+0000","last_clean":"2024-04-16T21:11:44.975586+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:11:44.975586+0000","last_undegraded":"2024-04-16T21:11:44.975586+0000","last_fullsized":"2024-04-16T21:11:44.975586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'362","last_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_deep_scrub":"52'362","last_deep_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.975535+0000","objects_scrubbed":21,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:50:56.598547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2185,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'18","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.687122+0000","last_change":"2024-04-16T21:11:48.687164+0000","last_active":"2024-04-16T21:11:48.687122+0000","last_peered":"2024-04-16T21:11:48.687122+0000","last_clean":"2024-04-16T21:11:48.687122+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:11:48.687122+0000","last_undegraded":"2024-04-16T21:11:48.687122+0000","last_fullsized":"2024-04-16T21:11:48.687122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.687069+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:52.625067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.386336+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:10:24.386336+0000","last_peered":"2024-04-16T21:10:24.386336+0000","last_clean":"2024-04-16T21:10:24.386336+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:10:24.386336+0000","last_undegraded":"2024-04-16T21:10:24.386336+0000","last_fullsized":"2024-04-16T21:10:24.386336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380925+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:10:19.380925+0000","last_peered":"2024-04-16T21:10:19.380925+0000","last_clean":"2024-04-16T21:10:19.380925+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:10:19.380925+0000","last_undegraded":"2024-04-16T21:10:19.380925+0000","last_fullsized":"2024-04-16T21:10:19.380925+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":458,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.528823+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:11:14.528823+0000","last_peered":"2024-04-16T21:11:14.528823+0000","last_clean":"2024-04-16T21:11:14.528823+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:11:14.528823+0000","last_undegraded":"2024-04-16T21:11:14.528823+0000","last_fullsized":"2024-04-16T21:11:14.528823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392184+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:10:59.392184+0000","last_peered":"2024-04-16T21:10:59.392184+0000","last_clean":"2024-04-16T21:10:59.392184+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:10:59.392184+0000","last_undegraded":"2024-04-16T21:10:59.392184+0000","last_fullsized":"2024-04-16T21:10:59.392184+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526201+0000","last_change":"2024-04-16T19:24:49.432294+0000","last_active":"2024-04-16T21:10:59.526201+0000","last_peered":"2024-04-16T21:10:59.526201+0000","last_clean":"2024-04-16T21:10:59.526201+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:10:59.526201+0000","last_undegraded":"2024-04-16T21:10:59.526201+0000","last_fullsized":"2024-04-16T21:10:59.526201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.847627+0000","last_change":"2024-04-16T21:12:03.847650+0000","last_active":"2024-04-16T21:12:03.847627+0000","last_peered":"2024-04-16T21:12:03.847627+0000","last_clean":"2024-04-16T21:12:03.847627+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:12:03.847627+0000","last_undegraded":"2024-04-16T21:12:03.847627+0000","last_fullsized":"2024-04-16T21:12:03.847627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.847563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:36.440589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744155+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:11:42.744155+0000","last_peered":"2024-04-16T21:11:42.744155+0000","last_clean":"2024-04-16T21:11:42.744155+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:11:42.744155+0000","last_undegraded":"2024-04-16T21:11:42.744155+0000","last_fullsized":"2024-04-16T21:11:42.744155+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2347,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293260+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:11:42.293260+0000","last_peered":"2024-04-16T21:11:42.293260+0000","last_clean":"2024-04-16T21:11:42.293260+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:11:42.293260+0000","last_undegraded":"2024-04-16T21:11:42.293260+0000","last_fullsized":"2024-04-16T21:11:42.293260+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.379523+0000","last_change":"2024-04-16T19:25:41.509053+0000","last_active":"2024-04-16T21:10:14.379523+0000","last_peered":"2024-04-16T21:10:14.379523+0000","last_clean":"2024-04-16T21:10:14.379523+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:10:14.379523+0000","last_undegraded":"2024-04-16T21:10:14.379523+0000","last_fullsized":"2024-04-16T21:10:14.379523+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"47'8","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392653+0000","last_change":"2024-04-16T19:25:15.486624+0000","last_active":"2024-04-16T21:11:34.392653+0000","last_peered":"2024-04-16T21:11:34.392653+0000","last_clean":"2024-04-16T21:11:34.392653+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:11:34.392653+0000","last_undegraded":"2024-04-16T21:11:34.392653+0000","last_fullsized":"2024-04-16T21:11:34.392653+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'705","reported_seq":2066,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.392404+0000","last_change":"2024-04-16T19:25:13.473118+0000","last_active":"2024-04-16T21:11:29.392404+0000","last_peered":"2024-04-16T21:11:29.392404+0000","last_clean":"2024-04-16T21:11:29.392404+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:11:29.392404+0000","last_undegraded":"2024-04-16T21:11:29.392404+0000","last_fullsized":"2024-04-16T21:11:29.392404+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'292","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.642477+0000","last_change":"2024-04-16T21:11:50.642568+0000","last_active":"2024-04-16T21:11:50.642477+0000","last_peered":"2024-04-16T21:11:50.642477+0000","last_clean":"2024-04-16T21:11:50.642477+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:11:50.642477+0000","last_undegraded":"2024-04-16T21:11:50.642477+0000","last_fullsized":"2024-04-16T21:11:50.642477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'292","last_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_deep_scrub":"52'292","last_deep_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.642400+0000","objects_scrubbed":23,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:29.527145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4142,"num_omap_keys":19,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.314574+0000","last_change":"2024-04-16T21:11:57.314619+0000","last_active":"2024-04-16T21:11:57.314574+0000","last_peered":"2024-04-16T21:11:57.314574+0000","last_clean":"2024-04-16T21:11:57.314574+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:11:57.314574+0000","last_undegraded":"2024-04-16T21:11:57.314574+0000","last_fullsized":"2024-04-16T21:11:57.314574+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.314511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:49.494822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380915+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:10:24.380915+0000","last_peered":"2024-04-16T21:10:24.380915+0000","last_clean":"2024-04-16T21:10:24.380915+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:10:24.380915+0000","last_undegraded":"2024-04-16T21:10:24.380915+0000","last_fullsized":"2024-04-16T21:10:24.380915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.379607+0000","last_change":"2024-04-16T19:25:41.503885+0000","last_active":"2024-04-16T21:10:19.379607+0000","last_peered":"2024-04-16T21:10:19.379607+0000","last_clean":"2024-04-16T21:10:19.379607+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:10:19.379607+0000","last_undegraded":"2024-04-16T21:10:19.379607+0000","last_fullsized":"2024-04-16T21:10:19.379607+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.522411+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:10:34.522411+0000","last_peered":"2024-04-16T21:10:34.522411+0000","last_clean":"2024-04-16T21:10:34.522411+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:10:34.522411+0000","last_undegraded":"2024-04-16T21:10:34.522411+0000","last_fullsized":"2024-04-16T21:10:34.522411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":261,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382728+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:10:34.382728+0000","last_peered":"2024-04-16T21:10:34.382728+0000","last_clean":"2024-04-16T21:10:34.382728+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:10:34.382728+0000","last_undegraded":"2024-04-16T21:10:34.382728+0000","last_fullsized":"2024-04-16T21:10:34.382728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391424+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:10:54.391424+0000","last_peered":"2024-04-16T21:10:54.391424+0000","last_clean":"2024-04-16T21:10:54.391424+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:10:54.391424+0000","last_undegraded":"2024-04-16T21:10:54.391424+0000","last_fullsized":"2024-04-16T21:10:54.391424+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.388598+0000","last_change":"2024-04-16T19:24:51.434984+0000","last_active":"2024-04-16T21:11:14.388598+0000","last_peered":"2024-04-16T21:11:14.388598+0000","last_clean":"2024-04-16T21:11:14.388598+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:11:14.388598+0000","last_undegraded":"2024-04-16T21:11:14.388598+0000","last_fullsized":"2024-04-16T21:11:14.388598+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293308+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:11:42.293308+0000","last_peered":"2024-04-16T21:11:42.293308+0000","last_clean":"2024-04-16T21:11:42.293308+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:11:42.293308+0000","last_undegraded":"2024-04-16T21:11:42.293308+0000","last_fullsized":"2024-04-16T21:11:42.293308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1680,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561550+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:11:44.561550+0000","last_peered":"2024-04-16T21:11:44.561550+0000","last_clean":"2024-04-16T21:11:44.561550+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:11:44.561550+0000","last_undegraded":"2024-04-16T21:11:44.561550+0000","last_fullsized":"2024-04-16T21:11:44.561550+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561496+0000","last_change":"2024-04-16T19:25:41.504293+0000","last_active":"2024-04-16T21:11:44.561496+0000","last_peered":"2024-04-16T21:11:44.561496+0000","last_clean":"2024-04-16T21:11:44.561496+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:11:44.561496+0000","last_undegraded":"2024-04-16T21:11:44.561496+0000","last_fullsized":"2024-04-16T21:11:44.561496+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'136","reported_seq":300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651212+0000","last_change":"2024-04-16T19:25:15.483254+0000","last_active":"2024-04-16T21:11:43.651212+0000","last_peered":"2024-04-16T21:11:43.651212+0000","last_clean":"2024-04-16T21:11:43.651212+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:11:43.651212+0000","last_undegraded":"2024-04-16T21:11:43.651212+0000","last_fullsized":"2024-04-16T21:11:43.651212+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.836583+0000","last_change":"2024-04-16T21:11:52.836606+0000","last_active":"2024-04-16T21:11:52.836583+0000","last_peered":"2024-04-16T21:11:52.836583+0000","last_clean":"2024-04-16T21:11:52.836583+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:11:52.836583+0000","last_undegraded":"2024-04-16T21:11:52.836583+0000","last_fullsized":"2024-04-16T21:11:52.836583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.836519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:26.725994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'371","reported_seq":758,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.959985+0000","last_change":"2024-04-16T21:11:46.960080+0000","last_active":"2024-04-16T21:11:46.959985+0000","last_peered":"2024-04-16T21:11:46.959985+0000","last_clean":"2024-04-16T21:11:46.959985+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:11:46.959985+0000","last_undegraded":"2024-04-16T21:11:46.959985+0000","last_fullsized":"2024-04-16T21:11:46.959985+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'371","last_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_deep_scrub":"52'371","last_deep_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.959920+0000","objects_scrubbed":24,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:28.129840+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2169,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'19","reported_seq":216,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.614888+0000","last_change":"2024-04-16T21:11:53.614931+0000","last_active":"2024-04-16T21:11:53.614888+0000","last_peered":"2024-04-16T21:11:53.614888+0000","last_clean":"2024-04-16T21:11:53.614888+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:11:53.614888+0000","last_undegraded":"2024-04-16T21:11:53.614888+0000","last_fullsized":"2024-04-16T21:11:53.614888+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.614826+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:49.084093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381758+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:10:24.381758+0000","last_peered":"2024-04-16T21:10:24.381758+0000","last_clean":"2024-04-16T21:10:24.381758+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:10:24.381758+0000","last_undegraded":"2024-04-16T21:10:24.381758+0000","last_fullsized":"2024-04-16T21:10:24.381758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":81,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377258+0000","last_change":"2024-04-16T19:25:41.506323+0000","last_active":"2024-04-16T21:10:04.377258+0000","last_peered":"2024-04-16T21:10:04.377258+0000","last_clean":"2024-04-16T21:10:04.377258+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:10:04.377258+0000","last_undegraded":"2024-04-16T21:10:04.377258+0000","last_fullsized":"2024-04-16T21:10:04.377258+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382104+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:10:34.382104+0000","last_peered":"2024-04-16T21:10:34.382104+0000","last_clean":"2024-04-16T21:10:34.382104+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:10:34.382104+0000","last_undegraded":"2024-04-16T21:10:34.382104+0000","last_fullsized":"2024-04-16T21:10:34.382104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'140","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524829+0000","last_change":"2024-04-16T19:24:47.426406+0000","last_active":"2024-04-16T21:10:49.524829+0000","last_peered":"2024-04-16T21:10:49.524829+0000","last_clean":"2024-04-16T21:10:49.524829+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:10:49.524829+0000","last_undegraded":"2024-04-16T21:10:49.524829+0000","last_fullsized":"2024-04-16T21:10:49.524829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'12","reported_seq":1312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.522431+0000","last_change":"2024-04-16T19:24:49.430773+0000","last_active":"2024-04-16T21:10:34.522431+0000","last_peered":"2024-04-16T21:10:34.522431+0000","last_clean":"2024-04-16T21:10:34.522431+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:10:34.522431+0000","last_undegraded":"2024-04-16T21:10:34.522431+0000","last_fullsized":"2024-04-16T21:10:34.522431+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.387880+0000","last_change":"2024-04-16T19:24:51.436754+0000","last_active":"2024-04-16T21:11:09.387880+0000","last_peered":"2024-04-16T21:11:09.387880+0000","last_clean":"2024-04-16T21:11:09.387880+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:11:09.387880+0000","last_undegraded":"2024-04-16T21:11:09.387880+0000","last_fullsized":"2024-04-16T21:11:09.387880+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389430+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:11:09.389430+0000","last_peered":"2024-04-16T21:11:09.389430+0000","last_clean":"2024-04-16T21:11:09.389430+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:11:09.389430+0000","last_undegraded":"2024-04-16T21:11:09.389430+0000","last_fullsized":"2024-04-16T21:11:09.389430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1872,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651400+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:11:43.651400+0000","last_peered":"2024-04-16T21:11:43.651400+0000","last_clean":"2024-04-16T21:11:43.651400+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:11:43.651400+0000","last_undegraded":"2024-04-16T21:11:43.651400+0000","last_fullsized":"2024-04-16T21:11:43.651400+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651360+0000","last_change":"2024-04-16T19:25:41.507626+0000","last_active":"2024-04-16T21:11:43.651360+0000","last_peered":"2024-04-16T21:11:43.651360+0000","last_clean":"2024-04-16T21:11:43.651360+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:11:43.651360+0000","last_undegraded":"2024-04-16T21:11:43.651360+0000","last_fullsized":"2024-04-16T21:11:43.651360+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'7","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391201+0000","last_change":"2024-04-16T19:25:15.477722+0000","last_active":"2024-04-16T21:11:29.391201+0000","last_peered":"2024-04-16T21:11:29.391201+0000","last_clean":"2024-04-16T21:11:29.391201+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:11:29.391201+0000","last_undegraded":"2024-04-16T21:11:29.391201+0000","last_fullsized":"2024-04-16T21:11:29.391201+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.625545+0000","last_change":"2024-04-16T21:11:53.625568+0000","last_active":"2024-04-16T21:11:53.625545+0000","last_peered":"2024-04-16T21:11:53.625545+0000","last_clean":"2024-04-16T21:11:53.625545+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:11:53.625545+0000","last_undegraded":"2024-04-16T21:11:53.625545+0000","last_fullsized":"2024-04-16T21:11:53.625545+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.625484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:35.360695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'485","reported_seq":911,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.722929+0000","last_change":"2024-04-16T21:11:47.722980+0000","last_active":"2024-04-16T21:11:47.722929+0000","last_peered":"2024-04-16T21:11:47.722929+0000","last_clean":"2024-04-16T21:11:47.722929+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:11:47.722929+0000","last_undegraded":"2024-04-16T21:11:47.722929+0000","last_fullsized":"2024-04-16T21:11:47.722929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'485","last_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_deep_scrub":"52'485","last_deep_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.722865+0000","objects_scrubbed":32,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:39.504569+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42883,"num_omap_keys":132,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'8","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106953+0000","last_change":"2024-04-16T19:25:41.507771+0000","last_active":"2024-04-16T21:11:44.106953+0000","last_peered":"2024-04-16T21:11:44.106953+0000","last_clean":"2024-04-16T21:11:44.106953+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:11:44.106953+0000","last_undegraded":"2024-04-16T21:11:44.106953+0000","last_fullsized":"2024-04-16T21:11:44.106953+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380981+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:10:24.380981+0000","last_peered":"2024-04-16T21:10:24.380981+0000","last_clean":"2024-04-16T21:10:24.380981+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:10:24.380981+0000","last_undegraded":"2024-04-16T21:10:24.380981+0000","last_fullsized":"2024-04-16T21:10:24.380981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384338+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:10:44.384338+0000","last_peered":"2024-04-16T21:10:44.384338+0000","last_clean":"2024-04-16T21:10:44.384338+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:10:44.384338+0000","last_undegraded":"2024-04-16T21:10:44.384338+0000","last_fullsized":"2024-04-16T21:10:44.384338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386376+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:10:54.386376+0000","last_peered":"2024-04-16T21:10:54.386376+0000","last_clean":"2024-04-16T21:10:54.386376+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:10:54.386376+0000","last_undegraded":"2024-04-16T21:10:54.386376+0000","last_fullsized":"2024-04-16T21:10:54.386376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":277,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382118+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:10:34.382118+0000","last_peered":"2024-04-16T21:10:34.382118+0000","last_clean":"2024-04-16T21:10:34.382118+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:10:34.382118+0000","last_undegraded":"2024-04-16T21:10:34.382118+0000","last_fullsized":"2024-04-16T21:10:34.382118+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'12","reported_seq":1312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382683+0000","last_change":"2024-04-16T19:24:49.428502+0000","last_active":"2024-04-16T21:10:34.382683+0000","last_peered":"2024-04-16T21:10:34.382683+0000","last_clean":"2024-04-16T21:10:34.382683+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:10:34.382683+0000","last_undegraded":"2024-04-16T21:10:34.382683+0000","last_fullsized":"2024-04-16T21:10:34.382683+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388664+0000","last_change":"2024-04-16T19:24:51.435114+0000","last_active":"2024-04-16T21:11:09.388664+0000","last_peered":"2024-04-16T21:11:09.388664+0000","last_clean":"2024-04-16T21:11:09.388664+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:11:09.388664+0000","last_undegraded":"2024-04-16T21:11:09.388664+0000","last_fullsized":"2024-04-16T21:11:09.388664+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385558+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:10:54.385558+0000","last_peered":"2024-04-16T21:10:54.385558+0000","last_clean":"2024-04-16T21:10:54.385558+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:10:54.385558+0000","last_undegraded":"2024-04-16T21:10:54.385558+0000","last_fullsized":"2024-04-16T21:10:54.385558+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385636+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:10:19.385636+0000","last_peered":"2024-04-16T21:10:19.385636+0000","last_clean":"2024-04-16T21:10:19.385636+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:10:19.385636+0000","last_undegraded":"2024-04-16T21:10:19.385636+0000","last_fullsized":"2024-04-16T21:10:19.385636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651359+0000","last_change":"2024-04-16T19:25:41.506250+0000","last_active":"2024-04-16T21:11:43.651359+0000","last_peered":"2024-04-16T21:11:43.651359+0000","last_clean":"2024-04-16T21:11:43.651359+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:11:43.651359+0000","last_undegraded":"2024-04-16T21:11:43.651359+0000","last_fullsized":"2024-04-16T21:11:43.651359+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391238+0000","last_change":"2024-04-16T19:25:15.479905+0000","last_active":"2024-04-16T21:11:29.391238+0000","last_peered":"2024-04-16T21:11:29.391238+0000","last_clean":"2024-04-16T21:11:29.391238+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:11:29.391238+0000","last_undegraded":"2024-04-16T21:11:29.391238+0000","last_fullsized":"2024-04-16T21:11:29.391238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'706","reported_seq":2474,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.688857+0000","last_change":"2024-04-16T21:11:58.688891+0000","last_active":"2024-04-16T21:11:58.688857+0000","last_peered":"2024-04-16T21:11:58.688857+0000","last_clean":"2024-04-16T21:11:58.688857+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:11:58.688857+0000","last_undegraded":"2024-04-16T21:11:58.688857+0000","last_fullsized":"2024-04-16T21:11:58.688857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.688787+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:58.866593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'332","reported_seq":603,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.669169+0000","last_change":"2024-04-16T21:11:50.669285+0000","last_active":"2024-04-16T21:11:50.669169+0000","last_peered":"2024-04-16T21:11:50.669169+0000","last_clean":"2024-04-16T21:11:50.669169+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:11:50.669169+0000","last_undegraded":"2024-04-16T21:11:50.669169+0000","last_fullsized":"2024-04-16T21:11:50.669169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'332","last_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_deep_scrub":"52'332","last_deep_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.669096+0000","objects_scrubbed":21,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:18.487063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2179,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'12","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.639977+0000","last_change":"2024-04-16T21:11:49.640051+0000","last_active":"2024-04-16T21:11:49.639977+0000","last_peered":"2024-04-16T21:11:49.639977+0000","last_clean":"2024-04-16T21:11:49.639977+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:11:49.639977+0000","last_undegraded":"2024-04-16T21:11:49.639977+0000","last_fullsized":"2024-04-16T21:11:49.639977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.639900+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:27.425866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383761+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:10:44.383761+0000","last_peered":"2024-04-16T21:10:44.383761+0000","last_clean":"2024-04-16T21:10:44.383761+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:10:44.383761+0000","last_undegraded":"2024-04-16T21:10:44.383761+0000","last_fullsized":"2024-04-16T21:10:44.383761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.523887+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:10:44.523887+0000","last_peered":"2024-04-16T21:10:44.523887+0000","last_clean":"2024-04-16T21:10:44.523887+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:10:44.523887+0000","last_undegraded":"2024-04-16T21:10:44.523887+0000","last_fullsized":"2024-04-16T21:10:44.523887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383477+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:10:34.383477+0000","last_peered":"2024-04-16T21:10:34.383477+0000","last_clean":"2024-04-16T21:10:34.383477+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:10:34.383477+0000","last_undegraded":"2024-04-16T21:10:34.383477+0000","last_fullsized":"2024-04-16T21:10:34.383477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":360,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383080+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:10:34.383080+0000","last_peered":"2024-04-16T21:10:34.383080+0000","last_clean":"2024-04-16T21:10:34.383080+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:10:34.383080+0000","last_undegraded":"2024-04-16T21:10:34.383080+0000","last_fullsized":"2024-04-16T21:10:34.383080+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383416+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:10:34.383416+0000","last_peered":"2024-04-16T21:10:34.383416+0000","last_clean":"2024-04-16T21:10:34.383416+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:10:34.383416+0000","last_undegraded":"2024-04-16T21:10:34.383416+0000","last_fullsized":"2024-04-16T21:10:34.383416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.387946+0000","last_change":"2024-04-16T19:24:51.434704+0000","last_active":"2024-04-16T21:11:09.387946+0000","last_peered":"2024-04-16T21:11:09.387946+0000","last_clean":"2024-04-16T21:11:09.387946+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:11:09.387946+0000","last_undegraded":"2024-04-16T21:11:09.387946+0000","last_fullsized":"2024-04-16T21:11:09.387946+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524809+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:10:49.524809+0000","last_peered":"2024-04-16T21:10:49.524809+0000","last_clean":"2024-04-16T21:10:49.524809+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:10:49.524809+0000","last_undegraded":"2024-04-16T21:10:49.524809+0000","last_fullsized":"2024-04-16T21:10:49.524809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'162","reported_seq":2097,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107487+0000","last_change":"2024-04-16T19:25:49.518828+0000","last_active":"2024-04-16T21:11:44.107487+0000","last_peered":"2024-04-16T21:11:44.107487+0000","last_clean":"2024-04-16T21:11:44.107487+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:11:44.107487+0000","last_undegraded":"2024-04-16T21:11:44.107487+0000","last_fullsized":"2024-04-16T21:11:44.107487+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391143+0000","last_change":"2024-04-16T19:25:41.510329+0000","last_active":"2024-04-16T21:11:29.391143+0000","last_peered":"2024-04-16T21:11:29.391143+0000","last_clean":"2024-04-16T21:11:29.391143+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:11:29.391143+0000","last_undegraded":"2024-04-16T21:11:29.391143+0000","last_fullsized":"2024-04-16T21:11:29.391143+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'12","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387932+0000","last_change":"2024-04-16T19:25:15.480613+0000","last_active":"2024-04-16T21:11:04.387932+0000","last_peered":"2024-04-16T21:11:04.387932+0000","last_clean":"2024-04-16T21:11:04.387932+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:11:04.387932+0000","last_undegraded":"2024-04-16T21:11:04.387932+0000","last_fullsized":"2024-04-16T21:11:04.387932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.634331+0000","last_change":"2024-04-16T21:11:56.634374+0000","last_active":"2024-04-16T21:11:56.634331+0000","last_peered":"2024-04-16T21:11:56.634331+0000","last_clean":"2024-04-16T21:11:56.634331+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:11:56.634331+0000","last_undegraded":"2024-04-16T21:11:56.634331+0000","last_fullsized":"2024-04-16T21:11:56.634331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.634253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:30.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'281","reported_seq":601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.734176+0000","last_change":"2024-04-16T21:11:44.734203+0000","last_active":"2024-04-16T21:11:44.734176+0000","last_peered":"2024-04-16T21:11:44.734176+0000","last_clean":"2024-04-16T21:11:44.734176+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:11:44.734176+0000","last_undegraded":"2024-04-16T21:11:44.734176+0000","last_fullsized":"2024-04-16T21:11:44.734176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'281","last_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_deep_scrub":"52'281","last_deep_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.734118+0000","objects_scrubbed":25,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:22.520084+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1975,"num_omap_keys":9,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'6","reported_seq":234,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.529520+0000","last_change":"2024-04-16T19:25:09.462147+0000","last_active":"2024-04-16T21:11:19.529520+0000","last_peered":"2024-04-16T21:11:19.529520+0000","last_clean":"2024-04-16T21:11:19.529520+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:11:19.529520+0000","last_undegraded":"2024-04-16T21:11:19.529520+0000","last_fullsized":"2024-04-16T21:11:19.529520+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390257+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:11:14.390257+0000","last_peered":"2024-04-16T21:11:14.390257+0000","last_clean":"2024-04-16T21:11:14.390257+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:11:14.390257+0000","last_undegraded":"2024-04-16T21:11:14.390257+0000","last_fullsized":"2024-04-16T21:11:14.390257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":292,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.379173+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:10:09.379173+0000","last_peered":"2024-04-16T21:10:09.379173+0000","last_clean":"2024-04-16T21:10:09.379173+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:10:09.379173+0000","last_undegraded":"2024-04-16T21:10:09.379173+0000","last_fullsized":"2024-04-16T21:10:09.379173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378376+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:10:04.378376+0000","last_peered":"2024-04-16T21:10:04.378376+0000","last_clean":"2024-04-16T21:10:04.378376+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:10:04.378376+0000","last_undegraded":"2024-04-16T21:10:04.378376+0000","last_fullsized":"2024-04-16T21:10:04.378376+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384510+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:10:49.384510+0000","last_peered":"2024-04-16T21:10:49.384510+0000","last_clean":"2024-04-16T21:10:49.384510+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:10:49.384510+0000","last_undegraded":"2024-04-16T21:10:49.384510+0000","last_fullsized":"2024-04-16T21:10:49.384510+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387912+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:11:04.387912+0000","last_peered":"2024-04-16T21:11:04.387912+0000","last_clean":"2024-04-16T21:11:04.387912+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:11:04.387912+0000","last_undegraded":"2024-04-16T21:11:04.387912+0000","last_fullsized":"2024-04-16T21:11:04.387912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":231,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387149+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:10:59.387149+0000","last_peered":"2024-04-16T21:10:59.387149+0000","last_clean":"2024-04-16T21:10:59.387149+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:10:59.387149+0000","last_undegraded":"2024-04-16T21:10:59.387149+0000","last_fullsized":"2024-04-16T21:10:59.387149+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392359+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:10:59.392359+0000","last_peered":"2024-04-16T21:10:59.392359+0000","last_clean":"2024-04-16T21:10:59.392359+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:10:59.392359+0000","last_undegraded":"2024-04-16T21:10:59.392359+0000","last_fullsized":"2024-04-16T21:10:59.392359+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387690+0000","last_change":"2024-04-16T19:24:51.442576+0000","last_active":"2024-04-16T21:11:04.387690+0000","last_peered":"2024-04-16T21:11:04.387690+0000","last_clean":"2024-04-16T21:11:04.387690+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:11:04.387690+0000","last_undegraded":"2024-04-16T21:11:04.387690+0000","last_fullsized":"2024-04-16T21:11:04.387690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385439+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:10:49.385439+0000","last_peered":"2024-04-16T21:10:49.385439+0000","last_clean":"2024-04-16T21:10:49.385439+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:10:49.385439+0000","last_undegraded":"2024-04-16T21:10:49.385439+0000","last_fullsized":"2024-04-16T21:10:49.385439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1952,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398096+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:11:34.398096+0000","last_peered":"2024-04-16T21:11:34.398096+0000","last_clean":"2024-04-16T21:11:34.398096+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:11:34.398096+0000","last_undegraded":"2024-04-16T21:11:34.398096+0000","last_fullsized":"2024-04-16T21:11:34.398096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561359+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:11:44.561359+0000","last_peered":"2024-04-16T21:11:44.561359+0000","last_clean":"2024-04-16T21:11:44.561359+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:11:44.561359+0000","last_undegraded":"2024-04-16T21:11:44.561359+0000","last_fullsized":"2024-04-16T21:11:44.561359+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'13","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393301+0000","last_change":"2024-04-16T19:25:15.485212+0000","last_active":"2024-04-16T21:11:04.393301+0000","last_peered":"2024-04-16T21:11:04.393301+0000","last_clean":"2024-04-16T21:11:04.393301+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:11:04.393301+0000","last_undegraded":"2024-04-16T21:11:04.393301+0000","last_fullsized":"2024-04-16T21:11:04.393301+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'706","reported_seq":2087,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.915788+0000","last_change":"2024-04-16T21:11:48.915841+0000","last_active":"2024-04-16T21:11:48.915788+0000","last_peered":"2024-04-16T21:11:48.915788+0000","last_clean":"2024-04-16T21:11:48.915788+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:11:48.915788+0000","last_undegraded":"2024-04-16T21:11:48.915788+0000","last_fullsized":"2024-04-16T21:11:48.915788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.915706+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.964684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'326","reported_seq":644,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.279300+0000","last_change":"2024-04-16T21:11:56.279412+0000","last_active":"2024-04-16T21:11:56.279300+0000","last_peered":"2024-04-16T21:11:56.279300+0000","last_clean":"2024-04-16T21:11:56.279300+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:11:56.279300+0000","last_undegraded":"2024-04-16T21:11:56.279300+0000","last_fullsized":"2024-04-16T21:11:56.279300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'326","last_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_deep_scrub":"52'326","last_deep_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.279222+0000","objects_scrubbed":20,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:03.299266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'20","reported_seq":201,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.117827+0000","last_change":"2024-04-16T21:11:45.117905+0000","last_active":"2024-04-16T21:11:45.117827+0000","last_peered":"2024-04-16T21:11:45.117827+0000","last_clean":"2024-04-16T21:11:45.117827+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:11:45.117827+0000","last_undegraded":"2024-04-16T21:11:45.117827+0000","last_fullsized":"2024-04-16T21:11:45.117827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.117757+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:23.210409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390704+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:10:49.390704+0000","last_peered":"2024-04-16T21:10:49.390704+0000","last_clean":"2024-04-16T21:10:49.390704+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:10:49.390704+0000","last_undegraded":"2024-04-16T21:10:49.390704+0000","last_fullsized":"2024-04-16T21:10:49.390704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.379206+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:10:09.379206+0000","last_peered":"2024-04-16T21:10:09.379206+0000","last_clean":"2024-04-16T21:10:09.379206+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:10:09.379206+0000","last_undegraded":"2024-04-16T21:10:09.379206+0000","last_fullsized":"2024-04-16T21:10:09.379206+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390485+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:10:49.390485+0000","last_peered":"2024-04-16T21:10:49.390485+0000","last_clean":"2024-04-16T21:10:49.390485+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:10:49.390485+0000","last_undegraded":"2024-04-16T21:10:49.390485+0000","last_fullsized":"2024-04-16T21:10:49.390485+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":592,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387164+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:10:59.387164+0000","last_peered":"2024-04-16T21:10:59.387164+0000","last_clean":"2024-04-16T21:10:59.387164+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:10:59.387164+0000","last_undegraded":"2024-04-16T21:10:59.387164+0000","last_fullsized":"2024-04-16T21:10:59.387164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1570,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387167+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:11:04.387167+0000","last_peered":"2024-04-16T21:11:04.387167+0000","last_clean":"2024-04-16T21:11:04.387167+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:11:04.387167+0000","last_undegraded":"2024-04-16T21:11:04.387167+0000","last_fullsized":"2024-04-16T21:11:04.387167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.390988+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:11:24.390988+0000","last_peered":"2024-04-16T21:11:24.390988+0000","last_clean":"2024-04-16T21:11:24.390988+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:11:24.390988+0000","last_undegraded":"2024-04-16T21:11:24.390988+0000","last_fullsized":"2024-04-16T21:11:24.390988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.671618+0000","last_change":"2024-04-16T21:11:57.671660+0000","last_active":"2024-04-16T21:11:57.671618+0000","last_peered":"2024-04-16T21:11:57.671618+0000","last_clean":"2024-04-16T21:11:57.671618+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:11:57.671618+0000","last_undegraded":"2024-04-16T21:11:57.671618+0000","last_fullsized":"2024-04-16T21:11:57.671618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.671545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:20.862941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385351+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:10:49.385351+0000","last_peered":"2024-04-16T21:10:49.385351+0000","last_clean":"2024-04-16T21:10:49.385351+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:10:49.385351+0000","last_undegraded":"2024-04-16T21:10:49.385351+0000","last_fullsized":"2024-04-16T21:10:49.385351+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2370,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392699+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:11:34.392699+0000","last_peered":"2024-04-16T21:11:34.392699+0000","last_clean":"2024-04-16T21:11:34.392699+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:11:34.392699+0000","last_undegraded":"2024-04-16T21:11:34.392699+0000","last_fullsized":"2024-04-16T21:11:34.392699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391873+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:11:29.391873+0000","last_peered":"2024-04-16T21:11:29.391873+0000","last_clean":"2024-04-16T21:11:29.391873+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:11:29.391873+0000","last_undegraded":"2024-04-16T21:11:29.391873+0000","last_fullsized":"2024-04-16T21:11:29.391873+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'16","reported_seq":154,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.391755+0000","last_change":"2024-04-16T19:25:15.484662+0000","last_active":"2024-04-16T21:11:24.391755+0000","last_peered":"2024-04-16T21:11:24.391755+0000","last_clean":"2024-04-16T21:11:24.391755+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:11:24.391755+0000","last_undegraded":"2024-04-16T21:11:24.391755+0000","last_fullsized":"2024-04-16T21:11:24.391755+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'21","reported_seq":982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.783889+0000","last_change":"2024-04-16T21:11:58.783920+0000","last_active":"2024-04-16T21:11:58.783889+0000","last_peered":"2024-04-16T21:11:58.783889+0000","last_clean":"2024-04-16T21:11:58.783889+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:11:58.783889+0000","last_undegraded":"2024-04-16T21:11:58.783889+0000","last_fullsized":"2024-04-16T21:11:58.783889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.783819+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:59.285203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'268","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.359130+0000","last_change":"2024-04-16T21:11:44.359183+0000","last_active":"2024-04-16T21:11:44.359130+0000","last_peered":"2024-04-16T21:11:44.359130+0000","last_clean":"2024-04-16T21:11:44.359130+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:11:44.359130+0000","last_undegraded":"2024-04-16T21:11:44.359130+0000","last_fullsized":"2024-04-16T21:11:44.359130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'268","last_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_deep_scrub":"52'268","last_deep_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.359071+0000","objects_scrubbed":19,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:18.502556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'13","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.427731+0000","last_change":"2024-04-16T21:11:53.427771+0000","last_active":"2024-04-16T21:11:53.427731+0000","last_peered":"2024-04-16T21:11:53.427731+0000","last_clean":"2024-04-16T21:11:53.427731+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:11:53.427731+0000","last_undegraded":"2024-04-16T21:11:53.427731+0000","last_fullsized":"2024-04-16T21:11:53.427731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.427656+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:14.929241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384655+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:10:49.384655+0000","last_peered":"2024-04-16T21:10:49.384655+0000","last_clean":"2024-04-16T21:10:49.384655+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:10:49.384655+0000","last_undegraded":"2024-04-16T21:10:49.384655+0000","last_fullsized":"2024-04-16T21:10:49.384655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194715+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:11:43.194715+0000","last_peered":"2024-04-16T21:11:43.194715+0000","last_clean":"2024-04-16T21:11:43.194715+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:11:43.194715+0000","last_undegraded":"2024-04-16T21:11:43.194715+0000","last_fullsized":"2024-04-16T21:11:43.194715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385426+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:10:54.385426+0000","last_peered":"2024-04-16T21:10:54.385426+0000","last_clean":"2024-04-16T21:10:54.385426+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:10:54.385426+0000","last_undegraded":"2024-04-16T21:10:54.385426+0000","last_fullsized":"2024-04-16T21:10:54.385426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526325+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:10:59.526325+0000","last_peered":"2024-04-16T21:10:59.526325+0000","last_clean":"2024-04-16T21:10:59.526325+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:10:59.526325+0000","last_undegraded":"2024-04-16T21:10:59.526325+0000","last_fullsized":"2024-04-16T21:10:59.526325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1514,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.390979+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:11:24.390979+0000","last_peered":"2024-04-16T21:11:24.390979+0000","last_clean":"2024-04-16T21:11:24.390979+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:11:24.390979+0000","last_undegraded":"2024-04-16T21:11:24.390979+0000","last_fullsized":"2024-04-16T21:11:24.390979+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393254+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:11:04.393254+0000","last_peered":"2024-04-16T21:11:04.393254+0000","last_clean":"2024-04-16T21:11:04.393254+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:11:04.393254+0000","last_undegraded":"2024-04-16T21:11:04.393254+0000","last_fullsized":"2024-04-16T21:11:04.393254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.388938+0000","last_change":"2024-04-16T19:24:51.438114+0000","last_active":"2024-04-16T21:10:39.388938+0000","last_peered":"2024-04-16T21:10:39.388938+0000","last_clean":"2024-04-16T21:10:39.388938+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:10:39.388938+0000","last_undegraded":"2024-04-16T21:10:39.388938+0000","last_fullsized":"2024-04-16T21:10:39.388938+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388785+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:11:04.388785+0000","last_peered":"2024-04-16T21:11:04.388785+0000","last_clean":"2024-04-16T21:11:04.388785+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:11:04.388785+0000","last_undegraded":"2024-04-16T21:11:04.388785+0000","last_fullsized":"2024-04-16T21:11:04.388785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1767,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392656+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:11:34.392656+0000","last_peered":"2024-04-16T21:11:34.392656+0000","last_clean":"2024-04-16T21:11:34.392656+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:11:34.392656+0000","last_undegraded":"2024-04-16T21:11:34.392656+0000","last_fullsized":"2024-04-16T21:11:34.392656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.699415+0000","last_change":"2024-04-16T21:11:59.699455+0000","last_active":"2024-04-16T21:11:59.699415+0000","last_peered":"2024-04-16T21:11:59.699415+0000","last_clean":"2024-04-16T21:11:59.699415+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:11:59.699415+0000","last_undegraded":"2024-04-16T21:11:59.699415+0000","last_fullsized":"2024-04-16T21:11:59.699415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.699336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'15","reported_seq":174,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393240+0000","last_change":"2024-04-16T19:25:15.487696+0000","last_active":"2024-04-16T21:11:04.393240+0000","last_peered":"2024-04-16T21:11:04.393240+0000","last_clean":"2024-04-16T21:11:04.393240+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:11:04.393240+0000","last_undegraded":"2024-04-16T21:11:04.393240+0000","last_fullsized":"2024-04-16T21:11:04.393240+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.891648+0000","last_change":"2024-04-16T21:12:03.891673+0000","last_active":"2024-04-16T21:12:03.891648+0000","last_peered":"2024-04-16T21:12:03.891648+0000","last_clean":"2024-04-16T21:12:03.891648+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:12:03.891648+0000","last_undegraded":"2024-04-16T21:12:03.891648+0000","last_fullsized":"2024-04-16T21:12:03.891648+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.891582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:27.082978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'720","reported_seq":1730,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.792306+0000","last_change":"2024-04-16T21:11:44.792348+0000","last_active":"2024-04-16T21:11:44.792306+0000","last_peered":"2024-04-16T21:11:44.792306+0000","last_clean":"2024-04-16T21:11:44.792306+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:11:44.792306+0000","last_undegraded":"2024-04-16T21:11:44.792306+0000","last_fullsized":"2024-04-16T21:11:44.792306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'720","last_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_deep_scrub":"52'720","last_deep_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.792244+0000","objects_scrubbed":30,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:07:06.408902+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2191,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'20","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.222406+0000","last_change":"2024-04-16T21:11:49.222443+0000","last_active":"2024-04-16T21:11:49.222406+0000","last_peered":"2024-04-16T21:11:49.222406+0000","last_clean":"2024-04-16T21:11:49.222406+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:11:49.222406+0000","last_undegraded":"2024-04-16T21:11:49.222406+0000","last_fullsized":"2024-04-16T21:11:49.222406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.222345+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:21.815371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385547+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:10:54.385547+0000","last_peered":"2024-04-16T21:10:54.385547+0000","last_clean":"2024-04-16T21:10:54.385547+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:10:54.385547+0000","last_undegraded":"2024-04-16T21:10:54.385547+0000","last_fullsized":"2024-04-16T21:10:54.385547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378349+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:10:04.378349+0000","last_peered":"2024-04-16T21:10:04.378349+0000","last_clean":"2024-04-16T21:10:04.378349+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:10:04.378349+0000","last_undegraded":"2024-04-16T21:10:04.378349+0000","last_fullsized":"2024-04-16T21:10:04.378349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385348+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:10:49.385348+0000","last_peered":"2024-04-16T21:10:49.385348+0000","last_clean":"2024-04-16T21:10:49.385348+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:10:49.385348+0000","last_undegraded":"2024-04-16T21:10:49.385348+0000","last_fullsized":"2024-04-16T21:10:49.385348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.396579+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:11:24.396579+0000","last_peered":"2024-04-16T21:11:24.396579+0000","last_clean":"2024-04-16T21:11:24.396579+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:11:24.396579+0000","last_undegraded":"2024-04-16T21:11:24.396579+0000","last_fullsized":"2024-04-16T21:11:24.396579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.386301+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:10:59.386301+0000","last_peered":"2024-04-16T21:10:59.386301+0000","last_clean":"2024-04-16T21:10:59.386301+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:10:59.386301+0000","last_undegraded":"2024-04-16T21:10:59.386301+0000","last_fullsized":"2024-04-16T21:10:59.386301+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387873+0000","last_change":"2024-04-16T19:24:49.432013+0000","last_active":"2024-04-16T21:10:59.387873+0000","last_peered":"2024-04-16T21:10:59.387873+0000","last_clean":"2024-04-16T21:10:59.387873+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:10:59.387873+0000","last_undegraded":"2024-04-16T21:10:59.387873+0000","last_fullsized":"2024-04-16T21:10:59.387873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387671+0000","last_change":"2024-04-16T19:24:51.436955+0000","last_active":"2024-04-16T21:11:04.387671+0000","last_peered":"2024-04-16T21:11:04.387671+0000","last_clean":"2024-04-16T21:11:04.387671+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:11:04.387671+0000","last_undegraded":"2024-04-16T21:11:04.387671+0000","last_fullsized":"2024-04-16T21:11:04.387671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561653+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:11:44.561653+0000","last_peered":"2024-04-16T21:11:44.561653+0000","last_clean":"2024-04-16T21:11:44.561653+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:11:44.561653+0000","last_undegraded":"2024-04-16T21:11:44.561653+0000","last_fullsized":"2024-04-16T21:11:44.561653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1336,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293264+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:11:42.293264+0000","last_peered":"2024-04-16T21:11:42.293264+0000","last_clean":"2024-04-16T21:11:42.293264+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:11:42.293264+0000","last_undegraded":"2024-04-16T21:11:42.293264+0000","last_fullsized":"2024-04-16T21:11:42.293264+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.392426+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:11:29.392426+0000","last_peered":"2024-04-16T21:11:29.392426+0000","last_clean":"2024-04-16T21:11:29.392426+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:11:29.392426+0000","last_undegraded":"2024-04-16T21:11:29.392426+0000","last_fullsized":"2024-04-16T21:11:29.392426+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'6","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.781379+0000","last_change":"2024-04-16T21:11:54.781414+0000","last_active":"2024-04-16T21:11:54.781379+0000","last_peered":"2024-04-16T21:11:54.781379+0000","last_clean":"2024-04-16T21:11:54.781379+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:11:54.781379+0000","last_undegraded":"2024-04-16T21:11:54.781379+0000","last_fullsized":"2024-04-16T21:11:54.781379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.781318+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.682337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'706","reported_seq":2169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.804811+0000","last_change":"2024-04-16T21:11:57.804839+0000","last_active":"2024-04-16T21:11:57.804811+0000","last_peered":"2024-04-16T21:11:57.804811+0000","last_clean":"2024-04-16T21:11:57.804811+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:11:57.804811+0000","last_undegraded":"2024-04-16T21:11:57.804811+0000","last_fullsized":"2024-04-16T21:11:57.804811+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.804739+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:38:55.231266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'264","reported_seq":495,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.746009+0000","last_change":"2024-04-16T21:11:46.746078+0000","last_active":"2024-04-16T21:11:46.746009+0000","last_peered":"2024-04-16T21:11:46.746009+0000","last_clean":"2024-04-16T21:11:46.746009+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:11:46.746009+0000","last_undegraded":"2024-04-16T21:11:46.746009+0000","last_fullsized":"2024-04-16T21:11:46.746009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.745946+0000","objects_scrubbed":17,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:43.933138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1751,"num_omap_keys":8,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'38","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.693339+0000","last_change":"2024-04-16T21:11:52.693388+0000","last_active":"2024-04-16T21:11:52.693339+0000","last_peered":"2024-04-16T21:11:52.693339+0000","last_clean":"2024-04-16T21:11:52.693339+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:11:52.693339+0000","last_undegraded":"2024-04-16T21:11:52.693339+0000","last_fullsized":"2024-04-16T21:11:52.693339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'38","last_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_deep_scrub":"52'38","last_deep_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.693273+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:15.884668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385497+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:10:54.385497+0000","last_peered":"2024-04-16T21:10:54.385497+0000","last_clean":"2024-04-16T21:10:54.385497+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:10:54.385497+0000","last_undegraded":"2024-04-16T21:10:54.385497+0000","last_fullsized":"2024-04-16T21:10:54.385497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388567+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:11:09.388567+0000","last_peered":"2024-04-16T21:11:09.388567+0000","last_clean":"2024-04-16T21:11:09.388567+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:11:09.388567+0000","last_undegraded":"2024-04-16T21:11:09.388567+0000","last_fullsized":"2024-04-16T21:11:09.388567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377330+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:10:04.377330+0000","last_peered":"2024-04-16T21:10:04.377330+0000","last_clean":"2024-04-16T21:10:04.377330+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:10:04.377330+0000","last_undegraded":"2024-04-16T21:10:04.377330+0000","last_fullsized":"2024-04-16T21:10:04.377330+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393177+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:11:04.393177+0000","last_peered":"2024-04-16T21:11:04.393177+0000","last_clean":"2024-04-16T21:11:04.393177+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:11:04.393177+0000","last_undegraded":"2024-04-16T21:11:04.393177+0000","last_fullsized":"2024-04-16T21:11:04.393177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":350,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393850+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:11:09.393850+0000","last_peered":"2024-04-16T21:11:09.393850+0000","last_clean":"2024-04-16T21:11:09.393850+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:11:09.393850+0000","last_undegraded":"2024-04-16T21:11:09.393850+0000","last_fullsized":"2024-04-16T21:11:09.393850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387828+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:11:04.387828+0000","last_peered":"2024-04-16T21:11:04.387828+0000","last_clean":"2024-04-16T21:11:04.387828+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:11:04.387828+0000","last_undegraded":"2024-04-16T21:11:04.387828+0000","last_fullsized":"2024-04-16T21:11:04.387828+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.345372+0000","last_change":"2024-04-16T21:12:01.345418+0000","last_active":"2024-04-16T21:12:01.345372+0000","last_peered":"2024-04-16T21:12:01.345372+0000","last_clean":"2024-04-16T21:12:01.345372+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:12:01.345372+0000","last_undegraded":"2024-04-16T21:12:01.345372+0000","last_fullsized":"2024-04-16T21:12:01.345372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.345289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:22.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195947+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:11:43.195947+0000","last_peered":"2024-04-16T21:11:43.195947+0000","last_clean":"2024-04-16T21:11:43.195947+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:11:43.195947+0000","last_undegraded":"2024-04-16T21:11:43.195947+0000","last_fullsized":"2024-04-16T21:11:43.195947+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2004,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561557+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:11:44.561557+0000","last_peered":"2024-04-16T21:11:44.561557+0000","last_clean":"2024-04-16T21:11:44.561557+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:11:44.561557+0000","last_undegraded":"2024-04-16T21:11:44.561557+0000","last_fullsized":"2024-04-16T21:11:44.561557+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":81,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:59.516817+0000","last_change":"2024-04-16T19:25:41.510087+0000","last_active":"2024-04-16T21:09:59.516817+0000","last_peered":"2024-04-16T21:09:59.516817+0000","last_clean":"2024-04-16T21:09:59.516817+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:09:59.516817+0000","last_undegraded":"2024-04-16T21:09:59.516817+0000","last_fullsized":"2024-04-16T21:09:59.516817+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'23","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393194+0000","last_change":"2024-04-16T19:25:15.487252+0000","last_active":"2024-04-16T21:11:04.393194+0000","last_peered":"2024-04-16T21:11:04.393194+0000","last_clean":"2024-04-16T21:11:04.393194+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:11:04.393194+0000","last_undegraded":"2024-04-16T21:11:04.393194+0000","last_fullsized":"2024-04-16T21:11:04.393194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.802564+0000","last_change":"2024-04-16T21:12:02.802585+0000","last_active":"2024-04-16T21:12:02.802564+0000","last_peered":"2024-04-16T21:12:02.802564+0000","last_clean":"2024-04-16T21:12:02.802564+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:12:02.802564+0000","last_undegraded":"2024-04-16T21:12:02.802564+0000","last_fullsized":"2024-04-16T21:12:02.802564+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.802497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:44.917953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'739","reported_seq":1609,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.752466+0000","last_change":"2024-04-16T21:11:47.752492+0000","last_active":"2024-04-16T21:11:47.752466+0000","last_peered":"2024-04-16T21:11:47.752466+0000","last_clean":"2024-04-16T21:11:47.752466+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:11:47.752466+0000","last_undegraded":"2024-04-16T21:11:47.752466+0000","last_fullsized":"2024-04-16T21:11:47.752466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'739","last_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_deep_scrub":"52'739","last_deep_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.752406+0000","objects_scrubbed":25,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:14.225249+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6099,"num_omap_keys":28,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'12","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.362460+0000","last_change":"2024-04-16T21:11:45.362519+0000","last_active":"2024-04-16T21:11:45.362460+0000","last_peered":"2024-04-16T21:11:45.362460+0000","last_clean":"2024-04-16T21:11:45.362460+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:11:45.362460+0000","last_undegraded":"2024-04-16T21:11:45.362460+0000","last_fullsized":"2024-04-16T21:11:45.362460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.362399+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:08.553795+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391347+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:10:54.391347+0000","last_peered":"2024-04-16T21:10:54.391347+0000","last_clean":"2024-04-16T21:10:54.391347+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:10:54.391347+0000","last_undegraded":"2024-04-16T21:10:54.391347+0000","last_fullsized":"2024-04-16T21:10:54.391347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384586+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:10:49.384586+0000","last_peered":"2024-04-16T21:10:49.384586+0000","last_clean":"2024-04-16T21:10:49.384586+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:10:49.384586+0000","last_undegraded":"2024-04-16T21:10:49.384586+0000","last_fullsized":"2024-04-16T21:10:49.384586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381733+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:10:24.381733+0000","last_peered":"2024-04-16T21:10:24.381733+0000","last_clean":"2024-04-16T21:10:24.381733+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:10:24.381733+0000","last_undegraded":"2024-04-16T21:10:24.381733+0000","last_fullsized":"2024-04-16T21:10:24.381733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387738+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:11:04.387738+0000","last_peered":"2024-04-16T21:11:04.387738+0000","last_clean":"2024-04-16T21:11:04.387738+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:11:04.387738+0000","last_undegraded":"2024-04-16T21:11:04.387738+0000","last_fullsized":"2024-04-16T21:11:04.387738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":347,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387067+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:11:04.387067+0000","last_peered":"2024-04-16T21:11:04.387067+0000","last_clean":"2024-04-16T21:11:04.387067+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:11:04.387067+0000","last_undegraded":"2024-04-16T21:11:04.387067+0000","last_fullsized":"2024-04-16T21:11:04.387067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388421+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:11:09.388421+0000","last_peered":"2024-04-16T21:11:09.388421+0000","last_clean":"2024-04-16T21:11:09.388421+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:11:09.388421+0000","last_undegraded":"2024-04-16T21:11:09.388421+0000","last_fullsized":"2024-04-16T21:11:09.388421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.393316+0000","last_change":"2024-04-16T21:12:00.393341+0000","last_active":"2024-04-16T21:12:00.393316+0000","last_peered":"2024-04-16T21:12:00.393316+0000","last_clean":"2024-04-16T21:12:00.393316+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:12:00.393316+0000","last_undegraded":"2024-04-16T21:12:00.393316+0000","last_fullsized":"2024-04-16T21:12:00.393316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.393227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:57.237156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524744+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:10:49.524744+0000","last_peered":"2024-04-16T21:10:49.524744+0000","last_clean":"2024-04-16T21:10:49.524744+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:10:49.524744+0000","last_undegraded":"2024-04-16T21:10:49.524744+0000","last_fullsized":"2024-04-16T21:10:49.524744+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1527,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.379314+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:10:09.379314+0000","last_peered":"2024-04-16T21:10:09.379314+0000","last_clean":"2024-04-16T21:10:09.379314+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:10:09.379314+0000","last_undegraded":"2024-04-16T21:10:09.379314+0000","last_fullsized":"2024-04-16T21:10:09.379314+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561391+0000","last_change":"2024-04-16T19:25:41.498506+0000","last_active":"2024-04-16T21:11:44.561391+0000","last_peered":"2024-04-16T21:11:44.561391+0000","last_clean":"2024-04-16T21:11:44.561391+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:11:44.561391+0000","last_undegraded":"2024-04-16T21:11:44.561391+0000","last_fullsized":"2024-04-16T21:11:44.561391+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'4","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.763978+0000","last_change":"2024-04-16T21:11:55.764028+0000","last_active":"2024-04-16T21:11:55.763978+0000","last_peered":"2024-04-16T21:11:55.763978+0000","last_clean":"2024-04-16T21:11:55.763978+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:11:55.763978+0000","last_undegraded":"2024-04-16T21:11:55.763978+0000","last_fullsized":"2024-04-16T21:11:55.763978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.763887+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.184478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.390995+0000","last_change":"2024-04-16T19:25:13.473234+0000","last_active":"2024-04-16T21:11:19.390995+0000","last_peered":"2024-04-16T21:11:19.390995+0000","last_clean":"2024-04-16T21:11:19.390995+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:11:19.390995+0000","last_undegraded":"2024-04-16T21:11:19.390995+0000","last_fullsized":"2024-04-16T21:11:19.390995+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'287","reported_seq":553,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.658905+0000","last_change":"2024-04-16T21:11:54.658986+0000","last_active":"2024-04-16T21:11:54.658905+0000","last_peered":"2024-04-16T21:11:54.658905+0000","last_clean":"2024-04-16T21:11:54.658905+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:11:54.658905+0000","last_undegraded":"2024-04-16T21:11:54.658905+0000","last_fullsized":"2024-04-16T21:11:54.658905+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'287","last_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_deep_scrub":"52'287","last_deep_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.658842+0000","objects_scrubbed":20,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.559862+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6346,"num_omap_keys":29,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'22","reported_seq":213,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.414533+0000","last_change":"2024-04-16T21:11:51.414602+0000","last_active":"2024-04-16T21:11:51.414533+0000","last_peered":"2024-04-16T21:11:51.414533+0000","last_clean":"2024-04-16T21:11:51.414533+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:11:51.414533+0000","last_undegraded":"2024-04-16T21:11:51.414533+0000","last_fullsized":"2024-04-16T21:11:51.414533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.414471+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:12.853053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392142+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:10:59.392142+0000","last_peered":"2024-04-16T21:10:59.392142+0000","last_clean":"2024-04-16T21:10:59.392142+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:10:59.392142+0000","last_undegraded":"2024-04-16T21:10:59.392142+0000","last_fullsized":"2024-04-16T21:10:59.392142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.524015+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:10:44.524015+0000","last_peered":"2024-04-16T21:10:44.524015+0000","last_clean":"2024-04-16T21:10:44.524015+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:10:44.524015+0000","last_undegraded":"2024-04-16T21:10:44.524015+0000","last_fullsized":"2024-04-16T21:10:44.524015+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742796+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:11:42.742796+0000","last_peered":"2024-04-16T21:11:42.742796+0000","last_clean":"2024-04-16T21:11:42.742796+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:11:42.742796+0000","last_undegraded":"2024-04-16T21:11:42.742796+0000","last_fullsized":"2024-04-16T21:11:42.742796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388551+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:11:04.388551+0000","last_peered":"2024-04-16T21:11:04.388551+0000","last_clean":"2024-04-16T21:11:04.388551+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:11:04.388551+0000","last_undegraded":"2024-04-16T21:11:04.388551+0000","last_fullsized":"2024-04-16T21:11:04.388551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":281,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388599+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:11:04.388599+0000","last_peered":"2024-04-16T21:11:04.388599+0000","last_clean":"2024-04-16T21:11:04.388599+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:11:04.388599+0000","last_undegraded":"2024-04-16T21:11:04.388599+0000","last_fullsized":"2024-04-16T21:11:04.388599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393149+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:11:04.393149+0000","last_peered":"2024-04-16T21:11:04.393149+0000","last_clean":"2024-04-16T21:11:04.393149+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:11:04.393149+0000","last_undegraded":"2024-04-16T21:11:04.393149+0000","last_fullsized":"2024-04-16T21:11:04.393149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.373196+0000","last_change":"2024-04-16T21:11:59.373240+0000","last_active":"2024-04-16T21:11:59.373196+0000","last_peered":"2024-04-16T21:11:59.373196+0000","last_clean":"2024-04-16T21:11:59.373196+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:11:59.373196+0000","last_undegraded":"2024-04-16T21:11:59.373196+0000","last_fullsized":"2024-04-16T21:11:59.373196+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.373112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:02.682886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":55,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.518285+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:10:09.518285+0000","last_peered":"2024-04-16T21:10:09.518285+0000","last_clean":"2024-04-16T21:10:09.518285+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:10:09.518285+0000","last_undegraded":"2024-04-16T21:10:09.518285+0000","last_fullsized":"2024-04-16T21:10:09.518285+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1522,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.380044+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:10:14.380044+0000","last_peered":"2024-04-16T21:10:14.380044+0000","last_clean":"2024-04-16T21:10:14.380044+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:10:14.380044+0000","last_undegraded":"2024-04-16T21:10:14.380044+0000","last_fullsized":"2024-04-16T21:10:14.380044+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.846610+0000","last_change":"2024-04-16T21:12:00.846636+0000","last_active":"2024-04-16T21:12:00.846610+0000","last_peered":"2024-04-16T21:12:00.846610+0000","last_clean":"2024-04-16T21:12:00.846610+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:12:00.846610+0000","last_undegraded":"2024-04-16T21:12:00.846610+0000","last_fullsized":"2024-04-16T21:12:00.846610+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.846545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:04.784544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'15","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.794385+0000","last_change":"2024-04-16T21:11:56.794414+0000","last_active":"2024-04-16T21:11:56.794385+0000","last_peered":"2024-04-16T21:11:56.794385+0000","last_clean":"2024-04-16T21:11:56.794385+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:11:56.794385+0000","last_undegraded":"2024-04-16T21:11:56.794385+0000","last_fullsized":"2024-04-16T21:11:56.794385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.794323+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.217325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'706","reported_seq":2090,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.681698+0000","last_change":"2024-04-16T21:11:49.681723+0000","last_active":"2024-04-16T21:11:49.681698+0000","last_peered":"2024-04-16T21:11:49.681698+0000","last_clean":"2024-04-16T21:11:49.681698+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:11:49.681698+0000","last_undegraded":"2024-04-16T21:11:49.681698+0000","last_fullsized":"2024-04-16T21:11:49.681698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.681634+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.077508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'335","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.410155+0000","last_change":"2024-04-16T21:11:50.410223+0000","last_active":"2024-04-16T21:11:50.410155+0000","last_peered":"2024-04-16T21:11:50.410155+0000","last_clean":"2024-04-16T21:11:50.410155+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:11:50.410155+0000","last_undegraded":"2024-04-16T21:11:50.410155+0000","last_fullsized":"2024-04-16T21:11:50.410155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'335","last_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_deep_scrub":"52'335","last_deep_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.410083+0000","objects_scrubbed":17,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:14.580127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'6","reported_seq":183,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525474+0000","last_change":"2024-04-16T19:25:09.461256+0000","last_active":"2024-04-16T21:10:54.525474+0000","last_peered":"2024-04-16T21:10:54.525474+0000","last_clean":"2024-04-16T21:10:54.525474+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:10:54.525474+0000","last_undegraded":"2024-04-16T21:10:54.525474+0000","last_fullsized":"2024-04-16T21:10:54.525474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386268+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:10:54.386268+0000","last_peered":"2024-04-16T21:10:54.386268+0000","last_clean":"2024-04-16T21:10:54.386268+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:10:54.386268+0000","last_undegraded":"2024-04-16T21:10:54.386268+0000","last_fullsized":"2024-04-16T21:10:54.386268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.389709+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:10:44.389709+0000","last_peered":"2024-04-16T21:10:44.389709+0000","last_clean":"2024-04-16T21:10:44.389709+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:10:44.389709+0000","last_undegraded":"2024-04-16T21:10:44.389709+0000","last_fullsized":"2024-04-16T21:10:44.389709+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:59.377807+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:09:59.377807+0000","last_peered":"2024-04-16T21:09:59.377807+0000","last_clean":"2024-04-16T21:09:59.377807+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:09:59.377807+0000","last_undegraded":"2024-04-16T21:09:59.377807+0000","last_fullsized":"2024-04-16T21:09:59.377807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527115+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:11:04.527115+0000","last_peered":"2024-04-16T21:11:04.527115+0000","last_clean":"2024-04-16T21:11:04.527115+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:11:04.527115+0000","last_undegraded":"2024-04-16T21:11:04.527115+0000","last_fullsized":"2024-04-16T21:11:04.527115+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":332,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387885+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:11:04.387885+0000","last_peered":"2024-04-16T21:11:04.387885+0000","last_clean":"2024-04-16T21:11:04.387885+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:11:04.387885+0000","last_undegraded":"2024-04-16T21:11:04.387885+0000","last_fullsized":"2024-04-16T21:11:04.387885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527173+0000","last_change":"2024-04-16T19:24:49.435265+0000","last_active":"2024-04-16T21:11:04.527173+0000","last_peered":"2024-04-16T21:11:04.527173+0000","last_clean":"2024-04-16T21:11:04.527173+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:11:04.527173+0000","last_undegraded":"2024-04-16T21:11:04.527173+0000","last_fullsized":"2024-04-16T21:11:04.527173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.383799+0000","last_change":"2024-04-16T19:24:51.438071+0000","last_active":"2024-04-16T21:10:39.383799+0000","last_peered":"2024-04-16T21:10:39.383799+0000","last_clean":"2024-04-16T21:10:39.383799+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:10:39.383799+0000","last_undegraded":"2024-04-16T21:10:39.383799+0000","last_fullsized":"2024-04-16T21:10:39.383799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.379660+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:09:44.379660+0000","last_peered":"2024-04-16T21:09:44.379660+0000","last_clean":"2024-04-16T21:09:44.379660+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:09:44.379660+0000","last_undegraded":"2024-04-16T21:09:44.379660+0000","last_fullsized":"2024-04-16T21:09:44.379660+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1570,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378464+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:10:04.378464+0000","last_peered":"2024-04-16T21:10:04.378464+0000","last_clean":"2024-04-16T21:10:04.378464+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:10:04.378464+0000","last_undegraded":"2024-04-16T21:10:04.378464+0000","last_fullsized":"2024-04-16T21:10:04.378464+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":75,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383616+0000","last_change":"2024-04-16T19:25:41.506596+0000","last_active":"2024-04-16T21:10:44.383616+0000","last_peered":"2024-04-16T21:10:44.383616+0000","last_clean":"2024-04-16T21:10:44.383616+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:10:44.383616+0000","last_undegraded":"2024-04-16T21:10:44.383616+0000","last_fullsized":"2024-04-16T21:10:44.383616+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'4","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380159+0000","last_change":"2024-04-16T19:25:15.480477+0000","last_active":"2024-04-16T21:10:19.380159+0000","last_peered":"2024-04-16T21:10:19.380159+0000","last_clean":"2024-04-16T21:10:19.380159+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:10:19.380159+0000","last_undegraded":"2024-04-16T21:10:19.380159+0000","last_fullsized":"2024-04-16T21:10:19.380159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.662892+0000","last_change":"2024-04-16T21:11:53.662915+0000","last_active":"2024-04-16T21:11:53.662892+0000","last_peered":"2024-04-16T21:11:53.662892+0000","last_clean":"2024-04-16T21:11:53.662892+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:11:53.662892+0000","last_undegraded":"2024-04-16T21:11:53.662892+0000","last_fullsized":"2024-04-16T21:11:53.662892+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.662831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:53.840637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'317","reported_seq":612,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.662747+0000","last_change":"2024-04-16T21:11:52.662773+0000","last_active":"2024-04-16T21:11:52.662747+0000","last_peered":"2024-04-16T21:11:52.662747+0000","last_clean":"2024-04-16T21:11:52.662747+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:11:52.662747+0000","last_undegraded":"2024-04-16T21:11:52.662747+0000","last_fullsized":"2024-04-16T21:11:52.662747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'317","last_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_deep_scrub":"52'317","last_deep_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.662688+0000","objects_scrubbed":25,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:03:07.759388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3966,"num_omap_keys":18,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'20","reported_seq":253,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.170966+0000","last_change":"2024-04-16T21:11:53.171045+0000","last_active":"2024-04-16T21:11:53.170966+0000","last_peered":"2024-04-16T21:11:53.170966+0000","last_clean":"2024-04-16T21:11:53.170966+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:11:53.170966+0000","last_undegraded":"2024-04-16T21:11:53.170966+0000","last_fullsized":"2024-04-16T21:11:53.170966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.170891+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:56.498676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.394059+0000","last_change":"2024-04-16T19:24:51.434685+0000","last_active":"2024-04-16T21:11:09.394059+0000","last_peered":"2024-04-16T21:11:09.394059+0000","last_clean":"2024-04-16T21:11:09.394059+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:11:09.394059+0000","last_undegraded":"2024-04-16T21:11:09.394059+0000","last_fullsized":"2024-04-16T21:11:09.394059+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106369+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:11:44.106369+0000","last_peered":"2024-04-16T21:11:44.106369+0000","last_clean":"2024-04-16T21:11:44.106369+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:11:44.106369+0000","last_undegraded":"2024-04-16T21:11:44.106369+0000","last_fullsized":"2024-04-16T21:11:44.106369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387132+0000","last_change":"2024-04-16T19:25:41.504766+0000","last_active":"2024-04-16T21:11:04.387132+0000","last_peered":"2024-04-16T21:11:04.387132+0000","last_clean":"2024-04-16T21:11:04.387132+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:11:04.387132+0000","last_undegraded":"2024-04-16T21:11:04.387132+0000","last_fullsized":"2024-04-16T21:11:04.387132+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":744,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374802+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:09:49.374802+0000","last_peered":"2024-04-16T21:09:49.374802+0000","last_clean":"2024-04-16T21:09:49.374802+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:09:49.374802+0000","last_undegraded":"2024-04-16T21:09:49.374802+0000","last_fullsized":"2024-04-16T21:09:49.374802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":259,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.515254+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:09:49.515254+0000","last_peered":"2024-04-16T21:09:49.515254+0000","last_clean":"2024-04-16T21:09:49.515254+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:09:49.515254+0000","last_undegraded":"2024-04-16T21:09:49.515254+0000","last_fullsized":"2024-04-16T21:09:49.515254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374733+0000","last_change":"2024-04-16T19:24:49.434342+0000","last_active":"2024-04-16T21:09:49.374733+0000","last_peered":"2024-04-16T21:09:49.374733+0000","last_clean":"2024-04-16T21:09:49.374733+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:09:49.374733+0000","last_undegraded":"2024-04-16T21:09:49.374733+0000","last_fullsized":"2024-04-16T21:09:49.374733+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:59.377157+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:09:59.377157+0000","last_peered":"2024-04-16T21:09:59.377157+0000","last_clean":"2024-04-16T21:09:59.377157+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:09:59.377157+0000","last_undegraded":"2024-04-16T21:09:59.377157+0000","last_fullsized":"2024-04-16T21:09:59.377157+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'118","reported_seq":1712,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.373774+0000","last_change":"2024-04-16T19:25:49.520399+0000","last_active":"2024-04-16T21:09:44.373774+0000","last_peered":"2024-04-16T21:09:44.373774+0000","last_clean":"2024-04-16T21:09:44.373774+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:09:44.373774+0000","last_undegraded":"2024-04-16T21:09:44.373774+0000","last_fullsized":"2024-04-16T21:09:44.373774+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.752918+0000","last_change":"2024-04-16T21:12:01.752964+0000","last_active":"2024-04-16T21:12:01.752918+0000","last_peered":"2024-04-16T21:12:01.752918+0000","last_clean":"2024-04-16T21:12:01.752918+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:12:01.752918+0000","last_undegraded":"2024-04-16T21:12:01.752918+0000","last_fullsized":"2024-04-16T21:12:01.752918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.752809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.500440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380100+0000","last_change":"2024-04-16T19:25:15.482484+0000","last_active":"2024-04-16T21:10:19.380100+0000","last_peered":"2024-04-16T21:10:19.380100+0000","last_clean":"2024-04-16T21:10:19.380100+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:10:19.380100+0000","last_undegraded":"2024-04-16T21:10:19.380100+0000","last_fullsized":"2024-04-16T21:10:19.380100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.380162+0000","last_change":"2024-04-16T19:25:13.472620+0000","last_active":"2024-04-16T21:10:14.380162+0000","last_peered":"2024-04-16T21:10:14.380162+0000","last_clean":"2024-04-16T21:10:14.380162+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:10:14.380162+0000","last_undegraded":"2024-04-16T21:10:14.380162+0000","last_fullsized":"2024-04-16T21:10:14.380162+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1163","reported_seq":2930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.718441+0000","last_change":"2024-04-16T21:11:46.718512+0000","last_active":"2024-04-16T21:11:46.718441+0000","last_peered":"2024-04-16T21:11:46.718441+0000","last_clean":"2024-04-16T21:11:46.718441+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:11:46.718441+0000","last_undegraded":"2024-04-16T21:11:46.718441+0000","last_fullsized":"2024-04-16T21:11:46.718441+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1163","last_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_deep_scrub":"52'1163","last_deep_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.718377+0000","objects_scrubbed":23,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:06.764370+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'16","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.594571+0000","last_change":"2024-04-16T21:11:51.594618+0000","last_active":"2024-04-16T21:11:51.594571+0000","last_peered":"2024-04-16T21:11:51.594571+0000","last_clean":"2024-04-16T21:11:51.594571+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:11:51.594571+0000","last_undegraded":"2024-04-16T21:11:51.594571+0000","last_fullsized":"2024-04-16T21:11:51.594571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.594510+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:52.095893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.782064+0000","last_change":"2024-04-16T21:12:01.782087+0000","last_active":"2024-04-16T21:12:01.782064+0000","last_peered":"2024-04-16T21:12:01.782064+0000","last_clean":"2024-04-16T21:12:01.782064+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:12:01.782064+0000","last_undegraded":"2024-04-16T21:12:01.782064+0000","last_fullsized":"2024-04-16T21:12:01.782064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.781996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:45.434631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743093+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:11:42.743093+0000","last_peered":"2024-04-16T21:11:42.743093+0000","last_clean":"2024-04-16T21:11:42.743093+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:11:42.743093+0000","last_undegraded":"2024-04-16T21:11:42.743093+0000","last_fullsized":"2024-04-16T21:11:42.743093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386872+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:10:54.386872+0000","last_peered":"2024-04-16T21:10:54.386872+0000","last_clean":"2024-04-16T21:10:54.386872+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:10:54.386872+0000","last_undegraded":"2024-04-16T21:10:54.386872+0000","last_fullsized":"2024-04-16T21:10:54.386872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374835+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:09:49.374835+0000","last_peered":"2024-04-16T21:09:49.374835+0000","last_clean":"2024-04-16T21:09:49.374835+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:09:49.374835+0000","last_undegraded":"2024-04-16T21:09:49.374835+0000","last_fullsized":"2024-04-16T21:09:49.374835+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":178,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.380595+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:09:49.380595+0000","last_peered":"2024-04-16T21:09:49.380595+0000","last_clean":"2024-04-16T21:09:49.380595+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:09:49.380595+0000","last_undegraded":"2024-04-16T21:09:49.380595+0000","last_fullsized":"2024-04-16T21:09:49.380595+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375904+0000","last_change":"2024-04-16T19:24:49.431431+0000","last_active":"2024-04-16T21:09:49.375904+0000","last_peered":"2024-04-16T21:09:49.375904+0000","last_clean":"2024-04-16T21:09:49.375904+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:09:49.375904+0000","last_undegraded":"2024-04-16T21:09:49.375904+0000","last_fullsized":"2024-04-16T21:09:49.375904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378320+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:10:04.378320+0000","last_peered":"2024-04-16T21:10:04.378320+0000","last_clean":"2024-04-16T21:10:04.378320+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:10:04.378320+0000","last_undegraded":"2024-04-16T21:10:04.378320+0000","last_fullsized":"2024-04-16T21:10:04.378320+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1704,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384537+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:10:49.384537+0000","last_peered":"2024-04-16T21:10:49.384537+0000","last_clean":"2024-04-16T21:10:49.384537+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:10:49.384537+0000","last_undegraded":"2024-04-16T21:10:49.384537+0000","last_fullsized":"2024-04-16T21:10:49.384537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561452+0000","last_change":"2024-04-16T19:25:41.499274+0000","last_active":"2024-04-16T21:11:44.561452+0000","last_peered":"2024-04-16T21:11:44.561452+0000","last_clean":"2024-04-16T21:11:44.561452+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:11:44.561452+0000","last_undegraded":"2024-04-16T21:11:44.561452+0000","last_fullsized":"2024-04-16T21:11:44.561452+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'20","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293061+0000","last_change":"2024-04-16T19:25:15.485235+0000","last_active":"2024-04-16T21:11:42.293061+0000","last_peered":"2024-04-16T21:11:42.293061+0000","last_clean":"2024-04-16T21:11:42.293061+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:11:42.293061+0000","last_undegraded":"2024-04-16T21:11:42.293061+0000","last_fullsized":"2024-04-16T21:11:42.293061+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392721+0000","last_change":"2024-04-16T19:25:13.476041+0000","last_active":"2024-04-16T21:11:34.392721+0000","last_peered":"2024-04-16T21:11:34.392721+0000","last_clean":"2024-04-16T21:11:34.392721+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:11:34.392721+0000","last_undegraded":"2024-04-16T21:11:34.392721+0000","last_fullsized":"2024-04-16T21:11:34.392721+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'291","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.385570+0000","last_change":"2024-04-16T21:11:52.385621+0000","last_active":"2024-04-16T21:11:52.385570+0000","last_peered":"2024-04-16T21:11:52.385570+0000","last_clean":"2024-04-16T21:11:52.385570+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:11:52.385570+0000","last_undegraded":"2024-04-16T21:11:52.385570+0000","last_fullsized":"2024-04-16T21:11:52.385570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'291","last_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_deep_scrub":"52'291","last_deep_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.385512+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:13.385713+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'16","reported_seq":262,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.131892+0000","last_change":"2024-04-16T21:11:52.131942+0000","last_active":"2024-04-16T21:11:52.131892+0000","last_peered":"2024-04-16T21:11:52.131892+0000","last_clean":"2024-04-16T21:11:52.131892+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:11:52.131892+0000","last_undegraded":"2024-04-16T21:11:52.131892+0000","last_fullsized":"2024-04-16T21:11:52.131892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.131829+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:55.304717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.381671+0000","last_change":"2024-04-16T19:24:51.435662+0000","last_active":"2024-04-16T21:09:54.381671+0000","last_peered":"2024-04-16T21:09:54.381671+0000","last_clean":"2024-04-16T21:09:54.381671+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:09:54.381671+0000","last_undegraded":"2024-04-16T21:09:54.381671+0000","last_fullsized":"2024-04-16T21:09:54.381671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194933+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:11:43.194933+0000","last_peered":"2024-04-16T21:11:43.194933+0000","last_clean":"2024-04-16T21:11:43.194933+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:11:43.194933+0000","last_undegraded":"2024-04-16T21:11:43.194933+0000","last_fullsized":"2024-04-16T21:11:43.194933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386323+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:10:54.386323+0000","last_peered":"2024-04-16T21:10:54.386323+0000","last_clean":"2024-04-16T21:10:54.386323+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:10:54.386323+0000","last_undegraded":"2024-04-16T21:10:54.386323+0000","last_fullsized":"2024-04-16T21:10:54.386323+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":612,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375292+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:09:49.375292+0000","last_peered":"2024-04-16T21:09:49.375292+0000","last_clean":"2024-04-16T21:09:49.375292+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:09:49.375292+0000","last_undegraded":"2024-04-16T21:09:49.375292+0000","last_fullsized":"2024-04-16T21:09:49.375292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'96","reported_seq":259,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.515303+0000","last_change":"2024-04-16T19:24:47.424747+0000","last_active":"2024-04-16T21:09:49.515303+0000","last_peered":"2024-04-16T21:09:49.515303+0000","last_clean":"2024-04-16T21:09:49.515303+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:09:49.515303+0000","last_undegraded":"2024-04-16T21:09:49.515303+0000","last_fullsized":"2024-04-16T21:09:49.515303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375791+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:09:49.375791+0000","last_peered":"2024-04-16T21:09:49.375791+0000","last_clean":"2024-04-16T21:09:49.375791+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:09:49.375791+0000","last_undegraded":"2024-04-16T21:09:49.375791+0000","last_fullsized":"2024-04-16T21:09:49.375791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":55,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378544+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:10:04.378544+0000","last_peered":"2024-04-16T21:10:04.378544+0000","last_clean":"2024-04-16T21:10:04.378544+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:10:04.378544+0000","last_undegraded":"2024-04-16T21:10:04.378544+0000","last_fullsized":"2024-04-16T21:10:04.378544+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390537+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:10:49.390537+0000","last_peered":"2024-04-16T21:10:49.390537+0000","last_clean":"2024-04-16T21:10:49.390537+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:10:49.390537+0000","last_undegraded":"2024-04-16T21:10:49.390537+0000","last_fullsized":"2024-04-16T21:10:49.390537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":76,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.516110+0000","last_change":"2024-04-16T19:25:41.508866+0000","last_active":"2024-04-16T21:09:54.516110+0000","last_peered":"2024-04-16T21:09:54.516110+0000","last_clean":"2024-04-16T21:09:54.516110+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:09:54.516110+0000","last_undegraded":"2024-04-16T21:09:54.516110+0000","last_fullsized":"2024-04-16T21:09:54.516110+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'22","reported_seq":174,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.392730+0000","last_change":"2024-04-16T19:25:15.480626+0000","last_active":"2024-04-16T21:11:39.392730+0000","last_peered":"2024-04-16T21:11:39.392730+0000","last_clean":"2024-04-16T21:11:39.392730+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:11:39.392730+0000","last_undegraded":"2024-04-16T21:11:39.392730+0000","last_fullsized":"2024-04-16T21:11:39.392730+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'706","reported_seq":2151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.887851+0000","last_change":"2024-04-16T21:12:02.887901+0000","last_active":"2024-04-16T21:12:02.887851+0000","last_peered":"2024-04-16T21:12:02.887851+0000","last_clean":"2024-04-16T21:12:02.887851+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:12:02.887851+0000","last_undegraded":"2024-04-16T21:12:02.887851+0000","last_fullsized":"2024-04-16T21:12:02.887851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.887759+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:37.031244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1038","reported_seq":1547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.224293+0000","last_change":"2024-04-16T21:11:54.224379+0000","last_active":"2024-04-16T21:11:54.224293+0000","last_peered":"2024-04-16T21:11:54.224293+0000","last_clean":"2024-04-16T21:11:54.224293+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:11:54.224293+0000","last_undegraded":"2024-04-16T21:11:54.224293+0000","last_fullsized":"2024-04-16T21:11:54.224293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1038","last_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_deep_scrub":"52'1038","last_deep_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.224224+0000","objects_scrubbed":21,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:22.282373+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2194,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'23","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.136772+0000","last_change":"2024-04-16T21:11:43.136819+0000","last_active":"2024-04-16T21:11:43.136772+0000","last_peered":"2024-04-16T21:11:43.136772+0000","last_clean":"2024-04-16T21:11:43.136772+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:11:43.136772+0000","last_undegraded":"2024-04-16T21:11:43.136772+0000","last_fullsized":"2024-04-16T21:11:43.136772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'23","last_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_deep_scrub":"52'23","last_deep_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.136721+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:06.328117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.376822+0000","last_change":"2024-04-16T19:24:51.438772+0000","last_active":"2024-04-16T21:09:54.376822+0000","last_peered":"2024-04-16T21:09:54.376822+0000","last_clean":"2024-04-16T21:09:54.376822+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:09:54.376822+0000","last_undegraded":"2024-04-16T21:09:54.376822+0000","last_fullsized":"2024-04-16T21:09:54.376822+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743080+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:11:42.743080+0000","last_peered":"2024-04-16T21:11:42.743080+0000","last_clean":"2024-04-16T21:11:42.743080+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:11:42.743080+0000","last_undegraded":"2024-04-16T21:11:42.743080+0000","last_fullsized":"2024-04-16T21:11:42.743080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391438+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:10:54.391438+0000","last_peered":"2024-04-16T21:10:54.391438+0000","last_clean":"2024-04-16T21:10:54.391438+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:10:54.391438+0000","last_undegraded":"2024-04-16T21:10:54.391438+0000","last_fullsized":"2024-04-16T21:10:54.391438+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375926+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:09:49.375926+0000","last_peered":"2024-04-16T21:09:49.375926+0000","last_clean":"2024-04-16T21:09:49.375926+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:09:49.375926+0000","last_undegraded":"2024-04-16T21:09:49.375926+0000","last_fullsized":"2024-04-16T21:09:49.375926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":226,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375830+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:09:49.375830+0000","last_peered":"2024-04-16T21:09:49.375830+0000","last_clean":"2024-04-16T21:09:49.375830+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:09:49.375830+0000","last_undegraded":"2024-04-16T21:09:49.375830+0000","last_fullsized":"2024-04-16T21:09:49.375830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.380679+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:09:49.380679+0000","last_peered":"2024-04-16T21:09:49.380679+0000","last_clean":"2024-04-16T21:09:49.380679+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:09:49.380679+0000","last_undegraded":"2024-04-16T21:09:49.380679+0000","last_fullsized":"2024-04-16T21:09:49.380679+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651543+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:11:43.651543+0000","last_peered":"2024-04-16T21:11:43.651543+0000","last_clean":"2024-04-16T21:11:43.651543+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:11:43.651543+0000","last_undegraded":"2024-04-16T21:11:43.651543+0000","last_fullsized":"2024-04-16T21:11:43.651543+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2029,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.375168+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:09:44.375168+0000","last_peered":"2024-04-16T21:09:44.375168+0000","last_clean":"2024-04-16T21:09:44.375168+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:09:44.375168+0000","last_undegraded":"2024-04-16T21:09:44.375168+0000","last_fullsized":"2024-04-16T21:09:44.375168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293163+0000","last_change":"2024-04-16T19:25:41.507140+0000","last_active":"2024-04-16T21:11:42.293163+0000","last_peered":"2024-04-16T21:11:42.293163+0000","last_clean":"2024-04-16T21:11:42.293163+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:11:42.293163+0000","last_undegraded":"2024-04-16T21:11:42.293163+0000","last_fullsized":"2024-04-16T21:11:42.293163+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'5","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.393820+0000","last_change":"2024-04-16T19:25:15.481140+0000","last_active":"2024-04-16T21:11:39.393820+0000","last_peered":"2024-04-16T21:11:39.393820+0000","last_clean":"2024-04-16T21:11:39.393820+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:11:39.393820+0000","last_undegraded":"2024-04-16T21:11:39.393820+0000","last_fullsized":"2024-04-16T21:11:39.393820+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.384792+0000","last_change":"2024-04-16T19:25:13.476134+0000","last_active":"2024-04-16T21:10:14.384792+0000","last_peered":"2024-04-16T21:10:14.384792+0000","last_clean":"2024-04-16T21:10:14.384792+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:10:14.384792+0000","last_undegraded":"2024-04-16T21:10:14.384792+0000","last_fullsized":"2024-04-16T21:10:14.384792+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'329","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.716248+0000","last_change":"2024-04-16T21:11:45.716329+0000","last_active":"2024-04-16T21:11:45.716248+0000","last_peered":"2024-04-16T21:11:45.716248+0000","last_clean":"2024-04-16T21:11:45.716248+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:11:45.716248+0000","last_undegraded":"2024-04-16T21:11:45.716248+0000","last_fullsized":"2024-04-16T21:11:45.716248+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'329","last_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_deep_scrub":"52'329","last_deep_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.716182+0000","objects_scrubbed":24,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:24.600927+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'11","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.675377+0000","last_change":"2024-04-16T21:11:50.675426+0000","last_active":"2024-04-16T21:11:50.675377+0000","last_peered":"2024-04-16T21:11:50.675377+0000","last_clean":"2024-04-16T21:11:50.675377+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:11:50.675377+0000","last_undegraded":"2024-04-16T21:11:50.675377+0000","last_fullsized":"2024-04-16T21:11:50.675377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_deep_scrub":"52'11","last_deep_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.675302+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:31.269067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.387000+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:10:54.387000+0000","last_peered":"2024-04-16T21:10:54.387000+0000","last_clean":"2024-04-16T21:10:54.387000+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:10:54.387000+0000","last_undegraded":"2024-04-16T21:10:54.387000+0000","last_fullsized":"2024-04-16T21:10:54.387000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525357+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:10:54.525357+0000","last_peered":"2024-04-16T21:10:54.525357+0000","last_clean":"2024-04-16T21:10:54.525357+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:10:54.525357+0000","last_undegraded":"2024-04-16T21:10:54.525357+0000","last_fullsized":"2024-04-16T21:10:54.525357+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.394031+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:11:09.394031+0000","last_peered":"2024-04-16T21:11:09.394031+0000","last_clean":"2024-04-16T21:11:09.394031+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:11:09.394031+0000","last_undegraded":"2024-04-16T21:11:09.394031+0000","last_fullsized":"2024-04-16T21:11:09.394031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":240,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374904+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:09:49.374904+0000","last_peered":"2024-04-16T21:09:49.374904+0000","last_clean":"2024-04-16T21:09:49.374904+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:09:49.374904+0000","last_undegraded":"2024-04-16T21:09:49.374904+0000","last_fullsized":"2024-04-16T21:09:49.374904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375999+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:09:49.375999+0000","last_peered":"2024-04-16T21:09:49.375999+0000","last_clean":"2024-04-16T21:09:49.375999+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:09:49.375999+0000","last_undegraded":"2024-04-16T21:09:49.375999+0000","last_fullsized":"2024-04-16T21:09:49.375999+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.732024+0000","last_change":"2024-04-16T21:11:59.732045+0000","last_active":"2024-04-16T21:11:59.732024+0000","last_peered":"2024-04-16T21:11:59.732024+0000","last_clean":"2024-04-16T21:11:59.732024+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:11:59.732024+0000","last_undegraded":"2024-04-16T21:11:59.732024+0000","last_fullsized":"2024-04-16T21:11:59.732024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.731957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:37.824609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.374987+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:09:44.374987+0000","last_peered":"2024-04-16T21:09:44.374987+0000","last_clean":"2024-04-16T21:09:44.374987+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:09:44.374987+0000","last_undegraded":"2024-04-16T21:09:44.374987+0000","last_fullsized":"2024-04-16T21:09:44.374987+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1827,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388716+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:11:09.388716+0000","last_peered":"2024-04-16T21:11:09.388716+0000","last_clean":"2024-04-16T21:11:09.388716+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:11:09.388716+0000","last_undegraded":"2024-04-16T21:11:09.388716+0000","last_fullsized":"2024-04-16T21:11:09.388716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561443+0000","last_change":"2024-04-16T19:25:41.504184+0000","last_active":"2024-04-16T21:11:44.561443+0000","last_peered":"2024-04-16T21:11:44.561443+0000","last_clean":"2024-04-16T21:11:44.561443+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:11:44.561443+0000","last_undegraded":"2024-04-16T21:11:44.561443+0000","last_fullsized":"2024-04-16T21:11:44.561443+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.393618+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:11:39.393618+0000","last_peered":"2024-04-16T21:11:39.393618+0000","last_clean":"2024-04-16T21:11:39.393618+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:11:39.393618+0000","last_undegraded":"2024-04-16T21:11:39.393618+0000","last_fullsized":"2024-04-16T21:11:39.393618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.518996+0000","last_change":"2024-04-16T19:25:13.472015+0000","last_active":"2024-04-16T21:10:14.518996+0000","last_peered":"2024-04-16T21:10:14.518996+0000","last_clean":"2024-04-16T21:10:14.518996+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:10:14.518996+0000","last_undegraded":"2024-04-16T21:10:14.518996+0000","last_fullsized":"2024-04-16T21:10:14.518996+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'138","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.719397+0000","last_change":"2024-04-16T21:11:48.719459+0000","last_active":"2024-04-16T21:11:48.719397+0000","last_peered":"2024-04-16T21:11:48.719397+0000","last_clean":"2024-04-16T21:11:48.719397+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:11:48.719397+0000","last_undegraded":"2024-04-16T21:11:48.719397+0000","last_fullsized":"2024-04-16T21:11:48.719397+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.719337+0000","objects_scrubbed":17,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.768315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4229,"num_omap_keys":20,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'8","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.405923+0000","last_change":"2024-04-16T21:11:49.405983+0000","last_active":"2024-04-16T21:11:49.405923+0000","last_peered":"2024-04-16T21:11:49.405923+0000","last_clean":"2024-04-16T21:11:49.405923+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:11:49.405923+0000","last_undegraded":"2024-04-16T21:11:49.405923+0000","last_fullsized":"2024-04-16T21:11:49.405923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.405858+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:30.100703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392340+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:10:59.392340+0000","last_peered":"2024-04-16T21:10:59.392340+0000","last_clean":"2024-04-16T21:10:59.392340+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:10:59.392340+0000","last_undegraded":"2024-04-16T21:10:59.392340+0000","last_fullsized":"2024-04-16T21:10:59.392340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.388626+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:11:14.388626+0000","last_peered":"2024-04-16T21:11:14.388626+0000","last_clean":"2024-04-16T21:11:14.388626+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:11:14.388626+0000","last_undegraded":"2024-04-16T21:11:14.388626+0000","last_fullsized":"2024-04-16T21:11:14.388626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388762+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:11:04.388762+0000","last_peered":"2024-04-16T21:11:04.388762+0000","last_clean":"2024-04-16T21:11:04.388762+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:11:04.388762+0000","last_undegraded":"2024-04-16T21:11:04.388762+0000","last_fullsized":"2024-04-16T21:11:04.388762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.376011+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:09:49.376011+0000","last_peered":"2024-04-16T21:09:49.376011+0000","last_clean":"2024-04-16T21:09:49.376011+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:09:49.376011+0000","last_undegraded":"2024-04-16T21:09:49.376011+0000","last_fullsized":"2024-04-16T21:09:49.376011+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374897+0000","last_change":"2024-04-16T19:24:49.432195+0000","last_active":"2024-04-16T21:09:49.374897+0000","last_peered":"2024-04-16T21:09:49.374897+0000","last_clean":"2024-04-16T21:09:49.374897+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:09:49.374897+0000","last_undegraded":"2024-04-16T21:09:49.374897+0000","last_fullsized":"2024-04-16T21:09:49.374897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.517904+0000","last_change":"2024-04-16T21:12:01.517927+0000","last_active":"2024-04-16T21:12:01.517904+0000","last_peered":"2024-04-16T21:12:01.517904+0000","last_clean":"2024-04-16T21:12:01.517904+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:12:01.517904+0000","last_undegraded":"2024-04-16T21:12:01.517904+0000","last_fullsized":"2024-04-16T21:12:01.517904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:38.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.379850+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:09:44.379850+0000","last_peered":"2024-04-16T21:09:44.379850+0000","last_clean":"2024-04-16T21:09:44.379850+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:09:44.379850+0000","last_undegraded":"2024-04-16T21:09:44.379850+0000","last_fullsized":"2024-04-16T21:09:44.379850+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.373973+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:09:44.373973+0000","last_peered":"2024-04-16T21:09:44.373973+0000","last_clean":"2024-04-16T21:09:44.373973+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:09:44.373973+0000","last_undegraded":"2024-04-16T21:09:44.373973+0000","last_fullsized":"2024-04-16T21:09:44.373973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393228+0000","last_change":"2024-04-16T19:25:41.509494+0000","last_active":"2024-04-16T21:11:04.393228+0000","last_peered":"2024-04-16T21:11:04.393228+0000","last_clean":"2024-04-16T21:11:04.393228+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:11:04.393228+0000","last_undegraded":"2024-04-16T21:11:04.393228+0000","last_fullsized":"2024-04-16T21:11:04.393228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'8","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.393581+0000","last_change":"2024-04-16T19:25:15.483611+0000","last_active":"2024-04-16T21:11:39.393581+0000","last_peered":"2024-04-16T21:11:39.393581+0000","last_clean":"2024-04-16T21:11:39.393581+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:11:39.393581+0000","last_undegraded":"2024-04-16T21:11:39.393581+0000","last_fullsized":"2024-04-16T21:11:39.393581+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.380113+0000","last_change":"2024-04-16T19:25:13.472723+0000","last_active":"2024-04-16T21:10:14.380113+0000","last_peered":"2024-04-16T21:10:14.380113+0000","last_clean":"2024-04-16T21:10:14.380113+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:10:14.380113+0000","last_undegraded":"2024-04-16T21:10:14.380113+0000","last_fullsized":"2024-04-16T21:10:14.380113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'341","reported_seq":879,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.634057+0000","last_change":"2024-04-16T21:11:48.634146+0000","last_active":"2024-04-16T21:11:48.634057+0000","last_peered":"2024-04-16T21:11:48.634057+0000","last_clean":"2024-04-16T21:11:48.634057+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:11:48.634057+0000","last_undegraded":"2024-04-16T21:11:48.634057+0000","last_fullsized":"2024-04-16T21:11:48.634057+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'341","last_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_deep_scrub":"52'341","last_deep_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.633983+0000","objects_scrubbed":20,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:05:52.976164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'19","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.631104+0000","last_change":"2024-04-16T21:11:55.631145+0000","last_active":"2024-04-16T21:11:55.631104+0000","last_peered":"2024-04-16T21:11:55.631104+0000","last_clean":"2024-04-16T21:11:55.631104+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:11:55.631104+0000","last_undegraded":"2024-04-16T21:11:55.631104+0000","last_fullsized":"2024-04-16T21:11:55.631104+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.631042+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.051632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.378954+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:09:39.378954+0000","last_peered":"2024-04-16T21:09:39.378954+0000","last_clean":"2024-04-16T21:09:39.378954+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:09:39.378954+0000","last_undegraded":"2024-04-16T21:09:39.378954+0000","last_fullsized":"2024-04-16T21:09:39.378954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106228+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:11:44.106228+0000","last_peered":"2024-04-16T21:11:44.106228+0000","last_clean":"2024-04-16T21:11:44.106228+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:11:44.106228+0000","last_undegraded":"2024-04-16T21:11:44.106228+0000","last_fullsized":"2024-04-16T21:11:44.106228+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375437+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:09:49.375437+0000","last_peered":"2024-04-16T21:09:49.375437+0000","last_clean":"2024-04-16T21:09:49.375437+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:09:49.375437+0000","last_undegraded":"2024-04-16T21:09:49.375437+0000","last_fullsized":"2024-04-16T21:09:49.375437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":265,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527257+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:11:04.527257+0000","last_peered":"2024-04-16T21:11:04.527257+0000","last_clean":"2024-04-16T21:11:04.527257+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:11:04.527257+0000","last_undegraded":"2024-04-16T21:11:04.527257+0000","last_fullsized":"2024-04-16T21:11:04.527257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388030+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:11:09.388030+0000","last_peered":"2024-04-16T21:11:09.388030+0000","last_clean":"2024-04-16T21:11:09.388030+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:11:09.388030+0000","last_undegraded":"2024-04-16T21:11:09.388030+0000","last_fullsized":"2024-04-16T21:11:09.388030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.482623+0000","last_change":"2024-04-16T21:12:03.482650+0000","last_active":"2024-04-16T21:12:03.482623+0000","last_peered":"2024-04-16T21:12:03.482623+0000","last_clean":"2024-04-16T21:12:03.482623+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:12:03.482623+0000","last_undegraded":"2024-04-16T21:12:03.482623+0000","last_fullsized":"2024-04-16T21:12:03.482623+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.482557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:28.918007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.374663+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:09:44.374663+0000","last_peered":"2024-04-16T21:09:44.374663+0000","last_clean":"2024-04-16T21:09:44.374663+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:09:44.374663+0000","last_undegraded":"2024-04-16T21:09:44.374663+0000","last_fullsized":"2024-04-16T21:09:44.374663+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1307,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:44.373940+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:09:44.373940+0000","last_peered":"2024-04-16T21:09:44.373940+0000","last_clean":"2024-04-16T21:09:44.373940+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:09:44.373940+0000","last_undegraded":"2024-04-16T21:09:44.373940+0000","last_fullsized":"2024-04-16T21:09:44.373940+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107598+0000","last_change":"2024-04-16T19:25:41.509274+0000","last_active":"2024-04-16T21:11:44.107598+0000","last_peered":"2024-04-16T21:11:44.107598+0000","last_clean":"2024-04-16T21:11:44.107598+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:11:44.107598+0000","last_undegraded":"2024-04-16T21:11:44.107598+0000","last_fullsized":"2024-04-16T21:11:44.107598+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"47'5","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.383730+0000","last_change":"2024-04-16T19:25:15.486177+0000","last_active":"2024-04-16T21:10:39.383730+0000","last_peered":"2024-04-16T21:10:39.383730+0000","last_clean":"2024-04-16T21:10:39.383730+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:10:39.383730+0000","last_undegraded":"2024-04-16T21:10:39.383730+0000","last_fullsized":"2024-04-16T21:10:39.383730+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.380014+0000","last_change":"2024-04-16T19:25:13.473925+0000","last_active":"2024-04-16T21:10:14.380014+0000","last_peered":"2024-04-16T21:10:14.380014+0000","last_clean":"2024-04-16T21:10:14.380014+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:10:14.380014+0000","last_undegraded":"2024-04-16T21:10:14.380014+0000","last_fullsized":"2024-04-16T21:10:14.380014+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'264","reported_seq":1136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.695963+0000","last_change":"2024-04-16T21:11:49.696035+0000","last_active":"2024-04-16T21:11:49.695963+0000","last_peered":"2024-04-16T21:11:49.695963+0000","last_clean":"2024-04-16T21:11:49.695963+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:11:49.695963+0000","last_undegraded":"2024-04-16T21:11:49.695963+0000","last_fullsized":"2024-04-16T21:11:49.695963+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.695875+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:11.637965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2924,"num_omap_keys":15,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'12","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.501554+0000","last_change":"2024-04-16T21:11:59.501597+0000","last_active":"2024-04-16T21:11:59.501554+0000","last_peered":"2024-04-16T21:11:59.501554+0000","last_clean":"2024-04-16T21:11:59.501554+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:11:59.501554+0000","last_undegraded":"2024-04-16T21:11:59.501554+0000","last_fullsized":"2024-04-16T21:11:59.501554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.501496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:51.681807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.513766+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:09:39.513766+0000","last_peered":"2024-04-16T21:09:39.513766+0000","last_clean":"2024-04-16T21:09:39.513766+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:09:39.513766+0000","last_undegraded":"2024-04-16T21:09:39.513766+0000","last_fullsized":"2024-04-16T21:09:39.513766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291882+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:11:42.291882+0000","last_peered":"2024-04-16T21:11:42.291882+0000","last_clean":"2024-04-16T21:11:42.291882+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:11:42.291882+0000","last_undegraded":"2024-04-16T21:11:42.291882+0000","last_fullsized":"2024-04-16T21:11:42.291882+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":631,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.374885+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:09:49.374885+0000","last_peered":"2024-04-16T21:09:49.374885+0000","last_clean":"2024-04-16T21:09:49.374885+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:09:49.374885+0000","last_undegraded":"2024-04-16T21:09:49.374885+0000","last_fullsized":"2024-04-16T21:09:49.374885+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389503+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:11:09.389503+0000","last_peered":"2024-04-16T21:11:09.389503+0000","last_clean":"2024-04-16T21:11:09.389503+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:11:09.389503+0000","last_undegraded":"2024-04-16T21:11:09.389503+0000","last_fullsized":"2024-04-16T21:11:09.389503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387890+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:11:04.387890+0000","last_peered":"2024-04-16T21:11:04.387890+0000","last_clean":"2024-04-16T21:11:04.387890+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:11:04.387890+0000","last_undegraded":"2024-04-16T21:11:04.387890+0000","last_fullsized":"2024-04-16T21:11:04.387890+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:54.376746+0000","last_change":"2024-04-16T19:24:51.438145+0000","last_active":"2024-04-16T21:09:54.376746+0000","last_peered":"2024-04-16T21:09:54.376746+0000","last_clean":"2024-04-16T21:09:54.376746+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:09:54.376746+0000","last_undegraded":"2024-04-16T21:09:54.376746+0000","last_fullsized":"2024-04-16T21:09:54.376746+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392508+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:11:34.392508+0000","last_peered":"2024-04-16T21:11:34.392508+0000","last_clean":"2024-04-16T21:11:34.392508+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:11:34.392508+0000","last_undegraded":"2024-04-16T21:11:34.392508+0000","last_fullsized":"2024-04-16T21:11:34.392508+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2305,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385281+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:10:49.385281+0000","last_peered":"2024-04-16T21:10:49.385281+0000","last_clean":"2024-04-16T21:10:49.385281+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:10:49.385281+0000","last_undegraded":"2024-04-16T21:10:49.385281+0000","last_fullsized":"2024-04-16T21:10:49.385281+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.389595+0000","last_change":"2024-04-16T19:25:41.508939+0000","last_active":"2024-04-16T21:10:44.389595+0000","last_peered":"2024-04-16T21:10:44.389595+0000","last_clean":"2024-04-16T21:10:44.389595+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:10:44.389595+0000","last_undegraded":"2024-04-16T21:10:44.389595+0000","last_fullsized":"2024-04-16T21:10:44.389595+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'11","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.379635+0000","last_change":"2024-04-16T19:25:15.480096+0000","last_active":"2024-04-16T21:10:19.379635+0000","last_peered":"2024-04-16T21:10:19.379635+0000","last_clean":"2024-04-16T21:10:19.379635+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:10:19.379635+0000","last_undegraded":"2024-04-16T21:10:19.379635+0000","last_fullsized":"2024-04-16T21:10:19.379635+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383372+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:10:34.383372+0000","last_peered":"2024-04-16T21:10:34.383372+0000","last_clean":"2024-04-16T21:10:34.383372+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:10:34.383372+0000","last_undegraded":"2024-04-16T21:10:34.383372+0000","last_fullsized":"2024-04-16T21:10:34.383372+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'557","reported_seq":1382,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.205288+0000","last_change":"2024-04-16T21:11:50.205349+0000","last_active":"2024-04-16T21:11:50.205288+0000","last_peered":"2024-04-16T21:11:50.205288+0000","last_clean":"2024-04-16T21:11:50.205288+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:11:50.205288+0000","last_undegraded":"2024-04-16T21:11:50.205288+0000","last_fullsized":"2024-04-16T21:11:50.205288+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'557","last_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_deep_scrub":"52'557","last_deep_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.205222+0000","objects_scrubbed":17,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:09.070331+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2200,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'9","reported_seq":212,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391836+0000","last_change":"2024-04-16T19:25:09.458347+0000","last_active":"2024-04-16T21:11:29.391836+0000","last_peered":"2024-04-16T21:11:29.391836+0000","last_clean":"2024-04-16T21:11:29.391836+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:11:29.391836+0000","last_undegraded":"2024-04-16T21:11:29.391836+0000","last_fullsized":"2024-04-16T21:11:29.391836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373802+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:09:39.373802+0000","last_peered":"2024-04-16T21:09:39.373802+0000","last_clean":"2024-04-16T21:09:39.373802+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:09:39.373802+0000","last_undegraded":"2024-04-16T21:09:39.373802+0000","last_fullsized":"2024-04-16T21:09:39.373802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560193+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:11:44.560193+0000","last_peered":"2024-04-16T21:11:44.560193+0000","last_clean":"2024-04-16T21:11:44.560193+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:11:44.560193+0000","last_undegraded":"2024-04-16T21:11:44.560193+0000","last_fullsized":"2024-04-16T21:11:44.560193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375354+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:09:49.375354+0000","last_peered":"2024-04-16T21:09:49.375354+0000","last_clean":"2024-04-16T21:09:49.375354+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:09:49.375354+0000","last_undegraded":"2024-04-16T21:09:49.375354+0000","last_fullsized":"2024-04-16T21:09:49.375354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391659+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:11:29.391659+0000","last_peered":"2024-04-16T21:11:29.391659+0000","last_clean":"2024-04-16T21:11:29.391659+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:11:29.391659+0000","last_undegraded":"2024-04-16T21:11:29.391659+0000","last_fullsized":"2024-04-16T21:11:29.391659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393824+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:11:09.393824+0000","last_peered":"2024-04-16T21:11:09.393824+0000","last_clean":"2024-04-16T21:11:09.393824+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:11:09.393824+0000","last_undegraded":"2024-04-16T21:11:09.393824+0000","last_fullsized":"2024-04-16T21:11:09.393824+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390326+0000","last_change":"2024-04-16T19:24:51.437809+0000","last_active":"2024-04-16T21:11:14.390326+0000","last_peered":"2024-04-16T21:11:14.390326+0000","last_clean":"2024-04-16T21:11:14.390326+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:11:14.390326+0000","last_undegraded":"2024-04-16T21:11:14.390326+0000","last_fullsized":"2024-04-16T21:11:14.390326+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.532076+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:11:34.532076+0000","last_peered":"2024-04-16T21:11:34.532076+0000","last_clean":"2024-04-16T21:11:34.532076+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:11:34.532076+0000","last_undegraded":"2024-04-16T21:11:34.532076+0000","last_fullsized":"2024-04-16T21:11:34.532076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384453+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:10:49.384453+0000","last_peered":"2024-04-16T21:10:49.384453+0000","last_clean":"2024-04-16T21:10:49.384453+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:10:49.384453+0000","last_undegraded":"2024-04-16T21:10:49.384453+0000","last_fullsized":"2024-04-16T21:10:49.384453+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384446+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:10:44.384446+0000","last_peered":"2024-04-16T21:10:44.384446+0000","last_clean":"2024-04-16T21:10:44.384446+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:10:44.384446+0000","last_undegraded":"2024-04-16T21:10:44.384446+0000","last_fullsized":"2024-04-16T21:10:44.384446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'8","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385556+0000","last_change":"2024-04-16T19:25:15.481698+0000","last_active":"2024-04-16T21:10:19.385556+0000","last_peered":"2024-04-16T21:10:19.385556+0000","last_clean":"2024-04-16T21:10:19.385556+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:10:19.385556+0000","last_undegraded":"2024-04-16T21:10:19.385556+0000","last_fullsized":"2024-04-16T21:10:19.385556+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'706","reported_seq":2082,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.853855+0000","last_change":"2024-04-16T21:12:01.853885+0000","last_active":"2024-04-16T21:12:01.853855+0000","last_peered":"2024-04-16T21:12:01.853855+0000","last_clean":"2024-04-16T21:12:01.853855+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:12:01.853855+0000","last_undegraded":"2024-04-16T21:12:01.853855+0000","last_fullsized":"2024-04-16T21:12:01.853855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.853792+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.601423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'542","reported_seq":41299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.597167+0000","last_change":"2024-04-16T21:11:54.597266+0000","last_active":"2024-04-16T21:11:54.597167+0000","last_peered":"2024-04-16T21:11:54.597167+0000","last_clean":"2024-04-16T21:11:54.597167+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:11:54.597167+0000","last_undegraded":"2024-04-16T21:11:54.597167+0000","last_fullsized":"2024-04-16T21:11:54.597167+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'542","last_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_deep_scrub":"52'542","last_deep_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.597093+0000","objects_scrubbed":23,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:16.539182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'18","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.434844+0000","last_change":"2024-04-16T21:11:54.434914+0000","last_active":"2024-04-16T21:11:54.434844+0000","last_peered":"2024-04-16T21:11:54.434844+0000","last_clean":"2024-04-16T21:11:54.434844+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:11:54.434844+0000","last_undegraded":"2024-04-16T21:11:54.434844+0000","last_fullsized":"2024-04-16T21:11:54.434844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.434778+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:53.824584+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:39.373759+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:09:39.373759+0000","last_peered":"2024-04-16T21:09:39.373759+0000","last_clean":"2024-04-16T21:09:39.373759+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:09:39.373759+0000","last_undegraded":"2024-04-16T21:09:39.373759+0000","last_fullsized":"2024-04-16T21:09:39.373759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560163+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:11:44.560163+0000","last_peered":"2024-04-16T21:11:44.560163+0000","last_clean":"2024-04-16T21:11:44.560163+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:11:44.560163+0000","last_undegraded":"2024-04-16T21:11:44.560163+0000","last_fullsized":"2024-04-16T21:11:44.560163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375858+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:09:49.375858+0000","last_peered":"2024-04-16T21:09:49.375858+0000","last_clean":"2024-04-16T21:09:49.375858+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:09:49.375858+0000","last_undegraded":"2024-04-16T21:09:49.375858+0000","last_fullsized":"2024-04-16T21:09:49.375858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393868+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:11:09.393868+0000","last_peered":"2024-04-16T21:11:09.393868+0000","last_clean":"2024-04-16T21:11:09.393868+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:11:09.393868+0000","last_undegraded":"2024-04-16T21:11:09.393868+0000","last_fullsized":"2024-04-16T21:11:09.393868+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.392356+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:11:29.392356+0000","last_peered":"2024-04-16T21:11:29.392356+0000","last_clean":"2024-04-16T21:11:29.392356+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:11:29.392356+0000","last_undegraded":"2024-04-16T21:11:29.392356+0000","last_fullsized":"2024-04-16T21:11:29.392356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.776294+0000","last_change":"2024-04-16T21:12:00.776335+0000","last_active":"2024-04-16T21:12:00.776294+0000","last_peered":"2024-04-16T21:12:00.776294+0000","last_clean":"2024-04-16T21:12:00.776294+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:12:00.776294+0000","last_undegraded":"2024-04-16T21:12:00.776294+0000","last_fullsized":"2024-04-16T21:12:00.776294+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.776223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:20.604086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398019+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:11:34.398019+0000","last_peered":"2024-04-16T21:11:34.398019+0000","last_clean":"2024-04-16T21:11:34.398019+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:11:34.398019+0000","last_undegraded":"2024-04-16T21:11:34.398019+0000","last_fullsized":"2024-04-16T21:11:34.398019+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'167","reported_seq":2112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524727+0000","last_change":"2024-04-16T19:25:49.519298+0000","last_active":"2024-04-16T21:10:49.524727+0000","last_peered":"2024-04-16T21:10:49.524727+0000","last_clean":"2024-04-16T21:10:49.524727+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:10:49.524727+0000","last_undegraded":"2024-04-16T21:10:49.524727+0000","last_fullsized":"2024-04-16T21:10:49.524727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383525+0000","last_change":"2024-04-16T19:25:41.507268+0000","last_active":"2024-04-16T21:10:44.383525+0000","last_peered":"2024-04-16T21:10:44.383525+0000","last_clean":"2024-04-16T21:10:44.383525+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:10:44.383525+0000","last_undegraded":"2024-04-16T21:10:44.383525+0000","last_fullsized":"2024-04-16T21:10:44.383525+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380829+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:10:19.380829+0000","last_peered":"2024-04-16T21:10:19.380829+0000","last_clean":"2024-04-16T21:10:19.380829+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:10:19.380829+0000","last_undegraded":"2024-04-16T21:10:19.380829+0000","last_fullsized":"2024-04-16T21:10:19.380829+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.652503+0000","last_change":"2024-04-16T21:11:59.652525+0000","last_active":"2024-04-16T21:11:59.652503+0000","last_peered":"2024-04-16T21:11:59.652503+0000","last_clean":"2024-04-16T21:11:59.652503+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:11:59.652503+0000","last_undegraded":"2024-04-16T21:11:59.652503+0000","last_fullsized":"2024-04-16T21:11:59.652503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.652420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:18.517529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'592","reported_seq":1363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.540243+0000","last_change":"2024-04-16T21:12:00.540298+0000","last_active":"2024-04-16T21:12:00.540243+0000","last_peered":"2024-04-16T21:12:00.540243+0000","last_clean":"2024-04-16T21:12:00.540243+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:12:00.540243+0000","last_undegraded":"2024-04-16T21:12:00.540243+0000","last_fullsized":"2024-04-16T21:12:00.540243+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'592","last_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_deep_scrub":"52'592","last_deep_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.540183+0000","objects_scrubbed":19,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:48.575472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'32","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.552154+0000","last_change":"2024-04-16T21:11:43.552192+0000","last_active":"2024-04-16T21:11:43.552154+0000","last_peered":"2024-04-16T21:11:43.552154+0000","last_clean":"2024-04-16T21:11:43.552154+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:11:43.552154+0000","last_undegraded":"2024-04-16T21:11:43.552154+0000","last_fullsized":"2024-04-16T21:11:43.552154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.552108+0000","objects_scrubbed":9,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.135567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395593+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:11:19.395593+0000","last_peered":"2024-04-16T21:11:19.395593+0000","last_clean":"2024-04-16T21:11:19.395593+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:11:19.395593+0000","last_undegraded":"2024-04-16T21:11:19.395593+0000","last_fullsized":"2024-04-16T21:11:19.395593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524792+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:10:49.524792+0000","last_peered":"2024-04-16T21:10:49.524792+0000","last_clean":"2024-04-16T21:10:49.524792+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:10:49.524792+0000","last_undegraded":"2024-04-16T21:10:49.524792+0000","last_fullsized":"2024-04-16T21:10:49.524792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388587+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:11:09.388587+0000","last_peered":"2024-04-16T21:11:09.388587+0000","last_clean":"2024-04-16T21:11:09.388587+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:11:09.388587+0000","last_undegraded":"2024-04-16T21:11:09.388587+0000","last_fullsized":"2024-04-16T21:11:09.388587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.380707+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:09:49.380707+0000","last_peered":"2024-04-16T21:09:49.380707+0000","last_clean":"2024-04-16T21:09:49.380707+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:09:49.380707+0000","last_undegraded":"2024-04-16T21:09:49.380707+0000","last_fullsized":"2024-04-16T21:09:49.380707+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375963+0000","last_change":"2024-04-16T19:24:49.431355+0000","last_active":"2024-04-16T21:09:49.375963+0000","last_peered":"2024-04-16T21:09:49.375963+0000","last_clean":"2024-04-16T21:09:49.375963+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:09:49.375963+0000","last_undegraded":"2024-04-16T21:09:49.375963+0000","last_fullsized":"2024-04-16T21:09:49.375963+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.514240+0000","last_change":"2024-04-16T21:12:02.514262+0000","last_active":"2024-04-16T21:12:02.514240+0000","last_peered":"2024-04-16T21:12:02.514240+0000","last_clean":"2024-04-16T21:12:02.514240+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:12:02.514240+0000","last_undegraded":"2024-04-16T21:12:02.514240+0000","last_fullsized":"2024-04-16T21:12:02.514240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.514179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:44.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.391931+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:11:34.391931+0000","last_peered":"2024-04-16T21:11:34.391931+0000","last_clean":"2024-04-16T21:11:34.391931+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:11:34.391931+0000","last_undegraded":"2024-04-16T21:11:34.391931+0000","last_fullsized":"2024-04-16T21:11:34.391931+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'127","reported_seq":1510,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524699+0000","last_change":"2024-04-16T19:25:49.517472+0000","last_active":"2024-04-16T21:10:49.524699+0000","last_peered":"2024-04-16T21:10:49.524699+0000","last_clean":"2024-04-16T21:10:49.524699+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:10:49.524699+0000","last_undegraded":"2024-04-16T21:10:49.524699+0000","last_fullsized":"2024-04-16T21:10:49.524699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383585+0000","last_change":"2024-04-16T19:25:41.502982+0000","last_active":"2024-04-16T21:10:44.383585+0000","last_peered":"2024-04-16T21:10:44.383585+0000","last_clean":"2024-04-16T21:10:44.383585+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:10:44.383585+0000","last_undegraded":"2024-04-16T21:10:44.383585+0000","last_fullsized":"2024-04-16T21:10:44.383585+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"47'10","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385504+0000","last_change":"2024-04-16T19:25:15.481921+0000","last_active":"2024-04-16T21:10:19.385504+0000","last_peered":"2024-04-16T21:10:19.385504+0000","last_clean":"2024-04-16T21:10:19.385504+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:10:19.385504+0000","last_undegraded":"2024-04-16T21:10:19.385504+0000","last_fullsized":"2024-04-16T21:10:19.385504+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.693490+0000","last_change":"2024-04-16T21:12:01.693513+0000","last_active":"2024-04-16T21:12:01.693490+0000","last_peered":"2024-04-16T21:12:01.693490+0000","last_clean":"2024-04-16T21:12:01.693490+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:12:01.693490+0000","last_undegraded":"2024-04-16T21:12:01.693490+0000","last_fullsized":"2024-04-16T21:12:01.693490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.693430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:04.866318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'249","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.401831+0000","last_change":"2024-04-16T21:11:48.401911+0000","last_active":"2024-04-16T21:11:48.401831+0000","last_peered":"2024-04-16T21:11:48.401831+0000","last_clean":"2024-04-16T21:11:48.401831+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:11:48.401831+0000","last_undegraded":"2024-04-16T21:11:48.401831+0000","last_fullsized":"2024-04-16T21:11:48.401831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.401765+0000","objects_scrubbed":19,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:02.153361+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'6","reported_seq":176,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391761+0000","last_change":"2024-04-16T19:25:09.458030+0000","last_active":"2024-04-16T21:11:29.391761+0000","last_peered":"2024-04-16T21:11:29.391761+0000","last_clean":"2024-04-16T21:11:29.391761+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:11:29.391761+0000","last_undegraded":"2024-04-16T21:11:29.391761+0000","last_fullsized":"2024-04-16T21:11:29.391761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387647+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:10:59.387647+0000","last_peered":"2024-04-16T21:10:59.387647+0000","last_clean":"2024-04-16T21:10:59.387647+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:10:59.387647+0000","last_undegraded":"2024-04-16T21:10:59.387647+0000","last_fullsized":"2024-04-16T21:10:59.387647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390741+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:10:49.390741+0000","last_peered":"2024-04-16T21:10:49.390741+0000","last_clean":"2024-04-16T21:10:49.390741+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:10:49.390741+0000","last_undegraded":"2024-04-16T21:10:49.390741+0000","last_fullsized":"2024-04-16T21:10:49.390741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391073+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:11:29.391073+0000","last_peered":"2024-04-16T21:11:29.391073+0000","last_clean":"2024-04-16T21:11:29.391073+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:11:29.391073+0000","last_undegraded":"2024-04-16T21:11:29.391073+0000","last_fullsized":"2024-04-16T21:11:29.391073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":223,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.375369+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:09:49.375369+0000","last_peered":"2024-04-16T21:09:49.375369+0000","last_clean":"2024-04-16T21:09:49.375369+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:09:49.375369+0000","last_undegraded":"2024-04-16T21:09:49.375369+0000","last_fullsized":"2024-04-16T21:09:49.375369+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:09:49.380663+0000","last_change":"2024-04-16T19:24:49.432869+0000","last_active":"2024-04-16T21:09:49.380663+0000","last_peered":"2024-04-16T21:09:49.380663+0000","last_clean":"2024-04-16T21:09:49.380663+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:09:49.380663+0000","last_undegraded":"2024-04-16T21:09:49.380663+0000","last_fullsized":"2024-04-16T21:09:49.380663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.379502+0000","last_change":"2024-04-16T19:24:51.442598+0000","last_active":"2024-04-16T21:10:14.379502+0000","last_peered":"2024-04-16T21:10:14.379502+0000","last_clean":"2024-04-16T21:10:14.379502+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:10:14.379502+0000","last_undegraded":"2024-04-16T21:10:14.379502+0000","last_fullsized":"2024-04-16T21:10:14.379502+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:50.420558+0000","last_clean_scrub_stamp":"2024-04-16T19:24:50.420558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'12","reported_seq":187,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525440+0000","last_change":"2024-04-16T19:25:09.464004+0000","last_active":"2024-04-16T21:10:54.525440+0000","last_peered":"2024-04-16T21:10:54.525440+0000","last_clean":"2024-04-16T21:10:54.525440+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:10:54.525440+0000","last_undegraded":"2024-04-16T21:10:54.525440+0000","last_fullsized":"2024-04-16T21:10:54.525440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384672+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:10:49.384672+0000","last_peered":"2024-04-16T21:10:49.384672+0000","last_clean":"2024-04-16T21:10:49.384672+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:10:49.384672+0000","last_undegraded":"2024-04-16T21:10:49.384672+0000","last_fullsized":"2024-04-16T21:10:49.384672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385227+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:10:49.385227+0000","last_peered":"2024-04-16T21:10:49.385227+0000","last_clean":"2024-04-16T21:10:49.385227+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:10:49.385227+0000","last_undegraded":"2024-04-16T21:10:49.385227+0000","last_fullsized":"2024-04-16T21:10:49.385227+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'18","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.679302+0000","last_change":"2024-04-16T21:12:03.679355+0000","last_active":"2024-04-16T21:12:03.679302+0000","last_peered":"2024-04-16T21:12:03.679302+0000","last_clean":"2024-04-16T21:12:03.679302+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:12:03.679302+0000","last_undegraded":"2024-04-16T21:12:03.679302+0000","last_fullsized":"2024-04-16T21:12:03.679302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.679242+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:31.737391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384640+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:10:49.384640+0000","last_peered":"2024-04-16T21:10:49.384640+0000","last_clean":"2024-04-16T21:10:49.384640+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:10:49.384640+0000","last_undegraded":"2024-04-16T21:10:49.384640+0000","last_fullsized":"2024-04-16T21:10:49.384640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524668+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:10:49.524668+0000","last_peered":"2024-04-16T21:10:49.524668+0000","last_clean":"2024-04-16T21:10:49.524668+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:10:49.524668+0000","last_undegraded":"2024-04-16T21:10:49.524668+0000","last_fullsized":"2024-04-16T21:10:49.524668+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'10","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.673321+0000","last_change":"2024-04-16T21:12:02.673359+0000","last_active":"2024-04-16T21:12:02.673321+0000","last_peered":"2024-04-16T21:12:02.673321+0000","last_clean":"2024-04-16T21:12:02.673321+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:12:02.673321+0000","last_undegraded":"2024-04-16T21:12:02.673321+0000","last_fullsized":"2024-04-16T21:12:02.673321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.673266+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:06.001051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385387+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:10:49.385387+0000","last_peered":"2024-04-16T21:10:49.385387+0000","last_clean":"2024-04-16T21:10:49.385387+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:10:49.385387+0000","last_undegraded":"2024-04-16T21:10:49.385387+0000","last_fullsized":"2024-04-16T21:10:49.385387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385866+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:10:49.385866+0000","last_peered":"2024-04-16T21:10:49.385866+0000","last_clean":"2024-04-16T21:10:49.385866+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:10:49.385866+0000","last_undegraded":"2024-04-16T21:10:49.385866+0000","last_fullsized":"2024-04-16T21:10:49.385866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'13","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.447533+0000","last_change":"2024-04-16T21:11:57.447598+0000","last_active":"2024-04-16T21:11:57.447533+0000","last_peered":"2024-04-16T21:11:57.447533+0000","last_clean":"2024-04-16T21:11:57.447533+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:11:57.447533+0000","last_undegraded":"2024-04-16T21:11:57.447533+0000","last_fullsized":"2024-04-16T21:11:57.447533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.447471+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:25.758892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384571+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:10:49.384571+0000","last_peered":"2024-04-16T21:10:49.384571+0000","last_clean":"2024-04-16T21:10:49.384571+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:10:49.384571+0000","last_undegraded":"2024-04-16T21:10:49.384571+0000","last_fullsized":"2024-04-16T21:10:49.384571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393952+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:11:09.393952+0000","last_peered":"2024-04-16T21:11:09.393952+0000","last_clean":"2024-04-16T21:11:09.393952+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:11:09.393952+0000","last_undegraded":"2024-04-16T21:11:09.393952+0000","last_fullsized":"2024-04-16T21:11:09.393952+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'11","reported_seq":250,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.379256+0000","last_change":"2024-04-16T19:25:09.466470+0000","last_active":"2024-04-16T21:10:09.379256+0000","last_peered":"2024-04-16T21:10:09.379256+0000","last_clean":"2024-04-16T21:10:09.379256+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:10:09.379256+0000","last_undegraded":"2024-04-16T21:10:09.379256+0000","last_fullsized":"2024-04-16T21:10:09.379256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.528113+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:11:09.528113+0000","last_peered":"2024-04-16T21:11:09.528113+0000","last_clean":"2024-04-16T21:11:09.528113+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:11:09.528113+0000","last_undegraded":"2024-04-16T21:11:09.528113+0000","last_fullsized":"2024-04-16T21:11:09.528113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383688+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:10:44.383688+0000","last_peered":"2024-04-16T21:10:44.383688+0000","last_clean":"2024-04-16T21:10:44.383688+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:10:44.383688+0000","last_undegraded":"2024-04-16T21:10:44.383688+0000","last_fullsized":"2024-04-16T21:10:44.383688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'20","reported_seq":265,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.518381+0000","last_change":"2024-04-16T19:25:09.460695+0000","last_active":"2024-04-16T21:10:09.518381+0000","last_peered":"2024-04-16T21:10:09.518381+0000","last_clean":"2024-04-16T21:10:09.518381+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:10:09.518381+0000","last_undegraded":"2024-04-16T21:10:09.518381+0000","last_fullsized":"2024-04-16T21:10:09.518381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386300+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:10:54.386300+0000","last_peered":"2024-04-16T21:10:54.386300+0000","last_clean":"2024-04-16T21:10:54.386300+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:10:54.386300+0000","last_undegraded":"2024-04-16T21:10:54.386300+0000","last_fullsized":"2024-04-16T21:10:54.386300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383706+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:10:44.383706+0000","last_peered":"2024-04-16T21:10:44.383706+0000","last_clean":"2024-04-16T21:10:44.383706+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:10:44.383706+0000","last_undegraded":"2024-04-16T21:10:44.383706+0000","last_fullsized":"2024-04-16T21:10:44.383706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'12","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.624929+0000","last_change":"2024-04-16T21:11:55.624974+0000","last_active":"2024-04-16T21:11:55.624929+0000","last_peered":"2024-04-16T21:11:55.624929+0000","last_clean":"2024-04-16T21:11:55.624929+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:11:55.624929+0000","last_undegraded":"2024-04-16T21:11:55.624929+0000","last_fullsized":"2024-04-16T21:11:55.624929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.624828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:36.218593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385469+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:10:54.385469+0000","last_peered":"2024-04-16T21:10:54.385469+0000","last_clean":"2024-04-16T21:10:54.385469+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:10:54.385469+0000","last_undegraded":"2024-04-16T21:10:54.385469+0000","last_fullsized":"2024-04-16T21:10:54.385469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.385174+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:10:44.385174+0000","last_peered":"2024-04-16T21:10:44.385174+0000","last_clean":"2024-04-16T21:10:44.385174+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:10:44.385174+0000","last_undegraded":"2024-04-16T21:10:44.385174+0000","last_fullsized":"2024-04-16T21:10:44.385174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'13","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.528161+0000","last_change":"2024-04-16T19:25:09.459396+0000","last_active":"2024-04-16T21:11:09.528161+0000","last_peered":"2024-04-16T21:11:09.528161+0000","last_clean":"2024-04-16T21:11:09.528161+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:11:09.528161+0000","last_undegraded":"2024-04-16T21:11:09.528161+0000","last_fullsized":"2024-04-16T21:11:09.528161+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386679+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:10:54.386679+0000","last_peered":"2024-04-16T21:10:54.386679+0000","last_clean":"2024-04-16T21:10:54.386679+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:10:54.386679+0000","last_undegraded":"2024-04-16T21:10:54.386679+0000","last_fullsized":"2024-04-16T21:10:54.386679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.524026+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:10:44.524026+0000","last_peered":"2024-04-16T21:10:44.524026+0000","last_clean":"2024-04-16T21:10:44.524026+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:10:44.524026+0000","last_undegraded":"2024-04-16T21:10:44.524026+0000","last_fullsized":"2024-04-16T21:10:44.524026+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'46","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.408711+0000","last_change":"2024-04-16T21:11:46.408756+0000","last_active":"2024-04-16T21:11:46.408711+0000","last_peered":"2024-04-16T21:11:46.408711+0000","last_clean":"2024-04-16T21:11:46.408711+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:11:46.408711+0000","last_undegraded":"2024-04-16T21:11:46.408711+0000","last_fullsized":"2024-04-16T21:11:46.408711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'46","last_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_deep_scrub":"52'46","last_deep_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.408648+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:38.312038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292194+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:11:42.292194+0000","last_peered":"2024-04-16T21:11:42.292194+0000","last_clean":"2024-04-16T21:11:42.292194+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:11:42.292194+0000","last_undegraded":"2024-04-16T21:11:42.292194+0000","last_fullsized":"2024-04-16T21:11:42.292194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387643+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:11:04.387643+0000","last_peered":"2024-04-16T21:11:04.387643+0000","last_clean":"2024-04-16T21:11:04.387643+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:11:04.387643+0000","last_undegraded":"2024-04-16T21:11:04.387643+0000","last_fullsized":"2024-04-16T21:11:04.387643+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'20","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.919349+0000","last_change":"2024-04-16T21:11:47.919430+0000","last_active":"2024-04-16T21:11:47.919349+0000","last_peered":"2024-04-16T21:11:47.919349+0000","last_clean":"2024-04-16T21:11:47.919349+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:11:47.919349+0000","last_undegraded":"2024-04-16T21:11:47.919349+0000","last_fullsized":"2024-04-16T21:11:47.919349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.919276+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:11:50.792760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106421+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:11:44.106421+0000","last_peered":"2024-04-16T21:11:44.106421+0000","last_clean":"2024-04-16T21:11:44.106421+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:11:44.106421+0000","last_undegraded":"2024-04-16T21:11:44.106421+0000","last_fullsized":"2024-04-16T21:11:44.106421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385533+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:10:19.385533+0000","last_peered":"2024-04-16T21:10:19.385533+0000","last_clean":"2024-04-16T21:10:19.385533+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:10:19.385533+0000","last_undegraded":"2024-04-16T21:10:19.385533+0000","last_fullsized":"2024-04-16T21:10:19.385533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'17","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.209595+0000","last_change":"2024-04-16T21:11:47.209653+0000","last_active":"2024-04-16T21:11:47.209595+0000","last_peered":"2024-04-16T21:11:47.209595+0000","last_clean":"2024-04-16T21:11:47.209595+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:11:47.209595+0000","last_undegraded":"2024-04-16T21:11:47.209595+0000","last_fullsized":"2024-04-16T21:11:47.209595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.209532+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:30.862167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.523274+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:10:39.523274+0000","last_peered":"2024-04-16T21:10:39.523274+0000","last_clean":"2024-04-16T21:10:39.523274+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:10:39.523274+0000","last_undegraded":"2024-04-16T21:10:39.523274+0000","last_fullsized":"2024-04-16T21:10:39.523274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380884+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:10:19.380884+0000","last_peered":"2024-04-16T21:10:19.380884+0000","last_clean":"2024-04-16T21:10:19.380884+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:10:19.380884+0000","last_undegraded":"2024-04-16T21:10:19.380884+0000","last_fullsized":"2024-04-16T21:10:19.380884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'27","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.894735+0000","last_change":"2024-04-16T21:11:49.894811+0000","last_active":"2024-04-16T21:11:49.894735+0000","last_peered":"2024-04-16T21:11:49.894735+0000","last_clean":"2024-04-16T21:11:49.894735+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:11:49.894735+0000","last_undegraded":"2024-04-16T21:11:49.894735+0000","last_fullsized":"2024-04-16T21:11:49.894735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.894658+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.290532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381700+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:10:24.381700+0000","last_peered":"2024-04-16T21:10:24.381700+0000","last_clean":"2024-04-16T21:10:24.381700+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:10:24.381700+0000","last_undegraded":"2024-04-16T21:10:24.381700+0000","last_fullsized":"2024-04-16T21:10:24.381700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391119+0000","last_change":"2024-04-16T19:25:41.505958+0000","last_active":"2024-04-16T21:11:29.391119+0000","last_peered":"2024-04-16T21:11:29.391119+0000","last_clean":"2024-04-16T21:11:29.391119+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:11:29.391119+0000","last_undegraded":"2024-04-16T21:11:29.391119+0000","last_fullsized":"2024-04-16T21:11:29.391119+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'10","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.658621+0000","last_change":"2024-04-16T21:11:58.658671+0000","last_active":"2024-04-16T21:11:58.658621+0000","last_peered":"2024-04-16T21:11:58.658621+0000","last_clean":"2024-04-16T21:11:58.658621+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:11:58.658621+0000","last_undegraded":"2024-04-16T21:11:58.658621+0000","last_fullsized":"2024-04-16T21:11:58.658621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.658555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:31.251581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380887+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:10:24.380887+0000","last_peered":"2024-04-16T21:10:24.380887+0000","last_clean":"2024-04-16T21:10:24.380887+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:10:24.380887+0000","last_undegraded":"2024-04-16T21:10:24.380887+0000","last_fullsized":"2024-04-16T21:10:24.380887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380933+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:10:19.380933+0000","last_peered":"2024-04-16T21:10:19.380933+0000","last_clean":"2024-04-16T21:10:19.380933+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:10:19.380933+0000","last_undegraded":"2024-04-16T21:10:19.380933+0000","last_fullsized":"2024-04-16T21:10:19.380933+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'22","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.811095+0000","last_change":"2024-04-16T21:11:53.811157+0000","last_active":"2024-04-16T21:11:53.811095+0000","last_peered":"2024-04-16T21:11:53.811095+0000","last_clean":"2024-04-16T21:11:53.811095+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:11:53.811095+0000","last_undegraded":"2024-04-16T21:11:53.811095+0000","last_fullsized":"2024-04-16T21:11:53.811095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.811028+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:31.711738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.386283+0000","last_change":"2024-04-16T19:25:41.508398+0000","last_active":"2024-04-16T21:10:24.386283+0000","last_peered":"2024-04-16T21:10:24.386283+0000","last_clean":"2024-04-16T21:10:24.386283+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:10:24.386283+0000","last_undegraded":"2024-04-16T21:10:24.386283+0000","last_fullsized":"2024-04-16T21:10:24.386283+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.384269+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:10:39.384269+0000","last_peered":"2024-04-16T21:10:39.384269+0000","last_clean":"2024-04-16T21:10:39.384269+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:10:39.384269+0000","last_undegraded":"2024-04-16T21:10:39.384269+0000","last_fullsized":"2024-04-16T21:10:39.384269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'6","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.660156+0000","last_change":"2024-04-16T21:12:03.660192+0000","last_active":"2024-04-16T21:12:03.660156+0000","last_peered":"2024-04-16T21:12:03.660156+0000","last_clean":"2024-04-16T21:12:03.660156+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:12:03.660156+0000","last_undegraded":"2024-04-16T21:12:03.660156+0000","last_fullsized":"2024-04-16T21:12:03.660156+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.660092+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:42.368715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380355+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:10:24.380355+0000","last_peered":"2024-04-16T21:10:24.380355+0000","last_clean":"2024-04-16T21:10:24.380355+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:10:24.380355+0000","last_undegraded":"2024-04-16T21:10:24.380355+0000","last_fullsized":"2024-04-16T21:10:24.380355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194855+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:11:43.194855+0000","last_peered":"2024-04-16T21:11:43.194855+0000","last_clean":"2024-04-16T21:11:43.194855+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:11:43.194855+0000","last_undegraded":"2024-04-16T21:11:43.194855+0000","last_fullsized":"2024-04-16T21:11:43.194855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'26","reported_seq":302,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.166590+0000","last_change":"2024-04-16T21:11:46.166636+0000","last_active":"2024-04-16T21:11:46.166590+0000","last_peered":"2024-04-16T21:11:46.166590+0000","last_clean":"2024-04-16T21:11:46.166590+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:11:46.166590+0000","last_undegraded":"2024-04-16T21:11:46.166590+0000","last_fullsized":"2024-04-16T21:11:46.166590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.166525+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:05.994388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389319+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:11:09.389319+0000","last_peered":"2024-04-16T21:11:09.389319+0000","last_clean":"2024-04-16T21:11:09.389319+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:11:09.389319+0000","last_undegraded":"2024-04-16T21:11:09.389319+0000","last_fullsized":"2024-04-16T21:11:09.389319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560276+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:11:44.560276+0000","last_peered":"2024-04-16T21:11:44.560276+0000","last_clean":"2024-04-16T21:11:44.560276+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:11:44.560276+0000","last_undegraded":"2024-04-16T21:11:44.560276+0000","last_fullsized":"2024-04-16T21:11:44.560276+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'22","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.232256+0000","last_change":"2024-04-16T21:11:55.232333+0000","last_active":"2024-04-16T21:11:55.232256+0000","last_peered":"2024-04-16T21:11:55.232256+0000","last_clean":"2024-04-16T21:11:55.232256+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:11:55.232256+0000","last_undegraded":"2024-04-16T21:11:55.232256+0000","last_fullsized":"2024-04-16T21:11:55.232256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.232184+0000","objects_scrubbed":10,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:19.898271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650107+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:11:43.650107+0000","last_peered":"2024-04-16T21:11:43.650107+0000","last_clean":"2024-04-16T21:11:43.650107+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:11:43.650107+0000","last_undegraded":"2024-04-16T21:11:43.650107+0000","last_fullsized":"2024-04-16T21:11:43.650107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395747+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:11:19.395747+0000","last_peered":"2024-04-16T21:11:19.395747+0000","last_clean":"2024-04-16T21:11:19.395747+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:11:19.395747+0000","last_undegraded":"2024-04-16T21:11:19.395747+0000","last_fullsized":"2024-04-16T21:11:19.395747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'3","reported_seq":169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.518325+0000","last_change":"2024-04-16T19:25:09.460170+0000","last_active":"2024-04-16T21:10:09.518325+0000","last_peered":"2024-04-16T21:10:09.518325+0000","last_clean":"2024-04-16T21:10:09.518325+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:10:09.518325+0000","last_undegraded":"2024-04-16T21:10:09.518325+0000","last_fullsized":"2024-04-16T21:10:09.518325+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:08.438975+0000","last_clean_scrub_stamp":"2024-04-16T19:25:08.438975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742979+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:11:42.742979+0000","last_peered":"2024-04-16T21:11:42.742979+0000","last_clean":"2024-04-16T21:11:42.742979+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:11:42.742979+0000","last_undegraded":"2024-04-16T21:11:42.742979+0000","last_fullsized":"2024-04-16T21:11:42.742979+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:10:59.387111+0000","last_peered":"2024-04-16T21:10:59.387111+0000","last_clean":"2024-04-16T21:10:59.387111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:10:59.387111+0000","last_undegraded":"2024-04-16T21:10:59.387111+0000","last_fullsized":"2024-04-16T21:10:59.387111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4203,"ondisk_log_size":4203,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3420,"ondisk_log_size":3420,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1086,"ondisk_log_size":1086,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16207,"ondisk_log_size":16207,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5667,"ondisk_log_size":5667,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641559,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":41132,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":104816468,"statfs":{"total":107374182400,"available":107332063232,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641560,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641559,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":55568,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802032,"statfs":{"total":107374182400,"available":107317280768,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641560,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59228,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798372,"statfs":{"total":107374182400,"available":107313532928,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641560,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":45348,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":104812252,"statfs":{"total":107374182400,"available":107327746048,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641559,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59960,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797640,"statfs":{"total":107374182400,"available":107312783360,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:12:05.447 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.447 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.447 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 9.37 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.448 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 9.33 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.449 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.450 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.450 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.450 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.450 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.450 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.451 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.451 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.451 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.451 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.451 INFO:tasks.ceph:pgid 9.3e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 9.38 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.452 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.453 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.453 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.453 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.453 INFO:tasks.ceph:pgid 13.1d last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.453 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.454 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.455 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.455 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.455 INFO:tasks.ceph:pgid 13.1c last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.455 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.455 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.456 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 14.1c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.457 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.458 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.458 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.458 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.458 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.458 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.459 INFO:tasks.ceph:pgid 14.2 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.460 INFO:tasks.ceph:pgid 13.1 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.460 INFO:tasks.ceph:pgid 12.0 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.460 INFO:tasks.ceph:pgid 11.7 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.460 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.460 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.461 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 13.0 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 12.1 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.462 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.463 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.463 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.463 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.463 INFO:tasks.ceph:pgid 14.0 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.463 INFO:tasks.ceph:pgid 13.3 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 12.2 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.464 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 14.1 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 13.2 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.465 INFO:tasks.ceph:pgid 11.4 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.466 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.466 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.466 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.466 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.466 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.467 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.467 INFO:tasks.ceph:pgid 14.6 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.467 INFO:tasks.ceph:pgid 13.5 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.467 INFO:tasks.ceph:pgid 12.4 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.467 INFO:tasks.ceph:pgid 11.3 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.468 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.469 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.469 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.469 INFO:tasks.ceph:pgid 13.4 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.469 INFO:tasks.ceph:pgid 12.5 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.469 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.470 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.470 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.470 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.470 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.470 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.471 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.471 INFO:tasks.ceph:pgid 14.4 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.471 INFO:tasks.ceph:pgid 13.7 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.471 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.471 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.472 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.472 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.472 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.472 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.472 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.473 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.473 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.473 INFO:tasks.ceph:pgid 14.5 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.473 INFO:tasks.ceph:pgid 13.6 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.473 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.474 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.474 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.474 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.474 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.474 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.475 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.475 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.475 INFO:tasks.ceph:pgid 14.f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.475 INFO:tasks.ceph:pgid 13.c last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.475 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.476 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.476 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.476 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T19:23:45.108082+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.476 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.476 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.477 INFO:tasks.ceph:pgid 14.c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.478 INFO:tasks.ceph:pgid 13.f last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.478 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.478 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.478 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.478 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.479 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.479 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.479 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.479 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.479 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 13.e last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.480 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.481 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.481 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.481 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.481 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.481 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.482 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.482 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.482 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.482 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.482 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.483 INFO:tasks.ceph:pgid 13.b last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.484 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.484 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.484 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.484 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.484 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.485 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.485 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.485 INFO:tasks.ceph:pgid 14.9 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.485 INFO:tasks.ceph:pgid 13.a last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.485 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.486 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.486 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.486 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.486 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.486 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.487 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.487 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.487 INFO:tasks.ceph:pgid 14.a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.487 INFO:tasks.ceph:pgid 13.9 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.487 INFO:tasks.ceph:pgid 11.f last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.488 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.488 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.488 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.488 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.488 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.489 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.489 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.489 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.489 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.489 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.490 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.490 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.490 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.490 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.490 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.491 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.491 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.491 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.491 INFO:tasks.ceph:pgid 13.1a last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.491 INFO:tasks.ceph:pgid 12.1b last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.492 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.492 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.492 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.492 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.492 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.493 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.493 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.493 INFO:tasks.ceph:pgid 14.18 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.493 INFO:tasks.ceph:pgid 12.1a last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.493 INFO:tasks.ceph:pgid 11.1d last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.494 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.494 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.494 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.494 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.494 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 13.18 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 11.1e last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.495 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.496 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.496 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.496 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.496 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.496 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.497 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.497 INFO:tasks.ceph:pgid 14.1a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.497 INFO:tasks.ceph:pgid 13.19 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.497 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.498 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.498 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.498 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.498 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 14.15 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 13.16 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.499 INFO:tasks.ceph:pgid 12.17 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.500 INFO:tasks.ceph:pgid 11.10 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.500 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.500 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.500 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.500 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 14.14 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 13.17 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.501 INFO:tasks.ceph:pgid 11.11 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.502 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.502 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.502 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.502 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.502 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.503 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.503 INFO:tasks.ceph:pgid 14.17 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.503 INFO:tasks.ceph:pgid 13.14 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.503 INFO:tasks.ceph:pgid 12.15 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.503 INFO:tasks.ceph:pgid 11.12 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.504 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.504 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.504 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.504 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.504 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.505 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.505 INFO:tasks.ceph:pgid 14.16 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.505 INFO:tasks.ceph:pgid 13.15 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.505 INFO:tasks.ceph:pgid 12.14 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.505 INFO:tasks.ceph:pgid 11.13 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.506 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.506 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.506 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.506 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.506 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 14.11 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 13.12 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.507 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.508 INFO:tasks.ceph:pgid 9.16 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.508 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.508 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.508 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.508 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.509 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.509 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.509 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.509 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.509 INFO:tasks.ceph:pgid 13.13 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.510 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.510 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.510 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.510 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.510 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.511 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.511 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.511 INFO:tasks.ceph:pgid 14.13 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.511 INFO:tasks.ceph:pgid 13.10 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.511 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.512 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.512 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.512 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.512 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.512 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 14.12 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 13.11 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.513 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.514 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.514 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.514 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.514 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.514 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-16T19:24:50.420558+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=50, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.515 INFO:tasks.ceph:pgid 9.2a last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.515 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.515 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.515 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.515 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.516 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.516 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.516 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.516 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.516 INFO:tasks.ceph:pgid 9.2e last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.517 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.517 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.517 INFO:tasks.ceph:pgid 9.2f last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.517 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.517 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.518 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.518 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.518 INFO:tasks.ceph:pgid 9.2d last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.518 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.518 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.519 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.519 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.519 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.519 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.519 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.520 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.520 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.520 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.520 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.520 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.521 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.521 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.521 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.521 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.521 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.522 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.522 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.522 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.522 INFO:tasks.ceph:pgid 9.3b last_scrub_stamp 2024-04-16T19:25:08.438975+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=8, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.522 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.523 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:05.523 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T21:12:25.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:12:25.880 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:12:25.898 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:12:25.901 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3306,"stamp":"2024-04-16T21:12:25.350615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":280847,"num_omap_keys":1280,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31942,"ondisk_log_size":31942,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":330048,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":318838,"kb_avail":628815552,"statfs":{"total":644245094400,"available":643907125248,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":326490785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":72802,"num_omap_keys":312,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005456"},"pg_stats":[{"pgid":"9.35","version":"52'41","reported_seq":316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.577077+0000","last_change":"2024-04-16T21:11:52.577152+0000","last_active":"2024-04-16T21:11:52.577077+0000","last_peered":"2024-04-16T21:11:52.577077+0000","last_clean":"2024-04-16T21:11:52.577077+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:11:52.577077+0000","last_undegraded":"2024-04-16T21:11:52.577077+0000","last_fullsized":"2024-04-16T21:11:52.577077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'41","last_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_deep_scrub":"52'41","last_deep_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.577012+0000","objects_scrubbed":13,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:52:59.327819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.386010+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:10:49.386010+0000","last_peered":"2024-04-16T21:10:49.386010+0000","last_clean":"2024-04-16T21:10:49.386010+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:10:49.386010+0000","last_undegraded":"2024-04-16T21:10:49.386010+0000","last_fullsized":"2024-04-16T21:10:49.386010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560425+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:11:44.560425+0000","last_peered":"2024-04-16T21:11:44.560425+0000","last_clean":"2024-04-16T21:11:44.560425+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:11:44.560425+0000","last_undegraded":"2024-04-16T21:11:44.560425+0000","last_fullsized":"2024-04-16T21:11:44.560425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"52'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.722895+0000","last_change":"2024-04-16T21:11:56.722928+0000","last_active":"2024-04-16T21:11:56.722895+0000","last_peered":"2024-04-16T21:11:56.722895+0000","last_clean":"2024-04-16T21:11:56.722895+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:11:56.722895+0000","last_undegraded":"2024-04-16T21:11:56.722895+0000","last_fullsized":"2024-04-16T21:11:56.722895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.722838+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:40.375473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384626+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:10:49.384626+0000","last_peered":"2024-04-16T21:10:49.384626+0000","last_clean":"2024-04-16T21:10:49.384626+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:10:49.384626+0000","last_undegraded":"2024-04-16T21:10:49.384626+0000","last_fullsized":"2024-04-16T21:10:49.384626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650348+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:11:43.650348+0000","last_peered":"2024-04-16T21:11:43.650348+0000","last_clean":"2024-04-16T21:11:43.650348+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:11:43.650348+0000","last_undegraded":"2024-04-16T21:11:43.650348+0000","last_fullsized":"2024-04-16T21:11:43.650348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'22","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.702468+0000","last_change":"2024-04-16T21:12:14.702518+0000","last_active":"2024-04-16T21:12:14.702468+0000","last_peered":"2024-04-16T21:12:14.702468+0000","last_clean":"2024-04-16T21:12:14.702468+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:12:14.702468+0000","last_undegraded":"2024-04-16T21:12:14.702468+0000","last_fullsized":"2024-04-16T21:12:14.702468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.702406+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:42.760555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390655+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:10:49.390655+0000","last_peered":"2024-04-16T21:10:49.390655+0000","last_clean":"2024-04-16T21:10:49.390655+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:10:49.390655+0000","last_undegraded":"2024-04-16T21:10:49.390655+0000","last_fullsized":"2024-04-16T21:10:49.390655+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106519+0000","last_change":"2024-04-16T19:24:35.400648+0000","last_active":"2024-04-16T21:11:44.106519+0000","last_peered":"2024-04-16T21:11:44.106519+0000","last_clean":"2024-04-16T21:11:44.106519+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:11:44.106519+0000","last_undegraded":"2024-04-16T21:11:44.106519+0000","last_fullsized":"2024-04-16T21:11:44.106519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'22","reported_seq":252,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.610592+0000","last_change":"2024-04-16T21:11:47.610645+0000","last_active":"2024-04-16T21:11:47.610592+0000","last_peered":"2024-04-16T21:11:47.610592+0000","last_clean":"2024-04-16T21:11:47.610592+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:11:47.610592+0000","last_undegraded":"2024-04-16T21:11:47.610592+0000","last_fullsized":"2024-04-16T21:11:47.610592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.610532+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:13.030892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386247+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:10:54.386247+0000","last_peered":"2024-04-16T21:10:54.386247+0000","last_clean":"2024-04-16T21:10:54.386247+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:10:54.386247+0000","last_undegraded":"2024-04-16T21:10:54.386247+0000","last_fullsized":"2024-04-16T21:10:54.386247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292197+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:11:42.292197+0000","last_peered":"2024-04-16T21:11:42.292197+0000","last_clean":"2024-04-16T21:11:42.292197+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:11:42.292197+0000","last_undegraded":"2024-04-16T21:11:42.292197+0000","last_fullsized":"2024-04-16T21:11:42.292197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'12","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.719672+0000","last_change":"2024-04-16T21:11:58.719715+0000","last_active":"2024-04-16T21:11:58.719672+0000","last_peered":"2024-04-16T21:11:58.719672+0000","last_clean":"2024-04-16T21:11:58.719672+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:11:58.719672+0000","last_undegraded":"2024-04-16T21:11:58.719672+0000","last_fullsized":"2024-04-16T21:11:58.719672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.719610+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:50.501313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650162+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:11:43.650162+0000","last_peered":"2024-04-16T21:11:43.650162+0000","last_clean":"2024-04-16T21:11:43.650162+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:11:43.650162+0000","last_undegraded":"2024-04-16T21:11:43.650162+0000","last_fullsized":"2024-04-16T21:11:43.650162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387029+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:10:59.387029+0000","last_peered":"2024-04-16T21:10:59.387029+0000","last_clean":"2024-04-16T21:10:59.387029+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:10:59.387029+0000","last_undegraded":"2024-04-16T21:10:59.387029+0000","last_fullsized":"2024-04-16T21:10:59.387029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'27","reported_seq":223,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.166051+0000","last_change":"2024-04-16T21:11:51.166108+0000","last_active":"2024-04-16T21:11:51.166051+0000","last_peered":"2024-04-16T21:11:51.166051+0000","last_clean":"2024-04-16T21:11:51.166051+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:11:51.166051+0000","last_undegraded":"2024-04-16T21:11:51.166051+0000","last_fullsized":"2024-04-16T21:11:51.166051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.165987+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:01.812779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560300+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:11:44.560300+0000","last_peered":"2024-04-16T21:11:44.560300+0000","last_clean":"2024-04-16T21:11:44.560300+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:11:44.560300+0000","last_undegraded":"2024-04-16T21:11:44.560300+0000","last_fullsized":"2024-04-16T21:11:44.560300+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386392+0000","last_change":"2024-04-16T19:24:35.400488+0000","last_active":"2024-04-16T21:10:54.386392+0000","last_peered":"2024-04-16T21:10:54.386392+0000","last_clean":"2024-04-16T21:10:54.386392+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:10:54.386392+0000","last_undegraded":"2024-04-16T21:10:54.386392+0000","last_fullsized":"2024-04-16T21:10:54.386392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'18","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.672517+0000","last_change":"2024-04-16T21:12:09.672592+0000","last_active":"2024-04-16T21:12:09.672517+0000","last_peered":"2024-04-16T21:12:09.672517+0000","last_clean":"2024-04-16T21:12:09.672517+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:12:09.672517+0000","last_undegraded":"2024-04-16T21:12:09.672517+0000","last_fullsized":"2024-04-16T21:12:09.672517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.672453+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.111035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650158+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:11:43.650158+0000","last_peered":"2024-04-16T21:11:43.650158+0000","last_clean":"2024-04-16T21:11:43.650158+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:11:43.650158+0000","last_undegraded":"2024-04-16T21:11:43.650158+0000","last_fullsized":"2024-04-16T21:11:43.650158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386334+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:10:54.386334+0000","last_peered":"2024-04-16T21:10:54.386334+0000","last_clean":"2024-04-16T21:10:54.386334+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:10:54.386334+0000","last_undegraded":"2024-04-16T21:10:54.386334+0000","last_fullsized":"2024-04-16T21:10:54.386334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'22","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.654742+0000","last_change":"2024-04-16T21:11:56.654830+0000","last_active":"2024-04-16T21:11:56.654742+0000","last_peered":"2024-04-16T21:11:56.654742+0000","last_clean":"2024-04-16T21:11:56.654742+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:11:56.654742+0000","last_undegraded":"2024-04-16T21:11:56.654742+0000","last_fullsized":"2024-04-16T21:11:56.654742+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.654666+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.077668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194861+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:11:43.194861+0000","last_peered":"2024-04-16T21:11:43.194861+0000","last_clean":"2024-04-16T21:11:43.194861+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:11:43.194861+0000","last_undegraded":"2024-04-16T21:11:43.194861+0000","last_fullsized":"2024-04-16T21:11:43.194861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525521+0000","last_change":"2024-04-16T19:24:35.391191+0000","last_active":"2024-04-16T21:10:54.525521+0000","last_peered":"2024-04-16T21:10:54.525521+0000","last_clean":"2024-04-16T21:10:54.525521+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:10:54.525521+0000","last_undegraded":"2024-04-16T21:10:54.525521+0000","last_fullsized":"2024-04-16T21:10:54.525521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'12","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.701548+0000","last_change":"2024-04-16T21:12:00.701594+0000","last_active":"2024-04-16T21:12:00.701548+0000","last_peered":"2024-04-16T21:12:00.701548+0000","last_clean":"2024-04-16T21:12:00.701548+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:12:00.701548+0000","last_undegraded":"2024-04-16T21:12:00.701548+0000","last_fullsized":"2024-04-16T21:12:00.701548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.701488+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:22.203073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106424+0000","last_change":"2024-04-16T19:24:35.400332+0000","last_active":"2024-04-16T21:11:44.106424+0000","last_peered":"2024-04-16T21:11:44.106424+0000","last_clean":"2024-04-16T21:11:44.106424+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:11:44.106424+0000","last_undegraded":"2024-04-16T21:11:44.106424+0000","last_fullsized":"2024-04-16T21:11:44.106424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390309+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:11:14.390309+0000","last_peered":"2024-04-16T21:11:14.390309+0000","last_clean":"2024-04-16T21:11:14.390309+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:11:14.390309+0000","last_undegraded":"2024-04-16T21:11:14.390309+0000","last_fullsized":"2024-04-16T21:11:14.390309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'12","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.436184+0000","last_change":"2024-04-16T21:11:55.436228+0000","last_active":"2024-04-16T21:11:55.436184+0000","last_peered":"2024-04-16T21:11:55.436184+0000","last_clean":"2024-04-16T21:11:55.436184+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:11:55.436184+0000","last_undegraded":"2024-04-16T21:11:55.436184+0000","last_fullsized":"2024-04-16T21:11:55.436184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.436126+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:58.763911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.914098+0000","last_change":"2024-04-16T21:12:16.914127+0000","last_active":"2024-04-16T21:12:16.914098+0000","last_peered":"2024-04-16T21:12:16.914098+0000","last_clean":"2024-04-16T21:12:16.914098+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:12:16.914098+0000","last_undegraded":"2024-04-16T21:12:16.914098+0000","last_fullsized":"2024-04-16T21:12:16.914098+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.914034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:05.877583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386917+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:10:54.386917+0000","last_peered":"2024-04-16T21:10:54.386917+0000","last_clean":"2024-04-16T21:10:54.386917+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:10:54.386917+0000","last_undegraded":"2024-04-16T21:10:54.386917+0000","last_fullsized":"2024-04-16T21:10:54.386917+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'10","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.882687+0000","last_change":"2024-04-16T21:11:51.882767+0000","last_active":"2024-04-16T21:11:51.882687+0000","last_peered":"2024-04-16T21:11:51.882687+0000","last_clean":"2024-04-16T21:11:51.882687+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:11:51.882687+0000","last_undegraded":"2024-04-16T21:11:51.882687+0000","last_fullsized":"2024-04-16T21:11:51.882687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.882609+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.077377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560427+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:11:44.560427+0000","last_peered":"2024-04-16T21:11:44.560427+0000","last_clean":"2024-04-16T21:11:44.560427+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:11:44.560427+0000","last_undegraded":"2024-04-16T21:11:44.560427+0000","last_fullsized":"2024-04-16T21:11:44.560427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386292+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:10:54.386292+0000","last_peered":"2024-04-16T21:10:54.386292+0000","last_clean":"2024-04-16T21:10:54.386292+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:10:54.386292+0000","last_undegraded":"2024-04-16T21:10:54.386292+0000","last_fullsized":"2024-04-16T21:10:54.386292+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'20","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.881532+0000","last_change":"2024-04-16T21:12:07.881609+0000","last_active":"2024-04-16T21:12:07.881532+0000","last_peered":"2024-04-16T21:12:07.881532+0000","last_clean":"2024-04-16T21:12:07.881532+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:12:07.881532+0000","last_undegraded":"2024-04-16T21:12:07.881532+0000","last_fullsized":"2024-04-16T21:12:07.881532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.881467+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.576312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526272+0000","last_change":"2024-04-16T19:24:35.394288+0000","last_active":"2024-04-16T21:10:59.526272+0000","last_peered":"2024-04-16T21:10:59.526272+0000","last_clean":"2024-04-16T21:10:59.526272+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:10:59.526272+0000","last_undegraded":"2024-04-16T21:10:59.526272+0000","last_fullsized":"2024-04-16T21:10:59.526272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386361+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:10:54.386361+0000","last_peered":"2024-04-16T21:10:54.386361+0000","last_clean":"2024-04-16T21:10:54.386361+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:10:54.386361+0000","last_undegraded":"2024-04-16T21:10:54.386361+0000","last_fullsized":"2024-04-16T21:10:54.386361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'16","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.428126+0000","last_change":"2024-04-16T21:11:47.428185+0000","last_active":"2024-04-16T21:11:47.428126+0000","last_peered":"2024-04-16T21:11:47.428126+0000","last_clean":"2024-04-16T21:11:47.428126+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:11:47.428126+0000","last_undegraded":"2024-04-16T21:11:47.428126+0000","last_fullsized":"2024-04-16T21:11:47.428126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.428059+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:23.278965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526292+0000","last_change":"2024-04-16T19:24:35.393490+0000","last_active":"2024-04-16T21:10:59.526292+0000","last_peered":"2024-04-16T21:10:59.526292+0000","last_clean":"2024-04-16T21:10:59.526292+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:10:59.526292+0000","last_undegraded":"2024-04-16T21:10:59.526292+0000","last_fullsized":"2024-04-16T21:10:59.526292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391334+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:10:54.391334+0000","last_peered":"2024-04-16T21:10:54.391334+0000","last_clean":"2024-04-16T21:10:54.391334+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:10:54.391334+0000","last_undegraded":"2024-04-16T21:10:54.391334+0000","last_fullsized":"2024-04-16T21:10:54.391334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'22","reported_seq":204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.947898+0000","last_change":"2024-04-16T21:12:11.947975+0000","last_active":"2024-04-16T21:12:11.947898+0000","last_peered":"2024-04-16T21:12:11.947898+0000","last_clean":"2024-04-16T21:12:11.947898+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:12:11.947898+0000","last_undegraded":"2024-04-16T21:12:11.947898+0000","last_fullsized":"2024-04-16T21:12:11.947898+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.947824+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:04.586823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387165+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:10:59.387165+0000","last_peered":"2024-04-16T21:10:59.387165+0000","last_clean":"2024-04-16T21:10:59.387165+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:10:59.387165+0000","last_undegraded":"2024-04-16T21:10:59.387165+0000","last_fullsized":"2024-04-16T21:10:59.387165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292016+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:11:42.292016+0000","last_peered":"2024-04-16T21:11:42.292016+0000","last_clean":"2024-04-16T21:11:42.292016+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:11:42.292016+0000","last_undegraded":"2024-04-16T21:11:42.292016+0000","last_fullsized":"2024-04-16T21:11:42.292016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385608+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:10:54.385608+0000","last_peered":"2024-04-16T21:10:54.385608+0000","last_clean":"2024-04-16T21:10:54.385608+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:10:54.385608+0000","last_undegraded":"2024-04-16T21:10:54.385608+0000","last_fullsized":"2024-04-16T21:10:54.385608+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1722,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651474+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:11:43.651474+0000","last_peered":"2024-04-16T21:11:43.651474+0000","last_clean":"2024-04-16T21:11:43.651474+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:11:43.651474+0000","last_undegraded":"2024-04-16T21:11:43.651474+0000","last_fullsized":"2024-04-16T21:11:43.651474+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.395179+0000","last_change":"2024-04-16T21:12:15.395220+0000","last_active":"2024-04-16T21:12:15.395179+0000","last_peered":"2024-04-16T21:12:15.395179+0000","last_clean":"2024-04-16T21:12:15.395179+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:12:15.395179+0000","last_undegraded":"2024-04-16T21:12:15.395179+0000","last_fullsized":"2024-04-16T21:12:15.395179+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.395104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:09.823839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'8","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.430759+0000","last_change":"2024-04-16T21:12:17.430800+0000","last_active":"2024-04-16T21:12:17.430759+0000","last_peered":"2024-04-16T21:12:17.430759+0000","last_clean":"2024-04-16T21:12:17.430759+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:12:17.430759+0000","last_undegraded":"2024-04-16T21:12:17.430759+0000","last_fullsized":"2024-04-16T21:12:17.430759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.430693+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:25.924925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.738443+0000","last_change":"2024-04-16T21:11:45.738465+0000","last_active":"2024-04-16T21:11:45.738443+0000","last_peered":"2024-04-16T21:11:45.738443+0000","last_clean":"2024-04-16T21:11:45.738443+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:11:45.738443+0000","last_undegraded":"2024-04-16T21:11:45.738443+0000","last_fullsized":"2024-04-16T21:11:45.738443+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.738344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:26.908264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'223","reported_seq":476,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.490112+0000","last_change":"2024-04-16T21:11:58.490168+0000","last_active":"2024-04-16T21:11:58.490112+0000","last_peered":"2024-04-16T21:11:58.490112+0000","last_clean":"2024-04-16T21:11:58.490112+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:11:58.490112+0000","last_undegraded":"2024-04-16T21:11:58.490112+0000","last_fullsized":"2024-04-16T21:11:58.490112+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'223","last_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_deep_scrub":"52'223","last_deep_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.490053+0000","objects_scrubbed":20,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:48.708953+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1971,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'29","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.926237+0000","last_change":"2024-04-16T21:11:50.926284+0000","last_active":"2024-04-16T21:11:50.926237+0000","last_peered":"2024-04-16T21:11:50.926237+0000","last_clean":"2024-04-16T21:11:50.926237+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:11:50.926237+0000","last_undegraded":"2024-04-16T21:11:50.926237+0000","last_fullsized":"2024-04-16T21:11:50.926237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'29","last_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_deep_scrub":"52'29","last_deep_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.926173+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:17.509632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.689691+0000","last_change":"2024-04-16T21:12:19.689715+0000","last_active":"2024-04-16T21:12:19.689691+0000","last_peered":"2024-04-16T21:12:19.689691+0000","last_clean":"2024-04-16T21:12:19.689691+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:12:19.689691+0000","last_undegraded":"2024-04-16T21:12:19.689691+0000","last_fullsized":"2024-04-16T21:12:19.689691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.689621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.705707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743114+0000","last_change":"2024-04-16T19:24:35.400133+0000","last_active":"2024-04-16T21:11:42.743114+0000","last_peered":"2024-04-16T21:11:42.743114+0000","last_clean":"2024-04-16T21:11:42.743114+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:11:42.743114+0000","last_undegraded":"2024-04-16T21:11:42.743114+0000","last_fullsized":"2024-04-16T21:11:42.743114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649927+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:11:43.649927+0000","last_peered":"2024-04-16T21:11:43.649927+0000","last_clean":"2024-04-16T21:11:43.649927+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:11:43.649927+0000","last_undegraded":"2024-04-16T21:11:43.649927+0000","last_fullsized":"2024-04-16T21:11:43.649927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'5","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388585+0000","last_change":"2024-04-16T19:24:45.414840+0000","last_active":"2024-04-16T21:11:09.388585+0000","last_peered":"2024-04-16T21:11:09.388585+0000","last_clean":"2024-04-16T21:11:09.388585+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:11:09.388585+0000","last_undegraded":"2024-04-16T21:11:09.388585+0000","last_fullsized":"2024-04-16T21:11:09.388585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.394046+0000","last_change":"2024-04-16T19:24:47.423393+0000","last_active":"2024-04-16T21:11:09.394046+0000","last_peered":"2024-04-16T21:11:09.394046+0000","last_clean":"2024-04-16T21:11:09.394046+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:11:09.394046+0000","last_undegraded":"2024-04-16T21:11:09.394046+0000","last_fullsized":"2024-04-16T21:11:09.394046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388641+0000","last_change":"2024-04-16T19:24:49.432621+0000","last_active":"2024-04-16T21:11:09.388641+0000","last_peered":"2024-04-16T21:11:09.388641+0000","last_clean":"2024-04-16T21:11:09.388641+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:11:09.388641+0000","last_undegraded":"2024-04-16T21:11:09.388641+0000","last_fullsized":"2024-04-16T21:11:09.388641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.196037+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:11:43.196037+0000","last_peered":"2024-04-16T21:11:43.196037+0000","last_clean":"2024-04-16T21:11:43.196037+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:11:43.196037+0000","last_undegraded":"2024-04-16T21:11:43.196037+0000","last_fullsized":"2024-04-16T21:11:43.196037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1837,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386348+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:10:54.386348+0000","last_peered":"2024-04-16T21:10:54.386348+0000","last_clean":"2024-04-16T21:10:54.386348+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:10:54.386348+0000","last_undegraded":"2024-04-16T21:10:54.386348+0000","last_fullsized":"2024-04-16T21:10:54.386348+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.873357+0000","last_change":"2024-04-16T21:12:15.873380+0000","last_active":"2024-04-16T21:12:15.873357+0000","last_peered":"2024-04-16T21:12:15.873357+0000","last_clean":"2024-04-16T21:12:15.873357+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:12:15.873357+0000","last_undegraded":"2024-04-16T21:12:15.873357+0000","last_fullsized":"2024-04-16T21:12:15.873357+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.873260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:01.509024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'8","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.369110+0000","last_change":"2024-04-16T21:12:14.369164+0000","last_active":"2024-04-16T21:12:14.369110+0000","last_peered":"2024-04-16T21:12:14.369110+0000","last_clean":"2024-04-16T21:12:14.369110+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:12:14.369110+0000","last_undegraded":"2024-04-16T21:12:14.369110+0000","last_fullsized":"2024-04-16T21:12:14.369110+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.369033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:29.838406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.686963+0000","last_change":"2024-04-16T21:11:57.686990+0000","last_active":"2024-04-16T21:11:57.686963+0000","last_peered":"2024-04-16T21:11:57.686963+0000","last_clean":"2024-04-16T21:11:57.686963+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:11:57.686963+0000","last_undegraded":"2024-04-16T21:11:57.686963+0000","last_fullsized":"2024-04-16T21:11:57.686963+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.686888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:39.802344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'264","reported_seq":1008,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.984736+0000","last_change":"2024-04-16T21:11:43.984786+0000","last_active":"2024-04-16T21:11:43.984736+0000","last_peered":"2024-04-16T21:11:43.984736+0000","last_clean":"2024-04-16T21:11:43.984736+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:11:43.984736+0000","last_undegraded":"2024-04-16T21:11:43.984736+0000","last_fullsized":"2024-04-16T21:11:43.984736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.984688+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:21.448843+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6375,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'10","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.702465+0000","last_change":"2024-04-16T21:11:54.702515+0000","last_active":"2024-04-16T21:11:54.702465+0000","last_peered":"2024-04-16T21:11:54.702465+0000","last_clean":"2024-04-16T21:11:54.702465+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:11:54.702465+0000","last_undegraded":"2024-04-16T21:11:54.702465+0000","last_fullsized":"2024-04-16T21:11:54.702465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.702406+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.795058+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.815226+0000","last_change":"2024-04-16T21:12:10.815250+0000","last_active":"2024-04-16T21:12:10.815226+0000","last_peered":"2024-04-16T21:12:10.815226+0000","last_clean":"2024-04-16T21:12:10.815226+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:12:10.815226+0000","last_undegraded":"2024-04-16T21:12:10.815226+0000","last_fullsized":"2024-04-16T21:12:10.815226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.815114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:27:54.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387060+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:10:59.387060+0000","last_peered":"2024-04-16T21:10:59.387060+0000","last_clean":"2024-04-16T21:10:59.387060+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:10:59.387060+0000","last_undegraded":"2024-04-16T21:10:59.387060+0000","last_fullsized":"2024-04-16T21:10:59.387060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291858+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:11:42.291858+0000","last_peered":"2024-04-16T21:11:42.291858+0000","last_clean":"2024-04-16T21:11:42.291858+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:11:42.291858+0000","last_undegraded":"2024-04-16T21:11:42.291858+0000","last_fullsized":"2024-04-16T21:11:42.291858+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.384073+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:10:09.384073+0000","last_peered":"2024-04-16T21:10:09.384073+0000","last_clean":"2024-04-16T21:10:09.384073+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:10:09.384073+0000","last_undegraded":"2024-04-16T21:10:09.384073+0000","last_fullsized":"2024-04-16T21:10:09.384073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'66","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388791+0000","last_change":"2024-04-16T19:24:47.422158+0000","last_active":"2024-04-16T21:11:09.388791+0000","last_peered":"2024-04-16T21:11:09.388791+0000","last_clean":"2024-04-16T21:11:09.388791+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:11:09.388791+0000","last_undegraded":"2024-04-16T21:11:09.388791+0000","last_fullsized":"2024-04-16T21:11:09.388791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.604337+0000","last_change":"2024-04-16T21:12:23.604359+0000","last_active":"2024-04-16T21:12:23.604337+0000","last_peered":"2024-04-16T21:12:23.604337+0000","last_clean":"2024-04-16T21:12:23.604337+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:12:23.604337+0000","last_undegraded":"2024-04-16T21:12:23.604337+0000","last_fullsized":"2024-04-16T21:12:23.604337+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.604274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:10.724532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377974+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:10:04.377974+0000","last_peered":"2024-04-16T21:10:04.377974+0000","last_clean":"2024-04-16T21:10:04.377974+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:10:04.377974+0000","last_undegraded":"2024-04-16T21:10:04.377974+0000","last_fullsized":"2024-04-16T21:10:04.377974+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'131","reported_seq":1872,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390599+0000","last_change":"2024-04-16T19:25:49.518182+0000","last_active":"2024-04-16T21:10:49.390599+0000","last_peered":"2024-04-16T21:10:49.390599+0000","last_clean":"2024-04-16T21:10:49.390599+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:10:49.390599+0000","last_undegraded":"2024-04-16T21:10:49.390599+0000","last_fullsized":"2024-04-16T21:10:49.390599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.746629+0000","last_change":"2024-04-16T21:12:00.746651+0000","last_active":"2024-04-16T21:12:00.746629+0000","last_peered":"2024-04-16T21:12:00.746629+0000","last_clean":"2024-04-16T21:12:00.746629+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:12:00.746629+0000","last_undegraded":"2024-04-16T21:12:00.746629+0000","last_fullsized":"2024-04-16T21:12:00.746629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.746571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:24:45.291359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'25","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.693813+0000","last_change":"2024-04-16T21:11:57.693880+0000","last_active":"2024-04-16T21:11:57.693813+0000","last_peered":"2024-04-16T21:11:57.693813+0000","last_clean":"2024-04-16T21:11:57.693813+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:11:57.693813+0000","last_undegraded":"2024-04-16T21:11:57.693813+0000","last_fullsized":"2024-04-16T21:11:57.693813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'25","last_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_deep_scrub":"52'25","last_deep_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.693743+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.739736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.385829+0000","last_change":"2024-04-16T21:12:04.385854+0000","last_active":"2024-04-16T21:12:04.385829+0000","last_peered":"2024-04-16T21:12:04.385829+0000","last_clean":"2024-04-16T21:12:04.385829+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:12:04.385829+0000","last_undegraded":"2024-04-16T21:12:04.385829+0000","last_fullsized":"2024-04-16T21:12:04.385829+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.385746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:20.771679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'716","reported_seq":41938,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.996557+0000","last_change":"2024-04-16T21:11:45.996631+0000","last_active":"2024-04-16T21:11:45.996557+0000","last_peered":"2024-04-16T21:11:45.996557+0000","last_clean":"2024-04-16T21:11:45.996557+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:11:45.996557+0000","last_undegraded":"2024-04-16T21:11:45.996557+0000","last_fullsized":"2024-04-16T21:11:45.996557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'716","last_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_deep_scrub":"52'716","last_deep_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.996464+0000","objects_scrubbed":20,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:40.351620+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'35","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.105334+0000","last_change":"2024-04-16T21:11:44.105372+0000","last_active":"2024-04-16T21:11:44.105334+0000","last_peered":"2024-04-16T21:11:44.105334+0000","last_clean":"2024-04-16T21:11:44.105334+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:11:44.105334+0000","last_undegraded":"2024-04-16T21:11:44.105334+0000","last_fullsized":"2024-04-16T21:11:44.105334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'35","last_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_deep_scrub":"52'35","last_deep_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.105287+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:44.283094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.409254+0000","last_change":"2024-04-16T21:12:03.409254+0000","last_active":"2024-04-16T21:12:03.409254+0000","last_peered":"2024-04-16T21:12:03.409254+0000","last_clean":"2024-04-16T21:12:03.409254+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:12:03.409254+0000","last_undegraded":"2024-04-16T21:12:03.409254+0000","last_fullsized":"2024-04-16T21:12:03.409254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.409137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:49.655116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392318+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:10:59.392318+0000","last_peered":"2024-04-16T21:10:59.392318+0000","last_clean":"2024-04-16T21:10:59.392318+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:10:59.392318+0000","last_undegraded":"2024-04-16T21:10:59.392318+0000","last_fullsized":"2024-04-16T21:10:59.392318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649938+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:11:43.649938+0000","last_peered":"2024-04-16T21:11:43.649938+0000","last_clean":"2024-04-16T21:11:43.649938+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:11:43.649938+0000","last_undegraded":"2024-04-16T21:11:43.649938+0000","last_fullsized":"2024-04-16T21:11:43.649938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'5","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.528197+0000","last_change":"2024-04-16T19:24:45.412108+0000","last_active":"2024-04-16T21:11:09.528197+0000","last_peered":"2024-04-16T21:11:09.528197+0000","last_clean":"2024-04-16T21:11:09.528197+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:11:09.528197+0000","last_undegraded":"2024-04-16T21:11:09.528197+0000","last_fullsized":"2024-04-16T21:11:09.528197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'110","reported_seq":276,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195294+0000","last_change":"2024-04-16T19:24:47.422596+0000","last_active":"2024-04-16T21:11:43.195294+0000","last_peered":"2024-04-16T21:11:43.195294+0000","last_clean":"2024-04-16T21:11:43.195294+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:11:43.195294+0000","last_undegraded":"2024-04-16T21:11:43.195294+0000","last_fullsized":"2024-04-16T21:11:43.195294+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'13","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.370369+0000","last_change":"2024-04-16T21:12:13.370396+0000","last_active":"2024-04-16T21:12:13.370369+0000","last_peered":"2024-04-16T21:12:13.370369+0000","last_clean":"2024-04-16T21:12:13.370369+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:12:13.370369+0000","last_undegraded":"2024-04-16T21:12:13.370369+0000","last_fullsized":"2024-04-16T21:12:13.370369+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.370308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:47.137634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.377342+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:10:04.377342+0000","last_peered":"2024-04-16T21:10:04.377342+0000","last_clean":"2024-04-16T21:10:04.377342+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:10:04.377342+0000","last_undegraded":"2024-04-16T21:10:04.377342+0000","last_fullsized":"2024-04-16T21:10:04.377342+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1473,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390635+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:10:49.390635+0000","last_peered":"2024-04-16T21:10:49.390635+0000","last_clean":"2024-04-16T21:10:49.390635+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:10:49.390635+0000","last_undegraded":"2024-04-16T21:10:49.390635+0000","last_fullsized":"2024-04-16T21:10:49.390635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.703764+0000","last_change":"2024-04-16T21:12:02.703764+0000","last_active":"2024-04-16T21:12:02.703764+0000","last_peered":"2024-04-16T21:12:02.703764+0000","last_clean":"2024-04-16T21:12:02.703764+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:12:02.703764+0000","last_undegraded":"2024-04-16T21:12:02.703764+0000","last_fullsized":"2024-04-16T21:12:02.703764+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.703681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:59.311639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'13","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.372480+0000","last_change":"2024-04-16T21:12:18.372514+0000","last_active":"2024-04-16T21:12:18.372480+0000","last_peered":"2024-04-16T21:12:18.372480+0000","last_clean":"2024-04-16T21:12:18.372480+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:12:18.372480+0000","last_undegraded":"2024-04-16T21:12:18.372480+0000","last_fullsized":"2024-04-16T21:12:18.372480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.372404+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:27.934072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.821906+0000","last_change":"2024-04-16T21:11:59.821929+0000","last_active":"2024-04-16T21:11:59.821906+0000","last_peered":"2024-04-16T21:11:59.821906+0000","last_clean":"2024-04-16T21:11:59.821906+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:11:59.821906+0000","last_undegraded":"2024-04-16T21:11:59.821906+0000","last_fullsized":"2024-04-16T21:11:59.821906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.821840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.493143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'366","reported_seq":690,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.574732+0000","last_change":"2024-04-16T21:11:46.574732+0000","last_active":"2024-04-16T21:11:46.574732+0000","last_peered":"2024-04-16T21:11:46.574732+0000","last_clean":"2024-04-16T21:11:46.574732+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:11:46.574732+0000","last_undegraded":"2024-04-16T21:11:46.574732+0000","last_fullsized":"2024-04-16T21:11:46.574732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'366","last_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_deep_scrub":"52'366","last_deep_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.574641+0000","objects_scrubbed":27,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:24.475352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'12","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.580757+0000","last_change":"2024-04-16T21:11:44.580812+0000","last_active":"2024-04-16T21:11:44.580757+0000","last_peered":"2024-04-16T21:11:44.580757+0000","last_clean":"2024-04-16T21:11:44.580757+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:11:44.580757+0000","last_undegraded":"2024-04-16T21:11:44.580757+0000","last_fullsized":"2024-04-16T21:11:44.580757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.580695+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:57.775463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.671998+0000","last_change":"2024-04-16T21:12:15.671998+0000","last_active":"2024-04-16T21:12:15.671998+0000","last_peered":"2024-04-16T21:12:15.671998+0000","last_clean":"2024-04-16T21:12:15.671998+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:12:15.671998+0000","last_undegraded":"2024-04-16T21:12:15.671998+0000","last_fullsized":"2024-04-16T21:12:15.671998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.671910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:43.983331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392386+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:10:59.392386+0000","last_peered":"2024-04-16T21:10:59.392386+0000","last_clean":"2024-04-16T21:10:59.392386+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:10:59.392386+0000","last_undegraded":"2024-04-16T21:10:59.392386+0000","last_fullsized":"2024-04-16T21:10:59.392386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560171+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:11:44.560171+0000","last_peered":"2024-04-16T21:11:44.560171+0000","last_clean":"2024-04-16T21:11:44.560171+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:11:44.560171+0000","last_undegraded":"2024-04-16T21:11:44.560171+0000","last_fullsized":"2024-04-16T21:11:44.560171+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389520+0000","last_change":"2024-04-16T19:24:45.413851+0000","last_active":"2024-04-16T21:11:09.389520+0000","last_peered":"2024-04-16T21:11:09.389520+0000","last_clean":"2024-04-16T21:11:09.389520+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:11:09.389520+0000","last_undegraded":"2024-04-16T21:11:09.389520+0000","last_fullsized":"2024-04-16T21:11:09.389520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389466+0000","last_change":"2024-04-16T19:24:47.419931+0000","last_active":"2024-04-16T21:11:09.389466+0000","last_peered":"2024-04-16T21:11:09.389466+0000","last_clean":"2024-04-16T21:11:09.389466+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:11:09.389466+0000","last_undegraded":"2024-04-16T21:11:09.389466+0000","last_fullsized":"2024-04-16T21:11:09.389466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'12","reported_seq":1310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:09.378158+0000","last_change":"2024-04-16T19:24:49.429270+0000","last_active":"2024-04-16T21:10:09.378158+0000","last_peered":"2024-04-16T21:10:09.378158+0000","last_clean":"2024-04-16T21:10:09.378158+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:10:09.378158+0000","last_undegraded":"2024-04-16T21:10:09.378158+0000","last_fullsized":"2024-04-16T21:10:09.378158+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386322+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:10:54.386322+0000","last_peered":"2024-04-16T21:10:54.386322+0000","last_clean":"2024-04-16T21:10:54.386322+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:10:54.386322+0000","last_undegraded":"2024-04-16T21:10:54.386322+0000","last_fullsized":"2024-04-16T21:10:54.386322+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'148","reported_seq":2008,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651384+0000","last_change":"2024-04-16T19:25:49.515352+0000","last_active":"2024-04-16T21:11:43.651384+0000","last_peered":"2024-04-16T21:11:43.651384+0000","last_clean":"2024-04-16T21:11:43.651384+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:11:43.651384+0000","last_undegraded":"2024-04-16T21:11:43.651384+0000","last_fullsized":"2024-04-16T21:11:43.651384+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.379178+0000","last_change":"2024-04-16T21:12:16.379217+0000","last_active":"2024-04-16T21:12:16.379178+0000","last_peered":"2024-04-16T21:12:16.379178+0000","last_clean":"2024-04-16T21:12:16.379178+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:12:16.379178+0000","last_undegraded":"2024-04-16T21:12:16.379178+0000","last_fullsized":"2024-04-16T21:12:16.379178+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.379105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:54.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'37","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.663863+0000","last_change":"2024-04-16T21:12:06.663901+0000","last_active":"2024-04-16T21:12:06.663863+0000","last_peered":"2024-04-16T21:12:06.663863+0000","last_clean":"2024-04-16T21:12:06.663863+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:12:06.663863+0000","last_undegraded":"2024-04-16T21:12:06.663863+0000","last_fullsized":"2024-04-16T21:12:06.663863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'37","last_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_deep_scrub":"52'37","last_deep_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.663802+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:20.415399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":158,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.716132+0000","last_change":"2024-04-16T21:12:07.716153+0000","last_active":"2024-04-16T21:12:07.716132+0000","last_peered":"2024-04-16T21:12:07.716132+0000","last_clean":"2024-04-16T21:12:07.716132+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:12:07.716132+0000","last_undegraded":"2024-04-16T21:12:07.716132+0000","last_fullsized":"2024-04-16T21:12:07.716132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.716070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.751360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1762","reported_seq":4632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.707742+0000","last_change":"2024-04-16T21:11:51.707808+0000","last_active":"2024-04-16T21:11:51.707742+0000","last_peered":"2024-04-16T21:11:51.707742+0000","last_clean":"2024-04-16T21:11:51.707742+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:11:51.707742+0000","last_undegraded":"2024-04-16T21:11:51.707742+0000","last_fullsized":"2024-04-16T21:11:51.707742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1762","last_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_deep_scrub":"52'1762","last_deep_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.707684+0000","objects_scrubbed":22,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:25.851168+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":89480,"num_omap_keys":462,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'27","reported_seq":288,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.189093+0000","last_change":"2024-04-16T21:11:48.189147+0000","last_active":"2024-04-16T21:11:48.189093+0000","last_peered":"2024-04-16T21:11:48.189093+0000","last_clean":"2024-04-16T21:11:48.189093+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:11:48.189093+0000","last_undegraded":"2024-04-16T21:11:48.189093+0000","last_fullsized":"2024-04-16T21:11:48.189093+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.189029+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:30.304485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385804+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:10:49.385804+0000","last_peered":"2024-04-16T21:10:49.385804+0000","last_clean":"2024-04-16T21:10:49.385804+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:10:49.385804+0000","last_undegraded":"2024-04-16T21:10:49.385804+0000","last_fullsized":"2024-04-16T21:10:49.385804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.383849+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:10:39.383849+0000","last_peered":"2024-04-16T21:10:39.383849+0000","last_clean":"2024-04-16T21:10:39.383849+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:10:39.383849+0000","last_undegraded":"2024-04-16T21:10:39.383849+0000","last_fullsized":"2024-04-16T21:10:39.383849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.386931+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:10:59.386931+0000","last_peered":"2024-04-16T21:10:59.386931+0000","last_clean":"2024-04-16T21:10:59.386931+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:10:59.386931+0000","last_undegraded":"2024-04-16T21:10:59.386931+0000","last_fullsized":"2024-04-16T21:10:59.386931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'184","reported_seq":395,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.390148+0000","last_change":"2024-04-16T19:24:47.419830+0000","last_active":"2024-04-16T21:11:19.390148+0000","last_peered":"2024-04-16T21:11:19.390148+0000","last_clean":"2024-04-16T21:11:19.390148+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:11:19.390148+0000","last_undegraded":"2024-04-16T21:11:19.390148+0000","last_fullsized":"2024-04-16T21:11:19.390148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.922097+0000","last_change":"2024-04-16T21:12:20.922119+0000","last_active":"2024-04-16T21:12:20.922097+0000","last_peered":"2024-04-16T21:12:20.922097+0000","last_clean":"2024-04-16T21:12:20.922097+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:12:20.922097+0000","last_undegraded":"2024-04-16T21:12:20.922097+0000","last_fullsized":"2024-04-16T21:12:20.922097+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.922033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:09.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.873344+0000","last_change":"2024-04-16T21:12:12.873367+0000","last_active":"2024-04-16T21:12:12.873344+0000","last_peered":"2024-04-16T21:12:12.873344+0000","last_clean":"2024-04-16T21:12:12.873344+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:12:12.873344+0000","last_undegraded":"2024-04-16T21:12:12.873344+0000","last_fullsized":"2024-04-16T21:12:12.873344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.873281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:06.480261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525424+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:10:54.525424+0000","last_peered":"2024-04-16T21:10:54.525424+0000","last_clean":"2024-04-16T21:10:54.525424+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:10:54.525424+0000","last_undegraded":"2024-04-16T21:10:54.525424+0000","last_fullsized":"2024-04-16T21:10:54.525424+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293176+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:11:42.293176+0000","last_peered":"2024-04-16T21:11:42.293176+0000","last_clean":"2024-04-16T21:11:42.293176+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:11:42.293176+0000","last_undegraded":"2024-04-16T21:11:42.293176+0000","last_fullsized":"2024-04-16T21:11:42.293176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.341198+0000","last_change":"2024-04-16T21:12:17.341220+0000","last_active":"2024-04-16T21:12:17.341198+0000","last_peered":"2024-04-16T21:12:17.341198+0000","last_clean":"2024-04-16T21:12:17.341198+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:12:17.341198+0000","last_undegraded":"2024-04-16T21:12:17.341198+0000","last_fullsized":"2024-04-16T21:12:17.341198+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.341136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:45.982431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'15","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.375445+0000","last_change":"2024-04-16T21:12:19.375473+0000","last_active":"2024-04-16T21:12:19.375445+0000","last_peered":"2024-04-16T21:12:19.375445+0000","last_clean":"2024-04-16T21:12:19.375445+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:12:19.375445+0000","last_undegraded":"2024-04-16T21:12:19.375445+0000","last_fullsized":"2024-04-16T21:12:19.375445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.375386+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.049075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.377316+0000","last_change":"2024-04-16T21:12:02.377338+0000","last_active":"2024-04-16T21:12:02.377316+0000","last_peered":"2024-04-16T21:12:02.377316+0000","last_clean":"2024-04-16T21:12:02.377316+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:12:02.377316+0000","last_undegraded":"2024-04-16T21:12:02.377316+0000","last_fullsized":"2024-04-16T21:12:02.377316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.377250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:49.251281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'325","reported_seq":659,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.419682+0000","last_change":"2024-04-16T21:11:56.419734+0000","last_active":"2024-04-16T21:11:56.419682+0000","last_peered":"2024-04-16T21:11:56.419682+0000","last_clean":"2024-04-16T21:11:56.419682+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:11:56.419682+0000","last_undegraded":"2024-04-16T21:11:56.419682+0000","last_fullsized":"2024-04-16T21:11:56.419682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'325","last_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_deep_scrub":"52'325","last_deep_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.419626+0000","objects_scrubbed":21,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:24.477776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6361,"num_omap_keys":29,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'16","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.568997+0000","last_change":"2024-04-16T21:11:45.569097+0000","last_active":"2024-04-16T21:11:45.568997+0000","last_peered":"2024-04-16T21:11:45.568997+0000","last_clean":"2024-04-16T21:11:45.568997+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:11:45.568997+0000","last_undegraded":"2024-04-16T21:11:45.568997+0000","last_fullsized":"2024-04-16T21:11:45.568997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.568923+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:31.758184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385305+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:10:49.385305+0000","last_peered":"2024-04-16T21:10:49.385305+0000","last_clean":"2024-04-16T21:10:49.385305+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:10:49.385305+0000","last_undegraded":"2024-04-16T21:10:49.385305+0000","last_fullsized":"2024-04-16T21:10:49.385305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380988+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:10:19.380988+0000","last_peered":"2024-04-16T21:10:19.380988+0000","last_clean":"2024-04-16T21:10:19.380988+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:10:19.380988+0000","last_undegraded":"2024-04-16T21:10:19.380988+0000","last_fullsized":"2024-04-16T21:10:19.380988+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"39'4","reported_seq":960,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526178+0000","last_change":"2024-04-16T19:24:45.410026+0000","last_active":"2024-04-16T21:10:59.526178+0000","last_peered":"2024-04-16T21:10:59.526178+0000","last_clean":"2024-04-16T21:10:59.526178+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:10:59.526178+0000","last_undegraded":"2024-04-16T21:10:59.526178+0000","last_fullsized":"2024-04-16T21:10:59.526178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'74","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382034+0000","last_change":"2024-04-16T19:24:47.423913+0000","last_active":"2024-04-16T21:10:34.382034+0000","last_peered":"2024-04-16T21:10:34.382034+0000","last_clean":"2024-04-16T21:10:34.382034+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:10:34.382034+0000","last_undegraded":"2024-04-16T21:10:34.382034+0000","last_fullsized":"2024-04-16T21:10:34.382034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.929170+0000","last_change":"2024-04-16T21:12:21.929194+0000","last_active":"2024-04-16T21:12:21.929170+0000","last_peered":"2024-04-16T21:12:21.929170+0000","last_clean":"2024-04-16T21:12:21.929170+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:12:21.929170+0000","last_undegraded":"2024-04-16T21:12:21.929170+0000","last_fullsized":"2024-04-16T21:12:21.929170+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.929105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.364555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.721391+0000","last_change":"2024-04-16T21:12:18.721414+0000","last_active":"2024-04-16T21:12:18.721391+0000","last_peered":"2024-04-16T21:12:18.721391+0000","last_clean":"2024-04-16T21:12:18.721391+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:12:18.721391+0000","last_undegraded":"2024-04-16T21:12:18.721391+0000","last_fullsized":"2024-04-16T21:12:18.721391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.721330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:06.756619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386715+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:10:54.386715+0000","last_peered":"2024-04-16T21:10:54.386715+0000","last_clean":"2024-04-16T21:10:54.386715+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:10:54.386715+0000","last_undegraded":"2024-04-16T21:10:54.386715+0000","last_fullsized":"2024-04-16T21:10:54.386715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'135","reported_seq":1431,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.902487+0000","last_change":"2024-04-16T21:12:20.902537+0000","last_active":"2024-04-16T21:12:20.902487+0000","last_peered":"2024-04-16T21:12:20.902487+0000","last_clean":"2024-04-16T21:12:20.902487+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:12:20.902487+0000","last_undegraded":"2024-04-16T21:12:20.902487+0000","last_fullsized":"2024-04-16T21:12:20.902487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'135","last_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_deep_scrub":"52'135","last_deep_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.902428+0000","objects_scrubbed":42,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:27:02.685242+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3879,"num_omap_keys":15,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.456127+0000","last_change":"2024-04-16T21:12:14.456172+0000","last_active":"2024-04-16T21:12:14.456127+0000","last_peered":"2024-04-16T21:12:14.456127+0000","last_clean":"2024-04-16T21:12:14.456127+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:12:14.456127+0000","last_undegraded":"2024-04-16T21:12:14.456127+0000","last_fullsized":"2024-04-16T21:12:14.456127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:43.097345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'24","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.378145+0000","last_change":"2024-04-16T21:12:23.378174+0000","last_active":"2024-04-16T21:12:23.378145+0000","last_peered":"2024-04-16T21:12:23.378145+0000","last_clean":"2024-04-16T21:12:23.378145+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:12:23.378145+0000","last_undegraded":"2024-04-16T21:12:23.378145+0000","last_fullsized":"2024-04-16T21:12:23.378145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.378087+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:34.980757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.647510+0000","last_change":"2024-04-16T21:11:51.647550+0000","last_active":"2024-04-16T21:11:51.647510+0000","last_peered":"2024-04-16T21:11:51.647510+0000","last_clean":"2024-04-16T21:11:51.647510+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:11:51.647510+0000","last_undegraded":"2024-04-16T21:11:51.647510+0000","last_fullsized":"2024-04-16T21:11:51.647510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.647441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:25.536915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2068","reported_seq":5847,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.355619+0000","last_change":"2024-04-16T21:11:58.355713+0000","last_active":"2024-04-16T21:11:58.355619+0000","last_peered":"2024-04-16T21:11:58.355619+0000","last_clean":"2024-04-16T21:11:58.355619+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:11:58.355619+0000","last_undegraded":"2024-04-16T21:11:58.355619+0000","last_fullsized":"2024-04-16T21:11:58.355619+0000","mapping_epoch":33,"log_start":"47'400","ondisk_log_start":"47'400","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2068","last_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_deep_scrub":"52'2068","last_deep_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.355545+0000","objects_scrubbed":24,"log_size":1668,"log_dups_size":400,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:46.390834+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16267,"num_omap_keys":93,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'26","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.715965+0000","last_change":"2024-04-16T21:11:55.716041+0000","last_active":"2024-04-16T21:11:55.715965+0000","last_peered":"2024-04-16T21:11:55.715965+0000","last_clean":"2024-04-16T21:11:55.715965+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:11:55.715965+0000","last_undegraded":"2024-04-16T21:11:55.715965+0000","last_fullsized":"2024-04-16T21:11:55.715965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.715893+0000","objects_scrubbed":12,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:15.543756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388499+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:11:09.388499+0000","last_peered":"2024-04-16T21:11:09.388499+0000","last_clean":"2024-04-16T21:11:09.388499+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:11:09.388499+0000","last_undegraded":"2024-04-16T21:11:09.388499+0000","last_fullsized":"2024-04-16T21:11:09.388499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380998+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:10:19.380998+0000","last_peered":"2024-04-16T21:10:19.380998+0000","last_clean":"2024-04-16T21:10:19.380998+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:10:19.380998+0000","last_undegraded":"2024-04-16T21:10:19.380998+0000","last_fullsized":"2024-04-16T21:10:19.380998+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"23'1","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.522362+0000","last_change":"2024-04-16T19:24:45.410217+0000","last_active":"2024-04-16T21:10:34.522362+0000","last_peered":"2024-04-16T21:10:34.522362+0000","last_clean":"2024-04-16T21:10:34.522362+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:10:34.522362+0000","last_undegraded":"2024-04-16T21:10:34.522362+0000","last_fullsized":"2024-04-16T21:10:34.522362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'140","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387605+0000","last_change":"2024-04-16T19:24:47.422060+0000","last_active":"2024-04-16T21:10:59.387605+0000","last_peered":"2024-04-16T21:10:59.387605+0000","last_clean":"2024-04-16T21:10:59.387605+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:10:59.387605+0000","last_undegraded":"2024-04-16T21:10:59.387605+0000","last_fullsized":"2024-04-16T21:10:59.387605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'12","reported_seq":1314,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392208+0000","last_change":"2024-04-16T19:24:49.427836+0000","last_active":"2024-04-16T21:10:59.392208+0000","last_peered":"2024-04-16T21:10:59.392208+0000","last_clean":"2024-04-16T21:10:59.392208+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:10:59.392208+0000","last_undegraded":"2024-04-16T21:10:59.392208+0000","last_fullsized":"2024-04-16T21:10:59.392208+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.405984+0000","last_change":"2024-04-16T21:12:07.406028+0000","last_active":"2024-04-16T21:12:07.405984+0000","last_peered":"2024-04-16T21:12:07.405984+0000","last_clean":"2024-04-16T21:12:07.405984+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:12:07.405984+0000","last_undegraded":"2024-04-16T21:12:07.405984+0000","last_fullsized":"2024-04-16T21:12:07.405984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.405900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:12.444844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195903+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:11:43.195903+0000","last_peered":"2024-04-16T21:11:43.195903+0000","last_clean":"2024-04-16T21:11:43.195903+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:11:43.195903+0000","last_undegraded":"2024-04-16T21:11:43.195903+0000","last_fullsized":"2024-04-16T21:11:43.195903+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'182","reported_seq":1946,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.937380+0000","last_change":"2024-04-16T21:12:22.937438+0000","last_active":"2024-04-16T21:12:22.937380+0000","last_peered":"2024-04-16T21:12:22.937380+0000","last_clean":"2024-04-16T21:12:22.937380+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:12:22.937380+0000","last_undegraded":"2024-04-16T21:12:22.937380+0000","last_fullsized":"2024-04-16T21:12:22.937380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'182","last_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_deep_scrub":"52'182","last_deep_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.937321+0000","objects_scrubbed":46,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:18.195145+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8599,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561358+0000","last_change":"2024-04-16T19:25:41.507673+0000","last_active":"2024-04-16T21:11:44.561358+0000","last_peered":"2024-04-16T21:11:44.561358+0000","last_clean":"2024-04-16T21:11:44.561358+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:11:44.561358+0000","last_undegraded":"2024-04-16T21:11:44.561358+0000","last_fullsized":"2024-04-16T21:11:44.561358+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'17","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.440078+0000","last_change":"2024-04-16T21:12:10.440115+0000","last_active":"2024-04-16T21:12:10.440078+0000","last_peered":"2024-04-16T21:12:10.440078+0000","last_clean":"2024-04-16T21:12:10.440078+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:12:10.440078+0000","last_undegraded":"2024-04-16T21:12:10.440078+0000","last_fullsized":"2024-04-16T21:12:10.440078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.440017+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:44.207343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.910624+0000","last_change":"2024-04-16T21:12:13.910647+0000","last_active":"2024-04-16T21:12:13.910624+0000","last_peered":"2024-04-16T21:12:13.910624+0000","last_clean":"2024-04-16T21:12:13.910624+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:12:13.910624+0000","last_undegraded":"2024-04-16T21:12:13.910624+0000","last_fullsized":"2024-04-16T21:12:13.910624+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.910562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:19.649702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'362","reported_seq":695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.975586+0000","last_change":"2024-04-16T21:11:44.975641+0000","last_active":"2024-04-16T21:11:44.975586+0000","last_peered":"2024-04-16T21:11:44.975586+0000","last_clean":"2024-04-16T21:11:44.975586+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:11:44.975586+0000","last_undegraded":"2024-04-16T21:11:44.975586+0000","last_fullsized":"2024-04-16T21:11:44.975586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'362","last_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_deep_scrub":"52'362","last_deep_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.975535+0000","objects_scrubbed":21,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:50:56.598547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2185,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'18","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.687122+0000","last_change":"2024-04-16T21:11:48.687164+0000","last_active":"2024-04-16T21:11:48.687122+0000","last_peered":"2024-04-16T21:11:48.687122+0000","last_clean":"2024-04-16T21:11:48.687122+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:11:48.687122+0000","last_undegraded":"2024-04-16T21:11:48.687122+0000","last_fullsized":"2024-04-16T21:11:48.687122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.687069+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:52.625067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.386336+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:10:24.386336+0000","last_peered":"2024-04-16T21:10:24.386336+0000","last_clean":"2024-04-16T21:10:24.386336+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:10:24.386336+0000","last_undegraded":"2024-04-16T21:10:24.386336+0000","last_fullsized":"2024-04-16T21:10:24.386336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380925+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:10:19.380925+0000","last_peered":"2024-04-16T21:10:19.380925+0000","last_clean":"2024-04-16T21:10:19.380925+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:10:19.380925+0000","last_undegraded":"2024-04-16T21:10:19.380925+0000","last_fullsized":"2024-04-16T21:10:19.380925+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'16","reported_seq":458,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.528823+0000","last_change":"2024-04-16T19:24:45.408396+0000","last_active":"2024-04-16T21:11:14.528823+0000","last_peered":"2024-04-16T21:11:14.528823+0000","last_clean":"2024-04-16T21:11:14.528823+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:11:14.528823+0000","last_undegraded":"2024-04-16T21:11:14.528823+0000","last_fullsized":"2024-04-16T21:11:14.528823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'82","reported_seq":247,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392184+0000","last_change":"2024-04-16T19:24:47.423337+0000","last_active":"2024-04-16T21:10:59.392184+0000","last_peered":"2024-04-16T21:10:59.392184+0000","last_clean":"2024-04-16T21:10:59.392184+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:10:59.392184+0000","last_undegraded":"2024-04-16T21:10:59.392184+0000","last_fullsized":"2024-04-16T21:10:59.392184+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.899124+0000","last_change":"2024-04-16T21:12:18.899147+0000","last_active":"2024-04-16T21:12:18.899124+0000","last_peered":"2024-04-16T21:12:18.899124+0000","last_clean":"2024-04-16T21:12:18.899124+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:12:18.899124+0000","last_undegraded":"2024-04-16T21:12:18.899124+0000","last_fullsized":"2024-04-16T21:12:18.899124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.899064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:17.822519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.847627+0000","last_change":"2024-04-16T21:12:03.847650+0000","last_active":"2024-04-16T21:12:03.847627+0000","last_peered":"2024-04-16T21:12:03.847627+0000","last_clean":"2024-04-16T21:12:03.847627+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:12:03.847627+0000","last_undegraded":"2024-04-16T21:12:03.847627+0000","last_fullsized":"2024-04-16T21:12:03.847627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.847563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:36.440589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744155+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:11:42.744155+0000","last_peered":"2024-04-16T21:11:42.744155+0000","last_clean":"2024-04-16T21:11:42.744155+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:11:42.744155+0000","last_undegraded":"2024-04-16T21:11:42.744155+0000","last_fullsized":"2024-04-16T21:11:42.744155+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'134","reported_seq":2347,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293260+0000","last_change":"2024-04-16T19:25:49.520834+0000","last_active":"2024-04-16T21:11:42.293260+0000","last_peered":"2024-04-16T21:11:42.293260+0000","last_clean":"2024-04-16T21:11:42.293260+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:11:42.293260+0000","last_undegraded":"2024-04-16T21:11:42.293260+0000","last_fullsized":"2024-04-16T21:11:42.293260+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.650119+0000","last_change":"2024-04-16T21:12:13.650119+0000","last_active":"2024-04-16T21:12:13.650119+0000","last_peered":"2024-04-16T21:12:13.650119+0000","last_clean":"2024-04-16T21:12:13.650119+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:12:13.650119+0000","last_undegraded":"2024-04-16T21:12:13.650119+0000","last_fullsized":"2024-04-16T21:12:13.650119+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.650036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:16.977821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"52'13","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.901211+0000","last_change":"2024-04-16T21:12:20.901247+0000","last_active":"2024-04-16T21:12:20.901211+0000","last_peered":"2024-04-16T21:12:20.901211+0000","last_clean":"2024-04-16T21:12:20.901211+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:12:20.901211+0000","last_undegraded":"2024-04-16T21:12:20.901211+0000","last_fullsized":"2024-04-16T21:12:20.901211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.901143+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:54.668469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'706","reported_seq":2075,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.405465+0000","last_change":"2024-04-16T21:12:08.405492+0000","last_active":"2024-04-16T21:12:08.405465+0000","last_peered":"2024-04-16T21:12:08.405465+0000","last_clean":"2024-04-16T21:12:08.405465+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:12:08.405465+0000","last_undegraded":"2024-04-16T21:12:08.405465+0000","last_fullsized":"2024-04-16T21:12:08.405465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.405399+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:20.200130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'292","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.642477+0000","last_change":"2024-04-16T21:11:50.642568+0000","last_active":"2024-04-16T21:11:50.642477+0000","last_peered":"2024-04-16T21:11:50.642477+0000","last_clean":"2024-04-16T21:11:50.642477+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:11:50.642477+0000","last_undegraded":"2024-04-16T21:11:50.642477+0000","last_fullsized":"2024-04-16T21:11:50.642477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'292","last_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_deep_scrub":"52'292","last_deep_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.642400+0000","objects_scrubbed":23,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:29.527145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4142,"num_omap_keys":19,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.314574+0000","last_change":"2024-04-16T21:11:57.314619+0000","last_active":"2024-04-16T21:11:57.314574+0000","last_peered":"2024-04-16T21:11:57.314574+0000","last_clean":"2024-04-16T21:11:57.314574+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:11:57.314574+0000","last_undegraded":"2024-04-16T21:11:57.314574+0000","last_fullsized":"2024-04-16T21:11:57.314574+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.314511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:49.494822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380915+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:10:24.380915+0000","last_peered":"2024-04-16T21:10:24.380915+0000","last_clean":"2024-04-16T21:10:24.380915+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:10:24.380915+0000","last_undegraded":"2024-04-16T21:10:24.380915+0000","last_fullsized":"2024-04-16T21:10:24.380915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.898110+0000","last_change":"2024-04-16T21:12:08.898133+0000","last_active":"2024-04-16T21:12:08.898110+0000","last_peered":"2024-04-16T21:12:08.898110+0000","last_clean":"2024-04-16T21:12:08.898110+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:12:08.898110+0000","last_undegraded":"2024-04-16T21:12:08.898110+0000","last_fullsized":"2024-04-16T21:12:08.898110+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.898047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:36.956196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"47'1","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.522411+0000","last_change":"2024-04-16T19:24:45.410741+0000","last_active":"2024-04-16T21:10:34.522411+0000","last_peered":"2024-04-16T21:10:34.522411+0000","last_clean":"2024-04-16T21:10:34.522411+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:10:34.522411+0000","last_undegraded":"2024-04-16T21:10:34.522411+0000","last_fullsized":"2024-04-16T21:10:34.522411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'96","reported_seq":261,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382728+0000","last_change":"2024-04-16T19:24:47.421491+0000","last_active":"2024-04-16T21:10:34.382728+0000","last_peered":"2024-04-16T21:10:34.382728+0000","last_clean":"2024-04-16T21:10:34.382728+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:10:34.382728+0000","last_undegraded":"2024-04-16T21:10:34.382728+0000","last_fullsized":"2024-04-16T21:10:34.382728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391424+0000","last_change":"2024-04-16T19:24:49.427104+0000","last_active":"2024-04-16T21:10:54.391424+0000","last_peered":"2024-04-16T21:10:54.391424+0000","last_clean":"2024-04-16T21:10:54.391424+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:10:54.391424+0000","last_undegraded":"2024-04-16T21:10:54.391424+0000","last_fullsized":"2024-04-16T21:10:54.391424+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.889383+0000","last_change":"2024-04-16T21:12:17.889407+0000","last_active":"2024-04-16T21:12:17.889383+0000","last_peered":"2024-04-16T21:12:17.889383+0000","last_clean":"2024-04-16T21:12:17.889383+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:12:17.889383+0000","last_undegraded":"2024-04-16T21:12:17.889383+0000","last_fullsized":"2024-04-16T21:12:17.889383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.889318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:23.628459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293308+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:11:42.293308+0000","last_peered":"2024-04-16T21:11:42.293308+0000","last_clean":"2024-04-16T21:11:42.293308+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:11:42.293308+0000","last_undegraded":"2024-04-16T21:11:42.293308+0000","last_fullsized":"2024-04-16T21:11:42.293308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1680,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561550+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:11:44.561550+0000","last_peered":"2024-04-16T21:11:44.561550+0000","last_clean":"2024-04-16T21:11:44.561550+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:11:44.561550+0000","last_undegraded":"2024-04-16T21:11:44.561550+0000","last_fullsized":"2024-04-16T21:11:44.561550+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.854832+0000","last_change":"2024-04-16T21:12:21.854876+0000","last_active":"2024-04-16T21:12:21.854832+0000","last_peered":"2024-04-16T21:12:21.854832+0000","last_clean":"2024-04-16T21:12:21.854832+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:12:21.854832+0000","last_undegraded":"2024-04-16T21:12:21.854832+0000","last_fullsized":"2024-04-16T21:12:21.854832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.854759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:37.324132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'138","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.362284+0000","last_change":"2024-04-16T21:12:22.362331+0000","last_active":"2024-04-16T21:12:22.362284+0000","last_peered":"2024-04-16T21:12:22.362284+0000","last_clean":"2024-04-16T21:12:22.362284+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:12:22.362284+0000","last_undegraded":"2024-04-16T21:12:22.362284+0000","last_fullsized":"2024-04-16T21:12:22.362284+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.362197+0000","objects_scrubbed":2,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:16.550872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9707,"num_omap_keys":56,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.836583+0000","last_change":"2024-04-16T21:11:52.836606+0000","last_active":"2024-04-16T21:11:52.836583+0000","last_peered":"2024-04-16T21:11:52.836583+0000","last_clean":"2024-04-16T21:11:52.836583+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:11:52.836583+0000","last_undegraded":"2024-04-16T21:11:52.836583+0000","last_fullsized":"2024-04-16T21:11:52.836583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.836519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:26.725994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'371","reported_seq":758,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.959985+0000","last_change":"2024-04-16T21:11:46.960080+0000","last_active":"2024-04-16T21:11:46.959985+0000","last_peered":"2024-04-16T21:11:46.959985+0000","last_clean":"2024-04-16T21:11:46.959985+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:11:46.959985+0000","last_undegraded":"2024-04-16T21:11:46.959985+0000","last_fullsized":"2024-04-16T21:11:46.959985+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'371","last_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_deep_scrub":"52'371","last_deep_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.959920+0000","objects_scrubbed":24,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:28.129840+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2169,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'19","reported_seq":216,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.614888+0000","last_change":"2024-04-16T21:11:53.614931+0000","last_active":"2024-04-16T21:11:53.614888+0000","last_peered":"2024-04-16T21:11:53.614888+0000","last_clean":"2024-04-16T21:11:53.614888+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:11:53.614888+0000","last_undegraded":"2024-04-16T21:11:53.614888+0000","last_fullsized":"2024-04-16T21:11:53.614888+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.614826+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:49.084093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381758+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:10:24.381758+0000","last_peered":"2024-04-16T21:10:24.381758+0000","last_clean":"2024-04-16T21:10:24.381758+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:10:24.381758+0000","last_undegraded":"2024-04-16T21:10:24.381758+0000","last_fullsized":"2024-04-16T21:10:24.381758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.827874+0000","last_change":"2024-04-16T21:12:04.827874+0000","last_active":"2024-04-16T21:12:04.827874+0000","last_peered":"2024-04-16T21:12:04.827874+0000","last_clean":"2024-04-16T21:12:04.827874+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:12:04.827874+0000","last_undegraded":"2024-04-16T21:12:04.827874+0000","last_fullsized":"2024-04-16T21:12:04.827874+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.827767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:23.692877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382104+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:10:34.382104+0000","last_peered":"2024-04-16T21:10:34.382104+0000","last_clean":"2024-04-16T21:10:34.382104+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:10:34.382104+0000","last_undegraded":"2024-04-16T21:10:34.382104+0000","last_fullsized":"2024-04-16T21:10:34.382104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'147","reported_seq":342,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.899689+0000","last_change":"2024-04-16T21:12:22.899755+0000","last_active":"2024-04-16T21:12:22.899689+0000","last_peered":"2024-04-16T21:12:22.899689+0000","last_clean":"2024-04-16T21:12:22.899689+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:12:22.899689+0000","last_undegraded":"2024-04-16T21:12:22.899689+0000","last_fullsized":"2024-04-16T21:12:22.899689+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.899611+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:49.019687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'13","reported_seq":1321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.856467+0000","last_change":"2024-04-16T21:12:19.856493+0000","last_active":"2024-04-16T21:12:19.856467+0000","last_peered":"2024-04-16T21:12:19.856467+0000","last_clean":"2024-04-16T21:12:19.856467+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:12:19.856467+0000","last_undegraded":"2024-04-16T21:12:19.856467+0000","last_fullsized":"2024-04-16T21:12:19.856467+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.856405+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.872492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.913945+0000","last_change":"2024-04-16T21:12:16.913967+0000","last_active":"2024-04-16T21:12:16.913945+0000","last_peered":"2024-04-16T21:12:16.913945+0000","last_clean":"2024-04-16T21:12:16.913945+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:12:16.913945+0000","last_undegraded":"2024-04-16T21:12:16.913945+0000","last_fullsized":"2024-04-16T21:12:16.913945+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.913879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:03.787910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389430+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:11:09.389430+0000","last_peered":"2024-04-16T21:11:09.389430+0000","last_clean":"2024-04-16T21:11:09.389430+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:11:09.389430+0000","last_undegraded":"2024-04-16T21:11:09.389430+0000","last_fullsized":"2024-04-16T21:11:09.389430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'122","reported_seq":1872,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651400+0000","last_change":"2024-04-16T19:25:49.519491+0000","last_active":"2024-04-16T21:11:43.651400+0000","last_peered":"2024-04-16T21:11:43.651400+0000","last_clean":"2024-04-16T21:11:43.651400+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:11:43.651400+0000","last_undegraded":"2024-04-16T21:11:43.651400+0000","last_fullsized":"2024-04-16T21:11:43.651400+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.368999+0000","last_change":"2024-04-16T21:12:24.369021+0000","last_active":"2024-04-16T21:12:24.368999+0000","last_peered":"2024-04-16T21:12:24.368999+0000","last_clean":"2024-04-16T21:12:24.368999+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:12:24.368999+0000","last_undegraded":"2024-04-16T21:12:24.368999+0000","last_fullsized":"2024-04-16T21:12:24.368999+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.368926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:12.501416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.879266+0000","last_change":"2024-04-16T21:12:15.879307+0000","last_active":"2024-04-16T21:12:15.879266+0000","last_peered":"2024-04-16T21:12:15.879266+0000","last_clean":"2024-04-16T21:12:15.879266+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:12:15.879266+0000","last_undegraded":"2024-04-16T21:12:15.879266+0000","last_fullsized":"2024-04-16T21:12:15.879266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.879198+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:41.314648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.625545+0000","last_change":"2024-04-16T21:11:53.625568+0000","last_active":"2024-04-16T21:11:53.625545+0000","last_peered":"2024-04-16T21:11:53.625545+0000","last_clean":"2024-04-16T21:11:53.625545+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:11:53.625545+0000","last_undegraded":"2024-04-16T21:11:53.625545+0000","last_fullsized":"2024-04-16T21:11:53.625545+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.625484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:35.360695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'485","reported_seq":911,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.722929+0000","last_change":"2024-04-16T21:11:47.722980+0000","last_active":"2024-04-16T21:11:47.722929+0000","last_peered":"2024-04-16T21:11:47.722929+0000","last_clean":"2024-04-16T21:11:47.722929+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:11:47.722929+0000","last_undegraded":"2024-04-16T21:11:47.722929+0000","last_fullsized":"2024-04-16T21:11:47.722929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'485","last_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_deep_scrub":"52'485","last_deep_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.722865+0000","objects_scrubbed":32,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:39.504569+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42883,"num_omap_keys":132,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'14","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.616163+0000","last_change":"2024-04-16T21:12:05.616245+0000","last_active":"2024-04-16T21:12:05.616163+0000","last_peered":"2024-04-16T21:12:05.616163+0000","last_clean":"2024-04-16T21:12:05.616163+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:12:05.616163+0000","last_undegraded":"2024-04-16T21:12:05.616163+0000","last_fullsized":"2024-04-16T21:12:05.616163+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.616057+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.466963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380981+0000","last_change":"2024-04-16T19:24:35.401480+0000","last_active":"2024-04-16T21:10:24.380981+0000","last_peered":"2024-04-16T21:10:24.380981+0000","last_clean":"2024-04-16T21:10:24.380981+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:10:24.380981+0000","last_undegraded":"2024-04-16T21:10:24.380981+0000","last_fullsized":"2024-04-16T21:10:24.380981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384338+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:10:44.384338+0000","last_peered":"2024-04-16T21:10:44.384338+0000","last_clean":"2024-04-16T21:10:44.384338+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:10:44.384338+0000","last_undegraded":"2024-04-16T21:10:44.384338+0000","last_fullsized":"2024-04-16T21:10:44.384338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386376+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:10:54.386376+0000","last_peered":"2024-04-16T21:10:54.386376+0000","last_clean":"2024-04-16T21:10:54.386376+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:10:54.386376+0000","last_undegraded":"2024-04-16T21:10:54.386376+0000","last_fullsized":"2024-04-16T21:10:54.386376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'110","reported_seq":277,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382118+0000","last_change":"2024-04-16T19:24:47.424300+0000","last_active":"2024-04-16T21:10:34.382118+0000","last_peered":"2024-04-16T21:10:34.382118+0000","last_clean":"2024-04-16T21:10:34.382118+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:10:34.382118+0000","last_undegraded":"2024-04-16T21:10:34.382118+0000","last_fullsized":"2024-04-16T21:10:34.382118+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'13","reported_seq":1321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.607231+0000","last_change":"2024-04-16T21:12:22.607256+0000","last_active":"2024-04-16T21:12:22.607231+0000","last_peered":"2024-04-16T21:12:22.607231+0000","last_clean":"2024-04-16T21:12:22.607231+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:12:22.607231+0000","last_undegraded":"2024-04-16T21:12:22.607231+0000","last_fullsized":"2024-04-16T21:12:22.607231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.607175+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:48.727251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.673878+0000","last_change":"2024-04-16T21:12:20.673922+0000","last_active":"2024-04-16T21:12:20.673878+0000","last_peered":"2024-04-16T21:12:20.673878+0000","last_clean":"2024-04-16T21:12:20.673878+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:12:20.673878+0000","last_undegraded":"2024-04-16T21:12:20.673878+0000","last_fullsized":"2024-04-16T21:12:20.673878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.673807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:17.517736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385558+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:10:54.385558+0000","last_peered":"2024-04-16T21:10:54.385558+0000","last_clean":"2024-04-16T21:10:54.385558+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:10:54.385558+0000","last_undegraded":"2024-04-16T21:10:54.385558+0000","last_fullsized":"2024-04-16T21:10:54.385558+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'121","reported_seq":2259,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293255+0000","last_change":"2024-04-16T19:25:49.517222+0000","last_active":"2024-04-16T21:11:42.293255+0000","last_peered":"2024-04-16T21:11:42.293255+0000","last_clean":"2024-04-16T21:11:42.293255+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:11:42.293255+0000","last_undegraded":"2024-04-16T21:11:42.293255+0000","last_fullsized":"2024-04-16T21:11:42.293255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.346082+0000","last_change":"2024-04-16T21:12:20.346107+0000","last_active":"2024-04-16T21:12:20.346082+0000","last_peered":"2024-04-16T21:12:20.346082+0000","last_clean":"2024-04-16T21:12:20.346082+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:12:20.346082+0000","last_undegraded":"2024-04-16T21:12:20.346082+0000","last_fullsized":"2024-04-16T21:12:20.346082+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.346020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:28.840252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.905708+0000","last_change":"2024-04-16T21:12:13.905733+0000","last_active":"2024-04-16T21:12:13.905708+0000","last_peered":"2024-04-16T21:12:13.905708+0000","last_clean":"2024-04-16T21:12:13.905708+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:12:13.905708+0000","last_undegraded":"2024-04-16T21:12:13.905708+0000","last_fullsized":"2024-04-16T21:12:13.905708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.905640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:17.215414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'706","reported_seq":2474,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.688857+0000","last_change":"2024-04-16T21:11:58.688891+0000","last_active":"2024-04-16T21:11:58.688857+0000","last_peered":"2024-04-16T21:11:58.688857+0000","last_clean":"2024-04-16T21:11:58.688857+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:11:58.688857+0000","last_undegraded":"2024-04-16T21:11:58.688857+0000","last_fullsized":"2024-04-16T21:11:58.688857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.688787+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:58.866593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'332","reported_seq":603,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.669169+0000","last_change":"2024-04-16T21:11:50.669285+0000","last_active":"2024-04-16T21:11:50.669169+0000","last_peered":"2024-04-16T21:11:50.669169+0000","last_clean":"2024-04-16T21:11:50.669169+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:11:50.669169+0000","last_undegraded":"2024-04-16T21:11:50.669169+0000","last_fullsized":"2024-04-16T21:11:50.669169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'332","last_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_deep_scrub":"52'332","last_deep_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.669096+0000","objects_scrubbed":21,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:18.487063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2179,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'12","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.639977+0000","last_change":"2024-04-16T21:11:49.640051+0000","last_active":"2024-04-16T21:11:49.639977+0000","last_peered":"2024-04-16T21:11:49.639977+0000","last_clean":"2024-04-16T21:11:49.639977+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:11:49.639977+0000","last_undegraded":"2024-04-16T21:11:49.639977+0000","last_fullsized":"2024-04-16T21:11:49.639977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.639900+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:27.425866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383761+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:10:44.383761+0000","last_peered":"2024-04-16T21:10:44.383761+0000","last_clean":"2024-04-16T21:10:44.383761+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:10:44.383761+0000","last_undegraded":"2024-04-16T21:10:44.383761+0000","last_fullsized":"2024-04-16T21:10:44.383761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.523887+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:10:44.523887+0000","last_peered":"2024-04-16T21:10:44.523887+0000","last_clean":"2024-04-16T21:10:44.523887+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:10:44.523887+0000","last_undegraded":"2024-04-16T21:10:44.523887+0000","last_fullsized":"2024-04-16T21:10:44.523887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383477+0000","last_change":"2024-04-16T19:24:45.414093+0000","last_active":"2024-04-16T21:10:34.383477+0000","last_peered":"2024-04-16T21:10:34.383477+0000","last_clean":"2024-04-16T21:10:34.383477+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:10:34.383477+0000","last_undegraded":"2024-04-16T21:10:34.383477+0000","last_fullsized":"2024-04-16T21:10:34.383477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'162","reported_seq":360,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383080+0000","last_change":"2024-04-16T19:24:47.420873+0000","last_active":"2024-04-16T21:10:34.383080+0000","last_peered":"2024-04-16T21:10:34.383080+0000","last_clean":"2024-04-16T21:10:34.383080+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:10:34.383080+0000","last_undegraded":"2024-04-16T21:10:34.383080+0000","last_fullsized":"2024-04-16T21:10:34.383080+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383416+0000","last_change":"2024-04-16T19:24:49.431047+0000","last_active":"2024-04-16T21:10:34.383416+0000","last_peered":"2024-04-16T21:10:34.383416+0000","last_clean":"2024-04-16T21:10:34.383416+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:10:34.383416+0000","last_undegraded":"2024-04-16T21:10:34.383416+0000","last_fullsized":"2024-04-16T21:10:34.383416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.903511+0000","last_change":"2024-04-16T21:12:19.903531+0000","last_active":"2024-04-16T21:12:19.903511+0000","last_peered":"2024-04-16T21:12:19.903511+0000","last_clean":"2024-04-16T21:12:19.903511+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:12:19.903511+0000","last_undegraded":"2024-04-16T21:12:19.903511+0000","last_fullsized":"2024-04-16T21:12:19.903511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.903448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:31.698180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524809+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:10:49.524809+0000","last_peered":"2024-04-16T21:10:49.524809+0000","last_clean":"2024-04-16T21:10:49.524809+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:10:49.524809+0000","last_undegraded":"2024-04-16T21:10:49.524809+0000","last_fullsized":"2024-04-16T21:10:49.524809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'216","reported_seq":2161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.661893+0000","last_change":"2024-04-16T21:12:21.661923+0000","last_active":"2024-04-16T21:12:21.661893+0000","last_peered":"2024-04-16T21:12:21.661893+0000","last_clean":"2024-04-16T21:12:21.661893+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:12:21.661893+0000","last_undegraded":"2024-04-16T21:12:21.661893+0000","last_fullsized":"2024-04-16T21:12:21.661893+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'216","last_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_deep_scrub":"52'216","last_deep_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.661832+0000","objects_scrubbed":54,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.097281+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14476,"num_omap_keys":58,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.921849+0000","last_change":"2024-04-16T21:12:14.921870+0000","last_active":"2024-04-16T21:12:14.921849+0000","last_peered":"2024-04-16T21:12:14.921849+0000","last_clean":"2024-04-16T21:12:14.921849+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:12:14.921849+0000","last_undegraded":"2024-04-16T21:12:14.921849+0000","last_fullsized":"2024-04-16T21:12:14.921849+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.921774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:11.765704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'17","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.682277+0000","last_change":"2024-04-16T21:12:08.682317+0000","last_active":"2024-04-16T21:12:08.682277+0000","last_peered":"2024-04-16T21:12:08.682277+0000","last_clean":"2024-04-16T21:12:08.682277+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:12:08.682277+0000","last_undegraded":"2024-04-16T21:12:08.682277+0000","last_fullsized":"2024-04-16T21:12:08.682277+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.682212+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:32.852256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.634331+0000","last_change":"2024-04-16T21:11:56.634374+0000","last_active":"2024-04-16T21:11:56.634331+0000","last_peered":"2024-04-16T21:11:56.634331+0000","last_clean":"2024-04-16T21:11:56.634331+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:11:56.634331+0000","last_undegraded":"2024-04-16T21:11:56.634331+0000","last_fullsized":"2024-04-16T21:11:56.634331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.634253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:30.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'281","reported_seq":601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.734176+0000","last_change":"2024-04-16T21:11:44.734203+0000","last_active":"2024-04-16T21:11:44.734176+0000","last_peered":"2024-04-16T21:11:44.734176+0000","last_clean":"2024-04-16T21:11:44.734176+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:11:44.734176+0000","last_undegraded":"2024-04-16T21:11:44.734176+0000","last_fullsized":"2024-04-16T21:11:44.734176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'281","last_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_deep_scrub":"52'281","last_deep_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.734118+0000","objects_scrubbed":25,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:22.520084+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1975,"num_omap_keys":9,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'12","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.876267+0000","last_change":"2024-04-16T21:12:09.876315+0000","last_active":"2024-04-16T21:12:09.876267+0000","last_peered":"2024-04-16T21:12:09.876267+0000","last_clean":"2024-04-16T21:12:09.876267+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:12:09.876267+0000","last_undegraded":"2024-04-16T21:12:09.876267+0000","last_fullsized":"2024-04-16T21:12:09.876267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.876206+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.314787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390257+0000","last_change":"2024-04-16T19:24:35.398140+0000","last_active":"2024-04-16T21:11:14.390257+0000","last_peered":"2024-04-16T21:11:14.390257+0000","last_clean":"2024-04-16T21:11:14.390257+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:11:14.390257+0000","last_undegraded":"2024-04-16T21:11:14.390257+0000","last_fullsized":"2024-04-16T21:11:14.390257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649866+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:11:43.649866+0000","last_peered":"2024-04-16T21:11:43.649866+0000","last_clean":"2024-04-16T21:11:43.649866+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:11:43.649866+0000","last_undegraded":"2024-04-16T21:11:43.649866+0000","last_fullsized":"2024-04-16T21:11:43.649866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560124+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:11:44.560124+0000","last_peered":"2024-04-16T21:11:44.560124+0000","last_clean":"2024-04-16T21:11:44.560124+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:11:44.560124+0000","last_undegraded":"2024-04-16T21:11:44.560124+0000","last_fullsized":"2024-04-16T21:11:44.560124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384510+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:10:49.384510+0000","last_peered":"2024-04-16T21:10:49.384510+0000","last_clean":"2024-04-16T21:10:49.384510+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:10:49.384510+0000","last_undegraded":"2024-04-16T21:10:49.384510+0000","last_fullsized":"2024-04-16T21:10:49.384510+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387912+0000","last_change":"2024-04-16T19:24:45.410438+0000","last_active":"2024-04-16T21:11:04.387912+0000","last_peered":"2024-04-16T21:11:04.387912+0000","last_clean":"2024-04-16T21:11:04.387912+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:11:04.387912+0000","last_undegraded":"2024-04-16T21:11:04.387912+0000","last_fullsized":"2024-04-16T21:11:04.387912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'68","reported_seq":231,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387149+0000","last_change":"2024-04-16T19:24:47.425795+0000","last_active":"2024-04-16T21:10:59.387149+0000","last_peered":"2024-04-16T21:10:59.387149+0000","last_clean":"2024-04-16T21:10:59.387149+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:10:59.387149+0000","last_undegraded":"2024-04-16T21:10:59.387149+0000","last_fullsized":"2024-04-16T21:10:59.387149+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392359+0000","last_change":"2024-04-16T19:24:49.428792+0000","last_active":"2024-04-16T21:10:59.392359+0000","last_peered":"2024-04-16T21:10:59.392359+0000","last_clean":"2024-04-16T21:10:59.392359+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:10:59.392359+0000","last_undegraded":"2024-04-16T21:10:59.392359+0000","last_fullsized":"2024-04-16T21:10:59.392359+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.689142+0000","last_change":"2024-04-16T21:12:06.689165+0000","last_active":"2024-04-16T21:12:06.689142+0000","last_peered":"2024-04-16T21:12:06.689142+0000","last_clean":"2024-04-16T21:12:06.689142+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:12:06.689142+0000","last_undegraded":"2024-04-16T21:12:06.689142+0000","last_fullsized":"2024-04-16T21:12:06.689142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.689080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:58.869391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385439+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:10:49.385439+0000","last_peered":"2024-04-16T21:10:49.385439+0000","last_clean":"2024-04-16T21:10:49.385439+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:10:49.385439+0000","last_undegraded":"2024-04-16T21:10:49.385439+0000","last_fullsized":"2024-04-16T21:10:49.385439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'135","reported_seq":1952,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398096+0000","last_change":"2024-04-16T19:25:49.514823+0000","last_active":"2024-04-16T21:11:34.398096+0000","last_peered":"2024-04-16T21:11:34.398096+0000","last_clean":"2024-04-16T21:11:34.398096+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:11:34.398096+0000","last_undegraded":"2024-04-16T21:11:34.398096+0000","last_fullsized":"2024-04-16T21:11:34.398096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561359+0000","last_change":"2024-04-16T19:25:41.509469+0000","last_active":"2024-04-16T21:11:44.561359+0000","last_peered":"2024-04-16T21:11:44.561359+0000","last_clean":"2024-04-16T21:11:44.561359+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:11:44.561359+0000","last_undegraded":"2024-04-16T21:11:44.561359+0000","last_fullsized":"2024-04-16T21:11:44.561359+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'16","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.513309+0000","last_change":"2024-04-16T21:12:20.513369+0000","last_active":"2024-04-16T21:12:20.513309+0000","last_peered":"2024-04-16T21:12:20.513309+0000","last_clean":"2024-04-16T21:12:20.513309+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:12:20.513309+0000","last_undegraded":"2024-04-16T21:12:20.513309+0000","last_fullsized":"2024-04-16T21:12:20.513309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.513228+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:32.115899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'706","reported_seq":2087,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.915788+0000","last_change":"2024-04-16T21:11:48.915841+0000","last_active":"2024-04-16T21:11:48.915788+0000","last_peered":"2024-04-16T21:11:48.915788+0000","last_clean":"2024-04-16T21:11:48.915788+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:11:48.915788+0000","last_undegraded":"2024-04-16T21:11:48.915788+0000","last_fullsized":"2024-04-16T21:11:48.915788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.915706+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.964684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'326","reported_seq":644,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.279300+0000","last_change":"2024-04-16T21:11:56.279412+0000","last_active":"2024-04-16T21:11:56.279300+0000","last_peered":"2024-04-16T21:11:56.279300+0000","last_clean":"2024-04-16T21:11:56.279300+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:11:56.279300+0000","last_undegraded":"2024-04-16T21:11:56.279300+0000","last_fullsized":"2024-04-16T21:11:56.279300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'326","last_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_deep_scrub":"52'326","last_deep_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.279222+0000","objects_scrubbed":20,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:03.299266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'20","reported_seq":201,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.117827+0000","last_change":"2024-04-16T21:11:45.117905+0000","last_active":"2024-04-16T21:11:45.117827+0000","last_peered":"2024-04-16T21:11:45.117827+0000","last_clean":"2024-04-16T21:11:45.117827+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:11:45.117827+0000","last_undegraded":"2024-04-16T21:11:45.117827+0000","last_fullsized":"2024-04-16T21:11:45.117827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.117757+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:23.210409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390704+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:10:49.390704+0000","last_peered":"2024-04-16T21:10:49.390704+0000","last_clean":"2024-04-16T21:10:49.390704+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:10:49.390704+0000","last_undegraded":"2024-04-16T21:10:49.390704+0000","last_fullsized":"2024-04-16T21:10:49.390704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649881+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:11:43.649881+0000","last_peered":"2024-04-16T21:11:43.649881+0000","last_clean":"2024-04-16T21:11:43.649881+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:11:43.649881+0000","last_undegraded":"2024-04-16T21:11:43.649881+0000","last_fullsized":"2024-04-16T21:11:43.649881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390485+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:10:49.390485+0000","last_peered":"2024-04-16T21:10:49.390485+0000","last_clean":"2024-04-16T21:10:49.390485+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:10:49.390485+0000","last_undegraded":"2024-04-16T21:10:49.390485+0000","last_fullsized":"2024-04-16T21:10:49.390485+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"47'4","reported_seq":592,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387164+0000","last_change":"2024-04-16T19:24:45.414986+0000","last_active":"2024-04-16T21:10:59.387164+0000","last_peered":"2024-04-16T21:10:59.387164+0000","last_clean":"2024-04-16T21:10:59.387164+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:10:59.387164+0000","last_undegraded":"2024-04-16T21:10:59.387164+0000","last_fullsized":"2024-04-16T21:10:59.387164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'165","reported_seq":1570,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387167+0000","last_change":"2024-04-16T19:24:47.423637+0000","last_active":"2024-04-16T21:11:04.387167+0000","last_peered":"2024-04-16T21:11:04.387167+0000","last_clean":"2024-04-16T21:11:04.387167+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:11:04.387167+0000","last_undegraded":"2024-04-16T21:11:04.387167+0000","last_fullsized":"2024-04-16T21:11:04.387167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.390988+0000","last_change":"2024-04-16T19:24:49.425901+0000","last_active":"2024-04-16T21:11:24.390988+0000","last_peered":"2024-04-16T21:11:24.390988+0000","last_clean":"2024-04-16T21:11:24.390988+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:11:24.390988+0000","last_undegraded":"2024-04-16T21:11:24.390988+0000","last_fullsized":"2024-04-16T21:11:24.390988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.671618+0000","last_change":"2024-04-16T21:11:57.671660+0000","last_active":"2024-04-16T21:11:57.671618+0000","last_peered":"2024-04-16T21:11:57.671618+0000","last_clean":"2024-04-16T21:11:57.671618+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:11:57.671618+0000","last_undegraded":"2024-04-16T21:11:57.671618+0000","last_fullsized":"2024-04-16T21:11:57.671618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.671545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:20.862941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385351+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:10:49.385351+0000","last_peered":"2024-04-16T21:10:49.385351+0000","last_clean":"2024-04-16T21:10:49.385351+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:10:49.385351+0000","last_undegraded":"2024-04-16T21:10:49.385351+0000","last_fullsized":"2024-04-16T21:10:49.385351+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2370,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392699+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:11:34.392699+0000","last_peered":"2024-04-16T21:11:34.392699+0000","last_clean":"2024-04-16T21:11:34.392699+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:11:34.392699+0000","last_undegraded":"2024-04-16T21:11:34.392699+0000","last_fullsized":"2024-04-16T21:11:34.392699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391873+0000","last_change":"2024-04-16T19:25:41.504482+0000","last_active":"2024-04-16T21:11:29.391873+0000","last_peered":"2024-04-16T21:11:29.391873+0000","last_clean":"2024-04-16T21:11:29.391873+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:11:29.391873+0000","last_undegraded":"2024-04-16T21:11:29.391873+0000","last_fullsized":"2024-04-16T21:11:29.391873+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'19","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.380245+0000","last_change":"2024-04-16T21:12:21.380273+0000","last_active":"2024-04-16T21:12:21.380245+0000","last_peered":"2024-04-16T21:12:21.380245+0000","last_clean":"2024-04-16T21:12:21.380245+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:12:21.380245+0000","last_undegraded":"2024-04-16T21:12:21.380245+0000","last_fullsized":"2024-04-16T21:12:21.380245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.380184+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:53.066886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":60,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'21","reported_seq":982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.783889+0000","last_change":"2024-04-16T21:11:58.783920+0000","last_active":"2024-04-16T21:11:58.783889+0000","last_peered":"2024-04-16T21:11:58.783889+0000","last_clean":"2024-04-16T21:11:58.783889+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:11:58.783889+0000","last_undegraded":"2024-04-16T21:11:58.783889+0000","last_fullsized":"2024-04-16T21:11:58.783889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.783819+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:59.285203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'268","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.359130+0000","last_change":"2024-04-16T21:11:44.359183+0000","last_active":"2024-04-16T21:11:44.359130+0000","last_peered":"2024-04-16T21:11:44.359130+0000","last_clean":"2024-04-16T21:11:44.359130+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:11:44.359130+0000","last_undegraded":"2024-04-16T21:11:44.359130+0000","last_fullsized":"2024-04-16T21:11:44.359130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'268","last_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_deep_scrub":"52'268","last_deep_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.359071+0000","objects_scrubbed":19,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:18.502556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'13","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.427731+0000","last_change":"2024-04-16T21:11:53.427771+0000","last_active":"2024-04-16T21:11:53.427731+0000","last_peered":"2024-04-16T21:11:53.427731+0000","last_clean":"2024-04-16T21:11:53.427731+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:11:53.427731+0000","last_undegraded":"2024-04-16T21:11:53.427731+0000","last_fullsized":"2024-04-16T21:11:53.427731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.427656+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:14.929241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384655+0000","last_change":"2024-04-16T19:24:35.399584+0000","last_active":"2024-04-16T21:10:49.384655+0000","last_peered":"2024-04-16T21:10:49.384655+0000","last_clean":"2024-04-16T21:10:49.384655+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:10:49.384655+0000","last_undegraded":"2024-04-16T21:10:49.384655+0000","last_fullsized":"2024-04-16T21:10:49.384655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194715+0000","last_change":"2024-04-16T19:23:47.119311+0000","last_active":"2024-04-16T21:11:43.194715+0000","last_peered":"2024-04-16T21:11:43.194715+0000","last_clean":"2024-04-16T21:11:43.194715+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:11:43.194715+0000","last_undegraded":"2024-04-16T21:11:43.194715+0000","last_fullsized":"2024-04-16T21:11:43.194715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385426+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:10:54.385426+0000","last_peered":"2024-04-16T21:10:54.385426+0000","last_clean":"2024-04-16T21:10:54.385426+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:10:54.385426+0000","last_undegraded":"2024-04-16T21:10:54.385426+0000","last_fullsized":"2024-04-16T21:10:54.385426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"47'5","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.526325+0000","last_change":"2024-04-16T19:24:45.410858+0000","last_active":"2024-04-16T21:10:59.526325+0000","last_peered":"2024-04-16T21:10:59.526325+0000","last_clean":"2024-04-16T21:10:59.526325+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:10:59.526325+0000","last_undegraded":"2024-04-16T21:10:59.526325+0000","last_fullsized":"2024-04-16T21:10:59.526325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'139","reported_seq":1514,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.390979+0000","last_change":"2024-04-16T19:24:47.419810+0000","last_active":"2024-04-16T21:11:24.390979+0000","last_peered":"2024-04-16T21:11:24.390979+0000","last_clean":"2024-04-16T21:11:24.390979+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:11:24.390979+0000","last_undegraded":"2024-04-16T21:11:24.390979+0000","last_fullsized":"2024-04-16T21:11:24.390979+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393254+0000","last_change":"2024-04-16T19:24:49.431666+0000","last_active":"2024-04-16T21:11:04.393254+0000","last_peered":"2024-04-16T21:11:04.393254+0000","last_clean":"2024-04-16T21:11:04.393254+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:11:04.393254+0000","last_undegraded":"2024-04-16T21:11:04.393254+0000","last_fullsized":"2024-04-16T21:11:04.393254+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.370511+0000","last_change":"2024-04-16T21:12:05.370546+0000","last_active":"2024-04-16T21:12:05.370511+0000","last_peered":"2024-04-16T21:12:05.370511+0000","last_clean":"2024-04-16T21:12:05.370511+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:12:05.370511+0000","last_undegraded":"2024-04-16T21:12:05.370511+0000","last_fullsized":"2024-04-16T21:12:05.370511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.370430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:17.165162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388785+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:11:04.388785+0000","last_peered":"2024-04-16T21:11:04.388785+0000","last_clean":"2024-04-16T21:11:04.388785+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:11:04.388785+0000","last_undegraded":"2024-04-16T21:11:04.388785+0000","last_fullsized":"2024-04-16T21:11:04.388785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1767,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392656+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:11:34.392656+0000","last_peered":"2024-04-16T21:11:34.392656+0000","last_clean":"2024-04-16T21:11:34.392656+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:11:34.392656+0000","last_undegraded":"2024-04-16T21:11:34.392656+0000","last_fullsized":"2024-04-16T21:11:34.392656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.699415+0000","last_change":"2024-04-16T21:11:59.699455+0000","last_active":"2024-04-16T21:11:59.699415+0000","last_peered":"2024-04-16T21:11:59.699415+0000","last_clean":"2024-04-16T21:11:59.699415+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:11:59.699415+0000","last_undegraded":"2024-04-16T21:11:59.699415+0000","last_fullsized":"2024-04-16T21:11:59.699415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.699336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'21","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.432050+0000","last_change":"2024-04-16T21:12:15.432112+0000","last_active":"2024-04-16T21:12:15.432050+0000","last_peered":"2024-04-16T21:12:15.432050+0000","last_clean":"2024-04-16T21:12:15.432050+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:12:15.432050+0000","last_undegraded":"2024-04-16T21:12:15.432050+0000","last_fullsized":"2024-04-16T21:12:15.432050+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.431977+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:24.993645+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.891648+0000","last_change":"2024-04-16T21:12:03.891673+0000","last_active":"2024-04-16T21:12:03.891648+0000","last_peered":"2024-04-16T21:12:03.891648+0000","last_clean":"2024-04-16T21:12:03.891648+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:12:03.891648+0000","last_undegraded":"2024-04-16T21:12:03.891648+0000","last_fullsized":"2024-04-16T21:12:03.891648+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.891582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:27.082978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'720","reported_seq":1730,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.792306+0000","last_change":"2024-04-16T21:11:44.792348+0000","last_active":"2024-04-16T21:11:44.792306+0000","last_peered":"2024-04-16T21:11:44.792306+0000","last_clean":"2024-04-16T21:11:44.792306+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:11:44.792306+0000","last_undegraded":"2024-04-16T21:11:44.792306+0000","last_fullsized":"2024-04-16T21:11:44.792306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'720","last_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_deep_scrub":"52'720","last_deep_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.792244+0000","objects_scrubbed":30,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:07:06.408902+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2191,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'20","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.222406+0000","last_change":"2024-04-16T21:11:49.222443+0000","last_active":"2024-04-16T21:11:49.222406+0000","last_peered":"2024-04-16T21:11:49.222406+0000","last_clean":"2024-04-16T21:11:49.222406+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:11:49.222406+0000","last_undegraded":"2024-04-16T21:11:49.222406+0000","last_fullsized":"2024-04-16T21:11:49.222406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.222345+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:21.815371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385547+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:10:54.385547+0000","last_peered":"2024-04-16T21:10:54.385547+0000","last_clean":"2024-04-16T21:10:54.385547+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:10:54.385547+0000","last_undegraded":"2024-04-16T21:10:54.385547+0000","last_fullsized":"2024-04-16T21:10:54.385547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560100+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:11:44.560100+0000","last_peered":"2024-04-16T21:11:44.560100+0000","last_clean":"2024-04-16T21:11:44.560100+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:11:44.560100+0000","last_undegraded":"2024-04-16T21:11:44.560100+0000","last_fullsized":"2024-04-16T21:11:44.560100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385348+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:10:49.385348+0000","last_peered":"2024-04-16T21:10:49.385348+0000","last_clean":"2024-04-16T21:10:49.385348+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:10:49.385348+0000","last_undegraded":"2024-04-16T21:10:49.385348+0000","last_fullsized":"2024-04-16T21:10:49.385348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.396579+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:11:24.396579+0000","last_peered":"2024-04-16T21:11:24.396579+0000","last_clean":"2024-04-16T21:11:24.396579+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:11:24.396579+0000","last_undegraded":"2024-04-16T21:11:24.396579+0000","last_fullsized":"2024-04-16T21:11:24.396579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'132","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.386301+0000","last_change":"2024-04-16T19:24:47.422209+0000","last_active":"2024-04-16T21:10:59.386301+0000","last_peered":"2024-04-16T21:10:59.386301+0000","last_clean":"2024-04-16T21:10:59.386301+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:10:59.386301+0000","last_undegraded":"2024-04-16T21:10:59.386301+0000","last_fullsized":"2024-04-16T21:10:59.386301+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.394475+0000","last_change":"2024-04-16T21:12:09.394517+0000","last_active":"2024-04-16T21:12:09.394475+0000","last_peered":"2024-04-16T21:12:09.394475+0000","last_clean":"2024-04-16T21:12:09.394475+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:12:09.394475+0000","last_undegraded":"2024-04-16T21:12:09.394475+0000","last_fullsized":"2024-04-16T21:12:09.394475+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.394403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:58.357952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.902684+0000","last_change":"2024-04-16T21:12:14.902708+0000","last_active":"2024-04-16T21:12:14.902684+0000","last_peered":"2024-04-16T21:12:14.902684+0000","last_clean":"2024-04-16T21:12:14.902684+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:12:14.902684+0000","last_undegraded":"2024-04-16T21:12:14.902684+0000","last_fullsized":"2024-04-16T21:12:14.902684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.902624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:23.524467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561653+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:11:44.561653+0000","last_peered":"2024-04-16T21:11:44.561653+0000","last_clean":"2024-04-16T21:11:44.561653+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:11:44.561653+0000","last_undegraded":"2024-04-16T21:11:44.561653+0000","last_fullsized":"2024-04-16T21:11:44.561653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1336,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293264+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:11:42.293264+0000","last_peered":"2024-04-16T21:11:42.293264+0000","last_clean":"2024-04-16T21:11:42.293264+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:11:42.293264+0000","last_undegraded":"2024-04-16T21:11:42.293264+0000","last_fullsized":"2024-04-16T21:11:42.293264+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.392426+0000","last_change":"2024-04-16T19:25:41.511560+0000","last_active":"2024-04-16T21:11:29.392426+0000","last_peered":"2024-04-16T21:11:29.392426+0000","last_clean":"2024-04-16T21:11:29.392426+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:11:29.392426+0000","last_undegraded":"2024-04-16T21:11:29.392426+0000","last_fullsized":"2024-04-16T21:11:29.392426+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'6","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.781379+0000","last_change":"2024-04-16T21:11:54.781414+0000","last_active":"2024-04-16T21:11:54.781379+0000","last_peered":"2024-04-16T21:11:54.781379+0000","last_clean":"2024-04-16T21:11:54.781379+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:11:54.781379+0000","last_undegraded":"2024-04-16T21:11:54.781379+0000","last_fullsized":"2024-04-16T21:11:54.781379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.781318+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.682337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'706","reported_seq":2169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.804811+0000","last_change":"2024-04-16T21:11:57.804839+0000","last_active":"2024-04-16T21:11:57.804811+0000","last_peered":"2024-04-16T21:11:57.804811+0000","last_clean":"2024-04-16T21:11:57.804811+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:11:57.804811+0000","last_undegraded":"2024-04-16T21:11:57.804811+0000","last_fullsized":"2024-04-16T21:11:57.804811+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.804739+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:38:55.231266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'264","reported_seq":495,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.746009+0000","last_change":"2024-04-16T21:11:46.746078+0000","last_active":"2024-04-16T21:11:46.746009+0000","last_peered":"2024-04-16T21:11:46.746009+0000","last_clean":"2024-04-16T21:11:46.746009+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:11:46.746009+0000","last_undegraded":"2024-04-16T21:11:46.746009+0000","last_fullsized":"2024-04-16T21:11:46.746009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.745946+0000","objects_scrubbed":17,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:43.933138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1751,"num_omap_keys":8,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'38","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.693339+0000","last_change":"2024-04-16T21:11:52.693388+0000","last_active":"2024-04-16T21:11:52.693339+0000","last_peered":"2024-04-16T21:11:52.693339+0000","last_clean":"2024-04-16T21:11:52.693339+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:11:52.693339+0000","last_undegraded":"2024-04-16T21:11:52.693339+0000","last_fullsized":"2024-04-16T21:11:52.693339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'38","last_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_deep_scrub":"52'38","last_deep_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.693273+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:15.884668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385497+0000","last_change":"2024-04-16T19:24:35.402034+0000","last_active":"2024-04-16T21:10:54.385497+0000","last_peered":"2024-04-16T21:10:54.385497+0000","last_clean":"2024-04-16T21:10:54.385497+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:10:54.385497+0000","last_undegraded":"2024-04-16T21:10:54.385497+0000","last_fullsized":"2024-04-16T21:10:54.385497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388567+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:11:09.388567+0000","last_peered":"2024-04-16T21:11:09.388567+0000","last_clean":"2024-04-16T21:11:09.388567+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:11:09.388567+0000","last_undegraded":"2024-04-16T21:11:09.388567+0000","last_fullsized":"2024-04-16T21:11:09.388567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742796+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:11:42.742796+0000","last_peered":"2024-04-16T21:11:42.742796+0000","last_clean":"2024-04-16T21:11:42.742796+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:11:42.742796+0000","last_undegraded":"2024-04-16T21:11:42.742796+0000","last_fullsized":"2024-04-16T21:11:42.742796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393177+0000","last_change":"2024-04-16T19:24:45.414277+0000","last_active":"2024-04-16T21:11:04.393177+0000","last_peered":"2024-04-16T21:11:04.393177+0000","last_clean":"2024-04-16T21:11:04.393177+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:11:04.393177+0000","last_undegraded":"2024-04-16T21:11:04.393177+0000","last_fullsized":"2024-04-16T21:11:04.393177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'148","reported_seq":350,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393850+0000","last_change":"2024-04-16T19:24:47.422079+0000","last_active":"2024-04-16T21:11:09.393850+0000","last_peered":"2024-04-16T21:11:09.393850+0000","last_clean":"2024-04-16T21:11:09.393850+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:11:09.393850+0000","last_undegraded":"2024-04-16T21:11:09.393850+0000","last_fullsized":"2024-04-16T21:11:09.393850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387828+0000","last_change":"2024-04-16T19:24:49.425384+0000","last_active":"2024-04-16T21:11:04.387828+0000","last_peered":"2024-04-16T21:11:04.387828+0000","last_clean":"2024-04-16T21:11:04.387828+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:11:04.387828+0000","last_undegraded":"2024-04-16T21:11:04.387828+0000","last_fullsized":"2024-04-16T21:11:04.387828+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.345372+0000","last_change":"2024-04-16T21:12:01.345418+0000","last_active":"2024-04-16T21:12:01.345372+0000","last_peered":"2024-04-16T21:12:01.345372+0000","last_clean":"2024-04-16T21:12:01.345372+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:12:01.345372+0000","last_undegraded":"2024-04-16T21:12:01.345372+0000","last_fullsized":"2024-04-16T21:12:01.345372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.345289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:22.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195947+0000","last_change":"2024-04-16T19:47:33.985478+0000","last_active":"2024-04-16T21:11:43.195947+0000","last_peered":"2024-04-16T21:11:43.195947+0000","last_clean":"2024-04-16T21:11:43.195947+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:11:43.195947+0000","last_undegraded":"2024-04-16T21:11:43.195947+0000","last_fullsized":"2024-04-16T21:11:43.195947+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'147","reported_seq":2004,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561557+0000","last_change":"2024-04-16T19:25:49.510845+0000","last_active":"2024-04-16T21:11:44.561557+0000","last_peered":"2024-04-16T21:11:44.561557+0000","last_clean":"2024-04-16T21:11:44.561557+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:11:44.561557+0000","last_undegraded":"2024-04-16T21:11:44.561557+0000","last_fullsized":"2024-04-16T21:11:44.561557+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.836131+0000","last_change":"2024-04-16T21:12:08.836131+0000","last_active":"2024-04-16T21:12:08.836131+0000","last_peered":"2024-04-16T21:12:08.836131+0000","last_clean":"2024-04-16T21:12:08.836131+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:12:08.836131+0000","last_undegraded":"2024-04-16T21:12:08.836131+0000","last_fullsized":"2024-04-16T21:12:08.836131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.836015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:33.006060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'30","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.417236+0000","last_change":"2024-04-16T21:12:16.417277+0000","last_active":"2024-04-16T21:12:16.417236+0000","last_peered":"2024-04-16T21:12:16.417236+0000","last_clean":"2024-04-16T21:12:16.417236+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:12:16.417236+0000","last_undegraded":"2024-04-16T21:12:16.417236+0000","last_fullsized":"2024-04-16T21:12:16.417236+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'30","last_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_deep_scrub":"52'30","last_deep_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.417171+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:08.090860+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.802564+0000","last_change":"2024-04-16T21:12:02.802585+0000","last_active":"2024-04-16T21:12:02.802564+0000","last_peered":"2024-04-16T21:12:02.802564+0000","last_clean":"2024-04-16T21:12:02.802564+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:12:02.802564+0000","last_undegraded":"2024-04-16T21:12:02.802564+0000","last_fullsized":"2024-04-16T21:12:02.802564+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.802497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:44.917953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'739","reported_seq":1609,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.752466+0000","last_change":"2024-04-16T21:11:47.752492+0000","last_active":"2024-04-16T21:11:47.752466+0000","last_peered":"2024-04-16T21:11:47.752466+0000","last_clean":"2024-04-16T21:11:47.752466+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:11:47.752466+0000","last_undegraded":"2024-04-16T21:11:47.752466+0000","last_fullsized":"2024-04-16T21:11:47.752466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'739","last_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_deep_scrub":"52'739","last_deep_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.752406+0000","objects_scrubbed":25,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:14.225249+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6099,"num_omap_keys":28,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'12","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.362460+0000","last_change":"2024-04-16T21:11:45.362519+0000","last_active":"2024-04-16T21:11:45.362460+0000","last_peered":"2024-04-16T21:11:45.362460+0000","last_clean":"2024-04-16T21:11:45.362460+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:11:45.362460+0000","last_undegraded":"2024-04-16T21:11:45.362460+0000","last_fullsized":"2024-04-16T21:11:45.362460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.362399+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:08.553795+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391347+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:10:54.391347+0000","last_peered":"2024-04-16T21:10:54.391347+0000","last_clean":"2024-04-16T21:10:54.391347+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:10:54.391347+0000","last_undegraded":"2024-04-16T21:10:54.391347+0000","last_fullsized":"2024-04-16T21:10:54.391347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384586+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:10:49.384586+0000","last_peered":"2024-04-16T21:10:49.384586+0000","last_clean":"2024-04-16T21:10:49.384586+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:10:49.384586+0000","last_undegraded":"2024-04-16T21:10:49.384586+0000","last_fullsized":"2024-04-16T21:10:49.384586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381733+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:10:24.381733+0000","last_peered":"2024-04-16T21:10:24.381733+0000","last_clean":"2024-04-16T21:10:24.381733+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:10:24.381733+0000","last_undegraded":"2024-04-16T21:10:24.381733+0000","last_fullsized":"2024-04-16T21:10:24.381733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387738+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:11:04.387738+0000","last_peered":"2024-04-16T21:11:04.387738+0000","last_clean":"2024-04-16T21:11:04.387738+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:11:04.387738+0000","last_undegraded":"2024-04-16T21:11:04.387738+0000","last_fullsized":"2024-04-16T21:11:04.387738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'154","reported_seq":347,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387067+0000","last_change":"2024-04-16T19:24:47.424720+0000","last_active":"2024-04-16T21:11:04.387067+0000","last_peered":"2024-04-16T21:11:04.387067+0000","last_clean":"2024-04-16T21:11:04.387067+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:11:04.387067+0000","last_undegraded":"2024-04-16T21:11:04.387067+0000","last_fullsized":"2024-04-16T21:11:04.387067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'12","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388421+0000","last_change":"2024-04-16T19:24:49.433872+0000","last_active":"2024-04-16T21:11:09.388421+0000","last_peered":"2024-04-16T21:11:09.388421+0000","last_clean":"2024-04-16T21:11:09.388421+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:11:09.388421+0000","last_undegraded":"2024-04-16T21:11:09.388421+0000","last_fullsized":"2024-04-16T21:11:09.388421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.393316+0000","last_change":"2024-04-16T21:12:00.393341+0000","last_active":"2024-04-16T21:12:00.393316+0000","last_peered":"2024-04-16T21:12:00.393316+0000","last_clean":"2024-04-16T21:12:00.393316+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:12:00.393316+0000","last_undegraded":"2024-04-16T21:12:00.393316+0000","last_fullsized":"2024-04-16T21:12:00.393316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.393227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:57.237156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524744+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:10:49.524744+0000","last_peered":"2024-04-16T21:10:49.524744+0000","last_clean":"2024-04-16T21:10:49.524744+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:10:49.524744+0000","last_undegraded":"2024-04-16T21:10:49.524744+0000","last_fullsized":"2024-04-16T21:10:49.524744+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'116","reported_seq":1528,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651445+0000","last_change":"2024-04-16T19:25:49.515636+0000","last_active":"2024-04-16T21:11:43.651445+0000","last_peered":"2024-04-16T21:11:43.651445+0000","last_clean":"2024-04-16T21:11:43.651445+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:11:43.651445+0000","last_undegraded":"2024-04-16T21:11:43.651445+0000","last_fullsized":"2024-04-16T21:11:43.651445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.939107+0000","last_change":"2024-04-16T21:12:19.939131+0000","last_active":"2024-04-16T21:12:19.939107+0000","last_peered":"2024-04-16T21:12:19.939107+0000","last_clean":"2024-04-16T21:12:19.939107+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:12:19.939107+0000","last_undegraded":"2024-04-16T21:12:19.939107+0000","last_fullsized":"2024-04-16T21:12:19.939107+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.939043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:23.022714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'4","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.763978+0000","last_change":"2024-04-16T21:11:55.764028+0000","last_active":"2024-04-16T21:11:55.763978+0000","last_peered":"2024-04-16T21:11:55.763978+0000","last_clean":"2024-04-16T21:11:55.763978+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:11:55.763978+0000","last_undegraded":"2024-04-16T21:11:55.763978+0000","last_fullsized":"2024-04-16T21:11:55.763978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.763887+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.184478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.416929+0000","last_change":"2024-04-16T21:12:12.416959+0000","last_active":"2024-04-16T21:12:12.416929+0000","last_peered":"2024-04-16T21:12:12.416929+0000","last_clean":"2024-04-16T21:12:12.416929+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:12:12.416929+0000","last_undegraded":"2024-04-16T21:12:12.416929+0000","last_fullsized":"2024-04-16T21:12:12.416929+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.416865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:15.500537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'287","reported_seq":553,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.658905+0000","last_change":"2024-04-16T21:11:54.658986+0000","last_active":"2024-04-16T21:11:54.658905+0000","last_peered":"2024-04-16T21:11:54.658905+0000","last_clean":"2024-04-16T21:11:54.658905+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:11:54.658905+0000","last_undegraded":"2024-04-16T21:11:54.658905+0000","last_fullsized":"2024-04-16T21:11:54.658905+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'287","last_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_deep_scrub":"52'287","last_deep_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.658842+0000","objects_scrubbed":20,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.559862+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6346,"num_omap_keys":29,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'22","reported_seq":213,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.414533+0000","last_change":"2024-04-16T21:11:51.414602+0000","last_active":"2024-04-16T21:11:51.414533+0000","last_peered":"2024-04-16T21:11:51.414533+0000","last_clean":"2024-04-16T21:11:51.414533+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:11:51.414533+0000","last_undegraded":"2024-04-16T21:11:51.414533+0000","last_fullsized":"2024-04-16T21:11:51.414533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.414471+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:12.853053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392142+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:10:59.392142+0000","last_peered":"2024-04-16T21:10:59.392142+0000","last_clean":"2024-04-16T21:10:59.392142+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:10:59.392142+0000","last_undegraded":"2024-04-16T21:10:59.392142+0000","last_fullsized":"2024-04-16T21:10:59.392142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.524015+0000","last_change":"2024-04-16T19:24:33.386068+0000","last_active":"2024-04-16T21:10:44.524015+0000","last_peered":"2024-04-16T21:10:44.524015+0000","last_clean":"2024-04-16T21:10:44.524015+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:10:44.524015+0000","last_undegraded":"2024-04-16T21:10:44.524015+0000","last_fullsized":"2024-04-16T21:10:44.524015+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742796+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:11:42.742796+0000","last_peered":"2024-04-16T21:11:42.742796+0000","last_clean":"2024-04-16T21:11:42.742796+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:11:42.742796+0000","last_undegraded":"2024-04-16T21:11:42.742796+0000","last_fullsized":"2024-04-16T21:11:42.742796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388551+0000","last_change":"2024-04-16T19:24:45.413345+0000","last_active":"2024-04-16T21:11:04.388551+0000","last_peered":"2024-04-16T21:11:04.388551+0000","last_clean":"2024-04-16T21:11:04.388551+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:11:04.388551+0000","last_undegraded":"2024-04-16T21:11:04.388551+0000","last_fullsized":"2024-04-16T21:11:04.388551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'110","reported_seq":281,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388599+0000","last_change":"2024-04-16T19:24:47.417633+0000","last_active":"2024-04-16T21:11:04.388599+0000","last_peered":"2024-04-16T21:11:04.388599+0000","last_clean":"2024-04-16T21:11:04.388599+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:11:04.388599+0000","last_undegraded":"2024-04-16T21:11:04.388599+0000","last_fullsized":"2024-04-16T21:11:04.388599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'12","reported_seq":1316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.393149+0000","last_change":"2024-04-16T19:24:49.432613+0000","last_active":"2024-04-16T21:11:04.393149+0000","last_peered":"2024-04-16T21:11:04.393149+0000","last_clean":"2024-04-16T21:11:04.393149+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:11:04.393149+0000","last_undegraded":"2024-04-16T21:11:04.393149+0000","last_fullsized":"2024-04-16T21:11:04.393149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.373196+0000","last_change":"2024-04-16T21:11:59.373240+0000","last_active":"2024-04-16T21:11:59.373196+0000","last_peered":"2024-04-16T21:11:59.373196+0000","last_clean":"2024-04-16T21:11:59.373196+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:11:59.373196+0000","last_undegraded":"2024-04-16T21:11:59.373196+0000","last_fullsized":"2024-04-16T21:11:59.373196+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.373112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:02.682886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.196003+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:11:43.196003+0000","last_peered":"2024-04-16T21:11:43.196003+0000","last_clean":"2024-04-16T21:11:43.196003+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:11:43.196003+0000","last_undegraded":"2024-04-16T21:11:43.196003+0000","last_fullsized":"2024-04-16T21:11:43.196003+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1522,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:14.380044+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:10:14.380044+0000","last_peered":"2024-04-16T21:10:14.380044+0000","last_clean":"2024-04-16T21:10:14.380044+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:10:14.380044+0000","last_undegraded":"2024-04-16T21:10:14.380044+0000","last_fullsized":"2024-04-16T21:10:14.380044+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.846610+0000","last_change":"2024-04-16T21:12:00.846636+0000","last_active":"2024-04-16T21:12:00.846610+0000","last_peered":"2024-04-16T21:12:00.846610+0000","last_clean":"2024-04-16T21:12:00.846610+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:12:00.846610+0000","last_undegraded":"2024-04-16T21:12:00.846610+0000","last_fullsized":"2024-04-16T21:12:00.846610+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.846545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:04.784544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'15","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.794385+0000","last_change":"2024-04-16T21:11:56.794414+0000","last_active":"2024-04-16T21:11:56.794385+0000","last_peered":"2024-04-16T21:11:56.794385+0000","last_clean":"2024-04-16T21:11:56.794385+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:11:56.794385+0000","last_undegraded":"2024-04-16T21:11:56.794385+0000","last_fullsized":"2024-04-16T21:11:56.794385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.794323+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.217325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'706","reported_seq":2090,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.681698+0000","last_change":"2024-04-16T21:11:49.681723+0000","last_active":"2024-04-16T21:11:49.681698+0000","last_peered":"2024-04-16T21:11:49.681698+0000","last_clean":"2024-04-16T21:11:49.681698+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:11:49.681698+0000","last_undegraded":"2024-04-16T21:11:49.681698+0000","last_fullsized":"2024-04-16T21:11:49.681698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.681634+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.077508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'335","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.410155+0000","last_change":"2024-04-16T21:11:50.410223+0000","last_active":"2024-04-16T21:11:50.410155+0000","last_peered":"2024-04-16T21:11:50.410155+0000","last_clean":"2024-04-16T21:11:50.410155+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:11:50.410155+0000","last_undegraded":"2024-04-16T21:11:50.410155+0000","last_fullsized":"2024-04-16T21:11:50.410155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'335","last_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_deep_scrub":"52'335","last_deep_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.410083+0000","objects_scrubbed":17,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:14.580127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'10","reported_seq":195,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.912474+0000","last_change":"2024-04-16T21:12:10.912515+0000","last_active":"2024-04-16T21:12:10.912474+0000","last_peered":"2024-04-16T21:12:10.912474+0000","last_clean":"2024-04-16T21:12:10.912474+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:12:10.912474+0000","last_undegraded":"2024-04-16T21:12:10.912474+0000","last_fullsized":"2024-04-16T21:12:10.912474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.912417+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.912619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386268+0000","last_change":"2024-04-16T19:24:35.401425+0000","last_active":"2024-04-16T21:10:54.386268+0000","last_peered":"2024-04-16T21:10:54.386268+0000","last_clean":"2024-04-16T21:10:54.386268+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:10:54.386268+0000","last_undegraded":"2024-04-16T21:10:54.386268+0000","last_fullsized":"2024-04-16T21:10:54.386268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.389709+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:10:44.389709+0000","last_peered":"2024-04-16T21:10:44.389709+0000","last_clean":"2024-04-16T21:10:44.389709+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:10:44.389709+0000","last_undegraded":"2024-04-16T21:10:44.389709+0000","last_fullsized":"2024-04-16T21:10:44.389709+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649924+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:11:43.649924+0000","last_peered":"2024-04-16T21:11:43.649924+0000","last_clean":"2024-04-16T21:11:43.649924+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:11:43.649924+0000","last_undegraded":"2024-04-16T21:11:43.649924+0000","last_fullsized":"2024-04-16T21:11:43.649924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527115+0000","last_change":"2024-04-16T19:24:45.409120+0000","last_active":"2024-04-16T21:11:04.527115+0000","last_peered":"2024-04-16T21:11:04.527115+0000","last_clean":"2024-04-16T21:11:04.527115+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:11:04.527115+0000","last_undegraded":"2024-04-16T21:11:04.527115+0000","last_fullsized":"2024-04-16T21:11:04.527115+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'134","reported_seq":332,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387885+0000","last_change":"2024-04-16T19:24:47.425725+0000","last_active":"2024-04-16T21:11:04.387885+0000","last_peered":"2024-04-16T21:11:04.387885+0000","last_clean":"2024-04-16T21:11:04.387885+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:11:04.387885+0000","last_undegraded":"2024-04-16T21:11:04.387885+0000","last_fullsized":"2024-04-16T21:11:04.387885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.903416+0000","last_change":"2024-04-16T21:12:17.903439+0000","last_active":"2024-04-16T21:12:17.903416+0000","last_peered":"2024-04-16T21:12:17.903416+0000","last_clean":"2024-04-16T21:12:17.903416+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:12:17.903416+0000","last_undegraded":"2024-04-16T21:12:17.903416+0000","last_fullsized":"2024-04-16T21:12:17.903416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.903352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:24.923396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.864530+0000","last_change":"2024-04-16T21:12:11.864551+0000","last_active":"2024-04-16T21:12:11.864530+0000","last_peered":"2024-04-16T21:12:11.864530+0000","last_clean":"2024-04-16T21:12:11.864530+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:12:11.864530+0000","last_undegraded":"2024-04-16T21:12:11.864530+0000","last_fullsized":"2024-04-16T21:12:11.864530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.864468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:37.984544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293397+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:11:42.293397+0000","last_peered":"2024-04-16T21:11:42.293397+0000","last_clean":"2024-04-16T21:11:42.293397+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:11:42.293397+0000","last_undegraded":"2024-04-16T21:11:42.293397+0000","last_fullsized":"2024-04-16T21:11:42.293397+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1570,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378464+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:10:04.378464+0000","last_peered":"2024-04-16T21:10:04.378464+0000","last_clean":"2024-04-16T21:10:04.378464+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:10:04.378464+0000","last_undegraded":"2024-04-16T21:10:04.378464+0000","last_fullsized":"2024-04-16T21:10:04.378464+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.869006+0000","last_change":"2024-04-16T21:12:06.869029+0000","last_active":"2024-04-16T21:12:06.869006+0000","last_peered":"2024-04-16T21:12:06.869006+0000","last_clean":"2024-04-16T21:12:06.869006+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:12:06.869006+0000","last_undegraded":"2024-04-16T21:12:06.869006+0000","last_fullsized":"2024-04-16T21:12:06.869006+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.868938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:10.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'8","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.650032+0000","last_change":"2024-04-16T21:12:10.650063+0000","last_active":"2024-04-16T21:12:10.650032+0000","last_peered":"2024-04-16T21:12:10.650032+0000","last_clean":"2024-04-16T21:12:10.650032+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:12:10.650032+0000","last_undegraded":"2024-04-16T21:12:10.650032+0000","last_fullsized":"2024-04-16T21:12:10.650032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.649972+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.650174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.662892+0000","last_change":"2024-04-16T21:11:53.662915+0000","last_active":"2024-04-16T21:11:53.662892+0000","last_peered":"2024-04-16T21:11:53.662892+0000","last_clean":"2024-04-16T21:11:53.662892+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:11:53.662892+0000","last_undegraded":"2024-04-16T21:11:53.662892+0000","last_fullsized":"2024-04-16T21:11:53.662892+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.662831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:53.840637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'317","reported_seq":612,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.662747+0000","last_change":"2024-04-16T21:11:52.662773+0000","last_active":"2024-04-16T21:11:52.662747+0000","last_peered":"2024-04-16T21:11:52.662747+0000","last_clean":"2024-04-16T21:11:52.662747+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:11:52.662747+0000","last_undegraded":"2024-04-16T21:11:52.662747+0000","last_fullsized":"2024-04-16T21:11:52.662747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'317","last_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_deep_scrub":"52'317","last_deep_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.662688+0000","objects_scrubbed":25,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:03:07.759388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3966,"num_omap_keys":18,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'20","reported_seq":253,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.170966+0000","last_change":"2024-04-16T21:11:53.171045+0000","last_active":"2024-04-16T21:11:53.170966+0000","last_peered":"2024-04-16T21:11:53.170966+0000","last_clean":"2024-04-16T21:11:53.170966+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:11:53.170966+0000","last_undegraded":"2024-04-16T21:11:53.170966+0000","last_fullsized":"2024-04-16T21:11:53.170966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.170891+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:56.498676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.474986+0000","last_change":"2024-04-16T21:12:09.475030+0000","last_active":"2024-04-16T21:12:09.474986+0000","last_peered":"2024-04-16T21:12:09.474986+0000","last_clean":"2024-04-16T21:12:09.474986+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:12:09.474986+0000","last_undegraded":"2024-04-16T21:12:09.474986+0000","last_fullsized":"2024-04-16T21:12:09.474986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.474907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:12.558579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106369+0000","last_change":"2024-04-16T19:24:35.398901+0000","last_active":"2024-04-16T21:11:44.106369+0000","last_peered":"2024-04-16T21:11:44.106369+0000","last_clean":"2024-04-16T21:11:44.106369+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:11:44.106369+0000","last_undegraded":"2024-04-16T21:11:44.106369+0000","last_fullsized":"2024-04-16T21:11:44.106369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.890729+0000","last_change":"2024-04-16T21:12:07.890754+0000","last_active":"2024-04-16T21:12:07.890729+0000","last_peered":"2024-04-16T21:12:07.890729+0000","last_clean":"2024-04-16T21:12:07.890729+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:12:07.890729+0000","last_undegraded":"2024-04-16T21:12:07.890729+0000","last_fullsized":"2024-04-16T21:12:07.890729+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.890665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:11.218450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":745,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743215+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:11:42.743215+0000","last_peered":"2024-04-16T21:11:42.743215+0000","last_clean":"2024-04-16T21:11:42.743215+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:11:42.743215+0000","last_undegraded":"2024-04-16T21:11:42.743215+0000","last_fullsized":"2024-04-16T21:11:42.743215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'96","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195249+0000","last_change":"2024-04-16T19:24:47.422106+0000","last_active":"2024-04-16T21:11:43.195249+0000","last_peered":"2024-04-16T21:11:43.195249+0000","last_clean":"2024-04-16T21:11:43.195249+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:11:43.195249+0000","last_undegraded":"2024-04-16T21:11:43.195249+0000","last_fullsized":"2024-04-16T21:11:43.195249+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.902196+0000","last_change":"2024-04-16T21:12:21.902219+0000","last_active":"2024-04-16T21:12:21.902196+0000","last_peered":"2024-04-16T21:12:21.902196+0000","last_clean":"2024-04-16T21:12:21.902196+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:12:21.902196+0000","last_undegraded":"2024-04-16T21:12:21.902196+0000","last_fullsized":"2024-04-16T21:12:21.902196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.902128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:26.941072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107591+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:11:44.107591+0000","last_peered":"2024-04-16T21:11:44.107591+0000","last_clean":"2024-04-16T21:11:44.107591+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:11:44.107591+0000","last_undegraded":"2024-04-16T21:11:44.107591+0000","last_fullsized":"2024-04-16T21:11:44.107591+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'159","reported_seq":1763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.889853+0000","last_change":"2024-04-16T21:12:18.889893+0000","last_active":"2024-04-16T21:12:18.889853+0000","last_peered":"2024-04-16T21:12:18.889853+0000","last_clean":"2024-04-16T21:12:18.889853+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:12:18.889853+0000","last_undegraded":"2024-04-16T21:12:18.889853+0000","last_fullsized":"2024-04-16T21:12:18.889853+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.889790+0000","objects_scrubbed":41,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:35.275723+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12681,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.752918+0000","last_change":"2024-04-16T21:12:01.752964+0000","last_active":"2024-04-16T21:12:01.752918+0000","last_peered":"2024-04-16T21:12:01.752918+0000","last_clean":"2024-04-16T21:12:01.752918+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:12:01.752918+0000","last_undegraded":"2024-04-16T21:12:01.752918+0000","last_fullsized":"2024-04-16T21:12:01.752918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.752809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.500440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.625386+0000","last_change":"2024-04-16T21:12:04.625407+0000","last_active":"2024-04-16T21:12:04.625386+0000","last_peered":"2024-04-16T21:12:04.625386+0000","last_clean":"2024-04-16T21:12:04.625386+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:12:04.625386+0000","last_undegraded":"2024-04-16T21:12:04.625386+0000","last_fullsized":"2024-04-16T21:12:04.625386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'2","last_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_deep_scrub":"45'2","last_deep_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.625306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.449220+0000","last_change":"2024-04-16T21:12:04.449242+0000","last_active":"2024-04-16T21:12:04.449220+0000","last_peered":"2024-04-16T21:12:04.449220+0000","last_clean":"2024-04-16T21:12:04.449220+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:12:04.449220+0000","last_undegraded":"2024-04-16T21:12:04.449220+0000","last_fullsized":"2024-04-16T21:12:04.449220+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.449160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:30.569236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1163","reported_seq":2930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.718441+0000","last_change":"2024-04-16T21:11:46.718512+0000","last_active":"2024-04-16T21:11:46.718441+0000","last_peered":"2024-04-16T21:11:46.718441+0000","last_clean":"2024-04-16T21:11:46.718441+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:11:46.718441+0000","last_undegraded":"2024-04-16T21:11:46.718441+0000","last_fullsized":"2024-04-16T21:11:46.718441+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1163","last_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_deep_scrub":"52'1163","last_deep_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.718377+0000","objects_scrubbed":23,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:06.764370+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'16","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.594571+0000","last_change":"2024-04-16T21:11:51.594618+0000","last_active":"2024-04-16T21:11:51.594571+0000","last_peered":"2024-04-16T21:11:51.594571+0000","last_clean":"2024-04-16T21:11:51.594571+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:11:51.594571+0000","last_undegraded":"2024-04-16T21:11:51.594571+0000","last_fullsized":"2024-04-16T21:11:51.594571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.594510+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:52.095893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.782064+0000","last_change":"2024-04-16T21:12:01.782087+0000","last_active":"2024-04-16T21:12:01.782064+0000","last_peered":"2024-04-16T21:12:01.782064+0000","last_clean":"2024-04-16T21:12:01.782064+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:12:01.782064+0000","last_undegraded":"2024-04-16T21:12:01.782064+0000","last_fullsized":"2024-04-16T21:12:01.782064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.781996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:45.434631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743093+0000","last_change":"2024-04-16T19:24:35.401716+0000","last_active":"2024-04-16T21:11:42.743093+0000","last_peered":"2024-04-16T21:11:42.743093+0000","last_clean":"2024-04-16T21:11:42.743093+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:11:42.743093+0000","last_undegraded":"2024-04-16T21:11:42.743093+0000","last_fullsized":"2024-04-16T21:11:42.743093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386872+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:10:54.386872+0000","last_peered":"2024-04-16T21:10:54.386872+0000","last_clean":"2024-04-16T21:10:54.386872+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:10:54.386872+0000","last_undegraded":"2024-04-16T21:10:54.386872+0000","last_fullsized":"2024-04-16T21:10:54.386872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743211+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:11:42.743211+0000","last_peered":"2024-04-16T21:11:42.743211+0000","last_clean":"2024-04-16T21:11:42.743211+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:11:42.743211+0000","last_undegraded":"2024-04-16T21:11:42.743211+0000","last_fullsized":"2024-04-16T21:11:42.743211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'46","reported_seq":179,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292383+0000","last_change":"2024-04-16T19:24:47.419319+0000","last_active":"2024-04-16T21:11:42.292383+0000","last_peered":"2024-04-16T21:11:42.292383+0000","last_clean":"2024-04-16T21:11:42.292383+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:11:42.292383+0000","last_undegraded":"2024-04-16T21:11:42.292383+0000","last_fullsized":"2024-04-16T21:11:42.292383+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.423193+0000","last_change":"2024-04-16T21:12:10.423217+0000","last_active":"2024-04-16T21:12:10.423193+0000","last_peered":"2024-04-16T21:12:10.423193+0000","last_clean":"2024-04-16T21:12:10.423193+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:12:10.423193+0000","last_undegraded":"2024-04-16T21:12:10.423193+0000","last_fullsized":"2024-04-16T21:12:10.423193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.423128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:15.462072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":50,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378320+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:10:04.378320+0000","last_peered":"2024-04-16T21:10:04.378320+0000","last_clean":"2024-04-16T21:10:04.378320+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:10:04.378320+0000","last_undegraded":"2024-04-16T21:10:04.378320+0000","last_fullsized":"2024-04-16T21:10:04.378320+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'130","reported_seq":1704,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384537+0000","last_change":"2024-04-16T19:25:49.524521+0000","last_active":"2024-04-16T21:10:49.384537+0000","last_peered":"2024-04-16T21:10:49.384537+0000","last_clean":"2024-04-16T21:10:49.384537+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:10:49.384537+0000","last_undegraded":"2024-04-16T21:10:49.384537+0000","last_fullsized":"2024-04-16T21:10:49.384537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.920212+0000","last_change":"2024-04-16T21:12:18.920252+0000","last_active":"2024-04-16T21:12:18.920212+0000","last_peered":"2024-04-16T21:12:18.920212+0000","last_clean":"2024-04-16T21:12:18.920212+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:12:18.920212+0000","last_undegraded":"2024-04-16T21:12:18.920212+0000","last_fullsized":"2024-04-16T21:12:18.920212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.920121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:25:34.146310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'26","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.477899+0000","last_change":"2024-04-16T21:12:19.477941+0000","last_active":"2024-04-16T21:12:19.477899+0000","last_peered":"2024-04-16T21:12:19.477899+0000","last_clean":"2024-04-16T21:12:19.477899+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:12:19.477899+0000","last_undegraded":"2024-04-16T21:12:19.477899+0000","last_fullsized":"2024-04-16T21:12:19.477899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.477827+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:13.666501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.766206+0000","last_change":"2024-04-16T21:12:09.766229+0000","last_active":"2024-04-16T21:12:09.766206+0000","last_peered":"2024-04-16T21:12:09.766206+0000","last_clean":"2024-04-16T21:12:09.766206+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:12:09.766206+0000","last_undegraded":"2024-04-16T21:12:09.766206+0000","last_fullsized":"2024-04-16T21:12:09.766206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.766140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:51.548954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'291","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.385570+0000","last_change":"2024-04-16T21:11:52.385621+0000","last_active":"2024-04-16T21:11:52.385570+0000","last_peered":"2024-04-16T21:11:52.385570+0000","last_clean":"2024-04-16T21:11:52.385570+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:11:52.385570+0000","last_undegraded":"2024-04-16T21:11:52.385570+0000","last_fullsized":"2024-04-16T21:11:52.385570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'291","last_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_deep_scrub":"52'291","last_deep_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.385512+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:13.385713+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'16","reported_seq":262,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.131892+0000","last_change":"2024-04-16T21:11:52.131942+0000","last_active":"2024-04-16T21:11:52.131892+0000","last_peered":"2024-04-16T21:11:52.131892+0000","last_clean":"2024-04-16T21:11:52.131892+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:11:52.131892+0000","last_undegraded":"2024-04-16T21:11:52.131892+0000","last_fullsized":"2024-04-16T21:11:52.131892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.131829+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:55.304717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.441339+0000","last_change":"2024-04-16T21:12:08.441339+0000","last_active":"2024-04-16T21:12:08.441339+0000","last_peered":"2024-04-16T21:12:08.441339+0000","last_clean":"2024-04-16T21:12:08.441339+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:12:08.441339+0000","last_undegraded":"2024-04-16T21:12:08.441339+0000","last_fullsized":"2024-04-16T21:12:08.441339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.441251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:03.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194933+0000","last_change":"2024-04-16T19:24:35.394167+0000","last_active":"2024-04-16T21:11:43.194933+0000","last_peered":"2024-04-16T21:11:43.194933+0000","last_clean":"2024-04-16T21:11:43.194933+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:11:43.194933+0000","last_undegraded":"2024-04-16T21:11:43.194933+0000","last_fullsized":"2024-04-16T21:11:43.194933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386323+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:10:54.386323+0000","last_peered":"2024-04-16T21:10:54.386323+0000","last_clean":"2024-04-16T21:10:54.386323+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:10:54.386323+0000","last_undegraded":"2024-04-16T21:10:54.386323+0000","last_fullsized":"2024-04-16T21:10:54.386323+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'12","reported_seq":613,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106580+0000","last_change":"2024-04-16T19:24:45.410534+0000","last_active":"2024-04-16T21:11:44.106580+0000","last_peered":"2024-04-16T21:11:44.106580+0000","last_clean":"2024-04-16T21:11:44.106580+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:11:44.106580+0000","last_undegraded":"2024-04-16T21:11:44.106580+0000","last_fullsized":"2024-04-16T21:11:44.106580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.893687+0000","last_change":"2024-04-16T21:12:23.893724+0000","last_active":"2024-04-16T21:12:23.893687+0000","last_peered":"2024-04-16T21:12:23.893687+0000","last_clean":"2024-04-16T21:12:23.893687+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:12:23.893687+0000","last_undegraded":"2024-04-16T21:12:23.893687+0000","last_fullsized":"2024-04-16T21:12:23.893687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.893616+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:17.500596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560696+0000","last_change":"2024-04-16T19:24:49.426662+0000","last_active":"2024-04-16T21:11:44.560696+0000","last_peered":"2024-04-16T21:11:44.560696+0000","last_clean":"2024-04-16T21:11:44.560696+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:11:44.560696+0000","last_undegraded":"2024-04-16T21:11:44.560696+0000","last_fullsized":"2024-04-16T21:11:44.560696+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":55,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:04.378544+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:10:04.378544+0000","last_peered":"2024-04-16T21:10:04.378544+0000","last_clean":"2024-04-16T21:10:04.378544+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:10:04.378544+0000","last_undegraded":"2024-04-16T21:10:04.378544+0000","last_fullsized":"2024-04-16T21:10:04.378544+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'88","reported_seq":1143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390537+0000","last_change":"2024-04-16T19:25:49.513235+0000","last_active":"2024-04-16T21:10:49.390537+0000","last_peered":"2024-04-16T21:10:49.390537+0000","last_clean":"2024-04-16T21:10:49.390537+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:10:49.390537+0000","last_undegraded":"2024-04-16T21:10:49.390537+0000","last_fullsized":"2024-04-16T21:10:49.390537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.867629+0000","last_change":"2024-04-16T21:12:04.867629+0000","last_active":"2024-04-16T21:12:04.867629+0000","last_peered":"2024-04-16T21:12:04.867629+0000","last_clean":"2024-04-16T21:12:04.867629+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:12:04.867629+0000","last_undegraded":"2024-04-16T21:12:04.867629+0000","last_fullsized":"2024-04-16T21:12:04.867629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.867541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.770932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'31","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.873726+0000","last_change":"2024-04-16T21:12:11.873787+0000","last_active":"2024-04-16T21:12:11.873726+0000","last_peered":"2024-04-16T21:12:11.873726+0000","last_clean":"2024-04-16T21:12:11.873726+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:12:11.873726+0000","last_undegraded":"2024-04-16T21:12:11.873726+0000","last_fullsized":"2024-04-16T21:12:11.873726+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'31","last_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_deep_scrub":"52'31","last_deep_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.873657+0000","objects_scrubbed":9,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:04.053968+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'706","reported_seq":2151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.887851+0000","last_change":"2024-04-16T21:12:02.887901+0000","last_active":"2024-04-16T21:12:02.887851+0000","last_peered":"2024-04-16T21:12:02.887851+0000","last_clean":"2024-04-16T21:12:02.887851+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:12:02.887851+0000","last_undegraded":"2024-04-16T21:12:02.887851+0000","last_fullsized":"2024-04-16T21:12:02.887851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.887759+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:37.031244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1038","reported_seq":1547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.224293+0000","last_change":"2024-04-16T21:11:54.224379+0000","last_active":"2024-04-16T21:11:54.224293+0000","last_peered":"2024-04-16T21:11:54.224293+0000","last_clean":"2024-04-16T21:11:54.224293+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:11:54.224293+0000","last_undegraded":"2024-04-16T21:11:54.224293+0000","last_fullsized":"2024-04-16T21:11:54.224293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1038","last_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_deep_scrub":"52'1038","last_deep_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.224224+0000","objects_scrubbed":21,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:22.282373+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2194,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'23","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.136772+0000","last_change":"2024-04-16T21:11:43.136819+0000","last_active":"2024-04-16T21:11:43.136772+0000","last_peered":"2024-04-16T21:11:43.136772+0000","last_clean":"2024-04-16T21:11:43.136772+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:11:43.136772+0000","last_undegraded":"2024-04-16T21:11:43.136772+0000","last_fullsized":"2024-04-16T21:11:43.136772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'23","last_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_deep_scrub":"52'23","last_deep_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.136721+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:06.328117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.766632+0000","last_change":"2024-04-16T21:12:08.766632+0000","last_active":"2024-04-16T21:12:08.766632+0000","last_peered":"2024-04-16T21:12:08.766632+0000","last_clean":"2024-04-16T21:12:08.766632+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:12:08.766632+0000","last_undegraded":"2024-04-16T21:12:08.766632+0000","last_fullsized":"2024-04-16T21:12:08.766632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.766544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:45.782631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743080+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:11:42.743080+0000","last_peered":"2024-04-16T21:11:42.743080+0000","last_clean":"2024-04-16T21:11:42.743080+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:11:42.743080+0000","last_undegraded":"2024-04-16T21:11:42.743080+0000","last_fullsized":"2024-04-16T21:11:42.743080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391438+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:10:54.391438+0000","last_peered":"2024-04-16T21:10:54.391438+0000","last_clean":"2024-04-16T21:10:54.391438+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:10:54.391438+0000","last_undegraded":"2024-04-16T21:10:54.391438+0000","last_fullsized":"2024-04-16T21:10:54.391438+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650430+0000","last_change":"2024-04-16T19:24:45.413354+0000","last_active":"2024-04-16T21:11:43.650430+0000","last_peered":"2024-04-16T21:11:43.650430+0000","last_clean":"2024-04-16T21:11:43.650430+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:11:43.650430+0000","last_undegraded":"2024-04-16T21:11:43.650430+0000","last_fullsized":"2024-04-16T21:11:43.650430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'74","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560664+0000","last_change":"2024-04-16T19:24:47.421544+0000","last_active":"2024-04-16T21:11:44.560664+0000","last_peered":"2024-04-16T21:11:44.560664+0000","last_clean":"2024-04-16T21:11:44.560664+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:11:44.560664+0000","last_undegraded":"2024-04-16T21:11:44.560664+0000","last_fullsized":"2024-04-16T21:11:44.560664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292469+0000","last_change":"2024-04-16T19:24:49.432387+0000","last_active":"2024-04-16T21:11:42.292469+0000","last_peered":"2024-04-16T21:11:42.292469+0000","last_clean":"2024-04-16T21:11:42.292469+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:11:42.292469+0000","last_undegraded":"2024-04-16T21:11:42.292469+0000","last_fullsized":"2024-04-16T21:11:42.292469+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651543+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:11:43.651543+0000","last_peered":"2024-04-16T21:11:43.651543+0000","last_clean":"2024-04-16T21:11:43.651543+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:11:43.651543+0000","last_undegraded":"2024-04-16T21:11:43.651543+0000","last_fullsized":"2024-04-16T21:11:43.651543+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'162","reported_seq":2030,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651440+0000","last_change":"2024-04-16T19:25:49.520473+0000","last_active":"2024-04-16T21:11:43.651440+0000","last_peered":"2024-04-16T21:11:43.651440+0000","last_clean":"2024-04-16T21:11:43.651440+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:11:43.651440+0000","last_undegraded":"2024-04-16T21:11:43.651440+0000","last_fullsized":"2024-04-16T21:11:43.651440+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.457685+0000","last_change":"2024-04-16T21:12:13.457709+0000","last_active":"2024-04-16T21:12:13.457685+0000","last_peered":"2024-04-16T21:12:13.457685+0000","last_clean":"2024-04-16T21:12:13.457685+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:12:13.457685+0000","last_undegraded":"2024-04-16T21:12:13.457685+0000","last_fullsized":"2024-04-16T21:12:13.457685+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.457615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:51.890971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'7","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.648645+0000","last_change":"2024-04-16T21:12:07.648674+0000","last_active":"2024-04-16T21:12:07.648645+0000","last_peered":"2024-04-16T21:12:07.648645+0000","last_clean":"2024-04-16T21:12:07.648645+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:12:07.648645+0000","last_undegraded":"2024-04-16T21:12:07.648645+0000","last_fullsized":"2024-04-16T21:12:07.648645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.648592+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.343437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.389539+0000","last_change":"2024-04-16T21:12:06.389563+0000","last_active":"2024-04-16T21:12:06.389539+0000","last_peered":"2024-04-16T21:12:06.389539+0000","last_clean":"2024-04-16T21:12:06.389539+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:12:06.389539+0000","last_undegraded":"2024-04-16T21:12:06.389539+0000","last_fullsized":"2024-04-16T21:12:06.389539+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.389467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:55.353016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'329","reported_seq":648,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.716248+0000","last_change":"2024-04-16T21:11:45.716329+0000","last_active":"2024-04-16T21:11:45.716248+0000","last_peered":"2024-04-16T21:11:45.716248+0000","last_clean":"2024-04-16T21:11:45.716248+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:11:45.716248+0000","last_undegraded":"2024-04-16T21:11:45.716248+0000","last_fullsized":"2024-04-16T21:11:45.716248+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'329","last_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_deep_scrub":"52'329","last_deep_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.716182+0000","objects_scrubbed":24,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:24.600927+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'11","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.675377+0000","last_change":"2024-04-16T21:11:50.675426+0000","last_active":"2024-04-16T21:11:50.675377+0000","last_peered":"2024-04-16T21:11:50.675377+0000","last_clean":"2024-04-16T21:11:50.675377+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:11:50.675377+0000","last_undegraded":"2024-04-16T21:11:50.675377+0000","last_fullsized":"2024-04-16T21:11:50.675377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_deep_scrub":"52'11","last_deep_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.675302+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:31.269067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.387000+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:10:54.387000+0000","last_peered":"2024-04-16T21:10:54.387000+0000","last_clean":"2024-04-16T21:10:54.387000+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:10:54.387000+0000","last_undegraded":"2024-04-16T21:10:54.387000+0000","last_fullsized":"2024-04-16T21:10:54.387000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525357+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:10:54.525357+0000","last_peered":"2024-04-16T21:10:54.525357+0000","last_clean":"2024-04-16T21:10:54.525357+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:10:54.525357+0000","last_undegraded":"2024-04-16T21:10:54.525357+0000","last_fullsized":"2024-04-16T21:10:54.525357+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.394031+0000","last_change":"2024-04-16T19:24:45.416047+0000","last_active":"2024-04-16T21:11:09.394031+0000","last_peered":"2024-04-16T21:11:09.394031+0000","last_clean":"2024-04-16T21:11:09.394031+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:11:09.394031+0000","last_undegraded":"2024-04-16T21:11:09.394031+0000","last_fullsized":"2024-04-16T21:11:09.394031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'82","reported_seq":241,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743416+0000","last_change":"2024-04-16T19:24:47.424208+0000","last_active":"2024-04-16T21:11:42.743416+0000","last_peered":"2024-04-16T21:11:42.743416+0000","last_clean":"2024-04-16T21:11:42.743416+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:11:42.743416+0000","last_undegraded":"2024-04-16T21:11:42.743416+0000","last_fullsized":"2024-04-16T21:11:42.743416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650659+0000","last_change":"2024-04-16T19:24:49.430180+0000","last_active":"2024-04-16T21:11:43.650659+0000","last_peered":"2024-04-16T21:11:43.650659+0000","last_clean":"2024-04-16T21:11:43.650659+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:11:43.650659+0000","last_undegraded":"2024-04-16T21:11:43.650659+0000","last_fullsized":"2024-04-16T21:11:43.650659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.732045+0000","last_change":"2024-04-16T21:11:59.732045+0000","last_active":"2024-04-16T21:11:59.732045+0000","last_peered":"2024-04-16T21:11:59.732045+0000","last_clean":"2024-04-16T21:11:59.732045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:11:59.732045+0000","last_undegraded":"2024-04-16T21:11:59.732045+0000","last_fullsized":"2024-04-16T21:11:59.732045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.731957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:37.824609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561671+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:11:44.561671+0000","last_peered":"2024-04-16T21:11:44.561671+0000","last_clean":"2024-04-16T21:11:44.561671+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:11:44.561671+0000","last_undegraded":"2024-04-16T21:11:44.561671+0000","last_fullsized":"2024-04-16T21:11:44.561671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'124","reported_seq":1827,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388716+0000","last_change":"2024-04-16T19:25:49.519043+0000","last_active":"2024-04-16T21:11:09.388716+0000","last_peered":"2024-04-16T21:11:09.388716+0000","last_clean":"2024-04-16T21:11:09.388716+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:11:09.388716+0000","last_undegraded":"2024-04-16T21:11:09.388716+0000","last_fullsized":"2024-04-16T21:11:09.388716+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.906298+0000","last_change":"2024-04-16T21:12:17.906319+0000","last_active":"2024-04-16T21:12:17.906298+0000","last_peered":"2024-04-16T21:12:17.906298+0000","last_clean":"2024-04-16T21:12:17.906298+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:12:17.906298+0000","last_undegraded":"2024-04-16T21:12:17.906298+0000","last_fullsized":"2024-04-16T21:12:17.906298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.906238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:22.945182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'15","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:39.393618+0000","last_change":"2024-04-16T19:25:15.483151+0000","last_active":"2024-04-16T21:11:39.393618+0000","last_peered":"2024-04-16T21:11:39.393618+0000","last_clean":"2024-04-16T21:11:39.393618+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:11:39.393618+0000","last_undegraded":"2024-04-16T21:11:39.393618+0000","last_fullsized":"2024-04-16T21:11:39.393618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.865487+0000","last_change":"2024-04-16T21:12:05.865507+0000","last_active":"2024-04-16T21:12:05.865487+0000","last_peered":"2024-04-16T21:12:05.865487+0000","last_clean":"2024-04-16T21:12:05.865487+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:12:05.865487+0000","last_undegraded":"2024-04-16T21:12:05.865487+0000","last_fullsized":"2024-04-16T21:12:05.865487+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.865437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.716343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'138","reported_seq":298,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.719397+0000","last_change":"2024-04-16T21:11:48.719459+0000","last_active":"2024-04-16T21:11:48.719397+0000","last_peered":"2024-04-16T21:11:48.719397+0000","last_clean":"2024-04-16T21:11:48.719397+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:11:48.719397+0000","last_undegraded":"2024-04-16T21:11:48.719397+0000","last_fullsized":"2024-04-16T21:11:48.719397+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.719337+0000","objects_scrubbed":17,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.768315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4229,"num_omap_keys":20,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'8","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.405923+0000","last_change":"2024-04-16T21:11:49.405983+0000","last_active":"2024-04-16T21:11:49.405923+0000","last_peered":"2024-04-16T21:11:49.405923+0000","last_clean":"2024-04-16T21:11:49.405923+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:11:49.405923+0000","last_undegraded":"2024-04-16T21:11:49.405923+0000","last_fullsized":"2024-04-16T21:11:49.405923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.405858+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:30.100703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392340+0000","last_change":"2024-04-16T19:24:35.396678+0000","last_active":"2024-04-16T21:10:59.392340+0000","last_peered":"2024-04-16T21:10:59.392340+0000","last_clean":"2024-04-16T21:10:59.392340+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:10:59.392340+0000","last_undegraded":"2024-04-16T21:10:59.392340+0000","last_fullsized":"2024-04-16T21:10:59.392340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.388626+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:11:14.388626+0000","last_peered":"2024-04-16T21:11:14.388626+0000","last_clean":"2024-04-16T21:11:14.388626+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:11:14.388626+0000","last_undegraded":"2024-04-16T21:11:14.388626+0000","last_fullsized":"2024-04-16T21:11:14.388626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"47'2","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388762+0000","last_change":"2024-04-16T19:24:45.413463+0000","last_active":"2024-04-16T21:11:04.388762+0000","last_peered":"2024-04-16T21:11:04.388762+0000","last_clean":"2024-04-16T21:11:04.388762+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:11:04.388762+0000","last_undegraded":"2024-04-16T21:11:04.388762+0000","last_fullsized":"2024-04-16T21:11:04.388762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'140","reported_seq":323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650484+0000","last_change":"2024-04-16T19:24:47.420323+0000","last_active":"2024-04-16T21:11:43.650484+0000","last_peered":"2024-04-16T21:11:43.650484+0000","last_clean":"2024-04-16T21:11:43.650484+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:11:43.650484+0000","last_undegraded":"2024-04-16T21:11:43.650484+0000","last_fullsized":"2024-04-16T21:11:43.650484+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.930468+0000","last_change":"2024-04-16T21:12:23.930491+0000","last_active":"2024-04-16T21:12:23.930468+0000","last_peered":"2024-04-16T21:12:23.930468+0000","last_clean":"2024-04-16T21:12:23.930468+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:12:23.930468+0000","last_undegraded":"2024-04-16T21:12:23.930468+0000","last_fullsized":"2024-04-16T21:12:23.930468+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.930404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:27.014075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.517927+0000","last_change":"2024-04-16T21:12:01.517927+0000","last_active":"2024-04-16T21:12:01.517927+0000","last_peered":"2024-04-16T21:12:01.517927+0000","last_clean":"2024-04-16T21:12:01.517927+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:12:01.517927+0000","last_undegraded":"2024-04-16T21:12:01.517927+0000","last_fullsized":"2024-04-16T21:12:01.517927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:38.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293399+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:11:42.293399+0000","last_peered":"2024-04-16T21:11:42.293399+0000","last_clean":"2024-04-16T21:11:42.293399+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:11:42.293399+0000","last_undegraded":"2024-04-16T21:11:42.293399+0000","last_fullsized":"2024-04-16T21:11:42.293399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'104","reported_seq":1285,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744079+0000","last_change":"2024-04-16T19:25:49.524319+0000","last_active":"2024-04-16T21:11:42.744079+0000","last_peered":"2024-04-16T21:11:42.744079+0000","last_clean":"2024-04-16T21:11:42.744079+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:11:42.744079+0000","last_undegraded":"2024-04-16T21:11:42.744079+0000","last_fullsized":"2024-04-16T21:11:42.744079+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.443129+0000","last_change":"2024-04-16T21:12:22.443154+0000","last_active":"2024-04-16T21:12:22.443129+0000","last_peered":"2024-04-16T21:12:22.443129+0000","last_clean":"2024-04-16T21:12:22.443129+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:12:22.443129+0000","last_undegraded":"2024-04-16T21:12:22.443129+0000","last_fullsized":"2024-04-16T21:12:22.443129+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.443057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:22.418023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'15","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.828587+0000","last_change":"2024-04-16T21:12:23.828622+0000","last_active":"2024-04-16T21:12:23.828587+0000","last_peered":"2024-04-16T21:12:23.828587+0000","last_clean":"2024-04-16T21:12:23.828587+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:12:23.828587+0000","last_undegraded":"2024-04-16T21:12:23.828587+0000","last_fullsized":"2024-04-16T21:12:23.828587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.828490+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:02.261846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.402446+0000","last_change":"2024-04-16T21:12:06.402484+0000","last_active":"2024-04-16T21:12:06.402446+0000","last_peered":"2024-04-16T21:12:06.402446+0000","last_clean":"2024-04-16T21:12:06.402446+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:12:06.402446+0000","last_undegraded":"2024-04-16T21:12:06.402446+0000","last_fullsized":"2024-04-16T21:12:06.402446+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.402344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:52.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'341","reported_seq":879,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.634057+0000","last_change":"2024-04-16T21:11:48.634146+0000","last_active":"2024-04-16T21:11:48.634057+0000","last_peered":"2024-04-16T21:11:48.634057+0000","last_clean":"2024-04-16T21:11:48.634057+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:11:48.634057+0000","last_undegraded":"2024-04-16T21:11:48.634057+0000","last_fullsized":"2024-04-16T21:11:48.634057+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'341","last_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_deep_scrub":"52'341","last_deep_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.633983+0000","objects_scrubbed":20,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:05:52.976164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'19","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.631104+0000","last_change":"2024-04-16T21:11:55.631145+0000","last_active":"2024-04-16T21:11:55.631104+0000","last_peered":"2024-04-16T21:11:55.631104+0000","last_clean":"2024-04-16T21:11:55.631104+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:11:55.631104+0000","last_undegraded":"2024-04-16T21:11:55.631104+0000","last_fullsized":"2024-04-16T21:11:55.631104+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.631042+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.051632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292189+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:11:42.292189+0000","last_peered":"2024-04-16T21:11:42.292189+0000","last_clean":"2024-04-16T21:11:42.292189+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:11:42.292189+0000","last_undegraded":"2024-04-16T21:11:42.292189+0000","last_fullsized":"2024-04-16T21:11:42.292189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106228+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:11:44.106228+0000","last_peered":"2024-04-16T21:11:44.106228+0000","last_clean":"2024-04-16T21:11:44.106228+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:11:44.106228+0000","last_undegraded":"2024-04-16T21:11:44.106228+0000","last_fullsized":"2024-04-16T21:11:44.106228+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106646+0000","last_change":"2024-04-16T19:24:45.411723+0000","last_active":"2024-04-16T21:11:44.106646+0000","last_peered":"2024-04-16T21:11:44.106646+0000","last_clean":"2024-04-16T21:11:44.106646+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:11:44.106646+0000","last_undegraded":"2024-04-16T21:11:44.106646+0000","last_fullsized":"2024-04-16T21:11:44.106646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":265,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527257+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:11:04.527257+0000","last_peered":"2024-04-16T21:11:04.527257+0000","last_clean":"2024-04-16T21:11:04.527257+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:11:04.527257+0000","last_undegraded":"2024-04-16T21:11:04.527257+0000","last_fullsized":"2024-04-16T21:11:04.527257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388030+0000","last_change":"2024-04-16T19:24:49.427034+0000","last_active":"2024-04-16T21:11:09.388030+0000","last_peered":"2024-04-16T21:11:09.388030+0000","last_clean":"2024-04-16T21:11:09.388030+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:11:09.388030+0000","last_undegraded":"2024-04-16T21:11:09.388030+0000","last_fullsized":"2024-04-16T21:11:09.388030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.482650+0000","last_change":"2024-04-16T21:12:03.482650+0000","last_active":"2024-04-16T21:12:03.482650+0000","last_peered":"2024-04-16T21:12:03.482650+0000","last_clean":"2024-04-16T21:12:03.482650+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:12:03.482650+0000","last_undegraded":"2024-04-16T21:12:03.482650+0000","last_fullsized":"2024-04-16T21:12:03.482650+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.482557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:28.918007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107592+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:11:44.107592+0000","last_peered":"2024-04-16T21:11:44.107592+0000","last_clean":"2024-04-16T21:11:44.107592+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:11:44.107592+0000","last_undegraded":"2024-04-16T21:11:44.107592+0000","last_fullsized":"2024-04-16T21:11:44.107592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'117","reported_seq":1308,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744067+0000","last_change":"2024-04-16T19:25:49.524231+0000","last_active":"2024-04-16T21:11:42.744067+0000","last_peered":"2024-04-16T21:11:42.744067+0000","last_clean":"2024-04-16T21:11:42.744067+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:11:42.744067+0000","last_undegraded":"2024-04-16T21:11:42.744067+0000","last_fullsized":"2024-04-16T21:11:42.744067+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.680232+0000","last_change":"2024-04-16T21:12:17.680254+0000","last_active":"2024-04-16T21:12:17.680232+0000","last_peered":"2024-04-16T21:12:17.680232+0000","last_clean":"2024-04-16T21:12:17.680232+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:12:17.680232+0000","last_undegraded":"2024-04-16T21:12:17.680232+0000","last_fullsized":"2024-04-16T21:12:17.680232+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.680171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:09.860482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"52'8","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.838545+0000","last_change":"2024-04-16T21:12:22.838571+0000","last_active":"2024-04-16T21:12:22.838545+0000","last_peered":"2024-04-16T21:12:22.838545+0000","last_clean":"2024-04-16T21:12:22.838545+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:12:22.838545+0000","last_undegraded":"2024-04-16T21:12:22.838545+0000","last_fullsized":"2024-04-16T21:12:22.838545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.838490+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:17.267225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.653884+0000","last_change":"2024-04-16T21:12:05.653905+0000","last_active":"2024-04-16T21:12:05.653884+0000","last_peered":"2024-04-16T21:12:05.653884+0000","last_clean":"2024-04-16T21:12:05.653884+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:12:05.653884+0000","last_undegraded":"2024-04-16T21:12:05.653884+0000","last_fullsized":"2024-04-16T21:12:05.653884+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.653825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:12.673868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'264","reported_seq":1136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.695963+0000","last_change":"2024-04-16T21:11:49.696035+0000","last_active":"2024-04-16T21:11:49.695963+0000","last_peered":"2024-04-16T21:11:49.695963+0000","last_clean":"2024-04-16T21:11:49.695963+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:11:49.695963+0000","last_undegraded":"2024-04-16T21:11:49.695963+0000","last_fullsized":"2024-04-16T21:11:49.695963+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.695875+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:11.637965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2924,"num_omap_keys":15,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'12","reported_seq":167,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.501554+0000","last_change":"2024-04-16T21:11:59.501597+0000","last_active":"2024-04-16T21:11:59.501554+0000","last_peered":"2024-04-16T21:11:59.501554+0000","last_clean":"2024-04-16T21:11:59.501554+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:11:59.501554+0000","last_undegraded":"2024-04-16T21:11:59.501554+0000","last_fullsized":"2024-04-16T21:11:59.501554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.501496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:51.681807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194943+0000","last_change":"2024-04-16T19:24:35.395992+0000","last_active":"2024-04-16T21:11:43.194943+0000","last_peered":"2024-04-16T21:11:43.194943+0000","last_clean":"2024-04-16T21:11:43.194943+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:11:43.194943+0000","last_undegraded":"2024-04-16T21:11:43.194943+0000","last_fullsized":"2024-04-16T21:11:43.194943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291882+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:11:42.291882+0000","last_peered":"2024-04-16T21:11:42.291882+0000","last_clean":"2024-04-16T21:11:42.291882+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:11:42.291882+0000","last_undegraded":"2024-04-16T21:11:42.291882+0000","last_fullsized":"2024-04-16T21:11:42.291882+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743303+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:11:42.743303+0000","last_peered":"2024-04-16T21:11:42.743303+0000","last_clean":"2024-04-16T21:11:42.743303+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:11:42.743303+0000","last_undegraded":"2024-04-16T21:11:42.743303+0000","last_fullsized":"2024-04-16T21:11:42.743303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'110","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389503+0000","last_change":"2024-04-16T19:24:47.419160+0000","last_active":"2024-04-16T21:11:09.389503+0000","last_peered":"2024-04-16T21:11:09.389503+0000","last_clean":"2024-04-16T21:11:09.389503+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:11:09.389503+0000","last_undegraded":"2024-04-16T21:11:09.389503+0000","last_fullsized":"2024-04-16T21:11:09.389503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'12","reported_seq":1316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387890+0000","last_change":"2024-04-16T19:24:49.433728+0000","last_active":"2024-04-16T21:11:04.387890+0000","last_peered":"2024-04-16T21:11:04.387890+0000","last_clean":"2024-04-16T21:11:04.387890+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:11:04.387890+0000","last_undegraded":"2024-04-16T21:11:04.387890+0000","last_fullsized":"2024-04-16T21:11:04.387890+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.424596+0000","last_change":"2024-04-16T21:12:05.424596+0000","last_active":"2024-04-16T21:12:05.424596+0000","last_peered":"2024-04-16T21:12:05.424596+0000","last_clean":"2024-04-16T21:12:05.424596+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:12:05.424596+0000","last_undegraded":"2024-04-16T21:12:05.424596+0000","last_fullsized":"2024-04-16T21:12:05.424596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.424465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:59.031445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392508+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:11:34.392508+0000","last_peered":"2024-04-16T21:11:34.392508+0000","last_clean":"2024-04-16T21:11:34.392508+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:11:34.392508+0000","last_undegraded":"2024-04-16T21:11:34.392508+0000","last_fullsized":"2024-04-16T21:11:34.392508+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'214","reported_seq":2305,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385281+0000","last_change":"2024-04-16T19:25:49.518910+0000","last_active":"2024-04-16T21:10:49.385281+0000","last_peered":"2024-04-16T21:10:49.385281+0000","last_clean":"2024-04-16T21:10:49.385281+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:10:49.385281+0000","last_undegraded":"2024-04-16T21:10:49.385281+0000","last_fullsized":"2024-04-16T21:10:49.385281+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.466819+0000","last_change":"2024-04-16T21:12:18.466844+0000","last_active":"2024-04-16T21:12:18.466819+0000","last_peered":"2024-04-16T21:12:18.466819+0000","last_clean":"2024-04-16T21:12:18.466819+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:12:18.466819+0000","last_undegraded":"2024-04-16T21:12:18.466819+0000","last_fullsized":"2024-04-16T21:12:18.466819+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.466741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:50.153442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'17","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.900870+0000","last_change":"2024-04-16T21:12:12.900906+0000","last_active":"2024-04-16T21:12:12.900870+0000","last_peered":"2024-04-16T21:12:12.900870+0000","last_clean":"2024-04-16T21:12:12.900870+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:12:12.900870+0000","last_undegraded":"2024-04-16T21:12:12.900870+0000","last_fullsized":"2024-04-16T21:12:12.900870+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.900807+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:00.936096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'705","reported_seq":2123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.383372+0000","last_change":"2024-04-16T19:25:13.473698+0000","last_active":"2024-04-16T21:10:34.383372+0000","last_peered":"2024-04-16T21:10:34.383372+0000","last_clean":"2024-04-16T21:10:34.383372+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:10:34.383372+0000","last_undegraded":"2024-04-16T21:10:34.383372+0000","last_fullsized":"2024-04-16T21:10:34.383372+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:12.453581+0000","last_clean_scrub_stamp":"2024-04-16T19:25:12.453581+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'557","reported_seq":1382,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.205288+0000","last_change":"2024-04-16T21:11:50.205349+0000","last_active":"2024-04-16T21:11:50.205288+0000","last_peered":"2024-04-16T21:11:50.205288+0000","last_clean":"2024-04-16T21:11:50.205288+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:11:50.205288+0000","last_undegraded":"2024-04-16T21:11:50.205288+0000","last_fullsized":"2024-04-16T21:11:50.205288+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'557","last_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_deep_scrub":"52'557","last_deep_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.205222+0000","objects_scrubbed":17,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:09.070331+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2200,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'17","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.702594+0000","last_change":"2024-04-16T21:12:11.702662+0000","last_active":"2024-04-16T21:12:11.702594+0000","last_peered":"2024-04-16T21:12:11.702594+0000","last_clean":"2024-04-16T21:12:11.702594+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:12:11.702594+0000","last_undegraded":"2024-04-16T21:12:11.702594+0000","last_fullsized":"2024-04-16T21:12:11.702594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.702519+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:33.204105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106455+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:11:44.106455+0000","last_peered":"2024-04-16T21:11:44.106455+0000","last_clean":"2024-04-16T21:11:44.106455+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:11:44.106455+0000","last_undegraded":"2024-04-16T21:11:44.106455+0000","last_fullsized":"2024-04-16T21:11:44.106455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560193+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:11:44.560193+0000","last_peered":"2024-04-16T21:11:44.560193+0000","last_clean":"2024-04-16T21:11:44.560193+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:11:44.560193+0000","last_undegraded":"2024-04-16T21:11:44.560193+0000","last_fullsized":"2024-04-16T21:11:44.560193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106597+0000","last_change":"2024-04-16T19:24:45.412314+0000","last_active":"2024-04-16T21:11:44.106597+0000","last_peered":"2024-04-16T21:11:44.106597+0000","last_clean":"2024-04-16T21:11:44.106597+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:11:44.106597+0000","last_undegraded":"2024-04-16T21:11:44.106597+0000","last_fullsized":"2024-04-16T21:11:44.106597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'140","reported_seq":324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391659+0000","last_change":"2024-04-16T19:24:47.422463+0000","last_active":"2024-04-16T21:11:29.391659+0000","last_peered":"2024-04-16T21:11:29.391659+0000","last_clean":"2024-04-16T21:11:29.391659+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:11:29.391659+0000","last_undegraded":"2024-04-16T21:11:29.391659+0000","last_fullsized":"2024-04-16T21:11:29.391659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393824+0000","last_change":"2024-04-16T19:24:49.427917+0000","last_active":"2024-04-16T21:11:09.393824+0000","last_peered":"2024-04-16T21:11:09.393824+0000","last_clean":"2024-04-16T21:11:09.393824+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:11:09.393824+0000","last_undegraded":"2024-04-16T21:11:09.393824+0000","last_fullsized":"2024-04-16T21:11:09.393824+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.443393+0000","last_change":"2024-04-16T21:12:07.443416+0000","last_active":"2024-04-16T21:12:07.443393+0000","last_peered":"2024-04-16T21:12:07.443393+0000","last_clean":"2024-04-16T21:12:07.443393+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:12:07.443393+0000","last_undegraded":"2024-04-16T21:12:07.443393+0000","last_fullsized":"2024-04-16T21:12:07.443393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.443331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:23.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.532076+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:11:34.532076+0000","last_peered":"2024-04-16T21:11:34.532076+0000","last_clean":"2024-04-16T21:11:34.532076+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:11:34.532076+0000","last_undegraded":"2024-04-16T21:11:34.532076+0000","last_fullsized":"2024-04-16T21:11:34.532076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384453+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:10:49.384453+0000","last_peered":"2024-04-16T21:10:49.384453+0000","last_clean":"2024-04-16T21:10:49.384453+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:10:49.384453+0000","last_undegraded":"2024-04-16T21:10:49.384453+0000","last_fullsized":"2024-04-16T21:10:49.384453+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384446+0000","last_change":"2024-04-16T19:25:41.507742+0000","last_active":"2024-04-16T21:10:44.384446+0000","last_peered":"2024-04-16T21:10:44.384446+0000","last_clean":"2024-04-16T21:10:44.384446+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:10:44.384446+0000","last_undegraded":"2024-04-16T21:10:44.384446+0000","last_fullsized":"2024-04-16T21:10:44.384446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:40.485060+0000","last_clean_scrub_stamp":"2024-04-16T19:25:40.485060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'10","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.431951+0000","last_change":"2024-04-16T21:12:23.431983+0000","last_active":"2024-04-16T21:12:23.431951+0000","last_peered":"2024-04-16T21:12:23.431951+0000","last_clean":"2024-04-16T21:12:23.431951+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:12:23.431951+0000","last_undegraded":"2024-04-16T21:12:23.431951+0000","last_fullsized":"2024-04-16T21:12:23.431951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.431886+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:37.702235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'706","reported_seq":2082,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.853855+0000","last_change":"2024-04-16T21:12:01.853885+0000","last_active":"2024-04-16T21:12:01.853855+0000","last_peered":"2024-04-16T21:12:01.853855+0000","last_clean":"2024-04-16T21:12:01.853855+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:12:01.853855+0000","last_undegraded":"2024-04-16T21:12:01.853855+0000","last_fullsized":"2024-04-16T21:12:01.853855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.853792+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.601423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'542","reported_seq":41299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.597167+0000","last_change":"2024-04-16T21:11:54.597266+0000","last_active":"2024-04-16T21:11:54.597167+0000","last_peered":"2024-04-16T21:11:54.597167+0000","last_clean":"2024-04-16T21:11:54.597167+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:11:54.597167+0000","last_undegraded":"2024-04-16T21:11:54.597167+0000","last_fullsized":"2024-04-16T21:11:54.597167+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'542","last_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_deep_scrub":"52'542","last_deep_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.597093+0000","objects_scrubbed":23,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:16.539182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'18","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.434844+0000","last_change":"2024-04-16T21:11:54.434914+0000","last_active":"2024-04-16T21:11:54.434844+0000","last_peered":"2024-04-16T21:11:54.434844+0000","last_clean":"2024-04-16T21:11:54.434844+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:11:54.434844+0000","last_undegraded":"2024-04-16T21:11:54.434844+0000","last_fullsized":"2024-04-16T21:11:54.434844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.434778+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:53.824584+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106370+0000","last_change":"2024-04-16T19:24:35.400462+0000","last_active":"2024-04-16T21:11:44.106370+0000","last_peered":"2024-04-16T21:11:44.106370+0000","last_clean":"2024-04-16T21:11:44.106370+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:11:44.106370+0000","last_undegraded":"2024-04-16T21:11:44.106370+0000","last_fullsized":"2024-04-16T21:11:44.106370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560163+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:11:44.560163+0000","last_peered":"2024-04-16T21:11:44.560163+0000","last_clean":"2024-04-16T21:11:44.560163+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:11:44.560163+0000","last_undegraded":"2024-04-16T21:11:44.560163+0000","last_fullsized":"2024-04-16T21:11:44.560163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'12","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560575+0000","last_change":"2024-04-16T19:24:45.414683+0000","last_active":"2024-04-16T21:11:44.560575+0000","last_peered":"2024-04-16T21:11:44.560575+0000","last_clean":"2024-04-16T21:11:44.560575+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:11:44.560575+0000","last_undegraded":"2024-04-16T21:11:44.560575+0000","last_fullsized":"2024-04-16T21:11:44.560575+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'74","reported_seq":225,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393868+0000","last_change":"2024-04-16T19:24:47.420795+0000","last_active":"2024-04-16T21:11:09.393868+0000","last_peered":"2024-04-16T21:11:09.393868+0000","last_clean":"2024-04-16T21:11:09.393868+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:11:09.393868+0000","last_undegraded":"2024-04-16T21:11:09.393868+0000","last_fullsized":"2024-04-16T21:11:09.393868+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.392356+0000","last_change":"2024-04-16T19:24:49.429979+0000","last_active":"2024-04-16T21:11:29.392356+0000","last_peered":"2024-04-16T21:11:29.392356+0000","last_clean":"2024-04-16T21:11:29.392356+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:11:29.392356+0000","last_undegraded":"2024-04-16T21:11:29.392356+0000","last_fullsized":"2024-04-16T21:11:29.392356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:48.412680+0000","last_clean_scrub_stamp":"2024-04-16T19:24:48.412680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.776294+0000","last_change":"2024-04-16T21:12:00.776335+0000","last_active":"2024-04-16T21:12:00.776294+0000","last_peered":"2024-04-16T21:12:00.776294+0000","last_clean":"2024-04-16T21:12:00.776294+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:12:00.776294+0000","last_undegraded":"2024-04-16T21:12:00.776294+0000","last_fullsized":"2024-04-16T21:12:00.776294+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.776223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:20.604086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398019+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:11:34.398019+0000","last_peered":"2024-04-16T21:11:34.398019+0000","last_clean":"2024-04-16T21:11:34.398019+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:11:34.398019+0000","last_undegraded":"2024-04-16T21:11:34.398019+0000","last_fullsized":"2024-04-16T21:11:34.398019+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'222","reported_seq":2177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.980519+0000","last_change":"2024-04-16T21:12:16.980550+0000","last_active":"2024-04-16T21:12:16.980519+0000","last_peered":"2024-04-16T21:12:16.980519+0000","last_clean":"2024-04-16T21:12:16.980519+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:12:16.980519+0000","last_undegraded":"2024-04-16T21:12:16.980519+0000","last_fullsized":"2024-04-16T21:12:16.980519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'222","last_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_deep_scrub":"52'222","last_deep_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.980456+0000","objects_scrubbed":55,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:41.646543+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14526,"num_omap_keys":59,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.913868+0000","last_change":"2024-04-16T21:12:10.913891+0000","last_active":"2024-04-16T21:12:10.913868+0000","last_peered":"2024-04-16T21:12:10.913868+0000","last_clean":"2024-04-16T21:12:10.913868+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:12:10.913868+0000","last_undegraded":"2024-04-16T21:12:10.913868+0000","last_fullsized":"2024-04-16T21:12:10.913868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.913806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:17.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'16","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380829+0000","last_change":"2024-04-16T19:25:15.481851+0000","last_active":"2024-04-16T21:10:19.380829+0000","last_peered":"2024-04-16T21:10:19.380829+0000","last_clean":"2024-04-16T21:10:19.380829+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:10:19.380829+0000","last_undegraded":"2024-04-16T21:10:19.380829+0000","last_fullsized":"2024-04-16T21:10:19.380829+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:14.462387+0000","last_clean_scrub_stamp":"2024-04-16T19:25:14.462387+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.652503+0000","last_change":"2024-04-16T21:11:59.652525+0000","last_active":"2024-04-16T21:11:59.652503+0000","last_peered":"2024-04-16T21:11:59.652503+0000","last_clean":"2024-04-16T21:11:59.652503+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:11:59.652503+0000","last_undegraded":"2024-04-16T21:11:59.652503+0000","last_fullsized":"2024-04-16T21:11:59.652503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.652420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:18.517529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'592","reported_seq":1363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.540243+0000","last_change":"2024-04-16T21:12:00.540298+0000","last_active":"2024-04-16T21:12:00.540243+0000","last_peered":"2024-04-16T21:12:00.540243+0000","last_clean":"2024-04-16T21:12:00.540243+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:12:00.540243+0000","last_undegraded":"2024-04-16T21:12:00.540243+0000","last_fullsized":"2024-04-16T21:12:00.540243+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'592","last_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_deep_scrub":"52'592","last_deep_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.540183+0000","objects_scrubbed":19,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:48.575472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'32","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.552154+0000","last_change":"2024-04-16T21:11:43.552192+0000","last_active":"2024-04-16T21:11:43.552154+0000","last_peered":"2024-04-16T21:11:43.552154+0000","last_clean":"2024-04-16T21:11:43.552154+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:11:43.552154+0000","last_undegraded":"2024-04-16T21:11:43.552154+0000","last_fullsized":"2024-04-16T21:11:43.552154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.552108+0000","objects_scrubbed":9,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.135567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395593+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:11:19.395593+0000","last_peered":"2024-04-16T21:11:19.395593+0000","last_clean":"2024-04-16T21:11:19.395593+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:11:19.395593+0000","last_undegraded":"2024-04-16T21:11:19.395593+0000","last_fullsized":"2024-04-16T21:11:19.395593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524792+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:10:49.524792+0000","last_peered":"2024-04-16T21:10:49.524792+0000","last_clean":"2024-04-16T21:10:49.524792+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:10:49.524792+0000","last_undegraded":"2024-04-16T21:10:49.524792+0000","last_fullsized":"2024-04-16T21:10:49.524792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388587+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:11:09.388587+0000","last_peered":"2024-04-16T21:11:09.388587+0000","last_clean":"2024-04-16T21:11:09.388587+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:11:09.388587+0000","last_undegraded":"2024-04-16T21:11:09.388587+0000","last_fullsized":"2024-04-16T21:11:09.388587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'30","reported_seq":158,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292404+0000","last_change":"2024-04-16T19:24:47.422336+0000","last_active":"2024-04-16T21:11:42.292404+0000","last_peered":"2024-04-16T21:11:42.292404+0000","last_clean":"2024-04-16T21:11:42.292404+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:11:42.292404+0000","last_undegraded":"2024-04-16T21:11:42.292404+0000","last_fullsized":"2024-04-16T21:11:42.292404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.384509+0000","last_change":"2024-04-16T21:12:11.384509+0000","last_active":"2024-04-16T21:12:11.384509+0000","last_peered":"2024-04-16T21:12:11.384509+0000","last_clean":"2024-04-16T21:12:11.384509+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:12:11.384509+0000","last_undegraded":"2024-04-16T21:12:11.384509+0000","last_fullsized":"2024-04-16T21:12:11.384509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.384424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:06.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.514240+0000","last_change":"2024-04-16T21:12:02.514262+0000","last_active":"2024-04-16T21:12:02.514240+0000","last_peered":"2024-04-16T21:12:02.514240+0000","last_clean":"2024-04-16T21:12:02.514240+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:12:02.514240+0000","last_undegraded":"2024-04-16T21:12:02.514240+0000","last_fullsized":"2024-04-16T21:12:02.514240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.514179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:44.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.391931+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:11:34.391931+0000","last_peered":"2024-04-16T21:11:34.391931+0000","last_clean":"2024-04-16T21:11:34.391931+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:11:34.391931+0000","last_undegraded":"2024-04-16T21:11:34.391931+0000","last_fullsized":"2024-04-16T21:11:34.391931+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'172","reported_seq":1564,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.949162+0000","last_change":"2024-04-16T21:12:12.949215+0000","last_active":"2024-04-16T21:12:12.949162+0000","last_peered":"2024-04-16T21:12:12.949162+0000","last_clean":"2024-04-16T21:12:12.949162+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:12:12.949162+0000","last_undegraded":"2024-04-16T21:12:12.949162+0000","last_fullsized":"2024-04-16T21:12:12.949162+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.949100+0000","objects_scrubbed":45,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.338906+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8810,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.869571+0000","last_change":"2024-04-16T21:12:09.869596+0000","last_active":"2024-04-16T21:12:09.869571+0000","last_peered":"2024-04-16T21:12:09.869571+0000","last_clean":"2024-04-16T21:12:09.869571+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:12:09.869571+0000","last_undegraded":"2024-04-16T21:12:09.869571+0000","last_fullsized":"2024-04-16T21:12:09.869571+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.869508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:34.535595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"52'16","reported_seq":159,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.491364+0000","last_change":"2024-04-16T21:12:21.491403+0000","last_active":"2024-04-16T21:12:21.491364+0000","last_peered":"2024-04-16T21:12:21.491364+0000","last_clean":"2024-04-16T21:12:21.491364+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:12:21.491364+0000","last_undegraded":"2024-04-16T21:12:21.491364+0000","last_fullsized":"2024-04-16T21:12:21.491364+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.491306+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:09.623796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.693490+0000","last_change":"2024-04-16T21:12:01.693513+0000","last_active":"2024-04-16T21:12:01.693490+0000","last_peered":"2024-04-16T21:12:01.693490+0000","last_clean":"2024-04-16T21:12:01.693490+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:12:01.693490+0000","last_undegraded":"2024-04-16T21:12:01.693490+0000","last_fullsized":"2024-04-16T21:12:01.693490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.693430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:04.866318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'249","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.401831+0000","last_change":"2024-04-16T21:11:48.401911+0000","last_active":"2024-04-16T21:11:48.401831+0000","last_peered":"2024-04-16T21:11:48.401831+0000","last_clean":"2024-04-16T21:11:48.401831+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:11:48.401831+0000","last_undegraded":"2024-04-16T21:11:48.401831+0000","last_fullsized":"2024-04-16T21:11:48.401831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.401765+0000","objects_scrubbed":19,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:02.153361+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'12","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.683222+0000","last_change":"2024-04-16T21:12:12.683275+0000","last_active":"2024-04-16T21:12:12.683222+0000","last_peered":"2024-04-16T21:12:12.683222+0000","last_clean":"2024-04-16T21:12:12.683222+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:12:12.683222+0000","last_undegraded":"2024-04-16T21:12:12.683222+0000","last_fullsized":"2024-04-16T21:12:12.683222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.683160+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.072966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387647+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:10:59.387647+0000","last_peered":"2024-04-16T21:10:59.387647+0000","last_clean":"2024-04-16T21:10:59.387647+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:10:59.387647+0000","last_undegraded":"2024-04-16T21:10:59.387647+0000","last_fullsized":"2024-04-16T21:10:59.387647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390741+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:10:49.390741+0000","last_peered":"2024-04-16T21:10:49.390741+0000","last_clean":"2024-04-16T21:10:49.390741+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:10:49.390741+0000","last_undegraded":"2024-04-16T21:10:49.390741+0000","last_fullsized":"2024-04-16T21:10:49.390741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"47'2","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:29.391073+0000","last_change":"2024-04-16T19:24:45.414074+0000","last_active":"2024-04-16T21:11:29.391073+0000","last_peered":"2024-04-16T21:11:29.391073+0000","last_clean":"2024-04-16T21:11:29.391073+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:11:29.391073+0000","last_undegraded":"2024-04-16T21:11:29.391073+0000","last_fullsized":"2024-04-16T21:11:29.391073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106674+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:11:44.106674+0000","last_peered":"2024-04-16T21:11:44.106674+0000","last_clean":"2024-04-16T21:11:44.106674+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:11:44.106674+0000","last_undegraded":"2024-04-16T21:11:44.106674+0000","last_fullsized":"2024-04-16T21:11:44.106674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.466444+0000","last_change":"2024-04-16T21:12:11.466444+0000","last_active":"2024-04-16T21:12:11.466444+0000","last_peered":"2024-04-16T21:12:11.466444+0000","last_clean":"2024-04-16T21:12:11.466444+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:12:11.466444+0000","last_undegraded":"2024-04-16T21:12:11.466444+0000","last_fullsized":"2024-04-16T21:12:11.466444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.466351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:26.935725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.718007+0000","last_change":"2024-04-16T21:12:16.718042+0000","last_active":"2024-04-16T21:12:16.718007+0000","last_peered":"2024-04-16T21:12:16.718007+0000","last_clean":"2024-04-16T21:12:16.718007+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:12:16.718007+0000","last_undegraded":"2024-04-16T21:12:16.718007+0000","last_fullsized":"2024-04-16T21:12:16.718007+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.717926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:06.936826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'19","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.886963+0000","last_change":"2024-04-16T21:12:06.887055+0000","last_active":"2024-04-16T21:12:06.886963+0000","last_peered":"2024-04-16T21:12:06.886963+0000","last_clean":"2024-04-16T21:12:06.886963+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:12:06.886963+0000","last_undegraded":"2024-04-16T21:12:06.886963+0000","last_fullsized":"2024-04-16T21:12:06.886963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.886889+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:22.161394+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384672+0000","last_change":"2024-04-16T19:24:35.401994+0000","last_active":"2024-04-16T21:10:49.384672+0000","last_peered":"2024-04-16T21:10:49.384672+0000","last_clean":"2024-04-16T21:10:49.384672+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:10:49.384672+0000","last_undegraded":"2024-04-16T21:10:49.384672+0000","last_fullsized":"2024-04-16T21:10:49.384672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385227+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:10:49.385227+0000","last_peered":"2024-04-16T21:10:49.385227+0000","last_clean":"2024-04-16T21:10:49.385227+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:10:49.385227+0000","last_undegraded":"2024-04-16T21:10:49.385227+0000","last_fullsized":"2024-04-16T21:10:49.385227+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'18","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.679302+0000","last_change":"2024-04-16T21:12:03.679355+0000","last_active":"2024-04-16T21:12:03.679302+0000","last_peered":"2024-04-16T21:12:03.679302+0000","last_clean":"2024-04-16T21:12:03.679302+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:12:03.679302+0000","last_undegraded":"2024-04-16T21:12:03.679302+0000","last_fullsized":"2024-04-16T21:12:03.679302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.679242+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:31.737391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384640+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:10:49.384640+0000","last_peered":"2024-04-16T21:10:49.384640+0000","last_clean":"2024-04-16T21:10:49.384640+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:10:49.384640+0000","last_undegraded":"2024-04-16T21:10:49.384640+0000","last_fullsized":"2024-04-16T21:10:49.384640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524668+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:10:49.524668+0000","last_peered":"2024-04-16T21:10:49.524668+0000","last_clean":"2024-04-16T21:10:49.524668+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:10:49.524668+0000","last_undegraded":"2024-04-16T21:10:49.524668+0000","last_fullsized":"2024-04-16T21:10:49.524668+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'10","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.673321+0000","last_change":"2024-04-16T21:12:02.673359+0000","last_active":"2024-04-16T21:12:02.673321+0000","last_peered":"2024-04-16T21:12:02.673321+0000","last_clean":"2024-04-16T21:12:02.673321+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:12:02.673321+0000","last_undegraded":"2024-04-16T21:12:02.673321+0000","last_fullsized":"2024-04-16T21:12:02.673321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.673266+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:06.001051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385387+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:10:49.385387+0000","last_peered":"2024-04-16T21:10:49.385387+0000","last_clean":"2024-04-16T21:10:49.385387+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:10:49.385387+0000","last_undegraded":"2024-04-16T21:10:49.385387+0000","last_fullsized":"2024-04-16T21:10:49.385387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385866+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:10:49.385866+0000","last_peered":"2024-04-16T21:10:49.385866+0000","last_clean":"2024-04-16T21:10:49.385866+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:10:49.385866+0000","last_undegraded":"2024-04-16T21:10:49.385866+0000","last_fullsized":"2024-04-16T21:10:49.385866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'13","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.447533+0000","last_change":"2024-04-16T21:11:57.447598+0000","last_active":"2024-04-16T21:11:57.447533+0000","last_peered":"2024-04-16T21:11:57.447533+0000","last_clean":"2024-04-16T21:11:57.447533+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:11:57.447533+0000","last_undegraded":"2024-04-16T21:11:57.447533+0000","last_fullsized":"2024-04-16T21:11:57.447533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.447471+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:25.758892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384571+0000","last_change":"2024-04-16T19:24:35.400964+0000","last_active":"2024-04-16T21:10:49.384571+0000","last_peered":"2024-04-16T21:10:49.384571+0000","last_clean":"2024-04-16T21:10:49.384571+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:10:49.384571+0000","last_undegraded":"2024-04-16T21:10:49.384571+0000","last_fullsized":"2024-04-16T21:10:49.384571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393952+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:11:09.393952+0000","last_peered":"2024-04-16T21:11:09.393952+0000","last_clean":"2024-04-16T21:11:09.393952+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:11:09.393952+0000","last_undegraded":"2024-04-16T21:11:09.393952+0000","last_fullsized":"2024-04-16T21:11:09.393952+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'20","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.649769+0000","last_change":"2024-04-16T21:12:04.649822+0000","last_active":"2024-04-16T21:12:04.649769+0000","last_peered":"2024-04-16T21:12:04.649769+0000","last_clean":"2024-04-16T21:12:04.649769+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:12:04.649769+0000","last_undegraded":"2024-04-16T21:12:04.649769+0000","last_fullsized":"2024-04-16T21:12:04.649769+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.649711+0000","objects_scrubbed":9,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:32.961132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.528113+0000","last_change":"2024-04-16T19:24:35.398409+0000","last_active":"2024-04-16T21:11:09.528113+0000","last_peered":"2024-04-16T21:11:09.528113+0000","last_clean":"2024-04-16T21:11:09.528113+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:11:09.528113+0000","last_undegraded":"2024-04-16T21:11:09.528113+0000","last_fullsized":"2024-04-16T21:11:09.528113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383688+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:10:44.383688+0000","last_peered":"2024-04-16T21:10:44.383688+0000","last_clean":"2024-04-16T21:10:44.383688+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:10:44.383688+0000","last_undegraded":"2024-04-16T21:10:44.383688+0000","last_fullsized":"2024-04-16T21:10:44.383688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'28","reported_seq":281,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.947883+0000","last_change":"2024-04-16T21:12:13.947946+0000","last_active":"2024-04-16T21:12:13.947883+0000","last_peered":"2024-04-16T21:12:13.947883+0000","last_clean":"2024-04-16T21:12:13.947883+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:12:13.947883+0000","last_undegraded":"2024-04-16T21:12:13.947883+0000","last_fullsized":"2024-04-16T21:12:13.947883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'28","last_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_deep_scrub":"52'28","last_deep_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.947818+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:20.063878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386300+0000","last_change":"2024-04-16T19:24:35.401509+0000","last_active":"2024-04-16T21:10:54.386300+0000","last_peered":"2024-04-16T21:10:54.386300+0000","last_clean":"2024-04-16T21:10:54.386300+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:10:54.386300+0000","last_undegraded":"2024-04-16T21:10:54.386300+0000","last_fullsized":"2024-04-16T21:10:54.386300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383706+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:10:44.383706+0000","last_peered":"2024-04-16T21:10:44.383706+0000","last_clean":"2024-04-16T21:10:44.383706+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:10:44.383706+0000","last_undegraded":"2024-04-16T21:10:44.383706+0000","last_fullsized":"2024-04-16T21:10:44.383706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'12","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.624929+0000","last_change":"2024-04-16T21:11:55.624974+0000","last_active":"2024-04-16T21:11:55.624929+0000","last_peered":"2024-04-16T21:11:55.624929+0000","last_clean":"2024-04-16T21:11:55.624929+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:11:55.624929+0000","last_undegraded":"2024-04-16T21:11:55.624929+0000","last_fullsized":"2024-04-16T21:11:55.624929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.624828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:36.218593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385469+0000","last_change":"2024-04-16T19:24:35.400686+0000","last_active":"2024-04-16T21:10:54.385469+0000","last_peered":"2024-04-16T21:10:54.385469+0000","last_clean":"2024-04-16T21:10:54.385469+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:10:54.385469+0000","last_undegraded":"2024-04-16T21:10:54.385469+0000","last_fullsized":"2024-04-16T21:10:54.385469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.385174+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:10:44.385174+0000","last_peered":"2024-04-16T21:10:44.385174+0000","last_clean":"2024-04-16T21:10:44.385174+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:10:44.385174+0000","last_undegraded":"2024-04-16T21:10:44.385174+0000","last_fullsized":"2024-04-16T21:10:44.385174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'26","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.939446+0000","last_change":"2024-04-16T21:12:15.939510+0000","last_active":"2024-04-16T21:12:15.939446+0000","last_peered":"2024-04-16T21:12:15.939446+0000","last_clean":"2024-04-16T21:12:15.939446+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:12:15.939446+0000","last_undegraded":"2024-04-16T21:12:15.939446+0000","last_fullsized":"2024-04-16T21:12:15.939446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.939383+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:49:32.014798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386679+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:10:54.386679+0000","last_peered":"2024-04-16T21:10:54.386679+0000","last_clean":"2024-04-16T21:10:54.386679+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:10:54.386679+0000","last_undegraded":"2024-04-16T21:10:54.386679+0000","last_fullsized":"2024-04-16T21:10:54.386679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.524026+0000","last_change":"2024-04-16T19:24:33.392662+0000","last_active":"2024-04-16T21:10:44.524026+0000","last_peered":"2024-04-16T21:10:44.524026+0000","last_clean":"2024-04-16T21:10:44.524026+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:10:44.524026+0000","last_undegraded":"2024-04-16T21:10:44.524026+0000","last_fullsized":"2024-04-16T21:10:44.524026+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'46","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.408711+0000","last_change":"2024-04-16T21:11:46.408756+0000","last_active":"2024-04-16T21:11:46.408711+0000","last_peered":"2024-04-16T21:11:46.408711+0000","last_clean":"2024-04-16T21:11:46.408711+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:11:46.408711+0000","last_undegraded":"2024-04-16T21:11:46.408711+0000","last_fullsized":"2024-04-16T21:11:46.408711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'46","last_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_deep_scrub":"52'46","last_deep_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.408648+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:38.312038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292194+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:11:42.292194+0000","last_peered":"2024-04-16T21:11:42.292194+0000","last_clean":"2024-04-16T21:11:42.292194+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:11:42.292194+0000","last_undegraded":"2024-04-16T21:11:42.292194+0000","last_fullsized":"2024-04-16T21:11:42.292194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387643+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:11:04.387643+0000","last_peered":"2024-04-16T21:11:04.387643+0000","last_clean":"2024-04-16T21:11:04.387643+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:11:04.387643+0000","last_undegraded":"2024-04-16T21:11:04.387643+0000","last_fullsized":"2024-04-16T21:11:04.387643+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'20","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.919349+0000","last_change":"2024-04-16T21:11:47.919430+0000","last_active":"2024-04-16T21:11:47.919349+0000","last_peered":"2024-04-16T21:11:47.919349+0000","last_clean":"2024-04-16T21:11:47.919349+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:11:47.919349+0000","last_undegraded":"2024-04-16T21:11:47.919349+0000","last_fullsized":"2024-04-16T21:11:47.919349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.919276+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:11:50.792760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106421+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:11:44.106421+0000","last_peered":"2024-04-16T21:11:44.106421+0000","last_clean":"2024-04-16T21:11:44.106421+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:11:44.106421+0000","last_undegraded":"2024-04-16T21:11:44.106421+0000","last_fullsized":"2024-04-16T21:11:44.106421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.385533+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:10:19.385533+0000","last_peered":"2024-04-16T21:10:19.385533+0000","last_clean":"2024-04-16T21:10:19.385533+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:10:19.385533+0000","last_undegraded":"2024-04-16T21:10:19.385533+0000","last_fullsized":"2024-04-16T21:10:19.385533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'17","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.209595+0000","last_change":"2024-04-16T21:11:47.209653+0000","last_active":"2024-04-16T21:11:47.209595+0000","last_peered":"2024-04-16T21:11:47.209595+0000","last_clean":"2024-04-16T21:11:47.209595+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:11:47.209595+0000","last_undegraded":"2024-04-16T21:11:47.209595+0000","last_fullsized":"2024-04-16T21:11:47.209595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.209532+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:30.862167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.523274+0000","last_change":"2024-04-16T19:24:35.395328+0000","last_active":"2024-04-16T21:10:39.523274+0000","last_peered":"2024-04-16T21:10:39.523274+0000","last_clean":"2024-04-16T21:10:39.523274+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:10:39.523274+0000","last_undegraded":"2024-04-16T21:10:39.523274+0000","last_fullsized":"2024-04-16T21:10:39.523274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380884+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:10:19.380884+0000","last_peered":"2024-04-16T21:10:19.380884+0000","last_clean":"2024-04-16T21:10:19.380884+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:10:19.380884+0000","last_undegraded":"2024-04-16T21:10:19.380884+0000","last_fullsized":"2024-04-16T21:10:19.380884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'27","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.894735+0000","last_change":"2024-04-16T21:11:49.894811+0000","last_active":"2024-04-16T21:11:49.894735+0000","last_peered":"2024-04-16T21:11:49.894735+0000","last_clean":"2024-04-16T21:11:49.894735+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:11:49.894735+0000","last_undegraded":"2024-04-16T21:11:49.894735+0000","last_fullsized":"2024-04-16T21:11:49.894735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.894658+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.290532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.381700+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:10:24.381700+0000","last_peered":"2024-04-16T21:10:24.381700+0000","last_clean":"2024-04-16T21:10:24.381700+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:10:24.381700+0000","last_undegraded":"2024-04-16T21:10:24.381700+0000","last_fullsized":"2024-04-16T21:10:24.381700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.863200+0000","last_change":"2024-04-16T21:12:05.863236+0000","last_active":"2024-04-16T21:12:05.863200+0000","last_peered":"2024-04-16T21:12:05.863200+0000","last_clean":"2024-04-16T21:12:05.863200+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:12:05.863200+0000","last_undegraded":"2024-04-16T21:12:05.863200+0000","last_fullsized":"2024-04-16T21:12:05.863200+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.863135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:16.509928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'10","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.658621+0000","last_change":"2024-04-16T21:11:58.658671+0000","last_active":"2024-04-16T21:11:58.658621+0000","last_peered":"2024-04-16T21:11:58.658621+0000","last_clean":"2024-04-16T21:11:58.658621+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:11:58.658621+0000","last_undegraded":"2024-04-16T21:11:58.658621+0000","last_fullsized":"2024-04-16T21:11:58.658621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.658555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:31.251581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380887+0000","last_change":"2024-04-16T19:24:35.398818+0000","last_active":"2024-04-16T21:10:24.380887+0000","last_peered":"2024-04-16T21:10:24.380887+0000","last_clean":"2024-04-16T21:10:24.380887+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:10:24.380887+0000","last_undegraded":"2024-04-16T21:10:24.380887+0000","last_fullsized":"2024-04-16T21:10:24.380887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:19.380933+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:10:19.380933+0000","last_peered":"2024-04-16T21:10:19.380933+0000","last_clean":"2024-04-16T21:10:19.380933+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:10:19.380933+0000","last_undegraded":"2024-04-16T21:10:19.380933+0000","last_fullsized":"2024-04-16T21:10:19.380933+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'22","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.811095+0000","last_change":"2024-04-16T21:11:53.811157+0000","last_active":"2024-04-16T21:11:53.811095+0000","last_peered":"2024-04-16T21:11:53.811095+0000","last_clean":"2024-04-16T21:11:53.811095+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:11:53.811095+0000","last_undegraded":"2024-04-16T21:11:53.811095+0000","last_fullsized":"2024-04-16T21:11:53.811095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.811028+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:31.711738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.431473+0000","last_change":"2024-04-16T21:12:12.431513+0000","last_active":"2024-04-16T21:12:12.431473+0000","last_peered":"2024-04-16T21:12:12.431473+0000","last_clean":"2024-04-16T21:12:12.431473+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:12:12.431473+0000","last_undegraded":"2024-04-16T21:12:12.431473+0000","last_fullsized":"2024-04-16T21:12:12.431473+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.431389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:06.860124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.384269+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:10:39.384269+0000","last_peered":"2024-04-16T21:10:39.384269+0000","last_clean":"2024-04-16T21:10:39.384269+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:10:39.384269+0000","last_undegraded":"2024-04-16T21:10:39.384269+0000","last_fullsized":"2024-04-16T21:10:39.384269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'6","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.660156+0000","last_change":"2024-04-16T21:12:03.660192+0000","last_active":"2024-04-16T21:12:03.660156+0000","last_peered":"2024-04-16T21:12:03.660156+0000","last_clean":"2024-04-16T21:12:03.660156+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:12:03.660156+0000","last_undegraded":"2024-04-16T21:12:03.660156+0000","last_fullsized":"2024-04-16T21:12:03.660156+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.660092+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:42.368715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:24.380355+0000","last_change":"2024-04-16T19:24:35.395875+0000","last_active":"2024-04-16T21:10:24.380355+0000","last_peered":"2024-04-16T21:10:24.380355+0000","last_clean":"2024-04-16T21:10:24.380355+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:10:24.380355+0000","last_undegraded":"2024-04-16T21:10:24.380355+0000","last_fullsized":"2024-04-16T21:10:24.380355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194855+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:11:43.194855+0000","last_peered":"2024-04-16T21:11:43.194855+0000","last_clean":"2024-04-16T21:11:43.194855+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:11:43.194855+0000","last_undegraded":"2024-04-16T21:11:43.194855+0000","last_fullsized":"2024-04-16T21:11:43.194855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'26","reported_seq":302,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.166590+0000","last_change":"2024-04-16T21:11:46.166636+0000","last_active":"2024-04-16T21:11:46.166590+0000","last_peered":"2024-04-16T21:11:46.166590+0000","last_clean":"2024-04-16T21:11:46.166590+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:11:46.166590+0000","last_undegraded":"2024-04-16T21:11:46.166590+0000","last_fullsized":"2024-04-16T21:11:46.166590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.166525+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:05.994388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389319+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:11:09.389319+0000","last_peered":"2024-04-16T21:11:09.389319+0000","last_clean":"2024-04-16T21:11:09.389319+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:11:09.389319+0000","last_undegraded":"2024-04-16T21:11:09.389319+0000","last_fullsized":"2024-04-16T21:11:09.389319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560276+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:11:44.560276+0000","last_peered":"2024-04-16T21:11:44.560276+0000","last_clean":"2024-04-16T21:11:44.560276+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:11:44.560276+0000","last_undegraded":"2024-04-16T21:11:44.560276+0000","last_fullsized":"2024-04-16T21:11:44.560276+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'22","reported_seq":227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.232256+0000","last_change":"2024-04-16T21:11:55.232333+0000","last_active":"2024-04-16T21:11:55.232256+0000","last_peered":"2024-04-16T21:11:55.232256+0000","last_clean":"2024-04-16T21:11:55.232256+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:11:55.232256+0000","last_undegraded":"2024-04-16T21:11:55.232256+0000","last_fullsized":"2024-04-16T21:11:55.232256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.232184+0000","objects_scrubbed":10,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:19.898271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650107+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:11:43.650107+0000","last_peered":"2024-04-16T21:11:43.650107+0000","last_clean":"2024-04-16T21:11:43.650107+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:11:43.650107+0000","last_undegraded":"2024-04-16T21:11:43.650107+0000","last_fullsized":"2024-04-16T21:11:43.650107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395747+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:11:19.395747+0000","last_peered":"2024-04-16T21:11:19.395747+0000","last_clean":"2024-04-16T21:11:19.395747+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:11:19.395747+0000","last_undegraded":"2024-04-16T21:11:19.395747+0000","last_fullsized":"2024-04-16T21:11:19.395747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'6","reported_seq":180,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.935999+0000","last_change":"2024-04-16T21:12:14.936034+0000","last_active":"2024-04-16T21:12:14.935999+0000","last_peered":"2024-04-16T21:12:14.935999+0000","last_clean":"2024-04-16T21:12:14.935999+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:12:14.935999+0000","last_undegraded":"2024-04-16T21:12:14.935999+0000","last_fullsized":"2024-04-16T21:12:14.935999+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.935939+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:02.653615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742979+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:11:42.742979+0000","last_peered":"2024-04-16T21:11:42.742979+0000","last_clean":"2024-04-16T21:11:42.742979+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:11:42.742979+0000","last_undegraded":"2024-04-16T21:11:42.742979+0000","last_fullsized":"2024-04-16T21:11:42.742979+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:10:59.387111+0000","last_peered":"2024-04-16T21:10:59.387111+0000","last_clean":"2024-04-16T21:10:59.387111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:10:59.387111+0000","last_undegraded":"2024-04-16T21:10:59.387111+0000","last_fullsized":"2024-04-16T21:10:59.387111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":62971,"num_omap_keys":252,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4486,"ondisk_log_size":4486,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3432,"ondisk_log_size":3432,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1190,"ondisk_log_size":1190,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16207,"ondisk_log_size":16207,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5668,"ondisk_log_size":5668,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9989,"num_omap_keys":62,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641563,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":45292,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":104812308,"statfs":{"total":107374182400,"available":107327803392,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641564,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":54924,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104802676,"statfs":{"total":107374182400,"available":107317940224,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641563,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59728,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797872,"statfs":{"total":107374182400,"available":107313020928,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641564,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":63388,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":104794212,"statfs":{"total":107374182400,"available":107309273088,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641564,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46756,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810844,"statfs":{"total":107374182400,"available":107326304256,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641563,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59960,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797640,"statfs":{"total":107374182400,"available":107312783360,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:12:25.924 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.924 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.924 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.925 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.925 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.925 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.925 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.925 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.926 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.926 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.926 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.926 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.927 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.927 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.927 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.927 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.927 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.928 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.928 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.928 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.928 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.928 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.929 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.929 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.929 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.929 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.929 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.930 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.930 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.930 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.930 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.930 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.931 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.931 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.931 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.931 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.931 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.932 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.932 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.932 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.932 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.932 INFO:tasks.ceph:pgid 14.1c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.933 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.933 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.933 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.933 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.933 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.934 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.934 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.934 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.934 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.934 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.935 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.935 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.935 INFO:tasks.ceph:pgid 14.2 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.935 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.935 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.936 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.936 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.936 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.936 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.936 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.937 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.937 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.937 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.937 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.937 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.938 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.938 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.938 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.938 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.938 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.939 INFO:tasks.ceph:pgid 13.2 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.939 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.939 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.939 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.939 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.940 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.940 INFO:tasks.ceph:pgid 14.6 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.940 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.940 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.940 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.941 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.941 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.941 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.941 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.941 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.942 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.942 INFO:tasks.ceph:pgid 14.4 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.942 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.942 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.942 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.943 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.943 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.943 INFO:tasks.ceph:pgid 14.5 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.943 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.944 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.944 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.944 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.944 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.944 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.945 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.945 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T19:23:45.108082+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.945 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.945 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.946 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.946 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.946 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.946 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.947 INFO:tasks.ceph:pgid 14.c last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.947 INFO:tasks.ceph:pgid 13.f last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.947 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.947 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.947 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.948 INFO:tasks.ceph:pgid 13.e last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.949 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.949 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.949 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.949 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.949 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.950 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.950 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.950 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.950 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.951 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.951 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.951 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.951 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.951 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.952 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.952 INFO:tasks.ceph:pgid 13.b last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.952 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.952 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.952 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.953 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.953 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.953 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.953 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.953 INFO:tasks.ceph:pgid 14.9 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.954 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.954 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.954 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.955 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.955 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.955 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.956 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.956 INFO:tasks.ceph:pgid 14.a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.957 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.957 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.957 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.958 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.958 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.958 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.959 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.959 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.960 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.960 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.960 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.961 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.961 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.962 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.962 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.963 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.963 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.963 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.963 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.963 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.964 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.964 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.964 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.964 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.964 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.965 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.965 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.965 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.965 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.965 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.966 INFO:tasks.ceph:pgid 14.1a last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.966 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.966 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.966 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.966 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.967 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.967 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.967 INFO:tasks.ceph:pgid 14.15 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.967 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.967 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 14.14 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.968 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.969 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.969 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.969 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.969 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.969 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.970 INFO:tasks.ceph:pgid 14.17 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.970 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.970 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.970 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.971 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.971 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.971 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.971 INFO:tasks.ceph:pgid 14.16 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.972 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.972 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.972 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.972 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.973 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.973 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.973 INFO:tasks.ceph:pgid 14.11 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.973 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2024-04-16T19:25:12.453581+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=12, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.974 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.974 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.974 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.974 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.974 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.975 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.975 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.975 INFO:tasks.ceph:pgid 13.13 last_scrub_stamp 2024-04-16T19:25:40.485060+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.975 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.975 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.976 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.976 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.976 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-04-16T19:24:48.412680+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.976 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.976 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-04-16T19:25:14.462387+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=14, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.977 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.977 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.977 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.977 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.978 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.978 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.978 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.978 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.979 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.979 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.979 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.979 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.979 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.980 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.980 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.980 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.980 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.980 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.981 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.982 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.982 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.982 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.983 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.983 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.983 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.983 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.984 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.984 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.984 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.984 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.984 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.985 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.985 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.985 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.985 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.985 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.986 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.986 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:25.986 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T21:12:45.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:12:46.345 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:12:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:12:46.368 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3316,"stamp":"2024-04-16T21:12:45.360162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":452071,"num_omap_keys":1979,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32815,"ondisk_log_size":32815,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":300928,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":289718,"kb_avail":628844672,"statfs":{"total":644245094400,"available":643936944128,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":296671905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":137585,"num_omap_keys":562,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005358"},"pg_stats":[{"pgid":"9.35","version":"52'41","reported_seq":316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.577077+0000","last_change":"2024-04-16T21:11:52.577152+0000","last_active":"2024-04-16T21:11:52.577077+0000","last_peered":"2024-04-16T21:11:52.577077+0000","last_clean":"2024-04-16T21:11:52.577077+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:11:52.577077+0000","last_undegraded":"2024-04-16T21:11:52.577077+0000","last_fullsized":"2024-04-16T21:11:52.577077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'41","last_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_deep_scrub":"52'41","last_deep_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.577012+0000","objects_scrubbed":13,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:52:59.327819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.386010+0000","last_change":"2024-04-16T19:24:33.393092+0000","last_active":"2024-04-16T21:10:49.386010+0000","last_peered":"2024-04-16T21:10:49.386010+0000","last_clean":"2024-04-16T21:10:49.386010+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:10:49.386010+0000","last_undegraded":"2024-04-16T21:10:49.386010+0000","last_fullsized":"2024-04-16T21:10:49.386010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560425+0000","last_change":"2024-04-16T19:24:35.397174+0000","last_active":"2024-04-16T21:11:44.560425+0000","last_peered":"2024-04-16T21:11:44.560425+0000","last_clean":"2024-04-16T21:11:44.560425+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:11:44.560425+0000","last_undegraded":"2024-04-16T21:11:44.560425+0000","last_fullsized":"2024-04-16T21:11:44.560425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"52'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.722895+0000","last_change":"2024-04-16T21:11:56.722928+0000","last_active":"2024-04-16T21:11:56.722895+0000","last_peered":"2024-04-16T21:11:56.722895+0000","last_clean":"2024-04-16T21:11:56.722895+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:11:56.722895+0000","last_undegraded":"2024-04-16T21:11:56.722895+0000","last_fullsized":"2024-04-16T21:11:56.722895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.722838+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:40.375473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384626+0000","last_change":"2024-04-16T19:24:33.384083+0000","last_active":"2024-04-16T21:10:49.384626+0000","last_peered":"2024-04-16T21:10:49.384626+0000","last_clean":"2024-04-16T21:10:49.384626+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:10:49.384626+0000","last_undegraded":"2024-04-16T21:10:49.384626+0000","last_fullsized":"2024-04-16T21:10:49.384626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650348+0000","last_change":"2024-04-16T19:24:35.393355+0000","last_active":"2024-04-16T21:11:43.650348+0000","last_peered":"2024-04-16T21:11:43.650348+0000","last_clean":"2024-04-16T21:11:43.650348+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:11:43.650348+0000","last_undegraded":"2024-04-16T21:11:43.650348+0000","last_fullsized":"2024-04-16T21:11:43.650348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'22","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.702468+0000","last_change":"2024-04-16T21:12:14.702518+0000","last_active":"2024-04-16T21:12:14.702468+0000","last_peered":"2024-04-16T21:12:14.702468+0000","last_clean":"2024-04-16T21:12:14.702468+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:12:14.702468+0000","last_undegraded":"2024-04-16T21:12:14.702468+0000","last_fullsized":"2024-04-16T21:12:14.702468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.702406+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:42.760555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390655+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:10:49.390655+0000","last_peered":"2024-04-16T21:10:49.390655+0000","last_clean":"2024-04-16T21:10:49.390655+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:10:49.390655+0000","last_undegraded":"2024-04-16T21:10:49.390655+0000","last_fullsized":"2024-04-16T21:10:49.390655+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.701798+0000","last_change":"2024-04-16T21:12:32.701821+0000","last_active":"2024-04-16T21:12:32.701798+0000","last_peered":"2024-04-16T21:12:32.701798+0000","last_clean":"2024-04-16T21:12:32.701798+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:12:32.701798+0000","last_undegraded":"2024-04-16T21:12:32.701798+0000","last_fullsized":"2024-04-16T21:12:32.701798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.701734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:27.130470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'22","reported_seq":252,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.610592+0000","last_change":"2024-04-16T21:11:47.610645+0000","last_active":"2024-04-16T21:11:47.610592+0000","last_peered":"2024-04-16T21:11:47.610592+0000","last_clean":"2024-04-16T21:11:47.610592+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:11:47.610592+0000","last_undegraded":"2024-04-16T21:11:47.610592+0000","last_fullsized":"2024-04-16T21:11:47.610592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.610532+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:13.030892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386247+0000","last_change":"2024-04-16T19:24:33.389974+0000","last_active":"2024-04-16T21:10:54.386247+0000","last_peered":"2024-04-16T21:10:54.386247+0000","last_clean":"2024-04-16T21:10:54.386247+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:10:54.386247+0000","last_undegraded":"2024-04-16T21:10:54.386247+0000","last_fullsized":"2024-04-16T21:10:54.386247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292197+0000","last_change":"2024-04-16T19:24:35.395080+0000","last_active":"2024-04-16T21:11:42.292197+0000","last_peered":"2024-04-16T21:11:42.292197+0000","last_clean":"2024-04-16T21:11:42.292197+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:11:42.292197+0000","last_undegraded":"2024-04-16T21:11:42.292197+0000","last_fullsized":"2024-04-16T21:11:42.292197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'12","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.719715+0000","last_change":"2024-04-16T21:11:58.719715+0000","last_active":"2024-04-16T21:11:58.719715+0000","last_peered":"2024-04-16T21:11:58.719715+0000","last_clean":"2024-04-16T21:11:58.719715+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:11:58.719715+0000","last_undegraded":"2024-04-16T21:11:58.719715+0000","last_fullsized":"2024-04-16T21:11:58.719715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.719610+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:50.501313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650162+0000","last_change":"2024-04-16T19:24:33.387737+0000","last_active":"2024-04-16T21:11:43.650162+0000","last_peered":"2024-04-16T21:11:43.650162+0000","last_clean":"2024-04-16T21:11:43.650162+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:11:43.650162+0000","last_undegraded":"2024-04-16T21:11:43.650162+0000","last_fullsized":"2024-04-16T21:11:43.650162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387029+0000","last_change":"2024-04-16T19:24:35.400622+0000","last_active":"2024-04-16T21:10:59.387029+0000","last_peered":"2024-04-16T21:10:59.387029+0000","last_clean":"2024-04-16T21:10:59.387029+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:10:59.387029+0000","last_undegraded":"2024-04-16T21:10:59.387029+0000","last_fullsized":"2024-04-16T21:10:59.387029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'27","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.166108+0000","last_change":"2024-04-16T21:11:51.166108+0000","last_active":"2024-04-16T21:11:51.166108+0000","last_peered":"2024-04-16T21:11:51.166108+0000","last_clean":"2024-04-16T21:11:51.166108+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:11:51.166108+0000","last_undegraded":"2024-04-16T21:11:51.166108+0000","last_fullsized":"2024-04-16T21:11:51.166108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.165987+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:01.812779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560300+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:11:44.560300+0000","last_peered":"2024-04-16T21:11:44.560300+0000","last_clean":"2024-04-16T21:11:44.560300+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:11:44.560300+0000","last_undegraded":"2024-04-16T21:11:44.560300+0000","last_fullsized":"2024-04-16T21:11:44.560300+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.679775+0000","last_change":"2024-04-16T21:12:34.679817+0000","last_active":"2024-04-16T21:12:34.679775+0000","last_peered":"2024-04-16T21:12:34.679775+0000","last_clean":"2024-04-16T21:12:34.679775+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:12:34.679775+0000","last_undegraded":"2024-04-16T21:12:34.679775+0000","last_fullsized":"2024-04-16T21:12:34.679775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.679697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:57.367334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'18","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.672517+0000","last_change":"2024-04-16T21:12:09.672592+0000","last_active":"2024-04-16T21:12:09.672517+0000","last_peered":"2024-04-16T21:12:09.672517+0000","last_clean":"2024-04-16T21:12:09.672517+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:12:09.672517+0000","last_undegraded":"2024-04-16T21:12:09.672517+0000","last_fullsized":"2024-04-16T21:12:09.672517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.672453+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.111035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650158+0000","last_change":"2024-04-16T19:24:33.393346+0000","last_active":"2024-04-16T21:11:43.650158+0000","last_peered":"2024-04-16T21:11:43.650158+0000","last_clean":"2024-04-16T21:11:43.650158+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:11:43.650158+0000","last_undegraded":"2024-04-16T21:11:43.650158+0000","last_fullsized":"2024-04-16T21:11:43.650158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386334+0000","last_change":"2024-04-16T19:24:35.396080+0000","last_active":"2024-04-16T21:10:54.386334+0000","last_peered":"2024-04-16T21:10:54.386334+0000","last_clean":"2024-04-16T21:10:54.386334+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:10:54.386334+0000","last_undegraded":"2024-04-16T21:10:54.386334+0000","last_fullsized":"2024-04-16T21:10:54.386334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'22","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.654742+0000","last_change":"2024-04-16T21:11:56.654830+0000","last_active":"2024-04-16T21:11:56.654742+0000","last_peered":"2024-04-16T21:11:56.654742+0000","last_clean":"2024-04-16T21:11:56.654742+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:11:56.654742+0000","last_undegraded":"2024-04-16T21:11:56.654742+0000","last_fullsized":"2024-04-16T21:11:56.654742+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.654666+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.077668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194861+0000","last_change":"2024-04-16T19:24:33.390382+0000","last_active":"2024-04-16T21:11:43.194861+0000","last_peered":"2024-04-16T21:11:43.194861+0000","last_clean":"2024-04-16T21:11:43.194861+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:11:43.194861+0000","last_undegraded":"2024-04-16T21:11:43.194861+0000","last_fullsized":"2024-04-16T21:11:43.194861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.815439+0000","last_change":"2024-04-16T21:12:39.815463+0000","last_active":"2024-04-16T21:12:39.815439+0000","last_peered":"2024-04-16T21:12:39.815439+0000","last_clean":"2024-04-16T21:12:39.815439+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:12:39.815439+0000","last_undegraded":"2024-04-16T21:12:39.815439+0000","last_fullsized":"2024-04-16T21:12:39.815439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.815374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:11.502076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'12","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.701548+0000","last_change":"2024-04-16T21:12:00.701594+0000","last_active":"2024-04-16T21:12:00.701548+0000","last_peered":"2024-04-16T21:12:00.701548+0000","last_clean":"2024-04-16T21:12:00.701548+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:12:00.701548+0000","last_undegraded":"2024-04-16T21:12:00.701548+0000","last_fullsized":"2024-04-16T21:12:00.701548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.701488+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:22.203073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.707007+0000","last_change":"2024-04-16T21:12:38.707030+0000","last_active":"2024-04-16T21:12:38.707007+0000","last_peered":"2024-04-16T21:12:38.707007+0000","last_clean":"2024-04-16T21:12:38.707007+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:12:38.707007+0000","last_undegraded":"2024-04-16T21:12:38.707007+0000","last_fullsized":"2024-04-16T21:12:38.707007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.706933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:10.393634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390309+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:11:14.390309+0000","last_peered":"2024-04-16T21:11:14.390309+0000","last_clean":"2024-04-16T21:11:14.390309+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:11:14.390309+0000","last_undegraded":"2024-04-16T21:11:14.390309+0000","last_fullsized":"2024-04-16T21:11:14.390309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'12","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.436184+0000","last_change":"2024-04-16T21:11:55.436228+0000","last_active":"2024-04-16T21:11:55.436184+0000","last_peered":"2024-04-16T21:11:55.436184+0000","last_clean":"2024-04-16T21:11:55.436184+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:11:55.436184+0000","last_undegraded":"2024-04-16T21:11:55.436184+0000","last_fullsized":"2024-04-16T21:11:55.436184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.436126+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:58.763911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.914098+0000","last_change":"2024-04-16T21:12:16.914127+0000","last_active":"2024-04-16T21:12:16.914098+0000","last_peered":"2024-04-16T21:12:16.914098+0000","last_clean":"2024-04-16T21:12:16.914098+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:12:16.914098+0000","last_undegraded":"2024-04-16T21:12:16.914098+0000","last_fullsized":"2024-04-16T21:12:16.914098+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.914034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:05.877583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386917+0000","last_change":"2024-04-16T19:24:33.387958+0000","last_active":"2024-04-16T21:10:54.386917+0000","last_peered":"2024-04-16T21:10:54.386917+0000","last_clean":"2024-04-16T21:10:54.386917+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:10:54.386917+0000","last_undegraded":"2024-04-16T21:10:54.386917+0000","last_fullsized":"2024-04-16T21:10:54.386917+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'10","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.882687+0000","last_change":"2024-04-16T21:11:51.882767+0000","last_active":"2024-04-16T21:11:51.882687+0000","last_peered":"2024-04-16T21:11:51.882687+0000","last_clean":"2024-04-16T21:11:51.882687+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:11:51.882687+0000","last_undegraded":"2024-04-16T21:11:51.882687+0000","last_fullsized":"2024-04-16T21:11:51.882687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.882609+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.077377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560427+0000","last_change":"2024-04-16T19:24:35.396129+0000","last_active":"2024-04-16T21:11:44.560427+0000","last_peered":"2024-04-16T21:11:44.560427+0000","last_clean":"2024-04-16T21:11:44.560427+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:11:44.560427+0000","last_undegraded":"2024-04-16T21:11:44.560427+0000","last_fullsized":"2024-04-16T21:11:44.560427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386292+0000","last_change":"2024-04-16T19:24:33.386706+0000","last_active":"2024-04-16T21:10:54.386292+0000","last_peered":"2024-04-16T21:10:54.386292+0000","last_clean":"2024-04-16T21:10:54.386292+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:10:54.386292+0000","last_undegraded":"2024-04-16T21:10:54.386292+0000","last_fullsized":"2024-04-16T21:10:54.386292+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'20","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.881532+0000","last_change":"2024-04-16T21:12:07.881609+0000","last_active":"2024-04-16T21:12:07.881532+0000","last_peered":"2024-04-16T21:12:07.881532+0000","last_clean":"2024-04-16T21:12:07.881532+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:12:07.881532+0000","last_undegraded":"2024-04-16T21:12:07.881532+0000","last_fullsized":"2024-04-16T21:12:07.881532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.881467+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.576312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.819466+0000","last_change":"2024-04-16T21:12:33.819489+0000","last_active":"2024-04-16T21:12:33.819466+0000","last_peered":"2024-04-16T21:12:33.819466+0000","last_clean":"2024-04-16T21:12:33.819466+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:12:33.819466+0000","last_undegraded":"2024-04-16T21:12:33.819466+0000","last_fullsized":"2024-04-16T21:12:33.819466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.819396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:28.248131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386361+0000","last_change":"2024-04-16T19:24:33.386562+0000","last_active":"2024-04-16T21:10:54.386361+0000","last_peered":"2024-04-16T21:10:54.386361+0000","last_clean":"2024-04-16T21:10:54.386361+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:10:54.386361+0000","last_undegraded":"2024-04-16T21:10:54.386361+0000","last_fullsized":"2024-04-16T21:10:54.386361+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'16","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.428185+0000","last_change":"2024-04-16T21:11:47.428185+0000","last_active":"2024-04-16T21:11:47.428185+0000","last_peered":"2024-04-16T21:11:47.428185+0000","last_clean":"2024-04-16T21:11:47.428185+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:11:47.428185+0000","last_undegraded":"2024-04-16T21:11:47.428185+0000","last_fullsized":"2024-04-16T21:11:47.428185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.428059+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:23.278965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.812506+0000","last_change":"2024-04-16T21:12:37.812549+0000","last_active":"2024-04-16T21:12:37.812506+0000","last_peered":"2024-04-16T21:12:37.812506+0000","last_clean":"2024-04-16T21:12:37.812506+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:12:37.812506+0000","last_undegraded":"2024-04-16T21:12:37.812506+0000","last_fullsized":"2024-04-16T21:12:37.812506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.812425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:29.486114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391334+0000","last_change":"2024-04-16T19:24:33.391793+0000","last_active":"2024-04-16T21:10:54.391334+0000","last_peered":"2024-04-16T21:10:54.391334+0000","last_clean":"2024-04-16T21:10:54.391334+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:10:54.391334+0000","last_undegraded":"2024-04-16T21:10:54.391334+0000","last_fullsized":"2024-04-16T21:10:54.391334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'22","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.947975+0000","last_change":"2024-04-16T21:12:11.947975+0000","last_active":"2024-04-16T21:12:11.947975+0000","last_peered":"2024-04-16T21:12:11.947975+0000","last_clean":"2024-04-16T21:12:11.947975+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:12:11.947975+0000","last_undegraded":"2024-04-16T21:12:11.947975+0000","last_fullsized":"2024-04-16T21:12:11.947975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.947824+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:04.586823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387165+0000","last_change":"2024-04-16T19:24:35.397475+0000","last_active":"2024-04-16T21:10:59.387165+0000","last_peered":"2024-04-16T21:10:59.387165+0000","last_clean":"2024-04-16T21:10:59.387165+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:10:59.387165+0000","last_undegraded":"2024-04-16T21:10:59.387165+0000","last_fullsized":"2024-04-16T21:10:59.387165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292016+0000","last_change":"2024-04-16T19:24:33.382424+0000","last_active":"2024-04-16T21:11:42.292016+0000","last_peered":"2024-04-16T21:11:42.292016+0000","last_clean":"2024-04-16T21:11:42.292016+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:11:42.292016+0000","last_undegraded":"2024-04-16T21:11:42.292016+0000","last_fullsized":"2024-04-16T21:11:42.292016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385608+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:10:54.385608+0000","last_peered":"2024-04-16T21:10:54.385608+0000","last_clean":"2024-04-16T21:10:54.385608+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:10:54.385608+0000","last_undegraded":"2024-04-16T21:10:54.385608+0000","last_fullsized":"2024-04-16T21:10:54.385608+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1722,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651474+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:11:43.651474+0000","last_peered":"2024-04-16T21:11:43.651474+0000","last_clean":"2024-04-16T21:11:43.651474+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:11:43.651474+0000","last_undegraded":"2024-04-16T21:11:43.651474+0000","last_fullsized":"2024-04-16T21:11:43.651474+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.395179+0000","last_change":"2024-04-16T21:12:15.395220+0000","last_active":"2024-04-16T21:12:15.395179+0000","last_peered":"2024-04-16T21:12:15.395179+0000","last_clean":"2024-04-16T21:12:15.395179+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:12:15.395179+0000","last_undegraded":"2024-04-16T21:12:15.395179+0000","last_fullsized":"2024-04-16T21:12:15.395179+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.395104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:09.823839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'8","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.430759+0000","last_change":"2024-04-16T21:12:17.430800+0000","last_active":"2024-04-16T21:12:17.430759+0000","last_peered":"2024-04-16T21:12:17.430759+0000","last_clean":"2024-04-16T21:12:17.430759+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:12:17.430759+0000","last_undegraded":"2024-04-16T21:12:17.430759+0000","last_fullsized":"2024-04-16T21:12:17.430759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.430693+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:25.924925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.738465+0000","last_change":"2024-04-16T21:11:45.738465+0000","last_active":"2024-04-16T21:11:45.738465+0000","last_peered":"2024-04-16T21:11:45.738465+0000","last_clean":"2024-04-16T21:11:45.738465+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:11:45.738465+0000","last_undegraded":"2024-04-16T21:11:45.738465+0000","last_fullsized":"2024-04-16T21:11:45.738465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.738344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:26.908264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'223","reported_seq":477,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.490168+0000","last_change":"2024-04-16T21:11:58.490168+0000","last_active":"2024-04-16T21:11:58.490168+0000","last_peered":"2024-04-16T21:11:58.490168+0000","last_clean":"2024-04-16T21:11:58.490168+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:11:58.490168+0000","last_undegraded":"2024-04-16T21:11:58.490168+0000","last_fullsized":"2024-04-16T21:11:58.490168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'223","last_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_deep_scrub":"52'223","last_deep_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.490053+0000","objects_scrubbed":20,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:48.708953+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1971,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'29","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.926237+0000","last_change":"2024-04-16T21:11:50.926284+0000","last_active":"2024-04-16T21:11:50.926237+0000","last_peered":"2024-04-16T21:11:50.926237+0000","last_clean":"2024-04-16T21:11:50.926237+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:11:50.926237+0000","last_undegraded":"2024-04-16T21:11:50.926237+0000","last_fullsized":"2024-04-16T21:11:50.926237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'29","last_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_deep_scrub":"52'29","last_deep_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.926173+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:17.509632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.689691+0000","last_change":"2024-04-16T21:12:19.689715+0000","last_active":"2024-04-16T21:12:19.689691+0000","last_peered":"2024-04-16T21:12:19.689691+0000","last_clean":"2024-04-16T21:12:19.689691+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:12:19.689691+0000","last_undegraded":"2024-04-16T21:12:19.689691+0000","last_fullsized":"2024-04-16T21:12:19.689691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.689621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.705707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.031454+0000","last_change":"2024-04-16T21:12:40.031477+0000","last_active":"2024-04-16T21:12:40.031454+0000","last_peered":"2024-04-16T21:12:40.031454+0000","last_clean":"2024-04-16T21:12:40.031454+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:12:40.031454+0000","last_undegraded":"2024-04-16T21:12:40.031454+0000","last_fullsized":"2024-04-16T21:12:40.031454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.031394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:26.329198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649927+0000","last_change":"2024-04-16T19:24:33.387104+0000","last_active":"2024-04-16T21:11:43.649927+0000","last_peered":"2024-04-16T21:11:43.649927+0000","last_clean":"2024-04-16T21:11:43.649927+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:11:43.649927+0000","last_undegraded":"2024-04-16T21:11:43.649927+0000","last_fullsized":"2024-04-16T21:11:43.649927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'6","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.901446+0000","last_change":"2024-04-16T21:12:42.901472+0000","last_active":"2024-04-16T21:12:42.901446+0000","last_peered":"2024-04-16T21:12:42.901446+0000","last_clean":"2024-04-16T21:12:42.901446+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:12:42.901446+0000","last_undegraded":"2024-04-16T21:12:42.901446+0000","last_fullsized":"2024-04-16T21:12:42.901446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.901389+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:59:18.553914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.351884+0000","last_change":"2024-04-16T21:12:33.351925+0000","last_active":"2024-04-16T21:12:33.351884+0000","last_peered":"2024-04-16T21:12:33.351884+0000","last_clean":"2024-04-16T21:12:33.351884+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:12:33.351884+0000","last_undegraded":"2024-04-16T21:12:33.351884+0000","last_fullsized":"2024-04-16T21:12:33.351884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.351821+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:30.327261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.757836+0000","last_change":"2024-04-16T21:12:30.757878+0000","last_active":"2024-04-16T21:12:30.757836+0000","last_peered":"2024-04-16T21:12:30.757836+0000","last_clean":"2024-04-16T21:12:30.757836+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:12:30.757836+0000","last_undegraded":"2024-04-16T21:12:30.757836+0000","last_fullsized":"2024-04-16T21:12:30.757836+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.757760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:42.360430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.196037+0000","last_change":"2024-04-16T19:47:33.988985+0000","last_active":"2024-04-16T21:11:43.196037+0000","last_peered":"2024-04-16T21:11:43.196037+0000","last_clean":"2024-04-16T21:11:43.196037+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:11:43.196037+0000","last_undegraded":"2024-04-16T21:11:43.196037+0000","last_fullsized":"2024-04-16T21:11:43.196037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1837,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386348+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:10:54.386348+0000","last_peered":"2024-04-16T21:10:54.386348+0000","last_clean":"2024-04-16T21:10:54.386348+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:10:54.386348+0000","last_undegraded":"2024-04-16T21:10:54.386348+0000","last_fullsized":"2024-04-16T21:10:54.386348+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.873357+0000","last_change":"2024-04-16T21:12:15.873380+0000","last_active":"2024-04-16T21:12:15.873357+0000","last_peered":"2024-04-16T21:12:15.873357+0000","last_clean":"2024-04-16T21:12:15.873357+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:12:15.873357+0000","last_undegraded":"2024-04-16T21:12:15.873357+0000","last_fullsized":"2024-04-16T21:12:15.873357+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.873260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:01.509024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'8","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.369110+0000","last_change":"2024-04-16T21:12:14.369164+0000","last_active":"2024-04-16T21:12:14.369110+0000","last_peered":"2024-04-16T21:12:14.369110+0000","last_clean":"2024-04-16T21:12:14.369110+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:12:14.369110+0000","last_undegraded":"2024-04-16T21:12:14.369110+0000","last_fullsized":"2024-04-16T21:12:14.369110+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.369033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:29.838406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.686990+0000","last_change":"2024-04-16T21:11:57.686990+0000","last_active":"2024-04-16T21:11:57.686990+0000","last_peered":"2024-04-16T21:11:57.686990+0000","last_clean":"2024-04-16T21:11:57.686990+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:11:57.686990+0000","last_undegraded":"2024-04-16T21:11:57.686990+0000","last_fullsized":"2024-04-16T21:11:57.686990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.686888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:39.802344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'264","reported_seq":1009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.984786+0000","last_change":"2024-04-16T21:11:43.984786+0000","last_active":"2024-04-16T21:11:43.984786+0000","last_peered":"2024-04-16T21:11:43.984786+0000","last_clean":"2024-04-16T21:11:43.984786+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:11:43.984786+0000","last_undegraded":"2024-04-16T21:11:43.984786+0000","last_fullsized":"2024-04-16T21:11:43.984786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.984688+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:21.448843+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6375,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'10","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.702465+0000","last_change":"2024-04-16T21:11:54.702515+0000","last_active":"2024-04-16T21:11:54.702465+0000","last_peered":"2024-04-16T21:11:54.702465+0000","last_clean":"2024-04-16T21:11:54.702465+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:11:54.702465+0000","last_undegraded":"2024-04-16T21:11:54.702465+0000","last_fullsized":"2024-04-16T21:11:54.702465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.702406+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.795058+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.815226+0000","last_change":"2024-04-16T21:12:10.815250+0000","last_active":"2024-04-16T21:12:10.815226+0000","last_peered":"2024-04-16T21:12:10.815226+0000","last_clean":"2024-04-16T21:12:10.815226+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:12:10.815226+0000","last_undegraded":"2024-04-16T21:12:10.815226+0000","last_fullsized":"2024-04-16T21:12:10.815226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.815114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:27:54.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387060+0000","last_change":"2024-04-16T19:24:35.397956+0000","last_active":"2024-04-16T21:10:59.387060+0000","last_peered":"2024-04-16T21:10:59.387060+0000","last_clean":"2024-04-16T21:10:59.387060+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:10:59.387060+0000","last_undegraded":"2024-04-16T21:10:59.387060+0000","last_fullsized":"2024-04-16T21:10:59.387060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291858+0000","last_change":"2024-04-16T19:24:33.392118+0000","last_active":"2024-04-16T21:11:42.291858+0000","last_peered":"2024-04-16T21:11:42.291858+0000","last_clean":"2024-04-16T21:11:42.291858+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:11:42.291858+0000","last_undegraded":"2024-04-16T21:11:42.291858+0000","last_fullsized":"2024-04-16T21:11:42.291858+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292295+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:11:42.292295+0000","last_peered":"2024-04-16T21:11:42.292295+0000","last_clean":"2024-04-16T21:11:42.292295+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:11:42.292295+0000","last_undegraded":"2024-04-16T21:11:42.292295+0000","last_fullsized":"2024-04-16T21:11:42.292295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'69","reported_seq":219,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.619780+0000","last_change":"2024-04-16T21:12:28.619811+0000","last_active":"2024-04-16T21:12:28.619780+0000","last_peered":"2024-04-16T21:12:28.619780+0000","last_clean":"2024-04-16T21:12:28.619780+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:12:28.619780+0000","last_undegraded":"2024-04-16T21:12:28.619780+0000","last_fullsized":"2024-04-16T21:12:28.619780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'69","last_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_deep_scrub":"52'69","last_deep_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.619713+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:56.443540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.604337+0000","last_change":"2024-04-16T21:12:23.604359+0000","last_active":"2024-04-16T21:12:23.604337+0000","last_peered":"2024-04-16T21:12:23.604337+0000","last_clean":"2024-04-16T21:12:23.604337+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:12:23.604337+0000","last_undegraded":"2024-04-16T21:12:23.604337+0000","last_fullsized":"2024-04-16T21:12:23.604337+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.604274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:10.724532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107642+0000","last_change":"2024-04-16T19:47:33.984561+0000","last_active":"2024-04-16T21:11:44.107642+0000","last_peered":"2024-04-16T21:11:44.107642+0000","last_clean":"2024-04-16T21:11:44.107642+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:11:44.107642+0000","last_undegraded":"2024-04-16T21:11:44.107642+0000","last_fullsized":"2024-04-16T21:11:44.107642+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'183","reported_seq":1934,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.306613+0000","last_change":"2024-04-16T21:12:38.306641+0000","last_active":"2024-04-16T21:12:38.306613+0000","last_peered":"2024-04-16T21:12:38.306613+0000","last_clean":"2024-04-16T21:12:38.306613+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:12:38.306613+0000","last_undegraded":"2024-04-16T21:12:38.306613+0000","last_fullsized":"2024-04-16T21:12:38.306613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.306555+0000","objects_scrubbed":52,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:07.296213+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":43,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.746629+0000","last_change":"2024-04-16T21:12:00.746651+0000","last_active":"2024-04-16T21:12:00.746629+0000","last_peered":"2024-04-16T21:12:00.746629+0000","last_clean":"2024-04-16T21:12:00.746629+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:12:00.746629+0000","last_undegraded":"2024-04-16T21:12:00.746629+0000","last_fullsized":"2024-04-16T21:12:00.746629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.746571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:24:45.291359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'25","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.693813+0000","last_change":"2024-04-16T21:11:57.693880+0000","last_active":"2024-04-16T21:11:57.693813+0000","last_peered":"2024-04-16T21:11:57.693813+0000","last_clean":"2024-04-16T21:11:57.693813+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:11:57.693813+0000","last_undegraded":"2024-04-16T21:11:57.693813+0000","last_fullsized":"2024-04-16T21:11:57.693813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'25","last_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_deep_scrub":"52'25","last_deep_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.693743+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.739736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.385829+0000","last_change":"2024-04-16T21:12:04.385854+0000","last_active":"2024-04-16T21:12:04.385829+0000","last_peered":"2024-04-16T21:12:04.385829+0000","last_clean":"2024-04-16T21:12:04.385829+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:12:04.385829+0000","last_undegraded":"2024-04-16T21:12:04.385829+0000","last_fullsized":"2024-04-16T21:12:04.385829+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.385746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:20.771679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'716","reported_seq":41938,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.996557+0000","last_change":"2024-04-16T21:11:45.996631+0000","last_active":"2024-04-16T21:11:45.996557+0000","last_peered":"2024-04-16T21:11:45.996557+0000","last_clean":"2024-04-16T21:11:45.996557+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:11:45.996557+0000","last_undegraded":"2024-04-16T21:11:45.996557+0000","last_fullsized":"2024-04-16T21:11:45.996557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'716","last_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_deep_scrub":"52'716","last_deep_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.996464+0000","objects_scrubbed":20,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:40.351620+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'35","reported_seq":324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.105372+0000","last_change":"2024-04-16T21:11:44.105372+0000","last_active":"2024-04-16T21:11:44.105372+0000","last_peered":"2024-04-16T21:11:44.105372+0000","last_clean":"2024-04-16T21:11:44.105372+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:11:44.105372+0000","last_undegraded":"2024-04-16T21:11:44.105372+0000","last_fullsized":"2024-04-16T21:11:44.105372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'35","last_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_deep_scrub":"52'35","last_deep_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.105287+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:44.283094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.409254+0000","last_change":"2024-04-16T21:12:03.409254+0000","last_active":"2024-04-16T21:12:03.409254+0000","last_peered":"2024-04-16T21:12:03.409254+0000","last_clean":"2024-04-16T21:12:03.409254+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:12:03.409254+0000","last_undegraded":"2024-04-16T21:12:03.409254+0000","last_fullsized":"2024-04-16T21:12:03.409254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.409137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:49.655116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392318+0000","last_change":"2024-04-16T19:24:35.396048+0000","last_active":"2024-04-16T21:10:59.392318+0000","last_peered":"2024-04-16T21:10:59.392318+0000","last_clean":"2024-04-16T21:10:59.392318+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:10:59.392318+0000","last_undegraded":"2024-04-16T21:10:59.392318+0000","last_fullsized":"2024-04-16T21:10:59.392318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649938+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:11:43.649938+0000","last_peered":"2024-04-16T21:11:43.649938+0000","last_clean":"2024-04-16T21:11:43.649938+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:11:43.649938+0000","last_undegraded":"2024-04-16T21:11:43.649938+0000","last_fullsized":"2024-04-16T21:11:43.649938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'6","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.842517+0000","last_change":"2024-04-16T21:12:38.842543+0000","last_active":"2024-04-16T21:12:38.842517+0000","last_peered":"2024-04-16T21:12:38.842517+0000","last_clean":"2024-04-16T21:12:38.842517+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:12:38.842517+0000","last_undegraded":"2024-04-16T21:12:38.842517+0000","last_fullsized":"2024-04-16T21:12:38.842517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.842450+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:47.336683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'115","reported_seq":289,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.884236+0000","last_change":"2024-04-16T21:12:25.884267+0000","last_active":"2024-04-16T21:12:25.884236+0000","last_peered":"2024-04-16T21:12:25.884236+0000","last_clean":"2024-04-16T21:12:25.884236+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:12:25.884236+0000","last_undegraded":"2024-04-16T21:12:25.884236+0000","last_fullsized":"2024-04-16T21:12:25.884236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.884183+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.270116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'13","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.370369+0000","last_change":"2024-04-16T21:12:13.370396+0000","last_active":"2024-04-16T21:12:13.370369+0000","last_peered":"2024-04-16T21:12:13.370369+0000","last_clean":"2024-04-16T21:12:13.370369+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:12:13.370369+0000","last_undegraded":"2024-04-16T21:12:13.370369+0000","last_fullsized":"2024-04-16T21:12:13.370369+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.370308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:47.137634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":74,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744200+0000","last_change":"2024-04-16T19:47:33.986306+0000","last_active":"2024-04-16T21:11:42.744200+0000","last_peered":"2024-04-16T21:11:42.744200+0000","last_clean":"2024-04-16T21:11:42.744200+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:11:42.744200+0000","last_undegraded":"2024-04-16T21:11:42.744200+0000","last_fullsized":"2024-04-16T21:11:42.744200+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1473,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390635+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:10:49.390635+0000","last_peered":"2024-04-16T21:10:49.390635+0000","last_clean":"2024-04-16T21:10:49.390635+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:10:49.390635+0000","last_undegraded":"2024-04-16T21:10:49.390635+0000","last_fullsized":"2024-04-16T21:10:49.390635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.703764+0000","last_change":"2024-04-16T21:12:02.703764+0000","last_active":"2024-04-16T21:12:02.703764+0000","last_peered":"2024-04-16T21:12:02.703764+0000","last_clean":"2024-04-16T21:12:02.703764+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:12:02.703764+0000","last_undegraded":"2024-04-16T21:12:02.703764+0000","last_fullsized":"2024-04-16T21:12:02.703764+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.703681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:59.311639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'13","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.372480+0000","last_change":"2024-04-16T21:12:18.372514+0000","last_active":"2024-04-16T21:12:18.372480+0000","last_peered":"2024-04-16T21:12:18.372480+0000","last_clean":"2024-04-16T21:12:18.372480+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:12:18.372480+0000","last_undegraded":"2024-04-16T21:12:18.372480+0000","last_fullsized":"2024-04-16T21:12:18.372480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.372404+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:27.934072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.821906+0000","last_change":"2024-04-16T21:11:59.821929+0000","last_active":"2024-04-16T21:11:59.821906+0000","last_peered":"2024-04-16T21:11:59.821906+0000","last_clean":"2024-04-16T21:11:59.821906+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:11:59.821906+0000","last_undegraded":"2024-04-16T21:11:59.821906+0000","last_fullsized":"2024-04-16T21:11:59.821906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.821840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.493143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'366","reported_seq":690,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.574732+0000","last_change":"2024-04-16T21:11:46.574732+0000","last_active":"2024-04-16T21:11:46.574732+0000","last_peered":"2024-04-16T21:11:46.574732+0000","last_clean":"2024-04-16T21:11:46.574732+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:11:46.574732+0000","last_undegraded":"2024-04-16T21:11:46.574732+0000","last_fullsized":"2024-04-16T21:11:46.574732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'366","last_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_deep_scrub":"52'366","last_deep_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.574641+0000","objects_scrubbed":27,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:24.475352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'12","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.580812+0000","last_change":"2024-04-16T21:11:44.580812+0000","last_active":"2024-04-16T21:11:44.580812+0000","last_peered":"2024-04-16T21:11:44.580812+0000","last_clean":"2024-04-16T21:11:44.580812+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:11:44.580812+0000","last_undegraded":"2024-04-16T21:11:44.580812+0000","last_fullsized":"2024-04-16T21:11:44.580812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.580695+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:57.775463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.671998+0000","last_change":"2024-04-16T21:12:15.671998+0000","last_active":"2024-04-16T21:12:15.671998+0000","last_peered":"2024-04-16T21:12:15.671998+0000","last_clean":"2024-04-16T21:12:15.671998+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:12:15.671998+0000","last_undegraded":"2024-04-16T21:12:15.671998+0000","last_fullsized":"2024-04-16T21:12:15.671998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.671910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:43.983331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392386+0000","last_change":"2024-04-16T19:24:35.399732+0000","last_active":"2024-04-16T21:10:59.392386+0000","last_peered":"2024-04-16T21:10:59.392386+0000","last_clean":"2024-04-16T21:10:59.392386+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:10:59.392386+0000","last_undegraded":"2024-04-16T21:10:59.392386+0000","last_fullsized":"2024-04-16T21:10:59.392386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560171+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:11:44.560171+0000","last_peered":"2024-04-16T21:11:44.560171+0000","last_clean":"2024-04-16T21:11:44.560171+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:11:44.560171+0000","last_undegraded":"2024-04-16T21:11:44.560171+0000","last_fullsized":"2024-04-16T21:11:44.560171+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.239994+0000","last_change":"2024-04-16T21:12:38.240020+0000","last_active":"2024-04-16T21:12:38.239994+0000","last_peered":"2024-04-16T21:12:38.239994+0000","last_clean":"2024-04-16T21:12:38.239994+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:12:38.239994+0000","last_undegraded":"2024-04-16T21:12:38.239994+0000","last_fullsized":"2024-04-16T21:12:38.239994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.239920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:04.921574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.327857+0000","last_change":"2024-04-16T21:12:33.327890+0000","last_active":"2024-04-16T21:12:33.327857+0000","last_peered":"2024-04-16T21:12:33.327857+0000","last_clean":"2024-04-16T21:12:33.327857+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:12:33.327857+0000","last_undegraded":"2024-04-16T21:12:33.327857+0000","last_fullsized":"2024-04-16T21:12:33.327857+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.327792+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:56.594542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'13","reported_seq":1320,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.937025+0000","last_change":"2024-04-16T21:12:24.937025+0000","last_active":"2024-04-16T21:12:24.937025+0000","last_peered":"2024-04-16T21:12:24.937025+0000","last_clean":"2024-04-16T21:12:24.937025+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:12:24.937025+0000","last_undegraded":"2024-04-16T21:12:24.937025+0000","last_fullsized":"2024-04-16T21:12:24.937025+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.936891+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:58.704216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386322+0000","last_change":"2024-04-16T19:47:33.981495+0000","last_active":"2024-04-16T21:10:54.386322+0000","last_peered":"2024-04-16T21:10:54.386322+0000","last_clean":"2024-04-16T21:10:54.386322+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:10:54.386322+0000","last_undegraded":"2024-04-16T21:10:54.386322+0000","last_fullsized":"2024-04-16T21:10:54.386322+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'188","reported_seq":2057,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.351576+0000","last_change":"2024-04-16T21:12:32.351645+0000","last_active":"2024-04-16T21:12:32.351576+0000","last_peered":"2024-04-16T21:12:32.351576+0000","last_clean":"2024-04-16T21:12:32.351576+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:12:32.351576+0000","last_undegraded":"2024-04-16T21:12:32.351576+0000","last_fullsized":"2024-04-16T21:12:32.351576+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'188","last_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_deep_scrub":"52'188","last_deep_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.351505+0000","objects_scrubbed":40,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:48.150801+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12543,"num_omap_keys":51,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.379178+0000","last_change":"2024-04-16T21:12:16.379217+0000","last_active":"2024-04-16T21:12:16.379178+0000","last_peered":"2024-04-16T21:12:16.379178+0000","last_clean":"2024-04-16T21:12:16.379178+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:12:16.379178+0000","last_undegraded":"2024-04-16T21:12:16.379178+0000","last_fullsized":"2024-04-16T21:12:16.379178+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.379105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:54.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'37","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.663863+0000","last_change":"2024-04-16T21:12:06.663901+0000","last_active":"2024-04-16T21:12:06.663863+0000","last_peered":"2024-04-16T21:12:06.663863+0000","last_clean":"2024-04-16T21:12:06.663863+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:12:06.663863+0000","last_undegraded":"2024-04-16T21:12:06.663863+0000","last_fullsized":"2024-04-16T21:12:06.663863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'37","last_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_deep_scrub":"52'37","last_deep_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.663802+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:20.415399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":158,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.716132+0000","last_change":"2024-04-16T21:12:07.716153+0000","last_active":"2024-04-16T21:12:07.716132+0000","last_peered":"2024-04-16T21:12:07.716132+0000","last_clean":"2024-04-16T21:12:07.716132+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:12:07.716132+0000","last_undegraded":"2024-04-16T21:12:07.716132+0000","last_fullsized":"2024-04-16T21:12:07.716132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.716070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.751360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1762","reported_seq":4632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.707742+0000","last_change":"2024-04-16T21:11:51.707808+0000","last_active":"2024-04-16T21:11:51.707742+0000","last_peered":"2024-04-16T21:11:51.707742+0000","last_clean":"2024-04-16T21:11:51.707742+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:11:51.707742+0000","last_undegraded":"2024-04-16T21:11:51.707742+0000","last_fullsized":"2024-04-16T21:11:51.707742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1762","last_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_deep_scrub":"52'1762","last_deep_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.707684+0000","objects_scrubbed":22,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:25.851168+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":89480,"num_omap_keys":462,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'27","reported_seq":288,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.189093+0000","last_change":"2024-04-16T21:11:48.189147+0000","last_active":"2024-04-16T21:11:48.189093+0000","last_peered":"2024-04-16T21:11:48.189093+0000","last_clean":"2024-04-16T21:11:48.189093+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:11:48.189093+0000","last_undegraded":"2024-04-16T21:11:48.189093+0000","last_fullsized":"2024-04-16T21:11:48.189093+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.189029+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:30.304485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385804+0000","last_change":"2024-04-16T19:24:35.392117+0000","last_active":"2024-04-16T21:10:49.385804+0000","last_peered":"2024-04-16T21:10:49.385804+0000","last_clean":"2024-04-16T21:10:49.385804+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:10:49.385804+0000","last_undegraded":"2024-04-16T21:10:49.385804+0000","last_fullsized":"2024-04-16T21:10:49.385804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.383849+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:10:39.383849+0000","last_peered":"2024-04-16T21:10:39.383849+0000","last_clean":"2024-04-16T21:10:39.383849+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:10:39.383849+0000","last_undegraded":"2024-04-16T21:10:39.383849+0000","last_fullsized":"2024-04-16T21:10:39.383849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.386931+0000","last_change":"2024-04-16T19:24:45.414530+0000","last_active":"2024-04-16T21:10:59.386931+0000","last_peered":"2024-04-16T21:10:59.386931+0000","last_clean":"2024-04-16T21:10:59.386931+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:10:59.386931+0000","last_undegraded":"2024-04-16T21:10:59.386931+0000","last_fullsized":"2024-04-16T21:10:59.386931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'193","reported_seq":412,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.895939+0000","last_change":"2024-04-16T21:12:38.895987+0000","last_active":"2024-04-16T21:12:38.895939+0000","last_peered":"2024-04-16T21:12:38.895939+0000","last_clean":"2024-04-16T21:12:38.895939+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:12:38.895939+0000","last_undegraded":"2024-04-16T21:12:38.895939+0000","last_fullsized":"2024-04-16T21:12:38.895939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'193","last_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_deep_scrub":"52'193","last_deep_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.895878+0000","objects_scrubbed":9,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:45.133526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.922097+0000","last_change":"2024-04-16T21:12:20.922119+0000","last_active":"2024-04-16T21:12:20.922097+0000","last_peered":"2024-04-16T21:12:20.922097+0000","last_clean":"2024-04-16T21:12:20.922097+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:12:20.922097+0000","last_undegraded":"2024-04-16T21:12:20.922097+0000","last_fullsized":"2024-04-16T21:12:20.922097+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.922033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:09.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.873344+0000","last_change":"2024-04-16T21:12:12.873367+0000","last_active":"2024-04-16T21:12:12.873344+0000","last_peered":"2024-04-16T21:12:12.873344+0000","last_clean":"2024-04-16T21:12:12.873344+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:12:12.873344+0000","last_undegraded":"2024-04-16T21:12:12.873344+0000","last_fullsized":"2024-04-16T21:12:12.873344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.873281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:06.480261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525424+0000","last_change":"2024-04-16T19:47:33.988332+0000","last_active":"2024-04-16T21:10:54.525424+0000","last_peered":"2024-04-16T21:10:54.525424+0000","last_clean":"2024-04-16T21:10:54.525424+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:10:54.525424+0000","last_undegraded":"2024-04-16T21:10:54.525424+0000","last_fullsized":"2024-04-16T21:10:54.525424+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293176+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:11:42.293176+0000","last_peered":"2024-04-16T21:11:42.293176+0000","last_clean":"2024-04-16T21:11:42.293176+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:11:42.293176+0000","last_undegraded":"2024-04-16T21:11:42.293176+0000","last_fullsized":"2024-04-16T21:11:42.293176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.341198+0000","last_change":"2024-04-16T21:12:17.341220+0000","last_active":"2024-04-16T21:12:17.341198+0000","last_peered":"2024-04-16T21:12:17.341198+0000","last_clean":"2024-04-16T21:12:17.341198+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:12:17.341198+0000","last_undegraded":"2024-04-16T21:12:17.341198+0000","last_fullsized":"2024-04-16T21:12:17.341198+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.341136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:45.982431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'15","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.375445+0000","last_change":"2024-04-16T21:12:19.375473+0000","last_active":"2024-04-16T21:12:19.375445+0000","last_peered":"2024-04-16T21:12:19.375445+0000","last_clean":"2024-04-16T21:12:19.375445+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:12:19.375445+0000","last_undegraded":"2024-04-16T21:12:19.375445+0000","last_fullsized":"2024-04-16T21:12:19.375445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.375386+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.049075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.377316+0000","last_change":"2024-04-16T21:12:02.377338+0000","last_active":"2024-04-16T21:12:02.377316+0000","last_peered":"2024-04-16T21:12:02.377316+0000","last_clean":"2024-04-16T21:12:02.377316+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:12:02.377316+0000","last_undegraded":"2024-04-16T21:12:02.377316+0000","last_fullsized":"2024-04-16T21:12:02.377316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.377250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:49.251281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'325","reported_seq":659,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.419682+0000","last_change":"2024-04-16T21:11:56.419734+0000","last_active":"2024-04-16T21:11:56.419682+0000","last_peered":"2024-04-16T21:11:56.419682+0000","last_clean":"2024-04-16T21:11:56.419682+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:11:56.419682+0000","last_undegraded":"2024-04-16T21:11:56.419682+0000","last_fullsized":"2024-04-16T21:11:56.419682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'325","last_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_deep_scrub":"52'325","last_deep_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.419626+0000","objects_scrubbed":21,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:24.477776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6361,"num_omap_keys":29,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'16","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.568997+0000","last_change":"2024-04-16T21:11:45.569097+0000","last_active":"2024-04-16T21:11:45.568997+0000","last_peered":"2024-04-16T21:11:45.568997+0000","last_clean":"2024-04-16T21:11:45.568997+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:11:45.568997+0000","last_undegraded":"2024-04-16T21:11:45.568997+0000","last_fullsized":"2024-04-16T21:11:45.568997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.568923+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:31.758184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385305+0000","last_change":"2024-04-16T19:24:35.394848+0000","last_active":"2024-04-16T21:10:49.385305+0000","last_peered":"2024-04-16T21:10:49.385305+0000","last_clean":"2024-04-16T21:10:49.385305+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:10:49.385305+0000","last_undegraded":"2024-04-16T21:10:49.385305+0000","last_fullsized":"2024-04-16T21:10:49.385305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560129+0000","last_change":"2024-04-16T19:24:33.388692+0000","last_active":"2024-04-16T21:11:44.560129+0000","last_peered":"2024-04-16T21:11:44.560129+0000","last_clean":"2024-04-16T21:11:44.560129+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:11:44.560129+0000","last_undegraded":"2024-04-16T21:11:44.560129+0000","last_fullsized":"2024-04-16T21:11:44.560129+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"52'6","reported_seq":970,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.813395+0000","last_change":"2024-04-16T21:12:30.813430+0000","last_active":"2024-04-16T21:12:30.813395+0000","last_peered":"2024-04-16T21:12:30.813395+0000","last_clean":"2024-04-16T21:12:30.813395+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:12:30.813395+0000","last_undegraded":"2024-04-16T21:12:30.813395+0000","last_fullsized":"2024-04-16T21:12:30.813395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.813330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.897002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'78","reported_seq":240,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.999104+0000","last_change":"2024-04-16T21:12:31.999135+0000","last_active":"2024-04-16T21:12:31.999104+0000","last_peered":"2024-04-16T21:12:31.999104+0000","last_clean":"2024-04-16T21:12:31.999104+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:12:31.999104+0000","last_undegraded":"2024-04-16T21:12:31.999104+0000","last_fullsized":"2024-04-16T21:12:31.999104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.999043+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:40.493275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.929170+0000","last_change":"2024-04-16T21:12:21.929194+0000","last_active":"2024-04-16T21:12:21.929170+0000","last_peered":"2024-04-16T21:12:21.929170+0000","last_clean":"2024-04-16T21:12:21.929170+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:12:21.929170+0000","last_undegraded":"2024-04-16T21:12:21.929170+0000","last_fullsized":"2024-04-16T21:12:21.929170+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.929105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.364555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.721391+0000","last_change":"2024-04-16T21:12:18.721414+0000","last_active":"2024-04-16T21:12:18.721391+0000","last_peered":"2024-04-16T21:12:18.721391+0000","last_clean":"2024-04-16T21:12:18.721391+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:12:18.721391+0000","last_undegraded":"2024-04-16T21:12:18.721391+0000","last_fullsized":"2024-04-16T21:12:18.721391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.721330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:06.756619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386715+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:10:54.386715+0000","last_peered":"2024-04-16T21:10:54.386715+0000","last_clean":"2024-04-16T21:10:54.386715+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:10:54.386715+0000","last_undegraded":"2024-04-16T21:10:54.386715+0000","last_fullsized":"2024-04-16T21:10:54.386715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'135","reported_seq":1431,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.902487+0000","last_change":"2024-04-16T21:12:20.902537+0000","last_active":"2024-04-16T21:12:20.902487+0000","last_peered":"2024-04-16T21:12:20.902487+0000","last_clean":"2024-04-16T21:12:20.902487+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:12:20.902487+0000","last_undegraded":"2024-04-16T21:12:20.902487+0000","last_fullsized":"2024-04-16T21:12:20.902487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'135","last_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_deep_scrub":"52'135","last_deep_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.902428+0000","objects_scrubbed":42,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:27:02.685242+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3879,"num_omap_keys":15,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.456127+0000","last_change":"2024-04-16T21:12:14.456172+0000","last_active":"2024-04-16T21:12:14.456127+0000","last_peered":"2024-04-16T21:12:14.456127+0000","last_clean":"2024-04-16T21:12:14.456127+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:12:14.456127+0000","last_undegraded":"2024-04-16T21:12:14.456127+0000","last_fullsized":"2024-04-16T21:12:14.456127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:43.097345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'24","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.378145+0000","last_change":"2024-04-16T21:12:23.378174+0000","last_active":"2024-04-16T21:12:23.378145+0000","last_peered":"2024-04-16T21:12:23.378145+0000","last_clean":"2024-04-16T21:12:23.378145+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:12:23.378145+0000","last_undegraded":"2024-04-16T21:12:23.378145+0000","last_fullsized":"2024-04-16T21:12:23.378145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.378087+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:34.980757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.647510+0000","last_change":"2024-04-16T21:11:51.647550+0000","last_active":"2024-04-16T21:11:51.647510+0000","last_peered":"2024-04-16T21:11:51.647510+0000","last_clean":"2024-04-16T21:11:51.647510+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:11:51.647510+0000","last_undegraded":"2024-04-16T21:11:51.647510+0000","last_fullsized":"2024-04-16T21:11:51.647510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.647441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:25.536915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2068","reported_seq":5847,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.355619+0000","last_change":"2024-04-16T21:11:58.355713+0000","last_active":"2024-04-16T21:11:58.355619+0000","last_peered":"2024-04-16T21:11:58.355619+0000","last_clean":"2024-04-16T21:11:58.355619+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:11:58.355619+0000","last_undegraded":"2024-04-16T21:11:58.355619+0000","last_fullsized":"2024-04-16T21:11:58.355619+0000","mapping_epoch":33,"log_start":"47'400","ondisk_log_start":"47'400","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2068","last_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_deep_scrub":"52'2068","last_deep_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.355545+0000","objects_scrubbed":24,"log_size":1668,"log_dups_size":400,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:46.390834+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16267,"num_omap_keys":93,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'26","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.715965+0000","last_change":"2024-04-16T21:11:55.716041+0000","last_active":"2024-04-16T21:11:55.715965+0000","last_peered":"2024-04-16T21:11:55.715965+0000","last_clean":"2024-04-16T21:11:55.715965+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:11:55.715965+0000","last_undegraded":"2024-04-16T21:11:55.715965+0000","last_fullsized":"2024-04-16T21:11:55.715965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.715893+0000","objects_scrubbed":12,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:15.543756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388499+0000","last_change":"2024-04-16T19:24:35.398619+0000","last_active":"2024-04-16T21:11:09.388499+0000","last_peered":"2024-04-16T21:11:09.388499+0000","last_clean":"2024-04-16T21:11:09.388499+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:11:09.388499+0000","last_undegraded":"2024-04-16T21:11:09.388499+0000","last_fullsized":"2024-04-16T21:11:09.388499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649904+0000","last_change":"2024-04-16T19:24:33.389193+0000","last_active":"2024-04-16T21:11:43.649904+0000","last_peered":"2024-04-16T21:11:43.649904+0000","last_clean":"2024-04-16T21:11:43.649904+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:11:43.649904+0000","last_undegraded":"2024-04-16T21:11:43.649904+0000","last_fullsized":"2024-04-16T21:11:43.649904+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"52'2","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.812592+0000","last_change":"2024-04-16T21:12:32.812623+0000","last_active":"2024-04-16T21:12:32.812592+0000","last_peered":"2024-04-16T21:12:32.812592+0000","last_clean":"2024-04-16T21:12:32.812592+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:12:32.812592+0000","last_undegraded":"2024-04-16T21:12:32.812592+0000","last_fullsized":"2024-04-16T21:12:32.812592+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.812531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:48.281904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'147","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334518+0000","last_change":"2024-04-16T21:12:30.334588+0000","last_active":"2024-04-16T21:12:30.334518+0000","last_peered":"2024-04-16T21:12:30.334518+0000","last_clean":"2024-04-16T21:12:30.334518+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:12:30.334518+0000","last_undegraded":"2024-04-16T21:12:30.334518+0000","last_fullsized":"2024-04-16T21:12:30.334518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334447+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:36.782836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'13","reported_seq":1323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.286038+0000","last_change":"2024-04-16T21:12:31.286066+0000","last_active":"2024-04-16T21:12:31.286038+0000","last_peered":"2024-04-16T21:12:31.286038+0000","last_clean":"2024-04-16T21:12:31.286038+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:12:31.286038+0000","last_undegraded":"2024-04-16T21:12:31.286038+0000","last_fullsized":"2024-04-16T21:12:31.286038+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.285973+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:54.848235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.405984+0000","last_change":"2024-04-16T21:12:07.406028+0000","last_active":"2024-04-16T21:12:07.405984+0000","last_peered":"2024-04-16T21:12:07.405984+0000","last_clean":"2024-04-16T21:12:07.405984+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:12:07.405984+0000","last_undegraded":"2024-04-16T21:12:07.405984+0000","last_fullsized":"2024-04-16T21:12:07.405984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.405900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:12.444844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.195903+0000","last_change":"2024-04-16T19:47:33.988606+0000","last_active":"2024-04-16T21:11:43.195903+0000","last_peered":"2024-04-16T21:11:43.195903+0000","last_clean":"2024-04-16T21:11:43.195903+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:11:43.195903+0000","last_undegraded":"2024-04-16T21:11:43.195903+0000","last_fullsized":"2024-04-16T21:11:43.195903+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'182","reported_seq":1946,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.937380+0000","last_change":"2024-04-16T21:12:22.937438+0000","last_active":"2024-04-16T21:12:22.937380+0000","last_peered":"2024-04-16T21:12:22.937380+0000","last_clean":"2024-04-16T21:12:22.937380+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:12:22.937380+0000","last_undegraded":"2024-04-16T21:12:22.937380+0000","last_fullsized":"2024-04-16T21:12:22.937380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'182","last_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_deep_scrub":"52'182","last_deep_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.937321+0000","objects_scrubbed":46,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:18.195145+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8599,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.724011+0000","last_change":"2024-04-16T21:12:29.724048+0000","last_active":"2024-04-16T21:12:29.724011+0000","last_peered":"2024-04-16T21:12:29.724011+0000","last_clean":"2024-04-16T21:12:29.724011+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:12:29.724011+0000","last_undegraded":"2024-04-16T21:12:29.724011+0000","last_fullsized":"2024-04-16T21:12:29.724011+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.723920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:23.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'17","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.440078+0000","last_change":"2024-04-16T21:12:10.440115+0000","last_active":"2024-04-16T21:12:10.440078+0000","last_peered":"2024-04-16T21:12:10.440078+0000","last_clean":"2024-04-16T21:12:10.440078+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:12:10.440078+0000","last_undegraded":"2024-04-16T21:12:10.440078+0000","last_fullsized":"2024-04-16T21:12:10.440078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.440017+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:44.207343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.910624+0000","last_change":"2024-04-16T21:12:13.910647+0000","last_active":"2024-04-16T21:12:13.910624+0000","last_peered":"2024-04-16T21:12:13.910624+0000","last_clean":"2024-04-16T21:12:13.910624+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:12:13.910624+0000","last_undegraded":"2024-04-16T21:12:13.910624+0000","last_fullsized":"2024-04-16T21:12:13.910624+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.910562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:19.649702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'362","reported_seq":695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.975586+0000","last_change":"2024-04-16T21:11:44.975641+0000","last_active":"2024-04-16T21:11:44.975586+0000","last_peered":"2024-04-16T21:11:44.975586+0000","last_clean":"2024-04-16T21:11:44.975586+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:11:44.975586+0000","last_undegraded":"2024-04-16T21:11:44.975586+0000","last_fullsized":"2024-04-16T21:11:44.975586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'362","last_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_deep_scrub":"52'362","last_deep_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.975535+0000","objects_scrubbed":21,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:50:56.598547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2185,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'18","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.687122+0000","last_change":"2024-04-16T21:11:48.687164+0000","last_active":"2024-04-16T21:11:48.687122+0000","last_peered":"2024-04-16T21:11:48.687122+0000","last_clean":"2024-04-16T21:11:48.687122+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:11:48.687122+0000","last_undegraded":"2024-04-16T21:11:48.687122+0000","last_fullsized":"2024-04-16T21:11:48.687122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.687069+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:52.625067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292230+0000","last_change":"2024-04-16T19:24:35.400311+0000","last_active":"2024-04-16T21:11:42.292230+0000","last_peered":"2024-04-16T21:11:42.292230+0000","last_clean":"2024-04-16T21:11:42.292230+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:11:42.292230+0000","last_undegraded":"2024-04-16T21:11:42.292230+0000","last_fullsized":"2024-04-16T21:11:42.292230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560194+0000","last_change":"2024-04-16T19:24:33.390726+0000","last_active":"2024-04-16T21:11:44.560194+0000","last_peered":"2024-04-16T21:11:44.560194+0000","last_clean":"2024-04-16T21:11:44.560194+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:11:44.560194+0000","last_undegraded":"2024-04-16T21:11:44.560194+0000","last_fullsized":"2024-04-16T21:11:44.560194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'19","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.913946+0000","last_change":"2024-04-16T21:12:26.913976+0000","last_active":"2024-04-16T21:12:26.913946+0000","last_peered":"2024-04-16T21:12:26.913946+0000","last_clean":"2024-04-16T21:12:26.913946+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:12:26.913946+0000","last_undegraded":"2024-04-16T21:12:26.913946+0000","last_fullsized":"2024-04-16T21:12:26.913946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.913887+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.708618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'87","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.302830+0000","last_change":"2024-04-16T21:12:35.302869+0000","last_active":"2024-04-16T21:12:35.302830+0000","last_peered":"2024-04-16T21:12:35.302830+0000","last_clean":"2024-04-16T21:12:35.302830+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:12:35.302830+0000","last_undegraded":"2024-04-16T21:12:35.302830+0000","last_fullsized":"2024-04-16T21:12:35.302830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.302758+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:01.984411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.899124+0000","last_change":"2024-04-16T21:12:18.899147+0000","last_active":"2024-04-16T21:12:18.899124+0000","last_peered":"2024-04-16T21:12:18.899124+0000","last_clean":"2024-04-16T21:12:18.899124+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:12:18.899124+0000","last_undegraded":"2024-04-16T21:12:18.899124+0000","last_fullsized":"2024-04-16T21:12:18.899124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.899064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:17.822519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.847627+0000","last_change":"2024-04-16T21:12:03.847650+0000","last_active":"2024-04-16T21:12:03.847627+0000","last_peered":"2024-04-16T21:12:03.847627+0000","last_clean":"2024-04-16T21:12:03.847627+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:12:03.847627+0000","last_undegraded":"2024-04-16T21:12:03.847627+0000","last_fullsized":"2024-04-16T21:12:03.847627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.847563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:36.440589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744155+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:11:42.744155+0000","last_peered":"2024-04-16T21:11:42.744155+0000","last_clean":"2024-04-16T21:11:42.744155+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:11:42.744155+0000","last_undegraded":"2024-04-16T21:11:42.744155+0000","last_fullsized":"2024-04-16T21:11:42.744155+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'186","reported_seq":2409,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.401091+0000","last_change":"2024-04-16T21:12:42.401122+0000","last_active":"2024-04-16T21:12:42.401091+0000","last_peered":"2024-04-16T21:12:42.401091+0000","last_clean":"2024-04-16T21:12:42.401091+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:12:42.401091+0000","last_undegraded":"2024-04-16T21:12:42.401091+0000","last_fullsized":"2024-04-16T21:12:42.401091+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'186","last_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_deep_scrub":"52'186","last_deep_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.401026+0000","objects_scrubbed":52,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:51.749803+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12956,"num_omap_keys":52,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.650119+0000","last_change":"2024-04-16T21:12:13.650119+0000","last_active":"2024-04-16T21:12:13.650119+0000","last_peered":"2024-04-16T21:12:13.650119+0000","last_clean":"2024-04-16T21:12:13.650119+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:12:13.650119+0000","last_undegraded":"2024-04-16T21:12:13.650119+0000","last_fullsized":"2024-04-16T21:12:13.650119+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.650036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:16.977821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"52'13","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.901211+0000","last_change":"2024-04-16T21:12:20.901247+0000","last_active":"2024-04-16T21:12:20.901211+0000","last_peered":"2024-04-16T21:12:20.901211+0000","last_clean":"2024-04-16T21:12:20.901211+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:12:20.901211+0000","last_undegraded":"2024-04-16T21:12:20.901211+0000","last_fullsized":"2024-04-16T21:12:20.901211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.901143+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:54.668469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'706","reported_seq":2075,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.405465+0000","last_change":"2024-04-16T21:12:08.405492+0000","last_active":"2024-04-16T21:12:08.405465+0000","last_peered":"2024-04-16T21:12:08.405465+0000","last_clean":"2024-04-16T21:12:08.405465+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:12:08.405465+0000","last_undegraded":"2024-04-16T21:12:08.405465+0000","last_fullsized":"2024-04-16T21:12:08.405465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.405399+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:20.200130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'292","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.642477+0000","last_change":"2024-04-16T21:11:50.642568+0000","last_active":"2024-04-16T21:11:50.642477+0000","last_peered":"2024-04-16T21:11:50.642477+0000","last_clean":"2024-04-16T21:11:50.642477+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:11:50.642477+0000","last_undegraded":"2024-04-16T21:11:50.642477+0000","last_fullsized":"2024-04-16T21:11:50.642477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'292","last_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_deep_scrub":"52'292","last_deep_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.642400+0000","objects_scrubbed":23,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:29.527145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4142,"num_omap_keys":19,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.314574+0000","last_change":"2024-04-16T21:11:57.314619+0000","last_active":"2024-04-16T21:11:57.314574+0000","last_peered":"2024-04-16T21:11:57.314574+0000","last_clean":"2024-04-16T21:11:57.314574+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:11:57.314574+0000","last_undegraded":"2024-04-16T21:11:57.314574+0000","last_fullsized":"2024-04-16T21:11:57.314574+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.314511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:49.494822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106353+0000","last_change":"2024-04-16T19:24:35.397465+0000","last_active":"2024-04-16T21:11:44.106353+0000","last_peered":"2024-04-16T21:11:44.106353+0000","last_clean":"2024-04-16T21:11:44.106353+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:11:44.106353+0000","last_undegraded":"2024-04-16T21:11:44.106353+0000","last_fullsized":"2024-04-16T21:11:44.106353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.898133+0000","last_change":"2024-04-16T21:12:08.898133+0000","last_active":"2024-04-16T21:12:08.898133+0000","last_peered":"2024-04-16T21:12:08.898133+0000","last_clean":"2024-04-16T21:12:08.898133+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:12:08.898133+0000","last_undegraded":"2024-04-16T21:12:08.898133+0000","last_fullsized":"2024-04-16T21:12:08.898133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.898047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:36.956196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"52'2","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.800274+0000","last_change":"2024-04-16T21:12:36.800302+0000","last_active":"2024-04-16T21:12:36.800274+0000","last_peered":"2024-04-16T21:12:36.800274+0000","last_clean":"2024-04-16T21:12:36.800274+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:12:36.800274+0000","last_undegraded":"2024-04-16T21:12:36.800274+0000","last_fullsized":"2024-04-16T21:12:36.800274+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.800213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:46.361882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'101","reported_seq":274,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.698884+0000","last_change":"2024-04-16T21:12:30.698919+0000","last_active":"2024-04-16T21:12:30.698884+0000","last_peered":"2024-04-16T21:12:30.698884+0000","last_clean":"2024-04-16T21:12:30.698884+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:12:30.698884+0000","last_undegraded":"2024-04-16T21:12:30.698884+0000","last_fullsized":"2024-04-16T21:12:30.698884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.698825+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.782497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.333039+0000","last_change":"2024-04-16T21:12:32.333063+0000","last_active":"2024-04-16T21:12:32.333039+0000","last_peered":"2024-04-16T21:12:32.333039+0000","last_clean":"2024-04-16T21:12:32.333039+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:12:32.333039+0000","last_undegraded":"2024-04-16T21:12:32.333039+0000","last_fullsized":"2024-04-16T21:12:32.333039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.332967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:32.295174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.889383+0000","last_change":"2024-04-16T21:12:17.889407+0000","last_active":"2024-04-16T21:12:17.889383+0000","last_peered":"2024-04-16T21:12:17.889383+0000","last_clean":"2024-04-16T21:12:17.889383+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:12:17.889383+0000","last_undegraded":"2024-04-16T21:12:17.889383+0000","last_fullsized":"2024-04-16T21:12:17.889383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.889318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:23.628459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293308+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:11:42.293308+0000","last_peered":"2024-04-16T21:11:42.293308+0000","last_clean":"2024-04-16T21:11:42.293308+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:11:42.293308+0000","last_undegraded":"2024-04-16T21:11:42.293308+0000","last_fullsized":"2024-04-16T21:11:42.293308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'134","reported_seq":1680,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561550+0000","last_change":"2024-04-16T19:25:49.518161+0000","last_active":"2024-04-16T21:11:44.561550+0000","last_peered":"2024-04-16T21:11:44.561550+0000","last_clean":"2024-04-16T21:11:44.561550+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:11:44.561550+0000","last_undegraded":"2024-04-16T21:11:44.561550+0000","last_fullsized":"2024-04-16T21:11:44.561550+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.854832+0000","last_change":"2024-04-16T21:12:21.854876+0000","last_active":"2024-04-16T21:12:21.854832+0000","last_peered":"2024-04-16T21:12:21.854832+0000","last_clean":"2024-04-16T21:12:21.854832+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:12:21.854832+0000","last_undegraded":"2024-04-16T21:12:21.854832+0000","last_fullsized":"2024-04-16T21:12:21.854832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.854759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:37.324132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'138","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.362284+0000","last_change":"2024-04-16T21:12:22.362331+0000","last_active":"2024-04-16T21:12:22.362284+0000","last_peered":"2024-04-16T21:12:22.362284+0000","last_clean":"2024-04-16T21:12:22.362284+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:12:22.362284+0000","last_undegraded":"2024-04-16T21:12:22.362284+0000","last_fullsized":"2024-04-16T21:12:22.362284+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.362197+0000","objects_scrubbed":2,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:16.550872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9707,"num_omap_keys":56,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.836583+0000","last_change":"2024-04-16T21:11:52.836606+0000","last_active":"2024-04-16T21:11:52.836583+0000","last_peered":"2024-04-16T21:11:52.836583+0000","last_clean":"2024-04-16T21:11:52.836583+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:11:52.836583+0000","last_undegraded":"2024-04-16T21:11:52.836583+0000","last_fullsized":"2024-04-16T21:11:52.836583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.836519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:26.725994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'371","reported_seq":758,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.959985+0000","last_change":"2024-04-16T21:11:46.960080+0000","last_active":"2024-04-16T21:11:46.959985+0000","last_peered":"2024-04-16T21:11:46.959985+0000","last_clean":"2024-04-16T21:11:46.959985+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:11:46.959985+0000","last_undegraded":"2024-04-16T21:11:46.959985+0000","last_fullsized":"2024-04-16T21:11:46.959985+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'371","last_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_deep_scrub":"52'371","last_deep_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.959920+0000","objects_scrubbed":24,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:28.129840+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2169,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'19","reported_seq":216,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.614888+0000","last_change":"2024-04-16T21:11:53.614931+0000","last_active":"2024-04-16T21:11:53.614888+0000","last_peered":"2024-04-16T21:11:53.614888+0000","last_clean":"2024-04-16T21:11:53.614888+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:11:53.614888+0000","last_undegraded":"2024-04-16T21:11:53.614888+0000","last_fullsized":"2024-04-16T21:11:53.614888+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.614826+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:49.084093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650238+0000","last_change":"2024-04-16T19:24:35.393387+0000","last_active":"2024-04-16T21:11:43.650238+0000","last_peered":"2024-04-16T21:11:43.650238+0000","last_clean":"2024-04-16T21:11:43.650238+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:11:43.650238+0000","last_undegraded":"2024-04-16T21:11:43.650238+0000","last_fullsized":"2024-04-16T21:11:43.650238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.827874+0000","last_change":"2024-04-16T21:12:04.827874+0000","last_active":"2024-04-16T21:12:04.827874+0000","last_peered":"2024-04-16T21:12:04.827874+0000","last_clean":"2024-04-16T21:12:04.827874+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:12:04.827874+0000","last_undegraded":"2024-04-16T21:12:04.827874+0000","last_fullsized":"2024-04-16T21:12:04.827874+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.827767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:23.692877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'19","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:34.382104+0000","last_change":"2024-04-16T19:24:45.408789+0000","last_active":"2024-04-16T21:10:34.382104+0000","last_peered":"2024-04-16T21:10:34.382104+0000","last_clean":"2024-04-16T21:10:34.382104+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:10:34.382104+0000","last_undegraded":"2024-04-16T21:10:34.382104+0000","last_fullsized":"2024-04-16T21:10:34.382104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'147","reported_seq":342,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.899689+0000","last_change":"2024-04-16T21:12:22.899755+0000","last_active":"2024-04-16T21:12:22.899689+0000","last_peered":"2024-04-16T21:12:22.899689+0000","last_clean":"2024-04-16T21:12:22.899689+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:12:22.899689+0000","last_undegraded":"2024-04-16T21:12:22.899689+0000","last_fullsized":"2024-04-16T21:12:22.899689+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.899611+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:49.019687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'13","reported_seq":1321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.856467+0000","last_change":"2024-04-16T21:12:19.856493+0000","last_active":"2024-04-16T21:12:19.856467+0000","last_peered":"2024-04-16T21:12:19.856467+0000","last_clean":"2024-04-16T21:12:19.856467+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:12:19.856467+0000","last_undegraded":"2024-04-16T21:12:19.856467+0000","last_fullsized":"2024-04-16T21:12:19.856467+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.856405+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.872492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.913945+0000","last_change":"2024-04-16T21:12:16.913967+0000","last_active":"2024-04-16T21:12:16.913945+0000","last_peered":"2024-04-16T21:12:16.913945+0000","last_clean":"2024-04-16T21:12:16.913945+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:12:16.913945+0000","last_undegraded":"2024-04-16T21:12:16.913945+0000","last_fullsized":"2024-04-16T21:12:16.913945+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.913879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:03.787910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389430+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:11:09.389430+0000","last_peered":"2024-04-16T21:11:09.389430+0000","last_clean":"2024-04-16T21:11:09.389430+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:11:09.389430+0000","last_undegraded":"2024-04-16T21:11:09.389430+0000","last_fullsized":"2024-04-16T21:11:09.389430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'166","reported_seq":1925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.247637+0000","last_change":"2024-04-16T21:12:40.247676+0000","last_active":"2024-04-16T21:12:40.247637+0000","last_peered":"2024-04-16T21:12:40.247637+0000","last_clean":"2024-04-16T21:12:40.247637+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:12:40.247637+0000","last_undegraded":"2024-04-16T21:12:40.247637+0000","last_fullsized":"2024-04-16T21:12:40.247637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'166","last_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_deep_scrub":"52'166","last_deep_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.247577+0000","objects_scrubbed":44,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:25.005335+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9289,"num_omap_keys":38,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.368999+0000","last_change":"2024-04-16T21:12:24.369021+0000","last_active":"2024-04-16T21:12:24.368999+0000","last_peered":"2024-04-16T21:12:24.368999+0000","last_clean":"2024-04-16T21:12:24.368999+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:12:24.368999+0000","last_undegraded":"2024-04-16T21:12:24.368999+0000","last_fullsized":"2024-04-16T21:12:24.368999+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.368926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:12.501416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.879266+0000","last_change":"2024-04-16T21:12:15.879307+0000","last_active":"2024-04-16T21:12:15.879266+0000","last_peered":"2024-04-16T21:12:15.879266+0000","last_clean":"2024-04-16T21:12:15.879266+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:12:15.879266+0000","last_undegraded":"2024-04-16T21:12:15.879266+0000","last_fullsized":"2024-04-16T21:12:15.879266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.879198+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:41.314648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.625545+0000","last_change":"2024-04-16T21:11:53.625568+0000","last_active":"2024-04-16T21:11:53.625545+0000","last_peered":"2024-04-16T21:11:53.625545+0000","last_clean":"2024-04-16T21:11:53.625545+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:11:53.625545+0000","last_undegraded":"2024-04-16T21:11:53.625545+0000","last_fullsized":"2024-04-16T21:11:53.625545+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.625484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:35.360695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'485","reported_seq":911,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.722929+0000","last_change":"2024-04-16T21:11:47.722980+0000","last_active":"2024-04-16T21:11:47.722929+0000","last_peered":"2024-04-16T21:11:47.722929+0000","last_clean":"2024-04-16T21:11:47.722929+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:11:47.722929+0000","last_undegraded":"2024-04-16T21:11:47.722929+0000","last_fullsized":"2024-04-16T21:11:47.722929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'485","last_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_deep_scrub":"52'485","last_deep_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.722865+0000","objects_scrubbed":32,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:39.504569+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42883,"num_omap_keys":132,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'14","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.616163+0000","last_change":"2024-04-16T21:12:05.616245+0000","last_active":"2024-04-16T21:12:05.616163+0000","last_peered":"2024-04-16T21:12:05.616163+0000","last_clean":"2024-04-16T21:12:05.616163+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:12:05.616163+0000","last_undegraded":"2024-04-16T21:12:05.616163+0000","last_fullsized":"2024-04-16T21:12:05.616163+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.616057+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.466963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.688471+0000","last_change":"2024-04-16T21:12:33.688471+0000","last_active":"2024-04-16T21:12:33.688471+0000","last_peered":"2024-04-16T21:12:33.688471+0000","last_clean":"2024-04-16T21:12:33.688471+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:12:33.688471+0000","last_undegraded":"2024-04-16T21:12:33.688471+0000","last_fullsized":"2024-04-16T21:12:33.688471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.688381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:51:41.315722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.384338+0000","last_change":"2024-04-16T19:24:33.385067+0000","last_active":"2024-04-16T21:10:44.384338+0000","last_peered":"2024-04-16T21:10:44.384338+0000","last_clean":"2024-04-16T21:10:44.384338+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:10:44.384338+0000","last_undegraded":"2024-04-16T21:10:44.384338+0000","last_fullsized":"2024-04-16T21:10:44.384338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386376+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:10:54.386376+0000","last_peered":"2024-04-16T21:10:54.386376+0000","last_clean":"2024-04-16T21:10:54.386376+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:10:54.386376+0000","last_undegraded":"2024-04-16T21:10:54.386376+0000","last_fullsized":"2024-04-16T21:10:54.386376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'115","reported_seq":290,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.013629+0000","last_change":"2024-04-16T21:12:29.013687+0000","last_active":"2024-04-16T21:12:29.013629+0000","last_peered":"2024-04-16T21:12:29.013629+0000","last_clean":"2024-04-16T21:12:29.013629+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:12:29.013629+0000","last_undegraded":"2024-04-16T21:12:29.013629+0000","last_fullsized":"2024-04-16T21:12:29.013629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.013556+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:57.654850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'13","reported_seq":1321,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.607231+0000","last_change":"2024-04-16T21:12:22.607256+0000","last_active":"2024-04-16T21:12:22.607231+0000","last_peered":"2024-04-16T21:12:22.607231+0000","last_clean":"2024-04-16T21:12:22.607231+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:12:22.607231+0000","last_undegraded":"2024-04-16T21:12:22.607231+0000","last_fullsized":"2024-04-16T21:12:22.607231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.607175+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:48.727251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.673878+0000","last_change":"2024-04-16T21:12:20.673922+0000","last_active":"2024-04-16T21:12:20.673878+0000","last_peered":"2024-04-16T21:12:20.673878+0000","last_clean":"2024-04-16T21:12:20.673878+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:12:20.673878+0000","last_undegraded":"2024-04-16T21:12:20.673878+0000","last_fullsized":"2024-04-16T21:12:20.673878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.673807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:17.517736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385558+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:10:54.385558+0000","last_peered":"2024-04-16T21:10:54.385558+0000","last_clean":"2024-04-16T21:10:54.385558+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:10:54.385558+0000","last_undegraded":"2024-04-16T21:10:54.385558+0000","last_fullsized":"2024-04-16T21:10:54.385558+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'162","reported_seq":2309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.337358+0000","last_change":"2024-04-16T21:12:34.337466+0000","last_active":"2024-04-16T21:12:34.337358+0000","last_peered":"2024-04-16T21:12:34.337358+0000","last_clean":"2024-04-16T21:12:34.337358+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:12:34.337358+0000","last_undegraded":"2024-04-16T21:12:34.337358+0000","last_fullsized":"2024-04-16T21:12:34.337358+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'162","last_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_deep_scrub":"52'162","last_deep_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.337295+0000","objects_scrubbed":41,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:21.592470+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11917,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.346082+0000","last_change":"2024-04-16T21:12:20.346107+0000","last_active":"2024-04-16T21:12:20.346082+0000","last_peered":"2024-04-16T21:12:20.346082+0000","last_clean":"2024-04-16T21:12:20.346082+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:12:20.346082+0000","last_undegraded":"2024-04-16T21:12:20.346082+0000","last_fullsized":"2024-04-16T21:12:20.346082+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.346020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:28.840252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.905708+0000","last_change":"2024-04-16T21:12:13.905733+0000","last_active":"2024-04-16T21:12:13.905708+0000","last_peered":"2024-04-16T21:12:13.905708+0000","last_clean":"2024-04-16T21:12:13.905708+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:12:13.905708+0000","last_undegraded":"2024-04-16T21:12:13.905708+0000","last_fullsized":"2024-04-16T21:12:13.905708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.905640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:17.215414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'706","reported_seq":2474,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.688857+0000","last_change":"2024-04-16T21:11:58.688891+0000","last_active":"2024-04-16T21:11:58.688857+0000","last_peered":"2024-04-16T21:11:58.688857+0000","last_clean":"2024-04-16T21:11:58.688857+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:11:58.688857+0000","last_undegraded":"2024-04-16T21:11:58.688857+0000","last_fullsized":"2024-04-16T21:11:58.688857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.688787+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:58.866593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'332","reported_seq":603,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.669169+0000","last_change":"2024-04-16T21:11:50.669285+0000","last_active":"2024-04-16T21:11:50.669169+0000","last_peered":"2024-04-16T21:11:50.669169+0000","last_clean":"2024-04-16T21:11:50.669169+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:11:50.669169+0000","last_undegraded":"2024-04-16T21:11:50.669169+0000","last_fullsized":"2024-04-16T21:11:50.669169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'332","last_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_deep_scrub":"52'332","last_deep_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.669096+0000","objects_scrubbed":21,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:18.487063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2179,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'12","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.639977+0000","last_change":"2024-04-16T21:11:49.640051+0000","last_active":"2024-04-16T21:11:49.639977+0000","last_peered":"2024-04-16T21:11:49.639977+0000","last_clean":"2024-04-16T21:11:49.639977+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:11:49.639977+0000","last_undegraded":"2024-04-16T21:11:49.639977+0000","last_fullsized":"2024-04-16T21:11:49.639977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.639900+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:27.425866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383761+0000","last_change":"2024-04-16T19:24:35.402984+0000","last_active":"2024-04-16T21:10:44.383761+0000","last_peered":"2024-04-16T21:10:44.383761+0000","last_clean":"2024-04-16T21:10:44.383761+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:10:44.383761+0000","last_undegraded":"2024-04-16T21:10:44.383761+0000","last_fullsized":"2024-04-16T21:10:44.383761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.523887+0000","last_change":"2024-04-16T19:24:33.388790+0000","last_active":"2024-04-16T21:10:44.523887+0000","last_peered":"2024-04-16T21:10:44.523887+0000","last_clean":"2024-04-16T21:10:44.523887+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:10:44.523887+0000","last_undegraded":"2024-04-16T21:10:44.523887+0000","last_fullsized":"2024-04-16T21:10:44.523887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.287510+0000","last_change":"2024-04-16T21:12:37.287531+0000","last_active":"2024-04-16T21:12:37.287510+0000","last_peered":"2024-04-16T21:12:37.287510+0000","last_clean":"2024-04-16T21:12:37.287510+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:12:37.287510+0000","last_undegraded":"2024-04-16T21:12:37.287510+0000","last_fullsized":"2024-04-16T21:12:37.287510+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.287448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:24.542624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'170","reported_seq":376,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.887678+0000","last_change":"2024-04-16T21:12:36.887713+0000","last_active":"2024-04-16T21:12:36.887678+0000","last_peered":"2024-04-16T21:12:36.887678+0000","last_clean":"2024-04-16T21:12:36.887678+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:12:36.887678+0000","last_undegraded":"2024-04-16T21:12:36.887678+0000","last_fullsized":"2024-04-16T21:12:36.887678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.887622+0000","objects_scrubbed":8,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:25.493915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.314201+0000","last_change":"2024-04-16T21:12:26.314224+0000","last_active":"2024-04-16T21:12:26.314201+0000","last_peered":"2024-04-16T21:12:26.314201+0000","last_clean":"2024-04-16T21:12:26.314201+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:12:26.314201+0000","last_undegraded":"2024-04-16T21:12:26.314201+0000","last_fullsized":"2024-04-16T21:12:26.314201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.314134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:40.584484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.903511+0000","last_change":"2024-04-16T21:12:19.903531+0000","last_active":"2024-04-16T21:12:19.903511+0000","last_peered":"2024-04-16T21:12:19.903511+0000","last_clean":"2024-04-16T21:12:19.903511+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:12:19.903511+0000","last_undegraded":"2024-04-16T21:12:19.903511+0000","last_fullsized":"2024-04-16T21:12:19.903511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.903448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:31.698180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524809+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:10:49.524809+0000","last_peered":"2024-04-16T21:10:49.524809+0000","last_clean":"2024-04-16T21:10:49.524809+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:10:49.524809+0000","last_undegraded":"2024-04-16T21:10:49.524809+0000","last_fullsized":"2024-04-16T21:10:49.524809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'216","reported_seq":2161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.661893+0000","last_change":"2024-04-16T21:12:21.661923+0000","last_active":"2024-04-16T21:12:21.661893+0000","last_peered":"2024-04-16T21:12:21.661893+0000","last_clean":"2024-04-16T21:12:21.661893+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:12:21.661893+0000","last_undegraded":"2024-04-16T21:12:21.661893+0000","last_fullsized":"2024-04-16T21:12:21.661893+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'216","last_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_deep_scrub":"52'216","last_deep_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.661832+0000","objects_scrubbed":54,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.097281+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14476,"num_omap_keys":58,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.921849+0000","last_change":"2024-04-16T21:12:14.921870+0000","last_active":"2024-04-16T21:12:14.921849+0000","last_peered":"2024-04-16T21:12:14.921849+0000","last_clean":"2024-04-16T21:12:14.921849+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:12:14.921849+0000","last_undegraded":"2024-04-16T21:12:14.921849+0000","last_fullsized":"2024-04-16T21:12:14.921849+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.921774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:11.765704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'17","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.682277+0000","last_change":"2024-04-16T21:12:08.682317+0000","last_active":"2024-04-16T21:12:08.682277+0000","last_peered":"2024-04-16T21:12:08.682277+0000","last_clean":"2024-04-16T21:12:08.682277+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:12:08.682277+0000","last_undegraded":"2024-04-16T21:12:08.682277+0000","last_fullsized":"2024-04-16T21:12:08.682277+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.682212+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:32.852256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.634331+0000","last_change":"2024-04-16T21:11:56.634374+0000","last_active":"2024-04-16T21:11:56.634331+0000","last_peered":"2024-04-16T21:11:56.634331+0000","last_clean":"2024-04-16T21:11:56.634331+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:11:56.634331+0000","last_undegraded":"2024-04-16T21:11:56.634331+0000","last_fullsized":"2024-04-16T21:11:56.634331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.634253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:30.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'281","reported_seq":601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.734176+0000","last_change":"2024-04-16T21:11:44.734203+0000","last_active":"2024-04-16T21:11:44.734176+0000","last_peered":"2024-04-16T21:11:44.734176+0000","last_clean":"2024-04-16T21:11:44.734176+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:11:44.734176+0000","last_undegraded":"2024-04-16T21:11:44.734176+0000","last_fullsized":"2024-04-16T21:11:44.734176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'281","last_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_deep_scrub":"52'281","last_deep_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.734118+0000","objects_scrubbed":25,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:22.520084+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1975,"num_omap_keys":9,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'12","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.876267+0000","last_change":"2024-04-16T21:12:09.876315+0000","last_active":"2024-04-16T21:12:09.876267+0000","last_peered":"2024-04-16T21:12:09.876267+0000","last_clean":"2024-04-16T21:12:09.876267+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:12:09.876267+0000","last_undegraded":"2024-04-16T21:12:09.876267+0000","last_fullsized":"2024-04-16T21:12:09.876267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.876206+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.314787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.204744+0000","last_change":"2024-04-16T21:12:43.204766+0000","last_active":"2024-04-16T21:12:43.204744+0000","last_peered":"2024-04-16T21:12:43.204744+0000","last_clean":"2024-04-16T21:12:43.204744+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:12:43.204744+0000","last_undegraded":"2024-04-16T21:12:43.204744+0000","last_fullsized":"2024-04-16T21:12:43.204744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.204682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:58.459466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":293,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649866+0000","last_change":"2024-04-16T19:23:46.115717+0000","last_active":"2024-04-16T21:11:43.649866+0000","last_peered":"2024-04-16T21:11:43.649866+0000","last_clean":"2024-04-16T21:11:43.649866+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:11:43.649866+0000","last_undegraded":"2024-04-16T21:11:43.649866+0000","last_fullsized":"2024-04-16T21:11:43.649866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:45.108082+0000","last_clean_scrub_stamp":"2024-04-16T19:23:45.108082+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560124+0000","last_change":"2024-04-16T19:23:47.119659+0000","last_active":"2024-04-16T21:11:44.560124+0000","last_peered":"2024-04-16T21:11:44.560124+0000","last_clean":"2024-04-16T21:11:44.560124+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:11:44.560124+0000","last_undegraded":"2024-04-16T21:11:44.560124+0000","last_fullsized":"2024-04-16T21:11:44.560124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384510+0000","last_change":"2024-04-16T19:24:33.386376+0000","last_active":"2024-04-16T21:10:49.384510+0000","last_peered":"2024-04-16T21:10:49.384510+0000","last_clean":"2024-04-16T21:10:49.384510+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:10:49.384510+0000","last_undegraded":"2024-04-16T21:10:49.384510+0000","last_fullsized":"2024-04-16T21:10:49.384510+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.706698+0000","last_change":"2024-04-16T21:12:37.706720+0000","last_active":"2024-04-16T21:12:37.706698+0000","last_peered":"2024-04-16T21:12:37.706698+0000","last_clean":"2024-04-16T21:12:37.706698+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:12:37.706698+0000","last_undegraded":"2024-04-16T21:12:37.706698+0000","last_fullsized":"2024-04-16T21:12:37.706698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.706633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:46.200865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'73","reported_seq":244,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.829156+0000","last_change":"2024-04-16T21:12:32.829190+0000","last_active":"2024-04-16T21:12:32.829156+0000","last_peered":"2024-04-16T21:12:32.829156+0000","last_clean":"2024-04-16T21:12:32.829156+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:12:32.829156+0000","last_undegraded":"2024-04-16T21:12:32.829156+0000","last_fullsized":"2024-04-16T21:12:32.829156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'73","last_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_deep_scrub":"52'73","last_deep_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.829094+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:32.804059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.313554+0000","last_change":"2024-04-16T21:12:28.313577+0000","last_active":"2024-04-16T21:12:28.313554+0000","last_peered":"2024-04-16T21:12:28.313554+0000","last_clean":"2024-04-16T21:12:28.313554+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:12:28.313554+0000","last_undegraded":"2024-04-16T21:12:28.313554+0000","last_fullsized":"2024-04-16T21:12:28.313554+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.313486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:34.551134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.689142+0000","last_change":"2024-04-16T21:12:06.689165+0000","last_active":"2024-04-16T21:12:06.689142+0000","last_peered":"2024-04-16T21:12:06.689142+0000","last_clean":"2024-04-16T21:12:06.689142+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:12:06.689142+0000","last_undegraded":"2024-04-16T21:12:06.689142+0000","last_fullsized":"2024-04-16T21:12:06.689142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.689080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:58.869391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385439+0000","last_change":"2024-04-16T19:47:33.980898+0000","last_active":"2024-04-16T21:10:49.385439+0000","last_peered":"2024-04-16T21:10:49.385439+0000","last_clean":"2024-04-16T21:10:49.385439+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:10:49.385439+0000","last_undegraded":"2024-04-16T21:10:49.385439+0000","last_fullsized":"2024-04-16T21:10:49.385439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'183","reported_seq":2009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334593+0000","last_change":"2024-04-16T21:12:30.334647+0000","last_active":"2024-04-16T21:12:30.334593+0000","last_peered":"2024-04-16T21:12:30.334593+0000","last_clean":"2024-04-16T21:12:30.334593+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:12:30.334593+0000","last_undegraded":"2024-04-16T21:12:30.334593+0000","last_fullsized":"2024-04-16T21:12:30.334593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334529+0000","objects_scrubbed":48,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:53.601279+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14209,"num_omap_keys":58,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.856959+0000","last_change":"2024-04-16T21:12:37.856984+0000","last_active":"2024-04-16T21:12:37.856959+0000","last_peered":"2024-04-16T21:12:37.856959+0000","last_clean":"2024-04-16T21:12:37.856959+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:12:37.856959+0000","last_undegraded":"2024-04-16T21:12:37.856959+0000","last_fullsized":"2024-04-16T21:12:37.856959+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.856878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:44.305267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'16","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.513309+0000","last_change":"2024-04-16T21:12:20.513369+0000","last_active":"2024-04-16T21:12:20.513309+0000","last_peered":"2024-04-16T21:12:20.513309+0000","last_clean":"2024-04-16T21:12:20.513309+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:12:20.513309+0000","last_undegraded":"2024-04-16T21:12:20.513309+0000","last_fullsized":"2024-04-16T21:12:20.513309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.513228+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:32.115899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'706","reported_seq":2087,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.915788+0000","last_change":"2024-04-16T21:11:48.915841+0000","last_active":"2024-04-16T21:11:48.915788+0000","last_peered":"2024-04-16T21:11:48.915788+0000","last_clean":"2024-04-16T21:11:48.915788+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:11:48.915788+0000","last_undegraded":"2024-04-16T21:11:48.915788+0000","last_fullsized":"2024-04-16T21:11:48.915788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.915706+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.964684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'326","reported_seq":644,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.279300+0000","last_change":"2024-04-16T21:11:56.279412+0000","last_active":"2024-04-16T21:11:56.279300+0000","last_peered":"2024-04-16T21:11:56.279300+0000","last_clean":"2024-04-16T21:11:56.279300+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:11:56.279300+0000","last_undegraded":"2024-04-16T21:11:56.279300+0000","last_fullsized":"2024-04-16T21:11:56.279300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'326","last_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_deep_scrub":"52'326","last_deep_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.279222+0000","objects_scrubbed":20,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:03.299266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'20","reported_seq":201,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.117827+0000","last_change":"2024-04-16T21:11:45.117905+0000","last_active":"2024-04-16T21:11:45.117827+0000","last_peered":"2024-04-16T21:11:45.117827+0000","last_clean":"2024-04-16T21:11:45.117827+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:11:45.117827+0000","last_undegraded":"2024-04-16T21:11:45.117827+0000","last_fullsized":"2024-04-16T21:11:45.117827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.117757+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:23.210409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390704+0000","last_change":"2024-04-16T19:24:35.397288+0000","last_active":"2024-04-16T21:10:49.390704+0000","last_peered":"2024-04-16T21:10:49.390704+0000","last_clean":"2024-04-16T21:10:49.390704+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:10:49.390704+0000","last_undegraded":"2024-04-16T21:10:49.390704+0000","last_fullsized":"2024-04-16T21:10:49.390704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649881+0000","last_change":"2024-04-16T19:23:47.119781+0000","last_active":"2024-04-16T21:11:43.649881+0000","last_peered":"2024-04-16T21:11:43.649881+0000","last_clean":"2024-04-16T21:11:43.649881+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:11:43.649881+0000","last_undegraded":"2024-04-16T21:11:43.649881+0000","last_fullsized":"2024-04-16T21:11:43.649881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390485+0000","last_change":"2024-04-16T19:24:33.383869+0000","last_active":"2024-04-16T21:10:49.390485+0000","last_peered":"2024-04-16T21:10:49.390485+0000","last_clean":"2024-04-16T21:10:49.390485+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:10:49.390485+0000","last_undegraded":"2024-04-16T21:10:49.390485+0000","last_fullsized":"2024-04-16T21:10:49.390485+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"52'8","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.885381+0000","last_change":"2024-04-16T21:12:40.885408+0000","last_active":"2024-04-16T21:12:40.885381+0000","last_peered":"2024-04-16T21:12:40.885381+0000","last_clean":"2024-04-16T21:12:40.885381+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:12:40.885381+0000","last_undegraded":"2024-04-16T21:12:40.885381+0000","last_fullsized":"2024-04-16T21:12:40.885381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.885321+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:11.985385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'175","reported_seq":1588,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.038998+0000","last_change":"2024-04-16T21:12:33.039035+0000","last_active":"2024-04-16T21:12:33.038998+0000","last_peered":"2024-04-16T21:12:33.038998+0000","last_clean":"2024-04-16T21:12:33.038998+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:12:33.038998+0000","last_undegraded":"2024-04-16T21:12:33.038998+0000","last_fullsized":"2024-04-16T21:12:33.038998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'175","last_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_deep_scrub":"52'175","last_deep_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.038938+0000","objects_scrubbed":10,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:04.725640+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.604589+0000","last_change":"2024-04-16T21:12:24.604613+0000","last_active":"2024-04-16T21:12:24.604589+0000","last_peered":"2024-04-16T21:12:24.604589+0000","last_clean":"2024-04-16T21:12:24.604589+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:12:24.604589+0000","last_undegraded":"2024-04-16T21:12:24.604589+0000","last_fullsized":"2024-04-16T21:12:24.604589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.604487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.850465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.671618+0000","last_change":"2024-04-16T21:11:57.671660+0000","last_active":"2024-04-16T21:11:57.671618+0000","last_peered":"2024-04-16T21:11:57.671618+0000","last_clean":"2024-04-16T21:11:57.671618+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:11:57.671618+0000","last_undegraded":"2024-04-16T21:11:57.671618+0000","last_fullsized":"2024-04-16T21:11:57.671618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.671545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:20.862941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385351+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:10:49.385351+0000","last_peered":"2024-04-16T21:10:49.385351+0000","last_clean":"2024-04-16T21:10:49.385351+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:10:49.385351+0000","last_undegraded":"2024-04-16T21:10:49.385351+0000","last_fullsized":"2024-04-16T21:10:49.385351+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2370,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392699+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:11:34.392699+0000","last_peered":"2024-04-16T21:11:34.392699+0000","last_clean":"2024-04-16T21:11:34.392699+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:11:34.392699+0000","last_undegraded":"2024-04-16T21:11:34.392699+0000","last_fullsized":"2024-04-16T21:11:34.392699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.830577+0000","last_change":"2024-04-16T21:12:25.830601+0000","last_active":"2024-04-16T21:12:25.830577+0000","last_peered":"2024-04-16T21:12:25.830577+0000","last_clean":"2024-04-16T21:12:25.830577+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:12:25.830577+0000","last_undegraded":"2024-04-16T21:12:25.830577+0000","last_fullsized":"2024-04-16T21:12:25.830577+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.830513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:35.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'19","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.380245+0000","last_change":"2024-04-16T21:12:21.380273+0000","last_active":"2024-04-16T21:12:21.380245+0000","last_peered":"2024-04-16T21:12:21.380245+0000","last_clean":"2024-04-16T21:12:21.380245+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:12:21.380245+0000","last_undegraded":"2024-04-16T21:12:21.380245+0000","last_fullsized":"2024-04-16T21:12:21.380245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.380184+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:53.066886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":60,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'21","reported_seq":982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.783889+0000","last_change":"2024-04-16T21:11:58.783920+0000","last_active":"2024-04-16T21:11:58.783889+0000","last_peered":"2024-04-16T21:11:58.783889+0000","last_clean":"2024-04-16T21:11:58.783889+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:11:58.783889+0000","last_undegraded":"2024-04-16T21:11:58.783889+0000","last_fullsized":"2024-04-16T21:11:58.783889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.783819+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:59.285203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'268","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.359130+0000","last_change":"2024-04-16T21:11:44.359183+0000","last_active":"2024-04-16T21:11:44.359130+0000","last_peered":"2024-04-16T21:11:44.359130+0000","last_clean":"2024-04-16T21:11:44.359130+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:11:44.359130+0000","last_undegraded":"2024-04-16T21:11:44.359130+0000","last_fullsized":"2024-04-16T21:11:44.359130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'268","last_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_deep_scrub":"52'268","last_deep_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.359071+0000","objects_scrubbed":19,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:18.502556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'13","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.427731+0000","last_change":"2024-04-16T21:11:53.427771+0000","last_active":"2024-04-16T21:11:53.427731+0000","last_peered":"2024-04-16T21:11:53.427731+0000","last_clean":"2024-04-16T21:11:53.427731+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:11:53.427731+0000","last_undegraded":"2024-04-16T21:11:53.427731+0000","last_fullsized":"2024-04-16T21:11:53.427731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.427656+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:14.929241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.080937+0000","last_change":"2024-04-16T21:12:42.080980+0000","last_active":"2024-04-16T21:12:42.080937+0000","last_peered":"2024-04-16T21:12:42.080937+0000","last_clean":"2024-04-16T21:12:42.080937+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:12:42.080937+0000","last_undegraded":"2024-04-16T21:12:42.080937+0000","last_fullsized":"2024-04-16T21:12:42.080937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.080859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:48.529247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.765443+0000","last_change":"2024-04-16T21:12:42.765465+0000","last_active":"2024-04-16T21:12:42.765443+0000","last_peered":"2024-04-16T21:12:42.765443+0000","last_clean":"2024-04-16T21:12:42.765443+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:12:42.765443+0000","last_undegraded":"2024-04-16T21:12:42.765443+0000","last_fullsized":"2024-04-16T21:12:42.765443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.765375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:30.897866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385426+0000","last_change":"2024-04-16T19:24:33.387787+0000","last_active":"2024-04-16T21:10:54.385426+0000","last_peered":"2024-04-16T21:10:54.385426+0000","last_clean":"2024-04-16T21:10:54.385426+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:10:54.385426+0000","last_undegraded":"2024-04-16T21:10:54.385426+0000","last_fullsized":"2024-04-16T21:10:54.385426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"52'6","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.801277+0000","last_change":"2024-04-16T21:12:34.801302+0000","last_active":"2024-04-16T21:12:34.801277+0000","last_peered":"2024-04-16T21:12:34.801277+0000","last_clean":"2024-04-16T21:12:34.801277+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:12:34.801277+0000","last_undegraded":"2024-04-16T21:12:34.801277+0000","last_fullsized":"2024-04-16T21:12:34.801277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.801209+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:13.234565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'147","reported_seq":1530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.863577+0000","last_change":"2024-04-16T21:12:39.863616+0000","last_active":"2024-04-16T21:12:39.863577+0000","last_peered":"2024-04-16T21:12:39.863577+0000","last_clean":"2024-04-16T21:12:39.863577+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:12:39.863577+0000","last_undegraded":"2024-04-16T21:12:39.863577+0000","last_fullsized":"2024-04-16T21:12:39.863577+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.863516+0000","objects_scrubbed":8,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:55.662812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.339655+0000","last_change":"2024-04-16T21:12:27.339693+0000","last_active":"2024-04-16T21:12:27.339655+0000","last_peered":"2024-04-16T21:12:27.339655+0000","last_clean":"2024-04-16T21:12:27.339655+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:12:27.339655+0000","last_undegraded":"2024-04-16T21:12:27.339655+0000","last_fullsized":"2024-04-16T21:12:27.339655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.339563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:33.787952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.370511+0000","last_change":"2024-04-16T21:12:05.370546+0000","last_active":"2024-04-16T21:12:05.370511+0000","last_peered":"2024-04-16T21:12:05.370511+0000","last_clean":"2024-04-16T21:12:05.370511+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:12:05.370511+0000","last_undegraded":"2024-04-16T21:12:05.370511+0000","last_fullsized":"2024-04-16T21:12:05.370511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.370430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:17.165162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388785+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:11:04.388785+0000","last_peered":"2024-04-16T21:11:04.388785+0000","last_clean":"2024-04-16T21:11:04.388785+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:11:04.388785+0000","last_undegraded":"2024-04-16T21:11:04.388785+0000","last_fullsized":"2024-04-16T21:11:04.388785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'117","reported_seq":1767,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392656+0000","last_change":"2024-04-16T19:25:49.519404+0000","last_active":"2024-04-16T21:11:34.392656+0000","last_peered":"2024-04-16T21:11:34.392656+0000","last_clean":"2024-04-16T21:11:34.392656+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:11:34.392656+0000","last_undegraded":"2024-04-16T21:11:34.392656+0000","last_fullsized":"2024-04-16T21:11:34.392656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.699415+0000","last_change":"2024-04-16T21:11:59.699455+0000","last_active":"2024-04-16T21:11:59.699415+0000","last_peered":"2024-04-16T21:11:59.699415+0000","last_clean":"2024-04-16T21:11:59.699415+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:11:59.699415+0000","last_undegraded":"2024-04-16T21:11:59.699415+0000","last_fullsized":"2024-04-16T21:11:59.699415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.699336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'21","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.432050+0000","last_change":"2024-04-16T21:12:15.432112+0000","last_active":"2024-04-16T21:12:15.432050+0000","last_peered":"2024-04-16T21:12:15.432050+0000","last_clean":"2024-04-16T21:12:15.432050+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:12:15.432050+0000","last_undegraded":"2024-04-16T21:12:15.432050+0000","last_fullsized":"2024-04-16T21:12:15.432050+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.431977+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:24.993645+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.891648+0000","last_change":"2024-04-16T21:12:03.891673+0000","last_active":"2024-04-16T21:12:03.891648+0000","last_peered":"2024-04-16T21:12:03.891648+0000","last_clean":"2024-04-16T21:12:03.891648+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:12:03.891648+0000","last_undegraded":"2024-04-16T21:12:03.891648+0000","last_fullsized":"2024-04-16T21:12:03.891648+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.891582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:27.082978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'720","reported_seq":1730,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.792306+0000","last_change":"2024-04-16T21:11:44.792348+0000","last_active":"2024-04-16T21:11:44.792306+0000","last_peered":"2024-04-16T21:11:44.792306+0000","last_clean":"2024-04-16T21:11:44.792306+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:11:44.792306+0000","last_undegraded":"2024-04-16T21:11:44.792306+0000","last_fullsized":"2024-04-16T21:11:44.792306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'720","last_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_deep_scrub":"52'720","last_deep_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.792244+0000","objects_scrubbed":30,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:07:06.408902+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2191,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'20","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.222406+0000","last_change":"2024-04-16T21:11:49.222443+0000","last_active":"2024-04-16T21:11:49.222406+0000","last_peered":"2024-04-16T21:11:49.222406+0000","last_clean":"2024-04-16T21:11:49.222406+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:11:49.222406+0000","last_undegraded":"2024-04-16T21:11:49.222406+0000","last_fullsized":"2024-04-16T21:11:49.222406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.222345+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:21.815371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.385547+0000","last_change":"2024-04-16T19:24:35.393242+0000","last_active":"2024-04-16T21:10:54.385547+0000","last_peered":"2024-04-16T21:10:54.385547+0000","last_clean":"2024-04-16T21:10:54.385547+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:10:54.385547+0000","last_undegraded":"2024-04-16T21:10:54.385547+0000","last_fullsized":"2024-04-16T21:10:54.385547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560100+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:11:44.560100+0000","last_peered":"2024-04-16T21:11:44.560100+0000","last_clean":"2024-04-16T21:11:44.560100+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:11:44.560100+0000","last_undegraded":"2024-04-16T21:11:44.560100+0000","last_fullsized":"2024-04-16T21:11:44.560100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385348+0000","last_change":"2024-04-16T19:24:33.381038+0000","last_active":"2024-04-16T21:10:49.385348+0000","last_peered":"2024-04-16T21:10:49.385348+0000","last_clean":"2024-04-16T21:10:49.385348+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:10:49.385348+0000","last_undegraded":"2024-04-16T21:10:49.385348+0000","last_fullsized":"2024-04-16T21:10:49.385348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:24.396579+0000","last_change":"2024-04-16T19:24:45.409274+0000","last_active":"2024-04-16T21:11:24.396579+0000","last_peered":"2024-04-16T21:11:24.396579+0000","last_clean":"2024-04-16T21:11:24.396579+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:11:24.396579+0000","last_undegraded":"2024-04-16T21:11:24.396579+0000","last_fullsized":"2024-04-16T21:11:24.396579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'138","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.993884+0000","last_change":"2024-04-16T21:12:33.993917+0000","last_active":"2024-04-16T21:12:33.993884+0000","last_peered":"2024-04-16T21:12:33.993884+0000","last_clean":"2024-04-16T21:12:33.993884+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:12:33.993884+0000","last_undegraded":"2024-04-16T21:12:33.993884+0000","last_fullsized":"2024-04-16T21:12:33.993884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.993823+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:28.182498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.394475+0000","last_change":"2024-04-16T21:12:09.394517+0000","last_active":"2024-04-16T21:12:09.394475+0000","last_peered":"2024-04-16T21:12:09.394475+0000","last_clean":"2024-04-16T21:12:09.394475+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:12:09.394475+0000","last_undegraded":"2024-04-16T21:12:09.394475+0000","last_fullsized":"2024-04-16T21:12:09.394475+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.394403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:58.357952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.902684+0000","last_change":"2024-04-16T21:12:14.902708+0000","last_active":"2024-04-16T21:12:14.902684+0000","last_peered":"2024-04-16T21:12:14.902684+0000","last_clean":"2024-04-16T21:12:14.902684+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:12:14.902684+0000","last_undegraded":"2024-04-16T21:12:14.902684+0000","last_fullsized":"2024-04-16T21:12:14.902684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.902624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:23.524467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561653+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:11:44.561653+0000","last_peered":"2024-04-16T21:11:44.561653+0000","last_clean":"2024-04-16T21:11:44.561653+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:11:44.561653+0000","last_undegraded":"2024-04-16T21:11:44.561653+0000","last_fullsized":"2024-04-16T21:11:44.561653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1336,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293264+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:11:42.293264+0000","last_peered":"2024-04-16T21:11:42.293264+0000","last_clean":"2024-04-16T21:11:42.293264+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:11:42.293264+0000","last_undegraded":"2024-04-16T21:11:42.293264+0000","last_fullsized":"2024-04-16T21:11:42.293264+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.320686+0000","last_change":"2024-04-16T21:12:28.320708+0000","last_active":"2024-04-16T21:12:28.320686+0000","last_peered":"2024-04-16T21:12:28.320686+0000","last_clean":"2024-04-16T21:12:28.320686+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:12:28.320686+0000","last_undegraded":"2024-04-16T21:12:28.320686+0000","last_fullsized":"2024-04-16T21:12:28.320686+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.320608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:14.618412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'6","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.781379+0000","last_change":"2024-04-16T21:11:54.781414+0000","last_active":"2024-04-16T21:11:54.781379+0000","last_peered":"2024-04-16T21:11:54.781379+0000","last_clean":"2024-04-16T21:11:54.781379+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:11:54.781379+0000","last_undegraded":"2024-04-16T21:11:54.781379+0000","last_fullsized":"2024-04-16T21:11:54.781379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.781318+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.682337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'706","reported_seq":2169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.804811+0000","last_change":"2024-04-16T21:11:57.804839+0000","last_active":"2024-04-16T21:11:57.804811+0000","last_peered":"2024-04-16T21:11:57.804811+0000","last_clean":"2024-04-16T21:11:57.804811+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:11:57.804811+0000","last_undegraded":"2024-04-16T21:11:57.804811+0000","last_fullsized":"2024-04-16T21:11:57.804811+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.804739+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:38:55.231266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'264","reported_seq":495,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.746009+0000","last_change":"2024-04-16T21:11:46.746078+0000","last_active":"2024-04-16T21:11:46.746009+0000","last_peered":"2024-04-16T21:11:46.746009+0000","last_clean":"2024-04-16T21:11:46.746009+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:11:46.746009+0000","last_undegraded":"2024-04-16T21:11:46.746009+0000","last_fullsized":"2024-04-16T21:11:46.746009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.745946+0000","objects_scrubbed":17,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:43.933138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1751,"num_omap_keys":8,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'38","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.693339+0000","last_change":"2024-04-16T21:11:52.693388+0000","last_active":"2024-04-16T21:11:52.693339+0000","last_peered":"2024-04-16T21:11:52.693339+0000","last_clean":"2024-04-16T21:11:52.693339+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:11:52.693339+0000","last_undegraded":"2024-04-16T21:11:52.693339+0000","last_fullsized":"2024-04-16T21:11:52.693339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'38","last_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_deep_scrub":"52'38","last_deep_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.693273+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:15.884668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.125696+0000","last_change":"2024-04-16T21:12:43.125718+0000","last_active":"2024-04-16T21:12:43.125696+0000","last_peered":"2024-04-16T21:12:43.125696+0000","last_clean":"2024-04-16T21:12:43.125696+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:12:43.125696+0000","last_undegraded":"2024-04-16T21:12:43.125696+0000","last_fullsized":"2024-04-16T21:12:43.125696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.125631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:49.363279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388567+0000","last_change":"2024-04-16T19:24:33.389003+0000","last_active":"2024-04-16T21:11:09.388567+0000","last_peered":"2024-04-16T21:11:09.388567+0000","last_clean":"2024-04-16T21:11:09.388567+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:11:09.388567+0000","last_undegraded":"2024-04-16T21:11:09.388567+0000","last_fullsized":"2024-04-16T21:11:09.388567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742796+0000","last_change":"2024-04-16T19:23:47.184291+0000","last_active":"2024-04-16T21:11:42.742796+0000","last_peered":"2024-04-16T21:11:42.742796+0000","last_clean":"2024-04-16T21:11:42.742796+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:11:42.742796+0000","last_undegraded":"2024-04-16T21:11:42.742796+0000","last_fullsized":"2024-04-16T21:11:42.742796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.378796+0000","last_change":"2024-04-16T21:12:43.378818+0000","last_active":"2024-04-16T21:12:43.378796+0000","last_peered":"2024-04-16T21:12:43.378796+0000","last_clean":"2024-04-16T21:12:43.378796+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:12:43.378796+0000","last_undegraded":"2024-04-16T21:12:43.378796+0000","last_fullsized":"2024-04-16T21:12:43.378796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.378722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:20.670430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'156","reported_seq":366,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.280884+0000","last_change":"2024-04-16T21:12:37.280951+0000","last_active":"2024-04-16T21:12:37.280884+0000","last_peered":"2024-04-16T21:12:37.280884+0000","last_clean":"2024-04-16T21:12:37.280884+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:12:37.280884+0000","last_undegraded":"2024-04-16T21:12:37.280884+0000","last_fullsized":"2024-04-16T21:12:37.280884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'156","last_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_deep_scrub":"52'156","last_deep_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.280808+0000","objects_scrubbed":8,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:22.038566+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.612982+0000","last_change":"2024-04-16T21:12:26.613002+0000","last_active":"2024-04-16T21:12:26.612982+0000","last_peered":"2024-04-16T21:12:26.612982+0000","last_clean":"2024-04-16T21:12:26.612982+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:12:26.612982+0000","last_undegraded":"2024-04-16T21:12:26.612982+0000","last_fullsized":"2024-04-16T21:12:26.612982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.612912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.407643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.345372+0000","last_change":"2024-04-16T21:12:01.345418+0000","last_active":"2024-04-16T21:12:01.345372+0000","last_peered":"2024-04-16T21:12:01.345372+0000","last_clean":"2024-04-16T21:12:01.345372+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:12:01.345372+0000","last_undegraded":"2024-04-16T21:12:01.345372+0000","last_fullsized":"2024-04-16T21:12:01.345372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.345289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:22.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.909918+0000","last_change":"2024-04-16T21:12:24.909962+0000","last_active":"2024-04-16T21:12:24.909918+0000","last_peered":"2024-04-16T21:12:24.909918+0000","last_clean":"2024-04-16T21:12:24.909918+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:12:24.909918+0000","last_undegraded":"2024-04-16T21:12:24.909918+0000","last_fullsized":"2024-04-16T21:12:24.909918+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.909836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:30.648976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'194","reported_seq":2060,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.906040+0000","last_change":"2024-04-16T21:12:41.906084+0000","last_active":"2024-04-16T21:12:41.906040+0000","last_peered":"2024-04-16T21:12:41.906040+0000","last_clean":"2024-04-16T21:12:41.906040+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:12:41.906040+0000","last_undegraded":"2024-04-16T21:12:41.906040+0000","last_fullsized":"2024-04-16T21:12:41.906040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'194","last_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_deep_scrub":"52'194","last_deep_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.905983+0000","objects_scrubbed":47,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:44.692749+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12654,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.836131+0000","last_change":"2024-04-16T21:12:08.836131+0000","last_active":"2024-04-16T21:12:08.836131+0000","last_peered":"2024-04-16T21:12:08.836131+0000","last_clean":"2024-04-16T21:12:08.836131+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:12:08.836131+0000","last_undegraded":"2024-04-16T21:12:08.836131+0000","last_fullsized":"2024-04-16T21:12:08.836131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.836015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:33.006060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'30","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.417236+0000","last_change":"2024-04-16T21:12:16.417277+0000","last_active":"2024-04-16T21:12:16.417236+0000","last_peered":"2024-04-16T21:12:16.417236+0000","last_clean":"2024-04-16T21:12:16.417236+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:12:16.417236+0000","last_undegraded":"2024-04-16T21:12:16.417236+0000","last_fullsized":"2024-04-16T21:12:16.417236+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'30","last_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_deep_scrub":"52'30","last_deep_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.417171+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:08.090860+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.802564+0000","last_change":"2024-04-16T21:12:02.802585+0000","last_active":"2024-04-16T21:12:02.802564+0000","last_peered":"2024-04-16T21:12:02.802564+0000","last_clean":"2024-04-16T21:12:02.802564+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:12:02.802564+0000","last_undegraded":"2024-04-16T21:12:02.802564+0000","last_fullsized":"2024-04-16T21:12:02.802564+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.802497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:44.917953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'739","reported_seq":1609,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.752466+0000","last_change":"2024-04-16T21:11:47.752492+0000","last_active":"2024-04-16T21:11:47.752466+0000","last_peered":"2024-04-16T21:11:47.752466+0000","last_clean":"2024-04-16T21:11:47.752466+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:11:47.752466+0000","last_undegraded":"2024-04-16T21:11:47.752466+0000","last_fullsized":"2024-04-16T21:11:47.752466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'739","last_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_deep_scrub":"52'739","last_deep_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.752406+0000","objects_scrubbed":25,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:14.225249+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6099,"num_omap_keys":28,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'12","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.362460+0000","last_change":"2024-04-16T21:11:45.362519+0000","last_active":"2024-04-16T21:11:45.362460+0000","last_peered":"2024-04-16T21:11:45.362460+0000","last_clean":"2024-04-16T21:11:45.362460+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:11:45.362460+0000","last_undegraded":"2024-04-16T21:11:45.362460+0000","last_fullsized":"2024-04-16T21:11:45.362460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.362399+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:08.553795+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391347+0000","last_change":"2024-04-16T19:24:35.401390+0000","last_active":"2024-04-16T21:10:54.391347+0000","last_peered":"2024-04-16T21:10:54.391347+0000","last_clean":"2024-04-16T21:10:54.391347+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:10:54.391347+0000","last_undegraded":"2024-04-16T21:10:54.391347+0000","last_fullsized":"2024-04-16T21:10:54.391347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384586+0000","last_change":"2024-04-16T19:24:33.385698+0000","last_active":"2024-04-16T21:10:49.384586+0000","last_peered":"2024-04-16T21:10:49.384586+0000","last_clean":"2024-04-16T21:10:49.384586+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:10:49.384586+0000","last_undegraded":"2024-04-16T21:10:49.384586+0000","last_fullsized":"2024-04-16T21:10:49.384586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649882+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:11:43.649882+0000","last_peered":"2024-04-16T21:11:43.649882+0000","last_clean":"2024-04-16T21:11:43.649882+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:11:43.649882+0000","last_undegraded":"2024-04-16T21:11:43.649882+0000","last_fullsized":"2024-04-16T21:11:43.649882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387738+0000","last_change":"2024-04-16T19:24:45.414283+0000","last_active":"2024-04-16T21:11:04.387738+0000","last_peered":"2024-04-16T21:11:04.387738+0000","last_clean":"2024-04-16T21:11:04.387738+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:11:04.387738+0000","last_undegraded":"2024-04-16T21:11:04.387738+0000","last_fullsized":"2024-04-16T21:11:04.387738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'161","reported_seq":362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.967720+0000","last_change":"2024-04-16T21:12:27.967756+0000","last_active":"2024-04-16T21:12:27.967720+0000","last_peered":"2024-04-16T21:12:27.967720+0000","last_clean":"2024-04-16T21:12:27.967720+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:12:27.967720+0000","last_undegraded":"2024-04-16T21:12:27.967720+0000","last_fullsized":"2024-04-16T21:12:27.967720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'161","last_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_deep_scrub":"52'161","last_deep_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.967658+0000","objects_scrubbed":7,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:06.401014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'13","reported_seq":1327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.766646+0000","last_change":"2024-04-16T21:12:27.766675+0000","last_active":"2024-04-16T21:12:27.766646+0000","last_peered":"2024-04-16T21:12:27.766646+0000","last_clean":"2024-04-16T21:12:27.766646+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:12:27.766646+0000","last_undegraded":"2024-04-16T21:12:27.766646+0000","last_fullsized":"2024-04-16T21:12:27.766646+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.766582+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:36.260814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.393316+0000","last_change":"2024-04-16T21:12:00.393341+0000","last_active":"2024-04-16T21:12:00.393316+0000","last_peered":"2024-04-16T21:12:00.393316+0000","last_clean":"2024-04-16T21:12:00.393316+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:12:00.393316+0000","last_undegraded":"2024-04-16T21:12:00.393316+0000","last_fullsized":"2024-04-16T21:12:00.393316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.393227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:57.237156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524744+0000","last_change":"2024-04-16T19:47:33.989459+0000","last_active":"2024-04-16T21:10:49.524744+0000","last_peered":"2024-04-16T21:10:49.524744+0000","last_clean":"2024-04-16T21:10:49.524744+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:10:49.524744+0000","last_undegraded":"2024-04-16T21:10:49.524744+0000","last_fullsized":"2024-04-16T21:10:49.524744+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'148","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.332144+0000","last_change":"2024-04-16T21:12:36.332178+0000","last_active":"2024-04-16T21:12:36.332144+0000","last_peered":"2024-04-16T21:12:36.332144+0000","last_clean":"2024-04-16T21:12:36.332144+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:12:36.332144+0000","last_undegraded":"2024-04-16T21:12:36.332144+0000","last_fullsized":"2024-04-16T21:12:36.332144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'148","last_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_deep_scrub":"52'148","last_deep_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.332084+0000","objects_scrubbed":32,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:33.307525+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11265,"num_omap_keys":45,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.939107+0000","last_change":"2024-04-16T21:12:19.939131+0000","last_active":"2024-04-16T21:12:19.939107+0000","last_peered":"2024-04-16T21:12:19.939107+0000","last_clean":"2024-04-16T21:12:19.939107+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:12:19.939107+0000","last_undegraded":"2024-04-16T21:12:19.939107+0000","last_fullsized":"2024-04-16T21:12:19.939107+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.939043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:23.022714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'4","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.763978+0000","last_change":"2024-04-16T21:11:55.764028+0000","last_active":"2024-04-16T21:11:55.763978+0000","last_peered":"2024-04-16T21:11:55.763978+0000","last_clean":"2024-04-16T21:11:55.763978+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:11:55.763978+0000","last_undegraded":"2024-04-16T21:11:55.763978+0000","last_fullsized":"2024-04-16T21:11:55.763978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.763887+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.184478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.416929+0000","last_change":"2024-04-16T21:12:12.416959+0000","last_active":"2024-04-16T21:12:12.416929+0000","last_peered":"2024-04-16T21:12:12.416929+0000","last_clean":"2024-04-16T21:12:12.416929+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:12:12.416929+0000","last_undegraded":"2024-04-16T21:12:12.416929+0000","last_fullsized":"2024-04-16T21:12:12.416929+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.416865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:15.500537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'287","reported_seq":553,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.658905+0000","last_change":"2024-04-16T21:11:54.658986+0000","last_active":"2024-04-16T21:11:54.658905+0000","last_peered":"2024-04-16T21:11:54.658905+0000","last_clean":"2024-04-16T21:11:54.658905+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:11:54.658905+0000","last_undegraded":"2024-04-16T21:11:54.658905+0000","last_fullsized":"2024-04-16T21:11:54.658905+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'287","last_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_deep_scrub":"52'287","last_deep_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.658842+0000","objects_scrubbed":20,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.559862+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6346,"num_omap_keys":29,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'22","reported_seq":213,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.414533+0000","last_change":"2024-04-16T21:11:51.414602+0000","last_active":"2024-04-16T21:11:51.414533+0000","last_peered":"2024-04-16T21:11:51.414533+0000","last_clean":"2024-04-16T21:11:51.414533+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:11:51.414533+0000","last_undegraded":"2024-04-16T21:11:51.414533+0000","last_fullsized":"2024-04-16T21:11:51.414533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.414471+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:12.853053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.392142+0000","last_change":"2024-04-16T19:24:35.396732+0000","last_active":"2024-04-16T21:10:59.392142+0000","last_peered":"2024-04-16T21:10:59.392142+0000","last_clean":"2024-04-16T21:10:59.392142+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:10:59.392142+0000","last_undegraded":"2024-04-16T21:10:59.392142+0000","last_fullsized":"2024-04-16T21:10:59.392142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.719411+0000","last_change":"2024-04-16T21:12:43.719433+0000","last_active":"2024-04-16T21:12:43.719411+0000","last_peered":"2024-04-16T21:12:43.719411+0000","last_clean":"2024-04-16T21:12:43.719411+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:12:43.719411+0000","last_undegraded":"2024-04-16T21:12:43.719411+0000","last_fullsized":"2024-04-16T21:12:43.719411+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.719343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:43.694308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742796+0000","last_change":"2024-04-16T19:23:47.122776+0000","last_active":"2024-04-16T21:11:42.742796+0000","last_peered":"2024-04-16T21:11:42.742796+0000","last_clean":"2024-04-16T21:11:42.742796+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:11:42.742796+0000","last_undegraded":"2024-04-16T21:11:42.742796+0000","last_fullsized":"2024-04-16T21:11:42.742796+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.211328+0000","last_change":"2024-04-16T21:12:42.211369+0000","last_active":"2024-04-16T21:12:42.211328+0000","last_peered":"2024-04-16T21:12:42.211328+0000","last_clean":"2024-04-16T21:12:42.211328+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:12:42.211328+0000","last_undegraded":"2024-04-16T21:12:42.211328+0000","last_fullsized":"2024-04-16T21:12:42.211328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.211237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:13.916029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'115","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.353424+0000","last_change":"2024-04-16T21:12:35.353459+0000","last_active":"2024-04-16T21:12:35.353424+0000","last_peered":"2024-04-16T21:12:35.353424+0000","last_clean":"2024-04-16T21:12:35.353424+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:12:35.353424+0000","last_undegraded":"2024-04-16T21:12:35.353424+0000","last_fullsized":"2024-04-16T21:12:35.353424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.353361+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:35.315569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'13","reported_seq":1325,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.412638+0000","last_change":"2024-04-16T21:12:24.412664+0000","last_active":"2024-04-16T21:12:24.412638+0000","last_peered":"2024-04-16T21:12:24.412638+0000","last_clean":"2024-04-16T21:12:24.412638+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:12:24.412638+0000","last_undegraded":"2024-04-16T21:12:24.412638+0000","last_fullsized":"2024-04-16T21:12:24.412638+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.412571+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:12.450226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.373196+0000","last_change":"2024-04-16T21:11:59.373240+0000","last_active":"2024-04-16T21:11:59.373196+0000","last_peered":"2024-04-16T21:11:59.373196+0000","last_clean":"2024-04-16T21:11:59.373196+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:11:59.373196+0000","last_undegraded":"2024-04-16T21:11:59.373196+0000","last_fullsized":"2024-04-16T21:11:59.373196+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.373112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:02.682886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.196003+0000","last_change":"2024-04-16T19:47:33.989351+0000","last_active":"2024-04-16T21:11:43.196003+0000","last_peered":"2024-04-16T21:11:43.196003+0000","last_clean":"2024-04-16T21:11:43.196003+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:11:43.196003+0000","last_undegraded":"2024-04-16T21:11:43.196003+0000","last_fullsized":"2024-04-16T21:11:43.196003+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1523,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561560+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:11:44.561560+0000","last_peered":"2024-04-16T21:11:44.561560+0000","last_clean":"2024-04-16T21:11:44.561560+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:11:44.561560+0000","last_undegraded":"2024-04-16T21:11:44.561560+0000","last_fullsized":"2024-04-16T21:11:44.561560+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.846610+0000","last_change":"2024-04-16T21:12:00.846636+0000","last_active":"2024-04-16T21:12:00.846610+0000","last_peered":"2024-04-16T21:12:00.846610+0000","last_clean":"2024-04-16T21:12:00.846610+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:12:00.846610+0000","last_undegraded":"2024-04-16T21:12:00.846610+0000","last_fullsized":"2024-04-16T21:12:00.846610+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.846545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:04.784544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'15","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.794385+0000","last_change":"2024-04-16T21:11:56.794414+0000","last_active":"2024-04-16T21:11:56.794385+0000","last_peered":"2024-04-16T21:11:56.794385+0000","last_clean":"2024-04-16T21:11:56.794385+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:11:56.794385+0000","last_undegraded":"2024-04-16T21:11:56.794385+0000","last_fullsized":"2024-04-16T21:11:56.794385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.794323+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.217325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'706","reported_seq":2090,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.681698+0000","last_change":"2024-04-16T21:11:49.681723+0000","last_active":"2024-04-16T21:11:49.681698+0000","last_peered":"2024-04-16T21:11:49.681698+0000","last_clean":"2024-04-16T21:11:49.681698+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:11:49.681698+0000","last_undegraded":"2024-04-16T21:11:49.681698+0000","last_fullsized":"2024-04-16T21:11:49.681698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.681634+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.077508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'335","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.410155+0000","last_change":"2024-04-16T21:11:50.410223+0000","last_active":"2024-04-16T21:11:50.410155+0000","last_peered":"2024-04-16T21:11:50.410155+0000","last_clean":"2024-04-16T21:11:50.410155+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:11:50.410155+0000","last_undegraded":"2024-04-16T21:11:50.410155+0000","last_fullsized":"2024-04-16T21:11:50.410155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'335","last_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_deep_scrub":"52'335","last_deep_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.410083+0000","objects_scrubbed":17,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:14.580127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'10","reported_seq":195,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.912474+0000","last_change":"2024-04-16T21:12:10.912515+0000","last_active":"2024-04-16T21:12:10.912474+0000","last_peered":"2024-04-16T21:12:10.912474+0000","last_clean":"2024-04-16T21:12:10.912474+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:12:10.912474+0000","last_undegraded":"2024-04-16T21:12:10.912474+0000","last_fullsized":"2024-04-16T21:12:10.912474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.912417+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.912619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.743575+0000","last_change":"2024-04-16T21:12:31.743600+0000","last_active":"2024-04-16T21:12:31.743575+0000","last_peered":"2024-04-16T21:12:31.743575+0000","last_clean":"2024-04-16T21:12:31.743575+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:12:31.743575+0000","last_undegraded":"2024-04-16T21:12:31.743575+0000","last_fullsized":"2024-04-16T21:12:31.743575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.743511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:22.862472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.389709+0000","last_change":"2024-04-16T19:24:33.381409+0000","last_active":"2024-04-16T21:10:44.389709+0000","last_peered":"2024-04-16T21:10:44.389709+0000","last_clean":"2024-04-16T21:10:44.389709+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:10:44.389709+0000","last_undegraded":"2024-04-16T21:10:44.389709+0000","last_fullsized":"2024-04-16T21:10:44.389709+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649924+0000","last_change":"2024-04-16T19:23:47.175146+0000","last_active":"2024-04-16T21:11:43.649924+0000","last_peered":"2024-04-16T21:11:43.649924+0000","last_clean":"2024-04-16T21:11:43.649924+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:11:43.649924+0000","last_undegraded":"2024-04-16T21:11:43.649924+0000","last_fullsized":"2024-04-16T21:11:43.649924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.901862+0000","last_change":"2024-04-16T21:12:28.901885+0000","last_active":"2024-04-16T21:12:28.901862+0000","last_peered":"2024-04-16T21:12:28.901862+0000","last_clean":"2024-04-16T21:12:28.901862+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:12:28.901862+0000","last_undegraded":"2024-04-16T21:12:28.901862+0000","last_fullsized":"2024-04-16T21:12:28.901862+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.901794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:33.940739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'142","reported_seq":348,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.606314+0000","last_change":"2024-04-16T21:12:27.606351+0000","last_active":"2024-04-16T21:12:27.606314+0000","last_peered":"2024-04-16T21:12:27.606314+0000","last_clean":"2024-04-16T21:12:27.606314+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:12:27.606314+0000","last_undegraded":"2024-04-16T21:12:27.606314+0000","last_fullsized":"2024-04-16T21:12:27.606314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'142","last_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_deep_scrub":"52'142","last_deep_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.606253+0000","objects_scrubbed":8,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:18.166355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.903416+0000","last_change":"2024-04-16T21:12:17.903439+0000","last_active":"2024-04-16T21:12:17.903416+0000","last_peered":"2024-04-16T21:12:17.903416+0000","last_clean":"2024-04-16T21:12:17.903416+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:12:17.903416+0000","last_undegraded":"2024-04-16T21:12:17.903416+0000","last_fullsized":"2024-04-16T21:12:17.903416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.903352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:24.923396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.864530+0000","last_change":"2024-04-16T21:12:11.864551+0000","last_active":"2024-04-16T21:12:11.864530+0000","last_peered":"2024-04-16T21:12:11.864530+0000","last_clean":"2024-04-16T21:12:11.864530+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:12:11.864530+0000","last_undegraded":"2024-04-16T21:12:11.864530+0000","last_fullsized":"2024-04-16T21:12:11.864530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.864468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:37.984544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293397+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:11:42.293397+0000","last_peered":"2024-04-16T21:11:42.293397+0000","last_clean":"2024-04-16T21:11:42.293397+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:11:42.293397+0000","last_undegraded":"2024-04-16T21:11:42.293397+0000","last_fullsized":"2024-04-16T21:11:42.293397+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1571,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651424+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:11:43.651424+0000","last_peered":"2024-04-16T21:11:43.651424+0000","last_clean":"2024-04-16T21:11:43.651424+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:11:43.651424+0000","last_undegraded":"2024-04-16T21:11:43.651424+0000","last_fullsized":"2024-04-16T21:11:43.651424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.869006+0000","last_change":"2024-04-16T21:12:06.869029+0000","last_active":"2024-04-16T21:12:06.869006+0000","last_peered":"2024-04-16T21:12:06.869006+0000","last_clean":"2024-04-16T21:12:06.869006+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:12:06.869006+0000","last_undegraded":"2024-04-16T21:12:06.869006+0000","last_fullsized":"2024-04-16T21:12:06.869006+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.868938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:10.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'8","reported_seq":106,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.650032+0000","last_change":"2024-04-16T21:12:10.650063+0000","last_active":"2024-04-16T21:12:10.650032+0000","last_peered":"2024-04-16T21:12:10.650032+0000","last_clean":"2024-04-16T21:12:10.650032+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:12:10.650032+0000","last_undegraded":"2024-04-16T21:12:10.650032+0000","last_fullsized":"2024-04-16T21:12:10.650032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.649972+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.650174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.662915+0000","last_change":"2024-04-16T21:11:53.662915+0000","last_active":"2024-04-16T21:11:53.662915+0000","last_peered":"2024-04-16T21:11:53.662915+0000","last_clean":"2024-04-16T21:11:53.662915+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:11:53.662915+0000","last_undegraded":"2024-04-16T21:11:53.662915+0000","last_fullsized":"2024-04-16T21:11:53.662915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.662831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:53.840637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'317","reported_seq":613,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.662773+0000","last_change":"2024-04-16T21:11:52.662773+0000","last_active":"2024-04-16T21:11:52.662773+0000","last_peered":"2024-04-16T21:11:52.662773+0000","last_clean":"2024-04-16T21:11:52.662773+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:11:52.662773+0000","last_undegraded":"2024-04-16T21:11:52.662773+0000","last_fullsized":"2024-04-16T21:11:52.662773+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'317","last_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_deep_scrub":"52'317","last_deep_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.662688+0000","objects_scrubbed":25,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:03:07.759388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3966,"num_omap_keys":18,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'20","reported_seq":253,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.170966+0000","last_change":"2024-04-16T21:11:53.171045+0000","last_active":"2024-04-16T21:11:53.170966+0000","last_peered":"2024-04-16T21:11:53.170966+0000","last_clean":"2024-04-16T21:11:53.170966+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:11:53.170966+0000","last_undegraded":"2024-04-16T21:11:53.170966+0000","last_fullsized":"2024-04-16T21:11:53.170966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.170891+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:56.498676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.474986+0000","last_change":"2024-04-16T21:12:09.475030+0000","last_active":"2024-04-16T21:12:09.474986+0000","last_peered":"2024-04-16T21:12:09.474986+0000","last_clean":"2024-04-16T21:12:09.474986+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:12:09.474986+0000","last_undegraded":"2024-04-16T21:12:09.474986+0000","last_fullsized":"2024-04-16T21:12:09.474986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.474907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:12.558579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.703966+0000","last_change":"2024-04-16T21:12:40.703987+0000","last_active":"2024-04-16T21:12:40.703966+0000","last_peered":"2024-04-16T21:12:40.703966+0000","last_clean":"2024-04-16T21:12:40.703966+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:12:40.703966+0000","last_undegraded":"2024-04-16T21:12:40.703966+0000","last_fullsized":"2024-04-16T21:12:40.703966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.703892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:52.306563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.890729+0000","last_change":"2024-04-16T21:12:07.890754+0000","last_active":"2024-04-16T21:12:07.890729+0000","last_peered":"2024-04-16T21:12:07.890729+0000","last_clean":"2024-04-16T21:12:07.890729+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:12:07.890729+0000","last_undegraded":"2024-04-16T21:12:07.890729+0000","last_fullsized":"2024-04-16T21:12:07.890729+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.890665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:11.218450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"32'3","reported_seq":745,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743215+0000","last_change":"2024-04-16T19:24:45.410600+0000","last_active":"2024-04-16T21:11:42.743215+0000","last_peered":"2024-04-16T21:11:42.743215+0000","last_clean":"2024-04-16T21:11:42.743215+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:11:42.743215+0000","last_undegraded":"2024-04-16T21:11:42.743215+0000","last_fullsized":"2024-04-16T21:11:42.743215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.782050+0000","last_change":"2024-04-16T21:12:40.782083+0000","last_active":"2024-04-16T21:12:40.782050+0000","last_peered":"2024-04-16T21:12:40.782050+0000","last_clean":"2024-04-16T21:12:40.782050+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:12:40.782050+0000","last_undegraded":"2024-04-16T21:12:40.782050+0000","last_fullsized":"2024-04-16T21:12:40.782050+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.781989+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:34.970663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.902196+0000","last_change":"2024-04-16T21:12:21.902219+0000","last_active":"2024-04-16T21:12:21.902196+0000","last_peered":"2024-04-16T21:12:21.902196+0000","last_clean":"2024-04-16T21:12:21.902196+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:12:21.902196+0000","last_undegraded":"2024-04-16T21:12:21.902196+0000","last_fullsized":"2024-04-16T21:12:21.902196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.902128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:26.941072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107591+0000","last_change":"2024-04-16T19:47:33.984316+0000","last_active":"2024-04-16T21:11:44.107591+0000","last_peered":"2024-04-16T21:11:44.107591+0000","last_clean":"2024-04-16T21:11:44.107591+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:11:44.107591+0000","last_undegraded":"2024-04-16T21:11:44.107591+0000","last_fullsized":"2024-04-16T21:11:44.107591+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'159","reported_seq":1763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.889853+0000","last_change":"2024-04-16T21:12:18.889893+0000","last_active":"2024-04-16T21:12:18.889853+0000","last_peered":"2024-04-16T21:12:18.889853+0000","last_clean":"2024-04-16T21:12:18.889853+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:12:18.889853+0000","last_undegraded":"2024-04-16T21:12:18.889853+0000","last_fullsized":"2024-04-16T21:12:18.889853+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.889790+0000","objects_scrubbed":41,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:35.275723+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12681,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.752918+0000","last_change":"2024-04-16T21:12:01.752964+0000","last_active":"2024-04-16T21:12:01.752918+0000","last_peered":"2024-04-16T21:12:01.752918+0000","last_clean":"2024-04-16T21:12:01.752918+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:12:01.752918+0000","last_undegraded":"2024-04-16T21:12:01.752918+0000","last_fullsized":"2024-04-16T21:12:01.752918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.752809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.500440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.625386+0000","last_change":"2024-04-16T21:12:04.625407+0000","last_active":"2024-04-16T21:12:04.625386+0000","last_peered":"2024-04-16T21:12:04.625386+0000","last_clean":"2024-04-16T21:12:04.625386+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:12:04.625386+0000","last_undegraded":"2024-04-16T21:12:04.625386+0000","last_fullsized":"2024-04-16T21:12:04.625386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'2","last_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_deep_scrub":"45'2","last_deep_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.625306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.449242+0000","last_change":"2024-04-16T21:12:04.449242+0000","last_active":"2024-04-16T21:12:04.449242+0000","last_peered":"2024-04-16T21:12:04.449242+0000","last_clean":"2024-04-16T21:12:04.449242+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:12:04.449242+0000","last_undegraded":"2024-04-16T21:12:04.449242+0000","last_fullsized":"2024-04-16T21:12:04.449242+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.449160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:30.569236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1163","reported_seq":2931,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.718512+0000","last_change":"2024-04-16T21:11:46.718512+0000","last_active":"2024-04-16T21:11:46.718512+0000","last_peered":"2024-04-16T21:11:46.718512+0000","last_clean":"2024-04-16T21:11:46.718512+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:11:46.718512+0000","last_undegraded":"2024-04-16T21:11:46.718512+0000","last_fullsized":"2024-04-16T21:11:46.718512+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1163","last_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_deep_scrub":"52'1163","last_deep_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.718377+0000","objects_scrubbed":23,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:06.764370+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'16","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.594571+0000","last_change":"2024-04-16T21:11:51.594618+0000","last_active":"2024-04-16T21:11:51.594571+0000","last_peered":"2024-04-16T21:11:51.594571+0000","last_clean":"2024-04-16T21:11:51.594571+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:11:51.594571+0000","last_undegraded":"2024-04-16T21:11:51.594571+0000","last_fullsized":"2024-04-16T21:11:51.594571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.594510+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:52.095893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.782064+0000","last_change":"2024-04-16T21:12:01.782087+0000","last_active":"2024-04-16T21:12:01.782064+0000","last_peered":"2024-04-16T21:12:01.782064+0000","last_clean":"2024-04-16T21:12:01.782064+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:12:01.782064+0000","last_undegraded":"2024-04-16T21:12:01.782064+0000","last_fullsized":"2024-04-16T21:12:01.782064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.781996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:45.434631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.052606+0000","last_change":"2024-04-16T21:12:39.052630+0000","last_active":"2024-04-16T21:12:39.052606+0000","last_peered":"2024-04-16T21:12:39.052606+0000","last_clean":"2024-04-16T21:12:39.052606+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:12:39.052606+0000","last_undegraded":"2024-04-16T21:12:39.052606+0000","last_fullsized":"2024-04-16T21:12:39.052606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.052538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:27.090193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386872+0000","last_change":"2024-04-16T19:24:33.389357+0000","last_active":"2024-04-16T21:10:54.386872+0000","last_peered":"2024-04-16T21:10:54.386872+0000","last_clean":"2024-04-16T21:10:54.386872+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:10:54.386872+0000","last_undegraded":"2024-04-16T21:10:54.386872+0000","last_fullsized":"2024-04-16T21:10:54.386872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'14","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743211+0000","last_change":"2024-04-16T19:24:45.413322+0000","last_active":"2024-04-16T21:11:42.743211+0000","last_peered":"2024-04-16T21:11:42.743211+0000","last_clean":"2024-04-16T21:11:42.743211+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:11:42.743211+0000","last_undegraded":"2024-04-16T21:11:42.743211+0000","last_fullsized":"2024-04-16T21:11:42.743211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'49","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.341873+0000","last_change":"2024-04-16T21:12:40.341940+0000","last_active":"2024-04-16T21:12:40.341873+0000","last_peered":"2024-04-16T21:12:40.341873+0000","last_clean":"2024-04-16T21:12:40.341873+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:12:40.341873+0000","last_undegraded":"2024-04-16T21:12:40.341873+0000","last_fullsized":"2024-04-16T21:12:40.341873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'49","last_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_deep_scrub":"52'49","last_deep_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.341796+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:55.596579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.423193+0000","last_change":"2024-04-16T21:12:10.423217+0000","last_active":"2024-04-16T21:12:10.423193+0000","last_peered":"2024-04-16T21:12:10.423193+0000","last_clean":"2024-04-16T21:12:10.423193+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:12:10.423193+0000","last_undegraded":"2024-04-16T21:12:10.423193+0000","last_fullsized":"2024-04-16T21:12:10.423193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.423128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:15.462072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561679+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:11:44.561679+0000","last_peered":"2024-04-16T21:11:44.561679+0000","last_clean":"2024-04-16T21:11:44.561679+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:11:44.561679+0000","last_undegraded":"2024-04-16T21:11:44.561679+0000","last_fullsized":"2024-04-16T21:11:44.561679+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'181","reported_seq":1765,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.009379+0000","last_change":"2024-04-16T21:12:35.009430+0000","last_active":"2024-04-16T21:12:35.009379+0000","last_peered":"2024-04-16T21:12:35.009379+0000","last_clean":"2024-04-16T21:12:35.009379+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:12:35.009379+0000","last_undegraded":"2024-04-16T21:12:35.009379+0000","last_fullsized":"2024-04-16T21:12:35.009379+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'181","last_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_deep_scrub":"52'181","last_deep_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.009303+0000","objects_scrubbed":51,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:46.611974+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8952,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.920212+0000","last_change":"2024-04-16T21:12:18.920252+0000","last_active":"2024-04-16T21:12:18.920212+0000","last_peered":"2024-04-16T21:12:18.920212+0000","last_clean":"2024-04-16T21:12:18.920212+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:12:18.920212+0000","last_undegraded":"2024-04-16T21:12:18.920212+0000","last_fullsized":"2024-04-16T21:12:18.920212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.920121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:25:34.146310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'26","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.477899+0000","last_change":"2024-04-16T21:12:19.477941+0000","last_active":"2024-04-16T21:12:19.477899+0000","last_peered":"2024-04-16T21:12:19.477899+0000","last_clean":"2024-04-16T21:12:19.477899+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:12:19.477899+0000","last_undegraded":"2024-04-16T21:12:19.477899+0000","last_fullsized":"2024-04-16T21:12:19.477899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.477827+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:13.666501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.766206+0000","last_change":"2024-04-16T21:12:09.766229+0000","last_active":"2024-04-16T21:12:09.766206+0000","last_peered":"2024-04-16T21:12:09.766206+0000","last_clean":"2024-04-16T21:12:09.766206+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:12:09.766206+0000","last_undegraded":"2024-04-16T21:12:09.766206+0000","last_fullsized":"2024-04-16T21:12:09.766206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.766140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:51.548954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'291","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.385570+0000","last_change":"2024-04-16T21:11:52.385621+0000","last_active":"2024-04-16T21:11:52.385570+0000","last_peered":"2024-04-16T21:11:52.385570+0000","last_clean":"2024-04-16T21:11:52.385570+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:11:52.385570+0000","last_undegraded":"2024-04-16T21:11:52.385570+0000","last_fullsized":"2024-04-16T21:11:52.385570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'291","last_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_deep_scrub":"52'291","last_deep_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.385512+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:13.385713+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'16","reported_seq":263,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.131942+0000","last_change":"2024-04-16T21:11:52.131942+0000","last_active":"2024-04-16T21:11:52.131942+0000","last_peered":"2024-04-16T21:11:52.131942+0000","last_clean":"2024-04-16T21:11:52.131942+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:11:52.131942+0000","last_undegraded":"2024-04-16T21:11:52.131942+0000","last_fullsized":"2024-04-16T21:11:52.131942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.131829+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:55.304717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.441339+0000","last_change":"2024-04-16T21:12:08.441339+0000","last_active":"2024-04-16T21:12:08.441339+0000","last_peered":"2024-04-16T21:12:08.441339+0000","last_clean":"2024-04-16T21:12:08.441339+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:12:08.441339+0000","last_undegraded":"2024-04-16T21:12:08.441339+0000","last_fullsized":"2024-04-16T21:12:08.441339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.441251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:03.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.805365+0000","last_change":"2024-04-16T21:12:35.805388+0000","last_active":"2024-04-16T21:12:35.805365+0000","last_peered":"2024-04-16T21:12:35.805365+0000","last_clean":"2024-04-16T21:12:35.805365+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:12:35.805365+0000","last_undegraded":"2024-04-16T21:12:35.805365+0000","last_fullsized":"2024-04-16T21:12:35.805365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.805297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:04.446592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386323+0000","last_change":"2024-04-16T19:24:33.386530+0000","last_active":"2024-04-16T21:10:54.386323+0000","last_peered":"2024-04-16T21:10:54.386323+0000","last_clean":"2024-04-16T21:10:54.386323+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:10:54.386323+0000","last_undegraded":"2024-04-16T21:10:54.386323+0000","last_fullsized":"2024-04-16T21:10:54.386323+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'14","reported_seq":623,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.736145+0000","last_change":"2024-04-16T21:12:39.736175+0000","last_active":"2024-04-16T21:12:39.736145+0000","last_peered":"2024-04-16T21:12:39.736145+0000","last_clean":"2024-04-16T21:12:39.736145+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:12:39.736145+0000","last_undegraded":"2024-04-16T21:12:39.736145+0000","last_fullsized":"2024-04-16T21:12:39.736145+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.736087+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:33.924762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.893687+0000","last_change":"2024-04-16T21:12:23.893724+0000","last_active":"2024-04-16T21:12:23.893687+0000","last_peered":"2024-04-16T21:12:23.893687+0000","last_clean":"2024-04-16T21:12:23.893687+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:12:23.893687+0000","last_undegraded":"2024-04-16T21:12:23.893687+0000","last_fullsized":"2024-04-16T21:12:23.893687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.893616+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:17.500596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.781919+0000","last_change":"2024-04-16T21:12:31.781950+0000","last_active":"2024-04-16T21:12:31.781919+0000","last_peered":"2024-04-16T21:12:31.781919+0000","last_clean":"2024-04-16T21:12:31.781919+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:12:31.781919+0000","last_undegraded":"2024-04-16T21:12:31.781919+0000","last_fullsized":"2024-04-16T21:12:31.781919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.781859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:19.914350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651552+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:11:43.651552+0000","last_peered":"2024-04-16T21:11:43.651552+0000","last_clean":"2024-04-16T21:11:43.651552+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:11:43.651552+0000","last_undegraded":"2024-04-16T21:11:43.651552+0000","last_fullsized":"2024-04-16T21:11:43.651552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'124","reported_seq":1188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.361236+0000","last_change":"2024-04-16T21:12:26.361300+0000","last_active":"2024-04-16T21:12:26.361236+0000","last_peered":"2024-04-16T21:12:26.361236+0000","last_clean":"2024-04-16T21:12:26.361236+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:12:26.361236+0000","last_undegraded":"2024-04-16T21:12:26.361236+0000","last_fullsized":"2024-04-16T21:12:26.361236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'124","last_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_deep_scrub":"52'124","last_deep_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.361168+0000","objects_scrubbed":36,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:14.967461+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7264,"num_omap_keys":29,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.867629+0000","last_change":"2024-04-16T21:12:04.867629+0000","last_active":"2024-04-16T21:12:04.867629+0000","last_peered":"2024-04-16T21:12:04.867629+0000","last_clean":"2024-04-16T21:12:04.867629+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:12:04.867629+0000","last_undegraded":"2024-04-16T21:12:04.867629+0000","last_fullsized":"2024-04-16T21:12:04.867629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.867541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.770932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'31","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.873726+0000","last_change":"2024-04-16T21:12:11.873787+0000","last_active":"2024-04-16T21:12:11.873726+0000","last_peered":"2024-04-16T21:12:11.873726+0000","last_clean":"2024-04-16T21:12:11.873726+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:12:11.873726+0000","last_undegraded":"2024-04-16T21:12:11.873726+0000","last_fullsized":"2024-04-16T21:12:11.873726+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'31","last_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_deep_scrub":"52'31","last_deep_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.873657+0000","objects_scrubbed":9,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:04.053968+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'706","reported_seq":2151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.887851+0000","last_change":"2024-04-16T21:12:02.887901+0000","last_active":"2024-04-16T21:12:02.887851+0000","last_peered":"2024-04-16T21:12:02.887851+0000","last_clean":"2024-04-16T21:12:02.887851+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:12:02.887851+0000","last_undegraded":"2024-04-16T21:12:02.887851+0000","last_fullsized":"2024-04-16T21:12:02.887851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.887759+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:37.031244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1038","reported_seq":1547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.224293+0000","last_change":"2024-04-16T21:11:54.224379+0000","last_active":"2024-04-16T21:11:54.224293+0000","last_peered":"2024-04-16T21:11:54.224293+0000","last_clean":"2024-04-16T21:11:54.224293+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:11:54.224293+0000","last_undegraded":"2024-04-16T21:11:54.224293+0000","last_fullsized":"2024-04-16T21:11:54.224293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1038","last_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_deep_scrub":"52'1038","last_deep_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.224224+0000","objects_scrubbed":21,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:22.282373+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2194,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'23","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.136819+0000","last_change":"2024-04-16T21:11:43.136819+0000","last_active":"2024-04-16T21:11:43.136819+0000","last_peered":"2024-04-16T21:11:43.136819+0000","last_clean":"2024-04-16T21:11:43.136819+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:11:43.136819+0000","last_undegraded":"2024-04-16T21:11:43.136819+0000","last_fullsized":"2024-04-16T21:11:43.136819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'23","last_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_deep_scrub":"52'23","last_deep_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.136721+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:06.328117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.766632+0000","last_change":"2024-04-16T21:12:08.766632+0000","last_active":"2024-04-16T21:12:08.766632+0000","last_peered":"2024-04-16T21:12:08.766632+0000","last_clean":"2024-04-16T21:12:08.766632+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:12:08.766632+0000","last_undegraded":"2024-04-16T21:12:08.766632+0000","last_fullsized":"2024-04-16T21:12:08.766632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.766544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:45.782631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743080+0000","last_change":"2024-04-16T19:24:35.394757+0000","last_active":"2024-04-16T21:11:42.743080+0000","last_peered":"2024-04-16T21:11:42.743080+0000","last_clean":"2024-04-16T21:11:42.743080+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:11:42.743080+0000","last_undegraded":"2024-04-16T21:11:42.743080+0000","last_fullsized":"2024-04-16T21:11:42.743080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391438+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:10:54.391438+0000","last_peered":"2024-04-16T21:10:54.391438+0000","last_clean":"2024-04-16T21:10:54.391438+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:10:54.391438+0000","last_undegraded":"2024-04-16T21:10:54.391438+0000","last_fullsized":"2024-04-16T21:10:54.391438+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.186769+0000","last_change":"2024-04-16T21:12:41.186808+0000","last_active":"2024-04-16T21:12:41.186769+0000","last_peered":"2024-04-16T21:12:41.186769+0000","last_clean":"2024-04-16T21:12:41.186769+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:12:41.186769+0000","last_undegraded":"2024-04-16T21:12:41.186769+0000","last_fullsized":"2024-04-16T21:12:41.186769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.186689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:10.176347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'78","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.901279+0000","last_change":"2024-04-16T21:12:35.901311+0000","last_active":"2024-04-16T21:12:35.901279+0000","last_peered":"2024-04-16T21:12:35.901279+0000","last_clean":"2024-04-16T21:12:35.901279+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:12:35.901279+0000","last_undegraded":"2024-04-16T21:12:35.901279+0000","last_fullsized":"2024-04-16T21:12:35.901279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.901215+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:22.199019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.365174+0000","last_change":"2024-04-16T21:12:25.365219+0000","last_active":"2024-04-16T21:12:25.365174+0000","last_peered":"2024-04-16T21:12:25.365174+0000","last_clean":"2024-04-16T21:12:25.365174+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:12:25.365174+0000","last_undegraded":"2024-04-16T21:12:25.365174+0000","last_fullsized":"2024-04-16T21:12:25.365174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.365094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:11.662898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651543+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:11:43.651543+0000","last_peered":"2024-04-16T21:11:43.651543+0000","last_clean":"2024-04-16T21:11:43.651543+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:11:43.651543+0000","last_undegraded":"2024-04-16T21:11:43.651543+0000","last_fullsized":"2024-04-16T21:11:43.651543+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'207","reported_seq":2084,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.250819+0000","last_change":"2024-04-16T21:12:44.250861+0000","last_active":"2024-04-16T21:12:44.250819+0000","last_peered":"2024-04-16T21:12:44.250819+0000","last_clean":"2024-04-16T21:12:44.250819+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:12:44.250819+0000","last_undegraded":"2024-04-16T21:12:44.250819+0000","last_fullsized":"2024-04-16T21:12:44.250819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'207","last_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_deep_scrub":"52'207","last_deep_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.250741+0000","objects_scrubbed":45,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:38.832019+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12815,"num_omap_keys":53,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.457685+0000","last_change":"2024-04-16T21:12:13.457709+0000","last_active":"2024-04-16T21:12:13.457685+0000","last_peered":"2024-04-16T21:12:13.457685+0000","last_clean":"2024-04-16T21:12:13.457685+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:12:13.457685+0000","last_undegraded":"2024-04-16T21:12:13.457685+0000","last_fullsized":"2024-04-16T21:12:13.457685+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.457615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:51.890971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'7","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.648645+0000","last_change":"2024-04-16T21:12:07.648674+0000","last_active":"2024-04-16T21:12:07.648645+0000","last_peered":"2024-04-16T21:12:07.648645+0000","last_clean":"2024-04-16T21:12:07.648645+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:12:07.648645+0000","last_undegraded":"2024-04-16T21:12:07.648645+0000","last_fullsized":"2024-04-16T21:12:07.648645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.648592+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.343437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.389563+0000","last_change":"2024-04-16T21:12:06.389563+0000","last_active":"2024-04-16T21:12:06.389563+0000","last_peered":"2024-04-16T21:12:06.389563+0000","last_clean":"2024-04-16T21:12:06.389563+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:12:06.389563+0000","last_undegraded":"2024-04-16T21:12:06.389563+0000","last_fullsized":"2024-04-16T21:12:06.389563+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.389467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:55.353016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'329","reported_seq":649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.716329+0000","last_change":"2024-04-16T21:11:45.716329+0000","last_active":"2024-04-16T21:11:45.716329+0000","last_peered":"2024-04-16T21:11:45.716329+0000","last_clean":"2024-04-16T21:11:45.716329+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:11:45.716329+0000","last_undegraded":"2024-04-16T21:11:45.716329+0000","last_fullsized":"2024-04-16T21:11:45.716329+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'329","last_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_deep_scrub":"52'329","last_deep_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.716182+0000","objects_scrubbed":24,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:24.600927+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'11","reported_seq":161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.675426+0000","last_change":"2024-04-16T21:11:50.675426+0000","last_active":"2024-04-16T21:11:50.675426+0000","last_peered":"2024-04-16T21:11:50.675426+0000","last_clean":"2024-04-16T21:11:50.675426+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:11:50.675426+0000","last_undegraded":"2024-04-16T21:11:50.675426+0000","last_fullsized":"2024-04-16T21:11:50.675426+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_deep_scrub":"52'11","last_deep_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.675302+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:31.269067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.387000+0000","last_change":"2024-04-16T19:24:35.397251+0000","last_active":"2024-04-16T21:10:54.387000+0000","last_peered":"2024-04-16T21:10:54.387000+0000","last_clean":"2024-04-16T21:10:54.387000+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:10:54.387000+0000","last_undegraded":"2024-04-16T21:10:54.387000+0000","last_fullsized":"2024-04-16T21:10:54.387000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.525357+0000","last_change":"2024-04-16T19:24:33.392065+0000","last_active":"2024-04-16T21:10:54.525357+0000","last_peered":"2024-04-16T21:10:54.525357+0000","last_clean":"2024-04-16T21:10:54.525357+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:10:54.525357+0000","last_undegraded":"2024-04-16T21:10:54.525357+0000","last_fullsized":"2024-04-16T21:10:54.525357+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.345176+0000","last_change":"2024-04-16T21:12:41.345218+0000","last_active":"2024-04-16T21:12:41.345176+0000","last_peered":"2024-04-16T21:12:41.345176+0000","last_clean":"2024-04-16T21:12:41.345176+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:12:41.345176+0000","last_undegraded":"2024-04-16T21:12:41.345176+0000","last_fullsized":"2024-04-16T21:12:41.345176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.345086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:35.926364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'87","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.024063+0000","last_change":"2024-04-16T21:12:30.024100+0000","last_active":"2024-04-16T21:12:30.024063+0000","last_peered":"2024-04-16T21:12:30.024063+0000","last_clean":"2024-04-16T21:12:30.024063+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:12:30.024063+0000","last_undegraded":"2024-04-16T21:12:30.024063+0000","last_fullsized":"2024-04-16T21:12:30.024063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.024002+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:39.585670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.356039+0000","last_change":"2024-04-16T21:12:27.356060+0000","last_active":"2024-04-16T21:12:27.356039+0000","last_peered":"2024-04-16T21:12:27.356039+0000","last_clean":"2024-04-16T21:12:27.356039+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:12:27.356039+0000","last_undegraded":"2024-04-16T21:12:27.356039+0000","last_fullsized":"2024-04-16T21:12:27.356039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.355975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:15.393630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.732045+0000","last_change":"2024-04-16T21:11:59.732045+0000","last_active":"2024-04-16T21:11:59.732045+0000","last_peered":"2024-04-16T21:11:59.732045+0000","last_clean":"2024-04-16T21:11:59.732045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:11:59.732045+0000","last_undegraded":"2024-04-16T21:11:59.732045+0000","last_fullsized":"2024-04-16T21:11:59.732045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.731957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:37.824609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561671+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:11:44.561671+0000","last_peered":"2024-04-16T21:11:44.561671+0000","last_clean":"2024-04-16T21:11:44.561671+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:11:44.561671+0000","last_undegraded":"2024-04-16T21:11:44.561671+0000","last_fullsized":"2024-04-16T21:11:44.561671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'170","reported_seq":1882,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.686097+0000","last_change":"2024-04-16T21:12:29.686173+0000","last_active":"2024-04-16T21:12:29.686097+0000","last_peered":"2024-04-16T21:12:29.686097+0000","last_clean":"2024-04-16T21:12:29.686097+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:12:29.686097+0000","last_undegraded":"2024-04-16T21:12:29.686097+0000","last_fullsized":"2024-04-16T21:12:29.686097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.686024+0000","objects_scrubbed":46,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:24.943848+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12193,"num_omap_keys":48,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.906298+0000","last_change":"2024-04-16T21:12:17.906319+0000","last_active":"2024-04-16T21:12:17.906298+0000","last_peered":"2024-04-16T21:12:17.906298+0000","last_clean":"2024-04-16T21:12:17.906298+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:12:17.906298+0000","last_undegraded":"2024-04-16T21:12:17.906298+0000","last_fullsized":"2024-04-16T21:12:17.906298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.906238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:22.945182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'18","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.785345+0000","last_change":"2024-04-16T21:12:24.785376+0000","last_active":"2024-04-16T21:12:24.785345+0000","last_peered":"2024-04-16T21:12:24.785345+0000","last_clean":"2024-04-16T21:12:24.785345+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:12:24.785345+0000","last_undegraded":"2024-04-16T21:12:24.785345+0000","last_fullsized":"2024-04-16T21:12:24.785345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.785282+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:53.426577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.865507+0000","last_change":"2024-04-16T21:12:05.865507+0000","last_active":"2024-04-16T21:12:05.865507+0000","last_peered":"2024-04-16T21:12:05.865507+0000","last_clean":"2024-04-16T21:12:05.865507+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:12:05.865507+0000","last_undegraded":"2024-04-16T21:12:05.865507+0000","last_fullsized":"2024-04-16T21:12:05.865507+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.865437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.716343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'138","reported_seq":299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.719459+0000","last_change":"2024-04-16T21:11:48.719459+0000","last_active":"2024-04-16T21:11:48.719459+0000","last_peered":"2024-04-16T21:11:48.719459+0000","last_clean":"2024-04-16T21:11:48.719459+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:11:48.719459+0000","last_undegraded":"2024-04-16T21:11:48.719459+0000","last_fullsized":"2024-04-16T21:11:48.719459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.719337+0000","objects_scrubbed":17,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.768315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4229,"num_omap_keys":20,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'8","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.405983+0000","last_change":"2024-04-16T21:11:49.405983+0000","last_active":"2024-04-16T21:11:49.405983+0000","last_peered":"2024-04-16T21:11:49.405983+0000","last_clean":"2024-04-16T21:11:49.405983+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:11:49.405983+0000","last_undegraded":"2024-04-16T21:11:49.405983+0000","last_fullsized":"2024-04-16T21:11:49.405983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.405858+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:30.100703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.341348+0000","last_change":"2024-04-16T21:12:44.341370+0000","last_active":"2024-04-16T21:12:44.341348+0000","last_peered":"2024-04-16T21:12:44.341348+0000","last_clean":"2024-04-16T21:12:44.341348+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:12:44.341348+0000","last_undegraded":"2024-04-16T21:12:44.341348+0000","last_fullsized":"2024-04-16T21:12:44.341348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.341283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:33.782350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.388626+0000","last_change":"2024-04-16T19:24:33.386066+0000","last_active":"2024-04-16T21:11:14.388626+0000","last_peered":"2024-04-16T21:11:14.388626+0000","last_clean":"2024-04-16T21:11:14.388626+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:11:14.388626+0000","last_undegraded":"2024-04-16T21:11:14.388626+0000","last_fullsized":"2024-04-16T21:11:14.388626+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"52'4","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.265214+0000","last_change":"2024-04-16T21:12:39.265243+0000","last_active":"2024-04-16T21:12:39.265214+0000","last_peered":"2024-04-16T21:12:39.265214+0000","last_clean":"2024-04-16T21:12:39.265214+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:12:39.265214+0000","last_undegraded":"2024-04-16T21:12:39.265214+0000","last_fullsized":"2024-04-16T21:12:39.265214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.265147+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:53.843686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'147","reported_seq":338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.377505+0000","last_change":"2024-04-16T21:12:31.377537+0000","last_active":"2024-04-16T21:12:31.377505+0000","last_peered":"2024-04-16T21:12:31.377505+0000","last_clean":"2024-04-16T21:12:31.377505+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:12:31.377505+0000","last_undegraded":"2024-04-16T21:12:31.377505+0000","last_fullsized":"2024-04-16T21:12:31.377505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.377449+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:37.615098+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.930468+0000","last_change":"2024-04-16T21:12:23.930491+0000","last_active":"2024-04-16T21:12:23.930468+0000","last_peered":"2024-04-16T21:12:23.930468+0000","last_clean":"2024-04-16T21:12:23.930468+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:12:23.930468+0000","last_undegraded":"2024-04-16T21:12:23.930468+0000","last_fullsized":"2024-04-16T21:12:23.930468+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.930404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:27.014075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.517927+0000","last_change":"2024-04-16T21:12:01.517927+0000","last_active":"2024-04-16T21:12:01.517927+0000","last_peered":"2024-04-16T21:12:01.517927+0000","last_clean":"2024-04-16T21:12:01.517927+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:12:01.517927+0000","last_undegraded":"2024-04-16T21:12:01.517927+0000","last_fullsized":"2024-04-16T21:12:01.517927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:38.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293399+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:11:42.293399+0000","last_peered":"2024-04-16T21:11:42.293399+0000","last_clean":"2024-04-16T21:11:42.293399+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:11:42.293399+0000","last_undegraded":"2024-04-16T21:11:42.293399+0000","last_fullsized":"2024-04-16T21:11:42.293399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'150","reported_seq":1340,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.054107+0000","last_change":"2024-04-16T21:12:31.054152+0000","last_active":"2024-04-16T21:12:31.054107+0000","last_peered":"2024-04-16T21:12:31.054107+0000","last_clean":"2024-04-16T21:12:31.054107+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:12:31.054107+0000","last_undegraded":"2024-04-16T21:12:31.054107+0000","last_fullsized":"2024-04-16T21:12:31.054107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'150","last_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_deep_scrub":"52'150","last_deep_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.054045+0000","objects_scrubbed":46,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:22.727735+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7793,"num_omap_keys":32,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.443129+0000","last_change":"2024-04-16T21:12:22.443154+0000","last_active":"2024-04-16T21:12:22.443129+0000","last_peered":"2024-04-16T21:12:22.443129+0000","last_clean":"2024-04-16T21:12:22.443129+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:12:22.443129+0000","last_undegraded":"2024-04-16T21:12:22.443129+0000","last_fullsized":"2024-04-16T21:12:22.443129+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.443057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:22.418023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'15","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.828587+0000","last_change":"2024-04-16T21:12:23.828622+0000","last_active":"2024-04-16T21:12:23.828587+0000","last_peered":"2024-04-16T21:12:23.828587+0000","last_clean":"2024-04-16T21:12:23.828587+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:12:23.828587+0000","last_undegraded":"2024-04-16T21:12:23.828587+0000","last_fullsized":"2024-04-16T21:12:23.828587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.828490+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:02.261846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.402484+0000","last_change":"2024-04-16T21:12:06.402484+0000","last_active":"2024-04-16T21:12:06.402484+0000","last_peered":"2024-04-16T21:12:06.402484+0000","last_clean":"2024-04-16T21:12:06.402484+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:12:06.402484+0000","last_undegraded":"2024-04-16T21:12:06.402484+0000","last_fullsized":"2024-04-16T21:12:06.402484+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.402344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:52.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'341","reported_seq":880,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.634146+0000","last_change":"2024-04-16T21:11:48.634146+0000","last_active":"2024-04-16T21:11:48.634146+0000","last_peered":"2024-04-16T21:11:48.634146+0000","last_clean":"2024-04-16T21:11:48.634146+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:11:48.634146+0000","last_undegraded":"2024-04-16T21:11:48.634146+0000","last_fullsized":"2024-04-16T21:11:48.634146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'341","last_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_deep_scrub":"52'341","last_deep_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.633983+0000","objects_scrubbed":20,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:05:52.976164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'19","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.631145+0000","last_change":"2024-04-16T21:11:55.631145+0000","last_active":"2024-04-16T21:11:55.631145+0000","last_peered":"2024-04-16T21:11:55.631145+0000","last_clean":"2024-04-16T21:11:55.631145+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:11:55.631145+0000","last_undegraded":"2024-04-16T21:11:55.631145+0000","last_fullsized":"2024-04-16T21:11:55.631145+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.631042+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.051632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292189+0000","last_change":"2024-04-16T19:24:35.395259+0000","last_active":"2024-04-16T21:11:42.292189+0000","last_peered":"2024-04-16T21:11:42.292189+0000","last_clean":"2024-04-16T21:11:42.292189+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:11:42.292189+0000","last_undegraded":"2024-04-16T21:11:42.292189+0000","last_fullsized":"2024-04-16T21:11:42.292189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106228+0000","last_change":"2024-04-16T19:24:33.382295+0000","last_active":"2024-04-16T21:11:44.106228+0000","last_peered":"2024-04-16T21:11:44.106228+0000","last_clean":"2024-04-16T21:11:44.106228+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:11:44.106228+0000","last_undegraded":"2024-04-16T21:11:44.106228+0000","last_fullsized":"2024-04-16T21:11:44.106228+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.723608+0000","last_change":"2024-04-16T21:12:41.723630+0000","last_active":"2024-04-16T21:12:41.723608+0000","last_peered":"2024-04-16T21:12:41.723608+0000","last_clean":"2024-04-16T21:12:41.723608+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:12:41.723608+0000","last_undegraded":"2024-04-16T21:12:41.723608+0000","last_fullsized":"2024-04-16T21:12:41.723608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.723545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:29.856036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'96","reported_seq":265,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.527257+0000","last_change":"2024-04-16T19:24:47.422350+0000","last_active":"2024-04-16T21:11:04.527257+0000","last_peered":"2024-04-16T21:11:04.527257+0000","last_clean":"2024-04-16T21:11:04.527257+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:11:04.527257+0000","last_undegraded":"2024-04-16T21:11:04.527257+0000","last_fullsized":"2024-04-16T21:11:04.527257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.928158+0000","last_change":"2024-04-16T21:12:25.928177+0000","last_active":"2024-04-16T21:12:25.928158+0000","last_peered":"2024-04-16T21:12:25.928158+0000","last_clean":"2024-04-16T21:12:25.928158+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:12:25.928158+0000","last_undegraded":"2024-04-16T21:12:25.928158+0000","last_fullsized":"2024-04-16T21:12:25.928158+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.928102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:41.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.482650+0000","last_change":"2024-04-16T21:12:03.482650+0000","last_active":"2024-04-16T21:12:03.482650+0000","last_peered":"2024-04-16T21:12:03.482650+0000","last_clean":"2024-04-16T21:12:03.482650+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:12:03.482650+0000","last_undegraded":"2024-04-16T21:12:03.482650+0000","last_fullsized":"2024-04-16T21:12:03.482650+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.482557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:28.918007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.107592+0000","last_change":"2024-04-16T19:47:33.983804+0000","last_active":"2024-04-16T21:11:44.107592+0000","last_peered":"2024-04-16T21:11:44.107592+0000","last_clean":"2024-04-16T21:11:44.107592+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:11:44.107592+0000","last_undegraded":"2024-04-16T21:11:44.107592+0000","last_fullsized":"2024-04-16T21:11:44.107592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'154","reported_seq":1354,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.959232+0000","last_change":"2024-04-16T21:12:26.959277+0000","last_active":"2024-04-16T21:12:26.959232+0000","last_peered":"2024-04-16T21:12:26.959232+0000","last_clean":"2024-04-16T21:12:26.959232+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:12:26.959232+0000","last_undegraded":"2024-04-16T21:12:26.959232+0000","last_fullsized":"2024-04-16T21:12:26.959232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'154","last_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_deep_scrub":"52'154","last_deep_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.959173+0000","objects_scrubbed":37,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:21.387909+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8816,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.680232+0000","last_change":"2024-04-16T21:12:17.680254+0000","last_active":"2024-04-16T21:12:17.680232+0000","last_peered":"2024-04-16T21:12:17.680232+0000","last_clean":"2024-04-16T21:12:17.680232+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:12:17.680232+0000","last_undegraded":"2024-04-16T21:12:17.680232+0000","last_fullsized":"2024-04-16T21:12:17.680232+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.680171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:09.860482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"52'8","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.838545+0000","last_change":"2024-04-16T21:12:22.838571+0000","last_active":"2024-04-16T21:12:22.838545+0000","last_peered":"2024-04-16T21:12:22.838545+0000","last_clean":"2024-04-16T21:12:22.838545+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:12:22.838545+0000","last_undegraded":"2024-04-16T21:12:22.838545+0000","last_fullsized":"2024-04-16T21:12:22.838545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.838490+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:17.267225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.653905+0000","last_change":"2024-04-16T21:12:05.653905+0000","last_active":"2024-04-16T21:12:05.653905+0000","last_peered":"2024-04-16T21:12:05.653905+0000","last_clean":"2024-04-16T21:12:05.653905+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:12:05.653905+0000","last_undegraded":"2024-04-16T21:12:05.653905+0000","last_fullsized":"2024-04-16T21:12:05.653905+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.653825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:12.673868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'264","reported_seq":1137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.696035+0000","last_change":"2024-04-16T21:11:49.696035+0000","last_active":"2024-04-16T21:11:49.696035+0000","last_peered":"2024-04-16T21:11:49.696035+0000","last_clean":"2024-04-16T21:11:49.696035+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:11:49.696035+0000","last_undegraded":"2024-04-16T21:11:49.696035+0000","last_fullsized":"2024-04-16T21:11:49.696035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.695875+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:11.637965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2924,"num_omap_keys":15,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'12","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.501597+0000","last_change":"2024-04-16T21:11:59.501597+0000","last_active":"2024-04-16T21:11:59.501597+0000","last_peered":"2024-04-16T21:11:59.501597+0000","last_clean":"2024-04-16T21:11:59.501597+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:11:59.501597+0000","last_undegraded":"2024-04-16T21:11:59.501597+0000","last_fullsized":"2024-04-16T21:11:59.501597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.501496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:51.681807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.856833+0000","last_change":"2024-04-16T21:12:29.856857+0000","last_active":"2024-04-16T21:12:29.856833+0000","last_peered":"2024-04-16T21:12:29.856833+0000","last_clean":"2024-04-16T21:12:29.856833+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:12:29.856833+0000","last_undegraded":"2024-04-16T21:12:29.856833+0000","last_fullsized":"2024-04-16T21:12:29.856833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.856754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:25.114578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291882+0000","last_change":"2024-04-16T19:24:33.387130+0000","last_active":"2024-04-16T21:11:42.291882+0000","last_peered":"2024-04-16T21:11:42.291882+0000","last_clean":"2024-04-16T21:11:42.291882+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:11:42.291882+0000","last_undegraded":"2024-04-16T21:11:42.291882+0000","last_fullsized":"2024-04-16T21:11:42.291882+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'3","reported_seq":632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.743303+0000","last_change":"2024-04-16T19:24:45.411363+0000","last_active":"2024-04-16T21:11:42.743303+0000","last_peered":"2024-04-16T21:11:42.743303+0000","last_clean":"2024-04-16T21:11:42.743303+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:11:42.743303+0000","last_undegraded":"2024-04-16T21:11:42.743303+0000","last_fullsized":"2024-04-16T21:11:42.743303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'115","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.359212+0000","last_change":"2024-04-16T21:12:34.359243+0000","last_active":"2024-04-16T21:12:34.359212+0000","last_peered":"2024-04-16T21:12:34.359212+0000","last_clean":"2024-04-16T21:12:34.359212+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:12:34.359212+0000","last_undegraded":"2024-04-16T21:12:34.359212+0000","last_fullsized":"2024-04-16T21:12:34.359212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.359152+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:57.921414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'13","reported_seq":1325,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.729138+0000","last_change":"2024-04-16T21:12:28.729186+0000","last_active":"2024-04-16T21:12:28.729138+0000","last_peered":"2024-04-16T21:12:28.729138+0000","last_clean":"2024-04-16T21:12:28.729138+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:12:28.729138+0000","last_undegraded":"2024-04-16T21:12:28.729138+0000","last_fullsized":"2024-04-16T21:12:28.729138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.729061+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:00.415763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.424596+0000","last_change":"2024-04-16T21:12:05.424596+0000","last_active":"2024-04-16T21:12:05.424596+0000","last_peered":"2024-04-16T21:12:05.424596+0000","last_clean":"2024-04-16T21:12:05.424596+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:12:05.424596+0000","last_undegraded":"2024-04-16T21:12:05.424596+0000","last_fullsized":"2024-04-16T21:12:05.424596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.424465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:59.031445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392508+0000","last_change":"2024-04-16T19:47:33.983688+0000","last_active":"2024-04-16T21:11:34.392508+0000","last_peered":"2024-04-16T21:11:34.392508+0000","last_clean":"2024-04-16T21:11:34.392508+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:11:34.392508+0000","last_undegraded":"2024-04-16T21:11:34.392508+0000","last_fullsized":"2024-04-16T21:11:34.392508+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'274","reported_seq":2375,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.635555+0000","last_change":"2024-04-16T21:12:25.635594+0000","last_active":"2024-04-16T21:12:25.635555+0000","last_peered":"2024-04-16T21:12:25.635555+0000","last_clean":"2024-04-16T21:12:25.635555+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:12:25.635555+0000","last_undegraded":"2024-04-16T21:12:25.635555+0000","last_fullsized":"2024-04-16T21:12:25.635555+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'274","last_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_deep_scrub":"52'274","last_deep_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.635495+0000","objects_scrubbed":60,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.021428+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17559,"num_omap_keys":71,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.466819+0000","last_change":"2024-04-16T21:12:18.466844+0000","last_active":"2024-04-16T21:12:18.466819+0000","last_peered":"2024-04-16T21:12:18.466819+0000","last_clean":"2024-04-16T21:12:18.466819+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:12:18.466819+0000","last_undegraded":"2024-04-16T21:12:18.466819+0000","last_fullsized":"2024-04-16T21:12:18.466819+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.466741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:50.153442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'17","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.900906+0000","last_change":"2024-04-16T21:12:12.900906+0000","last_active":"2024-04-16T21:12:12.900906+0000","last_peered":"2024-04-16T21:12:12.900906+0000","last_clean":"2024-04-16T21:12:12.900906+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:12:12.900906+0000","last_undegraded":"2024-04-16T21:12:12.900906+0000","last_fullsized":"2024-04-16T21:12:12.900906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.900807+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:00.936096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'706","reported_seq":2132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.326478+0000","last_change":"2024-04-16T21:12:25.326509+0000","last_active":"2024-04-16T21:12:25.326478+0000","last_peered":"2024-04-16T21:12:25.326478+0000","last_clean":"2024-04-16T21:12:25.326478+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:12:25.326478+0000","last_undegraded":"2024-04-16T21:12:25.326478+0000","last_fullsized":"2024-04-16T21:12:25.326478+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.326417+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:25.301383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'557","reported_seq":1382,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.205288+0000","last_change":"2024-04-16T21:11:50.205349+0000","last_active":"2024-04-16T21:11:50.205288+0000","last_peered":"2024-04-16T21:11:50.205288+0000","last_clean":"2024-04-16T21:11:50.205288+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:11:50.205288+0000","last_undegraded":"2024-04-16T21:11:50.205288+0000","last_fullsized":"2024-04-16T21:11:50.205288+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'557","last_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_deep_scrub":"52'557","last_deep_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.205222+0000","objects_scrubbed":17,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:09.070331+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2200,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'17","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.702594+0000","last_change":"2024-04-16T21:12:11.702662+0000","last_active":"2024-04-16T21:12:11.702594+0000","last_peered":"2024-04-16T21:12:11.702594+0000","last_clean":"2024-04-16T21:12:11.702594+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:12:11.702594+0000","last_undegraded":"2024-04-16T21:12:11.702594+0000","last_fullsized":"2024-04-16T21:12:11.702594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.702519+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:33.204105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106455+0000","last_change":"2024-04-16T19:24:35.398515+0000","last_active":"2024-04-16T21:11:44.106455+0000","last_peered":"2024-04-16T21:11:44.106455+0000","last_clean":"2024-04-16T21:11:44.106455+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:11:44.106455+0000","last_undegraded":"2024-04-16T21:11:44.106455+0000","last_fullsized":"2024-04-16T21:11:44.106455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560193+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:11:44.560193+0000","last_peered":"2024-04-16T21:11:44.560193+0000","last_clean":"2024-04-16T21:11:44.560193+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:11:44.560193+0000","last_undegraded":"2024-04-16T21:11:44.560193+0000","last_fullsized":"2024-04-16T21:11:44.560193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.762674+0000","last_change":"2024-04-16T21:12:43.762696+0000","last_active":"2024-04-16T21:12:43.762674+0000","last_peered":"2024-04-16T21:12:43.762674+0000","last_clean":"2024-04-16T21:12:43.762674+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:12:43.762674+0000","last_undegraded":"2024-04-16T21:12:43.762674+0000","last_fullsized":"2024-04-16T21:12:43.762674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.762572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:58.032922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'147","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.855560+0000","last_change":"2024-04-16T21:12:34.855594+0000","last_active":"2024-04-16T21:12:34.855560+0000","last_peered":"2024-04-16T21:12:34.855560+0000","last_clean":"2024-04-16T21:12:34.855560+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:12:34.855560+0000","last_undegraded":"2024-04-16T21:12:34.855560+0000","last_fullsized":"2024-04-16T21:12:34.855560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.855500+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:22.893155+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.297506+0000","last_change":"2024-04-16T21:12:29.297528+0000","last_active":"2024-04-16T21:12:29.297506+0000","last_peered":"2024-04-16T21:12:29.297506+0000","last_clean":"2024-04-16T21:12:29.297506+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:12:29.297506+0000","last_undegraded":"2024-04-16T21:12:29.297506+0000","last_fullsized":"2024-04-16T21:12:29.297506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.297434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:45.096730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.443393+0000","last_change":"2024-04-16T21:12:07.443416+0000","last_active":"2024-04-16T21:12:07.443393+0000","last_peered":"2024-04-16T21:12:07.443393+0000","last_clean":"2024-04-16T21:12:07.443393+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:12:07.443393+0000","last_undegraded":"2024-04-16T21:12:07.443393+0000","last_fullsized":"2024-04-16T21:12:07.443393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.443331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:23.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.532076+0000","last_change":"2024-04-16T19:47:33.988416+0000","last_active":"2024-04-16T21:11:34.532076+0000","last_peered":"2024-04-16T21:11:34.532076+0000","last_clean":"2024-04-16T21:11:34.532076+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:11:34.532076+0000","last_undegraded":"2024-04-16T21:11:34.532076+0000","last_fullsized":"2024-04-16T21:11:34.532076+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'116","reported_seq":1338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384453+0000","last_change":"2024-04-16T19:25:49.524732+0000","last_active":"2024-04-16T21:10:49.384453+0000","last_peered":"2024-04-16T21:10:49.384453+0000","last_clean":"2024-04-16T21:10:49.384453+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:10:49.384453+0000","last_undegraded":"2024-04-16T21:10:49.384453+0000","last_fullsized":"2024-04-16T21:10:49.384453+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.850753+0000","last_change":"2024-04-16T21:12:33.850776+0000","last_active":"2024-04-16T21:12:33.850753+0000","last_peered":"2024-04-16T21:12:33.850753+0000","last_clean":"2024-04-16T21:12:33.850753+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:12:33.850753+0000","last_undegraded":"2024-04-16T21:12:33.850753+0000","last_fullsized":"2024-04-16T21:12:33.850753+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.850667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:48.121017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'10","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.431983+0000","last_change":"2024-04-16T21:12:23.431983+0000","last_active":"2024-04-16T21:12:23.431983+0000","last_peered":"2024-04-16T21:12:23.431983+0000","last_clean":"2024-04-16T21:12:23.431983+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:12:23.431983+0000","last_undegraded":"2024-04-16T21:12:23.431983+0000","last_fullsized":"2024-04-16T21:12:23.431983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.431886+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:37.702235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'706","reported_seq":2082,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.853855+0000","last_change":"2024-04-16T21:12:01.853885+0000","last_active":"2024-04-16T21:12:01.853855+0000","last_peered":"2024-04-16T21:12:01.853855+0000","last_clean":"2024-04-16T21:12:01.853855+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:12:01.853855+0000","last_undegraded":"2024-04-16T21:12:01.853855+0000","last_fullsized":"2024-04-16T21:12:01.853855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.853792+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.601423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'542","reported_seq":41299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.597167+0000","last_change":"2024-04-16T21:11:54.597266+0000","last_active":"2024-04-16T21:11:54.597167+0000","last_peered":"2024-04-16T21:11:54.597167+0000","last_clean":"2024-04-16T21:11:54.597167+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:11:54.597167+0000","last_undegraded":"2024-04-16T21:11:54.597167+0000","last_fullsized":"2024-04-16T21:11:54.597167+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'542","last_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_deep_scrub":"52'542","last_deep_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.597093+0000","objects_scrubbed":23,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:16.539182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'18","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.434844+0000","last_change":"2024-04-16T21:11:54.434914+0000","last_active":"2024-04-16T21:11:54.434844+0000","last_peered":"2024-04-16T21:11:54.434844+0000","last_clean":"2024-04-16T21:11:54.434844+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:11:54.434844+0000","last_undegraded":"2024-04-16T21:11:54.434844+0000","last_fullsized":"2024-04-16T21:11:54.434844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.434778+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:53.824584+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.707330+0000","last_change":"2024-04-16T21:12:36.707352+0000","last_active":"2024-04-16T21:12:36.707330+0000","last_peered":"2024-04-16T21:12:36.707330+0000","last_clean":"2024-04-16T21:12:36.707330+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:12:36.707330+0000","last_undegraded":"2024-04-16T21:12:36.707330+0000","last_fullsized":"2024-04-16T21:12:36.707330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.707266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:58.955541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560163+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:11:44.560163+0000","last_peered":"2024-04-16T21:11:44.560163+0000","last_clean":"2024-04-16T21:11:44.560163+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:11:44.560163+0000","last_undegraded":"2024-04-16T21:11:44.560163+0000","last_fullsized":"2024-04-16T21:11:44.560163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'13","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.860330+0000","last_change":"2024-04-16T21:12:43.860357+0000","last_active":"2024-04-16T21:12:43.860330+0000","last_peered":"2024-04-16T21:12:43.860330+0000","last_clean":"2024-04-16T21:12:43.860330+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:12:43.860330+0000","last_undegraded":"2024-04-16T21:12:43.860330+0000","last_fullsized":"2024-04-16T21:12:43.860330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.860269+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:26.916002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.311803+0000","last_change":"2024-04-16T21:12:39.311841+0000","last_active":"2024-04-16T21:12:39.311803+0000","last_peered":"2024-04-16T21:12:39.311803+0000","last_clean":"2024-04-16T21:12:39.311803+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:12:39.311803+0000","last_undegraded":"2024-04-16T21:12:39.311803+0000","last_fullsized":"2024-04-16T21:12:39.311803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.311721+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:11.016514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.303819+0000","last_change":"2024-04-16T21:12:29.303842+0000","last_active":"2024-04-16T21:12:29.303819+0000","last_peered":"2024-04-16T21:12:29.303819+0000","last_clean":"2024-04-16T21:12:29.303819+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:12:29.303819+0000","last_undegraded":"2024-04-16T21:12:29.303819+0000","last_fullsized":"2024-04-16T21:12:29.303819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.303752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:17.910045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.776294+0000","last_change":"2024-04-16T21:12:00.776335+0000","last_active":"2024-04-16T21:12:00.776294+0000","last_peered":"2024-04-16T21:12:00.776294+0000","last_clean":"2024-04-16T21:12:00.776294+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:12:00.776294+0000","last_undegraded":"2024-04-16T21:12:00.776294+0000","last_fullsized":"2024-04-16T21:12:00.776294+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.776223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:20.604086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398019+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:11:34.398019+0000","last_peered":"2024-04-16T21:11:34.398019+0000","last_clean":"2024-04-16T21:11:34.398019+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:11:34.398019+0000","last_undegraded":"2024-04-16T21:11:34.398019+0000","last_fullsized":"2024-04-16T21:11:34.398019+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'222","reported_seq":2177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.980519+0000","last_change":"2024-04-16T21:12:16.980550+0000","last_active":"2024-04-16T21:12:16.980519+0000","last_peered":"2024-04-16T21:12:16.980519+0000","last_clean":"2024-04-16T21:12:16.980519+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:12:16.980519+0000","last_undegraded":"2024-04-16T21:12:16.980519+0000","last_fullsized":"2024-04-16T21:12:16.980519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'222","last_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_deep_scrub":"52'222","last_deep_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.980456+0000","objects_scrubbed":55,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:41.646543+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14526,"num_omap_keys":59,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.913868+0000","last_change":"2024-04-16T21:12:10.913891+0000","last_active":"2024-04-16T21:12:10.913868+0000","last_peered":"2024-04-16T21:12:10.913868+0000","last_clean":"2024-04-16T21:12:10.913868+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:12:10.913868+0000","last_undegraded":"2024-04-16T21:12:10.913868+0000","last_fullsized":"2024-04-16T21:12:10.913868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.913806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:17.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'24","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.799425+0000","last_change":"2024-04-16T21:12:26.799425+0000","last_active":"2024-04-16T21:12:26.799425+0000","last_peered":"2024-04-16T21:12:26.799425+0000","last_clean":"2024-04-16T21:12:26.799425+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:12:26.799425+0000","last_undegraded":"2024-04-16T21:12:26.799425+0000","last_fullsized":"2024-04-16T21:12:26.799425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.799297+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:18.472986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.652503+0000","last_change":"2024-04-16T21:11:59.652525+0000","last_active":"2024-04-16T21:11:59.652503+0000","last_peered":"2024-04-16T21:11:59.652503+0000","last_clean":"2024-04-16T21:11:59.652503+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:11:59.652503+0000","last_undegraded":"2024-04-16T21:11:59.652503+0000","last_fullsized":"2024-04-16T21:11:59.652503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.652420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:18.517529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'592","reported_seq":1363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.540243+0000","last_change":"2024-04-16T21:12:00.540298+0000","last_active":"2024-04-16T21:12:00.540243+0000","last_peered":"2024-04-16T21:12:00.540243+0000","last_clean":"2024-04-16T21:12:00.540243+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:12:00.540243+0000","last_undegraded":"2024-04-16T21:12:00.540243+0000","last_fullsized":"2024-04-16T21:12:00.540243+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'592","last_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_deep_scrub":"52'592","last_deep_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.540183+0000","objects_scrubbed":19,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:48.575472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'32","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.552154+0000","last_change":"2024-04-16T21:11:43.552192+0000","last_active":"2024-04-16T21:11:43.552154+0000","last_peered":"2024-04-16T21:11:43.552154+0000","last_clean":"2024-04-16T21:11:43.552154+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:11:43.552154+0000","last_undegraded":"2024-04-16T21:11:43.552154+0000","last_fullsized":"2024-04-16T21:11:43.552154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.552108+0000","objects_scrubbed":9,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.135567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395593+0000","last_change":"2024-04-16T19:24:35.391705+0000","last_active":"2024-04-16T21:11:19.395593+0000","last_peered":"2024-04-16T21:11:19.395593+0000","last_clean":"2024-04-16T21:11:19.395593+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:11:19.395593+0000","last_undegraded":"2024-04-16T21:11:19.395593+0000","last_fullsized":"2024-04-16T21:11:19.395593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524792+0000","last_change":"2024-04-16T19:24:33.387923+0000","last_active":"2024-04-16T21:10:49.524792+0000","last_peered":"2024-04-16T21:10:49.524792+0000","last_clean":"2024-04-16T21:10:49.524792+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:10:49.524792+0000","last_undegraded":"2024-04-16T21:10:49.524792+0000","last_fullsized":"2024-04-16T21:10:49.524792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"47'4","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.388587+0000","last_change":"2024-04-16T19:24:45.414116+0000","last_active":"2024-04-16T21:11:09.388587+0000","last_peered":"2024-04-16T21:11:09.388587+0000","last_clean":"2024-04-16T21:11:09.388587+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:11:09.388587+0000","last_undegraded":"2024-04-16T21:11:09.388587+0000","last_fullsized":"2024-04-16T21:11:09.388587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'32","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.284966+0000","last_change":"2024-04-16T21:12:36.284996+0000","last_active":"2024-04-16T21:12:36.284966+0000","last_peered":"2024-04-16T21:12:36.284966+0000","last_clean":"2024-04-16T21:12:36.284966+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:12:36.284966+0000","last_undegraded":"2024-04-16T21:12:36.284966+0000","last_fullsized":"2024-04-16T21:12:36.284966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.284903+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:50.863442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.384509+0000","last_change":"2024-04-16T21:12:11.384509+0000","last_active":"2024-04-16T21:12:11.384509+0000","last_peered":"2024-04-16T21:12:11.384509+0000","last_clean":"2024-04-16T21:12:11.384509+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:12:11.384509+0000","last_undegraded":"2024-04-16T21:12:11.384509+0000","last_fullsized":"2024-04-16T21:12:11.384509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.384424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:06.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.514240+0000","last_change":"2024-04-16T21:12:02.514262+0000","last_active":"2024-04-16T21:12:02.514240+0000","last_peered":"2024-04-16T21:12:02.514240+0000","last_clean":"2024-04-16T21:12:02.514240+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:12:02.514240+0000","last_undegraded":"2024-04-16T21:12:02.514240+0000","last_fullsized":"2024-04-16T21:12:02.514240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.514179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:44.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.391931+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:11:34.391931+0000","last_peered":"2024-04-16T21:11:34.391931+0000","last_clean":"2024-04-16T21:11:34.391931+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:11:34.391931+0000","last_undegraded":"2024-04-16T21:11:34.391931+0000","last_fullsized":"2024-04-16T21:11:34.391931+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'172","reported_seq":1564,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.949162+0000","last_change":"2024-04-16T21:12:12.949215+0000","last_active":"2024-04-16T21:12:12.949162+0000","last_peered":"2024-04-16T21:12:12.949162+0000","last_clean":"2024-04-16T21:12:12.949162+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:12:12.949162+0000","last_undegraded":"2024-04-16T21:12:12.949162+0000","last_fullsized":"2024-04-16T21:12:12.949162+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.949100+0000","objects_scrubbed":45,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.338906+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8810,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.869571+0000","last_change":"2024-04-16T21:12:09.869596+0000","last_active":"2024-04-16T21:12:09.869571+0000","last_peered":"2024-04-16T21:12:09.869571+0000","last_clean":"2024-04-16T21:12:09.869571+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:12:09.869571+0000","last_undegraded":"2024-04-16T21:12:09.869571+0000","last_fullsized":"2024-04-16T21:12:09.869571+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.869508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:34.535595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"52'16","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.491403+0000","last_change":"2024-04-16T21:12:21.491403+0000","last_active":"2024-04-16T21:12:21.491403+0000","last_peered":"2024-04-16T21:12:21.491403+0000","last_clean":"2024-04-16T21:12:21.491403+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:12:21.491403+0000","last_undegraded":"2024-04-16T21:12:21.491403+0000","last_fullsized":"2024-04-16T21:12:21.491403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.491306+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:09.623796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.693490+0000","last_change":"2024-04-16T21:12:01.693513+0000","last_active":"2024-04-16T21:12:01.693490+0000","last_peered":"2024-04-16T21:12:01.693490+0000","last_clean":"2024-04-16T21:12:01.693490+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:12:01.693490+0000","last_undegraded":"2024-04-16T21:12:01.693490+0000","last_fullsized":"2024-04-16T21:12:01.693490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.693430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:04.866318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'249","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.401831+0000","last_change":"2024-04-16T21:11:48.401911+0000","last_active":"2024-04-16T21:11:48.401831+0000","last_peered":"2024-04-16T21:11:48.401831+0000","last_clean":"2024-04-16T21:11:48.401831+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:11:48.401831+0000","last_undegraded":"2024-04-16T21:11:48.401831+0000","last_fullsized":"2024-04-16T21:11:48.401831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.401765+0000","objects_scrubbed":19,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:02.153361+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'12","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.683222+0000","last_change":"2024-04-16T21:12:12.683275+0000","last_active":"2024-04-16T21:12:12.683222+0000","last_peered":"2024-04-16T21:12:12.683222+0000","last_clean":"2024-04-16T21:12:12.683222+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:12:12.683222+0000","last_undegraded":"2024-04-16T21:12:12.683222+0000","last_fullsized":"2024-04-16T21:12:12.683222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.683160+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.072966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387647+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:10:59.387647+0000","last_peered":"2024-04-16T21:10:59.387647+0000","last_clean":"2024-04-16T21:10:59.387647+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:10:59.387647+0000","last_undegraded":"2024-04-16T21:10:59.387647+0000","last_fullsized":"2024-04-16T21:10:59.387647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390741+0000","last_change":"2024-04-16T19:24:33.389996+0000","last_active":"2024-04-16T21:10:49.390741+0000","last_peered":"2024-04-16T21:10:49.390741+0000","last_clean":"2024-04-16T21:10:49.390741+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:10:49.390741+0000","last_undegraded":"2024-04-16T21:10:49.390741+0000","last_fullsized":"2024-04-16T21:10:49.390741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"52'4","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.017072+0000","last_change":"2024-04-16T21:12:36.017103+0000","last_active":"2024-04-16T21:12:36.017072+0000","last_peered":"2024-04-16T21:12:36.017072+0000","last_clean":"2024-04-16T21:12:36.017072+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:12:36.017072+0000","last_undegraded":"2024-04-16T21:12:36.017072+0000","last_fullsized":"2024-04-16T21:12:36.017072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.017012+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:24.149502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'74","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106674+0000","last_change":"2024-04-16T19:24:47.419041+0000","last_active":"2024-04-16T21:11:44.106674+0000","last_peered":"2024-04-16T21:11:44.106674+0000","last_clean":"2024-04-16T21:11:44.106674+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:11:44.106674+0000","last_undegraded":"2024-04-16T21:11:44.106674+0000","last_fullsized":"2024-04-16T21:11:44.106674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:46.405599+0000","last_clean_scrub_stamp":"2024-04-16T19:24:46.405599+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.466444+0000","last_change":"2024-04-16T21:12:11.466444+0000","last_active":"2024-04-16T21:12:11.466444+0000","last_peered":"2024-04-16T21:12:11.466444+0000","last_clean":"2024-04-16T21:12:11.466444+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:12:11.466444+0000","last_undegraded":"2024-04-16T21:12:11.466444+0000","last_fullsized":"2024-04-16T21:12:11.466444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.466351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:26.935725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.718042+0000","last_change":"2024-04-16T21:12:16.718042+0000","last_active":"2024-04-16T21:12:16.718042+0000","last_peered":"2024-04-16T21:12:16.718042+0000","last_clean":"2024-04-16T21:12:16.718042+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:12:16.718042+0000","last_undegraded":"2024-04-16T21:12:16.718042+0000","last_fullsized":"2024-04-16T21:12:16.718042+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.717926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:06.936826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'19","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.886963+0000","last_change":"2024-04-16T21:12:06.887055+0000","last_active":"2024-04-16T21:12:06.886963+0000","last_peered":"2024-04-16T21:12:06.886963+0000","last_clean":"2024-04-16T21:12:06.886963+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:12:06.886963+0000","last_undegraded":"2024-04-16T21:12:06.886963+0000","last_fullsized":"2024-04-16T21:12:06.886963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.886889+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:22.161394+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.055428+0000","last_change":"2024-04-16T21:12:41.055453+0000","last_active":"2024-04-16T21:12:41.055428+0000","last_peered":"2024-04-16T21:12:41.055428+0000","last_clean":"2024-04-16T21:12:41.055428+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:12:41.055428+0000","last_undegraded":"2024-04-16T21:12:41.055428+0000","last_fullsized":"2024-04-16T21:12:41.055428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.055365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:29.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385227+0000","last_change":"2024-04-16T19:24:33.383470+0000","last_active":"2024-04-16T21:10:49.385227+0000","last_peered":"2024-04-16T21:10:49.385227+0000","last_clean":"2024-04-16T21:10:49.385227+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:10:49.385227+0000","last_undegraded":"2024-04-16T21:10:49.385227+0000","last_fullsized":"2024-04-16T21:10:49.385227+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'18","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.679302+0000","last_change":"2024-04-16T21:12:03.679355+0000","last_active":"2024-04-16T21:12:03.679302+0000","last_peered":"2024-04-16T21:12:03.679302+0000","last_clean":"2024-04-16T21:12:03.679302+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:12:03.679302+0000","last_undegraded":"2024-04-16T21:12:03.679302+0000","last_fullsized":"2024-04-16T21:12:03.679302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.679242+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:31.737391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.384640+0000","last_change":"2024-04-16T19:24:35.391856+0000","last_active":"2024-04-16T21:10:49.384640+0000","last_peered":"2024-04-16T21:10:49.384640+0000","last_clean":"2024-04-16T21:10:49.384640+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:10:49.384640+0000","last_undegraded":"2024-04-16T21:10:49.384640+0000","last_fullsized":"2024-04-16T21:10:49.384640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.524668+0000","last_change":"2024-04-16T19:24:33.386400+0000","last_active":"2024-04-16T21:10:49.524668+0000","last_peered":"2024-04-16T21:10:49.524668+0000","last_clean":"2024-04-16T21:10:49.524668+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:10:49.524668+0000","last_undegraded":"2024-04-16T21:10:49.524668+0000","last_fullsized":"2024-04-16T21:10:49.524668+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'10","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.673321+0000","last_change":"2024-04-16T21:12:02.673359+0000","last_active":"2024-04-16T21:12:02.673321+0000","last_peered":"2024-04-16T21:12:02.673321+0000","last_clean":"2024-04-16T21:12:02.673321+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:12:02.673321+0000","last_undegraded":"2024-04-16T21:12:02.673321+0000","last_fullsized":"2024-04-16T21:12:02.673321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.673266+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:06.001051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385387+0000","last_change":"2024-04-16T19:24:35.396348+0000","last_active":"2024-04-16T21:10:49.385387+0000","last_peered":"2024-04-16T21:10:49.385387+0000","last_clean":"2024-04-16T21:10:49.385387+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:10:49.385387+0000","last_undegraded":"2024-04-16T21:10:49.385387+0000","last_fullsized":"2024-04-16T21:10:49.385387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.385866+0000","last_change":"2024-04-16T19:24:33.386861+0000","last_active":"2024-04-16T21:10:49.385866+0000","last_peered":"2024-04-16T21:10:49.385866+0000","last_clean":"2024-04-16T21:10:49.385866+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:10:49.385866+0000","last_undegraded":"2024-04-16T21:10:49.385866+0000","last_fullsized":"2024-04-16T21:10:49.385866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'13","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.447533+0000","last_change":"2024-04-16T21:11:57.447598+0000","last_active":"2024-04-16T21:11:57.447533+0000","last_peered":"2024-04-16T21:11:57.447533+0000","last_clean":"2024-04-16T21:11:57.447533+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:11:57.447533+0000","last_undegraded":"2024-04-16T21:11:57.447533+0000","last_fullsized":"2024-04-16T21:11:57.447533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.447471+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:25.758892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.002410+0000","last_change":"2024-04-16T21:12:37.002453+0000","last_active":"2024-04-16T21:12:37.002410+0000","last_peered":"2024-04-16T21:12:37.002410+0000","last_clean":"2024-04-16T21:12:37.002410+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:12:37.002410+0000","last_undegraded":"2024-04-16T21:12:37.002410+0000","last_fullsized":"2024-04-16T21:12:37.002410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.002335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:36.977300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393952+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:11:09.393952+0000","last_peered":"2024-04-16T21:11:09.393952+0000","last_clean":"2024-04-16T21:11:09.393952+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:11:09.393952+0000","last_undegraded":"2024-04-16T21:11:09.393952+0000","last_fullsized":"2024-04-16T21:11:09.393952+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'20","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.649822+0000","last_change":"2024-04-16T21:12:04.649822+0000","last_active":"2024-04-16T21:12:04.649822+0000","last_peered":"2024-04-16T21:12:04.649822+0000","last_clean":"2024-04-16T21:12:04.649822+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:12:04.649822+0000","last_undegraded":"2024-04-16T21:12:04.649822+0000","last_fullsized":"2024-04-16T21:12:04.649822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.649711+0000","objects_scrubbed":9,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:32.961132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.869842+0000","last_change":"2024-04-16T21:12:27.869867+0000","last_active":"2024-04-16T21:12:27.869842+0000","last_peered":"2024-04-16T21:12:27.869842+0000","last_clean":"2024-04-16T21:12:27.869842+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:12:27.869842+0000","last_undegraded":"2024-04-16T21:12:27.869842+0000","last_fullsized":"2024-04-16T21:12:27.869842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.869765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:16.833314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383688+0000","last_change":"2024-04-16T19:26:41.916271+0000","last_active":"2024-04-16T21:10:44.383688+0000","last_peered":"2024-04-16T21:10:44.383688+0000","last_clean":"2024-04-16T21:10:44.383688+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:10:44.383688+0000","last_undegraded":"2024-04-16T21:10:44.383688+0000","last_fullsized":"2024-04-16T21:10:44.383688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'28","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.947946+0000","last_change":"2024-04-16T21:12:13.947946+0000","last_active":"2024-04-16T21:12:13.947946+0000","last_peered":"2024-04-16T21:12:13.947946+0000","last_clean":"2024-04-16T21:12:13.947946+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:12:13.947946+0000","last_undegraded":"2024-04-16T21:12:13.947946+0000","last_fullsized":"2024-04-16T21:12:13.947946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'28","last_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_deep_scrub":"52'28","last_deep_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.947818+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:20.063878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.699314+0000","last_change":"2024-04-16T21:12:35.699335+0000","last_active":"2024-04-16T21:12:35.699314+0000","last_peered":"2024-04-16T21:12:35.699314+0000","last_clean":"2024-04-16T21:12:35.699314+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:12:35.699314+0000","last_undegraded":"2024-04-16T21:12:35.699314+0000","last_fullsized":"2024-04-16T21:12:35.699314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.699249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:06:40.252661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.383706+0000","last_change":"2024-04-16T19:24:33.386845+0000","last_active":"2024-04-16T21:10:44.383706+0000","last_peered":"2024-04-16T21:10:44.383706+0000","last_clean":"2024-04-16T21:10:44.383706+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:10:44.383706+0000","last_undegraded":"2024-04-16T21:10:44.383706+0000","last_fullsized":"2024-04-16T21:10:44.383706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'12","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.624929+0000","last_change":"2024-04-16T21:11:55.624974+0000","last_active":"2024-04-16T21:11:55.624929+0000","last_peered":"2024-04-16T21:11:55.624929+0000","last_clean":"2024-04-16T21:11:55.624929+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:11:55.624929+0000","last_undegraded":"2024-04-16T21:11:55.624929+0000","last_fullsized":"2024-04-16T21:11:55.624929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.624828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:36.218593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.021978+0000","last_change":"2024-04-16T21:12:38.022020+0000","last_active":"2024-04-16T21:12:38.021978+0000","last_peered":"2024-04-16T21:12:38.021978+0000","last_clean":"2024-04-16T21:12:38.021978+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:12:38.021978+0000","last_undegraded":"2024-04-16T21:12:38.021978+0000","last_fullsized":"2024-04-16T21:12:38.021978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.021905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:52.292254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:44.385174+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:10:44.385174+0000","last_peered":"2024-04-16T21:10:44.385174+0000","last_clean":"2024-04-16T21:10:44.385174+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:10:44.385174+0000","last_undegraded":"2024-04-16T21:10:44.385174+0000","last_fullsized":"2024-04-16T21:10:44.385174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'26","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.939446+0000","last_change":"2024-04-16T21:12:15.939510+0000","last_active":"2024-04-16T21:12:15.939446+0000","last_peered":"2024-04-16T21:12:15.939446+0000","last_clean":"2024-04-16T21:12:15.939446+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:12:15.939446+0000","last_undegraded":"2024-04-16T21:12:15.939446+0000","last_fullsized":"2024-04-16T21:12:15.939446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.939383+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:49:32.014798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386679+0000","last_change":"2024-04-16T19:24:35.390105+0000","last_active":"2024-04-16T21:10:54.386679+0000","last_peered":"2024-04-16T21:10:54.386679+0000","last_clean":"2024-04-16T21:10:54.386679+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:10:54.386679+0000","last_undegraded":"2024-04-16T21:10:54.386679+0000","last_fullsized":"2024-04-16T21:10:54.386679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.764730+0000","last_change":"2024-04-16T21:12:41.764753+0000","last_active":"2024-04-16T21:12:41.764730+0000","last_peered":"2024-04-16T21:12:41.764730+0000","last_clean":"2024-04-16T21:12:41.764730+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:12:41.764730+0000","last_undegraded":"2024-04-16T21:12:41.764730+0000","last_fullsized":"2024-04-16T21:12:41.764730+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.764666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:53.367336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'46","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.408711+0000","last_change":"2024-04-16T21:11:46.408756+0000","last_active":"2024-04-16T21:11:46.408711+0000","last_peered":"2024-04-16T21:11:46.408711+0000","last_clean":"2024-04-16T21:11:46.408711+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:11:46.408711+0000","last_undegraded":"2024-04-16T21:11:46.408711+0000","last_fullsized":"2024-04-16T21:11:46.408711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'46","last_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_deep_scrub":"52'46","last_deep_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.408648+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:38.312038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292194+0000","last_change":"2024-04-16T19:24:35.392447+0000","last_active":"2024-04-16T21:11:42.292194+0000","last_peered":"2024-04-16T21:11:42.292194+0000","last_clean":"2024-04-16T21:11:42.292194+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:11:42.292194+0000","last_undegraded":"2024-04-16T21:11:42.292194+0000","last_fullsized":"2024-04-16T21:11:42.292194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387643+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:11:04.387643+0000","last_peered":"2024-04-16T21:11:04.387643+0000","last_clean":"2024-04-16T21:11:04.387643+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:11:04.387643+0000","last_undegraded":"2024-04-16T21:11:04.387643+0000","last_fullsized":"2024-04-16T21:11:04.387643+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'20","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.919349+0000","last_change":"2024-04-16T21:11:47.919430+0000","last_active":"2024-04-16T21:11:47.919349+0000","last_peered":"2024-04-16T21:11:47.919349+0000","last_clean":"2024-04-16T21:11:47.919349+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:11:47.919349+0000","last_undegraded":"2024-04-16T21:11:47.919349+0000","last_fullsized":"2024-04-16T21:11:47.919349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.919276+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:11:50.792760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.106421+0000","last_change":"2024-04-16T19:24:35.396482+0000","last_active":"2024-04-16T21:11:44.106421+0000","last_peered":"2024-04-16T21:11:44.106421+0000","last_clean":"2024-04-16T21:11:44.106421+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:11:44.106421+0000","last_undegraded":"2024-04-16T21:11:44.106421+0000","last_fullsized":"2024-04-16T21:11:44.106421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291939+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:11:42.291939+0000","last_peered":"2024-04-16T21:11:42.291939+0000","last_clean":"2024-04-16T21:11:42.291939+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:11:42.291939+0000","last_undegraded":"2024-04-16T21:11:42.291939+0000","last_fullsized":"2024-04-16T21:11:42.291939+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'17","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.209595+0000","last_change":"2024-04-16T21:11:47.209653+0000","last_active":"2024-04-16T21:11:47.209595+0000","last_peered":"2024-04-16T21:11:47.209595+0000","last_clean":"2024-04-16T21:11:47.209595+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:11:47.209595+0000","last_undegraded":"2024-04-16T21:11:47.209595+0000","last_fullsized":"2024-04-16T21:11:47.209595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.209532+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:30.862167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.844024+0000","last_change":"2024-04-16T21:12:31.844046+0000","last_active":"2024-04-16T21:12:31.844024+0000","last_peered":"2024-04-16T21:12:31.844024+0000","last_clean":"2024-04-16T21:12:31.844024+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:12:31.844024+0000","last_undegraded":"2024-04-16T21:12:31.844024+0000","last_fullsized":"2024-04-16T21:12:31.844024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.843954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:05.611280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560222+0000","last_change":"2024-04-16T19:24:33.386141+0000","last_active":"2024-04-16T21:11:44.560222+0000","last_peered":"2024-04-16T21:11:44.560222+0000","last_clean":"2024-04-16T21:11:44.560222+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:11:44.560222+0000","last_undegraded":"2024-04-16T21:11:44.560222+0000","last_fullsized":"2024-04-16T21:11:44.560222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'27","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.894735+0000","last_change":"2024-04-16T21:11:49.894811+0000","last_active":"2024-04-16T21:11:49.894735+0000","last_peered":"2024-04-16T21:11:49.894735+0000","last_clean":"2024-04-16T21:11:49.894735+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:11:49.894735+0000","last_undegraded":"2024-04-16T21:11:49.894735+0000","last_fullsized":"2024-04-16T21:11:49.894735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.894658+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.290532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650351+0000","last_change":"2024-04-16T19:24:35.394939+0000","last_active":"2024-04-16T21:11:43.650351+0000","last_peered":"2024-04-16T21:11:43.650351+0000","last_clean":"2024-04-16T21:11:43.650351+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:11:43.650351+0000","last_undegraded":"2024-04-16T21:11:43.650351+0000","last_fullsized":"2024-04-16T21:11:43.650351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.863200+0000","last_change":"2024-04-16T21:12:05.863236+0000","last_active":"2024-04-16T21:12:05.863200+0000","last_peered":"2024-04-16T21:12:05.863200+0000","last_clean":"2024-04-16T21:12:05.863200+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:12:05.863200+0000","last_undegraded":"2024-04-16T21:12:05.863200+0000","last_fullsized":"2024-04-16T21:12:05.863200+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.863135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:16.509928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'10","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.658621+0000","last_change":"2024-04-16T21:11:58.658671+0000","last_active":"2024-04-16T21:11:58.658621+0000","last_peered":"2024-04-16T21:11:58.658621+0000","last_clean":"2024-04-16T21:11:58.658621+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:11:58.658621+0000","last_undegraded":"2024-04-16T21:11:58.658621+0000","last_fullsized":"2024-04-16T21:11:58.658621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.658555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:31.251581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.764316+0000","last_change":"2024-04-16T21:12:42.764340+0000","last_active":"2024-04-16T21:12:42.764316+0000","last_peered":"2024-04-16T21:12:42.764316+0000","last_clean":"2024-04-16T21:12:42.764316+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:12:42.764316+0000","last_undegraded":"2024-04-16T21:12:42.764316+0000","last_fullsized":"2024-04-16T21:12:42.764316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.764250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:42.739215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650075+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:11:43.650075+0000","last_peered":"2024-04-16T21:11:43.650075+0000","last_clean":"2024-04-16T21:11:43.650075+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:11:43.650075+0000","last_undegraded":"2024-04-16T21:11:43.650075+0000","last_fullsized":"2024-04-16T21:11:43.650075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'22","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.811095+0000","last_change":"2024-04-16T21:11:53.811157+0000","last_active":"2024-04-16T21:11:53.811095+0000","last_peered":"2024-04-16T21:11:53.811095+0000","last_clean":"2024-04-16T21:11:53.811095+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:11:53.811095+0000","last_undegraded":"2024-04-16T21:11:53.811095+0000","last_fullsized":"2024-04-16T21:11:53.811095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.811028+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:31.711738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.431513+0000","last_change":"2024-04-16T21:12:12.431513+0000","last_active":"2024-04-16T21:12:12.431513+0000","last_peered":"2024-04-16T21:12:12.431513+0000","last_clean":"2024-04-16T21:12:12.431513+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:12:12.431513+0000","last_undegraded":"2024-04-16T21:12:12.431513+0000","last_fullsized":"2024-04-16T21:12:12.431513+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.431389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:06.860124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:39.384269+0000","last_change":"2024-04-16T19:24:33.388771+0000","last_active":"2024-04-16T21:10:39.384269+0000","last_peered":"2024-04-16T21:10:39.384269+0000","last_clean":"2024-04-16T21:10:39.384269+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:10:39.384269+0000","last_undegraded":"2024-04-16T21:10:39.384269+0000","last_fullsized":"2024-04-16T21:10:39.384269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'6","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.660156+0000","last_change":"2024-04-16T21:12:03.660192+0000","last_active":"2024-04-16T21:12:03.660156+0000","last_peered":"2024-04-16T21:12:03.660156+0000","last_clean":"2024-04-16T21:12:03.660156+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:12:03.660156+0000","last_undegraded":"2024-04-16T21:12:03.660156+0000","last_fullsized":"2024-04-16T21:12:03.660156+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.660092+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:42.368715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.153401+0000","last_change":"2024-04-16T21:12:44.153443+0000","last_active":"2024-04-16T21:12:44.153401+0000","last_peered":"2024-04-16T21:12:44.153401+0000","last_clean":"2024-04-16T21:12:44.153401+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:12:44.153401+0000","last_undegraded":"2024-04-16T21:12:44.153401+0000","last_fullsized":"2024-04-16T21:12:44.153401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.153324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:59.952621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.194855+0000","last_change":"2024-04-16T19:24:33.391585+0000","last_active":"2024-04-16T21:11:43.194855+0000","last_peered":"2024-04-16T21:11:43.194855+0000","last_clean":"2024-04-16T21:11:43.194855+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:11:43.194855+0000","last_undegraded":"2024-04-16T21:11:43.194855+0000","last_fullsized":"2024-04-16T21:11:43.194855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'26","reported_seq":302,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.166590+0000","last_change":"2024-04-16T21:11:46.166636+0000","last_active":"2024-04-16T21:11:46.166590+0000","last_peered":"2024-04-16T21:11:46.166590+0000","last_clean":"2024-04-16T21:11:46.166590+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:11:46.166590+0000","last_undegraded":"2024-04-16T21:11:46.166590+0000","last_fullsized":"2024-04-16T21:11:46.166590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.166525+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:05.994388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389319+0000","last_change":"2024-04-16T19:24:35.390747+0000","last_active":"2024-04-16T21:11:09.389319+0000","last_peered":"2024-04-16T21:11:09.389319+0000","last_clean":"2024-04-16T21:11:09.389319+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:11:09.389319+0000","last_undegraded":"2024-04-16T21:11:09.389319+0000","last_fullsized":"2024-04-16T21:11:09.389319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560276+0000","last_change":"2024-04-16T19:24:33.388665+0000","last_active":"2024-04-16T21:11:44.560276+0000","last_peered":"2024-04-16T21:11:44.560276+0000","last_clean":"2024-04-16T21:11:44.560276+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:11:44.560276+0000","last_undegraded":"2024-04-16T21:11:44.560276+0000","last_fullsized":"2024-04-16T21:11:44.560276+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'22","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.232333+0000","last_change":"2024-04-16T21:11:55.232333+0000","last_active":"2024-04-16T21:11:55.232333+0000","last_peered":"2024-04-16T21:11:55.232333+0000","last_clean":"2024-04-16T21:11:55.232333+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:11:55.232333+0000","last_undegraded":"2024-04-16T21:11:55.232333+0000","last_fullsized":"2024-04-16T21:11:55.232333+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.232184+0000","objects_scrubbed":10,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:19.898271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650107+0000","last_change":"2024-04-16T19:24:33.386775+0000","last_active":"2024-04-16T21:11:43.650107+0000","last_peered":"2024-04-16T21:11:43.650107+0000","last_clean":"2024-04-16T21:11:43.650107+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:11:43.650107+0000","last_undegraded":"2024-04-16T21:11:43.650107+0000","last_fullsized":"2024-04-16T21:11:43.650107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:19.395747+0000","last_change":"2024-04-16T19:24:35.396667+0000","last_active":"2024-04-16T21:11:19.395747+0000","last_peered":"2024-04-16T21:11:19.395747+0000","last_clean":"2024-04-16T21:11:19.395747+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:11:19.395747+0000","last_undegraded":"2024-04-16T21:11:19.395747+0000","last_fullsized":"2024-04-16T21:11:19.395747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'6","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.936034+0000","last_change":"2024-04-16T21:12:14.936034+0000","last_active":"2024-04-16T21:12:14.936034+0000","last_peered":"2024-04-16T21:12:14.936034+0000","last_clean":"2024-04-16T21:12:14.936034+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:12:14.936034+0000","last_undegraded":"2024-04-16T21:12:14.936034+0000","last_fullsized":"2024-04-16T21:12:14.936034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.935939+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:02.653615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.742979+0000","last_change":"2024-04-16T19:24:33.385570+0000","last_active":"2024-04-16T21:11:42.742979+0000","last_peered":"2024-04-16T21:11:42.742979+0000","last_clean":"2024-04-16T21:11:42.742979+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:11:42.742979+0000","last_undegraded":"2024-04-16T21:11:42.742979+0000","last_fullsized":"2024-04-16T21:11:42.742979+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387111+0000","last_change":"2024-04-16T19:24:35.401185+0000","last_active":"2024-04-16T21:10:59.387111+0000","last_peered":"2024-04-16T21:10:59.387111+0000","last_clean":"2024-04-16T21:10:59.387111+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:10:59.387111+0000","last_undegraded":"2024-04-16T21:10:59.387111+0000","last_fullsized":"2024-04-16T21:10:59.387111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":233901,"num_omap_keys":948,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5163,"ondisk_log_size":5163,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3590,"ondisk_log_size":3590,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1190,"ondisk_log_size":1190,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16207,"ondisk_log_size":16207,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5669,"ondisk_log_size":5669,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9989,"num_omap_keys":63,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641567,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28652,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104828948,"statfs":{"total":107374182400,"available":107344842752,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641568,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59084,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798516,"statfs":{"total":107374182400,"available":107313680384,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641567,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59728,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797872,"statfs":{"total":107374182400,"available":107313020928,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641568,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46748,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810852,"statfs":{"total":107374182400,"available":107326312448,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641568,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46756,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810844,"statfs":{"total":107374182400,"available":107326304256,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641567,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59960,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797640,"statfs":{"total":107374182400,"available":107312783360,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:12:46.390 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.391 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.391 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.391 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.391 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.392 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.392 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.392 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.392 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.392 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.393 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.393 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.393 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.393 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.394 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.394 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.394 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.394 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.394 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.395 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.395 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.395 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.395 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.395 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.396 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.396 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.396 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.396 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.396 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.397 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.397 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.397 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.397 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.397 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.398 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.398 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.398 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.398 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.399 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.399 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.399 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.399 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.399 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.400 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.400 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.400 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.400 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.400 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.401 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.401 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.401 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.401 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.402 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.402 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.402 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.402 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.402 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.403 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.403 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.403 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.403 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.403 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.404 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.404 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-16T19:23:45.108082+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=45, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.404 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.404 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.405 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.405 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.405 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.405 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.406 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.406 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.406 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.406 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.407 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.408 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.408 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.408 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.408 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.409 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.409 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.409 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.409 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.409 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.410 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.410 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.410 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.410 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.411 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.411 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.411 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.411 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.411 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.412 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.412 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.412 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.412 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.413 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.413 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.413 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.413 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.413 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.414 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.414 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.414 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.414 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.415 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.415 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.415 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.415 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.416 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.416 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.416 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.416 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.416 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.417 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.417 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.417 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.417 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.418 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.418 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.418 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.418 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-16T19:24:46.405599+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.419 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.420 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.420 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.420 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.420 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.420 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.421 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.421 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.421 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.421 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.421 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.422 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.422 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.422 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.422 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.422 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.423 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.423 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.423 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.423 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.423 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.424 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.424 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:12:46.424 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T21:13:06.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:13:06.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:13:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:13:06.801 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3326,"stamp":"2024-04-16T21:13:05.369966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":482244,"num_omap_keys":2103,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32959,"ondisk_log_size":32959,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":305088,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":293878,"kb_avail":628840512,"statfs":{"total":644245094400,"available":643932684288,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":300931745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18165,"num_omap_keys":75,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005395"},"pg_stats":[{"pgid":"9.35","version":"52'41","reported_seq":316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.577077+0000","last_change":"2024-04-16T21:11:52.577152+0000","last_active":"2024-04-16T21:11:52.577077+0000","last_peered":"2024-04-16T21:11:52.577077+0000","last_clean":"2024-04-16T21:11:52.577077+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:11:52.577077+0000","last_undegraded":"2024-04-16T21:11:52.577077+0000","last_fullsized":"2024-04-16T21:11:52.577077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'41","last_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_deep_scrub":"52'41","last_deep_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.577012+0000","objects_scrubbed":13,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:52:59.327819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.243008+0000","last_change":"2024-04-16T21:12:46.243008+0000","last_active":"2024-04-16T21:12:46.243008+0000","last_peered":"2024-04-16T21:12:46.243008+0000","last_clean":"2024-04-16T21:12:46.243008+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:12:46.243008+0000","last_undegraded":"2024-04-16T21:12:46.243008+0000","last_fullsized":"2024-04-16T21:12:46.243008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.242914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.242914+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.242914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:23.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.901218+0000","last_change":"2024-04-16T21:12:53.901241+0000","last_active":"2024-04-16T21:12:53.901218+0000","last_peered":"2024-04-16T21:12:53.901218+0000","last_clean":"2024-04-16T21:12:53.901218+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:12:53.901218+0000","last_undegraded":"2024-04-16T21:12:53.901218+0000","last_fullsized":"2024-04-16T21:12:53.901218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.901152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.901152+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.901152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:18:02.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"52'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.722895+0000","last_change":"2024-04-16T21:11:56.722928+0000","last_active":"2024-04-16T21:11:56.722895+0000","last_peered":"2024-04-16T21:11:56.722895+0000","last_clean":"2024-04-16T21:11:56.722895+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:11:56.722895+0000","last_undegraded":"2024-04-16T21:11:56.722895+0000","last_fullsized":"2024-04-16T21:11:56.722895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.722838+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:40.375473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.098255+0000","last_change":"2024-04-16T21:12:55.098278+0000","last_active":"2024-04-16T21:12:55.098255+0000","last_peered":"2024-04-16T21:12:55.098255+0000","last_clean":"2024-04-16T21:12:55.098255+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:12:55.098255+0000","last_undegraded":"2024-04-16T21:12:55.098255+0000","last_fullsized":"2024-04-16T21:12:55.098255+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.098190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.098190+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.098190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:10.352974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.303685+0000","last_change":"2024-04-16T21:12:56.303709+0000","last_active":"2024-04-16T21:12:56.303685+0000","last_peered":"2024-04-16T21:12:56.303685+0000","last_clean":"2024-04-16T21:12:56.303685+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:12:56.303685+0000","last_undegraded":"2024-04-16T21:12:56.303685+0000","last_fullsized":"2024-04-16T21:12:56.303685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.303621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.303621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.303621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:54.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'22","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.702468+0000","last_change":"2024-04-16T21:12:14.702518+0000","last_active":"2024-04-16T21:12:14.702468+0000","last_peered":"2024-04-16T21:12:14.702468+0000","last_clean":"2024-04-16T21:12:14.702468+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:12:14.702468+0000","last_undegraded":"2024-04-16T21:12:14.702468+0000","last_fullsized":"2024-04-16T21:12:14.702468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.702406+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:42.760555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292035+0000","last_change":"2024-04-16T19:24:33.389040+0000","last_active":"2024-04-16T21:11:42.292035+0000","last_peered":"2024-04-16T21:11:42.292035+0000","last_clean":"2024-04-16T21:11:42.292035+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:11:42.292035+0000","last_undegraded":"2024-04-16T21:11:42.292035+0000","last_fullsized":"2024-04-16T21:11:42.292035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.701798+0000","last_change":"2024-04-16T21:12:32.701821+0000","last_active":"2024-04-16T21:12:32.701798+0000","last_peered":"2024-04-16T21:12:32.701798+0000","last_clean":"2024-04-16T21:12:32.701798+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:12:32.701798+0000","last_undegraded":"2024-04-16T21:12:32.701798+0000","last_fullsized":"2024-04-16T21:12:32.701798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.701734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:27.130470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'22","reported_seq":252,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.610592+0000","last_change":"2024-04-16T21:11:47.610645+0000","last_active":"2024-04-16T21:11:47.610592+0000","last_peered":"2024-04-16T21:11:47.610592+0000","last_clean":"2024-04-16T21:11:47.610592+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:11:47.610592+0000","last_undegraded":"2024-04-16T21:11:47.610592+0000","last_fullsized":"2024-04-16T21:11:47.610592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.610532+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:13.030892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.920425+0000","last_change":"2024-04-16T21:12:50.920425+0000","last_active":"2024-04-16T21:12:50.920425+0000","last_peered":"2024-04-16T21:12:50.920425+0000","last_clean":"2024-04-16T21:12:50.920425+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:12:50.920425+0000","last_undegraded":"2024-04-16T21:12:50.920425+0000","last_fullsized":"2024-04-16T21:12:50.920425+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.920340+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:03:25.179453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.372858+0000","last_change":"2024-04-16T21:12:49.372882+0000","last_active":"2024-04-16T21:12:49.372858+0000","last_peered":"2024-04-16T21:12:49.372858+0000","last_clean":"2024-04-16T21:12:49.372858+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:12:49.372858+0000","last_undegraded":"2024-04-16T21:12:49.372858+0000","last_fullsized":"2024-04-16T21:12:49.372858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.372787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.372787+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.372787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:51.965470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'12","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.719715+0000","last_change":"2024-04-16T21:11:58.719715+0000","last_active":"2024-04-16T21:11:58.719715+0000","last_peered":"2024-04-16T21:11:58.719715+0000","last_clean":"2024-04-16T21:11:58.719715+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:11:58.719715+0000","last_undegraded":"2024-04-16T21:11:58.719715+0000","last_fullsized":"2024-04-16T21:11:58.719715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.719610+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:50.501313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.307424+0000","last_change":"2024-04-16T21:12:55.307464+0000","last_active":"2024-04-16T21:12:55.307424+0000","last_peered":"2024-04-16T21:12:55.307424+0000","last_clean":"2024-04-16T21:12:55.307424+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:12:55.307424+0000","last_undegraded":"2024-04-16T21:12:55.307424+0000","last_fullsized":"2024-04-16T21:12:55.307424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.307348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.307348+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.307348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:10:07.841672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.797340+0000","last_change":"2024-04-16T21:12:59.797381+0000","last_active":"2024-04-16T21:12:59.797340+0000","last_peered":"2024-04-16T21:12:59.797340+0000","last_clean":"2024-04-16T21:12:59.797340+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:12:59.797340+0000","last_undegraded":"2024-04-16T21:12:59.797340+0000","last_fullsized":"2024-04-16T21:12:59.797340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.797260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.797260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.797260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:41.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'27","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.166108+0000","last_change":"2024-04-16T21:11:51.166108+0000","last_active":"2024-04-16T21:11:51.166108+0000","last_peered":"2024-04-16T21:11:51.166108+0000","last_clean":"2024-04-16T21:11:51.166108+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:11:51.166108+0000","last_undegraded":"2024-04-16T21:11:51.166108+0000","last_fullsized":"2024-04-16T21:11:51.166108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.165987+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:01.812779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560300+0000","last_change":"2024-04-16T19:24:33.383360+0000","last_active":"2024-04-16T21:11:44.560300+0000","last_peered":"2024-04-16T21:11:44.560300+0000","last_clean":"2024-04-16T21:11:44.560300+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:11:44.560300+0000","last_undegraded":"2024-04-16T21:11:44.560300+0000","last_fullsized":"2024-04-16T21:11:44.560300+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.679817+0000","last_change":"2024-04-16T21:12:34.679817+0000","last_active":"2024-04-16T21:12:34.679817+0000","last_peered":"2024-04-16T21:12:34.679817+0000","last_clean":"2024-04-16T21:12:34.679817+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:12:34.679817+0000","last_undegraded":"2024-04-16T21:12:34.679817+0000","last_fullsized":"2024-04-16T21:12:34.679817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.679697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:57.367334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'18","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.672517+0000","last_change":"2024-04-16T21:12:09.672592+0000","last_active":"2024-04-16T21:12:09.672517+0000","last_peered":"2024-04-16T21:12:09.672517+0000","last_clean":"2024-04-16T21:12:09.672517+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:12:09.672517+0000","last_undegraded":"2024-04-16T21:12:09.672517+0000","last_fullsized":"2024-04-16T21:12:09.672517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.672453+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.111035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.239995+0000","last_change":"2024-04-16T21:12:48.240036+0000","last_active":"2024-04-16T21:12:48.239995+0000","last_peered":"2024-04-16T21:12:48.239995+0000","last_clean":"2024-04-16T21:12:48.239995+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:12:48.239995+0000","last_undegraded":"2024-04-16T21:12:48.239995+0000","last_fullsized":"2024-04-16T21:12:48.239995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.239912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.239912+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.239912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:01.243261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.835059+0000","last_change":"2024-04-16T21:12:47.835059+0000","last_active":"2024-04-16T21:12:47.835059+0000","last_peered":"2024-04-16T21:12:47.835059+0000","last_clean":"2024-04-16T21:12:47.835059+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:12:47.835059+0000","last_undegraded":"2024-04-16T21:12:47.835059+0000","last_fullsized":"2024-04-16T21:12:47.835059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.834965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.834965+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.834965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:33.241934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'22","reported_seq":255,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.654830+0000","last_change":"2024-04-16T21:11:56.654830+0000","last_active":"2024-04-16T21:11:56.654830+0000","last_peered":"2024-04-16T21:11:56.654830+0000","last_clean":"2024-04-16T21:11:56.654830+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:11:56.654830+0000","last_undegraded":"2024-04-16T21:11:56.654830+0000","last_fullsized":"2024-04-16T21:11:56.654830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.654666+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.077668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.751421+0000","last_change":"2024-04-16T21:12:47.751466+0000","last_active":"2024-04-16T21:12:47.751421+0000","last_peered":"2024-04-16T21:12:47.751421+0000","last_clean":"2024-04-16T21:12:47.751421+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:12:47.751421+0000","last_undegraded":"2024-04-16T21:12:47.751421+0000","last_fullsized":"2024-04-16T21:12:47.751421+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.751343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.751343+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.751343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:36.357636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.815463+0000","last_change":"2024-04-16T21:12:39.815463+0000","last_active":"2024-04-16T21:12:39.815463+0000","last_peered":"2024-04-16T21:12:39.815463+0000","last_clean":"2024-04-16T21:12:39.815463+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:12:39.815463+0000","last_undegraded":"2024-04-16T21:12:39.815463+0000","last_fullsized":"2024-04-16T21:12:39.815463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.815374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:11.502076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'12","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.701548+0000","last_change":"2024-04-16T21:12:00.701594+0000","last_active":"2024-04-16T21:12:00.701548+0000","last_peered":"2024-04-16T21:12:00.701548+0000","last_clean":"2024-04-16T21:12:00.701548+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:12:00.701548+0000","last_undegraded":"2024-04-16T21:12:00.701548+0000","last_fullsized":"2024-04-16T21:12:00.701548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.701488+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:22.203073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.707007+0000","last_change":"2024-04-16T21:12:38.707030+0000","last_active":"2024-04-16T21:12:38.707007+0000","last_peered":"2024-04-16T21:12:38.707007+0000","last_clean":"2024-04-16T21:12:38.707007+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:12:38.707007+0000","last_undegraded":"2024-04-16T21:12:38.707007+0000","last_fullsized":"2024-04-16T21:12:38.707007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.706933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:10.393634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:14.390309+0000","last_change":"2024-04-16T19:24:33.385848+0000","last_active":"2024-04-16T21:11:14.390309+0000","last_peered":"2024-04-16T21:11:14.390309+0000","last_clean":"2024-04-16T21:11:14.390309+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:11:14.390309+0000","last_undegraded":"2024-04-16T21:11:14.390309+0000","last_fullsized":"2024-04-16T21:11:14.390309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'12","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.436184+0000","last_change":"2024-04-16T21:11:55.436228+0000","last_active":"2024-04-16T21:11:55.436184+0000","last_peered":"2024-04-16T21:11:55.436184+0000","last_clean":"2024-04-16T21:11:55.436184+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:11:55.436184+0000","last_undegraded":"2024-04-16T21:11:55.436184+0000","last_fullsized":"2024-04-16T21:11:55.436184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.436126+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:58.763911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.914098+0000","last_change":"2024-04-16T21:12:16.914127+0000","last_active":"2024-04-16T21:12:16.914098+0000","last_peered":"2024-04-16T21:12:16.914098+0000","last_clean":"2024-04-16T21:12:16.914098+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:12:16.914098+0000","last_undegraded":"2024-04-16T21:12:16.914098+0000","last_fullsized":"2024-04-16T21:12:16.914098+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.914034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:05.877583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.324458+0000","last_change":"2024-04-16T21:12:53.324499+0000","last_active":"2024-04-16T21:12:53.324458+0000","last_peered":"2024-04-16T21:12:53.324458+0000","last_clean":"2024-04-16T21:12:53.324458+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:12:53.324458+0000","last_undegraded":"2024-04-16T21:12:53.324458+0000","last_fullsized":"2024-04-16T21:12:53.324458+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.324385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.324385+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.324385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:34.628013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'10","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.882687+0000","last_change":"2024-04-16T21:11:51.882767+0000","last_active":"2024-04-16T21:11:51.882687+0000","last_peered":"2024-04-16T21:11:51.882687+0000","last_clean":"2024-04-16T21:11:51.882687+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:11:51.882687+0000","last_undegraded":"2024-04-16T21:11:51.882687+0000","last_fullsized":"2024-04-16T21:11:51.882687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.882609+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.077377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.890648+0000","last_change":"2024-04-16T21:12:49.890672+0000","last_active":"2024-04-16T21:12:49.890648+0000","last_peered":"2024-04-16T21:12:49.890648+0000","last_clean":"2024-04-16T21:12:49.890648+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:12:49.890648+0000","last_undegraded":"2024-04-16T21:12:49.890648+0000","last_fullsized":"2024-04-16T21:12:49.890648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.890587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.890587+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.890587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:17.432217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.828110+0000","last_change":"2024-04-16T21:12:58.828130+0000","last_active":"2024-04-16T21:12:58.828110+0000","last_peered":"2024-04-16T21:12:58.828110+0000","last_clean":"2024-04-16T21:12:58.828110+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:12:58.828110+0000","last_undegraded":"2024-04-16T21:12:58.828110+0000","last_fullsized":"2024-04-16T21:12:58.828110+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.828052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.828052+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.828052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:01.420735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'20","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.881532+0000","last_change":"2024-04-16T21:12:07.881609+0000","last_active":"2024-04-16T21:12:07.881532+0000","last_peered":"2024-04-16T21:12:07.881532+0000","last_clean":"2024-04-16T21:12:07.881532+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:12:07.881532+0000","last_undegraded":"2024-04-16T21:12:07.881532+0000","last_fullsized":"2024-04-16T21:12:07.881532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.881467+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.576312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.819466+0000","last_change":"2024-04-16T21:12:33.819489+0000","last_active":"2024-04-16T21:12:33.819466+0000","last_peered":"2024-04-16T21:12:33.819466+0000","last_clean":"2024-04-16T21:12:33.819466+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:12:33.819466+0000","last_undegraded":"2024-04-16T21:12:33.819466+0000","last_fullsized":"2024-04-16T21:12:33.819466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.819396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:28.248131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.713176+0000","last_change":"2024-04-16T21:12:51.713200+0000","last_active":"2024-04-16T21:12:51.713176+0000","last_peered":"2024-04-16T21:12:51.713176+0000","last_clean":"2024-04-16T21:12:51.713176+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:12:51.713176+0000","last_undegraded":"2024-04-16T21:12:51.713176+0000","last_fullsized":"2024-04-16T21:12:51.713176+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.713107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.713107+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.713107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:38:57.545317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'16","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.428185+0000","last_change":"2024-04-16T21:11:47.428185+0000","last_active":"2024-04-16T21:11:47.428185+0000","last_peered":"2024-04-16T21:11:47.428185+0000","last_clean":"2024-04-16T21:11:47.428185+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:11:47.428185+0000","last_undegraded":"2024-04-16T21:11:47.428185+0000","last_fullsized":"2024-04-16T21:11:47.428185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.428059+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:23.278965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.812506+0000","last_change":"2024-04-16T21:12:37.812549+0000","last_active":"2024-04-16T21:12:37.812506+0000","last_peered":"2024-04-16T21:12:37.812506+0000","last_clean":"2024-04-16T21:12:37.812506+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:12:37.812506+0000","last_undegraded":"2024-04-16T21:12:37.812506+0000","last_fullsized":"2024-04-16T21:12:37.812506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.812425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:29.486114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.427331+0000","last_change":"2024-04-16T21:12:57.427374+0000","last_active":"2024-04-16T21:12:57.427331+0000","last_peered":"2024-04-16T21:12:57.427331+0000","last_clean":"2024-04-16T21:12:57.427331+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:12:57.427331+0000","last_undegraded":"2024-04-16T21:12:57.427331+0000","last_fullsized":"2024-04-16T21:12:57.427331+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.427251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.427251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.427251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:09.881504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'22","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.947975+0000","last_change":"2024-04-16T21:12:11.947975+0000","last_active":"2024-04-16T21:12:11.947975+0000","last_peered":"2024-04-16T21:12:11.947975+0000","last_clean":"2024-04-16T21:12:11.947975+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:12:11.947975+0000","last_undegraded":"2024-04-16T21:12:11.947975+0000","last_fullsized":"2024-04-16T21:12:11.947975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.947824+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:04.586823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.717931+0000","last_change":"2024-04-16T21:12:46.717970+0000","last_active":"2024-04-16T21:12:46.717931+0000","last_peered":"2024-04-16T21:12:46.717931+0000","last_clean":"2024-04-16T21:12:46.717931+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:12:46.717931+0000","last_undegraded":"2024-04-16T21:12:46.717931+0000","last_fullsized":"2024-04-16T21:12:46.717931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.717861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.717861+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.717861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:35.324154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.518893+0000","last_change":"2024-04-16T21:13:01.518917+0000","last_active":"2024-04-16T21:13:01.518893+0000","last_peered":"2024-04-16T21:13:01.518893+0000","last_clean":"2024-04-16T21:13:01.518893+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:13:01.518893+0000","last_undegraded":"2024-04-16T21:13:01.518893+0000","last_fullsized":"2024-04-16T21:13:01.518893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.518831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.518831+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.518831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:13:52.695386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":58,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744211+0000","last_change":"2024-04-16T19:47:33.986549+0000","last_active":"2024-04-16T21:11:42.744211+0000","last_peered":"2024-04-16T21:11:42.744211+0000","last_clean":"2024-04-16T21:11:42.744211+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:11:42.744211+0000","last_undegraded":"2024-04-16T21:11:42.744211+0000","last_fullsized":"2024-04-16T21:11:42.744211+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'105","reported_seq":1722,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651474+0000","last_change":"2024-04-16T19:25:49.525638+0000","last_active":"2024-04-16T21:11:43.651474+0000","last_peered":"2024-04-16T21:11:43.651474+0000","last_clean":"2024-04-16T21:11:43.651474+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:11:43.651474+0000","last_undegraded":"2024-04-16T21:11:43.651474+0000","last_fullsized":"2024-04-16T21:11:43.651474+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.395179+0000","last_change":"2024-04-16T21:12:15.395220+0000","last_active":"2024-04-16T21:12:15.395179+0000","last_peered":"2024-04-16T21:12:15.395179+0000","last_clean":"2024-04-16T21:12:15.395179+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:12:15.395179+0000","last_undegraded":"2024-04-16T21:12:15.395179+0000","last_fullsized":"2024-04-16T21:12:15.395179+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.395104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:09.823839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'8","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.430800+0000","last_change":"2024-04-16T21:12:17.430800+0000","last_active":"2024-04-16T21:12:17.430800+0000","last_peered":"2024-04-16T21:12:17.430800+0000","last_clean":"2024-04-16T21:12:17.430800+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:12:17.430800+0000","last_undegraded":"2024-04-16T21:12:17.430800+0000","last_fullsized":"2024-04-16T21:12:17.430800+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.430693+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:25.924925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.738465+0000","last_change":"2024-04-16T21:11:45.738465+0000","last_active":"2024-04-16T21:11:45.738465+0000","last_peered":"2024-04-16T21:11:45.738465+0000","last_clean":"2024-04-16T21:11:45.738465+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:11:45.738465+0000","last_undegraded":"2024-04-16T21:11:45.738465+0000","last_fullsized":"2024-04-16T21:11:45.738465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.738344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:26.908264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'223","reported_seq":477,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.490168+0000","last_change":"2024-04-16T21:11:58.490168+0000","last_active":"2024-04-16T21:11:58.490168+0000","last_peered":"2024-04-16T21:11:58.490168+0000","last_clean":"2024-04-16T21:11:58.490168+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:11:58.490168+0000","last_undegraded":"2024-04-16T21:11:58.490168+0000","last_fullsized":"2024-04-16T21:11:58.490168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'223","last_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_deep_scrub":"52'223","last_deep_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.490053+0000","objects_scrubbed":20,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:48.708953+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1971,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'29","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.926237+0000","last_change":"2024-04-16T21:11:50.926284+0000","last_active":"2024-04-16T21:11:50.926237+0000","last_peered":"2024-04-16T21:11:50.926237+0000","last_clean":"2024-04-16T21:11:50.926237+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:11:50.926237+0000","last_undegraded":"2024-04-16T21:11:50.926237+0000","last_fullsized":"2024-04-16T21:11:50.926237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'29","last_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_deep_scrub":"52'29","last_deep_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.926173+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:17.509632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.689691+0000","last_change":"2024-04-16T21:12:19.689715+0000","last_active":"2024-04-16T21:12:19.689691+0000","last_peered":"2024-04-16T21:12:19.689691+0000","last_clean":"2024-04-16T21:12:19.689691+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:12:19.689691+0000","last_undegraded":"2024-04-16T21:12:19.689691+0000","last_fullsized":"2024-04-16T21:12:19.689691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.689621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.705707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.031454+0000","last_change":"2024-04-16T21:12:40.031477+0000","last_active":"2024-04-16T21:12:40.031454+0000","last_peered":"2024-04-16T21:12:40.031454+0000","last_clean":"2024-04-16T21:12:40.031454+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:12:40.031454+0000","last_undegraded":"2024-04-16T21:12:40.031454+0000","last_fullsized":"2024-04-16T21:12:40.031454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.031394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:26.329198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.297056+0000","last_change":"2024-04-16T21:12:57.297097+0000","last_active":"2024-04-16T21:12:57.297056+0000","last_peered":"2024-04-16T21:12:57.297056+0000","last_clean":"2024-04-16T21:12:57.297056+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:12:57.297056+0000","last_undegraded":"2024-04-16T21:12:57.297056+0000","last_fullsized":"2024-04-16T21:12:57.297056+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.296983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.296983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.296983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:10.967137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'6","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.901446+0000","last_change":"2024-04-16T21:12:42.901472+0000","last_active":"2024-04-16T21:12:42.901446+0000","last_peered":"2024-04-16T21:12:42.901446+0000","last_clean":"2024-04-16T21:12:42.901446+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:12:42.901446+0000","last_undegraded":"2024-04-16T21:12:42.901446+0000","last_fullsized":"2024-04-16T21:12:42.901446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.901389+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:59:18.553914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.351884+0000","last_change":"2024-04-16T21:12:33.351925+0000","last_active":"2024-04-16T21:12:33.351884+0000","last_peered":"2024-04-16T21:12:33.351884+0000","last_clean":"2024-04-16T21:12:33.351884+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:12:33.351884+0000","last_undegraded":"2024-04-16T21:12:33.351884+0000","last_fullsized":"2024-04-16T21:12:33.351884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.351821+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:30.327261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.757836+0000","last_change":"2024-04-16T21:12:30.757878+0000","last_active":"2024-04-16T21:12:30.757836+0000","last_peered":"2024-04-16T21:12:30.757836+0000","last_clean":"2024-04-16T21:12:30.757836+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:12:30.757836+0000","last_undegraded":"2024-04-16T21:12:30.757836+0000","last_fullsized":"2024-04-16T21:12:30.757836+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.757760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:42.360430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.788930+0000","last_change":"2024-04-16T21:12:55.788953+0000","last_active":"2024-04-16T21:12:55.788930+0000","last_peered":"2024-04-16T21:12:55.788930+0000","last_clean":"2024-04-16T21:12:55.788930+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:12:55.788930+0000","last_undegraded":"2024-04-16T21:12:55.788930+0000","last_fullsized":"2024-04-16T21:12:55.788930+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.788863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.788863+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.788863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:43.044039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'154","reported_seq":1838,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561636+0000","last_change":"2024-04-16T19:26:41.581437+0000","last_active":"2024-04-16T21:11:44.561636+0000","last_peered":"2024-04-16T21:11:44.561636+0000","last_clean":"2024-04-16T21:11:44.561636+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:11:44.561636+0000","last_undegraded":"2024-04-16T21:11:44.561636+0000","last_fullsized":"2024-04-16T21:11:44.561636+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.873357+0000","last_change":"2024-04-16T21:12:15.873380+0000","last_active":"2024-04-16T21:12:15.873357+0000","last_peered":"2024-04-16T21:12:15.873357+0000","last_clean":"2024-04-16T21:12:15.873357+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:12:15.873357+0000","last_undegraded":"2024-04-16T21:12:15.873357+0000","last_fullsized":"2024-04-16T21:12:15.873357+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.873260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:01.509024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'8","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.369110+0000","last_change":"2024-04-16T21:12:14.369164+0000","last_active":"2024-04-16T21:12:14.369110+0000","last_peered":"2024-04-16T21:12:14.369110+0000","last_clean":"2024-04-16T21:12:14.369110+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:12:14.369110+0000","last_undegraded":"2024-04-16T21:12:14.369110+0000","last_fullsized":"2024-04-16T21:12:14.369110+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.369033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:29.838406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.686990+0000","last_change":"2024-04-16T21:11:57.686990+0000","last_active":"2024-04-16T21:11:57.686990+0000","last_peered":"2024-04-16T21:11:57.686990+0000","last_clean":"2024-04-16T21:11:57.686990+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:11:57.686990+0000","last_undegraded":"2024-04-16T21:11:57.686990+0000","last_fullsized":"2024-04-16T21:11:57.686990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.686888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:39.802344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'264","reported_seq":1009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.984786+0000","last_change":"2024-04-16T21:11:43.984786+0000","last_active":"2024-04-16T21:11:43.984786+0000","last_peered":"2024-04-16T21:11:43.984786+0000","last_clean":"2024-04-16T21:11:43.984786+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:11:43.984786+0000","last_undegraded":"2024-04-16T21:11:43.984786+0000","last_fullsized":"2024-04-16T21:11:43.984786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.984688+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:21.448843+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6375,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'10","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.702465+0000","last_change":"2024-04-16T21:11:54.702515+0000","last_active":"2024-04-16T21:11:54.702465+0000","last_peered":"2024-04-16T21:11:54.702465+0000","last_clean":"2024-04-16T21:11:54.702465+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:11:54.702465+0000","last_undegraded":"2024-04-16T21:11:54.702465+0000","last_fullsized":"2024-04-16T21:11:54.702465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.702406+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.795058+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.815226+0000","last_change":"2024-04-16T21:12:10.815250+0000","last_active":"2024-04-16T21:12:10.815226+0000","last_peered":"2024-04-16T21:12:10.815226+0000","last_clean":"2024-04-16T21:12:10.815226+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:12:10.815226+0000","last_undegraded":"2024-04-16T21:12:10.815226+0000","last_fullsized":"2024-04-16T21:12:10.815226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.815114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:27:54.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.821170+0000","last_change":"2024-04-16T21:12:55.821191+0000","last_active":"2024-04-16T21:12:55.821170+0000","last_peered":"2024-04-16T21:12:55.821170+0000","last_clean":"2024-04-16T21:12:55.821170+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:12:55.821170+0000","last_undegraded":"2024-04-16T21:12:55.821170+0000","last_fullsized":"2024-04-16T21:12:55.821170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.821108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.821108+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.821108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:44.476963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.491126+0000","last_change":"2024-04-16T21:12:55.491150+0000","last_active":"2024-04-16T21:12:55.491126+0000","last_peered":"2024-04-16T21:12:55.491126+0000","last_clean":"2024-04-16T21:12:55.491126+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:12:55.491126+0000","last_undegraded":"2024-04-16T21:12:55.491126+0000","last_fullsized":"2024-04-16T21:12:55.491126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.491061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.491061+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.491061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:12.884360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"47'3","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.292295+0000","last_change":"2024-04-16T19:24:45.413272+0000","last_active":"2024-04-16T21:11:42.292295+0000","last_peered":"2024-04-16T21:11:42.292295+0000","last_clean":"2024-04-16T21:11:42.292295+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:11:42.292295+0000","last_undegraded":"2024-04-16T21:11:42.292295+0000","last_fullsized":"2024-04-16T21:11:42.292295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'69","reported_seq":219,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.619780+0000","last_change":"2024-04-16T21:12:28.619811+0000","last_active":"2024-04-16T21:12:28.619780+0000","last_peered":"2024-04-16T21:12:28.619780+0000","last_clean":"2024-04-16T21:12:28.619780+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:12:28.619780+0000","last_undegraded":"2024-04-16T21:12:28.619780+0000","last_fullsized":"2024-04-16T21:12:28.619780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'69","last_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_deep_scrub":"52'69","last_deep_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.619713+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:56.443540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.604337+0000","last_change":"2024-04-16T21:12:23.604359+0000","last_active":"2024-04-16T21:12:23.604337+0000","last_peered":"2024-04-16T21:12:23.604337+0000","last_clean":"2024-04-16T21:12:23.604337+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:12:23.604337+0000","last_undegraded":"2024-04-16T21:12:23.604337+0000","last_fullsized":"2024-04-16T21:12:23.604337+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.604274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:10.724532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.836520+0000","last_change":"2024-04-16T21:13:02.836541+0000","last_active":"2024-04-16T21:13:02.836520+0000","last_peered":"2024-04-16T21:13:02.836520+0000","last_clean":"2024-04-16T21:13:02.836520+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:13:02.836520+0000","last_undegraded":"2024-04-16T21:13:02.836520+0000","last_fullsized":"2024-04-16T21:13:02.836520+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.836457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.836457+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.836457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:12.185234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'183","reported_seq":1934,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.306613+0000","last_change":"2024-04-16T21:12:38.306641+0000","last_active":"2024-04-16T21:12:38.306613+0000","last_peered":"2024-04-16T21:12:38.306613+0000","last_clean":"2024-04-16T21:12:38.306613+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:12:38.306613+0000","last_undegraded":"2024-04-16T21:12:38.306613+0000","last_fullsized":"2024-04-16T21:12:38.306613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.306555+0000","objects_scrubbed":52,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:07.296213+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":43,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.746629+0000","last_change":"2024-04-16T21:12:00.746651+0000","last_active":"2024-04-16T21:12:00.746629+0000","last_peered":"2024-04-16T21:12:00.746629+0000","last_clean":"2024-04-16T21:12:00.746629+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:12:00.746629+0000","last_undegraded":"2024-04-16T21:12:00.746629+0000","last_fullsized":"2024-04-16T21:12:00.746629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.746571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:24:45.291359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'25","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.693813+0000","last_change":"2024-04-16T21:11:57.693880+0000","last_active":"2024-04-16T21:11:57.693813+0000","last_peered":"2024-04-16T21:11:57.693813+0000","last_clean":"2024-04-16T21:11:57.693813+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:11:57.693813+0000","last_undegraded":"2024-04-16T21:11:57.693813+0000","last_fullsized":"2024-04-16T21:11:57.693813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'25","last_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_deep_scrub":"52'25","last_deep_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.693743+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.739736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.385829+0000","last_change":"2024-04-16T21:12:04.385854+0000","last_active":"2024-04-16T21:12:04.385829+0000","last_peered":"2024-04-16T21:12:04.385829+0000","last_clean":"2024-04-16T21:12:04.385829+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:12:04.385829+0000","last_undegraded":"2024-04-16T21:12:04.385829+0000","last_fullsized":"2024-04-16T21:12:04.385829+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.385746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:20.771679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'716","reported_seq":41938,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.996557+0000","last_change":"2024-04-16T21:11:45.996631+0000","last_active":"2024-04-16T21:11:45.996557+0000","last_peered":"2024-04-16T21:11:45.996557+0000","last_clean":"2024-04-16T21:11:45.996557+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:11:45.996557+0000","last_undegraded":"2024-04-16T21:11:45.996557+0000","last_fullsized":"2024-04-16T21:11:45.996557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'716","last_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_deep_scrub":"52'716","last_deep_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.996464+0000","objects_scrubbed":20,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:40.351620+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'35","reported_seq":324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.105372+0000","last_change":"2024-04-16T21:11:44.105372+0000","last_active":"2024-04-16T21:11:44.105372+0000","last_peered":"2024-04-16T21:11:44.105372+0000","last_clean":"2024-04-16T21:11:44.105372+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:11:44.105372+0000","last_undegraded":"2024-04-16T21:11:44.105372+0000","last_fullsized":"2024-04-16T21:11:44.105372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'35","last_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_deep_scrub":"52'35","last_deep_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.105287+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:44.283094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.409254+0000","last_change":"2024-04-16T21:12:03.409254+0000","last_active":"2024-04-16T21:12:03.409254+0000","last_peered":"2024-04-16T21:12:03.409254+0000","last_clean":"2024-04-16T21:12:03.409254+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:12:03.409254+0000","last_undegraded":"2024-04-16T21:12:03.409254+0000","last_fullsized":"2024-04-16T21:12:03.409254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.409137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:49.655116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.325581+0000","last_change":"2024-04-16T21:12:45.325602+0000","last_active":"2024-04-16T21:12:45.325581+0000","last_peered":"2024-04-16T21:12:45.325581+0000","last_clean":"2024-04-16T21:12:45.325581+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:12:45.325581+0000","last_undegraded":"2024-04-16T21:12:45.325581+0000","last_fullsized":"2024-04-16T21:12:45.325581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.325518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.325518+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.325518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:58.328868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649938+0000","last_change":"2024-04-16T19:24:33.383251+0000","last_active":"2024-04-16T21:11:43.649938+0000","last_peered":"2024-04-16T21:11:43.649938+0000","last_clean":"2024-04-16T21:11:43.649938+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:11:43.649938+0000","last_undegraded":"2024-04-16T21:11:43.649938+0000","last_fullsized":"2024-04-16T21:11:43.649938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'6","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.842517+0000","last_change":"2024-04-16T21:12:38.842543+0000","last_active":"2024-04-16T21:12:38.842517+0000","last_peered":"2024-04-16T21:12:38.842517+0000","last_clean":"2024-04-16T21:12:38.842517+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:12:38.842517+0000","last_undegraded":"2024-04-16T21:12:38.842517+0000","last_fullsized":"2024-04-16T21:12:38.842517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.842450+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:47.336683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'115","reported_seq":289,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.884236+0000","last_change":"2024-04-16T21:12:25.884267+0000","last_active":"2024-04-16T21:12:25.884236+0000","last_peered":"2024-04-16T21:12:25.884236+0000","last_clean":"2024-04-16T21:12:25.884236+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:12:25.884236+0000","last_undegraded":"2024-04-16T21:12:25.884236+0000","last_fullsized":"2024-04-16T21:12:25.884236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.884183+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.270116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'13","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.370369+0000","last_change":"2024-04-16T21:12:13.370396+0000","last_active":"2024-04-16T21:12:13.370369+0000","last_peered":"2024-04-16T21:12:13.370369+0000","last_clean":"2024-04-16T21:12:13.370369+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:12:13.370369+0000","last_undegraded":"2024-04-16T21:12:13.370369+0000","last_fullsized":"2024-04-16T21:12:13.370369+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.370308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:47.137634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.087222+0000","last_change":"2024-04-16T21:13:04.087271+0000","last_active":"2024-04-16T21:13:04.087222+0000","last_peered":"2024-04-16T21:13:04.087222+0000","last_clean":"2024-04-16T21:13:04.087222+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:13:04.087222+0000","last_undegraded":"2024-04-16T21:13:04.087222+0000","last_fullsized":"2024-04-16T21:13:04.087222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'9","last_scrub_stamp":"2024-04-16T21:13:04.087144+0000","last_deep_scrub":"49'9","last_deep_scrub_stamp":"2024-04-16T21:13:04.087144+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.087144+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:06.679827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'132","reported_seq":1473,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:49.390635+0000","last_change":"2024-04-16T19:25:49.522383+0000","last_active":"2024-04-16T21:10:49.390635+0000","last_peered":"2024-04-16T21:10:49.390635+0000","last_clean":"2024-04-16T21:10:49.390635+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:10:49.390635+0000","last_undegraded":"2024-04-16T21:10:49.390635+0000","last_fullsized":"2024-04-16T21:10:49.390635+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.703764+0000","last_change":"2024-04-16T21:12:02.703764+0000","last_active":"2024-04-16T21:12:02.703764+0000","last_peered":"2024-04-16T21:12:02.703764+0000","last_clean":"2024-04-16T21:12:02.703764+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:12:02.703764+0000","last_undegraded":"2024-04-16T21:12:02.703764+0000","last_fullsized":"2024-04-16T21:12:02.703764+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.703681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:59.311639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'13","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.372480+0000","last_change":"2024-04-16T21:12:18.372514+0000","last_active":"2024-04-16T21:12:18.372480+0000","last_peered":"2024-04-16T21:12:18.372480+0000","last_clean":"2024-04-16T21:12:18.372480+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:12:18.372480+0000","last_undegraded":"2024-04-16T21:12:18.372480+0000","last_fullsized":"2024-04-16T21:12:18.372480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.372404+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:27.934072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.821906+0000","last_change":"2024-04-16T21:11:59.821929+0000","last_active":"2024-04-16T21:11:59.821906+0000","last_peered":"2024-04-16T21:11:59.821906+0000","last_clean":"2024-04-16T21:11:59.821906+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:11:59.821906+0000","last_undegraded":"2024-04-16T21:11:59.821906+0000","last_fullsized":"2024-04-16T21:11:59.821906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.821840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.493143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'366","reported_seq":690,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.574732+0000","last_change":"2024-04-16T21:11:46.574732+0000","last_active":"2024-04-16T21:11:46.574732+0000","last_peered":"2024-04-16T21:11:46.574732+0000","last_clean":"2024-04-16T21:11:46.574732+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:11:46.574732+0000","last_undegraded":"2024-04-16T21:11:46.574732+0000","last_fullsized":"2024-04-16T21:11:46.574732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'366","last_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_deep_scrub":"52'366","last_deep_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.574641+0000","objects_scrubbed":27,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:24.475352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'12","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.580812+0000","last_change":"2024-04-16T21:11:44.580812+0000","last_active":"2024-04-16T21:11:44.580812+0000","last_peered":"2024-04-16T21:11:44.580812+0000","last_clean":"2024-04-16T21:11:44.580812+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:11:44.580812+0000","last_undegraded":"2024-04-16T21:11:44.580812+0000","last_fullsized":"2024-04-16T21:11:44.580812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.580695+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:57.775463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.671998+0000","last_change":"2024-04-16T21:12:15.671998+0000","last_active":"2024-04-16T21:12:15.671998+0000","last_peered":"2024-04-16T21:12:15.671998+0000","last_clean":"2024-04-16T21:12:15.671998+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:12:15.671998+0000","last_undegraded":"2024-04-16T21:12:15.671998+0000","last_fullsized":"2024-04-16T21:12:15.671998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.671910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:43.983331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.465807+0000","last_change":"2024-04-16T21:12:52.465832+0000","last_active":"2024-04-16T21:12:52.465807+0000","last_peered":"2024-04-16T21:12:52.465807+0000","last_clean":"2024-04-16T21:12:52.465807+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:12:52.465807+0000","last_undegraded":"2024-04-16T21:12:52.465807+0000","last_fullsized":"2024-04-16T21:12:52.465807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.465728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.465728+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.465728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:10:05.000052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560171+0000","last_change":"2024-04-16T19:24:33.385032+0000","last_active":"2024-04-16T21:11:44.560171+0000","last_peered":"2024-04-16T21:11:44.560171+0000","last_clean":"2024-04-16T21:11:44.560171+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:11:44.560171+0000","last_undegraded":"2024-04-16T21:11:44.560171+0000","last_fullsized":"2024-04-16T21:11:44.560171+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.239994+0000","last_change":"2024-04-16T21:12:38.240020+0000","last_active":"2024-04-16T21:12:38.239994+0000","last_peered":"2024-04-16T21:12:38.239994+0000","last_clean":"2024-04-16T21:12:38.239994+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:12:38.239994+0000","last_undegraded":"2024-04-16T21:12:38.239994+0000","last_fullsized":"2024-04-16T21:12:38.239994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.239920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:04.921574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.327857+0000","last_change":"2024-04-16T21:12:33.327890+0000","last_active":"2024-04-16T21:12:33.327857+0000","last_peered":"2024-04-16T21:12:33.327857+0000","last_clean":"2024-04-16T21:12:33.327857+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:12:33.327857+0000","last_undegraded":"2024-04-16T21:12:33.327857+0000","last_fullsized":"2024-04-16T21:12:33.327857+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.327792+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:56.594542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'13","reported_seq":1320,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.937025+0000","last_change":"2024-04-16T21:12:24.937025+0000","last_active":"2024-04-16T21:12:24.937025+0000","last_peered":"2024-04-16T21:12:24.937025+0000","last_clean":"2024-04-16T21:12:24.937025+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:12:24.937025+0000","last_undegraded":"2024-04-16T21:12:24.937025+0000","last_fullsized":"2024-04-16T21:12:24.937025+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.936891+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:58.704216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.816068+0000","last_change":"2024-04-16T21:12:58.816068+0000","last_active":"2024-04-16T21:12:58.816068+0000","last_peered":"2024-04-16T21:12:58.816068+0000","last_clean":"2024-04-16T21:12:58.816068+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:12:58.816068+0000","last_undegraded":"2024-04-16T21:12:58.816068+0000","last_fullsized":"2024-04-16T21:12:58.816068+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.815981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.815981+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.815981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:44.222950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'188","reported_seq":2057,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.351576+0000","last_change":"2024-04-16T21:12:32.351645+0000","last_active":"2024-04-16T21:12:32.351576+0000","last_peered":"2024-04-16T21:12:32.351576+0000","last_clean":"2024-04-16T21:12:32.351576+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:12:32.351576+0000","last_undegraded":"2024-04-16T21:12:32.351576+0000","last_fullsized":"2024-04-16T21:12:32.351576+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'188","last_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_deep_scrub":"52'188","last_deep_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.351505+0000","objects_scrubbed":40,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:48.150801+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12543,"num_omap_keys":51,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.379178+0000","last_change":"2024-04-16T21:12:16.379217+0000","last_active":"2024-04-16T21:12:16.379178+0000","last_peered":"2024-04-16T21:12:16.379178+0000","last_clean":"2024-04-16T21:12:16.379178+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:12:16.379178+0000","last_undegraded":"2024-04-16T21:12:16.379178+0000","last_fullsized":"2024-04-16T21:12:16.379178+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.379105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:54.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'37","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.663863+0000","last_change":"2024-04-16T21:12:06.663901+0000","last_active":"2024-04-16T21:12:06.663863+0000","last_peered":"2024-04-16T21:12:06.663863+0000","last_clean":"2024-04-16T21:12:06.663863+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:12:06.663863+0000","last_undegraded":"2024-04-16T21:12:06.663863+0000","last_fullsized":"2024-04-16T21:12:06.663863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'37","last_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_deep_scrub":"52'37","last_deep_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.663802+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:20.415399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":158,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.716132+0000","last_change":"2024-04-16T21:12:07.716153+0000","last_active":"2024-04-16T21:12:07.716132+0000","last_peered":"2024-04-16T21:12:07.716132+0000","last_clean":"2024-04-16T21:12:07.716132+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:12:07.716132+0000","last_undegraded":"2024-04-16T21:12:07.716132+0000","last_fullsized":"2024-04-16T21:12:07.716132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.716070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.751360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1762","reported_seq":4632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.707742+0000","last_change":"2024-04-16T21:11:51.707808+0000","last_active":"2024-04-16T21:11:51.707742+0000","last_peered":"2024-04-16T21:11:51.707742+0000","last_clean":"2024-04-16T21:11:51.707742+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:11:51.707742+0000","last_undegraded":"2024-04-16T21:11:51.707742+0000","last_fullsized":"2024-04-16T21:11:51.707742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1762","last_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_deep_scrub":"52'1762","last_deep_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.707684+0000","objects_scrubbed":22,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:25.851168+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":89480,"num_omap_keys":462,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'27","reported_seq":288,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.189093+0000","last_change":"2024-04-16T21:11:48.189147+0000","last_active":"2024-04-16T21:11:48.189093+0000","last_peered":"2024-04-16T21:11:48.189093+0000","last_clean":"2024-04-16T21:11:48.189093+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:11:48.189093+0000","last_undegraded":"2024-04-16T21:11:48.189093+0000","last_fullsized":"2024-04-16T21:11:48.189093+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.189029+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:30.304485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.281331+0000","last_change":"2024-04-16T21:12:54.281331+0000","last_active":"2024-04-16T21:12:54.281331+0000","last_peered":"2024-04-16T21:12:54.281331+0000","last_clean":"2024-04-16T21:12:54.281331+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:12:54.281331+0000","last_undegraded":"2024-04-16T21:12:54.281331+0000","last_fullsized":"2024-04-16T21:12:54.281331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.281207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.281207+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.281207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:10.616263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560158+0000","last_change":"2024-04-16T19:24:33.386018+0000","last_active":"2024-04-16T21:11:44.560158+0000","last_peered":"2024-04-16T21:11:44.560158+0000","last_clean":"2024-04-16T21:11:44.560158+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:11:44.560158+0000","last_undegraded":"2024-04-16T21:11:44.560158+0000","last_fullsized":"2024-04-16T21:11:44.560158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.849021+0000","last_change":"2024-04-16T21:12:44.849044+0000","last_active":"2024-04-16T21:12:44.849021+0000","last_peered":"2024-04-16T21:12:44.849021+0000","last_clean":"2024-04-16T21:12:44.849021+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:12:44.849021+0000","last_undegraded":"2024-04-16T21:12:44.849021+0000","last_fullsized":"2024-04-16T21:12:44.849021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.848958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.848958+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.848958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:26.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'193","reported_seq":412,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.895939+0000","last_change":"2024-04-16T21:12:38.895987+0000","last_active":"2024-04-16T21:12:38.895939+0000","last_peered":"2024-04-16T21:12:38.895939+0000","last_clean":"2024-04-16T21:12:38.895939+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:12:38.895939+0000","last_undegraded":"2024-04-16T21:12:38.895939+0000","last_fullsized":"2024-04-16T21:12:38.895939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'193","last_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_deep_scrub":"52'193","last_deep_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.895878+0000","objects_scrubbed":9,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:45.133526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.922119+0000","last_change":"2024-04-16T21:12:20.922119+0000","last_active":"2024-04-16T21:12:20.922119+0000","last_peered":"2024-04-16T21:12:20.922119+0000","last_clean":"2024-04-16T21:12:20.922119+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:12:20.922119+0000","last_undegraded":"2024-04-16T21:12:20.922119+0000","last_fullsized":"2024-04-16T21:12:20.922119+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.922033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:09.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.873344+0000","last_change":"2024-04-16T21:12:12.873367+0000","last_active":"2024-04-16T21:12:12.873344+0000","last_peered":"2024-04-16T21:12:12.873344+0000","last_clean":"2024-04-16T21:12:12.873344+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:12:12.873344+0000","last_undegraded":"2024-04-16T21:12:12.873344+0000","last_fullsized":"2024-04-16T21:12:12.873344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.873281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:06.480261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.705482+0000","last_change":"2024-04-16T21:12:52.705482+0000","last_active":"2024-04-16T21:12:52.705482+0000","last_peered":"2024-04-16T21:12:52.705482+0000","last_clean":"2024-04-16T21:12:52.705482+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:12:52.705482+0000","last_undegraded":"2024-04-16T21:12:52.705482+0000","last_fullsized":"2024-04-16T21:12:52.705482+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.705395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.705395+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.705395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:16.267657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'115","reported_seq":1930,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293176+0000","last_change":"2024-04-16T19:25:49.522347+0000","last_active":"2024-04-16T21:11:42.293176+0000","last_peered":"2024-04-16T21:11:42.293176+0000","last_clean":"2024-04-16T21:11:42.293176+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:11:42.293176+0000","last_undegraded":"2024-04-16T21:11:42.293176+0000","last_fullsized":"2024-04-16T21:11:42.293176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.341198+0000","last_change":"2024-04-16T21:12:17.341220+0000","last_active":"2024-04-16T21:12:17.341198+0000","last_peered":"2024-04-16T21:12:17.341198+0000","last_clean":"2024-04-16T21:12:17.341198+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:12:17.341198+0000","last_undegraded":"2024-04-16T21:12:17.341198+0000","last_fullsized":"2024-04-16T21:12:17.341198+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.341136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:45.982431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'15","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.375445+0000","last_change":"2024-04-16T21:12:19.375473+0000","last_active":"2024-04-16T21:12:19.375445+0000","last_peered":"2024-04-16T21:12:19.375445+0000","last_clean":"2024-04-16T21:12:19.375445+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:12:19.375445+0000","last_undegraded":"2024-04-16T21:12:19.375445+0000","last_fullsized":"2024-04-16T21:12:19.375445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.375386+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.049075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.377316+0000","last_change":"2024-04-16T21:12:02.377338+0000","last_active":"2024-04-16T21:12:02.377316+0000","last_peered":"2024-04-16T21:12:02.377316+0000","last_clean":"2024-04-16T21:12:02.377316+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:12:02.377316+0000","last_undegraded":"2024-04-16T21:12:02.377316+0000","last_fullsized":"2024-04-16T21:12:02.377316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.377250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:49.251281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'325","reported_seq":659,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.419682+0000","last_change":"2024-04-16T21:11:56.419734+0000","last_active":"2024-04-16T21:11:56.419682+0000","last_peered":"2024-04-16T21:11:56.419682+0000","last_clean":"2024-04-16T21:11:56.419682+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:11:56.419682+0000","last_undegraded":"2024-04-16T21:11:56.419682+0000","last_fullsized":"2024-04-16T21:11:56.419682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'325","last_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_deep_scrub":"52'325","last_deep_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.419626+0000","objects_scrubbed":21,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:24.477776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6361,"num_omap_keys":29,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'16","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.568997+0000","last_change":"2024-04-16T21:11:45.569097+0000","last_active":"2024-04-16T21:11:45.568997+0000","last_peered":"2024-04-16T21:11:45.568997+0000","last_clean":"2024-04-16T21:11:45.568997+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:11:45.568997+0000","last_undegraded":"2024-04-16T21:11:45.568997+0000","last_fullsized":"2024-04-16T21:11:45.568997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.568923+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:31.758184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.730052+0000","last_change":"2024-04-16T21:12:52.730052+0000","last_active":"2024-04-16T21:12:52.730052+0000","last_peered":"2024-04-16T21:12:52.730052+0000","last_clean":"2024-04-16T21:12:52.730052+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:12:52.730052+0000","last_undegraded":"2024-04-16T21:12:52.730052+0000","last_fullsized":"2024-04-16T21:12:52.730052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.729928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.729928+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.729928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:55.516694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.851893+0000","last_change":"2024-04-16T21:12:54.851929+0000","last_active":"2024-04-16T21:12:54.851893+0000","last_peered":"2024-04-16T21:12:54.851893+0000","last_clean":"2024-04-16T21:12:54.851893+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:12:54.851893+0000","last_undegraded":"2024-04-16T21:12:54.851893+0000","last_fullsized":"2024-04-16T21:12:54.851893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.851809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.851809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.851809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:06.030348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"52'6","reported_seq":970,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.813395+0000","last_change":"2024-04-16T21:12:30.813430+0000","last_active":"2024-04-16T21:12:30.813395+0000","last_peered":"2024-04-16T21:12:30.813395+0000","last_clean":"2024-04-16T21:12:30.813395+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:12:30.813395+0000","last_undegraded":"2024-04-16T21:12:30.813395+0000","last_fullsized":"2024-04-16T21:12:30.813395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.813330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.897002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'78","reported_seq":241,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.999135+0000","last_change":"2024-04-16T21:12:31.999135+0000","last_active":"2024-04-16T21:12:31.999135+0000","last_peered":"2024-04-16T21:12:31.999135+0000","last_clean":"2024-04-16T21:12:31.999135+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:12:31.999135+0000","last_undegraded":"2024-04-16T21:12:31.999135+0000","last_fullsized":"2024-04-16T21:12:31.999135+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.999043+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:40.493275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.929170+0000","last_change":"2024-04-16T21:12:21.929194+0000","last_active":"2024-04-16T21:12:21.929170+0000","last_peered":"2024-04-16T21:12:21.929170+0000","last_clean":"2024-04-16T21:12:21.929170+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:12:21.929170+0000","last_undegraded":"2024-04-16T21:12:21.929170+0000","last_fullsized":"2024-04-16T21:12:21.929170+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.929105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.364555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.721391+0000","last_change":"2024-04-16T21:12:18.721414+0000","last_active":"2024-04-16T21:12:18.721391+0000","last_peered":"2024-04-16T21:12:18.721391+0000","last_clean":"2024-04-16T21:12:18.721391+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:12:18.721391+0000","last_undegraded":"2024-04-16T21:12:18.721391+0000","last_fullsized":"2024-04-16T21:12:18.721391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.721330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:06.756619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":58,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651447+0000","last_change":"2024-04-16T19:47:33.987345+0000","last_active":"2024-04-16T21:11:43.651447+0000","last_peered":"2024-04-16T21:11:43.651447+0000","last_clean":"2024-04-16T21:11:43.651447+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:11:43.651447+0000","last_undegraded":"2024-04-16T21:11:43.651447+0000","last_fullsized":"2024-04-16T21:11:43.651447+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'135","reported_seq":1431,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.902487+0000","last_change":"2024-04-16T21:12:20.902537+0000","last_active":"2024-04-16T21:12:20.902487+0000","last_peered":"2024-04-16T21:12:20.902487+0000","last_clean":"2024-04-16T21:12:20.902487+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:12:20.902487+0000","last_undegraded":"2024-04-16T21:12:20.902487+0000","last_fullsized":"2024-04-16T21:12:20.902487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'135","last_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_deep_scrub":"52'135","last_deep_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.902428+0000","objects_scrubbed":42,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:27:02.685242+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3879,"num_omap_keys":15,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.456127+0000","last_change":"2024-04-16T21:12:14.456172+0000","last_active":"2024-04-16T21:12:14.456127+0000","last_peered":"2024-04-16T21:12:14.456127+0000","last_clean":"2024-04-16T21:12:14.456127+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:12:14.456127+0000","last_undegraded":"2024-04-16T21:12:14.456127+0000","last_fullsized":"2024-04-16T21:12:14.456127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:43.097345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'24","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.378145+0000","last_change":"2024-04-16T21:12:23.378174+0000","last_active":"2024-04-16T21:12:23.378145+0000","last_peered":"2024-04-16T21:12:23.378145+0000","last_clean":"2024-04-16T21:12:23.378145+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:12:23.378145+0000","last_undegraded":"2024-04-16T21:12:23.378145+0000","last_fullsized":"2024-04-16T21:12:23.378145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.378087+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:34.980757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.647510+0000","last_change":"2024-04-16T21:11:51.647550+0000","last_active":"2024-04-16T21:11:51.647510+0000","last_peered":"2024-04-16T21:11:51.647510+0000","last_clean":"2024-04-16T21:11:51.647510+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:11:51.647510+0000","last_undegraded":"2024-04-16T21:11:51.647510+0000","last_fullsized":"2024-04-16T21:11:51.647510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.647441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:25.536915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2068","reported_seq":5847,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.355619+0000","last_change":"2024-04-16T21:11:58.355713+0000","last_active":"2024-04-16T21:11:58.355619+0000","last_peered":"2024-04-16T21:11:58.355619+0000","last_clean":"2024-04-16T21:11:58.355619+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:11:58.355619+0000","last_undegraded":"2024-04-16T21:11:58.355619+0000","last_fullsized":"2024-04-16T21:11:58.355619+0000","mapping_epoch":33,"log_start":"47'400","ondisk_log_start":"47'400","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2068","last_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_deep_scrub":"52'2068","last_deep_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.355545+0000","objects_scrubbed":24,"log_size":1668,"log_dups_size":400,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:46.390834+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16267,"num_omap_keys":93,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'26","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.715965+0000","last_change":"2024-04-16T21:11:55.716041+0000","last_active":"2024-04-16T21:11:55.715965+0000","last_peered":"2024-04-16T21:11:55.715965+0000","last_clean":"2024-04-16T21:11:55.715965+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:11:55.715965+0000","last_undegraded":"2024-04-16T21:11:55.715965+0000","last_fullsized":"2024-04-16T21:11:55.715965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.715893+0000","objects_scrubbed":12,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:15.543756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.855470+0000","last_change":"2024-04-16T21:12:57.855492+0000","last_active":"2024-04-16T21:12:57.855470+0000","last_peered":"2024-04-16T21:12:57.855470+0000","last_clean":"2024-04-16T21:12:57.855470+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:12:57.855470+0000","last_undegraded":"2024-04-16T21:12:57.855470+0000","last_fullsized":"2024-04-16T21:12:57.855470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.855408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.855408+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.855408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:47.733563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.270377+0000","last_change":"2024-04-16T21:12:49.270399+0000","last_active":"2024-04-16T21:12:49.270377+0000","last_peered":"2024-04-16T21:12:49.270377+0000","last_clean":"2024-04-16T21:12:49.270377+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:12:49.270377+0000","last_undegraded":"2024-04-16T21:12:49.270377+0000","last_fullsized":"2024-04-16T21:12:49.270377+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.270275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.270275+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.270275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"52'2","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.812623+0000","last_change":"2024-04-16T21:12:32.812623+0000","last_active":"2024-04-16T21:12:32.812623+0000","last_peered":"2024-04-16T21:12:32.812623+0000","last_clean":"2024-04-16T21:12:32.812623+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:12:32.812623+0000","last_undegraded":"2024-04-16T21:12:32.812623+0000","last_fullsized":"2024-04-16T21:12:32.812623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.812531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:48.281904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'147","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334518+0000","last_change":"2024-04-16T21:12:30.334588+0000","last_active":"2024-04-16T21:12:30.334518+0000","last_peered":"2024-04-16T21:12:30.334518+0000","last_clean":"2024-04-16T21:12:30.334518+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:12:30.334518+0000","last_undegraded":"2024-04-16T21:12:30.334518+0000","last_fullsized":"2024-04-16T21:12:30.334518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334447+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:36.782836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'13","reported_seq":1323,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.286038+0000","last_change":"2024-04-16T21:12:31.286066+0000","last_active":"2024-04-16T21:12:31.286038+0000","last_peered":"2024-04-16T21:12:31.286038+0000","last_clean":"2024-04-16T21:12:31.286038+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:12:31.286038+0000","last_undegraded":"2024-04-16T21:12:31.286038+0000","last_fullsized":"2024-04-16T21:12:31.286038+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.285973+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:54.848235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.405984+0000","last_change":"2024-04-16T21:12:07.406028+0000","last_active":"2024-04-16T21:12:07.405984+0000","last_peered":"2024-04-16T21:12:07.405984+0000","last_clean":"2024-04-16T21:12:07.405984+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:12:07.405984+0000","last_undegraded":"2024-04-16T21:12:07.405984+0000","last_fullsized":"2024-04-16T21:12:07.405984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.405900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:12.444844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.748485+0000","last_change":"2024-04-16T21:12:54.748509+0000","last_active":"2024-04-16T21:12:54.748485+0000","last_peered":"2024-04-16T21:12:54.748485+0000","last_clean":"2024-04-16T21:12:54.748485+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:12:54.748485+0000","last_undegraded":"2024-04-16T21:12:54.748485+0000","last_fullsized":"2024-04-16T21:12:54.748485+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.748417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.748417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.748417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:51.723857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'182","reported_seq":1946,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.937380+0000","last_change":"2024-04-16T21:12:22.937438+0000","last_active":"2024-04-16T21:12:22.937380+0000","last_peered":"2024-04-16T21:12:22.937380+0000","last_clean":"2024-04-16T21:12:22.937380+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:12:22.937380+0000","last_undegraded":"2024-04-16T21:12:22.937380+0000","last_fullsized":"2024-04-16T21:12:22.937380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'182","last_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_deep_scrub":"52'182","last_deep_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.937321+0000","objects_scrubbed":46,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:18.195145+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8599,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.724011+0000","last_change":"2024-04-16T21:12:29.724048+0000","last_active":"2024-04-16T21:12:29.724011+0000","last_peered":"2024-04-16T21:12:29.724011+0000","last_clean":"2024-04-16T21:12:29.724011+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:12:29.724011+0000","last_undegraded":"2024-04-16T21:12:29.724011+0000","last_fullsized":"2024-04-16T21:12:29.724011+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.723920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:23.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'17","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.440078+0000","last_change":"2024-04-16T21:12:10.440115+0000","last_active":"2024-04-16T21:12:10.440078+0000","last_peered":"2024-04-16T21:12:10.440078+0000","last_clean":"2024-04-16T21:12:10.440078+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:12:10.440078+0000","last_undegraded":"2024-04-16T21:12:10.440078+0000","last_fullsized":"2024-04-16T21:12:10.440078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.440017+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:44.207343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.910624+0000","last_change":"2024-04-16T21:12:13.910647+0000","last_active":"2024-04-16T21:12:13.910624+0000","last_peered":"2024-04-16T21:12:13.910624+0000","last_clean":"2024-04-16T21:12:13.910624+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:12:13.910624+0000","last_undegraded":"2024-04-16T21:12:13.910624+0000","last_fullsized":"2024-04-16T21:12:13.910624+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.910562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:19.649702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'362","reported_seq":695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.975586+0000","last_change":"2024-04-16T21:11:44.975641+0000","last_active":"2024-04-16T21:11:44.975586+0000","last_peered":"2024-04-16T21:11:44.975586+0000","last_clean":"2024-04-16T21:11:44.975586+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:11:44.975586+0000","last_undegraded":"2024-04-16T21:11:44.975586+0000","last_fullsized":"2024-04-16T21:11:44.975586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'362","last_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_deep_scrub":"52'362","last_deep_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.975535+0000","objects_scrubbed":21,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:50:56.598547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2185,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'18","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.687122+0000","last_change":"2024-04-16T21:11:48.687164+0000","last_active":"2024-04-16T21:11:48.687122+0000","last_peered":"2024-04-16T21:11:48.687122+0000","last_clean":"2024-04-16T21:11:48.687122+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:11:48.687122+0000","last_undegraded":"2024-04-16T21:11:48.687122+0000","last_fullsized":"2024-04-16T21:11:48.687122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.687069+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:52.625067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.473384+0000","last_change":"2024-04-16T21:12:54.473405+0000","last_active":"2024-04-16T21:12:54.473384+0000","last_peered":"2024-04-16T21:12:54.473384+0000","last_clean":"2024-04-16T21:12:54.473384+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:12:54.473384+0000","last_undegraded":"2024-04-16T21:12:54.473384+0000","last_fullsized":"2024-04-16T21:12:54.473384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.473317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.473317+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.473317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:08.143470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.853921+0000","last_change":"2024-04-16T21:12:48.853954+0000","last_active":"2024-04-16T21:12:48.853921+0000","last_peered":"2024-04-16T21:12:48.853921+0000","last_clean":"2024-04-16T21:12:48.853921+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:12:48.853921+0000","last_undegraded":"2024-04-16T21:12:48.853921+0000","last_fullsized":"2024-04-16T21:12:48.853921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.853857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.853857+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.853857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:20.558650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'19","reported_seq":469,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.913946+0000","last_change":"2024-04-16T21:12:26.913976+0000","last_active":"2024-04-16T21:12:26.913946+0000","last_peered":"2024-04-16T21:12:26.913946+0000","last_clean":"2024-04-16T21:12:26.913946+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:12:26.913946+0000","last_undegraded":"2024-04-16T21:12:26.913946+0000","last_fullsized":"2024-04-16T21:12:26.913946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.913887+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.708618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'87","reported_seq":260,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.302830+0000","last_change":"2024-04-16T21:12:35.302869+0000","last_active":"2024-04-16T21:12:35.302830+0000","last_peered":"2024-04-16T21:12:35.302830+0000","last_clean":"2024-04-16T21:12:35.302830+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:12:35.302830+0000","last_undegraded":"2024-04-16T21:12:35.302830+0000","last_fullsized":"2024-04-16T21:12:35.302830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.302758+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:01.984411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.899124+0000","last_change":"2024-04-16T21:12:18.899147+0000","last_active":"2024-04-16T21:12:18.899124+0000","last_peered":"2024-04-16T21:12:18.899124+0000","last_clean":"2024-04-16T21:12:18.899124+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:12:18.899124+0000","last_undegraded":"2024-04-16T21:12:18.899124+0000","last_fullsized":"2024-04-16T21:12:18.899124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.899064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:17.822519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.847627+0000","last_change":"2024-04-16T21:12:03.847650+0000","last_active":"2024-04-16T21:12:03.847627+0000","last_peered":"2024-04-16T21:12:03.847627+0000","last_clean":"2024-04-16T21:12:03.847627+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:12:03.847627+0000","last_undegraded":"2024-04-16T21:12:03.847627+0000","last_fullsized":"2024-04-16T21:12:03.847627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.847563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:36.440589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744155+0000","last_change":"2024-04-16T19:47:33.989682+0000","last_active":"2024-04-16T21:11:42.744155+0000","last_peered":"2024-04-16T21:11:42.744155+0000","last_clean":"2024-04-16T21:11:42.744155+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:11:42.744155+0000","last_undegraded":"2024-04-16T21:11:42.744155+0000","last_fullsized":"2024-04-16T21:11:42.744155+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'186","reported_seq":2409,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.401091+0000","last_change":"2024-04-16T21:12:42.401122+0000","last_active":"2024-04-16T21:12:42.401091+0000","last_peered":"2024-04-16T21:12:42.401091+0000","last_clean":"2024-04-16T21:12:42.401091+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:12:42.401091+0000","last_undegraded":"2024-04-16T21:12:42.401091+0000","last_fullsized":"2024-04-16T21:12:42.401091+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'186","last_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_deep_scrub":"52'186","last_deep_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.401026+0000","objects_scrubbed":52,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:51.749803+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12956,"num_omap_keys":52,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.650119+0000","last_change":"2024-04-16T21:12:13.650119+0000","last_active":"2024-04-16T21:12:13.650119+0000","last_peered":"2024-04-16T21:12:13.650119+0000","last_clean":"2024-04-16T21:12:13.650119+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:12:13.650119+0000","last_undegraded":"2024-04-16T21:12:13.650119+0000","last_fullsized":"2024-04-16T21:12:13.650119+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.650036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:16.977821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"52'13","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.901211+0000","last_change":"2024-04-16T21:12:20.901247+0000","last_active":"2024-04-16T21:12:20.901211+0000","last_peered":"2024-04-16T21:12:20.901211+0000","last_clean":"2024-04-16T21:12:20.901211+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:12:20.901211+0000","last_undegraded":"2024-04-16T21:12:20.901211+0000","last_fullsized":"2024-04-16T21:12:20.901211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.901143+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:54.668469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'706","reported_seq":2075,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.405465+0000","last_change":"2024-04-16T21:12:08.405492+0000","last_active":"2024-04-16T21:12:08.405465+0000","last_peered":"2024-04-16T21:12:08.405465+0000","last_clean":"2024-04-16T21:12:08.405465+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:12:08.405465+0000","last_undegraded":"2024-04-16T21:12:08.405465+0000","last_fullsized":"2024-04-16T21:12:08.405465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.405399+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:20.200130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'292","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.642477+0000","last_change":"2024-04-16T21:11:50.642568+0000","last_active":"2024-04-16T21:11:50.642477+0000","last_peered":"2024-04-16T21:11:50.642477+0000","last_clean":"2024-04-16T21:11:50.642477+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:11:50.642477+0000","last_undegraded":"2024-04-16T21:11:50.642477+0000","last_fullsized":"2024-04-16T21:11:50.642477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'292","last_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_deep_scrub":"52'292","last_deep_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.642400+0000","objects_scrubbed":23,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:29.527145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4142,"num_omap_keys":19,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.314574+0000","last_change":"2024-04-16T21:11:57.314619+0000","last_active":"2024-04-16T21:11:57.314574+0000","last_peered":"2024-04-16T21:11:57.314574+0000","last_clean":"2024-04-16T21:11:57.314574+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:11:57.314574+0000","last_undegraded":"2024-04-16T21:11:57.314574+0000","last_fullsized":"2024-04-16T21:11:57.314574+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.314511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:49.494822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.778832+0000","last_change":"2024-04-16T21:12:54.778854+0000","last_active":"2024-04-16T21:12:54.778832+0000","last_peered":"2024-04-16T21:12:54.778832+0000","last_clean":"2024-04-16T21:12:54.778832+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:12:54.778832+0000","last_undegraded":"2024-04-16T21:12:54.778832+0000","last_fullsized":"2024-04-16T21:12:54.778832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.778773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.778773+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.778773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:37.834506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.898133+0000","last_change":"2024-04-16T21:12:08.898133+0000","last_active":"2024-04-16T21:12:08.898133+0000","last_peered":"2024-04-16T21:12:08.898133+0000","last_clean":"2024-04-16T21:12:08.898133+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:12:08.898133+0000","last_undegraded":"2024-04-16T21:12:08.898133+0000","last_fullsized":"2024-04-16T21:12:08.898133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.898047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:36.956196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"52'2","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.800302+0000","last_change":"2024-04-16T21:12:36.800302+0000","last_active":"2024-04-16T21:12:36.800302+0000","last_peered":"2024-04-16T21:12:36.800302+0000","last_clean":"2024-04-16T21:12:36.800302+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:12:36.800302+0000","last_undegraded":"2024-04-16T21:12:36.800302+0000","last_fullsized":"2024-04-16T21:12:36.800302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.800213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:46.361882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'101","reported_seq":275,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.698919+0000","last_change":"2024-04-16T21:12:30.698919+0000","last_active":"2024-04-16T21:12:30.698919+0000","last_peered":"2024-04-16T21:12:30.698919+0000","last_clean":"2024-04-16T21:12:30.698919+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:12:30.698919+0000","last_undegraded":"2024-04-16T21:12:30.698919+0000","last_fullsized":"2024-04-16T21:12:30.698919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.698825+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.782497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.333039+0000","last_change":"2024-04-16T21:12:32.333063+0000","last_active":"2024-04-16T21:12:32.333039+0000","last_peered":"2024-04-16T21:12:32.333039+0000","last_clean":"2024-04-16T21:12:32.333039+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:12:32.333039+0000","last_undegraded":"2024-04-16T21:12:32.333039+0000","last_fullsized":"2024-04-16T21:12:32.333039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.332967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:32.295174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.889383+0000","last_change":"2024-04-16T21:12:17.889407+0000","last_active":"2024-04-16T21:12:17.889383+0000","last_peered":"2024-04-16T21:12:17.889383+0000","last_clean":"2024-04-16T21:12:17.889383+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:12:17.889383+0000","last_undegraded":"2024-04-16T21:12:17.889383+0000","last_fullsized":"2024-04-16T21:12:17.889383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.889318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:23.628459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293308+0000","last_change":"2024-04-16T19:47:33.979630+0000","last_active":"2024-04-16T21:11:42.293308+0000","last_peered":"2024-04-16T21:11:42.293308+0000","last_clean":"2024-04-16T21:11:42.293308+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:11:42.293308+0000","last_undegraded":"2024-04-16T21:11:42.293308+0000","last_fullsized":"2024-04-16T21:11:42.293308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'172","reported_seq":1727,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.828813+0000","last_change":"2024-04-16T21:12:45.828865+0000","last_active":"2024-04-16T21:12:45.828813+0000","last_peered":"2024-04-16T21:12:45.828813+0000","last_clean":"2024-04-16T21:12:45.828813+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:12:45.828813+0000","last_undegraded":"2024-04-16T21:12:45.828813+0000","last_fullsized":"2024-04-16T21:12:45.828813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:45.828749+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:45.828749+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.828749+0000","objects_scrubbed":38,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:49:27.527910+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12008,"num_omap_keys":49,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.854832+0000","last_change":"2024-04-16T21:12:21.854876+0000","last_active":"2024-04-16T21:12:21.854832+0000","last_peered":"2024-04-16T21:12:21.854832+0000","last_clean":"2024-04-16T21:12:21.854832+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:12:21.854832+0000","last_undegraded":"2024-04-16T21:12:21.854832+0000","last_fullsized":"2024-04-16T21:12:21.854832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.854759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:37.324132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'138","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.362284+0000","last_change":"2024-04-16T21:12:22.362331+0000","last_active":"2024-04-16T21:12:22.362284+0000","last_peered":"2024-04-16T21:12:22.362284+0000","last_clean":"2024-04-16T21:12:22.362284+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:12:22.362284+0000","last_undegraded":"2024-04-16T21:12:22.362284+0000","last_fullsized":"2024-04-16T21:12:22.362284+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.362197+0000","objects_scrubbed":2,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:16.550872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9707,"num_omap_keys":56,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.836583+0000","last_change":"2024-04-16T21:11:52.836606+0000","last_active":"2024-04-16T21:11:52.836583+0000","last_peered":"2024-04-16T21:11:52.836583+0000","last_clean":"2024-04-16T21:11:52.836583+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:11:52.836583+0000","last_undegraded":"2024-04-16T21:11:52.836583+0000","last_fullsized":"2024-04-16T21:11:52.836583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.836519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:26.725994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'371","reported_seq":758,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.959985+0000","last_change":"2024-04-16T21:11:46.960080+0000","last_active":"2024-04-16T21:11:46.959985+0000","last_peered":"2024-04-16T21:11:46.959985+0000","last_clean":"2024-04-16T21:11:46.959985+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:11:46.959985+0000","last_undegraded":"2024-04-16T21:11:46.959985+0000","last_fullsized":"2024-04-16T21:11:46.959985+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'371","last_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_deep_scrub":"52'371","last_deep_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.959920+0000","objects_scrubbed":24,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:28.129840+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2169,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'19","reported_seq":216,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.614888+0000","last_change":"2024-04-16T21:11:53.614931+0000","last_active":"2024-04-16T21:11:53.614888+0000","last_peered":"2024-04-16T21:11:53.614888+0000","last_clean":"2024-04-16T21:11:53.614888+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:11:53.614888+0000","last_undegraded":"2024-04-16T21:11:53.614888+0000","last_fullsized":"2024-04-16T21:11:53.614888+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.614826+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:49.084093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.274919+0000","last_change":"2024-04-16T21:12:58.274953+0000","last_active":"2024-04-16T21:12:58.274919+0000","last_peered":"2024-04-16T21:12:58.274919+0000","last_clean":"2024-04-16T21:12:58.274919+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:12:58.274919+0000","last_undegraded":"2024-04-16T21:12:58.274919+0000","last_fullsized":"2024-04-16T21:12:58.274919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.274858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.274858+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.274858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:15.668157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.827874+0000","last_change":"2024-04-16T21:12:04.827874+0000","last_active":"2024-04-16T21:12:04.827874+0000","last_peered":"2024-04-16T21:12:04.827874+0000","last_clean":"2024-04-16T21:12:04.827874+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:12:04.827874+0000","last_undegraded":"2024-04-16T21:12:04.827874+0000","last_fullsized":"2024-04-16T21:12:04.827874+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.827767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:23.692877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'20","reported_seq":178,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.142643+0000","last_change":"2024-04-16T21:12:47.142643+0000","last_active":"2024-04-16T21:12:47.142643+0000","last_peered":"2024-04-16T21:12:47.142643+0000","last_clean":"2024-04-16T21:12:47.142643+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:12:47.142643+0000","last_undegraded":"2024-04-16T21:12:47.142643+0000","last_fullsized":"2024-04-16T21:12:47.142643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:47.142546+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:47.142546+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.142546+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:47.104753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'147","reported_seq":343,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.899755+0000","last_change":"2024-04-16T21:12:22.899755+0000","last_active":"2024-04-16T21:12:22.899755+0000","last_peered":"2024-04-16T21:12:22.899755+0000","last_clean":"2024-04-16T21:12:22.899755+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:12:22.899755+0000","last_undegraded":"2024-04-16T21:12:22.899755+0000","last_fullsized":"2024-04-16T21:12:22.899755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.899611+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:49.019687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'13","reported_seq":1322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.856493+0000","last_change":"2024-04-16T21:12:19.856493+0000","last_active":"2024-04-16T21:12:19.856493+0000","last_peered":"2024-04-16T21:12:19.856493+0000","last_clean":"2024-04-16T21:12:19.856493+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:12:19.856493+0000","last_undegraded":"2024-04-16T21:12:19.856493+0000","last_fullsized":"2024-04-16T21:12:19.856493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.856405+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.872492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.913945+0000","last_change":"2024-04-16T21:12:16.913967+0000","last_active":"2024-04-16T21:12:16.913945+0000","last_peered":"2024-04-16T21:12:16.913945+0000","last_clean":"2024-04-16T21:12:16.913945+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:12:16.913945+0000","last_undegraded":"2024-04-16T21:12:16.913945+0000","last_fullsized":"2024-04-16T21:12:16.913945+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.913879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:03.787910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.389430+0000","last_change":"2024-04-16T19:47:33.986086+0000","last_active":"2024-04-16T21:11:09.389430+0000","last_peered":"2024-04-16T21:11:09.389430+0000","last_clean":"2024-04-16T21:11:09.389430+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:11:09.389430+0000","last_undegraded":"2024-04-16T21:11:09.389430+0000","last_fullsized":"2024-04-16T21:11:09.389430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'166","reported_seq":1925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.247637+0000","last_change":"2024-04-16T21:12:40.247676+0000","last_active":"2024-04-16T21:12:40.247637+0000","last_peered":"2024-04-16T21:12:40.247637+0000","last_clean":"2024-04-16T21:12:40.247637+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:12:40.247637+0000","last_undegraded":"2024-04-16T21:12:40.247637+0000","last_fullsized":"2024-04-16T21:12:40.247637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'166","last_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_deep_scrub":"52'166","last_deep_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.247577+0000","objects_scrubbed":44,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:25.005335+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9289,"num_omap_keys":38,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.368999+0000","last_change":"2024-04-16T21:12:24.369021+0000","last_active":"2024-04-16T21:12:24.368999+0000","last_peered":"2024-04-16T21:12:24.368999+0000","last_clean":"2024-04-16T21:12:24.368999+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:12:24.368999+0000","last_undegraded":"2024-04-16T21:12:24.368999+0000","last_fullsized":"2024-04-16T21:12:24.368999+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.368926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:12.501416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.879266+0000","last_change":"2024-04-16T21:12:15.879307+0000","last_active":"2024-04-16T21:12:15.879266+0000","last_peered":"2024-04-16T21:12:15.879266+0000","last_clean":"2024-04-16T21:12:15.879266+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:12:15.879266+0000","last_undegraded":"2024-04-16T21:12:15.879266+0000","last_fullsized":"2024-04-16T21:12:15.879266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.879198+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:41.314648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.625545+0000","last_change":"2024-04-16T21:11:53.625568+0000","last_active":"2024-04-16T21:11:53.625545+0000","last_peered":"2024-04-16T21:11:53.625545+0000","last_clean":"2024-04-16T21:11:53.625545+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:11:53.625545+0000","last_undegraded":"2024-04-16T21:11:53.625545+0000","last_fullsized":"2024-04-16T21:11:53.625545+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.625484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:35.360695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'485","reported_seq":911,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.722929+0000","last_change":"2024-04-16T21:11:47.722980+0000","last_active":"2024-04-16T21:11:47.722929+0000","last_peered":"2024-04-16T21:11:47.722929+0000","last_clean":"2024-04-16T21:11:47.722929+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:11:47.722929+0000","last_undegraded":"2024-04-16T21:11:47.722929+0000","last_fullsized":"2024-04-16T21:11:47.722929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'485","last_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_deep_scrub":"52'485","last_deep_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.722865+0000","objects_scrubbed":32,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:39.504569+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42883,"num_omap_keys":132,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'14","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.616163+0000","last_change":"2024-04-16T21:12:05.616245+0000","last_active":"2024-04-16T21:12:05.616163+0000","last_peered":"2024-04-16T21:12:05.616163+0000","last_clean":"2024-04-16T21:12:05.616163+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:12:05.616163+0000","last_undegraded":"2024-04-16T21:12:05.616163+0000","last_fullsized":"2024-04-16T21:12:05.616163+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.616057+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.466963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.688471+0000","last_change":"2024-04-16T21:12:33.688471+0000","last_active":"2024-04-16T21:12:33.688471+0000","last_peered":"2024-04-16T21:12:33.688471+0000","last_clean":"2024-04-16T21:12:33.688471+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:12:33.688471+0000","last_undegraded":"2024-04-16T21:12:33.688471+0000","last_fullsized":"2024-04-16T21:12:33.688471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.688381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:51:41.315722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.702730+0000","last_change":"2024-04-16T21:12:50.702730+0000","last_active":"2024-04-16T21:12:50.702730+0000","last_peered":"2024-04-16T21:12:50.702730+0000","last_clean":"2024-04-16T21:12:50.702730+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:12:50.702730+0000","last_undegraded":"2024-04-16T21:12:50.702730+0000","last_fullsized":"2024-04-16T21:12:50.702730+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.702579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.702579+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.702579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:13.969329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'10","reported_seq":1362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.386376+0000","last_change":"2024-04-16T19:24:45.411451+0000","last_active":"2024-04-16T21:10:54.386376+0000","last_peered":"2024-04-16T21:10:54.386376+0000","last_clean":"2024-04-16T21:10:54.386376+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:10:54.386376+0000","last_undegraded":"2024-04-16T21:10:54.386376+0000","last_fullsized":"2024-04-16T21:10:54.386376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:44.395404+0000","last_clean_scrub_stamp":"2024-04-16T19:24:44.395404+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'115","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.013687+0000","last_change":"2024-04-16T21:12:29.013687+0000","last_active":"2024-04-16T21:12:29.013687+0000","last_peered":"2024-04-16T21:12:29.013687+0000","last_clean":"2024-04-16T21:12:29.013687+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:12:29.013687+0000","last_undegraded":"2024-04-16T21:12:29.013687+0000","last_fullsized":"2024-04-16T21:12:29.013687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.013556+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:57.654850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'13","reported_seq":1322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.607256+0000","last_change":"2024-04-16T21:12:22.607256+0000","last_active":"2024-04-16T21:12:22.607256+0000","last_peered":"2024-04-16T21:12:22.607256+0000","last_clean":"2024-04-16T21:12:22.607256+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:12:22.607256+0000","last_undegraded":"2024-04-16T21:12:22.607256+0000","last_fullsized":"2024-04-16T21:12:22.607256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.607175+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:48.727251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.673878+0000","last_change":"2024-04-16T21:12:20.673922+0000","last_active":"2024-04-16T21:12:20.673878+0000","last_peered":"2024-04-16T21:12:20.673878+0000","last_clean":"2024-04-16T21:12:20.673878+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:12:20.673878+0000","last_undegraded":"2024-04-16T21:12:20.673878+0000","last_fullsized":"2024-04-16T21:12:20.673878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.673807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:17.517736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":58,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.744139+0000","last_change":"2024-04-16T19:47:33.988373+0000","last_active":"2024-04-16T21:11:42.744139+0000","last_peered":"2024-04-16T21:11:42.744139+0000","last_clean":"2024-04-16T21:11:42.744139+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:11:42.744139+0000","last_undegraded":"2024-04-16T21:11:42.744139+0000","last_fullsized":"2024-04-16T21:11:42.744139+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'162","reported_seq":2309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.337358+0000","last_change":"2024-04-16T21:12:34.337466+0000","last_active":"2024-04-16T21:12:34.337358+0000","last_peered":"2024-04-16T21:12:34.337358+0000","last_clean":"2024-04-16T21:12:34.337358+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:12:34.337358+0000","last_undegraded":"2024-04-16T21:12:34.337358+0000","last_fullsized":"2024-04-16T21:12:34.337358+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'162","last_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_deep_scrub":"52'162","last_deep_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.337295+0000","objects_scrubbed":41,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:21.592470+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11917,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.346082+0000","last_change":"2024-04-16T21:12:20.346107+0000","last_active":"2024-04-16T21:12:20.346082+0000","last_peered":"2024-04-16T21:12:20.346082+0000","last_clean":"2024-04-16T21:12:20.346082+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:12:20.346082+0000","last_undegraded":"2024-04-16T21:12:20.346082+0000","last_fullsized":"2024-04-16T21:12:20.346082+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.346020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:28.840252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.905708+0000","last_change":"2024-04-16T21:12:13.905733+0000","last_active":"2024-04-16T21:12:13.905708+0000","last_peered":"2024-04-16T21:12:13.905708+0000","last_clean":"2024-04-16T21:12:13.905708+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:12:13.905708+0000","last_undegraded":"2024-04-16T21:12:13.905708+0000","last_fullsized":"2024-04-16T21:12:13.905708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.905640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:17.215414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'706","reported_seq":2474,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.688857+0000","last_change":"2024-04-16T21:11:58.688891+0000","last_active":"2024-04-16T21:11:58.688857+0000","last_peered":"2024-04-16T21:11:58.688857+0000","last_clean":"2024-04-16T21:11:58.688857+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:11:58.688857+0000","last_undegraded":"2024-04-16T21:11:58.688857+0000","last_fullsized":"2024-04-16T21:11:58.688857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.688787+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:58.866593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'332","reported_seq":603,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.669169+0000","last_change":"2024-04-16T21:11:50.669285+0000","last_active":"2024-04-16T21:11:50.669169+0000","last_peered":"2024-04-16T21:11:50.669169+0000","last_clean":"2024-04-16T21:11:50.669169+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:11:50.669169+0000","last_undegraded":"2024-04-16T21:11:50.669169+0000","last_fullsized":"2024-04-16T21:11:50.669169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'332","last_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_deep_scrub":"52'332","last_deep_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.669096+0000","objects_scrubbed":21,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:18.487063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2179,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'12","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.639977+0000","last_change":"2024-04-16T21:11:49.640051+0000","last_active":"2024-04-16T21:11:49.639977+0000","last_peered":"2024-04-16T21:11:49.639977+0000","last_clean":"2024-04-16T21:11:49.639977+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:11:49.639977+0000","last_undegraded":"2024-04-16T21:11:49.639977+0000","last_fullsized":"2024-04-16T21:11:49.639977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.639900+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:27.425866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.123064+0000","last_change":"2024-04-16T21:12:45.123064+0000","last_active":"2024-04-16T21:12:45.123064+0000","last_peered":"2024-04-16T21:12:45.123064+0000","last_clean":"2024-04-16T21:12:45.123064+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:12:45.123064+0000","last_undegraded":"2024-04-16T21:12:45.123064+0000","last_fullsized":"2024-04-16T21:12:45.123064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.122972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.122972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.122972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:08.389721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.789162+0000","last_change":"2024-04-16T21:12:46.789162+0000","last_active":"2024-04-16T21:12:46.789162+0000","last_peered":"2024-04-16T21:12:46.789162+0000","last_clean":"2024-04-16T21:12:46.789162+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:12:46.789162+0000","last_undegraded":"2024-04-16T21:12:46.789162+0000","last_fullsized":"2024-04-16T21:12:46.789162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.789085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.789085+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.789085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:33.086889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.287531+0000","last_change":"2024-04-16T21:12:37.287531+0000","last_active":"2024-04-16T21:12:37.287531+0000","last_peered":"2024-04-16T21:12:37.287531+0000","last_clean":"2024-04-16T21:12:37.287531+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:12:37.287531+0000","last_undegraded":"2024-04-16T21:12:37.287531+0000","last_fullsized":"2024-04-16T21:12:37.287531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.287448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:24.542624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'170","reported_seq":377,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.887713+0000","last_change":"2024-04-16T21:12:36.887713+0000","last_active":"2024-04-16T21:12:36.887713+0000","last_peered":"2024-04-16T21:12:36.887713+0000","last_clean":"2024-04-16T21:12:36.887713+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:12:36.887713+0000","last_undegraded":"2024-04-16T21:12:36.887713+0000","last_fullsized":"2024-04-16T21:12:36.887713+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.887622+0000","objects_scrubbed":8,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:25.493915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.314224+0000","last_change":"2024-04-16T21:12:26.314224+0000","last_active":"2024-04-16T21:12:26.314224+0000","last_peered":"2024-04-16T21:12:26.314224+0000","last_clean":"2024-04-16T21:12:26.314224+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:12:26.314224+0000","last_undegraded":"2024-04-16T21:12:26.314224+0000","last_fullsized":"2024-04-16T21:12:26.314224+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.314134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:40.584484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.903511+0000","last_change":"2024-04-16T21:12:19.903531+0000","last_active":"2024-04-16T21:12:19.903511+0000","last_peered":"2024-04-16T21:12:19.903511+0000","last_clean":"2024-04-16T21:12:19.903511+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:12:19.903511+0000","last_undegraded":"2024-04-16T21:12:19.903511+0000","last_fullsized":"2024-04-16T21:12:19.903511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.903448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:31.698180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.747319+0000","last_change":"2024-04-16T21:12:51.747319+0000","last_active":"2024-04-16T21:12:51.747319+0000","last_peered":"2024-04-16T21:12:51.747319+0000","last_clean":"2024-04-16T21:12:51.747319+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:12:51.747319+0000","last_undegraded":"2024-04-16T21:12:51.747319+0000","last_fullsized":"2024-04-16T21:12:51.747319+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.747234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.747234+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.747234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:15.013984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'216","reported_seq":2161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.661893+0000","last_change":"2024-04-16T21:12:21.661923+0000","last_active":"2024-04-16T21:12:21.661893+0000","last_peered":"2024-04-16T21:12:21.661893+0000","last_clean":"2024-04-16T21:12:21.661893+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:12:21.661893+0000","last_undegraded":"2024-04-16T21:12:21.661893+0000","last_fullsized":"2024-04-16T21:12:21.661893+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'216","last_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_deep_scrub":"52'216","last_deep_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.661832+0000","objects_scrubbed":54,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.097281+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14476,"num_omap_keys":58,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.921849+0000","last_change":"2024-04-16T21:12:14.921870+0000","last_active":"2024-04-16T21:12:14.921849+0000","last_peered":"2024-04-16T21:12:14.921849+0000","last_clean":"2024-04-16T21:12:14.921849+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:12:14.921849+0000","last_undegraded":"2024-04-16T21:12:14.921849+0000","last_fullsized":"2024-04-16T21:12:14.921849+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.921774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:11.765704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'17","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.682277+0000","last_change":"2024-04-16T21:12:08.682317+0000","last_active":"2024-04-16T21:12:08.682277+0000","last_peered":"2024-04-16T21:12:08.682277+0000","last_clean":"2024-04-16T21:12:08.682277+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:12:08.682277+0000","last_undegraded":"2024-04-16T21:12:08.682277+0000","last_fullsized":"2024-04-16T21:12:08.682277+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.682212+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:32.852256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.634331+0000","last_change":"2024-04-16T21:11:56.634374+0000","last_active":"2024-04-16T21:11:56.634331+0000","last_peered":"2024-04-16T21:11:56.634331+0000","last_clean":"2024-04-16T21:11:56.634331+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:11:56.634331+0000","last_undegraded":"2024-04-16T21:11:56.634331+0000","last_fullsized":"2024-04-16T21:11:56.634331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.634253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:30.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'281","reported_seq":601,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.734176+0000","last_change":"2024-04-16T21:11:44.734203+0000","last_active":"2024-04-16T21:11:44.734176+0000","last_peered":"2024-04-16T21:11:44.734176+0000","last_clean":"2024-04-16T21:11:44.734176+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:11:44.734176+0000","last_undegraded":"2024-04-16T21:11:44.734176+0000","last_fullsized":"2024-04-16T21:11:44.734176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'281","last_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_deep_scrub":"52'281","last_deep_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.734118+0000","objects_scrubbed":25,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:22.520084+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1975,"num_omap_keys":9,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'12","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.876267+0000","last_change":"2024-04-16T21:12:09.876315+0000","last_active":"2024-04-16T21:12:09.876267+0000","last_peered":"2024-04-16T21:12:09.876267+0000","last_clean":"2024-04-16T21:12:09.876267+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:12:09.876267+0000","last_undegraded":"2024-04-16T21:12:09.876267+0000","last_fullsized":"2024-04-16T21:12:09.876267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.876206+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.314787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.204744+0000","last_change":"2024-04-16T21:12:43.204766+0000","last_active":"2024-04-16T21:12:43.204744+0000","last_peered":"2024-04-16T21:12:43.204744+0000","last_clean":"2024-04-16T21:12:43.204744+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:12:43.204744+0000","last_undegraded":"2024-04-16T21:12:43.204744+0000","last_fullsized":"2024-04-16T21:12:43.204744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.204682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:58.459466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'119","reported_seq":303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.254459+0000","last_change":"2024-04-16T21:13:02.254490+0000","last_active":"2024-04-16T21:13:02.254459+0000","last_peered":"2024-04-16T21:13:02.254459+0000","last_clean":"2024-04-16T21:13:02.254459+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:13:02.254459+0000","last_undegraded":"2024-04-16T21:13:02.254459+0000","last_fullsized":"2024-04-16T21:13:02.254459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'119","last_scrub_stamp":"2024-04-16T21:13:02.254402+0000","last_deep_scrub":"52'119","last_deep_scrub_stamp":"2024-04-16T21:13:02.254402+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.254402+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:45.412979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.823040+0000","last_change":"2024-04-16T21:13:03.823065+0000","last_active":"2024-04-16T21:13:03.823040+0000","last_peered":"2024-04-16T21:13:03.823040+0000","last_clean":"2024-04-16T21:13:03.823040+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:13:03.823040+0000","last_undegraded":"2024-04-16T21:13:03.823040+0000","last_fullsized":"2024-04-16T21:13:03.823040+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.822971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.822971+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.822971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:13:18.427093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.091208+0000","last_change":"2024-04-16T21:12:59.091231+0000","last_active":"2024-04-16T21:12:59.091208+0000","last_peered":"2024-04-16T21:12:59.091208+0000","last_clean":"2024-04-16T21:12:59.091208+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:12:59.091208+0000","last_undegraded":"2024-04-16T21:12:59.091208+0000","last_fullsized":"2024-04-16T21:12:59.091208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.091143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.091143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.091143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:48.532211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.706698+0000","last_change":"2024-04-16T21:12:37.706720+0000","last_active":"2024-04-16T21:12:37.706698+0000","last_peered":"2024-04-16T21:12:37.706698+0000","last_clean":"2024-04-16T21:12:37.706698+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:12:37.706698+0000","last_undegraded":"2024-04-16T21:12:37.706698+0000","last_fullsized":"2024-04-16T21:12:37.706698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.706633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:46.200865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'73","reported_seq":244,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.829156+0000","last_change":"2024-04-16T21:12:32.829190+0000","last_active":"2024-04-16T21:12:32.829156+0000","last_peered":"2024-04-16T21:12:32.829156+0000","last_clean":"2024-04-16T21:12:32.829156+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:12:32.829156+0000","last_undegraded":"2024-04-16T21:12:32.829156+0000","last_fullsized":"2024-04-16T21:12:32.829156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'73","last_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_deep_scrub":"52'73","last_deep_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.829094+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:32.804059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.313554+0000","last_change":"2024-04-16T21:12:28.313577+0000","last_active":"2024-04-16T21:12:28.313554+0000","last_peered":"2024-04-16T21:12:28.313554+0000","last_clean":"2024-04-16T21:12:28.313554+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:12:28.313554+0000","last_undegraded":"2024-04-16T21:12:28.313554+0000","last_fullsized":"2024-04-16T21:12:28.313554+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.313486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:34.551134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.689142+0000","last_change":"2024-04-16T21:12:06.689165+0000","last_active":"2024-04-16T21:12:06.689142+0000","last_peered":"2024-04-16T21:12:06.689142+0000","last_clean":"2024-04-16T21:12:06.689142+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:12:06.689142+0000","last_undegraded":"2024-04-16T21:12:06.689142+0000","last_fullsized":"2024-04-16T21:12:06.689142+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.689080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:58.869391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.806497+0000","last_change":"2024-04-16T21:12:57.806497+0000","last_active":"2024-04-16T21:12:57.806497+0000","last_peered":"2024-04-16T21:12:57.806497+0000","last_clean":"2024-04-16T21:12:57.806497+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:12:57.806497+0000","last_undegraded":"2024-04-16T21:12:57.806497+0000","last_fullsized":"2024-04-16T21:12:57.806497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.806335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.806335+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.806335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:54:38.758338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'183","reported_seq":2009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334593+0000","last_change":"2024-04-16T21:12:30.334647+0000","last_active":"2024-04-16T21:12:30.334593+0000","last_peered":"2024-04-16T21:12:30.334593+0000","last_clean":"2024-04-16T21:12:30.334593+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:12:30.334593+0000","last_undegraded":"2024-04-16T21:12:30.334593+0000","last_fullsized":"2024-04-16T21:12:30.334593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334529+0000","objects_scrubbed":48,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:53.601279+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14209,"num_omap_keys":58,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.856959+0000","last_change":"2024-04-16T21:12:37.856984+0000","last_active":"2024-04-16T21:12:37.856959+0000","last_peered":"2024-04-16T21:12:37.856959+0000","last_clean":"2024-04-16T21:12:37.856959+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:12:37.856959+0000","last_undegraded":"2024-04-16T21:12:37.856959+0000","last_fullsized":"2024-04-16T21:12:37.856959+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.856878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:44.305267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'16","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.513309+0000","last_change":"2024-04-16T21:12:20.513369+0000","last_active":"2024-04-16T21:12:20.513309+0000","last_peered":"2024-04-16T21:12:20.513309+0000","last_clean":"2024-04-16T21:12:20.513309+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:12:20.513309+0000","last_undegraded":"2024-04-16T21:12:20.513309+0000","last_fullsized":"2024-04-16T21:12:20.513309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.513228+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:32.115899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'706","reported_seq":2087,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.915788+0000","last_change":"2024-04-16T21:11:48.915841+0000","last_active":"2024-04-16T21:11:48.915788+0000","last_peered":"2024-04-16T21:11:48.915788+0000","last_clean":"2024-04-16T21:11:48.915788+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:11:48.915788+0000","last_undegraded":"2024-04-16T21:11:48.915788+0000","last_fullsized":"2024-04-16T21:11:48.915788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.915706+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.964684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'326","reported_seq":644,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.279300+0000","last_change":"2024-04-16T21:11:56.279412+0000","last_active":"2024-04-16T21:11:56.279300+0000","last_peered":"2024-04-16T21:11:56.279300+0000","last_clean":"2024-04-16T21:11:56.279300+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:11:56.279300+0000","last_undegraded":"2024-04-16T21:11:56.279300+0000","last_fullsized":"2024-04-16T21:11:56.279300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'326","last_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_deep_scrub":"52'326","last_deep_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.279222+0000","objects_scrubbed":20,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:03.299266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'20","reported_seq":201,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.117827+0000","last_change":"2024-04-16T21:11:45.117905+0000","last_active":"2024-04-16T21:11:45.117827+0000","last_peered":"2024-04-16T21:11:45.117827+0000","last_clean":"2024-04-16T21:11:45.117827+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:11:45.117827+0000","last_undegraded":"2024-04-16T21:11:45.117827+0000","last_fullsized":"2024-04-16T21:11:45.117827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.117757+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:23.210409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.379965+0000","last_change":"2024-04-16T21:12:50.379965+0000","last_active":"2024-04-16T21:12:50.379965+0000","last_peered":"2024-04-16T21:12:50.379965+0000","last_clean":"2024-04-16T21:12:50.379965+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:12:50.379965+0000","last_undegraded":"2024-04-16T21:12:50.379965+0000","last_fullsized":"2024-04-16T21:12:50.379965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.379848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.379848+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.379848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:31.683476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.260284+0000","last_change":"2024-04-16T21:13:03.260307+0000","last_active":"2024-04-16T21:13:03.260284+0000","last_peered":"2024-04-16T21:13:03.260284+0000","last_clean":"2024-04-16T21:13:03.260284+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:13:03.260284+0000","last_undegraded":"2024-04-16T21:13:03.260284+0000","last_fullsized":"2024-04-16T21:13:03.260284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.260223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.260223+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.260223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:58.899912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.496213+0000","last_change":"2024-04-16T21:13:02.496238+0000","last_active":"2024-04-16T21:13:02.496213+0000","last_peered":"2024-04-16T21:13:02.496213+0000","last_clean":"2024-04-16T21:13:02.496213+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:13:02.496213+0000","last_undegraded":"2024-04-16T21:13:02.496213+0000","last_fullsized":"2024-04-16T21:13:02.496213+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.496135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.496135+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.496135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:32.552774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"52'8","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.885381+0000","last_change":"2024-04-16T21:12:40.885408+0000","last_active":"2024-04-16T21:12:40.885381+0000","last_peered":"2024-04-16T21:12:40.885381+0000","last_clean":"2024-04-16T21:12:40.885381+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:12:40.885381+0000","last_undegraded":"2024-04-16T21:12:40.885381+0000","last_fullsized":"2024-04-16T21:12:40.885381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.885321+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:11.985385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'175","reported_seq":1588,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.038998+0000","last_change":"2024-04-16T21:12:33.039035+0000","last_active":"2024-04-16T21:12:33.038998+0000","last_peered":"2024-04-16T21:12:33.038998+0000","last_clean":"2024-04-16T21:12:33.038998+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:12:33.038998+0000","last_undegraded":"2024-04-16T21:12:33.038998+0000","last_fullsized":"2024-04-16T21:12:33.038998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'175","last_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_deep_scrub":"52'175","last_deep_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.038938+0000","objects_scrubbed":10,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:04.725640+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.604589+0000","last_change":"2024-04-16T21:12:24.604613+0000","last_active":"2024-04-16T21:12:24.604589+0000","last_peered":"2024-04-16T21:12:24.604589+0000","last_clean":"2024-04-16T21:12:24.604589+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:12:24.604589+0000","last_undegraded":"2024-04-16T21:12:24.604589+0000","last_fullsized":"2024-04-16T21:12:24.604589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.604487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.850465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.671618+0000","last_change":"2024-04-16T21:11:57.671660+0000","last_active":"2024-04-16T21:11:57.671618+0000","last_peered":"2024-04-16T21:11:57.671618+0000","last_clean":"2024-04-16T21:11:57.671618+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:11:57.671618+0000","last_undegraded":"2024-04-16T21:11:57.671618+0000","last_fullsized":"2024-04-16T21:11:57.671618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.671545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:20.862941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":55,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561681+0000","last_change":"2024-04-16T19:47:33.986931+0000","last_active":"2024-04-16T21:11:44.561681+0000","last_peered":"2024-04-16T21:11:44.561681+0000","last_clean":"2024-04-16T21:11:44.561681+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:11:44.561681+0000","last_undegraded":"2024-04-16T21:11:44.561681+0000","last_fullsized":"2024-04-16T21:11:44.561681+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'203","reported_seq":2370,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.392699+0000","last_change":"2024-04-16T19:25:49.518501+0000","last_active":"2024-04-16T21:11:34.392699+0000","last_peered":"2024-04-16T21:11:34.392699+0000","last_clean":"2024-04-16T21:11:34.392699+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:11:34.392699+0000","last_undegraded":"2024-04-16T21:11:34.392699+0000","last_fullsized":"2024-04-16T21:11:34.392699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.830577+0000","last_change":"2024-04-16T21:12:25.830601+0000","last_active":"2024-04-16T21:12:25.830577+0000","last_peered":"2024-04-16T21:12:25.830577+0000","last_clean":"2024-04-16T21:12:25.830577+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:12:25.830577+0000","last_undegraded":"2024-04-16T21:12:25.830577+0000","last_fullsized":"2024-04-16T21:12:25.830577+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.830513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:35.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'19","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.380245+0000","last_change":"2024-04-16T21:12:21.380273+0000","last_active":"2024-04-16T21:12:21.380245+0000","last_peered":"2024-04-16T21:12:21.380245+0000","last_clean":"2024-04-16T21:12:21.380245+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:12:21.380245+0000","last_undegraded":"2024-04-16T21:12:21.380245+0000","last_fullsized":"2024-04-16T21:12:21.380245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.380184+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:53.066886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":60,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'21","reported_seq":982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.783889+0000","last_change":"2024-04-16T21:11:58.783920+0000","last_active":"2024-04-16T21:11:58.783889+0000","last_peered":"2024-04-16T21:11:58.783889+0000","last_clean":"2024-04-16T21:11:58.783889+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:11:58.783889+0000","last_undegraded":"2024-04-16T21:11:58.783889+0000","last_fullsized":"2024-04-16T21:11:58.783889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.783819+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:59.285203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'268","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.359130+0000","last_change":"2024-04-16T21:11:44.359183+0000","last_active":"2024-04-16T21:11:44.359130+0000","last_peered":"2024-04-16T21:11:44.359130+0000","last_clean":"2024-04-16T21:11:44.359130+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:11:44.359130+0000","last_undegraded":"2024-04-16T21:11:44.359130+0000","last_fullsized":"2024-04-16T21:11:44.359130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'268","last_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_deep_scrub":"52'268","last_deep_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.359071+0000","objects_scrubbed":19,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:18.502556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'13","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.427731+0000","last_change":"2024-04-16T21:11:53.427771+0000","last_active":"2024-04-16T21:11:53.427731+0000","last_peered":"2024-04-16T21:11:53.427731+0000","last_clean":"2024-04-16T21:11:53.427731+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:11:53.427731+0000","last_undegraded":"2024-04-16T21:11:53.427731+0000","last_fullsized":"2024-04-16T21:11:53.427731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.427656+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:14.929241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.080980+0000","last_change":"2024-04-16T21:12:42.080980+0000","last_active":"2024-04-16T21:12:42.080980+0000","last_peered":"2024-04-16T21:12:42.080980+0000","last_clean":"2024-04-16T21:12:42.080980+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:12:42.080980+0000","last_undegraded":"2024-04-16T21:12:42.080980+0000","last_fullsized":"2024-04-16T21:12:42.080980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.080859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:48.529247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.765443+0000","last_change":"2024-04-16T21:12:42.765465+0000","last_active":"2024-04-16T21:12:42.765443+0000","last_peered":"2024-04-16T21:12:42.765443+0000","last_clean":"2024-04-16T21:12:42.765443+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:12:42.765443+0000","last_undegraded":"2024-04-16T21:12:42.765443+0000","last_fullsized":"2024-04-16T21:12:42.765443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.765375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:30.897866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.114045+0000","last_change":"2024-04-16T21:12:52.114045+0000","last_active":"2024-04-16T21:12:52.114045+0000","last_peered":"2024-04-16T21:12:52.114045+0000","last_clean":"2024-04-16T21:12:52.114045+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:12:52.114045+0000","last_undegraded":"2024-04-16T21:12:52.114045+0000","last_fullsized":"2024-04-16T21:12:52.114045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.113954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.113954+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.113954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:36.871712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"52'6","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.801277+0000","last_change":"2024-04-16T21:12:34.801302+0000","last_active":"2024-04-16T21:12:34.801277+0000","last_peered":"2024-04-16T21:12:34.801277+0000","last_clean":"2024-04-16T21:12:34.801277+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:12:34.801277+0000","last_undegraded":"2024-04-16T21:12:34.801277+0000","last_fullsized":"2024-04-16T21:12:34.801277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.801209+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:13.234565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'147","reported_seq":1530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.863577+0000","last_change":"2024-04-16T21:12:39.863616+0000","last_active":"2024-04-16T21:12:39.863577+0000","last_peered":"2024-04-16T21:12:39.863577+0000","last_clean":"2024-04-16T21:12:39.863577+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:12:39.863577+0000","last_undegraded":"2024-04-16T21:12:39.863577+0000","last_fullsized":"2024-04-16T21:12:39.863577+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.863516+0000","objects_scrubbed":8,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:55.662812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.339655+0000","last_change":"2024-04-16T21:12:27.339693+0000","last_active":"2024-04-16T21:12:27.339655+0000","last_peered":"2024-04-16T21:12:27.339655+0000","last_clean":"2024-04-16T21:12:27.339655+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:12:27.339655+0000","last_undegraded":"2024-04-16T21:12:27.339655+0000","last_fullsized":"2024-04-16T21:12:27.339655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.339563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:33.787952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.370546+0000","last_change":"2024-04-16T21:12:05.370546+0000","last_active":"2024-04-16T21:12:05.370546+0000","last_peered":"2024-04-16T21:12:05.370546+0000","last_clean":"2024-04-16T21:12:05.370546+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:12:05.370546+0000","last_undegraded":"2024-04-16T21:12:05.370546+0000","last_fullsized":"2024-04-16T21:12:05.370546+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.370430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:17.165162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":54,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.388785+0000","last_change":"2024-04-16T19:47:33.987919+0000","last_active":"2024-04-16T21:11:04.388785+0000","last_peered":"2024-04-16T21:11:04.388785+0000","last_clean":"2024-04-16T21:11:04.388785+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:11:04.388785+0000","last_undegraded":"2024-04-16T21:11:04.388785+0000","last_fullsized":"2024-04-16T21:11:04.388785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'161","reported_seq":1820,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.785185+0000","last_change":"2024-04-16T21:12:56.785241+0000","last_active":"2024-04-16T21:12:56.785185+0000","last_peered":"2024-04-16T21:12:56.785185+0000","last_clean":"2024-04-16T21:12:56.785185+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:12:56.785185+0000","last_undegraded":"2024-04-16T21:12:56.785185+0000","last_fullsized":"2024-04-16T21:12:56.785185+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'161","last_scrub_stamp":"2024-04-16T21:12:56.785125+0000","last_deep_scrub":"52'161","last_deep_scrub_stamp":"2024-04-16T21:12:56.785125+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.785125+0000","objects_scrubbed":44,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:49:38.484287+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9743,"num_omap_keys":41,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.699415+0000","last_change":"2024-04-16T21:11:59.699455+0000","last_active":"2024-04-16T21:11:59.699415+0000","last_peered":"2024-04-16T21:11:59.699415+0000","last_clean":"2024-04-16T21:11:59.699415+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:11:59.699415+0000","last_undegraded":"2024-04-16T21:11:59.699415+0000","last_fullsized":"2024-04-16T21:11:59.699415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.699336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'21","reported_seq":188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.432050+0000","last_change":"2024-04-16T21:12:15.432112+0000","last_active":"2024-04-16T21:12:15.432050+0000","last_peered":"2024-04-16T21:12:15.432050+0000","last_clean":"2024-04-16T21:12:15.432050+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:12:15.432050+0000","last_undegraded":"2024-04-16T21:12:15.432050+0000","last_fullsized":"2024-04-16T21:12:15.432050+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.431977+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:24.993645+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.891648+0000","last_change":"2024-04-16T21:12:03.891673+0000","last_active":"2024-04-16T21:12:03.891648+0000","last_peered":"2024-04-16T21:12:03.891648+0000","last_clean":"2024-04-16T21:12:03.891648+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:12:03.891648+0000","last_undegraded":"2024-04-16T21:12:03.891648+0000","last_fullsized":"2024-04-16T21:12:03.891648+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.891582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:27.082978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'720","reported_seq":1730,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.792306+0000","last_change":"2024-04-16T21:11:44.792348+0000","last_active":"2024-04-16T21:11:44.792306+0000","last_peered":"2024-04-16T21:11:44.792306+0000","last_clean":"2024-04-16T21:11:44.792306+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:11:44.792306+0000","last_undegraded":"2024-04-16T21:11:44.792306+0000","last_fullsized":"2024-04-16T21:11:44.792306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'720","last_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_deep_scrub":"52'720","last_deep_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.792244+0000","objects_scrubbed":30,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:07:06.408902+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2191,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'20","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.222406+0000","last_change":"2024-04-16T21:11:49.222443+0000","last_active":"2024-04-16T21:11:49.222406+0000","last_peered":"2024-04-16T21:11:49.222406+0000","last_clean":"2024-04-16T21:11:49.222406+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:11:49.222406+0000","last_undegraded":"2024-04-16T21:11:49.222406+0000","last_fullsized":"2024-04-16T21:11:49.222406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.222345+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:21.815371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.118034+0000","last_change":"2024-04-16T21:12:48.118034+0000","last_active":"2024-04-16T21:12:48.118034+0000","last_peered":"2024-04-16T21:12:48.118034+0000","last_clean":"2024-04-16T21:12:48.118034+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:12:48.118034+0000","last_undegraded":"2024-04-16T21:12:48.118034+0000","last_fullsized":"2024-04-16T21:12:48.118034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.117906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.117906+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.117906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:45.093346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560100+0000","last_change":"2024-04-16T19:23:47.119737+0000","last_active":"2024-04-16T21:11:44.560100+0000","last_peered":"2024-04-16T21:11:44.560100+0000","last_clean":"2024-04-16T21:11:44.560100+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:11:44.560100+0000","last_undegraded":"2024-04-16T21:11:44.560100+0000","last_fullsized":"2024-04-16T21:11:44.560100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.707830+0000","last_change":"2024-04-16T21:12:47.707830+0000","last_active":"2024-04-16T21:12:47.707830+0000","last_peered":"2024-04-16T21:12:47.707830+0000","last_clean":"2024-04-16T21:12:47.707830+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:12:47.707830+0000","last_undegraded":"2024-04-16T21:12:47.707830+0000","last_fullsized":"2024-04-16T21:12:47.707830+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.707745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.707745+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.707745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:54.156133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.476532+0000","last_change":"2024-04-16T21:12:58.476558+0000","last_active":"2024-04-16T21:12:58.476532+0000","last_peered":"2024-04-16T21:12:58.476532+0000","last_clean":"2024-04-16T21:12:58.476532+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:12:58.476532+0000","last_undegraded":"2024-04-16T21:12:58.476532+0000","last_fullsized":"2024-04-16T21:12:58.476532+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.476463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.476463+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.476463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:31.485168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'138","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.993884+0000","last_change":"2024-04-16T21:12:33.993917+0000","last_active":"2024-04-16T21:12:33.993884+0000","last_peered":"2024-04-16T21:12:33.993884+0000","last_clean":"2024-04-16T21:12:33.993884+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:12:33.993884+0000","last_undegraded":"2024-04-16T21:12:33.993884+0000","last_fullsized":"2024-04-16T21:12:33.993884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.993823+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:28.182498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.394475+0000","last_change":"2024-04-16T21:12:09.394517+0000","last_active":"2024-04-16T21:12:09.394475+0000","last_peered":"2024-04-16T21:12:09.394475+0000","last_clean":"2024-04-16T21:12:09.394475+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:12:09.394475+0000","last_undegraded":"2024-04-16T21:12:09.394475+0000","last_fullsized":"2024-04-16T21:12:09.394475+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.394403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:58.357952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.902684+0000","last_change":"2024-04-16T21:12:14.902708+0000","last_active":"2024-04-16T21:12:14.902684+0000","last_peered":"2024-04-16T21:12:14.902684+0000","last_clean":"2024-04-16T21:12:14.902684+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:12:14.902684+0000","last_undegraded":"2024-04-16T21:12:14.902684+0000","last_fullsized":"2024-04-16T21:12:14.902684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.902624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:23.524467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561653+0000","last_change":"2024-04-16T19:47:33.982029+0000","last_active":"2024-04-16T21:11:44.561653+0000","last_peered":"2024-04-16T21:11:44.561653+0000","last_clean":"2024-04-16T21:11:44.561653+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:11:44.561653+0000","last_undegraded":"2024-04-16T21:11:44.561653+0000","last_fullsized":"2024-04-16T21:11:44.561653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'106","reported_seq":1336,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293264+0000","last_change":"2024-04-16T19:25:49.521305+0000","last_active":"2024-04-16T21:11:42.293264+0000","last_peered":"2024-04-16T21:11:42.293264+0000","last_clean":"2024-04-16T21:11:42.293264+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:11:42.293264+0000","last_undegraded":"2024-04-16T21:11:42.293264+0000","last_fullsized":"2024-04-16T21:11:42.293264+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.320686+0000","last_change":"2024-04-16T21:12:28.320708+0000","last_active":"2024-04-16T21:12:28.320686+0000","last_peered":"2024-04-16T21:12:28.320686+0000","last_clean":"2024-04-16T21:12:28.320686+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:12:28.320686+0000","last_undegraded":"2024-04-16T21:12:28.320686+0000","last_fullsized":"2024-04-16T21:12:28.320686+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.320608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:14.618412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'6","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.781379+0000","last_change":"2024-04-16T21:11:54.781414+0000","last_active":"2024-04-16T21:11:54.781379+0000","last_peered":"2024-04-16T21:11:54.781379+0000","last_clean":"2024-04-16T21:11:54.781379+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:11:54.781379+0000","last_undegraded":"2024-04-16T21:11:54.781379+0000","last_fullsized":"2024-04-16T21:11:54.781379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.781318+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.682337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'706","reported_seq":2169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.804811+0000","last_change":"2024-04-16T21:11:57.804839+0000","last_active":"2024-04-16T21:11:57.804811+0000","last_peered":"2024-04-16T21:11:57.804811+0000","last_clean":"2024-04-16T21:11:57.804811+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:11:57.804811+0000","last_undegraded":"2024-04-16T21:11:57.804811+0000","last_fullsized":"2024-04-16T21:11:57.804811+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.804739+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:38:55.231266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'264","reported_seq":495,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.746009+0000","last_change":"2024-04-16T21:11:46.746078+0000","last_active":"2024-04-16T21:11:46.746009+0000","last_peered":"2024-04-16T21:11:46.746009+0000","last_clean":"2024-04-16T21:11:46.746009+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:11:46.746009+0000","last_undegraded":"2024-04-16T21:11:46.746009+0000","last_fullsized":"2024-04-16T21:11:46.746009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.745946+0000","objects_scrubbed":17,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:43.933138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1751,"num_omap_keys":8,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'38","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.693339+0000","last_change":"2024-04-16T21:11:52.693388+0000","last_active":"2024-04-16T21:11:52.693339+0000","last_peered":"2024-04-16T21:11:52.693339+0000","last_clean":"2024-04-16T21:11:52.693339+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:11:52.693339+0000","last_undegraded":"2024-04-16T21:11:52.693339+0000","last_fullsized":"2024-04-16T21:11:52.693339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'38","last_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_deep_scrub":"52'38","last_deep_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.693273+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:15.884668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.125718+0000","last_change":"2024-04-16T21:12:43.125718+0000","last_active":"2024-04-16T21:12:43.125718+0000","last_peered":"2024-04-16T21:12:43.125718+0000","last_clean":"2024-04-16T21:12:43.125718+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:12:43.125718+0000","last_undegraded":"2024-04-16T21:12:43.125718+0000","last_fullsized":"2024-04-16T21:12:43.125718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.125631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:49.363279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.901348+0000","last_change":"2024-04-16T21:12:52.901371+0000","last_active":"2024-04-16T21:12:52.901348+0000","last_peered":"2024-04-16T21:12:52.901348+0000","last_clean":"2024-04-16T21:12:52.901348+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:12:52.901348+0000","last_undegraded":"2024-04-16T21:12:52.901348+0000","last_fullsized":"2024-04-16T21:12:52.901348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.901289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.901289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.901289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:59.588023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.102912+0000","last_change":"2024-04-16T21:12:54.102935+0000","last_active":"2024-04-16T21:12:54.102912+0000","last_peered":"2024-04-16T21:12:54.102912+0000","last_clean":"2024-04-16T21:12:54.102912+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:12:54.102912+0000","last_undegraded":"2024-04-16T21:12:54.102912+0000","last_fullsized":"2024-04-16T21:12:54.102912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.102847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.102847+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.102847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:25.807640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.378796+0000","last_change":"2024-04-16T21:12:43.378818+0000","last_active":"2024-04-16T21:12:43.378796+0000","last_peered":"2024-04-16T21:12:43.378796+0000","last_clean":"2024-04-16T21:12:43.378796+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:12:43.378796+0000","last_undegraded":"2024-04-16T21:12:43.378796+0000","last_fullsized":"2024-04-16T21:12:43.378796+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.378722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:20.670430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'156","reported_seq":366,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.280884+0000","last_change":"2024-04-16T21:12:37.280951+0000","last_active":"2024-04-16T21:12:37.280884+0000","last_peered":"2024-04-16T21:12:37.280884+0000","last_clean":"2024-04-16T21:12:37.280884+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:12:37.280884+0000","last_undegraded":"2024-04-16T21:12:37.280884+0000","last_fullsized":"2024-04-16T21:12:37.280884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'156","last_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_deep_scrub":"52'156","last_deep_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.280808+0000","objects_scrubbed":8,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:22.038566+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.612982+0000","last_change":"2024-04-16T21:12:26.613002+0000","last_active":"2024-04-16T21:12:26.612982+0000","last_peered":"2024-04-16T21:12:26.612982+0000","last_clean":"2024-04-16T21:12:26.612982+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:12:26.612982+0000","last_undegraded":"2024-04-16T21:12:26.612982+0000","last_fullsized":"2024-04-16T21:12:26.612982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.612912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.407643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.345418+0000","last_change":"2024-04-16T21:12:01.345418+0000","last_active":"2024-04-16T21:12:01.345418+0000","last_peered":"2024-04-16T21:12:01.345418+0000","last_clean":"2024-04-16T21:12:01.345418+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:12:01.345418+0000","last_undegraded":"2024-04-16T21:12:01.345418+0000","last_fullsized":"2024-04-16T21:12:01.345418+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.345289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:22.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.909918+0000","last_change":"2024-04-16T21:12:24.909962+0000","last_active":"2024-04-16T21:12:24.909918+0000","last_peered":"2024-04-16T21:12:24.909918+0000","last_clean":"2024-04-16T21:12:24.909918+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:12:24.909918+0000","last_undegraded":"2024-04-16T21:12:24.909918+0000","last_fullsized":"2024-04-16T21:12:24.909918+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.909836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:30.648976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'194","reported_seq":2060,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.906040+0000","last_change":"2024-04-16T21:12:41.906084+0000","last_active":"2024-04-16T21:12:41.906040+0000","last_peered":"2024-04-16T21:12:41.906040+0000","last_clean":"2024-04-16T21:12:41.906040+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:12:41.906040+0000","last_undegraded":"2024-04-16T21:12:41.906040+0000","last_fullsized":"2024-04-16T21:12:41.906040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'194","last_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_deep_scrub":"52'194","last_deep_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.905983+0000","objects_scrubbed":47,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:44.692749+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12654,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.836131+0000","last_change":"2024-04-16T21:12:08.836131+0000","last_active":"2024-04-16T21:12:08.836131+0000","last_peered":"2024-04-16T21:12:08.836131+0000","last_clean":"2024-04-16T21:12:08.836131+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:12:08.836131+0000","last_undegraded":"2024-04-16T21:12:08.836131+0000","last_fullsized":"2024-04-16T21:12:08.836131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.836015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:33.006060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'30","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.417236+0000","last_change":"2024-04-16T21:12:16.417277+0000","last_active":"2024-04-16T21:12:16.417236+0000","last_peered":"2024-04-16T21:12:16.417236+0000","last_clean":"2024-04-16T21:12:16.417236+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:12:16.417236+0000","last_undegraded":"2024-04-16T21:12:16.417236+0000","last_fullsized":"2024-04-16T21:12:16.417236+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'30","last_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_deep_scrub":"52'30","last_deep_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.417171+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:08.090860+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.802564+0000","last_change":"2024-04-16T21:12:02.802585+0000","last_active":"2024-04-16T21:12:02.802564+0000","last_peered":"2024-04-16T21:12:02.802564+0000","last_clean":"2024-04-16T21:12:02.802564+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:12:02.802564+0000","last_undegraded":"2024-04-16T21:12:02.802564+0000","last_fullsized":"2024-04-16T21:12:02.802564+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.802497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:44.917953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'739","reported_seq":1609,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.752466+0000","last_change":"2024-04-16T21:11:47.752492+0000","last_active":"2024-04-16T21:11:47.752466+0000","last_peered":"2024-04-16T21:11:47.752466+0000","last_clean":"2024-04-16T21:11:47.752466+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:11:47.752466+0000","last_undegraded":"2024-04-16T21:11:47.752466+0000","last_fullsized":"2024-04-16T21:11:47.752466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'739","last_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_deep_scrub":"52'739","last_deep_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.752406+0000","objects_scrubbed":25,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:14.225249+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6099,"num_omap_keys":28,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'12","reported_seq":283,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.362460+0000","last_change":"2024-04-16T21:11:45.362519+0000","last_active":"2024-04-16T21:11:45.362460+0000","last_peered":"2024-04-16T21:11:45.362460+0000","last_clean":"2024-04-16T21:11:45.362460+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:11:45.362460+0000","last_undegraded":"2024-04-16T21:11:45.362460+0000","last_fullsized":"2024-04-16T21:11:45.362460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.362399+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:08.553795+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.447089+0000","last_change":"2024-04-16T21:12:56.447089+0000","last_active":"2024-04-16T21:12:56.447089+0000","last_peered":"2024-04-16T21:12:56.447089+0000","last_clean":"2024-04-16T21:12:56.447089+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:12:56.447089+0000","last_undegraded":"2024-04-16T21:12:56.447089+0000","last_fullsized":"2024-04-16T21:12:56.447089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.446999+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:20.399918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.123738+0000","last_change":"2024-04-16T21:12:56.123791+0000","last_active":"2024-04-16T21:12:56.123738+0000","last_peered":"2024-04-16T21:12:56.123738+0000","last_clean":"2024-04-16T21:12:56.123738+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:12:56.123738+0000","last_undegraded":"2024-04-16T21:12:56.123738+0000","last_fullsized":"2024-04-16T21:12:56.123738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.123634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.123634+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.123634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:50.704912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.649882+0000","last_change":"2024-04-16T19:23:47.184206+0000","last_active":"2024-04-16T21:11:43.649882+0000","last_peered":"2024-04-16T21:11:43.649882+0000","last_clean":"2024-04-16T21:11:43.649882+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:11:43.649882+0000","last_undegraded":"2024-04-16T21:11:43.649882+0000","last_fullsized":"2024-04-16T21:11:43.649882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:23:46.108852+0000","last_clean_scrub_stamp":"2024-04-16T19:23:46.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.786089+0000","last_change":"2024-04-16T21:12:46.786111+0000","last_active":"2024-04-16T21:12:46.786089+0000","last_peered":"2024-04-16T21:12:46.786089+0000","last_clean":"2024-04-16T21:12:46.786089+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:12:46.786089+0000","last_undegraded":"2024-04-16T21:12:46.786089+0000","last_fullsized":"2024-04-16T21:12:46.786089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.786030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.786030+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.786030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:54:27.738032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'161","reported_seq":362,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.967720+0000","last_change":"2024-04-16T21:12:27.967756+0000","last_active":"2024-04-16T21:12:27.967720+0000","last_peered":"2024-04-16T21:12:27.967720+0000","last_clean":"2024-04-16T21:12:27.967720+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:12:27.967720+0000","last_undegraded":"2024-04-16T21:12:27.967720+0000","last_fullsized":"2024-04-16T21:12:27.967720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'161","last_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_deep_scrub":"52'161","last_deep_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.967658+0000","objects_scrubbed":7,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:06.401014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'13","reported_seq":1327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.766646+0000","last_change":"2024-04-16T21:12:27.766675+0000","last_active":"2024-04-16T21:12:27.766646+0000","last_peered":"2024-04-16T21:12:27.766646+0000","last_clean":"2024-04-16T21:12:27.766646+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:12:27.766646+0000","last_undegraded":"2024-04-16T21:12:27.766646+0000","last_fullsized":"2024-04-16T21:12:27.766646+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.766582+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:36.260814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.393316+0000","last_change":"2024-04-16T21:12:00.393341+0000","last_active":"2024-04-16T21:12:00.393316+0000","last_peered":"2024-04-16T21:12:00.393316+0000","last_clean":"2024-04-16T21:12:00.393316+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:12:00.393316+0000","last_undegraded":"2024-04-16T21:12:00.393316+0000","last_fullsized":"2024-04-16T21:12:00.393316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.393227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:57.237156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.796382+0000","last_change":"2024-04-16T21:12:57.796382+0000","last_active":"2024-04-16T21:12:57.796382+0000","last_peered":"2024-04-16T21:12:57.796382+0000","last_clean":"2024-04-16T21:12:57.796382+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:12:57.796382+0000","last_undegraded":"2024-04-16T21:12:57.796382+0000","last_fullsized":"2024-04-16T21:12:57.796382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.796292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.796292+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.796292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:12.374830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'148","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.332144+0000","last_change":"2024-04-16T21:12:36.332178+0000","last_active":"2024-04-16T21:12:36.332144+0000","last_peered":"2024-04-16T21:12:36.332144+0000","last_clean":"2024-04-16T21:12:36.332144+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:12:36.332144+0000","last_undegraded":"2024-04-16T21:12:36.332144+0000","last_fullsized":"2024-04-16T21:12:36.332144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'148","last_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_deep_scrub":"52'148","last_deep_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.332084+0000","objects_scrubbed":32,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:33.307525+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11265,"num_omap_keys":45,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.939107+0000","last_change":"2024-04-16T21:12:19.939131+0000","last_active":"2024-04-16T21:12:19.939107+0000","last_peered":"2024-04-16T21:12:19.939107+0000","last_clean":"2024-04-16T21:12:19.939107+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:12:19.939107+0000","last_undegraded":"2024-04-16T21:12:19.939107+0000","last_fullsized":"2024-04-16T21:12:19.939107+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.939043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:23.022714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'4","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.763978+0000","last_change":"2024-04-16T21:11:55.764028+0000","last_active":"2024-04-16T21:11:55.763978+0000","last_peered":"2024-04-16T21:11:55.763978+0000","last_clean":"2024-04-16T21:11:55.763978+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:11:55.763978+0000","last_undegraded":"2024-04-16T21:11:55.763978+0000","last_fullsized":"2024-04-16T21:11:55.763978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.763887+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.184478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.416929+0000","last_change":"2024-04-16T21:12:12.416959+0000","last_active":"2024-04-16T21:12:12.416929+0000","last_peered":"2024-04-16T21:12:12.416929+0000","last_clean":"2024-04-16T21:12:12.416929+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:12:12.416929+0000","last_undegraded":"2024-04-16T21:12:12.416929+0000","last_fullsized":"2024-04-16T21:12:12.416929+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.416865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:15.500537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'287","reported_seq":553,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.658905+0000","last_change":"2024-04-16T21:11:54.658986+0000","last_active":"2024-04-16T21:11:54.658905+0000","last_peered":"2024-04-16T21:11:54.658905+0000","last_clean":"2024-04-16T21:11:54.658905+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:11:54.658905+0000","last_undegraded":"2024-04-16T21:11:54.658905+0000","last_fullsized":"2024-04-16T21:11:54.658905+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'287","last_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_deep_scrub":"52'287","last_deep_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.658842+0000","objects_scrubbed":20,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.559862+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6346,"num_omap_keys":29,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'22","reported_seq":213,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.414533+0000","last_change":"2024-04-16T21:11:51.414602+0000","last_active":"2024-04-16T21:11:51.414533+0000","last_peered":"2024-04-16T21:11:51.414533+0000","last_clean":"2024-04-16T21:11:51.414533+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:11:51.414533+0000","last_undegraded":"2024-04-16T21:11:51.414533+0000","last_fullsized":"2024-04-16T21:11:51.414533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.414471+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:12.853053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.340615+0000","last_change":"2024-04-16T21:12:46.340615+0000","last_active":"2024-04-16T21:12:46.340615+0000","last_peered":"2024-04-16T21:12:46.340615+0000","last_clean":"2024-04-16T21:12:46.340615+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:12:46.340615+0000","last_undegraded":"2024-04-16T21:12:46.340615+0000","last_fullsized":"2024-04-16T21:12:46.340615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.340544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.340544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.340544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:58.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.719433+0000","last_change":"2024-04-16T21:12:43.719433+0000","last_active":"2024-04-16T21:12:43.719433+0000","last_peered":"2024-04-16T21:12:43.719433+0000","last_clean":"2024-04-16T21:12:43.719433+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:12:43.719433+0000","last_undegraded":"2024-04-16T21:12:43.719433+0000","last_fullsized":"2024-04-16T21:12:43.719433+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.719343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:43.694308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.134053+0000","last_change":"2024-04-16T21:12:53.134077+0000","last_active":"2024-04-16T21:12:53.134053+0000","last_peered":"2024-04-16T21:12:53.134053+0000","last_clean":"2024-04-16T21:12:53.134053+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:12:53.134053+0000","last_undegraded":"2024-04-16T21:12:53.134053+0000","last_fullsized":"2024-04-16T21:12:53.134053+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.133983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.133983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.133983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:22.123641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.211328+0000","last_change":"2024-04-16T21:12:42.211369+0000","last_active":"2024-04-16T21:12:42.211328+0000","last_peered":"2024-04-16T21:12:42.211328+0000","last_clean":"2024-04-16T21:12:42.211328+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:12:42.211328+0000","last_undegraded":"2024-04-16T21:12:42.211328+0000","last_fullsized":"2024-04-16T21:12:42.211328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.211237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:13.916029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'115","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.353424+0000","last_change":"2024-04-16T21:12:35.353459+0000","last_active":"2024-04-16T21:12:35.353424+0000","last_peered":"2024-04-16T21:12:35.353424+0000","last_clean":"2024-04-16T21:12:35.353424+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:12:35.353424+0000","last_undegraded":"2024-04-16T21:12:35.353424+0000","last_fullsized":"2024-04-16T21:12:35.353424+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.353361+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:35.315569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'13","reported_seq":1325,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.412638+0000","last_change":"2024-04-16T21:12:24.412664+0000","last_active":"2024-04-16T21:12:24.412638+0000","last_peered":"2024-04-16T21:12:24.412638+0000","last_clean":"2024-04-16T21:12:24.412638+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:12:24.412638+0000","last_undegraded":"2024-04-16T21:12:24.412638+0000","last_fullsized":"2024-04-16T21:12:24.412638+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.412571+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:12.450226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.373240+0000","last_change":"2024-04-16T21:11:59.373240+0000","last_active":"2024-04-16T21:11:59.373240+0000","last_peered":"2024-04-16T21:11:59.373240+0000","last_clean":"2024-04-16T21:11:59.373240+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:11:59.373240+0000","last_undegraded":"2024-04-16T21:11:59.373240+0000","last_fullsized":"2024-04-16T21:11:59.373240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.373112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:02.682886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.829001+0000","last_change":"2024-04-16T21:12:56.829025+0000","last_active":"2024-04-16T21:12:56.829001+0000","last_peered":"2024-04-16T21:12:56.829001+0000","last_clean":"2024-04-16T21:12:56.829001+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:12:56.829001+0000","last_undegraded":"2024-04-16T21:12:56.829001+0000","last_fullsized":"2024-04-16T21:12:56.829001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.828939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.828939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.828939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:23.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'112","reported_seq":1523,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561560+0000","last_change":"2024-04-16T19:26:41.570555+0000","last_active":"2024-04-16T21:11:44.561560+0000","last_peered":"2024-04-16T21:11:44.561560+0000","last_clean":"2024-04-16T21:11:44.561560+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:11:44.561560+0000","last_undegraded":"2024-04-16T21:11:44.561560+0000","last_fullsized":"2024-04-16T21:11:44.561560+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.846610+0000","last_change":"2024-04-16T21:12:00.846636+0000","last_active":"2024-04-16T21:12:00.846610+0000","last_peered":"2024-04-16T21:12:00.846610+0000","last_clean":"2024-04-16T21:12:00.846610+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:12:00.846610+0000","last_undegraded":"2024-04-16T21:12:00.846610+0000","last_fullsized":"2024-04-16T21:12:00.846610+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.846545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:04.784544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'15","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.794385+0000","last_change":"2024-04-16T21:11:56.794414+0000","last_active":"2024-04-16T21:11:56.794385+0000","last_peered":"2024-04-16T21:11:56.794385+0000","last_clean":"2024-04-16T21:11:56.794385+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:11:56.794385+0000","last_undegraded":"2024-04-16T21:11:56.794385+0000","last_fullsized":"2024-04-16T21:11:56.794385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.794323+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.217325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'706","reported_seq":2090,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.681698+0000","last_change":"2024-04-16T21:11:49.681723+0000","last_active":"2024-04-16T21:11:49.681698+0000","last_peered":"2024-04-16T21:11:49.681698+0000","last_clean":"2024-04-16T21:11:49.681698+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:11:49.681698+0000","last_undegraded":"2024-04-16T21:11:49.681698+0000","last_fullsized":"2024-04-16T21:11:49.681698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.681634+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.077508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'335","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.410155+0000","last_change":"2024-04-16T21:11:50.410223+0000","last_active":"2024-04-16T21:11:50.410155+0000","last_peered":"2024-04-16T21:11:50.410155+0000","last_clean":"2024-04-16T21:11:50.410155+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:11:50.410155+0000","last_undegraded":"2024-04-16T21:11:50.410155+0000","last_fullsized":"2024-04-16T21:11:50.410155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'335","last_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_deep_scrub":"52'335","last_deep_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.410083+0000","objects_scrubbed":17,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:14.580127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'10","reported_seq":195,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.912474+0000","last_change":"2024-04-16T21:12:10.912515+0000","last_active":"2024-04-16T21:12:10.912474+0000","last_peered":"2024-04-16T21:12:10.912474+0000","last_clean":"2024-04-16T21:12:10.912474+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:12:10.912474+0000","last_undegraded":"2024-04-16T21:12:10.912474+0000","last_fullsized":"2024-04-16T21:12:10.912474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.912417+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.912619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.743600+0000","last_change":"2024-04-16T21:12:31.743600+0000","last_active":"2024-04-16T21:12:31.743600+0000","last_peered":"2024-04-16T21:12:31.743600+0000","last_clean":"2024-04-16T21:12:31.743600+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:12:31.743600+0000","last_undegraded":"2024-04-16T21:12:31.743600+0000","last_fullsized":"2024-04-16T21:12:31.743600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.743511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:22.862472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.495161+0000","last_change":"2024-04-16T21:13:00.495206+0000","last_active":"2024-04-16T21:13:00.495161+0000","last_peered":"2024-04-16T21:13:00.495161+0000","last_clean":"2024-04-16T21:13:00.495161+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:13:00.495161+0000","last_undegraded":"2024-04-16T21:13:00.495161+0000","last_fullsized":"2024-04-16T21:13:00.495161+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.495078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.495078+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.495078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:35.162947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.220313+0000","last_change":"2024-04-16T21:13:04.220348+0000","last_active":"2024-04-16T21:13:04.220313+0000","last_peered":"2024-04-16T21:13:04.220313+0000","last_clean":"2024-04-16T21:13:04.220313+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:13:04.220313+0000","last_undegraded":"2024-04-16T21:13:04.220313+0000","last_fullsized":"2024-04-16T21:13:04.220313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:04.220225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:04.220225+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.220225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:13:55.396780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.901885+0000","last_change":"2024-04-16T21:12:28.901885+0000","last_active":"2024-04-16T21:12:28.901885+0000","last_peered":"2024-04-16T21:12:28.901885+0000","last_clean":"2024-04-16T21:12:28.901885+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:12:28.901885+0000","last_undegraded":"2024-04-16T21:12:28.901885+0000","last_fullsized":"2024-04-16T21:12:28.901885+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.901794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:33.940739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'142","reported_seq":348,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.606314+0000","last_change":"2024-04-16T21:12:27.606351+0000","last_active":"2024-04-16T21:12:27.606314+0000","last_peered":"2024-04-16T21:12:27.606314+0000","last_clean":"2024-04-16T21:12:27.606314+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:12:27.606314+0000","last_undegraded":"2024-04-16T21:12:27.606314+0000","last_fullsized":"2024-04-16T21:12:27.606314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'142","last_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_deep_scrub":"52'142","last_deep_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.606253+0000","objects_scrubbed":8,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:18.166355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.903416+0000","last_change":"2024-04-16T21:12:17.903439+0000","last_active":"2024-04-16T21:12:17.903416+0000","last_peered":"2024-04-16T21:12:17.903416+0000","last_clean":"2024-04-16T21:12:17.903416+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:12:17.903416+0000","last_undegraded":"2024-04-16T21:12:17.903416+0000","last_fullsized":"2024-04-16T21:12:17.903416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.903352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:24.923396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.864551+0000","last_change":"2024-04-16T21:12:11.864551+0000","last_active":"2024-04-16T21:12:11.864551+0000","last_peered":"2024-04-16T21:12:11.864551+0000","last_clean":"2024-04-16T21:12:11.864551+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:12:11.864551+0000","last_undegraded":"2024-04-16T21:12:11.864551+0000","last_fullsized":"2024-04-16T21:12:11.864551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.864468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:37.984544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293397+0000","last_change":"2024-04-16T19:47:33.980696+0000","last_active":"2024-04-16T21:11:42.293397+0000","last_peered":"2024-04-16T21:11:42.293397+0000","last_clean":"2024-04-16T21:11:42.293397+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:11:42.293397+0000","last_undegraded":"2024-04-16T21:11:42.293397+0000","last_fullsized":"2024-04-16T21:11:42.293397+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'113","reported_seq":1571,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651424+0000","last_change":"2024-04-16T19:25:49.520596+0000","last_active":"2024-04-16T21:11:43.651424+0000","last_peered":"2024-04-16T21:11:43.651424+0000","last_clean":"2024-04-16T21:11:43.651424+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:11:43.651424+0000","last_undegraded":"2024-04-16T21:11:43.651424+0000","last_fullsized":"2024-04-16T21:11:43.651424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:25:48.497865+0000","last_clean_scrub_stamp":"2024-04-16T19:25:48.497865+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.869006+0000","last_change":"2024-04-16T21:12:06.869029+0000","last_active":"2024-04-16T21:12:06.869006+0000","last_peered":"2024-04-16T21:12:06.869006+0000","last_clean":"2024-04-16T21:12:06.869006+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:12:06.869006+0000","last_undegraded":"2024-04-16T21:12:06.869006+0000","last_fullsized":"2024-04-16T21:12:06.869006+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.868938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:10.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'8","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.650063+0000","last_change":"2024-04-16T21:12:10.650063+0000","last_active":"2024-04-16T21:12:10.650063+0000","last_peered":"2024-04-16T21:12:10.650063+0000","last_clean":"2024-04-16T21:12:10.650063+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:12:10.650063+0000","last_undegraded":"2024-04-16T21:12:10.650063+0000","last_fullsized":"2024-04-16T21:12:10.650063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.649972+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.650174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.662915+0000","last_change":"2024-04-16T21:11:53.662915+0000","last_active":"2024-04-16T21:11:53.662915+0000","last_peered":"2024-04-16T21:11:53.662915+0000","last_clean":"2024-04-16T21:11:53.662915+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:11:53.662915+0000","last_undegraded":"2024-04-16T21:11:53.662915+0000","last_fullsized":"2024-04-16T21:11:53.662915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.662831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:53.840637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'317","reported_seq":613,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.662773+0000","last_change":"2024-04-16T21:11:52.662773+0000","last_active":"2024-04-16T21:11:52.662773+0000","last_peered":"2024-04-16T21:11:52.662773+0000","last_clean":"2024-04-16T21:11:52.662773+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:11:52.662773+0000","last_undegraded":"2024-04-16T21:11:52.662773+0000","last_fullsized":"2024-04-16T21:11:52.662773+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'317","last_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_deep_scrub":"52'317","last_deep_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.662688+0000","objects_scrubbed":25,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:03:07.759388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3966,"num_omap_keys":18,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'20","reported_seq":253,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.170966+0000","last_change":"2024-04-16T21:11:53.171045+0000","last_active":"2024-04-16T21:11:53.170966+0000","last_peered":"2024-04-16T21:11:53.170966+0000","last_clean":"2024-04-16T21:11:53.170966+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:11:53.170966+0000","last_undegraded":"2024-04-16T21:11:53.170966+0000","last_fullsized":"2024-04-16T21:11:53.170966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.170891+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:56.498676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.474986+0000","last_change":"2024-04-16T21:12:09.475030+0000","last_active":"2024-04-16T21:12:09.474986+0000","last_peered":"2024-04-16T21:12:09.474986+0000","last_clean":"2024-04-16T21:12:09.474986+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:12:09.474986+0000","last_undegraded":"2024-04-16T21:12:09.474986+0000","last_fullsized":"2024-04-16T21:12:09.474986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.474907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:12.558579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.703966+0000","last_change":"2024-04-16T21:12:40.703987+0000","last_active":"2024-04-16T21:12:40.703966+0000","last_peered":"2024-04-16T21:12:40.703966+0000","last_clean":"2024-04-16T21:12:40.703966+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:12:40.703966+0000","last_undegraded":"2024-04-16T21:12:40.703966+0000","last_fullsized":"2024-04-16T21:12:40.703966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.703892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:52.306563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.890729+0000","last_change":"2024-04-16T21:12:07.890754+0000","last_active":"2024-04-16T21:12:07.890729+0000","last_peered":"2024-04-16T21:12:07.890729+0000","last_clean":"2024-04-16T21:12:07.890729+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:12:07.890729+0000","last_undegraded":"2024-04-16T21:12:07.890729+0000","last_fullsized":"2024-04-16T21:12:07.890729+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.890665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:11.218450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"52'5","reported_seq":755,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.087803+0000","last_change":"2024-04-16T21:12:49.087850+0000","last_active":"2024-04-16T21:12:49.087803+0000","last_peered":"2024-04-16T21:12:49.087803+0000","last_clean":"2024-04-16T21:12:49.087803+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:12:49.087803+0000","last_undegraded":"2024-04-16T21:12:49.087803+0000","last_fullsized":"2024-04-16T21:12:49.087803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'5","last_scrub_stamp":"2024-04-16T21:12:49.087709+0000","last_deep_scrub":"52'5","last_deep_scrub_stamp":"2024-04-16T21:12:49.087709+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.087709+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:36.342884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.782050+0000","last_change":"2024-04-16T21:12:40.782083+0000","last_active":"2024-04-16T21:12:40.782050+0000","last_peered":"2024-04-16T21:12:40.782050+0000","last_clean":"2024-04-16T21:12:40.782050+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:12:40.782050+0000","last_undegraded":"2024-04-16T21:12:40.782050+0000","last_fullsized":"2024-04-16T21:12:40.782050+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.781989+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:34.970663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.902196+0000","last_change":"2024-04-16T21:12:21.902219+0000","last_active":"2024-04-16T21:12:21.902196+0000","last_peered":"2024-04-16T21:12:21.902196+0000","last_clean":"2024-04-16T21:12:21.902196+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:12:21.902196+0000","last_undegraded":"2024-04-16T21:12:21.902196+0000","last_fullsized":"2024-04-16T21:12:21.902196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.902128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:26.941072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.849129+0000","last_change":"2024-04-16T21:13:01.849151+0000","last_active":"2024-04-16T21:13:01.849129+0000","last_peered":"2024-04-16T21:13:01.849129+0000","last_clean":"2024-04-16T21:13:01.849129+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:13:01.849129+0000","last_undegraded":"2024-04-16T21:13:01.849129+0000","last_fullsized":"2024-04-16T21:13:01.849129+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.849065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.849065+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.849065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:03:36.108179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'159","reported_seq":1763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.889853+0000","last_change":"2024-04-16T21:12:18.889893+0000","last_active":"2024-04-16T21:12:18.889853+0000","last_peered":"2024-04-16T21:12:18.889853+0000","last_clean":"2024-04-16T21:12:18.889853+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:12:18.889853+0000","last_undegraded":"2024-04-16T21:12:18.889853+0000","last_fullsized":"2024-04-16T21:12:18.889853+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.889790+0000","objects_scrubbed":41,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:35.275723+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12681,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":83,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.752918+0000","last_change":"2024-04-16T21:12:01.752964+0000","last_active":"2024-04-16T21:12:01.752918+0000","last_peered":"2024-04-16T21:12:01.752918+0000","last_clean":"2024-04-16T21:12:01.752918+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:12:01.752918+0000","last_undegraded":"2024-04-16T21:12:01.752918+0000","last_fullsized":"2024-04-16T21:12:01.752918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.752809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.500440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.625407+0000","last_change":"2024-04-16T21:12:04.625407+0000","last_active":"2024-04-16T21:12:04.625407+0000","last_peered":"2024-04-16T21:12:04.625407+0000","last_clean":"2024-04-16T21:12:04.625407+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:12:04.625407+0000","last_undegraded":"2024-04-16T21:12:04.625407+0000","last_fullsized":"2024-04-16T21:12:04.625407+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'2","last_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_deep_scrub":"45'2","last_deep_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.625306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.449242+0000","last_change":"2024-04-16T21:12:04.449242+0000","last_active":"2024-04-16T21:12:04.449242+0000","last_peered":"2024-04-16T21:12:04.449242+0000","last_clean":"2024-04-16T21:12:04.449242+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:12:04.449242+0000","last_undegraded":"2024-04-16T21:12:04.449242+0000","last_fullsized":"2024-04-16T21:12:04.449242+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.449160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:30.569236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1163","reported_seq":2931,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.718512+0000","last_change":"2024-04-16T21:11:46.718512+0000","last_active":"2024-04-16T21:11:46.718512+0000","last_peered":"2024-04-16T21:11:46.718512+0000","last_clean":"2024-04-16T21:11:46.718512+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:11:46.718512+0000","last_undegraded":"2024-04-16T21:11:46.718512+0000","last_fullsized":"2024-04-16T21:11:46.718512+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1163","last_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_deep_scrub":"52'1163","last_deep_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.718377+0000","objects_scrubbed":23,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:06.764370+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'16","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.594571+0000","last_change":"2024-04-16T21:11:51.594618+0000","last_active":"2024-04-16T21:11:51.594571+0000","last_peered":"2024-04-16T21:11:51.594571+0000","last_clean":"2024-04-16T21:11:51.594571+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:11:51.594571+0000","last_undegraded":"2024-04-16T21:11:51.594571+0000","last_fullsized":"2024-04-16T21:11:51.594571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.594510+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:52.095893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.782064+0000","last_change":"2024-04-16T21:12:01.782087+0000","last_active":"2024-04-16T21:12:01.782064+0000","last_peered":"2024-04-16T21:12:01.782064+0000","last_clean":"2024-04-16T21:12:01.782064+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:12:01.782064+0000","last_undegraded":"2024-04-16T21:12:01.782064+0000","last_fullsized":"2024-04-16T21:12:01.782064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.781996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:45.434631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.052606+0000","last_change":"2024-04-16T21:12:39.052630+0000","last_active":"2024-04-16T21:12:39.052606+0000","last_peered":"2024-04-16T21:12:39.052606+0000","last_clean":"2024-04-16T21:12:39.052606+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:12:39.052606+0000","last_undegraded":"2024-04-16T21:12:39.052606+0000","last_fullsized":"2024-04-16T21:12:39.052606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.052538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:27.090193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.219480+0000","last_change":"2024-04-16T21:12:47.219500+0000","last_active":"2024-04-16T21:12:47.219480+0000","last_peered":"2024-04-16T21:12:47.219480+0000","last_clean":"2024-04-16T21:12:47.219480+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:12:47.219480+0000","last_undegraded":"2024-04-16T21:12:47.219480+0000","last_fullsized":"2024-04-16T21:12:47.219480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.219422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.219422+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.219422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:36.660489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'15","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.143911+0000","last_change":"2024-04-16T21:13:01.143941+0000","last_active":"2024-04-16T21:13:01.143911+0000","last_peered":"2024-04-16T21:13:01.143911+0000","last_clean":"2024-04-16T21:13:01.143911+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:13:01.143911+0000","last_undegraded":"2024-04-16T21:13:01.143911+0000","last_fullsized":"2024-04-16T21:13:01.143911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:13:01.143846+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:13:01.143846+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.143846+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:13.624978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'49","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.341873+0000","last_change":"2024-04-16T21:12:40.341940+0000","last_active":"2024-04-16T21:12:40.341873+0000","last_peered":"2024-04-16T21:12:40.341873+0000","last_clean":"2024-04-16T21:12:40.341873+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:12:40.341873+0000","last_undegraded":"2024-04-16T21:12:40.341873+0000","last_fullsized":"2024-04-16T21:12:40.341873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'49","last_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_deep_scrub":"52'49","last_deep_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.341796+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:55.596579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.423193+0000","last_change":"2024-04-16T21:12:10.423217+0000","last_active":"2024-04-16T21:12:10.423193+0000","last_peered":"2024-04-16T21:12:10.423193+0000","last_clean":"2024-04-16T21:12:10.423193+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:12:10.423193+0000","last_undegraded":"2024-04-16T21:12:10.423193+0000","last_fullsized":"2024-04-16T21:12:10.423193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.423128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:15.462072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":51,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561679+0000","last_change":"2024-04-16T19:47:33.986715+0000","last_active":"2024-04-16T21:11:44.561679+0000","last_peered":"2024-04-16T21:11:44.561679+0000","last_clean":"2024-04-16T21:11:44.561679+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:11:44.561679+0000","last_undegraded":"2024-04-16T21:11:44.561679+0000","last_fullsized":"2024-04-16T21:11:44.561679+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'181","reported_seq":1765,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.009379+0000","last_change":"2024-04-16T21:12:35.009430+0000","last_active":"2024-04-16T21:12:35.009379+0000","last_peered":"2024-04-16T21:12:35.009379+0000","last_clean":"2024-04-16T21:12:35.009379+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:12:35.009379+0000","last_undegraded":"2024-04-16T21:12:35.009379+0000","last_fullsized":"2024-04-16T21:12:35.009379+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'181","last_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_deep_scrub":"52'181","last_deep_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.009303+0000","objects_scrubbed":51,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:46.611974+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8952,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.920212+0000","last_change":"2024-04-16T21:12:18.920252+0000","last_active":"2024-04-16T21:12:18.920212+0000","last_peered":"2024-04-16T21:12:18.920212+0000","last_clean":"2024-04-16T21:12:18.920212+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:12:18.920212+0000","last_undegraded":"2024-04-16T21:12:18.920212+0000","last_fullsized":"2024-04-16T21:12:18.920212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.920121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:25:34.146310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'26","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.477899+0000","last_change":"2024-04-16T21:12:19.477941+0000","last_active":"2024-04-16T21:12:19.477899+0000","last_peered":"2024-04-16T21:12:19.477899+0000","last_clean":"2024-04-16T21:12:19.477899+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:12:19.477899+0000","last_undegraded":"2024-04-16T21:12:19.477899+0000","last_fullsized":"2024-04-16T21:12:19.477899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.477827+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:13.666501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.766206+0000","last_change":"2024-04-16T21:12:09.766229+0000","last_active":"2024-04-16T21:12:09.766206+0000","last_peered":"2024-04-16T21:12:09.766206+0000","last_clean":"2024-04-16T21:12:09.766206+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:12:09.766206+0000","last_undegraded":"2024-04-16T21:12:09.766206+0000","last_fullsized":"2024-04-16T21:12:09.766206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.766140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:51.548954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'291","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.385570+0000","last_change":"2024-04-16T21:11:52.385621+0000","last_active":"2024-04-16T21:11:52.385570+0000","last_peered":"2024-04-16T21:11:52.385570+0000","last_clean":"2024-04-16T21:11:52.385570+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:11:52.385570+0000","last_undegraded":"2024-04-16T21:11:52.385570+0000","last_fullsized":"2024-04-16T21:11:52.385570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'291","last_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_deep_scrub":"52'291","last_deep_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.385512+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:13.385713+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'16","reported_seq":263,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.131942+0000","last_change":"2024-04-16T21:11:52.131942+0000","last_active":"2024-04-16T21:11:52.131942+0000","last_peered":"2024-04-16T21:11:52.131942+0000","last_clean":"2024-04-16T21:11:52.131942+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:11:52.131942+0000","last_undegraded":"2024-04-16T21:11:52.131942+0000","last_fullsized":"2024-04-16T21:11:52.131942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.131829+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:55.304717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.441339+0000","last_change":"2024-04-16T21:12:08.441339+0000","last_active":"2024-04-16T21:12:08.441339+0000","last_peered":"2024-04-16T21:12:08.441339+0000","last_clean":"2024-04-16T21:12:08.441339+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:12:08.441339+0000","last_undegraded":"2024-04-16T21:12:08.441339+0000","last_fullsized":"2024-04-16T21:12:08.441339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.441251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:03.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.805365+0000","last_change":"2024-04-16T21:12:35.805388+0000","last_active":"2024-04-16T21:12:35.805365+0000","last_peered":"2024-04-16T21:12:35.805365+0000","last_clean":"2024-04-16T21:12:35.805365+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:12:35.805365+0000","last_undegraded":"2024-04-16T21:12:35.805365+0000","last_fullsized":"2024-04-16T21:12:35.805365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.805297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:04.446592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.803948+0000","last_change":"2024-04-16T21:13:00.803989+0000","last_active":"2024-04-16T21:13:00.803948+0000","last_peered":"2024-04-16T21:13:00.803948+0000","last_clean":"2024-04-16T21:13:00.803948+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:13:00.803948+0000","last_undegraded":"2024-04-16T21:13:00.803948+0000","last_fullsized":"2024-04-16T21:13:00.803948+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.803871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.803871+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.803871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:27.514468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'14","reported_seq":623,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.736145+0000","last_change":"2024-04-16T21:12:39.736175+0000","last_active":"2024-04-16T21:12:39.736145+0000","last_peered":"2024-04-16T21:12:39.736145+0000","last_clean":"2024-04-16T21:12:39.736145+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:12:39.736145+0000","last_undegraded":"2024-04-16T21:12:39.736145+0000","last_fullsized":"2024-04-16T21:12:39.736145+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.736087+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:33.924762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.893687+0000","last_change":"2024-04-16T21:12:23.893724+0000","last_active":"2024-04-16T21:12:23.893687+0000","last_peered":"2024-04-16T21:12:23.893687+0000","last_clean":"2024-04-16T21:12:23.893687+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:12:23.893687+0000","last_undegraded":"2024-04-16T21:12:23.893687+0000","last_fullsized":"2024-04-16T21:12:23.893687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.893616+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:17.500596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.781919+0000","last_change":"2024-04-16T21:12:31.781950+0000","last_active":"2024-04-16T21:12:31.781919+0000","last_peered":"2024-04-16T21:12:31.781919+0000","last_clean":"2024-04-16T21:12:31.781919+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:12:31.781919+0000","last_undegraded":"2024-04-16T21:12:31.781919+0000","last_fullsized":"2024-04-16T21:12:31.781919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.781859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:19.914350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651552+0000","last_change":"2024-04-16T19:47:42.514496+0000","last_active":"2024-04-16T21:11:43.651552+0000","last_peered":"2024-04-16T21:11:43.651552+0000","last_clean":"2024-04-16T21:11:43.651552+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:11:43.651552+0000","last_undegraded":"2024-04-16T21:11:43.651552+0000","last_fullsized":"2024-04-16T21:11:43.651552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'124","reported_seq":1188,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.361236+0000","last_change":"2024-04-16T21:12:26.361300+0000","last_active":"2024-04-16T21:12:26.361236+0000","last_peered":"2024-04-16T21:12:26.361236+0000","last_clean":"2024-04-16T21:12:26.361236+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:12:26.361236+0000","last_undegraded":"2024-04-16T21:12:26.361236+0000","last_fullsized":"2024-04-16T21:12:26.361236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'124","last_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_deep_scrub":"52'124","last_deep_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.361168+0000","objects_scrubbed":36,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:14.967461+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7264,"num_omap_keys":29,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.867629+0000","last_change":"2024-04-16T21:12:04.867629+0000","last_active":"2024-04-16T21:12:04.867629+0000","last_peered":"2024-04-16T21:12:04.867629+0000","last_clean":"2024-04-16T21:12:04.867629+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:12:04.867629+0000","last_undegraded":"2024-04-16T21:12:04.867629+0000","last_fullsized":"2024-04-16T21:12:04.867629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.867541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.770932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'31","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.873726+0000","last_change":"2024-04-16T21:12:11.873787+0000","last_active":"2024-04-16T21:12:11.873726+0000","last_peered":"2024-04-16T21:12:11.873726+0000","last_clean":"2024-04-16T21:12:11.873726+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:12:11.873726+0000","last_undegraded":"2024-04-16T21:12:11.873726+0000","last_fullsized":"2024-04-16T21:12:11.873726+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'31","last_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_deep_scrub":"52'31","last_deep_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.873657+0000","objects_scrubbed":9,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:04.053968+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'706","reported_seq":2151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.887851+0000","last_change":"2024-04-16T21:12:02.887901+0000","last_active":"2024-04-16T21:12:02.887851+0000","last_peered":"2024-04-16T21:12:02.887851+0000","last_clean":"2024-04-16T21:12:02.887851+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:12:02.887851+0000","last_undegraded":"2024-04-16T21:12:02.887851+0000","last_fullsized":"2024-04-16T21:12:02.887851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.887759+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:37.031244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1038","reported_seq":1547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.224293+0000","last_change":"2024-04-16T21:11:54.224379+0000","last_active":"2024-04-16T21:11:54.224293+0000","last_peered":"2024-04-16T21:11:54.224293+0000","last_clean":"2024-04-16T21:11:54.224293+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:11:54.224293+0000","last_undegraded":"2024-04-16T21:11:54.224293+0000","last_fullsized":"2024-04-16T21:11:54.224293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1038","last_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_deep_scrub":"52'1038","last_deep_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.224224+0000","objects_scrubbed":21,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:22.282373+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2194,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'23","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.136819+0000","last_change":"2024-04-16T21:11:43.136819+0000","last_active":"2024-04-16T21:11:43.136819+0000","last_peered":"2024-04-16T21:11:43.136819+0000","last_clean":"2024-04-16T21:11:43.136819+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:11:43.136819+0000","last_undegraded":"2024-04-16T21:11:43.136819+0000","last_fullsized":"2024-04-16T21:11:43.136819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'23","last_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_deep_scrub":"52'23","last_deep_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.136721+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:06.328117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.766632+0000","last_change":"2024-04-16T21:12:08.766632+0000","last_active":"2024-04-16T21:12:08.766632+0000","last_peered":"2024-04-16T21:12:08.766632+0000","last_clean":"2024-04-16T21:12:08.766632+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:12:08.766632+0000","last_undegraded":"2024-04-16T21:12:08.766632+0000","last_fullsized":"2024-04-16T21:12:08.766632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.766544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:45.782631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.107579+0000","last_change":"2024-04-16T21:12:46.107599+0000","last_active":"2024-04-16T21:12:46.107579+0000","last_peered":"2024-04-16T21:12:46.107579+0000","last_clean":"2024-04-16T21:12:46.107579+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:12:46.107579+0000","last_undegraded":"2024-04-16T21:12:46.107579+0000","last_fullsized":"2024-04-16T21:12:46.107579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.107523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.107523+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.107523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:09.669785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:54.391438+0000","last_change":"2024-04-16T19:24:33.388185+0000","last_active":"2024-04-16T21:10:54.391438+0000","last_peered":"2024-04-16T21:10:54.391438+0000","last_clean":"2024-04-16T21:10:54.391438+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:10:54.391438+0000","last_undegraded":"2024-04-16T21:10:54.391438+0000","last_fullsized":"2024-04-16T21:10:54.391438+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.186769+0000","last_change":"2024-04-16T21:12:41.186808+0000","last_active":"2024-04-16T21:12:41.186769+0000","last_peered":"2024-04-16T21:12:41.186769+0000","last_clean":"2024-04-16T21:12:41.186769+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:12:41.186769+0000","last_undegraded":"2024-04-16T21:12:41.186769+0000","last_fullsized":"2024-04-16T21:12:41.186769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.186689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:10.176347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'78","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.901279+0000","last_change":"2024-04-16T21:12:35.901311+0000","last_active":"2024-04-16T21:12:35.901279+0000","last_peered":"2024-04-16T21:12:35.901279+0000","last_clean":"2024-04-16T21:12:35.901279+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:12:35.901279+0000","last_undegraded":"2024-04-16T21:12:35.901279+0000","last_fullsized":"2024-04-16T21:12:35.901279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.901215+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:22.199019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.365174+0000","last_change":"2024-04-16T21:12:25.365219+0000","last_active":"2024-04-16T21:12:25.365174+0000","last_peered":"2024-04-16T21:12:25.365174+0000","last_clean":"2024-04-16T21:12:25.365174+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:12:25.365174+0000","last_undegraded":"2024-04-16T21:12:25.365174+0000","last_fullsized":"2024-04-16T21:12:25.365174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.365094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:11.662898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.651543+0000","last_change":"2024-04-16T19:47:33.986415+0000","last_active":"2024-04-16T21:11:43.651543+0000","last_peered":"2024-04-16T21:11:43.651543+0000","last_clean":"2024-04-16T21:11:43.651543+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:11:43.651543+0000","last_undegraded":"2024-04-16T21:11:43.651543+0000","last_fullsized":"2024-04-16T21:11:43.651543+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'207","reported_seq":2084,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.250819+0000","last_change":"2024-04-16T21:12:44.250861+0000","last_active":"2024-04-16T21:12:44.250819+0000","last_peered":"2024-04-16T21:12:44.250819+0000","last_clean":"2024-04-16T21:12:44.250819+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:12:44.250819+0000","last_undegraded":"2024-04-16T21:12:44.250819+0000","last_fullsized":"2024-04-16T21:12:44.250819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'207","last_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_deep_scrub":"52'207","last_deep_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.250741+0000","objects_scrubbed":45,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:38.832019+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12815,"num_omap_keys":53,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.457685+0000","last_change":"2024-04-16T21:12:13.457709+0000","last_active":"2024-04-16T21:12:13.457685+0000","last_peered":"2024-04-16T21:12:13.457685+0000","last_clean":"2024-04-16T21:12:13.457685+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:12:13.457685+0000","last_undegraded":"2024-04-16T21:12:13.457685+0000","last_fullsized":"2024-04-16T21:12:13.457685+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.457615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:51.890971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'7","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.648645+0000","last_change":"2024-04-16T21:12:07.648674+0000","last_active":"2024-04-16T21:12:07.648645+0000","last_peered":"2024-04-16T21:12:07.648645+0000","last_clean":"2024-04-16T21:12:07.648645+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:12:07.648645+0000","last_undegraded":"2024-04-16T21:12:07.648645+0000","last_fullsized":"2024-04-16T21:12:07.648645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.648592+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.343437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.389563+0000","last_change":"2024-04-16T21:12:06.389563+0000","last_active":"2024-04-16T21:12:06.389563+0000","last_peered":"2024-04-16T21:12:06.389563+0000","last_clean":"2024-04-16T21:12:06.389563+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:12:06.389563+0000","last_undegraded":"2024-04-16T21:12:06.389563+0000","last_fullsized":"2024-04-16T21:12:06.389563+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.389467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:55.353016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'329","reported_seq":649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.716329+0000","last_change":"2024-04-16T21:11:45.716329+0000","last_active":"2024-04-16T21:11:45.716329+0000","last_peered":"2024-04-16T21:11:45.716329+0000","last_clean":"2024-04-16T21:11:45.716329+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:11:45.716329+0000","last_undegraded":"2024-04-16T21:11:45.716329+0000","last_fullsized":"2024-04-16T21:11:45.716329+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'329","last_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_deep_scrub":"52'329","last_deep_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.716182+0000","objects_scrubbed":24,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:24.600927+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'11","reported_seq":161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.675426+0000","last_change":"2024-04-16T21:11:50.675426+0000","last_active":"2024-04-16T21:11:50.675426+0000","last_peered":"2024-04-16T21:11:50.675426+0000","last_clean":"2024-04-16T21:11:50.675426+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:11:50.675426+0000","last_undegraded":"2024-04-16T21:11:50.675426+0000","last_fullsized":"2024-04-16T21:11:50.675426+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_deep_scrub":"52'11","last_deep_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.675302+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:31.269067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.193468+0000","last_change":"2024-04-16T21:12:45.193468+0000","last_active":"2024-04-16T21:12:45.193468+0000","last_peered":"2024-04-16T21:12:45.193468+0000","last_clean":"2024-04-16T21:12:45.193468+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:12:45.193468+0000","last_undegraded":"2024-04-16T21:12:45.193468+0000","last_fullsized":"2024-04-16T21:12:45.193468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.193382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.193382+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.193382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:54.542159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.768914+0000","last_change":"2024-04-16T21:12:49.768937+0000","last_active":"2024-04-16T21:12:49.768914+0000","last_peered":"2024-04-16T21:12:49.768914+0000","last_clean":"2024-04-16T21:12:49.768914+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:12:49.768914+0000","last_undegraded":"2024-04-16T21:12:49.768914+0000","last_fullsized":"2024-04-16T21:12:49.768914+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.768845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.768845+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.768845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:56.006493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.345176+0000","last_change":"2024-04-16T21:12:41.345218+0000","last_active":"2024-04-16T21:12:41.345176+0000","last_peered":"2024-04-16T21:12:41.345176+0000","last_clean":"2024-04-16T21:12:41.345176+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:12:41.345176+0000","last_undegraded":"2024-04-16T21:12:41.345176+0000","last_fullsized":"2024-04-16T21:12:41.345176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.345086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:35.926364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'87","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.024063+0000","last_change":"2024-04-16T21:12:30.024100+0000","last_active":"2024-04-16T21:12:30.024063+0000","last_peered":"2024-04-16T21:12:30.024063+0000","last_clean":"2024-04-16T21:12:30.024063+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:12:30.024063+0000","last_undegraded":"2024-04-16T21:12:30.024063+0000","last_fullsized":"2024-04-16T21:12:30.024063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.024002+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:39.585670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.356039+0000","last_change":"2024-04-16T21:12:27.356060+0000","last_active":"2024-04-16T21:12:27.356039+0000","last_peered":"2024-04-16T21:12:27.356039+0000","last_clean":"2024-04-16T21:12:27.356039+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:12:27.356039+0000","last_undegraded":"2024-04-16T21:12:27.356039+0000","last_fullsized":"2024-04-16T21:12:27.356039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.355975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:15.393630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.732045+0000","last_change":"2024-04-16T21:11:59.732045+0000","last_active":"2024-04-16T21:11:59.732045+0000","last_peered":"2024-04-16T21:11:59.732045+0000","last_clean":"2024-04-16T21:11:59.732045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:11:59.732045+0000","last_undegraded":"2024-04-16T21:11:59.732045+0000","last_fullsized":"2024-04-16T21:11:59.732045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.731957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:37.824609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.561671+0000","last_change":"2024-04-16T19:47:33.985341+0000","last_active":"2024-04-16T21:11:44.561671+0000","last_peered":"2024-04-16T21:11:44.561671+0000","last_clean":"2024-04-16T21:11:44.561671+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:11:44.561671+0000","last_undegraded":"2024-04-16T21:11:44.561671+0000","last_fullsized":"2024-04-16T21:11:44.561671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'170","reported_seq":1882,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.686097+0000","last_change":"2024-04-16T21:12:29.686173+0000","last_active":"2024-04-16T21:12:29.686097+0000","last_peered":"2024-04-16T21:12:29.686097+0000","last_clean":"2024-04-16T21:12:29.686097+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:12:29.686097+0000","last_undegraded":"2024-04-16T21:12:29.686097+0000","last_fullsized":"2024-04-16T21:12:29.686097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.686024+0000","objects_scrubbed":46,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:24.943848+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12193,"num_omap_keys":48,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.906298+0000","last_change":"2024-04-16T21:12:17.906319+0000","last_active":"2024-04-16T21:12:17.906298+0000","last_peered":"2024-04-16T21:12:17.906298+0000","last_clean":"2024-04-16T21:12:17.906298+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:12:17.906298+0000","last_undegraded":"2024-04-16T21:12:17.906298+0000","last_fullsized":"2024-04-16T21:12:17.906298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.906238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:22.945182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'18","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.785345+0000","last_change":"2024-04-16T21:12:24.785376+0000","last_active":"2024-04-16T21:12:24.785345+0000","last_peered":"2024-04-16T21:12:24.785345+0000","last_clean":"2024-04-16T21:12:24.785345+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:12:24.785345+0000","last_undegraded":"2024-04-16T21:12:24.785345+0000","last_fullsized":"2024-04-16T21:12:24.785345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.785282+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:53.426577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.865507+0000","last_change":"2024-04-16T21:12:05.865507+0000","last_active":"2024-04-16T21:12:05.865507+0000","last_peered":"2024-04-16T21:12:05.865507+0000","last_clean":"2024-04-16T21:12:05.865507+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:12:05.865507+0000","last_undegraded":"2024-04-16T21:12:05.865507+0000","last_fullsized":"2024-04-16T21:12:05.865507+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.865437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.716343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'138","reported_seq":299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.719459+0000","last_change":"2024-04-16T21:11:48.719459+0000","last_active":"2024-04-16T21:11:48.719459+0000","last_peered":"2024-04-16T21:11:48.719459+0000","last_clean":"2024-04-16T21:11:48.719459+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:11:48.719459+0000","last_undegraded":"2024-04-16T21:11:48.719459+0000","last_fullsized":"2024-04-16T21:11:48.719459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.719337+0000","objects_scrubbed":17,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.768315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4229,"num_omap_keys":20,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'8","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.405983+0000","last_change":"2024-04-16T21:11:49.405983+0000","last_active":"2024-04-16T21:11:49.405983+0000","last_peered":"2024-04-16T21:11:49.405983+0000","last_clean":"2024-04-16T21:11:49.405983+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:11:49.405983+0000","last_undegraded":"2024-04-16T21:11:49.405983+0000","last_fullsized":"2024-04-16T21:11:49.405983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.405858+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:30.100703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.341370+0000","last_change":"2024-04-16T21:12:44.341370+0000","last_active":"2024-04-16T21:12:44.341370+0000","last_peered":"2024-04-16T21:12:44.341370+0000","last_clean":"2024-04-16T21:12:44.341370+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:12:44.341370+0000","last_undegraded":"2024-04-16T21:12:44.341370+0000","last_fullsized":"2024-04-16T21:12:44.341370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.341283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:33.782350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.076148+0000","last_change":"2024-04-16T21:12:57.076171+0000","last_active":"2024-04-16T21:12:57.076148+0000","last_peered":"2024-04-16T21:12:57.076148+0000","last_clean":"2024-04-16T21:12:57.076148+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:12:57.076148+0000","last_undegraded":"2024-04-16T21:12:57.076148+0000","last_fullsized":"2024-04-16T21:12:57.076148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.076066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.076066+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.076066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:06.424842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"52'4","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.265214+0000","last_change":"2024-04-16T21:12:39.265243+0000","last_active":"2024-04-16T21:12:39.265214+0000","last_peered":"2024-04-16T21:12:39.265214+0000","last_clean":"2024-04-16T21:12:39.265214+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:12:39.265214+0000","last_undegraded":"2024-04-16T21:12:39.265214+0000","last_fullsized":"2024-04-16T21:12:39.265214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.265147+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:53.843686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'147","reported_seq":338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.377505+0000","last_change":"2024-04-16T21:12:31.377537+0000","last_active":"2024-04-16T21:12:31.377505+0000","last_peered":"2024-04-16T21:12:31.377505+0000","last_clean":"2024-04-16T21:12:31.377505+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:12:31.377505+0000","last_undegraded":"2024-04-16T21:12:31.377505+0000","last_fullsized":"2024-04-16T21:12:31.377505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.377449+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:37.615098+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.930468+0000","last_change":"2024-04-16T21:12:23.930491+0000","last_active":"2024-04-16T21:12:23.930468+0000","last_peered":"2024-04-16T21:12:23.930468+0000","last_clean":"2024-04-16T21:12:23.930468+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:12:23.930468+0000","last_undegraded":"2024-04-16T21:12:23.930468+0000","last_fullsized":"2024-04-16T21:12:23.930468+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.930404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:27.014075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.517927+0000","last_change":"2024-04-16T21:12:01.517927+0000","last_active":"2024-04-16T21:12:01.517927+0000","last_peered":"2024-04-16T21:12:01.517927+0000","last_clean":"2024-04-16T21:12:01.517927+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:12:01.517927+0000","last_undegraded":"2024-04-16T21:12:01.517927+0000","last_fullsized":"2024-04-16T21:12:01.517927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:38.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":52,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.293399+0000","last_change":"2024-04-16T19:47:33.987742+0000","last_active":"2024-04-16T21:11:42.293399+0000","last_peered":"2024-04-16T21:11:42.293399+0000","last_clean":"2024-04-16T21:11:42.293399+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:11:42.293399+0000","last_undegraded":"2024-04-16T21:11:42.293399+0000","last_fullsized":"2024-04-16T21:11:42.293399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'150","reported_seq":1340,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.054107+0000","last_change":"2024-04-16T21:12:31.054152+0000","last_active":"2024-04-16T21:12:31.054107+0000","last_peered":"2024-04-16T21:12:31.054107+0000","last_clean":"2024-04-16T21:12:31.054107+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:12:31.054107+0000","last_undegraded":"2024-04-16T21:12:31.054107+0000","last_fullsized":"2024-04-16T21:12:31.054107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'150","last_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_deep_scrub":"52'150","last_deep_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.054045+0000","objects_scrubbed":46,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:22.727735+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7793,"num_omap_keys":32,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":86,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.443129+0000","last_change":"2024-04-16T21:12:22.443154+0000","last_active":"2024-04-16T21:12:22.443129+0000","last_peered":"2024-04-16T21:12:22.443129+0000","last_clean":"2024-04-16T21:12:22.443129+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:12:22.443129+0000","last_undegraded":"2024-04-16T21:12:22.443129+0000","last_fullsized":"2024-04-16T21:12:22.443129+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.443057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:22.418023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'15","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.828587+0000","last_change":"2024-04-16T21:12:23.828622+0000","last_active":"2024-04-16T21:12:23.828587+0000","last_peered":"2024-04-16T21:12:23.828587+0000","last_clean":"2024-04-16T21:12:23.828587+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:12:23.828587+0000","last_undegraded":"2024-04-16T21:12:23.828587+0000","last_fullsized":"2024-04-16T21:12:23.828587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.828490+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:02.261846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.402484+0000","last_change":"2024-04-16T21:12:06.402484+0000","last_active":"2024-04-16T21:12:06.402484+0000","last_peered":"2024-04-16T21:12:06.402484+0000","last_clean":"2024-04-16T21:12:06.402484+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:12:06.402484+0000","last_undegraded":"2024-04-16T21:12:06.402484+0000","last_fullsized":"2024-04-16T21:12:06.402484+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.402344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:52.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'341","reported_seq":880,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.634146+0000","last_change":"2024-04-16T21:11:48.634146+0000","last_active":"2024-04-16T21:11:48.634146+0000","last_peered":"2024-04-16T21:11:48.634146+0000","last_clean":"2024-04-16T21:11:48.634146+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:11:48.634146+0000","last_undegraded":"2024-04-16T21:11:48.634146+0000","last_fullsized":"2024-04-16T21:11:48.634146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'341","last_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_deep_scrub":"52'341","last_deep_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.633983+0000","objects_scrubbed":20,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:05:52.976164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'19","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.631145+0000","last_change":"2024-04-16T21:11:55.631145+0000","last_active":"2024-04-16T21:11:55.631145+0000","last_peered":"2024-04-16T21:11:55.631145+0000","last_clean":"2024-04-16T21:11:55.631145+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:11:55.631145+0000","last_undegraded":"2024-04-16T21:11:55.631145+0000","last_fullsized":"2024-04-16T21:11:55.631145+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.631042+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.051632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.384307+0000","last_change":"2024-04-16T21:12:47.384330+0000","last_active":"2024-04-16T21:12:47.384307+0000","last_peered":"2024-04-16T21:12:47.384307+0000","last_clean":"2024-04-16T21:12:47.384307+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:12:47.384307+0000","last_undegraded":"2024-04-16T21:12:47.384307+0000","last_fullsized":"2024-04-16T21:12:47.384307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.384240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.384240+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.384240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:55.538199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.701557+0000","last_change":"2024-04-16T21:12:48.701581+0000","last_active":"2024-04-16T21:12:48.701557+0000","last_peered":"2024-04-16T21:12:48.701557+0000","last_clean":"2024-04-16T21:12:48.701557+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:12:48.701557+0000","last_undegraded":"2024-04-16T21:12:48.701557+0000","last_fullsized":"2024-04-16T21:12:48.701557+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.701492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.701492+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.701492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:54.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.723608+0000","last_change":"2024-04-16T21:12:41.723630+0000","last_active":"2024-04-16T21:12:41.723608+0000","last_peered":"2024-04-16T21:12:41.723608+0000","last_clean":"2024-04-16T21:12:41.723608+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:12:41.723608+0000","last_undegraded":"2024-04-16T21:12:41.723608+0000","last_fullsized":"2024-04-16T21:12:41.723608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.723545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:29.856036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'101","reported_seq":278,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.758495+0000","last_change":"2024-04-16T21:12:50.758528+0000","last_active":"2024-04-16T21:12:50.758495+0000","last_peered":"2024-04-16T21:12:50.758495+0000","last_clean":"2024-04-16T21:12:50.758495+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:12:50.758495+0000","last_undegraded":"2024-04-16T21:12:50.758495+0000","last_fullsized":"2024-04-16T21:12:50.758495+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:50.758441+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:50.758441+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.758441+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:06.557737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.928158+0000","last_change":"2024-04-16T21:12:25.928177+0000","last_active":"2024-04-16T21:12:25.928158+0000","last_peered":"2024-04-16T21:12:25.928158+0000","last_clean":"2024-04-16T21:12:25.928158+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:12:25.928158+0000","last_undegraded":"2024-04-16T21:12:25.928158+0000","last_fullsized":"2024-04-16T21:12:25.928158+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.928102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:41.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.482650+0000","last_change":"2024-04-16T21:12:03.482650+0000","last_active":"2024-04-16T21:12:03.482650+0000","last_peered":"2024-04-16T21:12:03.482650+0000","last_clean":"2024-04-16T21:12:03.482650+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:12:03.482650+0000","last_undegraded":"2024-04-16T21:12:03.482650+0000","last_fullsized":"2024-04-16T21:12:03.482650+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.482557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:28.918007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.850860+0000","last_change":"2024-04-16T21:13:00.850882+0000","last_active":"2024-04-16T21:13:00.850860+0000","last_peered":"2024-04-16T21:13:00.850860+0000","last_clean":"2024-04-16T21:13:00.850860+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:13:00.850860+0000","last_undegraded":"2024-04-16T21:13:00.850860+0000","last_fullsized":"2024-04-16T21:13:00.850860+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.850797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.850797+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.850797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:28.392426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'154","reported_seq":1354,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.959232+0000","last_change":"2024-04-16T21:12:26.959277+0000","last_active":"2024-04-16T21:12:26.959232+0000","last_peered":"2024-04-16T21:12:26.959232+0000","last_clean":"2024-04-16T21:12:26.959232+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:12:26.959232+0000","last_undegraded":"2024-04-16T21:12:26.959232+0000","last_fullsized":"2024-04-16T21:12:26.959232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'154","last_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_deep_scrub":"52'154","last_deep_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.959173+0000","objects_scrubbed":37,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:21.387909+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8816,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.680232+0000","last_change":"2024-04-16T21:12:17.680254+0000","last_active":"2024-04-16T21:12:17.680232+0000","last_peered":"2024-04-16T21:12:17.680232+0000","last_clean":"2024-04-16T21:12:17.680232+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:12:17.680232+0000","last_undegraded":"2024-04-16T21:12:17.680232+0000","last_fullsized":"2024-04-16T21:12:17.680232+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.680171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:09.860482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"52'8","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.838571+0000","last_change":"2024-04-16T21:12:22.838571+0000","last_active":"2024-04-16T21:12:22.838571+0000","last_peered":"2024-04-16T21:12:22.838571+0000","last_clean":"2024-04-16T21:12:22.838571+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:12:22.838571+0000","last_undegraded":"2024-04-16T21:12:22.838571+0000","last_fullsized":"2024-04-16T21:12:22.838571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.838490+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:17.267225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.653905+0000","last_change":"2024-04-16T21:12:05.653905+0000","last_active":"2024-04-16T21:12:05.653905+0000","last_peered":"2024-04-16T21:12:05.653905+0000","last_clean":"2024-04-16T21:12:05.653905+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:12:05.653905+0000","last_undegraded":"2024-04-16T21:12:05.653905+0000","last_fullsized":"2024-04-16T21:12:05.653905+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.653825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:12.673868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'264","reported_seq":1137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.696035+0000","last_change":"2024-04-16T21:11:49.696035+0000","last_active":"2024-04-16T21:11:49.696035+0000","last_peered":"2024-04-16T21:11:49.696035+0000","last_clean":"2024-04-16T21:11:49.696035+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:11:49.696035+0000","last_undegraded":"2024-04-16T21:11:49.696035+0000","last_fullsized":"2024-04-16T21:11:49.696035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.695875+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:11.637965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2924,"num_omap_keys":15,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'12","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.501597+0000","last_change":"2024-04-16T21:11:59.501597+0000","last_active":"2024-04-16T21:11:59.501597+0000","last_peered":"2024-04-16T21:11:59.501597+0000","last_clean":"2024-04-16T21:11:59.501597+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:11:59.501597+0000","last_undegraded":"2024-04-16T21:11:59.501597+0000","last_fullsized":"2024-04-16T21:11:59.501597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.501496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:51.681807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.856833+0000","last_change":"2024-04-16T21:12:29.856857+0000","last_active":"2024-04-16T21:12:29.856833+0000","last_peered":"2024-04-16T21:12:29.856833+0000","last_clean":"2024-04-16T21:12:29.856833+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:12:29.856833+0000","last_undegraded":"2024-04-16T21:12:29.856833+0000","last_fullsized":"2024-04-16T21:12:29.856833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.856754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:25.114578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.489060+0000","last_change":"2024-04-16T21:13:03.489086+0000","last_active":"2024-04-16T21:13:03.489060+0000","last_peered":"2024-04-16T21:13:03.489060+0000","last_clean":"2024-04-16T21:13:03.489060+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:13:03.489060+0000","last_undegraded":"2024-04-16T21:13:03.489060+0000","last_fullsized":"2024-04-16T21:13:03.489060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.488997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.488997+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.488997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:16.660558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'6","reported_seq":643,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.083593+0000","last_change":"2024-04-16T21:12:51.083624+0000","last_active":"2024-04-16T21:12:51.083593+0000","last_peered":"2024-04-16T21:12:51.083593+0000","last_clean":"2024-04-16T21:12:51.083593+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:12:51.083593+0000","last_undegraded":"2024-04-16T21:12:51.083593+0000","last_fullsized":"2024-04-16T21:12:51.083593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:51.083530+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:51.083530+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.083530+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:05.662069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'115","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.359212+0000","last_change":"2024-04-16T21:12:34.359243+0000","last_active":"2024-04-16T21:12:34.359212+0000","last_peered":"2024-04-16T21:12:34.359212+0000","last_clean":"2024-04-16T21:12:34.359212+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:12:34.359212+0000","last_undegraded":"2024-04-16T21:12:34.359212+0000","last_fullsized":"2024-04-16T21:12:34.359212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.359152+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:57.921414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'13","reported_seq":1325,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.729138+0000","last_change":"2024-04-16T21:12:28.729186+0000","last_active":"2024-04-16T21:12:28.729138+0000","last_peered":"2024-04-16T21:12:28.729138+0000","last_clean":"2024-04-16T21:12:28.729138+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:12:28.729138+0000","last_undegraded":"2024-04-16T21:12:28.729138+0000","last_fullsized":"2024-04-16T21:12:28.729138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.729061+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:00.415763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.424596+0000","last_change":"2024-04-16T21:12:05.424596+0000","last_active":"2024-04-16T21:12:05.424596+0000","last_peered":"2024-04-16T21:12:05.424596+0000","last_clean":"2024-04-16T21:12:05.424596+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:12:05.424596+0000","last_undegraded":"2024-04-16T21:12:05.424596+0000","last_fullsized":"2024-04-16T21:12:05.424596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.424465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:59.031445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.849953+0000","last_change":"2024-04-16T21:12:59.849982+0000","last_active":"2024-04-16T21:12:59.849953+0000","last_peered":"2024-04-16T21:12:59.849953+0000","last_clean":"2024-04-16T21:12:59.849953+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:12:59.849953+0000","last_undegraded":"2024-04-16T21:12:59.849953+0000","last_fullsized":"2024-04-16T21:12:59.849953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.849879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.849879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.849879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:31.554672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'274","reported_seq":2375,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.635555+0000","last_change":"2024-04-16T21:12:25.635594+0000","last_active":"2024-04-16T21:12:25.635555+0000","last_peered":"2024-04-16T21:12:25.635555+0000","last_clean":"2024-04-16T21:12:25.635555+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:12:25.635555+0000","last_undegraded":"2024-04-16T21:12:25.635555+0000","last_fullsized":"2024-04-16T21:12:25.635555+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'274","last_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_deep_scrub":"52'274","last_deep_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.635495+0000","objects_scrubbed":60,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.021428+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17559,"num_omap_keys":71,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.466844+0000","last_change":"2024-04-16T21:12:18.466844+0000","last_active":"2024-04-16T21:12:18.466844+0000","last_peered":"2024-04-16T21:12:18.466844+0000","last_clean":"2024-04-16T21:12:18.466844+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:12:18.466844+0000","last_undegraded":"2024-04-16T21:12:18.466844+0000","last_fullsized":"2024-04-16T21:12:18.466844+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.466741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:50.153442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'17","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.900906+0000","last_change":"2024-04-16T21:12:12.900906+0000","last_active":"2024-04-16T21:12:12.900906+0000","last_peered":"2024-04-16T21:12:12.900906+0000","last_clean":"2024-04-16T21:12:12.900906+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:12:12.900906+0000","last_undegraded":"2024-04-16T21:12:12.900906+0000","last_fullsized":"2024-04-16T21:12:12.900906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.900807+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:00.936096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'706","reported_seq":2133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.326509+0000","last_change":"2024-04-16T21:12:25.326509+0000","last_active":"2024-04-16T21:12:25.326509+0000","last_peered":"2024-04-16T21:12:25.326509+0000","last_clean":"2024-04-16T21:12:25.326509+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:12:25.326509+0000","last_undegraded":"2024-04-16T21:12:25.326509+0000","last_fullsized":"2024-04-16T21:12:25.326509+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.326417+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:25.301383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'557","reported_seq":1382,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.205288+0000","last_change":"2024-04-16T21:11:50.205349+0000","last_active":"2024-04-16T21:11:50.205288+0000","last_peered":"2024-04-16T21:11:50.205288+0000","last_clean":"2024-04-16T21:11:50.205288+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:11:50.205288+0000","last_undegraded":"2024-04-16T21:11:50.205288+0000","last_fullsized":"2024-04-16T21:11:50.205288+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'557","last_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_deep_scrub":"52'557","last_deep_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.205222+0000","objects_scrubbed":17,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:09.070331+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2200,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'17","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.702594+0000","last_change":"2024-04-16T21:12:11.702662+0000","last_active":"2024-04-16T21:12:11.702594+0000","last_peered":"2024-04-16T21:12:11.702594+0000","last_clean":"2024-04-16T21:12:11.702594+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:12:11.702594+0000","last_undegraded":"2024-04-16T21:12:11.702594+0000","last_fullsized":"2024-04-16T21:12:11.702594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.702519+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:33.204105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.746552+0000","last_change":"2024-04-16T21:12:44.746576+0000","last_active":"2024-04-16T21:12:44.746552+0000","last_peered":"2024-04-16T21:12:44.746552+0000","last_clean":"2024-04-16T21:12:44.746552+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:12:44.746552+0000","last_undegraded":"2024-04-16T21:12:44.746552+0000","last_fullsized":"2024-04-16T21:12:44.746552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.746485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.746485+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.746485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:22:15.574936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560193+0000","last_change":"2024-04-16T19:24:33.385823+0000","last_active":"2024-04-16T21:11:44.560193+0000","last_peered":"2024-04-16T21:11:44.560193+0000","last_clean":"2024-04-16T21:11:44.560193+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:11:44.560193+0000","last_undegraded":"2024-04-16T21:11:44.560193+0000","last_fullsized":"2024-04-16T21:11:44.560193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.762674+0000","last_change":"2024-04-16T21:12:43.762696+0000","last_active":"2024-04-16T21:12:43.762674+0000","last_peered":"2024-04-16T21:12:43.762674+0000","last_clean":"2024-04-16T21:12:43.762674+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:12:43.762674+0000","last_undegraded":"2024-04-16T21:12:43.762674+0000","last_fullsized":"2024-04-16T21:12:43.762674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.762572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:58.032922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'147","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.855560+0000","last_change":"2024-04-16T21:12:34.855594+0000","last_active":"2024-04-16T21:12:34.855560+0000","last_peered":"2024-04-16T21:12:34.855560+0000","last_clean":"2024-04-16T21:12:34.855560+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:12:34.855560+0000","last_undegraded":"2024-04-16T21:12:34.855560+0000","last_fullsized":"2024-04-16T21:12:34.855560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.855500+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:22.893155+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.297506+0000","last_change":"2024-04-16T21:12:29.297528+0000","last_active":"2024-04-16T21:12:29.297506+0000","last_peered":"2024-04-16T21:12:29.297506+0000","last_clean":"2024-04-16T21:12:29.297506+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:12:29.297506+0000","last_undegraded":"2024-04-16T21:12:29.297506+0000","last_fullsized":"2024-04-16T21:12:29.297506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.297434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:45.096730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.443393+0000","last_change":"2024-04-16T21:12:07.443416+0000","last_active":"2024-04-16T21:12:07.443393+0000","last_peered":"2024-04-16T21:12:07.443393+0000","last_clean":"2024-04-16T21:12:07.443393+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:12:07.443393+0000","last_undegraded":"2024-04-16T21:12:07.443393+0000","last_fullsized":"2024-04-16T21:12:07.443393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.443331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:23.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.706740+0000","last_change":"2024-04-16T21:12:53.706762+0000","last_active":"2024-04-16T21:12:53.706740+0000","last_peered":"2024-04-16T21:12:53.706740+0000","last_clean":"2024-04-16T21:12:53.706740+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:12:53.706740+0000","last_undegraded":"2024-04-16T21:12:53.706740+0000","last_fullsized":"2024-04-16T21:12:53.706740+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.706674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.706674+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.706674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:53.668881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'159","reported_seq":1390,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.133236+0000","last_change":"2024-04-16T21:13:02.133278+0000","last_active":"2024-04-16T21:13:02.133236+0000","last_peered":"2024-04-16T21:13:02.133236+0000","last_clean":"2024-04-16T21:13:02.133236+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:13:02.133236+0000","last_undegraded":"2024-04-16T21:13:02.133236+0000","last_fullsized":"2024-04-16T21:13:02.133236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:13:02.133177+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:13:02.133177+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.133177+0000","objects_scrubbed":43,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:10.287136+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8422,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.850776+0000","last_change":"2024-04-16T21:12:33.850776+0000","last_active":"2024-04-16T21:12:33.850776+0000","last_peered":"2024-04-16T21:12:33.850776+0000","last_clean":"2024-04-16T21:12:33.850776+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:12:33.850776+0000","last_undegraded":"2024-04-16T21:12:33.850776+0000","last_fullsized":"2024-04-16T21:12:33.850776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.850667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:48.121017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'10","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.431983+0000","last_change":"2024-04-16T21:12:23.431983+0000","last_active":"2024-04-16T21:12:23.431983+0000","last_peered":"2024-04-16T21:12:23.431983+0000","last_clean":"2024-04-16T21:12:23.431983+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:12:23.431983+0000","last_undegraded":"2024-04-16T21:12:23.431983+0000","last_fullsized":"2024-04-16T21:12:23.431983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.431886+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:37.702235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'706","reported_seq":2082,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.853855+0000","last_change":"2024-04-16T21:12:01.853885+0000","last_active":"2024-04-16T21:12:01.853855+0000","last_peered":"2024-04-16T21:12:01.853855+0000","last_clean":"2024-04-16T21:12:01.853855+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:12:01.853855+0000","last_undegraded":"2024-04-16T21:12:01.853855+0000","last_fullsized":"2024-04-16T21:12:01.853855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.853792+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.601423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'542","reported_seq":41300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.597266+0000","last_change":"2024-04-16T21:11:54.597266+0000","last_active":"2024-04-16T21:11:54.597266+0000","last_peered":"2024-04-16T21:11:54.597266+0000","last_clean":"2024-04-16T21:11:54.597266+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:11:54.597266+0000","last_undegraded":"2024-04-16T21:11:54.597266+0000","last_fullsized":"2024-04-16T21:11:54.597266+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'542","last_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_deep_scrub":"52'542","last_deep_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.597093+0000","objects_scrubbed":23,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:16.539182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'18","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.434844+0000","last_change":"2024-04-16T21:11:54.434914+0000","last_active":"2024-04-16T21:11:54.434844+0000","last_peered":"2024-04-16T21:11:54.434844+0000","last_clean":"2024-04-16T21:11:54.434844+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:11:54.434844+0000","last_undegraded":"2024-04-16T21:11:54.434844+0000","last_fullsized":"2024-04-16T21:11:54.434844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.434778+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:53.824584+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.707330+0000","last_change":"2024-04-16T21:12:36.707352+0000","last_active":"2024-04-16T21:12:36.707330+0000","last_peered":"2024-04-16T21:12:36.707330+0000","last_clean":"2024-04-16T21:12:36.707330+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:12:36.707330+0000","last_undegraded":"2024-04-16T21:12:36.707330+0000","last_fullsized":"2024-04-16T21:12:36.707330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.707266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:58.955541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.560163+0000","last_change":"2024-04-16T19:24:33.384522+0000","last_active":"2024-04-16T21:11:44.560163+0000","last_peered":"2024-04-16T21:11:44.560163+0000","last_clean":"2024-04-16T21:11:44.560163+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:11:44.560163+0000","last_undegraded":"2024-04-16T21:11:44.560163+0000","last_fullsized":"2024-04-16T21:11:44.560163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'13","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.860330+0000","last_change":"2024-04-16T21:12:43.860357+0000","last_active":"2024-04-16T21:12:43.860330+0000","last_peered":"2024-04-16T21:12:43.860330+0000","last_clean":"2024-04-16T21:12:43.860330+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:12:43.860330+0000","last_undegraded":"2024-04-16T21:12:43.860330+0000","last_fullsized":"2024-04-16T21:12:43.860330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.860269+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:26.916002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.311803+0000","last_change":"2024-04-16T21:12:39.311841+0000","last_active":"2024-04-16T21:12:39.311803+0000","last_peered":"2024-04-16T21:12:39.311803+0000","last_clean":"2024-04-16T21:12:39.311803+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:12:39.311803+0000","last_undegraded":"2024-04-16T21:12:39.311803+0000","last_fullsized":"2024-04-16T21:12:39.311803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.311721+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:11.016514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.303819+0000","last_change":"2024-04-16T21:12:29.303842+0000","last_active":"2024-04-16T21:12:29.303819+0000","last_peered":"2024-04-16T21:12:29.303819+0000","last_clean":"2024-04-16T21:12:29.303819+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:12:29.303819+0000","last_undegraded":"2024-04-16T21:12:29.303819+0000","last_fullsized":"2024-04-16T21:12:29.303819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.303752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:17.910045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.776294+0000","last_change":"2024-04-16T21:12:00.776335+0000","last_active":"2024-04-16T21:12:00.776294+0000","last_peered":"2024-04-16T21:12:00.776294+0000","last_clean":"2024-04-16T21:12:00.776294+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:12:00.776294+0000","last_undegraded":"2024-04-16T21:12:00.776294+0000","last_fullsized":"2024-04-16T21:12:00.776294+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.776223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:20.604086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.398019+0000","last_change":"2024-04-16T19:47:33.980051+0000","last_active":"2024-04-16T21:11:34.398019+0000","last_peered":"2024-04-16T21:11:34.398019+0000","last_clean":"2024-04-16T21:11:34.398019+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:11:34.398019+0000","last_undegraded":"2024-04-16T21:11:34.398019+0000","last_fullsized":"2024-04-16T21:11:34.398019+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'222","reported_seq":2177,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.980519+0000","last_change":"2024-04-16T21:12:16.980550+0000","last_active":"2024-04-16T21:12:16.980519+0000","last_peered":"2024-04-16T21:12:16.980519+0000","last_clean":"2024-04-16T21:12:16.980519+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:12:16.980519+0000","last_undegraded":"2024-04-16T21:12:16.980519+0000","last_fullsized":"2024-04-16T21:12:16.980519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'222","last_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_deep_scrub":"52'222","last_deep_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.980456+0000","objects_scrubbed":55,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:41.646543+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14526,"num_omap_keys":59,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.913891+0000","last_change":"2024-04-16T21:12:10.913891+0000","last_active":"2024-04-16T21:12:10.913891+0000","last_peered":"2024-04-16T21:12:10.913891+0000","last_clean":"2024-04-16T21:12:10.913891+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:12:10.913891+0000","last_undegraded":"2024-04-16T21:12:10.913891+0000","last_fullsized":"2024-04-16T21:12:10.913891+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.913806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:17.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'24","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.799425+0000","last_change":"2024-04-16T21:12:26.799425+0000","last_active":"2024-04-16T21:12:26.799425+0000","last_peered":"2024-04-16T21:12:26.799425+0000","last_clean":"2024-04-16T21:12:26.799425+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:12:26.799425+0000","last_undegraded":"2024-04-16T21:12:26.799425+0000","last_fullsized":"2024-04-16T21:12:26.799425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.799297+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:18.472986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.652503+0000","last_change":"2024-04-16T21:11:59.652525+0000","last_active":"2024-04-16T21:11:59.652503+0000","last_peered":"2024-04-16T21:11:59.652503+0000","last_clean":"2024-04-16T21:11:59.652503+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:11:59.652503+0000","last_undegraded":"2024-04-16T21:11:59.652503+0000","last_fullsized":"2024-04-16T21:11:59.652503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.652420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:18.517529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'592","reported_seq":1363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.540243+0000","last_change":"2024-04-16T21:12:00.540298+0000","last_active":"2024-04-16T21:12:00.540243+0000","last_peered":"2024-04-16T21:12:00.540243+0000","last_clean":"2024-04-16T21:12:00.540243+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:12:00.540243+0000","last_undegraded":"2024-04-16T21:12:00.540243+0000","last_fullsized":"2024-04-16T21:12:00.540243+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'592","last_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_deep_scrub":"52'592","last_deep_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.540183+0000","objects_scrubbed":19,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:48.575472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'32","reported_seq":340,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.552192+0000","last_change":"2024-04-16T21:11:43.552192+0000","last_active":"2024-04-16T21:11:43.552192+0000","last_peered":"2024-04-16T21:11:43.552192+0000","last_clean":"2024-04-16T21:11:43.552192+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:11:43.552192+0000","last_undegraded":"2024-04-16T21:11:43.552192+0000","last_fullsized":"2024-04-16T21:11:43.552192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.552108+0000","objects_scrubbed":9,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.135567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.442869+0000","last_change":"2024-04-16T21:12:53.442890+0000","last_active":"2024-04-16T21:12:53.442869+0000","last_peered":"2024-04-16T21:12:53.442869+0000","last_clean":"2024-04-16T21:12:53.442869+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:12:53.442869+0000","last_undegraded":"2024-04-16T21:12:53.442869+0000","last_fullsized":"2024-04-16T21:12:53.442869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.442802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.442802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.442802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:51.858172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.797312+0000","last_change":"2024-04-16T21:12:45.797312+0000","last_active":"2024-04-16T21:12:45.797312+0000","last_peered":"2024-04-16T21:12:45.797312+0000","last_clean":"2024-04-16T21:12:45.797312+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:12:45.797312+0000","last_undegraded":"2024-04-16T21:12:45.797312+0000","last_fullsized":"2024-04-16T21:12:45.797312+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.797229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.797229+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.797229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:33.834884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"52'5","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.725079+0000","last_change":"2024-04-16T21:12:45.725111+0000","last_active":"2024-04-16T21:12:45.725079+0000","last_peered":"2024-04-16T21:12:45.725079+0000","last_clean":"2024-04-16T21:12:45.725079+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:12:45.725079+0000","last_undegraded":"2024-04-16T21:12:45.725079+0000","last_fullsized":"2024-04-16T21:12:45.725079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'5","last_scrub_stamp":"2024-04-16T21:12:45.725020+0000","last_deep_scrub":"52'5","last_deep_scrub_stamp":"2024-04-16T21:12:45.725020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.725020+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:32.022824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'32","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.284966+0000","last_change":"2024-04-16T21:12:36.284996+0000","last_active":"2024-04-16T21:12:36.284966+0000","last_peered":"2024-04-16T21:12:36.284966+0000","last_clean":"2024-04-16T21:12:36.284966+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:12:36.284966+0000","last_undegraded":"2024-04-16T21:12:36.284966+0000","last_fullsized":"2024-04-16T21:12:36.284966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.284903+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:50.863442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.384509+0000","last_change":"2024-04-16T21:12:11.384509+0000","last_active":"2024-04-16T21:12:11.384509+0000","last_peered":"2024-04-16T21:12:11.384509+0000","last_clean":"2024-04-16T21:12:11.384509+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:12:11.384509+0000","last_undegraded":"2024-04-16T21:12:11.384509+0000","last_fullsized":"2024-04-16T21:12:11.384509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.384424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:06.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.514240+0000","last_change":"2024-04-16T21:12:02.514262+0000","last_active":"2024-04-16T21:12:02.514240+0000","last_peered":"2024-04-16T21:12:02.514240+0000","last_clean":"2024-04-16T21:12:02.514240+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:12:02.514240+0000","last_undegraded":"2024-04-16T21:12:02.514240+0000","last_fullsized":"2024-04-16T21:12:02.514240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.514179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:44.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:34.391931+0000","last_change":"2024-04-16T19:47:33.987216+0000","last_active":"2024-04-16T21:11:34.391931+0000","last_peered":"2024-04-16T21:11:34.391931+0000","last_clean":"2024-04-16T21:11:34.391931+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:11:34.391931+0000","last_undegraded":"2024-04-16T21:11:34.391931+0000","last_fullsized":"2024-04-16T21:11:34.391931+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:47:32.964340+0000","last_clean_scrub_stamp":"2024-04-16T19:47:32.964340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'172","reported_seq":1564,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.949162+0000","last_change":"2024-04-16T21:12:12.949215+0000","last_active":"2024-04-16T21:12:12.949162+0000","last_peered":"2024-04-16T21:12:12.949162+0000","last_clean":"2024-04-16T21:12:12.949162+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:12:12.949162+0000","last_undegraded":"2024-04-16T21:12:12.949162+0000","last_fullsized":"2024-04-16T21:12:12.949162+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.949100+0000","objects_scrubbed":45,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.338906+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8810,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.869596+0000","last_change":"2024-04-16T21:12:09.869596+0000","last_active":"2024-04-16T21:12:09.869596+0000","last_peered":"2024-04-16T21:12:09.869596+0000","last_clean":"2024-04-16T21:12:09.869596+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:12:09.869596+0000","last_undegraded":"2024-04-16T21:12:09.869596+0000","last_fullsized":"2024-04-16T21:12:09.869596+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.869508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:34.535595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"52'16","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.491403+0000","last_change":"2024-04-16T21:12:21.491403+0000","last_active":"2024-04-16T21:12:21.491403+0000","last_peered":"2024-04-16T21:12:21.491403+0000","last_clean":"2024-04-16T21:12:21.491403+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:12:21.491403+0000","last_undegraded":"2024-04-16T21:12:21.491403+0000","last_fullsized":"2024-04-16T21:12:21.491403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.491306+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:09.623796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.693490+0000","last_change":"2024-04-16T21:12:01.693513+0000","last_active":"2024-04-16T21:12:01.693490+0000","last_peered":"2024-04-16T21:12:01.693490+0000","last_clean":"2024-04-16T21:12:01.693490+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:12:01.693490+0000","last_undegraded":"2024-04-16T21:12:01.693490+0000","last_fullsized":"2024-04-16T21:12:01.693490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.693430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:04.866318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'249","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.401831+0000","last_change":"2024-04-16T21:11:48.401911+0000","last_active":"2024-04-16T21:11:48.401831+0000","last_peered":"2024-04-16T21:11:48.401831+0000","last_clean":"2024-04-16T21:11:48.401831+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:11:48.401831+0000","last_undegraded":"2024-04-16T21:11:48.401831+0000","last_fullsized":"2024-04-16T21:11:48.401831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.401765+0000","objects_scrubbed":19,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:02.153361+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'12","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.683222+0000","last_change":"2024-04-16T21:12:12.683275+0000","last_active":"2024-04-16T21:12:12.683222+0000","last_peered":"2024-04-16T21:12:12.683222+0000","last_clean":"2024-04-16T21:12:12.683222+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:12:12.683222+0000","last_undegraded":"2024-04-16T21:12:12.683222+0000","last_fullsized":"2024-04-16T21:12:12.683222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.683160+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.072966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:10:59.387647+0000","last_change":"2024-04-16T19:24:35.396574+0000","last_active":"2024-04-16T21:10:59.387647+0000","last_peered":"2024-04-16T21:10:59.387647+0000","last_clean":"2024-04-16T21:10:59.387647+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:10:59.387647+0000","last_undegraded":"2024-04-16T21:10:59.387647+0000","last_fullsized":"2024-04-16T21:10:59.387647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:34.375642+0000","last_clean_scrub_stamp":"2024-04-16T19:24:34.375642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.501423+0000","last_change":"2024-04-16T21:12:59.501448+0000","last_active":"2024-04-16T21:12:59.501423+0000","last_peered":"2024-04-16T21:12:59.501423+0000","last_clean":"2024-04-16T21:12:59.501423+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:12:59.501423+0000","last_undegraded":"2024-04-16T21:12:59.501423+0000","last_fullsized":"2024-04-16T21:12:59.501423+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.501352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.501352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.501352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:42.659929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"52'4","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.017072+0000","last_change":"2024-04-16T21:12:36.017103+0000","last_active":"2024-04-16T21:12:36.017072+0000","last_peered":"2024-04-16T21:12:36.017072+0000","last_clean":"2024-04-16T21:12:36.017072+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:12:36.017072+0000","last_undegraded":"2024-04-16T21:12:36.017072+0000","last_fullsized":"2024-04-16T21:12:36.017072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.017012+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:24.149502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'78","reported_seq":236,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.736528+0000","last_change":"2024-04-16T21:12:55.736594+0000","last_active":"2024-04-16T21:12:55.736528+0000","last_peered":"2024-04-16T21:12:55.736528+0000","last_clean":"2024-04-16T21:12:55.736528+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:12:55.736528+0000","last_undegraded":"2024-04-16T21:12:55.736528+0000","last_fullsized":"2024-04-16T21:12:55.736528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:55.736452+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:55.736452+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.736452+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:37.460668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.466444+0000","last_change":"2024-04-16T21:12:11.466444+0000","last_active":"2024-04-16T21:12:11.466444+0000","last_peered":"2024-04-16T21:12:11.466444+0000","last_clean":"2024-04-16T21:12:11.466444+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:12:11.466444+0000","last_undegraded":"2024-04-16T21:12:11.466444+0000","last_fullsized":"2024-04-16T21:12:11.466444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.466351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:26.935725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.718042+0000","last_change":"2024-04-16T21:12:16.718042+0000","last_active":"2024-04-16T21:12:16.718042+0000","last_peered":"2024-04-16T21:12:16.718042+0000","last_clean":"2024-04-16T21:12:16.718042+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:12:16.718042+0000","last_undegraded":"2024-04-16T21:12:16.718042+0000","last_fullsized":"2024-04-16T21:12:16.718042+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.717926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:06.936826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'19","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.886963+0000","last_change":"2024-04-16T21:12:06.887055+0000","last_active":"2024-04-16T21:12:06.886963+0000","last_peered":"2024-04-16T21:12:06.886963+0000","last_clean":"2024-04-16T21:12:06.886963+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:12:06.886963+0000","last_undegraded":"2024-04-16T21:12:06.886963+0000","last_fullsized":"2024-04-16T21:12:06.886963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.886889+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:22.161394+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.055453+0000","last_change":"2024-04-16T21:12:41.055453+0000","last_active":"2024-04-16T21:12:41.055453+0000","last_peered":"2024-04-16T21:12:41.055453+0000","last_clean":"2024-04-16T21:12:41.055453+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:12:41.055453+0000","last_undegraded":"2024-04-16T21:12:41.055453+0000","last_fullsized":"2024-04-16T21:12:41.055453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.055365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:29.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.713970+0000","last_change":"2024-04-16T21:12:49.713970+0000","last_active":"2024-04-16T21:12:49.713970+0000","last_peered":"2024-04-16T21:12:49.713970+0000","last_clean":"2024-04-16T21:12:49.713970+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:12:49.713970+0000","last_undegraded":"2024-04-16T21:12:49.713970+0000","last_fullsized":"2024-04-16T21:12:49.713970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.713863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.713863+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.713863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:18:21.306923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'18","reported_seq":257,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.679302+0000","last_change":"2024-04-16T21:12:03.679355+0000","last_active":"2024-04-16T21:12:03.679302+0000","last_peered":"2024-04-16T21:12:03.679302+0000","last_clean":"2024-04-16T21:12:03.679302+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:12:03.679302+0000","last_undegraded":"2024-04-16T21:12:03.679302+0000","last_fullsized":"2024-04-16T21:12:03.679302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.679242+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:31.737391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.087447+0000","last_change":"2024-04-16T21:12:50.087447+0000","last_active":"2024-04-16T21:12:50.087447+0000","last_peered":"2024-04-16T21:12:50.087447+0000","last_clean":"2024-04-16T21:12:50.087447+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:12:50.087447+0000","last_undegraded":"2024-04-16T21:12:50.087447+0000","last_fullsized":"2024-04-16T21:12:50.087447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.087360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.087360+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.087360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:16.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.747433+0000","last_change":"2024-04-16T21:12:44.747433+0000","last_active":"2024-04-16T21:12:44.747433+0000","last_peered":"2024-04-16T21:12:44.747433+0000","last_clean":"2024-04-16T21:12:44.747433+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:12:44.747433+0000","last_undegraded":"2024-04-16T21:12:44.747433+0000","last_fullsized":"2024-04-16T21:12:44.747433+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.747346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.747346+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.747346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:59.017695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'10","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.673321+0000","last_change":"2024-04-16T21:12:02.673359+0000","last_active":"2024-04-16T21:12:02.673321+0000","last_peered":"2024-04-16T21:12:02.673321+0000","last_clean":"2024-04-16T21:12:02.673321+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:12:02.673321+0000","last_undegraded":"2024-04-16T21:12:02.673321+0000","last_fullsized":"2024-04-16T21:12:02.673321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.673266+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:06.001051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.894866+0000","last_change":"2024-04-16T21:12:51.894866+0000","last_active":"2024-04-16T21:12:51.894866+0000","last_peered":"2024-04-16T21:12:51.894866+0000","last_clean":"2024-04-16T21:12:51.894866+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:12:51.894866+0000","last_undegraded":"2024-04-16T21:12:51.894866+0000","last_fullsized":"2024-04-16T21:12:51.894866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.894777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.894777+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.894777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:01.243554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.228883+0000","last_change":"2024-04-16T21:12:59.228909+0000","last_active":"2024-04-16T21:12:59.228883+0000","last_peered":"2024-04-16T21:12:59.228883+0000","last_clean":"2024-04-16T21:12:59.228883+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:12:59.228883+0000","last_undegraded":"2024-04-16T21:12:59.228883+0000","last_fullsized":"2024-04-16T21:12:59.228883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.228813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.228813+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.228813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:23.181732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'13","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.447533+0000","last_change":"2024-04-16T21:11:57.447598+0000","last_active":"2024-04-16T21:11:57.447533+0000","last_peered":"2024-04-16T21:11:57.447533+0000","last_clean":"2024-04-16T21:11:57.447533+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:11:57.447533+0000","last_undegraded":"2024-04-16T21:11:57.447533+0000","last_fullsized":"2024-04-16T21:11:57.447533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.447471+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:25.758892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.002453+0000","last_change":"2024-04-16T21:12:37.002453+0000","last_active":"2024-04-16T21:12:37.002453+0000","last_peered":"2024-04-16T21:12:37.002453+0000","last_clean":"2024-04-16T21:12:37.002453+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:12:37.002453+0000","last_undegraded":"2024-04-16T21:12:37.002453+0000","last_fullsized":"2024-04-16T21:12:37.002453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.002335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:36.977300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:09.393952+0000","last_change":"2024-04-16T19:24:33.388804+0000","last_active":"2024-04-16T21:11:09.393952+0000","last_peered":"2024-04-16T21:11:09.393952+0000","last_clean":"2024-04-16T21:11:09.393952+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:11:09.393952+0000","last_undegraded":"2024-04-16T21:11:09.393952+0000","last_fullsized":"2024-04-16T21:11:09.393952+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'20","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.649822+0000","last_change":"2024-04-16T21:12:04.649822+0000","last_active":"2024-04-16T21:12:04.649822+0000","last_peered":"2024-04-16T21:12:04.649822+0000","last_clean":"2024-04-16T21:12:04.649822+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:12:04.649822+0000","last_undegraded":"2024-04-16T21:12:04.649822+0000","last_fullsized":"2024-04-16T21:12:04.649822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.649711+0000","objects_scrubbed":9,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:32.961132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.869842+0000","last_change":"2024-04-16T21:12:27.869867+0000","last_active":"2024-04-16T21:12:27.869842+0000","last_peered":"2024-04-16T21:12:27.869842+0000","last_clean":"2024-04-16T21:12:27.869842+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:12:27.869842+0000","last_undegraded":"2024-04-16T21:12:27.869842+0000","last_fullsized":"2024-04-16T21:12:27.869842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.869765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:16.833314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.069536+0000","last_change":"2024-04-16T21:13:03.069560+0000","last_active":"2024-04-16T21:13:03.069536+0000","last_peered":"2024-04-16T21:13:03.069536+0000","last_clean":"2024-04-16T21:13:03.069536+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:13:03.069536+0000","last_undegraded":"2024-04-16T21:13:03.069536+0000","last_fullsized":"2024-04-16T21:13:03.069536+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.069470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.069470+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.069470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:52.947626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'28","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.947946+0000","last_change":"2024-04-16T21:12:13.947946+0000","last_active":"2024-04-16T21:12:13.947946+0000","last_peered":"2024-04-16T21:12:13.947946+0000","last_clean":"2024-04-16T21:12:13.947946+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:12:13.947946+0000","last_undegraded":"2024-04-16T21:12:13.947946+0000","last_fullsized":"2024-04-16T21:12:13.947946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'28","last_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_deep_scrub":"52'28","last_deep_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.947818+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:20.063878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.699335+0000","last_change":"2024-04-16T21:12:35.699335+0000","last_active":"2024-04-16T21:12:35.699335+0000","last_peered":"2024-04-16T21:12:35.699335+0000","last_clean":"2024-04-16T21:12:35.699335+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:12:35.699335+0000","last_undegraded":"2024-04-16T21:12:35.699335+0000","last_fullsized":"2024-04-16T21:12:35.699335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.699249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:06:40.252661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.105048+0000","last_change":"2024-04-16T21:13:00.105072+0000","last_active":"2024-04-16T21:13:00.105048+0000","last_peered":"2024-04-16T21:13:00.105048+0000","last_clean":"2024-04-16T21:13:00.105048+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:13:00.105048+0000","last_undegraded":"2024-04-16T21:13:00.105048+0000","last_fullsized":"2024-04-16T21:13:00.105048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.104978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.104978+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.104978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:13.108327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'12","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.624929+0000","last_change":"2024-04-16T21:11:55.624974+0000","last_active":"2024-04-16T21:11:55.624929+0000","last_peered":"2024-04-16T21:11:55.624929+0000","last_clean":"2024-04-16T21:11:55.624929+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:11:55.624929+0000","last_undegraded":"2024-04-16T21:11:55.624929+0000","last_fullsized":"2024-04-16T21:11:55.624929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.624828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:36.218593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.022020+0000","last_change":"2024-04-16T21:12:38.022020+0000","last_active":"2024-04-16T21:12:38.022020+0000","last_peered":"2024-04-16T21:12:38.022020+0000","last_clean":"2024-04-16T21:12:38.022020+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:12:38.022020+0000","last_undegraded":"2024-04-16T21:12:38.022020+0000","last_fullsized":"2024-04-16T21:12:38.022020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.021905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:52.292254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650049+0000","last_change":"2024-04-16T19:24:33.385498+0000","last_active":"2024-04-16T21:11:43.650049+0000","last_peered":"2024-04-16T21:11:43.650049+0000","last_clean":"2024-04-16T21:11:43.650049+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:11:43.650049+0000","last_undegraded":"2024-04-16T21:11:43.650049+0000","last_fullsized":"2024-04-16T21:11:43.650049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'26","reported_seq":312,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.939446+0000","last_change":"2024-04-16T21:12:15.939510+0000","last_active":"2024-04-16T21:12:15.939446+0000","last_peered":"2024-04-16T21:12:15.939446+0000","last_clean":"2024-04-16T21:12:15.939446+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:12:15.939446+0000","last_undegraded":"2024-04-16T21:12:15.939446+0000","last_fullsized":"2024-04-16T21:12:15.939446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.939383+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:49:32.014798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.320242+0000","last_change":"2024-04-16T21:12:50.320242+0000","last_active":"2024-04-16T21:12:50.320242+0000","last_peered":"2024-04-16T21:12:50.320242+0000","last_clean":"2024-04-16T21:12:50.320242+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:12:50.320242+0000","last_undegraded":"2024-04-16T21:12:50.320242+0000","last_fullsized":"2024-04-16T21:12:50.320242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.320150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.320150+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.320150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:58.474109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.764753+0000","last_change":"2024-04-16T21:12:41.764753+0000","last_active":"2024-04-16T21:12:41.764753+0000","last_peered":"2024-04-16T21:12:41.764753+0000","last_clean":"2024-04-16T21:12:41.764753+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:12:41.764753+0000","last_undegraded":"2024-04-16T21:12:41.764753+0000","last_fullsized":"2024-04-16T21:12:41.764753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.764666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:53.367336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'46","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.408711+0000","last_change":"2024-04-16T21:11:46.408756+0000","last_active":"2024-04-16T21:11:46.408711+0000","last_peered":"2024-04-16T21:11:46.408711+0000","last_clean":"2024-04-16T21:11:46.408711+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:11:46.408711+0000","last_undegraded":"2024-04-16T21:11:46.408711+0000","last_fullsized":"2024-04-16T21:11:46.408711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'46","last_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_deep_scrub":"52'46","last_deep_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.408648+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:38.312038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.425196+0000","last_change":"2024-04-16T21:12:51.425238+0000","last_active":"2024-04-16T21:12:51.425196+0000","last_peered":"2024-04-16T21:12:51.425196+0000","last_clean":"2024-04-16T21:12:51.425196+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:12:51.425196+0000","last_undegraded":"2024-04-16T21:12:51.425196+0000","last_fullsized":"2024-04-16T21:12:51.425196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.425114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.425114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.425114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:07.760170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:04.387643+0000","last_change":"2024-04-16T19:24:33.384757+0000","last_active":"2024-04-16T21:11:04.387643+0000","last_peered":"2024-04-16T21:11:04.387643+0000","last_clean":"2024-04-16T21:11:04.387643+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:11:04.387643+0000","last_undegraded":"2024-04-16T21:11:04.387643+0000","last_fullsized":"2024-04-16T21:11:04.387643+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'20","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.919349+0000","last_change":"2024-04-16T21:11:47.919430+0000","last_active":"2024-04-16T21:11:47.919349+0000","last_peered":"2024-04-16T21:11:47.919349+0000","last_clean":"2024-04-16T21:11:47.919349+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:11:47.919349+0000","last_undegraded":"2024-04-16T21:11:47.919349+0000","last_fullsized":"2024-04-16T21:11:47.919349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.919276+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:11:50.792760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.763777+0000","last_change":"2024-04-16T21:12:53.763799+0000","last_active":"2024-04-16T21:12:53.763777+0000","last_peered":"2024-04-16T21:12:53.763777+0000","last_clean":"2024-04-16T21:12:53.763777+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:12:53.763777+0000","last_undegraded":"2024-04-16T21:12:53.763777+0000","last_fullsized":"2024-04-16T21:12:53.763777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.763711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.763711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.763711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:59:29.416236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:42.291939+0000","last_change":"2024-04-16T19:24:33.388528+0000","last_active":"2024-04-16T21:11:42.291939+0000","last_peered":"2024-04-16T21:11:42.291939+0000","last_clean":"2024-04-16T21:11:42.291939+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:11:42.291939+0000","last_undegraded":"2024-04-16T21:11:42.291939+0000","last_fullsized":"2024-04-16T21:11:42.291939+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'17","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.209595+0000","last_change":"2024-04-16T21:11:47.209653+0000","last_active":"2024-04-16T21:11:47.209595+0000","last_peered":"2024-04-16T21:11:47.209595+0000","last_clean":"2024-04-16T21:11:47.209595+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:11:47.209595+0000","last_undegraded":"2024-04-16T21:11:47.209595+0000","last_fullsized":"2024-04-16T21:11:47.209595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.209532+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:30.862167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.844046+0000","last_change":"2024-04-16T21:12:31.844046+0000","last_active":"2024-04-16T21:12:31.844046+0000","last_peered":"2024-04-16T21:12:31.844046+0000","last_clean":"2024-04-16T21:12:31.844046+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:12:31.844046+0000","last_undegraded":"2024-04-16T21:12:31.844046+0000","last_fullsized":"2024-04-16T21:12:31.844046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.843954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:05.611280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.827107+0000","last_change":"2024-04-16T21:13:02.827128+0000","last_active":"2024-04-16T21:13:02.827107+0000","last_peered":"2024-04-16T21:13:02.827107+0000","last_clean":"2024-04-16T21:13:02.827107+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:13:02.827107+0000","last_undegraded":"2024-04-16T21:13:02.827107+0000","last_fullsized":"2024-04-16T21:13:02.827107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.827047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.827047+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.827047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:43.172076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'27","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.894735+0000","last_change":"2024-04-16T21:11:49.894811+0000","last_active":"2024-04-16T21:11:49.894735+0000","last_peered":"2024-04-16T21:11:49.894735+0000","last_clean":"2024-04-16T21:11:49.894735+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:11:49.894735+0000","last_undegraded":"2024-04-16T21:11:49.894735+0000","last_fullsized":"2024-04-16T21:11:49.894735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.894658+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.290532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.206178+0000","last_change":"2024-04-16T21:13:00.206200+0000","last_active":"2024-04-16T21:13:00.206178+0000","last_peered":"2024-04-16T21:13:00.206178+0000","last_clean":"2024-04-16T21:13:00.206178+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:13:00.206178+0000","last_undegraded":"2024-04-16T21:13:00.206178+0000","last_fullsized":"2024-04-16T21:13:00.206178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.206117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.206117+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.206117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:12.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.863200+0000","last_change":"2024-04-16T21:12:05.863236+0000","last_active":"2024-04-16T21:12:05.863200+0000","last_peered":"2024-04-16T21:12:05.863200+0000","last_clean":"2024-04-16T21:12:05.863200+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:12:05.863200+0000","last_undegraded":"2024-04-16T21:12:05.863200+0000","last_fullsized":"2024-04-16T21:12:05.863200+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.863135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:16.509928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'10","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.658621+0000","last_change":"2024-04-16T21:11:58.658671+0000","last_active":"2024-04-16T21:11:58.658621+0000","last_peered":"2024-04-16T21:11:58.658621+0000","last_clean":"2024-04-16T21:11:58.658621+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:11:58.658621+0000","last_undegraded":"2024-04-16T21:11:58.658621+0000","last_fullsized":"2024-04-16T21:11:58.658621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.658555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:31.251581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.764316+0000","last_change":"2024-04-16T21:12:42.764340+0000","last_active":"2024-04-16T21:12:42.764316+0000","last_peered":"2024-04-16T21:12:42.764316+0000","last_clean":"2024-04-16T21:12:42.764316+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:12:42.764316+0000","last_undegraded":"2024-04-16T21:12:42.764316+0000","last_fullsized":"2024-04-16T21:12:42.764316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.764250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:42.739215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.650075+0000","last_change":"2024-04-16T19:24:33.385037+0000","last_active":"2024-04-16T21:11:43.650075+0000","last_peered":"2024-04-16T21:11:43.650075+0000","last_clean":"2024-04-16T21:11:43.650075+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:11:43.650075+0000","last_undegraded":"2024-04-16T21:11:43.650075+0000","last_fullsized":"2024-04-16T21:11:43.650075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T19:24:32.368281+0000","last_clean_scrub_stamp":"2024-04-16T19:24:32.368281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'22","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.811095+0000","last_change":"2024-04-16T21:11:53.811157+0000","last_active":"2024-04-16T21:11:53.811095+0000","last_peered":"2024-04-16T21:11:53.811095+0000","last_clean":"2024-04-16T21:11:53.811095+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:11:53.811095+0000","last_undegraded":"2024-04-16T21:11:53.811095+0000","last_fullsized":"2024-04-16T21:11:53.811095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.811028+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:31.711738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.431513+0000","last_change":"2024-04-16T21:12:12.431513+0000","last_active":"2024-04-16T21:12:12.431513+0000","last_peered":"2024-04-16T21:12:12.431513+0000","last_clean":"2024-04-16T21:12:12.431513+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:12:12.431513+0000","last_undegraded":"2024-04-16T21:12:12.431513+0000","last_fullsized":"2024-04-16T21:12:12.431513+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.431389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:06.860124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.343823+0000","last_change":"2024-04-16T21:12:51.343823+0000","last_active":"2024-04-16T21:12:51.343823+0000","last_peered":"2024-04-16T21:12:51.343823+0000","last_clean":"2024-04-16T21:12:51.343823+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:12:51.343823+0000","last_undegraded":"2024-04-16T21:12:51.343823+0000","last_fullsized":"2024-04-16T21:12:51.343823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.343738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.343738+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.343738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:41.221893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'6","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.660156+0000","last_change":"2024-04-16T21:12:03.660192+0000","last_active":"2024-04-16T21:12:03.660156+0000","last_peered":"2024-04-16T21:12:03.660156+0000","last_clean":"2024-04-16T21:12:03.660156+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:12:03.660156+0000","last_undegraded":"2024-04-16T21:12:03.660156+0000","last_fullsized":"2024-04-16T21:12:03.660156+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.660092+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:42.368715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.153401+0000","last_change":"2024-04-16T21:12:44.153443+0000","last_active":"2024-04-16T21:12:44.153401+0000","last_peered":"2024-04-16T21:12:44.153401+0000","last_clean":"2024-04-16T21:12:44.153401+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:12:44.153401+0000","last_undegraded":"2024-04-16T21:12:44.153401+0000","last_fullsized":"2024-04-16T21:12:44.153401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.153324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:59.952621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.744632+0000","last_change":"2024-04-16T21:12:48.744655+0000","last_active":"2024-04-16T21:12:48.744632+0000","last_peered":"2024-04-16T21:12:48.744632+0000","last_clean":"2024-04-16T21:12:48.744632+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:12:48.744632+0000","last_undegraded":"2024-04-16T21:12:48.744632+0000","last_fullsized":"2024-04-16T21:12:48.744632+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.744563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.744563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.744563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:55.192951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'26","reported_seq":302,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.166590+0000","last_change":"2024-04-16T21:11:46.166636+0000","last_active":"2024-04-16T21:11:46.166590+0000","last_peered":"2024-04-16T21:11:46.166590+0000","last_clean":"2024-04-16T21:11:46.166590+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:11:46.166590+0000","last_undegraded":"2024-04-16T21:11:46.166590+0000","last_fullsized":"2024-04-16T21:11:46.166590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.166525+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:05.994388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.322477+0000","last_change":"2024-04-16T21:12:52.322514+0000","last_active":"2024-04-16T21:12:52.322477+0000","last_peered":"2024-04-16T21:12:52.322477+0000","last_clean":"2024-04-16T21:12:52.322477+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:12:52.322477+0000","last_undegraded":"2024-04-16T21:12:52.322477+0000","last_fullsized":"2024-04-16T21:12:52.322477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.322391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.322391+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.322391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:54.915074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.830870+0000","last_change":"2024-04-16T21:12:56.830905+0000","last_active":"2024-04-16T21:12:56.830870+0000","last_peered":"2024-04-16T21:12:56.830870+0000","last_clean":"2024-04-16T21:12:56.830870+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:12:56.830870+0000","last_undegraded":"2024-04-16T21:12:56.830870+0000","last_fullsized":"2024-04-16T21:12:56.830870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.830806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.830806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.830806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:25:52.367671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'22","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.232333+0000","last_change":"2024-04-16T21:11:55.232333+0000","last_active":"2024-04-16T21:11:55.232333+0000","last_peered":"2024-04-16T21:11:55.232333+0000","last_clean":"2024-04-16T21:11:55.232333+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:11:55.232333+0000","last_undegraded":"2024-04-16T21:11:55.232333+0000","last_fullsized":"2024-04-16T21:11:55.232333+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.232184+0000","objects_scrubbed":10,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:19.898271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.214826+0000","last_change":"2024-04-16T21:13:01.214865+0000","last_active":"2024-04-16T21:13:01.214826+0000","last_peered":"2024-04-16T21:13:01.214826+0000","last_clean":"2024-04-16T21:13:01.214826+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:13:01.214826+0000","last_undegraded":"2024-04-16T21:13:01.214826+0000","last_fullsized":"2024-04-16T21:13:01.214826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.214754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.214754+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.214754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:34.223459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.395601+0000","last_change":"2024-04-16T21:12:48.395624+0000","last_active":"2024-04-16T21:12:48.395601+0000","last_peered":"2024-04-16T21:12:48.395601+0000","last_clean":"2024-04-16T21:12:48.395601+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:12:48.395601+0000","last_undegraded":"2024-04-16T21:12:48.395601+0000","last_fullsized":"2024-04-16T21:12:48.395601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.395534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.395534+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.395534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:38.273689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'6","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.936034+0000","last_change":"2024-04-16T21:12:14.936034+0000","last_active":"2024-04-16T21:12:14.936034+0000","last_peered":"2024-04-16T21:12:14.936034+0000","last_clean":"2024-04-16T21:12:14.936034+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:12:14.936034+0000","last_undegraded":"2024-04-16T21:12:14.936034+0000","last_fullsized":"2024-04-16T21:12:14.936034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.935939+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:02.653615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.048488+0000","last_change":"2024-04-16T21:12:58.048511+0000","last_active":"2024-04-16T21:12:58.048488+0000","last_peered":"2024-04-16T21:12:58.048488+0000","last_clean":"2024-04-16T21:12:58.048488+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:12:58.048488+0000","last_undegraded":"2024-04-16T21:12:58.048488+0000","last_fullsized":"2024-04-16T21:12:58.048488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.048410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.048410+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.048410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:35.340118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.804237+0000","last_change":"2024-04-16T21:13:01.804262+0000","last_active":"2024-04-16T21:13:01.804237+0000","last_peered":"2024-04-16T21:13:01.804237+0000","last_clean":"2024-04-16T21:13:01.804237+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:13:01.804237+0000","last_undegraded":"2024-04-16T21:13:01.804237+0000","last_fullsized":"2024-04-16T21:13:01.804237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.804169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.804169+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.804169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:57.074561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":264074,"num_omap_keys":1072,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5288,"ondisk_log_size":5288,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3599,"ondisk_log_size":3599,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1190,"ondisk_log_size":1190,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16207,"ondisk_log_size":16207,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5669,"ondisk_log_size":5669,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9989,"num_omap_keys":63,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641571,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28652,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104828948,"statfs":{"total":107374182400,"available":107344842752,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641572,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59084,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798516,"statfs":{"total":107374182400,"available":107313680384,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641571,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59728,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797872,"statfs":{"total":107374182400,"available":107313020928,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641572,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46748,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810852,"statfs":{"total":107374182400,"available":107326312448,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641572,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46756,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810844,"statfs":{"total":107374182400,"available":107326304256,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641571,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":64120,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":104793480,"statfs":{"total":107374182400,"available":107308523520,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:13:06.823 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.823 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.823 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.824 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.824 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.824 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.825 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.825 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.825 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.825 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.826 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.826 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.826 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.827 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.827 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.827 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.828 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-16T19:24:44.395404+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=44, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.828 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.829 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.829 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.829 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.829 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.830 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.830 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.830 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-16T19:23:46.108852+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=23, tm_sec=46, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.831 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.831 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.831 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-04-16T19:25:48.497865+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=25, tm_sec=48, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.832 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.832 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.833 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.833 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.833 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.833 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.834 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.834 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.835 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.835 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2024-04-16T19:47:32.964340+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=47, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.835 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-16T19:24:34.375642+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=34, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.836 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.836 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.836 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.836 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.837 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-16T19:24:32.368281+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=19, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=107, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=16, tm_hour=21, tm_min=11, tm_sec=40, tm_wday=1, tm_yday=107, tm_isdst=0) 2024-04-16T21:13:06.837 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-16T21:13:26.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-16T21:13:27.195 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-04-16T21:13:27.220 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-04-16T21:13:27.223 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":3336,"stamp":"2024-04-16T21:13:25.379692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930856,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":398442,"num_read_kb":309398,"num_write":36788,"num_write_kb":10719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1472,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":577214,"num_omap_keys":2487,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33297,"ondisk_log_size":33297,"up":1042,"acting":1042,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1042,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":629145600,"kb_used":292608,"kb_used_data":11152,"kb_used_omap":9,"kb_used_meta":281398,"kb_avail":628852992,"statfs":{"total":644245094400,"available":643945463808,"internally_reserved":0,"allocated":11419648,"data_stored":3891712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":288152225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":87130,"num_omap_keys":352,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005471"},"pg_stats":[{"pgid":"9.35","version":"52'41","reported_seq":316,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.577077+0000","last_change":"2024-04-16T21:11:52.577152+0000","last_active":"2024-04-16T21:11:52.577077+0000","last_peered":"2024-04-16T21:11:52.577077+0000","last_clean":"2024-04-16T21:11:52.577077+0000","last_became_active":"2024-04-16T19:25:09.465713+0000","last_became_peered":"2024-04-16T19:25:09.465713+0000","last_unstale":"2024-04-16T21:11:52.577077+0000","last_undegraded":"2024-04-16T21:11:52.577077+0000","last_fullsized":"2024-04-16T21:11:52.577077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'41","last_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_deep_scrub":"52'41","last_deep_scrub_stamp":"2024-04-16T21:11:52.577012+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.577012+0000","objects_scrubbed":13,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:52:59.327819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20710,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":452,"num_read_kb":401,"num_write":211,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.243008+0000","last_change":"2024-04-16T21:12:46.243008+0000","last_active":"2024-04-16T21:12:46.243008+0000","last_peered":"2024-04-16T21:12:46.243008+0000","last_clean":"2024-04-16T21:12:46.243008+0000","last_became_active":"2024-04-16T19:24:33.392720+0000","last_became_peered":"2024-04-16T19:24:33.392720+0000","last_unstale":"2024-04-16T21:12:46.243008+0000","last_undegraded":"2024-04-16T21:12:46.243008+0000","last_fullsized":"2024-04-16T21:12:46.243008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.242914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.242914+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.242914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:23.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.901218+0000","last_change":"2024-04-16T21:12:53.901241+0000","last_active":"2024-04-16T21:12:53.901218+0000","last_peered":"2024-04-16T21:12:53.901218+0000","last_clean":"2024-04-16T21:12:53.901218+0000","last_became_active":"2024-04-16T19:24:35.396922+0000","last_became_peered":"2024-04-16T19:24:35.396922+0000","last_unstale":"2024-04-16T21:12:53.901218+0000","last_undegraded":"2024-04-16T21:12:53.901218+0000","last_fullsized":"2024-04-16T21:12:53.901218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.901152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.901152+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.901152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:18:02.292925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.34","version":"52'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.722895+0000","last_change":"2024-04-16T21:11:56.722928+0000","last_active":"2024-04-16T21:11:56.722895+0000","last_peered":"2024-04-16T21:11:56.722895+0000","last_clean":"2024-04-16T21:11:56.722895+0000","last_became_active":"2024-04-16T19:25:09.467369+0000","last_became_peered":"2024-04-16T19:25:09.467369+0000","last_unstale":"2024-04-16T21:11:56.722895+0000","last_undegraded":"2024-04-16T21:11:56.722895+0000","last_fullsized":"2024-04-16T21:11:56.722895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:11:56.722838+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.722838+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:40.375473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.098255+0000","last_change":"2024-04-16T21:12:55.098278+0000","last_active":"2024-04-16T21:12:55.098255+0000","last_peered":"2024-04-16T21:12:55.098255+0000","last_clean":"2024-04-16T21:12:55.098255+0000","last_became_active":"2024-04-16T19:24:33.383824+0000","last_became_peered":"2024-04-16T19:24:33.383824+0000","last_unstale":"2024-04-16T21:12:55.098255+0000","last_undegraded":"2024-04-16T21:12:55.098255+0000","last_fullsized":"2024-04-16T21:12:55.098255+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.098190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.098190+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.098190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:04:10.352974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.303685+0000","last_change":"2024-04-16T21:12:56.303709+0000","last_active":"2024-04-16T21:12:56.303685+0000","last_peered":"2024-04-16T21:12:56.303685+0000","last_clean":"2024-04-16T21:12:56.303685+0000","last_became_active":"2024-04-16T19:24:35.393078+0000","last_became_peered":"2024-04-16T19:24:35.393078+0000","last_unstale":"2024-04-16T21:12:56.303685+0000","last_undegraded":"2024-04-16T21:12:56.303685+0000","last_fullsized":"2024-04-16T21:12:56.303685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.303621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.303621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.303621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:54.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"52'22","reported_seq":329,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.702468+0000","last_change":"2024-04-16T21:12:14.702518+0000","last_active":"2024-04-16T21:12:14.702468+0000","last_peered":"2024-04-16T21:12:14.702468+0000","last_clean":"2024-04-16T21:12:14.702468+0000","last_became_active":"2024-04-16T19:25:09.457177+0000","last_became_peered":"2024-04-16T19:25:09.457177+0000","last_unstale":"2024-04-16T21:12:14.702468+0000","last_undegraded":"2024-04-16T21:12:14.702468+0000","last_fullsized":"2024-04-16T21:12:14.702468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:14.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.702406+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:42.760555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":534,"num_read_kb":362,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:07.386318+0000","last_change":"2024-04-16T21:13:07.386338+0000","last_active":"2024-04-16T21:13:07.386318+0000","last_peered":"2024-04-16T21:13:07.386318+0000","last_clean":"2024-04-16T21:13:07.386318+0000","last_became_active":"2024-04-16T19:24:33.388768+0000","last_became_peered":"2024-04-16T19:24:33.388768+0000","last_unstale":"2024-04-16T21:13:07.386318+0000","last_undegraded":"2024-04-16T21:13:07.386318+0000","last_fullsized":"2024-04-16T21:13:07.386318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:07.386258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:07.386258+0000","last_clean_scrub_stamp":"2024-04-16T21:13:07.386258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:03:43.437682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.701798+0000","last_change":"2024-04-16T21:12:32.701821+0000","last_active":"2024-04-16T21:12:32.701798+0000","last_peered":"2024-04-16T21:12:32.701798+0000","last_clean":"2024-04-16T21:12:32.701798+0000","last_became_active":"2024-04-16T19:24:35.400370+0000","last_became_peered":"2024-04-16T19:24:35.400370+0000","last_unstale":"2024-04-16T21:12:32.701798+0000","last_undegraded":"2024-04-16T21:12:32.701798+0000","last_fullsized":"2024-04-16T21:12:32.701798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.701734+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.701734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:27.130470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"52'22","reported_seq":252,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.610592+0000","last_change":"2024-04-16T21:11:47.610645+0000","last_active":"2024-04-16T21:11:47.610592+0000","last_peered":"2024-04-16T21:11:47.610592+0000","last_clean":"2024-04-16T21:11:47.610592+0000","last_became_active":"2024-04-16T19:25:09.466801+0000","last_became_peered":"2024-04-16T19:25:09.466801+0000","last_unstale":"2024-04-16T21:11:47.610592+0000","last_undegraded":"2024-04-16T21:11:47.610592+0000","last_fullsized":"2024-04-16T21:11:47.610592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:47.610532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.610532+0000","objects_scrubbed":9,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:13.030892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":326,"num_read_kb":232,"num_write":119,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.920425+0000","last_change":"2024-04-16T21:12:50.920425+0000","last_active":"2024-04-16T21:12:50.920425+0000","last_peered":"2024-04-16T21:12:50.920425+0000","last_clean":"2024-04-16T21:12:50.920425+0000","last_became_active":"2024-04-16T19:24:33.389726+0000","last_became_peered":"2024-04-16T19:24:33.389726+0000","last_unstale":"2024-04-16T21:12:50.920425+0000","last_undegraded":"2024-04-16T21:12:50.920425+0000","last_fullsized":"2024-04-16T21:12:50.920425+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.920340+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:03:25.179453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.372858+0000","last_change":"2024-04-16T21:12:49.372882+0000","last_active":"2024-04-16T21:12:49.372858+0000","last_peered":"2024-04-16T21:12:49.372858+0000","last_clean":"2024-04-16T21:12:49.372858+0000","last_became_active":"2024-04-16T19:24:35.394817+0000","last_became_peered":"2024-04-16T19:24:35.394817+0000","last_unstale":"2024-04-16T21:12:49.372858+0000","last_undegraded":"2024-04-16T21:12:49.372858+0000","last_fullsized":"2024-04-16T21:12:49.372858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.372787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.372787+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.372787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:51.965470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.31","version":"52'12","reported_seq":294,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.719715+0000","last_change":"2024-04-16T21:11:58.719715+0000","last_active":"2024-04-16T21:11:58.719715+0000","last_peered":"2024-04-16T21:11:58.719715+0000","last_clean":"2024-04-16T21:11:58.719715+0000","last_became_active":"2024-04-16T19:25:09.459556+0000","last_became_peered":"2024-04-16T19:25:09.459556+0000","last_unstale":"2024-04-16T21:11:58.719715+0000","last_undegraded":"2024-04-16T21:11:58.719715+0000","last_fullsized":"2024-04-16T21:11:58.719715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:58.719610+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.719610+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:50.501313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":452,"num_read_kb":304,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.307424+0000","last_change":"2024-04-16T21:12:55.307464+0000","last_active":"2024-04-16T21:12:55.307424+0000","last_peered":"2024-04-16T21:12:55.307424+0000","last_clean":"2024-04-16T21:12:55.307424+0000","last_became_active":"2024-04-16T19:24:33.387467+0000","last_became_peered":"2024-04-16T19:24:33.387467+0000","last_unstale":"2024-04-16T21:12:55.307424+0000","last_undegraded":"2024-04-16T21:12:55.307424+0000","last_fullsized":"2024-04-16T21:12:55.307424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.307348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.307348+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.307348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:10:07.841672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.797340+0000","last_change":"2024-04-16T21:12:59.797381+0000","last_active":"2024-04-16T21:12:59.797340+0000","last_peered":"2024-04-16T21:12:59.797340+0000","last_clean":"2024-04-16T21:12:59.797340+0000","last_became_active":"2024-04-16T19:24:35.400350+0000","last_became_peered":"2024-04-16T19:24:35.400350+0000","last_unstale":"2024-04-16T21:12:59.797340+0000","last_undegraded":"2024-04-16T21:12:59.797340+0000","last_fullsized":"2024-04-16T21:12:59.797340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.797260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.797260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.797260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:41.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.30","version":"52'27","reported_seq":224,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.166108+0000","last_change":"2024-04-16T21:11:51.166108+0000","last_active":"2024-04-16T21:11:51.166108+0000","last_peered":"2024-04-16T21:11:51.166108+0000","last_clean":"2024-04-16T21:11:51.166108+0000","last_became_active":"2024-04-16T19:25:09.461569+0000","last_became_peered":"2024-04-16T19:25:09.461569+0000","last_unstale":"2024-04-16T21:11:51.166108+0000","last_undegraded":"2024-04-16T21:11:51.166108+0000","last_fullsized":"2024-04-16T21:11:51.166108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:51.165987+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.165987+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:01.812779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":169,"num_write":133,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:05.782726+0000","last_change":"2024-04-16T21:13:05.782748+0000","last_active":"2024-04-16T21:13:05.782726+0000","last_peered":"2024-04-16T21:13:05.782726+0000","last_clean":"2024-04-16T21:13:05.782726+0000","last_became_active":"2024-04-16T19:24:33.383070+0000","last_became_peered":"2024-04-16T19:24:33.383070+0000","last_unstale":"2024-04-16T21:13:05.782726+0000","last_undegraded":"2024-04-16T21:13:05.782726+0000","last_fullsized":"2024-04-16T21:13:05.782726+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:05.782665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:05.782665+0000","last_clean_scrub_stamp":"2024-04-16T21:13:05.782665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:18.236918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.679817+0000","last_change":"2024-04-16T21:12:34.679817+0000","last_active":"2024-04-16T21:12:34.679817+0000","last_peered":"2024-04-16T21:12:34.679817+0000","last_clean":"2024-04-16T21:12:34.679817+0000","last_became_active":"2024-04-16T19:24:35.400152+0000","last_became_peered":"2024-04-16T19:24:35.400152+0000","last_unstale":"2024-04-16T21:12:34.679817+0000","last_undegraded":"2024-04-16T21:12:34.679817+0000","last_fullsized":"2024-04-16T21:12:34.679817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:34.679697+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.679697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:07:57.367334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.33","version":"52'18","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.672517+0000","last_change":"2024-04-16T21:12:09.672592+0000","last_active":"2024-04-16T21:12:09.672517+0000","last_peered":"2024-04-16T21:12:09.672517+0000","last_clean":"2024-04-16T21:12:09.672517+0000","last_became_active":"2024-04-16T19:25:41.508225+0000","last_became_peered":"2024-04-16T19:25:41.508225+0000","last_unstale":"2024-04-16T21:12:09.672517+0000","last_undegraded":"2024-04-16T21:12:09.672517+0000","last_fullsized":"2024-04-16T21:12:09.672517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:09.672453+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.672453+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.111035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":290,"num_read_kb":196,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.239995+0000","last_change":"2024-04-16T21:12:48.240036+0000","last_active":"2024-04-16T21:12:48.239995+0000","last_peered":"2024-04-16T21:12:48.239995+0000","last_clean":"2024-04-16T21:12:48.239995+0000","last_became_active":"2024-04-16T19:24:33.393076+0000","last_became_peered":"2024-04-16T19:24:33.393076+0000","last_unstale":"2024-04-16T21:12:48.239995+0000","last_undegraded":"2024-04-16T21:12:48.239995+0000","last_fullsized":"2024-04-16T21:12:48.239995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.239912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.239912+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.239912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:01.243261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.835059+0000","last_change":"2024-04-16T21:12:47.835059+0000","last_active":"2024-04-16T21:12:47.835059+0000","last_peered":"2024-04-16T21:12:47.835059+0000","last_clean":"2024-04-16T21:12:47.835059+0000","last_became_active":"2024-04-16T19:24:35.395793+0000","last_became_peered":"2024-04-16T19:24:35.395793+0000","last_unstale":"2024-04-16T21:12:47.835059+0000","last_undegraded":"2024-04-16T21:12:47.835059+0000","last_fullsized":"2024-04-16T21:12:47.835059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.834965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.834965+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.834965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:33.241934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.32","version":"52'22","reported_seq":255,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.654830+0000","last_change":"2024-04-16T21:11:56.654830+0000","last_active":"2024-04-16T21:11:56.654830+0000","last_peered":"2024-04-16T21:11:56.654830+0000","last_clean":"2024-04-16T21:11:56.654830+0000","last_became_active":"2024-04-16T19:25:09.463265+0000","last_became_peered":"2024-04-16T19:25:09.463265+0000","last_unstale":"2024-04-16T21:11:56.654830+0000","last_undegraded":"2024-04-16T21:11:56.654830+0000","last_fullsized":"2024-04-16T21:11:56.654830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:56.654666+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.654666+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.077668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":329,"num_read_kb":235,"num_write":122,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.751421+0000","last_change":"2024-04-16T21:12:47.751466+0000","last_active":"2024-04-16T21:12:47.751421+0000","last_peered":"2024-04-16T21:12:47.751421+0000","last_clean":"2024-04-16T21:12:47.751421+0000","last_became_active":"2024-04-16T19:24:33.390071+0000","last_became_peered":"2024-04-16T19:24:33.390071+0000","last_unstale":"2024-04-16T21:12:47.751421+0000","last_undegraded":"2024-04-16T21:12:47.751421+0000","last_fullsized":"2024-04-16T21:12:47.751421+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.751343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.751343+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.751343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:36.357636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.815463+0000","last_change":"2024-04-16T21:12:39.815463+0000","last_active":"2024-04-16T21:12:39.815463+0000","last_peered":"2024-04-16T21:12:39.815463+0000","last_clean":"2024-04-16T21:12:39.815463+0000","last_became_active":"2024-04-16T19:24:35.390926+0000","last_became_peered":"2024-04-16T19:24:35.390926+0000","last_unstale":"2024-04-16T21:12:39.815463+0000","last_undegraded":"2024-04-16T21:12:39.815463+0000","last_fullsized":"2024-04-16T21:12:39.815463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.815374+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.815374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:11.502076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3d","version":"52'12","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.701548+0000","last_change":"2024-04-16T21:12:00.701594+0000","last_active":"2024-04-16T21:12:00.701548+0000","last_peered":"2024-04-16T21:12:00.701548+0000","last_clean":"2024-04-16T21:12:00.701548+0000","last_became_active":"2024-04-16T19:25:09.466812+0000","last_became_peered":"2024-04-16T19:25:09.466812+0000","last_unstale":"2024-04-16T21:12:00.701548+0000","last_undegraded":"2024-04-16T21:12:00.701548+0000","last_fullsized":"2024-04-16T21:12:00.701548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:00.701488+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.701488+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:22.203073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":178,"num_write":59,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.707007+0000","last_change":"2024-04-16T21:12:38.707030+0000","last_active":"2024-04-16T21:12:38.707007+0000","last_peered":"2024-04-16T21:12:38.707007+0000","last_clean":"2024-04-16T21:12:38.707007+0000","last_became_active":"2024-04-16T19:24:35.400064+0000","last_became_peered":"2024-04-16T19:24:35.400064+0000","last_unstale":"2024-04-16T21:12:38.707007+0000","last_undegraded":"2024-04-16T21:12:38.707007+0000","last_fullsized":"2024-04-16T21:12:38.707007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.706933+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.706933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:10.393634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:08.192780+0000","last_change":"2024-04-16T21:13:08.192801+0000","last_active":"2024-04-16T21:13:08.192780+0000","last_peered":"2024-04-16T21:13:08.192780+0000","last_clean":"2024-04-16T21:13:08.192780+0000","last_became_active":"2024-04-16T19:24:33.385585+0000","last_became_peered":"2024-04-16T19:24:33.385585+0000","last_unstale":"2024-04-16T21:13:08.192780+0000","last_undegraded":"2024-04-16T21:13:08.192780+0000","last_fullsized":"2024-04-16T21:13:08.192780+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:08.192719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:08.192719+0000","last_clean_scrub_stamp":"2024-04-16T21:13:08.192719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:34.108588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3c","version":"52'12","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.436184+0000","last_change":"2024-04-16T21:11:55.436228+0000","last_active":"2024-04-16T21:11:55.436184+0000","last_peered":"2024-04-16T21:11:55.436184+0000","last_clean":"2024-04-16T21:11:55.436184+0000","last_became_active":"2024-04-16T19:25:09.459153+0000","last_became_peered":"2024-04-16T19:25:09.459153+0000","last_unstale":"2024-04-16T21:11:55.436184+0000","last_undegraded":"2024-04-16T21:11:55.436184+0000","last_fullsized":"2024-04-16T21:11:55.436184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.436126+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.436126+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:58.763911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":385,"num_read_kb":251,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.914098+0000","last_change":"2024-04-16T21:12:16.914127+0000","last_active":"2024-04-16T21:12:16.914098+0000","last_peered":"2024-04-16T21:12:16.914098+0000","last_clean":"2024-04-16T21:12:16.914098+0000","last_became_active":"2024-04-16T19:25:41.503368+0000","last_became_peered":"2024-04-16T19:25:41.503368+0000","last_unstale":"2024-04-16T21:12:16.914098+0000","last_undegraded":"2024-04-16T21:12:16.914098+0000","last_fullsized":"2024-04-16T21:12:16.914098+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.914034+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.914034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:05.877583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.324499+0000","last_change":"2024-04-16T21:12:53.324499+0000","last_active":"2024-04-16T21:12:53.324499+0000","last_peered":"2024-04-16T21:12:53.324499+0000","last_clean":"2024-04-16T21:12:53.324499+0000","last_became_active":"2024-04-16T19:24:33.386426+0000","last_became_peered":"2024-04-16T19:24:33.386426+0000","last_unstale":"2024-04-16T21:12:53.324499+0000","last_undegraded":"2024-04-16T21:12:53.324499+0000","last_fullsized":"2024-04-16T21:12:53.324499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.324385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.324385+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.324385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:34.628013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3f","version":"52'10","reported_seq":211,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.882687+0000","last_change":"2024-04-16T21:11:51.882767+0000","last_active":"2024-04-16T21:11:51.882687+0000","last_peered":"2024-04-16T21:11:51.882687+0000","last_clean":"2024-04-16T21:11:51.882687+0000","last_became_active":"2024-04-16T19:25:09.464198+0000","last_became_peered":"2024-04-16T19:25:09.464198+0000","last_unstale":"2024-04-16T21:11:51.882687+0000","last_undegraded":"2024-04-16T21:11:51.882687+0000","last_fullsized":"2024-04-16T21:11:51.882687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:51.882609+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.882609+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:06:05.077377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":163,"num_write":54,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.890648+0000","last_change":"2024-04-16T21:12:49.890672+0000","last_active":"2024-04-16T21:12:49.890648+0000","last_peered":"2024-04-16T21:12:49.890648+0000","last_clean":"2024-04-16T21:12:49.890648+0000","last_became_active":"2024-04-16T19:24:35.395765+0000","last_became_peered":"2024-04-16T19:24:35.395765+0000","last_unstale":"2024-04-16T21:12:49.890648+0000","last_undegraded":"2024-04-16T21:12:49.890648+0000","last_fullsized":"2024-04-16T21:12:49.890648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.890587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.890587+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.890587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:17.432217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.828130+0000","last_change":"2024-04-16T21:12:58.828130+0000","last_active":"2024-04-16T21:12:58.828130+0000","last_peered":"2024-04-16T21:12:58.828130+0000","last_clean":"2024-04-16T21:12:58.828130+0000","last_became_active":"2024-04-16T19:24:33.386449+0000","last_became_peered":"2024-04-16T19:24:33.386449+0000","last_unstale":"2024-04-16T21:12:58.828130+0000","last_undegraded":"2024-04-16T21:12:58.828130+0000","last_fullsized":"2024-04-16T21:12:58.828130+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.828052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.828052+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.828052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:01.420735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3e","version":"52'20","reported_seq":249,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.881532+0000","last_change":"2024-04-16T21:12:07.881609+0000","last_active":"2024-04-16T21:12:07.881532+0000","last_peered":"2024-04-16T21:12:07.881532+0000","last_clean":"2024-04-16T21:12:07.881532+0000","last_became_active":"2024-04-16T19:25:09.463484+0000","last_became_peered":"2024-04-16T19:25:09.463484+0000","last_unstale":"2024-04-16T21:12:07.881532+0000","last_undegraded":"2024-04-16T21:12:07.881532+0000","last_fullsized":"2024-04-16T21:12:07.881532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:07.881467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.881467+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.576312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":306,"num_read_kb":199,"num_write":104,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.819489+0000","last_change":"2024-04-16T21:12:33.819489+0000","last_active":"2024-04-16T21:12:33.819489+0000","last_peered":"2024-04-16T21:12:33.819489+0000","last_clean":"2024-04-16T21:12:33.819489+0000","last_became_active":"2024-04-16T19:24:35.394025+0000","last_became_peered":"2024-04-16T19:24:35.394025+0000","last_unstale":"2024-04-16T21:12:33.819489+0000","last_undegraded":"2024-04-16T21:12:33.819489+0000","last_fullsized":"2024-04-16T21:12:33.819489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.819396+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.819396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:28.248131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.713200+0000","last_change":"2024-04-16T21:12:51.713200+0000","last_active":"2024-04-16T21:12:51.713200+0000","last_peered":"2024-04-16T21:12:51.713200+0000","last_clean":"2024-04-16T21:12:51.713200+0000","last_became_active":"2024-04-16T19:24:33.386293+0000","last_became_peered":"2024-04-16T19:24:33.386293+0000","last_unstale":"2024-04-16T21:12:51.713200+0000","last_undegraded":"2024-04-16T21:12:51.713200+0000","last_fullsized":"2024-04-16T21:12:51.713200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.713107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.713107+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.713107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:38:57.545317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.39","version":"52'16","reported_seq":206,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.428185+0000","last_change":"2024-04-16T21:11:47.428185+0000","last_active":"2024-04-16T21:11:47.428185+0000","last_peered":"2024-04-16T21:11:47.428185+0000","last_clean":"2024-04-16T21:11:47.428185+0000","last_became_active":"2024-04-16T19:25:09.462011+0000","last_became_peered":"2024-04-16T19:25:09.462011+0000","last_unstale":"2024-04-16T21:11:47.428185+0000","last_undegraded":"2024-04-16T21:11:47.428185+0000","last_fullsized":"2024-04-16T21:11:47.428185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:47.428059+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.428059+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:23.278965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":222,"num_read_kb":155,"num_write":76,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.812549+0000","last_change":"2024-04-16T21:12:37.812549+0000","last_active":"2024-04-16T21:12:37.812549+0000","last_peered":"2024-04-16T21:12:37.812549+0000","last_clean":"2024-04-16T21:12:37.812549+0000","last_became_active":"2024-04-16T19:24:35.393200+0000","last_became_peered":"2024-04-16T19:24:35.393200+0000","last_unstale":"2024-04-16T21:12:37.812549+0000","last_undegraded":"2024-04-16T21:12:37.812549+0000","last_fullsized":"2024-04-16T21:12:37.812549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.812425+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.812425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:29.486114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.427374+0000","last_change":"2024-04-16T21:12:57.427374+0000","last_active":"2024-04-16T21:12:57.427374+0000","last_peered":"2024-04-16T21:12:57.427374+0000","last_clean":"2024-04-16T21:12:57.427374+0000","last_became_active":"2024-04-16T19:24:33.389636+0000","last_became_peered":"2024-04-16T19:24:33.389636+0000","last_unstale":"2024-04-16T21:12:57.427374+0000","last_undegraded":"2024-04-16T21:12:57.427374+0000","last_fullsized":"2024-04-16T21:12:57.427374+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.427251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.427251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.427251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:09.881504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.38","version":"52'22","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.947975+0000","last_change":"2024-04-16T21:12:11.947975+0000","last_active":"2024-04-16T21:12:11.947975+0000","last_peered":"2024-04-16T21:12:11.947975+0000","last_clean":"2024-04-16T21:12:11.947975+0000","last_became_active":"2024-04-16T19:25:09.460495+0000","last_became_peered":"2024-04-16T19:25:09.460495+0000","last_unstale":"2024-04-16T21:12:11.947975+0000","last_undegraded":"2024-04-16T21:12:11.947975+0000","last_fullsized":"2024-04-16T21:12:11.947975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:12:11.947824+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.947824+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:04.586823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":158,"num_write":117,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.717970+0000","last_change":"2024-04-16T21:12:46.717970+0000","last_active":"2024-04-16T21:12:46.717970+0000","last_peered":"2024-04-16T21:12:46.717970+0000","last_clean":"2024-04-16T21:12:46.717970+0000","last_became_active":"2024-04-16T19:24:35.397170+0000","last_became_peered":"2024-04-16T19:24:35.397170+0000","last_unstale":"2024-04-16T21:12:46.717970+0000","last_undegraded":"2024-04-16T21:12:46.717970+0000","last_fullsized":"2024-04-16T21:12:46.717970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.717861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.717861+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.717861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:35.324154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.518893+0000","last_change":"2024-04-16T21:13:01.518917+0000","last_active":"2024-04-16T21:13:01.518893+0000","last_peered":"2024-04-16T21:13:01.518893+0000","last_clean":"2024-04-16T21:13:01.518893+0000","last_became_active":"2024-04-16T19:24:33.382071+0000","last_became_peered":"2024-04-16T19:24:33.382071+0000","last_unstale":"2024-04-16T21:13:01.518893+0000","last_undegraded":"2024-04-16T21:13:01.518893+0000","last_fullsized":"2024-04-16T21:13:01.518893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.518831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.518831+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.518831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:13:52.695386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:05.059034+0000","last_change":"2024-04-16T21:13:05.059055+0000","last_active":"2024-04-16T21:13:05.059034+0000","last_peered":"2024-04-16T21:13:05.059034+0000","last_clean":"2024-04-16T21:13:05.059034+0000","last_became_active":"2024-04-16T19:47:33.985994+0000","last_became_peered":"2024-04-16T19:47:33.985994+0000","last_unstale":"2024-04-16T21:13:05.059034+0000","last_undegraded":"2024-04-16T21:13:05.059034+0000","last_fullsized":"2024-04-16T21:13:05.059034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:05.058963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:05.058963+0000","last_clean_scrub_stamp":"2024-04-16T21:13:05.058963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:46.362591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1e","version":"52'142","reported_seq":1768,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:12.204553+0000","last_change":"2024-04-16T21:13:12.204590+0000","last_active":"2024-04-16T21:13:12.204553+0000","last_peered":"2024-04-16T21:13:12.204553+0000","last_clean":"2024-04-16T21:13:12.204553+0000","last_became_active":"2024-04-16T19:25:49.525406+0000","last_became_peered":"2024-04-16T19:25:49.525406+0000","last_unstale":"2024-04-16T21:13:12.204553+0000","last_undegraded":"2024-04-16T21:13:12.204553+0000","last_fullsized":"2024-04-16T21:13:12.204553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'142","last_scrub_stamp":"2024-04-16T21:13:12.204493+0000","last_deep_scrub":"52'142","last_deep_scrub_stamp":"2024-04-16T21:13:12.204493+0000","last_clean_scrub_stamp":"2024-04-16T21:13:12.204493+0000","objects_scrubbed":37,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:25.368471+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2688,"num_read_kb":2647,"num_write":160,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11326,"num_omap_keys":45,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.395220+0000","last_change":"2024-04-16T21:12:15.395220+0000","last_active":"2024-04-16T21:12:15.395220+0000","last_peered":"2024-04-16T21:12:15.395220+0000","last_clean":"2024-04-16T21:12:15.395220+0000","last_became_active":"2024-04-16T19:25:41.505567+0000","last_became_peered":"2024-04-16T19:25:41.505567+0000","last_unstale":"2024-04-16T21:12:15.395220+0000","last_undegraded":"2024-04-16T21:12:15.395220+0000","last_fullsized":"2024-04-16T21:12:15.395220+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.395104+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.395104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:09.823839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1c","version":"52'8","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.430800+0000","last_change":"2024-04-16T21:12:17.430800+0000","last_active":"2024-04-16T21:12:17.430800+0000","last_peered":"2024-04-16T21:12:17.430800+0000","last_clean":"2024-04-16T21:12:17.430800+0000","last_became_active":"2024-04-16T19:25:15.486215+0000","last_became_peered":"2024-04-16T19:25:15.486215+0000","last_unstale":"2024-04-16T21:12:17.430800+0000","last_undegraded":"2024-04-16T21:12:17.430800+0000","last_fullsized":"2024-04-16T21:12:17.430800+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:17.430693+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.430693+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:25.924925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.738465+0000","last_change":"2024-04-16T21:11:45.738465+0000","last_active":"2024-04-16T21:11:45.738465+0000","last_peered":"2024-04-16T21:11:45.738465+0000","last_clean":"2024-04-16T21:11:45.738465+0000","last_became_active":"2024-04-16T19:25:13.468989+0000","last_became_peered":"2024-04-16T19:25:13.468989+0000","last_unstale":"2024-04-16T21:11:45.738465+0000","last_undegraded":"2024-04-16T21:11:45.738465+0000","last_fullsized":"2024-04-16T21:11:45.738465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:45.738344+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.738344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:26.908264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"52'223","reported_seq":477,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.490168+0000","last_change":"2024-04-16T21:11:58.490168+0000","last_active":"2024-04-16T21:11:58.490168+0000","last_peered":"2024-04-16T21:11:58.490168+0000","last_clean":"2024-04-16T21:11:58.490168+0000","last_became_active":"2024-04-16T19:25:11.463979+0000","last_became_peered":"2024-04-16T19:25:11.463979+0000","last_unstale":"2024-04-16T21:11:58.490168+0000","last_undegraded":"2024-04-16T21:11:58.490168+0000","last_fullsized":"2024-04-16T21:11:58.490168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'223","last_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_deep_scrub":"52'223","last_deep_scrub_stamp":"2024-04-16T21:11:58.490053+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.490053+0000","objects_scrubbed":20,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:48.708953+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":336,"num_read_kb":329,"num_write":271,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1971,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.19","version":"52'29","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.926237+0000","last_change":"2024-04-16T21:11:50.926284+0000","last_active":"2024-04-16T21:11:50.926237+0000","last_peered":"2024-04-16T21:11:50.926237+0000","last_clean":"2024-04-16T21:11:50.926237+0000","last_became_active":"2024-04-16T19:25:09.464647+0000","last_became_peered":"2024-04-16T19:25:09.464647+0000","last_unstale":"2024-04-16T21:11:50.926237+0000","last_undegraded":"2024-04-16T21:11:50.926237+0000","last_fullsized":"2024-04-16T21:11:50.926237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'29","last_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_deep_scrub":"52'29","last_deep_scrub_stamp":"2024-04-16T21:11:50.926173+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.926173+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:17.509632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":460,"num_read_kb":296,"num_write":119,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.689691+0000","last_change":"2024-04-16T21:12:19.689715+0000","last_active":"2024-04-16T21:12:19.689691+0000","last_peered":"2024-04-16T21:12:19.689691+0000","last_clean":"2024-04-16T21:12:19.689691+0000","last_became_active":"2024-04-16T19:24:51.436092+0000","last_became_peered":"2024-04-16T19:24:51.436092+0000","last_unstale":"2024-04-16T21:12:19.689691+0000","last_undegraded":"2024-04-16T21:12:19.689691+0000","last_fullsized":"2024-04-16T21:12:19.689691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.689621+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.689621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.705707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.031454+0000","last_change":"2024-04-16T21:12:40.031477+0000","last_active":"2024-04-16T21:12:40.031454+0000","last_peered":"2024-04-16T21:12:40.031454+0000","last_clean":"2024-04-16T21:12:40.031454+0000","last_became_active":"2024-04-16T19:24:35.399888+0000","last_became_peered":"2024-04-16T19:24:35.399888+0000","last_unstale":"2024-04-16T21:12:40.031454+0000","last_undegraded":"2024-04-16T21:12:40.031454+0000","last_fullsized":"2024-04-16T21:12:40.031454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.031394+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.031394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:26.329198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.297056+0000","last_change":"2024-04-16T21:12:57.297097+0000","last_active":"2024-04-16T21:12:57.297056+0000","last_peered":"2024-04-16T21:12:57.297056+0000","last_clean":"2024-04-16T21:12:57.297056+0000","last_became_active":"2024-04-16T19:24:33.386552+0000","last_became_peered":"2024-04-16T19:24:33.386552+0000","last_unstale":"2024-04-16T21:12:57.297056+0000","last_undegraded":"2024-04-16T21:12:57.297056+0000","last_fullsized":"2024-04-16T21:12:57.297056+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.296983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.296983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.296983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:10.967137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"52'6","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.901446+0000","last_change":"2024-04-16T21:12:42.901472+0000","last_active":"2024-04-16T21:12:42.901446+0000","last_peered":"2024-04-16T21:12:42.901446+0000","last_clean":"2024-04-16T21:12:42.901446+0000","last_became_active":"2024-04-16T19:24:45.414498+0000","last_became_peered":"2024-04-16T19:24:45.414498+0000","last_unstale":"2024-04-16T21:12:42.901446+0000","last_undegraded":"2024-04-16T21:12:42.901446+0000","last_fullsized":"2024-04-16T21:12:42.901446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:42.901389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.901389+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:59:18.553914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":10,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.351884+0000","last_change":"2024-04-16T21:12:33.351925+0000","last_active":"2024-04-16T21:12:33.351884+0000","last_peered":"2024-04-16T21:12:33.351884+0000","last_clean":"2024-04-16T21:12:33.351884+0000","last_became_active":"2024-04-16T19:24:47.423115+0000","last_became_peered":"2024-04-16T19:24:47.423115+0000","last_unstale":"2024-04-16T21:12:33.351884+0000","last_undegraded":"2024-04-16T21:12:33.351884+0000","last_fullsized":"2024-04-16T21:12:33.351884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.351821+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.351821+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:30.327261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.757836+0000","last_change":"2024-04-16T21:12:30.757878+0000","last_active":"2024-04-16T21:12:30.757836+0000","last_peered":"2024-04-16T21:12:30.757836+0000","last_clean":"2024-04-16T21:12:30.757836+0000","last_became_active":"2024-04-16T19:24:49.432367+0000","last_became_peered":"2024-04-16T19:24:49.432367+0000","last_unstale":"2024-04-16T21:12:30.757836+0000","last_undegraded":"2024-04-16T21:12:30.757836+0000","last_fullsized":"2024-04-16T21:12:30.757836+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:30.757760+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.757760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:42.360430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.788930+0000","last_change":"2024-04-16T21:12:55.788953+0000","last_active":"2024-04-16T21:12:55.788930+0000","last_peered":"2024-04-16T21:12:55.788930+0000","last_clean":"2024-04-16T21:12:55.788930+0000","last_became_active":"2024-04-16T19:47:33.988690+0000","last_became_peered":"2024-04-16T19:47:33.988690+0000","last_unstale":"2024-04-16T21:12:55.788930+0000","last_undegraded":"2024-04-16T21:12:55.788930+0000","last_fullsized":"2024-04-16T21:12:55.788930+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.788863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.788863+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.788863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:43.044039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"52'205","reported_seq":1899,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:14.876933+0000","last_change":"2024-04-16T21:13:14.876970+0000","last_active":"2024-04-16T21:13:14.876933+0000","last_peered":"2024-04-16T21:13:14.876933+0000","last_clean":"2024-04-16T21:13:14.876933+0000","last_became_active":"2024-04-16T19:26:41.558234+0000","last_became_peered":"2024-04-16T19:26:41.558234+0000","last_unstale":"2024-04-16T21:13:14.876933+0000","last_undegraded":"2024-04-16T21:13:14.876933+0000","last_fullsized":"2024-04-16T21:13:14.876933+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'205","last_scrub_stamp":"2024-04-16T21:13:14.876875+0000","last_deep_scrub":"52'205","last_deep_scrub_stamp":"2024-04-16T21:13:14.876875+0000","last_clean_scrub_stamp":"2024-04-16T21:13:14.876875+0000","objects_scrubbed":51,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:24.127426+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2941,"num_read_kb":2881,"num_write":222,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11635,"num_omap_keys":48,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.873380+0000","last_change":"2024-04-16T21:12:15.873380+0000","last_active":"2024-04-16T21:12:15.873380+0000","last_peered":"2024-04-16T21:12:15.873380+0000","last_clean":"2024-04-16T21:12:15.873380+0000","last_became_active":"2024-04-16T19:25:41.503154+0000","last_became_peered":"2024-04-16T19:25:41.503154+0000","last_unstale":"2024-04-16T21:12:15.873380+0000","last_undegraded":"2024-04-16T21:12:15.873380+0000","last_fullsized":"2024-04-16T21:12:15.873380+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.873260+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.873260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:57:01.509024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"52'8","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.369164+0000","last_change":"2024-04-16T21:12:14.369164+0000","last_active":"2024-04-16T21:12:14.369164+0000","last_peered":"2024-04-16T21:12:14.369164+0000","last_clean":"2024-04-16T21:12:14.369164+0000","last_became_active":"2024-04-16T19:25:41.505622+0000","last_became_peered":"2024-04-16T19:25:41.505622+0000","last_unstale":"2024-04-16T21:12:14.369164+0000","last_undegraded":"2024-04-16T21:12:14.369164+0000","last_fullsized":"2024-04-16T21:12:14.369164+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:14.369033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.369033+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:29.838406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":64,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.686990+0000","last_change":"2024-04-16T21:11:57.686990+0000","last_active":"2024-04-16T21:11:57.686990+0000","last_peered":"2024-04-16T21:11:57.686990+0000","last_clean":"2024-04-16T21:11:57.686990+0000","last_became_active":"2024-04-16T19:25:13.471748+0000","last_became_peered":"2024-04-16T19:25:13.471748+0000","last_unstale":"2024-04-16T21:11:57.686990+0000","last_undegraded":"2024-04-16T21:11:57.686990+0000","last_fullsized":"2024-04-16T21:11:57.686990+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.686888+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.686888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:39.802344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"52'264","reported_seq":1009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.984786+0000","last_change":"2024-04-16T21:11:43.984786+0000","last_active":"2024-04-16T21:11:43.984786+0000","last_peered":"2024-04-16T21:11:43.984786+0000","last_clean":"2024-04-16T21:11:43.984786+0000","last_became_active":"2024-04-16T19:25:11.464753+0000","last_became_peered":"2024-04-16T19:25:11.464753+0000","last_unstale":"2024-04-16T21:11:43.984786+0000","last_undegraded":"2024-04-16T21:11:43.984786+0000","last_fullsized":"2024-04-16T21:11:43.984786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:43.984688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.984688+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:38:21.448843+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":999,"num_read_kb":849,"num_write":310,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6375,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"52'10","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.702465+0000","last_change":"2024-04-16T21:11:54.702515+0000","last_active":"2024-04-16T21:11:54.702465+0000","last_peered":"2024-04-16T21:11:54.702465+0000","last_clean":"2024-04-16T21:11:54.702465+0000","last_became_active":"2024-04-16T19:25:09.467371+0000","last_became_peered":"2024-04-16T19:25:09.467371+0000","last_unstale":"2024-04-16T21:11:54.702465+0000","last_undegraded":"2024-04-16T21:11:54.702465+0000","last_fullsized":"2024-04-16T21:11:54.702465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:54.702406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.702406+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.795058+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":128,"num_read_kb":79,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.815226+0000","last_change":"2024-04-16T21:12:10.815250+0000","last_active":"2024-04-16T21:12:10.815226+0000","last_peered":"2024-04-16T21:12:10.815226+0000","last_clean":"2024-04-16T21:12:10.815226+0000","last_became_active":"2024-04-16T19:24:51.437818+0000","last_became_peered":"2024-04-16T19:24:51.437818+0000","last_unstale":"2024-04-16T21:12:10.815226+0000","last_undegraded":"2024-04-16T21:12:10.815226+0000","last_fullsized":"2024-04-16T21:12:10.815226+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.815114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.815114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:27:54.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.821191+0000","last_change":"2024-04-16T21:12:55.821191+0000","last_active":"2024-04-16T21:12:55.821191+0000","last_peered":"2024-04-16T21:12:55.821191+0000","last_clean":"2024-04-16T21:12:55.821191+0000","last_became_active":"2024-04-16T19:24:35.397700+0000","last_became_peered":"2024-04-16T19:24:35.397700+0000","last_unstale":"2024-04-16T21:12:55.821191+0000","last_undegraded":"2024-04-16T21:12:55.821191+0000","last_fullsized":"2024-04-16T21:12:55.821191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.821108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.821108+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.821108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:44.476963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.491126+0000","last_change":"2024-04-16T21:12:55.491150+0000","last_active":"2024-04-16T21:12:55.491126+0000","last_peered":"2024-04-16T21:12:55.491126+0000","last_clean":"2024-04-16T21:12:55.491126+0000","last_became_active":"2024-04-16T19:24:33.391744+0000","last_became_peered":"2024-04-16T19:24:33.391744+0000","last_unstale":"2024-04-16T21:12:55.491126+0000","last_undegraded":"2024-04-16T21:12:55.491126+0000","last_fullsized":"2024-04-16T21:12:55.491126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:55.491061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:55.491061+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.491061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:12.884360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"52'4","reported_seq":217,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:08.347773+0000","last_change":"2024-04-16T21:13:08.347813+0000","last_active":"2024-04-16T21:13:08.347773+0000","last_peered":"2024-04-16T21:13:08.347773+0000","last_clean":"2024-04-16T21:13:08.347773+0000","last_became_active":"2024-04-16T19:24:45.412948+0000","last_became_peered":"2024-04-16T19:24:45.412948+0000","last_unstale":"2024-04-16T21:13:08.347773+0000","last_undegraded":"2024-04-16T21:13:08.347773+0000","last_fullsized":"2024-04-16T21:13:08.347773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:13:08.347704+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:13:08.347704+0000","last_clean_scrub_stamp":"2024-04-16T21:13:08.347704+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:09:12.547426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"52'69","reported_seq":219,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.619780+0000","last_change":"2024-04-16T21:12:28.619811+0000","last_active":"2024-04-16T21:12:28.619780+0000","last_peered":"2024-04-16T21:12:28.619780+0000","last_clean":"2024-04-16T21:12:28.619780+0000","last_became_active":"2024-04-16T19:24:47.421880+0000","last_became_peered":"2024-04-16T19:24:47.421880+0000","last_unstale":"2024-04-16T21:12:28.619780+0000","last_undegraded":"2024-04-16T21:12:28.619780+0000","last_fullsized":"2024-04-16T21:12:28.619780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'69","last_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_deep_scrub":"52'69","last_deep_scrub_stamp":"2024-04-16T21:12:28.619713+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.619713+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:56.443540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":96,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.604337+0000","last_change":"2024-04-16T21:12:23.604359+0000","last_active":"2024-04-16T21:12:23.604337+0000","last_peered":"2024-04-16T21:12:23.604337+0000","last_clean":"2024-04-16T21:12:23.604337+0000","last_became_active":"2024-04-16T19:24:49.426389+0000","last_became_peered":"2024-04-16T19:24:49.426389+0000","last_unstale":"2024-04-16T21:12:23.604337+0000","last_undegraded":"2024-04-16T21:12:23.604337+0000","last_fullsized":"2024-04-16T21:12:23.604337+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.604274+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.604274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:59:10.724532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.836520+0000","last_change":"2024-04-16T21:13:02.836541+0000","last_active":"2024-04-16T21:13:02.836520+0000","last_peered":"2024-04-16T21:13:02.836520+0000","last_clean":"2024-04-16T21:13:02.836520+0000","last_became_active":"2024-04-16T19:47:33.983430+0000","last_became_peered":"2024-04-16T19:47:33.983430+0000","last_unstale":"2024-04-16T21:13:02.836520+0000","last_undegraded":"2024-04-16T21:13:02.836520+0000","last_fullsized":"2024-04-16T21:13:02.836520+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.836457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.836457+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.836457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:12.185234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"52'183","reported_seq":1935,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.306641+0000","last_change":"2024-04-16T21:12:38.306641+0000","last_active":"2024-04-16T21:12:38.306641+0000","last_peered":"2024-04-16T21:12:38.306641+0000","last_clean":"2024-04-16T21:12:38.306641+0000","last_became_active":"2024-04-16T19:25:49.517926+0000","last_became_peered":"2024-04-16T19:25:49.517926+0000","last_unstale":"2024-04-16T21:12:38.306641+0000","last_undegraded":"2024-04-16T21:12:38.306641+0000","last_fullsized":"2024-04-16T21:12:38.306641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:38.306555+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.306555+0000","objects_scrubbed":52,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:07.296213+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":2781,"num_read_kb":2751,"num_write":187,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":43,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.746629+0000","last_change":"2024-04-16T21:12:00.746651+0000","last_active":"2024-04-16T21:12:00.746629+0000","last_peered":"2024-04-16T21:12:00.746629+0000","last_clean":"2024-04-16T21:12:00.746629+0000","last_became_active":"2024-04-16T19:25:41.502152+0000","last_became_peered":"2024-04-16T19:25:41.502152+0000","last_unstale":"2024-04-16T21:12:00.746629+0000","last_undegraded":"2024-04-16T21:12:00.746629+0000","last_fullsized":"2024-04-16T21:12:00.746629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.746571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.746571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:24:45.291359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"52'25","reported_seq":162,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.693813+0000","last_change":"2024-04-16T21:11:57.693880+0000","last_active":"2024-04-16T21:11:57.693813+0000","last_peered":"2024-04-16T21:11:57.693813+0000","last_clean":"2024-04-16T21:11:57.693813+0000","last_became_active":"2024-04-16T19:25:15.480155+0000","last_became_peered":"2024-04-16T19:25:15.480155+0000","last_unstale":"2024-04-16T21:11:57.693813+0000","last_undegraded":"2024-04-16T21:11:57.693813+0000","last_fullsized":"2024-04-16T21:11:57.693813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'25","last_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_deep_scrub":"52'25","last_deep_scrub_stamp":"2024-04-16T21:11:57.693743+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.693743+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:17.739736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":101,"num_write":50,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.385829+0000","last_change":"2024-04-16T21:12:04.385854+0000","last_active":"2024-04-16T21:12:04.385829+0000","last_peered":"2024-04-16T21:12:04.385829+0000","last_clean":"2024-04-16T21:12:04.385829+0000","last_became_active":"2024-04-16T19:25:13.475001+0000","last_became_peered":"2024-04-16T19:25:13.475001+0000","last_unstale":"2024-04-16T21:12:04.385829+0000","last_undegraded":"2024-04-16T21:12:04.385829+0000","last_fullsized":"2024-04-16T21:12:04.385829+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.385746+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.385746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:20.771679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"52'716","reported_seq":41938,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.996557+0000","last_change":"2024-04-16T21:11:45.996631+0000","last_active":"2024-04-16T21:11:45.996557+0000","last_peered":"2024-04-16T21:11:45.996557+0000","last_clean":"2024-04-16T21:11:45.996557+0000","last_became_active":"2024-04-16T19:25:11.462127+0000","last_became_peered":"2024-04-16T19:25:11.462127+0000","last_unstale":"2024-04-16T21:11:45.996557+0000","last_undegraded":"2024-04-16T21:11:45.996557+0000","last_fullsized":"2024-04-16T21:11:45.996557+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'716","last_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_deep_scrub":"52'716","last_deep_scrub_stamp":"2024-04-16T21:11:45.996464+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.996464+0000","objects_scrubbed":20,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:15:40.351620+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":123484,"num_read_kb":82529,"num_write":530,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"52'35","reported_seq":324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.105372+0000","last_change":"2024-04-16T21:11:44.105372+0000","last_active":"2024-04-16T21:11:44.105372+0000","last_peered":"2024-04-16T21:11:44.105372+0000","last_clean":"2024-04-16T21:11:44.105372+0000","last_became_active":"2024-04-16T19:25:09.464391+0000","last_became_peered":"2024-04-16T19:25:09.464391+0000","last_unstale":"2024-04-16T21:11:44.105372+0000","last_undegraded":"2024-04-16T21:11:44.105372+0000","last_fullsized":"2024-04-16T21:11:44.105372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'35","last_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_deep_scrub":"52'35","last_deep_scrub_stamp":"2024-04-16T21:11:44.105287+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.105287+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:44.283094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":486,"num_read_kb":329,"num_write":172,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.409254+0000","last_change":"2024-04-16T21:12:03.409254+0000","last_active":"2024-04-16T21:12:03.409254+0000","last_peered":"2024-04-16T21:12:03.409254+0000","last_clean":"2024-04-16T21:12:03.409254+0000","last_became_active":"2024-04-16T19:24:51.437767+0000","last_became_peered":"2024-04-16T19:24:51.437767+0000","last_unstale":"2024-04-16T21:12:03.409254+0000","last_undegraded":"2024-04-16T21:12:03.409254+0000","last_fullsized":"2024-04-16T21:12:03.409254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.409137+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.409137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:49.655116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.325602+0000","last_change":"2024-04-16T21:12:45.325602+0000","last_active":"2024-04-16T21:12:45.325602+0000","last_peered":"2024-04-16T21:12:45.325602+0000","last_clean":"2024-04-16T21:12:45.325602+0000","last_became_active":"2024-04-16T19:24:35.395797+0000","last_became_peered":"2024-04-16T19:24:35.395797+0000","last_unstale":"2024-04-16T21:12:45.325602+0000","last_undegraded":"2024-04-16T21:12:45.325602+0000","last_fullsized":"2024-04-16T21:12:45.325602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.325518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.325518+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.325518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:58.328868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:06.208717+0000","last_change":"2024-04-16T21:13:06.208742+0000","last_active":"2024-04-16T21:13:06.208717+0000","last_peered":"2024-04-16T21:13:06.208717+0000","last_clean":"2024-04-16T21:13:06.208717+0000","last_became_active":"2024-04-16T19:24:33.382960+0000","last_became_peered":"2024-04-16T19:24:33.382960+0000","last_unstale":"2024-04-16T21:13:06.208717+0000","last_undegraded":"2024-04-16T21:13:06.208717+0000","last_fullsized":"2024-04-16T21:13:06.208717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:06.208646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:06.208646+0000","last_clean_scrub_stamp":"2024-04-16T21:13:06.208646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:15:07.842582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"52'6","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.842517+0000","last_change":"2024-04-16T21:12:38.842543+0000","last_active":"2024-04-16T21:12:38.842517+0000","last_peered":"2024-04-16T21:12:38.842517+0000","last_clean":"2024-04-16T21:12:38.842517+0000","last_became_active":"2024-04-16T19:24:45.411697+0000","last_became_peered":"2024-04-16T19:24:45.411697+0000","last_unstale":"2024-04-16T21:12:38.842517+0000","last_undegraded":"2024-04-16T21:12:38.842517+0000","last_fullsized":"2024-04-16T21:12:38.842517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:38.842450+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.842450+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:47.336683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"52'115","reported_seq":289,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.884236+0000","last_change":"2024-04-16T21:12:25.884267+0000","last_active":"2024-04-16T21:12:25.884236+0000","last_peered":"2024-04-16T21:12:25.884236+0000","last_clean":"2024-04-16T21:12:25.884236+0000","last_became_active":"2024-04-16T19:24:47.421702+0000","last_became_peered":"2024-04-16T19:24:47.421702+0000","last_unstale":"2024-04-16T21:12:25.884236+0000","last_undegraded":"2024-04-16T21:12:25.884236+0000","last_fullsized":"2024-04-16T21:12:25.884236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:25.884183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.884183+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.270116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"52'13","reported_seq":1318,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.370369+0000","last_change":"2024-04-16T21:12:13.370396+0000","last_active":"2024-04-16T21:12:13.370369+0000","last_peered":"2024-04-16T21:12:13.370369+0000","last_clean":"2024-04-16T21:12:13.370369+0000","last_became_active":"2024-04-16T19:24:49.430827+0000","last_became_peered":"2024-04-16T19:24:49.430827+0000","last_unstale":"2024-04-16T21:12:13.370369+0000","last_undegraded":"2024-04-16T21:12:13.370369+0000","last_fullsized":"2024-04-16T21:12:13.370369+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:13.370308+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.370308+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:47.137634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"49'9","reported_seq":82,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.087222+0000","last_change":"2024-04-16T21:13:04.087271+0000","last_active":"2024-04-16T21:13:04.087222+0000","last_peered":"2024-04-16T21:13:04.087222+0000","last_clean":"2024-04-16T21:13:04.087222+0000","last_became_active":"2024-04-16T19:47:33.986025+0000","last_became_peered":"2024-04-16T19:47:33.986025+0000","last_unstale":"2024-04-16T21:13:04.087222+0000","last_undegraded":"2024-04-16T21:13:04.087222+0000","last_fullsized":"2024-04-16T21:13:04.087222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'9","last_scrub_stamp":"2024-04-16T21:13:04.087144+0000","last_deep_scrub":"49'9","last_deep_scrub_stamp":"2024-04-16T21:13:04.087144+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.087144+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:15:06.679827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"52'169","reported_seq":1520,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:11.364643+0000","last_change":"2024-04-16T21:13:11.364643+0000","last_active":"2024-04-16T21:13:11.364643+0000","last_peered":"2024-04-16T21:13:11.364643+0000","last_clean":"2024-04-16T21:13:11.364643+0000","last_became_active":"2024-04-16T19:25:49.522104+0000","last_became_peered":"2024-04-16T19:25:49.522104+0000","last_unstale":"2024-04-16T21:13:11.364643+0000","last_undegraded":"2024-04-16T21:13:11.364643+0000","last_fullsized":"2024-04-16T21:13:11.364643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'169","last_scrub_stamp":"2024-04-16T21:13:11.364546+0000","last_deep_scrub":"52'169","last_deep_scrub_stamp":"2024-04-16T21:13:11.364546+0000","last_clean_scrub_stamp":"2024-04-16T21:13:11.364546+0000","objects_scrubbed":37,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:10.156799+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2196,"num_read_kb":2198,"num_write":203,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13039,"num_omap_keys":53,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.703764+0000","last_change":"2024-04-16T21:12:02.703764+0000","last_active":"2024-04-16T21:12:02.703764+0000","last_peered":"2024-04-16T21:12:02.703764+0000","last_clean":"2024-04-16T21:12:02.703764+0000","last_became_active":"2024-04-16T19:25:41.501470+0000","last_became_peered":"2024-04-16T19:25:41.501470+0000","last_unstale":"2024-04-16T21:12:02.703764+0000","last_undegraded":"2024-04-16T21:12:02.703764+0000","last_fullsized":"2024-04-16T21:12:02.703764+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.703681+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.703681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:54:59.311639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"52'13","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.372480+0000","last_change":"2024-04-16T21:12:18.372514+0000","last_active":"2024-04-16T21:12:18.372480+0000","last_peered":"2024-04-16T21:12:18.372480+0000","last_clean":"2024-04-16T21:12:18.372480+0000","last_became_active":"2024-04-16T19:25:15.486797+0000","last_became_peered":"2024-04-16T19:25:15.486797+0000","last_unstale":"2024-04-16T21:12:18.372480+0000","last_undegraded":"2024-04-16T21:12:18.372480+0000","last_fullsized":"2024-04-16T21:12:18.372480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:18.372404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.372404+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:27.934072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1077,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":25,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.821906+0000","last_change":"2024-04-16T21:11:59.821929+0000","last_active":"2024-04-16T21:11:59.821906+0000","last_peered":"2024-04-16T21:11:59.821906+0000","last_clean":"2024-04-16T21:11:59.821906+0000","last_became_active":"2024-04-16T19:25:13.474495+0000","last_became_peered":"2024-04-16T19:25:13.474495+0000","last_unstale":"2024-04-16T21:11:59.821906+0000","last_undegraded":"2024-04-16T21:11:59.821906+0000","last_fullsized":"2024-04-16T21:11:59.821906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.821840+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.821840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.493143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"52'366","reported_seq":690,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.574732+0000","last_change":"2024-04-16T21:11:46.574732+0000","last_active":"2024-04-16T21:11:46.574732+0000","last_peered":"2024-04-16T21:11:46.574732+0000","last_clean":"2024-04-16T21:11:46.574732+0000","last_became_active":"2024-04-16T19:25:11.463443+0000","last_became_peered":"2024-04-16T19:25:11.463443+0000","last_unstale":"2024-04-16T21:11:46.574732+0000","last_undegraded":"2024-04-16T21:11:46.574732+0000","last_fullsized":"2024-04-16T21:11:46.574732+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'366","last_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_deep_scrub":"52'366","last_deep_scrub_stamp":"2024-04-16T21:11:46.574641+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.574641+0000","objects_scrubbed":27,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:24.475352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":546,"num_read_kb":533,"num_write":438,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"52'12","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.580812+0000","last_change":"2024-04-16T21:11:44.580812+0000","last_active":"2024-04-16T21:11:44.580812+0000","last_peered":"2024-04-16T21:11:44.580812+0000","last_clean":"2024-04-16T21:11:44.580812+0000","last_became_active":"2024-04-16T19:25:09.467902+0000","last_became_peered":"2024-04-16T19:25:09.467902+0000","last_unstale":"2024-04-16T21:11:44.580812+0000","last_undegraded":"2024-04-16T21:11:44.580812+0000","last_fullsized":"2024-04-16T21:11:44.580812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:44.580695+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.580695+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:05:57.775463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":54,"num_write":63,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.671998+0000","last_change":"2024-04-16T21:12:15.671998+0000","last_active":"2024-04-16T21:12:15.671998+0000","last_peered":"2024-04-16T21:12:15.671998+0000","last_clean":"2024-04-16T21:12:15.671998+0000","last_became_active":"2024-04-16T19:24:51.572723+0000","last_became_peered":"2024-04-16T19:24:51.572723+0000","last_unstale":"2024-04-16T21:12:15.671998+0000","last_undegraded":"2024-04-16T21:12:15.671998+0000","last_fullsized":"2024-04-16T21:12:15.671998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:15.671910+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.671910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:43.983331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.465832+0000","last_change":"2024-04-16T21:12:52.465832+0000","last_active":"2024-04-16T21:12:52.465832+0000","last_peered":"2024-04-16T21:12:52.465832+0000","last_clean":"2024-04-16T21:12:52.465832+0000","last_became_active":"2024-04-16T19:24:35.399479+0000","last_became_peered":"2024-04-16T19:24:35.399479+0000","last_unstale":"2024-04-16T21:12:52.465832+0000","last_undegraded":"2024-04-16T21:12:52.465832+0000","last_fullsized":"2024-04-16T21:12:52.465832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.465728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.465728+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.465728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:10:05.000052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:07.745529+0000","last_change":"2024-04-16T21:13:07.745553+0000","last_active":"2024-04-16T21:13:07.745529+0000","last_peered":"2024-04-16T21:13:07.745529+0000","last_clean":"2024-04-16T21:13:07.745529+0000","last_became_active":"2024-04-16T19:24:33.384153+0000","last_became_peered":"2024-04-16T19:24:33.384153+0000","last_unstale":"2024-04-16T21:13:07.745529+0000","last_undegraded":"2024-04-16T21:13:07.745529+0000","last_fullsized":"2024-04-16T21:13:07.745529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:07.745463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:07.745463+0000","last_clean_scrub_stamp":"2024-04-16T21:13:07.745463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:50.904041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.239994+0000","last_change":"2024-04-16T21:12:38.240020+0000","last_active":"2024-04-16T21:12:38.239994+0000","last_peered":"2024-04-16T21:12:38.239994+0000","last_clean":"2024-04-16T21:12:38.239994+0000","last_became_active":"2024-04-16T19:24:45.413165+0000","last_became_peered":"2024-04-16T19:24:45.413165+0000","last_unstale":"2024-04-16T21:12:38.239994+0000","last_undegraded":"2024-04-16T21:12:38.239994+0000","last_fullsized":"2024-04-16T21:12:38.239994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.239920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.239920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:04.921574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.327857+0000","last_change":"2024-04-16T21:12:33.327890+0000","last_active":"2024-04-16T21:12:33.327857+0000","last_peered":"2024-04-16T21:12:33.327857+0000","last_clean":"2024-04-16T21:12:33.327857+0000","last_became_active":"2024-04-16T19:24:47.419676+0000","last_became_peered":"2024-04-16T19:24:47.419676+0000","last_unstale":"2024-04-16T21:12:33.327857+0000","last_undegraded":"2024-04-16T21:12:33.327857+0000","last_fullsized":"2024-04-16T21:12:33.327857+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:33.327792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.327792+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:56.594542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"52'13","reported_seq":1320,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.937025+0000","last_change":"2024-04-16T21:12:24.937025+0000","last_active":"2024-04-16T21:12:24.937025+0000","last_peered":"2024-04-16T21:12:24.937025+0000","last_clean":"2024-04-16T21:12:24.937025+0000","last_became_active":"2024-04-16T19:24:49.428312+0000","last_became_peered":"2024-04-16T19:24:49.428312+0000","last_unstale":"2024-04-16T21:12:24.937025+0000","last_undegraded":"2024-04-16T21:12:24.937025+0000","last_fullsized":"2024-04-16T21:12:24.937025+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.936891+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.936891+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:58.704216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.816068+0000","last_change":"2024-04-16T21:12:58.816068+0000","last_active":"2024-04-16T21:12:58.816068+0000","last_peered":"2024-04-16T21:12:58.816068+0000","last_clean":"2024-04-16T21:12:58.816068+0000","last_became_active":"2024-04-16T19:47:33.981252+0000","last_became_peered":"2024-04-16T19:47:33.981252+0000","last_unstale":"2024-04-16T21:12:58.816068+0000","last_undegraded":"2024-04-16T21:12:58.816068+0000","last_fullsized":"2024-04-16T21:12:58.816068+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.815981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.815981+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.815981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:56:44.222950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"52'188","reported_seq":2057,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.351576+0000","last_change":"2024-04-16T21:12:32.351645+0000","last_active":"2024-04-16T21:12:32.351576+0000","last_peered":"2024-04-16T21:12:32.351576+0000","last_clean":"2024-04-16T21:12:32.351576+0000","last_became_active":"2024-04-16T19:25:49.514931+0000","last_became_peered":"2024-04-16T19:25:49.514931+0000","last_unstale":"2024-04-16T21:12:32.351576+0000","last_undegraded":"2024-04-16T21:12:32.351576+0000","last_fullsized":"2024-04-16T21:12:32.351576+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'188","last_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_deep_scrub":"52'188","last_deep_scrub_stamp":"2024-04-16T21:12:32.351505+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.351505+0000","objects_scrubbed":40,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:48.150801+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3125,"num_read_kb":3104,"num_write":246,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12543,"num_omap_keys":51,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.379178+0000","last_change":"2024-04-16T21:12:16.379217+0000","last_active":"2024-04-16T21:12:16.379178+0000","last_peered":"2024-04-16T21:12:16.379178+0000","last_clean":"2024-04-16T21:12:16.379178+0000","last_became_active":"2024-04-16T19:25:41.505674+0000","last_became_peered":"2024-04-16T19:25:41.505674+0000","last_unstale":"2024-04-16T21:12:16.379178+0000","last_undegraded":"2024-04-16T21:12:16.379178+0000","last_fullsized":"2024-04-16T21:12:16.379178+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.379105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.379105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:54.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"52'37","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.663863+0000","last_change":"2024-04-16T21:12:06.663901+0000","last_active":"2024-04-16T21:12:06.663863+0000","last_peered":"2024-04-16T21:12:06.663863+0000","last_clean":"2024-04-16T21:12:06.663863+0000","last_became_active":"2024-04-16T19:25:15.481911+0000","last_became_peered":"2024-04-16T19:25:15.481911+0000","last_unstale":"2024-04-16T21:12:06.663863+0000","last_undegraded":"2024-04-16T21:12:06.663863+0000","last_fullsized":"2024-04-16T21:12:06.663863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'37","last_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_deep_scrub":"52'37","last_deep_scrub_stamp":"2024-04-16T21:12:06.663802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.663802+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:20.415399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":260,"num_read_kb":207,"num_write":111,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":158,"num_omap_keys":2,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.716132+0000","last_change":"2024-04-16T21:12:07.716153+0000","last_active":"2024-04-16T21:12:07.716132+0000","last_peered":"2024-04-16T21:12:07.716132+0000","last_clean":"2024-04-16T21:12:07.716132+0000","last_became_active":"2024-04-16T19:25:13.474762+0000","last_became_peered":"2024-04-16T19:25:13.474762+0000","last_unstale":"2024-04-16T21:12:07.716132+0000","last_undegraded":"2024-04-16T21:12:07.716132+0000","last_fullsized":"2024-04-16T21:12:07.716132+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.716070+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.716070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.751360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"52'1762","reported_seq":4632,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.707742+0000","last_change":"2024-04-16T21:11:51.707808+0000","last_active":"2024-04-16T21:11:51.707742+0000","last_peered":"2024-04-16T21:11:51.707742+0000","last_clean":"2024-04-16T21:11:51.707742+0000","last_became_active":"2024-04-16T19:25:11.468510+0000","last_became_peered":"2024-04-16T19:25:11.468510+0000","last_unstale":"2024-04-16T21:11:51.707742+0000","last_undegraded":"2024-04-16T21:11:51.707742+0000","last_fullsized":"2024-04-16T21:11:51.707742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1762","last_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_deep_scrub":"52'1762","last_deep_scrub_stamp":"2024-04-16T21:11:51.707684+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.707684+0000","objects_scrubbed":22,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:25.851168+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52856,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5622,"num_read_kb":5324,"num_write":2699,"num_write_kb":1278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":89480,"num_omap_keys":462,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"52'27","reported_seq":288,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.189093+0000","last_change":"2024-04-16T21:11:48.189147+0000","last_active":"2024-04-16T21:11:48.189093+0000","last_peered":"2024-04-16T21:11:48.189093+0000","last_clean":"2024-04-16T21:11:48.189093+0000","last_became_active":"2024-04-16T19:25:09.468155+0000","last_became_peered":"2024-04-16T19:25:09.468155+0000","last_unstale":"2024-04-16T21:11:48.189093+0000","last_undegraded":"2024-04-16T21:11:48.189093+0000","last_fullsized":"2024-04-16T21:11:48.189093+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:48.189029+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.189029+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:30.304485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":406,"num_read_kb":257,"num_write":139,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.281331+0000","last_change":"2024-04-16T21:12:54.281331+0000","last_active":"2024-04-16T21:12:54.281331+0000","last_peered":"2024-04-16T21:12:54.281331+0000","last_clean":"2024-04-16T21:12:54.281331+0000","last_became_active":"2024-04-16T19:24:35.391835+0000","last_became_peered":"2024-04-16T19:24:35.391835+0000","last_unstale":"2024-04-16T21:12:54.281331+0000","last_undegraded":"2024-04-16T21:12:54.281331+0000","last_fullsized":"2024-04-16T21:12:54.281331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.281207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.281207+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.281207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:10.616263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:10.740431+0000","last_change":"2024-04-16T21:13:10.740454+0000","last_active":"2024-04-16T21:13:10.740431+0000","last_peered":"2024-04-16T21:13:10.740431+0000","last_clean":"2024-04-16T21:13:10.740431+0000","last_became_active":"2024-04-16T19:24:33.385728+0000","last_became_peered":"2024-04-16T19:24:33.385728+0000","last_unstale":"2024-04-16T21:13:10.740431+0000","last_undegraded":"2024-04-16T21:13:10.740431+0000","last_fullsized":"2024-04-16T21:13:10.740431+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:10.740367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:10.740367+0000","last_clean_scrub_stamp":"2024-04-16T21:13:10.740367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:16:21.319245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":127,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.849044+0000","last_change":"2024-04-16T21:12:44.849044+0000","last_active":"2024-04-16T21:12:44.849044+0000","last_peered":"2024-04-16T21:12:44.849044+0000","last_clean":"2024-04-16T21:12:44.849044+0000","last_became_active":"2024-04-16T19:24:45.414033+0000","last_became_peered":"2024-04-16T19:24:45.414033+0000","last_unstale":"2024-04-16T21:12:44.849044+0000","last_undegraded":"2024-04-16T21:12:44.849044+0000","last_fullsized":"2024-04-16T21:12:44.849044+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.848958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.848958+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.848958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:26.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"52'193","reported_seq":412,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.895939+0000","last_change":"2024-04-16T21:12:38.895987+0000","last_active":"2024-04-16T21:12:38.895939+0000","last_peered":"2024-04-16T21:12:38.895939+0000","last_clean":"2024-04-16T21:12:38.895939+0000","last_became_active":"2024-04-16T19:24:47.419543+0000","last_became_peered":"2024-04-16T19:24:47.419543+0000","last_unstale":"2024-04-16T21:12:38.895939+0000","last_undegraded":"2024-04-16T21:12:38.895939+0000","last_fullsized":"2024-04-16T21:12:38.895939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'193","last_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_deep_scrub":"52'193","last_deep_scrub_stamp":"2024-04-16T21:12:38.895878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.895878+0000","objects_scrubbed":9,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:45.133526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":281,"num_read_kb":270,"num_write":184,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.922119+0000","last_change":"2024-04-16T21:12:20.922119+0000","last_active":"2024-04-16T21:12:20.922119+0000","last_peered":"2024-04-16T21:12:20.922119+0000","last_clean":"2024-04-16T21:12:20.922119+0000","last_became_active":"2024-04-16T19:24:49.434211+0000","last_became_peered":"2024-04-16T19:24:49.434211+0000","last_unstale":"2024-04-16T21:12:20.922119+0000","last_undegraded":"2024-04-16T21:12:20.922119+0000","last_fullsized":"2024-04-16T21:12:20.922119+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.922033+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.922033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:09.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.873367+0000","last_change":"2024-04-16T21:12:12.873367+0000","last_active":"2024-04-16T21:12:12.873367+0000","last_peered":"2024-04-16T21:12:12.873367+0000","last_clean":"2024-04-16T21:12:12.873367+0000","last_became_active":"2024-04-16T19:24:51.436830+0000","last_became_peered":"2024-04-16T19:24:51.436830+0000","last_unstale":"2024-04-16T21:12:12.873367+0000","last_undegraded":"2024-04-16T21:12:12.873367+0000","last_fullsized":"2024-04-16T21:12:12.873367+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.873281+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.873281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:06.480261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.705482+0000","last_change":"2024-04-16T21:12:52.705482+0000","last_active":"2024-04-16T21:12:52.705482+0000","last_peered":"2024-04-16T21:12:52.705482+0000","last_clean":"2024-04-16T21:12:52.705482+0000","last_became_active":"2024-04-16T19:47:33.988024+0000","last_became_peered":"2024-04-16T19:47:33.988024+0000","last_unstale":"2024-04-16T21:12:52.705482+0000","last_undegraded":"2024-04-16T21:12:52.705482+0000","last_fullsized":"2024-04-16T21:12:52.705482+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.705395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.705395+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.705395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:16.267657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"52'158","reported_seq":1982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:10.348335+0000","last_change":"2024-04-16T21:13:10.348381+0000","last_active":"2024-04-16T21:13:10.348335+0000","last_peered":"2024-04-16T21:13:10.348335+0000","last_clean":"2024-04-16T21:13:10.348335+0000","last_became_active":"2024-04-16T19:25:49.522046+0000","last_became_peered":"2024-04-16T19:25:49.522046+0000","last_unstale":"2024-04-16T21:13:10.348335+0000","last_undegraded":"2024-04-16T21:13:10.348335+0000","last_fullsized":"2024-04-16T21:13:10.348335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'158","last_scrub_stamp":"2024-04-16T21:13:10.348272+0000","last_deep_scrub":"52'158","last_deep_scrub_stamp":"2024-04-16T21:13:10.348272+0000","last_clean_scrub_stamp":"2024-04-16T21:13:10.348272+0000","objects_scrubbed":43,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:47.360613+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2987,"num_read_kb":2942,"num_write":170,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10409,"num_omap_keys":43,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.341198+0000","last_change":"2024-04-16T21:12:17.341220+0000","last_active":"2024-04-16T21:12:17.341198+0000","last_peered":"2024-04-16T21:12:17.341198+0000","last_clean":"2024-04-16T21:12:17.341198+0000","last_became_active":"2024-04-16T19:25:41.502375+0000","last_became_peered":"2024-04-16T19:25:41.502375+0000","last_unstale":"2024-04-16T21:12:17.341198+0000","last_undegraded":"2024-04-16T21:12:17.341198+0000","last_fullsized":"2024-04-16T21:12:17.341198+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.341136+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.341136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:45.982431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"52'15","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.375445+0000","last_change":"2024-04-16T21:12:19.375473+0000","last_active":"2024-04-16T21:12:19.375445+0000","last_peered":"2024-04-16T21:12:19.375445+0000","last_clean":"2024-04-16T21:12:19.375445+0000","last_became_active":"2024-04-16T19:25:15.484831+0000","last_became_peered":"2024-04-16T19:25:15.484831+0000","last_unstale":"2024-04-16T21:12:19.375445+0000","last_undegraded":"2024-04-16T21:12:19.375445+0000","last_fullsized":"2024-04-16T21:12:19.375445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:19.375386+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.375386+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:11.049075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":50,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.377316+0000","last_change":"2024-04-16T21:12:02.377338+0000","last_active":"2024-04-16T21:12:02.377316+0000","last_peered":"2024-04-16T21:12:02.377316+0000","last_clean":"2024-04-16T21:12:02.377316+0000","last_became_active":"2024-04-16T19:25:13.471829+0000","last_became_peered":"2024-04-16T19:25:13.471829+0000","last_unstale":"2024-04-16T21:12:02.377316+0000","last_undegraded":"2024-04-16T21:12:02.377316+0000","last_fullsized":"2024-04-16T21:12:02.377316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.377250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.377250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:40:49.251281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"52'325","reported_seq":659,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.419682+0000","last_change":"2024-04-16T21:11:56.419734+0000","last_active":"2024-04-16T21:11:56.419682+0000","last_peered":"2024-04-16T21:11:56.419682+0000","last_clean":"2024-04-16T21:11:56.419682+0000","last_became_active":"2024-04-16T19:25:11.462727+0000","last_became_peered":"2024-04-16T19:25:11.462727+0000","last_unstale":"2024-04-16T21:11:56.419682+0000","last_undegraded":"2024-04-16T21:11:56.419682+0000","last_fullsized":"2024-04-16T21:11:56.419682+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'325","last_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_deep_scrub":"52'325","last_deep_scrub_stamp":"2024-04-16T21:11:56.419626+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.419626+0000","objects_scrubbed":21,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:24.477776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":444,"num_read_kb":431,"num_write":453,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6361,"num_omap_keys":29,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"52'16","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.568997+0000","last_change":"2024-04-16T21:11:45.569097+0000","last_active":"2024-04-16T21:11:45.568997+0000","last_peered":"2024-04-16T21:11:45.568997+0000","last_clean":"2024-04-16T21:11:45.568997+0000","last_became_active":"2024-04-16T19:25:09.467094+0000","last_became_peered":"2024-04-16T19:25:09.467094+0000","last_unstale":"2024-04-16T21:11:45.568997+0000","last_undegraded":"2024-04-16T21:11:45.568997+0000","last_fullsized":"2024-04-16T21:11:45.568997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:45.568923+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.568923+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:16:31.758184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":427,"num_read_kb":285,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.730052+0000","last_change":"2024-04-16T21:12:52.730052+0000","last_active":"2024-04-16T21:12:52.730052+0000","last_peered":"2024-04-16T21:12:52.730052+0000","last_clean":"2024-04-16T21:12:52.730052+0000","last_became_active":"2024-04-16T19:24:35.394572+0000","last_became_peered":"2024-04-16T19:24:35.394572+0000","last_unstale":"2024-04-16T21:12:52.730052+0000","last_undegraded":"2024-04-16T21:12:52.730052+0000","last_fullsized":"2024-04-16T21:12:52.730052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.729928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.729928+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.729928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:55.516694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.851893+0000","last_change":"2024-04-16T21:12:54.851929+0000","last_active":"2024-04-16T21:12:54.851893+0000","last_peered":"2024-04-16T21:12:54.851893+0000","last_clean":"2024-04-16T21:12:54.851893+0000","last_became_active":"2024-04-16T19:24:33.388414+0000","last_became_peered":"2024-04-16T19:24:33.388414+0000","last_unstale":"2024-04-16T21:12:54.851893+0000","last_undegraded":"2024-04-16T21:12:54.851893+0000","last_fullsized":"2024-04-16T21:12:54.851893+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.851809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.851809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.851809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:06.030348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"52'6","reported_seq":971,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.813430+0000","last_change":"2024-04-16T21:12:30.813430+0000","last_active":"2024-04-16T21:12:30.813430+0000","last_peered":"2024-04-16T21:12:30.813430+0000","last_clean":"2024-04-16T21:12:30.813430+0000","last_became_active":"2024-04-16T19:24:45.409718+0000","last_became_peered":"2024-04-16T19:24:45.409718+0000","last_unstale":"2024-04-16T21:12:30.813430+0000","last_undegraded":"2024-04-16T21:12:30.813430+0000","last_fullsized":"2024-04-16T21:12:30.813430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:30.813330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.813330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.897002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1143,"num_read_kb":1141,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"52'78","reported_seq":241,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.999135+0000","last_change":"2024-04-16T21:12:31.999135+0000","last_active":"2024-04-16T21:12:31.999135+0000","last_peered":"2024-04-16T21:12:31.999135+0000","last_clean":"2024-04-16T21:12:31.999135+0000","last_became_active":"2024-04-16T19:24:47.423601+0000","last_became_peered":"2024-04-16T19:24:47.423601+0000","last_unstale":"2024-04-16T21:12:31.999135+0000","last_undegraded":"2024-04-16T21:12:31.999135+0000","last_fullsized":"2024-04-16T21:12:31.999135+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:31.999043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.999043+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:40.493275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.929194+0000","last_change":"2024-04-16T21:12:21.929194+0000","last_active":"2024-04-16T21:12:21.929194+0000","last_peered":"2024-04-16T21:12:21.929194+0000","last_clean":"2024-04-16T21:12:21.929194+0000","last_became_active":"2024-04-16T19:24:49.429259+0000","last_became_peered":"2024-04-16T19:24:49.429259+0000","last_unstale":"2024-04-16T21:12:21.929194+0000","last_undegraded":"2024-04-16T21:12:21.929194+0000","last_fullsized":"2024-04-16T21:12:21.929194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.929105+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.929105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.364555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.721391+0000","last_change":"2024-04-16T21:12:18.721414+0000","last_active":"2024-04-16T21:12:18.721391+0000","last_peered":"2024-04-16T21:12:18.721391+0000","last_clean":"2024-04-16T21:12:18.721391+0000","last_became_active":"2024-04-16T19:24:51.436405+0000","last_became_peered":"2024-04-16T19:24:51.436405+0000","last_unstale":"2024-04-16T21:12:18.721391+0000","last_undegraded":"2024-04-16T21:12:18.721391+0000","last_fullsized":"2024-04-16T21:12:18.721391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.721330+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.721330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:06.756619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:15.217315+0000","last_change":"2024-04-16T21:13:15.217356+0000","last_active":"2024-04-16T21:13:15.217315+0000","last_peered":"2024-04-16T21:13:15.217315+0000","last_clean":"2024-04-16T21:13:15.217315+0000","last_became_active":"2024-04-16T19:47:33.987067+0000","last_became_peered":"2024-04-16T19:47:33.987067+0000","last_unstale":"2024-04-16T21:13:15.217315+0000","last_undegraded":"2024-04-16T21:13:15.217315+0000","last_fullsized":"2024-04-16T21:13:15.217315+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:15.217241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:15.217241+0000","last_clean_scrub_stamp":"2024-04-16T21:13:15.217241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:03:57.168052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"52'135","reported_seq":1431,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.902487+0000","last_change":"2024-04-16T21:12:20.902537+0000","last_active":"2024-04-16T21:12:20.902487+0000","last_peered":"2024-04-16T21:12:20.902487+0000","last_clean":"2024-04-16T21:12:20.902487+0000","last_became_active":"2024-04-16T19:26:41.559877+0000","last_became_peered":"2024-04-16T19:26:41.559877+0000","last_unstale":"2024-04-16T21:12:20.902487+0000","last_undegraded":"2024-04-16T21:12:20.902487+0000","last_fullsized":"2024-04-16T21:12:20.902487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'135","last_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_deep_scrub":"52'135","last_deep_scrub_stamp":"2024-04-16T21:12:20.902428+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.902428+0000","objects_scrubbed":42,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:27:02.685242+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2016,"num_read_kb":1983,"num_write":116,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3879,"num_omap_keys":15,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.456127+0000","last_change":"2024-04-16T21:12:14.456172+0000","last_active":"2024-04-16T21:12:14.456127+0000","last_peered":"2024-04-16T21:12:14.456127+0000","last_clean":"2024-04-16T21:12:14.456127+0000","last_became_active":"2024-04-16T19:25:41.508663+0000","last_became_peered":"2024-04-16T19:25:41.508663+0000","last_unstale":"2024-04-16T21:12:14.456127+0000","last_undegraded":"2024-04-16T21:12:14.456127+0000","last_fullsized":"2024-04-16T21:12:14.456127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.456050+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.456050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:43.097345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"52'24","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.378145+0000","last_change":"2024-04-16T21:12:23.378174+0000","last_active":"2024-04-16T21:12:23.378145+0000","last_peered":"2024-04-16T21:12:23.378145+0000","last_clean":"2024-04-16T21:12:23.378145+0000","last_became_active":"2024-04-16T19:25:15.481722+0000","last_became_peered":"2024-04-16T19:25:15.481722+0000","last_unstale":"2024-04-16T21:12:23.378145+0000","last_undegraded":"2024-04-16T21:12:23.378145+0000","last_fullsized":"2024-04-16T21:12:23.378145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:23.378087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.378087+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:34.980757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1707,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":81,"num_write":66,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.647510+0000","last_change":"2024-04-16T21:11:51.647550+0000","last_active":"2024-04-16T21:11:51.647510+0000","last_peered":"2024-04-16T21:11:51.647510+0000","last_clean":"2024-04-16T21:11:51.647510+0000","last_became_active":"2024-04-16T19:25:13.475105+0000","last_became_peered":"2024-04-16T19:25:13.475105+0000","last_unstale":"2024-04-16T21:11:51.647510+0000","last_undegraded":"2024-04-16T21:11:51.647510+0000","last_fullsized":"2024-04-16T21:11:51.647510+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:51.647441+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.647441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:25.536915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"52'2068","reported_seq":5847,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.355619+0000","last_change":"2024-04-16T21:11:58.355713+0000","last_active":"2024-04-16T21:11:58.355619+0000","last_peered":"2024-04-16T21:11:58.355619+0000","last_clean":"2024-04-16T21:11:58.355619+0000","last_became_active":"2024-04-16T19:25:11.468501+0000","last_became_peered":"2024-04-16T19:25:11.468501+0000","last_unstale":"2024-04-16T21:11:58.355619+0000","last_undegraded":"2024-04-16T21:11:58.355619+0000","last_fullsized":"2024-04-16T21:11:58.355619+0000","mapping_epoch":33,"log_start":"47'400","ondisk_log_start":"47'400","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2068","last_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_deep_scrub":"52'2068","last_deep_scrub_stamp":"2024-04-16T21:11:58.355545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.355545+0000","objects_scrubbed":24,"log_size":1668,"log_dups_size":400,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:46.390834+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":4888,"num_read_kb":4879,"num_write":1747,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16267,"num_omap_keys":93,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"52'26","reported_seq":344,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.715965+0000","last_change":"2024-04-16T21:11:55.716041+0000","last_active":"2024-04-16T21:11:55.715965+0000","last_peered":"2024-04-16T21:11:55.715965+0000","last_clean":"2024-04-16T21:11:55.715965+0000","last_became_active":"2024-04-16T19:25:09.467242+0000","last_became_peered":"2024-04-16T19:25:09.467242+0000","last_unstale":"2024-04-16T21:11:55.715965+0000","last_undegraded":"2024-04-16T21:11:55.715965+0000","last_fullsized":"2024-04-16T21:11:55.715965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:55.715893+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.715893+0000","objects_scrubbed":12,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:15.543756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1129,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":531,"num_read_kb":399,"num_write":149,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.855492+0000","last_change":"2024-04-16T21:12:57.855492+0000","last_active":"2024-04-16T21:12:57.855492+0000","last_peered":"2024-04-16T21:12:57.855492+0000","last_clean":"2024-04-16T21:12:57.855492+0000","last_became_active":"2024-04-16T19:24:35.398348+0000","last_became_peered":"2024-04-16T19:24:35.398348+0000","last_unstale":"2024-04-16T21:12:57.855492+0000","last_undegraded":"2024-04-16T21:12:57.855492+0000","last_fullsized":"2024-04-16T21:12:57.855492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.855408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.855408+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.855408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:47.733563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.270377+0000","last_change":"2024-04-16T21:12:49.270399+0000","last_active":"2024-04-16T21:12:49.270377+0000","last_peered":"2024-04-16T21:12:49.270377+0000","last_clean":"2024-04-16T21:12:49.270377+0000","last_became_active":"2024-04-16T19:24:33.388927+0000","last_became_peered":"2024-04-16T19:24:33.388927+0000","last_unstale":"2024-04-16T21:12:49.270377+0000","last_undegraded":"2024-04-16T21:12:49.270377+0000","last_fullsized":"2024-04-16T21:12:49.270377+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.270275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.270275+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.270275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"52'2","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.812623+0000","last_change":"2024-04-16T21:12:32.812623+0000","last_active":"2024-04-16T21:12:32.812623+0000","last_peered":"2024-04-16T21:12:32.812623+0000","last_clean":"2024-04-16T21:12:32.812623+0000","last_became_active":"2024-04-16T19:24:45.409939+0000","last_became_peered":"2024-04-16T19:24:45.409939+0000","last_unstale":"2024-04-16T21:12:32.812623+0000","last_undegraded":"2024-04-16T21:12:32.812623+0000","last_fullsized":"2024-04-16T21:12:32.812623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:32.812531+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.812531+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:48.281904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"52'147","reported_seq":345,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334588+0000","last_change":"2024-04-16T21:12:30.334588+0000","last_active":"2024-04-16T21:12:30.334588+0000","last_peered":"2024-04-16T21:12:30.334588+0000","last_clean":"2024-04-16T21:12:30.334588+0000","last_became_active":"2024-04-16T19:24:47.421757+0000","last_became_peered":"2024-04-16T19:24:47.421757+0000","last_unstale":"2024-04-16T21:12:30.334588+0000","last_undegraded":"2024-04-16T21:12:30.334588+0000","last_fullsized":"2024-04-16T21:12:30.334588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:30.334447+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334447+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:36.782836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"52'13","reported_seq":1324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.286066+0000","last_change":"2024-04-16T21:12:31.286066+0000","last_active":"2024-04-16T21:12:31.286066+0000","last_peered":"2024-04-16T21:12:31.286066+0000","last_clean":"2024-04-16T21:12:31.286066+0000","last_became_active":"2024-04-16T19:24:49.427541+0000","last_became_peered":"2024-04-16T19:24:49.427541+0000","last_unstale":"2024-04-16T21:12:31.286066+0000","last_undegraded":"2024-04-16T21:12:31.286066+0000","last_fullsized":"2024-04-16T21:12:31.286066+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:31.285973+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.285973+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:54.848235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.406028+0000","last_change":"2024-04-16T21:12:07.406028+0000","last_active":"2024-04-16T21:12:07.406028+0000","last_peered":"2024-04-16T21:12:07.406028+0000","last_clean":"2024-04-16T21:12:07.406028+0000","last_became_active":"2024-04-16T19:24:51.437684+0000","last_became_peered":"2024-04-16T19:24:51.437684+0000","last_unstale":"2024-04-16T21:12:07.406028+0000","last_undegraded":"2024-04-16T21:12:07.406028+0000","last_fullsized":"2024-04-16T21:12:07.406028+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.405900+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.405900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:12.444844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.748485+0000","last_change":"2024-04-16T21:12:54.748509+0000","last_active":"2024-04-16T21:12:54.748485+0000","last_peered":"2024-04-16T21:12:54.748485+0000","last_clean":"2024-04-16T21:12:54.748485+0000","last_became_active":"2024-04-16T19:47:33.987897+0000","last_became_peered":"2024-04-16T19:47:33.987897+0000","last_unstale":"2024-04-16T21:12:54.748485+0000","last_undegraded":"2024-04-16T21:12:54.748485+0000","last_fullsized":"2024-04-16T21:12:54.748485+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.748417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.748417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.748417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:51.723857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"52'182","reported_seq":1946,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.937380+0000","last_change":"2024-04-16T21:12:22.937438+0000","last_active":"2024-04-16T21:12:22.937380+0000","last_peered":"2024-04-16T21:12:22.937380+0000","last_clean":"2024-04-16T21:12:22.937380+0000","last_became_active":"2024-04-16T19:25:49.522832+0000","last_became_peered":"2024-04-16T19:25:49.522832+0000","last_unstale":"2024-04-16T21:12:22.937380+0000","last_undegraded":"2024-04-16T21:12:22.937380+0000","last_fullsized":"2024-04-16T21:12:22.937380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'182","last_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_deep_scrub":"52'182","last_deep_scrub_stamp":"2024-04-16T21:12:22.937321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.937321+0000","objects_scrubbed":46,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:18.195145+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2891,"num_read_kb":2841,"num_write":193,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8599,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.724011+0000","last_change":"2024-04-16T21:12:29.724048+0000","last_active":"2024-04-16T21:12:29.724011+0000","last_peered":"2024-04-16T21:12:29.724011+0000","last_clean":"2024-04-16T21:12:29.724011+0000","last_became_active":"2024-04-16T19:25:41.507408+0000","last_became_peered":"2024-04-16T19:25:41.507408+0000","last_unstale":"2024-04-16T21:12:29.724011+0000","last_undegraded":"2024-04-16T21:12:29.724011+0000","last_fullsized":"2024-04-16T21:12:29.724011+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.723920+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.723920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:23.912595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"52'17","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.440078+0000","last_change":"2024-04-16T21:12:10.440115+0000","last_active":"2024-04-16T21:12:10.440078+0000","last_peered":"2024-04-16T21:12:10.440078+0000","last_clean":"2024-04-16T21:12:10.440078+0000","last_became_active":"2024-04-16T19:25:15.480318+0000","last_became_peered":"2024-04-16T19:25:15.480318+0000","last_unstale":"2024-04-16T21:12:10.440078+0000","last_undegraded":"2024-04-16T21:12:10.440078+0000","last_fullsized":"2024-04-16T21:12:10.440078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:10.440017+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.440017+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:44.207343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1665,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.910624+0000","last_change":"2024-04-16T21:12:13.910647+0000","last_active":"2024-04-16T21:12:13.910624+0000","last_peered":"2024-04-16T21:12:13.910624+0000","last_clean":"2024-04-16T21:12:13.910624+0000","last_became_active":"2024-04-16T19:25:13.475788+0000","last_became_peered":"2024-04-16T19:25:13.475788+0000","last_unstale":"2024-04-16T21:12:13.910624+0000","last_undegraded":"2024-04-16T21:12:13.910624+0000","last_fullsized":"2024-04-16T21:12:13.910624+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.910562+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.910562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:19.649702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"52'362","reported_seq":695,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.975586+0000","last_change":"2024-04-16T21:11:44.975641+0000","last_active":"2024-04-16T21:11:44.975586+0000","last_peered":"2024-04-16T21:11:44.975586+0000","last_clean":"2024-04-16T21:11:44.975586+0000","last_became_active":"2024-04-16T19:25:11.467011+0000","last_became_peered":"2024-04-16T19:25:11.467011+0000","last_unstale":"2024-04-16T21:11:44.975586+0000","last_undegraded":"2024-04-16T21:11:44.975586+0000","last_fullsized":"2024-04-16T21:11:44.975586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'362","last_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_deep_scrub":"52'362","last_deep_scrub_stamp":"2024-04-16T21:11:44.975535+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.975535+0000","objects_scrubbed":21,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:50:56.598547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":685,"num_read_kb":535,"num_write":358,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2185,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"52'18","reported_seq":296,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.687122+0000","last_change":"2024-04-16T21:11:48.687164+0000","last_active":"2024-04-16T21:11:48.687122+0000","last_peered":"2024-04-16T21:11:48.687122+0000","last_clean":"2024-04-16T21:11:48.687122+0000","last_became_active":"2024-04-16T19:25:09.470186+0000","last_became_peered":"2024-04-16T19:25:09.470186+0000","last_unstale":"2024-04-16T21:11:48.687122+0000","last_undegraded":"2024-04-16T21:11:48.687122+0000","last_fullsized":"2024-04-16T21:11:48.687122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:48.687069+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.687069+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:39:52.625067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":445,"num_read_kb":284,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.473384+0000","last_change":"2024-04-16T21:12:54.473405+0000","last_active":"2024-04-16T21:12:54.473384+0000","last_peered":"2024-04-16T21:12:54.473384+0000","last_clean":"2024-04-16T21:12:54.473384+0000","last_became_active":"2024-04-16T19:24:35.400076+0000","last_became_peered":"2024-04-16T19:24:35.400076+0000","last_unstale":"2024-04-16T21:12:54.473384+0000","last_undegraded":"2024-04-16T21:12:54.473384+0000","last_fullsized":"2024-04-16T21:12:54.473384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.473317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.473317+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.473317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:08.143470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.853921+0000","last_change":"2024-04-16T21:12:48.853954+0000","last_active":"2024-04-16T21:12:48.853921+0000","last_peered":"2024-04-16T21:12:48.853921+0000","last_clean":"2024-04-16T21:12:48.853921+0000","last_became_active":"2024-04-16T19:24:33.390479+0000","last_became_peered":"2024-04-16T19:24:33.390479+0000","last_unstale":"2024-04-16T21:12:48.853921+0000","last_undegraded":"2024-04-16T21:12:48.853921+0000","last_fullsized":"2024-04-16T21:12:48.853921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.853857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.853857+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.853857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:20.558650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"52'19","reported_seq":470,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.913976+0000","last_change":"2024-04-16T21:12:26.913976+0000","last_active":"2024-04-16T21:12:26.913976+0000","last_peered":"2024-04-16T21:12:26.913976+0000","last_clean":"2024-04-16T21:12:26.913976+0000","last_became_active":"2024-04-16T19:24:45.408098+0000","last_became_peered":"2024-04-16T19:24:45.408098+0000","last_unstale":"2024-04-16T21:12:26.913976+0000","last_undegraded":"2024-04-16T21:12:26.913976+0000","last_fullsized":"2024-04-16T21:12:26.913976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:26.913887+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.913887+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.708618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2576,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":642,"num_write":26,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"52'87","reported_seq":261,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.302869+0000","last_change":"2024-04-16T21:12:35.302869+0000","last_active":"2024-04-16T21:12:35.302869+0000","last_peered":"2024-04-16T21:12:35.302869+0000","last_clean":"2024-04-16T21:12:35.302869+0000","last_became_active":"2024-04-16T19:24:47.423066+0000","last_became_peered":"2024-04-16T19:24:47.423066+0000","last_unstale":"2024-04-16T21:12:35.302869+0000","last_undegraded":"2024-04-16T21:12:35.302869+0000","last_fullsized":"2024-04-16T21:12:35.302869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:35.302758+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.302758+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:01.984411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.899147+0000","last_change":"2024-04-16T21:12:18.899147+0000","last_active":"2024-04-16T21:12:18.899147+0000","last_peered":"2024-04-16T21:12:18.899147+0000","last_clean":"2024-04-16T21:12:18.899147+0000","last_became_active":"2024-04-16T19:24:49.430503+0000","last_became_peered":"2024-04-16T19:24:49.430503+0000","last_unstale":"2024-04-16T21:12:18.899147+0000","last_undegraded":"2024-04-16T21:12:18.899147+0000","last_fullsized":"2024-04-16T21:12:18.899147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.899064+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.899064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:50:17.822519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.847650+0000","last_change":"2024-04-16T21:12:03.847650+0000","last_active":"2024-04-16T21:12:03.847650+0000","last_peered":"2024-04-16T21:12:03.847650+0000","last_clean":"2024-04-16T21:12:03.847650+0000","last_became_active":"2024-04-16T19:24:51.442242+0000","last_became_peered":"2024-04-16T19:24:51.442242+0000","last_unstale":"2024-04-16T21:12:03.847650+0000","last_undegraded":"2024-04-16T21:12:03.847650+0000","last_fullsized":"2024-04-16T21:12:03.847650+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.847563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.847563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:36.440589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:08.011944+0000","last_change":"2024-04-16T21:13:08.011967+0000","last_active":"2024-04-16T21:13:08.011944+0000","last_peered":"2024-04-16T21:13:08.011944+0000","last_clean":"2024-04-16T21:13:08.011944+0000","last_became_active":"2024-04-16T19:47:33.989408+0000","last_became_peered":"2024-04-16T19:47:33.989408+0000","last_unstale":"2024-04-16T21:13:08.011944+0000","last_undegraded":"2024-04-16T21:13:08.011944+0000","last_fullsized":"2024-04-16T21:13:08.011944+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:08.011877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:08.011877+0000","last_clean_scrub_stamp":"2024-04-16T21:13:08.011877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:21:06.427246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"52'186","reported_seq":2409,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.401091+0000","last_change":"2024-04-16T21:12:42.401122+0000","last_active":"2024-04-16T21:12:42.401091+0000","last_peered":"2024-04-16T21:12:42.401091+0000","last_clean":"2024-04-16T21:12:42.401091+0000","last_became_active":"2024-04-16T19:25:49.520490+0000","last_became_peered":"2024-04-16T19:25:49.520490+0000","last_unstale":"2024-04-16T21:12:42.401091+0000","last_undegraded":"2024-04-16T21:12:42.401091+0000","last_fullsized":"2024-04-16T21:12:42.401091+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'186","last_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_deep_scrub":"52'186","last_deep_scrub_stamp":"2024-04-16T21:12:42.401026+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.401026+0000","objects_scrubbed":52,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:51.749803+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":3703,"num_read_kb":3652,"num_write":193,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12956,"num_omap_keys":52,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.650119+0000","last_change":"2024-04-16T21:12:13.650119+0000","last_active":"2024-04-16T21:12:13.650119+0000","last_peered":"2024-04-16T21:12:13.650119+0000","last_clean":"2024-04-16T21:12:13.650119+0000","last_became_active":"2024-04-16T19:25:41.508491+0000","last_became_peered":"2024-04-16T19:25:41.508491+0000","last_unstale":"2024-04-16T21:12:13.650119+0000","last_undegraded":"2024-04-16T21:12:13.650119+0000","last_fullsized":"2024-04-16T21:12:13.650119+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.650036+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.650036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:16.977821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"52'13","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.901211+0000","last_change":"2024-04-16T21:12:20.901247+0000","last_active":"2024-04-16T21:12:20.901211+0000","last_peered":"2024-04-16T21:12:20.901211+0000","last_clean":"2024-04-16T21:12:20.901211+0000","last_became_active":"2024-04-16T19:25:15.486436+0000","last_became_peered":"2024-04-16T19:25:15.486436+0000","last_unstale":"2024-04-16T21:12:20.901211+0000","last_undegraded":"2024-04-16T21:12:20.901211+0000","last_fullsized":"2024-04-16T21:12:20.901211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:20.901143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.901143+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:54.668469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1626,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":135,"num_read_kb":103,"num_write":26,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"52'706","reported_seq":2075,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.405465+0000","last_change":"2024-04-16T21:12:08.405492+0000","last_active":"2024-04-16T21:12:08.405465+0000","last_peered":"2024-04-16T21:12:08.405465+0000","last_clean":"2024-04-16T21:12:08.405465+0000","last_became_active":"2024-04-16T19:25:13.472855+0000","last_became_peered":"2024-04-16T19:25:13.472855+0000","last_unstale":"2024-04-16T21:12:08.405465+0000","last_undegraded":"2024-04-16T21:12:08.405465+0000","last_fullsized":"2024-04-16T21:12:08.405465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:08.405399+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.405399+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:20.200130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"52'292","reported_seq":573,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.642477+0000","last_change":"2024-04-16T21:11:50.642568+0000","last_active":"2024-04-16T21:11:50.642477+0000","last_peered":"2024-04-16T21:11:50.642477+0000","last_clean":"2024-04-16T21:11:50.642477+0000","last_became_active":"2024-04-16T19:25:11.465171+0000","last_became_peered":"2024-04-16T19:25:11.465171+0000","last_unstale":"2024-04-16T21:11:50.642477+0000","last_undegraded":"2024-04-16T21:11:50.642477+0000","last_fullsized":"2024-04-16T21:11:50.642477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'292","last_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_deep_scrub":"52'292","last_deep_scrub_stamp":"2024-04-16T21:11:50.642400+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.642400+0000","objects_scrubbed":23,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:29.527145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":443,"num_read_kb":437,"num_write":387,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4142,"num_omap_keys":19,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"52'10","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.314574+0000","last_change":"2024-04-16T21:11:57.314619+0000","last_active":"2024-04-16T21:11:57.314574+0000","last_peered":"2024-04-16T21:11:57.314574+0000","last_clean":"2024-04-16T21:11:57.314574+0000","last_became_active":"2024-04-16T19:25:09.458002+0000","last_became_peered":"2024-04-16T19:25:09.458002+0000","last_unstale":"2024-04-16T21:11:57.314574+0000","last_undegraded":"2024-04-16T21:11:57.314574+0000","last_fullsized":"2024-04-16T21:11:57.314574+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:57.314511+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.314511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:49.494822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":279,"num_read_kb":187,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.778832+0000","last_change":"2024-04-16T21:12:54.778854+0000","last_active":"2024-04-16T21:12:54.778832+0000","last_peered":"2024-04-16T21:12:54.778832+0000","last_clean":"2024-04-16T21:12:54.778832+0000","last_became_active":"2024-04-16T19:24:35.397182+0000","last_became_peered":"2024-04-16T19:24:35.397182+0000","last_unstale":"2024-04-16T21:12:54.778832+0000","last_undegraded":"2024-04-16T21:12:54.778832+0000","last_fullsized":"2024-04-16T21:12:54.778832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.778773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.778773+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.778773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:37.834506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":143,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.898133+0000","last_change":"2024-04-16T21:12:08.898133+0000","last_active":"2024-04-16T21:12:08.898133+0000","last_peered":"2024-04-16T21:12:08.898133+0000","last_clean":"2024-04-16T21:12:08.898133+0000","last_became_active":"2024-04-16T19:25:41.503583+0000","last_became_peered":"2024-04-16T19:25:41.503583+0000","last_unstale":"2024-04-16T21:12:08.898133+0000","last_undegraded":"2024-04-16T21:12:08.898133+0000","last_fullsized":"2024-04-16T21:12:08.898133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.898047+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.898047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:36.956196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"52'2","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.800302+0000","last_change":"2024-04-16T21:12:36.800302+0000","last_active":"2024-04-16T21:12:36.800302+0000","last_peered":"2024-04-16T21:12:36.800302+0000","last_clean":"2024-04-16T21:12:36.800302+0000","last_became_active":"2024-04-16T19:24:45.410437+0000","last_became_peered":"2024-04-16T19:24:45.410437+0000","last_unstale":"2024-04-16T21:12:36.800302+0000","last_undegraded":"2024-04-16T21:12:36.800302+0000","last_fullsized":"2024-04-16T21:12:36.800302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'2","last_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_deep_scrub":"52'2","last_deep_scrub_stamp":"2024-04-16T21:12:36.800213+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.800213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:46.361882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"52'101","reported_seq":275,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.698919+0000","last_change":"2024-04-16T21:12:30.698919+0000","last_active":"2024-04-16T21:12:30.698919+0000","last_peered":"2024-04-16T21:12:30.698919+0000","last_clean":"2024-04-16T21:12:30.698919+0000","last_became_active":"2024-04-16T19:24:47.421222+0000","last_became_peered":"2024-04-16T19:24:47.421222+0000","last_unstale":"2024-04-16T21:12:30.698919+0000","last_undegraded":"2024-04-16T21:12:30.698919+0000","last_fullsized":"2024-04-16T21:12:30.698919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:30.698825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.698825+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:33.782497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.333063+0000","last_change":"2024-04-16T21:12:32.333063+0000","last_active":"2024-04-16T21:12:32.333063+0000","last_peered":"2024-04-16T21:12:32.333063+0000","last_clean":"2024-04-16T21:12:32.333063+0000","last_became_active":"2024-04-16T19:24:49.426831+0000","last_became_peered":"2024-04-16T19:24:49.426831+0000","last_unstale":"2024-04-16T21:12:32.333063+0000","last_undegraded":"2024-04-16T21:12:32.333063+0000","last_fullsized":"2024-04-16T21:12:32.333063+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:32.332967+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.332967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:32.295174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.889407+0000","last_change":"2024-04-16T21:12:17.889407+0000","last_active":"2024-04-16T21:12:17.889407+0000","last_peered":"2024-04-16T21:12:17.889407+0000","last_clean":"2024-04-16T21:12:17.889407+0000","last_became_active":"2024-04-16T19:24:51.434725+0000","last_became_peered":"2024-04-16T19:24:51.434725+0000","last_unstale":"2024-04-16T21:12:17.889407+0000","last_undegraded":"2024-04-16T21:12:17.889407+0000","last_fullsized":"2024-04-16T21:12:17.889407+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.889318+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.889318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:23.628459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:12.330666+0000","last_change":"2024-04-16T21:13:12.330704+0000","last_active":"2024-04-16T21:13:12.330666+0000","last_peered":"2024-04-16T21:13:12.330666+0000","last_clean":"2024-04-16T21:13:12.330666+0000","last_became_active":"2024-04-16T19:47:33.979364+0000","last_became_peered":"2024-04-16T19:47:33.979364+0000","last_unstale":"2024-04-16T21:13:12.330666+0000","last_undegraded":"2024-04-16T21:13:12.330666+0000","last_fullsized":"2024-04-16T21:13:12.330666+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:12.330576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:12.330576+0000","last_clean_scrub_stamp":"2024-04-16T21:13:12.330576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:31:58.794833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"52'172","reported_seq":1727,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.828813+0000","last_change":"2024-04-16T21:12:45.828865+0000","last_active":"2024-04-16T21:12:45.828813+0000","last_peered":"2024-04-16T21:12:45.828813+0000","last_clean":"2024-04-16T21:12:45.828813+0000","last_became_active":"2024-04-16T19:25:49.517830+0000","last_became_peered":"2024-04-16T19:25:49.517830+0000","last_unstale":"2024-04-16T21:12:45.828813+0000","last_undegraded":"2024-04-16T21:12:45.828813+0000","last_fullsized":"2024-04-16T21:12:45.828813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:45.828749+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:45.828749+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.828749+0000","objects_scrubbed":38,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:49:27.527910+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2660,"num_read_kb":2611,"num_write":196,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12008,"num_omap_keys":49,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.854832+0000","last_change":"2024-04-16T21:12:21.854876+0000","last_active":"2024-04-16T21:12:21.854832+0000","last_peered":"2024-04-16T21:12:21.854832+0000","last_clean":"2024-04-16T21:12:21.854832+0000","last_became_active":"2024-04-16T19:25:41.504026+0000","last_became_peered":"2024-04-16T19:25:41.504026+0000","last_unstale":"2024-04-16T21:12:21.854832+0000","last_undegraded":"2024-04-16T21:12:21.854832+0000","last_fullsized":"2024-04-16T21:12:21.854832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.854759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.854759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:37.324132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"52'138","reported_seq":310,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.362284+0000","last_change":"2024-04-16T21:12:22.362331+0000","last_active":"2024-04-16T21:12:22.362284+0000","last_peered":"2024-04-16T21:12:22.362284+0000","last_clean":"2024-04-16T21:12:22.362284+0000","last_became_active":"2024-04-16T19:25:15.482952+0000","last_became_peered":"2024-04-16T19:25:15.482952+0000","last_unstale":"2024-04-16T21:12:22.362284+0000","last_undegraded":"2024-04-16T21:12:22.362284+0000","last_fullsized":"2024-04-16T21:12:22.362284+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:22.362197+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.362197+0000","objects_scrubbed":2,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:16.550872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":341,"num_read_kb":612,"num_write":271,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9707,"num_omap_keys":56,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.836583+0000","last_change":"2024-04-16T21:11:52.836606+0000","last_active":"2024-04-16T21:11:52.836583+0000","last_peered":"2024-04-16T21:11:52.836583+0000","last_clean":"2024-04-16T21:11:52.836583+0000","last_became_active":"2024-04-16T19:25:13.471494+0000","last_became_peered":"2024-04-16T19:25:13.471494+0000","last_unstale":"2024-04-16T21:11:52.836583+0000","last_undegraded":"2024-04-16T21:11:52.836583+0000","last_fullsized":"2024-04-16T21:11:52.836583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:52.836519+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.836519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:26.725994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"52'371","reported_seq":758,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.959985+0000","last_change":"2024-04-16T21:11:46.960080+0000","last_active":"2024-04-16T21:11:46.959985+0000","last_peered":"2024-04-16T21:11:46.959985+0000","last_clean":"2024-04-16T21:11:46.959985+0000","last_became_active":"2024-04-16T19:25:11.466825+0000","last_became_peered":"2024-04-16T19:25:11.466825+0000","last_unstale":"2024-04-16T21:11:46.959985+0000","last_undegraded":"2024-04-16T21:11:46.959985+0000","last_fullsized":"2024-04-16T21:11:46.959985+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'371","last_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_deep_scrub":"52'371","last_deep_scrub_stamp":"2024-04-16T21:11:46.959920+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.959920+0000","objects_scrubbed":24,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:05:28.129840+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":569,"num_read_kb":555,"num_write":384,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2169,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"52'19","reported_seq":216,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.614888+0000","last_change":"2024-04-16T21:11:53.614931+0000","last_active":"2024-04-16T21:11:53.614888+0000","last_peered":"2024-04-16T21:11:53.614888+0000","last_clean":"2024-04-16T21:11:53.614888+0000","last_became_active":"2024-04-16T19:25:09.461204+0000","last_became_peered":"2024-04-16T19:25:09.461204+0000","last_unstale":"2024-04-16T21:11:53.614888+0000","last_undegraded":"2024-04-16T21:11:53.614888+0000","last_fullsized":"2024-04-16T21:11:53.614888+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:53.614826+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.614826+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:17:49.084093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":210,"num_read_kb":144,"num_write":88,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.274919+0000","last_change":"2024-04-16T21:12:58.274953+0000","last_active":"2024-04-16T21:12:58.274919+0000","last_peered":"2024-04-16T21:12:58.274919+0000","last_clean":"2024-04-16T21:12:58.274919+0000","last_became_active":"2024-04-16T19:24:35.393077+0000","last_became_peered":"2024-04-16T19:24:35.393077+0000","last_unstale":"2024-04-16T21:12:58.274919+0000","last_undegraded":"2024-04-16T21:12:58.274919+0000","last_fullsized":"2024-04-16T21:12:58.274919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.274858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.274858+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.274858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:37:15.668157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.827874+0000","last_change":"2024-04-16T21:12:04.827874+0000","last_active":"2024-04-16T21:12:04.827874+0000","last_peered":"2024-04-16T21:12:04.827874+0000","last_clean":"2024-04-16T21:12:04.827874+0000","last_became_active":"2024-04-16T19:25:41.505652+0000","last_became_peered":"2024-04-16T19:25:41.505652+0000","last_unstale":"2024-04-16T21:12:04.827874+0000","last_undegraded":"2024-04-16T21:12:04.827874+0000","last_fullsized":"2024-04-16T21:12:04.827874+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.827767+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.827767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:23.692877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"52'20","reported_seq":178,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.142643+0000","last_change":"2024-04-16T21:12:47.142643+0000","last_active":"2024-04-16T21:12:47.142643+0000","last_peered":"2024-04-16T21:12:47.142643+0000","last_clean":"2024-04-16T21:12:47.142643+0000","last_became_active":"2024-04-16T19:24:45.408515+0000","last_became_peered":"2024-04-16T19:24:45.408515+0000","last_unstale":"2024-04-16T21:12:47.142643+0000","last_undegraded":"2024-04-16T21:12:47.142643+0000","last_fullsized":"2024-04-16T21:12:47.142643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:47.142546+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:47.142546+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.142546+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:47.104753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1320,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":88,"num_write":39,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"52'147","reported_seq":343,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.899755+0000","last_change":"2024-04-16T21:12:22.899755+0000","last_active":"2024-04-16T21:12:22.899755+0000","last_peered":"2024-04-16T21:12:22.899755+0000","last_clean":"2024-04-16T21:12:22.899755+0000","last_became_active":"2024-04-16T19:24:47.426110+0000","last_became_peered":"2024-04-16T19:24:47.426110+0000","last_unstale":"2024-04-16T21:12:22.899755+0000","last_undegraded":"2024-04-16T21:12:22.899755+0000","last_fullsized":"2024-04-16T21:12:22.899755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:22.899611+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.899611+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:49.019687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"52'13","reported_seq":1322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.856493+0000","last_change":"2024-04-16T21:12:19.856493+0000","last_active":"2024-04-16T21:12:19.856493+0000","last_peered":"2024-04-16T21:12:19.856493+0000","last_clean":"2024-04-16T21:12:19.856493+0000","last_became_active":"2024-04-16T19:24:49.430364+0000","last_became_peered":"2024-04-16T19:24:49.430364+0000","last_unstale":"2024-04-16T21:12:19.856493+0000","last_undegraded":"2024-04-16T21:12:19.856493+0000","last_fullsized":"2024-04-16T21:12:19.856493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:19.856405+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.856405+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:56.872492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.913967+0000","last_change":"2024-04-16T21:12:16.913967+0000","last_active":"2024-04-16T21:12:16.913967+0000","last_peered":"2024-04-16T21:12:16.913967+0000","last_clean":"2024-04-16T21:12:16.913967+0000","last_became_active":"2024-04-16T19:24:51.436355+0000","last_became_peered":"2024-04-16T19:24:51.436355+0000","last_unstale":"2024-04-16T21:12:16.913967+0000","last_undegraded":"2024-04-16T21:12:16.913967+0000","last_fullsized":"2024-04-16T21:12:16.913967+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.913879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.913879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:41:03.787910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:13.231103+0000","last_change":"2024-04-16T21:13:13.231103+0000","last_active":"2024-04-16T21:13:13.231103+0000","last_peered":"2024-04-16T21:13:13.231103+0000","last_clean":"2024-04-16T21:13:13.231103+0000","last_became_active":"2024-04-16T19:47:33.985591+0000","last_became_peered":"2024-04-16T19:47:33.985591+0000","last_unstale":"2024-04-16T21:13:13.231103+0000","last_undegraded":"2024-04-16T21:13:13.231103+0000","last_fullsized":"2024-04-16T21:13:13.231103+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:13.231014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:13.231014+0000","last_clean_scrub_stamp":"2024-04-16T21:13:13.231014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:50.243355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"52'166","reported_seq":1925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.247637+0000","last_change":"2024-04-16T21:12:40.247676+0000","last_active":"2024-04-16T21:12:40.247637+0000","last_peered":"2024-04-16T21:12:40.247637+0000","last_clean":"2024-04-16T21:12:40.247637+0000","last_became_active":"2024-04-16T19:25:49.519156+0000","last_became_peered":"2024-04-16T19:25:49.519156+0000","last_unstale":"2024-04-16T21:12:40.247637+0000","last_undegraded":"2024-04-16T21:12:40.247637+0000","last_fullsized":"2024-04-16T21:12:40.247637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'166","last_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_deep_scrub":"52'166","last_deep_scrub_stamp":"2024-04-16T21:12:40.247577+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.247577+0000","objects_scrubbed":44,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:25.005335+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2895,"num_read_kb":2851,"num_write":169,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9289,"num_omap_keys":38,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.368999+0000","last_change":"2024-04-16T21:12:24.369021+0000","last_active":"2024-04-16T21:12:24.368999+0000","last_peered":"2024-04-16T21:12:24.368999+0000","last_clean":"2024-04-16T21:12:24.368999+0000","last_became_active":"2024-04-16T19:25:41.507056+0000","last_became_peered":"2024-04-16T19:25:41.507056+0000","last_unstale":"2024-04-16T21:12:24.368999+0000","last_undegraded":"2024-04-16T21:12:24.368999+0000","last_fullsized":"2024-04-16T21:12:24.368999+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.368926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.368926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:12.501416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"52'12","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.879266+0000","last_change":"2024-04-16T21:12:15.879307+0000","last_active":"2024-04-16T21:12:15.879266+0000","last_peered":"2024-04-16T21:12:15.879266+0000","last_clean":"2024-04-16T21:12:15.879266+0000","last_became_active":"2024-04-16T19:25:15.477387+0000","last_became_peered":"2024-04-16T19:25:15.477387+0000","last_unstale":"2024-04-16T21:12:15.879266+0000","last_undegraded":"2024-04-16T21:12:15.879266+0000","last_fullsized":"2024-04-16T21:12:15.879266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:15.879198+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.879198+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:41.314648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":892,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":76,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":102,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.625545+0000","last_change":"2024-04-16T21:11:53.625568+0000","last_active":"2024-04-16T21:11:53.625545+0000","last_peered":"2024-04-16T21:11:53.625545+0000","last_clean":"2024-04-16T21:11:53.625545+0000","last_became_active":"2024-04-16T19:25:13.475758+0000","last_became_peered":"2024-04-16T19:25:13.475758+0000","last_unstale":"2024-04-16T21:11:53.625545+0000","last_undegraded":"2024-04-16T21:11:53.625545+0000","last_fullsized":"2024-04-16T21:11:53.625545+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.625484+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.625484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:18:35.360695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"52'485","reported_seq":911,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.722929+0000","last_change":"2024-04-16T21:11:47.722980+0000","last_active":"2024-04-16T21:11:47.722929+0000","last_peered":"2024-04-16T21:11:47.722929+0000","last_clean":"2024-04-16T21:11:47.722929+0000","last_became_active":"2024-04-16T19:25:11.462325+0000","last_became_peered":"2024-04-16T19:25:11.462325+0000","last_unstale":"2024-04-16T21:11:47.722929+0000","last_undegraded":"2024-04-16T21:11:47.722929+0000","last_fullsized":"2024-04-16T21:11:47.722929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'485","last_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_deep_scrub":"52'485","last_deep_scrub_stamp":"2024-04-16T21:11:47.722865+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.722865+0000","objects_scrubbed":32,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:42:39.504569+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":850,"num_read_kb":835,"num_write":696,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42883,"num_omap_keys":132,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"52'14","reported_seq":210,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.616163+0000","last_change":"2024-04-16T21:12:05.616245+0000","last_active":"2024-04-16T21:12:05.616163+0000","last_peered":"2024-04-16T21:12:05.616163+0000","last_clean":"2024-04-16T21:12:05.616163+0000","last_became_active":"2024-04-16T19:25:41.506965+0000","last_became_peered":"2024-04-16T19:25:41.506965+0000","last_unstale":"2024-04-16T21:12:05.616163+0000","last_undegraded":"2024-04-16T21:12:05.616163+0000","last_fullsized":"2024-04-16T21:12:05.616163+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:05.616057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.616057+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.466963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":274,"num_read_kb":184,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.688471+0000","last_change":"2024-04-16T21:12:33.688471+0000","last_active":"2024-04-16T21:12:33.688471+0000","last_peered":"2024-04-16T21:12:33.688471+0000","last_clean":"2024-04-16T21:12:33.688471+0000","last_became_active":"2024-04-16T19:24:35.401178+0000","last_became_peered":"2024-04-16T19:24:35.401178+0000","last_unstale":"2024-04-16T21:12:33.688471+0000","last_undegraded":"2024-04-16T21:12:33.688471+0000","last_fullsized":"2024-04-16T21:12:33.688471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.688381+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.688381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:51:41.315722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.702730+0000","last_change":"2024-04-16T21:12:50.702730+0000","last_active":"2024-04-16T21:12:50.702730+0000","last_peered":"2024-04-16T21:12:50.702730+0000","last_clean":"2024-04-16T21:12:50.702730+0000","last_became_active":"2024-04-16T19:24:33.384761+0000","last_became_peered":"2024-04-16T19:24:33.384761+0000","last_unstale":"2024-04-16T21:12:50.702730+0000","last_undegraded":"2024-04-16T21:12:50.702730+0000","last_fullsized":"2024-04-16T21:12:50.702730+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.702579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.702579+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.702579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:13.969329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"52'13","reported_seq":1374,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:11.738200+0000","last_change":"2024-04-16T21:13:11.738248+0000","last_active":"2024-04-16T21:13:11.738200+0000","last_peered":"2024-04-16T21:13:11.738200+0000","last_clean":"2024-04-16T21:13:11.738200+0000","last_became_active":"2024-04-16T19:24:45.411132+0000","last_became_peered":"2024-04-16T19:24:45.411132+0000","last_unstale":"2024-04-16T21:13:11.738200+0000","last_undegraded":"2024-04-16T21:13:11.738200+0000","last_fullsized":"2024-04-16T21:13:11.738200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:13:11.738119+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:13:11.738119+0000","last_clean_scrub_stamp":"2024-04-16T21:13:11.738119+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:59:29.639763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":42,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"52'115","reported_seq":291,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.013687+0000","last_change":"2024-04-16T21:12:29.013687+0000","last_active":"2024-04-16T21:12:29.013687+0000","last_peered":"2024-04-16T21:12:29.013687+0000","last_clean":"2024-04-16T21:12:29.013687+0000","last_became_active":"2024-04-16T19:24:47.424015+0000","last_became_peered":"2024-04-16T19:24:47.424015+0000","last_unstale":"2024-04-16T21:12:29.013687+0000","last_undegraded":"2024-04-16T21:12:29.013687+0000","last_fullsized":"2024-04-16T21:12:29.013687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:29.013556+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.013556+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:57.654850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"52'13","reported_seq":1322,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.607256+0000","last_change":"2024-04-16T21:12:22.607256+0000","last_active":"2024-04-16T21:12:22.607256+0000","last_peered":"2024-04-16T21:12:22.607256+0000","last_clean":"2024-04-16T21:12:22.607256+0000","last_became_active":"2024-04-16T19:24:49.428278+0000","last_became_peered":"2024-04-16T19:24:49.428278+0000","last_unstale":"2024-04-16T21:12:22.607256+0000","last_undegraded":"2024-04-16T21:12:22.607256+0000","last_fullsized":"2024-04-16T21:12:22.607256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:22.607175+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.607175+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:48.727251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.673922+0000","last_change":"2024-04-16T21:12:20.673922+0000","last_active":"2024-04-16T21:12:20.673922+0000","last_peered":"2024-04-16T21:12:20.673922+0000","last_clean":"2024-04-16T21:12:20.673922+0000","last_became_active":"2024-04-16T19:24:51.434841+0000","last_became_peered":"2024-04-16T19:24:51.434841+0000","last_unstale":"2024-04-16T21:12:20.673922+0000","last_undegraded":"2024-04-16T21:12:20.673922+0000","last_fullsized":"2024-04-16T21:12:20.673922+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.673807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.673807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:17.517736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":66,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:07.032767+0000","last_change":"2024-04-16T21:13:07.032796+0000","last_active":"2024-04-16T21:13:07.032767+0000","last_peered":"2024-04-16T21:13:07.032767+0000","last_clean":"2024-04-16T21:13:07.032767+0000","last_became_active":"2024-04-16T19:47:33.988097+0000","last_became_peered":"2024-04-16T19:47:33.988097+0000","last_unstale":"2024-04-16T21:13:07.032767+0000","last_undegraded":"2024-04-16T21:13:07.032767+0000","last_fullsized":"2024-04-16T21:13:07.032767+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:07.032714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:07.032714+0000","last_clean_scrub_stamp":"2024-04-16T21:13:07.032714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:10:19.567038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"52'162","reported_seq":2309,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.337358+0000","last_change":"2024-04-16T21:12:34.337466+0000","last_active":"2024-04-16T21:12:34.337358+0000","last_peered":"2024-04-16T21:12:34.337358+0000","last_clean":"2024-04-16T21:12:34.337358+0000","last_became_active":"2024-04-16T19:25:49.516968+0000","last_became_peered":"2024-04-16T19:25:49.516968+0000","last_unstale":"2024-04-16T21:12:34.337358+0000","last_undegraded":"2024-04-16T21:12:34.337358+0000","last_fullsized":"2024-04-16T21:12:34.337358+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'162","last_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_deep_scrub":"52'162","last_deep_scrub_stamp":"2024-04-16T21:12:34.337295+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.337295+0000","objects_scrubbed":41,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:21.592470+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":3690,"num_read_kb":3704,"num_write":177,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11917,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.346082+0000","last_change":"2024-04-16T21:12:20.346107+0000","last_active":"2024-04-16T21:12:20.346082+0000","last_peered":"2024-04-16T21:12:20.346082+0000","last_clean":"2024-04-16T21:12:20.346082+0000","last_became_active":"2024-04-16T19:25:41.505479+0000","last_became_peered":"2024-04-16T19:25:41.505479+0000","last_unstale":"2024-04-16T21:12:20.346082+0000","last_undegraded":"2024-04-16T21:12:20.346082+0000","last_fullsized":"2024-04-16T21:12:20.346082+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:20.346020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.346020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:28.840252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":96,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.905708+0000","last_change":"2024-04-16T21:12:13.905733+0000","last_active":"2024-04-16T21:12:13.905708+0000","last_peered":"2024-04-16T21:12:13.905708+0000","last_clean":"2024-04-16T21:12:13.905708+0000","last_became_active":"2024-04-16T19:25:15.479628+0000","last_became_peered":"2024-04-16T19:25:15.479628+0000","last_unstale":"2024-04-16T21:12:13.905708+0000","last_undegraded":"2024-04-16T21:12:13.905708+0000","last_fullsized":"2024-04-16T21:12:13.905708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.905640+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.905640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:17.215414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"52'706","reported_seq":2474,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.688857+0000","last_change":"2024-04-16T21:11:58.688891+0000","last_active":"2024-04-16T21:11:58.688857+0000","last_peered":"2024-04-16T21:11:58.688857+0000","last_clean":"2024-04-16T21:11:58.688857+0000","last_became_active":"2024-04-16T19:25:13.469973+0000","last_became_peered":"2024-04-16T19:25:13.469973+0000","last_unstale":"2024-04-16T21:11:58.688857+0000","last_undegraded":"2024-04-16T21:11:58.688857+0000","last_fullsized":"2024-04-16T21:11:58.688857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:58.688787+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.688787+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:58.866593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"52'332","reported_seq":603,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.669169+0000","last_change":"2024-04-16T21:11:50.669285+0000","last_active":"2024-04-16T21:11:50.669169+0000","last_peered":"2024-04-16T21:11:50.669169+0000","last_clean":"2024-04-16T21:11:50.669169+0000","last_became_active":"2024-04-16T19:25:11.464479+0000","last_became_peered":"2024-04-16T19:25:11.464479+0000","last_unstale":"2024-04-16T21:11:50.669169+0000","last_undegraded":"2024-04-16T21:11:50.669169+0000","last_fullsized":"2024-04-16T21:11:50.669169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'332","last_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_deep_scrub":"52'332","last_deep_scrub_stamp":"2024-04-16T21:11:50.669096+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.669096+0000","objects_scrubbed":21,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:18.487063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":470,"num_read_kb":463,"num_write":383,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2179,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"52'12","reported_seq":264,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.639977+0000","last_change":"2024-04-16T21:11:49.640051+0000","last_active":"2024-04-16T21:11:49.639977+0000","last_peered":"2024-04-16T21:11:49.639977+0000","last_clean":"2024-04-16T21:11:49.639977+0000","last_became_active":"2024-04-16T19:25:09.465868+0000","last_became_peered":"2024-04-16T19:25:09.465868+0000","last_unstale":"2024-04-16T21:11:49.639977+0000","last_undegraded":"2024-04-16T21:11:49.639977+0000","last_fullsized":"2024-04-16T21:11:49.639977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:49.639900+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.639900+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:27.425866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":350,"num_read_kb":252,"num_write":74,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.123064+0000","last_change":"2024-04-16T21:12:45.123064+0000","last_active":"2024-04-16T21:12:45.123064+0000","last_peered":"2024-04-16T21:12:45.123064+0000","last_clean":"2024-04-16T21:12:45.123064+0000","last_became_active":"2024-04-16T19:24:35.402678+0000","last_became_peered":"2024-04-16T19:24:35.402678+0000","last_unstale":"2024-04-16T21:12:45.123064+0000","last_undegraded":"2024-04-16T21:12:45.123064+0000","last_fullsized":"2024-04-16T21:12:45.123064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.122972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.122972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.122972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:08.389721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.789162+0000","last_change":"2024-04-16T21:12:46.789162+0000","last_active":"2024-04-16T21:12:46.789162+0000","last_peered":"2024-04-16T21:12:46.789162+0000","last_clean":"2024-04-16T21:12:46.789162+0000","last_became_active":"2024-04-16T19:24:33.388551+0000","last_became_peered":"2024-04-16T19:24:33.388551+0000","last_unstale":"2024-04-16T21:12:46.789162+0000","last_undegraded":"2024-04-16T21:12:46.789162+0000","last_fullsized":"2024-04-16T21:12:46.789162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.789085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.789085+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.789085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:33.086889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.287531+0000","last_change":"2024-04-16T21:12:37.287531+0000","last_active":"2024-04-16T21:12:37.287531+0000","last_peered":"2024-04-16T21:12:37.287531+0000","last_clean":"2024-04-16T21:12:37.287531+0000","last_became_active":"2024-04-16T19:24:45.413526+0000","last_became_peered":"2024-04-16T19:24:45.413526+0000","last_unstale":"2024-04-16T21:12:37.287531+0000","last_undegraded":"2024-04-16T21:12:37.287531+0000","last_fullsized":"2024-04-16T21:12:37.287531+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.287448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.287448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:24.542624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"52'170","reported_seq":377,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.887713+0000","last_change":"2024-04-16T21:12:36.887713+0000","last_active":"2024-04-16T21:12:36.887713+0000","last_peered":"2024-04-16T21:12:36.887713+0000","last_clean":"2024-04-16T21:12:36.887713+0000","last_became_active":"2024-04-16T19:24:47.420588+0000","last_became_peered":"2024-04-16T19:24:47.420588+0000","last_unstale":"2024-04-16T21:12:36.887713+0000","last_undegraded":"2024-04-16T21:12:36.887713+0000","last_fullsized":"2024-04-16T21:12:36.887713+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:36.887622+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.887622+0000","objects_scrubbed":8,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:25.493915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":248,"num_read_kb":238,"num_write":162,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.314224+0000","last_change":"2024-04-16T21:12:26.314224+0000","last_active":"2024-04-16T21:12:26.314224+0000","last_peered":"2024-04-16T21:12:26.314224+0000","last_clean":"2024-04-16T21:12:26.314224+0000","last_became_active":"2024-04-16T19:24:49.430753+0000","last_became_peered":"2024-04-16T19:24:49.430753+0000","last_unstale":"2024-04-16T21:12:26.314224+0000","last_undegraded":"2024-04-16T21:12:26.314224+0000","last_fullsized":"2024-04-16T21:12:26.314224+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.314134+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.314134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:40.584484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.903531+0000","last_change":"2024-04-16T21:12:19.903531+0000","last_active":"2024-04-16T21:12:19.903531+0000","last_peered":"2024-04-16T21:12:19.903531+0000","last_clean":"2024-04-16T21:12:19.903531+0000","last_became_active":"2024-04-16T19:24:51.434439+0000","last_became_peered":"2024-04-16T19:24:51.434439+0000","last_unstale":"2024-04-16T21:12:19.903531+0000","last_undegraded":"2024-04-16T21:12:19.903531+0000","last_fullsized":"2024-04-16T21:12:19.903531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.903448+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.903448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:31.698180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.747319+0000","last_change":"2024-04-16T21:12:51.747319+0000","last_active":"2024-04-16T21:12:51.747319+0000","last_peered":"2024-04-16T21:12:51.747319+0000","last_clean":"2024-04-16T21:12:51.747319+0000","last_became_active":"2024-04-16T19:47:33.985334+0000","last_became_peered":"2024-04-16T19:47:33.985334+0000","last_unstale":"2024-04-16T21:12:51.747319+0000","last_undegraded":"2024-04-16T21:12:51.747319+0000","last_fullsized":"2024-04-16T21:12:51.747319+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.747234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.747234+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.747234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:59:15.013984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"52'216","reported_seq":2161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.661893+0000","last_change":"2024-04-16T21:12:21.661923+0000","last_active":"2024-04-16T21:12:21.661893+0000","last_peered":"2024-04-16T21:12:21.661893+0000","last_clean":"2024-04-16T21:12:21.661893+0000","last_became_active":"2024-04-16T19:25:49.518530+0000","last_became_peered":"2024-04-16T19:25:49.518530+0000","last_unstale":"2024-04-16T21:12:21.661893+0000","last_undegraded":"2024-04-16T21:12:21.661893+0000","last_fullsized":"2024-04-16T21:12:21.661893+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'216","last_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_deep_scrub":"52'216","last_deep_scrub_stamp":"2024-04-16T21:12:21.661832+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.661832+0000","objects_scrubbed":54,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:47.097281+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":3258,"num_read_kb":3202,"num_write":236,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14476,"num_omap_keys":58,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.921849+0000","last_change":"2024-04-16T21:12:14.921870+0000","last_active":"2024-04-16T21:12:14.921849+0000","last_peered":"2024-04-16T21:12:14.921849+0000","last_clean":"2024-04-16T21:12:14.921849+0000","last_became_active":"2024-04-16T19:25:41.510013+0000","last_became_peered":"2024-04-16T19:25:41.510013+0000","last_unstale":"2024-04-16T21:12:14.921849+0000","last_undegraded":"2024-04-16T21:12:14.921849+0000","last_fullsized":"2024-04-16T21:12:14.921849+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.921774+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.921774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:24:11.765704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"52'17","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.682317+0000","last_change":"2024-04-16T21:12:08.682317+0000","last_active":"2024-04-16T21:12:08.682317+0000","last_peered":"2024-04-16T21:12:08.682317+0000","last_clean":"2024-04-16T21:12:08.682317+0000","last_became_active":"2024-04-16T19:25:15.480310+0000","last_became_peered":"2024-04-16T19:25:15.480310+0000","last_unstale":"2024-04-16T21:12:08.682317+0000","last_undegraded":"2024-04-16T21:12:08.682317+0000","last_fullsized":"2024-04-16T21:12:08.682317+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:08.682212+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.682212+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:32.852256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":970,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":25,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.634331+0000","last_change":"2024-04-16T21:11:56.634374+0000","last_active":"2024-04-16T21:11:56.634331+0000","last_peered":"2024-04-16T21:11:56.634331+0000","last_clean":"2024-04-16T21:11:56.634331+0000","last_became_active":"2024-04-16T19:25:13.469351+0000","last_became_peered":"2024-04-16T19:25:13.469351+0000","last_unstale":"2024-04-16T21:11:56.634331+0000","last_undegraded":"2024-04-16T21:11:56.634331+0000","last_fullsized":"2024-04-16T21:11:56.634331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:56.634253+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.634253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:30.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"52'281","reported_seq":602,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.734203+0000","last_change":"2024-04-16T21:11:44.734203+0000","last_active":"2024-04-16T21:11:44.734203+0000","last_peered":"2024-04-16T21:11:44.734203+0000","last_clean":"2024-04-16T21:11:44.734203+0000","last_became_active":"2024-04-16T19:25:11.463141+0000","last_became_peered":"2024-04-16T19:25:11.463141+0000","last_unstale":"2024-04-16T21:11:44.734203+0000","last_undegraded":"2024-04-16T21:11:44.734203+0000","last_fullsized":"2024-04-16T21:11:44.734203+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'281","last_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_deep_scrub":"52'281","last_deep_scrub_stamp":"2024-04-16T21:11:44.734118+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.734118+0000","objects_scrubbed":25,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:56:22.520084+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":473,"num_read_kb":466,"num_write":312,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1975,"num_omap_keys":9,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"52'12","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.876267+0000","last_change":"2024-04-16T21:12:09.876315+0000","last_active":"2024-04-16T21:12:09.876267+0000","last_peered":"2024-04-16T21:12:09.876267+0000","last_clean":"2024-04-16T21:12:09.876267+0000","last_became_active":"2024-04-16T19:25:09.461900+0000","last_became_peered":"2024-04-16T19:25:09.461900+0000","last_unstale":"2024-04-16T21:12:09.876267+0000","last_undegraded":"2024-04-16T21:12:09.876267+0000","last_fullsized":"2024-04-16T21:12:09.876267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:09.876206+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.876206+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:31.314787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":231,"num_write":70,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.204766+0000","last_change":"2024-04-16T21:12:43.204766+0000","last_active":"2024-04-16T21:12:43.204766+0000","last_peered":"2024-04-16T21:12:43.204766+0000","last_clean":"2024-04-16T21:12:43.204766+0000","last_became_active":"2024-04-16T19:24:35.397870+0000","last_became_peered":"2024-04-16T19:24:35.397870+0000","last_unstale":"2024-04-16T21:12:43.204766+0000","last_undegraded":"2024-04-16T21:12:43.204766+0000","last_fullsized":"2024-04-16T21:12:43.204766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.204682+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.204682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:58.459466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"52'119","reported_seq":303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.254459+0000","last_change":"2024-04-16T21:13:02.254490+0000","last_active":"2024-04-16T21:13:02.254459+0000","last_peered":"2024-04-16T21:13:02.254459+0000","last_clean":"2024-04-16T21:13:02.254459+0000","last_became_active":"2024-04-16T19:23:46.115518+0000","last_became_peered":"2024-04-16T19:23:46.115518+0000","last_unstale":"2024-04-16T21:13:02.254459+0000","last_undegraded":"2024-04-16T21:13:02.254459+0000","last_fullsized":"2024-04-16T21:13:02.254459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'119","last_scrub_stamp":"2024-04-16T21:13:02.254402+0000","last_deep_scrub":"52'119","last_deep_scrub_stamp":"2024-04-16T21:13:02.254402+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.254402+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:45.412979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.823040+0000","last_change":"2024-04-16T21:13:03.823065+0000","last_active":"2024-04-16T21:13:03.823040+0000","last_peered":"2024-04-16T21:13:03.823040+0000","last_clean":"2024-04-16T21:13:03.823040+0000","last_became_active":"2024-04-16T19:23:47.119425+0000","last_became_peered":"2024-04-16T19:23:47.119425+0000","last_unstale":"2024-04-16T21:13:03.823040+0000","last_undegraded":"2024-04-16T21:13:03.823040+0000","last_fullsized":"2024-04-16T21:13:03.823040+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.822971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.822971+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.822971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:13:18.427093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.091208+0000","last_change":"2024-04-16T21:12:59.091231+0000","last_active":"2024-04-16T21:12:59.091208+0000","last_peered":"2024-04-16T21:12:59.091208+0000","last_clean":"2024-04-16T21:12:59.091208+0000","last_became_active":"2024-04-16T19:24:33.384597+0000","last_became_peered":"2024-04-16T19:24:33.384597+0000","last_unstale":"2024-04-16T21:12:59.091208+0000","last_undegraded":"2024-04-16T21:12:59.091208+0000","last_fullsized":"2024-04-16T21:12:59.091208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.091143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.091143+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.091143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:48.532211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.706720+0000","last_change":"2024-04-16T21:12:37.706720+0000","last_active":"2024-04-16T21:12:37.706720+0000","last_peered":"2024-04-16T21:12:37.706720+0000","last_clean":"2024-04-16T21:12:37.706720+0000","last_became_active":"2024-04-16T19:24:45.410168+0000","last_became_peered":"2024-04-16T19:24:45.410168+0000","last_unstale":"2024-04-16T21:12:37.706720+0000","last_undegraded":"2024-04-16T21:12:37.706720+0000","last_fullsized":"2024-04-16T21:12:37.706720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.706633+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.706633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:46.200865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"52'73","reported_seq":245,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:32.829190+0000","last_change":"2024-04-16T21:12:32.829190+0000","last_active":"2024-04-16T21:12:32.829190+0000","last_peered":"2024-04-16T21:12:32.829190+0000","last_clean":"2024-04-16T21:12:32.829190+0000","last_became_active":"2024-04-16T19:24:47.425566+0000","last_became_peered":"2024-04-16T19:24:47.425566+0000","last_unstale":"2024-04-16T21:12:32.829190+0000","last_undegraded":"2024-04-16T21:12:32.829190+0000","last_fullsized":"2024-04-16T21:12:32.829190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'73","last_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_deep_scrub":"52'73","last_deep_scrub_stamp":"2024-04-16T21:12:32.829094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:32.829094+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:32.804059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":106,"num_write":68,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.313577+0000","last_change":"2024-04-16T21:12:28.313577+0000","last_active":"2024-04-16T21:12:28.313577+0000","last_peered":"2024-04-16T21:12:28.313577+0000","last_clean":"2024-04-16T21:12:28.313577+0000","last_became_active":"2024-04-16T19:24:49.428526+0000","last_became_peered":"2024-04-16T19:24:49.428526+0000","last_unstale":"2024-04-16T21:12:28.313577+0000","last_undegraded":"2024-04-16T21:12:28.313577+0000","last_fullsized":"2024-04-16T21:12:28.313577+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.313486+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.313486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:34.551134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.689165+0000","last_change":"2024-04-16T21:12:06.689165+0000","last_active":"2024-04-16T21:12:06.689165+0000","last_peered":"2024-04-16T21:12:06.689165+0000","last_clean":"2024-04-16T21:12:06.689165+0000","last_became_active":"2024-04-16T19:24:51.442290+0000","last_became_peered":"2024-04-16T19:24:51.442290+0000","last_unstale":"2024-04-16T21:12:06.689165+0000","last_undegraded":"2024-04-16T21:12:06.689165+0000","last_fullsized":"2024-04-16T21:12:06.689165+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.689080+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.689080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:58.869391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.806497+0000","last_change":"2024-04-16T21:12:57.806497+0000","last_active":"2024-04-16T21:12:57.806497+0000","last_peered":"2024-04-16T21:12:57.806497+0000","last_clean":"2024-04-16T21:12:57.806497+0000","last_became_active":"2024-04-16T19:47:33.980633+0000","last_became_peered":"2024-04-16T19:47:33.980633+0000","last_unstale":"2024-04-16T21:12:57.806497+0000","last_undegraded":"2024-04-16T21:12:57.806497+0000","last_fullsized":"2024-04-16T21:12:57.806497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.806335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.806335+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.806335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:54:38.758338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.c","version":"52'183","reported_seq":2009,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.334593+0000","last_change":"2024-04-16T21:12:30.334647+0000","last_active":"2024-04-16T21:12:30.334593+0000","last_peered":"2024-04-16T21:12:30.334593+0000","last_clean":"2024-04-16T21:12:30.334593+0000","last_became_active":"2024-04-16T19:25:49.514533+0000","last_became_peered":"2024-04-16T19:25:49.514533+0000","last_unstale":"2024-04-16T21:12:30.334593+0000","last_undegraded":"2024-04-16T21:12:30.334593+0000","last_fullsized":"2024-04-16T21:12:30.334593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'183","last_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_deep_scrub":"52'183","last_deep_scrub_stamp":"2024-04-16T21:12:30.334529+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.334529+0000","objects_scrubbed":48,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:58:53.601279+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3161,"num_read_kb":3101,"num_write":202,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14209,"num_omap_keys":58,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.856959+0000","last_change":"2024-04-16T21:12:37.856984+0000","last_active":"2024-04-16T21:12:37.856959+0000","last_peered":"2024-04-16T21:12:37.856959+0000","last_clean":"2024-04-16T21:12:37.856959+0000","last_became_active":"2024-04-16T19:25:41.509188+0000","last_became_peered":"2024-04-16T19:25:41.509188+0000","last_unstale":"2024-04-16T21:12:37.856959+0000","last_undegraded":"2024-04-16T21:12:37.856959+0000","last_fullsized":"2024-04-16T21:12:37.856959+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.856878+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.856878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:44.305267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"52'16","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:20.513369+0000","last_change":"2024-04-16T21:12:20.513369+0000","last_active":"2024-04-16T21:12:20.513369+0000","last_peered":"2024-04-16T21:12:20.513369+0000","last_clean":"2024-04-16T21:12:20.513369+0000","last_became_active":"2024-04-16T19:25:15.484928+0000","last_became_peered":"2024-04-16T19:25:15.484928+0000","last_unstale":"2024-04-16T21:12:20.513369+0000","last_undegraded":"2024-04-16T21:12:20.513369+0000","last_fullsized":"2024-04-16T21:12:20.513369+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:20.513228+0000","last_clean_scrub_stamp":"2024-04-16T21:12:20.513228+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:32.115899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"52'706","reported_seq":2087,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.915788+0000","last_change":"2024-04-16T21:11:48.915841+0000","last_active":"2024-04-16T21:11:48.915788+0000","last_peered":"2024-04-16T21:11:48.915788+0000","last_clean":"2024-04-16T21:11:48.915788+0000","last_became_active":"2024-04-16T19:25:13.466964+0000","last_became_peered":"2024-04-16T19:25:13.466964+0000","last_unstale":"2024-04-16T21:11:48.915788+0000","last_undegraded":"2024-04-16T21:11:48.915788+0000","last_fullsized":"2024-04-16T21:11:48.915788+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:48.915706+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.915706+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.964684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"52'326","reported_seq":644,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.279300+0000","last_change":"2024-04-16T21:11:56.279412+0000","last_active":"2024-04-16T21:11:56.279300+0000","last_peered":"2024-04-16T21:11:56.279300+0000","last_clean":"2024-04-16T21:11:56.279300+0000","last_became_active":"2024-04-16T19:25:11.467012+0000","last_became_peered":"2024-04-16T19:25:11.467012+0000","last_unstale":"2024-04-16T21:11:56.279300+0000","last_undegraded":"2024-04-16T21:11:56.279300+0000","last_fullsized":"2024-04-16T21:11:56.279300+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'326","last_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_deep_scrub":"52'326","last_deep_scrub_stamp":"2024-04-16T21:11:56.279222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.279222+0000","objects_scrubbed":20,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:03.299266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":514,"num_read_kb":504,"num_write":350,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"52'20","reported_seq":202,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.117905+0000","last_change":"2024-04-16T21:11:45.117905+0000","last_active":"2024-04-16T21:11:45.117905+0000","last_peered":"2024-04-16T21:11:45.117905+0000","last_clean":"2024-04-16T21:11:45.117905+0000","last_became_active":"2024-04-16T19:25:09.463747+0000","last_became_peered":"2024-04-16T19:25:09.463747+0000","last_unstale":"2024-04-16T21:11:45.117905+0000","last_undegraded":"2024-04-16T21:11:45.117905+0000","last_fullsized":"2024-04-16T21:11:45.117905+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:45.117757+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.117757+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:23.210409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":189,"num_read_kb":134,"num_write":105,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.379965+0000","last_change":"2024-04-16T21:12:50.379965+0000","last_active":"2024-04-16T21:12:50.379965+0000","last_peered":"2024-04-16T21:12:50.379965+0000","last_clean":"2024-04-16T21:12:50.379965+0000","last_became_active":"2024-04-16T19:24:35.397035+0000","last_became_peered":"2024-04-16T19:24:35.397035+0000","last_unstale":"2024-04-16T21:12:50.379965+0000","last_undegraded":"2024-04-16T21:12:50.379965+0000","last_fullsized":"2024-04-16T21:12:50.379965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.379848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.379848+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.379848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:32:31.683476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.260284+0000","last_change":"2024-04-16T21:13:03.260307+0000","last_active":"2024-04-16T21:13:03.260284+0000","last_peered":"2024-04-16T21:13:03.260284+0000","last_clean":"2024-04-16T21:13:03.260284+0000","last_became_active":"2024-04-16T19:23:47.119553+0000","last_became_peered":"2024-04-16T19:23:47.119553+0000","last_unstale":"2024-04-16T21:13:03.260284+0000","last_undegraded":"2024-04-16T21:13:03.260284+0000","last_fullsized":"2024-04-16T21:13:03.260284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.260223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.260223+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.260223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:00:58.899912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.496213+0000","last_change":"2024-04-16T21:13:02.496238+0000","last_active":"2024-04-16T21:13:02.496213+0000","last_peered":"2024-04-16T21:13:02.496213+0000","last_clean":"2024-04-16T21:13:02.496213+0000","last_became_active":"2024-04-16T19:24:33.382953+0000","last_became_peered":"2024-04-16T19:24:33.382953+0000","last_unstale":"2024-04-16T21:13:02.496213+0000","last_undegraded":"2024-04-16T21:13:02.496213+0000","last_fullsized":"2024-04-16T21:13:02.496213+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.496135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.496135+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.496135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:32.552774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"52'8","reported_seq":605,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.885408+0000","last_change":"2024-04-16T21:12:40.885408+0000","last_active":"2024-04-16T21:12:40.885408+0000","last_peered":"2024-04-16T21:12:40.885408+0000","last_clean":"2024-04-16T21:12:40.885408+0000","last_became_active":"2024-04-16T19:24:45.414682+0000","last_became_peered":"2024-04-16T19:24:45.414682+0000","last_unstale":"2024-04-16T21:12:40.885408+0000","last_undegraded":"2024-04-16T21:12:40.885408+0000","last_fullsized":"2024-04-16T21:12:40.885408+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:40.885321+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.885321+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:38:11.985385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":470,"num_read_kb":481,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"52'175","reported_seq":1589,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.039035+0000","last_change":"2024-04-16T21:12:33.039035+0000","last_active":"2024-04-16T21:12:33.039035+0000","last_peered":"2024-04-16T21:12:33.039035+0000","last_clean":"2024-04-16T21:12:33.039035+0000","last_became_active":"2024-04-16T19:24:47.423353+0000","last_became_peered":"2024-04-16T19:24:47.423353+0000","last_unstale":"2024-04-16T21:12:33.039035+0000","last_undegraded":"2024-04-16T21:12:33.039035+0000","last_fullsized":"2024-04-16T21:12:33.039035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'175","last_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_deep_scrub":"52'175","last_deep_scrub_stamp":"2024-04-16T21:12:33.038938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.038938+0000","objects_scrubbed":10,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:04.725640+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":261,"num_read_kb":246,"num_write":158,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.604589+0000","last_change":"2024-04-16T21:12:24.604613+0000","last_active":"2024-04-16T21:12:24.604589+0000","last_peered":"2024-04-16T21:12:24.604589+0000","last_clean":"2024-04-16T21:12:24.604589+0000","last_became_active":"2024-04-16T19:24:49.425641+0000","last_became_peered":"2024-04-16T19:24:49.425641+0000","last_unstale":"2024-04-16T21:12:24.604589+0000","last_undegraded":"2024-04-16T21:12:24.604589+0000","last_fullsized":"2024-04-16T21:12:24.604589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.604487+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.604487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.850465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":114,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.671618+0000","last_change":"2024-04-16T21:11:57.671660+0000","last_active":"2024-04-16T21:11:57.671618+0000","last_peered":"2024-04-16T21:11:57.671618+0000","last_clean":"2024-04-16T21:11:57.671618+0000","last_became_active":"2024-04-16T19:24:51.573948+0000","last_became_peered":"2024-04-16T19:24:51.573948+0000","last_unstale":"2024-04-16T21:11:57.671618+0000","last_undegraded":"2024-04-16T21:11:57.671618+0000","last_fullsized":"2024-04-16T21:11:57.671618+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:57.671545+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.671545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:20.862941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:18.999270+0000","last_change":"2024-04-16T21:13:18.999293+0000","last_active":"2024-04-16T21:13:18.999270+0000","last_peered":"2024-04-16T21:13:18.999270+0000","last_clean":"2024-04-16T21:13:18.999270+0000","last_became_active":"2024-04-16T19:47:33.986675+0000","last_became_peered":"2024-04-16T19:47:33.986675+0000","last_unstale":"2024-04-16T21:13:18.999270+0000","last_undegraded":"2024-04-16T21:13:18.999270+0000","last_fullsized":"2024-04-16T21:13:18.999270+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:18.999198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:18.999198+0000","last_clean_scrub_stamp":"2024-04-16T21:13:18.999198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:13:47.151818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"52'249","reported_seq":2425,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:12.803829+0000","last_change":"2024-04-16T21:13:12.803888+0000","last_active":"2024-04-16T21:13:12.803829+0000","last_peered":"2024-04-16T21:13:12.803829+0000","last_clean":"2024-04-16T21:13:12.803829+0000","last_became_active":"2024-04-16T19:25:49.518196+0000","last_became_peered":"2024-04-16T19:25:49.518196+0000","last_unstale":"2024-04-16T21:13:12.803829+0000","last_undegraded":"2024-04-16T21:13:12.803829+0000","last_fullsized":"2024-04-16T21:13:12.803829+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:13:12.803767+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:13:12.803767+0000","last_clean_scrub_stamp":"2024-04-16T21:13:12.803767+0000","objects_scrubbed":46,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:15:14.437703+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3834,"num_read_kb":4124,"num_write":374,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22725,"num_omap_keys":90,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.830577+0000","last_change":"2024-04-16T21:12:25.830601+0000","last_active":"2024-04-16T21:12:25.830577+0000","last_peered":"2024-04-16T21:12:25.830577+0000","last_clean":"2024-04-16T21:12:25.830577+0000","last_became_active":"2024-04-16T19:25:41.503749+0000","last_became_peered":"2024-04-16T19:25:41.503749+0000","last_unstale":"2024-04-16T21:12:25.830577+0000","last_undegraded":"2024-04-16T21:12:25.830577+0000","last_fullsized":"2024-04-16T21:12:25.830577+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.830513+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.830513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:35.392181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"52'19","reported_seq":165,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.380245+0000","last_change":"2024-04-16T21:12:21.380273+0000","last_active":"2024-04-16T21:12:21.380245+0000","last_peered":"2024-04-16T21:12:21.380245+0000","last_clean":"2024-04-16T21:12:21.380245+0000","last_became_active":"2024-04-16T19:25:15.484454+0000","last_became_peered":"2024-04-16T19:25:15.484454+0000","last_unstale":"2024-04-16T21:12:21.380245+0000","last_undegraded":"2024-04-16T21:12:21.380245+0000","last_fullsized":"2024-04-16T21:12:21.380245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:21.380184+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.380184+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:53.066886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":92,"num_write":27,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":60,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"52'21","reported_seq":982,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.783889+0000","last_change":"2024-04-16T21:11:58.783920+0000","last_active":"2024-04-16T21:11:58.783889+0000","last_peered":"2024-04-16T21:11:58.783889+0000","last_clean":"2024-04-16T21:11:58.783889+0000","last_became_active":"2024-04-16T19:25:13.474433+0000","last_became_peered":"2024-04-16T19:25:13.474433+0000","last_unstale":"2024-04-16T21:11:58.783889+0000","last_undegraded":"2024-04-16T21:11:58.783889+0000","last_fullsized":"2024-04-16T21:11:58.783889+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:11:58.783819+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.783819+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:59.285203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.9","version":"52'268","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.359130+0000","last_change":"2024-04-16T21:11:44.359183+0000","last_active":"2024-04-16T21:11:44.359130+0000","last_peered":"2024-04-16T21:11:44.359130+0000","last_clean":"2024-04-16T21:11:44.359130+0000","last_became_active":"2024-04-16T19:25:11.460586+0000","last_became_peered":"2024-04-16T19:25:11.460586+0000","last_unstale":"2024-04-16T21:11:44.359130+0000","last_undegraded":"2024-04-16T21:11:44.359130+0000","last_fullsized":"2024-04-16T21:11:44.359130+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'268","last_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_deep_scrub":"52'268","last_deep_scrub_stamp":"2024-04-16T21:11:44.359071+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.359071+0000","objects_scrubbed":19,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:18.502556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":610,"num_read_kb":598,"num_write":303,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"52'13","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.427731+0000","last_change":"2024-04-16T21:11:53.427771+0000","last_active":"2024-04-16T21:11:53.427731+0000","last_peered":"2024-04-16T21:11:53.427731+0000","last_clean":"2024-04-16T21:11:53.427731+0000","last_became_active":"2024-04-16T19:25:09.460131+0000","last_became_peered":"2024-04-16T19:25:09.460131+0000","last_unstale":"2024-04-16T21:11:53.427731+0000","last_undegraded":"2024-04-16T21:11:53.427731+0000","last_fullsized":"2024-04-16T21:11:53.427731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:53.427656+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.427656+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:14.929241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":130,"num_write":84,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.080980+0000","last_change":"2024-04-16T21:12:42.080980+0000","last_active":"2024-04-16T21:12:42.080980+0000","last_peered":"2024-04-16T21:12:42.080980+0000","last_clean":"2024-04-16T21:12:42.080980+0000","last_became_active":"2024-04-16T19:24:35.399336+0000","last_became_peered":"2024-04-16T19:24:35.399336+0000","last_unstale":"2024-04-16T21:12:42.080980+0000","last_undegraded":"2024-04-16T21:12:42.080980+0000","last_fullsized":"2024-04-16T21:12:42.080980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.080859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.080859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:48.529247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.765443+0000","last_change":"2024-04-16T21:12:42.765465+0000","last_active":"2024-04-16T21:12:42.765443+0000","last_peered":"2024-04-16T21:12:42.765443+0000","last_clean":"2024-04-16T21:12:42.765443+0000","last_became_active":"2024-04-16T19:23:47.118972+0000","last_became_peered":"2024-04-16T19:23:47.118972+0000","last_unstale":"2024-04-16T21:12:42.765443+0000","last_undegraded":"2024-04-16T21:12:42.765443+0000","last_fullsized":"2024-04-16T21:12:42.765443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.765375+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.765375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:30.897866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.114045+0000","last_change":"2024-04-16T21:12:52.114045+0000","last_active":"2024-04-16T21:12:52.114045+0000","last_peered":"2024-04-16T21:12:52.114045+0000","last_clean":"2024-04-16T21:12:52.114045+0000","last_became_active":"2024-04-16T19:24:33.387518+0000","last_became_peered":"2024-04-16T19:24:33.387518+0000","last_unstale":"2024-04-16T21:12:52.114045+0000","last_undegraded":"2024-04-16T21:12:52.114045+0000","last_fullsized":"2024-04-16T21:12:52.114045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.113954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.113954+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.113954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:36.871712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"52'6","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.801302+0000","last_change":"2024-04-16T21:12:34.801302+0000","last_active":"2024-04-16T21:12:34.801302+0000","last_peered":"2024-04-16T21:12:34.801302+0000","last_clean":"2024-04-16T21:12:34.801302+0000","last_became_active":"2024-04-16T19:24:45.410556+0000","last_became_peered":"2024-04-16T19:24:45.410556+0000","last_unstale":"2024-04-16T21:12:34.801302+0000","last_undegraded":"2024-04-16T21:12:34.801302+0000","last_fullsized":"2024-04-16T21:12:34.801302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:34.801209+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.801209+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:13.234565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"52'147","reported_seq":1530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.863577+0000","last_change":"2024-04-16T21:12:39.863616+0000","last_active":"2024-04-16T21:12:39.863577+0000","last_peered":"2024-04-16T21:12:39.863577+0000","last_clean":"2024-04-16T21:12:39.863577+0000","last_became_active":"2024-04-16T19:24:47.419526+0000","last_became_peered":"2024-04-16T19:24:47.419526+0000","last_unstale":"2024-04-16T21:12:39.863577+0000","last_undegraded":"2024-04-16T21:12:39.863577+0000","last_fullsized":"2024-04-16T21:12:39.863577+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:39.863516+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.863516+0000","objects_scrubbed":8,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:55.662812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":202,"num_read_kb":194,"num_write":133,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.339693+0000","last_change":"2024-04-16T21:12:27.339693+0000","last_active":"2024-04-16T21:12:27.339693+0000","last_peered":"2024-04-16T21:12:27.339693+0000","last_clean":"2024-04-16T21:12:27.339693+0000","last_became_active":"2024-04-16T19:24:49.431370+0000","last_became_peered":"2024-04-16T19:24:49.431370+0000","last_unstale":"2024-04-16T21:12:27.339693+0000","last_undegraded":"2024-04-16T21:12:27.339693+0000","last_fullsized":"2024-04-16T21:12:27.339693+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.339563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.339563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:33.787952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.370546+0000","last_change":"2024-04-16T21:12:05.370546+0000","last_active":"2024-04-16T21:12:05.370546+0000","last_peered":"2024-04-16T21:12:05.370546+0000","last_clean":"2024-04-16T21:12:05.370546+0000","last_became_active":"2024-04-16T19:24:51.437742+0000","last_became_peered":"2024-04-16T19:24:51.437742+0000","last_unstale":"2024-04-16T21:12:05.370546+0000","last_undegraded":"2024-04-16T21:12:05.370546+0000","last_fullsized":"2024-04-16T21:12:05.370546+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.370430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.370430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:17.165162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:16.180898+0000","last_change":"2024-04-16T21:13:16.180898+0000","last_active":"2024-04-16T21:13:16.180898+0000","last_peered":"2024-04-16T21:13:16.180898+0000","last_clean":"2024-04-16T21:13:16.180898+0000","last_became_active":"2024-04-16T19:47:33.987653+0000","last_became_peered":"2024-04-16T19:47:33.987653+0000","last_unstale":"2024-04-16T21:13:16.180898+0000","last_undegraded":"2024-04-16T21:13:16.180898+0000","last_fullsized":"2024-04-16T21:13:16.180898+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:16.180813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:16.180813+0000","last_clean_scrub_stamp":"2024-04-16T21:13:16.180813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:31:34.669615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.e","version":"52'161","reported_seq":1820,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.785185+0000","last_change":"2024-04-16T21:12:56.785241+0000","last_active":"2024-04-16T21:12:56.785185+0000","last_peered":"2024-04-16T21:12:56.785185+0000","last_clean":"2024-04-16T21:12:56.785185+0000","last_became_active":"2024-04-16T19:25:49.519169+0000","last_became_peered":"2024-04-16T19:25:49.519169+0000","last_unstale":"2024-04-16T21:12:56.785185+0000","last_undegraded":"2024-04-16T21:12:56.785185+0000","last_fullsized":"2024-04-16T21:12:56.785185+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'161","last_scrub_stamp":"2024-04-16T21:12:56.785125+0000","last_deep_scrub":"52'161","last_deep_scrub_stamp":"2024-04-16T21:12:56.785125+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.785125+0000","objects_scrubbed":44,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:49:38.484287+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":2762,"num_read_kb":2731,"num_write":174,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9743,"num_omap_keys":41,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.699415+0000","last_change":"2024-04-16T21:11:59.699455+0000","last_active":"2024-04-16T21:11:59.699415+0000","last_peered":"2024-04-16T21:11:59.699415+0000","last_clean":"2024-04-16T21:11:59.699415+0000","last_became_active":"2024-04-16T19:25:41.505022+0000","last_became_peered":"2024-04-16T19:25:41.505022+0000","last_unstale":"2024-04-16T21:11:59.699415+0000","last_undegraded":"2024-04-16T21:11:59.699415+0000","last_fullsized":"2024-04-16T21:11:59.699415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.699336+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.699336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:24:41.370639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"52'21","reported_seq":189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.432112+0000","last_change":"2024-04-16T21:12:15.432112+0000","last_active":"2024-04-16T21:12:15.432112+0000","last_peered":"2024-04-16T21:12:15.432112+0000","last_clean":"2024-04-16T21:12:15.432112+0000","last_became_active":"2024-04-16T19:25:15.487260+0000","last_became_peered":"2024-04-16T19:25:15.487260+0000","last_unstale":"2024-04-16T21:12:15.432112+0000","last_undegraded":"2024-04-16T21:12:15.432112+0000","last_fullsized":"2024-04-16T21:12:15.432112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'21","last_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_deep_scrub":"52'21","last_deep_scrub_stamp":"2024-04-16T21:12:15.431977+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.431977+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:24.993645+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1997,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":226,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.891673+0000","last_change":"2024-04-16T21:12:03.891673+0000","last_active":"2024-04-16T21:12:03.891673+0000","last_peered":"2024-04-16T21:12:03.891673+0000","last_clean":"2024-04-16T21:12:03.891673+0000","last_became_active":"2024-04-16T19:25:13.471552+0000","last_became_peered":"2024-04-16T19:25:13.471552+0000","last_unstale":"2024-04-16T21:12:03.891673+0000","last_undegraded":"2024-04-16T21:12:03.891673+0000","last_fullsized":"2024-04-16T21:12:03.891673+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.891582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.891582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:27.082978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"52'720","reported_seq":1730,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:44.792306+0000","last_change":"2024-04-16T21:11:44.792348+0000","last_active":"2024-04-16T21:11:44.792306+0000","last_peered":"2024-04-16T21:11:44.792306+0000","last_clean":"2024-04-16T21:11:44.792306+0000","last_became_active":"2024-04-16T19:25:11.468541+0000","last_became_peered":"2024-04-16T19:25:11.468541+0000","last_unstale":"2024-04-16T21:11:44.792306+0000","last_undegraded":"2024-04-16T21:11:44.792306+0000","last_fullsized":"2024-04-16T21:11:44.792306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'720","last_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_deep_scrub":"52'720","last_deep_scrub_stamp":"2024-04-16T21:11:44.792244+0000","last_clean_scrub_stamp":"2024-04-16T21:11:44.792244+0000","objects_scrubbed":30,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:07:06.408902+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1723,"num_read_kb":1568,"num_write":646,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2191,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"52'20","reported_seq":185,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.222406+0000","last_change":"2024-04-16T21:11:49.222443+0000","last_active":"2024-04-16T21:11:49.222406+0000","last_peered":"2024-04-16T21:11:49.222406+0000","last_clean":"2024-04-16T21:11:49.222406+0000","last_became_active":"2024-04-16T19:25:09.462280+0000","last_became_peered":"2024-04-16T19:25:09.462280+0000","last_unstale":"2024-04-16T21:11:49.222406+0000","last_undegraded":"2024-04-16T21:11:49.222406+0000","last_fullsized":"2024-04-16T21:11:49.222406+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:49.222345+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.222345+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:21.815371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":112,"num_write":103,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.118034+0000","last_change":"2024-04-16T21:12:48.118034+0000","last_active":"2024-04-16T21:12:48.118034+0000","last_peered":"2024-04-16T21:12:48.118034+0000","last_clean":"2024-04-16T21:12:48.118034+0000","last_became_active":"2024-04-16T19:24:35.393008+0000","last_became_peered":"2024-04-16T19:24:35.393008+0000","last_unstale":"2024-04-16T21:12:48.118034+0000","last_undegraded":"2024-04-16T21:12:48.118034+0000","last_fullsized":"2024-04-16T21:12:48.118034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.117906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.117906+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.117906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:45.093346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.805409+0000","last_change":"2024-04-16T21:13:04.805431+0000","last_active":"2024-04-16T21:13:04.805409+0000","last_peered":"2024-04-16T21:13:04.805409+0000","last_clean":"2024-04-16T21:13:04.805409+0000","last_became_active":"2024-04-16T19:23:47.119416+0000","last_became_peered":"2024-04-16T19:23:47.119416+0000","last_unstale":"2024-04-16T21:13:04.805409+0000","last_undegraded":"2024-04-16T21:13:04.805409+0000","last_fullsized":"2024-04-16T21:13:04.805409+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:04.805347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:04.805347+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.805347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:45:39.967913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.707830+0000","last_change":"2024-04-16T21:12:47.707830+0000","last_active":"2024-04-16T21:12:47.707830+0000","last_peered":"2024-04-16T21:12:47.707830+0000","last_clean":"2024-04-16T21:12:47.707830+0000","last_became_active":"2024-04-16T19:24:33.380764+0000","last_became_peered":"2024-04-16T19:24:33.380764+0000","last_unstale":"2024-04-16T21:12:47.707830+0000","last_undegraded":"2024-04-16T21:12:47.707830+0000","last_fullsized":"2024-04-16T21:12:47.707830+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.707745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.707745+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.707745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:54.156133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.476532+0000","last_change":"2024-04-16T21:12:58.476558+0000","last_active":"2024-04-16T21:12:58.476532+0000","last_peered":"2024-04-16T21:12:58.476532+0000","last_clean":"2024-04-16T21:12:58.476532+0000","last_became_active":"2024-04-16T19:24:45.409000+0000","last_became_peered":"2024-04-16T19:24:45.409000+0000","last_unstale":"2024-04-16T21:12:58.476532+0000","last_undegraded":"2024-04-16T21:12:58.476532+0000","last_fullsized":"2024-04-16T21:12:58.476532+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.476463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.476463+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.476463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:31.485168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"52'138","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.993917+0000","last_change":"2024-04-16T21:12:33.993917+0000","last_active":"2024-04-16T21:12:33.993917+0000","last_peered":"2024-04-16T21:12:33.993917+0000","last_clean":"2024-04-16T21:12:33.993917+0000","last_became_active":"2024-04-16T19:24:47.421903+0000","last_became_peered":"2024-04-16T19:24:47.421903+0000","last_unstale":"2024-04-16T21:12:33.993917+0000","last_undegraded":"2024-04-16T21:12:33.993917+0000","last_fullsized":"2024-04-16T21:12:33.993917+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:12:33.993823+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.993823+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:28.182498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":198,"num_read_kb":192,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.394517+0000","last_change":"2024-04-16T21:12:09.394517+0000","last_active":"2024-04-16T21:12:09.394517+0000","last_peered":"2024-04-16T21:12:09.394517+0000","last_clean":"2024-04-16T21:12:09.394517+0000","last_became_active":"2024-04-16T19:24:49.431692+0000","last_became_peered":"2024-04-16T19:24:49.431692+0000","last_unstale":"2024-04-16T21:12:09.394517+0000","last_undegraded":"2024-04-16T21:12:09.394517+0000","last_fullsized":"2024-04-16T21:12:09.394517+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.394403+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.394403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:58.357952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.902708+0000","last_change":"2024-04-16T21:12:14.902708+0000","last_active":"2024-04-16T21:12:14.902708+0000","last_peered":"2024-04-16T21:12:14.902708+0000","last_clean":"2024-04-16T21:12:14.902708+0000","last_became_active":"2024-04-16T19:24:51.436671+0000","last_became_peered":"2024-04-16T19:24:51.436671+0000","last_unstale":"2024-04-16T21:12:14.902708+0000","last_undegraded":"2024-04-16T21:12:14.902708+0000","last_fullsized":"2024-04-16T21:12:14.902708+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:14.902624+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.902624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:15:23.524467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:15.896984+0000","last_change":"2024-04-16T21:13:15.897007+0000","last_active":"2024-04-16T21:13:15.896984+0000","last_peered":"2024-04-16T21:13:15.896984+0000","last_clean":"2024-04-16T21:13:15.896984+0000","last_became_active":"2024-04-16T19:47:33.981764+0000","last_became_peered":"2024-04-16T19:47:33.981764+0000","last_unstale":"2024-04-16T21:13:15.896984+0000","last_undegraded":"2024-04-16T21:13:15.896984+0000","last_fullsized":"2024-04-16T21:13:15.896984+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:15.896907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:15.896907+0000","last_clean_scrub_stamp":"2024-04-16T21:13:15.896907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:30:40.228145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.8","version":"52'144","reported_seq":1383,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:09.365421+0000","last_change":"2024-04-16T21:13:09.365467+0000","last_active":"2024-04-16T21:13:09.365421+0000","last_peered":"2024-04-16T21:13:09.365421+0000","last_clean":"2024-04-16T21:13:09.365421+0000","last_became_active":"2024-04-16T19:25:49.521046+0000","last_became_peered":"2024-04-16T19:25:49.521046+0000","last_unstale":"2024-04-16T21:13:09.365421+0000","last_undegraded":"2024-04-16T21:13:09.365421+0000","last_fullsized":"2024-04-16T21:13:09.365421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'144","last_scrub_stamp":"2024-04-16T21:13:09.365335+0000","last_deep_scrub":"52'144","last_deep_scrub_stamp":"2024-04-16T21:13:09.365335+0000","last_clean_scrub_stamp":"2024-04-16T21:13:09.365335+0000","objects_scrubbed":38,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:02:22.529312+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2036,"num_read_kb":1990,"num_write":141,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7840,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.320686+0000","last_change":"2024-04-16T21:12:28.320708+0000","last_active":"2024-04-16T21:12:28.320686+0000","last_peered":"2024-04-16T21:12:28.320686+0000","last_clean":"2024-04-16T21:12:28.320686+0000","last_became_active":"2024-04-16T19:25:41.511298+0000","last_became_peered":"2024-04-16T19:25:41.511298+0000","last_unstale":"2024-04-16T21:12:28.320686+0000","last_undegraded":"2024-04-16T21:12:28.320686+0000","last_fullsized":"2024-04-16T21:12:28.320686+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.320608+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.320608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:14.618412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"52'6","reported_seq":110,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.781379+0000","last_change":"2024-04-16T21:11:54.781414+0000","last_active":"2024-04-16T21:11:54.781379+0000","last_peered":"2024-04-16T21:11:54.781379+0000","last_clean":"2024-04-16T21:11:54.781379+0000","last_became_active":"2024-04-16T19:25:15.483215+0000","last_became_peered":"2024-04-16T19:25:15.483215+0000","last_unstale":"2024-04-16T21:11:54.781379+0000","last_undegraded":"2024-04-16T21:11:54.781379+0000","last_fullsized":"2024-04-16T21:11:54.781379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:11:54.781318+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.781318+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.682337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"52'706","reported_seq":2170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.804839+0000","last_change":"2024-04-16T21:11:57.804839+0000","last_active":"2024-04-16T21:11:57.804839+0000","last_peered":"2024-04-16T21:11:57.804839+0000","last_clean":"2024-04-16T21:11:57.804839+0000","last_became_active":"2024-04-16T19:25:13.475423+0000","last_became_peered":"2024-04-16T19:25:13.475423+0000","last_unstale":"2024-04-16T21:11:57.804839+0000","last_undegraded":"2024-04-16T21:11:57.804839+0000","last_fullsized":"2024-04-16T21:11:57.804839+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:57.804739+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.804739+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:38:55.231266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"52'264","reported_seq":496,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.746078+0000","last_change":"2024-04-16T21:11:46.746078+0000","last_active":"2024-04-16T21:11:46.746078+0000","last_peered":"2024-04-16T21:11:46.746078+0000","last_clean":"2024-04-16T21:11:46.746078+0000","last_became_active":"2024-04-16T19:25:11.465701+0000","last_became_peered":"2024-04-16T19:25:11.465701+0000","last_unstale":"2024-04-16T21:11:46.746078+0000","last_undegraded":"2024-04-16T21:11:46.746078+0000","last_fullsized":"2024-04-16T21:11:46.746078+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:46.745946+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.745946+0000","objects_scrubbed":17,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:58:43.933138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":359,"num_read_kb":351,"num_write":335,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1751,"num_omap_keys":8,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"52'38","reported_seq":267,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.693339+0000","last_change":"2024-04-16T21:11:52.693388+0000","last_active":"2024-04-16T21:11:52.693339+0000","last_peered":"2024-04-16T21:11:52.693339+0000","last_clean":"2024-04-16T21:11:52.693339+0000","last_became_active":"2024-04-16T19:25:09.469287+0000","last_became_peered":"2024-04-16T19:25:09.469287+0000","last_unstale":"2024-04-16T21:11:52.693339+0000","last_undegraded":"2024-04-16T21:11:52.693339+0000","last_fullsized":"2024-04-16T21:11:52.693339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'38","last_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_deep_scrub":"52'38","last_deep_scrub_stamp":"2024-04-16T21:11:52.693273+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.693273+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:15.884668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":206,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.125718+0000","last_change":"2024-04-16T21:12:43.125718+0000","last_active":"2024-04-16T21:12:43.125718+0000","last_peered":"2024-04-16T21:12:43.125718+0000","last_clean":"2024-04-16T21:12:43.125718+0000","last_became_active":"2024-04-16T19:24:35.401694+0000","last_became_peered":"2024-04-16T19:24:35.401694+0000","last_unstale":"2024-04-16T21:12:43.125718+0000","last_undegraded":"2024-04-16T21:12:43.125718+0000","last_fullsized":"2024-04-16T21:12:43.125718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.125631+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.125631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:49.363279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.901371+0000","last_change":"2024-04-16T21:12:52.901371+0000","last_active":"2024-04-16T21:12:52.901371+0000","last_peered":"2024-04-16T21:12:52.901371+0000","last_clean":"2024-04-16T21:12:52.901371+0000","last_became_active":"2024-04-16T19:24:33.388767+0000","last_became_peered":"2024-04-16T19:24:33.388767+0000","last_unstale":"2024-04-16T21:12:52.901371+0000","last_undegraded":"2024-04-16T21:12:52.901371+0000","last_fullsized":"2024-04-16T21:12:52.901371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.901289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.901289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.901289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:17:59.588023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:54.102912+0000","last_change":"2024-04-16T21:12:54.102935+0000","last_active":"2024-04-16T21:12:54.102912+0000","last_peered":"2024-04-16T21:12:54.102912+0000","last_clean":"2024-04-16T21:12:54.102912+0000","last_became_active":"2024-04-16T19:23:47.184005+0000","last_became_peered":"2024-04-16T19:23:47.184005+0000","last_unstale":"2024-04-16T21:12:54.102912+0000","last_undegraded":"2024-04-16T21:12:54.102912+0000","last_fullsized":"2024-04-16T21:12:54.102912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:54.102847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:54.102847+0000","last_clean_scrub_stamp":"2024-04-16T21:12:54.102847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:25.807640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.378818+0000","last_change":"2024-04-16T21:12:43.378818+0000","last_active":"2024-04-16T21:12:43.378818+0000","last_peered":"2024-04-16T21:12:43.378818+0000","last_clean":"2024-04-16T21:12:43.378818+0000","last_became_active":"2024-04-16T19:24:45.414001+0000","last_became_peered":"2024-04-16T19:24:45.414001+0000","last_unstale":"2024-04-16T21:12:43.378818+0000","last_undegraded":"2024-04-16T21:12:43.378818+0000","last_fullsized":"2024-04-16T21:12:43.378818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.378722+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.378722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:20.670430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"52'156","reported_seq":367,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.280951+0000","last_change":"2024-04-16T21:12:37.280951+0000","last_active":"2024-04-16T21:12:37.280951+0000","last_peered":"2024-04-16T21:12:37.280951+0000","last_clean":"2024-04-16T21:12:37.280951+0000","last_became_active":"2024-04-16T19:24:47.421796+0000","last_became_peered":"2024-04-16T19:24:47.421796+0000","last_unstale":"2024-04-16T21:12:37.280951+0000","last_undegraded":"2024-04-16T21:12:37.280951+0000","last_fullsized":"2024-04-16T21:12:37.280951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'156","last_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_deep_scrub":"52'156","last_deep_scrub_stamp":"2024-04-16T21:12:37.280808+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.280808+0000","objects_scrubbed":8,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:57:22.038566+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":232,"num_read_kb":220,"num_write":148,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.613002+0000","last_change":"2024-04-16T21:12:26.613002+0000","last_active":"2024-04-16T21:12:26.613002+0000","last_peered":"2024-04-16T21:12:26.613002+0000","last_clean":"2024-04-16T21:12:26.613002+0000","last_became_active":"2024-04-16T19:24:49.425118+0000","last_became_peered":"2024-04-16T19:24:49.425118+0000","last_unstale":"2024-04-16T21:12:26.613002+0000","last_undegraded":"2024-04-16T21:12:26.613002+0000","last_fullsized":"2024-04-16T21:12:26.613002+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:26.612912+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.612912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:44:38.407643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.345418+0000","last_change":"2024-04-16T21:12:01.345418+0000","last_active":"2024-04-16T21:12:01.345418+0000","last_peered":"2024-04-16T21:12:01.345418+0000","last_clean":"2024-04-16T21:12:01.345418+0000","last_became_active":"2024-04-16T19:24:51.571966+0000","last_became_peered":"2024-04-16T19:24:51.571966+0000","last_unstale":"2024-04-16T21:12:01.345418+0000","last_undegraded":"2024-04-16T21:12:01.345418+0000","last_fullsized":"2024-04-16T21:12:01.345418+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.345289+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.345289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:22.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.909918+0000","last_change":"2024-04-16T21:12:24.909962+0000","last_active":"2024-04-16T21:12:24.909918+0000","last_peered":"2024-04-16T21:12:24.909918+0000","last_clean":"2024-04-16T21:12:24.909918+0000","last_became_active":"2024-04-16T19:47:33.985197+0000","last_became_peered":"2024-04-16T19:47:33.985197+0000","last_unstale":"2024-04-16T21:12:24.909918+0000","last_undegraded":"2024-04-16T21:12:24.909918+0000","last_fullsized":"2024-04-16T21:12:24.909918+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:24.909836+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.909836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:07:30.648976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"52'194","reported_seq":2060,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.906040+0000","last_change":"2024-04-16T21:12:41.906084+0000","last_active":"2024-04-16T21:12:41.906040+0000","last_peered":"2024-04-16T21:12:41.906040+0000","last_clean":"2024-04-16T21:12:41.906040+0000","last_became_active":"2024-04-16T19:25:49.510532+0000","last_became_peered":"2024-04-16T19:25:49.510532+0000","last_unstale":"2024-04-16T21:12:41.906040+0000","last_undegraded":"2024-04-16T21:12:41.906040+0000","last_fullsized":"2024-04-16T21:12:41.906040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'194","last_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_deep_scrub":"52'194","last_deep_scrub_stamp":"2024-04-16T21:12:41.905983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.905983+0000","objects_scrubbed":47,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:49:44.692749+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2995,"num_read_kb":3006,"num_write":216,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12654,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.836131+0000","last_change":"2024-04-16T21:12:08.836131+0000","last_active":"2024-04-16T21:12:08.836131+0000","last_peered":"2024-04-16T21:12:08.836131+0000","last_clean":"2024-04-16T21:12:08.836131+0000","last_became_active":"2024-04-16T19:25:41.508462+0000","last_became_peered":"2024-04-16T19:25:41.508462+0000","last_unstale":"2024-04-16T21:12:08.836131+0000","last_undegraded":"2024-04-16T21:12:08.836131+0000","last_fullsized":"2024-04-16T21:12:08.836131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.836015+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.836015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:33.006060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"52'30","reported_seq":156,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.417277+0000","last_change":"2024-04-16T21:12:16.417277+0000","last_active":"2024-04-16T21:12:16.417277+0000","last_peered":"2024-04-16T21:12:16.417277+0000","last_clean":"2024-04-16T21:12:16.417277+0000","last_became_active":"2024-04-16T19:25:15.486936+0000","last_became_peered":"2024-04-16T19:25:15.486936+0000","last_unstale":"2024-04-16T21:12:16.417277+0000","last_undegraded":"2024-04-16T21:12:16.417277+0000","last_fullsized":"2024-04-16T21:12:16.417277+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'30","last_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_deep_scrub":"52'30","last_deep_scrub_stamp":"2024-04-16T21:12:16.417171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.417171+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:08.090860+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":135,"num_read_kb":105,"num_write":79,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.802585+0000","last_change":"2024-04-16T21:12:02.802585+0000","last_active":"2024-04-16T21:12:02.802585+0000","last_peered":"2024-04-16T21:12:02.802585+0000","last_clean":"2024-04-16T21:12:02.802585+0000","last_became_active":"2024-04-16T19:25:13.475539+0000","last_became_peered":"2024-04-16T19:25:13.475539+0000","last_unstale":"2024-04-16T21:12:02.802585+0000","last_undegraded":"2024-04-16T21:12:02.802585+0000","last_fullsized":"2024-04-16T21:12:02.802585+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.802497+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.802497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:45:44.917953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"52'739","reported_seq":1610,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.752492+0000","last_change":"2024-04-16T21:11:47.752492+0000","last_active":"2024-04-16T21:11:47.752492+0000","last_peered":"2024-04-16T21:11:47.752492+0000","last_clean":"2024-04-16T21:11:47.752492+0000","last_became_active":"2024-04-16T19:25:11.465204+0000","last_became_peered":"2024-04-16T19:25:11.465204+0000","last_unstale":"2024-04-16T21:11:47.752492+0000","last_undegraded":"2024-04-16T21:11:47.752492+0000","last_fullsized":"2024-04-16T21:11:47.752492+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'739","last_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_deep_scrub":"52'739","last_deep_scrub_stamp":"2024-04-16T21:11:47.752406+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.752406+0000","objects_scrubbed":25,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:14.225249+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48207,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":2302,"num_read_kb":1973,"num_write":1125,"num_write_kb":696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6099,"num_omap_keys":28,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"52'12","reported_seq":284,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.362519+0000","last_change":"2024-04-16T21:11:45.362519+0000","last_active":"2024-04-16T21:11:45.362519+0000","last_peered":"2024-04-16T21:11:45.362519+0000","last_clean":"2024-04-16T21:11:45.362519+0000","last_became_active":"2024-04-16T19:25:09.462626+0000","last_became_peered":"2024-04-16T19:25:09.462626+0000","last_unstale":"2024-04-16T21:11:45.362519+0000","last_undegraded":"2024-04-16T21:11:45.362519+0000","last_fullsized":"2024-04-16T21:11:45.362519+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:45.362399+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.362399+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:08.553795+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":422,"num_read_kb":277,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.447089+0000","last_change":"2024-04-16T21:12:56.447089+0000","last_active":"2024-04-16T21:12:56.447089+0000","last_peered":"2024-04-16T21:12:56.447089+0000","last_clean":"2024-04-16T21:12:56.447089+0000","last_became_active":"2024-04-16T19:24:35.401145+0000","last_became_peered":"2024-04-16T19:24:35.401145+0000","last_unstale":"2024-04-16T21:12:56.447089+0000","last_undegraded":"2024-04-16T21:12:56.447089+0000","last_fullsized":"2024-04-16T21:12:56.447089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.446999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.446999+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.446999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:20.399918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.123738+0000","last_change":"2024-04-16T21:12:56.123791+0000","last_active":"2024-04-16T21:12:56.123738+0000","last_peered":"2024-04-16T21:12:56.123738+0000","last_clean":"2024-04-16T21:12:56.123738+0000","last_became_active":"2024-04-16T19:24:33.384782+0000","last_became_peered":"2024-04-16T19:24:33.384782+0000","last_unstale":"2024-04-16T21:12:56.123738+0000","last_undegraded":"2024-04-16T21:12:56.123738+0000","last_fullsized":"2024-04-16T21:12:56.123738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.123634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.123634+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.123634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:50.704912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:05.243192+0000","last_change":"2024-04-16T21:13:05.243215+0000","last_active":"2024-04-16T21:13:05.243192+0000","last_peered":"2024-04-16T21:13:05.243192+0000","last_clean":"2024-04-16T21:13:05.243192+0000","last_became_active":"2024-04-16T19:23:47.184014+0000","last_became_peered":"2024-04-16T19:23:47.184014+0000","last_unstale":"2024-04-16T21:13:05.243192+0000","last_undegraded":"2024-04-16T21:13:05.243192+0000","last_fullsized":"2024-04-16T21:13:05.243192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:05.243128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:05.243128+0000","last_clean_scrub_stamp":"2024-04-16T21:13:05.243128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:59:23.144772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.786111+0000","last_change":"2024-04-16T21:12:46.786111+0000","last_active":"2024-04-16T21:12:46.786111+0000","last_peered":"2024-04-16T21:12:46.786111+0000","last_clean":"2024-04-16T21:12:46.786111+0000","last_became_active":"2024-04-16T19:24:45.413997+0000","last_became_peered":"2024-04-16T19:24:45.413997+0000","last_unstale":"2024-04-16T21:12:46.786111+0000","last_undegraded":"2024-04-16T21:12:46.786111+0000","last_fullsized":"2024-04-16T21:12:46.786111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.786030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.786030+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.786030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:54:27.738032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"52'161","reported_seq":363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.967756+0000","last_change":"2024-04-16T21:12:27.967756+0000","last_active":"2024-04-16T21:12:27.967756+0000","last_peered":"2024-04-16T21:12:27.967756+0000","last_clean":"2024-04-16T21:12:27.967756+0000","last_became_active":"2024-04-16T19:24:47.423818+0000","last_became_peered":"2024-04-16T19:24:47.423818+0000","last_unstale":"2024-04-16T21:12:27.967756+0000","last_undegraded":"2024-04-16T21:12:27.967756+0000","last_fullsized":"2024-04-16T21:12:27.967756+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'161","last_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_deep_scrub":"52'161","last_deep_scrub_stamp":"2024-04-16T21:12:27.967658+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.967658+0000","objects_scrubbed":7,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:06.401014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":224,"num_write":154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"52'13","reported_seq":1328,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.766675+0000","last_change":"2024-04-16T21:12:27.766675+0000","last_active":"2024-04-16T21:12:27.766675+0000","last_peered":"2024-04-16T21:12:27.766675+0000","last_clean":"2024-04-16T21:12:27.766675+0000","last_became_active":"2024-04-16T19:24:49.433630+0000","last_became_peered":"2024-04-16T19:24:49.433630+0000","last_unstale":"2024-04-16T21:12:27.766675+0000","last_undegraded":"2024-04-16T21:12:27.766675+0000","last_fullsized":"2024-04-16T21:12:27.766675+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:27.766582+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.766582+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:37:36.260814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.393341+0000","last_change":"2024-04-16T21:12:00.393341+0000","last_active":"2024-04-16T21:12:00.393341+0000","last_peered":"2024-04-16T21:12:00.393341+0000","last_clean":"2024-04-16T21:12:00.393341+0000","last_became_active":"2024-04-16T19:24:51.572173+0000","last_became_peered":"2024-04-16T19:24:51.572173+0000","last_unstale":"2024-04-16T21:12:00.393341+0000","last_undegraded":"2024-04-16T21:12:00.393341+0000","last_fullsized":"2024-04-16T21:12:00.393341+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.393227+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.393227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:23:57.237156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":63,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.796382+0000","last_change":"2024-04-16T21:12:57.796382+0000","last_active":"2024-04-16T21:12:57.796382+0000","last_peered":"2024-04-16T21:12:57.796382+0000","last_clean":"2024-04-16T21:12:57.796382+0000","last_became_active":"2024-04-16T19:47:33.989191+0000","last_became_peered":"2024-04-16T19:47:33.989191+0000","last_unstale":"2024-04-16T21:12:57.796382+0000","last_undegraded":"2024-04-16T21:12:57.796382+0000","last_fullsized":"2024-04-16T21:12:57.796382+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.796292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.796292+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.796292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:12.374830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"52'148","reported_seq":1569,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.332144+0000","last_change":"2024-04-16T21:12:36.332178+0000","last_active":"2024-04-16T21:12:36.332144+0000","last_peered":"2024-04-16T21:12:36.332144+0000","last_clean":"2024-04-16T21:12:36.332144+0000","last_became_active":"2024-04-16T19:25:49.514975+0000","last_became_peered":"2024-04-16T19:25:49.514975+0000","last_unstale":"2024-04-16T21:12:36.332144+0000","last_undegraded":"2024-04-16T21:12:36.332144+0000","last_fullsized":"2024-04-16T21:12:36.332144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'148","last_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_deep_scrub":"52'148","last_deep_scrub_stamp":"2024-04-16T21:12:36.332084+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.332084+0000","objects_scrubbed":32,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:56:33.307525+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2416,"num_read_kb":2419,"num_write":172,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11265,"num_omap_keys":45,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.939107+0000","last_change":"2024-04-16T21:12:19.939131+0000","last_active":"2024-04-16T21:12:19.939107+0000","last_peered":"2024-04-16T21:12:19.939107+0000","last_clean":"2024-04-16T21:12:19.939107+0000","last_became_active":"2024-04-16T19:25:41.498233+0000","last_became_peered":"2024-04-16T19:25:41.498233+0000","last_unstale":"2024-04-16T21:12:19.939107+0000","last_undegraded":"2024-04-16T21:12:19.939107+0000","last_fullsized":"2024-04-16T21:12:19.939107+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:19.939043+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.939043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:23.022714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"52'4","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.763978+0000","last_change":"2024-04-16T21:11:55.764028+0000","last_active":"2024-04-16T21:11:55.763978+0000","last_peered":"2024-04-16T21:11:55.763978+0000","last_clean":"2024-04-16T21:11:55.763978+0000","last_became_active":"2024-04-16T19:25:15.483380+0000","last_became_peered":"2024-04-16T19:25:15.483380+0000","last_unstale":"2024-04-16T21:11:55.763978+0000","last_undegraded":"2024-04-16T21:11:55.763978+0000","last_fullsized":"2024-04-16T21:11:55.763978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:11:55.763887+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.763887+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.184478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":27,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.416929+0000","last_change":"2024-04-16T21:12:12.416959+0000","last_active":"2024-04-16T21:12:12.416929+0000","last_peered":"2024-04-16T21:12:12.416929+0000","last_clean":"2024-04-16T21:12:12.416929+0000","last_became_active":"2024-04-16T19:25:13.472960+0000","last_became_peered":"2024-04-16T19:25:13.472960+0000","last_unstale":"2024-04-16T21:12:12.416929+0000","last_undegraded":"2024-04-16T21:12:12.416929+0000","last_fullsized":"2024-04-16T21:12:12.416929+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.416865+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.416865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:15.500537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"52'287","reported_seq":554,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.658986+0000","last_change":"2024-04-16T21:11:54.658986+0000","last_active":"2024-04-16T21:11:54.658986+0000","last_peered":"2024-04-16T21:11:54.658986+0000","last_clean":"2024-04-16T21:11:54.658986+0000","last_became_active":"2024-04-16T19:25:11.463241+0000","last_became_peered":"2024-04-16T19:25:11.463241+0000","last_unstale":"2024-04-16T21:11:54.658986+0000","last_undegraded":"2024-04-16T21:11:54.658986+0000","last_fullsized":"2024-04-16T21:11:54.658986+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'287","last_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_deep_scrub":"52'287","last_deep_scrub_stamp":"2024-04-16T21:11:54.658842+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.658842+0000","objects_scrubbed":20,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:15:24.559862+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":419,"num_read_kb":402,"num_write":346,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6346,"num_omap_keys":29,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"52'22","reported_seq":214,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.414602+0000","last_change":"2024-04-16T21:11:51.414602+0000","last_active":"2024-04-16T21:11:51.414602+0000","last_peered":"2024-04-16T21:11:51.414602+0000","last_clean":"2024-04-16T21:11:51.414602+0000","last_became_active":"2024-04-16T19:25:09.461382+0000","last_became_peered":"2024-04-16T19:25:09.461382+0000","last_unstale":"2024-04-16T21:11:51.414602+0000","last_undegraded":"2024-04-16T21:11:51.414602+0000","last_fullsized":"2024-04-16T21:11:51.414602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:51.414471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.414471+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:11:12.853053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":228,"num_read_kb":159,"num_write":116,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.340615+0000","last_change":"2024-04-16T21:12:46.340615+0000","last_active":"2024-04-16T21:12:46.340615+0000","last_peered":"2024-04-16T21:12:46.340615+0000","last_clean":"2024-04-16T21:12:46.340615+0000","last_became_active":"2024-04-16T19:24:35.396473+0000","last_became_peered":"2024-04-16T19:24:35.396473+0000","last_unstale":"2024-04-16T21:12:46.340615+0000","last_undegraded":"2024-04-16T21:12:46.340615+0000","last_fullsized":"2024-04-16T21:12:46.340615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.340544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.340544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.340544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:58.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.719433+0000","last_change":"2024-04-16T21:12:43.719433+0000","last_active":"2024-04-16T21:12:43.719433+0000","last_peered":"2024-04-16T21:12:43.719433+0000","last_clean":"2024-04-16T21:12:43.719433+0000","last_became_active":"2024-04-16T19:24:33.385046+0000","last_became_peered":"2024-04-16T19:24:33.385046+0000","last_unstale":"2024-04-16T21:12:43.719433+0000","last_undegraded":"2024-04-16T21:12:43.719433+0000","last_fullsized":"2024-04-16T21:12:43.719433+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.719343+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.719343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:43.694308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.134053+0000","last_change":"2024-04-16T21:12:53.134077+0000","last_active":"2024-04-16T21:12:53.134053+0000","last_peered":"2024-04-16T21:12:53.134053+0000","last_clean":"2024-04-16T21:12:53.134053+0000","last_became_active":"2024-04-16T19:23:47.122516+0000","last_became_peered":"2024-04-16T19:23:47.122516+0000","last_unstale":"2024-04-16T21:12:53.134053+0000","last_undegraded":"2024-04-16T21:12:53.134053+0000","last_fullsized":"2024-04-16T21:12:53.134053+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.133983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.133983+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.133983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:22.123641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.211369+0000","last_change":"2024-04-16T21:12:42.211369+0000","last_active":"2024-04-16T21:12:42.211369+0000","last_peered":"2024-04-16T21:12:42.211369+0000","last_clean":"2024-04-16T21:12:42.211369+0000","last_became_active":"2024-04-16T19:24:45.413022+0000","last_became_peered":"2024-04-16T19:24:45.413022+0000","last_unstale":"2024-04-16T21:12:42.211369+0000","last_undegraded":"2024-04-16T21:12:42.211369+0000","last_fullsized":"2024-04-16T21:12:42.211369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.211237+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.211237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:13.916029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"52'115","reported_seq":295,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.353459+0000","last_change":"2024-04-16T21:12:35.353459+0000","last_active":"2024-04-16T21:12:35.353459+0000","last_peered":"2024-04-16T21:12:35.353459+0000","last_clean":"2024-04-16T21:12:35.353459+0000","last_became_active":"2024-04-16T19:24:47.417332+0000","last_became_peered":"2024-04-16T19:24:47.417332+0000","last_unstale":"2024-04-16T21:12:35.353459+0000","last_undegraded":"2024-04-16T21:12:35.353459+0000","last_fullsized":"2024-04-16T21:12:35.353459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:35.353361+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.353361+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:35.315569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"52'13","reported_seq":1326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.412664+0000","last_change":"2024-04-16T21:12:24.412664+0000","last_active":"2024-04-16T21:12:24.412664+0000","last_peered":"2024-04-16T21:12:24.412664+0000","last_clean":"2024-04-16T21:12:24.412664+0000","last_became_active":"2024-04-16T19:24:49.431725+0000","last_became_peered":"2024-04-16T19:24:49.431725+0000","last_unstale":"2024-04-16T21:12:24.412664+0000","last_undegraded":"2024-04-16T21:12:24.412664+0000","last_fullsized":"2024-04-16T21:12:24.412664+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:24.412571+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.412571+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:12.450226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.373240+0000","last_change":"2024-04-16T21:11:59.373240+0000","last_active":"2024-04-16T21:11:59.373240+0000","last_peered":"2024-04-16T21:11:59.373240+0000","last_clean":"2024-04-16T21:11:59.373240+0000","last_became_active":"2024-04-16T19:24:51.573106+0000","last_became_peered":"2024-04-16T19:24:51.573106+0000","last_unstale":"2024-04-16T21:11:59.373240+0000","last_undegraded":"2024-04-16T21:11:59.373240+0000","last_fullsized":"2024-04-16T21:11:59.373240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.373112+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.373112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:51:02.682886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.829001+0000","last_change":"2024-04-16T21:12:56.829025+0000","last_active":"2024-04-16T21:12:56.829001+0000","last_peered":"2024-04-16T21:12:56.829001+0000","last_clean":"2024-04-16T21:12:56.829001+0000","last_became_active":"2024-04-16T19:47:33.989062+0000","last_became_peered":"2024-04-16T19:47:33.989062+0000","last_unstale":"2024-04-16T21:12:56.829001+0000","last_undegraded":"2024-04-16T21:12:56.829001+0000","last_fullsized":"2024-04-16T21:12:56.829001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.828939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.828939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.828939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:23.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"52'155","reported_seq":1575,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:13.821380+0000","last_change":"2024-04-16T21:13:13.821437+0000","last_active":"2024-04-16T21:13:13.821380+0000","last_peered":"2024-04-16T21:13:13.821380+0000","last_clean":"2024-04-16T21:13:13.821380+0000","last_became_active":"2024-04-16T19:26:41.557190+0000","last_became_peered":"2024-04-16T19:26:41.557190+0000","last_unstale":"2024-04-16T21:13:13.821380+0000","last_undegraded":"2024-04-16T21:13:13.821380+0000","last_fullsized":"2024-04-16T21:13:13.821380+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'155","last_scrub_stamp":"2024-04-16T21:13:13.821315+0000","last_deep_scrub":"52'155","last_deep_scrub_stamp":"2024-04-16T21:13:13.821315+0000","last_clean_scrub_stamp":"2024-04-16T21:13:13.821315+0000","objects_scrubbed":43,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:06:31.790327+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2420,"num_read_kb":2382,"num_write":161,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11043,"num_omap_keys":45,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":90,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.846610+0000","last_change":"2024-04-16T21:12:00.846636+0000","last_active":"2024-04-16T21:12:00.846610+0000","last_peered":"2024-04-16T21:12:00.846610+0000","last_clean":"2024-04-16T21:12:00.846610+0000","last_became_active":"2024-04-16T19:25:41.501758+0000","last_became_peered":"2024-04-16T19:25:41.501758+0000","last_unstale":"2024-04-16T21:12:00.846610+0000","last_undegraded":"2024-04-16T21:12:00.846610+0000","last_fullsized":"2024-04-16T21:12:00.846610+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.846545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.846545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:40:04.784544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"52'15","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:56.794385+0000","last_change":"2024-04-16T21:11:56.794414+0000","last_active":"2024-04-16T21:11:56.794385+0000","last_peered":"2024-04-16T21:11:56.794385+0000","last_clean":"2024-04-16T21:11:56.794385+0000","last_became_active":"2024-04-16T19:25:15.485693+0000","last_became_peered":"2024-04-16T19:25:15.485693+0000","last_unstale":"2024-04-16T21:11:56.794385+0000","last_undegraded":"2024-04-16T21:11:56.794385+0000","last_fullsized":"2024-04-16T21:11:56.794385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:11:56.794323+0000","last_clean_scrub_stamp":"2024-04-16T21:11:56.794323+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:43:05.217325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1232,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":35,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"52'706","reported_seq":2091,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.681723+0000","last_change":"2024-04-16T21:11:49.681723+0000","last_active":"2024-04-16T21:11:49.681723+0000","last_peered":"2024-04-16T21:11:49.681723+0000","last_clean":"2024-04-16T21:11:49.681723+0000","last_became_active":"2024-04-16T19:25:13.471097+0000","last_became_peered":"2024-04-16T19:25:13.471097+0000","last_unstale":"2024-04-16T21:11:49.681723+0000","last_undegraded":"2024-04-16T21:11:49.681723+0000","last_fullsized":"2024-04-16T21:11:49.681723+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:11:49.681634+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.681634+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.077508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"52'335","reported_seq":630,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.410155+0000","last_change":"2024-04-16T21:11:50.410223+0000","last_active":"2024-04-16T21:11:50.410155+0000","last_peered":"2024-04-16T21:11:50.410155+0000","last_clean":"2024-04-16T21:11:50.410155+0000","last_became_active":"2024-04-16T19:25:11.461642+0000","last_became_peered":"2024-04-16T19:25:11.461642+0000","last_unstale":"2024-04-16T21:11:50.410155+0000","last_undegraded":"2024-04-16T21:11:50.410155+0000","last_fullsized":"2024-04-16T21:11:50.410155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'335","last_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_deep_scrub":"52'335","last_deep_scrub_stamp":"2024-04-16T21:11:50.410083+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.410083+0000","objects_scrubbed":17,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:56:14.580127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":472,"num_read_kb":457,"num_write":383,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"52'10","reported_seq":196,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.912515+0000","last_change":"2024-04-16T21:12:10.912515+0000","last_active":"2024-04-16T21:12:10.912515+0000","last_peered":"2024-04-16T21:12:10.912515+0000","last_clean":"2024-04-16T21:12:10.912515+0000","last_became_active":"2024-04-16T19:25:09.461019+0000","last_became_peered":"2024-04-16T19:25:09.461019+0000","last_unstale":"2024-04-16T21:12:10.912515+0000","last_undegraded":"2024-04-16T21:12:10.912515+0000","last_fullsized":"2024-04-16T21:12:10.912515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:10.912417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.912417+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.912619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":161,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.743600+0000","last_change":"2024-04-16T21:12:31.743600+0000","last_active":"2024-04-16T21:12:31.743600+0000","last_peered":"2024-04-16T21:12:31.743600+0000","last_clean":"2024-04-16T21:12:31.743600+0000","last_became_active":"2024-04-16T19:24:35.401144+0000","last_became_peered":"2024-04-16T19:24:35.401144+0000","last_unstale":"2024-04-16T21:12:31.743600+0000","last_undegraded":"2024-04-16T21:12:31.743600+0000","last_fullsized":"2024-04-16T21:12:31.743600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.743511+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.743511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:22:22.862472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.495161+0000","last_change":"2024-04-16T21:13:00.495206+0000","last_active":"2024-04-16T21:13:00.495161+0000","last_peered":"2024-04-16T21:13:00.495161+0000","last_clean":"2024-04-16T21:13:00.495161+0000","last_became_active":"2024-04-16T19:24:33.381136+0000","last_became_peered":"2024-04-16T19:24:33.381136+0000","last_unstale":"2024-04-16T21:13:00.495161+0000","last_undegraded":"2024-04-16T21:13:00.495161+0000","last_fullsized":"2024-04-16T21:13:00.495161+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.495078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.495078+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.495078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:29:35.162947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.220313+0000","last_change":"2024-04-16T21:13:04.220348+0000","last_active":"2024-04-16T21:13:04.220313+0000","last_peered":"2024-04-16T21:13:04.220313+0000","last_clean":"2024-04-16T21:13:04.220313+0000","last_became_active":"2024-04-16T19:23:47.174900+0000","last_became_peered":"2024-04-16T19:23:47.174900+0000","last_unstale":"2024-04-16T21:13:04.220313+0000","last_undegraded":"2024-04-16T21:13:04.220313+0000","last_fullsized":"2024-04-16T21:13:04.220313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:04.220225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:04.220225+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.220225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:13:55.396780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.901885+0000","last_change":"2024-04-16T21:12:28.901885+0000","last_active":"2024-04-16T21:12:28.901885+0000","last_peered":"2024-04-16T21:12:28.901885+0000","last_clean":"2024-04-16T21:12:28.901885+0000","last_became_active":"2024-04-16T19:24:45.408808+0000","last_became_peered":"2024-04-16T19:24:45.408808+0000","last_unstale":"2024-04-16T21:12:28.901885+0000","last_undegraded":"2024-04-16T21:12:28.901885+0000","last_fullsized":"2024-04-16T21:12:28.901885+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:28.901794+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.901794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:33.940739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"52'142","reported_seq":349,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.606351+0000","last_change":"2024-04-16T21:12:27.606351+0000","last_active":"2024-04-16T21:12:27.606351+0000","last_peered":"2024-04-16T21:12:27.606351+0000","last_clean":"2024-04-16T21:12:27.606351+0000","last_became_active":"2024-04-16T19:24:47.425442+0000","last_became_peered":"2024-04-16T19:24:47.425442+0000","last_unstale":"2024-04-16T21:12:27.606351+0000","last_undegraded":"2024-04-16T21:12:27.606351+0000","last_fullsized":"2024-04-16T21:12:27.606351+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'142","last_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_deep_scrub":"52'142","last_deep_scrub_stamp":"2024-04-16T21:12:27.606253+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.606253+0000","objects_scrubbed":8,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:18.166355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":216,"num_read_kb":202,"num_write":134,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.903439+0000","last_change":"2024-04-16T21:12:17.903439+0000","last_active":"2024-04-16T21:12:17.903439+0000","last_peered":"2024-04-16T21:12:17.903439+0000","last_clean":"2024-04-16T21:12:17.903439+0000","last_became_active":"2024-04-16T19:24:49.434951+0000","last_became_peered":"2024-04-16T19:24:49.434951+0000","last_unstale":"2024-04-16T21:12:17.903439+0000","last_undegraded":"2024-04-16T21:12:17.903439+0000","last_fullsized":"2024-04-16T21:12:17.903439+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.903352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.903352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:24.923396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.864551+0000","last_change":"2024-04-16T21:12:11.864551+0000","last_active":"2024-04-16T21:12:11.864551+0000","last_peered":"2024-04-16T21:12:11.864551+0000","last_clean":"2024-04-16T21:12:11.864551+0000","last_became_active":"2024-04-16T19:24:51.437780+0000","last_became_peered":"2024-04-16T19:24:51.437780+0000","last_unstale":"2024-04-16T21:12:11.864551+0000","last_undegraded":"2024-04-16T21:12:11.864551+0000","last_fullsized":"2024-04-16T21:12:11.864551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.864468+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.864468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:37.984544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:14.412872+0000","last_change":"2024-04-16T21:13:14.412894+0000","last_active":"2024-04-16T21:13:14.412872+0000","last_peered":"2024-04-16T21:13:14.412872+0000","last_clean":"2024-04-16T21:13:14.412872+0000","last_became_active":"2024-04-16T19:47:33.980408+0000","last_became_peered":"2024-04-16T19:47:33.980408+0000","last_unstale":"2024-04-16T21:13:14.412872+0000","last_undegraded":"2024-04-16T21:13:14.412872+0000","last_fullsized":"2024-04-16T21:13:14.412872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:14.412802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:14.412802+0000","last_clean_scrub_stamp":"2024-04-16T21:13:14.412802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:25:45.113731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.19","version":"52'152","reported_seq":1619,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:11.220663+0000","last_change":"2024-04-16T21:13:11.220709+0000","last_active":"2024-04-16T21:13:11.220663+0000","last_peered":"2024-04-16T21:13:11.220663+0000","last_clean":"2024-04-16T21:13:11.220663+0000","last_became_active":"2024-04-16T19:25:49.520027+0000","last_became_peered":"2024-04-16T19:25:49.520027+0000","last_unstale":"2024-04-16T21:13:11.220663+0000","last_undegraded":"2024-04-16T21:13:11.220663+0000","last_fullsized":"2024-04-16T21:13:11.220663+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'152","last_scrub_stamp":"2024-04-16T21:13:11.220582+0000","last_deep_scrub":"52'152","last_deep_scrub_stamp":"2024-04-16T21:13:11.220582+0000","last_clean_scrub_stamp":"2024-04-16T21:13:11.220582+0000","objects_scrubbed":39,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:09:15.420303+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2416,"num_read_kb":2438,"num_write":149,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6953,"num_omap_keys":28,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.869029+0000","last_change":"2024-04-16T21:12:06.869029+0000","last_active":"2024-04-16T21:12:06.869029+0000","last_peered":"2024-04-16T21:12:06.869029+0000","last_clean":"2024-04-16T21:12:06.869029+0000","last_became_active":"2024-04-16T19:25:41.506060+0000","last_became_peered":"2024-04-16T19:25:41.506060+0000","last_unstale":"2024-04-16T21:12:06.869029+0000","last_undegraded":"2024-04-16T21:12:06.869029+0000","last_fullsized":"2024-04-16T21:12:06.869029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.868938+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.868938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:10.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"52'8","reported_seq":107,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.650063+0000","last_change":"2024-04-16T21:12:10.650063+0000","last_active":"2024-04-16T21:12:10.650063+0000","last_peered":"2024-04-16T21:12:10.650063+0000","last_clean":"2024-04-16T21:12:10.650063+0000","last_became_active":"2024-04-16T19:25:15.480210+0000","last_became_peered":"2024-04-16T19:25:15.480210+0000","last_unstale":"2024-04-16T21:12:10.650063+0000","last_undegraded":"2024-04-16T21:12:10.650063+0000","last_fullsized":"2024-04-16T21:12:10.650063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:10.649972+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.649972+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:31.650174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":947,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":11,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.662915+0000","last_change":"2024-04-16T21:11:53.662915+0000","last_active":"2024-04-16T21:11:53.662915+0000","last_peered":"2024-04-16T21:11:53.662915+0000","last_clean":"2024-04-16T21:11:53.662915+0000","last_became_active":"2024-04-16T19:25:13.468385+0000","last_became_peered":"2024-04-16T19:25:13.468385+0000","last_unstale":"2024-04-16T21:11:53.662915+0000","last_undegraded":"2024-04-16T21:11:53.662915+0000","last_fullsized":"2024-04-16T21:11:53.662915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:53.662831+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.662831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:24:53.840637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1d","version":"52'317","reported_seq":613,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.662773+0000","last_change":"2024-04-16T21:11:52.662773+0000","last_active":"2024-04-16T21:11:52.662773+0000","last_peered":"2024-04-16T21:11:52.662773+0000","last_clean":"2024-04-16T21:11:52.662773+0000","last_became_active":"2024-04-16T19:25:11.464039+0000","last_became_peered":"2024-04-16T19:25:11.464039+0000","last_unstale":"2024-04-16T21:11:52.662773+0000","last_undegraded":"2024-04-16T21:11:52.662773+0000","last_fullsized":"2024-04-16T21:11:52.662773+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'317","last_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_deep_scrub":"52'317","last_deep_scrub_stamp":"2024-04-16T21:11:52.662688+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.662688+0000","objects_scrubbed":25,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:03:07.759388+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":422,"num_read_kb":411,"num_write":428,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3966,"num_omap_keys":18,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"52'20","reported_seq":253,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.170966+0000","last_change":"2024-04-16T21:11:53.171045+0000","last_active":"2024-04-16T21:11:53.170966+0000","last_peered":"2024-04-16T21:11:53.170966+0000","last_clean":"2024-04-16T21:11:53.170966+0000","last_became_active":"2024-04-16T19:25:09.460154+0000","last_became_peered":"2024-04-16T19:25:09.460154+0000","last_unstale":"2024-04-16T21:11:53.170966+0000","last_undegraded":"2024-04-16T21:11:53.170966+0000","last_fullsized":"2024-04-16T21:11:53.170966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:53.170891+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.170891+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:46:56.498676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":314,"num_read_kb":212,"num_write":104,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":117,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.475030+0000","last_change":"2024-04-16T21:12:09.475030+0000","last_active":"2024-04-16T21:12:09.475030+0000","last_peered":"2024-04-16T21:12:09.475030+0000","last_clean":"2024-04-16T21:12:09.475030+0000","last_became_active":"2024-04-16T19:24:51.434318+0000","last_became_peered":"2024-04-16T19:24:51.434318+0000","last_unstale":"2024-04-16T21:12:09.475030+0000","last_undegraded":"2024-04-16T21:12:09.475030+0000","last_fullsized":"2024-04-16T21:12:09.475030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.474907+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.474907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:12.558579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.703966+0000","last_change":"2024-04-16T21:12:40.703987+0000","last_active":"2024-04-16T21:12:40.703966+0000","last_peered":"2024-04-16T21:12:40.703966+0000","last_clean":"2024-04-16T21:12:40.703966+0000","last_became_active":"2024-04-16T19:24:35.398560+0000","last_became_peered":"2024-04-16T19:24:35.398560+0000","last_unstale":"2024-04-16T21:12:40.703966+0000","last_undegraded":"2024-04-16T21:12:40.703966+0000","last_fullsized":"2024-04-16T21:12:40.703966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:40.703892+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.703892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:52.306563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.890754+0000","last_change":"2024-04-16T21:12:07.890754+0000","last_active":"2024-04-16T21:12:07.890754+0000","last_peered":"2024-04-16T21:12:07.890754+0000","last_clean":"2024-04-16T21:12:07.890754+0000","last_became_active":"2024-04-16T19:25:41.504190+0000","last_became_peered":"2024-04-16T19:25:41.504190+0000","last_unstale":"2024-04-16T21:12:07.890754+0000","last_undegraded":"2024-04-16T21:12:07.890754+0000","last_fullsized":"2024-04-16T21:12:07.890754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.890665+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.890665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:11.218450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"52'5","reported_seq":755,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.087803+0000","last_change":"2024-04-16T21:12:49.087850+0000","last_active":"2024-04-16T21:12:49.087803+0000","last_peered":"2024-04-16T21:12:49.087803+0000","last_clean":"2024-04-16T21:12:49.087803+0000","last_became_active":"2024-04-16T19:24:45.410346+0000","last_became_peered":"2024-04-16T19:24:45.410346+0000","last_unstale":"2024-04-16T21:12:49.087803+0000","last_undegraded":"2024-04-16T21:12:49.087803+0000","last_fullsized":"2024-04-16T21:12:49.087803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'5","last_scrub_stamp":"2024-04-16T21:12:49.087709+0000","last_deep_scrub":"52'5","last_deep_scrub_stamp":"2024-04-16T21:12:49.087709+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.087709+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:29:36.342884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1211,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":867,"num_read_kb":1338,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.782050+0000","last_change":"2024-04-16T21:12:40.782083+0000","last_active":"2024-04-16T21:12:40.782050+0000","last_peered":"2024-04-16T21:12:40.782050+0000","last_clean":"2024-04-16T21:12:40.782050+0000","last_became_active":"2024-04-16T19:24:47.421592+0000","last_became_peered":"2024-04-16T19:24:47.421592+0000","last_unstale":"2024-04-16T21:12:40.782050+0000","last_undegraded":"2024-04-16T21:12:40.782050+0000","last_fullsized":"2024-04-16T21:12:40.782050+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:40.781989+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.781989+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:34.970663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.902196+0000","last_change":"2024-04-16T21:12:21.902219+0000","last_active":"2024-04-16T21:12:21.902196+0000","last_peered":"2024-04-16T21:12:21.902196+0000","last_clean":"2024-04-16T21:12:21.902196+0000","last_became_active":"2024-04-16T19:24:49.434077+0000","last_became_peered":"2024-04-16T19:24:49.434077+0000","last_unstale":"2024-04-16T21:12:21.902196+0000","last_undegraded":"2024-04-16T21:12:21.902196+0000","last_fullsized":"2024-04-16T21:12:21.902196+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:21.902128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.902128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:26.941072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.849129+0000","last_change":"2024-04-16T21:13:01.849151+0000","last_active":"2024-04-16T21:13:01.849129+0000","last_peered":"2024-04-16T21:13:01.849129+0000","last_clean":"2024-04-16T21:13:01.849129+0000","last_became_active":"2024-04-16T19:47:33.984064+0000","last_became_peered":"2024-04-16T19:47:33.984064+0000","last_unstale":"2024-04-16T21:13:01.849129+0000","last_undegraded":"2024-04-16T21:13:01.849129+0000","last_fullsized":"2024-04-16T21:13:01.849129+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.849065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.849065+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.849065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:03:36.108179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"52'159","reported_seq":1763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.889853+0000","last_change":"2024-04-16T21:12:18.889893+0000","last_active":"2024-04-16T21:12:18.889853+0000","last_peered":"2024-04-16T21:12:18.889853+0000","last_clean":"2024-04-16T21:12:18.889853+0000","last_became_active":"2024-04-16T19:25:49.518261+0000","last_became_peered":"2024-04-16T19:25:49.518261+0000","last_unstale":"2024-04-16T21:12:18.889853+0000","last_undegraded":"2024-04-16T21:12:18.889853+0000","last_fullsized":"2024-04-16T21:12:18.889853+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:12:18.889790+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.889790+0000","objects_scrubbed":41,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:35.275723+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":2624,"num_read_kb":2580,"num_write":177,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12681,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.752964+0000","last_change":"2024-04-16T21:12:01.752964+0000","last_active":"2024-04-16T21:12:01.752964+0000","last_peered":"2024-04-16T21:12:01.752964+0000","last_clean":"2024-04-16T21:12:01.752964+0000","last_became_active":"2024-04-16T19:25:41.505518+0000","last_became_peered":"2024-04-16T19:25:41.505518+0000","last_unstale":"2024-04-16T21:12:01.752964+0000","last_undegraded":"2024-04-16T21:12:01.752964+0000","last_fullsized":"2024-04-16T21:12:01.752964+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.752809+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.752809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.500440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"45'2","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.625407+0000","last_change":"2024-04-16T21:12:04.625407+0000","last_active":"2024-04-16T21:12:04.625407+0000","last_peered":"2024-04-16T21:12:04.625407+0000","last_clean":"2024-04-16T21:12:04.625407+0000","last_became_active":"2024-04-16T19:25:15.482079+0000","last_became_peered":"2024-04-16T19:25:15.482079+0000","last_unstale":"2024-04-16T21:12:04.625407+0000","last_undegraded":"2024-04-16T21:12:04.625407+0000","last_fullsized":"2024-04-16T21:12:04.625407+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'2","last_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_deep_scrub":"45'2","last_deep_scrub_stamp":"2024-04-16T21:12:04.625306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.625306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.528697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":94,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.449242+0000","last_change":"2024-04-16T21:12:04.449242+0000","last_active":"2024-04-16T21:12:04.449242+0000","last_peered":"2024-04-16T21:12:04.449242+0000","last_clean":"2024-04-16T21:12:04.449242+0000","last_became_active":"2024-04-16T19:25:13.472354+0000","last_became_peered":"2024-04-16T19:25:13.472354+0000","last_unstale":"2024-04-16T21:12:04.449242+0000","last_undegraded":"2024-04-16T21:12:04.449242+0000","last_fullsized":"2024-04-16T21:12:04.449242+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.449160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.449160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:01:30.569236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"52'1163","reported_seq":2931,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.718512+0000","last_change":"2024-04-16T21:11:46.718512+0000","last_active":"2024-04-16T21:11:46.718512+0000","last_peered":"2024-04-16T21:11:46.718512+0000","last_clean":"2024-04-16T21:11:46.718512+0000","last_became_active":"2024-04-16T19:25:11.462489+0000","last_became_peered":"2024-04-16T19:25:11.462489+0000","last_unstale":"2024-04-16T21:11:46.718512+0000","last_undegraded":"2024-04-16T21:11:46.718512+0000","last_fullsized":"2024-04-16T21:11:46.718512+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1163","last_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_deep_scrub":"52'1163","last_deep_scrub_stamp":"2024-04-16T21:11:46.718377+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.718377+0000","objects_scrubbed":23,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:22:06.764370+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1791,"num_read_kb":1778,"num_write":1180,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1f","version":"52'16","reported_seq":208,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:51.594571+0000","last_change":"2024-04-16T21:11:51.594618+0000","last_active":"2024-04-16T21:11:51.594571+0000","last_peered":"2024-04-16T21:11:51.594571+0000","last_clean":"2024-04-16T21:11:51.594571+0000","last_became_active":"2024-04-16T19:25:09.465799+0000","last_became_peered":"2024-04-16T19:25:09.465799+0000","last_unstale":"2024-04-16T21:11:51.594571+0000","last_undegraded":"2024-04-16T21:11:51.594571+0000","last_fullsized":"2024-04-16T21:11:51.594571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:51.594510+0000","last_clean_scrub_stamp":"2024-04-16T21:11:51.594510+0000","objects_scrubbed":7,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:30:52.095893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":218,"num_read_kb":153,"num_write":84,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":119,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.782087+0000","last_change":"2024-04-16T21:12:01.782087+0000","last_active":"2024-04-16T21:12:01.782087+0000","last_peered":"2024-04-16T21:12:01.782087+0000","last_clean":"2024-04-16T21:12:01.782087+0000","last_became_active":"2024-04-16T19:24:51.442280+0000","last_became_peered":"2024-04-16T19:24:51.442280+0000","last_unstale":"2024-04-16T21:12:01.782087+0000","last_undegraded":"2024-04-16T21:12:01.782087+0000","last_fullsized":"2024-04-16T21:12:01.782087+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.781996+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.781996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:45.434631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.052606+0000","last_change":"2024-04-16T21:12:39.052630+0000","last_active":"2024-04-16T21:12:39.052606+0000","last_peered":"2024-04-16T21:12:39.052606+0000","last_clean":"2024-04-16T21:12:39.052606+0000","last_became_active":"2024-04-16T19:24:35.401366+0000","last_became_peered":"2024-04-16T19:24:35.401366+0000","last_unstale":"2024-04-16T21:12:39.052606+0000","last_undegraded":"2024-04-16T21:12:39.052606+0000","last_fullsized":"2024-04-16T21:12:39.052606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:39.052538+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.052538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:27.090193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.219500+0000","last_change":"2024-04-16T21:12:47.219500+0000","last_active":"2024-04-16T21:12:47.219500+0000","last_peered":"2024-04-16T21:12:47.219500+0000","last_clean":"2024-04-16T21:12:47.219500+0000","last_became_active":"2024-04-16T19:24:33.388402+0000","last_became_peered":"2024-04-16T19:24:33.388402+0000","last_unstale":"2024-04-16T21:12:47.219500+0000","last_undegraded":"2024-04-16T21:12:47.219500+0000","last_fullsized":"2024-04-16T21:12:47.219500+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.219422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.219422+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.219422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:36.660489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"52'15","reported_seq":155,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.143911+0000","last_change":"2024-04-16T21:13:01.143941+0000","last_active":"2024-04-16T21:13:01.143911+0000","last_peered":"2024-04-16T21:13:01.143911+0000","last_clean":"2024-04-16T21:13:01.143911+0000","last_became_active":"2024-04-16T19:24:45.413001+0000","last_became_peered":"2024-04-16T19:24:45.413001+0000","last_unstale":"2024-04-16T21:13:01.143911+0000","last_undegraded":"2024-04-16T21:13:01.143911+0000","last_fullsized":"2024-04-16T21:13:01.143911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:13:01.143846+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:13:01.143846+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.143846+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:20:13.624978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"52'49","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:40.341873+0000","last_change":"2024-04-16T21:12:40.341940+0000","last_active":"2024-04-16T21:12:40.341873+0000","last_peered":"2024-04-16T21:12:40.341873+0000","last_clean":"2024-04-16T21:12:40.341873+0000","last_became_active":"2024-04-16T19:24:47.419025+0000","last_became_peered":"2024-04-16T19:24:47.419025+0000","last_unstale":"2024-04-16T21:12:40.341873+0000","last_undegraded":"2024-04-16T21:12:40.341873+0000","last_fullsized":"2024-04-16T21:12:40.341873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'49","last_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_deep_scrub":"52'49","last_deep_scrub_stamp":"2024-04-16T21:12:40.341796+0000","last_clean_scrub_stamp":"2024-04-16T21:12:40.341796+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:03:55.596579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":66,"num_write":48,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.423193+0000","last_change":"2024-04-16T21:12:10.423217+0000","last_active":"2024-04-16T21:12:10.423193+0000","last_peered":"2024-04-16T21:12:10.423193+0000","last_clean":"2024-04-16T21:12:10.423193+0000","last_became_active":"2024-04-16T19:24:49.431102+0000","last_became_peered":"2024-04-16T19:24:49.431102+0000","last_unstale":"2024-04-16T21:12:10.423193+0000","last_undegraded":"2024-04-16T21:12:10.423193+0000","last_fullsized":"2024-04-16T21:12:10.423193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.423128+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.423128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:15.462072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":59,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:17.949377+0000","last_change":"2024-04-16T21:13:17.949398+0000","last_active":"2024-04-16T21:13:17.949377+0000","last_peered":"2024-04-16T21:13:17.949377+0000","last_clean":"2024-04-16T21:13:17.949377+0000","last_became_active":"2024-04-16T19:47:33.986438+0000","last_became_peered":"2024-04-16T19:47:33.986438+0000","last_unstale":"2024-04-16T21:13:17.949377+0000","last_undegraded":"2024-04-16T21:13:17.949377+0000","last_fullsized":"2024-04-16T21:13:17.949377+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:17.949312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:17.949312+0000","last_clean_scrub_stamp":"2024-04-16T21:13:17.949312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:28:11.394036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"52'181","reported_seq":1766,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.009430+0000","last_change":"2024-04-16T21:12:35.009430+0000","last_active":"2024-04-16T21:12:35.009430+0000","last_peered":"2024-04-16T21:12:35.009430+0000","last_clean":"2024-04-16T21:12:35.009430+0000","last_became_active":"2024-04-16T19:25:49.524259+0000","last_became_peered":"2024-04-16T19:25:49.524259+0000","last_unstale":"2024-04-16T21:12:35.009430+0000","last_undegraded":"2024-04-16T21:12:35.009430+0000","last_fullsized":"2024-04-16T21:12:35.009430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'181","last_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_deep_scrub":"52'181","last_deep_scrub_stamp":"2024-04-16T21:12:35.009303+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.009303+0000","objects_scrubbed":51,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:46.611974+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":2637,"num_read_kb":2599,"num_write":180,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8952,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.920212+0000","last_change":"2024-04-16T21:12:18.920252+0000","last_active":"2024-04-16T21:12:18.920212+0000","last_peered":"2024-04-16T21:12:18.920212+0000","last_clean":"2024-04-16T21:12:18.920212+0000","last_became_active":"2024-04-16T19:25:41.499022+0000","last_became_peered":"2024-04-16T19:25:41.499022+0000","last_unstale":"2024-04-16T21:12:18.920212+0000","last_undegraded":"2024-04-16T21:12:18.920212+0000","last_fullsized":"2024-04-16T21:12:18.920212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.920121+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.920121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:25:34.146310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"52'26","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:19.477899+0000","last_change":"2024-04-16T21:12:19.477941+0000","last_active":"2024-04-16T21:12:19.477899+0000","last_peered":"2024-04-16T21:12:19.477899+0000","last_clean":"2024-04-16T21:12:19.477899+0000","last_became_active":"2024-04-16T19:25:15.484940+0000","last_became_peered":"2024-04-16T19:25:15.484940+0000","last_unstale":"2024-04-16T21:12:19.477899+0000","last_undegraded":"2024-04-16T21:12:19.477899+0000","last_fullsized":"2024-04-16T21:12:19.477899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:19.477827+0000","last_clean_scrub_stamp":"2024-04-16T21:12:19.477827+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:13.666501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1617,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":94,"num_write":65,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.766206+0000","last_change":"2024-04-16T21:12:09.766229+0000","last_active":"2024-04-16T21:12:09.766206+0000","last_peered":"2024-04-16T21:12:09.766206+0000","last_clean":"2024-04-16T21:12:09.766206+0000","last_became_active":"2024-04-16T19:25:13.475714+0000","last_became_peered":"2024-04-16T19:25:13.475714+0000","last_unstale":"2024-04-16T21:12:09.766206+0000","last_undegraded":"2024-04-16T21:12:09.766206+0000","last_fullsized":"2024-04-16T21:12:09.766206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.766140+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.766140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:51.548954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1f","version":"52'291","reported_seq":604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.385570+0000","last_change":"2024-04-16T21:11:52.385621+0000","last_active":"2024-04-16T21:11:52.385570+0000","last_peered":"2024-04-16T21:11:52.385570+0000","last_clean":"2024-04-16T21:11:52.385570+0000","last_became_active":"2024-04-16T19:25:11.462338+0000","last_became_peered":"2024-04-16T19:25:11.462338+0000","last_unstale":"2024-04-16T21:11:52.385570+0000","last_undegraded":"2024-04-16T21:11:52.385570+0000","last_fullsized":"2024-04-16T21:11:52.385570+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'291","last_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_deep_scrub":"52'291","last_deep_scrub_stamp":"2024-04-16T21:11:52.385512+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.385512+0000","objects_scrubbed":17,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:09:13.385713+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":514,"num_read_kb":500,"num_write":301,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"52'16","reported_seq":263,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:52.131942+0000","last_change":"2024-04-16T21:11:52.131942+0000","last_active":"2024-04-16T21:11:52.131942+0000","last_peered":"2024-04-16T21:11:52.131942+0000","last_clean":"2024-04-16T21:11:52.131942+0000","last_became_active":"2024-04-16T19:25:09.460646+0000","last_became_peered":"2024-04-16T19:25:09.460646+0000","last_unstale":"2024-04-16T21:11:52.131942+0000","last_undegraded":"2024-04-16T21:11:52.131942+0000","last_fullsized":"2024-04-16T21:11:52.131942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:11:52.131829+0000","last_clean_scrub_stamp":"2024-04-16T21:11:52.131829+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:40:55.304717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":358,"num_read_kb":249,"num_write":89,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.441339+0000","last_change":"2024-04-16T21:12:08.441339+0000","last_active":"2024-04-16T21:12:08.441339+0000","last_peered":"2024-04-16T21:12:08.441339+0000","last_clean":"2024-04-16T21:12:08.441339+0000","last_became_active":"2024-04-16T19:24:51.435313+0000","last_became_peered":"2024-04-16T19:24:51.435313+0000","last_unstale":"2024-04-16T21:12:08.441339+0000","last_undegraded":"2024-04-16T21:12:08.441339+0000","last_fullsized":"2024-04-16T21:12:08.441339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.441251+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.441251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:03.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.805365+0000","last_change":"2024-04-16T21:12:35.805388+0000","last_active":"2024-04-16T21:12:35.805365+0000","last_peered":"2024-04-16T21:12:35.805365+0000","last_clean":"2024-04-16T21:12:35.805365+0000","last_became_active":"2024-04-16T19:24:35.393915+0000","last_became_peered":"2024-04-16T19:24:35.393915+0000","last_unstale":"2024-04-16T21:12:35.805365+0000","last_undegraded":"2024-04-16T21:12:35.805365+0000","last_fullsized":"2024-04-16T21:12:35.805365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.805297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.805297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:04.446592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.803989+0000","last_change":"2024-04-16T21:13:00.803989+0000","last_active":"2024-04-16T21:13:00.803989+0000","last_peered":"2024-04-16T21:13:00.803989+0000","last_clean":"2024-04-16T21:13:00.803989+0000","last_became_active":"2024-04-16T19:24:33.385911+0000","last_became_peered":"2024-04-16T19:24:33.385911+0000","last_unstale":"2024-04-16T21:13:00.803989+0000","last_undegraded":"2024-04-16T21:13:00.803989+0000","last_fullsized":"2024-04-16T21:13:00.803989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.803871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.803871+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.803871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:27.514468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"52'14","reported_seq":623,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.736145+0000","last_change":"2024-04-16T21:12:39.736175+0000","last_active":"2024-04-16T21:12:39.736145+0000","last_peered":"2024-04-16T21:12:39.736145+0000","last_clean":"2024-04-16T21:12:39.736145+0000","last_became_active":"2024-04-16T19:24:45.410292+0000","last_became_peered":"2024-04-16T19:24:45.410292+0000","last_unstale":"2024-04-16T21:12:39.736145+0000","last_undegraded":"2024-04-16T21:12:39.736145+0000","last_fullsized":"2024-04-16T21:12:39.736145+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'14","last_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_deep_scrub":"52'14","last_deep_scrub_stamp":"2024-04-16T21:12:39.736087+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.736087+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:19:33.924762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1412,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":498,"num_write":24,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"52'101","reported_seq":273,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.893687+0000","last_change":"2024-04-16T21:12:23.893724+0000","last_active":"2024-04-16T21:12:23.893687+0000","last_peered":"2024-04-16T21:12:23.893687+0000","last_clean":"2024-04-16T21:12:23.893687+0000","last_became_active":"2024-04-16T19:24:47.424473+0000","last_became_peered":"2024-04-16T19:24:47.424473+0000","last_unstale":"2024-04-16T21:12:23.893687+0000","last_undegraded":"2024-04-16T21:12:23.893687+0000","last_fullsized":"2024-04-16T21:12:23.893687+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:23.893616+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.893616+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:24:17.500596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.781919+0000","last_change":"2024-04-16T21:12:31.781950+0000","last_active":"2024-04-16T21:12:31.781919+0000","last_peered":"2024-04-16T21:12:31.781919+0000","last_clean":"2024-04-16T21:12:31.781919+0000","last_became_active":"2024-04-16T19:24:49.426381+0000","last_became_peered":"2024-04-16T19:24:49.426381+0000","last_unstale":"2024-04-16T21:12:31.781919+0000","last_undegraded":"2024-04-16T21:12:31.781919+0000","last_fullsized":"2024-04-16T21:12:31.781919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.781859+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.781859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:19.914350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":64,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:17.163016+0000","last_change":"2024-04-16T21:13:17.163037+0000","last_active":"2024-04-16T21:13:17.163016+0000","last_peered":"2024-04-16T21:13:17.163016+0000","last_clean":"2024-04-16T21:13:17.163016+0000","last_became_active":"2024-04-16T19:47:42.514149+0000","last_became_peered":"2024-04-16T19:47:42.514149+0000","last_unstale":"2024-04-16T21:13:17.163016+0000","last_undegraded":"2024-04-16T21:13:17.163016+0000","last_fullsized":"2024-04-16T21:13:17.163016+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:17.162953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:17.162953+0000","last_clean_scrub_stamp":"2024-04-16T21:13:17.162953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:25:47.863882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"52'124","reported_seq":1189,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.361300+0000","last_change":"2024-04-16T21:12:26.361300+0000","last_active":"2024-04-16T21:12:26.361300+0000","last_peered":"2024-04-16T21:12:26.361300+0000","last_clean":"2024-04-16T21:12:26.361300+0000","last_became_active":"2024-04-16T19:25:49.512987+0000","last_became_peered":"2024-04-16T19:25:49.512987+0000","last_unstale":"2024-04-16T21:12:26.361300+0000","last_undegraded":"2024-04-16T21:12:26.361300+0000","last_fullsized":"2024-04-16T21:12:26.361300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'124","last_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_deep_scrub":"52'124","last_deep_scrub_stamp":"2024-04-16T21:12:26.361168+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.361168+0000","objects_scrubbed":36,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:14.967461+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":1650,"num_read_kb":1619,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7264,"num_omap_keys":29,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.867629+0000","last_change":"2024-04-16T21:12:04.867629+0000","last_active":"2024-04-16T21:12:04.867629+0000","last_peered":"2024-04-16T21:12:04.867629+0000","last_clean":"2024-04-16T21:12:04.867629+0000","last_became_active":"2024-04-16T19:25:41.508586+0000","last_became_peered":"2024-04-16T19:25:41.508586+0000","last_unstale":"2024-04-16T21:12:04.867629+0000","last_undegraded":"2024-04-16T21:12:04.867629+0000","last_fullsized":"2024-04-16T21:12:04.867629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:04.867541+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.867541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:56.770932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.18","version":"52'31","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.873726+0000","last_change":"2024-04-16T21:12:11.873787+0000","last_active":"2024-04-16T21:12:11.873726+0000","last_peered":"2024-04-16T21:12:11.873726+0000","last_clean":"2024-04-16T21:12:11.873726+0000","last_became_active":"2024-04-16T19:25:15.480338+0000","last_became_peered":"2024-04-16T19:25:15.480338+0000","last_unstale":"2024-04-16T21:12:11.873726+0000","last_undegraded":"2024-04-16T21:12:11.873726+0000","last_fullsized":"2024-04-16T21:12:11.873726+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'31","last_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_deep_scrub":"52'31","last_deep_scrub_stamp":"2024-04-16T21:12:11.873657+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.873657+0000","objects_scrubbed":9,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:04.053968+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2271,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":202,"num_read_kb":149,"num_write":72,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"52'706","reported_seq":2151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.887851+0000","last_change":"2024-04-16T21:12:02.887901+0000","last_active":"2024-04-16T21:12:02.887851+0000","last_peered":"2024-04-16T21:12:02.887851+0000","last_clean":"2024-04-16T21:12:02.887851+0000","last_became_active":"2024-04-16T19:25:13.473093+0000","last_became_peered":"2024-04-16T19:25:13.473093+0000","last_unstale":"2024-04-16T21:12:02.887851+0000","last_undegraded":"2024-04-16T21:12:02.887851+0000","last_fullsized":"2024-04-16T21:12:02.887851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:02.887759+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.887759+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:37.031244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"52'1038","reported_seq":1547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.224293+0000","last_change":"2024-04-16T21:11:54.224379+0000","last_active":"2024-04-16T21:11:54.224293+0000","last_peered":"2024-04-16T21:11:54.224293+0000","last_clean":"2024-04-16T21:11:54.224293+0000","last_became_active":"2024-04-16T19:25:11.465892+0000","last_became_peered":"2024-04-16T19:25:11.465892+0000","last_unstale":"2024-04-16T21:11:54.224293+0000","last_undegraded":"2024-04-16T21:11:54.224293+0000","last_fullsized":"2024-04-16T21:11:54.224293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'1038","last_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_deep_scrub":"52'1038","last_deep_scrub_stamp":"2024-04-16T21:11:54.224224+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.224224+0000","objects_scrubbed":21,"log_size":1038,"log_dups_size":0,"ondisk_log_size":1038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:22.282373+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1711,"num_read_kb":1704,"num_write":1161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2194,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"52'23","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.136819+0000","last_change":"2024-04-16T21:11:43.136819+0000","last_active":"2024-04-16T21:11:43.136819+0000","last_peered":"2024-04-16T21:11:43.136819+0000","last_clean":"2024-04-16T21:11:43.136819+0000","last_became_active":"2024-04-16T19:25:09.467653+0000","last_became_peered":"2024-04-16T19:25:09.467653+0000","last_unstale":"2024-04-16T21:11:43.136819+0000","last_undegraded":"2024-04-16T21:11:43.136819+0000","last_fullsized":"2024-04-16T21:11:43.136819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'23","last_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_deep_scrub":"52'23","last_deep_scrub_stamp":"2024-04-16T21:11:43.136721+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.136721+0000","objects_scrubbed":9,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:33:06.328117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":195,"num_read_kb":162,"num_write":116,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":108,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:08.766632+0000","last_change":"2024-04-16T21:12:08.766632+0000","last_active":"2024-04-16T21:12:08.766632+0000","last_peered":"2024-04-16T21:12:08.766632+0000","last_clean":"2024-04-16T21:12:08.766632+0000","last_became_active":"2024-04-16T19:24:51.438488+0000","last_became_peered":"2024-04-16T19:24:51.438488+0000","last_unstale":"2024-04-16T21:12:08.766632+0000","last_undegraded":"2024-04-16T21:12:08.766632+0000","last_fullsized":"2024-04-16T21:12:08.766632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:08.766544+0000","last_clean_scrub_stamp":"2024-04-16T21:12:08.766544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:45.782631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:46.107579+0000","last_change":"2024-04-16T21:12:46.107599+0000","last_active":"2024-04-16T21:12:46.107579+0000","last_peered":"2024-04-16T21:12:46.107579+0000","last_clean":"2024-04-16T21:12:46.107579+0000","last_became_active":"2024-04-16T19:24:35.394498+0000","last_became_peered":"2024-04-16T19:24:35.394498+0000","last_unstale":"2024-04-16T21:12:46.107579+0000","last_undegraded":"2024-04-16T21:12:46.107579+0000","last_fullsized":"2024-04-16T21:12:46.107579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:46.107523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:46.107523+0000","last_clean_scrub_stamp":"2024-04-16T21:12:46.107523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:43:09.669785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:04.450793+0000","last_change":"2024-04-16T21:13:04.450793+0000","last_active":"2024-04-16T21:13:04.450793+0000","last_peered":"2024-04-16T21:13:04.450793+0000","last_clean":"2024-04-16T21:13:04.450793+0000","last_became_active":"2024-04-16T19:24:33.387929+0000","last_became_peered":"2024-04-16T19:24:33.387929+0000","last_unstale":"2024-04-16T21:13:04.450793+0000","last_undegraded":"2024-04-16T21:13:04.450793+0000","last_fullsized":"2024-04-16T21:13:04.450793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:04.450683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:04.450683+0000","last_clean_scrub_stamp":"2024-04-16T21:13:04.450683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:19:03.655955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.186769+0000","last_change":"2024-04-16T21:12:41.186808+0000","last_active":"2024-04-16T21:12:41.186769+0000","last_peered":"2024-04-16T21:12:41.186769+0000","last_clean":"2024-04-16T21:12:41.186769+0000","last_became_active":"2024-04-16T19:24:45.413041+0000","last_became_peered":"2024-04-16T19:24:45.413041+0000","last_unstale":"2024-04-16T21:12:41.186769+0000","last_undegraded":"2024-04-16T21:12:41.186769+0000","last_fullsized":"2024-04-16T21:12:41.186769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.186689+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.186689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:26:10.176347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"52'78","reported_seq":239,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.901279+0000","last_change":"2024-04-16T21:12:35.901311+0000","last_active":"2024-04-16T21:12:35.901279+0000","last_peered":"2024-04-16T21:12:35.901279+0000","last_clean":"2024-04-16T21:12:35.901279+0000","last_became_active":"2024-04-16T19:24:47.421269+0000","last_became_peered":"2024-04-16T19:24:47.421269+0000","last_unstale":"2024-04-16T21:12:35.901279+0000","last_undegraded":"2024-04-16T21:12:35.901279+0000","last_fullsized":"2024-04-16T21:12:35.901279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:35.901215+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.901215+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:22.199019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.365174+0000","last_change":"2024-04-16T21:12:25.365219+0000","last_active":"2024-04-16T21:12:25.365174+0000","last_peered":"2024-04-16T21:12:25.365174+0000","last_clean":"2024-04-16T21:12:25.365174+0000","last_became_active":"2024-04-16T19:24:49.431363+0000","last_became_peered":"2024-04-16T19:24:49.431363+0000","last_unstale":"2024-04-16T21:12:25.365174+0000","last_undegraded":"2024-04-16T21:12:25.365174+0000","last_fullsized":"2024-04-16T21:12:25.365174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.365094+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.365094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:11.662898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:14.198742+0000","last_change":"2024-04-16T21:13:14.198764+0000","last_active":"2024-04-16T21:13:14.198742+0000","last_peered":"2024-04-16T21:13:14.198742+0000","last_clean":"2024-04-16T21:13:14.198742+0000","last_became_active":"2024-04-16T19:47:33.986146+0000","last_became_peered":"2024-04-16T19:47:33.986146+0000","last_unstale":"2024-04-16T21:13:14.198742+0000","last_undegraded":"2024-04-16T21:13:14.198742+0000","last_fullsized":"2024-04-16T21:13:14.198742+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:14.198675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:14.198675+0000","last_clean_scrub_stamp":"2024-04-16T21:13:14.198675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:12.990928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"52'207","reported_seq":2084,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.250819+0000","last_change":"2024-04-16T21:12:44.250861+0000","last_active":"2024-04-16T21:12:44.250819+0000","last_peered":"2024-04-16T21:12:44.250819+0000","last_clean":"2024-04-16T21:12:44.250819+0000","last_became_active":"2024-04-16T19:25:49.519893+0000","last_became_peered":"2024-04-16T19:25:49.519893+0000","last_unstale":"2024-04-16T21:12:44.250819+0000","last_undegraded":"2024-04-16T21:12:44.250819+0000","last_fullsized":"2024-04-16T21:12:44.250819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'207","last_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_deep_scrub":"52'207","last_deep_scrub_stamp":"2024-04-16T21:12:44.250741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.250741+0000","objects_scrubbed":45,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:38.832019+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3256,"num_read_kb":3265,"num_write":272,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12815,"num_omap_keys":53,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.457685+0000","last_change":"2024-04-16T21:12:13.457709+0000","last_active":"2024-04-16T21:12:13.457685+0000","last_peered":"2024-04-16T21:12:13.457685+0000","last_clean":"2024-04-16T21:12:13.457685+0000","last_became_active":"2024-04-16T19:25:41.506833+0000","last_became_peered":"2024-04-16T19:25:41.506833+0000","last_unstale":"2024-04-16T21:12:13.457685+0000","last_undegraded":"2024-04-16T21:12:13.457685+0000","last_fullsized":"2024-04-16T21:12:13.457685+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:13.457615+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.457615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:51.890971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.17","version":"52'7","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.648645+0000","last_change":"2024-04-16T21:12:07.648674+0000","last_active":"2024-04-16T21:12:07.648645+0000","last_peered":"2024-04-16T21:12:07.648645+0000","last_clean":"2024-04-16T21:12:07.648645+0000","last_became_active":"2024-04-16T19:25:15.480864+0000","last_became_peered":"2024-04-16T19:25:15.480864+0000","last_unstale":"2024-04-16T21:12:07.648645+0000","last_undegraded":"2024-04-16T21:12:07.648645+0000","last_fullsized":"2024-04-16T21:12:07.648645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'7","last_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_deep_scrub":"52'7","last_deep_scrub_stamp":"2024-04-16T21:12:07.648592+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.648592+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:48.343437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.389563+0000","last_change":"2024-04-16T21:12:06.389563+0000","last_active":"2024-04-16T21:12:06.389563+0000","last_peered":"2024-04-16T21:12:06.389563+0000","last_clean":"2024-04-16T21:12:06.389563+0000","last_became_active":"2024-04-16T19:25:13.475848+0000","last_became_peered":"2024-04-16T19:25:13.475848+0000","last_unstale":"2024-04-16T21:12:06.389563+0000","last_undegraded":"2024-04-16T21:12:06.389563+0000","last_fullsized":"2024-04-16T21:12:06.389563+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.389467+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.389467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:31:55.353016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"52'329","reported_seq":649,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:45.716329+0000","last_change":"2024-04-16T21:11:45.716329+0000","last_active":"2024-04-16T21:11:45.716329+0000","last_peered":"2024-04-16T21:11:45.716329+0000","last_clean":"2024-04-16T21:11:45.716329+0000","last_became_active":"2024-04-16T19:25:11.465536+0000","last_became_peered":"2024-04-16T19:25:11.465536+0000","last_unstale":"2024-04-16T21:11:45.716329+0000","last_undegraded":"2024-04-16T21:11:45.716329+0000","last_fullsized":"2024-04-16T21:11:45.716329+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'329","last_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_deep_scrub":"52'329","last_deep_scrub_stamp":"2024-04-16T21:11:45.716182+0000","last_clean_scrub_stamp":"2024-04-16T21:11:45.716182+0000","objects_scrubbed":24,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:35:24.600927+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":484,"num_read_kb":476,"num_write":460,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"52'11","reported_seq":161,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.675426+0000","last_change":"2024-04-16T21:11:50.675426+0000","last_active":"2024-04-16T21:11:50.675426+0000","last_peered":"2024-04-16T21:11:50.675426+0000","last_clean":"2024-04-16T21:11:50.675426+0000","last_became_active":"2024-04-16T19:25:09.469330+0000","last_became_peered":"2024-04-16T19:25:09.469330+0000","last_unstale":"2024-04-16T21:11:50.675426+0000","last_undegraded":"2024-04-16T21:11:50.675426+0000","last_fullsized":"2024-04-16T21:11:50.675426+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_deep_scrub":"52'11","last_deep_scrub_stamp":"2024-04-16T21:11:50.675302+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.675302+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:31.269067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":108,"num_read_kb":85,"num_write":61,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.193468+0000","last_change":"2024-04-16T21:12:45.193468+0000","last_active":"2024-04-16T21:12:45.193468+0000","last_peered":"2024-04-16T21:12:45.193468+0000","last_clean":"2024-04-16T21:12:45.193468+0000","last_became_active":"2024-04-16T19:24:35.396419+0000","last_became_peered":"2024-04-16T19:24:35.396419+0000","last_unstale":"2024-04-16T21:12:45.193468+0000","last_undegraded":"2024-04-16T21:12:45.193468+0000","last_fullsized":"2024-04-16T21:12:45.193468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.193382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.193382+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.193382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:16:54.542159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":135,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.768937+0000","last_change":"2024-04-16T21:12:49.768937+0000","last_active":"2024-04-16T21:12:49.768937+0000","last_peered":"2024-04-16T21:12:49.768937+0000","last_clean":"2024-04-16T21:12:49.768937+0000","last_became_active":"2024-04-16T19:24:33.391814+0000","last_became_peered":"2024-04-16T19:24:33.391814+0000","last_unstale":"2024-04-16T21:12:49.768937+0000","last_undegraded":"2024-04-16T21:12:49.768937+0000","last_fullsized":"2024-04-16T21:12:49.768937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.768845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.768845+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.768845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:56.006493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.345218+0000","last_change":"2024-04-16T21:12:41.345218+0000","last_active":"2024-04-16T21:12:41.345218+0000","last_peered":"2024-04-16T21:12:41.345218+0000","last_clean":"2024-04-16T21:12:41.345218+0000","last_became_active":"2024-04-16T19:24:45.415725+0000","last_became_peered":"2024-04-16T19:24:45.415725+0000","last_unstale":"2024-04-16T21:12:41.345218+0000","last_undegraded":"2024-04-16T21:12:41.345218+0000","last_fullsized":"2024-04-16T21:12:41.345218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.345086+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.345086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:35.926364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"52'87","reported_seq":254,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:30.024063+0000","last_change":"2024-04-16T21:12:30.024100+0000","last_active":"2024-04-16T21:12:30.024063+0000","last_peered":"2024-04-16T21:12:30.024063+0000","last_clean":"2024-04-16T21:12:30.024063+0000","last_became_active":"2024-04-16T19:24:47.423335+0000","last_became_peered":"2024-04-16T19:24:47.423335+0000","last_unstale":"2024-04-16T21:12:30.024063+0000","last_undegraded":"2024-04-16T21:12:30.024063+0000","last_fullsized":"2024-04-16T21:12:30.024063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'87","last_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_deep_scrub":"52'87","last_deep_scrub_stamp":"2024-04-16T21:12:30.024002+0000","last_clean_scrub_stamp":"2024-04-16T21:12:30.024002+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:36:39.585670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":133,"num_read_kb":124,"num_write":82,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.356039+0000","last_change":"2024-04-16T21:12:27.356060+0000","last_active":"2024-04-16T21:12:27.356039+0000","last_peered":"2024-04-16T21:12:27.356039+0000","last_clean":"2024-04-16T21:12:27.356039+0000","last_became_active":"2024-04-16T19:24:49.429886+0000","last_became_peered":"2024-04-16T19:24:49.429886+0000","last_unstale":"2024-04-16T21:12:27.356039+0000","last_undegraded":"2024-04-16T21:12:27.356039+0000","last_fullsized":"2024-04-16T21:12:27.356039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.355975+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.355975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:15.393630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.732045+0000","last_change":"2024-04-16T21:11:59.732045+0000","last_active":"2024-04-16T21:11:59.732045+0000","last_peered":"2024-04-16T21:11:59.732045+0000","last_clean":"2024-04-16T21:11:59.732045+0000","last_became_active":"2024-04-16T19:24:51.573460+0000","last_became_peered":"2024-04-16T19:24:51.573460+0000","last_unstale":"2024-04-16T21:11:59.732045+0000","last_undegraded":"2024-04-16T21:11:59.732045+0000","last_fullsized":"2024-04-16T21:11:59.732045+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.731957+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.731957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:37.824609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:16.902123+0000","last_change":"2024-04-16T21:13:16.902161+0000","last_active":"2024-04-16T21:13:16.902123+0000","last_peered":"2024-04-16T21:13:16.902123+0000","last_clean":"2024-04-16T21:13:16.902123+0000","last_became_active":"2024-04-16T19:47:33.985090+0000","last_became_peered":"2024-04-16T19:47:33.985090+0000","last_unstale":"2024-04-16T21:13:16.902123+0000","last_undegraded":"2024-04-16T21:13:16.902123+0000","last_fullsized":"2024-04-16T21:13:16.902123+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:16.902054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:16.902054+0000","last_clean_scrub_stamp":"2024-04-16T21:13:16.902054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:29:54.903465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.14","version":"52'170","reported_seq":1882,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.686097+0000","last_change":"2024-04-16T21:12:29.686173+0000","last_active":"2024-04-16T21:12:29.686097+0000","last_peered":"2024-04-16T21:12:29.686097+0000","last_clean":"2024-04-16T21:12:29.686097+0000","last_became_active":"2024-04-16T19:25:49.518601+0000","last_became_peered":"2024-04-16T19:25:49.518601+0000","last_unstale":"2024-04-16T21:12:29.686097+0000","last_undegraded":"2024-04-16T21:12:29.686097+0000","last_fullsized":"2024-04-16T21:12:29.686097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'170","last_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_deep_scrub":"52'170","last_deep_scrub_stamp":"2024-04-16T21:12:29.686024+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.686024+0000","objects_scrubbed":46,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:24.943848+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2820,"num_read_kb":2776,"num_write":179,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12193,"num_omap_keys":48,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.906298+0000","last_change":"2024-04-16T21:12:17.906319+0000","last_active":"2024-04-16T21:12:17.906298+0000","last_peered":"2024-04-16T21:12:17.906298+0000","last_clean":"2024-04-16T21:12:17.906298+0000","last_became_active":"2024-04-16T19:25:41.503571+0000","last_became_peered":"2024-04-16T19:25:41.503571+0000","last_unstale":"2024-04-16T21:12:17.906298+0000","last_undegraded":"2024-04-16T21:12:17.906298+0000","last_fullsized":"2024-04-16T21:12:17.906298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.906238+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.906238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:09:22.945182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"52'18","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:24.785345+0000","last_change":"2024-04-16T21:12:24.785376+0000","last_active":"2024-04-16T21:12:24.785345+0000","last_peered":"2024-04-16T21:12:24.785345+0000","last_clean":"2024-04-16T21:12:24.785345+0000","last_became_active":"2024-04-16T19:25:15.482766+0000","last_became_peered":"2024-04-16T19:25:15.482766+0000","last_unstale":"2024-04-16T21:12:24.785345+0000","last_undegraded":"2024-04-16T21:12:24.785345+0000","last_fullsized":"2024-04-16T21:12:24.785345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:24.785282+0000","last_clean_scrub_stamp":"2024-04-16T21:12:24.785282+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:47:53.426577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":991,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":52,"num_write":55,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.865507+0000","last_change":"2024-04-16T21:12:05.865507+0000","last_active":"2024-04-16T21:12:05.865507+0000","last_peered":"2024-04-16T21:12:05.865507+0000","last_clean":"2024-04-16T21:12:05.865507+0000","last_became_active":"2024-04-16T19:25:13.471739+0000","last_became_peered":"2024-04-16T19:25:13.471739+0000","last_unstale":"2024-04-16T21:12:05.865507+0000","last_undegraded":"2024-04-16T21:12:05.865507+0000","last_fullsized":"2024-04-16T21:12:05.865507+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.865437+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.865437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:42:41.716343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"52'138","reported_seq":299,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.719459+0000","last_change":"2024-04-16T21:11:48.719459+0000","last_active":"2024-04-16T21:11:48.719459+0000","last_peered":"2024-04-16T21:11:48.719459+0000","last_clean":"2024-04-16T21:11:48.719459+0000","last_became_active":"2024-04-16T19:25:11.465557+0000","last_became_peered":"2024-04-16T19:25:11.465557+0000","last_unstale":"2024-04-16T21:11:48.719459+0000","last_undegraded":"2024-04-16T21:11:48.719459+0000","last_fullsized":"2024-04-16T21:11:48.719459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'138","last_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_deep_scrub":"52'138","last_deep_scrub_stamp":"2024-04-16T21:11:48.719337+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.719337+0000","objects_scrubbed":17,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:44.768315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":171,"num_read_kb":167,"num_write":178,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4229,"num_omap_keys":20,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"52'8","reported_seq":147,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.405983+0000","last_change":"2024-04-16T21:11:49.405983+0000","last_active":"2024-04-16T21:11:49.405983+0000","last_peered":"2024-04-16T21:11:49.405983+0000","last_clean":"2024-04-16T21:11:49.405983+0000","last_became_active":"2024-04-16T19:25:09.461868+0000","last_became_peered":"2024-04-16T19:25:09.461868+0000","last_unstale":"2024-04-16T21:11:49.405983+0000","last_undegraded":"2024-04-16T21:11:49.405983+0000","last_fullsized":"2024-04-16T21:11:49.405983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:11:49.405858+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.405858+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:48:30.100703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":59,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.341370+0000","last_change":"2024-04-16T21:12:44.341370+0000","last_active":"2024-04-16T21:12:44.341370+0000","last_peered":"2024-04-16T21:12:44.341370+0000","last_clean":"2024-04-16T21:12:44.341370+0000","last_became_active":"2024-04-16T19:24:35.396420+0000","last_became_peered":"2024-04-16T19:24:35.396420+0000","last_unstale":"2024-04-16T21:12:44.341370+0000","last_undegraded":"2024-04-16T21:12:44.341370+0000","last_fullsized":"2024-04-16T21:12:44.341370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.341283+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.341283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:25:33.782350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:57.076148+0000","last_change":"2024-04-16T21:12:57.076171+0000","last_active":"2024-04-16T21:12:57.076148+0000","last_peered":"2024-04-16T21:12:57.076148+0000","last_clean":"2024-04-16T21:12:57.076148+0000","last_became_active":"2024-04-16T19:24:33.385026+0000","last_became_peered":"2024-04-16T19:24:33.385026+0000","last_unstale":"2024-04-16T21:12:57.076148+0000","last_undegraded":"2024-04-16T21:12:57.076148+0000","last_fullsized":"2024-04-16T21:12:57.076148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:57.076066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:57.076066+0000","last_clean_scrub_stamp":"2024-04-16T21:12:57.076066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:06.424842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"52'4","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.265243+0000","last_change":"2024-04-16T21:12:39.265243+0000","last_active":"2024-04-16T21:12:39.265243+0000","last_peered":"2024-04-16T21:12:39.265243+0000","last_clean":"2024-04-16T21:12:39.265243+0000","last_became_active":"2024-04-16T19:24:45.413082+0000","last_became_peered":"2024-04-16T19:24:45.413082+0000","last_unstale":"2024-04-16T21:12:39.265243+0000","last_undegraded":"2024-04-16T21:12:39.265243+0000","last_fullsized":"2024-04-16T21:12:39.265243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:39.265147+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.265147+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:53.843686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2782,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":16,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"52'147","reported_seq":338,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.377505+0000","last_change":"2024-04-16T21:12:31.377537+0000","last_active":"2024-04-16T21:12:31.377505+0000","last_peered":"2024-04-16T21:12:31.377505+0000","last_clean":"2024-04-16T21:12:31.377505+0000","last_became_active":"2024-04-16T19:24:47.420053+0000","last_became_peered":"2024-04-16T19:24:47.420053+0000","last_unstale":"2024-04-16T21:12:31.377505+0000","last_undegraded":"2024-04-16T21:12:31.377505+0000","last_fullsized":"2024-04-16T21:12:31.377505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:31.377449+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.377449+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:37.615098+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.930468+0000","last_change":"2024-04-16T21:12:23.930491+0000","last_active":"2024-04-16T21:12:23.930468+0000","last_peered":"2024-04-16T21:12:23.930468+0000","last_clean":"2024-04-16T21:12:23.930468+0000","last_became_active":"2024-04-16T19:24:49.431933+0000","last_became_peered":"2024-04-16T19:24:49.431933+0000","last_unstale":"2024-04-16T21:12:23.930468+0000","last_undegraded":"2024-04-16T21:12:23.930468+0000","last_fullsized":"2024-04-16T21:12:23.930468+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:23.930404+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.930404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:27.014075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.517927+0000","last_change":"2024-04-16T21:12:01.517927+0000","last_active":"2024-04-16T21:12:01.517927+0000","last_peered":"2024-04-16T21:12:01.517927+0000","last_clean":"2024-04-16T21:12:01.517927+0000","last_became_active":"2024-04-16T19:24:51.572789+0000","last_became_peered":"2024-04-16T19:24:51.572789+0000","last_unstale":"2024-04-16T21:12:01.517927+0000","last_undegraded":"2024-04-16T21:12:01.517927+0000","last_fullsized":"2024-04-16T21:12:01.517927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.517839+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:33:38.533926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:15.437208+0000","last_change":"2024-04-16T21:13:15.437232+0000","last_active":"2024-04-16T21:13:15.437208+0000","last_peered":"2024-04-16T21:13:15.437208+0000","last_clean":"2024-04-16T21:13:15.437208+0000","last_became_active":"2024-04-16T19:47:33.987441+0000","last_became_peered":"2024-04-16T19:47:33.987441+0000","last_unstale":"2024-04-16T21:13:15.437208+0000","last_undegraded":"2024-04-16T21:13:15.437208+0000","last_fullsized":"2024-04-16T21:13:15.437208+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:15.437140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:15.437140+0000","last_clean_scrub_stamp":"2024-04-16T21:13:15.437140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:52:44.204195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"52'150","reported_seq":1340,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.054107+0000","last_change":"2024-04-16T21:12:31.054152+0000","last_active":"2024-04-16T21:12:31.054107+0000","last_peered":"2024-04-16T21:12:31.054107+0000","last_clean":"2024-04-16T21:12:31.054107+0000","last_became_active":"2024-04-16T19:25:49.523442+0000","last_became_peered":"2024-04-16T19:25:49.523442+0000","last_unstale":"2024-04-16T21:12:31.054107+0000","last_undegraded":"2024-04-16T21:12:31.054107+0000","last_fullsized":"2024-04-16T21:12:31.054107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'150","last_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_deep_scrub":"52'150","last_deep_scrub_stamp":"2024-04-16T21:12:31.054045+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.054045+0000","objects_scrubbed":46,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:22.727735+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1857,"num_read_kb":1827,"num_write":139,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7793,"num_omap_keys":32,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.443154+0000","last_change":"2024-04-16T21:12:22.443154+0000","last_active":"2024-04-16T21:12:22.443154+0000","last_peered":"2024-04-16T21:12:22.443154+0000","last_clean":"2024-04-16T21:12:22.443154+0000","last_became_active":"2024-04-16T19:25:41.509238+0000","last_became_peered":"2024-04-16T19:25:41.509238+0000","last_unstale":"2024-04-16T21:12:22.443154+0000","last_undegraded":"2024-04-16T21:12:22.443154+0000","last_fullsized":"2024-04-16T21:12:22.443154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:22.443057+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.443057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:22.418023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"52'15","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.828587+0000","last_change":"2024-04-16T21:12:23.828622+0000","last_active":"2024-04-16T21:12:23.828587+0000","last_peered":"2024-04-16T21:12:23.828587+0000","last_clean":"2024-04-16T21:12:23.828587+0000","last_became_active":"2024-04-16T19:25:15.483323+0000","last_became_peered":"2024-04-16T19:25:15.483323+0000","last_unstale":"2024-04-16T21:12:23.828587+0000","last_undegraded":"2024-04-16T21:12:23.828587+0000","last_fullsized":"2024-04-16T21:12:23.828587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'15","last_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_deep_scrub":"52'15","last_deep_scrub_stamp":"2024-04-16T21:12:23.828490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.828490+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:17:02.261846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1977,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":64,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.402484+0000","last_change":"2024-04-16T21:12:06.402484+0000","last_active":"2024-04-16T21:12:06.402484+0000","last_peered":"2024-04-16T21:12:06.402484+0000","last_clean":"2024-04-16T21:12:06.402484+0000","last_became_active":"2024-04-16T19:25:13.472466+0000","last_became_peered":"2024-04-16T19:25:13.472466+0000","last_unstale":"2024-04-16T21:12:06.402484+0000","last_undegraded":"2024-04-16T21:12:06.402484+0000","last_fullsized":"2024-04-16T21:12:06.402484+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:06.402344+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.402344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:52.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.13","version":"52'341","reported_seq":880,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.634146+0000","last_change":"2024-04-16T21:11:48.634146+0000","last_active":"2024-04-16T21:11:48.634146+0000","last_peered":"2024-04-16T21:11:48.634146+0000","last_clean":"2024-04-16T21:11:48.634146+0000","last_became_active":"2024-04-16T19:25:11.464136+0000","last_became_peered":"2024-04-16T19:25:11.464136+0000","last_unstale":"2024-04-16T21:11:48.634146+0000","last_undegraded":"2024-04-16T21:11:48.634146+0000","last_fullsized":"2024-04-16T21:11:48.634146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'341","last_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_deep_scrub":"52'341","last_deep_scrub_stamp":"2024-04-16T21:11:48.633983+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.633983+0000","objects_scrubbed":20,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:05:52.976164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":733,"num_read_kb":725,"num_write":327,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"52'19","reported_seq":182,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.631145+0000","last_change":"2024-04-16T21:11:55.631145+0000","last_active":"2024-04-16T21:11:55.631145+0000","last_peered":"2024-04-16T21:11:55.631145+0000","last_clean":"2024-04-16T21:11:55.631145+0000","last_became_active":"2024-04-16T19:25:09.465606+0000","last_became_peered":"2024-04-16T19:25:09.465606+0000","last_unstale":"2024-04-16T21:11:55.631145+0000","last_undegraded":"2024-04-16T21:11:55.631145+0000","last_fullsized":"2024-04-16T21:11:55.631145+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:11:55.631042+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.631042+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:08:57.051632+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20658,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":179,"num_write":89,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:47.384307+0000","last_change":"2024-04-16T21:12:47.384330+0000","last_active":"2024-04-16T21:12:47.384307+0000","last_peered":"2024-04-16T21:12:47.384307+0000","last_clean":"2024-04-16T21:12:47.384307+0000","last_became_active":"2024-04-16T19:24:35.394540+0000","last_became_peered":"2024-04-16T19:24:35.394540+0000","last_unstale":"2024-04-16T21:12:47.384307+0000","last_undegraded":"2024-04-16T21:12:47.384307+0000","last_fullsized":"2024-04-16T21:12:47.384307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:47.384240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:47.384240+0000","last_clean_scrub_stamp":"2024-04-16T21:12:47.384240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:55.538199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.701557+0000","last_change":"2024-04-16T21:12:48.701581+0000","last_active":"2024-04-16T21:12:48.701557+0000","last_peered":"2024-04-16T21:12:48.701557+0000","last_clean":"2024-04-16T21:12:48.701557+0000","last_became_active":"2024-04-16T19:24:33.381323+0000","last_became_peered":"2024-04-16T19:24:33.381323+0000","last_unstale":"2024-04-16T21:12:48.701557+0000","last_undegraded":"2024-04-16T21:12:48.701557+0000","last_fullsized":"2024-04-16T21:12:48.701557+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.701492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.701492+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.701492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:54.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.723608+0000","last_change":"2024-04-16T21:12:41.723630+0000","last_active":"2024-04-16T21:12:41.723608+0000","last_peered":"2024-04-16T21:12:41.723608+0000","last_clean":"2024-04-16T21:12:41.723608+0000","last_became_active":"2024-04-16T19:24:45.411455+0000","last_became_peered":"2024-04-16T19:24:45.411455+0000","last_unstale":"2024-04-16T21:12:41.723608+0000","last_undegraded":"2024-04-16T21:12:41.723608+0000","last_fullsized":"2024-04-16T21:12:41.723608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.723545+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.723545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:29.856036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"52'101","reported_seq":279,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.758528+0000","last_change":"2024-04-16T21:12:50.758528+0000","last_active":"2024-04-16T21:12:50.758528+0000","last_peered":"2024-04-16T21:12:50.758528+0000","last_clean":"2024-04-16T21:12:50.758528+0000","last_became_active":"2024-04-16T19:24:47.421336+0000","last_became_peered":"2024-04-16T19:24:47.421336+0000","last_unstale":"2024-04-16T21:12:50.758528+0000","last_undegraded":"2024-04-16T21:12:50.758528+0000","last_fullsized":"2024-04-16T21:12:50.758528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'101","last_scrub_stamp":"2024-04-16T21:12:50.758441+0000","last_deep_scrub":"52'101","last_deep_scrub_stamp":"2024-04-16T21:12:50.758441+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.758441+0000","objects_scrubbed":5,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:06.557737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":142,"num_write":96,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.928177+0000","last_change":"2024-04-16T21:12:25.928177+0000","last_active":"2024-04-16T21:12:25.928177+0000","last_peered":"2024-04-16T21:12:25.928177+0000","last_clean":"2024-04-16T21:12:25.928177+0000","last_became_active":"2024-04-16T19:24:49.426033+0000","last_became_peered":"2024-04-16T19:24:49.426033+0000","last_unstale":"2024-04-16T21:12:25.928177+0000","last_undegraded":"2024-04-16T21:12:25.928177+0000","last_fullsized":"2024-04-16T21:12:25.928177+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:25.928102+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.928102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:41.397476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.482650+0000","last_change":"2024-04-16T21:12:03.482650+0000","last_active":"2024-04-16T21:12:03.482650+0000","last_peered":"2024-04-16T21:12:03.482650+0000","last_clean":"2024-04-16T21:12:03.482650+0000","last_became_active":"2024-04-16T19:24:51.442127+0000","last_became_peered":"2024-04-16T19:24:51.442127+0000","last_unstale":"2024-04-16T21:12:03.482650+0000","last_undegraded":"2024-04-16T21:12:03.482650+0000","last_fullsized":"2024-04-16T21:12:03.482650+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:03.482557+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.482557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:28.918007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":60,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.850860+0000","last_change":"2024-04-16T21:13:00.850882+0000","last_active":"2024-04-16T21:13:00.850860+0000","last_peered":"2024-04-16T21:13:00.850860+0000","last_clean":"2024-04-16T21:13:00.850860+0000","last_became_active":"2024-04-16T19:47:33.983327+0000","last_became_peered":"2024-04-16T19:47:33.983327+0000","last_unstale":"2024-04-16T21:13:00.850860+0000","last_undegraded":"2024-04-16T21:13:00.850860+0000","last_fullsized":"2024-04-16T21:13:00.850860+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.850797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.850797+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.850797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:22:28.392426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"52'154","reported_seq":1354,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.959232+0000","last_change":"2024-04-16T21:12:26.959277+0000","last_active":"2024-04-16T21:12:26.959232+0000","last_peered":"2024-04-16T21:12:26.959232+0000","last_clean":"2024-04-16T21:12:26.959232+0000","last_became_active":"2024-04-16T19:25:49.523933+0000","last_became_peered":"2024-04-16T19:25:49.523933+0000","last_unstale":"2024-04-16T21:12:26.959232+0000","last_undegraded":"2024-04-16T21:12:26.959232+0000","last_fullsized":"2024-04-16T21:12:26.959232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'154","last_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_deep_scrub":"52'154","last_deep_scrub_stamp":"2024-04-16T21:12:26.959173+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.959173+0000","objects_scrubbed":37,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:21.387909+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1895,"num_read_kb":1847,"num_write":166,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8816,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:17.680232+0000","last_change":"2024-04-16T21:12:17.680254+0000","last_active":"2024-04-16T21:12:17.680232+0000","last_peered":"2024-04-16T21:12:17.680232+0000","last_clean":"2024-04-16T21:12:17.680232+0000","last_became_active":"2024-04-16T19:25:41.508226+0000","last_became_peered":"2024-04-16T19:25:41.508226+0000","last_unstale":"2024-04-16T21:12:17.680232+0000","last_undegraded":"2024-04-16T21:12:17.680232+0000","last_fullsized":"2024-04-16T21:12:17.680232+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:17.680171+0000","last_clean_scrub_stamp":"2024-04-16T21:12:17.680171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:45:09.860482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"52'8","reported_seq":123,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:22.838571+0000","last_change":"2024-04-16T21:12:22.838571+0000","last_active":"2024-04-16T21:12:22.838571+0000","last_peered":"2024-04-16T21:12:22.838571+0000","last_clean":"2024-04-16T21:12:22.838571+0000","last_became_active":"2024-04-16T19:25:15.485971+0000","last_became_peered":"2024-04-16T19:25:15.485971+0000","last_unstale":"2024-04-16T21:12:22.838571+0000","last_undegraded":"2024-04-16T21:12:22.838571+0000","last_fullsized":"2024-04-16T21:12:22.838571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'8","last_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_deep_scrub":"52'8","last_deep_scrub_stamp":"2024-04-16T21:12:22.838490+0000","last_clean_scrub_stamp":"2024-04-16T21:12:22.838490+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:17.267225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":97,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.653905+0000","last_change":"2024-04-16T21:12:05.653905+0000","last_active":"2024-04-16T21:12:05.653905+0000","last_peered":"2024-04-16T21:12:05.653905+0000","last_clean":"2024-04-16T21:12:05.653905+0000","last_became_active":"2024-04-16T19:25:13.473131+0000","last_became_peered":"2024-04-16T19:25:13.473131+0000","last_unstale":"2024-04-16T21:12:05.653905+0000","last_undegraded":"2024-04-16T21:12:05.653905+0000","last_fullsized":"2024-04-16T21:12:05.653905+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.653825+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.653825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:12.673868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"52'264","reported_seq":1137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.696035+0000","last_change":"2024-04-16T21:11:49.696035+0000","last_active":"2024-04-16T21:11:49.696035+0000","last_peered":"2024-04-16T21:11:49.696035+0000","last_clean":"2024-04-16T21:11:49.696035+0000","last_became_active":"2024-04-16T19:25:11.465522+0000","last_became_peered":"2024-04-16T19:25:11.465522+0000","last_unstale":"2024-04-16T21:11:49.696035+0000","last_undegraded":"2024-04-16T21:11:49.696035+0000","last_fullsized":"2024-04-16T21:11:49.696035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'264","last_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_deep_scrub":"52'264","last_deep_scrub_stamp":"2024-04-16T21:11:49.695875+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.695875+0000","objects_scrubbed":23,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:11.637965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":987,"num_read_kb":978,"num_write":355,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2924,"num_omap_keys":15,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"52'12","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.501597+0000","last_change":"2024-04-16T21:11:59.501597+0000","last_active":"2024-04-16T21:11:59.501597+0000","last_peered":"2024-04-16T21:11:59.501597+0000","last_clean":"2024-04-16T21:11:59.501597+0000","last_became_active":"2024-04-16T19:25:09.458291+0000","last_became_peered":"2024-04-16T19:25:09.458291+0000","last_unstale":"2024-04-16T21:11:59.501597+0000","last_undegraded":"2024-04-16T21:11:59.501597+0000","last_fullsized":"2024-04-16T21:11:59.501597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:59.501496+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.501496+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:44:51.681807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":59,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.856833+0000","last_change":"2024-04-16T21:12:29.856857+0000","last_active":"2024-04-16T21:12:29.856833+0000","last_peered":"2024-04-16T21:12:29.856833+0000","last_clean":"2024-04-16T21:12:29.856833+0000","last_became_active":"2024-04-16T19:24:35.395729+0000","last_became_peered":"2024-04-16T19:24:35.395729+0000","last_unstale":"2024-04-16T21:12:29.856833+0000","last_undegraded":"2024-04-16T21:12:29.856833+0000","last_fullsized":"2024-04-16T21:12:29.856833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.856754+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.856754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:25.114578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.489060+0000","last_change":"2024-04-16T21:13:03.489086+0000","last_active":"2024-04-16T21:13:03.489060+0000","last_peered":"2024-04-16T21:13:03.489060+0000","last_clean":"2024-04-16T21:13:03.489060+0000","last_became_active":"2024-04-16T19:24:33.385970+0000","last_became_peered":"2024-04-16T19:24:33.385970+0000","last_unstale":"2024-04-16T21:13:03.489060+0000","last_undegraded":"2024-04-16T21:13:03.489060+0000","last_fullsized":"2024-04-16T21:13:03.489060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.488997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.488997+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.488997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:16.660558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"52'6","reported_seq":643,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.083593+0000","last_change":"2024-04-16T21:12:51.083624+0000","last_active":"2024-04-16T21:12:51.083593+0000","last_peered":"2024-04-16T21:12:51.083593+0000","last_clean":"2024-04-16T21:12:51.083593+0000","last_became_active":"2024-04-16T19:24:45.411068+0000","last_became_peered":"2024-04-16T19:24:45.411068+0000","last_unstale":"2024-04-16T21:12:51.083593+0000","last_undegraded":"2024-04-16T21:12:51.083593+0000","last_fullsized":"2024-04-16T21:12:51.083593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:51.083530+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:51.083530+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.083530+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:47:05.662069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3297,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":652,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"52'115","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.359243+0000","last_change":"2024-04-16T21:12:34.359243+0000","last_active":"2024-04-16T21:12:34.359243+0000","last_peered":"2024-04-16T21:12:34.359243+0000","last_clean":"2024-04-16T21:12:34.359243+0000","last_became_active":"2024-04-16T19:24:47.418886+0000","last_became_peered":"2024-04-16T19:24:47.418886+0000","last_unstale":"2024-04-16T21:12:34.359243+0000","last_undegraded":"2024-04-16T21:12:34.359243+0000","last_fullsized":"2024-04-16T21:12:34.359243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'115","last_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_deep_scrub":"52'115","last_deep_scrub_stamp":"2024-04-16T21:12:34.359152+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.359152+0000","objects_scrubbed":5,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:57.921414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":160,"num_write":110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"52'13","reported_seq":1326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:28.729186+0000","last_change":"2024-04-16T21:12:28.729186+0000","last_active":"2024-04-16T21:12:28.729186+0000","last_peered":"2024-04-16T21:12:28.729186+0000","last_clean":"2024-04-16T21:12:28.729186+0000","last_became_active":"2024-04-16T19:24:49.432896+0000","last_became_peered":"2024-04-16T19:24:49.432896+0000","last_unstale":"2024-04-16T21:12:28.729186+0000","last_undegraded":"2024-04-16T21:12:28.729186+0000","last_fullsized":"2024-04-16T21:12:28.729186+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:28.729061+0000","last_clean_scrub_stamp":"2024-04-16T21:12:28.729061+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:24:00.415763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.424596+0000","last_change":"2024-04-16T21:12:05.424596+0000","last_active":"2024-04-16T21:12:05.424596+0000","last_peered":"2024-04-16T21:12:05.424596+0000","last_clean":"2024-04-16T21:12:05.424596+0000","last_became_active":"2024-04-16T19:24:51.437864+0000","last_became_peered":"2024-04-16T19:24:51.437864+0000","last_unstale":"2024-04-16T21:12:05.424596+0000","last_undegraded":"2024-04-16T21:12:05.424596+0000","last_fullsized":"2024-04-16T21:12:05.424596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.424465+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.424465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:23:59.031445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.849953+0000","last_change":"2024-04-16T21:12:59.849982+0000","last_active":"2024-04-16T21:12:59.849953+0000","last_peered":"2024-04-16T21:12:59.849953+0000","last_clean":"2024-04-16T21:12:59.849953+0000","last_became_active":"2024-04-16T19:47:33.983443+0000","last_became_peered":"2024-04-16T19:47:33.983443+0000","last_unstale":"2024-04-16T21:12:59.849953+0000","last_undegraded":"2024-04-16T21:12:59.849953+0000","last_fullsized":"2024-04-16T21:12:59.849953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.849879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.849879+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.849879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:31.554672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"52'274","reported_seq":2376,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.635594+0000","last_change":"2024-04-16T21:12:25.635594+0000","last_active":"2024-04-16T21:12:25.635594+0000","last_peered":"2024-04-16T21:12:25.635594+0000","last_clean":"2024-04-16T21:12:25.635594+0000","last_became_active":"2024-04-16T19:25:49.518658+0000","last_became_peered":"2024-04-16T19:25:49.518658+0000","last_unstale":"2024-04-16T21:12:25.635594+0000","last_undegraded":"2024-04-16T21:12:25.635594+0000","last_fullsized":"2024-04-16T21:12:25.635594+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'274","last_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_deep_scrub":"52'274","last_deep_scrub_stamp":"2024-04-16T21:12:25.635495+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.635495+0000","objects_scrubbed":60,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:42.021428+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3456,"num_read_kb":3520,"num_write":316,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17559,"num_omap_keys":71,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:18.466844+0000","last_change":"2024-04-16T21:12:18.466844+0000","last_active":"2024-04-16T21:12:18.466844+0000","last_peered":"2024-04-16T21:12:18.466844+0000","last_clean":"2024-04-16T21:12:18.466844+0000","last_became_active":"2024-04-16T19:25:41.508637+0000","last_became_peered":"2024-04-16T19:25:41.508637+0000","last_unstale":"2024-04-16T21:12:18.466844+0000","last_undegraded":"2024-04-16T21:12:18.466844+0000","last_fullsized":"2024-04-16T21:12:18.466844+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:18.466741+0000","last_clean_scrub_stamp":"2024-04-16T21:12:18.466741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:23:50.153442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"52'17","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.900906+0000","last_change":"2024-04-16T21:12:12.900906+0000","last_active":"2024-04-16T21:12:12.900906+0000","last_peered":"2024-04-16T21:12:12.900906+0000","last_clean":"2024-04-16T21:12:12.900906+0000","last_became_active":"2024-04-16T19:25:15.479828+0000","last_became_peered":"2024-04-16T19:25:15.479828+0000","last_unstale":"2024-04-16T21:12:12.900906+0000","last_undegraded":"2024-04-16T21:12:12.900906+0000","last_fullsized":"2024-04-16T21:12:12.900906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:12.900807+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.900807+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:00.936096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2025,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":115,"num_write":28,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"52'706","reported_seq":2133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:25.326509+0000","last_change":"2024-04-16T21:12:25.326509+0000","last_active":"2024-04-16T21:12:25.326509+0000","last_peered":"2024-04-16T21:12:25.326509+0000","last_clean":"2024-04-16T21:12:25.326509+0000","last_became_active":"2024-04-16T19:25:13.473437+0000","last_became_peered":"2024-04-16T19:25:13.473437+0000","last_unstale":"2024-04-16T21:12:25.326509+0000","last_undegraded":"2024-04-16T21:12:25.326509+0000","last_fullsized":"2024-04-16T21:12:25.326509+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:25.326417+0000","last_clean_scrub_stamp":"2024-04-16T21:12:25.326417+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:25.301383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"52'557","reported_seq":1382,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:50.205288+0000","last_change":"2024-04-16T21:11:50.205349+0000","last_active":"2024-04-16T21:11:50.205288+0000","last_peered":"2024-04-16T21:11:50.205288+0000","last_clean":"2024-04-16T21:11:50.205288+0000","last_became_active":"2024-04-16T19:25:11.465275+0000","last_became_peered":"2024-04-16T19:25:11.465275+0000","last_unstale":"2024-04-16T21:11:50.205288+0000","last_undegraded":"2024-04-16T21:11:50.205288+0000","last_fullsized":"2024-04-16T21:11:50.205288+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'557","last_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_deep_scrub":"52'557","last_deep_scrub_stamp":"2024-04-16T21:11:50.205222+0000","last_clean_scrub_stamp":"2024-04-16T21:11:50.205222+0000","objects_scrubbed":17,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:09.070331+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35459,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2043,"num_read_kb":1809,"num_write":810,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2200,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"52'17","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.702594+0000","last_change":"2024-04-16T21:12:11.702662+0000","last_active":"2024-04-16T21:12:11.702594+0000","last_peered":"2024-04-16T21:12:11.702594+0000","last_clean":"2024-04-16T21:12:11.702594+0000","last_became_active":"2024-04-16T19:25:09.457896+0000","last_became_peered":"2024-04-16T19:25:09.457896+0000","last_unstale":"2024-04-16T21:12:11.702594+0000","last_undegraded":"2024-04-16T21:12:11.702594+0000","last_fullsized":"2024-04-16T21:12:11.702594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:12:11.702519+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.702519+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:28:33.204105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":287,"num_read_kb":186,"num_write":81,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.746552+0000","last_change":"2024-04-16T21:12:44.746576+0000","last_active":"2024-04-16T21:12:44.746552+0000","last_peered":"2024-04-16T21:12:44.746552+0000","last_clean":"2024-04-16T21:12:44.746552+0000","last_became_active":"2024-04-16T19:24:35.397156+0000","last_became_peered":"2024-04-16T19:24:35.397156+0000","last_unstale":"2024-04-16T21:12:44.746552+0000","last_undegraded":"2024-04-16T21:12:44.746552+0000","last_fullsized":"2024-04-16T21:12:44.746552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.746485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.746485+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.746485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:22:15.574936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:09.710568+0000","last_change":"2024-04-16T21:13:09.710588+0000","last_active":"2024-04-16T21:13:09.710568+0000","last_peered":"2024-04-16T21:13:09.710568+0000","last_clean":"2024-04-16T21:13:09.710568+0000","last_became_active":"2024-04-16T19:24:33.385552+0000","last_became_peered":"2024-04-16T19:24:33.385552+0000","last_unstale":"2024-04-16T21:13:09.710568+0000","last_undegraded":"2024-04-16T21:13:09.710568+0000","last_fullsized":"2024-04-16T21:13:09.710568+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:09.710504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:09.710504+0000","last_clean_scrub_stamp":"2024-04-16T21:13:09.710504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:38:49.888988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.762674+0000","last_change":"2024-04-16T21:12:43.762696+0000","last_active":"2024-04-16T21:12:43.762674+0000","last_peered":"2024-04-16T21:12:43.762674+0000","last_clean":"2024-04-16T21:12:43.762674+0000","last_became_active":"2024-04-16T19:24:45.412128+0000","last_became_peered":"2024-04-16T19:24:45.412128+0000","last_unstale":"2024-04-16T21:12:43.762674+0000","last_undegraded":"2024-04-16T21:12:43.762674+0000","last_fullsized":"2024-04-16T21:12:43.762674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:43.762572+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.762572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:58.032922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"52'147","reported_seq":339,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:34.855560+0000","last_change":"2024-04-16T21:12:34.855594+0000","last_active":"2024-04-16T21:12:34.855560+0000","last_peered":"2024-04-16T21:12:34.855560+0000","last_clean":"2024-04-16T21:12:34.855560+0000","last_became_active":"2024-04-16T19:24:47.421792+0000","last_became_peered":"2024-04-16T19:24:47.421792+0000","last_unstale":"2024-04-16T21:12:34.855560+0000","last_undegraded":"2024-04-16T21:12:34.855560+0000","last_fullsized":"2024-04-16T21:12:34.855560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'147","last_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_deep_scrub":"52'147","last_deep_scrub_stamp":"2024-04-16T21:12:34.855500+0000","last_clean_scrub_stamp":"2024-04-16T21:12:34.855500+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:22.893155+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":215,"num_read_kb":206,"num_write":140,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.297506+0000","last_change":"2024-04-16T21:12:29.297528+0000","last_active":"2024-04-16T21:12:29.297506+0000","last_peered":"2024-04-16T21:12:29.297506+0000","last_clean":"2024-04-16T21:12:29.297506+0000","last_became_active":"2024-04-16T19:24:49.427641+0000","last_became_peered":"2024-04-16T19:24:49.427641+0000","last_unstale":"2024-04-16T21:12:29.297506+0000","last_undegraded":"2024-04-16T21:12:29.297506+0000","last_fullsized":"2024-04-16T21:12:29.297506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.297434+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.297434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:45.096730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:07.443393+0000","last_change":"2024-04-16T21:12:07.443416+0000","last_active":"2024-04-16T21:12:07.443393+0000","last_peered":"2024-04-16T21:12:07.443393+0000","last_clean":"2024-04-16T21:12:07.443393+0000","last_became_active":"2024-04-16T19:24:51.437535+0000","last_became_peered":"2024-04-16T19:24:51.437535+0000","last_unstale":"2024-04-16T21:12:07.443393+0000","last_undegraded":"2024-04-16T21:12:07.443393+0000","last_fullsized":"2024-04-16T21:12:07.443393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:07.443331+0000","last_clean_scrub_stamp":"2024-04-16T21:12:07.443331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:04:23.829264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.706740+0000","last_change":"2024-04-16T21:12:53.706762+0000","last_active":"2024-04-16T21:12:53.706740+0000","last_peered":"2024-04-16T21:12:53.706740+0000","last_clean":"2024-04-16T21:12:53.706740+0000","last_became_active":"2024-04-16T19:47:33.988052+0000","last_became_peered":"2024-04-16T19:47:33.988052+0000","last_unstale":"2024-04-16T21:12:53.706740+0000","last_undegraded":"2024-04-16T21:12:53.706740+0000","last_fullsized":"2024-04-16T21:12:53.706740+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.706674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.706674+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.706674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:48:53.668881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"52'159","reported_seq":1391,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.133278+0000","last_change":"2024-04-16T21:13:02.133278+0000","last_active":"2024-04-16T21:13:02.133278+0000","last_peered":"2024-04-16T21:13:02.133278+0000","last_clean":"2024-04-16T21:13:02.133278+0000","last_became_active":"2024-04-16T19:25:49.524476+0000","last_became_peered":"2024-04-16T19:25:49.524476+0000","last_unstale":"2024-04-16T21:13:02.133278+0000","last_undegraded":"2024-04-16T21:13:02.133278+0000","last_fullsized":"2024-04-16T21:13:02.133278+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'159","last_scrub_stamp":"2024-04-16T21:13:02.133177+0000","last_deep_scrub":"52'159","last_deep_scrub_stamp":"2024-04-16T21:13:02.133177+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.133177+0000","objects_scrubbed":43,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:39:10.287136+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1954,"num_read_kb":1906,"num_write":156,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8422,"num_omap_keys":34,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:33.850776+0000","last_change":"2024-04-16T21:12:33.850776+0000","last_active":"2024-04-16T21:12:33.850776+0000","last_peered":"2024-04-16T21:12:33.850776+0000","last_clean":"2024-04-16T21:12:33.850776+0000","last_became_active":"2024-04-16T19:25:41.507471+0000","last_became_peered":"2024-04-16T19:25:41.507471+0000","last_unstale":"2024-04-16T21:12:33.850776+0000","last_undegraded":"2024-04-16T21:12:33.850776+0000","last_fullsized":"2024-04-16T21:12:33.850776+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:33.850667+0000","last_clean_scrub_stamp":"2024-04-16T21:12:33.850667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:48.121017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"52'10","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:23.431983+0000","last_change":"2024-04-16T21:12:23.431983+0000","last_active":"2024-04-16T21:12:23.431983+0000","last_peered":"2024-04-16T21:12:23.431983+0000","last_clean":"2024-04-16T21:12:23.431983+0000","last_became_active":"2024-04-16T19:25:15.481396+0000","last_became_peered":"2024-04-16T19:25:15.481396+0000","last_unstale":"2024-04-16T21:12:23.431983+0000","last_undegraded":"2024-04-16T21:12:23.431983+0000","last_fullsized":"2024-04-16T21:12:23.431983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:23.431886+0000","last_clean_scrub_stamp":"2024-04-16T21:12:23.431886+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:37.702235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":15,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.15","version":"52'706","reported_seq":2082,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.853855+0000","last_change":"2024-04-16T21:12:01.853885+0000","last_active":"2024-04-16T21:12:01.853855+0000","last_peered":"2024-04-16T21:12:01.853855+0000","last_clean":"2024-04-16T21:12:01.853855+0000","last_became_active":"2024-04-16T19:25:13.474400+0000","last_became_peered":"2024-04-16T19:25:13.474400+0000","last_unstale":"2024-04-16T21:12:01.853855+0000","last_undegraded":"2024-04-16T21:12:01.853855+0000","last_fullsized":"2024-04-16T21:12:01.853855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'706","last_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_deep_scrub":"52'706","last_deep_scrub_stamp":"2024-04-16T21:12:01.853792+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.853792+0000","objects_scrubbed":1,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:14:14.601423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"52'542","reported_seq":41300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.597266+0000","last_change":"2024-04-16T21:11:54.597266+0000","last_active":"2024-04-16T21:11:54.597266+0000","last_peered":"2024-04-16T21:11:54.597266+0000","last_clean":"2024-04-16T21:11:54.597266+0000","last_became_active":"2024-04-16T19:25:11.464720+0000","last_became_peered":"2024-04-16T19:25:11.464720+0000","last_unstale":"2024-04-16T21:11:54.597266+0000","last_undegraded":"2024-04-16T21:11:54.597266+0000","last_fullsized":"2024-04-16T21:11:54.597266+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'542","last_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_deep_scrub":"52'542","last_deep_scrub_stamp":"2024-04-16T21:11:54.597093+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.597093+0000","objects_scrubbed":23,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:26:16.539182+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":122619,"num_read_kb":81813,"num_write":452,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"52'18","reported_seq":326,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:54.434844+0000","last_change":"2024-04-16T21:11:54.434914+0000","last_active":"2024-04-16T21:11:54.434844+0000","last_peered":"2024-04-16T21:11:54.434844+0000","last_clean":"2024-04-16T21:11:54.434844+0000","last_became_active":"2024-04-16T19:25:09.459653+0000","last_became_peered":"2024-04-16T19:25:09.459653+0000","last_unstale":"2024-04-16T21:11:54.434844+0000","last_undegraded":"2024-04-16T21:11:54.434844+0000","last_fullsized":"2024-04-16T21:11:54.434844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:11:54.434778+0000","last_clean_scrub_stamp":"2024-04-16T21:11:54.434778+0000","objects_scrubbed":9,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:53.824584+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":522,"num_read_kb":334,"num_write":93,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":129,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.707330+0000","last_change":"2024-04-16T21:12:36.707352+0000","last_active":"2024-04-16T21:12:36.707330+0000","last_peered":"2024-04-16T21:12:36.707330+0000","last_clean":"2024-04-16T21:12:36.707330+0000","last_became_active":"2024-04-16T19:24:35.400158+0000","last_became_peered":"2024-04-16T19:24:35.400158+0000","last_unstale":"2024-04-16T21:12:36.707330+0000","last_undegraded":"2024-04-16T21:12:36.707330+0000","last_fullsized":"2024-04-16T21:12:36.707330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:36.707266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.707266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:59:58.955541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:08.713883+0000","last_change":"2024-04-16T21:13:08.713904+0000","last_active":"2024-04-16T21:13:08.713883+0000","last_peered":"2024-04-16T21:13:08.713883+0000","last_clean":"2024-04-16T21:13:08.713883+0000","last_became_active":"2024-04-16T19:24:33.384273+0000","last_became_peered":"2024-04-16T19:24:33.384273+0000","last_unstale":"2024-04-16T21:13:08.713883+0000","last_undegraded":"2024-04-16T21:13:08.713883+0000","last_fullsized":"2024-04-16T21:13:08.713883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:08.713824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:08.713824+0000","last_clean_scrub_stamp":"2024-04-16T21:13:08.713824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:12:40.738369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"52'13","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:43.860330+0000","last_change":"2024-04-16T21:12:43.860357+0000","last_active":"2024-04-16T21:12:43.860330+0000","last_peered":"2024-04-16T21:12:43.860330+0000","last_clean":"2024-04-16T21:12:43.860330+0000","last_became_active":"2024-04-16T19:24:45.414409+0000","last_became_peered":"2024-04-16T19:24:45.414409+0000","last_unstale":"2024-04-16T21:12:43.860330+0000","last_undegraded":"2024-04-16T21:12:43.860330+0000","last_fullsized":"2024-04-16T21:12:43.860330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:12:43.860269+0000","last_clean_scrub_stamp":"2024-04-16T21:12:43.860269+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:18:26.916002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":24,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"52'78","reported_seq":237,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:39.311803+0000","last_change":"2024-04-16T21:12:39.311841+0000","last_active":"2024-04-16T21:12:39.311803+0000","last_peered":"2024-04-16T21:12:39.311803+0000","last_clean":"2024-04-16T21:12:39.311803+0000","last_became_active":"2024-04-16T19:24:47.420515+0000","last_became_peered":"2024-04-16T19:24:47.420515+0000","last_unstale":"2024-04-16T21:12:39.311803+0000","last_undegraded":"2024-04-16T21:12:39.311803+0000","last_fullsized":"2024-04-16T21:12:39.311803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:39.311721+0000","last_clean_scrub_stamp":"2024-04-16T21:12:39.311721+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:17:11.016514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":113,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:29.303819+0000","last_change":"2024-04-16T21:12:29.303842+0000","last_active":"2024-04-16T21:12:29.303819+0000","last_peered":"2024-04-16T21:12:29.303819+0000","last_clean":"2024-04-16T21:12:29.303819+0000","last_became_active":"2024-04-16T19:24:49.429669+0000","last_became_peered":"2024-04-16T19:24:49.429669+0000","last_unstale":"2024-04-16T21:12:29.303819+0000","last_undegraded":"2024-04-16T21:12:29.303819+0000","last_fullsized":"2024-04-16T21:12:29.303819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:29.303752+0000","last_clean_scrub_stamp":"2024-04-16T21:12:29.303752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:17.910045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.776294+0000","last_change":"2024-04-16T21:12:00.776335+0000","last_active":"2024-04-16T21:12:00.776294+0000","last_peered":"2024-04-16T21:12:00.776294+0000","last_clean":"2024-04-16T21:12:00.776294+0000","last_became_active":"2024-04-16T19:24:51.573685+0000","last_became_peered":"2024-04-16T19:24:51.573685+0000","last_unstale":"2024-04-16T21:12:00.776294+0000","last_undegraded":"2024-04-16T21:12:00.776294+0000","last_fullsized":"2024-04-16T21:12:00.776294+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:00.776223+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.776223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:20.604086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:13.376301+0000","last_change":"2024-04-16T21:13:13.376325+0000","last_active":"2024-04-16T21:13:13.376301+0000","last_peered":"2024-04-16T21:13:13.376301+0000","last_clean":"2024-04-16T21:13:13.376301+0000","last_became_active":"2024-04-16T19:47:33.979748+0000","last_became_peered":"2024-04-16T19:47:33.979748+0000","last_unstale":"2024-04-16T21:13:13.376301+0000","last_undegraded":"2024-04-16T21:13:13.376301+0000","last_fullsized":"2024-04-16T21:13:13.376301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:13.376234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:13.376234+0000","last_clean_scrub_stamp":"2024-04-16T21:13:13.376234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:31:31.865036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.13","version":"52'222","reported_seq":2178,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.980550+0000","last_change":"2024-04-16T21:12:16.980550+0000","last_active":"2024-04-16T21:12:16.980550+0000","last_peered":"2024-04-16T21:12:16.980550+0000","last_clean":"2024-04-16T21:12:16.980550+0000","last_became_active":"2024-04-16T19:25:49.518849+0000","last_became_peered":"2024-04-16T19:25:49.518849+0000","last_unstale":"2024-04-16T21:12:16.980550+0000","last_undegraded":"2024-04-16T21:12:16.980550+0000","last_fullsized":"2024-04-16T21:12:16.980550+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'222","last_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_deep_scrub":"52'222","last_deep_scrub_stamp":"2024-04-16T21:12:16.980456+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.980456+0000","objects_scrubbed":55,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:41.646543+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":3169,"num_read_kb":3193,"num_write":245,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14526,"num_omap_keys":59,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:10.913891+0000","last_change":"2024-04-16T21:12:10.913891+0000","last_active":"2024-04-16T21:12:10.913891+0000","last_peered":"2024-04-16T21:12:10.913891+0000","last_clean":"2024-04-16T21:12:10.913891+0000","last_became_active":"2024-04-16T19:25:41.506992+0000","last_became_peered":"2024-04-16T19:25:41.506992+0000","last_unstale":"2024-04-16T21:12:10.913891+0000","last_undegraded":"2024-04-16T21:12:10.913891+0000","last_fullsized":"2024-04-16T21:12:10.913891+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:10.913806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:10.913806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:11:17.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"52'24","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:26.799425+0000","last_change":"2024-04-16T21:12:26.799425+0000","last_active":"2024-04-16T21:12:26.799425+0000","last_peered":"2024-04-16T21:12:26.799425+0000","last_clean":"2024-04-16T21:12:26.799425+0000","last_became_active":"2024-04-16T19:25:15.481597+0000","last_became_peered":"2024-04-16T19:25:15.481597+0000","last_unstale":"2024-04-16T21:12:26.799425+0000","last_undegraded":"2024-04-16T21:12:26.799425+0000","last_fullsized":"2024-04-16T21:12:26.799425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'24","last_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_deep_scrub":"52'24","last_deep_scrub_stamp":"2024-04-16T21:12:26.799297+0000","last_clean_scrub_stamp":"2024-04-16T21:12:26.799297+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:53:18.472986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1607,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":49,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":1,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:59.652503+0000","last_change":"2024-04-16T21:11:59.652525+0000","last_active":"2024-04-16T21:11:59.652503+0000","last_peered":"2024-04-16T21:11:59.652503+0000","last_clean":"2024-04-16T21:11:59.652503+0000","last_became_active":"2024-04-16T19:25:13.471974+0000","last_became_peered":"2024-04-16T19:25:13.471974+0000","last_unstale":"2024-04-16T21:11:59.652503+0000","last_undegraded":"2024-04-16T21:11:59.652503+0000","last_fullsized":"2024-04-16T21:11:59.652503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:11:59.652420+0000","last_clean_scrub_stamp":"2024-04-16T21:11:59.652420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:38:18.517529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"52'592","reported_seq":1363,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:00.540243+0000","last_change":"2024-04-16T21:12:00.540298+0000","last_active":"2024-04-16T21:12:00.540243+0000","last_peered":"2024-04-16T21:12:00.540243+0000","last_clean":"2024-04-16T21:12:00.540243+0000","last_became_active":"2024-04-16T19:25:11.468632+0000","last_became_peered":"2024-04-16T19:25:11.468632+0000","last_unstale":"2024-04-16T21:12:00.540243+0000","last_undegraded":"2024-04-16T21:12:00.540243+0000","last_fullsized":"2024-04-16T21:12:00.540243+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'592","last_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_deep_scrub":"52'592","last_deep_scrub_stamp":"2024-04-16T21:12:00.540183+0000","last_clean_scrub_stamp":"2024-04-16T21:12:00.540183+0000","objects_scrubbed":19,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:48.575472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46991,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2098,"num_read_kb":1793,"num_write":1014,"num_write_kb":675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"52'32","reported_seq":340,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:43.552192+0000","last_change":"2024-04-16T21:11:43.552192+0000","last_active":"2024-04-16T21:11:43.552192+0000","last_peered":"2024-04-16T21:11:43.552192+0000","last_clean":"2024-04-16T21:11:43.552192+0000","last_became_active":"2024-04-16T19:25:09.468773+0000","last_became_peered":"2024-04-16T19:25:09.468773+0000","last_unstale":"2024-04-16T21:11:43.552192+0000","last_undegraded":"2024-04-16T21:11:43.552192+0000","last_fullsized":"2024-04-16T21:11:43.552192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:11:43.552108+0000","last_clean_scrub_stamp":"2024-04-16T21:11:43.552108+0000","objects_scrubbed":9,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:10.135567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":542,"num_read_kb":360,"num_write":179,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.442869+0000","last_change":"2024-04-16T21:12:53.442890+0000","last_active":"2024-04-16T21:12:53.442869+0000","last_peered":"2024-04-16T21:12:53.442869+0000","last_clean":"2024-04-16T21:12:53.442869+0000","last_became_active":"2024-04-16T19:24:35.389734+0000","last_became_peered":"2024-04-16T19:24:35.389734+0000","last_unstale":"2024-04-16T21:12:53.442869+0000","last_undegraded":"2024-04-16T21:12:53.442869+0000","last_fullsized":"2024-04-16T21:12:53.442869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.442802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.442802+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.442802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:20:51.858172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.797312+0000","last_change":"2024-04-16T21:12:45.797312+0000","last_active":"2024-04-16T21:12:45.797312+0000","last_peered":"2024-04-16T21:12:45.797312+0000","last_clean":"2024-04-16T21:12:45.797312+0000","last_became_active":"2024-04-16T19:24:33.387515+0000","last_became_peered":"2024-04-16T19:24:33.387515+0000","last_unstale":"2024-04-16T21:12:45.797312+0000","last_undegraded":"2024-04-16T21:12:45.797312+0000","last_fullsized":"2024-04-16T21:12:45.797312+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:45.797229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:45.797229+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.797229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:44:33.834884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"52'5","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:45.725079+0000","last_change":"2024-04-16T21:12:45.725111+0000","last_active":"2024-04-16T21:12:45.725079+0000","last_peered":"2024-04-16T21:12:45.725079+0000","last_clean":"2024-04-16T21:12:45.725079+0000","last_became_active":"2024-04-16T19:24:45.413805+0000","last_became_peered":"2024-04-16T19:24:45.413805+0000","last_unstale":"2024-04-16T21:12:45.725079+0000","last_undegraded":"2024-04-16T21:12:45.725079+0000","last_fullsized":"2024-04-16T21:12:45.725079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'5","last_scrub_stamp":"2024-04-16T21:12:45.725020+0000","last_deep_scrub":"52'5","last_deep_scrub_stamp":"2024-04-16T21:12:45.725020+0000","last_clean_scrub_stamp":"2024-04-16T21:12:45.725020+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:33:32.022824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"52'32","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.284966+0000","last_change":"2024-04-16T21:12:36.284996+0000","last_active":"2024-04-16T21:12:36.284966+0000","last_peered":"2024-04-16T21:12:36.284966+0000","last_clean":"2024-04-16T21:12:36.284966+0000","last_became_active":"2024-04-16T19:24:47.422065+0000","last_became_peered":"2024-04-16T19:24:47.422065+0000","last_unstale":"2024-04-16T21:12:36.284966+0000","last_undegraded":"2024-04-16T21:12:36.284966+0000","last_fullsized":"2024-04-16T21:12:36.284966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'32","last_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_deep_scrub":"52'32","last_deep_scrub_stamp":"2024-04-16T21:12:36.284903+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.284903+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:46:50.863442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":46,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.384509+0000","last_change":"2024-04-16T21:12:11.384509+0000","last_active":"2024-04-16T21:12:11.384509+0000","last_peered":"2024-04-16T21:12:11.384509+0000","last_clean":"2024-04-16T21:12:11.384509+0000","last_became_active":"2024-04-16T19:24:49.431063+0000","last_became_peered":"2024-04-16T19:24:49.431063+0000","last_unstale":"2024-04-16T21:12:11.384509+0000","last_undegraded":"2024-04-16T21:12:11.384509+0000","last_fullsized":"2024-04-16T21:12:11.384509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.384424+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.384424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:47:06.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":109,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.514240+0000","last_change":"2024-04-16T21:12:02.514262+0000","last_active":"2024-04-16T21:12:02.514240+0000","last_peered":"2024-04-16T21:12:02.514240+0000","last_clean":"2024-04-16T21:12:02.514240+0000","last_became_active":"2024-04-16T19:24:51.444645+0000","last_became_peered":"2024-04-16T19:24:51.444645+0000","last_unstale":"2024-04-16T21:12:02.514240+0000","last_undegraded":"2024-04-16T21:12:02.514240+0000","last_fullsized":"2024-04-16T21:12:02.514240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:02.514179+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.514179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:26:44.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":61,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:06.019384+0000","last_change":"2024-04-16T21:13:06.019427+0000","last_active":"2024-04-16T21:13:06.019384+0000","last_peered":"2024-04-16T21:13:06.019384+0000","last_clean":"2024-04-16T21:13:06.019384+0000","last_became_active":"2024-04-16T19:47:33.986925+0000","last_became_peered":"2024-04-16T19:47:33.986925+0000","last_unstale":"2024-04-16T21:13:06.019384+0000","last_undegraded":"2024-04-16T21:13:06.019384+0000","last_fullsized":"2024-04-16T21:13:06.019384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:06.019305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:06.019305+0000","last_clean_scrub_stamp":"2024-04-16T21:13:06.019305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:22.354360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"52'172","reported_seq":1565,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.949215+0000","last_change":"2024-04-16T21:12:12.949215+0000","last_active":"2024-04-16T21:12:12.949215+0000","last_peered":"2024-04-16T21:12:12.949215+0000","last_clean":"2024-04-16T21:12:12.949215+0000","last_became_active":"2024-04-16T19:25:49.516679+0000","last_became_peered":"2024-04-16T19:25:49.516679+0000","last_unstale":"2024-04-16T21:12:12.949215+0000","last_undegraded":"2024-04-16T21:12:12.949215+0000","last_fullsized":"2024-04-16T21:12:12.949215+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'172","last_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_deep_scrub":"52'172","last_deep_scrub_stamp":"2024-04-16T21:12:12.949100+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.949100+0000","objects_scrubbed":45,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.338906+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2284,"num_read_kb":2233,"num_write":170,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8810,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":87,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:09.869596+0000","last_change":"2024-04-16T21:12:09.869596+0000","last_active":"2024-04-16T21:12:09.869596+0000","last_peered":"2024-04-16T21:12:09.869596+0000","last_clean":"2024-04-16T21:12:09.869596+0000","last_became_active":"2024-04-16T19:25:41.502683+0000","last_became_peered":"2024-04-16T19:25:41.502683+0000","last_unstale":"2024-04-16T21:12:09.869596+0000","last_undegraded":"2024-04-16T21:12:09.869596+0000","last_fullsized":"2024-04-16T21:12:09.869596+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:09.869508+0000","last_clean_scrub_stamp":"2024-04-16T21:12:09.869508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:34.535595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"52'16","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:21.491403+0000","last_change":"2024-04-16T21:12:21.491403+0000","last_active":"2024-04-16T21:12:21.491403+0000","last_peered":"2024-04-16T21:12:21.491403+0000","last_clean":"2024-04-16T21:12:21.491403+0000","last_became_active":"2024-04-16T19:25:15.481652+0000","last_became_peered":"2024-04-16T19:25:15.481652+0000","last_unstale":"2024-04-16T21:12:21.491403+0000","last_undegraded":"2024-04-16T21:12:21.491403+0000","last_fullsized":"2024-04-16T21:12:21.491403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'16","last_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_deep_scrub":"52'16","last_deep_scrub_stamp":"2024-04-16T21:12:21.491306+0000","last_clean_scrub_stamp":"2024-04-16T21:12:21.491306+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:09.623796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":137,"num_write":30,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":95,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:01.693490+0000","last_change":"2024-04-16T21:12:01.693513+0000","last_active":"2024-04-16T21:12:01.693490+0000","last_peered":"2024-04-16T21:12:01.693490+0000","last_clean":"2024-04-16T21:12:01.693490+0000","last_became_active":"2024-04-16T19:25:13.473571+0000","last_became_peered":"2024-04-16T19:25:13.473571+0000","last_unstale":"2024-04-16T21:12:01.693490+0000","last_undegraded":"2024-04-16T21:12:01.693490+0000","last_fullsized":"2024-04-16T21:12:01.693490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:01.693430+0000","last_clean_scrub_stamp":"2024-04-16T21:12:01.693430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:41:04.866318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"52'249","reported_seq":676,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:48.401831+0000","last_change":"2024-04-16T21:11:48.401911+0000","last_active":"2024-04-16T21:11:48.401831+0000","last_peered":"2024-04-16T21:11:48.401831+0000","last_clean":"2024-04-16T21:11:48.401831+0000","last_became_active":"2024-04-16T19:25:11.460886+0000","last_became_peered":"2024-04-16T19:25:11.460886+0000","last_unstale":"2024-04-16T21:11:48.401831+0000","last_undegraded":"2024-04-16T21:11:48.401831+0000","last_fullsized":"2024-04-16T21:11:48.401831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'249","last_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_deep_scrub":"52'249","last_deep_scrub_stamp":"2024-04-16T21:11:48.401765+0000","last_clean_scrub_stamp":"2024-04-16T21:11:48.401765+0000","objects_scrubbed":19,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:23:02.153361+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":587,"num_read_kb":578,"num_write":302,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"52'12","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.683222+0000","last_change":"2024-04-16T21:12:12.683275+0000","last_active":"2024-04-16T21:12:12.683222+0000","last_peered":"2024-04-16T21:12:12.683222+0000","last_clean":"2024-04-16T21:12:12.683222+0000","last_became_active":"2024-04-16T19:25:09.457750+0000","last_became_peered":"2024-04-16T19:25:09.457750+0000","last_unstale":"2024-04-16T21:12:12.683222+0000","last_undegraded":"2024-04-16T21:12:12.683222+0000","last_fullsized":"2024-04-16T21:12:12.683222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:12:12.683160+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.683160+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:19:12.072966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":160,"num_write":77,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:10.154939+0000","last_change":"2024-04-16T21:13:10.154939+0000","last_active":"2024-04-16T21:13:10.154939+0000","last_peered":"2024-04-16T21:13:10.154939+0000","last_clean":"2024-04-16T21:13:10.154939+0000","last_became_active":"2024-04-16T19:24:35.396302+0000","last_became_peered":"2024-04-16T19:24:35.396302+0000","last_unstale":"2024-04-16T21:13:10.154939+0000","last_undegraded":"2024-04-16T21:13:10.154939+0000","last_fullsized":"2024-04-16T21:13:10.154939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:10.154846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:10.154846+0000","last_clean_scrub_stamp":"2024-04-16T21:13:10.154846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:08:59.750446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.501423+0000","last_change":"2024-04-16T21:12:59.501448+0000","last_active":"2024-04-16T21:12:59.501423+0000","last_peered":"2024-04-16T21:12:59.501423+0000","last_clean":"2024-04-16T21:12:59.501423+0000","last_became_active":"2024-04-16T19:24:33.389064+0000","last_became_peered":"2024-04-16T19:24:33.389064+0000","last_unstale":"2024-04-16T21:12:59.501423+0000","last_undegraded":"2024-04-16T21:12:59.501423+0000","last_fullsized":"2024-04-16T21:12:59.501423+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.501352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.501352+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.501352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:42.659929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"52'4","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:36.017072+0000","last_change":"2024-04-16T21:12:36.017103+0000","last_active":"2024-04-16T21:12:36.017072+0000","last_peered":"2024-04-16T21:12:36.017072+0000","last_clean":"2024-04-16T21:12:36.017072+0000","last_became_active":"2024-04-16T19:24:45.413773+0000","last_became_peered":"2024-04-16T19:24:45.413773+0000","last_unstale":"2024-04-16T21:12:36.017072+0000","last_undegraded":"2024-04-16T21:12:36.017072+0000","last_fullsized":"2024-04-16T21:12:36.017072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_deep_scrub":"52'4","last_deep_scrub_stamp":"2024-04-16T21:12:36.017012+0000","last_clean_scrub_stamp":"2024-04-16T21:12:36.017012+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:39:24.149502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1635,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"52'78","reported_seq":236,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:55.736528+0000","last_change":"2024-04-16T21:12:55.736594+0000","last_active":"2024-04-16T21:12:55.736528+0000","last_peered":"2024-04-16T21:12:55.736528+0000","last_clean":"2024-04-16T21:12:55.736528+0000","last_became_active":"2024-04-16T19:24:47.418752+0000","last_became_peered":"2024-04-16T19:24:47.418752+0000","last_unstale":"2024-04-16T21:12:55.736528+0000","last_undegraded":"2024-04-16T21:12:55.736528+0000","last_fullsized":"2024-04-16T21:12:55.736528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'78","last_scrub_stamp":"2024-04-16T21:12:55.736452+0000","last_deep_scrub":"52'78","last_deep_scrub_stamp":"2024-04-16T21:12:55.736452+0000","last_clean_scrub_stamp":"2024-04-16T21:12:55.736452+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:37.460668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":110,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:11.466444+0000","last_change":"2024-04-16T21:12:11.466444+0000","last_active":"2024-04-16T21:12:11.466444+0000","last_peered":"2024-04-16T21:12:11.466444+0000","last_clean":"2024-04-16T21:12:11.466444+0000","last_became_active":"2024-04-16T19:24:49.432099+0000","last_became_peered":"2024-04-16T19:24:49.432099+0000","last_unstale":"2024-04-16T21:12:11.466444+0000","last_undegraded":"2024-04-16T21:12:11.466444+0000","last_fullsized":"2024-04-16T21:12:11.466444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:11.466351+0000","last_clean_scrub_stamp":"2024-04-16T21:12:11.466351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:23:26.935725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:16.718042+0000","last_change":"2024-04-16T21:12:16.718042+0000","last_active":"2024-04-16T21:12:16.718042+0000","last_peered":"2024-04-16T21:12:16.718042+0000","last_clean":"2024-04-16T21:12:16.718042+0000","last_became_active":"2024-04-16T19:24:51.442294+0000","last_became_peered":"2024-04-16T19:24:51.442294+0000","last_unstale":"2024-04-16T21:12:16.718042+0000","last_undegraded":"2024-04-16T21:12:16.718042+0000","last_fullsized":"2024-04-16T21:12:16.718042+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:16.717926+0000","last_clean_scrub_stamp":"2024-04-16T21:12:16.717926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:06.936826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2a","version":"52'19","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:06.887055+0000","last_change":"2024-04-16T21:12:06.887055+0000","last_active":"2024-04-16T21:12:06.887055+0000","last_peered":"2024-04-16T21:12:06.887055+0000","last_clean":"2024-04-16T21:12:06.887055+0000","last_became_active":"2024-04-16T19:25:09.461662+0000","last_became_peered":"2024-04-16T19:25:09.461662+0000","last_unstale":"2024-04-16T21:12:06.887055+0000","last_undegraded":"2024-04-16T21:12:06.887055+0000","last_fullsized":"2024-04-16T21:12:06.887055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'19","last_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_deep_scrub":"52'19","last_deep_scrub_stamp":"2024-04-16T21:12:06.886889+0000","last_clean_scrub_stamp":"2024-04-16T21:12:06.886889+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:16:22.161394+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20671,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":203,"num_write":85,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.055453+0000","last_change":"2024-04-16T21:12:41.055453+0000","last_active":"2024-04-16T21:12:41.055453+0000","last_peered":"2024-04-16T21:12:41.055453+0000","last_clean":"2024-04-16T21:12:41.055453+0000","last_became_active":"2024-04-16T19:24:35.401714+0000","last_became_peered":"2024-04-16T19:24:35.401714+0000","last_unstale":"2024-04-16T21:12:41.055453+0000","last_undegraded":"2024-04-16T21:12:41.055453+0000","last_fullsized":"2024-04-16T21:12:41.055453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.055365+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.055365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T21:19:29.661658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:49.713970+0000","last_change":"2024-04-16T21:12:49.713970+0000","last_active":"2024-04-16T21:12:49.713970+0000","last_peered":"2024-04-16T21:12:49.713970+0000","last_clean":"2024-04-16T21:12:49.713970+0000","last_became_active":"2024-04-16T19:24:33.383230+0000","last_became_peered":"2024-04-16T19:24:33.383230+0000","last_unstale":"2024-04-16T21:12:49.713970+0000","last_undegraded":"2024-04-16T21:12:49.713970+0000","last_fullsized":"2024-04-16T21:12:49.713970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:49.713863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:49.713863+0000","last_clean_scrub_stamp":"2024-04-16T21:12:49.713863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:18:21.306923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"52'18","reported_seq":258,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.679355+0000","last_change":"2024-04-16T21:12:03.679355+0000","last_active":"2024-04-16T21:12:03.679355+0000","last_peered":"2024-04-16T21:12:03.679355+0000","last_clean":"2024-04-16T21:12:03.679355+0000","last_became_active":"2024-04-16T19:25:09.466177+0000","last_became_peered":"2024-04-16T19:25:09.466177+0000","last_unstale":"2024-04-16T21:12:03.679355+0000","last_undegraded":"2024-04-16T21:12:03.679355+0000","last_fullsized":"2024-04-16T21:12:03.679355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'18","last_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_deep_scrub":"52'18","last_deep_scrub_stamp":"2024-04-16T21:12:03.679242+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.679242+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:29:31.737391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":351,"num_read_kb":237,"num_write":91,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.087447+0000","last_change":"2024-04-16T21:12:50.087447+0000","last_active":"2024-04-16T21:12:50.087447+0000","last_peered":"2024-04-16T21:12:50.087447+0000","last_clean":"2024-04-16T21:12:50.087447+0000","last_became_active":"2024-04-16T19:24:35.391574+0000","last_became_peered":"2024-04-16T19:24:35.391574+0000","last_unstale":"2024-04-16T21:12:50.087447+0000","last_undegraded":"2024-04-16T21:12:50.087447+0000","last_fullsized":"2024-04-16T21:12:50.087447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.087360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.087360+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.087360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:28:16.769013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.747433+0000","last_change":"2024-04-16T21:12:44.747433+0000","last_active":"2024-04-16T21:12:44.747433+0000","last_peered":"2024-04-16T21:12:44.747433+0000","last_clean":"2024-04-16T21:12:44.747433+0000","last_became_active":"2024-04-16T19:24:33.384565+0000","last_became_peered":"2024-04-16T19:24:33.384565+0000","last_unstale":"2024-04-16T21:12:44.747433+0000","last_undegraded":"2024-04-16T21:12:44.747433+0000","last_fullsized":"2024-04-16T21:12:44.747433+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.747346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.747346+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.747346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:59.017695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.28","version":"52'10","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:02.673321+0000","last_change":"2024-04-16T21:12:02.673359+0000","last_active":"2024-04-16T21:12:02.673321+0000","last_peered":"2024-04-16T21:12:02.673321+0000","last_clean":"2024-04-16T21:12:02.673321+0000","last_became_active":"2024-04-16T19:25:09.466751+0000","last_became_peered":"2024-04-16T19:25:09.466751+0000","last_unstale":"2024-04-16T21:12:02.673321+0000","last_undegraded":"2024-04-16T21:12:02.673321+0000","last_fullsized":"2024-04-16T21:12:02.673321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:12:02.673266+0000","last_clean_scrub_stamp":"2024-04-16T21:12:02.673266+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:47:06.001051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":156,"num_write":57,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.894866+0000","last_change":"2024-04-16T21:12:51.894866+0000","last_active":"2024-04-16T21:12:51.894866+0000","last_peered":"2024-04-16T21:12:51.894866+0000","last_clean":"2024-04-16T21:12:51.894866+0000","last_became_active":"2024-04-16T19:24:35.396072+0000","last_became_peered":"2024-04-16T19:24:35.396072+0000","last_unstale":"2024-04-16T21:12:51.894866+0000","last_undegraded":"2024-04-16T21:12:51.894866+0000","last_fullsized":"2024-04-16T21:12:51.894866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.894777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.894777+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.894777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T04:17:01.243554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:59.228883+0000","last_change":"2024-04-16T21:12:59.228909+0000","last_active":"2024-04-16T21:12:59.228883+0000","last_peered":"2024-04-16T21:12:59.228883+0000","last_clean":"2024-04-16T21:12:59.228883+0000","last_became_active":"2024-04-16T19:24:33.386598+0000","last_became_peered":"2024-04-16T19:24:33.386598+0000","last_unstale":"2024-04-16T21:12:59.228883+0000","last_undegraded":"2024-04-16T21:12:59.228883+0000","last_fullsized":"2024-04-16T21:12:59.228883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:59.228813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:59.228813+0000","last_clean_scrub_stamp":"2024-04-16T21:12:59.228813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:17:23.181732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"52'13","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:57.447598+0000","last_change":"2024-04-16T21:11:57.447598+0000","last_active":"2024-04-16T21:11:57.447598+0000","last_peered":"2024-04-16T21:11:57.447598+0000","last_clean":"2024-04-16T21:11:57.447598+0000","last_became_active":"2024-04-16T19:25:09.458410+0000","last_became_peered":"2024-04-16T19:25:09.458410+0000","last_unstale":"2024-04-16T21:11:57.447598+0000","last_undegraded":"2024-04-16T21:11:57.447598+0000","last_fullsized":"2024-04-16T21:11:57.447598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'13","last_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_deep_scrub":"52'13","last_deep_scrub_stamp":"2024-04-16T21:11:57.447471+0000","last_clean_scrub_stamp":"2024-04-16T21:11:57.447471+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:25.758892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":66,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:37.002453+0000","last_change":"2024-04-16T21:12:37.002453+0000","last_active":"2024-04-16T21:12:37.002453+0000","last_peered":"2024-04-16T21:12:37.002453+0000","last_clean":"2024-04-16T21:12:37.002453+0000","last_became_active":"2024-04-16T19:24:35.400670+0000","last_became_peered":"2024-04-16T19:24:35.400670+0000","last_unstale":"2024-04-16T21:12:37.002453+0000","last_undegraded":"2024-04-16T21:12:37.002453+0000","last_fullsized":"2024-04-16T21:12:37.002453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:37.002335+0000","last_clean_scrub_stamp":"2024-04-16T21:12:37.002335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:36.977300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:06.426457+0000","last_change":"2024-04-16T21:13:06.426457+0000","last_active":"2024-04-16T21:13:06.426457+0000","last_peered":"2024-04-16T21:13:06.426457+0000","last_clean":"2024-04-16T21:13:06.426457+0000","last_became_active":"2024-04-16T19:24:33.388259+0000","last_became_peered":"2024-04-16T19:24:33.388259+0000","last_unstale":"2024-04-16T21:13:06.426457+0000","last_undegraded":"2024-04-16T21:13:06.426457+0000","last_fullsized":"2024-04-16T21:13:06.426457+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:06.426360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:06.426360+0000","last_clean_scrub_stamp":"2024-04-16T21:13:06.426360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:19.665763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2e","version":"52'20","reported_seq":268,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:04.649822+0000","last_change":"2024-04-16T21:12:04.649822+0000","last_active":"2024-04-16T21:12:04.649822+0000","last_peered":"2024-04-16T21:12:04.649822+0000","last_clean":"2024-04-16T21:12:04.649822+0000","last_became_active":"2024-04-16T19:25:09.466207+0000","last_became_peered":"2024-04-16T19:25:09.466207+0000","last_unstale":"2024-04-16T21:12:04.649822+0000","last_undegraded":"2024-04-16T21:12:04.649822+0000","last_fullsized":"2024-04-16T21:12:04.649822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:12:04.649711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:04.649711+0000","objects_scrubbed":9,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:26:32.961132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":370,"num_read_kb":236,"num_write":106,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:27.869867+0000","last_change":"2024-04-16T21:12:27.869867+0000","last_active":"2024-04-16T21:12:27.869867+0000","last_peered":"2024-04-16T21:12:27.869867+0000","last_clean":"2024-04-16T21:12:27.869867+0000","last_became_active":"2024-04-16T19:24:35.397493+0000","last_became_peered":"2024-04-16T19:24:35.397493+0000","last_unstale":"2024-04-16T21:12:27.869867+0000","last_undegraded":"2024-04-16T21:12:27.869867+0000","last_fullsized":"2024-04-16T21:12:27.869867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:27.869765+0000","last_clean_scrub_stamp":"2024-04-16T21:12:27.869765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:32:16.833314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:03.069536+0000","last_change":"2024-04-16T21:13:03.069560+0000","last_active":"2024-04-16T21:13:03.069536+0000","last_peered":"2024-04-16T21:13:03.069536+0000","last_clean":"2024-04-16T21:13:03.069536+0000","last_became_active":"2024-04-16T19:26:41.916035+0000","last_became_peered":"2024-04-16T19:26:41.916035+0000","last_unstale":"2024-04-16T21:13:03.069536+0000","last_undegraded":"2024-04-16T21:13:03.069536+0000","last_fullsized":"2024-04-16T21:13:03.069536+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:03.069470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:03.069470+0000","last_clean_scrub_stamp":"2024-04-16T21:13:03.069470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:52.947626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2f","version":"52'28","reported_seq":282,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:13.947946+0000","last_change":"2024-04-16T21:12:13.947946+0000","last_active":"2024-04-16T21:12:13.947946+0000","last_peered":"2024-04-16T21:12:13.947946+0000","last_clean":"2024-04-16T21:12:13.947946+0000","last_became_active":"2024-04-16T19:25:09.460055+0000","last_became_peered":"2024-04-16T19:25:09.460055+0000","last_unstale":"2024-04-16T21:12:13.947946+0000","last_undegraded":"2024-04-16T21:12:13.947946+0000","last_fullsized":"2024-04-16T21:12:13.947946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'28","last_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_deep_scrub":"52'28","last_deep_scrub_stamp":"2024-04-16T21:12:13.947818+0000","last_clean_scrub_stamp":"2024-04-16T21:12:13.947818+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:15:20.063878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":397,"num_read_kb":257,"num_write":149,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:35.699335+0000","last_change":"2024-04-16T21:12:35.699335+0000","last_active":"2024-04-16T21:12:35.699335+0000","last_peered":"2024-04-16T21:12:35.699335+0000","last_clean":"2024-04-16T21:12:35.699335+0000","last_became_active":"2024-04-16T19:24:35.401160+0000","last_became_peered":"2024-04-16T19:24:35.401160+0000","last_unstale":"2024-04-16T21:12:35.699335+0000","last_undegraded":"2024-04-16T21:12:35.699335+0000","last_fullsized":"2024-04-16T21:12:35.699335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:35.699249+0000","last_clean_scrub_stamp":"2024-04-16T21:12:35.699249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:06:40.252661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.105048+0000","last_change":"2024-04-16T21:13:00.105072+0000","last_active":"2024-04-16T21:13:00.105048+0000","last_peered":"2024-04-16T21:13:00.105048+0000","last_clean":"2024-04-16T21:13:00.105048+0000","last_became_active":"2024-04-16T19:24:33.386595+0000","last_became_peered":"2024-04-16T19:24:33.386595+0000","last_unstale":"2024-04-16T21:13:00.105048+0000","last_undegraded":"2024-04-16T21:13:00.105048+0000","last_fullsized":"2024-04-16T21:13:00.105048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.104978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.104978+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.104978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:13.108327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2c","version":"52'12","reported_seq":141,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.624974+0000","last_change":"2024-04-16T21:11:55.624974+0000","last_active":"2024-04-16T21:11:55.624974+0000","last_peered":"2024-04-16T21:11:55.624974+0000","last_clean":"2024-04-16T21:11:55.624974+0000","last_became_active":"2024-04-16T19:25:09.459632+0000","last_became_peered":"2024-04-16T19:25:09.459632+0000","last_unstale":"2024-04-16T21:11:55.624974+0000","last_undegraded":"2024-04-16T21:11:55.624974+0000","last_fullsized":"2024-04-16T21:11:55.624974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'12","last_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_deep_scrub":"52'12","last_deep_scrub_stamp":"2024-04-16T21:11:55.624828+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.624828+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:44:36.218593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":59,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:38.022020+0000","last_change":"2024-04-16T21:12:38.022020+0000","last_active":"2024-04-16T21:12:38.022020+0000","last_peered":"2024-04-16T21:12:38.022020+0000","last_clean":"2024-04-16T21:12:38.022020+0000","last_became_active":"2024-04-16T19:24:35.400262+0000","last_became_peered":"2024-04-16T19:24:35.400262+0000","last_unstale":"2024-04-16T21:12:38.022020+0000","last_undegraded":"2024-04-16T21:12:38.022020+0000","last_fullsized":"2024-04-16T21:12:38.022020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:38.021905+0000","last_clean_scrub_stamp":"2024-04-16T21:12:38.021905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T03:38:52.292254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:09.154337+0000","last_change":"2024-04-16T21:13:09.154356+0000","last_active":"2024-04-16T21:13:09.154337+0000","last_peered":"2024-04-16T21:13:09.154337+0000","last_clean":"2024-04-16T21:13:09.154337+0000","last_became_active":"2024-04-16T19:24:33.385199+0000","last_became_peered":"2024-04-16T19:24:33.385199+0000","last_unstale":"2024-04-16T21:13:09.154337+0000","last_undegraded":"2024-04-16T21:13:09.154337+0000","last_fullsized":"2024-04-16T21:13:09.154337+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:09.154272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:09.154272+0000","last_clean_scrub_stamp":"2024-04-16T21:13:09.154272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:22.393675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2d","version":"52'26","reported_seq":313,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:15.939510+0000","last_change":"2024-04-16T21:12:15.939510+0000","last_active":"2024-04-16T21:12:15.939510+0000","last_peered":"2024-04-16T21:12:15.939510+0000","last_clean":"2024-04-16T21:12:15.939510+0000","last_became_active":"2024-04-16T19:25:09.459127+0000","last_became_peered":"2024-04-16T19:25:09.459127+0000","last_unstale":"2024-04-16T21:12:15.939510+0000","last_undegraded":"2024-04-16T21:12:15.939510+0000","last_fullsized":"2024-04-16T21:12:15.939510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:12:15.939383+0000","last_clean_scrub_stamp":"2024-04-16T21:12:15.939383+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:49:32.014798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":473,"num_read_kb":293,"num_write":127,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:50.320242+0000","last_change":"2024-04-16T21:12:50.320242+0000","last_active":"2024-04-16T21:12:50.320242+0000","last_peered":"2024-04-16T21:12:50.320242+0000","last_clean":"2024-04-16T21:12:50.320242+0000","last_became_active":"2024-04-16T19:24:35.389809+0000","last_became_peered":"2024-04-16T19:24:35.389809+0000","last_unstale":"2024-04-16T21:12:50.320242+0000","last_undegraded":"2024-04-16T21:12:50.320242+0000","last_fullsized":"2024-04-16T21:12:50.320242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:50.320150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:50.320150+0000","last_clean_scrub_stamp":"2024-04-16T21:12:50.320150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T02:38:58.474109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:41.764753+0000","last_change":"2024-04-16T21:12:41.764753+0000","last_active":"2024-04-16T21:12:41.764753+0000","last_peered":"2024-04-16T21:12:41.764753+0000","last_clean":"2024-04-16T21:12:41.764753+0000","last_became_active":"2024-04-16T19:24:33.392325+0000","last_became_peered":"2024-04-16T19:24:33.392325+0000","last_unstale":"2024-04-16T21:12:41.764753+0000","last_undegraded":"2024-04-16T21:12:41.764753+0000","last_fullsized":"2024-04-16T21:12:41.764753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:41.764666+0000","last_clean_scrub_stamp":"2024-04-16T21:12:41.764666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:42:53.367336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.22","version":"52'46","reported_seq":327,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.408711+0000","last_change":"2024-04-16T21:11:46.408756+0000","last_active":"2024-04-16T21:11:46.408711+0000","last_peered":"2024-04-16T21:11:46.408711+0000","last_clean":"2024-04-16T21:11:46.408711+0000","last_became_active":"2024-04-16T19:25:09.462455+0000","last_became_peered":"2024-04-16T19:25:09.462455+0000","last_unstale":"2024-04-16T21:11:46.408711+0000","last_undegraded":"2024-04-16T21:11:46.408711+0000","last_fullsized":"2024-04-16T21:11:46.408711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'46","last_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_deep_scrub":"52'46","last_deep_scrub_stamp":"2024-04-16T21:11:46.408648+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.408648+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T00:50:38.312038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":475,"num_read_kb":306,"num_write":128,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.425196+0000","last_change":"2024-04-16T21:12:51.425238+0000","last_active":"2024-04-16T21:12:51.425196+0000","last_peered":"2024-04-16T21:12:51.425196+0000","last_clean":"2024-04-16T21:12:51.425196+0000","last_became_active":"2024-04-16T19:24:35.392137+0000","last_became_peered":"2024-04-16T19:24:35.392137+0000","last_unstale":"2024-04-16T21:12:51.425196+0000","last_undegraded":"2024-04-16T21:12:51.425196+0000","last_fullsized":"2024-04-16T21:12:51.425196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.425114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.425114+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.425114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:04:07.760170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:06.771504+0000","last_change":"2024-04-16T21:13:06.771504+0000","last_active":"2024-04-16T21:13:06.771504+0000","last_peered":"2024-04-16T21:13:06.771504+0000","last_clean":"2024-04-16T21:13:06.771504+0000","last_became_active":"2024-04-16T19:24:33.384032+0000","last_became_peered":"2024-04-16T19:24:33.384032+0000","last_unstale":"2024-04-16T21:13:06.771504+0000","last_undegraded":"2024-04-16T21:13:06.771504+0000","last_fullsized":"2024-04-16T21:13:06.771504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:06.771388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:06.771388+0000","last_clean_scrub_stamp":"2024-04-16T21:13:06.771388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:39.780093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"52'20","reported_seq":205,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.919349+0000","last_change":"2024-04-16T21:11:47.919430+0000","last_active":"2024-04-16T21:11:47.919349+0000","last_peered":"2024-04-16T21:11:47.919349+0000","last_clean":"2024-04-16T21:11:47.919349+0000","last_became_active":"2024-04-16T19:25:09.466916+0000","last_became_peered":"2024-04-16T19:25:09.466916+0000","last_unstale":"2024-04-16T21:11:47.919349+0000","last_undegraded":"2024-04-16T21:11:47.919349+0000","last_fullsized":"2024-04-16T21:11:47.919349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'20","last_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_deep_scrub":"52'20","last_deep_scrub_stamp":"2024-04-16T21:11:47.919276+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.919276+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:11:50.792760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":132,"num_write":107,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:53.763777+0000","last_change":"2024-04-16T21:12:53.763799+0000","last_active":"2024-04-16T21:12:53.763777+0000","last_peered":"2024-04-16T21:12:53.763777+0000","last_clean":"2024-04-16T21:12:53.763777+0000","last_became_active":"2024-04-16T19:24:35.396240+0000","last_became_peered":"2024-04-16T19:24:35.396240+0000","last_unstale":"2024-04-16T21:12:53.763777+0000","last_undegraded":"2024-04-16T21:12:53.763777+0000","last_fullsized":"2024-04-16T21:12:53.763777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:53.763711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:53.763711+0000","last_clean_scrub_stamp":"2024-04-16T21:12:53.763711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:59:29.416236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:05.470297+0000","last_change":"2024-04-16T21:13:05.470341+0000","last_active":"2024-04-16T21:13:05.470297+0000","last_peered":"2024-04-16T21:13:05.470297+0000","last_clean":"2024-04-16T21:13:05.470297+0000","last_became_active":"2024-04-16T19:24:33.383293+0000","last_became_peered":"2024-04-16T19:24:33.383293+0000","last_unstale":"2024-04-16T21:13:05.470297+0000","last_undegraded":"2024-04-16T21:13:05.470297+0000","last_fullsized":"2024-04-16T21:13:05.470297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:05.470223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:05.470223+0000","last_clean_scrub_stamp":"2024-04-16T21:13:05.470223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:06:23.439234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.20","version":"52'17","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:47.209595+0000","last_change":"2024-04-16T21:11:47.209653+0000","last_active":"2024-04-16T21:11:47.209595+0000","last_peered":"2024-04-16T21:11:47.209595+0000","last_clean":"2024-04-16T21:11:47.209595+0000","last_became_active":"2024-04-16T19:25:09.463758+0000","last_became_peered":"2024-04-16T19:25:09.463758+0000","last_unstale":"2024-04-16T21:11:47.209595+0000","last_undegraded":"2024-04-16T21:11:47.209595+0000","last_fullsized":"2024-04-16T21:11:47.209595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'17","last_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_deep_scrub":"52'17","last_deep_scrub_stamp":"2024-04-16T21:11:47.209532+0000","last_clean_scrub_stamp":"2024-04-16T21:11:47.209532+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:26:30.862167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":162,"num_read_kb":229,"num_write":96,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:31.844046+0000","last_change":"2024-04-16T21:12:31.844046+0000","last_active":"2024-04-16T21:12:31.844046+0000","last_peered":"2024-04-16T21:12:31.844046+0000","last_clean":"2024-04-16T21:12:31.844046+0000","last_became_active":"2024-04-16T19:24:35.395066+0000","last_became_peered":"2024-04-16T19:24:35.395066+0000","last_unstale":"2024-04-16T21:12:31.844046+0000","last_undegraded":"2024-04-16T21:12:31.844046+0000","last_fullsized":"2024-04-16T21:12:31.844046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:31.843954+0000","last_clean_scrub_stamp":"2024-04-16T21:12:31.843954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:18:05.611280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:02.827107+0000","last_change":"2024-04-16T21:13:02.827128+0000","last_active":"2024-04-16T21:13:02.827107+0000","last_peered":"2024-04-16T21:13:02.827107+0000","last_clean":"2024-04-16T21:13:02.827107+0000","last_became_active":"2024-04-16T19:24:33.385846+0000","last_became_peered":"2024-04-16T19:24:33.385846+0000","last_unstale":"2024-04-16T21:13:02.827107+0000","last_undegraded":"2024-04-16T21:13:02.827107+0000","last_fullsized":"2024-04-16T21:13:02.827107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:02.827047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:02.827047+0000","last_clean_scrub_stamp":"2024-04-16T21:13:02.827047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:22:43.172076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.21","version":"52'27","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:49.894735+0000","last_change":"2024-04-16T21:11:49.894811+0000","last_active":"2024-04-16T21:11:49.894735+0000","last_peered":"2024-04-16T21:11:49.894735+0000","last_clean":"2024-04-16T21:11:49.894735+0000","last_became_active":"2024-04-16T19:25:09.464977+0000","last_became_peered":"2024-04-16T19:25:09.464977+0000","last_unstale":"2024-04-16T21:11:49.894735+0000","last_undegraded":"2024-04-16T21:11:49.894735+0000","last_fullsized":"2024-04-16T21:11:49.894735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'27","last_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_deep_scrub":"52'27","last_deep_scrub_stamp":"2024-04-16T21:11:49.894658+0000","last_clean_scrub_stamp":"2024-04-16T21:11:49.894658+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:10:11.290532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":68,"num_write":108,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:00.206178+0000","last_change":"2024-04-16T21:13:00.206200+0000","last_active":"2024-04-16T21:13:00.206178+0000","last_peered":"2024-04-16T21:13:00.206178+0000","last_clean":"2024-04-16T21:13:00.206178+0000","last_became_active":"2024-04-16T19:24:35.394647+0000","last_became_peered":"2024-04-16T19:24:35.394647+0000","last_unstale":"2024-04-16T21:13:00.206178+0000","last_undegraded":"2024-04-16T21:13:00.206178+0000","last_fullsized":"2024-04-16T21:13:00.206178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:00.206117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:00.206117+0000","last_clean_scrub_stamp":"2024-04-16T21:13:00.206117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:25:12.660370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":88,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:05.863200+0000","last_change":"2024-04-16T21:12:05.863236+0000","last_active":"2024-04-16T21:12:05.863200+0000","last_peered":"2024-04-16T21:12:05.863200+0000","last_clean":"2024-04-16T21:12:05.863200+0000","last_became_active":"2024-04-16T19:25:41.505646+0000","last_became_peered":"2024-04-16T19:25:41.505646+0000","last_unstale":"2024-04-16T21:12:05.863200+0000","last_undegraded":"2024-04-16T21:12:05.863200+0000","last_fullsized":"2024-04-16T21:12:05.863200+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:05.863135+0000","last_clean_scrub_stamp":"2024-04-16T21:12:05.863135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T22:09:16.509928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.26","version":"52'10","reported_seq":157,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:58.658621+0000","last_change":"2024-04-16T21:11:58.658671+0000","last_active":"2024-04-16T21:11:58.658621+0000","last_peered":"2024-04-16T21:11:58.658621+0000","last_clean":"2024-04-16T21:11:58.658621+0000","last_became_active":"2024-04-16T19:25:09.467266+0000","last_became_peered":"2024-04-16T19:25:09.467266+0000","last_unstale":"2024-04-16T21:11:58.658621+0000","last_undegraded":"2024-04-16T21:11:58.658621+0000","last_fullsized":"2024-04-16T21:11:58.658621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'10","last_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_deep_scrub":"52'10","last_deep_scrub_stamp":"2024-04-16T21:11:58.658555+0000","last_clean_scrub_stamp":"2024-04-16T21:11:58.658555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:27:31.251581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":69,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:42.764316+0000","last_change":"2024-04-16T21:12:42.764340+0000","last_active":"2024-04-16T21:12:42.764316+0000","last_peered":"2024-04-16T21:12:42.764316+0000","last_clean":"2024-04-16T21:12:42.764316+0000","last_became_active":"2024-04-16T19:24:35.398209+0000","last_became_peered":"2024-04-16T19:24:35.398209+0000","last_unstale":"2024-04-16T21:12:42.764316+0000","last_undegraded":"2024-04-16T21:12:42.764316+0000","last_fullsized":"2024-04-16T21:12:42.764316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:42.764250+0000","last_clean_scrub_stamp":"2024-04-16T21:12:42.764250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:45:42.739215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:07.213767+0000","last_change":"2024-04-16T21:13:07.213808+0000","last_active":"2024-04-16T21:13:07.213767+0000","last_peered":"2024-04-16T21:13:07.213767+0000","last_clean":"2024-04-16T21:13:07.213767+0000","last_became_active":"2024-04-16T19:24:33.384615+0000","last_became_peered":"2024-04-16T19:24:33.384615+0000","last_unstale":"2024-04-16T21:13:07.213767+0000","last_undegraded":"2024-04-16T21:13:07.213767+0000","last_fullsized":"2024-04-16T21:13:07.213767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:07.213690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:07.213690+0000","last_clean_scrub_stamp":"2024-04-16T21:13:07.213690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:02:03.929917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.27","version":"52'22","reported_seq":215,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:53.811095+0000","last_change":"2024-04-16T21:11:53.811157+0000","last_active":"2024-04-16T21:11:53.811095+0000","last_peered":"2024-04-16T21:11:53.811095+0000","last_clean":"2024-04-16T21:11:53.811095+0000","last_became_active":"2024-04-16T19:25:09.463871+0000","last_became_peered":"2024-04-16T19:25:09.463871+0000","last_unstale":"2024-04-16T21:11:53.811095+0000","last_undegraded":"2024-04-16T21:11:53.811095+0000","last_fullsized":"2024-04-16T21:11:53.811095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:53.811028+0000","last_clean_scrub_stamp":"2024-04-16T21:11:53.811028+0000","objects_scrubbed":8,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:31.711738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":163,"num_write":121,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:12.431513+0000","last_change":"2024-04-16T21:12:12.431513+0000","last_active":"2024-04-16T21:12:12.431513+0000","last_peered":"2024-04-16T21:12:12.431513+0000","last_clean":"2024-04-16T21:12:12.431513+0000","last_became_active":"2024-04-16T19:25:41.508061+0000","last_became_peered":"2024-04-16T19:25:41.508061+0000","last_unstale":"2024-04-16T21:12:12.431513+0000","last_undegraded":"2024-04-16T21:12:12.431513+0000","last_fullsized":"2024-04-16T21:12:12.431513+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:12.431389+0000","last_clean_scrub_stamp":"2024-04-16T21:12:12.431389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:06.860124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:51.343823+0000","last_change":"2024-04-16T21:12:51.343823+0000","last_active":"2024-04-16T21:12:51.343823+0000","last_peered":"2024-04-16T21:12:51.343823+0000","last_clean":"2024-04-16T21:12:51.343823+0000","last_became_active":"2024-04-16T19:24:33.388073+0000","last_became_peered":"2024-04-16T19:24:33.388073+0000","last_unstale":"2024-04-16T21:12:51.343823+0000","last_undegraded":"2024-04-16T21:12:51.343823+0000","last_fullsized":"2024-04-16T21:12:51.343823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:51.343738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:51.343738+0000","last_clean_scrub_stamp":"2024-04-16T21:12:51.343738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:41.221893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.24","version":"52'6","reported_seq":152,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:03.660156+0000","last_change":"2024-04-16T21:12:03.660192+0000","last_active":"2024-04-16T21:12:03.660156+0000","last_peered":"2024-04-16T21:12:03.660156+0000","last_clean":"2024-04-16T21:12:03.660156+0000","last_became_active":"2024-04-16T19:25:41.499350+0000","last_became_peered":"2024-04-16T19:25:41.499350+0000","last_unstale":"2024-04-16T21:12:03.660156+0000","last_undegraded":"2024-04-16T21:12:03.660156+0000","last_fullsized":"2024-04-16T21:12:03.660156+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:03.660092+0000","last_clean_scrub_stamp":"2024-04-16T21:12:03.660092+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:56:42.368715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":101,"num_write":35,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":134,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:44.153401+0000","last_change":"2024-04-16T21:12:44.153443+0000","last_active":"2024-04-16T21:12:44.153401+0000","last_peered":"2024-04-16T21:12:44.153401+0000","last_clean":"2024-04-16T21:12:44.153401+0000","last_became_active":"2024-04-16T19:24:35.395567+0000","last_became_peered":"2024-04-16T19:24:35.395567+0000","last_unstale":"2024-04-16T21:12:44.153401+0000","last_undegraded":"2024-04-16T21:12:44.153401+0000","last_fullsized":"2024-04-16T21:12:44.153401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:44.153324+0000","last_clean_scrub_stamp":"2024-04-16T21:12:44.153324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:18:59.952621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.744632+0000","last_change":"2024-04-16T21:12:48.744655+0000","last_active":"2024-04-16T21:12:48.744632+0000","last_peered":"2024-04-16T21:12:48.744632+0000","last_clean":"2024-04-16T21:12:48.744632+0000","last_became_active":"2024-04-16T19:24:33.391338+0000","last_became_peered":"2024-04-16T19:24:33.391338+0000","last_unstale":"2024-04-16T21:12:48.744632+0000","last_undegraded":"2024-04-16T21:12:48.744632+0000","last_fullsized":"2024-04-16T21:12:48.744632+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.744563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.744563+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.744563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:50:55.192951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.25","version":"52'26","reported_seq":302,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:46.166590+0000","last_change":"2024-04-16T21:11:46.166636+0000","last_active":"2024-04-16T21:11:46.166590+0000","last_peered":"2024-04-16T21:11:46.166590+0000","last_clean":"2024-04-16T21:11:46.166590+0000","last_became_active":"2024-04-16T19:25:09.468171+0000","last_became_peered":"2024-04-16T19:25:09.468171+0000","last_unstale":"2024-04-16T21:11:46.166590+0000","last_undegraded":"2024-04-16T21:11:46.166590+0000","last_fullsized":"2024-04-16T21:11:46.166590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'26","last_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_deep_scrub":"52'26","last_deep_scrub_stamp":"2024-04-16T21:11:46.166525+0000","last_clean_scrub_stamp":"2024-04-16T21:11:46.166525+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:02:05.994388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":449,"num_read_kb":311,"num_write":145,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:52.322514+0000","last_change":"2024-04-16T21:12:52.322514+0000","last_active":"2024-04-16T21:12:52.322514+0000","last_peered":"2024-04-16T21:12:52.322514+0000","last_clean":"2024-04-16T21:12:52.322514+0000","last_became_active":"2024-04-16T19:24:35.390496+0000","last_became_peered":"2024-04-16T19:24:35.390496+0000","last_unstale":"2024-04-16T21:12:52.322514+0000","last_undegraded":"2024-04-16T21:12:52.322514+0000","last_fullsized":"2024-04-16T21:12:52.322514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:52.322391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:52.322391+0000","last_clean_scrub_stamp":"2024-04-16T21:12:52.322391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T05:14:54.915074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:56.830870+0000","last_change":"2024-04-16T21:12:56.830905+0000","last_active":"2024-04-16T21:12:56.830870+0000","last_peered":"2024-04-16T21:12:56.830870+0000","last_clean":"2024-04-16T21:12:56.830870+0000","last_became_active":"2024-04-16T19:24:33.388379+0000","last_became_peered":"2024-04-16T19:24:33.388379+0000","last_unstale":"2024-04-16T21:12:56.830870+0000","last_undegraded":"2024-04-16T21:12:56.830870+0000","last_fullsized":"2024-04-16T21:12:56.830870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:56.830806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:56.830806+0000","last_clean_scrub_stamp":"2024-04-16T21:12:56.830806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:25:52.367671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3a","version":"52'22","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:11:55.232333+0000","last_change":"2024-04-16T21:11:55.232333+0000","last_active":"2024-04-16T21:11:55.232333+0000","last_peered":"2024-04-16T21:11:55.232333+0000","last_clean":"2024-04-16T21:11:55.232333+0000","last_became_active":"2024-04-16T19:25:09.458721+0000","last_became_peered":"2024-04-16T19:25:09.458721+0000","last_unstale":"2024-04-16T21:11:55.232333+0000","last_undegraded":"2024-04-16T21:11:55.232333+0000","last_fullsized":"2024-04-16T21:11:55.232333+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'22","last_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_deep_scrub":"52'22","last_deep_scrub_stamp":"2024-04-16T21:11:55.232184+0000","last_clean_scrub_stamp":"2024-04-16T21:11:55.232184+0000","objects_scrubbed":10,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:12:19.898271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":239,"num_read_kb":199,"num_write":125,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.214826+0000","last_change":"2024-04-16T21:13:01.214865+0000","last_active":"2024-04-16T21:13:01.214826+0000","last_peered":"2024-04-16T21:13:01.214826+0000","last_clean":"2024-04-16T21:13:01.214826+0000","last_became_active":"2024-04-16T19:24:33.386490+0000","last_became_peered":"2024-04-16T19:24:33.386490+0000","last_unstale":"2024-04-16T21:13:01.214826+0000","last_undegraded":"2024-04-16T21:13:01.214826+0000","last_fullsized":"2024-04-16T21:13:01.214826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.214754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.214754+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.214754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:16:34.223459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":130,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:48.395601+0000","last_change":"2024-04-16T21:12:48.395624+0000","last_active":"2024-04-16T21:12:48.395601+0000","last_peered":"2024-04-16T21:12:48.395601+0000","last_clean":"2024-04-16T21:12:48.395601+0000","last_became_active":"2024-04-16T19:24:35.396426+0000","last_became_peered":"2024-04-16T19:24:35.396426+0000","last_unstale":"2024-04-16T21:12:48.395601+0000","last_undegraded":"2024-04-16T21:12:48.395601+0000","last_fullsized":"2024-04-16T21:12:48.395601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:48.395534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:48.395534+0000","last_clean_scrub_stamp":"2024-04-16T21:12:48.395534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T01:42:38.273689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3b","version":"52'6","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:14.936034+0000","last_change":"2024-04-16T21:12:14.936034+0000","last_active":"2024-04-16T21:12:14.936034+0000","last_peered":"2024-04-16T21:12:14.936034+0000","last_clean":"2024-04-16T21:12:14.936034+0000","last_became_active":"2024-04-16T19:25:09.459713+0000","last_became_peered":"2024-04-16T19:25:09.459713+0000","last_unstale":"2024-04-16T21:12:14.936034+0000","last_undegraded":"2024-04-16T21:12:14.936034+0000","last_fullsized":"2024-04-16T21:12:14.936034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'6","last_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_deep_scrub":"52'6","last_deep_scrub_stamp":"2024-04-16T21:12:14.935939+0000","last_clean_scrub_stamp":"2024-04-16T21:12:14.935939+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:54:02.653615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":114,"num_write":29,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":136,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:12:58.048488+0000","last_change":"2024-04-16T21:12:58.048511+0000","last_active":"2024-04-16T21:12:58.048488+0000","last_peered":"2024-04-16T21:12:58.048488+0000","last_clean":"2024-04-16T21:12:58.048488+0000","last_became_active":"2024-04-16T19:24:33.384924+0000","last_became_peered":"2024-04-16T19:24:33.384924+0000","last_unstale":"2024-04-16T21:12:58.048488+0000","last_undegraded":"2024-04-16T21:12:58.048488+0000","last_fullsized":"2024-04-16T21:12:58.048488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:12:58.048410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:12:58.048410+0000","last_clean_scrub_stamp":"2024-04-16T21:12:58.048410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T06:52:35.340118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":131,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-16T21:13:01.804262+0000","last_change":"2024-04-16T21:13:01.804262+0000","last_active":"2024-04-16T21:13:01.804262+0000","last_peered":"2024-04-16T21:13:01.804262+0000","last_clean":"2024-04-16T21:13:01.804262+0000","last_became_active":"2024-04-16T19:24:35.400884+0000","last_became_peered":"2024-04-16T19:24:35.400884+0000","last_unstale":"2024-04-16T21:13:01.804262+0000","last_undegraded":"2024-04-16T21:13:01.804262+0000","last_fullsized":"2024-04-16T21:13:01.804262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T21:13:01.804169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T21:13:01.804169+0000","last_clean_scrub_stamp":"2024-04-16T21:13:01.804169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T23:06:57.074561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":6},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":87473,"num_read_kb":86923,"num_write":6175,"num_write_kb":2636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":8,"num_objects_omap":1419,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":359044,"num_omap_keys":1456,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5622,"ondisk_log_size":5622,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":6},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":23143,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":4158,"num_read_kb":5103,"num_write":200,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":46286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":64,"acting":64,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":5281,"num_read_kb":5040,"num_write":3409,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":294,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3599,"ondisk_log_size":3599,"up":64,"acting":64,"num_store_stats":6},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":64,"acting":64,"num_store_stats":6},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":87805,"num_objects":448,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":17945,"num_read_kb":12650,"num_write":6112,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3670016,"data_stored":175610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1190,"ondisk_log_size":1190,"up":128,"acting":128,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":190672,"num_objects":699,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":280368,"num_read_kb":196750,"num_write":19424,"num_write_kb":4437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":207887,"num_omap_keys":966,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16207,"ondisk_log_size":16207,"up":64,"acting":64,"num_store_stats":6},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5669,"ondisk_log_size":5669,"up":64,"acting":64,"num_store_stats":6},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":35166,"num_objects":134,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":3060,"num_read_kb":2801,"num_write":1296,"num_write_kb":446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9989,"num_omap_keys":63,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1024000,"data_stored":70332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":64,"acting":64,"num_store_stats":6},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244641575,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28652,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104828948,"statfs":{"total":107374182400,"available":107344842752,"internally_reserved":0,"allocated":1675264,"data_stored":436509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244641576,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59084,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104798516,"statfs":{"total":107374182400,"available":107313680384,"internally_reserved":0,"allocated":1576960,"data_stored":457110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244641575,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":59728,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":104797872,"statfs":{"total":107374182400,"available":107313020928,"internally_reserved":0,"allocated":2236416,"data_stored":1042918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244641576,"num_pgs":170,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":46748,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810852,"statfs":{"total":107374182400,"available":107326312448,"internally_reserved":0,"allocated":1724416,"data_stored":436917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244641576,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":50916,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104806684,"statfs":{"total":107374182400,"available":107322044416,"internally_reserved":0,"allocated":1732608,"data_stored":390396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244641575,"num_pgs":180,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":47480,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":104810120,"statfs":{"total":107374182400,"available":107325562880,"internally_reserved":0,"allocated":2473984,"data_stored":1127862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":13136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":6437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":5255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":12632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":6764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":42635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":22290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":42862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":42338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":22965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":135794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":37874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":97106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":49595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":54666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":17235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":11753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":9316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":8011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T21:13:27.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-16T21:13:27.570 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' 2024-04-16T21:13:27.586 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T21:13:27.587 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T21:13:27.587 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-16T21:13:27.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.619 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-16T21:13:27.619 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-16T21:13:27.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.691 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-16T21:13:27.691 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-16T21:13:27.691 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.762 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-16T21:13:27.763 DEBUG:tasks.ceph.c1.osd.3:waiting for process to exit 2024-04-16T21:13:27.763 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.801 INFO:tasks.ceph.c1.osd.3:Stopped 2024-04-16T21:13:27.801 DEBUG:tasks.ceph.c1.osd.4:waiting for process to exit 2024-04-16T21:13:27.801 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.886 INFO:tasks.ceph.c1.osd.4:Stopped 2024-04-16T21:13:27.886 DEBUG:tasks.ceph.c1.osd.5:waiting for process to exit 2024-04-16T21:13:27.886 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.928 INFO:tasks.ceph.c1.osd.5:Stopped 2024-04-16T21:13:27.928 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T21:13:27.928 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-16T21:13:27.928 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.958 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-16T21:13:27.958 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T21:13:27.958 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-16T21:13:27.958 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T21:13:27.978 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-16T21:13:27.979 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T21:13:27.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-16T21:13:28.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T21:13:28.054 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-16T21:13:28.302 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T21:13:28.303 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-16T21:13:28.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T21:13:28.388 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-16T21:13:28.478 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-3 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T21:13:28.479 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c1-3 2024-04-16T21:13:28.637 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-4 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T21:13:28.637 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c1-4 2024-04-16T21:13:28.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-5 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T21:13:28.752 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/c1-5 2024-04-16T21:13:28.831 INFO:tasks.ceph:Archiving mon data... 2024-04-16T21:13:28.831 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/data/c1.mon.a.tgz 2024-04-16T21:13:28.831 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-04-16T21:13:28.847 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.mGdjJamUiX 2024-04-16T21:13:28.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.mGdjJamUiX 2024-04-16T21:13:29.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.mGdjJamUiX 2024-04-16T21:13:29.091 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.mGdjJamUiX is 359KB 2024-04-16T21:13:29.188 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.mGdjJamUiX 2024-04-16T21:13:29.203 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T21:13:29.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:13:29.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:13:29.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:13:29.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T21:13:29.326 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T21:13:29.344 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T21:13:29.363 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T21:13:29.381 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T21:13:29.382 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T21:13:29.384 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi026/crash 2024-04-16T21:13:29.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T21:13:29.412 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi069/crash 2024-04-16T21:13:29.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T21:13:29.463 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi119/crash 2024-04-16T21:13:29.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T21:13:29.530 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi151/crash 2024-04-16T21:13:29.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T21:13:29.595 INFO:tasks.ceph:Compressing logs... 2024-04-16T21:13:29.595 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T21:13:29.598 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T21:13:29.600 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T21:13:29.603 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T21:13:29.659 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35431.log 2024-04-16T21:13:29.659 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35471.log 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.31883.log 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35431.log.gz 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.35471.log: gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.1.log 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35471.log.gz 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.31883.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35511.log 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.31883.log.gz 2024-04-16T21:13:29.660 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/rgw.c2.client.1.log: gzip -5 --verbose -- /var/log/ceph/ops-log-c2-client.1.log 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.35511.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c2-client.admin.35511.log.gz 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/c2-client.admin.35391.log 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ops-log-c2-client.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32083.log 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr: 35.9% -- replaced with /var/log/ceph/ops-log-c2-client.1.log.gz 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.35391.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.35391.log.gz 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr: -5 --verbose -- /var/log/ceph/c2-osd.3.log 2024-04-16T21:13:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.32083.log: gzip -5 --verbose -- /var/log/ceph/c2-mgr.x.log 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.32083.log.gz 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.35534.log 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32003.log 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.1.35534.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.5.log 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr: 82.5% -- replaced with /var/log/ceph/c2-client.1.35534.log.gz 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.31963.log 2024-04-16T21:13:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.32003.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.32003.log.gz 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.31923.log 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/rgw.c2.client.1.log.gz 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.31963.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32043.log 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.31963.log.gz 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/c2-osd.4.log 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.31923.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.31923.log.gz 2024-04-16T21:13:29.663 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-client.admin.32043.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.32043.log.gz -5 2024-04-16T21:13:29.664 INFO:teuthology.orchestra.run.smithi151.stderr: --verbose -- /var/log/ceph/c2-client.1.36139.log 2024-04-16T21:13:29.664 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/c2-osd.4.log: /var/log/ceph/c2-client.1.36139.log: 43.9% -- replaced with /var/log/ceph/c2-client.1.36139.log.gz 2024-04-16T21:13:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32099.log 2024-04-16T21:13:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/c1-osd.4.log 2024-04-16T21:13:29.668 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.1.37520.log 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.32099.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.32099.log.gz 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c1-mgr.x.log 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.1.37520.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32059.log 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr: 93.9% -- replaced with /var/log/ceph/c1-client.1.37520.log.gz 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35304.log 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.32059.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32139.log 2024-04-16T21:13:29.669 INFO:teuthology.orchestra.run.smithi069.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.32059.log.gz 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.35304.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.admin.35304.log.gz 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ops-log-c1-client.1.log 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.32139.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35424.log 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.32139.log.gz 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ops-log-c1-client.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.36052.log 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr: 35.7% -- replaced with /var/log/ceph/ops-log-c1-client.1.log.gz 2024-04-16T21:13:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.35424.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35424.log.gz 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32019.log 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.1.36052.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.35447.log 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr: 43.9% -- replaced with /var/log/ceph/c1-client.1.36052.log.gz 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.32019.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.admin.32019.log.gz --verbose 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/c1-osd.5.log 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.1.35447.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.1.log 2024-04-16T21:13:29.671 INFO:teuthology.orchestra.run.smithi069.stderr: 82.5% -- replaced with /var/log/ceph/c1-client.1.35447.log.gz 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.3.log 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/rgw.c1.client.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35384.log 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.31939.log 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.31979.log 0.0% -- replaced with /var/log/ceph/c1-client.admin.35384.log.gz 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.31939.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.31939.log.gz 2024-04-16T21:13:29.672 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35344.log 2024-04-16T21:13:29.673 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.31979.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.31979.log.gz 2024-04-16T21:13:29.673 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/c1-client.admin.35344.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35344.log.gz 2024-04-16T21:13:29.673 INFO:teuthology.orchestra.run.smithi069.stderr: 94.1% -- replaced with /var/log/ceph/rgw.c1.client.1.log.gz 2024-04-16T21:13:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.40413.log 2024-04-16T21:13:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.89194.log 2024-04-16T21:13:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.172985.log 2024-04-16T21:13:29.910 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.40413.log.gz 2024-04-16T21:13:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.89194.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57633.log 2024-04-16T21:13:29.910 INFO:teuthology.orchestra.run.smithi026.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.89194.log.gz 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.172985.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92835.log 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.172985.log.gz 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.57633.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44240.log 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57633.log.gz 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.92835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36169.log 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92835.log.gz 2024-04-16T21:13:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.44240.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154713.log 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44240.log.gz 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36169.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54598.log 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36169.log.gz 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.154713.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.31965.log 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr: 69.8% -- replaced with /var/log/ceph/c1-client.0.154713.log.gz 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.54598.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.161393.log 2024-04-16T21:13:29.912 INFO:teuthology.orchestra.run.smithi026.stderr: 53.3% -- replaced with /var/log/ceph/c1-client.0.54598.log.gz 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.31965.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103752.log 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.31965.log.gz 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.161393.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52774.log 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.161393.log.gz 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.103752.log: gzip -5 --verbose -- /var/log/ceph/c1.audit.log 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103752.log.gz 2024-04-16T21:13:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.52774.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35265.log 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr: 50.1% -- replaced with /var/log/ceph/c1-client.0.52774.log.gz 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1.audit.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109824.log 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35265.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.181605.log 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35265.log.gz 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.109824.log: 89.3% -- replaced with /var/log/ceph/c1.audit.log.gz 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.130427.log 2024-04-16T21:13:29.914 INFO:teuthology.orchestra.run.smithi026.stderr: 58.4% -- replaced with /var/log/ceph/c1-client.0.109824.log.gz 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.181605.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114667.log 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.130427.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.135271.log 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.130427.log.gz 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr: 93.9% -- replaced with /var/log/ceph/c1-client.0.181605.log.gz 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.114667.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49694.log 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114667.log.gz 2024-04-16T21:13:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.135271.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94650.log 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.135271.log.gz 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.49694.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108002.log 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49694.log.gz 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.94650.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143180.log 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94650.log.gz 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.108002.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182965.log 2024-04-16T21:13:29.916 INFO:teuthology.orchestra.run.smithi026.stderr: 63.9% -- replaced with /var/log/ceph/c1-client.0.108002.log.gz 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.143180.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48482.log 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr: 78.7% -- replaced with /var/log/ceph/c1-client.0.143180.log.gz 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182965.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56418.log 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182965.log.gz 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.48482.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47877.log 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48482.log.gz 2024-04-16T21:13:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.56418.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.176616.log 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr: 50.4% -- replaced with /var/log/ceph/c1-client.0.56418.log.gz 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.47877.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117092.log 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47877.log.gz 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.176616.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.162002.log 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.176616.log.gz 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.117092.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.163819.log 2024-04-16T21:13:29.918 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117092.log.gz 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.162002.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.183134.log 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr: 54.1% -- replaced with /var/log/ceph/c1-client.0.162002.log.gz 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.163819.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35149.log 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.163819.log.gz 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.183134.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.167455.log 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.183134.log.gz 2024-04-16T21:13:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35149.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51515.log 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35149.log.gz 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.167455.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.159565.log 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.167455.log.gz 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.51515.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.136487.log 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51515.log.gz 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.159565.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182503.log 2024-04-16T21:13:29.920 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.159565.log.gz 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.136487.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.134057.log 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.136487.log.gz 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182503.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.160780.log 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182503.log.gz 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.134057.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.183049.log 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.134057.log.gz 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.160780.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45453.log 2024-04-16T21:13:29.921 INFO:teuthology.orchestra.run.smithi026.stderr: 67.5% -- replaced with /var/log/ceph/c1-client.0.160780.log.gz 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.183049.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76249.log 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.183049.log.gz 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.45453.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98294.log 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45453.log.gz 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.76249.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.2.log 2024-04-16T21:13:29.922 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76249.log.gz 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.98294.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91016.log 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98294.log.gz 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182377.log 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.91016.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117697.log 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91016.log.gz 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182377.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50299.log 2024-04-16T21:13:29.923 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182377.log.gz 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.117697.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72882.log 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117697.log.gz 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.50299.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86768.log 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr: 47.0% -- replaced with /var/log/ceph/c1-client.0.50299.log.gz 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.72882.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84949.log 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72882.log.gz 2024-04-16T21:13:29.924 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.86768.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80703.log 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86768.log.gz 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.84949.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93440.log 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr: 57.5% -- replaced with /var/log/ceph/c1-client.0.84949.log.gz 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.80703.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148648.log 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80703.log.gz 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.93440.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112852.log 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93440.log.gz 2024-04-16T21:13:29.925 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.148648.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.133452.log 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148648.log.gz 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.112852.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35853.log 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112852.log.gz 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.133452.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46667.log 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.133452.log.gz 2024-04-16T21:13:29.926 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35853.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35853.log.gz 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.96470.log 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.46667.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.125581.log 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46667.log.gz 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.96470.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.173590.log 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr: 70.5% -- replaced with /var/log/ceph/c1-client.0.96470.log.gz 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.125581.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.148043.log 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.125581.log.gz 2024-04-16T21:13:29.927 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.173590.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.173590.log.gz 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35975.log 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.148043.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69768.log 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.148043.log.gz 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35975.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120117.log 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35975.log.gz 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.69768.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c1-client.0.69768.log.gz /var/log/ceph/c1-client.admin.182293.log 2024-04-16T21:13:29.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.120117.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55205.log 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr: 69.8% -- replaced with /var/log/ceph/c1-client.0.120117.log.gz 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182293.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36578.log 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182293.log.gz 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.55205.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.179684.log 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr: 52.7% -- replaced with /var/log/ceph/c1-client.0.55205.log.gz 2024-04-16T21:13:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36578.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.158958.log 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36578.log.gz 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.179684.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.171085.log 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.179684.log.gz 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.158958.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.158958.log.gz -5 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.0.121935.log 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.171085.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.143789.log 2024-04-16T21:13:29.930 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.171085.log.gz 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.121935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70981.log 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr: 58.6% -- replaced with /var/log/ceph/c1-client.0.121935.log.gz 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.143789.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.177221.log 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.143789.log.gz 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.70981.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.70981.log.gz 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.100113.log 2024-04-16T21:13:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.177221.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35968.log 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.177221.log.gz 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.100113.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.137703.log -- replaced with /var/log/ceph/c1-client.0.100113.log.gz 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35968.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.174801.log 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35968.log.gz 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.137703.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.137703.log.gz 2024-04-16T21:13:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182335.log 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.174801.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.146826.log 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.174801.log.gz 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182335.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.112247.log -- replaced with /var/log/ceph/c1-client.admin.182335.log.gz 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.146826.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c1-client.0.146826.log.gz 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/c1-client.0.115877.log 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.112247.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.151676.log 2024-04-16T21:13:29.933 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112247.log.gz 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.115877.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58238.log 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115877.log.gz 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.151676.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.183174.log 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr: 68.0% -- replaced with /var/log/ceph/c1-client.0.151676.log.gz 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.58238.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.124370.log 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58238.log.gz 2024-04-16T21:13:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.183174.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.183174.log.gz 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1.log 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.124370.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.128612.log 0.0% 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.124370.log.gz 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182671.log 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.128612.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.128612.log.gz 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53381.log 2024-04-16T21:13:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182671.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76848.log 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182671.log.gz 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.53381.log: 52.7% -- replaced with /var/log/ceph/c1-client.0.53381.log.gz 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.129822.log 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.76848.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101932.log 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr: 7.2% -- replaced with /var/log/ceph/c1-client.0.76848.log.gz 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.129822.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.129822.log.gz 2024-04-16T21:13:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.140146.log 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.101932.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.158353.log 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101932.log.gz 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.140146.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.140146.log.gz 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70376.log 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.158353.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122542.log 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.158353.log.gz 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.70376.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70376.log.gz 2024-04-16T21:13:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182923.log 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.122542.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119512.log 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122542.log.gz 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182923.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182923.log.gz 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63701.log 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.119512.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.168665.log 2024-04-16T21:13:29.938 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119512.log.gz 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.63701.log: 46.3% -- replaced with /var/log/ceph/c1-client.0.63701.log.gz 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.124975.log 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.168665.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50906.log 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.168665.log.gz 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.124975.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.124975.log.gz 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36357.log 2024-04-16T21:13:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.50906.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.180303.log 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr: 71.0% -- replaced with /var/log/ceph/c1-client.0.50906.log.gz 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36357.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36357.log.gz 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109216.log 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.180303.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.141360.log 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.180303.log.gz 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.109216.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109216.log.gz 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35642.log 2024-04-16T21:13:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.141360.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66733.log 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.141360.log.gz 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35642.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35642.log.gz 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36167.log 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.66733.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.140752.log 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66733.log.gz 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36167.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36167.log.gz 2024-04-16T21:13:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100718.log 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.140752.log: gzip -5 --verbose -- /var/log/ceph/c1-mon.a.log 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr: 78.9% -- replaced with /var/log/ceph/c1-client.0.140752.log.gz 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.100718.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39755.log 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr: 56.7% -- replaced with /var/log/ceph/c1-client.0.100718.log.gz 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152283.log 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.39755.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39755.log.gz 2024-04-16T21:13:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.105573.log 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr: 89.7% -- replaced with /var/log/ceph/c1.log.gz 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/c1-client.0.152283.log: -5 --verbose -- /var/log/ceph/c1-client.admin.35703.log 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152283.log.gz 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.105573.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.105573.log.gz 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/c1-client.admin.182545.log 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35703.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.180921.log 2024-04-16T21:13:29.943 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35703.log.gz 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182545.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.126186.log 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182545.log.gz 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.180921.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.180921.log.gz 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.166850.log 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.126186.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42338.log 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr: 69.6% -- replaced with /var/log/ceph/c1-client.0.126186.log.gz 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.166850.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39156.log 2024-04-16T21:13:29.944 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.166850.log.gz 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.42338.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35873.log 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42338.log.gz 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.39156.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.171690.log 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr: 1.8% -- replaced with /var/log/ceph/c1-client.0.39156.log.gz 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35873.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35873.log.gz 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36315.log 2024-04-16T21:13:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.171690.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149253.log 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.171690.log.gz 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36315.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182839.log 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36315.log.gz 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.149253.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89801.log 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.149253.log.gz 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182839.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35444.log 2024-04-16T21:13:29.946 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182839.log.gz 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.89801.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.147436.log 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89801.log.gz 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35444.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35444.log.gz 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63094.log 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.147436.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92228.log 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.147436.log.gz 2024-04-16T21:13:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.63094.log: gzip 49.9% -5 --verbose -- replaced with /var/log/ceph/c1-client.0.63094.log.gz 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/c1-client.0.131032.log 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.92228.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53991.log 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr: 57.6% -- replaced with /var/log/ceph/c1-client.0.92228.log.gz 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.131032.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.131032.log.gz 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.150463.log 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.53991.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35223.log 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr: 52.4% -- replaced with /var/log/ceph/c1-client.0.53991.log.gz 2024-04-16T21:13:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.150463.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.150463.log.gz 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.170480.log 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35223.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.admin.35223.log.gz -5 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ops-log-c1-client.0.log 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.170480.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.138933.log 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.170480.log.gz 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ops-log-c1-client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.163214.log 2024-04-16T21:13:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.138933.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182797.log 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.138933.log.gz 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.163214.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.163214.log.gz 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35704.log 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84340.log 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182797.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182797.log.gz 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35704.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35704.log.gz 2024-04-16T21:13:29.950 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35051.log 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75640.log 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.84340.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84340.log.gz 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35051.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/c1-client.admin.35051.log.gz 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/c1-client.0.46062.log 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.75640.log: 14.6% -- replaced with /var/log/ceph/c1-client.0.75640.log.gz 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.116485.log 2024-04-16T21:13:29.951 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68553.log 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.46062.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.46062.log.gz 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.116485.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36273.log 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr: 57.2% -- replaced with /var/log/ceph/c1-client.0.116485.log.gz 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.68553.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.175406.log 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68553.log.gz 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36273.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36273.log.gz 2024-04-16T21:13:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.126797.log 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36105.log 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.175406.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.175406.log.gz 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.126797.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.126797.log.gz 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98899.log 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87978.log 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36105.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36105.log.gz 2024-04-16T21:13:29.953 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.98899.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64914.log 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr: 49.9% -- replaced with /var/log/ceph/c1-client.0.98899.log.gz 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.87978.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86163.log 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87978.log.gz 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.64914.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102537.log 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr: 50.1% -- replaced with /var/log/ceph/c1-client.0.64914.log.gz 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.86163.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65521.log 2024-04-16T21:13:29.954 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86163.log.gz 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.102537.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90406.log 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr: 86.5% -- replaced with /var/log/ceph/c1-client.0.102537.log.gz 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/c1-client.0.65521.log: -5 --verbose -- /var/log/ceph/c1-client.0.177826.log 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr: 46.3% -- replaced with /var/log/ceph/c1-client.0.65521.log.gz 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.90406.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.90406.log.gz 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.123759.log 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.110431.log 2024-04-16T21:13:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.177826.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.177826.log.gz 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.123759.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.123759.log.gz 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.132242.log 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.137093.log 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.110431.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.110431.log.gz 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.132242.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.132242.log.gz 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.123154.log 2024-04-16T21:13:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.101325.log 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.137093.log: 74.0% -- replaced with /var/log/ceph/c1-client.0.137093.log.gz 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.123154.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.123154.log.gz 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35657.log 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35107.log 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.101325.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.101325.log.gz 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35657.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.0.log 2024-04-16T21:13:29.957 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35657.log.gz 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118907.log 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35107.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35107.log.gz 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/rgw.c1.client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35307.log 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.118907.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.0.118907.log.gz 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.0.78136.log 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35307.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35307.log.gz 2024-04-16T21:13:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60665.log 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.78136.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35448.log 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr: 41.3% -- replaced with /var/log/ceph/c1-client.0.78136.log.gz 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.60665.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81309.log 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr: 60.5% -- replaced with /var/log/ceph/c1-client.0.60665.log.gz 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35440.log 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35448.log: /var/log/ceph/c1-client.0.81309.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.169875.log 2024-04-16T21:13:29.959 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81309.log.gz 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35448.log.gz 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35440.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182251.log 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35440.log.gz 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.169875.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64308.log 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.169875.log.gz 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182251.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/c1-client.admin.182251.log.gz 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/c1-client.0.72191.log 2024-04-16T21:13:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.64308.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.64308.log.gz 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.132847.log 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.72191.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72191.log.gz 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.151068.log 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.132847.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.0.132847.log.gz 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.admin.182713.log 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.151068.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.151068.log.gz 2024-04-16T21:13:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55812.log 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182713.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182713.log.gz 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.144394.log 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.55812.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55812.log.gz 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60060.log 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.144394.log: 67.8% -- replaced with /var/log/ceph/c1-client.0.144394.log.gz 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49087.log 2024-04-16T21:13:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.60060.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.60060.log.gz 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78735.log 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.49087.log: 63.4% -- replaced with /var/log/ceph/c1-client.0.49087.log.gz 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35974.log 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.78735.log: 4.8% -- replaced with /var/log/ceph/c1-client.0.78735.log.gz 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.138325.log 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35974.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35974.log.gz 2024-04-16T21:13:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.165634.log 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.138325.log: 76.0% -- replaced with /var/log/ceph/c1-client.0.138325.log.gz 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.164424.log 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.165634.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.165634.log.gz 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.128007.log 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.164424.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.164424.log.gz 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88583.log 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.128007.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.128007.log.gz 2024-04-16T21:13:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52167.log 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.88583.log: 84.9% -- replaced with /var/log/ceph/c1-client.0.88583.log.gz 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74960.log 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.52167.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.52167.log.gz 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.155321.log 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.74960.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74960.log.gz 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114062.log 2024-04-16T21:13:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.155321.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.155321.log.gz 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95860.log 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.114062.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.114062.log.gz 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40436.log 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.95860.log: 69.3% -- replaced with /var/log/ceph/c1-client.0.95860.log.gz 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118302.log 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.40436.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108610.log 2024-04-16T21:13:29.966 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40436.log.gz 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.118302.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87373.log 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118302.log.gz 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose/var/log/ceph/c1-client.0.108610.log: -- /var/log/ceph/c1-client.0.146221.log 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108610.log.gz 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.87373.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.87373.log.gz 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/c1-client.admin.35913.log 2024-04-16T21:13:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.146221.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.0.146221.log.gz 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.admin.35449.log 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35913.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35913.log.gz 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.106178.log 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35449.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35449.log.gz 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28530.log 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.106178.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.106178.log.gz 2024-04-16T21:13:29.968 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.94045.log 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.tmp-client.admin.28530.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28530.log.gz 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79383.log 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.94045.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.94045.log.gz 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73752.log 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.79383.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.79383.log.gz 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.91621.log 2024-04-16T21:13:29.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.73752.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.73752.log.gz 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.169270.log 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.91621.log: 58.1% -- replaced with /var/log/ceph/c1-client.0.91621.log.gz 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.103144.log 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.169270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182587.log 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.169270.log.gz 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.103144.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36620.log 2024-04-16T21:13:29.970 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103144.log.gz 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62488.log 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182587.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182587.log.gz 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36620.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35635.log 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36620.log.gz 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.62488.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/c1-client.0.62488.log.gz 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/c1-client.0.178447.log 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35635.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35635.log.gz 2024-04-16T21:13:29.971 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.71586.log 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.178447.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.178447.log.gz 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/c1-client.admin.183007.log 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.71586.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.71586.log.gz 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.157748.log 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.183007.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120725.log 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.183007.log.gz 2024-04-16T21:13:29.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.157748.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83729.log 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.157748.log.gz 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.120725.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37256.log 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120725.log.gz 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.83729.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67338.log 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83729.log.gz 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.37256.log: gzip -5 --verbose 43.9% -- replaced with /var/log/ceph/c1-client.0.37256.log.gz 2024-04-16T21:13:29.973 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/c1-client.0.141965.log 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.67338.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67338.log.gz 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80013.log 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.141965.log: 77.6% -- replaced with /var/log/ceph/c1-client.0.141965.log.gz 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.166245.log 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.80013.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.80013.log.gz 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.115272.log 2024-04-16T21:13:29.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.166245.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.166245.log.gz 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.135878.log 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.115272.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.115272.log.gz 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104357.log 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.135878.log: 55.3% -- replaced with /var/log/ceph/c1-client.0.135878.log.gz 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.153498.log 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose --/var/log/ceph/c1-client.0.104357.log: /var/log/ceph/c1-client.admin.35445.log 2024-04-16T21:13:29.975 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104357.log.gz 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.153498.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.153498.log.gz 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77496.log 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35445.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.admin.35445.log.gz 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.0.156537.log 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.77496.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77496.log.gz 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-osd.1.log 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.156537.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.156537.log.gz 2024-04-16T21:13:29.976 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35661.log 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97682.log 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35661.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35661.log.gz 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.183091.log 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.97682.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.97682.log.gz 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95255.log 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.183091.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.183091.log.gz 2024-04-16T21:13:29.977 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.38557.log 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.95255.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95255.log.gz 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182629.log 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.38557.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.38557.log.gz 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182881.log 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182629.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182629.log.gz 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.182461.log 2024-04-16T21:13:29.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182881.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182881.log.gz 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.162609.log 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182461.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182461.log.gz 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81914.log 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.162609.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.162609.log.gz 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74361.log 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.81914.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.81914.log.gz 2024-04-16T21:13:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69163.log 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.74361.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74361.log.gz 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85556.log 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.69163.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.69163.log.gz 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.129217.log 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.85556.log: 57.5% -- replaced with /var/log/ceph/c1-client.0.85556.log.gz 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.145614.log 2024-04-16T21:13:29.980 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5/var/log/ceph/c1-client.0.129217.log: --verbose -- /var/log/ceph/c1-client.0.113457.log 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.129217.log.gz 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.145614.log: gzip -5 --verbose -- 54.4% -- replaced with /var/log/ceph/c1-client.0.145614.log.gz 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/c1-client.0.145001.log 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.113457.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.113457.log.gz 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57025.log 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr: 95.2% -- replaced with /var/log/ceph/ops-log-c1-client.0.log.gz 2024-04-16T21:13:29.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.145001.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.0.145001.log.gz 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.0.160171.log 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.57025.log: 63.4% -- replaced with /var/log/ceph/c1-client.0.57025.log.gz 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.174196.log 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.160171.log: gzip -5 --verbose 0.0% -- /var/log/ceph/c1-client.admin.40371.log 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.160171.log.gz 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.174196.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.174196.log.gz 2024-04-16T21:13:29.982 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.176011.log 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.40371.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111642.log 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.40371.log.gz 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.176011.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66128.log 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.176011.log.gz 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.111642.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.157143.log 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111642.log.gz 2024-04-16T21:13:29.983 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.139538.log 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.66128.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66128.log.gz 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.157143.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47272.log 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.157143.log.gz 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.139538.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.142574.log 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr: 76.8% -- replaced with /var/log/ceph/c1-client.0.139538.log.gz 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.47272.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47272.log.gz 2024-04-16T21:13:29.984 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.131637.log 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.142574.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59453.log 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.142574.log.gz 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.131637.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36125.log 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.131637.log.gz 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.59453.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43030.log 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr: 46.7% -- replaced with /var/log/ceph/c1-client.0.59453.log.gz 2024-04-16T21:13:29.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36125.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.165029.log 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36125.log.gz 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.43030.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111036.log 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43030.log.gz 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.165029.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.172295.log 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.165029.log.gz 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.111036.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67943.log 2024-04-16T21:13:29.986 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111036.log.gz 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.172295.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.0.172295.log.gz 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.admin.182755.log 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.67943.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67943.log.gz 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43635.log 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182755.log: gzip -5 0.0% -- replaced with /var/log/ceph/c1-client.admin.182755.log.gz 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/c1-client.admin.182419.log 2024-04-16T21:13:29.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.43635.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.43635.log.gz 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.134662.log 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.182419.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.154104.log 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.182419.log.gz 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.134662.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36643.log 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr: 84.8% -- replaced with /var/log/ceph/c1-client.0.134662.log.gz 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.35447.log 2024-04-16T21:13:29.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.154104.log: /var/log/ceph/c1-client.0.36643.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.127402.log 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr: 83.1% -- replaced with /var/log/ceph/c1-client.0.154104.log.gz 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr: 83.7% -- replaced with /var/log/ceph/c1-client.0.36643.log.gz 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.99508.log 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.35447.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.35447.log.gz 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.127402.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106784.log 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.127402.log.gz 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.99508.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.168060.log 2024-04-16T21:13:29.989 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99508.log.gz 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.106784.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36496.log 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr: 58.3% -- replaced with /var/log/ceph/c1-client.0.106784.log.gz 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.168060.log: gzip -5 --verbose 0.0% -- /var/log/ceph/c1-client.0.61273.log 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.168060.log.gz 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36496.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36496.log.gz 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.121330.log 2024-04-16T21:13:29.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.61273.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104962.log 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61273.log.gz 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.121330.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.152888.log 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121330.log.gz 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.104962.log: gzip 0.0% -5 --verbose -- /var/log/ceph/c1-client.0.41035.log 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.104962.log.gz 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.152888.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.0.log 2024-04-16T21:13:29.991 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.152888.log.gz 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.41035.log: gzip -5 --verbose -- 17.8% -- replaced with /var/log/ceph/c1-client.0.41035.log.gz 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/c1-client.0.97077.log 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.179069.log 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.97077.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.97077.log.gz 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44845.log 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.179069.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.179069.log.gz 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.31773.log 2024-04-16T21:13:29.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.44845.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44845.log.gz 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.58848.log 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.31773.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.31773.log.gz 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83124.log 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.58848.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.58848.log.gz 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.107395.log 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.83124.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83124.log.gz 2024-04-16T21:13:29.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82519.log 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.107395.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.107395.log.gz 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.32039.log 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.82519.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.82519.log.gz 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.61879.log 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.32039.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.32039.log.gz 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.155927.log 2024-04-16T21:13:29.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.61879.log: 70.6%gzip -5 --verbose -- /var/log/ceph/c1-client.admin.36538.log 2024-04-16T21:13:29.995 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/c1-client.0.61879.log.gz 2024-04-16T21:13:29.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.155927.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.149858.log 2024-04-16T21:13:29.995 INFO:teuthology.orchestra.run.smithi026.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.155927.log.gz 2024-04-16T21:13:29.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.admin.36538.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.36538.log.gz 2024-04-16T21:13:29.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/c1-client.0.149858.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.149858.log.gz 2024-04-16T21:13:29.997 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.63822.log 2024-04-16T21:13:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.123312.log 2024-04-16T21:13:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.210793.log 2024-04-16T21:13:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.63822.log: /var/log/ceph/c2-client.0.123312.log: 92.2% -- replaced with /var/log/ceph/c2-client.0.63822.log.gz 2024-04-16T21:13:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.83905.log 2024-04-16T21:13:29.998 INFO:teuthology.orchestra.run.smithi119.stderr: 57.0% -- replaced with /var/log/ceph/c2-client.0.123312.log.gz 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.226897.log 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.210793.log: 92.1% -- replaced with /var/log/ceph/c2-client.0.210793.log.gz 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.83905.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.132543.log 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.83905.log.gz 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.226897.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.164312.log 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.226897.log.gz 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.132543.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.134972.log 2024-04-16T21:13:29.999 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.132543.log.gz 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.164312.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.96588.log 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.164312.log.gz 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.134972.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.232369.log 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr: 90.0% -- replaced with /var/log/ceph/c2-client.0.134972.log.gz 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.114183.log 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.96588.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.96588.log.gz 2024-04-16T21:13:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.232369.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.175750.log 2024-04-16T21:13:30.001 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.232369.log.gz 2024-04-16T21:13:30.001 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.114183.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.36806.log 2024-04-16T21:13:30.001 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.114183.log.gz 2024-04-16T21:13:30.001 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.175750.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.55319.log 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.175750.log.gz 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.36806.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36049.log 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr: 83.8% -- replaced with /var/log/ceph/c2-client.0.36806.log.gz 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.55319.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32636.log 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr: 50.4% -- replaced with /var/log/ceph/c2-client.0.55319.log.gz 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36049.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.191022.log 2024-04-16T21:13:30.002 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36049.log.gz 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.32636.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.144217.log 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.32636.log.gz 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.191022.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.143001.log 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.191022.log.gz 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.144217.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.62001.log 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.144217.log.gz 2024-04-16T21:13:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.143001.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.57758.log 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.143001.log.gz 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.62001.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.226289.log 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr: 49.6% -- replaced with /var/log/ceph/c2-client.0.62001.log.gz 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.57758.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.145434.log 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr: 53.3% -- replaced with /var/log/ceph/c2-client.0.57758.log.gz 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.226289.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.207761.log 2024-04-16T21:13:30.004 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.226289.log.gz 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.145434.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.222032.log 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.145434.log.gz 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.207761.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.121461.log 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.207761.log.gz 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.222032.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.56534.log 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.222032.log.gz 2024-04-16T21:13:30.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.121461.log: gzip -5 --verbose -- /var/log/ceph/c2-mon.a.log 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.121461.log.gz 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.56534.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.125138.log 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr: 91.2% -- replaced with /var/log/ceph/c2-client.0.56534.log.gz 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.66878.log 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.125138.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.186763.log 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.125138.log.gz 2024-04-16T21:13:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.66878.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.60791.log 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr: 50.4% -- replaced with /var/log/ceph/c2-client.0.66878.log.gz 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.186763.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.78368.log 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.186763.log.gz 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.60791.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.136826.log 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.60791.log.gz 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.78368.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.0.78368.log.gz -5 2024-04-16T21:13:30.007 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.178158.log 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.136826.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.91723.log 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr: 67.9% -- replaced with /var/log/ceph/c2-client.0.136826.log.gz 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.178158.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.0.178158.log.gz -5 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.221424.log 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.91723.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.156486.log 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.91723.log.gz 2024-04-16T21:13:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.221424.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.206547.log 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.221424.log.gz 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.156486.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35837.log 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.156486.log.gz 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.206547.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.0.206547.log.gz -5 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.230545.log 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35837.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.161904.log 2024-04-16T21:13:30.009 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35837.log.gz 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.230545.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.230545.log.gz 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.58364.log 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.224464.log 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.161904.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.161904.log.gz 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.58364.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36493.log 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.58364.log.gz 2024-04-16T21:13:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.224464.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.134368.log 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.224464.log.gz 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36493.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.157088.log 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36493.log.gz 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.134368.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35830.log 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr: 57.3% -- replaced with /var/log/ceph/c2-client.0.134368.log.gz 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.157088.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.157088.log.gz 2024-04-16T21:13:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36535.log 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.142393.log 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35830.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35830.log.gz 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36535.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.73550.log 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36535.log.gz 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.142393.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.104486.log 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.142393.log.gz 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.73550.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29140.log 2024-04-16T21:13:30.012 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.73550.log.gz 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.104486.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.0.104486.log.gz 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.99629.log 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.tmp-client.admin.29140.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.150911.log 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29140.log.gz 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.99629.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35825.log 2024-04-16T21:13:30.013 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.99629.log.gz 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.150911.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.109943.log 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.150911.log.gz 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35825.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35825.log.gz 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.202178.log 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.44955.log 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.109943.log: 78.6% -- replaced with /var/log/ceph/c2-client.0.109943.log.gz 2024-04-16T21:13:30.014 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.202178.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.223248.log 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.202178.log.gz 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.44955.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35437.log 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.44955.log.gz 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.223248.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.217766.log 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.223248.log.gz 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35437.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35437.log.gz 2024-04-16T21:13:30.015 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36741.log 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.217766.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36241.log 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.217766.log.gz 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36741.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36741.log.gz 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.41909.log 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36241.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.148479.log 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36241.log.gz 2024-04-16T21:13:30.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.41909.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.233041.log 46.1% 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.41909.log.gz 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.148479.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.115397.log 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.148479.log.gz 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233041.log: gzip -5 --verbose 8.2% -- -- replaced with /var/log/ceph/c2-client.0.233041.log.gz 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.200349.log 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.115397.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.114791.log 2024-04-16T21:13:30.017 INFO:teuthology.orchestra.run.smithi119.stderr: 89.9% -- replaced with /var/log/ceph/c2-client.0.115397.log.gz 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.200349.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.74756.log 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr: 88.8% -- replaced with /var/log/ceph/c2-client.0.200349.log.gz 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.114791.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.229937.log 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr: 71.5% -- replaced with /var/log/ceph/c2-client.0.114791.log.gz 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.74756.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.74756.log.gz 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.117822.log 2024-04-16T21:13:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.229937.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.179376.log 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.229937.log.gz 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.117822.log: 50.1% -- replaced with /var/log/ceph/c2-client.0.117822.log.gz 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.127071.log 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.179376.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.89063.log 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.179376.log.gz 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.127071.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.127071.log.gz 2024-04-16T21:13:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.68695.log 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.89063.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.213401.log 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr: 13.6% -- replaced with /var/log/ceph/c2-client.0.89063.log.gz 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.68695.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.68695.log.gz 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.40001.log 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.213401.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.171536.log 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.213401.log.gz 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.40001.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.184328.log 0.0% 2024-04-16T21:13:30.020 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.40001.log.gz 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.171536.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.211413.log 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.171536.log.gz 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.184328.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.182504.log 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.184328.log.gz 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.211413.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.39936.log 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr: 41.9% -- replaced with /var/log/ceph/c2-client.0.211413.log.gz 2024-04-16T21:13:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.182504.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.182504.log.gz 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.105093.log 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/c2-client.admin.39936.log: -5 --verbose -- /var/log/ceph/c2-client.0.64446.log 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.39936.log.gz 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.105093.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.210185.log 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.105093.log.gz 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.64446.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.64446.log.gz 2024-04-16T21:13:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.66270.log 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.210185.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.173342.log 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.210185.log.gz 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.66270.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.102062.log 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.66270.log.gz 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.173342.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35537.log 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.173342.log.gz 2024-04-16T21:13:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.102062.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.102062.log.gz 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.106310.log 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.87163.log 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35537.log.gz 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.106310.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.106310.log.gz 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.97196.log 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.87163.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.177556.log 2024-04-16T21:13:30.024 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.87163.log.gz 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.97196.log: gzip -5 46.1% --verbose -- -- replaced with /var/log/ceph/c2-client.0.97196.log.gz 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.143609.log 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36382.log 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.177556.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.177556.log.gz 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.143609.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.143609.log.gz 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.44347.log 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36382.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.122069.log 2024-04-16T21:13:30.025 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36382.log.gz 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.44347.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.93548.log 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.44347.log.gz 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.122069.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.202786.log 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr: 57.7% -- replaced with /var/log/ceph/c2-client.0.122069.log.gz 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.93548.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.93548.log.gz 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.69303.log 2024-04-16T21:13:30.026 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.90507.log 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.202786.log: 50.4% -- replaced with /var/log/ceph/c2-client.0.202786.log.gz 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.69303.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.158292.log 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr: 67.8% -- replaced with /var/log/ceph/c2-client.0.69303.log.gz 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.176954.log 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.90507.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.90507.log.gz 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.158292.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.76565.log 2024-04-16T21:13:30.027 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.158292.log.gz 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.176954.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.176954.log.gz 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36783.log 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.76565.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.183112.log 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.76565.log.gz 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36783.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36783.log.gz 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.218374.log 2024-04-16T21:13:30.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.183112.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.79656.log 2024-04-16T21:13:30.029 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.183112.log.gz 2024-04-16T21:13:30.029 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.218374.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.218374.log.gz 2024-04-16T21:13:30.029 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.43125.log 2024-04-16T21:13:30.029 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.79656.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.59576.log 2024-04-16T21:13:30.029 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.79656.log.gz 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.43125.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.70511.log 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.43125.log.gz 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.59576.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.155884.log 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr: 81.7% -- replaced with /var/log/ceph/c2-client.0.59576.log.gz 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.70511.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.149087.log 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.70511.log.gz 2024-04-16T21:13:30.030 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/c2-client.0.155884.log: -- /var/log/ceph/c2-client.0.100846.log 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.155884.log.gz 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.149087.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.201576.log 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.149087.log.gz 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.100846.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.105703.log 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.100846.log.gz 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.201576.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.129503.log 0.0% 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.201576.log.gz 2024-04-16T21:13:30.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.105703.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.164914.log 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.105703.log.gz 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.129503.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.113575.log 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.129503.log.gz 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.164914.log: gzip -5 --verbose -- /var/log/ceph/c2.log 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.164914.log.gz 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.113575.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.113575.log.gz 2024-04-16T21:13:30.032 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.83306.log 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.205940.log 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.83306.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.119638.log 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.83306.log.gz 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.205940.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.205940.log.gz 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2.audit.log 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.229329.log 2024-04-16T21:13:30.033 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.119638.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.119638.log.gz 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2.audit.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35834.log 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.229329.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.158894.log 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.229329.log.gz 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr: 89.9% -- replaced with /var/log/ceph/c2.audit.log.gz 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35834.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.122704.log 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35834.log.gz 2024-04-16T21:13:30.034 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.215228.log 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.158894.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.158894.log.gz 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.122704.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.77769.log 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.122704.log.gz 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.215228.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.135590.log 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr: 41.9% -- replaced with /var/log/ceph/c2-client.0.215228.log.gz 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.77769.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.65662.log 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.77769.log.gz 2024-04-16T21:13:30.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.135590.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.160700.log 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr: 41.7% -- replaced with /var/log/ceph/c2-client.0.135590.log.gz 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.65662.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36619.log 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.65662.log.gz 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.160700.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.152761.log 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.160700.log.gz 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36619.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36619.log.gz 2024-04-16T21:13:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.208368.log 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.162506.log 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.152761.log: 89.6% -- replaced with /var/log/ceph/c2-client.0.152761.log.gz 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr: 89.6% -- replaced with /var/log/ceph/c2.log.gz 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.208368.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.208368.log.gz 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.47387.log 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.86478.log 2024-04-16T21:13:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.162506.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.162506.log.gz 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.47387.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.74154.log 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.47387.log.gz 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.86478.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.86478.log.gz 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.69907.log 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.74154.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.85879.log 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.74154.log.gz 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.69907.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.186155.log 2024-04-16T21:13:30.038 INFO:teuthology.orchestra.run.smithi119.stderr: 46.3% -- replaced with /var/log/ceph/c2-client.0.69907.log.gz 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.85879.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c2-client.0.85879.log.gz 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.72335.log 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.186155.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36337.log 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.186155.log.gz 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.72335.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.208975.log 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.72335.log.gz 2024-04-16T21:13:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36337.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.admin.36337.log.gz -5 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.147259.log 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.208975.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.176352.log 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.208975.log.gz 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.147259.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.147259.log.gz 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35645.log 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.176352.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.204709.log 2024-04-16T21:13:30.040 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.176352.log.gz 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35645.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.35645.log.gz 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.102670.log 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.204709.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.214618.log 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr: 92.2% -- replaced with /var/log/ceph/c2-client.0.204709.log.gz 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.102670.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.0.102670.log.gz -5 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.198423.log 2024-04-16T21:13:30.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.214618.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.54103.log 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr: 55.6% -- replaced with /var/log/ceph/c2-client.0.214618.log.gz 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.198423.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.194671.log 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.198423.log.gz 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.54103.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36338.log 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.54103.log.gz 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.194671.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.108733.log 2024-04-16T21:13:30.042 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.194671.log.gz 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36338.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.admin.36338.log.gz 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/ops-log-c2-client.0.log 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.108733.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.77167.log 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr: 56.8% -- replaced with /var/log/ceph/c2-client.0.108733.log.gz 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ops-log-c2-client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.146042.log 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.77167.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.77167.log.gz 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.165516.log 2024-04-16T21:13:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.124525.log 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.146042.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.146042.log.gz 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.165516.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c2-client.0.165516.log.gz /var/log/ceph/c2-client.0.141177.log 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.124525.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.196597.log 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr: 86.6% -- replaced with /var/log/ceph/c2-client.0.124525.log.gz 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.141177.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.159496.log 2024-04-16T21:13:30.044 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.141177.log.gz 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.45563.log 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.196597.log: 89.0% -- replaced with /var/log/ceph/c2-client.0.196597.log.gz 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.159496.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.180684.log 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.159496.log.gz 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.45563.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.199031.log 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.45563.log.gz 2024-04-16T21:13:30.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.180684.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.120857.log 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr: 66.7% -- replaced with /var/log/ceph/c2-client.0.180684.log.gz 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.199031.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.233004.log 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr: 41.9% -- replaced with /var/log/ceph/c2-client.0.199031.log.gz 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.120857.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.98412.log 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr: 56.0% -- replaced with /var/log/ceph/c2-client.0.120857.log.gz 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233004.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.128287.log 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr: 8.2% -- replaced with /var/log/ceph/c2-client.0.233004.log.gz 2024-04-16T21:13:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.98412.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.49819.log 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.98412.log.gz 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.128287.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.55927.log 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.128287.log.gz 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.49819.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.97804.log 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr: 62.9% -- replaced with /var/log/ceph/c2-client.0.49819.log.gz 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.55927.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.181896.log 2024-04-16T21:13:30.047 INFO:teuthology.orchestra.run.smithi119.stderr: 72.8% -- replaced with /var/log/ceph/c2-client.0.55927.log.gz 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.97804.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36659.log 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.97804.log.gz 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.181896.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.71119.log 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.181896.log.gz 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36659.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.212731.log 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36659.log.gz 2024-04-16T21:13:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.71119.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.65054.log 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.71119.log.gz 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.212731.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.197821.log 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr: 97.2% -- replaced with /var/log/ceph/c2-client.0.212731.log.gz 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.65054.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.49211.log 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr: 60.5% -- replaced with /var/log/ceph/c2-client.0.65054.log.gz 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.197821.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.170934.log 2024-04-16T21:13:30.049 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.197821.log.gz 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.49211.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.166118.log 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.49211.log.gz 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/c2-client.0.170934.log: -- /var/log/ceph/c2-client.0.82698.log 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.170934.log.gz 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.166118.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.166118.log.gz 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.157690.log 2024-04-16T21:13:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.151519.log 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.82698.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.82698.log.gz 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.157690.log: 0.0%gzip -- replaced with /var/log/ceph/c2-client.0.157690.log.gz -5 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.91115.log 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.151519.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35579.log 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr: 55.1% -- replaced with /var/log/ceph/c2-client.0.151519.log.gz 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.91115.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.91115.log.gz 2024-04-16T21:13:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.154642.log 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35579.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.39978.log 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35579.log.gz 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.154642.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.170332.log 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr: 92.5% -- replaced with /var/log/ceph/c2-client.0.154642.log.gz 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32370.log 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.39978.log: 0.0%/var/log/ceph/c2-client.0.170332.log: -- replaced with /var/log/ceph/c2-client.admin.39978.log.gz 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.170332.log.gz 2024-04-16T21:13:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.131935.log 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.169128.log 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.32370.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.32370.log.gz 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.131935.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.110548.log 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.131935.log.gz 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.169128.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.131327.log 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.169128.log.gz 2024-04-16T21:13:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.110548.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c2-client.0.110548.log.gz /var/log/ceph/c2-client.0.57153.log 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.131327.log: 45.5%gzip -- replaced with /var/log/ceph/c2-client.0.131327.log.gz 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.192238.log 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35835.log 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.57153.log: 66.0% -- replaced with /var/log/ceph/c2-client.0.57153.log.gz 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.192238.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.192238.log.gz 2024-04-16T21:13:30.054 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36577.log 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.150303.log 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35835.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35835.log.gz 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36577.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36577.log.gz 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.138049.log 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.220815.log 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.150303.log: /var/log/ceph/c2-client.0.138049.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.150303.log.gz 2024-04-16T21:13:30.055 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.225681.log 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.138049.log.gz 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.220815.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.87771.log 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.220815.log.gz 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.225681.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36281.log 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.225681.log.gz 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.87771.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.2.log 2024-04-16T21:13:30.056 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.87771.log.gz 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36281.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36281.log.gz 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.231761.log 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.189803.log 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-osd.2.log: /var/log/ceph/c2-client.0.231761.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.231761.log.gz 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.46171.log 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.167924.log 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.189803.log: 46.8% -- replaced with /var/log/ceph/c2-client.0.189803.log.gz 2024-04-16T21:13:30.057 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.46171.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.136218.log 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.46171.log.gz 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.167924.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.133760.log 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.167924.log.gz 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.136218.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.60184.log 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr: 46.8% -- replaced with /var/log/ceph/c2-client.0.136218.log.gz 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.79048.log 2024-04-16T21:13:30.058 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.133760.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.133760.log.gz 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.60184.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.120249.log 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.60184.log.gz 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/c2-client.0.79048.log: -5 --verbose -- /var/log/ceph/c2-client.0.222640.log 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.79048.log.gz 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.120249.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.152128.log 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.120249.log.gz 2024-04-16T21:13:30.059 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/c2-client.0.222640.log: -- /var/log/ceph/c2-client.admin.36110.log 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.222640.log.gz 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.152128.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.48603.log 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr: 93.9% -- replaced with /var/log/ceph/c2-client.0.152128.log.gz 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36110.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.149695.log 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36110.log.gz 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.48603.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35493.log 2024-04-16T21:13:30.060 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.48603.log.gz 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.112970.log 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.149695.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.149695.log.gz 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35493.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.174546.log 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35493.log.gz 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.112970.log: gzip 57.1% -- replaced with /var/log/ceph/c2-client.0.112970.log.gz 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.190411.log 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.174546.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.51639.log 2024-04-16T21:13:30.061 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.174546.log.gz 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.190411.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.101454.log 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.190411.log.gz 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.51639.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.85189.log 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr: 47.0% -- replaced with /var/log/ceph/c2-client.0.51639.log.gz 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.101454.log: gzip -5 --verbose 46.1% -- replaced with /var/log/ceph/c2-client.0.101454.log.gz 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr: -- /var/log/ceph/c2-client.0.144826.log 2024-04-16T21:13:30.062 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.85189.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.85189.log.gz 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35695.log 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.144826.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.100237.log 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.144826.log.gz 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35695.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.75963.log 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35695.log.gz 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.38720.log 2024-04-16T21:13:30.063 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.100237.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.100237.log.gz 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.75963.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.0.75963.log.gz 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.193455.log 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.38720.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.38720.log.gz 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.188587.log 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.193455.log: gzip -5 --verbose -- 46.1% -- replaced with /var/log/ceph/c2-client.0.193455.log.gz 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.94764.log 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.188587.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.188587.log.gz 2024-04-16T21:13:30.064 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.88370.log 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.94764.log: gzip -5 46.1% -- replaced with /var/log/ceph/c2-client.0.94764.log.gz 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.233131.log 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.88370.log: 18.5% -- replaced with /var/log/ceph/c2-client.0.88370.log.gz 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.72943.log 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233131.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.153392.log 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr: 9.2% -- replaced with /var/log/ceph/c2-client.0.233131.log.gz 2024-04-16T21:13:30.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.72943.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c2-client.0.163710.log 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.72943.log.gz 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.203391.log 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.153392.log: /var/log/ceph/c2-client.0.163710.log: 94.3% -- replaced with /var/log/ceph/c2-client.0.153392.log.gz 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 0.0% -- replaced with /var/log/ceph/c2-client.0.163710.log.gz 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.233231.log 2024-04-16T21:13:30.066 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.203391.log: 41.9% -- replaced with /var/log/ceph/c2-client.0.203391.log.gz 2024-04-16T21:13:30.069 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.225072.log 2024-04-16T21:13:30.069 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233231.log: gzip -5 --verbose 8.2% -- replaced with /var/log/ceph/c2-client.0.233231.log.gz 2024-04-16T21:13:30.069 INFO:teuthology.orchestra.run.smithi119.stderr: -- /var/log/ceph/c2-client.0.52243.log 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.225072.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.225072.log.gz 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.58972.log 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.155264.log 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.52243.log: 70.9% -- replaced with /var/log/ceph/c2-client.0.52243.log.gz 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.58972.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.169730.log 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr: 50.4% -- replaced with /var/log/ceph/c2-client.0.58972.log.gz 2024-04-16T21:13:30.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.155264.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.103272.log 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr: 93.1% -- replaced with /var/log/ceph/c2-client.0.155264.log.gz 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.80872.log 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.169730.log: /var/log/ceph/c2-client.0.103272.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.169730.log.gz 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 0.0% -- replaced with /var/log/ceph/c2-client.0.103272.log.gz 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.172138.log 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.80872.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.163108.log 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.80872.log.gz 2024-04-16T21:13:30.071 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.172138.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c2-client.0.92940.log 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.172138.log.gz 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose --/var/log/ceph/c2-client.0.163108.log: /var/log/ceph/c2-client.0.130719.log 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.163108.log.gz 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.92940.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.118426.log 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.92940.log.gz 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.130719.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.227505.log 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.130719.log.gz 2024-04-16T21:13:30.072 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.118426.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.228113.log 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.118426.log.gz 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.227505.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.53460.log 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.227505.log.gz 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.228113.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.173944.log 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.228113.log.gz 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose --/var/log/ceph/c2-client.0.53460.log: /var/log/ceph/c2-client.admin.233276.log 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr: 91.3%/var/log/ceph/c2-client.0.173944.log: -- replaced with /var/log/ceph/c2-client.0.53460.log.gz 2024-04-16T21:13:30.073 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.173944.log.gz 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.82088.log 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.116005.log 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.233276.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.233276.log.gz 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.82088.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.168526.log 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.82088.log.gz 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.51031.log 2024-04-16T21:13:30.074 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.116005.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.116005.log.gz 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.168526.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.207154.log 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.168526.log.gz 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.80264.log 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.51031.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.51031.log.gz 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.207154.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.111757.log 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.207154.log.gz 2024-04-16T21:13:30.075 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.80264.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.139357.log 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.80264.log.gz 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.111757.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.181288.log 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.111757.log.gz 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.47995.log 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.139357.log: 82.5% -- replaced with /var/log/ceph/c2-client.0.139357.log.gz 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.181288.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.232977.log 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.181288.log.gz 2024-04-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose --/var/log/ceph/c2-client.0.47995.log: /var/log/ceph/c2-client.0.40600.log 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.47995.log.gz 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.232977.log: 8.2% -- replaced with /var/log/ceph/c2-client.0.232977.log.gz 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.147867.log 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.40600.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36034.log 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr: 19.1% -- replaced with /var/log/ceph/c2-client.0.40600.log.gz 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.161302.log 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.147867.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.147867.log.gz 2024-04-16T21:13:30.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36034.log: gzip 0.0% -- replaced with /var/log/ceph/c2-client.admin.36034.log.gz 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.84590.log 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.161302.log: gzip -5 0.0% -- replaced with /var/log/ceph/c2-client.0.161302.log.gz 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.admin.35988.log 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.84590.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.84590.log.gz 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.192846.log 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/c2-client.admin.35988.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/c2-client.admin.35988.log.gz 2024-04-16T21:13:30.078 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.46779.log 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.192846.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.192846.log.gz 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.68087.log 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.133152.log 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.46779.log: 46.8% -- replaced with /var/log/ceph/c2-client.0.46779.log.gz 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.68087.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.62606.log 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.68087.log.gz 2024-04-16T21:13:30.079 INFO:teuthology.orchestra.run.smithi119.stderr: 95.3% -- replaced with /var/log/ceph/ops-log-c2-client.0.log.gzgzip -5 --verbose -- /var/log/ceph/c2-client.0.75361.log 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.133152.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.133152.log.gz 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.62606.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.154022.log 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.62606.log.gz 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.52852.log 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.75361.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.75361.log.gz 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.154022.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.61393.log 2024-04-16T21:13:30.080 INFO:teuthology.orchestra.run.smithi119.stderr: 92.7% -- replaced with /var/log/ceph/c2-client.0.154022.log.gz 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.52852.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.141785.log 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.52852.log.gz 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.61393.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.187371.log 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.61393.log.gz 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.141785.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.172740.log 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.141785.log.gz 2024-04-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.187371.log: 46.1%gzip -- replaced with /var/log/ceph/c2-client.0.187371.log.gz 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/c2-client.0.95980.log 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.123923.log 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.172740.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.172740.log.gz 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.95980.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.43739.log 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.95980.log.gz 2024-04-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.123923.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.223856.log 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.123923.log.gz 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.43739.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.130111.log 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.43739.log.gz 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.223856.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.216546.log 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.223856.log.gz 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.130111.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.205333.log 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.130111.log.gz 2024-04-16T21:13:30.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.216546.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36701.log 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr: 91.4% -- replaced with /var/log/ceph/c2-client.0.216546.log.gz 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.205333.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.205333.log.gz 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.117214.log 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.108129.log 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36701.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36701.log.gz 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.117214.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.166720.log 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr: 45.5% -- replaced with /var/log/ceph/c2-client.0.117214.log.gz 2024-04-16T21:13:30.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.108129.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.111155.log 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr: 57.1% -- replaced with /var/log/ceph/c2-client.0.108129.log.gz 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.166720.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.166720.log.gz 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.94156.log 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.231153.log 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.111155.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.111155.log.gz 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.94156.log: gzip 46.1% -- replaced with /var/log/ceph/c2-client.0.94156.log.gz -5 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.217158.log 2024-04-16T21:13:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.214010.log 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.231153.log: 45.5% -- replaced with /var/log/ceph/c2-client.0.231153.log.gz 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.217158.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.217158.log.gz 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.92332.log 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.103879.log 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.214010.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.214010.log.gz/var/log/ceph/c2-client.0.92332.log: 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.92332.log.gz 2024-04-16T21:13:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.140569.log 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.178762.log 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.103879.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.103879.log.gz 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.140569.log: gzip 46.1% -- replaced with /var/log/ceph/c2-client.0.140569.log.gz -5 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.175148.log 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.200969.log 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.178762.log: /var/log/ceph/c2-client.0.175148.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.175148.log.gz 2024-04-16T21:13:30.087 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.50423.log 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.191630.log 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.200969.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.200969.log.gz 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.50423.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.42517.log 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.50423.log.gz 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.191630.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.128895.log 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.191630.log.gz 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.32565.log 2024-04-16T21:13:30.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.42517.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.42517.log.gz 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.128895.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.112365.log 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.128895.log.gz 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.32565.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.63214.log 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.32565.log.gz 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.67483.log 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.112365.log: 57.9% -- replaced with /var/log/ceph/c2-client.0.112365.log.gz 2024-04-16T21:13:30.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.63214.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.95372.log 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.63214.log.gz 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.67483.log: 46.3% -- replaced with /var/log/ceph/c2-client.0.67483.log.gz 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.0.log 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr: 97.0% -- replaced with /var/log/ceph/c2-client.0.178762.log.gz 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36341.log 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.95372.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.95372.log.gz 2024-04-16T21:13:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/rgw.c2.client.0.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.209577.log 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36341.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.187979.log 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36341.log.gz 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.209577.log: gzip -5 --verbose 45.5% -- /var/log/ceph/c2-client.0.160098.log 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.209577.log.gz 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.187979.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.71727.log 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.187979.log.gz 2024-04-16T21:13:30.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.160098.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.37419.log 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.160098.log.gz 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.71727.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.54711.log 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.71727.log.gz 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.37419.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.233067.log 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr: 43.2% -- replaced with /var/log/ceph/c2-client.0.37419.log.gz 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.54711.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.146650.log 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.54711.log.gz 2024-04-16T21:13:30.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233067.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.194063.log 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr: 9.2% -- replaced with /var/log/ceph/c2-client.0.233067.log.gz 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.146650.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.99020.log 46.1% -- replaced with /var/log/ceph/c2-client.0.146650.log.gz 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.194063.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.194063.log.gz 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.127679.log 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.99020.log: gzip -5 46.1% -- replaced with /var/log/ceph/c2-client.0.99020.log.gz 2024-04-16T21:13:30.093 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/c2-client.0.183720.log 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.127679.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.127679.log.gz 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.119034.log 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5/var/log/ceph/c2-client.0.183720.log: --verbose -- /var/log/ceph/c2-client.0.185547.log 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.183720.log.gz 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.119034.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.233103.log 51.7% -- replaced with /var/log/ceph/c2-client.0.119034.log.gz 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:30.094 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.185547.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.185547.log.gz 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.218991.log 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.233103.log: gzip -5 --verbose -- 8.2% -- replaced with /var/log/ceph/c2-client.0.233103.log.gz 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.116612.log 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.218991.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.218991.log.gz 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.109338.log 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.116612.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.126452.log 2024-04-16T21:13:30.095 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.116612.log.gz 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.109338.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.139961.log 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr: 75.5% -- replaced with /var/log/ceph/c2-client.0.109338.log.gz 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.126452.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.220207.log 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr: 88.5% -- replaced with /var/log/ceph/c2-client.0.126452.log.gz 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/c2-client.0.139961.log: -5 --verbose -- /var/log/ceph/c2-client.0.219599.log 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.139961.log.gz 2024-04-16T21:13:30.096 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.220207.log: gzip -5 --verbose -- 46.1% /var/log/ceph/c2-client.0.184936.log 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.220207.log.gz 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.219599.log: 45.5%gzip -5 --verbose -- /var/log/ceph/c2-client.0.89899.log 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.219599.log.gz 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36046.log 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.184936.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.184936.log.gz 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.89899.log: gzip -5 --verbose -- 46.1% -- replaced with /var/log/ceph/c2-client.0.89899.log.gz 2024-04-16T21:13:30.097 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/c2-client.0.228721.log 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.36046.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36046.log.gz 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.189195.log 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.228721.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.107521.log 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.228721.log.gz 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.189195.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.197214.log 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.189195.log.gz 2024-04-16T21:13:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.107521.log: gzip -5 --verbose -- 46.1% /var/log/ceph/c2-osd.0.log 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/c2-client.0.107521.log.gz 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.197214.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.197214.log.gz 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.106913.log 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.1.log 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.106913.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.167322.log 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr: 46.1% -- replaced with /var/log/ceph/c2-client.0.106913.log.gz 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.36111.log 2024-04-16T21:13:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.39320.log 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.167322.log: /var/log/ceph/c2-client.admin.36111.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.167322.log.gz 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.36111.log.gz 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.137438.log 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.35836.log 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.39320.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.39320.log.gz 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.137438.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.81480.log 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.admin.35836.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.35836.log.gz 2024-04-16T21:13:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.195279.log 2024-04-16T21:13:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.81480.log: 46.1% -- replaced with /var/log/ceph/c2-client.0.81480.log.gz 2024-04-16T21:13:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c2-client.0.195279.log: 41.9% -- replaced with /var/log/ceph/c2-client.0.195279.log.gz 2024-04-16T21:13:30.101 INFO:teuthology.orchestra.run.smithi119.stderr: 96.3% -- replaced with /var/log/ceph/c2-client.0.137438.log.gz 2024-04-16T21:13:30.102 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/c2-mgr.x.log.gz 2024-04-16T21:13:30.186 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/c1-mgr.x.log.gz 2024-04-16T21:13:31.447 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/c2-mon.a.log.gz 2024-04-16T21:13:31.947 INFO:teuthology.orchestra.run.smithi026.stderr: 91.6% -- replaced with /var/log/ceph/c1-mon.a.log.gz 2024-04-16T21:13:32.475 INFO:teuthology.orchestra.run.smithi026.stderr: 93.8% -- replaced with /var/log/ceph/rgw.c1.client.0.log.gz 2024-04-16T21:13:33.056 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/rgw.c2.client.0.log.gz 2024-04-16T21:13:37.810 INFO:teuthology.orchestra.run.smithi119.stderr: 93.3% -- replaced with /var/log/ceph/c2-osd.0.log.gz 2024-04-16T21:13:38.039 INFO:teuthology.orchestra.run.smithi151.stderr: 93.6% -- replaced with /var/log/ceph/c2-osd.4.log.gz 2024-04-16T21:13:38.145 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/c2-osd.5.log.gz 2024-04-16T21:13:38.610 INFO:teuthology.orchestra.run.smithi151.stderr: 93.5% -- replaced with /var/log/ceph/c2-osd.3.log.gz 2024-04-16T21:13:38.612 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-04-16T21:13:38.612 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m8.993s 2024-04-16T21:13:38.612 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m25.287s 2024-04-16T21:13:38.612 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.961s 2024-04-16T21:13:39.349 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/c2-osd.1.log.gz 2024-04-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi119.stderr: 93.9% -- replaced with /var/log/ceph/c2-osd.2.log.gz 2024-04-16T21:13:39.672 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-16T21:13:39.672 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m10.056s 2024-04-16T21:13:39.672 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m31.755s 2024-04-16T21:13:39.672 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m1.329s 2024-04-16T21:13:39.937 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/c1-osd.4.log.gz 2024-04-16T21:13:40.486 INFO:teuthology.orchestra.run.smithi069.stderr: 93.5% -- replaced with /var/log/ceph/c1-osd.5.log.gz 2024-04-16T21:13:40.581 INFO:teuthology.orchestra.run.smithi069.stderr: 93.5% -- replaced with /var/log/ceph/c1-osd.3.log.gz 2024-04-16T21:13:40.583 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-16T21:13:40.584 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m10.968s 2024-04-16T21:13:40.584 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m31.470s 2024-04-16T21:13:40.584 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m1.180s 2024-04-16T21:13:42.518 INFO:teuthology.orchestra.run.smithi026.stderr: 93.6% -- replaced with /var/log/ceph/c1-osd.0.log.gz 2024-04-16T21:13:44.614 INFO:teuthology.orchestra.run.smithi026.stderr: 94.2% -- replaced with /var/log/ceph/c1-osd.1.log.gz 2024-04-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi026.stderr: 94.3% -- replaced with /var/log/ceph/c1-osd.2.log.gz 2024-04-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-04-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m15.514s 2024-04-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m46.859s 2024-04-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m1.659s 2024-04-16T21:13:45.125 INFO:tasks.ceph:Archiving logs... 2024-04-16T21:13:45.125 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi026/log 2024-04-16T21:13:45.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T21:13:51.798 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi069/log 2024-04-16T21:13:51.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T21:13:56.694 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi119/log 2024-04-16T21:13:56.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T21:14:01.688 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi151/log 2024-04-16T21:14:01.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T21:14:05.721 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T21:14:05.731 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' 2024-04-16T21:14:05.736 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T21:14:05.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T21:14:05.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T21:14:05.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T21:14:05.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T21:14:05.892 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T21:14:05.902 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T21:14:05.902 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T21:14:05.906 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T21:14:05.909 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T21:14:05.911 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T21:14:05.920 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-04-16T21:14:05.923 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-16T21:14:05.925 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T21:14:05.925 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-04-16T21:14:05.925 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv01.front.sepia.ceph.com 3 7 377 2 +1836us[+1815us] +/- 60ms 2024-04-16T21:14:05.926 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 515 -1004us[-1060us] +/- 56ms 2024-04-16T21:14:05.926 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -1055us[-1076us] +/- 76ms 2024-04-16T21:14:05.926 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T21:14:05.926 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-16T21:14:05.928 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T21:14:05.929 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-16T21:14:05.929 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +1762us[+1756us] +/- 60ms 2024-04-16T21:14:05.929 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 70 -1056us[-1066us] +/- 63ms 2024-04-16T21:14:05.929 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -1093us[-1093us] +/- 76ms 2024-04-16T21:14:05.929 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T21:14:05.930 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-04-16T21:14:05.931 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T21:14:05.931 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-16T21:14:05.931 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +1723us[+1712us] +/- 59ms 2024-04-16T21:14:05.931 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 66 -1075us[-1079us] +/- 63ms 2024-04-16T21:14:05.932 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -1140us[-1140us] +/- 76ms 2024-04-16T21:14:05.932 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T21:14:05.934 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T21:14:05.935 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-04-16T21:14:05.935 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +1759us[+1751us] +/- 60ms 2024-04-16T21:14:05.935 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 139 -1052us[-1072us] +/- 62ms 2024-04-16T21:14:05.935 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -1090us[-1090us] +/- 76ms 2024-04-16T21:14:05.935 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T21:14:05.936 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T21:14:05.947 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T21:14:05.959 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T21:14:05.968 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T21:14:06.043 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T21:14:06.117 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T21:14:06.194 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T21:14:06.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T21:14:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:06.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T21:14:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:06.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T21:14:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:06.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T21:14:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:06.405 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T21:14:06.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T21:14:06.426 INFO:teuthology.task.internal:Duration was 7097.712771 seconds 2024-04-16T21:14:06.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T21:14:06.436 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T21:14:06.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T21:14:06.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T21:14:06.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T21:14:06.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T21:14:06.488 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T21:14:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T21:14:06.490 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T21:14:06.493 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T21:14:06.909 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T21:14:06.910 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-04-16T21:14:06.910 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T21:14:06.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-16T21:14:06.935 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T21:14:06.962 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-04-16T21:14:06.963 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T21:14:06.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-04-16T21:14:06.985 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T21:14:07.022 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T21:14:07.023 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T21:14:07.025 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T21:14:07.028 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T21:14:07.031 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T21:14:07.094 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T21:14:07.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T21:14:07.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T21:14:07.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T21:14:07.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T21:14:07.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T21:14:07.249 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T21:14:07.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T21:14:07.279 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T21:14:07.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T21:14:07.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T21:14:07.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T21:14:07.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.405 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-04-16T21:14:07.406 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-16T21:14:07.409 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-04-16T21:14:07.433 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-04-16T21:14:07.462 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:07.503 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:07.518 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:07.533 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T21:14:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:07.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T21:14:07.559 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T21:14:07.560 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi026 2024-04-16T21:14:07.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T21:14:07.620 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi069 2024-04-16T21:14:07.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T21:14:07.657 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi119 2024-04-16T21:14:07.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T21:14:07.696 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462/remote/smithi151 2024-04-16T21:14:07.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T21:14:07.733 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T21:14:07.733 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T21:14:07.736 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T21:14:07.738 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T21:14:07.740 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T21:14:07.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T21:14:07.799 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T21:14:07.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T21:14:07.810 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T21:14:07.810 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T21:14:07.813 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T21:14:07.815 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T21:14:07.818 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T21:14:07.826 INFO:teuthology.orchestra.run.smithi026.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 21:14 /home/ubuntu/cephtest 2024-04-16T21:14:07.828 INFO:teuthology.orchestra.run.smithi069.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 21:14 /home/ubuntu/cephtest 2024-04-16T21:14:07.831 INFO:teuthology.orchestra.run.smithi119.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 21:14 /home/ubuntu/cephtest 2024-04-16T21:14:07.839 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 21:14 /home/ubuntu/cephtest 2024-04-16T21:14:07.839 INFO:teuthology.orchestra.run.smithi151.stdout: 267187 0 srwxr-xr-x 1 root root 0 Apr 16 19:25 /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-16T21:14:07.839 INFO:teuthology.orchestra.run.smithi151.stdout: 267184 4 -rw-r--r-- 1 ceph root 42 Apr 16 19:25 /home/ubuntu/cephtest/url_file 2024-04-16T21:14:07.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T21:14:07.841 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T21:14:07.841 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 556, in task yield File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 364, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/rgw.py", line 269, in start_rgw rgwadmin(ctx, client, cmd=['gc', 'process', '--include-all'], check_status=True) File "/home/teuthworker/src/github.com_smanjara_ceph_9fbf23e1cbe3573736447d8751b80c9b42cadc24/qa/tasks/util/rgw.py", line 34, in rgwadmin proc = remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T21:14:07.856 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T21:14:07.868 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T21:14:07.880 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi119 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 gc process --include-all' 2024-04-16T21:14:07.888 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T21:14:07.922 DEBUG:teuthology.nuke:shortname: smithi026 2024-04-16T21:14:07.944 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T21:14:07.969 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-16T21:14:07.969 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T21:14:07.974 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.798557', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFeEZqyh9FanBgXANxWy0pvFnKNo2hNbUFcqc4b4YSTlDKxftRrW04MORnc+2d/6ALkEqGynnxTKwFB0BDqSt6E='} 2024-04-16T21:14:07.990 DEBUG:teuthology.nuke:shortname: smithi119 2024-04-16T21:14:07.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T21:14:07.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.799674', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM0TlHyXmVZ7jqn/rZpGycELmAJqs913ebWmQ2A0/heI3vGrv35mMKLp+X4G0DUqcRgFGEnaTPo3DjdbG/3Sps='} 2024-04-16T21:14:08.012 DEBUG:teuthology.nuke:shortname: smithi151 2024-04-16T21:14:08.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T21:14:08.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.800802', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBn36P5Qab1DiTDnz/Z06PhUxTXcAilRT0rHTDZBY3vuVopfxS/BCCXXESn35uqPsSYOEgNDMs9qS+wcQpSIaBQ='} 2024-04-16T21:14:08.029 INFO:teuthology.orchestra.console.smithi026:Power off 2024-04-16T21:14:08.029 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T21:14:08.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/smanjara-2024-04-16_19:05:08-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7658462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 19:06:27.796882', 'locked_by': 'scheduled_smanjara@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCVR1+NFSII3eGCZ2LQqmYNr8DPxw8UcAqsFGaEiPTDm+W8eOyUOvkV+Q0w306QFGyg9qJh1CvoFhvHYvh5aac='} 2024-04-16T21:14:08.071 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-04-16T21:14:08.071 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:08.101 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-16T21:14:08.102 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T21:14:08.128 INFO:teuthology.orchestra.console.smithi119:Power off 2024-04-16T21:14:08.129 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T21:14:08.144 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-16T21:14:08.144 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-16T21:14:08.144 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:08.157 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-04-16T21:14:08.157 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:08.178 INFO:teuthology.orchestra.console.smithi151:Power off 2024-04-16T21:14:08.178 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T21:14:08.191 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-16T21:14:08.192 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-16T21:14:08.205 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-04-16T21:14:08.205 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:08.240 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-04-16T21:14:12.145 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:12.193 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:12.205 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:12.241 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:12.259 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-04-16T21:14:12.307 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-16T21:14:12.320 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-16T21:14:12.355 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-04-16T21:14:16.261 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:16.308 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:16.321 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:16.356 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T21:14:16.376 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-04-16T21:14:16.423 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-04-16T21:14:16.434 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-16T21:14:16.469 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-04-16T21:14:16.477 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-04-16T21:14:16.524 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-04-16T21:14:16.534 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-16T21:14:16.570 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-04-16T21:14:16.674 INFO:teuthology.run:Summary data: description: rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi} duration: 7097.712770938873 failure_reason: rgw multisite test failures flavor: default owner: scheduled_smanjara@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b8fd33bc95804001836289ef42f77c90 status: fail success: false 2024-04-16T21:14:16.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T21:14:16.780 INFO:teuthology.run:FAIL